How to make sure my created filedownload is UTF-8? (and not UTF-8 without BOM)

Question

i've made a download function to download messages to a CSV file (code is below). Now when i open it up in notepad or notepad++ i see this:

é NY ø ╬ ║► ░ ê ö

(and that is what is in the database btw)

Now, when i open it up in Ms-Excel it shows this:

é NY ø ╬ ║► ░ ê ö

When i open it up in notepad++, it says it's encoded in 'UTF8 without BOM'. When i encode it (in notepad++) to UTF-8, all goes well (that is, Excel shows the right chars too)

But how can i make sure that the file i create from my code is UTF-8?

This is my code:

public ActionResult DownloadPersonalMessages()
{    
    StringBuilder myCsv = new StringBuilder();
    myCsv.Append(new DownloadService().GetPersonalMessages());

    this.Response.ContentType = "text/csv";
    Response.AddHeader("content-disposition", "attachment; filename=PersonalMessages.csv");
    Response.ContentEncoding = Encoding.UTF8;
    Response.Write(myCsv.ToString());
    Response.Flush();
    Response.HeaderEncoding = Encoding.UTF8;
    return Content("");
}

Edit:

my function now returns a ByteArray with this conversion

UTF8Encoding encoding = new UTF8Encoding();
return encoding.GetBytes(str);

and my download is now this:

Response.AddHeader("Content-Disposition", "attachment; filename=PersonalMessages.csv");
return File(new DownloadService().GetPersonalMessages(), "text/csv");

Solution 1

You might want to try using the UTF8Encoding class. The constructor has a parameter that determines if it should provide the BOM or not. You'll probably have to use the GetBytes-method and write the string as a series of bytes in the response, and not convert it back into a .net string object.

Solution 2

Zareth's answer helped the OP, but it didn't actually answer the question. Here's the correct solution, from this other post:

public ActionResult Download()
{
    var data = Encoding.UTF8.GetBytes("some data");
    var result = Encoding.UTF8.GetPreamble().Concat(data).ToArray();
    return File(result, "application/csv", "foo.csv");
}

The byte-order mark (while not technically required for UTF8) clues certain programs (e.g. Excel >2007) in to the fact that you're using UTF8. You have to manually include it via the GetPreamble() method.

Solution 3

You could simplify your code a little:

public ActionResult DownloadPersonalMessages()
{
    StringBuilder myCsv = new StringBuilder();
    myCsv.Append(new DownloadService().GetPersonalMessages());
    Response.AddHeader("Content-Disposition", "attachment; filename=PersonalMessages.csv");
    return File(Encoding.UTF8.GetBytes(myCsv.ToString()), "text/csv");
}

As far as the UTF-8 encoding is concerned I am afraid the problem might be in this GetPersonalMessages method. You might need to return a stream or byte array which you could directly return as file.

Comments

[Michel]

will try that. And the code became so large by trying all different things :)

[Michel]

hmm, when i return a byte array, will that make it UTF8? Wll, i'll give it a try. The GetPersonalMessgages() function reads a nvarchar field from the database btw

[Michel]

Thanks, changed my code to System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding(true); but still no luck...

[Michel]

you did set me on the right track by the getbytes and all. I was on the wrong track because notepad++ said it was a utf8 not bom file, and when i converted it to utf8, it worked. But of course utf8 was not a requirement on my side. I've tried it with a unicodeencoding and tadaa! it worked. Thanks for your thinking.

[StriplingWarrior]

FYI: For some reason the constructor parameter doesn't actually cause it to output the BOM. You have to prepend the result of .GetPreamble() manually.

[Ofir]

The MIME type should be: text/csv, see here (and if you want to be more precise then use: text/csv; charset=utf-8, see here).

[StriplingWarrior]

@fini007: I think it depends on the intent. According to this SO post, "If a server says 'This data is of type text/csv' the client can understand that can render that data internally, while if the server says 'This data is of type application/csv' the client knows that it needs to launch the application that is registered on the OS to open csv files." If you want the browser to launch Excel, then "application/csv" is probably better. Of course, if you're using content-disposition to make the file download then that's kind of moot.