Is it possible to force Excel recognize UTF-8 CSV files automatically?

Question

I'm developing a part of an application that's responsible for exporting some data into CSV files. The application always uses UTF-8 because of its multilingual nature at all levels. But opening such CSV files (containing e.g. diacritics, cyrillic letters, Greek letters) in Excel does not achieve the expected results showing something like Г„/Г¤, Г–/Г¶. And I don't know how to force Excel understand that the open CSV file is encoded in UTF-8. I also tried specifying UTF-8 BOM EF BB BF, but Excel ignores that.

Is there any workaround?

P.S. Which tools may potentially behave like Excel does?

---

UPDATE

I have to say that I've confused the community with the formulation of the question. When I was asking this question, I asked for a way of opening a UTF-8 CSV file in Excel without any problems for a user, in a fluent and transparent way. However, I used a wrong formulation asking for doing it automatically. That is very confusing and it clashes with VBA macro automation. There are two answers for this questions that I appreciate the most: the very first answer by Alex https://stackoverflow.com/a/6002338/166589, and I've accepted this answer; and the second one by Mark https://stackoverflow.com/a/6488070/166589 that have appeared a little later. From the usability point of view, Excel seemed to have lack of a good user-friendly UTF-8 CSV support, so I consider both answers are correct, and I have accepted Alex's answer first because it really stated that Excel was not able to do that transparently. That is what I confused with automatically here. Mark's answer promotes a more complicated way for more advanced users to achieve the expected result. Both answers are great, but Alex's one fits my not clearly specified question a little better.

---

UPDATE 2

Five months later after the last edit, I've noticed that Alex's answer has disappeared for some reason. I really hope it wasn't a technical issue and I hope there is no more discussion on which answer is greater now. So I'm accepting Mark's answer as the best one.

Solution 1

Alex is correct, but as you have to export to csv, you can give the users this advice when opening the csv files:

1. Save the exported file as a csv
2. Open Excel
3. Import the data using Data-->Import External Data --> Import Data
4. Select the file type of "csv" and browse to your file
5. In the import wizard change the File_Origin to "65001 UTF" (or choose correct language character identifier)
6. Change the Delimiter to comma
7. Select where to import to and Finish

This way the special characters should show correctly.

Solution 2

The UTF-8 Byte-order marker will clue Excel 2007+ in to the fact that you're using UTF-8. (See this SO post).

In case anybody is having the same issues I was, .NET's UTF8 encoding class does not output a byte-order marker in a GetBytes() call. You need to use streams (or use a workaround) to get the BOM to output.

Solution 3

The bug with ignored BOM seems to be fixed for Excel 2013. I had same problem with Cyrillic letters, but adding BOM character \uFEFF did help.

Solution 4

It is incredible that there are so many answers but none answers the question:

"When I was asking this question, I asked for a way of opening a UTF-8 CSV file in Excel without any problems for a user,..."

The answer marked as the accepted answer with 200+ up-votes is useless for me because I don't want to give my users a manual how to configure Excel. Apart from that: this manual will apply to one Excel version but other Excel versions have different menus and configuration dialogs. You would need a manual for each Excel version.

So the question is how to make Excel show UTF8 data with a simple double click?

Well at least in Excel 2007 this is not possible if you use CSV files because the UTF8 BOM is ignored and you will see only garbage. This is already part of the question of Lyubomyr Shaydariv:

"I also tried specifying UTF-8 BOM EF BB BF, but Excel ignores that."

I make the same experience: Writing russian or greek data into a UTF8 CSV file with BOM results in garbage in Excel:

Content of UTF8 CSV file:

Colum1;Column2
Val1;Val2
Авиабилет;Tλληνικ

Result in Excel 2007:

A solution is to not use CSV at all. This format is implemented so stupidly by Microsoft that it depends on the region settings in control panel if comma or semicolon is used as separator. So the same CSV file may open correctly on one computer but on anther computer not. "CSV" means "Comma Separated Values" but for example on a german Windows by default semicolon must be used as separator while comma does not work. (Here it should be named SSV = Semicolon Separated Values) CSV files cannot be interchanged between different language versions of Windows. This is an additional problem to the UTF-8 problem.

Excel exists since decades. It is a shame that Microsoft was not able to implement such a basic thing as CSV import in all these years.

---

However, if you put the same values into a HTML file and save that file as UTF8 file with BOM with the file extension XLS you will get the correct result.

Content of UTF8 XLS file:

<table>
<tr><td>Colum1</td><td>Column2</td></tr>
<tr><td>Val1</td><td>Val2</td></tr>
<tr><td>Авиабилет</td><td>Tλληνικ</td></tr>
</table>

Result in Excel 2007:

You can even use colors in HTML which Excel will show correctly.

<style>
.Head { background-color:gray; color:white; }
.Red  { color:red; }
</style>
<table border=1>
<tr><td class=Head>Colum1</td><td class=Head>Column2</td></tr>
<tr><td>Val1</td><td>Val2</td></tr>
<tr><td class=Red>Авиабилет</td><td class=Red>Tλληνικ</td></tr>
</table>

Result in Excel 2007:

In this case only the table itself has a black border and lines. If you want ALL cells to display gridlines this is also possible in HTML:

<html xmlns:x="urn:schemas-microsoft-com:office:excel">
    <head>
        <meta http-equiv="content-type" content="text/plain; charset=UTF-8"/>
        <xml>
            <x:ExcelWorkbook>
                <x:ExcelWorksheets>
                    <x:ExcelWorksheet>
                        <x:Name>MySuperSheet</x:Name>
                        <x:WorksheetOptions>
                            <x:DisplayGridlines/>
                        </x:WorksheetOptions>
                    </x:ExcelWorksheet>
                </x:ExcelWorksheets>
            </x:ExcelWorkbook>
        </xml>
    </head>
    <body>
        <table>
            <tr><td>Colum1</td><td>Column2</td></tr>
            <tr><td>Val1</td><td>Val2</td></tr>
            <tr><td>Авиабилет</td><td>Tλληνικ</td></tr>
        </table>
    </body>
</html>

This code even allows to specify the name of the worksheet (here "MySuperSheet")

Result in Excel 2007:

Solution 5

We have used this workaround:

1. Convert CSV to UTF-16 LE
2. Insert BOM at beginning of file
3. Use tab as field separator

View more solutions

Comments

[Lyubomyr Shaydariv]

Thanks. Unfortunately, this is the suggestion for really advanced Excel users, and what I'm aiming for, asking this question, is making Excel do it itself automatically.

[Lyubomyr Shaydariv]

Thanks for the post link (I didn't know about the UTF-16 case).

[Husky]

Thanks! I tried all the other answers in this thread, but converting to UTF-8 simply didn't work. When i tried UTF-16 with BOM it worked instantly.

[Lyubomyr Shaydariv]

It's an old question, but I'm happy that people still answer it. Thank you. :)

[Steve Jessop]

@LyubomyrShaydariv: since I answered this question, a colleague told me that the latest Excel does identify UTF-8 CSV files provided they have an initial BOM. So in a few years, when everyone in the world (or anyway, everyone we deliver the files to) is on that version of Excel or better, I might be able to change my encoding :-)

[Galdo]

Since I added \uFEFF at the beginning of my CSV file (generated in Java), Excel is able to open them correctly! Thx

[Abdullah]

that's solved my issue too , in php it looks like this: $utf8_with_bom = chr(239) . chr(187) . chr(191) . $csvText;

[Lyubomyr Shaydariv]

Which Excel version do you use?

[Auberon Vacher]

Brilliant, simply works and solves the problem (at least for me)

[nathan hayfield]

@updola thanks so much for this comment, saved me hours

[Ami]

Inserting a UTF-8 BOM seems to be the way to go. After that it's transparent.

[Veverke]

Nope, because the problem is to have Excel open the .csv file as a csv file. I mean, I have a process where I generate csv files. The user then would only be required to open the files. Here he/she is required to do more than that.

[Veverke]

Strange, for me it does not work. Turns the characters into chinese symbols... (they are supposed to be hebrew)

[Nelson]

UTF-8, by definition, does not use nor should use the BOM characters. The way Excel reads BOM screws up the advantage of UTF-8 over Unicode, which is backwards compatibility with ASCII. Adding the BOM will make Excel work, but break other proper UTF-8/ASCII file reads.

[Mark Ransom]

@Nelson the BOM shouldn't be necessary, but it is. Microsoft values backwards compatibility and Windows programs have always assumed a file would be encoded with a MBCS character set. Changing that assumption would break too many legacy systems. It would be nice if you could specify UTF-8 as your MBCS encoding, but they've never gotten that to work consistently.

[Mark Ransom]

@Veverke if you generate files on a regular basis this isn't the best solution. But if it's a one-off, this answer is perfect. It works because Notepad will put the BOM at the beginning of a UTF-8 or UTF-16 file.

[Nelson]

@MarkRansom but that's not UTF-8 specification at all. By definition, UTF-8 is suppose to be compatible with standard ASCII at the binary level, which means no characters above 127. Everything else under is suppose to be straight up "standard" with no special characters. What Microsoft should be doing is DEFAULT UTF-8 if there are no BOM, not default to their MS craptastic encoding.

[Mark Ransom]

@Nelson no, what Microsoft "should" be doing is making sure that things that used to work, continue to work. And that's exactly what they're doing. I know the spec says it isn't necessary, and in a perfect world it wouldn't be. But that's not the world we live in.

[Nelson]

@MarkRansom Microsoft does an atrocious job of keeping things working. Just look at this thorough report on the Excel inconsistencies in dealing with BOM: wiki.scn.sap.com/wiki/display/ABAP/…

[bgusach]

Thank you very much. This is way better than trying to guess what encoding excel fancies.

[Panagiotis Kanavos]

@Nelson Wikipedia seems to think that UTF-8 does have a BOM The UTF-8 representation of the BOM is the byte sequence 0xEF,0xBB,0xBF. Anyway, CSV is an imprecise format. Moreover, the link you provided shows limited knowledge of non-US cultures - ; is the list separators for several European languages because , is the decimal separator and can't be used as a list separator. If one wants fidelity, one should use the XLSX format - it's just a compressed XML with a very well defined schema.

[Henno]

I had chinese characters in Excel when I saved in "UTF-16 BE with BOM". When I tried "UTF-16 LE with BOM" it worked! I used Sublime Text for testing.

[Sébastien]

I use Notepad++ to easily convert the .csv from UTF-8 to UTF-8 with BOM

[Alveoli]

This should be the new accepted answer. Works like a dream!

[Steve]

@elmue care to elaborate a little? Surely outputting CSV using the correct encoding to start with ensures there's no compatibility issues with Excel later in the workflow?

[Elmue]

The code is wrong because you do not need output.Close() if you have a using() statement. Apart from that Excel CSV inport is very primitive. I would not use it at all. If you want to import in Excel use a HTML table instead and open it in Excel.

[multidynamic]

For Excel for Mac 2011 I was successful with UTF-16 little endian csv files

[LDW]

Thanks, this solution is excellent for giving my users a unicode csv that they can open in Excel.

[user369142]

Yes - best solution. Lots of confusing posts around setting headers in http responses. This solves the problem. Same can be seen when opening the file in notepad and saving right back down using the UTF-8 option. Also adds the byte-order marker.

[user369142]

Solution from the post referenced: private byte[] AddEncodingPreamble(byte[] byteArray, Encoding encoding) { return encoding.GetPreamble().Concat(byteArray).ToArray(); }

[Steve]

Thanks for clarifying @Elmue - unfortunately I've run into a number of scenarios (including files across network drives) where a close is required explicitly as well as the disposal. Not found a reason why as I think I remember seeing a call to Close() in the Dispose stack but there we go. You're also incorrect in your statement about CSV import being primitive as it does not need the inefficient HTML approach you've suggested. In fact creating additional programmatic steps to take plain-text data, transform into HTML then pull into Excel seems counter intuitive

[Steve]

continued... However, different scenarios have different requirements and my example correctly shows how to specify the encoding as the OP requested

[Elmue]

Please read the original question again: "I also tried specifying UTF-8 BOM EF BB BF, but Excel ignores that." I tested the same and come to the same result: Excel does NOT recognize UTf8 BOM. Try it! Write a CSV file and put greek or russian charcters into it. Then open that in Excel and you will get garbarge. So your answer does not solve anything.

[Elmue]

CSV is primitive because it depends on your settings in control panel if comma or semicolon is used as separator which is the most stupid design. A file with comma is detected correctly on some computers but on other computers not. If the data itself contains commas you must quote them. And UTF8 does NOT work. All these problems do not appear loading a HTML table into Excel.

[Elmue]

The programmatic steps to create an HTML table are ridiculous. Save the following into a file with the extension XLS and save it as UTF8 with BOM and open it in Excel: <table><tr><td>Colum1</td><td>Column2</td></tr> <tr><td>Авиабилет</td><td>Tλληνικ</td></tr></table>

[Elmue]

You did not read the question! Lyubomyr Shaydariv wrote: " I also tried specifying UTF-8 BOM EF BB BF, but Excel ignores that."

[Elmue]

You did not read the question! Lyubomyr Shaydariv wrote: " I also tried specifying UTF-8 BOM EF BB BF, but Excel ignores that."

[Lyubomyr Shaydariv]

And it's also incredible that people still answer it. :) However, I would argue that CSV is dead: CSV is a deadly simple plain streaming format, and, when I was on that project, we used it really heavily for components interaction too. I would never use HTML format because of its too-representive nature, and I couldn't ever make the users of that project export data to HTML just to make it Excel-friendly. Excel sucked at UTF-8 CSV, and I have no clue what's changed 5 years since then. But that comma-, semi-colon- and tab-delimited files is true madness, I guess.

[Elmue]

It is not incredible that people still answer your question because Google brings them here: All with the same question: How do I get stupid Excel to do what I want? Enter 3 words into Google: "excel csv utf8" and your question is the FIRST result world wide. You wrote a bestseller!

[Elmue]

By the way: Why did you mark an answer as accepted answer which does not answer what you wanted to know ? You wrote: "When I was asking this question, I asked for a way of opening a UTF-8 CSV file in Excel without any problems for a user,"

[Lyubomyr Shaydariv]

Just googled for "excel csv utf8", and I didn't know the question is at the top. :) Regarding accepting the answer: I marked the answer as the best one just because it really answered how it could be done in Excel. Maybe I was too strict when I said "without any problems". I think that import wizards are fine for me being a programmer, but it's shame that Excel required (or still requires) to do import rather than just open.

[StriplingWarrior]

@Elmue: I figured maybe (like me) he thought he was saving it with a byte-order marker, but wasn't because some library didn't work how he thought it did. I found it worthwhile to add this answer, and evidently a number of other people have found it useful too. At any rate, it seems like a better answer than saying not to use CSV, or instructing users how to open the file in some bizarre way.

[zax]

This is working code. If you write out UTF-8 files from .Net applications with this lead, Excel will recognize the file as UTF8.

[Steve]

I have read the OP - I've supplied a way to create the CSVs via Excel that solve the problem to start with, and also open forcing the specific encoding the OP has asked for. Seems to work just fine with Excel 2013 and .NET 4 so I'm guessing there's a version difference in the Office type libraries between our experiments? The HTML approach is inefficient and I'm still surprised you're trying to extol it's virtues - what you're describing as plaint text anomalies is down to the encoding. Try importing as plain text in a simple using {} and save as a new file forcing the encoding. It works.

[Elmue]

As I already said your approach does not work in Excel 2007 and as you never know which Excel versions your users use, it is not a generic solution.

[Elmue]

No. The cause that it did not work for him is not that he was unable to save a simple 3 bytes to a file. The cause that it does not work is that at least Excel 2007 IGNORES the BOM completely.

[Elmue]

May be in YOUR Excel it works. But at least in Excel 2007 the BOM is ignored completely. So if you intend to write a code that works on all Excel versions your code is not usefull.

[StriplingWarrior]

Since you've taken the time to point out how none of the other answers here cause Excel to open a CSV file correctly, it's worth mentioning that this answer doesn't do that either. The originally-accepted answer (which was deleted by a moderator) recommended exporting the data to an Excel file directly, or exporting an HTML file and allowing users to open the file in Excel. Either of those solutions are less hacky than saving an HTML file with an XLS file extension, but no answer has been proposed that actually answers the question.

[StriplingWarrior]

@Elmue: I'll grant that it's an imperfect solution. If there were any way to automatically make this work for older versions of Excel, I'd prefer them. But recognizing that there is no solution to the problem, and at least 69 people have found this answer to be helpful so far, I'm going to leave it here. Thank you for your observations.

[Elmue]

@Warrior: "..but no answer has been proposed that actually answers the question." This is not correct. I HAVE answered that question: At least with Excel 2007 it is NOT POSSIBLE. This is the definite answer. If Excel ignores the UTF8 BOM and interprets data as ANSI, there is no way to import for example greek or russian text into Excel via CSV by a simple double click on the file (which was the question: "without any problems for a user"). What I propose is a working solution that functions with ALL Excel versions with a double click and does not require additional steps in the menus of Excel.

[Sharunas Bielskis]

I tried to use this code snippet but it doesn't work for me. I use Excel 2013.

[Cyril Duchon-Doris]

Hell, like I have a choice to export as something else than CSV when it's the only button I can see in my interface... I agree that it is quite rubbish though, but we have to deal with it.

[Nakilon]

\FE\FF makes Mac OS Excel 2016 the whole file as Japanese hieroglyphs.

[srgstm]

I use plain old Notepad to easily convert .csv from UTF-8 to UTF-8 with BOM. Just open and save.

[Thomas Andrews]

The Mac version of Excel seems to still ignore the BOM. (Excel for Mac 2011.)

[Terence Chow]

Is there anyway to preserve the original excel cell lines instead of showing a blank page?

[Elmue]

I updated my answer with an additional example to show all gridlines.

[RandomGuy]

Thanks this fixed it. I have just converted the csv ';' separated to UTF-BOM in Notepad++ and know excels opens it correctly.

[Steve]

Seemed to work fine across 2007 and 2013 so not sure where you're going wrong

[Elmue]

An UTF8 BOM definitely does NOT work with Excel 2007. It is ignored.

[Jeremy]

Not only does this not answer the question, it's completely wrong to state CSV is "legacy" and "dead". Just because you doesn't see the use for it, does not mean it's dead. Also, you clearly don't understand it's history or how widely used it is today. This is better than XML (due to smaller packet sizes) and the issue here is how Microsoft simply does not follow a standard for the UTF8 format. The correct answer would be to use open office and not Microsoft as Microsoft has never been good at encoding in the first place.

[Elmue]

You are partly correct. I updated my answer. It is not the fault of CSV itself. It is the fault of Microsoft. But the quesion was about Microsoft Excel so you surely do not answer the original question if you recommend to use Open Office (which has other bugs which may even be worse). You surely will not write to the recipients of your emails: "Please install Open Office to see the attached CSV file correctly". There must be better solutions. And my answer is a far better solution than asking people to install Open Office just to open an CSV file.

[EFC]

Exactly what was needed! Here is a pointer to how to implement this in PHP stackoverflow.com/questions/25686191/…

[Dherik]

Works for me too. The BOM character needs to be at the beginning of the file.

[marsze]

.NET does output a BOM if you employ the streams correctly, see here: stackoverflow.com/questions/5266069/…

[StriplingWarrior]

@marsze: Thanks for pointing that out. There is another answer on the link that I posted which uses streams. I guess the lesson is that GetBytes() was never intended to be used to produce an entire file's contents by itself: we're supposed to use streams and let them call into GetPreamble() and GetBytes() and such.

[marsze]

@StriplingWarrior Very true, a BOM belongs only at the start of streams. What I like about that other answer (except that it's basically just another way of writing it: string.ToBytes(encoding) instead of encoding.ToBytes(string)) is that you can pass new UTF8Encoding(true) or new UTF8Encoding(false) for controlling the output of a BOM or not.

[pakalbekim]

Did what you said, didn't work at all. Converting/encoding, setting character sets, nothing worked in my case. Import file to google sheets, download as csv, worked flawlessly. Your welcome from 2045

[user2925795]

Works like a charm. ((:

[cobp]

This is the workaround for excel which could not recognize utf chareacters in csv files. So this correction to fix excel issue.

[fjsj]

Confirmed Excel for Mac 2011 reads the BOM as a wrong char in the first column and ignores the file is utf-8. So this solution won't work for that version.

[NameOfTheRose]

This worked for Excel 2007 Greek Version, with comma seperators. Thank you.

[EK0]

Just tested this with Excel for Mac (v 16.16) and it does recognize it, and reads the rest of the UTF-8 characters correctly. Without it you still get garbage.

[rahul shukla]

Changed encoding to 'UTF-16le' in fs.createWriteStream(fileName,{encoding: 'UTF-16le'}) and excel opened the CSV file correctly

[BiAiB]

when running the sed command, I get: "sed: invalid option -- 'I'", I think it should be 'i'

[Igor Vuković]

I had to convert it with Encoding -> Cinvert ti UTF-8-BOM Excel version 2016

[lakeweb]

This is absolutely the best answer. I write my csv to a std::ofstream. So if I first: file << "\xef\xbb\xbf";. I can then follow with utf8 data and it opens fine in excel 2007. Thanks!

[Will Martin]

Excel Office 365 correctly reads CSV files with a UTF-8 BOM. One side note is that if you have a TSV (tab-separated values) file, it will recognize the BOM, but will not recognize the tabs as separators. Removing the BOM makes it recognize the tabs correctly, but then multi-byte characters are mangled. As a workaround, import the TSV in LibreOffice Calc and save it as a UTF-8 encoded CSV.

[S.Serpooshan]

can you explain more exactly how to convert to UTF-16 LE and which BOM to use?

[Tom]

This is a workaround and does not answer the question. The question was how we can create csv files and excel automatically selects the right encoding.

[MSC]

Given the strong views on either side, I'd say that there should be an option in the Excel settings for how the user prefers to treat CSVs that are opened directly. I would like to be able to double-click a CSV and have Excel default to assuming it's Unicode.

[Simon East]

My testing with Excel 2007 (yes, quite an old version, I know) showed that a CSV with BOM did NOT open correctly when opening the CSV into Excel directly from Windows Explorer. However opening it within Excel (File > Open) launched the "Import Wizard" which did have UTF-8 set correctly by default, but required me to manually specify that it was comma separated. Will have to use that long-winded method each time I open the file.

[Newbielp]

It worked well, only I had to save with UTF-8 BOM. FInally, a way to work around.