C# Convert string from UTF-8 to ISO-8859-1 (Latin1) H

Question

I have googled on this topic and I have looked at every answer, but I still don't get it.

Basically I need to convert UTF-8 string to ISO-8859-1 and I do it using following code:

Encoding iso = Encoding.GetEncoding("ISO-8859-1");
Encoding utf8 = Encoding.UTF8;
string msg = iso.GetString(utf8.GetBytes(Message));

My source string is

Message = "ÄäÖöÕõÜü"

But unfortunately my result string becomes

msg = "�ä�ö�õ�ü

What I'm doing wrong here?

Solution 1

Use Encoding.Convert to adjust the byte array before attempting to decode it into your destination encoding.

Encoding iso = Encoding.GetEncoding("ISO-8859-1");
Encoding utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(Message);
byte[] isoBytes = Encoding.Convert(utf8, iso, utfBytes);
string msg = iso.GetString(isoBytes);

Solution 2

I think your problem is that you assume that the bytes that represent the utf8 string will result in the same string when interpreted as something else (iso-8859-1). And that is simply just not the case. I recommend that you read this excellent article by Joel spolsky.

Solution 3

Try this:

Encoding iso = Encoding.GetEncoding("ISO-8859-1");
Encoding utf8 = Encoding.UTF8;
byte[] utfBytes = utf8.GetBytes(Message);
byte[] isoBytes = Encoding.Convert(utf8,iso,utfBytes);
string msg = iso.GetString(isoBytes);

Solution 4

You need to fix the source of the string in the first place.

A string in .NET is actually just an array of 16-bit unicode code-points, characters, so a string isn't in any particular encoding.

It's when you take that string and convert it to a set of bytes that encoding comes into play.

In any case, the way you did it, encoded a string to a byte array with one character set, and then decoding it with another, will not work, as you see.

Can you tell us more about where that original string comes from, and why you think it has been encoded wrong?

Solution 5

Seems bit strange code. To get string from Utf8 byte stream all you need to do is:

string str = Encoding.UTF8.GetString(utf8ByteArray);

If you need to save iso-8859-1 byte stream to somewhere then just use: additional line of code for previous:

byte[] iso88591data = Encoding.GetEncoding("ISO-8859-1").GetBytes(str);

View more solutions

Comments

[mistertodd]

All strings in .NET internally store the strings using unicode characters. There is no notion of a String being "windows-1252", "iso-8859-1", "utf-8", etc. Are you trying to throw away any characters in your string that do not have a representation in the Windows-1252 code page?

[Tom Blodget]

@IanBoyd Actually, a String is a counted sequence of UTF-16 code units. (Unfortunately, the term Unicode has been misapplied in Encoding.Unicode and in the Win32 API. Unicode is a character set, not an encoding. UTF-16 is one of several encodings for Unicode.)

[StuS]

You make incorrect action: you make byte array in utf8 encoding, but read them by iso decode. If you want make string with encoded symbols it simple call string msg = iso.GetString(iso.GetBytes(Message));

[Rick James]

That's called Mojibake.

[Qwertie]

I guess what Daniil is saying is that Message was decoded from UTF-8. Assuming that part worked correctly, converting to Latin-1 is as simple as byte[] bytes = Encoding.GetEncoding("ISO-8859-1").GetBytes(Message). Then, like StuS says, you can convert the Latin-1 bytes back to UTF-16 with Encoding.GetEncoding("ISO-8859-1").GetString(bytes)

[Daniil Harik]

It's coming directly from App.config and I was thinking it's UTF8 by default. Thank You!

[Lasse V. Karlsen]

The encoding of that file might impact how the file gets interpreted, so I would look at that.

[Pantelis]

Excellent article indeed and with a sense of humor! I was facing an encoding issue today at work and this helped me out.

[user1237131]

why i am getting same utf-8 message?in place of message i passed string message=<name>sdjfhsjdf</name>.then same output getting in msg varieable.how to get latin data ?

[Spawnrider]

This works for me. Remember to include System.Text namespace.

[Jon Coombs]

Correct me if I'm wrong, but my understanding is that, while technically it "isn't in any particular encoding", a .NET string is a byte array that corresponds precisely to a UTF-16 file, byte for byte (excluding the BOM). It even uses surrogates in the same way (which seems like an encoding trick). Of course, you generally want to store files as UTF-8 but process the data in memory as 16-bit. (Or 32-bit, to avoid the complexity of surrogate pairs, though I'm not sure if that's really feasible.)

[Tertium]

Encoding.Convert throws fallback exception while converting if string has non-iso characters

[Admin]

The one liner is Encoding.GetEncoding("ISO-8859-1").GetString(Encoding.Conver‌​t(Encoding.UTF8, Encoding.GetEncoding("ISO-8859-1"), Encoding.UTF8.GetBytes(myString)))

[Nyerguds]

This is clearly the most straightforward answer. The problem in the code is indeed that the author seems to assume that a String in C# can already be stored "using" a certain encoding, which simply isn't true; they're always UTF16 internally.

[Sander A]

Fully agree. When you already have UTF-16, it is quite hard to make that into correct encoding, because when you converted byte array to string with wrong encoding there is already loss of information.

[goamn]

If you are creating the string yourself inside C#/.Net, then this code is not 100% correct, you need to encode from UTF-16 (which is the variable "Unicode"). Because this is the default. So UTF8 in the code above has to be changed to Unicode.

[Fuat]

I recommend to use this: Encoding iso = Encoding.GetEncoding("ISO-8859-9"); Because turkish encoding covers allmost all alphabet extended from Latin.

[Nyerguds]

You know, isoBytes is also just iso.GetBytes(Message);. There is no need to convert anything here. In fact, you can just skip all that and say string msg = Message. There's no real point to any of these conversions, since the start and end are both just a .Net String. And text encodings are irrelevant on a .Net String as long as you don't need to handle it as bytes.

[Nyerguds]

@JonCoombs I don't think that's correct. UTF-16 works with expanding opcodes. The .Net strings just use an array of 16-bit code points, without any expansion. As far as I know it only supports the 0000-FFFF range.