Why is this appearing in my c# strings: Â£

c# .net asp.net localization string

10,361

Solution 1

The default character set of URLs when used in HTML pages and in HTTP headers is called ISO-8859-1 or ISO Latin-1.

It's not the same as UTF-8, and it's not the same as ASCII, but it does fit into one-byte-per-character. The range 0 to 127 is a lot like ASCII, and the whole range 0 to 255 is the same as the range 0000-00FF of Unicode.

So you can generate it from a C# string by casting each character to a byte, or you can use Encoding.GetEncoding("iso-8859-1") to get an object to do the conversion for you.

(In this character set, the UK pound symbol is 163.)

Background

The RFC says that unencoded text must be limited to the traditional 7-bit US ASCII range, and anything else (plus the special URL delimiter characters) must be encoded. But it leaves open the question of what character set to use for the upper half of the 8-bit range, making it dependent on the context in which the URL appears.

And that context is defined by two other standards, HTTP and HTML, which do specify the default character set, and which together create a practically irresistable force on implementers to assume that the address bar contains percent-encodings that refer to ISO-8859-1.

ISO-8859-1 is the character set of text-based content sent via HTTP except where otherwise specified. So by the time a URL string appears in the HTTP GET header, it ought to be in ISO-8859-1.

The other factor is that HTML also uses ISO-8859-1 as its default, and URLs typically originate as links in HTML pages. So when you craft a simple minimal HTML page in Notepad, the URLs you type into that file are in ISO-8859-1.

It's sometimes described as "hole" in the standards, but it's not really; it's just that HTML/HTTP fill in the blank left by the RFC for URLs.

Hence, for example, the advice on this page:

URL encoding of a character consists of a "%" symbol, followed by the two-digit hexadecimal representation (case-insensitive) of the ISO-Latin code point for the character.

(ISO-Latin is another name for IS-8859-1).

So much for the theory. Paste this into notepad, save it as an .html file, and open it in a few browsers. Click the link and Google should search for UK pound.

<HTML>
  <BODY>
    <A href="http://www.google.com/search?q=%a3">Test</A>
  </BODY>
</HTML>

It works in IE, Firefox, Apple Safari, Google Chrome - I don't have any others available right now.

Solution 2

AppendAllText is writing out the text in UTF-8.

What are you using to look at it? Chances are it's something that doesn't understand UTF-8, or doesn't try UTF-8 first. Tell your editor/viewer that it's a UTF-8 file and all should be well. Alternatively, use the overload of AppendAllText which allows you to specify the encoding and use whichever encoding is going to be most convenient for you.

EDIT: In response to your edited question, the reason it fails when you encode with ASCII is that £ is not in the ASCII character set (which is Unicode 0-127).

URL encoding is also using UTF-8, by the looks of it. Again, if you want to use a different encoding, specify it to the HttpUtility.UrlEncode overload which accepts an encoding.

10,361

Author by

Tim

Updated on July 19, 2022

Comments

Tim almost 2 years
I have a a string in c# initialised as follows:
```
string strVal = "£2000";
```
However whenever I write this string out the following is written:

Â£2000

It does not do this with dollars.

An example bit of code I am using to write out the value:
```
System.IO.File.AppendAllText(HttpContext.Current.Server.MapPath("/logging.txt"), strVal);
```
I'm guessing it's something to do with localization but if c# strings are just unicode surely this should just work?

CLARIFICATION: Just a bit more info, Jon Skeet's answer is correct, however I also get the issue when I URLEncode the string. Is there a way of preventing this?

So the URL encoded string looks like this:

"%c2%a32000"

%c2 = Â %a3 = £

If I encode as ASCII the £ comes out as ?

Any more ideas?
Jon Skeet about 15 years

Do you have a source for the info about the default character encoding of URLs? I thought it was one of those annoyingly unspecified things. I'm not disputing it, but I'd like to see where it's specified as the default. Btw, you can also use Encoding.GetEncoding(28591) to get ISO-8859-1.
Tim about 15 years

I'm interested why URLEncode doesn't do this conversion automatically. As the strings in C# are UTF-8 so it would be fairly intuitive to have the URLEncode methods accept such a string and encode it correctly. Rather than falling over unless I manually specify the correct encoding?
Jon Skeet about 15 years

@tsaunders: No, the strings aren't UTF-8. They're UTF-16. UrlEncode is using UTF-8 by default. As I said, I don't think the URL spec specifies the encoding to be used for UrlEncoded characters, which is a pain.
Daniel Earwicker about 15 years

@Jon Skeet - added some background; it's not in the URL spec, but it is in the HTML and HTTP specs.
Daniel Earwicker about 15 years

Are C# strings in UTF-16? Note that 16-bit characters was enough for Unicode version 1, but later versions require more. The 16-bit subset is called UCS-2. UTF-16 can use multiple 16-bit codes to represent one character, just as UTF-8 does with 8-bit codes.
Daniel Earwicker about 15 years

i.e. if C# strings are in UTF-16, then the 'char' data type is possibly misnamed! :) It might only be half a character.
Daniel Earwicker about 15 years

It looks like (as with Win32 generally) it depends on what API the strings are passed to, and how Windows is configured. catch22.net/tuts/neatpad/11
Jon Skeet about 15 years

.NET encodes non-BMP characters using surrogate pairs - i.e. it uses UTF-16. Fortunately most C# developers don't usually come across this. This is exactly what Java does too.
Jon Skeet about 15 years

From the C# spec, section 1.3: "The char type represents a UTF-16 code unit, and the string type represents a sequence of UTF-16 code units."
Jon Skeet about 15 years

@Earwicker: I've been looking in RFC 2616 and 2396 btw, and I still can't see anything defining ISO-8859-1 as the default URL encoding. Could you give a spec/RFC reference?
Jon Skeet about 15 years

Where do they say that though? I'm failing to find any reference to ISO-8859-1 in any spec I look at... Who is "they"?
Daniel Earwicker about 15 years

That's odd. Can you see the string "ISO-8859-1" in this page? rfc-editor.org/rfc/rfc2616.txt - I count 9.
Jon Skeet about 15 years

Yup, excellent - I think I must have been looking at a version which was split up into sections, and not looking at the right section...
Jon Skeet about 15 years

(It's still not exactly well written, IMO, but at least it's pinned down to some extent. :)
Daniel Earwicker about 15 years

By the standard of some RFCs these are masterpieces! :)
Tim about 15 years

Irrespective of whether C# strings are UTF-8 or UTF-16 it is still not clear to me why URLEncode would be implemented in such a way that it doesn't work in its default state with strings defined in C#. Surely it would be more intuitive for it to encode correctly?
Amit Patil about 15 years

URLs' encoding of Unicode characters is not ISO-8559-1, it is undefined. This unfortunately means there is no ‘right’ answer to URL-encoding, but these days more and more web services are converging on using UTF-8, so it's definitely a reasonable default.
Daniel Earwicker about 15 years

For URLs generated by a web app, inserted in an HTML page as links and then sent back by the browser, they can use whatever character set they want. But for URLs generated from web form submission by the browser, you have to go with what browsers do.
Amit Patil about 15 years

What Browsers Do is dependent on the settings, but more likely UTF-8 than ISO-8859-1. The Google example above only works because Google picks up non-UTF-8 sequences and re-interprets them; if you use the URL “google.com/search?q=%C2%a3” you do get a search for a single ‘£’ and not ‘Â£’.