Python string encoding for a variable

Question

I'm aware of the fact that for Python < 3, unicode encoding for the string 'Plants vs. Zombies䋢 2' is as below:

u"Plants vs. Zombies䋢 2".encode("utf-8")

What if I have an variable (say appName) instead of a string can I do it like this:

  appName = "Plants vs. Zombies䋢 2"
 u+appName.encode("utf-8")

For:

 appName = appName.encode('utf-8');


 'ascii' codec can't decode byte 0xe4 in position 18: ordinal not in range(128)

Solution 1

No. The u notation is only for string literals. Variables containing string data don't need the u, because the variable contains an object that is either a unicode string or a byte string. (I'm assuming here that appName contains string data; if it doesn't, it doesn't make sense to try to encode it. Convert it to a bytestring or unicode first.)

So your variable either contains a unicode string or a byte string. If it is a unicode string you can just do appName.encode("utf-8").

If it is a byte string then it is already encoded with some encoding. If it's already encoded as UTF-8, then it's already how you want it and you don't need to do anything. If it's in some other encoding and you want to get it into UTF-8, you can do appName.decode('the-existing-encoding').encode("utf-8").

Note that if you do what you show in your edited, question, the result might not be what you expect. You have:

appName = "Plants vs. Zombies䋢 2"

Without the u on the string literal, you have created a bytestring in some encoding, namely the encoding of your source file. If your source file isn't in UTF-8, then you're in the last situation I described above. There is no way to "just make a string unicode" after you have created it as non-unicode. When you create it as non-unicode, you are creating it in a particular encoding, and you have to know what encoding that is in order to decode it to unicode (so you can then encode it to another encoding if you want).

Solution 2

No. the u prefix modifies the meaning of a string constant (making it a unicode constant). It is not an operator (which could be applied to any expression).

Comments

[freakish]

Your variable either contains a unicode string or a byte string Where is that stated?

[BrenBarn]

@freakish: It doesn't say it, but it's implied by the way he's using u and by his example with a string literal. I clarified my answer to say I'm assuming appName contains string data.