Converting from ascii to utf-8 with Python

Question

I have xmpp bot written in python. One of it's plugins is able to execute OS commands and send output to the user. As far as I know output should be unicode-like to send it over xmpp protocol. So I tried to handle it this way:

output = os.popen(cmd).read() 
if not isinstance(output, unicode):
   output = unicode(output,'utf-8','ignore')
bot.send(xmpp.Message(mess.getFrom(),output))

But when Russian symbols appear in output they aren't converted well.

sys.getdefaultencoding() 

says that default command prompt encoding is 'ascii', but when I try to do

output.decode('ascii') 

in python console I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position 1: 
ordinal not in range(128)

OS: Win XP, Python 2.5.4 PS: Sorry for my English :(

Solution 1

You say """sys.getdefaultencoding() says that default command prompt encoding is 'ascii'"""

sys.getdefaultencoding says NOTHING about the "command prompt" encoding.

On Windows, sys.stdout.encoding should do the job. On my machine, it contains cp850 when Python is run in a Command Prompt window, and cp1252 in IDLE. Yours should contain cp866 and cp1251 respectively.

Update You say that you still need cp866 in IDLE. Note this:

IDLE 2.6.4      
>>> import os
>>> os.popen('chcp').read()
'Active code page: 850\n'
>>>

So when your app starts up, check if you are on Windows and if so, parse the result of os.popen('chcp').read(). The text before the : is probably locale-dependent. codepage = result.split()[-1] may be good enough "parsing". On Unix, which doesn't have a Windows/MS-DOS split personality, sys.stdout.encoding should be OK.

Solution 2

sys.getdefaultencoding() returns python's default encoding - which is ASCII unless you have changed it. ASCII doesn't support Russian characters.

You need to work out what encoding the actual text is, either manually, or using the locale module.

Typically something like:

import locale
encoding = locale.getpreferredencoding(do_setlocale=True)¶

Solution 3

Ascii has no defined character values above 127 0x7F. Perhaps you mean the Cyrillic code page? It's 866

See http://en.wikipedia.org/wiki/Code_page

edit: since this answer was marked correct presumably 886 worked, but as other answers have pointed it, 886 is not the only Russian language code page. If you use a code page different from the one that was used when the Russian symbols were encoded, you will get the wrong result.

Comments

[Glenn Maynard]

Please use the real name, "KOI8-R", not the opaque Windows name "CP866".

[colriot]

But results of a.decode('cp866') and a.decode('koi8-r') are different

[John Knoeller]

If there is a portable identifier for the Cyrillic code page, it would be best to use it. Glenn, do you have a reference for KOI8-R ?

[Amit Patil]

Code page 866 is nothing like KOI8-R at all, or indeed any other Russian encoding. As a DOS code page you don't generally meet it much any more. See en.wikipedia.org/wiki/Code_page_866 vs en.wikipedia.org/wiki/KOI8-R vs the more usual en.wikipedia.org/wiki/Windows-1251.

[John Machin]

On Windows, that will give cp1251 in the OP's (Russian) setup even when Python is run at MS-DOS-emulating command prompt; the OP needs cp866.

[colriot]

Thank you! It is exactly what I wanted

[colriot]

Or not. How can I find out os.popen(command).read() default encoding? Or it depends on the command?

[John Machin]

os.popen("command").read() default encoding?? No such concept. The encoding of data being transmitted is chosen by (or forced upon) the WRITER; it has nothing to do with the READER, who needs to know or guess the encoding or obtain the encoding from a reliable source. Why are you asking? Why is sys.stdout.encoding not exactly what you wanted?

[colriot]

Because it does not matter whether you'll run python from Command Prompt or IDLE. 'cp866' is the right choise in both cases.

[colriot]

Thanks. This method seamed to be ideal. But when I tried to test bot with 'ipconfig' command... So 'cp1251' is real encoding of output in this case. Does this mean there is no universal method to solve my problem?