UnicodeEncodeError: 'ascii' codec can't encode character at special name

Question

My python (ver 2.7) script is running well to get some company name from local html files but when it comes to some specific country name, it gives this error "UnicodeEncodeError: 'ascii' codec can't encode character"

Specially getting error when this company name comes

Company Name: Kühlfix Kälteanlagen Ing.Gerhard Doczekal & Co. KG

The link cannot be processed

Traceback (most recent call last): 
  File "C:\Python27\Process2.py", line 261, in <module>
    flog.write("\nCompany Name: "+str(pCompanyName))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 9: ordinal not in range(128)

Error gives in this line of code:

if companyAlreadyKnown == 0:
   for hit in soup2.findAll("h1"):
       print "Company Name: "+hit.text
       pCompanyName = hit.text
       flog.write("\nCompany Name: "+str(pCompanyName))
       companyObj.setCompanyName(pCompanyName)

Solution 1

Try setting the system default encoding as utf-8 at the start of the script, so that all strings are encoded using that.

Example -

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

The above should set the default encoding as utf-8 .

Solution 2

You really want to do this

flog.write("\nCompany Name: "+ pCompanyName.encode('utf-8'))

This is the "encode late" strategy described in this unicode presentation (slides 32 through 35).

Comments

[rhb1]

now another error i am facing bro ! Traceback (most recent call last): File "C:\Python27\Process2.py", line 261, in <module> print "Company Name: "+hit.text File "C:\Python27\lib\encodings\cp437.py", line 12, in encode return codecs.charmap_encode(input,errors,encoding_map) UnicodeEncodeError: 'charmap' codec can't encode character u'\xae' in position 2 8: character maps to <undefined>

[Anand S Kumar]

You have changed the encoding to something else now - charmap so the issue occurs.

[kubudi]

You may be the most intelligent python dev I've ever encountered. Was it really that hard..

[MartyMacGyver]

This works for Python 2.x, but it's not a great way to go about this and is deprecated in Python 3. Better to actually decode/encode the data properly. See discussion at Why should we NOT use sys.setdefaultencoding(“utf-8”) in a py script

[Padraic Cunningham]

This is the correct approach and should be the accepted answer.

[Padraic Cunningham]

@MartyMacGyver, totally correct, this can break libraries and cause hard to find bugs. The fact that this is not even mentioned makes this a very dangerous answer.

[Rudresh Ajgaonkar]

Works like magic. Cant Thank Enough!

[Siddhartha]

Ah after half an hour of shuffling encode and decode and str statements in my script. Thank you very much.

[Kyle Kochis]

I agree with the other commenters that this is a potentially dangerous answer and you should instead decode/encode properly. However if the error is being raised from a library you have no control over (in my case pyspark), this is a handy short term work around.

[Dima Lituiev]

This does not work: module 'sys' has no attribute 'setdefaultencoding'

[Arnaud Bouchot]

you're the man!

[Yuseferi]

AttributeError: module 'sys' has no attribute 'setdefaultencoding'

[Yuseferi]

using export PYTHONIOENCODING=UTF-8 works for me

[M. Mariscal]

OMG I'm getting the f****** error for 3 days, and couldn't deal with it until set your code... It was so frustrating... THANKS YOU VERY MUCH!

[Krishan Kumar Mourya]

Nice and pretty simple, don't have to mess with encoding and decoding.

[tinker-tailor]

Correct approach - correct answer