Python 3: os.walk() file paths UnicodeEncodeError: 'utf-8' codec can't encode: surrogates not allowed

Question

This code:

for root, dirs, files in os.walk('.'):
    print(root)

Gives me this error:

UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 27: surrogates not allowed

How do I walk through a file tree without getting toxic strings like this?

Solution 1

On Linux, filenames are 'just a bunch of bytes', and are not necessarily encoded in a particular encoding. Python 3 tries to turn everything into Unicode strings. In doing so the developers came up with a scheme to translate byte strings to Unicode strings and back without loss, and without knowing the original encoding. They used partial surrogates to encode the 'bad' bytes, but the normal UTF8 encoder can't handle them when printing to the terminal.

For example, here's a non-UTF8 byte string:

>>> b'C\xc3N'.decode('utf8','surrogateescape')
'C\udcc3N'

It can be converted to and from Unicode without loss:

>>> b'C\xc3N'.decode('utf8','surrogateescape').encode('utf8','surrogateescape')
b'C\xc3N'

But it can't be printed:

>>> print(b'C\xc3N'.decode('utf8','surrogateescape'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 1: surrogates not allowed

You'll have to figure out what you want to do with file names with non-default encodings. Perhaps just encoding them back to original bytes and decode them with unknown replacement. Use this for display but keep the original name to access the file.

>>> b'C\xc3N'.decode('utf8','replace')
C�N

os.walk can also take a byte string and will return byte strings instead of Unicode strings:

for p,d,f in os.walk(b'.'):

Then you can decode as you like.

Solution 2

I ended up passing in a byte string to os.walk() which will apparently return byte strings instead of incorrect unicode strings

for root, dirs, files in os.walk(b'.'):
    print(root)

Solution 3

Try using this line of code:

"bad string".encode('utf-8', 'replace').decode()

Comments

[Collin Anderson]

I ended up doing bad_string.encode('utf-8', 'surrogateescape').decode('ISO-8859-1')

[DoTheEvo]

@Collin Anderson How did you detect the occurrence of the bad string, how did you catch error?

[Mabyn]

What worked for me was "bad string".encode('utf-8', 'surrogateescape').decode('utf-8')

[DomQ]

You get upvote, and Python gets -10 points for Gryffindor.

[M.D.]

@DoTheEvo Collins hack works on both good and bad strings. It works because every byte is a valid code point in 'ISO-8859-1'. However it will print weird things for characters that don't have the same utf-8 and 'ISO-8859-1' encoding.