Error while upgrading pip: UnicodeDecodeError: 'utf-8' codec can't decode byte

Question

I've just installed python on windows 10, and I'm trying to upgrade pip.

My windows user name has hebrew charecters...

When I try to run:

python -m pip install --upgrade pip

I get this error:

Collecting pip
Using cached pip-8.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 7.1.2
Exception:
Traceback (most recent call last):
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\basecommand.py", line 211, in main
    status = self.run(options, args)
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\commands\install.py", line 311, in run
    root=options.root_path,
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\req\req_set.py", line 640, in install
    requirement.uninstall(auto_confirm=True)
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\req\req_install.py", line 673, in uninstall
    for path in pip.wheel.uninstallation_paths(dist):
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\wheel.py", line 512, in unique
    for item in fn(*args, **kw):
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\wheel.py", line 531, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1619, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 1616, in get_metadata
    return self._get(self._fn(self.egg_info, name)).decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf2 in position 22365: invalid continuation byte
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

I'm geussing this has to do with my Hebrew windows user name, Is that correct?

Can I upgrade pip without opening a new windows user?

Solution

It looks like a bug in pip where it's assuming its metadata is stored as UTF-8. Instead, your username appears to be encoded as "windows-1255".

You could try the following:

1. Backup C:\Users\עדי\AppData\Local\Programs\Python\Python35-32\lib\site-packages\pip\_vendor\pkg_resources\__init__.py
2. Goto line: 1616
3. Change utf-8 to mbcs.
4. Re-run upgrade

Comments

[Eryk Sun]

The problem is that the distlib function _csv_open wrote the RECORD of installed files using the default locale encoding. Never depend on this, especially on Windows, since the ANSI locale is all but worthless. I'm amazed that PEPs 376 and 427 have nothing to say about enforcing UTF-8 for the contents of this file.

[Eryk Sun]

I recommend converting site-packages\pip-7.1.2.dist-info\RECORD to UTF-8, i.e. read it as 'cp1255' and write it back as 'utf-8'.

[Moberg]

Which line should be changed? This answer is outdated for Python36-32 unfortunately.

[Alastair McCormack]

@Moberg which version of Pip are you using?

[Moberg]

Oh wait, the upgrading of pip was not a problem, It was only when trying to install the package z3-solver :/ (running pip version 9.0.1)

[Alastair McCormack]

@Moberg Can you raise a new question please?

[Moberg]

I think I will :)

[Alastair McCormack]

Based on @eryksun 's research, I've created a pull request in distutils to force encoding of RECORD to utf-8 on Python 2 and 3 on all platforms: bitbucket.org/pypa/distlib/pull-requests/34/…