Upgrade python in a virtualenv

Question

Is there a way to upgrade the version of python used in a virtualenv (e.g. if a bugfix release comes out)?

I could pip freeze --local > requirements.txt, then remove the directory and pip install -r requirements.txt, but this requires a lot of reinstallation of large libraries, for instance, numpy, which I use a lot.

I can see this is an advantage when upgrading from, e.g., 2.6 -> 2.7, but what about 2.7.x -> 2.7.y?

Solution 1

Did you see this? If I haven't misunderstand that answer, you may try to create a new virtualenv on top of the old one. You just need to know which python is going to use your virtualenv (you will need to see your virtualenv version).

If your virtualenv is installed with the same python version of the old one and upgrading your virtualenv package is not an option, you may want to read this in order to install a virtualenv with the python version you want.

EDIT

I've tested this approach (the one that create a new virtualenv on top of the old one) and it worked fine for me. I think you may have some problems if you change from python 2.6 to 2.7 or 2.7 to 3.x but if you just upgrade inside the same version (staying at 2.7 as you want) you shouldn't have any problem, as all the packages are held in the same folders for both python versions (2.7.x and 2.7.y packages are inside your_env/lib/python2.7/).

If you change your virtualenv python version, you will need to install all your packages again for that version (or just link the packages you need into the new version packages folder, i.e: your_env/lib/python_newversion/site-packages)

Solution 2

If you happen to be using the venv module that comes with Python 3.3+, it supports an --upgrade option. Per the docs:

Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place

python3 -m venv --upgrade ENV_DIR

Solution 3

Updated again: The following method might not work in newer versions of virtualenv. Before you try to make modifications to the old virtualenv, you should save the dependencies in a requirement file (pip freeze > requirements.txt) and make a backup of it somewhere else. If anything goes wrong, you can still create a new virtualenv and install the old dependencies in it (pip install -r requirements.txt).

Updated: I changed the answer 5 months after I originally answered. The following method is more convenient and robust.

Side effect: it also fixes the Symbol not found: _SSLv2_method exception when you do import ssl in a virtual environment after upgrading Python to v2.7.8.

Notice: Currently, this is for Python 2.7.x only.

---

If you're using Homebrew Python on OS X, first deactivate all virtualenv, then upgrade Python:

brew update && brew upgrade python

Run the following commands (<EXISTING_ENV_PATH> is path of your virtual environment):

cd <EXISTING_ENV_PATH>
rm .Python
rm bin/pip{,2,2.7}
rm bin/python{,2,2.7}
rm -r include/python2.7
rm lib/python2.7/*
rm -r lib/python2.7/distutils
rm lib/python2.7/site-packages/easy_install.*
rm -r lib/python2.7/site-packages/pip
rm -r lib/python2.7/site-packages/pip-*.dist-info
rm -r lib/python2.7/site-packages/setuptools
rm -r lib/python2.7/site-packages/setuptools-*.dist-info

Finally, re-create your virtual environment:

virtualenv <EXISTING_ENV_PATH>

By doing so, old Python core files and standard libraries (plus setuptools and pip) are removed, while the custom libraries installed in site-packages are preserved and working, as soon as they are in pure Python. Binary libraries may or may not need to be reinstalled to function properly.

This worked for me on 5 virtual environments with Django installed.

BTW, if ./manage.py compilemessages is not working afterwards, try this:

brew install gettext && brew link gettext --force

Solution 4

Step 1: Freeze requirement & take a back-up of existing env

pip freeze > requirements.txt
deactivate
mv env env_old

Step 2: Install Python 3.7 & activate virutal environment

sudo apt-get install python3.7-venv
python3.7 -m venv env
source env/bin/activate
python --version

Step 3: Install requirements

sudo apt-get install python3.7-dev
pip3 install -r requirements.txt

Solution 5

How to upgrade the Python version for an existing virtualenvwrapper project and keep the same name

I'm adding an answer for anyone using Doug Hellmann's excellent virtualenvwrapper specifically since the existing answers didn't do it for me.

Some context:

• I work on some projects that are Python 2 and some that are Python 3; while I'd love to use python3 -m venv, it doesn't support Python 2 environments
• When I start a new project, I use mkproject which creates the virtual environment, creates an empty project directory, and cds into it
• I want to continue using virtualenvwrapper's workon command to activate any project irrespective of Python version

Directions:

Let's say your existing project is named foo and is currently running Python 2 (mkproject -p python2 foo), though the commands are the same whether upgrading from 2.x to 3.x, 3.6.0 to 3.6.1, etc. I'm also assuming you're currently inside the activated virtual environment.

1. Deactivate and remove the old virtual environment:

$ deactivate
$ rmvirtualenv foo

Note that if you've added any custom commands to the hooks (e.g., bin/postactivate) you'd need to save those before removing the environment.

2. Stash the real project in a temp directory:

$ cd ..
$ mv foo foo-tmp

3. Create the new virtual environment (and project dir) and activate:

$ mkproject -p python3 foo

4. Replace the empty generated project dir with real project, change back into project dir:

$ cd ..
$ mv -f foo-tmp foo
$ cdproject

5. Re-install dependencies, confirm new Python version, etc:

$ pip install -r requirements.txt
$ python --version

If this is a common use case, I'll consider opening a PR to add something like $ upgradevirtualenv / $ upgradeproject to virtualenvwrapper.

View more solutions

Comments

[Sam Dolan]

While you may be able to modify some paths, etc. to get it to work. The last thing you want is a slightly different environment from production. Just think.. if there is even one difference between how the various versions compile, you're going to loose out on all the time saved by tracking down the bug. I'd just take the time to create a brand new virtualenv and reinstall everything.

[Mohammed Shareef C]

@Simon Walker. Exactly same problem. I have to upgrade python 3.5.3 for a bug fix.

[Gonçalo Peres]

Does this answer your question? Can existing virtualenv be upgraded gracefully?

[Antti Haapala -- Слава Україні]

He specifically did not want to reinstall any packages if possible. Obviously you have never installed numpy from source if you do not know why it matters ;) hint: it takes a LOOOOOOOOOOOOOONG time.

[marianobianchi]

yeah, i understand that. But it seems that if he make a virtualenv in top of the other, packages are not lost, so he won't have to reinstall numpy or any other package. Nevertheless, i think he should try this in a new virtualenv in case it fails.

[Antti Haapala -- Слава Україні]

I rest my case, I believe that the new virtualenv versions do indeed upgrade in place, however for old versions, they did not even overwrite the python executable, which caused problems with at least one python bugfix release.

[Kentzo]

It didn't work for me and looks like it's not supposed to: github.com/pypa/virtualenv/issues/437

[marianobianchi]

Can you show us the error you get @Kentzo? It probably won't work in all cases, but it surely works in some cases. Maybe if we show us the error you get we can understand it and help you...

[Sergey Orshanskiy]

I tried to create a new virtualenv on top of the old one, and it did work. I did have to specify -p to point it to the right version of python.

[dpwr]

Whatever you do, don't remove the old python version from your system before upgrading the virtualenv because otherwise you can't do the install over itself trick.

[dpwr]

For most people I think making use of pip machinery to do the work makes sense, but for anyone who has customized their virtualenvs postactivate, predeactivate, etc, this is not a desirable option because all that is lost and has to be manually readded.

[Koen.]

The only file I had to delete was the bin/python file.

[Rockallite]

For some older version Python, deleting setuptools and pip is necessary.

[Clément]

I also had to delete <EXISTING_ENV_PATH>/.Python as that broke the virtualenv creation.

[Kurt]

Based on this answer I wrote a little script (with full attribution): burgundywall.com/post/update-all-python-virtual-environments

[rspeer]

Unfortunately, this doesn't work on Ubuntu 16.04, where they broke venv and refuse to fix it on the basis that "you could just use virtualenv". EDIT: actually, it does work! It just also shows an error message about "ensurepip".

[Nick Lucas]

You can actually easily remove your virtual environment by running rmvirtualenv <env_name> and will remove all its lined dependencies :) See more at: virtualenvwrapper.readthedocs.io/en/latest/…

[Rockallite]

@Nick Lucas In fact, I wanted to keep all the lined dependencies, while removing the core Python files and standard libraries :)

[Sarfraaz Ahmed]

I upgraded python from 3.5.1 to 3.6.4 on my system. And to upgrade python on my virtualenv, I had to explicitly mention python3.6 in the command : python3.6 -m venv --upgrade <path_of_existing_env>

[Scott Stevens]

The --upgrade flag is not necessary in my experience (3.5.2 to 3.5.4 on Windows/Git Bash).

[Jens]

In my case, bin/python and bin/python3 still linked to the old python3.4 binary, and I had to set the links manually. Also, is there a way to remove the now outdated python version?

[user305883]

I got Error: Command '['/Users/me/Sites/site/venv3/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. > what does it mean ? I see no active python processes, cannot upgrade.

[Mayur Patil]

awesome! Thanks for this gem

[AstroFloyd]

@user305883 I got a similar error after I upgraded my system Python version from 3.5 to 3.6 (and removed 3.5). I had to reinstall 3.5 and run python3.6 -m venv --upgrade ENV_DIR for it to work. I think I'll still have to update some symbolic links in ENV_DIR/bin. Hence, the trick was to update with the new Python version while the old version was still installed.

[Ash Berlin-Taylor]

py.test doesn't work because `~/.virtualenvs/myenv/bin/python3.7 isn't a symlink, so doesn't get updated.

[Ash Berlin-Taylor]

Ah actually, the problem is There is ~/.virtualenvs/myenv/.Python which is a symlink.

[user2682863]

@user305883 for those getting the subprocess error called python -Im ensurepip to see what the actual underlying error is

[Nic Scozzaro]

@user305883 I had a similar error and found that adding the --without-pip flag solved the issue for me.

[chris Frisina]

Yes, please. I got halfway through this and got major problems... A check would be nice because the first step is irreversible once deleted (to my current knowledge)

[Jens]

You refer to this pipenv?

[josemfc]

@Anupam it worked for me for python 3.9.5 python3.9 -m venv --upgrade venv_dir/.

[Allis Gao]

did not work for me: V3.6.8 -> V3.8.10. F:\MyCodes\python\dtprjops>python -m venv --upgrade venv Error: [Errno 13] Permission denied: 'F:\\MyCodes\\python\\dtprjops\\venv\\Scripts\\python.exe'

[KenBuckley]

I found it necessary to also install distutils for my python version to ensure that the new mkvirtualenv command worked without errors: sudo apt-get install python3.10-distutils