How to specify install order for python pip?

Question

I'm working with fabric(0.9.4)+pip(0.8.2) and I need to install some python modules for multiple servers. All servers have old version of setuptools (0.6c8) which needs to be upgraded for pymongo module. Pymongo requires setuptools>=0.6c9.

My problem is that pip starts installation with pymongo instead of setuptools which causes pip to stop. Shuffling module order in requirements file doesn't seem to help.

requirements.txt:

setuptools>=0.6c9
pymongo==1.9
simplejson==2.1.3

Is there a way to specify install order for pip as it doesn't seem to do it properly by itself?

This can be resolved with two separate requirements files but it would be nice if I didn't need to maintain multiple requirements files now or in the future.

Problem persists with pip 0.8.3.

Solution 1

This is a silly hack, but might just work. Write a bash script that reads from your requirements file line by line and runs the pip command on it.

#!/bin/bash
for line in $(cat requirements.txt)
do
  pip install $line -E /path/to/virtualenv
done

Solution 2

You can just use:

cat requirements.txt | xargs pip install

Solution 3

To allow all types of entries (for example packages from git repositories) in requirements.txt you need to use the following set of commands

cat requirements.txt | xargs -n 1 -L 1 pip install

-n 1 and -L 1 options are necessary to install packages one by one and treat every line in the requirements.txt file as a separate item.

Solution 4

Sadly the upgrade suggestion won't work. If you read the other details in https://github.com/pypa/pip/issues/24 you will see why

pip will build all packages first, before attempting to install them. So with a requirements file like the following

numpy==1.7.1
scipy==0.13.2
statsmodels==0.5.0

The build of statsmodels will fail with the following statement

ImportError: statsmodels requires numpy

The workaround given for manually calling pip for each entry in the requirements file (via a shell script) seems to be the only current solution.

Solution 5

Pymongo requires setuptools>=0.6c9

How do you know? Requires to build or to install? You don't say what version of Pymongo you were trying to install but looking at setup.py file for current (3.2.2) version there's no specification of neither what Pymongo requires to run setup.py (setup_requires) nor what it requires to install (install_requires). With no such information pip can't ensure specific version of setuptools. If Pymongo requires specific version of setuptools to run its setup.py (as opposed to requiring setuptools to run setup function itself) then the other problem is that until recently there was no way to specify this. Now there's specification – PEP 518 – Specifying Minimum Build System Requirements for Python Projects, which should be shortly implemented in pip – Implement PEP 518 support #3691.

As to order of installation, this was fixed in pip 6.1.0;

From pip install – Installation Order section of pip's documentation:

As of v6.1.0, pip installs dependencies before their dependents, i.e. in "topological order". This is the only commitment pip currently makes related to order.

And later:

Prior to v6.1.0, pip made no commitments about install order.

However, without proper specification of requirements by Pymongo it won't help either.

View more solutions

Comments

[Hugo Tavares]

It is OK to this case, but be careful about requirements file lines like: --find-links mypypi.com/pypi

[Seppo Erviälä]

pip 0.8.3 didn't help in this case

[Seppo Erviälä]

This script can be pythonized if using fabric: for line in open("requirements.txt", "r"): sudo("pip -E %s install %s" % (virtualenv_path, line))

[Venkat Kotra]

This works but causes a problem if you have PIL==1.1.7 --allow-external PIL --allow-unverified PIL as an entry in requirements.txt

[Piotr Dobrogost]

How is this supposed to allow packages from git repositories yet alone all types of entries (one example being PIL==1.1.7 --allow-external PIL --allow-unverified PIL)?

[Piotr Dobrogost]

With addition of topological sort to pip (Issue #2478: Topological installation order) that's no longer the case – pip installs each package's dependencies first and then the package itself.

[Sebastian Schrader]

This is Useless Use of Cat (UUoC). In addition, you must add -L 1 to ensure that only one line is used.xargs -L 1 pip install < requirements.txt

[jorgeh]

Sometimes there are comments in requirements.txt (lines starting with # are considered comments by pip). In such cases, you might prefer to use: grep -v '^#' requirements.txt | xargs pip install

[pythonator]

Also look out for such simple things as whitespace. wheel >= 0.34.0 will fail, whereas wheel>=0.34.0 will not.

[davidA]

I've still found this to be problematic with pip 21.3 when installing into a venv and a suitable build dependency is already installed on the host. For example, with a fresh venv but a host-installed numpy lurking around, if a requirements.txt file specifies numpy and something that has a compiled extension (like transformations) then transformations will be built against the host's numpy headers rather than those installed by numpy, resulting in a broken extension if the API versions don't match. Perhaps it's an issue with the way transformations is built, I'm not sure.

[davidA]

To continue my previous comment: if transformations is then force-reinstalled (after numpy from requirements.txt is installed), then the building of the extension picks up the new numpy's headers and builds with the correct API. To date, I haven't found a way around this except to create multiple requirements.txt for staged installations.