Is there a way to install all python modules at once using pip?

Question

I would like to install all available modules for Python 2.7.12 using a single pip command. Is there a way to do this without having to specify every single package name?

Solution 1

I highly recommend against doing this - the overwhelmingly supported best practice is to use a requirements.txt file, listing the packages you want to install specifically.

You then install it with pip install -r requirements.txt and it installs all the packages for your project.

This has several benefits:

• Repeatability by installing only the required packages
• Conciseness

However, if you really do want to install ALL python packages (note that there are thousands), you can do so via the following:

pip search * | grep ")[[:space:]]" | cut -f1 -d" "

I strongly recommend against this as it's likely to do horrible things to your system, as it will attempt to install every python package (and why it's in spoiler tags).

Solution 2

This is a terrible idea, but you could always use autoinstaller, which will automatically download packages via pip if you don't have them installed.

Solution 3

UPDATE: The other answers were right when they said that this was a terrible idea: Some of the packages I have installed using this method will not work because their fixed-old required dependencies have been updated by other packages. Next time, I'm using virtualenv. Some day, I'm going to produce a list of packages from packages.pypy.org with their dependencies, so that someone can make many environments with those packages quickly.

---

Well, I did find a way to do this, but only for the 1000 most popular packages at http://packages.pypy.org. But other pages can be processed too, it just needs some creativity. For that page you will need a spreadsheet program to process tha page. I used LibreOffice Calc. And I did do this on windows using the Command Prompt.

First, open the link I just posted and select/copy all the text (and do not expand any of the package names on the webpage).

Then, paste it on any cell in Calc. Open the Find & Replace dialog. If you use another spreadsheet program, make sure it's Find & Replace supports regexps. Select the checkbox next to Regular Expressions (or something like that), type downloads:\s+\d+ in the Search for field, make sure there is no text in the Replace with field, and leave all the other options at their defaults, click Replace all (assuming the program has this button). After that, you get rid of the trailing spaces by doing another replace with a space in the Search for field (Python package names do not have spaces). Once you do these two replaces, you have a package name in each cell.

Then you copy the entire spreadsheet into a text editor, any editor should work (I used Notepad). The text editor will put the contents of each cell (i.e. the package names) onto it's own line. There won't be any blank lines in the file (but it doesn't hurt to check). At this point you can close the spreadsheet, you don't have to save it's data.

Now save the text file. Open a command prompt and change to the folder where you saved the text file. Now type:

for /F %p in ('type TEXT_FILE_NAME.txt') DO pip.exe install %p

[If Command Prompt can't find the right pip to use, then you need to type the full path to pip e.g. C:\Python27\Scripts\pip.exe]

Voila, pip is now installing all of the packages listed on the link I gave.

You should redirect stdout (and maybe stderr too) to a file because command prompts don't have a large history buffer. For that, you put something like this at the end of the command:

>>PIP_STDOUT_FILE.txt 2>>PIP_STDERR_FILE.txt

Be careful about abruptly interrupting pip if you use this, because output no longer goes to the terminal. You might kill it in the middle of a package installation. In particular, if you use a text editor to view the output, reload the file frequently. It's safer to append --log LOG_FILE.txt instead of the above, but the log file becomes more verbose and also writes its messages to the command prompt as usual.

There are quite a number of

1. Linux-only packages
2. Packages built with a Microsoft C++ compiler

and using pip to install these will fail unless you do some additional work. But this should work for the majority of packages.