From virtualenv, pip freeze > requirements.txt give TONES of garbage! How to trim it out?

Question

I'm following this tutorial: http://devcenter.heroku.com/articles/django

At some point I'm suposed to do:

pip freeze > requirements.txt

(Ofc. from virtualenv created instance of python)

And I get this:

(venv)przemoli@ubuntu:~/Programowanie/hellodjango$ cat requirements.txt 
BeautifulSoup==3.2.0
Brlapi==0.5.5
CherryPy==3.1.2
ClientForm==0.2.10
Django==1.3
GnuPGInterface==0.3.2
PAM==0.4.2
PIL==1.1.7
Routes==1.12.3
Twisted-Core==11.0.0
Twisted-Names==11.0.0
Twisted-Web==11.0.0
WebOb==1.0.8
adium-theme-ubuntu==0.3.1
apt-xapian-index==0.44
apturl==0.5.1ubuntu1
chardet==2.0.1
command-not-found==0.2.44
configglue==1.0
cssutils==0.9.8a1
defer==1.0.2
distribute==0.6.19
django-tagging==0.3.1
dnspython==1.9.4
duplicity==0.6.15
gnome-app-install==0.4.7-nmu1ubuntu2
httplib2==0.7.2
jockey==0.9.4
keyring==0.6.2
launchpadlib==1.9.8
lazr.restfulclient==0.11.2
lazr.uri==1.0.2
louis==2.3.0
lxml==2.3
mechanize==0.1.11
nvidia-common==0.0.0
oauth==1.0.1
onboard==0.96.1
oneconf==0.2.6.7
papyon==0.5.5
pexpect==2.3
piston-mini-client==0.6
protobuf==2.4.0a
psycopg2==2.4.4
pyOpenSSL==0.12
pycrypto==2.3
pycups==1.9.59
pycurl==7.19.0
pyinotify==0.9.1
pyparsing==1.5.2
pyserial==2.5
pysmbc==1.0.10
python-apt==0.8.0ubuntu9
python-dateutil==1.4.1
python-debian==0.1.20ubuntu2
python-virtkey==0.60.0
pyxdg==0.19
sessioninstaller==0.0.0
simplejson==2.1.6
system-service==0.1.6
ubuntu-sso-client==1.4.0
ubuntuone-couch==0.3.0
ubuntuone-installer==2.0.0
ubuntuone-storage-protocol==2.0.0
ufw==0.30.1-2ubuntu1
unattended-upgrades==0.1
usb-creator==0.2.23
virtualenv==1.6.4
wadllib==1.2.0
wsgiref==0.1.2
xdiagnose==1.1
xkit==0.0.0
zope.interface==3.6.1

When deploying on heroku it fails at Brlapi .....

I see lots of stuff from my main python installation which is on ubuntu. Which is BAD since Ubuntu use python for quite a few thing itself (ubuntu-one, usb-creator, etc..).

I do not need them on heroku! I need only Django, psycopg2, and their dependencies. I do not even know if its fault of pip, or virutalenv. (If you want to know my setup look at link above I copied it into terminal)

Solution 1

That is one thing that has bugged me too quite a bit. This happens when you create a virtualenv without the --no-site-packages flag.

There are a couple of things you can do:

1. Create virtualenv with the --no-site-packages flag.
2. When installing apps, dont run pip install <name> directly, instead, add the library to your requirements.txt first, and then install the requirements. This is slower but makes sure your requirements are updated.
3. Manually delete libraries you dont need. A rule of thumb i follow for this is to add whatever is there in my INSTALLED_APPS, and database adapters. Most other required libraries will get installed automatically because of dependencies. I know its silly, but this is what I usually end up doing.

-- Edit --

I've since written a couple of scripts to help manage this. The first runs pip freeze and adds the found library to a provided requirements file, the other, runs pip install, and then adds it to the requirements file.

function pipa() {
    # Adds package to requirements file.
    # Usage: pipa <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    package_string=`pip freeze | grep -i $package_name`
    current_requirements=`cat $requirements_file`
    echo "$current_requirements\n$package_string" | LANG=C sort | uniq > $requirements_file
}

function pipia() {
    # Installs package and adds to requirements file.
    # Usage: pipia <package> <path to requirements file>
    package_name=$1
    requirements_file=$2
    if [[ -z $requirements_file ]]
    then
        requirements_file='./requirements.txt'
    fi
    pip install $package_name
    pipa $package_name $requirements_file
}

Solution 2

pipreqs solves the problem. It generates project-level requirement.txt file.

• Install pipreqs: pip install pipreqs
• Generate project-level requirement.txt file: pipreqs /path/to/your/project/ requirements file would be saved in /path/to/your/project/requirements.txt

Solution 3

If you care a lot about the cleanliness of your requirements.txt you should not only use the --no-site-packages option as already mentioned but also consider not to pipe the output of pip freeze directly to your requirements.txt. The reason for that is, that when doing a pip freeze not only packages specified by yourself show up, but also dependencies installed by these packages! It isn't necessary to keep them all in your requirements.txt as they will get installed automatically with the package that requires them... So if you add a new package to your virtualenv you probably should just add the line for this package to your requirements.txt...

See this example:

(demo)[~]$ pip freeze
distribute==0.6.19
wsgiref==0.1.2
(demo)[~]$ pip install django-blog-zinnia
Downloading/unpacking django-blog-zinnia
  Downloading django-blog-zinnia-0.9.tar.gz (523Kb): 523Kb downloaded
  Running setup.py egg_info for package django-blog-zinnia

    no previously-included directories found matching 'docs/api'
    no previously-included directories found matching 'docs/build'
    no previously-included directories found matching 'docs/coverage'
    no previously-included directories found matching 'zinnia/media/zinnia/css/.sass-cache'
Downloading/unpacking BeautifulSoup>=3.2.0 (from django-blog-zinnia)
  Downloading BeautifulSoup-3.2.1.tar.gz
  Running setup.py egg_info for package BeautifulSoup

  # truncated as it installs some more dependencies
Successfully installed django-blog-zinnia BeautifulSoup django-mptt django-tagging django-xmlrpc pyparsing
Cleaning up...
(demo)[~]$ pip freeze
BeautifulSoup==3.2.1
distribute==0.6.19
django-blog-zinnia==0.9
django-mptt==0.5.2
django-tagging==0.3.1
django-xmlrpc==0.1.3
pyparsing==1.5.6
wsgiref==0.1.2

(Though I should probably mentioned that in most cases it will not hurt that you have these dependencies there, just your file will grow and get harder to maintain.)

Solution 4

You can use:

pip freeze --local > requirement.txt

so only the packages installed locally in your virtualenv are listed in requirements.txt, not the globally-installed packages.

View more solutions

Comments

[Paolo]

Note: since v. 1.7, virtualenv takes --no-site-packages by default, so it's not required you specify that option. It's the default. Link: pypi.python.org/pypi/virtualenv#changes-news

[przemo_li]

Thx! --no-site-packages works fine! (Though I'm still puzzled why heroku crew did not mentioned it in theirs docs). (And yes I did used Ubuntu 11.10 provided package containing virtualenv 1.6.4 :( :( :( )

[przemo_li]

Nah! I just wanted quick solution to separate my OS related python libraries from my django development libraries that should be installed on deployment machine! (imho default behaviour of pip freeze should be changed...)

[Kenneth Reitz]

If you're using the latest version of virtualenv, --no-site-packages isn't necessary anymore. I highly recommend not relying on python modules from aptitude :)

[tobych]

I think you absolutely should keep all the indirect dependencies in your requirements.txt actually, along with their version numbers. That way your builds are repeatable. I used to keep my direct requirements at the top of the file, and my indirect ("transitive") dependencies at the bottom. Now I just keep them all in one file, using pip freeze to generate the information, but with comments, and in alphabetical order.

[Nick Zalutskiy]

You can also do pip freeze --local > requirements.txt This will output only the packages installed into your virtual env, without listing all dependencies (the packages themselves, handle those.)

[Roman]

Sadly it does not work, it creates a requirements but still some modules are missing

[Haifeng Zhang]

@Roman if you use virtualenv to run your project,pls make sure you use pip that belongs to the virtualenv install pipreqs

[SeedyROM]

Seems to be working as intended for me, would recommend as a quick fix.

[lmblanes]

I can't seem to get it working. "No module named pipreqs.__main__; 'pipreqs' is a package and cannot be directly executed"

[Haifeng Zhang]

@lmblanes check whether your pip outdated

[Faizan Ahmad]

and if you are in the current directory use pipreqs .