Check if my Python has all required packages

Question

I have a requirements.txt file with a list of packages that are required for my virtual environment. Is it possible to find out whether all the packages mentioned in the file are present. If some packages are missing, how to find out which are the missing packages?

Solution 1

UPDATE:

An up-to-date and improved way to do this is via distutils.text_file.TextFile. See Acumenus' answer below for details.

ORIGINAL:

The pythonic way of doing it is via the pkg_resources API. The requirements are written in a format understood by setuptools. E.g:

Werkzeug>=0.6.1
Flask
Django>=1.3

The example code:

import pkg_resources
from pkg_resources import DistributionNotFound, VersionConflict

# dependencies can be any iterable with strings, 
# e.g. file line-by-line iterator
dependencies = [
  'Werkzeug>=0.6.1',
  'Flask>=0.9',
]

# here, if a dependency is not met, a DistributionNotFound or VersionConflict
# exception is thrown. 
pkg_resources.require(dependencies)

Solution 2

Based on the answer by Zaur, assuming you indeed use a requirements file, you may want a unit test, perhaps in tests/test_requirements.py, that confirms the availability of packages.

Moreover, this approach uses a subtest to independently confirm each requirement. This is useful so that all failures are documented. Without subtests, only a single failure is documented.

"""Test availability of required packages."""

import unittest
from pathlib import Path

import pkg_resources

_REQUIREMENTS_PATH = Path(__file__).parent.with_name("requirements.txt")


class TestRequirements(unittest.TestCase):
    """Test availability of required packages."""

    def test_requirements(self):
        """Test that each required package is available."""
        # Ref: https://stackoverflow.com/a/45474387/
        requirements = pkg_resources.parse_requirements(_REQUIREMENTS_PATH.open())
        for requirement in requirements:
            requirement = str(requirement)
            with self.subTest(requirement=requirement):
                pkg_resources.require(requirement)

Solution 3

You can run pip freeze to see what you have installed and compare it to your requirements.txt file.

If you want to install missing modules you can run pip install -r requirements.txt and that will install any missing modules and tell you at the end which ones were missing and installed.

Solution 4

If you're interested in doing this from the command line, pip-missing-reqs will list missing packages. Example:

$ pip-missing-reqs directory
Missing requirements:
directory/exceptions.py:11 dist=grpcio module=grpc

(pip check and pipdeptree --warn fail only audit installed packages for compatibility with each other, without checking requirements.txt.)

Solution 5

You can use the -r option from pip freeze that verifies that. It generates a WARNING log for packages that are not installed. One appropriated verbose mode should be selected in order the WARNING message to be shown. For example:

$ pip -vvv freeze -r requirements.txt | grep "not installed"

WARNING: Requirement file [requirements.txt] contains six==1.15.0, but package 'six' is not installed

View more solutions

Comments

[Asclepius]

As a bonus, this automatically recursively detects conflicting version requirements -- these would be unsatisfiable.

[Zaur Nasibov]

This is truly a great idea!

[Shardj]

Loop over the require call with a try: except pkg_resources.DistributionNotFound wrapping the call. Then on exception you print. This way it'll tell you all missing dependencies and not die on the first missing one found.

[compman2408]

pip check does not verify the packages in a requirements file are installed. pip check verifies that already installed packages have compatible/non-broken dependencies.

[Zaur Nasibov]

Updated the accepted answer with a reference to your answer.

[Asclepius]

@R.Ilma With subtests you can know exactly the full subset of requirements that failed, so it's frequently useful while testing. Without subtests, you only know the first failure.

[Bilal]

When I run the test I get the error AttributeError: module 'distutils' has no attribute 'text_file' when I did check the distutils source code I didn't find text_file, I'm using Python 3.6.9

[Bilal]

@Acumenus, thank you for your prompt reply, I just added some details here with the solution I found. (sorry I did a mistake in the Python version, it's 3.6.9)

[sinoroc]

Instead of distutils.text_file.TextFile, one could use setuptools own pkg_resources.parse_requirements, like in this example: stackoverflow.com/a/59971236/11138259

[Asclepius]

@sinoroc Actually this answer used to use pkg_resources.parse_requirements as you can confirm in its edit history. I had to stop using it because its API was extremely wonky and used to keep breaking all the time. Now I have edited the answer again to go back to using it as per your suggestion.

[sinoroc]

I see, I think I'd already been told before that for a while that API was somewhat untrustworthy, don't remember the exact details anymore. Anyway, I appreciate when people make the effort to keep their answers up-to-date, thanks.

[Juan-Kabbali]

is it possible to install the missing packages inside the for ?

[Asclepius]

@Juan-Kabbali Even if it's possible, it's very non-standard. The packages are supposed to be managed by a package manager, e.g. pip, poetry, conda, etc., so the same package would have to be run. The standard workflow, however, is to first install the defined list of packages (as per the usage instructions of the chosen package manager), and only then run the unit tests.

[Sam Bull]

One thing this doesn't handle are referenced files (e.g. -r requirements.txt in a requirements-dev.txt file). This produces a parse error and requires a little more work to make it work.

[Asclepius]

@SamBull Noted, but then I have littler reason to reference requirements.txt in requirements-dev.txt because I can use pip install -U -r ./requirements.txt -r ./requirements-dev.txt to install both via a single command.

[Flimm]

The update links to Asclepius's answer. However, it doesn't use distutils.text_file.TextFile like the text of this answer indicates.