Python setup.py: How to get find_packages() to identify packages in subdirectories

Question

I'm trying to create a setup.py file where find_packages() recursively finds packages. In this example, foo, bar, and baz are all modules that I want to be installed and available on the python path. For example, I want to be able to do import foo, bar, baz. The bar-pack and foo-pack are just regular non-python directories that will contain various support files/dirs (such as tests, READMEs, etc. specific to the respective module).

├── bar-pack
│   └── bar
│       └── __init__.py
├── baz
│   └── __init__.py
├── foo-pack
│   └── foo
│       └── __init__.py
├── setup.py

Then say that setup.py is as follows:

from setuptools import setup, find_packages
setup(
    name="mypackage",
    version="0.1",
    packages=find_packages(),
)

However, when I run python setup.py install or python setup.py sdist, only the baz directory is identified and packaged.

I can simplify it down further, and run the following command, but again, only baz is identified.

python -c "from setuptools import setup, find_packages; print(find_packages())"
['baz']

Do you know how I might extend the search path (or manually hard-code the search path) of the find_packages()?

Any help is appreciated.

Solution

This is like using the src-layout for the "foo" and "bar" packages, but the flat layout for "baz". It's possible, but requires some custom configuration in the setup.py.

Setuptools' find_packages supports a "where" keyword (docs), you can use that.

setup(
    ...
    packages=(
        find_packages() +
        find_packages(where="./bar-pack") +
        find_packages(where="./foo-pack")
    ),
    ...
)

Since find_packages returns a plain old list, you could also just list your packages manually, and that's arguably easier / less magical.

setup(
    ...
    packages=["baz", "bar", "foo"],
    ...
)

The non-standard directory structure means you'll also want to specify the package_dir structure for distutils, which describes where to put the installed package(s).

Piecing it all together:

setup(
    name="mypackage",
    version="0.1",
    packages=["baz", "bar", "foo"],
    package_dir={
        "": ".",
        "bar": "./bar-pack/bar",
        "foo": "./foo-pack/foo",
    },
)

The above installer will create this directory structure in site-packages:

.venv/lib/python3.9/site-packages
├── bar
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── baz
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
├── foo
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-39.pyc
└── mypackage-0.1.dist-info
    ├── INSTALLER
    ├── METADATA
    ├── RECORD
    ├── REQUESTED
    ├── WHEEL
    ├── direct_url.json
    └── top_level.txt

Comments

[Sunil]

Interestingly for me, folders that were not packaged into wheel are the folders that do not have __init__.py file. As soon as I placed the empty __init__.py file in the folder which needs to go into wheel, folders/files were rolled into wheel package.

[chrisinmtown]

Is it safe to say that the where and the package_dir pieces are both needed? When I tried specifying find_packages() + find_packages(where="./bar-pack") without the package_dir structure, pip coughed up an error error: package directory 'bar-pack' does not exist. I think you're right that just specifying the package names directly (avoiding find_packages) yields a more understandable solution.

[wim]

@chrisinmtown Yes, I think they are both needed. One advantage of find_packages is that if you list packages manually it's easy to forget you need to add sub-packages - e.g. if there was a bar/subbar/__init__.py, you'd have to list both "bar" and "bar.subbar" as packages.