Using pytest with a src layer

python pytest python-packaging

14,142

Solution 1

Adjusting the PYTHONPATH (as suggested in the comments) is one possibility to solve the import issue. Another is adding an empty conftest.py file in the src directory:

$ touch src/conftest.py

and pytest will add src to sys.path. This is a simple way to trick pytest into adding codebase to sys.path.

However, the src layout is usually selected when you intend to build a distribution, e.g. providing a setup.py with (in this case) explicitly specifying the root package dir:

from setuptools import find_packages, setup


setup(
    ...
    package_dir={'': 'src'},
    packages=find_packages(where='src'),
    ...
)

and installing the package in the development mode (via python setup.py develop or pip install --editable .) while you're still developing it. This way, your package my_package is correctly integrated in the Python's site packages structure and there's no need to fiddle with PYTHONPATH.

Solution 2

PYTHONPATH updates weren't working for me when using github actions (known prob). Using this pytest-pythonpath install with pytest.ini file worked for me instead:

pip install pytest-pythonpath # accompany with python_path in pytest.ini, so PYTHONPATH is updated with location for modules under test

With this, basic 'pytest' command happily found all tests in subdirs, and found modules under test based on my pytest.ini (set to match source folders in pycharm)

14,142

Arne

I am interested in Cloud Services and Machine Learning, and generally programming in Python. When I am not writing code you can find me cooking, running, hiking, reading, doodling, writing, or loafing.

Updated on June 10, 2022

Comments

Arne almost 2 years
pytest recommends including an additional directory to separate the source code within a project:
```
my_package
├── src  # <-- no __init__.py on this layer
│   └── my_package
│       ├── __init__.py
│       └── util_module
│           ├── __init__.py
│           └── utils.py
└── tests
    ├── __init__.py
    └── test_util_module
        ├── __init__.py
        └── test_utils.py
```
Sadly, they say nothing^[1] about how imports in the test code should work in such a case, which work for my IDE just fine in this naive example^[2], but causes the following error with pytest:
```
my_package $ pytest

====================== test session starts ======================
platform linux -- Python 3.6.4, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/user/workspace/my_package, inifile:
collected 0 items / 1 errors     
                                                                                                                                                                      
============================ ERRORS =============================
___ ERROR collecting tests/test_util_module/test_utils.py ___
ImportError while importing test module '/home/user/workspace/my_package/tests/test_util_module/test_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_util_module/test_utils.py:1: in <module>
    from test.test_module.some_file import starify
E   ModuleNotFoundError: No module named 'my_package.util_module'
!!!! Interrupted: 1 errors during collection !!!!!
```
I can fix the issue by changing the import of the test to from src.my_package.util_module.utils import starify, but then my IDE complaints about the src part being redundant, so I'd like to keep it out.

^{[1]_{: Not the case any more. As of version 3.7.3, pytest recommends the editable install also featured in @hoefling's answer at the top of its good practices.}}

^{[2]_{: Setup is virtualenv env -p python3.6; source env/bin/activate; pip install pytest}}
- Tom Dalton almost 6 years
  
  Are your test directories plain directories or are they packages (containing an __init__.py file)?
- Arne almost 6 years
  
  @TomDalton Every dir except the root dir and the src dir is a package. Just a sec, I'm posting a tree .
- Tom Dalton almost 6 years
  
  If you export PYTHONPATH=".:src/" before running the tests, does that change anything?
- Tom Dalton almost 6 years
  
  (I'm assuming you run the tests from the root directory)
- Tom Dalton almost 6 years
  
  e.g. test $ pytest should be my_package $ pytest?
- Arne almost 6 years
  
  @TomDalton yes, i am renaming the example right now.. it was a bit confusing I guess. Also, exporting the PYTHONPATH works as well
- Tom Dalton almost 6 years
  
  Now the pytest output doesn't match your directory tree :D
- Arne almost 6 years
  
  Alright, I reproduced structure and all, it's reproducible as it is now in the post. Exporting PYTHONPATH solves the issue, and it is certainly better than adding src. in front of all test suite imports, but I'd hope for a slightly cleaner solution that would make the repo testable as it is now.
- Nils Werner almost 6 years
  
  Did you install my_package, i.e. does it contain a setup.py?
- Arne almost 6 years
  
  @NilsWerner No, and no. Should I always do that before testing? I'd rather run unit tests before build in the CI, but I might be wrong.
- Nils Werner almost 6 years
  
  Yes, of course you: 1) need to build the code in order to test it. 2) make the package importable by installing it
- Tom Dalton almost 6 years
  
  "of course you: 1) need to build the code in order to test it." - In the general case this is not true ,and for a lot of people (myself included) it is not desireable
- Tom Dalton almost 6 years
  
  And "of course you: ... 2) make the package importable by installing it", packages can be importable without being installed anywhere as long as python's package search path is configured correctly (e.g. by setting pythonpath or other tricks).
- wim almost 6 years
  
  @TomDalton And the right way to configure the search path correctly is by ... (drum roll) ... installing the package!
hoefling almost 6 years

Also, SO has an excellent answer on this topic.
Arne almost 6 years

I tested both solutions and both work =) thanks for the help!
Hunger over 2 years

This official doc might be helpful: pytest import mechanisms and sys.path/PYTHONPATH