coverage.py does not cover script if py.test executes it from another directory

Question

I got a python script which takes command line arguments, working with some files. I'm writing succeeding tests with py.test putting this script through its paces, executing it with subprocess.call.

Now I want to analyze code coverage with coverage.py. Coverage, when used via the pytest-cov plugin (which has subprocess-handling built-in), does not see/cover my script when it is called from a temporary testing directory created with py.test's tmpdir fixture. Coverage does see my script when it's called in the directory it resides in (and the filename argument points to a remote path).

In both situations, my tests pass! Coverage 3.6, pytest-2.3.5, pytest-cov 1.6, all from PyPi.

Question: How can I get coverage to recognize my script even if it's executed in another directory? Is this a bug in coverage, or something which is just not possible to do? Would be surprised if the latter, after all, tmpdir is a stock mechanism of py.test...

Minimal example:

I got a script my_script.py which just echoes the contents of a file arg_file.txt supplied via command-line argument. In two different tests, this is once called in a tmpdir, and once in the script's location. Both tests pass, but the in the tmpdir test, I get no coverage information!

Test run:

~/pytest_experiment$ py.test -s
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 2 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-52/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
--Contents of arg_file.txt--

.
tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-52/test_11
Running in directory /tmp/pytest-52/test_11
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

.

================================= 2 passed in 0.06 seconds =================================

Coverage:

~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py .
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

The files are here: https://gist.github.com/bilderbuchi/6412754

Edit: Interstingly, when running the coverage tests with -s, too, there's more curious output - coverage warns that No data was collected, when obviously it was collected, and in the tmpdir test warns that Module my_script.py was never imported.??

~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-63/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-63/test_10/arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: No data was collected.
.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-64/test_10
Running in directory /tmp/pytest-64/test_10
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

Solution 1

This turned out to be a problem of relative paths confusing coverage when the measured script is run from another directory. Coverage result files ended up in that directory, instead of the root directory of the project.

To solve this, I stopped using pytest-cov, and used pure coverage instead. I used full paths instead of relative paths wherever relevant.

So, e.g. define the environment variable necessary to enable subprocess coverage via export COVERAGE_PROCESS_START=/full/path/to/.coveragerc. In the .coveragerc, the coverage result file is specified via

     [run]
     data_file = /full/path/to/.coverage

and any --source and --include options should use full paths, too. Then it was possible to get correct coverage measurement.

Solution 2

I encountered the same issue when calling "py.test --cov ..." from tox. I found a hint on this page: http://blog.ionelmc.ro/2014/05/25/python-packaging/ even though it does not mention this explicitly. Using "--develop" for tox will make sure that coverage data gathering is called from the same directory as coverage analysis. This section in tox.ini made it work for me to have a test environment for coverage:

[tox]
envlist = ...,py34,cov

[testenv:cov]
# necessary to make cov find the .coverage file
# see http://blog.ionelmc.ro/2014/05/25/python-packaging/
usedevelop = true
commands = py.test --cov=<MODULE_NAME>
deps = pytest pytest-cov

Solution 3

according to this blog: https://thomas-cokelaer.info/blog/2017/01/pytest-cov-collects-no-data-on-travis/

You should add all the __init__.py files in tests folder! This solution works for me.