`py.test` and `__init__.py` files

Question

I thought py.test is "standalone" in a sense that it treats test_*.py files "as it is", and only imports modules specified in these files, with no respect to any surrounding files. It looks like I'm wrong. Here is my dialog with py.test:

$ ls
__init__.py    test_pytest.py
$ cat __init__.py 
$ cat test_pytest.py 
def test_pytest():
    assert True
$ py.test test_pytest.py 
========================================================= test session starts ==========================================================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 0 items / 1 errors 

================================================================ ERRORS ================================================================
___________________________________________________ ERROR collecting test_pytest.py ____________________________________________________
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.5-py2.7.egg/py/_path/local.py:529: in pyimport
>           mod = __import__(modname, None, None, ['__doc__'])
E           ImportError: No module named test_pytest
======================================================= 1 error in 0.01 seconds ========================================================
$ rm __init__.py 
$ py.test test_pytest.py 
========================================================= test session starts ==========================================================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 1 items 

test_pytest.py .

======================================================= 1 passed in 0.01 seconds =======================================================
$ 

How do I make py.test work and still have my __init__.py files?

UPDATE

In comments, Holger Krekel asked, what is the name of parent directory. And it turns out that I can reproduce the error above only having a certain parent directory name (e.g. the same name as one of the packages installed, like distutils). See here:

~/test_min 
$ tree
.
└── distutils
    ├── __init__.py
    └── test_pytest.py

1 directory, 2 files
~/test_min 
$ cat distutils/__init__.py 
~/test_min 
$ cat distutils/test_pytest.py 
def test_pytest():
    assert True
~/test_min 
$ py.test distutils/test_pytest.py 
======================== test session starts =========================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 0 items / 1 errors 

=============================== ERRORS ===============================
_____________ ERROR collecting distutils/test_pytest.py ______________
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.5-py2.7.egg/py/_path/local.py:529: in pyimport
>           mod = __import__(modname, None, None, ['__doc__'])
E           ImportError: No module named test_pytest
====================== 1 error in 0.01 seconds =======================
~/test_min 
$ rm distutils/__init__.py 
~/test_min 
$ py.test distutils/test_pytest.py 
======================== test session starts =========================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 1 items 

distutils/test_pytest.py .

====================== 1 passed in 0.01 seconds ======================
~/test_min 
$ touch __init__.py
~/test_min 
$ ls
__init__.py distutils
~/test_min 
$ touch distutils/__init__.py
~/test_min 
$ py.test distutils/test_pytest.py 
======================== test session starts =========================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 1 items 

distutils/test_pytest.py .

====================== 1 passed in 0.02 seconds ======================
~/test_min 
$ rm __init__.py 
~/test_min 
$ py.test distutils/test_pytest.py 
======================== test session starts =========================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 0 items / 1 errors 

=============================== ERRORS ===============================
_____________ ERROR collecting distutils/test_pytest.py ______________
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/py-1.4.5-py2.7.egg/py/_path/local.py:529: in pyimport
>           mod = __import__(modname, None, None, ['__doc__'])
E           ImportError: No module named test_pytest
====================== 1 error in 0.01 seconds =======================
~/test_min 
$ mv distutils foobar
~/test_min 
$ py.test foobar/test_pytest.py 
======================== test session starts =========================
platform darwin -- Python 2.7.2 -- pytest-2.1.3
collected 1 items 

foobar/test_pytest.py .

====================== 1 passed in 0.01 seconds ======================
~/test_min 
$ 

Hope this additional information will help.

Solution 1

Looks like py.test is using py._path.pyimport to open your file. If there is a __init__.py file in the directory, it treats your file as a module, otherwise it opens the file. Long story short, delete the __init__.py or put your tests in another directory outside your project code (<--- good idea).

https://py.readthedocs.io/en/latest/path.html#py._path.local.LocalPath.pyimport

Solution 2

I really suggest you to rename the directory to something not called "distutils". Why ? Because you are overriding an existing module. When "import distutils" or "from distutils import *" appear in the script (from another import or your own python file), it will prefer your directory instead the system one. If the module distutils have been already loaded before, your distutils will not be loaded, because the symbol already exists in global().

It would be really simpler to rename that directory (like tests) instead of trying to fight with py.text / python internals.

Solution 3

You can tell pytest to ignore specific files or glob patterns as described here. Put a conftest.py file in the root directory of your project that lists the files you want pytest to ignore:

However, many projects will have a setup.py which they don’t want to be imported. Moreover, there may files only importable by a specific python version. For such cases you can dynamically define files to be ignored by listing them in a conftest.py file:

 # content of conftest.py
 import sys

 collect_ignore = ["setup.py"]

Comments

[Vladimir Keleshev]

distutils is just an example to reproduce an error. I have problems with my own module with different name.

[tito]

Ok, one last question. When you add init.py on '.', the name of the '.' directory is taken as a module name too. What is it in your case ? (a side note also, even if by looking at your tests, i don't think you're hit this bug, but don't forget about the pyc generation, when removing py, pyc is still exist and importable.)

[Vladimir Keleshev]

if you mean, when I do $ touch __init__.py? It is done in ~/test_min as you can see from the line above. Also, I don't have any pyc files, as you can see from the tree command.

[Vladimir Keleshev]

I awarded you the bounty so that it will not automatically be given to tito, since his answer is incorrect. Though, I hoped to solve this problem without changing file structure.