How to keep Unit tests and Integrations tests separate in pytest

Question

According to Wikipedia and various articles it is best practice to divide tests into Unit tests (run first) and Integration tests (run second), where Unit tests are typically very fast and should be run with every build in a CI environment, however Integration tests take longer to run and should be more of a daily run.

Is there a way to divide these in pytest? Most projects don't seem to have multiple test folders, so is there a way to make sure I only run Unit, Integration or both according to the situtation (CI vs daily builds)? When calculating test coverage, I assume I will have to run both.

Am I going about this the right way in attempting to divide the tests into these categories? Is there a good example somewhere of a project that has done this?

Solution 1

Yes, you can mark tests with the pytest.mark decorator.

Example:

def unit_test_1():
    # assert here

def unit_test_2():
    # assert here

@pytest.mark.integtest
def integration_test():
    # assert here

Now, from the command line, you can run pytest -m "not integtest" for only the unit tests, pytest -m integtest for only the integration test and plain pytest for all.

(You can also decorate your unit tests with pytest.mark.unit if you want, but I find that slightly tedious/verbose)

See the documentation for more information.

Solution 2

You can also structurally separate unit and integration tests into specific directories. Here is a sample file structure from A. Shaw's article Getting Started With Testing in Python:

With a structural approach, you:

1. do not need to manually mark various tests with attributes or @pytest.mark.
2. are not limited to a specific test runner. See examples below.

---

Examples

Here we run various test runners on integration tests alone. See the sample project/ directory in the figure above.

With unittest from the standard library:

λ python -m unittest discover -s tests/integration

With nose:

λ nose tests/integration

With pytest:

λ pytest tests/integration

Many test runners have an auto test-discovery mechanism that can find tests in sub-directories. This offers the choice to run all tests with ease, e.g.

λ cd <root_dir>
λ pytest project/

Comments

[Tom Malkin]

Ah this looks perfect Marcus! Do you agree that dividing the tests into unit and integration is a good idea for a python project using pytest? Is that what you do?

[Tom Malkin]

Nice, I like the idea of not being bound to a specific test runner. Would most test runners be able to discover both unit and integration if you just passed the parent "tests" folder?

[pylang]

Possibly. I still use the older nose test runner, which can also handle this kind of discovery. Consult the docs per library, as the command-line invocation will vary.

[gmds]

@Harlekuin if you mean in terms of metadata allowing them to be run separately, certainly; I too face the problem of larger-scale tests (and benchmarking) taking really long to run. If you mean in terms of directories, I would say that really depends on the project and the nature of the test. Hope I helped!

[Tom Malkin]

You definitely helped Marcus, I really appreciate it

[pylang]

I can now confirm pytest also has auto test-discovery.

[Doopy]

When separated like this, you can also run pytest tests/unit tests/integration to execute the integration tests after the unit tests.

[Rose]

In a case like this, how would you import my_app/helloworld.py inside tests/unit/ test_helloworld. I'm trying to follow this directory structure but imports can't seem to resolve for me.

[yoyoog]

Note that custom markers such as "integtest" should be registered into a pytest.ini file (see documentation) otherwise, a warning will be raised

[Harry]

It's not a good idea to have unit tests and integration tests in the same file. Ideally they should be in a separate folder structure (as mentioned in the other answer)

[Ievgen]

you cannot get test coverage from both

[joebeeson]

@levgen toss them under "tox" and change the '--coverage' arguments to generate coverage for both.