How do I run all Python unit tests in a directory?

Question

I have a directory that contains my Python unit tests. Each unit test module is of the form test_*.py. I am attempting to make a file called all_test.py that will, you guessed it, run all files in the aforementioned test form and return the result. I have tried two methods so far; both have failed. I will show the two methods, and I hope someone out there knows how to actually do this correctly.

For my first valiant attempt, I thought "If I just import all my testing modules in the file, and then call this unittest.main() doodad, it will work, right?" Well, turns out I was wrong.

import glob
import unittest

testSuite = unittest.TestSuite()
test_file_strings = glob.glob('test_*.py')
module_strings = [str[0:len(str)-3] for str in test_file_strings]

if __name__ == "__main__":
     unittest.main()

This did not work, the result I got was:

$ python all_test.py 

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

For my second try, I though, ok, maybe I will try to do this whole testing thing in a more "manual" fashion. So I attempted to do that below:

import glob
import unittest

testSuite = unittest.TestSuite()
test_file_strings = glob.glob('test_*.py')
module_strings = [str[0:len(str)-3] for str in test_file_strings]
[__import__(str) for str in module_strings]
suites = [unittest.TestLoader().loadTestsFromName(str) for str in module_strings]
[testSuite.addTest(suite) for suite in suites]
print testSuite 

result = unittest.TestResult()
testSuite.run(result)
print result

#Ok, at this point I have a result
#How do I display it as the normal unit test command line output?
if __name__ == "__main__":
    unittest.main()

This also did not work, but it seems so close!

$ python all_test.py 
<unittest.TestSuite tests=[<unittest.TestSuite tests=[<unittest.TestSuite tests=[<test_main.TestMain testMethod=test_respondes_to_get>]>]>]>
<unittest.TestResult run=1 errors=0 failures=0>

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

I seem to have a suite of some sort, and I can execute the result. I am a little concerned about the fact that it says I have only run=1, seems like that should be run=2, but it is progress. But how do I pass and display the result to main? Or how do I basically get it working so I can just run this file, and in doing so, run all the unit tests in this directory?

Solution 1

With Python 2.7 and higher you don't have to write new code or use third-party tools to do this; recursive test execution via the command line is built-in. Put an __init__.py in your test directory and:

python -m unittest discover <test_directory>
# or
python -m unittest discover -s <directory> -p '*_test.py'

You can read more in the python 2.7 or python 3.x unittest documentation.

---

Update for 2021:

Lots of modern python projects use more advanced tools like pytest. For example, pull down matplotlib or scikit-learn and you will see they both use it.

It is important to know about these newer tools because when you have more than 7000 tests you need:

• more advanced ways to summarize what passes, skipped, warnings, errors
• easy ways to see how they failed
• percent complete as it is running
• total run time
• ways to generate a test report
• etc etc

Solution 2

In python 3, if you're using unittest.TestCase:

• You must have an empty (or otherwise) __init__.py file in your test directory (must be named test/)
• Your test files inside test/ match the pattern test_*.py. They can be inside a subdirectory under test/, and those subdirs can be named as anything.

Then, you can run all the tests with:

python -m unittest

Done! A solution less than 100 lines. Hopefully another python beginner saves time by finding this.

Solution 3

You could use a test runner that would do this for you. nose is very good for example. When run, it will find tests in the current tree and run them.

Updated:

Here's some code from my pre-nose days. You probably don't want the explicit list of module names, but maybe the rest will be useful to you.

testmodules = [
    'cogapp.test_makefiles',
    'cogapp.test_whiteutils',
    'cogapp.test_cogapp',
    ]

suite = unittest.TestSuite()

for t in testmodules:
    try:
        # If the module defines a suite() function, call it to get the suite.
        mod = __import__(t, globals(), locals(), ['suite'])
        suitefn = getattr(mod, 'suite')
        suite.addTest(suitefn())
    except (ImportError, AttributeError):
        # else, just load all the test cases from the module.
        suite.addTest(unittest.defaultTestLoader.loadTestsFromName(t))

unittest.TextTestRunner().run(suite)

Solution 4

This is now possible directly from unittest: unittest.TestLoader.discover.

import unittest
loader = unittest.TestLoader()
start_dir = 'path/to/your/test/files'
suite = loader.discover(start_dir)

runner = unittest.TextTestRunner()
runner.run(suite)

Solution 5

Well by studying the code above a bit (specifically using TextTestRunner and defaultTestLoader), I was able to get pretty close. Eventually I fixed my code by also just passing all test suites to a single suites constructor, rather than adding them "manually", which fixed my other problems. So here is my solution.

import glob
import unittest

test_files = glob.glob('test_*.py')
module_strings = [test_file[0:len(test_file)-3] for test_file in test_files]
suites = [unittest.defaultTestLoader.loadTestsFromName(test_file) for test_file in module_strings]
test_suite = unittest.TestSuite(suites)
test_runner = unittest.TextTestRunner().run(test_suite)

Yeah, it is probably easier to just use nose than to do this, but that is besides the point.

View more solutions

Comments

[Corey Porter]

Is the advantage of this approach over just explicitly importing all of your test modules in to one test_all.py module and calling unittest.main() that you can optionally declare a test suite in some modules and not in others?

[Jesse Webb]

I tried out nose and it works perfectly. It was easy to install and run in my project. I was even able to automate it with a few lines of script, running inside a virtualenv. +1 for nose!

[Larry Cai]

this is available for python2.7 only, I guess

[Larry Cai]

good, it works fine for the current directory, how to invoke the sub-directly ?

[rds]

@larrycai Maybe, I am usually on Python 3, sometimes Python 2.7. The question was not tied to a specific version.

[zinking]

problems include: ImportError: Start directory is not importable:

[Peter Kofler]

Larry, see the new answer (stackoverflow.com/a/24562019/104143) for recursive test discovery

[Dunes]

I'm on Python 3.4 and discover returns a suite, making the loop redundant.

[Two-Bit Alchemist]

From the unittest.TestLoader docs: "Normally, there is no need to create an instance of this class; the unittest module provides an instance that can be shared as unittest.defaultTestLoader." Also since TestSuite accepts an iterable as an argument, you can construct said iterable in a loop to avoid repeating loader.loadTestsFromTestCase.

[Two-Bit Alchemist]

For future Larry's: "Many new features were added to unittest in Python 2.7, including test discovery. unittest2 allows you to use these features with earlier versions of Python."

[user686249]

At least with Python 2.7.8 on Linux neither command line invocation gives me recursion. My project has several subprojects whose unit tests live in respective "unit_tests/<subproject>/python/" directories. If I specify such a path the unit tests for that subproject are run, but with just "unit_tests" as test directory argument no tests are found (instead of all tests for all subprojects, as I hoped). Any hint?

[chiffa]

Not always doable: sometimes importing structure of the project can lead to nose getting confused if it tries to run the imports on modules.

[demented hedgehog]

@Two-Bit Alchemist your second point in particular is nice. I'd change the code to include but I can't test it. (First mod would make it look like too much like Java for my liking.. though I realize I'm being irrational (screw them an their camel case variable names)).

[Emil Stenström]

About recursion: The first command without a <test_directory> defaults to "." and recurses to submodules. That is, all tests directories you want discovered needs to have a init.py. If they do, they will get found by the discover command. Just tried it, it worked.

[MarkII]

This is my fav, very clean. Was able to package this and make it an argument in my regular command line.

[Kurt Peek]

Note that nose has been "in maintenance mode for the past several years" and it is currently advised to use nose2, pytest, or just plain unittest / unittest2 for new projects.

[Corey Goldberg]

most of this answer has nothing to do with test discovery (i.e logging, etc). Stack Overflow is for answering questions, not showing off unrelated code.

[Charlie Parker]

did you ever try running the tests from an test instance object?

[Charlie Parker]

did you ever try running the tests from an test instance object?

[Charlie Parker]

did you ever try running the tests from an test instance object?

[Stefan]

The names of all test files and test methods should start with "test_". Otherwise the command "Run as -> Python unit test" wont find them.

[simkusr]

I have tried this method also, have couple tests, but works perfectly. Excellent!!! But I'm curious I have only 4 tests. Together they run 0.032s, but when I use this method to run them all, i get result .... ------------------------------------------------------------‌​---------- Ran 4 tests in 0.000s OK Why? The difference, where it comes from?

[Dustin Michels]

I am having trouble running a file that looks like this from the command line. How should it be invoked?

[slaughter98]

python file.py

[Shawabawa]

Note that by default it only searches for tests in filenames beginning with "test"

[tmck-code]

That's correct, the original question referred to the fact that "Each unit test module is of the form test_*.py.", so this answer in direct reply. I've now updated the answer to be more explicit

[qurban]

This works for me even if I don't specify -s and -p args. Just write python -m unittest discover

[jjwdesign]

Worked flawlessly! Just set it in your test/ dir and then set the start_id = "./" . IMHO, this answer is now (Python 3.7) the accepted way!

[Yan King Yin]

My tests seem to require -m pytest to run. The test functions are defined in the test file but there are no explicit function calls. If I try unittest discover it finds zero tests.

[Jeremy Cochoy]

Thanks, that what was missing for me to use Travis Bear's answer.

[Cameron Hudson]

Could you please update this for Python3? __init__.py files are deprecated in favor of namespace packages.

[bb1950328]

You can change the last line to ´res = runner.run(suite); sys.exit(0 if res.wasSuccessful() else 1)´ if you want a correct exit code

[Morris]

In test/ folder, OK: python3 ./deeply/nested/test_example.py But this shows import errors: python3 -m unittest Why?

[nope]

I also needed to add the init.py file to each subfolder for it to work, otherwise great. Thanks!

[eager2learn]

Can you update your answer to include that the subdirectories need to be packages as well, so that you need to add an init.py file to the subdirectories inside the test directory?

[Grodriguez]

This works but subdir does not need to be named "test"