Python unittest: cancel all tests if a specific test fails

Question

I am using unittest to test my Flask application, and nose to actually run the tests.

My first set of tests is to ensure the testing environment is clean and prevent running the tests on the Flask app's configured database. I'm confident that I've set up the test environment cleanly, but I'd like some assurance of that without running all the tests.

import unittest

class MyTestCase(unittest.TestCase):
    def setUp(self):
        # set some stuff up
        pass

    def tearDown(self):
        # do the teardown
        pass

class TestEnvironmentTest(MyTestCase):
    def test_environment_is_clean(self):
        # A failing test
        assert 0 == 1

class SomeOtherTest(MyTestCase):
    def test_foo(self):
        # A passing test
        assert 1 == 1

I'd like the TestEnvironmentTest to cause unittest or noseto bail if it fails, and prevent SomeOtherTest and any further tests from running. Is there some built-in method of doing so in either unittest (preferred) or nose that allows for that?

Solution 1

In order to get one test to execute first and only halt execution of the other tests in case of an error with that test, you'll need to put a call to the test in setUp() (because python does not guarantee test order) and then fail or skip the rest on failure.

I like skipTest() because it actually doesn't run the other tests whereas raising an exception seems to still attempt to run the tests.

def setUp(self):
    # set some stuff up
    self.environment_is_clean()

def environment_is_clean(self):
    try:
        # A failing test
        assert 0 == 1
    except AssertionError:
        self.skipTest("Test environment is not clean!")

Solution 2

For your use case there's setUpModule() function:

If an exception is raised in a setUpModule then none of the tests in the module will be run and the tearDownModule will not be run. If the exception is a SkipTest exception then the module will be reported as having been skipped instead of as an error.

Test your environment inside this function.

Solution 3

You can skip entire test cases by calling skipTest() in setUp(). This is a new feature in Python 2.7. Instead of failing the tests, it will simply skip them all.

Solution 4

I'm not quite sure whether it fits your needs, but you can make the execution of a second suite of unittests conditional on the result of a first suite of unittests:

envsuite = unittest.TestSuite()
moretests = unittest.TestSuite()
# fill suites with test cases ...
envresult = unittest.TextTestRunner().run(envsuite)
if envresult.wasSuccessful():
    unittest.TextTestRunner().run(moretests)

View more solutions

Comments

[Colin Dunklau]

That's a nice feature, but the final result is OK, when it should whine about errors.

[kichik]

That's what you have TestEnvironmentTest failure for. If you just want them to fail, what's wrong with the code you already have?

[Dennis Golomazov]

Or in some cases, setUpClass() will also work If an exception is raised during a setUpClass then the tests in the class are not run and the tearDownClass is not run. docs.python.org/2/library/…

[Stevoisiak]

If setUpModule() is defined as a function outside the TestCase class, how would you make an assertion?