How to stop all tests from inside a test or setUp using unittest?

Question

I'm extending the python 2.7 unittest framework to do some function testing. One of the things I would like to do is to stop all the tests from running inside of a test, and inside of a setUpClass() method. Sometimes if a test fails, the program is so broken it is no longer of any use to keep testing, so I want to stop the tests from running.

I noticed that a TestResult has a shouldStop attribute, and a stop() method, but I'm not sure how to get access to that inside of a test.

Does anyone have any ideas? Is there a better way?

Solution 1

Here's another answer I came up with after a while:

First, I added a new exception:

class StopTests(Exception):
"""
Raise this exception in a test to stop the test run.

"""
    pass

then I added a new assert to my child test class:

def assertStopTestsIfFalse(self, statement, reason=''):
    try:
        assert statement            
    except AssertionError:
        result.addFailure(self, sys.exc_info())

and last I overrode the run function to include this right below the testMethod() call:

except StopTests:
    result.addFailure(self, sys.exc_info())
    result.stop()

I like this better since any test now has the ability to stop all the tests, and there is no cpython-specific code.

Solution 2

In case you are interested, here is a simple example how you could make a decision yourself about exiting a test suite cleanly with py.test:

# content of test_module.py
import pytest
counter = 0
def setup_function(func):
    global counter
    counter += 1
    if counter >=3:
        pytest.exit("decided to stop the test run")

def test_one():
    pass
def test_two():
    pass
def test_three():
    pass

and if you run this you get:

$ pytest test_module.py 
============== test session starts =================
platform linux2 -- Python 2.6.5 -- pytest-1.4.0a1
test path 1: test_module.py

test_module.py ..

!!!! Exit: decided to stop the test run !!!!!!!!!!!!
============= 2 passed in 0.08 seconds =============

You can also put the py.test.exit() call inside a test or into a project-specific plugin.

Sidenote: py.test natively supports py.test --maxfail=NUM to implement stopping after NUM failures.

Sidenote2: py.test has only limited support for running tests in the traditional unittest.TestCase style.

Solution 3

Currently, you can only stop the tests at the suite level. Once you are in a TestCase, the stop() method for the TestResult is not used when iterating through the tests.

Somewhat related to your question, if you are using python 2.7, you can use the -f/--failfast flag when calling your test with python -m unittest. This will stop the test at the first failure.

See 25.3.2.1. failfast, catch and buffer command line options

You can also consider using Nose to run your tests and use the -x, --stop flag to stop the test early.

Solution 4

In the test loop of unittest.TestSuite, there is a break condition at the start:

class TestSuite(BaseTestSuite):

    def run(self, result, debug=False):
        topLevel = False
        if getattr(result, '_testRunEntered', False) is False:
            result._testRunEntered = topLevel = True

        for test in self:
            if result.shouldStop:
                break

So I am using a custom test suite like this:

class CustomTestSuite(unittest.TestSuite):
    """ This variant registers the test result object with all ScriptedTests,
        so that a failed Loign test can abort the test suite by setting result.shouldStop
        to True
    """
    def run(self, result, debug=False):
        for test in self._tests:
            test.result = result

        return super(CustomTestSuite, self).run(result, debug)

with a custom test result class like this:

class CustomTestResult(TextTestResult):
    def __init__(self, stream, descriptions, verbosity):
        super(CustomTestResult, self).__init__(stream, descriptions, verbosity)
        self.verbosity = verbosity
        self.shouldStop = False

and my test classes are like:

class ScriptedTest(unittest.TestCase):
    def __init__(self, environment, test_cfg, module, test):
        super(ScriptedTest, self).__init__()
        self.result = None

Under certain conditions, I then abort the test suite; for example, the test suite starts with a login, and if that fails, I do not have to try the rest:

    try:
        test_case.execute_script(test_command_list)
    except AssertionError as e:
        if test_case.module == 'session' and test_case.test == 'Login':
            test_case.result.shouldStop = True
            raise TestFatal('Login failed, aborting test.')
        else:
            raise sys.exc_info()

Then I use the test suite in the following way:

    suite = CustomTestSuite()

    self.add_tests(suite)

    result = unittest.TextTestRunner(verbosity=self.environment.verbosity, stream=UnitTestLoggerStream(self.logger),
                                     resultclass=CustomTestResult).run(suite)

I'm not sure if there is a better way to do it, but it behaves correctly for my tests.

Solution 5

Though you won't get the usual test reports of the tests run so far, a very easy way to stop the test run from within a TestCase method is simply to raise KeyboardInterrupt inside the method.

You can see how only KeyboardInterrupt is allowed to bubble up inside unittest's test runner by looking at CPython's code here inside testPartExecutor().

View more solutions