How do I handle multiple asserts within a single Python unittest?

Question

This is a problem that came up when performing a single test that had multiple independent failure modes, due to having multiple output streams. I also wanted to show the results of asserting the data on all those modes, regardless of which failed first. Python's unittest has no such feature outside of using a Suite to represent the single test, which was unacceptable since my single test always needed to be run as a single unit; it just doesn't capture the nature of the thing.

A practical example is testing an object that also generates a log. You want to assert the output of it's methods, but you also want to assert the log output. The two outputs require different tests, which can be neatly expressed as two of the stock asserts expressions, but you also don't want the failure of one to hide the possible failure of the other within the test. So you really need to test both at the same time.

I cobbled together this useful little widget to solve my problem.

def logFailures(fnList):
    failurelog = []
    for fn in fnList:
        try:
            fn()
        except AssertionError as e:
            failurelog.append("\nFailure %d: %s" % (len(failurelog)+1,str(e)))

    if len(failurelog) != 0:
        raise AssertionError(
            "%d failures within test.\n %s" % (len(failurelog),"\n".join(failurelog))
        )

Which is used like so:

def test__myTest():
    # do some work here
    logFailures([
        lambda: assert_(False,"This test failed."),
        lambda: assert_(False,"This test also failed."),
    ])

The result is that logFailures() will raise an exception that contains a log of all the assertions that were raised in methods within the list.

The question: While this does the job, I'm left wondering if there's a better way to handle this, other than having to go to the length of creating nested suites of tests and so forth?

Solution 1

With using a subtest, execution would not stop after the first failure https://docs.python.org/3/library/unittest.html#subtests

Here is example with two fail asserts:

class TestMultipleAsserts(unittest.TestCase):

    def test_multipleasserts(self):
        with self.subTest():
            self.assertEqual(1, 0)
        with self.subTest():
            self.assertEqual(2, 0)

Output will be:

======================================================================
FAIL: test_multipleasserts (__main__.TestMultipleAsserts) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 9, in test_multipleasserts
    self.assertEqual(1, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_multipleasserts (__main__.TestMultipleAsserts) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 11, in test_multipleasserts
    self.assertEqual(2, 0)
AssertionError: 2 != 0

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=2)

You can easy wrap subtest as following

class MyTestCase(unittest.TestCase):
    def expectEqual(self, first, second, msg=None):
        with self.subTest():
            self.assertEqual(first, second, msg)

class TestMA(MyTestCase):
    def test_ma(self):
        self.expectEqual(3, 0)
        self.expectEqual(4, 0)

Solution 2

I disagree with the dominant opinion that one should write a test method for each assertion. There are situations where you want to check multiple things in one test method. Here is my answer for how to do it:

# Works with unittest in Python 2.7
class ExpectingTestCase(unittest.TestCase):
    def run(self, result=None):
        self._result = result
        self._num_expectations = 0
        super(ExpectingTestCase, self).run(result)

    def _fail(self, failure):
        try:
            raise failure
        except failure.__class__:
            self._result.addFailure(self, sys.exc_info())

    def expect_true(self, a, msg):
        if not a:
            self._fail(self.failureException(msg))
        self._num_expectations += 1

    def expect_equal(self, a, b, msg=''):
        if a != b:
            msg = '({}) Expected {} to equal {}. '.format(self._num_expectations, a, b) + msg
            self._fail(self.failureException(msg))
        self._num_expectations += 1

And here are some situations where I think it's useful and not risky:

1) When you want to test code for different sets of data. Here we have an add() function and I want to test it with a few example inputs. To write 3 test methods for the 3 data sets means repeating yourself which is bad. Especially if the call was more elaborate.:

class MyTest(ExpectingTestCase):
    def test_multiple_inputs(self):
        for a, b, expect in ([1,1,2], [0,0,0], [2,2,4]):
            self.expect_equal(expect, add(a,b), 'inputs: {} {}'.format(a,b))

2) When you want to check multiple outputs of a function. I want to check each output but I don't want a first failure to mask out the other two.

class MyTest(ExpectingTestCase):
    def test_things_with_no_side_effects(self):
        a, b, c = myfunc()
        self.expect_equal('first value', a)
        self.expect_equal('second value', b)
        self.expect_equal('third value', c)

3) Testing things with heavy setup costs. Tests must run quickly or people stop using them. Some tests require a db or network connection that takes a second which would really slow down your test. If you are testing the db connection itself, then you probably need to take the speed hit. But if you are testing something unrelated, we want to do the slow setup once for a whole set of checks.

Solution 3

This feels like over-engineering to me. Either:

• Use two asserts in one test case. If the first assert fails, it's true, you won't know whether the second assert passed or not. But you're going to fix the code anyway, so fix it, and then you'll find out if the second assert passed.

• Write two tests, one to check each condition. If you fear duplicated code in the tests, put the bulk of the code in a helper method that you call from the tests.

Comments

[smheidrich]

Something like this should be present in unit test frameworks by default. Does anybody know one that has this functionality?

[TCulp]

It is important to note that this only applies to Python 3.4 and later, according to your link

[Tim Pederick]

Exactly what was asked for, and not coincidentally, just what I needed. I hope the asker comes back and accepts this.

[user3611]

is there a way to wrap the subtest with flexible assert statements, not just assertEqual? i.e., to use, assertTrue, assertFalse, etc.