Unit test script returns exit code = 0 even if tests fail

Question

My testing script looks as follows:

import os
import sys
from unittest import defaultTestLoader as loader, TextTestRunner

path_to_my_project = os.path.dirname(os.path.abspath(__file__)) + '/../'
sys.path.insert(0, path_to_my_project)

suite = loader.discover('my_project')
runner = TextTestRunner()
runner.run(suite)

If I run this script, the output is:

$ python3 runtest.py
.....F.....
======================================================================
FAIL: test_insert (fate.test.test_operators.OperatorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chiel/Projects/tfate/libs/fate/../fate/test/test_operators.py", line 16, in test_insert
    self.assertEqual(expected, self.session.text[:14])
AssertionError: 'Foo import sys$' != 'Foo import sys'
- Foo import sys$
?               -
+ Foo import sys


----------------------------------------------------------------------
Ran 12 tests in 0.030s

FAILED (failures=1)

And exit code zero:

$ echo $?
0

However, the Python documentation states that "By default main calls sys.exit() with an exit code indicating success or failure of the tests run."

What is wrong with my script?

Solution 1

The code is not using unittest.main. You need to check the result using TestResult.wasSuccessful and call sys.exit manually.

import sys

....

ret = not runner.run(suite).wasSuccessful()
sys.exit(ret)

Solution 2

I had some trouble getting TextTestRunner results. For those like me, here is how it works:

"""Run all tests inside of *_test.py modules located in the same directory."""


import sys
import unittest


if __name__ == '__main__':
    test_suite = unittest.defaultTestLoader.discover('.', '*_test.py')
    test_runner = unittest.TextTestRunner(resultclass=unittest.TextTestResult)
    result = test_runner.run(test_suite)
    sys.exit(not result.wasSuccessful())