pytest: How to get a list of all failed tests at the end of the session? (and while using xdist)
Solution 1
If you want results of the tests you can use hook runtest_makereport
:
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
rep = outcome.get_result()
if rep.when == 'call' and rep.failed:
mode = 'a' if os.path.exists('failures') else 'w'
try: # Just to not crash py.test reporting
pass # the test 'item' failed
except Exception as e:
pass
Solution 2
Run pytest with -rf
to get it to print a list of failed tests at the end.
From py.test --help
:
-r chars show extra test summary info as specified by chars
(f)ailed, (E)error, (s)skipped, (x)failed, (X)passed,
(p)passed, (P)passed with output, (a)all except pP.
Warnings are displayed at all times except when
--disable-warnings is set
Here's what you get:
$ py.test -rf
================= test session starts =================
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.6.0, pluggy-0.7.1
[...]
=============== short test summary info ===============
FAILED test_foo.py::test_foo_is_flar
FAILED test_spam.py::test_spam_is_mostly_pork
FAILED test_eggs.py::test_eggs_are_also_spam
=== 3 failed, 222 passed, 8 warnings in 12.52 seconds ==
Solution 3
--result-log
is deprecated. You can instead use -v
to output the test case names as they run. If you pipe that into a file, you can query it. So if you're running your tests from a script you can do something like:
pytest -v | tee log.txt
grep -E '::.*(FAILURE|ERROR)' log.txt
Solution 4
You can use command line option --result-log
:
test_dummy.py:
def test_dummy_success():
return
def test_dummy_fail():
raise Exception('Dummy fail')
Command line:
$ py.test --result-log=test_result.txt
Content of the test_result.txt
. test_dummy.py::test_dummy_success
F test_dummy.py::test_dummy_fail
def test_dummy_fail():
> raise Exception('Dummy fail')
E Exception: Dummy fail
test_dummy.py:6: Exception
Just search 'F' in first column and after that would be [file]::[test]
Solution 5
I wanted a concise report of failed tests and parametrized variations so went with pytest_terminal_summary
in conftest.py
:
def pytest_terminal_summary(terminalreporter, exitstatus, config):
terminalreporter.section('Failed tests')
failures = [report.nodeid.split('::')[-1]
for report in terminalreporter.stats.get('failed', [])]
terminalreporter.write('\n'.join(failures) + '\n')
If you inspect terminalreporter._session.items
, there's more information you can add to the report, this is just what I wanted.
Related videos on Youtube
Itay
Updated on January 30, 2021Comments
-
Itay over 3 years
I would like to have a list of all the tests that have failed to be used at the end of session.
Pytest lets you define a hook
pytest_sessionfinish(session, exitstatus)
, that is called at the end of the session, where I wish to have that list.session
is a_pytest.main.Session
instance that has the attributeitems
(typelist
), but I couldn't find whether the eachitem
in that list passed of failed.- How can a list of all failed tests could be retrieved at the end of the session?
How can it be done while using
pytest-xdist
plugin, where I would like to get that list in the master process. Using this plugin,session
does not even haveitems
attribute in the master:def pytest_sessionfinish(session, exitstatus): if os.environ.get("PYTEST_XDIST_WORKER", "master") == "master": print(hasattr(session, "items")) # False
-
Itay over 6 yearsThanks! That seems to work well when xdist is not in use, and I can save the failed tests to a global variable to be used by
pytest_sessionfinish
at the end. When using xdist,pytest_runtest_makereport
is called only for the workers and not for master, so they can communicate through file, but a better solution is usingpytest_report_teststatus
andpytest_runtest_logreport
that are called in the master and can be used to collect failed test. -
Itay over 6 yearsI would like to use this information in
pytest_sessionfinish
and this method means failed tests list is available only after the pytest process has exited. -
Arindam Roychowdhury over 3 yearsCould have given the whole function...I tried your solution..did not work for me..... perhaps mention the pytest version you have used....and also a print of the result....and the test case that was run......
-
Aleks Amirkhanov over 3 yearsWhat exactly does not work? Do you have skipped tests in your setup?
-
Arindam Roychowdhury over 3 yearsSays...item does not have rep_call
-
Aleks Amirkhanov over 3 yearsI guess it means that the failing test was not started. What if you replace
item.rep_call.outcome
withitem.rep_call.outcome if hasattr(session_item, 'rep_call') else item.rep_setup.outcome
-
cryanbhu over 3 yearsI use
pytest ... -rA
and it gives failures output before passed output, is there a way to make it give failures output at the end? So less scrolling up