How to get all failed tests _just once_ at the end of the session?

[lexicalunit]

I'd like to do something if certain tests have failed, just once, at the end of a test session.

Note that this code DOES NOT WORK:

@pytest.hookimpl(hookwrapper=True, tryfirst=True)
def pytest_runtest_logreport(report: pytest.TestReport) -> Generator[None, None]:
    outcome = yield
    if report.when == "call" and report.outcome == "failed":
        do_the_thing()

The problem with ☝🏻 is that it will execute for every single failed test. That's not what I want. I want the code in do_the_thing() to only run if a certain set of tests fail. But not for each one of those tests. Just once if any one of those tests fail.

So I need a higher level hook maybe? But this code ALSO DOES NOT WORK:

 def pytest_sessionfinish(session: pytest.Session, exitstatus: int) -> None:
    if exitstatus != pytest.ExitCode.OK:
        do_the_thing()

With ☝🏻 the code in do_the_thing() will execute if any test fails. Any test at all. Not just the specific tests I care about. And as far as I can tell, it's not possible to inspect session to find out what tests failed. I can iterate over session.items, but if I execute a dir() on the items, I only see these properties (edited for readability):

dir(item)=[
    '_ALLOW_MARKERS', 
    '__abstractmethods__', 
    '__annotations__', 
    '__class__', 
    '__delattr__', 
    '__dict__', 
    '__dir__', 
    '__doc__', 
    '__eq__', 
    '__format__', 
    '__ge__', 
    '__getattribute__', 
    '__getstate__', 
    '__gt__', 
    '__hash__', 
    '__init__', 
    '__init_subclass__', 
    '__le__', 
    '__lt__', 
    '__module__', 
    '__ne__', 
    '__new__', 
    '__reduce__', 
    '__reduce_ex__', 
    '__repr__', 
    '__setattr__', 
    '__sizeof__', 
    '__slots__', 
    '__str__', 
    '__subclasshook__', 
    '__weakref__', 
    '_abc_impl', 
    '_check_item_and_collector_diamond_inheritance', 
    '_fixtureinfo', 
    '_getobj', 
    '_initrequest', 
    '_nodeid', 
    '_obj', 
    '_pyfuncitem', 
    '_pytest_diamond_inheritance_warning_shown', 
    '_report_sections', 
    '_repr_failure_py', 
    '_request', 
    '_store', 
    '_traceback_filter', 
    'add_marker', 
    'add_report_section', 
    'addfinalizer', 
    'cls', 
    'config', 
    'extra_keyword_matches', 
    'fixturenames', 
    'from_parent', 
    'fspath', 
    'funcargs', 
    'function', 
    'get_closest_marker', 
    'getmodpath', 
    'getparent', 
    'ihook', 
    'instance', 
    'iter_markers', 
    'iter_markers_with_node', 
    'iter_parents', 
    'keywords', 
    'listchain', 
    'listextrakeywords', 
    'listnames', 
    'location', 
    'module', 
    'name', 
    'nextitem', 
    'nodeid', 
    'obj', 
    'originalname', 
    'own_markers', 
    'parent', 
    'path', 
    'reportinfo', 
    'repr_failure', 
    'runtest', 
    'session', 
    'setup', 
    'stash', 
    'teardown', 
    'user_properties', 
    'warn',
]

From some googling, I've seen that some people say there should be a rep_call property on the item where I could pull out the fail/pass state. But as you can see: This property simply does not exist.

So what is the lifetime hook, incantation, and properties that can tell me what tests failed at the end of running a test suite? Is such a thing even possible?

[lexicalunit]

I figured out a workaround but it seems like I'm doing something wrong here because I needed to import from _pytest and access a _session property to do it 🤷🏻‍♀️

import pytest
from _pytest.terminal import TerminalReporter

def pytest_terminal_summary(terminalreporter: TerminalReporter, exitstatus: int) -> None:
    if exitstatus == pytest.ExitCode.OK:
        return
    session: pytest.Session | None = terminalreporter._session
    if session is None:
        return

    should_do_the_thing = False
    failures: set[str] = {failure.nodeid for failure in terminalreporter.getreports("failed")}
    for item in session.items:
        if not item.get_closest_marker("failure_triggers_do_the_thing"):
            continue
        if item.nodeid in failures:
            should_do_the_thing = True
            break

    if should_do_the_thing:
        do_the_thing()

Also it's unclear if this will only trigger if there's a terminal report. What if the only report style selected is an HTML report? I guess maybe it won't run in that case? No idea.

---

Edit: This still doesn't work if you're using pytest-xdist because the do_the_thing() code will be executed n times where n is the number of xdist workers you have in your test run.

[nicoddemus]

Hi,

I think the proper way is to implement pytest_runtest_logreport like you did, and store the failed tests somewhere in your plugin. Then you can implement pytest_sessionfinished to do what you want with those failed tests.

pytest will not save the intermediate results anywhere by design: it is expected that each plugin interested in test results to implement pytest_runtest_logreport. Most plugins will not save the report at all, but do something else (show the status on the console, save to JUnit, XML, send it over the wire, etc). Some plugins will even run tests multiple times, generating multiple report logs for the same test, etc.

[lexicalunit]

@nicoddemus Actually, my accepted answer is still wrong. When using pytest-xdist I trigger the do_the_thing() code n times when there is a failure, instead of just once overall at the end of the tests. I'm not sure there's a workable solution here.

[nicoddemus]

If your hook is called N+1 times (one on each worker and once on master), perhaps you can use PYTEST_XDIST_WORKER to disambiguate and only call do_the_thing on master?

[lexicalunit]

This doesn't work. If I add something like this to the body of pytest_terminal_summary():

    xdist_worker = getenv("PYTEST_XDIST_WORKER")
    print(f"\n{xdist_worker=}", flush=True)

Then what I see when I use -n4 is (not necessarily in this order) these outputs:

• xdist_worker='gw0'
• xdist_worker='gw1'
• xdist_worker='gw2'
• xdist_worker='gw3'

And then a final one at the end of all output:

• xdist_worker=None

However, in this final call where xdist_worker is None, the value of session.items is also []. So I can't check if the items that failed have the pytest mark that I'm looking for.