Add --xfail-flaky to report exhausted flaky failures as xfail

[LouisDeconinck]

Summary

• Adds an opt-in --xfail-flaky flag: a test that still fails after all its reruns are exhausted is reported as xfailed instead of failed (functionality similar to pytest-ignore-flaky, as requested)
• Only applies to tests that were actually rerun (execution_count > 1); tests failing on first attempt or excluded by --only-rerun/--rerun-except still report as failed
• Implemented by converting the final call report to skipped + wasxfail, so pytest's standard xfail reporting applies

Closes #92

Test plan

• Always-failing test with --reruns 2 --xfail-flaky → 1 xfailed, 2 rerun, exit code 0
• Eventual pass unaffected; tests without reruns unaffected; marker-based reruns covered
• --only-rerun non-matching error still reports failed

[sylvesterkaczmarek]

--xfail-flaky only converts the final report when report.when == 'call'. Reruns can also be triggered by setup or teardown failures, so a fixture that fails on every attempt can exhaust its reruns and still finish as a normal failure even though the option says exhausted flaky failures are reported as xfailed. Could the terminal setup/teardown failure be handled too, or the option be explicitly narrowed, with a setup-failure regression test?

[LouisDeconinck]

Fixed in 53b61ec: the xfail conversion now applies to the terminal failed report in any phase, so a fixture that fails on every attempt ends as xfailed. Added a setup-failure regression test.

[LouisDeconinck]

Fixed — exhausted setup/teardown failures are now converted to xfail too, not just call reports, with a setup-failure regression test (53b61ec).

[icemac]

— Comment created by Claude

Code review findings for --xfail-flaky (see the inline comments).

[icemac]

— Comment created by Claude

Teardown failures are converted too, so a test is counted twice and the teardown error is hidden. This check runs on every report: setup, call and teardown. A test whose call passes but whose teardown always fails ends up as passed + xfailed. A test whose call and teardown both fail ends up as 2 xfailed. I reproduced it with --reruns 1 --xfail-flaky on two such tests: the run printed 2 passed, 3 xfailed, and the teardown error disappeared with exit code 0. Should this only convert the report that decides the test's outcome, the call report or a failed setup, and leave teardown errors as errors? A test for the teardown case would also help.

[icemac]

— Comment created by Claude

Using this with --fail-on-flaky gives exit code 7 for a test that never passed. Before this change, a test that failed every time made the session exit with 1, and pytest_sessionfinish returned early. Now the session status is 0, and pytest_sessionfinish sees execution_count > 1 and sets exit code 7. That code means "a flaky test passed on a rerun". Reproduced with def test_fail(): assert False and --reruns 1 --xfail-flaky --fail-on-flaky: the output is 1 xfailed, 1 rerun, exit 7. Should --fail-on-flaky skip items whose final report was converted, or should the two options be rejected together?

[icemac]

— Comment created by Claude

--max-suite-reruns bypasses the conversion. When the suite-wide cap is used up, the final failure is logged in the else branch ("Suite-wide limit exhausted"), not here. A test that was rerun and then hit the cap is still reported as failed. Reproduced with two always-failing tests and --reruns 2 --max-suite-reruns 1 --xfail-flaky: 2 failed, 1 rerun. The README says "Only tests that were actually re-run are affected", but this test was rerun and was not affected. The README should either document this case, or the cap-exhausted path should apply the same conversion.

[icemac]

— Comment created by Claude

Failures that come only from subtests are not converted. When a test fails only through subtests, the call report itself passed (report.failed is False). The failed subtest reports are logged by pytest's subtests plugin, not by this code. Reproduced with a test whose only subtest always fails, run with --reruns 1 --xfail-flaky: it still reports FAILED ... contains 1 failed subtest and the run fails. Either handle subtests or say in the docs that they are not covered.

[icemac]

— Comment created by Claude

Under xdist, a crash on the last attempt is not converted. When a worker crashes, XDistHooks.pytest_handlecrashitem builds the report on the controller, and this code never runs. A flaky test that crashes the worker on its final attempt is still failed, while the same test failing normally becomes xfailed. The two paths behave differently.

[icemac]

— Comment created by Claude

The failure reason is lost. Before, the short summary showed FAILED nodeid - AssertionError: .... Now it shows only XFAIL nodeid - test failed after N rerun(s), marked as xfail, and the test no longer appears in the FAILURES section. That makes the remaining flaky failures hard to diagnose. Consider adding the crash message (report.longrepr.reprcrash.message, when available) to wasxfail.

[icemac]

— Comment created by Claude

Nit: this repeats checks that _should_not_rerun already does (execution_count > reruns, _terminal_errors) and adds another special case to the main protocol loop. A small helper such as _mark_exhausted_failure_as_xfail(item, report, reruns) would be easier to read. The same helper could then be called from the suite-cap branch and limited to the deciding report, which also fixes the findings above.

[icemac]

— Comment created by Claude

Two blank lines are missing before def test_flaky_marker_with_zero_reruns_disables_rerun. The ruff-format pre-commit hook rewrites this, so the pre-commit check fails.