Annotate failing tests in the coverage job

[Mikola Lysenko (mikolalysenko)]

Problem

coverage is the gating Linux test run (test has no ubuntu leg). It is the most frequent non-macOS CI failure over the last 3 days: 5 of the 7 failed CI runs that weren't macOS failed there:

run	event	failing target
38273391986	merge_group (#1428, evicted)	socket-patch-cli --test in_process_edge_cases
38274384514	PR #1420	in_process_edge_cases (+ a deterministic core-lib arch test from #1420's own change)
38274549010 attempt 1	PR #1431	in_process_edge_cases; attempt 2 on the same SHA passed, so it's a flake
38218446419, 38184196576	PRs	coverage (not yet triaged)

None of the in_process_edge_cases failures can be diagnosed:

• The check run's only annotation is Process completed with exit code 101.
• The panic sits about 11k lines into a 16.3k-line log. The log API returns only the last 5,000 lines, and agent sessions can't reach the blob host for the full log.

So I couldn't find the root cause this run. Locally, 580 runs of the binary passed with 0 failures: 100 sequential and 480 under 12× parallel contention on 4 cores.

Fix

The coverage step now tees the test output to a file. If the tests fail, it runs a new script, scripts/ci-test-failures.py, over that file:

• one ::error annotation per failing test (at most 10, which is GitHub's per-step limit), titled <test binary> <test name> and carrying the panic text (up to 40 lines)
• a "Failing tests" block in the job summary

A test binary that dies without a report (signal, abort, stack overflow) gets an annotation of its own: cargo's to rerun pass error with the lines before it and the Caused by exit status. Doctest failures are credited to <crate> doctests.

The step's exit code is unchanged: pipefail, and the original status is re-raised. The next in_process_edge_cases failure (or any other) will then be readable from one check-runs/<id>/annotations call, and the flake can be fixed at its root.

Proof

• scripts/tests/test_ci_test_failures.py covers attribution to the binary, multiple sections, timestamped lines, the cap, escaping and the summary. It runs in the existing unittest discover -s scripts/tests job. The full suite passes locally (344 tests OK).
• Ran it on the real CI log of job 114896739060. It emitted the expected annotation for formats::architecture_tests::format_models_are_pure.
• Checked the shell wrapper's exit code: 0 when the tests pass, so the report steps still run, and 101 when they fail.
• actionlint 1.7.7 reports the same 36 pre-existing findings before and after. They are all YAML-anchor and shellcheck noise in other jobs.

Code review (/code-review high)

Fixed:

• crashes with no ---- stdout ---- section now get annotated
• doctests are credited to their crate
• the Running header needs its (path)
• lines are capped at 500 chars and the summary at 512 KiB
• failures past the 10-annotation cap are counted
• the code fence adapts to backtick runs in the body
• only Actions' 7-digit timestamps are stripped
• pipefail covers only the cargo pipeline

Not fixed:

• Annotate the test (macOS/Windows) and test-release shards too. That belongs in scripts/ci-test-shard.py, and Windows .exe paths need handling there. It's left for a follow-up so this PR stays one change.
• Captured output with a bare failures: line ends a section early. That needs a test to print libtest's own framing line. The fallback is a truncated body, never a wrong exit code.

Where tests run

No test moved, was removed, or was skipped. Required check names and ci-ok are unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RZFdPRj6Jyid4xSN71kCzP

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code