Repository navigation
scripts: name a failed stage's tests in the job log - #107
Merged
Merged
Conversation
A failed stage's tests were named only in its log, which CI uploads as an artifact, so a reader of the job log saw the command that failed but not the test. When a stage fails, validate-upstream.py now also prints the lines of that log that name what failed: the failed tests and where they panicked, the test counts, and the failed cargo or unittest target. A test's own output is never repeated, since its messages can carry a child's environment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
novelKR
marked this pull request as ready for review
October 15, 2026 11:13
novelKR
pushed a commit
that referenced
this pull request
Oct 15, 2026
#107 names a failed stage's tests in the job log, changing scripts/validate-upstream.py and docs/upstream-update.md; both merge cleanly with this branch's macOS operator tests and runbook. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
novelKR
pushed a commit
that referenced
this pull request
Oct 15, 2026
CI sets CARGO_TERM_COLOR=always, so cargo writes its own lines with terminal colors, and the summary that #107 added did not match cargo's "error: test failed, to rerun pass ..." line: #108's Integration failure printed the failed test and its panic but not the target to re-run. Colors are now removed before a line is matched. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Seongjae · project thread
Issue
None. Found while recording P1-RECOVERY's post-merge CI on the tracker #101.
Summary
Before: when a CI stage failed, its job log showed only the command that exited non-zero. The failed test, where it panicked and the test counts were only in
<stage>.loginside the uploaded artifact, which a reader of the job log alone cannot see. Main's post-merge CI failed one job on its first attempt three times in a row, and each passed on one re-run: macOS server DevGuard tests on1fda789(run 38686514388), macOS runner DevGuard tests on385e177(run 38708302887) and Linuxcargo test --locked --workspaceon950e2b2(run 38718331468). In none of them did the job log say which test failed.After: when a stage fails,
scripts/validate-upstream.pyalso prints the lines of that stage's log that name what failed: the failed tests (test … ... FAILED, and libtest's slow-test notice), where they panicked (thread '…' panicked at file:line:col:), the test counts, the cargo target to re-run or the test process that died with its signal, and unittest'sFAIL:andERROR:names. At most 40 lines are printed; the log keeps everything.A test's own output is never repeated. A line is printed only when its whole text has one of those formats, so a panic message, captured output or a child's environment stays in the log (the concern behind #94).
Contract changes
None. The report (
report.json), stage results, exit status and the CI workflow are unchanged, and a passing stage prints nothing new.Tests
Verified at
dea6c47, based on main950e2b2:python3 -B -m unittest discover -s scripts/tests: 105 tests OK, 3 of them new: a failed stage prints its failures and not the secret in their messages; which formats are and are not matched; an unreadable log and the 40-line cap.python3 -B -m unittest discover -s scripts/tests -p 'test_ci_*.py': OK.python3 -B scripts/check_docs.py: 20 paired documents verified, after reviewing theupstream-updatepair and recording its hashes.validate-upstream.py pythonwith a planted failing test printedFAIL: test_probe_fails (…)andFAILED (failures=1), not the secret in its assertion message. Without the planted test the stage passed and printed nothing new.Not verified locally: the docs-site build, which needs Node 24.21.0; the Docs workflow runs it. CI on this PR runs every leg, since the plan treats the script as a build input.
Security scenarios
Token logging: this repository's job logs are public, so the summary repeats only names, counts and source locations, never message text. A test checks that a secret in a panic message or an assertion message is not printed.
Out of scope
No test is changed, retried or skipped, and the flaky tests behind the three re-runs are not fixed here. Which stages run, the artifacts, and the DevGuard and Codex pins do not change.