Repository navigation
ci: a red scheduled main run must open or update an issue — a gate that fails unwatched is a fourth way to be dark #9830
Description
Split out of #9829 because it is independent of every counter question there,
and it is the change that would have saved nineteen days.
The failure mode
CLAUDE.md records three ways a gate can be unable to fail: continue-on-error,
missing from required contexts, and cancelled-shown-as-failed. #7856 added a
fourth, starvation. gc-ratchet has been demonstrating a fifth: failing,
correctly, and being watched by nobody.
Measured in #9829: gc-ratchet has been red on main since 2026-08-18 —
nineteen days. Of its last 100 scheduled runs, 15 succeeded (all on or before
08-17), 64 failed, 16 were cancelled. Nothing blocked, nothing notified, and
the failure grew from 19 regression cells to 43 in that window without anyone
noticing the second accrual.
This is worse than the three recorded modes in one specific respect: those
produce a green that is wrong, which is at least invisible. This produced a
red that is right, sitting in plain view in the Actions tab, and it still
told nobody. It also silently disarms the PR arm's usefulness: because the gate
was already red on main, the first PR to add run-extended-tests inherits an
unrelated red, which trains reviewers to treat that gate's failures as noise.
The consequence for anyone reading recent work: for nineteen days the one gate
that watches GC counters could not report a GC counter regression, during a
period when a lot of GC-adjacent code landed. Claims of the form "the collection
schedule is unchanged" made in that window rest on ad-hoc diag counters, not on
this gate — because this gate was already red and therefore uninformative.
The ask
When a scheduled (or tag/dispatch) run of a gate workflow on main
concludes failure, open an issue — or update the existing open one — with the
run URL, the head SHA and the failing rows. Close or comment when it goes green
again.
Properties worth having, learned from #7856 and #9829:
- One issue per workflow, updated, not one per run — 64 failures would
otherwise be 64 issues, which is its own way of being ignored. - Include the delta from the previous failing run. gc-ratchet has been RED on main since 2026-08-18 (19 days); every labelled PR inherits it, and the failure has grown from 19 to 43 cells #9829's most important
fact was not "red" but "19 cells became 43", i.e. a second accrual nobody
attributed. A bare red would have hidden that. - Record the last green SHA in the issue body. In gc-ratchet has been RED on main since 2026-08-18 (19 days); every labelled PR inherits it, and the failure has grown from 19 to 43 cells #9829 that single fact
(dc4bcf287) is what bounded the bisection window to ten commits. - Apply to the other scheduled/label-gated gates too —
gc-root-dominance,
gc-native-roots,gc-moving-witnesses,gc-parse-churn-gateshare this
shape.
Why not just re-enable push: branches: [main]
Because that is what #7856 removed, for good reasons that still hold: fourteen
workflows × ~29 jobs against ~9 concurrent slots at ~58 merges/day outran drain
and the gate went dark by starvation instead. The six-hourly schedule is the
right cadence. The missing piece is not more runs, it is that a failing run
must produce an artefact a human is obliged to look at.
Suggested check on any gate, worth writing into the contributing docs
Running is not the same as being watched. For every gate, ask: if this fails
onmaintonight, who finds out, and how? If the answer is "someone opens the
Actions tab", the gate is dark.
Cross-references #9829 (the nineteen-day red and its bisection), #7856 (the
starvation mode), #9774 (a shared false failure hiding real ones in
check_thread_locals.py).
Activity
proggeramlug commentedon Sep 6, 2026
Fixed by #9880, landed on
mainvia merge train #9888. The train merged as its own branch, so the close-keyword never fired — closing manually, verified on main.fix(ci): document why gate-failure-watch's workflow_run trigger is safe