You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 95b21b3
Browse filesBrowse the repository at this point in the historyBrowse files
ci(rerun-safety): measure the rerun property even when pass 1 is red (#18746)
Fixes#18590
Clause-②: no
`Rerun Safety` (workflow `323746634`) has been red on `main` and the
card's
finding was that it dies in `Test suite — pass 1`, so the rerun property
it
exists to measure has never been measured. The card marked two premises
**NOT
ESTABLISHED** and asked for them to be measured first. Both were. One of
them is
**refuted**, and that is what decided where the fix goes.
## Premise 1 — "do all ten failures share one cause?" — REFUTED
The card read only the newest run's log. Reading the whole streak back
through
the workflow-scoped runs API and each job's check-run annotations:
**The streak is 17 nights, not 10.** Last success was `33355711750`,
2026-08-31. And the failures do not share a cause — not the failing
package, not
even the failing *step*:
| nights | dates | step that died | failing turbo task | kind |
|---|---|---|---|---|
| 4 | 09-14 .. 09-17 | pass 1 | `@objectstack/cli#test` | `Test timed
out in 5000ms` |
| 6 | 09-08 .. 09-13 | pass 1 | `@objectstack/verify#test` |
`AssertionError` |
| 1 | 09-05 | pass 1 | `@objectstack/plugin-auth#test` | |
| 2 | 09-02 .. 09-03 | pass 1 | `@objectstack/cli#test` | |
| 4 | 09-01, 09-04, 09-06, 09-07 | **pass 2** |
`@objectstack/core#test`, `@objectstack/objectql#test` | |
Two consequences the card could not see from one log:
1. **Four of the seventeen nights did reach pass 2.** Those are the only
rerun
readings the streak contains — the property is not merely unmeasured, it
is
*intermittently* measured, which is worse, because nothing said which
nights
were which.
2. **The pass-1 killer MOVED.** `verify`'s
`harness.host-resolution.test.ts` was
the killer for six nights and **is green on today's `main`** (verified
locally: `Test Files 1 passed (11 tests)`). It was fixed, and `cli`
inherited
the job.
So "get pass 1 green" is not a reachable end state on a trunk that goes
red like
any other trunk — it was reached, repeatedly, and the instrument went
dark again
each time. That is why the fix is not a fix to whichever test is red
tonight.
## Premise 2 — "does that file time out outside this workflow?" — the
control fired
Run locally on an idle box holding the shared verify lock, `--project
integration`:
```
✓ NODE_ENV=development 3300ms
✓ NODE_ENV=test 2990ms
✓ NODE_ENV=production (the never-broken leg) 3073ms
× CONTROL: neutralising the declaration 5710ms Test timed out in 5000ms
✓ CONTROL: neutralised under production 3039ms
```
Green outside the workflow — but **a different leg failed**, and it
failed the
same way. Re-run with the budget lifted (`--testTimeout=120000`), all
five pass
and the true costs are `3000 / 2526 / 2742 / 3599 / 2467` ms.
⭐ This **refines** the card's lead rather than confirming it. The card
read
"all three legs time out regardless of `NODE_ENV`" as the signature of
the spawn
harness. The sharper reading: **all FIVE legs cost 2.5–3.6s against a
5000ms
budget nobody chose**, so which legs redden is decided by machine load.
In the
workflow the three non-neutralised legs died and both CONTROL legs
lived;
locally the reverse happened. Same file, same commit, disjoint
casualties — the
casualty list is a function of the runner, not of anything under test.
(The card's own reasoning that this is not about `NODE_ENV` stands, and
is
strengthened: `NODE_ENV` does not correlate with the casualties in
either run.)
## What this changes
### 1. The instrument — `pass 2` no longer depends on `pass 1` being
green
`.github/workflows/rerun-safety-nightly.yml`: pass 1 now **records** its
exit
code instead of ending the job with it, pass 2 runs `if: always()`, and
a new
verdict step owns the job's exit status.
`scripts/rerun-safety-verdict.mjs` compares the two passes' failing task
sets and
reports, in one line, **whether the rerun property was MEASURED** and
what it
said:
| pass 1 | pass 2 | verdict | measured | job |
|---|---|---|---|---|
| green | green | `RERUN_SAFE` | yes | pass |
| green | red | `RERUN_UNSAFE` | yes | fail |
| red | green | `BASELINE_FLAKY` | yes | fail |
| red | red, same set | `BASELINE_BROKEN` — rerun property CLEAN | yes |
fail |
| red | red, pass 2 worse | `RERUN_UNSAFE_ON_BROKEN_BASELINE` | yes |
fail |
| stall (75) | any | `STALL` | **no** | fail |
⛔ **Nothing is weakened and nothing is skipped.** A red pass 1 still
fails the
job, exactly as the workflow header intends. What changed is that a
broken trunk
now costs the instrument its **baseline** instead of its **reading** —
and a
stall is the only remaining state that reports NOT MEASURED, which is
the honest
answer for a stall.
The task-set parser reads three spellings because no one of them is
sufficient:
turbo's `Failed:` footer **under-reports** (measured on the 2026-09-08
run, it
named `verify` alone while `plugin-auth`, `runtime` and `client` had
each printed
ELIFECYCLE in the same run), so the footer, the per-task `ERROR command`
line and
the streaming `ELIFECYCLE` prefix are unioned.
### 2. The proximate cause — the shadow budget on the named file
`packages/cli/test/published-entry-node-env-source-reroute.test.ts` now
states
`CHILD_BUDGET_MS = 120_000` once and hands it to both its own
`spawnSync` and
each of its five legs.
**⛔ This is not "raise the timeout until it passes", and the card is
owed the
argument:**
**(a) Why it cannot mask a real rerun failure.** *It raises no bound.*
The
`spawnSync` in this file already declared 120s — chosen, and the same
number the
other spawning suites in `packages/cli/test/` pin (27 of 48
integration-tier
files here carry an explicit per-test timeout; this one was in the
minority that
does not). The child is still killed at exactly 120s. What is removed is
a
**second, lower, unchosen bound** — vitest's 5000ms default — that was
shadowing
the chosen one, and that could never have bounded a `spawnSync` anyway,
because a
synchronous spawn blocks the worker's event loop and vitest cannot
interrupt it.
The 5000ms number never bounded anything; it only relabelled a completed
slow
spawn as a failure after the fact.
**(b) The reading that still discriminates the two.** Previously a hung
child and
a slow child arrived as the *same* information-free line — `Error: Test
timed out
in 5000ms`, attributed to the `it()` and naming nothing about the child.
They are
now separated: `runPublishedEntry` raises a **named** error when the
child was
killed by its own bound (`error.code === 'ETIMEDOUT'` / `signal ===
'SIGTERM'`),
carrying the env, the neutralise flag and the elapsed ms, and saying in
the
message that this is a hung child rather than a slow one. A slow child
stays a
pass whose duration vitest prints, against the five measured costs
recorded in
the file's header.
### 3. The job backstop — a second latent way this instrument goes dark
`timeout-minutes: 120` → `170`. The last four **green** two-pass runs
took 74,
89, 103 and **115** minutes, so a healthy run was within five minutes of
being
killed by its own backstop — and this PR makes pass 2 run on the nights
pass 1 is
red, which is exactly when a 115-minute run is likely. ⛔ This is not a
stall
budget: `run-with-stall-guard` still declares a stall after 15 minutes
of frozen
output on either pass, so a genuine hang is still caught in 15 minutes
and never
at this number.
### 4. The self-test is wired at PR time, not in the nightly
`check:pm-dispatch-gates` refuses a **SCHEDULED-ONLY** gate family:
every
discovered family must reach a workflow declaring a PR-time event.
Invoking
`--self-test` from `rerun-safety-nightly.yml` (schedule +
`workflow_dispatch`
only) made it this tree's first — the gate caught it, by name. It is
wired as
`pnpm check:rerun-safety-verdict` in `lint.yml` instead.
## Reverse verification (ablation)
Through `scripts/ablation-replace.mjs`, so the mutation's landing and
the
restore's byte-identity are the tool's own verdict:
```
anchor "matchAll(ELIFECYCLE_RE)" x1 -> x0
blob 6275d79 -> 18fce43f6bda [mutation proven on disk]
FAIL parse: streaming ELIFECYCLE prefix
FAIL parse: union under-reporting footer (2026-09-08 shape) -- got @objectstack/verify#test
rerun-safety-verdict --self-test: 2 FAILED [exit 1]
blob after restore 6275d79 == blob at HEAD 6275d79, `git diff HEAD` empty
```
Direction as predicted (turns red), and the ablated union falls back to
exactly
the under-report the union exists to close.
## Verification
Measured at `24f5253ca`.
- `pnpm lint` **repo-wide** — `eslint . --no-inline-config`, full
population, no
narrowing: **exit 0**.
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack`
derived **92** commands at this HEAD; all run, exit codes recorded and
reconciled with `--ran`.
- `node scripts/rerun-safety-verdict.mjs --self-test` — 15 batteries, 15
registered, exit 0.
- `packages/cli` suite, plus `packages/verify`'s
`harness.host-resolution.test.ts`
as the premise-1 control.
**NOT MEASURED, declared:**
- ⛔ **This workflow runs only on `main`, so no PR can make it red or
green.**
The change is proven by local reproduction and by the ablation above; it
is
**not verified by the instrument itself** and must not be read as if it
were.
The first real reading arrives on the first nightly after this lands.
- `pnpm check:dual-build-cjs-loads` — **exit 3, PREREQUISITE NOT MET**:
it reads
built output and this tree has no full `pnpm build`. Not a pass and not
a red.
This diff adds no published bytes, so it cannot move that gate.
## Cross-face declaration
The claim comment warned that touching **vitest tiering configuration**
would be
a repo-wide shared predicate and therefore a cross-face effect. ⛔ **It
is not
touched.** `packages/cli/vitest-tiers.ts` and
`packages/cli/vitest.config.ts` are
both unmodified. The edited test file keeps both signals that place it
(`node:child_process` + the `run.js` entry basename), so it stays in the
`integration` tier and no file moves between tiers.
## Changeset
`skip-changeset`, **measured rather than assumed**: `@objectstack/cli`
ships
`["dist", "README.md", "CHANGELOG.md"]` — `test/` is not published; the
root
package is `private`, so `scripts/` and `.github/` are not published
either. Grep
of the built `packages/cli/dist` for the symbols this PR introduces
(`CHILD_BUDGET_MS`, `killedByOwnBound`, `rerun-safety-verdict`) returns
**zero
hits**, with a positive control (`enableAutoTranspile`) hitting three
files in
the same grep. No published bytes move.
## Acceptance notes
Noted, not filed — none of these is in this card's scope, and each is
recorded
here rather than acted on:
- The four pass-2 deaths in the streak (`@objectstack/core#test` ×3,
`@objectstack/objectql#test` ×1, on 09-01/04/06/07) are **real
rerun-safety
readings** that were taken and never acted on. Triage ruled explicitly
that
the rerun-unsafe tests are not this card's business while the instrument
is
mute. With the instrument repaired they will re-report themselves on the
next
nightly, attributed, which is the right time to pick them up. Carrier:
the
first nightly run after this lands.
- `packages/cli/test/published-entry-stderr-error-listener.test.ts` is
the other
spawning file in this directory with no explicit per-test timeout, so it
is the
next candidate to redden the same way under load. Not measured here, and
not
changed — naming it so the next occurrence is recognised rather than
rediscovered. Carrier: whoever picks up the next `cli#test` nightly red.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01DvvamiacK328idtBYJBxV3)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
0 commit comments