Skip to content

Commit b04f7d1

Browse files
ci(rerun-safety): a truthful verdict, passes bounded by the job's time, and a card on a red night (#23975)
Part of #23880 Clause-②: no `Rerun Safety` (the nightly that runs the whole suite twice in one working tree) has concluded `success` on one of its last thirty scheduled nights (2026-09-26), and nothing filed a card. This PR changes only that job's own shape and its reporting. No package code or package test is edited. - **`scripts/rerun-safety-verdict.mjs`** now separates a task turbo **failed** from a task turbo **interrupted** after another task failed. It also says how much of each pass actually ran: MEASURED, PARTIALLY MEASURED, or INCOMPLETE/NOT MEASURED. It flags a failure whose every failed test is a vitest timeout, and says how to tell load from a leak. - **`.github/workflows/rerun-safety-nightly.yml`** has three changes: - each pass runs under `timeout` set to the time the job has left, so a slow night ends in a verdict and a `failure` conclusion instead of a silent `cancelled`; - `--only` stops each pass re-executing the 69 build tasks that `Build packages` just built; - the header records the measured wall-time growth. - **`.github/workflows/scheduled-full-run-card.yml`** adds `'Rerun Safety'` to `workflow_run.workflows`, so a red night files or refreshes a card. - **`scripts/ci/scheduled-full-run.mjs`** (patch round, claim `6218695689`) gives that card its workflow's cadence. A closed map, `CARD_CADENCES`, sets `CI` and `Lint & Type Check` to hourly and `Rerun Safety` to nightly; a workflow outside the map is refused, never defaulted. For the two hourly workflows every byte the card carries is unchanged, so the open CI card #23885 is still found and refreshed. **Why `Part of`:** Done-when item 3 ("a nightly run concludes `success`, or names only failures that already have cards") can only be read on a nightly after this lands. Two full passes also no longer fit one 4-vCPU runner in 170 minutes. That capacity choice belongs to the maintainer, after the first nightly reading; it is set out under **Capacity** below. #23880 remains open for it. ## Readings: the failing tests, per night and pass **Source:** each run's job log via MCP `get_job_logs`. That tool returns the last 5,000 lines. The REST log zip answers 403 from this container (`results-receiver.actions.githubusercontent.com`, `connect_rejected`). A full log is 290k to 436k lines, so pass 1's detail is readable only on nights whose pass 2 stopped early. The other nights are marked NOT MEASURED. **Column meanings:** - **FAILED:** turbo's own report, from its `Failed:` footer and per-task `ERROR command … exited (1)` line. - **INTERRUPTED:** the task printed only `ELIFECYCLE Test failed.`, 0.1 to 0.5 s after the real failure's, with no ERROR line and no footer entry. | night | run | pass | FAILED | the failing test and its error line | INTERRUPTED | turbo: started / successful | |:--|:--|:--|:--|:--|:--|:--| | 10-02 | 36962761984 | 1 | dogfood | `activity-parent-read-gate.dogfood.test.ts:102` "list: every row returned is about a record the member can open…"; `activity-text-predicate.dogfood.test.ts:465` "control: the unmasking reader filters and groups by every column…"; `audit-log-parent-read-gate.dogfood.test.ts:144` "list: every row returned that names a record…". Each: `Error: Test timed out in 5000ms.` | cli | 138 / 136 | | 10-02 | | 2 | core | `src/security/platform-admin.test.ts:278` "nor by `@objectstack/core`'s security entry — the published surface lost them" (a dynamic `import('./index.js')`): `Test timed out in 5000ms.` | driver-sql, spec, lint | 42 / 20 | | 10-13 | 38344867129 | 1 | NOT MEASURED (verdict line: cli, dogfood) | | | | | 10-13 | | 2 | runtime | `src/declared-capability-absence-warn-once.test.ts` "the dispatcher door > reports a declared 501 ONCE across N requests…": `Test timed out in 30000ms.` | verify, http-conformance | 132 / 127 | | 10-14 | 38509291145 | 1 | dogfood | `activity-text-predicate.dogfood.test.ts:470` "control: the unmasking reader filters and groups by every column of a pinned parent, as before": `Test timed out in 5000ms.` | cli | 143 / 141 | | 10-14 | | 2 | core | `src/security/auth-gate.test.ts:571` "BOUNDARY — newly exempt is exactly POST on the four-segment hook route…": `Test timed out in 5000ms.` | spec, lint | 49 / 24 | | 10-15 | 38679312462 | 1 | NOT MEASURED (verdict line: cli, dogfood) | | | | | 10-15 | | 2 | runtime | `src/domains/share-links-refusal-order.test.ts:356` "email audience: the refusal is the one an unknown token and a password-less email link answer": `Test timed out in 5000ms.` | verify (it logged `Received SIGINT`), cli, organizations | 142 / 132 | | 10-16 | 38848504327 | 1 | NOT MEASURED: no verdict ran | | | | | 10-16 | | 2 | none: cancelled by the job timeout 67 min in, still printing | | | | Every real failure in the table is a vitest timeout. The pass-2 "newly failing" sets the old verdict printed are not rerun-safety findings either: - 10-14: core, lint, spec; - 10-15: cli, organizations, runtime, verify. Each was one real timeout plus the suites turbo stopped beside it. ## Pass 1's root cause - **`@objectstack/cli#test` is not a pass-1 failure.** On both nights whose pass 1 is readable (10-02, 10-14), turbo's footer names `@objectstack/dogfood#test` alone. cli printed ELIFECYCLE 0.16 to 0.23 s later, with no ERROR line: turbo, which runs without `--continue`, stopped it when dogfood failed. 10-13 and 10-15 print the same verdict shape; their detail is NOT MEASURED. The old verdict read every ELIFECYCLE as a failure. That was the defect, and this PR removes it. - **`@objectstack/dogfood#test` is vitest 5000 ms timeouts under this job's load.** I ran the failing files locally, unloaded, on an idle 4-core container at `98ba1bf1d2`. All passed: - activity-text-predicate's control test: 3,377 ms, which is 68% of vitest's default timeout. The file's 22 tests sum to 12.9 s. - activity-parent-read-gate's list test: 1,063 ms. - audit-log-parent-read-gate's list test: 974 ms. - **Why the load is higher here than in the hourly battery:** - The hourly `Dogfood` job runs dogfood alone, in three slices, one per 4-vCPU runner. - Here the whole dogfood suite runs beside up to three other heavy suites: 34 min on 10-14, starting 64 min into the pass. Turbo runs at `--concurrency=4`, each task at vitest's default of three workers, so twelve workers share four vCPUs. - The tail of the pass is where dogfood, cli, runtime and verify overlap. - That is this job's environment. This PR makes the job say so, and does not change the tests. How the job gets its capacity back is the decision under **Capacity**. ## The pass-2-only packages: each passes a second run in one tree Measured locally at `98ba1bf1d2`, unloaded, `--maxWorkers=2`, two consecutive runs in the same worktree: | package | run 1 | run 2 | |:--|:--|:--| | `@objectstack/core` | 108 files / 2,586 tests passed, 72 s | 108 / 2,586 passed, 70 s | | `@objectstack/runtime` (`--project local`) | 402 files / 6,122 tests passed, 679 s | 402 / 6,122 passed, 654 s | After both runs, `git status` showed no new or ignored state beyond `dist/` and `node_modules/`. **The specific tests, unloaded:** - auth-gate BOUNDARY: 1,055 ms. - share-links "email audience": 1,057 / 1,086 / 1,135 / 1,136 ms over four runs. - warn-once: 79 ms. The 10-13 warn-once test ran 79 ms here and hit its own 30 s bound in pass 2. That gap is beyond plain CPU contention. What held it is NOT MEASURED. It is one occurrence that passes twice locally, so it is noted here and not carded. ## The timeout's cause: wall-time growth, not a hang, not one new suite - **10-16 step timestamps:** - setup, install and build: 7 min (04:02:11 to 04:09:14); - pass 1: 100.4 min (04:09:14 to 05:49:39); - pass 2: started 05:49:39 and was cancelled at the job's 170-minute timeout (job end 06:57:11). Its log was still printing at 06:54:45 (verify and example-todo booting), and the 15-minute stall guard never fired. - **10-12, the same:** pass 1 113 min; pass 2 cancelled 56 min in. - **Pass 1 duration by night:** - 09-26, the last green night: 38 min (pass 2 also 38); - 10-02: 66; 10-03: 65.5; - 10-14: 98; 10-15: 97; 10-16: 100. - **Queue-tier test task time in `scripts/test-shard-timings.json`:** - by refresh date: 123.8 min (measured 09-28), 163.0 (10-05), 177.2 (10-08), 187.1 (10-12), 193.6 (10-13); - `@objectstack/cli` alone: 733 s, 1,703, 1,739, 2,109, 2,206. So two full passes need roughly 200 minutes of a 170-minute job. A night "finished" only because pass 2 stopped at its first failure: on 10-14 it ran 7 minutes and started 49 of 143 tasks. A night that did not fail early was `cancelled`, with no verdict. Raising `timeout-minutes` is ruled out by triage's scope and the test-budget rulings, and this PR does not raise it. ## What this PR does about it 1. **Each pass is bounded by the time the job has left.** - The bound is `budget = (deadline − now)/60 − PASS_RESERVE_MINUTES`, with `JOB_TIMEOUT_MINUTES: 170` set beside `timeout-minutes: 170`. - It is spelled `timeout --kill-after=2m "${budget}m"` and sits inside the stall guard, as the command it wraps. - Measured locally (guard, then `timeout 45s`, then turbo over core, types and formula): exit `124` came back through the guard. The log ended at `Shutting down Turborepo tasks...` / `Force killed Turborepo tasks: @objectstack/core#test`, with no `Tasks:` summary. No process was left behind. - The verdict reads 124 or 137 as a pass that did not finish: `INCOMPLETE`, rerun property `NOT MEASURED`, exit 1. The job concludes `failure`, which the card filer reads; `cancelled` it does not read. 2. **`--only` on both passes.** `turbo run test --force --dry=json` at `98ba1bf1d2` plans 83 `test`, 69 `build` and 1 `@objectstack/spec#gen:migration-registry` task. - All 69 builds are in `pnpm build`'s own plan (`Build packages`). - The spec's `test` script runs `pnpm gen:migration-registry` itself. - With `--only`, a pass runs only the 83 test tasks, and pass 2 no longer rebuilds `dist/` over what pass 1 left there. - **NOT MEASURED:** how many minutes this saves a nightly pass. That needs the first nightly after landing. 3. **The verdict tells the truth about every night above.** Here is run 38509291145 (10-14), read from its job log by the old script and by this one. Old (what the job printed): ```text Newly failing on pass 2: @objectstack/core#test, @objectstack/lint#test, @objectstack/spec#test rerun property: MEASURED pass 1 exit 1, failing tasks: @objectstack/cli#test, @objectstack/dogfood#test ``` New: ```text Newly failing on pass 2: @objectstack/core#test @objectstack/core#test: every failed test hit vitest's test timeout -- load-shaped. A leak fails an assertion, not a clock: run `pnpm --filter @objectstack/core run test` twice in one tree on its own; a leak fails the second run there, a load timeout does not. Failing in both passes: (none) Failed on pass 1 and never finished by pass 2 (NOT re-measured, not recovered): @objectstack/dogfood#test Interrupted by turbo in pass 1 after another task failed (not failures, unmeasured): @objectstack/cli#test Interrupted by turbo in pass 2 after another task failed (not failures, unmeasured): @objectstack/lint#test, @objectstack/spec#test Pass 1 stopped at its first failure (turbo runs without --continue); the tasks still running then were interrupted and are unmeasured in that pass. Pass 2 stopped at its first failure (turbo runs without --continue) after starting 49 of the 143 tasks the longer pass started; every task it did not finish is unmeasured in that pass. rerun property: PARTIALLY MEASURED pass 1 exit 1, failing tasks: @objectstack/dogfood#test (timeout-only) interrupted by turbo (unmeasured): @objectstack/cli#test turbo: 141 successful of 143 started pass 2 exit 1, failing tasks: @objectstack/core#test (timeout-only) interrupted by turbo (unmeasured): @objectstack/lint#test, @objectstack/spec#test turbo: 24 successful of 49 started ``` The 10-02 log (run 36962761984) reads the same way: newly failing `core` alone, not `core, driver-sql, lint, spec`. Pass 2 started 42 of 138 tasks. ## The reporting change: a card for a red night, in the night's own cadence `scheduled-full-run-card.yml` now lists `'Rerun Safety'`. I measured the existing mechanism against a once-a-day workflow: - **Identity and dedupe fit.** The card is per workflow name, so it is distinct from the CI and Lint cards. Refresh-in-place suits a nightly streak: one card, and no comment per night. - **The conclusion guard missed every timeout night.** The guard reads `failure || timed_out`, but a job timeout here concluded `cancelled`: runs 38848504327 and 38202695171 are `cancelled` at both run and job level. That gap is closed from the job side by change 1: an overrun night now concludes `failure`. - **The card body's cadence was hourly for every watched workflow.** This patch round closes that in `scripts/ci/scheduled-full-run.mjs`: - `CARD_CADENCES` is a closed map from the byte-exact workflow name to its cadence: `CI` hourly, `Lint & Type Check` hourly, `Rerun Safety` nightly. `cardIdentity` refuses a name outside it and names the map in the refusal, so adding a workflow to the card's watched list means adding its cadence in the same change. - The cadence drives the title prefix, the marker prefix and every cadence sentence in `renderBody`: the marker line, the "runs on `main`…" paragraph, the remedy lines, the "What landed since the last green … run" heading, and the empty-range flake reading. - `Rerun Safety` therefore files `nightly full run: red on main (Rerun Safety)` with marker `os-nightly-full-run-rerun-safety`. Its body says "every night", "the last green nightly run" and "green the night before", and has no hourly wording. - The hourly entry is the old text byte for byte. The CI card keeps `hourly full run: red on main (CI)`, marker `os-hourly-full-run-ci`, and the same marker line and body. - `renderBody` also refuses an identity built by hand without a cadence, rather than default one. - The range for `Rerun Safety` starts at its last green `schedule` run, 09-26 (run 36216601546). The compare API caps at 250 commits, and the body already says when it truncated. ## Capacity: the maintainer's choice, after the first nightly reading Under this PR, a night whose two passes do not fit gets a loud INCOMPLETE and no longer a silent `cancelled`. It is still not a measurement. Once the first nightly after landing prints its two pass durations (the budget lines print them), there are three ways to make two full passes fit again: - **A. Legs.** - What: a matrix of legs (dogfood | cli | the rest, split in two by `scripts/partition-test-shards.mjs` with cli and dogfood excluded), each running its own package set twice in its own tree. - Fit: each leg has an estimated 2 × 45 + 8 min of work at today's timings. - Cost: standard runners and about three more setup-and-build laps a night. - Loses: detection of pollution between legs. - Scales: add a leg. - **B. A larger runner.** - Keeps one tree and the full property. - Needs an organization larger-runner label and is billed per minute. Whether one is configured is NOT MEASURED; every `runs-on` in this repo is `ubuntu-latest`. - **C. Keep one runner under this PR's bound.** The nightly measures only what fits, and says so. **Recommendation, after that reading:** if pass 1 still exceeds about 78 minutes, choose **A**. That threshold is 170 minus 7 minutes of setup and build minus the 5-minute reserve, halved. - The leak this instrument was built for (#4065) wrote `.objectstack/data/memory-driver.json` under the suite's own working directory, and A keeps that. - Whether any other leak ever crossed packages is NOT MEASURED. - B buys the same answer with money. - C leaves the instrument mostly unmeasured. ## Split cards (self-split rule) One lane card was warranted. The seat filed it from the dev's draft in the report on #23880: - **Dogfood lane:** `packages/qa/dogfood/test/activity-text-predicate.dogfood.test.ts`. Its 22 tests sum to 12.9 s unloaded, over the 10 s per-file rule (ruling record 6181973363), and it declares no budget. Its "control: the unmasking reader filters and groups by every column…" test runs 3,377 ms of vitest's 5,000 ms default on an idle 4-core box. It timed out in this job's pass 1 on 10-02 and 10-14. The other timed-out tests are not carded. core and runtime each pass a second run in one tree. The 10-02 dogfood list tests run about 1 s unloaded and are load-only. ## Verification The branch merged `origin/main` at `68ffa5dc1` (a merge, no rebase). The gate union ran after the final commit, at `240a2a565`. **Round 1 (verdict and nightly):** - `pnpm check:rerun-safety-verdict`: `rerun-safety-verdict --self-test: OK (23 batteries, 23 registered)`, exit 0. The fixtures are the 10-14 and 10-15 log shapes, colour codes included. - **Ablation**, one behaviour, on committed code via `scripts/ablation-replace.mjs`: - Mutation: put back the pre-fix union, adding `failing.add(id)` on every ELIFECYCLE line. Landing proof: anchor 1 → 0, blob `050a7fc6ae10` → `aec8629e130f`. - Result: `--self-test: 9 FAILED`, including the end-to-end case printing `Newly failing on pass 2: @objectstack/core#test, @objectstack/lint#test, @objectstack/spec#test`. - Restored: blob == HEAD (`050a7fc6ae10`), `git diff HEAD` empty. - The first attempt was a no-op: its replacement re-contained the anchor, `ablation-replace` refused it as "a drop of 0", and that reading was discarded. - **Real logs:** both scripts were run on the 10-14 and 10-02 job logs; output as above. The old script reproduces the job's printed verdict exactly. **Patch round (cadence):** - `node scripts/ci/scheduled-full-run.mjs --self-test`: `scheduled-full-run: self-test OK`, exit 0. The identity battery now has 39 cases (28 before), and its floor rises from 23 to 34 for the eleven new ones: - the CI title and marker as the literals `hourly full run: red on main (CI)` and `os-hourly-full-run-ci`; - the CI marker line as a literal, in all three body shapes; - the open CI card exactly as filed (title and marker line), still found by its title, and still found by its marker alone after a rename; - `Lint & Type Check` hourly; - `Rerun Safety` nightly (title and marker); - a nightly body with no hourly wording in any of the three shapes, and an hourly body with no nightly wording; - a workflow outside the map (`Spec Coverage`) refused by name; - a hand-built identity with no cadence refused by `renderBody`; - the nightly and hourly identities never matching each other's cards. - **Byte-identity against the pre-change module** (one-off comparison, not a kept file): for `CI` and `Lint & Type Check`, title, marker, workflow name, labels and three body shapes match the module at `b48f908ad`. Result: `byte-identical: 14 of 14`. - `node scripts/ci/scheduled-full-run.mjs --check-concurrency`: exit 0. - **Ablations**, each on committed `240a2a565` via `scripts/ablation-replace.mjs`, each restored to blob == HEAD `c33af68bc2ee` with `git diff HEAD` empty. Every mutation landed (anchor 1 → 0) and each self-test went red: 1. The refusal made a default (`if (false …)` in place of the map check) → `Error: cadence: a workflow outside the map was not refused by name (accepted)`. 2. `CI` mapped to nightly → `Error: identity: CI title is 'nightly full run: red on main (CI)'`. 3. The hourly `every` word changed to `'each hour'` → `Error: cadence: the CI marker line moved: …`. The pre-existing cases do not pin this line; only the new marker-line pin catches it. **Gates and tier:** - **Gates at `240a2a565`:** `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derives 53 commands for this diff. Each was run with its exit code captured before any pipe, and all 53 exited 0. - These include `node scripts/ci/scheduled-full-run.mjs --self-test`, and `check:stall-guard-budget`, which still sees both rerun-safety sites: window 15m, cap 30m, budget 170m, slack 140m. - They also include `check:workflow-status-functions`, `check:agent-test-spelling`, `check:pm-dispatch-gates` (fast tier, 1,884 cases, 320 s) and `check:rerun-safety-verdict`. - `--ran` reconciliation: `53 derived, 53 run, 0 NOT-MEASURED, 0 UNRUN`, with a derived zero, because all 53 recorded an exit code. - `node scripts/pm/check-governed-merges.mjs --branch claude/issue-23880-rerun-safety-nightly` at `240a2a565`: GOVERNED, landing tier H, `.github/workflows/**` ×2. - **Changed lines: 1,089 (+876 / −213) over 4 files.** The round-1 body's "899 (+705 / −194)" was read at `58669da5e`. Round 1 ended at `fdc14b452` with 908 (+713 / −195); the cadence leg adds the rest. ## Acceptance notes - The verdict's old union reading was encoded in a self-test case named for the 2026-09-08 "footer under-reports" shape. Read against this card's logs, that shape is turbo's fail-fast interruption. The case is replaced, not respelled. - The 10-13 runtime warn-once timeout (79 ms unloaded, 30 s bound hit) is an observation. It is not carded: one occurrence, and it passes twice locally. - `Rerun Safety` runs `test` and not `test:repo`, while the root `pnpm test` runs both. That is out of this card's scope, noted only. - `CARD_CADENCES` and `scheduled-full-run-card.yml`'s `workflows:` list are kept in step by the refusal, at the first red run of an unmapped workflow. No self-test reads the workflow file to compare them, deliberately: that would add a cross-file check to this gate family, and no new gate is in scope. - **NOT MEASURED:** - the nightly effect of `--only` and of the bound (this seat cannot dispatch the workflow); - pass 1 detail on 10-13, 10-15 and 10-16; - whether a larger-runner label exists for this organization. ## 维护者速读(草稿) **改了什么**:只动「Rerun Safety」夜间任务自己的四处。 1. 判定脚本:分清「真失败」和「被 turbo 顺手停掉的」,并如实写出每一轮到底测了多少。 2. 夜间工作流:每一轮只用任务剩余的时间跑,超时就给出「未测完」的结论,不再被平台静默取消;两轮里也不再重复构建已构建好的包。 3. 红卡工作流:把这个夜间任务加入监听名单,红了会自动开卡或刷新卡。 4. 红卡文案:按工作流区分「每小时」和「每晚」。夜间任务的卡写「每晚」;CI 等每小时任务的卡一个字都不变,已开着的 CI 卡照常被找到、被刷新,不会重复开卡。 **为什么改**:这个任务 30 晚里只绿过 1 晚,却没人收到通知。读日志发现: - 每晚报的 `cli` 失败其实是被 turbo 中途停掉的,真正失败的是 dogfood,而且都是 5 秒超时,属于负载问题。 - 第二轮「新增失败」的包每晚都不同,同样只是第一个超时的包加上旁边被停掉的包。 - 整个测试集三周里变慢了约 2.6 倍,两轮已经塞不进 170 分钟。没测完的那几晚被平台直接取消,既没有结论,也没有发卡。 - 新开的夜间红卡如果沿用原来的文案,会写成「每小时跑一次」,误导看卡的人。 **风险与代价(含回滚)**: - 不改任何产品代码和测试,不新增门禁,不提高超时上限。 - 风险一:`--only` 依赖「构建」一步已建好全部依赖;已用 dry-run 核对过,69 个构建任务全部包含在内。 - 风险二:超时边界依赖 `JOB_TIMEOUT_MINUTES` 与 `timeout-minutes` 两个数字保持一致;注释已写明。 - 风险三:以后若给红卡新增监听的工作流,必须同时在节奏表里登记,否则该卡会报错而不是默认成「每小时」。这是有意为之。 - 回滚:revert 本 PR 即可。 **席位意见**: **你要做的**: 1. 批准(或否决)这四处改动,本 PR 属 Tier H。 2. 落地后看第一晚两轮的耗时,再在「容量」三选一中拍板:拆成多条腿 / 更大的 runner / 保持现状只报「未测完」。推荐先看数据,超过约 78 分钟就选拆腿。 --- _Generated by [Claude Code](https://claude.ai/code/session_016PVjjuVPte5XEJvaQx1E4t)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b15f8a1 commit b04f7d1

4 files changed

Lines changed: 876 additions & 213 deletions

File tree

‎.github/workflows/rerun-safety-nightly.yml‎

Lines changed: 76 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,17 @@
4646
# Turbo would replay cached results on the second pass and report green without
4747
# executing anything, which is precisely the wrong answer here. `--force`
4848
# bypasses the cache so the second pass genuinely re-runs.
49+
#
50+
# ## Why `--only` (#23880)
51+
#
52+
# `--force` applies to every task in the run, and `test` depends on `^build`, so
53+
# each pass also re-executed the whole build closure -- 69 `build` tasks and
54+
# `@objectstack/spec#gen:migration-registry` (`turbo run test --dry=json` at
55+
# 98ba1bf1d2) -- that `Build packages` had just built. `--only` runs the 83 `test`
56+
# tasks and nothing they depend on; all 69 of those builds are in `pnpm build`'s
57+
# own plan, and the spec's `test` script regenerates its migration registry
58+
# itself. It also stops pass 2 from rebuilding `dist/` over whatever pass 1 left
59+
# there, which is a working-tree state this job exists to see, not to repair.
4960
name: Rerun Safety
5061

5162
on:
@@ -77,8 +88,36 @@ jobs:
7788
# in 15 minutes and never at this number. Raising it can therefore hide no
7889
# hang; all it stops is a slow-but-live run being reported as a failure of
7990
# the thing under test.
91+
#
92+
# ⚠️ 170 no longer holds two full passes, and raising it again is not the
93+
# fix (#23880). Measured from the job's own step timestamps: pass 1 took 38
94+
# minutes on 2026-09-26 (the last green night), 66 on 10-02, and 97-113 on
95+
# every night from 10-12 to 10-16, while the queue-tier suite's measured
96+
# task time (scripts/test-shard-timings.json) grew from 124 to 194 minutes
97+
# between the 09-28 and 10-13 refreshes, the CLI's alone from 733 to 2206
98+
# seconds. Not a hang: on 10-16 pass 2 was still printing when the job
99+
# timeout cancelled it 67 minutes in. A night finished at all only because
100+
# pass 2 stopped at its first failure; one that did not fail was CANCELLED,
101+
# with no verdict and a conclusion the red-card filer does not read.
102+
#
103+
# So each pass now runs under the time the job has LEFT (JOB_TIMEOUT_MINUTES
104+
# below, which must equal this number, minus PASS_RESERVE_MINUTES for the
105+
# verdict), and a pass that runs out is stopped by `timeout` instead: the
106+
# verdict step still runs, reads the pass's 124/137 exit (or a log with no
107+
# turbo summary) as INCOMPLETE -- NOT MEASURED, and the job concludes
108+
# `failure`, never a silent `cancelled`. That bound decides only WHO reports
109+
# a slow night; how two full passes fit again is a capacity decision
110+
# recorded on #23880.
80111
timeout-minutes: 170
112+
env:
113+
JOB_TIMEOUT_MINUTES: 170
114+
# The verdict and the steps after it take seconds; the reserve also covers
115+
# `timeout --kill-after=2m`, the grace a pass gets between SIGTERM and SIGKILL.
116+
PASS_RESERVE_MINUTES: 5
81117
steps:
118+
- name: Record the job's deadline
119+
run: echo "JOB_DEADLINE_EPOCH=$(( $(date +%s) + JOB_TIMEOUT_MINUTES * 60 ))" >> "$GITHUB_ENV"
120+
82121
- name: Checkout repository
83122
uses: actions/checkout@v7
84123

@@ -142,6 +181,17 @@ jobs:
142181
# comment in ci.yml): grouped log order flushes a task's output only when
143182
# the task ends, so a solo tail task longer than the budget reads as
144183
# frozen while healthy. The guard refuses a turbo run without it.
184+
#
185+
# The time bound (#23880) sits INSIDE the guard, as the command it wraps:
186+
# the guard runs that command in its own process group, so `timeout` is
187+
# that group's leader and its SIGTERM reaches turbo, which stops its tasks.
188+
# Measured locally at 98ba1bf1d2 (guard -> timeout 45s -> turbo over core,
189+
# types and formula): exit 124 came back through the guard, and the log
190+
# ended at "Shutting down Turborepo tasks..." / "Force killed Turborepo
191+
# tasks: @objectstack/core#test" with no `Tasks:` summary. The verdict reads
192+
# 124 (or 137, after --kill-after) as a pass that did not finish. Wrapped
193+
# the other way round, `timeout` would kill only the guard and leave its
194+
# detached group running into the next step.
145195
- name: Test suite — pass 1
146196
run: |
147197
# Bound vitest's INNER worker pool (#11958). A no-op on a runner with
@@ -151,11 +201,23 @@ jobs:
151201
# vitest's own "use the default" signal. Needs turbo.json's
152202
# globalPassThroughEnv entry or turbo strips it — see the script header.
153203
export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)"
204+
budget=$(( (JOB_DEADLINE_EPOCH - $(date +%s)) / 60 - PASS_RESERVE_MINUTES ))
205+
echo "Pass 1 may run ${budget} minutes before the job's own timeout would end it."
154206
status=0
155-
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass1.log" --stall-minutes 15 -- \
156-
pnpm turbo run test --concurrency=4 --force --log-order=stream || status=$?
207+
if [ "$budget" -lt 1 ]; then
208+
: > "$RUNNER_TEMP/rerun-pass1.log"
209+
status=124
210+
else
211+
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass1.log" --stall-minutes 15 -- \
212+
timeout --kill-after=2m "${budget}m" \
213+
pnpm turbo run test --only --concurrency=4 --force --log-order=stream || status=$?
214+
fi
157215
echo "PASS1_EXIT=$status" >> "$GITHUB_ENV"
158-
if [ "$status" -ne 0 ]; then
216+
if [ "$status" -eq 124 ] || [ "$status" -eq 137 ]; then
217+
echo "::warning::Pass 1 ran out of the job's time (exit $status) before it finished."
218+
echo "::warning::That is a capacity reading, not a suite reading; the verdict step"
219+
echo "::warning::reports it as INCOMPLETE and will fail this job."
220+
elif [ "$status" -ne 0 ]; then
159221
echo "::warning::Pass 1 exited $status — the suite is red on main. Recorded as the"
160222
echo "::warning::BASELINE and NOT failed here: pass 2 runs anyway so the rerun"
161223
echo "::warning::property still gets measured tonight. The verdict step owns the"
@@ -197,9 +259,18 @@ jobs:
197259
# vitest's own "use the default" signal. Needs turbo.json's
198260
# globalPassThroughEnv entry or turbo strips it — see the script header.
199261
export VITEST_MAX_WORKERS="$(node scripts/vitest-worker-cap.mjs)"
262+
# Whatever pass 1 left of the job's time; see pass 1 for the bound.
263+
budget=$(( (JOB_DEADLINE_EPOCH - $(date +%s)) / 60 - PASS_RESERVE_MINUTES ))
264+
echo "Pass 2 may run ${budget} minutes before the job's own timeout would end it."
200265
status=0
201-
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass2.log" --stall-minutes 15 -- \
202-
pnpm turbo run test --concurrency=4 --force --log-order=stream || status=$?
266+
if [ "$budget" -lt 1 ]; then
267+
: > "$RUNNER_TEMP/rerun-pass2.log"
268+
status=124
269+
else
270+
node scripts/run-with-stall-guard.mjs --log "$RUNNER_TEMP/rerun-pass2.log" --stall-minutes 15 -- \
271+
timeout --kill-after=2m "${budget}m" \
272+
pnpm turbo run test --only --concurrency=4 --force --log-order=stream || status=$?
273+
fi
203274
echo "PASS2_EXIT=$status" >> "$GITHUB_ENV"
204275
205276
# ⭐ THE VERDICT, and the job's exit status (#18590).

‎.github/workflows/scheduled-full-run-card.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,14 @@ on:
7272
workflow_run:
7373
# Byte-exact workflow NAMES (the `name:` at the top of each file), not
7474
# paths. Quoted because `Lint & Type Check` starts a YAML alias unquoted.
75-
workflows: ['CI', 'Lint & Type Check']
75+
#
76+
# `Rerun Safety` (rerun-safety-nightly.yml, #23880) runs once a night, and
77+
# its scheduled runs from 2026-09-17 to 10-16 concluded `success` once in
78+
# thirty nights with no filer at all. It gets its own card by the same
79+
# identity rule, in its own cadence's words: every name listed here must
80+
# have an entry in `CARD_CADENCES` (scripts/ci/scheduled-full-run.mjs),
81+
# which refuses a name it does not hold rather than default one.
82+
workflows: ['CI', 'Lint & Type Check', 'Rerun Safety']
7683
types: [completed]
7784

7885
permissions: {}

0 commit comments

Comments
 (0)