Skip to content

Commit 6abd265

Browse files
ci: a per-package suite-duration ratchet — Test Core reds a suite over its measured ceiling (#22480)
Part of #16468 Clause-②: no The per-package suite-duration ratchet. `Test Core` now grades every executed package's test suite against a measured ceiling and goes red when one runs past it, naming the package and its slowest test files. The ceilings are generated by the shard-timings refresh into `scripts/test-shard-timings.json`, and a ceiling rises only by a ruling. This PR is `Part of` the card because no seat can write the first ceiling table (see "Bootstrap" below). The first scheduled run of `.github/workflows/shard-timings-refresh.yml` after this merges writes it. Until that refresh PR merges, the check prints NOT MEASURED on every run, with a warning annotation. What finishes the card: that refresh PR merged, and a `Test Core` run on `main` printing `suite-ceiling: OK` over graded packages. The held-versus-refreshed question below should also be answered before the second refresh. Authority (maintainer, quoted in the card body): 「同意你的建议,你负责执行派发所有可行的优化」 ## The ruling (comment 6079579826, quoted verbatim) > **A — the ceiling is the package's slowest executed run in the refresh window × 1.25; a provisional package is not red.** The ratchet's three ruled clauses stand: the ceilings are a generated table beside the timings file, refreshed with it, 25% headroom; the check runs after the shard's tests, reads the turbo summary, and reds a package over its ceiling naming the package and its slowest files; a ceiling rises only by a ruling. "The refreshed measurement" is read as the window maximum, so the window's own variation never trips the brake, and a pull request goes red only when a suite runs 25% slower than its slowest measured run. A package in the dataset's `provisional` list prints "no ceiling: provisional" and is not red; only a package absent from the dataset is red under clause 3, the new-package shape. Build order: the dev first measures A's false-red rate over the 21-run window (expected 0; if not 0, stop and report, do not force it), then builds the check; the CLI's three pull-request-time slices (#22075, PR #22415) are summed across shards or each compared to a third of the ceiling, an implementation detail the dev reports. ⛔ Not taken: B (31 packages already over the line with no code change; a brake that is red by default is learned around, the #16465 round-55 lesson), C (no red at the pull request, which is the card's core), D (no brake). ## What changed - **`scripts/measure-test-shard-timings.mjs` (the generator).** - New `ceilings` field, mapping each package to seconds. A package's ceiling is its slowest executed run in the refresh window × `CEILING_HEADROOM` (1.25), rounded up to the hundredth. For a file-sliced package, the "run" is the per-run sum of its slices. - New `uncapped` field, naming each package that gets no ceiling and why. `provisional` means fewer than 3 executed runs. `carried` means a cache HIT re-confirmed the package but no run in the window executed it. - `provenance.ceilingHeadroom` records the headroom. - The medians in `packages` are unchanged, and so is every reader of them (the partitioner and the drift step). - The spread table in the refresh PR body gains a `ceiling s` column. - **`scripts/check-test-suite-ceilings.mjs` (the check, new).** - Reads the per-shard timing captures that `report-test-timings.mjs --capture` already writes, and sums a sliced package's parts across shards. - Grades each executed package: - over its ceiling: red, naming the package and its 5 slowest files; - absent from the dataset: red; - `provisional`: prints `no ceiling: provisional`, not red; - `carried`: prints `no ceiling: carried`, not red; - a sliced package missing a part: NOT MEASURED for that package. - If the dataset has no `ceilings` field, the whole run reads NOT MEASURED and the step exits 0. - `RULED_CEILING_RAISES` is the ruled-raises record. Each entry is a package, its raised seconds and the ruling's comment URL. An entry without a well-formed ruling URL is refused, and the step goes red. - **`.github/workflows/ci.yml`.** - `Check the per-package suite-duration ceilings` is a new last step of the `Test Core` aggregator job, `if: always()`, with no `continue-on-error`. - The aggregator's comment block now names it as the one step there that does red the job. - Not changed: the job and step names, `needs:`, the matrix, the shard `timeout-minutes` (45), the drift step and its constants (1.5× red, 1.3× warning), the seven required contexts, and #22075's split (`FILE_SHARDED_PACKAGES`, the density cap, `planShards()`). - **`.github/workflows/lint.yml`.** New step `Suite-duration ceiling check self-test` (wiring for `check-self-test-wired`). - **`.github/workflows/shard-timings-refresh.yml`.** - The PR body gets a ceiling line. It says whether this refresh writes the FIRST table, or holds the existing one and adds new packages. It also names the uncapped packages, and any package that ran over its held ceiling in the window. - The header records that a refresh never moves a ceiling. - No change to selection, download, the generate loop or the write step. ## The table: a new field in the dataset, not a new file The claim left this choice to me. I chose a field, for three reasons: - **The pairing is structural.** One generator call writes the weights and the ceilings into one file, and the refresh lane's existing `cmp`, `git add` and PR body cover both. Nothing can refresh one without the other. - **"No table yet" is a property of the file.** An absent `ceilings` key means a dataset written before this generator version. The dataset already uses the same pattern for `provisional`. After the first refresh the key is always present, so deleting it would mean hand-editing a generated file. - **The ruling's "beside the timings file" holds as "beside the weights".** `packages` is untouched, and so is every reader of it. ## Bootstrap: no seat can write the first table (assumption 1, confirmed) - I confirmed it once, on artifact 11612571468 of run 37921147590. `GET /actions/artifacts/{id}/zip` answered `302` to `productionresultssa3.blob.core.windows.net`. The container's egress then answered `CONNECT tunnel failed, response 403`, and the proxy logged `connect_rejected ... 403` for that host. - Job logs redirect to the same host, and MCP `download_workflow_run_artifact` returns the same blob URL. - So the table is generated only by the refresh lane on a hosted runner. No ceiling is typed by hand, and none is seeded from PR #22368's body. - The bootstrap path: 1. This PR merges. The check reads NOT MEASURED, with a warning annotation, on every run. 2. The next refresh (scheduled Monday 05:30 UTC, or a manual dispatch by the maintainer) writes `ceilings`. It opens its PR with the "FIRST suite-duration ceiling table" paragraph and the ceiling column. 3. That PR merges, and the check grades from the next run on. - This PR's own `pull_request` run of the refresh lane (it touches the generator) is a dry run on real artifacts, so its step summary shows the table a scheduled run would write. Nothing is pushed from it. ## A's false-red rate (build order, before the check was built) **Over the 21-run window: 0, by construction.** Every reading in the window is at most the window maximum, and the ceiling is the window maximum × 1.25. **On the runs after the window** (assumption 2), the measurement that can say something: - **Runs:** all 13 hourly `schedule` runs of `ci.yml` after the window's newest run (37850732030). That is 37857021445 through 37921147590; 12 of them ran after the dataset refresh landed (`040184752c`). - **Readings:** each run's `Test Core` timing table, read from the aggregator job log (13 MCP `get_job_logs` reads). It lists the 10 slowest executed packages with turbo's own window. The same function reads those windows for the dataset, for the check, and for the PR #22368 spread table. - **Ceilings:** window maximum × 1.25, from PR #22368's spread table. The table was used as the measurement input only, not as table data. - **Density per run, by commit ancestry.** Round 2 of #22075 (`c64130bafa`, reverted at `806b03e2ae`) is in the commits of 37897571511 and 37902633826. Every negative ancestry check has a control leg (`040184752c` is an ancestor of every commit, exit 0). | runs | executed package-runs | read directly | over A's ceiling | bounded under by the 10th-slowest reading | provisional | not measured (below the top-10 cut) | |---|--:|--:|--:|--:|--:|--:| | 11 at the dataset's density | 497 | 108 | **0** | 46 | 4 | 339 | | 2 under round 2's density (37897571511, 37902633826) | 128 | 20 | **2** | 11 | 2 | 95 | - **The two exceedances** are both on 37902633826 (`081e6a09d4`), the one run in which round 2's split executed every package: - `@objectstack/service-automation`: 567.21 s against 497.55 s (1.14×); - `@objectstack/metadata-protocol`: 769.55 s against 717.28 s (1.07×). - On the 11 runs at the dataset's density those two packages read at most 420.65 s and 529.41 s. - Round 2 packed ~2641 s of predicted windows onto three shards (1.49× the density cap), so these readings are the split's packing, not an unchanged tree. I read them as the brake firing on a pull request that made suites slower, not as noise, and so I did not stop. - **Near misses at the normal density:** - `@objectstack/plugin-approvals` read 218.90 s against 222.47 s (0.98) on 37869710053. That is 1.23× its window maximum, within a day of the window. - `@objectstack/plugin-auth` read 522.15 s against 537.76 s (0.97) on 37886662177. - So A's 25% over the window maximum has thin margin on some packages. The week after the first table lands is when this gets read on graded runs. - **The coverage gap.** The logged table prints only the top 10 packages, so 339 small-package readings could not be bounded. The check prints every executed package's reading, ceiling and ratio in its log, which makes the next measurement of this kind complete. ## The CLI's slices: summed across shards (assumption 3) - The three slice windows are summed, and the sum is graded against the whole package's ceiling. That is the quantity the ceiling was measured in: the generator records a sliced package as the per-run sum of its slices, and takes the slowest of those sums. - Comparing each slice to a third of the ceiling was rejected on PR #22456's measured skew, 0.81 / 1.17 / 1.02 of an even third. With the CLI ceiling at 1844.58 × 1.25 = 2305.73 s, the heaviest slice would red at a whole-suite 1970.7 s. That is 6.8% over the window maximum instead of 25%. - vitest cuts the file list by a hash of each path. A new or renamed file can move a slow file into another slice while the suite gets no slower. - Summing needs all three shards, so the check runs in the `Test Core` aggregator, which already downloads every shard's capture. The ruling's "after the shard's tests" is met as "after all shards' tests". - A set missing a part is NOT MEASURED for that package. A part is never graded as the whole. ## Density (assumption 4) - The ceilings assume the packing density the dataset's own split runs at. The live split (PR #22456) holds that density under its density cap: no shard carries more predicted windows than the densest bin of the whole-package split, 1772.65 s today. - A package's window is contended wall clock, so it grows with its shard's density. Round 2 measured 1.5–1.9×, and the table above shows two packages over A's ceiling on that run. - A future split that raises the density will red this check on packages it did not touch. That red is right: the split made those suites slower. The remedy is the split, not a raise. - If a split that raises density is ever wanted on purpose, its PR should carry the ruled raises with it. ## Slowest files (assumption 5) These come from the per-file capture of #16454 that already runs in each shard, the vitest module lines in each capture's `files`. The check imports no reader of its own for them: it reads the captures the aggregator already downloads. ## A decision this PR makes, flagged for a ruling: a refresh HOLDS a ceiling The ruled clauses meet head-on at the second refresh: - "the ceiling is the package's slowest executed run in the refresh window × 1.25"; - "refreshed with it"; - "a ceiling rises only by a ruling". This PR **holds**: a refresh writes a ceiling only for a package that has none (new, or newly non-provisional), and never raises or lowers an existing one. Raising follows the ruling. Not lowering is my call, for one reason: a quieter window is variance, not a faster suite, and a ceiling ratcheted down on variance is red on the next ordinary run. The plugin-approvals reading above is that variance, 1.23× its window maximum within a day. The alternatives are a one-line change in `suiteCeilings()`: - shrink-only (lower on a quieter window, never raise), which is the line ratchets' rule; - free-follow (recompute every refresh, so ceilings rise with growth). The first table is identical under all three, so this only needs answering before the second refresh. The open question is in my report on the card. ## A consequence of clause 3 the card should know - A package absent from the dataset is red. A refresh measures only what runs on `main`, and a package whose PR is red never reaches `main`. - So a new package can never get a ceiling by waiting for "the next refresh". Its first ceiling comes from a ruled raise in `RULED_CEILING_RAISES`, which the check honours for a package absent from the dataset. - Renaming a package has the same effect. ## Gates (head `caffe8376d`, after one `origin/main` merge) - **Derivation.** `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` on `caffe8376d` derived 66 commands: 34 `pnpm`, 32 direct `node`, over 5 paths against the merge base `f66c440de`. That is the same list as before the merge. - **Runs.** All 66 ran on `caffe8376d`, each with its exit code captured before any pipe. The 22-minute `pnpm check:pm-dispatch-gates` battery ran in the background, wrote its exit code to a file, and I waited on it in the foreground: exit 0, `dispatch-gates self-test: 2011 cases pass`. - **Reconciliation.** `--ran`: `Run reconciliation — 66 derived, 66 run, 0 NOT-MEASURED, 0 UNRUN`, then `✓ dispatch-gates --ran: 66 derived famil(ies) accounted for — 66 run, 0 NOT-MEASURED (a DERIVED zero — all 66 recorded an exit code and none of them is 3)`. All 66 exited 0. They include: - `node scripts/check-test-suite-ceilings.mjs --self-test`: `self-test OK (35 cases across 7 batteries)`; - `node scripts/measure-test-shard-timings.mjs --self-test`: `self-test OK`. Its new battery `suite-duration ceilings (#16468)` has 14 cases, and the roster floor goes 3 to 4; - `node scripts/partition-test-shards.mjs --self-test`: `bins 1772.63/1771.02/1772.63/1770.95/1771.09/1772.46s within the 1772.65s density cap, file-level slices: @objectstack/cli x3`. The split is unchanged; - `node scripts/ci/select-shard-timings-run.mjs --self-test`, which drives the refresh lane's real generate step with the new generator; - `check-self-test-wired`, `check-self-test-workflow-commands`, `check:required-contexts`, `check:shard-attestation`, `check-aggregator-roster`, `check-step-collectors`, `check:stall-guard-budget`, `check:stall-guard-headroom` and `check:nul-bytes`. - **Before the merge, at `8ab0fb8f0a`.** The same battery went red, 1 of 2011 cases, on the import described in the acceptance notes. The check was changed at `68ca4e88d7`, and the rerun above is green. - **Lint, a declared narrowing, at `caffe8376d`.** `eslint --no-inline-config --format json` over the two changed scripts reported 2 files, 0 errors and 0 warnings. - The population comes from eslint's own config: `isPathIgnored` is false for both scripts and true for `ci.yml`, and workflow YAML is not an eslint input. - The computed config for both scripts has no `parserOptions.project` and no `projectService`. So no type-aware rule runs, and this diff cannot move an untouched file's verdict. - The full `pnpm lint` is CI's. ## Ablations (the code committed first; each through `node scripts/ablation-replace.mjs`, restore proven) Each mutation was made at head `68ca4e88d7`. Each was confirmed on disk by the tool (anchor 1 → 0, blob changed) before the self-test ran. Each was restored with blob == HEAD and an empty `git diff HEAD`. These are plain node scripts, with no build or `dist/` step. 1. **Over-ceiling red removed.** - Mutation: `row.status = seconds > row.ceiling ? 'over' : 'ok';` became `row.status = 'ok';` (blob `1f01cb9c9543` → `40c38e79e976`). - Result: RED, `Error: over: exit 0, verdict OK`. Restored to `1f01cb9c9543`. 2. **Provisional exemption removed.** - Mutation: `if (inDataset && table.provisional.has(name)) {` became `if (false) {` (blob → `e34689295a71`). - Result: RED, `Error: provisional: exit 0, uncapped`. Restored. - The direction is worth knowing. With the exemption gone, a provisional package that is `uncapped` falls to the uncapped row ("no ceiling: provisional"), still not red. A provisional package that still holds a ceiling turns red. - Measured with `--hold` on a one-off fixture (a provisional package at 9999 s holding a 12 s ceiling): restored code gave `NOT MEASURED 0`, the mutated code gave `OVER 1`. Restored, blob == HEAD. 3. **The ruled reading (window maximum, not median) removed from the generator.** - Mutation: `Math.max(...values)` became `median(values)` in `ceilingOf()` (blob `4c08f3d8463b` → `44bda0d56fe4`). - Result: RED, the spread row read `25.00` (median × 1.25) instead of `37.50`. Restored. 4. **The no-table reading removed.** - Mutation: the `if (!Object.hasOwn(dataset, 'ceilings')) return { table: null };` line deleted (blob → `94844d5c01e1`). - Result: RED. A pre-ceilings dataset becomes a refusal, `` `ceilings` is undefined ``, instead of NOT MEASURED. Restored. 5. **Slice completeness removed.** - Mutation: `complete: row.sliceCount === null || parts.length === row.sliceCount` became `complete: true` (blob → `4eb8e8b191c1`). - Result: RED, `Error: slices: an incomplete set read over / OVER`. Restored. ## Acceptance notes - **`check:pm-dispatch-gates` caught a real interaction.** The first draft of the check imported `mergeCaptures` from `report-test-timings.mjs`. That module is a gate file only through its own self-test, so a gate family importing it stops inheriting its declared populations in the dispatch-gates derivation. The battery's promotion invariant went red on exactly that (1 of 2011 cases). The check now folds the captures itself (`foldCaptures`): the same sum and completeness mark, and an unknown capture schema is named and left out. The turbo-summary reader is still the shared `samplesFromSummary`, which runs on the shard when the capture is taken. - **A stale comment, noted rather than edited.** `report-test-timings.mjs`'s header says `samplesFromSummary` "reads the `test` task only". Since the `test:repo` fold it sums both tasks. That file is outside this card's surface. Comment only; noted, not filed. - **`skip-changeset`:** root `scripts/` and workflows publish nothing. --- _Generated by [Claude Code](https://claude.ai/code/session_0115N1oNnQS5WqofZ2DzaT3q)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7895671 commit 6abd265

5 files changed

Lines changed: 1234 additions & 14 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1391,7 +1391,10 @@ jobs:
13911391
# redden it. Ordering after the verdict is what makes "cannot delay"
13921392
# true, and the two flags are what make "cannot redden" true — a
13931393
# diagnostics fault here (an artifact-service 403 is the measured one,
1394-
# #14469) must not evict a green run.
1394+
# #14469) must not evict a green run. The suite-duration ceiling check
1395+
# at the end of this job reads the same captures and is the one step
1396+
# here that DOES red the job, on purpose: it is a gate, not a report, and
1397+
# a capture that never arrived reads NOT MEASURED there, never red.
13951398
#
13961399
# ⛔ Steps, deliberately, NOT a new `needs:` entry: this job's
13971400
# `OS_AGGREGATOR_MEMBERS` / `OS_AGGREGATOR_NON_MEMBERS` block is asserted
@@ -1447,6 +1450,38 @@ jobs:
14471450
if-no-files-found: ignore
14481451
retention-days: 14
14491452

1453+
# ── The per-package suite-duration ratchet (#16468) ──────────────────
1454+
#
1455+
# A GATE, unlike the two reporting steps above: NO `continue-on-error`,
1456+
# so a package whose suite ran past its ceiling reds `Test Core`, naming
1457+
# the package and its slowest test files. The ceiling is the package's
1458+
# slowest executed run in the refresh window x 1.25, generated into
1459+
# scripts/test-shard-timings.json (`ceilings`) by the refresh lane and
1460+
# held by every later refresh; it rises only by a ruled entry in the
1461+
# script's RULED_CEILING_RAISES. A `provisional` package prints "no
1462+
# ceiling: provisional" and is not red; a package absent from the dataset
1463+
# is red (the new-package shape). See the script header for the ruling.
1464+
#
1465+
# HERE, in the aggregator, rather than in each shard, because the CLI's
1466+
# file-level slices land on three shards: the three windows are SUMMED
1467+
# and the sum graded against the whole package's ceiling -- the quantity
1468+
# the ceiling was measured in. It reads the shard captures the download
1469+
# step above already fetched (each package's turbo execution window via
1470+
# samplesFromSummary, and every vitest file line); a capture that did not
1471+
# arrive leaves its packages ungraded and says so, never red.
1472+
#
1473+
# NOT MEASURED, exit 0, while the dataset carries no `ceilings`: they are
1474+
# generated only where the run-summary artifacts are downloadable (the
1475+
# refresh lane), so the first refresh after this lands writes them, and
1476+
# until then every run prints a warning annotation saying so.
1477+
#
1478+
# LAST, after the reporting steps, so the timing table is published
1479+
# whatever this decides. `always()`: it reads only what the shards already
1480+
# captured, so a red shard's packages are graded too.
1481+
- name: Check the per-package suite-duration ceilings
1482+
if: always()
1483+
run: node scripts/check-test-suite-ceilings.mjs --captures "$RUNNER_TEMP/test-timings" --label 'Test Core'
1484+
14501485

14511486
# ── Temporal conformance against live, non-UTC servers (ADR-0053 D-A3) ─────
14521487
#

‎.github/workflows/lint.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4369,6 +4369,16 @@ jobs:
43694369
- name: Test-timing parser self-test
43704370
run: node scripts/report-test-timings.mjs --self-test
43714371

4372+
# The per-package suite-duration ratchet's own instrument (#16468). The
4373+
# gate runs in ci.yml's `Test Core` aggregator, where a clean tree with
4374+
# every suite under its ceiling reads OK whether or not the over-ceiling,
4375+
# absent-package and summed-slice rules still work -- so its self-test,
4376+
# which feeds those shapes in, runs here with teeth. Invoked as `node` for
4377+
# the GATE INVOCATION IDIOM reason at the top of this file. In-process
4378+
# fixtures, no IO, milliseconds.
4379+
- name: Suite-duration ceiling check self-test
4380+
run: node scripts/check-test-suite-ceilings.mjs --self-test
4381+
43724382
# Hand-written `.d.mts` mirrors (#10549). `scripts/js-comment-mask.mjs`
43734383
# and `scripts/check-regen-pending.mjs` are untyped `.mjs` that each ship
43744384
# a hand-written declaration beside them, and both files say "keep this in

‎.github/workflows/shard-timings-refresh.yml‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,16 @@
3333
# quoted verbatim into the PR body and a human decides. See "THE PINS" below.
3434
# ⛔ It never opens a PR when the regenerated file is byte-identical. A weekly
3535
# no-op PR trains everyone to ignore this PR.
36+
# ⛔ It never moves a suite-duration ceiling (#16468). The dataset also
37+
# carries `ceilings`, which `Test Core` grades every executed package
38+
# against (scripts/check-test-suite-ceilings.mjs). The generator writes
39+
# them in the same pass and HOLDS every ceiling the committed dataset
40+
# already has; a package gets a new one only when it had none (its slowest
41+
# executed run in this window x 1.25). A ceiling rises only by a ruling,
42+
# recorded in that script, so a refresh whose window ran slower REPORTS
43+
# the gap in the PR body instead of absorbing it. The first refresh after
44+
# the check landed writes the first table; until it merges the check reads
45+
# NOT MEASURED.
3646
#
3747
# HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE
3848
# ----------------------------------------------------------------------------
@@ -634,6 +644,53 @@ jobs:
634644
}
635645
')
636646
647+
# The suite-duration ceilings (#16468), read out of the dataset like the
648+
# two lines above. The prior table is the COMMITTED dataset, read from
649+
# git because the bins step has already copied the refreshed file over
650+
# it. A refresh never moves a held ceiling, so the line counts what is
651+
# new and what is uncapped, and the spread table below marks every
652+
# held ceiling this window would have set differently -- and every one
653+
# a run of this window already went over.
654+
CEILING_LINE=$(node -e '
655+
const fs = require("fs");
656+
const { execFileSync } = require("child_process");
657+
const d = JSON.parse(fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/refreshed.json", "utf8"));
658+
if (!d.ceilings || typeof d.ceilings !== "object" || !d.uncapped || typeof d.uncapped !== "object") {
659+
throw new Error("the refreshed dataset carries no `ceilings` / `uncapped`");
660+
}
661+
const prior = JSON.parse(execFileSync("git", ["show", "HEAD:scripts/test-shard-timings.json"], { encoding: "utf8" }));
662+
const held = prior.ceilings && typeof prior.ceilings === "object" ? prior.ceilings : null;
663+
const names = Object.keys(d.ceilings);
664+
const fresh = held === null ? names : names.filter((n) => !Object.hasOwn(held, n));
665+
const uncapped = Object.entries(d.uncapped).map(([n, why]) => n + " (" + why + ")");
666+
const h = d.provenance.ceilingHeadroom;
667+
const over = (fs.readFileSync(process.env.RUNNER_TEMP + "/refresh/spread.md", "utf8").match(/OVER its held ceiling/g) || []).length;
668+
const lines = [];
669+
if (held === null) {
670+
lines.push(
671+
"⚠️ This refresh writes the FIRST suite-duration ceiling table: " + names.length + " package(s) get a " +
672+
"ceiling, each its slowest executed run in this window x " + h + ". From the merge of this PR the `Test Core` " +
673+
"check (scripts/check-test-suite-ceilings.mjs) grades every executed package against it -- until then it " +
674+
"reads NOT MEASURED. Every later refresh HOLDS these numbers and a ceiling rises only by a ruling, so " +
675+
"review the ceiling column below before merging."
676+
);
677+
} else {
678+
lines.push(
679+
"Suite-duration ceilings: " + (names.length - fresh.length) + " held unchanged (a refresh never moves one; " +
680+
"a raise is a ruling), " + fresh.length + " new" + (fresh.length ? " (" + fresh.join(", ") + ")" : "") +
681+
", each new one its slowest executed run in this window x " + h + "."
682+
);
683+
}
684+
lines.push(uncapped.length ? "No ceiling: " + uncapped.join(", ") + "." : "Every package carries a ceiling.");
685+
if (over > 0) {
686+
lines.push(
687+
"⚠️ " + over + " package(s) ran OVER their held ceiling in this window (marked in the table): the brake " +
688+
"fired on main. This refresh does not raise them; that takes a ruling."
689+
);
690+
}
691+
console.log(lines.join("\n\n"));
692+
')
693+
637694
SHARDS=$(node -e '
638695
const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
639696
const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID);
@@ -675,10 +732,12 @@ jobs:
675732
echo "because a weekly lane cannot know which cards a given run ought to retire. If this is"
676733
echo "the first refresh to land, retire those two by hand as part of merging it."
677734
echo
678-
echo "## Sample depth and run-to-run spread"
735+
echo "## Sample depth, run-to-run spread and suite-duration ceilings"
679736
echo
680737
echo "$DEPTH_LINE"
681738
echo
739+
echo "$CEILING_LINE"
740+
echo
682741
cat "$RUNNER_TEMP/refresh/spread.md"
683742
echo
684743
echo "## Measured per-shard suite time on the newest run in the set"

0 commit comments

Comments
 (0)