Skip to content

Commit 16abea8

Browse files
feat(scripts): a test file of either ruled class is listed over its own declared budget, not the 10 s line (#23595)
Fixes #23534 Clause-②: no The refresh diff (`scripts/measure-test-shard-timings.mjs --diff`) now reads a test file's own measured budget and lists that file only above it, as ruling record 6181973363 (letter B, both classes) binds. A file without the line stays under the 10 s line, exactly as before. The `AGENTS.md` sentence is its own Tier H draft PR (linked from the card); this PR does not wait for it. ## The declaration One line in the test file's header: ```text // test-budget: SECONDS — CLASS: REASON ``` - **SECONDS**: the measured CI reading x 1.25, rounded up to the next 10 s. It must be a whole number, a multiple of 10, and above the 10 s line. A budget of 10 or less lifts nothing, so it is refused. - **CLASS**: exactly one of the ruling's two phrases, spelled as the ruling names them (`TEST_BUDGET_CLASSES`): - `spawns the real producer once per single-cause fixture` - `whole-package import` - **REASON**: must not be empty. - **Separators**: an em dash between single spaces, then a colon and a space after the class. - **Placement**: a `//` line at column 0 in the file's **header**. The header is the leading run of blank lines, `//` lines and block comments, up to the first line of code. The script header (`readTestBudget`'s docblock) states the same grammar. **Refused loudly, never skipped:** any comment line whose text opens with `test-budget:` counts as a declaration attempt, in any case and also spelled `test budget` or `test_budget`. Such a line is refused when it does not match the grammar, when it is a second one, or when it sits below the header. The refusal throws, so `--diff` exits non-zero, and the message names `path:line`. A file the window has over 10 s is the only file asked: a declaration cannot lift a file the line does not list. ## What the body shows - A declared file over its budget is listed, and its row shows which line applied: `| scripts/liveness/check-liveness.test.ts (local) | @objectstack/spec | r1 | 640.00 (declared 620) | 1 / 1 |`. The runs-over count is taken against the declared budget. - **Held to the budget it declares:** every declared file the window read over 10 s, with its reading against its budget (`902.10 s of its 1130 s`). - **No budget declaration could be read, so the 10 s line holds:** a captured file that is not at its path in this checkout (renamed or deleted since the window ran), or a package name the workspace does not have. The diff does not crash, and the file is not silently exempted. - The heading and the seat's duty item 2 now say "over budget (10 s of wall, unless the file declares a measured budget with its class)". ## PM mechanism assumptions, measured - **A1 holds: no workflow edit.** `shard-timings-refresh.yml` checks out the full tree (`actions/checkout`, line 260). It runs `node scripts/measure-test-shard-timings.mjs --diff …` from the repo root (line 799). No step between them switches or narrows the tree: the only `git switch` is at line 995, after the body is composed. The script resolves files through its own `REPO_ROOT`. - **A2: how a row names a file.** - A capture row is `{pkg, file, project}`. `file` is the path vitest printed, relative to the package directory (its root). The project is a separate field. - #23180's real rows show this shape: `scripts/build-schemas-check-mode.test.ts (repo)` under `@objectstack/spec`. - The path resolves as the script's existing workspace index (`packageDirForName`) plus `file`. - A real `--diff` on this tree, with captures naming those rows, found `packages/spec/scripts/build-schemas-check-mode.test.ts` and read it (no declaration, so 902.10 s is listed against 10 s, as today). It named the missing `src/nope-renamed.test.ts` as "no file at packages/spec/src/nope-renamed.test.ts in this checkout", and an unknown package as "no workspace package by that name". Exit 0. - **A3: the annotation went false, and was already half false; both halves are rewritten.** - The `dispatch-gates: no-path-population` line said `--diff` reads only argv paths. That is false now. - The line also said the `--self-test` "reads no tracked file". That was already false on `main`: the self-test's workspace-resolution cases index every package manifest and count `@objectstack/driver-turso`'s test files. That is also why an instrumented copy needs `packages/` beside it. - The new reason states both reads. The marker key is unchanged, and the derivation is unchanged: `--commands` derives the same 32 commands the dispatch lead listed, and `pnpm check:pm-dispatch-gates` exits 0. - **A4 is false as stated: there are five batteries, not one.** On `70d4c063`, an instrumented copy measured the registered counts as `self-test` 56, `env-carried slices` 11, `minimum executed runs` 14, `suite-duration ceilings` 14 and `refresh diff and per-file budget` 22. Each equals its pin, and the floor is 5. This PR adds a sixth battery, `per-file budget declarations (ruling 6181973363)`: 14 cases registered and pinned at 14, floor 5 to 6. The handshake (`SELF_TEST_VERDICT`) is unchanged. The new cases run in memory and in a temp dir. The CLI case passes `diffMain` a `packageDir` resolver pointed at the temp dir, so no tracked file is read. - **A5 holds.** An end-to-end run of the real CLI was measured in a replica tree whose spec test carried `whole-packge import` (a typo). It exited **1**, wrote no body, and its stderr opened `Error: packages/spec/src/shared/alias-integrity.test.ts:2: the class "whole-packge import" is neither class the ruling names. …`. The workflow's `|| DIFF_EXIT=$?` (lines 799–819) puts "Regression readings: NOT MEASURED -- the diff refused" plus that stderr into the refresh PR body. The same replica, with the two spec files declared at 1130 and 620, exited 0. It held `build-schemas-check-mode.test.ts` (902.10 s of 1130 s) out of the list and listed `check-liveness.test.ts` at `640.00 (declared 620)`. `FILE_BUDGET_SECONDS = 10` and the self-test's `!== 10` assertion are unchanged. No `packages/spec/**` file and no `.github/workflows/**` file is touched. The declarations themselves belong to the spec lane on #23180, which is not addressed here. ## Tests - `node scripts/measure-test-shard-timings.mjs --self-test` printed `measure-test-shard-timings: self-test OK`, exit 0. Six batteries registered 56/11/14/14/22/14 against pins 56/11/14/14/22/14. - Two existing assertions moved with the rendered text, and nothing else in them changed: - the NOT MEASURED heading substring; - the existing CLI round trip, which now passes `{ packageDir: () => null }` so it stays off the tracked tree. - **Ablation (one-off, on `40215f6c`, through `scripts/ablation-replace.mjs`):** - Deleted the class check (`if (!TEST_BUDGET_CLASSES.includes(cls)) refuse(…)`). The anchor count went 1 to 0 and the blob `e224fd82` to `0ea2558b`. - The self-test went **red**, exit 1: `Error: budget: a class naming neither read as null`. - Restored with `git checkout HEAD --`: blob `e224fd82` matches HEAD, `git diff HEAD` is empty, the anchor count is back to 1, and the self-test is OK again. No permanent mutation test. - `pnpm exec eslint --no-inline-config scripts/measure-test-shard-timings.mjs` exited 0. The config resolves for the path (`--print-config`). ## Gates (local, at `40215f6c`, the final head) - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 32 commands from git off merge base `70d4c063`. All 32 ran and all 32 exited 0, each exit captured before any pipe. - The 32 cover `pnpm check:pm-dispatch-gates` (detached, exit 0: `dispatch-gates self-test (fast tier): 1841 cases pass`), `check:nul-bytes`, `check:cross-package-test-inputs`, `check:entry-guard`, the four self-test-wiring and anchor gates with their self-tests, and the three shard-timing self-tests. - `--ran` printed `✓ dispatch-gates --ran: 32 derived famil(ies) accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero …)`. - Every family CI runs is read from this PR's check-runs. The shard-timings-refresh `pull_request` dry run is the end-to-end reading of the workflow path. ## Size +358 / −27 in one file, against the ruling's estimate of about 60 lines. The breakdown: - **The self-test:** about 180 lines (158 code, 22 comment). The 14 cases cover what the dispatch asked for, plus the near-miss spellings, the only-over-10 s read, and a CLI round trip from disk. - **Outside the self-test:** about 178 lines (118 code, 60 comment). The comment lines are the grammar's docblock, the rewritten annotation and the two section headers that described the list. ## Acceptance notes (not filed) - **Stale workflow prose.** The workflow's header prose still says "every test file over 10 s of wall in the window" (lines 178–181) and "a test file over 10 s of wall is a card for the owning lane" (190–191). It is a description, not a mechanism. Editing it is a governed `.github/workflows/**` change, so it is not made here. Carrier: none. - **Stale self-test docblock.** The self-test's docblock still reads "This file declares ONE battery". It was stale before this PR, which adds a sixth battery. Carrier: none. - **Sibling self-test reads the workspace index.** `scripts/ci/select-shard-timings-run.mjs --self-test` spawns the real `--diff` with a fixture package `@objectstack/a`. That run now reads the workspace index to look up the fixture's declaration: it finds none, the 10 s line holds, and its assertions are unchanged (it passed above). Carrier: none. - **No PR-time parse of a declaration.** A declaration is parsed only by the weekly refresh, so a malformed line in a test file lands green and refuses the next refresh. No new gate is added, by default. The spec lane can check its lines locally: ```bash node -e "import('./scripts/measure-test-shard-timings.mjs').then((m) => console.log(m.readTestBudget(require('fs').readFileSync(process.argv[1], 'utf8'), process.argv[1])))" packages/spec/scripts/build-schemas-check-mode.test.ts ``` --- _Generated by [Claude Code](https://claude.ai/code/session_01ESFcvW3staenyBCGq7kcDH)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 2372e0a commit 16abea8

1 file changed

Lines changed: 358 additions & 27 deletions

File tree

0 commit comments

Comments
 (0)