Repository navigation
Commit 16abea8
feat(scripts): a test file of either ruled class is listed over its own declared budget, not the 10 s line (#23595)
Fixes #23534
Clause-②: no
The refresh diff (`scripts/measure-test-shard-timings.mjs --diff`) now
reads a test file's own measured budget and lists that file only above
it, as ruling record 6181973363 (letter B, both classes) binds. A file
without the line stays under the 10 s line, exactly as before. The
`AGENTS.md` sentence is its own Tier H draft PR (linked from the card);
this PR does not wait for it.
## The declaration
One line in the test file's header:
```text
// test-budget: SECONDS — CLASS: REASON
```
- **SECONDS**: the measured CI reading x 1.25, rounded up to the next 10
s. It must be a whole number, a multiple of 10, and above the 10 s line.
A budget of 10 or less lifts nothing, so it is refused.
- **CLASS**: exactly one of the ruling's two phrases, spelled as the
ruling names them (`TEST_BUDGET_CLASSES`):
- `spawns the real producer once per single-cause fixture`
- `whole-package import`
- **REASON**: must not be empty.
- **Separators**: an em dash between single spaces, then a colon and a
space after the class.
- **Placement**: a `//` line at column 0 in the file's **header**. The
header is the leading run of blank lines, `//` lines and block comments,
up to the first line of code. The script header (`readTestBudget`'s
docblock) states the same grammar.
**Refused loudly, never skipped:** any comment line whose text opens
with `test-budget:` counts as a declaration attempt, in any case and
also spelled `test budget` or `test_budget`. Such a line is refused when
it does not match the grammar, when it is a second one, or when it sits
below the header. The refusal throws, so `--diff` exits non-zero, and
the message names `path:line`. A file the window has over 10 s is the
only file asked: a declaration cannot lift a file the line does not
list.
## What the body shows
- A declared file over its budget is listed, and its row shows which
line applied: `| scripts/liveness/check-liveness.test.ts (local) |
@objectstack/spec | r1 | 640.00 (declared 620) | 1 / 1 |`. The runs-over
count is taken against the declared budget.
- **Held to the budget it declares:** every declared file the window
read over 10 s, with its reading against its budget (`902.10 s of its
1130 s`).
- **No budget declaration could be read, so the 10 s line holds:** a
captured file that is not at its path in this checkout (renamed or
deleted since the window ran), or a package name the workspace does not
have. The diff does not crash, and the file is not silently exempted.
- The heading and the seat's duty item 2 now say "over budget (10 s of
wall, unless the file declares a measured budget with its class)".
## PM mechanism assumptions, measured
- **A1 holds: no workflow edit.** `shard-timings-refresh.yml` checks out
the full tree (`actions/checkout`, line 260). It runs `node
scripts/measure-test-shard-timings.mjs --diff …` from the repo root
(line 799). No step between them switches or narrows the tree: the only
`git switch` is at line 995, after the body is composed. The script
resolves files through its own `REPO_ROOT`.
- **A2: how a row names a file.**
- A capture row is `{pkg, file, project}`. `file` is the path vitest
printed, relative to the package directory (its root). The project is a
separate field.
- #23180's real rows show this shape:
`scripts/build-schemas-check-mode.test.ts (repo)` under
`@objectstack/spec`.
- The path resolves as the script's existing workspace index
(`packageDirForName`) plus `file`.
- A real `--diff` on this tree, with captures naming those rows, found
`packages/spec/scripts/build-schemas-check-mode.test.ts` and read it (no
declaration, so 902.10 s is listed against 10 s, as today). It named the
missing `src/nope-renamed.test.ts` as "no file at
packages/spec/src/nope-renamed.test.ts in this checkout", and an unknown
package as "no workspace package by that name". Exit 0.
- **A3: the annotation went false, and was already half false; both
halves are rewritten.**
- The `dispatch-gates: no-path-population` line said `--diff` reads only
argv paths. That is false now.
- The line also said the `--self-test` "reads no tracked file". That was
already false on `main`: the self-test's workspace-resolution cases
index every package manifest and count `@objectstack/driver-turso`'s
test files. That is also why an instrumented copy needs `packages/`
beside it.
- The new reason states both reads. The marker key is unchanged, and the
derivation is unchanged: `--commands` derives the same 32 commands the
dispatch lead listed, and `pnpm check:pm-dispatch-gates` exits 0.
- **A4 is false as stated: there are five batteries, not one.** On
`70d4c063`, an instrumented copy measured the registered counts as
`self-test` 56, `env-carried slices` 11, `minimum executed runs` 14,
`suite-duration ceilings` 14 and `refresh diff and per-file budget` 22.
Each equals its pin, and the floor is 5. This PR adds a sixth battery,
`per-file budget declarations (ruling 6181973363)`: 14 cases registered
and pinned at 14, floor 5 to 6. The handshake (`SELF_TEST_VERDICT`) is
unchanged. The new cases run in memory and in a temp dir. The CLI case
passes `diffMain` a `packageDir` resolver pointed at the temp dir, so no
tracked file is read.
- **A5 holds.** An end-to-end run of the real CLI was measured in a
replica tree whose spec test carried `whole-packge import` (a typo). It
exited **1**, wrote no body, and its stderr opened `Error:
packages/spec/src/shared/alias-integrity.test.ts:2: the class
"whole-packge import" is neither class the ruling names. …`. The
workflow's `|| DIFF_EXIT=$?` (lines 799–819) puts "Regression readings:
NOT MEASURED -- the diff refused" plus that stderr into the refresh PR
body. The same replica, with the two spec files declared at 1130 and
620, exited 0. It held `build-schemas-check-mode.test.ts` (902.10 s of
1130 s) out of the list and listed `check-liveness.test.ts` at `640.00
(declared 620)`.
`FILE_BUDGET_SECONDS = 10` and the self-test's `!== 10` assertion are
unchanged. No `packages/spec/**` file and no `.github/workflows/**` file
is touched. The declarations themselves belong to the spec lane on
#23180, which is not addressed here.
## Tests
- `node scripts/measure-test-shard-timings.mjs --self-test` printed
`measure-test-shard-timings: self-test OK`, exit 0. Six batteries
registered 56/11/14/14/22/14 against pins 56/11/14/14/22/14.
- Two existing assertions moved with the rendered text, and nothing else
in them changed:
- the NOT MEASURED heading substring;
- the existing CLI round trip, which now passes `{ packageDir: () =>
null }` so it stays off the tracked tree.
- **Ablation (one-off, on `40215f6c`, through
`scripts/ablation-replace.mjs`):**
- Deleted the class check (`if (!TEST_BUDGET_CLASSES.includes(cls))
refuse(…)`). The anchor count went 1 to 0 and the blob `e224fd82` to
`0ea2558b`.
- The self-test went **red**, exit 1: `Error: budget: a class naming
neither read as null`.
- Restored with `git checkout HEAD --`: blob `e224fd82` matches HEAD,
`git diff HEAD` is empty, the anchor count is back to 1, and the
self-test is OK again. No permanent mutation test.
- `pnpm exec eslint --no-inline-config
scripts/measure-test-shard-timings.mjs` exited 0. The config resolves
for the path (`--print-config`).
## Gates (local, at `40215f6c`, the final head)
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 32 commands from git off merge base
`70d4c063`. All 32 ran and all 32 exited 0, each exit captured before
any pipe.
- The 32 cover `pnpm check:pm-dispatch-gates` (detached, exit 0:
`dispatch-gates self-test (fast tier): 1841 cases pass`),
`check:nul-bytes`, `check:cross-package-test-inputs`,
`check:entry-guard`, the four self-test-wiring and anchor gates with
their self-tests, and the three shard-timing self-tests.
- `--ran` printed `✓ dispatch-gates --ran: 32 derived famil(ies)
accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero …)`.
- Every family CI runs is read from this PR's check-runs. The
shard-timings-refresh `pull_request` dry run is the end-to-end reading
of the workflow path.
## Size
+358 / −27 in one file, against the ruling's estimate of about 60 lines.
The breakdown:
- **The self-test:** about 180 lines (158 code, 22 comment). The 14
cases cover what the dispatch asked for, plus the near-miss spellings,
the only-over-10 s read, and a CLI round trip from disk.
- **Outside the self-test:** about 178 lines (118 code, 60 comment). The
comment lines are the grammar's docblock, the rewritten annotation and
the two section headers that described the list.
## Acceptance notes (not filed)
- **Stale workflow prose.** The workflow's header prose still says
"every test file over 10 s of wall in the window" (lines 178–181) and "a
test file over 10 s of wall is a card for the owning lane" (190–191). It
is a description, not a mechanism. Editing it is a governed
`.github/workflows/**` change, so it is not made here. Carrier: none.
- **Stale self-test docblock.** The self-test's docblock still reads
"This file declares ONE battery". It was stale before this PR, which
adds a sixth battery. Carrier: none.
- **Sibling self-test reads the workspace index.**
`scripts/ci/select-shard-timings-run.mjs --self-test` spawns the real
`--diff` with a fixture package `@objectstack/a`. That run now reads the
workspace index to look up the fixture's declaration: it finds none, the
10 s line holds, and its assertions are unchanged (it passed above).
Carrier: none.
- **No PR-time parse of a declaration.** A declaration is parsed only by
the weekly refresh, so a malformed line in a test file lands green and
refuses the next refresh. No new gate is added, by default. The spec
lane can check its lines locally:
```bash
node -e "import('./scripts/measure-test-shard-timings.mjs').then((m) =>
console.log(m.readTestBudget(require('fs').readFileSync(process.argv[1],
'utf8'), process.argv[1])))"
packages/spec/scripts/build-schemas-check-mode.test.ts
```
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ESFcvW3staenyBCGq7kcDH)_
Co-authored-by: Claude <noreply@anthropic.com>1 parent 2372e0a commit 16abea8
1 file changed
Lines changed: 358 additions & 27 deletions
0 commit comments