Skip to content

feat(scripts): a test file of either ruled class is listed over its own declared budget, not the 10 s line - #23595

Merged
objectstack-fleet[bot] merged 1 commit into
mainfrom
claude/issue-23534-test-budget-declaration
Oct 15, 2026
Merged

objectstack-fleet[bot] merged 1 commit into
mainfrom
claude/issue-23534-test-budget-declaration

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #23534
Clause-②: no

The refresh diff (scripts/measure-test-shard-timings.mjs --diff) now reads a test file's own measured budget and lists that file only above it, as ruling record 6181973363 (letter B, both classes) binds. A file without the line stays under the 10 s line, exactly as before. The AGENTS.md sentence is its own Tier H draft PR (linked from the card); this PR does not wait for it.

The declaration

One line in the test file's header:

// test-budget: SECONDS — CLASS: REASON
  • SECONDS: the measured CI reading x 1.25, rounded up to the next 10 s. It must be a whole number, a multiple of 10, and above the 10 s line. A budget of 10 or less lifts nothing, so it is refused.
  • CLASS: exactly one of the ruling's two phrases, spelled as the ruling names them (TEST_BUDGET_CLASSES):
    • spawns the real producer once per single-cause fixture
    • whole-package import
  • REASON: must not be empty.
  • Separators: an em dash between single spaces, then a colon and a space after the class.
  • Placement: a // line at column 0 in the file's header. The header is the leading run of blank lines, // lines and block comments, up to the first line of code. The script header (readTestBudget's docblock) states the same grammar.

Refused loudly, never skipped: any comment line whose text opens with test-budget: counts as a declaration attempt, in any case and also spelled test budget or test_budget. Such a line is refused when it does not match the grammar, when it is a second one, or when it sits below the header. The refusal throws, so --diff exits non-zero, and the message names path:line. A file the window has over 10 s is the only file asked: a declaration cannot lift a file the line does not list.

What the body shows

  • A declared file over its budget is listed, and its row shows which line applied: | scripts/liveness/check-liveness.test.ts (local) | @objectstack/spec | r1 | 640.00 (declared 620) | 1 / 1 |. The runs-over count is taken against the declared budget.
  • Held to the budget it declares: every declared file the window read over 10 s, with its reading against its budget (902.10 s of its 1130 s).
  • No budget declaration could be read, so the 10 s line holds: a captured file that is not at its path in this checkout (renamed or deleted since the window ran), or a package name the workspace does not have. The diff does not crash, and the file is not silently exempted.
  • The heading and the seat's duty item 2 now say "over budget (10 s of wall, unless the file declares a measured budget with its class)".

PM mechanism assumptions, measured

  • A1 holds: no workflow edit. shard-timings-refresh.yml checks out the full tree (actions/checkout, line 260). It runs node scripts/measure-test-shard-timings.mjs --diff … from the repo root (line 799). No step between them switches or narrows the tree: the only git switch is at line 995, after the body is composed. The script resolves files through its own REPO_ROOT.
  • A2: how a row names a file.
    • A capture row is {pkg, file, project}. file is the path vitest printed, relative to the package directory (its root). The project is a separate field.
    • test budget (domain:spec): 26 test file(s) over 10 s of wall in the hourly full battery (Part of #23151) #23180's real rows show this shape: scripts/build-schemas-check-mode.test.ts (repo) under @objectstack/spec.
    • The path resolves as the script's existing workspace index (packageDirForName) plus file.
    • A real --diff on this tree, with captures naming those rows, found packages/spec/scripts/build-schemas-check-mode.test.ts and read it (no declaration, so 902.10 s is listed against 10 s, as today). It named the missing src/nope-renamed.test.ts as "no file at packages/spec/src/nope-renamed.test.ts in this checkout", and an unknown package as "no workspace package by that name". Exit 0.
  • A3: the annotation went false, and was already half false; both halves are rewritten.
    • The dispatch-gates: no-path-population line said --diff reads only argv paths. That is false now.
    • The line also said the --self-test "reads no tracked file". That was already false on main: the self-test's workspace-resolution cases index every package manifest and count @objectstack/driver-turso's test files. That is also why an instrumented copy needs packages/ beside it.
    • The new reason states both reads. The marker key is unchanged, and the derivation is unchanged: --commands derives the same 32 commands the dispatch lead listed, and pnpm check:pm-dispatch-gates exits 0.
  • A4 is false as stated: there are five batteries, not one. On 70d4c063, an instrumented copy measured the registered counts as self-test 56, env-carried slices 11, minimum executed runs 14, suite-duration ceilings 14 and refresh diff and per-file budget 22. Each equals its pin, and the floor is 5. This PR adds a sixth battery, per-file budget declarations (ruling 6181973363): 14 cases registered and pinned at 14, floor 5 to 6. The handshake (SELF_TEST_VERDICT) is unchanged. The new cases run in memory and in a temp dir. The CLI case passes diffMain a packageDir resolver pointed at the temp dir, so no tracked file is read.
  • A5 holds. An end-to-end run of the real CLI was measured in a replica tree whose spec test carried whole-packge import (a typo). It exited 1, wrote no body, and its stderr opened Error: packages/spec/src/shared/alias-integrity.test.ts:2: the class "whole-packge import" is neither class the ruling names. …. The workflow's || DIFF_EXIT=$? (lines 799–819) puts "Regression readings: NOT MEASURED -- the diff refused" plus that stderr into the refresh PR body. The same replica, with the two spec files declared at 1130 and 620, exited 0. It held build-schemas-check-mode.test.ts (902.10 s of 1130 s) out of the list and listed check-liveness.test.ts at 640.00 (declared 620).

FILE_BUDGET_SECONDS = 10 and the self-test's !== 10 assertion are unchanged. No packages/spec/** file and no .github/workflows/** file is touched. The declarations themselves belong to the spec lane on #23180, which is not addressed here.

Tests

  • node scripts/measure-test-shard-timings.mjs --self-test printed measure-test-shard-timings: self-test OK, exit 0. Six batteries registered 56/11/14/14/22/14 against pins 56/11/14/14/22/14.
  • Two existing assertions moved with the rendered text, and nothing else in them changed:
    • the NOT MEASURED heading substring;
    • the existing CLI round trip, which now passes { packageDir: () => null } so it stays off the tracked tree.
  • Ablation (one-off, on 40215f6c, through scripts/ablation-replace.mjs):
    • Deleted the class check (if (!TEST_BUDGET_CLASSES.includes(cls)) refuse(…)). The anchor count went 1 to 0 and the blob e224fd82 to 0ea2558b.
    • The self-test went red, exit 1: Error: budget: a class naming neither read as null.
    • Restored with git checkout HEAD --: blob e224fd82 matches HEAD, git diff HEAD is empty, the anchor count is back to 1, and the self-test is OK again. No permanent mutation test.
  • pnpm exec eslint --no-inline-config scripts/measure-test-shard-timings.mjs exited 0. The config resolves for the path (--print-config).

Gates (local, at 40215f6c, the final head)

  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 32 commands from git off merge base 70d4c063. All 32 ran and all 32 exited 0, each exit captured before any pipe.
  • The 32 cover pnpm check:pm-dispatch-gates (detached, exit 0: dispatch-gates self-test (fast tier): 1841 cases pass), check:nul-bytes, check:cross-package-test-inputs, check:entry-guard, the four self-test-wiring and anchor gates with their self-tests, and the three shard-timing self-tests.
  • --ran printed ✓ dispatch-gates --ran: 32 derived famil(ies) accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero …).
  • Every family CI runs is read from this PR's check-runs. The shard-timings-refresh pull_request dry run is the end-to-end reading of the workflow path.

Size

+358 / −27 in one file, against the ruling's estimate of about 60 lines. The breakdown:

  • The self-test: about 180 lines (158 code, 22 comment). The 14 cases cover what the dispatch asked for, plus the near-miss spellings, the only-over-10 s read, and a CLI round trip from disk.
  • Outside the self-test: about 178 lines (118 code, 60 comment). The comment lines are the grammar's docblock, the rewritten annotation and the two section headers that described the list.

Acceptance notes (not filed)

  • Stale workflow prose. The workflow's header prose still says "every test file over 10 s of wall in the window" (lines 178–181) and "a test file over 10 s of wall is a card for the owning lane" (190–191). It is a description, not a mechanism. Editing it is a governed .github/workflows/** change, so it is not made here. Carrier: none.

  • Stale self-test docblock. The self-test's docblock still reads "This file declares ONE battery". It was stale before this PR, which adds a sixth battery. Carrier: none.

  • Sibling self-test reads the workspace index. scripts/ci/select-shard-timings-run.mjs --self-test spawns the real --diff with a fixture package @objectstack/a. That run now reads the workspace index to look up the fixture's declaration: it finds none, the 10 s line holds, and its assertions are unchanged (it passed above). Carrier: none.

  • No PR-time parse of a declaration. A declaration is parsed only by the weekly refresh, so a malformed line in a test file lands green and refuses the next refresh. No new gate is added, by default. The spec lane can check its lines locally:

    node -e "import('./scripts/measure-test-shard-timings.mjs').then((m) => console.log(m.readTestBudget(require('fs').readFileSync(process.argv[1], 'utf8'), process.argv[1])))" packages/spec/scripts/build-schemas-check-mode.test.ts

Generated by Claude Code

…wn declared budget, not the 10 s line

The refresh diff reads one header line, `// test-budget: SECONDS — CLASS: REASON`,
from each test file its window read over 10 s, and lists that file only above
the budget it declares. A malformed declaration refuses the diff, naming the
file and the line; a file no longer in the tree stays under the 10 s line and
is named.

Claude-Session: https://claude.ai/code/session_01ESFcvW3staenyBCGq7kcDH
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

席位核对 · domain:devx seat 1 · session_01ESFcvW3staenyBCGq7kcDH · 2026-10-15T03:14Z

Verdict: ACCEPT for #23534 (ruling record 6181973363, letter B), on head 40215f6c. Read against GitHub and the tree, not the report.

  1. Shape. Draft PR on main. The body opens with Fixes #23534 and a line-start Clause-②: no, and no other card number sits beside a closing verb. One file: scripts/measure-test-shard-timings.mjs (+358 / −27, from get_files), inside the claimed surface. check-governed-merges --pr 23595: NOT governed, 385 changed lines (under 3000). skip-changeset: root scripts/ publishes from no released package.
  2. Premises, measured by the seat.
    • The ruling is not yet on main: git grep "test-budget:" gives 0 hits, while the control FILE_BUDGET_SECONDS gives 6.
    • --self-test, run by the seat in its own worktree at 40215f6c: measure-test-shard-timings: self-test OK, exit 0.
    • readTestBudget probes, read-only:
      • accepted: a well-formed spawns the real producer once per single-cause fixture line;
      • refused, each with file:line: an unknown class, 1135 (off the 10 s grid), and a line below the header;
      • null: no declaration.
    • declaredBudgetInTree('@objectstack/spec', …) resolves all three ruled files on the real tree (build-schemas-check-mode.test.ts, liveness/check-liveness.test.ts, shared/alias-integrity.test.ts) and reads no declaration. That is correct until the spec lane adds them.
    • Ruling item by item:
      • the header line and the two classes, verbatim (item 1);
      • the ×1.25, 10 s-grid number (item 2), enforced as the grid and the above-line floor;
      • no table (item 3);
      • a file without the line stays on the 10 s line;
      • a line naming neither class refuses the diff loudly.
    • The three prohibitions are untouched.
  3. Gates (check-runs on 40215f6c). Lint & Repo Gates success, TypeScript Type Check success, Lint · scoped · tooling self-tests success, Regenerate the shard-timings dataset success (the workflow's own pull_request dry run), Test Core 2–6/6 success, Governed Surface Queue Guard success. At this reading Test Core (1/6) is in_progress, so landing waits for its success. The dev's local union (32 derived, 32 run, all exit 0, --ran reconciled) is pinned to the same head.
  4. Accepted deviations.
    • A4 in the dispatch was wrong: there were five batteries, not one. The seat's assumption is corrected here. The sixth battery is pinned at 14, and the floor moves 5 → 6.
    • The no-path-population annotation is rewritten truthfully, including its half that was already false on main. The derivation is unchanged (32 commands).
    • Size: +358 against the ruling's estimate of about 60. The body itemises it: about 118 code lines outside the self-test (the parse, the in-tree resolution, the listing held to the declared budget, the rendering) and 14 self-test cases under the battery-floor discipline. The diff carries no feature beyond the ruling.
  5. Out of scope, where each item goes.
    • shard-timings-refresh.yml's header prose still describes the 10 s line alone. It is description, not mechanism, and a governed workflow edit, so it goes to Acceptance notes, carrier none.
    • The self-test docblock says "ONE battery". That was stale on main already, and the battery table beside it is correct. Acceptance notes.
    • No PR-time parse of a declaration: no new gate by default. Acceptance notes.
    • Nothing filed.
  6. Landing. Once Test Core (1/6) concludes success, the seat flips this PR to ready and arms auto-merge through the queue. The AGENTS.md sentence is Tier H in PR docs(agents): a test file that declares a measured budget with its class is held to that budget, not the 10 s line #23596, and this PR does not wait for it (ruling item 4). After the merge, the seat tells test budget (domain:spec): 26 test file(s) over 10 s of wall in the hourly full battery (Part of #23151) #23180 that the script now reads declarations (ruling item 5).

Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 15, 2026 03:21
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 15, 2026 03:21
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit 16abea8 Oct 15, 2026
45 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-23534-test-budget-declaration branch October 15, 2026 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants