Repository navigation
Commit 2e3e276
ci(test-shards): widen Test Core from 6 to 8 shards (#25746)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} -->
_Requested by **Elon** · [project
thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DWDhLxhv4cQX1gMYwSzVYWk)_
Fixes #25376
Clause-②: no
Metadata-protocol: none
Test Core goes from 6 to 8 shards. It was sized on a fresh merge-group
timing table on current `main`, as the 2026-10-23T19:55Z unlock asked,
not on the card's 2026-10-13 arithmetic. Eight shards cut the slowest
shard by 10–15% on every replay below, so this clears the ~10% bar. The
Test Core timeout moves 50 → 60, because the stall-guard headroom tool
shows 50 is already too short at six.
## Step 1: the timing table this is sized from
**Merge-group run
[40155484566](https://github.com/objectstack-ai/objectstack/actions/runs/40155484566)**
(PR #25730's queue build, head `c882092b33` on `main` `ec4ef94ead`,
2026-10-25 02:42Z). 59 packages were executed and 1 replayed
(`@objectstack/spec`); the CLI ran as its 3 slices. Read from the `Test
Core` aggregator's timing table and suite-ceiling list, and from each
shard's `shard-timing-drift` line:
| shard | predicted bin | measured test windows | `Run this shard's
tests` step |
| :-- | --: | --: | --: |
| 1/6 | 809.4 s executed + spec (replayed, 1606.6 s) | 929.5 s | 666 s |
| 2/6 | 2416.5 s | 3211.4 s | 1043 s |
| 3/6 (CLI slice) | 2412.5 s | 2136.6 s | 1387 s |
| 4/6 (CLI slice) | 2419.2 s | 3109.2 s | **1794 s** |
| 5/6 (CLI slice) | 2419.5 s | 2661.2 s | 1498 s |
| 6/6 | 2414.5 s | 2368.7 s | 989 s |
Heaviest packages on that run, measured vs pinned: `cli` 3036.30 s vs
2899.02 s (all 3 slices, ~1012 s each), `objectql` 1044.70 vs 970.12,
`runtime` 936.14 vs 848.96, `plugin-auth` 789.01 vs 405.25, `rest`
735.83 vs 876.56, `plugin-security` 666.28 vs 360.02. Spec's heaviest
repo files after #25372: `build-schemas-check-mode.git-topology` 432.88
s, `.deleted-baseline` 403.48 s, `.deletion-proofs` 377.06 s.
A smaller merge-group run on the same day,
[40157571602](https://github.com/objectstack-ai/objectstack/actions/runs/40157571602)
(21 packages executed; core, spec and types replayed), had its slowest
step at 1381 s on the shard holding a CLI slice.
## Step 2: 6 vs 8, predicted
Bins are not wall time. On a shard, the whole packages run four at a
time, and a CLI slice then runs as a serial leg of its own. So a shard's
tests step reads close to `wholeSum / 2.9 + slice`. On run 40155484566
this model lands within 6.1% of the measured step on the four slowest
shards (the three slice shards and shard 2). It misreads the two light
shards, which never set the wall. Each list below is replayed through
the partitioner's own `planRun()` at each count:
| list | slowest shard at 6 | slowest shard at 8 | change |
| :-- | --: | --: | --: |
| run 40155484566 (60 packages) | ~1696 s (measured 1794 s) | ~1436 s |
**−15%** |
| full dataset list (75 packages) | ~1491 s | ~1277 s | −14% |
| run 40157571602 (24 packages) | ~1080 s (measured 1381 s) | ~967 s |
−10% |
The partitioner's own bins on the committed dataset (the CLI as 3
slices; `ratio` = max/mean):
```
6 shards -> max 2489s mean 2488s ratio 1.00x
7 shards -> max 2136s mean 2133s ratio 1.00x
8 shards -> max 1868s mean 1866s ratio 1.00x
9 shards -> max 1661s mean 1659s ratio 1.00x
10 shards -> max 1607s mean 1493s ratio 1.08x <- spec's whole weight is the floor
```
**What eight cannot move:** a CLI slice runs serially for about 960–1010
s. On small runs at eight, the slice shards carry almost nothing else,
so that is where the −10% stops. A ninth shard would mostly add a
runner. The `1.3×` balance bound is unchanged and still met: spec's 1607
s floor sits against `1.3 × 1866 = 2426 s`. Pin 3c (CLI n = 3) is
independent of the shard count and still holds.
**Cost:** over the window below, job start → tests step is 1.6 min
median (5.5 min p90), and the tail after the step is 0.2 min. Two more
shards cost about +3.6 runner-min per run at the median. The card's
maintainer directive accepts this.
## Step 3: the timeout, re-derived
`node scripts/measure-stall-guard-headroom.mjs` covered the 89
completed, non-cancelled CI runs created 2026-10-24 03:00Z → 2026-10-25
03:45Z (23 merge_group, 39 pull_request, 16 push, 11 schedule; 512 `Test
Core (N/6)` jobs):
```
.github/workflows/ci.yml job `test` step `Run this shard's tests`
W 10m · C 20m · T 50m (job timeout-minutes)
worst observed: p 2m03s + s 38m24s = 40m27s (522 observation(s), worst on `Test Core (1/6)`)
undeferred p+s+W = 50m27s vs T 50m -> UNCOVERED by 0m27s
deferred p+s+C = 60m27s vs T 50m -> UNCOVERED by 10m27s
```
So 50 is already short at six. The worst case is a red PR shard (run
40149547507). The healthy worst is 37m22s (p 7m28s + s 29m54s, run
40125226571 `Test Core (4/6)`, job wall 37m41s). Scaled to eight by
0.90, the smallest gain above (the safe side):
| reading | at eight | needs |
| :-- | --: | --: |
| window verdict, p + s + 10m | 2m03s + 34m34s + 10m = 46m37s | > 46.6 |
| cap-deferred verdict, p + s + 20m | 56m37s | > 56.6 |
| 80% wall margin, healthy worst wall | ~34m40s | ≥ 43.3 |
| #16465 red line, slowest shard ≤ 70% of timeout | ~34m40s | ≥ 49.5 |
| drift ceiling, healthy worst step at 1.5× prediction | 7m28s + 26m55s
× 1.5 = 47.8m | > 47.8 |
The largest reading wins, as in the #22075 derivation: **60** (3m23s
over the deferred path). The revert condition in `ci.yml` now reads
"below 60". The first full-battery run after landing re-takes these
readings on eight-shard jobs.
## What changed
- `scripts/partition-test-shards.mjs`: `SHARD_COUNT` 6 → 8. Its header
now carries the 6-vs-8 arithmetic above. The "#10472 asked for 6 → 8 and
the answer was no" paragraph is rewritten (kept, not deleted) to say why
the answer changed.
- `.github/workflows/ci.yml`, Test Core:
- the matrix is `1..8`;
- `name: Test Core (… /8)`, `--shard …/8`, the remote-cache and prune
`--site` labels, the drift `--label`, `--total 8`, and the `-of-8`
artifact names;
- the aggregator's `--leg "test/8:…"`;
- `timeout-minutes: 60`;
- the "WHY SIX" comment block is rewritten as "WHY EIGHT" / "WHY NOT
MORE THAN EIGHT" with today's numbers, and the history paragraphs are
kept.
- `scripts/ci/select-shard-timings-run.mjs`: `SHARD_COUNT` 6 → 8, now
only its matchers' default. Since the second commit, a candidate run is
judged at **its own width**: `runShardWidth` reads `M` from the run's
`Test Core (N/M)` job names, and the run is admitted when all `M` shards
concluded success and all `M` `-of-M` summaries are retained. A run
whose Test Core jobs disagree on `M`, or that has none, is refused by
name. Each candidate carries `shard_count` beside its `artifact_ids`.
- `.github/workflows/shard-timings-refresh.yml`: the summary download
loop and the PR body's per-shard line read the shard roster from the
selector's `artifact_ids` instead of spelling `1 2 3 4 5 6`. The
selector only admits a run when all of that run's shards are present, so
those keys are that run's roster, whatever its width. Wording that said
"six" now names the count generically.
- `scripts/check-shard-attestation.mjs`: the fixtures that edit the live
`ci.yml` (grow the matrix, shrink the roster, the name variants) use the
8-shard spellings. The replays of measured six-shard runs (38036705318,
34007386254) read the live workflow with the six-shard name, and that
substitution is asserted. The live-name pin now checks `Test Core
(1..8/8)`.
- `scripts/check-required-contexts.mjs`: the live-specimen battery
follows the `/8` name.
- `scripts/measure-test-shard-timings.mjs`: four `-of-6` spellings in
comments and output text now name the width generically (`-of-<width>`,
`N-of-M`), since a refresh window can mix widths.
Historical run fixtures elsewhere (`scripts/pm/*`, merge-queue triage)
keep the `(N/6)` names their runs really had.
**No refresh transition window.** The first head of this PR judged every
candidate at the live count, so `Regenerate the shard-timings dataset`
rejected all 24 recent schedule runs (each a whole, green 6-shard run)
as missing `7/8`, `8/8`, and exited 1 ([job
121075952437](https://github.com/objectstack-ai/objectstack/actions/runs/40162213660/job/121075952437)).
That would have held for every refresh until an 8-shard schedule run
existed with its one-day artifacts still retained. The dataset is per
package, and the width a run was split at is not an input to any
package's weight, so the fix reads each run at its own width rather than
reporting NOT MEASURED through the window.
Tests: a new self-test battery, `width change (#25376)` (9 cases: a
whole 6-shard run is eligible under the 8-shard matrix; the same run
pinned to 8 is rejected as job 121075952437 rejected it; an old-width
run missing `6-of-6` or with `6/6` cancelled is refused; 6 jobs with
`-of-8` summaries is refused; jobs disagreeing on `M` and a run with no
Test Core job are refused by name; an unanchored `Test Core (3/8) rerun`
does not move the width; `listCandidates` reports an 8-wide and a 6-wide
run each with its own width). The accumulation battery gains a leg that
drives the refresh workflow's real generate step over one 8-wide and two
6-wide runs, with a package measured only in each run's last shard. Both
were checked to fail against the old reading: restoring the fixed-width
default fails the first case, and restoring a fixed `1..8` download loop
fails the accumulation leg.
**Serial with #25697 / PR #25738:** #25738 landed first (`c61f75d`);
this branch merges it in, with no conflict.
## Acceptance (from the card)
- Partition self-test, `check-aggregator-roster`,
`check:ci-filter-parity` green locally (below) and on this PR.
- After landing, reported on #25376: the first full-battery run's
slowest `Test Core` tests step, and max shard wall / timeout (≤ 70%,
#16465).
## Local verification (head `8c254ff`)
Derived with `node scripts/pm/dispatch-gates.mjs --commands` (70
families) and reconciled with `--ran`: **66 exit 0, 4 NOT MEASURED**.
- NOT MEASURED (exit 3, PREREQUISITE NOT MET, they read built `dist/`
and this worktree has no full build; CI runs them): `pnpm
check:dts-closure`, `pnpm check:dual-build-cjs-loads`, `pnpm
check:lean-entry-closure`, `pnpm check:sourcemap-no-sources-content`.
- Among the green ones: `node scripts/partition-test-shards.mjs
--self-test` (75 measured packages → 77 items, 8 shards, max/mean 1.00x,
bins 1865.66–1867.52 s within the 2899.02 s density cap, CLI ×3), `node
scripts/check-aggregator-roster.mjs` and `--self-test`, `node
scripts/check-ci-filter-parity.mjs` and `--self-test`, `pnpm
check:shard-attestation`, `pnpm check:required-contexts`, `pnpm
check:pm-expected-skips`, `pnpm check:stall-guard-budget`, `pnpm
check:stall-guard-headroom`, `node
scripts/ci/select-shard-timings-run.mjs --self-test` (it executes the
refresh workflow's generate step, so the new roster loop ran), `node
scripts/measure-test-shard-timings.mjs --self-test`.
- `node scripts/check-commit-card-trailers.mjs --range
origin/main..HEAD`: OK.
<details><summary>All 70 derived commands and exit codes</summary>
```
node scripts/check-aggregator-roster.mjs :: exit 0
node scripts/check-aggregator-roster.mjs --self-test :: exit 0
node scripts/check-ci-filter-parity.mjs :: exit 0
node scripts/check-ci-filter-parity.mjs --self-test :: exit 0
node scripts/check-closing-keyword-parity.mjs :: exit 0
node scripts/check-closing-keyword-parity.mjs --self-test :: exit 0
node scripts/check-comment-mask-corpus.mjs :: exit 0
node scripts/check-declaration-mirrors.mjs :: exit 0
node scripts/check-declaration-mirrors.mjs --self-test :: exit 0
node scripts/check-dts-emitted.mjs --self-test :: exit 0
node scripts/check-position-name-fold-loaders.mjs :: exit 0
node scripts/check-position-name-fold-loaders.mjs --self-test :: exit 0
node scripts/check-scripts-symbol-anchors.mjs :: exit 0
node scripts/check-scripts-symbol-anchors.mjs --self-test :: exit 0
node scripts/check-self-test-wired.mjs :: exit 0
node scripts/check-self-test-wired.mjs --self-test :: exit 0
node scripts/check-self-test-workflow-commands.mjs :: exit 0
node scripts/check-self-test-workflow-commands.mjs --self-test :: exit 0
node scripts/check-step-collectors.mjs :: exit 0
node scripts/check-step-collectors.mjs --self-test :: exit 0
node scripts/check-whole-set-label-write.mjs :: exit 0
node scripts/check-whole-set-label-write.mjs --self-test :: exit 0
node scripts/ci/assert-turbo-remote-cache.mjs --self-test :: exit 0
node scripts/ci/prune-turbo-cache.mjs --self-test :: exit 0
node scripts/ci/scheduled-full-run.mjs --self-test :: exit 0
node scripts/ci/select-shard-timings-run.mjs --self-test :: exit 0
node scripts/docs-audit/check-drift-comment.mjs :: exit 0
node scripts/measure-test-shard-timings.mjs --self-test :: exit 0
node scripts/partition-test-shards.mjs --self-test :: exit 0
node scripts/pm/bare-root-worklist.mjs --self-test :: exit 0
node scripts/pm/ci-failure.mjs --self-test :: exit 0
pnpm check:agent-test-spelling :: exit 0
pnpm check:bash32-floor :: exit 0
pnpm check:cli-command-ids :: exit 0
pnpm check:console-injection :: exit 0
pnpm check:console-sha :: exit 0
pnpm check:cross-package-test-inputs :: exit 0
pnpm check:declared-population-live :: exit 0
pnpm check:driver-memory-census :: exit 0
pnpm check:dts-closure :: exit 3
pnpm check:dual-build-cjs-loads :: exit 3
pnpm check:entry-guard :: exit 0
pnpm check:gitlink-declared :: exit 0
pnpm check:lean-entry-closure :: exit 3
pnpm check:node-version :: exit 0
pnpm check:nul-bytes :: exit 0
pnpm check:parse-guard :: exit 0
pnpm check:pm-close-cards :: exit 0
pnpm check:pm-dispatch-gates :: exit 0
pnpm check:pm-expected-skips :: exit 0
pnpm check:pm-fleet-write-dispatch :: exit 0
pnpm check:pm-fleet-write-execute :: exit 0
pnpm check:pm-fleet-write-validate :: exit 0
pnpm check:pm-governed-merges :: exit 0
pnpm check:pm-issue-transfer :: exit 0
pnpm check:pm-post-stamped :: exit 0
pnpm check:pm-record-recognisers :: exit 0
pnpm check:pnpm-acquisition :: exit 0
pnpm check:pnpm-filter-targets :: exit 0
pnpm check:pnpm-flag-spellings :: exit 0
pnpm check:ratchet-remedy-authority :: exit 0
pnpm check:refd-timer-probe :: exit 0
pnpm check:required-contexts :: exit 0
pnpm check:shard-attestation :: exit 0
pnpm check:sourcemap-no-sources-content :: exit 3
pnpm check:stall-guard-budget :: exit 0
pnpm check:stall-guard-headroom :: exit 0
pnpm check:watch-hint-literal :: exit 0
pnpm check:workflow-status-functions :: exit 0
pnpm check:workflow-step-name-quoting :: exit 0
```
</details>
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 6ad226a commit 2e3e276
8 files changed
Lines changed: 430 additions & 169 deletions
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
11 | | - | |
| 10 | + | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
50 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
51 | 54 | | |
52 | 55 | | |
53 | 56 | | |
| |||
71 | 74 | | |
72 | 75 | | |
73 | 76 | | |
74 | | - | |
75 | | - | |
| 77 | + | |
| 78 | + | |
76 | 79 | | |
77 | 80 | | |
78 | 81 | | |
| |||
96 | 99 | | |
97 | 100 | | |
98 | 101 | | |
99 | | - | |
100 | | - | |
| 102 | + | |
| 103 | + | |
101 | 104 | | |
102 | 105 | | |
103 | 106 | | |
| |||
178 | 181 | | |
179 | 182 | | |
180 | 183 | | |
181 | | - | |
| 184 | + | |
182 | 185 | | |
183 | 186 | | |
184 | 187 | | |
| |||
432 | 435 | | |
433 | 436 | | |
434 | 437 | | |
435 | | - | |
436 | | - | |
| 438 | + | |
| 439 | + | |
437 | 440 | | |
438 | 441 | | |
439 | 442 | | |
440 | 443 | | |
441 | | - | |
442 | | - | |
| 444 | + | |
| 445 | + | |
443 | 446 | | |
444 | 447 | | |
445 | 448 | | |
| |||
482 | 485 | | |
483 | 486 | | |
484 | 487 | | |
485 | | - | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
486 | 495 | | |
487 | 496 | | |
488 | 497 | | |
| |||
636 | 645 | | |
637 | 646 | | |
638 | 647 | | |
639 | | - | |
| 648 | + | |
640 | 649 | | |
641 | 650 | | |
642 | 651 | | |
| |||
822 | 831 | | |
823 | 832 | | |
824 | 833 | | |
825 | | - | |
| 834 | + | |
| 835 | + | |
826 | 836 | | |
827 | 837 | | |
828 | 838 | | |
| |||
851 | 861 | | |
852 | 862 | | |
853 | 863 | | |
854 | | - | |
855 | | - | |
| 864 | + | |
| 865 | + | |
856 | 866 | | |
857 | 867 | | |
858 | 868 | | |
| |||
886 | 896 | | |
887 | 897 | | |
888 | 898 | | |
889 | | - | |
| 899 | + | |
890 | 900 | | |
891 | 901 | | |
892 | 902 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2026 | 2026 | | |
2027 | 2027 | | |
2028 | 2028 | | |
2029 | | - | |
| 2029 | + | |
2030 | 2030 | | |
2031 | 2031 | | |
2032 | 2032 | | |
2033 | 2033 | | |
2034 | 2034 | | |
2035 | | - | |
| 2035 | + | |
2036 | 2036 | | |
2037 | 2037 | | |
2038 | 2038 | | |
| |||
2041 | 2041 | | |
2042 | 2042 | | |
2043 | 2043 | | |
2044 | | - | |
| 2044 | + | |
2045 | 2045 | | |
2046 | 2046 | | |
2047 | 2047 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2085 | 2085 | | |
2086 | 2086 | | |
2087 | 2087 | | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
2088 | 2095 | | |
2089 | 2096 | | |
2090 | 2097 | | |
| |||
2109 | 2116 | | |
2110 | 2117 | | |
2111 | 2118 | | |
2112 | | - | |
| 2119 | + | |
2113 | 2120 | | |
2114 | 2121 | | |
2115 | 2122 | | |
| |||
2177 | 2184 | | |
2178 | 2185 | | |
2179 | 2186 | | |
2180 | | - | |
| 2187 | + | |
2181 | 2188 | | |
2182 | 2189 | | |
2183 | 2190 | | |
| |||
2480 | 2487 | | |
2481 | 2488 | | |
2482 | 2489 | | |
2483 | | - | |
2484 | | - | |
| 2490 | + | |
| 2491 | + | |
2485 | 2492 | | |
2486 | 2493 | | |
2487 | 2494 | | |
| |||
2573 | 2580 | | |
2574 | 2581 | | |
2575 | 2582 | | |
2576 | | - | |
| 2583 | + | |
2577 | 2584 | | |
2578 | 2585 | | |
2579 | 2586 | | |
2580 | 2587 | | |
2581 | 2588 | | |
2582 | | - | |
| 2589 | + | |
2583 | 2590 | | |
2584 | 2591 | | |
2585 | 2592 | | |
| |||
2591 | 2598 | | |
2592 | 2599 | | |
2593 | 2600 | | |
2594 | | - | |
| 2601 | + | |
2595 | 2602 | | |
2596 | 2603 | | |
2597 | 2604 | | |
2598 | | - | |
| 2605 | + | |
2599 | 2606 | | |
2600 | 2607 | | |
2601 | 2608 | | |
| |||
0 commit comments