Repository navigation
ci(test-shards): widen Test Core from 6 to 8 shards - #25746
Merged
Merged
Conversation
Sized on merge_group run 40155484566 on current main (59 packages executed, the CLI as 3 slices). Replayed through planRun() with a wall model calibrated on that run, the slowest shard drops from ~1696s at six (measured 1794s) to ~1436s at eight, -15%; -14% on the dataset's full list and -10% on a 21-package merge_group run. Predicted bins go from 2489s to 1868s, ratio 1.00x, with spec's 1607s floor binding only at ten. - SHARD_COUNT 6 -> 8 in partition-test-shards.mjs and the timings selector; ci.yml matrix, name, --shard, --total, artifact names and the aggregator's --leg move with it. - shard-timings-refresh.yml reads the shard roster from the selector's candidates instead of spelling 1..6. - Test Core timeout-minutes 50 -> 60: the stall-guard headroom tool over the last day's 512 shard jobs found 50 already short at six (worst p + s 40m27s); scaled to eight, the cap-deferred verdict lands at 56m37s. - Self-test fixtures that read the live ci.yml follow the new names; the replays of measured six-shard runs read the six-shard name. - The standing "6 is enough" comments are rewritten with the new numbers. Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb
objectstack-fleet
Bot
added
ci/cd
skip-changeset
The 6 -> 8 resize made select-shard-timings-run reject every recent schedule run (all 6-shard, all whole and green) as missing Test Core 7/8, 8/8, so the refresh exited 1 on this PR (job 121075952437) and would have kept doing so until an 8-shard schedule run existed with its artifacts retained. The dataset is per package; the width a run was split at is not an input to any package's weight. runIsEligible now reads the run's width from its own `Test Core (N/M)` job names and admits it when all M shards concluded success with all M summaries retained. A run whose Test Core jobs disagree on M, or that has none, is refused by name. The download loop already iterates each candidate's own artifact_ids, so a resize has no transition window. New self-test battery (width change, 9 cases) and an accumulation leg mixing 8-wide and 6-wide runs, both confirmed to fail against the old fixed-width reading. Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb
Contributor
Author
|
Seat review (domain:devx, #6023): ACCEPT at At Rest of the read:
Generated by Claude Code |
os-zhuang
approved these changes
Oct 25, 2026
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Elon · project thread
Fixes #25376
Clause-②: no
Metadata-protocol: none
Test Core goes from 6 to 8 shards. It was sized on a fresh merge-group timing table on current
main, as the 2026-10-23T19:55Z unlock asked, not on the card's 2026-10-13 arithmetic. Eight shards cut the slowest shard by 10–15% on every replay below, so this clears the ~10% bar. The Test Core timeout moves 50 → 60, because the stall-guard headroom tool shows 50 is already too short at six.Step 1: the timing table this is sized from
Merge-group run 40155484566 (PR #25730's queue build, head
c882092b33onmainec4ef94ead, 2026-10-25 02:42Z). 59 packages were executed and 1 replayed (@objectstack/spec); the CLI ran as its 3 slices. Read from theTest Coreaggregator's timing table and suite-ceiling list, and from each shard'sshard-timing-driftline:Run this shard's testsstepHeaviest packages on that run, measured vs pinned:
cli3036.30 s vs 2899.02 s (all 3 slices, ~1012 s each),objectql1044.70 vs 970.12,runtime936.14 vs 848.96,plugin-auth789.01 vs 405.25,rest735.83 vs 876.56,plugin-security666.28 vs 360.02. Spec's heaviest repo files after #25372:build-schemas-check-mode.git-topology432.88 s,.deleted-baseline403.48 s,.deletion-proofs377.06 s.A smaller merge-group run on the same day, 40157571602 (21 packages executed; core, spec and types replayed), had its slowest step at 1381 s on the shard holding a CLI slice.
Step 2: 6 vs 8, predicted
Bins are not wall time. On a shard, the whole packages run four at a time, and a CLI slice then runs as a serial leg of its own. So a shard's tests step reads close to
wholeSum / 2.9 + slice. On run 40155484566 this model lands within 6.1% of the measured step on the four slowest shards (the three slice shards and shard 2). It misreads the two light shards, which never set the wall. Each list below is replayed through the partitioner's ownplanRun()at each count:The partitioner's own bins on the committed dataset (the CLI as 3 slices;
ratio= max/mean):What eight cannot move: a CLI slice runs serially for about 960–1010 s. On small runs at eight, the slice shards carry almost nothing else, so that is where the −10% stops. A ninth shard would mostly add a runner. The
1.3×balance bound is unchanged and still met: spec's 1607 s floor sits against1.3 × 1866 = 2426 s. Pin 3c (CLI n = 3) is independent of the shard count and still holds.Cost: over the window below, job start → tests step is 1.6 min median (5.5 min p90), and the tail after the step is 0.2 min. Two more shards cost about +3.6 runner-min per run at the median. The card's maintainer directive accepts this.
Step 3: the timeout, re-derived
node scripts/measure-stall-guard-headroom.mjscovered the 89 completed, non-cancelled CI runs created 2026-10-24 03:00Z → 2026-10-25 03:45Z (23 merge_group, 39 pull_request, 16 push, 11 schedule; 512Test Core (N/6)jobs):So 50 is already short at six. The worst case is a red PR shard (run 40149547507). The healthy worst is 37m22s (p 7m28s + s 29m54s, run 40125226571
Test Core (4/6), job wall 37m41s). Scaled to eight by 0.90, the smallest gain above (the safe side):The largest reading wins, as in the #22075 derivation: 60 (3m23s over the deferred path). The revert condition in
ci.ymlnow reads "below 60". The first full-battery run after landing re-takes these readings on eight-shard jobs.What changed
scripts/partition-test-shards.mjs:SHARD_COUNT6 → 8. Its header now carries the 6-vs-8 arithmetic above. The "ci: rebalance the Test Core shards — shard 4/6 runs 13.6 min against 5.0–9.0 for the rest, and it is CI's critical path #10472 asked for 6 → 8 and the answer was no" paragraph is rewritten (kept, not deleted) to say why the answer changed..github/workflows/ci.yml, Test Core:1..8;name: Test Core (… /8),--shard …/8, the remote-cache and prune--sitelabels, the drift--label,--total 8, and the-of-8artifact names;--leg "test/8:…";timeout-minutes: 60;scripts/ci/select-shard-timings-run.mjs:SHARD_COUNT6 → 8, now only its matchers' default. Since the second commit, a candidate run is judged at its own width:runShardWidthreadsMfrom the run'sTest Core (N/M)job names, and the run is admitted when allMshards concluded success and allM-of-Msummaries are retained. A run whose Test Core jobs disagree onM, or that has none, is refused by name. Each candidate carriesshard_countbeside itsartifact_ids..github/workflows/shard-timings-refresh.yml: the summary download loop and the PR body's per-shard line read the shard roster from the selector'sartifact_idsinstead of spelling1 2 3 4 5 6. The selector only admits a run when all of that run's shards are present, so those keys are that run's roster, whatever its width. Wording that said "six" now names the count generically.scripts/check-shard-attestation.mjs: the fixtures that edit the liveci.yml(grow the matrix, shrink the roster, the name variants) use the 8-shard spellings. The replays of measured six-shard runs (38036705318, 34007386254) read the live workflow with the six-shard name, and that substitution is asserted. The live-name pin now checksTest Core (1..8/8).scripts/check-required-contexts.mjs: the live-specimen battery follows the/8name.scripts/measure-test-shard-timings.mjs: four-of-6spellings in comments and output text now name the width generically (-of-<width>,N-of-M), since a refresh window can mix widths.Historical run fixtures elsewhere (
scripts/pm/*, merge-queue triage) keep the(N/6)names their runs really had.No refresh transition window. The first head of this PR judged every candidate at the live count, so
Regenerate the shard-timings datasetrejected all 24 recent schedule runs (each a whole, green 6-shard run) as missing7/8,8/8, and exited 1 (job 121075952437). That would have held for every refresh until an 8-shard schedule run existed with its one-day artifacts still retained. The dataset is per package, and the width a run was split at is not an input to any package's weight, so the fix reads each run at its own width rather than reporting NOT MEASURED through the window.Tests: a new self-test battery,
width change (#25376)(9 cases: a whole 6-shard run is eligible under the 8-shard matrix; the same run pinned to 8 is rejected as job 121075952437 rejected it; an old-width run missing6-of-6or with6/6cancelled is refused; 6 jobs with-of-8summaries is refused; jobs disagreeing onMand a run with no Test Core job are refused by name; an unanchoredTest Core (3/8) rerundoes not move the width;listCandidatesreports an 8-wide and a 6-wide run each with its own width). The accumulation battery gains a leg that drives the refresh workflow's real generate step over one 8-wide and two 6-wide runs, with a package measured only in each run's last shard. Both were checked to fail against the old reading: restoring the fixed-width default fails the first case, and restoring a fixed1..8download loop fails the accumulation leg.Serial with #25697 / PR #25738: #25738 landed first (
c61f75d); this branch merges it in, with no conflict.Acceptance (from the card)
check-aggregator-roster,check:ci-filter-paritygreen locally (below) and on this PR.@objectstack/specis file-sharded — the "6 is enough" arithmetic predates the cli slices (full-battery slowest shard ≈ 1937 s → ≈ 1452 s) #25376: the first full-battery run's slowestTest Coretests step, and max shard wall / timeout (≤ 70%, ci: Test Core compares each shard's measured duration with its predicted weight — warning at 1.3x, red at 70% of the timeout (maintainer-directed, drift alarm) #16465).Local verification (head
8c254ff)Derived with
node scripts/pm/dispatch-gates.mjs --commands(70 families) and reconciled with--ran: 66 exit 0, 4 NOT MEASURED.dist/and this worktree has no full build; CI runs them):pnpm check:dts-closure,pnpm check:dual-build-cjs-loads,pnpm check:lean-entry-closure,pnpm check:sourcemap-no-sources-content.node scripts/partition-test-shards.mjs --self-test(75 measured packages → 77 items, 8 shards, max/mean 1.00x, bins 1865.66–1867.52 s within the 2899.02 s density cap, CLI ×3),node scripts/check-aggregator-roster.mjsand--self-test,node scripts/check-ci-filter-parity.mjsand--self-test,pnpm check:shard-attestation,pnpm check:required-contexts,pnpm check:pm-expected-skips,pnpm check:stall-guard-budget,pnpm check:stall-guard-headroom,node scripts/ci/select-shard-timings-run.mjs --self-test(it executes the refresh workflow's generate step, so the new roster loop ran),node scripts/measure-test-shard-timings.mjs --self-test.node scripts/check-commit-card-trailers.mjs --range origin/main..HEAD: OK.All 70 derived commands and exit codes
🤖 Generated with Claude Code
https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb