Repository navigation
Commit f36f619
ci(test-shards): place Test Core bins by predicted wall, and cut the CLI into 4 slices on the refreshed dataset (#25968)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} -->
_Requested by **Elon** · [project
thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DGuXpcS2tMB1v58Jdn3L7NW)_
Part of #25870. This PR carries two steps: (1) wall-based bin placement,
and (2) the refreshed dataset with the CLI cut into 4 slices. #25870
stays open, because its acceptance is measured on 3 consecutive
spec-touching PRs after landing. The prediction below is not that
measurement.
**Supersedes #25960.** It carries the shard-timings-refresh lane's
commit 92343f5 (`chore(ci): refresh the Test Core shard-timings
dataset`) unchanged. That refresh cannot land alone, because pin 3c
refuses it at 3 slices (step 2).
## Diagnosis — which of the three, with run ids
The long shards are always the three that carry a `@objectstack/cli`
file-level slice. Read from `gh api …/runs/N/jobs` step timings on 56
`pull_request` runs (2026-10-25 18:54Z – 2026-10-26 05:44Z), and from
the slice leg's own `Duration` / turbo `Time:` lines on the 18
slice-shard jobs of six `packages/spec`-touching runs.
The slowest shard on a spec-touching PR breaks down like this (minutes):
| run (spec PR) | slowest | setup | closure | whole-package leg | cli
slice leg (turbo `Time:`) |
|---|---|--:|--:|--:|--:|
| 40274911131 (#25903) | 5/8 28.8 | ~1.5 | 6.0 | 5.7 | 15.4 |
| 40271593540 (#25944) | 4/8 28.1 | ~1.5 | 5.5 | 3.4 | 17.3 |
| 40269339195 (#25813) | 4/8 28.5 | ~1.5 | 5.6 | 3.4 | 17.5 |
| 40266487909 (#25903) | 5/8 28.5 | ~1.5 | 5.9 | 5.6 | 15.4 |
| 40263851511 (#25813) | 3/8 26.8 | ~1.5 | 6.3 | 4.5 | 14.3 |
| 40261969569 (#25830) | 3/8 25.7 | ~1.5 | 6.0 | 4.3 | 13.6 |
(whole-package leg = `Run this shard's tests` minus the slice leg's
turbo `Time:`.)
Three causes, all real, in this order of size:
1. **The cli slice leg itself, 13.6–17.5 min.** vitest cuts the CLI by
file COUNT (`BaseSequencer.shard`: sha1 of the path, then equal-count
chunks), so the three slices are not equal: across the six runs they
measured 518–1047 s, the heaviest ~1.27× the mean (~820 s). Total cli
2406–2542 s against a pinned 2899 s (0.83–0.88×).
2. **The dependency closure, 5.5–6.3 min**, against 0.4–1.0 min on
shards that skip declarations. `@objectstack/cli` is in ci.yml's
`DTS_READERS`, so every slice shard builds the whole cli closure WITH
`.d.ts`, and a PR run reads no remote cache, so that build is cold
(ci.yml: local cold closure 5m59s with declarations, 1m24s without).
3. **A whole-package leg of 2.2–5.7 min run before the slice — the
partitioner's doing.** `partition()` balanced bins by their SUM of
predicted seconds. A slice runs serially after the four-wide whole leg,
so the step is ~ wholeSum / 2.9 + slice (the #25376 wall model).
Balancing sums filled every slice shard up to the mean with ~900 s of
whole packages: predicted step 1277 s on the three slice shards against
643 s on the other five (committed dataset, full list).
Stale weights are not the cause: on the refreshed dataset (#25960, the
shard-timings-refresh lane's 2026-10-26 output) the sum-placed split
still predicts 1107 s on the slice shards against 518 s elsewhere — the
same 2.1× gap.
## Step 1: wall-based placement (cause 3)
In `scripts/partition-test-shards.mjs`, with no workflow or package
edit:
- `partition()` places bins by **load** — the wall model in
whole-package seconds, a slice counting `WHOLE_LEG_PARALLELISM` (2.9,
the #25376 constant, reused rather than re-derived) times its weight. A
list with no slice is placed by exactly the same additions and
comparisons as before, so every non-sliced run (and the nightly, which
slices every package on every shard) gets a byte-identical split.
- `total` stays the plain sum, so the density cap and the drift report
mean what they meant. `refineToCap()` keeps `load` in step with `total`.
- `balanceOf()` also reports the walls; pin 2 grades the wall ratio
(with a slice in the split the sums are unequal by design); new pin 2b
is a fixture where sum-placement and wall-placement differ.
- The shard log prints the predicted walls under the bins line.
Predicted test steps, full list (`turbo ls`, dogfood excluded), via
`planRun()`:
| dataset | before (slice shards / others) | after (slice shards /
others) |
|---|---|---|
| committed (2026-10-23) | 1277 s / 643 s | 966 s / 830 s |
| #25960 refresh (2026-10-26) | 1107 s / 518 s | 899 s / 642 s |
Measured-based expectation for the six spec runs above: removing the
whole-package leg takes the slowest shard from 25.7–28.8 min to
**~22.3–25.1 min**. The other five shards absorb the moved packages
(their slowest was 17.5 min; +~2.5 min of four-wide work each).
## Step 2: the refreshed dataset, and the CLI at 4 slices
The refresh (92343f5, 14 runs, newest 40272142430) measures
`@objectstack/cli` at 2697.45s. Pin 3c (`sliceCountProblems`, the
serial-floor derivation of #22075) then refuses 3 slices:
| n | one slice | vs the heaviest other serial task
(`@objectstack/metadata-protocol` 670.12s; spec and runtime run as two
tasks) | limit 1.3x = 871.16s |
|--:|--:|--:|---|
| 3 | 899.15s | 1.34x | refused: "Raise it to 4" (this is #25960's red,
job 121437007857) |
| 4 | 674.36s | 1.01x | meets, and 3 does not, so 4 is the smallest |
So the change is `FILE_SHARDED_PACKAGES` cli 3 → 4, with
`PREVIOUS_FILE_SHARDED_PACKAGES` set to the outgoing `{cli: 3}` per the
map's own rule, so the refresh lane still decodes the 3-slice summaries
retained from before the change. The docblocks carry the new derivation
table. The bin-sum floor and the wall model agree on this dataset, and
no ratio or bound moved. Predicted test-step walls on the full list are
738s on all 8 shards (max/mean 1.00x), and every bin is within the
density cap.
**Predicted from measurement, not from the model.** vitest's
`BaseSequencer.shard` (sha1 of the path relative to the package root,
sorted, equal-count chunks) reproduced locally over `vitest list
--filesOnly` (506 files) matches run 40274911131's three slice jobs file
for file: 169/169/168 files on jobs 121415631670/…679/…708. From those
jobs' per-file times, with the wall taken as 0.39 × the per-file sum
(vitest's `Duration` over its `tests` sum on the same jobs:
0.403/0.381/0.390):
| split | predicted slice walls | measured (vitest `Duration`) |
heaviest / mean |
|---|---|---|--:|
| 3 ways | 789 / 691 / 926 s | 812.65 / 673.01 / 921.52 s | 1.15 |
| 4 ways | 487 / 616 / 673 / 630 s | — | 1.12 |
The heaviest slice falls 27%. Applied to the six spec runs (slowest =
setup ~1.5 + that run's cold DTS closure + ~1.1 min of whole leg, the
wall model's 738 − 674 s, + 0.727 × that run's heaviest slice):
| run (spec PR) | slowest before | predicted after both steps |
|---|--:|--:|
| 40274911131 (#25903) | 28.8 | 19.8 |
| 40271593540 (#25944) | 28.1 | 20.7 |
| 40269339195 (#25813) | 28.5 | 20.9 |
| 40266487909 (#25903) | 28.5 | 19.7 |
| 40263851511 (#25813) | 26.8 | 19.3 |
| 40261969569 (#25830) | 25.7 | 18.5 |
That gives 18.5–20.9 min, with three of six at or under 20. The cost: a
fourth shard now builds the CLI's DTS closure (~5.8 min cold). The 738s
walls above count test time only, not that build. The per-file read of
5/8 lacks 11 files that print before the 5,000-line log tail; their
172.2s (vitest `tests` minus the captured sum) is spread evenly over
them.
## If the readings stay above ~20 min
What is left on a slice shard is the cold DTS closure (5.5–6.3 min) plus
the heaviest slice. The next steps, neither taken here:
- **B. Declarations only where a slice reads them.** Three CLI test
files read built `.d.ts`. With the split now reproducible, a slice
holding none of them could build DTS-off (~1 min instead of ~5.8). This
edits `ci.yml`, so it is Tier H.
- **C. Weight-balanced CLI slices** (a sequencer splitting by measured
file time): heaviest/mean 1.12 → ~1.0. It needs a per-file weight source
that the dataset does not carry today.
## Acceptance notes
- **Deliberate leftovers in `.github/workflows/ci.yml` (Tier H,
untouched):** the Test Core header still says each slice is "placed at
its PLAIN weight so every bin keeps the whole-package split's density",
and its "3 slices" wording (the drift-step comment, "a shard carrying
one of its 3 slices is predicted a third of that"; the timeout history)
is now stale. Correcting either makes a PR Tier H, so both are left for
whichever step edits ci.yml. The partitioner's own headers carry the
corrected statements.
- Slice shards now come out as shards 1–4 on the full list (LPT places
the heaviest loads first). Shard numbers carry no meaning beyond the
per-shard Turbo seed key.
## Tests
- `node scripts/partition-test-shards.mjs --self-test` at b323410 →
`self-test OK (75 measured packages -> 78 shard items, 8 shards,
predicted test-step walls 738/738/738/738/738/738/738/739s, max/mean
1.00x <= 1.3x, …, file-level slices: @objectstack/cli x4)`.
- Step 1 revert proof (`scripts/ablation-replace.mjs`, bin choice back
to `total`, on the committed dataset): the self-test reds with `balance:
at 8 shards the slowest predicted test step is 1.45x the mean (1277s vs
881s)`. Restored blob == HEAD, and `git diff HEAD` is empty.
- Step 2 revert proof: the refreshed dataset with cli at 3 reds pin 3c
with `@objectstack/cli: sliced 3 ways at 2697.45s, one slice is 899s --
past 1.3x the heaviest other serial task (670s, so 871s) … Raise it to
4.` (the same red as #25960's job 121437007857).
- `scripts/test-shard-timings.json` is byte-identical to 92343f5 (`git
diff 92343f5 b323410 -- scripts/test-shard-timings.json` is empty).
- Sibling self-tests are green at b323410: `measure-test-shard-timings`
(decodes through both maps), `check-test-completeness`,
`ci/select-shard-timings-run`, `nightly-tiers`,
`check-test-suite-ceilings`, `ci/select-shard-packages.selftest.sh`.
## Gates
- Local derived set at b323410 (`node scripts/pm/dispatch-gates.mjs
--commands`, merge-base origin/main cdd4eea): all 35 families exit 0.
The `--ran` reconciliation reads `35 derived famil(ies) accounted for —
35 run, 0 NOT-MEASURED`. Step 1 alone (8ca31e0) ran its 33 families the
same way.
- Everything else (Lint & Repo Gates, Test Core, …) belongs to CI and is
read from this PR's check-runs.
Clause-②: no
Metadata-protocol: none
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01NPH84ZP1jWcst3tpRwfrJL
---
_Generated by [Claude
Code](https://claude.ai/code/session_01NPH84ZP1jWcst3tpRwfrJL)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>1 parent 62ba5ef commit f36f619
2 files changed
Lines changed: 550 additions & 276 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
160 | 168 | | |
161 | 169 | | |
162 | 170 | | |
| |||
354 | 362 | | |
355 | 363 | | |
356 | 364 | | |
357 | | - | |
358 | | - | |
| 365 | + | |
| 366 | + | |
359 | 367 | | |
360 | 368 | | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
361 | 382 | | |
362 | 383 | | |
363 | 384 | | |
| |||
378 | 399 | | |
379 | 400 | | |
380 | 401 | | |
381 | | - | |
| 402 | + | |
382 | 403 | | |
383 | 404 | | |
384 | 405 | | |
| |||
406 | 427 | | |
407 | 428 | | |
408 | 429 | | |
409 | | - | |
410 | | - | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
411 | 434 | | |
412 | 435 | | |
413 | 436 | | |
| |||
882 | 905 | | |
883 | 906 | | |
884 | 907 | | |
885 | | - | |
886 | | - | |
887 | | - | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
888 | 947 | | |
889 | 948 | | |
890 | | - | |
| 949 | + | |
891 | 950 | | |
892 | | - | |
| 951 | + | |
893 | 952 | | |
894 | 953 | | |
895 | 954 | | |
896 | | - | |
| 955 | + | |
897 | 956 | | |
898 | 957 | | |
899 | 958 | | |
| 959 | + | |
900 | 960 | | |
901 | 961 | | |
902 | 962 | | |
| |||
905 | 965 | | |
906 | 966 | | |
907 | 967 | | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
908 | 974 | | |
909 | 975 | | |
910 | 976 | | |
911 | 977 | | |
912 | 978 | | |
913 | 979 | | |
914 | | - | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
915 | 996 | | |
916 | 997 | | |
917 | 998 | | |
| |||
971 | 1052 | | |
972 | 1053 | | |
973 | 1054 | | |
974 | | - | |
| 1055 | + | |
975 | 1056 | | |
976 | 1057 | | |
977 | 1058 | | |
| |||
998 | 1079 | | |
999 | 1080 | | |
1000 | 1081 | | |
| 1082 | + | |
| 1083 | + | |
1001 | 1084 | | |
1002 | 1085 | | |
| 1086 | + | |
1003 | 1087 | | |
1004 | 1088 | | |
| 1089 | + | |
1005 | 1090 | | |
1006 | 1091 | | |
1007 | 1092 | | |
| 1093 | + | |
1008 | 1094 | | |
1009 | 1095 | | |
| 1096 | + | |
1010 | 1097 | | |
1011 | 1098 | | |
1012 | 1099 | | |
| |||
1484 | 1571 | | |
1485 | 1572 | | |
1486 | 1573 | | |
1487 | | - | |
| 1574 | + | |
| 1575 | + | |
| 1576 | + | |
1488 | 1577 | | |
1489 | 1578 | | |
1490 | 1579 | | |
| |||
1731 | 1820 | | |
1732 | 1821 | | |
1733 | 1822 | | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
1734 | 1827 | | |
1735 | | - | |
| 1828 | + | |
1736 | 1829 | | |
1737 | | - | |
1738 | | - | |
1739 | | - | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
1740 | 1834 | | |
1741 | 1835 | | |
1742 | 1836 | | |
1743 | 1837 | | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
1744 | 1872 | | |
1745 | 1873 | | |
1746 | 1874 | | |
| |||
3014 | 3142 | | |
3015 | 3143 | | |
3016 | 3144 | | |
3017 | | - | |
| 3145 | + | |
| 3146 | + | |
3018 | 3147 | | |
3019 | 3148 | | |
3020 | 3149 | | |
| |||
3137 | 3266 | | |
3138 | 3267 | | |
3139 | 3268 | | |
3140 | | - | |
| 3269 | + | |
3141 | 3270 | | |
3142 | 3271 | | |
3143 | 3272 | | |
| |||
3151 | 3280 | | |
3152 | 3281 | | |
3153 | 3282 | | |
| 3283 | + | |
| 3284 | + | |
| 3285 | + | |
| 3286 | + | |
| 3287 | + | |
| 3288 | + | |
| 3289 | + | |
3154 | 3290 | | |
3155 | 3291 | | |
3156 | 3292 | | |
| |||
0 commit comments