Repository navigation
Commit c64130b
ci(test-shards): grade the Test Core split on predicted shard wall and slice the CLI per run (#22415)
Part of #22075
Clause-②: no
Test Core's slowest shard set CI and merge-queue wall time because the
split was graded on the wrong quantity. Measurement showed the card's
mechanism is half right: the shard assignment is already computed per
run from the affected set, and what was wrong is the cost model. A bin's
summed package weight is not its wall time, because a shard runs its
whole packages four at a time (`--concurrency=4`) and each file-level
slice in a leg of its own after them. This PR grades the split on
predicted shard wall, configures `@objectstack/cli` at 3 slices (the
smallest count that meets 1.3x on the committed dataset), and lets each
run slice it only when its own split needs it. The Test Core wall stays
at 45 minutes: re-sizing it is owed after landing, from the measured
post-change job walls. This PR is therefore `Part of` the card. The
card's ratio pin and that re-size are both read after merge.
## Done when (quoted verbatim from the card)
> - The partition, or the slicing refusal, is derived against
affected-set runs, by one of two routes the dev measures on at least 3
`pull_request` and 2 `merge_group` runs before choosing:
> - (a) per-run partition: the shard assignment is computed at run time
from the affected package list, using the committed dataset as weights
(the matrix stays 6 wide; the generator's slice reassembly and
`OS_TEST_SHARD` wiring already exist);
> - (b) static shards with an affected-set-aware slice: the CLI slices
(`expandSlices`, the existing mechanism) when the affected set makes it
the only heavy item, and runs whole on full runs.
> - Pin: the measured slowest-shard / mean-of-the-others ratio on
affected-set runs is ≤ 1.3, read from the jobs API and quoted in the PR
body with run ids; `--check-drift` keeps its meaning.
> - The `Test Core` shard wall is re-sized from the measured
distribution (the #16445 raise was declared temporary), with the
rationale comment carrying the window and numbers.
> - ⛔ No reduction of what the shards test (every affected package still
runs); ⛔ the seven required contexts are unchanged; ⛔ dataset-quality
work stays in #22014 / #22022 and is not duplicated here.
## What measurement found
**Route (a) is already the status quo.** Every `Test Core (N/6)` job
runs `scripts/ci/select-shard-packages.sh` (which writes `turbo ls
--affected` for `pull_request` and `merge_group`) and then
`partition-test-shards.mjs` on that file, so the split is already
computed at run time from the run's own affected list with the dataset
as weights. Only the slicing refusal (`FILE_SHARDED_PACKAGES`, empty)
was derived on full-run sums.
**The defect is the cost model, and it shows on full runs too.** I
reproduced each sampled run's affected set locally (`turbo ls
--affected` between the run's base and head, plus the cross-package
union) and split it with the partitioner. On the runs whose affected set
fills six bins, the bin sums read 1.00-1.47x of the mean, inside the
bound. The jobs API reads 2.3-3.3x for the same runs. One example,
merge_group 37875522518: bin sums were 1739/1397/1399/1403/1400/1398 s,
and the `Run this shard's tests` steps took 1880/548/478/375/556/481 s.
Shard 2/6 summed 1036.6 s of package windows, and its turbo `Time:
9m8.239s` equals its heaviest single task (`@objectstack/spec`
`test:repo`, 547.04 s). Shard 1/6 was the CLI alone, 1867.26 s.
### The measured runs
These are all completed, successful CI runs after the 21-run dataset
(0401847), from 2026-10-09T00:03Z to 02:57Z: 6 `pull_request` and 8
`merge_group`. Walls come from the jobs API; bins come from this script
on each run's reproduced affected set.
| run | event | packages | CLI | bin-sum max/mean | Test Core job walls
1..6 (min) | slowest / mean of others | run wall (min) | after: plan |
after: predicted wall max/mean |
|---|---|--:|:-:|--:|---|--:|--:|---|--:|
| 37876969409 | pull_request | 12 | yes | 2.32x | 26.3 / 14.0 / 7.2 /
6.8 / 3.1 / 3.2 | 3.84x | 27.1 | CLI x3 | 1.21x |
| 37874898502 | pull_request | 38 | yes | 1.37x | 34.3 / 13.9 / 10.0 /
11.8 / 11.4 / 9.4 | 3.03x | 37.0 | CLI x3 | 1.06x |
| 37873731877 | pull_request | 11 | yes | 2.43x | 32.6 / 11.0 / 5.6 /
7.0 / 2.5 / 2.7 | 5.65x | 33.4 | CLI x3 | 1.21x |
| 37873634681 | pull_request | 56 | yes | 1.11x | 33.8 / 18.8 / 10.6 /
9.7 / 12.6 / 11.7 | 2.67x | 36.0 | CLI x3 | 1.02x |
| 37872770181 | pull_request | 75 | yes | 1.00x | 39.1 / 20.8 / 17.9 /
18.5 / 18.6 / 10.6 | 2.26x | 42.0 | CLI x3 | 1.01x |
| 37871447533 | pull_request | 3 | no | 4.10x | 9.5 / 4.5 / 1.4 / 1.5 /
1.5 / 1.3 | 4.64x | 11.1 | no slice | 4.05x |
| 37875522518 | merge_group | 48 | yes | 1.19x | 33.3 / 11.1 / 10.0 /
8.2 / 10.8 / 10.0 | 3.32x | 34.0 | CLI x3 | 1.01x |
| 37875521531 | merge_group | 67 | yes | 1.02x | 32.5 / 12.4 / 13.8 /
13.6 / 11.9 / 11.2 | 2.58x | 33.3 | CLI x3 | 1.02x |
| 37873846077 | merge_group | 48 | yes | 1.19x | 26.8 / 11.5 / 9.1 /
12.4 / 11.7 / 8.4 | 2.52x | 27.6 | CLI x3 | 1.01x |
| 37873791941 | merge_group | 67 | yes | 1.02x | 33.9 / 14.5 / 12.6 /
14.8 / 14.0 / 9.1 | 2.61x | 34.7 | CLI x3 | 1.02x |
| 37873694430 | merge_group | 33 | yes | 1.37x | 29.9 / 11.9 / 10.6 /
10.8 / 10.2 / 9.9 | 2.81x | 30.8 | CLI x3 | 1.06x |
| 37872756554 | merge_group | 3 | no | 4.10x | 13.8 / 3.9 / 1.5 / 0.9 /
1.3 / 1.6 | 7.55x | 15.2 | no slice | 4.05x |
| 37871575925 | merge_group | 31 | yes | 1.47x | 20.5 / 10.7 / 6.5 / 6.9
/ 8.8 / 8.4 | 2.48x | 21.3 | CLI x3 | 1.08x |
| 37870616843 | merge_group | 75 | yes | 1.00x | 35.6 / 18.0 / 16.0 /
15.8 / 12.4 / 11.1 | 2.43x | 36.5 | CLI x3 | 1.01x |
- The CLI was in the affected set of **12 of the 14 runs (86%)**. The
two runs without it are a 3-package docs diff (`spec`, `rest`,
`create-objectstack`). On those, the slowest shard is
`@objectstack/spec`, which cannot be sliced, so this change leaves them
untouched.
- The "after" columns are the new split's decision and its predicted
wall ratio, computed on the same reproduced affected sets.
## Route chosen: (b), graded on shard walls
- **Wall model** (`TEST_CONCURRENCY = 4`, `shardWalls()`). A shard's
predicted wall is the whole-package leg plus every slice leg after it.
The whole-package leg is max(heaviest serial task, summed weight / 4). A
package that runs `test` and `test:repo` concurrently counts half its
weight as its serial task, because the dataset holds only their sum. At
concurrency 1 this formula equals the bin sum, so the old model is the
special case. The pre-existing slice-count fixtures pass 1 explicitly
and keep their arithmetic.
- **Placement** (`partition()`). A slice costs its bin weight × 4,
because it holds the runner for its whole window after the whole-package
leg. Whole packages still cost their weight, so a split with no slices
places exactly as before.
- **Derivation, committed dataset** (pins 2/3/3c, now graded on walls):
| model | CLI whole | at 2 | at 3 |
|---|--:|--:|--:|
| bin sums (concurrency 1) | 1.00x meets | 1.00x meets | 1.00x meets |
| shard walls (4) | 2.52x | 1.31x | **1.01x meets** |
3 is therefore the smallest count that meets the bound, and pin 3c holds
it there. `PREVIOUS_FILE_SHARDED_PACKAGES` becomes the outgoing map
`{}`.
- **Per run** (`planShards()`). A run slices a configured package only
when two things hold: its whole suite is the item past the bound
(heavier than 1.3x the larger of the run's mean shard wall and any other
package's serial task), and its slices spread over the run's shard
count. Otherwise it runs whole. The decision prints in "Compute this
shard's package set", for example `slicing: @objectstack/cli: sliced x3
(whole would be 1739s against 818s)`.
- **Why not a free per-run count:** the generator decodes a slice from
the sha256 of `OS_TEST_SHARD` against the counts the map names. A
per-run derivation also gives 3 on all 12 CLI runs above, because 4
slices never spread there.
- **"Runs whole on full runs" (route (b)'s wording) does not survive
measurement.** On walls, the full list needs the slices as much as any
PR does (2.52x whole). The nightly tier run (CLI alone, 2 shards) cannot
spread 3 slices, so it keeps running the CLI whole, as it does today.
Assumption 5 (can the matrix take a run-time assignment without renaming
the required contexts?): yes. The matrix already takes one. The job
names (`Test Core (N/6)`, and the aggregate `Test Core`) are untouched,
and `pnpm check:required-contexts` is green.
## The ≤ 1.3 pin
- **In code:** pins 2/3/3c grade the committed dataset on walls: 1.01x
at 6 shards, floor 580 s (one CLI slice) against a 663 s mean wall. A
new self-test battery has 13 cases. It reproduces the measured shape
(bin sums inside the bound, one serial suite at ~5x the other walls) and
checks that the sum model keeps it whole while the wall model slices it
and meets the bound. It also covers the per-run cases (package absent,
another unsliceable task as the floor, slices that cannot spread) and
pins the concurrency against ci.yml's `--concurrency=4`.
- **Measured on this PR's own runs: not provable, and I'm saying so.**
This PR changes `scripts/partition-test-shards.mjs` and `ci.yml`. Its
affected set is `@objectstack/spec`, `@objectstack/client` and
`@objectstack/driver-sql` (cross-package union), with no CLI, so its CI
and its queue run exercise no slice. On the 12 CLI runs above, the
predicted wall ratio after the change is 1.01-1.21x.
- **How the seat reads it after landing.** Take the first ≥ 3
`pull_request` runs and ≥ 2 `merge_group` runs whose "Compute this
shard's package set" step prints `slicing: @objectstack/cli: sliced x3`.
From `GET /repos/objectstack-ai/objectstack/actions/runs/RUN_ID/jobs`,
read two things on the same runs:
1. **The ratio pin:** the slowest `Test Core (N/6)` job wall divided by
the mean of the other five, per run.
2. **The input for the follow-up wall re-size:** the post-change Test
Core job-wall distribution, meaning the slowest Test Core job wall of
each run, together with its range across the runs.
Post the run ids, the ratios and the slowest-job walls on the card. The
one-line `timeout-minutes` re-size under this card is sized from that
distribution. The 2 docs-only runs above show that the ratio cannot be
reached when an affected set has fewer heavy items than shards (pin 3's
floor). Those runs should be read as "slowest = spec", not as a breach.
## Before / after wall, one PR run and one queue run
| run | before: run wall | before: slowest Test Core job | after
(predicted, same affected set) |
|---|--:|--:|---|
| pull_request 37872770181 (full 75-package list) | 42.0 min | 39.1 min
(CLI alone) | CLI x3; predicted walls 667/667/659/659/659/659 s;
heaviest slice ≈ 12.2 min of tests |
| merge_group 37875522518 (48 packages) | 34.0 min | 33.3 min (CLI
alone) | CLI x3; predicted walls 591/583/583/581/581/583 s |
The "after" half is a model reading. It cannot be measured on this PR's
runs (see above), and it is what the seat's post-landing read replaces.
One known bias, measured: the model is a lower bound on whole-package
legs. On 37872770181 the five whole-package shards predicted
573/442/442/442/491 s and actually stepped 905/737/704/668/388 s.
## The Test Core wall: held at 45, re-size owed after landing
`timeout-minutes` stays at **45**. Removing the whole-CLI shard removes
the reason for the old raise, but a wall is re-sized from a measured
distribution. The post-change distribution does not exist until runs
execute the new split, and a miss would kill `merge_group` runs for
every lane. The rationale comment in `ci.yml` records the inputs for the
re-size and states that none of them has been applied. Window: the 14
runs above, 37870616843 to 37876969409.
- **Input 1 (measured before the change).** The job holding the whole
CLI took 20.5-39.1 min (12 runs). Every other Test Core job took at most
20.8 min (72 jobs). The closure build took at most 5.6 min, and fixed
setup at most 2.4 min.
- **Input 2 (predicted, not measured).**
- The heaviest CLI slice should take at most ~12.2 min of tests, or ~24
min of job. vitest's hash split puts 1.14x of an even third on slice
2/3.
- The busiest whole-package shard of a full run will carry ~2640 s of
windows, a size no run has executed. At the 2.0x packing measured on
~1400 s shards, that is at most ~30 min of job.
- The wall model reads low on whole-package legs. On 37872770181 it
predicted 573/442/442/442/491 s, and the shards took 905/737/704/668/388
s. So these figures are a floor for the post-landing reading, not a
substitute for it.
- **The re-size.** Size the wall from the slowest-job distribution in
the post-landing read above, using the existing 80% margin rule. The
revert condition is restated: back to 30 only when the slowest job stays
at or under 24 min on every scheduled run for a week.
- `pnpm check:stall-guard-budget` is green at 45: the cap is 20 min
against a 45-min budget, leaving 25 min of slack.
## `--check-drift` keeps its meaning
There is no code change on that path. Both the 1.5x red and the 1.3x
warning keep their values. On affected-set runs the ratio is still one
shard's executed windows against their own prediction: a shard carrying
a CLI slice is predicted a third of the CLI's weight, from the slice
count its own summary records (the `OS_TEST_SHARD` digest), not from the
config. The drift batteries (9 + 12 cases) are unchanged and green.
Slice skew, estimated from the 367-file list the CLI runs and the 18
slowest CLI files of run 37875522518, is 0.87/1.14/0.99 of an even
third. A slice-carrying shard should therefore stay under the 1.3x
warning.
## What #16468 will read differently
- The CLI's per-run test window now arrives as **three slice windows in
three shards' turbo summaries**, each about a third of the whole,
instead of one window on one shard. A per-shard ceiling check has two
options. It can compare a slice against its share (the ceiling divided
by the slice count the summary records, as `--check-drift` does through
`predictedSecondsFor`). Or it can sum the parts across shards:
`report-test-timings.mjs` already does this and marks a partial set.
- The dataset still holds the whole cost per package, and the generator
still reassembles slices within a run before recording them.
- The shard step's timing changes on the three shards that carry a
slice: they run the whole-package leg and then the slice leg in the same
`Run this shard's tests` step.
- Nothing of #16468's ceiling check is built here.
## Required contexts and coverage
- No job, step or matrix name changed, so the seven required contexts
are untouched.
- Every affected package still runs. The CLI's three slices are vitest's
`--shard` partition of its file list, so their union is the whole suite.
`check-test-completeness` still grades every scheduled item, and its
self-test is green.
- `scripts/test-shard-timings.json` is not touched.
## Gates (head 6e8b8f6; patch round 1, no `origin/main` merge this
round)
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` re-derived the same 58 commands on
6e8b8f6. All 58 ran, with exits captured before any pipe.
- `--ran` reconciliation: **58 derived, 54 run, 4 NOT-MEASURED, 0
UNRUN.**
- The 4 NOT MEASURED are `check:dts-closure`,
`check:dual-build-cjs-loads`, `check:lean-entry-closure` and
`check:sourcemap-no-sources-content`. All exited 3 (PREREQUISITE NOT
MET) because they load every package's built `dist/`, and this diff
touches no package source or build config.
- `pnpm check:pm-dispatch-gates` exited 1 on **1 of 2011 cases**, the
same as in round 0: "no mkdtempSync site in this tree takes a base the
scan cannot read". It names
`packages/qa/dogfood/test/security-catalog-cold-boot-environment-holder.dogfood.test.ts:108`.
That file came from e030d43, which is already on `main`, and this
diff does not touch it.
- Every other command exited 0. That includes `node
scripts/partition-test-shards.mjs --self-test`, `pnpm
check:stall-guard-budget` (cap 20 min against a 45-min budget, 25 min of
slack), `pnpm check:stall-guard-headroom` and `pnpm
check:required-contexts`.
- Round 0 (head 9156fd4) also ran these consumer self-tests, all
green: `measure-test-shard-timings.mjs`, `check-test-completeness.mjs`
and `report-test-timings.mjs`. They read `partition-test-shards.mjs`,
which is byte-identical at 6e8b8f6.
- **Lint, a declared narrowing (round 0; the script is unchanged
since):** `eslint --no-inline-config --format json
scripts/partition-test-shards.mjs` reported 1 file, 0 errors, 0
warnings.
- The population comes from eslint's own config: the file is not
ignored, and `ci.yml` is not an eslint input.
- Computed `parserOptions.project` is null, so this diff cannot move any
untouched file's verdict.
- The full `pnpm lint` is CI's.
## Ablation (run at 9156fd4; `scripts/partition-test-shards.mjs` is
byte-identical at the current head; fix committed first; restored by
`git checkout HEAD`, proven blob == HEAD and `git diff HEAD` empty)
Both ablations went through `node scripts/ablation-replace.mjs` in WRAP
mode, which confirms the anchor hits 1 → 0 and records the blob change
before running the command.
1. **Wall model reverted to bin sums.** I replaced the wall formula's
return line `Math.max(serial, whole / concurrency) + sliced` with `whole
+ sliced` (blob 739b927 → 22c7fd4d7ccd). The self-test went RED:
`slice spread: the committed dataset, split as CI splits it, carries no
slice -- @objectstack/cli: whole (whole fits: 1739s is within 2303s)`.
That is the old reading, which kept the CLI whole. Restored to
739b927.
2. **Per-run decision neutered.** I replaced `const needed = own >
target;` with `const needed = false;` (blob 739b927 →
7d1f33503974). The self-test went RED: `balance: at 6 shards the slowest
predicted shard wall is 2.52x the mean (1739s vs 689s), past the 1.3x
bound`. Restored to 739b927, and the self-test is green again on
the restored tree.
## Acceptance notes
- `test-nightly-tiers.yml`'s header still says `FILE_SHARDED_PACKAGES`
"cuts the CLI into two vitest slices". That has been stale since the map
emptied, and after this PR the CLI is configured at 3, which cannot
spread on that workflow's 2 shards, so it runs whole there. Comment
only, outside this card's file surface. Noted, not filed.
- A seventh shard would lower the predicted maximum from 667 s to the
580 s floor. Past seven nothing moves. I did not do this: it changes the
matrix, and it is not this card.
- The wall model's whole-package term is a lower bound (the bias is
measured above). If post-landing readings show whole-package shards
consistently above the slice shards, the next lever is the slot weight
given to a slice, not the bound.
- The Test Core `timeout-minutes` re-size is the post-landing half of
this card. Its inputs are in the `ci.yml` comment and in the section
above.
- `skip-changeset`: root `scripts/` and workflows publish nothing.
---
_Generated by [Claude
Code](https://claude.ai/code/session_0115N1oNnQS5WqofZ2DzaT3q)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: os-sales <sales@objectstack.ai>1 parent f05649f commit c64130b
2 files changed
Lines changed: 576 additions & 184 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
412 | 412 | | |
413 | 413 | | |
414 | 414 | | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
426 | 425 | | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
435 | 432 | | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | | - | |
445 | | - | |
446 | | - | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
447 | 439 | | |
448 | | - | |
449 | | - | |
450 | | - | |
451 | | - | |
452 | | - | |
453 | | - | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
454 | 450 | | |
455 | 451 | | |
456 | 452 | | |
| |||
478 | 474 | | |
479 | 475 | | |
480 | 476 | | |
481 | | - | |
482 | | - | |
483 | | - | |
484 | | - | |
485 | | - | |
486 | | - | |
487 | | - | |
488 | | - | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
489 | 512 | | |
490 | | - | |
491 | | - | |
492 | | - | |
493 | | - | |
494 | | - | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
499 | | - | |
500 | | - | |
501 | | - | |
502 | | - | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
503 | 521 | | |
504 | 522 | | |
505 | 523 | | |
| |||
662 | 680 | | |
663 | 681 | | |
664 | 682 | | |
665 | | - | |
666 | | - | |
| 683 | + | |
| 684 | + | |
667 | 685 | | |
668 | 686 | | |
669 | 687 | | |
| |||
677 | 695 | | |
678 | 696 | | |
679 | 697 | | |
680 | | - | |
681 | | - | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
682 | 701 | | |
683 | 702 | | |
684 | 703 | | |
| |||
759 | 778 | | |
760 | 779 | | |
761 | 780 | | |
762 | | - | |
763 | | - | |
764 | | - | |
765 | | - | |
766 | | - | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
767 | 788 | | |
768 | 789 | | |
769 | 790 | | |
| |||
882 | 903 | | |
883 | 904 | | |
884 | 905 | | |
885 | | - | |
886 | | - | |
887 | | - | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
888 | 911 | | |
889 | 912 | | |
890 | 913 | | |
| |||
1056 | 1079 | | |
1057 | 1080 | | |
1058 | 1081 | | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
1062 | 1091 | | |
1063 | 1092 | | |
1064 | 1093 | | |
| |||
0 commit comments