Skip to content

Commit c64130b

Browse files
objectstack-fleet[bot]claudeos-sales
authored
ci(test-shards): grade the Test Core split on predicted shard wall and slice the CLI per run (#22415)
Part of #22075 Clause-②: no Test Core's slowest shard set CI and merge-queue wall time because the split was graded on the wrong quantity. Measurement showed the card's mechanism is half right: the shard assignment is already computed per run from the affected set, and what was wrong is the cost model. A bin's summed package weight is not its wall time, because a shard runs its whole packages four at a time (`--concurrency=4`) and each file-level slice in a leg of its own after them. This PR grades the split on predicted shard wall, configures `@objectstack/cli` at 3 slices (the smallest count that meets 1.3x on the committed dataset), and lets each run slice it only when its own split needs it. The Test Core wall stays at 45 minutes: re-sizing it is owed after landing, from the measured post-change job walls. This PR is therefore `Part of` the card. The card's ratio pin and that re-size are both read after merge. ## Done when (quoted verbatim from the card) > - The partition, or the slicing refusal, is derived against affected-set runs, by one of two routes the dev measures on at least 3 `pull_request` and 2 `merge_group` runs before choosing: > - (a) per-run partition: the shard assignment is computed at run time from the affected package list, using the committed dataset as weights (the matrix stays 6 wide; the generator's slice reassembly and `OS_TEST_SHARD` wiring already exist); > - (b) static shards with an affected-set-aware slice: the CLI slices (`expandSlices`, the existing mechanism) when the affected set makes it the only heavy item, and runs whole on full runs. > - Pin: the measured slowest-shard / mean-of-the-others ratio on affected-set runs is ≤ 1.3, read from the jobs API and quoted in the PR body with run ids; `--check-drift` keeps its meaning. > - The `Test Core` shard wall is re-sized from the measured distribution (the #16445 raise was declared temporary), with the rationale comment carrying the window and numbers. > - ⛔ No reduction of what the shards test (every affected package still runs); ⛔ the seven required contexts are unchanged; ⛔ dataset-quality work stays in #22014 / #22022 and is not duplicated here. ## What measurement found **Route (a) is already the status quo.** Every `Test Core (N/6)` job runs `scripts/ci/select-shard-packages.sh` (which writes `turbo ls --affected` for `pull_request` and `merge_group`) and then `partition-test-shards.mjs` on that file, so the split is already computed at run time from the run's own affected list with the dataset as weights. Only the slicing refusal (`FILE_SHARDED_PACKAGES`, empty) was derived on full-run sums. **The defect is the cost model, and it shows on full runs too.** I reproduced each sampled run's affected set locally (`turbo ls --affected` between the run's base and head, plus the cross-package union) and split it with the partitioner. On the runs whose affected set fills six bins, the bin sums read 1.00-1.47x of the mean, inside the bound. The jobs API reads 2.3-3.3x for the same runs. One example, merge_group 37875522518: bin sums were 1739/1397/1399/1403/1400/1398 s, and the `Run this shard's tests` steps took 1880/548/478/375/556/481 s. Shard 2/6 summed 1036.6 s of package windows, and its turbo `Time: 9m8.239s` equals its heaviest single task (`@objectstack/spec` `test:repo`, 547.04 s). Shard 1/6 was the CLI alone, 1867.26 s. ### The measured runs These are all completed, successful CI runs after the 21-run dataset (0401847), from 2026-10-09T00:03Z to 02:57Z: 6 `pull_request` and 8 `merge_group`. Walls come from the jobs API; bins come from this script on each run's reproduced affected set. | run | event | packages | CLI | bin-sum max/mean | Test Core job walls 1..6 (min) | slowest / mean of others | run wall (min) | after: plan | after: predicted wall max/mean | |---|---|--:|:-:|--:|---|--:|--:|---|--:| | 37876969409 | pull_request | 12 | yes | 2.32x | 26.3 / 14.0 / 7.2 / 6.8 / 3.1 / 3.2 | 3.84x | 27.1 | CLI x3 | 1.21x | | 37874898502 | pull_request | 38 | yes | 1.37x | 34.3 / 13.9 / 10.0 / 11.8 / 11.4 / 9.4 | 3.03x | 37.0 | CLI x3 | 1.06x | | 37873731877 | pull_request | 11 | yes | 2.43x | 32.6 / 11.0 / 5.6 / 7.0 / 2.5 / 2.7 | 5.65x | 33.4 | CLI x3 | 1.21x | | 37873634681 | pull_request | 56 | yes | 1.11x | 33.8 / 18.8 / 10.6 / 9.7 / 12.6 / 11.7 | 2.67x | 36.0 | CLI x3 | 1.02x | | 37872770181 | pull_request | 75 | yes | 1.00x | 39.1 / 20.8 / 17.9 / 18.5 / 18.6 / 10.6 | 2.26x | 42.0 | CLI x3 | 1.01x | | 37871447533 | pull_request | 3 | no | 4.10x | 9.5 / 4.5 / 1.4 / 1.5 / 1.5 / 1.3 | 4.64x | 11.1 | no slice | 4.05x | | 37875522518 | merge_group | 48 | yes | 1.19x | 33.3 / 11.1 / 10.0 / 8.2 / 10.8 / 10.0 | 3.32x | 34.0 | CLI x3 | 1.01x | | 37875521531 | merge_group | 67 | yes | 1.02x | 32.5 / 12.4 / 13.8 / 13.6 / 11.9 / 11.2 | 2.58x | 33.3 | CLI x3 | 1.02x | | 37873846077 | merge_group | 48 | yes | 1.19x | 26.8 / 11.5 / 9.1 / 12.4 / 11.7 / 8.4 | 2.52x | 27.6 | CLI x3 | 1.01x | | 37873791941 | merge_group | 67 | yes | 1.02x | 33.9 / 14.5 / 12.6 / 14.8 / 14.0 / 9.1 | 2.61x | 34.7 | CLI x3 | 1.02x | | 37873694430 | merge_group | 33 | yes | 1.37x | 29.9 / 11.9 / 10.6 / 10.8 / 10.2 / 9.9 | 2.81x | 30.8 | CLI x3 | 1.06x | | 37872756554 | merge_group | 3 | no | 4.10x | 13.8 / 3.9 / 1.5 / 0.9 / 1.3 / 1.6 | 7.55x | 15.2 | no slice | 4.05x | | 37871575925 | merge_group | 31 | yes | 1.47x | 20.5 / 10.7 / 6.5 / 6.9 / 8.8 / 8.4 | 2.48x | 21.3 | CLI x3 | 1.08x | | 37870616843 | merge_group | 75 | yes | 1.00x | 35.6 / 18.0 / 16.0 / 15.8 / 12.4 / 11.1 | 2.43x | 36.5 | CLI x3 | 1.01x | - The CLI was in the affected set of **12 of the 14 runs (86%)**. The two runs without it are a 3-package docs diff (`spec`, `rest`, `create-objectstack`). On those, the slowest shard is `@objectstack/spec`, which cannot be sliced, so this change leaves them untouched. - The "after" columns are the new split's decision and its predicted wall ratio, computed on the same reproduced affected sets. ## Route chosen: (b), graded on shard walls - **Wall model** (`TEST_CONCURRENCY = 4`, `shardWalls()`). A shard's predicted wall is the whole-package leg plus every slice leg after it. The whole-package leg is max(heaviest serial task, summed weight / 4). A package that runs `test` and `test:repo` concurrently counts half its weight as its serial task, because the dataset holds only their sum. At concurrency 1 this formula equals the bin sum, so the old model is the special case. The pre-existing slice-count fixtures pass 1 explicitly and keep their arithmetic. - **Placement** (`partition()`). A slice costs its bin weight × 4, because it holds the runner for its whole window after the whole-package leg. Whole packages still cost their weight, so a split with no slices places exactly as before. - **Derivation, committed dataset** (pins 2/3/3c, now graded on walls): | model | CLI whole | at 2 | at 3 | |---|--:|--:|--:| | bin sums (concurrency 1) | 1.00x meets | 1.00x meets | 1.00x meets | | shard walls (4) | 2.52x | 1.31x | **1.01x meets** | 3 is therefore the smallest count that meets the bound, and pin 3c holds it there. `PREVIOUS_FILE_SHARDED_PACKAGES` becomes the outgoing map `{}`. - **Per run** (`planShards()`). A run slices a configured package only when two things hold: its whole suite is the item past the bound (heavier than 1.3x the larger of the run's mean shard wall and any other package's serial task), and its slices spread over the run's shard count. Otherwise it runs whole. The decision prints in "Compute this shard's package set", for example `slicing: @objectstack/cli: sliced x3 (whole would be 1739s against 818s)`. - **Why not a free per-run count:** the generator decodes a slice from the sha256 of `OS_TEST_SHARD` against the counts the map names. A per-run derivation also gives 3 on all 12 CLI runs above, because 4 slices never spread there. - **"Runs whole on full runs" (route (b)'s wording) does not survive measurement.** On walls, the full list needs the slices as much as any PR does (2.52x whole). The nightly tier run (CLI alone, 2 shards) cannot spread 3 slices, so it keeps running the CLI whole, as it does today. Assumption 5 (can the matrix take a run-time assignment without renaming the required contexts?): yes. The matrix already takes one. The job names (`Test Core (N/6)`, and the aggregate `Test Core`) are untouched, and `pnpm check:required-contexts` is green. ## The ≤ 1.3 pin - **In code:** pins 2/3/3c grade the committed dataset on walls: 1.01x at 6 shards, floor 580 s (one CLI slice) against a 663 s mean wall. A new self-test battery has 13 cases. It reproduces the measured shape (bin sums inside the bound, one serial suite at ~5x the other walls) and checks that the sum model keeps it whole while the wall model slices it and meets the bound. It also covers the per-run cases (package absent, another unsliceable task as the floor, slices that cannot spread) and pins the concurrency against ci.yml's `--concurrency=4`. - **Measured on this PR's own runs: not provable, and I'm saying so.** This PR changes `scripts/partition-test-shards.mjs` and `ci.yml`. Its affected set is `@objectstack/spec`, `@objectstack/client` and `@objectstack/driver-sql` (cross-package union), with no CLI, so its CI and its queue run exercise no slice. On the 12 CLI runs above, the predicted wall ratio after the change is 1.01-1.21x. - **How the seat reads it after landing.** Take the first ≥ 3 `pull_request` runs and ≥ 2 `merge_group` runs whose "Compute this shard's package set" step prints `slicing: @objectstack/cli: sliced x3`. From `GET /repos/objectstack-ai/objectstack/actions/runs/RUN_ID/jobs`, read two things on the same runs: 1. **The ratio pin:** the slowest `Test Core (N/6)` job wall divided by the mean of the other five, per run. 2. **The input for the follow-up wall re-size:** the post-change Test Core job-wall distribution, meaning the slowest Test Core job wall of each run, together with its range across the runs. Post the run ids, the ratios and the slowest-job walls on the card. The one-line `timeout-minutes` re-size under this card is sized from that distribution. The 2 docs-only runs above show that the ratio cannot be reached when an affected set has fewer heavy items than shards (pin 3's floor). Those runs should be read as "slowest = spec", not as a breach. ## Before / after wall, one PR run and one queue run | run | before: run wall | before: slowest Test Core job | after (predicted, same affected set) | |---|--:|--:|---| | pull_request 37872770181 (full 75-package list) | 42.0 min | 39.1 min (CLI alone) | CLI x3; predicted walls 667/667/659/659/659/659 s; heaviest slice ≈ 12.2 min of tests | | merge_group 37875522518 (48 packages) | 34.0 min | 33.3 min (CLI alone) | CLI x3; predicted walls 591/583/583/581/581/583 s | The "after" half is a model reading. It cannot be measured on this PR's runs (see above), and it is what the seat's post-landing read replaces. One known bias, measured: the model is a lower bound on whole-package legs. On 37872770181 the five whole-package shards predicted 573/442/442/442/491 s and actually stepped 905/737/704/668/388 s. ## The Test Core wall: held at 45, re-size owed after landing `timeout-minutes` stays at **45**. Removing the whole-CLI shard removes the reason for the old raise, but a wall is re-sized from a measured distribution. The post-change distribution does not exist until runs execute the new split, and a miss would kill `merge_group` runs for every lane. The rationale comment in `ci.yml` records the inputs for the re-size and states that none of them has been applied. Window: the 14 runs above, 37870616843 to 37876969409. - **Input 1 (measured before the change).** The job holding the whole CLI took 20.5-39.1 min (12 runs). Every other Test Core job took at most 20.8 min (72 jobs). The closure build took at most 5.6 min, and fixed setup at most 2.4 min. - **Input 2 (predicted, not measured).** - The heaviest CLI slice should take at most ~12.2 min of tests, or ~24 min of job. vitest's hash split puts 1.14x of an even third on slice 2/3. - The busiest whole-package shard of a full run will carry ~2640 s of windows, a size no run has executed. At the 2.0x packing measured on ~1400 s shards, that is at most ~30 min of job. - The wall model reads low on whole-package legs. On 37872770181 it predicted 573/442/442/442/491 s, and the shards took 905/737/704/668/388 s. So these figures are a floor for the post-landing reading, not a substitute for it. - **The re-size.** Size the wall from the slowest-job distribution in the post-landing read above, using the existing 80% margin rule. The revert condition is restated: back to 30 only when the slowest job stays at or under 24 min on every scheduled run for a week. - `pnpm check:stall-guard-budget` is green at 45: the cap is 20 min against a 45-min budget, leaving 25 min of slack. ## `--check-drift` keeps its meaning There is no code change on that path. Both the 1.5x red and the 1.3x warning keep their values. On affected-set runs the ratio is still one shard's executed windows against their own prediction: a shard carrying a CLI slice is predicted a third of the CLI's weight, from the slice count its own summary records (the `OS_TEST_SHARD` digest), not from the config. The drift batteries (9 + 12 cases) are unchanged and green. Slice skew, estimated from the 367-file list the CLI runs and the 18 slowest CLI files of run 37875522518, is 0.87/1.14/0.99 of an even third. A slice-carrying shard should therefore stay under the 1.3x warning. ## What #16468 will read differently - The CLI's per-run test window now arrives as **three slice windows in three shards' turbo summaries**, each about a third of the whole, instead of one window on one shard. A per-shard ceiling check has two options. It can compare a slice against its share (the ceiling divided by the slice count the summary records, as `--check-drift` does through `predictedSecondsFor`). Or it can sum the parts across shards: `report-test-timings.mjs` already does this and marks a partial set. - The dataset still holds the whole cost per package, and the generator still reassembles slices within a run before recording them. - The shard step's timing changes on the three shards that carry a slice: they run the whole-package leg and then the slice leg in the same `Run this shard's tests` step. - Nothing of #16468's ceiling check is built here. ## Required contexts and coverage - No job, step or matrix name changed, so the seven required contexts are untouched. - Every affected package still runs. The CLI's three slices are vitest's `--shard` partition of its file list, so their union is the whole suite. `check-test-completeness` still grades every scheduled item, and its self-test is green. - `scripts/test-shard-timings.json` is not touched. ## Gates (head 6e8b8f6; patch round 1, no `origin/main` merge this round) - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` re-derived the same 58 commands on 6e8b8f6. All 58 ran, with exits captured before any pipe. - `--ran` reconciliation: **58 derived, 54 run, 4 NOT-MEASURED, 0 UNRUN.** - The 4 NOT MEASURED are `check:dts-closure`, `check:dual-build-cjs-loads`, `check:lean-entry-closure` and `check:sourcemap-no-sources-content`. All exited 3 (PREREQUISITE NOT MET) because they load every package's built `dist/`, and this diff touches no package source or build config. - `pnpm check:pm-dispatch-gates` exited 1 on **1 of 2011 cases**, the same as in round 0: "no mkdtempSync site in this tree takes a base the scan cannot read". It names `packages/qa/dogfood/test/security-catalog-cold-boot-environment-holder.dogfood.test.ts:108`. That file came from e030d43, which is already on `main`, and this diff does not touch it. - Every other command exited 0. That includes `node scripts/partition-test-shards.mjs --self-test`, `pnpm check:stall-guard-budget` (cap 20 min against a 45-min budget, 25 min of slack), `pnpm check:stall-guard-headroom` and `pnpm check:required-contexts`. - Round 0 (head 9156fd4) also ran these consumer self-tests, all green: `measure-test-shard-timings.mjs`, `check-test-completeness.mjs` and `report-test-timings.mjs`. They read `partition-test-shards.mjs`, which is byte-identical at 6e8b8f6. - **Lint, a declared narrowing (round 0; the script is unchanged since):** `eslint --no-inline-config --format json scripts/partition-test-shards.mjs` reported 1 file, 0 errors, 0 warnings. - The population comes from eslint's own config: the file is not ignored, and `ci.yml` is not an eslint input. - Computed `parserOptions.project` is null, so this diff cannot move any untouched file's verdict. - The full `pnpm lint` is CI's. ## Ablation (run at 9156fd4; `scripts/partition-test-shards.mjs` is byte-identical at the current head; fix committed first; restored by `git checkout HEAD`, proven blob == HEAD and `git diff HEAD` empty) Both ablations went through `node scripts/ablation-replace.mjs` in WRAP mode, which confirms the anchor hits 1 → 0 and records the blob change before running the command. 1. **Wall model reverted to bin sums.** I replaced the wall formula's return line `Math.max(serial, whole / concurrency) + sliced` with `whole + sliced` (blob 739b927 → 22c7fd4d7ccd). The self-test went RED: `slice spread: the committed dataset, split as CI splits it, carries no slice -- @objectstack/cli: whole (whole fits: 1739s is within 2303s)`. That is the old reading, which kept the CLI whole. Restored to 739b927. 2. **Per-run decision neutered.** I replaced `const needed = own > target;` with `const needed = false;` (blob 739b927 → 7d1f33503974). The self-test went RED: `balance: at 6 shards the slowest predicted shard wall is 2.52x the mean (1739s vs 689s), past the 1.3x bound`. Restored to 739b927, and the self-test is green again on the restored tree. ## Acceptance notes - `test-nightly-tiers.yml`'s header still says `FILE_SHARDED_PACKAGES` "cuts the CLI into two vitest slices". That has been stale since the map emptied, and after this PR the CLI is configured at 3, which cannot spread on that workflow's 2 shards, so it runs whole there. Comment only, outside this card's file surface. Noted, not filed. - A seventh shard would lower the predicted maximum from 667 s to the 580 s floor. Past seven nothing moves. I did not do this: it changes the matrix, and it is not this card. - The wall model's whole-package term is a lower bound (the bias is measured above). If post-landing readings show whole-package shards consistently above the slice shards, the next lever is the slot weight given to a slice, not the bound. - The Test Core `timeout-minutes` re-size is the post-landing half of this card. Its inputs are in the `ci.yml` comment and in the section above. - `skip-changeset`: root `scripts/` and workflows publish nothing. --- _Generated by [Claude Code](https://claude.ai/code/session_0115N1oNnQS5WqofZ2DzaT3q)_ --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: os-sales <sales@objectstack.ai>
1 parent f05649f commit c64130b

2 files changed

Lines changed: 576 additions & 184 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 101 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -412,45 +412,41 @@ jobs:
412412
# shard, because every shard gets its own runner.
413413
#
414414
# SIX specifically, and not four or five: sharding is BY PACKAGE, so no
415-
# shard can finish faster than its single heaviest package. On the
416-
# current dataset (scripts/test-shard-timings.json as #21826 refreshed it,
417-
# provenance run 37262126122) that package is `@objectstack/cli` at
418-
# 1702.69s, ahead of `@objectstack/spec` at 1134.86s. At five shards or
419-
# fewer the partitioner must co-schedule it with others (heaviest bin
420-
# 1957s at five); at six it fills bin 1 alone and the other five sit at
421-
# 1615-1617s. Six is therefore the smallest count that isolates the
422-
# heaviest indivisible suite; past six, the CLI's shard cannot improve,
423-
# only the others can. (spec held that place until #21826 measured the
424-
# CLI whole: the dataset before it recorded the CLI at 733.33s, summed
425-
# from two file-level slices, and #21487 had already retired the slicing.)
415+
# shard can finish faster than the heaviest serial task it carries. On the
416+
# current dataset (scripts/test-shard-timings.json, the 21-run refresh at
417+
# 040184752c) that task is `@objectstack/cli` at 1738.88s, which no shard
418+
# count fixes by itself: whole, it is 2.52x the mean predicted shard wall at
419+
# six. So it is cut into 3 file-level slices whenever a run's own split
420+
# needs it (#22075; FILE_SHARDED_PACKAGES and planShards() in the script),
421+
# and the split is graded on predicted shard WALL, not on summed weights --
422+
# a shard runs its whole packages four at a time (`--concurrency=4` below)
423+
# and each slice after them, so a bin's sum is not its wall. Re-derived with
424+
# the partitioner's own planShards(), partition() and shardWalls():
426425
#
427-
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins
428-
# 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now
429-
# made in measured seconds, because #10472 replaced the weight input: the
430-
# file count was a proxy that ran ~2.8x off on @objectstack/cli, and the
431-
# six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/
432-
# 9.0 min in run 32428961038. The conclusion (six) survived the re-
433-
# measurement; only the units it is argued in changed. Weights now come
434-
# from scripts/test-shard-timings.json — see partition-test-shards.mjs.
426+
# 4 shards -> max 1071s mean 1011s ratio 1.06x
427+
# 5 shards -> max 910s mean 816s ratio 1.12x
428+
# 6 shards -> max 667s mean 663s ratio 1.01x
429+
# 7 shards -> max 580s mean 568s ratio 1.02x
430+
# 8 shards -> max 580s mean 517s ratio 1.12x
431+
# 10 shards -> max 580s mean 455s ratio 1.27x
435432
#
436-
# AND WHY NOT MORE THAN SIX (#10472 asked for 6 -> 8 to be considered).
437-
# The floor is the heaviest single package, and it does not move when
438-
# shards are added, while the mean falls with every shard — so max/mean,
439-
# which is what the acceptance bound is written in, gets WORSE past the
440-
# point where the floor becomes the max. Re-derived on the current dataset
441-
# with the partitioner's own partition() and balanceOf():
442-
#
443-
# 6 shards -> max 1703s mean 1630s ratio 1.04x
444-
# 7 shards -> max 1703s mean 1397s ratio 1.22x
445-
# 8 shards -> max 1703s mean 1223s ratio 1.39x (past the 1.3x bound)
446-
# 10 shards -> max 1703s mean 978s ratio 1.74x
433+
# ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS, then in summed
434+
# seconds. #10472 replaced the count (a proxy ~2.8x off on the CLI: six
435+
# count-balanced bins ran 5.0/6.2/6.3/13.6/6.3/9.0 min in run 32428961038);
436+
# #22075 replaced the sum, which read 1.00-1.47x on the runs where shard
437+
# 1/6 -- the CLI alone -- ran 2.3-7.6x the other five. Each time the
438+
# conclusion (six) survived and only the units it is argued in changed.
447439
#
448-
# Past six the critical path does not move at all — the CLI's 1703s is
449-
# the maximum at every count — so more shards buy only more runners'
450-
# fixed overhead and a worse ratio. The partitioner's self-test pins the
451-
# half of this that can fail: at SHARD_COUNT the heaviest package must
452-
# sit within 1.3x the mean, and when it stops fitting, slicing it below
453-
# package granularity is the remedy that failure names.
440+
# AND WHY NOT MORE THAN SIX (#10472 asked for 6 -> 8 to be considered).
441+
# The floor is the heaviest serial task -- one CLI slice, 580s, beside
442+
# `@objectstack/spec`'s 573s -- and it does not move when shards are added,
443+
# while the mean falls with every shard. A seventh shard would take the
444+
# predicted maximum from 667s to that floor; past seven the maximum does not
445+
# move at all, so more shards buy only more runners' fixed overhead and a
446+
# worse ratio. The partitioner's self-test pins the half of this that can
447+
# fail: at SHARD_COUNT the heaviest serial task must sit within 1.3x the mean
448+
# shard wall, and when it stops fitting, slicing it further is the remedy
449+
# that failure names.
454450
#
455451
# ⚠ THE COST, stated because it is real: per-shard fixed overhead
456452
# (checkout + pnpm/Turbo cache restore + install, ~60s measured on run
@@ -478,28 +474,50 @@ jobs:
478474
# margin for a cold Turbo cache; the old 45 left a hung job "running" for
479475
# half an hour past any plausible healthy finish.
480476
#
481-
# ⚠ RAISED 30 -> 45 (#16173), and the raise is still load-bearing. It was
482-
# made because shard 5/6 was being killed at the 30-minute wall (12+
483-
# observations at 30:16-30:21 with every other job green; the green band
484-
# topped out at 28:56, a ~1 minute margin), and so the tail could be
485-
# measured uncensored before anyone re-derived
486-
# `scripts/test-shard-timings.json`. Both are done: the dataset was
487-
# refreshed (#20388, then #21826) and the split now runs the whole CLI
488-
# alone on shard 1/6 (#21487).
477+
# ⚠ RAISED 30 -> 45 (#16173, the #16445 "temporary" raise): shard 5/6 was
478+
# being killed at the 30-minute wall, and after the dataset refreshes
479+
# (#20388, #21826) the split ran the whole CLI alone on shard 1/6, whose job
480+
# wall then reached 35m43s (run 37453598388).
481+
#
482+
# HELD AT 45 BY #22075; THE RE-SIZE IS OWED AFTER IT LANDS, carried by
483+
# #22075. That change stops running the whole CLI on one shard (it is cut
484+
# into 3 slices whenever a run carries it), which removes the reason for
485+
# this raise -- but a wall is re-sized from a MEASURED distribution, and
486+
# the post-change one does not exist until runs execute the new split.
487+
# What the re-size will start from, and nothing here has been applied:
488+
#
489+
# INPUT 1, measured before the change. Window: the 14 runs after the 21-run
490+
# dataset refresh (040184752c), 6 pull_request and 8 merge_group,
491+
# 37870616843 to 37876969409 (2026-10-09T00:03Z-02:57Z), from the jobs API:
492+
#
493+
# Test Core job holding the whole CLI 20.5-39.1 min (12 runs)
494+
# every other Test Core job <= 20.8 min (72 jobs)
495+
# `Build this shard's dependency closure` <= 5.6 min; fixed setup <= 2.4 min
496+
#
497+
# INPUT 2, PREDICTED, not measured. The CLI's test step read 18.7-32.2 min
498+
# whole, so its heaviest slice (vitest's hash split puts 1.14x of an even
499+
# third on slice 2/3) should read at most ~12.2 min, plus its own
500+
# `cli#build`, the closure and setup: <= ~24 min of job. The busiest
501+
# whole-package shard of a FULL run will carry ~2640s of windows, a size no
502+
# run has executed; at the 2.0x packing measured on ~1400s shards that is
503+
# <= ~22 min of tests, <= ~30 min of job. The wall model reads LOW on
504+
# whole-package legs (run 37872770181: predicted 573/442/442/442/491s,
505+
# stepped 905/737/704/668/388s), so these two figures are a floor for the
506+
# reading, never a substitute for it.
507+
#
508+
# THE RE-SIZE: once at least 3 pull_request and 2 merge_group runs print
509+
# `slicing: @objectstack/cli: sliced x3` in "Compute this shard's package
510+
# set", take the slowest Test Core JOB wall of each from the jobs API and
511+
# size this value from that distribution, with the margin rule below.
489512
#
490-
# ⛔ REVERT CONDITION, restated against measured wall time, not a card:
491-
# back to `30` only when the slowest shard's JOB wall time stays at or
492-
# under 24 minutes (80% of 30, so the margin the old wall lacked is
493-
# there) on every scheduled run for a week. It does not today. On the 16
494-
# main runs after the #21826 refresh (37413386379 to 37467882762), shard
495-
# 1/6 — the CLI alone — read 6m14s-35m43s of job wall time, over 30
496-
# minutes on 9 of them (34m39s run 37415122516, 35m43s run 37453598388,
497-
# 34m08s run 37460325624), so the condition as first written ("once
498-
# #16173 lands") would now kill that shard; and its slowest reading is
499-
# already 79% of this 45. Read the `Test Core (1/6)` job's started and
500-
# completed times before touching this value. It carries no other expiry
501-
# — a raise with no revert condition beside it becomes permanent by
502-
# forgetting — and the stall guard stays the primary hang detector.
513+
# ⛔ REVERT CONDITION, against measured wall time, not a card: back to `30`
514+
# only when the slowest Test Core JOB wall stays at or under 24 minutes (80%
515+
# of 30) on every scheduled run for a week. A miss here kills merge_group
516+
# runs for every lane, so this value moves on a reading, never on a
517+
# prediction. `partition-test-shards.mjs` prints each shard's predicted
518+
# wall in the "Compute this shard's package set" step, so the prediction
519+
# sits beside the measurement it is checked against. The stall guard (a
520+
# 20-minute cap) stays the primary hang detector.
503521
timeout-minutes: 45
504522
permissions:
505523
contents: read
@@ -662,8 +680,8 @@ jobs:
662680
# ⛔ THIS SHARD'S `^build` CLOSURE IS BUILT HERE, ON THE TURBO REMOTE
663681
# CACHE (#22077), so `Run this shard's tests` REPLAYS it from the local
664682
# cache instead of building it. Before this step the closure was built
665-
# inside the test step's own turbo run (the slice step below runs zero
666-
# iterations since #21487). Merge-queue run 37623284168, `Test Core
683+
# inside the test step's own turbo run (the slice step below ran zero
684+
# iterations from #21487 to #22075). Merge-queue run 37623284168, `Test Core
667685
# (3/6)`: `Tasks: 72 successful, 72 total` / `Cached: 1 cached, 72
668686
# total` / `Time: 11m43.373s` for the tests AND their closure in one
669687
# run. That run cannot read the remote, because `test` / `test:repo`
@@ -677,8 +695,9 @@ jobs:
677695
# Build Core's hash. A `test` task that `dependsOn: ["build"]` (cli and
678696
# metadata in turbo.json) also needs its OWN package's build. This filter
679697
# leaves that out on purpose, so a shard never builds a package its tests
680-
# do not need; the test step builds it as before. Today that is one task,
681-
# `cli#build` on shard 1/6.
698+
# do not need; the test step builds it as before. Today that is
699+
# `cli#build`, on each shard carrying a CLI slice: the slice step below
700+
# builds it there.
682701
#
683702
# Its own guarded step, for the reason the slice step below gives: a
684703
# guarded SITE is (file, job, step). No `--summarize`: `.turbo/runs/`
@@ -759,11 +778,13 @@ jobs:
759778
# and `pnpm check:stall-guard-headroom` both read this step, so it keeps
760779
# its own `--stall-minutes` and its own headroom row.
761780
#
762-
# A shard with no slice runs zero iterations here, and since #21487
763-
# emptied FILE_SHARDED_PACKAGES that is every shard: no package is sliced
764-
# today, so the paragraphs above describe the mechanism a slice would
765-
# use, not anything this job currently runs. Every shard still reaches
766-
# the step, so its name is a stable site for those two gates.
781+
# A shard with no slice runs zero iterations here. Since #22075 a run
782+
# that carries `@objectstack/cli` cuts it into 3 slices on 3 shards
783+
# whenever its own split needs them (partition-test-shards.mjs
784+
# planShards(), which prints its decision in "Compute this shard's
785+
# package set"), so those three shards run one iteration each. Every
786+
# shard still reaches the step, so its name is a stable site for those
787+
# two gates.
767788
- name: Build the sliced package's dependency closure
768789
env:
769790
NODE_OPTIONS: --report-on-signal --report-signal=SIGUSR2 --report-directory=${{ runner.temp }}/stall-reports
@@ -882,9 +903,11 @@ jobs:
882903
# an unmoved `PKG#test` across a change outside the package; the
883904
# completeness guard below reads both tasks' summaries. A slice leg
884905
# stays `test`-only: the CLI, the one package whose slice wiring is
885-
# in the tree, is not split. No package is sliced today
886-
# (FILE_SHARDED_PACKAGES is empty since #21487), so every shard runs
887-
# the whole-package leg alone and the slice branch below is idle.
906+
# in the tree, is not split. Since #22075 the CLI is cut into 3
907+
# slices whenever a run's own split needs it, so the three shards
908+
# carrying one run the whole-package leg and then the slice leg, in
909+
# that order -- which is why the partitioner weighs a slice as
910+
# holding its shard for its entire window.
888911
STATUS=0
889912
LOGS=""
890913
for LEG in __whole__ $SLICES; do
@@ -1056,9 +1079,15 @@ jobs:
10561079
# with no packages runs no turbo and writes no summary; that is NOT
10571080
# MEASURED here rather than a usage error from the script.
10581081
#
1059-
# The dataset it reads is scripts/test-shard-timings.json as #21826
1060-
# refreshed it (provenance run 37262126122): the whole CLI at 1702.69s,
1061-
# alone on shard 1/6 since FILE_SHARDED_PACKAGES went empty (#21487).
1082+
# The dataset it reads is scripts/test-shard-timings.json as the 21-run
1083+
# refresh left it (040184752c): the whole CLI at 1738.88s. Since #22075 a
1084+
# shard carrying one of its 3 slices is predicted a third of that, from
1085+
# the slice count the shard's own summary records (the OS_TEST_SHARD
1086+
# digest) rather than from the config, so the ratio stays one shard's
1087+
# executed windows against their own prediction on every run, whatever
1088+
# the affected set -- the meaning the 1.5x red and the 1.3x warning were
1089+
# written against. Measured before the slicing, on merge_group
1090+
# 37875522518: shard 1/6 (the whole CLI) 1.07x, shard 2/6 0.74x.
10621091
# The second sample taken before wiring this, executed windows only, from
10631092
# the `Test Core` timing tables of the main runs after that refresh:
10641093
# shard 1/6 read 0.61x-1.05x on all eight scheduled runs (37416453417 to

0 commit comments

Comments
 (0)