Skip to content

Commit 60c8f70

Browse files
ci(test-shards): refresh the Test Core dataset and cut the CLI into 4 slices on plain-sum placement, and pin the whole-package leg (#26186)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9DLXndrnCtL5skV6kUMFJ24w)_ Part of #25870. The card stays open until its after-landing reading: the first 10 full merge-queue Test Core builds with no load-shaped red. Clause-②: no Metadata-protocol: none ## What This is the second attempt at #25870, after #25968 was reverted by #26077. It keeps only the parts of #25968 that did not cause the merge-queue reds. Two files change. - **`scripts/test-shard-timings.json`**: the 2026-10-26 refresh (14 runs, newest 40272142430). The file is byte-identical to the one #25968 carried (`git diff f36f619 HEAD -- scripts/test-shard-timings.json` is empty). - **`scripts/partition-test-shards.mjs`**: - `FILE_SHARDED_PACKAGES` cli 3 → 4, and `PREVIOUS_FILE_SHARDED_PACKAGES` = `{cli: 3}`, so the refresh lane can still decode the 3-slice summaries it retains. The derivation table for 4 sits next to the map. - **Placement does not change.** `partition()` stays the plain-sum LPT. There is no wall model and no `loadOf`/`wallOf`. - **New pin 3d** in `--self-test` (`WHOLE_LEG_CAP_SECONDS`, `LOAD_SENSITIVE_APART`, `wholeLegProblems()`), described below. No `.github/**` edit, so this is not Tier H, and no governed path is touched. ## Why - **cli ×4 has to be on `main` before the 2026-11-02 05:30Z weekly refresh.** With the 10-26 weights, pin 3c refuses cli ×3: one slice is 899.15 s, which is past 1.3 × `@objectstack/metadata-protocol`'s 670.12 s = 871.16 s ("Raise it to 4"). If this does not land first, the refresh bot's PR goes red. - **#25968 failed because of its placement.** The weights and the slice count were not the problem. The wall model moved the whole packages off the slice shards, which left four shards with ~2141 s of four-wide work each, and put objectql and plugin-security on the same shard. Full queue builds then went red 5 times in 27, every time on a load-sensitive test (#26074, and the post-mortem on #25870, comment 6414712442). Keeping plain-sum placement with the refreshed weights makes every whole-package shard *lighter* than it is on main today. ## Shard table: `planRun()` on the full list These numbers come from `turbo ls --output=json` at 365fe5e with `--exclude @objectstack/dogfood`, run through `planRun()` exactly as ci.yml calls it. "whole" is the shard's whole-package seconds with the package count in brackets. "slice" is the CLI slice's seconds. "step" is the #25376 wall model (`whole / 2.9 + slice`), included for reference only. | shard | main (plain sum, 10-23 dataset, cli ×3) | this PR (plain sum, 10-26 dataset, cli ×4) | |---|---|---| | 1/8 | 1866 (6) · step 643 | 1499 (16) · step 517 | | 2/8 | 1866 (11) **objectql** · 643 | 1501 (10) · 518 | | 3/8 | 900 (8) + cli 1/3 966 · 1276 | 826 (9) + cli 1/4 674 · 959 | | 4/8 | 900 (9) + cli 2/3 966 · 1276 | 827 (9) + cli 2/4 674 · 959 | | 5/8 | 899 (8) + cli 3/3 966 **plugin-security** · 1276 | 826 (9) + cli 3/4 674 · 959 | | 6/8 | 1866 (10) · 643 | 826 (9) + cli 4/4 674 **plugin-security** · 959 | | 7/8 | 1868 (11) · 644 | 1501 (10) **objectql** · 518 | | 8/8 | 1866 (19) · 643 | 1500 (10) · 517 | The dispatch set three conditions, and all three hold: - Every whole-package shard is at or below today's ~1866 s. The heaviest is 1501 s, against 1868 s on main. - **objectql (7/8) and plugin-security (6/8) are on different shards.** plugin-security sits on a slice shard with 826 s of whole-package work, as it did before #25968 (899 s). - No shard's whole sum is above the pre-#25968 mean of 1866 s. Bin totals are 1499–1501 s (max/mean 1.00×), inside the 2697.45 s density cap. #25968's layout for comparison: 186/185/185/185 + 674 s slices, and 2141/2139/2141/2142 s whole. ## Pin 3d: the whole-package leg cap and the load-sensitive pair Pins 2–3c grade bin *sums* and slice counts, and #25968's 2141 s layout passed all of them. Pin 3d grades what actually broke, on the committed dataset's full list as CI splits it: - **`WHOLE_LEG_CAP_SECONDS = 1868`**: no shard's whole-package sum (slices excluded) may exceed 1868 s. That figure is the heaviest whole leg the queue has run green: shard 7/8 of the plain-sum layout on the 2026-10-23 dataset, red 0 times in 19 full builds before #25968 and 0 in 10 after #26077. 2141 s is the figure that went red. The pin's comment says that raising the cap needs a merge-queue red-rate reading, not a balance figure, and that the remedy for a refresh that outgrows the cap is a wider matrix or a finer slice. - **`LOAD_SENSITIVE_APART = [objectql, plugin-security]`**: these two may never be whole on the same shard. They are the two packages whose tests went red under load in #26074. The third red was a CLI slice, and a slice runs alone after its shard's whole leg. If either package disappears from the split, the pin goes red instead of passing on a pair it no longer grades. The battery has 5 cases: the committed-dataset grade, which first proves the split's whole-leg sums add up to the dataset's whole seconds, and 4 fixtures (pass, pair together, over cap, member missing). The battery roster floor goes from 15 to 16. ## Tests - `node scripts/partition-test-shards.mjs --self-test` at HEAD: `self-test OK (75 measured packages -> 78 shard items, 8 shards, max/mean 1.00x <= 1.3x, floor 1169s, bins 1499.31/1501.21/1499.98/1500.86/1499.98/1500.12/1501.22/1500.45s within the 2697.45s density cap, whole legs 1499/1501/826/827/826/826/1501/1500s within 1868s, file-level slices: @objectstack/cli x4)`. - **Ablation: the pin catches #25968's placement.** I put #25968's load-based `partition()` (slice × 2.9) back in temporarily and turned off pin 2, which on main already reds it at 1.43×. Pin 3d then reds with `shard 5/8 carries 2141s … 6/8 2139s … 7/8 2141s … 8/8 2142s of whole packages, past the 1868s cap` and `@objectstack/objectql and @objectstack/plugin-security share shard 7/8`. That is #25968's layout reproduced. After the ablation the file was restored and `cmp` confirms it matches. - **Ablation: cli ×3 on the refreshed dataset reds pin 3c** with `sliced 3 ways at 2697.45s, one slice is 899s … Raise it to 4.` Restored afterwards, `cmp` clean. - Sibling self-tests are green: `measure-test-shard-timings` (decodes through both maps), `check-test-completeness`, `ci/select-shard-timings-run`, `ci/select-shard-packages.selftest.sh` (32 cases). ## Gates - `node scripts/pm/dispatch-gates.mjs --commands` (merge base 365fe5e): all 35 derived families exit 0. `dispatch-gates.mjs --ran` reconciles: 35 derived, 35 run, 0 NOT-MEASURED. - Serial check before pushing: none of the 36 other open PRs on this repo touches `scripts/partition-test-shards.mjs` or `scripts/test-shard-timings.json`. I checked each with `git diff --name-only origin/main...pull/N/head`. - Everything else is left to this PR's CI. ## Acceptance notes - **After landing (the reading #25870 waits on):** read the first 10 full Test Core `merge_group` builds the way #26074's 13:35Z reading did. Any load-shaped red means stop and report, and a revert is on the table. Also: the slowest spec-touching `pull_request` shard on 3 readings, for information. The original goal was ≤ ~20 min. The predicted slice-shard step falls from 1276 s to 959 s, but the cold DTS closure (~5.8 min) still comes on top of that. - **Left alone because it is Tier H:** ci.yml's Test Core comments still say "3 slices" (the drift-step comment and the timeout history). Correcting them would make this PR Tier H. They are for whichever step edits ci.yml next. - The pin grades the dataset's full list, which is what a full queue build runs. A `pull_request` or partial `merge_group` run splits its own smaller list, which can put the two packages together on lighter shards. That was already true on the green layout, so this PR does not change it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CLvWYcHLpbpu8X8PvozD7F Co-authored-by: Claude <noreply@anthropic.com>
1 parent ea0327a commit 60c8f70

2 files changed

Lines changed: 530 additions & 261 deletions

File tree

‎scripts/partition-test-shards.mjs‎

Lines changed: 136 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -354,10 +354,25 @@ export function timingModeNote(event) {
354354
// n = 2 869.44s 1.52x still the floor
355355
// n = 3 579.63s 1.01x meets -- spec's task is the floor
356356
//
357-
// So n = 3, and pin 3c holds it there. Slices are placed at their PLAIN weight
357+
// So n = 3, and pin 3c held it there. Slices are placed at their PLAIN weight
358358
// (partition() is unchanged), so every bin keeps the density the whole-package
359359
// split had -- see densityCap() for the hard limit on that and its pin.
360360
//
361+
// RE-DERIVED AT 4 (#25870), on the 14-run refresh of 2026-10-26 (newest run
362+
// 40272142430): the CLI at 2697.45s. `@objectstack/spec` (1168.98s) and
363+
// `@objectstack/runtime` (793.70s) each run as two tasks, so the heaviest
364+
// other serial task is `@objectstack/metadata-protocol`'s 670.12s, and a slice
365+
// must sit within 1.3 x 670.12 = 871.16s.
366+
//
367+
// n = 3 899.15s 1.34x still the floor -- pin 3c refused 3
368+
// n = 4 674.36s 1.01x meets -- metadata-protocol is the floor
369+
//
370+
// Placement stays the plain-sum LPT above. #25968 placed slices by a wall model
371+
// instead, emptied the slice shards of whole packages, piled ~2141s of four-wide
372+
// work onto each of four shards (with objectql and plugin-security sharing one)
373+
// and turned load-sensitive tests red in the merge queue until #26077 reverted
374+
// it. Pin 3d (WHOLE_LEG_CAP_SECONDS, LOAD_SENSITIVE_APART) holds that line.
375+
//
361376
// ⛔ THE COUNT IS CONFIGURED; WHETHER A RUN SLICES IS DECIDED BY THAT RUN. A
362377
// pull_request or merge_group run splits its own affected set, so planShards()
363378
// asks, of the run's own list, whether a configured package is that run's
@@ -378,7 +393,7 @@ export function timingModeNote(event) {
378393
// a change to what that generator can DECODE, which is why the map it replaced
379394
// is kept below.
380395
export const FILE_SHARDED_PACKAGES = Object.freeze({
381-
'@objectstack/cli': 3,
396+
'@objectstack/cli': 4,
382397
});
383398

384399
// THE MAP AS IT STOOD BEFORE ITS LAST CHANGE, read only by the generator's
@@ -406,8 +421,10 @@ export const FILE_SHARDED_PACKAGES = Object.freeze({
406421
// `test-core-run-summary-*`, #25771), so four days after that PR lands no
407422
// retained summary predates the change and this map decodes nothing.
408423
// The outgoing map of #22075's third round was empty -- the runs before it
409-
// carried no slice -- so this one is empty too.
410-
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({});
424+
// carried no slice. #25870 raised the CLI from 3 slices to 4, so this holds 3.
425+
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({
426+
'@objectstack/cli': 3,
427+
});
411428

412429
// The item grammar. A shard item is a package (`@objectstack/cli`) or a SLICE
413430
// of one (`@objectstack/cli 1/2`), and this pair of functions is the only place
@@ -901,6 +918,71 @@ export function partition(items, shardCount) {
901918
return bins;
902919
}
903920

921+
// ── THE WHOLE-PACKAGE LEG IS CAPPED, AND TWO SUITES NEVER SHARE IT (#25870) ──
922+
//
923+
// A shard runs its whole packages four at a time, so a heavy whole leg is not
924+
// only a long step: every test in it runs beside three other suites, and the
925+
// tests with a per-test wall budget feel that. #25968 placed bins by a wall
926+
// model, moved the whole packages off the slice shards, and left four shards
927+
// at ~2141s of whole-package seconds each, with `@objectstack/objectql` and
928+
// `@objectstack/plugin-security` on one of them. Full merge_group Test Core
929+
// builds then went red 5 times in 27 (0 in 19 before), every red a
930+
// load-sensitive test: objectql hook-run-as on cold import, plugin-security
931+
// shipped-sets past its 5s budget, and the cli hook-timeout CONTROL. #26077
932+
// reverted it; the 10 full queue builds after the revert were green. Pins 2-3c
933+
// grade bin SUMS and slice counts, and 2141s passed every one of them.
934+
//
935+
// So two facts are held on the committed dataset's full list, as CI splits it:
936+
//
937+
// WHOLE_LEG_CAP_SECONDS no shard's whole-package sum (its slices excluded)
938+
// exceeds 1868s -- the heaviest whole leg the queue
939+
// has run green: shard 7/8 of the plain-sum layout on
940+
// the 2026-10-23 dataset (8 shards at a 1866s mean),
941+
// 0 red in 19 full builds before #25968 and 10 after
942+
// #26077. 2141s is what went red. ⛔ Raising this
943+
// number needs a merge-queue red-rate reading at the
944+
// new figure, not a balance figure: the shard wall
945+
// model cannot see co-run contention.
946+
// LOAD_SENSITIVE_APART these packages are never whole on the same shard.
947+
// They are the two whose tests went red under load in
948+
// #26074; the third red was a CLI slice, which runs
949+
// alone after its shard's whole leg.
950+
//
951+
// When a refresh grows the whole legs past the cap, the remedy is a wider
952+
// matrix or a finer slice, never a denser shard.
953+
export const WHOLE_LEG_CAP_SECONDS = 1868;
954+
export const LOAD_SENSITIVE_APART = Object.freeze(['@objectstack/objectql', '@objectstack/plugin-security']);
955+
956+
// `items` carry the weights the bins were filled from (planShards()' items).
957+
// A member of `apart` that is in no bin is a problem too: a pin on a pair the
958+
// split no longer carries would hold by testing nothing.
959+
export function wholeLegProblems(bins, items, { cap = WHOLE_LEG_CAP_SECONDS, apart = LOAD_SENSITIVE_APART } = {}) {
960+
const weight = new Map(items.map((i) => [i.name, i.weight]));
961+
const wholes = bins.map((b) => b.names.filter((n) => !parseShardItem(n).slice));
962+
const sums = wholes.map((names) => names.reduce((sum, n) => sum + (weight.get(n) ?? 0), 0));
963+
const problems = [];
964+
sums.forEach((sum, i) => {
965+
if (sum > cap) {
966+
problems.push(
967+
`shard ${i + 1}/${bins.length} carries ${sum.toFixed(0)}s of whole packages, past the ${cap}s cap ` +
968+
`(${wholes[i].length} packages run four at a time beside each other)`
969+
);
970+
}
971+
});
972+
const shardOf = new Map();
973+
for (const name of apart) {
974+
const at = wholes.findIndex((names) => names.includes(name));
975+
if (at === -1) {
976+
problems.push(`${name} is not a whole package in this split, so the pair it belongs to is not graded -- re-derive LOAD_SENSITIVE_APART`);
977+
continue;
978+
}
979+
const other = shardOf.get(at);
980+
if (other) problems.push(`${other} and ${name} share shard ${at + 1}/${bins.length}`);
981+
else shardOf.set(at, name);
982+
}
983+
return { sums, problems };
984+
}
985+
904986
// The one summary statistic #10472's acceptance criterion is written in.
905987
// `floor` is the heaviest single package: sharding is BY PACKAGE, so no split
906988
// at any shard count can put max below it, and comparing it to the mean says
@@ -1485,6 +1567,8 @@ const SELF_TEST_BATTERIES = Object.freeze({
14851567
// every one of them runs unconditionally: 3c's fixtures and 3b's cut of the
14861568
// heaviest package do not depend on what the live map slices.
14871569
'the balancing pins (#10472)': 25,
1570+
// 1 grades the committed dataset's split; 4 are fixtures, both directions.
1571+
'the whole-package leg cap and the load-sensitive pair (#25870)': 5,
14881572
'predicted-vs-measured drift (#16173)': 9,
14891573
'drift warning tier under the red (#16465)': 12,
14901574
'file-level slice items (#16173)': 20,
@@ -1501,7 +1585,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
15011585

15021586
// DELETING an entry silences that battery's floor exactly as effectively as
15031587
// zeroing it, so the roster's own size is pinned too.
1504-
const SELF_TEST_BATTERY_FLOOR = 15;
1588+
const SELF_TEST_BATTERY_FLOOR = 16;
15051589

15061590
// The key an assertion is filed under when no battery is open. It is not a
15071591
// declared battery, so it reds by the same set difference rather than silently
@@ -1981,6 +2065,52 @@ function selfTest() {
19812065
});
19822066
}
19832067

2068+
// 3d. THE WHOLE-PACKAGE LEG IS CAPPED AND THE LOAD-SENSITIVE PAIR IS APART
2069+
// (#25870) -- see WHOLE_LEG_CAP_SECONDS. The first case grades the
2070+
// committed dataset as CI splits it, and first proves the split carries
2071+
// whole-package seconds to grade; the fixtures hold each refusal.
2072+
battery('the whole-package leg cap and the load-sensitive pair (#25870)');
2073+
const wholeLeg = wholeLegProblems(real, datasetItems);
2074+
check(() => {
2075+
const wholeSeconds = datasetItems.filter((i) => !i.slice).reduce((sum, i) => sum + i.weight, 0);
2076+
if (wholeSeconds === 0 || Math.abs(wholeLeg.sums.reduce((a, b) => a + b, 0) - wholeSeconds) > 1e-6) {
2077+
throw new Error(`whole leg: the split's whole-package sums do not add up to the dataset's ${wholeSeconds}s`);
2078+
}
2079+
if (wholeLeg.problems.length > 0) {
2080+
throw new Error(
2081+
`whole leg, committed dataset at ${SHARD_COUNT} shards (sums ` +
2082+
wholeLeg.sums.map((t) => t.toFixed(0)).join('/') + 's):\n' +
2083+
wholeLeg.problems.map((p) => ` - ${p}`).join('\n')
2084+
);
2085+
}
2086+
});
2087+
// Two shards; `p` and `q` are the pair, 60s each. Split apart under a 100s cap
2088+
// it holds; a slice is not whole-leg weight, so 50s of slice beside 60s holds.
2089+
const legItems = [mk('p', 60), mk('q', 60), mk('r', 30), { ...mk('big 1/2', 50), slice: { index: 1, count: 2 } }];
2090+
const leg = (names, opts) => wholeLegProblems(names.map((n) => ({ names: n, total: 0 })), legItems, { cap: 100, apart: ['p', 'q'], ...opts });
2091+
check(() => {
2092+
const r = leg([['p', 'r', 'big 1/2'], ['q']]);
2093+
if (r.problems.length > 0) throw new Error(`whole leg: a split within the cap with the pair apart was refused: ${r.problems.join(' | ')}`);
2094+
});
2095+
check(() => {
2096+
const r = leg([['p', 'q'], ['r']]);
2097+
if (!r.problems.some((m) => m.includes('p and q share shard 1/2'))) {
2098+
throw new Error(`whole leg: the load-sensitive pair on one shard was accepted (${r.problems.join(' | ') || 'no problem named'})`);
2099+
}
2100+
});
2101+
check(() => {
2102+
const r = leg([['p', 'r', 'big 1/2'], ['q']], { cap: 80 });
2103+
if (!r.problems.some((m) => m.includes('shard 1/2 carries 90s of whole packages, past the 80s cap'))) {
2104+
throw new Error(`whole leg: a shard past the cap was accepted (${r.problems.join(' | ') || 'no problem named'})`);
2105+
}
2106+
});
2107+
check(() => {
2108+
const r = leg([['p', 'r'], ['big 1/2']], { apart: ['p', 'gone'] });
2109+
if (!r.problems.some((m) => m.includes('gone is not a whole package'))) {
2110+
throw new Error('whole leg: a pair member missing from the split was graded as apart');
2111+
}
2112+
});
2113+
19842114
// -- PREDICTED VS MEASURED (#16173) -------------------------------------
19852115
//
19862116
// The balancing pins above all read the dataset as GIVEN. None of them can
@@ -3016,6 +3146,7 @@ function selfTest() {
30163146
`-> ${datasetItems.length} shard items, ${SHARD_COUNT} shards, ` +
30173147
`max/mean ${balance.ratio.toFixed(2)}x <= ${MAX_SHARD_OVER_MEAN}x, floor ${balance.floor.toFixed(0)}s, ` +
30183148
`bins ${balance.totals.map((t) => t.toFixed(2)).join('/')}s within the ${datasetCap.toFixed(2)}s density cap, ` +
3149+
`whole legs ${wholeLeg.sums.map((t) => t.toFixed(0)).join('/')}s within ${WHOLE_LEG_CAP_SECONDS}s, ` +
30193150
`file-level slices: ` +
30203151
`${datasetPlan.decisions.filter((d) => d.count > 1).map((d) => `${d.name} x${d.count}`).join(', ') || 'none'})`
30213152
);

0 commit comments

Comments
 (0)