Skip to content

Commit 6e20eda

Browse files
committed
ci(test-shards): refresh the Test Core dataset and cut the CLI into 4 slices, keep plain-sum placement, and pin the whole-package leg
Retry of #25968 without its wall-model placement (reverted by #26077): the 2026-10-26 dataset refresh, `@objectstack/cli` at 4 slices with PREVIOUS_FILE_SHARDED_PACKAGES {cli: 3}, and pin 3d -- no shard's whole-package sum past 1868s, and objectql and plugin-security never whole on one shard. Co-authored-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CLvWYcHLpbpu8X8PvozD7F
1 parent 365fe5e commit 6e20eda

2 files changed

Lines changed: 530 additions & 261 deletions

File tree

‎scripts/partition-test-shards.mjs‎

Lines changed: 136 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -354,10 +354,25 @@ export function timingModeNote(event) {
354354
// n = 2 869.44s 1.52x still the floor
355355
// n = 3 579.63s 1.01x meets -- spec's task is the floor
356356
//
357-
// So n = 3, and pin 3c holds it there. Slices are placed at their PLAIN weight
357+
// So n = 3, and pin 3c held it there. Slices are placed at their PLAIN weight
358358
// (partition() is unchanged), so every bin keeps the density the whole-package
359359
// split had -- see densityCap() for the hard limit on that and its pin.
360360
//
361+
// RE-DERIVED AT 4 (#25870), on the 14-run refresh of 2026-10-26 (newest run
362+
// 40272142430): the CLI at 2697.45s. `@objectstack/spec` (1168.98s) and
363+
// `@objectstack/runtime` (793.70s) each run as two tasks, so the heaviest
364+
// other serial task is `@objectstack/metadata-protocol`'s 670.12s, and a slice
365+
// must sit within 1.3 x 670.12 = 871.16s.
366+
//
367+
// n = 3 899.15s 1.34x still the floor -- pin 3c refused 3
368+
// n = 4 674.36s 1.01x meets -- metadata-protocol is the floor
369+
//
370+
// Placement stays the plain-sum LPT above. #25968 placed slices by a wall model
371+
// instead, emptied the slice shards of whole packages, piled ~2141s of four-wide
372+
// work onto each of four shards (with objectql and plugin-security sharing one)
373+
// and turned load-sensitive tests red in the merge queue until #26077 reverted
374+
// it. Pin 3d (WHOLE_LEG_CAP_SECONDS, LOAD_SENSITIVE_APART) holds that line.
375+
//
361376
// ⛔ THE COUNT IS CONFIGURED; WHETHER A RUN SLICES IS DECIDED BY THAT RUN. A
362377
// pull_request or merge_group run splits its own affected set, so planShards()
363378
// asks, of the run's own list, whether a configured package is that run's
@@ -378,7 +393,7 @@ export function timingModeNote(event) {
378393
// a change to what that generator can DECODE, which is why the map it replaced
379394
// is kept below.
380395
export const FILE_SHARDED_PACKAGES = Object.freeze({
381-
'@objectstack/cli': 3,
396+
'@objectstack/cli': 4,
382397
});
383398

384399
// THE MAP AS IT STOOD BEFORE ITS LAST CHANGE, read only by the generator's
@@ -406,8 +421,10 @@ export const FILE_SHARDED_PACKAGES = Object.freeze({
406421
// `test-core-run-summary-*`, #25771), so four days after that PR lands no
407422
// retained summary predates the change and this map decodes nothing.
408423
// The outgoing map of #22075's third round was empty -- the runs before it
409-
// carried no slice -- so this one is empty too.
410-
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({});
424+
// carried no slice. #25870 raised the CLI from 3 slices to 4, so this holds 3.
425+
export const PREVIOUS_FILE_SHARDED_PACKAGES = Object.freeze({
426+
'@objectstack/cli': 3,
427+
});
411428

412429
// The item grammar. A shard item is a package (`@objectstack/cli`) or a SLICE
413430
// of one (`@objectstack/cli 1/2`), and this pair of functions is the only place
@@ -901,6 +918,71 @@ export function partition(items, shardCount) {
901918
return bins;
902919
}
903920

921+
// ── THE WHOLE-PACKAGE LEG IS CAPPED, AND TWO SUITES NEVER SHARE IT (#25870) ──
922+
//
923+
// A shard runs its whole packages four at a time, so a heavy whole leg is not
924+
// only a long step: every test in it runs beside three other suites, and the
925+
// tests with a per-test wall budget feel that. #25968 placed bins by a wall
926+
// model, moved the whole packages off the slice shards, and left four shards
927+
// at ~2141s of whole-package seconds each, with `@objectstack/objectql` and
928+
// `@objectstack/plugin-security` on one of them. Full merge_group Test Core
929+
// builds then went red 5 times in 27 (0 in 19 before), every red a
930+
// load-sensitive test: objectql hook-run-as on cold import, plugin-security
931+
// shipped-sets past its 5s budget, and the cli hook-timeout CONTROL. #26077
932+
// reverted it; the 10 full queue builds after the revert were green. Pins 2-3c
933+
// grade bin SUMS and slice counts, and 2141s passed every one of them.
934+
//
935+
// So two facts are held on the committed dataset's full list, as CI splits it:
936+
//
937+
// WHOLE_LEG_CAP_SECONDS no shard's whole-package sum (its slices excluded)
938+
// exceeds 1868s -- the heaviest whole leg the queue
939+
// has run green: shard 7/8 of the plain-sum layout on
940+
// the 2026-10-23 dataset (8 shards at a 1866s mean),
941+
// 0 red in 19 full builds before #25968 and 10 after
942+
// #26077. 2141s is what went red. ⛔ Raising this
943+
// number needs a merge-queue red-rate reading at the
944+
// new figure, not a balance figure: the shard wall
945+
// model cannot see co-run contention.
946+
// LOAD_SENSITIVE_APART these packages are never whole on the same shard.
947+
// They are the two whose tests went red under load in
948+
// #26074; the third red was a CLI slice, which runs
949+
// alone after its shard's whole leg.
950+
//
951+
// When a refresh grows the whole legs past the cap, the remedy is a wider
952+
// matrix or a finer slice, never a denser shard.
953+
export const WHOLE_LEG_CAP_SECONDS = 1868;
954+
export const LOAD_SENSITIVE_APART = Object.freeze(['@objectstack/objectql', '@objectstack/plugin-security']);
955+
956+
// `items` carry the weights the bins were filled from (planShards()' items).
957+
// A member of `apart` that is in no bin is a problem too: a pin on a pair the
958+
// split no longer carries would hold by testing nothing.
959+
export function wholeLegProblems(bins, items, { cap = WHOLE_LEG_CAP_SECONDS, apart = LOAD_SENSITIVE_APART } = {}) {
960+
const weight = new Map(items.map((i) => [i.name, i.weight]));
961+
const wholes = bins.map((b) => b.names.filter((n) => !parseShardItem(n).slice));
962+
const sums = wholes.map((names) => names.reduce((sum, n) => sum + (weight.get(n) ?? 0), 0));
963+
const problems = [];
964+
sums.forEach((sum, i) => {
965+
if (sum > cap) {
966+
problems.push(
967+
`shard ${i + 1}/${bins.length} carries ${sum.toFixed(0)}s of whole packages, past the ${cap}s cap ` +
968+
`(${wholes[i].length} packages run four at a time beside each other)`
969+
);
970+
}
971+
});
972+
const shardOf = new Map();
973+
for (const name of apart) {
974+
const at = wholes.findIndex((names) => names.includes(name));
975+
if (at === -1) {
976+
problems.push(`${name} is not a whole package in this split, so the pair it belongs to is not graded -- re-derive LOAD_SENSITIVE_APART`);
977+
continue;
978+
}
979+
const other = shardOf.get(at);
980+
if (other) problems.push(`${other} and ${name} share shard ${at + 1}/${bins.length}`);
981+
else shardOf.set(at, name);
982+
}
983+
return { sums, problems };
984+
}
985+
904986
// The one summary statistic #10472's acceptance criterion is written in.
905987
// `floor` is the heaviest single package: sharding is BY PACKAGE, so no split
906988
// at any shard count can put max below it, and comparing it to the mean says
@@ -1485,6 +1567,8 @@ const SELF_TEST_BATTERIES = Object.freeze({
14851567
// every one of them runs unconditionally: 3c's fixtures and 3b's cut of the
14861568
// heaviest package do not depend on what the live map slices.
14871569
'the balancing pins (#10472)': 25,
1570+
// 1 grades the committed dataset's split; 4 are fixtures, both directions.
1571+
'the whole-package leg cap and the load-sensitive pair (#25870)': 5,
14881572
'predicted-vs-measured drift (#16173)': 9,
14891573
'drift warning tier under the red (#16465)': 12,
14901574
'file-level slice items (#16173)': 20,
@@ -1501,7 +1585,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
15011585

15021586
// DELETING an entry silences that battery's floor exactly as effectively as
15031587
// zeroing it, so the roster's own size is pinned too.
1504-
const SELF_TEST_BATTERY_FLOOR = 15;
1588+
const SELF_TEST_BATTERY_FLOOR = 16;
15051589

15061590
// The key an assertion is filed under when no battery is open. It is not a
15071591
// declared battery, so it reds by the same set difference rather than silently
@@ -1981,6 +2065,52 @@ function selfTest() {
19812065
});
19822066
}
19832067

2068+
// 3d. THE WHOLE-PACKAGE LEG IS CAPPED AND THE LOAD-SENSITIVE PAIR IS APART
2069+
// (#25870) -- see WHOLE_LEG_CAP_SECONDS. The first case grades the
2070+
// committed dataset as CI splits it, and first proves the split carries
2071+
// whole-package seconds to grade; the fixtures hold each refusal.
2072+
battery('the whole-package leg cap and the load-sensitive pair (#25870)');
2073+
const wholeLeg = wholeLegProblems(real, datasetItems);
2074+
check(() => {
2075+
const wholeSeconds = datasetItems.filter((i) => !i.slice).reduce((sum, i) => sum + i.weight, 0);
2076+
if (wholeSeconds === 0 || Math.abs(wholeLeg.sums.reduce((a, b) => a + b, 0) - wholeSeconds) > 1e-6) {
2077+
throw new Error(`whole leg: the split's whole-package sums do not add up to the dataset's ${wholeSeconds}s`);
2078+
}
2079+
if (wholeLeg.problems.length > 0) {
2080+
throw new Error(
2081+
`whole leg, committed dataset at ${SHARD_COUNT} shards (sums ` +
2082+
wholeLeg.sums.map((t) => t.toFixed(0)).join('/') + 's):\n' +
2083+
wholeLeg.problems.map((p) => ` - ${p}`).join('\n')
2084+
);
2085+
}
2086+
});
2087+
// Two shards; `p` and `q` are the pair, 60s each. Split apart under a 100s cap
2088+
// it holds; a slice is not whole-leg weight, so 50s of slice beside 60s holds.
2089+
const legItems = [mk('p', 60), mk('q', 60), mk('r', 30), { ...mk('big 1/2', 50), slice: { index: 1, count: 2 } }];
2090+
const leg = (names, opts) => wholeLegProblems(names.map((n) => ({ names: n, total: 0 })), legItems, { cap: 100, apart: ['p', 'q'], ...opts });
2091+
check(() => {
2092+
const r = leg([['p', 'r', 'big 1/2'], ['q']]);
2093+
if (r.problems.length > 0) throw new Error(`whole leg: a split within the cap with the pair apart was refused: ${r.problems.join(' | ')}`);
2094+
});
2095+
check(() => {
2096+
const r = leg([['p', 'q'], ['r']]);
2097+
if (!r.problems.some((m) => m.includes('p and q share shard 1/2'))) {
2098+
throw new Error(`whole leg: the load-sensitive pair on one shard was accepted (${r.problems.join(' | ') || 'no problem named'})`);
2099+
}
2100+
});
2101+
check(() => {
2102+
const r = leg([['p', 'r', 'big 1/2'], ['q']], { cap: 80 });
2103+
if (!r.problems.some((m) => m.includes('shard 1/2 carries 90s of whole packages, past the 80s cap'))) {
2104+
throw new Error(`whole leg: a shard past the cap was accepted (${r.problems.join(' | ') || 'no problem named'})`);
2105+
}
2106+
});
2107+
check(() => {
2108+
const r = leg([['p', 'r'], ['big 1/2']], { apart: ['p', 'gone'] });
2109+
if (!r.problems.some((m) => m.includes('gone is not a whole package'))) {
2110+
throw new Error('whole leg: a pair member missing from the split was graded as apart');
2111+
}
2112+
});
2113+
19842114
// -- PREDICTED VS MEASURED (#16173) -------------------------------------
19852115
//
19862116
// The balancing pins above all read the dataset as GIVEN. None of them can
@@ -3016,6 +3146,7 @@ function selfTest() {
30163146
`-> ${datasetItems.length} shard items, ${SHARD_COUNT} shards, ` +
30173147
`max/mean ${balance.ratio.toFixed(2)}x <= ${MAX_SHARD_OVER_MEAN}x, floor ${balance.floor.toFixed(0)}s, ` +
30183148
`bins ${balance.totals.map((t) => t.toFixed(2)).join('/')}s within the ${datasetCap.toFixed(2)}s density cap, ` +
3149+
`whole legs ${wholeLeg.sums.map((t) => t.toFixed(0)).join('/')}s within ${WHOLE_LEG_CAP_SECONDS}s, ` +
30193150
`file-level slices: ` +
30203151
`${datasetPlan.decisions.filter((d) => d.count > 1).map((d) => `${d.name} x${d.count}`).join(', ') || 'none'})`
30213152
);

0 commit comments

Comments
 (0)