Skip to content

Commit 2301e17

Browse files
fix(ci): accumulate shard-timings runs until every package has three executions, provisional below that (#22022)
Part of #22014 Clause-②: no The shard-timings refresh now keeps adding scheduled runs until every package has been **executed** (not replayed) by at least three of them. Each weight is the median of those executions. When the retained runs cannot reach three for some package, the refresh is still opened, but it is marked **provisional** and does not get refused. The refresh PR body now records each package's run count and its min/median/max. #22014 stays open after this merges. Its other half is the first multi-run dataset landing on `main` through the refresh lane, which is the seat's act. Nothing in this PR touches `scripts/test-shard-timings.json`. ## Where the one-run dataset came from The cause was the regeneration loop in `.github/workflows/shard-timings-refresh.yml`, at `803764a36` lines 457-464 (comment at line 372). It ran `break` at the first accumulation that `--check-coverage` passed. Coverage counts a carried weight as covered, so the newest eligible run always passed it by itself. From then on every refresh was one run deep, which is why `provenance.runs` on `main` is `['37262126122']`. The selector was not the cause: it returns every eligible run in the window, up to 24. Retention was not the cause either: 20 eligible runs were in the window. The schedule-only rule was also not the cause. The median machinery from #16473 already worked. Nobody was feeding it more than one run. ## What changed - **`scripts/measure-test-shard-timings.mjs`** - `MINIMUM_EXECUTED_RUNS = 3`. - A new `provisional` list sits beside `carriedOver`. It names every package whose weight rests on fewer than that many executed runs. - A run counts for a package only when it executed that package. That means a MISS with exit 0, and for a sliced package, a complete slice set from that run. A cached run or a failed run does not count. - A carried weight keeps the depth it was measured at. A prior dataset with no `provisional` key never recorded a depth, so its carried weights are provisional. This includes the dataset on `main` today. - `provenance.minimumRuns` records the bar. - `--spread-out PATH` writes the per-package spread table for the PR body. `renderSpread` builds it. - `collectSamples` was split out of `buildDataset`, so the weights and the spread table read the same per-run readings. - **`.github/workflows/shard-timings-refresh.yml`** - The loop now stops only when the latest accumulation both covers the workspace and leaves `provisional` empty. - If the candidates run out first, the last covering accumulation is kept and a `::warning::` annotation is emitted. A coverage shortfall is still a refusal, unchanged. - The PR body gains a "Sample depth and run-to-run spread" section. - The header comments now say why the loop used to stop at one run. - **`scripts/ci/select-shard-timings-run.mjs`** - A note on `coverageReport`: coverage is necessary but not sufficient for the loop to stop. - A new self-test battery drives the real regeneration step (see Pins below). Untouched: the weights' shape (`packages` stays name to seconds), `partition-test-shards.mjs` and its constants, the 25% headroom, the 1.5x red, and the dataset file itself. ## Refuse or provisional: provisional, and why Triage left this choice to the PR. Each option is weighed on the four axes: - **Real need (measured).** Today the partitioner and the drift gate read the weights on every Test Core run, and they want the freshest numbers. Only a baseline reader, the per-package ceiling of #16468, is harmed by a shallow weight, and that reader can read the list. - Over the 20 eligible runs retained on 2026-10-06, all 72 packages reached three executions, so a provisional refresh should be rare. - When one does happen, refusing would throw away 71 packages' measurements because of one package and leave every PR balanced on the old dataset. - **Long-term soundness.** The workflow's own rule is that a refusal to open "would lose the measurement, which is the exact silent-rot failure this workflow exists to end". With provisional, the producer declares the depth in the file its consumers read. No consumer needs a fallback. - **Preventing AI mistakes.** The list is machine-readable and lives in the same file as the weights. - An empty list and a missing key are kept distinct. - A carried weight cannot become non-provisional just by being carried, and a depth that was never recorded counts as not met. - The dataset `note` says provisional weights are not baselines. - The remaining risk is a future consumer that ignores the list. That is a contract for #16468's ceiling generation to refuse provisional entries. It is stated here and is not built here. - **Startup focus.** The change is one constant, one list, one provenance number and one optional flag. It adds no new gate, no new exit code and no new workflow. A refusal path would instead add a weekly red job for someone to triage. ## Pins - `measure-test-shard-timings.mjs --self-test`, battery **`minimum executed runs per package`** (14 cases). It covers: - the one-run shape of the current dataset, kept and named provisional; - two runs (provisional) and three runs (empty list, present); - three runs where one package was replayed in two of them, so that package is provisional; - a failed run that does not count, and an incomplete slice set that does not count; - all three carry cases: a prior with no list, a prior with an empty list, and a prior that lists the package; - a malformed prior list, refused; - the bar recorded in `provenance`; - the spread table rows. - `select-shard-timings-run.mjs --self-test`, battery **`shard-timings-refresh accumulation depth`** (11 cases). It lifts the regeneration step's own `run:` block out of the YAML and runs it under `bash -e`. `curl` and `unzip` are stubbed with fixture summaries. The real generator and the real coverage check do the rest, from a scratch tree with a two-package committed fixture. The four legs: - **depth reached:** stops at 3 runs out of 4, not at 1, and spec is the median of three; - **executed, not replayed:** a replayed run does not count, so the loop takes 4; - **provisional, not refused:** with only 2 runs the step stays green, keeps the data, names both packages and annotates; - **coverage control:** a lost package still refuses with exit 1. ## The spread format This is a Markdown table in the refresh PR body, one row per package in the dataset: `package | runs | min s | median s | max s | max/min | provisional?` Rows are sorted widest spread first. A carried package shows `carried` instead of invented numbers. The spread is kept **out of the dataset JSON** because nothing machine-reads it. The partitioner and the drift gate read the weights, and the one machine fact a baseline needs is `provisional`. The only reader of the spread is the person reviewing the PR. This PR's own `pull_request` run of the refresh lane renders the would-be body, table included, into its step summary from the real retained runs, and pushes nothing. ## Per-package reachability of three executed runs within retention Run-summary artifacts are kept for 1 day. Per-package executed or replayed status was read from each run's `Test Core` timing table ("cache-replayed packages") in the aggregator job logs. Package names below drop the `@objectstack/` prefix. - **2026-10-05 17:01 to 2026-10-06 15:01** (all 20 eligible scheduled runs in retention when measured): - Every one of the 72 packages was executed by at least 3 runs. The newest 12 runs already reached 3 for every package. - Tightest: `spec` 7/20, `sdui-parser` 8/20, then `driver-sqlite-wasm`, `formula` and `plugin-hono-server` at 11/20. - Busy hours replay a lot: run 37475449497 replayed 54 of 72 packages. - **2026-10-04 05:31 to 2026-10-05 05:30**, the window the Monday 05:30 refresh reads (sampled, 9 of its 17 eligible runs): - Every package was executed by at least 4 of the 9 sampled runs. Tightest: `spec` 4/9. - The schedule fired late and irregularly that day, giving 19 runs in 24 hours, not 24. - **Not reachable: none measured.** `spec` is the package at risk. It is replayed in most hourly runs, and it is exactly the package whose single-run weight (1134.86 s) sat below its executed windows (1573-1651 s). If a quiet window ever leaves it below three, it is named in `provisional` rather than silently recorded from one run. ## How and when the first multi-run dataset reaches `main` 1. This PR merges. The scheduled trigger reads the workflow from `main`. 2. The schedule `30 5 * * 1` next fires **Monday 2026-10-12 05:30 UTC**. 3. The lane opens `claude/shard-timings-refresh-RUN_ID` as `github-actions[bot]`. Without `RELEASE_PUSH_TOKEN` that PR starts no CI by itself, so a maintainer push or update-branch is needed, as the previous refresh needed one. That step is unchanged here. 4. The PR lands through the queue as an ordinary non-governed diff. The maintainer or the seat could get it sooner with a manual `workflow_dispatch` of the refresh lane after merge. This PR does not do that. ## Evidence (at `68a312380`) - Self-tests: `node scripts/measure-test-shard-timings.mjs --self-test` printed `self-test OK`, exit 0. `node scripts/ci/select-shard-timings-run.mjs --self-test` printed `self-test OK`, exit 0, in about 9 s. - Ablations, each through `scripts/ablation-replace.mjs`, with the mutation proven on disk and the restore proven (blob equals HEAD and `git diff HEAD` empty): - Loop break reverted to coverage-only: the selector reds with `depth leg: expected the loop to stop at 3 runs, it fed 1`. - Generator rule that compares `values.length` with `minimumRuns` deleted: the generator reds with `a one-run refresh was not reported provisional ([])`, and the selector reds on the depth leg. - Carry rule inverted: the generator reds with `a weight carried from a depth-less dataset was not provisional ([])`. - Compose step: run once against a fake `RUNNER_TEMP` for the provisional branch, the full-depth branch and the absent-list branch. The absent list exits 1, as intended. - Gates: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 52 commands on the actual change set (3 paths vs merge base `803764a36`). All 52 were run. The 51 recorded with exit codes were all 0. `pnpm check:pm-dispatch-gates` printed `dispatch-gates self-test: 1976 cases pass`. Reconciliation with `--ran` reports 52 derived, 52 run, 0 unrun. - Lint, narrowed: eslint ran on the 2 changed `.mjs` files (the YAML is outside eslint's population) with 0 errors and 0 warnings. The repo config never enables type-aware linting (no `parserOptions.project`, per `eslint.config.mjs`), so this diff cannot move a verdict on any untouched file. The full `pnpm lint` is CI's. - `skip-changeset`: only the private root package's `scripts/` and a workflow change. No published `files[]` ships them. ## Acceptance notes - **#16468 contract (not built here).** Ceiling generation should refuse any package in the dataset's `provisional` list, and could also refuse a dataset whose `provenance.minimumRuns` is missing. Both readings are in the file now. - **Pre-existing, not changed.** When the generator refuses the very first candidate, `unset 'ACCEPTED[-1]'` leaves a sparse array. This path is unexercised, and no failure is measured. --- _Generated by [Claude Code](https://claude.ai/code/session_01VF48aw8RPG6wzDnMgp6rtw)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b88c356 commit 2301e17

3 files changed

Lines changed: 724 additions & 25 deletions

File tree

‎.github/workflows/shard-timings-refresh.yml‎

Lines changed: 101 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,28 @@
9898
# accumulated runs gets the median of three observations, which is the property
9999
# the dataset's own merge rule always claimed and could not previously deliver.
100100
#
101+
# ⚠ AND THE ACCUMULATION USED TO STOP AT ONE (#22014). The loop broke at the
102+
# first accumulation that COVERED the workspace — and coverage counts a carried
103+
# weight as covered, so the newest run always covered it. Every refresh since
104+
# the hourly run went live was therefore one run: the dataset on `main` named
105+
# `provenance.runs = ['37262126122']`, and recorded `@objectstack/spec` at
106+
# 1134.86s against 1573-1651s on the executed windows after it. The loop now
107+
# stops only when every package has been EXECUTED by at least the generator's
108+
# MINIMUM_EXECUTED_RUNS (three) of the accumulated runs — executed, not
109+
# replayed: an hourly run replays a different part of the workspace each time,
110+
# so three runs is not three samples of `spec`. Measured over the 20 eligible
111+
# hourly runs retained on 2026-10-06 (17:01 the day before to 15:01): every one
112+
# of the 72 packages was executed by at least three of them, the newest twelve
113+
# already sufficed, and `spec` was the tightest at 7 of 20.
114+
#
115+
# A refresh whose retained runs cannot give some package three executions is
116+
# NOT refused: it is opened PROVISIONAL, with those packages named in the
117+
# dataset's `provisional` list and in the PR body. Refusing would discard every
118+
# other package's measurement over one under-sampled one — the silent-rot
119+
# failure this lane exists to end — while the list keeps the shortfall
120+
# machine-readable for anything that needs a baseline rather than a balancing
121+
# input (a per-package ceiling must refuse a provisional weight).
122+
#
101123
# ⚠ AND ACCUMULATION ALONE STILL FALLS SHORT, so the pass MERGES rather than
102124
# replaces. Measured on the same live runs: the union of all seven retained runs
103125
# reaches 57 of 71 packages and converges there, because 14 packages are cache
@@ -266,11 +288,12 @@ jobs:
266288
# 24, not 15 (#16467). The window that matters is the artifact
267289
# retention window — 1 day — and the runs inside it are now the
268290
# hourly scheduled ones, so 24 is "everything still downloadable"
269-
# rather than an arbitrary depth. Coverage is ACCUMULATED across
270-
# runs (see the next step), so examining fewer than the window holds
271-
# is coverage left on the table; the cost is two API reads per run
272-
# examined and the loop below still stops at the first accumulation
273-
# that covers the workspace.
291+
# rather than an arbitrary depth. Coverage and sample depth are both
292+
# ACCUMULATED across runs (see the next step), so examining fewer
293+
# than the window holds is samples left on the table; the cost is two
294+
# API reads per run examined, and the loop below stops at the first
295+
# accumulation in which every package has been executed by the
296+
# minimum number of runs (#22014).
274297
#
275298
# ⭐ THREE EXITS, THREE READINGS (#16467). The selector answers 0, 1
276299
# or 3, and this step must not flatten them:
@@ -369,9 +392,16 @@ jobs:
369392
# observations rather than whichever run happened to be read last. Feeding
370393
# several runs WITHOUT that grouping is refused by the generator by name.
371394
#
372-
# The loop stops at the first accumulation that covers the workspace, so a
373-
# quiet week costs one run's download and a busy one costs a few.
374-
- name: Regenerate the dataset, accumulating runs until the workspace is covered
395+
# WHEN IT STOPS (#22014). Not at the first accumulation that covers the
396+
# workspace: coverage counts a carried weight as covered, so the newest
397+
# run alone always passed it and every refresh was one run deep. The loop
398+
# stops at the first accumulation that BOTH covers the workspace AND
399+
# leaves the dataset's `provisional` list empty — every package executed
400+
# by at least the generator's MINIMUM_EXECUTED_RUNS of the runs fed.
401+
# When the candidates run out first, the last covering accumulation is
402+
# kept and opened PROVISIONAL, its short packages named; only a coverage
403+
# shortfall is still a refusal.
404+
- name: Regenerate the dataset, accumulating runs until every package is measured enough times
375405
id: generate
376406
# Skipped on the NOT MEASURED reading: there is nothing to download.
377407
# Every step after `compare` is already gated on
@@ -384,7 +414,11 @@ jobs:
384414
set -euo pipefail
385415
WORK="$RUNNER_TEMP/refresh"
386416
mkdir -p "$WORK"
387-
CHOSEN=''
417+
# Both judged on the LAST accumulation the generator accepted, which
418+
# is the one `refreshed.json` holds: a run the generator refuses is
419+
# dropped and leaves the previous accumulation's file in place.
420+
COVERED=''
421+
SHORT=''
388422
ACCEPTED=()
389423
390424
RUN_COUNT=$(node -e 'console.log(JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8")).length)')
@@ -445,9 +479,13 @@ jobs:
445479
# HIT witnesses that its inputs are unchanged; anything else is left
446480
# out for the coverage check below to name. The workflow still writes
447481
# no byte the generator did not emit — the merge happens inside it.
482+
# `--spread-out`: every package's executed-run count and
483+
# min/median/max, for the PR body. Written beside the dataset,
484+
# never into it — nothing machine-reads a spread.
448485
if ! node scripts/measure-test-shard-timings.mjs "${ARGS[@]}" \
449486
--merge-into scripts/test-shard-timings.json \
450-
--out "$WORK/refreshed.json"; then
487+
--out "$WORK/refreshed.json" \
488+
--spread-out "$WORK/spread.md"; then
451489
echo "::warning::The generator refused the set including run $RUN_ID; dropping that run and continuing."
452490
unset 'ACCEPTED[-1]'
453491
echo "::endgroup::"; continue
@@ -459,17 +497,41 @@ jobs:
459497
--refreshed "$WORK/refreshed.json" \
460498
--workspace "$RUNNER_TEMP/turbo-ls.json" \
461499
--exclude @objectstack/dogfood; then
462-
CHOSEN="$RUN_ID"
500+
COVERED=1
501+
else
502+
COVERED=''
503+
fi
504+
# How many weights still rest on fewer than the minimum executed
505+
# runs. Read from the generator's own `provisional` list rather than
506+
# recounted here, and an absent list is an error, not a zero: a
507+
# zero read from nothing would end the loop at one run again.
508+
SHORT=$(node -e '
509+
const d = JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"));
510+
if (!Array.isArray(d.provisional)) throw new Error(process.argv[1] + " carries no `provisional` list");
511+
console.log(d.provisional.length);
512+
' "$WORK/refreshed.json")
513+
if [ -n "$COVERED" ] && [ "$SHORT" -eq 0 ]; then
514+
echo "Every package is covered and executed by at least the minimum number of these runs; stopping."
463515
echo "::endgroup::"
464516
break
465517
fi
518+
if [ -n "$COVERED" ]; then
519+
echo "Covered, but $SHORT package weight(s) rest on fewer than the minimum executed runs; adding the next older run."
520+
fi
466521
echo "::endgroup::"
467522
done
468523
469-
if [ -z "$CHOSEN" ]; then
524+
if [ -z "$COVERED" ]; then
470525
echo "::error::The ${#ACCEPTED[@]} eligible run(s) on main, accumulated and merged with the committed dataset, still leave a package that HAD a measured weight with neither a fresh measurement nor a turbo cache HIT to witness that it is unchanged — the shortfall above names them. Each would drop to the test-file-count ESTIMATE, which is the silent degradation this lane exists to prevent. NOTHING was regenerated and no PR was opened: this is a refusal, not a quiet success. A package that merely went unmeasured is NOT this error — that case is carried on its cache-hit witness — so a shortfall here means a suite failed, a package was renamed or removed, or its slices could not be assembled in any run."
471526
exit 1
472527
fi
528+
# PROVISIONAL, not refused (#22014): the candidates ran out before
529+
# every package reached the minimum. The measurement is kept, the
530+
# short packages are named in the dataset and the PR body, and this
531+
# annotation makes the reading visible on a green job.
532+
if [ "$SHORT" -gt 0 ]; then
533+
echo "::warning::Shard timings PROVISIONAL: $SHORT package weight(s) rest on fewer than the minimum executed runs across all ${#ACCEPTED[@]} retained run(s). They are named in the dataset's \`provisional\` list; read them as balancing inputs, not baselines."
534+
fi
473535
# Candidates arrive newest-first, so ACCEPTED[0] is the most recent run
474536
# in the set and its commit is the one the refresh is dated from. The
475537
# full list travels alongside it: every run in it contributed
@@ -553,6 +615,25 @@ jobs:
553615
}
554616
')
555617
618+
# Sample depth (#22014), read out of the dataset like the carry line
619+
# above. An absent list is an error rather than "none provisional".
620+
DEPTH_LINE=$(node -e '
621+
const d = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/refresh/refreshed.json", "utf8"));
622+
if (!Array.isArray(d.provisional)) throw new Error("the refreshed dataset carries no `provisional` list");
623+
const min = d.provenance.minimumRuns;
624+
const total = Object.keys(d.packages).length;
625+
if (d.provisional.length === 0) {
626+
console.log(`Every one of the ${total} weights rests on at least ${min} executed runs; none is provisional.`);
627+
} else {
628+
console.log(
629+
`⚠️ PROVISIONAL: ${d.provisional.length} of the ${total} weights rest on fewer than ${min} executed ` +
630+
`runs, because the retained runs replayed them more often than they executed them. They are ` +
631+
`named in the dataset \`provisional\` list and are fit to balance on, NOT to set a per-package ` +
632+
`ceiling from: ${d.provisional.join(", ")}.`
633+
);
634+
}
635+
')
636+
556637
SHARDS=$(node -e '
557638
const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
558639
const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID);
@@ -576,8 +657,8 @@ jobs:
576657
echo "per shard and only main pushes write it, so a"
577658
echo "package whose inputs have not changed is a HIT and the generator refuses hits rather"
578659
echo "than recording a replay as a duration. Runs are therefore accumulated, each fenced by"
579-
echo "its own \`--run\` group, until every package the committed dataset holds is measured"
580-
echo "again; a package seen in several of them gets the median of those observations."
660+
echo "its own \`--run\` group, until every package has been EXECUTED by at least the"
661+
echo "generator's minimum number of them, and each weight is the median of those executions."
581662
echo
582663
for R in $RUNS; do
583664
echo "- https://github.com/$GITHUB_REPOSITORY/actions/runs/$R"
@@ -594,6 +675,12 @@ jobs:
594675
echo "because a weekly lane cannot know which cards a given run ought to retire. If this is"
595676
echo "the first refresh to land, retire those two by hand as part of merging it."
596677
echo
678+
echo "## Sample depth and run-to-run spread"
679+
echo
680+
echo "$DEPTH_LINE"
681+
echo
682+
cat "$RUNNER_TEMP/refresh/spread.md"
683+
echo
597684
echo "## Measured per-shard suite time on the newest run in the set"
598685
echo
599686
echo "\`\`\`"

0 commit comments

Comments
 (0)