Repository navigation
Commit 2301e17
fix(ci): accumulate shard-timings runs until every package has three executions, provisional below that (#22022)
Part of #22014
Clause-②: no
The shard-timings refresh now keeps adding scheduled runs until every
package has been **executed** (not replayed) by at least three of them.
Each weight is the median of those executions. When the retained runs
cannot reach three for some package, the refresh is still opened, but it
is marked **provisional** and does not get refused. The refresh PR body
now records each package's run count and its min/median/max.
#22014 stays open after this merges. Its other half is the first
multi-run dataset landing on `main` through the refresh lane, which is
the seat's act. Nothing in this PR touches
`scripts/test-shard-timings.json`.
## Where the one-run dataset came from
The cause was the regeneration loop in
`.github/workflows/shard-timings-refresh.yml`, at `803764a36` lines
457-464 (comment at line 372). It ran `break` at the first accumulation
that `--check-coverage` passed. Coverage counts a carried weight as
covered, so the newest eligible run always passed it by itself. From
then on every refresh was one run deep, which is why `provenance.runs`
on `main` is `['37262126122']`. The selector was not the cause: it
returns every eligible run in the window, up to 24. Retention was not
the cause either: 20 eligible runs were in the window. The schedule-only
rule was also not the cause. The median machinery from #16473 already
worked. Nobody was feeding it more than one run.
## What changed
- **`scripts/measure-test-shard-timings.mjs`**
- `MINIMUM_EXECUTED_RUNS = 3`.
- A new `provisional` list sits beside `carriedOver`. It names every
package whose weight rests on fewer than that many executed runs.
- A run counts for a package only when it executed that package. That
means a MISS with exit 0, and for a sliced package, a complete slice set
from that run. A cached run or a failed run does not count.
- A carried weight keeps the depth it was measured at. A prior dataset
with no `provisional` key never recorded a depth, so its carried weights
are provisional. This includes the dataset on `main` today.
- `provenance.minimumRuns` records the bar.
- `--spread-out PATH` writes the per-package spread table for the PR
body. `renderSpread` builds it.
- `collectSamples` was split out of `buildDataset`, so the weights and
the spread table read the same per-run readings.
- **`.github/workflows/shard-timings-refresh.yml`**
- The loop now stops only when the latest accumulation both covers the
workspace and leaves `provisional` empty.
- If the candidates run out first, the last covering accumulation is
kept and a `::warning::` annotation is emitted. A coverage shortfall is
still a refusal, unchanged.
- The PR body gains a "Sample depth and run-to-run spread" section.
- The header comments now say why the loop used to stop at one run.
- **`scripts/ci/select-shard-timings-run.mjs`**
- A note on `coverageReport`: coverage is necessary but not sufficient
for the loop to stop.
- A new self-test battery drives the real regeneration step (see Pins
below).
Untouched: the weights' shape (`packages` stays name to seconds),
`partition-test-shards.mjs` and its constants, the 25% headroom, the
1.5x red, and the dataset file itself.
## Refuse or provisional: provisional, and why
Triage left this choice to the PR. Each option is weighed on the four
axes:
- **Real need (measured).** Today the partitioner and the drift gate
read the weights on every Test Core run, and they want the freshest
numbers. Only a baseline reader, the per-package ceiling of #16468, is
harmed by a shallow weight, and that reader can read the list.
- Over the 20 eligible runs retained on 2026-10-06, all 72 packages
reached three executions, so a provisional refresh should be rare.
- When one does happen, refusing would throw away 71 packages'
measurements because of one package and leave every PR balanced on the
old dataset.
- **Long-term soundness.** The workflow's own rule is that a refusal to
open "would lose the measurement, which is the exact silent-rot failure
this workflow exists to end". With provisional, the producer declares
the depth in the file its consumers read. No consumer needs a fallback.
- **Preventing AI mistakes.** The list is machine-readable and lives in
the same file as the weights.
- An empty list and a missing key are kept distinct.
- A carried weight cannot become non-provisional just by being carried,
and a depth that was never recorded counts as not met.
- The dataset `note` says provisional weights are not baselines.
- The remaining risk is a future consumer that ignores the list. That is
a contract for #16468's ceiling generation to refuse provisional
entries. It is stated here and is not built here.
- **Startup focus.** The change is one constant, one list, one
provenance number and one optional flag. It adds no new gate, no new
exit code and no new workflow. A refusal path would instead add a weekly
red job for someone to triage.
## Pins
- `measure-test-shard-timings.mjs --self-test`, battery **`minimum
executed runs per package`** (14 cases). It covers:
- the one-run shape of the current dataset, kept and named provisional;
- two runs (provisional) and three runs (empty list, present);
- three runs where one package was replayed in two of them, so that
package is provisional;
- a failed run that does not count, and an incomplete slice set that
does not count;
- all three carry cases: a prior with no list, a prior with an empty
list, and a prior that lists the package;
- a malformed prior list, refused;
- the bar recorded in `provenance`;
- the spread table rows.
- `select-shard-timings-run.mjs --self-test`, battery
**`shard-timings-refresh accumulation depth`** (11 cases). It lifts the
regeneration step's own `run:` block out of the YAML and runs it under
`bash -e`. `curl` and `unzip` are stubbed with fixture summaries. The
real generator and the real coverage check do the rest, from a scratch
tree with a two-package committed fixture. The four legs:
- **depth reached:** stops at 3 runs out of 4, not at 1, and spec is the
median of three;
- **executed, not replayed:** a replayed run does not count, so the loop
takes 4;
- **provisional, not refused:** with only 2 runs the step stays green,
keeps the data, names both packages and annotates;
- **coverage control:** a lost package still refuses with exit 1.
## The spread format
This is a Markdown table in the refresh PR body, one row per package in
the dataset:
`package | runs | min s | median s | max s | max/min | provisional?`
Rows are sorted widest spread first. A carried package shows `carried`
instead of invented numbers.
The spread is kept **out of the dataset JSON** because nothing
machine-reads it. The partitioner and the drift gate read the weights,
and the one machine fact a baseline needs is `provisional`. The only
reader of the spread is the person reviewing the PR. This PR's own
`pull_request` run of the refresh lane renders the would-be body, table
included, into its step summary from the real retained runs, and pushes
nothing.
## Per-package reachability of three executed runs within retention
Run-summary artifacts are kept for 1 day. Per-package executed or
replayed status was read from each run's `Test Core` timing table
("cache-replayed packages") in the aggregator job logs. Package names
below drop the `@objectstack/` prefix.
- **2026-10-05 17:01 to 2026-10-06 15:01** (all 20 eligible scheduled
runs in retention when measured):
- Every one of the 72 packages was executed by at least 3 runs. The
newest 12 runs already reached 3 for every package.
- Tightest: `spec` 7/20, `sdui-parser` 8/20, then `driver-sqlite-wasm`,
`formula` and `plugin-hono-server` at 11/20.
- Busy hours replay a lot: run 37475449497 replayed 54 of 72 packages.
- **2026-10-04 05:31 to 2026-10-05 05:30**, the window the Monday 05:30
refresh reads (sampled, 9 of its 17 eligible runs):
- Every package was executed by at least 4 of the 9 sampled runs.
Tightest: `spec` 4/9.
- The schedule fired late and irregularly that day, giving 19 runs in 24
hours, not 24.
- **Not reachable: none measured.** `spec` is the package at risk. It is
replayed in most hourly runs, and it is exactly the package whose
single-run weight (1134.86 s) sat below its executed windows (1573-1651
s). If a quiet window ever leaves it below three, it is named in
`provisional` rather than silently recorded from one run.
## How and when the first multi-run dataset reaches `main`
1. This PR merges. The scheduled trigger reads the workflow from `main`.
2. The schedule `30 5 * * 1` next fires **Monday 2026-10-12 05:30 UTC**.
3. The lane opens `claude/shard-timings-refresh-RUN_ID` as
`github-actions[bot]`. Without `RELEASE_PUSH_TOKEN` that PR starts no CI
by itself, so a maintainer push or update-branch is needed, as the
previous refresh needed one. That step is unchanged here.
4. The PR lands through the queue as an ordinary non-governed diff.
The maintainer or the seat could get it sooner with a manual
`workflow_dispatch` of the refresh lane after merge. This PR does not do
that.
## Evidence (at `68a312380`)
- Self-tests: `node scripts/measure-test-shard-timings.mjs --self-test`
printed `self-test OK`, exit 0. `node
scripts/ci/select-shard-timings-run.mjs --self-test` printed `self-test
OK`, exit 0, in about 9 s.
- Ablations, each through `scripts/ablation-replace.mjs`, with the
mutation proven on disk and the restore proven (blob equals HEAD and
`git diff HEAD` empty):
- Loop break reverted to coverage-only: the selector reds with `depth
leg: expected the loop to stop at 3 runs, it fed 1`.
- Generator rule that compares `values.length` with `minimumRuns`
deleted: the generator reds with `a one-run refresh was not reported
provisional ([])`, and the selector reds on the depth leg.
- Carry rule inverted: the generator reds with `a weight carried from a
depth-less dataset was not provisional ([])`.
- Compose step: run once against a fake `RUNNER_TEMP` for the
provisional branch, the full-depth branch and the absent-list branch.
The absent list exits 1, as intended.
- Gates: `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 52 commands on the actual change set
(3 paths vs merge base `803764a36`). All 52 were run. The 51 recorded
with exit codes were all 0. `pnpm check:pm-dispatch-gates` printed
`dispatch-gates self-test: 1976 cases pass`. Reconciliation with `--ran`
reports 52 derived, 52 run, 0 unrun.
- Lint, narrowed: eslint ran on the 2 changed `.mjs` files (the YAML is
outside eslint's population) with 0 errors and 0 warnings. The repo
config never enables type-aware linting (no `parserOptions.project`, per
`eslint.config.mjs`), so this diff cannot move a verdict on any
untouched file. The full `pnpm lint` is CI's.
- `skip-changeset`: only the private root package's `scripts/` and a
workflow change. No published `files[]` ships them.
## Acceptance notes
- **#16468 contract (not built here).** Ceiling generation should refuse
any package in the dataset's `provisional` list, and could also refuse a
dataset whose `provenance.minimumRuns` is missing. Both readings are in
the file now.
- **Pre-existing, not changed.** When the generator refuses the very
first candidate, `unset 'ACCEPTED[-1]'` leaves a sparse array. This path
is unexercised, and no failure is measured.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01VF48aw8RPG6wzDnMgp6rtw)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent b88c356 commit 2301e17
3 files changed
Lines changed: 724 additions & 25 deletions
File tree
- .github/workflows
- scripts
- ci
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
101 | 123 | | |
102 | 124 | | |
103 | 125 | | |
| |||
266 | 288 | | |
267 | 289 | | |
268 | 290 | | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
273 | | - | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
274 | 297 | | |
275 | 298 | | |
276 | 299 | | |
| |||
369 | 392 | | |
370 | 393 | | |
371 | 394 | | |
372 | | - | |
373 | | - | |
374 | | - | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
375 | 405 | | |
376 | 406 | | |
377 | 407 | | |
| |||
384 | 414 | | |
385 | 415 | | |
386 | 416 | | |
387 | | - | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
388 | 422 | | |
389 | 423 | | |
390 | 424 | | |
| |||
445 | 479 | | |
446 | 480 | | |
447 | 481 | | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
448 | 485 | | |
449 | 486 | | |
450 | | - | |
| 487 | + | |
| 488 | + | |
451 | 489 | | |
452 | 490 | | |
453 | 491 | | |
| |||
459 | 497 | | |
460 | 498 | | |
461 | 499 | | |
462 | | - | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
463 | 515 | | |
464 | 516 | | |
465 | 517 | | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
466 | 521 | | |
467 | 522 | | |
468 | 523 | | |
469 | | - | |
| 524 | + | |
470 | 525 | | |
471 | 526 | | |
472 | 527 | | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
473 | 535 | | |
474 | 536 | | |
475 | 537 | | |
| |||
553 | 615 | | |
554 | 616 | | |
555 | 617 | | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
556 | 637 | | |
557 | 638 | | |
558 | 639 | | |
| |||
576 | 657 | | |
577 | 658 | | |
578 | 659 | | |
579 | | - | |
580 | | - | |
| 660 | + | |
| 661 | + | |
581 | 662 | | |
582 | 663 | | |
583 | 664 | | |
| |||
594 | 675 | | |
595 | 676 | | |
596 | 677 | | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
597 | 684 | | |
598 | 685 | | |
599 | 686 | | |
| |||
0 commit comments