From ebc7c8ffcb09cfc9b9160d7f8f6062f512341312 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 25 Oct 2026 04:31:00 +0000 Subject: [PATCH 1/2] ci(test-shards): widen Test Core from 6 to 8 shards Sized on merge_group run 40155484566 on current main (59 packages executed, the CLI as 3 slices). Replayed through planRun() with a wall model calibrated on that run, the slowest shard drops from ~1696s at six (measured 1794s) to ~1436s at eight, -15%; -14% on the dataset's full list and -10% on a 21-package merge_group run. Predicted bins go from 2489s to 1868s, ratio 1.00x, with spec's 1607s floor binding only at ten. - SHARD_COUNT 6 -> 8 in partition-test-shards.mjs and the timings selector; ci.yml matrix, name, --shard, --total, artifact names and the aggregator's --leg move with it. - shard-timings-refresh.yml reads the shard roster from the selector's candidates instead of spelling 1..6. - Test Core timeout-minutes 50 -> 60: the stall-guard headroom tool over the last day's 512 shard jobs found 50 already short at six (worst p + s 40m27s); scaled to eight, the cap-deferred verdict lands at 56m37s. - Self-test fixtures that read the live ci.yml follow the new names; the replays of measured six-shard runs read the six-shard name. - The standing "6 is enough" comments are rewritten with the new numbers. Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb --- .github/workflows/ci.yml | 149 ++++++++++++-------- .github/workflows/shard-timings-refresh.yml | 45 +++--- scripts/check-required-contexts.mjs | 6 +- scripts/check-shard-attestation.mjs | 23 +-- scripts/ci/select-shard-timings-run.mjs | 88 ++++++------ scripts/measure-test-shard-timings.mjs | 8 +- scripts/partition-test-shards.mjs | 43 +++++- scripts/pm/check-expected-skips.mjs | 8 +- 8 files changed, 226 insertions(+), 144 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1ea57dcd62b..2fff3c28fea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -101,7 +101,7 @@ on: # fires on a push, and for everything else it is the policy. Two hourly runs # overlapping means one of them is over an hour old, and an hour-old # measurement of `main` is strictly the worse of the two. Test Core's wall clock -# is well inside the cadence (~39 shard-minutes across six shards), so this is +# is well inside the cadence (~39 shard-minutes across the shards), so this is # the exceptional path, not the normal one. concurrency: group: ci-${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.event.after || github.ref }} @@ -489,11 +489,11 @@ jobs: - 'packages/types/**' test: - # Sharded 6-way BY PACKAGE: a core-touching PR ran the affected suite - # ~11½ min on one 4-vCPU runner — the longest pole in the whole workflow — - # and at 2 shards the slower shard still ran ~10 min, keeping merge-queue - # builds at ~11 min end-to-end (#4859). - # scripts/partition-test-shards.mjs splits the package list into six + # Sharded 8-way BY PACKAGE (6-way until #25376): a core-touching PR ran + # the affected suite ~11½ min on one 4-vCPU runner — the longest pole in + # the whole workflow — and at 2 shards the slower shard still ran ~10 min, + # keeping merge-queue builds at ~11 min end-to-end (#4859). + # scripts/partition-test-shards.mjs splits the package list into eight # deterministic bins balanced on MEASURED per-package test duration # (scripts/test-shard-timings.json, generated — see #10472 below) and each # shard runs its slice through turbo. NOT the dogfood @@ -504,7 +504,7 @@ jobs: # on either shard (three packages have exactly one test file today). See # the script header for the verification. # - # ── WHY SIX AND NOT THREE (#10227) ──────────────────────────────────── + # ── WHY MORE THAN THREE (#10227) ────────────────────────────────────── # # The per-package durations this job's logs print are CONTENDED WALL # CLOCK, not package cost. `--concurrency=4` on a 4-vCPU runner means up @@ -522,59 +522,69 @@ jobs: # shards means fewer suites per runner AND four more vCPUs per added # shard, because every shard gets its own runner. # - # SIX specifically, and not four or five: sharding is BY PACKAGE, so no - # shard can finish faster than the heaviest serial task it carries. On the - # current dataset (scripts/test-shard-timings.json, the 21-run refresh at - # 040184752c) the heaviest package is `@objectstack/cli` at 1738.88s, and - # run whole it is one serial suite with nothing to overlap it: the job - # holding it ran 20.5-39.1 min on the 14 runs after that refresh while every - # other Test Core job finished within 20.8 (#22075). So it is cut into 3 + # EIGHT specifically (#25376; six from #22075 until then): sharding is BY + # PACKAGE, so no shard can finish faster than the heaviest serial task it + # carries. `@objectstack/cli` (2899.02s on the 2026-10-23 dataset) runs as 3 # file-level slices whenever a run's own list makes it the serial floor # (FILE_SHARDED_PACKAGES and planShards() in the script), each slice placed # at its PLAIN weight so every bin keeps the whole-package split's density, # and no slicing may put a shard past that split's densest bin (the density - # cap, 1772.65s today -- densityCap() in the script; the shard-timing drift - # check below is why). Re-derived with the partitioner's own planShards() - # and balanceOf(), bins of predicted test windows: + # cap, densityCap() in the script; the shard-timing drift check below is + # why). Re-derived with the partitioner's own planShards() and balanceOf() + # on scripts/test-shard-timings.json, bins of predicted test windows: # - # 4 shards -> max 2658s mean 2658s ratio 1.00x - # 5 shards -> max 2127s mean 2126s ratio 1.00x - # 6 shards -> max 1773s mean 1772s ratio 1.00x - # 7 shards -> max 1522s mean 1519s ratio 1.00x - # 8 shards -> max 1332s mean 1329s ratio 1.00x - # 10 shards -> max 1147s mean 1063s ratio 1.08x + # 4 shards -> max 3733s mean 3732s ratio 1.00x (the CLI runs whole) + # 6 shards -> max 2489s mean 2488s ratio 1.00x + # 7 shards -> max 2136s mean 2133s ratio 1.00x + # 8 shards -> max 1868s mean 1866s ratio 1.00x + # 9 shards -> max 1661s mean 1659s ratio 1.00x + # 10 shards -> max 1607s mean 1493s ratio 1.08x + # + # Bins are not wall. A shard's whole packages run four at a time while a + # CLI slice runs as a serial leg after them, so its tests step reads close + # to wholeSum / 2.9 + slice. On merge_group run 40155484566 (59 packages + # executed, the CLI as 3 slices) that model lands within 6.1% of the + # measured step on the four slowest shards (the two light ones never set + # the wall); replayed on that run's own list it puts the slowest shard + # at ~1696s at six (measured 1794s, a slice shard) and ~1436s at eight, + # -15%. Full dataset list -14%; a 21-package merge_group run (40157571602) + # -10%, its slice shards at eight carrying almost nothing else. The + # partitioner's SHARD_COUNT header carries the arithmetic. # # ⚠ THE ARGUMENT ABOVE USED TO BE MADE IN TEST-FILE COUNTS (bins # 415/389/389/389/389/388, spec carrying 415 of ~2360 files). It is now # made in measured seconds, because #10472 replaced the weight input: the # file count was a proxy that ran ~2.8x off on @objectstack/cli, and the # six perfectly count-balanced bins it produced ran 5.0/6.2/6.3/13.6/6.3/ - # 9.0 min in run 32428961038. The conclusion (six) survived the re- - # measurement; only the units it is argued in changed. Weights now come - # from scripts/test-shard-timings.json — see partition-test-shards.mjs. + # 9.0 min in run 32428961038. Weights now come from + # scripts/test-shard-timings.json — see partition-test-shards.mjs. # - # AND WHY NOT MORE THAN SIX (#10472 asked for 6 -> 8 to be considered). - # The floor is the heaviest single item -- `@objectstack/spec`, 1146.69s, - # once the CLI is sliced -- and it does not move when shards are added, - # while the mean falls with every shard, so max/mean gets WORSE past the - # point where the floor becomes the max (ten shards above). The + # AND WHY NOT MORE THAN EIGHT. #10472 asked for 6 -> 8 and was refused + # while spec's 1146.69s was the floor; #25376 widened once the CLI was + # sliced and spec's 842s repo file was split (#25372). Two floors remain. + # In bins, spec's whole 1606.55s binds at ten shards (above), and past it + # max/mean gets WORSE with every shard. In wall, a CLI slice (~960-1010s + # measured) is serial: at eight the slice shards of a small run already + # carry little else, so a ninth shard mostly adds a runner. The # partitioner's self-test pins the half of this that can fail: at # SHARD_COUNT the heaviest item must sit within 1.3x the mean, and when it # stops fitting, slicing it below package granularity is the remedy that # failure names. # - # ⚠ THE COST, stated because it is real: per-shard fixed overhead - # (checkout + pnpm/Turbo cache restore + install, ~60s measured on run - # 32392712980) is paid once PER SHARD, and the merge queue multiplies it - # by the speculative build count. Six shards trade roughly 1.7x the - # billed runner-minutes for a shorter critical path. `--concurrency=4` is - # deliberately UNCHANGED here: narrower shards may well want a lower - # value, but moving both at once makes the next measurement unreadable. + # ⚠ THE COST, stated because it is real: per-shard fixed overhead is paid + # once PER SHARD, and the merge queue multiplies it by the speculative + # build count. Measured over the 512 `Test Core (N/6)` jobs of the window + # below, job start to the tests step is 1.6 min median (5.5 p90) and the + # tail after it 0.2 min, so two more shards cost ≈ +3.6 runner-min per run + # at the median, accepted by the maintainer directive on #25376. + # `--concurrency=4` is deliberately UNCHANGED here: narrower shards may + # well want a lower value, but moving both at once makes the next + # measurement unreadable. # # Branch protection requires the bare "Test Core" context, which a matrix # can never publish again — the test-gate job below carries that name # (the #3622 lesson; see dogfood-gate). - name: Test Core (${{ matrix.shard }}/6) + name: Test Core (${{ matrix.shard }}/8) needs: filter # "Skip only when the filter EXPLICITLY said no core paths changed" — see # THE FILTER CONTRACT on the filter job's outputs (#4928). `scripts` is ORed @@ -614,25 +624,50 @@ jobs: # (70 packages executed, the CLI as all 3 slices), is within 80% of # 50 (40.0). # + # ⚠ RE-SIZED 50 -> 60 (#25376), with the matrix widened 6 -> 8. The same + # tool over the 89 completed, non-cancelled CI runs created 2026-10-24 + # 03:00Z-2026-10-25 03:45Z (23 merge_group, 39 pull_request, 16 push, + # 11 schedule; 512 `Test Core (N/6)` jobs) found 50 already short at six: + # worst p + s 40m27s (run 40149547507 `Test Core (1/6)`, a red PR shard; + # healthy worst 37m22s, run 40125226571 `Test Core (4/6)`), so the + # cap-deferred path needed 60.45m. The slowest shard is the one carrying + # a CLI slice; eight shards take it down by 10-15% on the replays in + # partition-test-shards.mjs's SHARD_COUNT header, so s is scaled by 0.90 + # (the smallest gain, the safe side) until a window of eight-shard runs + # exists. Five readings, the largest answer wins: + # * the window verdict: p + s + 10m = 2m03s + 34m34s + 10m = 46m37s; + # * the cap-deferred verdict: p + s + 20m = 56m37s, inside 60 by 3m23s; + # * the 80% wall margin: the healthy worst job wall, 37m41s at six, is + # ~34m40s at eight (7m28s + 26m55s + the 0.3m tail), within 80% of 60 + # (48.0); + # * the #16465 red line, the slowest shard at or under 70% of its + # timeout: ~34m40s against 42.0; + # * the drift ceiling: that healthy worst step running 1.5x its + # prediction (MAX_MEASURED_OVER_PREDICTED, the DRIFT verdict) lands at + # 7m28s + 26m55s x 1.5 = 47.8m, still a reading before the wall rather + # than a kill. + # The first full-battery run after landing re-takes these on eight-shard + # jobs and is reported on #25376. + # # ⛔ REVERT CONDITION, restated against measurement, not a card: a value - # below 50 is allowed only when, on a re-measured window of Test Core + # below 60 is allowed only when, on a re-measured window of Test Core # runs (`node scripts/measure-stall-guard-headroom.mjs` over the jobs # payloads of every completed pull_request, merge_group, push and # schedule run in it), BOTH guard verdicts -- the window path # (p + s + 10m) and the cap-deferred path (p + s + 20m) -- land before the - # new value on every observation, AND the slowest `Test Core (N/6)` job + # new value on every observation, AND the slowest `Test Core (N/8)` job # wall is at or under 80% of it. Read the pull_request and merge_group # runs, not the hourly schedule alone: scheduled runs mostly replay the # cache their commit's push run saved. It carries no other expiry — a # raise with no revert condition beside it becomes permanent by # forgetting — and the stall guard stays the primary hang detector. - timeout-minutes: 50 + timeout-minutes: 60 permissions: contents: read strategy: fail-fast: false matrix: - shard: [1, 2, 3, 4, 5, 6] + shard: [1, 2, 3, 4, 5, 6, 7, 8] steps: - name: Checkout repository @@ -780,7 +815,7 @@ jobs: run: | bash scripts/ci/select-shard-packages.sh node scripts/partition-test-shards.mjs "$RUNNER_TEMP/turbo-ls.json" \ - --shard ${{ matrix.shard }}/6 --exclude @objectstack/dogfood \ + --shard ${{ matrix.shard }}/8 --exclude @objectstack/dogfood \ > "$RUNNER_TEMP/shard-packages.txt" echo 'Items on this shard (a package name, or a package plus a k/n file-level slice):' cat "$RUNNER_TEMP/shard-packages.txt" @@ -884,7 +919,7 @@ jobs: # `--if-ran`: an empty shard exits the step above before turbo runs. - name: The Turbo remote cache was on for this shard's closure build run: | - node scripts/ci/assert-turbo-remote-cache.mjs --site 'Test Core (${{ matrix.shard }}/6) closure' \ + node scripts/ci/assert-turbo-remote-cache.mjs --site 'Test Core (${{ matrix.shard }}/8) closure' \ --log .turbo/remote-cache/test-core-closure.json --if-ran \ --move-summary-to "$RUNNER_TEMP/turbo-remote-cache" @@ -1226,14 +1261,14 @@ jobs: run: | node scripts/ci/report-test-leftovers.mjs report \ --before "$RUNNER_TEMP/test-leftovers/before.json" \ - --label "Test Core (${{ matrix.shard }}/6)" + --label "Test Core (${{ matrix.shard }}/8)" # --summarize above costs nothing at runtime and writes # `.turbo/runs/.json`: one per-task record with the execution window # turbo actually observed. That file IS the balancing input for the next # shard split (scripts/test-shard-timings.json is generated from it), and # publishing it here is what keeps the dataset re-measurable instead of - # hand-frozen — refreshing it becomes "download these six artifacts from + # hand-frozen — refreshing it becomes "download every shard's artifact from # any green run and re-run the generator", with no special measurement # run and no CI time spent measuring. # @@ -1265,7 +1300,7 @@ jobs: continue-on-error: true uses: actions/upload-artifact@v7 with: - name: test-core-run-summary-${{ matrix.shard }}-of-6 + name: test-core-run-summary-${{ matrix.shard }}-of-8 path: .turbo/runs/ if-no-files-found: ignore retention-days: 1 @@ -1347,11 +1382,11 @@ jobs: shopt -s nullglob SUMMARIES=(.turbo/runs/*.json) if [ "${#SUMMARIES[@]}" -eq 0 ]; then - echo "shard-timing-drift: NOT MEASURED -- Test Core (${{ matrix.shard }}/6) wrote no turbo run summary (no packages on this shard)." + echo "shard-timing-drift: NOT MEASURED -- Test Core (${{ matrix.shard }}/8) wrote no turbo run summary (no packages on this shard)." exit 0 fi node scripts/partition-test-shards.mjs --check-drift "${SUMMARIES[@]}" \ - --label "Test Core (${{ matrix.shard }}/6)" --event "$GITHUB_EVENT_NAME" + --label "Test Core (${{ matrix.shard }}/8)" --event "$GITHUB_EVENT_NAME" # Runs even when the suite failed — that is when it earns its keep. It # answers TWO questions about a red suite, and needs both to be able to @@ -1423,7 +1458,7 @@ jobs: # # Cutting tests is decided per test file BY COST, so the cost has to be # visible first. This shard reads the timings it already printed and - # hands them up; `Test Core` merges the six and publishes the table. + # hands them up; `Test Core` merges every shard's and publishes the table. # # WHERE THE NUMBERS COME FROM, and why no second test run is needed: # per FILE the vitest module lines already in `test-core.log`. @@ -1481,7 +1516,7 @@ jobs: node scripts/report-test-timings.mjs --capture \ --log "$RUNNER_TEMP/test-core.log" \ --summaries .turbo/runs \ - --shard "${{ matrix.shard }}/6" \ + --shard "${{ matrix.shard }}/8" \ --out "$RUNNER_TEMP/test-timings/shard-${{ matrix.shard }}.json" # 1 day, matching the run-summary upload above: this is an intermediate @@ -1493,7 +1528,7 @@ jobs: continue-on-error: true uses: actions/upload-artifact@v7 with: - name: test-core-timings-${{ matrix.shard }}-of-6 + name: test-core-timings-${{ matrix.shard }}-of-8 path: ${{ runner.temp }}/test-timings/ if-no-files-found: ignore retention-days: 1 @@ -1529,7 +1564,7 @@ jobs: id: prune-turbo-cache if: always() && github.event_name == 'push' run: | - node scripts/ci/prune-turbo-cache.mjs --site 'Test Core (${{ matrix.shard }}/6)' \ + node scripts/ci/prune-turbo-cache.mjs --site 'Test Core (${{ matrix.shard }}/8)' \ --cache .turbo/cache --run-summaries .turbo/runs \ --extra-summaries "$RUNNER_TEMP/turbo-remote-cache" @@ -1556,13 +1591,13 @@ jobs: - name: Attest this shard ran and passed run: | node scripts/check-shard-attestation.mjs --emit \ - --job test --shard ${{ matrix.shard }} --total 6 \ + --job test --shard ${{ matrix.shard }} --total 8 \ --out "$RUNNER_TEMP/shard-attestation" - name: Publish this shard's attestation uses: actions/upload-artifact@v7 with: - name: shard-attest-test-${{ matrix.shard }}-of-6 + name: shard-attest-test-${{ matrix.shard }}-of-8 path: ${{ runner.temp }}/shard-attestation/ if-no-files-found: error retention-days: 1 @@ -1671,7 +1706,7 @@ jobs: --dir "$OS_ATTEST_DIR" \ --filter-result "$OS_FILTER_RESULT" \ --download-outcome "$OS_DOWNLOAD_OUTCOME" \ - --leg "test/6:$OS_TEST_RESULT" + --leg "test/8:$OS_TEST_RESULT" # ── The timing table this gate publishes (#16454) ───────────────────── # diff --git a/.github/workflows/shard-timings-refresh.yml b/.github/workflows/shard-timings-refresh.yml index 6af4162ccd1..0f0183abe45 100644 --- a/.github/workflows/shard-timings-refresh.yml +++ b/.github/workflows/shard-timings-refresh.yml @@ -7,8 +7,8 @@ # # scripts/test-shard-timings.json is the per-package duration dataset the Test # Core split is binned from. It is GENERATED — scripts/measure-test-shard- -# timings.mjs turns a green run's six turbo summaries into it — but until now it -# was generated only when a person remembered to. It went 13 days without a +# timings.mjs turns a green run's per-shard turbo summaries into it — but until +# now it was generated only when a person remembered to. It went 13 days without a # refresh while ~700 test files were added, and the shard it mis-weighted was # killed by the job wall twelve times in one day and ejected from the merge queue # twice (#16173). @@ -46,8 +46,9 @@ # # HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE # ---------------------------------------------------------------------------- -# The documented refresh path downloads the six `test-core-run-summary-N-of-6` -# artifacts of a green run. `GET /actions/artifacts/{id}/zip` redirects to +# The documented refresh path downloads the `test-core-run-summary-N-of-M` +# artifacts of a green run, one per Test Core shard (M = SHARD_COUNT). +# `GET /actions/artifacts/{id}/zip` redirects to # `productionresultssa*.blob.core.windows.net`, and every agent container's # egress policy answers 403 to CONNECT for that host — reproduced four ways, on # two artifacts across two runs, on the pre-signed URL as well as the API one, on @@ -71,8 +72,8 @@ # Until #16467 a push to `main` re-ran the FULL Test Core battery, so "a green # push run on main" and "a measurement of the workspace" were the same thing. # They are not any more: a push run computes its package set with `--affected` -# against `github.event.before`. Its six shards still conclude `success` and -# still upload six run summaries, so NOTHING in the eligibility test would have +# against `github.event.before`. Its shards still conclude `success` and +# still upload every run summary, so NOTHING in the eligibility test would have # noticed — this lane would have kept regenerating the balancing dataset from # measurements of whatever the last merge happened to touch, and the coverage # check would not have caught it either (a package the affected set skipped is a @@ -96,8 +97,8 @@ # follows from the cache design rather than from luck — turbo's key is # namespaced per shard and only main pushes write it, so a package whose inputs # have not changed is a HIT, and the generator refuses hits rather than -# recording a replayed ~0.1s window as a suite's cost. "Download six artifacts -# from any green run" therefore measures a SLICE of the workspace. That number +# recording a replayed ~0.1s window as a suite's cost. "Download the shard +# artifacts from any green run" therefore measures a SLICE of the workspace. That number # should IMPROVE under the hourly run, because a push no longer seeds the whole # workspace's entries — but it is quoted as it was measured, and the next live # run is what re-measures it. @@ -178,7 +179,7 @@ # - DOWNS, in one line. # - FILES OVER BUDGET: every test file over 10 s of wall in the window, with # its package and its slowest run (#22998). The dataset holds no per-file -# reading; the window's runs do, in the per-shard `test-core-timings-N-of-6` +# reading; the window's runs do, in the per-shard `test-core-timings-N-of-M` # captures, which a step before the body downloads: the same runs, retained # exactly as long as their run summaries. # @@ -432,14 +433,14 @@ jobs: # not bad luck, it is the cache design: turbo's key is namespaced per shard # and only main pushes write it, so a package whose inputs have not changed # is a HIT — and the generator refuses hits rather than recording a - # replayed ~0.1s window as a suite's cost. "Download six artifacts from any - # green run" therefore measures a SLICE of the workspace, never all of it. + # replayed ~0.1s window as a suite's cost. "Download the shard artifacts + # from any green run" therefore measures a SLICE of the workspace, never all of it. # Since #16467 the runs read here are the HOURLY `schedule` runs, which are # the full battery; the accumulation stays because the cache argument above # is unchanged by which event ran the suite. # - # So runs are accumulated. Each contributes its six summaries under its own - # `--run ` group, which is exactly the grouping #16473 added: slices are + # So runs are accumulated. Each contributes its shards' summaries under its + # own `--run ` group, which is exactly the grouping #16473 added: slices are # summed within a run and the per-run sums are medianed across runs, so a # package measured by three of these runs gets a median of three # observations rather than whichever run happened to be read last. Feeding @@ -482,7 +483,12 @@ jobs: rm -rf "$SUMDIR"; mkdir -p "$SUMDIR" OK=1 - for N in 1 2 3 4 5 6; do + # One summary per shard the selector found: it admits a run only when + # every one of SHARD_COUNT (select-shard-timings-run.mjs) shards + # concluded success with its artifact retained, so these keys ARE the + # roster. A count spelled here would be a third copy of it (#25376). + SHARD_NS=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(Object.keys(r.artifact_ids).map(Number).sort((a,b)=>a-b).join(" "))' "$i") + for N in $SHARD_NS; do ART=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(r.artifact_ids[process.argv[2]])' "$i" "$N") # curl drops the Authorization header across the redirect to the # blob host, which is correct: that URL is pre-signed and the @@ -636,7 +642,7 @@ jobs: # The per-file half of the regression readings (#22949 ruling E, item 3). # The dataset holds no per-file durations; the window's runs do, in the # per-shard captures `report-test-timings.mjs --capture` uploads as - # `test-core-timings-N-of-6`, retained 1 day like the run summaries the + # `test-core-timings-N-of-M`, retained 1 day like the run summaries the # generator read. The selector listed their ids beside the summaries' # (`timing_artifact_ids`), so this is a read of the SAME runs, never a new # measurement, and it feeds the PR body alone. @@ -822,7 +828,8 @@ jobs: const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8")); const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID); const s = r.shard_seconds ?? {}; - console.log([1,2,3,4,5,6].map((n) => (s[n] == null ? "?" : `${n}: ${s[n]}s`)).join(" | ")); + const ns = Object.keys(r.artifact_ids ?? {}).map(Number).sort((a, b) => a - b); + console.log(ns.map((n) => (s[n] == null ? "?" : `${n}: ${s[n]}s`)).join(" | ")); ') { @@ -851,8 +858,8 @@ jobs: done echo echo "- Newest run in the set: \`$RUN_ID\`, commit \`$HEAD_SHA\` — the date this refresh carries." - echo "- Every run above had all six \`Test Core (N/6)\` jobs conclude \`success\` with its six" - echo " run-summary artifacts still retained; runs that were cancelled, failed or had lost" + echo "- Every run above had every \`Test Core (N/M)\` shard job conclude \`success\` with its" + echo " run-summary artifact still retained; runs that were cancelled, failed or had lost" echo " their artifacts were rejected by name in the log before any of these were used." echo echo "$CARRY_LINE" @@ -886,7 +893,7 @@ jobs: echo "## The partitioner's own pins RED on this refresh — read this before merging" echo echo "This is the designed behaviour, not a defect in the refresh: the acceptance bound is" - echo "a ratio, and a package that has grown past what any six-way split can bin makes the" + echo "a ratio, and a package that has grown past what the Test Core split can bin makes the" echo "pins fail with the arithmetic in the message. The remedy the partitioner names is to" echo "raise the file-level slice count for that package — ⛔ never to raise the bound, and" echo "⛔ never to hand-edit this dataset. This workflow deliberately does neither: it" diff --git a/scripts/check-required-contexts.mjs b/scripts/check-required-contexts.mjs index 81b3e2c7fc7..691405aac24 100644 --- a/scripts/check-required-contexts.mjs +++ b/scripts/check-required-contexts.mjs @@ -2026,13 +2026,13 @@ async function selfTest() { ); // ── (9) the shadowing collision, on the live specimen ──────────────────── - // ci.yml's sharded `test` job is named `Test Core (${{ matrix.shard }}/6)` + // ci.yml's sharded `test` job is named `Test Core (${{ matrix.shard }}/8)` // and its aggregate gate is named `Test Core`. Dropping the suffix makes two // jobs publish one context, and the surviving conclusion is whichever // finished last — a shard could satisfy the aggregate's required gate. battery('(9) the shadowing collision, on the live specimen'); const collided = fixture('collide the shard name with the gate name', 'ci.yml', (s) => - s.replace('name: Test Core (${{ matrix.shard }}/6)', 'name: Test Core'), + s.replace('name: Test Core (${{ matrix.shard }}/8)', 'name: Test Core'), ); assert( collided.problems.some((p) => p.includes("job 'test'") && p.includes("published by ci.yml:test-gate")), @@ -2041,7 +2041,7 @@ async function selfTest() { // The suffixed spelling is NOT a collision — the guard must not read a // prefix as a clash, or ci.yml is red on main today. assert( - baseline.problems.length === 0 && sources['ci.yml'].includes('name: Test Core (${{ matrix.shard }}/6)'), + baseline.problems.length === 0 && sources['ci.yml'].includes('name: Test Core (${{ matrix.shard }}/8)'), 'the real suffixed shard name coexists with the bare gate name (assertion 9 compares whole names, not prefixes)', ); diff --git a/scripts/check-shard-attestation.mjs b/scripts/check-shard-attestation.mjs index 28d73901a2f..d6121743050 100644 --- a/scripts/check-shard-attestation.mjs +++ b/scripts/check-shard-attestation.mjs @@ -2085,6 +2085,13 @@ async function selfTest() { // ── #22653: the per-shard cause read itself, offline ───────────────────── battery('#22653 the cause read, offline: one jobs list, then one annotations read per unattested shard'); const CI_TEXT = readFileSync(join(scriptRepoRoot(), '.github', 'workflows', 'ci.yml'), 'utf8'); + // The runs replayed below (38036705318, 34007386254) ran the SIX-shard matrix, + // which #25376 widened to eight, so they are read against the live workflow + // with the shard name as it stood then. The live names are pinned on their own + // at the end of this battery. + const LIVE_TEST_NAME = 'name: Test Core (${{ matrix.shard }}/8)'; + const CI_TEXT_SIX_SHARDS = CI_TEXT.replace(LIVE_TEST_NAME, 'name: Test Core (${{ matrix.shard }}/6)'); + assert(CI_TEXT_SIX_SHARDS !== CI_TEXT, `#22653: the live ci.yml no longer carries \`${LIVE_TEST_NAME}\` — the six-shard replays below would read the live roster`); const PARTIAL_JOBS = /\/repos\/o\/r\/actions\/runs\/38036705318\/attempts\/1\/jobs\?per_page=100&page=1$/; const ANY_ANNOTATIONS = /\/check-runs\/\d+\/annotations\?per_page=100$/; const shardCheckRun = (n) => 114168570850 + n; @@ -2109,7 +2116,7 @@ async function selfTest() { apiUrl: API, token: 'not-a-real-token', fetchJson, - workflowText: CI_TEXT, + workflowText: CI_TEXT_SIX_SHARDS, ...over, }); @@ -2177,7 +2184,7 @@ async function selfTest() { '#22653: a matrix name with no shard expression cannot locate a shard ⇒ unreadable, naming the shard', ); - assert([1, 2, 3, 4, 5, 6].every((n) => workflowJobDisplayName(CI_TEXT, 'test', n) === `Test Core (${n}/6)`), '#22653: the real ci.yml names each Test Core shard as the jobs API lists it'); + assert([1, 2, 3, 4, 5, 6, 7, 8].every((n) => workflowJobDisplayName(CI_TEXT, 'test', n) === `Test Core (${n}/8)`), '#22653: the real ci.yml names each Test Core shard as the jobs API lists it'); assert([1, 2, 3].every((n) => workflowJobDisplayName(CI_TEXT, 'dogfood', n) === `Dogfood Regression Gate (${n}/3)`), '#22653: …and each dogfood shard'); assert(workflowJobDisplayName(CI_TEXT, 'dogfood-verify') === 'Dogfood Verify CLI', "#22653 control: the single leg's literal name reads as before"); assert(workflowJobDisplayName(CI_TEXT, 'test') === undefined, '#22653: a matrix template asked WITHOUT a shard is still unresolvable ⇒ undefined'); @@ -2480,8 +2487,8 @@ async function selfTest() { // `String.replace` takes the first match. A short anchor would silently // start mutating dogfood's matrix instead — still red, still for a // plausible-looking reason, and no longer the assertion written here. - assert((await fixture('grow the matrix', (s) => s.replace('shard: [1, 2, 3, 4, 5, 6]', 'shard: [1, 2, 3, 4, 5, 6, 7]'))).problems.some((p) => p.includes('declares 7 shard')), 'growing the matrix without the gate ⇒ red'); - assert((await fixture('shrink the roster', (s) => s.replace('--leg "test/6', '--leg "test/2'))).problems.some((p) => p.includes('expects 2 attestation')), 'shrinking the gate roster without the matrix ⇒ red'); + assert((await fixture('grow the matrix', (s) => s.replace('shard: [1, 2, 3, 4, 5, 6, 7, 8]', 'shard: [1, 2, 3, 4, 5, 6, 7, 8, 9]'))).problems.some((p) => p.includes('declares 9 shard')), 'growing the matrix without the gate ⇒ red'); + assert((await fixture('shrink the roster', (s) => s.replace('--leg "test/8', '--leg "test/2'))).problems.some((p) => p.includes('expects 2 attestation')), 'shrinking the gate roster without the matrix ⇒ red'); assert((await fixture('uncounted intruder', (s) => s.replace(/^jobs:$/m, 'jobs:\n intruder:\n runs-on: ubuntu-latest\n steps:\n - run: node scripts/check-shard-attestation.mjs --emit'))).problems.some((p) => p.includes('no gate counts')), 'an attestation no gate counts ⇒ red'); assert((await fixture('rename test-gate', (s) => s.replace(' test-gate:', ' test-gate-renamed:'))).problems.some((p) => p.includes("'test-gate'")), 'losing a required-context gate ⇒ red'); assert((await fixture('typo the artifact name', (s) => s.replace('name: shard-attest-test-', 'name: shard-attest-typo-'))).problems.some((p) => p.includes('cannot count')), 'an artifact name the gate cannot match ⇒ red'); @@ -2573,13 +2580,13 @@ async function selfTest() { hasProblem(await fixture('drop the token (test-gate)', inJob('test-gate', TOKEN_LINE, '')), "gate 'test-gate'", 'GITHUB_TOKEN'), "#22653: test-gate's verify step without GITHUB_TOKEN ⇒ red", ); - const TEST_NAME = ' name: Test Core (${{ matrix.shard }}/6)\n'; + const TEST_NAME = ' name: Test Core (${{ matrix.shard }}/8)\n'; assert( hasProblem(await fixture('a matrix name without the shard expression', (s) => s.replace(TEST_NAME, ' name: Test Core shards\n')), '#22653', "job 'test' is named"), '#22653: a matrix name with no `${{ matrix.shard }}` ⇒ red — the platform would append the values itself, a name the reader cannot rebuild', ); assert( - hasProblem(await fixture('a matrix name with a second expression', (s) => s.replace(TEST_NAME, ' name: Test Core (${{ matrix.shard }}/6) ${{ github.event_name }}\n')), '#22653', "job 'test' is named"), + hasProblem(await fixture('a matrix name with a second expression', (s) => s.replace(TEST_NAME, ' name: Test Core (${{ matrix.shard }}/8) ${{ github.event_name }}\n')), '#22653', "job 'test' is named"), '#22653: any expression besides the shard ⇒ red — the text reading cannot evaluate it', ); assert( @@ -2591,11 +2598,11 @@ async function selfTest() { "#22653: dogfood's matrix leg is held to the same name rule — it is asked on a partial cancel too", ); assert( - (await fixture('the matrix name double-quoted', (s) => s.replace(TEST_NAME, ' name: "Test Core (${{ matrix.shard }}/6)"\n'))).problems.length === 0, + (await fixture('the matrix name double-quoted', (s) => s.replace(TEST_NAME, ' name: "Test Core (${{ matrix.shard }}/8)"\n'))).problems.length === 0, '#22653 positive limb: a quoted template reads the same from text and from YAML ⇒ green', ); assert( - (await fixture('the shard expression without inner spaces', (s) => s.replace(TEST_NAME, ' name: Test Core (${{matrix.shard}}/6)\n'))).problems.length === 0, + (await fixture('the shard expression without inner spaces', (s) => s.replace(TEST_NAME, ' name: Test Core (${{matrix.shard}}/8)\n'))).problems.length === 0, '#22653 positive limb: `${{matrix.shard}}` is the same expression ⇒ green', ); assert(baseline.problems.length === 0 && baseline.gateIds.includes('test-gate'), '#22653: the checked-in test-gate carries every prerequisite — the baseline is green with it counted as a gate'); diff --git a/scripts/ci/select-shard-timings-run.mjs b/scripts/ci/select-shard-timings-run.mjs index 615e43995de..0db4d3b5447 100644 --- a/scripts/ci/select-shard-timings-run.mjs +++ b/scripts/ci/select-shard-timings-run.mjs @@ -112,7 +112,7 @@ import { isEntrypoint } from '../invoked-as.mjs'; // complete run must present; partition-test-shards.mjs owns the split itself and // its own self-test reads ci.yml back, so this is a reader's expectation rather // than a second declaration of the split. -export const SHARD_COUNT = 6; +export const SHARD_COUNT = 8; const API = 'https://api.github.com'; @@ -121,8 +121,8 @@ const API = 'https://api.github.com'; // and would pass every eligibility and coverage check while measuring a slice. export const DEFAULT_RUN_EVENT = 'schedule'; -// `Test Core (3/6)` -> 3. Anchored on both ends: a job merely CONTAINING that -// text (a future "Test Core (3/6) rerun") is not this job, and reading it as one +// `Test Core (3/8)` -> 3. Anchored on both ends: a job merely CONTAINING that +// text (a future "Test Core (3/8) rerun") is not this job, and reading it as one // would let a run qualify on a job that never ran the suite. export function testCoreShard(jobName, shardCount = SHARD_COUNT) { const m = /^Test Core \((\d+)\/(\d+)\)$/.exec(String(jobName ?? '').trim()); @@ -132,7 +132,7 @@ export function testCoreShard(jobName, shardCount = SHARD_COUNT) { return index >= 1 && index <= shardCount ? index : null; } -// `test-core-run-summary-3-of-6` -> 3, by the same anchoring argument. +// `test-core-run-summary-3-of-8` -> 3, by the same anchoring argument. export function summaryArtifactShard(artifactName, shardCount = SHARD_COUNT) { const m = /^test-core-run-summary-(\d+)-of-(\d+)$/.exec(String(artifactName ?? '').trim()); if (!m) return null; @@ -141,7 +141,7 @@ export function summaryArtifactShard(artifactName, shardCount = SHARD_COUNT) { return index >= 1 && index <= shardCount ? index : null; } -// `test-core-timings-3-of-6` -> 3: the shard's per-FILE capture +// `test-core-timings-3-of-8` -> 3: the shard's per-FILE capture // (`report-test-timings.mjs --capture`), anchored like the summary matcher. // NOT an eligibility input -- the capture step is report-only // (`continue-on-error`), so a run can be a complete measurement without one -- @@ -590,28 +590,28 @@ async function selfTest() { // differently-sharded artifact would qualify a run on evidence from a run // that is not this one. check(() => { - if (testCoreShard('Test Core (3/6)') !== 3) throw new Error('job matcher: the plain shape did not parse'); + if (testCoreShard('Test Core (3/8)') !== 3) throw new Error('job matcher: the plain shape did not parse'); }); check(() => { - if (testCoreShard('Test Core (1/6)') !== 1 || testCoreShard('Test Core (6/6)') !== 6) { + if (testCoreShard('Test Core (1/8)') !== 1 || testCoreShard('Test Core (8/8)') !== 8) { throw new Error('job matcher: an endpoint shard did not parse'); } }); check(() => { - if (testCoreShard('Test Core (0/6)') !== null || testCoreShard('Test Core (7/6)') !== null) { + if (testCoreShard('Test Core (0/8)') !== null || testCoreShard('Test Core (9/8)') !== null) { throw new Error('job matcher: an out-of-range shard index was accepted'); } }); check(() => { - if (testCoreShard('Test Core (3/6) rerun') !== null) { + if (testCoreShard('Test Core (3/8) rerun') !== null) { throw new Error('job matcher: a job merely CONTAINING the shape was accepted'); } }); check(() => { - if (testCoreShard('Dogfood (3/6)') !== null) throw new Error('job matcher: a different job was accepted'); + if (testCoreShard('Dogfood (3/8)') !== null) throw new Error('job matcher: a different job was accepted'); }); check(() => { - if (testCoreShard('Test Core (3/8)') !== null) { + if (testCoreShard('Test Core (3/6)') !== null) { throw new Error('job matcher: a job from a different shard count was accepted'); } }); @@ -621,28 +621,28 @@ async function selfTest() { } }); check(() => { - if (summaryArtifactShard('test-core-run-summary-4-of-6') !== 4) { + if (summaryArtifactShard('test-core-run-summary-4-of-8') !== 4) { throw new Error('artifact matcher: the plain shape did not parse'); } }); check(() => { - if (summaryArtifactShard('test-core-run-summary-4-of-8') !== null) { + if (summaryArtifactShard('test-core-run-summary-4-of-6') !== null) { throw new Error('artifact matcher: a different shard count was accepted'); } }); check(() => { - if (summaryArtifactShard('test-core-run-summary-4-of-6-retry') !== null) { + if (summaryArtifactShard('test-core-run-summary-4-of-8-retry') !== null) { throw new Error('artifact matcher: a longer name was accepted'); } }); // #22949 ruling E: the per-file capture matcher, anchored the same way -- // and NOT the run-summary name, which is a different artifact. check(() => { - if (timingCaptureShard('test-core-timings-2-of-6') !== 2) throw new Error('capture matcher: the plain shape did not parse'); - if (timingCaptureShard('test-core-timings-2-of-8') !== null || timingCaptureShard('test-core-timings-2-of-6-retry') !== null) { + if (timingCaptureShard('test-core-timings-2-of-8') !== 2) throw new Error('capture matcher: the plain shape did not parse'); + if (timingCaptureShard('test-core-timings-2-of-6') !== null || timingCaptureShard('test-core-timings-2-of-8-retry') !== null) { throw new Error('capture matcher: a different shard count or a longer name was accepted'); } - if (timingCaptureShard('test-core-run-summary-2-of-6') !== null || summaryArtifactShard('test-core-timings-2-of-6') !== null) { + if (timingCaptureShard('test-core-run-summary-2-of-8') !== null || summaryArtifactShard('test-core-timings-2-of-8') !== null) { throw new Error('capture matcher: the run summary and the per-file capture were read as each other'); } }); @@ -650,69 +650,69 @@ async function selfTest() { // -- Eligibility. The fixtures are built from the two rejection shapes this // workflow actually meets: a cancelled shard, and an expired artifact. const jobs = (conclusions) => - conclusions.map((conclusion, i) => ({ name: `Test Core (${i + 1}/6)`, conclusion, status: 'completed' })); + conclusions.map((conclusion, i) => ({ name: `Test Core (${i + 1}/8)`, conclusion, status: 'completed' })); const artifacts = (present, expired = []) => - present.map((n) => ({ id: 1000 + n, name: `test-core-run-summary-${n}-of-6`, expired: expired.includes(n) })); - const allSix = ['success', 'success', 'success', 'success', 'success', 'success']; + present.map((n) => ({ id: 1000 + n, name: `test-core-run-summary-${n}-of-8`, expired: expired.includes(n) })); + const allEight = ['success', 'success', 'success', 'success', 'success', 'success', 'success', 'success']; check(() => { - const v = runIsEligible({ jobs: jobs(allSix), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const v = runIsEligible({ jobs: jobs(allEight), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (!v.eligible) throw new Error(`eligibility: a complete run was rejected (${v.reasons.join('; ')})`); }); check(() => { - const v = runIsEligible({ jobs: jobs(allSix), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const v = runIsEligible({ jobs: jobs(allEight), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (v.artifactIds['5'] !== 1005) throw new Error(`eligibility: artifact ids were not returned per shard (${JSON.stringify(v.artifactIds)})`); }); check(() => { // #22949 ruling E: the per-file captures ride along per shard, an expired // one is left out -- and a run WITHOUT them is still eligible, because the // capture step is report-only and the dataset never reads it. - const captures = [1, 2, 3, 4, 5].map((n) => ({ id: 2000 + n, name: `test-core-timings-${n}-of-6`, expired: n === 2 })); - const v = runIsEligible({ jobs: jobs(allSix), artifacts: [...artifacts([1, 2, 3, 4, 5, 6]), ...captures] }); + const captures = [1, 2, 3, 4, 5].map((n) => ({ id: 2000 + n, name: `test-core-timings-${n}-of-8`, expired: n === 2 })); + const v = runIsEligible({ jobs: jobs(allEight), artifacts: [...artifacts([1, 2, 3, 4, 5, 6, 7, 8]), ...captures] }); if (!v.eligible || JSON.stringify(v.timingArtifactIds) !== '{"1":2001,"3":2003,"4":2004,"5":2005}') { throw new Error(`eligibility: the per-file captures read ${JSON.stringify(v.timingArtifactIds)} (eligible ${v.eligible})`); } - const none = runIsEligible({ jobs: jobs(allSix), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const none = runIsEligible({ jobs: jobs(allEight), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (!none.eligible || Object.keys(none.timingArtifactIds).length !== 0) { throw new Error('eligibility: a run without per-file captures was rejected, or invented some'); } }); check(() => { - const cancelled = [...allSix]; + const cancelled = [...allEight]; cancelled[2] = 'cancelled'; - const v = runIsEligible({ jobs: jobs(cancelled), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const v = runIsEligible({ jobs: jobs(cancelled), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (v.eligible) throw new Error('eligibility: a run with a CANCELLED shard was accepted'); - if (!v.reasons.some((r) => r.includes('3/6'))) throw new Error(`eligibility: the cancelled shard was not named (${v.reasons.join('; ')})`); + if (!v.reasons.some((r) => r.includes('3/8'))) throw new Error(`eligibility: the cancelled shard was not named (${v.reasons.join('; ')})`); }); check(() => { // A cancelled shard that still uploaded its artifact is the exact censored // shape: "the artifacts exist" must not be enough on its own. - const cancelled = [...allSix]; + const cancelled = [...allEight]; cancelled[0] = 'cancelled'; - const v = runIsEligible({ jobs: jobs(cancelled), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const v = runIsEligible({ jobs: jobs(cancelled), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (v.eligible) throw new Error('eligibility: a censored run passed because its artifacts were complete'); }); check(() => { - const stillRunning = [...allSix]; + const stillRunning = [...allEight]; stillRunning[4] = null; - const v = runIsEligible({ jobs: jobs(stillRunning), artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const v = runIsEligible({ jobs: jobs(stillRunning), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (v.eligible) throw new Error('eligibility: a shard with no conclusion was read as a success'); }); check(() => { - const v = runIsEligible({ jobs: jobs(allSix), artifacts: artifacts([1, 2, 3, 4, 5]) }); + const v = runIsEligible({ jobs: jobs(allEight), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7]) }); if (v.eligible) throw new Error('eligibility: a run missing an artifact was accepted'); - if (!v.reasons.some((r) => r.includes('6-of-6'))) throw new Error(`eligibility: the missing artifact was not named (${v.reasons.join('; ')})`); + if (!v.reasons.some((r) => r.includes('8-of-8'))) throw new Error(`eligibility: the missing artifact was not named (${v.reasons.join('; ')})`); }); check(() => { - const v = runIsEligible({ jobs: jobs(allSix), artifacts: artifacts([1, 2, 3, 4, 5, 6], [2]) }); + const v = runIsEligible({ jobs: jobs(allEight), artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8], [2]) }); if (v.eligible) throw new Error('eligibility: an EXPIRED artifact was counted as retained'); }); check(() => { // A re-run leaves a failed attempt beside the successful one. The shard // qualifies on the attempt that succeeded, which is also the one that // uploaded the artifact. - const withRetry = [...jobs(allSix), { name: 'Test Core (2/6)', conclusion: 'failure', status: 'completed' }]; - const v = runIsEligible({ jobs: withRetry, artifacts: artifacts([1, 2, 3, 4, 5, 6]) }); + const withRetry = [...jobs(allEight), { name: 'Test Core (2/8)', conclusion: 'failure', status: 'completed' }]; + const v = runIsEligible({ jobs: withRetry, artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }); if (!v.eligible) throw new Error(`eligibility: a re-run's earlier failed attempt disqualified the shard (${v.reasons.join('; ')})`); }); @@ -872,8 +872,8 @@ async function selfTest() { const SCHEDULE_RUN_ID = 900; const PUSH_RUN_ID = 100; const eligiblePayload = { - jobs: { jobs: jobs(allSix) }, - artifacts: { artifacts: artifacts([1, 2, 3, 4, 5, 6]) }, + jobs: { jobs: jobs(allEight) }, + artifacts: { artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }, }; const makeFakeApi = () => { const requested = []; @@ -1007,11 +1007,11 @@ async function selfTest() { return { ok: true, status: 200, statusText: 'OK', json: async () => ({ workflow_runs: runsFor(count) }) }; } if (/\/actions\/runs\/\d+\/jobs$/.test(pathname)) { - const conclusions = shape === 'eligible' ? allSix : allSix.map(() => 'cancelled'); + const conclusions = shape === 'eligible' ? allEight : allEight.map(() => 'cancelled'); return { ok: true, status: 200, statusText: 'OK', json: async () => ({ jobs: jobs(conclusions) }) }; } if (/\/actions\/runs\/\d+\/artifacts$/.test(pathname)) { - return { ok: true, status: 200, statusText: 'OK', json: async () => ({ artifacts: artifacts([1, 2, 3, 4, 5, 6]) }) }; + return { ok: true, status: 200, statusText: 'OK', json: async () => ({ artifacts: artifacts([1, 2, 3, 4, 5, 6, 7, 8]) }) }; } return { ok: false, status: 404, statusText: 'Not Found', json: async () => ({}) }; }; @@ -1285,7 +1285,7 @@ async function selfTest() { const toolPath = (rel) => fileURLToPath(new URL(rel, import.meta.url)); // runs: newest first, each { id, tasks: [[package, seconds, cacheStatus]] }. - // Every task lands in shard 1's summary; shards 2-6 carry an empty task list, + // Every task lands in shard 1's summary; shards 2-8 carry an empty task list, // which is a valid summary that measures nothing. const driveGenerate = (runs, committedPackages, workspace = Object.keys(committedPackages)) => { const dir = mkdtempSync(join(tmpdir(), 'os-timings-generate-')); @@ -1306,11 +1306,11 @@ async function selfTest() { run_id: r.id, head_sha: String(r.id).padStart(40, 'c'), created_at: '2026-10-06T00:00:00Z', - artifact_ids: Object.fromEntries([1, 2, 3, 4, 5, 6].map((n) => [n, Number(`${r.id}${n}`)])), + artifact_ids: Object.fromEntries([1, 2, 3, 4, 5, 6, 7, 8].map((n) => [n, Number(`${r.id}${n}`)])), })); writeFileSync(join(runnerTemp, 'candidates.json'), JSON.stringify(candidatesList)); for (const r of runs) { - for (const n of [1, 2, 3, 4, 5, 6]) { + for (const n of [1, 2, 3, 4, 5, 6, 7, 8]) { const tasks = n !== 1 ? [] diff --git a/scripts/measure-test-shard-timings.mjs b/scripts/measure-test-shard-timings.mjs index 5e2b82584d3..a461cc4ffeb 100644 --- a/scripts/measure-test-shard-timings.mjs +++ b/scripts/measure-test-shard-timings.mjs @@ -27,7 +27,7 @@ // // From CI, no special run needed. Every Test Core shard passes --summarize // and, on merge_group builds, uploads `.turbo/runs/` as the -// `test-core-run-summary--of-6` artifact. +// `test-core-run-summary--of-8` artifact. // // ⚠ ONE RUN IS NOT ENOUGH, AND THIS SENTENCE USED TO SAY IT WAS. It read // "download all six from any green queue build and re-run the generator", @@ -986,7 +986,7 @@ export function renderSpread({ bySample, dataset }) { // if it were. The dataset holds per-PACKAGE weights only; the `Slowest // file:` line the ceiling check prints comes from the per-shard captures // `report-test-timings.mjs --capture` writes in every Test Core shard and -// uploads as `test-core-timings--of-6` -- the SAME runs this refresh reads, +// uploads as `test-core-timings--of-8` -- the SAME runs this refresh reads, // retained exactly as long as their run summaries (1 day). So the refresh lane // downloads those captures beside the summaries and this reads them: still a // read of an existing measurement, never a second test run. A run whose @@ -1558,7 +1558,7 @@ export function renderRefreshDiff({ diff, files = null, filesNotMeasured = null, ? ` No capture arrived for ${files.runsWithoutCapture.map((r) => `\`${r}\``).join(', ')}, so its files are not in the list.` : ''; out.push( - `**Per-file reading:** the window's per-shard captures (\`test-core-timings-N-of-6\`, written by ` + + `**Per-file reading:** the window's per-shard captures (\`test-core-timings-N-of-8\`, written by ` + `\`report-test-timings.mjs --capture\`), read for ${files.runsRead} of ${files.runsRead + files.runsWithoutCapture.length} run(s): ` + `${files.measuredRows} file line(s) measured here; ${files.replayedRows} replayed line(s) left out, because a cache ` + `HIT replays a duration measured outside this window.${missing}` @@ -3573,7 +3573,7 @@ function main() { refresh: 'node scripts/measure-test-shard-timings.mjs [--run ] ... ' + '--merge-into scripts/test-shard-timings.json --out scripts/test-shard-timings.json ' + - '(summaries: the `test-core-run-summary--of-6` artifacts of any green run, or a local ' + + '(summaries: the `test-core-run-summary--of-8` artifacts of any green run, or a local ' + '`pnpm exec turbo run test --concurrency=4 --summarize`. ⚠ ONE RUN COVERS ONLY 2-52 of ~71 ' + 'packages depending on cache warmth, so feed SEVERAL runs -- a `--run ` before each ' + 'run\'s summaries is REQUIRED, so a sliced package is assembled per run and then medianed ' + diff --git a/scripts/partition-test-shards.mjs b/scripts/partition-test-shards.mjs index cf45bf55c72..9ebc0474a3e 100644 --- a/scripts/partition-test-shards.mjs +++ b/scripts/partition-test-shards.mjs @@ -59,9 +59,12 @@ // ratio is what the acceptance bound is written in, so "add shards until it // balances" is not merely ineffective here -- it moves the number the wrong // way while looking like progress. #10472 asked for 6 -> 8 -// to be considered and the measured answer was no -- see SHARD_COUNT below, -// where the self-test now pins that arithmetic so the next person gets the -// answer from a failing assertion instead of from a CI run. +// to be considered and the measured answer was no, because spec's 1147s was +// then the floor. #25376 asked again once the CLI was sliced (#22075) and spec's +// 842s repo file was split (#25372), and the answer became yes -- see +// SHARD_COUNT below for both readings. The self-test pins that arithmetic, so +// the next person gets the answer from a failing assertion instead of from a +// CI run. // // Run-to-run variance is a SEPARATE and equally large effect, and it is not // placement: this job's Turbo cache key is namespaced per shard @@ -120,11 +123,39 @@ const TIMINGS_PATH = path.join(REPO_ROOT, 'scripts', 'test-shard-timings.json'); // The Test Core shard count, and the ONLY place it is reasoned about. // ci.yml spells it three more times (the matrix, the `name:`, and `--shard -// N/6`); the self-test below reads that workflow back and fails on drift, +// N/8`); the self-test below reads that workflow back and fails on drift, // because a partitioner splitting into a different number of bins than the // matrix declares is not a red step anywhere -- it is packages that no shard -// runs. -export const SHARD_COUNT = 6; +// runs. scripts/ci/select-shard-timings-run.mjs spells it once more, as the +// number of jobs and artifacts a complete run presents to the timings refresh. +// +// WHY EIGHT (#25376, 6 -> 8). On the committed dataset (the 2026-10-23 +// refresh), bins of predicted test windows, the CLI cut into its 3 slices: +// +// 6 shards -> max 2489s mean 2488s ratio 1.00x +// 8 shards -> max 1868s mean 1866s ratio 1.00x +// 9 shards -> max 1661s mean 1659s ratio 1.00x +// 10 shards -> max 1607s mean 1493s ratio 1.08x <- spec's whole weight is the floor +// +// Eight keeps one count of headroom below the point where the floor binds. Bin +// sums are not wall, though: whole packages share a shard four at a time while +// a CLI slice runs as a serial leg of its own (ci.yml, "Run this shard's +// tests"), so a shard's step reads close to wholeSum / 2.9 + slice. Calibrated +// on merge_group run 40155484566 (59 packages executed, the CLI as 3 slices), +// where it lands within 6.1% of the measured step on the four slowest shards +// (the three carrying a slice and the heaviest whole-package one; it misreads +// the two light shards by up to 24%, which never set the wall), and replayed on +// that run's own list through planRun(): +// +// 6 shards -> slowest shard ~1696s (measured 1794s, the shard with a slice) +// 8 shards -> slowest shard ~1436s, -15% +// +// The same model gives -14% on the dataset's full list and -10% on a 21-package +// merge_group run (40157571602), whose CLI slices at 8 carry almost nothing +// beside them. That last figure is the floor this count cannot move: a CLI +// slice (~960-1010s measured) is serial, so past eight the slowest shard is +// the slice and more shards only add runners. +export const SHARD_COUNT = 8; // The acceptance bound #10472 set: the slowest shard within ~1.3x the mean. // It is a RATIO, not a wall-clock target, which is what makes it stable as the diff --git a/scripts/pm/check-expected-skips.mjs b/scripts/pm/check-expected-skips.mjs index 114b35ed6c2..172476f8f55 100644 --- a/scripts/pm/check-expected-skips.mjs +++ b/scripts/pm/check-expected-skips.mjs @@ -314,7 +314,7 @@ export const EXPECTED_SKIPS = Object.freeze([ reason: `${FULL_BATTERY_REASON}; on a push it also needs the \`filter\` job's \`core\` output`, }, { - name: 'Test Core (${{ matrix.shard }}/6)', + name: 'Test Core (${{ matrix.shard }}/8)', workflow: 'ci.yml', job: 'test', gate: { kind: 'filter-output', outputs: ['core', 'crosspkg'] }, @@ -1249,7 +1249,9 @@ function fixtureRun(name, conclusion, suite = 1, extra = {}) { * Measured 2026-09-16 on PR #18315's head `b671f83b` (the AGENTS.md-only * landing): 39 check-runs, 20 success / 19 skipped, four `PR Automation` runs * on one head. Names, conclusions and check-suite ids VERBATIM from the - * listing. This is the real shape the roster must accept, kept here so the + * listing, but for one name: the Test Core matrix template read `/6` then and + * is re-spelled `/8` here, the matrix #25376 widened, because the roster + * grades the live name. This is the real shape the roster must accept, kept here so the * measured family and the declared family cannot drift apart unnoticed. */ const MEASURED_18315 = [ @@ -1285,7 +1287,7 @@ const MEASURED_18315 = [ ['Part-of PR must not also close its card', 'success', 94769267707], ['Temporal Conformance (live PG + MySQL)', 'skipped', 94769267631], ['Test Core', 'success', 94769267631], - ['Test Core (${{ matrix.shard }}/6)', 'skipped', 94769267631], + ['Test Core (${{ matrix.shard }}/8)', 'skipped', 94769267631], ['The card this PR closes must claim this branch', 'success', 94769267827], ['Type Check · consumer gates', 'success', 94769267736], ['Type Check · debt ledger', 'success', 94769267736], From 56ccd0cae966a7795fd29a032718599202866be5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 25 Oct 2026 04:38:22 +0000 Subject: [PATCH 2/2] ci(shard-timings): judge each refresh candidate at its own shard width The 6 -> 8 resize made select-shard-timings-run reject every recent schedule run (all 6-shard, all whole and green) as missing Test Core 7/8, 8/8, so the refresh exited 1 on this PR (job 121075952437) and would have kept doing so until an 8-shard schedule run existed with its artifacts retained. The dataset is per package; the width a run was split at is not an input to any package's weight. runIsEligible now reads the run's width from its own `Test Core (N/M)` job names and admits it when all M shards concluded success with all M summaries retained. A run whose Test Core jobs disagree on M, or that has none, is refused by name. The download loop already iterates each candidate's own artifact_ids, so a resize has no transition window. New self-test battery (width change, 9 cases) and an accumulation leg mixing 8-wide and 6-wide runs, both confirmed to fail against the old fixed-width reading. Co-authored-by: Claude Claude-Session: https://claude.ai/code/session_01M781cxsyRAqT49LvLkJKXb --- .github/workflows/shard-timings-refresh.yml | 11 +- scripts/ci/select-shard-timings-run.mjs | 230 +++++++++++++++++--- scripts/measure-test-shard-timings.mjs | 8 +- scripts/partition-test-shards.mjs | 6 +- 4 files changed, 217 insertions(+), 38 deletions(-) diff --git a/.github/workflows/shard-timings-refresh.yml b/.github/workflows/shard-timings-refresh.yml index 0f0183abe45..218846ee817 100644 --- a/.github/workflows/shard-timings-refresh.yml +++ b/.github/workflows/shard-timings-refresh.yml @@ -47,7 +47,9 @@ # HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE # ---------------------------------------------------------------------------- # The documented refresh path downloads the `test-core-run-summary-N-of-M` -# artifacts of a green run, one per Test Core shard (M = SHARD_COUNT). +# artifacts of a green run, one per Test Core shard. M is the RUN's own width, +# read from its `Test Core (N/M)` job names, so runs from before and after a +# matrix resize both read (#25376). # `GET /actions/artifacts/{id}/zip` redirects to # `productionresultssa*.blob.core.windows.net`, and every agent container's # egress policy answers 403 to CONNECT for that host — reproduced four ways, on @@ -484,9 +486,10 @@ jobs: OK=1 # One summary per shard the selector found: it admits a run only when - # every one of SHARD_COUNT (select-shard-timings-run.mjs) shards - # concluded success with its artifact retained, so these keys ARE the - # roster. A count spelled here would be a third copy of it (#25376). + # every shard OF THAT RUN concluded success with its artifact + # retained, at the run's own width (select-shard-timings-run.mjs, + # runShardWidth), so these keys ARE that run's roster. A count + # spelled here would break the first refresh after a resize (#25376). SHARD_NS=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(Object.keys(r.artifact_ids).map(Number).sort((a,b)=>a-b).join(" "))' "$i") for N in $SHARD_NS; do ART=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(r.artifact_ids[process.argv[2]])' "$i" "$N") diff --git a/scripts/ci/select-shard-timings-run.mjs b/scripts/ci/select-shard-timings-run.mjs index 0db4d3b5447..2258e312bf5 100644 --- a/scripts/ci/select-shard-timings-run.mjs +++ b/scripts/ci/select-shard-timings-run.mjs @@ -18,12 +18,12 @@ // next run in its concurrency group starts, and merges arrive faster than // Test Core finishes: 36 of the last 60 push runs were censored that way. // Such a run leaves SOME artifacts, so "the artifacts exist" is not the -// test -- every one of the six Test Core jobs must have concluded +// test -- every one of the run's Test Core jobs must have concluded // `success`. The hourly run this now reads has its OWN concurrency group // (#16467, `github.event_name` is in ci.yml's key), so a merge cannot // censor it -- but a second hourly run can, which is why this test stays. // -// 2. CACHE REPLAYS. A run whose six jobs all concluded `success` can still +// 2. CACHE REPLAYS. A run whose shard jobs all concluded `success` can still // have replayed most of the workspace from the turbo cache: shards were // measured at 7s and 98s against a 672s prediction (0.01x, 0.15x). The // generator REFUSES cached tasks, so those packages simply are not in the @@ -51,8 +51,8 @@ // push to `main` computes its package set with `--affected` against // `github.event.before`, so a push run measures the packages that merge // touched and NOTHING ELSE. Nothing in the eligibility test below would have -// noticed. Six shard jobs still conclude `success`, six run-summary artifacts -// are still uploaded, and every guard reads green -- over a measurement of a +// noticed. Every shard job still concludes `success`, every run-summary artifact +// is still uploaded, and every guard reads green -- over a measurement of a // slice of the workspace that nobody asked for. // // `coverageReport` is not the backstop for that either. It refuses a package @@ -108,10 +108,11 @@ import { fileURLToPath } from 'node:url'; import { EXIT_FINDINGS, EXIT_PREREQUISITE_NOT_MET } from '../import-prerequisite.mjs'; import { isEntrypoint } from '../invoked-as.mjs'; -// The Test Core shard count. Spelled here as the number of jobs and artifacts a -// complete run must present; partition-test-shards.mjs owns the split itself and -// its own self-test reads ci.yml back, so this is a reader's expectation rather -// than a second declaration of the split. +// The Test Core shard count of the LIVE matrix. partition-test-shards.mjs owns +// the split itself and its own self-test reads ci.yml back, so this is a +// reader's default rather than a second declaration of the split -- and since +// #25376 it is NOT what a candidate run is judged against. See "A RUN IS +// COMPLETE AT ITS OWN WIDTH" below. export const SHARD_COUNT = 8; const API = 'https://api.github.com'; @@ -168,6 +169,42 @@ export function testStepSeconds(job, stepName = TEST_STEP_NAME) { return Number.isFinite(seconds) && seconds >= 0 ? seconds : null; } +// --------------------------------------------------------------------------- +// A RUN IS COMPLETE AT ITS OWN WIDTH (#25376) +// --------------------------------------------------------------------------- +// +// Measured on #25376's own PR, job 121075952437: the matrix went 6 -> 8 and +// this lane, still judging every candidate against the LIVE count, rejected all +// 24 recent schedule runs -- each one a whole, green 6-shard run -- as "missing +// Test Core 7/8, 8/8", and exited 1. Every refresh would have read the same +// until an 8-shard schedule run existed with its artifacts still retained. +// +// The dataset is PER PACKAGE. Which shard a package ran on, and how many shards +// there were, is not an input to its weight. So a run is a complete +// measurement when every shard OF THAT RUN concluded success and every one of +// THAT RUN's summaries is retained, and the width is read from the run itself: +// the `M` of its `Test Core (N/M)` job names. A width change then has no +// transition window at all -- the runs before it stay readable, the runs after +// it are readable -- rather than a window this lane would have to report as +// NOT MEASURED. +// +// ⛔ A run whose Test Core jobs disagree on `M` is REJECTED, by name. That is +// not a run of any width; reading it at either one would admit a run that +// never ran a whole matrix. A run with no Test Core job at all is rejected too. +export function runShardWidth(jobs) { + const widths = new Set(); + for (const job of jobs ?? []) { + const m = /^Test Core \((\d+)\/(\d+)\)$/.exec(String(job?.name ?? '').trim()); + if (m && Number(m[2]) >= 1) widths.add(Number(m[2])); + } + if (widths.size === 1) return { width: [...widths][0], reason: null }; + if (widths.size === 0) return { width: null, reason: 'no `Test Core (N/M)` job in the run' }; + return { + width: null, + reason: `its Test Core jobs disagree on the shard count (${[...widths].sort((a, b) => a - b).join(' vs ')})`, + }; +} + // Is this run a complete, uncensored measurement? Both halves are required and // they fail for different reasons, so both are named separately. // @@ -176,8 +213,18 @@ export function testStepSeconds(job, stepName = TEST_STEP_NAME) { // censored run this refuses. A job that is still running is likewise not a // success -- it may yet fail -- so anything other than the literal string is a // rejection. -export function runIsEligible({ jobs, artifacts }, shardCount = SHARD_COUNT) { +// +// `shardCount` defaults to the run's OWN width (runShardWidth above); passing +// one pins the reading to that width, which only a test does. +export function runIsEligible({ jobs, artifacts }, shardCount) { const reasons = []; + if (shardCount === undefined) { + const read = runShardWidth(jobs); + if (read.width === null) { + return { eligible: false, reasons: [read.reason], shardCount: null, artifactIds: {}, timingArtifactIds: {}, shardSeconds: {} }; + } + shardCount = read.width; + } const byShard = new Map(); for (const job of jobs ?? []) { @@ -224,6 +271,7 @@ export function runIsEligible({ jobs, artifacts }, shardCount = SHARD_COUNT) { return { eligible: reasons.length === 0, reasons, + shardCount, artifactIds: Object.fromEntries([...artifactByShard.entries()].sort((a, b) => a[0] - b[0])), timingArtifactIds: Object.fromEntries([...timingByShard.entries()].sort((a, b) => a[0] - b[0])), shardSeconds, @@ -340,7 +388,8 @@ export async function listCandidates({ repo, token, workflow = 'ci.yml', - shardCount = SHARD_COUNT, + // Unset: each run is judged at its own width (#25376). Set only by a test. + shardCount, limit = 12, event = DEFAULT_RUN_EVENT, fetchImpl = fetch, @@ -367,6 +416,7 @@ export async function listCandidates({ html_url: run.html_url, eligible: verdict.eligible, reasons: verdict.reasons, + shard_count: verdict.shardCount, artifact_ids: verdict.artifactIds, timing_artifact_ids: verdict.timingArtifactIds, shard_seconds: verdict.shardSeconds, @@ -458,7 +508,7 @@ export async function runCandidates({ repo, token, limit, event, workflow = 'ci. for (const run of examined) { notes.push( run.eligible - ? ` ELIGIBLE ${run.run_id} ${run.created_at} ${String(run.head_sha).slice(0, 10)}` + ? ` ELIGIBLE ${run.run_id} ${run.created_at} ${String(run.head_sha).slice(0, 10)} (${run.shard_count} shards)` : ` rejected ${run.run_id} ${run.created_at} ${run.reasons.join('; ')}` ); } @@ -537,6 +587,10 @@ const SELF_TEST_BATTERIES = Object.freeze({ // reports a battery that registered zero cases by name, and a battery folded // into another only makes a number smaller. 'select-shard-timings-run candidate selection': 13, + // #25376. A run is judged at its OWN width, so a matrix resize has no + // transition window. Its own battery so that "an old-width run stopped + // reading" is a NAMED breach. + 'select-shard-timings-run width change (#25376)': 9, // #16467, patch round. Its own battery so that "the empty/ineligible // distinction stopped being tested" is a NAMED breach rather than a smaller // number folded into a neighbour. @@ -549,12 +603,12 @@ const SELF_TEST_BATTERIES = Object.freeze({ // it must stop at the minimum depth rather than at the first covering run, // count executed runs rather than runs, and keep a short refresh as // PROVISIONAL rather than refusing it. - 'shard-timings-refresh accumulation depth': 11, + 'shard-timings-refresh accumulation depth': 13, // #22949 ruling E. The per-file capture download step, driven like the // battery above, and the real `--diff` reading what it laid out. 'shard-timings-refresh per-file captures (#22949 E)': 5, }); -const SELF_TEST_BATTERY_FLOOR = 6; +const SELF_TEST_BATTERY_FLOOR = 7; const UNATTRIBUTED_BATTERY = '(no battery open)'; // Returned by `selfTest()` only after its verdict is printed, so a `return` that @@ -854,6 +908,99 @@ async function selfTest() { if (!threw(() => workspaceNames({ packages: { items: [] } }))) throw new Error('workspace: an EMPTY package list was accepted'); }); + // ------------------------------------------------------------------------- + battery('select-shard-timings-run width change (#25376)'); + // ------------------------------------------------------------------------- + // Measured on #25376's own PR: with the live count at 8, all 24 candidate + // schedule runs -- whole, green 6-shard runs -- were rejected as incomplete + // and the lane exited 1. A run is judged at ITS OWN width now; these pin + // both directions: an old-width run reads, and a run that is not whole at + // any width still does not. + const jobsAt = (width, conclusion = 'success') => + Array.from({ length: width }, (_, i) => ({ name: `Test Core (${i + 1}/${width})`, conclusion, status: 'completed' })); + const artifactsAt = (width, present = Array.from({ length: width }, (_, i) => i + 1)) => + present.map((n) => ({ id: 3000 + n, name: `test-core-run-summary-${n}-of-${width}`, expired: false })); + const sixShardRun = { jobs: jobsAt(6), artifacts: artifactsAt(6) }; + + check(() => { + const v = runIsEligible(sixShardRun); + if (!v.eligible) throw new Error(`width: a whole 6-shard run was rejected under an 8-shard matrix (${v.reasons.join('; ')})`); + if (v.shardCount !== 6) throw new Error(`width: the run's own width read as ${v.shardCount}, not 6`); + if (Object.keys(v.artifactIds).join(',') !== '1,2,3,4,5,6') { + throw new Error(`width: the 6-shard run's artifacts read as ${JSON.stringify(v.artifactIds)}`); + } + }); + check(() => { + // FIRING CONTROL: pinned to the live count, the same run is the rejection + // job 121075952437 printed -- so the pass above is the width reading, not a + // fixture that would pass anyway. + const v = runIsEligible(sixShardRun, SHARD_COUNT); + if (SHARD_COUNT === 6) return; + if (v.eligible || !v.reasons.some((r) => r.includes(`${SHARD_COUNT}/${SHARD_COUNT}`))) { + throw new Error(`width: pinned to ${SHARD_COUNT}, the 6-shard run was not rejected as the live lane rejected it (${v.reasons.join('; ')})`); + } + }); + check(() => { + const v = runIsEligible({ jobs: jobsAt(6), artifacts: artifactsAt(6, [1, 2, 3, 4, 5]) }); + if (v.eligible || !v.reasons.some((r) => r.includes('6-of-6'))) { + throw new Error(`width: an old-width run missing its last summary was not refused by name (${v.reasons.join('; ')})`); + } + }); + check(() => { + const cancelled = jobsAt(6); + cancelled[5].conclusion = 'cancelled'; + const v = runIsEligible({ jobs: cancelled, artifacts: artifactsAt(6) }); + if (v.eligible || !v.reasons.some((r) => r.includes('6/6'))) { + throw new Error(`width: a censored old-width run was accepted (${v.reasons.join('; ')})`); + } + }); + check(() => { + // Jobs at one width, summaries at another: not a whole run at either. + const v = runIsEligible({ jobs: jobsAt(6), artifacts: artifactsAt(8) }); + if (v.eligible) throw new Error('width: 6 shard jobs with 8-shard summaries were accepted'); + }); + check(() => { + const v = runIsEligible({ jobs: [...jobsAt(6), ...jobsAt(8)], artifacts: [...artifactsAt(6), ...artifactsAt(8)] }); + if (v.eligible || v.shardCount !== null || !v.reasons.some((r) => r.includes('disagree') && r.includes('6 vs 8'))) { + throw new Error(`width: a run whose Test Core jobs disagree on the count was not refused by name (${v.reasons.join('; ')})`); + } + }); + check(() => { + const v = runIsEligible({ jobs: [{ name: 'Lint', conclusion: 'success' }], artifacts: artifactsAt(8) }); + if (v.eligible || !v.reasons.some((r) => r.includes('no `Test Core (N/M)` job'))) { + throw new Error(`width: a run with no Test Core job was not refused by name (${v.reasons.join('; ')})`); + } + }); + check(() => { + // The width reader is anchored like the shard matcher: a job merely + // containing the shape is not a shard of any width. + const read = runShardWidth([...jobsAt(6), { name: 'Test Core (3/8) rerun', conclusion: 'success' }]); + if (read.width !== 6) throw new Error(`width: an unanchored job name moved the width (${JSON.stringify(read)})`); + }); + await (async () => { + // Through listCandidates: one 8-wide and one 6-wide run, both eligible, + // each reported with its own width for the download loop to iterate. + const byRun = { + 801: { jobs: jobsAt(8), artifacts: artifactsAt(8) }, + 802: { jobs: jobsAt(6), artifacts: artifactsAt(6) }, + }; + const fetchImpl = async (url) => { + const { pathname } = new URL(String(url)); + if (/\/actions\/workflows\/[^/]+\/runs$/.test(pathname)) { + const workflow_runs = [801, 802].map((id) => ({ id, head_sha: `${id}`.padStart(40, 'a'), created_at: '2026-10-25T00:00:00Z' })); + return { ok: true, status: 200, statusText: 'OK', json: async () => ({ workflow_runs }) }; + } + const m = /\/actions\/runs\/(\d+)\/(jobs|artifacts)$/.exec(pathname); + if (m) return { ok: true, status: 200, statusText: 'OK', json: async () => ({ [m[2]]: byRun[m[1]][m[2]] }) }; + return { ok: false, status: 404, statusText: 'Not Found', json: async () => ({}) }; + }; + const examined = await listCandidates({ repo: 'o/r', token: 't', fetchImpl }); + check(() => { + const shape = examined.map((r) => `${r.run_id}:${r.eligible}:${r.shard_count}:${Object.keys(r.artifact_ids).length}`).join(' '); + if (shape !== '801:true:8:8 802:true:6:6') throw new Error(`width: listCandidates read mixed widths as ${shape}`); + }); + })(); + // ------------------------------------------------------------------------- battery('select-shard-timings-run candidate selection'); // ------------------------------------------------------------------------- @@ -1284,9 +1431,12 @@ async function selfTest() { const TYPES = '@objectstack/types'; const toolPath = (rel) => fileURLToPath(new URL(rel, import.meta.url)); - // runs: newest first, each { id, tasks: [[package, seconds, cacheStatus]] }. - // Every task lands in shard 1's summary; shards 2-8 carry an empty task list, - // which is a valid summary that measures nothing. + // runs: newest first, each { id, tasks: [[package, seconds, cacheStatus]] }, + // optionally `width` (default 8) and `lastShardTasks`. `tasks` land in shard + // 1's summary and `lastShardTasks` in the run's LAST shard's; every other + // shard carries an empty task list, which is a valid summary that measures + // nothing. + const shardsOf = (r) => Array.from({ length: r.width ?? 8 }, (_, i) => i + 1); const driveGenerate = (runs, committedPackages, workspace = Object.keys(committedPackages)) => { const dir = mkdtempSync(join(tmpdir(), 'os-timings-generate-')); try { @@ -1306,21 +1456,21 @@ async function selfTest() { run_id: r.id, head_sha: String(r.id).padStart(40, 'c'), created_at: '2026-10-06T00:00:00Z', - artifact_ids: Object.fromEntries([1, 2, 3, 4, 5, 6, 7, 8].map((n) => [n, Number(`${r.id}${n}`)])), + artifact_ids: Object.fromEntries(shardsOf(r).map((n) => [n, Number(`${r.id}${n}`)])), })); writeFileSync(join(runnerTemp, 'candidates.json'), JSON.stringify(candidatesList)); for (const r of runs) { - for (const n of [1, 2, 3, 4, 5, 6, 7, 8]) { - const tasks = - n !== 1 - ? [] - : r.tasks.map(([pkg, seconds, status = 'MISS']) => ({ - taskId: `${pkg}#test`, - task: 'test', - package: pkg, - cache: { status }, - execution: { startTime: 0, endTime: status === 'MISS' ? seconds * 1000 : 90, exitCode: 0 }, - })); + const last = shardsOf(r).length; + for (const n of shardsOf(r)) { + const tasks = [...(n === 1 ? r.tasks : []), ...(n === last ? r.lastShardTasks ?? [] : [])].map( + ([pkg, seconds, status = 'MISS']) => ({ + taskId: `${pkg}#test`, + task: 'test', + package: pkg, + cache: { status }, + execution: { startTime: 0, endTime: status === 'MISS' ? seconds * 1000 : 90, exitCode: 0 }, + }) + ); writeFileSync(join(fixtures, `${r.id}${n}.json`), JSON.stringify({ tasks })); } } @@ -1432,6 +1582,30 @@ async function selfTest() { } }); + // LEG 1b -- A WIDTH CHANGE (#25376). The newest run is 8 shards wide, the two + // before it 6, as on the first refresh after the matrix grew. Types is + // measured ONLY in each run's last shard, so the loop must download that + // run's own shards -- 8 for one, 6 for the others -- and no more: a fixed + // 1..8 loop asks for a 6-wide run's shard 7, which has no artifact. + const widened = driveGenerate( + [ + { id: 921, width: 8, tasks: [[SPEC, 1600]], lastShardTasks: [[TYPES, 60]] }, + { id: 922, width: 6, tasks: [[SPEC, 1500]], lastShardTasks: [[TYPES, 62]] }, + { id: 923, width: 6, tasks: [[SPEC, 1700]], lastShardTasks: [[TYPES, 61]] }, + ], + committedTwo + ); + await check(() => { + if (widened.status !== 0 || widened.runCount !== '3') { + throw new Error(`width leg: runs of two widths did not accumulate to 3 (got ${widened.runCount}, exit ${widened.status}):\n${widened.log}`); + } + }); + await check(() => { + if (widened.refreshed?.packages?.[SPEC] !== 1600 || widened.refreshed?.packages?.[TYPES] !== 61) { + throw new Error(`width leg: a run's last shard was not read at its own width (${JSON.stringify(widened.refreshed?.packages)})`); + } + }); + // LEG 2 -- EXECUTED, NOT REPLAYED. Run 912 replayed spec, so three runs give // spec only two samples and the loop has to take a fourth. const replayed = driveGenerate( diff --git a/scripts/measure-test-shard-timings.mjs b/scripts/measure-test-shard-timings.mjs index a461cc4ffeb..4cc60f2efbf 100644 --- a/scripts/measure-test-shard-timings.mjs +++ b/scripts/measure-test-shard-timings.mjs @@ -27,7 +27,7 @@ // // From CI, no special run needed. Every Test Core shard passes --summarize // and, on merge_group builds, uploads `.turbo/runs/` as the -// `test-core-run-summary--of-8` artifact. +// `test-core-run-summary--of-` artifact. // // ⚠ ONE RUN IS NOT ENOUGH, AND THIS SENTENCE USED TO SAY IT WAS. It read // "download all six from any green queue build and re-run the generator", @@ -986,7 +986,7 @@ export function renderSpread({ bySample, dataset }) { // if it were. The dataset holds per-PACKAGE weights only; the `Slowest // file:` line the ceiling check prints comes from the per-shard captures // `report-test-timings.mjs --capture` writes in every Test Core shard and -// uploads as `test-core-timings--of-8` -- the SAME runs this refresh reads, +// uploads as `test-core-timings--of-` -- the SAME runs this refresh reads, // retained exactly as long as their run summaries (1 day). So the refresh lane // downloads those captures beside the summaries and this reads them: still a // read of an existing measurement, never a second test run. A run whose @@ -1558,7 +1558,7 @@ export function renderRefreshDiff({ diff, files = null, filesNotMeasured = null, ? ` No capture arrived for ${files.runsWithoutCapture.map((r) => `\`${r}\``).join(', ')}, so its files are not in the list.` : ''; out.push( - `**Per-file reading:** the window's per-shard captures (\`test-core-timings-N-of-8\`, written by ` + + `**Per-file reading:** the window's per-shard captures (\`test-core-timings-N-of-M\`, written by ` + `\`report-test-timings.mjs --capture\`), read for ${files.runsRead} of ${files.runsRead + files.runsWithoutCapture.length} run(s): ` + `${files.measuredRows} file line(s) measured here; ${files.replayedRows} replayed line(s) left out, because a cache ` + `HIT replays a duration measured outside this window.${missing}` @@ -3573,7 +3573,7 @@ function main() { refresh: 'node scripts/measure-test-shard-timings.mjs [--run ] ... ' + '--merge-into scripts/test-shard-timings.json --out scripts/test-shard-timings.json ' + - '(summaries: the `test-core-run-summary--of-8` artifacts of any green run, or a local ' + + '(summaries: the `test-core-run-summary--of-` artifacts of any green run, or a local ' + '`pnpm exec turbo run test --concurrency=4 --summarize`. ⚠ ONE RUN COVERS ONLY 2-52 of ~71 ' + 'packages depending on cache warmth, so feed SEVERAL runs -- a `--run ` before each ' + 'run\'s summaries is REQUIRED, so a sliced package is assembled per run and then medianed ' + diff --git a/scripts/partition-test-shards.mjs b/scripts/partition-test-shards.mjs index 9ebc0474a3e..57943b2fe92 100644 --- a/scripts/partition-test-shards.mjs +++ b/scripts/partition-test-shards.mjs @@ -126,8 +126,10 @@ const TIMINGS_PATH = path.join(REPO_ROOT, 'scripts', 'test-shard-timings.json'); // N/8`); the self-test below reads that workflow back and fails on drift, // because a partitioner splitting into a different number of bins than the // matrix declares is not a red step anywhere -- it is packages that no shard -// runs. scripts/ci/select-shard-timings-run.mjs spells it once more, as the -// number of jobs and artifacts a complete run presents to the timings refresh. +// runs. scripts/ci/select-shard-timings-run.mjs spells it once more, but only +// as its matchers' default: the timings refresh judges each run at the width +// that run's own `Test Core (N/M)` jobs carry, so a resize leaves the runs +// from before it readable (#25376). // // WHY EIGHT (#25376, 6 -> 8). On the committed dataset (the 2026-10-23 // refresh), bins of predicted test windows, the CLI cut into its 3 slices: