|
7 | 7 | # |
8 | 8 | # scripts/test-shard-timings.json is the per-package duration dataset the Test |
9 | 9 | # Core split is binned from. It is GENERATED — scripts/measure-test-shard- |
10 | | -# timings.mjs turns a green run's six turbo summaries into it — but until now it |
11 | | -# was generated only when a person remembered to. It went 13 days without a |
| 10 | +# timings.mjs turns a green run's per-shard turbo summaries into it — but until |
| 11 | +# now it was generated only when a person remembered to. It went 13 days without a |
12 | 12 | # refresh while ~700 test files were added, and the shard it mis-weighted was |
13 | 13 | # killed by the job wall twelve times in one day and ejected from the merge queue |
14 | 14 | # twice (#16173). |
|
46 | 46 | # |
47 | 47 | # HOW IT GETS ITS INPUTS, AND WHY IT RUNS HERE RATHER THAN IN AN AGENT WORKTREE |
48 | 48 | # ---------------------------------------------------------------------------- |
49 | | -# The documented refresh path downloads the six `test-core-run-summary-N-of-6` |
50 | | -# artifacts of a green run. `GET /actions/artifacts/{id}/zip` redirects to |
| 49 | +# The documented refresh path downloads the `test-core-run-summary-N-of-M` |
| 50 | +# artifacts of a green run, one per Test Core shard (M = SHARD_COUNT). |
| 51 | +# `GET /actions/artifacts/{id}/zip` redirects to |
51 | 52 | # `productionresultssa*.blob.core.windows.net`, and every agent container's |
52 | 53 | # egress policy answers 403 to CONNECT for that host — reproduced four ways, on |
53 | 54 | # two artifacts across two runs, on the pre-signed URL as well as the API one, on |
|
71 | 72 | # Until #16467 a push to `main` re-ran the FULL Test Core battery, so "a green |
72 | 73 | # push run on main" and "a measurement of the workspace" were the same thing. |
73 | 74 | # They are not any more: a push run computes its package set with `--affected` |
74 | | -# against `github.event.before`. Its six shards still conclude `success` and |
75 | | -# still upload six run summaries, so NOTHING in the eligibility test would have |
| 75 | +# against `github.event.before`. Its shards still conclude `success` and |
| 76 | +# still upload every run summary, so NOTHING in the eligibility test would have |
76 | 77 | # noticed — this lane would have kept regenerating the balancing dataset from |
77 | 78 | # measurements of whatever the last merge happened to touch, and the coverage |
78 | 79 | # check would not have caught it either (a package the affected set skipped is a |
|
96 | 97 | # follows from the cache design rather than from luck — turbo's key is |
97 | 98 | # namespaced per shard and only main pushes write it, so a package whose inputs |
98 | 99 | # have not changed is a HIT, and the generator refuses hits rather than |
99 | | -# recording a replayed ~0.1s window as a suite's cost. "Download six artifacts |
100 | | -# from any green run" therefore measures a SLICE of the workspace. That number |
| 100 | +# recording a replayed ~0.1s window as a suite's cost. "Download the shard |
| 101 | +# artifacts from any green run" therefore measures a SLICE of the workspace. That number |
101 | 102 | # should IMPROVE under the hourly run, because a push no longer seeds the whole |
102 | 103 | # workspace's entries — but it is quoted as it was measured, and the next live |
103 | 104 | # run is what re-measures it. |
|
178 | 179 | # - DOWNS, in one line. |
179 | 180 | # - FILES OVER BUDGET: every test file over 10 s of wall in the window, with |
180 | 181 | # its package and its slowest run (#22998). The dataset holds no per-file |
181 | | -# reading; the window's runs do, in the per-shard `test-core-timings-N-of-6` |
| 182 | +# reading; the window's runs do, in the per-shard `test-core-timings-N-of-M` |
182 | 183 | # captures, which a step before the body downloads: the same runs, retained |
183 | 184 | # exactly as long as their run summaries. |
184 | 185 | # |
@@ -432,14 +433,14 @@ jobs: |
432 | 433 | # not bad luck, it is the cache design: turbo's key is namespaced per shard |
433 | 434 | # and only main pushes write it, so a package whose inputs have not changed |
434 | 435 | # is a HIT — and the generator refuses hits rather than recording a |
435 | | - # replayed ~0.1s window as a suite's cost. "Download six artifacts from any |
436 | | - # green run" therefore measures a SLICE of the workspace, never all of it. |
| 436 | + # replayed ~0.1s window as a suite's cost. "Download the shard artifacts |
| 437 | + # from any green run" therefore measures a SLICE of the workspace, never all of it. |
437 | 438 | # Since #16467 the runs read here are the HOURLY `schedule` runs, which are |
438 | 439 | # the full battery; the accumulation stays because the cache argument above |
439 | 440 | # is unchanged by which event ran the suite. |
440 | 441 | # |
441 | | - # So runs are accumulated. Each contributes its six summaries under its own |
442 | | - # `--run <id>` group, which is exactly the grouping #16473 added: slices are |
| 442 | + # So runs are accumulated. Each contributes its shards' summaries under its |
| 443 | + # own `--run <id>` group, which is exactly the grouping #16473 added: slices are |
443 | 444 | # summed within a run and the per-run sums are medianed across runs, so a |
444 | 445 | # package measured by three of these runs gets a median of three |
445 | 446 | # observations rather than whichever run happened to be read last. Feeding |
@@ -482,7 +483,12 @@ jobs: |
482 | 483 | rm -rf "$SUMDIR"; mkdir -p "$SUMDIR" |
483 | 484 |
|
484 | 485 | OK=1 |
485 | | - for N in 1 2 3 4 5 6; do |
| 486 | + # One summary per shard the selector found: it admits a run only when |
| 487 | + # every one of SHARD_COUNT (select-shard-timings-run.mjs) shards |
| 488 | + # concluded success with its artifact retained, so these keys ARE the |
| 489 | + # roster. A count spelled here would be a third copy of it (#25376). |
| 490 | + SHARD_NS=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(Object.keys(r.artifact_ids).map(Number).sort((a,b)=>a-b).join(" "))' "$i") |
| 491 | + for N in $SHARD_NS; do |
486 | 492 | ART=$(node -e 'const r=JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP+"/candidates.json","utf8"))[Number(process.argv[1])];console.log(r.artifact_ids[process.argv[2]])' "$i" "$N") |
487 | 493 | # curl drops the Authorization header across the redirect to the |
488 | 494 | # blob host, which is correct: that URL is pre-signed and the |
@@ -636,7 +642,7 @@ jobs: |
636 | 642 | # The per-file half of the regression readings (#22949 ruling E, item 3). |
637 | 643 | # The dataset holds no per-file durations; the window's runs do, in the |
638 | 644 | # per-shard captures `report-test-timings.mjs --capture` uploads as |
639 | | - # `test-core-timings-N-of-6`, retained 1 day like the run summaries the |
| 645 | + # `test-core-timings-N-of-M`, retained 1 day like the run summaries the |
640 | 646 | # generator read. The selector listed their ids beside the summaries' |
641 | 647 | # (`timing_artifact_ids`), so this is a read of the SAME runs, never a new |
642 | 648 | # measurement, and it feeds the PR body alone. |
@@ -822,7 +828,8 @@ jobs: |
822 | 828 | const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8")); |
823 | 829 | const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID); |
824 | 830 | const s = r.shard_seconds ?? {}; |
825 | | - console.log([1,2,3,4,5,6].map((n) => (s[n] == null ? "?" : `${n}: ${s[n]}s`)).join(" | ")); |
| 831 | + const ns = Object.keys(r.artifact_ids ?? {}).map(Number).sort((a, b) => a - b); |
| 832 | + console.log(ns.map((n) => (s[n] == null ? "?" : `${n}: ${s[n]}s`)).join(" | ")); |
826 | 833 | ') |
827 | 834 |
|
828 | 835 | { |
@@ -851,8 +858,8 @@ jobs: |
851 | 858 | done |
852 | 859 | echo |
853 | 860 | echo "- Newest run in the set: \`$RUN_ID\`, commit \`$HEAD_SHA\` — the date this refresh carries." |
854 | | - echo "- Every run above had all six \`Test Core (N/6)\` jobs conclude \`success\` with its six" |
855 | | - echo " run-summary artifacts still retained; runs that were cancelled, failed or had lost" |
| 861 | + echo "- Every run above had every \`Test Core (N/M)\` shard job conclude \`success\` with its" |
| 862 | + echo " run-summary artifact still retained; runs that were cancelled, failed or had lost" |
856 | 863 | echo " their artifacts were rejected by name in the log before any of these were used." |
857 | 864 | echo |
858 | 865 | echo "$CARRY_LINE" |
@@ -886,7 +893,7 @@ jobs: |
886 | 893 | echo "## The partitioner's own pins RED on this refresh — read this before merging" |
887 | 894 | echo |
888 | 895 | echo "This is the designed behaviour, not a defect in the refresh: the acceptance bound is" |
889 | | - echo "a ratio, and a package that has grown past what any six-way split can bin makes the" |
| 896 | + echo "a ratio, and a package that has grown past what the Test Core split can bin makes the" |
890 | 897 | echo "pins fail with the arithmetic in the message. The remedy the partitioner names is to" |
891 | 898 | echo "raise the file-level slice count for that package — ⛔ never to raise the bound, and" |
892 | 899 | echo "⛔ never to hand-edit this dataset. This workflow deliberately does neither: it" |
|
0 commit comments