CI perf: e2e fan-out — 23 e2e-macos + ~118 Linux e2e jobs/run, 82–87% under 2 min, 30–58% of job time is setup (~1,200 macOS + ~6,000 Linux job-min/day) #1172

Measurement

Window: 2026-10-08 02:16Z to 19:36Z. All CI runs, from runs/<id>/jobs.

* Setup-step share is the time in Set up job, Checkout, Download the e2e binaries, Complete job and Post steps. It does not count runner provisioning or queue time.

e2e-macos in merge_group runs: 23 legs. Summing each leg's p50, the jobs take 25.7 min, of which the test steps (Run …) are only 10.8 min. Examples per leg (p50 job / p50 test steps, n=68):

The 3 Maven legs each install Maven separately.

Linux runner saturation. When Linux jobs started exceed about 3,000 per hour, Linux queue wait climbs:

Those queue minutes land directly on merge_group critical paths. e2e-macos itself waits p50 0.35 min, p90 5.2 min.

Example run: 37828932401, with 23 e2e-macos jobs and about 106 Linux e2e jobs.

Where the time goes

Each matrix leg pays these fixed costs before running about 5–60 s of tests:

• Runner provisioning.
• Set up job (about 0.1 min).
• Checkout.
• Downloading the 200 MB e2e-bin artifact (about 0.1 min on Linux, more on macOS).
• A toolchain install (Maven, dotnet, pip/uv, bun, vlt).

Root cause

The e2e, e2e-macos and e2e-full matrices give one job to each (suite × tool-version) pair. Since #1133, Linux merge_group runs have about 106–117 e2e jobs. Most legs finish in under a minute of test time, so per-job overhead dominates. Each leg also takes a separate runner slot from the ~20-slot macOS pool and the Linux pool.

Proposed fix

In .github/workflows/ci.yml, bin-pack the short legs. Leave the long Gradle legs alone; those are tracked separately.

1. e2e-macos: collapse the 23 rows into about 4 bins of roughly 3 test-minutes each, grouped by toolchain:

   • JVM: the 3 Maven legs share one Maven install, plus the dotnet leg.
   • JS: bun, pnpm, vlt ×5, rush.
   • Python: uv, pypi, vex pip/pipenv/hatch/poetry/pdm.
   • npm.

   Each bin runs its suites sequentially in one job, with one Run e2e tests step per suite, so the logs stay attributable. Use if: always() on each step so one failure doesn't hide the others.

2. Linux e2e and e2e-full: apply the same packing to rows whose p90 test time is under 2 min. That is about 80% of the 117 rows. Target bins of about 6–8 min, which stays well under the Gradle legs (10–14 min) so the critical path doesn't grow. Generate the bins from a duration table checked into scripts/ (like ci-test-shard.py) rather than by hand.

3. Keep the matrix row → suite mapping under test (scripts/test_ci_*.py) so every suite/version combination still runs exactly once.

Expected saving

• macOS: about 19 fewer jobs × about 0.45 min fixed cost ≈ 8.5 macOS min per run. At 141 runs/day (merge_group, push and nightly), that is ~1,200 macOS job-min/day, or about 900/day if CI perf: CI on push to main re-tests the exact SHA the merge queue just passed (~16,000 job-min/day, ~1,800 macOS) #1170 removes the push re-run. macOS slots per run drop from 23 to 4.
• Linux: packing about 33,000 of the sub-2-min jobs a day roughly 4:1 removes about three quarters of their about 9,000 min/day of setup-step time, so ~6,000 Linux job-min/day. Runner provisioning, which these numbers don't count, comes on top. It also removes about 25,000 Linux job starts a day, which is the load behind the 5–10 min busy-hour queue waits.
• Critical path: neutral to slightly positive, from less queue wait for the Gradle legs.

Coverage and risk

• Every suite and tool-version still runs in the same events (PR / merge_group / push / nightly). Only the job grouping changes.
• Failure attribution gets coarser, which per-suite steps mitigate.
• A flaky suite now fails a bigger job. Re-runs cost more per failure, but there are fewer jobs.
• Required checks ci-ok and clippy are unchanged. ci-ok keeps needs: on the same job ids.

Effort

L. It restructures the matrices and needs a generator script plus test updates.

ROI

ROI = weighted saving × confidence / effort. Weighted saving is (Linux 6.0 + 3 × macOS 1.2) k job-min/day = 9.6. Confidence is 0.6. Effort L counts as 3.

ROI = 9.6 × 0.6 / 3 = 1.9