CI perf: ci.yml e2e-macos — 23 one-minute macOS jobs per merge_group/push run (~2,300 macOS job starts/day) #1176

Measurement

Window: 2026-10-01 → 2026-10-08 (jobs sampled: 643 successful e2e-macos legs from merge_group/push/schedule runs).

• e2e-macos (ci.yml:1734) is a 23-row matrix, all macos-latest. It runs on every merge_group run (21.8 legs/run in the sample), every push to main (23/run) and nightly — if: github.event_name != 'pull_request'.
• Per leg: run time p50 0.83 min; queue wait p50 1.1 min, p90 20.8 min (macOS pool saturation).
• Last 24h: 115 merge_group CI runs + 26 push runs ⇒ roughly 100 runs/day × 23 = ~2,300 macOS job starts/day for ~9.4 min of actual test work per run.
• Pre-Cut merge-group CI from ~46 to ~20 min: shard Gradle e2e and test legs, skip test-release in queue, cancel orphaned runs #1133 sample: on merge_group runs, 11 e2e (macos-latest, …) legs had finish offset p50 ~45–49 min with queue wait p50 ~20 min / p90 ~50 min against a run time of ~1 min — i.e. the legs were waiting for a mac runner, not testing. Example run: https://github.com/SocketDev/socket-patch/actions/runs/37726213794/job/113147648897
• macOS queue wait across all jobs, last 24h: p50 22.0 min, p90 55.5 min; worst hour 2026-10-07T20 p50 81 min.

Where the time goes (per leg, averaged over 643 legs)

So ~56% of each leg is fixed overhead, and GitHub bills each job rounded up to a whole minute (23 billed macOS minutes per run for ~9.4 min of tests).

Root cause

Each suite × toolchain combination is its own matrix job. That was cheap when macOS legs ran on idle runners; now that the macOS pool (~20 concurrent, shared by every PR push and merge_group run) is the bottleneck, every extra job is another trip through the macOS queue.

Proposed fix

In .github/workflows/ci.yml, collapse e2e-macos from 23 rows to ~3 grouped rows, reusing the existing multi-suite loop in the shared *e2e-steps "Run e2e tests" step (it already iterates for suite in $E2E_SUITE and checks each suite ran ≥1 test):

1. JS: e2e_safety_pnpm, e2e_redirect_npm_build, e2e_redirect_rush_sim, e2e_redirect_pnpm_build, bun ×3 (bun 1.4.2), vlt ×5 (vlt 1.2.0).
2. Python: e2e_redirect_uv_build, e2e_vendor_pypi_build, e2e_vex_build for poetry/pdm/hatch/pipenv/pip.
3. JVM/.NET: e2e_redirect_maven_build, e2e_vendor_maven_build, e2e_vendor_jvm_build (maven_reactor), e2e_nuget_dotnet_build.

Rows that need distinct test_filters per suite need a small extension: allow test_filter per suite (e.g. a suite=filter list, or run the step once per sub-row with a bash loop over a JSON list in the matrix). The vlt legs' separate "Run vlt e2e tests" step and vlt_store_linker: hardlink (e2e_safety_vlt) can stay as an extra step in group 1. Keep fail-fast: false and keep the per-suite ::error:: reporting so a failure still names the suite. e2e-full (same steps) is unaffected.

Expected saving

• macOS job starts: 23 → 3 per run, ~2,000 fewer macOS jobs/day (~100 merge_group+push runs/day).
• macOS job-minutes: ~20 legs × ~0.5 min overhead ≈ ~10 min/run ≈ 1,000 macOS job-min/day actual; billed minutes drop more (per-job rounding: 23 → ~3 roundings).
• Merge-queue critical path: macOS queue contention from these legs is what pushed the mac legs' finish to ~45–49 min p50 pre-Cut merge-group CI from ~46 to ~20 min: shard Gradle e2e and test legs, skip test-release in queue, cancel orphaned runs #1133; with 3 jobs the e2e-macos group needs 3 slots instead of 23. Expected ~5–15 min off merge_group p90 when the mac pool is saturated (to be confirmed post-Cut merge-group CI from ~46 to ~20 min: shard Gradle e2e and test legs, skip test-release in queue, cancel orphaned runs #1133).
• Also cuts macOS queue wait for every other macOS job (PR compat legs on push/nightly, test (macos-latest)).

Coverage and risk

Every suite still runs on macOS on every merge_group, push to main, and nightly — only the job boundaries change. A group's wall time becomes ~4–6 min (still far under the ~24 min merge_group run). Risk: one failing suite marks the whole group red, so the job name no longer pinpoints the suite — mitigated by the existing per-suite ::error:: annotations. Required checks ci-ok / clippy are unchanged (ci-ok already depends on e2e-macos as a job, not on leg names).

Effort

S/M (matrix rewrite + per-suite filter plumbing in the shared step).

ROI

Saving ~1,000 macOS job-min/day + ~2,000 fewer macOS queue entries/day (weighted ×3 for the scarce macOS pool ≈ 3,000 pts) × confidence 0.8 / effort 1.5 ≈ 1600.