CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th)

Filed by the CI profiler routine (observer only). This is a ruleset change, so a human has to decide; the implementer routine can't apply it.

Measurement

Window: 2026-10-08 08:17 → 2026-10-09 08:17 UTC. Sources: ruleset 24668460, 157 CI merge_group runs, job detail for 52 of them, and REST timelines of the 14 most recently merged PRs.

Current merge_queue rule: max_entries_to_build: 5, max_entries_to_merge: 5, min_entries_to_merge: 1, grouping_strategy: ALLGREEN, check_response_timeout_minutes: 120.

metric	value
PRs merged / 24h	69
merge_group `CI` success, created → done, p50 / p90	23.4 / 46.1 min (n=80); 21.5 / 23.0 min since 04:17 (n=16)
Last enqueue → merge, p50 / p90 (14 PRs merged 03:14–08:01)	64 / 69 min
First enqueue → merge, p50 / p90	65 / 82 min

What happened in the 06:19–08:01 burst. 13 PRs were enqueued between 06:19 and 06:52: #1215, #1209, #1208, #1217, #1193, #1218, #1036, #1041, #1226, #1230, #1229, #1228, plus #1007. They were tested in waves of at most 5 CI runs, and each wave started only after the previous one finished:

wave created	runs	outcome
06:19–06:20	5	all cancelled. `test (windows-latest, 1)` failed in 37892957346 and evicted #1215; the other 4 were rebuilds (37892958452 etc.)
06:34–06:35	5	2 ok, 3 cancelled (`coverage-docker (sbt)` failure in 37894279963)
06:56	5	all ok → merged 06:56 / 07:19
07:17–07:19	5	all ok → merged 07:38–07:41
07:38–07:41	4 (+1 cancelled)	merged 08:01

So the queue's throughput is 5 PRs per ~21 min (≈14/h). The 6th–13th PRs of a burst wait one or two full waves before their own run even starts. Of the 64-min p50, about 21 min is the PR's own run and ~40 min is waiting behind earlier waves.

Where the time goes

Per wave: test (windows-latest, 2) (Run tests 15.0 min p50 + Build 2.2), coverage-docker (sbt) → coverage-merge (CI perf: coverage-docker (sbt) — 5.6-min image rebuild warms 3 unused toolchains; merge-queue critical path (~2.5 min/merge-group run) #1225), and the Gradle e2e_redirect_gradle_build … gradle_hosted_b legs (13.4 min p50, CI perf: Gradle e2e shards unbalanced — b-p leg is the merge-queue critical path (~3 min/merge-group run) #1171) all finish at 20–22 min. Queue wait for those jobs is ≤0.1 min (Linux/Windows) and 0.2 min p50 (macOS), so the waves are not runner-starved. The limit is the build depth.
Rebuild cost when an entry is evicted (52-run sample):
- 15 runs cancelled purely as rebuilds behind an evicted entry: 4,311 job-min, 287 avg
- 10 runs with a genuine failure: 4,110 job-min
- 27 successful runs: 12,795 job-min, 474 avg
- A deeper build window raises the first number.

Root cause

max_entries_to_build: 5 caps the speculative pipeline at 5 entries. With ALLGREEN groups of up to 5 and every wave taking about the same ~21 min, the queue advances in lock-step waves. During the agent bursts that now produce most merges, every PR past the 5th waits at least one extra full wave.

Proposed fix (ruleset 24668460, merge_queue rule)

Raise max_entries_to_build 5 → 8, and keep max_entries_to_merge at 5 (or raise it to 8 too, so a fully green window lands in one merge).
Leave grouping_strategy, timeouts and the required checks (ci-ok, clippy) unchanged.
Roll back if any of these hold over the following 24h:
- macOS job queue-wait p90 goes above ~5 min (it is 1.8 min now)
- the merge_group eviction rate stays above ~20%. Then the extra rebuilds cost more than the latency buys; prefer landing the ci-janitor flake fixes (Fix Windows flake in read-set replacement test #1237, Retry HTTP errors in sbt/composer Docker image downloads #1238, Retry Docker e2e maven/cargo fixture fetches past registry blips #1222, Retry Bun backtest cells on hosted-fetch 5xx blips #1244) first.

Expected saving

Latency: ~21 min off enqueue → merge for every PR that is 6th–8th in line. In the window above that was 8 of 13 PRs. Over the day, assuming about a third of 69 merges arrive in bursts, that is ~20 PRs × ~21 min ≈ 7 PR-hours/day. Merge-queue p50 should fall from ~64 toward ~45 min during bursts.
Critical path per affected PR: −21 min (one wave).
Cost:
- Each eviction during a burst can rebuild up to 7 entries instead of 4: ≈ +3 × 287 ≈ +860 job-min per eviction.
- At ~5–8 burst evictions/day that is +4–7k job-min/day, ~11% of it macOS (51 mac-min per full run).
- Peak concurrent macOS jobs from the queue would go from 50 (measured at 06:42 with 5 entries; 28.9 mac jobs per run) to roughly 80.
- CI perf: ci.yml e2e-macos — 23 one-minute macOS jobs per merge_group/push run (~2,300 macOS job starts/day) #1176 / Run e2e suites that share a toolchain in one leg #1234 (fewer e2e-macos legs) would offset most of that.

Coverage and risk

No test moves; every job still runs on every merge_group run.
Required check names are unchanged.
Main risk: macOS pool pressure. It is shared with push-to-main runs (42/day × ~56 mac-min) and every other merge_group run. Watch macOS queue wait on the CI performance dashboard after the change.
Second risk: more cancelled rebuild minutes while the eviction rate is still ~20–30%.

Effort

S (one ruleset field, reversible in seconds).

ROI

About 7: (≈21 critical-path min − ≈7 weighted for +5k job-min) × confidence 0.5 ÷ effort 1. Same scale as the dashboard.

[agent] Triaged p3. This is a merge-queue ruleset change (max_entries_to_build), which agents may not apply, so it needs a maintainer decision. Labeled agent:needs-human.

Generated by Claude Code

More evidence from the CI janitor, 2026-10-09 18:38Z: runner capacity is starving the merge queue.

154 runs queued and 95 in progress (235 after dedup). 187 of them are pull_request runs on current heads of 80 open PRs. 28 more are Copilot dynamic runs.
The 6 merge_group CI runs created at 18:14Z and 18:20Z (pr-1284, 1286, 1288, 1247, 1290, 1274) were all still queued at 18:38Z, 18 to 24 minutes after creation, before any of their test jobs started.
The main push CI run for 9ab72d4 (run 37970501466, created 18:02Z) was still queued 36 minutes later.
When an entry was dequeued, rebuilt runs waited for runners all over again. Example: Follow the republished minimist canary patch #1302 was closed at about 18:05Z, the queue rebuilt at 18:14Z, and the Merge queue janitor sweep for the 18:02Z group itself waited for a runner until 18:17Z.

I cancelled only clearly wasted runs: 4 on superseded PR SHAs and 5 duplicate runs on the same SHA. That is negligible next to the backlog. This is a capacity problem rather than a reliability one. Options for the owner: merge_group priority over PR runs (for example, a separate runner group or larger runners for merge_group), or limiting PR pushes per agent. #1362 (Depot Linux runners) is the open PR in this direction.

Generated by Claude Code

CI janitor evidence, 2026-10-09 ~19:40 UTC:

121 queued + 90 in-progress Actions runs. The oldest queued PR run (CI on agent/fix-vendored-reresolved-version-drift) was created 18:23, so it has waited about 75 min.
Merge-group wall time under this backlog: the four groups created 18:14 (Skip the vlt compat matrix on ci.yml-only changes #1284, Retry CI toolchain installs past rustup download blips #1286, Wire vendored nuget.config through formats::nuget (#594) #1288, Fix bun.lockb shared bundled pin being unmanageable (#1243) #1247) finished between 19:31 and 19:35, about 77–81 min each. The two groups created 18:20 (Fix gem checks judging unused system gem homes (#1098, #1109) #1290, Fix yarn classic empty-range lock keys (#1271) #1274) finished at 19:38, about 78 min each. Every CI job in those groups succeeded. The time was spent waiting for runners, not running tests.
Three groups (Fix scan human/JSON exit and prune forks (#1062) #1298, Fix hosted requirements rewrite of user direct refs (#542) #1333, Document vendor --force as variant-probe bypass (#923) #1346) were created between 19:32 and 19:37 and are still all queued.
The only merge_group CI failures in the last 24h were one semantic conflict (15:20, clippy compile error on Fix bun.lockb shared bundled pin being unmanageable (#1243) #1247's group, which cascaded to six entries) and one Windows gem e2e flake on 2026-10-08 (since fixed by Fix Windows flake in gem global-gemfile refusal e2e #1169). The queue is not losing entries to flakes. It is starved of runners because PR pushes compete with it.

I cancelled 3 runs to relieve the backlog: 2 on a superseded PR head and 1 on a draft PR.

Generated by Claude Code

CI profiler, 2026-10-09 20:16 UTC.

Settings changed since this issue was filed. Ruleset 24668460 now has max_entries_to_build 8 (was 5) and check_response_timeout_minutes 90 (was 120). max_entries_to_merge is still 5. The new values can't be judged on today's data:

Last enqueue → merge:
- 111–112 min for Skip the vlt compat matrix on ci.yml-only changes #1284, Retry CI toolchain installs past rustup download blips #1286, Wire vendored nuget.config through formats::nuget (#594) #1288 and Fix bun.lockb shared bundled pin being unmanageable (#1243) #1247 (enqueued 17:43–17:44, merged 19:35).
- 78 min for Fix yarn classic empty-range lock keys (#1271) #1274 and Fix gem checks judging unused system gem homes (#1098, #1109) #1290 (enqueued 18:20).
- Earlier today it was 76 / 94 min p50 / p90.
Merge-group run time: the six groups that passed took 77–81 min, against ~22 min unloaded. About 62 of those minutes were queue wait for ubuntu-latest runners, not test time. ubuntu-22.04 jobs in the same runs waited under 1.5 min.
Fixes in flight: CI perf: ci-ok — waits 9–18 min for an ubuntu-latest runner while ubuntu-22.04 starts in <1 min (merge-queue critical path, backlog hours) #1372 (ci-ok waits 9–18 min for a runner, S fix) and CI perf: PR CI — 23% of full PR runs re-test unchanged diffs after a merge-main push (~20–30k Linux job-min/day, feeds the ubuntu-latest backlog) #1373 (merge-main pushes drive 23% of full PR runs). Depot runners (Run Linux CI jobs on Depot runners #1362) would remove the backlog itself.

Building 8 entries at once adds about 3 × 310 Linux job-min to the same backlog in a burst. Worth re-checking once #1362 or #1373 lands.

Generated by Claude Code

CI profiler, 2026-10-11 12:17 UTC. Window: the last 24h.

Settings now: ruleset 24668460 reads max_entries_to_build 5. It was 8 at the last profile, so it has been set back. max_entries_to_merge is 5, min_entries_to_merge_wait_minutes 5, the timeout 90 min, grouping ALLGREEN.

What changed underneath: since #1362 (Depot Linux) and #1375 (lean gate) merged, a merge_group CI run costs 29 Linux + 7.5 Windows + 0 macOS job-min. It cost 312 / 69 / 60 before. It runs created → ci-ok in p50 9.6 / p90 9.95 min (n=30). Queue wait is p50/p90 0.02/0.02 min on Depot (n=2,814 jobs) and 0.03/0.03 min on Windows. The only hosted job per group is one e2e-build-windows. Building more entries at once no longer competes for a scarce pool.

What 5 still costs:

Enqueue → merge for 14 PRs merged in the window: p50 10.3, max 18.8 min.
The one outlier came from build concurrency. Fix project-mode Maven crawl listing all of ~/.m2 (#265) #1361, Warn and fail --check when Go vendor/ or go.mod need a resync (#343, #618) #1357, Fix vendored Pipenv sibling requirements.txt (#612) #1309, Fix vendored-ledger root under --manifest-path (#745) #1345 and Add CI watch script, API budget rules and /code-review note for agents #1164 were enqueued 17:30–17:31 and filled the 5 build slots. Fix NuGet mapping exclusivity and inherited sources (#354, #462) #1341 (enqueued 17:31:11) got its merge_group run only at 17:39:49 and merged at 17:50, after 18.8 min, about 8.5 min of it waiting for a slot. Fix hosted Maven pom edits outside live markup (#259, #262, #342) #1327 and Stop warm NuGet cache shadowing patches (#352) #1344 waited the same way.
Evictions: one in the window, a clippy failure on Restore NuGet contentHash, not catalog packageHash (#624) #1343's group, evicted in 1.4 min by fail-fast. The re-queue merged in 10.7 min.

Suggestion for a human: with ~37 job-min per group and no queue wait, max_entries_to_build 10 (or more) costs about 37 extra job-min per extra entry in a burst. Bursts of 8+ entries would then merge in one ~10-min wave instead of two. The Windows job in each group is the only hosted runner, and #1385 would cut its time.

Generated by Claude Code

CI profiler, 2026-10-12 00:16 UTC. Window: the last 24h.

Settings now: unchanged since the last comment. max_entries_to_build 5, max_entries_to_merge 5, min_entries_to_merge_wait_minutes 5, ALLGREEN, timeout 90.

This window:

20 merge_group runs; 19 finished, all green. 0 evictions in the 15 merged-PR timelines read.
Enqueue → merge: p50 7.9 / p90 8.3 / max 8.7 min (n=15).
Largest burst: 5 entries at 19:10–19:15 (run), exactly max_entries_to_build. All 5 merged in 7.6–8.7 min. The other bursts were 3 entries (22:47–22:54, 23:48–23:49).
merge_group created → ci-ok: p50 7.3 min after CI: type-check Windows in the queue, in parallel (#1385) #1386 (n=16).

What this means for the decision: no burst above 5 in 48h, so a second build round has not happened since 10-11 17:31. If one does, an entry past the 5th now waits about one extra CI run, ~7.3 min (it was ~9.7 before #1386). The case for raising max_entries_to_build is weaker than when this was filed, though raising it is still free on Depot runners (Linux queue wait p90 0.02 min, n=2,075 jobs).

Generated by Claude Code

CI profiler, 2026-10-13 00:16 UTC. Window: the last 24h (2026-10-12 00:16 → 2026-10-13 00:16).

Settings now: unchanged since the last comment (read 00:20). max_entries_to_build 5, max_entries_to_merge 5, min_entries_to_merge_wait_minutes 5, ALLGREEN, timeout 90.

The 15:44 burst is the only one in the window that hit the cap:

11 entries were enqueued between 15:44 and 15:52. The merge_group CI runs were created in waves: 15:44–15:45 (5), 15:52–15:53 (4), 16:01–16:02 (6), 16:10–16:11 (4).
Four entries (Fix macOS go.work member match through ./x/../app #1411, CI: run Windows pnpm e2e tests instead of skipping (#1054) #1407, Fix Poetry rewrite of a source = "pypi" dep (#1282) #1413, Fix gem agent crawl under a moved BUNDLE_GEMFILE (#952) #1430) waited for the third wave and merged 27.6–31.9 min after their last enqueue. The other 11 timelines read last profile took 7.9–10.4 min.
Roughly ~8 min of that extra wait came from the 5-entry build cap, and roughly ~9 min from the rebuild after Fix vendored Hatch refs in %/#/;/? checkouts (#547) #1428's coverage flake (38273391986, fixed by Fix env race flake in cache_env scrub self-tests #1418). That flake also cancelled 3 runs.
Later bursts were 5 entries at 17:49, 2 at 19:49, 3 at 20:49 and 1 each at 19:27, 20:17, 20:34 and 22:49. None was larger than 5. Enqueue → merge for those 15 PRs was p50 8.8 / p90 10.5 / max 10.5 min, with 0 evictions.

What this means for the setting: a merge_group CI run takes p50 8.15 / p90 9.2 min from creation to ci-ok (n=36). Linux queue wait on Depot is p90 0.02 min, so runner capacity is not the limit. Raising max_entries_to_build to 8–10 would have cut ~8 min from the 4 burst entries past the 5th. That is about 32 entry-minutes in one burst per day, and nothing in the other 31 merges. The case is weaker than when this issue was filed: bursts of more than 5 now happen about once a day, not continuously. Leaving this open for a human decision.

Generated by Claude Code

Navigation Menu

Repository navigation

CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th) #1248

Measurement

Where the time goes

Root cause

Proposed fix (ruleset 24668460, merge_queue rule)

Expected saving

Coverage and risk

Effort

ROI

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI perf (settings): merge queue builds 5 entries at a time — a 13-PR burst took 100 min to drain on a 21-min CI run (~21 min per PR past the 5th) #1248

Description

Measurement

Where the time goes

Root cause

Proposed fix (ruleset 24668460, merge_queue rule)

Expected saving

Coverage and risk

Effort

ROI

Activity

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 11, 2026

mikolalysenko commented on Oct 12, 2026

mikolalysenko commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions