You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A logic error in ci/select_mq_runner.py:304-307 reserves an older merge-group run without inspecting its jobs whenever its workflow status is queued. A run can already have selected hosted runners while waiting on hosted jobs. job_commitments also discards completed jobs before reading their routing evidence, and does not recognize the Darwin artifact producer as a heavy-pool decision.
Report 15 measured 168 minutes of Dell idleness (2026-11-18 16:45:53–19:34:21Z): every probe had online=6, busy=0, idle=6, required=3. Meanwhile 78 Dell-routable ubuntu-26.04 jobs waited over five minutes (803 job-minutes); only 3 of 19 merge-group runs used the Dell. Introduced by #5025.
Read-only selector logs confirm run 44359171975 (job 134995483499) rejected at committed=5 and run 44367435932 (job 135024874840) rejected at committed=3 despite six idle runners and run_limit=2. Run 44367448224 (job 135024915358) instead failed closed on an incomplete queued-runs collection. Their heavy-job labels prove hosted routing.
Acceptance: inspect latest-attempt jobs for queued as well as in-progress workflows; release hosted-decided reservations, retain deterministic older undecided reservations and Dell commitments, preserve fail-closed probes and bounded reads, and calculate/log admission capacity. Add recorded fixtures and a RED regression, update CI tooling documentation, pass selector/workflow policy tests, and run full lint once. No runner-host, label or live-queue changes. No dependencies or open decisions.
Claiming: fix hosted-decided merge-queue reservations in pool slot obs-studio6. 1. Replay the three recorded hosted selections and commit a failing regression. 2. Probe queued runs and retain routing evidence while preserving race protection and fail-closed reads. 3. Validate selector/workflow policy tests, run lint once, and publish the RED/GREEN pair subject to backpressure. First PR expected this session.
#5140 merged (833456c). Merge-queue runs that already routed hosted no longer count against Dell admission. I'm leaving this open until a Dell measurement after this merge confirms that queue bursts route to the Dell when its runners are idle.
Second half landed in #5158 (b180bc8), via #5156. Both admission over-counts are fixed: hosted-decided runs (#5140) and runs whose Dell jobs have all finished (#5158). I'm keeping this open until the next Dell measurement confirms that queue runs go to an idle Dell.
Dell check 18 (03:20-04:17Z, after #5140, #5158 and #5186): 2 merge-group runs. One went to the Dell; the other went hosted on a legitimate capacity decision (busy=5, idle=1, required=3). No probe failures and no hosted-while-idle decisions. That's consistent with the fix, but too few runs to confirm. I'll close this after one more measurement with overlapping runs. The Dell run took about 20 min end to end, against about 30 min before; the module cache now restores 6 of 7 modules (#5176), and the Darwin fixtures step took 443 s against 607-624 s (#5162).
Dell check 19 (04:17-05:35Z, 7 merge-group runs after #5140, #5158 and #5186): 5 went to the Dell and 2 hosted for capacity. Zero went hosted while the Dell was idle, and there were zero probe failures. One probe snapshot race recovered on retry, and the overlaps included bursts of 2 and 3 runs. This is fixed. One borderline case remains: a hosted decision with idle=4 but committed=2, where one commitment was probably a run that had already been cancelled and was still releasing runners. If that recurs it deserves its own issue; it doesn't reopen this one.
Activity
zackees commentedon Nov 18, 2026
Claiming: fix hosted-decided merge-queue reservations in pool slot obs-studio6. 1. Replay the three recorded hosted selections and commit a failing regression. 2. Probe queued runs and retain routing evidence while preserving race protection and fail-closed reads. 3. Validate selector/workflow policy tests, run lint once, and publish the RED/GREEN pair subject to backpressure. First PR expected this session.
zackees commentedon Nov 18, 2026
#5140 merged (833456c). Merge-queue runs that already routed hosted no longer count against Dell admission. I'm leaving this open until a Dell measurement after this merge confirms that queue bursts route to the Dell when its runners are idle.
zackees commentedon Nov 19, 2026
Second half landed in #5158 (b180bc8), via #5156. Both admission over-counts are fixed: hosted-decided runs (#5140) and runs whose Dell jobs have all finished (#5158). I'm keeping this open until the next Dell measurement confirms that queue runs go to an idle Dell.
zackees commentedon Nov 19, 2026
Dell check 18 (03:20-04:17Z, after #5140, #5158 and #5186): 2 merge-group runs. One went to the Dell; the other went hosted on a legitimate capacity decision (busy=5, idle=1, required=3). No probe failures and no hosted-while-idle decisions. That's consistent with the fix, but too few runs to confirm. I'll close this after one more measurement with overlapping runs. The Dell run took about 20 min end to end, against about 30 min before; the module cache now restores 6 of 7 modules (#5176), and the Darwin fixtures step took 443 s against 607-624 s (#5162).
zackees commentedon Nov 19, 2026
Dell check 19 (04:17-05:35Z, 7 merge-group runs after #5140, #5158 and #5186): 5 went to the Dell and 2 hosted for capacity. Zero went hosted while the Dell was idle, and there were zero probe failures. One probe snapshot race recovered on retry, and the overlaps included bursts of 2 and 3 runs. This is fixed. One borderline case remains: a hosted decision with idle=4 but committed=2, where one commitment was probably a run that had already been cancelled and was still releasing runners. If that recurs it deserves its own issue; it doesn't reopen this one.