Skip to content

bug: ci: hosted backlog holds idle Dell merge-queue capacity #5135

Closed

Description

@zackees

A logic error in ci/select_mq_runner.py:304-307 reserves an older merge-group run without inspecting its jobs whenever its workflow status is queued. A run can already have selected hosted runners while waiting on hosted jobs. job_commitments also discards completed jobs before reading their routing evidence, and does not recognize the Darwin artifact producer as a heavy-pool decision.

Report 15 measured 168 minutes of Dell idleness (2026-11-18 16:45:53–19:34:21Z): every probe had online=6, busy=0, idle=6, required=3. Meanwhile 78 Dell-routable ubuntu-26.04 jobs waited over five minutes (803 job-minutes); only 3 of 19 merge-group runs used the Dell. Introduced by #5025.

Read-only selector logs confirm run 44359171975 (job 134995483499) rejected at committed=5 and run 44367435932 (job 135024874840) rejected at committed=3 despite six idle runners and run_limit=2. Run 44367448224 (job 135024915358) instead failed closed on an incomplete queued-runs collection. Their heavy-job labels prove hosted routing.

Acceptance: inspect latest-attempt jobs for queued as well as in-progress workflows; release hosted-decided reservations, retain deterministic older undecided reservations and Dell commitments, preserve fail-closed probes and bounded reads, and calculate/log admission capacity. Add recorded fixtures and a RED regression, update CI tooling documentation, pass selector/workflow policy tests, and run full lint once. No runner-host, label or live-queue changes. No dependencies or open decisions.

Activity

  1. added
    bugSomething isn't working
    readyMeets the definition of Ready; claimable
    P1Next up
    ciContinuous integration
    on Nov 18, 2026
  2. zackees commented on Nov 18, 2026

    @zackees
    MemberAuthor

    Claiming: fix hosted-decided merge-queue reservations in pool slot obs-studio6. 1. Replay the three recorded hosted selections and commit a failing regression. 2. Probe queued runs and retain routing evidence while preserving race protection and fail-closed reads. 3. Validate selector/workflow policy tests, run lint once, and publish the RED/GREEN pair subject to backpressure. First PR expected this session.

  3. zackees commented on Nov 18, 2026

    @zackees
    MemberAuthor

    #5140 merged (833456c). Merge-queue runs that already routed hosted no longer count against Dell admission. I'm leaving this open until a Dell measurement after this merge confirms that queue bursts route to the Dell when its runners are idle.

  4. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Second half landed in #5158 (b180bc8), via #5156. Both admission over-counts are fixed: hosted-decided runs (#5140) and runs whose Dell jobs have all finished (#5158). I'm keeping this open until the next Dell measurement confirms that queue runs go to an idle Dell.

  5. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Dell check 18 (03:20-04:17Z, after #5140, #5158 and #5186): 2 merge-group runs. One went to the Dell; the other went hosted on a legitimate capacity decision (busy=5, idle=1, required=3). No probe failures and no hosted-while-idle decisions. That's consistent with the fix, but too few runs to confirm. I'll close this after one more measurement with overlapping runs. The Dell run took about 20 min end to end, against about 30 min before; the module cache now restores 6 of 7 modules (#5176), and the Darwin fixtures step took 443 s against 607-624 s (#5162).

  6. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Dell check 19 (04:17-05:35Z, 7 merge-group runs after #5140, #5158 and #5186): 5 went to the Dell and 2 hosted for capacity. Zero went hosted while the Dell was idle, and there were zero probe failures. One probe snapshot race recovered on retry, and the overlaps included bursts of 2 and 3 runs. This is fixed. One borderline case remains: a hosted decision with idle=4 but committed=2, where one commitment was probably a run that had already been cancelled and was still releasing runners. If that recurs it deserves its own issue; it doesn't reopen this one.

  7. moved this from In progress to Done in obs-rust Multiviewon Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1Next upbugSomething isn't workingciContinuous integrationreadyMeets the definition of Ready; claimablesize:SHours

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions