Skip to content

Pool loop optimization: continuous coordinator/manager/worker loop, client-side prooft attestation, Dell E640 MQ runner #4896

Open

Description

@zackees

Goal

Get the most merged PRs per hour on master without breaking any rule: RED then GREEN, the four test tiers, no edits to C tests or public headers, and owner holds respected. Three things get us there:

  1. A continuous coordinator, manager and worker loop that keeps every worker slot busy and the merge queue moving.
  2. Client-side attestation. The client runs and attests the heavy checks (clippy and Dylint for every platform, lint and the local gate) through prooft, so PR checks only verify those attestations and PRs reach the merge queue quickly.
  3. The Dell E640 merge-queue runner working and registered as an org GitHub Actions runner, with hosted fallback.

This issue is the umbrella, and its body is also the operating prompt for the loop session. Refs #4743, #4744, #4745, #4748, #443, #4005, #4838.


Operating prompt (paste into /loop)

Read AGENTS.md, .claude/skills/obs-coordinator/SKILL.md, .claude/skills/obs-worker/SKILL.md, .claude/skills/obs-worker-loop/SKILL.md, the org obs-org-coordinator/obs-org-worker skills (obs-rust/.github), and the workspace pool-worker skill (~/dev/obs-rust/.claude/skills/pool-worker/SKILL.md). You act as coordinator and pool manager together. You never write port code yourself. Each tick:

  1. Board and health. board_read. Run .worker-pool/dell_mq_health.py. Check master CI health, and run ci/backpressure.py --check.
  2. Coordinate. Start from uv run -m workflow.coordinator. Put green PRs in the queue, pinned to their head commit. Rebase PRs that were ejected or are stacked, push fixups, review, and close stalled follow-ups. A merge watcher (chaperone) per queued PR uses pr_merge_watch.py.
  3. Collect workers. Run session_status on every live session in state.json, not on a listing of the newest report files. Read each response file, check its slot is clean on detached origin/master with no process inside it, and mark the slot free.
  4. Fill slots. There are eight slots, ~/dev/obs-rust/obs-studio and obs-studio2 through obs-studio8. Triage before assigning: read the thread, holds, claims and linked PRs. Give each worker a named unit and a set of files that no other live worker touches. When back pressure is closed, launch only rebase, fix and review work, or branch-only work. Each worker gets a complete, self-contained prompt and pushes through ./push, which carries the attestation trailers.
  5. Measure. Append one line per tick to .worker-pool/metrics.tsv: timestamp, busy slots, launched, QUEUED/MERGED/BLOCKED/DUPLICATE/NEEDS-HELP counts, PRs merged since the last tick, queue depth, the median time from push to merge, rebases, and runner state.
  6. Learn. Add lessons to the pool-worker skill, and post a short status line on the board.

Workstream A: loop throughput

The main metric is PRs merged per hour. The secondary metrics are slot utilization, the share of workers that finish QUEUED rather than BLOCKED, DUPLICATE or NEEDS-HELP, the median time from push to merge, rebases per merged PR, and tokens per merged PR.

Experiments to run, one at a time, comparing each against the metrics.tsv baseline:

  • Tick driven by events: wake on session completion and chaperone reports instead of a fixed interval.
  • Assign files without overlap: one file or area per slot, and roll several appenders to the same file into one PR. Check overlap with git merge-tree --write-tree --name-only before launching.
  • Remove shared append points (test-graph.json, rust/obs-build/tests/native.rs, build-group lists) that make every merge rebase the next PR.
  • Cut wasted triage: the manager pre-reads ready issues so workers never get units that are already done or blocked.
  • Make use of a jammed queue: branch-only and stacked work while back pressure is closed, with one batched rebase worker per merge wave.
  • Machine contention: no parallel Windows link preflight batches, stale target/mcp lock cleanup, a 50 GB free-disk floor, and a cap on parallel cache-bypassed ./lint runs.
  • Worker model and prompt size: compare Sol against Opus on hard units, and measure prompt tokens against outcome.

Workstream B: client-side attestation with prooft

The client proves everything heavy, and PR CI only verifies the proof.

Workstream C: the Dell E640 merge-queue runner

  • The host is healthy: iDRAC 192.168.1.169 is reachable, it boots unattended, the NixOS netboot from the dev box works, and the hardware faults are known (DIMM A2, PSU1 without AC) and tracked.
  • It is registered as an org-level runner in group Dell-NixOS, with obs-ubuntu-2604 labels and all runner instances online. The group admits gh-readonly-queue/master/* merge-group refs.
  • The repo variables MQ_RUNNER_LABELS and MQ_RUNNER_STATUS_TOKEN are set. Before setting them, prove the runner with one real Actions job.
  • Hosted overflow works when the Dell is saturated or offline. Refs bug: CI: saturated Dell merge-queue runners prevent hosted overflow #4838.
  • Every loop tick runs dell_mq_health.py. Exit 2 means a Dell job failed: clear MQ_RUNNER_LABELS, then investigate. Exit 3 means fewer than 4 runners are online.
  • Measure merge-queue wall time on the Dell against hosted runners. Refs Merge queue wall time is now ~100% real work; the remaining lever is a larger queue runner (infrastructure) #4005.
  • The dev box must never be registered as a runner.

Done when

  • The loop runs unattended for 24 hours with at least 80% slot utilization, and metrics.tsv shows merged PRs per hour rising compared with the baseline.
  • Attested PRs from org members reach the queue with no heavy PR-side jobs.
  • Merge-queue runs land on the Dell, with a verified hosted fallback.

Activity

  1. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: workstream B ./push re-stamp publication fix in slot obs-studio2, branch fix/4896-push-restamp. 1. Record local RED with temporary-bare-remote regression coverage for new branches, exact-head re-stamps, remote movement and divergent history. 2. Permit only a pure re-stamp with an exact pre-stamp SHA lease; preserve ordinary pushes and refusals. 3. Run ./lint once, dogfood ./push, open a Refs-only PR and arm it pinned to HEAD. First PR expected this session. Maintainer explicitly authorized this pool-attestation unblocker despite closed back pressure.

  2. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Progress, 2026-11-16 (pool loop ticks 1-7)

    Workstream B: the client-attested fast path works end to end

    Gaps found

    1. ./push can't re-attest a branch that already exists. It amends HEAD to stamp and sign, then runs a plain git push, which is rejected as non-fast-forward after 11-26 min of gates. A fix is in progress: a --force-with-lease pinned to the pre-stamp remote SHA, used only for a pure re-stamp.
    2. The client gate costs 11-26 min per ./push (client-lint 107-644 s, client-test 514-774 s). This is the next thing to cut.
    3. Cold slots: a fresh checkout pays a full Tier 2 configure (7,526 edges) and about 6 min of lint.

    Workstream C: Dell

    Workstream A: the loop

    • Tick 1 wasted 4 of 8 launches: a hold, a duplicate and 2 back-pressure stops. Fix: read PR bodies for holds, cross-check the claim-conflict list, and launch no more producers than backpressure.py headroom allows. Fill the remaining slots with branch-only or stacked units and reviews (one contributor review took 4 min from approval to queue).
    • The bottleneck has now moved to the merge queue itself: 4 entries are awaiting checks, and the last merge was at 14:37Z.
  3. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: workstream B client-gate wall-time profiling and one measured optimization in slot obs-studio3, branch infra/4896-client-gate-speed.

    1. Profile two unchanged full client gate runs on origin/master without publication, preserving all 19 gates.
    2. Select the largest safe measured win and record local RED then GREEN regression coverage.
    3. Run ./lint once, push through ./push, and open a draft PR with Refs Pool loop optimization: continuous coordinator/manager/worker loop, client-side prooft attestation, Dell E640 MQ runner #4896; leave auto-merge off under the maintainer's explicit hold. First draft expected this session after the measurements.
  4. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: backpressure relaxation in slot obs-studio4, branch fix/4896-backpressure-relax. 1. Record fixture-based local RED for merge-queue width, stall/normal limits, starvation and unreadable state. 2. Implement the width-aware limit and update its documentation without changing parked-red accounting or starvation. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm it pinned to HEAD. First PR expected this session. The maintainer directed this change on 2026-11-16: the closed backpressure gate is too conservative; this fix reopens it, so opening and arming this PR is explicitly authorized despite the closed gate.

  5. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: workstream B merge-base attestation admission in slot obs-studio5, branch fix/4896-attest-mergebase. 1. Record local RED with temporary git repositories for master-only trust edits, PR-owned trust edits, merged master and unavailable merge bases, including fallback scope. 2. Share merge-base-relative changed paths between trust rejection and admission scope, preserving immutable BASE policy and queue heavy checks. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm auto-merge pinned to HEAD. First PR expected this session. Backpressure limit 10 per maintainer direction.

  6. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: workstream C Dell persistent Tier 2 build caches in slot obs-studio4, branch infra/4896-dell-persistent-cache. 1. Record local RED workflow contracts for marker-gated runner slots and unchanged hosted behavior. 2. Wire compiler outputs and apt archives only, preserving cost/module/verdict state and master-only save gates. 3. Run ./lint once, rebase immediately before ./push, push RED and GREEN together, and open a Refs-only PR with the proposal timing table; arm pinned to HEAD. First PR expected this session. Persistent Dell caches approved by the maintainer on 2026-11-16; backpressure limit 10.

  7. zackees commented on Nov 16, 2026

    @zackees
    MemberAuthor

    Claiming: rebase-bot cooperation in slot obs-studio9, branch fix/4896-rebase-bot-coop. 1. Record local RED tests for queue arming, worker ownership and failed merge-group ejections. 2. Preserve lease pushes, stacked landing, holds, FIFO and headroom while adding the cooperation guards and documenting pool-active/queue-failed. 3. Run ./lint once and push RED and GREEN together through ./push, then open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned on 2026-11-16, backpressure limit 10.

  8. zackees commented on Nov 17, 2026

    @zackees
    MemberAuthor

    Claiming: workstream B explicit ./push history replacement in slot obs-studio4, branch fix/4896-push-rewrite. 1. Record local RED with real bare-remote regressions for default rewrite refusal, matching and mismatched expected remote heads, and concurrent movement. 2. Add --replace-remote-head using an exact fully qualified force-with-lease, preserving default publication behavior, and document it. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned 2026-11-16, backpressure limit 10.

  9. zackees commented on Nov 17, 2026

    @zackees
    MemberAuthor

    Claiming: workstream C mq-selector in slot obs-studio3, branch fix/4896-mq-selector. 1. Compare recent merge-group job timings and record local RED for the six-online/five-busy/one-idle snapshot. 2. Make the smallest safe capacity-default correction while preserving outage fallback and all ci_scope assertions. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned, backpressure limit 10.

  10. zackees commented on Nov 17, 2026

    @zackees
    MemberAuthor

    Claiming: automatic merge-queue re-arm in slot obs-studio5, branch fix/4896-mq-requeue, building on #4932. 1. Record local RED fixture tests for ejection classes, unchanged heads, holds and the per-head retry cap. 2. Add a GITHUB_TOKEN-only requeue mode to the existing ai-rebase workflow and document it. 3. Run ./lint once and ./push with RED and GREEN together, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned, backpressure limit 10.

  11. 26 remaining items

  12. zackees commented on Nov 18, 2026

    @zackees
    MemberAuthor

    Claiming: merge-queue selector request budget and probe diagnostics (pool slot obs-studio4), branch infra/select-budget. 1. Record local RED fake-API regressions for nine-run regroups, exhausted budgets, terminal runs/jobs and warning reasons. 2. Size the shared budget and conservatively reserve unprobed active runs, building on #5043. 3. Run ./lint, push RED/GREEN together and open the focused PR this session; arm only if backpressure is open. Refs #4839.

  13. moved this from In progress to Ready in obs-rust Multiviewon Nov 19, 2026
  14. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Claiming: workstream C member-only same-repository PR routing, explicitly assigned by the maintainer on 2026-11-19; slot obs-studio3, branch infra/dell-member-prs. 1. Record local RED eligibility, reserve, base-trust and secret-isolation tests. 2. Add default-off DELL_PR_ROUTING and MQ_PR_RESERVE while preserving merge-queue routing. 3. Run ./lint once, push RED and GREEN together through ./push, and open the requested Refs-only PR this session; arm only if backpressure is open. Decisions: membership/API uncertainty stays hosted; routing code and membership verification are loaded from immutable PR BASE.

  15. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Claiming: local-gate timeout reporting fix in slot obs-studio3, branch fix/4896-local-gate-timeout (maintainer assignment local-gate-timeout). 1. Record local RED for bytes/None/string subprocess timeout output and complete failure aggregation. 2. Measure admission-verifier subprocess overhead under the parallel gate and implement the smallest supported resource/timeout fix. 3. Run ./lint and ./push, publish RED and GREEN together, and arm only if backpressure is open. First PR expected this session. Refs #4896.

  16. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Claiming: Dell merge-queue overcommit and selector snapshot KeyError, branch infra/select-overcommit (select-overcommit pool assignment). 1. Record local RED fake-API cases for busy Dell admission and the missing-current-run snapshot. 2. Add a bounded MQ_RUNNER_MAX_RUNS cap (default 3), conservative snapshot handling and routing documentation, preserving #5279 PR reserves. 3. Run ./lint and focused tests, push RED/GREEN together and open the focused CI PR this session. Refs #4839.

  17. zackees commented on Nov 19, 2026

    @zackees
    MemberAuthor

    Claiming: admission-verifier self-test subprocess reduction in slot obs-studio3, branch infra/4896-attest-test-speed (maintainer assignment attest-test-speed). 1. Record local RED subprocess counts and serial/parallel timing. 2. Reuse isolated fixture objects without removing assertions; measure GREEN and compare assertion coverage. 3. Run ./lint and ./push with RED/GREEN together; arm only with open backpressure. First PR expected this session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

infraBuild, gate, CI and toolingreadyMeets the definition of Ready; claimable

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions