Repository navigation
Description
Goal
Get the most merged PRs per hour on master without breaking any rule: RED then GREEN, the four test tiers, no edits to C tests or public headers, and owner holds respected. Three things get us there:
- A continuous coordinator, manager and worker loop that keeps every worker slot busy and the merge queue moving.
- Client-side attestation. The client runs and attests the heavy checks (clippy and Dylint for every platform, lint and the local gate) through prooft, so PR checks only verify those attestations and PRs reach the merge queue quickly.
- The Dell E640 merge-queue runner working and registered as an org GitHub Actions runner, with hosted fallback.
This issue is the umbrella, and its body is also the operating prompt for the loop session. Refs #4743, #4744, #4745, #4748, #443, #4005, #4838.
Operating prompt (paste into /loop)
Read
AGENTS.md,.claude/skills/obs-coordinator/SKILL.md,.claude/skills/obs-worker/SKILL.md,.claude/skills/obs-worker-loop/SKILL.md, the orgobs-org-coordinator/obs-org-workerskills (obs-rust/.github), and the workspacepool-workerskill (~/dev/obs-rust/.claude/skills/pool-worker/SKILL.md). You act as coordinator and pool manager together. You never write port code yourself. Each tick:
- Board and health.
board_read. Run.worker-pool/dell_mq_health.py. Check master CI health, and runci/backpressure.py --check.- Coordinate. Start from
uv run -m workflow.coordinator. Put green PRs in the queue, pinned to their head commit. Rebase PRs that were ejected or are stacked, push fixups, review, and close stalled follow-ups. A merge watcher (chaperone) per queued PR usespr_merge_watch.py.- Collect workers. Run
session_statuson every live session instate.json, not on a listing of the newest report files. Read each response file, check its slot is clean on detachedorigin/masterwith no process inside it, and mark the slot free.- Fill slots. There are eight slots,
~/dev/obs-rust/obs-studioandobs-studio2throughobs-studio8. Triage before assigning: read the thread, holds, claims and linked PRs. Give each worker a named unit and a set of files that no other live worker touches. When back pressure is closed, launch only rebase, fix and review work, or branch-only work. Each worker gets a complete, self-contained prompt and pushes through./push, which carries the attestation trailers.- Measure. Append one line per tick to
.worker-pool/metrics.tsv: timestamp, busy slots, launched, QUEUED/MERGED/BLOCKED/DUPLICATE/NEEDS-HELP counts, PRs merged since the last tick, queue depth, the median time from push to merge, rebases, and runner state.- Learn. Add lessons to the
pool-workerskill, and post a short status line on the board.
Workstream A: loop throughput
The main metric is PRs merged per hour. The secondary metrics are slot utilization, the share of workers that finish QUEUED rather than BLOCKED, DUPLICATE or NEEDS-HELP, the median time from push to merge, rebases per merged PR, and tokens per merged PR.
Experiments to run, one at a time, comparing each against the metrics.tsv baseline:
- Tick driven by events: wake on session completion and chaperone reports instead of a fixed interval.
- Assign files without overlap: one file or area per slot, and roll several appenders to the same file into one PR. Check overlap with
git merge-tree --write-tree --name-onlybefore launching. - Remove shared append points (
test-graph.json,rust/obs-build/tests/native.rs, build-group lists) that make every merge rebase the next PR. - Cut wasted triage: the manager pre-reads
readyissues so workers never get units that are already done or blocked. - Make use of a jammed queue: branch-only and stacked work while back pressure is closed, with one batched rebase worker per merge wave.
- Machine contention: no parallel Windows link preflight batches, stale
target/mcplock cleanup, a 50 GB free-disk floor, and a cap on parallel cache-bypassed./lintruns. - Worker model and prompt size: compare Sol against Opus on hard units, and measure prompt tokens against outcome.
Workstream B: client-side attestation with prooft
The client proves everything heavy, and PR CI only verifies the proof.
-
./pushruns clippy and Dylint for all six targets (Linux, macOS and Windows, x86_64 and aarch64) before it stamps the commit. Refs CI: make ./push prove clippy and Dylint for all six targets before stamping #4745. - The local gate (
local-gate.toml) attests the Linux unit-test lanes and Tier 2. Local native lanes or cross-compilation attest the macOS and Windows jobs. Refs ci: Let local native lanes attest the macOS and Windows jobs #443. - PR checks are attestation-only for trusted verified org members: a verifier owned by the base branch checks the prooft trailers. Outside contributors keep a real CI fallback. Refs CI: make PR checks attestation-only; client runs lint/clippy/Dylint (all targets); merge queue is the heavy gate #4743, CI: add base-owned per-gate attestation trust verifier for verified org members #4744, CI: cut trusted attested PRs over to verification-only admission with real outsider fallback #4748.
- Make
PUSH_REQUIRE_ATTESTATION=1the pool default, so a worker push without an attestation fails loudly. - Measure the time from opening a PR to entering the queue, attested against unattested. The target is under 5 minutes for an attested PR.
- The merge queue stays the heavy gate that decides the merge: nothing attested is trusted there.
Workstream C: the Dell E640 merge-queue runner
- The host is healthy: iDRAC 192.168.1.169 is reachable, it boots unattended, the NixOS netboot from the dev box works, and the hardware faults are known (DIMM A2, PSU1 without AC) and tracked.
- It is registered as an org-level runner in group Dell-NixOS, with
obs-ubuntu-2604labels and all runner instances online. The group admitsgh-readonly-queue/master/*merge-group refs. - The repo variables
MQ_RUNNER_LABELSandMQ_RUNNER_STATUS_TOKENare set. Before setting them, prove the runner with one real Actions job. - Hosted overflow works when the Dell is saturated or offline. Refs bug: CI: saturated Dell merge-queue runners prevent hosted overflow #4838.
- Every loop tick runs
dell_mq_health.py. Exit 2 means a Dell job failed: clearMQ_RUNNER_LABELS, then investigate. Exit 3 means fewer than 4 runners are online. - Measure merge-queue wall time on the Dell against hosted runners. Refs Merge queue wall time is now ~100% real work; the remaining lever is a larger queue runner (infrastructure) #4005.
- The dev box must never be registered as a runner.
Done when
- The loop runs unattended for 24 hours with at least 80% slot utilization, and
metrics.tsvshows merged PRs per hour rising compared with the baseline. - Attested PRs from org members reach the queue with no heavy PR-side jobs.
- Merge-queue runs land on the Dell, with a verified hosted fallback.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsReady
Activity
zackees commentedon Nov 16, 2026
Claiming: workstream B ./push re-stamp publication fix in slot obs-studio2, branch fix/4896-push-restamp. 1. Record local RED with temporary-bare-remote regression coverage for new branches, exact-head re-stamps, remote movement and divergent history. 2. Permit only a pure re-stamp with an exact pre-stamp SHA lease; preserve ordinary pushes and refusals. 3. Run ./lint once, dogfood ./push, open a Refs-only PR and arm it pinned to HEAD. First PR expected this session. Maintainer explicitly authorized this pool-attestation unblocker despite closed back pressure.
zackees commentedon Nov 16, 2026
Progress, 2026-11-16 (pool loop ticks 1-7)
Workstream B: the client-attested fast path works end to end
Ci-Attestationrecords, including six-target clippy and Dylint, plus about 8,600 host tests.added_to_merge_queue19:02:56Z, so 66 s from push to queue. rust: Port canvas scene migration to obs-core (Refs #4774) #4899 was the same.Gaps found
./pushcan't re-attest a branch that already exists. It amends HEAD to stamp and sign, then runs a plaingit push, which is rejected as non-fast-forward after 11-26 min of gates. A fix is in progress: a--force-with-leasepinned to the pre-stamp remote SHA, used only for a pure re-stamp../push(client-lint 107-644 s, client-test 514-774 s). This is the next thing to cut.Workstream C: Dell
MQ_RUNNER_LABELSwas re-set at 17:42Z, since the ci: overflow saturated merge-queue runners to hosted (Refs #4838) #4839 overflow fix had merged.dell-fsx3mn2-4failedsetup-soldr3 times ("spawn cargo ... Exec format error") while the other 5 passed. It was stopped at about 19:05Z; root cause is still open.Workstream A: the loop
backpressure.pyheadroom allows. Fill the remaining slots with branch-only or stacked units and reviews (one contributor review took 4 min from approval to queue).zackees commentedon Nov 16, 2026
Claiming: workstream B client-gate wall-time profiling and one measured optimization in slot obs-studio3, branch infra/4896-client-gate-speed.
zackees commentedon Nov 16, 2026
Claiming: backpressure relaxation in slot obs-studio4, branch fix/4896-backpressure-relax. 1. Record fixture-based local RED for merge-queue width, stall/normal limits, starvation and unreadable state. 2. Implement the width-aware limit and update its documentation without changing parked-red accounting or starvation. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm it pinned to HEAD. First PR expected this session. The maintainer directed this change on 2026-11-16: the closed backpressure gate is too conservative; this fix reopens it, so opening and arming this PR is explicitly authorized despite the closed gate.
zackees commentedon Nov 16, 2026
Claiming: workstream B merge-base attestation admission in slot obs-studio5, branch fix/4896-attest-mergebase. 1. Record local RED with temporary git repositories for master-only trust edits, PR-owned trust edits, merged master and unavailable merge bases, including fallback scope. 2. Share merge-base-relative changed paths between trust rejection and admission scope, preserving immutable BASE policy and queue heavy checks. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm auto-merge pinned to HEAD. First PR expected this session. Backpressure limit 10 per maintainer direction.
zackees commentedon Nov 16, 2026
Claiming: workstream C Dell persistent Tier 2 build caches in slot obs-studio4, branch infra/4896-dell-persistent-cache. 1. Record local RED workflow contracts for marker-gated runner slots and unchanged hosted behavior. 2. Wire compiler outputs and apt archives only, preserving cost/module/verdict state and master-only save gates. 3. Run ./lint once, rebase immediately before ./push, push RED and GREEN together, and open a Refs-only PR with the proposal timing table; arm pinned to HEAD. First PR expected this session. Persistent Dell caches approved by the maintainer on 2026-11-16; backpressure limit 10.
zackees commentedon Nov 16, 2026
Claiming: rebase-bot cooperation in slot obs-studio9, branch fix/4896-rebase-bot-coop. 1. Record local RED tests for queue arming, worker ownership and failed merge-group ejections. 2. Preserve lease pushes, stacked landing, holds, FIFO and headroom while adding the cooperation guards and documenting pool-active/queue-failed. 3. Run ./lint once and push RED and GREEN together through ./push, then open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned on 2026-11-16, backpressure limit 10.
zackees commentedon Nov 17, 2026
Claiming: workstream B explicit ./push history replacement in slot obs-studio4, branch fix/4896-push-rewrite. 1. Record local RED with real bare-remote regressions for default rewrite refusal, matching and mismatched expected remote heads, and concurrent movement. 2. Add --replace-remote-head using an exact fully qualified force-with-lease, preserving default publication behavior, and document it. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned 2026-11-16, backpressure limit 10.
zackees commentedon Nov 17, 2026
Claiming: workstream C mq-selector in slot obs-studio3, branch fix/4896-mq-selector. 1. Compare recent merge-group job timings and record local RED for the six-online/five-busy/one-idle snapshot. 2. Make the smallest safe capacity-default correction while preserving outage fallback and all ci_scope assertions. 3. Run ./lint once, push RED and GREEN together through ./push, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned, backpressure limit 10.
zackees commentedon Nov 17, 2026
Claiming: automatic merge-queue re-arm in slot obs-studio5, branch fix/4896-mq-requeue, building on #4932. 1. Record local RED fixture tests for ejection classes, unchanged heads, holds and the per-head retry cap. 2. Add a GITHUB_TOKEN-only requeue mode to the existing ai-rebase workflow and document it. 3. Run ./lint once and ./push with RED and GREEN together, open a Refs-only PR and arm pinned to HEAD. First PR expected this session; maintainer-assigned, backpressure limit 10.
26 remaining items
ci: reuse the prepared Tier 2 runtime image instead of re-downloading it
ci: reuse the prepared Tier 2 runtime image instead of re-downloading it
zackees commentedon Nov 18, 2026
Claiming: merge-queue selector request budget and probe diagnostics (pool slot obs-studio4), branch infra/select-budget. 1. Record local RED fake-API regressions for nine-run regroups, exhausted budgets, terminal runs/jobs and warning reasons. 2. Size the shared budget and conservatively reserve unprobed active runs, building on #5043. 3. Run ./lint, push RED/GREEN together and open the focused PR this session; arm only if backpressure is open. Refs #4839.
ci: size the merge-queue runner probe for queue regroups and log prob…
ci: fingerprint the running Python interpreter (Refs #4954 #4896)
zackees commentedon Nov 19, 2026
Claiming: workstream C member-only same-repository PR routing, explicitly assigned by the maintainer on 2026-11-19; slot obs-studio3, branch infra/dell-member-prs. 1. Record local RED eligibility, reserve, base-trust and secret-isolation tests. 2. Add default-off DELL_PR_ROUTING and MQ_PR_RESERVE while preserving merge-queue routing. 3. Run ./lint once, push RED and GREEN together through ./push, and open the requested Refs-only PR this session; arm only if backpressure is open. Decisions: membership/API uncertainty stays hosted; routing code and membership verification are loaded from immutable PR BASE.
test(ci): pin safe kept-tree configure reuse (Refs #4954 #4896)
ci: skip the redundant Tier 2 configure on the Dell's kept build tree
ci: fingerprint the running Python interpreter (Refs #4954 #4896)
zackees commentedon Nov 19, 2026
Claiming: local-gate timeout reporting fix in slot obs-studio3, branch fix/4896-local-gate-timeout (maintainer assignment local-gate-timeout). 1. Record local RED for bytes/None/string subprocess timeout output and complete failure aggregation. 2. Measure admission-verifier subprocess overhead under the parallel gate and implement the smallest supported resource/timeout fix. 3. Run ./lint and ./push, publish RED and GREEN together, and arm only if backpressure is open. First PR expected this session. Refs #4896.
zackees commentedon Nov 19, 2026
Claiming: Dell merge-queue overcommit and selector snapshot KeyError, branch infra/select-overcommit (select-overcommit pool assignment). 1. Record local RED fake-API cases for busy Dell admission and the missing-current-run snapshot. 2. Add a bounded MQ_RUNNER_MAX_RUNS cap (default 3), conservative snapshot handling and routing documentation, preserving #5279 PR reserves. 3. Run ./lint and focused tests, push RED/GREEN together and open the focused CI PR this session. Refs #4839.
zackees commentedon Nov 19, 2026
Claiming: admission-verifier self-test subprocess reduction in slot obs-studio3, branch infra/4896-attest-test-speed (maintainer assignment attest-test-speed). 1. Record local RED subprocess counts and serial/parallel timing. 2. Reuse isolated fixture objects without removing assertions; measure GREEN and compare assertion coverage. 3. Run ./lint and ./push with RED/GREEN together; arm only with open backpressure. First PR expected this session.