scripts, docs: the independent Runner's qualification harness and protocol (P1R-C06)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: P1R-C06, the sixth and last P1-RECOVERY unit. This is its harness and protocol; the qualification run on the Mac follows the merge, and a docs-only report PR records the result. This PR does not close an issue.

Item	Value
Base	main at 345864d (P1R-C05 and the P1R-C04 follow-up #111)
Head	8de5fdb
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged
Decisions	D3: the harness starts each independent Runner itself, in its own session (setsid), with no LaunchAgent and no policy change. D7: Linux off is CI functional verification only, never a Linux qualification, and the CS-RG suites rerun on the same candidate once the Runner is drained. D12: the codespace consumer keeps one instance, so each measurement interval registers a fresh Runner and drains it before the next

Summary

Before: the independent Runner that P1R-C01 to C05 built had no qualification. The harness covered only the CS-RG modes, nothing exercised a gateway restart against a real Runner with DevGuard or measured restarts while processes ran, and the docs said the mode was not supported yet without saying what would make it supported.

After: scripts/qualify-devguard.py has six more suites. Five functional suites, one per unit, test each unit's behavior end to end against a real independent Runner that the harness starts as an operator would: 34 scenarios with participation off everywhere, 21 more with required on macOS against DevGuard's fixture authority, and two gaps that every run records as not run. The recovery suite runs them all and then measures gateway restarts against the installed DevGuard service for whole idle and loaded intervals, three repetitions of 10 and 30 minutes under the full protocol. The new Gateway recovery qualification protocol, in English and Korean, says what is claimed, how each claim is shown, what passes and what evidence a run leaves; it is frozen with this harness. Operations gains "Independent Runner adoption and rollback". The mode stays unsupported everywhere until the Mac run qualifies a combination.

How.

• Functional suites (scripts/qualify-devguard.py): recovery-lifecycle (every way a gateway ends, original deadlines across restarts, the Runner's own SIGTERM, refused settings), recovery-store (records, the writer lock, schema refusal, records list and release), recovery-fence (a foreign connection, stale epochs, a stopped gateway, eight concurrent Claims, another database, concurrent gateways), recovery-reconcile (running and ended processes, approvals with a resume in flight, an authority outage, a missing record, gateways killed 0 to 160 ms into exec_command and terminate_process) and recovery-observation (ring overflow, retention, Runner loss, a replaced Runner). Each scenario has its own Runner, operations database and work directory, and with required its own fixture authority with a one-instance consumer. Commands carry tags whose marker files count how often each ran.
• Restart measurement (recovery): each interval starts a fresh Runner that registers as the consumer's one instance and holds a governed cat (pipe) and sleep (PTY), a PTY that writes numbered lines and a pipe that bursts past the 256 KiB output ring on cue. Gateways are ended in turn with their input closed, SIGTERM, SIGUSR1 and SIGKILL. In every cycle the next gateway claims under the next epoch and finds the processes running; another connection's Claim and its mutating requests under three epochs must be refused as stale; a rival gateway must give up for the writer; the approval granted before the restart is resumed once by the next gateway, also when the gateway before was killed 0 to 160 ms after sending its resume; and a timeout target's deadline falls either while no gateway runs or while one serves. The control path is sampled on the control suite's schedules while a gateway serves, and slots without one are recorded as restarting. The interval ends by draining the Runner through a gateway with the exit intent stop.
• Verdicts (scripts/qualify_analysis.py): restart, fence, rival, timeout (0.25 s early, 2 s late, recorded deadline unchanged), approval, execution and output-continuity metrics, the drain, and coverage shortfalls. verify recomputes them from the raw rows. A run that is not governed on macOS is named independent-off and never qualifies.
• Host helpers (scripts/qualify_host.py): the Runner process (started with setsid, its log parsed for Claims), a minimal client of the runner wire (protocol 11) for the other connection, gateway ends by signal, a PID watch, running (a PID that is not a zombie), and read-only queries of the operations database.
• Claims and wording: CLAIMS gains the independent mode's four cells. The guides (runner isolation, architecture, security model, agent integration, DevGuard integration, operations), the gateway's --runner-lifecycle help and its Claim log line now say that the mode is supported only on the combinations its qualification qualifies, so the report PR stays docs-only and the candidate frozen at this merge holds.
• Docs in English and Korean, with the registry hashes recorded.

Contract changes

None to MCP tools, error codes, status values or IDs. Two texts change: the gateway's --runner-lifecycle help with the RunnerLifecycleMode::Independent doc comment, and the gateway's log line for a Claim, which no longer adds "it is not a supported mode yet". The harness gains suites and --participation; the existing suites and their evidence are unchanged.

Tests

Verified here at 8de5fdb, in a Linux cloud container running as root, with the Codex recorded patch applied:

• python3 -B -m unittest discover -s scripts/tests: 117 passed, 11 of them new in test_qualify_devguard.py (Runner log parsing, the wire client against a fake socket, output continuity, timeouts, restarts, approvals and markers, the drain, the Serving gate, argument checks, interval verdicts, and verify of a synthetic sealed recovery run, as written and falsified).
• pyflakes on the four scripts; cargo fmt --check; cargo clippy --all-targets -- -D warnings for the workspace and for codespace-server with devguard.
• scripts/check_docs.py (21 paired documents), and the docs site with Node 24.21.0: npm test (5 passed), npm run build and site.py check (45 HTML pages).
• The harness on Linux against this branch's debug binaries, recovery --rehearsal --participation off: all 34 off scenarios passed, and the two gaps were recorded as not run. In the 2-minute idle interval, 5 restarts ended gateways in all four ways; 90 fenced requests changed nothing; none of the 6 rival gateways served; 4 timeout targets ended at most 65 ms after their original deadlines, 2 of them while no gateway ran; 5 approvals were resumed once each, one after its gateway was killed with the resume in flight; no tagged command ran twice; the PTY's 627 lines arrived whole, and the pipe's burst lost only the 22,524 lines that the Runner declared lost to ring_overflow; and the drain claimed under epoch 7 and stopped the Runner with nothing left to terminate. The load interval was inconclusive, as it must be without DevGuard's service to run DG-1's load, so the run is inconclusive (exit 2), and verify recomputed every verdict unchanged.

Not verified here: anything with required, since DevGuard governs only on macOS; the measurement against the installed service and under DG-1's load; and the full protocol's durations. The Mac run does these.

Security scenarios

• Credentials: the harness never reads a credential. Each Runner is given the credential file's path, and the fixture authorities use throwaway credentials in their own scratch directories. No evidence directory holds a credential, its digest or a child's environment.
• The installed service: nothing is installed and no policy or consumer setting changes. Each interval drains its Runner so that no codespace instance or charge is left, and an interval that finds one active as it begins is inconclusive.
• No duplicate execution and no stale mutation are counted, not assumed: a marker line per run of each tagged command, with the service the launches each gateway logged per execution, and the answer to every fenced request.
• Rollback honesty: the runbook's rollback stops the gateways in independent mode, drains the Runner (Stop, instance retired) and returns to the previously qualified modes with the same operations database; a lost Runner goes to the Runner loss runbook, never to a bulk close.

Out of scope

• The qualification run and its report, which follow this merge.
• Restoring processes after the Runner itself restarts (unsupported, recorded as a gap), a real host restart (a gap), and making the independent mode the default (ADR-002, a separate review).
• Any DevGuard release, policy, consumer or pin change, and any Codex pin change.