runner, server, store: reconcile the independent Runner per workspace (P1R-C04)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: P1R-C04, the fourth P1-RECOVERY unit. This PR does not close an issue.

Item	Value
Base	main at 1fda789, with P1R-C03 merged
Head	7cabcba
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged
Decisions	No new decision. D11 is kept: PTY handles are released exactly when they were before. D1–D15 at their recommended options (the decision column was left blank, checked again at the start)

Summary

Before: a gateway that claimed an independent Runner after any controller had started work on it, its own connection before a lost connection included, refused every new execution, patch and approval resume with RECOVERY_PENDING until the Runner was stopped and a new one started: in every workspace, and even after the inherited processes had ended. Its process records were written but never read back. An approval left resuming stayed APPROVAL_AMBIGUOUS, a process that was still running held no workspace in the gateway that inherited it, and an exit that no gateway received was kept by the Runner only until the next controller's exit event.

After: after each Claim, the gateway reconciles what earlier controllers left, workspace by workspace, from three accounts: its process records, what the Runner holds and remembers of each process it was asked to start, and for a required execution its DevGuard attempt, which the Runner looks up with its own session. Until a workspace is reconciled, the gateway and the Runner both refuse new executions, patches and approval resumes there with RECOVERY_PENDING, before the approval is claimed, so it stays as it was. Other workspaces take work at once, and inherited processes can always be queried, read, written to, resized and terminated, DevGuard down included. A process found running holds its workspace again, so new work there gets WORKSPACE_BUSY; one that ended is recorded with how it ended; one that the Runner proves it never received or created is recorded as not started. An approval left resuming is consumed with the process the Runner created for it, granted again when the Runner proves that it created none, and otherwise left resuming. A workspace whose accounts disagree, whose record names another Runner, or where the Runner holds a running process that no record accounts for stays refused, with the reason in the error and the log. The mode stays unsupported.

How.

• Runner memory and retention (crates/runner, process.rs): an independent Runner remembers, for its whole life and by ID alone, whether it created a process for each process_id it was asked to start, or received the request and created nothing. It keeps each ended process, with its status and output, until a controller acknowledges that its record says how the process ended. Handles are released exactly when they were before, after 15 minutes or beyond 64 ended processes (D11, risk 7); of the processes whose handles were released it keeps at most 64 (DEFAULT_MAX_UNACKNOWLEDGED), dropping the oldest first, and of a dropped one it remembers only that it created it.
• Wire (wire.rs, uds.rs, admission.rs): new controller requests recall, reconciled, acknowledge and, with DevGuard, lookup, which looks an attempt up only when this Runner admitted it for that process, with the admission it asked for. The barrier is now per workspace: the Claim's answer names the workspaces where a controller before it started work, and the Runner refuses new work in each until the controller states that it reconciled it. stop answers with how each process ended. The client skips exit events that the Claim's inventory already reflects (seq ≤ events_through), since the reconciliation accounts for them. A write that fails, as one sent after the Runner closed the connection does, no longer discards the answers the Runner wrote before it closed: an exit's acknowledgement sent while the gateway stopped its Runner could cost the stop its answer, and the gateway then exited with an error.
• Reconciliation (crates/server/src/recovery.rs): at each Claim the gateway bars each workspace of the barrier, each one where the Runner holds a running process, and each one that a record left to reconcile names: a record that is not terminal and was not sent to this Runner under this Claim or a later one, or one of an approval still resuming. Each pass recalls those processes and the running ones no record accounts for in one request, looks up the governed attempts, decides each record by the brief's table, settles the approvals, tells the Runner which workspaces are reconciled and acknowledges the ends it recorded. A record that names another Runner is never asked about, since a Runner proves "never received" only for itself; it keeps its workspace blocked for P1R-C05. Passes run again after each exit while a workspace is refused, and after a pause growing from 500 ms to 30 s while one waits for an answer.
• Approvals and leases (mcp.rs, records.rs, crates/store): operation_resume checks the approval's workspace before claim_resume (risk 15). An interrupted exec resume whose record proves that the Runner started it is answered with that process_id instead of APPROVAL_AMBIGUOUS. settle_interrupted_resume consumes or gives back only a resuming approval, under the in-flight guard, and never one that an execution under the current Claim resumes. hold_process holds a workspace for a process found running and fails queued waiters with WORKSPACE_BUSY; exits noted during a pass keep a process that exited after the Runner reported it running from being held again.
• Records (records.rs, crates/store): each end recorded from an exit event is acknowledged to the Runner, a stop records how each process ended, and records_to_reconcile now takes the Runner: records of other Runners are always reconciled, because epochs are per Runner.
• Docs, English and Korean: runner isolation gains "Reconciliation" with the table and "Ended processes"; error codes, operations, architecture, agent integration, the execution substrate, the security model, Codex reuse and DevGuard integration are updated, with the registry hashes recorded.

Contract changes

These need the owner's approval through the merge (D15). No new MCP error code, field, tool or option.

• Worker protocol 10 → 11, still matched exactly.
  • New controller requests: recall (process_ids), answered recalled with, per process, not_received, refused, held (workspace_id, status) or dropped; reconciled (workspace_ids); acknowledge (process_ids), answered acknowledged; with DevGuard, lookup (attempts: [{attempt_id, process_id}]), answered looked_up with not_admitted, not_found, found (the attempt's state) or failed. A managed worker refuses all four; a connection that does not control the Runner gets STALE_CONTROLLER; reconciled and acknowledge carry the epoch.
  • The Claim's answer gains barrier (absent reads as none), and previous_controller now means that barrier is not empty.
  • stop answers stopped with processes, as the Claim lists them.
  • The Runner refuses new work with RECOVERY_PENDING only in a workspace of the barrier, until reconciled names it.
• MCP behavior (independent mode only):
  • RECOVERY_PENDING is per workspace and lifts once the gateway has reconciled the workspace. Its message says why it waits: when the gateway waits for an answer it tries again by itself, so retrying after a pause helps; when the accounts disagree or a record names another Runner, an operator resolves it.
  • A process found running holds its workspace again: new work there returns WORKSPACE_BUSY instead of RECOVERY_PENDING.
  • operation_resume of an approval left resuming by an exec: consumed with the execution's process_id and dispatch_status: confirmed when its record proves that the Runner started it; granted again, resumable, when the Runner proved that it created none; APPROVAL_AMBIGUOUS otherwise, as before.
• MCP instructions (independent mode only): the lifetime paragraph replaces "refused until the Runner is stopped and a new one is started" with the reconciliation, the operator case and WORKSPACE_BUSY for a process found running.
• Behavior change to note: a workspace whose unsettled record names a Runner that stopped without reporting or was lost (SIGKILL, a crash, a reboot) now stays refused with RECOVERY_PENDING for every later Runner's gateway, as the brief requires (another identity: keep the occupancy, P1R-C05). Before this PR, a new Runner started work there. A stop through the gateway (--runner-exit-intent stop) records how each process ended and leaves nothing to block. The operator's release comes with P1R-C05's runbook.
• Logs: the Runner's Claim log says that it starts nothing where an earlier controller did until that workspace is reconciled; the gateway's warning names the barrier, and it logs each workspace that it refuses, with the reason, and each one it reconciles.
• Operations database: no schema change; the records are now read back, and records_to_reconcile includes other Runners' records whatever their epoch.

Tests

The brief's cases and the tests that cover them.

Case	Tests
Normal running reconnect	gateway an_independent_gateway_after_a_detach_holds_the_process_it_inherits (busy for exec and apply_patch, released once terminated, a new exec under epoch 2), a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it; gateway binary an_independent_runner_keeps_a_killed_gateways_process_to_its_original_timeout, a_claim_is_refused_while_a_stopped_gateway_holds_the_runner_and_granted_once_it_ends (now WORKSPACE_BUSY); runner the_next_gateway_inherits_the_processes_and_starts_work_once_it_reconciled
Terminal reconnect	gateway an_end_while_no_gateway_is_connected_is_recorded_by_the_next_gateway, an_end_a_gateway_missed_is_recorded_when_the_next_gateway_reconciles; runner a_lost_connection_keeps_the_processes_and_the_next_claim_accounts_for_exits, client an_independent_claim_accounts_for_the_exits_before_it
Stale approval	gateway an_independent_resume_left_resuming_is_consumed_with_the_process_it_started, an_independent_gateway_consumes_an_interrupted_resume_with_its_process, an_independent_gateway_gives_back_an_approval_whose_execution_never_reached_the_runner; store an_interrupted_resume_is_settled_once
Authority failure	gateway an_independent_gateway_waits_for_the_resource_authority_before_it_reconciles (DevGuard: lookups fail, the workspace stays refused while the process stays readable, then the attempt is found launched and the workspace is held for it); runner an_independent_runner_looks_up_only_the_attempts_it_admitted (a test authority's answers: launched, released unlaunched, not found, unknown, unreachable, and no registration), an_independent_runner_looks_up_attempts_for_its_controller_only
Record mismatch	gateway an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree (a record that says running for a process this Runner never received keeps its workspace refused; a running process that no record accounts for keeps its own refused until it is terminated; a third workspace takes work at once)
Another Runner identity	gateway an_independent_gateway_blocks_a_workspace_another_runner_left_work_in; store a_gateway_reconciles_what_earlier_claims_left_unsettled (other Runners' records at any epoch)
Exits and events overlapping new requests while reconnecting	gateway a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it (one process exits during the disconnect and one keeps running: refused, then admitted in one workspace and busy until the exit in the other); store a_process_found_again_holds_its_workspace
The approval stays after a barrier refusal	gateway an_independent_gateway_blocks_a_workspace_another_runner_left_work_in (the approval stays granted), a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it
Zero duplicate executions	the resume tests above (the command's marker file is written once across two resumes and the reconciliation)
A late request while the Runner stops	client an_answer_written_before_the_runner_closed_outlives_a_later_failed_write; gateway binary a_gateway_refuses_independent_settings_or_a_runner_that_do_not_fit and a_sandboxed_command_cannot_reach_the_runner_socket, which failed about one run in seven before the fix and passed 40 runs of 40 after it
Retention (risk 7, D11)	runner an_independent_runner_keeps_an_ended_process_until_it_is_acknowledged, an_independent_runner_evicts_an_acknowledged_process_as_before, an_independent_runner_drops_the_oldest_released_process_beyond_its_bound, an_independent_runner_remembers_each_process_it_was_asked_to_start, an_independent_runner_recalls_whether_it_created_each_process; worker binary an_independent_runner_outlives_its_gateways_until_one_stops_it (the barrier and the stop's report)

Verified here (Linux container running as root, toolchain 1.99.0, on 7cabcba):

• cargo fmt --check for the root workspace and the worker: clean.
• cargo clippy --all-targets -- -D warnings in the six CI configurations (root; gateway, Runner and domain with DevGuard; worker with and without DevGuard): clean.
• cargo test --workspace with the worker, patch and Linux sandbox helpers: every target passes, among them the Runner lib (152), the gateway lib (52), the gateway binary's independent Runner tests (8) and the store (71).
• Gateway with DevGuard and DevGuard's binaries (CODESPACE_REQUIRE_DEVGUARD_BINS=1): 163 passed; the 5 ignored are child processes that their parent tests run.
• Runner lib with DevGuard: 193 passed. launch::tests::nothing_is_committed_when_the_owner_cannot_launch fails only as root, because it needs a group other than root's; this PR does not touch launch.rs.
• Domain lib with DevGuard: 32 passed. Worker tests, with and without DevGuard: pass.
• scripts/check_docs.py: 20 paired documents verified.

Not verified here: the hosted macOS job, and the docs site build, which needs Node 24.21.0; CI runs both.

Security scenarios

• No duplicate execution. Nothing is replayed: a record is settled only from what the Runner holds or remembers, and an approval is granted again only when the Runner proves that it created nothing for it (not_received or refused of a dispatching record, with DevGuard also no attempt or one that ended unlaunched).
• Proof stays with its Runner. A Runner proves "never received" only for itself; a record that names another Runner identity keeps its workspace refused, and its approval stays as it is.
• Approval honesty. The barrier is checked before claim_resume; a consumed approval, or one whose execution may have started, is never granted again; settling an approval takes the in-flight guard and leaves one that an execution under the current Claim resumes to that resume.
• No silent release. A workspace that cannot be reconciled is never released in bulk: its process holds it, or it stays refused, with the reason in RECOVERY_PENDING and the log. Inherited processes stay readable and terminable whatever the reconciliation's state, DevGuard down included.
• Same controller only. recall, reconciled, acknowledge and lookup are answered to the controlling connection only; reconciled and acknowledge carry its epoch. The D6 exposure, any process of the same user claiming the Runner while no gateway controls it, is unchanged and stays documented.
• No secret in records or answers. recall answers with IDs and statuses, lookup with attempt IDs and states; DevGuard's credential stays with the Runner.
• Rollback. Stop starting new work in the independent mode, query, terminate and reconcile the existing processes, and stop the Runner through its gateway so it reports how each process ended; do not release busy workspaces in bulk. Then run the earlier version's Runner and gateway. Keep the operations database.

Out of scope

• What follows the loss of a Runner, the operator's runbook to release a workspace that a lost Runner's record keeps refused, and the gap and uncertainty reason fields (P1R-C05).
• Changing when PTY handles are released (OX-02, D11): handles go when they did before, and only status and output are kept longer.
• The qualification harness (P1R-C06) and supporting the mode.
• Any Codex or DevGuard pin change.