runner, server, store: Claim and controller epoch for the independent Runner (P1R-C03)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: P1R-C03, the third P1-RECOVERY unit. This PR does not close an issue.

Item	Value
Base	main at 5976df0, with P1R-C02 merged
Head	4e54309
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged
Decisions	D5, D6 and D15 at their recommended options (the decision column was left blank, checked again at the start)

Summary

Before: an independent Runner served one connection at a time: the next gateway's connection waited, unanswered, until every request of the previous one had ended, and nothing checked who connected beyond the socket directory's permissions. Requests carried nothing that tied them to a controller. A gateway that lost its connection refused every later request with RUNNER_DISCONNECTED until it was restarted.

After: one connection at a time controls the Runner, after a Claim that names the gateway's operations database, and every request that changes a process or the workspace carries the controller epoch of its Claim. The Runner refuses a request from an earlier epoch with STALE_CONTROLLER, and nothing it asked for is done. A gateway that loses its connection claims the same Runner again without a restart, refusing requests meanwhile with RUNNER_RECONNECTING. A request refused with one of these codes never ran: it consumes no approval and leaves no patch result for its operation_key. The PR1 barrier stays global: a controller after one that started work, the same gateway after a lost connection included, starts nothing (RECOVERY_PENDING). While no gateway controls the Runner, any process of the same user that can reach its socket can claim it; the operator docs and the new mode's MCP instructions say so. The mode stays unsupported.

How.

• Claim (crates/runner, wire.rs): the Runner serves every connection it accepts. It grants a Claim only when the connection's peer credentials (Tokio's peer_cred: SO_PEERCRED on Linux, getpeereid on macOS) show its own user, and only for the operations database of its first granted Claim. It refuses a Claim while another connection controls it and is open, a stopped gateway's included (D6: nothing pushes a live controller out). Once that connection has ended, a Claim waits up to CLAIM_WAIT_BOUND (10 s) for the requests it sent, the Prepare and Launch that run beside the others included, and is refused past that bound with nothing changed. A granted Claim raises the epoch by one (it never decreases while the Runner lives) and answers with every process the Runner holds, each with its workspace and status, and events_through, the number of the last exit event that list reflects. Refusals name a reason: peer_uid, store_mismatch, controller_active, requests_running or stopping. The Runner logs each Claim it grants or refuses with the peer's pid.
• Fence (wire.rs): a connection that does not control the Runner gets answers to Hello, Claim and Detach only. exec, apply_patch, write_stdin, resize, terminate, TerminateWorkspace, Stop, and with DevGuard Prepare and Launch must carry the current epoch, or the Runner answers STALE_CONTROLLER. Exit events are numbered, go to the controller only, and an event whose write failed goes to the next controller.
• store id (crates/store): the operations database keeps a random identity in a new store_identity table, created the first time it is asked for; the schema version stays 1. It guards against a gateway started with the wrong database and is not a credential, as the database's owner can read it.
• Gateway (uds.rs, crates/server): at start the gateway claims the Runner, asking again with a growing pause for up to 30 s while another connection controls it or that connection's requests run, and refuses to start on any other refusal. When its connection is lost, or it closed it after a call passed its 120 s deadline, it connects again, checks that the same Runner (by its identity) answers, and claims it again: at once, then after 100 ms, doubling up to 5 s. Meanwhile requests that need the Runner get RUNNER_RECONNECTING. When nothing accepts connections at the socket, another Runner answers there, or the Runner refuses for a reason waiting cannot end, the gateway stops and refuses them with RUNNER_DISCONNECTED. Each exit is handed on once, by its event number. Each execution's record (PR2) keeps the epoch it was sent under.
• Approvals and idempotency (crates/server/src/mcp.rs, crates/store): whether new work can be sent is checked before claim_resume (risk 15) and again once the workspace is held. A refusal with STALE_CONTROLLER, RECOVERY_PENDING, RUNNER_RECONNECTING or RUNNER_DISCONNECTED proves that nothing ran: a resuming approval is given back with return_resume, as CSRG-U5 does for a governed refusal, a queued one stays queued, a patch's operation is forgotten so its operation_key begins afresh, and an execution is recorded as not started.
• Worker binary (crates/codex-runtime): the independent Runner binds Tokio's UnixListener, which codex-uds wraps on Unix, to read each connection's peer credentials, and logs each Claim.
• Docs, English and Korean: runner isolation gains "Claim and controller epoch", the reconnect, and "Who can take control" (the D6 exposure); error codes, operations, architecture, agent integration, the security model, Codex reuse and DevGuard integration are updated, with the registry hashes recorded.

Contract changes

These need the owner's approval through the merge (D15).

• Worker protocol 9 → 10, still matched exactly.
  • New request claim ({"store_id": …}), answered claimed (epoch, previous_controller, processes: [{workspace_id, status}], events_through) or claim_refused (refusal: peer_uid | store_mismatch | controller_active | requests_running | stopping, message). A managed worker refuses claim as it refuses detach and stop.
  • The envelope gains epoch on requests and seq on events, both left out when absent.
  • hello no longer carries previous_controller and inherited_processes; the Claim's answer replaces them.
  • An independent Runner answers a request outside the current Claim with STALE_CONTROLLER.
• MCP error codes (independent mode only):
  • New RUNNER_RECONNECTING: the gateway is claiming its Runner again; the request was not sent and consumed no approval; retry after a pause.
  • New STALE_CONTROLLER: the Runner refused a request sent under an earlier Claim; nothing was done, and the approval it resumed is granted again.
  • RUNNER_DISCONNECTED now means that the gateway cannot claim its Runner again; a lost connection alone gives RUNNER_RECONNECTING.
  • RECOVERY_PENDING also follows a reconnect after this gateway started work.
• MCP instructions (independent mode only): the lifetime paragraph now states the single controller, the reconnect and the codes above, that these refusals consume no approval and keep no patch result, and that while no gateway controls the Runner, any process of the same user that can reach its socket can take control of it.
• Approvals: these four codes no longer consume an approval being resumed.
• Operations database: new table store_identity; schema version unchanged; processes.controller_epoch is now filled.
• Gateway start: with --runner-lifecycle independent, the gateway waits up to 30 s for another controller, then refuses to start, and refuses at once a Runner claimed for another operations database. No option changes.

Tests

The brief's cases and the tests that cover them.

Case	Tests
Normal handover	runner the_next_gateway_inherits_the_processes_but_starts_nothing (next epoch, the inherited processes, the original timeout), a_claim_states_the_inventory_and_the_exits_it_reflects, a_controller_that_starts_nothing_leaves_no_barrier; worker binary an_independent_runner_outlives_its_gateways_until_one_stops_it (epochs 1 and 2 with the inventory and the Runner's Claim log)
Wrong UID (injected for the test)	runner a_claim_from_another_user_is_refused
Reuse of an earlier session, and a late mutation from the previous controller	runner requests_outside_the_current_claim_are_refused (no epoch, another epoch, and a request prepared under the earlier Claim that arrives after the next one), a_client_claims_its_runner_again_after_a_lost_connection; gateway a_request_refused_as_stale_never_ran_and_gives_its_approval_back
Another store id	runner a_claim_for_another_operations_database_is_refused; store an_operations_database_keeps_its_identity; gateway binary a_gateway_with_another_operations_database_is_refused
Two gateways claiming at once	runner of_claims_made_at_once_one_wins
Claim refused while a live controller exists; handover after a SIGSTOPped gateway ends	runner a_claim_is_refused_while_the_controller_is_open; gateway binary a_claim_is_refused_while_a_stopped_gateway_holds_the_runner_and_granted_once_it_ends
Retry after a lost Claim answer	runner a_claim_whose_answer_is_lost_is_asked_again
Claim wait bound (risk 12)	runner a_claim_waits_for_the_requests_of_the_controller_before_it, a_claim_past_its_wait_bound_is_refused_and_changes_nothing
A command in the Linux sandbox cannot reach the Runner socket	gateway binary a_sandboxed_command_cannot_reach_the_runner_socket (restricted and enabled network; the same command outside the sandbox connects)
Zero duplicate executions and reservations while reconnecting	runner a_client_claims_its_runner_again_after_a_lost_connection (nothing runs twice, each exit once), a_runner_claimed_by_another_connection_meanwhile_is_claimed_once_it_ends, a_client_that_cannot_claim_its_runner_again_is_disconnected; client an_exit_event_is_delivered_once; gateway a_gateway_refuses_new_work_while_it_claims_its_runner_again (no record, no reservation, the approval unconsumed), a_gateway_that_started_work_claims_its_runner_again_and_starts_nothing
Approvals and idempotency	store queued_scheduler_errors_do_not_consume_approval (the four codes), an_abandoned_operation_begins_afresh; gateway a_request_refused_as_stale_never_ran_and_gives_its_approval_back
Managed worker unchanged	runner a_managed_worker_refuses_claim_detach_and_stop, an_independent_runner_answers_registration_before_the_next_claim

Claim latency (risk 12), measured in this container:

• A handover after a detach: the next Claim was granted in 0.23 ms (the_next_gateway_inherits_the_processes_but_starts_nothing).
• A Claim asked while the earlier controller's write still ran: it waited 1.18 s, until that write ended with its process's 1.5 s timeout, within the 10 s bound (a_claim_waits_for_the_requests_of_the_controller_before_it).
• Past a wait bound of 300 ms set for the test: refused after 320 ms, with nothing changed (a_claim_past_its_wait_bound_is_refused_and_changes_nothing).
• A Claim whose answer was lost: granted again after 115 ms, the gateway's first 100 ms pause included (a_claim_whose_answer_is_lost_is_asked_again).
• Real binaries: a SIGSTOPped gateway kept control and another Claim was refused; once that gateway was killed, a Claim was granted 2.2 ms later (2.9 ms in an earlier run) (a_claim_is_refused_while_a_stopped_gateway_holds_the_runner_and_granted_once_it_ends).

Verified locally (cloud Linux, Rust 1.99.0):

• cargo fmt --check for the root and every adapter, python3 scripts/check_docs.py ("Documentation registry verified: 20 paired documents"), the script tests (102), scripts/check-no-model-deps.sh and scripts/check-upstream-pin.sh (Codex 6b9826e with its 1 recorded patch).
• clippy with -D warnings: the root with --all-targets; the server, runner and domain with devguard; the worker with and without devguard; the patch adapter.
• cargo test --workspace, with the worker, the patch helper and the Linux sandbox helper built and bubblewrap installed, so commands ran sandboxed as in CI's integration stage: 413 passed, none failed, independent_runner (8) included.
• cargo test -p codespace-server --features devguard, with the DevGuard worker, fixture authority and launch helper built as CI's integration stage builds them and CODESPACE_REQUIRE_DEVGUARD_BINS=1: 157 passed, none failed. The 5 ignored tests are the existing child-process tests that their parent tests run.
• The worker's tests (runtime_binary: 9 passed, its handshake test now expecting protocol 10) and its DevGuard binary tests (2 passed); cargo test -p codespace-domain --features devguard --lib: 32 passed; the patch and DevGuard adapters' tests passed.
• cargo test -p codespace-runner --features devguard --lib: 184 passed and 1 failed, launch::tests::nothing_is_committed_when_the_owner_cannot_launch. As on runner, server: opt-in independent Runner that outlives its gateways (P1R-C01) #102 and store, runner, server: process records for the independent Runner (P1R-C02) #103, it fails here only because the cloud container runs as root; CI runs it as a non-root user.
• After the last change, the Runner's wire tests ran four more times with and without devguard (31 and 59 tests each time), and the gateway's independent tests once more (15): none failed.

Not verified locally: the macOS legs and the documentation site build; CI covers both.

CI: passed on 4e54309: every job of run 38631018229 (17 jobs, the integration stage, Linux isolation and the macOS contracts included) and the documentation site's run 38631019062.

Security scenarios

• Same user only. The Runner reads the peer's user from the connection's credentials and refuses another user's Claim; a connection that never claimed gets nothing but Hello, Claim and Detach answers, and no exit event.
• No push-out (D6). A live controller keeps control, even a stopped one: a Claim is refused until its connection ends, and the next gateway asks again. Pushing a live controller out and checking the peer's executable are not built.
• Exposure that remains (D6), stated. While no gateway controls the Runner (between gateways, or while one reconnects), any process of the same user that can reach the socket can claim it: it then holds the Runner's processes, and can start work when no controller before it did. A command that the Linux sandbox runs cannot connect to the socket (tested with a restricted and an enabled network); an unsandboxed command can, on macOS and on Linux without the helper. The runner isolation guide ("Who can take control"), the security model, the DevGuard integration status and the new mode's MCP instructions say so.
• store id is not a secret. It is a mistake guard; the docs say so, and it appears in refusal messages and the Runner's log.
• Stale requests do nothing. A request under an earlier epoch is refused before the Runner acts on it, a stop included.
• Approval and idempotency honesty. Only refusals that prove nothing ran give an approval back or forget a patch operation; a request whose answer was lost keeps its uncertain result (dispatch_status: unknown, APPROVAL_AMBIGUOUS).
• Rollback. Stop starting gateways in the independent mode (no new Claims), stop the Runner through its current controller (--runner-exit-intent stop) or with SIGTERM so it ends and drains its processes, then start the earlier version's Runner and gateway. The epoch is never decreased: a Runner's epochs only rise, and a new Runner is a new identity in the records. Keep the operations database; the earlier version leaves store_identity alone.

Out of scope

• Reconciling occupancy and approvals from the records, and a per-workspace barrier in place of the global one (P1R-C04).
• Pushing out a live controller and checking the peer's executable (D6's other options); two lanes in place of one connection (D5's other option).
• The unknown phase and gap reasons (P1R-C05), the qualification harness (P1R-C06), and supporting the mode.
• Any Codex or DevGuard pin change.