runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: P1R-C05, the fifth P1-RECOVERY unit. This PR does not close an issue.

Item	Value
Base	main at 3fac934: P1R-C04 with its follow-up #106, and #107, merged into this branch at 5a41aaf
Head	5deb102: the change in 272ca06, one runbook sentence in 8dbab2b, the merge of main in 5a41aaf, and the writer lock fix in 5deb102
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged
Decisions	D8, D12 and D15 at their recommended options (the decision column was blank when this unit started): an operator releases a lost Runner's occupation per process (D8); a required record is released only once DevGuard no longer charges its attempt, which the runbook has the operator read from DevGuard's administrator quiescence (D12); the reason field, its values and the status after a Runner loss are an MCP contract change for the owner to approve with the merge (D15)

Summary

Before: a gateway whose independent Runner was killed, crashed or stopped refused every request with RUNNER_DISCONNECTED, process_status and read_process included, though some of the Runner's processes might still run. The next gateway kept each workspace with an unended record of that Runner refused with RECOVERY_PENDING, and nothing could release it. An ended process that the Runner no longer kept was PROCESS_NOT_FOUND, and nothing said why output was missing.

After: read_process and process_status say why output is missing, or why what became of a process cannot be learned, in a new lost_reason: ring_overflow, retention_expired, runner_lost or host_rebooted. A gateway finds its Runner lost when nothing accepts connections at its socket or another worker answers there, marks each unended record of that Runner unknown, and answers for it from the record: state: unknown with runner_lost, which is never an end, or, after a restart of the host, exited with termination unknown and host_rebooted. Each such record keeps its workspace refused, and nothing of it runs again, until an operator confirms that nothing of it runs and releases it with the new codespace-mcp records list and records release. A new operations runbook, in English and Korean, says how to find what still runs, when a required record may be released given DevGuard's charges, and why a new Runner cannot register while DevGuard keeps the lost Runner's instance. The mode stays unsupported.

How.

• Contract types (crates/domain): LostReason, and lost_reason on ReadProcessResult and ProcessStatusResult, absent when there is no reason; ProcessState::Unknown, whose invariants require runner_lost and no termination, exit code or signal.
• Loss detection (crates/runner/src/uds.rs): claiming the Runner again after a lost connection now tells a lost Runner (nothing accepts connections at the socket, or another identity or another worker answers) from a Claim refused for a reason that waiting cannot end. The loss is reported once, as LinkEvent::RunnerLost and through UdsRunner::lost(), and every later request is refused with a RUNNER_DISCONNECTED that says the Runner was lost. lifecycle.rs gains host_boot_id, process_runs, whether a PID still runs as the process with a given start identity, and pid_runs, whether any process holds a PID; neither counts an exited process that is not reaped yet, on Linux and macOS.
• Records (crates/store): two columns, lost and released_at_ms, added in place with the schema version unchanged. record_runner_lost marks a Runner's unended records unknown for runner_lost or host_rebooted; an end recorded afterwards still ends a record; release_record releases one, and released records are no longer reconciled. unsettled_records_at reads a database read-only, so listing works while a gateway runs.
• Gateway (crates/server): ProcessRecords::follow_loss marks the records when the client loses the Runner, and process_status and read_process answer from the record when the Runner answers PROCESS_NOT_FOUND for a recorded execution or was lost (records.rs, mcp.rs). Reconciliation (recovery.rs) marks the unended records of any other Runner lost, as host_rebooted when that Runner ran on an earlier boot, and keeps their workspaces refused with a RECOVERY_PENDING that names the process and codespace-mcp records. An approval that such an execution resumed stays resuming, and operation_resume answers it with APPROVAL_AMBIGUOUS once the record is released.
• Operator commands (crates/server/src/operator.rs, config.rs, main.rs): codespace-mcp records list prints, as JSON, every record that may still hold a workspace, with whether its root process still runs as itself and whether a process holds its Runner's PID. records release <process_id>... releases a lost Runner's record once its root process no longer runs, only while no gateway holds the database's lock. It marks the records of a Runner that died while no gateway ran lost only once no process holds that Runner's PID, and those of a Runner of an earlier boot as host_rebooted. It ends no process and asks DevGuard nothing.
• Docs, English and Korean: runner isolation gains "Runner loss" with its answer table; operations gains the "Runner loss" runbook; error codes, agent integration, the execution substrate and DevGuard integration are updated, with the registry hashes recorded.
• Writer lock (crates/store/src/writer.rs): the database's writer lock now lets go when it is dropped. Closing its descriptor did not while a child that another thread was forking held a copy of it, so a records release right after a test let go of the lock could be refused; loops of the operator tests found this.
• CI (scripts/validate-upstream.py): the macOS leg also runs the gateway's operator tests, which read macOS's boot session and process start.
• A test fix outside the unit: the P1R-C02 test a_start_identity_names_one_process_until_it_is_reaped compared a child's start identity with its parent's. On Linux two processes that start within one clock tick have the same start time, so it failed when the test binary and its child started in the same tick, as a filtered run here showed; it now compares identities only with the same PID.

Contract changes

These need the owner's approval through the merge (D15).

• New MCP field lost_reason on read_process and process_status results, absent when there is no reason:
  • ring_overflow: only the most recent output is kept (output_lost on a read, output_retained_from above 0 on a status), in every mode;
  • retention_expired: an independent Runner no longer keeps the ended process, and the answer comes from its record, without output;
  • runner_lost: the Runner that ran the process was lost;
  • host_rebooted: the host restarted since the process started.
• New status value state: unknown, only after an independent Runner was lost before the process's end was recorded: no termination, exit code or signal, and lost_reason: runner_lost. It is not an end and not success.
• Answers from records (independent mode only): process_status of a process that the Runner no longer holds, or that a lost Runner held, answers from its record: exited as recorded (signaled without a recorded signal becomes termination unknown), unknown, or after a host restart exited with termination unknown; eof is true. read_process answers an empty chunk with eof and output_lost true, retained_from at the furthest output offset the gateway saw, and the cursor at that offset, or at the cursor asked for when that is further. Both were PROCESS_NOT_FOUND or RUNNER_DISCONNECTED before.
• Errors: no new code. RUNNER_DISCONNECTED for a lost Runner says so and that the records answer. RECOVERY_PENDING for a workspace with a lost Runner's record names the process, why, and codespace-mcp records, where before it said that another Runner was sent the process.
• Tool descriptions and instructions: read_process and process_status describe lost_reason and unknown; the independent mode's lifetime paragraph says what follows a Runner loss and not to start a duplicate.
• New operator commands: codespace-mcp records list and codespace-mcp records release <process_id>..., with --operations-db or CODESPACE_OPERATIONS_DB, JSON on stdout, and exit status 1 when a release was refused. Without a subcommand the gateway starts as before.
• Operations database: the processes table gains lost and released_at_ms when a gateway opens it; the schema version is unchanged. An earlier CodeSpace ignores the columns, so a record released here keeps its workspace refused there again, the safe direction.
• Behavior change to note: the next gateway now marks an unended record of another Runner unknown, with why; P1R-C04 left it running. Such a record becomes releasable.

Tests

The brief's cases and the tests that cover them.

Case	Tests
Output and exit re-queried normally, then retention expiry	gateway an_ended_process_is_answered_from_its_record_once_the_runner_let_it_go (the Runner's answer while it keeps the process, then the record's with retention_expired, the recorded exit and no output)
Runner crash: SIGKILL, a surviving pipe process	gateway binary a_killed_runners_processes_stay_held_until_an_operator_releases_them (SIGKILL of the real Runner; unknown with runner_lost, RUNNER_DISCONNECTED for new work, records list and release through the binary, refused while the gateway runs and while the root runs, released once it ended, then a new Runner and gateway work in the workspace). Without the Linux command sandbox the pipe process survives the Runner; under it, in a workspace without network, the command ends with the Runner, which the test checks instead
Host reboot (boot ID fixture)	gateway an_independent_gateway_answers_for_a_lost_runners_processes_until_released, an_independent_gateway_blocks_a_workspace_another_runner_left_work_in; operator records_release_takes_a_restart_as_the_end; store a_loss_names_a_restart_only_across_two_known_boots
Lost Runner's records until released	gateway an_independent_gateway_answers_for_a_lost_runners_processes_until_released (unknown/runner_lost, exited/host_rebooted, an end recorded before the loss, RECOVERY_PENDING naming the processes, release, then new work); store a_lost_runners_unended_records_wait_for_a_release
Races: exit and loss, reconnect, eviction	store a_lost_runners_unended_records_wait_for_a_release (an end recorded after the loss still ends the record); client a_client_that_cannot_claim_its_runner_again_is_disconnected (another Runner, then nothing at the socket: lost, not reclaimed); gateway an_ended_process_is_answered_from_its_record_once_the_runner_let_it_go (eviction)
PID reuse	operator records_release_waits_for_the_gateway_and_the_root (a root PID that a later process took), records_release_takes_a_restart_as_the_end; lifecycle a_process_runs_as_itself_until_it_exits (identity, zombie, reaped)
A Runner that died while no gateway ran	operator records_release_takes_a_runner_gone_unseen_as_lost, and records_release_waits_for_the_gateway_and_the_root (refused while a process holds the Runner's PID)
Original deadline unchanged, zero automatic executions	gateway an_independent_gateway_answers_for_a_lost_runners_processes_until_released (the deadline and the execution count stay), an_approval_whose_execution_a_lost_runner_held_is_never_run_again (APPROVAL_AMBIGUOUS after the release; the command's marker is written once)
Writer lock let go on drop	store a_writer_lets_go_though_a_copy_of_its_descriptor_lives_on (keeps a copy of the descriptor open across the drop; fails without the fix)
Contract types	domain reasons_and_the_unknown_state_cross_as_names, an_unknown_state_says_why_and_is_no_end; config the_records_commands_take_the_operations_database; store records_from_before_losses_are_listed_as_they_are_and_gain_the_columns, listing_records_reads_only_a_database_it_knows; operator records_list_shows_what_may_still_hold_a_workspace, records_commands_need_their_database

Verified here at 5deb102, in a Linux cloud container running as root, on the Codex checkout with its recorded patch applied and the worker, patch and Linux sandbox binaries built:

• cargo fmt --check for the workspace and for the worker.
• cargo clippy --locked --all-targets -- -D warnings in CI's six configurations: the workspace; the server, Runner and domain with devguard; the worker with and without devguard.
• cargo test --locked --workspace: 446 passed, none failed, in 34 test binaries, among them the 9 independent_runner tests and the 5 operator tests.
• -p codespace-server --features devguard: 173 passed. -p codespace-domain --features devguard --lib: 34 passed. The worker: 11 passed, and 2 with devguard.
• -p codespace-runner --features devguard --lib: 194 passed and 1 failed, launch::tests::nothing_is_committed_when_the_owner_cannot_launch, which fails only as root; this PR does not touch launch.rs, and the same test binary passed it when run as an unprivileged user.
• python3 scripts/check_docs.py check (20 paired documents), and after merging main, which brought only scripts: name a failed stage's tests in the job log #107's script and docs change, the scripts' Python tests (105, and the 36 CI policy tests).
• Under load: 32 runs of the gateway's library tests, 4 at a time, with no failure of this PR's tests. The P1R-C04 test an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree failed in 4 of them, as it does on main; its cause and a proposed test patch are in the PR's comment.

Not verified here: the hosted macOS job, which now also runs the operator tests; the docs site build, which needs Node 24.21.0; a real host restart (tests use a boot ID fixture); and a real DevGuard authority, so the runbook's DevGuard steps rest on DevGuard's source and docs at 7cd86e4.

Security scenarios

• No duplicate execution. Nothing is run again after a loss: a lost record is never settled as an end or as not started, and the approval its execution resumed stays resuming and is APPROVAL_AMBIGUOUS once released; the tests count executions and marker writes.
• No bulk release. An operator releases records one process ID at a time, only while no gateway runs, only a lost Runner's records, and never while the root process runs as itself. A later process with the root's PID is not taken for it, and a host restart ended everything of the boot before.
• A running Runner is not taken as lost. records release marks a Runner's records lost only once no process holds its PID. A PID that another process took keeps the records held, the safe direction; a gateway then finds the Runner lost at its socket.
• Uncertainty is never an end. unknown carries no termination or exit code, and host_rebooted is exited with termination unknown, never an exit code.
• DevGuard accounting. CodeSpace releases nothing in DevGuard and asks it nothing about a lost Runner's attempts; the runbook has the operator wait for DevGuard's own settlement before releasing a required record. records list prints IDs, PIDs, boot IDs and times, no argv, environment or secret.
• Rollback. Stop starting executions in the independent mode and keep process_status, read_process and the records; never close unresolved records as success in bulk. An earlier CodeSpace reads the database and ignores the new columns; a record released here keeps its workspace refused there.

Out of scope

• Restoring a lost Runner's I/O after a Runner restart, and any re-execution.
• Changing the codespace consumer's instance count, or anything in DevGuard. DevGuard has no command that only reads its quiescence (devguard admission --open also reopens a closed admission); that is proposed to the lead thread and the owner, not made here.
• A narrower operator path for a workspace whose records, Runner and DevGuard disagree while the Runner runs; the runbook says that stopping that Runner turns it into the lost-Runner case.
• The P1R-C04 test that terminates a sandboxed command while its sandbox may still be setting up, and the Runner's termination of such a command, which leaves it running; both are proposed to the lead thread with the PR's comment.
• The qualification harness (P1R-C06) and supporting the mode.
• Any Codex or DevGuard pin change.

[novelKR]

CI: the Linux Integration failure on 5a41aaf is a race in a P1R-C04 test, not this PR's change.

• Failing check: Rust / Integration (run 38728457392): mcp::independent_tests::an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree panicked in admitted (crates/server/src/mcp.rs:4706) after its 20 s bound, with RECOVERY_PENDING for proc-unrecorded.
• Why it is not this PR's: the test is P1R-C04's, and the race is between the Runner's termination and the Linux sandbox, which this PR does not change. It fails on main at 3fac934 as well: 5 of 64 runs with 8 copies in parallel in a Linux container, against 4 of 64 on this branch. It is probably also main's Integration failure on 950e2b2 (run 38718331468), whose test the job log did not name then.
• Cause: the test terminates its sandboxed /bin/sleep 30 about 100 ms after the Runner starts it. Under load the kill can land while bubblewrap is still setting up the sandbox. bubblewrap's sandbox process sets its parent-death signal (--die-with-parent) only just before it runs the command, so killing the launcher in that window leaves the sandbox running under PID 1. The sandbox runs sleep 30 to its end and keeps the Runner's output pipes open, and the Runner reports the exit only once the output ends, so the gateway keeps demo refused past the test's bound. A process listing taken after a failing run showed the sandbox's inner stage with parent PID 1 and its /bin/sleep 30 still running 20 s later. In passing runs the output ends within a few milliseconds of the kill.
• Proposed patch, not in this PR: the command prints a line, and the test terminates it only once that line is read, as the other independent tests already wait for their command before terminating it (an_independent_gateway_consumes_an_interrupted_resume_with_its_process). With it the test passed 80 of 80 runs at 8 in parallel on main.

Patch against main 3fac934

diff --git a/crates/server/src/mcp.rs b/crates/server/src/mcp.rs
index c7b35a7..12d2f8c 100644
--- a/crates/server/src/mcp.rs
+++ b/crates/server/src/mcp.rs
@@ -6287,7 +6287,7 @@ mod independent_tests {
         // A record that says this Runner started an execution it never received.
         let this = runner_ref(RunnerIdentity::this_process());
         left_running(&database, &this, "proc-phantom", None, "gov");
-        // A process that a controller started without recording it.
+        // A process that a controller started without recording it. It says when it runs.
         let unrecorded = ProcessId("proc-unrecorded".into());
         {
             let (client, server) = UnixStream::pair().unwrap();
@@ -6302,7 +6302,11 @@ mod independent_tests {
                 &raw,
                 ws,
                 RunnerExecRequest::for_host(
-                    vec["/bin/sleep".into(), "30".into()],
+                    vec[
+                        "/bin/sh".into(),
+                        "-c".into(),
+                        "echo started; exec /bin/sleep 30".into(),
+                    ],
                     unrecorded.clone(),
                     ws.profile,
                 ),
@@ -6353,6 +6357,24 @@ mod independent_tests {
             .0;
         assert_eq!(status.state, ProcessState::Running);
 
+        // Terminate it once it runs: a sandboxed command whose launcher is killed while its
+        // sandbox is still being set up runs on outside it, and keeps its output open.
+        let deadline = Instant::now() + Duration::from_secs(10);
+        loop {
+            let read = gateway
+                .read_process(Parameters(ReadProcessParams {
+                    process_id: unrecorded.clone(),
+                    cursor: 0,
+                }))
+                .await
+                .unwrap()
+                .0;
+            if read.chunk.contains("started") {
+                break;
+            }
+            assert!(Instant::now() < deadline, "{read:?}");
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
         gateway
             .terminate_process(Parameters(TerminateProcessParams {
                 process_id: unrecorded.clone(),

• The product side: a sandboxed Linux pipe process terminated inside that window keeps running where the Runner can no longer end it. Ending the sandbox's PID namespace rather than only its launcher would fix it, but that changes termination, which this unit must not, so it goes to the P1R lead thread with the test patch instead of widening this PR.

Separately, loops of this PR's new operator tests found a race of their own: the writer lock stayed held while a child that another thread was forking kept a copy of its descriptor. 5deb102 lets go of the lock when it is dropped, with a regression test. With it, 32 runs of the gateway's library tests at 4 in parallel had no operator failure, where 4 of 20 had one before; the C04 test above failed in 4 of the 32. CI runs again on that push.

---

Generated by Claude Code