store, runner, server: process records for the independent Runner (P1R-C02)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: P1R-C02, the second P1-RECOVERY unit. This PR does not close an issue.

Item	Value
Base	main at feb5f52, with P1R-C01 merged
Head	a4f362c: the change 82262c8 and a test fix for the Linux sandbox
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged
Decisions	D4 at its recommended option (the decision column was left blank)

Summary

Before: a gateway connected to an independent Runner knew its executions only in memory. Once it was restarted or killed, nothing said which executions it had sent, to which Runner, for which approval, or how they ended. Any number of gateways could write to the same operations database, and the database did not say which CodeSpace version wrote it.

After: a gateway with an independent Runner records each execution in its operations database before it sends the execution to the Runner, and follows it to its end. A gateway that lost its memory finds the same execution by its process_id: the Runner it was sent to, its workspace, approval and DevGuard attempt, its phase, its deadline, its root process and how it ended. A record never revives a handle and never alone proves that an execution started or how it ended; the Runner stays the authority. Such a gateway needs --operations-db, and it is the database's only writer: it waits up to 15 seconds for a gateway that is still ending, then refuses to start. The database now states its schema version, and a newer one is refused. Nothing changes outside the independent mode, which stays unsupported. Using the records to restore occupancy and approvals comes in P1R-C04.

codespace-mcp --runner uds --runner-socket "$RUNNER_DIR/runner.sock" \
  --runner-lifecycle independent --runner-exit-intent detach \
  --operations-db "$STATE_DIR/operations.sqlite"

How.

• Records (crates/store): a new processes table. A record holds the process_id; the Runner's identity from its Hello (boot id, pid, start time, nonce); the controller epoch, empty until P1R-C03; the workspace, the approval it resumes, the DevGuard attempt, the participation (off or required), the tty flag and the argv's SHA-256 digest; the phase; the deadline the Runner reported; the root process's pid and start identity; the termination, exit code and signal; the furthest output cursor a gateway observed; and when it was created and last changed. The phase is dispatching until the Runner answers that it started the execution, running after that, and terminal once the Runner reports the end or answers that it did not start it. Nothing records unknown until P1R-C05. Only a gateway on the Runner with the same full identity changes a record, so a later Runner that reuses a pid changes nothing. An end reported before the start answer stays the end, and how an execution ended is recorded once.
• Write order (crates/server): the record is written before the request goes to the Runner, the DevGuard preparation and launch included. A resumed approval is marked resuming in the same transaction. When the record cannot be written, the request is refused with INTERNAL and nothing is sent. The record becomes running with the Runner's answer, or with a governed launch once its helper exists, and stays dispatching when the answer is lost (dispatch_status: unknown). Each exit event releases the lease as before and is then recorded with what process_status reports. The next gateway records the exits that happened while no gateway was connected, and process_status and read_process record what the Runner reports.
• Schema version: the database keeps it in PRAGMA user_version, now 1. A database without one, which every earlier CodeSpace wrote, is brought up to it, and the version is written last, so an upgrade cut short runs again. A database with a newer version is refused before anything is written. The patch ledger, works, intents and approvals keep their columns.
• One writer: before it opens the database, a gateway with an independent Runner takes an exclusive flock on <database>.gateway.lock (0600), writes its pid there, and holds it until it exits, however it exits. A gateway that finds it held logs the holder, retries for up to 15 seconds and then refuses to start, naming the holder. Other modes take no lock.
• Runner (crates/runner): the exec answer and the governed launch report when the process times out and its root process, by pid and, on Linux and macOS, a start identity read before the reaper runs. A PTY root is not named, since the pinned Codex PTY does not give its pid. The worker protocol is now 9.
• Docs, English and Korean: runner isolation gains "Process records" and "One writer", and operations gains the schema version and why an earlier CodeSpace must not share the database. Architecture, Codex reuse, DevGuard integration, error codes and the security model are updated, with the registry hashes recorded.
• CI: the macos-core stage of scripts/validate-upstream.py also runs the store tests and the runner's root report test on hosted macOS.

Contract changes

• MCP: none. Tool schemas, error codes, status values and instructions are unchanged. In the independent mode only, exec_command and operation_resume can answer the existing INTERNAL code when the record cannot be written ("the execution could not be recorded before it was sent, so nothing was started"). A resume refused this way is consumed with that error, as any failed resume is today.
• Gateway CLI: --runner-lifecycle independent now requires --operations-db (or CODESPACE_OPERATIONS_DB) and a Runner whose Hello states its identity, which every P1R-C01 Runner does.
• Lock file: <database>.gateway.lock beside the operations database, mode 0600, holding the gateway's pid; a 15-second wait, then refusal. Independent mode only.
• Operations database schema version 1, in PRAGMA user_version. Version 0 (no version) is upgraded in place, and a newer version is refused. New table processes with the index idx_processes_workspace_phase; the other tables are unchanged.
• Worker protocol 8 → 9, still matched exactly. The exec answer (RunnerExecResult) and the governed launch (ManagedLaunch) gain the optional fields root (pid, started) and deadline_at_ms.

Tests

The brief's cases and the tests that cover them.

Case	Tests
Normal re-lookup and terminal records	store a_new_gateway_finds_each_execution_from_dispatch_to_its_end, an_execution_the_runner_did_not_start_ends_as_not_started, an_end_without_details_is_completed_once; gateway a_gateway_records_an_execution_from_its_dispatch_to_its_end, a_resumed_execution_is_recorded_with_its_approval
Torn write	store a_crash_while_writing_leaves_neither_the_record_nor_the_resume (aborts a child process between the record's statements and its commit), a_damaged_database_is_refused_and_kept; gateway an_execution_that_cannot_be_recorded_is_not_sent
Unknown schema version	store a_newer_database_is_refused_and_left_as_it_is
PID reuse	store a_runner_with_the_same_pid_is_another_runner, a_root_is_the_recorded_process_only_with_its_start_identity; runner a_start_identity_names_one_process_until_it_is_reaped, a_spawn_reports_its_root_and_its_deadline
Overlapping gateway restart on one lock: the bounded wait and the refusal after it	store a_second_writer_waits_up_to_its_bound_and_is_refused, a_writer_that_lets_go_within_the_bound_is_waited_for, a_writer_killed_holding_the_lock_lets_go_of_it, the_lock_is_a_private_file_beside_the_database_naming_its_holder; gateway binary a_gateway_with_an_independent_runner_is_its_databases_only_writer, a_restarted_gateway_waits_for_the_one_before_it_and_finds_its_records
Crash at spawn	gateway an_execution_is_recorded_before_the_runner_receives_it (the gateway process aborts as the Runner receives the request, and the record is already dispatching), an_execution_whose_answer_is_lost_stays_dispatching
Crash at save	the torn-write tests above
Crash at exit and the re-lookup race	gateway an_end_while_no_gateway_is_connected_is_recorded_by_the_next_gateway, an_end_a_gateway_missed_is_recorded_when_the_next_gateway_asks_the_runner, an_execution_that_ends_at_once_is_recorded_as_it_ended (24 rounds); store an_end_reported_before_the_start_answer_stays_the_end; gateway binary a_restarted_gateway_waits_for_the_one_before_it_and_finds_its_records
Old and new database fixtures; an earlier binary ignores the new table	store a_database_from_before_schema_versions_is_brought_up_to_this_one, an_earlier_codespace_leaves_the_records_it_does_not_know (opens a database this version wrote with feb5f52's statements and adds a patch operation)
The patch ledger is unchanged and no sensitive value is stored	store the_patch_ledger_and_the_approvals_keep_their_columns, a_record_keeps_no_argv_and_no_secret (searches every file of the database for the argv and a secret)
Settings	gateway only_an_independent_runner_with_an_exit_intent_is_accepted (now also needs --operations-db); gateway binary a_gateway_with_an_independent_runner_is_its_databases_only_writer

Verified locally (cloud Linux, Rust 1.99.0):

• cargo fmt --check for the root and every adapter, python3 scripts/check_docs.py ("Documentation registry verified: 20 paired documents"), the script tests (102), scripts/check-no-model-deps.sh and scripts/check-upstream-pin.sh (Codex 6b9826e with its 1 recorded patch).
• clippy with -D warnings: the root with --all-targets; the server, runner and domain with devguard; the worker with and without devguard.
• cargo test --workspace, with the worker and the patch helper built: 391 passed, none failed, the new store, runner and gateway tests and independent_runner included.
• cargo test -p codespace-server --features devguard, with the DevGuard worker, fixture authority and launch helper built as CI's integration stage builds them and CODESPACE_REQUIRE_DEVGUARD_BINS=1: 151 passed, none failed. The 5 ignored tests are the existing child-process tests that their parent tests run.
• The worker's tests (runtime_binary: 9 passed, its handshake test now expecting protocol 9) and its DevGuard binary tests (2 passed); cargo test -p codespace-domain --features devguard --lib: 32 passed.
• cargo test -p codespace-runner --features devguard --lib: 170 passed and 1 failed, launch::tests::nothing_is_committed_when_the_owner_cannot_launch. As on runner, server: opt-in independent Runner that outlives its gateways (P1R-C01) #102, it fails here only because the cloud container runs as root, so the test's group-writable helper belongs to group 0, which counts as private. CI runs it as a non-root user.
• The integration stage's tests again with the Linux sandbox on, as CI runs them (bubblewrap installed, the sandbox helper built and named): cargo test --workspace --no-fail-fast 391 passed, cargo test -p codespace-server --features devguard 151 passed, none failed.

Not verified locally: the macOS legs and the documentation site build (it needs Node 24.21.0); CI covers both.

CI: passed on a4f362c in run 38608571156: all 17 jobs, Integration, Linux isolation and macOS contracts included. The hosted macOS leg ran the store tests, the runner's root report test, the independent gateway tests and independent_runner in its macos-core stage, and its 400 fork-race trials had no failure. The documentation site's run 38608572440 built the site.

The first run on 82262c8, 38606709531, failed only Integration: two record tests compared the shell's $$ with the recorded root's pid, and inside the Linux sandbox $$ is the shell's pid in the sandbox's own PID namespace while the root is the sandbox helper. a4f362c compares them only without the sandbox, and checks, sandboxed or not, that the running execution's recorded root is a live child of the Runner.

Security scenarios

• No sensitive values. A record keeps the argv's SHA-256 digest, never the argv, its environment or a secret, and the test searches the database's files for them. The lock file holds only a pid.
• Records are not authority. A record never revives a handle. An execution whose answer was lost stays dispatching, and only the Runner's answer or exit report makes it running or terminal. A record is changed only by a gateway on the Runner with the same full identity, so a reused pid changes nothing. For the reconciliation in P1R-C04, the store counts a root as the recorded process only with the same start identity.
• Nothing runs unrecorded. When the record cannot be written, nothing is sent to the Runner, and the lease is released.
• One writer, honestly bounded. Two gateways never record into one database at once: the second one waits up to 15 seconds and then refuses, naming the holder. The lock is advisory: an earlier CodeSpace takes no lock and checks no version, and the docs say not to run one on such a database.
• Rollback honesty. A database a newer CodeSpace wrote is refused before anything is written, never downgraded. A file that is not a database is refused and kept.
• Rollback. Stop starting the independent mode and keep the database. Restoring an earlier copy would drop the patch idempotency records, confirmation holds and process records written since.

Out of scope

• Restoring occupancy or approvals from the records (P1R-C04), issuing controller epochs (P1R-C03), gap and uncertainty reasons and the unknown phase (P1R-C05), and the qualification harness (P1R-C06).
• Pruning records: nothing removes them yet.
• Supporting the mode: it stays unsupported until P1R-C06 qualifies it.
• Any Codex or DevGuard pin change.