server: record an exit heard while reconnecting as reported, and fix a test's early terminate (P1R-C04 follow-up)

[novelKR]

Requested by Seongjae · project thread

Issue

Part of #101: a follow-up to P1R-C04 (#105, #106) for two races that loops found while P1R-C05 (#108) was in CI, both on main before #108 too. The P1R lead thread asked for them in one PR, so that P1R-C06 qualifies a main with both fixed. This PR does not close an issue: #109, the termination problem behind the test's race, stays open.

Item	Value
Base	main at dadb58c, with P1R-C05 (#108) merged
Head	57908bc
Pins	Codex gitlink with its recorded patch, and DevGuard 6e7e065; both unchanged

Summary

Before:

• An exit that an independent Runner reported right after its answer to a new Claim could be recorded without how the process ended. The records asked the Runner for the process's status before the client had taken the new connection as its own, the client refused the request with RUNNER_RECONNECTING, and the records ended the process with no termination or exit code, which reconciliation does not correct. a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it failed this way in about 1 of 100 runs at 16 in parallel, on main and on runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's branch.
• an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree terminated its sandboxed sleep 30 about 100 ms after the Runner started it. When the kill landed while bubblewrap was still setting up the sandbox, the sandbox ran on under PID 1 with the Runner's output pipes open, so the Runner did not report the exit within the test's 20 s bound. It failed in 5 of 64 runs at 8 in parallel on main, and failed runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's first Linux Integration job (comment).

After:

• The records ask the Runner again while the client refuses with RUNNER_RECONNECTING, and record the end as the Runner reports it once the gateway has claimed the Runner, which keeps the ended process until its end is recorded.
• The test's command prints a line first, and the test terminates it once it has read that line, as the other independent tests already wait for their command before terminating it.

How.

• crates/server/src/records.rs: status_after_exit asks for the status of a process the Runner reported exited, and while the answer is RUNNER_RECONNECTING asks again after 10 ms, then twice as long each time, up to 500 ms. Every other answer is handled as before. The client refuses that way only while it claims the Runner again; once it has lost the Runner or stopped trying, it refuses with RUNNER_DISCONNECTED, and the records end the process without saying how, as before.
• crates/server/src/mcp.rs, tests: the new an_exit_heard_while_claiming_again_is_recorded_as_the_runner_reports_it, and the command and the wait in an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree.

Contract changes

None: no tool schema, error code, status value or ID changes. A record of a process that exits as the gateway claims its Runner again now holds the termination and exit code the Runner reports, as process records already describe, instead of none.

Tests

Verified here at 57908bc, in a Linux cloud container running as root, on the Codex checkout with its recorded patch applied and the worker, patch and Linux sandbox binaries built:

• The new test answers the first status request with RUNNER_RECONNECTING, as the client does while it claims the Runner again, then with the exit. It passes, and without the change in records.rs it fails with the record the race left: terminal, no outcome, no exit code.
• Loops of the two tests named in Before, 16 at a time: 0 of 128 failed each. On main before this change, the first failed in 1 of 96 runs at 16 at a time, and the second in 5 of 64 at 8 at a time.
• cargo fmt --check for the workspace and for the worker, and cargo clippy --locked --all-targets -- -D warnings in CI's six configurations.
• cargo test --locked --workspace: 447 passed, none failed. -p codespace-server --features devguard: 174 passed. -p codespace-domain --features devguard --lib: 34 passed. The worker: 11 passed, and 2 with devguard.
• -p codespace-runner --features devguard --lib: 194 passed and 1 failed, launch::tests::nothing_is_committed_when_the_owner_cannot_launch, which fails only as root, as on main; the same test binary passed it when run as an unprivileged user.
• python3 scripts/check_docs.py check (20 paired documents).

Not verified here: the hosted macOS job, which CI runs.

Security scenarios

• No duplicate execution. The change only asks the Runner again about a process it reported exited; nothing is started or sent again.
• Rollback honesty. A record still says how a process ended only as the Runner reported it. While the gateway claims the Runner again the record stays unended rather than ended without a termination, and the process's lease is released at the exit report, as before.
• Workspace isolation, path escape, token logging. Unchanged.

Out of scope

• runner: a sandboxed command terminated during its sandbox setup keeps running (Linux) #109: terminating a sandboxed command while its sandbox is still being set up leaves the command running outside the Runner's reach. This PR only makes the test wait; fixing it changes termination.
• P1R-C06, and supporting the independent mode.