Skip to content

Repository navigation

Tracker: P1-RECOVERY, Gateway recovery while an independent Runner survives #101

Open
Open
Tracker: P1-RECOVERY, Gateway recovery while an independent Runner survives#101
Labels
documentationImprovements or additions to documentation

Description

@novelKR
novelKR
opened on Oct 14, 2026

What this issue is

The CodeSpace tracker for P1-RECOVERY, started on the owner's direction of 2026-10-14. It is a living tracker: when the state changes, add a comment (date, what changed, evidence link) and tick the boxes below.

Scope

In an operator-selected mode, an independent Runner keeps its processes, I/O and original deadlines when the Gateway exits, and a new Gateway takes over the same executions. Existing modes keep their termination behavior. InProcess, restoring I/O after the Runner itself restarts, and replaying argv are out of scope (ADR-002).

Units

One CodeSpace PR per unit, merged in order. Each unit starts on the owner's approval after the previous one is merged and its post-merge CI is green.

Unit What it adds After
P1R-C01 Opt-in independent Runner lifecycle: serial connections, exit intents (detach or stop), Runner identity in Hello; a re-attached Gateway may only observe, write stdin, resize and terminate DevGuard ledger start
P1R-C02 Process records in the operations database, with a schema version and a single-writer lock P1R-C01
P1R-C03 Claim on the existing connection, with an epoch fence on every mutating request P1R-C02
P1R-C04 Reconciliation of records, Runner inventory and DevGuard attempts; a per-workspace barrier P1R-C03
P1R-C05 Output gaps, uncertainty and Runner loss; the operator runbook P1R-C04
P1R-C06 Qualification harness and the macOS runs, then the report P1R-C05

Rules

  • The default mode does not change. The new mode is opt-in and is not a supported mode until P1R-C06 qualifies it.
  • No Codex pin, DevGuard pin, DevGuard release, installed service or policy change without the owner's approval. A Codex fix is a recorded patch file only (upstream patches).
  • MCP contract changes (new error codes, gap and uncertainty reasons, mode-specific tool text) are listed in each PR's "Contract changes" and need the owner's approval at merge.
  • The known defects OX-01 and OX-02 (Handoff: CS-RG managed-execution suspension (#75) and the DevGuard boundary revalidation (2026-09-27) #76) are not fixed here; P1-RECOVERY does not change when a PTY handle is dropped.

Checklist

  • DevGuard ledger: P1-RECOVERY in-progress and the plan re-baselined (docs: start P1-RECOVERY and re-baseline its plan DevGuard#36)
  • P1R-C01 merged and verified
  • P1R-C02 merged and verified
  • P1R-C03 merged and verified
  • P1R-C04 merged and verified
  • P1R-C05 merged and verified
  • P1R-C06 merged; qualification run on macOS; report merged
  • DevGuard ledger: P1-RECOVERY implemented and its qualification recorded

Activity

  1. novelKR
    added
    documentationImprovements or additions to documentation
    on Oct 14, 2026

  2. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, DevGuard ledger started. novelKR/DevGuard#36 was merged on the owner's approval at head 9b19078, as DevGuard main 165cb84 (documentation only; the merge tree equals the head). Post-merge CI run 38573601167 is green.

    The ledger now records P1-RECOVERY as in-progress with P1R-C01 active, the plan carries the dated re-baseline, the integration specification's recovery mode section is excepted from the suspension for P1-RECOVERY, and ADR-001 has the amendment for the independent Runner's credential file. CodeSpace's DevGuard pin stays 6e7e065.

    P1R-C01 may ask for its merge once its pull request is green at its exact head.

    Generated by Claude Code

  3. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, P1R-C01 merged and verified; P1R-C02 started. #102 was merged on the owner's approval at head c22e823, as main feb5f52. Post-merge CI run 38592385183 passed all 17 jobs, hosted macOS included. The approval also covered the contract changes listed in #102. With it the owner started P1R-C02. The DevGuard pin stays 6e7e065.

    The DevGuard ledger update, P1R-C01 merged-verified and P1R-C02 active, is novelKR/DevGuard#37 (head a623275, CI green), waiting for the owner's merge approval.

    Generated by Claude Code

  4. novelKR commented on Oct 14, 2026

    @novelKR
    novelKR
    on Oct 14, 2026 – with Claude
    OwnerAuthor

    2026-10-14, P1R-C02 merged and verified; P1R-C03 started. #103 was merged on the owner's approval at head a4f362c, as main 5976df0. Post-merge CI run 38612003338 passed all 17 jobs, hosted macOS included. The approval also covered the contract changes listed in #103. With it the owner started P1R-C03. The DevGuard pin stays 6e7e065.

    DevGuard ledger: novelKR/DevGuard#37 (P1R-C01 merged-verified, P1R-C02 active) was merged on the owner's approval as DevGuard main 0f4bf73, post-merge CI green. The next update, novelKR/DevGuard#38 (P1R-C02 merged-verified, P1R-C03 active), was merged on the owner's approval at head bd078f5 as DevGuard main 77293a2; post-merge CI run 38616346872 is green.

    Generated by Claude Code

  5. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C03 merged and verified; P1R-C04 started. #104 was merged on the owner's approval at head 4e54309, as main 1fda789. Post-merge CI run 38686514388 passed all 17 jobs, hosted macOS included. In its first attempt the macOS contracts job failed in its "macOS contracts and dependencies" step, on a tree whose PR run 38631018229 had passed; the one re-run of that job passed. The approval also covered the contract changes listed in #104. With it the owner started P1R-C04. The DevGuard pin stays 6e7e065.

    DevGuard ledger: novelKR/DevGuard#39 (P1R-C03 merged-verified, P1R-C04 active) was merged on the owner's approval at head 760041a as DevGuard main 7cd86e4; post-merge CI run 38708338123 is green.

    Generated by Claude Code

  6. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C04 merged and verified; P1R-C05 started. #105 was merged on the owner's approval at head 7cabcba, as main 385e177. Post-merge CI run 38708302887 passed, with Linux isolation skipped by the plan as on the PR. In its first attempt the macOS contracts job failed in the runner's DevGuard tests (cargo test -p codespace-runner --features devguard --lib -- registration wire admission launch independent lifecycle exited 101), on a tree whose PR run 38704449082 had passed; the one re-run of that job passed. Which test failed is in that attempt's artifact only. The approval also covered the contract changes listed in #105. With it the owner started P1R-C05. The DevGuard pin stays 6e7e065.

    Follow-up, tests only: #106 (head 41db94c), opened while looking for that failure. A P1R-C03 test in the same command, an_independent_runner_answers_registration_before_the_next_claim, failed 1 run in 25 locally; #106 fixes it and makes five tests from #105 wait for a process's end rather than its exit. It is not a unit; it was merged on the owner's approval as main 950e2b2. Its post-merge CI run 38718331468 passed after one re-run: in the first attempt only the Linux integration job failed (cargo test --locked --workspace exited 101; the failing test is in that attempt's artifact only).

    DevGuard ledger: novelKR/DevGuard#40 (P1R-C04 merged-verified, P1R-C05 active) was merged on the owner's approval at head 7b6ad73 as DevGuard main 494793a; post-merge CI run 38718406648 is green.

    Generated by Claude Code

  7. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15: what P1R-C05 found outside its unit

    While driving #108, the P1R-C05 work found:

    1. A race in a P1R-C04 test. mcp::independent_tests::an_independent_gateway_blocks_a_workspace_whose_records_and_runner_disagree failed runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's Linux Integration job on 5a41aaf (run 38728457392), whose log named it thanks to scripts: name a failed stage's tests in the job log #107. It fails on main 3fac934 too (5 of 64 runs at 8 in parallel) and is probably main 950e2b2's Integration failure (run 38718331468). The test terminates its sandboxed command while the sandbox is still being set up. A test-only fix, terminating once the command has printed a line, passed 80 of 80 runs on main. Details: comment on #108.
    2. A race in P1R-C04's reclaim. a_gateway_that_started_work_claims_its_runner_again_and_reconciles_it fails about 1 in 100 runs at 16 in parallel, on main and on runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108's branch. An exit that the Runner sends right after the new Claim's answer reaches the records' exit-following loop (crates/server/src/records.rs) before reclaim_loop (crates/runner/src/uds.rs) makes the link current. The status query is then refused as reconnecting, and the record ends as "exited, how not known", which reconciliation cannot correct. In use, this loses the exit code of a process that exits around a reconnection; nothing runs twice.
    3. Terminating a sandboxed command during sandbox setup on Linux leaves it running outside the Runner's reach: runner: a sandboxed command terminated during its sandbox setup keeps running (Linux) #109.
    4. Open from runner, server, store: answer for a lost Runner's processes and let an operator release them (P1R-C05) #108: a read-only DevGuard admission status command, since --open reopens admission.

    Plan: 1 and 2 go into one follow-up PR after #108 merges, as #106 did for P1R-C04, so P1R-C06's Mac qualification runs on a main that has them. #109 stays outside the units unless the owner adds it.

    Generated by Claude Code

  8. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C05 merged and verified; P1R-C06 started. #108 was merged on the owner's approval at head 5deb102, as main dadb58c. Post-merge CI run 38739506757 passed all 17 jobs on the first attempt, macOS contracts and Linux isolation included, and the documentation site run 38739507537 passed. The approval also covered the contract changes listed in #108. With it the owner started P1R-C06. The DevGuard pin stays 6e7e065.

    CI tooling, not a unit: #107 and #110 make a failed CI stage list its failed tests and cargo's rerun targets in the job log, never their output. Both were merged on the owner's approval, #107 at head dea6c47 as main 3fac934 and #110 at head dfe9144 as main e947e80; their post-merge CI runs 38728182277 and 38739632977 passed on the first attempt.

    Follow-up for findings 1 and 2 above: #111 (head 57908bc), from the P1R-C05 work, was merged on the owner's approval as main 345864d; its post-merge CI run 38743295778 passed on the first attempt. P1R-C06's macOS qualification runs on a main that includes it.

    DevGuard ledger: novelKR/DevGuard#41 (P1R-C05 merged-verified, P1R-C06 active) was merged on the owner's approval at head cf3e206 as DevGuard main ec66b12; post-merge CI run 38743349871 is green.

    Generated by Claude Code

  9. novelKR commented on Oct 15, 2026

    @novelKR
    novelKR
    on Oct 15, 2026 – with Claude
    OwnerAuthor

    2026-10-15, P1R-C06's harness and protocol merged. #112 was merged on the owner's approval at head 8de5fdb, as main b2cb042. Post-merge CI run 38767662399 passed all 17 jobs on the first attempt, and the documentation site run 38767664121 passed. It adds the recovery-* functional suites and the recovery restart measurement to scripts/qualify-devguard.py, the Gateway recovery qualification protocol in English and Korean, and the independent Runner's adoption and rollback to the operations guide. It changes no MCP tool, error code or status value, and no pin. The independent mode stays unsupported until the macOS run qualifies a combination.

    Next, in P1R-C06's thread: the macOS check and candidate freeze on b2cb042 or a later main, the measurement, then a docs-only report PR. The P1R-C06 box is ticked when that report merges, and the DevGuard ledger then records P1-RECOVERY's result.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions