From 8de5fdbe99e71d25abe590c7a554a88f90a873b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 15 Oct 2026 15:27:32 +0000 Subject: [PATCH] scripts, docs: qualify the independent Runner (P1R-C06) The qualification harness gains P1R-C06's recovery suites. recovery-lifecycle, recovery-store, recovery-fence, recovery-reconcile and recovery-observation test the five P1-RECOVERY units against an independent Runner that the harness starts in its own session, as an operator would: with participation off everywhere, and required on macOS against DevGuard's fixture authority. recovery runs them all and then measures gateway restarts. In each idle and loaded interval a fresh Runner holds governed pipe and PTY processes while gateways end in turn (input closed, SIGTERM, SIGUSR1, SIGKILL); every cycle another connection and a rival gateway probe the fence, a timeout falls with or without a gateway, and an approval is resumed across the restart, in flight when the gateway is killed. Each interval ends by draining its Runner. verify recomputes every recovery verdict from the raw rows, and CLAIMS gains the independent mode's cells. docs/recovery-qualification.md, in English and Korean, is the protocol, frozen with the harness. Operations gains the independent Runner's adoption and rollback. The guides, the gateway's flag help and its Claim log line no longer say that the mode is not supported yet: it is supported only on the combinations its qualification qualifies, none so far. Co-Authored-By: Claude Opus 5.5 (1M context) --- crates/server/src/config.rs | 7 +- crates/server/src/runtime.rs | 2 +- docs/agent-integration.md | 2 +- docs/architecture.md | 2 +- docs/devguard-integration.md | 2 +- docs/ko/agent-integration.md | 2 +- docs/ko/architecture.md | 2 +- docs/ko/devguard-integration.md | 2 +- docs/ko/operations.md | 35 +- docs/ko/recovery-qualification.md | 224 ++ docs/ko/runner-isolation.md | 2 +- docs/ko/security-model.md | 2 +- docs/operations.md | 35 +- docs/recovery-qualification.md | 224 ++ docs/runner-isolation.md | 2 +- docs/security-model.md | 2 +- docs/translations.json | 93 +- scripts/qualify-devguard.py | 2966 +++++++++++++++++++++++- scripts/qualify_analysis.py | 248 +- scripts/qualify_host.py | 342 ++- scripts/tests/test_qualify_devguard.py | 416 +++- 21 files changed, 4509 insertions(+), 103 deletions(-) create mode 100644 docs/ko/recovery-qualification.md create mode 100644 docs/recovery-qualification.md diff --git a/crates/server/src/config.rs b/crates/server/src/config.rs index fbb0d9b..4bc879d 100644 --- a/crates/server/src/config.rs +++ b/crates/server/src/config.rs @@ -24,8 +24,8 @@ pub enum RunnerLifecycleMode { #[default] Managed, /// An independent Runner that an operator or a harness started: the gateway only connects - /// to it through `--runner-socket`, and its processes outlive the gateway. Not a supported - /// mode yet. + /// to it through `--runner-socket`, and its processes outlive the gateway. Supported only on + /// the combinations its qualification qualifies (P1R-C06). Independent, } @@ -135,7 +135,8 @@ pub struct Cli { /// The UDS Runner's lifecycle: `managed` (default), or `independent` for a Runner that an /// operator or a harness started, which the gateway only connects to through - /// `--runner-socket` (P1R-C01). `independent` is not a supported mode yet. + /// `--runner-socket` (P1R-C01). `independent` is supported only on the combinations its + /// qualification qualifies (P1R-C06). #[arg(long, value_enum, default_value_t = RunnerLifecycleMode::Managed)] pub runner_lifecycle: RunnerLifecycleMode, diff --git a/crates/server/src/runtime.rs b/crates/server/src/runtime.rs index 9af5fd7..f5218bc 100644 --- a/crates/server/src/runtime.rs +++ b/crates/server/src/runtime.rs @@ -177,7 +177,7 @@ impl RuntimeProcess { previous_controller = claim.previous_controller, processes = claim.processes.len(), events_through = claim.events_through, - "claimed an independent Runner; it is not a supported mode yet" + "claimed an independent Runner" ); warn_recovery_pending(&claim); Ok((runner, claim)) diff --git a/docs/agent-integration.md b/docs/agent-integration.md index a85c924..d286006 100644 --- a/docs/agent-integration.md +++ b/docs/agent-integration.md @@ -150,7 +150,7 @@ Interactive input and cancellation use the same handle: A live command occupies the workspace. Wait for it to end or terminate it before applying a patch or starting another command. Reads and searches remain available. Long-lived development servers therefore require a workflow that stops them before edits. -`workspace_info.execution.process.capabilities.lifetime` advertises the owner: the runner instance, not the MCP session. Streamable HTTP or MCP client disconnect keeps the process running; reconnect with the saved `process_id`. Losing the UDS worker connection or shutting down the gateway (including stdio EOF) terminates the owned subtree and drops handles. Restart does not restore `process_id`. With an [independent Runner](runner-isolation.md#independent-runner), which is not a supported mode yet, `runner_disconnect` is `keep_running`, and `gateway_shutdown` is `keep_running` when the gateway detaches and `terminate` when it stops the Runner; read `initialize.instructions` for what that gateway does. While that gateway claims its Runner again after a lost connection, it refuses requests with `RUNNER_RECONNECTING`, which sent nothing and consumed no approval: retry after a pause. Then, in a workspace where work was started before, it refuses new commands, patches and approval resumes with `RECOVERY_PENDING` until it has [reconciled](runner-isolation.md#reconciliation) that workspace with the Runner; its message says whether retrying after a pause helps. A process it finds running there holds the workspace as before. If the gateway loses the Runner itself, it answers `process_status` and `read_process` for that Runner's processes from its records, with `lost_reason`, refuses other requests that need the Runner with `RUNNER_DISCONNECTED`, and keeps their workspaces refused, with `RECOVERY_PENDING` in the next gateway, until an operator releases them ([Runner loss](runner-isolation.md#runner-loss)). If the spawn response is lost before you receive `process_id`, do not search with `process_status` and do not start a duplicate command. +`workspace_info.execution.process.capabilities.lifetime` advertises the owner: the runner instance, not the MCP session. Streamable HTTP or MCP client disconnect keeps the process running; reconnect with the saved `process_id`. Losing the UDS worker connection or shutting down the gateway (including stdio EOF) terminates the owned subtree and drops handles. Restart does not restore `process_id`. With an [independent Runner](runner-isolation.md#independent-runner), supported only where [qualified](recovery-qualification.md), `runner_disconnect` is `keep_running`, and `gateway_shutdown` is `keep_running` when the gateway detaches and `terminate` when it stops the Runner; read `initialize.instructions` for what that gateway does. While that gateway claims its Runner again after a lost connection, it refuses requests with `RUNNER_RECONNECTING`, which sent nothing and consumed no approval: retry after a pause. Then, in a workspace where work was started before, it refuses new commands, patches and approval resumes with `RECOVERY_PENDING` until it has [reconciled](runner-isolation.md#reconciliation) that workspace with the Runner; its message says whether retrying after a pause helps. A process it finds running there holds the workspace as before. If the gateway loses the Runner itself, it answers `process_status` and `read_process` for that Runner's processes from its records, with `lost_reason`, refuses other requests that need the Runner with `RUNNER_DISCONNECTED`, and keeps their workspaces refused, with `RECOVERY_PENDING` in the next gateway, until an operator releases them ([Runner loss](runner-isolation.md#runner-loss)). If the spawn response is lost before you receive `process_id`, do not search with `process_status` and do not start a duplicate command. ## Confirm a held mutation diff --git a/docs/architecture.md b/docs/architecture.md index 3032687..8dc80bf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -51,7 +51,7 @@ If helper application fails, the Runner attempts per-file snapshot restoration. ## Process lifetime -MCP request completion does not end a managed process. Clients continue with its `process_id`. Server restart loses those handles. In UDS mode the gateway owns the worker: internal disconnect/shutdown ends the worker and its children, with no reconnect. An [independent Runner](runner-isolation.md#independent-runner), not a supported mode yet, outlives its gateways instead, and a gateway that lost its connection to it claims it again and reconciles what earlier controllers left before it starts new work where they did. When the Runner itself is lost, the gateway answers for its executions from their records, and their workspaces wait for an operator ([Runner loss](runner-isolation.md#runner-loss)). See [runner isolation](runner-isolation.md) for the distinct transport and isolation boundaries. +MCP request completion does not end a managed process. Clients continue with its `process_id`. Server restart loses those handles. In UDS mode the gateway owns the worker: internal disconnect/shutdown ends the worker and its children, with no reconnect. An [independent Runner](runner-isolation.md#independent-runner), supported only where [qualified](recovery-qualification.md), outlives its gateways instead, and a gateway that lost its connection to it claims it again and reconciles what earlier controllers left before it starts new work where they did. When the Runner itself is lost, the gateway answers for its executions from their records, and their workspaces wait for an operator ([Runner loss](runner-isolation.md#runner-loss)). See [runner isolation](runner-isolation.md) for the distinct transport and isolation boundaries. ## Planned execution coordination diff --git a/docs/devguard-integration.md b/docs/devguard-integration.md index 6c2b2c7..a5ca463 100644 --- a/docs/devguard-integration.md +++ b/docs/devguard-integration.md @@ -338,7 +338,7 @@ Default participation remains `off`; `required` is an explicit operator choice i ## Initial Gateway recovery scope -**Status, P1-RECOVERY (owner directive 2026-10-14):** P1-RECOVERY is implemented one pull request per unit, as re-baselined in DevGuard's [P1-RECOVERY plan](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/planning/milestones/P1-RECOVERY.md#re-baseline-2026-10-14). Its first unit, P1R-C01, adds the [independent Runner](runner-isolation.md#independent-runner), an opt-in mode that is not supported yet: an operator or a harness starts the Runner, and its processes, output and timeouts outlive each gateway, which detaches from it or stops it as its exit intent says. The second unit, P1R-C02, adds [process records](runner-isolation.md#process-records): the gateway records each execution in its operations database before it sends the execution to the Runner and follows it to its end, as the database's only writer. The third unit, P1R-C03, adds the [Claim](runner-isolation.md#claim): one connection at a time controls the Runner, under a controller epoch that each request changing a process or the workspace carries, and a gateway that lost its connection claims the Runner again instead of being restarted. While no gateway controls the Runner, any process of the same user that can reach its socket can claim it, an exposure the [Runner's documentation](runner-isolation.md#claim-exposure) states. The fourth unit, P1R-C04, adds [reconciliation](runner-isolation.md#reconciliation): after each Claim, the gateway compares its records, what the Runner holds and remembers, and for a `required` execution its DevGuard attempt, which the Runner looks up, and refuses new executions, patches and approval resumes with `RECOVERY_PENDING` only in the workspaces where an earlier controller left work, until it has reconciled them. A process found running holds its workspace again, an approval left resuming is consumed with the process the Runner created for it or granted again when the Runner proves that it created none, and a workspace whose accounts do not agree, or whose record names another Runner, stays refused for an operator. The fifth unit, P1R-C05, adds [Runner loss](runner-isolation.md#runner-loss): when the Runner is killed, crashes or stops, or the host restarts, the gateway answers `process_status` and `read_process` for its executions from their records, with a `lost_reason`, and each execution that had not ended becomes `unknown` and holds its workspace until an operator finds its processes and releases its record with `codespace-mcp records` ([Operations](operations.md#runner-loss)). The operator releases a `required` execution's record only once DevGuard no longer charges its attempt, which only DevGuard's administrator quiescence shows, since DevGuard answers a lookup only to the attempt's own instance; until then DevGuard keeps the lost Runner's instance, so a new Runner cannot register as the `codespace` consumer's one instance. The mode is supported only once the last unit qualifies it. In this mode the Runner is DevGuard's execution owner and reads the consumer secret from its own credential file, as DevGuard's [ADR-001](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/planning/decisions.md#adr-001--one-registration-by-the-execution-owner) now provides; its gateway admits `required` workspaces only when the Runner states that it registers and launches through a launch helper. The rest of this section is the scope as planned. +**Status, P1-RECOVERY (owner directive 2026-10-14):** P1-RECOVERY is implemented one pull request per unit, as re-baselined in DevGuard's [P1-RECOVERY plan](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/planning/milestones/P1-RECOVERY.md#re-baseline-2026-10-14). Its first unit, P1R-C01, adds the [independent Runner](runner-isolation.md#independent-runner), an opt-in mode: an operator or a harness starts the Runner, and its processes, output and timeouts outlive each gateway, which detaches from it or stops it as its exit intent says. The second unit, P1R-C02, adds [process records](runner-isolation.md#process-records): the gateway records each execution in its operations database before it sends the execution to the Runner and follows it to its end, as the database's only writer. The third unit, P1R-C03, adds the [Claim](runner-isolation.md#claim): one connection at a time controls the Runner, under a controller epoch that each request changing a process or the workspace carries, and a gateway that lost its connection claims the Runner again instead of being restarted. While no gateway controls the Runner, any process of the same user that can reach its socket can claim it, an exposure the [Runner's documentation](runner-isolation.md#claim-exposure) states. The fourth unit, P1R-C04, adds [reconciliation](runner-isolation.md#reconciliation): after each Claim, the gateway compares its records, what the Runner holds and remembers, and for a `required` execution its DevGuard attempt, which the Runner looks up, and refuses new executions, patches and approval resumes with `RECOVERY_PENDING` only in the workspaces where an earlier controller left work, until it has reconciled them. A process found running holds its workspace again, an approval left resuming is consumed with the process the Runner created for it or granted again when the Runner proves that it created none, and a workspace whose accounts do not agree, or whose record names another Runner, stays refused for an operator. The fifth unit, P1R-C05, adds [Runner loss](runner-isolation.md#runner-loss): when the Runner is killed, crashes or stops, or the host restarts, the gateway answers `process_status` and `read_process` for its executions from their records, with a `lost_reason`, and each execution that had not ended becomes `unknown` and holds its workspace until an operator finds its processes and releases its record with `codespace-mcp records` ([Operations](operations.md#runner-loss)). The operator releases a `required` execution's record only once DevGuard no longer charges its attempt, which only DevGuard's administrator quiescence shows, since DevGuard answers a lookup only to the attempt's own instance; until then DevGuard keeps the lost Runner's instance, so a new Runner cannot register as the `codespace` consumer's one instance. The sixth and last unit, P1R-C06, adds the [qualification protocol](recovery-qualification.md) with its harness suites, and the mode is supported only on the combinations that its run qualifies; none is qualified yet. [Operations](operations.md#independent-runner-adoption) describes how to adopt the mode and roll it back. In this mode the Runner is DevGuard's execution owner and reads the consumer secret from its own credential file, as DevGuard's [ADR-001](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/planning/decisions.md#adr-001--one-registration-by-the-execution-owner) now provides; its gateway admits `required` workspaces only when the Runner states that it registers and launches through a launch helper. The rest of this section is the scope as planned. P1-RECOVERY introduces an operator-selected independent Runner mode and explicit capability. Existing modes keep their current shutdown/disconnect contracts. InProcess shares the Gateway PID and is excluded from live Gateway restart recovery. diff --git a/docs/ko/agent-integration.md b/docs/ko/agent-integration.md index d572904..76f7c91 100644 --- a/docs/ko/agent-integration.md +++ b/docs/ko/agent-integration.md @@ -150,7 +150,7 @@ MCP 클라이언트 SDK로 stdio 또는 Streamable HTTP를 초기화하고, 초 실행 중인 명령은 작업 공간을 점유합니다. 패치를 적용하거나 다른 명령을 시작하려면 기존 명령이 끝날 때까지 기다리거나 종료하세요. 읽기와 검색은 계속 가능합니다. 개발 서버를 오래 실행하는 작업 흐름이라면 수정 전에 서버를 멈추는 절차가 필요합니다. -`workspace_info.execution.process.capabilities.lifetime`은 소유자가 MCP 세션이 아니라 러너 인스턴스임을 알립니다. Streamable HTTP나 MCP 클라이언트 끊김은 프로세스를 유지하므로, 저장한 `process_id`로 다시 연결하면 됩니다. UDS worker 연결이 끊기거나 게이트웨이가 종료되면(stdio EOF 포함) 소유 서브트리가 종료되고 핸들이 사라집니다. 재시작은 `process_id`를 복구하지 않습니다. 아직 지원하는 모드가 아닌 [독립 Runner](runner-isolation.md#independent-runner)에서는 `runner_disconnect`가 `keep_running`이고, `gateway_shutdown`은 게이트웨이가 detach하면 `keep_running`, Runner를 중지하면 `terminate`입니다. 그 게이트웨이가 하는 일은 `initialize.instructions`에서 확인하세요. 그 게이트웨이는 연결을 잃은 뒤 Runner를 다시 Claim하는 동안 요청을 `RUNNER_RECONNECTING`으로 거부합니다. 이 거부는 아무것도 보내지 않았고 승인도 소모하지 않았으므로, 잠시 뒤 재시도하세요. 그 뒤 이전에 작업을 시작한 workspace에서는 Runner와 그 workspace를 [대조](runner-isolation.md#reconciliation)할 때까지 새 명령, 패치, 승인 재개를 `RECOVERY_PENDING`으로 거부합니다. 잠시 뒤 재시도하면 되는지는 그 메시지가 알려 줍니다. 그곳에서 실행 중으로 확인한 프로세스는 지금처럼 workspace를 점유합니다. 게이트웨이가 Runner 자체를 잃으면 그 Runner의 프로세스에 대한 `process_status`와 `read_process`에는 기록으로 `lost_reason`과 함께 답하고, Runner가 필요한 다른 요청은 `RUNNER_DISCONNECTED`로 거부하며, 운영자가 그 기록을 풀 때까지 그 workspace를 계속 거부합니다. 다음 게이트웨이에서는 `RECOVERY_PENDING`으로 거부합니다([Runner 손실](runner-isolation.md#runner-loss)). spawn 응답을 받기 전에 `process_id`를 잃으면 `process_status`로 찾지 말고 명령을 중복 시작하지 마세요. +`workspace_info.execution.process.capabilities.lifetime`은 소유자가 MCP 세션이 아니라 러너 인스턴스임을 알립니다. Streamable HTTP나 MCP 클라이언트 끊김은 프로세스를 유지하므로, 저장한 `process_id`로 다시 연결하면 됩니다. UDS worker 연결이 끊기거나 게이트웨이가 종료되면(stdio EOF 포함) 소유 서브트리가 종료되고 핸들이 사라집니다. 재시작은 `process_id`를 복구하지 않습니다. [qualification](recovery-qualification.md)을 통과한 곳에서만 지원하는 [독립 Runner](runner-isolation.md#independent-runner)에서는 `runner_disconnect`가 `keep_running`이고, `gateway_shutdown`은 게이트웨이가 detach하면 `keep_running`, Runner를 중지하면 `terminate`입니다. 그 게이트웨이가 하는 일은 `initialize.instructions`에서 확인하세요. 그 게이트웨이는 연결을 잃은 뒤 Runner를 다시 Claim하는 동안 요청을 `RUNNER_RECONNECTING`으로 거부합니다. 이 거부는 아무것도 보내지 않았고 승인도 소모하지 않았으므로, 잠시 뒤 재시도하세요. 그 뒤 이전에 작업을 시작한 workspace에서는 Runner와 그 workspace를 [대조](runner-isolation.md#reconciliation)할 때까지 새 명령, 패치, 승인 재개를 `RECOVERY_PENDING`으로 거부합니다. 잠시 뒤 재시도하면 되는지는 그 메시지가 알려 줍니다. 그곳에서 실행 중으로 확인한 프로세스는 지금처럼 workspace를 점유합니다. 게이트웨이가 Runner 자체를 잃으면 그 Runner의 프로세스에 대한 `process_status`와 `read_process`에는 기록으로 `lost_reason`과 함께 답하고, Runner가 필요한 다른 요청은 `RUNNER_DISCONNECTED`로 거부하며, 운영자가 그 기록을 풀 때까지 그 workspace를 계속 거부합니다. 다음 게이트웨이에서는 `RECOVERY_PENDING`으로 거부합니다([Runner 손실](runner-isolation.md#runner-loss)). spawn 응답을 받기 전에 `process_id`를 잃으면 `process_status`로 찾지 말고 명령을 중복 시작하지 마세요. ## 보류된 변경 확인 diff --git a/docs/ko/architecture.md b/docs/ko/architecture.md index 82dae0e..f61fdb6 100644 --- a/docs/ko/architecture.md +++ b/docs/ko/architecture.md @@ -58,7 +58,7 @@ UDS worker와 기본 러너는 모두 같은 호스트에서 실행합니다. ## 프로세스 수명 -MCP 요청이 끝나도 관리 중인 프로세스는 유지됩니다. 클라이언트는 `process_id`로 후속 호출을 수행합니다. 서버 재시작 후에는 핸들이 사라집니다. UDS 모드에서는 게이트웨이가 worker를 관리하므로 내부 연결 종료나 서버 종료 시 worker와 자식 프로세스가 종료되며 재접속은 지원하지 않습니다. 아직 지원하는 모드가 아닌 [독립 Runner](runner-isolation.md#independent-runner)는 이와 달리 게이트웨이보다 오래 살고, Runner와의 연결을 잃은 게이트웨이는 Runner를 다시 Claim한 뒤 앞선 controller들이 작업을 시작한 곳에서는 그들이 남긴 것을 대조하고 나서 새 작업을 시작합니다. Runner 자체를 잃으면 게이트웨이는 그 실행들에 대해 기록으로 답하고, 그 workspace는 운영자를 기다립니다([Runner 손실](runner-isolation.md#runner-loss)). [러너 격리](runner-isolation.md)에서 전송 경계와 격리 경계를 구분해 설명합니다. +MCP 요청이 끝나도 관리 중인 프로세스는 유지됩니다. 클라이언트는 `process_id`로 후속 호출을 수행합니다. 서버 재시작 후에는 핸들이 사라집니다. UDS 모드에서는 게이트웨이가 worker를 관리하므로 내부 연결 종료나 서버 종료 시 worker와 자식 프로세스가 종료되며 재접속은 지원하지 않습니다. [qualification](recovery-qualification.md)을 통과한 곳에서만 지원하는 [독립 Runner](runner-isolation.md#independent-runner)는 이와 달리 게이트웨이보다 오래 살고, Runner와의 연결을 잃은 게이트웨이는 Runner를 다시 Claim한 뒤 앞선 controller들이 작업을 시작한 곳에서는 그들이 남긴 것을 대조하고 나서 새 작업을 시작합니다. Runner 자체를 잃으면 게이트웨이는 그 실행들에 대해 기록으로 답하고, 그 workspace는 운영자를 기다립니다([Runner 손실](runner-isolation.md#runner-loss)). [러너 격리](runner-isolation.md)에서 전송 경계와 격리 경계를 구분해 설명합니다. ## 계획된 실행 조정 구조 diff --git a/docs/ko/devguard-integration.md b/docs/ko/devguard-integration.md index 12d7543..9e99201 100644 --- a/docs/ko/devguard-integration.md +++ b/docs/ko/devguard-integration.md @@ -338,7 +338,7 @@ contract/client pin, daemon/helper artifact와 CodeSpace Runner wire는 각각 ## 최초 Gateway 복구 범위 -**상태, P1-RECOVERY(소유자 지시 2026-10-14):** P1-RECOVERY는 DevGuard의 [P1-RECOVERY 계획](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/ko/planning/milestones/P1-RECOVERY.md#재기준-2026-10-14)에서 다시 잡은 기준대로 단위마다 PR 하나로 구현한다. 첫 단위 P1R-C01은 아직 지원하지 않는 opt-in 모드인 [독립 Runner](runner-isolation.md#independent-runner)를 더한다. 운영자나 하네스가 Runner를 시작하고, Runner의 프로세스, 출력, timeout은 모든 gateway보다 오래 산다. gateway는 종료 의도대로 Runner에서 detach하거나 Runner를 중지한다. 두 번째 단위 P1R-C02는 [실행 기록](runner-isolation.md#process-records)을 더한다. gateway는 작업 데이터베이스에 쓰는 유일한 gateway로서 각 실행을 Runner에 보내기 전에 기록하고 끝날 때까지 따라간다. 세 번째 단위 P1R-C03은 [Claim](runner-isolation.md#claim)을 더한다. 한 번에 한 연결만 controller epoch 아래에서 Runner를 제어하고, 프로세스나 workspace를 바꾸는 요청은 모두 그 epoch를 함께 보낸다. 연결을 잃은 gateway는 재시작하지 않고 Runner를 다시 Claim한다. Runner를 제어하는 gateway가 없는 동안에는 Runner의 소켓에 닿을 수 있는 같은 사용자의 프로세스라면 무엇이든 Runner를 Claim할 수 있으며, 이 노출은 [Runner 문서](runner-isolation.md#claim-exposure)에 적었다. 네 번째 단위 P1R-C04는 [대조](runner-isolation.md#reconciliation)를 더한다. gateway는 Claim할 때마다 자기 기록, Runner가 가진 것과 기억하는 것, `required` 실행이면 Runner가 조회한 DevGuard attempt를 맞춰 보고, 앞선 controller가 작업을 남긴 workspace에서만 대조를 마칠 때까지 새 실행, 패치, 승인 재개를 `RECOVERY_PENDING`으로 거부한다. 실행 중으로 확인한 프로세스는 다시 workspace를 점유하고, resuming으로 남은 승인은 Runner가 그 승인을 위해 만든 프로세스로 consumed가 되거나, Runner가 아무것도 만들지 않았음이 증명되면 다시 granted가 된다. 세 가지가 맞지 않거나 기록이 다른 Runner를 가리키는 workspace는 운영자가 판단할 때까지 계속 거부된다. 다섯 번째 단위 P1R-C05는 [Runner 손실](runner-isolation.md#runner-loss) 처리를 더한다. Runner가 kill되거나, 크래시하거나, 멈추거나, 호스트가 재시작하면 gateway는 그 실행들에 대한 `process_status`와 `read_process`에 기록으로 `lost_reason`과 함께 답하고, 끝나지 않은 실행은 `unknown`이 되어 운영자가 그 프로세스를 찾고 `codespace-mcp records`로 기록을 풀 때까지 workspace를 점유한다([운영](operations.md#runner-loss)). `required` 실행의 기록은 DevGuard가 그 attempt를 더는 과금하지 않을 때에만 운영자가 푼다. DevGuard는 attempt를 소유한 인스턴스의 조회에만 답하므로, 이는 DevGuard 관리자의 quiescence로만 확인할 수 있다. 그때까지 DevGuard는 잃은 Runner의 인스턴스를 유지하므로, 새 Runner는 `codespace` 소비자의 유일한 인스턴스로 등록할 수 없다. 마지막 단위가 qualification한 뒤에야 이 모드를 지원한다. 이 모드에서는 Runner가 DevGuard의 실행 소유자이며, DevGuard [ADR-001](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/ko/planning/decisions.md#adr-001--실행-소유자인-runner의-단일-등록)의 보완대로 소비자 secret을 자기 자격 파일에서 직접 읽는다. 그 gateway는 Runner가 등록하고 launch helper로 실행한다고 알릴 때만 `required` workspace를 받아들인다. 이 절의 나머지는 계획한 범위다. +**상태, P1-RECOVERY(소유자 지시 2026-10-14):** P1-RECOVERY는 DevGuard의 [P1-RECOVERY 계획](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/ko/planning/milestones/P1-RECOVERY.md#재기준-2026-10-14)에서 다시 잡은 기준대로 단위마다 PR 하나로 구현한다. 첫 단위 P1R-C01은 opt-in 모드인 [독립 Runner](runner-isolation.md#independent-runner)를 더한다. 운영자나 하네스가 Runner를 시작하고, Runner의 프로세스, 출력, timeout은 모든 gateway보다 오래 산다. gateway는 종료 의도대로 Runner에서 detach하거나 Runner를 중지한다. 두 번째 단위 P1R-C02는 [실행 기록](runner-isolation.md#process-records)을 더한다. gateway는 작업 데이터베이스에 쓰는 유일한 gateway로서 각 실행을 Runner에 보내기 전에 기록하고 끝날 때까지 따라간다. 세 번째 단위 P1R-C03은 [Claim](runner-isolation.md#claim)을 더한다. 한 번에 한 연결만 controller epoch 아래에서 Runner를 제어하고, 프로세스나 workspace를 바꾸는 요청은 모두 그 epoch를 함께 보낸다. 연결을 잃은 gateway는 재시작하지 않고 Runner를 다시 Claim한다. Runner를 제어하는 gateway가 없는 동안에는 Runner의 소켓에 닿을 수 있는 같은 사용자의 프로세스라면 무엇이든 Runner를 Claim할 수 있으며, 이 노출은 [Runner 문서](runner-isolation.md#claim-exposure)에 적었다. 네 번째 단위 P1R-C04는 [대조](runner-isolation.md#reconciliation)를 더한다. gateway는 Claim할 때마다 자기 기록, Runner가 가진 것과 기억하는 것, `required` 실행이면 Runner가 조회한 DevGuard attempt를 맞춰 보고, 앞선 controller가 작업을 남긴 workspace에서만 대조를 마칠 때까지 새 실행, 패치, 승인 재개를 `RECOVERY_PENDING`으로 거부한다. 실행 중으로 확인한 프로세스는 다시 workspace를 점유하고, resuming으로 남은 승인은 Runner가 그 승인을 위해 만든 프로세스로 consumed가 되거나, Runner가 아무것도 만들지 않았음이 증명되면 다시 granted가 된다. 세 가지가 맞지 않거나 기록이 다른 Runner를 가리키는 workspace는 운영자가 판단할 때까지 계속 거부된다. 다섯 번째 단위 P1R-C05는 [Runner 손실](runner-isolation.md#runner-loss) 처리를 더한다. Runner가 kill되거나, 크래시하거나, 멈추거나, 호스트가 재시작하면 gateway는 그 실행들에 대한 `process_status`와 `read_process`에 기록으로 `lost_reason`과 함께 답하고, 끝나지 않은 실행은 `unknown`이 되어 운영자가 그 프로세스를 찾고 `codespace-mcp records`로 기록을 풀 때까지 workspace를 점유한다([운영](operations.md#runner-loss)). `required` 실행의 기록은 DevGuard가 그 attempt를 더는 과금하지 않을 때에만 운영자가 푼다. DevGuard는 attempt를 소유한 인스턴스의 조회에만 답하므로, 이는 DevGuard 관리자의 quiescence로만 확인할 수 있다. 그때까지 DevGuard는 잃은 Runner의 인스턴스를 유지하므로, 새 Runner는 `codespace` 소비자의 유일한 인스턴스로 등록할 수 없다. 여섯 번째이자 마지막 단위 P1R-C06은 [qualification 절차](recovery-qualification.md)와 그 하네스 suite를 더하며, 이 모드는 그 실행이 통과시킨 조합에서만 지원한다. 아직 통과한 조합은 없다. 이 모드를 도입하고 되돌리는 방법은 [운영](operations.md#independent-runner-adoption)에 있다. 이 모드에서는 Runner가 DevGuard의 실행 소유자이며, DevGuard [ADR-001](https://github.com/novelKR/DevGuard/blob/165cb84a991fbc941b822681b47ef7dd07363878/docs/ko/planning/decisions.md#adr-001--실행-소유자인-runner의-단일-등록)의 보완대로 소비자 secret을 자기 자격 파일에서 직접 읽는다. 그 gateway는 Runner가 등록하고 launch helper로 실행한다고 알릴 때만 `required` workspace를 받아들인다. 이 절의 나머지는 계획한 범위다. P1-RECOVERY는 운영자가 선택하는 독립 Runner 모드와 명시적인 capability를 도입한다. 기존 모드의 종료·연결 소실 계약은 유지한다. InProcess는 Gateway와 PID를 공유하므로 Gateway 재시작 중 live 복구 대상에서 제외한다. diff --git a/docs/ko/operations.md b/docs/ko/operations.md index 4c596f6..a623640 100644 --- a/docs/ko/operations.md +++ b/docs/ko/operations.md @@ -115,7 +115,7 @@ export CODESPACE_RUNNER=uds export CODESPACE_RUNTIME_BIN="$PWD/dist/codespace-codex-runtime" ``` -worker는 같은 호스트에서 실행하는 별도 프로세스이며 컨테이너가 아닙니다. 게이트웨이가 전용 소켓 디렉터리를 만들고 자식 프로세스를 관리합니다. worker 연결이 끊기거나 게이트웨이가 종료되면 해당 worker의 프로세스도 종료됩니다. 재접속과 프로세스 복구는 지원하지 않습니다. MCP/HTTP 클라이언트 끊김은 그 프로세스를 죽이지 않습니다. 기본값은 `in-process`입니다. 프로세스가 게이트웨이보다 오래 살고, 게이트웨이가 그 프로세스를 작업 데이터베이스에 기록하며, 연결을 잃은 게이트웨이가 다시 Claim한 뒤 자기 기록과 대조하는 [독립 Runner](runner-isolation.md#independent-runner)는 아직 지원하는 모드가 아닙니다. 이런 Runner 자체를 잃으면 운영자가 `codespace-mcp records`로 그 Runner가 실행하던 것을 정리합니다([Runner 손실](#runner-loss)). +worker는 같은 호스트에서 실행하는 별도 프로세스이며 컨테이너가 아닙니다. 게이트웨이가 전용 소켓 디렉터리를 만들고 자식 프로세스를 관리합니다. worker 연결이 끊기거나 게이트웨이가 종료되면 해당 worker의 프로세스도 종료됩니다. 재접속과 프로세스 복구는 지원하지 않습니다. MCP/HTTP 클라이언트 끊김은 그 프로세스를 죽이지 않습니다. 기본값은 `in-process`입니다. 프로세스가 게이트웨이보다 오래 살고, 게이트웨이가 그 프로세스를 작업 데이터베이스에 기록하며, 연결을 잃은 게이트웨이가 다시 Claim한 뒤 자기 기록과 대조하는 [독립 Runner](runner-isolation.md#independent-runner)는 [qualification](recovery-qualification.md)이 통과시킨 조합에서만 지원하며, 아직 통과한 조합은 없습니다([도입과 되돌리기](#independent-runner-adoption)). 이런 Runner 자체를 잃으면 운영자가 `codespace-mcp records`로 그 Runner가 실행하던 것을 정리합니다([Runner 손실](#runner-loss)). @@ -141,11 +141,42 @@ worker는 같은 호스트에서 실행하는 별도 프로세스이며 컨테 패치 응답을 받지 못했다면 `operation_id` 또는 `operation_key` 중 하나만 지정해 `operation_status`를 조회합니다. 재시작 후 미완료 기록은 `unknown`이 되므로 파일을 확인한 뒤 다음 행동을 결정하세요. 프로세스는 `process_id`로 관리하며 `operation_status`로 복구할 수 없습니다. 프로세스가 죽을 때 `queued`이던 확인 홀드는 다시 acquire합니다. `resuming`이던 홀드는 패치 원장에서 복구하거나, 저장된 단말 결과를 재현하거나, `APPROVAL_AMBIGUOUS`를 반환할 수 있습니다. exec는 다시 spawn하지 않습니다. 독립 Runner에 연결한 게이트웨이는 `resuming`으로 남은 exec 홀드를 [대조](runner-isolation.md#reconciliation)할 때 정리합니다. Runner가 그 홀드를 위해 만든 프로세스로 consumed가 되거나, Runner가 아무것도 만들지 않았음이 증명되면 다시 granted가 됩니다. [재시도와 복구 규칙](agent-integration.md)을 참고하세요. + + +## 독립 Runner 도입과 되돌리기 + +[독립 Runner](runner-isolation.md#independent-runner)는 [qualification](recovery-qualification.md)이 통과시킨 조합에서만 지원하며, 아직 통과한 조합은 없습니다. 이 모드는 opt-in이고 기본값은 계속 `in-process`입니다. 통과한 조합에서만, 그것도 qualification을 받은 CodeSpace 빌드, DevGuard release, 정책 그대로 도입하세요. 그 밖의 것은 모두 새 후보입니다. + +1. **준비합니다.** 모든 workspace 밖에 상태 디렉터리를 두고, 실행 기록이 생기기 전의 CodeSpace 버전이 열지 않을 작업 데이터베이스([스키마 버전](#operations-database))와 Runner용 mode 0700 소켓 디렉터리를 준비합니다. DevGuard를 쓰면 Runner가 실행 소유자로서 `codespace` 소비자의 인스턴스 하나로 등록하므로, 그 소비자로 등록하는 다른 게이트웨이가 있으면 먼저 멈춥니다. +2. **Runner를 시작합니다.** qualification이 시작하듯 Runner를 자기 세션에서 시작해, 터미널이 닫히거나 어떤 게이트웨이가 끝나도 Runner에 닿지 않게 합니다. Linux에서는 `setsid`가 이렇게 하며, 아래 Python 줄은 macOS와 Linux 모두에서 같은 일을 합니다. DevGuard를 쓰면 [상태 전용 연결](devguard-integration.md#devguard-status-connection)의 소비자 설정과 release의 `devguard-launch`를 줍니다. DevGuard 없이 쓰면 `--devguard-*` 설정을 빼며, 이때 `required` workspace는 거부됩니다. + + ```bash + python3 -c 'import os, sys; os.setsid(); os.execv(sys.argv[1], sys.argv[1:])' \ + "$PWD/dist/codespace-codex-runtime" "$RUNNER_DIR/runner.sock" --lifecycle independent \ + --devguard-socket "$DEVGUARD_SOCKET" --devguard-consumer codespace \ + --devguard-generation "$DEVGUARD_GENERATION" \ + --devguard-credential-file "$DEVGUARD_CREDENTIAL_FILE" \ + --devguard-launch-helper "$DEVGUARD_LAUNCH" >>"$STATE_DIR/runner.log" 2>&1 ## Runner 손실 -[독립 Runner](runner-isolation.md#independent-runner)에 연결한 게이트웨이에만 해당하며, 이 모드는 아직 지원하지 않습니다. 그 Runner가 kill되거나, 크래시하거나, 멈추거나, 호스트가 재시작하면 Runner가 실행하던 것을 Runner에서 더는 알아낼 수 없고, 그 프로세스는 Runner보다 오래 살 수 있습니다([Runner 손실](runner-isolation.md#runner-loss)). 게이트웨이는 그 프로세스에 대해 기록으로 `lost_reason`과 함께 `unknown`이라고 답합니다. 그 프로세스가 점유하던 workspace는 운영자가 그중 아무것도 실행 중이지 않음을 확인하고 기록을 풀 때까지 계속 거부되며, 다음 게이트웨이에서는 `RECOVERY_PENDING`으로 거부됩니다. CodeSpace는 이 프로세스를 하나도 종료하지 않고, 아무것도 다시 시작하지 않으며, 이 프로세스에 대해 DevGuard에 묻지도 않습니다. +[독립 Runner](runner-isolation.md#independent-runner)에 연결한 게이트웨이에만 해당하며, 이 모드는 [qualification](recovery-qualification.md)을 통과한 곳에서만 지원합니다. 그 Runner가 kill되거나, 크래시하거나, 멈추거나, 호스트가 재시작하면 Runner가 실행하던 것을 Runner에서 더는 알아낼 수 없고, 그 프로세스는 Runner보다 오래 살 수 있습니다([Runner 손실](runner-isolation.md#runner-loss)). 게이트웨이는 그 프로세스에 대해 기록으로 `lost_reason`과 함께 `unknown`이라고 답합니다. 그 프로세스가 점유하던 workspace는 운영자가 그중 아무것도 실행 중이지 않음을 확인하고 기록을 풀 때까지 계속 거부되며, 다음 게이트웨이에서는 `RECOVERY_PENDING`으로 거부됩니다. CodeSpace는 이 프로세스를 하나도 종료하지 않고, 아무것도 다시 시작하지 않으며, 이 프로세스에 대해 DevGuard에 묻지도 않습니다. 1. **기록을 나열합니다.** 아직 workspace를 점유할 수 있는 기록을 보여 줍니다. 데이터베이스를 읽기만 하므로 게이트웨이가 실행 중이어도 됩니다. diff --git a/docs/ko/recovery-qualification.md b/docs/ko/recovery-qualification.md new file mode 100644 index 0000000..292a200 --- /dev/null +++ b/docs/ko/recovery-qualification.md @@ -0,0 +1,224 @@ + + +# Gateway 복구 qualification 절차 + +[English](../recovery-qualification.md) | [한국어](recovery-qualification.md) + +이 문서는 Gateway 복구(P1-RECOVERY)의 opt-in 모드인 [독립 Runner](runner-isolation.md#independent-runner)를 qualification하는 절차입니다. 어떤 조합을 지원한다고 주장하는지, 각 주장을 어떻게 시험하고 측정하는지, 무엇이 통과인지, 실행이 어떤 증거를 남기는지를 정합니다. [CS-RG qualification 절차](cs-rg-qualification.md)를 따르며, 그 후보 조합, 증거, 확인 방식을 그대로 쓰고, 하네스도 같은 [`scripts/qualify-devguard.py`](../../scripts/qualify-devguard.py)입니다. 이 절차는 첫 qualification 실행 전에 하네스와 함께 고정합니다. 첫 실행 뒤에 둘 중 하나라도 바꾸면 새 절차가 되며, 다른 변경처럼 검토하고 영향받는 실행은 다시 합니다. 각 단위가 무엇을 만들었는지는 [결합 로드맵](devguard-integration.md#resource-gateway-recovery)에 있고, 이 문서는 그 결과가 qualification을 통과하는지를 판단합니다. + +**현재 상태:** 첫 qualification 실행 전에 하네스와 함께 고정했습니다. 아직 통과한 조합이 없으므로, 독립 Runner는 어디에서도 지원하는 모드가 아닙니다. + + + +## 후보 조합 + +한 번의 실행은 정확히 한 조합을 qualification합니다. 조합은 CS-RG처럼 하네스의 `candidate` 스위트가 기록합니다([후보 조합](cs-rg-qualification.md#candidate)). CodeSpace commit과 그 code tree, DevGuard client pin, 기록된 패치를 적용한 Codex pin, 바이너리, 설치된 DevGuard release와 그 정책, 도구와 호스트입니다. 독립 Runner는 `uds` worker와 같은 `codespace-codex-runtime` 바이너리입니다. 실행은 설치된 DevGuard release와 정책을 있는 그대로 쓰며, 둘 다 바꾸지 않고 아무것도 설치하거나 provisioning하지 않습니다. 코드, 바이너리, release, 정책, Codex, 도구 가운데 무엇이든 바뀌면 새 후보가 되고 영향받는 실행을 다시 합니다. 문서만 바꾼 commit은 후보를 유지합니다. + + + +## 지원 모드 + +주장하는 칸은 각각 따로 시험합니다. 어떤 칸도 다른 칸에서 추론하지 않습니다. CS-RG가 qualification한 모드는 그 주장을 유지하며, 실행은 그 주장이 여전히 맞음을 보여 줍니다([회귀](#recovery-regressions)). + +| 플랫폼 | runner | I/O | 참여 | 주장 | 근거 | +| --- | --- | --- | --- | --- | --- | +| macOS, 실제 호스트 증거가 있는 설치 서비스 | `--runner-lifecycle independent`를 쓴 `uds`, 운영자가 시작한 독립 Runner | pipe, PTY | `off`와 `required` | qualification | 두 참여 모두로 실행하는 recovery 스위트와 gateway 재시작 측정 | +| Linux | 같음 | pipe, PTY | `off` | CI 기능 시험만 | Linux CI에서 실행하는 CodeSpace의 독립 Runner 시험. 하네스의 `off` 시나리오도 Linux에서 실행됨. 기능 확인일 뿐 Linux qualification이 아님 | +| Linux | 같음 | pipe, PTY | `required` | DG-LINUX 전까지 지원하지 않음 | Linux의 모든 `required` 실행처럼 아무것도 실행하기 전에 거부 | +| macOS, Linux | 같음 | pipe, PTY | Runner 자신이 재시작한 뒤 프로세스 복원 | 지원하지 않음 | P1-RECOVERY 범위 밖. gateway가 잃은 Runner 자리에 새 Runner를 받아들이지 않음을 `runner-replaced`가 보여 줌 | + +독립 모드는 opt-in입니다. 이 모드를 기본값으로 바꾸는 것은 이 절차 밖의 별도 결정입니다. + + + +## 하네스 실행 + +하네스는 CS-RG 스위트처럼 qualification 호스트에서 실행합니다([하네스 실행](cs-rg-qualification.md#running)). 시스템 Python을 쓰고, 외부 패키지를 쓰지 않으며, 하네스 자신과 하네스가 시작하는 모든 프로세스는 최소 환경만 받습니다. 독립 Runner는 하네스가 운영자처럼 직접 시작합니다. `codespace-codex-runtime --lifecycle independent`를 자기 세션에서, 전용 소켓 디렉터리로 실행하고, 그 로그를 증거로 남깁니다. `required`이면 쓰는 authority의 DevGuard 설정을 주며, credential 파일은 경로만 넘기고 하네스는 읽지 않습니다. LaunchAgent로 설치하는 것은 없고 정책도 바꾸지 않습니다. gateway는 `--runner uds --runner-socket --runner-lifecycle independent`, 종료 의도, 작업 데이터베이스로 실행하며, `required`이면 `--devguard register`도 줍니다. + +| 스위트 | 보여 주는 것 | 추가 인자 | +| --- | --- | --- | +| `recovery-lifecycle` | gateway가 끝나는 모든 방식을 지나는 Runner, timeout, 설정([생명주기](#recovery-lifecycle)) | `--participation`. macOS에서 `required`이면 `faults`처럼 `--fixture-bin`과 `--fixture-helper` | +| `recovery-store` | 실행 기록과 작업 데이터베이스의 단일 writer([실행 기록](#recovery-store)) | `recovery-lifecycle`과 같음 | +| `recovery-fence` | Claim, controller epoch, 소켓에 닿은 다른 프로세스([Claim과 fence](#recovery-fence)) | `recovery-lifecycle`과 같음 | +| `recovery-reconcile` | 재시작 뒤 대조, 그 경쟁, authority 장애([대조](#recovery-reconcile)) | `recovery-lifecycle`과 같음 | +| `recovery-observation` | 출력 공백, 보관, Runner 손실([관찰과 Runner 손실](#recovery-observation)) | `recovery-lifecycle`과 같음 | +| `recovery` | 위 스위트 모두, 그 뒤 [gateway 재시작 측정](#recovery-measurement) | 위와 같고, macOS에서 `required`이면 `control` 스위트의 `--release`, `--work`, `--qualify-bin`, `--devguard-checkout`, `--load`, `--rehearsal`, `--repetitions` | + +`--participation`은 `off`, `required`, 또는 둘 다(기본)를 받습니다. `off` 시나리오는 authority 없이 모든 플랫폼에서 실행합니다. macOS에서 `required` 시나리오는 CS-RG 장애 사례처럼 DevGuard의 fixture authority를 대상으로 실행하며, 시나리오마다 새 fixture authority가 설치 정책처럼 인스턴스 하나짜리 `codespace` 소비자를 스스로 provisioning합니다. 관리 대상 workspace는 50 mCPU, 32 MiB, 4 tasks를 요청합니다. Linux에서 `required` 시나리오는 이유와 함께 실행하지 않은 것으로 기록하고 세지 않습니다. 스위트는 실행한 시나리오가 모두 통과하고 하나 이상 실행했을 때 통과합니다. 종료 상태는 CS-RG 스위트와 같습니다. + +```bash +env -i HOME="$HOME" TMPDIR="$TMPDIR" PATH="/bin:/usr/bin:/bin:/usr/sbin:/sbin" \ + /usr/bin/python3 scripts/qualify-devguard.py recovery --out evidence/<새 디렉터리> \ + --mcp-bin --runtime-bin \ + --fixture-bin --fixture-helper \ + --release --work <새 디렉터리> --qualify-bin \ + --devguard-checkout +``` + + + +## 기능 스위트 + +시나리오마다 작업 디렉터리, 작업 데이터베이스, Runner를 따로 쓰고, `required`이면 fixture authority도 따로 씁니다. 시나리오가 남긴 실행 중인 것은 그 뒤에 끝냅니다. 시나리오는 확인 항목이 모두 통과해야 통과합니다. 시나리오가 따로 말하지 않으면 gateway의 프로세스 timeout은 10분이고 종료 의도는 `detach`입니다. I/O 칸의 "둘 다"는 pipe로 한 번, PTY로 한 번 실행한다는 뜻이고, 참여 칸은 그 시나리오를 실행하는 참여를 나열합니다. + + + +### 생명주기 + +| 시나리오 | I/O | 참여 | 통과 조건 | +| --- | --- | --- | --- | +| exit-kinds | 둘 다 | `off`, `required` | 프로세스가 실행되는 동안 gateway가 끝나는 모든 방식으로 gateway를 끝내고, 매번 다음 gateway를 시작합니다. 종료 의도 `detach`로 입력을 닫거나 SIGTERM을 보내고, `stop`으로 SIGUSR1을 보내고(재시작은 언제나 detach), SIGKILL을 보냅니다. Runner와 프로세스는 그 모두보다 오래 살고, 다음 gateway마다 다음 epoch로 Runner를 Claim해 같은 프로세스가 실행 중임을 확인하며 앞 gateway가 멈춘 곳부터 출력을 이어 읽습니다. 그 뒤 `stop`인 gateway의 입력을 닫으면 프로세스가 끝나고, Runner는 소켓을 지운 뒤 종료 상태 0으로 끝납니다 | +| original-timeout | 둘 다 | `off`, `required` | 두 프로세스가 gateway 재시작을 지나 timeout에 이릅니다. 첫째는 gateway가 하나도 실행되지 않을 때, 둘째는 프로세스 timeout이 훨씬 긴 뒤 gateway가 요청을 받는 동안입니다. 각각 시작할 때 기록에 있던 deadline에 끝나고, 재시작이 옮기거나 늘린 deadline에 끝나지 않으며, `timeout`으로 보고됩니다 | +| runner-sigterm | | `off`, `required` | 운영자가 멈추듯 Runner에 SIGTERM을 보내면 controller의 Stop처럼 pipe와 PTY 프로세스를 끝내고, 종료 상태 0으로 끝나며 소켓을 지웁니다. 그 gateway는 두 프로세스 모두 실행 중이 아니라고 보고하고 새 작업을 거부하며, authority가 있으면 두 attempt가 해제되고 인스턴스가 retire됩니다 | +| invalid-settings | | `off` | 서로 맞지 않는 설정은 Runner에 닿기 전에 거부됩니다. 독립 생명주기 없이 준 종료 의도, 종료 의도 없이 준 독립 생명주기, in-process, 시작할 worker가 있는 경우, 소켓이나 작업 데이터베이스가 없는 경우, 독립 Runner에 연결한 managed gateway, managed worker에 연결한 독립 gateway입니다. Runner는 전용이 아닌 소켓 디렉터리, 공유 디렉터리, 소켓이 없는 경우, 알 수 없는 생명주기, 건네받은 credential, 다른 Runner가 듣고 있는 소켓을 거부합니다. 듣고 있던 Runner는 그대로입니다. Claim을 기록하지 않았고, 그 뒤 gateway가 epoch 1로 Runner를 Claim해 명령을 실행합니다 | + + + +### 실행 기록 + +| 시나리오 | I/O | 참여 | 통과 조건 | +| --- | --- | --- | --- | +| records | 둘 다 | `off`, `required` | Runner에 보낸 실행에 대해, gateway 재시작 뒤에도 작업 데이터베이스가 다음을 가집니다. Hello가 밝힌 Runner의 정체, controller epoch, workspace, argv가 아닌 argv의 digest, pipe 실행의 root 프로세스(PTY의 root는 적지 않음), 원래 deadline, 그리고 뒤 gateway가 종료한 뒤의 끝난 방식 | +| writer-lock | | `off` | 첫 gateway가 요청을 받는 동안 같은 데이터베이스로 시작한 둘째 gateway는 정해진 시간만 기다린 뒤 lock을 가진 쪽을 밝히며 시작하지 않고, Runner에 닿지 않습니다. 첫 gateway가 끝나는 동안 시작한 gateway는 그것을 기다린 뒤 요청을 받으며, 그 뒤에 Runner를 Claim합니다 | +| schema | | `off` | 새 스키마 버전의 데이터베이스와 데이터베이스가 아닌 파일은 Runner에 닿기 전에 거부되고 바뀌지 않습니다 | +| records-command | | `off` | `codespace-mcp records list`는 아직 workspace를 점유할 수 있는 기록을 나열하고, `records release`는 gateway가 실행되지 않을 때만, 끝나지 않은 실행만, 그리고 그 Runner의 PID가 어떤 프로세스로 실행 중이 아닐 때만 기록을 풉니다 | + + + +### Claim과 fence + +| 시나리오 | I/O | 참여 | 통과 조건 | +| --- | --- | --- | --- | +| foreign-connection | 둘 다 | `off`, `required` | gateway가 Runner를 제어하는 동안 같은 사용자의 다른 프로세스가 Runner의 Hello를 읽습니다. 그 Claim은 controller가 연결되어 있어 거부되고, controller의 epoch든 다른 epoch든 프로세스를 바꾸려는 요청은 모두 stale로 거부되며, Detach는 아무것도 바꾸지 않습니다. gateway는 그 뒤에도 Runner와 그 프로세스를 계속 제어합니다 | +| stale-epoch | | `off` | 두 gateway가 차례로 detach한 뒤, 다른 작업 데이터베이스를 위한 Claim은 거부됩니다. gateway가 Runner를 제어하지 않는 동안 같은 사용자의 프로세스라면 누구나 할 수 있는, Runner 자신의 데이터베이스를 위한 Claim([노출](runner-isolation.md#claim-exposure), 결정 D6이 받아들임)은 다음 epoch로 허용되고 그 PID와 함께 기록되며, 그 연결이 이전 epoch나 epoch 없이 보낸 요청은 stale로 거부됩니다. 다음 gateway는 그다음 epoch로 Runner를 Claim해 같은 프로세스를 찾고, 그 출력을 있던 곳부터 이어 읽습니다 | +| stopped-gateway | | `off` | SIGSTOP으로 멈춘 controller gateway는 연결을 유지하므로 아무도 밀어내지 못합니다. 다른 프로세스의 Claim은 거부되고, 둘째 gateway는 데이터베이스 writer 때문에 시작하지 않습니다. 멈춘 gateway를 죽이면 다음 gateway가 epoch 2로 Runner를 Claim해 프로세스가 실행 중임을 확인합니다 | +| concurrent-claims | | `off` | gateway가 Runner를 제어하지 않는 동안 연결 여덟 개가 Runner 자신의 데이터베이스로 한꺼번에 Claim하면, 정확히 하나만 허용되고 나머지는 controller가 연결되어 있어 거부됩니다. 그 연결이 detach하면 다음 gateway가 그다음 epoch로 Runner를 Claim합니다 | +| other-database | | `off` | Runner의 첫 Claim이 밝힌 것과 다른 작업 데이터베이스를 쓰는 gateway는, gateway가 Runner를 제어하든 아니든 거부되고 아무것도 시작하지 않습니다 | +| concurrent-gateways | | `off` | Runner의 데이터베이스로 한꺼번에 시작한 gateway 둘 가운데 하나가 요청을 받으며 Runner를 Claim하고, 다른 하나는 writer를 기다린 뒤 시작하지 않으며 Runner에 닿지 않습니다 | + + + +### 대조 + +| 시나리오 | I/O | 참여 | 통과 조건 | +| --- | --- | --- | --- | +| running | 둘 다 | `off`, `required` | 프로세스가 실행 중일 때 gateway를 죽이면, 다음 gateway는 그 workspace를 대조할 때까지 새 작업을 `RECOVERY_PENDING`으로 거부하고, 그 뒤 그 프로세스를 위해 workspace를 다시 점유합니다(`WORKSPACE_BUSY`). 남은 작업이 없던 workspace는 곧바로 새 작업을 받습니다. 물려받은 프로세스를 종료하면 그 workspace가 풀리며, 거부된 명령은 실행되지 않았습니다 | +| terminal | 둘 다 | `off`, `required` | gateway가 하나도 실행되지 않는 동안 프로세스가 끝나면, 다음 gateway는 끝난 방식을 Runner에서 알아내 기록하고 workspace를 풉니다 | +| approval | | `off`, `required` | gateway를 죽이기 전에 잡아 두고 허가한 승인은 다음 gateway가 한 번 재개하며, 다시 재개하면 같은 프로세스를 돌려줍니다. gateway를 죽일 때 재개가 진행 중이던 승인은(재개를 보낸 뒤 0에서 160 ms 사이의 여러 시점) 다음 gateway가 Runner가 그 승인을 위해 만든 프로세스로 재개하거나, Runner가 아무것도 만들지 않았음이 증명되면 그때 시작합니다. 각 명령은 정확히 한 번 실행되었습니다 | +| authority-outage | | `required` | fixture authority가 lock을 잡고 있는 동안 죽인 gateway를 바꿉니다. 물려받은 관리 대상 프로세스는 곧바로 답하고, Runner가 그 attempt를 authority에 물을 수 없는 동안 그 workspace는 `RECOVERY_PENDING`으로 거부되며, 관리하지 않는 작업은 실행됩니다. authority가 답하면 gateway가 이를 대조해 각 workspace를 그 프로세스를 위해 다시 점유하고, 그 프로세스를 종료하면 attempt가 해제됩니다 | +| missing-record | | `off` | gateway가 실행되지 않는 동안 데이터베이스에서 기록 하나를 지우면, 다음 gateway는 Runner가 가진 그 프로세스를 여전히 조회할 수 있고, 그 workspace는 무엇을 시작하는 대신 운영자를 위해 `RECOVERY_PENDING`으로 거부하며, 다른 workspace는 새 작업을 받습니다 | +| kill-during-exec | 둘 다 | `off`, `required` | `exec_command`가 진행 중일 때 0에서 160 ms 사이의 여러 시점에 gateway를 죽이면, 다음 gateway가 각각 남긴 것을 대조하며, 어떤 명령도 두 번 이상 실행되지 않았습니다 | +| kill-during-terminate | 둘 다 | `off`, `required` | `terminate_process`가 진행 중일 때 같은 시점들에 gateway를 죽이면, 다음 gateway는 프로세스를 종료됨으로 보고하거나, 아직 실행 중으로 보고한 뒤 종료합니다. 각 프로세스는 한 번 실행되고 한 번 종료로 끝나며, 그 뒤 그 workspace는 새 작업을 받습니다 | + + + +### 관찰과 Runner 손실 + +| 시나리오 | I/O | 참여 | 통과 조건 | +| --- | --- | --- | --- | +| ring-overflow | 둘 다 | `off` | gateway가 하나도 실행되지 않는 동안 프로세스가 Runner의 256 KiB 출력 ring보다 많이 씁니다. 다음 gateway는 마지막 읽기가 멈춘 곳부터 이어 읽고, 돌려주는 출력 앞의 출력을 잃었다는 것과 그 이유(`ring_overflow`)를 알리며, 가장 최근 출력은 온전히 돌려줍니다. 상태도 이를 알립니다 | +| retention | | `off` | Runner가 보관하는 것보다 많은 프로세스가 끝나면(64개, 그리고 핸들을 놓은 64개 더), 가장 오래된 것에 대해 `process_status`와 `read_process`는 기록으로 답합니다. 출력 없이 끝난 방식과 `retention_expired`로 답하며, 다음 gateway에서도 같습니다 | +| runner-loss | 둘 다 | `off`, `required` | Runner를 SIGKILL로 죽이면 그 gateway는 기록으로 답합니다. 실행 중이던 프로세스는 `unknown`, 끝난 프로세스는 끝난 그대로, 둘 다 `runner_lost`와 함께이며, 새 작업은 `RUNNER_DISCONNECTED`로 거부합니다. 출력하지 않는 pipe 프로세스는 죽은 Runner보다 오래 삽니다. `records list`는 그 기록을 보여 주고, `records release`는 gateway가 실행 중인 동안과 그 프로세스가 실행 중인 동안 그 기록을 거부합니다. 운영자가 프로세스를 끝낸 뒤(authority가 있으면 DevGuard가 attempt를 해제하고 잃은 Runner의 인스턴스를 retire한 뒤) 기록이 풀리고, 새 Runner와 gateway가 그 workspace에서 새 작업을 받습니다. PTY의 root 프로세스는 기록에 적지 않으므로, 운영자가 풀기 전에 끝냅니다 | +| runner-replaced | | `off` | 첫 Runner를 죽인 뒤 다른 Runner가 그 소켓에서 답하면, gateway는 그것을 옛 Runner로 받아들이지 않고, 옛 Runner의 프로세스를 `runner_lost`와 함께 `unknown`으로 보고하며 새 작업을 거부합니다. 다음 gateway는 운영자가 그 기록을 풀 때까지 그 workspace를 계속 거부하고, 다른 곳에서는 새 작업을 받습니다 | + + + +### 공백 + +다음은 실행할 때마다 이유와 함께 실행하지 않은 시나리오로 기록하거나 여기에 적으며, 어느 것도 통과로 세지 않습니다. + +| 공백 | 이유와, 대신 그 동작을 보여 주는 것 | +| --- | --- | +| 호스트 재시작 | 실행 중에 만들 수 없습니다. CodeSpace 시험은 기록에 다른 boot ID를 주어([`crates/server/src/mcp.rs`](../../crates/server/src/mcp.rs)) 재시작이 `host_rebooted`로 보고됨을 보여 주며, 실제 재시작은 qualification하지 않은 채로 남습니다 | +| Runner 자신이 재시작한 뒤 프로세스 복원 | 지원하지 않으며 P1-RECOVERY 범위 밖입니다. `runner-replaced`가 거부를 보여 줍니다 | +| gateway가 Runner를 제어하지 않는 동안 다른 프로세스의 Claim | Runner 사용자의 프로세스라면 설계상 가능합니다(결정 D6이 이 노출을 받아들임). `stale-epoch`는 이를 막는 것이 아니라 기록하고 fence함을 보여 줍니다 | +| gateway가 읽지 않는 동안 256 KiB ring을 넘는 출력 | 설계상 잃습니다. `ring-overflow`와 측정은 손실이 조용히 일어나지 않고 늘 알려짐을 보여 줍니다 | +| Runner를 잃은 뒤의 PTY 프로세스 | 기록에 root를 적지 않으므로 `records release`는 그것이 실행 중인지 알 수 없습니다. 운영자가 먼저 끝냅니다 | +| runner 통신에서 준비나 launch 메시지의 재전송 | CS-RG처럼 주입하지 않습니다. epoch fence가 이전 controller의 메시지를 거부합니다(`foreign-connection`, `stale-epoch`) | +| Linux `required` | DG-LINUX 전까지 지원하지 않습니다 | + + + +## gateway 재시작 측정 + +이 측정은 관리 대상 pipe와 PTY 프로세스가 구간 내내 실행되는 동안, 유휴 상태와 DG-1의 개발 부하 아래에서, 설치된 서비스를 대상으로 gateway를 거듭 재시작하고, 재시작할 때마다 [완료 증거](#recovery-completion)가 요구하는 것을 확인합니다. macOS에서 `required`로 관리 대상으로 실행하며, 그 밖의 곳에서는 같은 실행이 `independent-off`라는 이름의 하네스 자체 시험이 되고 qualification을 통과하지 않습니다. + + + +### 구간, 주기, 표본 + +한 번의 반복은 10분 유휴 구간과 그 뒤 30분 이상의 부하 구간이고, 절차는 세 번 반복합니다. 구간마다 새 Runner를 시작하며, 이 Runner는 `codespace` 소비자의 유일한 인스턴스로 등록합니다(결정 D12). 첫 gateway는 구간 내내 실행되는 프로세스를 시작합니다. 각각 25 mCPU, 16 MiB, 2 tasks를 요청하는 관리 대상 workspace의 `cat`(pipe)과 `sleep`(PTY), 그리고 관리하지 않는 workspace에서 0.2초마다 번호 붙은 줄을 쓰는 PTY와, 하네스가 신호를 줄 때마다 출력 ring이 담는 것보다 많은 번호 붙은 줄 40,000개를 한꺼번에 쓰는 pipe입니다. 요청을 받는 gateway가 무엇이든 그것을 통해 `control` 스위트의 일정대로 제어 경로를 표본으로 잽니다([구간과 표본](cs-rg-qualification.md#control-intervals)). status는 1초마다, 새 관리 대상 종료는 20초마다, I/O echo는 5초마다, 관리 대상 거부는 30초마다, 그리고 host, service, footprint(Runner와 gateway의 것)입니다. 두 출력 흐름은 2초마다 읽습니다. 요청을 받는 gateway가 없을 때 돌아온 일정은 표본이 아닌 `restarting`으로 기록하며, 기대하는 표본 수는 gateway가 요청을 받은 시간을 따릅니다. + +구간은 gateway 주기의 연속입니다. 첫 주기는 20초이고, 그 뒤의 gateway는 모두 프로세스 timeout 20초로 실행을 시작하며, 그 주기의 길이는 그 주기의 timeout 대상이 정합니다. 각 주기에서: + +1. 요청을 받는 gateway는 그 주기의 epoch로 Runner를 Claim했고, 둘째 주기부터는 오래 실행되는 프로세스가 모두 `running`임을 확인합니다. +2. 같은 사용자의 다른 프로세스가 소켓에 연결합니다. 그 Claim은 거부되고, 쓰기, 크기 변경, 프로세스 종료, workspace의 프로세스 종료, Runner 중지 요청을 이전 epoch, 현재 epoch, epoch 없이 보내면 각각 `STALE_CONTROLLER`로 거부됩니다. +3. 같은 작업 데이터베이스로 경쟁 gateway를 시작합니다. 경쟁 gateway는 writer를 기다린 뒤 포기해야 하며 Runner에 닿지 않습니다. 주기가 끝날 때까지 기다리고 있으면 죽입니다. +4. 재시작 전에 허가한 승인은 gateway가 그 workspace를 대조한 뒤 재개하고, 다시 재개해 같은 프로세스를 돌려받아야 합니다. 그 명령은 한 번 실행됩니다. +5. deadline이 지난 timeout 대상마다, 그 프로세스가 처음 사라진 것으로 보인 시각을 시작할 때 기록한 deadline과 비교하고, 요청을 받는 gateway가 어떻게 보고하는지 판단합니다. +6. 구간이 끝나지 않으면 gateway는 그 주기의 timeout 대상(관리 대상이거나 아닌, pipe이거나 PTY인 `sleep`을 번갈아)을 시작하고, 재시작 2초 전에 승인을 잡아 두고 허가한 뒤 끝납니다. 입력 닫기, SIGTERM, SIGUSR1, SIGKILL을 번갈아 쓰며, 모두 종료 의도 `detach`입니다. SIGKILL 전에는 승인의 재개를 보내 0에서 160 ms 동안 진행 중으로 둡니다. 네 주기씩 묶어, gateway가 대상의 deadline 2초 전에 끝나고 다음 gateway가 4초 뒤에 시작해 gateway가 없는 동안 deadline이 오게 하거나, deadline 3초 뒤에 끝나고 다음 gateway가 곧바로 시작합니다. 넷째 주기와 그 뒤 대략 아홉 주기마다, gateway가 읽지 않는 동안 pipe가 한꺼번에 씁니다. + +구간이 끝나면 마지막 gateway가 두 흐름을 한 번 더 읽고, 마지막 승인을 재개하고, 오래 실행되는 프로세스를 모두 종료하며(각 종료는 효과가 있어야 함), 입력을 닫아 끝납니다. 그 뒤 운영자가 하듯 Runner를 drain합니다([도입과 되돌리기](operations.md#independent-runner-adoption)). 종료 의도 `stop`인 gateway가 다음 epoch로 Runner를 Claim해 중지하면, Runner는 종료 상태 0으로 끝나고 소켓을 지우며, 하네스가 프로세스를 먼저 끝냈으므로 종료하는 것이 없습니다. 서비스가 있으면 그 인스턴스가 retire되고 소비자의 자원은 아무것도 남지 않습니다. 부하는 `control` 스위트의 것이고([부하](cs-rg-qualification.md#control-load)) 기본은 cold입니다. 실행하는 동안 화면과 시스템이 잠들지 않게 합니다. + + + +### 판정 + +구간은 다음과 같으면 잘못된 관찰인 **판정 불가**입니다. Runner가 소비자의 인스턴스로 등록하지 않았거나, 오래 실행되는 프로세스를 시작하지 못했거나, 시작할 때 다른 작업이 자원을 차지하고 있었거나 `codespace` 인스턴스가 활성이었거나, journal을 읽을 수 없었거나, 원시 증거를 쓰지 못했거나, 서비스를 관찰할 수 없었거나 다른 release가 선택되었거나, 부하가 한도 안에 끝나지 않은 경우입니다. + +그렇지 않으면, gateway가 요청을 받은 시간에 대해 `control` 스위트의 [규칙](cs-rg-qualification.md#control-verdicts)으로 제어 경로가 실패하거나, 다음 중 하나일 때 **실패**입니다. + +- gateway가 요청한 것과 다르게 끝났거나, gateway가 다음 epoch가 아닌 epoch로 Runner를 Claim했거나, 오래 실행되는 프로세스가 다음 gateway에게 `running`이 아님 +- 다른 프로세스의 요청이 무엇인가를 바꿨거나(stale mutation), stale이 아닌 다른 이유로 거부되었거나, 그 Claim이 허용됨 +- 경쟁 gateway가 Runner에 닿았거나, 요청을 받았거나, writer가 아닌 다른 이유로 끝남 +- timeout 대상이 원래 deadline보다 2초 넘게 더 살았거나 기록된 deadline이 바뀜(연장), deadline보다 0.25초 넘게 먼저 끝남, 또는 `timeout`으로 보고되지 않음 +- 승인이 재개되지 않았거나, 두 번째 재개가 다른 프로세스를 돌려줬거나, 그 명령이 두 번 실행되었거나 한 번도 실행되지 않음 +- 태그를 붙인 명령이 두 번 실행되었거나, 거부된 명령이 실행됨 +- Runner가 공백을 알리지 않은 곳에서 출력 흐름이 줄을 건너뛰거나 되풀이함 +- drain이 실패했거나, 마지막 gateway가 입력을 닫았을 때 종료 상태 0으로 끝나지 않음 +- 주기가 오류로 중단됨 + +관찰이 너무 적어도 **판정 불가**입니다. 25초마다 한 번꼴로 계획한 재시작의 80%보다 적게 재시작했거나, gateway가 없는 동안 온 deadline이 없거나, 재개가 진행 중인 SIGKILL이 없거나, 어떤 출력 흐름에서 읽은 것이 없거나, 끝을 관찰하지 못한 timeout 대상이 있거나, `control` 스위트의 관찰 부족이 있거나, 부하가 DG-1의 것이 아닌 경우입니다. 그 밖에는 **통과**입니다. + +반복, 조합, 실행 전체는 `control`처럼 판정합니다. 조합은 세 반복이 모두 통과하면 qualification을 통과하고, 실행 전체는 전체 절차(관리 대상으로, 10분과 30분을 세 번)에서만 qualification을 통과합니다. `--rehearsal`(유휴 2분, 부하 3분, 한 번 반복)은 실행을 시험해 보는 것이며 qualification을 통과하지 않습니다. `recovery` 스위트는 기능 시나리오가 통과하고 측정이 qualification을 통과했을 때만 qualification을 통과합니다. + + + +## 완료 증거 + +| P1-RECOVERY가 보여야 하는 것 | 보여 주는 곳 | +| --- | --- | +| 재시작 뒤 같은 실행에 다시 닿음 | 모든 gateway를 지나는 주기마다의 `running` 상태와 출력 흐름. 생명주기 `exit-kinds`, fence `stale-epoch` | +| 두 번 실행된 실행 없음 | 측정의 승인과 태그를 붙인 명령, 서비스가 있으면 gateway들이 실행마다 기록한 launch. 대조 `approval`, `kill-during-exec`, `kill-during-terminate` | +| stale mutation 없음 | 주기마다의 fence 확인과 경쟁 gateway. fence 스위트 | +| timeout 연장 없음 | 측정의 timeout 대상. 생명주기 `original-timeout` | +| 명확한 공백 | 실행마다 기록하는 [공백](#recovery-gaps) | +| 기존 모드가 계속 통과 | [회귀](#recovery-regressions) | +| 지원 모드 | [표](#recovery-supported-modes). Linux `off`는 CI 기능 확인일 뿐 | + + + +## qualification 실행 + +이 하네스가 들어간 병합된 main으로, qualification 호스트에서 다음 순서로 합니다. + +1. **고정하기 전에** Mac checkout을 `--prune`으로 fetch해 main으로 fast-forward하고, 메모리 pressure가 정상인지, 설치된 DevGuard release와 정책이 CS-RG가 qualification한 그대로인지 확인합니다. +2. **후보를 고정합니다.** `candidate`가 통과해야 합니다. +3. **`recovery`를 실행합니다.** 두 참여 모두로, 기능 스위트를 실행한 뒤 측정을 세 번 반복합니다. +4. **drain합니다.** 구간마다 그 Runner를 drain하며, 그 뒤 DevGuard에 활성 `codespace` 인스턴스도 차지한 자원도 없어야 합니다. 소비자는 인스턴스 하나만 허용하므로 살아 있는 Runner가 있으면 CS-RG 스위트의 gateway가 등록하지 못합니다. +5. **CS-RG 스위트** `candidate`, `matrix`, `faults`, `authority`를 같은 후보로 실행합니다. +6. **확인합니다.** 봉인한 디렉터리마다 `verify`를 실행합니다. + + + +## 회귀 + +5단계의 CS-RG 스위트는 qualification을 마친 모드가 그 후보에서도 qualification대로 동작함을 보여 주며, 그중 하나라도 실패하면 qualification은 실패합니다. 후보 commit의 CI가 독립 Runner 시험을 포함한 Linux와 macOS의 회귀 단계를 보여 줍니다. + + + +## 증거 + +모든 스위트는 CS-RG처럼 새 디렉터리를 만들어 쓰고 봉인합니다([증거](cs-rg-qualification.md#evidence)). recovery 디렉터리에는 Runner마다 argv와 로그, gateway마다 argv, 로그, MCP 대화 기록이 들어가고, 측정이면 구간마다 원시 행이 들어갑니다. 표본, 모든 주기(`restarts.jsonl`), timeout 대상, 승인, 태그를 붙인 모든 명령(`execs.jsonl`), 출력 읽기, 그리고 각 workspace의 명령이 실행마다 한 줄씩 쓴 marker 파일입니다. 어떤 디렉터리에도 credential, credential digest, 자식 프로세스의 환경은 들어가지 않습니다. + +`qualify-devguard.py verify <디렉터리>`는 manifest를 확인하고 원시 파일에서 모든 판정을 다시 계산합니다. 시나리오 판정은 그 확인 항목에서, recovery 구간의 지표와 판정은 그 행에서, 그다음 반복, 조합, 실행 전체 순서로 계산합니다. 아무것도 쓰지 않습니다. diff --git a/docs/ko/runner-isolation.md b/docs/ko/runner-isolation.md index 54f4e3c..af6690a 100644 --- a/docs/ko/runner-isolation.md +++ b/docs/ko/runner-isolation.md @@ -23,7 +23,7 @@ ## 독립 Runner -> **아직 지원하는 모드가 아닙니다.** 독립 Runner와 그 실행의 기록, 한 게이트웨이에서 다음 게이트웨이로 제어권을 넘기는 Claim, 그리고 Claim마다 뒤따르는 대조는 게이트웨이 복구의 첫 단계입니다. 게이트웨이는 연결을 잃으면 Runner를 다시 Claim하고, 이전 controller가 작업을 시작한 곳에서는 그 controller가 남긴 것을 대조한 뒤에야 새 작업을 시작합니다. Runner 자체를 잃으면 게이트웨이는 그 실행들에 대해 기록으로 답하고, 운영자가 풀 때까지 그 workspace를 점유한 채로 둡니다([Runner 손실](#runner-loss)). 이 모드는 qualification을 마친 뒤에야 지원합니다. +> **qualification을 통과한 곳에서만 지원합니다.** 독립 Runner와 그 실행의 기록, 한 게이트웨이에서 다음 게이트웨이로 제어권을 넘기는 Claim, 그리고 Claim마다 뒤따르는 대조가 게이트웨이 복구를 이룹니다. 게이트웨이는 연결을 잃으면 Runner를 다시 Claim하고, 이전 controller가 작업을 시작한 곳에서는 그 controller가 남긴 것을 대조한 뒤에야 새 작업을 시작합니다. Runner 자체를 잃으면 게이트웨이는 그 실행들에 대해 기록으로 답하고, 운영자가 풀 때까지 그 workspace를 점유한 채로 둡니다([Runner 손실](#runner-loss)). 이 모드는 [qualification](recovery-qualification.md)이 통과시킨 조합에서만 지원하며, 아직 통과한 조합은 없습니다. 도입하고 되돌리는 방법은 [운영](operations.md#independent-runner-adoption)에 있습니다. 독립 Runner는 자신에게 연결하는 게이트웨이보다 오래 삽니다. 게이트웨이가 종료되거나 재시작되거나 강제로 종료되어도 프로세스와 그 출력, timeout은 계속됩니다. 운영자나 하네스가 독립 Runner를 시작하고, 게이트웨이는 연결만 합니다. diff --git a/docs/ko/security-model.md b/docs/ko/security-model.md index 81b90ae..0579b0d 100644 --- a/docs/ko/security-model.md +++ b/docs/ko/security-model.md @@ -20,7 +20,7 @@ Runner는 허용된 요청을 실행합니다. `PathSandbox`는 논리적인 파 선택적 UDS worker는 같은 호스트에서 프로세스를 분리합니다. 소켓을 전용 디렉터리에 두지만 프로세스 분리 자체가 명령 샌드박스는 아닙니다. Linux 도우미의 사용 가능 검사가 성공하면 명령 샌드박스를 사용합니다. 조건과 실패 시 동작은 [러너 격리](runner-isolation.md)에 설명합니다. -아직 지원하는 모드가 아닌 [독립 Runner](runner-isolation.md#claim)는 같은 사용자의 연결을 모두 받고, Claim한 연결 하나에만 한 번에 제어를 맡깁니다. 게이트웨이가 Runner를 제어하지 않는 동안에는 소켓에 닿을 수 있는 그 사용자의 프로세스라면 무엇이든 Runner를 Claim할 수 있으며, Linux 샌드박스 없이 실행한 명령도 여기에 포함됩니다. [누가 제어권을 얻을 수 있는가](runner-isolation.md#claim-exposure)를 참고하세요. +[qualification](recovery-qualification.md)을 통과한 곳에서만 지원하는 [독립 Runner](runner-isolation.md#claim)는 같은 사용자의 연결을 모두 받고, Claim한 연결 하나에만 한 번에 제어를 맡깁니다. 게이트웨이가 Runner를 제어하지 않는 동안에는 소켓에 닿을 수 있는 그 사용자의 프로세스라면 무엇이든 Runner를 Claim할 수 있으며, Linux 샌드박스 없이 실행한 명령도 여기에 포함됩니다. [누가 제어권을 얻을 수 있는가](runner-isolation.md#claim-exposure)를 참고하세요. diff --git a/docs/operations.md b/docs/operations.md index f0bb7fa..6d0396d 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -109,7 +109,7 @@ export CODESPACE_RUNNER=uds export CODESPACE_RUNTIME_BIN="$PWD/dist/codespace-codex-runtime" ``` -The worker runs on the same host and is not a container. The gateway creates a private socket directory and owns the child. Worker connection loss or gateway shutdown ends that worker's processes; reconnect and process recovery are not supported. MCP/HTTP client disconnect does not kill those processes. The default remains `in-process`. An [independent Runner](runner-isolation.md#independent-runner), whose processes outlive the gateway, whose gateway records them in the operations database, and which a gateway claims again after a lost connection and then reconciles with its records, is not a supported mode yet; when such a Runner itself is lost, an operator settles what it ran with `codespace-mcp records` ([Runner loss](#runner-loss)). +The worker runs on the same host and is not a container. The gateway creates a private socket directory and owns the child. Worker connection loss or gateway shutdown ends that worker's processes; reconnect and process recovery are not supported. MCP/HTTP client disconnect does not kill those processes. The default remains `in-process`. An [independent Runner](runner-isolation.md#independent-runner), whose processes outlive the gateway, whose gateway records them in the operations database, and which a gateway claims again after a lost connection and then reconciles with its records, is supported only on the combinations that its [qualification](recovery-qualification.md) qualifies, and none is qualified yet ([adoption and rollback](#independent-runner-adoption)); when such a Runner itself is lost, an operator settles what it ran with `codespace-mcp records` ([Runner loss](#runner-loss)). @@ -133,11 +133,42 @@ Store logs and the database outside the managed workspace, with tokens and gatew After losing a patch response, query `operation_status` with exactly one of `operation_id` or `operation_key`. An unfinished record becomes `unknown` after restart; inspect files before deciding what to do. Processes use `process_id` and cannot be recovered through `operation_status`. A confirmation hold that was `queued` when the process died can be resumed and will reacquire. A hold that was `resuming` may recover a patch from that ledger, replay a stored terminal result, or return `APPROVAL_AMBIGUOUS`; exec is not respawned. A gateway with an independent Runner settles an exec hold left `resuming` when it [reconciles](runner-isolation.md#reconciliation): consumed with the process that the Runner created for it, or granted again when the Runner proves that it created none. See [retry and recovery rules](agent-integration.md). + + +## Independent Runner adoption and rollback + +The [independent Runner](runner-isolation.md#independent-runner) is supported only on the combinations that its [qualification](recovery-qualification.md) qualifies, and none is qualified yet. It is opt-in, and the default stays `in-process`. Adopt it only on a qualified combination, with the CodeSpace build, the DevGuard release and the policy that were qualified: anything else is a new candidate. + +1. **Prepare** a state directory outside every workspace, with an operations database that no CodeSpace version from before the process records will open ([schema version](#operations-database)), and a socket directory of mode 0700 for the Runner. With DevGuard, the Runner is the execution owner and registers as the `codespace` consumer's one instance, so stop any other gateway that registers as that consumer first. +2. **Start the Runner** in its own session, as the qualification starts it, so that neither your terminal nor any gateway's end reaches it. On Linux `setsid` does this; the Python line below does it on macOS and Linux alike. With DevGuard, give it the consumer settings of the [status-only connection](devguard-integration.md#devguard-status-connection) and the release's `devguard-launch`; without DevGuard, leave out the `--devguard-*` settings, and `required` workspaces are refused. + + ```bash + python3 -c 'import os, sys; os.setsid(); os.execv(sys.argv[1], sys.argv[1:])' \ + "$PWD/dist/codespace-codex-runtime" "$RUNNER_DIR/runner.sock" --lifecycle independent \ + --devguard-socket "$DEVGUARD_SOCKET" --devguard-consumer codespace \ + --devguard-generation "$DEVGUARD_GENERATION" \ + --devguard-credential-file "$DEVGUARD_CREDENTIAL_FILE" \ + --devguard-launch-helper "$DEVGUARD_LAUNCH" >>"$STATE_DIR/runner.log" 2>&1 ## Runner loss -This applies to a gateway with an [independent Runner](runner-isolation.md#independent-runner), which is not a supported mode yet. When that Runner is killed, crashes or stops, or the host restarts, what it was running can no longer be learned from it, and its processes may outlive it ([Runner loss](runner-isolation.md#runner-loss)). The gateway answers for them from their records, as `unknown` with a `lost_reason`, and each workspace they held stays refused, with `RECOVERY_PENDING` in the next gateway, until you confirm that nothing of them runs and release their records. CodeSpace ends none of these processes, starts nothing again, and asks DevGuard nothing about them. +This applies to a gateway with an [independent Runner](runner-isolation.md#independent-runner), which is supported only where [qualified](recovery-qualification.md). When that Runner is killed, crashes or stops, or the host restarts, what it was running can no longer be learned from it, and its processes may outlive it ([Runner loss](runner-isolation.md#runner-loss)). The gateway answers for them from their records, as `unknown` with a `lost_reason`, and each workspace they held stays refused, with `RECOVERY_PENDING` in the next gateway, until you confirm that nothing of them runs and release their records. CodeSpace ends none of these processes, starts nothing again, and asks DevGuard nothing about them. 1. **List the records** that may still hold a workspace. This only reads the database, so it also works while a gateway runs: diff --git a/docs/recovery-qualification.md b/docs/recovery-qualification.md new file mode 100644 index 0000000..193a7b8 --- /dev/null +++ b/docs/recovery-qualification.md @@ -0,0 +1,224 @@ + + +# Gateway recovery qualification protocol + +[English](recovery-qualification.md) | [한국어](ko/recovery-qualification.md) + +This page is the protocol for qualifying the [independent Runner](runner-isolation.md#independent-runner), the opt-in mode of Gateway recovery (P1-RECOVERY): which combinations are claimed, how each claim is tested and measured, what makes a run pass, and what evidence a run leaves. It follows the [CS-RG qualification protocol](cs-rg-qualification.md), whose candidate, evidence and verification it reuses, and its harness is the same [`scripts/qualify-devguard.py`](../scripts/qualify-devguard.py). It is frozen together with the harness before any qualification run; a change to either after the first run is a new protocol, reviewed like any other change, and the runs it affects are repeated. The [integration roadmap](devguard-integration.md#resource-gateway-recovery) describes what each unit built; this page decides whether the result qualifies. + +**Status:** frozen with its harness before the first qualification run. No combination is qualified yet, so the independent Runner is not a supported mode anywhere. + + + +## The candidate + +A run qualifies one exact combination, recorded by the harness's `candidate` suite as for CS-RG ([the candidate](cs-rg-qualification.md#candidate)): the CodeSpace commit and its code tree, the DevGuard client pin, the Codex pin with its recorded patch, the binaries, the installed DevGuard release and its policy, and the toolchain and host. The independent Runner is the same `codespace-codex-runtime` binary as the `uds` worker. The run uses the installed DevGuard release and the policy as they are: it changes neither, installs nothing and provisions nothing. Any change to code, binaries, the release, the policy, Codex or the toolchain is a new candidate and repeats the runs it affects; a commit that changes only documentation keeps it. + + + +## Supported modes + +Each claimed cell is tested on its own; no cell is inferred from another. The modes that CS-RG qualified keep their claims, and the run shows that they still hold ([regressions](#recovery-regressions)). + +| Platform | Runner | I/O | Participation | Claim | Shown by | +| --- | --- | --- | --- | --- | --- | +| macOS, installed service with native host evidence | `uds` with `--runner-lifecycle independent`, an independent Runner the operator starts | pipe, PTY | `off` and `required` | qualify | the recovery suites with both participations and the gateway restart measurement | +| Linux | the same | pipe, PTY | `off` | CI functional tests only | CodeSpace's tests of the independent Runner in Linux CI; the harness's `off` scenarios also run on Linux. This is functional verification, never a Linux qualification | +| Linux | the same | pipe, PTY | `required` | unsupported until DG-LINUX | refused before anything runs, as every Linux `required` execution is | +| macOS, Linux | the same | pipe, PTY | restoring processes after the Runner itself restarts | unsupported | outside P1-RECOVERY's scope; `runner-replaced` shows that a gateway never takes a new Runner for the one it lost | + +The independent mode is opt-in. Making it the default is a separate decision, outside this protocol. + + + +## Running the harness + +The harness runs on the qualification host as the CS-RG suites do ([running the harness](cs-rg-qualification.md#running)): the system Python, no third-party packages, and a minimal environment for itself and every process it starts. It starts each independent Runner itself, as an operator would: `codespace-codex-runtime --lifecycle independent` in its own session, with a private socket directory, its log kept as evidence, and for `required` the DevGuard settings of the authority it uses, the credential file given by its path, which the harness never reads. Nothing is installed as a LaunchAgent and no policy changes. Its gateways run with `--runner uds --runner-socket --runner-lifecycle independent`, an exit intent, an operations database and, for `required`, `--devguard register`. + +| Suite | What it shows | Extra arguments | +| --- | --- | --- | +| `recovery-lifecycle` | the Runner across every way a gateway ends, timeouts and settings ([lifecycle](#recovery-lifecycle)) | `--participation`; on macOS with `required`, `--fixture-bin` and `--fixture-helper` as for `faults` | +| `recovery-store` | process records and the operations database's one writer ([store](#recovery-store)) | as `recovery-lifecycle` | +| `recovery-fence` | Claims, controller epochs and other processes at the socket ([fence](#recovery-fence)) | as `recovery-lifecycle` | +| `recovery-reconcile` | reconciliation after a restart, its races and an authority outage ([reconcile](#recovery-reconcile)) | as `recovery-lifecycle` | +| `recovery-observation` | output gaps, retention and a lost Runner ([observation](#recovery-observation)) | as `recovery-lifecycle` | +| `recovery` | every suite above, then the [gateway restart measurement](#recovery-measurement) | as above, and on macOS with `required` the `control` suite's `--release`, `--work`, `--qualify-bin`, `--devguard-checkout`, `--load`, `--rehearsal` and `--repetitions` | + +`--participation` takes `off`, `required` or both, the default. `off` scenarios run on every platform without an authority. On macOS a `required` scenario runs against DevGuard's fixture authority, as the CS-RG fault cases do, each with a fresh one that provisions its own `codespace` consumer with one instance, as the installed policy does; its governed workspaces request 50 mCPU, 32 MiB and 4 tasks. On Linux `required` scenarios are recorded as not run, with the reason, and are not counted. A suite passes when every scenario that ran passed and at least one ran. Exit statuses are those of the CS-RG suites. + +```bash +env -i HOME="$HOME" TMPDIR="$TMPDIR" PATH="/bin:/usr/bin:/bin:/usr/sbin:/sbin" \ + /usr/bin/python3 scripts/qualify-devguard.py recovery --out evidence/ \ + --mcp-bin --runtime-bin \ + --fixture-bin --fixture-helper \ + --release --work --qualify-bin \ + --devguard-checkout +``` + + + +## Functional suites + +Each scenario gets its own work directory, operations database and Runner, and with `required` its own fixture authority; whatever a scenario leaves running is ended afterwards. A scenario passes only when each of its checks passed. Unless a scenario says otherwise, a gateway's process timeout is 10 minutes and its exit intent is `detach`. "Both" in the I/O column means that the scenario runs once with a pipe and once with a PTY, and the participation column lists the participations the scenario runs with. + + + +### Lifecycle + +| Scenario | I/O | Participation | Passes when | +| --- | --- | --- | --- | +| exit-kinds | both | `off`, `required` | a process runs while its gateway ends in every way a gateway ends: its input closed and SIGTERM with the exit intent `detach`, SIGUSR1 with `stop` (a restart always detaches) and SIGKILL, each followed by the next gateway. The Runner and the process outlive each of them; each next gateway claims the Runner under the next epoch, finds the same process running and reads its output on from where the gateway before stopped. Then a gateway with `stop` whose input closes ends the process, and the Runner removes its socket and exits 0 | +| original-timeout | both | `off`, `required` | two processes reach their timeouts across gateway restarts, the first while no gateway runs and the second while a later gateway with a much longer process timeout serves: each ends at the deadline its record held when it started, never at one that a restart moved or lengthened, and is reported `timeout` | +| runner-sigterm | | `off`, `required` | the Runner's own SIGTERM, as an operator stops it, ends its pipe and PTY processes as a controller's Stop does; it exits 0 and removes its socket, its gateway reports neither process running and refuses new work, and with an authority both attempts are released and the instance retires | +| invalid-settings | | `off` | settings that do not fit together are refused before anything reaches the Runner: an exit intent without the independent lifecycle, the lifecycle without an exit intent, in process, with a worker to start, without a socket or without an operations database, a managed gateway at an independent Runner and an independent gateway at a managed worker. A Runner refuses a socket directory that is not private, a shared one, no socket, an unknown lifecycle, a credential handed to it and a socket where another Runner listens. The listening Runner is left as it was: it logged no Claim, and a gateway then claims it under epoch 1 and runs a command | + + + +### Process records + +| Scenario | I/O | Participation | Passes when | +| --- | --- | --- | --- | +| records | both | `off`, `required` | the operations database holds, of an execution sent to the Runner and after a gateway restart, the Runner's identity as its Hello states it, the controller epoch, the workspace, the argv's digest and never the argv, the root process of a pipe execution (a PTY's is not named), the original deadline, and how the process ended once a later gateway terminated it | +| writer-lock | | `off` | a second gateway started on the database while the first serves waits a bounded time and refuses to start, naming the holder, and never reaches the Runner; one started while the first ends waits for it and then serves, claiming the Runner after it | +| schema | | `off` | a database of a newer schema version, and a file that is not a database, are refused before anything reaches the Runner and left unchanged | +| records-command | | `off` | `codespace-mcp records list` lists the records that may still hold a workspace, and `records release` releases one only while no gateway runs, only for an execution that has not ended, and not while its Runner's PID runs a process | + + + +### Claim and fence + +| Scenario | I/O | Participation | Passes when | +| --- | --- | --- | --- | +| foreign-connection | both | `off`, `required` | while a gateway controls the Runner, another process of the same user reads the Runner's Hello, its Claim is refused because a controller is connected, every request it sends that would change a process, under the controller's epoch or any other, is refused as stale, and its Detach changes nothing; the gateway goes on controlling the Runner and its processes | +| stale-epoch | | `off` | after two gateways detached in turn, a Claim for another operations database is refused; one for the Runner's own database, which any process of the same user can make while no gateway controls the Runner (the [exposure](runner-isolation.md#claim-exposure) that decision D6 accepts), is granted under the next epoch and logged with its PID, and that connection's requests under an earlier epoch or none are refused as stale. The next gateway claims the Runner under the epoch after it and finds the same process, its output on from where it was | +| stopped-gateway | | `off` | a controlling gateway stopped with SIGSTOP keeps its connection, so nothing pushes it out: another process's Claim is refused and a second gateway refuses to start for the database's writer. Once the stopped gateway is killed, the next one claims the Runner under epoch 2 and finds the process running | +| concurrent-claims | | `off` | eight connections claim the Runner at once for its own database while no gateway controls it: exactly one is granted and the others are refused because a controller is connected; once it detaches, the next gateway claims the Runner under the epoch after it | +| other-database | | `off` | a gateway with another operations database than the Runner's first Claim named is refused and starts nothing, whether or not a gateway controls the Runner | +| concurrent-gateways | | `off` | of two gateways started at once on the Runner's database, one serves and claims the Runner, and the other waits for the writer and refuses to start, never reaching the Runner | + + + +### Reconciliation + +| Scenario | I/O | Participation | Passes when | +| --- | --- | --- | --- | +| running | both | `off`, `required` | after a gateway is killed while its process runs, the next gateway refuses new work in that workspace with `RECOVERY_PENDING` until it has reconciled it, then holds the workspace for the process again (`WORKSPACE_BUSY`), while a workspace where no work was left takes new work at once; terminating the inherited process frees its workspace, and no refused command ran | +| terminal | both | `off`, `required` | a process that ends while no gateway runs: the next gateway learns how it ended from the Runner, records it and frees the workspace | +| approval | | `off`, `required` | an approval held and granted before a gateway was killed is resumed by the next gateway once, and resuming it again returns the same process; one whose resume was in flight when the gateway was killed, at moments from 0 to 160 ms after the resume was sent, is resumed by the next gateway with the process the Runner created for it, or started then when the Runner proved that it created none. Each command ran exactly once | +| authority-outage | | `required` | while the fixture authority holds its lock, a killed gateway is replaced: the inherited governed processes answer at once, their workspaces stay refused with `RECOVERY_PENDING` while the Runner cannot ask the authority about their attempts, and ungoverned work runs. Once the authority answers, the gateway reconciles them, holds each workspace for its process again, and terminating them releases their attempts | +| missing-record | | `off` | a record removed from the database while no gateway ran: the next gateway can still query the process the Runner holds, keeps its workspace refused with `RECOVERY_PENDING` for an operator rather than start anything there, and other workspaces take new work | +| kill-during-exec | both | `off`, `required` | gateways killed while `exec_command` is in flight, at moments from 0 to 160 ms: the next gateway reconciles what each left, and no command ran more than once | +| kill-during-terminate | both | `off`, `required` | gateways killed while `terminate_process` is in flight, at the same moments: the next gateway reports the process terminated, or still running and then terminates it; each process ends once, terminated, after running once, and its workspace takes new work afterwards | + + + +### Observation and Runner loss + +| Scenario | I/O | Participation | Passes when | +| --- | --- | --- | --- | +| ring-overflow | both | `off` | a process writes more than the Runner's 256 KiB output ring while no gateway runs: the next gateway reads on from where the last read stopped, says that output was lost before what it returns and why (`ring_overflow`), returns the newest output whole, and its status says so too | +| retention | | `off` | of more ended processes than the Runner keeps (64, then 64 more whose handles it released), `process_status` and `read_process` answer for the oldest from their records, how each ended with no output and `retention_expired`, also through the next gateway | +| runner-loss | both | `off`, `required` | the Runner killed with SIGKILL: its gateway answers from the records, the running process as `unknown` and the ended one as it ended, both with `runner_lost`, and refuses new work with `RUNNER_DISCONNECTED`. A pipe process that does not write outlives the killed Runner; `records list` shows its record, and `records release` refuses it while a gateway runs and while the process runs. Once the operator ended it (with an authority, once DevGuard released its attempt and retired the lost Runner's instance), the record is released, and a new Runner and gateway take new work in that workspace. A PTY's root process is not named, so the operator ends it before releasing | +| runner-replaced | | `off` | another Runner answers at the socket after the first was killed: the gateway does not take it for the old one, reports the old Runner's process `unknown` with `runner_lost` and refuses new work; the next gateway keeps that workspace refused until an operator releases the record, and takes new work elsewhere | + + + +### Gaps + +These are recorded in every run as scenarios not run, with the reason, or stated here; none counts as passed. + +| Gap | Why, and what shows the behavior instead | +| --- | --- | +| A host restart | it cannot be made during a run. CodeSpace's tests give a record another boot ID ([`crates/server/src/mcp.rs`](../crates/server/src/mcp.rs)), so a restart is reported with `host_rebooted`; a real restart stays unqualified | +| Restoring processes after the Runner itself restarts | unsupported and outside P1-RECOVERY's scope; `runner-replaced` shows the refusal | +| Another process claiming the Runner while no gateway controls it | possible by design for any process of the Runner's user (decision D6 accepts this exposure); `stale-epoch` shows that it is logged and fenced, not prevented | +| Output beyond the 256 KiB ring while no gateway reads | lost by design; `ring-overflow` and the measurement show that the loss is declared, never silent | +| A PTY process after the Runner's loss | its root is not named in its record, so `records release` cannot tell whether it runs; the operator ends it first | +| A replay of a preparation or launch message on the runner wire | not injected, as in CS-RG; epoch fencing refuses one from an earlier controller (`foreign-connection`, `stale-epoch`) | +| Linux `required` | unsupported until DG-LINUX | + + + +## Gateway restart measurement + +The measurement restarts the gateway again and again while governed pipe and PTY processes run for a whole interval, idle and under DG-1's development load, against the installed service, and checks after every restart what the [completion evidence](#recovery-completion) asks for. It runs governed on macOS with `required`; anywhere else the same run is a self-test of the harness, named `independent-off`, which never qualifies. + + + +### Intervals, cycles and samples + +A repetition is a 10-minute idle interval followed by a load interval of at least 30 minutes; the protocol has three repetitions. Each interval starts a fresh Runner, which registers as the `codespace` consumer's one instance (decision D12), and its first gateway starts the processes that run the whole interval: a `cat` (pipe) and a `sleep` (PTY) in governed workspaces, which request 25 mCPU, 16 MiB and 2 tasks each, and in ungoverned ones a PTY that writes a numbered line every 0.2 seconds and a pipe that writes 40,000 numbered lines at once, more than the output ring holds, whenever the harness cues it. Through whichever gateway serves, the control path is sampled on the `control` suite's schedules ([intervals and samples](cs-rg-qualification.md#control-intervals)): status every second, a fresh governed termination every 20 seconds, the I/O echo every 5 seconds, a governed refusal every 30 seconds, and the host, service and footprint (the Runner's and the gateway's). Both output streams are read every 2 seconds. A slot that falls while no gateway serves is recorded as `restarting`, not as a sample, and the samples expected follow the time a gateway served. + +The interval is a sequence of gateway cycles. The first lasts 20 seconds; every later gateway starts its executions with a 20-second process timeout, and its cycle is timed by that cycle's timeout target. In each cycle: + +1. The serving gateway claimed the Runner under the cycle's epoch, and from the second cycle on it finds every long-running process `running`. +2. Another process of the same user connects to the socket: its Claim is refused, and its requests to write, resize, terminate a process, terminate a workspace's processes and stop the Runner, under the previous, the current and no epoch, are each refused `STALE_CONTROLLER`. +3. A rival gateway starts on the same operations database. It must wait for the writer and give up, never reaching the Runner; one still waiting when the cycle ends is killed. +4. The approval granted before the restart is resumed once the gateway has reconciled its workspace, and resumed again, which must return the same process; its command runs once. +5. Each timeout target whose deadline has passed is judged: when its process was first seen gone against the deadline recorded when it started, and how the serving gateway reports it. +6. Unless the interval ends, the gateway starts the cycle's timeout target (a `sleep`, governed or not, pipe or PTY, in turn), holds an approval and grants it 2 seconds before the restart, and is ended: in turn with its input closed, SIGTERM, SIGUSR1 or SIGKILL, all with the exit intent `detach`; before a SIGKILL the approval's resume is sent and left in flight for 0 to 160 ms. In blocks of four cycles the gateway ends 2 seconds before the target's deadline and the next one starts 4 seconds later, so the deadline falls while no gateway runs, or it ends 3 seconds after the deadline and the next one starts at once. In the fourth cycle and about every ninth after it, the pipe writes its burst while no gateway reads it. + +At the interval's end the last gateway reads both streams once more, resumes the last approval, terminates every long-running process, which must be effective, and ends with its input closed. Then the Runner is drained as an operator drains it ([adoption and rollback](operations.md#independent-runner-adoption)): a gateway with the exit intent `stop` claims it under the next epoch and stops it, and the Runner exits 0, removes its socket, and terminates nothing, since the harness ended its processes first; with the service, its instance retires and nothing of the consumer stays charged. The load is the `control` suite's ([load](cs-rg-qualification.md#control-load)), cold by default. The display and the system are kept awake for the run. + + + +### Verdicts + +An interval is **inconclusive**, an invalid observation, when the Runner did not register as the consumer's instance, a long-running process could not be started, other work was charged or a `codespace` instance was active as it began, the journal could not be read, raw evidence could not be written, the service could not be observed or had another release selected, or the load did not finish within its limit. + +Otherwise it **fails** when the control path fails by the `control` suite's [rules](cs-rg-qualification.md#control-verdicts) over the time a gateway served, or when: + +- a gateway ended otherwise than asked, a gateway claimed the Runner under another epoch than the next, or a long-running process was not `running` for the next gateway; +- any request of the other process changed something (a stale mutation), was refused other than as stale, or its Claim was granted; +- a rival gateway reached the Runner, served, or ended other than for the writer; +- a timeout target outlived its original deadline by more than 2 seconds or its recorded deadline moved (an extension), ended more than 0.25 seconds before it, or was not reported `timeout`; +- an approval was not resumed, a second resume returned another process, or its command ran twice or never; +- a tagged command ran twice, or one that was refused ran; +- an output stream skipped or repeated a line where the Runner declared no gap; +- the drain failed, or the last gateway did not exit 0 when its input closed; +- a cycle broke with an error. + +It is also **inconclusive** when too little was observed: fewer restarts than 80% of one every 25 seconds, no deadline that fell while no gateway ran, no SIGKILL with a resume in flight, nothing read from an output stream, a timeout target whose end was not observed, the `control` suite's shortfalls, or a load that was not DG-1's. Otherwise it **passes**. + +A repetition, the combination and the run are judged as in `control`: the combination qualifies when all three repetitions pass, and the run qualifies only under the full protocol (three repetitions of 10 and 30 minutes, governed). `--rehearsal` (2 minutes idle, 3 minutes load, one repetition) tries the run and never qualifies. The `recovery` suite qualifies only when its functional scenarios passed and its measurement qualified. + + + +## Completion evidence + +| What P1-RECOVERY must show | Where it is shown | +| --- | --- | +| The same execution is reached again after a restart | every cycle's `running` statuses and output streams through every gateway; lifecycle `exit-kinds`; fence `stale-epoch` | +| No execution runs twice | the measurement's approvals and tagged commands, and with the service the launches that the gateways logged for each execution; reconcile `approval`, `kill-during-exec`, `kill-during-terminate` | +| No stale mutation | every cycle's fence probe and rival; fence suite | +| No timeout extension | the measurement's timeout targets; lifecycle `original-timeout` | +| Explicit gaps | [gaps](#recovery-gaps), recorded in every run | +| Existing modes still pass | [regressions](#recovery-regressions) | +| Supported modes | [the table](#recovery-supported-modes); Linux `off` is CI functional verification only | + + + +## The qualification run + +On the qualification host, on the merged main that holds this harness: + +1. **Before freezing**, the Mac checkout is fetched with `--prune` and fast-forwarded to main, memory pressure is normal, and the installed DevGuard release and policy are the ones CS-RG qualified, unchanged. +2. **Freeze the candidate** with `candidate`, which must pass. +3. **Run `recovery`** with both participations, the functional suites and then the measurement in three repetitions. +4. **Drain.** Each interval drains its Runner; afterwards DevGuard must show no active `codespace` instance and nothing charged, since the consumer allows one instance and a live Runner would keep the CS-RG suites' gateways from registering. +5. **Run the CS-RG suites** `candidate`, `matrix`, `faults` and `authority` on the same candidate. +6. **Verify** every sealed directory with `verify`. + + + +## Regressions + +The CS-RG suites in step 5 show that the qualified modes still behave as qualified on the candidate, and a failure in any of them fails the qualification. CI on the candidate's commit shows the regression stages on Linux and macOS, the independent Runner's tests included. + + + +## Evidence + +Every suite writes a new directory and seals it as for CS-RG ([evidence](cs-rg-qualification.md#evidence)). A recovery directory also holds each Runner's argv and log, each gateway's argv, log and MCP transcript, and for the measurement each interval's raw rows: the samples, every cycle (`restarts.jsonl`), the timeout targets, the approvals, every tagged command (`execs.jsonl`), the output reads, and the marker files each workspace's commands wrote, one line per run. Nothing in a directory holds a credential, a credential digest or a child's environment. + +`qualify-devguard.py verify ` checks the manifest and recomputes every verdict from the raw files: each scenario's verdict from its checks, each recovery interval's metrics and verdict from its rows, then each repetition, the combination and the run. It writes nothing. diff --git a/docs/runner-isolation.md b/docs/runner-isolation.md index 334ba0c..cb7c7d5 100644 --- a/docs/runner-isolation.md +++ b/docs/runner-isolation.md @@ -20,7 +20,7 @@ Managed process lifetime is owned by the runner instance, not the MCP connection ## Independent Runner -> **Not a supported mode yet.** The independent Runner, the records of its executions, the Claim that hands its control from one gateway to the next and the reconciliation after each Claim are the first steps of Gateway recovery. A gateway claims the Runner again when its connection is lost, and starts new work where an earlier controller started work only once it has reconciled what that controller left. When the Runner itself is lost, the gateway answers for its executions from their records and keeps their workspaces until an operator releases them ([Runner loss](#runner-loss)). The mode is supported only once it is qualified. +> **Supported only where qualified.** The independent Runner, the records of its executions, the Claim that hands its control from one gateway to the next and the reconciliation after each Claim make up Gateway recovery. A gateway claims the Runner again when its connection is lost, and starts new work where an earlier controller started work only once it has reconciled what that controller left. When the Runner itself is lost, the gateway answers for its executions from their records and keeps their workspaces until an operator releases them ([Runner loss](#runner-loss)). The mode is supported only on the combinations that its [qualification](recovery-qualification.md) qualifies, and none is qualified yet. [Operations](operations.md#independent-runner-adoption) describes how to adopt it and roll it back. An independent Runner outlives the gateways that connect to it: its processes, their output and their timeouts carry on when a gateway exits, restarts or is killed. An operator or a harness starts it, and a gateway only connects to it: diff --git a/docs/security-model.md b/docs/security-model.md index 6dafbe0..0af2945 100644 --- a/docs/security-model.md +++ b/docs/security-model.md @@ -15,7 +15,7 @@ The Runner executes approved requests. `PathSandbox` checks logical file scope; The optional UDS worker separates processes on the same host. Its socket lives in a private directory, but process separation is not an OS command sandbox. The Linux helper supplies the command sandbox when its probe succeeds. [Runner isolation](runner-isolation.md) defines those conditions and failure behavior. -An [independent Runner](runner-isolation.md#claim), not a supported mode yet, serves every connection of its own user and lets one at a time control it after a Claim. While no gateway controls it, any process of that user that can reach its socket can claim it, a command run without the Linux sandbox included; see [who can take control](runner-isolation.md#claim-exposure). +An [independent Runner](runner-isolation.md#claim), supported only where [qualified](recovery-qualification.md), serves every connection of its own user and lets one at a time control it after a Claim. While no gateway controls it, any process of that user that can reach its socket can claim it, a command run without the Linux sandbox included; see [who can take control](runner-isolation.md#claim-exposure). diff --git a/docs/translations.json b/docs/translations.json index 2384194..76f540e 100644 --- a/docs/translations.json +++ b/docs/translations.json @@ -78,8 +78,8 @@ "최소-도입-조건", "최초-gateway-복구-범위" ], - "source_sha256": "77bada859adefafb165febb3273f179c6135fcb5b6f757396dd639eb22c3ca0f", - "translation_sha256": "319176abbe5b1e8029493d5e3a637e712287a40c518068c6cd0e8b5e38f7b40b" + "source_sha256": "030dcad13f96bb7f9d9ae48e93cf9381b3774e61322218770fb353277ad3c1c6", + "translation_sha256": "e924382d4b0571900d5ef8b1c562b4445031d2d785ca25eeae3224b1a03569a5" }, { "id": "cs-rg-qualification", @@ -186,6 +186,72 @@ "source_sha256": "8b05c00a7ecd9d0960e1bfb69bf2c6fcb9d22686392c0b4264e2ca2d41b727a7", "translation_sha256": "c344d8142f13a2acc3e9c3e3e511423b3da0b8da399418679302696ab3c62a59" }, + { + "id": "recovery-qualification", + "section": "project", + "order": 4, + "route": "/guide/recovery-qualification", + "source": "docs/recovery-qualification.md", + "translation": "docs/ko/recovery-qualification.md", + "anchors": [ + "claim-and-fence", + "claim과-fence", + "completion-evidence", + "evidence", + "functional-suites", + "gaps", + "gateway-recovery-qualification-protocol", + "gateway-restart-measurement", + "gateway-복구-qualification-절차", + "gateway-재시작-측정", + "intervals-cycles-and-samples", + "lifecycle", + "observation-and-runner-loss", + "process-records", + "qualification-실행", + "reconciliation", + "recovery-candidate", + "recovery-completion", + "recovery-evidence", + "recovery-fence", + "recovery-functional", + "recovery-gaps", + "recovery-intervals", + "recovery-lifecycle", + "recovery-measurement", + "recovery-measurement-verdicts", + "recovery-observation", + "recovery-qualification", + "recovery-reconcile", + "recovery-regressions", + "recovery-run", + "recovery-running", + "recovery-store", + "recovery-supported-modes", + "regressions", + "running-the-harness", + "supported-modes", + "the-candidate", + "the-qualification-run", + "verdicts", + "공백", + "관찰과-runner-손실", + "구간-주기-표본", + "기능-스위트", + "대조", + "생명주기", + "실행-기록", + "완료-증거", + "증거", + "지원-모드", + "판정", + "하네스-실행", + "회귀", + "후보-조합" + ], + "source_sha256": "e49e67542a40b866f58e973546541fd3abf5790416db9726fab9e884c586f43f", + "translation_sha256": "01ccea2d97e827ea9d210909ff903b68c5e2bd5d73704a130293e3180ec7632f" + }, { "id": "overview", "section": "start", @@ -257,8 +323,8 @@ "재시도와-복구", "파일-읽기와-패치" ], - "source_sha256": "a3a4705eb2a425b423aa936b0c3f9b448fbb217f58ec4bbbb8ca7031333ff8d2", - "translation_sha256": "6cb63bc559a6b7e9f6fc9c1fa425113c67a08970faef1f84eb98094ef41e6b10" + "source_sha256": "dc29caa33ca71dad3eb6b2319c84ea12fa567596e601ab2c781306d8490b4c6c", + "translation_sha256": "3b38428e8b09c3983ae58100d5a8e4f1d7d2b8110091a56965493cfc88d18944" }, { "id": "operations", @@ -268,6 +334,8 @@ "source": "docs/operations.md", "translation": "docs/ko/operations.md", "anchors": [ + "independent-runner-adoption", + "independent-runner-adoption-and-rollback", "install", "installation-and-operations", "linux-isolation-fixture", @@ -298,6 +366,7 @@ "워크스페이스-레지스트리", "이-문서가-검증하지-않는-것", "게이트웨이-실행", + "독립-runner-도입과-되돌리기", "로그", "로그와-제한-복구", "문제-해결", @@ -312,8 +381,8 @@ "작업-공간-등록", "첫-연결-확인" ], - "source_sha256": "5950758a72f81664dce25afd7c281c9201fab11fa4ae4d78461bdaeb00ce978b", - "translation_sha256": "09ca3ed836f6b34f0528e9f750882af013389b47a50047cae4bd512a64250355" + "source_sha256": "da04d1227b65a31be67e5b841d42934ed410a9ab775033e1eed7d4c87264ddb5", + "translation_sha256": "dbd583fe19271b9db950726cbc1363d789c186405988539b3cdd1aa48b8d7871" }, { "id": "chatgpt-connector", @@ -402,8 +471,8 @@ "현재-실행-구조", "확장-시-유지할-경계" ], - "source_sha256": "686f0cd9906db317058fa6128080362e74da25bbdd8863095ac849a566d99bb8", - "translation_sha256": "02d127df0e182eab1c9c504194143ad9f282162ffed7ca3a8cf53cc3d3d75d6b" + "source_sha256": "2dfac81dda34cc09cc6b1b572bc2bd74a2222566efd16c2f31c169fe43095cb1", + "translation_sha256": "41953807f5d3dfcc275ad0afef6a7819731cba2a79734791d81af3341451c2a1" }, { "id": "execution-substrate", @@ -578,8 +647,8 @@ "프로세스-정직성", "프로필-mvp" ], - "source_sha256": "c9ca06af27ed21c9cce3be7528c9d5aa5cdbe03db171dc02ae24880ffc755859", - "translation_sha256": "80cd04a917bbbaf09f0c9f3191fa16b6ef79c1222cf65428a5d0801c5435e70f" + "source_sha256": "109db0ce732d4bcf67812424b65ca36d6dfb569f0cd3292ebb9b3539c6652738", + "translation_sha256": "2d5dbd91c260e88108497823307b7e457f5f57a3c175fafb9adfa3b8ab3b863a" }, { "id": "runner-isolation", @@ -616,8 +685,8 @@ "컨테이너-실험-구성과-검증", "호스트와-worker-실행" ], - "source_sha256": "ada35d6e3907abe03287e595acf28acfda9ba546ede51d5d65ce4115db67268a", - "translation_sha256": "3336edb2d573475407bb95ff5b8220b3195836e42debb39f1feff0ae61c5a429" + "source_sha256": "9f8118e96dc863babaae5e629fa924ab4ce7f6942eb968d96dd766f24f8851c7", + "translation_sha256": "a99a0c9ee574e71a8eb8035c7541f3e1019cc2226e58a39a0080e1cc05113768" }, { "id": "error-codes", diff --git a/scripts/qualify-devguard.py b/scripts/qualify-devguard.py index 397aa82..ec1b65b 100644 --- a/scripts/qualify-devguard.py +++ b/scripts/qualify-devguard.py @@ -1,21 +1,34 @@ #!/usr/bin/env python3 -"""CSRG-U6 qualification of CodeSpace with DevGuard; docs/cs-rg-qualification.md is the protocol. - - candidate record the exact combination under test - matrix the claimed modes against the installed DevGuard service, each launch checked - against DevGuard's own record of its attempt - faults the cases that would leave the installed service with Suspect or charged work, - against DevGuard's fixture authority at CodeSpace's pin - control control-path latency under DG-1's development load, against the installed service - authority control-path latency while the fixture authority stalls, pauses, restarts and stops - verify check a sealed run and recompute its verdicts from its raw evidence +"""CSRG-U6 and P1R-C06 qualification of CodeSpace with DevGuard. + +docs/cs-rg-qualification.md and docs/recovery-qualification.md are the protocols. + + candidate record the exact combination under test + matrix the claimed modes against the installed DevGuard service, each launch + checked against DevGuard's own record of its attempt + faults the cases that would leave the installed service with Suspect or charged + work, against DevGuard's fixture authority at CodeSpace's pin + control control-path latency under DG-1's development load, against the + installed service + authority control-path latency while the fixture authority stalls, pauses, + restarts and stops + recovery-lifecycle the independent Runner across every way a gateway ends (P1R-C01) + recovery-store process records and the operations database's one writer (P1R-C02) + recovery-fence Claims, controller epochs and other processes at the socket (P1R-C03) + recovery-reconcile reconciliation after a restart: approvals, in-flight calls, an + authority outage, a lost record (P1R-C04) + recovery-observation output gaps, retention and a lost Runner (P1R-C05) + recovery every recovery suite, then gateway restarts measured over idle and + loaded intervals against the installed service + verify check a sealed run and recompute its verdicts from its raw evidence Every suite but verify writes a new evidence directory and seals it; verify writes nothing. Nothing here edits DevGuard's configuration, restarts its service or reads its credential: the -gateway is given that file's path and reads it itself, and the fault cases use DevGuard's fixture -authority and throwaway credentials in their own scratch directories. Exit status: 0 passed or -qualified, 1 failed, 2 incomplete or inconclusive, 3 the harness failed, 130 interrupted; verify -exits 0 when the sealed run checks out and 1 when it does not. +gateway, or the independent Runner the harness starts, is given that file's path and reads it +itself, and the fault cases use DevGuard's fixture authority and throwaway credentials in their +own scratch directories. Exit status: 0 passed or qualified, 1 failed, 2 incomplete or +inconclusive, 3 the harness failed, 130 interrupted; verify exits 0 when the sealed run checks out +and 1 when it does not. """ import argparse import hashlib @@ -29,6 +42,7 @@ import secrets import shutil import signal +import sqlite3 import subprocess import sys import tempfile @@ -64,6 +78,11 @@ {'platform': 'macos', 'runner': 'supported', 'io': ['pipe', 'pty'], 'participation': 'off and required in one gateway', 'claim': 'qualify'}, {'platform': 'macos, linux', 'runner': 'all', 'io': ['pipe', 'pty'], 'participation': 'off', 'claim': 'existing behavior'}, {'platform': 'linux', 'runner': 'all', 'io': ['pipe', 'pty'], 'participation': 'required', 'claim': 'unsupported until DG-LINUX'}, + # The independent Runner (P1-RECOVERY, docs/recovery-qualification.md). + {'platform': 'macos', 'runner': 'uds --runner-lifecycle independent', 'io': ['pipe', 'pty'], 'participation': 'off and required', 'claim': 'qualify'}, + {'platform': 'linux', 'runner': 'uds --runner-lifecycle independent', 'io': ['pipe', 'pty'], 'participation': 'off', 'claim': 'CI functional tests only'}, + {'platform': 'linux', 'runner': 'uds --runner-lifecycle independent', 'io': ['pipe', 'pty'], 'participation': 'required', 'claim': 'unsupported until DG-LINUX'}, + {'platform': 'macos, linux', 'runner': 'uds --runner-lifecycle independent', 'io': ['pipe', 'pty'], 'participation': "restoring processes after the Runner's own restart", 'claim': 'unsupported'}, ] @@ -71,11 +90,12 @@ class Live: """What a run started, settled however it ends: every process a gateway started is ended and - waited for before the gateway's input closes, then workers and fixture authorities stop, and - DG-1's load owners are asked to settle their scopes.""" + waited for before the gateway's input closes, then independent Runners stop, which ends what + they still hold, workers and fixture authorities stop, and DG-1's load owners are asked to + settle their scopes.""" def __init__(self): - self.gateways, self.fixtures, self.workers = [], [], [] + self.gateways, self.fixtures, self.workers, self.runners = [], [], [], [] self.measure = None self.reports = [] @@ -88,6 +108,9 @@ def settle(self): if gateway.process.poll() is None: left = gateway.settle(limit=20) self.reports.append({'gateway': gateway.pid, 'left_running': left, **gateway.close()}) + for runner in self.runners: + if runner.alive(): + self.reports.append({'runner': runner.pid, **runner.stop(limit=20)}) for worker in self.workers: if worker.poll() is None: worker.terminate() @@ -353,23 +376,34 @@ def teardown(self): shutil.rmtree(self.work, ignore_errors=True) -def run_scenario(results, evidence, mode, io, name, function, *args): - """Run one scenario and keep its checks; a scenario that breaks is a failed check.""" +def run_scenario(results, evidence, mode, io, name, function, *args, suite=None, participation=None, not_run=None): + """Run one scenario and keep its checks; a scenario that breaks is a failed check. One that + cannot run here is kept as not run, with `not_run` saying why.""" began = time.time() - try: - checks, detail = function(*args) - except Exception as error: # noqa: BLE001 - every broken scenario is evidence - checks = [check('the scenario completed', False, error=f'{type(error).__name__}: {error}', - trace=traceback.format_exc()[-3000:])] - detail = {} - entry = {'scenario': name, 'mode': mode, 'io': io, 'verdict': analysis.scenario_verdict(checks), + checks, detail = [], {} + if not not_run: + try: + checks, detail = function(*args) + except Exception as error: # noqa: BLE001 - every broken scenario is evidence + checks = [check('the scenario completed', False, error=f'{type(error).__name__}: {error}', + trace=traceback.format_exc()[-3000:])] + detail = {} + entry = {'scenario': name, 'mode': mode, 'io': io, 'verdict': analysis.scenario_verdict(checks, not_run), 'checks': checks, 'detail': detail, 'started_unix': began, 'ended_unix': time.time()} + for key, value in (('suite', suite), ('participation', participation), ('not_run', not_run)): + if value: + entry[key] = value results.append(entry) evidence.append('scenarios.jsonl', entry) - print(f"qualify: {mode} {io or '-'} {name}: {entry['verdict']}", flush=True) + print(f"qualify: {scenario_label(entry)}: {entry['verdict']}", flush=True) return entry +def scenario_label(entry): + return ' '.join(str(part) for part in (entry.get('suite'), entry['mode'], entry.get('participation'), + entry['io'] or '-', entry['scenario']) if part) + + def io_name(tty): return 'pty' if tty else 'pipe' @@ -2041,6 +2075,2680 @@ def control(args, live, evidence): 'plan': plan, 'combinations': combinations} +# ---------------------------------------------------------------- recovery (P1R-C06): the independent Runner +# docs/recovery-qualification.md is the protocol. The harness starts an independent Runner as an +# operator would, in its own session (D3), and drives gateways that connect to it, end in every +# way a gateway ends and connect again, all over one operations database. With `required` the +# Runner registers itself as the consumer's one instance (D12) and launches through the +# authority's helper; with `off` nothing reaches an authority. Each command the harness runs +# there is tagged: before it becomes the command, it appends its tag to its workspace's marker +# file and writes its PID beside it, so that every time it ran shows and its own end can be timed. + +RECOVERY_UNITS = ('recovery-lifecycle', 'recovery-store', 'recovery-fence', 'recovery-reconcile', + 'recovery-observation') +PARTICIPATIONS = ('off', 'required') +# The process timeout of a recovery gateway, unless a scenario gives its own. +RECOVERY_TIMEOUT_S = 600 +MARKERS = '.qualify-markers' +NOT_NATIVE = ('the fixture authority governs only with native host evidence (macOS); Linux `required` stays ' + 'unsupported until DG-LINUX') +NO_AUTHORITY = {'socket': None, 'credential_file': None, 'helper': None, 'consumer': host.CONSUMER, + 'generation': host.GENERATION, 'journal': None} +# A line every 0.2 s, numbered at a fixed width, so that a gap or a repeat in the output shows. +BEAT = ['/bin/sh', '-c', "i=0; while :; do i=$((i+1)); printf 'beat %08d\\n' \"$i\"; sleep 0.2; done"] +# More than the Runner's 256 KiB output ring holds, at once. +BURST_LINES = 40000 +# Commands a tagged process can be, for the cleanup of what a case left running. +TAGGED_COMMANDS = ('sleep', 'sh', 'cat', 'awk', 'bash', 'dash', 'zsh') +RUNNER_LINES = {'detached': 'the controlling gateway detached', 'lost': 'connection ended without a detach or a stop'} + + +def burst(cue, lines=BURST_LINES): + """Each time `cue` appears: remove it and write `lines` numbered lines at once, numbered on from + the burst before.""" + script = ('k=0; while :; do while [ ! -e "$1" ]; do sleep 0.05; done; rm -f "$1"; ' + f'awk -v k="$k" \'BEGIN {{ for (i = 1; i <= {lines}; i++) printf "burst %08d\\n", k * {lines} + i }}\'; ' + 'k=$((k+1)); done') + return ['/bin/sh', '-c', script, 'burst', str(cue)] + + +def tagged(root, tag, command): + """`command`, run so that it first appends `tag` to its workspace's marker file and writes its + PID beside it, then becomes `command` itself, keeping that PID.""" + script = f'echo "$0" >> "$1/{MARKERS}"; echo $$ > "$1/.qualify-pid-$0"; shift; exec "$@"' + return ['/bin/sh', '-c', script, tag, str(root), *[str(part) for part in command]] + + +def markers(root): + """The tags of a workspace's commands, in the order they started; a tag twice is a command + that ran twice.""" + try: + return [line.strip() for line in (Path(root) / MARKERS).read_text().splitlines() if line.strip()] + except OSError: + return [] + + +def tag_pid(root, tag, limit=10.0): + return host.read_pid(Path(root) / f'.qualify-pid-{tag}', limit) + + +def kill_leftovers(roots, since_unix): + """Kill the tagged processes a case left running: only a process whose PID a tagged command + wrote, that started after the case began and runs one of the commands the harness starts, so + that a PID used again by another process is never touched.""" + killed = [] + for root in roots: + for path in sorted(Path(root).glob('.qualify-pid-*')): + pid = host.read_pid(path, limit=0) + if not pid or not host.pid_alive(pid): + continue + facts = host.process_facts(pid) + if (not facts or facts['started_unix'] is None or facts['started_unix'] < since_unix - 2 + or Path(facts['command'] or '').name not in TAGGED_COMMANDS): + continue + try: + os.kill(pid, signal.SIGKILL) + killed.append(pid) + except ProcessLookupError: + pass + return killed + + +class RecoveryBench(Bench): + """An independent Runner at one socket, in a directory it creates and removes, and the gateways + that connect to it, all over one operations database. Without an authority (`NO_AUTHORITY`) + the Runner registers with nothing; with one, it registers from the credential file it is + given (its path only) and launches through the helper, and its gateways take part with + `--devguard register`.""" + + def __init__(self, args, live, evidence, writer, work, workspaces, authority, prefix, fixture=None): + super().__init__(args, live, evidence, writer, 'independent', work, workspaces, authority, prefix=prefix, + fixture=fixture) + self.socket = work / 'rv' / 'runner.sock' + self.operations = work / 'operations.sqlite' + self.governed = authority.get('socket') is not None + self.runners = [] + self.began_unix = time.time() + + def runner_flags(self): + if not self.governed: + return [] + authority = self.authority + return ['--devguard-socket', str(authority['socket']), '--devguard-consumer', authority['consumer'], + '--devguard-generation', authority['generation'], + '--devguard-credential-file', str(authority['credential_file']), + '--devguard-launch-helper', str(authority['helper'])] + + def runner(self, name='runner'): + index = len(self.runners) + 1 + runner = host.RunnerProcess(self.args.runtime_bin, self.socket, host.minimal_env(self.work), + self.evidence.file(f'{self.prefix}/runner-{index}-{name}.log'), + self.runner_flags()) + self.live.runners.append(runner) + self.runners.append(runner) + self.evidence.append(f'{self.prefix}/runners.jsonl', {'name': name, 'pid': runner.pid, 'argv': runner.argv, + 'listening': runner.listening, + 'started_unix': time.time()}) + if not runner.listening: + raise RuntimeError(f'the independent Runner did not listen: {runner.log()[-500:]}') + return runner + + def flags(self, intent='detach'): + flags = ['--runner', 'uds', '--runner-socket', str(self.socket), '--runner-lifecycle', 'independent'] + if intent: + flags += ['--runner-exit-intent', intent] + if self.governed: + authority = self.authority + flags += host.devguard_flags('register', authority['socket'], authority['credential_file'], None, + authority['consumer'], authority['generation']) + return flags + + def environment(self, timeout_s=None, database=None): + """A gateway's environment: the registry, the process timeout and the operations database + (`database` False: none).""" + env = {'CODESPACE_CONFIG': str(self.config), + 'CODESPACE_PROCESS_TIMEOUT_SECS': str(timeout_s or RECOVERY_TIMEOUT_S)} + if database is not False: + env['CODESPACE_OPERATIONS_DB'] = str(database or self.operations) + return host.minimal_env(self.work, env) + + def gateway(self, name, intent='detach', timeout_s=None, database=None, initialize=True): + self.count += 1 + prefix = f'{self.prefix}/{self.count:02d}-{name}' + transcript = self.evidence.file(prefix + '.mcp.jsonl') + gateway = host.Gateway(self.args.mcp_bin, self.flags(intent), self.environment(timeout_s, database), + self.evidence.file(prefix + '.gateway.log'), + record=lambda row: self.writer.write(transcript, row)) + self.live.gateways.append(gateway) + self.gateways.append(gateway) + self.evidence.append(f'{self.prefix}/gateways.jsonl', { + 'name': name, 'pid': gateway.pid, 'argv': gateway.argv, 'intent': intent, + 'timeout_s': timeout_s or RECOVERY_TIMEOUT_S, 'started_unix': time.time()}) + if initialize: + gateway.initialize() + return gateway + + def end(self, gateway, how, limit=60): + ended = host.end_gateway(gateway, how, limit) + self.evidence.append(f'{self.prefix}/gateways.jsonl', {'ended': gateway.pid, **ended}) + return ended + + def refused(self, name, flags=None, env=None, limit=90): + """A gateway that should not start: its exit status and what it logged. Its input is empty, + so one that does start detaches at once and exits 0.""" + argv = [str(self.args.mcp_bin), *(self.flags('detach') if flags is None else flags)] + began = time.monotonic() + try: + result = subprocess.run(argv, stdin=subprocess.DEVNULL, capture_output=True, text=True, timeout=limit, + env=env or self.environment(), start_new_session=True) + returncode, output = result.returncode, result.stderr + except subprocess.TimeoutExpired as error: + output = error.stderr.decode(errors='replace') if isinstance(error.stderr, bytes) else (error.stderr or '') + returncode = None + self.evidence.file(f'{self.prefix}/{name}.refused.log').write_text(output) + entry = {'name': name, 'argv': argv, 'returncode': returncode, 'elapsed_s': round(time.monotonic() - began, 3)} + self.evidence.append(f'{self.prefix}/refused.jsonl', entry) + return dict(entry, output=output) + + def teardown(self): + """End whatever a case left: gateways killed, Runners stopped, which ends what they hold, + and any tagged process a killed Runner left; then the fixture authority and the work + directory.""" + for gateway in self.gateways: + if gateway.process.poll() is None: + gateway.process.kill() + gateway.process.wait() + gateway.log_reader.join(timeout=10) + stopped = [runner.stop(limit=20) for runner in self.runners if runner.alive()] + killed = kill_leftovers(self.roots.values(), self.began_unix) + self.evidence.append(f'{self.prefix}/teardown.jsonl', {'runners_stopped': stopped, 'leftovers_killed': killed}) + if self.fixture is not None: + self.fixture.stop() + shutil.rmtree(self.work, ignore_errors=True) + + +def recovery_workspaces(work, participation): + """`a`, `b` and `c` of the bench's participation, `confirm` that holds each execution for + approval, and `off`, which never takes part.""" + resources = host.required(**FIXTURE_REQUEST) if participation == 'required' else None + spaces = {name: (work / f'ws/{name}', resources) for name in ('a', 'b', 'c')} + spaces['confirm'] = (work / 'ws/confirm', resources, {'approvals': 'confirm'}) + spaces['off'] = (work / 'ws/off', None) + return spaces + + +def recovery_bench(args, live, evidence, writer, participation, label, workspaces=recovery_workspaces): + """A fresh bench for one scenario: its own work directory, operations database and Runner + socket; with `required`, its own fixture authority with one instance per consumer (D12).""" + work = scratch() + prefix = f'{participation}/{label}' + if participation == 'off': + return RecoveryBench(args, live, evidence, writer, work, workspaces(work, participation), dict(NO_AUTHORITY), + prefix) + try: + fixture = host.FixtureAuthority(args.fixture_bin, work / 'fx', host.minimal_env(work), + evidence.file(f'{prefix}/fixture.log'), limit=1) + except Exception: + shutil.rmtree(work, ignore_errors=True) + raise + live.fixtures.append(fixture) + settings = fixture.settings + if not settings.get('native'): + fixture.stop() + shutil.rmtree(work, ignore_errors=True) + raise NotNative(NOT_NATIVE) + authority = {'socket': settings['socket'], 'credential_file': settings['credential_file'], + 'helper': args.fixture_helper, 'consumer': settings['consumer'], + 'generation': settings['generation'], 'journal': fixture.journal()} + return RecoveryBench(args, live, evidence, writer, work, workspaces(work, participation), authority, prefix, + fixture=fixture) + + +# -------------------------------- what the scenarios share + +def claimed_by(runner, pid, limit=10.0): + """The Claim the Runner logged granting control to process `pid`, once it is logged.""" + deadline = time.monotonic() + limit + while True: + found = [claim for claim in runner.claims() if claim['granted'] and claim['pid'] == pid] + if found or time.monotonic() >= deadline: + return found[-1] if found else None + time.sleep(0.05) + + +def runner_log_counts(runner): + text = runner.log() + return {key: text.count(needle) for key, needle in RUNNER_LINES.items()} + + +def instance_of(bench, pid): + """The consumer's instances in the authority's journal, and the active ones that are process `pid`.""" + instances = host.journal_instances(bench.authority['consumer'], bench.database) + return [entry for entry in instances if entry['state'] == 'active' and entry['pid'] == pid], instances + + +def wait_registered(bench, runner, limit=30.0): + deadline = time.monotonic() + limit + while True: + active, instances = instance_of(bench, runner.pid) + if active or time.monotonic() >= deadline: + return (active[0] if active else None), instances + time.sleep(0.1) + + +def wait_retired(bench, pid, limit=60.0): + deadline = time.monotonic() + limit + while True: + active, instances = instance_of(bench, pid) + if not active or time.monotonic() >= deadline: + return not active, instances + time.sleep(0.5) + + +def registered_checks(bench, runner): + """The Runner listens and, with an authority, is the consumer's one active instance.""" + checks = [check('the Runner listens', runner.listening, log=runner.log()[-500:])] + if bench.governed: + instance, instances = wait_registered(bench, runner) + active = [entry for entry in instances if entry['state'] == 'active'] + checks.append(check("the Runner is the consumer's one active instance", instance is not None and len(active) == 1, + instances=instances)) + return checks + + +def stopped_checks(bench, runner, label, terminated=None): + """The Runner stopped, by its controller's Stop or its own SIGTERM: it exited 0, said how many + processes it terminated, removed its socket and the directory it created for it, and with an + authority its instance retired.""" + code = runner.wait(30) + found = re.findall(r'stopped; (\d+) processes were terminated', runner.log()) + checks = [check(f'{label}: the Runner exited 0', code == 0, returncode=code), + check(f'{label}: it removed its socket and the directory it created', + not bench.socket.exists() and not bench.socket.parent.exists())] + if terminated is not None: + checks.append(check(f'{label}: it terminated {terminated} processes', + bool(found) and int(found[-1]) == terminated, logged=found)) + if bench.governed: + retired, instances = wait_retired(bench, runner.pid) + checks.append(check(f'{label}: its DevGuard instance retired', retired, instances=instances)) + return checks + + +def once_reconciled(call, limit=30.0, period=0.05): + """`call()` again while it is refused with RECOVERY_PENDING, for up to `limit` seconds: its + last result, the refusals before it and how long that took (ms).""" + began = time.monotonic() + pending = 0 + while True: + result = call() + if result.get('code') != 'RECOVERY_PENDING' or time.monotonic() - began >= limit: + return result, pending, round((time.monotonic() - began) * 1000, 3) + pending += 1 + time.sleep(period) + + +def resume_result(gateway, approval, timeout=host.CALL_TIMEOUT_S): + """operation_resume of a granted approval as an outcome: started, with its process, refused + with CodeSpace's code, or an error of the transport.""" + try: + payload, latency = gateway.resume(approval, timeout) + except ToolError as error: + return {'outcome': 'error' if error.transport else 'refused', 'code': error.code, + 'message': error.message[:500], 'latency_ms': error.latency_ms} + started = payload.get('exec_command') or {} + return {'outcome': 'started' if started.get('process_id') else 'answered', 'process_id': started.get('process_id'), + 'dispatch': started.get('dispatch_status'), 'latency_ms': latency} + + +def status_of(gateway, process_id, timeout=10): + try: + payload, latency = gateway.status(process_id, timeout) + except ToolError as error: + return {'error': error.code, 'message': error.message[:500]} + return dict(payload, latency_ms=latency) + + +def wait_state(gateway, process_id, accept, limit=15.0, period=0.1): + """The process's status once `accept(status)` holds, or the last one after `limit` seconds.""" + deadline = time.monotonic() + limit + while True: + status = status_of(gateway, process_id) + if accept(status) or time.monotonic() >= deadline: + return status + time.sleep(period) + + +def exited(status): + return status.get('state') == 'exited' + + +def not_running(status): + return status.get('state') not in (None, 'running') or bool(status.get('error')) + + +def refused_code(result, code, limit_message=None): + return result.get('outcome') == 'refused' and result.get('code') == code + + +def record_of(bench, process_id): + try: + return host.process_records(bench.operations).get(process_id) + except Exception as error: # noqa: BLE001 - an unreadable record is evidence + return {'error': f'{type(error).__name__}: {error}'} + + +def argv_digest(argv): + """The digest a process record keeps of its argv: SHA-256 of the argv as a compact JSON array.""" + return 'sha256:' + hashlib.sha256(json.dumps(list(argv), separators=(',', ':'), ensure_ascii=False).encode()).hexdigest() + + +class OutputReader: + """Reads one process's output on from where it left off, through whichever gateway it is + given, keeping each answer: the cursor it asked from, the next one, the chunk, and what was + lost before it (analysis.continuity_metrics).""" + + def __init__(self, process_id): + self.process_id, self.cursor, self.rows = process_id, 0, [] + + def read(self, gateway, timeout=10): + payload, latency = gateway.call('read_process', {'process_id': self.process_id, 'cursor': self.cursor}, timeout) + row = {'from': self.cursor, 'to': payload.get('cursor', self.cursor), 'chunk': payload.get('chunk') or '', + 'eof': bool(payload.get('eof')), 'output_lost': bool(payload.get('output_lost')), + 'retained_from': payload.get('retained_from'), 'lost_reason': payload.get('lost_reason'), + 'latency_ms': latency} + self.rows.append(row) + self.cursor = row['to'] + return row + + +def continuity_checks(label, rows, name, declared=False): + metrics = analysis.continuity_metrics(rows, name) + checks = [check(f'{label}: the output went on without a gap or a repeat', metrics['passed'], metrics=metrics)] + if declared: + checks.append(check(f'{label}: the overflow was declared as ring_overflow', + metrics['declared_gaps'] >= 1 and metrics['gap_reasons'] == ['ring_overflow'], + metrics=metrics)) + return checks, metrics + + +def attempt_checks(bench, gateway, process_id, label): + """With an authority: the execution's attempt as DevGuard settled it once its scope ended.""" + if not bench.governed: + return [] + attempt = attempt_of(gateway, process_id) + view = settled_view(bench, attempt) + return [check(f'{label}: DevGuard released its attempt, holding nothing', released_cleanly(view), + attempt_id=attempt, view=view)] + +# -------------------------------- recovery-lifecycle (P1R-C01) + +# Each gateway's exit intent and how it ends; the last one stops the Runner. +EXIT_STEPS = (('detach', 'eof'), ('detach', 'sigterm'), ('stop', 'sigusr1'), ('detach', 'sigkill'), ('stop', 'eof')) +# The process timeout of the timeout scenario's first gateway. +TIMEOUT_CASE_S = 8 + + +def recovery_exit_kinds(bench, tty): + """A process runs while its gateway ends in every way a gateway ends, each time followed by + the next gateway: its input closed or SIGTERM with the exit intent `detach`, SIGUSR1 with + `stop` (a restart always detaches), SIGKILL, which says nothing; then a gateway with `stop` + whose input closes stops the Runner. The Runner and the process outlive every gateway but the + last; each gateway claims the Runner under the next epoch, finds the same process running and + reads its output on from where the one before stopped; the last one's Stop ends the process, + and the Runner removes its socket and exits 0 (P1R-C01, P1R-C03).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + gateway = bench.gateway('g1', intent=EXIT_STEPS[0][0]) + first = gateway + started = gateway.exec('a', tagged(bench.roots['a'], 'beat', BEAT), tty=tty) + checks.append(started_checks('the process', started)) + if started['outcome'] != 'started': + return checks, {'started': started} + process_id = started['process_id'] + pid = tag_pid(bench.roots['a'], 'beat') + claim = claimed_by(runner, gateway.pid) + checks.append(check('gateway 1 claimed the Runner under epoch 1', bool(claim and claim['epoch'] == 1), claim=claim)) + reader, steps = OutputReader(process_id), [] + for step, (intent, how) in enumerate(EXIT_STEPS): + time.sleep(1.0) + reader.read(gateway) + label = f'gateway {step + 1} ({intent}, {how})' + before = runner_log_counts(runner) + ended = bench.end(gateway, how) + steps.append({'intent': intent, 'how': how, 'ended': ended}) + checks.append(check(f'{label}: exited as a gateway ended so exits', + ended['returncode'] == analysis.GATEWAY_EXIT[how] and not ended['stopped'], ended=ended)) + if step == len(EXIT_STEPS) - 1: + break + time.sleep(0.5) + after = runner_log_counts(runner) + said = 'lost' if how == 'sigkill' else 'detached' + logged = 'left without a word' if said == 'lost' else 'detached' + checks += [check(f'{label}: the Runner logged that its controller {logged}', after[said] == before[said] + 1, + before=before, after=after), + check(f'{label}: the Runner and the process still run', runner.alive() and host.running(pid), + pid=pid)] + gateway = bench.gateway(f'g{step + 2}', intent=EXIT_STEPS[step + 1][0]) + claim = claimed_by(runner, gateway.pid) + status = status_of(gateway, process_id) + checks += [check(f'gateway {step + 2} claimed the Runner under epoch {step + 2}, after a controller with work', + bool(claim and claim['epoch'] == step + 2 and claim['previous_controller']), claim=claim), + check(f'gateway {step + 2}: the same process runs', status.get('state') == 'running', status=status)] + time.sleep(0.5) + reader.read(gateway) + checks += stopped_checks(bench, runner, 'the last gateway stopped the Runner', terminated=1) + checks.append(check('the process ended with the Runner', gone(pid, limit=10), pid=pid)) + continuity, metrics = continuity_checks('the process', reader.rows, 'beat') + checks += continuity + checks += attempt_checks(bench, first, process_id, 'the process') + return checks, {'steps': steps, 'claims': runner.claims(), 'continuity': metrics} + + +def deadline_of(bench, process_id, limit=5.0): + """The deadline (Unix ms) the process's record holds, once it holds one.""" + deadline = time.monotonic() + limit + while True: + record = record_of(bench, process_id) or {} + if record.get('deadline_at_ms') or time.monotonic() >= deadline: + return record.get('deadline_at_ms') + time.sleep(0.05) + + +def recovery_original_timeout(bench, tty): + """Two processes reach their timeouts across gateway restarts: the first while no gateway + runs, the second while a later gateway with a much longer process timeout serves. Each ends at + the deadline its record held when it started, as the Runner timed it, never at one a gateway + restarted or lengthened, and is reported timed out (P1R-C01, P1R-C02).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + watch = host.PidWatch() + detail = {} + try: + g1 = bench.gateway('g1', timeout_s=TIMEOUT_CASE_S) + first = g1.exec('a', tagged(bench.roots['a'], 'first', ['/bin/sleep', '600']), tty=tty) + checks.append(started_checks('the first process', first)) + if first['outcome'] != 'started': + return checks, {'first': first} + watch.watch('first', tag_pid(bench.roots['a'], 'first')) + d1 = deadline_of(bench, first['process_id']) + time.sleep(2.0) + detail['g1'] = bench.end(g1, 'sigkill') + time.sleep(1.0) + g2 = bench.gateway('g2', timeout_s=TIMEOUT_CASE_S - 1) + second, _, _ = once_reconciled(lambda: g2.exec('b', tagged(bench.roots['b'], 'second', ['/bin/sleep', '600']), + tty=tty)) + checks.append(started_checks('the second process', second)) + if second['outcome'] != 'started' or not d1: + return checks, dict(detail, second=second, d1=d1) + watch.watch('second', tag_pid(bench.roots['b'], 'second')) + d2 = deadline_of(bench, second['process_id']) + until_unix(d1 / 1000 - 2.0) + detail['g2'] = bench.end(g2, 'sigterm') + until_unix(d1 / 1000 + 1.0) + g3 = bench.gateway('g3', timeout_s=RECOVERY_TIMEOUT_S) + rows = [] + for key, started, deadline, in_gap in (('first', first, d1, True), ('second', second, d2, False)): + gone_unix = watch.wait(key, max(0.0, (deadline or 0) / 1000 + 5.0 - time.time())) + status = wait_state(g3, started['process_id'], exited) + later = (record_of(bench, started['process_id']) or {}).get('deadline_at_ms') + rows.append({'tag': key, 'deadline_at_ms': deadline, 'deadline_after_ms': later, 'gone_unix': gone_unix, + 'termination': status.get('termination'), 'in_gap': in_gap}) + checks.append(check(f"the {key} process timed out at its own deadline{' while no gateway ran' if in_gap else ''}", + analysis.timeout_metrics([rows[-1]])['passed'], row=rows[-1], status=status)) + checks += attempt_checks(bench, g1 if key == 'first' else g2, started['process_id'], f'the {key} process') + detail['timeouts'] = analysis.timeout_metrics(rows) + detail['rows'] = rows + finally: + watch.stop() + return checks, detail + + +def until_unix(moment): + time.sleep(max(0.0, moment - time.time())) + + +def recovery_runner_sigterm(bench): + """The Runner's own SIGTERM, as an operator stops it, stops it as a controller's Stop does: it + terminates its pipe and PTY processes, exits 0 and removes its socket; its gateway, which no + longer finds it, reports neither process running and refuses new work; with an authority, both + attempts are released and the instance retires (P1R-C01).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + gateway = bench.gateway('g1') + started, pids = [], [] + for workspace, tty in (('a', False), ('b', True)): + entry = gateway.exec(workspace, tagged(bench.roots[workspace], f'beat-{io_name(tty)}', BEAT), tty=tty) + checks.append(started_checks(f'the {io_name(tty)} process', entry)) + started.append(entry) + pids.append(tag_pid(bench.roots[workspace], f'beat-{io_name(tty)}') if entry['outcome'] == 'started' else None) + if any(entry['outcome'] != 'started' for entry in started): + return checks, {'started': started} + time.sleep(1.0) + runner.process.send_signal(signal.SIGTERM) + checks += stopped_checks(bench, runner, 'SIGTERM', terminated=2) + statuses = [] + for entry, pid in zip(started, pids): + status = wait_state(gateway, entry['process_id'], not_running) + statuses.append(status) + checks += [check(f"{entry['process_id']}: the process ended", gone(pid, limit=10), pid=pid), + check(f"{entry['process_id']}: the gateway no longer reports it running", not_running(status), + status=status)] + checks += attempt_checks(bench, gateway, entry['process_id'], entry['process_id']) + later = wait_refusal(gateway, 'c', 'RUNNER_DISCONNECTED') + checks.append(check('new work is refused with RUNNER_DISCONNECTED', refused_code(later, 'RUNNER_DISCONNECTED'), + result=later)) + return checks, {'statuses': statuses, 'later': later} + + +def wait_refusal(gateway, workspace, code, limit=20.0): + """exec_command in `workspace` until it is refused with `code`, for up to `limit` seconds.""" + deadline = time.monotonic() + limit + while True: + result = gateway.exec(workspace, ['/bin/echo', 'refused']) + if result.get('code') == code or result['outcome'] == 'started' or time.monotonic() >= deadline: + return result + time.sleep(0.2) + + +def recovery_invalid_settings(bench): + """Settings that do not fit together are refused before anything reaches the Runner: an exit + intent without the independent lifecycle, the lifecycle without an exit intent, in process, + with a worker to start, without a socket or without an operations database; a managed gateway + finds an independent Runner, and an independent gateway a managed worker. A Runner refuses a + socket directory that is not private, a shared one, no socket, an unknown lifecycle, a handed + credential and a socket where another Runner listens. The Runner that listens is left as it + was: it logged no Claim, and a gateway then claims it under epoch 1 and runs a command + (P1R-C01).""" + runner = bench.runner() + socket = str(bench.socket) + env = bench.environment() + detach = ['--runner-exit-intent', 'detach'] + independent = ['--runner-lifecycle', 'independent'] + managed_dir = bench.work / 'mw' + managed_dir.mkdir(mode=0o700) + managed_socket = managed_dir / 'runner.sock' + worker = subprocess.Popen([str(bench.args.runtime_bin), str(managed_socket)], stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=open(bench.evidence.file(f'{bench.prefix}/managed-worker.log'), 'ab'), + env=host.minimal_env(bench.work), start_new_session=True) + bench.live.workers.append(worker) + # A managed worker serves the first connection it accepts and no other, so it is not probed. + deadline = time.monotonic() + 15 + while not managed_socket.exists() and time.monotonic() < deadline and worker.poll() is None: + time.sleep(0.05) + gateways = [ + ('an exit intent without the independent lifecycle', ['--runner', 'uds', '--runner-socket', socket, *detach], env, + '--runner-exit-intent is only for --runner-lifecycle independent'), + ('the independent lifecycle without an exit intent', ['--runner', 'uds', '--runner-socket', socket, *independent], + env, 'needs --runner-exit-intent detach or stop'), + ('in process', ['--runner', 'in-process', *independent, *detach], env, 'needs --runner uds'), + ('with a worker to start', ['--runner', 'uds', '--runner-socket', socket, '--runtime-bin', + str(bench.args.runtime_bin), *independent, *detach], env, + '--runtime-bin and --runner-dir are for a worker the gateway starts'), + ('without a socket', ['--runner', 'uds', *independent, *detach], env, 'needs --runner-socket'), + ('without an operations database', bench.flags('detach'), bench.environment(database=False), + 'needs --operations-db'), + ('a managed gateway at the independent Runner', ['--runner', 'uds', '--runner-socket', socket], env, + 'is an independent Runner'), + ('an independent gateway at a managed worker', ['--runner', 'uds', '--runner-socket', str(managed_socket), + *independent, *detach], env, 'not an independent Runner')] + checks, detail = [], {} + for index, (label, flags, environment, needle) in enumerate(gateways): + result = bench.refused(f'gateway-{index + 1}', flags, environment, limit=60) + detail[label] = result + checks.append(check(f'{label}: refused', result['returncode'] not in (0, None) and needle in result['output'], + returncode=result['returncode'], output=result['output'][-600:])) + if worker.poll() is None: + worker.terminate() + worker.wait(timeout=10) + open_dir = bench.work / 'open' + open_dir.mkdir(mode=0o755) + os.chmod(open_dir, 0o755) + lifecycle = ['--lifecycle', 'independent'] + runners = [ + ('a socket directory that is not private', [str(open_dir / 'runner.sock'), *lifecycle], 1, + 'must be a directory of this user with mode 0700'), + ('a shared socket directory', [f'/tmp/csq-{os.getpid()}.sock', *lifecycle], 1, 'must not be /, /tmp'), + ('no socket', lifecycle, 2, "needs the socket's path first"), + ('an unknown lifecycle', [socket, '--lifecycle', 'forever'], 2, '--lifecycle must be managed or independent'), + ('a handed credential', [socket, *lifecycle, '--devguard-socket', '/nonexistent/authority.sock', + '--devguard-consumer', 'codespace', '--devguard-generation', 'g1', + '--devguard-credential-fd', '3'], 2, '--devguard-credential-file only'), + ('a socket where a Runner listens', [socket, *lifecycle], 1, 'already in use')] + for label, arguments, code, needle in runners: + try: + result = subprocess.run([str(bench.args.runtime_bin), *arguments], stdin=subprocess.DEVNULL, + capture_output=True, text=True, timeout=30, env=host.minimal_env(bench.work), + start_new_session=True) + returncode, output = result.returncode, result.stderr + except subprocess.TimeoutExpired: + returncode, output = None, 'did not exit within 30 s' + detail[label] = {'returncode': returncode, 'output': output[-600:]} + checks.append(check(f'a Runner with {label}: refused with exit status {code}', + returncode == code and needle in output, returncode=returncode, output=output[-600:])) + checks += [check('the Runner logged no Claim and still runs', runner.claims() == [] and runner.alive(), + claims=runner.claims()), + check('its socket directory is unchanged', stat_mode(bench.socket.parent) == 0o700, + mode=stat_mode(bench.socket.parent))] + gateway = bench.gateway('g1') + claim = claimed_by(runner, gateway.pid) + ran_checks, _, _ = ran(gateway, gateway.exec('a', ['/bin/echo', 'fine']), 'fine\n', 'a command') + checks.append(check('a gateway then claimed the Runner under epoch 1', bool(claim and claim['epoch'] == 1), claim=claim)) + return checks + ran_checks, detail + + +def stat_mode(path): + try: + return os.stat(path).st_mode & 0o777 + except OSError: + return None + + +# -------------------------------- recovery-store (P1R-C02) + +def recovery_records(bench, tty): + """What the operations database holds of an execution sent to the independent Runner, and + after a gateway restart: the Runner's identity as its Hello states it, the controller epoch, + the workspace, the argv's digest and never the argv, the root process of a pipe execution (a + PTY's is not named), the original deadline, and how it ended once a later gateway terminated + it (P1R-C02).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + probe = host.WireClient(bench.socket) + try: + hello = probe.hello() + finally: + probe.close() + identity = (hello.get('data') or {}).get('identity') or {} + token = f'qualify-argv-{secrets.token_hex(8)}' + command = tagged(bench.roots['a'], 'record', ['/bin/sh', '-c', 'sleep 600', token]) + g1 = bench.gateway('g1') + started = g1.exec('a', command, tty=tty) + checks.append(started_checks('the process', started)) + if started['outcome'] != 'started': + return checks, {'started': started} + process_id = started['process_id'] + pid = tag_pid(bench.roots['a'], 'record') + row = record_of(bench, process_id) or {} + stored = [name for name, value in row.items() if isinstance(value, str) and token in value] + database = b''.join(path.read_bytes() for path in bench.work.glob('operations.sqlite*') if path.is_file()) + expected_participation = 'required' if bench.governed else 'off' + checks += [ + check('the record is running', row.get('phase') == 'running', record=row), + check("it names the Runner by its Hello's identity", + row.get('runner_pid') == identity.get('pid') == runner.pid + and row.get('runner_boot_id') == identity.get('boot_id') + and row.get('runner_started_at_ms') == identity.get('started_at_ms') + and row.get('runner_nonce') == identity.get('nonce'), identity=identity), + check('under controller epoch 1, in its workspace, as sent', row.get('controller_epoch') == 1 + and row.get('workspace_id') == 'a' and bool(row.get('tty')) == tty + and row.get('participation') == expected_participation, record=row), + check("it holds the argv's digest", row.get('argv_digest') == argv_digest(command), + recorded=row.get('argv_digest'), expected=argv_digest(command)), + check('neither the record nor the database holds the argv', not stored and token.encode() not in database, + columns=stored), + check('the root process is the pipe process, and a PTY root is not named', + row.get('root_pid') == (None if tty else pid), root_pid=row.get('root_pid'), pid=pid), + check('its deadline is the process timeout from its start', + row.get('deadline_at_ms') is not None and row.get('created_at_ms') is not None + and abs(row['deadline_at_ms'] - row['created_at_ms'] - RECOVERY_TIMEOUT_S * 1000) <= 2000, record=row)] + if bench.governed: + checks.append(check('it names its DevGuard attempt', row.get('attempt_id') is not None + and row.get('attempt_id') == attempt_of(g1, process_id), record=row)) + bench.end(g1, 'sigkill') + g2 = bench.gateway('g2') + g2.terminate(process_id) + status = wait_state(g2, process_id, exited) + after = record_of(bench, process_id) or {} + checks += [check('a later gateway terminated it', status.get('termination') == 'terminated', status=status), + check('its record says it was terminated, its deadline unchanged', + after.get('phase') == 'terminal' and after.get('termination') == 'terminated' + and after.get('deadline_at_ms') == row.get('deadline_at_ms') and not after.get('lost'), record=after)] + checks += attempt_checks(bench, g1, process_id, 'the process') + return checks, {'record': row, 'after': after, 'hello': hello} + + +def recovery_writer_lock(bench): + """One gateway at a time writes the operations database (P1R-C02): a second gateway started + on it while the first serves waits a bounded time and refuses to start, naming the holder, + and never reaches the Runner; one started while the first ends waits for it and then serves, + claiming the Runner after it.""" + runner = bench.runner() + checks = [] + g1 = bench.gateway('g1') + second = bench.refused('second-writer') + claims = runner.claims() + checks += [check('a second gateway on the database refused to start, naming the holder', + second['returncode'] not in (0, None) and 'has another writer' in second['output'] + and str(g1.pid) in second['output'], returncode=second['returncode'], output=second['output'][-600:]), + check('after waiting a bounded time', 14.0 <= second['elapsed_s'] <= 45.0, elapsed_s=second['elapsed_s']), + check('it never reached the Runner', [claim['pid'] for claim in claims] == [g1.pid], claims=claims)] + ran_checks, _, _ = ran(g1, g1.exec('a', ['/bin/echo', 'still']), 'still\n', 'the first gateway') + checks += ran_checks + g3 = bench.gateway('g3', initialize=False) + time.sleep(1.5) + ended = bench.end(g1, 'eof') + g3.initialize(timeout=60) + claim = claimed_by(runner, g3.pid) + log = g3.log_path.read_text(errors='replace') + checks += [check('a gateway started while the first one ended waited for it, then served', + 'waiting for it to end' in log and 'the only writer of the operations database' in log + and ended['returncode'] == 0, ended=ended), + check('it claimed the Runner under epoch 2', bool(claim and claim['epoch'] == 2), claim=claim)] + later, _, _ = ran(g3, g3.exec('a', ['/bin/echo', 'next']), 'next\n', 'the next gateway') + return checks + later, {'second': {key: second[key] for key in ('returncode', 'elapsed_s')}} + + +def sqlite_write(path, statement): + connection = sqlite3.connect(str(path), timeout=5) + try: + connection.execute(statement) + connection.commit() + finally: + connection.close() + + +def recovery_schema(bench): + """An operations database the gateway cannot use is refused before anything reaches the + Runner, and left unchanged: one of a newer schema version, and one that is not a database + (P1R-C02).""" + runner = bench.runner() + g1 = bench.gateway('g1') + checks, _, _ = ran(g1, g1.exec('a', ['/bin/echo', 'first']), 'first\n', 'the first gateway') + bench.end(g1, 'eof') + newer = bench.work / 'newer.sqlite' + shutil.copy(bench.operations, newer) + sqlite_write(newer, 'PRAGMA user_version = 2') + garbage = bench.work / 'garbage.sqlite' + garbage.write_bytes(secrets.token_bytes(8192)) + before = runner.claims() + detail = {} + for label, path, needle in (('a newer schema', newer, 'schema version 2, newer than this'), + ('not a database', garbage, '')): + digest = host.sha256(path) + result = bench.refused(label.replace(' ', '-'), env=bench.environment(database=path), limit=60) + detail[label] = {key: result[key] for key in ('returncode', 'elapsed_s')} + checks += [check(f'{label}: refused', result['returncode'] not in (0, None) and needle in result['output'], + returncode=result['returncode'], output=result['output'][-600:]), + check(f'{label}: the file is unchanged', host.sha256(path) == digest)] + checks.append(check('nothing reached the Runner', runner.claims() == before and runner.alive(), claims=runner.claims())) + g2 = bench.gateway('g2') + claim = claimed_by(runner, g2.pid) + checks.append(check('the gateway of the database the Runner knows claimed it under epoch 2', + bool(claim and claim['epoch'] == 2), claim=claim)) + later, _, _ = ran(g2, g2.exec('a', ['/bin/echo', 'next']), 'next\n', 'it') + return checks + later, detail + + +def recovery_records_command(bench): + """`codespace-mcp records` (P1R-C05) lists the records that may still hold a workspace and + releases a record only while no gateway runs, only for an execution that has not ended, and + not while its Runner's PID runs a process, which may be that Runner.""" + runner = bench.runner() + g1 = bench.gateway('g1') + held = g1.exec('a', tagged(bench.roots['a'], 'held', ['/bin/sleep', '600'])) + done = g1.exec('b', tagged(bench.roots['b'], 'done', ['/bin/sh', '-c', 'exit 4'])) + checks = [started_checks('a running process', held), started_checks('an ending process', done)] + if held['outcome'] != 'started' or done['outcome'] != 'started': + return checks, {'held': held, 'done': done} + wait_state(g1, done['process_id'], exited) + env = host.minimal_env(bench.work) + listed = host.records_command(bench.args.mcp_bin, bench.operations, env, 'list') + found = {entry['process_id']: entry for entry in ((listed['report'] or {}).get('records') or [])} + entry = found.get(held['process_id']) or {} + while_serving = host.records_command(bench.args.mcp_bin, bench.operations, env, 'release', held['process_id']) + bench.end(g1, 'eof') + releases = {label: host.records_command(bench.args.mcp_bin, bench.operations, env, 'release', process_id) + for label, process_id in (('unknown', 'no-such-process'), ('ended', done['process_id']), + ('running', held['process_id']))} + + def refusal(result): + return ' '.join(item.get('reason', '') for item in ((result['report'] or {}).get('refused') or [])) + checks += [ + check('records list shows the running one, with its Runner and root, and not the ended one', + listed['done'] and held['process_id'] in found and done['process_id'] not in found + and entry.get('phase') == 'running' and (entry.get('runner') or {}).get('pid') == runner.pid + and (entry.get('root') or {}).get('running') is True, listed=listed['report']), + check('release is refused while a gateway runs', not while_serving['done'] + and 'only while no gateway runs' in while_serving['stderr'], result=while_serving), + check('an unknown process is refused', not releases['unknown']['done'] + and 'no record has this process ID' in refusal(releases['unknown']), result=releases['unknown']), + check('an ended execution is refused', not releases['ended']['done'] + and 'its execution ended' in refusal(releases['ended']), result=releases['ended']), + check("a running one is refused while its Runner's PID runs", not releases['running']['done'] + and f"Runner's PID {runner.pid} runs" in refusal(releases['running']), result=releases['running'])] + g2 = bench.gateway('g2') + status = status_of(g2, held['process_id']) + checks.append(check('the next gateway still finds it running', status.get('state') == 'running', status=status)) + g2.terminate(held['process_id']) + return checks, {'listed': listed['report'], 'releases': {key: value['report'] for key, value in releases.items()}} + +# -------------------------------- recovery-fence (P1R-C03) + +def fence_requests(client, targets, epochs): + """Requests that would change running processes, under each of `epochs` (None: none): the + Runner must refuse each with STALE_CONTROLLER and change nothing.""" + found = [] + for epoch in epochs: + found += [client.request('write_stdin', {'request': {'process_id': targets['write'], 'data': ''}}, epoch=epoch), + client.request('resize', {'process_id': targets['resize'], 'rows': 30, 'cols': 90}, epoch=epoch), + client.request('terminate', {'process_id': targets['terminate']}, epoch=epoch), + client.request('terminate_workspace', {'workspace_id': targets['workspace']}, epoch=epoch), + client.request('stop', epoch=epoch)] + return [{'op': entry['op'], 'epoch': epoch, 'ok': entry['ok'], 'code': entry['code'], 'type': entry['type'], + 'message': (entry['message'] or '')[:300]} for entry, epoch in zip(found, [e for e in epochs for _ in range(5)])] + + +def stale_checks(label, requests): + accepted = [request for request in requests if request['ok']] + unfenced = [request for request in requests if not request['ok'] and request['code'] != 'STALE_CONTROLLER'] + return [check(f'{label}: every request that would change a process was refused as stale, nothing changed', + not accepted and not unfenced and bool(requests), accepted=accepted, unfenced=unfenced)] + + +def two_targets(bench, gateway, tty): + """A pipe or PTY beat in `a` and a `cat` in `b`, both of the bench's participation.""" + beat = gateway.exec('a', tagged(bench.roots['a'], 'beat', BEAT), tty=tty) + cat = gateway.exec('b', tagged(bench.roots['b'], 'cat', ['/bin/cat'])) + return beat, cat + + +def recovery_foreign_connection(bench, tty): + """While a gateway controls the Runner, another process of the same user connects to its + socket: it reads the Runner's Hello, its Claim is refused because a controller is connected, + and every request it sends that would change a process, under the controller's epoch or any + other, is refused as stale; a Detach changes nothing. The gateway goes on controlling the + Runner and its processes as before (P1R-C03).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + gateway = bench.gateway('g1') + beat, cat = two_targets(bench, gateway, tty) + checks += [started_checks('the beat', beat), started_checks('the cat', cat)] + if beat['outcome'] != 'started' or cat['outcome'] != 'started': + return checks, {'beat': beat, 'cat': cat} + reader = OutputReader(beat['process_id']) + time.sleep(0.6) + reader.read(gateway) + store = host.store_id(bench.operations) + targets = {'write': cat['process_id'], 'resize': beat['process_id'], 'terminate': beat['process_id'], + 'workspace': 'a'} + client = host.WireClient(bench.socket) + try: + hello = client.hello() + claim = client.claim(store) + requests = fence_requests(client, targets, (1, None, 2)) + detach = client.detach() + finally: + client.close() + refusals = [entry for entry in runner.claims() if not entry['granted']] + checks += [check("it reads the Runner's Hello", hello['ok'] and hello['type'] == 'hello', hello=hello), + check('its Claim is refused while the controller is connected', claim['type'] == 'claim_refused' + and (claim['data'] or {}).get('refusal') == 'controller_active', claim=claim), + check('the Runner logged the refused Claim with its PID', any(entry['pid'] == os.getpid() + and entry['refusal'] == 'controller_active' + for entry in refusals), refusals=refusals), + check('its Detach is answered and changes nothing', detach['ok'], detach=detach)] + checks += stale_checks('the other connection', requests) + time.sleep(0.6) + reader.read(gateway) + beat_status, cat_status = status_of(gateway, beat['process_id']), status_of(gateway, cat['process_id']) + gateway.call('write_stdin', {'process_id': cat['process_id'], 'data': 'still-here\n'}) + echoed = gateway.read_until(cat['process_id'], 'still-here')[3] + checks += [check('both processes still run', beat_status.get('state') == 'running' + and cat_status.get('state') == 'running', beat=beat_status, cat=cat_status), + check('the gateway still writes to its process', echoed), + check('the Runner logged no other Claim', [entry['pid'] for entry in runner.claims() if entry['granted']] + == [gateway.pid], claims=runner.claims())] + continuity, metrics = continuity_checks('the beat', reader.rows, 'beat') + checks += continuity + for started in (beat, cat): + gateway.terminate(started['process_id']) + wait_state(gateway, started['process_id'], exited) + checks += attempt_checks(bench, gateway, started['process_id'], started['process_id']) + return checks, {'requests': requests, 'continuity': metrics} + + +def recovery_stale_epoch(bench): + """Controller epochs across gateways and another process: once two gateways in turn detached, + a Claim for another operations database is refused; one for the Runner's own, which any + process of the same user can make while no gateway controls the Runner (the exposure D6 + accepts, logged with its PID), is granted under the next epoch, and that connection's + requests that would change a process under an earlier epoch or none are refused as stale. + The next gateway claims the Runner under the epoch after it and finds the same process, + its output on from where it was (P1R-C03).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + g1 = bench.gateway('g1') + beat, cat = two_targets(bench, g1, False) + checks += [started_checks('the beat', beat), started_checks('the cat', cat)] + if beat['outcome'] != 'started' or cat['outcome'] != 'started': + return checks, {'beat': beat, 'cat': cat} + reader = OutputReader(beat['process_id']) + time.sleep(0.6) + reader.read(g1) + bench.end(g1, 'eof') + g2 = bench.gateway('g2') + time.sleep(0.6) + reader.read(g2) + bench.end(g2, 'sigusr1') + store = host.store_id(bench.operations) + targets = {'write': cat['process_id'], 'resize': beat['process_id'], 'terminate': beat['process_id'], + 'workspace': 'a'} + client = host.WireClient(bench.socket) + try: + wrong = client.claim('another-operations-database') + claim = client.claim(store) + requests = fence_requests(client, targets, (1, 2, None)) + detach = client.detach() + finally: + client.close() + granted = [entry for entry in runner.claims() if entry['granted']] + checks += [check('a Claim for another operations database is refused', wrong['type'] == 'claim_refused' + and (wrong['data'] or {}).get('refusal') == 'store_mismatch', claim=wrong), + check("a Claim for the Runner's database while no gateway controls it is granted under epoch 3", + claim['type'] == 'claimed' and (claim['data'] or {}).get('epoch') == 3, claim=claim), + check('the Runner logged that Claim with the PID that made it', + bool(granted) and granted[-1]['pid'] == os.getpid() and granted[-1]['epoch'] == 3, claims=granted), + check('its Detach is answered', detach['ok'] and detach['type'] == 'detached', detach=detach)] + checks += stale_checks('requests under epochs 1, 2 and none', requests) + g3 = bench.gateway('g3') + claim3 = claimed_by(runner, g3.pid) + status = status_of(g3, beat['process_id']) + time.sleep(0.6) + reader.read(g3) + checks += [check('the next gateway claimed the Runner under epoch 4', bool(claim3 and claim3['epoch'] == 4), + claim=claim3), + check('the same process runs', status.get('state') == 'running', status=status)] + continuity, metrics = continuity_checks('the beat', reader.rows, 'beat') + checks += continuity + for started in (beat, cat): + g3.terminate(started['process_id']) + wait_state(g3, started['process_id'], exited) + return checks, {'requests': requests, 'claims': runner.claims(), 'continuity': metrics} + + +def recovery_stopped_gateway(bench): + """A controlling gateway stopped with SIGSTOP keeps its connection open, so nothing pushes it + out: another process's Claim is refused, and a second gateway waits for the operations + database's writer and refuses to start. Once the stopped gateway is killed, the next gateway + claims the Runner under epoch 2 and finds the process running (P1R-C02, P1R-C03).""" + runner = bench.runner() + g1 = bench.gateway('g1') + beat = g1.exec('a', tagged(bench.roots['a'], 'beat', BEAT)) + checks = [started_checks('the beat', beat)] + if beat['outcome'] != 'started': + return checks, {'beat': beat} + reader = OutputReader(beat['process_id']) + time.sleep(0.6) + reader.read(g1) + os.kill(g1.pid, signal.SIGSTOP) + try: + client = host.WireClient(bench.socket) + try: + claim = client.claim(host.store_id(bench.operations)) + finally: + client.close() + second = bench.refused('second-gateway') + finally: + os.kill(g1.pid, signal.SIGCONT) + bench.end(g1, 'sigkill') + g3 = bench.gateway('g3') + claim3 = claimed_by(runner, g3.pid) + status = status_of(g3, beat['process_id']) + time.sleep(0.6) + reader.read(g3) + checks += [check("another process's Claim is refused while the stopped gateway's connection is open", + claim['type'] == 'claim_refused' and (claim['data'] or {}).get('refusal') == 'controller_active', + claim=claim), + check('a second gateway refused to start, waiting for the writer', second['returncode'] not in (0, None) + and 'has another writer' in second['output'], returncode=second['returncode'], + output=second['output'][-600:]), + check('once the stopped gateway was killed, the next one claimed the Runner under epoch 2', + bool(claim3 and claim3['epoch'] == 2), claim=claim3), + check('and found the process running', status.get('state') == 'running', status=status)] + continuity, metrics = continuity_checks('the beat', reader.rows, 'beat') + g3.terminate(beat['process_id']) + return checks + continuity, {'continuity': metrics} + + +CONCURRENT_CLAIMS = 8 + + +def recovery_concurrent_claims(bench): + """While no gateway controls the Runner, eight connections claim it at once for its own + operations database: exactly one is granted, the others are refused because a controller is + connected; once it detaches, the next gateway claims the Runner under the epoch after it and + finds the process (P1R-C03).""" + runner = bench.runner() + g1 = bench.gateway('g1') + beat = g1.exec('a', tagged(bench.roots['a'], 'beat', BEAT)) + checks = [started_checks('the beat', beat)] + if beat['outcome'] != 'started': + return checks, {'beat': beat} + bench.end(g1, 'eof') + store = host.store_id(bench.operations) + clients = [host.WireClient(bench.socket) for _ in range(CONCURRENT_CLAIMS)] + gate = threading.Barrier(CONCURRENT_CLAIMS) + answers = [None] * CONCURRENT_CLAIMS + + def claim(index): + gate.wait(timeout=10) + try: + answers[index] = clients[index].claim(store) + except (OSError, ValueError) as error: + answers[index] = {'type': 'error', 'message': str(error)} + threads = [threading.Thread(target=claim, args=(index,), daemon=True) for index in range(CONCURRENT_CLAIMS)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=30) + winners = [index for index, answer in enumerate(answers) if (answer or {}).get('type') == 'claimed'] + refused = [answer for answer in answers if (answer or {}).get('type') == 'claim_refused'] + detached = clients[winners[0]].detach() if len(winners) == 1 else None + for client in clients: + client.close() + g2 = bench.gateway('g2') + claim2 = claimed_by(runner, g2.pid) + status = status_of(g2, beat['process_id']) + checks += [check('exactly one Claim was granted, under epoch 2', len(winners) == 1 + and (answers[winners[0]]['data'] or {}).get('epoch') == 2, answers=answers), + check('the others were refused because a controller was connected', len(refused) == CONCURRENT_CLAIMS - 1 + and all((answer['data'] or {}).get('refusal') == 'controller_active' for answer in refused), + refused=refused), + check('the winner detached', bool(detached and detached['ok']), detach=detached), + check('the next gateway claimed the Runner under epoch 3', bool(claim2 and claim2['epoch'] == 3), claim=claim2), + check('and found the process running', status.get('state') == 'running', status=status)] + g2.terminate(beat['process_id']) + return checks, {'answers': answers} + + +def recovery_other_database(bench): + """A gateway whose operations database is not the one the Runner was first claimed for is + refused and starts nothing, whether or not a gateway controls the Runner: two gateways with + two databases never both write to one Runner (P1R-C03).""" + runner = bench.runner() + g1 = bench.gateway('g1') + beat = g1.exec('a', tagged(bench.roots['a'], 'beat', BEAT)) + checks = [started_checks('the beat', beat)] + if beat['outcome'] != 'started': + return checks, {'beat': beat} + other = bench.work / 'other.sqlite' + during = bench.refused('other-database-while-controlled', env=bench.environment(database=other)) + bench.end(g1, 'eof') + after = bench.refused('other-database-after', env=bench.environment(database=other)) + g2 = bench.gateway('g2') + claim2 = claimed_by(runner, g2.pid) + status = status_of(g2, beat['process_id']) + for label, result in (('while a gateway controls the Runner', during), ('while none does', after)): + checks.append(check(f'a gateway with another database is refused {label}', + result['returncode'] not in (0, None) and 'store_mismatch' in result['output'], + returncode=result['returncode'], output=result['output'][-600:])) + checks += [check("the Runner's own gateway then claimed it under epoch 2", bool(claim2 and claim2['epoch'] == 2), + claim=claim2), + check('and found the process running', status.get('state') == 'running', status=status), + check('the other database never controlled the Runner', + [entry['pid'] for entry in runner.claims() if entry['granted']] == [g1.pid, g2.pid], + claims=runner.claims())] + g2.terminate(beat['process_id']) + return checks, {'during': during['returncode'], 'after': after['returncode']} + + +def recovery_concurrent_gateways(bench): + """Two gateways started at once on the Runner's operations database: one serves and claims the + Runner; the other waits for the writer and refuses to start, never reaching the Runner + (P1R-C02, P1R-C03).""" + runner = bench.runner() + g1 = bench.gateway('g1') + beat = g1.exec('a', tagged(bench.roots['a'], 'beat', BEAT)) + checks = [started_checks('the beat', beat)] + if beat['outcome'] != 'started': + return checks, {'beat': beat} + bench.end(g1, 'eof') + pair = [bench.gateway(f'pair-{index}', initialize=False) for index in (1, 2)] + outcomes = [None, None] + + def start(index): + try: + pair[index].initialize(timeout=60) + outcomes[index] = 'serving' + except ToolError as error: + outcomes[index] = error.code + threads = [threading.Thread(target=start, args=(index,), daemon=True) for index in (0, 1)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=90) + serving = [gateway for gateway, outcome in zip(pair, outcomes) if outcome == 'serving'] + losers = [gateway for gateway, outcome in zip(pair, outcomes) if outcome != 'serving'] + for gateway in losers: + gateway.process.wait(timeout=60) + granted = [entry['pid'] for entry in runner.claims() if entry['granted']] + checks += [check('exactly one of them serves', len(serving) == 1, outcomes=outcomes), + check('the other refused to start, waiting for the writer', + len(losers) == 1 and losers[0].process.returncode not in (0, None) + and 'has another writer' in losers[0].log_path.read_text(errors='replace'), + returncode=[gateway.process.returncode for gateway in losers]), + check('only the serving one claimed the Runner, under epoch 2', + len(serving) == 1 and granted == [g1.pid, serving[0].pid], claims=runner.claims())] + if serving: + status = status_of(serving[0], beat['process_id']) + checks.append(check('it found the process running', status.get('state') == 'running', status=status)) + serving[0].terminate(beat['process_id']) + return checks, {'outcomes': outcomes} + + +# -------------------------------- recovery-reconcile (P1R-C04) + +def recovery_reconcile_running(bench, tty): + """A gateway killed while its process runs: the next gateway refuses new work in that + workspace with RECOVERY_PENDING until it has reconciled it, then holds the workspace for the + process again (WORKSPACE_BUSY), while a workspace where no work was left takes new work at + once; terminating the inherited process frees its workspace. No refused command ran + (P1R-C04).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + g1 = bench.gateway('g1') + held = g1.exec('a', tagged(bench.roots['a'], 'held', ['/bin/sleep', '600']), tty=tty) + checks.append(started_checks('the process', held)) + if held['outcome'] != 'started': + return checks, {'held': held} + bench.end(g1, 'sigkill') + g2 = bench.gateway('g2') + elsewhere = g2.exec('b', tagged(bench.roots['b'], 'elsewhere', ['/bin/echo', 'elsewhere']), tty=tty) + busy, pending, waited = once_reconciled(lambda: g2.exec('a', tagged(bench.roots['a'], 'refused', ['/bin/echo', 'no']), + tty=tty)) + status = status_of(g2, held['process_id']) + g2.terminate(held['process_id']) + ended = wait_state(g2, held['process_id'], exited) + fresh, _, _ = once_reconciled(lambda: g2.exec('a', tagged(bench.roots['a'], 'fresh', ['/bin/echo', 'fresh']), tty=tty)) + fresh_checks, _, _ = ran(g2, fresh, 'fresh\n', 'new work once it was free') + checks += [check('a workspace where no work was left took new work at once', elsewhere['outcome'] == 'started', + result=elsewhere), + check('new work where the process runs was held for it again (WORKSPACE_BUSY), after reconciling', + refused_code(busy, 'WORKSPACE_BUSY'), result=busy, pending_refusals=pending, waited_ms=waited), + check('the inherited process runs', status.get('state') == 'running', status=status), + check('terminating it ended it', ended.get('termination') == 'terminated', status=ended), + check('no refused command ran', 'refused' not in markers(bench.roots['a']) + and markers(bench.roots['a']).count('held') == 1, markers=markers(bench.roots['a']))] + checks += fresh_checks + checks += attempt_checks(bench, g1, held['process_id'], 'the inherited process') + return checks, {'pending_refusals': pending, 'waited_ms': waited} + + +def recovery_reconcile_terminal(bench, tty): + """A process that ends while no gateway runs: the next gateway learns how it ended from the + Runner, which kept its end for the controller, records it and frees the workspace (P1R-C04).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + g1 = bench.gateway('g1') + started = g1.exec('a', tagged(bench.roots['a'], 'ends', ['/bin/sh', '-c', 'sleep 1.5; exit 7']), tty=tty) + checks.append(started_checks('the process', started)) + if started['outcome'] != 'started': + return checks, {'started': started} + bench.end(g1, 'sigkill') + time.sleep(3.0) + g2 = bench.gateway('g2') + status = wait_state(g2, started['process_id'], exited) + fresh, pending, waited = once_reconciled(lambda: g2.exec('a', tagged(bench.roots['a'], 'fresh', ['/bin/echo', 'fresh']), + tty=tty)) + fresh_checks, _, _ = ran(g2, fresh, 'fresh\n', 'new work in its workspace') + record = record_of(bench, started['process_id']) or {} + checks += [check('the next gateway reports how it ended', status.get('termination') == 'exited' + and status.get('exit_code') == 7, status=status), + check('its record says so', record.get('phase') == 'terminal' and record.get('exit_code') == 7, + record=record)] + checks += fresh_checks + checks += attempt_checks(bench, g1, started['process_id'], 'the process') + return checks, {'pending_refusals': pending, 'waited_ms': waited} + + +KILL_DELAYS_S = (0.0, 0.002, 0.005, 0.01, 0.02, 0.04, 0.08, 0.16) + + +def recovery_reconcile_approval(bench): + """Approvals across gateway restarts (P1R-C04): one held and granted before a gateway was + killed is resumed by the next gateway once, and resuming it again returns the same process; + one resumed while the gateway was killed, at a range of moments, is resumed by the next + gateway with the process the Runner created for it, or started then when the Runner proved + it created none: each command ran exactly once.""" + runner = bench.runner() + checks = registered_checks(bench, runner) + root = bench.roots['confirm'] + gateway = bench.gateway('g1') + rounds = [] + for index, delay in enumerate((None,) + KILL_DELAYS_S): + tag = f'approval-{index}' + held = gateway.exec('confirm', tagged(root, tag, ['/bin/sh', '-c', 'exit 0'])) + approval = held.get('approval_id') + row = {'tag': tag, 'held': held.get('code'), 'approval_id': approval, 'delay_s': delay} + rounds.append(row) + if not approval: + continue + gateway.call('approval_resolve', {'approval_id': approval, 'decision': 'grant'}) + if delay is not None: + flight = {} + thread = threading.Thread(target=lambda: flight.update(resume_result(gateway, approval)), daemon=True) + thread.start() + time.sleep(delay) + bench.end(gateway, 'sigkill') + if delay is not None: + thread.join(timeout=10) + row['in_flight'] = flight + gateway = bench.gateway(f'g{index + 2}') + first, pending, waited = once_reconciled(lambda: resume_result(gateway, approval)) + again = resume_result(gateway, approval) + if first.get('process_id'): + wait_state(gateway, first['process_id'], exited) + row.update(resume=first, again=again, pending_refusals=pending, ready_ms=waited, + runs=markers(root).count(tag)) + for row in rounds: + label = f"approval {row['tag']}" + if row['delay_s'] is not None: + label += f" (killed {row['delay_s'] * 1000:.0f} ms into a resume)" + checks += [check(f'{label}: held for approval', row['held'] == 'APPROVAL_REQUIRED' and row['approval_id'], row=row), + check(f'{label}: the next gateway resumed it', bool((row.get('resume') or {}).get('process_id')), + row=row), + check(f'{label}: resuming again returned the same process', + (row.get('again') or {}).get('process_id') == (row.get('resume') or {}).get('process_id'), + row=row), + check(f'{label}: its command ran exactly once', row.get('runs') == 1, row=row)] + return checks, {'rounds': rounds} + + +def recovery_authority_outage(bench): + """The authority holds its lock while a gateway that was killed is replaced: the inherited + governed processes answer at once, and their workspaces stay refused with RECOVERY_PENDING + while the Runner cannot ask the authority about their attempts; ungoverned work runs. Once + the authority answers, the gateway reconciles them, holds each workspace for its process + again, and terminating them releases their attempts (P1R-C04).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + g1 = bench.gateway('g1') + held = [g1.exec('a', tagged(bench.roots['a'], 'pipe', ['/bin/sleep', '600'])), + g1.exec('b', tagged(bench.roots['b'], 'pty', ['/bin/sleep', '600']), tty=True)] + for started in held: + checks.append(started_checks(started.get('process_id') or 'a governed process', started)) + if any(started['outcome'] != 'started' for started in held): + return checks, {'held': held} + bench.end(g1, 'sigkill') + bench.fixture.command('hold 6000') + hold = time.monotonic() + g2 = bench.gateway('g2') + statuses = [status_of(g2, started['process_id']) for started in held] + during = g2.exec('a', tagged(bench.roots['a'], 'during', ['/bin/echo', 'no'])) + off = g2.exec('off', ['/bin/echo', 'off']) + off_checks, _, _ = ran(g2, off, 'off\n', 'ungoverned work during the hold') + during_s = time.monotonic() - hold + until(hold + 7.0) + busy, pending, waited = once_reconciled(lambda: g2.exec('a', tagged(bench.roots['a'], 'after', ['/bin/echo', 'no'])), + limit=60) + checks += [check('the inherited processes answered during the hold', all(status.get('state') == 'running' + and status.get('latency_ms', 1e9) < 1000 + for status in statuses), statuses=statuses), + check('new work in their workspaces was refused with RECOVERY_PENDING during the hold', + refused_code(during, 'RECOVERY_PENDING') and during_s < 6.0, result=during, + during_s=round(during_s, 3)), + check('once the authority answered, the gateway reconciled and held the workspace for its process', + refused_code(busy, 'WORKSPACE_BUSY'), result=busy, pending_refusals=pending, waited_ms=waited), + check('no refused command ran', 'during' not in markers(bench.roots['a']) + and 'after' not in markers(bench.roots['a']), markers=markers(bench.roots['a']))] + checks += off_checks + for started in held: + g2.terminate(started['process_id']) + wait_state(g2, started['process_id'], exited) + checks += attempt_checks(bench, g1, started['process_id'], started['process_id']) + return checks, {'statuses': statuses, 'during': during, 'busy': busy} + + +def recovery_missing_record(bench): + """A record removed from the operations database while no gateway ran, as a damaged database + could lose one: the Runner still holds the process, so the next gateway can query it, and + keeps its workspace refused with RECOVERY_PENDING for an operator rather than start anything + there; other workspaces take new work (P1R-C04).""" + bench.runner() + g1 = bench.gateway('g1') + held = g1.exec('a', tagged(bench.roots['a'], 'held', ['/bin/sleep', '600'])) + checks = [started_checks('the process', held)] + if held['outcome'] != 'started': + return checks, {'held': held} + bench.end(g1, 'eof') + sqlite_write(bench.operations, f"DELETE FROM processes WHERE process_id = '{held['process_id']}'") + g2 = bench.gateway('g2') + time.sleep(3.0) + blocked = g2.exec('a', tagged(bench.roots['a'], 'refused', ['/bin/echo', 'no'])) + elsewhere = g2.exec('b', ['/bin/echo', 'elsewhere']) + elsewhere_checks, _, _ = ran(g2, elsewhere, 'elsewhere\n', 'other work') + status = status_of(g2, held['process_id']) + checks += [check('its workspace stays refused with RECOVERY_PENDING for an operator', + refused_code(blocked, 'RECOVERY_PENDING') and 'operator' in (blocked.get('message') or ''), + result=blocked), + check('the process the Runner holds can be queried', status.get('state') == 'running', status=status), + check('no refused command ran', 'refused' not in markers(bench.roots['a']), markers=markers(bench.roots['a']))] + checks += elsewhere_checks + g2.terminate(held['process_id']) + after = wait_state(g2, held['process_id'], exited) + return checks, {'blocked': blocked, 'terminated': after} + + +def recovery_kill_during_exec(bench, tty): + """Gateways killed while exec_command is in flight, at a range of moments: the next gateway + reconciles what each left, and no command ran more than once: nothing is replayed (P1R-C04).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + gateway = bench.gateway('g1') + rounds = [] + for index, delay in enumerate(KILL_DELAYS_S): + tag = f'exec-{index}' + flight = {} + thread = threading.Thread(target=lambda: flight.update(gateway.exec('a', tagged(bench.roots['a'], tag, + ['/bin/sleep', '0.3']), tty=tty)), + daemon=True) + thread.start() + time.sleep(delay) + bench.end(gateway, 'sigkill') + thread.join(timeout=10) + gateway = bench.gateway(f'g{index + 2}') + probe_tag = f'probe-{index}' + probe, pending, waited = once_reconciled(lambda: gateway.exec('a', tagged(bench.roots['a'], probe_tag, + ['/bin/echo', 'probe']), tty=tty)) + deadline = time.monotonic() + 10 + while refused_code(probe, 'WORKSPACE_BUSY') and time.monotonic() < deadline: + time.sleep(0.2) + probe_tag = f'probe-{index}-{int(time.monotonic() * 1000)}' + probe, _, _ = once_reconciled(lambda: gateway.exec('a', tagged(bench.roots['a'], probe_tag, + ['/bin/echo', 'probe']), tty=tty)) + if probe['outcome'] == 'started': + wait_state(gateway, probe['process_id'], exited) + rounds.append({'tag': tag, 'delay_s': delay, 'in_flight': flight, 'probe': probe.get('outcome'), + 'probe_code': probe.get('code'), 'pending_refusals': pending, 'ready_ms': waited}) + tags = markers(bench.roots['a']) + runs = {row['tag']: tags.count(row['tag']) for row in rounds} + checks += [check('no command ran twice', all(count <= 1 for count in runs.values()) + and all(tags.count(tag) == 1 for tag in tags), runs=runs, markers=tags), + check('after each kill, the next gateway reconciled and took new work', + all(row['probe'] == 'started' for row in rounds), rounds=rounds)] + if bench.governed: + found, charged = host.settle_attempts(gateway_attempts(bench), bench.database, 30, bench.authority['consumer']) + checks.append(check('nothing of it stays charged', not charged, charged=charged)) + return checks, {'rounds': rounds, 'runs': runs} + + +def recovery_kill_during_terminate(bench, tty): + """Gateways killed while terminate_process is in flight, at a range of moments: the next + gateway reports the process terminated, or still running and then terminates it; each process + ends once, terminated, and ran once; its workspace takes new work afterwards (P1R-C04).""" + runner = bench.runner() + checks = registered_checks(bench, runner) + gateway = bench.gateway('g1') + rounds = [] + for index, delay in enumerate(KILL_DELAYS_S): + tag = f'held-{index}' + started, _, _ = once_reconciled(lambda: gateway.exec('a', tagged(bench.roots['a'], tag, ['/bin/sleep', '600']), + tty=tty)) + if started['outcome'] != 'started': + rounds.append({'tag': tag, 'delay_s': delay, 'started': started}) + break + pid = tag_pid(bench.roots['a'], tag) + flight = {} + + def terminate(process_id=started['process_id'], target=gateway, result=flight): + try: + result['ack_ms'] = target.terminate(process_id, timeout=10) + except ToolError as error: + result['error'] = error.code + thread = threading.Thread(target=terminate, daemon=True) + thread.start() + time.sleep(delay) + bench.end(gateway, 'sigkill') + thread.join(timeout=10) + gateway = bench.gateway(f'g{index + 2}') + status = wait_state(gateway, started['process_id'], not_running, limit=3) + again = None + if status.get('state') == 'running': + again = gateway.terminate(started['process_id']) + status = wait_state(gateway, started['process_id'], exited) + rounds.append({'tag': tag, 'delay_s': delay, 'in_flight': flight, 'terminated_again': again is not None, + 'state': status.get('state'), 'termination': status.get('termination'), + 'pid_gone': gone(pid, limit=10) if pid else None}) + tags = markers(bench.roots['a']) + checks += [check('each process ended once, terminated, whichever gateway terminated it', + len(rounds) == len(KILL_DELAYS_S) and all(row.get('termination') == 'terminated' and row.get('pid_gone') + for row in rounds), rounds=rounds), + check('each process ran once', all(tags.count(row['tag']) == 1 for row in rounds), markers=tags)] + fresh, _, _ = once_reconciled(lambda: gateway.exec('a', ['/bin/echo', 'fresh'], tty=tty)) + fresh_checks, _, _ = ran(gateway, fresh, 'fresh\n', 'new work in its workspace') + return checks + fresh_checks, {'rounds': rounds} + + +def gateway_attempts(bench): + return sorted({attempt for gateway in bench.gateways for attempt in gateway.attempts()}) + + +# -------------------------------- recovery-observation (P1R-C05) + +def recovery_ring_overflow(bench, tty): + """A process writes more than the Runner's 256 KiB output ring while no gateway runs: the next + gateway reads on from where the last read stopped, says that output was lost before what it + returns and why (`ring_overflow`), and returns the newest output whole; the status says so + too (P1R-C05).""" + bench.runner() + cue = bench.work / 'cue' + g1 = bench.gateway('g1') + started = g1.exec('off', tagged(bench.roots['off'], 'burst', burst(cue)), tty=tty) + checks = [started_checks('the burst', started)] + if started['outcome'] != 'started': + return checks, {'started': started} + reader = OutputReader(started['process_id']) + cue.touch() + time.sleep(1.5) + first = reader.read(g1) + bench.end(g1, 'sigterm') + cue.touch() + time.sleep(1.5) + g2 = bench.gateway('g2') + second = reader.read(g2) + status = status_of(g2, started['process_id']) + continuity, metrics = continuity_checks('the burst', reader.rows, 'burst', declared=True) + checks += [check('the first gateway read the end of its first burst', f'burst {BURST_LINES:08d}' in first['chunk']), + check('the next gateway says output was lost before what it returns, because the ring overflowed', + second['output_lost'] and second['lost_reason'] == 'ring_overflow' + and (second['retained_from'] or 0) > second['from'], + row={key: second[key] for key in ('from', 'to', 'output_lost', 'retained_from', 'lost_reason')}), + check('and returns the newest output whole', f'burst {2 * BURST_LINES:08d}' in second['chunk']), + check('the status says the same', status.get('lost_reason') == 'ring_overflow' + and (status.get('output_retained_from') or 0) > 0, status=status)] + checks += continuity + g2.terminate(started['process_id']) + return checks, {'continuity': metrics} + + +RETENTION_COUNT = 140 + + +def recovery_retention(bench): + """More ended processes than the Runner keeps (64, then 64 released): once it no longer keeps + the oldest, process_status and read_process answer from the records, how it ended with no + output retained and `retention_expired`, also through the next gateway (P1R-C05).""" + bench.runner() + g1 = bench.gateway('g1') + ids = [] + for index in range(RETENTION_COUNT): + started = g1.exec('off', ['/bin/sh', '-c', f'echo {index}; exit {index % 7}']) + if started['outcome'] != 'started': + return [started_checks(f'process {index}', started)], {} + ids.append(started['process_id']) + wait_state(g1, started['process_id'], exited, limit=10, period=0.02) + oldest = ids[0] + status = status_of(g1, oldest) + read, _ = g1.call('read_process', {'process_id': oldest, 'cursor': 0}) + bench.end(g1, 'eof') + g2 = bench.gateway('g2') + again = status_of(g2, oldest) + newest = status_of(g2, ids[-1]) + checks = [check('the oldest is answered from its record: how it ended, and why its output is gone', + status.get('state') == 'exited' and status.get('exit_code') == 0 + and status.get('lost_reason') == 'retention_expired', status=status), + check('reading it returns no output, declared lost for retention', + read.get('chunk') == '' and read.get('output_lost') is True + and read.get('lost_reason') == 'retention_expired', read=read), + check('the next gateway answers the same', again.get('lost_reason') == 'retention_expired' + and again.get('exit_code') == 0, status=again), + check('the newest is still kept', newest.get('state') == 'exited' and newest.get('lost_reason') is None + and newest.get('exit_code') == (RETENTION_COUNT - 1) % 7, status=newest)] + return checks, {} + + +def recovery_runner_loss(bench, tty): + """The Runner killed with SIGKILL (P1R-C05, D8): its gateway answers from the records, the + running process as `unknown` and the ended one as it ended, both with `runner_lost`, and + refuses new work. A pipe process that does not write outlives its killed Runner, as nothing + ends it (one that writes meets a closed pipe); the operator finds it with `records list`, + cannot release its record while a gateway runs or while it runs, ends it, and releases the + record (with an authority, once DevGuard released its attempt and retired the lost Runner's + instance). A PTY's root is not named, so `release` cannot tell whether it runs: the operator + ends it first. A new Runner and gateway then take new work in that workspace.""" + runner = bench.runner() + checks = registered_checks(bench, runner) + g1 = bench.gateway('g1') + held = g1.exec('a', tagged(bench.roots['a'], 'held', ['/bin/sleep', '600']), tty=tty) + done = g1.exec('b', tagged(bench.roots['b'], 'done', ['/bin/sh', '-c', 'exit 5'])) + checks += [started_checks('the running process', held), started_checks('the ending process', done)] + if held['outcome'] != 'started' or done['outcome'] != 'started': + return checks, {'held': held, 'done': done} + wait_state(g1, done['process_id'], exited) + pid = tag_pid(bench.roots['a'], 'held') + attempt = attempt_of(g1, held['process_id']) + runner.kill() + status = wait_state(g1, held['process_id'], lambda s: s.get('state') == 'unknown', limit=30) + ended = status_of(g1, done['process_id']) + read, _ = g1.call('read_process', {'process_id': held['process_id'], 'cursor': 0}) + refused = wait_refusal(g1, 'c', 'RUNNER_DISCONNECTED') + alive = host.running(pid) + checks += [check('the running process is unknown, lost with its Runner', status.get('state') == 'unknown' + and status.get('lost_reason') == 'runner_lost', status=status), + check('the ended one is reported as it ended, with runner_lost', ended.get('state') == 'exited' + and ended.get('exit_code') == 5 and ended.get('lost_reason') == 'runner_lost', status=ended), + check('reading the running one says its output was lost with the Runner', + read.get('output_lost') is True and read.get('lost_reason') == 'runner_lost', read=read), + check('new work is refused with RUNNER_DISCONNECTED', refused_code(refused, 'RUNNER_DISCONNECTED'), + result=refused)] + if not tty: + checks.append(check('the pipe process outlived its killed Runner', alive, pid=pid)) + env = host.minimal_env(bench.work) + listed = host.records_command(bench.args.mcp_bin, bench.operations, env, 'list') + serving = host.records_command(bench.args.mcp_bin, bench.operations, env, 'release', held['process_id']) + bench.end(g1, 'eof') + running = None + if alive and not tty: + running = host.records_command(bench.args.mcp_bin, bench.operations, env, 'release', held['process_id']) + if host.pid_alive(pid): + os.kill(pid, signal.SIGKILL) + checks += [check('records list shows it, its Runner gone', held['process_id'] in + [entry['process_id'] for entry in ((listed['report'] or {}).get('records') or [])], + report=listed['report']), + check('release is refused while a gateway runs', not serving['done'], result=serving)] + if running is not None: + checks.append(check('release is refused while its root process runs', not running['done'] and + 'still runs' in json.dumps(running['report']), result=running)) + checks.append(check('the process ended once the operator ended it', gone(pid, limit=10), pid=pid)) + if bench.governed: + view = settled_view(bench, attempt, limit=60) + retired, instances = wait_retired(bench, runner.pid) + checks += [check("DevGuard released the lost process's attempt once its scope ended", + bool(view and not view['charged']), view=view), + check("DevGuard retired the lost Runner's instance", retired, instances=instances)] + released = host.records_command(bench.args.mcp_bin, bench.operations, env, 'release', held['process_id']) + checks.append(check('then the record is released', released['done'] and held['process_id'] in + ((released['report'] or {}).get('released') or []), result=released)) + fresh_runner = bench.runner('replacement') + checks += registered_checks(bench, fresh_runner) + g2 = bench.gateway('g2') + fresh, _, _ = once_reconciled(lambda: g2.exec('a', tagged(bench.roots['a'], 'fresh', ['/bin/echo', 'fresh']), tty=tty)) + fresh_checks, _, _ = ran(g2, fresh, 'fresh\n', 'new work in its workspace') + return checks + fresh_checks, {'status': status, 'ended': ended, 'pipe_outlived': alive, 'listed': listed['report']} + + +def recovery_runner_replaced(bench): + """Another Runner answers at the socket after the first was killed: restoring a Runner's + processes after it restarts is not supported, so the gateway does not take the new Runner + for the old one: it reports the old Runner's process `unknown` with `runner_lost` and refuses + new work; the next gateway keeps that workspace refused until an operator releases the + record, and takes new work elsewhere (P1R-C05).""" + runner = bench.runner() + g1 = bench.gateway('g1') + held = g1.exec('a', tagged(bench.roots['a'], 'held', ['/bin/sleep', '600'])) + checks = [started_checks('the process', held)] + if held['outcome'] != 'started': + return checks, {'held': held} + pid = tag_pid(bench.roots['a'], 'held') + runner.kill() + second = bench.runner('replacement') + status = wait_state(g1, held['process_id'], lambda s: s.get('state') == 'unknown', limit=30) + refused = wait_refusal(g1, 'b', 'RUNNER_DISCONNECTED') + bench.end(g1, 'eof') + if host.pid_alive(pid): + os.kill(pid, signal.SIGKILL) + g2 = bench.gateway('g2') + time.sleep(2.0) + blocked = g2.exec('a', tagged(bench.roots['a'], 'refused', ['/bin/echo', 'no'])) + elsewhere = g2.exec('b', ['/bin/echo', 'elsewhere']) + elsewhere_checks, _, _ = ran(g2, elsewhere, 'elsewhere\n', 'work elsewhere') + checks += [check('the gateway reports the process unknown, lost with its Runner', status.get('state') == 'unknown' + and status.get('lost_reason') == 'runner_lost', status=status), + check('and refuses new work', refused_code(refused, 'RUNNER_DISCONNECTED'), result=refused), + check("the next gateway keeps the lost process's workspace refused for an operator", + refused_code(blocked, 'RECOVERY_PENDING') and 'records' in (blocked.get('message') or ''), + result=blocked), + check('no refused command ran', 'refused' not in markers(bench.roots['a']), + markers=markers(bench.roots['a']))] + return checks + elsewhere_checks, {'status': status, 'blocked': blocked, 'second_runner': second.pid} + + +# What each unit's suite runs: (name, IO kinds or None, participations, function). +RECOVERY_CASES = { + 'recovery-lifecycle': [('exit-kinds', IO, PARTICIPATIONS, recovery_exit_kinds), + ('original-timeout', IO, PARTICIPATIONS, recovery_original_timeout), + ('runner-sigterm', None, PARTICIPATIONS, recovery_runner_sigterm), + ('invalid-settings', None, ('off',), recovery_invalid_settings)], + 'recovery-store': [('records', IO, PARTICIPATIONS, recovery_records), + ('writer-lock', None, ('off',), recovery_writer_lock), + ('schema', None, ('off',), recovery_schema), + ('records-command', None, ('off',), recovery_records_command)], + 'recovery-fence': [('foreign-connection', IO, PARTICIPATIONS, recovery_foreign_connection), + ('stale-epoch', None, ('off',), recovery_stale_epoch), + ('stopped-gateway', None, ('off',), recovery_stopped_gateway), + ('concurrent-claims', None, ('off',), recovery_concurrent_claims), + ('other-database', None, ('off',), recovery_other_database), + ('concurrent-gateways', None, ('off',), recovery_concurrent_gateways)], + 'recovery-reconcile': [('running', IO, PARTICIPATIONS, recovery_reconcile_running), + ('terminal', IO, PARTICIPATIONS, recovery_reconcile_terminal), + ('approval', None, PARTICIPATIONS, recovery_reconcile_approval), + ('authority-outage', None, ('required',), recovery_authority_outage), + ('missing-record', None, ('off',), recovery_missing_record), + ('kill-during-exec', IO, PARTICIPATIONS, recovery_kill_during_exec), + ('kill-during-terminate', IO, PARTICIPATIONS, recovery_kill_during_terminate)], + 'recovery-observation': [('ring-overflow', IO, ('off',), recovery_ring_overflow), + ('retention', None, ('off',), recovery_retention), + ('runner-loss', IO, PARTICIPATIONS, recovery_runner_loss), + ('runner-replaced', None, ('off',), recovery_runner_replaced)]} +# Gaps the functional suites state instead of passing over: why each cannot run here. +RECOVERY_GAPS = { + 'recovery-observation': [ + ('host-reboot', 'a host restart cannot be made during a run; CodeSpace tests it with a boot ID fixture ' + '(crates/server/src/mcp.rs), so a host restart stays unqualified on the Mac'), + ('runner-restart-restore', "restoring a Runner's processes after the Runner itself restarts is out of P1-RECOVERY's " + 'scope and unsupported: runner-replaced shows the gateway refusing to take a new Runner ' + 'for the old one')]} + + +def run_recovery_case(results, args, live, evidence, writer, suite, participation, io, name, function): + """One scenario on its own bench, torn down however it ends; `required` off macOS is not run.""" + label = f'{suite}/{name}' if io is None else f'{suite}/{name}-{io}' + if participation == 'required' and platform.system() != 'Darwin': + return run_scenario(results, evidence, 'independent', io, name, None, suite=suite, participation=participation, + not_run=NOT_NATIVE) + holder = {} + + def case(): + bench = holder['bench'] = recovery_bench(args, live, evidence, writer, participation, label) + return function(bench) if io is None else function(bench, io == 'pty') + try: + return run_scenario(results, evidence, 'independent', io, name, case, suite=suite, participation=participation) + finally: + if 'bench' in holder: + holder['bench'].teardown() + + +def recovery_functional(args, live, evidence, suites): + """The scenarios of `suites` for each chosen participation, then the gaps they state.""" + writer = host.Writer() + results = [] + try: + for suite in suites: + for name, ios, participations, function in RECOVERY_CASES[suite]: + for participation in args.participation: + if participation not in participations: + continue + for tty in (ios or (None,)): + run_recovery_case(results, args, live, evidence, writer, suite, participation, + None if tty is None else io_name(tty), name, function) + for name, reason in RECOVERY_GAPS.get(suite, []): + run_scenario(results, evidence, 'independent', None, name, None, suite=suite, not_run=reason) + finally: + evidence.write_json('functional-writer.json', writer.close()) + return results + + +def scenario_summary(results): + return [{key: entry.get(key) for key in ('suite', 'scenario', 'participation', 'io', 'verdict')} + for entry in results] + + +def recovery_unit(suite): + """The command of one unit's suite: its functional scenarios alone.""" + def run(args, live, evidence): + results = recovery_functional(args, live, evidence, [suite]) + return {'verdict': analysis.suite_verdict([entry['verdict'] for entry in results]), + 'scenarios': scenario_summary(results)} + return run + +# -------------------------------- recovery: the measurement (P1R-C06) +# The gateway restarted again and again while governed pipe and PTY processes run for a whole +# interval, idle and under DG-1's development load, against the installed service (D7): a fresh +# independent Runner per interval, registered as the codespace consumer's one instance (D12) and +# drained with its controller's Stop at the interval's end. Each restart ends the serving gateway +# in one of the four ways a gateway ends and starts the next, which must claim the Runner under +# the next epoch and find every long-running process running. Through whichever gateway serves, +# the control path is sampled as the control suite samples it; a slot that falls in a restart is +# recorded `restarting`, not as a sample, and the samples expected follow the time a gateway +# served. Every restart also has another process probe the epoch fence, a rival gateway start on +# the same operations database, an approval granted before it resumed after it (with a resume in +# flight when the gateway is killed), and a timeout target reach its deadline while no gateway +# runs or while one serves; now and then a burst overflows a pipe process's output ring while no +# gateway reads it. Without the installed service (Linux, or `off` only) the same run is a +# self-test of the harness, never a qualification. + +RECOVERY_PROTOCOL = {'idle_s': 600, 'load_s': 1800, 'repetitions': 3, 'combinations': ['independent']} +RECOVERY_REHEARSAL = {'idle_s': 120, 'load_s': 180, 'repetitions': 1, 'combinations': ['independent']} +RECOVERY_PERIODS = dict(PERIODS, read=2.0) +# Every gateway after the first starts its executions with this process timeout, so that each +# restart's timeout target reaches its deadline around the restart; the first starts the +# processes that run the whole interval with the control suite's. +CYCLE_TIMEOUT_S = 20 +FIRST_CYCLE_S = 20 +# A restart cycle's length as planned, from which follow the restarts an interval should hold. +PLANNED_CYCLE_S = 25 +# A timeout target whose deadline falls in the restart: its gateway ends this long before the +# deadline and the next one starts this long after that. Otherwise the gateway ends this long +# after the deadline and the next one starts at once. +IN_GAP_LEAD_S = 2.0 +GAP_S = 4.0 +CONNECTED_LAG_S = 3.0 +# The approval resumed after a restart is held and granted this long before it. +APPROVAL_LEAD_S = 2.0 +# A timeout target's end is awaited this long past its deadline. +TIMEOUT_WAIT_S = 10.0 +# Refusals a tagged command meets while capacity, a reconciliation or the workspace's last +# process stands in its way: it is tried again under a new tag. +RETRIED = ('RESOURCE_UNAVAILABLE', 'RECOVERY_PENDING', 'WORKSPACE_BUSY') +# The workspace and terminal of each restart's timeout target, in turn. +TIMEOUT_KINDS = (('timeout-a', False), ('timeout-off', False), ('timeout-a', True), ('timeout-off', True)) +RECOVERY_SAMPLES = analysis.SAMPLE_FILES + ('read_beat', 'read_burst') +RECOVERY_RECORDS = ('restarts', 'timeouts', 'approvals', 'execs') + + +def measurement_workspaces(work, governed, capacity=None): + """`status-pipe`, `status-pty`, `term`, `timeout-a` and `confirm` (which holds every execution + for approval) take part with `required` when the run is governed; `hog` asks for more than the + whole work capacity; `beat-pty`, `burst-pipe` and `timeout-off` never take part.""" + resources = host.required(**CONTROL_REQUEST) if governed else None + spaces = {name: (work / f'ws/{name}', resources) for name in ('status-pipe', 'status-pty', 'term', 'timeout-a')} + spaces['confirm'] = (work / 'ws/confirm', resources, {'approvals': 'confirm'}) + spaces.update({name: (work / f'ws/{name}', None) for name in ('beat-pty', 'burst-pipe', 'timeout-off')}) + if governed: + hog = dict(CONTROL_REQUEST, cpu_milli=((capacity or {}).get('cpu_milli') or 1_000_000) + 1) + spaces['hog'] = (work / 'ws/hog', host.required(**hog)) + return spaces + + +class Serving: + """The gateway that serves now. A sampled action enters while one serves; a restart closes the + gate and waits for the actions in flight to leave before it ends the gateway, and opens it + for the next one. The time the gate stood open is the time the interval was served.""" + + def __init__(self): + self.condition = threading.Condition() + self.gateway = None + self.inflight = 0 + self.opened = None + self.served_s = 0.0 + + def serve(self, gateway): + with self.condition: + self.gateway = gateway + self.opened = time.monotonic() + + def enter(self): + with self.condition: + if self.gateway is None: + return None + self.inflight += 1 + return self.gateway + + def leave(self): + with self.condition: + self.inflight -= 1 + self.condition.notify_all() + + def close(self, limit=30.0): + """No action enters any more; the actions still in flight after `limit` seconds.""" + with self.condition: + self.gateway = None + if self.opened is not None: + self.served_s += time.monotonic() - self.opened + self.opened = None + deadline = time.monotonic() + limit + while self.inflight and time.monotonic() < deadline: + self.condition.wait(max(0.0, deadline - time.monotonic())) + return self.inflight + + +def gated(serving, action): + """`action(gateway, slot)` through the gateway that serves; a slot while none does is + `restarting`.""" + def run(slot): + gateway = serving.enter() + if gateway is None: + return {'outcome': 'restarting'} + try: + return dict(action(gateway, slot), gateway=gateway.pid) + finally: + serving.leave() + return run + + +def status_sample(process_id): + def sample(gateway, slot): + payload, latency = gateway.status(process_id, timeout=10) + return {'outcome': 'answered', 'latency_ms': latency, 'state': payload.get('state')} + return sample + + +class GatedEcho: + """Echo through whichever gateway serves: the cursor is the Runner's, so each gateway reads on + from where the one before stopped.""" + + def __init__(self, process_id): + self.process_id, self.cursor = process_id, 0 + + def __call__(self, gateway, slot): + line = f'io-{slot}\n' + began = time.monotonic() + gateway.call('write_stdin', {'process_id': self.process_id, 'data': line}, timeout=10) + _, self.cursor, eof, found = gateway.read_until(self.process_id, line, limit=10, period=0.01, + cursor=self.cursor, timeout=10) + return {'outcome': 'echoed' if found else 'not_echoed', + 'latency_ms': round((time.monotonic() - began) * 1000, 3), 'eof': eof} + + +def read_sample(reader): + """A read of a long-running process's output on from the last one, kept whole for its + continuity (analysis.continuity_metrics).""" + def sample(gateway, slot): + return dict(reader.read(gateway, timeout=10), outcome='read') + return sample + + +class Execs: + """Every tagged command an interval sent, its tag and outcome kept as raw evidence + (`execs.jsonl`) against the workspaces' marker files (analysis.marker_metrics).""" + + def __init__(self, bench, writer, path): + self.bench, self.writer, self.path = bench, writer, path + + def run(self, gateway, workspace, tag, command, tty=False, retry=0.0, approval=False): + """exec_command of a tagged command. Refused for capacity, for a reconciliation or for its + workspace's last process, it is sent again under a new tag for up to `retry` seconds; the + result names the tag that was sent last.""" + deadline = time.monotonic() + retry + for attempt in itertools.count(): + name = tag if attempt == 0 else f'{tag}-{attempt}' + result = gateway.exec(workspace, tagged(self.bench.roots[workspace], name, command), tty=tty) + self.writer.write(self.path, {'tag': name, 'workspace': workspace, 'outcome': result['outcome'], + 'code': result.get('code'), 'process_id': result.get('process_id'), + 'approval': approval, 'gateway': gateway.pid, 'unix': time.time()}) + if result.get('code') not in RETRIED or time.monotonic() >= deadline: + return dict(result, tag=name) + time.sleep(0.2) + + +class GatedTerminations: + """Each slot starts a fresh target in `term` through the gateway that serves, alternating pipe + and PTY, terminates it and times it as Terminations does: a governed target's end by its + scope's process group, an ungoverned one's by its own PID, its release by the watcher.""" + + def __init__(self, bench, execs, watcher): + self.bench, self.execs, self.watcher = bench, execs, watcher + + def __call__(self, gateway, slot): + tty = bool(slot % 2) + started = self.execs.run(gateway, 'term', f'term-{slot}', ['/bin/sleep', '600'], tty=tty) + if started['outcome'] != 'started': + code = started.get('code') + return {'outcome': 'not_started' if code == 'RESOURCE_UNAVAILABLE' else 'error', 'code': code, + 'attempt_id': started.get('attempt_id'), 'latency_ms': started.get('latency_ms'), 'tty': tty} + process_id = started['process_id'] + entry = (gateway.execution(process_id) or {}) if self.bench.governed else {} + root, attempt = entry.get('scope_root_pid'), entry.get('attempt_id') + pid = tag_pid(self.bench.roots['term'], started['tag'], limit=5) + asked = time.monotonic() + ack = gateway.terminate(process_id, timeout=10) + if attempt and self.watcher is not None: + self.watcher.watch(slot, attempt, asked) + status = gateway.wait_exited(process_id, limit=30, period=0.02) + exited = round((time.monotonic() - asked) * 1000, 3) if status else None + ended = host.wait_group_gone(root, limit=30) is not None if root else bool(pid) and gone(pid, limit=30) + return {'outcome': 'acknowledged', 'tty': tty, 'ack_ms': ack, 'termination': (status or {}).get('termination'), + 'exited_ms': exited, 'scope_exit_ms': round((time.monotonic() - asked) * 1000, 3) if ended else None, + 'attempt_id': attempt, 'scope_root_pid': root, 'pid': pid} + + +def gated_admission(execs): + """A governed execution in `hog`, which the authority must refuse for capacity; a command it + refused must never run (its tag never in the marker file).""" + def sample(gateway, slot): + result = execs.run(gateway, 'hog', f'hog-{slot}', ['/usr/bin/true']) + row = {'latency_ms': result.get('latency_ms'), 'code': result.get('code'), 'attempt_id': result.get('attempt_id')} + if result['outcome'] == 'started': + row.update(outcome='admitted', attempt_id=attempt_of(gateway, result['process_id'])) + gateway.wait_exited(result['process_id'], limit=30) + else: + row['outcome'] = 'refused' if result['outcome'] == 'refused' else 'error' + return row + return sample + + +def runner_footprint(runner, serving): + def sample(): + gateway = serving.gateway + return {'owner': host.process_footprint(runner.pid), + 'gateway': host.process_footprint(gateway.pid) if gateway is not None else None} + return sample + + +class Rival: + """A second gateway started on the serving gateway's operations database: it must wait for the + writer and give up, never reaching the Runner. One still waiting when its cycle ends is + killed before the serving gateway ends, so that it never takes the writer's place.""" + + def __init__(self, bench, name): + self.log_path = bench.evidence.file(f'{bench.prefix}/{name}.gateway.log') + argv = [str(bench.args.mcp_bin), *bench.flags('detach')] + with open(self.log_path, 'ab') as log: + self.process = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=log, + env=bench.environment(CYCLE_TIMEOUT_S), start_new_session=True) + bench.live.workers.append(self.process) + self.pid = self.process.pid + + def finish(self, runner, limit=0.0): + killed = False + try: + self.process.wait(timeout=max(0.0, limit)) + except subprocess.TimeoutExpired: + killed = True + self.process.kill() + self.process.wait() + try: + self.process.stdin.close() + except OSError: + pass + text = self.log_path.read_text(errors='replace') + return {'pid': self.pid, 'returncode': self.process.returncode, 'killed': killed, + 'another_writer': 'has another writer' in text, + 'claims': [claim for claim in runner.claims() if claim['pid'] == self.pid]} + + +class Gateways: + """The attempts and launches all of an interval's gateways logged, as one gateway's.""" + + def __init__(self, gateways): + self.gateways = gateways + + def attempts(self): + return sorted({attempt for gateway in self.gateways for attempt in gateway.attempts()}) + + def launches(self): + found = {} + for gateway in self.gateways: + for process_id, count in gateway.launches().items(): + found[process_id] = found.get(process_id, 0) + count + return found + + +def recovery_journal(database, gateways, rows, launched_before): + """journal_findings over every gateway of an interval. A launch the Runner made for a gateway + killed before it logged it is in the journal and in no log, so the difference is reported, + not counted as a duplicate: the marker files judge whether a command ran twice.""" + merged = Gateways(gateways) + found = journal_findings(database, merged, rows, launched_before) + if 'error' not in found: + found['duplicate_launches'] = sum(1 for count in merged.launches().values() if count > 1) + found['unlogged_launches'] = max(0, (found['launched_in_journal'] or 0) - found['launched_in_log']) + return found + + +class RecoveryInterval: + """One interval of the measurement; see the section's comment.""" + + def __init__(self, context, kind, seconds, key): + self.context, self.kind, self.seconds, self.key = context, kind, seconds, key + self.args, self.live, self.evidence = context['args'], context['live'], context['evidence'] + self.governed = context['governed'] + self.prefix = f'{key}/{kind}' + self.raw = self.evidence.file(self.prefix + '/x').parent + self.record = {'kind': kind, 'combination': context['combination'], 'governed': self.governed, 'invalid': [], + 'planned_restarts': max(1, int(seconds / PLANNED_CYCLE_S))} + self.invalid = self.record['invalid'] + self.writer = host.Writer() + self.serving = Serving() + self.watch = host.PidWatch() + self.bench = self.runner = self.gateway = self.execs = self.cue = None + self.serving_ms = None + self.targets, self.readers = {}, {} + self.pending_timeouts, self.pending_approval = [], None + self.number = 0 + self.store_id = None + + # ---------------------------------------------------------- the interval + + def run(self): + record, invalid, args = self.record, self.invalid, self.args + launched_before = self.journal_before() + directory = scratch() + capacity = host.work_capacity(args.service_log) if self.governed else None + authority = installed_authority(args) if self.governed else dict(NO_AUTHORITY) + self.bench = RecoveryBench(args, self.live, self.evidence, self.writer, directory, + measurement_workspaces(directory, self.governed, capacity), authority, self.prefix) + self.execs = Execs(self.bench, self.writer, self.raw / 'execs.jsonl') + self.cue = self.bench.work / 'cue' + schedules, samplers, load, watcher, rows, sampled = {}, {}, [], None, {}, {} + try: + self.runner = self.bench.runner() + if self.governed: + record['instance'], _ = wait_registered(self.bench, self.runner) + if record['instance'] is None: + invalid.append("the Runner did not register as the consumer's instance") + if not invalid: + self.start_gateway(CONTROL_TIMEOUT_S) + self.start_targets() + if not invalid: + watcher = ReleaseWatcher(self.bench, self.writer, self.raw / 'releases.jsonl') if self.governed else None + if watcher is not None: + watcher.start() + begin = time.monotonic() + 0.5 + schedules = self.schedules(watcher, begin) + samplers = self.samplers() + if self.kind == 'load' and self.context.get('measure') is not None: + load = self.start_load(begin) + until(begin) + self.serving.serve(self.gateway) + record['started_unix'] = time.time() + for thread in list(schedules.values()) + list(samplers.values()): + thread.start() + for consumer in load: + consumer.start() + self.end_at = begin + self.seconds + self.cycles() + until(self.end_at) + if load: + self.join_load(load, begin) + record['ended_unix'] = time.time() + record['seconds'] = round(time.monotonic() - begin, 3) + rows = {name: schedules[name].stop() if name in schedules else [] for name in RECOVERY_SAMPLES + if name != 'releases'} + sampled = {name: sampler.stop() for name, sampler in samplers.items()} + self.serving.close() + record['served_s'] = round(self.serving.served_s, 3) + self.conclude_targets() + rows['releases'] = watcher.finish(limit=60) if watcher is not None else [] + self.end_last() + record['drain'] = self.drain() + except BaseException: + for thread in list(schedules.values()) + list(samplers.values()) + ([watcher] if watcher else []): + thread.halt.set() + if load: + self.context['measure'].settle_consumers(load) + self.watch.stop() + self.writer.close(limit=10) + self.bench.teardown() + raise + self.watch.stop() + self.evidence.write_json(self.prefix + '/markers.json', + {name: markers(root) for name, root in self.bench.roots.items()}) + self.bench.teardown() + record.setdefault('served_s', round(self.serving.served_s, 3)) + if self.governed and 'started_unix' in record: + record['journal'] = recovery_journal(args.journal, self.bench.gateways, rows, launched_before) + if 'error' in record['journal']: + invalid.append(f"the journal could not be read: {record['journal']['error']}") + runs = [run for consumer in load for run in consumer.runs] + if load: + self.load_record(load, runs) + written = self.writer.close() + record['evidence_writer'] = written + if written['error_count']: + invalid.append(f"raw evidence could not be written: {written['errors'][:3]}") + sampled = {name: read_rows(self.raw, name) for name in ('host', 'service', 'footprint')} + release, service_pid = self.context.get('release'), self.context.get('service_pid') + if self.governed and 'started_unix' in record: + service_invalid, _ = analysis.service_findings(sampled.get('service') or [], release.id, service_pid) + invalid += service_invalid + record['raw'] = raw_hashes(self.raw) + if 'started_unix' in record: + records = {name: read_rows(self.raw, name) for name in RECOVERY_RECORDS} + found = read_json(self.raw / 'markers.json') + metrics = recovery_interval_metrics(record, {name: read_rows(self.raw, name) for name in RECOVERY_SAMPLES}, + sampled, records, found, runs, release.id if release else None, + service_pid, self.context.get('reservation')) + record['verdict'], record['reasons'] = analysis.recovery_interval_verdict(self.kind, invalid, metrics) + record['metrics'] = metrics + else: + record['verdict'], record['reasons'] = 'inconclusive', list(invalid) or ['the interval did not begin'] + self.evidence.write_json(self.prefix + '/interval.json', record) + print(f"qualify: {self.key} {self.kind}: {record['verdict']} {record['reasons']}", flush=True) + return record + + def journal_before(self): + if not self.governed: + return None + try: + before = host.journal_charged(self.args.journal) + launched = launched_count(self.args.journal) + active = [entry for entry in host.journal_instances(host.CONSUMER, self.args.journal) + if entry['state'] == 'active'] + except Exception as error: # noqa: BLE001 - an unreadable journal invalidates the interval + self.record['journal_before'] = {'error': str(error)} + self.invalid.append(f'the journal could not be read: {error}') + return None + self.record['journal_before'] = dict(before, active_instances=active) + if sum(before['attempts'].values()) or before['leases']: + self.invalid.append("work other than the harness's was charged when the interval began") + if active: + self.invalid.append('a codespace instance was active when the interval began') + return launched + + def start_gateway(self, timeout_s): + """The next gateway, serving once initialized; the time that took.""" + self.number += 1 + began = time.monotonic() + self.gateway = self.bench.gateway(f'g{self.number:03d}', timeout_s=timeout_s) + self.serving_ms = round((time.monotonic() - began) * 1000, 3) + return self.gateway + + def start_targets(self): + """The processes that run the whole interval, started by the first gateway: a `cat` (status + and I/O) and a PTY `sleep` (status), governed when the run is, and an ungoverned PTY beat + and pipe burst, whose output is read across every restart.""" + commands = {'status-pipe': (['/bin/cat'], False), 'status-pty': (['/bin/sleep', '86400'], True), + 'beat-pty': (BEAT, True), 'burst-pipe': (burst(self.cue), False)} + for name, (command, tty) in commands.items(): + started = self.execs.run(self.gateway, name, name, command, tty=tty, retry=30.0) + if started['outcome'] != 'started': + self.invalid.append(f"{name} could not be started: {started.get('code')}") + continue + process_id = started['process_id'] + governed = self.bench.requests.get(name) is not None + self.targets[name] = { + 'process_id': process_id, 'tag': started['tag'], 'tty': tty, 'governed': governed, + 'pid': tag_pid(self.bench.roots[name], started['tag']), + 'scope_root_pid': (self.gateway.execution(process_id) or {}).get('scope_root_pid') if governed else None} + self.record['long_running'] = self.targets + self.readers = {name: OutputReader(self.targets[name]['process_id']) for name in ('beat-pty', 'burst-pipe') + if name in self.targets} + + def schedules(self, watcher, begin): + writer, raw, serving, targets = self.writer, self.raw, self.serving, self.targets + periods = RECOVERY_PERIODS + found = { + 'status_pipe': Schedule(writer, raw / 'status_pipe.jsonl', + gated(serving, status_sample(targets['status-pipe']['process_id'])), begin, + period=periods['status']), + 'status_pty': Schedule(writer, raw / 'status_pty.jsonl', + gated(serving, status_sample(targets['status-pty']['process_id'])), begin, + period=periods['status']), + 'io': Schedule(writer, raw / 'io.jsonl', gated(serving, GatedEcho(targets['status-pipe']['process_id'])), + begin + 0.25, period=periods['io']), + 'terminations': Schedule(writer, raw / 'terminations.jsonl', + gated(serving, GatedTerminations(self.bench, self.execs, watcher)), + begin + periods['termination'] / 2, period=periods['termination']), + 'read_beat': Schedule(writer, raw / 'read_beat.jsonl', gated(serving, read_sample(self.readers['beat-pty'])), + begin + 0.4, period=periods['read']), + 'read_burst': Schedule(writer, raw / 'read_burst.jsonl', + gated(serving, read_sample(self.readers['burst-pipe'])), begin + 1.4, + period=periods['read'])} + if self.governed: + found['refusals'] = Schedule(writer, raw / 'refusals.jsonl', gated(serving, gated_admission(self.execs)), + begin + periods['refusal'] / 3, period=periods['refusal']) + return found + + def samplers(self): + found = {'host': host.Sampler(self.writer, self.raw / 'host.jsonl', HOST_PERIOD_S, host.host_sample), + 'footprint': host.Sampler(self.writer, self.raw / 'footprint.jsonl', FOOTPRINT_PERIOD_S, + runner_footprint(self.runner, self.serving))} + if self.governed: + found['service'] = host.Sampler(self.writer, self.raw / 'service.jsonl', SERVICE_PERIOD_S, + self.context['release'].state) + return found + + def start_load(self, begin): + measure, args = self.context['measure'], self.args + (self.raw / 'runs').mkdir() + return measure.consumers(SimpleNamespace(qualify_bin=Path(args.qualify_bin)), self.context['dg_release'], + args.load, self.context['work'], self.raw / 'runs', begin + self.seconds, self.key, + args.rehearsal) + + def join_load(self, load, begin): + measure = self.context['measure'] + limit = begin + self.seconds + measure.LOAD_COMPLETION_LIMIT_S + for consumer in load: + consumer.join(timeout=max(0.0, limit - time.monotonic())) + if any(consumer.is_alive() for consumer in load): + self.invalid.append('the load did not complete within the limit') + measure.settle_consumers(load) + + def load_record(self, load, runs): + measure, record = self.context['measure'], self.record + (self.raw / 'runs.json').write_text(json.dumps(runs, indent=1) + '\n') + record['consumers'] = [consumer.name for consumer in load] + errors = {consumer.name: consumer.errors for consumer in load if consumer.errors} + record['load_issues'] = ['a consumer stopped with an error'] if errors else [] + record['consumer_errors'] = errors + ids = {attempt for run in runs for attempt in run.get('attempt_ids') or []} + record['load_journal'] = measure.settle_journal(ids, runs) + record['load_failures'] = (['load attempts still charged after the interval'] + if record['load_journal'].get('still_charged') else []) + + # ---------------------------------------------------------- restarts + + def cycles(self): + """Restart after restart while a whole cycle still fits in the interval; the last gateway + serves to its end.""" + while True: + restart = self.end_at - time.monotonic() > PLANNED_CYCLE_S + 5 + if not self.cycle(restart) or not restart: + return + + def cycle(self, restart): + """The serving gateway's cycle: its Claim and what it inherited, the fence probe and a + rival, the approval granted before the restart resumed, the timeout target that ended + around the restart concluded; then, when it restarts, the next timeout target started, + the next approval held and granted, the gateway ended as this cycle's number says and + the next one started after the gap. Whether a gateway serves afterwards.""" + number, gateway = self.number, self.gateway + row = {'cycle': number, 'gateway': gateway.pid, 'expected_epoch': number, 'serving_ms': self.serving_ms, + 'restart': restart} + rival = None + end_at = time.monotonic() + PLANNED_CYCLE_S + try: + row['claim'] = claimed_by(self.runner, gateway.pid) + if number > 1: + row['inherited'] = {name: status_of(gateway, target['process_id']).get('state') + for name, target in self.targets.items()} + row['fence'] = self.fence(number) + rival = Rival(self.bench, f'{number:03d}-rival') + if self.pending_approval is not None: + self.resume_approval(gateway) + self.conclude_timeouts(gateway) + if not restart: + until(self.end_at) + return True + if number == 1: + end_at, gap, target = time.monotonic() + FIRST_CYCLE_S, 0.0, None + row['timeout'] = {'alignment': None} + else: + end_at, gap, target = self.timeout_target(gateway, number, row) + until(end_at - APPROVAL_LEAD_S) + approval = self.hold_approval(gateway, number) + until(end_at) + how = host.GATEWAY_ENDS[number % len(host.GATEWAY_ENDS)] + row['how'] = how + row['inflight_at_close'] = self.serving.close() + row['rival'] = rival.finish(self.runner) + rival = None + if row['timeout'].get('alignment') == 'in_gap' and (number // 8) % 4 == number % 4: + self.cue.touch() + row['burst'] = True + flight = self.resume_in_flight(gateway, approval, number) if how == 'sigkill' and approval else None + row['ended'] = self.bench.end(gateway, how) + if target is not None: + target['gateway_ended_unix'] = time.time() + if flight is not None: + thread, result, delay = flight + thread.join(timeout=30) + self.pending_approval.update(killed_during_resume=True, kill_delay_s=delay, in_flight=dict(result)) + time.sleep(gap) + serves = self.next_gateway(row) + if serves and target is not None: + target['next_served_unix'] = time.time() + return serves + except Exception as error: # noqa: BLE001 - a cycle that breaks is evidence, and the run goes on + row['error'] = f'{type(error).__name__}: {error}' + row['trace'] = traceback.format_exc()[-2000:] + until(min(end_at, self.end_at)) + if self.gateway is gateway and gateway.process.poll() is None: + self.serving.serve(gateway) + return True + return self.serving.gateway is not None or self.next_gateway(row) + finally: + if rival is not None: + row['rival'] = rival.finish(self.runner, limit=30) + self.writer.write(self.raw / 'restarts.jsonl', row) + + def next_gateway(self, row): + """The next gateway, serving; tried twice. Whether one serves.""" + for attempt in range(2): + try: + self.start_gateway(CYCLE_TIMEOUT_S) + self.serving.serve(self.gateway) + return True + except Exception as error: # noqa: BLE001 - recorded with the restart + row.setdefault('start_errors', []).append(f'{type(error).__name__}: {error}') + row['error'] = row.get('error') or 'the next gateway did not start' + time.sleep(2.0) + return False + + def fence(self, epoch): + """Another process of the same user at the Runner's socket while the gateway controls it: + its Claim, and its requests that would change a long-running process or the Runner under + the previous, the current and no epoch.""" + targets = {'write': self.targets['status-pipe']['process_id'], + 'resize': self.targets['status-pty']['process_id'], + 'terminate': self.targets['beat-pty']['process_id'], 'workspace': 'beat-pty'} + try: + client = host.WireClient(self.bench.socket, timeout=10) + try: + claim = client.claim(self.store()) + requests = fence_requests(client, targets, (epoch - 1, epoch, None)) + finally: + client.close() + except (OSError, ValueError) as error: + return {'claim': None, 'requests': [], 'error': f'{type(error).__name__}: {error}'} + refusal = (claim.get('data') or {}).get('refusal') if claim.get('type') == 'claim_refused' else claim.get('type') + return {'claim': refusal, 'requests': requests} + + def store(self): + if self.store_id is None: + self.store_id = host.store_id(self.bench.operations) + return self.store_id + + def timeout_target(self, gateway, number, row): + """This restart's timeout target, when the gateway ends and the gap before the next: the + gateway ends before the deadline, so that it falls while no gateway runs (`in_gap`), or + after it (`connected`). A target that could not be started leaves the restart a fixed + length.""" + workspace, tty = TIMEOUT_KINDS[(number + number // 8) % len(TIMEOUT_KINDS)] + alignment = 'in_gap' if (number // 4) % 2 else 'connected' + started = self.execs.run(gateway, workspace, f'timeout-{number:03d}', ['/bin/sleep', '600'], tty=tty, + retry=10.0) + row['timeout'] = {'tag': started['tag'], 'workspace': workspace, 'tty': tty, 'alignment': alignment, + 'outcome': started['outcome'], 'code': started.get('code')} + if started['outcome'] != 'started': + return time.monotonic() + CYCLE_TIMEOUT_S, 0.0, None + deadline = deadline_of(self.bench, started['process_id']) + pid = tag_pid(self.bench.roots[workspace], started['tag']) + if pid: + self.watch.watch(started['tag'], pid) + target = {'tag': started['tag'], 'cycle': number, 'workspace': workspace, 'tty': tty, + 'process_id': started['process_id'], 'pid': pid, 'deadline_at_ms': deadline, 'alignment': alignment} + self.pending_timeouts.append(target) + if not deadline: + return time.monotonic() + CYCLE_TIMEOUT_S, 0.0, target + remaining = deadline / 1000 - time.time() + if alignment == 'in_gap': + return time.monotonic() + remaining - IN_GAP_LEAD_S, GAP_S, target + return time.monotonic() + remaining + CONNECTED_LAG_S, 0.0, target + + def conclude_timeouts(self, gateway, final=False): + """Each timeout target whose deadline has passed: when its process was seen gone, how the + gateway that serves now reports it, and the deadline its record holds now.""" + for entry in list(self.pending_timeouts): + deadline = entry['deadline_at_ms'] + if not final and deadline and time.time() < deadline / 1000: + continue + self.pending_timeouts.remove(entry) + limit = max(0.0, (deadline or 0) / 1000 + TIMEOUT_WAIT_S - time.time()) + gone_unix = self.watch.wait(entry['tag'], limit) if entry['pid'] else None + status = wait_state(gateway, entry['process_id'], exited, limit=15) + later = (record_of(self.bench, entry['process_id']) or {}).get('deadline_at_ms') + ended, served = entry.get('gateway_ended_unix'), entry.get('next_served_unix') + entry.update(gone_unix=gone_unix, termination=status.get('termination'), status_error=status.get('error'), + deadline_after_ms=later, reported_by=gateway.pid, + in_gap=bool(deadline and ended and served and ended < deadline / 1000 < served)) + self.writer.write(self.raw / 'timeouts.jsonl', entry) + + def hold_approval(self, gateway, number): + """An approval held and granted before the restart, for the next gateway to resume.""" + held = self.execs.run(gateway, 'confirm', f'approval-{number:03d}', ['/bin/sh', '-c', 'exit 0'], retry=5.0, + approval=True) + approval = held.get('approval_id') + entry = {'tag': held['tag'], 'cycle': number, 'approval_id': approval, 'held': held.get('code')} + if approval: + try: + gateway.call('approval_resolve', {'approval_id': approval, 'decision': 'grant'}) + except ToolError as error: + entry['grant_error'] = error.code + if approval and 'grant_error' not in entry: + self.pending_approval = entry + return approval + self.writer.write(self.raw / 'approvals.jsonl', entry) + return None + + def resume_in_flight(self, gateway, approval, number): + """The approval resumed in a thread, the gateway then killed after a delay of the range.""" + delay = KILL_DELAYS_S[(number // 4) % len(KILL_DELAYS_S)] + result = {} + thread = threading.Thread(target=lambda: result.update(resume_result(gateway, approval, timeout=30)), + daemon=True) + thread.start() + time.sleep(delay) + return thread, result, delay + + def resume_approval(self, gateway): + """The approval granted before this restart, resumed once this gateway reconciled its + workspace, and resumed again: the same process both times, its command run once.""" + entry, self.pending_approval = self.pending_approval, None + approval = entry['approval_id'] + first, pending, waited = once_reconciled(lambda: resume_result(gateway, approval)) + deadline = time.monotonic() + 10 + while first.get('code') == 'RESOURCE_UNAVAILABLE' and time.monotonic() < deadline: + time.sleep(0.5) + first = resume_result(gateway, approval) + again = resume_result(gateway, approval) if first.get('process_id') else None + if first.get('process_id'): + wait_state(gateway, first['process_id'], exited) + entry.update(resumed_by=gateway.pid, resume=first, again=again, pending_refusals=pending, ready_ms=waited, + not_admitted=first.get('code') == 'RESOURCE_UNAVAILABLE') + self.writer.write(self.raw / 'approvals.jsonl', entry) + + # ---------------------------------------------------------- the interval's end + + def conclude_targets(self): + """Through the last gateway: the final reads, the pending approval and timeout target, then + every long-running process terminated, effective when reported terminated and its process + (a governed one's scope) gone.""" + gateway = self.gateway + if gateway is None or gateway.process.poll() is not None: + self.record['targets'] = [] + return + for name, path in (('beat-pty', 'read_beat'), ('burst-pipe', 'read_burst')): + try: + row = self.readers[name].read(gateway, timeout=10) + self.writer.write(self.raw / f'{path}.jsonl', dict(row, slot='final', outcome='read', + gateway=gateway.pid)) + except ToolError as error: + self.writer.write(self.raw / f'{path}.jsonl', {'slot': 'final', 'outcome': 'error', 'code': error.code}) + if self.pending_approval is not None: + self.resume_approval(gateway) + self.conclude_timeouts(gateway, final=True) + self.record['targets'] = [self.end_target(gateway, name) for name in self.targets] + + def end_target(self, gateway, name): + target = self.targets[name] + try: + ack = gateway.terminate(target['process_id'], timeout=10) + status = gateway.wait_exited(target['process_id'], limit=30) + except ToolError as error: + return {'name': name, 'process_id': target['process_id'], 'error': error.code, 'effective': False} + if target['scope_root_pid']: + ended = host.wait_group_gone(target['scope_root_pid'], limit=30) is not None + else: + ended = bool(target['pid']) and gone(target['pid'], limit=30) + termination = (status or {}).get('termination') + return {'name': name, 'process_id': target['process_id'], 'ack_ms': ack, 'termination': termination, + 'scope_gone': ended, 'effective': termination == 'terminated' and ended} + + def end_last(self): + """The last gateway ends with its input closed and detaches, as the others did.""" + gateway = self.gateway + if gateway is None or gateway.process.poll() is not None: + return + ended = self.bench.end(gateway, 'eof') + self.writer.write(self.raw / 'restarts.jsonl', { + 'cycle': self.number, 'gateway': gateway.pid, 'last': True, 'how': 'eof', 'ended': ended}) + + def drain(self): + """The Runner drained as an operator drains it: a gateway with the exit intent `stop` + claims it under the next epoch, and its input closed stops the Runner, which exits 0 and + removes its socket; with an authority, its instance retires and nothing of the consumer + stays charged.""" + drain = {'governed': self.governed, 'expected_epoch': self.number + 1} + try: + gateway = self.bench.gateway('drain', intent='stop', timeout_s=CYCLE_TIMEOUT_S) + claim = claimed_by(self.runner, gateway.pid) + drain['claim_epoch'] = claim['epoch'] if claim else None + drain['gateway'] = self.bench.end(gateway, 'eof') + except Exception as error: # noqa: BLE001 - a failed drain is evidence + drain['error'] = f'{type(error).__name__}: {error}' + drain['runner_exit'] = self.runner.wait(30) + drain['socket_left'] = self.bench.socket.exists() + found = re.findall(r'stopped; (\d+) processes were terminated', self.runner.log()) + drain['terminated'] = int(found[-1]) if found else None + if self.governed: + drain['retired'], drain['instances'] = wait_retired(self.bench, self.runner.pid) + deadline = time.monotonic() + 60 + while True: + charged = host.journal_charged(self.args.journal)['attempts'].get(host.CONSUMER, 0) + if not charged or time.monotonic() >= deadline: + break + time.sleep(1.0) + drain['charged'] = charged + return drain + + +def recovery_interval_metrics(record, rows, sampled, records, found, runs, release_id, service_pid, reservation): + """A recovery interval's metrics from its raw files, as the run computes them and `verify` + recomputes them: the control path over the time a gateway served, and the recovery evidence.""" + governed = record['governed'] + metrics = analysis.control_metrics(rows, record['served_s'], RECOVERY_PERIODS, release_expected=governed) + targets = record.get('targets') or [] + metrics['termination']['status_target_effective'] = (all(t.get('effective') for t in targets) + if targets else None) + if governed: + _, failure = analysis.service_findings(sampled.get('service') or [], release_id, service_pid) + metrics['service'] = {'rows': len(sampled.get('service') or []), 'failure': failure} + footprints = sampled.get('footprint') or [] + metrics['host'] = analysis.host_peaks(sampled.get('host') or []) + metrics['footprint'] = { + 'runner': analysis.footprint_metrics([r['owner'] for r in footprints if r.get('owner')], reservation), + 'gateway': analysis.footprint_metrics([r['gateway'] for r in footprints if r.get('gateway')], None)} + metrics['journal'] = record.get('journal') or {} + if record['kind'] == 'load': + if record.get('consumers') is None: + metrics['load'] = {'failures': [], 'issues': ['no load ran: there is no DevGuard service to run it']} + else: + metrics['load'] = analysis.load_metrics(runs, record['started_unix'], record['ended_unix'], + record['consumers']) + metrics['load']['issues'] += record.get('load_issues') or [] + metrics['load']['failures'] += record.get('load_failures') or [] + metrics['recovery'] = { + 'restarts': analysis.restart_metrics([row for row in records['restarts'] if not row.get('last')]), + 'timeouts': analysis.timeout_metrics(records['timeouts']), + 'approvals': analysis.approval_metrics(records['approvals'], found), + 'markers': analysis.marker_metrics(found, records['execs']), + 'continuity': {'beat_pty': analysis.continuity_metrics(rows['read_beat'], 'beat'), + 'burst_pipe': analysis.continuity_metrics(rows['read_burst'], 'burst')}, + 'last_gateway': next((row.get('ended') for row in records['restarts'] if row.get('last')), None), + 'drain': record.get('drain'), 'planned_restarts': record['planned_restarts']} + return metrics + + +def recovery_interval(context, kind, seconds, key): + return RecoveryInterval(context, kind, seconds, key).run() + + +def recovery_repetition_outcome(report, idle, load): + """A repetition's verdict: as a control repetition's when governed, otherwise from its two + intervals alone.""" + if report.get('governed'): + return repetition_outcome(report, idle, load) + verdict, reasons = analysis.repetition_verdict(idle, load) + if report['rehearsal']: + verdict, reasons = 'inconclusive', reasons + ['rehearsal durations are below the protocol'] + return verdict, reasons + + +def recovery_repetition(context, plan, repetition): + args, evidence, governed, release = (context[name] for name in ('args', 'evidence', 'governed', 'release')) + key = f"{context['combination']}-{repetition}" + report = {'schema': 'codespace-recovery-repetition/v1', 'combination': context['combination'], + 'repetition': repetition, 'rehearsal': args.rehearsal, 'governed': governed, + 'load': args.load if governed else None, 'service_before': release.state() if governed else None} + + def conclude(outcome): + report['verdict'], report['reasons'] = outcome + evidence.write_json(f'{key}/report.json', report) + print(f"qualify: {key} {report['verdict']} {report['reasons']}", flush=True) + return report + + if governed: + context['service_pid'] = report['service_before']['service_pid'] + if report['service_before']['runs_release'] and args.load == 'warm': + report['prewarm'] = context['measure'].prewarm(SimpleNamespace(qualify_bin=Path(args.qualify_bin)), + context['dg_release'], context['work'], + evidence.file(f'{key}/x').parent) + early = early_outcome(report) + if early: + report['service_after'] = release.state() + return conclude(early) + idle = recovery_interval(context, 'idle', plan['idle_s'], key) + load = recovery_interval(context, 'load', plan['load_s'], key) + report['intervals'] = {'idle': {k: idle[k] for k in ('verdict', 'reasons')}, + 'load': {k: load[k] for k in ('verdict', 'reasons')}} + if governed: + for target in sorted((context['work'] / 'targets.noindex').glob(f'cold-{key}-*')): + shutil.rmtree(target, ignore_errors=True) + report['service_after'] = release.state() + return conclude(recovery_repetition_outcome(report, idle, load)) + + +def active_instances(args): + return [entry for entry in host.journal_instances(host.CONSUMER, args.journal) if entry['state'] == 'active'] + + +def recovery_measurement(args, live, evidence): + """The measurement: governed against the installed service on macOS with `required`, a + self-test of the harness otherwise.""" + governed = platform.system() == 'Darwin' and 'required' in args.participation + combination = 'independent' if governed else 'independent-off' + plan = dict(RECOVERY_REHEARSAL if args.rehearsal else RECOVERY_PROTOCOL, combinations=[combination], + load=args.load if governed else None, rehearsal=args.rehearsal) + if args.repetitions is not None: + plan['repetitions'] = args.repetitions + context = {'args': args, 'live': live, 'evidence': evidence, 'governed': governed, 'combination': combination, + 'release': None, 'measure': None, 'reservation': None} + run = {'schema': 'codespace-recovery-run/v1', 'argv': sys.argv, 'plan': plan, 'protocol': RECOVERY_PROTOCOL, + 'periods': RECOVERY_PERIODS, 'request': CONTROL_REQUEST, 'governed': governed, + 'cycle': {'timeout_s': CYCLE_TIMEOUT_S, 'first_s': FIRST_CYCLE_S, 'planned_s': PLANNED_CYCLE_S, + 'gap_s': GAP_S, 'in_gap_lead_s': IN_GAP_LEAD_S, 'connected_lag_s': CONNECTED_LAG_S, + 'ends': list(host.GATEWAY_ENDS), 'kill_delays_s': list(KILL_DELAYS_S)}, + 'targets': {'status_p99_ms': analysis.STATUS_P99_MS, 'termination_ack_p99_ms': analysis.TERMINATION_P99_MS}, + 'host': host.host_facts(), 'harness': {'commit': git('rev-parse', 'HEAD'), 'dirty': dirty()}} + if governed: + measure = load_measure(args.devguard_checkout) + live.measure = measure + release = host.Release(args.release) + problems = control_preconditions(args, release, measure) + if active_instances(args): + problems.append('a codespace instance is active: drain the independent Runner or stop the gateway first') + work = Path(args.work).resolve() + if work.exists(): + problems.append(f'{work} already exists') + if problems: + return {'verdict': 'inconclusive', 'problems': problems} + work.mkdir(parents=True) + manifest = json.loads(release.manifest.read_text()) + measure.export_source(manifest['source']['commit'], work) + reservation = consumer_settings(Path(args.host_config).read_text()).get('control_reservation') + context.update(measure=measure, release=release, dg_release=measure.Release(args.release), work=work, + reservation=reservation) + run.update(release={'id': release.id, **release.artifacts(), 'state': release.state()}, + policy=policy(release, args), work_capacity=host.work_capacity(args.service_log), + reservation=reservation) + run['harness'].update(measure_sha256=host.sha256(Path(measure.__file__)), + devguard_checkout=git('rev-parse', 'HEAD', root=measure.ROOT), + qualify_sha256=host.sha256(args.qualify_bin), rustc=host.run_text('rustc', '--version')) + run['started_unix'] = time.time() + evidence.write_json('run.json', run) + awake = None + if platform.system() == 'Darwin': + # The display and the system stay awake for the run; caffeinate ends with this process. + awake = subprocess.Popen(['/usr/bin/caffeinate', '-d', '-i', '-w', str(os.getpid())], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + live.workers.append(awake) + try: + reports = [recovery_repetition(context, plan, repetition) for repetition in range(1, plan['repetitions'] + 1)] + finally: + if awake is not None: + awake.terminate() + combinations = {combination: { + 'verdict': analysis.combination_verdict([report['verdict'] for report in reports], plan['repetitions']), + 'repetitions': [{'repetition': report['repetition'], 'verdict': report['verdict'], 'reasons': report['reasons'], + 'report_sha256': host.sha256(evidence.path / f"{combination}-{report['repetition']}/report.json")} + for report in reports]}} + return {'verdict': analysis.overall_verdict(combinations, plan, RECOVERY_PROTOCOL, args.rehearsal), 'plan': plan, + 'combinations': combinations} + + +def recovery(args, live, evidence): + """Every unit's functional scenarios, then the measurement: qualified only when both are.""" + results = recovery_functional(args, live, evidence, RECOVERY_UNITS) + functional = {'verdict': analysis.suite_verdict([entry['verdict'] for entry in results]), + 'scenarios': scenario_summary(results)} + measurement = recovery_measurement(args, live, evidence) + return {'verdict': analysis.recovery_verdict(functional['verdict'], measurement['verdict']), + 'functional': functional, 'measurement': measurement} + + # ---------------------------------------------------------------- verify def normalized(value): @@ -2069,11 +4777,11 @@ def verify_candidate(directory, problems): def verify_scenarios(directory, problems): - """matrix and faults: each scenario's verdict from its recorded checks.""" + """matrix, faults and the recovery units: each scenario's verdict from its recorded checks.""" verdicts = [] for entry in read_rows(directory, 'scenarios'): - verdict = analysis.scenario_verdict(entry['checks']) - compare(problems, f"{entry['mode']} {entry['io'] or '-'} {entry['scenario']} verdict", entry['verdict'], verdict) + verdict = analysis.scenario_verdict(entry['checks'], entry.get('not_run')) + compare(problems, f'{scenario_label(entry)} verdict', entry['verdict'], verdict) verdicts.append(verdict) return analysis.suite_verdict(verdicts) @@ -2159,8 +4867,87 @@ def verify_control(directory, problems): return analysis.overall_verdict(combinations, plan, CONTROL_PROTOCOL, plan['rehearsal']) +def verify_recovery_interval(directory, key, kind, run, report, problems): + """A recovery interval's metrics and verdict from its raw files.""" + raw = directory / key / kind + record = read_json(raw / 'interval.json') + label = f'{key} {kind}' + compare(problems, f'{label} raw files', record['raw'], raw_hashes(raw)) + if 'metrics' not in record: + compare(problems, f'{label} verdict', record['verdict'], 'inconclusive') + return {'verdict': record['verdict'], 'reasons': record['reasons']} + rows = {name: read_rows(raw, name) for name in RECOVERY_SAMPLES} + sampled = {name: read_rows(raw, name) for name in ('host', 'service', 'footprint')} + records = {name: read_rows(raw, name) for name in RECOVERY_RECORDS} + found = read_json(raw / 'markers.json') + runs = read_json(raw / 'runs.json') if (raw / 'runs.json').exists() else [] + release_id = (run.get('release') or {}).get('id') + service_pid = (report.get('service_before') or {}).get('service_pid') + if record['governed']: + service_invalid, _ = analysis.service_findings(sampled['service'], release_id, service_pid) + for reason in service_invalid: + if reason not in record['invalid']: + problems.append(f'{label}: the service samples make it invalid ({reason}), the record does not say so') + metrics = recovery_interval_metrics(record, rows, sampled, records, found, runs, release_id, service_pid, + run.get('reservation')) + verdict, reasons = analysis.recovery_interval_verdict(kind, record['invalid'], metrics) + compare(problems, f'{label} metrics', record['metrics'], metrics) + compare(problems, f'{label} verdict', [record['verdict'], record['reasons']], [verdict, reasons]) + return {'verdict': verdict, 'reasons': reasons} + + +def verify_measurement(directory, problems): + """Every recovery interval from its raw files, then each repetition, the combination and the + measurement.""" + if not (directory / 'run.json').exists(): + # The measurement stopped at its preconditions, before any interval. + return 'inconclusive' + run = read_json(directory / 'run.json') + plan = run['plan'] + compare(problems, 'the recovery protocol', run['protocol'], RECOVERY_PROTOCOL) + combinations = {} + for combination in plan['combinations']: + verdicts = [] + for repetition in range(1, plan['repetitions'] + 1): + key = f'{combination}-{repetition}' + if not (directory / key / 'report.json').exists(): + continue + report = read_json(directory / key / 'report.json') + if 'intervals' in report: + idle = verify_recovery_interval(directory, key, 'idle', run, report, problems) + load = verify_recovery_interval(directory, key, 'load', run, report, problems) + outcome = recovery_repetition_outcome(report, idle, load) + else: + outcome = early_outcome(report) or ('inconclusive', ['the repetition has no intervals']) + compare(problems, f'{key} verdict', [report['verdict'], report['reasons']], list(outcome)) + verdicts.append(outcome[0]) + combinations[combination] = {'verdict': analysis.combination_verdict(verdicts, plan['repetitions'])} + measured = (read_json(directory / 'report.json').get('result') or {}).get('measurement') or {} + if 'combinations' in measured: + compare(problems, 'combination verdicts', {name: entry['verdict'] for name, entry in measured['combinations'].items()}, + {name: entry['verdict'] for name, entry in combinations.items()}) + for name, entry in measured['combinations'].items(): + for repetition in entry['repetitions']: + path = directory / f"{name}-{repetition['repetition']}" / 'report.json' + if not path.exists() or host.sha256(path) != repetition['report_sha256']: + problems.append(f"{name}-{repetition['repetition']}: the report differs from the one the run recorded") + return analysis.overall_verdict(combinations, plan, RECOVERY_PROTOCOL, plan['rehearsal']) + + +def verify_recovery(directory, problems): + """The recovery suite: its functional scenarios and its measurement.""" + functional = verify_scenarios(directory, problems) + measurement = verify_measurement(directory, problems) + result = read_json(directory / 'report.json').get('result') or {} + if result: + compare(problems, 'functional verdict', (result.get('functional') or {}).get('verdict'), functional) + compare(problems, 'measurement verdict', (result.get('measurement') or {}).get('verdict'), measurement) + return analysis.recovery_verdict(functional, measurement) + + VERIFIERS = {'candidate': verify_candidate, 'matrix': verify_scenarios, 'faults': verify_scenarios, - 'authority': verify_authority, 'control': verify_control} + 'authority': verify_authority, 'control': verify_control, 'recovery': verify_recovery, + **{unit: verify_scenarios for unit in RECOVERY_UNITS}} HARNESS = ('qualify-devguard.py', 'qualify_analysis.py', 'qualify_host.py') @@ -2202,37 +4989,72 @@ def verify(directory): # ---------------------------------------------------------------- command line -SUITES = {'candidate': candidate, 'matrix': matrix, 'faults': faults, 'authority': authority, 'control': control} - - -def modes(text): - chosen = [part.strip() for part in text.split(',') if part.strip()] - unknown = [mode for mode in chosen if mode not in MODES] - if unknown or not chosen: - raise argparse.ArgumentTypeError(f"modes are a comma-separated subset of {', '.join(MODES)}") - return chosen +SUITES = {'candidate': candidate, 'matrix': matrix, 'faults': faults, 'authority': authority, 'control': control, + 'recovery': recovery, **{unit: recovery_unit(unit) for unit in RECOVERY_UNITS}} + + +def subset(allowed, name): + def parse(text): + chosen = [part.strip() for part in text.split(',') if part.strip()] + if not chosen or any(part not in allowed for part in chosen): + raise argparse.ArgumentTypeError(f"{name} are a comma-separated subset of {', '.join(allowed)}") + return chosen + return parse + + +modes = subset(MODES, 'modes') +participations = subset(PARTICIPATIONS, 'participations') +RECOVERY_HELP = { + 'recovery-lifecycle': 'the independent Runner across every way a gateway ends (P1R-C01)', + 'recovery-store': "process records and the operations database's one writer (P1R-C02)", + 'recovery-fence': 'Claims, controller epochs and other processes at the socket (P1R-C03)', + 'recovery-reconcile': 'reconciliation after a restart, its races and an authority outage (P1R-C04)', + 'recovery-observation': 'output gaps, retention and a lost Runner (P1R-C05)'} + + +def installed_options(required): + """The installed DevGuard service's options: required for the CS-RG suites, and for `recovery` + only when it measures against the service.""" + options = argparse.ArgumentParser(add_help=False) + options.add_argument('--release', required=required, help='the installed DevGuard release ID') + options.add_argument('--socket', type=Path, default=host.default_socket()) + options.add_argument('--credential-file', type=Path, default=host.CREDENTIAL_FILE, + help="the codespace consumer's credential file; its path is passed, never read here") + options.add_argument('--journal', type=Path, default=host.JOURNAL) + options.add_argument('--service-log', type=Path, default=host.SERVICE_LOG) + options.add_argument('--host-config', type=Path, default=host.HOST_CONFIG) + return options + + +def measurement_options(parser, required): + """DG-1's load and the run's length, as `control` and `recovery` measure.""" + parser.add_argument('--work', type=Path, required=required, + help="a new directory for the load's DevGuard source and target directories") + parser.add_argument('--qualify-bin', type=Path, required=required, + help="DevGuard's devguard-qualify, as DG-1's measurement uses it") + parser.add_argument('--devguard-checkout', type=Path, required=required, + help="a DevGuard checkout holding the release's source commit, for scripts/measure.py") + parser.add_argument('--load', choices=('cold', 'warm'), default='cold') + parser.add_argument('--rehearsal', action='store_true', help='short intervals to try the run; never qualifies') + parser.add_argument('--repetitions', type=int, help='repetitions (protocol: 3)') def arguments(argv=None): parser = argparse.ArgumentParser(description=__doc__.split('\n', 1)[0], - epilog='docs/cs-rg-qualification.md describes each suite and its evidence.') + epilog='docs/cs-rg-qualification.md and docs/recovery-qualification.md describe ' + 'each suite and its evidence.') suites = parser.add_subparsers(dest='suite', metavar='suite') suites.required = True - common = argparse.ArgumentParser(add_help=False) - common.add_argument('--out', type=Path, required=True, help='a new evidence directory') - common.add_argument('--mcp-bin', type=Path, required=True, help='codespace-mcp built with --features devguard') - common.add_argument('--runtime-bin', type=Path, required=True, - help='codespace-codex-runtime built with --features devguard (the uds worker)') + base = argparse.ArgumentParser(add_help=False) + base.add_argument('--out', type=Path, required=True, help='a new evidence directory') + base.add_argument('--mcp-bin', type=Path, required=True, help='codespace-mcp built with --features devguard') + base.add_argument('--runtime-bin', type=Path, required=True, + help='codespace-codex-runtime built with --features devguard (the uds worker, or the ' + 'independent Runner)') + common = argparse.ArgumentParser(add_help=False, parents=[base]) common.add_argument('--modes', type=modes, default=list(MODES), help='in-process, uds or both (default both)') - installed = argparse.ArgumentParser(add_help=False) - installed.add_argument('--release', required=True, help='the installed DevGuard release ID') - installed.add_argument('--socket', type=Path, default=host.default_socket()) - installed.add_argument('--credential-file', type=Path, default=host.CREDENTIAL_FILE, - help="the codespace consumer's credential file; its path is passed, never read here") - installed.add_argument('--journal', type=Path, default=host.JOURNAL) - installed.add_argument('--service-log', type=Path, default=host.SERVICE_LOG) - installed.add_argument('--host-config', type=Path, default=host.HOST_CONFIG) + installed = installed_options(True) fixture = argparse.ArgumentParser(add_help=False) fixture.add_argument('--fixture-bin', type=Path, required=True, help="DevGuard's fixture authority at CodeSpace's pin (examples/fixture_authority)") @@ -2248,19 +5070,33 @@ def arguments(argv=None): help='control-path latency while the fixture authority stalls, pauses, restarts and stops') control_parser = suites.add_parser('control', parents=[common, installed], help="control-path latency under DG-1's load against the installed service") - control_parser.add_argument('--work', type=Path, required=True, - help="a new directory for the load's DevGuard source and target directories") - control_parser.add_argument('--qualify-bin', type=Path, required=True, - help="DevGuard's devguard-qualify, as DG-1's measurement uses it") - control_parser.add_argument('--devguard-checkout', type=Path, required=True, - help="a DevGuard checkout holding the release's source commit, for scripts/measure.py") - control_parser.add_argument('--load', choices=('cold', 'warm'), default='cold') - control_parser.add_argument('--rehearsal', action='store_true', - help='short intervals to try the run; never qualifies') - control_parser.add_argument('--repetitions', type=int, help='repetitions per combination (protocol: 3)') + measurement_options(control_parser, required=True) + recovering = argparse.ArgumentParser(add_help=False, parents=[base]) + recovering.add_argument('--participation', type=participations, default=list(PARTICIPATIONS), + help="off, required or both (default both); required runs on macOS only, against " + "DevGuard's fixture authority") + recovering.add_argument('--fixture-bin', type=Path, + help="DevGuard's fixture authority at CodeSpace's pin, for required (macOS)") + recovering.add_argument('--fixture-helper', type=Path, + help="devguard-launch built from CodeSpace's DevGuard pin, for required (macOS)") + for unit, text in RECOVERY_HELP.items(): + suites.add_parser(unit, parents=[recovering], help=text) + recovery_parser = suites.add_parser( + 'recovery', parents=[recovering, installed_options(False)], + help='every recovery suite, then gateway restarts measured against the installed service (required, ' + 'macOS) or as a self-test of the harness') + measurement_options(recovery_parser, required=False) check_parser = suites.add_parser('verify', help='check a sealed run and recompute its verdicts') check_parser.add_argument('directory', type=Path) - return parser.parse_args(argv) + args = parser.parse_args(argv) + if args.suite in (*RECOVERY_UNITS, 'recovery') and platform.system() == 'Darwin' and 'required' in args.participation: + needed = ['fixture_bin', 'fixture_helper'] + if args.suite == 'recovery': + needed += ['release', 'work', 'qualify_bin', 'devguard_checkout'] + missing = ['--' + name.replace('_', '-') for name in needed if getattr(args, name) is None] + if missing: + parser.error(f"{args.suite} with required participation needs {', '.join(missing)}") + return args def interrupted(number, frame): diff --git a/scripts/qualify_analysis.py b/scripts/qualify_analysis.py index c4af30f..7c94814 100644 --- a/scripts/qualify_analysis.py +++ b/scripts/qualify_analysis.py @@ -1,8 +1,9 @@ -"""CSRG-U6 qualification analysis: pure functions over recorded samples, so that every verdict -can be recomputed from the raw evidence alone. Targets and rules follow DevGuard's SLO protocol -(DevGuard docs/planning/verification.md and scripts/measure.py) wherever they apply; nothing here -redefines them.""" +"""CSRG-U6 and P1R-C06 qualification analysis: pure functions over recorded samples, so that every +verdict can be recomputed from the raw evidence alone. Targets and rules follow DevGuard's SLO +protocol (DevGuard docs/planning/verification.md and scripts/measure.py) wherever they apply; +nothing here redefines them.""" import math +import re # The established targets (DevGuard docs/planning/verification.md, "SLO protocol"). STATUS_P99_MS = 500 @@ -356,3 +357,242 @@ def suite_verdict(verdicts): if ran and all(v == 'pass' for v in ran): return 'passed' return 'incomplete' + + +# ---------------------------------------------------------------- recovery (P1R-C06) + +# A timeout target may be seen gone this much before its recorded deadline (the Runner's timer +# and the gateway's clock) and this much after it (the process's own end, observed every 50 ms). +TIMEOUT_EARLY_S = 0.25 +TIMEOUT_LATE_S = 2.0 +# Fewer restarts than this share of those planned is not a complete observation. +MIN_RESTARTS = 0.8 +# How a gateway that ends this way exits: its input closed, SIGTERM and SIGUSR1 end it normally. +GATEWAY_EXIT = {'eof': 0, 'sigterm': 0, 'sigusr1': 0, 'sigkill': -9} + + +def percentiles(values): + """p50, p99 and the maximum of raw values, nearest rank.""" + return {'p50': number(nearest_rank(values, 0.5)), 'p99': number(nearest_rank(values, 0.99)), + 'max': number(max(values, default=None))} + + +def continuity_metrics(rows, label): + """A numbered output stream across every read of it, through any number of gateways. Each line + is `label` and a fixed-width number. Within a stretch of retained output each line is the one + after the last; a read that asked from before what the Runner still retains (`output_lost`, + `retained_from` above its cursor) declares a gap, after which the numbers may only move on. A + line seen again, a number that goes back, or one skipped where no gap was declared fails.""" + pattern = re.compile(re.escape(label) + r' (\d{8})') + segments, text, gaps = [], '', [] + for row in rows: + retained = row.get('retained_from') + if row.get('output_lost') and retained is not None and row['from'] < retained: + segments.append(text) + text = '' + gaps.append({'from': row['from'], 'retained_from': retained, 'reason': row.get('lost_reason')}) + text += row.get('chunk') or '' + segments.append(text) + lines = undeclared = repeated = declared = 0 + first = last = None + for index, segment in enumerate(segments): + for position, value in enumerate(pattern.findall(segment)): + value = int(value) + lines += 1 + if last is not None and value <= last: + repeated += 1 + continue + expected = 1 if last is None else last + 1 + if value != expected: + if position == 0 and index > 0: + declared += value - expected + else: + undeclared += value - expected + first = value if first is None else first + last = value + return {'reads': len(rows), 'lines': lines, 'first': first, 'last': last, 'declared_gaps': len(gaps), + 'gap_reasons': sorted({str(gap['reason']) for gap in gaps}), 'declared_lost_lines': declared, + 'undeclared_lost_lines': undeclared, 'repeated_lines': repeated, + 'passed': None if not lines else (undeclared == 0 and repeated == 0)} + + +def timeout_metrics(rows): + """Timeout targets, each started with the gateway's process timeout: when its process was first + seen gone against the deadline its record held when it started. Extended: the recorded + deadline moved, or the process outlived it by more than TIMEOUT_LATE_S; early: it ended more + than TIMEOUT_EARLY_S before it; misreported: CodeSpace did not report it timed out. A target + whose end was not observed is missing.""" + offsets, extended, early, misreported, missing = [], 0, 0, 0, 0 + for row in rows: + deadline, gone = row.get('deadline_at_ms'), row.get('gone_unix') + if deadline is None or gone is None: + missing += 1 + continue + offset = gone - deadline / 1000 + offsets.append(offset * 1000) + if row.get('deadline_after_ms') != deadline or offset > TIMEOUT_LATE_S: + extended += 1 + elif offset < -TIMEOUT_EARLY_S: + early += 1 + if row.get('termination') != 'timeout': + misreported += 1 + return {'targets': len(rows), 'in_gap': sum(1 for row in rows if row.get('in_gap')), + 'extended': extended, 'early': early, 'misreported': misreported, 'missing': missing, + 'end_offset_ms': percentiles(offsets), + 'passed': None if not rows else not (extended or early or misreported)} + + +def restart_metrics(rows): + """Each gateway's cycle: it claimed the Runner under the next epoch and served, inherited every + long-running target still running, and every request of another process (a Claim, and + requests that would change a target under an earlier, the current or no epoch) was refused, + changing nothing (a stale mutation otherwise); a rival gateway started on the same operations + database meanwhile never claimed the Runner and gave up for the writer lock; and, when the + cycle ended in a restart, the gateway ended as asked.""" + restarted = [row for row in rows if row.get('restart')] + serving = [row['serving_ms'] for row in rows if row.get('serving_ms') is not None] + ends = {} + for row in restarted: + ends[row.get('how')] = ends.get(row.get('how'), 0) + 1 + unexpected_exit = sum(1 for row in restarted + if (row.get('ended') or {}).get('returncode') != GATEWAY_EXIT.get(row.get('how')) + or (row.get('ended') or {}).get('stopped')) + epochs = sum(1 for row in rows if not row.get('claim') or row['claim'].get('epoch') != row.get('expected_epoch')) + lost_targets = sum(1 for row in rows for state in (row.get('inherited') or {}).values() if state != 'running') + fence = [request for row in rows for request in ((row.get('fence') or {}).get('requests') or [])] + stale = sum(1 for request in fence if request.get('ok')) + unfenced = sum(1 for request in fence if not request.get('ok') and request.get('code') != 'STALE_CONTROLLER') + granted = sum(1 for row in rows if (row.get('fence') or {}).get('claim') != 'controller_active') + rivals = [row['rival'] for row in rows if row.get('rival')] + rival_claims = sum(1 for rival in rivals if rival.get('claims')) + rival_served = sum(1 for rival in rivals if not rival.get('killed') and rival.get('returncode') == 0) + rival_unrefused = sum(1 for rival in rivals if not rival.get('killed') and not rival.get('another_writer')) + errors = [row['error'] for row in rows if row.get('error')] + return {'cycles': len(rows), 'restarts': len(restarted), 'ends': ends, 'serving_ms': percentiles(serving), + 'unexpected_exits': unexpected_exit, 'epoch_mismatches': epochs, 'targets_not_running': lost_targets, + 'fence_requests': len(fence), 'stale_mutations': stale, 'unfenced_refusals': unfenced, + 'fence_claims_not_refused': granted, 'rivals': len(rivals), 'rival_claims': rival_claims, + 'rivals_served': rival_served, 'rivals_not_refused_for_the_lock': rival_unrefused, + 'rivals_killed_waiting': sum(1 for rival in rivals if rival.get('killed')), 'errors': errors[:20], + 'passed': None if not rows else not (unexpected_exit or epochs or lost_targets or stale or unfenced + or granted or rival_claims or rival_served or rival_unrefused + or errors)} + + +def approval_metrics(rows, markers): + """Approvals held before a restart, granted, and resumed by the next gateway, some while a + resume was in flight when the gateway was killed: each was resumed, its command ran exactly + once (its tag in the marker file), and a second resume returned the process the first one + did. One the authority refused for capacity when it was resumed is reported, not failed.""" + counts = {} + for tags in markers.values(): + for tag in tags: + counts[tag] = counts.get(tag, 0) + 1 + resumed = [row for row in rows if (row.get('resume') or {}).get('process_id')] + not_admitted = sum(1 for row in rows if row.get('not_admitted')) + duplicates = sum(1 for row in rows if counts.get(row['tag'], 0) > 1) + unmatched = sum(1 for row in resumed if (row.get('again') or {}).get('process_id') != row['resume']['process_id']) + unresumed = sum(1 for row in rows if not (row.get('resume') or {}).get('process_id') and not row.get('not_admitted')) + not_ran = sum(1 for row in resumed if counts.get(row['tag'], 0) == 0) + return {'approvals': len(rows), 'killed_during_resume': sum(1 for row in rows if row.get('killed_during_resume')), + 'resumed': len(resumed), 'unresumed': unresumed, 'not_admitted': not_admitted, 'duplicates': duplicates, + 'second_resume_differs': unmatched, 'resumed_but_never_ran': not_ran, + 'ready_ms': percentiles([row['ready_ms'] for row in rows if row.get('ready_ms') is not None]), + 'passed': None if not rows else not (duplicates or unmatched or not_ran or unresumed)} + + +def marker_metrics(markers, execs): + """Every tagged command the harness sent: none ran twice, and none that was refused ran. An + approval's command is judged by its resumes (approval_metrics).""" + counts = {} + for tags in markers.values(): + for tag in tags: + counts[tag] = counts.get(tag, 0) + 1 + refused = [row['tag'] for row in execs if row.get('outcome') != 'started' and not row.get('approval')] + return {'tagged': len(execs), 'started': sum(1 for row in execs if row.get('outcome') == 'started'), + 'refused': len(refused), 'ran_twice': sorted(tag for tag, count in counts.items() if count > 1), + 'refused_but_ran': sorted(tag for tag in refused if counts.get(tag)), + 'passed': not any(count > 1 for count in counts.values()) and not any(counts.get(tag) for tag in refused)} + + +def drain_failures(drain): + """What the drain at an interval's end failed to show: a gateway claimed the Runner under the + next epoch and stopped it with its controller's Stop, which found nothing the harness had not + already ended; the Runner exited 0 and removed its socket; with an authority, its instance + retired and nothing of the consumer stayed charged.""" + failures = [] + if not drain: + return ['the Runner was not drained'] + if drain.get('error'): + failures.append(f"the drain failed: {drain['error']}") + if drain.get('claim_epoch') != drain.get('expected_epoch'): + failures.append(f"the draining gateway claimed the Runner under epoch {drain.get('claim_epoch')}, " + f"not {drain.get('expected_epoch')}") + ended = drain.get('gateway') + if ended is not None and (ended.get('returncode') != 0 or ended.get('stopped')): + failures.append(f"the draining gateway exited {ended.get('returncode')} when its input closed") + if drain.get('terminated') is None: + failures.append('the Runner did not log its Stop') + elif drain['terminated']: + failures.append(f"the Stop terminated {drain['terminated']} processes the harness had already ended") + if drain.get('runner_exit') != 0: + failures.append(f"the Runner exited {drain.get('runner_exit')} after Stop") + if drain.get('socket_left'): + failures.append("the Runner's socket was left after Stop") + if drain.get('governed') and not drain.get('retired'): + failures.append("the Runner's DevGuard instance was not retired") + if drain.get('governed') and drain.get('charged'): + failures.append('work of the consumer stayed charged after the drain') + return failures + + +def recovery_interval_verdict(kind, invalid, metrics): + """A recovery interval: the control path as a control interval judges it, over the time a + gateway served, and the recovery evidence. inconclusive when the observation is invalid; fail + on any failure; inconclusive when too little was observed.""" + verdict, reasons = interval_verdict(kind, invalid, metrics) + if invalid: + return verdict, reasons + failures = list(reasons) if verdict == 'fail' else [] + shortfalls = list(reasons) if verdict == 'inconclusive' else [] + recovery = metrics['recovery'] + for name in ('restarts', 'timeouts', 'approvals', 'markers'): + if recovery[name]['passed'] is False: + failures.append(f'recovery {name}') + for name, entry in recovery['continuity'].items(): + if entry['passed'] is False: + failures.append(f'output continuity of {name}') + elif entry['passed'] is None: + shortfalls.append(f'output continuity of {name}: nothing was read') + failures += drain_failures(recovery.get('drain')) + last = recovery.get('last_gateway') + if last is not None and (last.get('returncode') != 0 or last.get('stopped')): + failures.append(f"the last gateway exited {last.get('returncode')} when its input closed") + restarts = recovery['restarts'] + if restarts['restarts'] < MIN_RESTARTS * recovery['planned_restarts']: + shortfalls.append(f"restarts {restarts['restarts']} of {recovery['planned_restarts']}") + timeouts, approvals = recovery['timeouts'], recovery['approvals'] + if timeouts['passed'] is None: + shortfalls.append('timeouts: nothing was measured') + else: + if timeouts['missing']: + shortfalls.append(f"timeouts: {timeouts['missing']} targets unobserved") + if not timeouts['in_gap']: + shortfalls.append('timeouts: no deadline fell while no gateway ran') + if approvals['passed'] is None: + shortfalls.append('approvals: nothing was measured') + elif not approvals['killed_during_resume']: + shortfalls.append('approvals: no gateway was killed with a resume in flight') + if failures: + return 'fail', failures + return ('inconclusive', shortfalls) if shortfalls else ('pass', []) + + +def recovery_verdict(functional, measurement): + """The `recovery` suite: qualified only when its functional scenarios passed and its + measurement qualified.""" + if 'failed' in (functional, measurement): + return 'failed' + if functional == 'passed' and measurement == 'qualified': + return 'qualified' + return 'inconclusive' diff --git a/scripts/qualify_host.py b/scripts/qualify_host.py index d86851b..11b5ba5 100644 --- a/scripts/qualify_host.py +++ b/scripts/qualify_host.py @@ -1,5 +1,6 @@ -"""CSRG-U6 qualification on the host: CodeSpace's gateway driven over its MCP stdio, DevGuard's -installed service and fixture authority observed, and the evidence written and sealed. +"""CSRG-U6 and P1R-C06 qualification on the host: CodeSpace's gateway driven over its MCP stdio, +an independent Runner started and probed as an operator would, DevGuard's installed service and +fixture authority observed, and the evidence written and sealed. Nothing here reads, prints or passes a credential. A gateway is given the credential file's path and opens the file itself; DevGuard's journal is opened read-only and only the fields a check @@ -14,7 +15,10 @@ import platform import queue import re +import signal +import socket as sockets import sqlite3 +import struct import subprocess import threading import time @@ -600,6 +604,38 @@ def close(self, limit=60): 'malformed_lines': self.client.malformed} +# How a gateway can end: its input closed, as a client that goes away does; SIGTERM; SIGUSR1, +# which ends it for a restart and always detaches an independent Runner; or SIGKILL, a loss. +GATEWAY_ENDS = ('eof', 'sigterm', 'sigusr1', 'sigkill') + + +def end_gateway(gateway, how, limit=60): + """End a gateway as `how` says and wait for it to exit: its exit status, how long that took, + and whether it had to be killed. Its log is complete afterwards.""" + began = time.monotonic() + if how == 'eof': + try: + gateway.process.stdin.close() + except OSError: + pass + elif how == 'sigkill': + gateway.process.kill() + elif how in ('sigterm', 'sigusr1'): + gateway.process.send_signal(signal.SIGTERM if how == 'sigterm' else signal.SIGUSR1) + else: + raise ValueError(f'a gateway ends by one of {GATEWAY_ENDS}, not {how!r}') + stopped = None + try: + gateway.process.wait(timeout=limit) + except subprocess.TimeoutExpired: + stopped = 'SIGKILL' + gateway.process.kill() + gateway.process.wait() + gateway.log_reader.join(timeout=10) + return {'how': how, 'returncode': gateway.process.returncode, 'stopped': stopped, + 'exit_ms': round((time.monotonic() - began) * 1000, 3)} + + # ---------------------------------------------------------------- processes and descriptors def group_alive(pgid): @@ -679,6 +715,308 @@ def sysctl(name): return facts +def pid_alive(pid): + """Whether a process with this PID exists, a zombie included.""" + try: + os.kill(pid, 0) + except ProcessLookupError: + return False + except PermissionError: + return True + return True + + +def running(pid): + """Whether a process with this PID runs: it exists and is not a zombie, as `ps` shows it on + macOS and Linux alike.""" + state = run_text('ps', '-o', 'stat=', '-p', str(pid), timeout=10, env=dict(os.environ, LC_ALL='C', LANG='C')) + return bool(state) and not state.startswith('Z') + + +def process_facts(pid): + """When a process started (Unix seconds) and its command's name, from ps in the C locale; + None when it is gone.""" + env = dict(os.environ, LC_ALL='C', LANG='C') + started = run_text('ps', '-o', 'lstart=', '-p', str(pid), timeout=10, env=env) + command = run_text('ps', '-o', 'comm=', '-p', str(pid), timeout=10, env=env) + if not started and not command: + return None + try: + started_unix = time.mktime(time.strptime(' '.join((started or '').split()), '%a %b %d %H:%M:%S %Y')) + except ValueError: + started_unix = None + return {'started_unix': started_unix, 'command': command} + + +def read_pid(path, limit=10.0): + """The PID a command wrote to `path`, once it has written it, or None after `limit`.""" + deadline = time.monotonic() + limit + while True: + try: + text = Path(path).read_text().strip() + if text.isdigit(): + return int(text) + except OSError: + pass + if time.monotonic() >= deadline: + return None + time.sleep(0.02) + + +class PidWatch(threading.Thread): + """Notes when each watched process is first seen gone, polling every `period` seconds, so + that a timeout is timed by the process's own end and not by when a gateway reports it.""" + + def __init__(self, period=0.05): + super().__init__(daemon=True) + self.period = period + self.lock = threading.Lock() + self.watched = {} + self.halt = threading.Event() + self.start() + + def watch(self, key, pid): + with self.lock: + self.watched[key] = {'pid': pid, 'gone_unix': None if pid_alive(pid) else time.time()} + + def gone(self, key): + """When the process was first seen gone (Unix seconds), or None while it runs.""" + with self.lock: + return (self.watched.get(key) or {}).get('gone_unix') + + def wait(self, key, limit): + deadline = time.monotonic() + limit + while self.gone(key) is None and time.monotonic() < deadline: + time.sleep(self.period) + return self.gone(key) + + def run(self): + while not self.halt.wait(self.period): + with self.lock: + pending = [(key, entry['pid']) for key, entry in self.watched.items() if entry['gone_unix'] is None] + for key, pid in pending: + if not pid_alive(pid): + with self.lock: + self.watched[key]['gone_unix'] = time.time() + + def stop(self): + self.halt.set() + self.join(timeout=5) + + +# ---------------------------------------------------------------- the independent Runner (P1R) + +def accepts(path, timeout=1.0): + """Whether something accepts connections at the Unix socket `path`. A connection that says + nothing leaves an independent Runner as it was.""" + connection = sockets.socket(sockets.AF_UNIX, sockets.SOCK_STREAM) + connection.settimeout(timeout) + try: + connection.connect(str(path)) + return True + except OSError: + return False + finally: + connection.close() + + +CLAIMED = re.compile(r'process (\S+) claimed the Runner under epoch (\d+)(; an earlier controller)?') +REFUSED_CLAIM = re.compile(r'refused the Claim of process (\S+) \((\w+)\)') + + +def claim_lines(text): + """The Claims an independent Runner logged, granted and refused, in order, with the PID of + the process that made each (P1R-C03).""" + found = [] + for line in text.splitlines(): + granted = CLAIMED.search(line) + refused = None if granted else REFUSED_CLAIM.search(line) + match = granted or refused + if not match: + continue + pid = int(match[1]) if match[1].isdigit() else None + if granted: + found.append({'granted': True, 'pid': pid, 'epoch': int(granted[2]), + 'previous_controller': bool(granted[3])}) + else: + found.append({'granted': False, 'pid': pid, 'refusal': refused[2]}) + return found + + +class RunnerProcess: + """An independent Runner (`codespace-codex-runtime SOCKET --lifecycle independent`) that the + harness starts as an operator would (D3): in its own session, so its pipe processes share its + process group, in a minimal environment, with its log kept as evidence. Its DevGuard settings + name the authority's socket, the consumer, its credential file (the path only: the Runner + reads it, the harness never does) and the launch helper.""" + + def __init__(self, binary, socket, env, log_path, devguard=(), limit=30.0): + self.socket = Path(socket) + self.argv = [str(binary), str(socket), '--lifecycle', 'independent', *[str(flag) for flag in devguard]] + self.log_path = Path(log_path) + with open(self.log_path, 'ab') as log: + self.process = subprocess.Popen(self.argv, stdin=subprocess.DEVNULL, stdout=log, stderr=log, env=env, + start_new_session=True) + self.pid = self.process.pid + self.listening = self.wait_listening(limit) + + def wait_listening(self, limit): + deadline = time.monotonic() + limit + while time.monotonic() < deadline: + if self.process.poll() is not None: + return False + if accepts(self.socket): + return True + time.sleep(0.02) + return False + + def log(self): + try: + return self.log_path.read_text(errors='replace') + except OSError: + return '' + + def claims(self): + return claim_lines(self.log()) + + def alive(self): + return self.process.poll() is None + + def wait(self, limit=30.0): + """Its exit status once it has exited, or None after `limit` seconds.""" + try: + return self.process.wait(timeout=limit) + except subprocess.TimeoutExpired: + return None + + def stop(self, limit=20.0): + """SIGTERM, which stops the Runner as a controller's Stop does; killed if it still runs + after `limit` seconds.""" + if self.alive(): + self.process.send_signal(signal.SIGTERM) + code = self.wait(limit) + killed = code is None + if killed: + self.process.kill() + code = self.process.wait() + return {'returncode': code, 'killed': killed} + + def kill(self): + if self.alive(): + self.process.kill() + return self.process.wait() + + +WIRE_PROTOCOL = 11 + + +class WireClient: + """A connection to an independent Runner speaking its internal protocol (version 11: a + big-endian u32 length, then a JSON envelope), as any process of the Runner's user can. The + fence probes use it; exit events, which the Runner sends only to its controller, are passed + over.""" + + def __init__(self, path, timeout=30.0): + self.connection = sockets.socket(sockets.AF_UNIX, sockets.SOCK_STREAM) + self.connection.settimeout(timeout) + try: + self.connection.connect(str(path)) + except OSError: + self.connection.close() + raise + self.count = 0 + + def request(self, op, data=None, epoch=None): + """The Runner's answer to one request: whether it succeeded, its result's type and data, + or its error's code and message, and the round trip.""" + self.count += 1 + ident = f'qualify-{os.getpid()}-{self.count}' + envelope = {'protocol': WIRE_PROTOCOL, 'kind': 'request', 'request_id': ident, + 'op': {'op': op} if data is None else {'op': op, 'data': data}} + if epoch is not None: + envelope['epoch'] = epoch + body = json.dumps(envelope).encode() + began = time.monotonic() + self.connection.sendall(struct.pack('>I', len(body)) + body) + while True: + message = self._receive() + if message.get('kind') == 'response' and message.get('request_id') == ident: + break + result, error = message.get('result') or {}, message.get('error') or {} + return {'op': op, 'ok': bool(message.get('ok')), 'type': result.get('type'), 'data': result.get('data'), + 'code': error.get('code'), 'message': error.get('message'), + 'latency_ms': round((time.monotonic() - began) * 1000, 3)} + + def _exactly(self, count): + data = b'' + while len(data) < count: + chunk = self.connection.recv(count - len(data)) + if not chunk: + raise ConnectionError('the Runner closed the connection') + data += chunk + return data + + def _receive(self): + (length,) = struct.unpack('>I', self._exactly(4)) + return json.loads(self._exactly(length)) + + def hello(self): + return self.request('hello') + + def claim(self, store_id): + return self.request('claim', {'store_id': store_id}) + + def detach(self): + return self.request('detach') + + def close(self): + try: + self.connection.close() + except OSError: + pass + + +def operations_query(database, query, parameters=()): + """Rows of a gateway's operations database, read-only, as dictionaries.""" + connection = sqlite3.connect(f'file:{database}?mode=ro', uri=True, timeout=5) + try: + connection.row_factory = sqlite3.Row + return [dict(row) for row in connection.execute(query, parameters).fetchall()] + finally: + connection.close() + + +def store_id(database): + """The operations database's identity, which an independent Runner keeps from its first + Claim (P1R-C03); no secret from the database's owner.""" + rows = operations_query(database, 'SELECT store_id FROM store_identity') + return rows[0]['store_id'] if rows else None + + +def process_records(database): + """Every process record of an operations database, read-only, by process ID (P1R-C02). A + record holds the argv's digest, never the argv, its environment or a secret.""" + return {row['process_id']: row for row in operations_query(database, 'SELECT * FROM processes')} + + +def records_command(binary, database, env, *args, timeout=60): + """`codespace-mcp records ARGS` on an operations database (P1R-C05): its JSON report (None + when it printed none), whether it did all it was asked, and the end of what it wrote to its + standard error.""" + env = dict(env, CODESPACE_OPERATIONS_DB=str(database)) + try: + result = subprocess.run([str(binary), 'records', *args], stdin=subprocess.DEVNULL, capture_output=True, + text=True, timeout=timeout, env=env) + except subprocess.TimeoutExpired: + return {'report': None, 'done': False, 'returncode': None, 'stderr': f'no answer within {timeout} s'} + try: + report = json.loads(result.stdout) if result.stdout.strip() else None + except ValueError: + report = None + return {'report': report, 'done': result.returncode == 0, 'returncode': result.returncode, + 'stderr': result.stderr[-2000:]} + + # ---------------------------------------------------------------- the installed DevGuard service class Release: diff --git a/scripts/tests/test_qualify_devguard.py b/scripts/tests/test_qualify_devguard.py index 297d664..b1c689e 100644 --- a/scripts/tests/test_qualify_devguard.py +++ b/scripts/tests/test_qualify_devguard.py @@ -1,5 +1,6 @@ -"""The CSRG-U6 harness without a host: its analysis, evidence, journal reads, MCP client and -verdict recomputation, on synthetic inputs (docs/cs-rg-qualification.md).""" +"""The CSRG-U6 and P1R-C06 harness without a host: its analysis, evidence, journal reads, MCP and +Runner clients and verdict recomputation, on synthetic inputs (docs/cs-rg-qualification.md, +docs/recovery-qualification.md).""" import contextlib import importlib.util import io @@ -7,11 +8,14 @@ import math import os from pathlib import Path +import socket import sqlite3 +import struct import subprocess import sys import tempfile import textwrap +import threading import time from types import SimpleNamespace import unittest @@ -57,6 +61,58 @@ def admitted(count, latency=60.0): for n in range(count)] +def stream(label, first, last, per_read=3, cursor=0): + """Reads of a numbered output stream, `per_read` lines each, as the measurement records them.""" + rows = [] + for start in range(first, last + 1, per_read): + chunk = ''.join(f'{label} {n:08d}\n' for n in range(start, min(start + per_read, last + 1))) + rows.append({'slot': len(rows), 'outcome': 'read', 'from': cursor, 'to': cursor + len(chunk), 'chunk': chunk, + 'output_lost': False, 'retained_from': cursor, 'lost_reason': None}) + cursor += len(chunk) + return rows + + +def lost(row, retained_from, reason='ring_overflow'): + """A read that asked from before what the Runner still retains.""" + return dict(row, output_lost=True, retained_from=retained_from, lost_reason=reason) + + +def cycles(count): + """A recovery interval's gateway cycles: each claimed under its epoch, inherited its targets, + saw every fence request refused and a rival give up; all but the last restart, ending in turn + as GATEWAY_ENDS says; then the last gateway's end.""" + rows = [] + for number in range(1, count + 1): + row = {'cycle': number, 'gateway': 100 + number, 'expected_epoch': number, 'serving_ms': 30.0, + 'restart': number < count, + 'claim': {'granted': True, 'pid': 100 + number, 'epoch': number, 'previous_controller': number > 1}, + 'fence': {'claim': 'controller_active', 'requests': [ + {'op': op, 'epoch': epoch, 'ok': False, 'code': 'STALE_CONTROLLER'} + for op in ('write_stdin', 'terminate', 'stop') for epoch in (number - 1, number, None)]}, + 'rival': {'pid': 200 + number, 'returncode': 1, 'killed': False, 'another_writer': True, 'claims': []}} + if number > 1: + row['inherited'] = {'status-pipe': 'running', 'beat-pty': 'running'} + if row['restart']: + how = host.GATEWAY_ENDS[number % len(host.GATEWAY_ENDS)] + row.update(how=how, ended={'how': how, 'returncode': analysis.GATEWAY_EXIT[how], 'stopped': None}) + rows.append(row) + rows.append({'cycle': count, 'gateway': 100 + count, 'last': True, 'how': 'eof', + 'ended': {'how': 'eof', 'returncode': 0, 'stopped': None}}) + return rows + + +def timed_out(deadline_s, gone_s, in_gap=False, **changes): + deadline = int(deadline_s * 1000) + return dict({'tag': 'timeout', 'deadline_at_ms': deadline, 'deadline_after_ms': deadline, 'gone_unix': gone_s, + 'termination': 'timeout', 'in_gap': in_gap}, **changes) + + +def resumed(tag, process_id, killed=False, **changes): + resume = {'outcome': 'started', 'process_id': process_id} + return dict({'tag': tag, 'resume': resume, 'again': dict(resume), 'ready_ms': 12.0, 'killed_during_resume': killed, + 'not_admitted': False}, **changes) + + class AnalysisTests(unittest.TestCase): def test_a_missing_sample_ranks_above_every_value(self): self.assertEqual(analysis.nearest_rank([10, 20], 0.99, missing=1), math.inf) @@ -760,5 +816,361 @@ def test_verify_refuses_an_unsealed_directory(self): self.assertEqual(qualify.verify(self.dir / 'open')['verdict'], 'failed') +class RecoveryTests(unittest.TestCase): + """P1R-C06: the recovery suites' analysis, the Runner's log and wire protocol, the serving gate, + the command line and `verify` of a recovery run.""" + + def setUp(self): + self.scratch = tempfile.TemporaryDirectory() + self.dir = Path(self.scratch.name) + + def tearDown(self): + for path in self.dir.rglob('*'): + os.chmod(path, 0o700 if path.is_dir() else 0o600) + self.scratch.cleanup() + + def test_claims_from_the_runner_log(self): + log = '\n'.join(( + 'codespace-codex-runtime: independent Runner 7 listening on /x/runner.sock', + 'codespace-codex-runtime: process 11 claimed the Runner under epoch 1', + 'codespace-codex-runtime: refused the Claim of process 12 (controller_active): another connection ' + 'controls this Runner and is still open', + 'codespace-codex-runtime: the controlling gateway detached; its processes go on', + 'codespace-codex-runtime: process 13 claimed the Runner under epoch 2; an earlier controller started ' + 'work, so it starts nothing where that controller did until it reconciled it')) + self.assertEqual(host.claim_lines(log), [ + {'granted': True, 'pid': 11, 'epoch': 1, 'previous_controller': False}, + {'granted': False, 'pid': 12, 'refusal': 'controller_active'}, + {'granted': True, 'pid': 13, 'epoch': 2, 'previous_controller': True}]) + + def test_the_wire_client_frames_requests_and_passes_events_over(self): + path = self.dir / 'r.sock' + server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + server.bind(str(path)) + server.listen(1) + seen = [] + + def exactly(connection, count): + data = b'' + while len(data) < count: + chunk = connection.recv(count - len(data)) + if not chunk: + raise ConnectionError('closed') + data += chunk + return data + + def send(connection, message): + body = json.dumps(message).encode() + connection.sendall(struct.pack('>I', len(body)) + body) + + def serve(): + connection, _ = server.accept() + with connection: + for answer in ({'ok': True, 'result': {'type': 'claimed', 'data': {'epoch': 4}}}, + {'ok': False, 'error': {'code': 'STALE_CONTROLLER', 'message': 'not the controller'}}): + (length,) = struct.unpack('>I', exactly(connection, 4)) + request = json.loads(exactly(connection, length)) + seen.append(request) + # An exit event, which the Runner sends its controller, comes before the answer. + send(connection, {'protocol': host.WIRE_PROTOCOL, 'kind': 'event', 'event': {'type': 'exited'}}) + send(connection, dict(answer, protocol=host.WIRE_PROTOCOL, kind='response', + request_id=request['request_id'])) + thread = threading.Thread(target=serve, daemon=True) + thread.start() + client = host.WireClient(path, timeout=10) + try: + claim = client.claim('store-1') + stale = client.request('terminate', {'process_id': 'proc-1'}, epoch=3) + finally: + client.close() + thread.join(timeout=10) + server.close() + self.assertEqual([request['op'] for request in seen], [{'op': 'claim', 'data': {'store_id': 'store-1'}}, + {'op': 'terminate', 'data': {'process_id': 'proc-1'}}]) + self.assertEqual([request.get('epoch') for request in seen], [None, 3]) + self.assertEqual({request['protocol'] for request in seen}, {host.WIRE_PROTOCOL}) + self.assertEqual((claim['ok'], claim['type'], claim['data']), (True, 'claimed', {'epoch': 4})) + self.assertEqual((stale['ok'], stale['code']), (False, 'STALE_CONTROLLER')) + + def test_output_continuity(self): + whole = stream('beat', 1, 10) + # A line split between two reads is still one line. + split = ([dict(whole[0], chunk=whole[0]['chunk'][:-5]), + dict(whole[1], chunk=whole[0]['chunk'][-5:] + whole[1]['chunk'])] + whole[2:]) + for rows in (whole, split): + metrics = analysis.continuity_metrics(rows, 'beat') + self.assertEqual((metrics['lines'], metrics['first'], metrics['last'], metrics['passed']), + (10, 1, 10, True)) + # A gap the Runner declared: the numbers only move on after it. + burst = stream('burst', 1, 6) + burst += [lost(row, row['from'] + 4096) for row in stream('burst', 40, 42, cursor=burst[-1]['to'])] + metrics = analysis.continuity_metrics(burst, 'burst') + self.assertEqual((metrics['declared_gaps'], metrics['declared_lost_lines'], metrics['gap_reasons'], + metrics['undeclared_lost_lines'], metrics['passed']), (1, 33, ['ring_overflow'], 0, True)) + # Lines skipped or seen again where no gap was declared fail. + skipped = analysis.continuity_metrics(stream('beat', 1, 6) + stream('beat', 9, 12, cursor=1000), 'beat') + self.assertEqual((skipped['undeclared_lost_lines'], skipped['passed']), (2, False)) + again = analysis.continuity_metrics(stream('beat', 1, 6) + stream('beat', 4, 9, cursor=1000), 'beat') + self.assertEqual((again['repeated_lines'], again['passed']), (3, False)) + empty = [{'slot': 0, 'outcome': 'read', 'from': 0, 'to': 0, 'chunk': '', 'output_lost': False, + 'retained_from': 0}] + self.assertIsNone(analysis.continuity_metrics(empty, 'burst')['passed']) + + def test_timeouts(self): + metrics = analysis.timeout_metrics([timed_out(1000, 1000.05, in_gap=True), timed_out(2000, 2000.4)]) + self.assertEqual((metrics['targets'], metrics['in_gap'], metrics['passed']), (2, 1, True)) + for row, field in ((timed_out(1000, 1003.0), 'extended'), + (timed_out(1000, 1000.05, deadline_after_ms=1_005_000), 'extended'), + (timed_out(1000, 999.5), 'early'), + (timed_out(1000, 1000.05, termination='exited'), 'misreported')): + metrics = analysis.timeout_metrics([row]) + self.assertEqual((metrics[field], metrics['passed']), (1, False), field) + unobserved = analysis.timeout_metrics([timed_out(1000, None)]) + self.assertEqual((unobserved['missing'], unobserved['passed']), (1, True)) + self.assertIsNone(analysis.timeout_metrics([])['passed']) + + def test_restarts(self): + rows = cycles(4)[:-1] + metrics = analysis.restart_metrics(rows) + self.assertEqual((metrics['cycles'], metrics['restarts'], metrics['fence_requests'], metrics['passed']), + (4, 3, 36, True)) + self.assertEqual(metrics['ends'], {'sigterm': 1, 'sigusr1': 1, 'sigkill': 1}) + cases = { + 'stale_mutations': lambda rows: rows[1]['fence']['requests'][0].update(ok=True, code=None), + 'unfenced_refusals': lambda rows: rows[1]['fence']['requests'][4].update(code='NOT_FOUND'), + 'fence_claims_not_refused': lambda rows: rows[2]['fence'].update(claim='claimed'), + 'epoch_mismatches': lambda rows: rows[2]['claim'].update(epoch=2), + 'targets_not_running': lambda rows: rows[3]['inherited'].update({'beat-pty': 'exited'}), + 'unexpected_exits': lambda rows: rows[0]['ended'].update(returncode=1), + 'rival_claims': lambda rows: rows[0]['rival'].update(claims=[{'granted': False, 'pid': 201}]), + 'rivals_served': lambda rows: rows[0]['rival'].update(returncode=0, another_writer=True), + 'rivals_not_refused_for_the_lock': lambda rows: rows[0]['rival'].update(another_writer=False)} + for field, change in cases.items(): + changed = json.loads(json.dumps(rows)) + change(changed) + metrics = analysis.restart_metrics(changed) + self.assertEqual((metrics[field], metrics['passed']), (1, False), field) + # A rival still waiting for the writer when its cycle ended was killed; that fails nothing. + waiting = json.loads(json.dumps(rows)) + waiting[0]['rival'].update(killed=True, returncode=-9, another_writer=False) + self.assertEqual(analysis.restart_metrics(waiting)['rivals_killed_waiting'], 1) + self.assertTrue(analysis.restart_metrics(waiting)['passed']) + self.assertIsNone(analysis.restart_metrics([])['passed']) + + def test_approvals_and_markers(self): + markers = {'confirm': ['approval-001', 'approval-002'], 'term': ['term-0']} + rows = [resumed('approval-001', 'proc-1'), resumed('approval-002', 'proc-2', killed=True)] + metrics = analysis.approval_metrics(rows, markers) + self.assertEqual((metrics['resumed'], metrics['killed_during_resume'], metrics['passed']), (2, 1, True)) + twice = analysis.approval_metrics(rows, dict(markers, again=['approval-002'])) + self.assertEqual((twice['duplicates'], twice['passed']), (1, False)) + differs = analysis.approval_metrics([resumed('approval-001', 'proc-1', again={'process_id': 'proc-9'})], + markers) + self.assertEqual((differs['second_resume_differs'], differs['passed']), (1, False)) + never = analysis.approval_metrics([resumed('approval-003', 'proc-3')], markers) + self.assertEqual((never['resumed_but_never_ran'], never['passed']), (1, False)) + unresumed = analysis.approval_metrics([{'tag': 'approval-004', 'resume': {'outcome': 'error'}}], markers) + self.assertEqual((unresumed['unresumed'], unresumed['passed']), (1, False)) + capacity = analysis.approval_metrics([{'tag': 'approval-004', 'not_admitted': True}], markers) + self.assertEqual((capacity['not_admitted'], capacity['unresumed'], capacity['passed']), (1, 0, True)) + execs = [{'tag': 'term-0', 'outcome': 'started'}, {'tag': 'hog-1', 'outcome': 'refused'}, + {'tag': 'approval-001', 'outcome': 'refused', 'approval': True}] + self.assertTrue(analysis.marker_metrics(markers, execs)['passed']) + ran = analysis.marker_metrics(dict(markers, hog=['hog-1']), execs) + self.assertEqual((ran['refused_but_ran'], ran['passed']), (['hog-1'], False)) + self.assertEqual(analysis.marker_metrics(dict(markers, again=['term-0']), execs)['ran_twice'], ['term-0']) + + def test_drain_failures(self): + drain = {'governed': True, 'expected_epoch': 5, 'claim_epoch': 5, 'terminated': 0, 'runner_exit': 0, + 'socket_left': False, 'retired': True, 'charged': 0, + 'gateway': {'how': 'eof', 'returncode': 0, 'stopped': None}} + self.assertEqual(analysis.drain_failures(drain), []) + self.assertEqual(analysis.drain_failures(None), ['the Runner was not drained']) + for change, failure in ( + ({'claim_epoch': 4}, 'the draining gateway claimed the Runner under epoch 4, not 5'), + ({'gateway': {'how': 'eof', 'returncode': -9, 'stopped': 'SIGKILL'}}, + 'the draining gateway exited -9 when its input closed'), + ({'terminated': None}, 'the Runner did not log its Stop'), + ({'terminated': 2}, 'the Stop terminated 2 processes the harness had already ended'), + ({'runner_exit': None}, 'the Runner exited None after Stop'), + ({'socket_left': True}, "the Runner's socket was left after Stop"), + ({'retired': False}, "the Runner's DevGuard instance was not retired"), + ({'charged': 1}, 'work of the consumer stayed charged after the drain'), + ({'error': 'OSError: boom'}, 'the drain failed: OSError: boom')): + self.assertEqual(analysis.drain_failures(dict(drain, **change)), [failure]) + self.assertEqual(analysis.drain_failures(dict(drain, governed=False, retired=None, charged=None)), []) + + def test_the_serving_gate(self): + serving = qualify.Serving() + sample = qualify.gated(serving, lambda gateway, slot: {'outcome': 'answered', 'slot': slot}) + self.assertEqual(sample(0), {'outcome': 'restarting'}) + serving.serve(SimpleNamespace(pid=41)) + self.assertEqual(sample(1), {'outcome': 'answered', 'slot': 1, 'gateway': 41}) + entered, release = threading.Event(), threading.Event() + + def slow(gateway, slot): + entered.set() + release.wait(10) + return {'outcome': 'answered'} + thread = threading.Thread(target=qualify.gated(serving, slow), args=(2,)) + thread.start() + self.assertTrue(entered.wait(10)) + # A restart closes the gate at once, and waits for what is in flight up to its limit. + self.assertEqual(serving.close(limit=0.05), 1) + self.assertEqual(sample(3), {'outcome': 'restarting'}) + release.set() + thread.join(timeout=10) + served = serving.served_s + self.assertGreater(served, 0) + time.sleep(0.02) + self.assertEqual(serving.close(), 0) + self.assertEqual(serving.served_s, served) + + def test_recovery_arguments(self): + common = ['--out', 'x', '--mcp-bin', 'm', '--runtime-bin', 'r'] + self.assertEqual(qualify.arguments(['recovery-fence', *common, '--participation', 'off']).participation, + ['off']) + self.assertEqual(qualify.arguments(['recovery-store', *common]).participation, list(qualify.PARTICIPATIONS)) + with contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit): + qualify.arguments(['recovery-fence', *common, '--participation', 'sometimes']) + with mock.patch.object(qualify.platform, 'system', return_value='Darwin'): + with contextlib.redirect_stderr(io.StringIO()) as printed, self.assertRaises(SystemExit): + qualify.arguments(['recovery-fence', *common]) + self.assertIn('needs --fixture-bin, --fixture-helper', printed.getvalue()) + fixture = ['--fixture-bin', 'f', '--fixture-helper', 'h'] + self.assertEqual(qualify.arguments(['recovery-fence', *common, *fixture]).fixture_bin, Path('f')) + self.assertEqual(qualify.arguments(['recovery', *common, '--participation', 'off']).release, None) + with contextlib.redirect_stderr(io.StringIO()) as printed, self.assertRaises(SystemExit): + qualify.arguments(['recovery', *common, *fixture]) + self.assertIn('needs --release, --work, --qualify-bin, --devguard-checkout', printed.getvalue()) + + # ------------------------------------------------------------ verify + + def write_rows(self, directory, rows): + directory.mkdir(parents=True, exist_ok=True) + for name, values in rows.items(): + (directory / f'{name}.jsonl').write_text(''.join(json.dumps(row) + '\n' for row in values)) + + def recovery_interval(self, evidence, key, kind, served_s=60.0): + """An interval of four gateway cycles that saw nothing wrong, as a self-test records it.""" + raw = evidence.file(f'{key}/{kind}/x').parent + terminations, _ = terminated(3) + burst = stream('burst', 1, 6) + burst += [lost(row, row['from'] + 4096) for row in stream('burst', 900, 902, cursor=burst[-1]['to'])] + rows = {'status_pipe': answered(60) + [{'slot': 60, 'outcome': 'restarting'}], 'status_pty': answered(60), + 'terminations': terminations, 'releases': [], 'io': echoed(12), 'refusals': [], + 'read_beat': stream('beat', 1, 30), 'read_burst': burst} + records = {'restarts': cycles(4), + 'timeouts': [timed_out(1000, 1000.04, in_gap=True), timed_out(1030, 1030.02)], + 'approvals': [resumed('approval-001', 'proc-a1'), resumed('approval-002', 'proc-a2', killed=True)], + 'execs': [{'tag': 'status-pipe', 'outcome': 'started'}, + {'tag': 'approval-001', 'outcome': 'refused', 'code': 'APPROVAL_REQUIRED', + 'approval': True}]} + found = {'status-pipe': ['status-pipe'], 'confirm': ['approval-001', 'approval-002']} + sampled = {'host': [{'pressure_level': '1', 'swap_used_mib': 10.0, 'load_average': [2.0, 1.0, 1.0]}], + 'footprint': [{'owner': {'rss_bytes': 1 << 20, 'cpu_percent': 1.0, 'threads': 9}, + 'gateway': {'rss_bytes': 1 << 21, 'cpu_percent': 2.0, 'threads': 8}}]} + self.write_rows(raw, dict(rows, **records, **sampled)) + (raw / 'markers.json').write_text(json.dumps(found)) + record = {'kind': kind, 'combination': 'independent-off', 'governed': False, 'invalid': [], + 'planned_restarts': 3, 'served_s': served_s, 'started_unix': 990.0, 'ended_unix': 1060.0, + 'seconds': 70.0, 'targets': [{'effective': True}, {'effective': True}], + 'drain': {'governed': False, 'expected_epoch': 5, 'claim_epoch': 5, 'runner_exit': 0, + 'socket_left': False, 'terminated': 0, + 'gateway': {'how': 'eof', 'returncode': 0, 'stopped': None}}} + record['raw'] = qualify.raw_hashes(raw) + metrics = qualify.recovery_interval_metrics(record, rows, sampled, records, found, [], None, None, None) + record['verdict'], record['reasons'] = analysis.recovery_interval_verdict(kind, [], metrics) + record['metrics'] = metrics + evidence.write_json(f'{key}/{kind}/interval.json', record) + return record + + def recovery_run(self, name='recovery'): + evidence = host.Evidence(self.dir / name) + check = analysis.check + entries = [{'suite': 'recovery-fence', 'scenario': 'stale-epoch', 'mode': 'independent', + 'participation': 'off', 'io': None, 'checks': [check('a stale epoch is refused', True)]}, + {'suite': 'recovery-fence', 'scenario': 'stale-epoch', 'mode': 'independent', + 'participation': 'required', 'io': None, 'checks': [], 'not_run': qualify.NOT_NATIVE}] + for entry in entries: + entry['verdict'] = analysis.scenario_verdict(entry['checks'], entry.get('not_run')) + evidence.append('scenarios.jsonl', entry) + functional = analysis.suite_verdict([entry['verdict'] for entry in entries]) + plan = dict(qualify.RECOVERY_REHEARSAL, combinations=['independent-off'], load=None, rehearsal=True) + evidence.write_json('run.json', {'plan': plan, 'protocol': qualify.RECOVERY_PROTOCOL, 'governed': False}) + key = 'independent-off-1' + records = {kind: self.recovery_interval(evidence, key, kind) for kind in ('idle', 'load')} + report = {'combination': 'independent-off', 'repetition': 1, 'rehearsal': True, 'governed': False, + 'intervals': {kind: {'verdict': record['verdict'], 'reasons': record['reasons']} + for kind, record in records.items()}} + report['verdict'], report['reasons'] = qualify.recovery_repetition_outcome(report, records['idle'], + records['load']) + evidence.write_json(f'{key}/report.json', report) + combination = analysis.combination_verdict([report['verdict']], 1) + measurement = analysis.overall_verdict({'independent-off': {'verdict': combination}}, plan, + qualify.RECOVERY_PROTOCOL, True) + verdict = analysis.recovery_verdict(functional, measurement) + evidence.write_json('report.json', { + 'suite': 'recovery', 'verdict': verdict, 'harness': qualify.harness(), + 'result': {'verdict': verdict, 'functional': {'verdict': functional}, 'measurement': { + 'verdict': measurement, 'combinations': {'independent-off': {'verdict': combination, 'repetitions': [{ + 'repetition': 1, 'verdict': report['verdict'], 'reasons': report['reasons'], + 'report_sha256': host.sha256(evidence.path / key / 'report.json')}]}}}}}) + evidence.seal() + return evidence.path, records + + def test_recovery_interval_verdicts(self): + _, records = self.recovery_run() + self.assertEqual((records['idle']['verdict'], records['idle']['reasons']), ('pass', [])) + self.assertEqual(records['load']['reasons'], ['no load ran: there is no DevGuard service to run it']) + metrics = records['idle']['metrics'] + + def judged(change): + changed = json.loads(json.dumps(metrics)) + change(changed['recovery']) + return analysis.recovery_interval_verdict('idle', [], changed) + for change, expected in ( + (lambda found: found['timeouts'].update(in_gap=0), + ('inconclusive', ['timeouts: no deadline fell while no gateway ran'])), + (lambda found: found['approvals'].update(killed_during_resume=0), + ('inconclusive', ['approvals: no gateway was killed with a resume in flight'])), + (lambda found: found['restarts'].update(restarts=2), ('inconclusive', ['restarts 2 of 3'])), + (lambda found: found['continuity']['burst_pipe'].update(passed=None), + ('inconclusive', ['output continuity of burst_pipe: nothing was read'])), + (lambda found: found['restarts'].update(passed=False), ('fail', ['recovery restarts'])), + (lambda found: found['continuity']['beat_pty'].update(passed=False), + ('fail', ['output continuity of beat_pty'])), + (lambda found: found['last_gateway'].update(returncode=-9), + ('fail', ['the last gateway exited -9 when its input closed'])), + (lambda found: found['drain'].update(socket_left=True), + ('fail', ["the Runner's socket was left after Stop"]))): + self.assertEqual(judged(change), expected) + self.assertEqual(analysis.recovery_interval_verdict('idle', ['the Runner did not register'], metrics), + ('inconclusive', ['the Runner did not register'])) + self.assertEqual(analysis.recovery_verdict('passed', 'qualified'), 'qualified') + self.assertEqual(analysis.recovery_verdict('passed', 'inconclusive'), 'inconclusive') + self.assertEqual(analysis.recovery_verdict('incomplete', 'qualified'), 'inconclusive') + self.assertEqual(analysis.recovery_verdict('passed', 'failed'), 'failed') + + def test_verify_recovery(self): + path, _ = self.recovery_run() + result = qualify.verify(path) + self.assertEqual((result['verdict'], result['recomputed_verdict']), ('verified', 'inconclusive'), result) + self.assertEqual(qualify.scenario_label(qualify.read_rows(path, 'scenarios')[1]), + 'recovery-fence independent required - stale-epoch') + # A stale mutation the record leaves out is found again from the raw restarts. + restarts = path / 'independent-off-1/idle/restarts.jsonl' + rows = qualify.read_rows(restarts.parent, 'restarts') + rows[1]['fence']['requests'][0].update(ok=True, code=None) + os.chmod(restarts, 0o600) + restarts.write_text(''.join(json.dumps(row) + '\n' for row in rows)) + result = qualify.verify(path) + self.assertEqual(result['verdict'], 'failed') + for prefix in ('changed independent-off-1/idle/restarts.jsonl', 'independent-off-1 idle metrics', + 'independent-off-1 idle verdict'): + self.assertTrue(any(problem.startswith(prefix) for problem in result['problems']), + (prefix, result['problems'])) + + if __name__ == '__main__': unittest.main()