diff --git a/docs/reference/specs/execution.md b/docs/reference/specs/execution.md index 871e6af1d..558f1170b 100644 --- a/docs/reference/specs/execution.md +++ b/docs/reference/specs/execution.md @@ -35,7 +35,7 @@ Tools never touch the bot host: every `bash`/`read_file`/`write_file` runs throu 23. **A thread's first request never carries the container's start (`sandbox-starting`).** The SDK's first exec on a container that is not running does the start inside the call — the platform's instance grant (its default wait 30 s), the image pull on a machine that has not seen the image, the microVM boot and the runtime's port (90 s) — so a slow start can outlive the executor's per-send deadline for a 60 s command (90 s, item 11). The deaths that led here — every run that fell to a fresh sandbox dying with "gave no answer within 90s", seven in one hour once the leaked fleet drained — turned out to be item 24's held output pipes, not the start (the Worker's logs showed the container answering commands in 230 ms ten seconds after the ask); the gate stays as the guard for a genuinely slow start, which the SDK's own allowances say can take two minutes. The contract: (a) the **Worker's** Durable Object runs every route through a start gate (`src/execution/sandboxStart.ts`, node-free, bundled; `StartGate`, wired in `SwitchboardSandbox`): when `ctx.container.running` is not `true` and no warm-up is in flight, it begins one — a single `true` through the SDK, which does the start, run inside the idle ledger so the guard (item 22) sees a request in flight for as long as the start takes — and answers at once; `/exec` finishes its streamed body with `{error: "sandbox-starting: ()", reason: "sandbox-starting", stdout: "", stderr: , exitCode: 127}` (the item-3 dual shape), `/read` and `/write` answer HTTP 503 `{error, reason: "sandbox-starting"}`; while the warm-up is in flight every request gets the same answer and no second warm-up begins; once the runtime answered, requests run. A warm-up that fails hands its error to the next request, once, where it is classified as any SDK error is — a full fleet is `fleet-busy` (item 14), a silent control port `runtime-unreachable` (item 9) — and the request after that starts a fresh warm-up if the container is still not running; a container the idle guard destroyed is started again the same way by the thread's next request. Every start is two JSON lines in Workers Logs, `sandbox.starting` and `sandbox.started` with `durationMs` (or `sandbox.start-failed` with the error), so the start time is a measured fact. (b) The **executor** (`CloudflareSandboxExecutor.call`) matches the machine token only (`reason === "sandbox-starting"`, in-body or on a 503 — an older Worker changes nothing) and re-sends the IDENTICAL request after 5 s, 10 s, then 15 s (`SANDBOX_START_BACKOFF_MS`) until the total wait reaches `SANDBOX_START_WAIT_MAX_MS` (10 min — under a midday burst the platform granted containers 2.5 to 5 minutes late, and two threads that died at five minutes had theirs a minute away; the ladder is the token's, restarting at its first step when the answer's token changes, so a start that becomes a refused start is re-sent after the fleet's 10 s, not the 30 s the start's ladder had reached; the Worker asks the SDK for the instance grant with a 10 s limit, `INSTANCE_GET_TIMEOUT_MS`, so a refusal the platform took 63 s to voice under the SDK's 30 s default and retries reaches the gate in seconds) — **whatever the command's own budget**: the start is not the command's time, and a 60 s command survives a two-minute start it did not cause. The two tokens share one wait ledger; the plan (budget, backoff, exhausted message) is the last token's, so a start that turns into a full fleet is judged by the fleet's budget from then on with the time already waited counted. (c) When the wait is spent it throws `ExecCapacityError` — `sandbox not ready — the thread's container did not finish starting within 600s; try again in a few minutes` — never `ExecInfraError`: a slow start is not a dead sandbox. Invariant: **the first command a thread runs on a fresh or restarted container is judged against its own budget from the moment the container is up, never against the start.** Deploy order: the Worker before the bot, as for item 14 — an old Worker's first exec still carries the start, today's behaviour. 24. **A command's output ends when the command does; a detached child never holds the answer.** On the 0.13 line an exec's answer is read from the process's log stream, which the runtime ends only when every holder of the command's stdout and stderr is gone. `setsid -f sh -c '…'` forks a wrapper out of the exec's process group but not out of its file descriptors, so a detached shell that inherited them kept the stream open for its own lifetime — the harness seam's pi start did exactly that, the start command never answered, and every run that fell to a fresh sandbox died at the executor's 90 s deadline while its container, and its pi, ran on (the same shape at 05:08Z that day and after each deploy; run 8aff9e50 is the receipt — the container answered its first commands in 230 ms and the start command was `canceled` 90 s later). Three parts: (a) the **seam's** start script (`startScript`, `src/core/harness/container.ts`) runs the wrapper as `setsid -f sh -c '' /dev/null 2>&1` (`DETACHED_STDIO`); the process's own stdin is the FIFO and its stdout and stderr the run's files as before, so nothing the process needs changes, and the exec's descriptors close when the exec's shell exits. (b) The **Worker** waits for the output to end only until the command's deadline plus `OUTPUT_AFTER_EXIT_MS` (20 s, `src/execution/sandboxLifecycle.ts`; inside the executor's 30 s per-send margin so this answer always arrives first), then asks the runtime for the process's status: **exited** → the answer is its exit code, empty output and `heldOutputNote` on stderr — what happened, that the output is lost to this call and the job runs on, and how to detach (`DETACH_HINT`); **running** → the process is killed and answered as the shell-level timeout, as before. (c) The exit-124 **hint** (`timeoutNote`) spells the detach with the same redirects, `DETACH_HINT` = `setsid -f sh -c ' > /tmp/job.log 2>&1' /dev/null 2>&1`, one string shared by the seam (`DETACHED_STDIO` equals `DETACH_REDIRECTS`, held by a test) and the docs ([agent-explore.md](agent-explore.md) item 5). Invariant: **the Worker answers every `/exec` by the command's deadline plus 20 s, whatever the command left running** — a held pipe costs the call its output, never the run. 25. **A cold sandbox is seeded from the resident's snapshot before its first command (`POST /seed`).** A resident holds a stamped snapshot of its repository in R2 — the checkout without its dependency view ([resident-repos.md](resident-repos.md) item 7) and one archive per lockfile key holding the view (item 61) — and its `/status` publishes the handle, `snapshot.{checkoutBackupId, depsBackupId, ref, sha}`. The bot forwards it as the body's `seed` (`src/execution/seedPlan.ts`: `{slug, checkoutBackupId, depsBackupId?, ref, sha, fetchRef?, fetchSha?}`, every field checked — a backup id is a UUID, a slug is `owner/name`, a ref a branch name, a sha forty hex — and a bad one refused 400 by field name), with the run's env as on every route. The Durable Object, inside the idle ledger and behind the start gate (items 22, 23), restores the checkout into `/workspace/checkout` and the deps entry into `/workspace/.seed-deps` **presigned only** — the container downloads the archive over a URL the object signs, so a gigabyte never crosses the isolate; a Worker without the four presigned values answers `seed-unconfigured`, and `/healthz` says `backupTransfer` — each restore judged by bytes arriving against one shared `SEED_RESTORE_MAX_MS` deadline (8 min) and extracted onto the disk the resident's way (`unsquashfs` from the downloaded archive; the image carries squashfs-tools), while the image also carries apt's `ripgrep` and proves `rg` at build time — seed restore changes disk contents, not image tools, so fresh and seeded sandboxes inherit the same binary without an agent download — then runs the fix-up as one failing-fast script under `SEED_FIXUP_TIMEOUT_MS` (3 min): the tree becomes root's, origin points at GitHub, the deps view replaces `node_modules`, the thread's `fetchRef` is fetched (the credential is the exec env's `GH_TOKEN` through the image's helper, never text in the script) and checked out at `fetchSha` when the fetch holds it else at the tip, and the head is printed last as the answer's `sha`. The marker `/workspace/.switchboard-seed` names the handle the container was seeded from: the same handle again (a retry, a re-attach) answers `cached` at once and never restores over a live tree. Every outcome is named in the streamed body (item 3), so the executor never reads a seed that did not happen as a dead sandbox: `{seeded: true, cached, slug, ref, sha, from, steps: {restore, deps, fixup}, phases: {checkout: {download, extract}, deps: {download, extract} | null}, ms}` — each restore split into the archive's download (judged by bytes) and its extraction onto the disk, logged as `sandbox.seed-restore` too, so a slow seed's owner is read off the answer, or `{seeded: false, reason, detail, step}` with `reason` one of `seed-missing` (the handle's objects are gone — a rotation or an offboard took them; the caller re-reads `/status` once and retries), `seed-failed` (a step failed; the run goes cold with the note), `seed-unconfigured`; a start or a full fleet met at the warm-up keeps its wait token (items 14, 23). A failed seed sweeps what it wrote — mounts, a half tree, the marker — so the cold run that follows clones into an empty workspace. The client is `CloudflareSandboxExecutor.seed`, under `SEED_BUDGET_MS`; the bot's selection of a seeded sandbox is the next item's work. The sandbox Worker binds the RESIDENT's cache bucket (`{{resident.script}}-cache`) and names it in `BACKUP_BUCKET_NAME`, inside an `{{#if resident}}` block of its template — a profile without a resident renders no seed bindings — and the `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` secrets are the resident's own, provisioned to the sandbox too. Deploy: the Worker before the bot, as for items 14 and 23. -26. **A run the resident refuses lands in a seeded sandbox when the probe carried the handle (`ExecutorSelection.seeded`).** In the factory's resident fallback — the probe answered a state that is not serviceable, or the attach failed — a probe body that carried `seed` (item 25) makes the per-thread sandbox call `POST /seed` before the run: `seedForThread` (`src/execution/seedPlan.ts`) puts the thread's own ref and resolved head on the handle as `fetchRef` / `fetchSha` (a thread with no ref stays on the snapshot's branch). A `seeded` answer puts `{slug, ref, sha, workspace: /workspace/checkout, cached, ms}` on the selection, and the card's note reads ` — seeded sandbox · from resident snapshot · owner/name · ref@sha7` (`seededSandboxNote`), so a reader tells the three paths — resident, seeded, cold — apart from Slack alone; the `resident` discriminant stays unset. `seed-missing` re-reads `/status` once (`seedRetryDecision`) and retries with the newer handle when the resident published one; the same handle again, a second miss, `seed-failed`, `seed-unconfigured`, or a Worker with no `/seed` (an older release — the client's infra error) send the run cold with the refusal on the note: ` — using fresh sandbox (seed …)`. Nothing to seed from — an unreachable resident (no body), a repository not onboarded (404), an execution type that is not the cloudflare sandbox — is the cold path unchanged. The dispatcher composes the seeded prompt variant after executor resolution exactly as the resident's ([resident-repos.md](resident-repos.md) item 31): `AgentDef.seededSystem` (`CODING_SYSTEM_SEEDED`, `REVIEW_SYSTEM_SEEDED` — the repository already cloned at the checkout, no clone, no install, `gh` and Docker present unlike the resident's) plus a line naming the repository and the checkout; the REVIEW TARGET block's seeded branch names the checkout, forbids a second clone, pins the first command to the HEAD check there and diffs against the base already present. `resident` and `seeded` never both hold; resident wins. +26. **A run the resident refuses lands in a seeded sandbox when the probe carried the handle (`ExecutorSelection.seeded`).** In the factory's resident fallback — the probe answered a state that is not serviceable, or the attach failed — a probe body that carried `seed` (item 25) makes the per-thread sandbox call `POST /seed` before the run. Because this fallback stands behind the first attach, a drained fleet is waited for under the FALLBACK's own cost, never the deploy's ([resident-repos.md](resident-repos.md) item 69, issue 2101): the attach passes `drainBoundMs` = `DRAIN_FALLBACK_WAIT_MS` (3 min — a seeded sandbox stands up in about two), the typed draining error carries the drain's share, and the fallback selection's `drainWaitMs` publishes the run's `drain_wait` note. `seedForThread` (`src/execution/seedPlan.ts`) puts the thread's own ref and resolved head on the handle as `fetchRef` / `fetchSha` (a thread with no ref stays on the snapshot's branch). A `seeded` answer puts `{slug, ref, sha, workspace: /workspace/checkout, cached, ms}` on the selection, and the card's note reads ` — seeded sandbox · from resident snapshot · owner/name · ref@sha7` (`seededSandboxNote`), so a reader tells the three paths — resident, seeded, cold — apart from Slack alone; the `resident` discriminant stays unset. `seed-missing` re-reads `/status` once (`seedRetryDecision`) and retries with the newer handle when the resident published one; the same handle again, a second miss, `seed-failed`, `seed-unconfigured`, or a Worker with no `/seed` (an older release — the client's infra error) send the run cold with the refusal on the note: ` — using fresh sandbox (seed …)`. Nothing to seed from — an unreachable resident (no body), a repository not onboarded (404), an execution type that is not the cloudflare sandbox — is the cold path unchanged. The dispatcher composes the seeded prompt variant after executor resolution exactly as the resident's ([resident-repos.md](resident-repos.md) item 31): `AgentDef.seededSystem` (`CODING_SYSTEM_SEEDED`, `REVIEW_SYSTEM_SEEDED` — the repository already cloned at the checkout, no clone, no install, `gh` and Docker present unlike the resident's) plus a line naming the repository and the checkout; the REVIEW TARGET block's seeded branch names the checkout, forbids a second clone, pins the first command to the HEAD check there and diffs against the base already present. `resident` and `seeded` never both hold; resident wins. 27. **A restoring resident is a short wait, never a cold fall-through (`/await-restore`).** A resident deploy restores every resident from its snapshot, and every dispatch that probed one mid-restore used to fall to the per-thread fleet at once — under load that burst exhausted `max_instances` and the runs died on capacity while a warm resident was seconds away (the leaked-slot half of the same incident was fixed separately). Event-driven, with **no polling and no retry timer on either side**: (a) the resident Worker gains `POST /await-restore` (operator scope, body `{resource}`; a not-onboarded resource is 404) whose Durable Object registers a waiter on an in-memory ledger (`RestoreWaiters`, `src/execution/restoreWaiters.ts`) BEFORE it reads the state — the read awaits storage, and a transition landing in that gap would otherwise publish to a ledger the request is not on yet and strand it — then answers a non-`restoring` state at once, withdrawing the waiter, and otherwise holds the ONE request until `setResidentState` transitions out of `restoring` — the transition itself publishes the restore event to every held waiter with the state landed on. The answer streams like `/attach` (heartbeat whitespace then one JSON document over HTTP 200) so a restore lasting minutes cannot lose the connection; a DO restart drops the held RPCs with the ledger and the bot's deadline names the fallback. (b) The bot's factory: a `/status` probe that finds the resident `restoring` opens that one held request (`ResidentExecutor.awaitRestore`, bounded by `AWAIT_RESTORE_TIMEOUT_MS` = 10 min — the outer safety net, not a retry timer) instead of falling cold; when the answer is a serviceable state it attaches as any serviceable probe does and the card names the wait (`· after waiting for the resident's restore`); a non-serviceable landing, an attach that fails after the wait, a failed wait, or an older Worker's 404 (a bundle that predates the route) takes item 26's fallback — a sandbox seeded from the probe's handle when it carried one, a fresh one otherwise — with the wait named on the note, never silently. The run's stop rides into the held request (`awaitRestore`'s signal): a run stopped during the hold is the stop's typed shape (`aborted`), decided by the signal before the error's name, and is never provisioned cold. (c) The resumed-run re-attach holds the same request (issue 1364 part 1; [run-history.md](run-history.md) item 54): a re-attach's probe that finds the resident `restoring` opens the held request too — a resumed run's worktree lives on that resident or nowhere, so a refusal at once would close the run and restart it from its request, losing the tree, the transcript and a coordinator child's round to a rehydrate that ends on its own — bounded by the same ceiling clipped to the run's lease less its write-up reserve (`restoreHoldBudget`), the lease read again after the hold so its end under the hold is the run's end on its budget (item 9), never a refusal; a serviceable landing re-attaches the run's own worktree with the wait named on the note, and every landing short of that — a non-serviceable state, an older Worker's 404, a failed wait — is the typed re-attach refusal naming the wait: a re-attach never falls to a sandbox, whichever way the hold ends. Deploy order: the resident Worker before the bot, as always — an old Worker's 404 is the named cold fallback on the fresh path, today's behaviour. 28. **A container that did not accept the connection is a wait, never a dead sandbox (`runtime-busy`).** The container's runtime accepts one control connection per SDK call, and every `/exec`, `/read` and `/write` opens one. When the platform's fetch to the container's port is not accepted inside the platform's own allowance — a few seconds, well under the SDK's 30 s connect timeout — it throws a plain `Error`: `Container is taking too long to accept the connection; the application could be overwhelmed with load`. Nothing ran: the SDK was still connecting, before any process was started. The platform's words name a cause it never measured, and the token names the refusal, never its cause: live, the harness's one-second poll of its transcript log met it three minutes into a whole repository's `verify` saturating every core of the item-16 instance, AND on an idle container — a review thread whose every command was a `git clone`, two `git diff`s, a `grep` and a `sed`, each completing in under 31 ms, the runtime accepting a connection 0.9 s before the refused one and 0.8 s after it, the fleet at 7 of 25 and no deploy rolling. Both times the failure reached the bot as an ordinary in-body error (`answered`, item 9), the harness read it as the run's failure, killed the process and tore the workspace down — and the container accepted the kill a second later. The contract: (a) the **Worker** names the refusal only where no process was started — the spawn (`spawnFailure`) and the gate's warm-up — with the token `runtime-busy` in the item-3 dual shape (`runtimeBusyExecAnswer`); the file routes throw the typed `SandboxRuntimeBusyError` and the fetch handler answers HTTP 503 with the token; the recognizer is the platform's wording alone (`isRuntimeBusySignal`, `RUNTIME_BUSY_WORDING`, `src/execution/sandboxErrors.ts`), since the platform gives no type; a failure of a running command's output is never named so — the process exists, and a re-send would run it twice. (b) The **executor** re-sends the identical request on the token exactly as on `fleet-busy` (item 14) and `sandbox-starting` (item 23), after 3 s, 5 s, then 10 s, inside the operation's own budget capped at `RUNTIME_BUSY_WAIT_MAX_MS` (5 min) — when a command of the thread's own holds the container, it is bounded like every command — and throws `ExecCapacityError` naming the refusal when the wait is spent, never `ExecInfraError`; neither the token's explanation nor that message asserts what held the container or tells the reader to wait for a command that may not be running. (c) An older Worker's bare platform text stays an ordinary in-body error after one send: the token decides, never the words. The instance size is not the fix: an idle container has met the refusal, so no size rules it out, and a poll must survive it whatever the size; the SDK's connect timeout is not the knob either, since the platform's allowance runs out first and the SDK exposes none. diff --git a/docs/reference/specs/resident-repos.md b/docs/reference/specs/resident-repos.md index f61fdef96..3993fc302 100644 --- a/docs/reference/specs/resident-repos.md +++ b/docs/reference/specs/resident-repos.md @@ -97,7 +97,7 @@ Base URL for all `[agent]` checks: the resident Worker's own hostname, written ` 66. **A resumed run's attach reuses the thread's worktree as it stands and refuses by name a tree it cannot keep; nothing is wiped** (pure `parseReuse` and `decideWorktree` in `src/execution/residentReuse.ts`, imported by the Worker like `residentReadonly`; the bot side is `ResidentExecutorOptions.reuse`, `ResidentReuseRefusedError` and the factory's `reattachWorkspace`). Background: item 17's discipline treats every tree as a leftover of an earlier run, which is right for a fresh run and wrong for one that comes back after a bot restart ([run-history.md](run-history.md) item 38): the attach the resumed run made wiped its own dirty tree under a still-living pi, or raced that run's in-flight install, failed at `worktree-clean`, and the bot moved the run onto a cold sandbox silently. The shape: the bot sends `reuse: true` on a resume alone (a fresh attach, and an older resident, see the body they always did); the Worker parses it before anything runs (absent → false, a boolean → itself, anything else → 400) and hands it down `attachThread` → `attachThreadBody` → `attachThreadCreate` → `ensureThreadWorktree`, which measures the tree as the thread user (exists, readable, dirty, HEAD; the ancestry probe only for a provisioning attach whose HEAD is not the target) and runs the pure decision: reusing plus readable → `reuse`, whatever the dirt or the HEAD, the tree untouched; reusing plus missing, unreadable or built for the other mode (item 50) → `refuse` with the reason, thrown as `ReuseRefusedError` before the one `rm -rf` is reached and answered 409 `{error: "reuse-refused: ", needs: "recreate"}` after the allocation's rollback; provisioning → item 17 byte for byte. The client types the 409 as `ResidentReuseRefusedError`; the factory turns every failure of a re-attach into `WorkspaceReattachRefusedError` and provisions nothing in its place (item 24). The answer also names the container the tree is in (`container`: the kernel's boot id, read once per incarnation and memoized, so a replaced runtime re-reads it), for the run's row ([harness-pi.md](harness-pi.md) item 8). 67. **A cycle that keeps failing in the resident's own steps heals itself; one that fails in the repository's command keeps parking** (pure `REPO_COMMAND_STEPS`, `isRepoCommandStep`, `stepOfFailedReason`, `parksOnRepeat`, `INFRA_STREAK_RECREATE_AT` = 3, `INFRA_STREAK_DOWN_AT` = 5, `infraStreakRung`, `infraStreakReason` in `src/execution/residentInfraStreak.ts`; `RefreshFailure.step`; the Worker's `refreshFailed`, `noteInfraStreak`, the gate's park condition, `refreshComplete`): the split item 64 drew for one signature, drawn for every refresh failure. A step that runs the repository's own command (the onboard-time command table: `deps-install`, `build`, `test`) failing is evidence about the repository — the head is broken — and nothing a rebuild does changes that: provisioning would run the same command and fail the same way, and a `down` runs no cycle, so the resident would never notice the head being fixed. Those stay `degraded(-failed …)`, the gate's park streak (`DEGRADED_PARK_AFTER_CYCLES`, item 16b) slows them to the idle cadence, and they heal on the first cycle after the head is fixed. Every other step is the resident's machinery — git against the mirror (`fetch`, `checkout-update`), the probes and markers, the restores, a failure between steps (`refresh`) — and a failure there that repeats is the container or the disk gone wrong, which a rebuild does fix. The week before this rule: one resident sat `degraded(deps-install-failed: … configured to use bun)` for three cycles and parked — right, a command-table fix was the cure; another failed one cycle with `checkout-update-failed: fatal: Could not parse object` and self-healed — had it kept failing, nothing would have escalated, ever. **The ladder** over `resident:infraStreak` (`{step, count, firstAt, lastAt}`, one writer `noteInfraStreak`): counts 1–2 record and let the next cycle try (`degraded(-failed …)`, as before); count 3 recreates the container (item 64's rung 3, `recreateContainer`: the VM destroyed, snapshots kept, `degraded(infra-streak: 3 consecutive cycles failed in the resident's own steps (last: , since ) — the container was destroyed, snapshots kept; the next cycle restores from the snapshot)`, and the next cycle's wake restores mirror, checkout and deps from R2 onto a fresh disk); count 4 is the recreated container's one cycle of its own; count ≥ 5 destroys the VM and goes `down(infra-streak: … a recreated container failed the same way; rebuilding)` — a reason `REHYDRATION_FAILURE_RE` matches, so item 36's transition rebuild fires at once, under its budget. A completed cycle, a repo-command failure (our machinery got that far) and the down rung clear the row. `infra-streak:` is a non-evidence reason (`NON_EVIDENCE_REASON`): the gate never parks on it and always runs the next cycle; and the park streak itself now counts only reasons `parksOnRepeat` allows — a repo-command step's failure, or a reason naming no step (`github-unreachable`) — so a resident-step failure is never parked, since parking would only slow its own ladder. A disk-full failure keeps its recovery (item 54), before any counting. **The fetch step names GitHub only for GitHub's failures**: `refreshFetch` used to record every failed mirror fetch as `github-unreachable` — a serviceable reason (item 24) that parks on repeat — so a mirror broken on disk read as GitHub being down and never climbed; the first live row of this item found exactly that (`degraded(github-unreachable: 'origin' does not appear to be a git repository …)`, streak unmoved). Now the pure `fetchFailureIsMirrors` (`MIRROR_FAILURE_WORDING`: the remote gone from `config`, `not a git repository`, `bad object`, `corrupt`, unreadable or empty objects, a `.git/` file it cannot open) routes a fetch that failed on the mirror through `refreshFailed` as the resident step `fetch` — `degraded(fetch-failed: …)`, never serviceable, counted by the ladder; everything else (DNS, timeouts, HTTP errors, refused credentials, a repository GitHub says is not there) stays `github-unreachable`, GitHub's to fix and the resident's to wait out. `/debug info` carries `infraStreak: {step, count, firstAt, lastAt, rung} | null`. Fault injection (admin scope, test residents only): `POST /debug {"op":"infra-streak","count":N}` writes the row (0 deletes it), so one real failure lands on the rung under test; `{"op":"break-mirror"}` removes the mirror's `config` on disk (its remote with it), so the next cycle's `fetch` fails in a resident step — the corrupt disk the recreate rung restores from the snapshot. Not the objects: the wake check (`readyStamp`) tests `objects/` and the checkout's `.git`, and a missing directory is a disk to restore, never a fault that reaches the fetch — the first live attempt healed itself that way before any step failed. 68. **A container that did not accept the connection is a wait on the resident too (`runtime-busy`).** The thread sandbox's [execution.md](execution.md) item 28, on the resident: when the platform's own accept allowance for the SDK's control-port connect runs out before the SDK's 30 s timeout — under a command saturating the cores, and on an idle container too (item 28) — the container port's fetch throws a plain `Error`, `Container is taking too long to accept the connection; the application could be overwhelmed with load` — inside the wake path, before any process started; the token names the refusal, never the cause the platform's words guess at. It was seen on a resident settling after a roll, as `resident /exec:` followed by the platform's words, and it ended the review. It is neither a replacement (the runtime did not change) nor the silent port of item 64 (the container accepts again in moments), so it is counted as neither. The contract: (a) the **Worker** names it at the exec choke point's SPAWN alone (`run()`, after the reset and the replacement checks, before the unreachable count) as the typed `SandboxRuntimeBusyError` (`src/execution/sandboxErrors.ts`, the recognizer `isRuntimeBusySignal` over the cause chain); the thread data plane (`execThreadImpl`, `readThreadFileImpl`, `readThreadBytes`, `writeThreadFileImpl`, and `threadRejectionErr` for a throw that crossed the stub) answers it as `runtimeBusyErr` — a 503 with `reason: "runtime-busy"` and the `system` cause, streamed in-body on `/exec` like every failure — and the catch-all reads the word as the platform's transient; a running command's output failure is never named so, since the process exists and a re-send would run it twice. (b) The **client** (`callWaitingOutBusy`, around every send `opWithReattach` makes) re-sends the identical request on the token after 3 s, 5 s, then 10 s, inside the command's own budget capped at `RUNTIME_BUSY_WAIT_MAX_MS` (5 min), and throws `ExecCapacityError` naming the refusal when the wait is spent — never `ExecInfraError`, never a re-attach (the worktree is as it was); a hard stop ends the pause at once with the run's one typed `aborted` error. (c) An older Worker's bare platform words — the typed 500 with no token — stay an ordinary infra answer after one send: the token decides, never the words. (d) The **refresh cycle**'s own execs meet the same typed word from `run()`, and the cycle **yields** to it: `classifyRefreshFailure` reads the token anywhere in the failure's text (`carriesRuntimeBusyToken` — a `StepError` or the fetch step's mint prefix keeps it; the platform's bare words alone are not the token) as `busy`, `refresh-yielded: …`, decided before the disk inference and with the disk probe skipped; the instance step answers `stopped (runtime-busy)`, releases the cycle lease and puts back the settled state the fetch step recorded under its `refreshing` (`resident:refreshingFrom`, rewritten before every `refreshing` write) — warm, or the degraded an earlier cycle earned — and the Workflow skips the sweep and the measure, which exec on the same container; the next cron firing tries again. Never `degraded`, never a rung of item 67's ladder (three cycles of it once destroyed the container under the very run that kept it busy), never the ladder of item 64, and never a retry into the same busy container. -69. **The fleet drain — a deploy closes the fleet to new runs, the runs in flight end, the swap lands, the containers reconcile, the fleet reopens** ([record 0059](../../decisions/0059-a-deploy-drains-the-resident-fleet.md); [`drain.ts`](../../../deploy/cloudflare-resident/drain.ts), the routes and the gate in [`worker.ts`](../../../deploy/cloudflare-resident/worker.ts), the client's wait in [`resident.ts`](../../../src/execution/resident.ts); the deploy side is [release-and-deploy.md](release-and-deploy.md) item 31). Item 44's preflight refuses while any resident has a run in flight, and under steady traffic — ship children holding a repo for up to their 45-minute lease, new ones starting as old ones end — no minute is ever quiet: three releases of one evening lost five 30-minute waits and left the resident two releases behind the bot. So the wait is made to end by closing the door it waits on. **The drain** is one record in the registry Durable Object — `{ since, until, by, reason }`, key `drain`, outside the `resident:` prefix — set by `POST /drain {minutes?, reason?, by?}` (`minutes` an integer in [1, 90], default 60; a second drain replaces the first) and cleared by `POST /undrain` (idempotent, answers `cleared`) — both of the **drain scope**, with `POST /reconcile` beside them: a fourth bearer, `RESIDENT_DRAIN_TOKEN`, that passes these three routes and nothing else (admin passes them too), so the release deploy holds no more than the drain and its swap need: a leaked CI secret can close a fleet — 90 minutes per post, for as long as its holder keeps posting until the token is rotated — or restart idle stale containers (state the disk never holds), and can never offboard, rebuild or read one. **The reopen comes after the reconcile, never at preflight-ok**: the deploy runner posts `/reconcile` once its Worker deploy landed and before its `/undrain` ([release-and-deploy.md](release-and-deploy.md) item 31) — the handler walks the registry and asks every resident's Durable Object to reconcile its container onto the current image (`reconcileForDeploy`, item 16b's reconcile under the deploy's own name; a failing resident degrades to its own `error` row) — because a fleet reopened seconds after preflight-ok admitted runs into containers the reconcile then restarted under them; a resident that answers `deferred` (something in flight, or a registered run) or `error` restarts on its own next quiet refresh cycle (never on an attach, item 16b / issue 2101 — and while the report is owed every new attach is refused onto the seeded sandbox, so the resident can go quiet); its pending report remains through the stop and is sent only after the replacement's own hydration reaches `warm`, and the reconcile's deferral on a live registration (item 16b) is what keeps even that later restart from landing under a run. `GET /residents` carries it as `draining` (null for none). **The gate** is the Durable Object's, in `attachThreadTraced` after hydration and before the image reconcile: while the drain is in force an attach for a thread with **no run registration** (item 44's row, held from an attach to its release) is answered as the streamed refusal `{ error: "draining: …", status: 503, draining: }` — a new attach is a new run — and an attach for a thread whose run IS registered passes: a rolled container's re-attach (item 65), an evicted worktree's recovery attach, a resumed run's `reuse` attach are the runs the drain waits for, and refusing them would hold the fleet closed on the very run it is closed for. `/exec`, `/read`, `/write`, `/op` and `/detach` are untouched, so the runs in flight finish and release as they always did. (A registration whose release never came — the clean-idle sweep's case — lets one new run of that thread through; thread-scoped and rare, and the sweep clears it. A registry the Durable Object cannot read at the attach is **no drain**, said in the log: a run never fails on a flag it could not read, and the deploy's preflight fails closed on its own side, so the failure lands on the deploy.) **The end is the record's own**: `liveDrain(stored, now)` reads a record whose `until` has passed, or a shape this build cannot read, as no drain — Durable Object storage survives the isolate swap the deploy performs, so a drain nobody lifted must expire by itself, and it does within its own minutes (at most 90). **The client waits** (`ResidentExecutor.attach`): a `draining` answer — read by its record on a 503, never by its words, so it is never the platform's transient of item 27 — is waited through with one re-attach every `DRAIN_POLL_MS` (30 s) under the run's own lease less `DRAIN_LEASE_RESERVE_MS` (10 min, what the attach and the work after it need), `DRAIN_WAIT_MAX_MS` (60 min) with no lease; the run's stop ends the wait at once; the binding carries the wait as `wokeAfterMs`, so the card's opening note says the run started late and for how long; an answer that is no longer the drain is judged as a first answer is (admitted, transient → the wake wait with what the budget left, else the attach's own refusal); the budget ending with the fleet still drained is the typed `ResidentDrainingError` (`refused`), naming how long the run could wait and when the drain says it ends. Inside a wake wait's re-attach (item 65) a `draining` answer hands the wait to this drain wait rather than spending the wake budget on it: the deploy's own isolate swap lands inside its own drain, so an attach that meets the Durable Object reset enters the wake wait — but the drain record survives the swap, the re-attach that follows meets it, and the run then waits out the drain under its lease instead of striking at the wake budget's minute and falling back cold on exactly the requests the drain exists to hold. **The hold has liveness** (issue 2044: the reopen gate's first live deploy held the whole fleet 34 minutes on a container the platform never cycled): the first hold stamps `holdsUntil` — the cycle's measured bound (`DRAIN.cycleBoundMinutes`, item 65's three-to-ten-minute roll), clamped to `until` — and `liveDrain` reads a held record past it as no drain: the fleet reopens by construction, whoever died between the reconcile and the lift, with the registry's alarm (armed at the bound, the earlier end) naming the stale containers in a warning (`staleHolds`); each restarts on its own next quiet refresh cycle. A replacement start, rebuild or fresh provision counts as the report only after it reaches `warm`: the hydration path and `runProvisioning` then report their own fresh container current — each runs on the image the deployed Worker pins by construction — so a mid-drain replacement or rebuild reopens the fleet instead of leaving the hold standing. **The waits are legible** (issue 2044's second and third defects): the client's drain wait paints `waiting for the deploy to finish · N min` through `onSetupNote` on every poll and clears it at the wait's end, the binding carries the drain's share as `drainWaitMs` beside `wokeAfterMs`, and the dispatcher publishes it as the run's `drain_wait` note ([run-friction.md](run-friction.md) reads it as its own category); `GET /residents` carries each resident's `imageReport` (`pending` names a container whose post-deploy report is owed) and `repo list` leads with the drain and marks every row `drained — attach refused`, so `warm` is never printed bare for a repository no attach can reach; a typed `repo test`/`repo build` on a drained fleet is answered `op-refused: the resident fleet is drained …` at once instead of running or hanging. Why the drain lives in the Worker and not the bot: the bot is one of the fleet's callers, not the only one (the CLI, a load harness, a second bot generation mid-roll), and the deploy that needs the door closed runs outside every bot; the registry is the one place every attach already passes. For the orchestration plane ([orchestration-plane.md](orchestration-plane.md) item 9) the registry posts the drain as a `drain` level — the set posts `above` and arms ONE alarm at the record's `until`, `/undrain` posts `below`, and the alarm posts `below` for a drain nobody lifted — each post riding the registry's answers (`planeOutbox`) until superseded, so the plane's `resident-drain` window opens and lifts on the drain's own life. +69. **The fleet drain — a deploy closes the fleet to new runs, the runs in flight end, the swap lands, the containers reconcile, the fleet reopens** ([record 0059](../../decisions/0059-a-deploy-drains-the-resident-fleet.md); [`drain.ts`](../../../deploy/cloudflare-resident/drain.ts), the routes and the gate in [`worker.ts`](../../../deploy/cloudflare-resident/worker.ts), the client's wait in [`resident.ts`](../../../src/execution/resident.ts); the deploy side is [release-and-deploy.md](release-and-deploy.md) item 31). Item 44's preflight refuses while any resident has a run in flight, and under steady traffic — ship children holding a repo for up to their 45-minute lease, new ones starting as old ones end — no minute is ever quiet: three releases of one evening lost five 30-minute waits and left the resident two releases behind the bot. So the wait is made to end by closing the door it waits on. **The drain** is one record in the registry Durable Object — `{ since, until, by, reason }`, key `drain`, outside the `resident:` prefix — set by `POST /drain {minutes?, reason?, by?}` (`minutes` an integer in [1, 90], default 60; a second drain replaces the first) and cleared by `POST /undrain` (idempotent, answers `cleared`) — both of the **drain scope**, with `POST /reconcile` beside them: a fourth bearer, `RESIDENT_DRAIN_TOKEN`, that passes these three routes and nothing else (admin passes them too), so the release deploy holds no more than the drain and its swap need: a leaked CI secret can close a fleet — 90 minutes per post, for as long as its holder keeps posting until the token is rotated — or restart idle stale containers (state the disk never holds), and can never offboard, rebuild or read one. **The reopen comes after the reconcile, never at preflight-ok**: the deploy runner posts `/reconcile` once its Worker deploy landed and before its `/undrain` ([release-and-deploy.md](release-and-deploy.md) item 31) — the handler walks the registry and asks every resident's Durable Object to reconcile its container onto the current image (`reconcileForDeploy`, item 16b's reconcile under the deploy's own name; a failing resident degrades to its own `error` row) — because a fleet reopened seconds after preflight-ok admitted runs into containers the reconcile then restarted under them; a resident that answers `deferred` (something in flight, or a registered run) or `error` restarts on its own next quiet refresh cycle (never on an attach, item 16b / issue 2101 — and while the report is owed every new attach is refused onto the seeded sandbox, so the resident can go quiet); its pending report remains through the stop and is sent only after the replacement's own hydration reaches `warm`, and the reconcile's deferral on a live registration (item 16b) is what keeps even that later restart from landing under a run. `GET /residents` carries it as `draining` (null for none). **The gate** is the Durable Object's, in `attachThreadTraced` after hydration and before the image reconcile: while the drain is in force an attach for a thread with **no run registration** (item 44's row, held from an attach to its release) is answered as the streamed refusal `{ error: "draining: …", status: 503, draining: }` — a new attach is a new run — and an attach for a thread whose run IS registered passes: a rolled container's re-attach (item 65), an evicted worktree's recovery attach, a resumed run's `reuse` attach are the runs the drain waits for, and refusing them would hold the fleet closed on the very run it is closed for. `/exec`, `/read`, `/write`, `/op` and `/detach` are untouched, so the runs in flight finish and release as they always did. (A registration whose release never came — the clean-idle sweep's case — lets one new run of that thread through; thread-scoped and rare, and the sweep clears it. A registry the Durable Object cannot read at the attach is **no drain**, said in the log: a run never fails on a flag it could not read, and the deploy's preflight fails closed on its own side, so the failure lands on the deploy.) **The end is the record's own**: `liveDrain(stored, now)` reads a record whose `until` has passed, or a shape this build cannot read, as no drain — Durable Object storage survives the isolate swap the deploy performs, so a drain nobody lifted must expire by itself, and it does within its own minutes (at most 90). **The client waits** (`ResidentExecutor.attach`): a `draining` answer — read by its record on a 503, never by its words, so it is never the platform's transient of item 27 — is waited through with one re-attach every `DRAIN_POLL_MS` (30 s) under the run's own lease less `DRAIN_LEASE_RESERVE_MS` (10 min, what the attach and the work after it need), `DRAIN_WAIT_MAX_MS` (60 min) with no lease — **and under the fallback's own cost where one stands behind the attach** (issue 2101: a run refused by the drain waited 18 minutes at its attach for the whole deploy — the preflight's wait, the swap, the image gate — then did the job in a seeded sandbox that stands up in about two): the factory's first attach passes `drainBoundMs` = `DRAIN_FALLBACK_WAIT_MS` (`DRAIN.fallbackWaitMs`, 3 min), and the first drain turns that bound into one absolute deadline carried through every drain/wake hand-off, so a drain refusal falls to the seed after at most a few minutes even across repeated deploy resets, while a resumed run's re-attach and a mid-run recovery — which have no fallback, their worktree lives on the resident or nowhere — pass none and keep the lease's bound; every error thrown out of the wait carries the drain's share (`drainWaitOf`), so the fallback's selection still publishes the run's `drain_wait` note; the run's stop ends the wait at once; the binding carries the wait as `wokeAfterMs`, so the card's opening note says the run started late and for how long; an answer that is no longer the drain is judged as a first answer is (admitted, transient → the wake wait with what the budget left, else the attach's own refusal); the budget ending with the fleet still drained is the typed `ResidentDrainingError` (`refused`), naming how long the run could wait and when the drain says it ends. Inside a wake wait's re-attach (item 65) a `draining` answer hands the wait to this drain wait rather than spending the wake budget on it: the deploy's own isolate swap lands inside its own drain, so an attach that meets the Durable Object reset enters the wake wait — but the drain record survives the swap, the re-attach that follows meets it, and the run then waits out the drain under its lease instead of striking at the wake budget's minute and falling back cold on exactly the requests the drain exists to hold. **The hold has liveness** (issue 2044: the reopen gate's first live deploy held the whole fleet 34 minutes on a container the platform never cycled): the first hold stamps `holdsUntil` — the cycle's measured bound (`DRAIN.cycleBoundMinutes`, item 65's three-to-ten-minute roll), clamped to `until` — and `liveDrain` reads a held record past it as no drain: the fleet reopens by construction, whoever died between the reconcile and the lift, with the registry's alarm (armed at the bound, the earlier end) naming the stale containers in a warning (`staleHolds`); each restarts on its own next quiet refresh cycle. A replacement start, rebuild or fresh provision counts as the report only after it reaches `warm`: the hydration path and `runProvisioning` then report their own fresh container current — each runs on the image the deployed Worker pins by construction — so a mid-drain replacement or rebuild reopens the fleet instead of leaving the hold standing. **The waits are legible** (issue 2044's second and third defects; issue 2101's third: 18 minutes with zero events between the attach span's start and its end): the client's drain wait paints `waiting for the deploy to finish · N min` through `onSetupNote` on every poll and clears it at the wait's end; the drain wait and the attach's wake wait are each one child span under the attach span (`dispatch.workspace.attach.drain-wait`, `dispatch.workspace.attach.wake-wait`, named on the run page by `ATTACH_WAIT_NAMES` in `displayNames.ts`), failing with the typed error's classification — the refusal's words stay on the card note and the attach error, never on a span ([tracing.md](tracing.md) item 2); the binding carries the drain's share as `drainWaitMs` beside `wokeAfterMs` — and the sandbox fallback after a drained attach carries it on the selection (`ExecutorSelection.drainWaitMs`) — and the dispatcher publishes either as the run's `drain_wait` note ([run-friction.md](run-friction.md) reads it as its own category); `GET /residents` carries each resident's `imageReport` (`pending` names a container whose post-deploy report is owed) and `repo list` leads with the drain and marks every row `drained — attach refused`, so `warm` is never printed bare for a repository no attach can reach; a typed `repo test`/`repo build` on a drained fleet is answered `op-refused: the resident fleet is drained …` at once instead of running or hanging. Why the drain lives in the Worker and not the bot: the bot is one of the fleet's callers, not the only one (the CLI, a load harness, a second bot generation mid-roll), and the deploy that needs the door closed runs outside every bot; the registry is the one place every attach already passes. For the orchestration plane ([orchestration-plane.md](orchestration-plane.md) item 9) the registry posts the drain as a `drain` level — the set posts `above` and arms ONE alarm at the record's `until`, `/undrain` posts `below`, and the alarm posts `below` for a drain nobody lifted — each post riding the registry's answers (`planeOutbox`) until superseded, so the plane's `resident-drain` window opens and lifts on the drain's own life. 70. **The resident guards its own memory** (pure [`memoryGuard.ts`](../../../deploy/cloudflare-resident/memoryGuard.ts); the wiring in `worker.ts`; the token `MEMORY_PRESSURE_REASON` in `src/execution/sandboxErrors.ts`, read by both sides like `runtime-busy`). A container at its cgroup memory cap wedges its own control port and every run on it loses its work, and before this item no log line carried a memory number. **The reader**: one cgroup v2 reading — `memory.current`, `memory.max`, `cpu.stat`'s `usage_usec` from `/sys/fs/cgroup` — taken as ONE exec (`CGROUP_READ_ARGV` through the exec choke point `run()`, so it shares item 68's classification) behind the small `CgroupReader` interface (a fake in tests), at every `/exec` and `/attach` start and on the measure step of the refresh instance the watchdog cron creates (item 7 — the tick that exists; the watchdog's own pass reads storage alone, item 55's rule, and a sleeping container is never woken to be measured). Each sample is ONE structured log line — used bytes, cap bytes, percent — so Workers Logs has the number; no polling loop anywhere. **The gate**, two named constants over the last reading: at or above `MEMORY_SOFT_LIMIT_PCT` (80) a NEW `/attach` is refused `503 {error: "memory-pressure: resident near its memory cap, N% used (… bytes), queued …", state, stateReason, reason: "memory-pressure"}` — the same shape as `mirror-busy`, so the bot falls back cold or waits legibly and the card says why (item 24; `ANSWER_OWN_WORDS` keeps the token out of the lifecycle reading); at or above `MEMORY_HARD_LIMIT_PCT` (90) a NEW `/exec` is refused by name with the same numbers — while every command already running finishes, never killed: the gate sits at each route's start (exec before its preflight; attach after the drain gate and before the image reconcile, so a refusal never restarts a container) and kills nothing. A run already in flight — registered from attach to release (item 44) — re-attaches THROUGH the attach gate exactly as it passes the drain (one registration read decides both): a rolled container's re-attach, an evicted worktree's recovery and a resume belong to the runs the gate holds the resident open for, and refusing them would strand the warm worktree and lose the very work the gate protects; the sample (and its log line) is still taken. **A reading gates only the container it measured**: an inactive runtime invalidates the guard's reading before the gate asks (`sampleMemory` → `invalidate()` — a container that no longer runs holds no memory, and a fresh hydration memo can skip the restore that would wake it), and a typed runtime replacement under the probe (`RuntimeReplacedError` from `run()`) travels as `MemorySampleUnavailable(…, invalidates)` and forgets the reading too, so the route falls through to its own `runtime-replaced` answer instead of a `memory-pressure` off a dead container's numbers. An UNREADABLE cgroup (a missing file, unparsable output) logs once and disables the gate for the incarnation rather than refusing everything; any other transient read failure (the container busy under the probe, `MemorySampleUnavailable`) keeps the last reading and the gate. **The gauges**: the last reading is persisted (`resident:memory`) and carried on `GET /status` (`memory`), `GET /residents` (`getResidentInfo`) and the watchdog answer, so the residents page and the bot can show it; `GET /healthz` deliberately stays without it — it touches no DO (item 1), and the reading is per-resident. The soft side is also the resident's `memory` level for the orchestration plane ([orchestration-plane.md](orchestration-plane.md) item 9): at or past the soft line the level reads `above`, carried on every answer's `levels` document. ## Validation criteria @@ -113,6 +113,7 @@ Base URL for all `[agent]` checks: the resident Worker's own hostname, written ` | 69 (issue 2044): the hold's liveness — the first hold stamps the cycle bound clamped to `until` and later holds keep it, a held record past the bound is no drain (liftAsked or not), `staleHolds` names the reopen's stale containers and nothing otherwise, a pre-bound record keeps `until` as its only end, and the incident's replay (drain, one held container that never reports, the lift asked, the bound elapsing) ends in a reopened fleet with the container named | `[unit]` `deploy/cloudflare-resident/drain.test.ts::the hold's liveness — a cycle that never happens reopens the fleet with the container named, never a silence to the record's own until (issue 2044)::*` | | 69 (issue 2044): the wiring — `holdDrainFor` stamps the bound and arms the registry alarm at it, the alarm's reopen warns naming the stale containers, a fresh provision reports its own container current and clears the marker, `/op` answers a drained fleet with `op-refused: … drained …` before any work, and the admin view carries `imageReport` | `[unit]` `deploy/cloudflare-resident/drain.test.ts::the Worker's wiring (by scan)::the hold's liveness is wired (issue 2044)…` | | 69 (issue 2044): the waits are legible — the drain wait paints `waiting for the deploy to finish · N min` through `onSetupNote` on each poll and clears it at the wait's end, the binding carries `drainWaitMs` beside `wokeAfterMs` from the first draining answer and from the wake path's drain hand-off alike, the dispatcher publishes it as the `drain_wait` run note with the rounded minutes, and `repo list` leads with the drain and marks every row instead of printing a bare `warm` | `[unit]` `src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::the wait says so on the card (issue 2044)…`, `::ResidentExecutor.attach during a fleet drain (item 69)::a drained fleet is waited for…`, `::ResidentExecutor.attach during a fleet drain (item 69)::a Durable Object reset that lands during a live drain…`, `src/core/dispatcher.test.ts::executor provisioning by agent resources::a binding that carries drainWaitMs publishes the run's drain_wait note…`, `src/core/commands/repo.test.ts::repo.list::issue 2044: a drained fleet leads the reply…` | +| 69 (issue 2101): the drain wait is bounded by the fallback's cost and visible — a first attach with a fallback behind it passes `drainBoundMs`; the first drain turns it into one absolute deadline carried through every drain/wake hand-off, so repeated deploy resets cannot renew the allowance, and the wait ends in the typed error after `DRAIN_FALLBACK_WAIT_MS`, never the deploy's whole length, the error carrying the full drain share; the factory's fallback selection carries `drainWaitMs` and the dispatcher publishes the `drain_wait` note from it; the drain wait and the attach's wake wait are each one child span under the attach span, ending ok when the fleet reopens and failing with the typed classification from every stop point, including the wake's initial probe, never the refusal's free text; both span names have display names | `[unit]` `src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::a fallback bounds the drain wait by its own cost (issue 2101)…`, `::ResidentExecutor.attach during a fleet drain (item 69)::one fallback deadline survives drain/wake hand-offs…`, `::ResidentExecutor.attach during a fleet drain (item 69)::the drain wait is one span under the attach span (issue 2101)…`, `::ResidentExecutor.attach during a fleet drain (item 69)::a drain wait that runs out fails its span with the typed classification and no remote words (issue 2101)`, `::ResidentExecutor.attach during a fleet drain (item 69)::the wake wait after a transient refusal is its own span under the attach span (issue 2101)…`, `::ResidentExecutor.attach during a fleet drain (item 69)::a stop during the wake wait's initial status probe closes the wake span as failed`, `src/execution/factory.test.ts::makeExecutor resident selection::a drained fleet's first attach falls to the sandbox after the fallback's own bound…`, `src/core/dispatcher.test.ts::executor provisioning by agent resources::a selection that carries drainWaitMs without a binding…`, `src/core/trace/displayNames.test.ts::display names::the attach's drain and wake waits have display names under the attach prefix` | | 69: the client — a `draining` answer is keyed on the record, not the words; a drained fleet is waited for with one re-attach per poll and the binding carries the wait; the wait is the run's lease less the reserve and ends in the typed `ResidentDrainingError` naming the wait and the drain's end; an answer that is no longer the drain ends the wait as the attach's own refusal; a Durable Object reset met during a live drain enters the wake wait but its re-attach's `draining` answer hands the wait to the drain wait — the lease's budget and the typed error, never the wake budget's strike and a cold fallback; the run's stop ends it at once | `[unit]` `src/execution/resident.test.ts::ResidentExecutor.attach during a fleet drain (item 69)::*` | | 69: live — a deploy under running work: the resident step logs `fleet drained … until …`, a run asked during the drain shows `attaching the workspace…` until the deploy lands and then a note `after waiting Ns for the resident`, the runs that were in flight finish, the step deploys once `/residents` reads zero in flight, and `fleet reopened` closes it | `[agent]` On a release that lands while a coding child is live: read the deploy job's resident lines and the card of a review asked mid-drain. | | 65: the refusals that name a rolling container (the incident's `The container just exited`, `Container is starting`, the runtime-replacement wordings, attach's `image-stale:`) are recognized, and none of the refusals other rules own | `[unit]` `src/execution/residentWake.test.ts::isContainerRolling: the refusals that name a container gone for a moment::*` | diff --git a/src/core/budgets.ts b/src/core/budgets.ts index c0decd52f..814defbc5 100644 --- a/src/core/budgets.ts +++ b/src/core/budgets.ts @@ -104,11 +104,18 @@ export const HARNESS_PROBE_WAIT_MS = 5 * MINUTE_MS; * `maxMinutes` so one nobody lifted is an hour and a half, not a day. A run * asked during a drain waits at its attach one `pollMs` at a time under its * own lease less `leaseReserveMs` (what the attach and the work after it - * need), `waitMaxMs` with no lease to clip it. */ + * need), `waitMaxMs` with no lease to clip it — unless a fallback stands + * behind the attach: `fallbackWaitMs` bounds the wait by the FALLBACK's own + * cost, never the deploy's (issue 2101: a run refused by the drain waited + * 18 minutes for the whole deploy, then did the job in a seeded sandbox that + * stands up in about two), so a drain refusal on a first attach falls to the + * seeded sandbox after at most a few minutes; a re-attach with no fallback — + * a resumed run, a mid-run recovery — keeps the lease's bound. */ export const DRAIN = { pollMs: 30_000, waitMaxMs: 60 * MINUTE_MS, leaseReserveMs: 10 * MINUTE_MS, + fallbackWaitMs: 3 * MINUTE_MS, deployWaitMaxMs: 60 * MINUTE_MS, marginMinutes: 5, maxMinutes: 90, diff --git a/src/core/dispatcher.test.ts b/src/core/dispatcher.test.ts index 6072dcdbc..70987588b 100644 --- a/src/core/dispatcher.test.ts +++ b/src/core/dispatcher.test.ts @@ -993,6 +993,30 @@ describe("executor provisioning by agent resources", () => { expect(quietEvents.some((e) => e.type === "run_note" && e.kind === "drain_wait")).toBe(false); }); + // Feature: docs/reference/specs/resident-repos.md item 69 (issue 2101) — a run + // the drain sent to the sandbox fallback carries the wait on the SELECTION + // (no binding exists), and the dispatcher publishes the same `drain_wait` + // note: the incident's run fell cold and `runs friction` counted zero. + it("a selection that carries drainWaitMs without a binding — the drain's sandbox fallback — publishes the run's drain_wait note too", async () => { + vi.stubEnv("SANDBOX_TOKEN", "tok"); + vi.stubEnv("GITHUB_APP_ID", ""); + const registry = new RunRegistry({ genId: () => "run-fell-cold", genToken: () => "tok" }); + const deps = makeDeps(REMOTE_YAML_FIXTURE, capturingProvider()); + deps.runRegistry = registry; + const fake = { exec: async () => "", readFile: async () => "", writeFile: async () => "" }; + vi.mocked(makeExecutor).mockImplementationOnce(async () => ({ + executor: fake, + backend: "sandbox" as const, + note: "resident attach failed (… drained …) — using fresh sandbox", + drainWaitMs: 3 * 60_000, + })); + await dispatch(deps, msg("agent:coding fix it", "slack:UADMIN"), fakeIO().io); + const events = registry.snapshot("run-fell-cold", "tok")!.events; + expect(events.find((e) => e.type === "run_note" && e.kind === "drain_wait")).toMatchObject({ + summary: "waited 3 min at the fleet drain for a deploy to finish", + }); + }); + // Feature: docs/reference/specs/tracing.md item 19 — a resident attach that FAILS still // grafts the steps it ran under the (failed) attach span; no run exists, so // they reach the process sinks. diff --git a/src/core/dispatcher.ts b/src/core/dispatcher.ts index 66f65ae81..6b1ed29bb 100644 --- a/src/core/dispatcher.ts +++ b/src/core/dispatcher.ts @@ -1647,8 +1647,10 @@ export async function dispatch( // the note is what `runs friction` reads as the wait's category and the // plane's table shows as the run's cause — a half-hour wait with a card // that counted "attaching the workspace…" and a friction verdict of - // "none" was the incident's shape. - const drainWaitMs = round.selection.binding?.drainWaitMs ?? 0; + // "none" was the incident's shape. A run the drain sent to the sandbox + // fallback carries the wait on the selection instead of a binding (issue + // 2101: the incident's run fell cold and `runs friction` counted zero). + const drainWaitMs = round.selection.binding?.drainWaitMs ?? round.selection.drainWaitMs ?? 0; if (drainWaitMs > 0) registry.publish(run.id, { type: "run_note", diff --git a/src/core/trace/displayNames.test.ts b/src/core/trace/displayNames.test.ts index dbf6c4b9d..fdd1b4acc 100644 --- a/src/core/trace/displayNames.test.ts +++ b/src/core/trace/displayNames.test.ts @@ -41,6 +41,14 @@ describe("display names", () => { // every streamed prefix family has a rule for (const prefix of Object.keys(STREAMED_PREFIXES)) expect(displayNameOf(`${prefix}x`)).not.toBe(`${prefix}x`); }); + + // The bot client's own attach waits (issue 2101): named beside the Worker's + // step vocabulary, never inside it — the step table is scanned against the + // Worker's source, and these spans are the bot's. + it("the attach's drain and wake waits have display names under the attach prefix", () => { + expect(displayNameOf("dispatch.workspace.attach.drain-wait")).toBe("waiting for the deploy to finish"); + expect(displayNameOf("dispatch.workspace.attach.wake-wait")).toBe("waiting for the resident to wake"); + }); }); // The resident Worker names each command it runs (`runOk(argv, "", …)`, diff --git a/src/core/trace/displayNames.ts b/src/core/trace/displayNames.ts index 2c7561f80..60b7d26b9 100644 --- a/src/core/trace/displayNames.ts +++ b/src/core/trace/displayNames.ts @@ -46,6 +46,15 @@ export const DISPLAY_NAMES = { /** The fallback for a prefixed span whose leaf we cannot name. */ export const GENERIC_STEP_NAME = "a Switchboard step"; +/** The bot client's own attach waits (issue 2101): spans under the attach + * span, named here rather than in the resident step table — that table is + * the Worker's vocabulary, scanned against its source, and these waits run + * in the bot. */ +export const ATTACH_WAIT_NAMES = { + "drain-wait": "waiting for the deploy to finish", + "wake-wait": "waiting for the resident to wake", +} as const; + /** The display name for any span name: the table for an enumerated name; for a * prefix family the tool's own name (`tool.bash` → `bash`), the MCP tool's own * name (`mcp..` → ``), or the resident step's label, with @@ -59,7 +68,11 @@ export function displayNameOf(name: string): string { return leaf && leaf !== "mcp" ? leaf : GENERIC_STEP_NAME; } for (const prefix of ["dispatch.workspace.attach.", "run.command."]) { - if (name.startsWith(prefix)) return residentStepLabel(name.slice(prefix.length)) ?? GENERIC_STEP_NAME; + if (name.startsWith(prefix)) { + const leaf = name.slice(prefix.length); + if (leaf in ATTACH_WAIT_NAMES) return ATTACH_WAIT_NAMES[leaf as keyof typeof ATTACH_WAIT_NAMES]; + return residentStepLabel(leaf) ?? GENERIC_STEP_NAME; + } } return GENERIC_STEP_NAME; } diff --git a/src/execution/factory.test.ts b/src/execution/factory.test.ts index 3bbacd7e8..b369aab22 100644 --- a/src/execution/factory.test.ts +++ b/src/execution/factory.test.ts @@ -7,7 +7,7 @@ import { AGENTS, type AgentDef } from "../agents/registry.js"; import { declaredProfile } from "../config/profile.js"; import { CloudflareSandboxExecutor } from "./cloudflareSandbox.js"; import { ExecInfraError, LocalExecutor } from "./executor.js"; -import { ResidentExecutor, ResidentNeedsRefError } from "./resident.js"; +import { DRAIN_FALLBACK_WAIT_MS, DRAIN_POLL_MS, ResidentExecutor, ResidentNeedsRefError } from "./resident.js"; import { gitIdentityEnvs, makeExecutor, @@ -866,6 +866,42 @@ describe("makeExecutor resident selection", () => { // /status probe and /attach (503 mirror-busy, 429 pool-exhausted). Any attach // failure that is NOT needs-ref must fall back to the per-thread backend with // a NAMED note — never a silent stall or a raw ⚠️. + // Issue 2101: the first attach has the sandbox fallback below to fall to, so + // a drained fleet is waited for under the FALLBACK's own cost — never the + // deploy's — and the drain's wait rides the selection so the run's + // `drain_wait` note still publishes (the incident's run counted zero). + it("a drained fleet's first attach falls to the sandbox after the fallback's own bound, the wait riding the selection as drainWaitMs (issue 2101)", async () => { + vi.useFakeTimers(); + try { + stubEnvs(); + const draining = { + status: 503, + body: { + error: "draining: the resident fleet is closed to new runs for deploy 62e4e9a — the run waits at its attach", + status: 503, + draining: { since: "2026-09-18T05:00:00.000Z", until: "2026-09-18T06:00:00.000Z", by: "deploy all" }, + }, + }; + const polls = DRAIN_FALLBACK_WAIT_MS / DRAIN_POLL_MS; + const { calls } = stubFetch( + { body: { state: "warm", reason: "" } }, + ...Array.from({ length: polls + 1 }, () => draining), + ); + const p = makeExecutor(residentOpts(), repoCtx()); + await vi.advanceTimersByTimeAsync(DRAIN_FALLBACK_WAIT_MS + 1_000); + const { executor, resident, note, drainWaitMs } = await p; + expect(executor).toBeInstanceOf(CloudflareSandboxExecutor); + expect(resident).toBeFalsy(); + expect(drainWaitMs).toBe(DRAIN_FALLBACK_WAIT_MS); + expect(note).toMatch( + /^resident attach failed \(.*did not reopen within the 180s this run could wait.*\) — using fresh sandbox$/, + ); + expect(calls).toEqual(["/status", ...Array.from({ length: polls + 1 }, () => "/attach")]); + } finally { + vi.useRealTimers(); + } + }); + it("warm probe then a NON-needs-ref attach failure → per-thread executor WITH a named 'resident attach failed' note", async () => { stubEnvs(); const { calls } = stubFetch( diff --git a/src/execution/factory.ts b/src/execution/factory.ts index 5d9219128..24d9a3e90 100644 --- a/src/execution/factory.ts +++ b/src/execution/factory.ts @@ -23,6 +23,8 @@ import { type SeededSandbox, } from "./seedPlan.js"; import { + DRAIN_FALLBACK_WAIT_MS, + drainWaitOf, ResidentExecutor, ResidentLeaseSpentError, ResidentNeedsRefError, @@ -303,6 +305,11 @@ export interface ExecutorSelection { * path — the dispatcher names the path to the model and checks the sha * against the PR head before a review runs. Unset on every other path. */ binding?: ResidentBinding; + /** The drain's share of a failed attach's wait (issue 2101): on the sandbox + * fallback after a drained fleet refused the run, so the dispatcher still + * publishes the run's `drain_wait` note — the resident path carries it on + * the binding instead. */ + drainWaitMs?: number; } // Resident lifecycle states the bot attaches in — `isServiceable` in @@ -403,6 +410,8 @@ export async function makeExecutor( let reason: string | undefined; /** The steps of a resident attach that failed before the sandbox fallback. */ let failedAttach: ResidentTrace | undefined; + /** The drain's share of a failed attach's wait, for the run's `drain_wait` note. */ + let drainWaitMs: number | undefined; if (ctx.repo && opts.execution?.resident) { const resident = opts.execution.resident; const tokenEnv = resident.tokenEnv ?? "RESIDENT_OPERATOR_TOKEN"; @@ -495,6 +504,10 @@ export async function makeExecutor( signal: ctx.stopSignal, budgetMs: Math.max(0, FIRST_ATTACH_WAIT_MS - probeWaitMs), waitedMs: probeWaitMs, + // A drained fleet is waited for under the FALLBACK's own cost, not + // the deploy's (issue 2101): this attach has the seeded sandbox + // below to fall to, so the drain wait ends in minutes. + drainBoundMs: DRAIN_FALLBACK_WAIT_MS, }, ); // Item 27: the wait is on the card whichever state the restore landed on. @@ -512,6 +525,7 @@ export async function makeExecutor( // is that failure, and falls cold as it always did. if (isRunStopError(err)) throw err; failedAttach = residentTraceOf(err); + drainWaitMs = drainWaitOf(err); // Item 27: an attach that fails after a wait names the wait too — the // probe's through a typed blip (item 9) and the restore's alike, so a // run that started late says why even when it then fell cold. @@ -561,6 +575,7 @@ export async function makeExecutor( backend: perThreadBackend(opts), seeded: outcome.seeded, ...(failedAttach ? { trace: failedAttach.steps } : {}), + ...(drainWaitMs !== undefined ? { drainWaitMs } : {}), }; } return { @@ -568,6 +583,7 @@ export async function makeExecutor( note: `${reason} — using fresh sandbox${outcome ? ` (${outcome.why})` : ""}`, backend: perThreadBackend(opts), ...(failedAttach ? { trace: failedAttach.steps } : {}), + ...(drainWaitMs !== undefined ? { drainWaitMs } : {}), }; } } @@ -823,8 +839,9 @@ async function openResident( * stop, which ends the wait at once, and the budget past which the attach * fails with the wake's strike; `waitedMs` is a wait the caller already * spent before this attach (the selection probe's), added to the total the - * card names. */ - wait: { signal?: AbortSignal; budgetMs?: number; waitedMs?: number } = {}, + * card names; `drainBoundMs` bounds a drained fleet's wait by the caller's + * fallback cost (issue 2101). */ + wait: { signal?: AbortSignal; budgetMs?: number; waitedMs?: number; drainBoundMs?: number } = {}, ): Promise { let executor = new ResidentExecutor(opts); let binding: ResidentBinding; diff --git a/src/execution/resident.test.ts b/src/execution/resident.test.ts index 50de6a68e..a91bbf2c8 100644 --- a/src/execution/resident.test.ts +++ b/src/execution/resident.test.ts @@ -18,6 +18,8 @@ import { ResidentReuseRefusedError, DRAIN_LEASE_RESERVE_MS, DRAIN_POLL_MS, + DRAIN_FALLBACK_WAIT_MS, + drainWaitOf, ResidentDrainingError, isDrainingRefusal, } from "./resident.js"; @@ -2700,4 +2702,146 @@ describe("ResidentExecutor.attach during a fleet drain (item 69)", () => { expect(err).toBeInstanceOf(Error); expect(calls).toHaveLength(1); }); + + // Issue 2101: a run refused by the drain waited 18 minutes for the whole + // deploy, then did the job in a seeded sandbox that stands up in about two. + // Where a fallback stands behind the attach, its cost bounds the wait. + it("a fallback bounds the drain wait by its own cost (issue 2101): drainBoundMs ends the wait in the typed error after minutes, with most of the lease left, and the error carries the drain's share", async () => { + const polls = DRAIN_FALLBACK_WAIT_MS / DRAIN_POLL_MS; + stubFetch(...Array.from({ length: polls + 2 }, () => draining)); + const ex = new ResidentExecutor({ ...OPTS, remainingMs: () => 45 * 60_000 }); + const p = ex.attach(undefined, { drainBoundMs: DRAIN_FALLBACK_WAIT_MS }).catch((e: unknown) => e); + await vi.advanceTimersByTimeAsync(DRAIN_FALLBACK_WAIT_MS + 1_000); + const err = await p; + expect(err).toBeInstanceOf(ResidentDrainingError); + expect((err as Error).message).toContain( + `did not reopen within the ${DRAIN_FALLBACK_WAIT_MS / 1000}s this run could wait`, + ); + // The drain's share rides the error, so the factory's fallback still + // publishes the run's `drain_wait` note (the incident counted zero). + expect(drainWaitOf(err)).toBe(DRAIN_FALLBACK_WAIT_MS); + }); + + it("one fallback deadline survives drain/wake hand-offs, so a second drain gets only the first wait's remaining allowance", async () => { + const transient = { + body: { + error: "attach-failed: Durable Object reset because its code was updated.", + status: 500, + transient: true, + }, + }; + const firstDrainPolls = DRAIN_FALLBACK_WAIT_MS / DRAIN_POLL_MS - 2; + const { calls } = stubFetch( + draining, + ...Array.from({ length: firstDrainPolls }, () => draining), + transient, + { body: { state: "warm", reason: "", inFlight: 0 } }, + draining, + draining, + ); + const p = new ResidentExecutor(OPTS) + .attach(undefined, { drainBoundMs: DRAIN_FALLBACK_WAIT_MS }) + .catch((e: unknown) => e); + + await vi.advanceTimersByTimeAsync(DRAIN_FALLBACK_WAIT_MS); + const err = await p; + + expect(err).toBeInstanceOf(ResidentDrainingError); + expect(drainWaitOf(err)).toBe(DRAIN_FALLBACK_WAIT_MS); + expect(calls.map(route)).toEqual([ + "/attach", + ...Array.from({ length: firstDrainPolls + 1 }, () => "/attach"), + "/status", + "/attach", + "/attach", + ]); + }); + + it("the drain wait is one span under the attach span (issue 2101): dispatch.workspace.attach.drain-wait ends ok when the fleet reopens, its duration the wait's own", async () => { + const log = recordingSink(); + const root = createTracer({ clock: () => Date.now() }).start("request", { sinks: [log] }); + const attachSpan = root.start("dispatch.workspace.attach"); + stubFetch(draining, draining, { raw: "\n" + JSON.stringify(ATTACH_OK) }); + const p = new ResidentExecutor(OPTS).attach(attachSpan); + await vi.advanceTimersByTimeAsync(2 * DRAIN_POLL_MS); + await p; + const span = log.ended("dispatch.workspace.attach.drain-wait"); + expect(span).toMatchObject({ parentSpanId: attachSpan.id, status: "ok", durationMs: 2 * DRAIN_POLL_MS }); + }); + + it("a drain wait that runs out fails its span with the typed classification and no remote words (issue 2101)", async () => { + const log = recordingSink(); + const root = createTracer({ clock: () => Date.now() }).start("request", { sinks: [log] }); + const attachSpan = root.start("dispatch.workspace.attach"); + const polls = DRAIN_FALLBACK_WAIT_MS / DRAIN_POLL_MS; + stubFetch(...Array.from({ length: polls + 2 }, () => draining)); + const p = new ResidentExecutor(OPTS) + .attach(attachSpan, { drainBoundMs: DRAIN_FALLBACK_WAIT_MS }) + .catch((e: unknown) => e); + await vi.advanceTimersByTimeAsync(DRAIN_FALLBACK_WAIT_MS + 1_000); + await p; + const span = log.ended("dispatch.workspace.attach.drain-wait"); + // The typed error's classification, never the refusal's free text: the + // words stay on the card note and the attach error (tracing.md item 2). + expect(span).toMatchObject({ status: "error", errorKind: "infra", errorCode: "attach" }); + expect(JSON.stringify(span)).not.toContain("draining:"); + }); + + it("the wake wait after a transient refusal is its own span under the attach span (issue 2101): dispatch.workspace.attach.wake-wait", async () => { + const log = recordingSink(); + const root = createTracer({ clock: () => Date.now() }).start("request", { sinks: [log] }); + const attachSpan = root.start("dispatch.workspace.attach"); + stubFetch( + { body: { error: "attach-failed: Network connection lost.", status: 500, transient: true } }, + { body: { state: "warm", reason: "", inFlight: 0 } }, + { raw: "\n" + JSON.stringify(ATTACH_OK) }, + ); + const p = new ResidentExecutor(OPTS).attach(attachSpan); + await vi.advanceTimersByTimeAsync(1_000); + await p; + const span = log.ended("dispatch.workspace.attach.wake-wait"); + expect(span).toMatchObject({ parentSpanId: attachSpan.id, status: "ok" }); + }); + + it("a stop during the wake wait's initial status probe closes the wake span as failed", async () => { + const calls: Array<{ url: string; init: RequestInit }> = []; + vi.stubGlobal( + "fetch", + vi.fn((url: unknown, init?: RequestInit) => { + calls.push({ url: String(url), init: init ?? {} }); + if (calls.length === 1) { + return Promise.resolve( + new Response( + JSON.stringify({ error: "attach-failed: Network connection lost.", status: 500, transient: true }), + { status: 200 }, + ), + ); + } + return new Promise((_resolve, reject) => { + const signal = init?.signal; + if (signal?.aborted) reject(signal.reason); + else signal?.addEventListener("abort", () => reject(signal.reason), { once: true }); + }); + }), + ); + const log = recordingSink(); + const root = createTracer({ clock: () => Date.now() }).start("request", { sinks: [log] }); + const attachSpan = root.start("dispatch.workspace.attach"); + const control = new AbortController(); + const outcome = new ResidentExecutor(OPTS).attach(attachSpan, { signal: control.signal }).catch((e: unknown) => e); + await vi.advanceTimersByTimeAsync(0); + expect(calls.map(route)).toEqual(["/attach", "/status"]); + + control.abort(); + await vi.advanceTimersByTimeAsync(0); + const err = await outcome; + + expect(err).toBeInstanceOf(ExecInfraError); + expect((err as ExecInfraError).reason).toBe("aborted"); + expect(log.ended("dispatch.workspace.attach.wake-wait")).toMatchObject({ + parentSpanId: attachSpan.id, + status: "error", + errorKind: "transport", + }); + }); }); diff --git a/src/execution/resident.ts b/src/execution/resident.ts index 63d86894a..ee91abc3b 100644 --- a/src/execution/resident.ts +++ b/src/execution/resident.ts @@ -189,6 +189,31 @@ export const DRAIN_WAIT_MAX_MS: number = DRAIN.waitMaxMs; /** What the wait leaves of the run's lease for the attach and the work after * it: a drained run that would start with less has nothing to start for. */ export const DRAIN_LEASE_RESERVE_MS: number = DRAIN.leaseReserveMs; +/** The drain wait where a fallback stands behind the attach (issue 2101): the + * factory's first attach falls to a seeded sandbox, which stands up in about + * two minutes, so the wait is bounded by the fallback's own cost — never the + * deploy's. Passed as `drainBoundMs`; an attach with no fallback (a resumed + * run's re-attach, a mid-run recovery) passes none and keeps the lease's bound. */ +export const DRAIN_FALLBACK_WAIT_MS: number = DRAIN.fallbackWaitMs; + +/** Stamp the drain's share of a failed wait on the error, so the factory's + * fallback can publish the run's `drain_wait` note even though the attach + * never answered a binding to carry it (issue 2101: the incident's run fell + * to the sandbox and `runs friction` counted `drain_wait: 0`). */ +function stampDrainWait(err: E, waitedMs: number): E { + if (err !== null && typeof err === "object") (err as { drainWaitMs?: number }).drainWaitMs = waitedMs; + return err; +} + +/** The drain's share stamped on an error thrown out of the drain wait — the + * typed `ResidentDrainingError`, the attach's own refusal after the drain + * ended, or the wake hand-off's strike; undefined when the attach met no + * drain. */ +export function drainWaitOf(err: unknown): number | undefined { + if (err === null || typeof err !== "object") return undefined; + const stamped = (err as { drainWaitMs?: unknown }).drainWaitMs; + return typeof stamped === "number" && stamped > 0 ? stamped : undefined; +} export function isDrainingRefusal(answer: { status: number; data: Record }): boolean { const d = answer.data.draining; @@ -1127,7 +1152,10 @@ export class ResidentExecutor implements Executor { * bounds the wait's PROBING alone (item 65's ceiling too), never a request: * an operation's wall clock is the command's, and a recovery attach is not * the command. */ - async attach(span?: Span, opts: { signal?: AbortSignal; budgetMs?: number } = {}): Promise { + async attach( + span?: Span, + opts: { signal?: AbortSignal; budgetMs?: number; drainBoundMs?: number } = {}, + ): Promise { const answer = await this.attachOnce(span, this.attachBoundMs("/attach"), opts.signal); if (answer.ok) return answer.binding; if (isDrainingRefusal(answer)) { @@ -1139,6 +1167,7 @@ export class ResidentExecutor implements Executor { origin: "transient-refusal", signal: opts.signal, budgetMs: opts.budgetMs, + drainBoundMs: opts.drainBoundMs, span, }); // The spread keeps a `drainWaitMs` the wake path stamped when its @@ -1153,25 +1182,43 @@ export class ResidentExecutor implements Executor { /** The wait for a drained fleet to reopen (item 69): one re-attach every * `DRAIN_POLL_MS` until the fleet admits the run, under the run's own lease * less what the attach and the work need (`DRAIN_LEASE_RESERVE_MS`), or the - * ceiling with no lease. The run's stop ends it at once (the pause and the - * re-attach both carry its signal). An answer that is no longer the drain is - * judged as `attach` judges a first answer: admitted, the platform's transient - * (handed to the wake wait with what the budget left), or the attach's own - * refusal. The budget ending with the fleet still drained is the typed + * ceiling with no lease — and under `drainBoundMs` where the caller has a + * fallback whose cost bounds the wait (issue 2101: the factory's first + * attach falls to a seeded sandbox in minutes, so it never waits out the + * deploy). The run's stop ends it at once (the pause and the re-attach both + * carry its signal). An answer that is no longer the drain is judged as + * `attach` judges a first answer: admitted, the platform's transient (handed + * to the wake wait with what the budget left), or the attach's own refusal. + * The budget ending with the fleet still drained is the typed * `ResidentDrainingError`, naming how long the run could wait and when the - * drain says it ends. */ + * drain says it ends. Every error thrown out of the wait carries the + * drain's share (`stampDrainWait`), so the fallback still publishes the + * run's `drain_wait` note. The wait is one child span under the attach span + * (`dispatch.workspace.attach.drain-wait`, issue 2101): an 18-minute hold + * with zero events between the attach's start and its end was the incident's + * shape; the span carries the typed error's classification — the refusal's + * words stay on the card note and the attach error (tracing.md item 2 keeps + * remote free text off spans). */ private async awaitDrainEnd( first: { status: number; data: Record }, - opts: { signal?: AbortSignal; budgetMs?: number }, + opts: { + signal?: AbortSignal; + budgetMs?: number; + drainBoundMs?: number; + /** The effective deadline an earlier drain wait opened. A wake hand-off + * carries it back here so no later drain can renew any allowance. */ + drainDeadlineMs?: number; + }, span?: Span, ): Promise<{ binding: ResidentBinding; waitedMs: number }> { const t0 = systemClock(); const left = this.opts.remainingMs?.(); - const budget = Math.min( - DRAIN_WAIT_MAX_MS, - left === undefined ? DRAIN_WAIT_MAX_MS : Math.max(0, left - DRAIN_LEASE_RESERVE_MS), + const deadline = Math.min( + opts.drainDeadlineMs ?? Number.POSITIVE_INFINITY, + t0 + DRAIN_WAIT_MAX_MS, + t0 + (opts.drainBoundMs ?? DRAIN_WAIT_MAX_MS), + t0 + (left === undefined ? DRAIN_WAIT_MAX_MS : Math.max(0, left - DRAIN_LEASE_RESERVE_MS)), ); - const deadline = t0 + budget; let answer = first; // The card's one line while the run waits (issue 2044): user words, the // minutes counted up on every poll, cleared however the wait ends — the @@ -1181,6 +1228,7 @@ export class ResidentExecutor implements Executor { this.opts.onSetupNote?.( `waiting for the deploy to finish · ${Math.max(1, Math.round((now - t0) / MINUTE_MS))} min`, ); + const waitSpan = span?.start("dispatch.workspace.attach.drain-wait"); try { for (;;) { const now = systemClock(); @@ -1188,21 +1236,45 @@ export class ResidentExecutor implements Executor { if (now >= deadline) throw new ResidentDrainingError(this.opts.resource, now - t0, drainingUntil(answer), refusalWords(answer)); await wakePause(Math.min(DRAIN_POLL_MS, deadline - now), opts.signal, "/attach"); - const next = await this.attachOnce(span, this.attachBoundMs("/attach"), opts.signal); - if (next.ok) return { binding: next.binding, waitedMs: systemClock() - t0 }; + const next = await this.attachOnce(waitSpan ?? span, this.attachBoundMs("/attach"), opts.signal); + if (next.ok) { + waitSpan?.end("ok"); + return { binding: next.binding, waitedMs: systemClock() - t0 }; + } answer = next; if (isDrainingRefusal(answer)) continue; if (isTransientRefusal(answer)) { - const woke = await this.awaitWake("/attach", refusalWords(answer), { - origin: "transient-refusal", - signal: opts.signal, - budgetMs: Math.max(0, deadline - systemClock()), - span, - }); - return { binding: woke.binding, waitedMs: systemClock() - t0 }; + // The drain's own share ends here; the wake wait that follows is its + // own span and its own budget's, and a failure inside it still + // carries the drain's share for the fallback's note. + const drainShareMs = systemClock() - t0; + waitSpan?.end("ok"); + try { + const woke = await this.awaitWake("/attach", refusalWords(answer), { + origin: "transient-refusal", + signal: opts.signal, + budgetMs: Math.max(0, deadline - systemClock()), + drainBoundMs: opts.drainBoundMs, + drainDeadlineMs: deadline, + span, + }); + return { binding: woke.binding, waitedMs: systemClock() - t0 }; + } catch (err) { + throw stampDrainWait(err, drainShareMs + (drainWaitOf(err) ?? 0)); + } } throw this.attachRefusal(answer); } + } catch (err) { + // The wake hand-off stamped the drain's own share already; everything + // else — the typed draining error, the attach's refusal after the drain + // ended, the run's stop — spent the whole wait on the drain. + if (drainWaitOf(err) === undefined) stampDrainWait(err, systemClock() - t0); + if (waitSpan !== undefined && !waitSpan.ended) { + waitSpan.fail(err); + waitSpan.end(); + } + throw err; } finally { this.opts.onSetupNote?.(undefined); } @@ -1589,9 +1661,27 @@ export class ResidentExecutor implements Executor { origin: WakeOrigin; signal?: AbortSignal; budgetMs?: number; + /** The drain wait's bound where a fallback stands behind the attach + * (issue 2101): handed to the drain wait a re-attach's `draining` + * answer opens, so the hand-off keeps the fallback's bound too. */ + drainBoundMs?: number; + /** The effective deadline of a drain wait that handed off to this wake. + * A later draining answer keeps this deadline rather than opening a new bound. */ + drainDeadlineMs?: number; span?: Span; }, ): Promise<{ waitedMs: number; binding: ResidentBinding }> { + // The attach's wake wait is one child span under the attach span (issue + // 2101, as the drain wait is): the incident's run spent the wake budget + // here with nothing on the timeline. Other routes' waits stay log-only — + // their parent is the command's span, not the attach's. + const waitSpan = route === "/attach" ? opts.span?.start("dispatch.workspace.attach.wake-wait") : undefined; + const failWaitSpan = (err: unknown): void => { + if (waitSpan !== undefined && !waitSpan.ended) { + waitSpan.fail(err); + waitSpan.end(); + } + }; // The wait's one clock, before the first probe: the loop reads it for its // budget check and hands it to `judge` as `spent()`, so every `waited Ns` — // the strikes', the binding's — is the whole wait, never short by one @@ -1620,12 +1710,20 @@ export class ResidentExecutor implements Executor { opts.span, signal, ); - const first = await probe(opts.signal); - // The run's stop rides into the probe as into every send: a stop during - // one is the stop's own typed error, as the pause throws it — never a - // strike on an "unreachable" view the stop itself produced. (The loop - // makes this check after each of its own probes.) - if (opts.signal?.aborted) throw wakeStopped(route); + let first: ResidentStatusProbe; + try { + first = await probe(opts.signal); + // The run's stop rides into the probe as into every send: a stop during + // one is the stop's own typed error, as the pause throws it — never a + // strike on an "unreachable" view the stop itself produced. (The loop + // makes this check after each of its own probes.) + if (opts.signal?.aborted) throw wakeStopped(route); + } catch (err) { + // The initial probe precedes `waitOnStatus`, but it is still part of the + // visible wake wait and must close the same span on a stop or failure. + failWaitSpan(err); + throw err; + } // Each turn of the wake path's one loop (`waitOnStatus`), the first view // included: judge the view — a definite one is the strike at once, before // any pause — then re-attach when it says the resident serves, else (or @@ -1637,7 +1735,7 @@ export class ResidentExecutor implements Executor { // the exit, so the first re-attach follows the first pause, never a full // /attach into a container still starting. let firstView = true; - return waitOnStatus<{ waitedMs: number; binding: ResidentBinding }>({ + const wait = waitOnStatus<{ waitedMs: number; binding: ResidentBinding }>({ first, since: t0, probe, @@ -1694,7 +1792,16 @@ export class ResidentExecutor implements Executor { // and a binding without the share would publish no `drain_wait` // note — the silence the field exists to end. if (isDrainingRefusal(answer)) { - const reopened = await this.awaitDrainEnd(answer, { signal: opts.signal }, opts.span); + waitSpan?.end("ok"); + const reopened = await this.awaitDrainEnd( + answer, + { + signal: opts.signal, + drainBoundMs: opts.drainBoundMs, + drainDeadlineMs: opts.drainDeadlineMs, + }, + opts.span, + ); return { end: { waitedMs: spent(), binding: { ...reopened.binding, drainWaitMs: reopened.waitedMs } }, }; @@ -1720,6 +1827,15 @@ export class ResidentExecutor implements Executor { throw residentWakeBudgetStrike(route, refusal, spentMs, last, { origin: opts.origin, transient }); }, }); + if (waitSpan === undefined) return wait; + try { + const out = await wait; + waitSpan.end("ok"); + return out; + } catch (err) { + failWaitSpan(err); + throw err; + } } async exec(command: string, opts?: ExecOptions): Promise {