Skip to content

fix(resident): bound drain waits by the fallback deadline - #2113

Merged
justinhelmer merged 1 commit into
mainfrom
plan/fix-issue-2101-defects-1-4f2b60/u1
Sep 21, 2026
Merged

justinhelmer merged 1 commit into
mainfrom
plan/fix-issue-2101-defects-1-4f2b60/u1

Conversation

@coreplane-switchboard

@coreplane-switchboard coreplane-switchboard Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

A drained first attach now falls back to the seeded sandbox on one three-minute deadline, even across wake/drain hand-offs. Drain and wake waits remain visible through spans and a drain_wait run finding without new card or health vocabulary.

Why: Issue #2101 exposed an 18-minute drain hold before the same run used a two-minute seed. This keeps record 0059's drain semantics while bounding the first resident attach by its cheaper fallback.

Where to look

  1. Fallback-bound first attach The only attach with a seeded fallback supplies the fallback-cost bound; resumed and recovery attaches keep lease bounds.
  2. Single effective drain deadline The first drain computes one effective deadline from fallback, lease, and global ceilings instead of renewing a duration. ⚠ A renewed deadline can hold new runs through an entire deploy.
  3. Deadline across wake hand-offs A transient hand-off carries the same deadline forward and accumulates each drain segment for friction reporting. ⚠ Losing the deadline or a segment defeats the fallback bound or undercounts delay.
  4. Initial-probe span finalization A stop during the first status probe now takes the wake span's typed failure path before propagating.
  5. Drain friction publication The dispatcher publishes drain_wait from either a successful resident binding or a cold-fallback selection.
  6. Hand-off deadline regression The replay spends most of the allowance before a wake hand-off and proves the second drain gets only the remainder.
  7. Resident drain contract Item 69 binds the absolute deadline, typed spans, fallback timing, and friction finding to executable proofs.

Feedback wanted: Please judge the absolute-deadline propagation across repeated drain/wake cycles and the span finalization for every initial-probe exit.

Risk: 444 changed lines across one resident attach path. Splitting visibility from the bound was rejected because friction derives from the same timing data; rollback is the single commit.

Verified: 145 resident tests, scoped TypeScript, Prettier, hygiene, spec integrity, and spec coverage passed. CI and live deploy timing remain human-gated.

Decisions (2)
  • Carry an absolute deadline. Passing a fresh duration was rejected because each wake/drain cycle could renew it. One effective deadline also preserves the stricter lease or global ceiling selected by the first drain.
  • Keep span failures typed. Remote refusal text was rejected on spans because tracing's privacy contract forbids it. The wait spans carry typed classifications; user-facing words remain in the existing card line and attach error.
Validation (7 criteria)
Criterion Proof
Repeated drain/wake hand-offs cannot renew the fallback allowance src/execution/resident.test.ts — “one fallback deadline survives drain/wake hand-offs…” passed in npx vitest run src/execution/resident.test.ts (145/145).
Stopping during the initial status probe closes the wake span src/execution/resident.test.ts — “a stop during the wake wait's initial status probe closes the wake span as failed” passed.
Touched TypeScript remains type-safe NODE_OPTIONS=--max-old-space-size=6144 npx tsc --noEmit -p tsconfig.json passed.
Changed files retain repository formatting npx prettier --check on every origin/main...HEAD changed file passed.
Specs and public-tree constraints remain valid npm run specs:check and npm run hygiene:check passed.
Every changed source path has spec coverage and tests were not removed npm run specs:coverage -- --changed origin/main...HEAD --test-guard --require passed.
PR title satisfies the changelog gate npm run check:pr-title -- "fix(resident): bound drain waits by the fallback deadline" passed.
For agents

Review fixes F1 and F2 are folded into the original coherent commit. Branch was rebased onto origin/main at db6003c and force-pushed at 1ec9053. The two new regressions first failed on the reviewed head: deadline renewal timed out and the wake span had no end record. No generated files changed; no new plane health word or card string was introduced.
Requested by @justinhelmer in slack:C0BRRHKFLCB

🤖 Generated with Claude Code

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested: The fallback deadline resets across wake/drain hand-offs, and one stop path leaves the new wake span open.

Warning

Changes requested · head 4c420e7 · 2 findings: 1 major, 1 minor

Severity Finding Where
major F1 Preserve one fallback deadline across wake/drain hand-offs src/execution/resident.ts:1771
minor F2 Close the wake span when the initial probe is stopped src/execution/resident.ts:1701
Full review

F1: awaitWake starts a fresh awaitDrainEnd with the full drainBoundMs. If the first drain wait spends 2½ minutes, hands off through a transient refusal, and the wake re-attach encounters the drain again, the second drain wait receives another three minutes. Repeated deploy resets can keep renewing the bound toward the lease limit. Carry one absolute fallback deadline—or only the remaining allowance—across every drain/wake hand-off.

F2: When the run stops during the initial /status probe, probeStatus returns an unreachable result and the abort check throws before reaching the span-closing try/catch. The dispatch.workspace.attach.wake-wait span therefore never ends. Wrap the initial probe and abort check in the same failure/finalization path, with a regression test for stopping during that probe.

@coreplane-switchboard
coreplane-switchboard Bot force-pushed the plan/fix-issue-2101-defects-1-4f2b60/u1 branch from 4c420e7 to 1ec9053 Compare September 21, 2026 03:52
@coreplane-switchboard coreplane-switchboard Bot changed the title fix(resident): a drain refusal falls to the seeded sandbox in minutes fix(resident): bound drain waits by the fallback deadline Sep 21, 2026

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: Both prior findings are resolved and no new issues remain in the full change.

Note

Approved · head 1ec9053 · no findings

Full review

Both prior findings are resolved, with no new issues found.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

A run refused by the fleet drain waited at its attach for the whole
deploy (issue #2101 defect 1: 18 minutes, DRAIN.waitMaxMs or the lease
less the reserve), then did the job in a seeded sandbox that stands up
in about two. The factory's first attach now passes drainBoundMs =
DRAIN.fallbackWaitMs (3 min), so the wait is bounded by the fallback's
own cost, never the deploy's; a resumed run's re-attach keeps the
lease's bound (its worktree lives on the resident or nowhere).

The wait is also visible (defect 3): the drain wait and the attach's
wake wait are each one child span under the attach span
(dispatch.workspace.attach.drain-wait / .wake-wait, display names in
ATTACH_WAIT_NAMES), failing with the typed classification; the typed
draining error carries the drain's share (drainWaitOf), the fallback
selection carries it as ExecutorSelection.drainWaitMs, and the
dispatcher publishes the drain_wait run note from either source, so
runs friction counts the wait the incident counted as zero.

Spec: resident-repos.md item 69 (prose + a new proof row), execution.md
item 26.

Co-Authored-By: coreplane-switchboard[bot] <318072483+coreplane-switchboard[bot]@users.noreply.github.com>
@justinhelmer
justinhelmer force-pushed the plan/fix-issue-2101-defects-1-4f2b60/u1 branch from 1ec9053 to 2e1891f Compare September 21, 2026 15:28
@justinhelmer
justinhelmer merged commit afd2b51 into main Sep 21, 2026
29 checks passed
@justinhelmer
justinhelmer deleted the plan/fix-issue-2101-defects-1-4f2b60/u1 branch September 21, 2026 15:36

@coreplane-switchboard coreplane-switchboard Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: The rebase preserves the reviewed source patch and correctly combines both sides of the spec conflicts.

Note

Approved · head 2e1891f · no findings

Full review

The rebase preserves the reviewed implementation and both sides of the spec updates, with no new issues.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: coreplane-switchboard[bot] reviewed this PR and posted an LGTM verdict (see its review). This repository opted in through its REVIEW_BOT_LOGIN and REVIEW_BOT_ID variables.

@justinhelmer

Copy link
Copy Markdown
Contributor

Live receipt 2026-09-21 (1.260.2 deploy, the first deploy with #2113 in the bot): review child 7296aa2f-c3a9-4ad4-8be4-f3f73cd66265 (PR #2168's review, thread slack:C0BRRHKFLCB:1790016553.187339) was admitted at 21:02:54Z while the resident drain held for the swap. Its run log carries the wait as its own span — dispatch.workspace.attach.drain-wait seq 27→28, started 21:02:56.077Z, durationMs: 128350 (2 min 8 s), status ok — inside the 3-minute bound; the swap landed (resident on 95c0153 at 21:03:01Z), the drain lifted, and the attach continued on the resident (unmount-restoresmirror-restore-extractcheckout-restore-extractgit-setup, backend resident) instead of falling back. The card said ◓ review · 31s — waiting for the deploy to finish · 1 min for the wait (tracker item 2: the drain-wait span and the card line, both present). Receipt runnable when: worker:bot ≥ 95c0153 and event: a run admitted while the drain holds — met.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant