Skip to content

fix(server): a cancelled queued Claude continuation no longer drops later wakes - #30114

Open
juliusmarminge wants to merge 5 commits into
mainfrom
fix/issue-30052
Open

juliusmarminge wants to merge 5 commits into
mainfrom
fix/issue-30052

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Problem

When a Claude background task wakes the agent while another run is active, T3 queues a continuation run for that wake. If that queued run is cancelled before it starts (the queue's cancel, t3_queue with action: "cancel", or Stop cancelling queued wakes), the Claude adapter's requestedContinuations latch for that native thread is never released. Every later wake on that thread is then dropped: Claude keeps working, but no T3 run owns the turn, its messages never reach the thread, and T3 MCP tools in that turn fail with parent_not_active. It lasts until the provider query is reopened or the runtime restarts.

Reproduced on main (cefb544) with an integration test that drives the real Claude adapter, orchestrator, effect worker and ProviderContinuationService with replayed SDK frames:

  1. A user turn starts a background task and ends.
  2. The task finishes and Claude wakes, so a continuation run is queued.
  3. queued-run.cancel cancels the queued run.
  4. A second user turn starts another background task.
  5. That task's wake offers no continuation.

Change

  • ProviderContinuationRequest gets an optional onRunEndedWithoutStarting callback. When a request supplies it, ProviderContinuationService watches the run its dispatch created (stored events from the dispatch's own sequence) and calls the callback if that run reaches a terminal status with startedAt === null. Only Claude supplies it, so ACP/Grok, Muse, Pi and OpenCode are unchanged.
  • The Claude latch becomes a per-offer token, so a stale request cannot release a newer wake's latch. The Claude offer also supplies clearIfCurrent (archived thread, disabled instance) with the same retire behaviour.
  • Retiring a cancelled wake drops exactly the frames and reports that wake's continuation would have shown, then releases the latch. If the wake turn is still running, the rest of its root frames are dropped until its root result. A later wake already buffered behind it gets its own continuation. Subagent task_started frames are kept only for subagents that did not finish inside the dropped frames, and are removed with their child frames if a dropped frame settles them later. Empty lifecycle results (num_turns: 0) do not count as the wake's end.
  • If the cancelled wake never opens, the next notification that wakes Claude releases the latch, so the following wake is not lost.

Scope and approval

Fixes #30052

Verification

Focused tests (server-only change, no render path):

Commit ClaudeAutomaticDelivery.integration.test.ts
d556521 (red test only, on main cefb544) 1 failed, 4 passed: the wake after a cancelled queued continuation must offer a new continuation: expected [ { …(5) } ] to have a length of 2 but got 1
953644f (head) all passed
  • New tests in Adapters/ClaudeAdapterV2.test.ts ("background wake turns"):

    • a stale request leaves a newer latch alone;
    • a cancelled wake's release that arrives after the next wake opened still offers that wake;
    • a wake cancelled before its first root frame is not offered twice;
    • a cancelled wake that never opens does not swallow the next wake;
    • a subagent that finished inside a cancelled wake is not replayed as running, whether it ended by notification, by tool result, or by a tool result that arrived after the cancel;
    • an empty lifecycle result before the real turn does not make the cancelled wake get a second continuation;
    • a subagent's notification behind a cancelled wake gets its own continuation, before or after the cancelled wake's result.

    Each fails on the commit before it and passes at head.

  • The integration test also asserts that the next continuation carries only its own wake's output, and that two wakes yield exactly two continuation offers.

  • ClaudeAdapterV2.test.ts, ClaudeAutomaticDelivery.integration.test.ts and ProviderContinuationService.test.ts all pass (322 tests). The Pi, Muse, ACP and OpenCode adapter suites pass unchanged.

  • vp run check:changed: format, lint, knip, typecheck (26 packages) and 4 test files pass.

Not checked: a live Claude CLI session. The verification VM has no Claude account, so every Claude ordering here is replayed from synthetic SDK frames.

Model and harness: Claude Opus 5.5 (claude-opus-5-5) in Claude Code via T3 Code, with GPT-6.1-Sol in Codex reviewing.

🤖 Generated with Claude Code

juliusmarminge and others added 4 commits November 3, 2026 09:53
…drops later wakes

A wake offered while another run is active becomes a queued continuation
run. Cancelling it with queued-run.cancel never reaches the Claude adapter,
so its requestedContinuations latch stays set and the next wake on that
native thread offers no continuation. These tests reproduce #30052.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ater wakes

The Claude adapter latches each native thread after offering a wake
continuation and released it only when that continuation attached. A
queued continuation that ended without starting (cancelled, stopped,
settled) never attached, so every later wake on the thread was dropped.

ProviderContinuationService now follows the run it dispatched and calls
the request's clearIfCurrent when that run ends without starting. The
Claude offer supplies a per-offer token so a stale request cannot release
a newer wake's latch, and it defers the release to the cancelled wake
turn's end so that turn's remaining frames cannot offer it again.

Fixes #30052

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releasing the wake latch judged the wake from whatever was buffered, so a
late release could swallow the next wake, an early one re-offered the
cancelled wake, and the cancelled wake's frames replayed into the next
continuation. The latch now records where its wake ends; release drops
that wake's frames and reports and re-offers a later wake, or drops the
wake's root frames until its result. The continuation service calls a new
opt-in onRunEndedWithoutStarting callback, so only Claude changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bagents

Retiring a cancelled wake kept every subagent start from the discarded
frames, so the next continuation replayed a subagent that had already
finished as running forever. Retirement now keeps only starts with no
terminal notification or settling root tool result in those frames.

A wake cancelled before Claude opened it held its latch until the next
root result, which dropped a later task's wake entirely. While the
retired wake is unopened, a new task notification now releases the
latch so that wake gets its own continuation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Nov 3, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 953644f

Macroscope's review found this PR approvable — This restores later Claude background wakes after a queued continuation is cancelled, with per-offer guards preventing stale cleanup from affecting newer wakes. The unresolved Medium subagent-state finding remains subject to the separate blocking gate.

Notes:

  • This verdict was updated automatically after the outstanding correctness findings were resolved. Macroscope did not re-review the code.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB +16 B (+0.3%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB +16 B (+1.3%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.6 KiB +41 B (+0.2%) 24.4 KiB ✅
Codex Live turn messages 1 2 +1 (+100.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: cefb544 · PR result: 953644f · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The change adds callbacks for continuation runs that end without starting. Claude uses these callbacks to retire wake latches, filter buffered frames, and offer eligible later wakes. Tests cover cancellation, wake ordering, subagent completion, and later continuations.

Changes: Claude wake continuation lifecycle

Layer / File(s) Summary
Report continuations that never start
packages/provider-core/src/server/ProviderContinuationRequests.ts, apps/server/src/orchestration-v2/ProviderContinuationService.ts, apps/server/src/orchestration-v2/testkit/ProviderReplayHarness.ts
Continuation requests can provide an optional callback for runs that end without starting. The service watches dispatched runs for terminal events without a start and invokes the callback.
Retire and reoffer Claude wake continuations
apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
Claude tracks tokenized per-thread wake latches, retires latches when requests clear or end without starting, filters retired wake frames, and offers eligible later continuations. Shared predicates handle subagent settlement and empty task-notification results.
Verify cancellation and later wake handling
apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts, apps/server/src/orchestration-v2/ClaudeAutomaticDelivery.integration.test.ts
Tests cover wake ordering, cancellation, completed subagents, empty lifecycle results, and a later wake receiving a new continuation with only its own output.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant ClaudeAdapterV2
  participant ProviderContinuationService
  participant Orchestrator
  ClaudeAdapterV2->>ProviderContinuationService: offer continuation with retirement callback
  ProviderContinuationService->>Orchestrator: dispatch continuation
  Orchestrator-->>ProviderContinuationService: terminal run event without start
  ProviderContinuationService->>ClaudeAdapterV2: invoke retirement callback
  ClaudeAdapterV2->>ClaudeAdapterV2: retire latch and process eligible later wake frames
Loading

Possibly related issues

Suggested reviewers: maria-rcks, t3dotgg

Merge Risk: Minimal · up to 95364

No concrete merge-blocking regression was established. The change can merge after normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title is concise, uses the repository’s conventional format, and clearly identifies the main fix: cancelled queued Claude continuations no longer drop later wakes.
Description check Passed The description includes Problem, Change, Scope and approval, and Verification sections. It explains the failure, implementation, linked issue, focused test results, broader checks, and the limitation…
Linked Issues check Passed Directly linked issue [#30052] requires that a Claude continuation run which ends without starting releases the wake latch for its native thread. Later wakes must then open a run and deliver output.

…
Out of Scope Changes check Passed Changes map to the wake-latch fix for [#30052]:
- packages/provider-core/src/server/ProviderContinuationRequests.ts: adds the optional onRunEndedWithoutStarting callback. This is the request-level…
Full details: Description check

Explanation

The description includes Problem, Change, Scope and approval, and Verification sections. It explains the failure, implementation, linked issue, focused test results, broader checks, and the limitation that live Claude CLI testing was not performed.

Full details: Linked Issues check

Explanation

Directly linked issue [#30052] requires that a Claude continuation run which ends without starting releases the wake latch for its native thread. Later wakes must then open a run and deliver output.

Evidence at head 953644f:

  • ProviderContinuationService.ts adds releaseIfNeverStarted. After threads.dispatch, it finds the run.created event for the dispatched messageId. It watches run.updated events after the dispatch sequence. When that run reaches a terminal status with startedAt === null, it calls request.onRunEndedWithoutStarting. Watcher failures are logged.
  • ClaudeAdapterV2.ts stores the wake latch as a per-offer token. Its retire callback is passed as both clearIfCurrent and onRunEndedWithoutStarting. retireOfferedWake checks the token before it changes state, so a stale request cannot release a newer latch.
  • The retire path drops the cancelled wake's frames and releases the latch. It then offers the next buffered wake. If the wake never opens, the next wake notification releases the latch.
  • ClaudeAutomaticDelivery.integration.test.ts reproduces the issue's cancel sequence. It asserts that a second wake yields a second continuation offer and that the continuation carries only its own output. The PR reports this test fails on the red commit and passes at head.
  • ClaudeAdapterV2.test.ts adds cases for stale requests, cancelled wakes, unopened wakes, and settled subagents.

Residual uncertainty: [#30052] also names promote_to_steer as covered by the same path. The callback is generic. It fires for any dispatched run that ends unstarted. I did not verify that promote_to_steer ends the queued run with a terminal status and startedAt === null, and no test covers that path. The cancel path, which the issue reports as observed, is covered.

Full details: Out of Scope Changes check

Explanation

Changes map to the wake-latch fix for [#30052]:

  • packages/provider-core/src/server/ProviderContinuationRequests.ts: adds the optional onRunEndedWithoutStarting callback. This is the request-level hook the fix needs.
  • ProviderContinuationService.ts: adds the run watcher that calls the callback. The dispatch .pipe(Effect.tap(...)) wrapper is required for the watcher. Its formatting change is a side effect of that wrapper.
  • ClaudeAdapterV2.ts: changes the latch to a per-offer token map. It adds frame retirement, subagent settlement helpers, and the unopened-wake release. The subagent settlement helper is used by the retire path. The summary states this rule is shared with wake-frame processing. The diff output was truncated at 16000 bytes, so I could not confirm that no non-wake path uses the changed settlement rule. The PR states the rule applies to wake frames.
  • testkit/ProviderReplayHarness.ts: exposes streamStoredEventsFrom to the mocked thread service. This supports the new integration test.
  • Test files: the new unit and integration tests cover the cases described in the issue and PR.

No unrelated production behavior was demonstrated. ACP, Muse, Pi, and OpenCode do not supply the new callback, so their behavior is unchanged by the callback path.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 959da91 and 953644f.

Files selected for processing: 2
  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

…re-offers itself

A cancelled wake's continuation could still go wrong in three ways. A root
tool result dropped while the wake was retired left the subagent it settled
kept as running. An empty lifecycle result before the wake's real turn was
taken as the wake's end, so the real turn got a second continuation. And a
subagent notification kept behind the released wake never got its own offer.

The retired branch now drops the settled subagent's start, child frames and
report; empty task-notification results neither end the latch nor release a
retired one; and both release paths offer a buffered subagent notification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Member Author

Thanks for the handoff and the real-Claude reproduction on cefb544.

The two patches differ only in what happens to the cancelled wake's output:

  • dd0da1d releases the latch and keeps the buffer. That is the shape this branch had first (b81d635). Two internal reviews found problems with it in replayed SDK-frame tests:
    1. If wake B's continuation starts while B is still streaming, the drain replays A's buffered frames, including A's result. The continuation settles on that stale result, and B's real result then opens an extra continuation. Two wakes gave three continuation requests.
    2. Calling clearIfCurrent for every adapter makes ACP/Grok re-offer the cancelled wake on its next chunk, because its flag clears immediately. On Pi, clearIfCurrent is stopPendingWake, which terminates the connection.
  • This PR (953644f) discards the cancelled wake's buffered output. Only Claude supplies the new onRunEndedWithoutStarting hook, so other adapters are unchanged. The cost is size, about 1,000 lines with tests. It also hides a native turn Claude already ran, including its tool calls and edits.

Whether a cancelled wake's output should be shown with the next wake or dropped is still open. This PR has only been verified with replayed frames; no live Claude session was available here.

@juliusmarminge juliusmarminge added bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed labels Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Cancelling a queued Claude continuation run leaves requestedContinuations set, so every later wake on that thread is dropped

1 participant