From 595bc698601e793313a7bc0064557083d3b3c145 Mon Sep 17 00:00:00 2001 From: Christian Byrne Date: Sat, 31 Oct 2026 17:08:15 +0000 Subject: [PATCH 1/4] fix(agent): restore pre-reload thinking when the turn finishes (FRONT-3659) (#22244) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reload while the agent is still working and the rounds it had already narrated do not come back when the turn finishes. Only what streamed after the reload shows, and the earlier reasoning reappears once you leave the chat and open it again. Fixes [FRONT-3659](https://linear.app/comfyorg/issue/FRONT-3659). The reload restores the running turn from the transcript, and a running row carries no reasoning: the backend writes `content.thinking` when the turn finishes (Comfy-Org/cloud#14903, consumed here by #21600). So the panel follows the rest of the turn over the socket and settles on its terminal frame with nothing fetched — the one completion path that never reads the stored row back. The reconnect recovery *does* read it, through `settleTurn`, which is why a dropped socket keeps the whole trace and a reload does not. `agentConversationStore` already knows which turns came from a snapshot (`ActiveTurnSlot.origin`), so it announces one once nothing is still publishing the row, and `useAgentSession` makes the same read the recovery makes, once. - **After the settle, not instead of it.** The turn is already finished, so holding it open for a round trip would delay the composer's handback and strand it outright if the read failed. Reconciling afterwards means a failed read leaves the turn exactly as it is today, rather than locking the composer. - **The announcement waits for the transport to stop publishing.** A settled turn can still hold a tool-call part pending canvas catch-up (PM-1575), and such a turn keeps its transport reachable in `settledActiveTransports`. That transport owns its own copy of the row and republishes it through `replaceActive` on the next `notifyCanvasCaughtUp` — which would discard everything the reconcile folded in. So the registration carries the transport and the announcement is deferred until nothing is held: the reconcile only ever runs when no transport can still publish that row. Every transport emit routes through `publishMessage`, which covers the catch-up settle and the transport's own stale fallback alike, and the announcement is one-shot. - **The stored row decides the finished turn's thinking.** After the settled read the panel renders the same thinking that reopening the chat renders; live thinking is kept only when the stored row holds no thinking at all. An earlier head of this PR merged live and stored rounds on the client, and review rejected that: the same finished turn showed different reasoning depending on whether you watched it settle. Only two things carry over from the live copy, because only the client has them: `canvasConfirmed`, and the tool offset that places post-reload tab links. Any reasoning the backend cap drops is a backend contract problem, tracked in https://linear.app/comfyorg/issue/BE-24672/agent-stored-thinking-is-cut-off-with-no-signal-past-the-per-round-and. - **The read repeats while the row still reads as streaming.** The terminal write can trail the socket frame, so the settled read follows `TURN_RECOVERY_DELAYS_MS` until the row is terminal or the deadline passes. A failed read is reported once and not retried, and every exit that does not apply a result releases the registration and its transport. - **A hydrate the read deferred is still honored.** Leaving for a new chat and reopening the conversation while the settled read is in flight clears the snapshot registration, so the read's own result is discarded when it lands. The hydrate it deferred is then the only thing that can settle the row hydration restored, so the read tears down through `finishRecovery`, which reruns only while the turn is live -- a no-op on the ordinary settled path, since a settled turn is not live. - **`settleTurn` is unchanged.** The recovery read keeps its plain stored-row handling, so this PR only touches the socket-settled path that had no read at all. - **The settled publisher is released at the announcement.** Its stale fallback can clear the held part without removing it from `settledActiveTransports`, and a later ask retirement would then republish the transport's own copy of the row over the reconciled one. - **`reconcileSettledTurn` writes a fresh object**, because `replaceActive`'s emit is what every other write to a settled row relies on. - **Identity.** A snapshot keys a turn by **row id** (`ActiveTurnSlot.turnId`) while its panel message is keyed by **turn id**, so the registration holds both and the store unit tests use different values for each — the browser fixture deliberately makes them equal, which would hide a wrong lookup. - **The pre-reload rounds still do not show *while* the turn is running.** That would need the backend to replay or store in-flight thinking, which is a separate and larger change. The reload's recovery poll *can* win the race and settle the turn through the reading path, which would hide the defect and make the spec flaky-green on the parent. `AgentTurnLockHarness.waitForRecoveryExhausted()` therefore waits out the poll schedule — one hydration read plus one per `TURN_RECOVERY_DELAYS_MS` entry, counted through the fixture's transcript-read counter — so the socket is deterministically what settles the turn. The schedule lives in its own leaf module (`services/agent/turnRecoverySchedule.ts`) so the fixture derives that count from the real value instead of hard-coding it: if the schedule grew, a hard-coded wait would be too short and the spec would keep passing for the wrong reason. **Black-box proof, red on the parent and green here.** New spec `browser_tests/tests/agent/agentThinkingAcrossMidTurnReload.spec.ts` narrates a round, reloads mid-turn, narrates another round, finishes the turn over the socket, and asserts the whole trace. On the parent (`d85f282401`, product change reverted, test files kept) the trace holds only the post-reload round — the earlier reasoning *and* the tool call are both gone: ``` Error: expect(locator).toHaveText(expected) failed Locator: locator('#agent-panel-root').getByRole('listitem') - Expected - 2 + Received + 0 Array [ - "Wiring the audio output node.", - /Add node/, "Back online, so I can finish the wiring.", ] ``` | Check | Result | | --- | --- | | New spec on the parent with the product change reverted | **Failed** as quoted above | | New spec on an earlier head of this branch | Passed, and 3/3 with `--repeat-each=3` | | Focused conversation-store and session unit suites | Green, 1 expected fail | | Broader Agent unit sweep | Green apart from one recovery-timer assertion that fails only under full parallel load and passes alone, identically on the parent | | `pnpm typecheck`, `pnpm typecheck:browser`, `oxfmt`, `oxlint --type-aware`, `depcruise`, `knip`, `fallow:audit`, pre-commit and pre-push hooks | Clean | Session coverage adds the leave-and-reopen interleaving: the settled read is held open across a New Chat and a reopen with a still-streaming snapshot, and the reopened row must still settle once the read lands. It is red on the previous teardown with the row stranded as streaming. Store coverage exercises announce-and-fold, one-shot reconciliation, stored thinking winning over a live round that is longer, a suffix, missing from the row, a repeat of an earlier round or keyed differently, live thinking kept when the row holds none, confirmed canvas outcomes, canvas-catch-up deferral, teardown ordering, a late ask retirement after a stale held tool call, and a failed post-settlement history read without reclaiming the composer. Not verified: no live in-browser turn against a real backend, and no Windows run. The agent API is mocked by the browser fixture throughout. The browser spec is proven on this head by the cloud E2E job, not by a local run. One pre-existing unrelated failure: whole-repo `pnpm oxlint` (`--type-aware`) crashes in `tsgolint` on this machine, identically on a pristine parent checkout. The staged-file `oxlint --type-aware` that the pre-commit hook runs passes. - Earlier thinking still does not show while the turn is running: https://linear.app/comfyorg/issue/BE-24497 - Reasoning past the stored caps is dropped with no signal: https://linear.app/comfyorg/issue/BE-24672/agent-stored-thinking-is-cut-off-with-no-signal-past-the-per-round-and - Backports: the 1.58 and 1.59 targets are labelled. 1.58 will probably need a manual port because main's #18949 recovery rework is not on that line. --------- Co-authored-by: GitHub Action (cherry picked from commit 9a1f6dc2bc5fda194f9d78bf07124e70fd5ca68e) --- .../fixtures/agentTurnLockFixture.ts | 32 +- .../fixtures/components/AgentPanel.ts | 9 +- .../agentThinkingAcrossMidTurnReload.spec.ts | 61 +++ .../composables/agent/useAgentSession.test.ts | 191 ++++++++- .../composables/agent/useAgentSession.ts | 83 +++- .../services/agent/turnRecoverySchedule.ts | 1 + .../agent/agentConversationStore.test.ts | 372 ++++++++++++++++++ .../stores/agent/agentConversationStore.ts | 171 +++++++- 8 files changed, 900 insertions(+), 20 deletions(-) create mode 100644 browser_tests/tests/agent/agentThinkingAcrossMidTurnReload.spec.ts create mode 100644 src/workbench/extensions/agent/services/agent/turnRecoverySchedule.ts diff --git a/browser_tests/fixtures/agentTurnLockFixture.ts b/browser_tests/fixtures/agentTurnLockFixture.ts index 982b32ad4b5..ca8a09f61e6 100644 --- a/browser_tests/fixtures/agentTurnLockFixture.ts +++ b/browser_tests/fixtures/agentTurnLockFixture.ts @@ -17,6 +17,7 @@ import { escapeRegExp } from 'es-toolkit' import { z } from 'zod' import enMessages from '@/locales/en/main.json' with { type: 'json' } +import { TURN_RECOVERY_DELAYS_MS } from '@/workbench/extensions/agent/services/agent/turnRecoverySchedule' import type { AgentWsEvent, zTurnInProgressError @@ -51,7 +52,8 @@ const TURN_IN_PROGRESS: TurnInProgressBody = { turn_id: TURN_ID } -const TURN_THINKING_TEXT = 'Wiring the audio output node.' +export const TURN_THINKING_TEXT = 'Wiring the audio output node.' +export const LATE_THINKING_TEXT = 'Back online, so I can finish the wiring.' export const POST_RECONNECT_TEXT = 'Reconnected, and the graph is ready.' export const PERSISTED_BEFORE_TOOL_TEXT = 'Prepared the graph. ' export const PERSISTED_AFTER_TOOL_TEXT = 'Audio output connected.' @@ -73,6 +75,11 @@ const TURN_TOOL_EVENT: AgentWsEvent = { } } +export const LATE_THINKING_EVENT: AgentWsEvent = { + type: 'agent_thinking', + data: { delta: LATE_THINKING_TEXT, message_id: TURN_ID, thread_id: THREAD_ID } +} + export const POST_RECONNECT_EVENT: AgentWsEvent = { type: 'agent_message_delta', data: { @@ -178,6 +185,7 @@ class TurnLockServer { private foreignPrompt: string | undefined private rejected = 0 private posts = 0 + private transcriptReads = 0 private readonly answered: string[][] = [] private pendingAsk: AgentPendingAsk | undefined private heldTranscript: @@ -203,6 +211,14 @@ class TurnLockServer { return this.posts } + get transcriptReadCount(): number { + return this.transcriptReads + } + + countTranscriptRead(): void { + this.transcriptReads++ + } + countPost(): void { this.posts++ } @@ -324,6 +340,10 @@ class TurnLockServer { tool_name: 'add_node', status: 'success' } + ], + thinking: [ + { text: TURN_THINKING_TEXT, tool_call_id: 'call-add-node' }, + { text: LATE_THINKING_TEXT } ] } }, @@ -438,6 +458,7 @@ async function routeTurnLock( const path = new URL(route.request().url()).pathname.split('/') const threadId = decodeURIComponent(path.at(-2) ?? '') await server.waitForTranscriptRelease(threadId) + server.countTranscriptRead() return route.fulfill(jsonRoute(server.transcript(threadId))) } server.countPost() @@ -512,9 +533,9 @@ export class AgentTurnLockHarness { public readonly liveProgressRow: Locator public readonly userBubbles: Locator public readonly turnInProgressNotice: Locator + public readonly agentPanel: AgentPanel private readonly entryButton: Locator private readonly dock: Locator - private readonly agentPanel: AgentPanel constructor( private readonly page: Page, @@ -580,6 +601,13 @@ export class AgentTurnLockHarness { return this.server.postAttempts } + async waitForRecoveryExhausted(): Promise { + const hydrationRead = 1 + await expect + .poll(() => this.server.transcriptReadCount, { timeout: 90_000 }) + .toBeGreaterThanOrEqual(TURN_RECOVERY_DELAYS_MS.length + hydrationRead) + } + /** Opens the panel on a blank workflow and points the composer at that tab. */ async openOnBlankWorkflow(): Promise { await expect( diff --git a/browser_tests/fixtures/components/AgentPanel.ts b/browser_tests/fixtures/components/AgentPanel.ts index 1f1348cdbb6..c3d04d11560 100644 --- a/browser_tests/fixtures/components/AgentPanel.ts +++ b/browser_tests/fixtures/components/AgentPanel.ts @@ -124,7 +124,14 @@ export class AgentPanel { name: 'Workflow JSON' }) this.workSummary = this.root.getByRole('button', { - name: new RegExp(`^${escapeRegExp(enMessages.agent.worked)}`) + name: new RegExp( + `^(${[ + enMessages.agent.worked, + enMessages.agent.workedNotConfirmedOnCanvas + ] + .map(escapeRegExp) + .join('|')})` + ) }) } diff --git a/browser_tests/tests/agent/agentThinkingAcrossMidTurnReload.spec.ts b/browser_tests/tests/agent/agentThinkingAcrossMidTurnReload.spec.ts new file mode 100644 index 00000000000..feda5605858 --- /dev/null +++ b/browser_tests/tests/agent/agentThinkingAcrossMidTurnReload.spec.ts @@ -0,0 +1,61 @@ +import { expect } from '@playwright/test' + +import { + LATE_THINKING_EVENT, + LATE_THINKING_TEXT, + PERSISTED_AFTER_TOOL_TEXT, + TURN_THINKING_TEXT, + agentTurnLockTest as test +} from '@e2e/fixtures/agentTurnLockFixture' + +test.describe.configure({ timeout: 180_000 }) +test.use({ connectWebSocketToServer: false }) + +test( + 'restores the thinking from before a mid-turn reload when the turn finishes', + { + tag: ['@cloud', '@ui'], + annotation: { + type: 'source', + description: 'https://linear.app/comfyorg/issue/FRONT-3659' + } + }, + async ({ turnLock, getAgentSocket }) => { + const prompt = 'add an audio output node' + + await test.step('narrate a round, then reload mid-turn', async () => { + await turnLock.openOnBlankWorkflow() + await turnLock.startTurn(prompt) + + await turnLock.agentPanel.reload() + await expect(turnLock.userBubbles).toHaveText([prompt]) + await expect(turnLock.stopButton).toBeVisible() + }) + + const socket = + await test.step('narrate another round after the reload', async () => { + await turnLock.waitForRecoveryExhausted() + + const live = await getAgentSocket() + turnLock.push(live, LATE_THINKING_EVENT) + await expect(turnLock.panel.getByText(LATE_THINKING_TEXT)).toBeVisible() + return live + }) + + await test.step('finishing the turn shows both rounds beside the tool call', async () => { + turnLock.finishTurn(socket) + + await expect(turnLock.sendButton).toBeVisible({ timeout: 30_000 }) + + await turnLock.agentPanel.openWorkSummary() + await expect(turnLock.agentPanel.activityRows).toHaveText([ + TURN_THINKING_TEXT, + /Add node/, + LATE_THINKING_TEXT + ]) + await expect( + turnLock.panel.getByText(PERSISTED_AFTER_TOOL_TEXT) + ).toBeVisible() + }) + } +) diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts index 7434bf0237b..dd86d2255ea 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts @@ -33,7 +33,8 @@ import type { import { zAgentAdmissionError, zAgentTurnAccepted, - zAgentWsEvent + zAgentWsEvent, + toTurnId } from '../../schemas/agentApiSchema' import { AgentApiError, @@ -819,6 +820,194 @@ describe('useAgentSession (v1 composition root)', () => { expect(conversation.activeTurnId).toBe('msg-1') }) + it('keeps socket parts and releases the composer when a settled snapshot read fails', async () => { + const readFailure = new AgentApiError('history unavailable', 503, undefined) + let rejectSettledRead: ((error: AgentApiError) => void) | undefined + const getMessages = vi + .fn() + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' + } + ]) + .mockImplementationOnce(hangingGetMessages) + .mockImplementationOnce( + () => + new Promise((_resolve, reject) => { + rejectSettledRead = reject + }) + ) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + + expect(session.isStreaming.value).toBe(false) + expect(session.entries.value.at(-1)).toMatchObject({ + role: 'assistant', + parts: [ + { + type: 'thinking', + text: 'live after reload', + state: 'done' + } + ] + }) + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(3)) + assert.exists(rejectSettledRead) + rejectSettledRead(readFailure) + + await vi.waitFor(() => + expect(reportError).toHaveBeenCalledWith(readFailure, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + ) + expect(session.entries.value.at(-1)).toMatchObject({ + role: 'assistant', + parts: [ + { + type: 'thinking', + text: 'live after reload', + state: 'done' + } + ] + }) + + conversation.reconcileSettledTurn( + { threadId: 'th-1', messageId: toTurnId('msg-1') }, + [{ type: 'text', text: 'a later read', state: 'done' }] + ) + expect(session.entries.value.at(-1)).toMatchObject({ + parts: [{ type: 'thinking', text: 'live after reload' }] + }) + }) + + it('reads a settled snapshot turn again while its row still reads as streaming', async () => { + const streamingRow = { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' as const + } + const getMessages = vi + .fn() + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + streamingRow + ]) + .mockImplementationOnce(hangingGetMessages) + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + streamingRow + ]) + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', 'Added it.', 'msg-1'), + content: { + text: 'Added it.', + thinking: [{ text: 'before the reload' }] + } + } + ]) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + + await vi.waitFor( + () => { + const last = session.entries.value.at(-1) + expect( + last?.role === 'assistant' + ? last.parts.flatMap((part) => + part.type === 'thinking' ? [part.text] : [] + ) + : [] + ).toContain('before the reload') + }, + { timeout: 5000 } + ) + }) + + it('honors a hydrate deferred behind an in-flight settled snapshot read', async () => { + const restored: AgentMessages = [ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' + } + ] + const finished: AgentMessages = [ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', 'Added it.', 'msg-1'), + content: { + text: 'Added it.', + thinking: [{ text: 'before the reload' }] + } + } + ] + let settleRead: ((history: AgentMessages) => void) | undefined + const getMessages = vi + .fn() + .mockResolvedValueOnce(restored) + .mockImplementationOnce(hangingGetMessages) + .mockImplementationOnce( + () => + new Promise((resolve) => { + settleRead = resolve + }) + ) + .mockResolvedValueOnce(restored) + .mockResolvedValue(finished) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(3)) + + session.newChat('new_chat_button') + void session.loadThread('th-1') + await vi.waitFor(() => expect(session.isStreaming.value).toBe(true)) + + assert.exists(settleRead) + settleRead(finished) + + await vi.waitFor(() => expect(session.isStreaming.value).toBe(false), { + timeout: 2000 + }) + expect(conversation.activeTurnId).toBeNull() + }) + it('re-attaches to the running turn when a send is refused with TURN_IN_PROGRESS', async () => { const conversation = useAgentConversationStore() const postMessage = vi diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts index da1a88ed9eb..44c3469aaad 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts @@ -63,6 +63,7 @@ import { useAgentWorkflowTabBindingStore } from '../../stores/agent/agentWorkflo import type { ComposerAttachment } from '../../types/composerAttachment' import type { WorkflowReference } from '../../types/workflowReference' import { serializeWorkflowReferences } from '../../utils/workflowReferenceText' +import { TURN_RECOVERY_DELAYS_MS } from '../../services/agent/turnRecoverySchedule' export interface AgentEventSource { subscribe(listener: (raw: unknown) => void): () => void @@ -185,12 +186,6 @@ const HYDRATION_MAILBOX_MS = 5 * 60_000 const MAX_HYDRATION_EVENTS = 256 const MAX_HYDRATION_MAILBOXES = 32 -/** - * After a reconnect the server may still be finishing the turn, and its - * terminal event may or may not reach the new socket. Poll the persisted row - * with backoff; once the schedule is exhausted the socket alone is trusted. - */ -const TURN_RECOVERY_DELAYS_MS = [0, 1000, 2000, 4000, 8000, 16000] /** Upper bound on one recovery job, including any history fetch still in flight. */ const TURN_RECOVERY_DEADLINE_MS = 60_000 @@ -198,7 +193,7 @@ type TurnOutcome = | { kind: 'terminal'; parts: AssistantMessage['parts'] | undefined } | { kind: 'thread-missing' } | { kind: 'streaming' } - | { kind: 'error'; message: string } + | { kind: 'error'; cause: unknown } function isTerminalTurnStatus( status: AgentMessages[number]['status'] @@ -259,10 +254,7 @@ function recoveryKey(turn: LiveTurn): string { function turnOutcomeFromError(error: unknown): TurnOutcome { if (error instanceof AgentApiError && error.status === 404) return { kind: 'thread-missing' } - return { - kind: 'error', - message: error instanceof Error ? error.message : String(error) - } + return { kind: 'error', cause: error } } const NON_RETRYABLE_REQUEST_STATUSES = new Set([ @@ -580,6 +572,7 @@ export function useAgentSession(deps: AgentSessionDeps) { let unsubscribe: (() => void) | null = null let unsubscribeStatus: (() => void) | null = null + let unsubscribeSnapshotSettled: (() => void) | null = null let ownedGeneration = 0 let connection: SocketConnection = 'initial' const recoveringTurns = new Map() @@ -641,6 +634,9 @@ export function useAgentSession(deps: AgentSessionDeps) { initializeWorkflowContext(initialThreadId !== null) unsubscribe = events.subscribe(onRaw) if (events.onStatus) unsubscribeStatus = events.onStatus(onStatus) + unsubscribeSnapshotSettled = conversationStore.onSnapshotTurnSettled( + (turn) => void reconcileSettledSnapshotTurn(turn) + ).off if (!restore) return if (initialThreadId === null) { onThreadActivated?.(null) @@ -924,8 +920,10 @@ export function useAgentSession(deps: AgentSessionDeps) { readyThreadId.value = null unsubscribe?.() unsubscribeStatus?.() + unsubscribeSnapshotSettled?.() unsubscribe = null unsubscribeStatus = null + unsubscribeSnapshotSettled = null for (const recovery of recoveringTurns.values()) recovery.abort() recoveringTurns.clear() const stoppedGeneration = ownedGeneration @@ -1889,6 +1887,69 @@ export function useAgentSession(deps: AgentSessionDeps) { } } + async function fetchSettledTurnOutcome( + turn: LiveTurn, + generation: number, + signal: AbortSignal + ): Promise { + let outcome = await fetchTurnOutcome(turn, signal) + for (const ms of TURN_RECOVERY_DELAYS_MS.slice(1)) { + // A failed read is retried like a still-streaming row: the turn has + // already settled, so only its stored thinking is at stake, and one + // transient failure would otherwise drop it for good. + const retryable = outcome.kind === 'streaming' || outcome.kind === 'error' + if (!retryable || generation !== sessionGeneration) return outcome + await delay(ms, { signal }) + outcome = await fetchTurnOutcome(turn, signal) + } + return outcome + } + + async function reconcileSettledSnapshotTurn(turn: LiveTurn): Promise { + const generation = ownedGeneration + const key = recoveryKey(turn) + recoveringTurns.get(key)?.abort() + const recovery = new AbortController() + recoveringTurns.set(key, recovery) + const deadline = setTimeout( + () => recovery.abort(), + TURN_RECOVERY_DEADLINE_MS + ) + try { + const outcome = await fetchSettledTurnOutcome( + turn, + generation, + recovery.signal + ) + if (generation !== sessionGeneration) return + if (outcome.kind === 'error') { + reportError(outcome.cause, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + return + } + if (outcome.kind !== 'terminal') return + conversationStore.reconcileSettledTurn(turn, outcome.parts) + } catch (error) { + if (!recovery.signal.aborted) + reportError(error, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + } finally { + clearTimeout(deadline) + if (recoveringTurns.get(key) === recovery) { + recoveringTurns.delete(key) + if (generation === sessionGeneration) + conversationStore.reconcileSettledTurn(turn, undefined) + // A hydrate while this read held the key skipped recovering the row + // it restored (reconcileLiveTurns), so recover it now if it is live. + if (isTurnLive(turn, ownedGeneration)) void reconcileTurn(turn) + } + } + } + async function recoverTurn( turn: LiveTurn, generation: number, diff --git a/src/workbench/extensions/agent/services/agent/turnRecoverySchedule.ts b/src/workbench/extensions/agent/services/agent/turnRecoverySchedule.ts new file mode 100644 index 00000000000..574d6e1d0f4 --- /dev/null +++ b/src/workbench/extensions/agent/services/agent/turnRecoverySchedule.ts @@ -0,0 +1 @@ +export const TURN_RECOVERY_DELAYS_MS = [0, 1000, 2000, 4000, 8000, 16000] diff --git a/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts b/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts index 5aea3138bfb..eacf01d0b7c 100644 --- a/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts +++ b/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts @@ -8,6 +8,7 @@ import type { TurnId } from '@/workbench/extensions/agent/schemas/agentApiSchema' import { + toTurnId, zAgentMessages, zAgentWsEvent } from '@/workbench/extensions/agent/schemas/agentApiSchema' @@ -155,6 +156,8 @@ const activeTab = ( data: { workflow_id: workflowId, message_id: id, thread_id: threadId } }) +type ConversationStore = ReturnType + const tabLinkIds = (store: ReturnType) => store.messages.flatMap((m) => m.parts.flatMap((p) => (p.type === 'tabLink' ? [p.workflowId] : [])) @@ -1955,6 +1958,375 @@ describe('useAgentConversationStore', () => { } ) + describe('reconciling a snapshot-restored turn settled on the socket', () => { + const SNAPSHOT_ROW_ID = toTurnId('assistant-row-7') + const SNAPSHOT_TURN_ID = toTurnId('turn-7') + const liveSnapshot = (): AgentMessages => + zAgentMessages.parse([ + historyRow(1, 'user', SNAPSHOT_TURN_ID, 'add a loader', 'user-row-1'), + { + id: SNAPSHOT_ROW_ID, + thread_id: 'th', + seq: 2, + role: 'assistant', + status: 'streaming', + turn_id: SNAPSHOT_TURN_ID + } + ]) + const storedTool = (): AssistantMessage['parts'][number] => ({ + type: 'tool', + callId: 'call-add_node', + name: 'add_node', + state: 'done', + ok: true + }) + const storedParts = (): AssistantMessage['parts'] => [ + { type: 'thinking', text: 'before the reload', state: 'done' }, + storedTool(), + { type: 'thinking', text: 'after the reload', state: 'done' }, + { type: 'text', text: 'Added it.', state: 'done' } + ] + const reconciledParts = ( + store: ConversationStore + ): AssistantMessage['parts'] | undefined => + store.messages.find((message) => message.id === SNAPSHOT_TURN_ID)?.parts + const restoredAndSettled = () => { + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest(thinking(SNAPSHOT_ROW_ID, 'after the reload')) + store.ingest(done(SNAPSHOT_ROW_ID)) + return { store, settled } + } + + it('announces the turn and folds the stored row in', () => { + const { store, settled } = restoredAndSettled() + expect(settled).toEqual([SNAPSHOT_ROW_ID]) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + const message = store.messages.find((m) => m.id === SNAPSHOT_TURN_ID) + expect(message?.streaming).toBe(false) + expect(message?.parts).toEqual(storedParts()) + }) + + it('reconciles once, so a second read cannot re-apply the row', () => { + const { store } = restoredAndSettled() + const turn = { threadId: 'th', messageId: SNAPSHOT_ROW_ID } + store.reconcileSettledTurn(turn, storedParts()) + + store.reconcileSettledTurn(turn, [ + { type: 'text', text: 'a later read', state: 'done' } + ]) + + expect( + store.messages.find((m) => m.id === SNAPSHOT_TURN_ID)?.parts + ).toEqual(storedParts()) + }) + + it.for([ + { + reason: 'is longer than the stored copy', + live: 'after the reload continues' + }, + { reason: 'is a suffix of the stored copy', live: ' reload' }, + { + reason: 'is missing from the stored row', + live: 'a round the row dropped' + }, + { reason: 'repeats an earlier stored round', live: 'before the reload' } + ])( + 'renders the stored thinking when the live round $reason', + ({ live }) => { + const store = useAgentConversationStore() + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest(thinking(SNAPSHOT_ROW_ID, live)) + store.ingest(done(SNAPSHOT_ROW_ID)) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + expect(reconciledParts(store)).toEqual(storedParts()) + } + ) + + it('renders the stored thinking when the row keys the live call differently', () => { + const store = useAgentConversationStore() + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest(thinking(SNAPSHOT_ROW_ID, 'after the reload, partial')) + store.ingest(toolCall(SNAPSHOT_ROW_ID, 'save', 'success')) + store.ingest(done(SNAPSHOT_ROW_ID)) + const persisted: AssistantMessage['parts'] = [ + { type: 'thinking', text: 'before the reload', state: 'done' }, + storedTool(), + { type: 'thinking', text: 'after the reload', state: 'done' }, + { + type: 'tool', + callId: 'row-fallback-id', + name: 'save', + state: 'done', + ok: true + }, + { type: 'text', text: 'Saved it.', state: 'done' } + ] + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + persisted + ) + + expect(reconciledParts(store)).toEqual(persisted) + }) + + it('keeps live thinking when the stored row holds no thinking at all', () => { + const { store } = restoredAndSettled() + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + [storedTool(), { type: 'text', text: 'Added it.', state: 'done' }] + ) + + expect(reconciledParts(store)).toEqual([ + storedTool(), + { type: 'thinking', text: 'after the reload', state: 'done' }, + { type: 'text', text: 'Added it.', state: 'done' } + ]) + }) + + it('places a post-reload tab link after the stored pre-reload tool', () => { + const store = useAgentConversationStore() + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest(activeTab('wf-1', SNAPSHOT_ROW_ID)) + store.ingest(thinking(SNAPSHOT_ROW_ID, 'after the reload')) + store.ingest(done(SNAPSHOT_ROW_ID)) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + expect(reconciledParts(store)?.map((part) => part.type)).toEqual([ + 'thinking', + 'tool', + 'thinking', + 'tabLink', + 'text' + ]) + }) + + it('keeps a live canvas confirmation for a call the stored row keys by row id', () => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const store = useAgentConversationStore() + store.setCanvasSyncGate(() => 'awaiting') + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(done(SNAPSHOT_ROW_ID)) + store.notifyCanvasCaughtUp(liveCanvasEvidence(workflowId, ['op-add'])) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + [ + { + type: 'tool', + callId: 'row-fallback-id', + name: 'add_node', + state: 'done', + ok: true, + canvasConfirmed: false + }, + { type: 'text', text: 'Added it.', state: 'done' } + ] + ) + + expect( + reconciledParts(store)?.find((part) => part.type === 'tool') + ).toMatchObject({ callId: 'row-fallback-id', canvasConfirmed: true }) + }) + + it('lets stored thinking win when the recovery read settles the turn', () => { + const store = useAgentConversationStore() + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest(thinking(SNAPSHOT_ROW_ID, 'a round the row dropped')) + + store.settleTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + expect(reconciledParts(store)).toEqual(storedParts()) + }) + + it('waits for a held tool call to clear before announcing the turn', () => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setCanvasSyncGate(() => 'awaiting') + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(done(SNAPSHOT_ROW_ID)) + expect(settled).toEqual([]) + + store.notifyCanvasCaughtUp(liveCanvasEvidence(workflowId, ['op-add'])) + expect(settled).toEqual([SNAPSHOT_ROW_ID]) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts().map((part) => + part.type === 'tool' ? { ...part, canvasConfirmed: false } : part + ) + ) + + expect( + store.messages + .find((message) => message.id === SNAPSHOT_TURN_ID) + ?.parts.find((part) => part.type === 'tool') + ).toMatchObject({ + callId: 'call-add_node', + ok: true, + canvasConfirmed: true + }) + }) + + it.for([ + { through: 'reset', leave: (store: ConversationStore) => store.reset() }, + { + through: 'hydrate', + leave: (store: ConversationStore) => store.hydrate([]) + } + ])( + 'does not announce a held turn when the thread is left through $through', + ({ leave }) => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setCanvasSyncGate(() => 'awaiting') + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(done(SNAPSHOT_ROW_ID)) + + leave(store) + + expect(settled).toEqual([]) + } + ) + + it('keeps the reconciled row when a late ask retirement follows a stale held tool call', () => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setCanvasSyncGate(() => 'awaiting') + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(runApproval(SNAPSHOT_ROW_ID, 'turn-7:call-1')) + store.ingest(done(SNAPSHOT_ROW_ID)) + expect(settled).toEqual([]) + + vi.advanceTimersByTime(30_000) + expect(settled).toEqual([SNAPSHOT_ROW_ID]) + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + store.retireAsk('turn-7:call-1', { status: 'retired' }) + + expect(reconciledParts(store)).toEqual(storedParts()) + }) + + it('announces a held turn only once when the canvas reports twice', () => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setCanvasSyncGate(() => 'awaiting') + store.setThreadId('th') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(done(SNAPSHOT_ROW_ID)) + + store.notifyCanvasCaughtUp(liveCanvasEvidence(workflowId, ['op-add'])) + store.notifyCanvasCaughtUp(liveCanvasEvidence(workflowId, ['op-add'])) + + expect(settled).toEqual([SNAPSHOT_ROW_ID]) + }) + + it('leaves a turn this client started alone', () => { + const store = useAgentConversationStore() + const settled: TurnId[] = [] + store.onSnapshotTurnSettled((turn) => settled.push(turn.messageId)) + store.setThreadId('th') + store.startTurn(T1) + store.ingest(delta('t1', 'local reply')) + store.ingest(done('t1')) + expect(settled).toEqual([]) + + store.reconcileSettledTurn({ threadId: 'th', messageId: T1 }, [ + { type: 'text', text: 'persisted reply', state: 'done' } + ]) + + expect(messageTexts(store, T1)).toEqual(['local reply']) + }) + + it('drops the registration when the thread is left', () => { + const { store } = restoredAndSettled() + store.reset() + store.setThreadId('th') + store.startTurn(SNAPSHOT_TURN_ID) + + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + expect( + store.messages.find((m) => m.id === SNAPSHOT_TURN_ID)?.parts + ).toEqual([]) + }) + }) + it('keeps a local tab link between persisted text and tool parts', () => { const store = useAgentConversationStore() store.setThreadId('th') diff --git a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts index bd220485023..ae625db5367 100644 --- a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts +++ b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts @@ -1,4 +1,5 @@ import type { AgentAnswerRequest } from '@comfyorg/ingest-types' +import { createEventHook } from '@vueuse/core' import { defineStore } from 'pinia' import { computed, ref, shallowRef } from 'vue' @@ -93,6 +94,13 @@ export interface LiveTurn { messageId: TurnId } +interface UnreconciledSnapshotTurn { + turn: LiveTurn + messageId: string + transport: AgentEventTransport + announced: boolean +} + interface AnchoredLocalPart { part: AssistantMessage['parts'][number] toolCount: number @@ -203,9 +211,88 @@ function interleaveLocalParts( return mergedParts } +function toolCallIdsOf(parts: AssistantMessage['parts']): string[] { + return parts.flatMap((part) => (part.type === 'tool' ? [part.callId] : [])) +} + +function liveToolOffset( + liveParts: AssistantMessage['parts'], + persistedParts: AssistantMessage['parts'] +): number { + const live = toolCallIdsOf(liveParts) + const persisted = toolCallIdsOf(persistedParts) + if (live.length === 0) return persisted.length + for (const [index, callId] of live.entries()) { + const at = persisted.indexOf(callId) + if (at >= index) return at - index + } + return Math.max(0, persisted.length - live.length) +} + +function storedCallIdOf( + liveParts: AssistantMessage['parts'], + persistedParts: AssistantMessage['parts'], + offset: number +): (callId: string) => string { + const liveCallIds = toolCallIdsOf(liveParts) + const persistedCallIds = toolCallIdsOf(persistedParts) + const persisted = new Set(persistedCallIds) + const live = new Set(liveCallIds) + return (callId) => { + if (persisted.has(callId)) return callId + const position = liveCallIds.indexOf(callId) + offset + const atPosition = + position < persistedCallIds.length + ? persistedCallIds.at(position) + : undefined + return atPosition !== undefined && !live.has(atPosition) + ? atPosition + : callId + } +} + +function preserveCanvasConfirmation( + liveParts: AssistantMessage['parts'], + persistedParts: AssistantMessage['parts'], + storedCallId: (callId: string) => string +): AssistantMessage['parts'] { + const confirmedCallIds = new Set( + liveParts.flatMap((part) => + part.type === 'tool' && part.canvasConfirmed === true + ? [storedCallId(part.callId)] + : [] + ) + ) + return persistedParts.map((part) => + part.type === 'tool' && + part.ok === true && + confirmedCallIds.has(part.callId) + ? { ...part, canvasConfirmed: true } + : part + ) +} + +function finishSnapshotTurn( + message: AssistantMessage, + persistedParts: AssistantMessage['parts'] +): void { + const liveParts = message.parts + const offset = liveToolOffset(liveParts, persistedParts) + finishWithPersistedParts( + message, + preserveCanvasConfirmation( + liveParts, + persistedParts, + storedCallIdOf(liveParts, persistedParts, offset) + ), + offset + ) +} + function finishWithPersistedParts( message: AssistantMessage, - persistedParts: AssistantMessage['parts'] | undefined + persistedParts: AssistantMessage['parts'] | undefined, + localToolOffset = 0 ): void { if (persistedParts === undefined) { message.parts = message.parts.filter((part) => !isOpenAskPart(part)) @@ -227,9 +314,12 @@ function finishWithPersistedParts( ) message.parts = interleaveLocalParts( persistedParts, - anchorLocalParts(message.parts).filter( - ({ part }) => !persistedHasThinking || part.type !== 'thinking' - ) + anchorLocalParts(message.parts) + .filter(({ part }) => !persistedHasThinking || part.type !== 'thinking') + .map((anchor) => ({ + ...anchor, + toolCount: anchor.toolCount + localToolOffset + })) ) } @@ -276,6 +366,11 @@ export const useAgentConversationStore = defineStore( // has nothing left pending. const settledActiveTransports = new Set() const backgroundTurns = new Map() + const unreconciledSnapshotTurns = new Map< + TurnId, + UnreconciledSnapshotTurn + >() + const snapshotTurnSettled = createEventHook() let hydratedMessageIds = new Set() let hydratedTurnIds = new Map() let hydratedAssistantTurnIds = new Set() @@ -327,6 +422,18 @@ export const useAgentConversationStore = defineStore( if (index >= 0) messages.value[index] = message } + // The snapshot turn's transport publishes through here, so a settled + // snapshot turn is announced once its last held tool call clears. + function publishSnapshotMessage( + turnId: TurnId, + message: AssistantMessage + ): void { + replaceActive(message) + const unreconciled = unreconciledSnapshotTurns.get(turnId) + if (unreconciled !== undefined) + announceReconcilableSnapshotTurn(unreconciled) + } + function rememberAttachmentNames(attachments: UserAttachment[]): void { const currentThreadId = threadId.value if (currentThreadId === null) return @@ -612,9 +719,59 @@ export const useAgentConversationStore = defineStore( function settleActiveTurn(activeTransport: AgentEventTransport): void { rememberDepartedActiveTurn('settled-turn') activeTransport.settle() + const restoredFromSnapshot = recordUnreconciledSnapshotTurn() if (activeTransport.hasPendingCanvasSync()) settledActiveTransports.add(activeTransport) clearActive() + if (restoredFromSnapshot) + announceReconcilableSnapshotTurn(restoredFromSnapshot) + } + + function recordUnreconciledSnapshotTurn(): + | UnreconciledSnapshotTurn + | undefined { + const slot = activeSlot.value + if (slot?.origin !== 'snapshot' || slot.threadId === null) + return undefined + const unreconciled: UnreconciledSnapshotTurn = { + turn: { threadId: slot.threadId, messageId: slot.turnId }, + messageId: slot.message.id, + transport: slot.transport, + announced: false + } + unreconciledSnapshotTurns.set(slot.turnId, unreconciled) + return unreconciled + } + + function announceReconcilableSnapshotTurn( + unreconciled: UnreconciledSnapshotTurn + ): void { + if ( + unreconciled.announced || + unreconciled.transport.hasPendingCanvasSync() + ) + return + settledActiveTransports.delete(unreconciled.transport) + unreconciled.announced = true + void snapshotTurnSettled.trigger(unreconciled.turn) + } + + function reconcileSettledTurn( + turn: LiveTurn, + persistedParts: AssistantMessage['parts'] | undefined + ): void { + const unreconciled = unreconciledSnapshotTurns.get(turn.messageId) + if (unreconciled === undefined) return + unreconciledSnapshotTurns.delete(turn.messageId) + if (persistedParts === undefined || threadId.value !== turn.threadId) + return + const index = messages.value.findIndex( + (message) => message.id === unreconciled.messageId + ) + if (index < 0) return + const reconciled = { ...messages.value[index] } + finishSnapshotTurn(reconciled, persistedParts) + messages.value[index] = reconciled } function ingestActiveTabEvent( @@ -1048,6 +1205,7 @@ export const useAgentConversationStore = defineStore( } function reset(): void { + unreconciledSnapshotTurns.clear() disposeActiveAndSettledTransports() messages.value = [] userTexts.value = new Map() @@ -1091,6 +1249,7 @@ export const useAgentConversationStore = defineStore( function hydrate(history: AgentMessages): void { if (activeSlot.value) rememberDepartedActiveTurn('no-live-turn') + unreconciledSnapshotTurns.clear() disposeActiveAndSettledTransports() clearActive() const transcript = normalizeAgentTranscript(history) @@ -1147,7 +1306,7 @@ export const useAgentConversationStore = defineStore( message: pending.message, transport: createAgentEventTransport( pending.message, - replaceActive, + (message) => publishSnapshotMessage(pending.messageId, message), (workflowId) => canvasSyncGate(workflowId), () => canvasSyncOutcomeCount(), reportUndeliverableAskData @@ -1255,6 +1414,7 @@ export const useAgentConversationStore = defineStore( entries, activeTurnId, activeMessageId, + onSnapshotTurnSettled: snapshotTurnSettled.on, threadId, isStreaming, status, @@ -1288,6 +1448,7 @@ export const useAgentConversationStore = defineStore( turnUsesSkill, historySkillNames, settleTurn, + reconcileSettledTurn, reset, hydrate } From 334d709cb37a0616bb1430c8241d7506b0ea6fe2 Mon Sep 17 00:00:00 2001 From: bymyself Date: Mon, 2 Nov 2026 01:03:35 +0000 Subject: [PATCH 2/4] fix(agent): keep a settled snapshot read from taking a later turn's registration Follow-up to #22244 (FRONT-3659), from its adversarial review: - reconcileSettledTurn only consumes an announced registration, so a read still finishing for an earlier settle cannot take or drop a re-registered turn whose own read has not started yet. - The settled read retries a failed read on the same schedule as a row that still reads as streaming, and reports only if the last read fails. - The read's teardown clears the recovery ledgers only while it still owns the key, before any deferred recovery reruns. (cherry picked from commit 60f9354f4fa92fdb78ee848fa3ce3169b93cbec8) --- .../composables/agent/useAgentSession.test.ts | 75 +++++++++++++++++-- .../agent/agentConversationStore.test.ts | 31 ++++++++ .../stores/agent/agentConversationStore.ts | 4 +- 3 files changed, 102 insertions(+), 8 deletions(-) diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts index dd86d2255ea..bd51b753737 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts @@ -840,6 +840,7 @@ describe('useAgentSession (v1 composition root)', () => { rejectSettledRead = reject }) ) + .mockRejectedValue(readFailure) const conversation = useAgentConversationStore() conversation.setThreadId('th-1') const { source, emit } = fakeEvents() @@ -866,14 +867,20 @@ describe('useAgentSession (v1 composition root)', () => { }) await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(3)) assert.exists(rejectSettledRead) - rejectSettledRead(readFailure) + vi.useFakeTimers() + try { + rejectSettledRead(readFailure) + await vi.advanceTimersByTimeAsync(40_000) + } finally { + vi.useRealTimers() + } - await vi.waitFor(() => - expect(reportError).toHaveBeenCalledWith(readFailure, { - surface: 'agent', - errorType: 'failure_recovering_agent_turn' - }) - ) + expect(getMessages).toHaveBeenCalledTimes(8) + expect(reportError).toHaveBeenCalledTimes(1) + expect(reportError).toHaveBeenCalledWith(readFailure, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) expect(session.entries.value.at(-1)).toMatchObject({ role: 'assistant', parts: [ @@ -894,6 +901,60 @@ describe('useAgentSession (v1 composition root)', () => { }) }) + it('reads a settled snapshot turn again after a transient read failure', async () => { + const getMessages = vi + .fn() + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' + } + ]) + .mockImplementationOnce(hangingGetMessages) + .mockRejectedValueOnce( + new AgentApiError('history unavailable', 503, undefined) + ) + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', 'Added it.', 'msg-1'), + content: { + text: 'Added it.', + thinking: [{ text: 'before the reload' }] + } + } + ]) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + + await vi.waitFor( + () => { + const last = session.entries.value.at(-1) + expect( + last?.role === 'assistant' + ? last.parts.flatMap((part) => + part.type === 'thinking' ? [part.text] : [] + ) + : [] + ).toEqual(['before the reload']) + }, + { timeout: 5000 } + ) + expect(reportError).not.toHaveBeenCalled() + }) + it('reads a settled snapshot turn again while its row still reads as streaming', async () => { const streamingRow = { ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), diff --git a/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts b/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts index eacf01d0b7c..e661aa983a9 100644 --- a/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts +++ b/src/workbench/extensions/agent/stores/agent/agentConversationStore.test.ts @@ -2271,6 +2271,37 @@ describe('useAgentConversationStore', () => { expect(reconciledParts(store)).toEqual(storedParts()) }) + it('keeps a re-registered held turn for its own read when an earlier read lands late', () => { + const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' + const { store, settled } = restoredAndSettled() + const earlierRead = { threadId: 'th', messageId: SNAPSHOT_ROW_ID } + store.setCanvasSyncGate(() => 'awaiting') + store.hydrate(liveSnapshot()) + store.ingest( + toolCall(SNAPSHOT_ROW_ID, 'add_node', 'success', { + workflow_id: workflowId, + op_ids: ['op-add'] + }) + ) + store.ingest(done(SNAPSHOT_ROW_ID)) + expect(settled).toEqual([SNAPSHOT_ROW_ID]) + + store.reconcileSettledTurn(earlierRead, undefined) + store.notifyCanvasCaughtUp(liveCanvasEvidence(workflowId, ['op-add'])) + expect(settled).toEqual([SNAPSHOT_ROW_ID, SNAPSHOT_ROW_ID]) + store.reconcileSettledTurn( + { threadId: 'th', messageId: SNAPSHOT_ROW_ID }, + storedParts() + ) + + expect(reconciledParts(store)?.map((part) => part.type)).toEqual([ + 'thinking', + 'tool', + 'thinking', + 'text' + ]) + }) + it('announces a held turn only once when the canvas reports twice', () => { const workflowId = 'f2b4c6d8-1a3e-4b5c-9d7e-0f1a2b3c4d5e' const store = useAgentConversationStore() diff --git a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts index ae625db5367..3ce1d70af8f 100644 --- a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts +++ b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts @@ -761,7 +761,9 @@ export const useAgentConversationStore = defineStore( persistedParts: AssistantMessage['parts'] | undefined ): void { const unreconciled = unreconciledSnapshotTurns.get(turn.messageId) - if (unreconciled === undefined) return + // An unannounced registration belongs to a later settle whose read has + // not started; a read still finishing for an earlier one must not take it. + if (unreconciled === undefined || !unreconciled.announced) return unreconciledSnapshotTurns.delete(turn.messageId) if (persistedParts === undefined || threadId.value !== turn.threadId) return From 6f2c9d0ae0c3cba450da16c07224fc6eb516d96e Mon Sep 17 00:00:00 2001 From: bymyself Date: Mon, 2 Nov 2026 01:45:01 +0000 Subject: [PATCH 3/4] fix(agent): retry only retryable settled reads and report one failing at its deadline Review follow-ups on the settled snapshot read: - Retry an 'error' outcome only when isRetryableRequestFailure says a retry can help, so a refused or malformed read is reported at once instead of costing five more thread downloads. - When the deadline aborts the read while it is still failing, report the last failure rather than swallowing it as an ordinary abort. - Apply a successful read only while this read still owns the turn's key. - Clear the recovery ledgers after finishRecovery again, so a deferred rerun still sees this recovery's observation, and only while this read owned the key. - Drop the code comments the previous commit added. --- .../composables/agent/useAgentSession.test.ts | 86 +++++++++++++++ .../composables/agent/useAgentSession.ts | 103 ++++++++++++------ .../stores/agent/agentConversationStore.ts | 4 - 3 files changed, 158 insertions(+), 35 deletions(-) diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts index bd51b753737..3d5bda67fac 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts @@ -901,6 +901,92 @@ describe('useAgentSession (v1 composition root)', () => { }) }) + it('does not retry a settled snapshot read the server refused', async () => { + const refused = new AgentApiError('unauthorized', 401, undefined) + const getMessages = vi + .fn() + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' + } + ]) + .mockImplementationOnce(hangingGetMessages) + .mockRejectedValue(refused) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + vi.useFakeTimers() + try { + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + await vi.advanceTimersByTimeAsync(40_000) + } finally { + vi.useRealTimers() + } + + expect(getMessages).toHaveBeenCalledTimes(3) + expect(reportError).toHaveBeenCalledTimes(1) + expect(reportError).toHaveBeenCalledWith(refused, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + }) + + it('reports a settled snapshot read that is still failing when its deadline passes', async () => { + const slowFailure = new AgentApiError('gateway timeout', 504, undefined) + const getMessages = vi + .fn() + .mockResolvedValueOnce([ + historyRow(1, 'user', 'turn-1', 'add an audio output node'), + { + ...historyRow(2, 'assistant', 'turn-1', '', 'msg-1'), + content: {}, + status: 'streaming' + } + ]) + .mockImplementationOnce(hangingGetMessages) + .mockImplementation( + () => + new Promise((_resolve, reject) => { + setTimeout(() => reject(slowFailure), 25_000) + }) + ) + const conversation = useAgentConversationStore() + conversation.setThreadId('th-1') + const { source, emit } = fakeEvents() + const session = useAgentSession({ + rest: fakeRest({ getMessages }), + events: source + }) + session.start() + await vi.waitFor(() => expect(getMessages).toHaveBeenCalledTimes(2)) + + vi.useFakeTimers() + try { + emit(thinking('msg-1', 'live after reload')) + emit(done('msg-1')) + await vi.advanceTimersByTimeAsync(90_000) + } finally { + vi.useRealTimers() + } + + expect(reportError).toHaveBeenCalledTimes(1) + expect(reportError).toHaveBeenCalledWith(slowFailure, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + }) + it('reads a settled snapshot turn again after a transient read failure', async () => { const getMessages = vi .fn() diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts index 44c3469aaad..33b7a626a47 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts @@ -1887,66 +1887,107 @@ export function useAgentSession(deps: AgentSessionDeps) { } } + function isRetryableSettledRead(outcome: TurnOutcome): boolean { + if (outcome.kind === 'streaming') return true + return ( + outcome.kind === 'error' && + isRetryableRequestFailure(outcome.cause, false) + ) + } + async function fetchSettledTurnOutcome( turn: LiveTurn, generation: number, - signal: AbortSignal + signal: AbortSignal, + observe: (outcome: TurnOutcome) => void ): Promise { let outcome = await fetchTurnOutcome(turn, signal) + observe(outcome) for (const ms of TURN_RECOVERY_DELAYS_MS.slice(1)) { - // A failed read is retried like a still-streaming row: the turn has - // already settled, so only its stored thinking is at stake, and one - // transient failure would otherwise drop it for good. - const retryable = outcome.kind === 'streaming' || outcome.kind === 'error' - if (!retryable || generation !== sessionGeneration) return outcome + if (!isRetryableSettledRead(outcome) || generation !== sessionGeneration) + return outcome await delay(ms, { signal }) outcome = await fetchTurnOutcome(turn, signal) + observe(outcome) } return outcome } + function reportSettledReadFailure(cause: unknown): void { + reportError(cause, { + surface: 'agent', + errorType: 'failure_recovering_agent_turn' + }) + } + + function applySettledRead(turn: LiveTurn, outcome: TurnOutcome): void { + if (outcome.kind === 'error') reportSettledReadFailure(outcome.cause) + else if (outcome.kind === 'terminal') + conversationStore.reconcileSettledTurn(turn, outcome.parts) + } + + function settledReadAbortFailure( + error: unknown, + signal: AbortSignal, + deadlineReason: unknown, + latest: TurnOutcome | undefined + ): { cause: unknown } | undefined { + if (!signal.aborted) return { cause: error } + if (signal.reason === deadlineReason && latest?.kind === 'error') + return { cause: latest.cause } + return undefined + } + + function releaseSettledRead( + key: string, + recovery: AbortController, + turn: LiveTurn, + generation: number + ): void { + if (recoveringTurns.get(key) !== recovery) return + recoveringTurns.delete(key) + if (generation === sessionGeneration) + conversationStore.reconcileSettledTurn(turn, undefined) + if (isTurnLive(turn, ownedGeneration)) void reconcileTurn(turn) + } + async function reconcileSettledSnapshotTurn(turn: LiveTurn): Promise { const generation = ownedGeneration const key = recoveryKey(turn) recoveringTurns.get(key)?.abort() const recovery = new AbortController() recoveringTurns.set(key, recovery) + const latest: { outcome?: TurnOutcome } = {} + const deadlineReason = new Error('settled read deadline') const deadline = setTimeout( - () => recovery.abort(), + () => recovery.abort(deadlineReason), TURN_RECOVERY_DEADLINE_MS ) try { const outcome = await fetchSettledTurnOutcome( turn, generation, - recovery.signal + recovery.signal, + (observed) => { + latest.outcome = observed + } ) - if (generation !== sessionGeneration) return - if (outcome.kind === 'error') { - reportError(outcome.cause, { - surface: 'agent', - errorType: 'failure_recovering_agent_turn' - }) - return - } - if (outcome.kind !== 'terminal') return - conversationStore.reconcileSettledTurn(turn, outcome.parts) + if ( + generation === sessionGeneration && + recoveringTurns.get(key) === recovery + ) + applySettledRead(turn, outcome) } catch (error) { - if (!recovery.signal.aborted) - reportError(error, { - surface: 'agent', - errorType: 'failure_recovering_agent_turn' - }) + const failure = settledReadAbortFailure( + error, + recovery.signal, + deadlineReason, + latest.outcome + ) + if (failure) reportSettledReadFailure(failure.cause) } finally { clearTimeout(deadline) - if (recoveringTurns.get(key) === recovery) { - recoveringTurns.delete(key) - if (generation === sessionGeneration) - conversationStore.reconcileSettledTurn(turn, undefined) - // A hydrate while this read held the key skipped recovering the row - // it restored (reconcileLiveTurns), so recover it now if it is live. - if (isTurnLive(turn, ownedGeneration)) void reconcileTurn(turn) - } + releaseSettledRead(key, recovery, turn, generation) } } diff --git a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts index 3ce1d70af8f..8820a8dbf3c 100644 --- a/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts +++ b/src/workbench/extensions/agent/stores/agent/agentConversationStore.ts @@ -422,8 +422,6 @@ export const useAgentConversationStore = defineStore( if (index >= 0) messages.value[index] = message } - // The snapshot turn's transport publishes through here, so a settled - // snapshot turn is announced once its last held tool call clears. function publishSnapshotMessage( turnId: TurnId, message: AssistantMessage @@ -761,8 +759,6 @@ export const useAgentConversationStore = defineStore( persistedParts: AssistantMessage['parts'] | undefined ): void { const unreconciled = unreconciledSnapshotTurns.get(turn.messageId) - // An unannounced registration belongs to a later settle whose read has - // not started; a read still finishing for an earlier one must not take it. if (unreconciled === undefined || !unreconciled.announced) return unreconciledSnapshotTurns.delete(turn.messageId) if (persistedParts === undefined || threadId.value !== turn.threadId) From 831cac7647873397b898b9cbe23f94e9efb61fc9 Mon Sep 17 00:00:00 2001 From: bymyself Date: Mon, 2 Nov 2026 02:48:10 +0000 Subject: [PATCH 4/4] refactor(agent): let the settled read return its last failure at the deadline --- .../composables/agent/useAgentSession.test.ts | 15 ++---- .../composables/agent/useAgentSession.ts | 51 +++++++------------ 2 files changed, 24 insertions(+), 42 deletions(-) diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts index 3d5bda67fac..49484914980 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts @@ -1026,16 +1026,11 @@ describe('useAgentSession (v1 composition root)', () => { emit(done('msg-1')) await vi.waitFor( - () => { - const last = session.entries.value.at(-1) - expect( - last?.role === 'assistant' - ? last.parts.flatMap((part) => - part.type === 'thinking' ? [part.text] : [] - ) - : [] - ).toEqual(['before the reload']) - }, + () => + expect(session.entries.value.at(-1)).toMatchObject({ + role: 'assistant', + parts: [{ type: 'thinking', text: 'before the reload' }, {}] + }), { timeout: 5000 } ) expect(reportError).not.toHaveBeenCalled() diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts index 33b7a626a47..70ee094920e 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts @@ -1899,18 +1899,26 @@ export function useAgentSession(deps: AgentSessionDeps) { turn: LiveTurn, generation: number, signal: AbortSignal, - observe: (outcome: TurnOutcome) => void + deadlineReason: Error ): Promise { - let outcome = await fetchTurnOutcome(turn, signal) - observe(outcome) - for (const ms of TURN_RECOVERY_DELAYS_MS.slice(1)) { - if (!isRetryableSettledRead(outcome) || generation !== sessionGeneration) - return outcome - await delay(ms, { signal }) + let outcome: TurnOutcome | undefined + try { outcome = await fetchTurnOutcome(turn, signal) - observe(outcome) + for (const ms of TURN_RECOVERY_DELAYS_MS.slice(1)) { + if ( + !isRetryableSettledRead(outcome) || + generation !== sessionGeneration + ) + return outcome + await delay(ms, { signal }) + outcome = await fetchTurnOutcome(turn, signal) + } + return outcome + } catch (error) { + if (signal.reason === deadlineReason && outcome?.kind === 'error') + return outcome + throw error } - return outcome } function reportSettledReadFailure(cause: unknown): void { @@ -1926,18 +1934,6 @@ export function useAgentSession(deps: AgentSessionDeps) { conversationStore.reconcileSettledTurn(turn, outcome.parts) } - function settledReadAbortFailure( - error: unknown, - signal: AbortSignal, - deadlineReason: unknown, - latest: TurnOutcome | undefined - ): { cause: unknown } | undefined { - if (!signal.aborted) return { cause: error } - if (signal.reason === deadlineReason && latest?.kind === 'error') - return { cause: latest.cause } - return undefined - } - function releaseSettledRead( key: string, recovery: AbortController, @@ -1957,7 +1953,6 @@ export function useAgentSession(deps: AgentSessionDeps) { recoveringTurns.get(key)?.abort() const recovery = new AbortController() recoveringTurns.set(key, recovery) - const latest: { outcome?: TurnOutcome } = {} const deadlineReason = new Error('settled read deadline') const deadline = setTimeout( () => recovery.abort(deadlineReason), @@ -1968,9 +1963,7 @@ export function useAgentSession(deps: AgentSessionDeps) { turn, generation, recovery.signal, - (observed) => { - latest.outcome = observed - } + deadlineReason ) if ( generation === sessionGeneration && @@ -1978,13 +1971,7 @@ export function useAgentSession(deps: AgentSessionDeps) { ) applySettledRead(turn, outcome) } catch (error) { - const failure = settledReadAbortFailure( - error, - recovery.signal, - deadlineReason, - latest.outcome - ) - if (failure) reportSettledReadFailure(failure.cause) + if (!recovery.signal.aborted) reportSettledReadFailure(error) } finally { clearTimeout(deadline) releaseSettledRead(key, recovery, turn, generation)