From 99f2306972b3fa6e1efb92a75fae4e532f73f8f9 Mon Sep 17 00:00:00 2001 From: Christian Byrne Date: Fri, 9 Oct 2026 01:11:16 +0000 Subject: [PATCH] fix(agent): stop a run-mode change from overtaking a message already sent (#18706) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run-mode change picked right after Send used to govern the message already sent: the composer clears on the click, but `performSend` awaits `prepareWorkflow()` before issuing the POST, and the server pins a turn's run mode when that POST *arrives* (`resolveRunMode`, run inside `postMessage` before the `202`). `agentSendGateStore` now counts this page's unsettled message POSTs and `agentRunModeStore.save()` waits on that count before writing, bounded only by the gate's own backstop — `MAX_HOLD_MS`, 190s, derived from `api.fetchApi`'s auth-init and response-headers timeouts plus the remint and prepare budgets rather than hand-summed. Page-scoped: a second tab's write is ordered against its own sends only, and closing that needs the ordering point to move to the server. Fixes PM-1660, with the regression test PM-1661 asks for. - **What**: new `agentSendGateStore` — a page-global in-flight count, one single-use release per hold, and a backstop so a send that never settles cannot strand the control. `sendMessage` takes a hold before `prepareWorkflow()` and releases however the POST settles; `save()` waits on it inline. `load()` now snapshots `appliedSaveRevision` as well, so a load started inside the hold cannot put the pre-write mode back. `api.fetchApi`'s two request bounds move to `src/scripts/apiTimeouts.ts` so a caller that must *outlast* a request can derive from them: read from `api.ts` they arrive `undefined` under the manual mock, and `NaN` arithmetic arms a timer that fires immediately. - **Breaking**: none. - **Dependencies**: none. - `MAX_HOLD_MS` is a liveness escape hatch, not the ordering guarantee — the guarantee is the release in `sendMessage`'s `finally`. The budget covering `getAuthHeader()` and the unified-user wait is explicit, not provable; a provable bound needs an overall send deadline or a release tied to dispatch, and both change send behaviour beyond this fix. There is deliberately no test asserting the budget covers the window: the constant is derived from the same timeouts the window is built from, so raising one raises both sides and such a test cannot fail. - Two residuals documented in `agentSendGateStore`'s docblock rather than closed: a client-side abort or network error ends the hold without proving the server discarded the POST (it turns on whether cloud cancels on client disconnect, which needs an experiment against the service), and the hold starts in `sendMessage` rather than at `startSubmission`, leaving one consent round trip on an account's first message. Local, at `5b66a27` (branch merged with `main`): - `pnpm vitest run src/workbench/extensions/agent src/scripts` — 190 files, 3548 passed, 0 failures. - `pnpm typecheck` and `pnpm format:check` clean; `oxlint --type-aware` and `eslint` clean on the changed files. The whole-repo `oxlint --type-aware` pass is killed by the dev host on an unmodified tree too, so it ran per-file through the pre-commit hook and in CI. - Every new unit test was confirmed red against the code it protects: removing the gate wiring from `sendMessage` fails both `useAgentSession.test.ts` cases; removing the superseded-revision check fails `sends only the latest of two picks parked on one send`; reverting `load()`'s second snapshot fails `does not let a load started during the hold undo the saved mode`; removing the backstop fails both `agentSendGateStore.test.ts` cases by assertion rather than by timeout. - The `@cloud` Playwright spec is not runnable on the dev host (the agent panel's entry button does not render against a local CPU backend), so CI is its only real run. Its setup assertion was racing `agentRunModeStore.load()` and reaching its own assertion only on a fourth attempt; `db9a0c9` removes the race, and the no-retry `playwright-video-new-tests (cloud)` job passed at that head. Tests live in `agentSendGateStore.test.ts` (new), `agentRunModeStore.test.ts`, `useAgentSession.test.ts`, and `browser_tests/tests/agent/agentRunModePopover.spec.ts` (PM-1661). --------- Co-authored-by: glary (cherry picked from commit 8bcbd9f3cd912fe28e21d05b02439f71591d250d) --- .../tests/agent/agentRunModePopover.spec.ts | 98 ++++++++++++ src/scripts/api.ts | 14 +- src/scripts/apiTimeouts.ts | 22 +++ .../composables/agent/useAgentSession.test.ts | 45 ++++++ .../composables/agent/useAgentSession.ts | 4 + .../stores/agent/agentRunModeStore.test.ts | 141 ++++++++++++++++++ .../agent/stores/agent/agentRunModeStore.ts | 36 ++++- .../stores/agent/agentSendGateStore.test.ts | 32 ++++ .../agent/stores/agent/agentSendGateStore.ts | 110 ++++++++++++++ 9 files changed, 495 insertions(+), 7 deletions(-) create mode 100644 src/scripts/apiTimeouts.ts create mode 100644 src/workbench/extensions/agent/stores/agent/agentSendGateStore.test.ts create mode 100644 src/workbench/extensions/agent/stores/agent/agentSendGateStore.ts diff --git a/browser_tests/tests/agent/agentRunModePopover.spec.ts b/browser_tests/tests/agent/agentRunModePopover.spec.ts index bcf1476abf9..12035d61a30 100644 --- a/browser_tests/tests/agent/agentRunModePopover.spec.ts +++ b/browser_tests/tests/agent/agentRunModePopover.spec.ts @@ -196,4 +196,102 @@ test.describe('Agent run permissions popover', { tag: '@cloud' }, () => { expect(savedModes).toEqual(['auto', 'auto']) }) }) + + // PM-1660/PM-1661, reproduced from the reporter's recording: the POST that + // carries the message waits on the cloud-workflow refresh first. The server + // pins a turn's run mode when that POST arrives, so a mode written inside + // that window re-authorized a turn sent under the previous mode. + test('a mode picked while a message is still in flight is saved after it', async ({ + agentPanel, + comfyPage + }) => { + const page = comfyPage.page + const reachedServer: string[] = [] + let holdTheSend = false + let releaseTheSend: () => void = () => {} + const sendHeld = new Promise((resolve) => { + releaseTheSend = resolve + }) + + // Both halves of the send are held. Holding the workflow refresh alone + // would not be enough: prepareWorkflow() abandons it after + // PREPARE_TIMEOUT_MS and issues the POST anyway, which then waits here. + await page.route('**/api/workflows?*', async (route) => { + if (holdTheSend) await sendHeld + await route.fallback() + }) + await page.route('**/api/agent/threads/*/messages', async (route) => { + if (route.request().method() !== 'POST') return route.fallback() + // Recorded where the POST is FORWARDED, not where it is intercepted. + // Recording on interception would let a PUT that overtook a still-held + // POST still read back as ['message', 'run-mode'] and pass. + if (holdTheSend) await sendHeld + await route.fallback() + reachedServer.push('message') + }) + await page.route('**/api/agent/run-mode', async (route) => { + const request = route.request() + // ask_approval, which is also agentRunModeStore's own default, so the + // starting mode does not depend on this route winning a race. + // DockedAgentPanel fires load() on setup, and asserting a mode only this + // mock can produce was right about one attempt in four: it failed the + // initial run and two retries in CI and was reported 'flaky' only + // because a fourth attempt passed. + if (request.method() !== 'PUT') + return route.fulfill( + jsonRoute({ + mode: 'ask_approval', + credit_limit: null + } satisfies AgentRunModePreference) + ) + reachedServer.push('run-mode') + return route.fulfill( + jsonRoute(zAgentRunMode.parse(request.postDataJSON())) + ) + }) + + await agentPanel.open() + await agentPanel.selectWorkflow() + const panel = agentPanel.root + const askTrigger = panel.getByRole('button', { + name: enMessages.agent.runModeTriggerAsk, + exact: true + }) + const autoTrigger = panel.getByRole('button', { + name: enMessages.agent.runModeTriggerAuto, + exact: true + }) + await expect(askTrigger).toBeVisible() + + holdTheSend = true + await agentPanel.sendMessage('run the wf') + + // Picked in the ESCALATING direction: the turn was sent under + // ask_approval, so a PUT that overtook it would let an already-sent turn + // run without a consent card. + await test.step('switching mode now does not overtake the message', async () => { + await askTrigger.click() + await page + .getByRole('menuitemradio', { + name: new RegExp(enMessages.agent.runModeAuto) + }) + .click() + + // A cheap sanity guard, not the regression detector: reachedServer is + // pushed from a Node-side route handler, which the DOM assertion above + // it does not order against. The detector is the final assertion. + // Nothing at all should have been forwarded while the send is held. + await expect( + page.getByRole('menuitemradio', { + name: new RegExp(enMessages.agent.runModeAuto) + }) + ).toHaveAttribute('aria-busy', 'true') + expect(reachedServer).toEqual([]) + + releaseTheSend() + await expect(autoTrigger).toBeVisible() + // The load-bearing assertion: the write landed, and it landed second. + await expect.poll(() => reachedServer).toEqual(['message', 'run-mode']) + }) + }) }) diff --git a/src/scripts/api.ts b/src/scripts/api.ts index ff6def8a6ae..f6d4424d69e 100644 --- a/src/scripts/api.ts +++ b/src/scripts/api.ts @@ -23,6 +23,10 @@ import { fetchWithUnifiedRemint, shouldRemintCloudRequest } from '@/platform/auth/unified/remintRetry' +import { + AUTH_INIT_TIMEOUT_MS, + FETCH_RESPONSE_HEADERS_TIMEOUT_MS +} from '@/scripts/apiTimeouts' import { getDevOverride } from '@/utils/devFeatureFlagOverride' import { getSessionOverride } from '@/utils/sessionFeatureFlagOverride' import type { @@ -143,8 +147,6 @@ interface QueuePromptRequestBody { number?: number } -const FETCH_RESPONSE_HEADERS_TIMEOUT_MS = 60_000 - /** * Which cloud auth path a request actually took, for error diagnostics (PM-1802). * @@ -636,7 +638,7 @@ export class ComfyApi extends EventTarget { /** * Waits for Firebase auth to be initialized before proceeding. - * Includes 10-second timeout to prevent infinite hanging. + * Bounded by AUTH_INIT_TIMEOUT_MS to prevent infinite hanging. */ private async waitForAuthInitialization(): Promise { if (isCloud) { @@ -649,10 +651,12 @@ export class ComfyApi extends EventTarget { try { await Promise.race([ until(isInitialized).toBe(true), - promiseTimeout(10000) + promiseTimeout(AUTH_INIT_TIMEOUT_MS) ]) } catch { - console.warn('Firebase auth initialization timeout after 10 seconds') + console.warn( + `Firebase auth initialization timeout after ${AUTH_INIT_TIMEOUT_MS / 1000} seconds` + ) } } } diff --git a/src/scripts/apiTimeouts.ts b/src/scripts/apiTimeouts.ts new file mode 100644 index 00000000000..3c03c85e4bd --- /dev/null +++ b/src/scripts/apiTimeouts.ts @@ -0,0 +1,22 @@ +/** + * The request bounds `api.fetchApi` enforces, in their own leaf module so that + * a caller which must OUTLAST a request can derive its bound from the same + * numbers instead of restating them. + * + * Deliberately not exported from `api.ts`: `src/scripts/__mocks__/api.ts` is + * the manual mock every `vi.mock('@/scripts/api')` resolves to, and it exports + * only `api` — so a constant read through that path arrives `undefined` and any + * arithmetic on it silently becomes `NaN`. A `setTimeout` armed with `NaN` + * fires immediately, which is exactly the failure a derived timeout must not + * have. Importing from here keeps the real values under a mocked `api`. + */ + +/** Ceiling on `ComfyApi.waitForAuthInitialization`, spent before the response timer below is armed. */ +export const AUTH_INIT_TIMEOUT_MS = 10_000 + +/** + * Response-HEADERS timeout for one `fetchApi` attempt — not the body. A 401 + * remint clears this timer and arms a fresh one for the retry, so a single + * cloud request can spend it twice (`fetchWithUnifiedRemint`). + */ +export const FETCH_RESPONSE_HEADERS_TIMEOUT_MS = 60_000 diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts index adb90b660ea..0c81279063c 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.test.ts @@ -37,6 +37,7 @@ import type { PostMessageInput } from '../../services/agent/agentRestClient' import { useAgentConversationStore } from '../../stores/agent/agentConversationStore' +import { useAgentSendGateStore } from '../../stores/agent/agentSendGateStore' import { useAgentWorkflowTabBindingStore } from '../../stores/agent/agentWorkflowTabBindingStore' import type { SelectedNode } from './useCanvasSelection' @@ -5367,6 +5368,50 @@ describe('useAgentSession (v1 composition root)', () => { ]) expect(session.notices.value).toHaveLength(0) }) + + // PM-1660: the gate has to be taken in sendMessage, before performSend + // awaits prepareWorkflow(), and released however the POST settles. Without + // these two cases the wiring can be removed or moved into the success branch + // with every agent unit suite still green — only the @cloud spec notices, + // and only on the success path. + it.for([ + ['resolves', (resolve: () => void) => resolve()], + ['rejects', (_resolve: () => void, reject: () => void) => reject()] + ] as const)( + 'holds the send gate from send until the POST %s, across New Chat', + async ([, settlePost]) => { + let settle: () => void = () => {} + const postMessage = vi.fn( + () => + new Promise((resolve, reject) => { + settle = () => + settlePost( + () => + resolve({ + thread_id: 'th-1', + message_id: 'msg-1', + workflow_id: 'wf-1' + }), + () => reject(new Error('post failed')) + ) + }) + ) + const rest = fakeRest({ postMessage }) + const session = useAgentSession({ rest, events: fakeEvents().source }) + session.start() + const gate = useAgentSendGateStore() + + const sent = session.sendMessage('go') + await vi.waitFor(() => expect(postMessage).toHaveBeenCalled()) + session.newChat('new_chat_button') + + expect(gate.isSending).toBe(true) + settle() + await sent + + expect(gate.isSending).toBe(false) + } + ) }) describe('thread resume (B17)', () => { diff --git a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts index 1130fd5956b..4c9e45b5390 100644 --- a/src/workbench/extensions/agent/composables/agent/useAgentSession.ts +++ b/src/workbench/extensions/agent/composables/agent/useAgentSession.ts @@ -47,6 +47,7 @@ import type { AssistantMessage } from '../../services/agent/agentMessageParts' import { normalizeAgentTranscript } from '../../services/agent/agentTranscript' import type { LiveTurn } from '../../stores/agent/agentConversationStore' import { useAgentConversationStore } from '../../stores/agent/agentConversationStore' +import { useAgentSendGateStore } from '../../stores/agent/agentSendGateStore' import { useAgentWorkflowTabBindingStore } from '../../stores/agent/agentWorkflowTabBindingStore' import type { WorkflowReference } from '../../types/workflowReference' import { serializeWorkflowReferences } from '../../utils/workflowReferenceText' @@ -374,6 +375,7 @@ export function useAgentSession(deps: AgentSessionDeps) { clearLegacyAgentStorage() const conversationStore = useAgentConversationStore() + const sendGateStore = useAgentSendGateStore() const bindingStore = useAgentWorkflowTabBindingStore() /** * The workflow the session is bound to (set on turn ack or an active-tab @@ -1152,6 +1154,7 @@ export function useAgentSession(deps: AgentSessionDeps) { } promptEditState.value = { phase: 'idle' } sending.value = true + const releaseSendGate = sendGateStore.begin() sendInFlight = true stopPendingAck = null try { @@ -1165,6 +1168,7 @@ export function useAgentSession(deps: AgentSessionDeps) { ) } finally { sending.value = false + releaseSendGate() sendInFlight = false } } diff --git a/src/workbench/extensions/agent/stores/agent/agentRunModeStore.test.ts b/src/workbench/extensions/agent/stores/agent/agentRunModeStore.test.ts index a85228bbe00..aeb521d45b1 100644 --- a/src/workbench/extensions/agent/stores/agent/agentRunModeStore.test.ts +++ b/src/workbench/extensions/agent/stores/agent/agentRunModeStore.test.ts @@ -1,10 +1,14 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' +import { nextTick } from 'vue' import { api } from '@/scripts/api' +import { createMockLoadedWorkflow } from '@/utils/__tests__/litegraphTestUtils' vi.mock(import('@/scripts/api')) +import { useAgentComposerStore } from './agentComposerStore' import { useAgentRunModeStore } from './agentRunModeStore' +import { useAgentSendGateStore } from './agentSendGateStore' function jsonResponse(status: number, body: unknown): Response { return new Response(JSON.stringify(body), { @@ -272,6 +276,143 @@ describe('agentRunModeStore', () => { } ) + // PM-1660: the composer clears on the send click but the POST waits on the + // cloud-workflow refresh, so a mode saved in that window used to reach the + // server first and re-authorize the turn the user had already sent. + it('holds a mode change back until the in-flight send settles', async () => { + localStorage.setItem( + 'Comfy.Agent.RunModePreference', + JSON.stringify({ mode: 'auto', credit_limit: null }) + ) + vi.mocked(api.fetchApi).mockResolvedValue( + jsonResponse(200, { mode: 'ask_approval', credit_limit: null }) + ) + const sendGate = useAgentSendGateStore() + const releaseSend = sendGate.begin() + const store = useAgentRunModeStore() + + const save = store.save('ask_approval', null) + await nextTick() + + expect(vi.mocked(api.fetchApi)).not.toHaveBeenCalled() + expect(store.mode).toBe('auto') + + releaseSend() + await save + + expect(vi.mocked(api.fetchApi)).toHaveBeenCalledWith( + '/agent/run-mode', + expect.objectContaining({ method: 'PUT' }) + ) + expect(store.mode).toBe('ask_approval') + }) + + // Abandoning the VIEW does not abandon the turn: newChat/loadThread clear the + // composer's submission but stash the turn, which keeps running and keeps + // spending. Keying the wait on the draft let a later mode change overtake it. + it('keeps holding a mode change after the composer submission is discarded', async () => { + vi.mocked(api.fetchApi).mockResolvedValue( + jsonResponse(200, { mode: 'auto', credit_limit: null }) + ) + const composer = useAgentComposerStore() + composer.startSubmission({ + prompt: composer.prompt, + attachments: [], + nodes: [], + target: createMockLoadedWorkflow({ path: 'workflows/target.json' }) + }) + const sendGate = useAgentSendGateStore() + const releaseSend = sendGate.begin() + const store = useAgentRunModeStore() + + composer.invalidateSubmission() + const save = store.save('auto', null) + await nextTick() + + expect(vi.mocked(api.fetchApi)).not.toHaveBeenCalled() + + releaseSend() + await save + + expect(vi.mocked(api.fetchApi)).toHaveBeenCalledOnce() + }) + + // A send is only bounded as far as its response HEADERS, so a stalled body + // would hold the gate for the page's lifetime. The gate's own backstop is + // the only thing that ends such a hold, so this is what keeps the control + // usable without a reload. The assertion on isSending comes BEFORE the + // await: a missing backstop then fails here rather than by test timeout. + it('recovers the run-mode control from a send that never settles', async () => { + vi.mocked(api.fetchApi).mockResolvedValue( + jsonResponse(200, { mode: 'ask_approval', credit_limit: null }) + ) + const sendGate = useAgentSendGateStore() + sendGate.begin() + const store = useAgentRunModeStore() + + const save = store.save('ask_approval', null) + await vi.runAllTimersAsync() + + expect(sendGate.isSending).toBe(false) + await save + expect(store.mode).toBe('ask_approval') + }) + + // A save that parks on the gate does not bump saveRevision again while it + // waits, so load()'s revision check alone still matches when the PUT lands + // first — and a GET taken before that write would put the old mode back + // while the server enforces the new one. The hold widens that window from + // one PUT round trip to the length of a send. + it('does not let a load started during the hold undo the saved mode', async () => { + let resolveGet!: (response: Response) => void + vi.mocked(api.fetchApi).mockImplementation((_route, init) => { + if (init?.method === 'GET') + return new Promise((resolve) => { + resolveGet = resolve + }) + return Promise.resolve(jsonResponse(200, JSON.parse(String(init?.body)))) + }) + const releaseSend = useAgentSendGateStore().begin() + const store = useAgentRunModeStore() + + const save = store.save('auto_limited', 40) + const load = store.load() + await nextTick() + releaseSend() + await save + + resolveGet(jsonResponse(200, { mode: 'ask_approval', credit_limit: null })) + await load + + expect(store.mode).toBe('auto_limited') + expect(store.creditLimit).toBe(40) + }) + + // Two picks can park on one hold (two popover instances, or one remounted + // mid-write) and both wake on the same release, so without the superseded- + // revision check both PUTs go out and the server keeps whichever landed + // last — which is not necessarily the mode the user picked last. + it('sends only the latest of two picks parked on one send', async () => { + vi.mocked(api.fetchApi).mockImplementation(async (_route, init) => + jsonResponse(200, JSON.parse(String(init?.body))) + ) + const releaseSend = useAgentSendGateStore().begin() + const store = useAgentRunModeStore() + + const first = store.save('auto', null) + const second = store.save('auto_limited', 40) + await nextTick() + releaseSend() + await Promise.all([first, second]) + + expect( + vi + .mocked(api.fetchApi) + .mock.calls.map(([, init]) => JSON.parse(String(init?.body))) + ).toEqual([{ mode: 'auto_limited', credit_limit: 40 }]) + expect(store.mode).toBe('auto_limited') + }) + it('surfaces non-404 failures without changing the saved preference', async () => { vi.mocked(api.fetchApi).mockResolvedValueOnce( jsonResponse(500, { error: 'failed' }) diff --git a/src/workbench/extensions/agent/stores/agent/agentRunModeStore.ts b/src/workbench/extensions/agent/stores/agent/agentRunModeStore.ts index ebf464d9683..18256e89371 100644 --- a/src/workbench/extensions/agent/stores/agent/agentRunModeStore.ts +++ b/src/workbench/extensions/agent/stores/agent/agentRunModeStore.ts @@ -1,4 +1,4 @@ -import { useLocalStorage } from '@vueuse/core' +import { until, useLocalStorage } from '@vueuse/core' import { defineStore } from 'pinia' import { computed } from 'vue' @@ -12,6 +12,7 @@ import { AgentApiError, createAgentRestClient } from '../../services/agent/agentRestClient' +import { useAgentSendGateStore } from './agentSendGateStore' const DEFAULT_PREFERENCE: AgentRunModePreference = { mode: 'ask_approval', @@ -103,9 +104,18 @@ export const useAgentRunModeStore = defineStore('agentRunMode', () => { async function load(): Promise { const revision = saveRevision + const appliedRevision = appliedSaveRevision try { const serverPreference = await api.getRunMode() - if (revision === saveRevision) apply(serverPreference) + // BOTH counters. A save that starts before this load and then parks on + // the send gate does not bump saveRevision again while it waits, so the + // revision check alone still matches when its PUT lands first — and + // this GET, taken before that write, would put the old mode back while + // the server enforces the new one. The gate widens that window from one + // PUT round trip to the length of a send, which is what makes it worth + // closing here. + if (revision === saveRevision && appliedRevision === appliedSaveRevision) + apply(serverPreference) } catch (error) { if (!(error instanceof AgentApiError && error.status === 404)) throw error localPreference() @@ -126,7 +136,29 @@ export const useAgentRunModeStore = defineStore('agentRunMode', () => { appliedSaveRevision = revision apply(savedPreference) } + // The picked mode is deliberately NOT applied here: this control shows + // what is SAVED, with the popover's own spinner covering the write (see + // Composer.test.ts, 'blocks a second pick while the write is in flight'). + // Waiting for a send in flight widens that window; it does not change it. try { + // The server pins a turn's run mode when that turn's POST ARRIVES, not + // when the user pressed send, so a mode written while the composer + // already looks sent overtakes the message and re-authorizes it + // (PM-1660). Bounded by the gate's own MAX_HOLD_MS and nothing else: a + // second timeout here could only fire while overlapping sends were + // legitimately holding the gate, and firing meant rejecting the user's + // change behind an error toast with the mode left as it was. + // + // The `isSending` check is load-bearing, not an optimisation: awaiting + // unconditionally adds a microtask, and that is enough for a + // back-to-back second save to supersede this one before it reaches the + // revision check below, so the first never sends its PUT. + const sendGate = useAgentSendGateStore() + if (sendGate.isSending) await until(() => sendGate.isSending).toBe(false) + // A second pick can park on the same gate (two popover instances, or + // one remounted mid-write) and both wake on the same release, so their + // PUTs would race and the server would keep whichever landed last. + if (revision !== saveRevision) return applySaved(await api.putRunMode(next)) } catch (error) { if (!(error instanceof AgentApiError && error.status === 404)) throw error diff --git a/src/workbench/extensions/agent/stores/agent/agentSendGateStore.test.ts b/src/workbench/extensions/agent/stores/agent/agentSendGateStore.test.ts new file mode 100644 index 00000000000..5e3a4870150 --- /dev/null +++ b/src/workbench/extensions/agent/stores/agent/agentSendGateStore.test.ts @@ -0,0 +1,32 @@ +import { describe, expect, it, vi } from 'vitest' + +import { MAX_HOLD_MS, useAgentSendGateStore } from './agentSendGateStore' + +// There is deliberately no test asserting MAX_HOLD_MS exceeds the request +// window it has to outlast. It is derived from the same api.ts timeouts the +// window is built from, so raising one of those raises both sides and the +// assertion cannot fail; the only terms left are literals it would compare +// against literals. The derivation is the guard — see agentSendGateStore.ts. +describe('agentSendGateStore', () => { + it('counts overlapping holds and releases each one once', () => { + const gate = useAgentSendGateStore() + const first = gate.begin() + gate.begin() + + first() + first() + + expect(gate.isSending).toBe(true) + }) + + it('releases a hold that never settles, and not before the backstop', async () => { + const gate = useAgentSendGateStore() + gate.begin() + + await vi.advanceTimersByTimeAsync(MAX_HOLD_MS - 1) + expect(gate.isSending).toBe(true) + + await vi.advanceTimersByTimeAsync(1) + expect(gate.isSending).toBe(false) + }) +}) diff --git a/src/workbench/extensions/agent/stores/agent/agentSendGateStore.ts b/src/workbench/extensions/agent/stores/agent/agentSendGateStore.ts new file mode 100644 index 00000000000..89f3637f595 --- /dev/null +++ b/src/workbench/extensions/agent/stores/agent/agentSendGateStore.ts @@ -0,0 +1,110 @@ +import { defineStore } from 'pinia' +import { computed, ref } from 'vue' + +import { + AUTH_INIT_TIMEOUT_MS, + FETCH_RESPONSE_HEADERS_TIMEOUT_MS +} from '@/scripts/apiTimeouts' + +/** + * A 401 remint, which runs BETWEEN the two response-headers timers: `api.ts` + * clears the first one before re-minting and arms the second after. Budgeted + * rather than imported — the 15s ceiling is `DEFAULT_MINT_TIMEOUT_MS`, private + * to `packages/account-core/src/core/session.ts`, and the unified-user wait in + * front of it has no ceiling at all. + */ +const REMINT_BUDGET_MS = 30_000 + +/** + * Everything the hold covers before the first response timer is armed: + * `prepareWorkflow()`'s own 3s race, plus `getAuthHeader()` and + * `shouldRemintCloudRequest()`'s dynamic import, neither of which has a + * ceiling. A budget, not a bound — making it provable needs an overall + * deadline on the request or a release tied to its dispatch. + */ +const SEND_PRELUDE_BUDGET_MS = 30_000 + +/** + * Deliberately longer than every path that can still deliver the POST, so the + * backstop cannot release the gate while the message is still on its way and + * let a run-mode write overtake it. + * + * DERIVED rather than transcribed, because a hand-summed 150s here sat BELOW + * its own window: 3s of prepare + 10s of auth init + 60s + a 15s remint + a + * second 60s is 148s, and a slower remint makes it 163s, so a retried POST + * could land after the backstop had already let a parked PUT through — the + * PM-1660 ordering again. + * + * Past this the only unbounded case left is a stalled response BODY, and + * headers having arrived means the server already ran `resolveRunMode`: cloud + * `services/agent/server/agent_handler.go` pins the turn's mode inside + * `postMessage` before it writes `202 Accepted`. The ordering this gate + * protects is settled by then, so releasing is safe. + */ +export const MAX_HOLD_MS = + AUTH_INIT_TIMEOUT_MS + + 2 * FETCH_RESPONSE_HEADERS_TIMEOUT_MS + + REMINT_BUDGET_MS + + SEND_PRELUDE_BUDGET_MS + +/** + * Counts the messages currently on their way to the server: held from entry to + * useAgentSession.sendMessage until the POST that carries the message has + * settled. + * + * NOT from the send click, and the gap is not nothing. + * useAgentDraftSubmission.submit clears the draft before it calls `send`, so + * anything AgentPanelRoot awaits in between runs with the composer already + * looking sent and no hold taken — on main that is the first-message consent + * check and the coach tour it waits on. A mode picked inside that window still + * precedes the POST. It is one consent round trip on an account's first + * message; closing it means taking the hold at startSubmission and passing the + * release through. + * + * It exists because that window is not visible anywhere else. The composer's + * submission looks like it, but New Chat and switching threads both clear it + * (agentComposerStore.invalidateSubmission) while the POST is still in flight — + * and neither cancels the turn. The POST keeps running server-side and keeps + * spending; what is discarded is the client's view of it, since + * stashActiveTurn() has no slot to stash before the ack and performSend's + * generation check drops the late one. Anything that must not overtake a sent + * message has to track the request, not the draft it came from (PM-1660). + * + * It orders writes from THIS page only. The server resolves run mode per + * user, so a second tab's write is ordered against its own sends and can + * still overtake this one's — closing that needs the ordering point to move + * to the server, which is where the mode is pinned. + * + * And a hold ends on ANY settle, including a client-side abort: a headers + * timeout or a network error releases it, which does not prove the server + * discarded a request it may already have queued. Unresolved deliberately — + * it turns on whether cloud cancels on client disconnect, which needs an + * experiment against the service rather than a reading of this file, and + * holding on every ambiguous failure would change behaviour for every failed + * send. + */ +export const useAgentSendGateStore = defineStore('agentSendGate', () => { + const inFlight = ref(0) + + /** + * Marks one message as on its way and returns ITS release. The release is + * single-use, so a caller cannot decrement a hold it does not own, and it + * fires on its own after MAX_HOLD_MS, so a send that never settles cannot + * strand the gate. Both matter: `isSending` reading false during a live send + * silently reopens the ordering hole this store exists to close. + */ + function begin(): () => void { + inFlight.value += 1 + let released = false + const release = (): void => { + if (released) return + released = true + clearTimeout(backstop) + inFlight.value = Math.max(0, inFlight.value - 1) + } + const backstop = setTimeout(release, MAX_HOLD_MS) + return release + } + + return { isSending: computed(() => inFlight.value > 0), begin } +})