Repository navigation
[Backport core/1.55] Fix agent sender retry accounting and transport recovery - #19590
Conversation
… and the new-chat fixture (#18538) ## Summary Follow-up to #18064, narrowed after #18700. Fixes four defects in the agent op sender's late-result accounting and transport handling, makes the new-chat e2e fixture fail on an acknowledgement it could previously swallow, and adds an e2e that proves a human edit survives a transport throw. Each sender fix landed as a red test first. ## Changes - **What**: - Late-result credits are reserved per send that actually left the client, not per resend attempted. `batch.resent` was set before `transmit()`, so a batch retired at that resend reserved two credits for one transmission and the surplus swallowed the next batch's own result. - A credit is retired only on the aborted batch's own identified result. The sender remembers the op ids of every retired transmitted batch; a result naming ops it never minted no longer spends the credit. - An acknowledged batch reserves a credit for a send left unanswered. A batch resent after silence has two sends on the wire, so settling it on the first answer previously left the second to settle its successor. - A throwing transport is a failed send, not a stalled sender. `pump()` installs the in-flight record before `sendOps`, so a synchronous throw left the slot occupied with no timer armed and every later batch queued forever. The send now runs inside the sender's own boundary, is reported once per delivery as `failure_sending_agent_human_ops`, and is retried within the existing budget. - The new-chat fixture records a failed turn acknowledgement instead of skewing the thread list. `recordAccepted()` ran unawaited from `page.on('response')`; the handlers are now filtered to turn acks, drained before the thread list is served and before the teardown assertion, and an unreadable body is reported separately from a malformed one. - An e2e covers the throwing transport end to end: the first outbound `doc_ops` frame throws, and a renamed node survives the retry and a tab switch. The fault lives in `fixtures/utils/throwOnFirstDocOpsSend.ts` and records itself on the document, so the spec asserts the injection fired rather than passing as a plain rename test. ## Review Focus - `onOpsResult` is restructured rather than patched. It captures the addressed batch, then picks between the in-flight batch's own identified result, a retired batch's identified result (which drains its credit), and an anonymous failure (the in-flight batch's only once no credit could explain it). - **Known cost, deliberately kept.** A credit reserved for a result that never arrives is never drained, so the next batch's own answer pays for it and reserves another on its resend: once one result is lost, every later anonymous failure costs a `RESULT_TIMEOUT_MS` resend. Charging the swallow back to the batch was tried and reverted — the sender cannot tell whether an anonymous result is the retired batch's or the in-flight batch's, and charging it as both under-reserves and settles a successor on its predecessor's failure, reverting edits the host never rejected. Closing it needs a credit lifetime the result contract does not define, or a per-send id `DocFrameClient.sendOps` does not mint and `DocOpsResult` does not echo. `never settles a batch with a result whose ops it does not own` pins the invariant meanwhile. - `staleAnonymousBudget` and `retiredOpIds` model the same retired sends with separate lifetimes, and the set only sheds entries when the budget reaches zero. Per-batch `{ opIds, credits }` records would fix that and the cost above in one change; deliberately left as a follow-up. ## Verification - Every `test:` commit fails against its parent. Against current `main`'s `opSender.ts`, **nine** of this branch's sender tests fail and 40 pass. - Mutation-checked: removing the identified-branch reservation, removing the resend's re-arm of the throw report, adding a re-arm to `resume()`, and changing the telemetry slug each turn exactly one test red. - `typecheck`, `typecheck:browser`, `lint`, `format:check`, `knip` and `fallow audit` clean. - Both agent specs pass in the `cloud` project against a locally built cloud bundle on a real backend. Two negative controls: forcing the new-chat acknowledgement to a malformed 200 fails through the fixture's teardown guard as `body unreadable`, and breaking the `doc_ops` frame match fails the transport-throw spec on its injection assertion. - The new-chat fixture's failure branches are not exercised by any committed spec — its only consumer always gets a valid 2xx. They were checked by hand; a table-driven test for them needs `browser_tests/**` inside Vitest or the helper extracted, so it is worth its own change. --------- Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎨 Storybook: ✅ Built — View Storybook🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (9 changed files).
Pre-existing findings are baselined in Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since f692a7272fcab0d18e8e61beed7e55eb1d6c0211
Full docs: ADR-DEVEX-LINT-0015 🎭 Playwright: ✅ 1858 passed, 0 failed📊 Browser Reports
🧩 Custom Nodes Ecosystem Matrix: ✅ PassedExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight
On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff. Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md |
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @christian-byrne.
Found 10 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 1 |
| 🟡 Medium | 4 |
| 🟢 Low | 4 |
| ⚪ Nit | 1 |
Panel: 8/8 reviewers contributed findings.
| return | ||
| } | ||
| settle({ state: 'acknowledged', ops: inFlight.ops, result }) | ||
| retire(batch, 1) |
There was a problem hiding this comment.
🟠 High — The anonymous-failure path settles the in-flight batch acknowledged without checking that the batch was ever transmitted, and retire(batch, 1) silently no-ops when sends === 0 (outstanding is -1). A batch parked by suspend() or sitting in the new 500 ms trySend retry window therefore gets settled by a stray result belonging to someone else, and settleHumanOps then runs revertRejectedOps with an empty applied set, reverting an edit that never left the client. The new sends counter makes the guard cheap: only attribute an anonymous result when batch.sends > 0, otherwise keep retrying. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max edge-case, kimi-k3-max adversarial).
| function drainStaleCredit(): void { | ||
| if (staleAnonymousBudget === 0) return | ||
| staleAnonymousBudget-- | ||
| if (staleAnonymousBudget === 0) retiredOpIds.clear() |
There was a problem hiding this comment.
🟡 Medium — retiredOpIds is emptied only when staleAnonymousBudget happens to hit exactly 0, and neither the set nor the counter has a cap, eviction, or time bound. A single credit that is never drained (a permanently lost result, or a host that stops emitting doc_ops_result entirely) pins the budget above zero for the sender's lifetime, after which every retired or resent batch adds up to MAX_DOC_OPS_PER_FRAME more op ids that are never removed — unbounded growth for as long as the agent panel stays mounted. Track credits and ids per batch, or bound/expire them, rather than clearing on a global zero. Raised by 6 of 8 reviewers (claude-opus-5-thinking-max adversarial, gpt-5.6-sol-max adversarial, gpt-5.6-sol-max edge-case, gemini-3.1-pro adversarial, kimi-k3-max adversarial, kimi-k3-max edge-case).
| if (staleAnonymousBudget > 0) staleAnonymousBudget-- | ||
| return | ||
| if (batch && identified.some((opId) => batch.opIds.has(opId))) { | ||
| retire(batch, 1) |
There was a problem hiding this comment.
🟡 Medium — This newly reserves a late-result credit on the acknowledged path whenever sends === 2, but the most common reason a resend happened at all is that the first result was genuinely lost, so that credit can never be drained. The budget then stays at or above 1 for the session: each later batch has its first anonymous failure swallowed at line 339, burns a full RESULT_TIMEOUT_MS resend, and reserves a fresh credit when it settles — the human write path degrades to roughly one batch per 10 s with a growing queue and no recovery short of a reload. Expiring credits by age or scoping them to the batch would bound the cost the in-code comment currently just accepts. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).
| // no batch waiting, one addressed to another workflow, or one a stale | ||
| // credit could explain drains that credit so it cannot swallow a future | ||
| // batch's own result. Only then is it the in-flight batch's. | ||
| if (batch === null || staleAnonymousBudget > 0) { |
There was a problem hiding this comment.
🟡 Medium — Credits are global even though results carry a workflowId. A credit reserved for a retired batch on workflow A swallows an anonymous failure whose workflowId matches the current in-flight batch on workflow B, even though the workflow id proves the result cannot be A's — costing B an unnecessary 10 s resend and carrying the phantom credit forward. Key credits by workflowId so a result that provably cannot belong to the retired batch is not spent on it. Raised by 2 of 8 reviewers (gpt-5.6-sol-max adversarial, gpt-5.6-sol-max edge-case).
| if (batch && identified.some((opId) => batch.opIds.has(opId))) { | ||
| retire(batch, 1) | ||
| settle({ state: 'acknowledged', ops: batch.ops, result }) | ||
| } else if (identified.some((opId) => retiredOpIds.has(opId))) { |
There was a problem hiding this comment.
🟡 Medium — A late identified rejection (ok:false with failed.op_id) for a batch that already settled unacknowledged takes this branch and only drains a credit, so onBatchSettled never sees it and revertRejectedOps never runs — the host rejected the write but the edit stays in the local graph permanently. The same information is lost when a batch with two sends settles on the first answer and the second, contradictory answer arrives afterwards. Now that retired op ids are tracked, this branch has enough context to reconcile the rejection instead of discarding it. Raised by 1 of 8 reviewers (gpt-5.6-sol-max edge-case).
| failure_kind: 'caught_unexpected', | ||
| feature_area: 'agent', | ||
| operation: 'sync', | ||
| outcome: 'recovered' |
There was a problem hiding this comment.
🟢 Low — outcome: 'recovered' is hard-coded at throw time, before anything is known about recovery. A transport that keeps throwing past SEND_RETRY_LIMIT reaches settleUnbound with sends === 0 and settles undeliverable, dropping the user's ops, yet the only telemetry for that failure reports it as recovered. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
| try { | ||
| return deps.sendOps(batch.workflowId, deps.tab, batch.ops) | ||
| } catch (error) { | ||
| if (!batch.reportedThrow) { |
There was a problem hiding this comment.
🟢 Low — reportedThrow dedupes only within one delivery and is deliberately re-armed for the resend, so a persistently throwing transport (say a payload shape it cannot serialize) emits one or two reportError calls per batch — each fanning out to Sentry, Datadog, and the console — for every user edit, with no session-level cap. Sibling code in this module dedupes for the client's lifetime (reportedTypes in docFrameClient.ts); a sender-scoped flag would match. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).
| } | ||
|
|
||
| const nativeSend = WebSocket.prototype.send | ||
| let throwsLeft = 1 |
There was a problem hiding this comment.
🟢 Low — addInitScript re-executes in every new document and every same-origin frame, and throwsLeft lives in that per-document closure, so a reload, popup, or child frame re-arms the fault. The helper therefore injects "first doc_ops send per document", not the per-test contract the doc comment promises; hoisting the counter onto a page-level object shared across documents would make the contract hold. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
| void record.finally(() => this.recording.delete(record)) | ||
| }) | ||
| await this.page.route('**/api/agent/threads', async (route) => { | ||
| await this.settled().catch((error: unknown) => { |
There was a problem hiding this comment.
🟢 Low — settled() only awaits ack recordings whose response event has already fired, so a GET /api/agent/threads that races a POST .../messages still in flight sees an empty recording set, returns immediately, and serves a list missing that thread. Closing the race needs posts accepted in answerPost to be tracked until their response has been recorded. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
| INJECTED_DOC_OPS_THROWS_ATTR, | ||
| '1' | ||
| ) | ||
| expect( |
There was a problem hiding this comment.
⚪ Nit — This is a non-retrying assertion on a session-global doc_ops frame count, unlike the sibling agentHumanRenameMintsSetNodeField.spec.ts, which baselines with opsBeforeRename. Any frame the replay emits before the rename would consume the single injected throw (so the rename never exercises the retry path the spec is named for) and break the count. Capturing the frame count before renameNode and asserting the delta pins the fault to the rename. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
Summary
Manual
core/1.55backport of #18538. The automatic carrier was not created because the release label was added after the source PR merged.This fixes late-result accounting and synchronous transport-throw recovery in the agent op sender, hardens the new-chat fixture, and adds an end-to-end recovery scenario.
Validation
opSender.test.ts: 49 tests passed