Repository navigation
[Backport cloud/1.55] Fix agent sender retry accounting and transport recovery - #19591
Conversation
… and the new-chat fixture (#18538) ## Summary Follow-up to #18064, narrowed after #18700. Fixes four defects in the agent op sender's late-result accounting and transport handling, makes the new-chat e2e fixture fail on an acknowledgement it could previously swallow, and adds an e2e that proves a human edit survives a transport throw. Each sender fix landed as a red test first. ## Changes - **What**: - Late-result credits are reserved per send that actually left the client, not per resend attempted. `batch.resent` was set before `transmit()`, so a batch retired at that resend reserved two credits for one transmission and the surplus swallowed the next batch's own result. - A credit is retired only on the aborted batch's own identified result. The sender remembers the op ids of every retired transmitted batch; a result naming ops it never minted no longer spends the credit. - An acknowledged batch reserves a credit for a send left unanswered. A batch resent after silence has two sends on the wire, so settling it on the first answer previously left the second to settle its successor. - A throwing transport is a failed send, not a stalled sender. `pump()` installs the in-flight record before `sendOps`, so a synchronous throw left the slot occupied with no timer armed and every later batch queued forever. The send now runs inside the sender's own boundary, is reported once per delivery as `failure_sending_agent_human_ops`, and is retried within the existing budget. - The new-chat fixture records a failed turn acknowledgement instead of skewing the thread list. `recordAccepted()` ran unawaited from `page.on('response')`; the handlers are now filtered to turn acks, drained before the thread list is served and before the teardown assertion, and an unreadable body is reported separately from a malformed one. - An e2e covers the throwing transport end to end: the first outbound `doc_ops` frame throws, and a renamed node survives the retry and a tab switch. The fault lives in `fixtures/utils/throwOnFirstDocOpsSend.ts` and records itself on the document, so the spec asserts the injection fired rather than passing as a plain rename test. ## Review Focus - `onOpsResult` is restructured rather than patched. It captures the addressed batch, then picks between the in-flight batch's own identified result, a retired batch's identified result (which drains its credit), and an anonymous failure (the in-flight batch's only once no credit could explain it). - **Known cost, deliberately kept.** A credit reserved for a result that never arrives is never drained, so the next batch's own answer pays for it and reserves another on its resend: once one result is lost, every later anonymous failure costs a `RESULT_TIMEOUT_MS` resend. Charging the swallow back to the batch was tried and reverted — the sender cannot tell whether an anonymous result is the retired batch's or the in-flight batch's, and charging it as both under-reserves and settles a successor on its predecessor's failure, reverting edits the host never rejected. Closing it needs a credit lifetime the result contract does not define, or a per-send id `DocFrameClient.sendOps` does not mint and `DocOpsResult` does not echo. `never settles a batch with a result whose ops it does not own` pins the invariant meanwhile. - `staleAnonymousBudget` and `retiredOpIds` model the same retired sends with separate lifetimes, and the set only sheds entries when the budget reaches zero. Per-batch `{ opIds, credits }` records would fix that and the cost above in one change; deliberately left as a follow-up. ## Verification - Every `test:` commit fails against its parent. Against current `main`'s `opSender.ts`, **nine** of this branch's sender tests fail and 40 pass. - Mutation-checked: removing the identified-branch reservation, removing the resend's re-arm of the throw report, adding a re-arm to `resume()`, and changing the telemetry slug each turn exactly one test red. - `typecheck`, `typecheck:browser`, `lint`, `format:check`, `knip` and `fallow audit` clean. - Both agent specs pass in the `cloud` project against a locally built cloud bundle on a real backend. Two negative controls: forcing the new-chat acknowledgement to a malformed 200 fails through the fixture's teardown guard as `body unreadable`, and breaking the `doc_ops` frame match fails the transport-throw spec on its injection assertion. - The new-chat fixture's failure branches are not exercised by any committed spec — its only consumer always gets a valid 2xx. They were checked by hand; a table-driven test for them needs `browser_tests/**` inside Vitest or the helper extracted, so it is worth its own change. --------- Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎭 Playwright: ✅ 2121 passed, 0 failed · 1 flaky📊 Browser Reports
🎨 Storybook: ✅ Built — View Storybook🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (9 changed files).
Pre-existing findings are baselined in Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 32838ae557648bb9d2066c4a376889df7dd9f665
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: ✅ PassedExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight
On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff. Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## cloud/1.55 #19591 +/- ##
=============================================
Coverage ? 80.55%
=============================================
Files ? 2210
Lines ? 117125
Branches ? 35455
=============================================
Hits ? 94347
Misses ? 22391
Partials ? 387
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @christian-byrne.
Found 7 finding(s).
| Severity | Count |
|---|---|
| 🟠 High | 2 |
| 🟡 Medium | 2 |
| 🟢 Low | 2 |
| ⚪ Nit | 1 |
Panel: 8/8 reviewers contributed findings.
| return | ||
| } | ||
| settle({ state: 'acknowledged', ops: inFlight.ops, result }) | ||
| retire(batch, 1) |
There was a problem hiding this comment.
🟠 High — When an anonymous failure (empty applied/skipped, no failed op_id) arrives while the in-flight batch has sends === 0 — parked by suspend(), or still waiting on a send-retry timer after a refused/throwing send — retire(batch, 1) computes outstanding = -1 and no-ops, then the batch is settled acknowledged, telling downstream code the host applied ops that never left the client. The suspension tests construct exactly this state via parkSecondBatch, and settle() pumps the next queued batch into the same parked slot, so a run of anonymous failures during a suspension can silently discard the whole queue. Guard this path with batch.sends > 0 (ignore the frame, or settle undeliverable). Raised by 3 of 8 reviewers (claude-opus-5-thinking-max edge-case, gemini-3.1-pro edge-case, kimi-k3-max adversarial).
| function retire(batch: InFlight, answered: number): void { | ||
| const outstanding = batch.sends - answered | ||
| if (outstanding <= 0) return | ||
| staleAnonymousBudget += outstanding |
There was a problem hiding this comment.
🟠 High — retire(batch, 0) on the unacknowledged timeout (line 251) and in settleUnbound (line 239) reserves credits for answers that by definition never arrive, and nothing caps, ages, or resets them — so staleAnonymousBudget never returns to 0, retiredOpIds.clear() never runs, and the set keeps accruing op ids from every later retire (unbounded growth keyed to user edit volume for the life of the session). Worse, a nonzero budget is self-sustaining: each new batch's own anonymous answer is swallowed, it resends, the next answer is swallowed, it settles unacknowledged and re-adds credits — so revertRejectedOps never runs and host-rejected ops stay silently diverged on the canvas while every batch costs a RESULT_TIMEOUT_MS stall. Expiring credits with the batch they were reserved for, or bounding the budget, would make the state recoverable. Raised by 5 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case, gpt-5.6-sol-max adversarial, gemini-3.1-pro adversarial, kimi-k3-max adversarial).
| function drainStaleCredit(): void { | ||
| if (staleAnonymousBudget === 0) return | ||
| staleAnonymousBudget-- | ||
| if (staleAnonymousBudget === 0) retiredOpIds.clear() |
There was a problem hiding this comment.
🟡 Medium — The ledger is one global counter plus one flat id set, so a credit is consumed by whichever result arrives next rather than by the batch it was reserved for: an anonymous answer for the in-flight batch drains retired batch A's credit, and drainStaleCredit() then clears the set wholesale (dropping retired batch B's ids too), after which A's or B's genuine late anonymous result is attributed to and settles the current batch with someone else's result — the exact mis-settle this scheme exists to prevent. Credits are also not scoped by workflowId, so a stale credit minted before a workflow switch is spent on the new workflow's answer and the resend recreates a phantom credit. A per-batch map from op id to its reserved credit would make the accounting exact. Raised by 3 of 8 reviewers (gpt-5.6-sol-max edge-case, gpt-5.6-sol-max adversarial, claude-opus-5-thinking-max edge-case).
| level: 'error' | ||
| }) | ||
| } | ||
| return false |
There was a problem hiding this comment.
🟡 Medium — trySend catches every exception and treats it as a transient refusal, so a deterministic failure — e.g. JSON.stringify in DocFrameClient.send throwing on a BigInt or circular widget value from a custom node — throws on all retries, burns the whole SEND_RETRY_LIMIT budget, and settles the batch undeliverable, a state settleHumanOps neither reverts nor surfaces. The edit stays on the canvas, never reaches the server, and disappears on reload with no user-visible signal. Consider distinguishing transport refusals from serialization/programming errors, or surfacing terminal exhaustion to the user. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, kimi-k3-max adversarial).
| failure_kind: 'caught_unexpected', | ||
| feature_area: 'agent', | ||
| operation: 'sync', | ||
| outcome: 'recovered' |
There was a problem hiding this comment.
🟢 Low — outcome: 'recovered' is stamped at the moment of the first throw, before any retry has run, and reportedThrow suppresses any later report for that delivery — so a batch that throws on every attempt and settles undeliverable/unconfirmed (a human edit lost to the server) is still reported as recovered. Dashboards filtered on this errorType/outcome will therefore show zero unrecovered send failures. Report the terminal outcome once the retry budget is exhausted rather than asserting recovery up front. Raised by 3 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case, kimi-k3-max adversarial).
|
|
||
| async settled(): Promise<void> { | ||
| for (let pass = 0; pass < SETTLE_PASSES; pass++) { | ||
| if (this.recording.size === 0) return |
There was a problem hiding this comment.
🟢 Low — settled() returns immediately when recording is empty, which is precisely the race it exists to close: the GET /api/agent/threads handler awaits it, but Playwright dispatches the POST's response event asynchronously, so if the GET route fires first nothing has been added to recording, the barrier passes instantly, and the list is fulfilled without the just-acked thread. Tracking in-flight POSTs from the route/request handler (which runs before the response event) would give a real happens-before. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
| // One silent-result resend of the SAME minted ops: idempotent at the | ||
| // applier through the op_id gate. | ||
| batch.resent = true | ||
| batch.reportedThrow = false |
There was a problem hiding this comment.
⚪ Nit — reportedThrow only dedupes within one delivery and is reset on every silence resend, so a persistently throwing transport emits one reportError per batch — and each call also writes a console.error and drains the pending report queue. Batches are produced per coalesced user edit and per chunkWireOps chunk, so a bad socket during a drag or a large paste floods telemetry and the console exactly during the incident you most need to observe; a sender-scoped throttle would bound it. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).
Summary
Manual
cloud/1.55backport of #18538. The automatic carrier was not created because the release label was added after the source PR merged.This fixes late-result accounting and synchronous transport-throw recovery in the agent op sender, hardens the new-chat fixture, and adds an end-to-end recovery scenario.
Validation
opSender.test.ts: 49 tests passed