Skip to content

[Backport cloud/1.55] Fix agent sender retry accounting and transport recovery - #19591

Merged
christian-byrne merged 1 commit into
cloud/1.55from
backport-18538-cloud-1.55
Sep 30, 2026
Merged

christian-byrne merged 1 commit into
cloud/1.55from
backport-18538-cloud-1.55

Conversation

@christian-byrne

Copy link
Copy Markdown
Contributor

Summary

Manual cloud/1.55 backport of #18538. The automatic carrier was not created because the release label was added after the source PR merged.

This fixes late-result accounting and synchronous transport-throw recovery in the agent op sender, hardens the new-chat fixture, and adds an end-to-end recovery scenario.

Validation

  • opSender.test.ts: 49 tests passed
  • app typecheck passed
  • source PR CI passed with no failures
  • cherry-pick applied without conflicts

… and the new-chat fixture (#18538)

## Summary

Follow-up to #18064, narrowed after #18700. Fixes four defects in the
agent op sender's late-result accounting and transport handling, makes
the new-chat e2e fixture fail on an acknowledgement it could previously
swallow, and adds an e2e that proves a human edit survives a transport
throw. Each sender fix landed as a red test first.

## Changes

- **What**:
- Late-result credits are reserved per send that actually left the
client, not per resend attempted. `batch.resent` was set before
`transmit()`, so a batch retired at that resend reserved two credits for
one transmission and the surplus swallowed the next batch's own result.
- A credit is retired only on the aborted batch's own identified result.
The sender remembers the op ids of every retired transmitted batch; a
result naming ops it never minted no longer spends the credit.
- An acknowledged batch reserves a credit for a send left unanswered. A
batch resent after silence has two sends on the wire, so settling it on
the first answer previously left the second to settle its successor.
- A throwing transport is a failed send, not a stalled sender. `pump()`
installs the in-flight record before `sendOps`, so a synchronous throw
left the slot occupied with no timer armed and every later batch queued
forever. The send now runs inside the sender's own boundary, is reported
once per delivery as `failure_sending_agent_human_ops`, and is retried
within the existing budget.
- The new-chat fixture records a failed turn acknowledgement instead of
skewing the thread list. `recordAccepted()` ran unawaited from
`page.on('response')`; the handlers are now filtered to turn acks,
drained before the thread list is served and before the teardown
assertion, and an unreadable body is reported separately from a
malformed one.
- An e2e covers the throwing transport end to end: the first outbound
`doc_ops` frame throws, and a renamed node survives the retry and a tab
switch. The fault lives in `fixtures/utils/throwOnFirstDocOpsSend.ts`
and records itself on the document, so the spec asserts the injection
fired rather than passing as a plain rename test.

## Review Focus

- `onOpsResult` is restructured rather than patched. It captures the
addressed batch, then picks between the in-flight batch's own identified
result, a retired batch's identified result (which drains its credit),
and an anonymous failure (the in-flight batch's only once no credit
could explain it).
- **Known cost, deliberately kept.** A credit reserved for a result that
never arrives is never drained, so the next batch's own answer pays for
it and reserves another on its resend: once one result is lost, every
later anonymous failure costs a `RESULT_TIMEOUT_MS` resend. Charging the
swallow back to the batch was tried and reverted — the sender cannot
tell whether an anonymous result is the retired batch's or the in-flight
batch's, and charging it as both under-reserves and settles a successor
on its predecessor's failure, reverting edits the host never rejected.
Closing it needs a credit lifetime the result contract does not define,
or a per-send id `DocFrameClient.sendOps` does not mint and
`DocOpsResult` does not echo. `never settles a batch with a result whose
ops it does not own` pins the invariant meanwhile.
- `staleAnonymousBudget` and `retiredOpIds` model the same retired sends
with separate lifetimes, and the set only sheds entries when the budget
reaches zero. Per-batch `{ opIds, credits }` records would fix that and
the cost above in one change; deliberately left as a follow-up.

## Verification

- Every `test:` commit fails against its parent. Against current
`main`'s `opSender.ts`, **nine** of this branch's sender tests fail and
40 pass.
- Mutation-checked: removing the identified-branch reservation, removing
the resend's re-arm of the throw report, adding a re-arm to `resume()`,
and changing the telemetry slug each turn exactly one test red.
- `typecheck`, `typecheck:browser`, `lint`, `format:check`, `knip` and
`fallow audit` clean.
- Both agent specs pass in the `cloud` project against a locally built
cloud bundle on a real backend. Two negative controls: forcing the
new-chat acknowledgement to a malformed 200 fails through the fixture's
teardown guard as `body unreadable`, and breaking the `doc_ops` frame
match fails the transport-throw spec on its injection assertion.
- The new-chat fixture's failure branches are not exercised by any
committed spec — its only consumer always gets a valid 2xx. They were
checked by hand; a table-driven test for them needs `browser_tests/**`
inside Vitest or the helper extracted, so it is worth its own change.

---------

Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (2)
  • [release]
  • [backport

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Team

Run ID: 70edd7cf-b4ae-47e3-80b6-f67f0035d9cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@christian-byrne christian-byrne added backport Backporting a PR onto a release candidate cursor-review Trigger multi-model Cursor agent review on this PR labels Sep 30, 2026
@github-actions
github-actions Bot requested a review from ryan-di September 30, 2026 07:25
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Playwright: ✅ 2121 passed, 0 failed · 1 flaky

📊 Browser Reports
  • chromium: View Report (✅ 2100 / ❌ 0 / ⚠️ 1 / ⏭️ 13)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • New-test walkthrough (chromium, recorded video): View Report

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 09/30/2026, 07:28:28 AM UTC

Links

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (9 changed files).

Kind Finding Where Usual remedy
Complexity unsubscribe() — cyclomatic 14, CRAP 56.3 src/workbench/extensions/agent/crdt/opSender.ts:314 Extract a branch, or raise branch coverage — CRAP is CC² × (1−cov)³ + CC, so either moves it.

Pre-existing findings are baselined in .fallow-baselines/ and are not your
responsibility — see ADR-DEVEX-LINT-0015.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 32838ae557648bb9d2066c4a376889df7dd9f665

Full docs: ADR-DEVEX-LINT-0015

🧩 Custom Nodes Ecosystem Matrix: ✅ Passed

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

Stage Result What a failure here means
pin-status success Informational. It never fails the build; it warns when the corpus snapshot is over 30 days old.
corpus success Infrastructure: the pack mirror could not be fetched, validated or cached. Not your diff.
shards success Harness integrity, not pack breakage. A shard only checks that every built spec wrote its row; vitest's own exit code is deliberately ignored because pack code leaks unhandled rejections.
verdicts success The population verdict per renderer. This is the stage that judges packs. Exit 1 is FAIL; exit 2 is withheld — rows were missing, so nothing was measured, which is not the same as your change breaking packs.
detection-proofs success The instrument self-test: the matrix must FAIL on a deliberately poisoned corpus. A failure here means the gate cannot be trusted this run, in either direction.

On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff.

Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

@christian-byrne
christian-byrne merged commit e268690 into cloud/1.55 Sep 30, 2026
130 checks passed
@christian-byrne
christian-byrne deleted the backport-18538-cloud-1.55 branch September 30, 2026 07:33
@github-actions github-actions Bot added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 30, 2026
@codecov

codecov Bot commented Sep 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@              Coverage Diff              @@
##             cloud/1.55   #19591   +/-   ##
=============================================
  Coverage              ?   80.55%           
=============================================
  Files                 ?     2210           
  Lines                 ?   117125           
  Branches              ?    35455           
=============================================
  Hits                  ?    94347           
  Misses                ?    22391           
  Partials              ?      387           
Flag Coverage Δ
unit 80.55% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/workbench/extensions/agent/crdt/opSender.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @christian-byrne.

Found 7 finding(s).

Severity Count
🟠 High 2
🟡 Medium 2
🟢 Low 2
⚪ Nit 1

Panel: 8/8 reviewers contributed findings.

return
}
settle({ state: 'acknowledged', ops: inFlight.ops, result })
retire(batch, 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — When an anonymous failure (empty applied/skipped, no failed op_id) arrives while the in-flight batch has sends === 0 — parked by suspend(), or still waiting on a send-retry timer after a refused/throwing send — retire(batch, 1) computes outstanding = -1 and no-ops, then the batch is settled acknowledged, telling downstream code the host applied ops that never left the client. The suspension tests construct exactly this state via parkSecondBatch, and settle() pumps the next queued batch into the same parked slot, so a run of anonymous failures during a suspension can silently discard the whole queue. Guard this path with batch.sends > 0 (ignore the frame, or settle undeliverable). Raised by 3 of 8 reviewers (claude-opus-5-thinking-max edge-case, gemini-3.1-pro edge-case, kimi-k3-max adversarial).

function retire(batch: InFlight, answered: number): void {
const outstanding = batch.sends - answered
if (outstanding <= 0) return
staleAnonymousBudget += outstanding

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — retire(batch, 0) on the unacknowledged timeout (line 251) and in settleUnbound (line 239) reserves credits for answers that by definition never arrive, and nothing caps, ages, or resets them — so staleAnonymousBudget never returns to 0, retiredOpIds.clear() never runs, and the set keeps accruing op ids from every later retire (unbounded growth keyed to user edit volume for the life of the session). Worse, a nonzero budget is self-sustaining: each new batch's own anonymous answer is swallowed, it resends, the next answer is swallowed, it settles unacknowledged and re-adds credits — so revertRejectedOps never runs and host-rejected ops stay silently diverged on the canvas while every batch costs a RESULT_TIMEOUT_MS stall. Expiring credits with the batch they were reserved for, or bounding the budget, would make the state recoverable. Raised by 5 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case, gpt-5.6-sol-max adversarial, gemini-3.1-pro adversarial, kimi-k3-max adversarial).

function drainStaleCredit(): void {
if (staleAnonymousBudget === 0) return
staleAnonymousBudget--
if (staleAnonymousBudget === 0) retiredOpIds.clear()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The ledger is one global counter plus one flat id set, so a credit is consumed by whichever result arrives next rather than by the batch it was reserved for: an anonymous answer for the in-flight batch drains retired batch A's credit, and drainStaleCredit() then clears the set wholesale (dropping retired batch B's ids too), after which A's or B's genuine late anonymous result is attributed to and settles the current batch with someone else's result — the exact mis-settle this scheme exists to prevent. Credits are also not scoped by workflowId, so a stale credit minted before a workflow switch is spent on the new workflow's answer and the resend recreates a phantom credit. A per-batch map from op id to its reserved credit would make the accounting exact. Raised by 3 of 8 reviewers (gpt-5.6-sol-max edge-case, gpt-5.6-sol-max adversarial, claude-opus-5-thinking-max edge-case).

level: 'error'
})
}
return false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — trySend catches every exception and treats it as a transient refusal, so a deterministic failure — e.g. JSON.stringify in DocFrameClient.send throwing on a BigInt or circular widget value from a custom node — throws on all retries, burns the whole SEND_RETRY_LIMIT budget, and settles the batch undeliverable, a state settleHumanOps neither reverts nor surfaces. The edit stays on the canvas, never reaches the server, and disappears on reload with no user-visible signal. Consider distinguishing transport refusals from serialization/programming errors, or surfacing terminal exhaustion to the user. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, kimi-k3-max adversarial).

failure_kind: 'caught_unexpected',
feature_area: 'agent',
operation: 'sync',
outcome: 'recovered'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — outcome: 'recovered' is stamped at the moment of the first throw, before any retry has run, and reportedThrow suppresses any later report for that delivery — so a batch that throws on every attempt and settles undeliverable/unconfirmed (a human edit lost to the server) is still reported as recovered. Dashboards filtered on this errorType/outcome will therefore show zero unrecovered send failures. Report the terminal outcome once the retry budget is exhausted rather than asserting recovery up front. Raised by 3 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case, kimi-k3-max adversarial).


async settled(): Promise<void> {
for (let pass = 0; pass < SETTLE_PASSES; pass++) {
if (this.recording.size === 0) return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — settled() returns immediately when recording is empty, which is precisely the race it exists to close: the GET /api/agent/threads handler awaits it, but Playwright dispatches the POST's response event asynchronously, so if the GET route fires first nothing has been added to recording, the barrier passes instantly, and the list is fulfilled without the just-acked thread. Tracking in-flight POSTs from the route/request handler (which runs before the response event) would give a real happens-before. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

// One silent-result resend of the SAME minted ops: idempotent at the
// applier through the op_id gate.
batch.resent = true
batch.reportedThrow = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Nit — reportedThrow only dedupes within one delivery and is reset on every silence resend, so a persistently throwing transport emits one reportError per batch — and each call also writes a console.error and drains the pending report queue. Batches are produced per coalesced user edit and per chunkWireOps chunk, so a bad socket during a drag or a large paste floods telemetry and the console exactly during the incident you most need to observe; a sender-scoped throttle would bound it. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Backporting a PR onto a release candidate cursor-review Trigger multi-model Cursor agent review on this PR risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants