Skip to content

[Backport core/1.55] Fix agent sender retry accounting and transport recovery - #19590

Merged
christian-byrne merged 1 commit into
core/1.55from
backport-18538-core-1.55
Sep 30, 2026
Merged

christian-byrne merged 1 commit into
core/1.55from
backport-18538-core-1.55

Conversation

@christian-byrne

Copy link
Copy Markdown
Contributor

Summary

Manual core/1.55 backport of #18538. The automatic carrier was not created because the release label was added after the source PR merged.

This fixes late-result accounting and synchronous transport-throw recovery in the agent op sender, hardens the new-chat fixture, and adds an end-to-end recovery scenario.

Validation

  • opSender.test.ts: 49 tests passed
  • app typecheck passed
  • changed-file ESLint and formatting checks passed
  • cherry-pick applied without conflicts

… and the new-chat fixture (#18538)

## Summary

Follow-up to #18064, narrowed after #18700. Fixes four defects in the
agent op sender's late-result accounting and transport handling, makes
the new-chat e2e fixture fail on an acknowledgement it could previously
swallow, and adds an e2e that proves a human edit survives a transport
throw. Each sender fix landed as a red test first.

## Changes

- **What**:
- Late-result credits are reserved per send that actually left the
client, not per resend attempted. `batch.resent` was set before
`transmit()`, so a batch retired at that resend reserved two credits for
one transmission and the surplus swallowed the next batch's own result.
- A credit is retired only on the aborted batch's own identified result.
The sender remembers the op ids of every retired transmitted batch; a
result naming ops it never minted no longer spends the credit.
- An acknowledged batch reserves a credit for a send left unanswered. A
batch resent after silence has two sends on the wire, so settling it on
the first answer previously left the second to settle its successor.
- A throwing transport is a failed send, not a stalled sender. `pump()`
installs the in-flight record before `sendOps`, so a synchronous throw
left the slot occupied with no timer armed and every later batch queued
forever. The send now runs inside the sender's own boundary, is reported
once per delivery as `failure_sending_agent_human_ops`, and is retried
within the existing budget.
- The new-chat fixture records a failed turn acknowledgement instead of
skewing the thread list. `recordAccepted()` ran unawaited from
`page.on('response')`; the handlers are now filtered to turn acks,
drained before the thread list is served and before the teardown
assertion, and an unreadable body is reported separately from a
malformed one.
- An e2e covers the throwing transport end to end: the first outbound
`doc_ops` frame throws, and a renamed node survives the retry and a tab
switch. The fault lives in `fixtures/utils/throwOnFirstDocOpsSend.ts`
and records itself on the document, so the spec asserts the injection
fired rather than passing as a plain rename test.

## Review Focus

- `onOpsResult` is restructured rather than patched. It captures the
addressed batch, then picks between the in-flight batch's own identified
result, a retired batch's identified result (which drains its credit),
and an anonymous failure (the in-flight batch's only once no credit
could explain it).
- **Known cost, deliberately kept.** A credit reserved for a result that
never arrives is never drained, so the next batch's own answer pays for
it and reserves another on its resend: once one result is lost, every
later anonymous failure costs a `RESULT_TIMEOUT_MS` resend. Charging the
swallow back to the batch was tried and reverted — the sender cannot
tell whether an anonymous result is the retired batch's or the in-flight
batch's, and charging it as both under-reserves and settles a successor
on its predecessor's failure, reverting edits the host never rejected.
Closing it needs a credit lifetime the result contract does not define,
or a per-send id `DocFrameClient.sendOps` does not mint and
`DocOpsResult` does not echo. `never settles a batch with a result whose
ops it does not own` pins the invariant meanwhile.
- `staleAnonymousBudget` and `retiredOpIds` model the same retired sends
with separate lifetimes, and the set only sheds entries when the budget
reaches zero. Per-batch `{ opIds, credits }` records would fix that and
the cost above in one change; deliberately left as a follow-up.

## Verification

- Every `test:` commit fails against its parent. Against current
`main`'s `opSender.ts`, **nine** of this branch's sender tests fail and
40 pass.
- Mutation-checked: removing the identified-branch reservation, removing
the resend's re-arm of the throw report, adding a re-arm to `resume()`,
and changing the telemetry slug each turn exactly one test red.
- `typecheck`, `typecheck:browser`, `lint`, `format:check`, `knip` and
`fallow audit` clean.
- Both agent specs pass in the `cloud` project against a locally built
cloud bundle on a real backend. Two negative controls: forcing the
new-chat acknowledgement to a malformed 200 fails through the fixture's
teardown guard as `body unreadable`, and breaking the `doc_ops` frame
match fails the transport-throw spec on its injection assertion.
- The new-chat fixture's failure branches are not exercised by any
committed spec — its only consumer always gets a valid 2xx. They were
checked by hand; a table-driven test for them needs `browser_tests/**`
inside Vitest or the helper extracted, so it is worth its own change.

---------

Co-authored-by: Glary-Bot <glary-bot@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (2)
  • [release]
  • [backport

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Team

Run ID: 9323a515-a8e0-4026-ae65-5d266925cb32

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 09/30/2026, 07:24:07 AM UTC

Links

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

Fallow gates on new findings in changed files only, so everything below was introduced or moved by this PR (9 changed files).

Kind Finding Where Usual remedy
Complexity unsubscribe() — cyclomatic 14, CRAP 56.3 src/workbench/extensions/agent/crdt/opSender.ts:314 Extract a branch, or raise branch coverage — CRAP is CC² × (1−cov)³ + CC, so either moves it.

Pre-existing findings are baselined in .fallow-baselines/ and are not your
responsibility — see ADR-DEVEX-LINT-0015.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since f692a7272fcab0d18e8e61beed7e55eb1d6c0211

Full docs: ADR-DEVEX-LINT-0015

🎭 Playwright: ✅ 1858 passed, 0 failed

📊 Browser Reports
  • chromium: View Report (✅ 1838 / ❌ 0 / ⚠️ 0 / ⏭️ 13)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: ❌ Deployment failed
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

🧩 Custom Nodes Ecosystem Matrix: ✅ Passed

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

Stage Result What a failure here means
pin-status success Informational. It never fails the build; it warns when the corpus snapshot is over 30 days old.
corpus success Infrastructure: the pack mirror could not be fetched, validated or cached. Not your diff.
shards success Harness integrity, not pack breakage. A shard only checks that every built spec wrote its row; vitest's own exit code is deliberately ignored because pack code leaks unhandled rejections.
verdicts success The population verdict per renderer. This is the stage that judges packs. Exit 1 is FAIL; exit 2 is withheld — rows were missing, so nothing was measured, which is not the same as your change breaking packs.
detection-proofs success The instrument self-test: the matrix must FAIL on a deliberately poisoned corpus. A failure here means the gate cannot be trusted this run, in either direction.

On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff.

Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

@github-actions
github-actions Bot requested a review from ryan-di September 30, 2026 07:22
@christian-byrne christian-byrne added backport Backporting a PR onto a release candidate cursor-review Trigger multi-model Cursor agent review on this PR labels Sep 30, 2026
@github-actions github-actions Bot added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 30, 2026
@christian-byrne
christian-byrne merged commit 51bf8b8 into core/1.55 Sep 30, 2026
124 of 130 checks passed
@christian-byrne
christian-byrne deleted the backport-18538-core-1.55 branch September 30, 2026 07:33

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @christian-byrne.

Found 10 finding(s).

Severity Count
🟠 High 1
🟡 Medium 4
🟢 Low 4
⚪ Nit 1

Panel: 8/8 reviewers contributed findings.

return
}
settle({ state: 'acknowledged', ops: inFlight.ops, result })
retire(batch, 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — The anonymous-failure path settles the in-flight batch acknowledged without checking that the batch was ever transmitted, and retire(batch, 1) silently no-ops when sends === 0 (outstanding is -1). A batch parked by suspend() or sitting in the new 500 ms trySend retry window therefore gets settled by a stray result belonging to someone else, and settleHumanOps then runs revertRejectedOps with an empty applied set, reverting an edit that never left the client. The new sends counter makes the guard cheap: only attribute an anonymous result when batch.sends > 0, otherwise keep retrying. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max edge-case, kimi-k3-max adversarial).

function drainStaleCredit(): void {
if (staleAnonymousBudget === 0) return
staleAnonymousBudget--
if (staleAnonymousBudget === 0) retiredOpIds.clear()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — retiredOpIds is emptied only when staleAnonymousBudget happens to hit exactly 0, and neither the set nor the counter has a cap, eviction, or time bound. A single credit that is never drained (a permanently lost result, or a host that stops emitting doc_ops_result entirely) pins the budget above zero for the sender's lifetime, after which every retired or resent batch adds up to MAX_DOC_OPS_PER_FRAME more op ids that are never removed — unbounded growth for as long as the agent panel stays mounted. Track credits and ids per batch, or bound/expire them, rather than clearing on a global zero. Raised by 6 of 8 reviewers (claude-opus-5-thinking-max adversarial, gpt-5.6-sol-max adversarial, gpt-5.6-sol-max edge-case, gemini-3.1-pro adversarial, kimi-k3-max adversarial, kimi-k3-max edge-case).

if (staleAnonymousBudget > 0) staleAnonymousBudget--
return
if (batch && identified.some((opId) => batch.opIds.has(opId))) {
retire(batch, 1)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This newly reserves a late-result credit on the acknowledged path whenever sends === 2, but the most common reason a resend happened at all is that the first result was genuinely lost, so that credit can never be drained. The budget then stays at or above 1 for the session: each later batch has its first anonymous failure swallowed at line 339, burns a full RESULT_TIMEOUT_MS resend, and reserves a fresh credit when it settles — the human write path degrades to roughly one batch per 10 s with a growing queue and no recovery short of a reload. Expiring credits by age or scoping them to the batch would bound the cost the in-code comment currently just accepts. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

// no batch waiting, one addressed to another workflow, or one a stale
// credit could explain drains that credit so it cannot swallow a future
// batch's own result. Only then is it the in-flight batch's.
if (batch === null || staleAnonymousBudget > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — Credits are global even though results carry a workflowId. A credit reserved for a retired batch on workflow A swallows an anonymous failure whose workflowId matches the current in-flight batch on workflow B, even though the workflow id proves the result cannot be A's — costing B an unnecessary 10 s resend and carrying the phantom credit forward. Key credits by workflowId so a result that provably cannot belong to the retired batch is not spent on it. Raised by 2 of 8 reviewers (gpt-5.6-sol-max adversarial, gpt-5.6-sol-max edge-case).

if (batch && identified.some((opId) => batch.opIds.has(opId))) {
retire(batch, 1)
settle({ state: 'acknowledged', ops: batch.ops, result })
} else if (identified.some((opId) => retiredOpIds.has(opId))) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — A late identified rejection (ok:false with failed.op_id) for a batch that already settled unacknowledged takes this branch and only drains a credit, so onBatchSettled never sees it and revertRejectedOps never runs — the host rejected the write but the edit stays in the local graph permanently. The same information is lost when a batch with two sends settles on the first answer and the second, contradictory answer arrives afterwards. Now that retired op ids are tracked, this branch has enough context to reconcile the rejection instead of discarding it. Raised by 1 of 8 reviewers (gpt-5.6-sol-max edge-case).

failure_kind: 'caught_unexpected',
feature_area: 'agent',
operation: 'sync',
outcome: 'recovered'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — outcome: 'recovered' is hard-coded at throw time, before anything is known about recovery. A transport that keeps throwing past SEND_RETRY_LIMIT reaches settleUnbound with sends === 0 and settles undeliverable, dropping the user's ops, yet the only telemetry for that failure reports it as recovered. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

try {
return deps.sendOps(batch.workflowId, deps.tab, batch.ops)
} catch (error) {
if (!batch.reportedThrow) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — reportedThrow dedupes only within one delivery and is deliberately re-armed for the resend, so a persistently throwing transport (say a payload shape it cannot serialize) emits one or two reportError calls per batch — each fanning out to Sentry, Datadog, and the console — for every user edit, with no session-level cap. Sibling code in this module dedupes for the client's lifetime (reportedTypes in docFrameClient.ts); a sender-scoped flag would match. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

}

const nativeSend = WebSocket.prototype.send
let throwsLeft = 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — addInitScript re-executes in every new document and every same-origin frame, and throwsLeft lives in that per-document closure, so a reload, popup, or child frame re-arms the fault. The helper therefore injects "first doc_ops send per document", not the per-test contract the doc comment promises; hoisting the counter onto a page-level object shared across documents would make the contract hold. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

void record.finally(() => this.recording.delete(record))
})
await this.page.route('**/api/agent/threads', async (route) => {
await this.settled().catch((error: unknown) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — settled() only awaits ack recordings whose response event has already fired, so a GET /api/agent/threads that races a POST .../messages still in flight sees an empty recording set, returns immediately, and serves a list missing that thread. Closing the race needs posts accepted in answerPost to be tracked until their response has been recorded. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

INJECTED_DOC_OPS_THROWS_ATTR,
'1'
)
expect(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Nit — This is a non-retrying assertion on a session-global doc_ops frame count, unlike the sibling agentHumanRenameMintsSetNodeField.spec.ts, which baselines with opsBeforeRename. Any frame the replay emits before the rename would consume the single injected throw (so the rename never exercises the retry path the spec is named for) and break the count. Capturing the frame count before renameNode and asserting the delta pins the fault to the rename. Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport Backporting a PR onto a release candidate cursor-review Trigger multi-model Cursor agent review on this PR risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants