Repository navigation
fix(agent): stop retrying a permanently refused CRDT doc subscribe - #18704
Conversation
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎭 Playwright: ✅ 2144 passed, 0 failed📊 Browser Reports
🎨 Storybook: ✅ Built — View Storybook🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. No new findings in the changed files. Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 74e47713db1c15ce49aa09733718e242df5e7dc6
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: ✅ PassedExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight
On a pin-bump PR, a red gate means the ecosystem moved — not that your diff broke something. That is what the bump PR is for. Outside a pin bump or a cold rebuild of the unpinned tail, a red means the diff. Note the corpus size and the executed count are different numbers, and the verdict is against the executed count: roughly 3,100 of ~5,100 packs ship no extension-shaped JS and are skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md 📦 Bundle: 10 MB gzip 🟢 -3.75 kBDetailsSummary
Category Glance App Entry Points — 3.77 kB (baseline 3.77 kB) • ⚪ 0 BMain entry bundles and manifests
Status: 1 added / 1 removed Graph Workspace — 1.45 MB (baseline 1.45 MB) • 🟢 -3.29 kBGraph editor runtime, canvas, workflow orchestration
Status: 2 added / 2 removed / 1 unchanged Views & Navigation — 134 kB (baseline 134 kB) • ⚪ 0 BTop-level views, pages, and routed surfaces
Status: 14 added / 14 removed / 4 unchanged Panels & Settings — 639 kB (baseline 639 kB) • ⚪ 0 BConfiguration panels, inspectors, and settings screens
Status: 10 added / 10 removed / 18 unchanged User & Accounts — 39.4 kB (baseline 39.4 kB) • ⚪ 0 BAuthentication, profile, and account management bundles
Status: 7 added / 7 removed / 4 unchanged Editors & Dialogs — 126 kB (baseline 126 kB) • ⚪ 0 BModals, dialogs, drawers, and in-app editors
Status: 7 added / 7 removed / 1 unchanged UI Components — 125 kB (baseline 125 kB) • ⚪ 0 BReusable component library chunks
Status: 7 added / 7 removed / 9 unchanged Data & Services — 4 MB (baseline 4.01 MB) • 🟢 -6.9 kBStores, services, APIs, and repositories
Status: 12 added / 12 removed / 3 unchanged Utilities & Hooks — 578 kB (baseline 578 kB) • ⚪ 0 BHelpers, composables, and utility bundles
Status: 20 added / 20 removed / 16 unchanged Vendor & Third-Party — 17.4 MB (baseline 17.4 MB) • ⚪ 0 BExternal libraries and shared vendor chunks Status: 18 unchanged Other — 17.9 MB (baseline 17.9 MB) • 🟢 -905 BBundles that do not match a named category
Status: 82 added / 82 removed / 219 unchanged ⚡ Performance
|
The CRDT doc-subscribe lifecycle retried every subscribe refusal with the same exponential backoff regardless of error code, so a schema_version_mismatch refusal (the doc-host's terminal classification for a document this build can never read) retried forever instead of failing fast. Gate on the refusal code: schema_version_mismatch now latches the same give-up exit an unanswered ack times out into and surfaces a toast; every other code keeps the existing bounded retry.
e6c968a to
4020736
Compare
Fallow flagged onSubscribed() at cyclomatic 12 / CRAP 43.1 after this PR added the schema_version_mismatch check and onSyncError call. Pull that branch (and the code/message derivation) into a standalone helper, matching the module's existing pattern of pure top-level handlers (notifyAgentMaterialization, emitPendingMaterializations). No behavior change.
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #18704 +/- ##
==========================================
+ Coverage 84.11% 85.03% +0.92%
==========================================
Files 3164 2315 -849
Lines 154880 133574 -21306
Branches 42062 36550 -5512
==========================================
- Hits 130280 113587 -16693
+ Misses 22996 19669 -3327
+ Partials 1604 318 -1286
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 858 files with indirect coverage changes 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @christian-byrne.
Found 8 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 5 |
| 🟢 Low | 3 |
Panel: 8/8 reviewers contributed findings.
- giveUpPermanently is now idempotent (guards on already-latched
gaveUp) so a redelivered/duplicate refusal frame cannot re-run the
report/toast a second time.
- giveUpPermanently now cancels any already-armed subscribeRetryTimer
from an earlier retryable refusal, so it cannot outlive the
permanent latch and fire a resubscribe anyway.
- useAgentCrdtFollower re-imports SCHEMA_VERSION_MISMATCH_CODE instead
of re-spelling the literal, and runs releaseHeldOps/abortIfUnbound
before notifying onSyncError (matching onDocReset's cleanup-before-
notify order), so a throw from consumer code can no longer strand an
in-flight op batch.
- AgentPanelRoot's permanent-desync toast is now sticky (life: 0)
instead of auto-dismissing after 5s, and its detail always leads
with the localized copy, appending the doc-host's raw message
(clamped to 200 chars) as extra context instead of replacing it.
Not fixed, left unresolved for review:
- Gating the latch on which subscribe attempt a refusal answers: the
wire frame (docFrameClient.ts DocSubscribed) carries no per-attempt
correlation id, and answers are not guaranteed to arrive in send
order, so no same-file heuristic can reliably tell a stale answer
from a current one. Confirmed a same-file heuristic (gate on whether
an ack is currently outstanding) breaks the existing
useAgentCrdtFollower.test.ts:766 contract test. Needs a wire-level
fix spanning the relay/backend.
- Making the permanent latch survive a reconnect: agentCrdtDocLifecycle
.test.ts already has a dedicated, intentional regression test ("a
reconnect releases the permanent give-up latch") asserting the
opposite behavior. Reversing it is a product decision, not a small
addition.
christian-byrne
left a comment
There was a problem hiding this comment.
Reviewed against current main (0f8c42c035) and against the backend that produces the code this gate branches on, because the whole change hangs on one wire string.
The premise checks out, and the code string is real
schema_version_mismatch is genuinely reachable end-to-end, which is worth recording since nothing on main referenced the string before this PR:
cloudservices/agent/server/events_doc_frames.go→socketDocCode()returns("schema_version_mismatch", "the document schema is incompatible with this document service; retrying will not help"), and that pair goes straight onto the refusal frame viadocSubscribedRejection().cloudservices/ingest/server/services/websocket/doc_relay.go→docErrorCode()threads the agent's 422codethrough verbatim rather than collapsing it, with a comment namingschema_version_mismatchspecifically.DocSubscribedFrame.Reasoncarries the JSON tagmessage, so thedetail.messagethis PR reads is the right field.
And the classification really is terminal rather than a collapsed transient: a stored document at an older schema is accepted by staleSchemaSubscribe(), not refused, and PM-1604's own backend test is pm1604_subgraph_resync_schema_misclassification_test.go — a misclassification fix. The backend has already excluded the recoverable cases before the FE ever sees this code, so latching on it does not trade a spin for a silent give-up. That was the thing most worth checking here and it holds.
Two implementation traps are handled, and both are non-obvious enough to call out: giveUpPermanently() cancels an already-armed retryable backoff (whose callback checks only workflowId()), and it de-dupes a redelivered refusal frame. Both have tests. No new write path to the shared doc, so KEEP-ALIVE #6 is intact.
1. The permanent-code set is under-inclusive, and the codes it misses are the ones that spin hardest
catalog_mismatch sits in the same switch arm one line above schema_version_mismatch in socketDocCode(), with the identical contract: "the document was minted under a different widget catalog; retrying will not help". Ingest's docErrorCode() says it even more bluntly for the 422 case — "retrying cannot fix it, and a client that retries a permanent failure just amplifies it". unsupported (docSurfaceOffReason, sent by doc_relay.go when the doc surface is off for the deployment) is permanent for the life of the connection too.
After this PR both still take the retry ladder, and the ladder's exhaustion is not a stopping state. scheduleSubscribeRetry() returns at this.subscribeRetryAttempt >= SUBSCRIBE_RETRY_MAX_ATTEMPTS without setting gaveUp and without arming a timer — pinned by the existing test in agentCrdtDocLifecycle.test.ts ("refusal exhaustion"), which asserts exactly shouldDeferSubscribe() === false, onGaveUp not called, reportError not called. Because onSocketActivity in useAgentCrdtFollower.ts only bails while shouldDeferSubscribe() is true, and the bridge cleared sentWorkflowId on the refusal so hasPendingSubscribe stays true, every subsequent ComfyUI status frame re-drives bridge.reconcile() and re-sends the subscribe with no backoff at all — indefinitely, and silently.
So for catalog_mismatch the behaviour this PR is named after survives untouched. Suggest a set rather than a constant:
export const PERMANENT_SUBSCRIBE_REFUSAL_CODES: ReadonlySet<string> = new Set([
'schema_version_mismatch',
'catalog_mismatch'
])2. Please separate "stop retrying" from "tell the person"
giveUpPermanently() currently couples three things: the latch, reportError, and onGaveUp() → the sticky toast. That coupling is why (1) cannot simply be widened: unsupported most needs the latch (it is the code every user on a deployment with the doc surface off will receive, on every panel open), and most needs no toast — "This workflow was created with an incompatible document version" would be actively wrong copy for it.
giveUpPermanently(code, { notify: boolean }), or keeping the notify decision at the handleSubscribeRefusal() seam where it already lives, would let the latch cover every permanent code while the toast stays scoped to the one that really is a document-version desync.
3. A flaky socket stacks unbounded sticky toasts
onReconnected() → clearForRetarget() sets gaveUp = false. The reconnect resubscribes, the host refuses again with the same permanent code, and giveUpPermanently() runs from scratch: another reportError, another onGaveUp(), another life: 0 toast. Nothing dismisses the previous one, so N reconnects leave N stacked permanent-error toasts and N telemetry events for one unchanged fact.
Releasing the latch on reconnect is right — a new socket may reach a redeployed host. Re-notifying on it is not. Remembering the workflow id last notified (and re-arming on retarget / doc_reset rather than on a bare reconnect) would keep the toast once-per-lineage, which is what "permanent" implies.
4. Nit: the retryable-code fixture uses a code the server never sends
dispatchFrame('doc_subscribed', { ok: false, code: 'doc_not_found' }) — the wire code for that case is not_found (the docstore.ErrNotFound arm in both socketDocCode() and docErrorCode()). Any non-matching string exercises the retryable arm, so nothing is broken; but the fixture is the closest thing this repo has to a written record of the refusal vocabulary, and it is worth it being accurate — especially given (1), where a reader checking "did anyone look at the other codes?" will look here first.
5. PR body: "retried indefinitely" is true, but not for the reason a reader will check
The ladder is bounded — 6 attempts, ~31.5 s. The unbounded retry is the status-frame path in (1). A reader who checks SUBSCRIBE_RETRY_MAX_ATTEMPTS first will conclude the premise is wrong and stop. Worth one sentence naming the actual mechanism.
Disposition against the FE-2504 rewrite
4 of 9 files overlap #18700 — AgentPanelRoot.vue, crdt/devPanelLog.ts, crdt/useAgentCrdtFollower.ts, crdt/useAgentCrdtFollower.test.ts — all modified, none deleted. The gate itself (agentCrdtDocLifecycle.ts, workflowSyncErrorDetail.ts, docFrameClient.ts) is not touched by the rewrite at all, so the fix survives it; only the onSyncError wiring through useAgentCrdtFollower.ts into AgentPanelRoot.vue will need re-deriving, and AgentPanelRoot.vue loses 126 lines there.
Not merging this yet, and the reason is not the diff: src/workbench/extensions/agent/crdt/** is inside the informal freeze, and this is new behaviour (a new gate, a new event, a new toast) rather than a test extraction. There is a documented route — a named owner plus an exceptions row stating what happens to the fix when the refactor lands — and two such rows already exist for #18404 and #18766. This needs that decision, from Christian or Alex, not a quiet merge. Findings 1–3 are worth fixing either way, since 1 leaves the named behaviour unfixed for a sibling code.
…ying on reconnect Widens the permanent-refusal latch from a single schema_version_mismatch constant to PERMANENT_SUBSCRIBE_REFUSAL_CODES (schema_version_mismatch, catalog_mismatch, unsupported), so the sibling codes stop hitting the infinite-retry bug this gate exists to fix. giveUpPermanently() now takes a notify flag so unsupported (doc surface off for the whole deployment) latches without toasting every user, and the lifecycle remembers the last notified workflow id so a flaky socket reconnecting to the same workflow with the same permanent code reports once, not once per reconnect; retarget/doc_reset re-arm it. Also fixes a test fixture using the wrong wire code (doc_not_found instead of not_found). Addresses review comments from christian-byrne on PR #18704.
knip flags PERMANENT_SUBSCRIBE_REFUSAL_CODES as an unused export now that nothing outside agentCrdtDocLifecycle.ts imports it.
|
Thanks for the precise backend citations, Christian — All 3 design findings plus the nit are addressed in 6ae63be:
New/updated tests cover all of the above in Per this repo's Generated by Claude Code |
christian-byrne
left a comment
There was a problem hiding this comment.
The permanent-refusal notification state still has a blocking recovery bug. The existing stale-ack race thread remains applicable and is not duplicated here; the other two comments are non-blocking follow-ups.
…race, thread refusal code Christian's second review (pr18704-f2, blocking): a successful confirm after a permanent refusal left `lastNotifiedGiveUpWorkflowId` pointing at the workflow, so a LATER independent permanent refusal for the same workflow was silently suppressed (no reportError, no onGaveUp, no toast). `onSubscribeConfirmed()` now calls `resetNotifiedGiveUp()`, ending the notification-dedupe episode on actual recovery while still leaving it latched across a reconnect that never recovers. Stale-ack race (root-caused, previously left open for a human call): an ack-timeout resubscribe can leave an older subscribe attempt outstanding while a newer one (same lineage) confirms first. A delayed permanent refusal answering that older attempt then arrived and gave up on an otherwise healthy channel, tearing down the stale probe the confirm had just armed. Root cause: the latch had no notion of which attempt an incoming answer belonged to. Fix: track `confirmedSinceLastSend`, set on `onSubscribeConfirmed()` and cleared on the next `onSubscribeSent()`. A confirm and a permanent refusal can never both answer the same subscribe, so a permanent refusal arriving while this is true must be a stale answer to a superseded attempt - `onSubscribeRefused` now discards it before touching any timer. This is scoped to the permanent-refusal path only: the existing "a confirm between refusals restarts the backoff" contract for retryable refusals is untouched, and the general "giveUpPermanently cancels an already-armed retryable backoff" contract still holds when no confirm intervened. f3 (non-blocking): `onSyncError` collapsed every permanent refusal code to the same localized copy, so `catalog_mismatch` reached the "incompatible document version" message written for `schema_version_mismatch`. `handleSubscribeRefusal`/`onSyncError` now thread the refusal `code` through, and `formatWorkflowSyncErrorDetail` picks a dedicated `workflowSyncFailedDetailCatalogMismatch` copy for it. f5 (non-blocking): added a narrow AgentPanelRoot.vue component test proving the onSyncError wiring end-to-end (error toast, formatted detail, life: 0) - the follower and formatter each had unit coverage, but nothing asserted they were actually connected. Fixed the test harness's `ws.emit` helper to dispatch a real CustomEvent (the CRDT doc-frame pipeline checks `instanceof CustomEvent` before reading `.detail`), which the new test needed to reach the follower at all; verified the new test by temporarily removing the onSyncError wiring and observing it fail, then restored it.
e3aaf47 to
e6f668a
Compare
christian-byrne
left a comment
There was a problem hiding this comment.
Re-reviewed at 0a94625 after merging current main. The schema-refusal path remains fail-closed, focused tests cover the lifecycle and UI wiring, and the reconnect tradeoff is documented. Local focused suite: 436 passed; typecheck, lint, format, Knip, and Fallow gates pass.
|
Exact-head landing judgment: |
|
|
|
|
|
Release follow-up: automated backports failed on all four targets. Core/cloud 1.54 do not contain the CRDT lifecycle files this retry gate changes, so this fix is not applicable there. Target-native 1.55 adaptations are open as #19182 (core/1.55) and #19183 (cloud/1.55); both retain the 1.55 reset path while adding the permanent-refusal gate and durable sync-error notification. Local validation: 381 focused tests, full workspace typecheck, formatting, and pre-push dependency checks passed. |
…ackport #18704 to the 1.55 release branch with its native reset path.\nPreserve the permanent schema-mismatch refusal gate, durable sync-error\nnotification, and reset of the one-shot give-up latch.
Backport #18704 to the 1.55 release branch with its native reset path. Preserve the permanent schema-mismatch refusal gate, durable sync-error notification, and reset of the one-shot give-up latch.
…ackport #18704 to the 1.55 release branch with its native reset path.\nPreserve the permanent schema-mismatch refusal gate, durable sync-error\nnotification, and reset of the one-shot give-up latch.
Backport #18704 to the 1.55 release branch with its native reset path. Preserve the permanent schema-mismatch refusal gate, durable sync-error notification, and reset of the one-shot give-up latch.
Summary
The CRDT doc-subscribe lifecycle retried every
doc_subscribed {ok:false}refusal with the same bounded exponential backoff, regardless of the refusal's error code. When the doc-host classifies a resync asschema_version_mismatch(a terminal, permanent refusal for a document this build can never read — the backend half of this landed incloud#10215), retrying cannot change the outcome, so the frontend kept retrying indefinitely instead of failing fast, and the workflow canvas eventually went stale with no specific explanation.Changes
AgentCrdtDocLifecycle.onSubscribeRefusednow takes the refusal's errorcode.schema_version_mismatchlatches the same give-up exit an unanswered subscribe ack already times out into (stops the retry loop, reports it once viareportError); every other code keeps the existing bounded-backoff retry unchanged.useAgentCrdtFollowerpasses the code through and, only forschema_version_mismatch, invokes a new optionalonSyncErrorevent so the panel can tell the person their canvas is out of sync.AgentPanelRoot.vuewires that into the existing toast surface.Review Focus
Confirm the permanent-refusal path (
schema_version_mismatch) stays fail-closed (no further retries until a reconnect) while every other refusal code retains its bounded retry — see the new tests inagentCrdtDocLifecycle.test.tsanduseAgentCrdtFollower.test.ts.Related
main— the intended approach is the same, but implemented as a small, self-contained gate rather than resurrecting that branch. Leaving the decision to close fix(agent): explain and gate workflow schema mismatches #16590 to a human reviewer.Testing
pnpm exec vitest run src/workbench/extensions/agent/crdt/agentCrdtDocLifecycle.test.ts src/workbench/extensions/agent/crdt/useAgentCrdtFollower.test.ts src/workbench/extensions/agent/AgentPanelRoot.test.ts— 363 passedpnpm typecheck— passedpnpm exec oxlint/pnpm exec eslint/pnpm exec oxfmt --checkon changed files — passed (also enforced by the pre-commit hook)pnpm exec knip— no new dead code/exportsScreenshots (if applicable)
Not applicable — the change is a retry/error-code gate plus a toast that reuses the existing toast surface; covered by unit tests.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YLFTHrz4bPiD5ZHomCADTP
Generated by Claude Code