Repository navigation
test(agent): cover template-replace scenario for agent delete-all race - #18702
claude[bot] wants to merge 3 commits into
Conversation
Adds a regression test replaying Jo Zhang's exact staging report: load one template, then ask the agent to replace it with a different one. Before this PR's fix, the old template's nodes were never deleted and stayed on screen next to the new template's nodes forever (proven red against the pre-fix EcsFollowerAdapter/AgentCrdtProjection). With the fix, the self-driven reconcile retry replays the dropped replace once scope recovers, without any further frame arriving, and the old template is actually removed while the new one is actually added. Lands next to the existing "self-driven reconcile retry" describe block in agentCrdtProjection.integration.test.ts, driven through the same adapter/projection chain, since a true Playwright agent-conversation replay of this scenario needs a recorded fixture from the cloud agent's non-standalone stack (Postgres/Redis/doc host + a real model), which is not reachable from this environment.
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎭 Playwright: ✅ 2117 passed, 0 failed📊 Browser Reports
🎨 Storybook: ✅ Built — View Storybook🌾 Fallow: ✅ PassedFallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened. No new findings in the changed files. Reproduce locally# the base the gate compared against
npx fallow audit --root . --quiet --format json \
--changed-since 255dbbbf3cda1103c861864c5b68002f8745ad9b
Full docs: ADR-DEVEX-LINT-0015 🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measureExecutes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight This candidate changes nothing the matrix measures, so every expensive stage was skipped. Reproduce locally# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
npx vitest run --config vitest.matrix.config.mts
Full docs: scripts/registry-census/README.md 📦 Bundle Size
|
There was a problem hiding this comment.
🔍 Cursor Review — Consolidated panel
Triggered by @christian-byrne.
Found 5 finding(s).
| Severity | Count |
|---|---|
| 🟡 Medium | 1 |
| 🟢 Low | 3 |
| ⚪ Nit | 1 |
Panel: 8/8 reviewers contributed findings.
| } | ||
| } satisfies Op | ||
| applyOps(host, [insertOp], CATALOG) | ||
| scopeAvailable = false |
There was a problem hiding this comment.
🟡 Medium — Injecting the failure by flipping scopeAvailable makes createGraphMutations().batch() bail on its getScope() check before the projection callback runs, so the incremental delete+add path a real replace turn takes is never exercised; the retry then re-reads the doc through the full-reconcile branch, which is convergent by construction and also prepares node 3 via reconcileNode rather than addNode. As written this test cannot fail for the PM-1500/PM-1504 "insert instead of replace" defect it is meant to pin — inject the rejection downstream of the scope check so the replace batch itself is what gets replayed. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).
There was a problem hiding this comment.
Confirmed real, and confirmed by tracing the actual code (not just re-reading): createGraphMutations().batch() calls getScope() and returns false before it ever invokes the define() callback that builds the incremental ops. So flipping scopeAvailable meant the rejected "replace" frame never reached ecsFollowerAdapter's nodeActions-driven delete+add builder at all. And since reconcileNextFrame is set unconditionally on any rejected batch, the self-driven retry always replays through the full-reconcile branch (reconcileNode, not addNode) — exactly as you described.
Fix: split into two tests. One now delivers the replace frame with scope resolved throughout, so it genuinely exercises the incremental delete+add path a successful replace turn takes. The other keeps the scope-race/retry scenario (now honestly commented as proving full-reconcile recovery, which is the actual #18404 fix — the retry not sweeping the live graph).
Verified with a mutation test: skipping the incremental delete in ecsFollowerAdapter.ts makes the new single-batch test fail red, while the old retry-based test still passes green — proving the original test gave zero coverage of that path, matching your finding exactly. Re-verified red/green: the retry test is red against the pre-fix commit (3a4b4a9, before the retry+live-sweep fix) and green on this branch; the new single-batch test passes on both (it doesn't depend on that particular fix, since the incremental path was already correct).
Pushed in ce277bd.
Generated by Claude Code
| } | ||
|
|
||
| // Turn 1: the old template loads and catches up onto the live graph. | ||
| deliver(Y.encodeStateAsUpdate(host)) |
There was a problem hiding this comment.
🟢 Low — Turn 1's deliver() result is discarded, unlike bindAndCatchUp elsewhere in this suite which asserts .toBe(true), and the two assertions that follow are already satisfied by buildLiveGraph() alone. If catch-up silently stopped applying, reconcileNextFrame would stay armed and turn 2 would take the full-reconcile branch instead of the incremental one, yet the test would still go green; assert expect(deliver(...)).toBe(true). Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).
There was a problem hiding this comment.
Fixed — turn 1's deliver(...) calls in both tests now assert .toBe(true). Pushed in ce277bd.
Generated by Claude Code
| outputs: [{ name: 'image', type: 'IMAGE', links: [] }] | ||
| } | ||
| } satisfies Op | ||
| applyOps(host, [insertOp], CATALOG) |
There was a problem hiding this comment.
🟢 Low — applyOps's result is dropped here, unlike other call sites in this suite that assert outcomes[0].outcome === 'applied', and the op carries a hand-picked base_version: 10/stamp well ahead of the freshly minted doc's version. If the applier ever rejects or LWW-drops the insert, the scenario silently degrades to the plain delete-all case an earlier test already covers and the failure surfaces 20 lines later at the unrelated title assertion. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).
There was a problem hiding this comment.
Fixed. applyOps(host, [insertOp], CATALOG)'s result is now captured and asserted: expect(applied.outcomes).toEqual([{ op_id: insertOp.op_id, outcome: 'applied' }]). Also fixed the base_version/stamp: instead of the hand-picked 10, it's now computed as appliedOpIds(host).length right before building the op, so it actually reflects the freshly-minted doc's applied-op count instead of an arbitrary number that happened not to collide. Pushed in ce277bd.
Generated by Claude Code
| .getGraphNodesFor(scope.rootGraphId, scope.owningGraphId) | ||
| .map(({ id }) => id) | ||
| ).toEqual([toNodeId(3)]) | ||
| expect(layout.deleteNodes.mock.calls.flatMap(([, ids]) => ids)).toEqual( |
There was a problem hiding this comment.
🟢 Low — expect.arrayContaining is an unordered superset check, so this still passes if the retry also churns node 3 through a delete, deletes unrelated ids, or issues duplicate deletes — weaker than the stated goal of proving "not that only half of the replace ever lands"; the commit path deletes exactly [1] then [2], so toEqual([toNodeId(1), toNodeId(2)]) is provable. The expect.anything() layout args on the createNode assertion below likewise never check where the replacement node lands, which is part of the reported symptom. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).
There was a problem hiding this comment.
Fixed both. layout.deleteNodes.mock.calls.flatMap(([, ids]) => ids) now asserts toEqual([toNodeId(1), toNodeId(2)]) (exact array, not arrayContaining) in both tests. The layout.createNode assertion now checks the real layout payload instead of expect.anything(): { position: { x: 400, y: 20 }, size: { width: 180, height: 90 } }, matching where the replacement node (pos: [400, 20], size: [180, 90]) actually lands. Pushed in ce277bd.
Generated by Claude Code
| expect.anything() | ||
| ) | ||
|
|
||
| projection.destroy() |
There was a problem hiding this comment.
⚪ Nit — The three destroy() calls sit inside the try after the assertions, so any assertion failure skips them and leaves the EcsFollowerAdapter session subscribed, a retry timer possibly pending, and two Y.Docs undisposed — compounding across each configured CI retry. Move them into the finally alongside vi.useRealTimers(). Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).
There was a problem hiding this comment.
Fixed — projection.destroy() / follower.destroy() / host.destroy() are now in a finally block (nested inside the vi.useFakeTimers() try/finally for the retry test) instead of trailing after the assertions. Pushed in ce277bd.
Generated by Claude Code
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## fix/agent-delete-all-scope-drop-retry #18702 +/- ##
=========================================================================
- Coverage 84.36% 84.08% -0.29%
=========================================================================
Files 2284 2294 +10
Lines 130580 134893 +4313
Branches 35504 36657 +1153
=========================================================================
+ Hits 110170 113422 +3252
- Misses 20114 21170 +1056
- Partials 296 301 +5
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
createGraphMutations().batch() checks getScope() before ever invoking the define() callback that builds the incremental delete+add ops, so flipping scopeAvailable to fail the batch meant the rejected "replace" frame never reached that builder at all. The self-driven retry then always replays through the full-reconcile branch (reconcileNode, not addNode), so the original test could not fail on a defect specific to the incremental delete+add composition a successful replace turn actually takes. Split the scenario into two tests: one delivers the replace frame with scope resolved throughout, proving the incremental path itself correctly removes the old template and adds the new one; the other keeps the scope-race + self-driven-retry scenario, now accurately commented as proving full-reconcile recovery, with exact (not arrayContaining) node-id assertions, a real position payload on the layout.createNode assertion instead of expect.anything(), a computed base_version/stamp instead of a hand-picked one, asserted applyOps/deliver outcomes instead of discarded results, and destroy() calls moved into a finally block. Verified: the new single-batch test fails when the incremental delete is mutated out (confirming it actually exercises that path, unlike the original), both tests are green on this branch, and the retry test alone is red against the pre-fix commit (3a4b4a9, before the retry/live-sweep fix) while the single-batch test passes there unmodified, since it does not depend on that fix.
…template-replace-delete-race # Conflicts: # src/workbench/extensions/agent/crdt/agentCrdtProjection.integration.test.ts
|
Closing as superseded after revalidating #18700. Commit 6ec9537 deletes agentCrdtProjection and moves delete-all through direct graph-API apply; the #18700 fail→pass ledger records the durable delete-all behavior as absorbed by agentDeleteAndClearWorkflow.spec.ts from #18404. The duplicate-insert/opaque-widget black-box symptom is also retained on main via merged #18703, so this projection-only carrier has no remaining coverage to preserve. |
…on canvas (Comfy-Org#18969) ## Summary Re-authors the regression guard from Comfy-Org#18702 — closed without merging — as a black-box `browser_tests/` case, so the live bug it covered (ask the agent to clear the canvas, the turn reports the tab is empty, the nodes stay on screen — PM-1500, PM-1504) has a test again. ## Changes - **What**: new `browser_tests/tests/agent/agentClearedWorkflowStaysCleared.spec.ts` (two cases) plus one `browser_tests/types/globals.d.ts` declaration. Test-only; no production code touched. ## Why re-authored rather than reopened Comfy-Org#18702's base was `fix/agent-delete-all-scope-drop-retry` (Comfy-Org#18404), which is not landing, and it asserted against `AgentCrdtProjection` and `EcsFollowerAdapter` — both deleted by Comfy-Org#18700. Rebasing it would have restored a test onto a base that cannot merge, stated in code that is being removed. This states the same property where the user reads it: the rendered canvas. Every API it touches was checked against Comfy-Org#18700 and survives — `agentConversationTest` / `conversationCase`, `addedNodeIds`, `runTurns`, `subscribeCount`, `VueNodeHelpers.nodes` / `getNodeLocator`, `Topbar`, and the `agent-rec-clear-workflow` fixture. It asserts nothing in `agentCrdtProjection`, `ecsFollowerAdapter`, `graphMutations`, `agentNodeMaterializer` or `agentSubgraphHostSlots`, and `globals.d.ts` is untouched by Comfy-Org#18700, so there is no conflict there either. ## Review Focus **It closes a vacuous-pass hole, which is most of the point.** The recorded turn adds one node and then clears the workflow, so asserting only that the canvas ends empty cannot distinguish *the clear landed* from *neither operation did*. That is the failure mode flagged as PM-1755 — a recorded replay can skip a clear and still report a pass — and it is real here, measured, not hypothetical: | fault injected into the replay | `agentConversationReplay` (recorded case) | `agentDeleteAndClearWorkflow` on Comfy-Org#18700 | this spec | | --- | --- | --- | --- | | none | pass | pass | pass | | clear frame withheld from the client (the live bug) | **fail** | — | **fail** | | whole turn's graph ops withheld | pass | pass | **fail** | The distinguishing assertion records every `data-node-id` the DOM mounts, so it reads only what the user sees and does not care which graph API removed the node. **This lands green — it is a pin, not a live repro, and the PR does not claim a fix is needed.** The mechanism Comfy-Org#18702 drove the bug through was a mutation batch rejected because the app briefly could not resolve which workflow tab the edit was bound to. Comfy-Org#18700 removes that mechanism rather than fixing it: `AgentPanelRoot.vue` there has no `getScope` and no `createGraphMutations`, and `graphMutations.ts` / `ecsFollowerAdapter.ts` are deleted. So there is no surviving surface to inject the race from, and Comfy-Org#18404's `dropWorkflowScope()` — which reached into Pinia to delete `tab.activeState.id` — would be asserting against the very thing being deleted. What is worth holding across that change is the user-visible outcome, on both sides of it. **Second case, tab round trip.** Returning to the tab re-subscribes the follower, which replays the bound document over the tab's own snapshot; the case waits for that subscribe before asserting, so it is about the catch-up rather than about a canvas nothing has touched yet. `agentConversationReplay` never switches tabs, so this was unpinned. ## Verification Local, against a `DISTRIBUTION=cloud` dev server over a containerised ComfyUI backend with devtools: ``` PLAYWRIGHT_LOCAL=1 DISTRIBUTION=cloud pnpm exec playwright test \ browser_tests/tests/agent/agentClearedWorkflowStaysCleared.spec.ts --project=cloud ``` - `main` (9cb7f5b): 2 passed - Comfy-Org#18700 (540a2b8), spec copied in unchanged: 2 passed - red/green table above reproduced by temporary, uncommitted edits to `replayResponse` in `agentConversationFixture.ts` - `oxfmt`, `eslint`, `oxlint --type-aware`, `pnpm typecheck:browser`: clean (pre-commit hook ran all four) ## Note for Comfy-Org#18700 Comfy-Org#18700 adds its own `browser_tests/tests/agent/agentDeleteAndClearWorkflow.spec.ts`. This is a **different file**, deliberately, so there is no add/add conflict on rebase. Its `clear_canvas` case is the one that passes vacuously in the table above; the two can coexist, or that case can be dropped in favour of this one. --------- Co-authored-by: Amp <amp@ampcode.com>
<!-- ccr-slack-attribution -->
Requested by Christian Byrne · Slack thread
Summary
Stacked on #18404. Adds a regression test that replays QA's exact staging report (Jo Zhang) for the agent delete-all/scope-race bug #18404 fixes: load one template, then ask the agent to replace it with a different template. On staging, the old template's nodes were never deleted — the new template's nodes just got inserted alongside them, which looked like "insert instead of replace".
Linear: PM-1500, PM-1504
Why not a Playwright agent-conversation replay
The task asked for a
browser_tests/Playwright replay of this scenario using.claude/skills/agent-integration-replay/SKILL.md's conventions. That skill's fixtures must be recorded against the real cloud agent's non-standalone stack (Postgres/Redis/doc host + a real model) perbrowser_tests/fixtures/data/agent/README.md— "Do not writegraph_opsby hand". That stack, a running ComfyUI backend, and Playwright's browsers are not reachable from this environment (notestcloud/local stack, noANTHROPIC_API_KEY, no~/.cache/ms-playwright), and none of the existing recorded fixtures underconversations/exercise a template load or a template replace (checked all 14:add_node/connect/set_widget/delete_nodeonly).So this PR proves the fix at the same layer #18404's own tests do —
EcsFollowerAdapter/AgentCrdtProjection, the exact code the race lives in — but shaped around the template-replace user story instead of a bare delete-all, inagentCrdtProjection.integration.test.tsnext to the existing "self-driven reconcile retry" block.What the test proves
buildLiveGraph()'s two wired nodes) catches up onto the live graph.graph.getNodeByIdnull for both its nodes, and gone from the node-data store) and the new template's node is actually present and registered — not just that something got added alongside the stale nodes.Verification
fix/agent-delete-all-scope-drop-retryforked from (pre-EcsFollowerAdapter/AgentCrdtProjectionretry):expect(graph.getNodeById(toNodeId(1))).toBeNull()fails — the old template's node is still live with nothing to remove it.pnpm exec vitest run src/workbench/extensions/agent/crdt/agentCrdtProjection.integration.test.ts→ 4 passed, 0 failed.pnpm exec eslinton the changed file: clean.pnpm exec oxfmt --check: clean.pnpm typecheck(full workspace, all sub-packages): exit 0.lint-staged(oxfmt, oxlint --type-aware, eslint, typecheck) ran clean on commit.Test plan
agentCrdtProjection.integration.test.ts, confirmed red against the pre-fix base and green on this stacked branch🤖 Generated with Claude Code
https://claude.ai/code/session_01UqyourCBfi97xzpBx2W7oY
Generated by Claude Code