Skip to content

test(agent): cover template-replace scenario for agent delete-all race - #18702

Closed
claude[bot] wants to merge 3 commits into
fix/agent-delete-all-scope-drop-retryfrom
test/agent-template-replace-delete-race
Closed

claude[bot] wants to merge 3 commits into
fix/agent-delete-all-scope-drop-retryfrom
test/agent-template-replace-delete-race

Conversation

@claude

@claude claude Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

<!-- ccr-slack-attribution -->
Requested by Christian Byrne · Slack thread

Summary

Stacked on #18404. Adds a regression test that replays QA's exact staging report (Jo Zhang) for the agent delete-all/scope-race bug #18404 fixes: load one template, then ask the agent to replace it with a different template. On staging, the old template's nodes were never deleted — the new template's nodes just got inserted alongside them, which looked like "insert instead of replace".

Linear: PM-1500, PM-1504

Why not a Playwright agent-conversation replay

The task asked for a browser_tests/ Playwright replay of this scenario using .claude/skills/agent-integration-replay/SKILL.md's conventions. That skill's fixtures must be recorded against the real cloud agent's non-standalone stack (Postgres/Redis/doc host + a real model) per browser_tests/fixtures/data/agent/README.md — "Do not write graph_ops by hand". That stack, a running ComfyUI backend, and Playwright's browsers are not reachable from this environment (no testcloud/local stack, no ANTHROPIC_API_KEY, no ~/.cache/ms-playwright), and none of the existing recorded fixtures under conversations/ exercise a template load or a template replace (checked all 14: add_node/connect/set_widget/delete_node only).

So this PR proves the fix at the same layer #18404's own tests do — EcsFollowerAdapter/AgentCrdtProjection, the exact code the race lives in — but shaped around the template-replace user story instead of a bare delete-all, in agentCrdtProjection.integration.test.ts next to the existing "self-driven reconcile retry" block.

What the test proves

  • Turn 1: the old template ("NetaYume Lumina Text to Image" stood in by buildLiveGraph()'s two wired nodes) catches up onto the live graph.
  • Turn 2 ("replace that with a different template"): one CRDT frame both clears the old template and adds the new template's node — the agent's "replace" landing as one batch. Scope briefly can't resolve the bound workflow tab (the tab-binding race fix: agent delete-all doesn't clear canvas until next message #18404 fixes), so the whole batch is rejected and nothing is swept.
  • No further frame ever arrives (the user doesn't send another message) — confirmed red against the pre-fix code: the old template's node stays live forever with nothing to recover it.
  • The self-driven retry (fix: agent delete-all doesn't clear canvas until next message #18404) replays the dropped frame once scope recovers. Asserts the old template is actually gone (graph.getNodeById null for both its nodes, and gone from the node-data store) and the new template's node is actually present and registered — not just that something got added alongside the stale nodes.

Verification

  • Red, confirmed against the pre-fix base. Temporarily built this test against the commit fix/agent-delete-all-scope-drop-retry forked from (pre-EcsFollowerAdapter/AgentCrdtProjection retry): expect(graph.getNodeById(toNodeId(1))).toBeNull() fails — the old template's node is still live with nothing to remove it.
  • Green on this branch (stacked on fix: agent delete-all doesn't clear canvas until next message #18404's fix): pnpm exec vitest run src/workbench/extensions/agent/crdt/agentCrdtProjection.integration.test.ts → 4 passed, 0 failed.
  • pnpm exec eslint on the changed file: clean.
  • pnpm exec oxfmt --check: clean.
  • pnpm typecheck (full workspace, all sub-packages): exit 0.
  • Pre-commit lint-staged (oxfmt, oxlint --type-aware, eslint, typecheck) ran clean on commit.

Test plan

  • New regression test in agentCrdtProjection.integration.test.ts, confirmed red against the pre-fix base and green on this stacked branch
  • Existing tests in the same file still pass (4/4)
  • Lint, format, typecheck clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01UqyourCBfi97xzpBx2W7oY


Generated by Claude Code

Adds a regression test replaying Jo Zhang's exact staging report: load
one template, then ask the agent to replace it with a different one.
Before this PR's fix, the old template's nodes were never deleted and
stayed on screen next to the new template's nodes forever (proven red
against the pre-fix EcsFollowerAdapter/AgentCrdtProjection). With the
fix, the self-driven reconcile retry replays the dropped replace once
scope recovers, without any further frame arriving, and the old
template is actually removed while the new one is actually added.

Lands next to the existing "self-driven reconcile retry" describe
block in agentCrdtProjection.integration.test.ts, driven through the
same adapter/projection chain, since a true Playwright agent-conversation
replay of this scenario needs a recorded fixture from the cloud agent's
non-standalone stack (Postgres/Redis/doc host + a real model), which is
not reachable from this environment.
@coderabbitai

coderabbitai Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: Comfy-Org/ComfyUI_frontend/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 8118628f-2651-47b8-9b52-094c3062bd74

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot added the cursor-review Trigger multi-model Cursor agent review on this PR label Sep 24, 2026
@claude
claude Bot requested a review from christian-byrne September 24, 2026 08:32
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Playwright: ✅ 2117 passed, 0 failed

📊 Browser Reports
  • chromium: View Report (✅ 2096 / ❌ 0 / ⚠️ 0 / ⏭️ 13)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

🎨 Storybook: ✅ Built — View Storybook

Details

⏰ Completed at: 09/24/2026, 08:18:18 PM UTC

Links

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since 255dbbbf3cda1103c861864c5b68002f8745ad9b

Full docs: ADR-DEVEX-LINT-0015

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

This candidate changes nothing the matrix measures, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

📦 Bundle Size

⏳ Size data collection in progress…

@github-actions github-actions Bot added the risk:high PR risk grade (advisory shadow check; grader-owned) label Sep 24, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Cursor Review — Consolidated panel

Triggered by @christian-byrne.

Found 5 finding(s).

Severity Count
🟡 Medium 1
🟢 Low 3
⚪ Nit 1

Panel: 8/8 reviewers contributed findings.

}
} satisfies Op
applyOps(host, [insertOp], CATALOG)
scopeAvailable = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — Injecting the failure by flipping scopeAvailable makes createGraphMutations().batch() bail on its getScope() check before the projection callback runs, so the incremental delete+add path a real replace turn takes is never exercised; the retry then re-reads the doc through the full-reconcile branch, which is convergent by construction and also prepares node 3 via reconcileNode rather than addNode. As written this test cannot fail for the PM-1500/PM-1504 "insert instead of replace" defect it is meant to pin — inject the rejection downstream of the scope check so the replace batch itself is what gets replayed. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed real, and confirmed by tracing the actual code (not just re-reading): createGraphMutations().batch() calls getScope() and returns false before it ever invokes the define() callback that builds the incremental ops. So flipping scopeAvailable meant the rejected "replace" frame never reached ecsFollowerAdapter's nodeActions-driven delete+add builder at all. And since reconcileNextFrame is set unconditionally on any rejected batch, the self-driven retry always replays through the full-reconcile branch (reconcileNode, not addNode) — exactly as you described.

Fix: split into two tests. One now delivers the replace frame with scope resolved throughout, so it genuinely exercises the incremental delete+add path a successful replace turn takes. The other keeps the scope-race/retry scenario (now honestly commented as proving full-reconcile recovery, which is the actual #18404 fix — the retry not sweeping the live graph).

Verified with a mutation test: skipping the incremental delete in ecsFollowerAdapter.ts makes the new single-batch test fail red, while the old retry-based test still passes green — proving the original test gave zero coverage of that path, matching your finding exactly. Re-verified red/green: the retry test is red against the pre-fix commit (3a4b4a9, before the retry+live-sweep fix) and green on this branch; the new single-batch test passes on both (it doesn't depend on that particular fix, since the incremental path was already correct).

Pushed in ce277bd.

Generated by Claude Code

}

// Turn 1: the old template loads and catches up onto the live graph.
deliver(Y.encodeStateAsUpdate(host))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — Turn 1's deliver() result is discarded, unlike bindAndCatchUp elsewhere in this suite which asserts .toBe(true), and the two assertions that follow are already satisfied by buildLiveGraph() alone. If catch-up silently stopped applying, reconcileNextFrame would stay armed and turn 2 would take the full-reconcile branch instead of the incremental one, yet the test would still go green; assert expect(deliver(...)).toBe(true). Raised by 1 of 8 reviewers (claude-opus-5-thinking-max edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — turn 1's deliver(...) calls in both tests now assert .toBe(true). Pushed in ce277bd.

Generated by Claude Code

outputs: [{ name: 'image', type: 'IMAGE', links: [] }]
}
} satisfies Op
applyOps(host, [insertOp], CATALOG)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — applyOps's result is dropped here, unlike other call sites in this suite that assert outcomes[0].outcome === 'applied', and the op carries a hand-picked base_version: 10/stamp well ahead of the freshly minted doc's version. If the applier ever rejects or LWW-drops the insert, the scenario silently degrades to the plain delete-all case an earlier test already covers and the failure surfaces 20 lines later at the unrelated title assertion. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. applyOps(host, [insertOp], CATALOG)'s result is now captured and asserted: expect(applied.outcomes).toEqual([{ op_id: insertOp.op_id, outcome: 'applied' }]). Also fixed the base_version/stamp: instead of the hand-picked 10, it's now computed as appliedOpIds(host).length right before building the op, so it actually reflects the freshly-minted doc's applied-op count instead of an arbitrary number that happened not to collide. Pushed in ce277bd.

Generated by Claude Code

.getGraphNodesFor(scope.rootGraphId, scope.owningGraphId)
.map(({ id }) => id)
).toEqual([toNodeId(3)])
expect(layout.deleteNodes.mock.calls.flatMap(([, ids]) => ids)).toEqual(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low — expect.arrayContaining is an unordered superset check, so this still passes if the retry also churns node 3 through a delete, deletes unrelated ids, or issues duplicate deletes — weaker than the stated goal of proving "not that only half of the replace ever lands"; the commit path deletes exactly [1] then [2], so toEqual([toNodeId(1), toNodeId(2)]) is provable. The expect.anything() layout args on the createNode assertion below likewise never check where the replacement node lands, which is part of the reported symptom. Raised by 2 of 8 reviewers (claude-opus-5-thinking-max adversarial, claude-opus-5-thinking-max edge-case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed both. layout.deleteNodes.mock.calls.flatMap(([, ids]) => ids) now asserts toEqual([toNodeId(1), toNodeId(2)]) (exact array, not arrayContaining) in both tests. The layout.createNode assertion now checks the real layout payload instead of expect.anything(): { position: { x: 400, y: 20 }, size: { width: 180, height: 90 } }, matching where the replacement node (pos: [400, 20], size: [180, 90]) actually lands. Pushed in ce277bd.

Generated by Claude Code

expect.anything()
)

projection.destroy()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ Nit — The three destroy() calls sit inside the try after the assertions, so any assertion failure skips them and leaves the EcsFollowerAdapter session subscribed, a retry timer possibly pending, and two Y.Docs undisposed — compounding across each configured CI retry. Move them into the finally alongside vi.useRealTimers(). Raised by 1 of 8 reviewers (claude-opus-5-thinking-max adversarial).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — projection.destroy() / follower.destroy() / host.destroy() are now in a finally block (nested inside the vi.useFakeTimers() try/finally for the retry test) instead of trailing after the assertions. Pushed in ce277bd.

Generated by Claude Code

@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@                            Coverage Diff                            @@
##           fix/agent-delete-all-scope-drop-retry   #18702      +/-   ##
=========================================================================
- Coverage                                  84.36%   84.08%   -0.29%     
=========================================================================
  Files                                       2284     2294      +10     
  Lines                                     130580   134893    +4313     
  Branches                                   35504    36657    +1153     
=========================================================================
+ Hits                                      110170   113422    +3252     
- Misses                                     20114    21170    +1056     
- Partials                                     296      301       +5     
Flag Coverage Δ
unit 80.74% <ø> (+0.74%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 184 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

createGraphMutations().batch() checks getScope() before ever invoking the
define() callback that builds the incremental delete+add ops, so flipping
scopeAvailable to fail the batch meant the rejected "replace" frame never
reached that builder at all. The self-driven retry then always replays
through the full-reconcile branch (reconcileNode, not addNode), so the
original test could not fail on a defect specific to the incremental
delete+add composition a successful replace turn actually takes.

Split the scenario into two tests: one delivers the replace frame with
scope resolved throughout, proving the incremental path itself correctly
removes the old template and adds the new one; the other keeps the
scope-race + self-driven-retry scenario, now accurately commented as
proving full-reconcile recovery, with exact (not arrayContaining) node-id
assertions, a real position payload on the layout.createNode assertion
instead of expect.anything(), a computed base_version/stamp instead of a
hand-picked one, asserted applyOps/deliver outcomes instead of discarded
results, and destroy() calls moved into a finally block.

Verified: the new single-batch test fails when the incremental delete is
mutated out (confirming it actually exercises that path, unlike the
original), both tests are green on this branch, and the retry test alone
is red against the pre-fix commit (3a4b4a9, before the retry/live-sweep
fix) while the single-batch test passes there unmodified, since it does
not depend on that fix.
…template-replace-delete-race

# Conflicts:
#	src/workbench/extensions/agent/crdt/agentCrdtProjection.integration.test.ts
@christian-byrne

Copy link
Copy Markdown
Contributor

Closing as superseded after revalidating #18700. Commit 6ec9537 deletes agentCrdtProjection and moves delete-all through direct graph-API apply; the #18700 fail→pass ledger records the durable delete-all behavior as absorbed by agentDeleteAndClearWorkflow.spec.ts from #18404. The duplicate-insert/opaque-widget black-box symptom is also retained on main via merged #18703, so this projection-only carrier has no remaining coverage to preserve.

@DrJKL DrJKL added agent Comfy Agent product, integration, tests or product documentation agent-crdt Agent document synchronization, graph projection or CRDT operations agent-test-only Agent tests, fixtures and test support only; no production changes agent-test-unit Touches Agent-related unit/component tests or their support labels Sep 27, 2026
christian-byrne added a commit to christian-byrne/ComfyUI_frontend that referenced this pull request Sep 28, 2026
…on canvas (Comfy-Org#18969)

## Summary

Re-authors the regression guard from Comfy-Org#18702 — closed without merging —
as a black-box `browser_tests/` case, so the live bug it covered (ask
the agent to clear the canvas, the turn reports the tab is empty, the
nodes stay on screen — PM-1500, PM-1504) has a test again.

## Changes

- **What**: new
`browser_tests/tests/agent/agentClearedWorkflowStaysCleared.spec.ts`
(two cases) plus one `browser_tests/types/globals.d.ts` declaration.
Test-only; no production code touched.

## Why re-authored rather than reopened

Comfy-Org#18702's base was `fix/agent-delete-all-scope-drop-retry` (Comfy-Org#18404),
which is not landing, and it asserted against `AgentCrdtProjection` and
`EcsFollowerAdapter` — both deleted by Comfy-Org#18700. Rebasing it would have
restored a test onto a base that cannot merge, stated in code that is
being removed.

This states the same property where the user reads it: the rendered
canvas. Every API it touches was checked against Comfy-Org#18700 and survives —
`agentConversationTest` / `conversationCase`, `addedNodeIds`,
`runTurns`, `subscribeCount`, `VueNodeHelpers.nodes` / `getNodeLocator`,
`Topbar`, and the `agent-rec-clear-workflow` fixture. It asserts nothing
in `agentCrdtProjection`, `ecsFollowerAdapter`, `graphMutations`,
`agentNodeMaterializer` or `agentSubgraphHostSlots`, and `globals.d.ts`
is untouched by Comfy-Org#18700, so there is no conflict there either.

## Review Focus

**It closes a vacuous-pass hole, which is most of the point.** The
recorded turn adds one node and then clears the workflow, so asserting
only that the canvas ends empty cannot distinguish *the clear landed*
from *neither operation did*. That is the failure mode flagged as
PM-1755 — a recorded replay can skip a clear and still report a pass —
and it is real here, measured, not hypothetical:

| fault injected into the replay | `agentConversationReplay` (recorded
case) | `agentDeleteAndClearWorkflow` on Comfy-Org#18700 | this spec |
| --- | --- | --- | --- |
| none | pass | pass | pass |
| clear frame withheld from the client (the live bug) | **fail** | — |
**fail** |
| whole turn's graph ops withheld | pass | pass | **fail** |

The distinguishing assertion records every `data-node-id` the DOM
mounts, so it reads only what the user sees and does not care which
graph API removed the node.

**This lands green — it is a pin, not a live repro, and the PR does not
claim a fix is needed.** The mechanism Comfy-Org#18702 drove the bug through was
a mutation batch rejected because the app briefly could not resolve
which workflow tab the edit was bound to. Comfy-Org#18700 removes that mechanism
rather than fixing it: `AgentPanelRoot.vue` there has no `getScope` and
no `createGraphMutations`, and `graphMutations.ts` /
`ecsFollowerAdapter.ts` are deleted. So there is no surviving surface to
inject the race from, and Comfy-Org#18404's `dropWorkflowScope()` — which reached
into Pinia to delete `tab.activeState.id` — would be asserting against
the very thing being deleted. What is worth holding across that change
is the user-visible outcome, on both sides of it.

**Second case, tab round trip.** Returning to the tab re-subscribes the
follower, which replays the bound document over the tab's own snapshot;
the case waits for that subscribe before asserting, so it is about the
catch-up rather than about a canvas nothing has touched yet.
`agentConversationReplay` never switches tabs, so this was unpinned.

## Verification

Local, against a `DISTRIBUTION=cloud` dev server over a containerised
ComfyUI backend with devtools:

```
PLAYWRIGHT_LOCAL=1 DISTRIBUTION=cloud pnpm exec playwright test \
  browser_tests/tests/agent/agentClearedWorkflowStaysCleared.spec.ts --project=cloud
```

- `main` (9cb7f5b): 2 passed
- Comfy-Org#18700 (540a2b8), spec copied in unchanged: 2 passed
- red/green table above reproduced by temporary, uncommitted edits to
`replayResponse` in `agentConversationFixture.ts`
- `oxfmt`, `eslint`, `oxlint --type-aware`, `pnpm typecheck:browser`:
clean (pre-commit hook ran all four)

## Note for Comfy-Org#18700

Comfy-Org#18700 adds its own
`browser_tests/tests/agent/agentDeleteAndClearWorkflow.spec.ts`. This is
a **different file**, deliberately, so there is no add/add conflict on
rebase. Its `clear_canvas` case is the one that passes vacuously in the
table above; the two can coexist, or that case can be dropped in favour
of this one.

---------

Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Comfy Agent product, integration, tests or product documentation agent-crdt Agent document synchronization, graph projection or CRDT operations agent-test-only Agent tests, fixtures and test support only; no production changes agent-test-unit Touches Agent-related unit/component tests or their support cursor-review Trigger multi-model Cursor agent review on this PR risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants