Repository navigation
test: custom-node E2E regression suite - Core depth and Cloud breadth - #15225
Conversation
The overlay validator now owns 'tiers must include run' instead of deferring it to the vendored-file test, and the loader guard that makes T1 registration safe (run-enrolled rows always carry a workflow) gets its regression assertions.
The workflow-required rule fires only for run-enrolled rows, so the regression assertion has to mutate tiers too or it pins nothing.
rgthree's author will not support Vue Nodes 2.0, so the pack leaves the suite: its manifest row, curated workflow asset, geometry baseline, and every calibration entry (auto-run excludes, roundtrip allowlists, curated sink type) are removed, and the doc scale figures recount to 6 packs / 799 nodes. Parser fixtures and captured-incident records that merely name rgthree stay - they are historical fact, not manifest dependencies.
An earlier step (build, backend boot, or the suite crashing) can fail before Playwright writes its json; both gates now surface that plainly instead of a jq no-such-file error on an already-red job.
…riable node Run 30116507105 disproved the exact-equality bet its own comment made: on the identical runner image (ubuntu24/20260720.247) and pinned bundled Chromium, 3187 node widths jittered by up to 2e-4px between two ephemeral runners - hardware/session float the browser+image pin cannot reach. Per that comment's own 'if jitter is observed, that red justifies a tolerance' clause, diffGeometry now compares numeric values within GEOMETRY_EPSILON_PX (0.01px): ~47x above the observed 2e-4px noise ceiling, still failing on any shift >=1/100th px. The one real delta in that run, LoadAndResizeImage height 566->354 (it read 566 in the record and a compare run, 354 in another), is content- variable - its height follows the input-dir preview - so it joins the GEOMETRY_UNSTABLE_NODES ledger and leaves the baseline, same as the editor-race nodes. Docs and the pure spec updated to the tolerance.
Remove branch-added narration and what-not-why comments across the custom-node cloud prong; keep 11 one-line why-comments that each record a real cross-boundary or must-equal constraint. Comment-only: no code lines changed; typecheck, vitest, and the pure-spec suite stay green.
The cloud gate, geometry-record workflow, and smokeAuth helper read invented SMOKE_* secrets and a nonexistent CLOUD_BACKEND_URL, so the gate never opened. Use the actual CLOUD_TEST_EMAIL / CLOUD_TEST_PASSWORD and the confirmed staging/test Firebase project (dreamboothy-dev dev web key, served at testcloud.comfy.org). Wiring only: CI shows the stored credentials are still rejected by Firebase (INVALID_LOGIN_CREDENTIALS), so end-to-end sign-in stays blocked on valid cloud test credentials.
Add clientType: CLIENT_TYPE_WEB to the cloud test-user signInWithPassword call, matching Comfy-Org/cloud's mint-smoke-api-key firebaseSignIn (the reference implementation this suite's cloud auth mirrors). Cloud-only path; inert unless CUSTOM_NODES_ENV=cloud.
Revert the cloud custom-node suite to SMOKE_ACCOUNT_EMAIL / SMOKE_ACCOUNT_PASSWORD, the Cloud smoke user the plan specifies. The CLOUD_TEST_* secrets are unrelated orphans from a removed workflow and are left untouched.
…n gaps Panel review of the cloud prong found a live credential leak and two gates that could pass without checking what they exist to check. What - turn Playwright tracing off under CUSTOM_NODES_ENV=cloud: the seeded Firebase record rides page.evaluate arguments into the trace, which CI uploads as a public artifact, so a long-lived refresh token would ship with the first failing cloud run - fail the forbid-skips step closed when the suite succeeded but wrote no results json; previously the skip gate silently no-opped and the job went green - pin the geometry tolerance between 0.009px and 0.011px, so widening it fails a test instead of silently loosening the tier - let preview:cloud-e2e own DEV_SERVER_COMFYUI_URL, removing a path where a dropped workflow env silently proxied to stagingcloud - correct ARCHITECTURE G15, which still promised exact comparison and a two-entry ledger, and widen the ledger criterion to cover content-variable layout across all six sites that state it - name the Firebase project in the sign-in failure, record the LoadAndResizeImage observation inline, and mark the two DETECTION_PROOF rows whose offender pack left the corpus Why - the leak becomes real the moment valid smoke credentials land - a gate that cannot run must not report success
The trace fix that stops a seeded cloud session reaching a public CI artifact was a single untested ternary; a rename or config restructure would have silently reinstated the leak. What - add scripts/playwright-cloud-trace.test.ts: asserts the custom-nodes project traces 'off' under CUSTOM_NODES_ENV=cloud and 'retain-on-failure' otherwise - relabel the G15 evidence from '3187 node widths' to measured geometry values; the corpus holds 1592 widths, so the noun could not be right - widen the geometry triage list from three causes to four, the last site still routing a content-variable delta to 'fix the code' - stop claiming the cloud and core report artifacts are identical; the cloud report ships without traces - correct the README bullet that described playwright.chrome.config.ts as pinning trace off when it retains on failure - disclose that DETECTION_PROOF row 11 names a node since ledgered and excluded from measurement Why - the highest-consequence invariant in this change set was the only one without a regression guard
The trace kill switch is set per Playwright project, but the seeding is keyed on CUSTOM_NODES_ENV and fires from the shared fixture. Running CUSTOM_NODES_ENV=cloud without --project lands 78 pure specs in the chromium project, which inherits retain-on-failure and would write a live refresh token into a local trace. CI always passes --project=custom-nodes, so this closes the local path. What - throw from the cloud branch of the comfyPage fixture when the running project does not trace 'off', naming the project and the flag to pass - pin the trace guard test to jsdom; under the repo's happy-dom default the @playwright/test import dumps an unhandled ECONNREFUSED on every unit run, reproduced on the pinned Node 25 - describe playwright.chrome.config.ts by what it actually buys (system Chrome, no bundled download); its trace never reaches this project - name the trace mode the custom-nodes project really uses in the core gate comment - four geometry causes resolve to three actions Why - a guard keyed on a different axis than the thing it guards is not a guard
testInfo.project.use.trace misses a spec-level test.use({ trace }), which
turns tracing on without changing the project value - the guard would
read 'off' and seed a live refresh token into a trace that was actually
recording. The built-in trace option resolves config, project, CLI and
test.use in one read, and matches the value Playwright itself feeds the
trace recorder.
Also handles the documented object form ({ mode: 'off' }), which the
previous string compare would have false-thrown on, and names --trace in
the remedy.
The epsilon change routed numbers to a new branch and left the exact-equality leg without a test: replacing it with 'return null' kept the whole pure suite green. That leg carries widgets[].name, the only cross-run widget rename detector the tier has - the in-run checks compare before and after within one run, so a rename at identical coordinates moves both sides together. Mutation-checked: the new case fails on 'return null', passes on clean code.
The cloud gate section enumerated two cloud-specific differences and omitted the third: tracing is off under CUSTOM_NODES_ENV=cloud so the seeded smoke session cannot ride the public report artifact. That is the one invariant here whose failure leaks a credential instead of reddening a test, and it changes how a cloud red is triaged.
The block comment still described the pre-rename secret set; the step below it reads SMOKE_ACCOUNT_EMAIL and SMOKE_ACCOUNT_PASSWORD only, and the two names it implied no longer exist anywhere in the repo.
Last stale pack count from the rgthree removal; the manifest has held 6 rows since 83e389f.
Add DETECTION_PROOF rows 12 and 13. Both tiers shipped without a falsification row, so neither had been proven to catch its class. Each red was captured in ISOLATION - one break on a branch off the shipped state - because PR #13534 carries ten live breaks and two of them (rows 4, 5) sabotage connection validity, which would make an autogrow red unattributable. - row 12 (S11): dropping ComfyUI-KJNodes from the extension import list reds the load tier's expectedExtensions assert (run 30292373406) - row 13 (S12): suppressing the live-connect onConnectionsChange call reds the dynamic-inputs tier (run 30292391836)
The cloud prong was waiting on an authenticated /object_info probe. Comfy-Org/cloud already checks the snapshot in at services/ingest/data/object_info.json (3573 nodes, 88 pack modules, newer than supported_nodes.yaml), so no credential is needed to calibrate. What - generate customNodeManifest.cloud.json: 87 pack rows, 2778 expected nodes, 14 core-disabled nodes - record yaml packs the snapshot has no nodes for instead of throwing on them; one dud pack must not block the other 87. The list ships in the manifest and is asserted against the LIVE backend, so a pack that starts registering nodes reds rather than silently losing coverage - keep failing closed when more packs go unjoined than joined - that is the dirname mapping breaking, not a few empty packs - fix the curated overlay key: Cloud installs VideoHelperSuite lowercase, and overlay keys must be snapshot dirnames - invert the two pre-calibration assertions now that the manifest exists Why - comfyui-fl-path-animator is declared supported but registers zero nodes: absent from the snapshot, from Cloud's own node_pack_mapping, and from /api/extensions
The per-row count assert catches drift inside a known pack, but the suite only iterates manifest rows - a pack the backend registers that no row covers lands with zero coverage while every assertion stays green. Cloud deploys packs on its own cadence, so that gap is reachable. Close the set both ways: the live custom-node pack set must equal the manifest's, or the gate reds naming the uncovered packs. Packs under custom_nodes/ that are not community packs are ledgered by mechanism rather than hardcoded past: ComfyUI_devtools is the harness, and websocket_image_save ships inside ComfyUI core's own custom_nodes/ (verified in comfyanonymous/ComfyUI), so neither carries a row. Dry-run against the real data: cloud live 87 == manifest 87; core installs exactly its 6 rows plus devtools.
Every generated cloud row shipped expectedExtensions: [], which the schema defines as 'this pack ships no frontend JS'. That is false for the packs Cloud serves extensions for, so the extension-load assert had nothing to check on the cloud prong. Derive the real names without credentials: /api/extensions is reachable unauthenticated, and so are the extension files it lists, so the registered name can be read out of each registerExtension call. One sentinel per pack, not the full set - the assert only has to prove the pack's JS loaded, and a conditionally-registered extra would red a healthy run. 17 packs calibrated. Names that could not be confirmed as registerExtension names are omitted rather than guessed (LTXDirector, makadi_iTools, vewd, 'Basic data handling: dynamic input'); a wrong sentinel reds the gate forever. Cross-check: the derived name for videohelpersuite is VideoHelperSuite.Core, byte-identical to the core manifest's independently calibrated value.
What - pre-push exits early when every pushed ref is a deletion; there is no source state to check, so knip only added latency to 'git push --delete' (and forced a HUSKY=0 bypass to delete a branch) - calibrate four more cloud packs' expectedExtensions, taking the count from 17 to 21 of 87 Why - the earlier four were dropped on a 'name contains a dot' heuristic, which is a naming convention rather than evidence; all four are genuine registerExtension names, confirmed at their call sites in the JS Cloud serves Notes - knip still runs on every real push, on a mixed delete+update push, and canonically in CI (.github/actions/lint-format-verify)
The field ships in the manifest and the live assert reads it, but assertCloudManifestShape never checked it, so a hand-edited or stale manifest could carry a missing or malformed list and the assert would silently check nothing - the exact silent-coverage-loss the field exists to prevent. Require it: an array of unique non-empty strings, where an empty array is the deliberate 'every yaml pack joined' declaration. Also pin that a well-formed top level still validates each row, so the per-entry deployRef error cannot get masked by the shape check.
Verified against run 30297797189: row 12 reproduces verbatim here (8 occurrences), but row 13 does not. The dynamic-inputs tier still reds, at an earlier precondition - 'consumer instantiates with at least one input' - because row 1's break leaves the consumer with zero inputs, so the test dies before reaching the autogrow assert. The break is still proven: run 30292391836 captured row 13's message in isolation. What was wrong was implying a reader could reproduce that exact text from this PR's HEAD. Same reason rows 12 and 13 were captured on isolated branches at all - rows 4 and 5 sabotage connection validity and slot hit-testing, so an autogrow red taken here could not be attributed to row 13's break.
Core rows calibrate expectedExtensions by grepping the pinned source. Cloud packs are never checked out locally, so the 21 sentinels were read off the /extensions files Cloud serves unauthenticated, taking the name from each registerExtension call site. That call site proves the name exists; it does NOT prove the extension registers unconditionally at boot, which is what the assert requires. Document the failure mode for the first cloud burn-in so a conditional sentinel gets swapped rather than the assert deleted or the list emptied - [] declares 'this pack ships no frontend JS', which would be false for these packs.
Run 30309274120, the first real cloud run, failed all 359 tests on one call: setupUser's node-side GET /api/users carries no bearer, and cloud rejects it before any test starts. Same class for the devtools settings POST, silently no-oping in its try/catch. What - skip the local multi-user registry on cloud; the smoke account is the user, the worker username stands in as the id - skip the node-side devtools settings POST on cloud; apply the same startup settings through the booted app's authenticated session Why - /api/users and /api/devtools/set_settings sit behind the cloud session, which only the browser holds; a Playwright request context is same-origin but carries no Authorization header
What - scripts/run-custom-nodes-local.mjs: boots the core backend (venv auto-detect, video asset staged) + dev server, or the cloud build + preview, waits on real readiness endpoints, runs the suite, tears down only what it started - test:custom-nodes:local and test:custom-nodes:local:cloud package scripts; extra args pass through to Playwright - README Running section leads with the orchestrators Why - the manual recipes were three terminals of commands in a fixed order that nobody remembers; already-running services are reused rather than killed, so the fast iteration path stays fast
…-e2e-suite # Conflicts: # browser_tests/fixtures/ComfyPage.ts
The bespoke 180-line launcher script reimplemented what start-server-and-test already does (start, wait on readiness URLs, reuse already-running services, tear down). Replaced with chained package scripts; the backend launcher is one sh line with venv auto-detect and the video asset staged.
What - outputHashes.ts: PNG IDAT-only sha256 (ComfyUI embeds prompt metadata in tEXt, so whole-file hashes false-fail on identical pixels), strict compare naming node + both digests; missing, changed, and NEW outputs all fail - T1 hook: after the sink-payload proof, fetch each image via /api/view and compare against curatedOutputHashes.core.json; unenrolled curated workflows fail with the record remedy - no silent opt-out - record mode (RECORD_OUTPUT_HASHES=1) writes the observed table and deliberately fails each test, per the CN_GEOMETRY record convention - record workflow gains the S15 step + artifact on the same backend boot - core only: the fetch is node-side, which on cloud carries no session (run 30309274120); cloud enrolls when an in-app fetch lands Why - execution_success proves a workflow ran; nothing proved it produced the same pixels - a serialization regression could silently change outputs. Also raises the cloud gate timeout to 150: run 30312432225 proved the full manifest cannot finish in 60 unsharded
|
Head
The preceding Core red was isolated to |
This comment has been minimized.
This comment has been minimized.
Nightly schedule and manual dispatch only. PRs neither trigger the workflow nor wait on any of its checks, and nothing is required in branch protection. The path-gate job and fork-PR policy skip existed only to make a pull_request trigger required-safe, so they go with it. Dispatch with the branch input to test a PR branch.
comfyui_essentials@1.1.0 failed to download during a cdn.comfy.org outage that outlived three 2s retries, then fetched cleanly minutes later at the same URL. Retry all errors with backoff for up to 60s and drop partial files so a resumed retry cannot unzip a truncated artifact.
main added an eager isCloud import to StorageKeys, which the custom-node fixtures import in Playwright's Node collection process. Vite replaces __DISTRIBUTION__ in browser bundles but nothing defined it under Node, so every spec aborted before collection. Mirror vite.config.mts's derivation at config load, the same way vitest.setup.ts already does.
DrJKL
left a comment
There was a problem hiding this comment.
Requesting changes. A parallel correctness/security/CI review plus a thermo-nuclear maintainability pass found blocking issues that should be addressed before this becomes a long-lived gate.
The highest-risk problems are the PR-controlled reusable workflow inheriting repository secrets, persisted checkout credentials remaining available while third-party Python executes, non-reproducible manifest generation, and run-event attribution that can produce false passes. Structurally, this adds a 1,863-line spec framework, a 751-line shell workflow, 4,126 lines of mostly-default interaction snapshots, and cast-first boundary validation. These are expensive foundations to normalize.
The test corpus also conflicts with the repository's own guidance: unit logic belongs in Vitest, not Playwright (AGENTS.md); specs should contain tests/hooks rather than helper frameworks (browser_tests/README.md); inline test types/casts are forbidden (README); and mutation assertions must poll (README). The 16 *.pure.spec.ts files contain 142 tests/3,178 lines; 134 run without a browser and should be Vitest tests. The fake Page/response systems, literal-ledger change detectors, and repeated pairing permutations make the suite test its scaffolding rather than behavior.
The focused product regressions for empty audio and unset image values are good and should remain. Verification: browser and script typechecks passed; focused Vitest passed 27/27; script tests passed 13/13. Of the pure Playwright tests, 134 passed and the remaining 8 could not launch because this environment lacks /opt/google/chrome/chrome. Green CI does not resolve the design and correctness findings below.
| source_pr: ${{ format('{0}', github.event.pull_request.number) }} | ||
| source_run_id: ${{ format('{0}', github.run_id) }} | ||
| source_sha: ${{ github.event.pull_request.head.sha || github.sha }} | ||
| secrets: inherit |
There was a problem hiding this comment.
Blocking — remove this PR-context secret handoff. For same-repository PRs, the local reusable workflow is loaded from the PR commit, so PR-authored workflow code receives every inherited repository secret. The fork check does not protect same-repo branches. Move coverage upload to a trusted default-branch workflow_run path, or otherwise call only immutable trusted code without inherited secrets.
There was a problem hiding this comment.
Addressed in #15590: the coverage workflow no longer accepts workflow_call or inherited secrets. It runs only from the trusted default-branch workflow_run path and excludes fork PRs.
| echo "SHARD_SLUG=${CUSTOM_NODES_MANIFEST}-${CUSTOM_NODES_SHARD/\//-of-}" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Checkout repository | ||
| uses: actions/checkout@v7 |
There was a problem hiding this comment.
Set persist-credentials: false. This checkout leaves the repository token in Git config while the later install step fetches and executes third-party custom-node setup/Python code. That untrusted code must not have access to checkout credentials.
There was a problem hiding this comment.
Addressed in #15590: the frontend checkout now sets persist-credentials: false before any third-party custom-node code can execute.
| uses: actions/checkout@v7 | ||
| with: | ||
| repository: 'comfyanonymous/ComfyUI' | ||
| ref: ${{ inputs.comfyui_ref }} |
There was a problem hiding this comment.
This second checkout also needs persist-credentials: false: its credential helper remains reachable when arbitrary custom-node installers and imports execute. Remove both persisted tokens before running third-party code.
There was a problem hiding this comment.
Addressed in #15590: the ComfyUI checkout now sets persist-credentials: false as well.
| const currentValue: unknown = JSON.parse(readFileSync(manifestPath, 'utf8')) | ||
| const current = assertCloudManifestShape(currentValue, manifestPath) | ||
| const snapshotValue: unknown = JSON.parse(readFileSync(snapshotPath, 'utf8')) | ||
| const { overlay, sentinels, cannotRunAlone } = regenerationSidecars(current) |
There was a problem hiding this comment.
The generated manifest is feeding authored policy back into its own regeneration. Overlays, sentinels, and calibration are read from the previous output and then overwritten, so a clean regeneration is not reproducible and deleting/corrupting the artifact loses source data. Keep authored policy in explicit sidecars and generate the manifest solely from source inputs.
There was a problem hiding this comment.
Addressed in #15590: authored policy is now stored in curatedWorkflows.json, extensionSentinels.json, and cannotRunAlone.json. Manifest generation no longer reads policy back from its own generated output.
| // `executing` strings - stay, and graph membership still vets them); | ||
| // otherwise the legacy seen-set exclusion. | ||
| capturedPromptId !== undefined | ||
| ? event.prompt_id === undefined || event.prompt_id === capturedPromptId |
There was a problem hiding this comment.
This retains every prompt-less executing event, including late/concurrent events from another run. classifyRun only applies graph membership to error attribution, so reused node IDs can satisfy expected execution and yield a false PASS. Scope all event kinds to the active attempt/graph and add a regression with a late prompt-less event.
There was a problem hiding this comment.
Addressed in #15590 with a pure prompt-event scope boundary and regression test. Prompt-less events are accepted only inside the captured prompt execution window; a foreign execution_start closes attribution even when node IDs are reused.
| } | ||
| } | ||
| return { | ||
| page: page as unknown as Page, |
There was a problem hiding this comment.
Do not build a fake Playwright runtime and force it through as unknown as Page. This hand-implements evaluate, response listeners, and request/response objects (the same pattern appears in promptError.pure.spec.ts), coupling tests to a mock framework we do not own. Extract typed pure transitions for unit tests and cover the lifecycle once with a real Playwright page. This also violates the cast rule in the test guidance.
There was a problem hiding this comment.
Addressed in #15590: the fake Page/Response runtime suites are removed. Prompt submission and response classification are pure typed boundaries with Vitest tests; the remaining browser test uses real Playwright objects.
|
|
||
| test.use({ initialSettings: customNodeSuiteSettings }) | ||
|
|
||
| interface ProbePlan { |
There was a problem hiding this comment.
This spec starts by declaring copied domain types, constants, and helper algorithms; the same architecture appears in allNodes, connectivity, and dynamicInputs. Specs should contain imports, Playwright configuration/hooks, and tests—not implementation frameworks (guidance). Move the plan/producers into typed fixture modules and import the real domain types rather than inline declarations.
There was a problem hiding this comment.
Addressed in #15590: interaction probing and profile comparison helpers moved out of the spec into interactionProbe.ts with direct Vitest coverage.
| }, | ||
| [String(consumer.id), String(inIndex)] as const | ||
| ) | ||
| expect(linked, `${key} with VueNodes=${vueNodesEnabled}`).toBe(true) |
There was a problem hiding this comment.
This snapshots graph state immediately after a drag, which is explicitly listed as a flake pattern in the Playwright guidance. Use await expect.poll(() => page.evaluate(...)).toBe(true). The same snapshot-after-mutation issue exists in customNode.regression.spec.ts and coreSmoke.spec.ts.
There was a problem hiding this comment.
Addressed in #15590: post-drag graph mutations in the affected connectivity and smoke tests now use expect.poll rather than immediate snapshots.
|
|
||
| test.describe('cloud roundtrip expectations', () => { | ||
| test('pins only the observed widget indices', () => { | ||
| expect(ROUNDTRIP_VALUE_ALLOWED_INDICES_LITEGRAPH).toMatchObject({ |
There was a problem hiding this comment.
These assertions lock literal ledger contents rather than consumer behavior; the following test does the same for a specific temporary entry, and consoleErrorLedger.pure.spec.ts repeats the pattern. Repository guidance explicitly forbids change-detector/default-value tests. Test schema and observable classification behavior, not today's exact calibration list.
There was a problem hiding this comment.
Addressed in #15590: literal ledger-content assertions were replaced by synthetic schema, stale-entry, and behavioral tests. Authored contents can evolve without rewriting change-detector tests.
| ]) | ||
| }) | ||
|
|
||
| test('COMBO slots with different vocabularies stay excluded', () => { |
There was a problem hiding this comment.
This 424-line unit suite repeats large node definitions and near-identical COMBO/output-name permutations. Make the meaningful vocabulary/schema cases table-driven and share one ledger-integrity contract rather than duplicating exhaustive fixtures across suites. The testing guidance requires parsimonious, composable behavioral tests—not another maintenance surface proportional to manifest size.
There was a problem hiding this comment.
Addressed in #15590: type-pairing cases are table-driven, preserving the behavioral cases while removing repetitive setup and assertions.
The custom-node console ledger (#15225, landed on main after this branch was cut) requires the ColorOverlay duplicate-registration error to be observed at startup: requiredStartupId: 'duplicate-color-overlay' pattern: /\[vite:preloadError\].*Extension named 'ColorOverlay' already registered\./ That error was the throw this PR removes. Registering a duplicate name now warns and skips, so dz_node_palette.js finishes evaluating and emits no preloadError - the pattern can never match again, since the message it matches no longer exists in the frontend. Left in place, staleRequiredStartupErrorRulesForPacks would report ComfyUI_LayerStyle_Advance/duplicate-color-overlay and fail the nightly suite's stale-rule assertion. Removing the entry is the ledger's documented lifecycle for a mechanism that stops firing, and it tightens rather than loosens the gate: the pack no longer carries a suppression, so a duplicate-registration error reappearing would surface as unallowlisted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The custom-node console ledger (#15225, landed on main after this branch was cut) requires the ColorOverlay duplicate-registration error to be observed at startup: requiredStartupId: 'duplicate-color-overlay' pattern: /\[vite:preloadError\].*Extension named 'ColorOverlay' already registered\./ That error was the throw this PR removes. Registering a duplicate name now warns and skips, so dz_node_palette.js finishes evaluating and emits no preloadError - the pattern can never match again, since the message it matches no longer exists in the frontend. Left in place, staleRequiredStartupErrorRulesForPacks would report ComfyUI_LayerStyle_Advance/duplicate-color-overlay and fail the nightly suite's stale-rule assertion. Removing the entry is the ledger's documented lifecycle for a mechanism that stops firing, and it tightens rather than loosens the gate: the pack no longer carries a suppression, so a duplicate-registration error reappearing would surface as unallowlisted.
## Summary Makes the custom-node browser suite and dual-renderer ecosystem matrix fail-closed pull-request and merge-queue gates. ## Changes - Adds `pull_request` and `merge_group` triggers without path filters. - Exposes stable aggregate checks named `E2E Custom Nodes Test` and `Custom Nodes Ecosystem Matrix test`. - Keeps retries disabled and fails on failed, skipped, flaky, missing, mixed-renderer, or incomplete results. - Runs the ecosystem corpus in legacy and Nodes 2.0 modes with independent detection proofs. - Preserves the custom-node S1/S2/S3/S9/S15 detection proofs on scheduled and manual proof runs. - Removes persisted checkout credentials before third-party pack setup and prevents PR cache publication. - Includes the reviewed ECS compatibility, malformed registry-name, broken preview-image, S3 readiness, and exclusion-report fixes needed for a green gate. Follow-up to Comfy-Org#15225. ## Validation - 213 focused unit tests pass across the custom-node harness, matrix runner, ECS widget compatibility, registry guard, preview renderer, and result reporters. - Frontend, browser, and scripts typechecks pass. - Lint, format, Knip, YAML parsing, and exact Playwright collection pass. - Exact-head gate runs: pending on commit `336548bd43`. --------- Co-authored-by: Alexander Brown <drjkl@comfy.org>
## Summary Makes the custom-node browser suite and dual-renderer ecosystem matrix fail-closed pull-request and merge-queue gates. ## Changes - Adds `pull_request` and `merge_group` triggers without path filters. - Exposes stable aggregate checks named `E2E Custom Nodes Test` and `Custom Nodes Ecosystem Matrix test`. - Keeps retries disabled and fails on failed, skipped, flaky, missing, mixed-renderer, or incomplete results. - Runs the ecosystem corpus in legacy and Nodes 2.0 modes with independent detection proofs. - Preserves the custom-node S1/S2/S3/S9/S15 detection proofs on scheduled and manual proof runs. - Removes persisted checkout credentials before third-party pack setup and prevents PR cache publication. - Includes the reviewed ECS compatibility, malformed registry-name, broken preview-image, S3 readiness, and exclusion-report fixes needed for a green gate. Follow-up to #15225. ## Validation - 213 focused unit tests pass across the custom-node harness, matrix runner, ECS widget compatibility, registry guard, preview renderer, and result reporters. - Frontend, browser, and scripts typechecks pass. - Lint, format, Knip, YAML parsing, and exact Playwright collection pass. - Exact-head gate runs: pending on commit `336548bd43`. --------- Co-authored-by: Alexander Brown <drjkl@comfy.org>
Main's Codecov upload step already sets override_branch, override_commit and override_pr (#15225, hardened for forks in #16404). Keeping this branch's copies would produce duplicate keys in the step's `with:` map, which GitHub Actions rejects as an invalid workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Adds a local-backend custom-node E2E suite with two complementary populations. Not a PR gate yet: the workflow runs on a nightly schedule and manual dispatch only - PRs neither trigger it nor wait on any of its checks, and none of its checks are required in branch protection. Within a run it fails closed (an install failure or skipped tier is red).
S15 is restored inside the six Core curated workflow tests that already execute. It adds output comparison, not another prompt or Playwright test, so its incremental runtime is negligible.
Changes
Tier coverage and applicability
loadloadloadrunrowsrunrowThe suite contains no
test.skiportest.fixme, uses one worker and--retries=0, and independently rejects Playwright-reported skips or flaky results.Explicit coverage debt
comfyui-fl-path-animatordoes not join the pinned Cloud snapshot.Imathrequirement. Their upstream fixes are tracked by FE-1660.comfyui-itools@0.6.8is a banned registry artifact whoseiToolsCropImagehook has two terminal race outcomes under the same pin. Only that node is excluded from S1-S8; the pack count remains exact and its other 21 nodes run. Restoration is FE-1675.VHS_SelectLatestrequires the pack-owned prompt transformation and is the one model-free node not executed by Cloud S9. Restoration is FE-1661.was-node-suite-comfyui/Text Random Promptperforms an unbounded public Lexica API request and is excluded only from Core S9. Deterministic restoration is FE-1682.Review focus
Supersedes #13389 and #15200. Existing review follow-ups remain tracked in FE-1611.