Skip to content

test: custom-node E2E regression suite - Core depth and Cloud breadth - #15225

Merged
benjcooley merged 365 commits into
mainfrom
benjcooley/custom-node-e2e-landing
Aug 21, 2026
Merged

benjcooley merged 365 commits into
mainfrom
benjcooley/custom-node-e2e-landing

Conversation

@benjcooley

@benjcooley benjcooley commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a local-backend custom-node E2E suite with two complementary populations. Not a PR gate yet: the workflow runs on a nightly schedule and manual dispatch only - PRs neither trigger it nor wait on any of its checks, and none of its checks are required in branch protection. Within a run it fails closed (an install failure or skipped tier is red).

  • Core depth: the original six pinned packs retain S1-S13 and S15. S14 geometry snapshots remain removed by team decision.
  • Cloud breadth: the pinned Cloud snapshot has 87 joined manifest rows; 83 run across five fixed shards on a local CPU ComfyUI backend. One source row is unjoined and four rows are explicitly quarantined.

S15 is restored inside the six Core curated workflow tests that already execute. It adds output comparison, not another prompt or Playwright test, so its incremental runtime is negligible.

Changes

  • Pins ComfyUI core, pack sources, registry artifacts, staged inputs, worker count, retry count, and shard composition.
  • Keeps each cloud pack in a stable shared Python environment; changing a quarantine entry cannot reshuffle other packs.
  • Runs Core as its own matrix entry and Cloud as five weight-balanced breadth shards.
  • Fails on any Playwright failure, skip, flaky result, count mismatch, dirty backend teardown, or stale exact expectation.
  • Records visible errors for the page lifetime, so a transient toast that clears before the assertion still fails S7.
  • Reports every coverage exclusion in bold in the GitHub Actions summary with its mechanism and removal condition.
  • Includes two bounded frontend fixes surfaced by the suite: unset image-upload combos no longer request filename=undefined, and empty audio-upload sentinels no longer request a preview. These are the only live src behavior changes in this PR.
  • Provides deliberate-break proofs for S1, S2, S3, S9, and S15.

Tier coverage and applicability

Tier Core Cloud breadth Assertion
S1 6 packs rows declaring load Every enrolled registered node instantiates in LiteGraph with exact declared slot materialization.
S2 6 packs rows declaring load The enrolled S1 corpus mounts under Vue Nodes with its visible widgets and slots represented in the DOM.
S3 6 packs rows declaring load Enrolled node identity and type, initialized live widget topology, and serialized widget values survive save/reload; exact pinned pack divergences are two-way ledgered.
S4 retained retained Representative type-correct connections among enrolled nodes survive graph connection and round-trip validation.
S5 retained retained Curated anchor links and one materialized in-pack link per applicable pack use real drag/connection APIs in both renderers, excluding only explicitly reported nodes.
S6 retained retained Connectivity round-trips reach prompt conversion and validate the serialized edge contract.
S7 retained, strengthened retained, strengthened All user-visible error surfaces are sampled every animation frame from initial navigation; transient and final-state errors fail.
S8 retained retained Console errors and uncaught page errors are collected across startup and operations; only exact attributed signatures are accepted.
S9 all Core run rows VideoHelperSuite, the only Cloud run row Exact calibrated model-free corpora queue against the real backend and must execute or produce an observable output.
S10 retained retained Manifest shape, exact local node counts, registered-pack attribution, and collection counts are sentinels.
S11 retained retained where declared Expected frontend extensions and served web-directory assets must register.
S12 Impact case Impact case Dynamic list input grows and shrinks through programmatic and real drag connections in both renderers.
S13 6 pinned Core profiles not enrolled Existing Core interaction-delta profiles compare at the exact recorded pack refs. Cloud expansion is FE-1659.
S14 removed removed Team-approved removal of full node geometry/position/size snapshots.
S15 6 Core curated workflows not enrolled Deterministic sink payload hashes detect valid-but-wrong serialized output. Full-pack expansion is FE-1657.

The suite contains no test.skip or test.fixme, uses one worker and --retries=0, and independently rejects Playwright-reported skips or flaky results.

Explicit coverage debt

  • comfyui-fl-path-animator does not join the pinned Cloud snapshot.
  • LivePortraitKJ has an unfetchable SHA and radiance has an unsatisfiable Imath requirement. Their upstream fixes are tracked by FE-1660.
  • SeedVR2 and NVIDIA RTX register zero nodes on a CPU runner. GPU-backed restoration is FE-1658.
  • comfyui-itools@0.6.8 is a banned registry artifact whose iToolsCropImage hook has two terminal race outcomes under the same pin. Only that node is excluded from S1-S8; the pack count remains exact and its other 21 nodes run. Restoration is FE-1675.
  • VHS_SelectLatest requires the pack-owned prompt transformation and is the one model-free node not executed by Cloud S9. Restoration is FE-1661.
  • was-node-suite-comfyui/Text Random Prompt performs an unbounded public Lexica API request and is excluded only from Core S9. Deterministic restoration is FE-1682.
  • Exact known pack defects remain exercised under two-way stale ledgers; they are not skipped. A fixed or changed outcome fails until the expectation is removed or recalibrated with evidence.

Review focus

  • Whether the two disclosed preview guards are correct and appropriately scoped; all remaining changes are tests, fixtures, scripts, tooling, documentation, or CI.
  • Whether every tier claim above matches its assertion and applicability.
  • Whether each temporary exclusion is specific, visible, owned, and removable.
  • Whether exact expectation ledgers describe attributable pack behavior without weakening the asserted contract.
  • Whether representative-per-slot connectivity plus per-pack two-renderer drags is the right bounded surface; this does not claim a producer-by-consumer cross-product or cross-shard pairing.
  • Whether the fixed-shard dependency environment and manifest provenance are sufficiently deterministic.

Supersedes #13389 and #15200. Existing review follow-ups remain tracked in FE-1611.

The overlay validator now owns 'tiers must include run' instead of
deferring it to the vendored-file test, and the loader guard that
makes T1 registration safe (run-enrolled rows always carry a
workflow) gets its regression assertions.
The workflow-required rule fires only for run-enrolled rows, so the
regression assertion has to mutate tiers too or it pins nothing.
rgthree's author will not support Vue Nodes 2.0, so the pack leaves
the suite: its manifest row, curated workflow asset, geometry
baseline, and every calibration entry (auto-run excludes, roundtrip
allowlists, curated sink type) are removed, and the doc scale figures
recount to 6 packs / 799 nodes. Parser fixtures and captured-incident
records that merely name rgthree stay - they are historical fact, not
manifest dependencies.
An earlier step (build, backend boot, or the suite crashing) can fail
before Playwright writes its json; both gates now surface that plainly
instead of a jq no-such-file error on an already-red job.
…riable node

Run 30116507105 disproved the exact-equality bet its own comment made:
on the identical runner image (ubuntu24/20260720.247) and pinned
bundled Chromium, 3187 node widths jittered by up to 2e-4px between
two ephemeral runners - hardware/session float the browser+image pin
cannot reach. Per that comment's own 'if jitter is observed, that red
justifies a tolerance' clause, diffGeometry now compares numeric
values within GEOMETRY_EPSILON_PX (0.01px): ~47x above the observed
2e-4px noise ceiling, still failing on any shift >=1/100th px. The one
real delta in that run, LoadAndResizeImage height 566->354 (it read
566 in the record and a compare run, 354 in another), is content-
variable - its height follows the input-dir preview - so it joins the
GEOMETRY_UNSTABLE_NODES ledger and leaves the baseline, same as the
editor-race nodes. Docs and the pure spec updated to the tolerance.
Remove branch-added narration and what-not-why comments across the
custom-node cloud prong; keep 11 one-line why-comments that each
record a real cross-boundary or must-equal constraint.

Comment-only: no code lines changed; typecheck, vitest, and the
pure-spec suite stay green.
The cloud gate, geometry-record workflow, and smokeAuth helper read
invented SMOKE_* secrets and a nonexistent CLOUD_BACKEND_URL, so the gate
never opened. Use the actual CLOUD_TEST_EMAIL / CLOUD_TEST_PASSWORD and
the confirmed staging/test Firebase project (dreamboothy-dev dev web key,
served at testcloud.comfy.org).

Wiring only: CI shows the stored credentials are still rejected by
Firebase (INVALID_LOGIN_CREDENTIALS), so end-to-end sign-in stays blocked
on valid cloud test credentials.
Add clientType: CLIENT_TYPE_WEB to the cloud test-user signInWithPassword
call, matching Comfy-Org/cloud's mint-smoke-api-key firebaseSignIn (the
reference implementation this suite's cloud auth mirrors). Cloud-only path;
inert unless CUSTOM_NODES_ENV=cloud.
Revert the cloud custom-node suite to SMOKE_ACCOUNT_EMAIL /
SMOKE_ACCOUNT_PASSWORD, the Cloud smoke user the plan specifies. The
CLOUD_TEST_* secrets are unrelated orphans from a removed workflow and are
left untouched.
…n gaps

Panel review of the cloud prong found a live credential leak and two
gates that could pass without checking what they exist to check.

What

- turn Playwright tracing off under CUSTOM_NODES_ENV=cloud: the seeded
  Firebase record rides page.evaluate arguments into the trace, which CI
  uploads as a public artifact, so a long-lived refresh token would ship
  with the first failing cloud run
- fail the forbid-skips step closed when the suite succeeded but wrote no
  results json; previously the skip gate silently no-opped and the job
  went green
- pin the geometry tolerance between 0.009px and 0.011px, so widening it
  fails a test instead of silently loosening the tier
- let preview:cloud-e2e own DEV_SERVER_COMFYUI_URL, removing a path where
  a dropped workflow env silently proxied to stagingcloud
- correct ARCHITECTURE G15, which still promised exact comparison and a
  two-entry ledger, and widen the ledger criterion to cover
  content-variable layout across all six sites that state it
- name the Firebase project in the sign-in failure, record the
  LoadAndResizeImage observation inline, and mark the two DETECTION_PROOF
  rows whose offender pack left the corpus

Why

- the leak becomes real the moment valid smoke credentials land
- a gate that cannot run must not report success
The trace fix that stops a seeded cloud session reaching a public CI
artifact was a single untested ternary; a rename or config restructure
would have silently reinstated the leak.

What

- add scripts/playwright-cloud-trace.test.ts: asserts the custom-nodes
  project traces 'off' under CUSTOM_NODES_ENV=cloud and
  'retain-on-failure' otherwise
- relabel the G15 evidence from '3187 node widths' to measured geometry
  values; the corpus holds 1592 widths, so the noun could not be right
- widen the geometry triage list from three causes to four, the last
  site still routing a content-variable delta to 'fix the code'
- stop claiming the cloud and core report artifacts are identical; the
  cloud report ships without traces
- correct the README bullet that described playwright.chrome.config.ts
  as pinning trace off when it retains on failure
- disclose that DETECTION_PROOF row 11 names a node since ledgered and
  excluded from measurement

Why

- the highest-consequence invariant in this change set was the only one
  without a regression guard
The trace kill switch is set per Playwright project, but the seeding is
keyed on CUSTOM_NODES_ENV and fires from the shared fixture. Running
CUSTOM_NODES_ENV=cloud without --project lands 78 pure specs in the
chromium project, which inherits retain-on-failure and would write a
live refresh token into a local trace. CI always passes
--project=custom-nodes, so this closes the local path.

What

- throw from the cloud branch of the comfyPage fixture when the running
  project does not trace 'off', naming the project and the flag to pass
- pin the trace guard test to jsdom; under the repo's happy-dom default
  the @playwright/test import dumps an unhandled ECONNREFUSED on every
  unit run, reproduced on the pinned Node 25
- describe playwright.chrome.config.ts by what it actually buys (system
  Chrome, no bundled download); its trace never reaches this project
- name the trace mode the custom-nodes project really uses in the core
  gate comment
- four geometry causes resolve to three actions

Why

- a guard keyed on a different axis than the thing it guards is not a
  guard
testInfo.project.use.trace misses a spec-level test.use({ trace }), which
turns tracing on without changing the project value - the guard would
read 'off' and seed a live refresh token into a trace that was actually
recording. The built-in trace option resolves config, project, CLI and
test.use in one read, and matches the value Playwright itself feeds the
trace recorder.

Also handles the documented object form ({ mode: 'off' }), which the
previous string compare would have false-thrown on, and names --trace in
the remedy.
The epsilon change routed numbers to a new branch and left the
exact-equality leg without a test: replacing it with 'return null' kept
the whole pure suite green. That leg carries widgets[].name, the only
cross-run widget rename detector the tier has - the in-run checks compare
before and after within one run, so a rename at identical coordinates
moves both sides together.

Mutation-checked: the new case fails on 'return null', passes on clean
code.
The cloud gate section enumerated two cloud-specific differences and
omitted the third: tracing is off under CUSTOM_NODES_ENV=cloud so the
seeded smoke session cannot ride the public report artifact. That is the
one invariant here whose failure leaks a credential instead of reddening
a test, and it changes how a cloud red is triaged.
The block comment still described the pre-rename secret set; the step
below it reads SMOKE_ACCOUNT_EMAIL and SMOKE_ACCOUNT_PASSWORD only, and
the two names it implied no longer exist anywhere in the repo.
Last stale pack count from the rgthree removal; the manifest has held 6
rows since 83e389f.
Add DETECTION_PROOF rows 12 and 13. Both tiers shipped without a
falsification row, so neither had been proven to catch its class.

Each red was captured in ISOLATION - one break on a branch off the
shipped state - because PR #13534 carries ten live breaks and two of
them (rows 4, 5) sabotage connection validity, which would make an
autogrow red unattributable.

- row 12 (S11): dropping ComfyUI-KJNodes from the extension import list
  reds the load tier's expectedExtensions assert (run 30292373406)
- row 13 (S12): suppressing the live-connect onConnectionsChange call
  reds the dynamic-inputs tier (run 30292391836)
The cloud prong was waiting on an authenticated /object_info probe.
Comfy-Org/cloud already checks the snapshot in at
services/ingest/data/object_info.json (3573 nodes, 88 pack modules,
newer than supported_nodes.yaml), so no credential is needed to
calibrate.

What

- generate customNodeManifest.cloud.json: 87 pack rows, 2778 expected
  nodes, 14 core-disabled nodes
- record yaml packs the snapshot has no nodes for instead of throwing on
  them; one dud pack must not block the other 87. The list ships in the
  manifest and is asserted against the LIVE backend, so a pack that
  starts registering nodes reds rather than silently losing coverage
- keep failing closed when more packs go unjoined than joined - that is
  the dirname mapping breaking, not a few empty packs
- fix the curated overlay key: Cloud installs VideoHelperSuite lowercase,
  and overlay keys must be snapshot dirnames
- invert the two pre-calibration assertions now that the manifest exists

Why

- comfyui-fl-path-animator is declared supported but registers zero
  nodes: absent from the snapshot, from Cloud's own node_pack_mapping,
  and from /api/extensions
The per-row count assert catches drift inside a known pack, but the
suite only iterates manifest rows - a pack the backend registers that
no row covers lands with zero coverage while every assertion stays
green. Cloud deploys packs on its own cadence, so that gap is reachable.

Close the set both ways: the live custom-node pack set must equal the
manifest's, or the gate reds naming the uncovered packs.

Packs under custom_nodes/ that are not community packs are ledgered by
mechanism rather than hardcoded past: ComfyUI_devtools is the harness,
and websocket_image_save ships inside ComfyUI core's own custom_nodes/
(verified in comfyanonymous/ComfyUI), so neither carries a row.

Dry-run against the real data: cloud live 87 == manifest 87; core
installs exactly its 6 rows plus devtools.
Every generated cloud row shipped expectedExtensions: [], which the
schema defines as 'this pack ships no frontend JS'. That is false for
the packs Cloud serves extensions for, so the extension-load assert had
nothing to check on the cloud prong.

Derive the real names without credentials: /api/extensions is reachable
unauthenticated, and so are the extension files it lists, so the
registered name can be read out of each registerExtension call.

One sentinel per pack, not the full set - the assert only has to prove
the pack's JS loaded, and a conditionally-registered extra would red a
healthy run. 17 packs calibrated.

Names that could not be confirmed as registerExtension names are
omitted rather than guessed (LTXDirector, makadi_iTools, vewd, 'Basic
data handling: dynamic input'); a wrong sentinel reds the gate forever.

Cross-check: the derived name for videohelpersuite is
VideoHelperSuite.Core, byte-identical to the core manifest's
independently calibrated value.
What

- pre-push exits early when every pushed ref is a deletion; there is no
  source state to check, so knip only added latency to
  'git push --delete' (and forced a HUSKY=0 bypass to delete a branch)
- calibrate four more cloud packs' expectedExtensions, taking the count
  from 17 to 21 of 87

Why

- the earlier four were dropped on a 'name contains a dot' heuristic,
  which is a naming convention rather than evidence; all four are
  genuine registerExtension names, confirmed at their call sites in the
  JS Cloud serves

Notes

- knip still runs on every real push, on a mixed delete+update push, and
  canonically in CI (.github/actions/lint-format-verify)
The field ships in the manifest and the live assert reads it, but
assertCloudManifestShape never checked it, so a hand-edited or stale
manifest could carry a missing or malformed list and the assert would
silently check nothing - the exact silent-coverage-loss the field
exists to prevent.

Require it: an array of unique non-empty strings, where an empty array
is the deliberate 'every yaml pack joined' declaration.

Also pin that a well-formed top level still validates each row, so the
per-entry deployRef error cannot get masked by the shape check.
Verified against run 30297797189: row 12 reproduces verbatim here (8
occurrences), but row 13 does not. The dynamic-inputs tier still reds,
at an earlier precondition - 'consumer instantiates with at least one
input' - because row 1's break leaves the consumer with zero inputs, so
the test dies before reaching the autogrow assert.

The break is still proven: run 30292391836 captured row 13's message in
isolation. What was wrong was implying a reader could reproduce that
exact text from this PR's HEAD.

Same reason rows 12 and 13 were captured on isolated branches at all -
rows 4 and 5 sabotage connection validity and slot hit-testing, so an
autogrow red taken here could not be attributed to row 13's break.
Core rows calibrate expectedExtensions by grepping the pinned source.
Cloud packs are never checked out locally, so the 21 sentinels were read
off the /extensions files Cloud serves unauthenticated, taking the name
from each registerExtension call site.

That call site proves the name exists; it does NOT prove the extension
registers unconditionally at boot, which is what the assert requires.
Document the failure mode for the first cloud burn-in so a conditional
sentinel gets swapped rather than the assert deleted or the list
emptied - [] declares 'this pack ships no frontend JS', which would be
false for these packs.
Run 30309274120, the first real cloud run, failed all 359 tests on one
call: setupUser's node-side GET /api/users carries no bearer, and cloud
rejects it before any test starts. Same class for the devtools settings
POST, silently no-oping in its try/catch.

What

- skip the local multi-user registry on cloud; the smoke account is the
  user, the worker username stands in as the id
- skip the node-side devtools settings POST on cloud; apply the same
  startup settings through the booted app's authenticated session

Why

- /api/users and /api/devtools/set_settings sit behind the cloud
  session, which only the browser holds; a Playwright request context
  is same-origin but carries no Authorization header
What

- scripts/run-custom-nodes-local.mjs: boots the core backend (venv
  auto-detect, video asset staged) + dev server, or the cloud build +
  preview, waits on real readiness endpoints, runs the suite, tears
  down only what it started
- test:custom-nodes:local and test:custom-nodes:local:cloud package
  scripts; extra args pass through to Playwright
- README Running section leads with the orchestrators

Why

- the manual recipes were three terminals of commands in a fixed order
  that nobody remembers; already-running services are reused rather
  than killed, so the fast iteration path stays fast
…-e2e-suite

# Conflicts:
#	browser_tests/fixtures/ComfyPage.ts
The bespoke 180-line launcher script reimplemented what
start-server-and-test already does (start, wait on readiness URLs,
reuse already-running services, tear down). Replaced with chained
package scripts; the backend launcher is one sh line with venv
auto-detect and the video asset staged.
What

- outputHashes.ts: PNG IDAT-only sha256 (ComfyUI embeds prompt metadata
  in tEXt, so whole-file hashes false-fail on identical pixels), strict
  compare naming node + both digests; missing, changed, and NEW outputs
  all fail
- T1 hook: after the sink-payload proof, fetch each image via /api/view
  and compare against curatedOutputHashes.core.json; unenrolled curated
  workflows fail with the record remedy - no silent opt-out
- record mode (RECORD_OUTPUT_HASHES=1) writes the observed table and
  deliberately fails each test, per the CN_GEOMETRY record convention
- record workflow gains the S15 step + artifact on the same backend boot
- core only: the fetch is node-side, which on cloud carries no session
  (run 30309274120); cloud enrolls when an in-app fetch lands

Why

- execution_success proves a workflow ran; nothing proved it produced
  the same pixels - a serialization regression could silently change
  outputs. Also raises the cloud gate timeout to 150: run 30312432225
  proved the full manifest cannot finish in 60 unsharded
@benjcooley

Copy link
Copy Markdown
Collaborator Author

Head 350ffdf481 is fully green and the raw Playwright artifacts have been audited.

  • Custom-node run 32322122794: 184/184 passed, 0 failed, 0 skipped, 0 flaky, max retry 0.
  • Per slice: Cloud 30 / 31 / 31 / 29 / 30, Core 33.
  • Wall time: Cloud 7m50s–12m01s, Core 15m44s.
  • Ordinary E2E run 32322122951: all browser variants and 16 Chromium shards passed.
  • Codecov's e2e upload is attached to exact head 350ffdf481, names run 32322122951 as its build, and is not carried forward. All PR checks are complete: 76 passing, 0 failing, 0 pending (7 expected path-level skips).

The preceding Core red was isolated to was-node-suite-comfyui/Text Random Prompt. At the pinned source it executes requests.get("https://lexica.art/api/v1/search?...") with no timeout. The fix does not raise a timeout or accept a failed result: that exact external-network node is excluded from S9 and WAS's exact runnable-corpus contract changes from 163 to 162. All other WAS nodes and applicable tiers still run. The Actions summary reports the omission as S9 - SKIP - NODE NOT EXECUTED, and deterministic restoration is owned by FE-1682.

@datadog-official

This comment has been minimized.

Nightly schedule and manual dispatch only. PRs neither trigger the
workflow nor wait on any of its checks, and nothing is required in
branch protection. The path-gate job and fork-PR policy skip existed
only to make a pull_request trigger required-safe, so they go with it.
Dispatch with the branch input to test a PR branch.
comfyui_essentials@1.1.0 failed to download during a cdn.comfy.org
outage that outlived three 2s retries, then fetched cleanly minutes
later at the same URL. Retry all errors with backoff for up to 60s and
drop partial files so a resumed retry cannot unzip a truncated
artifact.
main added an eager isCloud import to StorageKeys, which the custom-node
fixtures import in Playwright's Node collection process. Vite replaces
__DISTRIBUTION__ in browser bundles but nothing defined it under Node,
so every spec aborted before collection. Mirror vite.config.mts's
derivation at config load, the same way vitest.setup.ts already does.
DrJKL
DrJKL previously requested changes Aug 21, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. A parallel correctness/security/CI review plus a thermo-nuclear maintainability pass found blocking issues that should be addressed before this becomes a long-lived gate.

The highest-risk problems are the PR-controlled reusable workflow inheriting repository secrets, persisted checkout credentials remaining available while third-party Python executes, non-reproducible manifest generation, and run-event attribution that can produce false passes. Structurally, this adds a 1,863-line spec framework, a 751-line shell workflow, 4,126 lines of mostly-default interaction snapshots, and cast-first boundary validation. These are expensive foundations to normalize.

The test corpus also conflicts with the repository's own guidance: unit logic belongs in Vitest, not Playwright (AGENTS.md); specs should contain tests/hooks rather than helper frameworks (browser_tests/README.md); inline test types/casts are forbidden (README); and mutation assertions must poll (README). The 16 *.pure.spec.ts files contain 142 tests/3,178 lines; 134 run without a browser and should be Vitest tests. The fake Page/response systems, literal-ledger change detectors, and repeated pairing permutations make the suite test its scaffolding rather than behavior.

The focused product regressions for empty audio and unset image values are good and should remain. Verification: browser and script typechecks passed; focused Vitest passed 27/27; script tests passed 13/13. Of the pure Playwright tests, 134 passed and the remaining 8 could not launch because this environment lacks /opt/google/chrome/chrome. Green CI does not resolve the design and correctness findings below.

source_pr: ${{ format('{0}', github.event.pull_request.number) }}
source_run_id: ${{ format('{0}', github.run_id) }}
source_sha: ${{ github.event.pull_request.head.sha || github.sha }}
secrets: inherit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — remove this PR-context secret handoff. For same-repository PRs, the local reusable workflow is loaded from the PR commit, so PR-authored workflow code receives every inherited repository secret. The fork check does not protect same-repo branches. Move coverage upload to a trusted default-branch workflow_run path, or otherwise call only immutable trusted code without inherited secrets.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: the coverage workflow no longer accepts workflow_call or inherited secrets. It runs only from the trusted default-branch workflow_run path and excludes fork PRs.

echo "SHARD_SLUG=${CUSTOM_NODES_MANIFEST}-${CUSTOM_NODES_SHARD/\//-of-}" >> "$GITHUB_ENV"

- name: Checkout repository
uses: actions/checkout@v7

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set persist-credentials: false. This checkout leaves the repository token in Git config while the later install step fetches and executes third-party custom-node setup/Python code. That untrusted code must not have access to checkout credentials.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: the frontend checkout now sets persist-credentials: false before any third-party custom-node code can execute.

uses: actions/checkout@v7
with:
repository: 'comfyanonymous/ComfyUI'
ref: ${{ inputs.comfyui_ref }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This second checkout also needs persist-credentials: false: its credential helper remains reachable when arbitrary custom-node installers and imports execute. Remove both persisted tokens before running third-party code.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: the ComfyUI checkout now sets persist-credentials: false as well.

Comment thread scripts/cloud-manifest.ts
const currentValue: unknown = JSON.parse(readFileSync(manifestPath, 'utf8'))
const current = assertCloudManifestShape(currentValue, manifestPath)
const snapshotValue: unknown = JSON.parse(readFileSync(snapshotPath, 'utf8'))
const { overlay, sentinels, cannotRunAlone } = regenerationSidecars(current)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated manifest is feeding authored policy back into its own regeneration. Overlays, sentinels, and calibration are read from the previous output and then overwritten, so a clean regeneration is not reproducible and deleting/corrupting the artifact loses source data. Keep authored policy in explicit sidecars and generate the manifest solely from source inputs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: authored policy is now stored in curatedWorkflows.json, extensionSentinels.json, and cannotRunAlone.json. Manifest generation no longer reads policy back from its own generated output.

// `executing` strings - stay, and graph membership still vets them);
// otherwise the legacy seen-set exclusion.
capturedPromptId !== undefined
? event.prompt_id === undefined || event.prompt_id === capturedPromptId

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This retains every prompt-less executing event, including late/concurrent events from another run. classifyRun only applies graph membership to error attribution, so reused node IDs can satisfy expected execution and yield a false PASS. Scope all event kinds to the active attempt/graph and add a regression with a late prompt-less event.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590 with a pure prompt-event scope boundary and regression test. Prompt-less events are accepted only inside the captured prompt execution window; a foreign execution_start closes attribution even when node IDs are reused.

}
}
return {
page: page as unknown as Page,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not build a fake Playwright runtime and force it through as unknown as Page. This hand-implements evaluate, response listeners, and request/response objects (the same pattern appears in promptError.pure.spec.ts), coupling tests to a mock framework we do not own. Extract typed pure transitions for unit tests and cover the lifecycle once with a real Playwright page. This also violates the cast rule in the test guidance.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: the fake Page/Response runtime suites are removed. Prompt submission and response classification are pure typed boundaries with Vitest tests; the remaining browser test uses real Playwright objects.


test.use({ initialSettings: customNodeSuiteSettings })

interface ProbePlan {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This spec starts by declaring copied domain types, constants, and helper algorithms; the same architecture appears in allNodes, connectivity, and dynamicInputs. Specs should contain imports, Playwright configuration/hooks, and tests—not implementation frameworks (guidance). Move the plan/producers into typed fixture modules and import the real domain types rather than inline declarations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: interaction probing and profile comparison helpers moved out of the spec into interactionProbe.ts with direct Vitest coverage.

},
[String(consumer.id), String(inIndex)] as const
)
expect(linked, `${key} with VueNodes=${vueNodesEnabled}`).toBe(true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This snapshots graph state immediately after a drag, which is explicitly listed as a flake pattern in the Playwright guidance. Use await expect.poll(() => page.evaluate(...)).toBe(true). The same snapshot-after-mutation issue exists in customNode.regression.spec.ts and coreSmoke.spec.ts.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: post-drag graph mutations in the affected connectivity and smoke tests now use expect.poll rather than immediate snapshots.


test.describe('cloud roundtrip expectations', () => {
test('pins only the observed widget indices', () => {
expect(ROUNDTRIP_VALUE_ALLOWED_INDICES_LITEGRAPH).toMatchObject({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These assertions lock literal ledger contents rather than consumer behavior; the following test does the same for a specific temporary entry, and consoleErrorLedger.pure.spec.ts repeats the pattern. Repository guidance explicitly forbids change-detector/default-value tests. Test schema and observable classification behavior, not today's exact calibration list.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: literal ledger-content assertions were replaced by synthetic schema, stale-entry, and behavioral tests. Authored contents can evolve without rewriting change-detector tests.

])
})

test('COMBO slots with different vocabularies stay excluded', () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 424-line unit suite repeats large node definitions and near-identical COMBO/output-name permutations. Make the meaningful vocabulary/schema cases table-driven and share one ledger-integrity contract rather than duplicating exhaustive fixtures across suites. The testing guidance requires parsimonious, composable behavioral tests—not another maintenance surface proportional to manifest size.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #15590: type-pairing cases are table-driven, preserving the behavioral cases while removing repetitive setup and assertions.

@DrJKL
DrJKL dismissed their stale review August 21, 2026 18:19

Not going to block, but :-/

@benjcooley
benjcooley added this pull request to the merge queue Aug 21, 2026
Merged via the queue into main with commit 92c249d Aug 21, 2026
93 checks passed
@benjcooley
benjcooley deleted the benjcooley/custom-node-e2e-landing branch August 21, 2026 19:44
mattmillerai added a commit that referenced this pull request Aug 22, 2026
The custom-node console ledger (#15225, landed on main after this branch
was cut) requires the ColorOverlay duplicate-registration error to be
observed at startup:

  requiredStartupId: 'duplicate-color-overlay'
  pattern: /\[vite:preloadError\].*Extension named 'ColorOverlay' already registered\./

That error was the throw this PR removes. Registering a duplicate name now
warns and skips, so dz_node_palette.js finishes evaluating and emits no
preloadError - the pattern can never match again, since the message it
matches no longer exists in the frontend. Left in place,
staleRequiredStartupErrorRulesForPacks would report
ComfyUI_LayerStyle_Advance/duplicate-color-overlay and fail the nightly
suite's stale-rule assertion.

Removing the entry is the ledger's documented lifecycle for a mechanism
that stops firing, and it tightens rather than loosens the gate: the pack
no longer carries a suppression, so a duplicate-registration error
reappearing would surface as unallowlisted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mattmillerai added a commit that referenced this pull request Aug 23, 2026
The custom-node console ledger (#15225, landed on main after this branch
was cut) requires the ColorOverlay duplicate-registration error to be
observed at startup:

  requiredStartupId: 'duplicate-color-overlay'
  pattern: /\[vite:preloadError\].*Extension named 'ColorOverlay' already registered\./

That error was the throw this PR removes. Registering a duplicate name now
warns and skips, so dz_node_palette.js finishes evaluating and emits no
preloadError - the pattern can never match again, since the message it
matches no longer exists in the frontend. Left in place,
staleRequiredStartupErrorRulesForPacks would report
ComfyUI_LayerStyle_Advance/duplicate-color-overlay and fail the nightly
suite's stale-rule assertion.

Removing the entry is the ledger's documented lifecycle for a mechanism
that stops firing, and it tightens rather than loosens the gate: the pack
no longer carries a suppression, so a duplicate-registration error
reappearing would surface as unallowlisted.
pull Bot pushed a commit to Penguinjanator/ComfyUI_frontend that referenced this pull request Aug 27, 2026
## Summary

Makes the custom-node browser suite and dual-renderer ecosystem matrix
fail-closed pull-request and merge-queue gates.

## Changes

- Adds `pull_request` and `merge_group` triggers without path filters.
- Exposes stable aggregate checks named `E2E Custom Nodes Test` and
`Custom Nodes Ecosystem Matrix test`.
- Keeps retries disabled and fails on failed, skipped, flaky, missing,
mixed-renderer, or incomplete results.
- Runs the ecosystem corpus in legacy and Nodes 2.0 modes with
independent detection proofs.
- Preserves the custom-node S1/S2/S3/S9/S15 detection proofs on
scheduled and manual proof runs.
- Removes persisted checkout credentials before third-party pack setup
and prevents PR cache publication.
- Includes the reviewed ECS compatibility, malformed registry-name,
broken preview-image, S3 readiness, and exclusion-report fixes needed
for a green gate.

Follow-up to Comfy-Org#15225.

## Validation

- 213 focused unit tests pass across the custom-node harness, matrix
runner, ECS widget compatibility, registry guard, preview renderer, and
result reporters.
- Frontend, browser, and scripts typechecks pass.
- Lint, format, Knip, YAML parsing, and exact Playwright collection
pass.
- Exact-head gate runs: pending on commit `336548bd43`.

---------

Co-authored-by: Alexander Brown <drjkl@comfy.org>
mattmillerai pushed a commit that referenced this pull request Aug 29, 2026
## Summary

Makes the custom-node browser suite and dual-renderer ecosystem matrix
fail-closed pull-request and merge-queue gates.

## Changes

- Adds `pull_request` and `merge_group` triggers without path filters.
- Exposes stable aggregate checks named `E2E Custom Nodes Test` and
`Custom Nodes Ecosystem Matrix test`.
- Keeps retries disabled and fails on failed, skipped, flaky, missing,
mixed-renderer, or incomplete results.
- Runs the ecosystem corpus in legacy and Nodes 2.0 modes with
independent detection proofs.
- Preserves the custom-node S1/S2/S3/S9/S15 detection proofs on
scheduled and manual proof runs.
- Removes persisted checkout credentials before third-party pack setup
and prevents PR cache publication.
- Includes the reviewed ECS compatibility, malformed registry-name,
broken preview-image, S3 readiness, and exclusion-report fixes needed
for a green gate.

Follow-up to #15225.

## Validation

- 213 focused unit tests pass across the custom-node harness, matrix
runner, ECS widget compatibility, registry guard, preview renderer, and
result reporters.
- Frontend, browser, and scripts typechecks pass.
- Lint, format, Knip, YAML parsing, and exact Playwright collection
pass.
- Exact-head gate runs: pending on commit `336548bd43`.

---------

Co-authored-by: Alexander Brown <drjkl@comfy.org>
@huang47 huang47 added the risk:xhigh PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026 — with ChatGPT Codex Connector
@huang47 huang47 removed the risk:R3 PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026
mattmillerai added a commit that referenced this pull request Oct 8, 2026
Main's Codecov upload step already sets override_branch, override_commit
and override_pr (#15225, hardened for forks in #16404). Keeping this
branch's copies would produce duplicate keys in the step's `with:` map,
which GitHub Actions rejects as an invalid workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:xhigh PR risk grade (advisory shadow check; grader-owned) size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants