Repository navigation
test: custom-node E2E regression suite - Core depth and Cloud breadth #15225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
365 commits
Select commit
Hold shift + click to select a range
2085f11
test: pin the run-enrollment and workflow-when-run invariants
CodeJuggernaut 20c24ce
test: enroll the mutated cloud fixture in the run tier
CodeJuggernaut 83e389f
test: drop rgthree-comfy from the custom-node regression suite
CodeJuggernaut 130d0b1
ci: guard the forbid-skips step against a missing results file
CodeJuggernaut 04376b7
test: compare geometry within a 0.01px tolerance, ledger a content-va…
CodeJuggernaut d351d9e
docs: trim churned comments from cloud-prong files
CodeJuggernaut 2cc96e9
fix: point cloud custom-node suite at the real repo secrets
CodeJuggernaut d161812
test: match cloud smoke tool's Firebase sign-in payload exactly
CodeJuggernaut 27b1339
test: use the smoke user for cloud auth
CodeJuggernaut 5a1ab12
fix: stop cloud auth tokens reaching CI artifacts, close two fail-ope…
CodeJuggernaut 4283d41
test: guard the cloud trace kill switch, correct four doc claims
CodeJuggernaut 93e22d4
fix: refuse to seed a cloud session into a traced project
CodeJuggernaut 14d7c0e
fix: read the resolved trace option in the cloud seeding guard
CodeJuggernaut 163af5d
test: cover the non-numeric compare in diffGeometry
CodeJuggernaut 01052a5
docs: record the cloud trace kill switch as a cloud-vs-core difference
CodeJuggernaut f8a2261
docs: the cloud gate step reads two credentials, not four
CodeJuggernaut e4e6afd
docs: the custom-nodes repro needs 6 packs, not 7
CodeJuggernaut 085901e
docs: falsify the extension-load (S11) and autogrow (S12) tiers
CodeJuggernaut c342c56
feat: generate the cloud manifest from Cloud's checked-in object_info
CodeJuggernaut 6ee6055
test: red when a pack registers with no manifest row
CodeJuggernaut 7f1494c
feat: calibrate cloud expectedExtensions from the JS Cloud serves
CodeJuggernaut e17da78
chore: skip the pre-push knip run on branch deletions
CodeJuggernaut a5fafdd
fix: validate unjoinedYamlPacks in the cloud manifest shape check
CodeJuggernaut 564ea72
docs: disclose that row 13's red is not reproducible from this PR
CodeJuggernaut 65800c9
docs: record where cloud expectedExtensions come from, and their caveat
CodeJuggernaut e53b2c9
fix: stop cloud fixture init dying on unauthenticated harness calls
CodeJuggernaut 5ca032c
feat: one-command local runs for the custom-node suite
CodeJuggernaut 14f3d1f
Merge remote-tracking branch 'origin/main' into nathaniel/custom-node…
CodeJuggernaut d5e1d2e
refactor: compose the local suite runners from start-server-and-test
CodeJuggernaut f9b2485
feat: S15 output-regression tier - pixel hashes for curated runs
CodeJuggernaut fe6fb6d
ci: let the record workflow run from a record/* ref push
CodeJuggernaut f652e60
fix: S15 record mode survives per-failure worker restarts
CodeJuggernaut 4ff89eb
fix: S15 hashes the sink payloads, not just image files
CodeJuggernaut 2e28b60
fix: S15 messages say output, not pixel - value sinks are not images
CodeJuggernaut 9a49e05
test: commit the CI-recorded S15 output hashes
CodeJuggernaut 4fa4ab9
ci: give cloud calibration dispatches their own concurrency group
CodeJuggernaut 501e955
docs: module header describes payload hashing, not the retired pixel-…
CodeJuggernaut a37be48
fix: apply the CORE panel's findings to the S15 tier
CodeJuggernaut 3743d7f
test: re-record S15 hashes under the reviewed canonicalization
CodeJuggernaut 129d1b2
docs: record the cloud calibration result - unsharded cannot pass
CodeJuggernaut ec8a726
fix: loud off-CI S15 skip, fixture schema guard, Step 5c scoped to CI
CodeJuggernaut 5780b9f
feat: S13 interaction-profile tier - lock every node's interaction de…
CodeJuggernaut ece3d94
ci: record S13 cloud interaction profiles on the cloud record boot
CodeJuggernaut 0678041
test: commit the CI-recorded S13 interaction baselines
CodeJuggernaut 7fa7d97
ci: let the cloud record workflow run from a record ref push
CodeJuggernaut 22b1bc2
ci: raise the cloud gate timeout to GitHub's 360-minute wall
CodeJuggernaut 6840d65
perf: one app boot per pack - merge the three every-node tiers
CodeJuggernaut 9d0f147
docs: add the custom-node suite Technical Design Doc
CodeJuggernaut c408ebc
fix: cloud custom-nodes tests get a 60s base timeout
CodeJuggernaut 550faf5
docs: the Technical Design Doc's source of truth moves to Notion
CodeJuggernaut e242a12
ci: add the cloud drift leg (canary-cloud-drift) to the nightly canary
CodeJuggernaut ba9653b
ci: reground cloud concurrency comments on the account queue
CodeJuggernaut c42b2c3
ci: reground the cloud sharding comment on the measured cost
CodeJuggernaut 8a56815
test: cancel only our own prompts when draining the backend
CodeJuggernaut 2bb932c
ci: raise the cloud record ceiling above what cloud actually costs
CodeJuggernaut 4ec9c4d
test: enroll S15 on cloud by downloading outputs in-page
CodeJuggernaut 7177d68
test: stop cloud from retrying an unrecoverable hook, and time it
CodeJuggernaut 2bdc727
test: make an unready app say why, permanently
CodeJuggernaut ba922f9
test: remove the cloud per-test cap; ceilings are the only fuse
CodeJuggernaut 59f7760
test: bound the app-ready wait so a hang explains itself
CodeJuggernaut d9df7b7
test: satisfy cloud's onboarding gate before waiting for the app
CodeJuggernaut efeec0b
test: answer cloud's onboarding gate so the graph view can load
CodeJuggernaut d8c2d77
test: drop the over-explained comment on the survey bypass
CodeJuggernaut b3c823a
test: trace every cloud page event so a hang names its cause
CodeJuggernaut 62d3c35
test: seed the cloud session where the app actually reads it
CodeJuggernaut de0ecdc
test: fail loudly if the cloud session does not persist
CodeJuggernaut 35abe84
fix: restore working cloud seeding auth, drop the form-login churn
CodeJuggernaut d249de8
test: tolerate testcloud's unserved global_subgraphs, calibrate Impac…
CodeJuggernaut 226b0b2
test: calibrate the cloud auto-run baseline, defer S13/S14 until reco…
CodeJuggernaut 152dd18
fix: anchor telemetry blocking to third-party hosts, not URL substrings
CodeJuggernaut 476e036
fix: fulfill telemetry blocks with 204 instead of aborting
CodeJuggernaut 2a3c07a
fix: answer blocked telemetry with 200 {} so SDK json parsing stays q…
CodeJuggernaut 80204d5
test: instrument cloud auth seams to attribute the boot 403s
CodeJuggernaut 2290c85
test: keep the cloud fail-fast out of the settings-write 403
CodeJuggernaut b91e56b
fix: seed a real workspace JWT - testcloud no longer accepts Firebase…
CodeJuggernaut 56d7714
fix: classify inputs declaring options.widgetType as widgets in auto-run
CodeJuggernaut 0cd2aa0
fix: keep /api/features anonymous in the cloud smoke auth rewrite
CodeJuggernaut 736c996
test: defer S15 output hashes on cloud until baselines are recorded
CodeJuggernaut cb26844
test: recalibrate the cloud auto-run baseline, counts, and run sentinels
CodeJuggernaut e2cb69d
test: stop the stale-ledger guards firing on env-absent nodes
CodeJuggernaut dda1137
test: defer the execution-error self-check instead of skipping it
CodeJuggernaut 269d47c
fix: ledger the cloud console noise and match pack keys case-insensit…
CodeJuggernaut 64e2574
fix: count cache-served nodes as executed in the run classifier
CodeJuggernaut a909286
fix: stop an unset image-upload combo requesting filename=undefined
CodeJuggernaut 5521a2e
fix: apply the console-error ledger to the curated load tier
CodeJuggernaut 5c20163
test: bind the cloud run tier to the path the upload actually stored
CodeJuggernaut ca5746c
test: chunk the connectivity sweep and budget it per pair
CodeJuggernaut b517d61
Merge remote-tracking branch 'origin/main' into tmp/cn-inline-merge
CodeJuggernaut 54f5cda
test: disconnect swept nodes and rank the pair gate first
CodeJuggernaut 7697e53
test: give the load-widget fixture a real missing video to fail on
CodeJuggernaut 85e38ee
fix(custom-nodes): resolve ledger pack keys case-folded; make tier re…
IAMtheIAM f5082d9
fix: case-fold ledger pack keys and make tier reds readable
CodeJuggernaut 069838b
test: re-derive the cloud cannotRunAlone baseline with cache accounting
CodeJuggernaut ac60028
fix(custom-nodes): restore core sweep and cloud session auth
CodeJuggernaut 5339bae
revert: restore plain graph.clear teardown in the connectivity sweep
CodeJuggernaut 4a71dce
merge: keep verified custom-node harness fixes
CodeJuggernaut 7e83eaf
fix: count only widgets the Vue renderer actually draws in S2
CodeJuggernaut 23aa042
test: correct cloud S2 and S9 expectations
CodeJuggernaut 4f8d757
test: classify verified cloud custom-node findings
CodeJuggernaut 7b27bfc
test: tighten cloud custom-node failure evidence
CodeJuggernaut 4740168
test: correct final core and cloud custom-node reds
CodeJuggernaut 4507ca7
ci: remove custom-node core timeout
CodeJuggernaut a9b6fa1
test: scope custom-node startup settings by environment
CodeJuggernaut 1dd9510
test: seed blank custom-node workflow before first boot
CodeJuggernaut fcec620
test: keep boot guard cleanup private
CodeJuggernaut f38f251
test: reconcile remaining cloud custom-node failures
CodeJuggernaut 683ae49
test: classify state-variable cloud outcomes
CodeJuggernaut 07e3834
test: settle exact custom-node runtime mechanisms
CodeJuggernaut 22c91be
test: gate exact custom-node runtime variance
CodeJuggernaut e636aa1
Merge origin/main into custom node E2E suite
CodeJuggernaut 937f0db
Merge latest origin/main into custom node E2E suite
CodeJuggernaut d46f7fc
Merge origin/main and harden custom node E2E proof
CodeJuggernaut 2cd2c34
Merge latest origin/main into custom node E2E suite
CodeJuggernaut f134a3f
test: align LTXVideo Cloud node count
CodeJuggernaut 9637805
test: keep Cloud evidence key constant local
CodeJuggernaut d58cbae
test: make Cloud 502 tracing observational
CodeJuggernaut fc3675d
test: ledger VHS preview geometry instability
CodeJuggernaut b19924e
test: baseline FlashVSR Cloud auto-run failures
CodeJuggernaut abfe4b0
test: baseline GIMM Cloud model loader
CodeJuggernaut 6bf3943
test: surface required prompt rejections
CodeJuggernaut 96bd9e0
test: stream custom-node failure summaries
CodeJuggernaut 90b206f
test: fail fast on required Cloud boot 5xx
CodeJuggernaut 21b86b2
test: wait for WhatDreams widget initialization
CodeJuggernaut 6217d84
test: clarify custom-node CI labels
CodeJuggernaut 96c6809
test: a 5xx prompt rejection is an environment fault, not a pack verd…
benjcooley db21287
Merge remote-tracking branch 'origin/main' into nathaniel/custom-node…
CodeJuggernaut a062fe0
test: skip coverage teardown on a page the test legitimately closed (…
benjcooley 03b11ef
test: route the incident 5xx mock through the path production takes (…
benjcooley 2d6578e
ci: dispatch inputs and a results table for the core custom-node gate…
benjcooley 498a99a
test: ledger the WAS vue-height flip and the MathExpression sweep cra…
benjcooley f82454a
ci: serialize every cloud-gate run and stop the per-PR stampede (#15060)
benjcooley 191f9f7
test: exclude the impact-add-queue emitters and ledger their victim (…
benjcooley e3ff0d7
ci: cache the core gate's setup - pip wheels and pinned pack trees (#…
benjcooley d02f676
test: fail error surfaces with their visible text, not a bare count (…
benjcooley 5592728
test(custom-nodes): AudioConcatenate cannot run alone on Cloud (#15162)
benjcooley c6531c8
Merge origin/main into nathaniel/custom-node-e2e-suite
benjcooley 6cad289
chore: mark recorded custom-node fixture data as generated
benjcooley 8ad9a47
test: single-source the public Firebase web key in cloud fixtures
benjcooley 56ea1d1
fix: keep Firebase fixture independent of Vite config
CodeJuggernaut b4494ec
test: isolate custom-node E2E tier results (#15155)
CodeJuggernaut bf66f00
test: remove the live-cloud surface from the custom-node suite
benjcooley fb4e6fb
test: remove the S14 geometry tier and its recorded baselines
benjcooley 12e0cf8
chore: drop unrelated drift this branch carried
benjcooley bfa059b
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley 7cbfc7e
test: report pack pin age, and give the bump a command and an action
benjcooley 3149fbc
test: address the automated review on the custom-node suite
benjcooley 1f4e125
test: run the cloud pack set against the local backend, sharded ten ways
benjcooley 15b6f18
chore: mark the generated cloud manifest as generated
benjcooley 3a0a5e9
test: drop the unconsumed cloud-backend surface, and lock the shard p…
benjcooley 4b8afea
test: rebalance to 18 shards, sized from measured suite time
benjcooley bc8a532
test: run the tiers a pack asks for, not every tier against every pack
benjcooley 3c430f6
test: four shards - the fewest that still lands inside the target window
benjcooley 8e4bd54
test: one pack's broken requirements must not erase the shard's coverage
benjcooley 9e8e209
test: quarantine packs their author has to fix, and prove the quarant…
benjcooley 2a7fd35
test: balance shards by node count, not by pack count
benjcooley ddf1273
test: S13 covers packs with a baseline recorded at the ref under test
benjcooley 84295dc
test: calibrate the cloud pack set against a local backend
benjcooley fdb207e
test: complete the local calibration, and treat GPU-gated packs as en…
benjcooley 6ecbfb8
test: fix shard composition to the manifest, not to the exclusion led…
benjcooley 0b95ae5
test: calibrate on stable shard composition
benjcooley 299bbd2
test: exclude packs that register nothing on a CPU runner
benjcooley a5e7bb6
test: calibrate cloud custom-node coverage
benjcooley 5b712ad
test: ledger cloud node behaviors
benjcooley bca38dc
test: finish cloud renderer calibration
benjcooley b740297
test: finish cloud execution coverage
benjcooley cfcfc9f
test: contain Fill-Nodes dynamic cleanup
benjcooley 27ba93b
test: scope isolated pair diagnostics
benjcooley b46bdb9
test: reject disappearing error evidence
benjcooley ac34c75
test: fail on visible error evidence
benjcooley b798dcd
test: remove snapshot-only interaction gate
benjcooley f8498dd
test: require complete custom-node execution
benjcooley 8d8054b
test: make cloud applicability explicit
benjcooley cf75904
test: expose every custom-node coverage exclusion
benjcooley adab982
ci: make the custom-node gate fail closed
benjcooley 987c336
test: make custom-node coverage exceptions explicit
benjcooley 0cec18c
test: require exact custom-node behavior
benjcooley 7b67559
[automated] Apply ESLint and Oxfmt fixes
actions-user 3082920
test: harden custom-node behavior contracts
benjcooley ea9fccc
test: state connectivity coverage precisely
benjcooley bdef938
ci: fix shared path-filter predicate
benjcooley c796a16
docs: describe cloud manifest maintenance honestly
benjcooley ba4d7e3
fix: avoid empty audio preview requests
benjcooley bef5c8d
test: surface conditional console acceptances
benjcooley fd75f63
test: make custom-node controls deterministic
benjcooley b881d79
test: exercise loaded audio preview
benjcooley ec617a6
[automated] Update test expectations
invalid-email-address 6599f5f
test: keep audio snapshot update scoped
benjcooley 1ab0597
docs: describe stable shard tie behavior
benjcooley c4bbd07
chore: remove retired suite drift
benjcooley d78acaa
test: ground local baselines in evidence
benjcooley 073c1d0
test: break an exercised S9 witness
benjcooley 09413c5
test: keep unset image preview case concise
benjcooley b7da864
test: isolate custom-node detection witnesses
benjcooley 620dfc6
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley e8a87de
test: follow automatic Vitest cleanup
benjcooley 95e701b
docs: report measured custom-node runtime
benjcooley 7c37a15
test: exercise local autogrow coverage
benjcooley 084eba4
test: make custom-node exclusions fail closed
benjcooley 05c54a1
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley 2f005ca
test: stabilize custom-node CI runtime
benjcooley 5489d6b
test: compare serialized widget topology
benjcooley 8e3fa06
test: stabilize five-shard custom-node gate
benjcooley 280da20
test: make custom-node roundtrip readiness exact
benjcooley 75bffe4
test: restore core custom-node depth coverage
benjcooley ec3920c
test: fix custom-node gate failures
benjcooley 7a9162a
test: harden custom-node isolation checks
benjcooley f5495fa
test: require exact custom-node failure contracts
benjcooley 121de07
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley 9732298
test: attribute deferred node lifecycle failures
benjcooley 54d6b4f
test: keep visible-error proof self-describing
benjcooley bb1176a
test: pin iTools lifecycle readiness
benjcooley a12c66e
test: declare cloud summary fixture context
benjcooley d5417d9
test: pin renderer-specific iTools readiness
benjcooley 885238b
test: name every accepted coverage gap by S-tier
benjcooley 9ae3592
test: report observed value when roundtrip readiness times out
benjcooley f6b2ccd
fix: harden custom node e2e contracts
benjcooley bcc21cf
fix: validate cross-shard connectivity endpoints
benjcooley 0037995
fix: scope visible error mutation scans
benjcooley 14b9072
perf: collapse visible error selector walks
benjcooley 9d98479
perf: skip empty error subtrees
benjcooley 1cd723c
perf: avoid error-surface subtree rescans
benjcooley 226fd9f
perf: ignore unrelated error-surface mutations
benjcooley afddc27
perf: observe only error candidates
benjcooley db343ed
perf: observe inserted error surfaces
benjcooley acb69fd
perf: collapse nested error mutation roots
benjcooley 431c640
fix: retain bounded error attribute detection
benjcooley d765ece
perf: bound visible error sampling by frame
benjcooley 6cd5bae
perf: cap visible error sampling rate
benjcooley 2c47a09
perf: collapse visible error selector scans
benjcooley 50a5041
test: expose connectivity sweep progress
benjcooley 2408ad8
test: bound known connectivity hangs
benjcooley 34924b0
test: terminate hung connectivity targets
benjcooley 272861d
test: interrupt hung connectivity execution
benjcooley 4c330c9
test: detect stalled connectivity cleanup
benjcooley 69ae928
test: classify dangling roundtrip links
benjcooley a056163
test: isolate nondeterministic itools crop node
benjcooley 788b0ee
test: name enrolled custom node coverage
benjcooley 48d3472
test: await restored image preview topology
benjcooley 3332e66
test: await exact roundtrip media state
benjcooley 12c743a
test: scope roundtrip readiness to each batch
benjcooley b4ef7fc
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley 5f71f55
fix: attribute E2E coverage to tested commit
benjcooley 6fd9c76
ci: upload PR E2E coverage from originating run
benjcooley 350ffdf
test: exclude network-dependent WAS prompt from S9
benjcooley 281463d
Merge branch 'main' into benjcooley/custom-node-e2e-landing
benjcooley dfe2c57
Merge remote-tracking branch 'origin/main' into benjcooley/custom-nod…
benjcooley deb8ec6
ci: isolate custom-node jobs from pnpm cache
benjcooley bebb24d
ci: take the custom-node suite off PR triggers
benjcooley 2f67d21
ci: ride out transient CDN errors on registry artifact downloads
benjcooley c2f5a40
test: define compile-time globals for Playwright collection
benjcooley File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This second checkout also needs
persist-credentials: false: its credential helper remains reachable when arbitrary custom-node installers and imports execute. Remove both persisted tokens before running third-party code.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in #15590: the ComfyUI checkout now sets persist-credentials: false as well.