Repository navigation
ci: ecosystem matrix advisory - execute every registry pack with a fail-closed verdict and detection proof - #15175
Conversation
Vendors the census scanner from ComfyUI_ECS_Compat_Check (pure stdlib, adapted to a single cacheable CENSUS_ROOT) and runs it weekly: registry snapshot re-pinned, corpus restored from the actions cache, drifted packs revalidated per-pack by tarball ETag HEAD-check (--revalidate, added here), scan table in the run Summary, results + lockfile as artifact, cache saved under a lockfile-hash key so it updates exactly when a pack moves. Radar, never a gate: the job fails only on mechanical errors. Validated locally against the study's corpus: byte-identical census (4,983 packs, 701/14.1% confirmed-break, 37.3% of downloads); warm scan ~50s; revalidation ~1s per 30 packs at 16 workers.
…nsus tar --include is bsdtar-only: on GNU tar (ubuntu CI) it errors with the return code discarded, producing an empty corpus that scanned 100% clean - caught in review, invisible to the macOS validation. Extraction now uses only universal flags (extract whole, filter extensions in Python), checks the return code, and stages the replacement tree before swapping, so a failed refetch never evicts the previous copy. --revalidate treats a missing recorded ETag as drift (refetch once to establish it) and the HEAD check follows redirects taking the last ETag, matching the GET path. The scanner excludes unfetched packs from the scanned population via the .done marker (disk ground truth; unknown is never clean) and reports them as their own census line; the dead shell=True fetch() is deleted; empty-registry and missing-snapshot paths exit with messages instead of ZeroDivisionError. The workflow pins the cache sub-actions by SHA (pinact matches names exactly; the actions/cache ignore entry does not cover restore/save) and gates the save on the fetch step plus a present lockfile so a partial .census can never save under the empty-hash key, which equals the restore-keys prefix. Revalidated end-to-end locally: two packs force-refetched through the new extraction (extension-filtered trees, markers written), full census byte-identical on the break figures (701 / 14.1% / 37.3%) over 4,974 scanned packs with 9 unscannable now explicitly excluded.
Paths-filtered pull_request trigger: self-testing for census changes, and the registration path that lets workflow_dispatch resolve before the file reaches the default branch.
…ntend runtime The execution rung between the census (reads all packs) and the E2E gates (browser + backend): build_matrix.py generates one isolated vitest spec per JS-shipping pack from the census corpus, rewriting ComfyUI import specifiers to frontend aliases; matrix_runner.ts loads the pack's real entry files, drives the registration lifecycle against real node defs, runs the user-operation battery (widget edits, mute/bypass, reorder, splice, convert, reconnect), serialize/reload, and writes one JSON row per pack. Vendored from ComfyUI_ECS_Compat_Check's L2.6 matrix layer, typed to repo rules with behavior verified semantically identical before/after. Sharded 4 ways in CI after the census job (same-run corpus cache); vitest parallelizes per core within a shard - measured ~4.7s of test time per pack, 42 packs in 18s wall locally. Radar, never a gate: pack failures live in rows and the summary (no-JS packs ignored beyond a count); the job fails only mechanically, on a spec that produced no row. Generated fixtures live in gitignored src/__ecs_matrix__, excluded from test:unit; the dedicated config assigns include/exclude after the merge because mergeConfig concatenates arrays.
Shard jobs check harness integrity only (every built spec wrote its row); a new matrix-verdict job merges all shards' rows and applies the PASS criteria - loads clean >= 95%, registerNodeDef OK >= 99%, every operation >= 99% clean - with its exit code as the verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scan's false-positive/negative rate was judged too high to act on; packs are measured by executing their code (the ecosystem matrix, stacked PR) instead. The weekly job now maintains the pack corpus - registry re-pin, fetch, ETag revalidation, lockfile-keyed cache - as the matrix's input. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atrix L1 static scan dropped on the base branch; the workflow is now corpus refresh -> sharded matrix -> combined verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seven synthetic packs, each breaking one measurement channel, plus a clean control. The matrix-detection-proof job passes only if every channel fires with its exact poison message (verify_detection.py) and the combined verdict FAILs on the poison corpus. Building it surfaced two blind channels, now handled honestly: throwing extension hooks are contained by extensionService and could never fail registration - the runner now records the containment signature as hookErrors row data - and the store-vs-live desync comparator needs Vue widget-store wiring absent in this harness, so widget mutations surface via the serialized signature instead. The poison fixture is corpus data, not repo code (eslint/oxlint/knip ignores), and the generated src/__ecs_matrix__ build output is excluded from vue-tsc so a locally built fixture cannot fail typecheck. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…criteria (#15146) Follow-through on the maintainability audit: the weekly must not red (or hollow out) for reasons unrelated to custom nodes. Stacked on #15145 (-> #15110 -> #15081) so this diff is only the hardening. ## Plumbing (no measurement change) - **The runner is now typechecked** (`tsconfig` includes `scripts/registry-census/**/*.ts`; 6 real errors fixed, all inside the harness file). A frontend rename of `registerNodeDef`/`createReroute`/`graphToPrompt` now breaks the renaming PR loudly instead of red-ing the next weekly as a false ecosystem FAIL. Generated `src/__ecs_matrix__` stays excluded. - **Registry refresh survives the network**: curl `--retry 5`, falls back to the cached snapshot on failure, and refuses a snapshot that shrank >10% (API shape drift, not mass unpublishing). Previously one 5xx killed corpus -> shards -> verdict for the week. - **Corpus fetch refuses partial completeness**: curl retries; >5% failed fetches fails the run instead of green-shipping a shrunken denominator. - **Cache economics**: superseded ~1GB `registry-corpus-*` entries are pruned after each save (repo budget is 10GB shared; unpruned, the census would evict other workflows' caches within weeks). A Thursday cron keeps the cache inside GitHub's 7-day eviction window. - **`matrix-verdict` runs `if: always()`**: one failed shard degrades to a partial verdict, not a skipped check. - **build_matrix**: the rewrite regex required a non-empty relative prefix (it could hijack a pack's own `./scripts/utils.js` to OUR module - wrong-code-under-test); entry-cap truncation is now recorded in the manifest (the comment claimed it already was); missing corpus fails friendly. ## Measurement changes - **Real timers.** `setupFiles: []` was a mergeConfig-concatenation no-op, so all 1,879 packs executed under the unit suite's `vi.useFakeTimers()` by accident. Now assigned explicitly: `vitest.setup.ts` kept, fake timers dropped. Poison proof re-validated under real timers (14/14 channels). - **Criteria reworked against measured full-registry baselines** (1,879 packs, run 31563501583): | criterion | floor | baseline | |---|---|---| | entry JS loads (any entry per pack) | >= 95% | 97.4% | | **entry files load clean (new)** | >= 91% | 94.1% | | registerNodeDef OK | >= 99% | 100% | | every operation clean | >= 99% | 99.8% | | **entry-clean delta vs previous run (new)** | >= -1.5pp | metrics carried by actions cache | The old gated metric was only "pack had >=1 entry load" - 59-of-60 entries failing still counted as loaded. The delta gate catches gradual erosion (97.4 -> 95.9) that absolute floors ignore. - **Harness gate before any criterion**: the runner self-checks that the default workflow materialized (>=7 nodes, KSampler with widgets - the op-break poison proved a 0-node graph passes 15 if-guarded ops "clean"). If most packs fail self-check the verdict is **withheld as HARNESS FAILURE (exit 2)** - a degraded harness can no longer report a vacuously clean ecosystem or a false FAIL. ## Validated locally - `pnpm typecheck` green with the runner included - Poison corpus: `DETECTION PROOF PASSED` (14 checks incl. the two new self-check assertions), summarizer exit 1 naming three breaches - Real 1,879-row corpus: **PASS** exit 0, metrics.json written (anyPct 97.392 / entryPct 94.091 / worstOp load 99.84) - Simulated delta breach (prev entryPct 96.5): exit 1, `entry files clean dropped 2.4pp` - Simulated broad self-check failure: exit 2, verdict withheld Note: the fake-timer removal shifts measurement conditions; the first CI run of this PR re-baselines under real timers, and if entry-clean lands under the 91% floor the floor gets adjusted here before merge (that is what the review run is for). Deferred by design (discussed): shard autoscaling vs registry growth; registerNodeDef floor kept as a pure harness canary; V1 DEFS drift is now loud via the self-check. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The matrix shards and the detection proof execute unreviewed third-party pack JS; the default persisted checkout credential is the one thing worth stealing in those jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🎭 Playwright: ✅ 1841 passed, 0 failed📊 Browser Reports
📦 Bundle: 9.13 MB gzip 🟢 -188 BDetailsSummary
Category Glance App Entry Points — 3.71 kB (baseline 3.71 kB) • ⚪ 0 BMain entry bundles and manifests
Status: 1 added / 1 removed Graph Workspace — 1.37 MB (baseline 1.37 MB) • 🔴 +62 BGraph editor runtime, canvas, workflow orchestration
Status: 2 added / 2 removed / 1 unchanged Views & Navigation — 124 kB (baseline 124 kB) • ⚪ 0 BTop-level views, pages, and routed surfaces
Status: 13 added / 13 removed / 4 unchanged Panels & Settings — 566 kB (baseline 566 kB) • ⚪ 0 BConfiguration panels, inspectors, and settings screens
Status: 11 added / 11 removed / 15 unchanged User & Accounts — 27.5 kB (baseline 27.5 kB) • ⚪ 0 BAuthentication, profile, and account management bundles
Status: 6 added / 6 removed / 5 unchanged Editors & Dialogs — 125 kB (baseline 125 kB) • ⚪ 0 BModals, dialogs, drawers, and in-app editors
Status: 7 added / 7 removed / 1 unchanged UI Components — 67.1 kB (baseline 67.1 kB) • ⚪ 0 BReusable component library chunks
Status: 8 added / 8 removed / 6 unchanged Data & Services — 3.53 MB (baseline 3.53 MB) • 🟢 -1 BStores, services, APIs, and repositories
Status: 14 added / 14 removed / 3 unchanged Utilities & Hooks — 549 kB (baseline 549 kB) • ⚪ 0 BHelpers, composables, and utility bundles
Status: 24 added / 24 removed / 13 unchanged Vendor & Third-Party — 18.1 MB (baseline 18.1 MB) • 🟢 -18 BExternal libraries and shared vendor chunks
Status: 1 added / 1 removed / 17 unchanged Other — 14.2 MB (baseline 14.2 MB) • 🟢 -1 BBundles that do not match a named category
Status: 82 added / 82 removed / 206 unchanged ⚡ Performance Report
Show regressions
All metrics
Historical variance (last 15 runs)
Trend (last 15 commits on main)
Raw data{
"timestamp": "2026-08-21T04:26:27.882Z",
"gitSha": "0d262c4f59cc642f842f3982141b746f651e1ad2",
"branch": "benjcooley/matrix-detection-proof",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2061.0319999999547,
"styleRecalcs": 8,
"styleRecalcDurationMs": 6.301000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 557.503,
"heapDeltaBytes": 15397560,
"heapUsedBytes": 76813188,
"domNodes": -284,
"jsHeapTotalBytes": 4448256,
"scriptDurationMs": 6.199,
"eventListeners": -151,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-idle",
"durationMs": 2068.324999999959,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.73,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 477.13899999999995,
"heapDeltaBytes": 347704,
"heapUsedBytes": 62490952,
"domNodes": -284,
"jsHeapTotalBytes": 4448256,
"scriptDurationMs": 6.276000000000001,
"eventListeners": -181,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1799.2049999999722,
"styleRecalcs": 71,
"styleRecalcDurationMs": 32.972,
"layouts": 12,
"layoutDurationMs": 3.2830000000000004,
"taskDurationMs": 817.769,
"heapDeltaBytes": -3699300,
"heapUsedBytes": 58845044,
"domNodes": -283,
"jsHeapTotalBytes": 4972544,
"scriptDurationMs": 110.34400000000001,
"eventListeners": -151,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1756.7289999999502,
"styleRecalcs": 71,
"styleRecalcDurationMs": 32.292,
"layouts": 12,
"layoutDurationMs": 3.7059999999999995,
"taskDurationMs": 821.975,
"heapDeltaBytes": -6018940,
"heapUsedBytes": 56695432,
"domNodes": -281,
"jsHeapTotalBytes": 4972544,
"scriptDurationMs": 107.934,
"eventListeners": -151,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1723.3349999999632,
"styleRecalcs": 33,
"styleRecalcDurationMs": 15.807000000000002,
"layouts": 6,
"layoutDurationMs": 0.6609999999999999,
"taskDurationMs": 340.692,
"heapDeltaBytes": 3219104,
"heapUsedBytes": 65452532,
"domNodes": 77,
"jsHeapTotalBytes": 5242880,
"scriptDurationMs": 8.488000000000003,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1721.7180000000099,
"styleRecalcs": 31,
"styleRecalcDurationMs": 14.702000000000002,
"layouts": 6,
"layoutDurationMs": 0.5640000000000001,
"taskDurationMs": 327.935,
"heapDeltaBytes": 3024300,
"heapUsedBytes": 65173252,
"domNodes": 76,
"jsHeapTotalBytes": 4980736,
"scriptDurationMs": 8.045000000000002,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "dom-widget-clipping",
"durationMs": 629.4330000000059,
"styleRecalcs": 10,
"styleRecalcDurationMs": 7.204999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 368.588,
"heapDeltaBytes": 9963388,
"heapUsedBytes": 72462764,
"domNodes": 16,
"jsHeapTotalBytes": 4456448,
"scriptDurationMs": 53.559,
"eventListeners": 2,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "dom-widget-clipping",
"durationMs": 567.888000000039,
"styleRecalcs": 12,
"styleRecalcDurationMs": 7.909000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 337.99600000000004,
"heapDeltaBytes": 10615820,
"heapUsedBytes": 73370264,
"domNodes": 20,
"jsHeapTotalBytes": 4980736,
"scriptDurationMs": 51.601,
"eventListeners": 2,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-idle",
"durationMs": 2056.752999999958,
"styleRecalcs": 8,
"styleRecalcDurationMs": 8.726999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 586.884,
"heapDeltaBytes": -3678780,
"heapUsedBytes": 73330832,
"domNodes": -277,
"jsHeapTotalBytes": -1314816,
"scriptDurationMs": 12.428,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "large-graph-idle",
"durationMs": 2021.4330000000018,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.850999999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 583.119,
"heapDeltaBytes": -13966328,
"heapUsedBytes": 62611068,
"domNodes": -281,
"jsHeapTotalBytes": 3141632,
"scriptDurationMs": 11.216000000000005,
"eventListeners": -181,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-pan",
"durationMs": 2166.5419999999926,
"styleRecalcs": 70,
"styleRecalcDurationMs": 15.859000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1179.9740000000002,
"heapDeltaBytes": -3885872,
"heapUsedBytes": 73822336,
"domNodes": -233,
"jsHeapTotalBytes": -303104,
"scriptDurationMs": 333.28999999999996,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-pan",
"durationMs": 2185.2409999999054,
"styleRecalcs": 69,
"styleRecalcDurationMs": 13.231000000000003,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1174.75,
"heapDeltaBytes": 11498156,
"heapUsedBytes": 88958636,
"domNodes": -284,
"jsHeapTotalBytes": 4415488,
"scriptDurationMs": 335.35200000000003,
"eventListeners": -179,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-zoom",
"durationMs": 3130.051999999978,
"styleRecalcs": 64,
"styleRecalcDurationMs": 13.396999999999995,
"layouts": 60,
"layoutDurationMs": 7.311,
"taskDurationMs": 1302.604,
"heapDeltaBytes": -7807376,
"heapUsedBytes": 69967436,
"domNodes": 10,
"jsHeapTotalBytes": 5804032,
"scriptDurationMs": 391.137,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666696,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-zoom",
"durationMs": 3103.2519999999977,
"styleRecalcs": 64,
"styleRecalcDurationMs": 12.748999999999997,
"layouts": 60,
"layoutDurationMs": 6.83,
"taskDurationMs": 1307.0110000000002,
"heapDeltaBytes": -1425196,
"heapUsedBytes": 77256408,
"domNodes": -277,
"jsHeapTotalBytes": -2363392,
"scriptDurationMs": 368.43699999999995,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "legacy-node-drag",
"durationMs": 2171.6200000000185,
"styleRecalcs": 45,
"styleRecalcDurationMs": 8.599000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1379.085,
"heapDeltaBytes": -15252480,
"heapUsedBytes": 69819468,
"domNodes": 10,
"jsHeapTotalBytes": 7376896,
"scriptDurationMs": 447.974,
"eventListeners": 186,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "legacy-node-drag",
"durationMs": 2248.5339999999496,
"styleRecalcs": 47,
"styleRecalcDurationMs": 9.995000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1368.973,
"heapDeltaBytes": -14469580,
"heapUsedBytes": 70320636,
"domNodes": 14,
"jsHeapTotalBytes": 8425472,
"scriptDurationMs": 441.60299999999995,
"eventListeners": 184,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "minimap-idle",
"durationMs": 2026.9480000000044,
"styleRecalcs": 8,
"styleRecalcDurationMs": 7.013000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 589.5930000000001,
"heapDeltaBytes": -13146204,
"heapUsedBytes": 69896480,
"domNodes": -281,
"jsHeapTotalBytes": 4714496,
"scriptDurationMs": 13.225999999999997,
"eventListeners": -177,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "minimap-idle",
"durationMs": 2028.8679999999886,
"styleRecalcs": 7,
"styleRecalcDurationMs": 5.928000000000003,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 578.919,
"heapDeltaBytes": -12790464,
"heapUsedBytes": 70340560,
"domNodes": -285,
"jsHeapTotalBytes": 4714496,
"scriptDurationMs": 11.679000000000002,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 596.2000000000103,
"styleRecalcs": 46,
"styleRecalcDurationMs": 9.772000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 390.795,
"heapDeltaBytes": 11571804,
"heapUsedBytes": 73854556,
"domNodes": 18,
"jsHeapTotalBytes": 4718592,
"scriptDurationMs": 123.67999999999998,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 557.7150000000302,
"styleRecalcs": 46,
"styleRecalcDurationMs": 9.314,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 369.901,
"heapDeltaBytes": 11418764,
"heapUsedBytes": 74233856,
"domNodes": 18,
"jsHeapTotalBytes": 6029312,
"scriptDurationMs": 113.834,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "subgraph-idle",
"durationMs": 2038.7590000000273,
"styleRecalcs": 9,
"styleRecalcDurationMs": 8.212999999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 468.981,
"heapDeltaBytes": -5526404,
"heapUsedBytes": 57086388,
"domNodes": -280,
"jsHeapTotalBytes": 4972544,
"scriptDurationMs": 5.851999999999999,
"eventListeners": -151,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-idle",
"durationMs": 2030.2849999999353,
"styleRecalcs": 8,
"styleRecalcDurationMs": 6.995000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 469.042,
"heapDeltaBytes": 6855940,
"heapUsedBytes": 69544260,
"domNodes": -283,
"jsHeapTotalBytes": 3923968,
"scriptDurationMs": 5.764000000000002,
"eventListeners": -181,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1715.831000000037,
"styleRecalcs": 75,
"styleRecalcDurationMs": 32.967000000000006,
"layouts": 16,
"layoutDurationMs": 3.679,
"taskDurationMs": 753.737,
"heapDeltaBytes": 5778884,
"heapUsedBytes": 68501688,
"domNodes": -281,
"jsHeapTotalBytes": 4448256,
"scriptDurationMs": 81.292,
"eventListeners": -183,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1726.1150000000498,
"styleRecalcs": 74,
"styleRecalcDurationMs": 34.249,
"layouts": 16,
"layoutDurationMs": 4.141,
"taskDurationMs": 777.854,
"heapDeltaBytes": 6028052,
"heapUsedBytes": 68506596,
"domNodes": -284,
"jsHeapTotalBytes": 4710400,
"scriptDurationMs": 83.66499999999999,
"eventListeners": -183,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-transition-enter",
"durationMs": 1400.0690000000304,
"styleRecalcs": 19,
"styleRecalcDurationMs": 31.177999999999997,
"layouts": 14,
"layoutDurationMs": 14.288000000000002,
"taskDurationMs": 899.8840000000001,
"heapDeltaBytes": -7842412,
"heapUsedBytes": 84917380,
"domNodes": 13673,
"jsHeapTotalBytes": 11010048,
"scriptDurationMs": 15.395,
"eventListeners": 2375,
"totalBlockingTimeMs": 142,
"frameDurationMs": 16.666666666666696,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8281.773999999985,
"styleRecalcs": 250,
"styleRecalcDurationMs": 36.222,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 3972.852,
"heapDeltaBytes": 15919616,
"heapUsedBytes": 91563164,
"domNodes": -281,
"jsHeapTotalBytes": 4153344,
"scriptDurationMs": 1008.6409999999998,
"eventListeners": -163,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8229.939000000059,
"styleRecalcs": 251,
"styleRecalcDurationMs": 36.898,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 3983.476,
"heapDeltaBytes": -12018660,
"heapUsedBytes": 63699908,
"domNodes": -282,
"jsHeapTotalBytes": 3141632,
"scriptDurationMs": 1020.9209999999999,
"eventListeners": -163,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-idle",
"durationMs": 16149.591999999984,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 15130.125999999998,
"heapDeltaBytes": -42654664,
"heapUsedBytes": 175885864,
"domNodes": -8312,
"jsHeapTotalBytes": -22700032,
"scriptDurationMs": 87.56499999999998,
"eventListeners": -16391,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.223333333333358,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-idle",
"durationMs": 16239.584000000035,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 15155.125999999998,
"heapDeltaBytes": -31292844,
"heapUsedBytes": 180799348,
"domNodes": -8312,
"jsHeapTotalBytes": -10788864,
"scriptDurationMs": 96.664,
"eventListeners": -16391,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.776666666666763,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 19922.033999999996,
"styleRecalcs": 172,
"styleRecalcDurationMs": 16.168000000000017,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 19479.296,
"heapDeltaBytes": -26580880,
"heapUsedBytes": 194014448,
"domNodes": -8312,
"jsHeapTotalBytes": -25370624,
"scriptDurationMs": 390.618,
"eventListeners": -16389,
"totalBlockingTimeMs": 50,
"frameDurationMs": 17.77333333333336,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 20090.540000000034,
"styleRecalcs": 166,
"styleRecalcDurationMs": 16.413999999999984,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 19596.43,
"heapDeltaBytes": -22997352,
"heapUsedBytes": 192838748,
"domNodes": -8312,
"jsHeapTotalBytes": -20541440,
"scriptDurationMs": 395.683,
"eventListeners": -16389,
"totalBlockingTimeMs": 41,
"frameDurationMs": 17.779999999999927,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "workflow-execution",
"durationMs": 98.71400000008634,
"styleRecalcs": 7,
"styleRecalcDurationMs": 13.019000000000002,
"layouts": 2,
"layoutDurationMs": 0.3280000000000002,
"taskDurationMs": 72.27899999999998,
"heapDeltaBytes": 3163304,
"heapUsedBytes": 65518848,
"domNodes": 111,
"jsHeapTotalBytes": 262144,
"scriptDurationMs": 5.256,
"eventListeners": 49,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.663333333333338,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "workflow-execution",
"durationMs": 470.70099999996273,
"styleRecalcs": 17,
"styleRecalcDurationMs": 19.662,
"layouts": 3,
"layoutDurationMs": 1.043,
"taskDurationMs": 107.711,
"heapDeltaBytes": 5087676,
"heapUsedBytes": 67525808,
"domNodes": 147,
"jsHeapTotalBytes": 262144,
"scriptDurationMs": 7.705,
"eventListeners": 99,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.799999999999272
}
]
}🎨 Storybook: ✅ Built — View Storybook |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR adds a registry census pipeline. It refreshes and pins registry-pack corpora, generates isolated ecosystem matrix tests, aggregates quality verdicts, verifies detection fixtures, and runs the process through GitHub Actions. ChangesRegistry census pipeline
Estimated code review effort: 5 (Critical) | ~120 minutes Merge Risk: 🟡 Moderate · up to This PR adds broad ecosystem compatibility reporting, but several current paths can mismeasure packs, treat failed reads as clean, or omit unusable results; related workflow and input-handling issues can also produce unreliable CI outcomes. These risks could lead to false ecosystem verdicts, so the change is not merge-ready until the major findings are fixed or explicitly accepted. Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant fetch_corpus
participant build_matrix
participant Vitest
participant summarize_matrix
GitHubActions->>fetch_corpus: Refresh or restore pinned corpus
fetch_corpus->>build_matrix: Provide available pack assets
build_matrix->>Vitest: Generate and run matrix specifications
Vitest->>summarize_matrix: Upload shard rows and manifests
summarize_matrix->>GitHubActions: Return combined verdict and metrics
Possibly related PRs
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔗 Website PreviewWebsite Preview: https://comfy-website-preview-pr-15145.vercel.app This commit: https://website-frontend-rlmuvcb9c-comfyui.vercel.app Last updated: 2026-08-18T16:15:48Z for |
There was a problem hiding this comment.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-weekly-registry-census.yaml:
- Around line 35-45: Adjust the pull_request workflow path to run a lightweight
smoke build using build_matrix.py’s existing --limit option instead of fetching
the full corpus and all four shards. Keep the complete registry census for
scheduled runs, and preserve the existing registry-corpus concurrency behavior
for the full pipeline.
- Around line 209-236: Update the “Merge shard rows” download-artifact step to
continue when no matching artifacts exist and ensure the matrix-rows directory
is created before “Verdict” runs. Preserve the existing paths and invoke
summarize_matrix.py so its no-rows handling returns the documented exit code 2.
In `@scripts/registry-census/build_matrix.py`:
- Around line 115-131: In the anchor-search loop around the __PACKROOT__
replacement, simplify the conditional that checks for a web directory by
removing the redundant any(q == 'web' ...) term and its trailing conditional
expression. Retain the existing os.path.isdir(os.path.join(probe, 'web')) check
and preserve the anchor-selection behavior.
- Around line 143-144: Update the spec generation and file write around
SPEC.format so the pack value is embedded as a JSON string literal, safely
handling quotes, backslashes, and newlines instead of inserting the raw pack
name into JavaScript strings. Open the generated matrix file with
encoding='utf-8' while preserving the existing destination and write behavior.
In `@scripts/registry-census/fetch_corpus.py`:
- Around line 68-74: Update tarball_url to parse the normalized repository
hostname and compare it exactly against the supported host allowlist, rather
than using substring checks. Preserve the existing GitHub and GitLab archive URL
construction for exact github.com and gitlab.com hosts, and return None for
lookalike or unsupported hosts.
- Around line 105-108: Validate each registry pack ID before using it in
filesystem operations, rejecting traversal or otherwise invalid values at the
registry boundary. Update the destination mapping around pack_id and dest to use
a deterministic collision-resistant safe name, such as a digest of the original
ID, while retaining the original ID only as lockfile metadata; ensure cleanup
paths use this validated mapped destination.
In `@scripts/registry-census/matrix_runner.ts`:
- Around line 295-306: Replace the `as never` assertion in the
`svc.registerNodeDef` call within the DEFS loop with the actual node-definition
type, importing that type as needed and typing DEFS/object construction against
it so contract changes are checked. Apply the same replacement to the other `as
never` assertions at the corresponding node-definition registration sites around
the referenced lines, without introducing broader assertions.
- Around line 567-572: Update the generated spec and matrix runner flow so the
de-duplicated safe spec name from build_matrix.py is passed as the fourth SPEC
argument and used as the row filename basename in matrix_runner.ts. Replace the
local pack-based sanitization while preserving the existing output directory and
JSON writing behavior.
In `@scripts/registry-census/refresh_registry.py`:
- Around line 65-85: Update main() to validate the registry response schema
inside the existing guarded fetch flow, ensuring totalPages is an integer and
every nodes item is a mapping with the expected fields before pagination or
snapshot construction. Convert any malformed payload or field type into
RegistryUnavailable so the existing except path calls keep_cached(), while
preserving normal processing for valid responses.
In `@scripts/registry-census/summarize_matrix.py`:
- Around line 174-176: Guard the previous-metrics load in the prev_path branch,
catching both ValueError and OSError from opening or parsing metrics.json and
treating the result as unavailable. Ensure prev_entry remains unset so the
existing else path reports “no previous metrics” instead of propagating an
exception.
In `@scripts/registry-census/verify_detection.py`:
- Around line 25-28: Update the row-loading loop that populates rows to catch
unreadable JSON and records missing the pack key, skipping those entries instead
of terminating. Preserve the existing harness-integrity check so it reports the
condition as a MISS, and derive EXPECTED_PACKS from the fixture directory rather
than maintaining it manually.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 3d6c77fb-71db-4919-9a54-81876940d544
📒 Files selected for processing (23)
.github/workflows/ci-weekly-registry-census.yaml.gitignore.oxlintrc.jsoneslint.config.tsknip.config.tsscripts/registry-census/README.mdscripts/registry-census/build_matrix.pyscripts/registry-census/detection-proof/corpus/registry_js/clean-control/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-customnodes-throw/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-desync/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-load-throw/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-op-break/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-regdef-throw/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/poison-serialize-throw/js/main.jsscripts/registry-census/fetch_corpus.pyscripts/registry-census/matrix_runner.tsscripts/registry-census/paths.pyscripts/registry-census/refresh_registry.pyscripts/registry-census/summarize_matrix.pyscripts/registry-census/verify_detection.pytsconfig.jsonvite.config.mtsvitest.matrix.config.mts
benjcooley
left a comment
There was a problem hiding this comment.
Automated review — posted by an AI agent (Claude Code)
The full census stack landing on main (~1.9k lines: workflow, Python tooling, vitest matrix runner, lint/tsconfig plumbing).
Cleared after checking (not assumed): every litegraph/service API the runner touches exists and matches (onDeprecationWarning, alwaysRepeatWarnings, createReroute, isInput/OutputConnected, the links Map, rootGraphInternal, registerNodeDef, the Error calling extension log text); browser_tests/assets/default.json resolves identically from scripts/registry-census/ and the copied src/__ecs_matrix__/; codeload returns identical ETags for HEAD and GET, so --revalidate genuinely works; resolveJsonModule is on; and a sample of 100 registry ids contains nothing that would collide row filenames.
I also chased a cache-poisoning path from executing untrusted pack code and dropped it — ACTIONS_RUNTIME_TOKEN isn't exposed to run steps, and the matrix jobs use persist-credentials: false with no GH_TOKEN. That design is sound.
The theme: two systematic miscounts inflate the ecosystem break rate, which is the headline number this whole stack exists to produce.
| Severity | Count |
|---|---|
| BLOCKER | 2 |
| SHOULD FIX | 3 |
| TRIVIAL | 1 |
| Total | 6 |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #15175 +/- ##
==========================================
+ Coverage 79.38% 81.74% +2.35%
==========================================
Files 2214 1887 -327
Lines 123826 118923 -4903
Branches 38871 38171 -700
==========================================
- Hits 98299 97211 -1088
+ Misses 24985 21269 -3716
+ Partials 542 443 -99
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 335 files with indirect coverage changes 🚀 New features to boost your workflow:
|
… (review) The spec glob now covers .mjs entries and the corpus keeps the assets packs import (css/json/svg/woff2/png, 2MB cap) - both previously scored affected packs as load failures they never had; corpus cache key bumped to v2 so cached asset-less trees refetch once. Two new proof controls pin both fixes. The runner writes a stub row before measuring so a pack that hangs or crashes reads as counted pack noise instead of a shard harness failure, with a majority-stub harness gate; row filenames reuse the spec's deduplicated safe name; pack names embed as JSON; registry pack ids are validated before becoming filesystem deletion targets; tarball hosts match exactly; malformed registry payloads and corrupt previous-metrics fall back instead of crashing; self-check floors tighten to 6/6; concurrency is ref-scoped so PR self-tests cannot delay the weekly; verdict download tolerates missing shard artifacts; the generated src/__ecs_matrix__ dir is excluded from eslint/oxlint/knip; as-never casts replaced with signature-bound conversions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
benjcooley
left a comment
There was a problem hiding this comment.
APPROVED — posted by an AI agent (Claude Code)
(GitHub blocks a formal Approve on your own PR, so this is a comment review.)
c5e42ad7 resolves all six, and adds hardening I didn't ask for and should have.
| # | Severity | Status |
|---|---|---|
| 1 | BLOCKER | ✅ Glob is now ./packs/{safe}/**/*.{js,mjs}, pinned by a new clean-mjs-control proof pack. |
| 2 | BLOCKER | ✅ INCLUDES extended to css/json/svg/woff2/png with a 2MB cap, pinned by clean-asset-control. The important detail is the registry-corpus-v2- key bump — without it, already-cached asset-less trees would persist and the fix would silently not take effect for exactly the packs it was written for. I checked: v2 is applied consistently across restore, save, the matrix key, and the gh cache list prune. |
| 3 | SHOULD FIX | ✅ Self-check floors 4 → 6 against an expected 7, and the README no longer states a contradicting number. A 6/7 graph passing is a reasonable tolerance band. |
| 4 | SHOULD FIX | ✅ Write-ahead stub rows, so a hung/crashed pack reads as counted pack noise rather than a shard harness failure — with a majority-stub gate that still calls harness failure when most packs hang. That's the right split; I'd only suggested distinguishing the two cases, and the majority threshold is the better version. |
| 5 | SHOULD FIX | ✅ src/__ecs_matrix__/ excluded from all three — eslint, oxlint, and knip. |
| 6 | TRIVIAL | ✅ seen set over id while accumulating pages. |
Hardening beyond the review
Three of these close real holes I did not flag, and they're worth noting because they're all in the "untrusted input reaching a dangerous sink" class:
- Pack id validated (
_PACK_ID_RE) before becoming a directory name and a deletion target on refetch — an unvalidated..would have resolved to the census root. This is the one I most regret missing. - Host matching is now exact (
host == 'github.com') rather than substring ('github.com' in repo), sogithub.com.evil.exampleno longer matches the trusted-host branch. - Pack names embed as JSON (
PACK = {pack_json}) instead of interpolating into a generated string literal, so a pack name containing a quote can't break out into the spec.
Also confirmed replace('/', '_') is gone everywhere under scripts/registry-census/, so id→path handling is consistent across fetch and scan. One consequence worth being aware of rather than fixing: an id containing / now returns bad-id and is excluded — which fails safe, since excluded packs report as unscannable rather than clean.
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/registry-census/fetch_corpus.py (1)
186-202: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftValidate archive members before extraction.
Line 186 extracts every member before Lines 195-198 filter paths and sizes. A
pack archive can retain a symlink namedentry.js.build_matrix.pythen
follows that link when it opens the retained file. A compressed oversized member
can also exhaust runner disk beforeMAX_FILE_BYTESis checked.Extract only selected regular files into
staged. Reject absolute paths,
traversal paths, symlinks, hard links, and special files. Enforce per-member and
total uncompressed byte limits before writing files.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/registry-census/fetch_corpus.py` around lines 186 - 202, Update the archive handling around the extraction subprocess and staged-file walk to inspect tar members before writing them: retain only included regular files, reject absolute or traversal paths, symlinks, hard links, and special files, and enforce both MAX_FILE_BYTES per member and a cumulative uncompressed-byte limit. Extract selected members directly into staged using validated relative paths, without first unpacking the entire archive into extracted.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-weekly-registry-census.yaml:
- Around line 110-111: Update the cache-pruning command in the workflow to
restrict gh cache list to "$GITHUB_REF" using --ref, and ensure the query
examines enough entries by paginating or setting a sufficient --limit. Preserve
the existing behavior of retaining the newest cache while deleting older
matching caches through the loop.
In `@scripts/registry-census/README.md`:
- Around line 65-67: Update the incomplete-row threshold description in the
README text around “summarize_matrix.py” to state that a harness failure occurs
only when incomplete rows exceed complete rows, meaning more than half of all
rows are incomplete; describe lower incomplete counts as remaining in the normal
verdict calculation.
Apply the same fix in `@scripts/registry-census/README.md` around lines 78 - 82:
Covered by the same README accuracy correction.
In `@scripts/registry-census/refresh_registry.py`:
- Around line 70-90: Validate the registry response in the guarded refresh flow
before building the snapshot: require totalPages to be a positive integer, every
node row to be a mapping, each id to be a non-empty string, and repository
values to be strings. Convert any validation or conversion failure to
RegistryUnavailable so keep_cached() retains the previous snapshot, while
preserving deduplication and projected-field construction for valid rows.
In `@scripts/registry-census/summarize_matrix.py`:
- Around line 196-231: Guard the same-run comparison in the previous-metrics
handling block with isinstance(prev, dict) before calling prev.get('runId').
Preserve the existing “previous metrics are from this run” message for matching
dictionary metrics, and allow non-object JSON values to reach the existing
malformed-metrics branch.
---
Outside diff comments:
In `@scripts/registry-census/fetch_corpus.py`:
- Around line 186-202: Update the archive handling around the extraction
subprocess and staged-file walk to inspect tar members before writing them:
retain only included regular files, reject absolute or traversal paths,
symlinks, hard links, and special files, and enforce both MAX_FILE_BYTES per
member and a cumulative uncompressed-byte limit. Extract selected members
directly into staged using validated relative paths, without first unpacking the
entire archive into extracted.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 2f919d7c-5d60-4991-9c29-a325bd26a0a9
📒 Files selected for processing (15)
.github/workflows/ci-weekly-registry-census.yaml.oxlintrc.jsoneslint.config.tsknip.config.tsscripts/registry-census/README.mdscripts/registry-census/build_matrix.pyscripts/registry-census/detection-proof/corpus/registry_js/clean-asset-control/js/config.jsonscripts/registry-census/detection-proof/corpus/registry_js/clean-asset-control/js/main.jsscripts/registry-census/detection-proof/corpus/registry_js/clean-asset-control/js/style.cssscripts/registry-census/detection-proof/corpus/registry_js/clean-mjs-control/js/main.mjsscripts/registry-census/fetch_corpus.pyscripts/registry-census/matrix_runner.tsscripts/registry-census/refresh_registry.pyscripts/registry-census/summarize_matrix.pyscripts/registry-census/verify_detection.py
The asset size cap added a follow-stat that a broken tarball symlink turns into FileNotFoundError, and one worker exception killed the whole 5,112-pack fetch (run 31664684227, quick-connections). Symlinks are skipped outright, the stat is guarded, and any unexpected per-pack exception now degrades to that pack's 'failed' status - feeding the mass-failure gate instead of aborting the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…m/Comfy-Org/ComfyUI_frontend into benjcooley/matrix-detection-proof
|
The first cold-fetch run under the new corpus rules (31664684227) failed in the corpus job: the new asset size cap follow-stats every kept file, and a broken symlink in one pack tarball ( |
christian-byrne
left a comment
There was a problem hiding this comment.
I said in #frontend-code-reviews that I was on board with the direction and would read the code separately. This is that read. The direction still looks right to me and I want this instrument to exist. Most of what follows is about the gap between what it measures and what it gates.
The through-line
The instrument records much more than it judges, and the detection proof certifies the record rather than the judgment.
registerCustomNodes, hookErrors, desync and the per-op sig are all collected, printed, and never consulted by any criterion. The verdict is four ratios, and matrix-detection-proof passes when the combined verdict goes FAIL, not when the specific channel a poison pack targets is the one that breached. So a channel can be certified as firing while contributing nothing to the gate. Three currently are.
The sharpest instance: the widget-desync channel is structurally dead. matrix_runner.ts:200 gates on store?.getNodeWidgetIds, which exists nowhere in src/ (the store exports getNodeWidgets), and the double cast at line 360 keeps pnpm typecheck quiet about it. desync is permanently '' and the summarizer's desync column is permanently 0, which reads as "no desync in the ecosystem" rather than "not measured". The poison-desync leg still passes, because it asserts on wn, which is built from live widgets and is true either way.
That is the same defect as the vue_row_desync column in the previous compat rig, reproduced in the thing built to replace it, and it is the class of breakage with the largest install-weighted reach. Worth fixing before this gates a release.
The symlink crash, already fixed mid-review
I started this review against dbf0570, where corpus and matrix-verdict were red on run 31664684227: an unguarded FileNotFoundError on a dangling symlink in quick-connections aborted all ~5,000 packs through ex.map, and because the crash never became a failed status it also bypassed the 5% mass-failure threshold. 2e5c3ba06d fixes both halves, corpus is green again, and I have dropped that comment rather than posting it.
One thing worth carrying forward from it. The trigger was c5e42ad7b0 widening INCLUDES to css/json/svg/woff2/png, and both green runs cited in the PR description predate that commit. The description still cites them as the evidence for the current code, and it is now two commits further along again. Re-pointing those links at a run of the merge candidate would make the evidence match the claim.
Merge mechanics, before anything else
The repo is squash-merge with delete_branch_on_merge: true, and #15145 has the same head branch as this PR (dbf057027a in both). When this merges, #15145 auto-closes as unmerged and #15110 / #15081 become permanently-open zombies, since squashing gives new SHAs and GitHub never marks them merged. Close all three manually pointing here first.
Four config collisions to hand off after this lands, none blocking: #14179 (eslint.config.ts, same allowlist anchor, and it extends linting past src/ onto matrix_runner.ts), #14839 (tsconfig.json include anchor), #14484 (.gitignore .amp line, contested by three PRs), and #15170, which merges clean but turns on typecheck:scripts over matrix_runner.ts.
Also: this has no Linear issue. Given CONTRIBUTING's "deferred work must have a tracking issue" and that several items below are reasonable follow-ups rather than blockers, somewhere to hang them would help.
Two process notes
The ~1,065 lines of Python here are not linted, formatted, type-checked, or syntax-checked by anything. ci-python-validation.yaml is path-filtered to tools/devtools/** and only runs compileall. Adding scripts/registry-census/** to that filter is a one-line change and would at least catch syntax.
Second, Slack described this as a PR gate at ~14m; the workflow as written is a weekly cron plus a self-test on census-touching paths. The workflow is the better shape, but the team is expecting the other one, so worth restating where it landed.
Method, and what I dropped
Ten review passes plus CodeRabbit, then a cross-model adversarial pass over the findings. That pass refuted two of my own before they reached you: the selfCheck default-to-OK and the worst_pct initialiser both looked like fail-open defaults but are unreachable, because incomplete stubs are filtered out before either is read and completed rows always populate both fields. Dropped rather than posted. Two more were downgraded, and I have noted the corrections inline where relevant.
I also checked and am not raising: action pinning (matches .pinact.yaml, CodeRabbit's flag is wrong for this repo), the concurrency group (present and ref-scoped, which closes #11027 for this workflow up front), persist-credentials: false on both pack-executing jobs (present, and correctly absent from the two that execute nothing), tar-slip and path traversal on extraction (GNU tar refuses .., absolute paths and symlink traversal; verified empirically), SSRF (host allowlist, URL reconstructed from a regex-extracted slug rather than passed through), pack-id traversal (_PACK_ID_RE.fullmatch blocks it before rmtree), poison-corpus leakage into build/test/lint/bundle (every glob in the diff checked, none reach it), the sharding math, the floor-boundary comparisons, zero-denominator handling on entryPct and regPct (both fail closed), the runId re-run self-comparison guard, and the empty-hash cache-key guard at line 94. Those are all correct as written and the comments explaining them are accurate.
None of the above is a merge blocker on its own except the red corpus job. The desync channel and the recorded-but-not-gated set are the ones I would want resolved before this verdict is cited as a release gate, which the Community Nodes 2.0 plan has it doing for the ECS launch.
Executes the frontend JS of every registry pack (~5,100) against the commit under review and reports an advisory population verdict. It is the execution rung between the corpus and the E2E gates: registration lifecycle, a 20-operation user battery, serialize/reload, one JSON row per pack, sharded four ways.
Changes
An advisory PR check, not a required gate.
pull_requestand main-push runs coversrc/**and the matrix tooling.matrix-verdictis not inProtectMain, so red informs but does not block. Promotion requires both a required-check setting and amerge_grouptrigger; adding only the former would stall the merge queue.Warm-cache work is bounded. An exact corpus-cache hit skips the registry crawl and pack fetch entirely. Fork-owned workflow runs are skipped. The four execution shards remain the dominant cost.
Cold fetches fail closed. At least 95% of registry targets must be available before
corpus.ready.jsonis written. Cache save requires that readiness marker, not merely the lockfile also written for failed fetches. The cache namespace is bumped so no earlier partial cache can survive this fix.Corpus pins are checked in; the resolved corpus is proved at runtime.
corpus.pins.jsonnames the commit of 4,970 packs. Packs added since the last bump track their registry ref when a cache is built. The cold build publishescorpus.lock.json,corpus.ready.json, and the registry snapshot as a required provenance artifact;matrix-verdictdownloads and validates that artifact before accepting shard output. Bumping remains manual throughupdate-corpus-pins.yaml.The corpus is 0.32 GB compressed, down from 6.0 GB. Only the executable surface is kept:
.js/.mjsplus capped text assets, with empty placeholders at the path of every binary asset.Verdict (
summarize_matrix.py, exit 0/1/2) is a pure function with 31 focused unit tests (44 registry-census tests total). Missing producer keys withhold instead of merely printing a warning, and an empty operation map fails instead of reading as 100% clean. The remaining criteria cover entry loads, registration stages, contained hook errors, completed rows, per-operation cleanliness (errordesync), and entry-clean delta.Detection proof uses seven poison packs plus three clean controls. Each poison must record its own breakage and breach every criterion it targets. The verifier also checks the reverse mapping: every criterion needs poison counter-evidence or an explicit, reported exemption backed by verdict unit tests.
The final exact-head run passed provenance validation, every detection-proof row, all four execution shards, and the combined verdict. The last complete cold run built 5,180 targets, validated 4,998 available targets (96.5%), and produced a
PASSverdict over 1,907 extension-JS packs. It measured 99.1% of packs free of contained hook errors against the 98% floor and 99.8% cleanliness for the worst operation against the 99% floor.Review focus
The load-bearing earlier bug was the widget-store desync comparator calling nonexistent
getNodeWidgetIds; it now uses the real typedgetNodeWidgetsAPI and gates.The deliberate review question remains the
packs free of contained hook errors >= 98%floor. The latest complete execution measured 99.1%, leaving 1.1 percentage points of headroom.