Repository navigation
ci: ecosystem matrix - execute every JS-shipping pack against the frontend runtime - #15110
benjcooley wants to merge 4 commits into
Conversation
…ntend runtime The execution rung between the census (reads all packs) and the E2E gates (browser + backend): build_matrix.py generates one isolated vitest spec per JS-shipping pack from the census corpus, rewriting ComfyUI import specifiers to frontend aliases; matrix_runner.ts loads the pack's real entry files, drives the registration lifecycle against real node defs, runs the user-operation battery (widget edits, mute/bypass, reorder, splice, convert, reconnect), serialize/reload, and writes one JSON row per pack. Vendored from ComfyUI_ECS_Compat_Check's L2.6 matrix layer, typed to repo rules with behavior verified semantically identical before/after. Sharded 4 ways in CI after the census job (same-run corpus cache); vitest parallelizes per core within a shard - measured ~4.7s of test time per pack, 42 packs in 18s wall locally. Radar, never a gate: pack failures live in rows and the summary (no-JS packs ignored beyond a count); the job fails only mechanically, on a spec that produced no row. Generated fixtures live in gitignored src/__ecs_matrix__, excluded from test:unit; the dedicated config assigns include/exclude after the merge because mergeConfig concatenates arrays.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🎭 Playwright: ✅ 1806 passed, 0 failed📊 Browser Reports
📦 Bundle Size
⚡ Performance Report
Absolute values
Raw data{
"timestamp": "2026-08-12T04:43:00.886Z",
"gitSha": "d0f0be3a0959b2f2557397ac62331f5c74cb54cb",
"branch": "benjcooley/ecosystem-matrix",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2064.1729999999825,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.293000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 468.55300000000005,
"heapDeltaBytes": 5833728,
"heapUsedBytes": 70325576,
"domNodes": 18,
"jsHeapTotalBytes": 24117248,
"scriptDurationMs": 19.496,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-idle",
"durationMs": 2018.7329999999974,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.442,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 489.88700000000006,
"heapDeltaBytes": -18435436,
"heapUsedBytes": 45858256,
"domNodes": 20,
"jsHeapTotalBytes": 23568384,
"scriptDurationMs": 14.631000000000002,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1920.8380000000034,
"styleRecalcs": 73,
"styleRecalcDurationMs": 34.169,
"layouts": 12,
"layoutDurationMs": 3.4709999999999996,
"taskDurationMs": 1035.721,
"heapDeltaBytes": -12033024,
"heapUsedBytes": 52323352,
"domNodes": -279,
"jsHeapTotalBytes": 24616960,
"scriptDurationMs": 135.855,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.670000000000012,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1803.1210000000328,
"styleRecalcs": 74,
"styleRecalcDurationMs": 32.93,
"layouts": 12,
"layoutDurationMs": 3.107,
"taskDurationMs": 849.3329999999999,
"heapDeltaBytes": -14221336,
"heapUsedBytes": 50175732,
"domNodes": -282,
"jsHeapTotalBytes": 23830528,
"scriptDurationMs": 119.92200000000001,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.670000000000012,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1671.720999999934,
"styleRecalcs": 32,
"styleRecalcDurationMs": 16.033,
"layouts": 6,
"layoutDurationMs": 0.6120000000000001,
"taskDurationMs": 362.323,
"heapDeltaBytes": 9070864,
"heapUsedBytes": 73510312,
"domNodes": 78,
"jsHeapTotalBytes": 24379392,
"scriptDurationMs": 17.299000000000003,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1720.4840000000559,
"styleRecalcs": 32,
"styleRecalcDurationMs": 16.241,
"layouts": 6,
"layoutDurationMs": 0.6639999999999999,
"taskDurationMs": 361.11699999999996,
"heapDeltaBytes": 8858684,
"heapUsedBytes": 73171636,
"domNodes": 76,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 17.882,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "dom-widget-clipping",
"durationMs": 558.4670000000074,
"styleRecalcs": 10,
"styleRecalcDurationMs": 6.4830000000000005,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 360.74100000000004,
"heapDeltaBytes": -11059384,
"heapUsedBytes": 53160600,
"domNodes": 16,
"jsHeapTotalBytes": 25165824,
"scriptDurationMs": 58.294,
"eventListeners": 0,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "dom-widget-clipping",
"durationMs": 593.588000000068,
"styleRecalcs": 12,
"styleRecalcDurationMs": 9.250000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 381.013,
"heapDeltaBytes": -11232028,
"heapUsedBytes": 53064552,
"domNodes": 20,
"jsHeapTotalBytes": 25165824,
"scriptDurationMs": 60.22,
"eventListeners": 0,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-idle",
"durationMs": 2037.1430000000146,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.783999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 658.019,
"heapDeltaBytes": 6954256,
"heapUsedBytes": 66373004,
"domNodes": -281,
"jsHeapTotalBytes": 3510272,
"scriptDurationMs": 100.93800000000002,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-idle",
"durationMs": 2020.762999999988,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.487000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 675.02,
"heapDeltaBytes": 6313388,
"heapUsedBytes": 65823228,
"domNodes": -281,
"jsHeapTotalBytes": 3510272,
"scriptDurationMs": 105.266,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-pan",
"durationMs": 2171.7820000000074,
"styleRecalcs": 67,
"styleRecalcDurationMs": 11.730999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1240.155,
"heapDeltaBytes": 3650384,
"heapUsedBytes": 65344792,
"domNodes": -290,
"jsHeapTotalBytes": 4489216,
"scriptDurationMs": 417.96700000000004,
"eventListeners": -149,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-pan",
"durationMs": 2150.3119999999853,
"styleRecalcs": 69,
"styleRecalcDurationMs": 13.367999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1277.5939999999998,
"heapDeltaBytes": 5126684,
"heapUsedBytes": 65757508,
"domNodes": -283,
"jsHeapTotalBytes": 3440640,
"scriptDurationMs": 443.69800000000004,
"eventListeners": -149,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "large-graph-zoom",
"durationMs": 3179.7639999999774,
"styleRecalcs": 65,
"styleRecalcDurationMs": 13.661,
"layouts": 60,
"layoutDurationMs": 6.896,
"taskDurationMs": 1521.702,
"heapDeltaBytes": -3229504,
"heapUsedBytes": 60158564,
"domNodes": -286,
"jsHeapTotalBytes": 7704576,
"scriptDurationMs": 569.0229999999999,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "large-graph-zoom",
"durationMs": 3180.9039999999413,
"styleRecalcs": 66,
"styleRecalcDurationMs": 14.650000000000004,
"layouts": 60,
"layoutDurationMs": 6.783000000000001,
"taskDurationMs": 1465.8179999999998,
"heapDeltaBytes": -2794188,
"heapUsedBytes": 59236464,
"domNodes": -285,
"jsHeapTotalBytes": 6131712,
"scriptDurationMs": 509.058,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "minimap-idle",
"durationMs": 2039.062000000058,
"styleRecalcs": 9,
"styleRecalcDurationMs": 7.866999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 662.6229999999999,
"heapDeltaBytes": 6879388,
"heapUsedBytes": 67586244,
"domNodes": -283,
"jsHeapTotalBytes": 3510272,
"scriptDurationMs": 102.944,
"eventListeners": -181,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "minimap-idle",
"durationMs": 2025.987999999984,
"styleRecalcs": 8,
"styleRecalcDurationMs": 6.855000000000003,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 666.2379999999998,
"heapDeltaBytes": 6567200,
"heapUsedBytes": 67259816,
"domNodes": -284,
"jsHeapTotalBytes": 3510272,
"scriptDurationMs": 103.28399999999999,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 597.3580000000993,
"styleRecalcs": 48,
"styleRecalcDurationMs": 10.771999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 412.139,
"heapDeltaBytes": -10934968,
"heapUsedBytes": 53546408,
"domNodes": 22,
"jsHeapTotalBytes": 26476544,
"scriptDurationMs": 118.971,
"eventListeners": 6,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 612.5549999999294,
"styleRecalcs": 46,
"styleRecalcDurationMs": 10.005999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 415.4789999999999,
"heapDeltaBytes": -10670044,
"heapUsedBytes": 53957280,
"domNodes": 18,
"jsHeapTotalBytes": 25427968,
"scriptDurationMs": 128.85100000000003,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "subgraph-idle",
"durationMs": 1991.47099999999,
"styleRecalcs": 8,
"styleRecalcDurationMs": 6.731000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 432.8879999999999,
"heapDeltaBytes": 5868108,
"heapUsedBytes": 70390040,
"domNodes": 16,
"jsHeapTotalBytes": 23855104,
"scriptDurationMs": 12.822,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-idle",
"durationMs": 1994.6069999999736,
"styleRecalcs": 10,
"styleRecalcDurationMs": 7.006000000000002,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 449.94900000000007,
"heapDeltaBytes": 5897424,
"heapUsedBytes": 70454336,
"domNodes": 20,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 12.303,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1709.4060000000582,
"styleRecalcs": 76,
"styleRecalcDurationMs": 36.105,
"layouts": 16,
"layoutDurationMs": 4.156,
"taskDurationMs": 794.8820000000001,
"heapDeltaBytes": -16501200,
"heapUsedBytes": 48028984,
"domNodes": -279,
"jsHeapTotalBytes": 23044096,
"scriptDurationMs": 89.088,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1689.0670000000227,
"styleRecalcs": 75,
"styleRecalcDurationMs": 33.793,
"layouts": 16,
"layoutDurationMs": 4.314,
"taskDurationMs": 752.4029999999999,
"heapDeltaBytes": -3563924,
"heapUsedBytes": 60686844,
"domNodes": 60,
"jsHeapTotalBytes": 25165824,
"scriptDurationMs": 88.45899999999999,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-transition-enter",
"durationMs": 1416.8829999999843,
"styleRecalcs": 17,
"styleRecalcDurationMs": 28.925999999999995,
"layouts": 13,
"layoutDurationMs": 11.931000000000001,
"taskDurationMs": 889.9359999999998,
"heapDeltaBytes": 8934040,
"heapUsedBytes": 100797664,
"domNodes": 13673,
"jsHeapTotalBytes": 11272192,
"scriptDurationMs": 33.81099999999999,
"eventListeners": 2375,
"totalBlockingTimeMs": 136,
"frameDurationMs": 16.666666666666696,
"p95FrameDurationMs": 16.80000000000109
},
{
"name": "viewport-pan-sweep",
"durationMs": 8263.331999999991,
"styleRecalcs": 250,
"styleRecalcDurationMs": 35.342999999999996,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 4362.78,
"heapDeltaBytes": 5397792,
"heapUsedBytes": 64816152,
"domNodes": -282,
"jsHeapTotalBytes": 6062080,
"scriptDurationMs": 1352.6150000000002,
"eventListeners": -163,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.670000000000012,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "viewport-pan-sweep",
"durationMs": 8314.675999999963,
"styleRecalcs": 250,
"styleRecalcDurationMs": 35.852,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 4425.6539999999995,
"heapDeltaBytes": 12896548,
"heapUsedBytes": 72531212,
"domNodes": -281,
"jsHeapTotalBytes": 6062080,
"scriptDurationMs": 1429.6419999999998,
"eventListeners": -133,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.80000000000109
},
{
"name": "vue-large-graph-idle",
"durationMs": 16633.11399999998,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 16602.809999999998,
"heapDeltaBytes": -36514064,
"heapUsedBytes": 170619432,
"domNodes": -8312,
"jsHeapTotalBytes": -5971968,
"scriptDurationMs": 544.115,
"eventListeners": -16387,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.776666666666763,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-idle",
"durationMs": 16932.32999999998,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 16904.65,
"heapDeltaBytes": -59856592,
"heapUsedBytes": 167558868,
"domNodes": -8316,
"jsHeapTotalBytes": -6496256,
"scriptDurationMs": 553.5570000000001,
"eventListeners": -16393,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.776666666666642,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 20646.919000000024,
"styleRecalcs": 143,
"styleRecalcDurationMs": 17.989000000000033,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 20620.78,
"heapDeltaBytes": -28796812,
"heapUsedBytes": 176806380,
"domNodes": -8312,
"jsHeapTotalBytes": -10498048,
"scriptDurationMs": 888.296,
"eventListeners": -16387,
"totalBlockingTimeMs": 229,
"frameDurationMs": 17.776666666666642,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "vue-large-graph-pan",
"durationMs": 20596.720000000005,
"styleRecalcs": 143,
"styleRecalcDurationMs": 16.942999999999987,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 20565.674000000003,
"heapDeltaBytes": -36623620,
"heapUsedBytes": 169922072,
"domNodes": -8316,
"jsHeapTotalBytes": -9187328,
"scriptDurationMs": 863.3219999999999,
"eventListeners": -16381,
"totalBlockingTimeMs": 203,
"frameDurationMs": 17.776666666666642,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "workflow-execution",
"durationMs": 474.0399999999454,
"styleRecalcs": 12,
"styleRecalcDurationMs": 18.285999999999998,
"layouts": 3,
"layoutDurationMs": 0.6869999999999999,
"taskDurationMs": 115.55900000000001,
"heapDeltaBytes": 5065744,
"heapUsedBytes": 68536068,
"domNodes": 124,
"jsHeapTotalBytes": 4718592,
"scriptDurationMs": 9.033,
"eventListeners": 99,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "workflow-execution",
"durationMs": 451.67000000003554,
"styleRecalcs": 13,
"styleRecalcDurationMs": 21.085,
"layouts": 2,
"layoutDurationMs": 0.4109999999999999,
"taskDurationMs": 106.928,
"heapDeltaBytes": 5041716,
"heapUsedBytes": 68498796,
"domNodes": 123,
"jsHeapTotalBytes": 4980736,
"scriptDurationMs": 8.955,
"eventListeners": 97,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.799999999999272
}
]
} |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## benjcooley/registry-census #15110 +/- ##
==============================================================
+ Coverage 80.71% 80.81% +0.10%
==============================================================
Files 1882 1873 -9
Lines 122466 121954 -512
Branches 37739 37569 -170
==============================================================
- Hits 98847 98558 -289
+ Misses 23056 22849 -207
+ Partials 563 547 -16
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
First full-registry execution (run 31558519779, all green): census 5.6m, then 4 shards at 10.1-11.1m each - the ~9-10m projection held. Shard 1 as the sample: 470 JS-shipping packs executed, 470/470 rows written (the mechanical gate), 766 no-JS packs ignored beyond the count. Substance: 97.9% of pack entry JS loads clean against the current frontend runtime, Across the four shards that is roughly 1,900 real packs executed against the real frontend runtime in ~11 minutes of wall clock - the rung between the 6-minute census and the 16-minute core gate now exists and works at full registry scale on its first CI run. |
Shard jobs check harness integrity only (every built spec wrote its row); a new matrix-verdict job merges all shards' rows and applies the PASS criteria - loads clean >= 95%, registerNodeDef OK >= 99%, every operation >= 99% clean - with its exit code as the verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atrix L1 static scan dropped on the base branch; the workflow is now corpus refresh -> sharded matrix -> combined verdict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Full-registry evidence for the combined-verdict design - run 31563501583, all jobs green, 13m03s end-to-end:
The six operation errors are real per-pack findings ( |
…criteria (#15146) Follow-through on the maintainability audit: the weekly must not red (or hollow out) for reasons unrelated to custom nodes. Stacked on #15145 (-> #15110 -> #15081) so this diff is only the hardening. ## Plumbing (no measurement change) - **The runner is now typechecked** (`tsconfig` includes `scripts/registry-census/**/*.ts`; 6 real errors fixed, all inside the harness file). A frontend rename of `registerNodeDef`/`createReroute`/`graphToPrompt` now breaks the renaming PR loudly instead of red-ing the next weekly as a false ecosystem FAIL. Generated `src/__ecs_matrix__` stays excluded. - **Registry refresh survives the network**: curl `--retry 5`, falls back to the cached snapshot on failure, and refuses a snapshot that shrank >10% (API shape drift, not mass unpublishing). Previously one 5xx killed corpus -> shards -> verdict for the week. - **Corpus fetch refuses partial completeness**: curl retries; >5% failed fetches fails the run instead of green-shipping a shrunken denominator. - **Cache economics**: superseded ~1GB `registry-corpus-*` entries are pruned after each save (repo budget is 10GB shared; unpruned, the census would evict other workflows' caches within weeks). A Thursday cron keeps the cache inside GitHub's 7-day eviction window. - **`matrix-verdict` runs `if: always()`**: one failed shard degrades to a partial verdict, not a skipped check. - **build_matrix**: the rewrite regex required a non-empty relative prefix (it could hijack a pack's own `./scripts/utils.js` to OUR module - wrong-code-under-test); entry-cap truncation is now recorded in the manifest (the comment claimed it already was); missing corpus fails friendly. ## Measurement changes - **Real timers.** `setupFiles: []` was a mergeConfig-concatenation no-op, so all 1,879 packs executed under the unit suite's `vi.useFakeTimers()` by accident. Now assigned explicitly: `vitest.setup.ts` kept, fake timers dropped. Poison proof re-validated under real timers (14/14 channels). - **Criteria reworked against measured full-registry baselines** (1,879 packs, run 31563501583): | criterion | floor | baseline | |---|---|---| | entry JS loads (any entry per pack) | >= 95% | 97.4% | | **entry files load clean (new)** | >= 91% | 94.1% | | registerNodeDef OK | >= 99% | 100% | | every operation clean | >= 99% | 99.8% | | **entry-clean delta vs previous run (new)** | >= -1.5pp | metrics carried by actions cache | The old gated metric was only "pack had >=1 entry load" - 59-of-60 entries failing still counted as loaded. The delta gate catches gradual erosion (97.4 -> 95.9) that absolute floors ignore. - **Harness gate before any criterion**: the runner self-checks that the default workflow materialized (>=7 nodes, KSampler with widgets - the op-break poison proved a 0-node graph passes 15 if-guarded ops "clean"). If most packs fail self-check the verdict is **withheld as HARNESS FAILURE (exit 2)** - a degraded harness can no longer report a vacuously clean ecosystem or a false FAIL. ## Validated locally - `pnpm typecheck` green with the runner included - Poison corpus: `DETECTION PROOF PASSED` (14 checks incl. the two new self-check assertions), summarizer exit 1 naming three breaches - Real 1,879-row corpus: **PASS** exit 0, metrics.json written (anyPct 97.392 / entryPct 94.091 / worstOp load 99.84) - Simulated delta breach (prev entryPct 96.5): exit 1, `entry files clean dropped 2.4pp` - Simulated broad self-check failure: exit 2, verdict withheld Note: the fake-timer removal shifts measurement conditions; the first CI run of this PR re-baselines under real timers, and if entry-clean lands under the 91% floor the floor gets adjusted here before merge (that is what the review run is for). Deferred by design (discussed): shard autoscaling vs registry growth; registerNodeDef floor kept as a pure harness canary; V1 DEFS drift is now loud via the self-check. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
I have read and agree to the Contributor License Agreement |
christian-byrne
left a comment
There was a problem hiding this comment.
Four gating correctness blockers found. The current PASS can be vacuous or incomplete.
| + (f' (no-JS packs ignored: {skipped})' if skipped else '') | ||
| ] | ||
| loaded = sum(1 for r in rows if r.get('loadedOk')) | ||
| lines.append(f'{"entry JS loaded":22s} {loaded:5d} ({loaded / len(rows) * 100:5.1f}%)') |
There was a problem hiding this comment.
issue: A pack is counted as loaded when any entry imports successfully. loadedOk is a count, so one OK plus 59 failures becomes a clean pack here and can satisfy the 95% gate. Could this require every selected entry to be OK (while preserving the count separately for reporting)?
| reg_measured = sum(1 for r in rows if 'registerNodeDef' in r) | ||
| reg_ok = sum(1 for r in rows if r.get('registerNodeDef') == 'OK') | ||
| reg_pct = reg_ok / reg_measured * 100 if reg_measured else 0.0 | ||
| worst_op, worst_pct = 'n/a', 100.0 |
There was a problem hiding this comment.
issue: The operation gate defaults to 100% when no operations were observed, and desyncs collected above do not contribute to op_err. A row set with ops: {} or desync-only results therefore passes “every operation clean.” Could missing operations and either err or desync make this criterion fail?
| rows.append(json.load(open(path))) | ||
| except (OSError, ValueError): | ||
| print(f'unreadable row: {name}', file=sys.stderr) | ||
| if not rows: |
There was a problem hiding this comment.
issue: This proves only that at least one row survived. The verdict never requires all four manifests or reconciles manifest pack identities with row identities, so a missing shard or a sanitized filename collision across shards can still produce PASS from a subset. Could the verdict reject missing manifests and require a one-to-one match between expected and observed packs?
| try { | ||
| const { useExtensionService } = await import('@/services/extensionService') | ||
| await useExtensionService().invokeExtensionsAsync('registerCustomNodes') | ||
| row.registerCustomNodes = 'OK' |
There was a problem hiding this comment.
issue: invokeExtensionsAsync catches each extension callback rejection and only logs it, while this runner silences console.error. A thrown registerCustomNodes callback is therefore recorded as OK; beforeRegisterNodeDef failures are swallowed through the same service path. Could the harness capture lifecycle errors and include them in the row status instead of treating the resolved wrapper promise as success?
Summary
The execution rung of the custom-node coverage stack: every JS-shipping registry pack's real extension code imported and driven against the real frontend runtime under vitest, with a single combined PASS/FAIL verdict over all shards. Stacked on #15081 (the corpus); retarget to main when that merges. With L1 (static idiom scan) dropped for its false-positive/negative rate, this matrix is the full-registry measurement.
Changes
scripts/registry-census/build_matrix.py- one isolated vitest spec per JS-shipping pack from the corpus (--shard I/Npartitions; no-JS packs skipped and reported only as a count); ComfyUI import specifiers rewritten to frontend aliases.scripts/registry-census/matrix_runner.ts- loads pack entry files, runs the registration lifecycle, a 14-op user-operation battery (per-op error/signature/desync capture), serialize/reload, graphToPrompt; one JSON row per pack.scripts/registry-census/summarize_matrix.py- merges all shards' rows + manifests into one table and applies the PASS criteria; its exit code is the verdict.ci-weekly-registry-census.yaml- corpus job -> 4 matrix shards ->matrix-verdict. Shards check harness integrity only (every built spec wrote its row; vitest's own exit code ignored - pack code leaks unhandled rejections). The verdict job downloads all shard rows and gates.vitest.matrix.config.mts, knip/eslint registration, gitignore entries.PASS criteria
All must hold, over all shards combined; each floor sits under the measured baseline so a handful of broken pack HEADs cannot flip the verdict, while a frontend regression that breaks pack integration craters all three at once:
Review Focus
matrix-verdictfails only on criteria breach (ecosystem).summarize_matrix.pymerges shard manifests by pack name, so skip counts cannot double-count.🤖 Generated with Claude Code