Skip to content

ci: ecosystem matrix - execute every JS-shipping pack against the frontend runtime - #15110

Open
benjcooley wants to merge 4 commits into
benjcooley/registry-censusfrom
benjcooley/ecosystem-matrix
Open

benjcooley wants to merge 4 commits into
benjcooley/registry-censusfrom
benjcooley/ecosystem-matrix

Conversation

@benjcooley

@benjcooley benjcooley commented Aug 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The execution rung of the custom-node coverage stack: every JS-shipping registry pack's real extension code imported and driven against the real frontend runtime under vitest, with a single combined PASS/FAIL verdict over all shards. Stacked on #15081 (the corpus); retarget to main when that merges. With L1 (static idiom scan) dropped for its false-positive/negative rate, this matrix is the full-registry measurement.

Changes

  • What:
    • scripts/registry-census/build_matrix.py - one isolated vitest spec per JS-shipping pack from the corpus (--shard I/N partitions; no-JS packs skipped and reported only as a count); ComfyUI import specifiers rewritten to frontend aliases.
    • scripts/registry-census/matrix_runner.ts - loads pack entry files, runs the registration lifecycle, a 14-op user-operation battery (per-op error/signature/desync capture), serialize/reload, graphToPrompt; one JSON row per pack.
    • scripts/registry-census/summarize_matrix.py - merges all shards' rows + manifests into one table and applies the PASS criteria; its exit code is the verdict.
    • ci-weekly-registry-census.yaml - corpus job -> 4 matrix shards -> matrix-verdict. Shards check harness integrity only (every built spec wrote its row; vitest's own exit code ignored - pack code leaks unhandled rejections). The verdict job downloads all shard rows and gates.
    • vitest.matrix.config.mts, knip/eslint registration, gitignore entries.

PASS criteria

All must hold, over all shards combined; each floor sits under the measured baseline so a handful of broken pack HEADs cannot flip the verdict, while a frontend regression that breaks pack integration craters all three at once:

criterion floor measured baseline (run 31558519779)
entry JS loads clean >= 95% 97.9%
registerNodeDef OK >= 99% 100%
every operation clean >= 99% of packs ~100%

Review Focus

  • The harness-vs-ecosystem split: shard jobs fail only on missing rows (harness); matrix-verdict fails only on criteria breach (ecosystem).
  • summarize_matrix.py merges shard manifests by pack name, so skip counts cannot double-count.

🤖 Generated with Claude Code

…ntend runtime

The execution rung between the census (reads all packs) and the E2E
gates (browser + backend): build_matrix.py generates one isolated vitest
spec per JS-shipping pack from the census corpus, rewriting ComfyUI
import specifiers to frontend aliases; matrix_runner.ts loads the pack's
real entry files, drives the registration lifecycle against real node
defs, runs the user-operation battery (widget edits, mute/bypass,
reorder, splice, convert, reconnect), serialize/reload, and writes one
JSON row per pack. Vendored from ComfyUI_ECS_Compat_Check's L2.6 matrix
layer, typed to repo rules with behavior verified semantically identical
before/after.

Sharded 4 ways in CI after the census job (same-run corpus cache);
vitest parallelizes per core within a shard - measured ~4.7s of test
time per pack, 42 packs in 18s wall locally. Radar, never a gate: pack
failures live in rows and the summary (no-JS packs ignored beyond a
count); the job fails only mechanically, on a spec that produced no
row. Generated fixtures live in gitignored src/__ecs_matrix__, excluded
from test:unit; the dedicated config assigns include/exclude after the
merge because mergeConfig concatenates arrays.
@dosubot dosubot Bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Aug 12, 2026
@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5a330d2b-03dd-498c-9708-7af75d202b95

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Playwright: ✅ 1806 passed, 0 failed

📊 Browser Reports
  • chromium: View Report (✅ 1785 / ❌ 0 / ⚠️ 0 / ⏭️ 5)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

📦 Bundle Size

⏳ Size data collection in progress…

⚡ Performance Report

canvas-idle: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 55.4 MB heap
canvas-mouse-sweep: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 48.9 MB heap
canvas-zoom-sweep: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 69.9 MB heap
dom-widget-clipping: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 50.7 MB heap
large-graph-idle: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 63.0 MB heap
large-graph-pan: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 62.5 MB heap
large-graph-zoom: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 56.9 MB heap
minimap-idle: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 64.3 MB heap
subgraph-dom-widget-clipping: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 51.3 MB heap
subgraph-idle: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 67.2 MB heap
subgraph-mouse-sweep: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 51.8 MB heap
subgraph-transition-enter: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 136ms TBT · 96.1 MB heap
viewport-pan-sweep: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 65.5 MB heap
vue-large-graph-idle: · 56.3 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 161.3 MB heap
vue-large-graph-pan: · 56.3 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 216ms TBT · 165.3 MB heap
workflow-execution: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 65.3 MB heap

ℹ️ No baseline found — significance unavailable.

Absolute values
Metric Value
canvas-idle: avg frame time 17ms
canvas-idle: p95 frame time 17ms
canvas-idle: layout duration 0ms
canvas-idle: style recalc duration 8ms
canvas-idle: layout count 0
canvas-idle: style recalc count 10
canvas-idle: task duration 479ms
canvas-idle: script duration 17ms
canvas-idle: TBT 0ms
canvas-idle: heap used 55.4 MB
canvas-idle: DOM nodes 19
canvas-idle: event listeners 4
canvas-mouse-sweep: avg frame time 17ms
canvas-mouse-sweep: p95 frame time 17ms
canvas-mouse-sweep: layout duration 3ms
canvas-mouse-sweep: style recalc duration 34ms
canvas-mouse-sweep: layout count 12
canvas-mouse-sweep: style recalc count 74
canvas-mouse-sweep: task duration 943ms
canvas-mouse-sweep: script duration 128ms
canvas-mouse-sweep: TBT 0ms
canvas-mouse-sweep: heap used 48.9 MB
canvas-mouse-sweep: DOM nodes -281
canvas-mouse-sweep: event listeners -153
canvas-zoom-sweep: avg frame time 17ms
canvas-zoom-sweep: p95 frame time 17ms
canvas-zoom-sweep: layout duration 1ms
canvas-zoom-sweep: style recalc duration 16ms
canvas-zoom-sweep: layout count 6
canvas-zoom-sweep: style recalc count 32
canvas-zoom-sweep: task duration 362ms
canvas-zoom-sweep: script duration 18ms
canvas-zoom-sweep: TBT 0ms
canvas-zoom-sweep: heap used 69.9 MB
canvas-zoom-sweep: DOM nodes 77
canvas-zoom-sweep: event listeners 19
dom-widget-clipping: avg frame time 17ms
dom-widget-clipping: p95 frame time 17ms
dom-widget-clipping: layout duration 0ms
dom-widget-clipping: style recalc duration 8ms
dom-widget-clipping: layout count 0
dom-widget-clipping: style recalc count 11
dom-widget-clipping: task duration 371ms
dom-widget-clipping: script duration 59ms
dom-widget-clipping: TBT 0ms
dom-widget-clipping: heap used 50.7 MB
dom-widget-clipping: DOM nodes 18
dom-widget-clipping: event listeners 0
large-graph-idle: avg frame time 17ms
large-graph-idle: p95 frame time 17ms
large-graph-idle: layout duration 0ms
large-graph-idle: style recalc duration 8ms
large-graph-idle: layout count 0
large-graph-idle: style recalc count 9
large-graph-idle: task duration 667ms
large-graph-idle: script duration 103ms
large-graph-idle: TBT 0ms
large-graph-idle: heap used 63.0 MB
large-graph-idle: DOM nodes -281
large-graph-idle: event listeners -147
large-graph-pan: avg frame time 17ms
large-graph-pan: p95 frame time 17ms
large-graph-pan: layout duration 0ms
large-graph-pan: style recalc duration 13ms
large-graph-pan: layout count 0
large-graph-pan: style recalc count 68
large-graph-pan: task duration 1259ms
large-graph-pan: script duration 431ms
large-graph-pan: TBT 0ms
large-graph-pan: heap used 62.5 MB
large-graph-pan: DOM nodes -287
large-graph-pan: event listeners -149
large-graph-zoom: avg frame time 17ms
large-graph-zoom: p95 frame time 17ms
large-graph-zoom: layout duration 7ms
large-graph-zoom: style recalc duration 14ms
large-graph-zoom: layout count 60
large-graph-zoom: style recalc count 66
large-graph-zoom: task duration 1494ms
large-graph-zoom: script duration 539ms
large-graph-zoom: TBT 0ms
large-graph-zoom: heap used 56.9 MB
large-graph-zoom: DOM nodes -286
large-graph-zoom: event listeners -153
minimap-idle: avg frame time 17ms
minimap-idle: p95 frame time 17ms
minimap-idle: layout duration 0ms
minimap-idle: style recalc duration 7ms
minimap-idle: layout count 0
minimap-idle: style recalc count 9
minimap-idle: task duration 664ms
minimap-idle: script duration 103ms
minimap-idle: TBT 0ms
minimap-idle: heap used 64.3 MB
minimap-idle: DOM nodes -284
minimap-idle: event listeners -164
subgraph-dom-widget-clipping: avg frame time 17ms
subgraph-dom-widget-clipping: p95 frame time 17ms
subgraph-dom-widget-clipping: layout duration 0ms
subgraph-dom-widget-clipping: style recalc duration 10ms
subgraph-dom-widget-clipping: layout count 0
subgraph-dom-widget-clipping: style recalc count 47
subgraph-dom-widget-clipping: task duration 414ms
subgraph-dom-widget-clipping: script duration 124ms
subgraph-dom-widget-clipping: TBT 0ms
subgraph-dom-widget-clipping: heap used 51.3 MB
subgraph-dom-widget-clipping: DOM nodes 20
subgraph-dom-widget-clipping: event listeners 7
subgraph-idle: avg frame time 17ms
subgraph-idle: p95 frame time 17ms
subgraph-idle: layout duration 0ms
subgraph-idle: style recalc duration 7ms
subgraph-idle: layout count 0
subgraph-idle: style recalc count 9
subgraph-idle: task duration 441ms
subgraph-idle: script duration 13ms
subgraph-idle: TBT 0ms
subgraph-idle: heap used 67.2 MB
subgraph-idle: DOM nodes 18
subgraph-idle: event listeners 4
subgraph-mouse-sweep: avg frame time 17ms
subgraph-mouse-sweep: p95 frame time 17ms
subgraph-mouse-sweep: layout duration 4ms
subgraph-mouse-sweep: style recalc duration 35ms
subgraph-mouse-sweep: layout count 16
subgraph-mouse-sweep: style recalc count 76
subgraph-mouse-sweep: task duration 774ms
subgraph-mouse-sweep: script duration 89ms
subgraph-mouse-sweep: TBT 0ms
subgraph-mouse-sweep: heap used 51.8 MB
subgraph-mouse-sweep: DOM nodes -110
subgraph-mouse-sweep: event listeners -75
subgraph-transition-enter: avg frame time 17ms
subgraph-transition-enter: p95 frame time 17ms
subgraph-transition-enter: layout duration 12ms
subgraph-transition-enter: style recalc duration 29ms
subgraph-transition-enter: layout count 13
subgraph-transition-enter: style recalc count 17
subgraph-transition-enter: task duration 890ms
subgraph-transition-enter: script duration 34ms
subgraph-transition-enter: TBT 136ms
subgraph-transition-enter: heap used 96.1 MB
subgraph-transition-enter: DOM nodes 13673
subgraph-transition-enter: event listeners 2375
viewport-pan-sweep: avg frame time 17ms
viewport-pan-sweep: p95 frame time 17ms
viewport-pan-sweep: layout duration 0ms
viewport-pan-sweep: style recalc duration 36ms
viewport-pan-sweep: layout count 0
viewport-pan-sweep: style recalc count 250
viewport-pan-sweep: task duration 4394ms
viewport-pan-sweep: script duration 1391ms
viewport-pan-sweep: TBT 0ms
viewport-pan-sweep: heap used 65.5 MB
viewport-pan-sweep: DOM nodes -282
viewport-pan-sweep: event listeners -148
vue-large-graph-idle: avg frame time 18ms
vue-large-graph-idle: p95 frame time 17ms
vue-large-graph-idle: layout duration 0ms
vue-large-graph-idle: style recalc duration 0ms
vue-large-graph-idle: layout count 0
vue-large-graph-idle: style recalc count 0
vue-large-graph-idle: task duration 16754ms
vue-large-graph-idle: script duration 549ms
vue-large-graph-idle: TBT 0ms
vue-large-graph-idle: heap used 161.3 MB
vue-large-graph-idle: DOM nodes -8314
vue-large-graph-idle: event listeners -16390
vue-large-graph-pan: avg frame time 18ms
vue-large-graph-pan: p95 frame time 17ms
vue-large-graph-pan: layout duration 0ms
vue-large-graph-pan: style recalc duration 17ms
vue-large-graph-pan: layout count 0
vue-large-graph-pan: style recalc count 143
vue-large-graph-pan: task duration 20593ms
vue-large-graph-pan: script duration 876ms
vue-large-graph-pan: TBT 216ms
vue-large-graph-pan: heap used 165.3 MB
vue-large-graph-pan: DOM nodes -8314
vue-large-graph-pan: event listeners -16384
workflow-execution: avg frame time 17ms
workflow-execution: p95 frame time 17ms
workflow-execution: layout duration 1ms
workflow-execution: style recalc duration 20ms
workflow-execution: layout count 3
workflow-execution: style recalc count 13
workflow-execution: task duration 111ms
workflow-execution: script duration 9ms
workflow-execution: TBT 0ms
workflow-execution: heap used 65.3 MB
workflow-execution: DOM nodes 124
workflow-execution: event listeners 98
Raw data
{
  "timestamp": "2026-08-12T04:43:00.886Z",
  "gitSha": "d0f0be3a0959b2f2557397ac62331f5c74cb54cb",
  "branch": "benjcooley/ecosystem-matrix",
  "measurements": [
    {
      "name": "canvas-idle",
      "durationMs": 2064.1729999999825,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 7.293000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 468.55300000000005,
      "heapDeltaBytes": 5833728,
      "heapUsedBytes": 70325576,
      "domNodes": 18,
      "jsHeapTotalBytes": 24117248,
      "scriptDurationMs": 19.496,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "canvas-idle",
      "durationMs": 2018.7329999999974,
      "styleRecalcs": 10,
      "styleRecalcDurationMs": 9.442,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 489.88700000000006,
      "heapDeltaBytes": -18435436,
      "heapUsedBytes": 45858256,
      "domNodes": 20,
      "jsHeapTotalBytes": 23568384,
      "scriptDurationMs": 14.631000000000002,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "canvas-mouse-sweep",
      "durationMs": 1920.8380000000034,
      "styleRecalcs": 73,
      "styleRecalcDurationMs": 34.169,
      "layouts": 12,
      "layoutDurationMs": 3.4709999999999996,
      "taskDurationMs": 1035.721,
      "heapDeltaBytes": -12033024,
      "heapUsedBytes": 52323352,
      "domNodes": -279,
      "jsHeapTotalBytes": 24616960,
      "scriptDurationMs": 135.855,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.670000000000012,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "canvas-mouse-sweep",
      "durationMs": 1803.1210000000328,
      "styleRecalcs": 74,
      "styleRecalcDurationMs": 32.93,
      "layouts": 12,
      "layoutDurationMs": 3.107,
      "taskDurationMs": 849.3329999999999,
      "heapDeltaBytes": -14221336,
      "heapUsedBytes": 50175732,
      "domNodes": -282,
      "jsHeapTotalBytes": 23830528,
      "scriptDurationMs": 119.92200000000001,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.670000000000012,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "canvas-zoom-sweep",
      "durationMs": 1671.720999999934,
      "styleRecalcs": 32,
      "styleRecalcDurationMs": 16.033,
      "layouts": 6,
      "layoutDurationMs": 0.6120000000000001,
      "taskDurationMs": 362.323,
      "heapDeltaBytes": 9070864,
      "heapUsedBytes": 73510312,
      "domNodes": 78,
      "jsHeapTotalBytes": 24379392,
      "scriptDurationMs": 17.299000000000003,
      "eventListeners": 19,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "canvas-zoom-sweep",
      "durationMs": 1720.4840000000559,
      "styleRecalcs": 32,
      "styleRecalcDurationMs": 16.241,
      "layouts": 6,
      "layoutDurationMs": 0.6639999999999999,
      "taskDurationMs": 361.11699999999996,
      "heapDeltaBytes": 8858684,
      "heapUsedBytes": 73171636,
      "domNodes": 76,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 17.882,
      "eventListeners": 19,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "dom-widget-clipping",
      "durationMs": 558.4670000000074,
      "styleRecalcs": 10,
      "styleRecalcDurationMs": 6.4830000000000005,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 360.74100000000004,
      "heapDeltaBytes": -11059384,
      "heapUsedBytes": 53160600,
      "domNodes": 16,
      "jsHeapTotalBytes": 25165824,
      "scriptDurationMs": 58.294,
      "eventListeners": 0,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "dom-widget-clipping",
      "durationMs": 593.588000000068,
      "styleRecalcs": 12,
      "styleRecalcDurationMs": 9.250000000000002,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 381.013,
      "heapDeltaBytes": -11232028,
      "heapUsedBytes": 53064552,
      "domNodes": 20,
      "jsHeapTotalBytes": 25165824,
      "scriptDurationMs": 60.22,
      "eventListeners": 0,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "large-graph-idle",
      "durationMs": 2037.1430000000146,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 7.783999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 658.019,
      "heapDeltaBytes": 6954256,
      "heapUsedBytes": 66373004,
      "domNodes": -281,
      "jsHeapTotalBytes": 3510272,
      "scriptDurationMs": 100.93800000000002,
      "eventListeners": -147,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "large-graph-idle",
      "durationMs": 2020.762999999988,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 7.487000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 675.02,
      "heapDeltaBytes": 6313388,
      "heapUsedBytes": 65823228,
      "domNodes": -281,
      "jsHeapTotalBytes": 3510272,
      "scriptDurationMs": 105.266,
      "eventListeners": -147,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "large-graph-pan",
      "durationMs": 2171.7820000000074,
      "styleRecalcs": 67,
      "styleRecalcDurationMs": 11.730999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 1240.155,
      "heapDeltaBytes": 3650384,
      "heapUsedBytes": 65344792,
      "domNodes": -290,
      "jsHeapTotalBytes": 4489216,
      "scriptDurationMs": 417.96700000000004,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "large-graph-pan",
      "durationMs": 2150.3119999999853,
      "styleRecalcs": 69,
      "styleRecalcDurationMs": 13.367999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 1277.5939999999998,
      "heapDeltaBytes": 5126684,
      "heapUsedBytes": 65757508,
      "domNodes": -283,
      "jsHeapTotalBytes": 3440640,
      "scriptDurationMs": 443.69800000000004,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "large-graph-zoom",
      "durationMs": 3179.7639999999774,
      "styleRecalcs": 65,
      "styleRecalcDurationMs": 13.661,
      "layouts": 60,
      "layoutDurationMs": 6.896,
      "taskDurationMs": 1521.702,
      "heapDeltaBytes": -3229504,
      "heapUsedBytes": 60158564,
      "domNodes": -286,
      "jsHeapTotalBytes": 7704576,
      "scriptDurationMs": 569.0229999999999,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.699999999999818
    },
    {
      "name": "large-graph-zoom",
      "durationMs": 3180.9039999999413,
      "styleRecalcs": 66,
      "styleRecalcDurationMs": 14.650000000000004,
      "layouts": 60,
      "layoutDurationMs": 6.783000000000001,
      "taskDurationMs": 1465.8179999999998,
      "heapDeltaBytes": -2794188,
      "heapUsedBytes": 59236464,
      "domNodes": -285,
      "jsHeapTotalBytes": 6131712,
      "scriptDurationMs": 509.058,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "minimap-idle",
      "durationMs": 2039.062000000058,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 7.866999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 662.6229999999999,
      "heapDeltaBytes": 6879388,
      "heapUsedBytes": 67586244,
      "domNodes": -283,
      "jsHeapTotalBytes": 3510272,
      "scriptDurationMs": 102.944,
      "eventListeners": -181,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "minimap-idle",
      "durationMs": 2025.987999999984,
      "styleRecalcs": 8,
      "styleRecalcDurationMs": 6.855000000000003,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 666.2379999999998,
      "heapDeltaBytes": 6567200,
      "heapUsedBytes": 67259816,
      "domNodes": -284,
      "jsHeapTotalBytes": 3510272,
      "scriptDurationMs": 103.28399999999999,
      "eventListeners": -147,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "subgraph-dom-widget-clipping",
      "durationMs": 597.3580000000993,
      "styleRecalcs": 48,
      "styleRecalcDurationMs": 10.771999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 412.139,
      "heapDeltaBytes": -10934968,
      "heapUsedBytes": 53546408,
      "domNodes": 22,
      "jsHeapTotalBytes": 26476544,
      "scriptDurationMs": 118.971,
      "eventListeners": 6,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-dom-widget-clipping",
      "durationMs": 612.5549999999294,
      "styleRecalcs": 46,
      "styleRecalcDurationMs": 10.005999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 415.4789999999999,
      "heapDeltaBytes": -10670044,
      "heapUsedBytes": 53957280,
      "domNodes": 18,
      "jsHeapTotalBytes": 25427968,
      "scriptDurationMs": 128.85100000000003,
      "eventListeners": 8,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "subgraph-idle",
      "durationMs": 1991.47099999999,
      "styleRecalcs": 8,
      "styleRecalcDurationMs": 6.731000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 432.8879999999999,
      "heapDeltaBytes": 5868108,
      "heapUsedBytes": 70390040,
      "domNodes": 16,
      "jsHeapTotalBytes": 23855104,
      "scriptDurationMs": 12.822,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "subgraph-idle",
      "durationMs": 1994.6069999999736,
      "styleRecalcs": 10,
      "styleRecalcDurationMs": 7.006000000000002,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 449.94900000000007,
      "heapDeltaBytes": 5897424,
      "heapUsedBytes": 70454336,
      "domNodes": 20,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 12.303,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-mouse-sweep",
      "durationMs": 1709.4060000000582,
      "styleRecalcs": 76,
      "styleRecalcDurationMs": 36.105,
      "layouts": 16,
      "layoutDurationMs": 4.156,
      "taskDurationMs": 794.8820000000001,
      "heapDeltaBytes": -16501200,
      "heapUsedBytes": 48028984,
      "domNodes": -279,
      "jsHeapTotalBytes": 23044096,
      "scriptDurationMs": 89.088,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "subgraph-mouse-sweep",
      "durationMs": 1689.0670000000227,
      "styleRecalcs": 75,
      "styleRecalcDurationMs": 33.793,
      "layouts": 16,
      "layoutDurationMs": 4.314,
      "taskDurationMs": 752.4029999999999,
      "heapDeltaBytes": -3563924,
      "heapUsedBytes": 60686844,
      "domNodes": 60,
      "jsHeapTotalBytes": 25165824,
      "scriptDurationMs": 88.45899999999999,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-transition-enter",
      "durationMs": 1416.8829999999843,
      "styleRecalcs": 17,
      "styleRecalcDurationMs": 28.925999999999995,
      "layouts": 13,
      "layoutDurationMs": 11.931000000000001,
      "taskDurationMs": 889.9359999999998,
      "heapDeltaBytes": 8934040,
      "heapUsedBytes": 100797664,
      "domNodes": 13673,
      "jsHeapTotalBytes": 11272192,
      "scriptDurationMs": 33.81099999999999,
      "eventListeners": 2375,
      "totalBlockingTimeMs": 136,
      "frameDurationMs": 16.666666666666696,
      "p95FrameDurationMs": 16.80000000000109
    },
    {
      "name": "viewport-pan-sweep",
      "durationMs": 8263.331999999991,
      "styleRecalcs": 250,
      "styleRecalcDurationMs": 35.342999999999996,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 4362.78,
      "heapDeltaBytes": 5397792,
      "heapUsedBytes": 64816152,
      "domNodes": -282,
      "jsHeapTotalBytes": 6062080,
      "scriptDurationMs": 1352.6150000000002,
      "eventListeners": -163,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.670000000000012,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "viewport-pan-sweep",
      "durationMs": 8314.675999999963,
      "styleRecalcs": 250,
      "styleRecalcDurationMs": 35.852,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 4425.6539999999995,
      "heapDeltaBytes": 12896548,
      "heapUsedBytes": 72531212,
      "domNodes": -281,
      "jsHeapTotalBytes": 6062080,
      "scriptDurationMs": 1429.6419999999998,
      "eventListeners": -133,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.80000000000109
    },
    {
      "name": "vue-large-graph-idle",
      "durationMs": 16633.11399999998,
      "styleRecalcs": 0,
      "styleRecalcDurationMs": 0,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 16602.809999999998,
      "heapDeltaBytes": -36514064,
      "heapUsedBytes": 170619432,
      "domNodes": -8312,
      "jsHeapTotalBytes": -5971968,
      "scriptDurationMs": 544.115,
      "eventListeners": -16387,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 17.776666666666763,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "vue-large-graph-idle",
      "durationMs": 16932.32999999998,
      "styleRecalcs": 0,
      "styleRecalcDurationMs": 0,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 16904.65,
      "heapDeltaBytes": -59856592,
      "heapUsedBytes": 167558868,
      "domNodes": -8316,
      "jsHeapTotalBytes": -6496256,
      "scriptDurationMs": 553.5570000000001,
      "eventListeners": -16393,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 17.776666666666642,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "vue-large-graph-pan",
      "durationMs": 20646.919000000024,
      "styleRecalcs": 143,
      "styleRecalcDurationMs": 17.989000000000033,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 20620.78,
      "heapDeltaBytes": -28796812,
      "heapUsedBytes": 176806380,
      "domNodes": -8312,
      "jsHeapTotalBytes": -10498048,
      "scriptDurationMs": 888.296,
      "eventListeners": -16387,
      "totalBlockingTimeMs": 229,
      "frameDurationMs": 17.776666666666642,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "vue-large-graph-pan",
      "durationMs": 20596.720000000005,
      "styleRecalcs": 143,
      "styleRecalcDurationMs": 16.942999999999987,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 20565.674000000003,
      "heapDeltaBytes": -36623620,
      "heapUsedBytes": 169922072,
      "domNodes": -8316,
      "jsHeapTotalBytes": -9187328,
      "scriptDurationMs": 863.3219999999999,
      "eventListeners": -16381,
      "totalBlockingTimeMs": 203,
      "frameDurationMs": 17.776666666666642,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "workflow-execution",
      "durationMs": 474.0399999999454,
      "styleRecalcs": 12,
      "styleRecalcDurationMs": 18.285999999999998,
      "layouts": 3,
      "layoutDurationMs": 0.6869999999999999,
      "taskDurationMs": 115.55900000000001,
      "heapDeltaBytes": 5065744,
      "heapUsedBytes": 68536068,
      "domNodes": 124,
      "jsHeapTotalBytes": 4718592,
      "scriptDurationMs": 9.033,
      "eventListeners": 99,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "workflow-execution",
      "durationMs": 451.67000000003554,
      "styleRecalcs": 13,
      "styleRecalcDurationMs": 21.085,
      "layouts": 2,
      "layoutDurationMs": 0.4109999999999999,
      "taskDurationMs": 106.928,
      "heapDeltaBytes": 5041716,
      "heapUsedBytes": 68498796,
      "domNodes": 123,
      "jsHeapTotalBytes": 4980736,
      "scriptDurationMs": 8.955,
      "eventListeners": 97,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.799999999999272
    }
  ]
}

@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@                      Coverage Diff                       @@
##           benjcooley/registry-census   #15110      +/-   ##
==============================================================
+ Coverage                       80.71%   80.81%   +0.10%     
==============================================================
  Files                            1882     1873       -9     
  Lines                          122466   121954     -512     
  Branches                        37739    37569     -170     
==============================================================
- Hits                            98847    98558     -289     
+ Misses                          23056    22849     -207     
+ Partials                          563      547      -16     
Flag Coverage Δ
unit 72.52% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 29 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benjcooley

Copy link
Copy Markdown
Collaborator Author

First full-registry execution (run 31558519779, all green): census 5.6m, then 4 shards at 10.1-11.1m each - the ~9-10m projection held.

Shard 1 as the sample: 470 JS-shipping packs executed, 470/470 rows written (the mechanical gate), 766 no-JS packs ignored beyond the count. Substance: 97.9% of pack entry JS loads clean against the current frontend runtime, registerNodeDef 470/470 OK, and the operation battery nearly spotless (1 load error, honestly recorded in its row). 12 vitest-level unhandled errors from pack code were tolerated exactly as designed while row completeness held.

Across the four shards that is roughly 1,900 real packs executed against the real frontend runtime in ~11 minutes of wall clock - the rung between the 6-minute census and the 16-minute core gate now exists and works at full registry scale on its first CI run.

benjcooley and others added 2 commits August 11, 2026 21:13
Shard jobs check harness integrity only (every built spec wrote its row);
a new matrix-verdict job merges all shards' rows and applies the PASS
criteria - loads clean >= 95%, registerNodeDef OK >= 99%, every operation
>= 99% clean - with its exit code as the verdict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atrix

L1 static scan dropped on the base branch; the workflow is now corpus
refresh -> sharded matrix -> combined verdict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@benjcooley

Copy link
Copy Markdown
Collaborator Author

Full-registry evidence for the combined-verdict design - run 31563501583, all jobs green, 13m03s end-to-end:

job time
corpus (warm restore, registry re-pin, ETag revalidation) 1m42s
ecosystem-matrix shards 1-4 (parallel) 10.1-11.0m
matrix-verdict (merge 4 shards into one table, one verdict) 11s
packs with extension JS executed: 1879 (no-JS packs ignored: 3062)
entry JS loaded         1830 ( 97.4%)
registerNodeDef         1879 of 1879 OK
registerCustomNodes     1879 of 1879 OK

operation battery: 6 errors across 20 ops x 1879 packs
  load 1876/1879   addNode 1877/1879   reload 1878/1879
  (the other 17 operations: 1879/1879 clean)

PASS criteria - all must hold, all shards combined:
  entry JS loads clean                        97.4%  (floor 95%)  OK
  registerNodeDef OK                         100.0%  (floor 99%)  OK
  every operation clean (worst: load)         99.8%  (floor 99%)  OK
VERDICT: PASS

The six operation errors are real per-pack findings (ReferenceError: LGraphGroup is not defined, TypeError: Cannot read properties of undefined (reading 'setting') / ('setDirty') / ('id')), preserved in the ecosystem-matrix-shard-* and ecosystem-matrix-combined artifacts.

benjcooley added a commit that referenced this pull request Aug 13, 2026
…criteria (#15146)

Follow-through on the maintainability audit: the weekly must not red (or
hollow out) for reasons unrelated to custom nodes. Stacked on #15145 (->
#15110 -> #15081) so this diff is only the hardening.

## Plumbing (no measurement change)
- **The runner is now typechecked** (`tsconfig` includes
`scripts/registry-census/**/*.ts`; 6 real errors fixed, all inside the
harness file). A frontend rename of
`registerNodeDef`/`createReroute`/`graphToPrompt` now breaks the
renaming PR loudly instead of red-ing the next weekly as a false
ecosystem FAIL. Generated `src/__ecs_matrix__` stays excluded.
- **Registry refresh survives the network**: curl `--retry 5`, falls
back to the cached snapshot on failure, and refuses a snapshot that
shrank >10% (API shape drift, not mass unpublishing). Previously one 5xx
killed corpus -> shards -> verdict for the week.
- **Corpus fetch refuses partial completeness**: curl retries; >5%
failed fetches fails the run instead of green-shipping a shrunken
denominator.
- **Cache economics**: superseded ~1GB `registry-corpus-*` entries are
pruned after each save (repo budget is 10GB shared; unpruned, the census
would evict other workflows' caches within weeks). A Thursday cron keeps
the cache inside GitHub's 7-day eviction window.
- **`matrix-verdict` runs `if: always()`**: one failed shard degrades to
a partial verdict, not a skipped check.
- **build_matrix**: the rewrite regex required a non-empty relative
prefix (it could hijack a pack's own `./scripts/utils.js` to OUR module
- wrong-code-under-test); entry-cap truncation is now recorded in the
manifest (the comment claimed it already was); missing corpus fails
friendly.

## Measurement changes
- **Real timers.** `setupFiles: []` was a mergeConfig-concatenation
no-op, so all 1,879 packs executed under the unit suite's
`vi.useFakeTimers()` by accident. Now assigned explicitly:
`vitest.setup.ts` kept, fake timers dropped. Poison proof re-validated
under real timers (14/14 channels).
- **Criteria reworked against measured full-registry baselines** (1,879
packs, run 31563501583):

| criterion | floor | baseline |
|---|---|---|
| entry JS loads (any entry per pack) | >= 95% | 97.4% |
| **entry files load clean (new)** | >= 91% | 94.1% |
| registerNodeDef OK | >= 99% | 100% |
| every operation clean | >= 99% | 99.8% |
| **entry-clean delta vs previous run (new)** | >= -1.5pp | metrics
carried by actions cache |

The old gated metric was only "pack had >=1 entry load" - 59-of-60
entries failing still counted as loaded. The delta gate catches gradual
erosion (97.4 -> 95.9) that absolute floors ignore.
- **Harness gate before any criterion**: the runner self-checks that the
default workflow materialized (>=7 nodes, KSampler with widgets - the
op-break poison proved a 0-node graph passes 15 if-guarded ops "clean").
If most packs fail self-check the verdict is **withheld as HARNESS
FAILURE (exit 2)** - a degraded harness can no longer report a vacuously
clean ecosystem or a false FAIL.

## Validated locally
- `pnpm typecheck` green with the runner included
- Poison corpus: `DETECTION PROOF PASSED` (14 checks incl. the two new
self-check assertions), summarizer exit 1 naming three breaches
- Real 1,879-row corpus: **PASS** exit 0, metrics.json written (anyPct
97.392 / entryPct 94.091 / worstOp load 99.84)
- Simulated delta breach (prev entryPct 96.5): exit 1, `entry files
clean dropped 2.4pp`
- Simulated broad self-check failure: exit 2, verdict withheld

Note: the fake-timer removal shifts measurement conditions; the first CI
run of this PR re-baselines under real timers, and if entry-clean lands
under the 91% floor the floor gets adjusted here before merge (that is
what the review run is for).

Deferred by design (discussed): shard autoscaling vs registry growth;
registerNodeDef floor kept as a pure harness canary; V1 DEFS drift is
now loud via the self-check.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@benjcooley

Copy link
Copy Markdown
Collaborator Author

I have read and agree to the Contributor License Agreement

@christian-byrne christian-byrne self-assigned this Aug 26, 2026

@christian-byrne christian-byrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four gating correctness blockers found. The current PASS can be vacuous or incomplete.

+ (f' (no-JS packs ignored: {skipped})' if skipped else '')
]
loaded = sum(1 for r in rows if r.get('loadedOk'))
lines.append(f'{"entry JS loaded":22s} {loaded:5d} ({loaded / len(rows) * 100:5.1f}%)')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: A pack is counted as loaded when any entry imports successfully. loadedOk is a count, so one OK plus 59 failures becomes a clean pack here and can satisfy the 95% gate. Could this require every selected entry to be OK (while preserving the count separately for reporting)?

reg_measured = sum(1 for r in rows if 'registerNodeDef' in r)
reg_ok = sum(1 for r in rows if r.get('registerNodeDef') == 'OK')
reg_pct = reg_ok / reg_measured * 100 if reg_measured else 0.0
worst_op, worst_pct = 'n/a', 100.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: The operation gate defaults to 100% when no operations were observed, and desyncs collected above do not contribute to op_err. A row set with ops: {} or desync-only results therefore passes “every operation clean.” Could missing operations and either err or desync make this criterion fail?

rows.append(json.load(open(path)))
except (OSError, ValueError):
print(f'unreadable row: {name}', file=sys.stderr)
if not rows:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: This proves only that at least one row survived. The verdict never requires all four manifests or reconciles manifest pack identities with row identities, so a missing shard or a sanitized filename collision across shards can still produce PASS from a subset. Could the verdict reject missing manifests and require a one-to-one match between expected and observed packs?

try {
const { useExtensionService } = await import('@/services/extensionService')
await useExtensionService().invokeExtensionsAsync('registerCustomNodes')
row.registerCustomNodes = 'OK'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: invokeExtensionsAsync catches each extension callback rejection and only logs it, while this runner silences console.error. A thrown registerCustomNodes callback is therefore recorded as OK; beforeRegisterNodeDef failures are swallowed through the same service path. Could the harness capture lifecycle errors and include them in the row status instead of treating the resolved wrapper promise as success?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants