Skip to content

ci: harden the corpus/matrix against non-node failures; tighten pass criteria - #15146

Merged
benjcooley merged 2 commits into
benjcooley/matrix-detection-prooffrom
benjcooley/matrix-hardening
Aug 13, 2026
Merged

benjcooley merged 2 commits into
benjcooley/matrix-detection-prooffrom
benjcooley/matrix-hardening

Conversation

@benjcooley

Copy link
Copy Markdown
Collaborator

Follow-through on the maintainability audit: the weekly must not red (or hollow out) for reasons unrelated to custom nodes. Stacked on #15145 (-> #15110 -> #15081) so this diff is only the hardening.

Plumbing (no measurement change)

  • The runner is now typechecked (tsconfig includes scripts/registry-census/**/*.ts; 6 real errors fixed, all inside the harness file). A frontend rename of registerNodeDef/createReroute/graphToPrompt now breaks the renaming PR loudly instead of red-ing the next weekly as a false ecosystem FAIL. Generated src/__ecs_matrix__ stays excluded.
  • Registry refresh survives the network: curl --retry 5, falls back to the cached snapshot on failure, and refuses a snapshot that shrank >10% (API shape drift, not mass unpublishing). Previously one 5xx killed corpus -> shards -> verdict for the week.
  • Corpus fetch refuses partial completeness: curl retries; >5% failed fetches fails the run instead of green-shipping a shrunken denominator.
  • Cache economics: superseded ~1GB registry-corpus-* entries are pruned after each save (repo budget is 10GB shared; unpruned, the census would evict other workflows' caches within weeks). A Thursday cron keeps the cache inside GitHub's 7-day eviction window.
  • matrix-verdict runs if: always(): one failed shard degrades to a partial verdict, not a skipped check.
  • build_matrix: the rewrite regex required a non-empty relative prefix (it could hijack a pack's own ./scripts/utils.js to OUR module - wrong-code-under-test); entry-cap truncation is now recorded in the manifest (the comment claimed it already was); missing corpus fails friendly.

Measurement changes

  • Real timers. setupFiles: [] was a mergeConfig-concatenation no-op, so all 1,879 packs executed under the unit suite's vi.useFakeTimers() by accident. Now assigned explicitly: vitest.setup.ts kept, fake timers dropped. Poison proof re-validated under real timers (14/14 channels).
  • Criteria reworked against measured full-registry baselines (1,879 packs, run 31563501583):
criterion floor baseline
entry JS loads (any entry per pack) >= 95% 97.4%
entry files load clean (new) >= 91% 94.1%
registerNodeDef OK >= 99% 100%
every operation clean >= 99% 99.8%
entry-clean delta vs previous run (new) >= -1.5pp metrics carried by actions cache

The old gated metric was only "pack had >=1 entry load" - 59-of-60 entries failing still counted as loaded. The delta gate catches gradual erosion (97.4 -> 95.9) that absolute floors ignore.

  • Harness gate before any criterion: the runner self-checks that the default workflow materialized (>=7 nodes, KSampler with widgets - the op-break poison proved a 0-node graph passes 15 if-guarded ops "clean"). If most packs fail self-check the verdict is withheld as HARNESS FAILURE (exit 2) - a degraded harness can no longer report a vacuously clean ecosystem or a false FAIL.

Validated locally

  • pnpm typecheck green with the runner included
  • Poison corpus: DETECTION PROOF PASSED (14 checks incl. the two new self-check assertions), summarizer exit 1 naming three breaches
  • Real 1,879-row corpus: PASS exit 0, metrics.json written (anyPct 97.392 / entryPct 94.091 / worstOp load 99.84)
  • Simulated delta breach (prev entryPct 96.5): exit 1, entry files clean dropped 2.4pp
  • Simulated broad self-check failure: exit 2, verdict withheld

Note: the fake-timer removal shifts measurement conditions; the first CI run of this PR re-baselines under real timers, and if entry-clean lands under the 91% floor the floor gets adjusted here before merge (that is what the review run is for).

Deferred by design (discussed): shard autoscaling vs registry growth; registerNodeDef floor kept as a pure harness canary; V1 DEFS drift is now loud via the self-check.

…criteria

Maintainability audit follow-through. Plumbing: the runner is now under
vue-tsc (tsconfig include) so frontend renames break the harness at PR
time instead of red-ing the weekly as a false ecosystem FAIL; registry
refresh retries and falls back to the cached snapshot (with a 90% shrink
guard against API shape drift); corpus fetch retries and refuses to ship
a partial corpus when >5% of fetches fail; superseded ~1GB corpus caches
are pruned (10GB repo budget); a Thursday cron keeps the cache inside
GitHub's 7-day eviction window; matrix-verdict runs on partial shards
instead of skipping; build_matrix no longer rewrites a pack's own
scripts/* imports to ours, records entry truncation, and fails friendly
on a missing corpus.

Measurement: packs now execute under real timers (setupFiles was a
mergeConfig-concatenation no-op that smuggled in the unit suite's fake
timers). Criteria reworked against baselines measured over the full
1,879-pack registry: any-entry-loaded >= 95% (97.4), NEW entry-files-
clean >= 91% (94.1), registerNodeDef >= 99%, worst-op >= 99%, plus a
week-over-week delta gate (entry-clean must not drop >1.5pp; metrics
carried between runs by the actions cache) and a harness gate - the
runner self-checks that the default workflow materialized, and the
verdict is withheld as HARNESS FAILURE (exit 2) when most packs fail it,
so a degraded harness can never report a vacuously clean ecosystem or a
false FAIL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Aug 12, 2026
@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 55af39e9-3f18-48ad-b5f0-b03a6bb4e93c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Playwright: ✅ 1802 passed, 0 failed · 2 flaky

📊 Browser Reports
  • chromium: View Report (✅ 1781 / ❌ 0 / ⚠️ 2 / ⏭️ 5)
  • chromium-2x: View Report (✅ 2 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • chromium-0.5x: View Report (✅ 1 / ❌ 0 / ⚠️ 0 / ⏭️ 0)
  • mobile-chrome: View Report (✅ 18 / ❌ 0 / ⚠️ 0 / ⏭️ 0)

📦 Bundle Size

⏳ Size data collection in progress…

⚡ Performance Report

canvas-idle: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 55.3 MB heap
canvas-mouse-sweep: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 55.3 MB heap
canvas-zoom-sweep: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 69.9 MB heap
dom-widget-clipping: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 50.8 MB heap
large-graph-idle: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 63.8 MB heap
large-graph-pan: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 62.3 MB heap
large-graph-zoom: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 66.9 MB heap
minimap-idle: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 64.8 MB heap
subgraph-dom-widget-clipping: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 51.1 MB heap
subgraph-idle: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 55.6 MB heap
subgraph-mouse-sweep: · 60.0 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 58.0 MB heap
subgraph-transition-enter: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 149ms TBT · 94.5 MB heap
viewport-pan-sweep: · 60.0 avg FPS · 59.9 P5 FPS ✅ (target: ≥52) · 0ms TBT · 69.2 MB heap
vue-large-graph-idle: · 54.6 avg FPS · 59.7 P5 FPS ✅ (target: ≥52) · 0ms TBT · 160.0 MB heap
vue-large-graph-pan: · 55.4 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 440ms TBT · 157.6 MB heap
workflow-execution: · 60.0 avg FPS · 59.5 P5 FPS ✅ (target: ≥52) · 0ms TBT · 65.7 MB heap

ℹ️ No baseline found — significance unavailable.

Absolute values
Metric Value
canvas-idle: avg frame time 17ms
canvas-idle: p95 frame time 17ms
canvas-idle: layout duration 0ms
canvas-idle: style recalc duration 9ms
canvas-idle: layout count 0
canvas-idle: style recalc count 9
canvas-idle: task duration 546ms
canvas-idle: script duration 18ms
canvas-idle: TBT 0ms
canvas-idle: heap used 55.3 MB
canvas-idle: DOM nodes -132
canvas-idle: event listeners -75
canvas-mouse-sweep: avg frame time 17ms
canvas-mouse-sweep: p95 frame time 17ms
canvas-mouse-sweep: layout duration 4ms
canvas-mouse-sweep: style recalc duration 46ms
canvas-mouse-sweep: layout count 12
canvas-mouse-sweep: style recalc count 76
canvas-mouse-sweep: task duration 942ms
canvas-mouse-sweep: script duration 134ms
canvas-mouse-sweep: TBT 0ms
canvas-mouse-sweep: heap used 55.3 MB
canvas-mouse-sweep: DOM nodes -111
canvas-mouse-sweep: event listeners -75
canvas-zoom-sweep: avg frame time 17ms
canvas-zoom-sweep: p95 frame time 17ms
canvas-zoom-sweep: layout duration 1ms
canvas-zoom-sweep: style recalc duration 18ms
canvas-zoom-sweep: layout count 6
canvas-zoom-sweep: style recalc count 32
canvas-zoom-sweep: task duration 383ms
canvas-zoom-sweep: script duration 19ms
canvas-zoom-sweep: TBT 0ms
canvas-zoom-sweep: heap used 69.9 MB
canvas-zoom-sweep: DOM nodes 78
canvas-zoom-sweep: event listeners 19
dom-widget-clipping: avg frame time 17ms
dom-widget-clipping: p95 frame time 17ms
dom-widget-clipping: layout duration 0ms
dom-widget-clipping: style recalc duration 8ms
dom-widget-clipping: layout count 0
dom-widget-clipping: style recalc count 11
dom-widget-clipping: task duration 395ms
dom-widget-clipping: script duration 62ms
dom-widget-clipping: TBT 0ms
dom-widget-clipping: heap used 50.8 MB
dom-widget-clipping: DOM nodes 18
dom-widget-clipping: event listeners 2
large-graph-idle: avg frame time 17ms
large-graph-idle: p95 frame time 17ms
large-graph-idle: layout duration 0ms
large-graph-idle: style recalc duration 8ms
large-graph-idle: layout count 0
large-graph-idle: style recalc count 8
large-graph-idle: task duration 695ms
large-graph-idle: script duration 112ms
large-graph-idle: TBT 0ms
large-graph-idle: heap used 63.8 MB
large-graph-idle: DOM nodes -281
large-graph-idle: event listeners -149
large-graph-pan: avg frame time 17ms
large-graph-pan: p95 frame time 17ms
large-graph-pan: layout duration 0ms
large-graph-pan: style recalc duration 13ms
large-graph-pan: layout count 0
large-graph-pan: style recalc count 68
large-graph-pan: task duration 1342ms
large-graph-pan: script duration 462ms
large-graph-pan: TBT 0ms
large-graph-pan: heap used 62.3 MB
large-graph-pan: DOM nodes -287
large-graph-pan: event listeners -148
large-graph-zoom: avg frame time 17ms
large-graph-zoom: p95 frame time 17ms
large-graph-zoom: layout duration 8ms
large-graph-zoom: style recalc duration 17ms
large-graph-zoom: layout count 60
large-graph-zoom: style recalc count 65
large-graph-zoom: task duration 1655ms
large-graph-zoom: script duration 588ms
large-graph-zoom: TBT 0ms
large-graph-zoom: heap used 66.9 MB
large-graph-zoom: DOM nodes -288
large-graph-zoom: event listeners -153
minimap-idle: avg frame time 17ms
minimap-idle: p95 frame time 17ms
minimap-idle: layout duration 0ms
minimap-idle: style recalc duration 8ms
minimap-idle: layout count 0
minimap-idle: style recalc count 8
minimap-idle: task duration 719ms
minimap-idle: script duration 115ms
minimap-idle: TBT 0ms
minimap-idle: heap used 64.8 MB
minimap-idle: DOM nodes -284
minimap-idle: event listeners -165
subgraph-dom-widget-clipping: avg frame time 17ms
subgraph-dom-widget-clipping: p95 frame time 17ms
subgraph-dom-widget-clipping: layout duration 0ms
subgraph-dom-widget-clipping: style recalc duration 10ms
subgraph-dom-widget-clipping: layout count 0
subgraph-dom-widget-clipping: style recalc count 46
subgraph-dom-widget-clipping: task duration 409ms
subgraph-dom-widget-clipping: script duration 123ms
subgraph-dom-widget-clipping: TBT 0ms
subgraph-dom-widget-clipping: heap used 51.1 MB
subgraph-dom-widget-clipping: DOM nodes 17
subgraph-dom-widget-clipping: event listeners 7
subgraph-idle: avg frame time 17ms
subgraph-idle: p95 frame time 17ms
subgraph-idle: layout duration 0ms
subgraph-idle: style recalc duration 9ms
subgraph-idle: layout count 0
subgraph-idle: style recalc count 9
subgraph-idle: task duration 510ms
subgraph-idle: script duration 15ms
subgraph-idle: TBT 0ms
subgraph-idle: heap used 55.6 MB
subgraph-idle: DOM nodes 8
subgraph-idle: event listeners -75
subgraph-mouse-sweep: avg frame time 17ms
subgraph-mouse-sweep: p95 frame time 17ms
subgraph-mouse-sweep: layout duration 5ms
subgraph-mouse-sweep: style recalc duration 39ms
subgraph-mouse-sweep: layout count 16
subgraph-mouse-sweep: style recalc count 75
subgraph-mouse-sweep: task duration 782ms
subgraph-mouse-sweep: script duration 94ms
subgraph-mouse-sweep: TBT 0ms
subgraph-mouse-sweep: heap used 58.0 MB
subgraph-mouse-sweep: DOM nodes 61
subgraph-mouse-sweep: event listeners 4
subgraph-transition-enter: avg frame time 17ms
subgraph-transition-enter: p95 frame time 17ms
subgraph-transition-enter: layout duration 13ms
subgraph-transition-enter: style recalc duration 31ms
subgraph-transition-enter: layout count 14
subgraph-transition-enter: style recalc count 18
subgraph-transition-enter: task duration 975ms
subgraph-transition-enter: script duration 45ms
subgraph-transition-enter: TBT 149ms
subgraph-transition-enter: heap used 94.5 MB
subgraph-transition-enter: DOM nodes 13673
subgraph-transition-enter: event listeners 2375
viewport-pan-sweep: avg frame time 17ms
viewport-pan-sweep: p95 frame time 17ms
viewport-pan-sweep: layout duration 0ms
viewport-pan-sweep: style recalc duration 38ms
viewport-pan-sweep: layout count 0
viewport-pan-sweep: style recalc count 250
viewport-pan-sweep: task duration 4785ms
viewport-pan-sweep: script duration 1541ms
viewport-pan-sweep: TBT 0ms
viewport-pan-sweep: heap used 69.2 MB
viewport-pan-sweep: DOM nodes -284
viewport-pan-sweep: event listeners -148
vue-large-graph-idle: avg frame time 18ms
vue-large-graph-idle: p95 frame time 17ms
vue-large-graph-idle: layout duration 0ms
vue-large-graph-idle: style recalc duration 0ms
vue-large-graph-idle: layout count 0
vue-large-graph-idle: style recalc count 0
vue-large-graph-idle: task duration 17625ms
vue-large-graph-idle: script duration 604ms
vue-large-graph-idle: TBT 0ms
vue-large-graph-idle: heap used 160.0 MB
vue-large-graph-idle: DOM nodes -8312
vue-large-graph-idle: event listeners -16389
vue-large-graph-pan: avg frame time 18ms
vue-large-graph-pan: p95 frame time 17ms
vue-large-graph-pan: layout duration 0ms
vue-large-graph-pan: style recalc duration 19ms
vue-large-graph-pan: layout count 0
vue-large-graph-pan: style recalc count 148
vue-large-graph-pan: task duration 21530ms
vue-large-graph-pan: script duration 942ms
vue-large-graph-pan: TBT 440ms
vue-large-graph-pan: heap used 157.6 MB
vue-large-graph-pan: DOM nodes -8314
vue-large-graph-pan: event listeners -16383
workflow-execution: avg frame time 17ms
workflow-execution: p95 frame time 17ms
workflow-execution: layout duration 1ms
workflow-execution: style recalc duration 23ms
workflow-execution: layout count 4
workflow-execution: style recalc count 17
workflow-execution: task duration 130ms
workflow-execution: script duration 13ms
workflow-execution: TBT 0ms
workflow-execution: heap used 65.7 MB
workflow-execution: DOM nodes 138
workflow-execution: event listeners 98
Raw data
{
  "timestamp": "2026-08-12T17:59:44.837Z",
  "gitSha": "a2433fdb7cba04fef9da6392562e46f24bc71566",
  "branch": "benjcooley/matrix-hardening",
  "measurements": [
    {
      "name": "canvas-idle",
      "durationMs": 2083.6530000000266,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 8.408000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 553.132,
      "heapDeltaBytes": -18601152,
      "heapUsedBytes": 45715976,
      "domNodes": -282,
      "jsHeapTotalBytes": 22781952,
      "scriptDurationMs": 17.774,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "canvas-idle",
      "durationMs": 2051.8560000000434,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 8.610000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 539.454,
      "heapDeltaBytes": 5876992,
      "heapUsedBytes": 70191200,
      "domNodes": 18,
      "jsHeapTotalBytes": 24379392,
      "scriptDurationMs": 18.825000000000003,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "canvas-mouse-sweep",
      "durationMs": 2012.7650000000017,
      "styleRecalcs": 75,
      "styleRecalcDurationMs": 46.675000000000004,
      "layouts": 12,
      "layoutDurationMs": 3.7160000000000006,
      "taskDurationMs": 968.4029999999999,
      "heapDeltaBytes": -12841672,
      "heapUsedBytes": 51663044,
      "domNodes": -281,
      "jsHeapTotalBytes": 24092672,
      "scriptDurationMs": 138.587,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "canvas-mouse-sweep",
      "durationMs": 1856.5029999999751,
      "styleRecalcs": 76,
      "styleRecalcDurationMs": 44.892,
      "layouts": 12,
      "layoutDurationMs": 3.7120000000000006,
      "taskDurationMs": 914.7449999999999,
      "heapDeltaBytes": -150356,
      "heapUsedBytes": 64220088,
      "domNodes": 59,
      "jsHeapTotalBytes": 25690112,
      "scriptDurationMs": 130.195,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "canvas-zoom-sweep",
      "durationMs": 1721.1070000000177,
      "styleRecalcs": 32,
      "styleRecalcDurationMs": 17.561,
      "layouts": 6,
      "layoutDurationMs": 0.5139999999999999,
      "taskDurationMs": 384.40999999999997,
      "heapDeltaBytes": 8853696,
      "heapUsedBytes": 73291148,
      "domNodes": 77,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 18.970000000000002,
      "eventListeners": 19,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "canvas-zoom-sweep",
      "durationMs": 1740.762000000018,
      "styleRecalcs": 31,
      "styleRecalcDurationMs": 18.285,
      "layouts": 6,
      "layoutDurationMs": 0.6839999999999999,
      "taskDurationMs": 381.41600000000005,
      "heapDeltaBytes": 8805928,
      "heapUsedBytes": 73287952,
      "domNodes": 78,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 19.455,
      "eventListeners": 19,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "dom-widget-clipping",
      "durationMs": 606.9229999999948,
      "styleRecalcs": 11,
      "styleRecalcDurationMs": 7.268,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 388.505,
      "heapDeltaBytes": -11203232,
      "heapUsedBytes": 53244132,
      "domNodes": 18,
      "jsHeapTotalBytes": 25427968,
      "scriptDurationMs": 62.832,
      "eventListeners": 2,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "dom-widget-clipping",
      "durationMs": 601.8990000000031,
      "styleRecalcs": 11,
      "styleRecalcDurationMs": 8.554999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 402.3349999999999,
      "heapDeltaBytes": -11144296,
      "heapUsedBytes": 53282464,
      "domNodes": 18,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 61.779999999999994,
      "eventListeners": 2,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "large-graph-idle",
      "durationMs": 2009.7840000000247,
      "styleRecalcs": 9,
      "styleRecalcDurationMs": 9.179000000000004,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 682.1239999999999,
      "heapDeltaBytes": 7062244,
      "heapUsedBytes": 66462000,
      "domNodes": -281,
      "jsHeapTotalBytes": 3510272,
      "scriptDurationMs": 104.682,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "large-graph-idle",
      "durationMs": 2038.7190000000146,
      "styleRecalcs": 7,
      "styleRecalcDurationMs": 6.437999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 708.5319999999999,
      "heapDeltaBytes": 7311888,
      "heapUsedBytes": 67346676,
      "domNodes": -281,
      "jsHeapTotalBytes": 4296704,
      "scriptDurationMs": 118.64200000000001,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "large-graph-pan",
      "durationMs": 2176.536999999996,
      "styleRecalcs": 69,
      "styleRecalcDurationMs": 14.249999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 1304.6470000000002,
      "heapDeltaBytes": 4081692,
      "heapUsedBytes": 64805648,
      "domNodes": -284,
      "jsHeapTotalBytes": 2916352,
      "scriptDurationMs": 441.461,
      "eventListeners": -147,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "large-graph-pan",
      "durationMs": 2262.878999999998,
      "styleRecalcs": 67,
      "styleRecalcDurationMs": 12.544,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 1379.235,
      "heapDeltaBytes": 4633264,
      "heapUsedBytes": 65776564,
      "domNodes": -289,
      "jsHeapTotalBytes": 4227072,
      "scriptDurationMs": 482.26099999999997,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "large-graph-zoom",
      "durationMs": 3314.5789999999806,
      "styleRecalcs": 65,
      "styleRecalcDurationMs": 16.045,
      "layouts": 60,
      "layoutDurationMs": 8.135,
      "taskDurationMs": 1627.186,
      "heapDeltaBytes": 5675060,
      "heapUsedBytes": 67988348,
      "domNodes": -287,
      "jsHeapTotalBytes": 5869568,
      "scriptDurationMs": 579.5310000000001,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.699999999999818
    },
    {
      "name": "large-graph-zoom",
      "durationMs": 3394.386000000054,
      "styleRecalcs": 65,
      "styleRecalcDurationMs": 17.354,
      "layouts": 60,
      "layoutDurationMs": 7.804,
      "taskDurationMs": 1682.4580000000003,
      "heapDeltaBytes": 10231896,
      "heapUsedBytes": 72356324,
      "domNodes": -289,
      "jsHeapTotalBytes": 5607424,
      "scriptDurationMs": 597.318,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.670000000000012,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "minimap-idle",
      "durationMs": 2056.3250000000153,
      "styleRecalcs": 8,
      "styleRecalcDurationMs": 8.408999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 768.754,
      "heapDeltaBytes": 7216724,
      "heapUsedBytes": 67546312,
      "domNodes": -283,
      "jsHeapTotalBytes": 3772416,
      "scriptDurationMs": 125.67,
      "eventListeners": -181,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "minimap-idle",
      "durationMs": 2036.9079999999258,
      "styleRecalcs": 8,
      "styleRecalcDurationMs": 7.795999999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 669.1010000000001,
      "heapDeltaBytes": 7559368,
      "heapUsedBytes": 68253520,
      "domNodes": -284,
      "jsHeapTotalBytes": 2723840,
      "scriptDurationMs": 104.946,
      "eventListeners": -149,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "subgraph-dom-widget-clipping",
      "durationMs": 601.1570000000006,
      "styleRecalcs": 46,
      "styleRecalcDurationMs": 10.777,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 418.30500000000006,
      "heapDeltaBytes": -10910564,
      "heapUsedBytes": 53588884,
      "domNodes": 18,
      "jsHeapTotalBytes": 26214400,
      "scriptDurationMs": 123.531,
      "eventListeners": 8,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-dom-widget-clipping",
      "durationMs": 567.5900000001093,
      "styleRecalcs": 45,
      "styleRecalcDurationMs": 9.735999999999999,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 398.90299999999996,
      "heapDeltaBytes": -11027732,
      "heapUsedBytes": 53588968,
      "domNodes": 16,
      "jsHeapTotalBytes": 26738688,
      "scriptDurationMs": 122.62100000000001,
      "eventListeners": 6,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "subgraph-idle",
      "durationMs": 2039.6939999999972,
      "styleRecalcs": 8,
      "styleRecalcDurationMs": 8.504,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 500.449,
      "heapDeltaBytes": 5827932,
      "heapUsedBytes": 70431188,
      "domNodes": 16,
      "jsHeapTotalBytes": 24641536,
      "scriptDurationMs": 15.368,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-idle",
      "durationMs": 2011.7979999999989,
      "styleRecalcs": 10,
      "styleRecalcDurationMs": 9.619000000000002,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 519.879,
      "heapDeltaBytes": -18305380,
      "heapUsedBytes": 46116128,
      "domNodes": -1,
      "jsHeapTotalBytes": 22257664,
      "scriptDurationMs": 14.022,
      "eventListeners": -153,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-mouse-sweep",
      "durationMs": 1693.7880000000405,
      "styleRecalcs": 75,
      "styleRecalcDurationMs": 36.479,
      "layouts": 16,
      "layoutDurationMs": 4.4719999999999995,
      "taskDurationMs": 769.9149999999998,
      "heapDeltaBytes": -3409456,
      "heapUsedBytes": 61009640,
      "domNodes": 60,
      "jsHeapTotalBytes": 25427968,
      "scriptDurationMs": 92.984,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333335,
      "p95FrameDurationMs": 16.800000000000182
    },
    {
      "name": "subgraph-mouse-sweep",
      "durationMs": 1713.2299999999532,
      "styleRecalcs": 75,
      "styleRecalcDurationMs": 41.481,
      "layouts": 16,
      "layoutDurationMs": 4.814000000000001,
      "taskDurationMs": 794.8779999999999,
      "heapDeltaBytes": -3589820,
      "heapUsedBytes": 60671436,
      "domNodes": 61,
      "jsHeapTotalBytes": 24903680,
      "scriptDurationMs": 95.001,
      "eventListeners": 4,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "subgraph-transition-enter",
      "durationMs": 1434.6939999999222,
      "styleRecalcs": 18,
      "styleRecalcDurationMs": 31.151999999999997,
      "layouts": 14,
      "layoutDurationMs": 13.481999999999998,
      "taskDurationMs": 974.8440000000003,
      "heapDeltaBytes": 31041528,
      "heapUsedBytes": 99113860,
      "domNodes": 13673,
      "jsHeapTotalBytes": 16252928,
      "scriptDurationMs": 44.510000000000005,
      "eventListeners": 2375,
      "totalBlockingTimeMs": 149,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "viewport-pan-sweep",
      "durationMs": 8575.391000000025,
      "styleRecalcs": 249,
      "styleRecalcDurationMs": 38.01200000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 4920.692000000001,
      "heapDeltaBytes": 17359812,
      "heapUsedBytes": 76974756,
      "domNodes": -284,
      "jsHeapTotalBytes": 7897088,
      "scriptDurationMs": 1633.764,
      "eventListeners": -133,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "viewport-pan-sweep",
      "durationMs": 8359.383000000093,
      "styleRecalcs": 250,
      "styleRecalcDurationMs": 37.914,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 4649.485000000001,
      "heapDeltaBytes": 8694984,
      "heapUsedBytes": 68116776,
      "domNodes": -284,
      "jsHeapTotalBytes": 6324224,
      "scriptDurationMs": 1447.607,
      "eventListeners": -163,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.66333333333332,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "vue-large-graph-idle",
      "durationMs": 17911.49699999994,
      "styleRecalcs": 0,
      "styleRecalcDurationMs": 0,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 17864.126,
      "heapDeltaBytes": -33103708,
      "heapUsedBytes": 167619232,
      "domNodes": -8312,
      "jsHeapTotalBytes": -7282688,
      "scriptDurationMs": 608.245,
      "eventListeners": -16391,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 18.886666666666617,
      "p95FrameDurationMs": 16.700000000000728
    },
    {
      "name": "vue-large-graph-idle",
      "durationMs": 17439.45700000006,
      "styleRecalcs": 0,
      "styleRecalcDurationMs": 0,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 17386.266999999996,
      "heapDeltaBytes": -52339048,
      "heapUsedBytes": 167866676,
      "domNodes": -8312,
      "jsHeapTotalBytes": -7278592,
      "scriptDurationMs": 599.689,
      "eventListeners": -16387,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 17.776666666666763,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "vue-large-graph-pan",
      "durationMs": 21376.580999999987,
      "styleRecalcs": 145,
      "styleRecalcDurationMs": 18.35399999999998,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 21313.078,
      "heapDeltaBytes": -52956552,
      "heapUsedBytes": 153875984,
      "domNodes": -8316,
      "jsHeapTotalBytes": -9641984,
      "scriptDurationMs": 921.387,
      "eventListeners": -16385,
      "totalBlockingTimeMs": 383,
      "frameDurationMs": 17.776666666666642,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "vue-large-graph-pan",
      "durationMs": 21811.20599999997,
      "styleRecalcs": 151,
      "styleRecalcDurationMs": 19.55000000000001,
      "layouts": 0,
      "layoutDurationMs": 0,
      "taskDurationMs": 21746.98,
      "heapDeltaBytes": -48092368,
      "heapUsedBytes": 176709760,
      "domNodes": -8312,
      "jsHeapTotalBytes": -7348224,
      "scriptDurationMs": 963.0049999999999,
      "eventListeners": -16381,
      "totalBlockingTimeMs": 497,
      "frameDurationMs": 18.333333333333332,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "workflow-execution",
      "durationMs": 490.7990000000382,
      "styleRecalcs": 22,
      "styleRecalcDurationMs": 26.659000000000002,
      "layouts": 5,
      "layoutDurationMs": 1.954,
      "taskDurationMs": 150.673,
      "heapDeltaBytes": 5519396,
      "heapUsedBytes": 69218648,
      "domNodes": 154,
      "jsHeapTotalBytes": 5505024,
      "scriptDurationMs": 15.573,
      "eventListeners": 99,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.670000000000012,
      "p95FrameDurationMs": 16.799999999999272
    },
    {
      "name": "workflow-execution",
      "durationMs": 471.05099999998856,
      "styleRecalcs": 12,
      "styleRecalcDurationMs": 18.768,
      "layouts": 2,
      "layoutDurationMs": 0.4630000000000001,
      "taskDurationMs": 109.489,
      "heapDeltaBytes": 5047684,
      "heapUsedBytes": 68636112,
      "domNodes": 121,
      "jsHeapTotalBytes": 4980736,
      "scriptDurationMs": 10.068000000000001,
      "eventListeners": 97,
      "totalBlockingTimeMs": 0,
      "frameDurationMs": 16.666666666666668,
      "p95FrameDurationMs": 16.799999999999272
    }
  ]
}

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — posted by an AI agent (Claude Code)

CI/tooling hardening for the weekly census: retries, cache fallbacks, a harness self-check gate, two new PASS criteria, a week-over-week delta gate, corpus pruning, real timers for the matrix suite.

Verified correct (not assumed): the setupFiles fix is real — base vite.config.mts does set ['./vitest.timer.setup.ts','./vitest.setup.ts'], so the old setupFiles: [] really was a mergeConfig concat no-op; tsconfig already excludes src/__ecs_matrix__; the wReorder/graph.add type changes are behaviour-preserving; String(i).split(':').pop() ?? '' is a no-op (split always yields ≥1 element); and the poison-corpus job's exit-1 assertion holds (7 packs, only poison-op-break trips the self-check, so 86% clears the 50% harness gate).

The theme of the findings: three of the guards this PR adds don't actually guard — each passes trivially in the case it was written to catch.

Severity Count
BLOCKER 3
SHOULD FIX 5
TRIVIAL 1
Total 9

Comment thread scripts/registry-census/fetch_corpus.py Resolved
Comment thread scripts/registry-census/build_matrix.py Resolved
@codecov

codecov Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@                        Coverage Diff                         @@
##           benjcooley/matrix-detection-proof   #15146   +/-   ##
==================================================================
  Coverage                              80.81%   80.81%           
==================================================================
  Files                                   1873     1873           
  Lines                                 121954   121954           
  Branches                               38028    38028           
==================================================================
  Hits                                   98558    98558           
  Misses                                 22849    22849           
  Partials                                 547      547           
Flag Coverage Δ
unit 72.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@benjcooley

Copy link
Copy Markdown
Collaborator Author

Re-baseline under real timers - run 31624366070, all seven jobs green, VERDICT: PASS, no floor adjustment needed:

criterion fake timers (before) real timers (this run) floor
entry JS loads (any entry per pack) 97.4% 97.3% 95%
entry files load clean 94.1% 94.0% 91%
registerNodeDef OK 100% 100% 99%
every operation clean (worst: load) 99.8% 99.8% 99%

The fake-timer removal moved the ecosystem numbers by ~0.1pp - measurement conditions changed safely. Corpus grew to 1,883 packs / 5,630 entries between runs (registry drift, revalidation working).

Two new instruments reporting for the first time:

  • contained hook errors: 14 pack(s) - real packs whose extension hooks throw and are silently swallowed by the app; previously invisible, now recorded per-row.
  • harness self-check OK: 1,880 of 1,883 - the 3 exceptions are the same packs with load-op errors (LGraphGroup is not defined etc.), i.e. pack-caused graph degradation, correctly attributed and nowhere near the 50% verdict-withhold threshold.

Delta gate shows n/a (no previous metrics) as expected for a first run; this run saved metrics.json to the cache, so the next run on this PR (or the weekly, post-merge) compares against 94.0%.

Review follow-ups: metrics saves carry a run_attempt suffix and the
summarizer skips the delta when the restored metrics came from this same
run (a re-run can no longer compare a run against itself); the corpus
mass-failure gate divides by attempted fetches instead of the mostly
cached target list, with a >=5 floor; the import-rewrite prefix no longer
hijacks a pack's own ./scripts/ modules and again matches ./../ parent
references; metrics are written only on a PASSing verdict so a failing
run cannot ratchet the baseline down to its own eroded numbers; non-JSON
registry responses fall back to the cached snapshot; unreadable shard
manifests are skipped instead of killing the verdict; runner self-check
uses floors rather than pins on today's workflow shape; the verdict job
header documents the current five-criteria contract; cache pruning runs
only from the default branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@benjcooley benjcooley left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVED — posted by an AI agent (Claude Code)

(GitHub blocks a formal Approve on your own PR, so this is a comment review.)

95b983f9 resolves all nine. The three BLOCKERs I re-verified by execution rather than by reading:

# Severity Status
1 BLOCKER ✅ Delta self-comparison. Note the run_attempt suffix alone would not have fixed this — restore-keys: matrix-metrics- still serves attempt 1's save to attempt 2. The actual fix is the runId stamp plus the summarizer's skip branch; the unique key just stops the overwrite. Both together are correct.
2 BLOCKER ✅ Denominator is now ok+empty+failed+drifted, excluding cached, so a 100% outage of an incremental run's attempted fetches trips. The >= 5 floor is a sensible small-sample guard.
3 BLOCKER ✅ PREFIX — ran all six cases against the new regex: bare "scripts/app.js" and pack-local "./scripts/app.js" correctly NOT rewritten; "../", "./../", "/" and "../../" all correctly rewritten. Both the hijack and the regression I flagged are closed.
4 SHOULD FIX ✅ Metrics written only when not breaches, and the save step gated on steps.verdict.outcome. The comment about the restored previous metrics still sitting in the directory on FAIL is the subtle part — a file-presence guard alone would have re-saved the stale baseline under a newer key. Good catch on your side.
5 SHOULD FIX ✅ try/except (OSError, ValueError) with continue — a bad shard manifest is skipped, not fatal.
6 SHOULD FIX ✅ Prune now requires github.ref_name == github.event.repository.default_branch.
7 SHOULD FIX ✅ json.loads wrapped and re-raised as RegistryUnavailable, so an HTML body falls back to the cached snapshot.
8 SHOULD FIX ✅ Floors (4) rather than pins (7), with the rationale in the comment. Catches "nothing loaded" without failing on a legitimate workflow edit.
9 TRIVIAL ✅ Header now documents all five criteria, the harness withhold gate, and the 0/1/2 exit codes.

The measured-baseline callout in the new header (run 31624366070, real timers) is the right way to make those floors auditable later.

@benjcooley
benjcooley merged commit 1172789 into benjcooley/matrix-detection-proof Aug 13, 2026
74 checks passed
@benjcooley
benjcooley deleted the benjcooley/matrix-hardening branch August 13, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant