Repository navigation
ci: stop snapshot regeneration laundering bad baselines and dead shards - #14686
Conversation
continue-on-error is required on the update-snapshots run because a rewritten baseline surfaces as a test failure, but it also swallowed genuine breakage: if a shard died (server never came up, wrong container, Playwright crashed) no snapshots changed, merge-and-commit logged 'No changes to commit', and the run went green having done nothing. Keep the JSON report and check it. No report at all means the process never completed. Failures with zero rewritten files is a broken run, not an absence of drift. Zero matched tests stays legitimate - Playwright exits non-zero and records a top-level error for it, so neither the exit code nor the error list can be trusted alone. Parsed with node rather than jq: this step runs inside the CI container, where node is guaranteed but jq is not.
Regeneration has no notion of whether a baseline actually drifted: it rewrites whatever it captured. Screenshot capture is not bit-deterministic - repeated runs against the same container and dist produce images that differ in text glyph rasterization with layout byte-identical. Normal e2e absorbs this via retries; regeneration captures once under --update-snapshots and commits the result. So a baseline that never changed can be rewritten, and a baseline failing from a genuine visual regression can be laundered into an accepted one. The only guard was a human eyeballing binary files, which is the review binary diffs defeat. Measure each rewritten snapshot against HEAD with ImageMagick and publish the table to the step summary and the commit body, so there is a number to review instead of a PNG. Warn on rewrites under 2000 pixels, which is the rasterization-noise signature rather than an intended change. Degrades to byte sizes if compare is unavailable and never fails the job. The commit body is assembled with cat, never a heredoc: the table contains backticks and file names that must not be evaluated.
📄 Knowledge reviewDosu skipped reviewing this PR because your organization has used its |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🎭 Playwright: ✅ 1788 passed, 0 failed · 1 flaky📊 Browser Reports
📦 Bundle Size
⚡ Performance Report
Absolute values
Raw data{
"timestamp": "2026-08-04T09:58:20.941Z",
"gitSha": "47427bc946e004268b12eb2cdfe4698eb7d9bc5f",
"branch": "ci/snapshot-regeneration-guards",
"measurements": [
{
"name": "canvas-idle",
"durationMs": 2031.3400000000001,
"styleRecalcs": 8,
"styleRecalcDurationMs": 6.558999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 531.342,
"heapDeltaBytes": 5607092,
"heapUsedBytes": 69787668,
"domNodes": 16,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 18.628,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.799999999999272
},
{
"name": "canvas-idle",
"durationMs": 2037.9709999999704,
"styleRecalcs": 9,
"styleRecalcDurationMs": 8.425,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 516.2750000000001,
"heapDeltaBytes": 5634840,
"heapUsedBytes": 69772264,
"domNodes": 18,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 23.733,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1872.556999999972,
"styleRecalcs": 76,
"styleRecalcDurationMs": 40.49999999999999,
"layouts": 12,
"layoutDurationMs": 3.731,
"taskDurationMs": 867.7479999999999,
"heapDeltaBytes": -109580,
"heapUsedBytes": 63742072,
"domNodes": 58,
"jsHeapTotalBytes": 26214400,
"scriptDurationMs": 124.334,
"eventListeners": 6,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "canvas-mouse-sweep",
"durationMs": 1830.5139999999938,
"styleRecalcs": 74,
"styleRecalcDurationMs": 38.031,
"layouts": 12,
"layoutDurationMs": 3.5899999999999994,
"taskDurationMs": 940.086,
"heapDeltaBytes": -13873160,
"heapUsedBytes": 50045580,
"domNodes": -281,
"jsHeapTotalBytes": 24367104,
"scriptDurationMs": 132.078,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1743.325000000027,
"styleRecalcs": 30,
"styleRecalcDurationMs": 18.258,
"layouts": 6,
"layoutDurationMs": 0.576,
"taskDurationMs": 407.78400000000005,
"heapDeltaBytes": 8587276,
"heapUsedBytes": 72632896,
"domNodes": 76,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 23.344000000000005,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66999999999998,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "canvas-zoom-sweep",
"durationMs": 1753.4359999999651,
"styleRecalcs": 32,
"styleRecalcDurationMs": 19.891,
"layouts": 6,
"layoutDurationMs": 0.6680000000000001,
"taskDurationMs": 410.28,
"heapDeltaBytes": 8658184,
"heapUsedBytes": 72780352,
"domNodes": 78,
"jsHeapTotalBytes": 24641536,
"scriptDurationMs": 23.079999999999995,
"eventListeners": 19,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "dom-widget-clipping",
"durationMs": 578.044000000034,
"styleRecalcs": 11,
"styleRecalcDurationMs": 8.181000000000001,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 384.48199999999997,
"heapDeltaBytes": -11077528,
"heapUsedBytes": 52930616,
"domNodes": 18,
"jsHeapTotalBytes": 25427968,
"scriptDurationMs": 61.10900000000001,
"eventListeners": 2,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "dom-widget-clipping",
"durationMs": 620.0389999999061,
"styleRecalcs": 12,
"styleRecalcDurationMs": 11.883,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 409.06600000000003,
"heapDeltaBytes": -10953896,
"heapUsedBytes": 52982824,
"domNodes": 20,
"jsHeapTotalBytes": 24903680,
"scriptDurationMs": 68.801,
"eventListeners": 2,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-idle",
"durationMs": 2012.8690000000233,
"styleRecalcs": 9,
"styleRecalcDurationMs": 9.461999999999998,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 698.1170000000001,
"heapDeltaBytes": 6347708,
"heapUsedBytes": 65719124,
"domNodes": -283,
"jsHeapTotalBytes": 2736128,
"scriptDurationMs": 112.53399999999999,
"eventListeners": -147,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-idle",
"durationMs": 2023.0020000000195,
"styleRecalcs": 10,
"styleRecalcDurationMs": 10.452,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 781.3050000000001,
"heapDeltaBytes": 6573696,
"heapUsedBytes": 65856932,
"domNodes": -282,
"jsHeapTotalBytes": 3260416,
"scriptDurationMs": 125.70399999999998,
"eventListeners": -179,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "large-graph-pan",
"durationMs": 2254.8979999999688,
"styleRecalcs": 69,
"styleRecalcDurationMs": 15.496999999999996,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1371.6200000000001,
"heapDeltaBytes": 2010192,
"heapUsedBytes": 62473712,
"domNodes": -286,
"jsHeapTotalBytes": 4239360,
"scriptDurationMs": 457.464,
"eventListeners": -149,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66999999999998,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "large-graph-pan",
"durationMs": 2401.971000000003,
"styleRecalcs": 68,
"styleRecalcDurationMs": 14.796999999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 1441.1450000000002,
"heapDeltaBytes": -5136668,
"heapUsedBytes": 55458988,
"domNodes": -285,
"jsHeapTotalBytes": 4571136,
"scriptDurationMs": 491.351,
"eventListeners": -179,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "large-graph-zoom",
"durationMs": 3313.5049999999637,
"styleRecalcs": 65,
"styleRecalcDurationMs": 15.654000000000002,
"layouts": 60,
"layoutDurationMs": 7.816999999999999,
"taskDurationMs": 1575.0040000000001,
"heapDeltaBytes": 24321048,
"heapUsedBytes": 86379100,
"domNodes": 12,
"jsHeapTotalBytes": 7077888,
"scriptDurationMs": 561.327,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333332,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "large-graph-zoom",
"durationMs": 3362.2349999999415,
"styleRecalcs": 64,
"styleRecalcDurationMs": 14.705,
"layouts": 60,
"layoutDurationMs": 8.021,
"taskDurationMs": 1629.74,
"heapDeltaBytes": 5149036,
"heapUsedBytes": 68034608,
"domNodes": -287,
"jsHeapTotalBytes": 6144000,
"scriptDurationMs": 570.236,
"eventListeners": -183,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
},
{
"name": "minimap-idle",
"durationMs": 2011.4599999999996,
"styleRecalcs": 7,
"styleRecalcDurationMs": 6.619999999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 685.4529999999999,
"heapDeltaBytes": 6476252,
"heapUsedBytes": 67597120,
"domNodes": -286,
"jsHeapTotalBytes": 3522560,
"scriptDurationMs": 111.69999999999999,
"eventListeners": -151,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "minimap-idle",
"durationMs": 2075.476999999978,
"styleRecalcs": 7,
"styleRecalcDurationMs": 6.964999999999999,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 807.9640000000002,
"heapDeltaBytes": 7534836,
"heapUsedBytes": 67981856,
"domNodes": -285,
"jsHeapTotalBytes": 2473984,
"scriptDurationMs": 136.999,
"eventListeners": -149,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.670000000000012,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 594.807000000003,
"styleRecalcs": 45,
"styleRecalcDurationMs": 9.482,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 401.05400000000003,
"heapDeltaBytes": -10471600,
"heapUsedBytes": 53663084,
"domNodes": 16,
"jsHeapTotalBytes": 25952256,
"scriptDurationMs": 120.31700000000001,
"eventListeners": 8,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-dom-widget-clipping",
"durationMs": 628.05800000001,
"styleRecalcs": 46,
"styleRecalcDurationMs": 10.381,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 418.84,
"heapDeltaBytes": -10326084,
"heapUsedBytes": 53646976,
"domNodes": 18,
"jsHeapTotalBytes": 26214400,
"scriptDurationMs": 122.092,
"eventListeners": 6,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-idle",
"durationMs": 2004.9710000000118,
"styleRecalcs": 10,
"styleRecalcDurationMs": 9.655,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 475.63899999999995,
"heapDeltaBytes": 5196420,
"heapUsedBytes": 69249348,
"domNodes": 20,
"jsHeapTotalBytes": 25165824,
"scriptDurationMs": 16.087,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.66333333333335,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "subgraph-idle",
"durationMs": 2024.405999999999,
"styleRecalcs": 11,
"styleRecalcDurationMs": 10.206,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 518.568,
"heapDeltaBytes": 5171628,
"heapUsedBytes": 69340192,
"domNodes": 22,
"jsHeapTotalBytes": 25427968,
"scriptDurationMs": 20.869,
"eventListeners": 4,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1713.666000000046,
"styleRecalcs": 77,
"styleRecalcDurationMs": 38.796,
"layouts": 16,
"layoutDurationMs": 4.733,
"taskDurationMs": 877.509,
"heapDeltaBytes": -17517492,
"heapUsedBytes": 46503260,
"domNodes": 19,
"jsHeapTotalBytes": 24629248,
"scriptDurationMs": 102.79500000000002,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-mouse-sweep",
"durationMs": 1739.7110000000566,
"styleRecalcs": 75,
"styleRecalcDurationMs": 38.241,
"layouts": 16,
"layoutDurationMs": 4.534,
"taskDurationMs": 883.529,
"heapDeltaBytes": -16878548,
"heapUsedBytes": 47143136,
"domNodes": -281,
"jsHeapTotalBytes": 23580672,
"scriptDurationMs": 99.32500000000002,
"eventListeners": -153,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "subgraph-transition-enter",
"durationMs": 1401.6010000000279,
"styleRecalcs": 17,
"styleRecalcDurationMs": 32.715999999999994,
"layouts": 13,
"layoutDurationMs": 13.379999999999999,
"taskDurationMs": 1041.1380000000001,
"heapDeltaBytes": 30613704,
"heapUsedBytes": 99206260,
"domNodes": 13673,
"jsHeapTotalBytes": 15466496,
"scriptDurationMs": 40.407,
"eventListeners": 2375,
"totalBlockingTimeMs": 140,
"frameDurationMs": 16.666666666666636,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8554.101000000002,
"styleRecalcs": 249,
"styleRecalcDurationMs": 39.31,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 4797.217,
"heapDeltaBytes": 7395120,
"heapUsedBytes": 67277764,
"domNodes": -283,
"jsHeapTotalBytes": 4501504,
"scriptDurationMs": 1468.934,
"eventListeners": -161,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "viewport-pan-sweep",
"durationMs": 8823.124000000007,
"styleRecalcs": 249,
"styleRecalcDurationMs": 39.506,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 4918.768,
"heapDeltaBytes": 14569524,
"heapUsedBytes": 74175040,
"domNodes": -285,
"jsHeapTotalBytes": 4501504,
"scriptDurationMs": 1484.681,
"eventListeners": -133,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "vue-large-graph-idle",
"durationMs": 17496.185999999965,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 17480.828,
"heapDeltaBytes": -48931952,
"heapUsedBytes": 166932120,
"domNodes": -8312,
"jsHeapTotalBytes": -13307904,
"scriptDurationMs": 556.575,
"eventListeners": -16387,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.776666666666642,
"p95FrameDurationMs": 16.700000000000728
},
{
"name": "vue-large-graph-idle",
"durationMs": 17793.32599999998,
"styleRecalcs": 0,
"styleRecalcDurationMs": 0,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 17761.368000000002,
"heapDeltaBytes": -59950692,
"heapUsedBytes": 167645436,
"domNodes": -8312,
"jsHeapTotalBytes": -8331264,
"scriptDurationMs": 583.916,
"eventListeners": -16385,
"totalBlockingTimeMs": 0,
"frameDurationMs": 17.780000000000047,
"p95FrameDurationMs": 16.80000000000291
},
{
"name": "vue-large-graph-pan",
"durationMs": 21151.419999999973,
"styleRecalcs": 146,
"styleRecalcDurationMs": 19.562999999999995,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 21125.556000000004,
"heapDeltaBytes": -44092916,
"heapUsedBytes": 156115320,
"domNodes": -8312,
"jsHeapTotalBytes": -15933440,
"scriptDurationMs": 899.7249999999999,
"eventListeners": -16379,
"totalBlockingTimeMs": 357,
"frameDurationMs": 17.780000000000047,
"p95FrameDurationMs": 16.80000000000291
},
{
"name": "vue-large-graph-pan",
"durationMs": 21500.721999999994,
"styleRecalcs": 149,
"styleRecalcDurationMs": 19.89599999999997,
"layouts": 0,
"layoutDurationMs": 0,
"taskDurationMs": 21445.21,
"heapDeltaBytes": -50031520,
"heapUsedBytes": 154061024,
"domNodes": -8312,
"jsHeapTotalBytes": -10948608,
"scriptDurationMs": 924.19,
"eventListeners": -16383,
"totalBlockingTimeMs": 262,
"frameDurationMs": 18.33666666666674,
"p95FrameDurationMs": 16.80000000000291
},
{
"name": "workflow-execution",
"durationMs": 466.4189999999735,
"styleRecalcs": 14,
"styleRecalcDurationMs": 20.602000000000004,
"layouts": 3,
"layoutDurationMs": 0.685,
"taskDurationMs": 123.81,
"heapDeltaBytes": 5326876,
"heapUsedBytes": 68448084,
"domNodes": 134,
"jsHeapTotalBytes": 5242880,
"scriptDurationMs": 11.45,
"eventListeners": 99,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.800000000000182
},
{
"name": "workflow-execution",
"durationMs": 454.01700000002165,
"styleRecalcs": 11,
"styleRecalcDurationMs": 17.209000000000003,
"layouts": 3,
"layoutDurationMs": 0.7240000000000001,
"taskDurationMs": 103.97599999999998,
"heapDeltaBytes": 5066620,
"heapUsedBytes": 68257592,
"domNodes": 121,
"jsHeapTotalBytes": 5505024,
"scriptDurationMs": 8.984,
"eventListeners": 97,
"totalBlockingTimeMs": 0,
"frameDurationMs": 16.666666666666668,
"p95FrameDurationMs": 16.699999999999818
}
]
} |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## ci/fix-update-playwright-expectations-triggers #14686 +/- ##
================================================================================
Coverage 78.48% 78.48%
================================================================================
Files 1790 1790
Lines 105427 105427
Branches 33059 32185 -874
================================================================================
Hits 82742 82742
Misses 22243 22243
Partials 442 442
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
7eb94db
into
ci/fix-update-playwright-expectations-triggers
Follow-up to #14684, stacked on it (base is
ci/fix-update-playwright-expectations-triggers, so the diff here shows only these two fixes). It auto-retargets tomainwhen #14684 merges. Merge #14684 first.Two more fail-open paths in the same workflow. Same defect class as #14684's bug 1: the run reports success having not done the thing it claims. Neither is a trigger problem, which is why they are here and not there.
A. Regeneration rewrites baselines that never drifted
Screenshot capture is not bit-deterministic. 12 runs of one mobile pan test against the same container and the same
distproduced 2 distinct images, minority roughly 1-in-6, differing by 795 pixels — and those pixels are purely text glyph rasterization, with layout byte-identical.Normal e2e absorbs this because the project sets
retries: 3and re-captures until a comparison passes. Regeneration does not get that protection:--update-snapshotswrites on mismatch, so the committed bytes are one unvalidated capture rather than a stable one.The consequence that matters is not the noise. It is that this path has no notion of whether a baseline drifted at all. It rewrites whatever differs and commits it. So it will just as readily overwrite a baseline that is failing because of a genuine visual regression, laundering the regression into an accepted baseline. The only guard was a human eyeballing binary files — exactly the review that binary diffs defeat.
Fix chosen: report per-file diff magnitude. Each rewritten snapshot is measured against
HEADwith ImageMagickcompare -metric AE, and the table is published to the step summary and into the commit body, so the numbers travel with the binary diff. Rewrites under 2000 pixels raise a::warning::— that is the rasterization-noise signature, not an intended visual change.This would have caught the reported instance outright: 33 of 34 rewritten files corresponded to failing tests; the lone outlier that did not was the bad one, and it is the one that shows up as a small-magnitude rewrite.
Rejected for now: capture-twice-and-write-on-agreement. It is the more thorough fix and it removes the noise rather than reporting it. I did not take it here because it roughly doubles a regeneration that already runs ~40 minutes across 5 legs, and because it is a behavioural change to what gets committed that I cannot exercise before merging. Reporting is additive, cannot change what is committed, and turns an unreviewable binary diff into a sortable number today. Capture-twice is the right next step if the noise persists.
Note this fix is deliberately advisory: it does not block the commit. Making a small-magnitude rewrite fail would reject legitimate 1-pixel intentional changes, and I would rather not add a false-failure mode to a workflow I cannot test end-to-end.
B. A dead shard reports success
continue-on-error: trueon the update-snapshots run is required — under--update-snapshotsa rewritten baseline surfaces as a test failure, so without it every successful regeneration would fail. But it also swallowed genuine breakage. If a shard died (ComfyUI server never came up, wrong container, Playwright crashed), zero snapshots changed,merge-and-commitlogged "No changes to commit", and the whole run went green having done nothing.Now the JSON report is retained and checked per shard:
statsThat last row is why the exit code alone cannot be used, and I only found this by running it:
--grepwith no matches exits 1 and records a top-level Playwright error, identical on both counts to a real failure. Thecloudleg can legitimately match nothing. The discriminator that actually works is the combination of "did the report get written" and "did failures coincide with zero rewrites".Parsed with
node, notjq: this step runs inside the CI container, where node is guaranteed (it just ran pnpm and Playwright) but jq is not.Verification
The workflow itself cannot be exercised without merging, for the reason established in #14684. What I could execute, I did — against real Playwright output and real images, not mocks:
.stats.{expected,unexpected,flaky,skipped}plus.errors. The no-match exit code and the dead-run "no JSON file at all" behaviour were both observed, not assumed — and the first of those invalidated my original design.100and raised the warning; a 3600-pixel change reported exactly3600and did not; an untracked file reportednew; an unchanged file was correctly absent.`a$(touch /tmp/PWNED)b.png`and backtick-wrapped names produced the literal text with no evaluation. This is why it is built withcatand not a heredoc.yamllint(the repo'sCI: YAML Validationgate) passes;actionlintv1.7.11 reports no expression or context errors and the same 6 pre-existingSC2086:infofindings asmain— zero new.bash -npasses on every inline script. Both commits were checked individually, so the intermediate commit is not broken.Unverifiable until merge: that ImageMagick
compareis present on the runner (documented in the ubuntu runner image; the step degrades to byte sizes and warns if absent, and never fails); thatnodeis present in the CI container (it necessarily is, since the preceding steps run pnpm and Playwright); that--reporter=json,htmlalongsidePLAYWRIGHT_JSON_OUTPUT_NAMEbehaves in the container as it does locally; and the real end-to-end regeneration.Threshold caveat: 2000 pixels is a heuristic chosen against a single reported 795-pixel instance, not a measured distribution. It only gates an advisory warning, never the commit, so being wrong about it costs noise and not correctness — but it should be revisited once there is more data.