ci: compiler-output-regression has been red on every main sweep since 2026-09-18 — six structural verdicts, and nobody has established whether they are regressions or stale expectations #10784

Activity

1. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Correcting my own framing above before anyone acts on it. I wrote "the gate was green at some point before 2026-09-18T17:56, and the transition commit names the cause." I did not verify that, and it does not hold within any window I can query.

   Sampling 60 consecutive test.yml runs on main, compiler-output-regression is failure in every one that ran it, back to fcd108bfb0 (2026-09-16T07:55, v0.5.1579) — four days, no green. The job itself is long-standing (it traces back to the v0.5.1117 era), so it was surely green once, but the last green is outside what gh run list will return.

   So there is no known-good endpoint, and a bisect has nothing to bisect between. Starting one would burn a release build per hop searching a range whose upper bound is unknown. Ignore that instruction.

   Do this instead

   Answer the regression-vs-stale-expectation question directly from the artifacts, which needs no known-good commit:

   1. For each of the three vectorization_expectation workloads, read the fixture and the emitted IR and decide whether control_flow is an honest reason for the vectorizer to decline that loop. If the loop genuinely contains control flow the vectorizer cannot handle, the expectation is stale. If it does not, it is a regression.
   2. Same for raw_numeric_field_get_guard / native_reps_required_raw_class_field_get_* on raw_numeric_object_fields, and no_dynamic_property_runtime on native_owned_typed_views: does the compiler's current output actually violate the property the verdict names, or has the property's definition drifted?
   3. Only if (1) or (2) says "regression" is a bisect worth running — and by then you have a bounded hypothesis about which subsystem changed, which makes git log -S over that subsystem far cheaper than a build-per-hop bisect.

   The standing caution still holds, and is now more important, not less: do not refresh the expectations to make the gate green. A four-day-red gate with six stable verdicts is exactly where re-baselining turns a compiler regression into permanent invisible behaviour. If they turn out to be stale, say why each one is stale in the PR that refreshes it.

   One more thing this changes: a gate nobody can show green in four days is not obviously providing signal. Whether it should remain a main-gate input while in this state is a maintainer call — but leaving it required-and-always-red means every merge to main is already bypassing it, which is the "gate theatre" pattern CLAUDE.md documents.
2. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   My retraction above was itself wrong. There is a known-good endpoint, and I have verified it.

   I said "the last green is outside what gh run list will return, so there is no known-good endpoint — ignore the bisect instruction." The REST API pages past gh run list's cap. Walking /actions/workflows/test.yml/runs?branch=main and checking the job conclusion on each run:

   run	head	compiler-output-regression
   33626227278	0b24670dd9 2026-09-02T11:45Z	success
   33657570663	3a7c0b0a81 2026-09-02T16:52Z	failure

   git rev-list --count 0b24670dd9..3a7c0b0a81 = 45. So the gate has been red for 18 days, not four, and a bounded 45-commit window existed the whole time. Two wrong claims from me on this gate in one hour, both from treating an exhausted gh run list as proof of absence.

   It is three events, not one — which is why the failing set looked stable

   onset	verdict(s)	cause
   2026-09-02, window 0b24670dd9..3a7c0b0a81	vectorization_expectation × 3	b13a20095c (#9441) — proven by IR A/B
   ≤ 2026-09-05	no_dynamic_property_runtime	not attributed; window is 267 commits
   2026-09-12 → 09-15	raw_numeric_field_get_guard + 3 × native_reps_…	fc54e1b181 — self-documented

   Two artefacts of the job's own history hid this. Until #8855 added if: !cancelled(), the job aborted at the first failing step, so from 09-02 to 09-04 only the native-region step ran and no_dynamic_property_runtime was structurally invisible — its onset window cannot be narrowed from artifacts at all. And the failing set churns: fdc437f966 also failed h1_native_rep_equivalence, which does not fail now. The "six stable verdicts" in the issue body is the last-4-days shape, not the original one.

   Verdict: all six are stale expectations. None is a compiler regression.

   The three vectorization_expectation failures are not about the fixtures at all. The control_flow remark comes from main's compiler-emitted event loop, which is in every Perry binary. dynamic_fractional_array_index.ts contains no loop and still reports control_flow: 1; a bare console.log(1) produces the same five remarks; opt -passes='loop-simplify,print<loops>' finds exactly one loop in main (event_loop.header/check_pending/body/body_check/body_wait/exit), and excising it drops the remark count 29 → 0. The reason is honest: uncomputable trip count, no induction variable, two uncountable early exits, 25 side-effecting calls, and a predicated body_wait that cannot be if-converted. Vectorizing a program's event loop is meaningless. All three positive halves pass.

   b13a20095c (#9441, "the event loop looks before it sleeps") added the body_check/body_wait diamond; reconstructing the pre-#9441 3-block loop in the IR removes exactly the Control flow cannot be substituted for a select remark and leaves the other four, which are already allowed. That was a real fix (1082 ms vs node's 132 ms on a 20 ms setTimeout). The expectation should move, not the codegen. Corroborating: control_flow is already allowed in 22 of 25 workloads — the only three lacking it are the three that fail, and their vectorization blocks have not been touched since June/July while the others were widened piecemeal (#8490, #8854 are the precedent).

   The four raw_numeric_object_fields failures are one changed string, seen twice. That workload is a member of both suites with identical arguments, so 4 errors × 2 suites = the 8 occurrences in my table above — one defect, not two and not four. fc54e1b181 collapsed the class-field GET tower to one exit (−7.09 % .text on @babel/parser), updated its own codegen unit tests, and missed workloads.toml. Every one of the nine required facts is still recorded; the only field that differs is consumer: expected js_object_get_field_by_name_f64, actual js_class_field_get_ic. The property raw_numeric_field_get_guard names still holds — the sibling check that asserts the structure directly, raw_numeric_field_get_scoped_raw_load, passes.

   no_dynamic_property_runtime is an over-broad detector. It is a raw substring sweep of the whole module; exactly one helper hits, js_dyn_index_set_strict, in the cold tav.set.slow arm of a guarded typed-array store — the fixture's designed negative case, whose own allow_materialization_reasons lists "mutable_alias", the very reason recorded. The workload declares the fallback legal in one knob and illegal in another. Tellingly, the matching TypedArrayGet also falls back for mutable_alias but its slow arm calls js_typed_array_get, which is not on the watched list — so the read is "clean" and the write is not, for no semantic reason.

   This is the weakest of the six calls and I am flagging it as such rather than presenting six equal answers: const alias = view is a trivial copy, so if the native-owned-view analysis should see through it, this would be a minor analysis regression rather than a designed fallback. Leading suspect for the routing change is a2ffa80a4c ("retire receiver fact tables", 2026-09-04), which migrates exactly the eligibility predicate in expr/index_set.rs and claims to be behaviour-neutral — unconfirmed, and confirming it costs two debug builds.

   One incidental correction

   The existing #8489 comment in workloads.toml attributes native_owned_typed_views' control_flow to #8457's -O3 → -Os switch. Measured at -O3, -O2, -Os, -Oz: it appears identically at -O3, -O2 and -Os; only -Oz suppresses it, by disabling the vectorizer wholesale. The opt-level is not the cause. Anyone editing that comment should fix it.

   The structural finding worth more than the fix

   allowed_missed_reason_kinds is a module-wide sweep, and main's event loop guarantees a floor of {control_flow, call_instruction, generic_not_vectorized ×2, unknown_trip_count} in every program. For any fixture without un-vectorizable loops of its own, that list is a detector of compiler boilerplate, not of the fixture — which is exactly why 22 of 25 workloads have quietly accumulated the same five entries (13 → 22 over time). Scoping the remark pass to the fixture's own functions, excluding main, would make it a real detector again.

   And the obvious one: required, and red for 18 days, means every merge in that period bypassed it.
3. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Fix implemented and pushed — fix/10784-stale-compiler-output-expectations @ 2079501cdc, 4 files, +493/−22. It will land via a merge train.

   Both suite --suite native-region-proof and --suite native-abi-proof now exit 0 with failed_workloads: [], and python3 -m unittest tests.test_compiler_output_regression is 85 OK. run_lint_gates.sh is 78 of 79, the one failure being the known-red public-baseline step, verified identical on a pristine tree.

   Every refreshed expectation was sabotage-tested back to red, because a refreshed expectation that cannot fail is worse than the stale one it replaced:

   • dropping a different allowed kind (call_instruction) → red; raising min_vectorized_loops to 99 → red on the positive half. Both halves stay live.
   • reverting one consumer string → red; pointing the guard regex at a non-emitted symbol → red; keeping the guard load but requiring a branch into a non-existent block → red.
   • injecting js_dyn_index_get into nativeOwnedPositive → red, naming the function; into nativeOwnedKinds → red; into disposedFallback (which declares use_after_dispose) → green, as intended; removing mutable_alias from allow_materialization_reasons → red, so the licence comes from the workload's written allowance rather than a hardcoded name.

   no_dynamic_property_runtime was scoped, not blanket-allowed — verification.py now sweeps function bodies and exempts a function whose own native-rep record declares a fallback the workload already allows. No allow_dynamic_property_runtime = true, and deliberately no exclusion by block label, since a label exclusion would hide the same degradation in nativeOwnedPositive. Twenty untargeted workloads are byte-identical; the twenty-first differs only in a temp scratch path echoed into a detail string.

   Three corrections to my diagnosis comment above

   I posted that comment, so these are mine to correct.

   1. native_owned_typed_views' onset was wrong — 2026-09-15, not "≤2026-09-05", and the window is ~20 commits, not 267. Bracketed by job log: 6bab431189 (17:18Z) passes, ea42912d50 (21:26Z) fails, and the workload appears in failed_workloads for the first time there.

   2. a2ffa80a4c is not the cause. Its buffer_views.rs diff is a mechanical buffer_view_slots → receiver_descriptors table migration, and the gate was green for eleven days after it. I named it as "leading suspect" on the strength of it touching the right predicate; that was pattern-matching, not evidence.

   3. The real cause is d792c0a761 ("inline element access on declared typed arrays with unproven indexes", 2026-09-15). I verified independently that it is the only commit in 6bab431189..ea42912d50 — 27 commits — that touches expr/index_set_typed_array.rs.

   And the substantive point: nothing regressed, and the analysis never saw through const alias = view in the first place. At the last-green commit 0b24670dd9, mutableAliasFallback's TypedArraySet record was identical in every field to today's — access_mode: dynamic_fallback, materialization_reason: mutable_alias, fallback_reason: mutable_alias. Only the helper it lowered to changed: a bare js_typed_array_set, which is not on the watched list, became the guarded inline arms whose cold exit is js_dyn_index_set_strict, which is. A codegen improvement turned a helper's name into a watched prefix. So my "weakest call" was also the one I got most wrong — in the safe direction, but wrong.

   A caveat on the structural finding, which I also understated

   I wrote that scoping the remark pass to exclude main would make allowed_missed_reason_kinds a real detector again. Necessary but not sufficient. The GC loop back-edge poll (load volatile @PERRY_GC_POLL_ARMED guarding js_gc_loop_safepoint(), default-on since #7721) injects compiler-emitted control flow into the fixture's own functions — it accounts for 2 of native_owned_typed_views' 3 control_flow remarks, inside nativeOwnedPositive. A function-scoped list would still have to accept those, or the poll needs excluding structurally too. Anyone budgeting that work as "just drop main" will find it does not close.

   The mechanism for doing it properly does exist: -fsave-optimization-record=yaml -foptimization-record-passes=loop-vectorize gives exact per-function attribution with stable machine-readable keys, strictly better than the current English-prose regexes over <unknown>:0:0 stderr. ~150–250 lines. The real cost is the re-baseline — it changes what all 25 workloads measure, so every list must be re-derived and each surviving entry re-justified, including the three this PR just widened. Its own issue, so the 25-workload diff is reviewable on its own.

   Also corrected in the file: #8489's comment blaming -O3 → -Os for native_owned_typed_views' control_flow. Measured at -O3, -O2, -Os, -Oz — it appears identically at the first three; only -Oz suppresses it, by disabling the vectorizer wholesale. The three remarks are 2 from the GC safepoint poll and 1 from main's event loop.
4. proggeramlug

   added a commit that references this issue on Sep 20, 2026

   ci: refresh three stale compiler-output-regression expectations (#10784)

   ...

   84a0342
5. proggeramlug commented on Sep 20, 2026

   

   proggeramlug

   on Sep 20, 2026

   ContributorAuthor

   More actions

   Fixed on main as of v0.5.1616 (8e9f6f09e8), via merge train 237 (#10792).

   compiler-output-regression now passes both suites — the first time since 2026-09-02, eighteen days. It is a main-gate input, so every merge in that window was bypassing it.

   The refreshed gate was proven able to fail, on the assembled tree rather than only per-branch:

   clean      → rc=0, "failed_workloads": []
   sabotaged  → rc=1   (one require_records consumer reverted to js_object_get_field_by_name_f64)
   

   That check took four attempts, and the three that failed are worth recording because each one looked like a pass: BSD sed has no 0,/re/ address so the file was never modified; the next edit landed on a comment rather than the consumer = line; and the third used the wrong entry point, which produced zero bytes of output and exited 0. Asserting that an edit landed is not sufficient — it has to land on the load-bearing line, and the harness has to actually be running.

   Both fixes are in the train body in full, including the reasoning that PERRY_NO_AUTO_OPTIMIZE=1 is set at the HIR probe and deliberately not at the linking compile, since runtime_budgets are maxima derived from the executed binary's PERRY_GC_TRACE output and would pass vacuously against a runtime that emits no trace.

   Two things left open rather than quietly closed:

   The cold build still happens, just in the linking compile's 300 s budget instead of the probe's. Only one perry-auto hash is created across all 23 workloads, so CI pays exactly one cold build — the fix changes which step absorbs it, not whether 300 s is enough. The durable answer is a pre-warm step in test.yml with its own explicit budget, asserting target/perry-auto-* exists afterwards or it is a no-op nobody notices.

   --no-link could decline to auto-optimize outright — a two-line guard at run_pipeline.rs:6169 — which fixes the class rather than this instance. The risk is the undefined-symbol stub scan at :6179, which consumes optimized_libs.runtime to seed its defined-symbol set; skipping the rebuild falls back to find_runtime_library and could change synthesized stubs in an emitted object. repsel_determinism and repsel_knob_isolation hash --no-link objects, so their baselines would move once. Worth its own issue.

   And the structural finding, which this PR deliberately did not attempt: allowed_missed_reason_kinds is a module-wide sweep and main's event loop guarantees a five-remark floor in every program, so for any fixture without un-vectorizable loops of its own the list detects compiler boilerplate rather than the fixture. Excluding main is necessary but not sufficient — the GC loop back-edge poll injects control flow into the fixture's own functions, accounting for 2 of native_owned_typed_views' 3 control_flow remarks. -fsave-optimization-record=yaml gives exact per-function attribution with stable keys and is the right mechanism, ~150–250 lines, but the real cost is re-deriving all 25 workloads' lists. Its own issue, so that diff is reviewable on its own.
6. proggeramlug

   closed this as completedon Sep 20, 2026