Skip to content

conformance-smoke fails 8/8 shards: 12 real regressions, 10 newly-visible node_fail transitions, 1 fix the snapshot hasn't accepted #8117

Closed

Description

@proggeramlug

Summary

conformance-smoke is a required status context and it fails 8 of 8 shards on every recent PR. I pulled all eight shard logs from #8090's completed run and tallied the actual gate output. It is not flake, and it is not one cause — there are 23 status changes in four distinct categories.

CLAUDE.md warns that these shards are flaky and that a red shard should be re-run and A/B'd before being believed. Eight of eight is not the flake signature, which is why I went looking.

The tally

 10  node_fail  -> parity_fail
  8  pass       -> parity_fail
  3  pass       -> crash
  1  pass       -> compile_fail
  1  parity_fail -> pass

10 × node_fail -> parity_fail — previously invisible, now exposed

test_gap_4510_enum_forward_ref          test_gap_moment_methods
test_gap_backoff_options                test_gap_prop_plan_cache_invalidation
test_gap_cron_cronjob                   test_gap_ratelimiter_memory
test_gap_dayjs_factory_arg              test_gap_slugify_options
test_gap_derived_param_props            test_gap_enum_in_function_body

This is exactly the #6364 pattern CLAUDE.md documents: when the oracle cannot run a test it exits non-zero, the harness classifies it node_fail, and the test is silently dropped from the gate rather than going red. These ten were in that hidden state and are now running — so these are not new breakage, they are newly visible breakage. The shard header says so itself: "STATUS CHANGES — including tests the oracle stopped covering".

That makes them the least urgent of the four groups and arguably good news, but they must be triaged and recorded rather than left to hold the gate red.

12 × genuine regressions from pass

pass -> crash          test_gap_6386_dataview_concat_regex_fastpaths
                       test_gap_7238_i64_specialization_exactness
                       test_gap_buffer_own_props
pass -> compile_fail   test_gap_zlib_4917_level
pass -> parity_fail    test_gap_array_methods
                       test_gap_builtin_alias_construct_7524
                       test_gap_collection_foreach_member_receiver_thisarg
                       test_gap_dynamic_builtin_construct_dispatch
                       test_gap_new_globalthis_builtin_6726
                       test_gap_set_map_foreach_fused_receiver
                       test_gap_specabi_reassign
                       test_gap_sso_concat_string_index

These are the ones that matter. Three pass -> crash are the worst of them. test_gap_specabi_reassign is #8100, fixed by #8109 — so at least one of the twelve is already resolved, which suggests the rest are individually tractable.

I have not attributed these to a commit. They are red with similar counts on every PR I checked, so they are almost certainly main-side rather than any one PR's doing, but that is an inference and someone should bisect rather than trust it.

1 × parity_fail -> pass — a fix that fails the gate

test_gap_iterator_helpers_2874 now passes and the snapshot still lists it as failing. The snapshot gates divergence in either direction by design, so an improvement turns the gate red until the snapshot is regenerated. Working as intended, but worth naming: part of why this context is red is that something got better.

Why this matters beyond the tests themselves

conformance-smoke-complete is one of seven required contexts. With it red, no PR can merge without admin bypass, which is the "required and always red trains bypass" failure mode CLAUDE.md calls out — and it is how #8100 (a silent wrong-answer miscompile in the shipped default) sat unnoticed. Its shard was cancelled early on recent runs, so nothing surfaced it.

Tracking the required-context picture in #8092. The others: lint is fixed as of #8087; cargo-test fails on four ELF-only byte-identity tests; parity/compile-smoke report SKIPPED by design and are continue-on-error.

Suggested split

  1. Triage the 10 node_fail -> parity_fail and record them in test-parity/gap_snapshot.json with issue links — they were always broken and are cheap to classify.
  2. Bisect the 12 genuine regressions. Three crashes first.
  3. Regenerate the snapshot for test_gap_iterator_helpers_2874.
  4. Only then is the gate meaningful enough to keep required.

Reproduce with ./scripts/run_gap_tests.sh --shard N/8.

Activity

  1. proggeramlug commented on Aug 14, 2026

    @proggeramlug
    ContributorAuthor

    Independent corroboration from a local gap run

    An agent running the full gap suite locally (a different worktree, a different build, no CI involvement) hit the same set. Two of the twelve pass -> regressions reproduced by name:

    test_gap_array_methods                                Int32Array.toSorted/.toReversed/.with -> "[]"
    test_gap_collection_foreach_member_receiver_thisarg   "set member no-thisArg:" -> "[]"
    

    and five of the ten node_fail -> parity_fail transitions also reproduced (enum_forward_ref, backoff_options, cron_cronjob, dayjs_factory_arg, derived_param_props).

    That matters for the attribution I left open above. Two independent methods — CI's sharded run and a local full run on an unrelated branch — agreeing on the same names is much stronger evidence that these are main-side than my inference from "red on every PR I checked".

    The local run also adds a control I could not get from CI logs: both regressions reproduce identically under PERRY_SPECIALIZED_ABI=0, so they are not representation-selection or specialized-ABI effects.

    Two notes on the middle group: the agent characterised those five as environment drift rather than code — the snapshot recorded node refusing TS enums and failing to resolve exponential-backoff without node_modules, and here node executes both. Consistent with the #6364 reading above, and it means a snapshot refresh clears them rather than a fix.

    One of the twelve now has an owner: test_gap_array_methods is the sort/with/toReversed family that #8090's changelog names as "same-class cases still open", tracked as #8096 with a fix in flight. That leaves eleven unowned, three of them crashes.

  2. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    Second corroboration, and a lead on two of the twelve

    A full 559/559 local gap run (different host, different branch, GAP_HARNESS_EXIT=1, no partial) reported seven regressions. All seven are in the twelve catalogued above, including both non-parity classes:

    array_methods                              pass -> parity_fail
    collection_foreach_member_receiver_thisarg pass -> parity_fail
    set_map_foreach_fused_receiver             pass -> parity_fail
    specabi_reassign                           pass -> parity_fail   (fixed since, by #8109)
    sso_concat_string_index                    pass -> parity_fail   (fixed since, by #8109)
    7238_i64_specialization_exactness          pass -> crash
    zlib_4917_level                            pass -> compile_fail
    

    It also independently confirmed the improvement: test_gap_iterator_helpers_2874: parity_fail -> pass.

    The lead: two of them fail only in-suite

    test_gap_7238_i64_specialization_exactness and test_gap_zlib_4917_level do not reproduce standalone. Run by hand they are byte-for-byte identical to node with exit 0.

    The first reading was host load — that run executed above load 100 with auto-optimize lock serialization, and the harness's own wording for the crash class is "died from a signal or timed out". But CI classified them the same way (crash, compile_fail) on GitHub runners, which is a different host under different load. Two independent environments producing identical classifications is not the usual signature of a timeout artifact.

    So the better characterisation is "fails in-suite, passes standalone, on two independent hosts" — which points at test isolation or ordering rather than the code under test. That is a much cheaper thing to chase than a miscompile, and it should be checked before anyone bisects those two: if they are ordering-dependent, a bisect will produce a confident wrong answer.

    Progress on the twelve

    Three now have fixes on main: specabi_reassign and sso_concat_string_index (#8109), array_methods (#8119, closing #8096). That leaves nine, of which collection_foreach_member_receiver_thisarg and set_map_foreach_fused_receiver are the same empty-collection family and look like one bug rather than two.

  3. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    Status of the twelve after tonight's merges — measured, not inferred

    A post-rebase retest on 83b6b8c69 (which now carries #8106, #8109, #8118, #8119, #8120) re-ran the affected entries individually. Two are fixed; ten remain.

    # test state
    1 6386_dataview_concat_regex_fastpaths red — crash
    2 7238_i64_specialization_exactness red in-suite, passes standalone
    3 array_methods ✅ fixed by #8119 (#8096)
    4 buffer_own_props red — crash
    5 builtin_alias_construct_7524 red
    6 collection_foreach_member_receiver_thisarg red, unowned
    7 dynamic_builtin_construct_dispatch red
    8 new_globalthis_builtin_6726 red
    9 set_map_foreach_fused_receiver red, unowned
    10 specabi_reassign ✅ fixed by #8109 (#8100)
    11 sso_concat_string_index red — see below
    12 zlib_4917_level red in-suite, passes standalone

    sso_concat_string_index was predicted fixed and is not

    Worth recording because the reasoning failed in an instructive way. #8109's commit message says it re-tags "a heap string so it reaches the string arm rather than the ObjectHeader walk", which reads like general coverage of string indexing. It is not — that describes one arm of that change, and this case is outside it.

    Reading a commit message is not reading the diff. The prediction was recorded in advance, which is the only reason the miss was visible at all; without it the observation would have been "three still red" with no signal that an expectation had been violated.

    This one now looks like it wants an owner: it is red standalone, and the two fixes that landed in its neighbourhood did not touch it.

    Remaining shape

    • Two crashes (6386_dataview_concat_regex_fastpaths, buffer_own_props) — most severe, unowned.
    • Two in-suite-only (7238_i64_specialization_exactness, zlib_4917_level) — do not bisect these before establishing isolation; an ordering-dependent failure will hand a bisect a confident wrong answer.
    • Two in the same empty-collection family (collection_foreach_member_receiver_thisarg, set_map_foreach_fused_receiver) — plausibly one bug.
    • Four others red standalone.

    So the ten split into roughly six investigations, not ten.

  4. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    The two pass -> crash entries: both are SIGSEGV, both are 12 days old, and they bisect to #7314 — nothing merged tonight

    Taking 6386_dataview_concat_regex_fastpaths and buffer_own_props. Three things are measured below; one reading at the end is explicitly labelled inferred.

    1. They are signals, not timeouts

    From the eight shard logs of #8090's run (31825115698), pulled with gh api repos/:owner/:repo/actions/jobs/<id>/logs:

    CRASH test_gap_6386_dataview_concat_regex_fastpaths (SIGSEGV (exit 139))
          Perry died after printing 13 line(s); Node exited 0.
    CRASH test_gap_buffer_own_props (SIGSEGV (exit 139))
          Perry died after printing 11 line(s); Node exited 0.
    

    The 13th/11th "line" is bash's own Segmentation fault message, captured by the harness's 2>&1. So the real counts are 12 and 10.

    For contrast, in that same run 7238_i64_specialization_exactness is TIMEOUT (killed after 10s) — a genuinely different class, consistent with the "in-suite only" reading already in this thread.

    2. A zero-build bisect over the archived CI logs

    For each historical PR run I downloaded its eight conformance-smoke shard logs and read the PASS/CRASH verdict for these two names, then resolved each run's head sha to the main commit it branched from (gh api repos/:owner/:repo/compare/main...<sha> --jq .merge_base_commit). No builds involved.

    Both tests PASS on every probed run from 2026-07-28 through 2026-08-03 13:49, and CRASH from 2026-08-03 15:41Z onward:

    run branch main merge-base base UTC buffer dataview
    30826172101 gc/7280-dep-corpus b50e857c2 13:37 PASS PASS
    30828209038 exp/stackmap-viability b50e857c2 13:37 CRASH CRASH
    30826965091 fix/7311-multi-operand-reload 2fc383a6f 15:13 PASS PASS
    30828385716 feat/gc-schedule-seed-fuzzing 6bedb2574 15:20 PASS PASS
    30829166093 docs/7314-plan-status 61a789f7c 15:41 CRASH CRASH
    30829805015 fix/node-module-node26-parity 6fe493920 15:48 PASS CRASH
    30830497975 gc/7314-followup-cross-function-match 6fe493920 15:48 CRASH CRASH
    30830932181 fix/loop-safepoint-purity-knob 93f5029e6 16:06 CRASH CRASH

    git log 6bedb2574..6fe493920 is exactly two commits and the later one is docs-only, so the window holds one code commit:

    61a789f7c — Native-frame GC roots via LLVM statepoints, opt-in (#7173, #7174) (#7314)

    Two rows make that airtight rather than circumstantial:

    Controls: the runner image is identical across the boundary (ubuntu-24.04 / 20260720.247.2, provisioner 20260707.563) on every row above, so this is not an image rotation.

    They are also intermittent. After 08-03 each test independently flips: buffer PASSes on 08-10, dataview PASSes on 08-08/08-09/08-13. Roughly 65–80% crash rate each. That is the signature CLAUDE.md gives for an unrooted register — "goes bad only when a collection lands in its window" — not a deterministic miscompile, and not an unrooted cache (which would be perfectly reproducible).

    3. Where each one dies

    The line counts are trustworthy: js_console_log prints through Rust's println!, and Rust's Stdout is a LineWriter — it flushes on every newline whether or not stdout is a TTY — so nothing is lost in a buffer when the process dies.

    • buffer_own_props printed all 10 of its lines, then faulted. That is the generated exit epilogue (codegen/entry.rs loop_exit): js_process_emit_before_exit → js_promise_run_microtasks_event_loop (an outermost microtask-pump GC safepoint) → timers → js_process_run_finalization_exit → … → js_gc_release_current_thread_collection_side_allocations.
    • 6386_dataview printed 12 — through dv shadowed: 43, the last statement of the DataView block — and faulted before concat hot:. What follows is 200 pushes and 500 × [].concat(x, y) over 200 elements, the file's first heavy-allocation loop, i.e. the first loop back-edge poll / moving minor after that block.

    Shared shape: the first moving minor after an own property is stored onto a Buffer/DataView. 6386's v3 is const inside a { … } block, so by the concat loop it is dead while buffer_props[addr(v3)] still holds its entry (a closure) keyed by a dead address. buffer_own_props keeps its Buffers module-level, so its entries stay live and its crash lands at the exit-boundary pump instead. Of the 559 gap fixtures, a grep for an own-property assignment onto a Buffer/DataView receiver matches essentially only these two.

    4. They do not reproduce on macOS/arm64

    Release build at 83b6b8c69, PERRY_RUNTIME_DIR pinned, grep -c "Compiling perry-runtime v" = 1. Both fixtures are byte-identical to node v26.5.1 with exit 0, and stable across 30 consecutive runs each, with auto-optimize and without, and under PERRY_GC_ZEAL=1, PERRY_GC_ZEAL_ALLOC_KB=0, PERRY_GC_PROTECT_FROMSPACE=1 …_DEPTH=800, PERRY_GC_FORCE_EVACUATE=1, PERRY_GC_VERIFY_EVACUATION=1, PERRY_GC_FROMSPACE_SCAN=1, PERRY_GC_SCHEDULE_RATE=1 across seeds, and PERRY_GEN_GC=0. The from-space scan reports dangling=0 owners=0 every time. This is not "statepoints are off locally" — otool -l shows a real __PERRY_GCMAP/__perry_gcmap section in the compiled fixture, so the native-root backend is live here too.

    So the axis is Linux, not in-suite-vs-standalone — a different answer from the one #7238/#zlib got.

    5. The mechanism — this part is inferred, not measured

    Every behavioural change in #7314 is gated behind statepoints_enabled() / rs4gc_enabled() / native_stack_roots_enabled() / gc_safepoint_only_contract_enabled() / stack_map_requested, all default-off at that commit. I read every hunk; function.rs has 4 removed lines and 1197 additive gated ones, module.rs's per-bucket global emission and linker.rs's clang concurrency only fire for multi-unit modules (render_codegen_units(1) == to_ir(), and a one-file gap fixture is one unit), and contract_scan_heal_guard() / visit_stack_map_root_slots both early-return when off.

    There is exactly one unconditional addition: initialize_stack_maps() in js_gc_init. On macOS it resolves through _dyld_get_image_header and allocates nothing. On Linux, at that commit, it is:

    #[cfg(target_os = "linux")]
    fn loaded_stack_map_section() -> Option<&'static [u8]> {
        let bytes = std::fs::read("/proc/self/exe").ok()?;   // the ENTIRE executable
        let (addr, size) = elf_section_vaddr(&bytes, b".perry_gcmap")?;
        ...
    }

    So from 61a789f7c on, every Perry binary on Linux reads its own 5–13 MB image into a heap Vec at startup and frees it, before any user code runs, then returns None because a shadow-stack build has no such section. That moves every subsequent mmap and arena-block placement, it is Linux-only, and it lands exactly at the boundary. It is the best available explanation for "the diff is gated off and the tests flipped anyway, on Linux only" — i.e. #7314 most likely perturbed a latent bug rather than introducing a new code path, the same "allocator property, not a program property" class #7747 named for this exact test family.

    Two consequences if that reading holds: reverting #7314 is not the fix (it is the default root lowering now), and the real defect is older than 08-03 and still unnamed.

    Worth a separate look regardless: current main has expanded that startup read — loaded_stack_map_sections (#8075) now std::fs::reads every dl_iterate_phdr image, the executable plus every shared object, on every Linux start.

    6. Two concrete defects in that neighbourhood, found by reading

    Neither is proven to be the crash; both are real.

    1. finalize_collected_dead_buffer (buffer/header.rs:675) never prunes the buffer own-property table. Its docstring is "Drop every registry/side-table entry keyed by a dead buffer's address" and it prunes ten of them — BUFFER_REGISTRY, ARRAY_BUFFER_REGISTRY, SHARED_ARRAY_BUFFER_REGISTRY, DATA_VIEW_REGISTRY, BUFFER_AB_ALIAS, CRYPTO_KEY_META_REGISTRY, SECRET_KEY_REGISTRY, UINT8ARRAY_FROM_CTOR, the three external_* maps, plus detach:: and view:: — but not own_props::clear_buffer_own_props. Its only caller is register_buffer, so an entry is dropped only when the address is re-registered as a buffer. Result: an unbounded leak, scan_buffer_own_props_roots_mut retaining a dead buffer's closure forever, and an entry whose key visit_metadata_usize_slot keeps re-resolving against whatever now owns that address.
    2. scan_buffer_own_props_roots_mut diverges from the sibling it claims to mirror. own_props.rs says it mirrors closure::dynamic_props "(same locked side table + GC root scanner contract)". The closure version does two things it does not: it falls back to forwarded_heap_owner(owner) when the visitor did not rewrite the key, and it re-inserts through merge_closure_prop_map so a post-rewrite key collision merges instead of clobbering. The buffer version does a bare props.insert(new_owner, entries).

    Next

    A Linux-side probe is running (diag/8117-probe, a manual-dispatch-only workflow that hammers just these two fixtures on ubuntu-latest and takes a gdb backtrace on the first signal), plus the same thing in a local Linux aarch64 container. Whatever the backtrace says, the three facts above stand: SIGSEGV, dating to #7314 on 2026-08-03, intermittent, and Linux-only.

  5. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    Root cause found. Both pass -> crash entries are one missing receiver arm — fix in #8141.

    Correcting my own earlier comment on two points, both because I now have a Linux reproduction rather than an inference.

    1. They DO reproduce standalone. "Passes standalone" was a property of my host.

    host / mode buffer_own_props 6386_dataview
    macOS arm64, release, no auto-optimize, ×30 exit 0 exit 0
    macOS arm64, auto-optimize exit 0 exit 0
    ubuntu 24.04 aarch64 container, no auto-optimize exit 0 ×25 SIGSEGV 25/25
    ubuntu 24.04 aarch64 container, auto-optimize SIGSEGV —
    ubuntu-latest x86-64 (GitHub runner), no auto-optimize, standalone SIGSEGV (139) —

    The x86-64 row is a manual-dispatch job on a clean checkout of fa83ecab2, compiling and running the fixture directly — no gap harness, no shard, no timeout wrapper. So this is not test isolation or ordering, and it is not the 7238/zlib class.

    2. The bug

    #0  js_array_length                                        <- SIGSEGV, x0 = 0x12b00003aa1f03e2
    #1  perry_runtime::object::reflect_support::obj_value_has_own_key
    #2  perry_runtime::proxy::own_set_descriptor
    #3  perry_runtime::proxy::ordinary_set_with_receiver
    #4  js_put_value_set
    #5  js_put_value_set_dyn_ic_miss
    #6  main
    

    Faulting instruction ldurb w9, [x20, #-8], then cmp w9, #0x9 and a compare against 0x4C5A5841 — GC_TYPE_LAZY_ARRAY and LAZY_ARRAY_MAGIC. x0 is not an address.

    obj_value_has_own_key has arms for a registry typed array, a GC_TYPE_ARRAY/GC_TYPE_LAZY_ARRAY, a closure and a native-module namespace — and none for a Buffer / ArrayBuffer / DataView. A buffer is a BufferHeader: no class_id, no keys_array. So the ordinary arm read (*obj).keys_array out of the bytes that follow a buffer header and handed it to js_array_length, with nothing in between but a < 0x10000 magnitude floor that payload bytes clear routinely.

    Four lines, 10/10 on Linux:

    const b: any = Buffer.alloc(8);
    b.readUInt8 = function () { return "shadowed"; };
    const k = "readUInt8";
    b[k](0);

    Same "ask the receiver question before the generic path claims it" shape as #8090 / #8109 / #8119 / #8120 — on the has-own-key / [[Set]] path this time. #8130 and #8140 are the same family again, which is starting to look like a category worth a sweep rather than one arm at a time.

    3. Why the bisect landed on #7314, and why that was still worth doing

    The defect is older than #7314. The garbage keys_array has to clear the floor and land unmapped; macOS's 2 TB heap floor makes it read as null, so the same call there silently answers "no own key" for a property the buffer really owns — wrong, but survivable.

    #7314's one unconditional addition is initialize_stack_maps() in js_gc_init, whose Linux arm is std::fs::read("/proc/self/exe") — the entire executable into a heap Vec, before any user code, on every Perry binary. That moved the heap enough to turn a pre-existing wrong answer into a fault. So the bisect was correct about when it started landing and wrong about what is broken, which is exactly the failure mode a bisect on a layout-sensitive bug has. Reverting #7314 would not have fixed it.

    (Independently worth a look: current main has widened that to loaded_stack_map_sections (#8075), which now std::fs::reads every dl_iterate_phdr image — the executable plus every shared object — at every Linux start.)

    4. Fix and verification — #8141

    A registered-buffer arm beside the typed-array arm, answering from the #6406 expando table. Prototype methods are inherited, not own, which is what lets buf.readUInt8 = fn install a shadowing own property; canonical integer indices are deliberately not folded in. Plus the keys_array guard becomes addr_class::is_plausible_heap_addr instead of the bare floor, as defence in depth for the next missing arm.

    Linux container, before → after (grep -c "Compiling perry-runtime v" = 1 on the rebuild):

    fixture before after
    the 4-line repro 10/10 SIGSEGV 20/20 exit 0
    test_gap_6386_dataview_concat_regex_fastpaths 25/25 SIGSEGV 20/20 exit 0
    test_gap_buffer_own_props SIGSEGV 20/20 exit 0

    Both gap fixtures are byte-identical to node v26.5.1 afterwards. cargo test -p perry-runtime --lib → 2390 passed, 0 failed, 4 ignored (baseline 2389 + the new test). The test asserts the answer, not "did not crash", so it fails on macOS too — watched fail with the arm removed: "a buffer's own expando property must be reported as an own key".

    Two side findings from the same read

    That leaves eight of the original twelve, and no crashes among them.

  6. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    Recount against current main (fb87d8923): 9 of the 12 now pass

    Re-ran every one of the twelve pass -> X regressions on a fresh perry-dev build of current main, PERRY_NO_AUTO_OPTIMIZE=1, PERRY_RUNTIME_DIR pinned to that build's own .a files, byte-compared against node 26.5.1 (.node-version).

    test filed as now
    test_gap_6386_dataview_concat_regex_fastpaths crash pass
    test_gap_7238_i64_specialization_exactness crash pass
    test_gap_buffer_own_props crash pass
    test_gap_zlib_4917_level compile_fail pass
    test_gap_array_methods parity_fail pass
    test_gap_builtin_alias_construct_7524 parity_fail pass
    test_gap_dynamic_builtin_construct_dispatch parity_fail pass
    test_gap_new_globalthis_builtin_6726 parity_fail pass
    test_gap_specabi_reassign parity_fail pass
    test_gap_collection_foreach_member_receiver_thisarg parity_fail parity_fail
    test_gap_set_map_foreach_fused_receiver parity_fail parity_fail
    test_gap_sso_concat_string_index parity_fail parity_fail

    All three pass -> crash entries are gone. Two of them (buffer_own_props, 6386_dataview) were one bug — a missing Buffer/DataView arm in obj_value_has_own_key, fixed in #8141. specabi_reassign was #8100/#8109 as this issue predicted. The rest were cleared by the #8090/#8109/#8119/#8120/#8134/#8135 run.

    This issue's own note that the twelve "are almost certainly main-side rather than any one PR's doing" was right, and it is the reason they cleared in a batch.

    The three that remain

    test_gap_collection_foreach_member_receiver_thisarg
      < set member no-thisArg: []
      > set member no-thisArg: ["n:10","n:20"]
    
    test_gap_set_map_foreach_fused_receiver
      < [] 3                        > [10,11,12] 3
      <                             > a=1,b=2
      < [] 2                        > true true 1
      < [] 3                        > [1,2] 0
    
    test_gap_sso_concat_string_index
      < undefined undefined undefined      > a b c
      < undefinedundefinedundefined        > abc
      < undefined undefined undefined      > % . "\n"
    

    The first two are almost certainly one bug — Set/Map forEach with a member-expression callback or a fused receiver collects nothing. Note the second also loses whole output lines, so a callback is likely never invoked rather than merely returning empty.

    The third is separate and the worst of the three: indexing a string returns undefined after an SSO concat. Silent wrong values, not a crash, and not confined to an exotic receiver.

    All three are in flight.

    Caveat on this measurement

    My harness is a direct compile-and-cmp against node, not the gap runner. The real harness applies normalize_output, so a test I call parity_fail could normalize to a pass, and in principle the reverse. Treat the nine as "very likely clear" and confirm on the next full conformance-smoke run rather than closing this issue on my numbers alone. I would rather state that than have someone close it and find the gate still red.

    The 10 node_fail -> parity_fail entries are untouched by this recount — they were always broken and still need triaging into test-parity/gap_snapshot.json with issue links, as this issue's step 1 says.

  7. proggeramlug commented on Aug 15, 2026

    @proggeramlug
    ContributorAuthor

    All 12 genuine regressions are now fixed

    The last three landed:

    Both are the same family as #8090/#8109/#8119/#8120/#8124/#8141/#8148: a receiver-specific path claiming an operation before the receiver-kind question is asked. That is now eight instances in two days, which is worth treating as a design signal rather than eight coincidences.

    Verified on a build of main with both merged — all three byte-identical to node 26.5.1, exit 0:

    PASS  test_gap_collection_foreach_member_receiver_thisarg
    PASS  test_gap_set_map_foreach_fused_receiver
    PASS  test_gap_sso_concat_string_index
    

    cargo test -p perry-runtime --lib: 2399 passed, 0 failed, 4 ignored.

    Not closing this yet — two things remain

    1. The normalize_output caveat still applies. My verification compiles and cmps directly; the real gate normalizes. The next full conformance-smoke run on main is the authority, not my numbers. Leaving this open until one goes green.

    2. The 10 node_fail -> parity_fail entries are untouched. They were always broken and only became visible; this issue's own step 1 asks for them to be triaged into test-parity/gap_snapshot.json with issue links. Nobody has done that, and they will hold the gate red on their own.

    A finding that changes what fixing this issue buys

    Fixing conformance-smoke will not stop merges needing admin bypass, so this issue is not the last obstacle to #8101.

    parity and compile-smoke are in the required contexts but both are gated to push / workflow_dispatch / a run-extended-tests label. On an ordinary PR they never run, so they sit permanently at "Expected — waiting for status" and every PR is structurally BLOCKED no matter what else is green. Confirmed against the live required list and against PRs merged today: of the seven required contexts, only a couple ever report.

    That is CLAUDE.md's own hazard 2/4 aimed at the required list itself. Two ways out, and it is a maintainer decision:

    • drop parity and compile-smoke from the required contexts, since they are tag-gated by design; or
    • give them a PR-side job that reports a skip.

    Worth deciding deliberately rather than discovering it at #8101's promotion time.

  8. proggeramlug commented on Aug 16, 2026

    @proggeramlug
    ContributorAuthor

    All 23 status changes are now accounted for on 3be2016c1. Evidence is a full unsharded run_parity_tests.sh --filter test_gap_ run (562 tests, 01:26) — conformance-smoke runs ./scripts/run_gap_tests.sh --shard N/8, i.e. this same suite, so a full run is a superset of all eight shards.

    The 12 genuine regressions — all 12 PASS

    test resolution
    6386_dataview_concat_regex_fastpaths (was crash) #8141
    buffer_own_props (was crash) #8141
    collection_foreach_member_receiver_thisarg #8130
    set_map_foreach_fused_receiver #8130 (one bug with the above, as predicted)
    sso_concat_string_index #8151
    array_methods #8119
    specabi_reassign #8109
    builtin_alias_construct_7524 now passes
    dynamic_builtin_construct_dispatch now passes
    new_globalthis_builtin_6726 now passes
    7238_i64_specialization_exactness (was crash) now passes
    zlib_4917_level (was compile_fail) now passes

    The three indirect-builtin-construct tests were expected to need one shared fix; they now pass without one having been written for them specifically, so something landed in the last day that covers the family (#8153's new name resolution and #8177's handle-method dispatch are the plausible candidates — neither is proven, and it no longer matters for this issue).

    7238_i64_specialization_exactness and zlib_4917_level were the two flagged as in-suite red, standalone green on two independent hosts. Both are green in-suite now.

    The 1 × parity_fail -> pass

    iterator_helpers_2874 — confirmed passing. It is still listed in gap_snapshot.json, so it currently reads as an improvement diff and holds the gate red. Fixed by the snapshot regeneration below.

    The 10 × node_fail -> parity_fail — root-caused, and worse than "newly visible"

    This issue read them as previously-hidden breakage now exposed. That is half right. The full mechanism:

    run_parity_tests.sh cannot emit node_fail for a plain exit 1 at all. The only record_result "$test_id" "node_fail" site (~:1344) is guarded by perry_abnormal_exit, which matches signals and timeout only (124, 132/134/136/138/139, >128). These tests fail with plain exit 1 — ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX for the enum/parameter-property tests (strip-only mode cannot run them, and Node 26 removed --experimental-transform-types), and MODULE_NOT_FOUND for the npm ones.

    So those ten snapshot entries record a status the harness is structurally incapable of producing. Every run yields changed, and any diff makes gap_snapshot.py return 1. The gap gate could never be green — the mirror of CLAUDE.md's "four ways a gate can be unable to fail". It survived because parity is tag-gated and invisible per-PR; conformance-smoke running the same suite is what surfaced it.

    Fixes, verified individually through the real harness (9 PASS):

    • 3 TS-syntax tests (4510_enum_forward_ref, derived_param_props, enum_in_function_body) — node can never run these, so they get test-parity/expected/<name>.txt fixtures, the channel the test_decorators_* tests already use for the same reason. Oracle built with esbuild → node; output matches each test's own documented expectations, and Perry is byte-identical to all three. Note derived_param_props's header already said "this is a perry-only expected-output test" — the fixture was simply never created.
    • 6 npm tests (backoff_options, cron_cronjob, dayjs_factory_arg, moment_methods, ratelimiter_memory, slugify_options) — the packages were never declared in package.json, so the oracle could not run in any environment, CI included; npm ci would not install them either. .npmrc says the root npm install exists precisely to "materialize the parity-test fixture deps". Declared them; all six now run under node with exit 0 and all six match Perry. Live parity, better than a frozen fixture.
    • 1 remaining — a real Perry bug, not a snapshot artifact. prop_plan_cache_invalidation is written for sloppy mode ("writes must be dropped") but runs as ESM, which is strict, so the frozen write throws. Node and Perry both correctly throw; they differ only in the message — node '#<H>', Perry '#<Object>'. js_throw_type_error_immutable_write (error.rs:1726) takes no receiver and hardcodes Object. 14 callers, all behind one wrapper, so it is fixable without a codegen ABI change. Being fixed separately.

    Also fixed while here: SKIP_TESTS is exact-match and contains only legacy test_* names, so it can never match a test_gap_* test.

    What remains before this closes

    The snapshot is being regenerated from a real run (UPDATE_SNAPSHOT=1) rather than hand-edited — the _schema says "do not hand-edit statuses", and hand-editing is almost certainly how ten impossible statuses got there. When that PR lands, all 23 changes are resolved.

    Two caveats recorded honestly:

    1. My run was unsharded; conformance-smoke shards 8 ways, which changes test ordering. The two "in-suite red, standalone green" tests are exactly the shape that could be ordering-sensitive, so a sharded confirmation is worth doing before trusting the gate green.
    2. webcrypto_async_threadpool failed once in this run and is not a regression — the test asserts an async digest crosses a macrotask boundary, node printed false at load ~64, and 5/5 re-runs at load ~45 print true, matching Perry. The oracle flaked. It must not be parked in the regenerated snapshot; I will check the output before committing it.
  9. proggeramlug commented on Aug 16, 2026

    @proggeramlug
    ContributorAuthor

    All 23 status changes are resolved on main @ 6e05295ed. Closing.

    conformance-smoke runs ./scripts/run_gap_tests.sh --shard N/8 — the same gap suite — so a full unsharded 562-test run is a superset of all eight shards. That run reports 99.1% parity, 5 non-passing, 0 crashes, and gap_snapshot.py check exits 0.

    12 × genuine regressions — all pass. Seven by named fixes (#8141 ×2 for the two crashes, #8130 ×2, #8151, #8119, #8109); the other five — builtin_alias_construct_7524, dynamic_builtin_construct_dispatch, new_globalthis_builtin_6726, and both "in-suite red, standalone green" oddities (7238_i64_specialization_exactness, zlib_4917_level) — now pass without a fix aimed at them specifically. The three indirect-builtin-construct tests were expected to need one shared fix; something in the last day covers the family.

    1 × parity_fail -> pass (iterator_helpers_2874) — accepted by the regenerated snapshot.

    10 × node_fail -> parity_fail — this issue read them as previously-hidden breakage now exposed. That was half right, and the full mechanism was worse: run_parity_tests.sh cannot emit node_fail for a plain exit 1 at all. The only record_result "node_fail" site is guarded by perry_abnormal_exit, which matches signals and timeout only (124, 132/134/136/138/139, >128). These tests fail with a plain exit 1 — ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX for the enum and parameter-property cases (strip-only mode cannot run them, and Node 26 removed --experimental-transform-types), and MODULE_NOT_FOUND for the npm ones.

    So ten snapshot entries recorded a status the harness is structurally incapable of producing. diff() returned changed for every one, and any diff makes the gate exit 1. The gap gate could never be green, on any tree — the mirror of CLAUDE.md's "four ways a gate can be unable to fail". It survived because parity is tag-gated and invisible per-PR; this issue's 8/8 red shards are what surfaced it.

    Fixed by category in #8198 (verified on main):

    • 3 TypeScript-syntax tests now have test-parity/expected/*.txt fixtures — the channel the test_decorators_* tests already use for the same reason. Oracle built with esbuild → node, cross-checked against the expectations in each test's own header; Perry byte-identical to all three. test_gap_derived_param_props already said "this is a perry-only expected-output test" — the fixture was simply never created, so it had verified nothing since May.
    • 6 npm tests have their packages declared as devDependencies. They were never in package.json, so the oracle could not run in any environment, CI included — npm ci would not have installed them either. All six now run under node with exit 0 and all six match Perry: live parity rather than a frozen fixture.
    • 1 re-triaged as a real Perry bug. prop_plan_cache_invalidation is written for sloppy mode but runs as ESM, which is strict, so its frozen-object writes throw. Node and Perry both correctly throw and differ only in the message — node names the constructor ('#<H>'), Perry hardcodes '#<Object>' at error.rs:1726 because js_throw_type_error_immutable_write takes no receiver. Recorded with that root cause; 14 callers behind one wrapper, so a receiver-aware variant fixes it with no codegen ABI change.

    The snapshot was regenerated from a full run rather than hand-edited — the _schema says not to, and hand-editing is how ten impossible statuses got there. It now holds 5 entries, all parity_fail, zero node_fail.

    Two things recorded rather than quietly done. SKIP_TESTS is exact-match and holds only legacy test_* names, so it can never match a test_gap_* test — left alone because every current case is better served by the expected-output channel. And test_gap_webcrypto_async_threadpool failed once during verification at load ~64 and was deliberately not parked: it asserts an async digest crosses a macrotask boundary, node printed false, and 5/5 re-runs at load ~45 print true, matching Perry. The oracle flaked; parking it would have converted a load artifact into a permanently accepted failure.

    One caveat left open honestly: the verification run was unsharded, and conformance-smoke shards 8 ways, which changes ordering. The two formerly-flaky tests are exactly the shape that could be ordering-sensitive, so a sharded confirmation is worth having before treating the gate as trustworthy — but nothing in this issue's tally remains unaccounted for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions