Skip to content

gc: test_gap_repsel_gc_stress SIGSEGVs intermittently in a copying minor with the conservative stack scan ON (PERRY_GC_INCREMENTAL=0) #6982

Closed

Description

@proggeramlug

Separate from the precise-roots family: this one crashes with the conservative native-stack scan ON, so it is not an unrooted-temporary bug.

Repro

Release build, macOS arm64, origin/main + #6977 (the fix in #6977 is not on this path — PERRY_GC_INCREMENTAL=0 bypasses the budgeted stepper entirely, so this reproduces on plain main given the same escape hatch).

$ perry test-files/test_gap_repsel_gc_stress.ts -o gc_stress
$ for i in 1 2; do PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 ./gc_stress >/dev/null; echo "rc=$?"; done
rc=139
rc=139

Intermittent — roughly 2 of 3 runs. With PERRY_GC_TRACE=1 PERRY_GC_DIAG=1 added:

run1 rc=139  output mismatch
run2 rc=0    output matches the Node oracle
run3 rc=139  output mismatch

PERRY_GC_INCREMENTAL=0 is a documented escape hatch, and it is what makes gc_check_trigger take the #5476 direct-collection arm instead of handing the trigger to the budgeted stepper. On this path the collection is a real copying minor: the passing runs report [gc-copy-minor] ran copied_objects=... in the thousands.

Why it is a distinct defect

Every other corpus file passes on this exact configuration while relocating thousands of objects (see the table in the precise-roots issue — 19/20 OK in the conservative-scan column). test_gap_repsel_gc_stress is the one corpus member deliberately built to be GC-live, i.e. the one that drives the collector hardest, and it is the only one that crashes with conservative pinning still in force. So this is not "precise roots lost a value"; something in the copying minor itself is unsound under sustained churn.

The nondeterminism (same binary, same env, different outcome per run) points at a timing/ordering window rather than a fixed miscompile.

Why nobody saw it

test_gap_repsel_gc_stress is green in the matrix's shipped-configuration arms because in the shipped configuration it never takes this path: with incremental mode on, the budgeted cycle handles the trigger. #6950 documents that the shipped automatic paths never produce a copying minor at all. So the only way to reach this crash is the escape hatch, and nothing was running the escape hatch against the corpus until #6977 added the arm.

Suggested next step

Bisect against the copying-minor phases with PERRY_GC_VERIFY_EVACUATION=1 and PERRY_GC_VERIFY_MARK=1, and check whether the crash survives PERRY_GC_FORCE_EVACUATE=0 — i.e. whether it needs actual relocation or only the copying-minor code path. A core dump / lldb backtrace on a repro run should localise it quickly given how reliably it fires.

Related: #6950, #6977, #6892 (minor-sweep finalization), #6910.

Activity

  1. proggeramlug commented on Jul 29, 2026

    @proggeramlug
    ContributorAuthor

    Evidence from the #6981 investigation — and an explicit warning against over-reading it

    While fixing #6981 (PR #6990), five of that issue's fourteen red files turned out to be GC-internal crashes rather than the typed-array-constructor bug: repsel_canonical_i32, repsel_ptr_shape_locals, repsel_proven_this_frozen, ta_param_numeric_read, typedarray_param_read.

    The concrete part

    In two crashes examined in detail (an ad-hoc probe and typedarray_param_read), the faulting instruction was identical:

    ldrb w8, [x21, #-0x8]!      ; faulting addresses 0x300000020b, 0x15
    

    That is a GC-header obj_type read at addr-8 — the collector validating a pointer that should never have reached it. So the defect is not in the read; it is in whatever installs that pointer. typedarray_param_read was bisected to a single line, console.log("u8", xorU8(u8, 8)), whose init sequence and param-read shape each pass in isolation.

    The part that is NOT established — please do not inherit this as fact

    An earlier summary (mine) described these as "frame-for-frame identical stacks — one shared collector defect, not five bugs." That overstates the evidence:

    • The stacks were matched by offset-into-unnamed-symbol in a stripped binary (+46704, +156, +72, +60, +14656, +8048), not by symbol name. Strong pattern-matching, not proof of a shared path.
    • Only two stacks were compared. The other three crashers were grouped by crash class, not by stack.

    So "5 sites, 1 defect" is a hypothesis with n=2, and any claim that it subsumes this issue is downstream of that hypothesis.

    Cheapest next step

    Rebuild with symbols and re-collect stacks for all five. That settles in one build whether this is one defect or several, and it should come before any fix attempt — scoping a fix around a shared root cause that turns out to be three separate bugs would waste the work.

    Reproduce on the evacuating precise-roots arm, and confirm relocation actually happened ([gc-copy-minor] copied_objects=N, N > 0):

    PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off
    

    A green run where nothing moved proves nothing — the failure mode #6942/#6946/#6950 documented and #6977 removed.

    Related: #6981, #6990, #6968, #6977.

  2. added a commit that references this issue on Jul 29, 2026
  3. proggeramlug commented on Jul 29, 2026

    @proggeramlug
    ContributorAuthor

    Settled: symbolized stacks for all five, and the premise does not survive them

    You were right to insist on this before any fix. No symbols build was needed — post_link.rs::strip_final_binary skips strip when PERRY_DEBUG_SYMBOLS=1 is set, so the (already unstripped) runtime symbols survive into the linked test binary and the macOS crash report symbolizes directly.

    All five reproduce from aa1c15028 with copied_objects matching #6981's table exactly (4 579 / 4 640 / 4 614 / 5 948 / 5 965), and without auto-optimize.

    The stacks

    repsel_canonical_i32 / repsel_ptr_shape_locals   SIGBUS  0x18400001fc
      0  js_array_length + 488
      1  js_object_set_field_by_name + 5964
      2  populate_global_this_builtins + 45592
      3  js_get_global_this + 156
      4  js_get_global_this_builtin_value + 72
    
    repsel_proven_this_frozen                        SIGSEGV 0x434c4f5300000010
      0  descriptor_state::get_accessor_descriptor + 136
      1  js_object_set_field_by_name + 5728
      2  populate_global_this_builtins + 45592
    
    ta_param_numeric_read                            SIGSEGV 0x004e614e00000008
      0  descriptor_state::set_builtin_property_attrs + 456
      1  populate_global_this_builtins + 46820
    
    typedarray_param_read                            SIGBUS  0x300000020b
      0  get_field_by_name_tail::get_field_by_name_object_tail + 10444
      1  js_object_get_field_by_name + 8048
      2  js_get_global_this_builtin_value + 396
    

    Two corrections to the issue

    1. These are not GC-internal crashes. There is not one GC frame in any of the five — no mark, trace, copying or evacuate. The ldrb w8, [x21, #-0x8]! is js_array_length reading an array header at arr-8: an ordinary mutator read. The collector is not the code validating a bad pointer; it is the code that moved the object out from under the mutator. (is_plausible_heap_addr would have rejected both quoted addresses anyway — on macOS HEAP_MIN is 2 TB and both are below it — which is a hint the faulting read was never a validation path.)

    2. The real faulting addresses carry ASCII. 0x434c4f53 = "CLOS", 0x004e614e = "NaN" — relocated string payload being read where a pointer/descriptor was expected. That is the signature of a stale from-space pointer whose bytes have been recycled, not of a small integer or tag remnant.

    One defect or several?

    The five crashes shared one blocking defect, and it was masking at least two more. All five fault downstream of the same js_get_global_this_builtin_value -> populate_global_this_builtins spine, at four different sites — the site varies only with where the recycled bytes landed. So "one shared defect" was directionally right, but it is one shared first hurdle, not the whole story.

    Root cause: js_get_global_this registers two root slots for the singleton so an evacuating collector rewrites them, then passes the raw pre-GC pointer by value into populate_global_this_builtins, which allocates on nearly every step. Fixed in #6994 by rooting it in a RuntimeHandleScope and re-reading through the handle.

    After that fix, crashes on the arm go 6 -> 2, repsel_canonical_i32 flips to byte-identical-to-Node, and two independent defects become observable: #6991 (compiled constructor receiver stale — the #6983 ctor-receiver class) and #6992 (proven-this method value lost).

    Also worth knowing: the per-PR matrix arms cannot reproduce this class at all. On the unfixed binaries, default, verify_evac, cons_scan_off and shipped_default are all green; the evacuating base needs PERRY_GC_INCREMENTAL=0 and PERRY_CONSERVATIVE_STACK_SCAN=off together, which only appears in --arms all (push-only). Filed as #6993.

  4. proggeramlug commented on Jul 30, 2026

    @proggeramlug
    ContributorAuthor

    Likely resolved by #7050, but not closing — this is an intermittent bug and one clean pass is not enough

    The matrix on e279b2d54 (post-#7050) reports test_gap_repsel_gc_stress PASS in all 20 arms, including the conservative-scan-ON arms this issue is specifically about. #7050's root cause plausibly covers it: Arena::alloc(&mut self, ..) called gc_check_trigger() inside its own &mut self borrow, so a collection allocating into the same arena could grow self.blocks underneath the live borrow — silent malloc-heap corruption whose surfacing point is wherever the next allocation lands. That mechanism is indifferent to whether the conservative scan is on; it only needs a collection to install a block, which is why the scan-ON population would go quiet too.

    But this issue documented a ~2-in-3 failure rate, and the evidence above is a single matrix pass. For an intermittent fault that is weak evidence — the same pass would have looked clean by luck roughly 1 time in 27 even with the bug fully present.

    Before closing, someone should run the scan-ON arms repeatedly — 20+ runs, the standard this campaign used for #7022 (139 × 20 → 0 × 20) — and confirm zero failures. If that comes back clean, close it citing #7050.

    Recorded rather than closed because a wrongly-closed intermittent GC bug is close to unrecoverable: it comes back months later as an unrelated-looking crash, and the search starts from zero.

  5. proggeramlug commented on Aug 2, 2026

    @proggeramlug
    ContributorAuthor

    Re-tested on main = d4ac65f21 and deliberately NOT closed. The reproducer now passes 10/10 — and that result is worthless as evidence. Recording it so nobody closes this on the same run.

    What happened

    Exactly the arm this issue specifies (conservative native-stack scan ON):

    PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 ./gc_stress
    

    10/10 rc=0, byte-exact against node 26.5.1 — versus the rc=139 roughly 2-in-3 reported here.

    Why that proves nothing

    PERRY_GC_DIAG=1 on the same binary and the same arm reports 0 copying minors. Adding PERRY_GC_FORCE_EVACUATE=1 still gives 0.

    This issue is a crash inside a copying minor ("the passing runs report [gc-copy-minor] ran copied_objects=... in the thousands"). A configuration that runs zero copying minors cannot exhibit it, so 10/10 green here means only that the code path was never entered. Under CLAUDE.md's fourth way a gate can be unable to fail, the subject never ran.

    What it is blocked on

    #7255 — four of the six PR-gating matrix arms run zero copying minors corpus-wide, and this is the same inertness reaching this issue's arm. Until the requires=scavenge route relocates again, this issue is untestable as written, not fixed.

    What would actually close it

    Either (a) #7255 is resolved and this arm relocates again, then re-run 10× and compare; or (b) this issue is re-scoped onto an arm that demonstrably moves — +PERRY_CONSERVATIVE_STACK_SCAN=off does relocate on this build, but that changes the premise, since "conservative pinning still in force" is the distinguishing feature of this report versus the precise-roots family.

    Part of the sweep in #7262.

  6. proggeramlug commented on Aug 2, 2026

    @proggeramlug
    ContributorAuthor

    Still not safely judgeable — and the reason is measurable, not "it's intermittent"

    Measured while fixing #7255, on 61410a222, release, macOS arm64, node 26.5.1.

    This issue's configuration is now intermittently inert. Ten runs of
    test_gap_repsel_gc_stress under exactly the env in the report — conservative
    stack scan ON, which is the whole point of it:

    PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_GC_TRACE=1 PERRY_GC_DIAG=1 ./gc_stress
    
    run rc copying minors objects copied output
    1–7, 9, 10 0 0 0 byte-exact
    8 0 1 141 711 byte-exact

    So "passes 10/10" here is really "passes once, and the other nine runs did not
    execute the subject at all". That is the #7255 pathology at the level of a single
    issue, and it is why the earlier sweep read this as fixed.

    Half of the mechanism is established: with PERRY_GC_MOVING_LOOP_POLLS
    default-OFF (#7161's stopgap for #7154) the alloc-point minor runs behind
    ManualGcScanGuard::force_full_scan(), which makes the copying minor ineligible
    (fallback=conservative_stack) — so the alloc-point route contributes zero. Which
    entry supplies the one-in-ten copying minor is not established here; the
    obvious candidate is the microtask-pump safepoint (gc_safepoint_moving_minor,
    gated by gc_moving_safepoint_enabled, default ON), whose timing depends on when
    the workload happens to yield, but I did not confirm it.

    The neighbouring deterministic arm is clean. evac_minor — the same env plus
    PERRY_CONSERVATIVE_STACK_SCAN=off — runs 10 copying minors and copies 1 385 325
    objects on every run, and is 10/10 byte-exact. But that is a different claim:
    this issue is specifically about a crash with conservative pinning still in
    force, so a green there does not settle it.

    Deliberately not added as a matrix arm. An incr_off arm carrying this exact
    env would satisfy its requires=scavenge about one run in ten per file, i.e. a
    flaky gate — and #7255's sequencing constraint is that no new permanent (or
    intermittent) red gets created. What is needed first is for the collector to
    reach a copying minor with the conservative scan on reliably; then the arm is
    worth adding and this issue becomes judgeable in one run instead of ten.

    Leaving open. Next step for whoever picks it up: reproduce with the table above
    in hand, and count copying minors before believing any verdict.

  7. proggeramlug commented on Aug 10, 2026

    @proggeramlug
    ContributorAuthor

    Closing — 40/40 clean on current main (v0.5.1432), with the arm live in every single run.

    PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 ./gc_stress     × 40
      → nonzero exits: 0
      → runs with a live copying minor (copied_objects > 0): 40/40
    

    The number that matters is 40/40 live, not 40/40 clean

    This issue's own last comment described the configuration as "intermittently inert", and an earlier 150-run pass confirmed it precisely: 145 of 150 runs performed zero copying minors (eligible_false_count=23 each), with only 5 firing — and those 5 were clean. A 150-run soak in which 145 runs exercised nothing is not evidence the bug is gone.

    On current main every run collects. What changed is #7721, which flipped PERRY_GC_MOVING_LOOP_POLLS to default ON — the back-edge poll is the precise collection point a compute-only program reaches, and it was default-OFF from #7161 until then. The five that did fire previously almost certainly came through the microtask-pump safepoint, the only route that needs no compile-time polls.

    So this is a genuinely different experiment from the one that left the issue open, and it is the one the issue was waiting for: the arm is live 40/40 and the SIGSEGV does not occur.

    Reopen with the copied_objects count if it returns — a run that collected nothing cannot distinguish a fixed bug from an unexercised one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions