You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gc: test_gap_repsel_gc_stress SIGSEGVs intermittently in a copying minor with the conservative stack scan ON (PERRY_GC_INCREMENTAL=0) #6982
Separate from the precise-roots family: this one crashes with the conservative native-stack scan ON, so it is not an unrooted-temporary bug.
Repro
Release build, macOS arm64, origin/main + #6977 (the fix in #6977 is not on this path — PERRY_GC_INCREMENTAL=0 bypasses the budgeted stepper entirely, so this reproduces on plain main given the same escape hatch).
$ perry test-files/test_gap_repsel_gc_stress.ts -o gc_stress
$ for i in 1 2; do PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 ./gc_stress >/dev/null; echo "rc=$?"; done
rc=139
rc=139
Intermittent — roughly 2 of 3 runs. With PERRY_GC_TRACE=1 PERRY_GC_DIAG=1 added:
PERRY_GC_INCREMENTAL=0 is a documented escape hatch, and it is what makes gc_check_trigger take the #5476 direct-collection arm instead of handing the trigger to the budgeted stepper. On this path the collection is a real copying minor: the passing runs report [gc-copy-minor] ran copied_objects=... in the thousands.
Why it is a distinct defect
Every other corpus file passes on this exact configuration while relocating thousands of objects (see the table in the precise-roots issue — 19/20 OK in the conservative-scan column). test_gap_repsel_gc_stress is the one corpus member deliberately built to be GC-live, i.e. the one that drives the collector hardest, and it is the only one that crashes with conservative pinning still in force. So this is not "precise roots lost a value"; something in the copying minor itself is unsound under sustained churn.
The nondeterminism (same binary, same env, different outcome per run) points at a timing/ordering window rather than a fixed miscompile.
Why nobody saw it
test_gap_repsel_gc_stress is green in the matrix's shipped-configuration arms because in the shipped configuration it never takes this path: with incremental mode on, the budgeted cycle handles the trigger. #6950 documents that the shipped automatic paths never produce a copying minor at all. So the only way to reach this crash is the escape hatch, and nothing was running the escape hatch against the corpus until #6977 added the arm.
Suggested next step
Bisect against the copying-minor phases with PERRY_GC_VERIFY_EVACUATION=1 and PERRY_GC_VERIFY_MARK=1, and check whether the crash survives PERRY_GC_FORCE_EVACUATE=0 — i.e. whether it needs actual relocation or only the copying-minor code path. A core dump / lldb backtrace on a repro run should localise it quickly given how reliably it fires.
Evidence from the #6981 investigation — and an explicit warning against over-reading it
While fixing #6981 (PR #6990), five of that issue's fourteen red files turned out to be GC-internal crashes rather than the typed-array-constructor bug: repsel_canonical_i32, repsel_ptr_shape_locals, repsel_proven_this_frozen, ta_param_numeric_read, typedarray_param_read.
The concrete part
In two crashes examined in detail (an ad-hoc probe and typedarray_param_read), the faulting instruction was identical:
That is a GC-header obj_type read at addr-8 — the collector validating a pointer that should never have reached it. So the defect is not in the read; it is in whatever installs that pointer. typedarray_param_read was bisected to a single line, console.log("u8", xorU8(u8, 8)), whose init sequence and param-read shape each pass in isolation.
The part that is NOT established — please do not inherit this as fact
An earlier summary (mine) described these as "frame-for-frame identical stacks — one shared collector defect, not five bugs." That overstates the evidence:
The stacks were matched by offset-into-unnamed-symbol in a stripped binary (+46704, +156, +72, +60, +14656, +8048), not by symbol name. Strong pattern-matching, not proof of a shared path.
Only two stacks were compared. The other three crashers were grouped by crash class, not by stack.
So "5 sites, 1 defect" is a hypothesis with n=2, and any claim that it subsumes this issue is downstream of that hypothesis.
Cheapest next step
Rebuild with symbols and re-collect stacks for all five. That settles in one build whether this is one defect or several, and it should come before any fix attempt — scoping a fix around a shared root cause that turns out to be three separate bugs would waste the work.
Reproduce on the evacuating precise-roots arm, and confirm relocation actually happened ([gc-copy-minor] copied_objects=N, N > 0):
Settled: symbolized stacks for all five, and the premise does not survive them
You were right to insist on this before any fix. No symbols build was needed — post_link.rs::strip_final_binary skips strip when PERRY_DEBUG_SYMBOLS=1 is set, so the (already unstripped) runtime symbols survive into the linked test binary and the macOS crash report symbolizes directly.
All five reproduce from aa1c15028 with copied_objects matching #6981's table exactly (4 579 / 4 640 / 4 614 / 5 948 / 5 965), and without auto-optimize.
These are not GC-internal crashes. There is not one GC frame in any of the five — no mark, trace, copying or evacuate. The ldrb w8, [x21, #-0x8]! is js_array_length reading an array header at arr-8: an ordinary mutator read. The collector is not the code validating a bad pointer; it is the code that moved the object out from under the mutator. (is_plausible_heap_addr would have rejected both quoted addresses anyway — on macOS HEAP_MIN is 2 TB and both are below it — which is a hint the faulting read was never a validation path.)
The real faulting addresses carry ASCII.0x434c4f53 = "CLOS", 0x004e614e = "NaN" — relocated string payload being read where a pointer/descriptor was expected. That is the signature of a stale from-space pointer whose bytes have been recycled, not of a small integer or tag remnant.
One defect or several?
The five crashes shared one blocking defect, and it was masking at least two more. All five fault downstream of the same js_get_global_this_builtin_value -> populate_global_this_builtins spine, at four different sites — the site varies only with where the recycled bytes landed. So "one shared defect" was directionally right, but it is one shared first hurdle, not the whole story.
Root cause: js_get_global_this registers two root slots for the singleton so an evacuating collector rewrites them, then passes the raw pre-GC pointer by value into populate_global_this_builtins, which allocates on nearly every step. Fixed in #6994 by rooting it in a RuntimeHandleScope and re-reading through the handle.
After that fix, crashes on the arm go 6 -> 2, repsel_canonical_i32 flips to byte-identical-to-Node, and two independent defects become observable: #6991 (compiled constructor receiver stale — the #6983 ctor-receiver class) and #6992 (proven-this method value lost).
Also worth knowing: the per-PR matrix arms cannot reproduce this class at all. On the unfixed binaries, default, verify_evac, cons_scan_off and shipped_default are all green; the evacuating base needs PERRY_GC_INCREMENTAL=0andPERRY_CONSERVATIVE_STACK_SCAN=off together, which only appears in --arms all (push-only). Filed as #6993.
Likely resolved by #7050, but not closing — this is an intermittent bug and one clean pass is not enough
The matrix on e279b2d54 (post-#7050) reports test_gap_repsel_gc_stressPASS in all 20 arms, including the conservative-scan-ON arms this issue is specifically about. #7050's root cause plausibly covers it: Arena::alloc(&mut self, ..) called gc_check_trigger() inside its own &mut self borrow, so a collection allocating into the same arena could grow self.blocks underneath the live borrow — silent malloc-heap corruption whose surfacing point is wherever the next allocation lands. That mechanism is indifferent to whether the conservative scan is on; it only needs a collection to install a block, which is why the scan-ON population would go quiet too.
But this issue documented a ~2-in-3 failure rate, and the evidence above is a single matrix pass. For an intermittent fault that is weak evidence — the same pass would have looked clean by luck roughly 1 time in 27 even with the bug fully present.
Before closing, someone should run the scan-ON arms repeatedly — 20+ runs, the standard this campaign used for #7022 (139 × 20 → 0 × 20) — and confirm zero failures. If that comes back clean, close it citing #7050.
Recorded rather than closed because a wrongly-closed intermittent GC bug is close to unrecoverable: it comes back months later as an unrelated-looking crash, and the search starts from zero.
Re-tested on main = d4ac65f21 and deliberately NOT closed. The reproducer now passes 10/10 — and that result is worthless as evidence. Recording it so nobody closes this on the same run.
What happened
Exactly the arm this issue specifies (conservative native-stack scan ON):
10/10 rc=0, byte-exact against node 26.5.1 — versus the rc=139 roughly 2-in-3 reported here.
Why that proves nothing
PERRY_GC_DIAG=1 on the same binary and the same arm reports 0 copying minors. Adding PERRY_GC_FORCE_EVACUATE=1 still gives 0.
This issue is a crash inside a copying minor ("the passing runs report [gc-copy-minor] ran copied_objects=... in the thousands"). A configuration that runs zero copying minors cannot exhibit it, so 10/10 green here means only that the code path was never entered. Under CLAUDE.md's fourth way a gate can be unable to fail, the subject never ran.
What it is blocked on
#7255 — four of the six PR-gating matrix arms run zero copying minors corpus-wide, and this is the same inertness reaching this issue's arm. Until the requires=scavenge route relocates again, this issue is untestable as written, not fixed.
What would actually close it
Either (a) #7255 is resolved and this arm relocates again, then re-run 10× and compare; or (b) this issue is re-scoped onto an arm that demonstrably moves — +PERRY_CONSERVATIVE_STACK_SCAN=off does relocate on this build, but that changes the premise, since "conservative pinning still in force" is the distinguishing feature of this report versus the precise-roots family.
Still not safely judgeable — and the reason is measurable, not "it's intermittent"
Measured while fixing #7255, on 61410a222, release, macOS arm64, node 26.5.1.
This issue's configuration is now intermittently inert. Ten runs of test_gap_repsel_gc_stress under exactly the env in the report — conservative
stack scan ON, which is the whole point of it:
So "passes 10/10" here is really "passes once, and the other nine runs did not
execute the subject at all". That is the #7255 pathology at the level of a single
issue, and it is why the earlier sweep read this as fixed.
Half of the mechanism is established: with PERRY_GC_MOVING_LOOP_POLLS
default-OFF (#7161's stopgap for #7154) the alloc-point minor runs behind ManualGcScanGuard::force_full_scan(), which makes the copying minor ineligible
(fallback=conservative_stack) — so the alloc-point route contributes zero. Which
entry supplies the one-in-ten copying minor is not established here; the
obvious candidate is the microtask-pump safepoint (gc_safepoint_moving_minor,
gated by gc_moving_safepoint_enabled, default ON), whose timing depends on when
the workload happens to yield, but I did not confirm it.
The neighbouring deterministic arm is clean.evac_minor — the same env plus PERRY_CONSERVATIVE_STACK_SCAN=off — runs 10 copying minors and copies 1 385 325
objects on every run, and is 10/10 byte-exact. But that is a different claim:
this issue is specifically about a crash with conservative pinning still in
force, so a green there does not settle it.
Deliberately not added as a matrix arm. An incr_off arm carrying this exact
env would satisfy its requires=scavenge about one run in ten per file, i.e. a
flaky gate — and #7255's sequencing constraint is that no new permanent (or
intermittent) red gets created. What is needed first is for the collector to
reach a copying minor with the conservative scan on reliably; then the arm is
worth adding and this issue becomes judgeable in one run instead of ten.
Leaving open. Next step for whoever picks it up: reproduce with the table above
in hand, and count copying minors before believing any verdict.
Closing — 40/40 clean on current main (v0.5.1432), with the arm live in every single run.
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 ./gc_stress × 40
→ nonzero exits: 0
→ runs with a live copying minor (copied_objects > 0): 40/40
The number that matters is 40/40 live, not 40/40 clean
This issue's own last comment described the configuration as "intermittently inert", and an earlier 150-run pass confirmed it precisely: 145 of 150 runs performed zero copying minors (eligible_false_count=23 each), with only 5 firing — and those 5 were clean. A 150-run soak in which 145 runs exercised nothing is not evidence the bug is gone.
On current main every run collects. What changed is #7721, which flipped PERRY_GC_MOVING_LOOP_POLLS to default ON — the back-edge poll is the precise collection point a compute-only program reaches, and it was default-OFF from #7161 until then. The five that did fire previously almost certainly came through the microtask-pump safepoint, the only route that needs no compile-time polls.
So this is a genuinely different experiment from the one that left the issue open, and it is the one the issue was waiting for: the arm is live 40/40 and the SIGSEGV does not occur.
Reopen with the copied_objects count if it returns — a run that collected nothing cannot distinguish a fixed bug from an unexercised one.
Activity
test(gc): keep the evacuating arms out of the PR subset until #6981 c…
fix(gc): a compiled program can now actually reach an evacuating minor (
proggeramlug commentedon Jul 29, 2026
Evidence from the #6981 investigation — and an explicit warning against over-reading it
While fixing #6981 (PR #6990), five of that issue's fourteen red files turned out to be GC-internal crashes rather than the typed-array-constructor bug:
repsel_canonical_i32,repsel_ptr_shape_locals,repsel_proven_this_frozen,ta_param_numeric_read,typedarray_param_read.The concrete part
In two crashes examined in detail (an ad-hoc probe and
typedarray_param_read), the faulting instruction was identical:That is a GC-header
obj_typeread ataddr-8— the collector validating a pointer that should never have reached it. So the defect is not in the read; it is in whatever installs that pointer.typedarray_param_readwas bisected to a single line,console.log("u8", xorU8(u8, 8)), whose init sequence and param-read shape each pass in isolation.The part that is NOT established — please do not inherit this as fact
An earlier summary (mine) described these as "frame-for-frame identical stacks — one shared collector defect, not five bugs." That overstates the evidence:
+46704,+156,+72,+60,+14656,+8048), not by symbol name. Strong pattern-matching, not proof of a shared path.So "5 sites, 1 defect" is a hypothesis with n=2, and any claim that it subsumes this issue is downstream of that hypothesis.
Cheapest next step
Rebuild with symbols and re-collect stacks for all five. That settles in one build whether this is one defect or several, and it should come before any fix attempt — scoping a fix around a shared root cause that turns out to be three separate bugs would waste the work.
Reproduce on the evacuating precise-roots arm, and confirm relocation actually happened (
[gc-copy-minor] copied_objects=N, N > 0):A green run where nothing moved proves nothing — the failure mode #6942/#6946/#6950 documented and #6977 removed.
Related: #6981, #6990, #6968, #6977.
docs(changelog): add the #6982 globalThis population rooting fragment
proggeramlug commentedon Jul 29, 2026
Settled: symbolized stacks for all five, and the premise does not survive them
You were right to insist on this before any fix. No symbols build was needed —
post_link.rs::strip_final_binaryskipsstripwhenPERRY_DEBUG_SYMBOLS=1is set, so the (already unstripped) runtime symbols survive into the linked test binary and the macOS crash report symbolizes directly.All five reproduce from
aa1c15028withcopied_objectsmatching #6981's table exactly (4 579 / 4 640 / 4 614 / 5 948 / 5 965), and without auto-optimize.The stacks
Two corrections to the issue
These are not GC-internal crashes. There is not one GC frame in any of the five — no mark, trace, copying or evacuate. The
ldrb w8, [x21, #-0x8]!isjs_array_lengthreading an array header atarr-8: an ordinary mutator read. The collector is not the code validating a bad pointer; it is the code that moved the object out from under the mutator. (is_plausible_heap_addrwould have rejected both quoted addresses anyway — on macOSHEAP_MINis 2 TB and both are below it — which is a hint the faulting read was never a validation path.)The real faulting addresses carry ASCII.
0x434c4f53="CLOS",0x004e614e="NaN"— relocated string payload being read where a pointer/descriptor was expected. That is the signature of a stale from-space pointer whose bytes have been recycled, not of a small integer or tag remnant.One defect or several?
The five crashes shared one blocking defect, and it was masking at least two more. All five fault downstream of the same
js_get_global_this_builtin_value -> populate_global_this_builtinsspine, at four different sites — the site varies only with where the recycled bytes landed. So "one shared defect" was directionally right, but it is one shared first hurdle, not the whole story.Root cause:
js_get_global_thisregisters two root slots for the singleton so an evacuating collector rewrites them, then passes the raw pre-GC pointer by value intopopulate_global_this_builtins, which allocates on nearly every step. Fixed in #6994 by rooting it in aRuntimeHandleScopeand re-reading through the handle.After that fix, crashes on the arm go 6 -> 2,
repsel_canonical_i32flips to byte-identical-to-Node, and two independent defects become observable: #6991 (compiled constructor receiver stale — the #6983 ctor-receiver class) and #6992 (proven-thismethod value lost).Also worth knowing: the per-PR matrix arms cannot reproduce this class at all. On the unfixed binaries,
default,verify_evac,cons_scan_offandshipped_defaultare all green; the evacuating base needsPERRY_GC_INCREMENTAL=0andPERRY_CONSERVATIVE_STACK_SCAN=offtogether, which only appears in--arms all(push-only). Filed as #6993.fix(gc): root globalThis across lazy builtin population (#6982)
docs(changelog): add the #6982 globalThis population rooting fragment
fix(gc): root globalThis across its own lazy builtin population (#6982)…
proggeramlug commentedon Jul 30, 2026
Likely resolved by #7050, but not closing — this is an intermittent bug and one clean pass is not enough
The matrix on
e279b2d54(post-#7050) reportstest_gap_repsel_gc_stressPASS in all 20 arms, including the conservative-scan-ON arms this issue is specifically about. #7050's root cause plausibly covers it:Arena::alloc(&mut self, ..)calledgc_check_trigger()inside its own&mut selfborrow, so a collection allocating into the same arena could growself.blocksunderneath the live borrow — silent malloc-heap corruption whose surfacing point is wherever the next allocation lands. That mechanism is indifferent to whether the conservative scan is on; it only needs a collection to install a block, which is why the scan-ON population would go quiet too.But this issue documented a ~2-in-3 failure rate, and the evidence above is a single matrix pass. For an intermittent fault that is weak evidence — the same pass would have looked clean by luck roughly 1 time in 27 even with the bug fully present.
Before closing, someone should run the scan-ON arms repeatedly — 20+ runs, the standard this campaign used for #7022 (
139 × 20 → 0 × 20) — and confirm zero failures. If that comes back clean, close it citing #7050.Recorded rather than closed because a wrongly-closed intermittent GC bug is close to unrecoverable: it comes back months later as an unrelated-looking crash, and the search starts from zero.
fix(gc): run the lazy realm bootstraps in a no-move window (#7217)
fix(gc): run the globalThis bootstrap in a no-move window (#7217)
fix(gc): run the globalThis bootstrap in a no-move window (#7217)
fix(gc): run the globalThis bootstrap in a no-move window (#7217)
fix(gc): run the globalThis bootstrap in a no-move window (#7217) (#7249
proggeramlug commentedon Aug 2, 2026
Re-tested on
main=d4ac65f21and deliberately NOT closed. The reproducer now passes 10/10 — and that result is worthless as evidence. Recording it so nobody closes this on the same run.What happened
Exactly the arm this issue specifies (conservative native-stack scan ON):
10/10
rc=0, byte-exact against node 26.5.1 — versus therc=139roughly 2-in-3 reported here.Why that proves nothing
PERRY_GC_DIAG=1on the same binary and the same arm reports 0 copying minors. AddingPERRY_GC_FORCE_EVACUATE=1still gives 0.This issue is a crash inside a copying minor ("the passing runs report
[gc-copy-minor] ran copied_objects=...in the thousands"). A configuration that runs zero copying minors cannot exhibit it, so 10/10 green here means only that the code path was never entered. Under CLAUDE.md's fourth way a gate can be unable to fail, the subject never ran.What it is blocked on
#7255 — four of the six PR-gating matrix arms run zero copying minors corpus-wide, and this is the same inertness reaching this issue's arm. Until the
requires=scavengeroute relocates again, this issue is untestable as written, not fixed.What would actually close it
Either (a) #7255 is resolved and this arm relocates again, then re-run 10× and compare; or (b) this issue is re-scoped onto an arm that demonstrably moves —
+PERRY_CONSERVATIVE_STACK_SCAN=offdoes relocate on this build, but that changes the premise, since "conservative pinning still in force" is the distinguishing feature of this report versus the precise-roots family.Part of the sweep in #7262.
proggeramlug commentedon Aug 2, 2026
Still not safely judgeable — and the reason is measurable, not "it's intermittent"
Measured while fixing #7255, on
61410a222, release, macOS arm64, node 26.5.1.This issue's configuration is now intermittently inert. Ten runs of
test_gap_repsel_gc_stressunder exactly the env in the report — conservativestack scan ON, which is the whole point of it:
So "passes 10/10" here is really "passes once, and the other nine runs did not
execute the subject at all". That is the #7255 pathology at the level of a single
issue, and it is why the earlier sweep read this as fixed.
Half of the mechanism is established: with
PERRY_GC_MOVING_LOOP_POLLSdefault-OFF (#7161's stopgap for #7154) the alloc-point minor runs behind
ManualGcScanGuard::force_full_scan(), which makes the copying minor ineligible(
fallback=conservative_stack) — so the alloc-point route contributes zero. Whichentry supplies the one-in-ten copying minor is not established here; the
obvious candidate is the microtask-pump safepoint (
gc_safepoint_moving_minor,gated by
gc_moving_safepoint_enabled, default ON), whose timing depends on whenthe workload happens to yield, but I did not confirm it.
The neighbouring deterministic arm is clean.
evac_minor— the same env plusPERRY_CONSERVATIVE_STACK_SCAN=off— runs 10 copying minors and copies 1 385 325objects on every run, and is 10/10 byte-exact. But that is a different claim:
this issue is specifically about a crash with conservative pinning still in
force, so a green there does not settle it.
Deliberately not added as a matrix arm. An
incr_offarm carrying this exactenv would satisfy its
requires=scavengeabout one run in ten per file, i.e. aflaky gate — and #7255's sequencing constraint is that no new permanent (or
intermittent) red gets created. What is needed first is for the collector to
reach a copying minor with the conservative scan on reliably; then the arm is
worth adding and this issue becomes judgeable in one run instead of ten.
Leaving open. Next step for whoever picks it up: reproduce with the table above
in hand, and count copying minors before believing any verdict.
proggeramlug commentedon Aug 10, 2026
Closing — 40/40 clean on current
main(v0.5.1432), with the arm live in every single run.The number that matters is 40/40 live, not 40/40 clean
This issue's own last comment described the configuration as "intermittently inert", and an earlier 150-run pass confirmed it precisely: 145 of 150 runs performed zero copying minors (
eligible_false_count=23each), with only 5 firing — and those 5 were clean. A 150-run soak in which 145 runs exercised nothing is not evidence the bug is gone.On current main every run collects. What changed is #7721, which flipped
PERRY_GC_MOVING_LOOP_POLLSto default ON — the back-edge poll is the precise collection point a compute-only program reaches, and it was default-OFF from #7161 until then. The five that did fire previously almost certainly came through the microtask-pump safepoint, the only route that needs no compile-time polls.So this is a genuinely different experiment from the one that left the issue open, and it is the one the issue was waiting for: the arm is live 40/40 and the SIGSEGV does not occur.
Reopen with the
copied_objectscount if it returns — a run that collected nothing cannot distinguish a fixed bug from an unexercised one.