Repository navigation
gc(layer 3): from-space quarantine catches 55 stale dereferences across the gap suite — the instrument is in CI but aimed at one synthetic fixture #7341
Description
Summary
PERRY_GC_PROTECT_FROMSPACE catches 55 stale from-space dereferences across
the 479-test gap suite — 44 of them in programs that otherwise exit cleanly and
produce correct output. The instrument already exists, is well built, and is
already wired into CI. It is simply pointed at the wrong workload, so none of
these have ever been visible.
This is layer 3 of docs/engine-plan.md (rooting in hand-written runtime Rust),
the layer that #7280's surviving fault moved into and that no static gate can
see.
Why these were invisible
scripts/gc_instrument_smoke.sh is a good gate — it asserts both knob
directions and has a real non-vacuity check (zeal must produce strictly more
retirements than no-zeal). But it drives the instrument with:
PERRY_GC_MOVING_LOOP_POLLS=1 ... $WORK/fixture
over one synthetic fixture. Back-edge polls fire only while user JS runs, so
that configuration structurally cannot expose an unrooted pointer in runtime code
that does not re-enter user JS — which is most of the runtime. That limitation is
already documented from #7249; the smoke script predates applying it here.
Driving the same instrument by allocation-point pressure over real
programs instead:
PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64 \
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off \
./test_binary
Results
Every gap test compiled and run twice — quarantine OFF as a control, quarantine
ON — so a pre-existing crash cannot be miscounted as a catch.
measured 479
QUARANTINE-ONLY FAULTS (control ok, exp SIGBUS) 55
...of which the control exited 0 (clean) 44
faulted in BOTH arms (pre-existing) 9
no fault in either arm 415
The faults are self-diagnosing:
[gc-fromspace-protect] FAULT: signal 10 at 0x5cb232a03a9
This address is RETIRED FROM-SPACE. The evacuating minor moved or
freed the object here and the holder kept the pre-collection address.
block=0x5cb232a0000 +937 retired_bytes=1048536 retired_by_minor=#0
last-known object: user_ptr=0x5cb232a03b0 obj_type=1 size=8208
This is not 55 independent bugs
The signatures are heavily concentrated, which is the useful part:
| dimension | distribution |
|---|---|
| collection | 49 of 57 are retired_by_minor=#0 — the first minor, 7 at #1, 1 at #2 |
| object type | 34 are GC_TYPE_STRING, 8 GC_TYPE_PROMISE, 4 GC_TYPE_OBJECT, 2 GC_TYPE_CLOSURE |
| size | size=32 dominates (22) |
A stale *mut StringHeader across the first minor collection is the dominant
shape. That is consistent with the runtime exposing 218 functions returning
*mut StringHeader with nothing obliging a caller to root the result, and with
#7247 (js_regexp_new holding &str borrows into movable StringHeaders).
The minor-#0 concentration also rhymes with #7217, where the first collection
landed inside a bootstrap holding a raw object graph — worth checking whether
these share that cause before treating them as separate.
Three of the catches are tests named for GC rooting, which is a useful
sanity check that the instrument is finding the right class:
test_gap_gc_assign_string_source_rooting, test_gap_gc_pointer_publish,
test_gap_gc_process_env_cache_rooting.
Verification
test_gap_gc_pointer_publishfaults 20/20 runs — deterministic, not a
schedule lottery.- Cross-checked against
PERRY_GC_PROTECT_FROMSPACE=poison, which poisons
retired bytes without changing page protection. Several tests change
observable behaviour under poison alone (test_gap_json_advancedgoes
rc=0→rc=1), which confirms retired memory is genuinely being read rather
than the mprotect being over-eager.
Proposed work
- Triage the minor-#0 /
GC_TYPE_STRINGcluster first — it is very likely a
small number of causes behind ~60% of the catches. - Add a CI arm that runs the quarantine over the gap suite by the
allocation-point route, alongside the existing loop-polls smoke. The
instrument is already correct; only its aim needs fixing.
Full list of the 44 clean-program catches:
test_gap_6300_const_cast_assigntest_gap_6375_legacy_url_parse_formattest_gap_6386_dataview_concat_regex_fastpathstest_gap_6812_object_write_loop_generalizationtest_gap_asynchooks_3089_3090_3091_3093test_gap_bind_replace_2840_2867test_gap_class_forward_capture_6523test_gap_computed_namespace_dispatchtest_gap_concise_method_explicit_thistest_gap_cp_3079plustest_gap_crypto_cipher_3382_3381_2954test_gap_crypto_state_2944_2945_2962test_gap_diagchannel_3082_3084_3085_3086test_gap_dynamic_import_init_timetest_gap_encoding_timerstest_gap_fetch_responsetest_gap_field_lane_semanticstest_gap_fs_write_error_propagationtest_gap_gc_assign_string_source_rootingtest_gap_gc_pointer_publishtest_gap_gc_process_env_cache_rootingtest_gap_http2_settingstest_gap_in_operator_numeric_keytest_gap_in_operator_to_property_keytest_gap_json_advancedtest_gap_json_array_element_overflow_fieldstest_gap_node_crypto_buffertest_gap_node_fstest_gap_node_processtest_gap_object_assign_collectiontest_gap_process_misc_3045plustest_gap_put_value_plan_cachetest_gap_readable_stream_tee_pulltest_gap_reflect_semantics2test_gap_repsel_canonical_str_localstest_gap_request_binary_body_5483test_gap_stream_async_transform_tick_paritytest_gap_stream_id_band_dynamic_dispatchtest_gap_stream_livetee_tick_paritytest_gap_stream_tee_tick_paritytest_gap_string_index_get_object_receivertest_gap_transform_stream_deferred_flushtest_gap_worker_channels_3157plustest_gap_zlib_fs_assert_2935_2752_2971
Activity
proggeramlug commentedon Aug 3, 2026
Post-fix measurement: 55 → 54, and the remaining cluster is sharper
#7342 fixed the array-store-receiver cluster. Re-running the same two-arm sweep
against it:
So that fix closed exactly one test (
test_gap_gc_pointer_publish) andintroduced none. Worth stating plainly: the array-store receiver was a real bug
with a deterministic reproducer, but it is a small slice of this issue.
The remaining 54 are dominated by one shape
retired_by_minor=#0, 7 at #1, 1 at #2GC_TYPE_STRING, 8GC_TYPE_PROMISE, 5GC_TYPE_OBJECT, 2GC_TYPE_CLOSUREA worked example, because it redirects the search
Smallest string-cluster case is
test_gap_object_assign_collection(20 lines).The interesting part is that
js_object_assign_oneis already carefullyrooted — #7200 gave it a
RuntimeHandleScope, rootstarget,src,sourceand
arr, and reads each back. The array branch does:There is no collection point between the root and the read-back, so
arrcanonly be stale if
src_rawwas already stale when the scope was opened — i.e.the argument arrived stale. The last line of that test is
Object.assign({ a: 1 }, { b: 2 }, [10, 20]): the array literal is allocated,then further allocation happens before the helper is called.
That means a meaningful part of this issue is not layer 3 at all. Rooting a
pointer that is already dangling publishes the dangling value — the fix has to be
at the call site, in the same family as #6969/#7114 argument rooting and #7342's
store receiver, not inside the runtime helper.
Anyone picking this up: get the backtrace first
(
--debug-symbols+ the allocation-point route below), and check whether thefaulting helper already has a scope. If it does, the bug is in its caller.
#7342 also adds arm 4 to
gc_instrument_smoke.sh, which runs this route over thegc_ratchetprobes — so anything reproduced as a probe is gated from then on.fix(gc): root the array-store receiver across an allocating RHS (#7342)
docs(gc): plan status after #7339/#7340/#7342 — the 0→2 spine is done (…
fix(gc): stop reading a string's bytes after the string moved (#7363)
proggeramlug commentedon Aug 4, 2026
Cluster breakdown of the remaining catches, by faulting site
Backtraced all 31 catches with
--debug-symbolsand grouped byframe#0 ← frame#1. They are not 31 bugs:DirectParser::parse_value←js_json_parselookup_fancy_regex← 4 different callersjs_promise_state←mainjs_symbol_to_string←js_jsvalue_to_stringjs_object_literal_infer_computed_function_namejs_object_get_field_by_nameThe method that works
Two clusters in, the reliable procedure is:
--debug-symbols, run underPERRY_GC_PROTECT_FROMSPACE=1+ the allocation-point route, take a backtrace.&strtaken from a heap object, then a collection point, then a use.Two things that do not work, both measured:
The regex cluster (5) is caller-side, not layer 3
lookup_fancy_regexappears under four different callers, which suggested one shared bug. It is not in that helper:string_as_str+to_string()pair closed 0/5.obj_type=3(a 32-byte string) atlookup_fancy_regex + 184, which isstring_as_str((*re).pattern_ptr).RegExpHeader's string slots are correctly traced —regex_gc_slot_ptrsexposespattern_ptr/flags_ptras an exact 2-slot range, gated on the header magic (gc/layout.rs).Those slots are only rewritten if the header itself is visited, i.e. reachable from a root at collection time. And nothing in
js_string_replace_regexallocates before the call — frame #2 is generated code. Sorewas already stale on entry.Hypothesis for whoever picks this up: a
RegExp-typed local is not being classified as pointer-bearing by codegen's root-set analysis, so it never gets a root slot and the header is invisible to the collector. That would make this a layer-2 gap that statepoints inherit rather than fix — the same shape as #7342, where "both root backends faulted identically" because the value had no root slot at all.Cheap way to test it: compile one of the five with
--trace llvmand check whether the regex local gets a shadow/statepoint root.proggeramlug commentedon Aug 4, 2026
23 of 31 closed; here is the triage for the remaining 8
Five clusters fixed across four PRs — #7373 (JSON, 8), #7374 (regex flags, 5), #7375 (await, 4), #7376 (symbol description, 6).
Every fix was an ORDERING bug, not a missing root
Every one already had rooting of some kind — the pattern's root, the receiver's root, the per-block unbox, the registry for registered symbols. What was missing in each case was ordering the root relative to the collection point. "Add roots" would have been the wrong lesson; "root before the allocation, re-read after" is the right one.
Two of the four were a long-lived object's field written after the allocation that moved it, which is why the most productive rule turned out to be: if a field of a long-lived object is stale, suspect the constructor, not the reader.
Methods that measurably do NOT work
Recorded because each looked obvious and each closed exactly zero:
lookup_fancy_regexclosed 0/5.replacelowering closed 0/5, even after confirming in the IR that the root landed and moving the re-read to the point of use.objacross the key allocation inconstructor_dynamic_prototypeclosed 0/3.Grouping by frame #0 over-counts clusters
js_object_literal_infer_computed_function_namewas triaged as its own cluster. It reaches the same stale(*sym_ptr).descriptionthroughinfer_symbol_function_name, so #7376 closed it too. One bad field, two readers, two apparent clusters.The remaining 8, by stale object type
field_lane_semanticsjs_object_get_field_by_name←..._f64put_value_plan_cachestream_id_band_dynamic_dispatchjs_object_get_field_by_name←fetch_subclass_handle_iddiagchannel_3082_…decode_string_value←throw_tracing_channel_symbolgc_assign_string_source_rootingmirror_class_object_static_write←js_object_set_field_by_namegc_process_env_cache_rootingjs_array_push_f64←make_iter_resultobject_assign_collectionjs_array_length←js_object_assign_onefetch_responsejs_promise_resolve←js_request_array_bufferThe three OBJECT ones are one group and are harder than what is fixed so far. The fault is at
js_object_get_field_by_name + 664— inside the function, on a 72-byte OBJECT — and rooting the caller's receiver does not close it. That points at the prototype-chain walk holding a link across an allocation, rather than a stale argument. Worth doing next because it is three catches and a hot path; worth budgeting more than the others because the ordering lens alone does not locate it.object_assign_collectionis the case analysed earlier in this issue:js_object_assign_oneis already carefully rooted (#7200), so the argument arrives stale — caller-side.proggeramlug commentedon Aug 4, 2026
The 3-catch OBJECT group: exact faulting instruction, and two more failed hypotheses
Disassembled it rather than guessing further:
So the fault is the fast lane's GC-header type probe in
js_object_get_field_by_name, dereferencing a stale receiver atuser_ptr - 8. The fast lane's own comment describes it as resolving a field "with no rooting scope, no exotic-registry probes, and no key hashing" — the absence of rooting is a deliberate design property, which is what makes a stale receiver fatal here rather than merely wrong.Two hypotheses tested and killed
Both measured 0/3:
constructor_dynamic_prototype. Itsjs_string_from_bytes(b"constructor")is a real collection point withobjused after it — a textbook instance of the ordering bug fixed four times elsewhere in this issue. Rootingobjacross it and re-reading closed nothing.process_env_get_fieldprobe at the top ofjs_object_get_field_by_name, which does an OS lookup and materializes a StringHeader. Rootingobjandkeyacross it and re-reading closed nothing.Since neither in-function collection point is responsible,
objarrives stale at the call. Frame #1 differs across the three tests —js_object_get_field_by_name_f64(the typed-thisIC-miss fallback) for two of them,fetch_subclass_handle_idfor the third — so the next step is the callers, not this function.Why this group is worth a separate budget
The four clusters already fixed shared one shape: a collection point with a live raw pointer either side of it, locatable by reading ~30 lines around the faulting frame. This one is not that. The receiver is already dead on entry, three different callers reach it, and the consuming code is a fast path that deliberately omits rooting — so the fix is either at each caller or a decision that the fast lane must root after all, which has a measurable cost on the hottest property-read path in the runtime.
Recording the disassembly so whoever picks this up starts from the instruction rather than re-deriving it.
proggeramlug commentedon Aug 4, 2026
The remaining 8 are a DIFFERENT class from the 23 fixed, and here is the evidence
Disassembled each rather than guessing. Every one faults on an early dereference of an argument — the pointer is already dead on entry:
gc_assign_string_source_rootingldr w8, [x0]+28(*obj).object_type, the function's FIRST derefjs_object_set_field_by_name +8708object_assign_collectionldurb w9, [x20,#-0x8]+468js_object_assign_one +1456diagchannel_3082_…ldr w1, [x0,#0x4]+260byte_lenof a stale StringHeaderthrow_tracing_channel_symbol +28field_lane_semanticsldurb w8, [x22,#-0x8]+664js_object_get_field_by_name_f64put_value_plan_cachestream_id_band_dynamic_dispatchfetch_subclass_handle_idWhy this matters for planning
The 23 already fixed shared one shape: a collection point visible inside the faulting function, with a live raw pointer on either side of it. That is locatable by reading ~30 lines around the frame, and four such fixes landed (#7373–#7376).
These 8 are not that. The argument is stale before the callee runs, so there is nothing to fix in the function the backtrace names. I confirmed this the expensive way — four consecutive callee-side fixes measured exactly zero:
lookup_fancy_regex— 0/5replacelowering — 0/5objacross the key allocation inconstructor_dynamic_prototype— 0/3obj+keyacross theprocess_env_get_fieldprobe — 0/3mirror_class_object_static_write— 0/1 (the fault is at +28, before that borrow)What this class actually needs
Each one requires walking up to find where the caller lost the pointer, and the callers differ per test. Two structural notes:
js_object_assign_oneis already carefully rooted (GC:{ ...src }with an accessor source SIGSEGVs under PERRY_GC_MOVING_LOOP_POLLS=1 #7200), so its argument arriving stale means the caller above it is at fault — two levels up.js_object_get_field_by_namefast lane deliberately has no rooting scope (its comment says so). Making it root has a measurable cost on the hottest property-read path, so that is a design decision rather than a patch.Recommendation: treat these 8 as a separate work item from the 23. They are not more of the same, and the method that closed the first four does not apply. The disassembly above is the starting point so nobody re-derives it.
fix(gc): JSON.parse read its input after the source string moved (#7373)
fix(gc): a RegExp's flags string was stored into the header already s…
fix(gc): await polled a promise the event-loop pump had already moved (…
fix(gc): a Symbol's description was stored into the header already st…
docs(rfc): score rooting-by-construction against four bugs found afte…
fix(gc): Object.assign type-confused a RegExp source as a plain object (
fix(gc): refresh obj before the class-static mirror on both own-data-…
9 remaining items
proggeramlug commentedon Aug 11, 2026
Full sweep re-run on
main@db44b31b7— all 523 gap tests, both arms, exactly the configuration this issue specifies. The population is essentially gone: 55 → 1, and the 44 that mattered most → 0.Method unchanged from the issue: compile once, run control (
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off) then experiment (+ PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64), and count only faults present in the quarantine arm and absent from the control.The one remaining catch is not the shape this issue is about
Its control exit is 1 — the program was already failing before the quarantine was armed.
test_gap_backoff_optionsis one of the 19 pre-existing failures in the release gap sweep (output mismatch). So it is not a case of "clean output hiding a stale dereference", which is the category this issue was opened for and which stood at 44.The result is not vacuous
The instrument's arming was measured separately rather than assumed, since that is this issue's own thesis — the smoke gate was pointed at a workload that structurally could not expose these. Across a 45-test subsample: 42 of 45 armed (≥1 retired page-set), 65 retired sets total, with a single-test probe showing
mode=ProtectPages retired_set=#0 blocks=1 bytes_protected=104857alongside a copying minor. And the full sweep did catch one, so the detector was live end to end.Why the population went away
Nothing here targeted these 55 directly. The layer-3 campaign did: #7206 (call receiver across sibling operands), #7214 (
js_closure_callN's callee/this/args), #7217 (the lazyglobalThisbootstrap window), #6943 (thejs_string_coerce-as-property-key family), and RS4GC operand re-reads — on #7248 I measured the previously-stalejs_eqleft operand now being reloaded from its slot below the allocating sibling, which is precisely this failure mode being fixed structurally rather than site-by-site.Closing: the measured claim in this issue no longer holds. The instrument-aiming half also moved — #7793 (merged) drives the same quarantine at
DEPTH=800over the 81-module zod corpus ingc-moving-witnesses.ymland asserts its own arming, so CI now points it at dependency-scale real code rather than one synthetic fixture. If a future sweep finds this rate climbing again, the two-arm script in this thread is the reproduction.refactor(gc): use the sanctioned handle accessors in the arraylike loops
fix(gc): root heap values held across JS-running calls in runtime and…
refactor(gc): sanctioned handle accessors in the bound-method builders
fix(intl): root Segmenter handle reads via the #7341 wrappers (#8376)
merge: land #8630 (class semantics tail) with six audit fixes (#8643)
fix(runtime): honor Array toString dispatch (#8732)
batch: land #8784, #8788, #8785, #8786 (#8803)
perf(regex): construct a RegExp without allocating its flags string
perf(regex): construct a RegExp without allocating its flags string
fix(train): split regex.rs and its tests for the file cap; classify t…
test(gc): read the lazy header's backing through its handle scope
test(gc): read the lazy header's backing through its handle scope
fix(gc): scope the static-accessor key read with with_const_ptr (#7341)