Skip to content

gc(layer 3): from-space quarantine catches 55 stale dereferences across the gap suite — the instrument is in CI but aimed at one synthetic fixture #7341

Closed

Description

@proggeramlug

Summary

PERRY_GC_PROTECT_FROMSPACE catches 55 stale from-space dereferences across
the 479-test gap suite
— 44 of them in programs that otherwise exit cleanly and
produce correct output. The instrument already exists, is well built, and is
already wired into CI. It is simply pointed at the wrong workload, so none of
these have ever been visible.

This is layer 3 of docs/engine-plan.md (rooting in hand-written runtime Rust),
the layer that #7280's surviving fault moved into and that no static gate can
see.

Why these were invisible

scripts/gc_instrument_smoke.sh is a good gate — it asserts both knob
directions and has a real non-vacuity check (zeal must produce strictly more
retirements than no-zeal). But it drives the instrument with:

PERRY_GC_MOVING_LOOP_POLLS=1 ... $WORK/fixture

over one synthetic fixture. Back-edge polls fire only while user JS runs, so
that configuration structurally cannot expose an unrooted pointer in runtime code
that does not re-enter user JS — which is most of the runtime. That limitation is
already documented from #7249; the smoke script predates applying it here.

Driving the same instrument by allocation-point pressure over real
programs
instead:

PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64 \
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off \
  ./test_binary

Results

Every gap test compiled and run twice — quarantine OFF as a control, quarantine
ON — so a pre-existing crash cannot be miscounted as a catch.

measured                                     479
QUARANTINE-ONLY FAULTS (control ok, exp SIGBUS)  55
  ...of which the control exited 0 (clean)       44
faulted in BOTH arms (pre-existing)               9
no fault in either arm                          415

The faults are self-diagnosing:

[gc-fromspace-protect] FAULT: signal 10 at 0x5cb232a03a9
  This address is RETIRED FROM-SPACE. The evacuating minor moved or
  freed the object here and the holder kept the pre-collection address.
  block=0x5cb232a0000 +937 retired_bytes=1048536 retired_by_minor=#0
  last-known object: user_ptr=0x5cb232a03b0 obj_type=1 size=8208

This is not 55 independent bugs

The signatures are heavily concentrated, which is the useful part:

dimension distribution
collection 49 of 57 are retired_by_minor=#0 — the first minor, 7 at #1, 1 at #2
object type 34 are GC_TYPE_STRING, 8 GC_TYPE_PROMISE, 4 GC_TYPE_OBJECT, 2 GC_TYPE_CLOSURE
size size=32 dominates (22)

A stale *mut StringHeader across the first minor collection is the dominant
shape. That is consistent with the runtime exposing 218 functions returning
*mut StringHeader
with nothing obliging a caller to root the result, and with
#7247 (js_regexp_new holding &str borrows into movable StringHeaders).

The minor-#0 concentration also rhymes with #7217, where the first collection
landed inside a bootstrap holding a raw object graph — worth checking whether
these share that cause before treating them as separate.

Three of the catches are tests named for GC rooting, which is a useful
sanity check that the instrument is finding the right class:
test_gap_gc_assign_string_source_rooting, test_gap_gc_pointer_publish,
test_gap_gc_process_env_cache_rooting.

Verification

  • test_gap_gc_pointer_publish faults 20/20 runs — deterministic, not a
    schedule lottery.
  • Cross-checked against PERRY_GC_PROTECT_FROMSPACE=poison, which poisons
    retired bytes without changing page protection. Several tests change
    observable behaviour under poison alone (test_gap_json_advanced goes
    rc=0 → rc=1), which confirms retired memory is genuinely being read rather
    than the mprotect being over-eager.

Proposed work

  1. Triage the minor-#0 / GC_TYPE_STRING cluster first — it is very likely a
    small number of causes behind ~60% of the catches.
  2. Add a CI arm that runs the quarantine over the gap suite by the
    allocation-point route, alongside the existing loop-polls smoke. The
    instrument is already correct; only its aim needs fixing.

Full list of the 44 clean-program catches:

  • test_gap_6300_const_cast_assign
  • test_gap_6375_legacy_url_parse_format
  • test_gap_6386_dataview_concat_regex_fastpaths
  • test_gap_6812_object_write_loop_generalization
  • test_gap_asynchooks_3089_3090_3091_3093
  • test_gap_bind_replace_2840_2867
  • test_gap_class_forward_capture_6523
  • test_gap_computed_namespace_dispatch
  • test_gap_concise_method_explicit_this
  • test_gap_cp_3079plus
  • test_gap_crypto_cipher_3382_3381_2954
  • test_gap_crypto_state_2944_2945_2962
  • test_gap_diagchannel_3082_3084_3085_3086
  • test_gap_dynamic_import_init_time
  • test_gap_encoding_timers
  • test_gap_fetch_response
  • test_gap_field_lane_semantics
  • test_gap_fs_write_error_propagation
  • test_gap_gc_assign_string_source_rooting
  • test_gap_gc_pointer_publish
  • test_gap_gc_process_env_cache_rooting
  • test_gap_http2_settings
  • test_gap_in_operator_numeric_key
  • test_gap_in_operator_to_property_key
  • test_gap_json_advanced
  • test_gap_json_array_element_overflow_fields
  • test_gap_node_crypto_buffer
  • test_gap_node_fs
  • test_gap_node_process
  • test_gap_object_assign_collection
  • test_gap_process_misc_3045plus
  • test_gap_put_value_plan_cache
  • test_gap_readable_stream_tee_pull
  • test_gap_reflect_semantics2
  • test_gap_repsel_canonical_str_locals
  • test_gap_request_binary_body_5483
  • test_gap_stream_async_transform_tick_parity
  • test_gap_stream_id_band_dynamic_dispatch
  • test_gap_stream_livetee_tick_parity
  • test_gap_stream_tee_tick_parity
  • test_gap_string_index_get_object_receiver
  • test_gap_transform_stream_deferred_flush
  • test_gap_worker_channels_3157plus
  • test_gap_zlib_fs_assert_2935_2752_2971

Activity

  1. proggeramlug commented on Aug 3, 2026

    @proggeramlug
    ContributorAuthor

    Post-fix measurement: 55 → 54, and the remaining cluster is sharper

    #7342 fixed the array-store-receiver cluster. Re-running the same two-arm sweep
    against it:

                            before   after
    quarantine-only faults      55      54
    faulted in both arms         9       9
    clean                      415     416
    correctness (vs node)   447/19/13 → 447/19/13   (unchanged, zero regressions)
    

    So that fix closed exactly one test (test_gap_gc_pointer_publish) and
    introduced none. Worth stating plainly: the array-store receiver was a real bug
    with a deterministic reproducer, but it is a small slice of this issue.

    The remaining 54 are dominated by one shape

    dimension distribution
    collection 48 of 56 at retired_by_minor=#0, 7 at #1, 1 at #2
    object type 41 GC_TYPE_STRING, 8 GC_TYPE_PROMISE, 5 GC_TYPE_OBJECT, 2 GC_TYPE_CLOSURE

    A worked example, because it redirects the search

    Smallest string-cluster case is test_gap_object_assign_collection (20 lines).

    frame #0: js_array_length + 468
    frame #1: js_object_assign_one + 1456
    frame #2: main + 2084
    

    The interesting part is that js_object_assign_one is already carefully
    rooted
    — #7200 gave it a RuntimeHandleScope, roots target, src, source
    and arr, and reads each back. The array branch does:

    let arr_h = scope.root_raw_const_ptr(src_raw as *const ArrayHeader);
    let arr = arr_h.get_raw_const_ptr::<ArrayHeader>();
    let n = crate::array::js_array_length(arr);   // ← faults here

    There is no collection point between the root and the read-back, so arr can
    only be stale if src_raw was already stale when the scope was opened — i.e.
    the argument arrived stale. The last line of that test is
    Object.assign({ a: 1 }, { b: 2 }, [10, 20]): the array literal is allocated,
    then further allocation happens before the helper is called.

    That means a meaningful part of this issue is not layer 3 at all. Rooting a
    pointer that is already dangling publishes the dangling value — the fix has to be
    at the call site, in the same family as #6969/#7114 argument rooting and #7342's
    store receiver, not inside the runtime helper.

    Anyone picking this up: get the backtrace first
    (--debug-symbols + the allocation-point route below), and check whether the
    faulting helper already has a scope. If it does, the bug is in its caller.

    PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64 \
    PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off \
      ./binary
    

    #7342 also adds arm 4 to gc_instrument_smoke.sh, which runs this route over the
    gc_ratchet probes — so anything reproduced as a probe is gated from then on.

  2. proggeramlug commented on Aug 4, 2026

    @proggeramlug
    ContributorAuthor

    Cluster breakdown of the remaining catches, by faulting site

    Backtraced all 31 catches with --debug-symbols and grouped by frame#0 ← frame#1. They are not 31 bugs:

    count faulting site status
    8 DirectParser::parse_value ← js_json_parse ✅ fixed in #7373
    5 lookup_fancy_regex ← 4 different callers caller-side, see below
    4 js_promise_state ← main
    3 js_symbol_to_string ← js_jsvalue_to_string
    3 js_object_literal_infer_computed_function_name
    2 js_object_get_field_by_name
    6 singletons

    The method that works

    Two clusters in, the reliable procedure is:

    1. Compile with --debug-symbols, run under PERRY_GC_PROTECT_FROMSPACE=1 + the allocation-point route, take a backtrace.
    2. Look for derive-then-collect ordering: a raw pointer or &str taken from a heap object, then a collection point, then a use.
    3. If the faulting helper already has a scope or root, the bug is in its caller.

    Two things that do not work, both measured:

    • Pattern-matching for the shape. A regex for "derive a payload pointer, then allocate within 25 lines" found 34 sites; applying a no-move window to the 28 non-loop-carrying ones closed zero catches (fix(gc): stop reading a string's bytes after the string moved #7363). Only backtrace-found sites were real.
    • Fixing the helper when the caller is at fault. See the regex cluster below.

    The regex cluster (5) is caller-side, not layer 3

    lookup_fancy_regex appears under four different callers, which suggested one shared bug. It is not in that helper:

    • Adding a no-move window around its string_as_str + to_string() pair closed 0/5.
    • The fault is obj_type=3 (a 32-byte string) at lookup_fancy_regex + 184, which is string_as_str((*re).pattern_ptr).
    • RegExpHeader's string slots are correctly traced — regex_gc_slot_ptrs exposes pattern_ptr/flags_ptr as an exact 2-slot range, gated on the header magic (gc/layout.rs).

    Those slots are only rewritten if the header itself is visited, i.e. reachable from a root at collection time. And nothing in js_string_replace_regex allocates before the call — frame #2 is generated code. So re was already stale on entry.

    Hypothesis for whoever picks this up: a RegExp-typed local is not being classified as pointer-bearing by codegen's root-set analysis, so it never gets a root slot and the header is invisible to the collector. That would make this a layer-2 gap that statepoints inherit rather than fix — the same shape as #7342, where "both root backends faulted identically" because the value had no root slot at all.

    Cheap way to test it: compile one of the five with --trace llvm and check whether the regex local gets a shadow/statepoint root.

  3. proggeramlug commented on Aug 4, 2026

    @proggeramlug
    ContributorAuthor

    23 of 31 closed; here is the triage for the remaining 8

    Five clusters fixed across four PRs — #7373 (JSON, 8), #7374 (regex flags, 5), #7375 (await, 4), #7376 (symbol description, 6).

    Every fix was an ORDERING bug, not a missing root

    cluster root cause
    JSON (8) root pushed after the collection point
    regex flags (5) field stored already stale at construction
    await (4) value re-derived from a pre-collection register
    symbol description (6) field stored already stale at construction

    Every one already had rooting of some kind — the pattern's root, the receiver's root, the per-block unbox, the registry for registered symbols. What was missing in each case was ordering the root relative to the collection point. "Add roots" would have been the wrong lesson; "root before the allocation, re-read after" is the right one.

    Two of the four were a long-lived object's field written after the allocation that moved it, which is why the most productive rule turned out to be: if a field of a long-lived object is stale, suspect the constructor, not the reader.

    Methods that measurably do NOT work

    Recorded because each looked obvious and each closed exactly zero:

    1. Pattern-matching the shape. A regex for "derive a payload pointer, then allocate within 25 lines" found 34 sites; windowing the 28 non-loop-carrying ones closed 0.
    2. Fixing the faulting helper when the constructor is at fault. A no-move window inside lookup_fancy_regex closed 0/5.
    3. Rooting the call-site operand for the same cluster. Rooting the search value in codegen's replace lowering closed 0/5, even after confirming in the IR that the root landed and moving the re-read to the point of use.
    4. Rooting a parameter when the fault is deeper. Rooting obj across the key allocation in constructor_dynamic_prototype closed 0/3.

    Grouping by frame #0 over-counts clusters

    js_object_literal_infer_computed_function_name was triaged as its own cluster. It reaches the same stale (*sym_ptr).description through infer_symbol_function_name, so #7376 closed it too. One bad field, two readers, two apparent clusters.

    The remaining 8, by stale object type

    test obj_type frame #0 ← #1
    field_lane_semantics 2 OBJECT 72 js_object_get_field_by_name ← ..._f64
    put_value_plan_cache 2 OBJECT 72 same
    stream_id_band_dynamic_dispatch 2 OBJECT 72 js_object_get_field_by_name ← fetch_subclass_handle_id
    diagchannel_3082_… 3 STRING 32 decode_string_value ← throw_tracing_channel_symbol
    gc_assign_string_source_rooting 3 STRING 56 mirror_class_object_static_write ← js_object_set_field_by_name
    gc_process_env_cache_rooting 3 STRING 40 js_array_push_f64 ← make_iter_result
    object_assign_collection 3 STRING 32 js_array_length ← js_object_assign_one
    fetch_response 5 PROMISE 72 js_promise_resolve ← js_request_array_buffer

    The three OBJECT ones are one group and are harder than what is fixed so far. The fault is at js_object_get_field_by_name + 664 — inside the function, on a 72-byte OBJECT — and rooting the caller's receiver does not close it. That points at the prototype-chain walk holding a link across an allocation, rather than a stale argument. Worth doing next because it is three catches and a hot path; worth budgeting more than the others because the ordering lens alone does not locate it.

    object_assign_collection is the case analysed earlier in this issue: js_object_assign_one is already carefully rooted (#7200), so the argument arrives stale — caller-side.

  4. proggeramlug commented on Aug 4, 2026

    @proggeramlug
    ContributorAuthor

    The 3-catch OBJECT group: exact faulting instruction, and two more failed hypotheses

    Disassembled it rather than guessing further:

    ->  0x1005fca38 <+664>: ldurb  w8, [x22, #-0x8]     ; GC header obj_type at user_ptr-8
        0x1005fca3c <+668>: cmp    w8, #0x2             ; == GC_TYPE_OBJECT
        0x1005fca40 <+672>: b.ne   <+1292>
    

    So the fault is the fast lane's GC-header type probe in js_object_get_field_by_name, dereferencing a stale receiver at user_ptr - 8. The fast lane's own comment describes it as resolving a field "with no rooting scope, no exotic-registry probes, and no key hashing" — the absence of rooting is a deliberate design property, which is what makes a stale receiver fatal here rather than merely wrong.

    Two hypotheses tested and killed

    Both measured 0/3:

    1. Stale receiver from constructor_dynamic_prototype. Its js_string_from_bytes(b"constructor") is a real collection point with obj used after it — a textbook instance of the ordering bug fixed four times elsewhere in this issue. Rooting obj across it and re-reading closed nothing.
    2. Stale receiver from the process_env_get_field probe at the top of js_object_get_field_by_name, which does an OS lookup and materializes a StringHeader. Rooting obj and key across it and re-reading closed nothing.

    Since neither in-function collection point is responsible, obj arrives stale at the call. Frame #1 differs across the three tests — js_object_get_field_by_name_f64 (the typed-this IC-miss fallback) for two of them, fetch_subclass_handle_id for the third — so the next step is the callers, not this function.

    Why this group is worth a separate budget

    The four clusters already fixed shared one shape: a collection point with a live raw pointer either side of it, locatable by reading ~30 lines around the faulting frame. This one is not that. The receiver is already dead on entry, three different callers reach it, and the consuming code is a fast path that deliberately omits rooting — so the fix is either at each caller or a decision that the fast lane must root after all, which has a measurable cost on the hottest property-read path in the runtime.

    Recording the disassembly so whoever picks this up starts from the instruction rather than re-deriving it.

  5. proggeramlug commented on Aug 4, 2026

    @proggeramlug
    ContributorAuthor

    The remaining 8 are a DIFFERENT class from the 23 fixed, and here is the evidence

    Disassembled each rather than guessing. Every one faults on an early dereference of an argument — the pointer is already dead on entry:

    test faulting instruction reads frame #1
    gc_assign_string_source_rooting ldr w8, [x0] +28 (*obj).object_type, the function's FIRST deref js_object_set_field_by_name +8708
    object_assign_collection ldurb w9, [x20,#-0x8] +468 GC header of a stale array js_object_assign_one +1456
    diagchannel_3082_… ldr w1, [x0,#0x4] +260 byte_len of a stale StringHeader throw_tracing_channel_symbol +28
    field_lane_semantics ldurb w8, [x22,#-0x8] +664 GC header, fast-lane type probe js_object_get_field_by_name_f64
    put_value_plan_cache same same same
    stream_id_band_dynamic_dispatch same same fetch_subclass_handle_id

    Why this matters for planning

    The 23 already fixed shared one shape: a collection point visible inside the faulting function, with a live raw pointer on either side of it. That is locatable by reading ~30 lines around the frame, and four such fixes landed (#7373–#7376).

    These 8 are not that. The argument is stale before the callee runs, so there is nothing to fix in the function the backtrace names. I confirmed this the expensive way — four consecutive callee-side fixes measured exactly zero:

    • no-move window in lookup_fancy_regex — 0/5
    • rooting the search-value operand in codegen's replace lowering — 0/5
    • rooting obj across the key allocation in constructor_dynamic_prototype — 0/3
    • rooting obj+key across the process_env_get_field probe — 0/3
    • no-move window around the name borrow in mirror_class_object_static_write — 0/1 (the fault is at +28, before that borrow)

    What this class actually needs

    Each one requires walking up to find where the caller lost the pointer, and the callers differ per test. Two structural notes:

    • js_object_assign_one is already carefully rooted (GC: { ...src } with an accessor source SIGSEGVs under PERRY_GC_MOVING_LOOP_POLLS=1 #7200), so its argument arriving stale means the caller above it is at fault — two levels up.
    • The js_object_get_field_by_name fast lane deliberately has no rooting scope (its comment says so). Making it root has a measurable cost on the hottest property-read path, so that is a design decision rather than a patch.

    Recommendation: treat these 8 as a separate work item from the 23. They are not more of the same, and the method that closed the first four does not apply. The disassembly above is the starting point so nobody re-derives it.

  6. 9 remaining items

  7. proggeramlug commented on Aug 11, 2026

    @proggeramlug
    ContributorAuthor

    Full sweep re-run on main @ db44b31b7 — all 523 gap tests, both arms, exactly the configuration this issue specifies. The population is essentially gone: 55 → 1, and the 44 that mattered most → 0.

    this issue now
    measured 479 523
    QUARANTINE-ONLY FAULTS 55 1
    …of which the control exited 0 (clean program, silent corruption) 44 0
    faulted in BOTH arms (pre-existing) 9 0
    no fault in either 415 522

    Method unchanged from the issue: compile once, run control (PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off) then experiment (+ PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64), and count only faults present in the quarantine arm and absent from the control.

    The one remaining catch is not the shape this issue is about

    QUARANTINE-ONLY: test_gap_backoff_options (control exit 1, quarantine exit 138)
    

    Its control exit is 1 — the program was already failing before the quarantine was armed. test_gap_backoff_options is one of the 19 pre-existing failures in the release gap sweep (output mismatch). So it is not a case of "clean output hiding a stale dereference", which is the category this issue was opened for and which stood at 44.

    The result is not vacuous

    The instrument's arming was measured separately rather than assumed, since that is this issue's own thesis — the smoke gate was pointed at a workload that structurally could not expose these. Across a 45-test subsample: 42 of 45 armed (≥1 retired page-set), 65 retired sets total, with a single-test probe showing mode=ProtectPages retired_set=#0 blocks=1 bytes_protected=104857 alongside a copying minor. And the full sweep did catch one, so the detector was live end to end.

    Why the population went away

    Nothing here targeted these 55 directly. The layer-3 campaign did: #7206 (call receiver across sibling operands), #7214 (js_closure_callN's callee/this/args), #7217 (the lazy globalThis bootstrap window), #6943 (the js_string_coerce-as-property-key family), and RS4GC operand re-reads — on #7248 I measured the previously-stale js_eq left operand now being reloaded from its slot below the allocating sibling, which is precisely this failure mode being fixed structurally rather than site-by-site.

    Closing: the measured claim in this issue no longer holds. The instrument-aiming half also moved — #7793 (merged) drives the same quarantine at DEPTH=800 over the 81-module zod corpus in gc-moving-witnesses.yml and asserts its own arming, so CI now points it at dependency-scale real code rather than one synthetic fixture. If a future sweep finds this rate climbing again, the two-arm script in this thread is the reproduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions