Repository navigation
gc: incremental mode starts a budgeted cycle and never completes it — a small compiled program still performs ZERO automatic collections (#6950 residue) #6978
Description
Split out of #6950. #6977 fixed the arena-triggered half of this; the malloc-triggered half survives, and it is why a small compiled program still performs ZERO automatic collections in the shipped (incremental) configuration.
Symptom
A compiled program in the default configuration reaches a due GC trigger, starts a budgeted cycle, and never finishes it. Measured on origin/main + #6977, release build, macOS arm64, pinned Node 26.5.0, PERRY_GC_TRACE=1 PERRY_GC_DIAG=1:
| corpus test | PERRY_GC_HEAP_LIMIT=8 (default/incremental) |
+ PERRY_GC_INCREMENTAL=0 |
|---|---|---|
test_gap_repsel_canonical_i32 |
0 cycles | 3 cycles |
test_gap_repsel_ptr_shape_locals |
0 cycles | 4 cycles |
A trigger is due in both columns — turning incremental off collects, so the arming path agrees there is work. The difference is entirely in who runs the collection.
Mechanism
gc_check_trigger's direct-collection arm (the #5476 patch) is gated on registered_root_scanners_block_budgeted_gc(). Under gc_incremental_enabled() that predicate reduces to "is any copy-only scanner registered":
if super::gc_incremental_enabled() {
return has_copy_only;
}A compiled program registers none (copy-only scanners are the FFI copy-only API plus tests — the comment in gc/roots.rs says so). So the predicate is false, the direct arm is skipped, and the trigger is handed to the budgeted stepper, which is driven only by bounded mutator assists.
For an arena-triggered cycle #6977 made those assists scale: the debt term was reading the raw GC_NEXT_TRIGGER_BYTES cell instead of effective_next_arena_trigger() and therefore reported zero. For a malloc-triggered cycle the debt term is already correct —
malloc_debt_objects: malloc_count.saturating_sub(next_malloc_trigger)— it is just small on a small program, so gc_mutator_assist_scaled_work_units() stays near its 256-unit floor and the cycle still never reaches completion. Same outcome, different cause: the cycle is started and then starved.
Why it matters
- Correctness of the memory bound. "Started but never completed" is indistinguishable from "no collector" to RSS. This is the same shape as Memory not free in benchmark #5476 and as the debt-pacing regression fix(gc): a compiled program can now actually reach an evacuating minor (#6950) #6977 fixed; it keeps reappearing because the budgeted stepper has no completion guarantee, only a budget.
- Every GC gate that runs on small programs is inert.
scripts/gc_repsel_matrix.sh'sdefault/verify_evac/gen_gc_off/wb_off/cons_scan_offarms requirecollect, and on 18 of 20 corpus files they get zero cycles in the shipped configuration. fix(gc): a compiled program can now actually reach an evacuating minor (#6950) #6977 made themovearms live by addingPERRY_GC_INCREMENTAL=0; thecollectarms deliberately were not changed, because they are supposed to describe the shipped configuration — and the shipped configuration genuinely does not collect. That honest UNVER is this issue.
Suggested direction (not prescriptive)
The budgeted stepper needs a completion guarantee, not a bigger budget. Something shaped like: if a cycle has been active across N assists while its arming trigger is still due, finish it synchronously — the graceful degradation the pacing doc already describes ("under extreme allocation pressure incremental degrades gracefully toward synchronous behavior rather than toward unbounded memory"), applied to elapsed assists rather than to measured debt alone. A starvation counter is cheap and cannot regress the case where the cycle does complete on budget.
Whatever the fix, the acceptance signal is liveness, not a green table: PERRY_GC_TRACE=1 must show cycles > 0 for test_gap_repsel_canonical_i32 under PERRY_GC_HEAP_LIMIT=8 with no other GC env var set.
Activity
proggeramlug commentedon Jul 29, 2026
Full
--arms allre-measure of the combination of today's 13 GC PRsThe last recorded matrix was taken immediately after #6977, before #6972/#6975, #6983, #6990, #6994, #7007, #6997 and #6974 landed. Each of those was verified against a different base, several within minutes of each other; nobody had measured the combination. This is that measurement.
Headline: the evacuating arms went from 14 red corpus files to 2, and there is not a single previously-green-now-red cell. The 180 UNVERIFIED cells this issue tracks are unchanged at 180 — none of the seven PRs touches the incremental stepper, and the measurement confirms it did not move by accident.
Setup
393a36aa4(currentorigin/main; #7009/#7008 are its ancestors, PR numbers are not merge-ordered)--release,-p perry -p perry-runtime -p perry-stdlib -p perry-runtime-static -p perry-stdlib-static(9m33s).node-version)scripts/gc_repsel_matrix.sh --arms all --pressure 8 --jobs 6 --json(--pressure 8= the script default and what #6977/#6981 used)Freshness was verified functionally, not by mtime (
CLAUDE.mdstale-artifact trap).test_gap_gc_ta_ctor_source_rootingunder the evacuating base is byte-exact with 6 558 objects copied — that cell failed before #6990 (literal: 0 undefined undefined), so a stale.acould not produce this result. The runtime archive also exportsjs_gc_temp_root_{push,set,get,truncate}.Counts
Delta vs 45 / 125 / 180 — and an honest note on the denominator
45+125+180 = 350, which matches no--arms alldenominator in this window (20 corpus rows → 400 cells, 21 → 420, 22 → 440, 23 → 460 today). I cannot reconstruct that triple's exact population, so I am not going to publish a fabricated delta. What it fits cleanly is the 10movearms plus the collect-arm inert block. Restricted to the 21 corpus rows that existed at #6977, so the comparison is apples-to-apples:movearms (10 × 21) — PASSmovearms (10 × 21) — FAILcollectarms (9 × 21) — UNVERThe 180 landing exactly on 180 is the load-bearing part for this issue: the collect arms are still inert on 21 of 23 corpus rows, and that is #6978 verbatim.
Per-cell: what changed on the evacuating arm
Diffed against #6981's recorded
evac_minorcolumn. 12 of the 14 previously-red cells are now green; nothing regressed.evac_minorrepsel_canonical_i32specabi_polymorphic_coexistspecabi_reassignta_param_numeric_readtypedarray_param_readrepsel_ptr_shape_localsrepsel_ptr_shape_barriersrepsel_p4a_inline_tiersrepsel_p4a_logical_numericrepsel_p4b_field_store_elisionrepsel_proven_this_frozenrepsel_gc_stressrepsel_p4a3_numarray_barriersrepsel_p4a3_ptr_numarrayint_valued_ta_locals,specabi_recursion_escape,specabi_view_detach,repsel_canonical_str_locals,repsel_p4a_holes_axisrepsel_p4a3_numarray_growthPreviously-green-now-red: none, and this is airtight
The red set today is exactly 20 cells: 2 corpus files × the 10
movearms. I rebuilt the #6977 baseline (e70ffe29d) and re-ran those same 10 arms on those rows through the same harness. Reverse-freshness on that build was proven first — #6981's own reproducer still printsafter: undefined:All 20 of today's red cells were already red at the baseline, on the same arms. Since a regression requires a cell that was green and is now red, and every red cell today is accounted for as already-red, the combination introduced zero regressions. That conclusion does not depend on the unmeasured cells, because a green cell cannot be a regression.
One retraction. I initially flagged
numarray_barriersas having degraded from mismatch to crash, on the strength of #6981's table (mismatch (15 297)). My own re-measure ate70ffe29dgives exit=139 SIGSEGV, identical to today. The failure mode did not change; #6981's recorded entry for that one row disagrees with a direct re-measure at that commit. Withdrawing the concern.Liveness — every arm, with evidence
Largest
copied_objects:scalar_replaced_locals1 501 975,gc_stress1 026 015. The evacuating arms are genuinely live — no green cell in this run comes from an inert arm.Decomposition of the 240 PASS, so the number is not over-read:
movearms withcopied_objects > 0— evacuation genuinely verified.collectarms, on the only 2 rows that collect (gc_stress,scalar_replaced_locals).shipped_default, whose requirement isnone. It collected 0/23. Those 23 cells assert byte-exactness only and carry no GC evidence whatsoever — they should not be read as green GC results.Still-UNVERIFIED: 199, and why
collectarms are inert on 21 of 23 rows: the shipped incremental configuration performs zero collections on a small program. Not fixable by any rooting PR; it needs the budgeted stepper's completion guarantee.test_gap_repsel_p4a3_numarray_growth, which reports 0 cycles / 0 moved on every one of the 20 arms, including the evacuating base at--pressure 8. This is a corpus gap rather than a collector one: that file's churn budget never reaches a trigger, so itsmovecells assert nothing. Its cells have been honestly UNVER since the matrix landed and will stay that way until its allocation volume is re-tuned — same remedy the header prescribes forgc_stress.The 2 remaining red cells → #6981
Both map to #6981, which stays open. Both are deterministic (8/8 trials, identical output) — not flakes, not harness-induced.
p4a3_numarray_barriersp4a3_ptr_numarrayhistogram()before its 4th line158vs159, total4999vs5000The #6981 discriminator reproduces exactly: both relocate thousands of objects and pass with the conservative stack scan on; they fail only under precise roots. Both are the identical source shape —
counts[v] = (counts[v] || 0) + 1over anew Array(size)local, with the collection driven by a neighbouringdata.push(seed)loop.New diagnostic worth recording:
rep_ptr_numarray_off(PERRY_PTR_NUMARRAY_LOCALS=0) does not rescue either test — it is red on all 10movearms including the representation's own off-switch. So the stale base pointer is not thePtr<NumArray>promotion itself; #6981's remaining surface is the generic array element read-modify-write path. That should narrow the fix hunt considerably, and it distinguishes this from #6976 / #6984, which are off-switch defects.The single XFAIL (
ptr_shape_locals × rep_ptr_shape_off) is #6976, still correctly triaged.Unrelated defect found: a duplicated corpus row
test-parity/gc_repsel_corpus.txtregisterstest_gap_repsel_proven_this_frozentwice — 23 rows, 22 unique. #6977 added it under "Phase 5a" near the top and again at the bottom. It is compiled and run twice per arm, inflating every count by 20 cells and double-weighting that one test in the liveness denominators (the22/23and2/23figures above). Harmless to the conclusions, but it should be de-duplicated so the totals mean what they say.What I could not measure
--arms allat the fix(gc): a compiled program can now actually reach an evacuating minor (#6950) #6977 baseline (disk headroom on the host); I ran the targeted 10-move-arm ×p4a3subset. As argued above this is sufficient for the regression question, but it means I have no baseline figures for the cells that are green today.numarray_growthasserts nothing on any arm, so all 20 representations remain unverified against that file's shape specifically.fix(gc): charge the host-safepoint allowance per cycle, and keep it w…
proggeramlug commentedon Jul 29, 2026
Re-measured on current
main(#7019 merged): this no longer reproducesorigin/main@4340bffc9("gc: default-on evacuating young-gen scavenge (moving minor at precise safepoints)", #7019). Release build ofperry+perry-runtime+perry-stdlib+ both-staticwrappers, rustc 1.97.1, macOS arm64, oracle node 26.5.0 pinned. The compiler was built on one host and functionally re-verified on the host it ran on (byte-exact against the oracle before any measurement) — not trusted by mtime.1. #6978's own acceptance line now passes
PERRY_GC_HEAP_LIMIT=8,PERRY_GC_TRACE=1, no other GC env var — the configuration this issue specifies:main+ PERRY_GC_INCREMENTAL=0test_gap_repsel_canonical_i32test_gap_repsel_ptr_shape_localsA/B'd through #7019's own kill switch, same source and same compiler,
PERRY_GC_MOVING_LOOP_POLLS=0set at both compile and run time (the codegen and runtime halves must agree):So the symptom is gone, and #7019 is what removed it.
Mechanism (source-confirmed, and it is not a fix so much as a re-route).
gc_moving_loop_polls_enabled()is now default-on, sogc_check_triggerenters its direct arm and anArenaBytes/MallocCounttrigger is taken away from the budgeted stepper before it can arm a cycle. There is no longer a budgeted cycle for nursery pressure to park in. The starvation path #6978 describes still exists in the code; nursery pressure simply no longer travels down it.Both collections land mid-program — on an interleaved stdout/stderr stream the first
[gc] cyclemarker precedes the last program output line. That makes them stronger evidence than the teardown-boundary collections #7017 was filed against.2. What that means for #7015
Its stated symptom no longer reproduces, so as a fix for this issue it looks obsolete and I'd close it — with two things kept:
ArenaBytes, not malloc (these programs never callgc_malloc;malloc_count=0against anext_malloc=100000trigger, so that path is never armed, not merely small), and the binding constraint was the number ofgc_check_triggeropportunities — two in the whole process — not the assist budget's 256-unit floor.gc_safepoint_moving_minorstill early-returns ongc_budgeted_cycle_active(), and so doesgc_check_trigger's defer arm. A budgeted cycle armed by a non-nursery trigger would still park and would still disable gc: default-on evacuating young-gen scavenge (moving minor at precise safepoints) #7019's moving minor for the rest of the process. I did not construct that case. What I can say is that it does not occur on this corpus: in the shipped configurationtest_gap_repsel_gc_stressperforms 260 cycles with 251[gc-copy-minor] ranand the last copy at line 1033 of 1038 — the moving minor keeps running to the end of the process. If fix(gc): a budgeted incremental cycle that nothing drives now completes (#6978) #7015 is closed, that guarantee is worth re-filing on its own merits rather than losing.3. Matrix re-run — 440 cells / 22 rows (
--arms all --pressure 8)4ad27a15b(#7015's pre-fix arm)main@4340bffc9The 190 UNVERIFIED moved wholesale — 99 of them left the column — but in both directions. The
collectarms went from ~2/22 live to 13/22. I do not have the pre-#7019 per-cell JSON (only the aggregates in #7015's body), so I am not publishing a cell-by-cell transition table; the arm/row accounting below is what the +26 decomposes into.4. ★ 26 newly-red cells. They are real, and both new rows are caused by #7019
test_gap_repsel_gc_stresstest_gap_repsel_scalar_replaced_localsp4a3_numarray_barriers+p4a3_ptr_numarraygc_stresswas the corpus's green, live member (gc_repsel_triage.txtrecords itscons_scan_offcells as hard gates since #6951), so that is a PASS → FAIL regression, not an inert cell going red. It dies immediately after a successful evacuating minor (copied_objects=101215 promoted_objects=44933,fallback=none).scalar_replaced_localsis not #7018 — it faults with noPERRY_GC_TRACEand noPERRY_GC_DIAGat all.Both failing sets are dominated by the same discriminator as #6981: the arms carrying
PERRY_CONSERVATIVE_STACK_SCAN=off.5. Per-arm liveness, with
copied_objectsseparated from the C4b counterThe harness's
movedfigure sums two different collectors —moved_objects=(the mark-sweep C4b evacuation policy) and[gc-copy-minor] ran copied_objects=(#7019's scavenge). Separated, over all 22 rows:[gc-copy-minor] randefaultverify_evacgen_gc_offgen_off_verifycons_scan_offcons_scan_off_forceevac_minorforce_evacforce_verifyshipped_default(no env)Two things fall out, both filed:
defaultarm runs zero copying minors on all 22 rows —--pressure 8collapsesgc_moving_defer_hard_cap_dyn_bytes()and the arena-trigger ceiling to the samebudget_scaled(_, 1, 4, 2 MB)value, soarena_total_bytes() < hard_capis false exactly when the trigger fires and the defer arm is skipped ([gc-copy-minor] eligible=false fallback=conservative_stack). Sweeping the knob oncanonical_i32/ptr_shape_locals: limits 2 and 8 give cycles but zero copy-minors, limits 16/32/64/128 give zero cycles at all. The band is empty. → gc-matrix: --pressure disables the very path #7019 added — the 'default' arm runs ZERO copying minors on all 22 corpus rows #7024movedcounter lets arequires=movearm be green with zero scavenged objects. → gc-matrix: the 'moved' liveness counter sums the C4b mark-sweep evacuation with the copying minor, so a move-arm can be green with zero scavenged objects #7025The un-pressured shipped default does work as advertised: 251 copy runs,
fallback=none, 13 501 581 objects copied ongc_stress.test_gap_repsel_p4a3_numarray_growthis 0 cycles / 0 copies in every arm includingevac_minor, which is #7016 exactly — 19 of the 91 remaining UNVER cells are that one row.6. Things I got wrong along the way, and corrected
Recorded because they are cheap to repeat:
$E, soenv $E binaryset a single garbage variable and the GC env vars were never applied — the control was silently measuring the shipped default. Re-run underbash, the failures reproduce.scalar_replaced_locals. Retracted; the numbers in §4 are from re-measurement with explicit exit codes.7. Not measured
gc_safepoint_moving_minorsuppression directly — only the observation that it does not occur on this corpus.proggeramlug commentedon Jul 30, 2026
Status update: the symptom is gone, but this issue's stated mechanism is wrong and one hazard survives
PR #7015 (the fix) has been closed as obsolete — see its closing comment for the measurements. Summary here so this issue is self-contained.
The symptom no longer reproduces
Re-measured on
4340bffc9(post-#7019, default-on evacuating young-gen scavenge), on this issue's own acceptance rows:test_gap_repsel_canonical_i32test_gap_repsel_ptr_shape_localsPERRY_GC_MOVING_LOOP_POLLS=0restores 0/0, attributing the change to #7019. It removed the starvation by re-routing nursery triggers away from the budgeted stepper before a cycle can arm — not by fixing the completion path. Both collections now land mid-program, not at teardown (cf. #7017).★ This issue's mechanism is wrong in both particulars
Correcting the text above, because it will otherwise keep misdirecting readers:
ArenaBytes. The two programs named in this issue never callgc_mallocat all —malloc_count=0against anext_malloc=100000trigger. The malloc path was not "small"; it was never armed.gc_check_triggerruns exactly twice in the entire process. The second call arms the cycle, pays one assist, and no allocation-point opportunity ever comes again (3 arena blocks total). The 7 steps that follow are host safepoints at the fixed 2048-unit slice; they reachRootScanand the process exits. A fatter per-step budget could not have fixed this — completion needs a bounded number of calls, not more units.What this issue should stay open for
gc_safepoint_moving_minorearly-returns ongc_budgeted_cycle_active(), so a parked budgeted cycle disables the moving path for the rest of the process. #7019 removed the nursery-armed route into that state, but a non-nursery-armed budgeted cycle could still reach it.Not observed on the repsel corpus (
gc_stress: 251 copy runs, last at line 1033/1038), and not constructed — so this is a live hazard with no reproducer, which is the hardest kind to keep honest.Per the GC knob kill-policy now in
CLAUDE.md(every GC knob either has a required CI arm exercising its off-state, or it is deleted after one release of soak), the resolution is one of:A reachable state with no test covering it is exactly what that policy is meant to eliminate.
proggeramlug commentedon Aug 2, 2026
Confirmed still open on
main=d4ac65f21. Re-verified in the sweep #7262.This issue's own stated acceptance signal is
cycles > 0fortest_gap_repsel_canonical_i32underPERRY_GC_HEAP_LIMIT=8with no other GC env var set. Measured,PERRY_GC_TRACE=1, compiled with no compile-time GC env:PERRY_GC_HEAP_LIMIT=8only+ PERRY_GC_INCREMENTAL=0Unchanged from the table here. A trigger is due in both columns — turning incremental off collects — so the arming path still agrees there is work and the budgeted stepper still never finishes the cycle it starts.
Worth connecting to #7255, which measured the downstream consequence corpus-wide rather than on two files: the
requires=scavengearms (default,verify_evac,cons_scan_off,cons_scan_off_force) now reportcopy-minor 0/49. This issue's point 2 — "every GC gate that runs on small programs is inert", and the honestUNVERis this issue — is now the state of four of the six PR-gating arms, not just of the--arms alltable.The suggested direction (a completion guarantee via a starvation counter rather than a bigger budget) still looks right and is untouched in the tree.
proggeramlug commentedon Aug 9, 2026
Closing — measured on v0.5.1414 and the observable does not reproduce.
What I ran
A compiled program allocating
class N { v: number; s: string }in a loop,PERRY_NO_AUTO_OPTIMIZE=1, no zeal, no loop polls — i.e. the default configuration this issue is about.[gc]cycles, peak RSS 201 MBAnd all three incremental arms behave identically at 4M: unset → 12,
PERRY_GC_INCREMENTAL=1→ 12,PERRY_GC_INCREMENTAL=0→ 12.The 400k case is correct behaviour, not the bug. It frees 36.7 MB against a 64 MB initial arena trigger threshold, so it never reaches a trigger. "A small program performs zero automatic collections" is true of any program that stays under the threshold, by design.
The control, because a zero is worthless without one
0 diag linescould equally mean the diag knob prints nothing. It does not: the same binary with an explicitgc()prints 3 lines ([gc] blocks: …,[gc-old-free],[gc-tenuring]), and withPERRY_GC_MOVING_LOOP_POLLS=1+PERRY_GC_ZEAL=1prints 22. The instrument works; the small program simply does not collect because it should not.Why this issue was hard to settle by reading
It is the clearest instance I have seen of a claim that silently stopped being true, and the history is worth keeping:
5a06970b6("disable evacuating minor by default pending GC: evacuating minor drops an old-to-young field[1] edge, crashing with 'value is not a function' #7154") plausibly re-broke it;Four changes of state, none of which updated the issue. Any percentage or behaviour claim attached to this ticket that predates today should be treated as unverified.
If automatic collection genuinely fails on a workload that does exceed the threshold, please reopen with that workload and a
PERRY_GC_DIAG=1capture — the diag-line count against a known-good control is the whole test.