Repository navigation
Old-gen growth from large-object births never arms a full collection: RSS climbs unbounded on repeated wide-object parse #10123
Description
Old-generation growth from large-object births does not arm a full collection
A workload that repeatedly allocates and drops a large pointer-bearing object grows old-gen without bound. Every minor promotes essentially the whole nursery, copies nothing and frees nothing, and the full collection that would reclaim it never arrives.
records_array_wide_1m:parse — a flat object of 50 000 integer keys, parsed in a loop:
old_in_use=6578096 survival_permille=997 copied_objects=0 freed_bytes=0
old_in_use=28272432 survival_permille=997 copied_objects=0 freed_bytes=0
old_in_use=59873112 survival_permille=997 copied_objects=0 freed_bytes=0
old_in_use=112274032 survival_permille=997 copied_objects=0 freed_bytes=0
Peak RSS 228 MiB against Node's 121 and Bun's 95.
The same signature on records_array_16k:scan before #10098's lazy-array work, with the trigger state visible:
old_in_use=48385432 old_threshold=50331648 old_pending=false
[gc-arena-rebaseline] step=134217728 -> 268435456 -> 536870912 -> 1073741824
An old-gen threshold of 48 MB exists and old_in_use reaches 48.3 MB with old_pending=false; meanwhile the arena trigger rebaselines itself upward exponentially. So the arm that would notice old-gen filling keeps stepping out of the way of the arm that is filling it.
Why this is one cause, not two
Each object is individually reasonable. arena_alloc_gc puts a pointer-bearing object over LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB) directly into non-moving old-gen, and the reasoning in its own comment is sound: for a pointer-bearing object the cost of tenuring "is not its own bytes, it is every object it can reach, held live through the remembered set by a container nothing refers to any more".
That reasoning assumes a full collection eventually runs. When it does not, the policy inverts: the objects chosen for old-gen because they are expensive to hold are exactly the ones nothing reclaims.
Two of these are now fixed by moving the allocation rather than the policy — #10098 made the lazy JSON array cluster nursery-resident and took records_array_16k:scan from 205 MiB to 51 MiB and records_array_1m:scan from 159 to 75, both now below Node and Bun. That works because a lazy cluster's cache is small enough to sit under the line. A 50 000-key object's field storage is ~400 KB and cannot.
What to look at
- Whether old-gen growth from large-object births feeds the full-collection trigger at all, or only growth observed through promotion.
- Whether the arena trigger's exponential rebaseline should be bounded while
old_in_useis climbing — the two arms currently move in opposite directions. old_pending=falseatold_in_use≈old_thresholdis the specific observation to explain first.
Scope note
This is GC policy, which CLAUDE.md routes separately and warns about, so it wants its own change with the ratchet corpus run before and after rather than riding along with JSON work. Filing it with the evidence rather than attempting it inside a JSON performance pass.
Rows currently affected: wide_1m:parse RSS 228 vs 95 MiB (2.4x), and the residual records_array_8m:scan 187 vs 134 and records_array_20m:scan 256 vs 226, whose clusters are genuinely over the pointer-bearing line.
Activity
proggeramlug commentedon Sep 12, 2026
Root cause found: the RETAINING veto is fed by the survival its own pinning creates
A full collection reclaims all of it.
PERRY_GEN_GC=0onwide_1m:parse, same binary:So nothing is unreclaimable and no allocation needs to move. The generational regime simply never runs the collection that would free it. At 55 MiB Perry would be below Node (121) and Bun (95) on this row.
The veto
gc::policy::old_reclaim_pressure_due:GC_MAJOR_PACING_RETAININGis armed from measured young-survival, which on this workload readssurvival_permille=997on every minor. #7937's reasoning for the exemption is that "a heap whose young generation is not dying is retaining live data, and a full mark-sweep cannot lower the number being watched."That reasoning inverts here. The young generation is not dying because large old-gen containers pin it through the remembered set — every parse's 400 KB field storage is born tenured, and a minor never sweeps old-gen, so it holds its 50 000 key strings live. High survival is the symptom of the condition a full collection would clear, and the veto reads it as proof that one would be futile. The measurement above is the counter-example: the full sweep lowers exactly the number being watched, by 4x.
The proportional arm does still bound the exposure, but the RETAINING multiplier widens the band to 128 MB, and
old_in_usereaches ~112 MB over 64 iterations — so the bound is real but sits above where this workload ends.The width cliff confirms the container
RSS growth from 8 to 64 iterations, by object width — the step is exactly at
LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES(128 KB), i.e. 16 384 keys at 8 bytes each:Under the line the storage is nursery-resident and a minor reclaims it. Over the line it is born tenured and only a full collection can, which is the one that never comes.
Suggested shape
Young-survival cannot distinguish "live data" from "dead data pinned by a born-tenured container", so it should not be the sole veto on the arm that would tell them apart. Two options, either of which is testable against this fixture:
credit_promoted_bytes_to_old_baselinecredits the baseline only from promotion, so a born-tenured object movesold_in_usewithout movingbaseline— the two quantities the arm compares already disagree in exactly the informative way. Tracking born-tenured bytes since the last full collection and letting them bypass the RETAINING veto is a small change with a direct signal.Either needs the gc-ratchet corpus run before and after, which is why this is filed rather than attempted inside a JSON pass.
Note this is the same root cause as the residual
records_array_8m:scan(187 vs 134 MiB) andrecords_array_20m:scan(256 vs 226), whose clusters are also genuinely over the pointer-bearing line. #10098 fixed the two rows whose clusters fit under it by making them nursery-resident; that approach cannot reach these.proggeramlug commentedon Sep 12, 2026
Attempted the trigger fix. It does not work, and the measurements say why: the trigger is only half the chain.
Three attempts, each measured on
wide_1m:parse(64 iterations, Node 121 MiB, Bun 96 MiB):RETAININGvetoMaking the collection fire is achievable — a born-tenured byte counter reset at each reclaim, exempted from the
RETAININGveto, produces collections where main has none. It does not lower peak RSS, and two fulls make it worse than zero.Why: the arena never returns capacity
Right-sizing is armed (
disarmed=false) and never starts an episode. So the chain is:totalreaches 141 MB, rebaselining its own trigger 536 MB → 1 GB along the way.That is why
PERRY_GEN_GC=0holds 55 MiB flat: with full sweeps throughout, old-gen never accumulates, so the arena never grows in the first place. It is not that fulls shrink the heap — it is that they stop it expanding.Two late fulls are therefore strictly worse than none: the arena has already grown, and the collections add their own cost on top.
What this means for the fix
The trigger work above is necessary but not sufficient. Either:
threshold/2, and that trades CPU against a benchmark this issue has not measured. The gc-ratchet corpus is the right instrument.arena-right-sizeactually run after a productive full.episodes=0withdisarmed=falseis the specific thing to explain:arena_live=81 MBagainstarena_capacity=90 MBis only 10% slack, so a ratio-based gate may simply never see enough headroom — while the 141 MB the arena reached, versus the 55 MB the workload needs, is where the memory actually is.Option 2 looks like the higher-value half, and it is independent of the trigger.
Also worth noting
old_reclaim_pressure_duehas three consumers, one of themcopied_minor_promotion_handoff_pressure_due. I split the births arm out of theshared predicate on the theory that scheduling a full before a promotion was the
#7592 futile-handoff shape. It made no difference — 284 MiB either way — so the
handoff is not implicated here, and a future change need not carry that
complication.
Everything above is reverted; nothing from this investigation is on a branch.
proggeramlug commentedon Sep 12, 2026
The complete chain: two independent heuristics, both fed by the pathology they would cure
Same fixture, same binary, only
PERRY_GEN_GCdiffers:arena_livearena_capacityThe workload's true live set is 6.4 MB. Everything above that in the generational column is dead storage that no collection has adjudicated: born tenured, over
LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, and a minor never sweeps old-gen.That inflated number is then consumed by two separate gates, and both read it as "this heap is busy":
old_reclaim_pressure_due—GC_MAJOR_PACING_RETAININGis armed from young-survival, which reads 997permille because the born-tenured containers pin the young graph through the remembered set. The veto blocks the collection that would disprove it.arena_right_size— opens an episode only at or belowARENA_RIGHT_SIZE_TRIGGER_PCT(50%) live/capacity. It sees 90% and declines, soepisodes=0 starts=0 released_capacity_bytes=0withdisarmed=false. Capacity is never returned.So the chain is: born-tenured storage accumulates ->
arena_livereads 25x the true live set -> the reclaim trigger declines AND right-sizing declines -> the arena grows to hold the garbage and never gives it back -> peak RSS is 4x the true requirement.PERRY_GEN_GC=0holds 55 MiB not because full sweeps shrink the heap, but because the arena never grows in the first place: live stays at 6.4 MB, so nothing ever asks for 175 MB of capacity.Why the trigger fix alone cannot work
This is the measured reason attempts 2 and 3 made things worse rather than better. By the time a births-driven trigger fires, the arena has already expanded, and right-sizing still will not open because
arena_liveis still inflated -- the two fulls left it at 81 MB against a 6.4 MB true live set. Two late fulls are strictly worse than none: no capacity comes back, and the collections cost their own time.Where a fix has to start
The shared defect is that both gates consult a liveness number that includes old-gen nothing has visited since it was born. Options, in the order I would try them:
arena/allocators.rscosts one add). A heap whose capacity is dominated by un-adjudicated births should be able to open an episode regardless of apparent utilization, because apparent utilization is precisely what cannot be trusted there.threshold/2. That trades CPU, and the gc-ratchet corpus is the instrument.Option 1 looks independent of the trigger work and addresses the half that actually holds the memory.
Dead ends, so nobody re-derives them
RETAININGveto: produces collections (0 -> 1), no RSS change.old_reclaim_pressure_due: 2 collections, RSS 222 -> 284 MiB.copied_minor_promotion_handoff_pressure_duekeeps the original: identical 284 MiB, so the handoff is not implicated and a future change need not carry that complication.All reverted; nothing is on a branch.
proggeramlug commentedon Sep 12, 2026
Root cause found: promotion is structurally invisible to old-gen pressure
Correcting my three earlier comments on this issue. I framed this as a trigger-tuning
problem and measured three dead ends against that framing. It is not a tuning problem,
and that is why none of them moved.
The mechanism
credit_promoted_bytes_to_old_baseline(gc/policy.rs) adds a minor's promoted bytesstraight onto the old-reclaim baseline:
So
old_in_use - old_baselineis invariant under promotion, by construction. Promotionraises both sides in lockstep. Only allocation born directly into old-gen can ever create
old-gen pressure.
That is deliberate and I am not proposing to change it — #7902 and #7965 both settled it,
and the argument in the code is right: the baseline is the base of a growth measurement,
not a liveness claim, and withholding the credit pins that base at 0 on exactly the
workloads that reach it.
But it leaves one question unanswerable: how much of old-gen arrived because a minor
could not tell garbage from survivors? On
wide_1m:parseevery minor promotes at999 permille. Nothing dies. The entire nursery is laundered into old-gen, where no
minor will ever look at it again — and the only thing scheduling fulls was incidental
born-tenured allocation elsewhere in the parser.
How that was proved
JSON.parsere-mints every key of a wide object as a born-tenured string on every call:PARSE_KEY_CACHEholds 4096 entries, a 50k-field object overflows it on the first parse,and the parse-boundary clear then wipes it before the second parse can hit it. So 50,000
js_string_from_bytes_longlivedallocations per iteration.Fixing that — raising the bound so the schema stays cached — makes the row much worse:
Removing the garbage removed the only thing that was scheduling collections. That is the
cleanest possible demonstration of the mechanism, and it is a trap for anyone who fixes
the parser side first.
The fix
Record a second, independent signal alongside the baseline credit rather than correcting
it. Bytes promoted by a minor at >= 950 permille survival accumulate in
GC_OLD_GARBAGE_SUSPECT_BYTES; 16 MB of it makes an old reclaim due on its own; a reclaimresets it. The baseline is untouched, so #7902 and #7965 both still hold.
Split off
old_reclaim_pressure_dueon purpose rather than folded into it — that predicateis also what
copied_minor_promotion_handoff_pressure_dueasks, and an arm there schedulesa full immediately before a promotion, over survivors it cannot reclaim, which is the
futile-handoff shape #7592 removed. (That was one of my earlier dead ends.)
Floor is the measured knee of a 2/4/8/16/32/64/128 MB sweep at 256 iterations:
wide_1m:parsepeak RSS 387 MiB -> 243 MiB (-37%) at baseline CPU. With the pacing armin place the key-cache fix is measurably redundant (243 MiB either way), so it is not
included.
Branch
gc/old-born-bytes-arm-full, commit14ceaef3d6. Not opened as a PR — this is aGC pacing change and needs the gc-ratchet corpus run before/after by someone with that setup.
What still stands between this row and parity
Not pacing. RSS floors at 243 MiB however many fulls run — because freed pages are never
returned to the OS:
The process holds 243 MiB while its arena is 67 MB and its live set is 56.7 MB. Block
release rejects 40 of 41 blocks at
snapshot.data == 0(arena/reset.rs:700). Targets areNode 121 MiB / Bun 96 MiB, and 56.7 MB of live data is comfortably inside both — so this is
the whole remaining gap on the row, and it is independent of everything above.
Dead ends, so nobody re-derives them
RETAININGveto — 1 full, 222 MiB, unchanged.futile-handoff shape).
never implicated.
because it buys RSS with extra whole-heap marks. The suspect-promotion arm gets most of
the same RSS for free. Superseded.
Revert "perf(gc): schedule old reclaims on suspect promotion (#10123)"
proggeramlug commentedon Sep 12, 2026
Correction, and the actual root cause
First, a correction to my previous comment. I said the suspect-promotion arm delivered
"387 MiB -> 243 MiB at baseline CPU". That CPU claim was measured on
wide_1m:parseonly.Measured across the rest of the JSON matrix against clean main, it is not baseline CPU:
records_array_1m:scanrecords_array_8m:scanrecords_array_20m:parseWorse, the regression does not respond to the floor — swept at 16/64/256 MB,
records_array_1m:scanstays at 1.26–1.30s even at 256 MB where the arm should be inert. Sothe cost is the predicate, not the extra collections:
old_reclaim_pressure_or_suspect_duesits in
gc_budgeted_due_trigger, which runs at every safepoint, and the arm adds athread-local read on the common path. Reverted in
9e1c80e060. The diagnosis in thatcommit stands (promotion credits the baseline, so only born-tenured allocation can create
old-gen pressure); the remedy does not.
The real cause: wide objects are born tenured, and a minor never sweeps old-gen
arena/allocators.rsalready states the failure mode exactly:A wide JSON document's shape-keys array is that container. Above 16,384 fields its keys
array crosses
LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES(128 KB), is born tenured, andthen holds its whole key set live after every instance has died.
Confirmed to the byte —
PERRY_GC_CENSUS, 64 parses, fixtures straddling 131,072 bytes:The census names it directly:
shape_keys_arrays {bytes: 6000240, count: 15},slot_tags {string: 750000}— 15 arrays × 50,000 keys = 750,000 live strings, on a documentwhose live set is ~0. Same binary under
PERRY_GEN_GC=0: live 336 bytes, 50 MiB RSS.Ruled out by direct experiment (each rebuilt and re-measured, all zero effect):
transition-cache size (raised 16,384 -> 65,536 on both the runtime constant and the mask
codegen bakes in
proxy_reflect.rs); parse-key identity (PARSE_KEY_CACHEbound raised4,096 -> 65,536); the shape table (
table_len=12< 15 live arrays); registered mutable rootscanners (all report
slots=0 ptr_roots=0).What a fix is worth, and why I am not proposing one yet
Raising the threshold 128 KB -> 512 KB (still inside the copier's 1 MiB
MAX_YOUNG_MOVE_BYTESand nursery-block ceilings) on wide_1m:parse:
0.62s / 284 MiB -> 0.15s / 40 MiB — −76% CPU and −86% RSS, better on both axes, against
Node 121 MiB and Bun 96 MiB.
perry-runtimeis green (3628 passed) after retargeting twotests that hardcoded a field count and asserted old-gen residency — they now derive their
width from the constant, so the old-gen path stays covered at any threshold.
But globally it is not shippable: the same change moves ordinary 128 KB–512 KB data arrays
into the nursery, and on rows that do not need it that costs RSS —
records_array_8m:scan444M -> 710M (+60%),records_array_20m:parse267M -> 314M (+18%)(CPU on those two improves, −29% and −44%).
So the widening has to be scoped to wide-object JSON construction, not global. I built
half of that: a
WideShapeKeysBirthScopearoundallocate_parse_shape_keys_array, gated inthe cold large-object branch of
arena_alloc_gcso the hot path is untouched. It firescorrectly (
born young past threshold: 400016 bytesevery parse) but only getswide_1m to 176 MiB, because the keys array is half the story — a 50,000-field object's own
property storage is also 400 KB and also born tenured. The scope needs to cover the
object-storage allocation on the JSON construction path too;
records_array_*is unaffectedby that because its large allocation is array element storage, not wide-object storage.
That is the next step, and it wants the quiet-mini matrix plus the gc-ratchet corpus rather
than more measurement on a box at load 42.
proggeramlug commentedon Sep 12, 2026
Fixed — PR #10145
The scoped fix is finished and open as draft #10145 (gated on the gc-ratchet corpus, since it changes birth generation).
The half I built earlier didn't work because the scope was open in the wrong place. A mask probe at each large allocation showed the keys array was admitted (
mask=2) but the object's property storage sawmask=0. That storage is minted inobject/json_construction.rsviajs_object_alloc_class_inline_keys_stamped, not insidejs_json_parse_result. I found the site by tagging all nineGC_TYPE_OBJECTallocation sites inobject/alloc.rs. Scoping it there completes the fix.Measured on current
mainagainst a same-tree baseline (the commit reverted and rebuilt):wide_1m:parserecords_array_8m:scanrecords_array_20m:parserecords_array_1m:scanrecords_array_16k:scanOne more correction to my earlier comments. The "main" figures I quoted before (283 MiB for
wide_1m, 444 MiB for8m:scan) came from a frozen binary whose runtime turned out to be a different source state. The compiler refused to cross-link it (library source 111389820894vsa511bd884a7f). The numbers above use a baseline built from the same tree as the fix. That also shrinks the global-widening regression I reported on8m:scanfrom "+60%" to 643 → 710 MiB. It's still a regression, and it's still why this is scoped.perf(json): allocate large record arrays once, at their estimated size (
fix(codegen): drop the trip-count obligation for a non-counter index
perf(gc): keep wide JSON document storage in the nursery (#10123)
perf(json): allocate large record arrays once, at their estimated size (
proggeramlug commentedon Sep 13, 2026
Closed by #10171 and #10145 ("Closes #10123"), landed via merge train #10188 (main
5cec2fbbc9).