Skip to content

Old-gen growth from large-object births never arms a full collection: RSS climbs unbounded on repeated wide-object parse #10123

Closed

Description

@proggeramlug

Old-generation growth from large-object births does not arm a full collection

A workload that repeatedly allocates and drops a large pointer-bearing object grows old-gen without bound. Every minor promotes essentially the whole nursery, copies nothing and frees nothing, and the full collection that would reclaim it never arrives.

records_array_wide_1m:parse — a flat object of 50 000 integer keys, parsed in a loop:

old_in_use=6578096    survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=28272432   survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=59873112   survival_permille=997  copied_objects=0  freed_bytes=0
old_in_use=112274032  survival_permille=997  copied_objects=0  freed_bytes=0

Peak RSS 228 MiB against Node's 121 and Bun's 95.

The same signature on records_array_16k:scan before #10098's lazy-array work, with the trigger state visible:

old_in_use=48385432  old_threshold=50331648  old_pending=false
[gc-arena-rebaseline] step=134217728 -> 268435456 -> 536870912 -> 1073741824

An old-gen threshold of 48 MB exists and old_in_use reaches 48.3 MB with old_pending=false; meanwhile the arena trigger rebaselines itself upward exponentially. So the arm that would notice old-gen filling keeps stepping out of the way of the arm that is filling it.

Why this is one cause, not two

Each object is individually reasonable. arena_alloc_gc puts a pointer-bearing object over LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB) directly into non-moving old-gen, and the reasoning in its own comment is sound: for a pointer-bearing object the cost of tenuring "is not its own bytes, it is every object it can reach, held live through the remembered set by a container nothing refers to any more".

That reasoning assumes a full collection eventually runs. When it does not, the policy inverts: the objects chosen for old-gen because they are expensive to hold are exactly the ones nothing reclaims.

Two of these are now fixed by moving the allocation rather than the policy — #10098 made the lazy JSON array cluster nursery-resident and took records_array_16k:scan from 205 MiB to 51 MiB and records_array_1m:scan from 159 to 75, both now below Node and Bun. That works because a lazy cluster's cache is small enough to sit under the line. A 50 000-key object's field storage is ~400 KB and cannot.

What to look at

  • Whether old-gen growth from large-object births feeds the full-collection trigger at all, or only growth observed through promotion.
  • Whether the arena trigger's exponential rebaseline should be bounded while old_in_use is climbing — the two arms currently move in opposite directions.
  • old_pending=false at old_in_use ≈ old_threshold is the specific observation to explain first.

Scope note

This is GC policy, which CLAUDE.md routes separately and warns about, so it wants its own change with the ratchet corpus run before and after rather than riding along with JSON work. Filing it with the evidence rather than attempting it inside a JSON performance pass.

Rows currently affected: wide_1m:parse RSS 228 vs 95 MiB (2.4x), and the residual records_array_8m:scan 187 vs 134 and records_array_20m:scan 256 vs 226, whose clusters are genuinely over the pointer-bearing line.

Refs #10098, #793.

Activity

  1. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    Root cause found: the RETAINING veto is fed by the survival its own pinning creates

    A full collection reclaims all of it. PERRY_GEN_GC=0 on wide_1m:parse, same binary:

    iterations generational full-sweep-only
    8 56 MiB 54 MiB
    64 222 MiB 55 MiB (flat)

    So nothing is unreclaimable and no allocation needs to move. The generational regime simply never runs the collection that would free it. At 55 MiB Perry would be below Node (121) and Bun (95) on this row.

    The veto

    gc::policy::old_reclaim_pressure_due:

    let crossed_absolute_threshold = old_in_use >= threshold
        && baseline < threshold
        && !GC_MAJOR_PACING_RETAINING.with(|c| c.get());

    GC_MAJOR_PACING_RETAINING is armed from measured young-survival, which on this workload reads survival_permille=997 on every minor. #7937's reasoning for the exemption is that "a heap whose young generation is not dying is retaining live data, and a full mark-sweep cannot lower the number being watched."

    That reasoning inverts here. The young generation is not dying because large old-gen containers pin it through the remembered set — every parse's 400 KB field storage is born tenured, and a minor never sweeps old-gen, so it holds its 50 000 key strings live. High survival is the symptom of the condition a full collection would clear, and the veto reads it as proof that one would be futile. The measurement above is the counter-example: the full sweep lowers exactly the number being watched, by 4x.

    The proportional arm does still bound the exposure, but the RETAINING multiplier widens the band to 128 MB, and old_in_use reaches ~112 MB over 64 iterations — so the bound is real but sits above where this workload ends.

    The width cliff confirms the container

    RSS growth from 8 to 64 iterations, by object width — the step is exactly at LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB), i.e. 16 384 keys at 8 bytes each:

    keys keys array growth
    8 000 62 KiB 16 MiB
    14 000 109 KiB 21 MiB
    17 000 132 KiB 67 MiB
    24 000 187 KiB 85 MiB
    50 000 390 KiB 166 MiB

    Under the line the storage is nursery-resident and a minor reclaims it. Over the line it is born tenured and only a full collection can, which is the one that never comes.

    Suggested shape

    Young-survival cannot distinguish "live data" from "dead data pinned by a born-tenured container", so it should not be the sole veto on the arm that would tell them apart. Two options, either of which is testable against this fixture:

    1. Exempt old-gen growth that came from births rather than promotion. credit_promoted_bytes_to_old_baseline credits the baseline only from promotion, so a born-tenured object moves old_in_use without moving baseline — the two quantities the arm compares already disagree in exactly the informative way. Tracking born-tenured bytes since the last full collection and letting them bypass the RETAINING veto is a small change with a direct signal.
    2. Make the veto self-correcting: apply it only while the previous full collection actually failed to reclaim. Here the first one would succeed and disarm it.

    Either needs the gc-ratchet corpus run before and after, which is why this is filed rather than attempted inside a JSON pass.

    Note this is the same root cause as the residual records_array_8m:scan (187 vs 134 MiB) and records_array_20m:scan (256 vs 226), whose clusters are also genuinely over the pointer-bearing line. #10098 fixed the two rows whose clusters fit under it by making them nursery-resident; that approach cannot reach these.

  2. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    Attempted the trigger fix. It does not work, and the measurements say why: the trigger is only half the chain.

    Three attempts, each measured on wide_1m:parse (64 iterations, Node 121 MiB, Bun 96 MiB):

    change full collections peak RSS
    main 0 222 MiB
    exempt births from the RETAINING veto 1 222 MiB
    + recurring births arm, shared predicate 2 284 MiB
    + births arm split off the shared predicate 2 284 MiB

    Making the collection fire is achievable — a born-tenured byte counter reset at each reclaim, exempted from the RETAINING veto, produces collections where main has none. It does not lower peak RSS, and two fulls make it worse than zero.

    Why: the arena never returns capacity

    [gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0
                          low_collections=0 fulls_remaining=0 disarmed=false
                          arena_live=81407888 arena_capacity=90177536
    [gc-arena-rebaseline] total=141557760 step=1073741824 -> 1073741824
    

    Right-sizing is armed (disarmed=false) and never starts an episode. So the chain is:

    1. Old-gen fills with born-tenured storage that no minor sweeps.
    2. The arena grows to hold it — total reaches 141 MB, rebaselining its own trigger 536 MB → 1 GB along the way.
    3. A full collection, when it finally runs, frees objects but returns no capacity.
    4. Peak RSS is set by arena capacity, which only ever grows.

    That is why PERRY_GEN_GC=0 holds 55 MiB flat: with full sweeps throughout, old-gen never accumulates, so the arena never grows in the first place. It is not that fulls shrink the heap — it is that they stop it expanding.

    Two late fulls are therefore strictly worse than none: the arena has already grown, and the collections add their own cost on top.

    What this means for the fix

    The trigger work above is necessary but not sufficient. Either:

    1. Fire much earlier, before the arena expands — the born counter makes that expressible, but the floor would have to be well below the current threshold/2, and that trades CPU against a benchmark this issue has not measured. The gc-ratchet corpus is the right instrument.
    2. Make arena-right-size actually run after a productive full. episodes=0 with disarmed=false is the specific thing to explain: arena_live=81 MB against arena_capacity=90 MB is only 10% slack, so a ratio-based gate may simply never see enough headroom — while the 141 MB the arena reached, versus the 55 MB the workload needs, is where the memory actually is.

    Option 2 looks like the higher-value half, and it is independent of the trigger.

    Also worth noting

    old_reclaim_pressure_due has three consumers, one of them
    copied_minor_promotion_handoff_pressure_due. I split the births arm out of the
    shared predicate on the theory that scheduling a full before a promotion was the
    #7592 futile-handoff shape. It made no difference — 284 MiB either way — so the
    handoff is not implicated here, and a future change need not carry that
    complication.

    Everything above is reverted; nothing from this investigation is on a branch.

  3. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    The complete chain: two independent heuristics, both fed by the pathology they would cure

    Same fixture, same binary, only PERRY_GEN_GC differs:

    regime arena_live arena_capacity utilization peak RSS
    generational 158 MB 175 MB 90% 222 MiB
    full-sweep-only 6.4 MB 14.7 MB 44% 55 MiB

    The workload's true live set is 6.4 MB. Everything above that in the generational column is dead storage that no collection has adjudicated: born tenured, over LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, and a minor never sweeps old-gen.

    That inflated number is then consumed by two separate gates, and both read it as "this heap is busy":

    1. old_reclaim_pressure_due — GC_MAJOR_PACING_RETAINING is armed from young-survival, which reads 997permille because the born-tenured containers pin the young graph through the remembered set. The veto blocks the collection that would disprove it.
    2. arena_right_size — opens an episode only at or below ARENA_RIGHT_SIZE_TRIGGER_PCT (50%) live/capacity. It sees 90% and declines, so episodes=0 starts=0 released_capacity_bytes=0 with disarmed=false. Capacity is never returned.

    So the chain is: born-tenured storage accumulates -> arena_live reads 25x the true live set -> the reclaim trigger declines AND right-sizing declines -> the arena grows to hold the garbage and never gives it back -> peak RSS is 4x the true requirement.

    PERRY_GEN_GC=0 holds 55 MiB not because full sweeps shrink the heap, but because the arena never grows in the first place: live stays at 6.4 MB, so nothing ever asks for 175 MB of capacity.

    Why the trigger fix alone cannot work

    This is the measured reason attempts 2 and 3 made things worse rather than better. By the time a births-driven trigger fires, the arena has already expanded, and right-sizing still will not open because arena_live is still inflated -- the two fulls left it at 81 MB against a 6.4 MB true live set. Two late fulls are strictly worse than none: no capacity comes back, and the collections cost their own time.

    Where a fix has to start

    The shared defect is that both gates consult a liveness number that includes old-gen nothing has visited since it was born. Options, in the order I would try them:

    1. Give the right-size gate a birth-aware path. Born-tenured bytes since the last reclaim are already trackable (a counter at the two birth sites in arena/allocators.rs costs one add). A heap whose capacity is dominated by un-adjudicated births should be able to open an episode regardless of apparent utilization, because apparent utilization is precisely what cannot be trusted there.
    2. Fire the reclaim early enough that the arena never expands, which means a floor far below threshold/2. That trades CPU, and the gc-ratchet corpus is the instrument.

    Option 1 looks independent of the trigger work and addresses the half that actually holds the memory.

    Dead ends, so nobody re-derives them

    • Exempting births from the RETAINING veto: produces collections (0 -> 1), no RSS change.
    • A recurring births arm in old_reclaim_pressure_due: 2 collections, RSS 222 -> 284 MiB.
    • Splitting that arm off the shared predicate so copied_minor_promotion_handoff_pressure_due keeps the original: identical 284 MiB, so the handoff is not implicated and a future change need not carry that complication.

    All reverted; nothing is on a branch.

  4. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    Root cause found: promotion is structurally invisible to old-gen pressure

    Correcting my three earlier comments on this issue. I framed this as a trigger-tuning
    problem and measured three dead ends against that framing. It is not a tuning problem,
    and that is why none of them moved.

    The mechanism

    credit_promoted_bytes_to_old_baseline (gc/policy.rs) adds a minor's promoted bytes
    straight onto the old-reclaim baseline:

    GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|b| b.set(b.get().saturating_add(promoted_bytes)));

    So old_in_use - old_baseline is invariant under promotion, by construction. Promotion
    raises both sides in lockstep. Only allocation born directly into old-gen can ever create
    old-gen pressure.

    That is deliberate and I am not proposing to change it — #7902 and #7965 both settled it,
    and the argument in the code is right: the baseline is the base of a growth measurement,
    not a liveness claim, and withholding the credit pins that base at 0 on exactly the
    workloads that reach it.

    But it leaves one question unanswerable: how much of old-gen arrived because a minor
    could not tell garbage from survivors?
    On wide_1m:parse every minor promotes at
    999 permille. Nothing dies. The entire nursery is laundered into old-gen, where no
    minor will ever look at it again — and the only thing scheduling fulls was incidental
    born-tenured allocation elsewhere in the parser.

    How that was proved

    JSON.parse re-mints every key of a wide object as a born-tenured string on every call:
    PARSE_KEY_CACHE holds 4096 entries, a 50k-field object overflows it on the first parse,
    and the parse-boundary clear then wipes it before the second parse can hit it. So 50,000
    js_string_from_bytes_longlived allocations per iteration.

    Fixing that — raising the bound so the schema stays cached — makes the row much worse:

    peak RSS @256 it fulls
    main 387 MiB 8
    key-cache bound raised 833 MiB 0

    Removing the garbage removed the only thing that was scheduling collections. That is the
    cleanest possible demonstration of the mechanism, and it is a trap for anyone who fixes
    the parser side first.

    The fix

    Record a second, independent signal alongside the baseline credit rather than correcting
    it. Bytes promoted by a minor at >= 950 permille survival accumulate in
    GC_OLD_GARBAGE_SUSPECT_BYTES; 16 MB of it makes an old reclaim due on its own; a reclaim
    resets it. The baseline is untouched, so #7902 and #7965 both still hold.

    Split off old_reclaim_pressure_due on purpose rather than folded into it — that predicate
    is also what copied_minor_promotion_handoff_pressure_due asks, and an arm there schedules
    a full immediately before a promotion, over survivors it cannot reclaim, which is the
    futile-handoff shape #7592 removed. (That was one of my earlier dead ends.)

    Floor is the measured knee of a 2/4/8/16/32/64/128 MB sweep at 256 iterations:

    floor peak RSS CPU fulls
    8 MB 243 MiB 3.14 s 14
    16 MB 243 MiB 3.00 s 12
    32 MB 274 MiB 3.00 s 11
    64 MB 510 MiB 2.71 s 5
    (unpatched) 387 MiB 3.03 s 8

    wide_1m:parse peak RSS 387 MiB -> 243 MiB (-37%) at baseline CPU. With the pacing arm
    in place the key-cache fix is measurably redundant (243 MiB either way), so it is not
    included.

    Branch gc/old-born-bytes-arm-full, commit 14ceaef3d6. Not opened as a PR — this is a
    GC pacing change and needs the gc-ratchet corpus run before/after by someone with that setup.

    What still stands between this row and parity

    Not pacing. RSS floors at 243 MiB however many fulls run — because freed pages are never
    returned to the OS:

    arena_live=56678536  arena_capacity=67108864      <- 56.7 MB live in a 67 MB arena
    [gc-general-reclaim] examined=41 released=0 rejected: no_snapshot=40 ...
    [gc-arena-right-size] episodes=0 starts=0 released_capacity_bytes=0
    

    The process holds 243 MiB while its arena is 67 MB and its live set is 56.7 MB. Block
    release rejects 40 of 41 blocks at snapshot.data == 0 (arena/reset.rs:700). Targets are
    Node 121 MiB / Bun 96 MiB, and 56.7 MB of live data is comfortably inside both — so this is
    the whole remaining gap on the row, and it is independent of everything above.

    Dead ends, so nobody re-derives them

    1. Exempting births from the RETAINING veto — 1 full, 222 MiB, unchanged.
    2. A recurring births arm on the shared predicate — 2 fulls, 284 MiB (worse; the
      futile-handoff shape).
    3. Splitting that arm off the shared predicate — identical 284 MiB, so the handoff was
      never implicated.
    4. A born-tenured-bytes floor — this does work (387 -> 154 MiB) but costs +7.6% CPU,
      because it buys RSS with extra whole-heap marks. The suspect-promotion arm gets most of
      the same RSS for free. Superseded.
    5. Raising the parse-key cache bound alone — 833 MiB, as above.
  5. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    Correction, and the actual root cause

    First, a correction to my previous comment. I said the suspect-promotion arm delivered
    "387 MiB -> 243 MiB at baseline CPU". That CPU claim was measured on wide_1m:parse only.
    Measured across the rest of the JSON matrix against clean main, it is not baseline CPU:

    row CPU main -> arm RSS main -> arm
    records_array_1m:scan 0.59s -> 1.30s (+120%) 410M -> 118M
    records_array_8m:scan 1.73s -> 2.60s (+50%) 444M -> 178M
    records_array_20m:parse 0.73s -> 1.05s (+44%) 267M -> 196M

    Worse, the regression does not respond to the floor — swept at 16/64/256 MB,
    records_array_1m:scan stays at 1.26–1.30s even at 256 MB where the arm should be inert. So
    the cost is the predicate, not the extra collections: old_reclaim_pressure_or_suspect_due
    sits in gc_budgeted_due_trigger, which runs at every safepoint, and the arm adds a
    thread-local read on the common path. Reverted in 9e1c80e060. The diagnosis in that
    commit stands (promotion credits the baseline, so only born-tenured allocation can create
    old-gen pressure); the remedy does not.

    The real cause: wide objects are born tenured, and a minor never sweeps old-gen

    arena/allocators.rs already states the failure mode exactly:

    It is TYPE-DEPENDENT, because the price of crossing it is: this object is also stamped
    GC_FLAG_TENURED, and a minor never sweeps old-gen — so for a POINTER-BEARING object the
    cost is not its own bytes, it is every object it can reach, held live through the
    remembered set by a container nothing refers to any more
    .

    A wide JSON document's shape-keys array is that container. Above 16,384 fields its keys
    array crosses LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES (128 KB), is born tenured, and
    then holds its whole key set live after every instance has died.

    Confirmed to the byte — PERRY_GC_CENSUS, 64 parses, fixtures straddling 131,072 bytes:

    fields keys array shapes retained live peak RSS
    16,000 128,016 B 0 0.0 MB 29 MiB
    16,300 130,416 B 0 0.0 MB 29 MiB
    16,500 132,016 B 30 22.7 MB 98 MiB
    50,000 400,016 B 15 34.3 MB 177 MiB

    The census names it directly: shape_keys_arrays {bytes: 6000240, count: 15},
    slot_tags {string: 750000} — 15 arrays × 50,000 keys = 750,000 live strings, on a document
    whose live set is ~0. Same binary under PERRY_GEN_GC=0: live 336 bytes, 50 MiB RSS.

    Ruled out by direct experiment (each rebuilt and re-measured, all zero effect):
    transition-cache size (raised 16,384 -> 65,536 on both the runtime constant and the mask
    codegen bakes in proxy_reflect.rs); parse-key identity (PARSE_KEY_CACHE bound raised
    4,096 -> 65,536); the shape table (table_len=12 < 15 live arrays); registered mutable root
    scanners (all report slots=0 ptr_roots=0).

    What a fix is worth, and why I am not proposing one yet

    Raising the threshold 128 KB -> 512 KB (still inside the copier's 1 MiB MAX_YOUNG_MOVE_BYTES
    and nursery-block ceilings) on wide_1m:parse:

    0.62s / 284 MiB -> 0.15s / 40 MiB — −76% CPU and −86% RSS, better on both axes, against
    Node 121 MiB and Bun 96 MiB. perry-runtime is green (3628 passed) after retargeting two
    tests that hardcoded a field count and asserted old-gen residency — they now derive their
    width from the constant, so the old-gen path stays covered at any threshold.

    But globally it is not shippable: the same change moves ordinary 128 KB–512 KB data arrays
    into the nursery, and on rows that do not need it that costs RSS —
    records_array_8m:scan 444M -> 710M (+60%), records_array_20m:parse 267M -> 314M (+18%)
    (CPU on those two improves, −29% and −44%).

    So the widening has to be scoped to wide-object JSON construction, not global. I built
    half of that: a WideShapeKeysBirthScope around allocate_parse_shape_keys_array, gated in
    the cold large-object branch of arena_alloc_gc so the hot path is untouched. It fires
    correctly (born young past threshold: 400016 bytes every parse) but only gets
    wide_1m to 176 MiB, because the keys array is half the story — a 50,000-field object's own
    property storage is also 400 KB and also born tenured. The scope needs to cover the
    object-storage allocation on the JSON construction path too; records_array_* is unaffected
    by that because its large allocation is array element storage, not wide-object storage.

    That is the next step, and it wants the quiet-mini matrix plus the gc-ratchet corpus rather
    than more measurement on a box at load 42.

  6. proggeramlug commented on Sep 12, 2026

    @proggeramlug
    ContributorAuthor

    Fixed — PR #10145

    The scoped fix is finished and open as draft #10145 (gated on the gc-ratchet corpus, since it changes birth generation).

    The half I built earlier didn't work because the scope was open in the wrong place. A mask probe at each large allocation showed the keys array was admitted (mask=2) but the object's property storage saw mask=0. That storage is minted in object/json_construction.rs via js_object_alloc_class_inline_keys_stamped, not inside js_json_parse_result. I found the site by tagging all nine GC_TYPE_OBJECT allocation sites in object/alloc.rs. Scoping it there completes the fix.

    Measured on current main against a same-tree baseline (the commit reverted and rebuilt):

    row node bun baseline #10145
    wide_1m:parse 136 MiB 71 MiB 220 MiB / 0.20s 40 MiB / 0.15s
    records_array_8m:scan 156 166 643 / 0.94s 643 / 0.93s
    records_array_20m:parse 422 170 313 / 0.40s 313 / 0.41s
    records_array_1m:scan 93 95 410 / 0.58s 410 / 0.59s
    records_array_16k:scan 57 77 117 / 0.13s 117 / 0.13s

    One more correction to my earlier comments. The "main" figures I quoted before (283 MiB for wide_1m, 444 MiB for 8m:scan) came from a frozen binary whose runtime turned out to be a different source state. The compiler refused to cross-link it (library source 111389820894 vs a511bd884a7f). The numbers above use a baseline built from the same tree as the fix. That also shrinks the global-widening regression I reported on 8m:scan from "+60%" to 643 → 710 MiB. It's still a regression, and it's still why this is scoped.

  7. proggeramlug commented on Sep 13, 2026

    @proggeramlug
    ContributorAuthor

    Closed by #10171 and #10145 ("Closes #10123"), landed via merge train #10188 (main 5cec2fbbc9).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugConfirmed defect or regression

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions