Repository navigation
GC: tree.ts scavenge-on peak RSS (221 MB) still loses to scavenge-off (103 MB) — old-gen churn + young capacity high-water #7438
Description
Follow-up to #7432 (adaptive tenuring + young-scoped scavenge cap), which fixed the large-live-set scavenge wall-time regression but leaves one acceptance cell open from gc-handoff/PROMPT-scavenge-large-live-set.md: on tree.ts, scavenge-on must beat PERRY_GC_SCAVENGE=0 on peak RSS as well as wall.
Current state (same binary, arms back-to-back, macOS arm64)
| arm | wall | peak RSS |
|---|---|---|
| scavenge ON (#7432) | 10.26 s | 221 MB |
| scavenge OFF | 10.70 s | 103 MB |
| ON before #7432 | 48.06 s | 172 MB |
Max arena in-use across the 43 collections is only ~101 MB; the RSS above that is capacity high-water (mapped-ever pages, mimalloc MADV_FREE retention — heap is Memory Tag 240):
- Young: up to 64 MB — the influx-driven cap scale legitimately holds ×4 on tree (influx ~3 MB/cycle), and probing showed shrinking it is a bad trade: base=8 MB (eff 32 MB) measured 34 s wall for RSS still ≈220 MB — the young cap is NOT the RSS driver.
- Old-gen churn: high-water ~70-100 MB in-use under the pre-existing reclaim pacing (
GC_OLD_GEN_RECLAIM_THRESHOLD_BYTES = 48 MB, growth 32 MB), cycling 122 MB of promoted-then-dropped tree nodes through old blocks. 37 of tree's 43 collections are already old-reclaim-driven fulls, so the pacing is near its policy floor — lowering the global thresholds trades pause for RSS across every workload and belongs to old-gen policy work, not the tenuring PR. - Old-gen fragmentation under scattered survivors compounds this — filed separately (GC: old-gen fragmentation — scattered survivors pin 105 MB of blocks for a ~1 MB live set #7437).
The OFF arm's 103 MB is structurally cheap: the non-moving minor never promotes, so its old-gen is near-empty and its young high-water is one 16 MB cap. Any promoting scavenger pays a young-capacity + old-churn footprint; closing the cell needs old-gen reclaim/defrag to keep ever-touched old capacity near the live set (~35 MB), plus possibly returning young blocks more eagerly after the cap scale shrinks.
Acceptance
tree.ts scavenge-on peak RSS ≤ the PERRY_GC_SCAVENGE=0 arm on the same binary, without giving back #7432's wall win (≤ ~10.5 s) or the churn/cycles RSS results (24 MB / 29 MB).
Activity
proggeramlug commentedon Aug 5, 2026
Diagnosis update, measured on the dedicated bench mini (M1/8 GB,
perry-macos.local) at main5e236e6e2(post-#7443):The old-gen churn theory is retired. With the hole free list in, tree's steady state reclaims 37.7 of 37.9 MB per full via whole-block reclaim (dropped trees die contiguously; holes are a minor 838 KB term here), and per-region traces show old-gen in-use settling at ~1 MB with a one-time early spike to 36.7 MB (the first trees promote before the survival-rate lock and old-reclaim settle). The longlived arena holds ~1 MB — the earlier 84-block reading was a mislabeled DIAG line (
longlived=actually counts every non-general block; relabeled on the WIP branch).Where tree's ON-arm RSS actually is (
vmmapmid-run + per-region traces; ON ≈ 236 MB vs OFF ≈ 102 MB):ru_maxrssis a high-water metric, so it never un-counts.MADV_FREEpages stay resident; the union of ever-dirtied segments exceeds the concurrent arena sum.Open experiment (running): a clean cap matrix (effective 64/32/16 MB) on the quiet mini. The earlier "eff-32 = 34 s wall" reading was taken on a loaded MacBook and is not trusted. If eff-16 walls are competitive under the corrected young-scoped cadence, the Pareto cell closes by capping the scale when RSS-bound; if not, the trade is structural and this issue becomes "young cap scale needs an RSS-aware ceiling".
Side observation while profiling: the #6882 mimalloc VM-tag retag (
Memory Tag 240) is not covering the heap on current main — ~230 MB of heap mappings render asIOAccelerator(tag 100) again. Labeling-only, butdocs/src/internals/memory-model.md's profiling recipe currently misleads.proggeramlug commentedon Aug 5, 2026
PR #7449: recycled-block pool. tree.ts 225 -> 190 MB peak RSS at wall parity (same-session A/B on the bench mini); peak commit 257.5 -> 225.8 MiB; cap measured (no pool 225 / 64 MB 190 / 128 MB 210). The remaining ~88 MB vs the scavenge-off arm is structural (young-cap committed high-water 64 vs 16 MB, one-time early promotion spike, live-set double-residency during transitions) — next lever if this cell must fully close: ramping the tenuring lock in to damp early promotion volume.
gc(diag): the sweep blocks line labeled every non-general block 'long…
arena: recycled-block pool — released blocks are reused, not round-tr…
arena: recycled-block pool — block round-trips through the allocator …
proggeramlug commentedon Aug 5, 2026
Closing after a full experimental campaign on the promotion-damping lever. The verdict, with instruments: the remaining gap is the price of the promoting design, and every mechanism that closes it on tree.ts damages a mixed workload the collector must protect. All measurements on the dedicated bench mini (M1, Spotlight off), same-session A/B.
Where this issue ends up
gc/7438-close-rss-gap, NOT merged)The experiment (7 iterations, every transition DIAG-witnessed)
The last ~90 MB is the startup transient: the survival-lock promotes two consecutive live ~35 MB trees (98-100% dead within two collections —
[gc-promotion-waste] veto ENGAGED (promoted=17928832 retained=210496 died=98%)), spiking old-gen's committed high-water to 68 MB, before the K× major-pacing escalation converges the steady state onto in-place fulls (which is the scavenge-off arm's behaviour, and optimal here). A promotion-waste veto — promote only if promoted cohorts tend to stay live — was built and iterated:And the gc-ratchet rejected every airtight variant:
12_large_live_set(permanent list + ring churn — the mixed regime) produces genuinely clean ≥90%-dead promotion windows from its transients, so the veto engages, the accumulating list then ages in young, the trajectory release fires, the veto re-engages — and the probe pays +19% promoted / +14% peak RSS / +14% wall against its pinned baseline, at every engagement-bar setting. The three regimes (live-set-bound, accumulating, mixed) are not separable by any decision-time signal tried: waste ratio, young-live trajectory, tiny-young, full-freed-fraction — each has a measured counterexample in this thread.Structural floor
With the veto ruled out, tree.ts scavenge-on decomposes as: young committed high-water 68 MB (the cap matrix showed every smaller cap doubles wall before RSS moves: 64/32/16/8 MB → 235/221/226/168 MB at 9.8/18.7/20.2/19.6 s), first-flush old transient ~18 MB, runtime base ~25 MB, allocator/pool slack ~20 MB (pool curve also measured: 0/64/128 MB caps → 225/190/210). Floor ≈ 130 with a perfect veto, ≈ 190 without one — against the OFF arm's ~101, which never promotes and never pays young-capacity for wall.
Disposition
gc/7438-close-rss-gap(unmerged) preserves the veto implementation and its tests for any future attempt with a genuinely predictive signal (e.g., allocation-site tenuring feedback à la HotSpot's NUMA/AOT profiles — a per-site history could separate the ring transients from the list nodes where a global ratio cannot).