Repository navigation
perf(gc): whole-block in-place promotion — kill the ~243 ns/object promotion tax on a fully-live nursery #7742
Description
Activity
- added 4 commits that reference this issue
on Aug 10, 2026 Measurement of the residue named in #7744's "Not closed", on
main@ 0a2bf15 (perry-dev, M1 Max),retain.tsunderPERRY_GC_TRACE=1.The one full mark-sweep is 36% of all GC pause, and its premise does not hold on this workload.
6 cycles, 504.4 ms total pause. Five minors; one full at 183.7 ms.
What that full did:
arena in-use before → after 67.0 MB → 59.0 MB reclaimed 8.37 MB = 11.9% retained_forwarded_stub_bytes0 retained_forwarded_stub_objects0 Where its time went:
trace_worklist93.1 ms (55%),atomic_finalize32.9 ms (19%),remembered_set_marking28.9 ms (17%),sweep9.5 ms (6%).So more than half of it is re-tracing the 3M-object live worklist to free 8 MB, and it freed zero forwarding stubs — which is the entire documented justification for the arena-growth escalation ("a non-moving minor sweep cannot free array-growth forwarding stubs … only a FULL mark-sweep reclaims them"). Being under the 20% productive threshold, it is also what pushes
GC_MAJOR_PACING_BACKOFF_SHIFT0 → 1, i.e. the yield-adaptive backoff (#7726) only learns the bet was bad after paying 183.7 ms for it, once per run.The signal is available before escalating.
set_forwarding_addresshas exactly two production callers:array/push_pop.rs:162, which leaves the retained array-growth stub, andgc/copying.rs:744, whose forwarding is transient and cleared by the same cycle. So retained stub debt is countable with one add on the growth path and a reset when a full sweeps them, rather than the object walkevacuation_policyuses today.arena_growth_full_escalation_duecould then require that there is stub debt to reclaim, instead of discovering there was none by doing the work — and old-gen garbage is unaffected either way, sinceold_reclaim_pressure_dueis a separate trigger.Note this got cheaper to justify after #7744: whole-block in-place promotion does not copy, so it does not leave forwarding stubs at all on exactly the high-survival workloads where the escalation is most expensive.
I have not implemented this — it is a cross-cutting collector change that needs the bench corpus and a
gc_ratchetpass to land safely, and I did not want to start it inside an unrelated PR. Posting the measurement so whoever picks it up starts from numbers.(The other half of the residue #7744 names — the remembered-set scan over the array's dirty element slots — is untouched by the above.)
Closing: the promotion tax this issue was opened for is gone, measured three ways
243 ns/object → 8.9 ns/object. Not an estimate —
PERRY_GC_TRACE=1per-cycle
records onretain.ts: cycles 1–4 promote 2,109,820 objects in 18.7 ms of
in_place_promotion(gc-handoff/RETAIN4-NOTES.md). The intermediate step was
105 ns/object when #7744 landed whole-block promotion; #7888 removed the trace on
fully-live promoting cycles, which is the rest of it.End to end, on the pinned quiet M1 mini (
gc-handoff/MEASUREMENT-PROTOCOL.md,
baseline round 2026-08-12 @14e291f3a), the two benchmarks this issue names:bench at filing now node now vs node retain0.80 s 0.153 s 0.137 1.12× retain_wide1.33 s 0.197 s 0.157 1.25× The opening premise — "the only two benchmarks still far off Node" at 6.7× and
8.9× — no longer holds; both are inside the 1.3× band that 15 of the 19 corpus
programs sit in.Each of the four "things this has to get right" has a named mechanism, and each
was measured rather than argued:- The promotion decision —
gc/promote_in_place.rs, per cycle from the
previous cycle's measured young-survival ratio. Sound rather than
optimistic because a promoting cycle still traces, so it re-measures; the
observed population is bimodal with a three-order-of-magnitude gap (0.999–1.000
vs 0.000–0.004), so the 95% threshold is justified by footprint, not by
classification. - Page-generation metadata and the remembered set —
arena/promote.rsretags
the young blocks toOld/PromotedYoungbefore the trace, so the barrier
predicates read old-gen semantics while the trace runs and record the right
edges;Old ⟹ TENURED(perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) is restored by the finish walk's header stamp,
and the old-gen page index is registered in per-page bulk runs (now as a
described run rather than a stored list, perf(gc): describe a promoted block's page-object list instead of storing it #7914). - Pacing — the exact hazard the issue predicted did fire: the first working
version collapsed Eden's high-water 57 → 18 MB and turned a 0.81 s program into
1.52 s. Fixed by returning the freed capacity as one-shot headroom to the
next trigger rather than re-reserving blocks. - Footprint —
PROMOTED_DEAD_BUDGET_BYTES(32 MB of promoted dead bytes since
the last full) plus the untraced arm'smax(128 MB, old-gen at last measurement)budget, both reset by a full collection. Measured
in_place_dead_bytesandin_place_sparse_blocksare zero across the
benchmark set.
The residue, named rather than left attached to this issue
None of it is promotion cost, and three of the four already have their own
trackers:- memory(gc): untraced promotion can retain an old-heap-sized phase-change cohort as live #7902 — untraced promotion can retain a phase-change cohort as live.
- perf(gc): the nursery/promotion budget is denominated in BYTES but the collector's cost is per-OBJECT — compaction is taxed back #7929 — the nursery/promotion budget is denominated in bytes while the
collector's cost is per-object. - perf(gc): time-budgeted steps still contain unbounded atomic work #7903 — time-budgeted steps still contain unbounded atomic work.
- Unfiled until now: cycle 0 cannot promote in place, because nothing has
been measured yet — so the first copying minor evacuates object-by-object and
is 58% ofretain's GC pause, 81% ofretain1's, 71% ofretain_wide1's,
and cycle 1 then promotes those very blocks in place. The obvious fix (a
smaller first nursery) is measured unsafe as a global change: at a 2 MB cap
asyncpipegoes from 0 collections to 4, costing 385 ms on a 127 ms program.
Filed as perf(gc): the first copying minor cannot promote in place — 58-81% of remaining GC pause on the live-set cluster #7937.
Also worth recording for whoever picks up the next round: on this cluster the
mutator is now the binding constraint, not the collector —retain_wideand
shapeslose to scriptc with a collector that costs zero, andretainneeds it
to cost exactly that. A further round aimed here that works on the collector is
working on the smaller half.- The promotion decision —
retain(3M records, none ever dropped) andretain_wideare the only twobenchmarks still far off Node — 0.80 s vs 0.12 and 1.33 s vs 0.15 on the pinned
M1 mini. Everything else is within 1.2×–3.6×.
The cause is that every survivor gets promoted, object by object. The GC
trace shows 2,097,155 promotions across four copying minors at 509.7 ms of
pause — 243 ns per object — spent on
arena_alloc_gc_old, thememcpy,layout_transfer,old_page_account_promoted_object, theGcMoveHookKindhooks, and the forwarding/rewrite that follows. The measured young-survival
ratio on those cycles is 1.000: not one of those objects had a reason to
move.
The structural answer is V8-style whole-page promotion: when a block is
(near-)entirely live, relabel the block as old-gen instead of evacuating it.
Nothing moves, so there is no copy, no old-gen allocation, no layout transfer,
no move hook, no forwarding pointer, and no slot anywhere in the heap or in any
address-keyed side table needs rewriting.
Things this has to get right:
and Eden blocks are recycled at offset 0, so per-block history means nothing.
A per-cycle decision from the previous cycle's measured survival ratio works
only if a promoting cycle still measures.
generation in place — this is where correctness lives. In particular
Old ⟹ TENURED(perf(gc): write barriers cost 16% on an all-numeric store workload — elide on provably-non-pointer stores #7511) and the old-gen page-object index, which is what theremembered-set dirty scan uses to reach an object at all.
The arena-bytes trigger fires on
arena_total_bytes(), so the allocationrunway collapses unless the capacity is given back.
full. The per-cycle cost is
(1 − liveness) × young bytes; the steady stateneeds its own bound.