Repository navigation
perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902) #7965
Description
Activity
Bisected to one commit and one line: #7944 (#7902) stopped crediting untraced promotion to the old-reclaim baseline
Independent confirmation, plus the bisect. Four builds, each a full
-p perry -p perry-runtime-static -p perry-stdlib-staticin an isolated target
dir; corpus byte-exact vsnode --experimental-strip-types26.5.1 and exit 0 at
every hop. Instructions retired (load-independent — the dev box sat at load
77–92),full=collection_kind:"full"in thePERRY_GC_TRACE=1JSON.commit deeplist retain1 retain retain_wide 0809ada92(before all three)fulls / instr 0 / 1117 M 0 / 1286 M 0 / 2567 M 0 / 2967 M e886b56dd(#7900 + #7901)fulls / instr 0 / 1106 M 0 / 1294 M 0 / 2596 M 0 / 3026 M 1bd5eeb6b(+ #7902)fulls / instr 1 / 3910 M 1 / 3720 M 2 / 12 431 M 2 / 8786 M ac52a5c38(current main)fulls / instr 1 / 3928 M 1 / 3734 M 2 / 12 457 M 2 / 8755 M #7900 and #7901 are exonerated — flat to ≤0.6%, which is this A/B's noise
floor. The whole regression appears at #7944 (#7902) and nothing after it
moves. Peak RSS moves with it:deeplist75.5 → 103.5 MB (+37%),retain1
87.0 → 116.6 MB (+34%),retain249 → 347 MB (+39%).★ It is NOT promotion granularity
The leading hypothesis was that #7902's bounded untraced budget produces more,
smaller promotion steps, newly straddlingold_reclaim_pressure_due's absolute
arm. Measured, that is false.deeplist's promotion trace is byte-identical
across the boundary — same cycles, same untraced classification, same counts:e886b56dd: untraced=false untraced_cycles=0 untraced_objects=0 untraced=true untraced_cycles=1 untraced_objects=337035 untraced=true untraced_cycles=2 untraced_objects=954928 1bd5eeb6b: (identical, all three lines)The actual mechanism — one line
- credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes); + if !untraced { + credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes); + }
deeplistand theretain*family promote through the untraced in-place
path. Post-#7902 those bytes no longer advance
GC_LAST_OLD_RECLAIM_IN_USE_BYTES, soold_in_useclimbs to 53.5 MB while
baselinestays pinned at 0 — and then both arms of(old_in_use >= T && baseline < T) || old_in_use.saturating_sub(baseline) >= band(baseline)
are true for the first time (T =
GC_OLD_GEN_RECLAIM_THRESHOLD_BYTES= 48 MB).
The resulting full mark-sweep reportssweep.freed_bytes = 0: it is a
whole-heap collection over an old-gen that is live by construction. On
deeplistthat one futile cycle is 72% of the program's instruction count.So the arm's shape is not the bug — its input was silently zeroed. #7902's
reasoning ("an untraced promotion makes no marked-liveness claim, so do not tell
old-reclaim pacing that cohort is clean") is sound about liveness, but the
baseline is not a liveness claim, it is the denominator of a growth band.
Zeroing it does not make pacing more conservative, it makes the band meaningless:
old_in_use - 0exceeds any band as soon as old-gen exists at all.Two shapes for a fix, both preserving #7902's intent:
- Credit untraced promotions to the baseline but keep them accounted as the
uncertain class elsewhere — the baseline records where old-gen was, not
what was proven clean. - Keep the guard but make the absolute arm unreachable when the last cycle
measured a ~100%-live young generation (young_survival_permillewas 1000
on every one of these cycles), i.e. do not schedule a reclaim that the
collector has just finished proving cannot free anything.
Interaction with #7929 / PR #7961
Recorded so nobody double-counts: PR #7961 lowers the nursery band on
small-object programs, which keepsdeeplistunder the 48 MB threshold and so
dodges this regression without fixing it. Against currentmainthat reads
as −72.3%; against the pre-regression basee886b56ddthe same binary is
−1.6% ondeeplist— i.e. essentially all of that headline is #7965, not
#7929.retain1/retain/retain_widestill carry the full with #7961 applied.
The #7961 numbers are being restated both ways on that PR.- Credit untraced promotions to the baseline but keep them accounted as the
★ #7961 (merged) is a second, independent trigger for the same latch — this must be fixed together
Follow-up to the bisect above. I rebuilt #7961's change on the pre-regression
base (e886b56dd, before #7902) to measure the term in isolation, and it
reproduces this issue's symptom on its own:program e886b56ddfulls / instre886b56dd+ #7961Δ retain1 0 / 1293.7 M 1 / 3342.4 M +158%, RSS +25.4% retain 0 / 2596.4 M 1 / 4635.8 M +78.5%, RSS +2.6% deeplist 0 / 1107.2 M 0 / 1093.5 M −1.2% baseline(GC_LAST_OLD_RECLAIM_IN_USE_BYTES) is a running sum of promotion
deltas, so the absolute arm'sbaseline < Tis decided by promotion step
size. Onretain1:steps final baseline latch base 18.74 + 34.60 MB 53.34 MB > 48 disarmed by overshoot +#7961 18.74 + 26.21 MB 44.95 MB < 48 still armed retain1'sallarray is a large object born straight into old-gen; its growth
never passes through promotion, so it never credits the baseline. It then carries
old_in_usepast 48 MB and the still-armed latch fires —freed_bytes = 8 257 632, a whole-heap collection for 8 MB of dead array backing stores.So the arm is disarmed only by accident — by a promotion step large enough to
overshoot T. #7902 removed one source of credit; #7961 shrinks the steps. Neither
is wrong about its own subject; the latch is wrong to depend on either.Ordering hazard, please do not miss it: on current main both arms already
carry this full, so #7961 measures as a win there (retain1−10.9%,retain
−63.1%). The moment this issue is fixed, those two flip to +158% / +78.5%.
A fix for #7965 that does not also re-measure #7961's nursery band will land a
regression onretain1/retainthat looks like it came from nowhere.Both are closed by the same durable change — make disarming a function of heap
state rather than of step size. Either credit old-gen growth to the baseline
however it arrived, or refuse to schedule a reclaim when the last completed cycle
measuredyoung_survival_permille = 1000(it is 1000 on every cycle in every row
above, i.e. the collector had already proved the reclaim could free nothing).Fixed in PR #7968 — and the mechanism proposed in this issue is refuted
Bisect (on the full count, one run per commit):
0809ada92 → e886b56dd
(#7900 + #7901) is flat at ≤0.6% — both exonerated;e886b56dd → 1bd5eeb6b
(#7902) carries the entire regression. Confirmed independently here:
cdb67021ereproduces8260a9e50's census exactly (retain5 cycles / 0
fulls / 2 841 M instr / 250.1 MB).★ The granularity hypothesis is false here
This issue proposed that
old_reclaim_pressure_due's absolute arm is decided by
promotion step size. That is a real property of that arm — it is why #7960
exempted it — but it is not what happened:deeplist's untraced-promotion
trace is byte-identical across the boundary. Step size did not move.The actual cause is #7902's
if !untraced { credit_… }GC_LAST_OLD_RECLAIM_IN_USE_BYTESis not a liveness claim, it is the base of
a growth measurement. A fully-live young generation promotes untraced on every
cycle, so nothing else credits the baseline on exactly the workloads that reach
this path and it stays pinned at 0 — whereuponold_in_use - baseline
collapses into absolute occupancy andgc_old_reclaim_growth_band_bytes's
proportional half (baseline / 2) collapses with it, leaving the constant floor.So this is the fifth instance of the rule, but reached from underneath: not
"paced on a quantity the collection does not move" but its corollary — a
constant band pacing a collector whose per-cycle cost is O(live), the shape
#7592 removed here and #7594 removed one generation down.Result
19/19 byte-exact + exit 0 both arms.
retain1 full → 0, 8 237 M → 2 176 M
instructions (−73.6%), RSS 311 → 249 MB;retain_wide1 full → 0,
5 741 M → 2 859 M (−50.2%). No program gains a cycle or a full. Against the
pre-regression base the fix lands below it (retain−23.4%,retain1−31.5%,
deeplist−17.5%) — i.e. #7960's independently-reported −14%…−31%, which is the
arithmetic check that the regression is gone rather than masked.Gated by
an_untraced_promotion_credits_the_old_reclaim_baseline: a real
untraced-promoting minor driven through the production entry point must advance
the baseline by exactly the bytes it relocated, plus a replay ofretain's
measured promotion series asserting that an uncredited baseline does fire —
so a green run proves the credit is load-bearing rather than that nothing was
tried. That second assertion failed on its own first draft.Note for #7929/#7961: its band merged on top of this regression and its
reported wins were the spurious fulls moving. Re-measured with the fix in both
arms and posted to #7929 — ±0.2% on the retain cluster, and +42…83% collection
count oncycles/interp/iso_miss.- added a commit that references this issue
on Aug 12, 2026 - added a commit that references this issue
on Aug 12, 2026
Summary
The
retaincluster regressed 2.2–4.8× onmainbetween8260a9e50andd78efca41. Independently confirmed by a second party on a separate build.The load-independent evidence is the collection census —
mainnow runs full mark-sweeps on programs that previously ran none:collection_kind:"full"minorretainretain1deeplist(measured at
cf9999855,PERRY_GC_TRACE=1 PERRY_GC_DIAG=1)The last clean quiet-mini sweep at
8039fa798recordedretain0.153 s (1.13× node),retain10.068 s,deeplist0.057 s — and that commit predates both suspects, so the regression is invisible to it. Corroborating timing on a loaded dev box (magnitude only, not a benchmark):retain0.83 s,retain10.24 s,deeplist0.24 s.The original reporter measured
retain2,840 M → 12,421 M instructions with fulls costing 190 ms + 605 ms.Suspected cause — not bisected
Two PRs merged in that window, both in the promotion/survival accounting:
e886b56dd— "subtract live from-space bytes, not the block high-water, in a copied minor". Changes what the survival ratio is computed from.1bd5eeb6b— "bound and heap-scale the untraced-promotion budget". Bounds the untraced budget ⟹ more, smaller promotion steps.Both are plausible on mechanism, and they interact: the survival ratio feeds the in-place-promotion decision, and promotion step size feeds the old-gen trigger.
The mechanism this most likely trips
A third party independently instrumented
old_reclaim_pressure_due's absolute arm (old_in_use >= T && baseline < T) and found it races two quantities that both move with promotion, so whether it fires depends on promotion step size, not on the heap: same program, same live set, same total promotion, and changing only the schedule from (18.7, 34.6) to (17.7, 17.8) makes it fire twice and buy 588 ms of futile fulls.#7592 exempted the proportional arm via
GC_MAJOR_PACING_RETAININGand left this one. #7902 changing promotion step size is exactly the input that would newly trip it.That also makes this an instance of the rule CLAUDE.md already carries, for the fifth time: never pace a collection on a quantity that collection does not move — and its corollary that a constant band cannot pace an O(live) process.
Asks
8260a9e50..d78efca41on the full count (a counter, so a single run per commit suffices — no quiet host needed).Note on downstream measurements
Any A/B taken against a base after
e886b56ddis measuring against a regressed baseline. #7960 (first-cycle in-place promotion) reports −41% to −77% against currentmainbut −14% to −31% against the pre-regression base; the latter is the honest figure and its author flagged this unprompted.