Repository navigation
gc-ratchet has been RED on main since 2026-08-18 (19 days); every labelled PR inherits it, and the failure has grown from 19 to 43 cells #9829
Description
Diagnosed after gc-ratchet and gc-root-dominance failed identically on
#9827, #9823, #9816 and #9807 — branches touching unrelated code. Unrelated
changes cannot produce identical failures, so the question was whether main
fails its own ratchet with no PR involved. It does.
1. Main fails its own ratchet
The workflow's main-line arm is a six-hourly schedule (push: branches: [main] was removed in #7856 because it starved the queue). That arm is red:
09-05T20:15 schedule failure d36a1af0c
09-05T15:05 schedule failure c7361c87c
09-05T10:28 schedule failure 1d63fa91f
Run 33989581881 (scheduled, main @ d36a1af0c, no PR) fails with 43
regression rows.
2. The PRs inherit it exactly — they contribute nothing
Extracting the regression rows from main's own scheduled run and from PR
#9823's run:
$ diff main_rows.txt pr9823_rows.txt
(no output — all 43 rows byte-for-byte identical)
So the campaign's GC-adjacent work is not the cause, and re-pinning on any
of those PRs would be absorbing main's problem into a branch that did not
create it.
3. It started 2026-08-18 and has been red for 19 days
Every scheduled run since is red; the last green is dc4bcf287:
08-18T01:46 failure 7441e1f73 <- first red
08-17T18:52 success dc4bcf287 <- last green
Of the last 100 scheduled runs (08-11 → 09-05): 15 success — all of them on
or before 08-17 — 64 failure, 16 cancelled.
4. It is NOT one regression: the failure has grown 19 -> 43 cells
The first red run (7441e1f73) had 19 regression cells. Today's has 43.
14 have been red the whole time; 29 were added since, including whole
probes that were clean on 08-18:
01_nursery_churn copied_bytes, copied_objects
04_dead_after_deep_stack copied_bytes, promoted_objects
05_closure_capture copied_bytes, promoted_bytes, promoted_objects
06_string_retention copied_bytes, copied_objects, freed_bytes, promoted_*
07_array_grow_evacuate copied_objects, heap_total_bytes, heap_used_bytes
09_try_catch_roots copied_bytes, copied_objects, promoted_bytes, promoted_objects
10_store_receiver_across_alloc copied_bytes, copied_objects, freed_bytes, heap_used_bytes
11_collect_at_depth copied_bytes, copied_objects
13_large_eden_survivors copied_bytes, copied_objects, heap_total_bytes, promoted_objects
At least two accrual events, 19 days apart. That is the single most
important fact here, because it rules out a one-line re-pin.
Representative rows from today (main, scheduled):
| probe | counter | baseline | current | delta |
|---|---|---|---|---|
10_store_receiver_across_alloc |
heap_used_bytes | 220,384 | 464,072 | +110.57 % |
10_store_receiver_across_alloc |
copied_objects | 8,160 | 0 | −100 % |
10_store_receiver_across_alloc |
freed_bytes | 8,930,928 | 0 | −100 % |
09_try_catch_roots |
promoted_objects | 6,240 | 0 | −100 % |
13_large_eden_survivors |
copied_objects | 0 | 377,569 | — |
12_large_live_set |
freed_bytes | 76,315,456 | 63,732,784 | −16.49 % |
The shape — byte counters moving together, and some probes going to zero
copied/promoted/freed — reads as probes whose collection schedule changed
(they stopped triggering a minor at all), not as a retention leak.
5. Candidate for the FIRST accrual, offered as a lead not a conclusion
Ten commits sit between the last green and the first red:
7441e1f73 feat(native): expose u8 and byte profile types (#8330)
2775ffe21 runtime: add zero-copy bun:ffi memory views (#6562) (#8329)
765900869 fix(worker_threads): preserve channel listener semantics (#8327)
1ed959122 ci: shard parity 8 ways (#8244)
6da121d02 test(ci): close out #8264 (#8335)
3f2fffc4f fix(dyn_eval): root global this before env_new_root allocates (#8333)
de70a433e fix(tls): match Node server identity semantics (#8334)
e3e0105e4 fix(node:test): expose test context assertions (#8332)
770b6223d chore(make): simplify fix target to clippy --fix (#8331)
e2e6f4a1c perf(object): shrink common objects to 40 bytes (#8313)
e2e6f4a1c — "shrink common objects to 40 bytes" — is the obvious
candidate: changing the common object size moves every byte-derived counter
and, by changing how fast the nursery fills, the collection schedule too, which
is exactly the failure shape. It is also plausibly a win that simply needs
the baseline re-pinned with that justification. I have not bisected it and I
am not asserting it. The second accrual is unattributed.
6. Why nobody saw it — the same class as #9774 and #7856
- The PR arm is opt-in on
run-extended-tests, so almost every PR shows
skippingand proves nothing. - The main arm is a schedule, and a scheduled failure blocks nothing and
notifies nobody.
So main can drift past its own pinned baseline indefinitely, and the first PR
to ask for the gate inherits 19 days of accumulated red. A gate that has failed
64 of its last 100 runs while blocking nothing is dark in CLAUDE.md's sense —
a fourth way, alongside continue-on-error, missing-from-required-contexts,
cancelled and starved: failing but unwatched.
7. Do NOT re-pin to clear this
Re-pinning is how a real regression gets absorbed, and here it would absorb an
unknown number of them: 43 cells, at least two accrual events, 19 days and
~200 commits. Nobody can currently say which of those values are correct.
Note 423975f6f (Sep 5) touched the baseline file but changed
probe_overrides / rationale metadata only — no one has re-pinned counter
values, so this red is not a hidden re-pin returning.
Suggested order of work
- Bisect the first accrual across the 10 commits above (the ratchet can be
run on a single commit;#8313first). - Bisect the second, which lands somewhere in 08-18 → 09-05.
- Only then re-pin, per-cell, with the justification the gate's own docs
demand: which counters moved, why the new values are correct, and what would
have failed under the old pin. - Make the scheduled arm visible — a red six-hourly
mainrun should open
or update an issue automatically. That is the part that would have saved 19
days, and it is independent of whatever the counters turn out to mean.
Filed from the keystroke lane while unblocking #9807/#9823/#9828; happy to take
the bisection if it is wanted, but which counter values are correct is a
judgement I should not make alone.
Activity
proggeramlug commentedon Sep 6, 2026
Bisected: the first accrual is
e2e6f4a1c perf(object): shrink common objects to 40 bytes (#8313)— 19 of 19 cells, byte-exactScoped to the first accrual as asked.
e2e6f4a1cis the immediate successorof the last green commit
dc4bcf287, so one measurement either side settlesit without a search.
Method — no baseline reasoning involved. I measured
gc_ratchet.py measureat
dc4bcf287and ate2e6f4a1con the same host and diffed the raw counters,rather than checking either against a pinned baseline. The gated counters are
documented as "semantic and transfer across machine classes", and the
measurement confirms it: spread across repeats is 0 % on every gated cell,
i.e. bit-identical.
Result
e2e6f4a1cacross all 14 probese2e6f4a1cdid not moveThe local
dc4bcf287measurement also reproduces the pinned baseline valueexactly for 13 of the 19; the other 6 differ by 0.02–0.15 % (e.g. 13,201,144 vs
13,181,528), which is inside their documented bands — and for all 6 the post
value still matches CI exactly.
Nineteen byte-exact matches on a different machine class from the CI runner is
about as firm as attribution gets.
Representative rows (CI current == local
e2e6f4a1c, exactly):e2e6f4a1c12_large_live_set.freed_bytes12_large_live_set.promoted_bytes13_large_eden_survivors.freed_bytes04_dead_after_deep_stack.freed_bytes03_cross_gen_writes.freed_bytesWhat the direction means
Overwhelmingly a win that was never re-pinned, and the shape is exactly
what "objects got smaller" predicts:
freed_bytesdown 9–18 % in nearly every probe — less garbage produced,because each object is smaller.
copied_bytes,promoted_bytes,heap_total_bytes,heap_used_bytesmostly down (
heap_total−5 % to −14 %).copied_objectsup 7–17 % in several probes. That is not retention: withsmaller objects more of them fit in the nursery before it fills, so more
survive per cycle. A schedule shift, which is why probes elsewhere in the
43 look like they "stopped collecting".
Two cells I am NOT calling a win, per the standing constraint
12_large_live_set.heap_used_bytes1,104,848 → 1,681,712 (+52.2 %) —live heap up by half on the large-live-set probe, while everything else on
that probe went down. Smaller objects reducing live bytes is expected;
increasing them is not, and I do not have an explanation.
14_grow_then_churn.copied_bytes+11.1 % andcopied_objects+10.8 % — theonly probe where copying rose in bytes as well as count.
Both need a reason from someone who knows #8313's layout change before any
re-pin. Everything else on this list I would be comfortable calling correct.
Scope
The second accrual is untouched — the 24 cells that appeared between
2026-08-18 and today are still unattributed, and I did not chase them in this
pass. Whether they are one event or several is the next question.
Re-pinning still needs per-cell reasons, and this comment supplies them for 17
of the 19; the two above are the ones to resolve first.
Filed alongside: #9830, for the automation that would have surfaced this on
2026-08-18 instead of nineteen days later.
https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m
proggeramlug commentedon Sep 6, 2026
Both unexplained cells resolved. Neither is a regression — and
12_large_live_setis a net 9.9 MB memory improvement, not a 52 % increase12_large_live_set.heap_used_bytes+52.2 %The decisive fact is what that counter is. Reading the probe:
heap_used_bytesisprocess.memoryUsage().heapUsedsampled after therelease phase and an explicit
gc()— 63 of every 64LNodes dropped, ~10,938kept. It is the probe's stranded-in-old-gen detector, not its memory figure.
The probe's memory figure is
heap_total_bytes, and that went down 12.7 %.I isolated the release phase and ran it at both commits on one host:
dc4bcf287e2e6f4a1cgc()So objects did NOT get bigger — they got 12 % smaller, exactly as #8313
intends. The commit removes the per-object
keys_arraymirror, takingObjectHeader32 → 16 bytes and a two-slot object 56 → 48 → 40. The live setshrinks accordingly; what rose is only what survives the release.
Putting both of that probe's retention counters together:
#8313 makes this probe use ~9.9 MB less memory. The cell that rose is a
~1 MB tail on a ~80 MB workload, and it rose because the objects are smaller:
more of them pack per page, so a cohort dropped uniformly (every 64th node
kept) leaves proportionally more partially-occupied pages that cannot be
reclaimed whole. That last clause is a hypothesis about the mechanism — the
measurement above establishes the facts (smaller objects, smaller live heap,
larger post-release residue) without depending on it.
So this is neither of the two options in the brief. Not "accounting moved" and
not "objects got bigger": objects got smaller, and the release leaves more
behind. It is a real, small, and strictly-dominated effect — worth pinning
with that reason recorded, not worth calling a regression.
I would still flag one thing for whoever owns #8313: the residue scales with
the drop pattern, not with the live set. A workload that frees most of a large
cohort and keeps a sparse remainder is the shape that pays it. cc does that, so
if anyone wants to chase footprint further, "post-release page occupancy after a
sparse survivor cohort" is a real lead — but it is a lead, not this commit's
defect, and it predates any question of re-pinning.
14_grow_then_churncopied up in both terms — confirmed schedule, not sizeAsked to confirm rather than assume, and the average settles it in one line:
The copied population's average size is unchanged (+0.32 %). These are
~4 KB objects — arrays, not the 40-byte objects #8313 touched — so their size
could not have moved, and 33 more of them were copied. That is the same
schedule shift that explains the other 17 cells: smaller objects mean more fit
in the nursery before it fills, so more survive per cycle. Confirmed, not
assumed.
Verdict
All 19 of 19 first-accrual cells now have a stated reason:
freed_bytes,copied_bytes,promoted_bytes,heap_total_bytesdown) or of the schedule shift thatfollows (
copied_objectsup).12_large_live_set.heap_used_bytes— post-release residue up ~0.58 MB whilethe same probe's total falls 10.5 MB; net −9.9 MB.
14_grow_then_churn— schedule, confirmed by invariant average object size.Nothing here needs filing as a regression, so the re-pin does not need to
carve anything out. A re-pin PR for these 19 cells can now be written with a
per-cell reason, as the gate's own docs require. I have not touched the 24
second-accrual cells and they must not be swept into the same re-pin.
Method note, same as the bisect: every comparison is two commits measured
against each other on one host, never against a pinned baseline.
https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m
proggeramlug commentedon Sep 6, 2026
Second accrual mapped: it is not one event — it is at least eight, and the cell set churns
Mapped without a single build, from evidence that already existed: each of the
59 failing scheduled runs since 08-18 carries its own regression rows, so
counting cells per run brackets every accrual to one six-hour window.
The timeline
7441e1f73526e0b502f4a7559d6c76b43930ead6364646d10e8a1d3b9e786d6b92c101b63a9c45801ef79a0211d36a1af0cThe count goes down as often as it needs to (32→28, 43→42, 48→43), so this
is not accumulation — it is several independent changes moving the same
counters, some partially offsetting.
My earlier arithmetic was wrong, and here is the correction
I said "19 attributed, 24 remaining". That assumed the original 19 persisted.
They did not:
02_survivor_promotion.copied_objects,09_try_catch_roots.freed_bytes,11_collect_at_depth.freed_bytes,14_grow_then_churn.copied_bytes, and12_large_live_set.heap_used_bytes;So today's 43 = 14 (from #8313) + 29 (later), not 19 + 24.
And that includes the cell I was asked to prioritise.
12_large_live_set.heap_used_bytesis not in today's failing set — it wentred at #8313 and has since come back into band. My analysis of why it moved
stands and produced a real fragmentation lead, but it was never blocking the
gate, and I should have checked that it was still failing before spending two
builds on it. Process lesson: confirm a cell is in the current failing set
before investigating it.
The cell that now deserves that treatment instead is
10_store_receiver_across_alloc.heap_used_bytes, baseline 220,384 →464,072, +110.57 % — today's largest direction-contradicting retention cell,
and it is in the accrued-29.
The cell names identify the culprits — no blind bisect needed
For the three smallest windows the match is close to unambiguous:
06_string_retention.*f1d231fa5..526e0b5025f9aa1f40 perf(codegen): retain string accumulators in concat chains (#8417)04_dead_after_deep_stack,05_closure_capture,07_array_grow_evacuate,14_grow_then_churn— copy/promote across four probesc2da03439..f4a7559d606e1ab349 merge: land #8652/#8651/#8647/#8646/#8650 (#8657)(five PRs in one commit) or2382a9f15 perf(codegen): delete six instructions from the generic property-read hit path (#8665)13_large_eden_survivors.{copied_bytes,copied_objects,heap_total_bytes,promoted_objects}+heap_usedon 09/112d6a05038..b92c101b63d1adbc4c gc: give the copied minor the liveness probes it never had (#9359)— a GC change to the copied minor, and the cells are copied/promoted on eden survivorsFull appeared/cleared lists for all eight transitions are in this session's
artefacts and can be pasted in if wanted.
Worth naming: most of these windows are perf work — much of it this
campaign's. That is expected and not an accusation: perf changes move GC
counters, and this gate exists to make you justify them. It was dark, so nobody
was asked to.
The judgement I am bringing back rather than making
the shortlists above is ~8 builds (one per window, testing the named
candidate). Both are real work; neither is a re-pin.
have only a window and a candidate, not an attribution.
So: do you want the ~8 confirmations, or is the map itself — every cell
bracketed to a six-hour window with a named candidate — sufficient justification
for a re-pin that says "the gate was dark for nineteen days; here is what moved,
when, and by what"? I do not think I should choose that alone, and I have not
started either.
Method unchanged: two commits measured against each other on one host, counter
definitions read before interpreting direction, structure printed before
trusting a diff.
https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m
proggeramlug commentedon Sep 6, 2026
Classification done, and it stopped short of the pin — for a reason worth the delay
Split by what a wrong answer would cost, as instructed. The sweep for cells
whose direction contradicts their window's mechanism found four retention
cells rising, and confirming them needed zero builds — the gate's own
measurement artifact from main's scheduled run (33989581881,
d36a1af0c)carries
minor_cyclesandstep_cyclesper probe, which settles all four.The blocker:
10_store_receiver_across_allocis inert — filed as #9832minor_cyclescopied_objects/copied_bytes/freed_bytesheap_used_bytesfreed_bytes = 0rules out "a minor ran and found nothing" — no collectionran at all. Every other probe still collects (
minor_cycles1–22, ±1).That probe is the only cover for the stale-root-across-evacuation class
(#6970 / #9523), and its own header says dropping the allocating RHS "makes it
silently measure nothing". The +110.57 % is a consequence of nothing being
collected, not a retention regression — the same reading error that
12_large_live_set.heap_used_bytesinvited.So I did not write the pin. Pinning
minor_cycles = 0would permanentlybless an inert probe: a green gate that proves nothing, which is the exact
failure this suite's header warns about, and the exact thing you said would
destroy the gate's value. #9832 has the two possible orders and the decision I
should not make alone.
The other three rising retention cells — explained, not regressions
13_large_eden_survivors.heap_total_bytes+3.51 % — the probe also showscopied 0 → 377,569whilepromoted −28.7 %. That is a tenuring-policyshift: objects that were promoted straight out are now copied within the
nursery first. Fewer promotions, slightly larger nursery. Explained by the
mechanism, dominated by improvements.
07_array_grow_evacuate.heap_used_bytes+8.90 % /heap_total_bytes+9.09 % with
copied_objects −21.86 %— same family, and itscopied_bytesand
freed_bytescells already carry a documentedprobe_overridesexclusionfrom GC Ratchet cannot gate 07_array_grow_evacuate: copied_bytes/freed_bytes are nondeterministic across traced runs #9790 for precisely this promotion-order/packing interaction.
04_dead_after_deep_stack(+6720 % promoted_objects) and05_closure_capture(0 → 3,390 promoted) — both are survivorship rising onliveness probes, which is the shape that would hide a stale-root bug. But
both keep collecting (
minor_cycles6→7 and 3→3), both showfreed_bytesstill falling (−12.6 %, −17.2 %), and the absolute magnitudes are tiny
(149 KB, 122 KB). Consistent with more objects surviving per cycle because
they are smaller — the same schedule shift as the other 17. I would not call
these regressions, and I would note them in the pin rather than measure
further.
Evidence grades, ready for whoever writes the pin
minor_cycles/step_cyclesfrom CI's own artifactOne thing the pin must say
The count went 43 → 48 → 43 in the last two days. This is not a ratchet
accumulating debt; it is several independent changes moving shared counters
with partial offsets. So the pin cannot be "the values as of today" without
saying that today is one sample of a moving set: it must record the SHA
(
d36a1af0c), the run it came from (33989581881), and that the gate's failureswere unwatched from 2026-08-18 to 2026-09-06. Otherwise the next person reads it
as a considered baseline rather than the artefact of a dark gate.
I am stopping here on this thread, per your steer — the retention-tail lead is
written up in
HANDOFF_keystroke_retention_tail.mdand is where I would gonext.
https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m
proggeramlug commentedon Sep 6, 2026
Probe repaired (#9833). The pin is now blocked on one more thing, and it is a tool bug (#9834)
#9833 fixes the inert probe:
minor_cycles0 → 9, sabotage-verified(removing the allocating RHS returns the exact inert signature), and
heap_used_bytesreturns 464,072 → 244,648 against a baseline of 220,384 —confirming that cell was never retention.
The sweep that came with it is the bigger find: six of fourteen probes were
pinned at
minor_cycles == 1, and one had already fallen to 0.01,02,03,09,11are each one allocation win from going silently inert, whilethis campaign lands allocation wins deliberately.
gc_ratchet.pyrefuses to pinminor_cycles < 1— it catches a probe that is already dead and blesses thestate that produces one. Detail in #9832.
Why I have not committed a baseline
I measured 7 repeats on
main@d36a1af0c+ #9833 — 14 probes, no inertprobe, no correctness failure — and
assemblerefused it:The measurement is fine.
distribution()is not idempotent under its ownrounding: the stored
samplesare rounded to 6 dp, the recordedspread_pctwas computed from the raw ones, and
validaterecomputes from the rounded ones.Three probes differed by one unit in the last place. It is fatal rather than
deferrable, and it is nondeterministic — a different run hits a different
subset. Filed as #9834 with the one-line fix and the missing regression
test.
So the pin needs, in order:
tool enforces this, so the 43-cell pin would have been rejected outright).
my local GC counters have matched CI byte-for-byte throughout this
investigation, so either host will do for the gated cells.
The pin's content is ready
Grades, unchanged from the previous comment:
minor_cycles/step_cyclesfrom CI's artifactAnd the sentence the pin must carry: the failing-cell count went 43 → 48 →
43 in the two days before it, so this is one sample of a moving set — pin
it with the SHA (
d36a1af0c), the run (33989581881), and the fact that thegate's failures were unwatched from 2026-08-18 to 2026-09-06.
I am stopping here and moving to the retention tail
(
HANDOFF_keystroke_retention_tail.md), per the standing steer. Happy to writethe pin the moment #9833 and #9834 are in.
https://claude.ai/code/session_014UZWia6L37DpA93VLtNK9m
proggeramlug commentedon Sep 6, 2026
Fixed by #9837, landed on
mainvia merge train #9867. The train merged as its own branch, so the close-keyword never fired — closing manually, verified on main.