Repository navigation
gc-ratchet has measured nothing since 2026-08-05: harness validation fails before the probes run (12_large_live_set spread 6768) #7554
Description
The GC ratchet has measured nothing since 2026-08-05
gc-ratchet fails on main in test_pinned_artifact_retention_is_deterministic:
AssertionError: 6768 != 0 : 12_large_live_set.heap_used_bytes was not
deterministic when pinned; it must not be in a gating family
That assertion lives in the "Harness unit tests and artifact validation" step, which runs before the measurement step. So the probes themselves — all twelve — have not executed on any branch since this broke. The job is red, and the thing it exists to measure is not running at all. That is CLAUDE.md's fourth failure mode ("the gate runs but its subject never did") in its most expensive form, compounded by gc-ratchet not being a required context, so the red blocks nothing either.
It has failed on every completed main run since 2026-08-05T15:07Z, which is the window in which #7446 re-pinned the artifact.
The assertion is right; the artifact is unfit
This is not an over-strict test. benchmarks/gc_ratchet/tolerances.json justifies gating heap_used_bytes on exactly this basis:
"Observed spread 0.000% over 21 runs on all 8 probes: bit-identical. The band is therefore pure anti-brittleness margin … not a noise allowance."
The pinned artifact contradicts its own premise. Every other probe still holds it:
| probe | heap_used_bytes spread |
|---|---|
01_nursery_churn … 11_collect_at_depth (11 probes) |
0 |
12_large_live_set |
6768 |
and the samples show it is a single outlier, not noise:
[59943824, 59943824, 59950592, 59943824, 59943824, 59943824, 59943824]
^^^^^^^^ one run, +6768 bytes
heap_total_bytes on the same probe is spread 0 across all seven.
Two possible causes, and they need different fixes
- The re-pin captured a bad sample. gc-ratchet: re-pin baseline on the dedicated bench host (ratchets #7443's retention win) #7446 ran on the dedicated bench host; one run in seven landing 6,768 bytes high could be incidental (a lazily-created runtime global, an interned string) rather than a property of the probe. Fix: re-pin
12_large_live_set, ideally with more repeats. 12_large_live_setgenuinely became non-deterministic. 6,768 bytes on a 60 MB heap is 0.011% — small, but the gating premise is bit-identical, and the other eleven probes still are. If retention on the largest live set is now sample-dependent, that is a real finding about the collector and worth knowing on its own.
Distinguishing them is cheap: re-run that probe ~21× on a quiet host and see whether the outlier recurs.
Why this matters right now
Roughly fifteen GC-affecting changes landed today — layout installation at allocation (#7532), the shape-install memo (#7535), write-barrier gating (#7536), the tape leaving the old generation (#7553), plus a spill-path rooting fix (#7546). None of them were checked by the ratchet, because it has not measured anything since before they were written. Several carried their own PERRY_GC_DIAG A/Bs, which is why I merged them, but that is per-PR diligence rather than a standing gate.
Suggested acceptance
12_large_live_set.heap_used_bytesdeterministic across ≥21 runs on the pinned host, or the probe explicitly moved out of the gating family with the reason recorded next to it (the mechanism the assertion's own message points at).- The ratchet's measurement step observed green on
mainat least once, so we know the probes run. - Consider promoting
gc-ratchetto a required context after that first green — a new gate has never been green, and promoting it while red would block every open PR.
Activity
proggeramlug commentedon Aug 7, 2026
Experiment: it is cause 2 — genuinely non-deterministic
Ran
12_large_live_set21 times on the pinned quiet mini (load 1.8,main@52f7dae1f, release build), readingheap_used_bytesfromPERRY_GC_DIAG=1:So this is not a bad sample captured by #7446's re-pin. The probe's retention is genuinely sample-dependent, and the pinned artifact's single outlier in seven was simply the first time the harness happened to catch it. The gating premise in
tolerances.json— "observed spread 0.000% over 21 runs on all 8 probes: bit-identical" — was written when there were 8 probes;12_large_live_setis the 12th, and it does not share that property.Note the assertion is still doing exactly its job: it refused to gate on a metric that cannot carry a gate. The bug is that the artifact was pinned as if it could.
The suggested fix is not currently expressible
tolerances.jsonis keyed per metric per profile, not per probe:So "take
12_large_live_set.heap_used_bytesout of the gating family" — what the assertion's own message instructs — cannot be said today. The only available lever turnsheap_used_bytesgating off for all twelve probes, which would discard a genuinely bit-identical signal on the other eleven. That is too blunt to accept.Three options, in my order of preference:
probes: { "12_large_live_set": { "heap_used_bytes": { "gating": false, "rationale": "…" } } }), keeping the other eleven gated. Smallest change that makes the artifact honest and gets the probes running again.One more observation, offered carefully
Today's runs read ~42.6 MB where the pinned artifact records a median of 59.94 MB — about 29% lower retention. I am not claiming that as a win: the artifact was captured on the dedicated bench host and this was the mac mini, and I have not controlled for host. But if a re-pin on the correct host reproduces it, the ratchet has a substantial retention improvement to record from today's GC work (#7532, #7535, #7536, #7546, #7553), and that is worth capturing deliberately rather than discovering by accident later.
docs(gc-ratchet): record the measured cause of 12_large_live_set's sp…
docs(changelog): add the #7557 fragment
docs(gc-ratchet): point the override's evidence at the cause issue
proggeramlug commentedon Aug 7, 2026
#7557 adds the per-probe override and gets the probes running again. Four
things worth having here.
The probes run again, and the gate has real output. Full
measure+checkend to end on two machine classes — the pinned quiet host(
perry-macos, load 2.2,pinned_host) and a MacBook Pro under load 21(
shared_ci). Twelve probes each, all compiled, all ran, all passed the Nodeoracle diff, and retention plus the counters reproduced bit-for-bit across the
two hosts. Both arms report the same ten gating breaches, invisible since
2026-08-05. Most look like improvements wearing a two-sided band (
03and04shed 40–95% of their copy/promote work while retention fell 49% and 22%), but
two are not:
05_closure_captureretains +16.44% with every collector counterat +0.00% — same work, more retained — and
02_survivor_promotionis +2.77%on the same shape. The PR deliberately does not re-pin; those two want a look
before they are accepted.
A second hole, found on the way. Six of the twelve probes pin
minor_cyclesat 1 and the allowance floor is also 1, so a collapse from 1 to 0was
delta == -allowanceand scoredok. A collector that stops runningcopying minors — the largest regression this ratchet exists to catch — was
being reported as passing.
checknow asserts liveness instead of inferring itfrom the bands.
The non-determinism is a real finding: filed as #7558. It is not the
collector's steady state, it is the measurement. Every probe reads
process.memoryUsage()after an explicitgc(), and an explicitgc()runs afull mark-sweep with a forced conservative stack scan
(
[gc-scan-fallback] site=manual_collect automatic=false, on every run).Diffing two disagreeing traces shows the minors, tenuring decisions, step cycles
and every copy/promote counter matching exactly, with the sole difference in the
last collection's
freed_bytes. UnderPERRY_CONSERVATIVE_STACK_SCAN=offtheprobe reports 51,668,688 bytes on 8 consecutive runs, bit-identical, against
59,943,080–59,952,824 by default. So the variance is entirely false roots — and
the conservative scan is systematically retaining 8.28 MB, 16% of what that
probe reports as its retained heap. The eleven small probes stay bit-identical
because their live sets are one to two orders of magnitude smaller.
The ~29% retention win does not reproduce, and one premise was off. The
pinned artifact was captured on
perry-macos.fritz.box— the same Mac mini asyour 21 runs, not a separate bench host; the artifact's own
notescall it "thededicated bench host … which replaces the shared MacBook". So there was no host
difference to control for. Measured with the harness protocol:
12_large_live_set.heap_used_bytes5e236e6e2, 2026-08-05perry-macos52f7dae1f(the commit you read 42.6 MB at)be1fc80f8(main + #7557)perry-macosRetention on that probe has not moved at all across the 2026-08-06 batch.
PERRY_GC_DIAG=1does not change it (the harness's traced/untraced split stillholds) and neither does the auto-optimizer, so there is no 29% win to record —
whatever produced 42.6 MB was not this probe under the harness protocol at that
commit.
CI has a deep runner backlog and may not report on #7557; the validation above
is local and is what it rests on.
test(gc-ratchet): per-probe gating overrides so the probes run again (#…
test(gc-ratchet): classify a retention breach instead of guessing at …
test(gc-ratchet): classify a retention breach instead of guessing at …
docs(plan): sync to v0.5.1345 — the 2026-08-07/08 wave (#7607)
proggeramlug commentedon Aug 7, 2026
#7609 does the repair. One correction to the framing first, because it changes
what the fix had to be.
The preflight failure this issue describes was already fixed by #7557.
validate+ the 63 unit tests pass atmaintoday (verified locally, exit 0).But the gate is still not producing a verdict, and the outage is a week long, not
three days:
gc-ratchetonmain30686263856)mainruns, 15 cancelled, 0 greencheck, against a0.5.1280artifact that no longer describes the collectorThe preflight abort was an episode inside a longer failure. And because
gc-ratchetis not in branch protection's required contexts (they arelint, cargo-test, parity, compile-smoke, api-docs-drift, security-audit, conformance-smoke-complete), none of those 179 red runs blocked anything —hazard 2 sitting underneath hazard 4.
What #7609 changes
Fail open per cell. Defects now carry a scope.
artifact(unreadable,tampered) stays fatal in preflight;
probeandcelldefects demote theirsubject and are reported as failures, so
checkstill measures all twelveprobes and still names a regression elsewhere in the matrix. The suggested
acceptance here was "moved out of the gating family with the reason recorded"
— that mechanism existed (test(gc-ratchet): per-probe gating overrides so the probes run again (#7554) #7557) but the validator could still veto the
whole run before it, which is the actual hole.
Baseline re-pinned at
main26b9c9d59on this box, with per-cellattribution in the artifact's own
notes.03/04's counter collapse isperf(gc): break the survivor-promotion handoff livelock (#7592) #7594 + perf(gc): live-proportional collection budgets at both generations (#7592) #7596;
02and05's retention breaches are conservative-scan falseroots (
classifyevidence on gc: 05_closure_capture retains +16.44% with every collector counter at +0.00% — surfaced when the ratchet started measuring again #7559).Two things flagged rather than absorbed: gc: 12_large_live_set is 13.6% slower at 0.5.1346 than 0.5.1280 with every collector counter flat #7610 (
12_large_live_set.wall_ms+13.58%, unexplained, while other probes got faster) and gc: PERRY_GEN_GC_EVACUATE=0 moves zero cells on all 12 gc-ratchet probes — the knob's off-state is unexercised #7611
(
PERRY_GEN_GC_EVACUATE=0moves zero cells on all twelve probes — my firstsabotage attempt was vacuous and I am reporting it as such).
Also: #7596's accepted
12_large_live_set.heap_total_bytes+36% did notreproduce under the harness protocol on this host — 110,100,480 → 110,100,480,
+0.00%. Nothing was re-pinned for it.
On the last bullet of the acceptance
Still exactly right, and still admin-only. #7609 is the prerequisite: it is what
makes a green
mainrun possible for the first time since 2026-08-01. CI has adeep backlog right now, so the validation on that PR is local and says so.
test(gc-ratchet): fail open per cell so one bad cell cannot zero the …
test(gc-ratchet): re-pin the baseline at main 26b9c9d on the pinned h…
fix(gc-ratchet): fail open per cell, fail closed on the verdict; re-p…
proggeramlug commentedon Aug 9, 2026
Closed — the ratchet measures again. Repaired by #7609 (fail open per cell, fail closed on the verdict; re-pin), and the artifact has since been fully re-pinned twice more: #7657 (all twelve probes after the conservative-scan removal) and #7666 (a thirteenth probe).
Verified in the merged tree: 13 probes pinned,
validate --scope allstructurally valid, and a post-merge--checkon the pinned mini against currentmainreportsgc-ratchet: OKwith every cell in band — the run this issue said had not happened since 2026-08-05.Two structural repairs came with it, both worth more than the re-pin:
probe_overridesis now{}.12_large_live_set.heap_used_bytescarried a one-waygating: falsespent on conservative-scan residue; fix(gc): explicit gc() runs on precise roots — remove the forced conservative stack scan (#7558) #7657 removed the cause and the cell gates again. Reclaiming a one-way exemption is the strongest available evidence the cause was real rather than tolerated.copied_objects + promoted_objects > 0. fix(gc): explicit gc() runs on precise roots — remove the forced conservative stack scan (#7558) #7657 re-enabled the adaptive-tenuring seed, which drovecopied_objectsto 0 on two probes — the re-pin would otherwise have pinned that at zero and made the guard permanently false exactly where it last fired.Provenance is single and uniform again (#7652 closed), and #7668 added a rebase-stable
code_treefield so a future pin's commit being orphaned by the merge that ships it no longer loses the reader.Not closed by this: the
12_large_live_set.wall_ms+13.58% that the #7609 re-pin flagged as unexplained by any merged decision. That is #7610, which stays open and wants a bisect over 0.5.1280..0.5.1346.