Repository navigation
gc: asyncpipe collects at 1200-1650 ns/object, including a 122 ms minor that handled zero objects #7915
Description
Activity
- added a commit that references this issue
on Aug 12, 2026 Lead from #7909: the per-cycle cost is the ROOT SCAN, and the denominator is wrong
Investigating #7909 I had to unblock
asyncpipe's collector (a starved budgeted
incremental cycle was rejecting every precise safepoint via
gc_safepoint_moving_minor'sbudgetedentry guard). That produced two real
copying minors on the 1× program, traced withPERRY_GC_TRACE=1 PERRY_GC_DIAG=1:minor #1 minor #2 pause_us61 400 134 076 copied_objects153 056 0 copied_bytes12 053 704 0 dominant phase copying_nursery55 682 µscopying_nursery132 791 µsruntime_mutable_scanners.registered_scanners82 82 runtime_mutable_scanners.slots_scanned187 670 234 837 runtime_mutable_scanners.pointer_roots171 068 218 455 runtime_mutable_scanners.rewritten_slots85 404 109 091 layout_scans.pointer_slots_read263 828 536 873 Minor #2 is a 134 ms collection that copied zero objects and zero bytes —
longer than the entire 127 ms program, with no per-object work at all. Its time
is 82 registered runtime mutable root scanners exposing 218 455 pointer
roots, all scanned and roughly half rewritten.That is the fixed overhead this issue is looking for, and it says the
ns/object figure has the wrong denominator: the work is per registered
root, not per copied object.asyncpipeallocates ~95 600 promises and
~24 000 closures; each registry entry is reached through a registered mutable
root scanner, so the root set of a young collection is effectively the whole
heap — young collection stops being O(survivors) and becomes O(all registered
objects).Two consequences worth checking before optimising the copy loop:
- a minor's cost here is independent of how much it reclaims, so scavenging
more often (a smaller nursery cap) multiplies the fixed cost with no
corresponding win — consistent with the nursery-shrink result already
measured on this program; - on
asyncpipe, collecting is worse on every axis: 1927 → 2918 M
instructions and 48.9 → 76.7 MB RSS. The scavenge nursery cap, whose purpose
is to hold RSS down, is counterproductive on this shape.
Full trace, counters and reproduce commands:
gc-handoff/GC7909-NOTES.md§4.1.
The[gc-incremental]diagnostic that made the stall visible is PR #7923.- a minor's cost here is independent of how much it reclaims, so scavenging
Both halves of the premise are counter artefacts — and the real cause is bigger
Measured on
origin/main(ebefba51a),asyncpipeat 240 batches,
PERRY_GC_TRACE=1 PERRY_GC_DIAG=1. Draft PR #7931 ships the instrument this used.1. The "minor that handled zero objects" relocated 200 201 of them
[gc-copy-minor] ran … survival_permille=895 copied_objects=0 copied_bytes=0 promoted_objects=200201 promoted_bytes=15860160 …copied_objectscounts survivor-space copies only, so on a promoting cycle it
is structurally zero. 101 ms / 200 201 objects = 505 ns/object — not a fixed
per-cycle cost. Same shape as CLAUDE.md's "a gate must assert its subject was
live": a counter that cannot be non-zero on the path being measured.2. "82 scanners over 218 455 roots" is one scanner
Per-scanner attribution (
[gc-scanner-profile], #7931), minor #1:scanner µs slot visits ptr roots rewrites crate::box::scan_box_roots_mut18 452 186 776 186 760 93 380 scan_closure_dynamic_props_roots_mut745 4 458 4 196 2 065 descriptor_state::scan_descriptor_roots_mut523 342 342 171 promise_mutable_root_scanner43 12 10 5 other 78 scanners < 25 each 92 % of the pointer roots and 89.5 % of root-scan time is one registry. The
suspected promise registry visits 12 slots. (Counts sum both passes; #7645
already skips the third.)3. …and that 89.5 % is the object trace, not root-scan overhead
A cheap slot visit costs ~4 ns (
intern_table_mutable_root_scanner: 16 384 slots,
1 144 pointer roots, 68 µs). The box scanner costs 98.8 ns/slot because
essentially every box slot is a from-space pointer, so visiting it in
CopyingMarkevacuates the object it names — 93 380 of minor #1's 156 236
evacuations come straight out of box roots, and 98.8 ns/pointer-root is the same
price as a measured full evacuation (123 ns/object).The fixed per-root overhead this issue targets is ~4 ns × 265 000 ≈ 1 ms of a
65–101 ms pause. All three suggested directions (generational filtering, dirty
tracking, cheaper representation) share that 1 ms budget. I built the third —
slab-allocated box cells so the scan streams instead of chasing scattered
mallocs in hash order — and it measured +0.03 % instructions, −1.8 % cycles, no
RSS change, overlapping pause distributions over 7 runs each. Reverted; the
mechanism above explains why it cannot pay.4. The real cause: 96 % of the young generation is retained by dead activations
One-env-var probe (temporary),
scan_box_roots_mutskips marking but still
rewrites:control box roots do not mark minor #1 young_survival_permille695 24 minor #1 copied_objects156 236 6 634 (−95.8 %) minor #1 freed_bytes5 456 696 17 451 568 (3.2×) minor #2 200 201 promoted, 15.9 MB never happens exit 0, correct output 134 (SIGABRT) asyncpipe's real live young set is ~6 600 objects. Everything else is alive
only because a completed async activation's boxed locals are still GC roots —
the async-to-generator transform boxes every body local of everyasync
function, and nothing ever frees or clears one, so every local of all 48 000
completed activations is a permanent root. At 6 634 live objects a minor costs
~0.8 ms instead of 62 ms.For context,
asyncpipeis a churn program by construction (each batch is built,
folded into six numbers and dropped) yet survives at 695–895 ‰, against 0–4 ‰ for
every other churn program ingc/promote_in_place.rs's own census.The SIGABRT is the control, not a failure: not marking is unsound, which is why
the fix is clearing an activation's boxes at its terminal state, not skipping
them. Clearing keepsBOX_REGISTRYmonotonic and the address valid — so
is_registered_box_ptr's perry#4898 rejection and the positive pointer cache are
untouched — while removing the retention. It is a codegen change in the async
transform and deserves its own issue.Full write-up with reproduction:
gc-handoff/ROOTS-NOTES.md§2–8.Suggested disposition
Keep this issue open only if it is re-scoped to the ~1 ms of genuine root-scan
overhead; the 60× lever is the activation-local retention above.- added a commit that references this issue
on Aug 12, 2026
Summary
Measured while establishing the per-cycle GC picture for the retain cluster (
gc-handoff/RETAIN4-NOTES.md):asyncpipecollects at 1,200–1,650 ns/object when it collects at all — including a 122 ms minor that handled ZERO objects.For scale, the retain cluster's in-place promotion runs at 8.9 ns/object and a real evacuation of 245,752 objects costs 30.2 ms. So
asyncpipe's collections are two orders of magnitude more expensive per object, and one of them did no per-object work whatsoever yet still cost 122 ms — longer than the entire program's 127 ms runtime.A collection whose cost is independent of the number of objects it processes is a fixed overhead being paid per cycle, not collection work. That points at root scanning, side-table maintenance, or a per-cycle setup/teardown cost that scales with something other than the live set.
Relationship to #7909
#7909 covers the incremental old-gen work
asyncpipepays while running zero cycles. This issue is different: it is about the cost of the cycles that do run, once the workload is scaled far enough to trigger them (pergc-handoff/ASYNC2-NOTES.md, 120 batches → zero cycles, 240 → two copying minors, 1200 → GC-dominated).The two may share a root cause. They should be investigated together but neither should be closed by the other without evidence.
Acceptance
Note on measuring this
gc-handoff/apps/asyncpipe_big.tsis not a valid scaled proxy forasyncpipe— the zero-GC property is a knife-edge of the exact batch count. Any scaling used to trigger collections must be reported explicitly, along with confirmation that the mechanism under study survived the scaling.