Repository navigation
[Next.js/dylib] Warm App Route server retains ~1,100 async box cells per request: 113 KB/req live Rust heap, minor-GC pause grows 59 ms -> 0.5 s #8213
Description
Activity
Measured input from the #8208 side: async box cells account for roughly a quarter of the per-request growth, and the
post_in_usehalf cannot be them at all. Posting here rather than in a side channel so whoever picks this up has it.Box cells are 38–42 bytes, measured two independent ways
- Marginal RSS across
asyncpipeBATCHES 120 → 1200: 42.2 B/cell vmmaptag-240 (mimalloc) at BATCHES 1200: 37.8 B/cell
That is cell + registry + allocator overhead, not the payload alone.
At the ~1,100 cells/request reported here, that is 41–45 KB/request of the ~150 KB — about 27–30%.
#8208 removes that share, and it should be harvested in this workload: a warm event-driven server reaches an empty task queue between requests, which is the flush boundary the release path keys on. On
asyncpipethe same fix takes malloc residue from linear-in-batches to constant 65,915 across a 10× range (slope zero), peak RSS 148.9 → 79.7 MB,vmmaptag-240 141.1 → 71.6 MB.The
post_in_use5.4 → 39.7 MB is a different bug, and it is the larger halfBox cells cannot explain any of the arena growth, for two independent reasons:
- They are not in the arena. Box cells are
std::alloc, entirely malloc-side. In theasyncpipereproduction the arena sat at ~250 KB while 118 MB accumulated in mimalloc and no full collection ever fired — the tracked-malloc registry read 0 for the whole run. Arena occupancy and box-cell retention are disjoint accounting. - A retained cell holds
undefined. Post-perf(codegen): async-to-generator boxes are never cleared — completed activations retain 96% of asyncpipe's young generation #7933 a completed activation's cell contains no GC reference, and the control cells arei32/bool. So a retained box cell cannot keep a GC object alive even in principle.
A 34 MB rise in
post_in_useis therefore a second retention path, in the GC heap, that #8208 does not touch.Suggested next measurement
Split the two axes rather than treating ~150 KB/request as one number:
- malloc-side (
vmmaptag 240, or a counting#[global_allocator]) — expect ~45 KB/req, and expect it to go away with fix(async): release and reuse a completed plain-async activation's box cells (#7933 follow-up) #8208. - arena-side (
[gc-step] post_in_use,PERRY_GC_DIAG=1) — expect the remaining ~105 KB/req, which is the real subject of this issue.
If those two numbers come out as ~45 / ~105, the box-cell share is accounted for and the hunt narrows to whatever is holding GC objects live across requests. If the arena side is much smaller than ~105 KB/req, then something else is malloc-side too and #8208 is not the whole of that half either.
The per-root-source retained-BFS instrument described in the issue body is the right tool for the arena half — it attributes which root keeps the objects, which neither RSS nor
post_in_usecan.Caveats on these numbers
Measured on
asyncpipe, not on this fixture, so the 38–42 B/cell figure transfers but the ~1,100 cells/request does not — that is this issue's own measurement and I have not reproduced it. The conclusion that box cells are outside the arena is structural rather than workload-dependent.- Marginal RSS across
Status check, 2026-08-16 — not fixed, and mechanism (b) has a sharper edge than this issue records.
Mechanism (a), box cells (55% of the retained JS heap): unchanged on
main. PR #8208 is still open. The caveat in the issue body stands — its release skips closure-visible locals and Next's minified route code is closure-dense, so theasyncpipe"releases == allocs" result does not transfer automatically. Acceptance is still the fixture slope, before/after.Mechanism (b), the append-only Proxy registry: unchanged on
main(js_proxy_newstill pushes with no removal path). While confirming that, the registry turned out to fail hard, not just leak:The band is
[PROXY_ID_BAND_START, HANDLE_BAND_MAX)=[0xF0000, 0x100000)— exactly 65,536 ids — andjs_proxy_newnever bound-checkedPROXIES.len(). Id 65,536 encodes toHANDLE_BAND_MAXitself, whichaddr_class::is_proxy_id_bandrejects andaddr_class::is_above_handle_bandaccepts, so every classifier in the tree reads it as a dereferenceable heap address. Measured: 65,600 proxies created then read back segfaults (exit 139) at exactly the first out-of-band id,keep[65535].Because the registry is cumulative rather than a live count, the ~4 proxies/request measured here put that ceiling at roughly 16k requests — the warm route was already at 10,037 entries (15% of the band) after 2,478. So the retention documented above is not only an RSS curve; it terminates in a wild pointer.
Filed as #8218 and fixed in #8219: refuse to mint past the edge (catchable
RangeError, the #5067 trade) and makedecode_proxy_idagree withaddr_classabout what a proxy id is. That makes the failure safe and named — it does not close this issue. The ceiling exists because slots are never reclaimed.On the durable fix for (b). Having looked at what it takes: reclamation requires the collector to observe proxy-band payloads in traced slots, because a proxy handle is an id-band value rather than a heap object and there is no death signal today. The cheap-looking shortcuts do not work:
- Deciding an entry is dead from its target's or handler's reachability is circular — the entry is what keeps them alive.
- Backing a proxy with a shadow GC object gives it no referents, so it dies immediately; making the handle a real heap object drops the band's whole purpose (generic object dispatch would silently read fields off the proxy instead of trapping).
- Widening the band buys a linear factor against an unbounded leak, and
HANDLE_BAND_MAX's own doc requires auditing everyis_handle_bandcaller before it moves.
So it is observation or nothing, and observation has a correctness cliff: under-approximate the live set by one holder and a live proxy is reclaimed, after which
lookupstops recognising it. That needs (i) hooks in each of the three funnels the collector actually visits slots through, (ii) pruning restricted to full traces, where the whole heap is scanned, and (iii) something covering a proxy id held only in a native frame across a trap call. Worth doing, but it is its own piece of work, not a follow-up commit — and it should land after #8208, so the box-cell share stops masking the measurement.- changed the title
[-][Next.js/dylib] Warm App Route server retains per request: ~1,100 box cells + 4 immortal Proxy entries each, 113 KB/req live Rust heap, minor-GC pause grows 59 ms -> 0.5 s[/-][+][Next.js/dylib] Warm App Route server retains ~1,100 async box cells per request: 113 KB/req live Rust heap, minor-GC pause grows 59 ms -> 0.5 s[/+]on Aug 16, 2026 The retention has a user-visible consequence, not just a footprint one — measured
From an in-flight 300-pass warm soak on the native backend, matched tree
183d30c53, default GC, no knobs (the #8163 close-condition run; 211/300 passes at time of writing). Posting now rather than at the end because it bears on whether this is scoped as a footprint defect.It isn't. A warm process gets slow enough to fail requests outright.
Eight of the run's requests failed with
UND_ERR_HEADERS_TIMEOUT— undici's client-side headers timeout, i.e. the caller gave up and got nothing. Not a wrong answer, not a slow-but-correct answer: a failed request from the client's point of view. The server's own log is clean at those moments (TypeError: value is not a function0,E180 failed to pipe response0, empty-bodyUnexpected end of JSON input0 across 6,267 copying minors), so this is purely the process being unresponsive.All eight are in a single pass, and that pass is an outlier by 7.7×:
pass 200: 84 minors pass 201: 84 minors pass 202: 84 minors pass 203: 644 minors <-- all 8 timeouts here pass 204: 84 minors pass 205: 84 minorsSo the failure mode is a collection storm, not chronic slowness. One pass ran 644 copying minors and starved the event loop long enough that 8 of its 21 concurrent requests blew a default HTTP client timeout.
The steady state that produces it:
at pass ~1 at pass ~203 live heap ( post_in_use)5.0 MB 82.7 MB copying minors per pass ~0.2 84 RSS — 545 MB And the yield is the damning number: a representative minor at pass 203 reports
pre_in_use=82,771,712 post_in_use=82,732,720 sweep_freed=38,992— 39 KB reclaimed against 82 MB live, ~0.05%. The collector is running constantly and recovering essentially nothing, because the retained set is retained, not garbage. That is what turns a memory graph into a throughput collapse: every collection pays a full root-scan and mark cost over a live set that only grows.Why the existing arms cannot see this. A 10-pass cold-start process never leaves the small-heap regime (
post_in_use5.2 → 5.6 MB, ~2 minors), so nothing in the fixture's default coverage — or in either transport's cold-start arms — reaches the knee. It requires one warm process past ~120 passes.I'm not arguing this must block #8040; that's a scope call for whoever owns it. I am saying the grounds should be accurate: this is not "a footprint/throughput defect rather than a correctness one". Requests fail. If #8040's DoD says anything about hangs or served requests, this is inside it, and if it's tracked separately it should be tracked as a defect with user-visible failures rather than as an RSS number.
Final counts and the full cadence curve when the soak completes.
Corroboration on a second host, and the sweep yield is a decay curve rather than a constant
From the native-backend warm soak now running on the bench mini (matched tree
183d30c53, default GC, no knobs, noPERRY_LLVM_INPROCESS— the same arm as the previous comment, continued on a different machine after the original host ran out of disk at pass 265).Two figures have been quoted for the sweep yield in the saturated regime — ~0.15% and ~0.05%. Both are right; they are the two ends of the same curve. Computed over 3,151
[gc-step]records at pass ≥ 150, so this is a distribution, not a sample:sweep_freed / pre_in_usemedian 0.064% mean 0.071% min / max 0.014% / 0.255% And it decays monotonically with depth:
pass 150 (first saturated): pre_in_use=51,183,184 post_in_use=51,092,344 sweep_freed=90,704 -> 0.177% pass 168 : pre_in_use=61,100,448 post_in_use=61,054,752 sweep_freed=43,424 -> 0.071% pass 187 : pre_in_use=72,426,472 post_in_use=72,391,400 sweep_freed=35,072 -> 0.048%The absolute bytes reclaimed fall (90.7 KB → 35.1 KB) while the live set rises (51.2 MB → 72.4 MB), so the yield roughly halves every ~35 passes. Extrapolating the trend, it does not asymptote to a useful floor within the range measured.
The pacing is already maxed out. Every one of those records also carries
pct=0% step=1073741824→1073741824— the step is pinned at its 1 GiB ceiling and the reclaim percentage rounds to zero. The collector is running at maximum stride and recovering essentially nothing, because the retained set is retained rather than garbage. That is the mechanism behind the collection storms: there is no configuration in which this workload's collector "catches up", so the only outcomes available to it are more frequent collections and longer pauses.Three independent measurements — two hosts, and the text and native backends — now agree that past the knee these collections are near-pure overhead. Taken with the storm at pass 203 (644 collections against a steady 84, where all 8 client timeouts landed), the picture is a throughput collapse with user-visible request failures, not a footprint number.
Scope note, stated as a correction to my own earlier advice: I had been arguing this issue was out of scope for #8040 on the grounds that it was "footprint, not correctness". On this evidence that framing was wrong. Whether it blocks #8040 is the maintainer's call, but it should be made knowing that a warm process in this regime fails requests outright, and that the collector has no remaining headroom to absorb it.
Independently and importantly: no #8163 residual signature has appeared in this run — empty body, host
TypeError: value is not a function, andE180are all still 0 at pass 187 with 3,381+ copying minors. The retention above is a separate defect from the rooting bug that was fixed; this comment is not evidence against #8220.- added a commit that references this issue
on Aug 17, 2026 The collection storms are stochastic in placement — one clean run proves nothing about them
Two 300-pass native-backend warm soaks, same tree
183d30c53, same instrument, same prebuilt-artifact lineage, on two different hosts:host passes copying minors storm passes UND_ERR_HEADERS_TIMEOUTA 265 (ended on disk exhaustion) 10,995 203 (644 minors vs a steady 84) and 238 12 B (bench mini) 300 13,089 none 0 Host B ran more passes and more collections and saw no storm at all. So the storm is not a deterministic function of pass index or live-heap size — the trigger is probabilistic. Two consequences worth stating explicitly:
- Do not conclude from a single clean run that a storm fix worked. Host B's run would have been read as "no timeouts, therefore no latency problem" if taken alone. Any fix here needs either repeated runs or a deterministic trigger, not one green soak.
- The severity is not bounded by what has been observed. A storm is 7.7× the steady collection rate landing inside a single pass; nothing measured so far says 644 minors is the ceiling.
The underlying steady state remains as described in the previous comment — sweep yield decaying (median 0.064%, 0.177% → 0.048% with depth) while
pct=0%and the pacing step sits pinned at its 1 GiB ceiling. Host B's live heap grew 5.2 MB → 123.4 MB over its 300 passes.Both runs are clean on the #8163 residual signatures (0 empty bodies, 0 host
TypeError: value is not a function, 0E180across 24,084 collections), so none of this is evidence against #8220 — the retention and its latency consequence are a separate defect from the rooting bug that was fixed.- added a commit that references this issue
on Aug 17, 2026
Serving the pinned #8034 production App Route fixture (release layout: providers + app dylib from clean
main3c95020f8, C host), one warm process's RSS climbs 160 MB -> 510-620 MB and[gc-step] post_in_useclimbs 5.4 -> 39.7 MB over 100 verifier passes (2,100 requests) with no plateau — first noted in the #8040 DoD audit and #8163's 2026-08-16 comment ("it wants its own home"). This issue is that home, with the diagnosis. All numbers measured 2026-08-16 on the bench mini (M1, 8 GB), 21 requests/pass (20 concurrent GETs + 1 POST viaverify.mjs),PERRY_GC_DIAG=1(+PERRY_GC_TRACE=1where noted). Logs:~/perry-bench.noindex/logs-rss/on the mini; throwaway instrumented providers (counting#[global_allocator]wrapper + per-root-source retained-BFS, runtime crate only) in~/perry-bench.noindex/build-rss-diag/, patch in the session scratchpad — not shipped.1. The growth is per-REQUEST (~150 KB/req), not per-connection (~0.6 KB/conn)
verify.mjsloop (21 fresh conns/pass)http.Agent {keepAlive, maxSockets:1}, sequentialOne keep-alive socket reproduces the stock slope exactly, and connection churn alone is noise-level. ext-http's per-connection tracking is clean (its GC scanner exposes only 10 live pointer roots at pass ~92).
2. The retained heap is real, rooted retention — 11 full collections reclaimed 0.00 MB
post_in_useclimbs 5.4 -> ~50.4 MB over the first ~50 passes, then creeps ~4 KB/cycle. Across two 140-pass runs, all 11 fulls (arena_bytes+old_gen_bytesmajor pacing) reclaimed nothing from the arena:The fulls do clear the malloc-object registry (97,950 -> 400 objects at one full — almost all
GC_TYPE_REGEXPheaders; 12.6 MB allocated / 12.2 MB freed over a run), so RegExp churn collects fine and is not the retainer.3. Off-heap: 100% of the RSS delta is mimalloc-owned VM, and most of it is LIVE Rust heap
vmmap --summary(one process, pass 0/10/50/100):Memory Tag 240(mimalloc,os_tag=240) resident grows 95.0 -> 225.6 -> 421.8 -> 526.3 MB; libsystem MALLOC zones stay ~0.5 MB; stack/__DATA/__PERRY_GCMAPflat.The counting-allocator build decomposes Tag 240 (values at GC cycle 1 ≈ pass 2 vs cycle 33 ≈ pass 119, ~2,450 requests apart,
rss-idiag-host.log):The committed-but-free overlay is real but reclaimable: in a 400-pass run, when a concurrent build put the host under memory pressure, RSS fell 710 -> 405-413 MB in ~2 passes with no behavior change, then re-inflated when pressure eased — macOS MADV_FREE lag + fragmentation, the same mechanism PR #8208 measured on
asyncpipe_big.Live-histogram growth by size class (cycle 1 -> 33):
hashbrownusizetable at 8M-bucket capacity — the box-registry/GC-census family; not individually named by this instrument.4. Which structures retain (counted, not guessed)
Side-table growth per request (cycle 1 -> 33, all monotone, none per-connection):
BOX_REGISTRY(+i32/bool)PROXIES(proxy.rs)fn.x = ...)Per-root-source reachability over the live heap (retained-BFS, cycle 20 ≈ pass 92, live = 36.78 MB / 548k objects; independent walk then marginal in the same order):
crate::r#box::scan_box_roots_mut(2.10 M slots, 291,736 live ptrs)crate::proxy::scan_proxy_roots_mutSo the "retained requests" decompose into two mechanisms:
(a) Async-activation box cells — the dominant term in all three symptoms. Every request leaks ~1,100 8-byte box cells whose registry entries are permanent (#7933 cleared values but kept cells+entries). They account for 55% of the retained JS heap (stale values in cells the clearing pass could not touch), ~25 MB of cells + tens of MB of registry table on the Rust heap, and 96% of the minor-GC root-scan slots — which is why minor pause grows 59 ms -> ~460-550 ms over 140 passes (scan slots 261k -> 6.5M; the collector rescans every box ever allocated, every minor). In a long run this compounds: once the trigger regime tightens, the host runs ~70 collections/pass at ~0.5 s each and throughput collapses ~15x.
(b) The Proxy registry is append-only and roots its targets forever.
js_proxy_newpushes into the thread-localPROXIES: Vec<Option<Box<ProxyEntry>>>(id = v.len(), no slot reuse); nothing but explicit revocation ever detaches an entry (the onlyslot.take()is#[cfg(test)]), andscan_proxy_roots_mutmarks every entry'starget+handleras strong GC roots each cycle. A Perry proxy is an id-band handle, not a heap object, so the GC has no death signal for it — in Node an unreferenced Proxy is ordinary garbage. Next creates ~4 proxies/request on this route (headers/cookies/mutableCookies/searchParams adapters;chunks/430.jsalone has 34new Proxy(sites), each pinning its request's header/cookie graph: ~10k immortal proxies by 2,500 requests, 2.9 MB retained at ~1,900, growing without bound. The symbol/descriptor side-table growth is largely downstream of (a)+(b): #8174 gave owner-keyed tables death stories, but these owners never die.Does PR #8208 cover this?
Mechanism (a), yes —
fix/async-state-rss-accumulationreleases + pools exactly these cells, de-registers them, and its[box-stats]gate asserts constant residue. If it works on this workload it should remove the box share of the Rust heap, the box 55% of the JS-heap retention, and the pause growth. Caveat that must be A/B'd on this fixture: its release only covers closure-unobservable locals and skips a body entirely when the escape scan is poisoned; Next's minified route code is closure-dense, so the asyncpipe numbers (releases == allocs) do not transfer automatically. Acceptance for this issue should be the fixture slope: RSS and[rss-diag-tables] box=over 100+ passes, before/after.Mechanism (b) is NOT covered by #8208 and needs its own fix: give Proxy entries a lifetime — either back proxies with a real GC heap object (death hook drops the registry slot) or key the registry entries weakly so an unreferenced proxy id can be collected. Not a small change; the registry is load-bearing for
lookup()misclassification defense.Not covered by either: the ~100-215 MB mimalloc committed-but-free overlay (returned only under OS memory pressure). Once (a)+(b) stop the churn's live tail,
purge_delay/page-retirement tuning can be evaluated separately.Reproduction
~/mini-batch-rss.sh <tag> 100 PERRY_GC_DIAG=1(mini; runsverify.mjsN times against one warm host, records per-pass RSS + snapshots).~/mini-batch-rss2.sh <tag> <stock|ka1|ka20|conn|partial> <N> [env..]withPROVIDERS=~/perry-bench.noindex/build-rss-diagandPERRY_RSS_DIAG=1 PERRY_RSS_DIAG_RETAINED_EVERY=20.Refs: #8040 (DoD audit that surfaced this), #8037 ("do not retain a request" acceptance bullet), #8163 (separate defect, same runs), #7933 / PR #8208 (box-cell accumulation and its in-flight fix).