Repository navigation
coop/in_process: second Next app dies with TypeError: value is not a function (GC rooting, multi-heap) #8546
Description
Activity
Re-tested on
0.5.1516(3885ba491) — still fails, and #8552 is not the causeI expected #8552 ("register mutable root scanners per thread") to fix this: a
process-global "scanner already registered" latch would mean the first heap
registers its scanners and every later heap silently skips, leaving app #2's
roots unscanned. That would have matched the symptom exactly.It does not. Re-ran the full chain on main
3885ba491(which includes #8552),
with provider provenance verified by content — the removed
ERR_PERRY_PATH_MODULE_THREADstring appears 0 times in the packaged
libperry_runtime.dylib, and nowhere in main's source, so the check is
meaningful rather than vacuous.Result is byte-identical to the
0.5.1514run:application library preloaded on dedicated Perry thread x2 (both apps init) rejected duplicate path-module initializer 0 bench-000 -> 200 {"runtime":"next","iterations":100,"checksum":3726872593} bench-001 -> 500 coop: 500 deployment dispatch failed ERROR deployment dispatch failed deployment=bench-001 error=Runtime(invoking deployment HTTP handler: handler promise rejected: )Which app fails varies between runs (it was bench-000 in the harness run and
bench-001 in the manual daemon run of the same build), consistent with "whichever
is second", not with anything specific to a deployment.What this rules out
- per-thread scanner registration latches (fix(gc): register mutable root scanners per thread #8552)
- the path-module registry (fix(runtime): make the path-module registry per-heap so one process can host several apps #8528) — both apps register their own initializers,
zero rejections - app load / module init — both complete, with timings logged
Still the best hypothesis
Some other process-global runtime/stdlib state assumes one application per
process. The per-request registries remain the suspects, several of which are
Mutex<HashMap<usize, ...>>and UNCOVERED by the holders census:
REQUEST_REGISTRY,BLOB_REGISTRY,FETCH_RESPONSES,TRANSFORM_PAIRS,
STREAM_HANDLES, plusbuffer/own_props.rs'sPROPSand
buffer/header.rs'sEXTERNAL_*_REGISTRY(keyed by RAW heap address).The blocking next step is diagnosability
handler promise rejected:carries an empty reason. Until that surfaces the
underlying JS error there is nothing to attribute, and every further step here is
guesswork. I'd fix that first — it is likely a small change at the rejection
site, and it converts this issue from "one of ~7 suspects" into a named failure.The rejection reason, finally:
TypeError: value is not a functionCoop was reporting an empty reason, and it turned out to be reporting it
wrongly. Two bugs on the host side, now fixed locally (PR: PerryTS/coop#14):read_perry_stringtrusted a contract that does not exist. Its comment says
js_get_string_pointer_unified"Returns 0 if not a string", but that function
deliberately returns the payload for a POINTER_TAG (0x7ffd) value too
("used for cross-module returns",value/nanbox.rs:296-299). Handing it an
Errorobject yields a non-null pointer that is not aStringHeader, and
reading through it is a wild read — it fabricated a one-character reason
("\t") out of unrelated heap bytes.- The hex fallback only fired for
None, so the garbage string took the
"success" path and printed as if it were the reason.
With the tag checked and
js_jsvalue_to_stringused for non-strings, the real
error appears:handler promise rejected: TypeError: value is not a function [raw=0x7ffd0007ca3bf5f0 tag=0x7ffd]What that changes
This is not a registry logic bug.
TypeError: value is not a functionis the
signature of a GC rooting failure — CLAUDE.md: "A 'GC value live but not rooted
across a collection point' bug is invisible at collection time … It surfaces one
or more cycles later, in a different function, asTypeError: value is not a function."So: in a two-application
in_processdaemon, something live in the second
application's heap is being collected.The reproducibility narrows it further
The failure is deterministic — the second application to be dispatched fails
every run (which application that is varies, so it tracks ordering, not identity).
Per the rule of thumb in the same docs, an unrooted register goes bad only
when a collection lands in its window and is therefore intermittent, while an
unrooted table goes bad at collection #0 and stays bad. A perfectly
reproducible failure points at a table, not a register — which is exactly the
process-global-registry hypothesis, now with evidence behind it rather than
suspicion.Suggested instruments
The docs name the tools for this class, and they apply directly:
PERRY_GC_SCHEDULE_SEED=<u64> PERRY_GC_PROTECT_FROMSPACE=1to turn the stale
deref into an immediate precise faultPERRY_GC_PROTECT_FROMSPACE_DEPTHraised, since the value may cross many
collections between its last valid observation and the stale usescripts/gc_runtime_root_holders.py— the per-request registries that remain
UNCOVERED are the shortlist:REQUEST_REGISTRY,BLOB_REGISTRY,
FETCH_RESPONSES,TRANSFORM_PAIRS,STREAM_HANDLES, plus
buffer/own_props.rs'sPROPSandbuffer/header.rs'sEXTERNAL_*_REGISTRY
Note #8552 (per-thread scanner registration latches) is already in this build and
does not fix it, so the defect is not simply a skipped registration.- changed the title
[-]coop/in_process: second Next app loads but its handler promise rejects at dispatch (500)[/-][+]coop/in_process: second Next app dies with TypeError: value is not a function (GC rooting, multi-heap)[/+]on Aug 22, 2026 Reopening — the close was a keyword accident, the bug is still live
#8562 carried
Fixes #8546and auto-closed this at 04:47 on 2026-08-22. #8562 is a regression test for #8552's per-thread scanner latch. My comment five hours later (09:18) established that #8552 is already in the failing build and does not fix this; PerryTS/coop#15, opened the same morning, says the same. Nothing that landed since claims to address it.Where it stands after the two Coop-side fixes
- fix(worker): never report an empty handler rejection reason coop#14 — the rejection reason is no longer empty (the host was reading an
Errorobject as aStringHeader). - fix(worker): root the awaited handler Promise across perry_poll coop#15 — one real unrooted holder removed on the host side:
await_promisecached the raw Promise pointer acrossperry_poll.PERRY_GC_SCHEDULE_SEED=12345 PERRY_GC_SCHEDULE_RATE=0.5 PERRY_GC_PROTECT_FROMSPACE=1faulted on it directly (obj_type=5=GC_TYPE_PROMISE, "the faulting instruction IS the stale use"); rooting it throughjs_ffi_root_scope_enter/js_ffi_root_push_nanbox/js_ffi_root_get_nanboxtook the same seed from FAULT to 0 faults with 107 copying minors still running (anti-vacuity).
Both are merged. Two Next apps
in_processstill fail, but the failures are now named:bench-000: TypeError: _onUserlandLoaded is not a function bench-001: TypeError: value is not a function_onUserlandLoadedis a Node bootstrap internal, so at least one more holder of the same class exists — a JS value cached across something that can run JS — and it is not on the Coop host side anymore, which only touches the Promise. The process-global stdlib registries from the original report remain the shortlist (REQUEST_REGISTRY,BLOB_REGISTRY,FETCH_RESPONSES,TRANSFORM_PAIRS,STREAM_HANDLES,buffer/own_props.rsPROPS,buffer/header.rsEXTERNAL_*_REGISTRY).Measured against Perry
0.5.1512(fc6b3378, Coop's current pin); next step is re-running the from-space-protect hunt against currentmainonce Coop's pin and the pairedrusqlitebump land.- fix(worker): never report an empty handler rejection reason coop#14 — the rejection reason is no longer empty (the host was reading an
- added a commit that references this issue
on Aug 27, 2026 Reproduced on 0.5.1516 with coop#14/#15 in place — and the rule is now precise: only the app whose init finished LAST works
Coop
mainat3e7180e(worker has #14's reason reporting and #15's Promise rooting), Perry0.5.1516(3885ba491) — chosen because on0.5.1519the fixture does not load at all (perry#8882). Harness patched locally to warm every app before judging (it used to stop at the first 500, which is what hid the pattern) and to relay the daemon's stdout/stderr live.run init-completion order result 2 apps, preload_concurrency=1000 → 001 000: 500, 001: 200 3 apps, preload_concurrency=1000 → 001 → 002 000: 500, 001: 500, 002: 200 3 apps, concurrent 000 → 002 → 001 (by log order) 000: 500, 002: 500, 001: 200 2 apps, concurrent, inits finishing 0.5 ms apart 001 → 000 both 500 So it was never "the second one dispatched": every application except the last one to finish init fails, whatever order they are dispatched in. Two inits interleaving closely enough break both. That is last-writer-wins on process-global state written during module init.
Every failure is
handler promise rejected: TypeError: value is not a function(onceTypeError: _onUserlandLoaded is not a function).What from-space protection says: nothing, and that is informative
PERRY_GC_SCHEDULE_SEED=12345 PERRY_GC_PROTECT_FROMSPACE=1atRATE=0.5 DEPTH=64and atRATE=1 DEPTH=256(382 safepoints / 216 scheduled collections in the RATE=0.5 arm): no fault, same 500. Before #15 the same seed faulted immediately on the un-rooted Promise (obj_type=5, backtrace ininvoke_buffer_handler), so the instrument is armed and #15 is confirmed as a real fix of a real instance. What remains does not deref retired from-space — the value it reads is live memory in the other application's heap, which the quarantine cannot object to. A cross-heap value, not a stale one.Where it dies
_onUserlandLoadedis Next's, not Perry's:AppRouteRouteModuleconstructsnew LazyModule(userland, (module1) => this._onUserlandLoaded(module1)), and the route bundle'suserlandis webpack's lazy() => __webpack_require__(3870)— evaluated on the first request, not at init. So the earlier-initialised app survives init, and dies the first time it resolves something lazily by name: the arrow's capturedthis(or the module it is handed) is the last-initialised image's, and_onUserlandLoadedis not a function on it.That narrows the suspect list from "any of ~7 per-request registries" to "process-global tables keyed by name that module init writes with
insert(last wins), whose values are heap pointers or ids used to resolve properties / modules / closures". In0.5.1516crates/perry-runtime/src/symbol.rs:116SYMBOL_REGISTRY: Mutex<Option<HashMap<String, usize>>>is one such (a string → heap pointer map, process-global while the path-module initializer registry from #8528 isperry_thread_local!); I have not yet confirmed it is the one. Next step on my side is a census of name-keyed process-global statics written during init; on Perry's side, making whichever it is per-thread like #8528 did for initializers is the shape of the fix.Harness changes (report every app; relay daemon output) are going into PerryTS/coop as a PR so this table is reproducible with one env var.
Mechanism: the class registries are process-global and keyed by compile-time class id — N identical images overwrite each other
crates/perry-runtime/src/object/class_registry/state.rs(checked on3885ba491and unchanged on41e8479a5):pub static CLASS_VTABLE_REGISTRY: RwLock<Option<HashMap<u32, ClassVTable>>> // methods/getters/setters by class id pub static CLASS_STATIC_METHODS: RwLock<Option<HashMap<u32, HashMap<String, (usize, u32, bool)>>>> pub static CLASS_METHOD_BIND_LENGTHS / CLASS_STATIC_METHOD_BIND_LENGTHS / REGISTERED_CLASS_IDS / CLASS_STATIC_ACCESSORS
js_register_class_methoddoesvtable.methods.insert(name, VTableMethodEntry { func_ptr, .. })— last writer wins. Class ids are assigned by codegen (next_class_idthreaded through lowering), so two dlopen'd copies of the same app dylib register the same ids with differentfunc_ptrs (each image's own code addresses). After the last image's init, every class of every earlier image dispatches into the last image's code: earlier apps fail at their first by-name resolution (Next's lazyLazyModule→this._onUserlandLoaded), the last one serves. That is exactly the table from my previous comment, and it explains the from-space silence — afunc_ptris not a heap pointer. (CLASS_SYMBOL_METHODS/CLASS_SYMBOL_ACCESSORSin the same file are alreadyperry_thread_local!; the path-initializer registry was made per-thread in #8528. These string-keyed tables were left behind.)Three experiments, one patch each on a local 0.5.1516 provider (app image unchanged)
registration policy serial 3 apps reading stock ( insert, last wins)000 ✗ 001 ✗ 002 ✓ last image owns every table instance methods first-wins only 000 ✗ 001 ✗ 002 ✗ methods from image 1, getters/setters/statics from image 3 — every vtable is a mix, nobody works every js_register_class_*/js_class_register_*entry point (24 of them) ignores registrations for a class id from any thread but the first to register it000 boots; 001 dies during its own init — TypeError: Cannot read properties of undefined (reading 'createBaggage')— and takes the daemon downlater images cannot even initialise on another image's tables: the tables are per-image state that every image both writes and depends on So the registries are not merely "polluted" by a second image — there is no ordering of writes to a shared table that works for more than one image. They have to be per image, and since Coop gives each application its own dedicated Perry thread, per-thread is the natural key — the same shape as #8528's fix for the path-initializer map.
Scope of the change
CLASS_VTABLE_REGISTRYhas 50 use sites acrossclass_registry/*.rs,proxy.rs,typed_feedback/guards.rs,value/dynamic_object.rs,object/{property_key,prop_plan,native_module,native_call_method,descriptors}.rs; the other five statics add ~45. Plus whatever backsjs_register_class_constructor/_flags,js_class_register_capture_values(object/class_constructors.rs),js_register_class_{id,name}(class_meta.rs),js_register_class_parent{,_dynamic}(parent_static.rs),js_class_register_static_field, theclass_meta_registry.rsfour,data_view_registry.rs, andsymbol/properties.rs'sjs_class_register_static_symbol— every one of the 24 entry points is keyed by class id. Aperry_thread_local!conversion, or a per-image registry handle selected by the loading thread, are the two shapes I see; the second keeps the RwLock code paths intact.Diagnostic patches were not committed anywhere; the reproduction is PerryTS/coop
main+ coop#18's harness (COOP_BENCH_APP_COUNTS=3 COOP_BENCH_PRELOAD_CONCURRENCY=1 COOP_BENCH_TRACE_DAEMON=1, path/api/benchmark) on any Perry that loads the fixture — i.e. 0.5.1516, until perry#8882 is fixed.Closed the loop on Linux too
With #8893 (per-image class registries) plus the two Linux-only defects that were hiding behind it — #8942 (promoted string constants collided across split modules under GNU ld) and #8948 (the compact stack-map rewrite dropped the
.hidden/.weakonDW.ref.perry_eh_personality, so every caughtthrowin a multi-module Linux program called a junk personality pointer) — Coop's three-appin_processtable on a fresh Ubuntu 24.04 box reads 200 / 200 / 200 (three trials; single app 200 as well), matching the macOS result recorded on PerryTS/coop#19. Fixture compile on 16 cores: ~6 minutes.Coop side: PerryTS/coop#19 (pin + ext wrappers built coherently), #20 (
-lmon the Linux app link) are merged; a final pin bump follows once #8942/#8948 land.
Summary
With #8528 merged, a Coop
in_processdaemon can load and initialize twoNext.js applications in one process — but the second application's HTTP
handler rejects at dispatch, so it serves 500 while the first serves 200.
Evidence
Measured on the quiet M1 mini, Perry
7c22189aa(0.5.1514), Coopin_process,two deployments of the same Next fixture. Provider provenance verified by
content, not by recorded metadata: the merged fix deletes
ERR_PERRY_PATH_MODULE_THREAD, andstringson the packagedlibperry_runtime.dylibfinds 0 occurrences of it (the string also appearsnowhere in main's source, so the check is meaningful rather than vacuous).
Both applications initialize cleanly:
Then:
The rejection reason is empty, which is itself worth fixing — an empty
handler promise rejected:gives an operator nothing to act on.What this is NOT
rejected duplicate path-module initializermessages in this run; both apps register their own initializerson their own dedicated Perry threads. (An earlier iteration of fix(runtime): make the path-module registry per-heap so one process can host several apps #8528 that kept
the initializer-address map process-wide DID produce 115 such rejections;
that is fixed.)
dispatched succeeds and the second fails.
Hypothesis
Some other process-global runtime/stdlib state assumes a single application per
process. Until #8528 the fail-closed thread guard made multi-app hosting
impossible, so none of that state has ever been exercised by two heaps. The
per-request registries are the obvious suspects — several are
Mutex<HashMap<usize, ...>>and are UNCOVERED by
scripts/gc_runtime_root_holders.py, i.e. nobody hasdetermined whether they hold heap pointers:
REQUEST_REGISTRY,BLOB_REGISTRY,FETCH_RESPONSES,TRANSFORM_PAIRS,STREAM_HANDLES(crates/perry-stdlib/src/fetch/mod.rs,streams/transform.rs), plusbuffer/own_props.rs'sPROPSandbuffer/header.rs'sEXTERNAL_*_REGISTRY(the latter keyed by RAW heapaddress).
Suggested first steps
is process-global.
Reproduction
Needs a Coop checkout whose
perry-main.lockpoints at a Perry with #8528, pluscoop-daemon'srusqlitebumped 0.37 -> 0.39 to matchperry-stdlib(they linkthe same native
sqlite3; see the companion Coop change). Thenscripts/prepare-next-benchmark.shand theresource_benchmarkharness withCOOP_BENCH_APP_COUNTS=2,COOP_BENCH_EXECUTION_MODE=in_process.