Repository navigation
[codegen] In-process LLVM backend fails the 5 biggest Next App Route modules on current main (silent per-module failure; regression in 3c95020f8..07c8040bf) #8228
Description
Activity
Attribution, from hitting this while building the #8163 residual A/B on merged main
b8d32ab19.The offending instruction is the module-init header-image compose, and Perry's own IR dialect parser has no case for it.
Error compiling module '.next/server/chunks/2.js' … with --backend llvm: native codegen unit 2/3 failed: unit 1: in line: %r7 = insertelement <2 x i64> <i64 207232172546, i64 0>, i64 %r6, i32 1: bad binary op `insertelement` operands: <2 x i64> <i64 207232172546, i64 0>, i64 %r6, i32 1- Emitter:
crates/perry-codegen/src/function.rs:586andcrates/perry-codegen/src/codegen/string_pool.rs:514, both"{} = insertelement <2 x i64> <i64 {}, i64 0>, i64 {}, i32 1". - Rejecter:
crates/perry-codegen/src/dialect/mod.rs:1014—bail!("bad binary op \{op}` operands: {rest}").insertelement` is not in the dialect's instruction table, so it falls through to the binary-op arm. - Introduced by
bf8fd868e— perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (perf(object): remove the derivable object_type and field_count header words (56 B -> 48 B) [HELD: #8157 refuted; footprint-coupled residual + new #8094 guard cost] #8122) (perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204).git grep -c 'insertelement <2 x i64>' 3c95020f8 -- crates/perry-codegen/srcreturns nothing; the same grep atb8d32ab19returns four files. That is inside this issue's stated3c95020f8..07c8040bfrange and narrows it to one commit.
All five failures are the same instruction, and all five are modules large enough to be split across native codegen units (
unit 2/3,4/6,2/2,8/10,3/4) — consistent with the compose only being round-tripped through the dialect parser on the multi-unit path.Two consequences worth recording:
-
The [Next.js/dylib] Add a pinned production App Route parity fixture and CI gate #8034/[Next.js/dylib] Full production App Route compatibility tracker #8040 App Route fixture cannot build its app dylib on merged main.
perry compile perry-host.js --output-type dylib --no-auto-optimize --no-cacherefuses to link with "5 module(s) failed to compile", and the failed set includesnext-server/app-route.runtime.prod.js— the route runtime itself — soPERRY_ALLOW_PARTIAL_CODEGEN=1is not a usable workaround. -
An app dylib compiled before
bf8fd868edoes not run against providers built after it.build8163's dylib (compiled at3b01efba4, whose parent is3c95020f8) dies during Next's module init againstb8d32ab19providers withTypeError: Cannot set properties of null or undefined (setting 'id'). A clean-main control with no other change fails identically, so it is the provider/compiler pair, not any local patch. Since perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204 is a coordinated codegen+runtime change to the object header image, that is the expected shape of a co-versioning break — worth deciding explicitly whether the shared-provider model is supposed to tolerate a compiler/runtime skew, because the fixture always rebuilds both together and would never notice.
This blocked re-baselining the #8163 residual A/B onto merged main; that measurement was taken on
3b01efba4instead (wherecrates/perry-stdlib/src/fetch/headers.rsis byte-identical tob8d32ab19).- Emitter:
Interim workaround, measured:
PERRY_LLVM_INPROCESS=offConfirmed on the bench mini against unfixed
main(537f74a461), same fixture, same
runtime archives, only the env var added:cd tests/release/packages/next-app-route PERRY_LLVM_INPROCESS=off \ PERRY_MODULE_JOBS=2 PERRY_CODEGEN_UNIT_JOBS=2 PERRY_RUNTIME_DIR=<runtime dir> \ perry compile perry-host.js --output-type dylib --no-auto-optimize --no-cache -o app.dylib- unset (default, in-process) — exit 1 in 14m19s, the 5 modules from the report,
every onebad binary op \insertelement` operands`. =off(text transport) — exit 0 in 9m15s,Error compiling modulecount 0,
all 104 modules, dylib written.
So anyone blocked on this (#8215 re-validation, #8220 end-to-end numbers, #8210 HEAD
validation) can move today with that one variable.native_units_mode()
(crates/perry-codegen/src/native_emit.rs:61) maps0|off|falseto the text
transport, and the missinginsertelementcase is only in the dialect reader on the
native path — which is also why the report's ownPERRY_LLVM_INPROCESS=0observation
held.It is not free. The text path materialized 222.6 MB of IR for a single module and
produced a 206 MB dylib where the native path produces 87 MB — the oversized
modules fall to-O0there. Expect a slower, larger artifact, and do not compare
performance numbers across the two backends.Fix
PR #8241 (draft) teaches the dialect reader the closed set of vector forms
perry-codegen actually emits:insertelement,extractelement,shufflevector,
vector-typedadd/mul, constant vector literals, and vector
poison/undef/zeroinitializer. That also coversexpr/channel.rs's<4 x i32>
SIMD reduction, which was the same latent bug and had simply never landed in a module
big enough to split. With the fix, the same fixture compiles in-process: exit 0,
0 errors, dylib linked.On the report's aggravating detail #1 — "no per-module error text is emitted"
The per-module
Error compiling module …lines are emitted; they were in this
run at lines 500-504, above the refusing-to-link summary. What is genuinely
misleading is the progress output: for a module whose unit fails, the surviving units
still printLLVM unit N/M finished (… ; K complete)and the counter reachesM, so
the log reads exactly like a module that succeeded..next/server/chunks/2.jsprinted
"3 complete" out of 3 units and had in fact failed unit 1. That is worth a separate
issue if anyone wants it — it cost me half a run of confusion before the summary
appeared.Root cause of why this shipped green
The dialect reader's only gate is
dialect::tests::corpus_*, which round-trips
frozen.llsnapshots underexperiments/llvm-inprocess-spike/. A form the
emitters started producing after those files were captured is invisible to them by
construction — all three corpus tests pass with and without the fix. Combined with the
reader being the default only for split modules (every gap/parity fixture is a single
unit and keeps the text path), nothing per-PR could have caught #8204's new
insertelement. #8241 replaces the snapshot with a live emit → re-parse test.- unset (default, in-process) — exit 1 in 14m19s, the 5 modules from the report,
The
PERRY_LLVM_INPROCESS=offworkaround: mechanism confirmed, and an operational cost worth knowing before you take itTwo independent measurements, on two machines and two trees, agree on both halves.
It works, and we now know why rather than just that. Counting
freeze/LLVM pipeline startedin the compile log: 0 occurrences with the flag, 5 without — and those 5 are exactly the failing modules, with their unit splits matching one-to-one:module units fails without the flag chunks/2.js3 ✔ chunks/430.js6 ✔ next/dist/compiled/jsonwebtoken/index.js2 ✔ next-server/app-page.runtime.prod.js10 ✔ next-server/app-route.runtime.prod.js4 ✔ So the flag works by routing precisely the multi-unit modules off the reader path — mechanism, not correlation. (Credit for the workaround hypothesis and this measurement to the parallel session working #8163.)
Three-arm result, same tree/fixture/runtime, only the compiler and flag differing:
arm outcome main, in-process (default)fails — 5 modules, bad binary op \insertelement`,REFUSING TO LINK`, 14m19smain+PERRY_LLVM_INPROCESS=offsucceeds, 9m15s, dylib 206 MB main+ #8241, in-processsucceeds, 15m24s, dylib 87 MB, 104/104 modules ⚠️ The workaround is not free, and on a constrained machine it can fail outright. The text path materialises enormous IR for the big modules — 222.6 MB for one module on one host, 171.3 MB forapp-page.runtime.prod.json another, both at-O0— and then links a ~206 MB dylib. One of the two hosts ran out of disk mid-compile. Budget several GB of free space, or use #8241 instead.Do not compare performance or footprint across the two backends. Oversized modules drop to
-O0on the text path (15 of 104 modules compile below-O3in one measured build), so a binary produced with=offdiffers from the shipped configuration in code quality, size, and plausibly in allocation behaviour and GC scheduling. Any GC or perf measurement taken under the workaround should say so explicitly; it is evidence about the text path only, and the native path is the default — it is what users actually get.Correction to this issue's description: the per-module
Error compiling module …lines are emitted. What misleads is the progress output — a module whose unit failed still printsLLVM unit N/M finished (… complete)all the way toM, so a failed module reads as "3 complete of 3". That deserves its own issue.Fix is up as #8241 (ready for review).
Addendum: how the disk exhaustion actually presents — it does not say "no space"
I was the host that ran out of disk mid-compile under the workaround. Recording the failure signature because it is actively misleading, and the compile-time cost that produces it.
The scratch is not cleaned up on failure. My failed run left 124
perry_llvm_scratch_*directories totalling 4.9 GB in$TMPDIR. That is one run. The text path materialises 100–200 MB of IR per oversized module and leaves it behind when the module fails, so a couple of failed attempts on a box near its limit will bury it — and the second attempt fails because of the first.Under ENOSPC,
clang -cfails with an EMPTY stderr, so all 105 modules report:Error compiling module '…' with --backend llvm: clang -c failed (stderr empty) clang: /usr/bin/clang clang --version: Apple clang version 21.0.0 (clang-2100.1.1.101) requested -target: arm64-apple-macosx15.0.0 … If the failure is a triple/ABI mismatch, set PERRY_LLVM_CLANG to a clang whose default Target: matches arm64-apple-macosx15.0.0 (run `perry doctor` to verify).There is no
os error 28anywhere in the log, and the diagnostic that is printed points at a triple/ABI mismatch — a plausible-sounding wrong answer that would cost real time to chase. The tells that it is disk and not toolchain: the failure hits every module rather than a subset, stderr is empty rather than containing a clang diagnostic, andbad binary op/freeze/LLVM pipeline startedcounts are both 0 (i.e. the flag was working and #8228 was not the cause).Worth considering independently of #8241: an empty clang stderr plus a non-zero exit is a case where checking free space and saying so would be strictly better than suggesting a triple mismatch. Filing that thought here rather than as its own issue since it is one line of diagnosis in the same code path — happy to split it out if preferred.
Practical guidance if you take the workaround: budget several GB beyond the ~206 MB dylib, check
dffirst, and clean$TMPDIR/perry_llvm_scratch_*between attempts — but only your own, matched by the pid-derived prefix in the error message (perry_llvm_scratch_<prefix>_N). Another session's compile in the same$TMPDIRlooks identical, and there were six live ones on my box when I cleaned mine.- added a commit that references this issue
on Aug 16, 2026
Symptom
On current
main-equivalent (07c8040bf,perry-devprofile), compiling the pinned production Next App Route fixture (tests/release/packages/next-app-route/, 104 modules) with the in-process LLVM backend — the default wheneverPERRY_LLVM_INPROCESSis unset, i.e. whatfixture.shstep[5/7]runs — fails deterministically:Those are the app's 5 biggest modules (the 4 that "run exclusively to cap peak memory" plus jsonwebtoken). Reproduced 5/5 times on the bench mini (macOS 26.5.1 arm64, 8 GB, LLVM 22.1.8): four full
fixture.shruns (withPERRY_MODULE_JOBS=2and bothPERRY_CODEGEN_UNIT_JOBS=2and=1) and one bare compile:Two aggravating details
Error compiling module)" — but no such line exists anywhere in the output (grepped the full logs). The modules visibly make progress (LLVM unit 1/10 finished,froze 3/10 units, the fix(next): pass production App Route dylib gate #8082 RS4GC-optnone fallback fires forperry_closure_…app_page_runtime_prod_js__5118) and then are reported failed with nothing to act on. Whatever fails should say what failed.fixture.shcompiled the same app clean, in-process, on this same box earlier today at3c95020f8(the [Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163 validation runs, and [Next.js/dylib] Full production App Route compatibility tracker #8040's DoD audit — 500 serving batches). The window is3c95020f8..07c8040bf; commits touching codegen in it: perf(codegen): decide scalar parameter descriptors with the typed-abi leaf guards #8201 (scalar parameter descriptors via typed-abi leaf guards), perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204 (recover the 56→48 B header-shrink instruction cost), gc: verify GC_STORE_AUDIT claims against emitted IR and source structure, not comments (#8185) #8206 (verify GC_STORE_AUDIT claims against emitted IR).PERRY_LLVM_INPROCESS=0(external clang/opt — the configurationtests/test_next_app_route_dylib.shpins) compiles all 104 modules clean at the same commit on the same box, and the resulting app serves. So the release-tier fixture and anything else on the in-process default is blocked, while the external-backend path is fine.Evidence on the bench mini
~/perry-bench.noindex/tmp-8040-knob/probe-inprocess.log— bare-compile repro (exit 1, full output).~/perry-bench.noindex/tmp-8040-knob/fixture-v2.log,fixture-v10.log,fixture-v10-diag.log,rerun-v2.log— fourfixture.shfailures (FAIL next-app-route — Perry dylib compile failed). Note: in three of these the process also died without printing the refusing-to-link summary at all.~/perry-bench.noindex/tmp-8040-knob/ext-v2.log— same fixture, same commit,PERRY_LLVM_INPROCESS=0: compile clean,PASS next-app-route (20 batches, …).Found while validating #8209/#8210; the serving-side results there are unaffected (validated on the external backend).