Repository navigation
tracking: close the remaining 7 losing rows vs Node (12/19 won, geomean 0.7775x) #8410
Description
Purpose
Tracking issue for closing the remaining gap to Node on the 19-row benchmark corpus. Perry
currently beats Node on 12 of 19 rows with a geomean of 0.7775x. This enumerates the
seven rows that still lose, each with its own ticket, evidence and owner-ready repro.
Measured on 8bfdac447 (2026-08-19 sweep, quiet M1 mini, verdict CLEAN, load 1.42 -> 2.08,
best-of-five interleaved, byte-exact output re-checked on every run).
Current standing
| value | |
|---|---|
| Perry beats Node | 12/19 |
| geomean Perry/Node | 0.7775x |
| geomean Perry/scriptc | 0.359x |
| geomean Perry/Porffor | 0.271x |
| peak RSS vs Node | lower on all 19 rows |
The seven remaining rows
| row | P/Node | instr ratio | dominant cost | ticket |
|---|---|---|---|---|
cycles |
1.01 | 2.51 | init_typed_shape_layout 20.1% |
#8405 |
shapes |
1.06 | 1.25 | try_read_tracked_gc_header + shape_descriptor_by_id = 100% |
#8406 |
asyncpipe |
1.12 | 1.71 | not CPU-bound — event-loop/timer latency | #8409 |
churn |
1.16 | 0.99 | memory stalls; lowest IPC in corpus (13.35) | #8408 |
pipeline |
1.26 | 1.88 | libm fmod 7.3%, plus user-code method dispatch |
#8407 |
interp |
1.55 | 1.60 | layout bookkeeping ~20.8% | #8405 |
iso_miss |
2.02 | 1.97 | string concat 17.9% (#8394), layout bookkeeping | #8394, #8405 |
The structural finding
Splitting all 19 rows by instruction ratio is unambiguous:
- every row Perry wins has instructions/Node <= 1.07, mostly far below
(fib400.22,churn_read0.27,tree_wide0.37) - every row Perry loses has >= 1.25, except
churn
Perry's codegen quality is not the problem. Where it emits comparable work it wins
comfortably. It loses precisely where it executes more work. That makes each remaining row a
question of "what work can be deleted", which is why every ticket above names a specific
frame and a percentage rather than a vague direction.
churn is the sole exception and therefore the most interesting: 0.99 instruction ratio and
still 1.16x slower. That one is locality, not work.
Ordering suggestion
- perf: per-allocation GC layout bookkeeping is 20.1% of cycles self time and ~21% of interp #8405 (layout bookkeeping) has the widest blast radius — it is 20.1% of
cycles,
~20.8% ofinterp, and visible oniso_missandpipeline. It alone should flipcycles. - perf: shapes spends 100% of self time in try_read_tracked_gc_header + shape_descriptor_by_id (1.06x Node) #8406 (shapes) is a single row but its profile is 100% two helpers, so it is unusually
well localized. - perf: JS % lowers to a libm fmod call — 7.3% of pipeline self time #8407 (
fmod) is the cheapest concrete win and has a clear precedent in perf(runtime): give the relational operators the numeric fast path + and === already have #8395. - perf: no rope/cons-string representation makes accumulator concatenation quadratic (514x Node at n=16000; 17.9% of iso_miss) #8394 (ropes) is the largest single item on the worst row but is a genuine
architectural change; a staged design is already committed there. - perf: churn loses 1.16x to Node while executing FEWER instructions (0.99 ratio, lowest IPC in the corpus) #8408 (
churn) and perf: asyncpipe is 1.12x Node but NOT CPU-bound — needs event-loop/timer counters, not a sampler #8409 (asyncpipe) need different instruments than the others
— hardware counters and event-loop counters respectively.
Method notes for anyone picking these up
Every ticket carries the same build/measure/profile setup and the same constraints. Two that
have cost real time on this corpus and are worth repeating here:
- Build the
-staticwrapper crates or you link a stale runtime and both A/B arms behave
identically — a vacuous "no change". - A compelling diff read is a hypothesis, not a result. fix(async_hooks): track scheduler resource lifecycles #8328 looked obviously guilty on
this exact corpus and measured +0.2%.
Activity
proggeramlug commentedon Aug 20, 2026
Fresh four-engine sweep on the quiet mini, Perry @
415f4e22e(includes #8482, #8483, #8484, #8487, #8488). Harness verdict CLEAN, load 1.46 → 2.12, 5 shuffled interleaved repeats, every run's output re-checked. Node oracle v26.5.1.12/19 → 15/19 rows won. Geomean 0.7083x wall.
Geomean instructions 0.8086x, geomean peak RSS 0.2565x — about a quarter of Node's memory across the corpus, with the compute win intact.
Against the previous mini sweep (
14468dcbc), same harness:Biggest movers:
tree_wide−75.9%,tree−59.8%,churn−59.2%,fib40−55.1%,pipeline−46.0%,cycles−39.2%,retain_wide−37.8%,interp−33.2%,deeplist−24.5%,retain−23.3%,iso_miss−21.5%.The four remaining losses split into two different problems
interp(1.50x) andiso_miss(1.88x) are genuinely CPU-bound. Instruction ratio tracks the wall ratio almost exactly (1.51x / 1.99x), and cycles track too. These need fewer instructions, not better scheduling — the dynamic numeric envelope work on #8485 is aimed here.asyncpipe(1.10x) andpipeline(1.12x) are not. Both execute ~70–75% more instructions than Node yet spend only 5–7% more cycles, so they retire at a much better IPC and the wall gap is not coming from the instruction count. Confirms #8409's read thatasyncpipeneeds event-loop/timer work rather than codegen.pipelinedeserves the same treatment and is separately tracked by #8243.shapessits at exactly 1.00x with a 1.21x instruction ratio — one good pass on #8406'stry_read_tracked_gc_headerhot spot should flip it.proggeramlug commentedon Aug 20, 2026
Re-swept on the quiet mini after #8491 landed. Perry
8d1cca158, verdict CLEAN, load 1.49 → 2.16, 5 shuffled interleaved repeats. Now a 20-row corpus — see below.The 19 original rows are unchanged. Every instruction ratio is flat to within 0.02x of the previous sweep, confirming #8491 costs the corpus nothing. Two wall numbers moved more than noise would suggest —
churn_alloc0.80x → 0.92x andshapes1.00x → 1.04x — but both have flat instruction ratios (1.03→1.04 and 1.21→1.22), so that is host contention, not a regression.Geomean 0.7999x over 20 rows, 14/20 won. On the unchanged 19-row basis it is 0.7083x, 15/19 — identical to the previous sweep.
The corpus was missing its worst row
typed_arrayenters at 5.25x Node, 8.66x instructions — roughly three times worse than anything else Perry runs, and it has been invisible to every sweep ever taken.The reason is structural, not an oversight:
benchmarks/suite/bench_typed_array_untyped_access.tsprints its own wall-clock timings, and the sweep's correctness gate is a byte-exact stdout compare against the Node oracle, so a self-timing program can never be admitted. I addedgc-handoff/apps/typed_array.ts— identical workload, emitting onlychecksum:-821955270, verified deterministic across runs — and registered it as row 20. The suite version stays as the #5525 ratio metric; the corpus one is the regression tripwire.Worth stating plainly: #8491 improved that workload by −44.1% wall and −34.7% instructions, and no sweep would have noticed either the win or a later regression. Any benchmark whose own output encodes timing is structurally excluded from the gate that protects it.
Tracked as #8496, with an agent on it.
Where the remaining six losses stand
typed_array5.25x,iso_miss1.94x,interp1.51x — genuinely CPU-bound; the instruction ratio tracks or exceeds the wall ratio. Agents are on all three.asyncpipe1.13x,pipeline1.14x — not CPU-bound. Both retire ~70–77% more instructions than Node while spending only ~5–7% more cycles, so they are waiting, not computing. A sampling profiler cannot see this; these need event-loop and timer counters (perf: asyncpipe is 1.12x Node but NOT CPU-bound — needs event-loop/timer counters, not a sampler #8409, perf: pipeline's +3.95% at #8084 is unattributed — it is NOT the callee rooting (IR is byte-identical) #8243), which is a different instrument, not a codegen pass.shapes1.04x — nearest to flipping; perf: shapes spends 100% of self time in try_read_tracked_gc_header + shape_descriptor_by_id (1.06x Node) #8406 has the profile (100% of self time in two runtime helpers) and an agent.proggeramlug commentedon Aug 21, 2026
Summary
s = s + a + b + ...for proven string accumulators in local, captured, and module-global slotsRefs #8410.
Profile
The macOS
sampleutility hung before collecting a sample on this host, including when reproduced againstyes. I used the same symbol-preserving build (PERRY_KEEP_SYMBOLS=1 PERRY_DEBUG_SYMBOLS=1, about 17.7k symbols in the generated fixture) with Instruments Time Profiler over a private 10xiso_missfixture instead.Baseline sampled CPU was 8,422 ms. The largest leaves were
evalNode(38.90%),js_string_concat_chain(14.15%),lookup(12.19%),memmove(4.19%), arena allocation (3.72%), and the separate accumulator append/storage/rooting envelope (6.80% combined).Candidate sampled CPU was 7,716 ms (-8.38%). The suffix concat plus separate append envelope becomes one
js_string_append_chainleaf (15.15%);evalNoderemained dominant (42.69%) andlookupremained essentially unchanged (12.08%).Performance
Five shuffled, interleaved repeats on macOS, comparing a locally built
8d1cca158baseline with this branch. Both arms used the identical-p perry -p perry-runtime-static -p perry-stdlib-staticrelease build set, and archive mtimes were verified after each build. Medians are shown.iso_missThe wall ranges overlap widely (1.41-5.00 s before, 1.24-7.07 s after), so the wall median is not treated as a real 28.9% result. Instructions retired are the primary signal. The observed RSS cost is 48 KiB; the implementation does not add a cache or retained reserve and allocates the complete result once, but the measured peak is reported rather than rounded away.
Whole-corpus medians:
No non-target row moved more than 0.41% in instructions. All non-target wall ranges overlapped; the large wall/cycle swings track host contention rather than instruction count.
Validation
cargo test --release -p perry-runtime --lib: 2,608 passed, 4 ignoredcargo test --release -p perry-codegen --lib: 1,121 passedcargo test --release -p perry --bin perry: 1,008 passedHost status at handoff: 6 GiB free after deleting this worktree's generated
target/and reducing the reserved external target to about 832 MiB. The task requires stopping below 15 GiB, so this PR remains draft until those two acceptance runs can complete.proggeramlug commentedon Aug 21, 2026
interpfollow-up shipped in #8539.The profile did not support the dynamic-numeric-envelope hypothesis: across 30 sampled runs,
evalNodeheld 61.1% of self samples. The actionable cost was tiny pointer-bearing environment arrays keeping per-object GC layout masks alive. Raising the minimum mask-bearing payload from 2 to 4 slots makes two- and three-slot mixed objects use the exact tag scan instead.Measured result: -4.61% retired instructions on
interp(9,013,545,066 -> 8,598,251,738), 19/19 byte-exact, no other corpus row moved by more than 1%, and no peak-RSS regression. The cutoff is 4 rather than 8 because 4 captured the full instruction win; 8 retired no fewer instructions and would broaden the tracing-policy change to larger objects.Post-rebase validation is green: serialized release runtime tests passed (2,608 passed / 4 ignored / 0 failed), and the full 575-test gap suite matched its snapshot (570 pass / 5 known mismatches / 0 compile failures / 0 crashes).
proggeramlug commentedon Aug 22, 2026
Status update — the row count in the title is stale, and the measurement behind it is too
49 commits have landed on
mainsince this issue was last updated, including nine perf changes that bear directly on the losing rows. The most recent sweep predates almost all of them.Where the last CLEAN measurement left us
Quiet mini, 5 shuffled interleaved repeats,
perry @ 4ee1d04b5, Node 26.5.1 — geomean 0.7773x, 15/20 rows won, peak RSS 0.2395x:Two regressions from that sweep are still unattributed
typed_array5.25x → 6.23x (+18.6%) — the corpus's worst row got worseiso_miss1.94x → 2.09x (+7.9%)Both appeared between two sweeps whose merged work I had audited for correctness but not measured. They are the strongest argument in this issue for why perf PRs need a corpus timing before merge, not just a compile check.
The corpus was missing its worst row until recently
typed_arrayonly entered the corpus on 2026-08-21. It could not join earlier becausebenchmarks/suite/bench_typed_array_untyped_access.tsprints its own wall-clock timings, and the sweep's correctness gate is a byte-exact stdout compare against the Node oracle.gc-handoff/apps/typed_array.tsis the corpus variant — same workload, emits onlychecksum:-821955270.Worth stating plainly: #8491 improved that workload by −44.1% wall and −34.7% instructions, and no sweep would have noticed either the win or a later regression. Any benchmark whose own output encodes timing is structurally excluded from the gate meant to protect it.
Landed since, unmeasured on the corpus
#8581(join + repeat + pad/replace, closes #8434),#8580(proven-nonnegative index specialization),#8573(inline monomorphic shape guards),#8570(asyncpipe microtask suppression, fixes #8409),#8568/#8571(superseded by #8581),#8560(PIC descriptor gate, 6.49x on an ECS query),#8539(layout-mask cutoff,interp−4.61%),#8540(number-only local roots).#8570targetsasyncpipeand#8581targets the string-heavy rows, soasyncpipe,interpandiso_missare the rows most likely to have moved. None of it is measured yet.What the instruction ratio still says about the remaining losses
typed_array8.49x,iso_miss1.98x,interp1.44x — genuinely CPU-bound; the instruction ratio tracks or exceeds the wall ratio. These need fewer instructions.asyncpipe1.71x andpipeline1.75x instructions but only 1.07x/1.10x wall — not CPU-bound. They retire far more instructions at much better IPC, so the gap is waiting, not computing. A sampling profiler cannot see it; these need event-loop and timer counters (perf: asyncpipe is 1.12x Node but NOT CPU-bound — needs event-loop/timer counters, not a sampler #8409, perf: pipeline's +3.95% at #8084 is unattributed — it is NOT the callee rooting (IR is byte-identical) #8243).Next
A fresh sweep at current
main(fefdc367b) is the blocking item, and it should be run before any further perf work — both to capture nine landed changes and to attribute the two open regressions. The 20-row corpus, the Node-oracle byte gate and the harness are all staged; the rebuild is the only cost.