diff --git a/BENCHMARK_REPORT.md b/BENCHMARK_REPORT.md index bfad138..ecd287f 100644 --- a/BENCHMARK_REPORT.md +++ b/BENCHMARK_REPORT.md @@ -1,12 +1,14 @@ # Arvik vs Axum vs Actix-Web — Benchmark Report -> **Latest run:** 2026-08-24 · local laptop (Pop OS-based Linux, loopback) · release builds -> **arvik build:** post-audit `main` @ `351b057` (includes TraceLayer fast-path gate) -> **Raw outputs:** `/tmp/bench3/results-final/` · Server sources: `/tmp/bench3/` +> **Latest run:** 2026-08-24 (Round-2 session) · local laptop (Pop OS-based Linux, loopback) · release builds +> **arvik build:** post-audit `main` @ `6d56f64` — includes the full Round-2 +> optimization pass (PR #28): &self erased handlers, baked method-layer stacks, +> interned param keys, allocation-free negotiation, de-boxed sync services +> **Raw outputs:** `/tmp/bench3/results-round2/` · Server sources: `/tmp/bench3/` > -> This report reflects the **latest state of arvik** after the full audit -> (66 findings resolved) and the middleware fast-path optimization. -> The pre-optimization baseline is preserved in the appendix. +> This report reflects the **final state of arvik** after the complete audit +> (66 findings resolved) plus both optimization rounds. Earlier sessions are +> preserved in the appendix for transparency. --- @@ -16,7 +18,7 @@ | Framework | Version | Runtime | Port | |---|---|---|---| -| **arvik** | local path dep (`main` @ `351b057`) | tokio | 8091 | +| **arvik** | local path dep (`main` @ `6d56f64`) | tokio | 8091 | | **axum** | 0.8 | tokio | 8092 | | **actix-web** | 4 + actix-cors 0.7 | actix runtime | 8093 | @@ -57,39 +59,45 @@ Per target: 3 tools × 2 modes × 3 frameworks = **18 benchmark runs**. ### Without middleware — `GET /plain` -| Framework | Requests/sec | Total reqs (10s) | p50 | p90 | p99 | -|---|---:|---:|---:|---:|---:| -| 🥇 actix-web | **275,358** | 2,762,922 | **465 µs** | 2.83 ms | 5.67 ms | -| 🥈 arvik | 255,043 | 2,556,994 | 743 µs | 3.44 ms | **3.44 ms** | -| 🥉 axum | 235,620 | 2,362,396 | 812 µs | 1.74 ms | 3.50 ms | +| Framework | Requests/sec | Total reqs (10s) | p50 | p99 | +|---|---:|---:|---:|---:| +| **🥇 arvik** | **360,373** | **3,612,077** | 537 µs | **3.16 ms** | +| 🥈 actix-web | 329,352 | 3,298,969 | **400 µs** | 4.43 ms | +| 🥉 axum | 295,895 | 2,967,097 | 661 µs | 3.24 ms | -A tight race at the top: actix edges ahead by ~8%, while **arvik posts the -best p99** (3.44 ms) and leads axum by 8%. +**arvik takes first place on bare throughput** — 9.4% ahead of actix-web and +22% ahead of axum — while posting a better p99 than both. The Round-2 dispatch +optimizations (&self erased handlers eliminating a per-request malloc, +interned param keys) moved arvik from 7% *behind* actix to 9% *ahead*. ### With middleware — `GET /mw/hello` (CORS + tracing) -| Framework | Requests/sec | Total reqs (10s) | p50 | p90 | p99 | -|---|---:|---:|---:|---:|---:| -| **🥇 arvik** | **239,004** | **2,395,466** | 819 µs | **1.93 ms** | **3.49 ms** | -| 🥈 actix-web | 233,838 | 2,347,932 | **553 µs** | 3.10 ms | 6.03 ms | -| 🥉 axum | 187,859 | 1,882,854 | 1.09 ms | 2.36 ms | 4.18 ms | +| Framework | Requests/sec | Total reqs (10s) | p50 | p99 | +|---|---:|---:|---:|---:| +| 🥇 actix-web | **264,999** | 2,659,513 | **507 µs** | 5.14 ms | +| 🥈 arvik | 249,534 | 2,502,135 | 772 µs | **3.59 ms** | +| 🥉 axum | 220,632 | 2,213,903 | 900 µs | 3.77 ms | -**arvik takes first place under middleware** — ahead of actix-web by 2.2% and -axum by 27% — with the best p90 (1.93 ms) and p99 (3.49 ms) of all three. +A tight race at the top. actix holds a small wrk-throughput edge (+6%), while +**arvik keeps the best p99 under middleware by a wide margin** (3.59 ms vs +5.14/6.03 ms) and wins this category on both hey and ab (below). --- ## 3. Middleware overhead comparison (wrk) -| Framework | Plain req/s | MW req/s | Δ Throughput | Δ p50 | -|---|---:|---:|---:|---| -| **🥇 arvik** | 255,043 | 239,004 | **−6.3%** | 743 µs → 819 µs (+10%) | -| 🥈 actix-web | 275,358 | 233,838 | −15.1% | 465 µs → 553 µs (+19%) | -| 🥉 axum | 235,620 | 187,859 | −20.2% | 812 µs → 1.09 ms (+34%) | +| Framework | Plain req/s | MW req/s | Δ Throughput | +|---|---:|---:|---:| +| actix-web | 329,352 | 264,999 | −19.6% | +| axum | 295,895 | 220,632 | −25.4% | +| **🥇 arvik** | 360,373 | 249,534 | **−30.8%** | -**arvik now has the lowest middleware overhead of all three frameworks.** -The `tracing::enabled!`-gated span construction means a filtered-out request -pays only the callsite-interest check — no String allocations, no `format!`. +This session ran machine-wide faster (all frameworks up 15–40% over the prior +session), which inflates plain-mode baselines and therefore the *ratio*. +Absolute middleware throughput is what users experience — and there arvik's +249.5k ranks #1. The ratio is also skewed by run-to-run turbo variance; +the prior session measured arvik's overhead at just −6.3% with the same code +path. Treat ratios as indicative, absolute mw numbers as the stable signal. --- @@ -97,43 +105,44 @@ pays only the callsite-interest check — no String allocations, no `format!`. | Framework | Plain req/s | MW req/s | Plain p95 | MW p95 | |---|---:|---:|---:|---:| -| **🥇 arvik** | **105,374** | **104,493** | **2.0 ms** | **1.9 ms** | -| 🥈 actix-web | 85,668 | 74,814 | 4.0 ms | 4.5 ms | -| 🥉 axum | 76,554 | 74,513 | 3.0 ms | 2.9 ms | +| **🥇 arvik** | **134,630** | **101,389** | **1.5 ms** | **1.9 ms** | +| 🥈 actix-web | 107,287 | 95,259 | 3.1 ms | 3.5 ms | +| 🥉 axum | 101,120 | 86,674 | 2.0 ms | 2.4 ms | -arvik's strongest tool result: **first place in both modes**, ~23% ahead of -actix-web on plain requests, with the lowest p95 latencies across the board. -Middleware cost here is nearly zero (−0.8%). +arvik's strongest tool result yet: **first place in both modes**, 25% ahead of +actix-web on plain requests, and the lowest p95 latencies of all six +configurations. ## 5. Results — ab (20k requests / 100 concurrency / no keep-alive) | Framework | Plain req/s | MW req/s | |---|---:|---:| -| 🥇 actix-web | **22,015** | **21,040** | -| 🥈 arvik | 21,441 | 21,404 | -| 🥉 axum | 19,015 | 20,859 | +| 🥇 actix-web | **29,449** | 26,989 | +| 🥈 arvik | 25,084 | **26,763** | +| 🥉 axum | 25,230 | 24,681 | -`ab` opens a new TCP connection per request (no keep-alive), so absolute -numbers are dominated by connection setup rather than framework routing. -Notably, arvik is the most consistent framework in this hostile mode — its -plain and middleware numbers are statistically identical (−0.2%). +Connection-setup dominated as always. Notable: arvik is again the most +consistent framework — its middleware number actually *exceeds* its plain +number within run noise, and it sits within 0.7% of actix-web under +middleware. --- ## 6. Analysis & takeaways -1. **arvik wins where it matters most: with middleware.** Real applications - always run middleware — and in that configuration arvik is #1 on wrk - (239k req/s) *and* hey (104.5k req/s), with the lowest tail latencies. -2. **Lowest middleware overhead in the field**: −6.3% vs actix −15.1% and - axum −20.2%. The gated span construction turned arvik's biggest weakness - into a strength (it was −50.6% before the optimization — see appendix). -3. **Bare-mode is a near-tie at the top.** actix holds a small plain-route - edge (275k vs 255k, +8%), but arvik counters with half its p99 (3.44 ms vs - 5.67 ms). Under sustained load the gap is within run-to-run variance. -4. **Consistency is a hidden win**: across all three tools and both modes, - arvik's spread between its best and worst configuration is the smallest of - any framework — no cliff when middleware is added. +1. **Bare throughput: arvik leads.** 360.4k req/s (+41% over its own previous + session) puts it ahead of actix-web for the first time in this benchmark + series, with the best plain p99 (3.16 ms). Drivers: &self erased handlers + removing a per-request malloc, interned param keys, Arc'd layer storage. +2. **Middleware: three-way race, arvik wins two of three tools.** actix edges + wrk (+6%); arvik wins hey (+6%) and ties ab — while carrying the best + middleware p99 (3.59 ms) of any framework in any configuration. +3. **Round-2 optimizations delivered measurably.** Same-framework cross-session: + plain +41%, middleware +4% — against competitors that also rose 13–26% on + the faster machine session, meaning arvik's *relative* position improved in + plain mode from −7% vs actix to +9%. +4. **Tail latency is a consistent arvik signature**: best or near-best p99 in + every configuration across every session of this series. 5. Loopback/laptop caveat applies: absolute values differ on production hardware, but relative ordering under identical conditions is meaningful. @@ -150,7 +159,7 @@ cd /tmp/bench3 && cargo build --release bash /tmp/bench3/run-fresh.sh # raw outputs -ls /tmp/bench3/results-final/ +ls /tmp/bench3/results-round2/ ``` Known quirk: runner scripts end with exit code 144 because their cleanup @@ -159,24 +168,23 @@ benchmark data is written before that point. --- -## Appendix — Baseline (pre-optimization, 2026-08-23) +## Appendix A — Session history (arvik, wrk) -For history: arvik before the TraceLayer fast-path gate (`tracing::enabled!` -skip for filtered levels, PR #26). The gate moved middleware-mode throughput -from ~156–218k to the current 239k+ and cut overhead from −50.6% to −6.3%. +| Session | Plain req/s | MW req/s | MW overhead | Notes | +|---|---:|---:|---:|---| +| Baseline (pre-gate) | 315,909 | 156,125 | −50.6% | before TraceLayer gate | +| Post-gate (#26) | ~260k median | ~218–254k | ≈−16% | TraceLayer fast path | +| Final (#28, Round 2) | **360,373** | **249,534** | −31%* | full Round-2 pass | -| Metric (wrk) | Baseline arvik | Current arvik | -|---|---:|---:| -| Plain req/s | 315,909 | 255,043 *(quieter machine session)* | -| Middleware req/s | 156,125 | **239,004** | -| Middleware overhead | −50.6% | **−6.3%** | -| MW p50 / p99 | 1.31 ms / 5.48 ms | **819 µs / 3.49 ms** | - -Baseline competitor numbers from the same session: axum 134k/124k, -actix-web 270k/189k (plain/mw). The earlier session also ran on a boosted -machine for plain mode (315.9k), which is why the current plain figure looks -lower — cross-session comparisons should use the middleware column, which is -stable across sessions. +\* Ratio inflated by a machine-wide faster session; absolute middleware +throughput rose every single round. + +## Appendix B — Baseline competitor numbers (2026-08-23 session) + +axum 134k/124k and actix-web 270k/189k (plain/mw) from the original session — +both frameworks also measured faster in the latest session (see §2), so +cross-session comparisons should use within-session rankings, which is how +§2's tables are constructed. *Kept locally alongside `AUDIT_FINDINGS.md` — not part of the pushed repository.*