diff --git a/CHANGELOG.md b/CHANGELOG.md index ea34f27f..5104040c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,11 +52,15 @@ is in the [`1.0.0-rc.0`](#100-rc0---2026-05-26) entry below. ## [1.0.2] - 2026-07-28 ### Changed -- `Lua.new/1` is ~100x faster (roughly 40µs down to 0.4µs) for the default and - fully-custom-sandbox configurations. Installing the standard library is pure - and deterministic, so the boot-time VM template is now built once per node - and memoized in `:persistent_term`; every later `Lua.new/1` starts from the - shared template copy-on-write. In `:interactive` mode (dev, IEx, tests) the +- `Lua.new/1` is ~60x faster for the default configuration — 36.7µs down to + 0.6µs median, with per-call allocation down from ~92KB to under 1KB — and + ~5.5x faster when a custom sandbox is passed (36.0µs down to 6.5µs), as + measured by `benchmarks/vm_new.exs` under `mix run` + ([full figures](https://github.com/tv-labs/lua/blob/main/bench_results/versions-2026-07-28.md)). + Installing the + standard library is pure and deterministic, so the boot-time VM template is + now built once per node and memoized in `:persistent_term`; every later + `Lua.new/1` starts from the shared template copy-on-write. In `:interactive` mode (dev, IEx, tests) the cache self-invalidates when the modules that built it are recompiled; hosts that hot-load new code in `:embedded` mode (releases) can force a rebuild with `Lua.VM.Bootstrap.reset/0` (#398). diff --git a/Dockerfile b/Dockerfile index 7e74e943..9f5f0e33 100644 --- a/Dockerfile +++ b/Dockerfile @@ -56,6 +56,10 @@ COPY website/priv priv COPY website/lib lib COPY website/assets assets +# Recorded benchmark results, read at compile time by lib/website/benchmarks.ex +# (@external_resource) to render /benchmarks. Compile fails loudly without it. +COPY bench_results /app/bench_results + # mix compile must run BEFORE assets.deploy because Phoenix's LiveView # colocated-hooks compiler generates files under _build/ that esbuild # resolves via NODE_PATH (`phoenix-colocated/website`). diff --git a/bench_results/README.md b/bench_results/README.md new file mode 100644 index 00000000..07cbc16c --- /dev/null +++ b/bench_results/README.md @@ -0,0 +1,137 @@ +# `bench_results/` + +Recorded benchmark runs and the reports written from them. + +The benchmark **scripts** live in [`benchmarks/`](../benchmarks/). This +directory holds their **output**: raw stdout, a parsed JSON summary, environment +probes, and the human-readable report for each measurement campaign. Nothing +here is used by the library at runtime, and none of it ships in the Hex package. + +## Contents + +| Path | What it is | +|---|---| +| [`versions-2026-07-28.md`](./versions-2026-07-28.md) | Cross-version comparison — v0.4.0 vs v1.0.0 vs 1.0.2 (`main` @ `3a0d392`), with Luerl 1.5.1 as a same-run control in every table. Supersedes the 1.0.0-era numbers in [`benchmarks/BASELINE.md`](../benchmarks/BASELINE.md). | +| [`v0.4.0/`](./v0.4.0/), [`v1.0.0/`](./v1.0.0/), [`v1.0.2/`](./v1.0.2/) | The data behind that report — one directory per released version. This is the ongoing convention: each release gets its own directory here, measured with the full suite of its day. | + +### Layout of a version directory + +``` +/ + environment.md # ref, commit, mode, CPU, OTP/Elixir, timestamp, command form + summary.json # parsed results: workload -> case -> jobs + comparison lines + .txt # verbatim stdout of one `mix run benchmarks/.exs` + cpu.txt versions.txt timestamp.txt commit.txt +``` + +`summary.json` schema, as produced for the 2026-07-28 run: + +- Top level is keyed by **workload** (`fibonacci`, `table_ops`, `vm_new`, …). +- Each workload maps to its **case banners** (`"default"` for single-case + workloads, otherwise the banner the script printed, e.g. + `"patterns: gsub template substitution (n=200)"`). +- Each case has `jobs` — a list of `{name, ips, average, deviation, median, + p99, memory}` — plus the verbatim Benchee `comparison` lines and, when memory + measurement was on, `memory_comparison` or `memory_note`. +- `table_ops` cases nest one level deeper under `by_input` (or `inputs` on the + v0.4.0 run) keyed by input label — `small (n=10)`, `medium (n=100)`, + `large (n=1000)`. +- `vm_new` carries `cold_call` and `second_call` alongside its jobs: the + first-ever and second `Lua.new()` on the node, measured before Benchee starts. +- `encode_decode` is `{"raw": "..."}`. That script uses its own `:timer.tc` + harness rather than Benchee and prints a per-element-nanoseconds table, so it + is stored unparsed. + +Case-name keys are **not** byte-identical across refs: the v0.4.0 run appends +`" (mode: full)"` to banners and uses `"(single case)"` where later runs use +`"default"`. Normalise by stripping the mode suffix before joining across refs. + +## Reproducing a run + +Full per-ref instructions — including the three adaptations v0.4.0 needs and +the language constraints a cross-version workload must respect — are in the +[Reproduction section of the report](./versions-2026-07-28.md#reproduction). +The short version: + +```sh +MIX_ENV=benchmark mix deps.get +for w in fibonacci closures oop string_ops string_format table_ops \ + patterns metamethods pcall_varargs vm_new encode_decode; do + LUA_BENCH_MODE=full MIX_ENV=benchmark mix run "benchmarks/$w.exs" +done +``` + +Two rules are not optional: + +- **Serially, one `mix run` at a time, on a quiet machine.** Concurrent load + inflates deviation badly — the table and OOP cases swing enough to flip + orderings. +- **`LUA_BENCH_MODE=full` for anything published.** The default `quick` mode + uses short windows, skips memory measurement, and collapses the table + workloads to a single input size. It is for "did my change move the needle" + iteration, not for numbers anyone reads. + +### Older refs + +Each ref is measured in a throwaway detached worktree so the main checkout is +never modified: + +```sh +git worktree add --detach /tmp/lua- +``` + +- **v1.0.0** — copy in the four workloads that postdate it + (`patterns`, `metamethods`, `pcall_varargs`, `vm_new`), then + `MIX_ENV=benchmark mix deps.get`. +- **v0.4.0** — copy in the whole `benchmarks/` directory (the tag has none) and + add `{:benchee, "~> 1.3", only: :benchmark}` to `deps/0`. `luerl` is already + an unconditional dependency there, so the control rows need nothing. + +Remove the worktree when done (`git worktree remove --force /tmp/lua-`). + +## Benchmarking a new release + +The convention: **every released version gets a directory here**, so the +series grows one column per release. + +1. After tagging, run the full suite against the tag (serially, full mode, + quiet machine — see above) and put the outputs in + `bench_results//` with the same file layout as the existing + directories (`environment.md`, `summary.json`, one `.txt` per workload, + plus the env probes). +2. Include the Luerl control rows — they are what make the new column + comparable to the old ones despite machine/OTP drift between sittings. +3. If the suite gained workloads since the last release, note in + `environment.md` which workloads are new (older version directories will + simply lack those files). +4. Write or extend a report named `-.md` quoting + **medians**, not averages — several workloads have allocation-driven GC + pauses that pull the mean around. +5. Add a row to the Contents table above. +6. Do not edit `benchmarks/BASELINE.md`. It is the historical 1.0.0 gate + record; a newer report supersedes it by saying so. + +### What updates itself + +The hosted page at [`/benchmarks`](../website/lib/website_web/controllers/page_html/benchmarks.html.heex) +reads these directories directly, so steps 1–5 are the whole job: + +- **A new `/summary.json` becomes a new column.** Version directories + are found by glob and ordered with `Version.compare/2` — no list to extend. +- **A new `versions-.md` becomes the linked report**, and its date becomes + the page's dated eyebrow. The newest report filename wins. +- **Headline tiles and the "still behind Luerl" figures re-derive** from the new + column, including the "N× faster than " deltas. + +Two things still need a human: + +- **A new workload needs a row spec** in `Website.Benchmarks` (`@rows`) before it + appears — which cases are worth showing, and under what name, is editorial. + A version that lacks a workload another version has renders as `—`. +- **The prose** — the headline claim and the analysis paragraphs — is written, + not generated. Re-read it when the story changes. + +`Website.Benchmarks` registers each `summary.json` as an `@external_resource`, +so editing recorded results recompiles the page in dev. The container build +copies this directory in (see `Dockerfile`); compilation fails loudly rather +than shipping an empty page if it is missing. diff --git a/bench_results/v0.4.0/closures.txt b/bench_results/v0.4.0/closures.txt new file mode 100644 index 00000000..8e16e2ef --- /dev/null +++ b/bench_results/v0.4.0/closures.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 2.69 K 372.35 μs ±7.42% 368.92 μs 507.15 μs +lua (eval) 2.52 K 397.20 μs ±8.30% 390.13 μs 515.81 μs +luerl 2.50 K 400.26 μs ±9.75% 391.42 μs 536.69 μs + +Comparison: +lua (chunk) 2.69 K +lua (eval) 2.52 K - 1.07x slower +24.85 μs +luerl 2.50 K - 1.07x slower +27.91 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.89 MB +lua (eval) 1.90 MB - 1.01x memory usage +0.0101 MB +luerl 1.90 MB - 1.00x memory usage +0.00886 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/cpu.txt b/bench_results/v0.4.0/cpu.txt new file mode 100644 index 00000000..de5c8ad6 --- /dev/null +++ b/bench_results/v0.4.0/cpu.txt @@ -0,0 +1 @@ +Apple M4 diff --git a/bench_results/v0.4.0/encode_decode.txt b/bench_results/v0.4.0/encode_decode.txt new file mode 100644 index 00000000..0f54da60 --- /dev/null +++ b/bench_results/v0.4.0/encode_decode.txt @@ -0,0 +1,128 @@ +lua 0.4.0 — encode!/decode! decomposition +(decode+deep_cast column: enabled) +================================================================================== +op shape N total_us per_elem_ns +encode int_list 8 0.27 33.3 +decode int_list 8 0.09 11.2 +dec+cast int_list 8 0.11 13.7 +encode int_list 64 1.61 25.1 +decode int_list 64 0.54 8.5 +dec+cast int_list 64 0.96 15.0 +encode int_list 512 26.38 51.5 +decode int_list 512 5.11 10.0 +dec+cast int_list 512 8.60 16.8 +encode int_list 4096 226.97 55.4 +decode int_list 4096 43.47 10.6 +dec+cast int_list 4096 88.67 21.6 +---------------------------------------------------------------------------------- +encode float_list 8 0.22 27.7 +decode float_list 8 0.09 11.2 +dec+cast float_list 8 0.11 13.2 +encode float_list 64 1.64 25.7 +decode float_list 64 0.55 8.5 +dec+cast float_list 64 0.94 14.7 +encode float_list 512 27.00 52.7 +decode float_list 512 5.85 11.4 +dec+cast float_list 512 9.00 17.6 +encode float_list 4096 239.42 58.5 +decode float_list 4096 64.74 15.8 +dec+cast float_list 4096 98.40 24.0 +---------------------------------------------------------------------------------- +encode bool_list 8 0.21 26.4 +decode bool_list 8 0.08 10.0 +dec+cast bool_list 8 0.10 12.6 +encode bool_list 64 1.60 25.0 +decode bool_list 64 0.56 8.7 +dec+cast bool_list 64 0.97 15.1 +encode bool_list 512 25.68 50.1 +decode bool_list 512 4.80 9.4 +dec+cast bool_list 512 8.38 16.4 +encode bool_list 4096 224.95 54.9 +decode bool_list 4096 38.99 9.5 +dec+cast bool_list 4096 84.05 20.5 +---------------------------------------------------------------------------------- +encode short_string_list 8 0.21 26.7 +decode short_string_list 8 0.08 10.5 +dec+cast short_string_list 8 0.10 12.9 +encode short_string_list 64 1.62 25.3 +decode short_string_list 64 0.55 8.6 +dec+cast short_string_list 64 0.98 15.4 +encode short_string_list 512 26.45 51.7 +decode short_string_list 512 4.86 9.5 +dec+cast short_string_list 512 9.05 17.7 +encode short_string_list 4096 202.93 49.5 +decode short_string_list 4096 43.46 10.6 +dec+cast short_string_list 4096 73.06 17.8 +---------------------------------------------------------------------------------- +encode long_string_list 8 0.22 27.2 +decode long_string_list 8 0.08 10.4 +dec+cast long_string_list 8 0.10 12.7 +encode long_string_list 64 1.67 26.1 +decode long_string_list 64 0.59 9.2 +dec+cast long_string_list 64 1.03 16.1 +encode long_string_list 512 28.42 55.5 +decode long_string_list 512 5.16 10.1 +dec+cast long_string_list 512 9.09 17.7 +encode long_string_list 4096 465.96 113.8 +decode long_string_list 4096 54.39 13.3 +dec+cast long_string_list 4096 108.40 26.5 +---------------------------------------------------------------------------------- +encode string_map 8 0.52 65.3 +decode string_map 8 0.07 8.2 +dec+cast string_map 8 0.17 21.1 +encode string_map 64 6.56 102.5 +decode string_map 64 0.40 6.2 +dec+cast string_map 64 3.21 50.2 +encode string_map 512 113.10 220.9 +decode string_map 512 4.27 8.3 +dec+cast string_map 512 39.39 76.9 +encode string_map 4096 1295.09 316.2 +decode string_map 4096 42.02 10.3 +dec+cast string_map 4096 367.76 89.8 +---------------------------------------------------------------------------------- +encode int_map 8 0.25 31.2 +decode int_map 8 0.09 10.7 +dec+cast int_map 8 0.10 12.9 +encode int_map 64 3.56 55.6 +decode int_map 64 0.57 8.9 +dec+cast int_map 64 0.97 15.1 +encode int_map 512 51.12 99.8 +decode int_map 512 4.79 9.4 +dec+cast int_map 512 8.76 17.1 +encode int_map 4096 443.17 108.2 +decode int_map 4096 40.30 9.8 +dec+cast int_map 4096 87.24 21.3 +---------------------------------------------------------------------------------- +encode record_list 8 2.07 258.7 +decode record_list 8 0.40 49.9 +dec+cast record_list 8 0.80 100.1 +encode record_list 64 31.54 492.8 +decode record_list 64 4.54 70.9 +dec+cast record_list 64 7.32 114.4 +encode record_list 512 287.96 562.4 +decode record_list 512 52.67 102.9 +dec+cast record_list 512 77.64 151.6 +encode record_list 4096 2346.38 572.8 +decode record_list 4096 406.88 99.3 +dec+cast record_list 4096 658.70 160.8 +---------------------------------------------------------------------------------- +nested chain (depth sweep) — isolates recursion/traversal from fan-out +op shape N total_us per_elem_ns +encode nested_chain 4 0.51 257.3 +decode nested_chain 4 0.12 61.7 +dec+cast nested_chain 4 0.24 118.7 +encode nested_chain 16 2.06 1032.0 +decode nested_chain 16 0.63 314.9 +dec+cast nested_chain 16 1.10 550.2 +encode nested_chain 64 8.53 4265.3 +decode nested_chain 64 4.16 2079.7 +dec+cast nested_chain 64 6.63 3317.0 +encode nested_chain 256 34.34 17170.4 +decode nested_chain 256 37.82 18909.7 +dec+cast nested_chain 256 47.43 23716.3 +================================================================================== +composite anchor — the PR's `original_nested` (matches the 18us/108us figure) +op shape N total_us per_elem_ns +encode original_nested 75 3.48 870.0 +decode original_nested 75 0.58 145.4 +dec+cast original_nested 75 1.32 329.3 diff --git a/bench_results/v0.4.0/environment.md b/bench_results/v0.4.0/environment.md new file mode 100644 index 00000000..897695d4 --- /dev/null +++ b/bench_results/v0.4.0/environment.md @@ -0,0 +1,53 @@ +# Benchmark environment — lua v0.4.0 (full mode) + +- **Ref**: `v0.4.0` (tag), commit `5bf2069` +- **CPU**: Apple M4 +- **OTP**: Erlang/OTP 29 [erts-17.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] +- **Elixir**: 1.20.0 (compiled with Erlang/OTP 29) +- **Mode**: `LUA_BENCH_MODE=full MIX_ENV=benchmark` +- **Setup timestamp**: 2026-07-28T13:59:36Z (UTC) +- **luaport (C Lua)**: not installed — skipped by all workloads as expected +- **luerl**: 1.5.1 (unconditional dep at this tag) + +## Worktree setup + +``` +git -C worktree add --detach v0.4.0 + +cp -R /benchmarks /benchmarks + +# mix.exs deps edit (worktree only): added +# {:benchee, "~> 1.3", only: :benchmark} +# alongside the existing {:luerl, "~> 1.5.1"} unconditional dep. + +MIX_ENV=benchmark mix deps.get +MIX_ENV=benchmark mix compile +``` + +No lockfile/dep conflicts were encountered; `deps.get` resolved cleanly (benchee 1.5.1, deep_merge 1.0.2, statistex 1.1.1 added; luerl/ex_doc/dialyxir unchanged). + +## Measurement commands + +Each workload run strictly serially, one at a time, via: + +``` +(cd && LUA_BENCH_MODE=full MIX_ENV=benchmark mix run benchmarks/.exs) \ + > /.txt 2>&1 +``` + +Order: fibonacci, closures, oop, string_ops, string_format, table_ops, patterns, metamethods, pcall_varargs, vm_new, encode_decode. + +Excluded: `dispatcher_vs_interpreter.exs` (crashes on v0.4.0), `array_vs_map_probe.exs` (runs no Lua). + +`encode_decode.exs` uses its own `:timer.tc` harness (not Benchee) and prints its own table directly. + +## Cleanup + +``` +git -C worktree remove --force +``` + +## Results + +Parsed into `summary.json` in this directory. Raw Benchee/harness stdout for each +workload is preserved verbatim in `.txt`. diff --git a/bench_results/v0.4.0/fibonacci.txt b/bench_results/v0.4.0/fibonacci.txt new file mode 100644 index 00000000..62e549cf --- /dev/null +++ b/bench_results/v0.4.0/fibonacci.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 1.42 703.34 ms ±1.00% 702.84 ms 715.45 ms +lua (eval) 1.40 712.63 ms ±1.13% 711.69 ms 732.64 ms +luerl 1.38 724.80 ms ±3.25% 720.40 ms 801.56 ms + +Comparison: +lua (chunk) 1.42 +lua (eval) 1.40 - 1.01x slower +9.29 ms +luerl 1.38 - 1.03x slower +21.46 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 2.45 GB +lua (eval) 2.45 GB - 1.00x memory usage +0.00001 GB +luerl 2.45 GB - 1.00x memory usage +0.00001 GB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/metamethods.txt b/bench_results/v0.4.0/metamethods.txt new file mode 100644 index 00000000..a60332a0 --- /dev/null +++ b/bench_results/v0.4.0/metamethods.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== metamethods: self-call method dispatch (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 2.04 K 489.21 μs ±10.11% 476.58 μs 672.87 μs +lua (eval) 1.99 K 503.66 μs ±9.40% 488.92 μs 678.61 μs +luerl 1.98 K 505.65 μs ±10.20% 492.79 μs 680.89 μs + +Comparison: +lua (chunk) 2.04 K +lua (eval) 1.99 K - 1.03x slower +14.45 μs +luerl 1.98 K - 1.03x slower +16.44 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.42 MB +lua (eval) 1.43 MB - 1.01x memory usage +0.0105 MB +luerl 1.43 MB - 1.01x memory usage +0.0103 MB + +**All measurements for memory usage were the same** + +=== metamethods: 3-level __index chain (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 4.58 K 218.46 μs ±4.01% 217.92 μs 233.19 μs +lua (chunk) 4.57 K 219.03 μs ±10.32% 216.42 μs 254.20 μs +lua (eval) 4.52 K 221.04 μs ±5.51% 219.33 μs 245.33 μs + +Comparison: +luerl 4.58 K +lua (chunk) 4.57 K - 1.00x slower +0.57 μs +lua (eval) 4.52 K - 1.01x slower +2.57 μs + +Memory usage statistics: + +Name Memory usage +luerl 668.59 KB +lua (chunk) 658.30 KB - 0.98x memory usage -10.28906 KB +lua (eval) 668.80 KB - 1.00x memory usage +0.22 KB + +**All measurements for memory usage were the same** + +=== metamethods: arithmetic/relational metamethods (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 519.76 1.92 ms ±9.38% 1.88 ms 2.25 ms +lua (chunk) 513.47 1.95 ms ±15.04% 1.88 ms 2.30 ms +lua (eval) 508.61 1.97 ms ±10.60% 1.89 ms 2.93 ms + +Comparison: +luerl 519.76 +lua (chunk) 513.47 - 1.01x slower +0.0236 ms +lua (eval) 508.61 - 1.02x slower +0.0422 ms + +Memory usage statistics: + +Name Memory usage +luerl 6.24 MB +lua (chunk) 6.23 MB - 1.00x memory usage -0.01392 MB +lua (eval) 6.24 MB - 1.00x memory usage -0.00303 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/oop.txt b/bench_results/v0.4.0/oop.txt new file mode 100644 index 00000000..b1671978 --- /dev/null +++ b/bench_results/v0.4.0/oop.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 9.11 K 109.76 μs ±13.66% 106.50 μs 179.25 μs +lua (eval) 8.85 K 113.01 μs ±18.72% 108.58 μs 193.98 μs +lua (chunk) 8.80 K 113.70 μs ±14.10% 110.88 μs 195.87 μs + +Comparison: +luerl 9.11 K +lua (eval) 8.85 K - 1.03x slower +3.25 μs +lua (chunk) 8.80 K - 1.04x slower +3.94 μs + +Memory usage statistics: + +Name Memory usage +luerl 381.45 KB +lua (eval) 382.52 KB - 1.00x memory usage +1.07 KB +lua (chunk) 372.03 KB - 0.98x memory usage -9.42188 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/patterns.txt b/bench_results/v0.4.0/patterns.txt new file mode 100644 index 00000000..cc66c55d --- /dev/null +++ b/bench_results/v0.4.0/patterns.txt @@ -0,0 +1,139 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== patterns: find/match field extraction (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 947.52 1.06 ms ±3.75% 1.06 ms 1.18 ms +luerl 937.83 1.07 ms ±6.93% 1.05 ms 1.22 ms +lua (chunk) 919.82 1.09 ms ±4.39% 1.07 ms 1.22 ms + +Comparison: +lua (eval) 947.52 +luerl 937.83 - 1.01x slower +0.0109 ms +lua (chunk) 919.82 - 1.03x slower +0.0318 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 6.70 MB +luerl 6.70 MB - 1.00x memory usage -0.00021 MB +lua (chunk) 6.69 MB - 1.00x memory usage -0.00996 MB + +**All measurements for memory usage were the same** + +=== patterns: find-based tokenizer (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 731.44 1.37 ms ±3.23% 1.36 ms 1.51 ms +lua (chunk) 721.24 1.39 ms ±4.36% 1.37 ms 1.57 ms +lua (eval) 715.30 1.40 ms ±4.16% 1.39 ms 1.49 ms + +Comparison: +luerl 731.44 +lua (chunk) 721.24 - 1.01x slower +0.0193 ms +lua (eval) 715.30 - 1.02x slower +0.0308 ms + +Memory usage statistics: + +Name Memory usage +luerl 6.65 MB +lua (chunk) 6.64 MB - 1.00x memory usage -0.00977 MB +lua (eval) 6.65 MB - 1.00x memory usage +0.00024 MB + +**All measurements for memory usage were the same** + +=== patterns: gsub template substitution (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 452.66 2.21 ms ±4.59% 2.21 ms 2.50 ms +lua (eval) 446.40 2.24 ms ±8.19% 2.19 ms 2.75 ms +lua (chunk) 436.67 2.29 ms ±10.44% 2.21 ms 3.18 ms + +Comparison: +luerl 452.66 +lua (eval) 446.40 - 1.01x slower +0.0310 ms +lua (chunk) 436.67 - 1.04x slower +0.0809 ms + +Memory usage statistics: + +Name average deviation median 99th % +luerl 11.75 MB ±0.00% 11.75 MB 11.75 MB +lua (eval) 11.75 MB ±0.00% 11.75 MB 11.75 MB +lua (chunk) 11.74 MB ±0.00% 11.74 MB 11.74 MB + +Comparison: +luerl 11.75 MB +lua (eval) 11.75 MB - 1.00x memory usage +0.00021 MB +lua (chunk) 11.74 MB - 1.00x memory usage -0.00975 MB diff --git a/bench_results/v0.4.0/pcall_varargs.txt b/bench_results/v0.4.0/pcall_varargs.txt new file mode 100644 index 00000000..efb881e1 --- /dev/null +++ b/bench_results/v0.4.0/pcall_varargs.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== call protocol: pcall, success path (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 3.44 K 291.08 μs ±3.72% 289.67 μs 313 μs +lua (eval) 3.42 K 292.45 μs ±3.54% 291.17 μs 315.44 μs +lua (chunk) 3.40 K 293.76 μs ±7.51% 290.50 μs 338.21 μs + +Comparison: +luerl 3.44 K +lua (eval) 3.42 K - 1.00x slower +1.36 μs +lua (chunk) 3.40 K - 1.01x slower +2.68 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.11 MB +lua (eval) 1.11 MB - 1.00x memory usage +0.00021 MB +lua (chunk) 1.10 MB - 0.99x memory usage -0.00993 MB + +**All measurements for memory usage were the same** + +=== call protocol: pcall, raise + catch (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 2.13 K 468.84 μs ±5.39% 466.38 μs 505.90 μs +lua (eval) 2.13 K 470.00 μs ±3.58% 467.50 μs 498.92 μs +lua (chunk) 2.11 K 474.55 μs ±9.26% 469.29 μs 583.95 μs + +Comparison: +luerl 2.13 K +lua (eval) 2.13 K - 1.00x slower +1.15 μs +lua (chunk) 2.11 K - 1.01x slower +5.70 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.55 MB +lua (eval) 1.55 MB - 1.00x memory usage +0.00021 MB +lua (chunk) 1.54 MB - 0.99x memory usage -0.01002 MB + +**All measurements for memory usage were the same** + +=== call protocol: varargs + multiple returns (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 418.17 2.39 ms ±4.44% 2.36 ms 2.70 ms +lua (eval) 400.02 2.50 ms ±11.76% 2.39 ms 3.48 ms +lua (chunk) 391.27 2.56 ms ±14.33% 2.40 ms 3.52 ms + +Comparison: +luerl 418.17 +lua (eval) 400.02 - 1.05x slower +0.109 ms +lua (chunk) 391.27 - 1.07x slower +0.164 ms + +Memory usage statistics: + +Name Memory usage +luerl 8.73 MB +lua (eval) 8.73 MB - 1.00x memory usage -0.00005 MB +lua (chunk) 8.72 MB - 1.00x memory usage -0.00979 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/string_format.txt b/bench_results/v0.4.0/string_format.txt new file mode 100644 index 00000000..b992107a --- /dev/null +++ b/bench_results/v0.4.0/string_format.txt @@ -0,0 +1,139 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== string.format: long literal-heavy format string (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 249.24 4.01 ms ±6.43% 3.91 ms 4.54 ms +luerl 242.35 4.13 ms ±6.17% 4.25 ms 4.54 ms +lua (eval) 187.95 5.32 ms ±9.84% 5.58 ms 6.77 ms + +Comparison: +lua (chunk) 249.24 +luerl 242.35 - 1.03x slower +0.114 ms +lua (eval) 187.95 - 1.33x slower +1.31 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 20.36 MB +luerl 22.59 MB - 1.11x memory usage +2.23 MB +lua (eval) 20.38 MB - 1.00x memory usage +0.0172 MB + +**All measurements for memory usage were the same** + +=== string.format: width-flagged specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 582.97 1.72 ms ±3.52% 1.72 ms 1.86 ms +lua (eval) 576.12 1.74 ms ±8.52% 1.70 ms 2.41 ms +luerl 572.77 1.75 ms ±8.30% 1.73 ms 2.14 ms + +Comparison: +lua (chunk) 582.97 +lua (eval) 576.12 - 1.01x slower +0.0204 ms +luerl 572.77 - 1.02x slower +0.0306 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 7.53 MB +lua (eval) 7.55 MB - 1.00x memory usage +0.0192 MB +luerl 7.55 MB - 1.00x memory usage +0.0161 MB + +**All measurements for memory usage were the same** + +=== string.format: many specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 370.09 2.70 ms ±4.67% 2.78 ms 2.89 ms +lua (chunk) 367.47 2.72 ms ±4.88% 2.75 ms 2.99 ms +luerl 362.27 2.76 ms ±3.93% 2.80 ms 2.99 ms + +Comparison: +lua (eval) 370.09 +lua (chunk) 367.47 - 1.01x slower +0.0193 ms +luerl 362.27 - 1.02x slower +0.0584 ms + +Memory usage statistics: + +Name average deviation median 99th % +lua (eval) 13.76 MB ±0.00% 13.76 MB 13.76 MB +lua (chunk) 13.73 MB ±0.00% 13.73 MB 13.73 MB +luerl 13.76 MB ±0.00% 13.76 MB 13.76 MB + +Comparison: +lua (eval) 13.76 MB +lua (chunk) 13.73 MB - 1.00x memory usage -0.03384 MB +luerl 13.76 MB - 1.00x memory usage -0.00677 MB diff --git a/bench_results/v0.4.0/string_ops.txt b/bench_results/v0.4.0/string_ops.txt new file mode 100644 index 00000000..7823a7a2 --- /dev/null +++ b/bench_results/v0.4.0/string_ops.txt @@ -0,0 +1,91 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== String Concatenation via table.concat (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 25.94 K 38.55 μs ±12.21% 38.38 μs 58.63 μs +lua (eval) 25.35 K 39.45 μs ±6.04% 39.54 μs 45.04 μs +luerl 25.19 K 39.69 μs ±19.25% 39.42 μs 46.88 μs + +Comparison: +lua (chunk) 25.94 K +lua (eval) 25.35 K - 1.02x slower +0.89 μs +luerl 25.19 K - 1.03x slower +1.14 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 161.98 KB +lua (eval) 172.77 KB - 1.07x memory usage +10.79 KB +luerl 172.42 KB - 1.06x memory usage +10.44 KB + +**All measurements for memory usage were the same** + +=== String Formatting via string.format (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 9.75 K 102.58 μs ±11.83% 101.46 μs 124.58 μs +lua (eval) 9.70 K 103.12 μs ±7.60% 102.33 μs 124.83 μs +luerl 9.63 K 103.90 μs ±9.83% 102.83 μs 127.33 μs + +Comparison: +lua (chunk) 9.75 K +lua (eval) 9.70 K - 1.01x slower +0.54 μs +luerl 9.63 K - 1.01x slower +1.31 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 577.43 KB +lua (eval) 587.10 KB - 1.02x memory usage +9.67 KB +luerl 588.84 KB - 1.02x memory usage +11.41 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/summary.json b/bench_results/v0.4.0/summary.json new file mode 100644 index 00000000..d0fda5d8 --- /dev/null +++ b/bench_results/v0.4.0/summary.json @@ -0,0 +1,1257 @@ +{ + "fibonacci": { + "(single case)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "1.42", + "average": "703.34 ms", + "deviation": "±1.00%", + "median": "702.84 ms", + "p99": "715.45 ms", + "memory": "2.45 GB" + }, + { + "name": "lua (eval)", + "ips": "1.40", + "average": "712.63 ms", + "deviation": "±1.13%", + "median": "711.69 ms", + "p99": "732.64 ms", + "memory": "2.45 GB" + }, + { + "name": "luerl", + "ips": "1.38", + "average": "724.80 ms", + "deviation": "±3.25%", + "median": "720.40 ms", + "p99": "801.56 ms", + "memory": "2.45 GB" + } + ], + "comparison": [ + "lua (chunk) 1.42", + "lua (eval) 1.40 - 1.01x slower +9.29 ms", + "luerl 1.38 - 1.03x slower +21.46 ms" + ] + } + }, + "closures": { + "(single case)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "2.69 K", + "average": "372.35 μs", + "deviation": "±7.42%", + "median": "368.92 μs", + "p99": "507.15 μs", + "memory": "1.89 MB" + }, + { + "name": "lua (eval)", + "ips": "2.52 K", + "average": "397.20 μs", + "deviation": "±8.30%", + "median": "390.13 μs", + "p99": "515.81 μs", + "memory": "1.90 MB" + }, + { + "name": "luerl", + "ips": "2.50 K", + "average": "400.26 μs", + "deviation": "±9.75%", + "median": "391.42 μs", + "p99": "536.69 μs", + "memory": "1.90 MB" + } + ], + "comparison": [ + "lua (chunk) 2.69 K", + "lua (eval) 2.52 K - 1.07x slower +24.85 μs", + "luerl 2.50 K - 1.07x slower +27.91 μs" + ] + } + }, + "oop": { + "(single case)": { + "jobs": [ + { + "name": "luerl", + "ips": "9.11 K", + "average": "109.76 μs", + "deviation": "±13.66%", + "median": "106.50 μs", + "p99": "179.25 μs", + "memory": "381.45 KB" + }, + { + "name": "lua (eval)", + "ips": "8.85 K", + "average": "113.01 μs", + "deviation": "±18.72%", + "median": "108.58 μs", + "p99": "193.98 μs", + "memory": "382.52 KB" + }, + { + "name": "lua (chunk)", + "ips": "8.80 K", + "average": "113.70 μs", + "deviation": "±14.10%", + "median": "110.88 μs", + "p99": "195.87 μs", + "memory": "372.03 KB" + } + ], + "comparison": [ + "luerl 9.11 K", + "lua (eval) 8.85 K - 1.03x slower +3.25 μs", + "lua (chunk) 8.80 K - 1.04x slower +3.94 μs" + ] + } + }, + "string_ops": { + "String Concatenation via table.concat (n=100) (mode: full)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "25.94 K", + "average": "38.55 μs", + "deviation": "±12.21%", + "median": "38.38 μs", + "p99": "58.63 μs", + "memory": "161.98 KB" + }, + { + "name": "lua (eval)", + "ips": "25.35 K", + "average": "39.45 μs", + "deviation": "±6.04%", + "median": "39.54 μs", + "p99": "45.04 μs", + "memory": "172.77 KB" + }, + { + "name": "luerl", + "ips": "25.19 K", + "average": "39.69 μs", + "deviation": "±19.25%", + "median": "39.42 μs", + "p99": "46.88 μs", + "memory": "172.42 KB" + } + ], + "comparison": [ + "lua (chunk) 25.94 K", + "lua (eval) 25.35 K - 1.02x slower +0.89 μs", + "luerl 25.19 K - 1.03x slower +1.14 μs" + ] + }, + "String Formatting via string.format (n=100) (mode: full)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "9.75 K", + "average": "102.58 μs", + "deviation": "±11.83%", + "median": "101.46 μs", + "p99": "124.58 μs", + "memory": "577.43 KB" + }, + { + "name": "lua (eval)", + "ips": "9.70 K", + "average": "103.12 μs", + "deviation": "±7.60%", + "median": "102.33 μs", + "p99": "124.83 μs", + "memory": "587.10 KB" + }, + { + "name": "luerl", + "ips": "9.63 K", + "average": "103.90 μs", + "deviation": "±9.83%", + "median": "102.83 μs", + "p99": "127.33 μs", + "memory": "588.84 KB" + } + ], + "comparison": [ + "lua (chunk) 9.75 K", + "lua (eval) 9.70 K - 1.01x slower +0.54 μs", + "luerl 9.63 K - 1.01x slower +1.31 μs" + ] + } + }, + "string_format": { + "string.format: long literal-heavy format string (n=1000) (mode: full)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "249.24", + "average": "4.01 ms", + "deviation": "±6.43%", + "median": "3.91 ms", + "p99": "4.54 ms", + "memory": "20.36 MB" + }, + { + "name": "luerl", + "ips": "242.35", + "average": "4.13 ms", + "deviation": "±6.17%", + "median": "4.25 ms", + "p99": "4.54 ms", + "memory": "22.59 MB" + }, + { + "name": "lua (eval)", + "ips": "187.95", + "average": "5.32 ms", + "deviation": "±9.84%", + "median": "5.58 ms", + "p99": "6.77 ms", + "memory": "20.38 MB" + } + ], + "comparison": [ + "lua (chunk) 249.24", + "luerl 242.35 - 1.03x slower +0.114 ms", + "lua (eval) 187.95 - 1.33x slower +1.31 ms" + ] + }, + "string.format: width-flagged specifiers (n=1000) (mode: full)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "582.97", + "average": "1.72 ms", + "deviation": "±3.52%", + "median": "1.72 ms", + "p99": "1.86 ms", + "memory": "7.53 MB" + }, + { + "name": "lua (eval)", + "ips": "576.12", + "average": "1.74 ms", + "deviation": "±8.52%", + "median": "1.70 ms", + "p99": "2.41 ms", + "memory": "7.55 MB" + }, + { + "name": "luerl", + "ips": "572.77", + "average": "1.75 ms", + "deviation": "±8.30%", + "median": "1.73 ms", + "p99": "2.14 ms", + "memory": "7.55 MB" + } + ], + "comparison": [ + "lua (chunk) 582.97", + "lua (eval) 576.12 - 1.01x slower +0.0204 ms", + "luerl 572.77 - 1.02x slower +0.0306 ms" + ] + }, + "string.format: many specifiers (n=1000) (mode: full)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "370.09", + "average": "2.70 ms", + "deviation": "±4.67%", + "median": "2.78 ms", + "p99": "2.89 ms", + "memory": "13.76 MB" + }, + { + "name": "lua (chunk)", + "ips": "367.47", + "average": "2.72 ms", + "deviation": "±4.88%", + "median": "2.75 ms", + "p99": "2.99 ms", + "memory": "13.73 MB" + }, + { + "name": "luerl", + "ips": "362.27", + "average": "2.76 ms", + "deviation": "±3.93%", + "median": "2.80 ms", + "p99": "2.99 ms", + "memory": "13.76 MB" + } + ], + "comparison": [ + "lua (eval) 370.09", + "lua (chunk) 367.47 - 1.01x slower +0.0193 ms", + "luerl 362.27 - 1.02x slower +0.0584 ms" + ], + "memory_comparison": [ + "lua (eval) 13.76 MB", + "lua (chunk) 13.73 MB - 1.00x memory usage -0.03384 MB", + "luerl 13.76 MB - 1.00x memory usage -0.00677 MB" + ] + } + }, + "table_ops": { + "Table Build (mode: full)": { + "inputs": { + "With input large (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "6.50 K", + "average": "153.83 μs", + "deviation": "±11.45%", + "median": "151.96 μs", + "p99": "195.78 μs", + "memory": "993.93 KB" + }, + { + "name": "luerl", + "ips": "6.44 K", + "average": "155.20 μs", + "deviation": "±14.07%", + "median": "152.33 μs", + "p99": "259.75 μs", + "memory": "996.96 KB" + }, + { + "name": "lua (chunk)", + "ips": "6.35 K", + "average": "157.51 μs", + "deviation": "±18.23%", + "median": "149.92 μs", + "p99": "262.79 μs", + "memory": "983.38 KB" + } + ], + "comparison": [ + "lua (eval) 6.50 K", + "luerl 6.44 K - 1.01x slower +1.37 μs", + "lua (chunk) 6.35 K - 1.02x slower +3.68 μs" + ] + }, + "With input medium (n=100)": { + "jobs": [ + { + "name": "luerl", + "ips": "59.29 K", + "average": "16.87 μs", + "deviation": "±38.96%", + "median": "16.25 μs", + "p99": "26.79 μs", + "memory": "110.78 KB" + }, + { + "name": "lua (eval)", + "ips": "51.13 K", + "average": "19.56 μs", + "deviation": "±34.91%", + "median": "16.46 μs", + "p99": "41.71 μs", + "memory": "111.19 KB" + }, + { + "name": "lua (chunk)", + "ips": "43.75 K", + "average": "22.86 μs", + "deviation": "±38.05%", + "median": "25.96 μs", + "p99": "47.33 μs", + "memory": "100.13 KB" + } + ], + "comparison": [ + "luerl 59.29 K", + "lua (eval) 51.13 K - 1.16x slower +2.69 μs", + "lua (chunk) 43.75 K - 1.36x slower +5.99 μs" + ] + }, + "With input small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "405.22 K", + "average": "2.47 μs", + "deviation": "±243.17%", + "median": "2 μs", + "p99": "5.83 μs", + "memory": "12.45 KB" + }, + { + "name": "luerl", + "ips": "321.06 K", + "average": "3.11 μs", + "deviation": "±187.65%", + "median": "3 μs", + "p99": "4.54 μs", + "memory": "22.62 KB" + }, + { + "name": "lua (eval)", + "ips": "282.18 K", + "average": "3.54 μs", + "deviation": "±178.79%", + "median": "3.13 μs", + "p99": "9.29 μs", + "memory": "23.02 KB" + } + ], + "comparison": [ + "lua (chunk) 405.22 K", + "luerl 321.06 K - 1.26x slower +0.65 μs", + "lua (eval) 282.18 K - 1.44x slower +1.08 μs" + ] + } + } + }, + "Table Sort (mode: full)": { + "inputs": { + "With input large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "5.61 K", + "average": "178.37 μs", + "deviation": "±12.59%", + "median": "174.29 μs", + "p99": "234.12 μs", + "memory": "1.17 MB" + }, + { + "name": "lua (eval)", + "ips": "5.45 K", + "average": "183.60 μs", + "deviation": "±106.15%", + "median": "177.67 μs", + "p99": "334.99 μs", + "memory": "1.18 MB" + }, + { + "name": "luerl", + "ips": "4.88 K", + "average": "205.11 μs", + "deviation": "±25.14%", + "median": "180.59 μs", + "p99": "336.78 μs", + "memory": "1.18 MB" + } + ], + "comparison": [ + "lua (chunk) 5.61 K", + "lua (eval) 5.45 K - 1.03x slower +5.23 μs", + "luerl 4.88 K - 1.15x slower +26.74 μs" + ] + }, + "With input medium (n=100)": { + "jobs": [ + { + "name": "luerl", + "ips": "51.03 K", + "average": "19.60 μs", + "deviation": "±20.90%", + "median": "19.08 μs", + "p99": "28.42 μs", + "memory": "133.34 KB" + }, + { + "name": "lua (chunk)", + "ips": "47.42 K", + "average": "21.09 μs", + "deviation": "±42.23%", + "median": "18.04 μs", + "p99": "51.33 μs", + "memory": "122.98 KB" + }, + { + "name": "lua (eval)", + "ips": "46.64 K", + "average": "21.44 μs", + "deviation": "±39.09%", + "median": "19.33 μs", + "p99": "45.92 μs", + "memory": "133.59 KB" + } + ], + "comparison": [ + "luerl 51.03 K", + "lua (chunk) 47.42 K - 1.08x slower +1.49 μs", + "lua (eval) 46.64 K - 1.09x slower +1.85 μs" + ] + }, + "With input small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "324.41 K", + "average": "3.08 μs", + "deviation": "±194.26%", + "median": "2.54 μs", + "p99": "7.08 μs", + "memory": "15.84 KB" + }, + { + "name": "luerl", + "ips": "271.87 K", + "average": "3.68 μs", + "deviation": "±124.45%", + "median": "3.54 μs", + "p99": "10 μs", + "memory": "25.98 KB" + }, + { + "name": "lua (eval)", + "ips": "245.19 K", + "average": "4.08 μs", + "deviation": "±136.48%", + "median": "3.67 μs", + "p99": "8.25 μs", + "memory": "26.25 KB" + } + ], + "comparison": [ + "lua (chunk) 324.41 K", + "luerl 271.87 K - 1.19x slower +0.60 μs", + "lua (eval) 245.19 K - 1.32x slower +1.00 μs" + ] + } + } + }, + "Table Iterate/Sum (mode: full)": { + "inputs": { + "With input large (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "3.81 K", + "average": "262.24 μs", + "deviation": "±19.86%", + "median": "243.42 μs", + "p99": "444.12 μs", + "memory": "1.35 MB" + }, + { + "name": "luerl", + "ips": "3.75 K", + "average": "266.52 μs", + "deviation": "±20.59%", + "median": "243.13 μs", + "p99": "416.21 μs", + "memory": "1.35 MB" + }, + { + "name": "lua (chunk)", + "ips": "3.60 K", + "average": "278.11 μs", + "deviation": "±24.12%", + "median": "242.42 μs", + "p99": "456.96 μs", + "memory": "1.34 MB" + } + ], + "comparison": [ + "lua (eval) 3.81 K", + "luerl 3.75 K - 1.02x slower +4.28 μs", + "lua (chunk) 3.60 K - 1.06x slower +15.87 μs" + ] + }, + "With input medium (n=100)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "37.84 K", + "average": "26.43 μs", + "deviation": "±9.28%", + "median": "26.08 μs", + "p99": "29.92 μs", + "memory": "149.42 KB" + }, + { + "name": "luerl", + "ips": "37.50 K", + "average": "26.66 μs", + "deviation": "±86.90%", + "median": "26.21 μs", + "p99": "36.50 μs", + "memory": "149.28 KB" + }, + { + "name": "lua (chunk)", + "ips": "35.12 K", + "average": "28.47 μs", + "deviation": "±26.41%", + "median": "25.17 μs", + "p99": "53.58 μs", + "memory": "139.25 KB" + } + ], + "comparison": [ + "lua (eval) 37.84 K", + "luerl 37.50 K - 1.01x slower +0.24 μs", + "lua (chunk) 35.12 K - 1.08x slower +2.05 μs" + ] + }, + "With input small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "245.60 K", + "average": "4.07 μs", + "deviation": "±313.25%", + "median": "3.29 μs", + "p99": "9.79 μs", + "memory": "16.70 KB" + }, + { + "name": "luerl", + "ips": "227.18 K", + "average": "4.40 μs", + "deviation": "±102.94%", + "median": "4.29 μs", + "p99": "10.88 μs", + "memory": "26.80 KB" + }, + { + "name": "lua (eval)", + "ips": "206.12 K", + "average": "4.85 μs", + "deviation": "±120.13%", + "median": "4.33 μs", + "p99": "9.83 μs", + "memory": "27.20 KB" + } + ], + "comparison": [ + "lua (chunk) 245.60 K", + "luerl 227.18 K - 1.08x slower +0.33 μs", + "lua (eval) 206.12 K - 1.19x slower +0.78 μs" + ] + } + } + }, + "Table Map + Reduce (mode: full)": { + "inputs": { + "With input large (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "2.21 K", + "average": "452.53 μs", + "deviation": "±7.79%", + "median": "446.25 μs", + "p99": "614.89 μs", + "memory": "2.45 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.19 K", + "average": "455.66 μs", + "deviation": "±9.09%", + "median": "444.34 μs", + "p99": "628.64 μs", + "memory": "2.43 MB" + }, + { + "name": "luerl", + "ips": "2.19 K", + "average": "456.58 μs", + "deviation": "±12.12%", + "median": "442.21 μs", + "p99": "659.87 μs", + "memory": "2.45 MB" + } + ], + "comparison": [ + "lua (eval) 2.21 K", + "lua (chunk) 2.19 K - 1.01x slower +3.14 μs", + "luerl 2.19 K - 1.01x slower +4.05 μs" + ] + }, + "With input medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "21.79 K", + "average": "45.90 μs", + "deviation": "±12.29%", + "median": "45.25 μs", + "p99": "53.13 μs", + "memory": "252.38 KB" + }, + { + "name": "luerl", + "ips": "21.47 K", + "average": "46.58 μs", + "deviation": "±5.77%", + "median": "46.42 μs", + "p99": "52.54 μs", + "memory": "262.48 KB" + }, + { + "name": "lua (eval)", + "ips": "21.44 K", + "average": "46.65 μs", + "deviation": "±6.16%", + "median": "46.50 μs", + "p99": "51.96 μs", + "memory": "263.02 KB" + } + ], + "comparison": [ + "lua (chunk) 21.79 K", + "luerl 21.47 K - 1.01x slower +0.68 μs", + "lua (eval) 21.44 K - 1.02x slower +0.75 μs" + ] + }, + "With input small (n=10)": { + "jobs": [ + { + "name": "luerl", + "ips": "148.83 K", + "average": "6.72 μs", + "deviation": "±69.12%", + "median": "6.58 μs", + "p99": "14.71 μs", + "memory": "39.39 KB" + }, + { + "name": "lua (chunk)", + "ips": "145.39 K", + "average": "6.88 μs", + "deviation": "±90.79%", + "median": "5.54 μs", + "p99": "14.96 μs", + "memory": "29.06 KB" + }, + { + "name": "lua (eval)", + "ips": "131.69 K", + "average": "7.59 μs", + "deviation": "±88.05%", + "median": "6.71 μs", + "p99": "16.42 μs", + "memory": "39.80 KB" + } + ], + "comparison": [ + "luerl 148.83 K", + "lua (chunk) 145.39 K - 1.02x slower +0.159 μs", + "lua (eval) 131.69 K - 1.13x slower +0.87 μs" + ] + } + } + }, + "Table Pairs (hash) (mode: full)": { + "inputs": { + "With input large (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "760.68", + "average": "1.31 ms", + "deviation": "±5.02%", + "median": "1.32 ms", + "p99": "1.45 ms", + "memory": "2.48 MB" + }, + { + "name": "lua (chunk)", + "ips": "759.91", + "average": "1.32 ms", + "deviation": "±5.40%", + "median": "1.32 ms", + "p99": "1.49 ms", + "memory": "2.47 MB" + }, + { + "name": "luerl", + "ips": "754.19", + "average": "1.33 ms", + "deviation": "±6.28%", + "median": "1.33 ms", + "p99": "1.45 ms", + "memory": "2.48 MB" + } + ], + "comparison": [ + "lua (eval) 760.68", + "lua (chunk) 759.91 - 1.00x slower +0.00133 ms", + "luerl 754.19 - 1.01x slower +0.0113 ms" + ] + }, + "With input medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "10.96 K", + "average": "91.23 μs", + "deviation": "±12.10%", + "median": "89.92 μs", + "p99": "135.19 μs", + "memory": "238.58 KB" + }, + { + "name": "lua (eval)", + "ips": "10.20 K", + "average": "97.99 μs", + "deviation": "±11.84%", + "median": "97.42 μs", + "p99": "138.25 μs", + "memory": "249.40 KB" + }, + { + "name": "luerl", + "ips": "9.96 K", + "average": "100.37 μs", + "deviation": "±20.92%", + "median": "99.96 μs", + "p99": "144.67 μs", + "memory": "248.99 KB" + } + ], + "comparison": [ + "lua (chunk) 10.96 K", + "lua (eval) 10.20 K - 1.07x slower +6.76 μs", + "luerl 9.96 K - 1.10x slower +9.14 μs" + ] + }, + "With input small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "126.80 K", + "average": "7.89 μs", + "deviation": "±150.76%", + "median": "6.50 μs", + "p99": "18.79 μs", + "memory": "25.83 KB" + }, + { + "name": "luerl", + "ips": "125.64 K", + "average": "7.96 μs", + "deviation": "±72.08%", + "median": "7.29 μs", + "p99": "24.29 μs", + "memory": "36.16 KB" + }, + { + "name": "lua (eval)", + "ips": "109.11 K", + "average": "9.17 μs", + "deviation": "±61.17%", + "median": "9.38 μs", + "p99": "28.33 μs", + "memory": "36.56 KB" + } + ], + "comparison": [ + "lua (chunk) 126.80 K", + "luerl 125.64 K - 1.01x slower +0.0728 μs", + "lua (eval) 109.11 K - 1.16x slower +1.28 μs" + ] + } + } + } + }, + "patterns": { + "patterns: find/match field extraction (n=200) (mode: full)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "947.52", + "average": "1.06 ms", + "deviation": "±3.75%", + "median": "1.06 ms", + "p99": "1.18 ms", + "memory": "6.70 MB" + }, + { + "name": "luerl", + "ips": "937.83", + "average": "1.07 ms", + "deviation": "±6.93%", + "median": "1.05 ms", + "p99": "1.22 ms", + "memory": "6.70 MB" + }, + { + "name": "lua (chunk)", + "ips": "919.82", + "average": "1.09 ms", + "deviation": "±4.39%", + "median": "1.07 ms", + "p99": "1.22 ms", + "memory": "6.69 MB" + } + ], + "comparison": [ + "lua (eval) 947.52", + "luerl 937.83 - 1.01x slower +0.0109 ms", + "lua (chunk) 919.82 - 1.03x slower +0.0318 ms" + ] + }, + "patterns: find-based tokenizer (n=200) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "731.44", + "average": "1.37 ms", + "deviation": "±3.23%", + "median": "1.36 ms", + "p99": "1.51 ms", + "memory": "6.65 MB" + }, + { + "name": "lua (chunk)", + "ips": "721.24", + "average": "1.39 ms", + "deviation": "±4.36%", + "median": "1.37 ms", + "p99": "1.57 ms", + "memory": "6.64 MB" + }, + { + "name": "lua (eval)", + "ips": "715.30", + "average": "1.40 ms", + "deviation": "±4.16%", + "median": "1.39 ms", + "p99": "1.49 ms", + "memory": "6.65 MB" + } + ], + "comparison": [ + "luerl 731.44", + "lua (chunk) 721.24 - 1.01x slower +0.0193 ms", + "lua (eval) 715.30 - 1.02x slower +0.0308 ms" + ] + }, + "patterns: gsub template substitution (n=200) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "452.66", + "average": "2.21 ms", + "deviation": "±4.59%", + "median": "2.21 ms", + "p99": "2.50 ms", + "memory": "11.75 MB" + }, + { + "name": "lua (eval)", + "ips": "446.40", + "average": "2.24 ms", + "deviation": "±8.19%", + "median": "2.19 ms", + "p99": "2.75 ms", + "memory": "11.75 MB" + }, + { + "name": "lua (chunk)", + "ips": "436.67", + "average": "2.29 ms", + "deviation": "±10.44%", + "median": "2.21 ms", + "p99": "3.18 ms", + "memory": "11.74 MB" + } + ], + "comparison": [ + "luerl 452.66", + "lua (eval) 446.40 - 1.01x slower +0.0310 ms", + "lua (chunk) 436.67 - 1.04x slower +0.0809 ms" + ], + "memory_comparison": [ + "luerl 11.75 MB", + "lua (eval) 11.75 MB - 1.00x memory usage +0.00021 MB", + "lua (chunk) 11.74 MB - 1.00x memory usage -0.00975 MB" + ] + } + }, + "metamethods": { + "metamethods: self-call method dispatch (n=200) (mode: full)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "2.04 K", + "average": "489.21 μs", + "deviation": "±10.11%", + "median": "476.58 μs", + "p99": "672.87 μs", + "memory": "1.42 MB" + }, + { + "name": "lua (eval)", + "ips": "1.99 K", + "average": "503.66 μs", + "deviation": "±9.40%", + "median": "488.92 μs", + "p99": "678.61 μs", + "memory": "1.43 MB" + }, + { + "name": "luerl", + "ips": "1.98 K", + "average": "505.65 μs", + "deviation": "±10.20%", + "median": "492.79 μs", + "p99": "680.89 μs", + "memory": "1.43 MB" + } + ], + "comparison": [ + "lua (chunk) 2.04 K", + "lua (eval) 1.99 K - 1.03x slower +14.45 μs", + "luerl 1.98 K - 1.03x slower +16.44 μs" + ] + }, + "metamethods: 3-level __index chain (n=200) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "4.58 K", + "average": "218.46 μs", + "deviation": "±4.01%", + "median": "217.92 μs", + "p99": "233.19 μs", + "memory": "668.59 KB" + }, + { + "name": "lua (chunk)", + "ips": "4.57 K", + "average": "219.03 μs", + "deviation": "±10.32%", + "median": "216.42 μs", + "p99": "254.20 μs", + "memory": "658.30 KB" + }, + { + "name": "lua (eval)", + "ips": "4.52 K", + "average": "221.04 μs", + "deviation": "±5.51%", + "median": "219.33 μs", + "p99": "245.33 μs", + "memory": "668.80 KB" + } + ], + "comparison": [ + "luerl 4.58 K", + "lua (chunk) 4.57 K - 1.00x slower +0.57 μs", + "lua (eval) 4.52 K - 1.01x slower +2.57 μs" + ] + }, + "metamethods: arithmetic/relational metamethods (n=200) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "519.76", + "average": "1.92 ms", + "deviation": "±9.38%", + "median": "1.88 ms", + "p99": "2.25 ms", + "memory": "6.24 MB" + }, + { + "name": "lua (chunk)", + "ips": "513.47", + "average": "1.95 ms", + "deviation": "±15.04%", + "median": "1.88 ms", + "p99": "2.30 ms", + "memory": "6.23 MB" + }, + { + "name": "lua (eval)", + "ips": "508.61", + "average": "1.97 ms", + "deviation": "±10.60%", + "median": "1.89 ms", + "p99": "2.93 ms", + "memory": "6.24 MB" + } + ], + "comparison": [ + "luerl 519.76", + "lua (chunk) 513.47 - 1.01x slower +0.0236 ms", + "lua (eval) 508.61 - 1.02x slower +0.0422 ms" + ] + } + }, + "pcall_varargs": { + "call protocol: pcall, success path (n=500) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "3.44 K", + "average": "291.08 μs", + "deviation": "±3.72%", + "median": "289.67 μs", + "p99": "313 μs", + "memory": "1.11 MB" + }, + { + "name": "lua (eval)", + "ips": "3.42 K", + "average": "292.45 μs", + "deviation": "±3.54%", + "median": "291.17 μs", + "p99": "315.44 μs", + "memory": "1.11 MB" + }, + { + "name": "lua (chunk)", + "ips": "3.40 K", + "average": "293.76 μs", + "deviation": "±7.51%", + "median": "290.50 μs", + "p99": "338.21 μs", + "memory": "1.10 MB" + } + ], + "comparison": [ + "luerl 3.44 K", + "lua (eval) 3.42 K - 1.00x slower +1.36 μs", + "lua (chunk) 3.40 K - 1.01x slower +2.68 μs" + ] + }, + "call protocol: pcall, raise + catch (n=500) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "2.13 K", + "average": "468.84 μs", + "deviation": "±5.39%", + "median": "466.38 μs", + "p99": "505.90 μs", + "memory": "1.55 MB" + }, + { + "name": "lua (eval)", + "ips": "2.13 K", + "average": "470.00 μs", + "deviation": "±3.58%", + "median": "467.50 μs", + "p99": "498.92 μs", + "memory": "1.55 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.11 K", + "average": "474.55 μs", + "deviation": "±9.26%", + "median": "469.29 μs", + "p99": "583.95 μs", + "memory": "1.54 MB" + } + ], + "comparison": [ + "luerl 2.13 K", + "lua (eval) 2.13 K - 1.00x slower +1.15 μs", + "lua (chunk) 2.11 K - 1.01x slower +5.70 μs" + ] + }, + "call protocol: varargs + multiple returns (n=500) (mode: full)": { + "jobs": [ + { + "name": "luerl", + "ips": "418.17", + "average": "2.39 ms", + "deviation": "±4.44%", + "median": "2.36 ms", + "p99": "2.70 ms", + "memory": "8.73 MB" + }, + { + "name": "lua (eval)", + "ips": "400.02", + "average": "2.50 ms", + "deviation": "±11.76%", + "median": "2.39 ms", + "p99": "3.48 ms", + "memory": "8.73 MB" + }, + { + "name": "lua (chunk)", + "ips": "391.27", + "average": "2.56 ms", + "deviation": "±14.33%", + "median": "2.40 ms", + "p99": "3.52 ms", + "memory": "8.72 MB" + } + ], + "comparison": [ + "luerl 418.17", + "lua (eval) 400.02 - 1.05x slower +0.109 ms", + "lua (chunk) 391.27 - 1.07x slower +0.164 ms" + ] + } + }, + "vm_new": { + "VM instantiation: Lua.new/1 vs :luerl.init/0 (mode: full)": { + "jobs": [ + { + "name": "lua (new, no sandbox)", + "ips": "66.32 K", + "average": "15.08 μs", + "deviation": "±21.55%", + "median": "14.88 μs", + "p99": "22.58 μs", + "memory": "51.77 KB" + }, + { + "name": "luerl (init)", + "ips": "64.85 K", + "average": "15.42 μs", + "deviation": "±46.27%", + "median": "15.04 μs", + "p99": "30.25 μs", + "memory": "51.64 KB" + }, + { + "name": "lua (new, custom exclude)", + "ips": "47.89 K", + "average": "20.88 μs", + "deviation": "±63.00%", + "median": "20.63 μs", + "p99": "31.67 μs", + "memory": "72.27 KB" + }, + { + "name": "lua (new)", + "ips": "46.57 K", + "average": "21.47 μs", + "deviation": "±15.57%", + "median": "21.33 μs", + "p99": "31.25 μs", + "memory": "73.78 KB" + } + ], + "comparison": [ + "lua (new, no sandbox) 66.32 K", + "luerl (init) 64.85 K - 1.02x slower +0.34 μs", + "lua (new, custom exclude) 47.89 K - 1.38x slower +5.80 μs", + "lua (new) 46.57 K - 1.42x slower +6.40 μs" + ] + }, + "_cold_start": { + "first_call": "10070.0 us", + "second_call": "48.0 us" + } + }, + "encode_decode": { + "raw": "lua 0.4.0 — encode!/decode! decomposition\n(decode+deep_cast column: enabled)\n==================================================================================\nop shape N total_us per_elem_ns\nencode int_list 8 0.27 33.3\ndecode int_list 8 0.09 11.2\ndec+cast int_list 8 0.11 13.7\nencode int_list 64 1.61 25.1\ndecode int_list 64 0.54 8.5\ndec+cast int_list 64 0.96 15.0\nencode int_list 512 26.38 51.5\ndecode int_list 512 5.11 10.0\ndec+cast int_list 512 8.60 16.8\nencode int_list 4096 226.97 55.4\ndecode int_list 4096 43.47 10.6\ndec+cast int_list 4096 88.67 21.6\n----------------------------------------------------------------------------------\nencode float_list 8 0.22 27.7\ndecode float_list 8 0.09 11.2\ndec+cast float_list 8 0.11 13.2\nencode float_list 64 1.64 25.7\ndecode float_list 64 0.55 8.5\ndec+cast float_list 64 0.94 14.7\nencode float_list 512 27.00 52.7\ndecode float_list 512 5.85 11.4\ndec+cast float_list 512 9.00 17.6\nencode float_list 4096 239.42 58.5\ndecode float_list 4096 64.74 15.8\ndec+cast float_list 4096 98.40 24.0\n----------------------------------------------------------------------------------\nencode bool_list 8 0.21 26.4\ndecode bool_list 8 0.08 10.0\ndec+cast bool_list 8 0.10 12.6\nencode bool_list 64 1.60 25.0\ndecode bool_list 64 0.56 8.7\ndec+cast bool_list 64 0.97 15.1\nencode bool_list 512 25.68 50.1\ndecode bool_list 512 4.80 9.4\ndec+cast bool_list 512 8.38 16.4\nencode bool_list 4096 224.95 54.9\ndecode bool_list 4096 38.99 9.5\ndec+cast bool_list 4096 84.05 20.5\n----------------------------------------------------------------------------------\nencode short_string_list 8 0.21 26.7\ndecode short_string_list 8 0.08 10.5\ndec+cast short_string_list 8 0.10 12.9\nencode short_string_list 64 1.62 25.3\ndecode short_string_list 64 0.55 8.6\ndec+cast short_string_list 64 0.98 15.4\nencode short_string_list 512 26.45 51.7\ndecode short_string_list 512 4.86 9.5\ndec+cast short_string_list 512 9.05 17.7\nencode short_string_list 4096 202.93 49.5\ndecode short_string_list 4096 43.46 10.6\ndec+cast short_string_list 4096 73.06 17.8\n----------------------------------------------------------------------------------\nencode long_string_list 8 0.22 27.2\ndecode long_string_list 8 0.08 10.4\ndec+cast long_string_list 8 0.10 12.7\nencode long_string_list 64 1.67 26.1\ndecode long_string_list 64 0.59 9.2\ndec+cast long_string_list 64 1.03 16.1\nencode long_string_list 512 28.42 55.5\ndecode long_string_list 512 5.16 10.1\ndec+cast long_string_list 512 9.09 17.7\nencode long_string_list 4096 465.96 113.8\ndecode long_string_list 4096 54.39 13.3\ndec+cast long_string_list 4096 108.40 26.5\n----------------------------------------------------------------------------------\nencode string_map 8 0.52 65.3\ndecode string_map 8 0.07 8.2\ndec+cast string_map 8 0.17 21.1\nencode string_map 64 6.56 102.5\ndecode string_map 64 0.40 6.2\ndec+cast string_map 64 3.21 50.2\nencode string_map 512 113.10 220.9\ndecode string_map 512 4.27 8.3\ndec+cast string_map 512 39.39 76.9\nencode string_map 4096 1295.09 316.2\ndecode string_map 4096 42.02 10.3\ndec+cast string_map 4096 367.76 89.8\n----------------------------------------------------------------------------------\nencode int_map 8 0.25 31.2\ndecode int_map 8 0.09 10.7\ndec+cast int_map 8 0.10 12.9\nencode int_map 64 3.56 55.6\ndecode int_map 64 0.57 8.9\ndec+cast int_map 64 0.97 15.1\nencode int_map 512 51.12 99.8\ndecode int_map 512 4.79 9.4\ndec+cast int_map 512 8.76 17.1\nencode int_map 4096 443.17 108.2\ndecode int_map 4096 40.30 9.8\ndec+cast int_map 4096 87.24 21.3\n----------------------------------------------------------------------------------\nencode record_list 8 2.07 258.7\ndecode record_list 8 0.40 49.9\ndec+cast record_list 8 0.80 100.1\nencode record_list 64 31.54 492.8\ndecode record_list 64 4.54 70.9\ndec+cast record_list 64 7.32 114.4\nencode record_list 512 287.96 562.4\ndecode record_list 512 52.67 102.9\ndec+cast record_list 512 77.64 151.6\nencode record_list 4096 2346.38 572.8\ndecode record_list 4096 406.88 99.3\ndec+cast record_list 4096 658.70 160.8\n----------------------------------------------------------------------------------\nnested chain (depth sweep) — isolates recursion/traversal from fan-out\nop shape N total_us per_elem_ns\nencode nested_chain 4 0.51 257.3\ndecode nested_chain 4 0.12 61.7\ndec+cast nested_chain 4 0.24 118.7\nencode nested_chain 16 2.06 1032.0\ndecode nested_chain 16 0.63 314.9\ndec+cast nested_chain 16 1.10 550.2\nencode nested_chain 64 8.53 4265.3\ndecode nested_chain 64 4.16 2079.7\ndec+cast nested_chain 64 6.63 3317.0\nencode nested_chain 256 34.34 17170.4\ndecode nested_chain 256 37.82 18909.7\ndec+cast nested_chain 256 47.43 23716.3\n==================================================================================\ncomposite anchor — the PR's `original_nested` (matches the 18us/108us figure)\nop shape N total_us per_elem_ns\nencode original_nested 75 3.48 870.0\ndecode original_nested 75 0.58 145.4\ndec+cast original_nested 75 1.32 329.3\n" + } +} \ No newline at end of file diff --git a/bench_results/v0.4.0/table_ops.txt b/bench_results/v0.4.0/table_ops.txt new file mode 100644 index 00000000..d0e203b2 --- /dev/null +++ b/bench_results/v0.4.0/table_ops.txt @@ -0,0 +1,461 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== Table Build (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (eval) 6.50 K 153.83 μs ±11.45% 151.96 μs 195.78 μs +luerl 6.44 K 155.20 μs ±14.07% 152.33 μs 259.75 μs +lua (chunk) 6.35 K 157.51 μs ±18.23% 149.92 μs 262.79 μs + +Comparison: +lua (eval) 6.50 K +luerl 6.44 K - 1.01x slower +1.37 μs +lua (chunk) 6.35 K - 1.02x slower +3.68 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 993.93 KB +luerl 996.96 KB - 1.00x memory usage +3.03 KB +lua (chunk) 983.38 KB - 0.99x memory usage -10.54688 KB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +luerl 59.29 K 16.87 μs ±38.96% 16.25 μs 26.79 μs +lua (eval) 51.13 K 19.56 μs ±34.91% 16.46 μs 41.71 μs +lua (chunk) 43.75 K 22.86 μs ±38.05% 25.96 μs 47.33 μs + +Comparison: +luerl 59.29 K +lua (eval) 51.13 K - 1.16x slower +2.69 μs +lua (chunk) 43.75 K - 1.36x slower +5.99 μs + +Memory usage statistics: + +Name Memory usage +luerl 110.78 KB +lua (eval) 111.19 KB - 1.00x memory usage +0.41 KB +lua (chunk) 100.13 KB - 0.90x memory usage -10.64844 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 405.22 K 2.47 μs ±243.17% 2 μs 5.83 μs +luerl 321.06 K 3.11 μs ±187.65% 3 μs 4.54 μs +lua (eval) 282.18 K 3.54 μs ±178.79% 3.13 μs 9.29 μs + +Comparison: +lua (chunk) 405.22 K +luerl 321.06 K - 1.26x slower +0.65 μs +lua (eval) 282.18 K - 1.44x slower +1.08 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 12.45 KB +luerl 22.62 KB - 1.82x memory usage +10.16 KB +lua (eval) 23.02 KB - 1.85x memory usage +10.57 KB + +**All measurements for memory usage were the same** + +=== Table Sort (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 5.61 K 178.37 μs ±12.59% 174.29 μs 234.12 μs +lua (eval) 5.45 K 183.60 μs ±106.15% 177.67 μs 334.99 μs +luerl 4.88 K 205.11 μs ±25.14% 180.59 μs 336.78 μs + +Comparison: +lua (chunk) 5.61 K +lua (eval) 5.45 K - 1.03x slower +5.23 μs +luerl 4.88 K - 1.15x slower +26.74 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.17 MB +lua (eval) 1.18 MB - 1.01x memory usage +0.0107 MB +luerl 1.18 MB - 1.01x memory usage +0.00999 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +luerl 51.03 K 19.60 μs ±20.90% 19.08 μs 28.42 μs +lua (chunk) 47.42 K 21.09 μs ±42.23% 18.04 μs 51.33 μs +lua (eval) 46.64 K 21.44 μs ±39.09% 19.33 μs 45.92 μs + +Comparison: +luerl 51.03 K +lua (chunk) 47.42 K - 1.08x slower +1.49 μs +lua (eval) 46.64 K - 1.09x slower +1.85 μs + +Memory usage statistics: + +Name Memory usage +luerl 133.34 KB +lua (chunk) 122.98 KB - 0.92x memory usage -10.35156 KB +lua (eval) 133.59 KB - 1.00x memory usage +0.25 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 324.41 K 3.08 μs ±194.26% 2.54 μs 7.08 μs +luerl 271.87 K 3.68 μs ±124.45% 3.54 μs 10 μs +lua (eval) 245.19 K 4.08 μs ±136.48% 3.67 μs 8.25 μs + +Comparison: +lua (chunk) 324.41 K +luerl 271.87 K - 1.19x slower +0.60 μs +lua (eval) 245.19 K - 1.32x slower +1.00 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 15.84 KB +luerl 25.98 KB - 1.64x memory usage +10.13 KB +lua (eval) 26.25 KB - 1.66x memory usage +10.41 KB + +**All measurements for memory usage were the same** + +=== Table Iterate/Sum (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (eval) 3.81 K 262.24 μs ±19.86% 243.42 μs 444.12 μs +luerl 3.75 K 266.52 μs ±20.59% 243.13 μs 416.21 μs +lua (chunk) 3.60 K 278.11 μs ±24.12% 242.42 μs 456.96 μs + +Comparison: +lua (eval) 3.81 K +luerl 3.75 K - 1.02x slower +4.28 μs +lua (chunk) 3.60 K - 1.06x slower +15.87 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 1.35 MB +luerl 1.35 MB - 1.00x memory usage -0.00027 MB +lua (chunk) 1.34 MB - 0.99x memory usage -0.01018 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (eval) 37.84 K 26.43 μs ±9.28% 26.08 μs 29.92 μs +luerl 37.50 K 26.66 μs ±86.90% 26.21 μs 36.50 μs +lua (chunk) 35.12 K 28.47 μs ±26.41% 25.17 μs 53.58 μs + +Comparison: +lua (eval) 37.84 K +luerl 37.50 K - 1.01x slower +0.24 μs +lua (chunk) 35.12 K - 1.08x slower +2.05 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 149.42 KB +luerl 149.28 KB - 1.00x memory usage -0.14063 KB +lua (chunk) 139.25 KB - 0.93x memory usage -10.17188 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 245.60 K 4.07 μs ±313.25% 3.29 μs 9.79 μs +luerl 227.18 K 4.40 μs ±102.94% 4.29 μs 10.88 μs +lua (eval) 206.12 K 4.85 μs ±120.13% 4.33 μs 9.83 μs + +Comparison: +lua (chunk) 245.60 K +luerl 227.18 K - 1.08x slower +0.33 μs +lua (eval) 206.12 K - 1.19x slower +0.78 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 16.70 KB +luerl 26.80 KB - 1.61x memory usage +10.10 KB +lua (eval) 27.20 KB - 1.63x memory usage +10.51 KB + +**All measurements for memory usage were the same** + +=== Table Map + Reduce (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (eval) 2.21 K 452.53 μs ±7.79% 446.25 μs 614.89 μs +lua (chunk) 2.19 K 455.66 μs ±9.09% 444.34 μs 628.64 μs +luerl 2.19 K 456.58 μs ±12.12% 442.21 μs 659.87 μs + +Comparison: +lua (eval) 2.21 K +lua (chunk) 2.19 K - 1.01x slower +3.14 μs +luerl 2.19 K - 1.01x slower +4.05 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 2.45 MB +lua (chunk) 2.43 MB - 1.00x memory usage -0.01137 MB +luerl 2.45 MB - 1.00x memory usage -0.00098 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 21.79 K 45.90 μs ±12.29% 45.25 μs 53.13 μs +luerl 21.47 K 46.58 μs ±5.77% 46.42 μs 52.54 μs +lua (eval) 21.44 K 46.65 μs ±6.16% 46.50 μs 51.96 μs + +Comparison: +lua (chunk) 21.79 K +luerl 21.47 K - 1.01x slower +0.68 μs +lua (eval) 21.44 K - 1.02x slower +0.75 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 252.38 KB +luerl 262.48 KB - 1.04x memory usage +10.11 KB +lua (eval) 263.02 KB - 1.04x memory usage +10.65 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +luerl 148.83 K 6.72 μs ±69.12% 6.58 μs 14.71 μs +lua (chunk) 145.39 K 6.88 μs ±90.79% 5.54 μs 14.96 μs +lua (eval) 131.69 K 7.59 μs ±88.05% 6.71 μs 16.42 μs + +Comparison: +luerl 148.83 K +lua (chunk) 145.39 K - 1.02x slower +0.159 μs +lua (eval) 131.69 K - 1.13x slower +0.87 μs + +Memory usage statistics: + +Name Memory usage +luerl 39.39 KB +lua (chunk) 29.06 KB - 0.74x memory usage -10.32813 KB +lua (eval) 39.80 KB - 1.01x memory usage +0.41 KB + +**All measurements for memory usage were the same** + +=== Table Pairs (hash) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (eval) 760.68 1.31 ms ±5.02% 1.32 ms 1.45 ms +lua (chunk) 759.91 1.32 ms ±5.40% 1.32 ms 1.49 ms +luerl 754.19 1.33 ms ±6.28% 1.33 ms 1.45 ms + +Comparison: +lua (eval) 760.68 +lua (chunk) 759.91 - 1.00x slower +0.00133 ms +luerl 754.19 - 1.01x slower +0.0113 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 2.48 MB +lua (chunk) 2.47 MB - 1.00x memory usage -0.01072 MB +luerl 2.48 MB - 1.00x memory usage +0.00007 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 10.96 K 91.23 μs ±12.10% 89.92 μs 135.19 μs +lua (eval) 10.20 K 97.99 μs ±11.84% 97.42 μs 138.25 μs +luerl 9.96 K 100.37 μs ±20.92% 99.96 μs 144.67 μs + +Comparison: +lua (chunk) 10.96 K +lua (eval) 10.20 K - 1.07x slower +6.76 μs +luerl 9.96 K - 1.10x slower +9.14 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 238.58 KB +lua (eval) 249.40 KB - 1.05x memory usage +10.82 KB +luerl 248.99 KB - 1.04x memory usage +10.41 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 126.80 K 7.89 μs ±150.76% 6.50 μs 18.79 μs +luerl 125.64 K 7.96 μs ±72.08% 7.29 μs 24.29 μs +lua (eval) 109.11 K 9.17 μs ±61.17% 9.38 μs 28.33 μs + +Comparison: +lua (chunk) 126.80 K +luerl 125.64 K - 1.01x slower +0.0728 μs +lua (eval) 109.11 K - 1.16x slower +1.28 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 25.83 KB +luerl 36.16 KB - 1.40x memory usage +10.33 KB +lua (eval) 36.56 KB - 1.42x memory usage +10.73 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v0.4.0/timestamp.txt b/bench_results/v0.4.0/timestamp.txt new file mode 100644 index 00000000..0c188fe0 --- /dev/null +++ b/bench_results/v0.4.0/timestamp.txt @@ -0,0 +1 @@ +2026-07-28T13:59:36Z diff --git a/bench_results/v0.4.0/versions.txt b/bench_results/v0.4.0/versions.txt new file mode 100644 index 00000000..184e5f9a --- /dev/null +++ b/bench_results/v0.4.0/versions.txt @@ -0,0 +1,3 @@ +Erlang/OTP 29 [erts-17.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] + +Elixir 1.20.0 (compiled with Erlang/OTP 29) diff --git a/bench_results/v0.4.0/vm_new.txt b/bench_results/v0.4.0/vm_new.txt new file mode 100644 index 00000000..7d4449c6 --- /dev/null +++ b/bench_results/v0.4.0/vm_new.txt @@ -0,0 +1,56 @@ +=== VM instantiation: Lua.new/1 vs :luerl.init/0 (mode: full) === + +Lua.new() one-time vs repeat cost (single samples, informational): + first call on this node : 10070.0 us + second call : 48.0 us + +The first figure includes any one-time template build and first-time module +loading. Benchee's steady-state numbers below are the per-request cost after +that point. + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 52 s +Excluding outliers: false + +Benchmarking lua (new) ... +Benchmarking lua (new, custom exclude) ... +Benchmarking lua (new, no sandbox) ... +Benchmarking luerl (init) ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (new, no sandbox) 66.32 K 15.08 μs ±21.55% 14.88 μs 22.58 μs +luerl (init) 64.85 K 15.42 μs ±46.27% 15.04 μs 30.25 μs +lua (new, custom exclude) 47.89 K 20.88 μs ±63.00% 20.63 μs 31.67 μs +lua (new) 46.57 K 21.47 μs ±15.57% 21.33 μs 31.25 μs + +Comparison: +lua (new, no sandbox) 66.32 K +luerl (init) 64.85 K - 1.02x slower +0.34 μs +lua (new, custom exclude) 47.89 K - 1.38x slower +5.80 μs +lua (new) 46.57 K - 1.42x slower +6.40 μs + +Memory usage statistics: + +Name Memory usage +lua (new, no sandbox) 51.77 KB +luerl (init) 51.64 KB - 1.00x memory usage -0.12500 KB +lua (new, custom exclude) 72.27 KB - 1.40x memory usage +20.51 KB +lua (new) 73.78 KB - 1.43x memory usage +22.02 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/closures.txt b/bench_results/v1.0.0/closures.txt new file mode 100644 index 00000000..244e4660 --- /dev/null +++ b/bench_results/v1.0.0/closures.txt @@ -0,0 +1,46 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 2.58 K 388.21 μs ±7.60% 382.50 μs 546.24 μs +lua (chunk) 2.09 K 478.08 μs ±6.68% 473.54 μs 613.46 μs +lua (eval) 2.01 K 498.43 μs ±16.91% 482.54 μs 1019.38 μs + +Comparison: +luerl 2.58 K +lua (chunk) 2.09 K - 1.23x slower +89.87 μs +lua (eval) 2.01 K - 1.28x slower +110.23 μs + +Memory usage statistics: + +Name average deviation median 99th % +luerl 1.90 MB ±0.00% 1.90 MB 1.90 MB +lua (chunk) 2.65 MB ±0.08% 2.65 MB 2.65 MB +lua (eval) 2.65 MB ±0.09% 2.65 MB 2.66 MB + +Comparison: +luerl 1.90 MB +lua (chunk) 2.65 MB - 1.39x memory usage +0.75 MB +lua (eval) 2.65 MB - 1.40x memory usage +0.76 MB diff --git a/bench_results/v1.0.0/cpu.txt b/bench_results/v1.0.0/cpu.txt new file mode 100644 index 00000000..de5c8ad6 --- /dev/null +++ b/bench_results/v1.0.0/cpu.txt @@ -0,0 +1 @@ +Apple M4 diff --git a/bench_results/v1.0.0/encode_decode.txt b/bench_results/v1.0.0/encode_decode.txt new file mode 100644 index 00000000..f1b2d16b --- /dev/null +++ b/bench_results/v1.0.0/encode_decode.txt @@ -0,0 +1,128 @@ +lua 1.0.0 — encode!/decode! decomposition +(decode+deep_cast column: enabled) +================================================================================== +op shape N total_us per_elem_ns +encode int_list 8 0.90 113.0 +decode int_list 8 0.28 34.7 +dec+cast int_list 8 0.31 38.5 +encode int_list 64 13.52 211.2 +decode int_list 64 4.23 66.0 +dec+cast int_list 64 7.14 111.6 +encode int_list 512 151.37 295.6 +decode int_list 512 48.44 94.6 +dec+cast int_list 512 73.99 144.5 +encode int_list 4096 1601.11 390.9 +decode int_list 4096 583.70 142.5 +dec+cast int_list 4096 785.61 191.8 +---------------------------------------------------------------------------------- +encode float_list 8 0.78 97.1 +decode float_list 8 0.24 29.8 +dec+cast float_list 8 0.26 32.9 +encode float_list 64 11.47 179.2 +decode float_list 64 3.55 55.4 +dec+cast float_list 64 7.03 109.9 +encode float_list 512 156.43 305.5 +decode float_list 512 45.74 89.3 +dec+cast float_list 512 72.74 142.1 +encode float_list 4096 1611.38 393.4 +decode float_list 4096 551.08 134.5 +dec+cast float_list 4096 839.19 204.9 +---------------------------------------------------------------------------------- +encode bool_list 8 0.77 96.7 +decode bool_list 8 0.23 28.7 +dec+cast bool_list 8 0.25 31.3 +encode bool_list 64 11.21 175.2 +decode bool_list 64 3.46 54.1 +dec+cast bool_list 64 7.00 109.4 +encode bool_list 512 153.74 300.3 +decode bool_list 512 47.42 92.6 +dec+cast bool_list 512 74.01 144.6 +encode bool_list 4096 1569.25 383.1 +decode bool_list 4096 571.91 139.6 +dec+cast bool_list 4096 775.34 189.3 +---------------------------------------------------------------------------------- +encode short_string_list 8 0.78 96.9 +decode short_string_list 8 0.24 29.8 +dec+cast short_string_list 8 0.26 32.6 +encode short_string_list 64 11.40 178.1 +decode short_string_list 64 3.53 55.2 +dec+cast short_string_list 64 7.08 110.6 +encode short_string_list 512 155.05 302.8 +decode short_string_list 512 49.88 97.4 +dec+cast short_string_list 512 72.83 142.2 +encode short_string_list 4096 1715.16 418.7 +decode short_string_list 4096 718.28 175.4 +dec+cast short_string_list 4096 763.05 186.3 +---------------------------------------------------------------------------------- +encode long_string_list 8 0.78 96.9 +decode long_string_list 8 0.24 30.5 +dec+cast long_string_list 8 0.26 33.0 +encode long_string_list 64 11.20 174.9 +decode long_string_list 64 3.49 54.5 +dec+cast long_string_list 64 7.31 114.3 +encode long_string_list 512 157.18 307.0 +decode long_string_list 512 42.76 83.5 +dec+cast long_string_list 512 76.03 148.5 +encode long_string_list 4096 2488.38 607.5 +decode long_string_list 4096 1495.20 365.0 +dec+cast long_string_list 4096 1707.00 416.7 +---------------------------------------------------------------------------------- +encode string_map 8 1.60 199.5 +decode string_map 8 0.10 12.4 +dec+cast string_map 8 0.20 25.5 +encode string_map 64 30.47 476.1 +decode string_map 64 0.74 11.6 +dec+cast string_map 64 3.95 61.7 +encode string_map 512 240.62 470.0 +decode string_map 512 11.04 21.6 +dec+cast string_map 512 39.76 77.7 +encode string_map 4096 2194.00 535.6 +decode string_map 4096 71.08 17.4 +dec+cast string_map 4096 335.83 82.0 +---------------------------------------------------------------------------------- +encode int_map 8 0.83 103.4 +decode int_map 8 0.24 29.8 +dec+cast int_map 8 0.26 32.2 +encode int_map 64 10.97 171.5 +decode int_map 64 3.68 57.5 +dec+cast int_map 64 5.96 93.1 +encode int_map 512 109.35 213.6 +decode int_map 512 27.78 54.3 +dec+cast int_map 512 72.41 141.4 +encode int_map 4096 1307.14 319.1 +decode int_map 4096 576.06 140.6 +dec+cast int_map 4096 881.09 215.1 +---------------------------------------------------------------------------------- +encode record_list 8 5.07 633.8 +decode record_list 8 0.78 98.1 +dec+cast record_list 8 1.20 149.9 +encode record_list 64 47.83 747.4 +decode record_list 64 8.26 129.1 +dec+cast record_list 64 14.64 228.8 +encode record_list 512 565.88 1105.2 +decode record_list 512 118.02 230.5 +dec+cast record_list 512 196.29 383.4 +encode record_list 4096 4732.25 1155.3 +decode record_list 4096 1094.36 267.2 +dec+cast record_list 4096 1628.34 397.5 +---------------------------------------------------------------------------------- +nested chain (depth sweep) — isolates recursion/traversal from fan-out +op shape N total_us per_elem_ns +encode nested_chain 4 1.07 535.2 +decode nested_chain 4 0.21 103.7 +dec+cast nested_chain 4 0.32 161.9 +encode nested_chain 16 4.19 2094.1 +decode nested_chain 16 0.91 456.5 +dec+cast nested_chain 16 1.33 665.1 +encode nested_chain 64 16.94 8469.5 +decode nested_chain 64 3.92 1961.9 +dec+cast nested_chain 64 6.28 3140.2 +encode nested_chain 256 68.77 34386.2 +decode nested_chain 256 18.23 9114.9 +dec+cast nested_chain 256 32.31 16156.7 +================================================================================== +composite anchor — the PR's `original_nested` (matches the 18us/108us figure) +op shape N total_us per_elem_ns +encode original_nested 75 17.11 4276.4 +decode original_nested 75 3.21 802.7 +dec+cast original_nested 75 5.26 1314.8 diff --git a/bench_results/v1.0.0/environment.md b/bench_results/v1.0.0/environment.md new file mode 100644 index 00000000..bd4c848d --- /dev/null +++ b/bench_results/v1.0.0/environment.md @@ -0,0 +1,36 @@ +# Benchmark environment — v1.0.0 + +- **Ref**: `v1.0.0` (commit `69e13a6`) +- **Mode**: `full` (`LUA_BENCH_MODE=full`) +- **CPU**: Apple M4 (see `cpu.txt`) +- **Elixir / OTP**: Elixir 1.20.0, Erlang/OTP 29 [erts-17.0] [64-bit] [jit] (see `versions.txt`) +- **Worktree setup timestamp**: Tue Jul 28 10:48:16 EDT 2026 (see `timestamp.txt`) +- **Run date (this document)**: Tue Jul 28 11:20:58 EDT 2026 + +## Command form + +Each workload was run serially, one process per file, on an otherwise quiet +machine, from the worktree root: + +``` +LUA_BENCH_MODE=full MIX_ENV=benchmark mix run benchmarks/.exs +``` + +Workloads run: `fibonacci`, `closures`, `oop`, `string_ops`, `string_format`, +`table_ops`, `patterns`, `metamethods`, `pcall_varargs`, `vm_new`, +`encode_decode`. + +`encode_decode.exs` is not Benchee-based (it uses a `:timer.tc` harness +directly), but was invoked with the same command form for consistency. + +C Lua via `luaport` was not available in this environment (no local luaport +build) and was skipped by each script's own fallback path; all comparisons +below are lua (chunk)/lua (eval) vs. luerl only. + +## Artifacts + +- `cpu.txt`, `versions.txt`, `timestamp.txt` — raw environment probes captured + at worktree setup time. +- `.txt` — raw stdout of each `mix run` invocation (11 files). +- `summary.json` — parsed structured form of the above (workload -> case -> + jobs/comparison/memory). diff --git a/bench_results/v1.0.0/fibonacci.txt b/bench_results/v1.0.0/fibonacci.txt new file mode 100644 index 00000000..61c6c972 --- /dev/null +++ b/bench_results/v1.0.0/fibonacci.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 1.37 730.63 ms ±0.64% 730.11 ms 743.77 ms +lua (chunk) 1.26 794.25 ms ±1.02% 792.42 ms 811.45 ms +lua (eval) 1.24 807.88 ms ±1.38% 805.75 ms 831.46 ms + +Comparison: +luerl 1.37 +lua (chunk) 1.26 - 1.09x slower +63.62 ms +lua (eval) 1.24 - 1.11x slower +77.25 ms + +Memory usage statistics: + +Name Memory usage +luerl 2.45 GB +lua (chunk) 2.90 GB - 1.18x memory usage +0.44 GB +lua (eval) 2.90 GB - 1.18x memory usage +0.44 GB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/metamethods.txt b/bench_results/v1.0.0/metamethods.txt new file mode 100644 index 00000000..07e5541c --- /dev/null +++ b/bench_results/v1.0.0/metamethods.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== metamethods: self-call method dispatch (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 2.06 K 484.65 μs ±9.07% 472.63 μs 658.27 μs +lua (eval) 1.77 K 565.06 μs ±8.07% 552.29 μs 755.54 μs +lua (chunk) 1.75 K 569.99 μs ±12.20% 553.75 μs 849.77 μs + +Comparison: +luerl 2.06 K +lua (eval) 1.77 K - 1.17x slower +80.41 μs +lua (chunk) 1.75 K - 1.18x slower +85.34 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.43 MB +lua (eval) 1.85 MB - 1.30x memory usage +0.43 MB +lua (chunk) 1.84 MB - 1.29x memory usage +0.42 MB + +**All measurements for memory usage were the same** + +=== metamethods: 3-level __index chain (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 4.19 K 238.51 μs ±3.78% 237.13 μs 257.04 μs +lua (eval) 4.01 K 249.27 μs ±5.91% 248.25 μs 273.67 μs +luerl 3.95 K 253.10 μs ±42.54% 221.29 μs 343.51 μs + +Comparison: +lua (chunk) 4.19 K +lua (eval) 4.01 K - 1.05x slower +10.76 μs +luerl 3.95 K - 1.06x slower +14.59 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 788.91 KB +lua (eval) 798.82 KB - 1.01x memory usage +9.91 KB +luerl 668.59 KB - 0.85x memory usage -120.32031 KB + +**All measurements for memory usage were the same** + +=== metamethods: arithmetic/relational metamethods (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 523.21 1.91 ms ±4.47% 1.88 ms 2.20 ms +lua (chunk) 455.21 2.20 ms ±4.68% 2.23 ms 2.54 ms +lua (eval) 453.68 2.20 ms ±6.62% 2.22 ms 2.63 ms + +Comparison: +luerl 523.21 +lua (chunk) 455.21 - 1.15x slower +0.29 ms +lua (eval) 453.68 - 1.15x slower +0.29 ms + +Memory usage statistics: + +Name Memory usage +luerl 6.24 MB +lua (chunk) 7.07 MB - 1.13x memory usage +0.83 MB +lua (eval) 7.08 MB - 1.13x memory usage +0.84 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/oop.txt b/bench_results/v1.0.0/oop.txt new file mode 100644 index 00000000..e5607b9d --- /dev/null +++ b/bench_results/v1.0.0/oop.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 8.38 K 119.27 μs ±16.72% 116.13 μs 217.71 μs +lua (chunk) 7.88 K 126.88 μs ±17.97% 123.50 μs 232.21 μs +lua (eval) 7.59 K 131.77 μs ±16.80% 126.29 μs 228.05 μs + +Comparison: +luerl 8.38 K +lua (chunk) 7.88 K - 1.06x slower +7.62 μs +lua (eval) 7.59 K - 1.10x slower +12.50 μs + +Memory usage statistics: + +Name Memory usage +luerl 381.45 KB +lua (chunk) 502.34 KB - 1.32x memory usage +120.89 KB +lua (eval) 511.52 KB - 1.34x memory usage +130.07 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/patterns.txt b/bench_results/v1.0.0/patterns.txt new file mode 100644 index 00000000..442f7901 --- /dev/null +++ b/bench_results/v1.0.0/patterns.txt @@ -0,0 +1,139 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== patterns: find/match field extraction (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 972.71 1.03 ms ±9.96% 1.01 ms 1.24 ms +lua (eval) 970.99 1.03 ms ±3.16% 1.02 ms 1.13 ms +luerl 906.75 1.10 ms ±5.98% 1.09 ms 1.26 ms + +Comparison: +lua (chunk) 972.71 +lua (eval) 970.99 - 1.00x slower +0.00182 ms +luerl 906.75 - 1.07x slower +0.0748 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 3.72 MB +lua (eval) 3.73 MB - 1.00x memory usage +0.00805 MB +luerl 6.70 MB - 1.80x memory usage +2.98 MB + +**All measurements for memory usage were the same** + +=== patterns: find-based tokenizer (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 733.20 1.36 ms ±5.05% 1.35 ms 1.48 ms +lua (chunk) 626.48 1.60 ms ±4.81% 1.57 ms 1.73 ms +lua (eval) 611.43 1.64 ms ±8.70% 1.61 ms 1.99 ms + +Comparison: +luerl 733.20 +lua (chunk) 626.48 - 1.17x slower +0.23 ms +lua (eval) 611.43 - 1.20x slower +0.27 ms + +Memory usage statistics: + +Name average deviation median 99th % +luerl 6.65 MB ±0.00% 6.65 MB 6.65 MB +lua (chunk) 10.11 MB ±0.00% 10.11 MB 10.11 MB +lua (eval) 10.12 MB ±0.00% 10.12 MB 10.12 MB + +Comparison: +luerl 6.65 MB +lua (chunk) 10.11 MB - 1.52x memory usage +3.46 MB +lua (eval) 10.12 MB - 1.52x memory usage +3.48 MB + +=== patterns: gsub template substitution (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 491.94 2.03 ms ±2.07% 2.02 ms 2.14 ms +lua (eval) 488.05 2.05 ms ±4.20% 2.04 ms 2.22 ms +luerl 450.89 2.22 ms ±4.50% 2.17 ms 2.47 ms + +Comparison: +lua (chunk) 491.94 +lua (eval) 488.05 - 1.01x slower +0.0162 ms +luerl 450.89 - 1.09x slower +0.185 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 5.05 MB +lua (eval) 5.06 MB - 1.00x memory usage +0.00848 MB +luerl 11.75 MB - 2.33x memory usage +6.71 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/pcall_varargs.txt b/bench_results/v1.0.0/pcall_varargs.txt new file mode 100644 index 00000000..9048cf74 --- /dev/null +++ b/bench_results/v1.0.0/pcall_varargs.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== call protocol: pcall, success path (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 3.49 K 286.44 μs ±12.19% 282.13 μs 352.91 μs +luerl 3.44 K 290.86 μs ±3.49% 289.50 μs 309.63 μs +lua (eval) 3.34 K 299.49 μs ±5.85% 297.75 μs 329.07 μs + +Comparison: +lua (chunk) 3.49 K +luerl 3.44 K - 1.02x slower +4.42 μs +lua (eval) 3.34 K - 1.05x slower +13.05 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.31 MB +luerl 1.11 MB - 0.85x memory usage -0.20367 MB +lua (eval) 1.32 MB - 1.01x memory usage +0.00981 MB + +**All measurements for memory usage were the same** + +=== call protocol: pcall, raise + catch (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 1.87 K 533.92 μs ±29.41% 485.38 μs 1138.98 μs +lua (chunk) 1.11 K 904.68 μs ±17.56% 877.33 μs 1891.77 μs +lua (eval) 1.00 K 1001.11 μs ±31.79% 899.42 μs 2144.79 μs + +Comparison: +luerl 1.87 K +lua (chunk) 1.11 K - 1.69x slower +370.76 μs +lua (eval) 1.00 K - 1.88x slower +467.19 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.55 MB +lua (chunk) 3.35 MB - 2.15x memory usage +1.79 MB +lua (eval) 3.36 MB - 2.16x memory usage +1.81 MB + +**All measurements for memory usage were the same** + +=== call protocol: varargs + multiple returns (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 410.14 2.44 ms ±4.02% 2.40 ms 2.73 ms +lua (eval) 253.78 3.94 ms ±6.97% 3.89 ms 4.79 ms +lua (chunk) 237.27 4.21 ms ±16.27% 4.01 ms 8.17 ms + +Comparison: +luerl 410.14 +lua (eval) 253.78 - 1.62x slower +1.50 ms +lua (chunk) 237.27 - 1.73x slower +1.78 ms + +Memory usage statistics: + +Name Memory usage +luerl 8.73 MB +lua (eval) 19.95 MB - 2.29x memory usage +11.22 MB +lua (chunk) 19.94 MB - 2.28x memory usage +11.21 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/string_format.txt b/bench_results/v1.0.0/string_format.txt new file mode 100644 index 00000000..39bfaf51 --- /dev/null +++ b/bench_results/v1.0.0/string_format.txt @@ -0,0 +1,139 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== string.format: long literal-heavy format string (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 1063.59 0.94 ms ±10.28% 0.90 ms 1.17 ms +lua (chunk) 966.41 1.03 ms ±6.22% 1.03 ms 1.16 ms +luerl 241.01 4.15 ms ±10.25% 4.32 ms 4.71 ms + +Comparison: +lua (eval) 1063.59 +lua (chunk) 966.41 - 1.10x slower +0.0945 ms +luerl 241.01 - 4.41x slower +3.21 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 3.27 MB +lua (chunk) 3.26 MB - 1.00x memory usage -0.00974 MB +luerl 22.59 MB - 6.92x memory usage +19.32 MB + +**All measurements for memory usage were the same** + +=== string.format: width-flagged specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 668.09 1.50 ms ±2.73% 1.49 ms 1.62 ms +lua (chunk) 662.60 1.51 ms ±4.32% 1.50 ms 1.63 ms +luerl 549.59 1.82 ms ±5.86% 1.79 ms 2.09 ms + +Comparison: +lua (eval) 668.09 +lua (chunk) 662.60 - 1.01x slower +0.0124 ms +luerl 549.59 - 1.22x slower +0.32 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 5.22 MB +lua (chunk) 5.21 MB - 1.00x memory usage -0.01022 MB +luerl 7.55 MB - 1.45x memory usage +2.33 MB + +**All measurements for memory usage were the same** + +=== string.format: many specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 418.38 2.39 ms ±8.03% 2.39 ms 3.24 ms +lua (chunk) 413.90 2.42 ms ±8.21% 2.35 ms 3.05 ms +luerl 358.58 2.79 ms ±4.13% 2.82 ms 3.00 ms + +Comparison: +lua (eval) 418.38 +lua (chunk) 413.90 - 1.01x slower +0.0259 ms +luerl 358.58 - 1.17x slower +0.40 ms + +Memory usage statistics: + +Name average deviation median 99th % +lua (eval) 14.86 MB ±0.00% 14.86 MB 14.86 MB +lua (chunk) 14.86 MB ±0.00% 14.86 MB 14.86 MB +luerl 13.76 MB ±0.00% 13.76 MB 13.76 MB + +Comparison: +lua (eval) 14.86 MB +lua (chunk) 14.86 MB - 1.00x memory usage -0.00894 MB +luerl 13.76 MB - 0.93x memory usage -1.10735 MB diff --git a/bench_results/v1.0.0/string_ops.txt b/bench_results/v1.0.0/string_ops.txt new file mode 100644 index 00000000..bf8b3b9a --- /dev/null +++ b/bench_results/v1.0.0/string_ops.txt @@ -0,0 +1,91 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== String Concatenation via table.concat (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 27.97 K 35.76 μs ±9.62% 35.67 μs 41.97 μs +lua (eval) 25.79 K 38.77 μs ±11.58% 38.58 μs 45.21 μs +luerl 24.73 K 40.44 μs ±7.76% 40.17 μs 46.08 μs + +Comparison: +lua (chunk) 27.97 K +lua (eval) 25.79 K - 1.08x slower +3.01 μs +luerl 24.73 K - 1.13x slower +4.68 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 171.33 KB +lua (eval) 180.13 KB - 1.05x memory usage +8.80 KB +luerl 172.42 KB - 1.01x memory usage +1.09 KB + +**All measurements for memory usage were the same** + +=== String Formatting via string.format (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 12.44 K 80.35 μs ±7.41% 80.21 μs 93.29 μs +lua (eval) 11.53 K 86.74 μs ±27.72% 85.58 μs 111.49 μs +luerl 9.55 K 104.75 μs ±11.48% 103.38 μs 122.63 μs + +Comparison: +lua (chunk) 12.44 K +lua (eval) 11.53 K - 1.08x slower +6.39 μs +luerl 9.55 K - 1.30x slower +24.39 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 333.77 KB +lua (eval) 343.45 KB - 1.03x memory usage +9.68 KB +luerl 588.84 KB - 1.76x memory usage +255.07 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/summary.json b/bench_results/v1.0.0/summary.json new file mode 100644 index 00000000..09df2029 --- /dev/null +++ b/bench_results/v1.0.0/summary.json @@ -0,0 +1,1290 @@ +{ + "fibonacci": { + "default": { + "jobs": [ + { + "name": "luerl", + "ips": "1.37", + "average": "730.63 ms", + "deviation": "±0.64%", + "median": "730.11 ms", + "p99": "743.77 ms", + "memory": "2.45 GB" + }, + { + "name": "lua (chunk)", + "ips": "1.26", + "average": "794.25 ms", + "deviation": "±1.02%", + "median": "792.42 ms", + "p99": "811.45 ms", + "memory": "2.90 GB" + }, + { + "name": "lua (eval)", + "ips": "1.24", + "average": "807.88 ms", + "deviation": "±1.38%", + "median": "805.75 ms", + "p99": "831.46 ms", + "memory": "2.90 GB" + } + ], + "comparison": [ + "luerl 1.37", + "lua (chunk) 1.26 - 1.09x slower +63.62 ms", + "lua (eval) 1.24 - 1.11x slower +77.25 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "closures": { + "default": { + "jobs": [ + { + "name": "luerl", + "ips": "2.58 K", + "average": "388.21 μs", + "deviation": "±7.60%", + "median": "382.50 μs", + "p99": "546.24 μs", + "memory": "1.90 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.09 K", + "average": "478.08 μs", + "deviation": "±6.68%", + "median": "473.54 μs", + "p99": "613.46 μs", + "memory": "2.65 MB" + }, + { + "name": "lua (eval)", + "ips": "2.01 K", + "average": "498.43 μs", + "deviation": "±16.91%", + "median": "482.54 μs", + "p99": "1019.38 μs", + "memory": "2.65 MB" + } + ], + "comparison": [ + "luerl 2.58 K", + "lua (chunk) 2.09 K - 1.23x slower +89.87 μs", + "lua (eval) 2.01 K - 1.28x slower +110.23 μs" + ], + "memory_comparison": [ + "luerl 1.90 MB", + "lua (chunk) 2.65 MB - 1.39x memory usage +0.75 MB", + "lua (eval) 2.65 MB - 1.40x memory usage +0.76 MB" + ] + } + }, + "oop": { + "default": { + "jobs": [ + { + "name": "luerl", + "ips": "8.38 K", + "average": "119.27 μs", + "deviation": "±16.72%", + "median": "116.13 μs", + "p99": "217.71 μs", + "memory": "381.45 KB" + }, + { + "name": "lua (chunk)", + "ips": "7.88 K", + "average": "126.88 μs", + "deviation": "±17.97%", + "median": "123.50 μs", + "p99": "232.21 μs", + "memory": "502.34 KB" + }, + { + "name": "lua (eval)", + "ips": "7.59 K", + "average": "131.77 μs", + "deviation": "±16.80%", + "median": "126.29 μs", + "p99": "228.05 μs", + "memory": "511.52 KB" + } + ], + "comparison": [ + "luerl 8.38 K", + "lua (chunk) 7.88 K - 1.06x slower +7.62 μs", + "lua (eval) 7.59 K - 1.10x slower +12.50 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "string_ops": { + "String Concatenation via table.concat (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "27.97 K", + "average": "35.76 μs", + "deviation": "±9.62%", + "median": "35.67 μs", + "p99": "41.97 μs", + "memory": "171.33 KB" + }, + { + "name": "lua (eval)", + "ips": "25.79 K", + "average": "38.77 μs", + "deviation": "±11.58%", + "median": "38.58 μs", + "p99": "45.21 μs", + "memory": "180.13 KB" + }, + { + "name": "luerl", + "ips": "24.73 K", + "average": "40.44 μs", + "deviation": "±7.76%", + "median": "40.17 μs", + "p99": "46.08 μs", + "memory": "172.42 KB" + } + ], + "comparison": [ + "lua (chunk) 27.97 K", + "lua (eval) 25.79 K - 1.08x slower +3.01 μs", + "luerl 24.73 K - 1.13x slower +4.68 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "String Formatting via string.format (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "12.44 K", + "average": "80.35 μs", + "deviation": "±7.41%", + "median": "80.21 μs", + "p99": "93.29 μs", + "memory": "333.77 KB" + }, + { + "name": "lua (eval)", + "ips": "11.53 K", + "average": "86.74 μs", + "deviation": "±27.72%", + "median": "85.58 μs", + "p99": "111.49 μs", + "memory": "343.45 KB" + }, + { + "name": "luerl", + "ips": "9.55 K", + "average": "104.75 μs", + "deviation": "±11.48%", + "median": "103.38 μs", + "p99": "122.63 μs", + "memory": "588.84 KB" + } + ], + "comparison": [ + "lua (chunk) 12.44 K", + "lua (eval) 11.53 K - 1.08x slower +6.39 μs", + "luerl 9.55 K - 1.30x slower +24.39 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "string_format": { + "string.format: long literal-heavy format string (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "1063.59", + "average": "0.94 ms", + "deviation": "±10.28%", + "median": "0.90 ms", + "p99": "1.17 ms", + "memory": "3.27 MB" + }, + { + "name": "lua (chunk)", + "ips": "966.41", + "average": "1.03 ms", + "deviation": "±6.22%", + "median": "1.03 ms", + "p99": "1.16 ms", + "memory": "3.26 MB" + }, + { + "name": "luerl", + "ips": "241.01", + "average": "4.15 ms", + "deviation": "±10.25%", + "median": "4.32 ms", + "p99": "4.71 ms", + "memory": "22.59 MB" + } + ], + "comparison": [ + "lua (eval) 1063.59", + "lua (chunk) 966.41 - 1.10x slower +0.0945 ms", + "luerl 241.01 - 4.41x slower +3.21 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "string.format: width-flagged specifiers (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "668.09", + "average": "1.50 ms", + "deviation": "±2.73%", + "median": "1.49 ms", + "p99": "1.62 ms", + "memory": "5.22 MB" + }, + { + "name": "lua (chunk)", + "ips": "662.60", + "average": "1.51 ms", + "deviation": "±4.32%", + "median": "1.50 ms", + "p99": "1.63 ms", + "memory": "5.21 MB" + }, + { + "name": "luerl", + "ips": "549.59", + "average": "1.82 ms", + "deviation": "±5.86%", + "median": "1.79 ms", + "p99": "2.09 ms", + "memory": "7.55 MB" + } + ], + "comparison": [ + "lua (eval) 668.09", + "lua (chunk) 662.60 - 1.01x slower +0.0124 ms", + "luerl 549.59 - 1.22x slower +0.32 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "string.format: many specifiers (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "418.38", + "average": "2.39 ms", + "deviation": "±8.03%", + "median": "2.39 ms", + "p99": "3.24 ms", + "memory": "14.86 MB" + }, + { + "name": "lua (chunk)", + "ips": "413.90", + "average": "2.42 ms", + "deviation": "±8.21%", + "median": "2.35 ms", + "p99": "3.05 ms", + "memory": "14.86 MB" + }, + { + "name": "luerl", + "ips": "358.58", + "average": "2.79 ms", + "deviation": "±4.13%", + "median": "2.82 ms", + "p99": "3.00 ms", + "memory": "13.76 MB" + } + ], + "comparison": [ + "lua (eval) 418.38", + "lua (chunk) 413.90 - 1.01x slower +0.0259 ms", + "luerl 358.58 - 1.17x slower +0.40 ms" + ], + "memory_comparison": [ + "lua (eval) 14.86 MB", + "lua (chunk) 14.86 MB - 1.00x memory usage -0.00894 MB", + "luerl 13.76 MB - 0.93x memory usage -1.10735 MB" + ] + } + }, + "table_ops": { + "Table Build": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "luerl", + "ips": "6.37 K", + "average": "157.04 μs", + "deviation": "±11.82%", + "median": "154.21 μs", + "p99": "218.16 μs", + "memory": "0.97 MB" + }, + { + "name": "lua (chunk)", + "ips": "6.23 K", + "average": "160.45 μs", + "deviation": "±8.91%", + "median": "157.88 μs", + "p99": "198.69 μs", + "memory": "1.00 MB" + }, + { + "name": "lua (eval)", + "ips": "5.90 K", + "average": "169.48 μs", + "deviation": "±14.14%", + "median": "164.58 μs", + "p99": "287.61 μs", + "memory": "1.01 MB" + } + ], + "comparison": [ + "luerl 6.37 K", + "lua (chunk) 6.23 K - 1.02x slower +3.41 μs", + "lua (eval) 5.90 K - 1.08x slower +12.44 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "60.98 K", + "average": "16.40 μs", + "deviation": "±32.33%", + "median": "16.17 μs", + "p99": "24.29 μs", + "memory": "105.41 KB" + }, + { + "name": "luerl", + "ips": "58.70 K", + "average": "17.04 μs", + "deviation": "±26.63%", + "median": "16.50 μs", + "p99": "26.67 μs", + "memory": "110.77 KB" + }, + { + "name": "lua (eval)", + "ips": "53.06 K", + "average": "18.85 μs", + "deviation": "±33.45%", + "median": "18.42 μs", + "p99": "27.04 μs", + "memory": "114.74 KB" + } + ], + "comparison": [ + "lua (chunk) 60.98 K", + "luerl 58.70 K - 1.04x slower +0.64 μs", + "lua (eval) 53.06 K - 1.15x slower +2.45 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "437.49 K", + "average": "2.29 μs", + "deviation": "±237.41%", + "median": "2.25 μs", + "p99": "3.42 μs", + "memory": "13.80 KB" + }, + { + "name": "luerl", + "ips": "317.31 K", + "average": "3.15 μs", + "deviation": "±243.81%", + "median": "3.04 μs", + "p99": "4.71 μs", + "memory": "22.62 KB" + }, + { + "name": "lua (eval)", + "ips": "205.32 K", + "average": "4.87 μs", + "deviation": "±231.75%", + "median": "4.46 μs", + "p99": "10.42 μs", + "memory": "23.45 KB" + } + ], + "comparison": [ + "lua (chunk) 437.49 K", + "luerl 317.31 K - 1.38x slower +0.87 μs", + "lua (eval) 205.32 K - 2.13x slower +2.58 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Sort": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "luerl", + "ips": "5.52 K", + "average": "181.29 μs", + "deviation": "±10.01%", + "median": "178.00 μs", + "p99": "220.42 μs", + "memory": "1.18 MB" + }, + { + "name": "lua (chunk)", + "ips": "4.27 K", + "average": "234.42 μs", + "deviation": "±13.69%", + "median": "227.63 μs", + "p99": "411.33 μs", + "memory": "1.33 MB" + }, + { + "name": "lua (eval)", + "ips": "4.19 K", + "average": "238.72 μs", + "deviation": "±9.85%", + "median": "234.88 μs", + "p99": "335.73 μs", + "memory": "1.34 MB" + } + ], + "comparison": [ + "luerl 5.52 K", + "lua (chunk) 4.27 K - 1.29x slower +53.13 μs", + "lua (eval) 4.19 K - 1.32x slower +57.42 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "luerl", + "ips": "50.83 K", + "average": "19.67 μs", + "deviation": "±20.44%", + "median": "19.13 μs", + "p99": "28.17 μs", + "memory": "133.33 KB" + }, + { + "name": "lua (chunk)", + "ips": "42.73 K", + "average": "23.40 μs", + "deviation": "±35.27%", + "median": "23.21 μs", + "p99": "26.75 μs", + "memory": "141.38 KB" + }, + { + "name": "lua (eval)", + "ips": "38.35 K", + "average": "26.07 μs", + "deviation": "±37.69%", + "median": "25.58 μs", + "p99": "35.33 μs", + "memory": "151.13 KB" + } + ], + "comparison": [ + "luerl 50.83 K", + "lua (chunk) 42.73 K - 1.19x slower +3.73 μs", + "lua (eval) 38.35 K - 1.33x slower +6.40 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "311.25 K", + "average": "3.21 μs", + "deviation": "±183.71%", + "median": "3.17 μs", + "p99": "4.50 μs", + "memory": "19.06 KB" + }, + { + "name": "luerl", + "ips": "266.96 K", + "average": "3.75 μs", + "deviation": "±163.56%", + "median": "3.58 μs", + "p99": "9.50 μs", + "memory": "25.98 KB" + }, + { + "name": "lua (eval)", + "ips": "169.47 K", + "average": "5.90 μs", + "deviation": "±170.93%", + "median": "5.50 μs", + "p99": "15.75 μs", + "memory": "28.78 KB" + } + ], + "comparison": [ + "lua (chunk) 311.25 K", + "luerl 266.96 K - 1.17x slower +0.53 μs", + "lua (eval) 169.47 K - 1.84x slower +2.69 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Iterate/Sum": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "4.10 K", + "average": "243.68 μs", + "deviation": "±11.81%", + "median": "235.09 μs", + "p99": "342.27 μs", + "memory": "1.47 MB" + }, + { + "name": "lua (eval)", + "ips": "4.07 K", + "average": "245.42 μs", + "deviation": "±12.09%", + "median": "241.67 μs", + "p99": "362.47 μs", + "memory": "1.48 MB" + }, + { + "name": "luerl", + "ips": "4.06 K", + "average": "246.09 μs", + "deviation": "±7.76%", + "median": "243.06 μs", + "p99": "342.83 μs", + "memory": "1.35 MB" + } + ], + "comparison": [ + "lua (chunk) 4.10 K", + "lua (eval) 4.07 K - 1.01x slower +1.75 μs", + "luerl 4.06 K - 1.01x slower +2.41 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "40.08 K", + "average": "24.95 μs", + "deviation": "±25.49%", + "median": "24.25 μs", + "p99": "52.88 μs", + "memory": "154.27 KB" + }, + { + "name": "lua (eval)", + "ips": "36.72 K", + "average": "27.23 μs", + "deviation": "±17.19%", + "median": "26.67 μs", + "p99": "38.21 μs", + "memory": "164.09 KB" + }, + { + "name": "luerl", + "ips": "36.18 K", + "average": "27.64 μs", + "deviation": "±35.39%", + "median": "27.00 μs", + "p99": "57.79 μs", + "memory": "149.14 KB" + } + ], + "comparison": [ + "lua (chunk) 40.08 K", + "lua (eval) 36.72 K - 1.09x slower +2.28 μs", + "luerl 36.18 K - 1.11x slower +2.69 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "306.46 K", + "average": "3.26 μs", + "deviation": "±172.56%", + "median": "3.21 μs", + "p99": "4.38 μs", + "memory": "19.87 KB" + }, + { + "name": "luerl", + "ips": "231.29 K", + "average": "4.32 μs", + "deviation": "±138.54%", + "median": "4.25 μs", + "p99": "6.33 μs", + "memory": "26.80 KB" + }, + { + "name": "lua (eval)", + "ips": "177.92 K", + "average": "5.62 μs", + "deviation": "±101.11%", + "median": "5.42 μs", + "p99": "9.25 μs", + "memory": "29.51 KB" + } + ], + "comparison": [ + "lua (chunk) 306.46 K", + "luerl 231.29 K - 1.33x slower +1.06 μs", + "lua (eval) 177.92 K - 1.72x slower +2.36 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Map + Reduce": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "luerl", + "ips": "2.15 K", + "average": "464.93 μs", + "deviation": "±7.90%", + "median": "449.96 μs", + "p99": "600.72 μs", + "memory": "2.44 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.05 K", + "average": "487.99 μs", + "deviation": "±6.39%", + "median": "480.24 μs", + "p99": "622.53 μs", + "memory": "2.92 MB" + }, + { + "name": "lua (eval)", + "ips": "2.01 K", + "average": "496.87 μs", + "deviation": "±13.21%", + "median": "487.17 μs", + "p99": "772.35 μs", + "memory": "2.93 MB" + } + ], + "comparison": [ + "luerl 2.15 K", + "lua (chunk) 2.05 K - 1.05x slower +23.06 μs", + "lua (eval) 2.01 K - 1.07x slower +31.94 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "luerl", + "ips": "21.41 K", + "average": "46.71 μs", + "deviation": "±6.09%", + "median": "46.50 μs", + "p99": "52.29 μs", + "memory": "262.35 KB" + }, + { + "name": "lua (chunk)", + "ips": "20.87 K", + "average": "47.91 μs", + "deviation": "±8.43%", + "median": "47.54 μs", + "p99": "53.79 μs", + "memory": "303.87 KB" + }, + { + "name": "lua (eval)", + "ips": "19.56 K", + "average": "51.12 μs", + "deviation": "±6.66%", + "median": "50.58 μs", + "p99": "60.46 μs", + "memory": "314.32 KB" + } + ], + "comparison": [ + "luerl 21.41 K", + "lua (chunk) 20.87 K - 1.03x slower +1.20 μs", + "lua (eval) 19.56 K - 1.09x slower +4.40 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "168.91 K", + "average": "5.92 μs", + "deviation": "±76.95%", + "median": "5.83 μs", + "p99": "13.38 μs", + "memory": "37.09 KB" + }, + { + "name": "luerl", + "ips": "145.75 K", + "average": "6.86 μs", + "deviation": "±94.69%", + "median": "6.59 μs", + "p99": "15.96 μs", + "memory": "39.39 KB" + }, + { + "name": "lua (eval)", + "ips": "102.03 K", + "average": "9.80 μs", + "deviation": "±71.35%", + "median": "8.21 μs", + "p99": "22.96 μs", + "memory": "46.72 KB" + } + ], + "comparison": [ + "lua (chunk) 168.91 K", + "luerl 145.75 K - 1.16x slower +0.94 μs", + "lua (eval) 102.03 K - 1.66x slower +3.88 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Pairs (hash)": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "1.02 K", + "average": "984.62 μs", + "deviation": "±7.98%", + "median": "995.70 μs", + "p99": "1143.86 μs", + "memory": "2.06 MB" + }, + { + "name": "lua (eval)", + "ips": "1.01 K", + "average": "990.11 μs", + "deviation": "±8.33%", + "median": "1002.18 μs", + "p99": "1157.35 μs", + "memory": "2.05 MB" + }, + { + "name": "luerl", + "ips": "0.75 K", + "average": "1332.28 μs", + "deviation": "±5.93%", + "median": "1334.35 μs", + "p99": "1489.09 μs", + "memory": "2.48 MB" + } + ], + "comparison": [ + "lua (chunk) 1.02 K", + "lua (eval) 1.01 K - 1.01x slower +5.49 μs", + "luerl 0.75 K - 1.35x slower +347.66 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "13.46 K", + "average": "74.27 μs", + "deviation": "±10.12%", + "median": "72.92 μs", + "p99": "99.63 μs", + "memory": "211.49 KB" + }, + { + "name": "lua (eval)", + "ips": "12.33 K", + "average": "81.10 μs", + "deviation": "±25.10%", + "median": "78.13 μs", + "p99": "167.55 μs", + "memory": "221.46 KB" + }, + { + "name": "luerl", + "ips": "10.92 K", + "average": "91.56 μs", + "deviation": "±15.90%", + "median": "86.83 μs", + "p99": "142.16 μs", + "memory": "248.84 KB" + } + ], + "comparison": [ + "lua (chunk) 13.46 K", + "lua (eval) 12.33 K - 1.09x slower +6.83 μs", + "luerl 10.92 K - 1.23x slower +17.29 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "130.58 K", + "average": "7.66 μs", + "deviation": "±70.51%", + "median": "7.33 μs", + "p99": "24.42 μs", + "memory": "25.10 KB" + }, + { + "name": "luerl", + "ips": "130.19 K", + "average": "7.68 μs", + "deviation": "±68.45%", + "median": "7.25 μs", + "p99": "23.38 μs", + "memory": "36.16 KB" + }, + { + "name": "lua (eval)", + "ips": "94.66 K", + "average": "10.56 μs", + "deviation": "±56.52%", + "median": "9.92 μs", + "p99": "29.25 μs", + "memory": "34.91 KB" + } + ], + "comparison": [ + "lua (chunk) 130.58 K", + "luerl 130.19 K - 1.00x slower +0.0229 μs", + "lua (eval) 94.66 K - 1.38x slower +2.91 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + } + }, + "patterns": { + "patterns: find/match field extraction (n=200)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "972.71", + "average": "1.03 ms", + "deviation": "±9.96%", + "median": "1.01 ms", + "p99": "1.24 ms", + "memory": "3.72 MB" + }, + { + "name": "lua (eval)", + "ips": "970.99", + "average": "1.03 ms", + "deviation": "±3.16%", + "median": "1.02 ms", + "p99": "1.13 ms", + "memory": "3.73 MB" + }, + { + "name": "luerl", + "ips": "906.75", + "average": "1.10 ms", + "deviation": "±5.98%", + "median": "1.09 ms", + "p99": "1.26 ms", + "memory": "6.70 MB" + } + ], + "comparison": [ + "lua (chunk) 972.71", + "lua (eval) 970.99 - 1.00x slower +0.00182 ms", + "luerl 906.75 - 1.07x slower +0.0748 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "patterns: find-based tokenizer (n=200)": { + "jobs": [ + { + "name": "luerl", + "ips": "733.20", + "average": "1.36 ms", + "deviation": "±5.05%", + "median": "1.35 ms", + "p99": "1.48 ms", + "memory": "6.65 MB" + }, + { + "name": "lua (chunk)", + "ips": "626.48", + "average": "1.60 ms", + "deviation": "±4.81%", + "median": "1.57 ms", + "p99": "1.73 ms", + "memory": "10.11 MB" + }, + { + "name": "lua (eval)", + "ips": "611.43", + "average": "1.64 ms", + "deviation": "±8.70%", + "median": "1.61 ms", + "p99": "1.99 ms", + "memory": "10.12 MB" + } + ], + "comparison": [ + "luerl 733.20", + "lua (chunk) 626.48 - 1.17x slower +0.23 ms", + "lua (eval) 611.43 - 1.20x slower +0.27 ms" + ], + "memory_comparison": [ + "luerl 6.65 MB", + "lua (chunk) 10.11 MB - 1.52x memory usage +3.46 MB", + "lua (eval) 10.12 MB - 1.52x memory usage +3.48 MB" + ] + }, + "patterns: gsub template substitution (n=200)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "491.94", + "average": "2.03 ms", + "deviation": "±2.07%", + "median": "2.02 ms", + "p99": "2.14 ms", + "memory": "5.05 MB" + }, + { + "name": "lua (eval)", + "ips": "488.05", + "average": "2.05 ms", + "deviation": "±4.20%", + "median": "2.04 ms", + "p99": "2.22 ms", + "memory": "5.06 MB" + }, + { + "name": "luerl", + "ips": "450.89", + "average": "2.22 ms", + "deviation": "±4.50%", + "median": "2.17 ms", + "p99": "2.47 ms", + "memory": "11.75 MB" + } + ], + "comparison": [ + "lua (chunk) 491.94", + "lua (eval) 488.05 - 1.01x slower +0.0162 ms", + "luerl 450.89 - 1.09x slower +0.185 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "metamethods": { + "metamethods: self-call method dispatch (n=200)": { + "jobs": [ + { + "name": "luerl", + "ips": "2.06 K", + "average": "484.65 μs", + "deviation": "±9.07%", + "median": "472.63 μs", + "p99": "658.27 μs", + "memory": "1.43 MB" + }, + { + "name": "lua (eval)", + "ips": "1.77 K", + "average": "565.06 μs", + "deviation": "±8.07%", + "median": "552.29 μs", + "p99": "755.54 μs", + "memory": "1.85 MB" + }, + { + "name": "lua (chunk)", + "ips": "1.75 K", + "average": "569.99 μs", + "deviation": "±12.20%", + "median": "553.75 μs", + "p99": "849.77 μs", + "memory": "1.84 MB" + } + ], + "comparison": [ + "luerl 2.06 K", + "lua (eval) 1.77 K - 1.17x slower +80.41 μs", + "lua (chunk) 1.75 K - 1.18x slower +85.34 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "metamethods: 3-level __index chain (n=200)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "4.19 K", + "average": "238.51 μs", + "deviation": "±3.78%", + "median": "237.13 μs", + "p99": "257.04 μs", + "memory": "788.91 KB" + }, + { + "name": "lua (eval)", + "ips": "4.01 K", + "average": "249.27 μs", + "deviation": "±5.91%", + "median": "248.25 μs", + "p99": "273.67 μs", + "memory": "798.82 KB" + }, + { + "name": "luerl", + "ips": "3.95 K", + "average": "253.10 μs", + "deviation": "±42.54%", + "median": "221.29 μs", + "p99": "343.51 μs", + "memory": "668.59 KB" + } + ], + "comparison": [ + "lua (chunk) 4.19 K", + "lua (eval) 4.01 K - 1.05x slower +10.76 μs", + "luerl 3.95 K - 1.06x slower +14.59 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "metamethods: arithmetic/relational metamethods (n=200)": { + "jobs": [ + { + "name": "luerl", + "ips": "523.21", + "average": "1.91 ms", + "deviation": "±4.47%", + "median": "1.88 ms", + "p99": "2.20 ms", + "memory": "6.24 MB" + }, + { + "name": "lua (chunk)", + "ips": "455.21", + "average": "2.20 ms", + "deviation": "±4.68%", + "median": "2.23 ms", + "p99": "2.54 ms", + "memory": "7.07 MB" + }, + { + "name": "lua (eval)", + "ips": "453.68", + "average": "2.20 ms", + "deviation": "±6.62%", + "median": "2.22 ms", + "p99": "2.63 ms", + "memory": "7.08 MB" + } + ], + "comparison": [ + "luerl 523.21", + "lua (chunk) 455.21 - 1.15x slower +0.29 ms", + "lua (eval) 453.68 - 1.15x slower +0.29 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "pcall_varargs": { + "call protocol: pcall, success path (n=500)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "3.49 K", + "average": "286.44 μs", + "deviation": "±12.19%", + "median": "282.13 μs", + "p99": "352.91 μs", + "memory": "1.31 MB" + }, + { + "name": "luerl", + "ips": "3.44 K", + "average": "290.86 μs", + "deviation": "±3.49%", + "median": "289.50 μs", + "p99": "309.63 μs", + "memory": "1.11 MB" + }, + { + "name": "lua (eval)", + "ips": "3.34 K", + "average": "299.49 μs", + "deviation": "±5.85%", + "median": "297.75 μs", + "p99": "329.07 μs", + "memory": "1.32 MB" + } + ], + "comparison": [ + "lua (chunk) 3.49 K", + "luerl 3.44 K - 1.02x slower +4.42 μs", + "lua (eval) 3.34 K - 1.05x slower +13.05 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "call protocol: pcall, raise + catch (n=500)": { + "jobs": [ + { + "name": "luerl", + "ips": "1.87 K", + "average": "533.92 μs", + "deviation": "±29.41%", + "median": "485.38 μs", + "p99": "1138.98 μs", + "memory": "1.55 MB" + }, + { + "name": "lua (chunk)", + "ips": "1.11 K", + "average": "904.68 μs", + "deviation": "±17.56%", + "median": "877.33 μs", + "p99": "1891.77 μs", + "memory": "3.35 MB" + }, + { + "name": "lua (eval)", + "ips": "1.00 K", + "average": "1001.11 μs", + "deviation": "±31.79%", + "median": "899.42 μs", + "p99": "2144.79 μs", + "memory": "3.36 MB" + } + ], + "comparison": [ + "luerl 1.87 K", + "lua (chunk) 1.11 K - 1.69x slower +370.76 μs", + "lua (eval) 1.00 K - 1.88x slower +467.19 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "call protocol: varargs + multiple returns (n=500)": { + "jobs": [ + { + "name": "luerl", + "ips": "410.14", + "average": "2.44 ms", + "deviation": "±4.02%", + "median": "2.40 ms", + "p99": "2.73 ms", + "memory": "8.73 MB" + }, + { + "name": "lua (eval)", + "ips": "253.78", + "average": "3.94 ms", + "deviation": "±6.97%", + "median": "3.89 ms", + "p99": "4.79 ms", + "memory": "19.95 MB" + }, + { + "name": "lua (chunk)", + "ips": "237.27", + "average": "4.21 ms", + "deviation": "±16.27%", + "median": "4.01 ms", + "p99": "8.17 ms", + "memory": "19.94 MB" + } + ], + "comparison": [ + "luerl 410.14", + "lua (eval) 253.78 - 1.62x slower +1.50 ms", + "lua (chunk) 237.27 - 1.73x slower +1.78 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "vm_new": { + "VM instantiation: Lua.new/1 vs :luerl.init/0": { + "jobs": [ + { + "name": "luerl (init)", + "ips": "64.86 K", + "average": "15.42 μs", + "deviation": "±23.43%", + "median": "15.21 μs", + "p99": "24.25 μs", + "memory": "51.64 KB" + }, + { + "name": "lua (new, no sandbox)", + "ips": "33.30 K", + "average": "30.03 μs", + "deviation": "±11.23%", + "median": "29.71 μs", + "p99": "38.46 μs", + "memory": "68.76 KB" + }, + { + "name": "lua (new, custom exclude)", + "ips": "27.37 K", + "average": "36.54 μs", + "deviation": "±8.96%", + "median": "35.96 μs", + "p99": "42.63 μs", + "memory": "90.97 KB" + }, + { + "name": "lua (new)", + "ips": "26.58 K", + "average": "37.63 μs", + "deviation": "±62.22%", + "median": "36.67 μs", + "p99": "71.96 μs", + "memory": "91.88 KB" + } + ], + "comparison": [ + "luerl (init) 64.86 K", + "lua (new, no sandbox) 33.30 K - 1.95x slower +14.62 μs", + "lua (new, custom exclude) 27.37 K - 2.37x slower +21.12 μs", + "lua (new) 26.58 K - 2.44x slower +22.21 μs" + ], + "memory_note": "**All measurements for memory usage were the same**", + "cold_call": "6654.0 us", + "second_call": "93.0 us" + } + }, + "encode_decode": { + "raw": "lua 1.0.0 — encode!/decode! decomposition\n(decode+deep_cast column: enabled)\n==================================================================================\nop shape N total_us per_elem_ns\nencode int_list 8 0.90 113.0\ndecode int_list 8 0.28 34.7\ndec+cast int_list 8 0.31 38.5\nencode int_list 64 13.52 211.2\ndecode int_list 64 4.23 66.0\ndec+cast int_list 64 7.14 111.6\nencode int_list 512 151.37 295.6\ndecode int_list 512 48.44 94.6\ndec+cast int_list 512 73.99 144.5\nencode int_list 4096 1601.11 390.9\ndecode int_list 4096 583.70 142.5\ndec+cast int_list 4096 785.61 191.8\n----------------------------------------------------------------------------------\nencode float_list 8 0.78 97.1\ndecode float_list 8 0.24 29.8\ndec+cast float_list 8 0.26 32.9\nencode float_list 64 11.47 179.2\ndecode float_list 64 3.55 55.4\ndec+cast float_list 64 7.03 109.9\nencode float_list 512 156.43 305.5\ndecode float_list 512 45.74 89.3\ndec+cast float_list 512 72.74 142.1\nencode float_list 4096 1611.38 393.4\ndecode float_list 4096 551.08 134.5\ndec+cast float_list 4096 839.19 204.9\n----------------------------------------------------------------------------------\nencode bool_list 8 0.77 96.7\ndecode bool_list 8 0.23 28.7\ndec+cast bool_list 8 0.25 31.3\nencode bool_list 64 11.21 175.2\ndecode bool_list 64 3.46 54.1\ndec+cast bool_list 64 7.00 109.4\nencode bool_list 512 153.74 300.3\ndecode bool_list 512 47.42 92.6\ndec+cast bool_list 512 74.01 144.6\nencode bool_list 4096 1569.25 383.1\ndecode bool_list 4096 571.91 139.6\ndec+cast bool_list 4096 775.34 189.3\n----------------------------------------------------------------------------------\nencode short_string_list 8 0.78 96.9\ndecode short_string_list 8 0.24 29.8\ndec+cast short_string_list 8 0.26 32.6\nencode short_string_list 64 11.40 178.1\ndecode short_string_list 64 3.53 55.2\ndec+cast short_string_list 64 7.08 110.6\nencode short_string_list 512 155.05 302.8\ndecode short_string_list 512 49.88 97.4\ndec+cast short_string_list 512 72.83 142.2\nencode short_string_list 4096 1715.16 418.7\ndecode short_string_list 4096 718.28 175.4\ndec+cast short_string_list 4096 763.05 186.3\n----------------------------------------------------------------------------------\nencode long_string_list 8 0.78 96.9\ndecode long_string_list 8 0.24 30.5\ndec+cast long_string_list 8 0.26 33.0\nencode long_string_list 64 11.20 174.9\ndecode long_string_list 64 3.49 54.5\ndec+cast long_string_list 64 7.31 114.3\nencode long_string_list 512 157.18 307.0\ndecode long_string_list 512 42.76 83.5\ndec+cast long_string_list 512 76.03 148.5\nencode long_string_list 4096 2488.38 607.5\ndecode long_string_list 4096 1495.20 365.0\ndec+cast long_string_list 4096 1707.00 416.7\n----------------------------------------------------------------------------------\nencode string_map 8 1.60 199.5\ndecode string_map 8 0.10 12.4\ndec+cast string_map 8 0.20 25.5\nencode string_map 64 30.47 476.1\ndecode string_map 64 0.74 11.6\ndec+cast string_map 64 3.95 61.7\nencode string_map 512 240.62 470.0\ndecode string_map 512 11.04 21.6\ndec+cast string_map 512 39.76 77.7\nencode string_map 4096 2194.00 535.6\ndecode string_map 4096 71.08 17.4\ndec+cast string_map 4096 335.83 82.0\n----------------------------------------------------------------------------------\nencode int_map 8 0.83 103.4\ndecode int_map 8 0.24 29.8\ndec+cast int_map 8 0.26 32.2\nencode int_map 64 10.97 171.5\ndecode int_map 64 3.68 57.5\ndec+cast int_map 64 5.96 93.1\nencode int_map 512 109.35 213.6\ndecode int_map 512 27.78 54.3\ndec+cast int_map 512 72.41 141.4\nencode int_map 4096 1307.14 319.1\ndecode int_map 4096 576.06 140.6\ndec+cast int_map 4096 881.09 215.1\n----------------------------------------------------------------------------------\nencode record_list 8 5.07 633.8\ndecode record_list 8 0.78 98.1\ndec+cast record_list 8 1.20 149.9\nencode record_list 64 47.83 747.4\ndecode record_list 64 8.26 129.1\ndec+cast record_list 64 14.64 228.8\nencode record_list 512 565.88 1105.2\ndecode record_list 512 118.02 230.5\ndec+cast record_list 512 196.29 383.4\nencode record_list 4096 4732.25 1155.3\ndecode record_list 4096 1094.36 267.2\ndec+cast record_list 4096 1628.34 397.5\n----------------------------------------------------------------------------------\nnested chain (depth sweep) — isolates recursion/traversal from fan-out\nop shape N total_us per_elem_ns\nencode nested_chain 4 1.07 535.2\ndecode nested_chain 4 0.21 103.7\ndec+cast nested_chain 4 0.32 161.9\nencode nested_chain 16 4.19 2094.1\ndecode nested_chain 16 0.91 456.5\ndec+cast nested_chain 16 1.33 665.1\nencode nested_chain 64 16.94 8469.5\ndecode nested_chain 64 3.92 1961.9\ndec+cast nested_chain 64 6.28 3140.2\nencode nested_chain 256 68.77 34386.2\ndecode nested_chain 256 18.23 9114.9\ndec+cast nested_chain 256 32.31 16156.7\n==================================================================================\ncomposite anchor — the PR's `original_nested` (matches the 18us/108us figure)\nop shape N total_us per_elem_ns\nencode original_nested 75 17.11 4276.4\ndecode original_nested 75 3.21 802.7\ndec+cast original_nested 75 5.26 1314.8\n" + } +} diff --git a/bench_results/v1.0.0/table_ops.txt b/bench_results/v1.0.0/table_ops.txt new file mode 100644 index 00000000..5b463f7d --- /dev/null +++ b/bench_results/v1.0.0/table_ops.txt @@ -0,0 +1,461 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== Table Build (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +luerl 6.37 K 157.04 μs ±11.82% 154.21 μs 218.16 μs +lua (chunk) 6.23 K 160.45 μs ±8.91% 157.88 μs 198.69 μs +lua (eval) 5.90 K 169.48 μs ±14.14% 164.58 μs 287.61 μs + +Comparison: +luerl 6.37 K +lua (chunk) 6.23 K - 1.02x slower +3.41 μs +lua (eval) 5.90 K - 1.08x slower +12.44 μs + +Memory usage statistics: + +Name Memory usage +luerl 0.97 MB +lua (chunk) 1.00 MB - 1.03x memory usage +0.0267 MB +lua (eval) 1.01 MB - 1.04x memory usage +0.0361 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 60.98 K 16.40 μs ±32.33% 16.17 μs 24.29 μs +luerl 58.70 K 17.04 μs ±26.63% 16.50 μs 26.67 μs +lua (eval) 53.06 K 18.85 μs ±33.45% 18.42 μs 27.04 μs + +Comparison: +lua (chunk) 60.98 K +luerl 58.70 K - 1.04x slower +0.64 μs +lua (eval) 53.06 K - 1.15x slower +2.45 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 105.41 KB +luerl 110.77 KB - 1.05x memory usage +5.35 KB +lua (eval) 114.74 KB - 1.09x memory usage +9.33 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 437.49 K 2.29 μs ±237.41% 2.25 μs 3.42 μs +luerl 317.31 K 3.15 μs ±243.81% 3.04 μs 4.71 μs +lua (eval) 205.32 K 4.87 μs ±231.75% 4.46 μs 10.42 μs + +Comparison: +lua (chunk) 437.49 K +luerl 317.31 K - 1.38x slower +0.87 μs +lua (eval) 205.32 K - 2.13x slower +2.58 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 13.80 KB +luerl 22.62 KB - 1.64x memory usage +8.82 KB +lua (eval) 23.45 KB - 1.70x memory usage +9.66 KB + +**All measurements for memory usage were the same** + +=== Table Sort (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +luerl 5.52 K 181.29 μs ±10.01% 178.00 μs 220.42 μs +lua (chunk) 4.27 K 234.42 μs ±13.69% 227.63 μs 411.33 μs +lua (eval) 4.19 K 238.72 μs ±9.85% 234.88 μs 335.73 μs + +Comparison: +luerl 5.52 K +lua (chunk) 4.27 K - 1.29x slower +53.13 μs +lua (eval) 4.19 K - 1.32x slower +57.42 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.18 MB +lua (chunk) 1.33 MB - 1.13x memory usage +0.156 MB +lua (eval) 1.34 MB - 1.14x memory usage +0.164 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +luerl 50.83 K 19.67 μs ±20.44% 19.13 μs 28.17 μs +lua (chunk) 42.73 K 23.40 μs ±35.27% 23.21 μs 26.75 μs +lua (eval) 38.35 K 26.07 μs ±37.69% 25.58 μs 35.33 μs + +Comparison: +luerl 50.83 K +lua (chunk) 42.73 K - 1.19x slower +3.73 μs +lua (eval) 38.35 K - 1.33x slower +6.40 μs + +Memory usage statistics: + +Name Memory usage +luerl 133.33 KB +lua (chunk) 141.38 KB - 1.06x memory usage +8.05 KB +lua (eval) 151.13 KB - 1.13x memory usage +17.80 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 311.25 K 3.21 μs ±183.71% 3.17 μs 4.50 μs +luerl 266.96 K 3.75 μs ±163.56% 3.58 μs 9.50 μs +lua (eval) 169.47 K 5.90 μs ±170.93% 5.50 μs 15.75 μs + +Comparison: +lua (chunk) 311.25 K +luerl 266.96 K - 1.17x slower +0.53 μs +lua (eval) 169.47 K - 1.84x slower +2.69 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 19.06 KB +luerl 25.98 KB - 1.36x memory usage +6.91 KB +lua (eval) 28.78 KB - 1.51x memory usage +9.72 KB + +**All measurements for memory usage were the same** + +=== Table Iterate/Sum (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 4.10 K 243.68 μs ±11.81% 235.09 μs 342.27 μs +lua (eval) 4.07 K 245.42 μs ±12.09% 241.67 μs 362.47 μs +luerl 4.06 K 246.09 μs ±7.76% 243.06 μs 342.83 μs + +Comparison: +lua (chunk) 4.10 K +lua (eval) 4.07 K - 1.01x slower +1.75 μs +luerl 4.06 K - 1.01x slower +2.41 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.47 MB +lua (eval) 1.48 MB - 1.01x memory usage +0.0110 MB +luerl 1.35 MB - 0.92x memory usage -0.12469 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 40.08 K 24.95 μs ±25.49% 24.25 μs 52.88 μs +lua (eval) 36.72 K 27.23 μs ±17.19% 26.67 μs 38.21 μs +luerl 36.18 K 27.64 μs ±35.39% 27.00 μs 57.79 μs + +Comparison: +lua (chunk) 40.08 K +lua (eval) 36.72 K - 1.09x slower +2.28 μs +luerl 36.18 K - 1.11x slower +2.69 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 154.27 KB +lua (eval) 164.09 KB - 1.06x memory usage +9.81 KB +luerl 149.14 KB - 0.97x memory usage -5.13281 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 306.46 K 3.26 μs ±172.56% 3.21 μs 4.38 μs +luerl 231.29 K 4.32 μs ±138.54% 4.25 μs 6.33 μs +lua (eval) 177.92 K 5.62 μs ±101.11% 5.42 μs 9.25 μs + +Comparison: +lua (chunk) 306.46 K +luerl 231.29 K - 1.33x slower +1.06 μs +lua (eval) 177.92 K - 1.72x slower +2.36 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 19.87 KB +luerl 26.80 KB - 1.35x memory usage +6.93 KB +lua (eval) 29.51 KB - 1.49x memory usage +9.64 KB + +**All measurements for memory usage were the same** + +=== Table Map + Reduce (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +luerl 2.15 K 464.93 μs ±7.90% 449.96 μs 600.72 μs +lua (chunk) 2.05 K 487.99 μs ±6.39% 480.24 μs 622.53 μs +lua (eval) 2.01 K 496.87 μs ±13.21% 487.17 μs 772.35 μs + +Comparison: +luerl 2.15 K +lua (chunk) 2.05 K - 1.05x slower +23.06 μs +lua (eval) 2.01 K - 1.07x slower +31.94 μs + +Memory usage statistics: + +Name Memory usage +luerl 2.44 MB +lua (chunk) 2.92 MB - 1.19x memory usage +0.47 MB +lua (eval) 2.93 MB - 1.20x memory usage +0.48 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +luerl 21.41 K 46.71 μs ±6.09% 46.50 μs 52.29 μs +lua (chunk) 20.87 K 47.91 μs ±8.43% 47.54 μs 53.79 μs +lua (eval) 19.56 K 51.12 μs ±6.66% 50.58 μs 60.46 μs + +Comparison: +luerl 21.41 K +lua (chunk) 20.87 K - 1.03x slower +1.20 μs +lua (eval) 19.56 K - 1.09x slower +4.40 μs + +Memory usage statistics: + +Name Memory usage +luerl 262.35 KB +lua (chunk) 303.87 KB - 1.16x memory usage +41.52 KB +lua (eval) 314.32 KB - 1.20x memory usage +51.97 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 168.91 K 5.92 μs ±76.95% 5.83 μs 13.38 μs +luerl 145.75 K 6.86 μs ±94.69% 6.59 μs 15.96 μs +lua (eval) 102.03 K 9.80 μs ±71.35% 8.21 μs 22.96 μs + +Comparison: +lua (chunk) 168.91 K +luerl 145.75 K - 1.16x slower +0.94 μs +lua (eval) 102.03 K - 1.66x slower +3.88 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 37.09 KB +luerl 39.39 KB - 1.06x memory usage +2.30 KB +lua (eval) 46.72 KB - 1.26x memory usage +9.63 KB + +**All measurements for memory usage were the same** + +=== Table Pairs (hash) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 1.02 K 984.62 μs ±7.98% 995.70 μs 1143.86 μs +lua (eval) 1.01 K 990.11 μs ±8.33% 1002.18 μs 1157.35 μs +luerl 0.75 K 1332.28 μs ±5.93% 1334.35 μs 1489.09 μs + +Comparison: +lua (chunk) 1.02 K +lua (eval) 1.01 K - 1.01x slower +5.49 μs +luerl 0.75 K - 1.35x slower +347.66 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 2.06 MB +lua (eval) 2.05 MB - 0.99x memory usage -0.01064 MB +luerl 2.48 MB - 1.20x memory usage +0.42 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 13.46 K 74.27 μs ±10.12% 72.92 μs 99.63 μs +lua (eval) 12.33 K 81.10 μs ±25.10% 78.13 μs 167.55 μs +luerl 10.92 K 91.56 μs ±15.90% 86.83 μs 142.16 μs + +Comparison: +lua (chunk) 13.46 K +lua (eval) 12.33 K - 1.09x slower +6.83 μs +luerl 10.92 K - 1.23x slower +17.29 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 211.49 KB +lua (eval) 221.46 KB - 1.05x memory usage +9.97 KB +luerl 248.84 KB - 1.18x memory usage +37.34 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 130.58 K 7.66 μs ±70.51% 7.33 μs 24.42 μs +luerl 130.19 K 7.68 μs ±68.45% 7.25 μs 23.38 μs +lua (eval) 94.66 K 10.56 μs ±56.52% 9.92 μs 29.25 μs + +Comparison: +lua (chunk) 130.58 K +luerl 130.19 K - 1.00x slower +0.0229 μs +lua (eval) 94.66 K - 1.38x slower +2.91 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 25.10 KB +luerl 36.16 KB - 1.44x memory usage +11.05 KB +lua (eval) 34.91 KB - 1.39x memory usage +9.81 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.0/timestamp.txt b/bench_results/v1.0.0/timestamp.txt new file mode 100644 index 00000000..ff5029ff --- /dev/null +++ b/bench_results/v1.0.0/timestamp.txt @@ -0,0 +1 @@ +Tue Jul 28 10:48:16 EDT 2026 diff --git a/bench_results/v1.0.0/versions.txt b/bench_results/v1.0.0/versions.txt new file mode 100644 index 00000000..184e5f9a --- /dev/null +++ b/bench_results/v1.0.0/versions.txt @@ -0,0 +1,3 @@ +Erlang/OTP 29 [erts-17.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] + +Elixir 1.20.0 (compiled with Erlang/OTP 29) diff --git a/bench_results/v1.0.0/vm_new.txt b/bench_results/v1.0.0/vm_new.txt new file mode 100644 index 00000000..9ade4d52 --- /dev/null +++ b/bench_results/v1.0.0/vm_new.txt @@ -0,0 +1,56 @@ +=== VM instantiation: Lua.new/1 vs :luerl.init/0 (mode: full) === + +Lua.new() one-time vs repeat cost (single samples, informational): + first call on this node : 6654.0 us + second call : 93.0 us + +The first figure includes any one-time template build and first-time module +loading. Benchee's steady-state numbers below are the per-request cost after +that point. + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 52 s +Excluding outliers: false + +Benchmarking lua (new) ... +Benchmarking lua (new, custom exclude) ... +Benchmarking lua (new, no sandbox) ... +Benchmarking luerl (init) ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl (init) 64.86 K 15.42 μs ±23.43% 15.21 μs 24.25 μs +lua (new, no sandbox) 33.30 K 30.03 μs ±11.23% 29.71 μs 38.46 μs +lua (new, custom exclude) 27.37 K 36.54 μs ±8.96% 35.96 μs 42.63 μs +lua (new) 26.58 K 37.63 μs ±62.22% 36.67 μs 71.96 μs + +Comparison: +luerl (init) 64.86 K +lua (new, no sandbox) 33.30 K - 1.95x slower +14.62 μs +lua (new, custom exclude) 27.37 K - 2.37x slower +21.12 μs +lua (new) 26.58 K - 2.44x slower +22.21 μs + +Memory usage statistics: + +Name Memory usage +luerl (init) 51.64 KB +lua (new, no sandbox) 68.76 KB - 1.33x memory usage +17.12 KB +lua (new, custom exclude) 90.97 KB - 1.76x memory usage +39.33 KB +lua (new) 91.88 KB - 1.78x memory usage +40.23 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/closures.txt b/bench_results/v1.0.2/closures.txt new file mode 100644 index 00000000..c2a370a5 --- /dev/null +++ b/bench_results/v1.0.2/closures.txt @@ -0,0 +1,46 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 2.68 K 372.50 μs ±10.92% 365.04 μs 532.92 μs +lua (chunk) 2.63 K 379.93 μs ±8.39% 373.71 μs 512.83 μs +luerl 2.53 K 395.50 μs ±7.19% 389.21 μs 548.26 μs + +Comparison: +lua (eval) 2.68 K +lua (chunk) 2.63 K - 1.02x slower +7.43 μs +luerl 2.53 K - 1.06x slower +23.01 μs + +Memory usage statistics: + +Name average deviation median 99th % +lua (eval) 2.12 MB ±0.11% 2.12 MB 2.13 MB +lua (chunk) 2.11 MB ±0.11% 2.11 MB 2.12 MB +luerl 1.90 MB ±0.00% 1.90 MB 1.90 MB + +Comparison: +lua (eval) 2.12 MB +lua (chunk) 2.11 MB - 1.00x memory usage -0.01023 MB +luerl 1.90 MB - 0.89x memory usage -0.22699 MB diff --git a/bench_results/v1.0.2/commit.txt b/bench_results/v1.0.2/commit.txt new file mode 100644 index 00000000..27bf7c3e --- /dev/null +++ b/bench_results/v1.0.2/commit.txt @@ -0,0 +1 @@ +3a0d3923249cbb05380a3f17d3920a0b3ed55cb0 diff --git a/bench_results/v1.0.2/cpu.txt b/bench_results/v1.0.2/cpu.txt new file mode 100644 index 00000000..de5c8ad6 --- /dev/null +++ b/bench_results/v1.0.2/cpu.txt @@ -0,0 +1 @@ +Apple M4 diff --git a/bench_results/v1.0.2/encode_decode.txt b/bench_results/v1.0.2/encode_decode.txt new file mode 100644 index 00000000..204913bc --- /dev/null +++ b/bench_results/v1.0.2/encode_decode.txt @@ -0,0 +1,128 @@ +lua 1.0.2 — encode!/decode! decomposition +(decode+deep_cast column: enabled) +================================================================================== +op shape N total_us per_elem_ns +encode int_list 8 0.81 101.5 +decode int_list 8 0.29 35.9 +dec+cast int_list 8 0.33 40.9 +encode int_list 64 14.59 228.0 +decode int_list 64 4.94 77.1 +dec+cast int_list 64 7.11 111.0 +encode int_list 512 158.65 309.9 +decode int_list 512 47.04 91.9 +dec+cast int_list 512 72.78 142.1 +encode int_list 4096 1568.66 383.0 +decode int_list 4096 565.55 138.1 +dec+cast int_list 4096 892.14 217.8 +---------------------------------------------------------------------------------- +encode float_list 8 0.78 98.0 +decode float_list 8 0.25 31.8 +dec+cast float_list 8 0.27 33.6 +encode float_list 64 11.77 183.9 +decode float_list 64 3.76 58.7 +dec+cast float_list 64 6.99 109.2 +encode float_list 512 162.64 317.6 +decode float_list 512 47.30 92.4 +dec+cast float_list 512 75.86 148.2 +encode float_list 4096 1614.23 394.1 +decode float_list 4096 547.77 133.7 +dec+cast float_list 4096 805.73 196.7 +---------------------------------------------------------------------------------- +encode bool_list 8 0.79 99.1 +decode bool_list 8 0.24 29.9 +dec+cast bool_list 8 0.27 33.9 +encode bool_list 64 14.65 228.8 +decode bool_list 64 4.69 73.3 +dec+cast bool_list 64 6.88 107.4 +encode bool_list 512 158.50 309.6 +decode bool_list 512 45.36 88.6 +dec+cast bool_list 512 71.69 140.0 +encode bool_list 4096 1572.86 384.0 +decode bool_list 4096 581.14 141.9 +dec+cast bool_list 4096 792.78 193.6 +---------------------------------------------------------------------------------- +encode short_string_list 8 0.78 97.8 +decode short_string_list 8 0.24 30.0 +dec+cast short_string_list 8 0.27 34.2 +encode short_string_list 64 11.49 179.6 +decode short_string_list 64 3.51 54.9 +dec+cast short_string_list 64 6.93 108.3 +encode short_string_list 512 162.32 317.0 +decode short_string_list 512 48.13 94.0 +dec+cast short_string_list 512 74.26 145.0 +encode short_string_list 4096 1664.81 406.4 +decode short_string_list 4096 550.34 134.4 +dec+cast short_string_list 4096 810.06 197.8 +---------------------------------------------------------------------------------- +encode long_string_list 8 0.79 98.9 +decode long_string_list 8 0.25 31.1 +dec+cast long_string_list 8 0.26 33.0 +encode long_string_list 64 11.67 182.4 +decode long_string_list 64 3.66 57.3 +dec+cast long_string_list 64 6.75 105.5 +encode long_string_list 512 169.63 331.3 +decode long_string_list 512 47.95 93.7 +dec+cast long_string_list 512 75.67 147.8 +encode long_string_list 4096 2667.06 651.1 +decode long_string_list 4096 1339.42 327.0 +dec+cast long_string_list 4096 1705.52 416.4 +---------------------------------------------------------------------------------- +encode string_map 8 0.72 89.9 +decode string_map 8 0.11 13.2 +dec+cast string_map 8 0.21 26.1 +encode string_map 64 12.18 190.3 +decode string_map 64 0.74 11.6 +dec+cast string_map 64 3.58 56.0 +encode string_map 512 103.35 201.9 +decode string_map 512 9.32 18.2 +dec+cast string_map 512 39.57 77.3 +encode string_map 4096 1046.22 255.4 +decode string_map 4096 79.30 19.4 +dec+cast string_map 4096 343.00 83.7 +---------------------------------------------------------------------------------- +encode int_map 8 0.81 100.9 +decode int_map 8 0.25 31.6 +dec+cast int_map 8 0.27 33.9 +encode int_map 64 11.36 177.6 +decode int_map 64 3.45 54.0 +dec+cast int_map 64 5.64 88.1 +encode int_map 512 111.00 216.8 +decode int_map 512 32.55 63.6 +dec+cast int_map 512 72.18 141.0 +encode int_map 4096 1503.89 367.2 +decode int_map 4096 599.78 146.4 +dec+cast int_map 4096 850.84 207.7 +---------------------------------------------------------------------------------- +encode record_list 8 3.10 387.3 +decode record_list 8 0.89 111.3 +dec+cast record_list 8 1.60 199.4 +encode record_list 64 45.68 713.8 +decode record_list 64 12.94 202.3 +dec+cast record_list 64 18.97 296.4 +encode record_list 512 422.71 825.6 +decode record_list 512 115.71 226.0 +dec+cast record_list 512 154.32 301.4 +encode record_list 4096 3814.88 931.4 +decode record_list 4096 1139.67 278.2 +dec+cast record_list 4096 1656.06 404.3 +---------------------------------------------------------------------------------- +nested chain (depth sweep) — isolates recursion/traversal from fan-out +op shape N total_us per_elem_ns +encode nested_chain 4 0.68 338.0 +decode nested_chain 4 0.24 122.2 +dec+cast nested_chain 4 0.36 179.7 +encode nested_chain 16 2.77 1387.2 +decode nested_chain 16 1.11 552.9 +dec+cast nested_chain 16 1.54 768.1 +encode nested_chain 64 10.94 5468.5 +decode nested_chain 64 6.00 3002.0 +dec+cast nested_chain 64 8.37 4183.6 +encode nested_chain 256 44.55 22273.9 +decode nested_chain 256 28.00 13998.0 +dec+cast nested_chain 256 37.24 18618.2 +================================================================================== +composite anchor — the PR's `original_nested` (matches the 18us/108us figure) +op shape N total_us per_elem_ns +encode original_nested 75 12.26 3065.6 +decode original_nested 75 3.29 821.3 +dec+cast original_nested 75 5.66 1413.8 diff --git a/bench_results/v1.0.2/environment.md b/bench_results/v1.0.2/environment.md new file mode 100644 index 00000000..453867ba --- /dev/null +++ b/bench_results/v1.0.2/environment.md @@ -0,0 +1,36 @@ +# Benchmark environment — main (1.0.2) + +- **Ref**: `main` — the main checkout itself, commit `3a0d3923249cbb05380a3f17d3920a0b3ed55cb0` (the 1.0.2 release). No worktree was created or removed for this run; the main checkout was read but not modified. +- **Mode**: `full` (`LUA_BENCH_MODE=full`) +- **CPU**: Apple M4 (see `cpu.txt`) +- **Elixir / OTP**: Elixir 1.20.0, Erlang/OTP 29 [erts-17.0] [64-bit] [jit] (see `versions.txt`) +- **Run timestamp**: Tue Jul 28 11:14:31 EDT 2026 (see `timestamp.txt`) +- **This document written**: Tue Jul 28 11:37:22 EDT 2026 + +## Command form + +Each workload was run serially, one process per file, on an otherwise quiet +machine, from the main checkout root: + +``` +LUA_BENCH_MODE=full MIX_ENV=benchmark mix run benchmarks/.exs +``` + +Workloads run: `fibonacci`, `closures`, `oop`, `string_ops`, `string_format`, +`table_ops`, `patterns`, `metamethods`, `pcall_varargs`, `vm_new`, +`encode_decode`. + +`encode_decode.exs` is not Benchee-based (it uses a `:timer.tc` harness +directly), but was invoked with the same command form for consistency. + +C Lua via `luaport` was not available in this environment (no local luaport +build) and was skipped by each script's own fallback path; all comparisons +below are lua (chunk)/lua (eval) vs. luerl only. + +## Artifacts + +- `cpu.txt`, `versions.txt`, `timestamp.txt`, `commit.txt` — raw environment + probes. +- `.txt` — raw stdout of each `mix run` invocation (11 files). +- `summary.json` — parsed structured form of the above (workload -> case -> + jobs/comparison/memory), same schema as `results/v1.0.0/summary.json`. diff --git a/bench_results/v1.0.2/fibonacci.txt b/bench_results/v1.0.2/fibonacci.txt new file mode 100644 index 00000000..153dfe11 --- /dev/null +++ b/bench_results/v1.0.2/fibonacci.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 2.32 431.14 ms ±1.78% 429.30 ms 454.01 ms +lua (chunk) 2.30 435.33 ms ±2.05% 434.12 ms 469.44 ms +luerl 1.35 742.61 ms ±1.33% 741.95 ms 764.71 ms + +Comparison: +lua (eval) 2.32 +lua (chunk) 2.30 - 1.01x slower +4.18 ms +luerl 1.35 - 1.72x slower +311.46 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 1016.62 MB +lua (chunk) 1016.63 MB - 1.00x memory usage +0.00925 MB +luerl 2513.67 MB - 2.47x memory usage +1497.05 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/metamethods.txt b/bench_results/v1.0.2/metamethods.txt new file mode 100644 index 00000000..5079db11 --- /dev/null +++ b/bench_results/v1.0.2/metamethods.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== metamethods: self-call method dispatch (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 2.10 K 476.79 μs ±13.40% 462.04 μs 730.43 μs +luerl 2.05 K 487.93 μs ±8.87% 474.63 μs 657.89 μs +lua (chunk) 2.04 K 490.34 μs ±7.16% 485.83 μs 642.74 μs + +Comparison: +lua (eval) 2.10 K +luerl 2.05 K - 1.02x slower +11.14 μs +lua (chunk) 2.04 K - 1.03x slower +13.55 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 1.37 MB +luerl 1.43 MB - 1.04x memory usage +0.0544 MB +lua (chunk) 1.36 MB - 0.99x memory usage -0.01070 MB + +**All measurements for memory usage were the same** + +=== metamethods: 3-level __index chain (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 5.24 K 190.75 μs ±4.32% 190.08 μs 206.47 μs +lua (eval) 5.09 K 196.41 μs ±6.91% 194.58 μs 221.08 μs +luerl 4.60 K 217.23 μs ±4.39% 216 μs 233.25 μs + +Comparison: +lua (chunk) 5.24 K +lua (eval) 5.09 K - 1.03x slower +5.65 μs +luerl 4.60 K - 1.14x slower +26.48 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 576.76 KB +lua (eval) 587.70 KB - 1.02x memory usage +10.95 KB +luerl 668.59 KB - 1.16x memory usage +91.83 KB + +**All measurements for memory usage were the same** + +=== metamethods: arithmetic/relational metamethods (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 533.57 1.87 ms ±5.87% 1.92 ms 2.18 ms +luerl 528.86 1.89 ms ±5.87% 1.85 ms 2.20 ms +lua (chunk) 521.21 1.92 ms ±12.56% 1.93 ms 3.10 ms + +Comparison: +lua (eval) 533.57 +luerl 528.86 - 1.01x slower +0.0167 ms +lua (chunk) 521.21 - 1.02x slower +0.0444 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 5.71 MB +luerl 6.24 MB - 1.09x memory usage +0.53 MB +lua (chunk) 5.70 MB - 1.00x memory usage -0.01050 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/oop.txt b/bench_results/v1.0.2/oop.txt new file mode 100644 index 00000000..1e6a3aeb --- /dev/null +++ b/bench_results/v1.0.2/oop.txt @@ -0,0 +1,43 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 10.47 K 95.55 μs ±13.61% 93.21 μs 154.21 μs +lua (eval) 10.27 K 97.33 μs ±13.81% 94.67 μs 153.75 μs +luerl 7.14 K 140.03 μs ±21.43% 133.50 μs 231.96 μs + +Comparison: +lua (chunk) 10.47 K +lua (eval) 10.27 K - 1.02x slower +1.78 μs +luerl 7.14 K - 1.47x slower +44.48 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 368.40 KB +lua (eval) 378.94 KB - 1.03x memory usage +10.54 KB +luerl 381.45 KB - 1.04x memory usage +13.05 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/patterns.txt b/bench_results/v1.0.2/patterns.txt new file mode 100644 index 00000000..9bd65efa --- /dev/null +++ b/bench_results/v1.0.2/patterns.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== patterns: find/match field extraction (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 998.95 1.00 ms ±3.02% 1.00 ms 1.06 ms +lua (eval) 986.14 1.01 ms ±10.37% 1.00 ms 1.36 ms +luerl 923.97 1.08 ms ±3.61% 1.08 ms 1.18 ms + +Comparison: +lua (chunk) 998.95 +lua (eval) 986.14 - 1.01x slower +0.0130 ms +luerl 923.97 - 1.08x slower +0.0812 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 3.21 MB +lua (eval) 3.22 MB - 1.00x memory usage +0.0110 MB +luerl 6.70 MB - 2.09x memory usage +3.49 MB + +**All measurements for memory usage were the same** + +=== patterns: find-based tokenizer (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 729.19 1.37 ms ±3.06% 1.36 ms 1.48 ms +lua (chunk) 673.77 1.48 ms ±3.57% 1.46 ms 1.61 ms +lua (eval) 661.88 1.51 ms ±4.44% 1.49 ms 1.76 ms + +Comparison: +luerl 729.19 +lua (chunk) 673.77 - 1.08x slower +0.113 ms +lua (eval) 661.88 - 1.10x slower +0.139 ms + +Memory usage statistics: + +Name Memory usage +luerl 6.65 MB +lua (chunk) 9.10 MB - 1.37x memory usage +2.45 MB +lua (eval) 9.11 MB - 1.37x memory usage +2.46 MB + +**All measurements for memory usage were the same** + +=== patterns: gsub template substitution (n=200) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 513.54 1.95 ms ±2.83% 1.95 ms 2.06 ms +lua (chunk) 480.12 2.08 ms ±16.12% 1.98 ms 3.12 ms +luerl 418.97 2.39 ms ±17.21% 2.20 ms 3.51 ms + +Comparison: +lua (eval) 513.54 +lua (chunk) 480.12 - 1.07x slower +0.136 ms +luerl 418.97 - 1.23x slower +0.44 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 4.50 MB +lua (chunk) 4.49 MB - 1.00x memory usage -0.01124 MB +luerl 11.75 MB - 2.61x memory usage +7.25 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/pcall_varargs.txt b/bench_results/v1.0.2/pcall_varargs.txt new file mode 100644 index 00000000..ebeacdd5 --- /dev/null +++ b/bench_results/v1.0.2/pcall_varargs.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== call protocol: pcall, success path (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 4.47 K 223.70 μs ±4.04% 222.88 μs 241.97 μs +lua (eval) 4.44 K 225.34 μs ±6.09% 224.42 μs 243.03 μs +luerl 3.40 K 294.17 μs ±6.42% 291.54 μs 317.97 μs + +Comparison: +lua (chunk) 4.47 K +lua (eval) 4.44 K - 1.01x slower +1.63 μs +luerl 3.40 K - 1.31x slower +70.46 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 895.93 KB +lua (eval) 911.78 KB - 1.02x memory usage +15.85 KB +luerl 1137.10 KB - 1.27x memory usage +241.17 KB + +**All measurements for memory usage were the same** + +=== call protocol: pcall, raise + catch (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 2.12 K 471.09 μs ±3.42% 468.42 μs 503.21 μs +lua (chunk) 1.36 K 733.85 μs ±2.60% 727.13 μs 781.15 μs +lua (eval) 1.35 K 742.90 μs ±5.35% 734.58 μs 824.44 μs + +Comparison: +luerl 2.12 K +lua (chunk) 1.36 K - 1.56x slower +262.76 μs +lua (eval) 1.35 K - 1.58x slower +271.81 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.55 MB +lua (chunk) 2.67 MB - 1.72x memory usage +1.12 MB +lua (eval) 2.68 MB - 1.72x memory usage +1.12 MB + +**All measurements for memory usage were the same** + +=== call protocol: varargs + multiple returns (n=500) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +luerl 409.82 2.44 ms ±4.21% 2.41 ms 2.74 ms +lua (eval) 306.73 3.26 ms ±1.81% 3.27 ms 3.44 ms +lua (chunk) 300.00 3.33 ms ±10.97% 3.28 ms 5.01 ms + +Comparison: +luerl 409.82 +lua (eval) 306.73 - 1.34x slower +0.82 ms +lua (chunk) 300.00 - 1.37x slower +0.89 ms + +Memory usage statistics: + +Name Memory usage +luerl 8.73 MB +lua (eval) 16.71 MB - 1.91x memory usage +7.98 MB +lua (chunk) 16.70 MB - 1.91x memory usage +7.97 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/string_format.txt b/bench_results/v1.0.2/string_format.txt new file mode 100644 index 00000000..97284a35 --- /dev/null +++ b/bench_results/v1.0.2/string_format.txt @@ -0,0 +1,136 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== string.format: long literal-heavy format string (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 1.38 K 722.93 μs ±4.47% 716.04 μs 787.74 μs +lua (eval) 1.23 K 815.14 μs ±12.46% 759.94 μs 989.38 μs +luerl 0.25 K 3990.98 μs ±7.83% 3921.50 μs 5009.73 μs + +Comparison: +lua (chunk) 1.38 K +lua (eval) 1.23 K - 1.13x slower +92.22 μs +luerl 0.25 K - 5.52x slower +3268.06 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 2.03 MB +lua (eval) 2.04 MB - 1.01x memory usage +0.0106 MB +luerl 22.59 MB - 11.11x memory usage +20.56 MB + +**All measurements for memory usage were the same** + +=== string.format: width-flagged specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 812.70 1.23 ms ±5.82% 1.22 ms 1.33 ms +lua (eval) 804.55 1.24 ms ±2.99% 1.24 ms 1.32 ms +luerl 571.68 1.75 ms ±3.48% 1.74 ms 1.86 ms + +Comparison: +lua (chunk) 812.70 +lua (eval) 804.55 - 1.01x slower +0.0125 ms +luerl 571.68 - 1.42x slower +0.52 ms + +Memory usage statistics: + +Name Memory usage +lua (chunk) 3.52 MB +lua (eval) 3.53 MB - 1.00x memory usage +0.0132 MB +luerl 7.55 MB - 2.14x memory usage +4.03 MB + +**All measurements for memory usage were the same** + +=== string.format: many specifiers (n=1000) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (eval) 577.23 1.73 ms ±12.08% 1.70 ms 2.30 ms +lua (chunk) 566.54 1.77 ms ±3.99% 1.77 ms 1.94 ms +luerl 359.44 2.78 ms ±7.17% 2.78 ms 3.20 ms + +Comparison: +lua (eval) 577.23 +lua (chunk) 566.54 - 1.02x slower +0.0327 ms +luerl 359.44 - 1.61x slower +1.05 ms + +Memory usage statistics: + +Name Memory usage +lua (eval) 6.64 MB +lua (chunk) 6.63 MB - 1.00x memory usage -0.01004 MB +luerl 13.76 MB - 2.07x memory usage +7.11 MB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/string_ops.txt b/bench_results/v1.0.2/string_ops.txt new file mode 100644 index 00000000..9ac629f1 --- /dev/null +++ b/bench_results/v1.0.2/string_ops.txt @@ -0,0 +1,91 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== String Concatenation via table.concat (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 31.46 K 31.79 μs ±14.86% 31.21 μs 41.21 μs +lua (eval) 29.39 K 34.03 μs ±10.70% 33.63 μs 42.46 μs +luerl 24.71 K 40.47 μs ±7.41% 40.17 μs 45.88 μs + +Comparison: +lua (chunk) 31.46 K +lua (eval) 29.39 K - 1.07x slower +2.24 μs +luerl 24.71 K - 1.27x slower +8.68 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 147.68 KB +lua (eval) 161.46 KB - 1.09x memory usage +13.78 KB +luerl 172.42 KB - 1.17x memory usage +24.74 KB + +**All measurements for memory usage were the same** + +=== String Formatting via string.format (n=100) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 39 s +Excluding outliers: false + +Benchmarking lua (chunk) ... +Benchmarking lua (eval) ... +Benchmarking luerl ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (chunk) 14.08 K 71.03 μs ±8.11% 72.21 μs 80.46 μs +lua (eval) 13.20 K 75.76 μs ±16.78% 76.17 μs 92.94 μs +luerl 9.54 K 104.79 μs ±7.40% 103.04 μs 123.13 μs + +Comparison: +lua (chunk) 14.08 K +lua (eval) 13.20 K - 1.07x slower +4.73 μs +luerl 9.54 K - 1.48x slower +33.75 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 241.46 KB +lua (eval) 252.21 KB - 1.04x memory usage +10.75 KB +luerl 588.84 KB - 2.44x memory usage +347.38 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/summary.json b/bench_results/v1.0.2/summary.json new file mode 100644 index 00000000..c146681e --- /dev/null +++ b/bench_results/v1.0.2/summary.json @@ -0,0 +1,1282 @@ +{ + "fibonacci": { + "default": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "2.32", + "average": "431.14 ms", + "deviation": "±1.78%", + "median": "429.30 ms", + "p99": "454.01 ms", + "memory": "1016.62 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.30", + "average": "435.33 ms", + "deviation": "±2.05%", + "median": "434.12 ms", + "p99": "469.44 ms", + "memory": "1016.63 MB" + }, + { + "name": "luerl", + "ips": "1.35", + "average": "742.61 ms", + "deviation": "±1.33%", + "median": "741.95 ms", + "p99": "764.71 ms", + "memory": "2513.67 MB" + } + ], + "comparison": [ + "lua (eval) 2.32", + "lua (chunk) 2.30 - 1.01x slower +4.18 ms", + "luerl 1.35 - 1.72x slower +311.46 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "closures": { + "default": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "2.68 K", + "average": "372.50 μs", + "deviation": "±10.92%", + "median": "365.04 μs", + "p99": "532.92 μs", + "memory": "2.12 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.63 K", + "average": "379.93 μs", + "deviation": "±8.39%", + "median": "373.71 μs", + "p99": "512.83 μs", + "memory": "2.11 MB" + }, + { + "name": "luerl", + "ips": "2.53 K", + "average": "395.50 μs", + "deviation": "±7.19%", + "median": "389.21 μs", + "p99": "548.26 μs", + "memory": "1.90 MB" + } + ], + "comparison": [ + "lua (eval) 2.68 K", + "lua (chunk) 2.63 K - 1.02x slower +7.43 μs", + "luerl 2.53 K - 1.06x slower +23.01 μs" + ], + "memory_comparison": [ + "lua (eval) 2.12 MB", + "lua (chunk) 2.11 MB - 1.00x memory usage -0.01023 MB", + "luerl 1.90 MB - 0.89x memory usage -0.22699 MB" + ] + } + }, + "oop": { + "default": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "10.47 K", + "average": "95.55 μs", + "deviation": "±13.61%", + "median": "93.21 μs", + "p99": "154.21 μs", + "memory": "368.40 KB" + }, + { + "name": "lua (eval)", + "ips": "10.27 K", + "average": "97.33 μs", + "deviation": "±13.81%", + "median": "94.67 μs", + "p99": "153.75 μs", + "memory": "378.94 KB" + }, + { + "name": "luerl", + "ips": "7.14 K", + "average": "140.03 μs", + "deviation": "±21.43%", + "median": "133.50 μs", + "p99": "231.96 μs", + "memory": "381.45 KB" + } + ], + "comparison": [ + "lua (chunk) 10.47 K", + "lua (eval) 10.27 K - 1.02x slower +1.78 μs", + "luerl 7.14 K - 1.47x slower +44.48 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "string_ops": { + "String Concatenation via table.concat (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "31.46 K", + "average": "31.79 μs", + "deviation": "±14.86%", + "median": "31.21 μs", + "p99": "41.21 μs", + "memory": "147.68 KB" + }, + { + "name": "lua (eval)", + "ips": "29.39 K", + "average": "34.03 μs", + "deviation": "±10.70%", + "median": "33.63 μs", + "p99": "42.46 μs", + "memory": "161.46 KB" + }, + { + "name": "luerl", + "ips": "24.71 K", + "average": "40.47 μs", + "deviation": "±7.41%", + "median": "40.17 μs", + "p99": "45.88 μs", + "memory": "172.42 KB" + } + ], + "comparison": [ + "lua (chunk) 31.46 K", + "lua (eval) 29.39 K - 1.07x slower +2.24 μs", + "luerl 24.71 K - 1.27x slower +8.68 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "String Formatting via string.format (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "14.08 K", + "average": "71.03 μs", + "deviation": "±8.11%", + "median": "72.21 μs", + "p99": "80.46 μs", + "memory": "241.46 KB" + }, + { + "name": "lua (eval)", + "ips": "13.20 K", + "average": "75.76 μs", + "deviation": "±16.78%", + "median": "76.17 μs", + "p99": "92.94 μs", + "memory": "252.21 KB" + }, + { + "name": "luerl", + "ips": "9.54 K", + "average": "104.79 μs", + "deviation": "±7.40%", + "median": "103.04 μs", + "p99": "123.13 μs", + "memory": "588.84 KB" + } + ], + "comparison": [ + "lua (chunk) 14.08 K", + "lua (eval) 13.20 K - 1.07x slower +4.73 μs", + "luerl 9.54 K - 1.48x slower +33.75 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "string_format": { + "string.format: long literal-heavy format string (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "1.38 K", + "average": "722.93 μs", + "deviation": "±4.47%", + "median": "716.04 μs", + "p99": "787.74 μs", + "memory": "2.03 MB" + }, + { + "name": "lua (eval)", + "ips": "1.23 K", + "average": "815.14 μs", + "deviation": "±12.46%", + "median": "759.94 μs", + "p99": "989.38 μs", + "memory": "2.04 MB" + }, + { + "name": "luerl", + "ips": "0.25 K", + "average": "3990.98 μs", + "deviation": "±7.83%", + "median": "3921.50 μs", + "p99": "5009.73 μs", + "memory": "22.59 MB" + } + ], + "comparison": [ + "lua (chunk) 1.38 K", + "lua (eval) 1.23 K - 1.13x slower +92.22 μs", + "luerl 0.25 K - 5.52x slower +3268.06 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "string.format: width-flagged specifiers (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "812.70", + "average": "1.23 ms", + "deviation": "±5.82%", + "median": "1.22 ms", + "p99": "1.33 ms", + "memory": "3.52 MB" + }, + { + "name": "lua (eval)", + "ips": "804.55", + "average": "1.24 ms", + "deviation": "±2.99%", + "median": "1.24 ms", + "p99": "1.32 ms", + "memory": "3.53 MB" + }, + { + "name": "luerl", + "ips": "571.68", + "average": "1.75 ms", + "deviation": "±3.48%", + "median": "1.74 ms", + "p99": "1.86 ms", + "memory": "7.55 MB" + } + ], + "comparison": [ + "lua (chunk) 812.70", + "lua (eval) 804.55 - 1.01x slower +0.0125 ms", + "luerl 571.68 - 1.42x slower +0.52 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "string.format: many specifiers (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "577.23", + "average": "1.73 ms", + "deviation": "±12.08%", + "median": "1.70 ms", + "p99": "2.30 ms", + "memory": "6.64 MB" + }, + { + "name": "lua (chunk)", + "ips": "566.54", + "average": "1.77 ms", + "deviation": "±3.99%", + "median": "1.77 ms", + "p99": "1.94 ms", + "memory": "6.63 MB" + }, + { + "name": "luerl", + "ips": "359.44", + "average": "2.78 ms", + "deviation": "±7.17%", + "median": "2.78 ms", + "p99": "3.20 ms", + "memory": "13.76 MB" + } + ], + "comparison": [ + "lua (eval) 577.23", + "lua (chunk) 566.54 - 1.02x slower +0.0327 ms", + "luerl 359.44 - 1.61x slower +1.05 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "table_ops": { + "Table Build": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "6.74 K", + "average": "148.28 μs", + "deviation": "±10.00%", + "median": "145.92 μs", + "p99": "180.79 μs", + "memory": "1023.05 KB" + }, + { + "name": "lua (eval)", + "ips": "6.55 K", + "average": "152.58 μs", + "deviation": "±18.06%", + "median": "148.88 μs", + "p99": "259.21 μs", + "memory": "1033.39 KB" + }, + { + "name": "luerl", + "ips": "6.48 K", + "average": "154.42 μs", + "deviation": "±9.20%", + "median": "152.71 μs", + "p99": "189.18 μs", + "memory": "996.96 KB" + } + ], + "comparison": [ + "lua (chunk) 6.74 K", + "lua (eval) 6.55 K - 1.03x slower +4.30 μs", + "luerl 6.48 K - 1.04x slower +6.15 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "65.36 K", + "average": "15.30 μs", + "deviation": "±46.45%", + "median": "15 μs", + "p99": "24.63 μs", + "memory": "104.51 KB" + }, + { + "name": "luerl", + "ips": "58.37 K", + "average": "17.13 μs", + "deviation": "±38.71%", + "median": "16.63 μs", + "p99": "27.96 μs", + "memory": "110.78 KB" + }, + { + "name": "lua (eval)", + "ips": "56.97 K", + "average": "17.55 μs", + "deviation": "±21.90%", + "median": "17.21 μs", + "p99": "26.13 μs", + "memory": "115.14 KB" + } + ], + "comparison": [ + "lua (chunk) 65.36 K", + "luerl 58.37 K - 1.12x slower +1.83 μs", + "lua (eval) 56.97 K - 1.15x slower +2.25 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "473.88 K", + "average": "2.11 μs", + "deviation": "±259.11%", + "median": "2.04 μs", + "p99": "3.21 μs", + "memory": "12.89 KB" + }, + { + "name": "luerl", + "ips": "319.03 K", + "average": "3.13 μs", + "deviation": "±185.41%", + "median": "3.04 μs", + "p99": "4.54 μs", + "memory": "22.62 KB" + }, + { + "name": "lua (eval)", + "ips": "234.54 K", + "average": "4.26 μs", + "deviation": "±124.39%", + "median": "4.13 μs", + "p99": "11.13 μs", + "memory": "23.84 KB" + } + ], + "comparison": [ + "lua (chunk) 473.88 K", + "luerl 319.03 K - 1.49x slower +1.02 μs", + "lua (eval) 234.54 K - 2.02x slower +2.15 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Sort": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "luerl", + "ips": "5.41 K", + "average": "184.80 μs", + "deviation": "±18.74%", + "median": "179.09 μs", + "p99": "413.57 μs", + "memory": "1.17 MB" + }, + { + "name": "lua (chunk)", + "ips": "4.88 K", + "average": "205.09 μs", + "deviation": "±13.18%", + "median": "199.34 μs", + "p99": "322.32 μs", + "memory": "1.22 MB" + }, + { + "name": "lua (eval)", + "ips": "4.82 K", + "average": "207.48 μs", + "deviation": "±16.82%", + "median": "202.59 μs", + "p99": "278.13 μs", + "memory": "1.24 MB" + } + ], + "comparison": [ + "luerl 5.41 K", + "lua (chunk) 4.88 K - 1.11x slower +20.29 μs", + "lua (eval) 4.82 K - 1.12x slower +22.68 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "luerl", + "ips": "51.19 K", + "average": "19.54 μs", + "deviation": "±19.26%", + "median": "19.04 μs", + "p99": "28.13 μs", + "memory": "133.16 KB" + }, + { + "name": "lua (chunk)", + "ips": "47.57 K", + "average": "21.02 μs", + "deviation": "±75.79%", + "median": "20.38 μs", + "p99": "44.00 μs", + "memory": "129.16 KB" + }, + { + "name": "lua (eval)", + "ips": "43.10 K", + "average": "23.20 μs", + "deviation": "±19.83%", + "median": "22.79 μs", + "p99": "28.67 μs", + "memory": "140.11 KB" + } + ], + "comparison": [ + "luerl 51.19 K", + "lua (chunk) 47.57 K - 1.08x slower +1.49 μs", + "lua (eval) 43.10 K - 1.19x slower +3.67 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "343.00 K", + "average": "2.92 μs", + "deviation": "±179.73%", + "median": "2.88 μs", + "p99": "4.17 μs", + "memory": "16.81 KB" + }, + { + "name": "luerl", + "ips": "267.77 K", + "average": "3.73 μs", + "deviation": "±158.30%", + "median": "3.63 μs", + "p99": "5.67 μs", + "memory": "25.98 KB" + }, + { + "name": "lua (eval)", + "ips": "193.98 K", + "average": "5.16 μs", + "deviation": "±94.77%", + "median": "5.04 μs", + "p99": "8.29 μs", + "memory": "27.75 KB" + } + ], + "comparison": [ + "lua (chunk) 343.00 K", + "luerl 267.77 K - 1.28x slower +0.82 μs", + "lua (eval) 193.98 K - 1.77x slower +2.24 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Iterate/Sum": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "4.61 K", + "average": "216.80 μs", + "deviation": "±10.43%", + "median": "212.71 μs", + "p99": "315.74 μs", + "memory": "1.47 MB" + }, + { + "name": "lua (eval)", + "ips": "4.45 K", + "average": "224.56 μs", + "deviation": "±17.51%", + "median": "216.17 μs", + "p99": "421.98 μs", + "memory": "1.48 MB" + }, + { + "name": "luerl", + "ips": "4.01 K", + "average": "249.67 μs", + "deviation": "±8.67%", + "median": "245.63 μs", + "p99": "340.05 μs", + "memory": "1.35 MB" + } + ], + "comparison": [ + "lua (chunk) 4.61 K", + "lua (eval) 4.45 K - 1.04x slower +7.76 μs", + "luerl 4.01 K - 1.15x slower +32.87 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "45.22 K", + "average": "22.12 μs", + "deviation": "±15.19%", + "median": "21.75 μs", + "p99": "31.46 μs", + "memory": "152.91 KB" + }, + { + "name": "lua (eval)", + "ips": "40.41 K", + "average": "24.74 μs", + "deviation": "±35.77%", + "median": "24.29 μs", + "p99": "34.58 μs", + "memory": "164.10 KB" + }, + { + "name": "luerl", + "ips": "37.45 K", + "average": "26.70 μs", + "deviation": "±12.27%", + "median": "26.50 μs", + "p99": "36.50 μs", + "memory": "149.13 KB" + } + ], + "comparison": [ + "lua (chunk) 45.22 K", + "lua (eval) 40.41 K - 1.12x slower +2.63 μs", + "luerl 37.45 K - 1.21x slower +4.58 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "335.95 K", + "average": "2.98 μs", + "deviation": "±201.50%", + "median": "2.88 μs", + "p99": "7 μs", + "memory": "18.55 KB" + }, + { + "name": "luerl", + "ips": "220.48 K", + "average": "4.54 μs", + "deviation": "±148.78%", + "median": "4.29 μs", + "p99": "11.21 μs", + "memory": "26.80 KB" + }, + { + "name": "lua (eval)", + "ips": "197.51 K", + "average": "5.06 μs", + "deviation": "±103.13%", + "median": "4.96 μs", + "p99": "12.13 μs", + "memory": "29.64 KB" + } + ], + "comparison": [ + "lua (chunk) 335.95 K", + "luerl 220.48 K - 1.52x slower +1.56 μs", + "lua (eval) 197.51 K - 1.70x slower +2.09 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Map + Reduce": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "2.22 K", + "average": "450.65 μs", + "deviation": "±6.37%", + "median": "443.17 μs", + "p99": "596.71 μs", + "memory": "2.91 MB" + }, + { + "name": "luerl", + "ips": "2.20 K", + "average": "454.46 μs", + "deviation": "±6.67%", + "median": "446.63 μs", + "p99": "600.22 μs", + "memory": "2.44 MB" + }, + { + "name": "lua (eval)", + "ips": "2.20 K", + "average": "455.45 μs", + "deviation": "±6.47%", + "median": "447.96 μs", + "p99": "597.34 μs", + "memory": "2.92 MB" + } + ], + "comparison": [ + "lua (chunk) 2.22 K", + "luerl 2.20 K - 1.01x slower +3.81 μs", + "lua (eval) 2.20 K - 1.01x slower +4.80 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "22.40 K", + "average": "44.64 μs", + "deviation": "±9.68%", + "median": "44.21 μs", + "p99": "55.09 μs", + "memory": "302.05 KB" + }, + { + "name": "luerl", + "ips": "21.26 K", + "average": "47.05 μs", + "deviation": "±8.41%", + "median": "46.63 μs", + "p99": "56.42 μs", + "memory": "262.49 KB" + }, + { + "name": "lua (eval)", + "ips": "20.96 K", + "average": "47.70 μs", + "deviation": "±17.39%", + "median": "46.71 μs", + "p99": "67.33 μs", + "memory": "313.15 KB" + } + ], + "comparison": [ + "lua (chunk) 22.40 K", + "luerl 21.26 K - 1.05x slower +2.41 μs", + "lua (eval) 20.96 K - 1.07x slower +3.07 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "186.34 K", + "average": "5.37 μs", + "deviation": "±89.27%", + "median": "5.21 μs", + "p99": "12.79 μs", + "memory": "35.40 KB" + }, + { + "name": "luerl", + "ips": "149.86 K", + "average": "6.67 μs", + "deviation": "±89.30%", + "median": "6.58 μs", + "p99": "13.71 μs", + "memory": "39.39 KB" + }, + { + "name": "lua (eval)", + "ips": "132.59 K", + "average": "7.54 μs", + "deviation": "±81.79%", + "median": "7.42 μs", + "p99": "15 μs", + "memory": "46.17 KB" + } + ], + "comparison": [ + "lua (chunk) 186.34 K", + "luerl 149.86 K - 1.24x slower +1.31 μs", + "lua (eval) 132.59 K - 1.41x slower +2.18 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + }, + "Table Pairs (hash)": { + "by_input": { + "large (n=1000)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "1.02 K", + "average": "982.50 μs", + "deviation": "±9.43%", + "median": "999.49 μs", + "p99": "1154.53 μs", + "memory": "2.06 MB" + }, + { + "name": "lua (chunk)", + "ips": "1.00 K", + "average": "998.42 μs", + "deviation": "±13.14%", + "median": "1001.12 μs", + "p99": "1477.94 μs", + "memory": "2.05 MB" + }, + { + "name": "luerl", + "ips": "0.74 K", + "average": "1347.55 μs", + "deviation": "±4.60%", + "median": "1349.77 μs", + "p99": "1465.70 μs", + "memory": "2.48 MB" + } + ], + "comparison": [ + "lua (eval) 1.02 K", + "lua (chunk) 1.00 K - 1.02x slower +15.92 μs", + "luerl 0.74 K - 1.37x slower +365.05 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "medium (n=100)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "13.32 K", + "average": "75.07 μs", + "deviation": "±9.83%", + "median": "74.46 μs", + "p99": "96.75 μs", + "memory": "209.77 KB" + }, + { + "name": "lua (eval)", + "ips": "12.97 K", + "average": "77.10 μs", + "deviation": "±11.95%", + "median": "75.33 μs", + "p99": "106.04 μs", + "memory": "221.48 KB" + }, + { + "name": "luerl", + "ips": "11.37 K", + "average": "87.93 μs", + "deviation": "±10.51%", + "median": "86.75 μs", + "p99": "123.60 μs", + "memory": "248.85 KB" + } + ], + "comparison": [ + "lua (chunk) 13.32 K", + "lua (eval) 12.97 K - 1.03x slower +2.04 μs", + "luerl 11.37 K - 1.17x slower +12.86 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "small (n=10)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "134.58 K", + "average": "7.43 μs", + "deviation": "±85.96%", + "median": "7.13 μs", + "p99": "18.33 μs", + "memory": "24.09 KB" + }, + { + "name": "luerl", + "ips": "128.74 K", + "average": "7.77 μs", + "deviation": "±71.32%", + "median": "7.38 μs", + "p99": "23.96 μs", + "memory": "36.16 KB" + }, + { + "name": "lua (eval)", + "ips": "98.68 K", + "average": "10.13 μs", + "deviation": "±86.13%", + "median": "9.54 μs", + "p99": "24.92 μs", + "memory": "34.87 KB" + } + ], + "comparison": [ + "lua (chunk) 134.58 K", + "luerl 128.74 K - 1.05x slower +0.34 μs", + "lua (eval) 98.68 K - 1.36x slower +2.70 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + } + } + }, + "patterns": { + "patterns: find/match field extraction (n=200)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "998.95", + "average": "1.00 ms", + "deviation": "±3.02%", + "median": "1.00 ms", + "p99": "1.06 ms", + "memory": "3.21 MB" + }, + { + "name": "lua (eval)", + "ips": "986.14", + "average": "1.01 ms", + "deviation": "±10.37%", + "median": "1.00 ms", + "p99": "1.36 ms", + "memory": "3.22 MB" + }, + { + "name": "luerl", + "ips": "923.97", + "average": "1.08 ms", + "deviation": "±3.61%", + "median": "1.08 ms", + "p99": "1.18 ms", + "memory": "6.70 MB" + } + ], + "comparison": [ + "lua (chunk) 998.95", + "lua (eval) 986.14 - 1.01x slower +0.0130 ms", + "luerl 923.97 - 1.08x slower +0.0812 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "patterns: find-based tokenizer (n=200)": { + "jobs": [ + { + "name": "luerl", + "ips": "729.19", + "average": "1.37 ms", + "deviation": "±3.06%", + "median": "1.36 ms", + "p99": "1.48 ms", + "memory": "6.65 MB" + }, + { + "name": "lua (chunk)", + "ips": "673.77", + "average": "1.48 ms", + "deviation": "±3.57%", + "median": "1.46 ms", + "p99": "1.61 ms", + "memory": "9.10 MB" + }, + { + "name": "lua (eval)", + "ips": "661.88", + "average": "1.51 ms", + "deviation": "±4.44%", + "median": "1.49 ms", + "p99": "1.76 ms", + "memory": "9.11 MB" + } + ], + "comparison": [ + "luerl 729.19", + "lua (chunk) 673.77 - 1.08x slower +0.113 ms", + "lua (eval) 661.88 - 1.10x slower +0.139 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "patterns: gsub template substitution (n=200)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "513.54", + "average": "1.95 ms", + "deviation": "±2.83%", + "median": "1.95 ms", + "p99": "2.06 ms", + "memory": "4.50 MB" + }, + { + "name": "lua (chunk)", + "ips": "480.12", + "average": "2.08 ms", + "deviation": "±16.12%", + "median": "1.98 ms", + "p99": "3.12 ms", + "memory": "4.49 MB" + }, + { + "name": "luerl", + "ips": "418.97", + "average": "2.39 ms", + "deviation": "±17.21%", + "median": "2.20 ms", + "p99": "3.51 ms", + "memory": "11.75 MB" + } + ], + "comparison": [ + "lua (eval) 513.54", + "lua (chunk) 480.12 - 1.07x slower +0.136 ms", + "luerl 418.97 - 1.23x slower +0.44 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "metamethods": { + "metamethods: self-call method dispatch (n=200)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "2.10 K", + "average": "476.79 μs", + "deviation": "±13.40%", + "median": "462.04 μs", + "p99": "730.43 μs", + "memory": "1.37 MB" + }, + { + "name": "luerl", + "ips": "2.05 K", + "average": "487.93 μs", + "deviation": "±8.87%", + "median": "474.63 μs", + "p99": "657.89 μs", + "memory": "1.43 MB" + }, + { + "name": "lua (chunk)", + "ips": "2.04 K", + "average": "490.34 μs", + "deviation": "±7.16%", + "median": "485.83 μs", + "p99": "642.74 μs", + "memory": "1.36 MB" + } + ], + "comparison": [ + "lua (eval) 2.10 K", + "luerl 2.05 K - 1.02x slower +11.14 μs", + "lua (chunk) 2.04 K - 1.03x slower +13.55 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "metamethods: 3-level __index chain (n=200)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "5.24 K", + "average": "190.75 μs", + "deviation": "±4.32%", + "median": "190.08 μs", + "p99": "206.47 μs", + "memory": "576.76 KB" + }, + { + "name": "lua (eval)", + "ips": "5.09 K", + "average": "196.41 μs", + "deviation": "±6.91%", + "median": "194.58 μs", + "p99": "221.08 μs", + "memory": "587.70 KB" + }, + { + "name": "luerl", + "ips": "4.60 K", + "average": "217.23 μs", + "deviation": "±4.39%", + "median": "216 μs", + "p99": "233.25 μs", + "memory": "668.59 KB" + } + ], + "comparison": [ + "lua (chunk) 5.24 K", + "lua (eval) 5.09 K - 1.03x slower +5.65 μs", + "luerl 4.60 K - 1.14x slower +26.48 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "metamethods: arithmetic/relational metamethods (n=200)": { + "jobs": [ + { + "name": "lua (eval)", + "ips": "533.57", + "average": "1.87 ms", + "deviation": "±5.87%", + "median": "1.92 ms", + "p99": "2.18 ms", + "memory": "5.71 MB" + }, + { + "name": "luerl", + "ips": "528.86", + "average": "1.89 ms", + "deviation": "±5.87%", + "median": "1.85 ms", + "p99": "2.20 ms", + "memory": "6.24 MB" + }, + { + "name": "lua (chunk)", + "ips": "521.21", + "average": "1.92 ms", + "deviation": "±12.56%", + "median": "1.93 ms", + "p99": "3.10 ms", + "memory": "5.70 MB" + } + ], + "comparison": [ + "lua (eval) 533.57", + "luerl 528.86 - 1.01x slower +0.0167 ms", + "lua (chunk) 521.21 - 1.02x slower +0.0444 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "pcall_varargs": { + "call protocol: pcall, success path (n=500)": { + "jobs": [ + { + "name": "lua (chunk)", + "ips": "4.47 K", + "average": "223.70 μs", + "deviation": "±4.04%", + "median": "222.88 μs", + "p99": "241.97 μs", + "memory": "895.93 KB" + }, + { + "name": "lua (eval)", + "ips": "4.44 K", + "average": "225.34 μs", + "deviation": "±6.09%", + "median": "224.42 μs", + "p99": "243.03 μs", + "memory": "911.78 KB" + }, + { + "name": "luerl", + "ips": "3.40 K", + "average": "294.17 μs", + "deviation": "±6.42%", + "median": "291.54 μs", + "p99": "317.97 μs", + "memory": "1137.10 KB" + } + ], + "comparison": [ + "lua (chunk) 4.47 K", + "lua (eval) 4.44 K - 1.01x slower +1.63 μs", + "luerl 3.40 K - 1.31x slower +70.46 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "call protocol: pcall, raise + catch (n=500)": { + "jobs": [ + { + "name": "luerl", + "ips": "2.12 K", + "average": "471.09 μs", + "deviation": "±3.42%", + "median": "468.42 μs", + "p99": "503.21 μs", + "memory": "1.55 MB" + }, + { + "name": "lua (chunk)", + "ips": "1.36 K", + "average": "733.85 μs", + "deviation": "±2.60%", + "median": "727.13 μs", + "p99": "781.15 μs", + "memory": "2.67 MB" + }, + { + "name": "lua (eval)", + "ips": "1.35 K", + "average": "742.90 μs", + "deviation": "±5.35%", + "median": "734.58 μs", + "p99": "824.44 μs", + "memory": "2.68 MB" + } + ], + "comparison": [ + "luerl 2.12 K", + "lua (chunk) 1.36 K - 1.56x slower +262.76 μs", + "lua (eval) 1.35 K - 1.58x slower +271.81 μs" + ], + "memory_note": "**All measurements for memory usage were the same**" + }, + "call protocol: varargs + multiple returns (n=500)": { + "jobs": [ + { + "name": "luerl", + "ips": "409.82", + "average": "2.44 ms", + "deviation": "±4.21%", + "median": "2.41 ms", + "p99": "2.74 ms", + "memory": "8.73 MB" + }, + { + "name": "lua (eval)", + "ips": "306.73", + "average": "3.26 ms", + "deviation": "±1.81%", + "median": "3.27 ms", + "p99": "3.44 ms", + "memory": "16.71 MB" + }, + { + "name": "lua (chunk)", + "ips": "300.00", + "average": "3.33 ms", + "deviation": "±10.97%", + "median": "3.28 ms", + "p99": "5.01 ms", + "memory": "16.70 MB" + } + ], + "comparison": [ + "luerl 409.82", + "lua (eval) 306.73 - 1.34x slower +0.82 ms", + "lua (chunk) 300.00 - 1.37x slower +0.89 ms" + ], + "memory_note": "**All measurements for memory usage were the same**" + } + }, + "vm_new": { + "VM instantiation: Lua.new/1 vs :luerl.init/0": { + "jobs": [ + { + "name": "lua (new, no sandbox)", + "ips": "1.90 M", + "average": "0.53 μs", + "deviation": "±805.41%", + "median": "0.50 μs", + "p99": "0.63 μs", + "memory": "0.95 KB" + }, + { + "name": "lua (new)", + "ips": "1.59 M", + "average": "0.63 μs", + "deviation": "±1209.11%", + "median": "0.58 μs", + "p99": "0.75 μs", + "memory": "0.88 KB" + }, + { + "name": "lua (new, custom exclude)", + "ips": "0.151 M", + "average": "6.63 μs", + "deviation": "±88.13%", + "median": "6.54 μs", + "p99": "8.79 μs", + "memory": "22.57 KB" + }, + { + "name": "luerl (init)", + "ips": "0.0656 M", + "average": "15.24 μs", + "deviation": "±36.68%", + "median": "15.13 μs", + "p99": "22.42 μs", + "memory": "51.64 KB" + } + ], + "comparison": [ + "lua (new, no sandbox) 1.90 M", + "lua (new) 1.59 M - 1.20x slower +0.104 μs", + "lua (new, custom exclude) 0.151 M - 12.61x slower +6.10 μs", + "luerl (init) 0.0656 M - 29.00x slower +14.71 μs" + ], + "memory_note": "**All measurements for memory usage were the same**", + "cold_call": "6892.0 us", + "second_call": "1.0 us" + } + }, + "encode_decode": { + "raw": "lua 1.0.2 — encode!/decode! decomposition\n(decode+deep_cast column: enabled)\n==================================================================================\nop shape N total_us per_elem_ns\nencode int_list 8 0.81 101.5\ndecode int_list 8 0.29 35.9\ndec+cast int_list 8 0.33 40.9\nencode int_list 64 14.59 228.0\ndecode int_list 64 4.94 77.1\ndec+cast int_list 64 7.11 111.0\nencode int_list 512 158.65 309.9\ndecode int_list 512 47.04 91.9\ndec+cast int_list 512 72.78 142.1\nencode int_list 4096 1568.66 383.0\ndecode int_list 4096 565.55 138.1\ndec+cast int_list 4096 892.14 217.8\n----------------------------------------------------------------------------------\nencode float_list 8 0.78 98.0\ndecode float_list 8 0.25 31.8\ndec+cast float_list 8 0.27 33.6\nencode float_list 64 11.77 183.9\ndecode float_list 64 3.76 58.7\ndec+cast float_list 64 6.99 109.2\nencode float_list 512 162.64 317.6\ndecode float_list 512 47.30 92.4\ndec+cast float_list 512 75.86 148.2\nencode float_list 4096 1614.23 394.1\ndecode float_list 4096 547.77 133.7\ndec+cast float_list 4096 805.73 196.7\n----------------------------------------------------------------------------------\nencode bool_list 8 0.79 99.1\ndecode bool_list 8 0.24 29.9\ndec+cast bool_list 8 0.27 33.9\nencode bool_list 64 14.65 228.8\ndecode bool_list 64 4.69 73.3\ndec+cast bool_list 64 6.88 107.4\nencode bool_list 512 158.50 309.6\ndecode bool_list 512 45.36 88.6\ndec+cast bool_list 512 71.69 140.0\nencode bool_list 4096 1572.86 384.0\ndecode bool_list 4096 581.14 141.9\ndec+cast bool_list 4096 792.78 193.6\n----------------------------------------------------------------------------------\nencode short_string_list 8 0.78 97.8\ndecode short_string_list 8 0.24 30.0\ndec+cast short_string_list 8 0.27 34.2\nencode short_string_list 64 11.49 179.6\ndecode short_string_list 64 3.51 54.9\ndec+cast short_string_list 64 6.93 108.3\nencode short_string_list 512 162.32 317.0\ndecode short_string_list 512 48.13 94.0\ndec+cast short_string_list 512 74.26 145.0\nencode short_string_list 4096 1664.81 406.4\ndecode short_string_list 4096 550.34 134.4\ndec+cast short_string_list 4096 810.06 197.8\n----------------------------------------------------------------------------------\nencode long_string_list 8 0.79 98.9\ndecode long_string_list 8 0.25 31.1\ndec+cast long_string_list 8 0.26 33.0\nencode long_string_list 64 11.67 182.4\ndecode long_string_list 64 3.66 57.3\ndec+cast long_string_list 64 6.75 105.5\nencode long_string_list 512 169.63 331.3\ndecode long_string_list 512 47.95 93.7\ndec+cast long_string_list 512 75.67 147.8\nencode long_string_list 4096 2667.06 651.1\ndecode long_string_list 4096 1339.42 327.0\ndec+cast long_string_list 4096 1705.52 416.4\n----------------------------------------------------------------------------------\nencode string_map 8 0.72 89.9\ndecode string_map 8 0.11 13.2\ndec+cast string_map 8 0.21 26.1\nencode string_map 64 12.18 190.3\ndecode string_map 64 0.74 11.6\ndec+cast string_map 64 3.58 56.0\nencode string_map 512 103.35 201.9\ndecode string_map 512 9.32 18.2\ndec+cast string_map 512 39.57 77.3\nencode string_map 4096 1046.22 255.4\ndecode string_map 4096 79.30 19.4\ndec+cast string_map 4096 343.00 83.7\n----------------------------------------------------------------------------------\nencode int_map 8 0.81 100.9\ndecode int_map 8 0.25 31.6\ndec+cast int_map 8 0.27 33.9\nencode int_map 64 11.36 177.6\ndecode int_map 64 3.45 54.0\ndec+cast int_map 64 5.64 88.1\nencode int_map 512 111.00 216.8\ndecode int_map 512 32.55 63.6\ndec+cast int_map 512 72.18 141.0\nencode int_map 4096 1503.89 367.2\ndecode int_map 4096 599.78 146.4\ndec+cast int_map 4096 850.84 207.7\n----------------------------------------------------------------------------------\nencode record_list 8 3.10 387.3\ndecode record_list 8 0.89 111.3\ndec+cast record_list 8 1.60 199.4\nencode record_list 64 45.68 713.8\ndecode record_list 64 12.94 202.3\ndec+cast record_list 64 18.97 296.4\nencode record_list 512 422.71 825.6\ndecode record_list 512 115.71 226.0\ndec+cast record_list 512 154.32 301.4\nencode record_list 4096 3814.88 931.4\ndecode record_list 4096 1139.67 278.2\ndec+cast record_list 4096 1656.06 404.3\n----------------------------------------------------------------------------------\nnested chain (depth sweep) — isolates recursion/traversal from fan-out\nop shape N total_us per_elem_ns\nencode nested_chain 4 0.68 338.0\ndecode nested_chain 4 0.24 122.2\ndec+cast nested_chain 4 0.36 179.7\nencode nested_chain 16 2.77 1387.2\ndecode nested_chain 16 1.11 552.9\ndec+cast nested_chain 16 1.54 768.1\nencode nested_chain 64 10.94 5468.5\ndecode nested_chain 64 6.00 3002.0\ndec+cast nested_chain 64 8.37 4183.6\nencode nested_chain 256 44.55 22273.9\ndecode nested_chain 256 28.00 13998.0\ndec+cast nested_chain 256 37.24 18618.2\n==================================================================================\ncomposite anchor — the PR's `original_nested` (matches the 18us/108us figure)\nop shape N total_us per_elem_ns\nencode original_nested 75 12.26 3065.6\ndecode original_nested 75 3.29 821.3\ndec+cast original_nested 75 5.66 1413.8\n" + } +} diff --git a/bench_results/v1.0.2/table_ops.txt b/bench_results/v1.0.2/table_ops.txt new file mode 100644 index 00000000..d6f11809 --- /dev/null +++ b/bench_results/v1.0.2/table_ops.txt @@ -0,0 +1,461 @@ +luaport not available ({:luaport, {~c"no such file or directory", ~c"luaport.app"}}) — skipping C Lua benchmarks + +=== Table Build (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 6.74 K 148.28 μs ±10.00% 145.92 μs 180.79 μs +lua (eval) 6.55 K 152.58 μs ±18.06% 148.88 μs 259.21 μs +luerl 6.48 K 154.42 μs ±9.20% 152.71 μs 189.18 μs + +Comparison: +lua (chunk) 6.74 K +lua (eval) 6.55 K - 1.03x slower +4.30 μs +luerl 6.48 K - 1.04x slower +6.15 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1023.05 KB +lua (eval) 1033.39 KB - 1.01x memory usage +10.34 KB +luerl 996.96 KB - 0.97x memory usage -26.09375 KB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 65.36 K 15.30 μs ±46.45% 15 μs 24.63 μs +luerl 58.37 K 17.13 μs ±38.71% 16.63 μs 27.96 μs +lua (eval) 56.97 K 17.55 μs ±21.90% 17.21 μs 26.13 μs + +Comparison: +lua (chunk) 65.36 K +luerl 58.37 K - 1.12x slower +1.83 μs +lua (eval) 56.97 K - 1.15x slower +2.25 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 104.51 KB +luerl 110.78 KB - 1.06x memory usage +6.27 KB +lua (eval) 115.14 KB - 1.10x memory usage +10.63 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 473.88 K 2.11 μs ±259.11% 2.04 μs 3.21 μs +luerl 319.03 K 3.13 μs ±185.41% 3.04 μs 4.54 μs +lua (eval) 234.54 K 4.26 μs ±124.39% 4.13 μs 11.13 μs + +Comparison: +lua (chunk) 473.88 K +luerl 319.03 K - 1.49x slower +1.02 μs +lua (eval) 234.54 K - 2.02x slower +2.15 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 12.89 KB +luerl 22.62 KB - 1.75x memory usage +9.73 KB +lua (eval) 23.84 KB - 1.85x memory usage +10.95 KB + +**All measurements for memory usage were the same** + +=== Table Sort (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +luerl 5.41 K 184.80 μs ±18.74% 179.09 μs 413.57 μs +lua (chunk) 4.88 K 205.09 μs ±13.18% 199.34 μs 322.32 μs +lua (eval) 4.82 K 207.48 μs ±16.82% 202.59 μs 278.13 μs + +Comparison: +luerl 5.41 K +lua (chunk) 4.88 K - 1.11x slower +20.29 μs +lua (eval) 4.82 K - 1.12x slower +22.68 μs + +Memory usage statistics: + +Name Memory usage +luerl 1.17 MB +lua (chunk) 1.22 MB - 1.04x memory usage +0.0500 MB +lua (eval) 1.24 MB - 1.05x memory usage +0.0602 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +luerl 51.19 K 19.54 μs ±19.26% 19.04 μs 28.13 μs +lua (chunk) 47.57 K 21.02 μs ±75.79% 20.38 μs 44.00 μs +lua (eval) 43.10 K 23.20 μs ±19.83% 22.79 μs 28.67 μs + +Comparison: +luerl 51.19 K +lua (chunk) 47.57 K - 1.08x slower +1.49 μs +lua (eval) 43.10 K - 1.19x slower +3.67 μs + +Memory usage statistics: + +Name Memory usage +luerl 133.16 KB +lua (chunk) 129.16 KB - 0.97x memory usage -3.99219 KB +lua (eval) 140.11 KB - 1.05x memory usage +6.95 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 343.00 K 2.92 μs ±179.73% 2.88 μs 4.17 μs +luerl 267.77 K 3.73 μs ±158.30% 3.63 μs 5.67 μs +lua (eval) 193.98 K 5.16 μs ±94.77% 5.04 μs 8.29 μs + +Comparison: +lua (chunk) 343.00 K +luerl 267.77 K - 1.28x slower +0.82 μs +lua (eval) 193.98 K - 1.77x slower +2.24 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 16.81 KB +luerl 25.98 KB - 1.55x memory usage +9.16 KB +lua (eval) 27.75 KB - 1.65x memory usage +10.94 KB + +**All measurements for memory usage were the same** + +=== Table Iterate/Sum (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 4.61 K 216.80 μs ±10.43% 212.71 μs 315.74 μs +lua (eval) 4.45 K 224.56 μs ±17.51% 216.17 μs 421.98 μs +luerl 4.01 K 249.67 μs ±8.67% 245.63 μs 340.05 μs + +Comparison: +lua (chunk) 4.61 K +lua (eval) 4.45 K - 1.04x slower +7.76 μs +luerl 4.01 K - 1.15x slower +32.87 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 1.47 MB +lua (eval) 1.48 MB - 1.01x memory usage +0.0119 MB +luerl 1.35 MB - 0.91x memory usage -0.12543 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 45.22 K 22.12 μs ±15.19% 21.75 μs 31.46 μs +lua (eval) 40.41 K 24.74 μs ±35.77% 24.29 μs 34.58 μs +luerl 37.45 K 26.70 μs ±12.27% 26.50 μs 36.50 μs + +Comparison: +lua (chunk) 45.22 K +lua (eval) 40.41 K - 1.12x slower +2.63 μs +luerl 37.45 K - 1.21x slower +4.58 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 152.91 KB +lua (eval) 164.10 KB - 1.07x memory usage +11.20 KB +luerl 149.13 KB - 0.98x memory usage -3.78125 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 335.95 K 2.98 μs ±201.50% 2.88 μs 7 μs +luerl 220.48 K 4.54 μs ±148.78% 4.29 μs 11.21 μs +lua (eval) 197.51 K 5.06 μs ±103.13% 4.96 μs 12.13 μs + +Comparison: +lua (chunk) 335.95 K +luerl 220.48 K - 1.52x slower +1.56 μs +lua (eval) 197.51 K - 1.70x slower +2.09 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 18.55 KB +luerl 26.80 KB - 1.44x memory usage +8.25 KB +lua (eval) 29.64 KB - 1.60x memory usage +11.09 KB + +**All measurements for memory usage were the same** + +=== Table Map + Reduce (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (chunk) 2.22 K 450.65 μs ±6.37% 443.17 μs 596.71 μs +luerl 2.20 K 454.46 μs ±6.67% 446.63 μs 600.22 μs +lua (eval) 2.20 K 455.45 μs ±6.47% 447.96 μs 597.34 μs + +Comparison: +lua (chunk) 2.22 K +luerl 2.20 K - 1.01x slower +3.81 μs +lua (eval) 2.20 K - 1.01x slower +4.80 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 2.91 MB +luerl 2.44 MB - 0.84x memory usage -0.46843 MB +lua (eval) 2.92 MB - 1.00x memory usage +0.0121 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 22.40 K 44.64 μs ±9.68% 44.21 μs 55.09 μs +luerl 21.26 K 47.05 μs ±8.41% 46.63 μs 56.42 μs +lua (eval) 20.96 K 47.70 μs ±17.39% 46.71 μs 67.33 μs + +Comparison: +lua (chunk) 22.40 K +luerl 21.26 K - 1.05x slower +2.41 μs +lua (eval) 20.96 K - 1.07x slower +3.07 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 302.05 KB +luerl 262.49 KB - 0.87x memory usage -39.55469 KB +lua (eval) 313.15 KB - 1.04x memory usage +11.10 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 186.34 K 5.37 μs ±89.27% 5.21 μs 12.79 μs +luerl 149.86 K 6.67 μs ±89.30% 6.58 μs 13.71 μs +lua (eval) 132.59 K 7.54 μs ±81.79% 7.42 μs 15 μs + +Comparison: +lua (chunk) 186.34 K +luerl 149.86 K - 1.24x slower +1.31 μs +lua (eval) 132.59 K - 1.41x slower +2.18 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 35.40 KB +luerl 39.39 KB - 1.11x memory usage +3.99 KB +lua (eval) 46.17 KB - 1.30x memory usage +10.77 KB + +**All measurements for memory usage were the same** + +=== Table Pairs (hash) (mode: full) === + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: large (n=1000), medium (n=100), small (n=10) +Estimated total run time: 1 min 57 s +Excluding outliers: false + +Benchmarking lua (chunk) with input large (n=1000) ... +Benchmarking lua (chunk) with input medium (n=100) ... +Benchmarking lua (chunk) with input small (n=10) ... +Benchmarking lua (eval) with input large (n=1000) ... +Benchmarking lua (eval) with input medium (n=100) ... +Benchmarking lua (eval) with input small (n=10) ... +Benchmarking luerl with input large (n=1000) ... +Benchmarking luerl with input medium (n=100) ... +Benchmarking luerl with input small (n=10) ... +Calculating statistics... +Formatting results... + +##### With input large (n=1000) ##### +Name ips average deviation median 99th % +lua (eval) 1.02 K 982.50 μs ±9.43% 999.49 μs 1154.53 μs +lua (chunk) 1.00 K 998.42 μs ±13.14% 1001.12 μs 1477.94 μs +luerl 0.74 K 1347.55 μs ±4.60% 1349.77 μs 1465.70 μs + +Comparison: +lua (eval) 1.02 K +lua (chunk) 1.00 K - 1.02x slower +15.92 μs +luerl 0.74 K - 1.37x slower +365.05 μs + +Memory usage statistics: + +Name Memory usage +lua (eval) 2.06 MB +lua (chunk) 2.05 MB - 0.99x memory usage -0.01048 MB +luerl 2.48 MB - 1.20x memory usage +0.42 MB + +**All measurements for memory usage were the same** + +##### With input medium (n=100) ##### +Name ips average deviation median 99th % +lua (chunk) 13.32 K 75.07 μs ±9.83% 74.46 μs 96.75 μs +lua (eval) 12.97 K 77.10 μs ±11.95% 75.33 μs 106.04 μs +luerl 11.37 K 87.93 μs ±10.51% 86.75 μs 123.60 μs + +Comparison: +lua (chunk) 13.32 K +lua (eval) 12.97 K - 1.03x slower +2.04 μs +luerl 11.37 K - 1.17x slower +12.86 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 209.77 KB +lua (eval) 221.48 KB - 1.06x memory usage +11.72 KB +luerl 248.85 KB - 1.19x memory usage +39.09 KB + +**All measurements for memory usage were the same** + +##### With input small (n=10) ##### +Name ips average deviation median 99th % +lua (chunk) 134.58 K 7.43 μs ±85.96% 7.13 μs 18.33 μs +luerl 128.74 K 7.77 μs ±71.32% 7.38 μs 23.96 μs +lua (eval) 98.68 K 10.13 μs ±86.13% 9.54 μs 24.92 μs + +Comparison: +lua (chunk) 134.58 K +luerl 128.74 K - 1.05x slower +0.34 μs +lua (eval) 98.68 K - 1.36x slower +2.70 μs + +Memory usage statistics: + +Name Memory usage +lua (chunk) 24.09 KB +luerl 36.16 KB - 1.50x memory usage +12.06 KB +lua (eval) 34.87 KB - 1.45x memory usage +10.77 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/v1.0.2/timestamp.txt b/bench_results/v1.0.2/timestamp.txt new file mode 100644 index 00000000..2a012924 --- /dev/null +++ b/bench_results/v1.0.2/timestamp.txt @@ -0,0 +1 @@ +Tue Jul 28 11:14:31 EDT 2026 diff --git a/bench_results/v1.0.2/versions.txt b/bench_results/v1.0.2/versions.txt new file mode 100644 index 00000000..184e5f9a --- /dev/null +++ b/bench_results/v1.0.2/versions.txt @@ -0,0 +1,3 @@ +Erlang/OTP 29 [erts-17.0] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] + +Elixir 1.20.0 (compiled with Erlang/OTP 29) diff --git a/bench_results/v1.0.2/vm_new.txt b/bench_results/v1.0.2/vm_new.txt new file mode 100644 index 00000000..fb12a496 --- /dev/null +++ b/bench_results/v1.0.2/vm_new.txt @@ -0,0 +1,56 @@ +=== VM instantiation: Lua.new/1 vs :luerl.init/0 (mode: full) === + +Lua.new() one-time vs repeat cost (single samples, informational): + first call on this node : 6892.0 us + second call : 1.0 us + +The first figure includes any one-time template build and first-time module +loading. Benchee's steady-state numbers below are the per-request cost after +that point. + +Operating System: macOS +CPU Information: Apple M4 +Number of Available Cores: 10 +Available memory: 32 GB +Elixir 1.20.0 +Erlang 29.0 +JIT enabled: true + +Benchmark suite executing with the following configuration: +warmup: 2 s +time: 10 s +memory time: 1 s +reduction time: 0 ns +parallel: 1 +inputs: none specified +Estimated total run time: 52 s +Excluding outliers: false + +Benchmarking lua (new) ... +Benchmarking lua (new, custom exclude) ... +Benchmarking lua (new, no sandbox) ... +Benchmarking luerl (init) ... +Calculating statistics... +Formatting results... + +Name ips average deviation median 99th % +lua (new, no sandbox) 1.90 M 0.53 μs ±805.41% 0.50 μs 0.63 μs +lua (new) 1.59 M 0.63 μs ±1209.11% 0.58 μs 0.75 μs +lua (new, custom exclude) 0.151 M 6.63 μs ±88.13% 6.54 μs 8.79 μs +luerl (init) 0.0656 M 15.24 μs ±36.68% 15.13 μs 22.42 μs + +Comparison: +lua (new, no sandbox) 1.90 M +lua (new) 1.59 M - 1.20x slower +0.104 μs +lua (new, custom exclude) 0.151 M - 12.61x slower +6.10 μs +luerl (init) 0.0656 M - 29.00x slower +14.71 μs + +Memory usage statistics: + +Name Memory usage +lua (new, no sandbox) 0.95 KB +lua (new) 0.88 KB - 0.92x memory usage -0.07813 KB +lua (new, custom exclude) 22.57 KB - 23.68x memory usage +21.62 KB +luerl (init) 51.64 KB - 54.18x memory usage +50.69 KB + +**All measurements for memory usage were the same** diff --git a/bench_results/versions-2026-07-28.md b/bench_results/versions-2026-07-28.md new file mode 100644 index 00000000..e04a1b66 --- /dev/null +++ b/bench_results/versions-2026-07-28.md @@ -0,0 +1,574 @@ +# Cross-version benchmarks — v0.4.0 vs v1.0.0 vs 1.0.2 + +Recorded 2026-07-28 for the comparative perf run tracked in +[#267](https://github.com/tv-labs/lua/issues/267). + +This report **supersedes the 1.0.0-era figures in +[`benchmarks/BASELINE.md`](../benchmarks/BASELINE.md)**. That file remains in +the tree as the historical record of the 1.0.0 perf gate; every number in it +predates the call-convention, peephole and bootstrap work that landed after +1.0.0, and several of its conclusions no longer hold (notably its "every +workload is within 25% of Luerl" gate statement — see +[Where we are still behind](#where-we-are-still-behind)). Use this document, +not `BASELINE.md`, for current numbers. + +## What is being compared + +Three points in this library's history, each measured in its own process, on +the same machine, in the same sitting: + +| Column | Ref | What it is | +|---|---|---| +| **v0.4.0** | tag `v0.4.0` | The last release before the native VM. `Lua` was a thin Elixir wrapper over [Luerl](https://github.com/rvirding/luerl) 1.5.1 — parsing, evaluation and the whole standard library were Luerl's. | +| **v1.0.0** | tag `v1.0.0`, commit `69e13a6` | The first release running this project's own Lua 5.3 VM: own lexer, parser, compiler, bytecode dispatcher and standard library. No Luerl on the execution path. | +| **1.0.2** | `main`, commit `3a0d392` | Current release. Adds the peephole/fused-opcode pass ([#403](https://github.com/tv-labs/lua/pull/403)), the one-allocation register file and static-arity call convention ([#405](https://github.com/tv-labs/lua/pull/405)), and the memoized boot template ([#398](https://github.com/tv-labs/lua/pull/398)). | + +### Why Luerl appears in every table + +Every benchmark script runs a Luerl 1.5.1 job **in the same process, in the +same Benchee run, against the same Lua source** as the `lua` jobs. Luerl is +therefore not a competitor here so much as a **control**: it is the one thing +held constant across all three refs, so it absorbs machine drift, thermal +state, and OTP-level variation. When a ratio moves between columns, the Luerl +control tells you whether the library moved or the machine did. + +The control held very steady. Across the three independent runs, Luerl's +`:luerl.init()` median was 15.04 µs / 15.21 µs / 15.13 µs, and its +`string.format` many-specifier median was 2.80 ms / 2.82 ms / 2.78 ms — under +1% spread. Where the control *did* drift, +it is called out inline. + +Ratios below are always **`lua (chunk)` median ÷ same-run `luerl` median**. +Lower is better; **below 1.00 means faster than Luerl in that same run**. + +### Environment + +- Apple M4 (arm64), Elixir 1.20.0, Erlang/OTP 29 [erts-17.0] [64-bit] [jit] +- `LUA_BENCH_MODE=full` (2 s warmup, 10 s measurement, memory measurement on, + and the multi-size sweep for the table workloads) +- Comparison control: Luerl `~> 1.5` (1.5.1) +- C Lua via `:luaport` was **not** available in this environment; each script's + own fallback path skipped it. There are no C Lua rows anywhere in this report. + +### Measurement discipline + +Every workload was run **serially, one `mix run` process at a time, on an +otherwise quiet machine**. This is not optional. Running these concurrently +with tests or other agents inflates deviation to the point where the table and +OOP cases swing wildly and orderings flip. All figures below are the +quiet-machine read. + +**Medians are quoted as primary throughout**, not averages. On workloads with +allocation-driven GC pauses the mean is pulled around by a small number of +long iterations; the median is the number an embedding host actually +experiences per call. Averages, p99s, deviations and memory figures are all in +the `summary.json` files if you want them. + +--- + +## Numeric code: recursion and call overhead + +`fibonacci` computes `fib(30)` recursively through a global function — roughly +2.7 million calls per iteration, each one a small integer compare, two +recursive calls and an add. It is deliberately the worst case for call +overhead: almost nothing happens between calls, so the measurement is +dominated by frame setup, argument passing and return. This is the workload +the static-arity call convention and one-allocation register file +([#405](https://github.com/tv-labs/lua/pull/405)) were built for. + +| Ref | lua (chunk) median | ips | luerl median (same run) | ratio | +|---|---|---|---|---| +| v0.4.0 | 702.84 ms | 1.42 | 720.40 ms | 0.98 | +| v1.0.0 | 792.42 ms | 1.26 | 730.11 ms | 1.09 | +| **1.0.2** | **434.12 ms** | **2.30** | 741.95 ms | **0.59** | + +1.0.0 shipped this workload 9% *slower* than the Luerl control. 1.0.2 runs it +**1.7× faster than Luerl** and 1.83× faster than 1.0.0. Allocation moved with +it: 2.90 GB per iteration on 1.0.0 → 1016.63 MB on 1.0.2, against Luerl's +2513.67 MB in the same run. + +`lua (eval)` is within 1% of `lua (chunk)` here (429.30 ms vs 434.12 ms) — at +this workload size the one-time parse is invisible next to 400 ms of +execution. + +## Closures and upvalues + +`closures` builds 100 counter closures through a factory function, each +capturing and mutating its own upvalue, then calls each one ten times. It +exercises closure allocation, upvalue capture, and mutation of a captured +local through a closure boundary — the machinery `BASELINE.md` flagged as the +weakest area at 1.0.0. + +| Ref | lua (chunk) median | ips | luerl median (same run) | ratio | +|---|---|---|---|---| +| v0.4.0 | 368.92 µs | 2.69 K | 391.42 µs | 0.94 | +| v1.0.0 | 473.54 µs | 2.09 K | 382.50 µs | 1.24 | +| **1.0.2** | **373.71 µs** | **2.63 K** | 389.21 µs | **0.96** | + +This is the clearest single vindication of the post-1.0 work: the 1.25× gap +`BASELINE.md` recorded as "at the bar" reproduces exactly (1.24× here) and is +now closed — 1.0.2 is marginally faster than the control and 1.27× faster than +1.0.0. + +## OOP via metatables + +Two workloads cover this. `oop` is the shallow, construction-dominated end: 50 +instances per iteration, each two field writes plus a `setmetatable`, then one +field-style method call whose body concatenates two fields. `metamethods` +covers dispatch: `:` self-calls (including a chained call on a freshly +constructed receiver), a three-level `__index` prototype chain resolving at +depths 1/2/3, and arithmetic/relational metamethods (`__add`, `__sub`, `__lt`, +`__eq`, `__tostring`) which route through the arithmetic opcodes' metamethod +fallback rather than through table indexing. + +| Case | Ref | lua (chunk) median | ips | luerl median | ratio | +|---|---|---|---|---|---| +| oop (construction) | v0.4.0 | 110.88 µs | 8.80 K | 106.50 µs | 1.04 | +| | v1.0.0 | 123.50 µs | 7.88 K | 116.13 µs | 1.06 | +| | **1.0.2** | **93.21 µs** | **10.47 K** | 133.50 µs | **0.70** | +| self-call dispatch | v0.4.0 | 476.58 µs | 2.04 K | 492.79 µs | 0.97 | +| | v1.0.0 | 553.75 µs | 1.75 K | 472.63 µs | 1.17 | +| | **1.0.2** | **485.83 µs** | **2.04 K** | 474.63 µs | **1.02** | +| 3-level `__index` chain | v0.4.0 | 216.42 µs | 4.57 K | 217.92 µs | 0.99 | +| | v1.0.0 | 237.13 µs | 4.19 K | 221.29 µs | 1.07 | +| | **1.0.2** | **190.08 µs** | **5.24 K** | 216 µs | **0.88** | +| arithmetic metamethods | v0.4.0 | 1.88 ms | 513.47 | 1.88 ms | 1.00 | +| | v1.0.0 | 2.23 ms | 455.21 | 1.88 ms | 1.19 | +| | **1.0.2** | **1.93 ms** | **521.21** | 1.85 ms | **1.04** | + +Prototype-chain walking is now 12% faster than the control and self-call +dispatch has essentially reached parity (1.02×) from 1.17× at 1.0.0 — the +`call_self` fusion in [#405](https://github.com/tv-labs/lua/pull/405) doing its +job. Arithmetic metamethods improved from 1.19× to 1.04× but have not quite +closed. + +**Control drift warning:** the `oop` Luerl median moved 106.50 → 116.13 → +133.50 µs across the three runs (deviations ±13.7% to ±21.4%), a 25% spread on +the one job that should be identical. `oop`'s ratio column is the least +trustworthy in this report; the raw `lua (chunk)` medians (110.88 → 123.50 → +93.21 µs) tell the story more reliably than the ratios do. + +## Strings: building and formatting + +`string_ops` covers string building: `table.concat` over 100 parts, and +`string.format("item_%d=%f")` in a 100-iteration loop. `string_format` pushes +the formatter on three separate axes at n=1000: a long literal-heavy format +string (~430 characters of literal text around three specifiers, so the cost +is copying literal bytes), many width-flagged specifiers (`%-20s`, `%8d`, +`%12.4f`, `%6x` — the padding path on every conversion), and a dozen +specifiers interleaved with short literals (the conversion-heavy counterpart). + +| Case | Ref | lua (chunk) median | ips | luerl median | ratio | +|---|---|---|---|---|---| +| `table.concat` (n=100) | v0.4.0 | 38.38 µs | 25.94 K | 39.42 µs | 0.97 | +| | v1.0.0 | 35.67 µs | 27.97 K | 40.17 µs | 0.89 | +| | **1.0.2** | **31.21 µs** | **31.46 K** | 40.17 µs | **0.78** | +| `string.format` loop (n=100) | v0.4.0 | 101.46 µs | 9.75 K | 102.83 µs | 0.99 | +| | v1.0.0 | 80.21 µs | 12.44 K | 103.38 µs | 0.78 | +| | **1.0.2** | **72.21 µs** | **14.08 K** | 103.04 µs | **0.70** | +| format: long literal-heavy | v0.4.0 | 3.91 ms | 249.24 | 4.25 ms | 0.92 | +| | v1.0.0 | 1.03 ms | 966.41 | 4.32 ms | 0.24 | +| | **1.0.2** | **716.04 µs** | **1.38 K** | 3921.50 µs | **0.18** | +| format: width-flagged | v0.4.0 | 1.72 ms | 582.97 | 1.73 ms | 0.99 | +| | v1.0.0 | 1.50 ms | 662.60 | 1.79 ms | 0.84 | +| | **1.0.2** | **1.22 ms** | **812.70** | 1.74 ms | **0.70** | +| format: many specifiers | v0.4.0 | 2.75 ms | 367.47 | 2.80 ms | 0.98 | +| | v1.0.0 | 2.35 ms | 413.90 | 2.82 ms | 0.83 | +| | **1.0.2** | **1.77 ms** | **566.54** | 2.78 ms | **0.64** | + +`string.format` is this library's strongest area. The literal-heavy case runs +**5.5× faster than Luerl** (716.04 µs vs 3921.50 µs) and allocates 2.03 MB +against Luerl's 22.59 MB — an 11× allocation reduction — thanks to the +bare-specifier fast path, the exact bignum fixed-precision float formatter, and +the parsed-template cache. Every string case improved monotonically across all +three versions. + +## Tables: array and hash parts + +`table_ops` covers five operations, swept over n=10/100/1000 in full mode: +building an array of squares, sorting a reverse-ordered array (worst case), +summing by index, a two-pass map-then-reduce, and a full `pairs` walk over a +**string-keyed** table, which forces every entry into the hash part and drives +the memoized hash-iteration path. + +The **n=1000** column is quoted below. The n=10 and n=100 cells are in the +JSON but carry deviations of ±20–46% at 2–25 µs, which is not enough signal to +publish a ratio from; one such cell (v0.4.0 build at n=100) reads 1.60× purely +from noise, on a ref where the `lua` and `luerl` rows are the same engine. + +| Operation (n=1000) | Ref | lua (chunk) median | ips | luerl median | ratio | +|---|---|---|---|---|---| +| build | v0.4.0 | 149.92 µs | 6.35 K | 152.33 µs | 0.98 | +| | v1.0.0 | 157.88 µs | 6.23 K | 154.21 µs | 1.02 | +| | **1.0.2** | **145.92 µs** | **6.74 K** | 152.71 µs | **0.96** | +| sort | v0.4.0 | 174.29 µs | 5.61 K | 180.59 µs | 0.97 | +| | v1.0.0 | 227.63 µs | 4.27 K | 178.00 µs | 1.28 | +| | **1.0.2** | **199.34 µs** | **4.88 K** | 179.09 µs | **1.11** | +| iterate / sum | v0.4.0 | 242.42 µs | 3.60 K | 243.13 µs | 1.00 | +| | v1.0.0 | 235.09 µs | 4.10 K | 243.06 µs | 0.97 | +| | **1.0.2** | **212.71 µs** | **4.61 K** | 245.63 µs | **0.87** | +| map + reduce | v0.4.0 | 444.34 µs | 2.19 K | 442.21 µs | 1.00 | +| | v1.0.0 | 480.24 µs | 2.05 K | 449.96 µs | 1.07 | +| | **1.0.2** | **443.17 µs** | **2.22 K** | 446.63 µs | **0.99** | +| `pairs` (hash part) | v0.4.0 | 1.32 ms | 759.91 | 1.33 ms | 0.99 | +| | v1.0.0 | 995.70 µs | 1.02 K | 1334.35 µs | 0.75 | +| | **1.0.2** | **1001.12 µs** | **1.00 K** | 1349.77 µs | **0.74** | + +Hash-part iteration is the standout: **1.35× faster than Luerl** at n=1000, +where the memoized order-index makes each `next` step O(1) instead of rescanning +an order list. `table.sort` is the one table operation still behind (1.11×, +improved from 1.28× at 1.0.0) — the array part is rebuilt with a single +`:array.from_list/2` write-back, but the comparison-driven sort itself remains +more expensive than Luerl's. + +## String patterns + +`patterns` exercises the Lua pattern engine, which had no coverage at all +before this report. Three cases: field extraction from a log line with +`string.find` and `string.match` (character classes, quantifiers, escaped magic +characters, captures); a tokenizer that re-enters `string.find(s, "[^,]+", pos)` +at an advancing init offset; and template substitution using all three `gsub` +replacement kinds — a table replacement driven by a capture, a string +replacement over `%s+`, and a function replacement invoked per word. + +The tokenizer deliberately avoids `string.gmatch`. Luerl 1.5.1 raises +`{:badarg, :gmatch, ...}` for any `gmatch` call, so a `gmatch`-based tokenizer +could not be measured against the control on identical Lua source, and keeping +the source identical across engines is this suite's fairness contract. + +| Case | Ref | lua (chunk) median | ips | luerl median | ratio | +|---|---|---|---|---|---| +| find/match extraction | v0.4.0 | 1.07 ms | 919.82 | 1.05 ms | 1.02 | +| | v1.0.0 | 1.01 ms | 972.71 | 1.09 ms | 0.93 | +| | **1.0.2** | **1.00 ms** | **998.95** | 1.08 ms | **0.93** | +| find-based tokenizer | v0.4.0 | 1.37 ms | 721.24 | 1.36 ms | 1.01 | +| | v1.0.0 | 1.57 ms | 626.48 | 1.35 ms | 1.16 | +| | **1.0.2** | **1.46 ms** | **673.77** | 1.36 ms | **1.07** | +| gsub template substitution | v0.4.0 | 2.21 ms | 436.67 | 2.21 ms | 1.00 | +| | v1.0.0 | 2.02 ms | 491.94 | 2.17 ms | 0.93 | +| | **1.0.2** | **1.98 ms** | **480.12** | 2.20 ms | **0.90** | + +Extraction and `gsub` are 7–10% faster than the control. The tokenizer is 7% +behind: it restarts the matcher once per token with a fresh start position, and +that per-restart setup is where the remaining gap sits. Allocation is mixed — +`gsub` allocates 4.49 MB against Luerl's 11.75 MB, but the tokenizer allocates +9.10 MB against Luerl's 6.65 MB. + +## Call protocol: protected calls, varargs, multiple returns + +`pcall_varargs` covers the parts of the call protocol that every host +embedding depends on and that had no coverage before this report. Three cases, +all n=500: protected calls that all succeed (the production-common case, which +isolates the cost of entering and leaving a protected frame); protected calls +that all raise a string error and are caught (error construction, unwinding, +returning `false, err`); and variadic handling — `select("#", ...)`, +positional `select(i, ...)`, tail-position `f(...)` forwarding, +`table.pack`/`table.unpack` round-tripping, and multiple-return destructuring. + +| Case | Ref | lua (chunk) median | ips | luerl median | ratio | +|---|---|---|---|---|---| +| pcall, success path | v0.4.0 | 290.50 µs | 3.40 K | 289.67 µs | 1.00 | +| | v1.0.0 | 282.13 µs | 3.49 K | 289.50 µs | 0.97 | +| | **1.0.2** | **222.88 µs** | **4.47 K** | 291.54 µs | **0.76** | +| pcall, raise + catch | v0.4.0 | 469.29 µs | 2.11 K | 466.38 µs | 1.01 | +| | v1.0.0 | 877.33 µs | 1.11 K | 485.38 µs | 1.81 | +| | **1.0.2** | **727.13 µs** | **1.36 K** | 468.42 µs | **1.55** | +| varargs + multiple returns | v0.4.0 | 2.40 ms | 391.27 | 2.36 ms | 1.02 | +| | v1.0.0 | 4.01 ms | 237.27 | 2.40 ms | 1.67 | +| | **1.0.2** | **3.28 ms** | **300.00** | 2.41 ms | **1.36** | + +The success path — the one that actually runs on every request in a host that +wraps script entry points in `pcall` — is **1.31× faster than Luerl** and 1.27× +faster than 1.0.0. The other two are the subject of +[Where we are still behind](#where-we-are-still-behind). + +## `Lua.new/1`: VM instantiation + +`vm_new` measures the cost of standing up a VM before any Lua runs. A host +that builds a fresh sandbox per request — the recommended isolation model — +pays this on every request. Three instantiation shapes are measured, chosen +because `:sandboxed` and `:exclude` are the only `new/1` options that exist +across the whole version history: + +- **`Lua.new()`** — the default deny-list sandbox. What most embedders call. +- **`Lua.new(sandboxed: [])`** — standard library installed, zero sandbox + passes. The closest like-for-like analogue of a bare `:luerl.init()`, which + also performs no sandboxing. **Compare the Luerl row against this row.** +- **`Lua.new(exclude: [[:require]])`** — the default deny-list minus one entry. + "I want the sandbox but need one thing back." + +| Shape | v0.4.0 median | v1.0.0 median | 1.0.2 median | +|---|---|---|---| +| `Lua.new()` | 21.33 µs | 36.67 µs | **0.58 µs** | +| `Lua.new(sandboxed: [])` | 14.88 µs | 29.71 µs | **0.50 µs** | +| `Lua.new(exclude: [[:require]])` | 20.63 µs | 35.96 µs | **6.54 µs** | +| `:luerl.init()` (control) | 15.04 µs | 15.21 µs | 15.13 µs | + +Allocation for `Lua.new()` fell from 91.88 KB (1.0.0) to **0.88 KB** (1.0.2) — +104× less — which is the expected signature of returning a shared memoized +template rather than rebuilding one. + +### Validating the "~100×" claim + +The 1.0.2 changelog and [#419](https://github.com/tv-labs/lua/pull/419) claim +"`Lua.new/1` ~100× faster via a memoized boot-time VM template +([#398](https://github.com/tv-labs/lua/pull/398))". The claim holds in +substance, with four qualifications that anyone quoting it should carry: + +1. **Measured 63× here, not 100×, and that is expected.** 36.67 µs → 0.58 µs + by median is **63×**; by average (37.63 µs → 0.63 µs) it is **60×**. These + runs are under `mix run`, which puts the VM in `:interactive` code-loading + mode. In that mode a cache hit re-verifies the module-reload fingerprint + that #398 describes as costing ~0.2 µs of a ~0.42 µs hit. Under `:embedded` + mode — releases — that check is skipped entirely and the ~100× figure is + credible. **Quote the mode alongside the multiplier.** +2. **It is shape-specific.** Only the exactly-default `Lua.new()` hits the + fully-sandboxed template. Pass any custom `:sandboxed` or `:exclude` and you + get the shared pre-sandbox install but still pay your own sandbox pass: + 35.96 µs → 6.54 µs, i.e. **5.5×, not 63×**. Passing `sandboxed:` is common, + so an unqualified "100× faster `Lua.new`" overstates it for a real fraction + of users. +3. **The history is non-monotonic.** v1.0.0 *regressed* instantiation relative + to v0.4.0 (36.67 µs vs 21.33 µs — 1.72× slower), because the native VM's + standard-library install cost more than Luerl's `init`. 1.0.2 does not + merely recover that; it beats v0.4.0 by **36.8×**. A two-point + "0.4.0 → 1.0.2" comparison would hide a real regression that existed in + between. +4. **There is a one-time cost, and it is not zero.** The template is written + once per node. The first `Lua.new()` on a node measured 6892 µs on 1.0.2, + the second 1.0 µs. That first figure is an **upper bound, not the template + build cost** — under `mix run` it also absorbs first-time loading of the + standard-library modules, which a release has already done at boot. (For + scale, the same cold measurement is 6654 µs on v1.0.0 and 10070 µs on + v0.4.0, i.e. the *larger* cold number belongs to the *slower* library.) Do + not publish "the first call costs 6.9 ms" as a property of the memoization. + +**Unequal work across refs.** `Lua.new()` does not sandbox the same number of +paths on every ref: v0.4.0's default deny-list has 14 entries, v1.0.0 and 1.0.2 +have 27 (the single `[:io]` entry was split into 14 per-function paths). The +`Lua.new()` row is therefore "what the public API costs on that release", not +identical work. The `sandboxed: []` row is the identical-work comparison, and +it is the one the Luerl control should be read against. + +**Deviation on the sub-microsecond rows.** 1.0.2's `Lua.new()` reports +±1209% deviation. This is Benchee measuring an operation near timer +resolution and batching to compensate, not instability — the median (0.58 µs) +and p99 (0.75 µs) are tight. Use medians for these rows and do not reproduce +the deviation column without this note. + +## Host boundary: `encode!` / `decode!` + +`encode_decode` measures the Elixir↔Lua data boundary — `Lua.encode!/2`, +`Lua.decode!/2`, and `Lua.Table.deep_cast/1` — across container shapes (integer +/ float / boolean / short-string / long-string lists, string-keyed maps, +integer-keyed maps, record-shaped maps) at N = 8/64/512/4096, plus a +nested-chain depth sweep and a composite anchor. It uses its own `:timer.tc` +harness rather than Benchee, reports per-element nanoseconds so a super-linear +curve is visible, and **has no Luerl comparator** — it measures this library's +own boundary code, which on v0.4.0 happens to be Luerl's term conversion. + +This is the one area where the native VM is **materially behind the +Luerl-backed v0.4.0**, and it should be published as such. + +| Operation | v0.4.0 | v1.0.0 | 1.0.2 | 1.0.2 vs v0.4.0 | +|---|---|---|---|---| +| `encode` int_list, N=4096 | 226.97 µs | 1601.11 µs | 1568.66 µs | **6.91× slower** | +| `decode` int_list, N=4096 | 43.47 µs | 583.70 µs | 565.55 µs | **13.01× slower** | +| `decode` long_string_list, N=4096 | 54.39 µs | 1495.20 µs | 1339.42 µs | **24.63× slower** | +| `encode` record_list, N=4096 | 2346.38 µs | 4732.25 µs | 3814.88 µs | 1.63× slower | +| `encode` string_map, N=4096 | 1295.09 µs | 2194.00 µs | 1046.22 µs | **0.81× (faster)** | +| `dec+cast` string_map, N=4096 | 367.76 µs | 335.83 µs | 343.00 µs | 0.93× (faster) | +| `encode` original_nested (composite) | 3.48 µs | 17.11 µs | 12.26 µs | 3.52× slower | +| `decode` original_nested (composite) | 0.58 µs | 3.21 µs | 3.29 µs | 5.67× slower | + +Two things are true at once. **List-shaped payloads regressed badly** at the +boundary and have barely improved since 1.0.0 — integer lists ~7× slower to +encode and ~13× slower to decode than v0.4.0, long-string lists ~25× slower to +decode. **String-keyed maps went the other way**: 1.0.2 encodes them 1.24× +faster than v0.4.0 and 2.1× faster than 1.0.0 (2194.00 → 1046.22 µs), and +`decode + deep_cast` on them is at parity or better across all three refs. + +If your host passes large lists across the boundary per request, v0.4.0 was +faster at that specific thing and this is a known gap, not a measurement +artifact. It is the clearest optimisation target this report surfaces. + +## Where we are still behind + +Three cases where 1.0.2 is slower than the same-run Luerl control. All three +improved relative to 1.0.0; none has closed. + +| Case | v1.0.0 ratio | 1.0.2 ratio | 1.0.2 median vs control | +|---|---|---|---| +| pcall, raise + catch | 1.81× | **1.55×** | 727.13 µs vs 468.42 µs | +| varargs + multiple returns | 1.67× | **1.36×** | 3.28 ms vs 2.41 ms | +| `table.sort` (n=1000) | 1.28× | **1.11×** | 199.34 µs vs 179.09 µs | +| patterns: find-based tokenizer | 1.16× | **1.07×** | 1.46 ms vs 1.36 ms | + +**The raise path is not an apples-to-apples work comparison.** On 1.0.2, +`error("negative")` produces `":1: negative"` — position-prefixed, which +is what PUC-Lua does at error level 1. Luerl (and therefore v0.4.0) produces a +bare `"negative"` with no position information. 1.0.2 is doing strictly more +work per raise, and the extra work is the *conformant* behaviour. The 1.55× +should be read as "we pay 1.55× for a more correct error value", not as pure +overhead. It is still worth optimising — position capture need not cost this +much — but it is not a like-for-like loss. + +**Varargs is a genuine loss.** 1.36× slower and allocating 16.70 MB against +the control's 8.73 MB (1.91×) on the same source. Variadic collection and +`table.pack`/`unpack` round-tripping are doing measurably more allocation than +they need to. This is the most actionable pure-performance gap in the suite. + +For the record, `BASELINE.md`'s 1.0.0 gate statement — "every workload is +within 25% of Luerl on the chunk path" — was true of the workloads that +existed when it was written. It is not a property of the current, wider suite: +the pcall-raise and varargs cases were added for this report and both exceed +that band, on 1.0.0 as well as on 1.0.2. + +## Caveats + +1. **v0.4.0 *is* Luerl.** The v0.4.0 column is a thin Elixir wrapper over + `luerl 1.5.1`, so its `lua` rows and its own `luerl` control row measure + substantially the same engine. The data shows this directly: + `Lua.new(sandboxed: [])` at 14.88 µs vs `:luerl.init()` at 15.04 µs (1.1% + apart); `table.concat` at 38.38 µs vs 39.42 µs; `gsub` substitution at + 2.21 ms vs 2.21 ms; arithmetic metamethods at 1.88 ms vs 1.88 ms. **v0.4.0 + is not an independent data point.** Read it as "what an embedder got before + the native VM", and read the v0.4.0 ratio column as a noise floor — it + should sit at 1.00, and where it strays (0.94 on closures, 1.04 on oop) that + is the measurement error budget for this whole report. + +2. **Coroutines are absent by design and are not benchmarked.** The + `coroutine` library is not implemented on v1.0.0 or 1.0.2 — an explicit 1.0 + capability exclusion, recorded as such in the Lua 5.3 suite skip list. Any + coroutine workload would run only on v0.4.0. This is a real gap in + "representative Lua" coverage and is disclosed rather than benchmarked + around. If your embedding needs coroutines, this library does not currently + provide them. + +3. **Four of these workloads are new as of this report.** `patterns`, + `metamethods`, `pcall_varargs` and `vm_new` were added specifically for this + comparison, to cover the pattern engine, self-call/metamethod dispatch, the + protected-call and variadic protocol, and instantiation — none of which the + pre-existing suite touched. They are run against all three refs from the + same source files, so the cross-version numbers are valid, but they have no + history before 2026-07-28 and no `BASELINE.md` counterpart. + +4. **Single machine, single OTP version.** Everything here is one Apple M4 on + Elixir 1.20.0 / OTP 29. Ratios against the same-run Luerl control should + travel reasonably well; absolute microsecond figures will not. Nothing here + has been reproduced on x86_64, on another OTP release, or under a different + scheduler configuration. + +5. **Sub-microsecond rows need medians.** The `Lua.new()` rows on 1.0.2 report + deviations in the hundreds to over a thousand percent because the operation + is near timer resolution and Benchee batches to compensate. Medians and p99s + are tight and meaningful; the deviation column is not, and should not be + republished for those rows without explanation. + +6. **`table_ops` small and medium sizes are too noisy to publish ratios from.** + At n=10 and n=100 the deviations run ±20–46% on operations taking 2–25 µs. + One such cell (v0.4.0 build at n=100) reads 1.60× purely from noise on a ref + where both rows are the same engine. Only the n=1000 column is quoted above. + +7. **No C Lua reference.** `:luaport` was unavailable in this environment, so + there is no PUC-Lua/C row anywhere here. Nothing in this report should be + read as a comparison against reference Lua; it is a comparison of this + library against itself over time, with Luerl as the control. + +8. **`lua (eval)` vs `lua (chunk)`.** The `chunk` path — compile once, run many + — is quoted throughout because it is the production embedding path. The + `eval` path (parse on every call) is within a few percent on most workloads; + the gap is largest where the script is small relative to its parse, and it + inverts on a few cases where measurement noise exceeds the difference. Full + `eval` figures are in the JSON. + +## Reproduction + +All raw stdout, parsed JSON and environment probes for this run are under +[`v0.4.0/`](./v0.4.0/), [`v1.0.0/`](./v1.0.0/) and [`v1.0.2/`](./v1.0.2/), +one directory per released version. Every figure in this +report is traceable to the `median` field of a job in the corresponding +`summary.json`. To regenerate: + +### 1.0.2 / main + +Benchee and Luerl are gated to the `:benchmark` env, so `MIX_ENV=benchmark` is +mandatory. + +```sh +MIX_ENV=benchmark mix deps.get +for w in fibonacci closures oop string_ops string_format table_ops \ + patterns metamethods pcall_varargs vm_new encode_decode; do + LUA_BENCH_MODE=full MIX_ENV=benchmark mix run "benchmarks/$w.exs" +done +``` + +Run these **serially**, as written — not in parallel, and not alongside a test +suite. `mix lua.bench --workload ` is a convenience wrapper that sets the +env for you, but note that invoking it with no `--workload` also picks up +`array_vs_map_probe` (a data-structure probe that runs no Lua) and +`dispatcher_vs_interpreter` (an internal A/B that reaches into +`Lua.Compiler.Prototype` and `Lua.VM.State`). Neither belongs in a +cross-version comparison, and the latter cannot run on v0.4.0 at all. + +### v1.0.0 + +`benchmarks/` at tag `v1.0.0` is byte-identical to the 1.0.2 tree for the +pre-existing workloads, and `mix.exs` deps are the same. Copy in the four +workloads added for this report, then run as above: + +```sh +git worktree add --detach /tmp/lua-v1.0.0 v1.0.0 +cp benchmarks/{patterns,metamethods,pcall_varargs,vm_new}.exs \ + /tmp/lua-v1.0.0/benchmarks/ +cd /tmp/lua-v1.0.0 && MIX_ENV=benchmark mix deps.get +# then the same loop as above +``` + +### v0.4.0 + +Three adaptations are required: + +1. **There is no `benchmarks/` directory at that tag.** Copy the whole + directory in from the current tree. +2. **There is no `benchee` dependency.** Add + `{:benchee, "~> 1.3", only: :benchmark}` to `deps/0` in the worktree's + `mix.exs`, then `MIX_ENV=benchmark mix deps.get`. (`luerl` is an + unconditional runtime dependency at v0.4.0, so the control rows need no + adaptation on any ref.) +3. **Nothing else.** In particular the `load_chunk!/2` state threading that + v0.4.0 requires is **already committed** in every benchmark file. At v0.4.0 + a `%Lua.Chunk{}` holds a `:ref` into the state it was loaded against, so + discarding that state invalidates the chunk and the `lua (chunk)` job dies + with `key N not found`. Every file now writes + `{chunk, lua} = Lua.load_chunk!(lua, ...)`, which is correct and free on all + three refs. + +```sh +git worktree add --detach /tmp/lua-v0.4.0 v0.4.0 +cp -R benchmarks /tmp/lua-v0.4.0/benchmarks +# add {:benchee, "~> 1.3", only: :benchmark} to deps/0 in /tmp/lua-v0.4.0/mix.exs +cd /tmp/lua-v0.4.0 && MIX_ENV=benchmark mix deps.get +# then the same loop as above +``` + +### Language constraints when editing workloads + +Empirically verified against all three refs. A workload that violates any of +these cannot be measured across the whole history: + +- **Use global `function f(...)`, not `local function f(...)`, for anything + self-recursive.** `local function` self-recursion fails on v0.4.0 + (`undefined function nil`). A forward-declared `local f; f = function ...` + also works. +- **No `string.gmatch`.** Raises `{:badarg, :gmatch, ...}` in Luerl 1.5.1, so + it is broken on v0.4.0 and unusable for the control row on any ref. It works + correctly on v1.0.0 and 1.0.2. +- **No `goto`/labels** (parse error on v0.4.0), **no `xpcall`** (broken on + v0.4.0), **no `coroutine.*`** (absent on v1.0.0 and 1.0.2). +- **No non-string `error()` values.** v0.4.0 stringifies them; 1.0.2 preserves + the table. +- **Beware `select("#", ...)`** — returns a float on v0.4.0/Luerl and an + integer on v1.0.0/1.0.2. Harmless in `pcall_varargs` (same iteration count), + but it makes a dedicated integer-arithmetic benchmark measure float math on + one column and integer math on another. diff --git a/benchmarks/closures.exs b/benchmarks/closures.exs index 8910e607..3e6c350e 100644 --- a/benchmarks/closures.exs +++ b/benchmarks/closures.exs @@ -53,9 +53,13 @@ end call_closures = "return run_closures(100)" # --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. lua = Lua.new() {_, lua} = Lua.eval!(lua, closure_def) -{closure_chunk, _} = Lua.load_chunk!(lua, call_closures) +{closure_chunk, lua} = Lua.load_chunk!(lua, call_closures) # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/fibonacci.exs b/benchmarks/fibonacci.exs index 502f889f..88ee743d 100644 --- a/benchmarks/fibonacci.exs +++ b/benchmarks/fibonacci.exs @@ -27,9 +27,13 @@ end call_fib = "return fib(30)" # --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. lua = Lua.new() {_, lua} = Lua.eval!(lua, fib_def) -{fib_chunk, _} = Lua.load_chunk!(lua, call_fib) +{fib_chunk, lua} = Lua.load_chunk!(lua, call_fib) # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/metamethods.exs b/benchmarks/metamethods.exs new file mode 100644 index 00000000..57f54803 --- /dev/null +++ b/benchmarks/metamethods.exs @@ -0,0 +1,193 @@ +# Run with: mix run benchmarks/metamethods.exs +# +# Benchmarks metatable-driven dispatch. oop.exs already covers the shallow +# case — one `setmetatable` + a single-level `__index` table lookup, with +# methods invoked as plain field calls (`Animal.speak(a)`). This script covers +# the three things that shape real Lua OOP code and that the shallow case +# leaves untouched: +# +# - methods: method calls written with the `:` sugar (`v:len2()`), so the +# receiver is threaded as an implicit `self` argument, plus a +# chained call on a freshly constructed receiver +# (`v:scaled(2):len2()`). This is the self-call path; the plain +# `T.f(obj)` form in oop.exs does not reach it. +# - inherit: a three-level prototype chain (Rect -> Polygon -> Shape) built +# the idiomatic way, with `setmetatable` on the class tables +# themselves. Each call resolves at a different depth: one hit on +# the leaf, one two hops up, one three hops up — so the cost of +# walking a chain is separated from the cost of a single hit. +# - arith: arithmetic and relational metamethods (`__add`, `__sub`, +# `__lt`, `__eq`) plus `__tostring`. These route through the +# metamethod fallback in the arithmetic/comparison opcodes rather +# than through table indexing, which is a different VM path from +# `__index` entirely. +# +# Each workload runs n=200 iterations per invocation. +# +# Compares: +# - This Lua implementation (eval with string, eval with pre-compiled chunk) +# - Luerl (Erlang-based Lua 5.3 implementation) +# - C Lua 5.4 via luaport (port-based; results include IPC overhead) +# +# NOTE: luaport requires C Lua 5.4 development headers and a small in-tree +# patch (its 1.6.3 release defaults to LuaJIT and uses LUA_GLOBALSINDEX which +# was removed in Lua 5.2). On macOS: +# brew install lua@5.4 +# ./benchmarks/setup_luaport.sh # idempotent; patches + builds +# MIX_ENV=benchmark mix run benchmarks/metamethods.exs +# If luaport fails to start, the benchmark prints a notice and skips it. +# +# Run modes (see benchmarks/helpers.exs): +# default — quick mode (~4 s per Benchee.run) +# LUA_BENCH_MODE=full — long windows + memory_time, for publishable numbers + +Code.require_file("helpers.exs", __DIR__) + +Application.ensure_all_started(:luerl) + +meta_def = """ +-- Self-call dispatch via the `:` sugar, including a chained call. +Vec = {} +Vec.__index = Vec + +function Vec.new(x, y) + return setmetatable({ x = x, y = y }, Vec) +end + +function Vec:len2() + return self.x * self.x + self.y * self.y +end + +function Vec:scaled(k) + return Vec.new(self.x * k, self.y * k) +end + +function run_methods(n) + local v = Vec.new(3, 4) + local acc = 0 + for i = 1, n do + acc = acc + v:len2() + acc = acc + v:scaled(2):len2() + end + return acc +end + +-- Three-level prototype chain; the three calls below resolve at depth 1, 2 +-- and 3 respectively, so a chain walk is measured alongside a direct hit. +Shape = {} +Shape.__index = Shape + +function Shape:kind() return "shape" end +function Shape:area() return 0 end + +Polygon = setmetatable({}, { __index = Shape }) +Polygon.__index = Polygon + +function Polygon:sides() return 0 end + +Rect = setmetatable({}, { __index = Polygon }) +Rect.__index = Rect + +function Rect.new(w, h) + return setmetatable({ w = w, h = h }, Rect) +end + +function Rect:sides() return 4 end + +function run_inherit(n) + local r = Rect.new(3, 4) + local acc = 0 + for i = 1, n do + acc = acc + r:sides() + acc = acc + r:area() + acc = acc + #r:kind() + end + return acc +end + +-- Arithmetic / relational / tostring metamethods. +Money = {} +Money.__index = Money +Money.__add = function(a, b) return Money.new(a.cents + b.cents) end +Money.__sub = function(a, b) return Money.new(a.cents - b.cents) end +Money.__lt = function(a, b) return a.cents < b.cents end +Money.__eq = function(a, b) return a.cents == b.cents end +Money.__tostring = function(m) return "$" .. tostring(m.cents) end + +function Money.new(cents) + return setmetatable({ cents = cents }, Money) +end + +function run_arith(n) + local acc = Money.new(0) + local flags = 0 + local last = "" + for i = 1, n do + acc = acc + Money.new(i) + acc = acc - Money.new(1) + if Money.new(i) < Money.new(i + 1) then flags = flags + 1 end + if Money.new(i) == Money.new(i) then flags = flags + 1 end + last = tostring(acc) + end + return acc.cents, flags, #last +end +""" + +call_methods = "return run_methods(200)" +call_inherit = "return run_inherit(200)" +call_arith = "return run_arith(200)" + +# --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. +lua = Lua.new() +{_, lua} = Lua.eval!(lua, meta_def) +{methods_chunk, lua} = Lua.load_chunk!(lua, call_methods) +{inherit_chunk, lua} = Lua.load_chunk!(lua, call_inherit) +{arith_chunk, lua} = Lua.load_chunk!(lua, call_arith) + +# --- Luerl --- +luerl_state = :luerl.init() +{:ok, _, luerl_state} = :luerl.do(meta_def, luerl_state) + +# --- C Lua via luaport (optional) --- +{c_lua, c_lua_cleanup} = + case Application.ensure_all_started(:luaport) do + {:ok, _} -> + scripts_dir = Path.join(__DIR__, "scripts") + {:ok, port_pid, _} = :luaport.spawn(:meta_bench, to_charlist(scripts_dir)) + :luaport.load(port_pid, meta_def) + + { + fn func -> %{"C Lua (luaport)" => fn -> :luaport.call(port_pid, func, [200]) end} end, + fn -> :luaport.despawn(:meta_bench) end + } + + {:error, reason} -> + IO.puts("luaport not available (#{inspect(reason)}) — skipping C Lua benchmarks") + {fn _func -> %{} end, fn -> :ok end} + end + +bench = fn name, call_str, chunk, c_lua_func -> + Bench.banner(name) + + Benchee.run( + Map.merge( + %{ + "lua (eval)" => fn -> Lua.eval!(lua, call_str) end, + "lua (chunk)" => fn -> Lua.eval!(lua, chunk) end, + "luerl" => fn -> :luerl.do(call_str, luerl_state) end + }, + c_lua.(c_lua_func) + ), + Bench.opts() + ) +end + +bench.("metamethods: self-call method dispatch (n=200)", call_methods, methods_chunk, :run_methods) +bench.("metamethods: 3-level __index chain (n=200)", call_inherit, inherit_chunk, :run_inherit) +bench.("metamethods: arithmetic/relational metamethods (n=200)", call_arith, arith_chunk, :run_arith) + +c_lua_cleanup.() diff --git a/benchmarks/oop.exs b/benchmarks/oop.exs index e6790e58..2cd4be3e 100644 --- a/benchmarks/oop.exs +++ b/benchmarks/oop.exs @@ -1,14 +1,21 @@ # Run with: mix run benchmarks/oop.exs # -# Benchmarks object-oriented patterns using Lua tables and metatables. -# Uses assignment-style method definitions (e.g. Animal.speak = function(self) ... end) -# which are compatible with this Lua implementation's current feature set. -# Creates 50 Animal instances per iteration and calls a method on each. +# Benchmarks the shallow, construction-dominated end of Lua OOP: 50 instances +# per iteration, each built with two field writes and a `setmetatable`, then +# sent one method whose body concatenates two of its fields. +# +# Methods are defined and invoked in assignment/field-call style +# (`Animal.speak = function(self) ... end`, called as `Animal.speak(a)`) rather +# than with the `:` sugar. That is not a language limitation — `:` method +# definitions and self-calls work — it holds dispatch to its simplest form so +# this workload stays dominated by construction cost. The dispatch-heavy +# counterpart (`:` self-calls, multi-level `__index` chains, arithmetic +# metamethods) is benchmarks/metamethods.exs. # # Patterns tested: # - Table creation and field assignment -# - setmetatable / __index prototype chain lookup -# - Closure creation per object (factory pattern variant) +# - setmetatable, plus __index lookup one level up the prototype chain +# - String concatenation and tostring inside a method body # # Compares: # - This Lua implementation (eval with string, eval with pre-compiled chunk) @@ -60,9 +67,13 @@ end call_oop = "return run_oop(50)" # --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. lua = Lua.new() {_, lua} = Lua.eval!(lua, oop_def) -{oop_chunk, _} = Lua.load_chunk!(lua, call_oop) +{oop_chunk, lua} = Lua.load_chunk!(lua, call_oop) # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/patterns.exs b/benchmarks/patterns.exs new file mode 100644 index 00000000..05ffd113 --- /dev/null +++ b/benchmarks/patterns.exs @@ -0,0 +1,156 @@ +# Run with: mix run benchmarks/patterns.exs +# +# Benchmarks Lua's string-pattern engine — the part of the string library that +# compiles and matches Lua patterns, as opposed to the byte-copying and +# formatting paths covered by string_ops.exs / string_format.exs. +# +# - scan: repeated `string.find` + `string.match` over one log line. Drives +# character classes (%a, %d, %w), quantifiers, escaped magic +# characters (%[ %]) and single-capture extraction. This is the +# shape of nearly every "pull fields out of a line" script. +# - split: tokenises a comma-separated list with `string.find(s, "[^,]+", pos)` +# advanced by an explicit init offset. Exercises the anchor-free +# restart path — the matcher is re-entered once per token with a +# fresh start position. +# - gsub: template substitution in three passes: `%${(%w+)}` with a table +# replacement, `%s+` whitespace squeezing with a string +# replacement, and `(%a+)` with a *function* replacement. Covers +# all three gsub replacement kinds plus capture-driven lookup. +# +# `split` deliberately uses `string.find` with an init offset rather than +# `string.gmatch`: `gmatch` raises `badarg` in Luerl 1.5.x, so a gmatch-based +# tokeniser could not be measured against the Luerl reference on the same Lua +# source. Keeping the source identical across VMs is the fairness contract of +# this suite, so the iterator-free idiom is used instead. +# +# Each workload runs n=200 pattern-heavy iterations per invocation so the +# per-match cost is visible above harness overhead. +# +# Compares: +# - This Lua implementation (eval with string, eval with pre-compiled chunk) +# - Luerl (Erlang-based Lua 5.3 implementation) +# - C Lua 5.4 via luaport (port-based; results include IPC overhead) +# +# NOTE: luaport requires C Lua 5.4 development headers and a small in-tree +# patch (its 1.6.3 release defaults to LuaJIT and uses LUA_GLOBALSINDEX which +# was removed in Lua 5.2). On macOS: +# brew install lua@5.4 +# ./benchmarks/setup_luaport.sh # idempotent; patches + builds +# MIX_ENV=benchmark mix run benchmarks/patterns.exs +# If luaport fails to start, the benchmark prints a notice and skips it. +# +# Run modes (see benchmarks/helpers.exs): +# default — quick mode (~4 s per Benchee.run) +# LUA_BENCH_MODE=full — long windows + memory_time, for publishable numbers + +Code.require_file("helpers.exs", __DIR__) + +Application.ensure_all_started(:luerl) + +pattern_def = """ +local LOG = "2024-05-01 12:34:56 [warn] request_id=a1b2c3 latency=42ms status=503 path=/api/v1/items" + +-- Field extraction: an escaped-magic-character find plus three captures. +function run_scan(n) + local hits = 0 + for i = 1, n do + local s, e = string.find(LOG, "%[%a+%]") + if s then hits = hits + (e - s) end + local status = string.match(LOG, "status=(%d+)") + local id = string.match(LOG, "request_id=(%w+)") + local lat = string.match(LOG, "latency=(%d+)ms") + if status and id and lat then hits = hits + #status + #id + #lat end + end + return hits +end + +local CSV = "alpha,beta,gamma,delta,epsilon,zeta,eta,theta,iota,kappa" + +-- Tokenise by re-entering the matcher at an advancing init offset. +function run_split(n) + local total = 0 + for i = 1, n do + local pos = 1 + while true do + local s, e = string.find(CSV, "[^,]+", pos) + if not s then break end + total = total + (e - s + 1) + pos = e + 1 + end + end + return total +end + +local TEMPLATE = "Hello ${name}, you have ${count} new ${kind} since ${when}. Visit ${url} for details." +local VALUES = { name = "Ada", count = "7", kind = "messages", when = "Tuesday", url = "/inbox" } + +-- All three gsub replacement kinds: table, string, function. +function run_gsub(n) + local last = "" + for i = 1, n do + local filled = string.gsub(TEMPLATE, "%${(%w+)}", VALUES) + local squeezed = string.gsub(filled, "%s+", " ") + last = string.gsub(squeezed, "(%a+)", function(w) return w end) + end + return #last +end +""" + +call_scan = "return run_scan(200)" +call_split = "return run_split(200)" +call_gsub = "return run_gsub(200)" + +# --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. +lua = Lua.new() +{_, lua} = Lua.eval!(lua, pattern_def) +{scan_chunk, lua} = Lua.load_chunk!(lua, call_scan) +{split_chunk, lua} = Lua.load_chunk!(lua, call_split) +{gsub_chunk, lua} = Lua.load_chunk!(lua, call_gsub) + +# --- Luerl --- +luerl_state = :luerl.init() +{:ok, _, luerl_state} = :luerl.do(pattern_def, luerl_state) + +# --- C Lua via luaport (optional) --- +{c_lua, c_lua_cleanup} = + case Application.ensure_all_started(:luaport) do + {:ok, _} -> + scripts_dir = Path.join(__DIR__, "scripts") + {:ok, port_pid, _} = :luaport.spawn(:pattern_bench, to_charlist(scripts_dir)) + :luaport.load(port_pid, pattern_def) + + { + fn func -> %{"C Lua (luaport)" => fn -> :luaport.call(port_pid, func, [200]) end} end, + fn -> :luaport.despawn(:pattern_bench) end + } + + {:error, reason} -> + IO.puts("luaport not available (#{inspect(reason)}) — skipping C Lua benchmarks") + {fn _func -> %{} end, fn -> :ok end} + end + +bench = fn name, call_str, chunk, c_lua_func -> + Bench.banner(name) + + Benchee.run( + Map.merge( + %{ + "lua (eval)" => fn -> Lua.eval!(lua, call_str) end, + "lua (chunk)" => fn -> Lua.eval!(lua, chunk) end, + "luerl" => fn -> :luerl.do(call_str, luerl_state) end + }, + c_lua.(c_lua_func) + ), + Bench.opts() + ) +end + +bench.("patterns: find/match field extraction (n=200)", call_scan, scan_chunk, :run_scan) +bench.("patterns: find-based tokenizer (n=200)", call_split, split_chunk, :run_split) +bench.("patterns: gsub template substitution (n=200)", call_gsub, gsub_chunk, :run_gsub) + +c_lua_cleanup.() diff --git a/benchmarks/pcall_varargs.exs b/benchmarks/pcall_varargs.exs new file mode 100644 index 00000000..75ba3192 --- /dev/null +++ b/benchmarks/pcall_varargs.exs @@ -0,0 +1,161 @@ +# Run with: mix run benchmarks/pcall_varargs.exs +# +# Benchmarks the call protocol: protected calls and variadic/multiple-return +# argument handling. Both are pervasive in embedded Lua — host integrations +# routinely wrap every script entry point in `pcall`, and `...`/multiple +# returns are how Lua code passes argument lists around — and neither appears +# in the comparative workloads otherwise. +# +# - pcall_ok: n protected calls that all succeed. Isolates the cost of +# entering and leaving a protected frame from the cost of +# actually raising, which is the common case in production. +# - pcall_raise: n protected calls that all raise a string error and are +# caught. Drives error-value construction, stack unwinding +# and the return of `false, err` to the caller. +# - varargs: variadic collection (`select("#", ...)`), positional +# variadic access (`select(i, ...)`), variadic forwarding +# (`f(...)` in tail position), `table.pack`/`table.unpack` +# round-tripping, and multiple-return destructuring +# (`local a, b, c = triple(i)`). +# +# Each workload runs n=500 iterations per invocation. +# +# Compares: +# - This Lua implementation (eval with string, eval with pre-compiled chunk) +# - Luerl (Erlang-based Lua 5.3 implementation) +# - C Lua 5.4 via luaport (port-based; results include IPC overhead) +# +# NOTE: luaport requires C Lua 5.4 development headers and a small in-tree +# patch (its 1.6.3 release defaults to LuaJIT and uses LUA_GLOBALSINDEX which +# was removed in Lua 5.2). On macOS: +# brew install lua@5.4 +# ./benchmarks/setup_luaport.sh # idempotent; patches + builds +# MIX_ENV=benchmark mix run benchmarks/pcall_varargs.exs +# If luaport fails to start, the benchmark prints a notice and skips it. +# +# Run modes (see benchmarks/helpers.exs): +# default — quick mode (~4 s per Benchee.run) +# LUA_BENCH_MODE=full — long windows + memory_time, for publishable numbers + +Code.require_file("helpers.exs", __DIR__) + +Application.ensure_all_started(:luerl) + +call_def = """ +-- Raises for negative input, returns normally otherwise, so the same callee +-- drives both the success and the error path below. +function classify(v) + if v < 0 then + error("negative") + end + return v * 2 +end + +function run_pcall_ok(n) + local acc = 0 + for i = 1, n do + local ok, v = pcall(classify, i) + if ok then acc = acc + v end + end + return acc +end + +function run_pcall_raise(n) + local caught = 0 + for i = 1, n do + local ok, err = pcall(classify, -i) + if not ok and type(err) == "string" then caught = caught + 1 end + end + return caught +end + +-- Variadic collection and positional access. +function tally(...) + local count = select("#", ...) + local acc = 0 + for i = 1, count do + acc = acc + select(i, ...) + end + return acc, count +end + +-- Variadic forwarding in tail position. +function forward(...) + return tally(...) +end + +function triple(i) + return i, i + 1, i + 2 +end + +function run_varargs(n) + local acc = 0 + for i = 1, n do + local a, b, c = triple(i) + local sum, count = forward(a, b, c, i, i * 2) + acc = acc + sum + local packed = table.pack(a, b, c) + acc = acc + tally(table.unpack(packed, 1, packed.n)) + end + return acc +end +""" + +call_pcall_ok = "return run_pcall_ok(500)" +call_pcall_raise = "return run_pcall_raise(500)" +call_varargs = "return run_varargs(500)" + +# --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. +lua = Lua.new() +{_, lua} = Lua.eval!(lua, call_def) +{pcall_ok_chunk, lua} = Lua.load_chunk!(lua, call_pcall_ok) +{pcall_raise_chunk, lua} = Lua.load_chunk!(lua, call_pcall_raise) +{varargs_chunk, lua} = Lua.load_chunk!(lua, call_varargs) + +# --- Luerl --- +luerl_state = :luerl.init() +{:ok, _, luerl_state} = :luerl.do(call_def, luerl_state) + +# --- C Lua via luaport (optional) --- +{c_lua, c_lua_cleanup} = + case Application.ensure_all_started(:luaport) do + {:ok, _} -> + scripts_dir = Path.join(__DIR__, "scripts") + {:ok, port_pid, _} = :luaport.spawn(:call_bench, to_charlist(scripts_dir)) + :luaport.load(port_pid, call_def) + + { + fn func -> %{"C Lua (luaport)" => fn -> :luaport.call(port_pid, func, [500]) end} end, + fn -> :luaport.despawn(:call_bench) end + } + + {:error, reason} -> + IO.puts("luaport not available (#{inspect(reason)}) — skipping C Lua benchmarks") + {fn _func -> %{} end, fn -> :ok end} + end + +bench = fn name, call_str, chunk, c_lua_func -> + Bench.banner(name) + + Benchee.run( + Map.merge( + %{ + "lua (eval)" => fn -> Lua.eval!(lua, call_str) end, + "lua (chunk)" => fn -> Lua.eval!(lua, chunk) end, + "luerl" => fn -> :luerl.do(call_str, luerl_state) end + }, + c_lua.(c_lua_func) + ), + Bench.opts() + ) +end + +bench.("call protocol: pcall, success path (n=500)", call_pcall_ok, pcall_ok_chunk, :run_pcall_ok) +bench.("call protocol: pcall, raise + catch (n=500)", call_pcall_raise, pcall_raise_chunk, :run_pcall_raise) +bench.("call protocol: varargs + multiple returns (n=500)", call_varargs, varargs_chunk, :run_varargs) + +c_lua_cleanup.() diff --git a/benchmarks/string_format.exs b/benchmarks/string_format.exs index 0dd1e16d..70cde337 100644 --- a/benchmarks/string_format.exs +++ b/benchmarks/string_format.exs @@ -78,11 +78,15 @@ call_width = "return run_width_format(1000)" call_many = "return run_many_specs(1000)" # --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. lua = Lua.new() {_, lua} = Lua.eval!(lua, string_def) -{long_chunk, _} = Lua.load_chunk!(lua, call_long) -{width_chunk, _} = Lua.load_chunk!(lua, call_width) -{many_chunk, _} = Lua.load_chunk!(lua, call_many) +{long_chunk, lua} = Lua.load_chunk!(lua, call_long) +{width_chunk, lua} = Lua.load_chunk!(lua, call_width) +{many_chunk, lua} = Lua.load_chunk!(lua, call_many) # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/string_ops.exs b/benchmarks/string_ops.exs index 50e66881..2c6431d0 100644 --- a/benchmarks/string_ops.exs +++ b/benchmarks/string_ops.exs @@ -43,10 +43,14 @@ call_concat = "return run_concat(100)" call_format = "return run_format(100)" # --- This Lua implementation --- +# The state returned by `load_chunk!/2` is threaded through each call rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. Threading it +# is correct on every release and costs nothing. lua = Lua.new() {_, lua} = Lua.eval!(lua, string_def) -{concat_chunk, _} = Lua.load_chunk!(lua, call_concat) -{format_chunk, _} = Lua.load_chunk!(lua, call_format) +{concat_chunk, lua} = Lua.load_chunk!(lua, call_concat) +{format_chunk, lua} = Lua.load_chunk!(lua, call_format) # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/table_ops.exs b/benchmarks/table_ops.exs index 684bf376..94deb475 100644 --- a/benchmarks/table_ops.exs +++ b/benchmarks/table_ops.exs @@ -97,37 +97,30 @@ lua = Lua.new() # Pre-compile chunks per (operation, n) pair so the chunk path doesn't # pay the compile cost during measurement. Inputs ship through Benchee's # `inputs:` mechanism so all sizes share warmup/measurement state. +# +# The state returned by `load_chunk!/2` is threaded through every load rather +# than discarded: a loaded chunk may be a reference *into* the state it was +# loaded against, so dropping that state can invalidate the chunk. All five +# chunk maps are therefore built against one accumulating state, and that final +# state is the one the benchmarks below evaluate against. sizes = Bench.table_inputs() -build_chunks = - Map.new(sizes, fn {label, n} -> - {chunk, _} = Lua.load_chunk!(lua, "return run_table_build(#{n})") - {label, {chunk, "return run_table_build(#{n})", n}} - end) - -sort_chunks = - Map.new(sizes, fn {label, n} -> - {chunk, _} = Lua.load_chunk!(lua, "return run_table_sort(#{n})") - {label, {chunk, "return run_table_sort(#{n})", n}} - end) - -sum_chunks = - Map.new(sizes, fn {label, n} -> - {chunk, _} = Lua.load_chunk!(lua, "return run_table_sum(#{n})") - {label, {chunk, "return run_table_sum(#{n})", n}} - end) - -map_reduce_chunks = - Map.new(sizes, fn {label, n} -> - {chunk, _} = Lua.load_chunk!(lua, "return run_table_map_reduce(#{n})") - {label, {chunk, "return run_table_map_reduce(#{n})", n}} - end) - -pairs_hash_chunks = - Map.new(sizes, fn {label, n} -> - {chunk, _} = Lua.load_chunk!(lua, "return run_table_pairs_hash(#{n})") - {label, {chunk, "return run_table_pairs_hash(#{n})", n}} - end) +load_chunks = fn lua, func -> + {loaded, lua} = + Enum.map_reduce(sizes, lua, fn {label, n}, acc -> + call = "return #{func}(#{n})" + {chunk, acc} = Lua.load_chunk!(acc, call) + {{label, {chunk, call, n}}, acc} + end) + + {Map.new(loaded), lua} +end + +{build_chunks, lua} = load_chunks.(lua, "run_table_build") +{sort_chunks, lua} = load_chunks.(lua, "run_table_sort") +{sum_chunks, lua} = load_chunks.(lua, "run_table_sum") +{map_reduce_chunks, lua} = load_chunks.(lua, "run_table_map_reduce") +{pairs_hash_chunks, lua} = load_chunks.(lua, "run_table_pairs_hash") # --- Luerl --- luerl_state = :luerl.init() diff --git a/benchmarks/vm_new.exs b/benchmarks/vm_new.exs new file mode 100644 index 00000000..10d0ad18 --- /dev/null +++ b/benchmarks/vm_new.exs @@ -0,0 +1,94 @@ +# Run with: mix run benchmarks/vm_new.exs +# +# Benchmarks VM instantiation — `Lua.new/1` — the cost an embedding host pays +# before a single line of Lua runs. Hosts that build a fresh sandbox per +# request (the recommended isolation model) pay this on every request, so it +# sits directly in the request path and is worth measuring separately from +# script execution. +# +# Three instantiation shapes are measured, chosen because all three are valid +# on every release of this library (`:sandboxed` and `:exclude` are the only +# `new/1` options that exist across the whole history; the limit options +# `:max_call_depth` / `:max_string_bytes` / `:max_instructions` / `:debug` are +# newer and would raise on older releases): +# +# - new — `Lua.new()`. The default deny-list sandbox. This is +# what >90% of embedders call. +# - new, no sandbox — `Lua.new(sandboxed: [])`. Standard library +# installed, zero sandbox passes. This is the closest +# like-for-like analogue of a bare `:luerl.init()`, +# which also performs no sandboxing — compare the +# luerl row against *this* row, not against `new`. +# - new, custom exclude — `Lua.new(exclude: [[:require]])`. The default +# deny-list minus one entry. Represents "I want the +# sandbox but need one thing back", and on releases +# that memoize instantiation it is the shape that +# still pays a per-call sandbox pass. +# +# --------------------------------------------------------------------------- +# Cold vs steady state +# --------------------------------------------------------------------------- +# Newer releases memoize the boot-time VM template in `:persistent_term`, +# written once per node. That makes the *first* `Lua.new()` on a node more +# expensive than every subsequent one, so a benchmark could mislead in either +# direction: measuring only the first call would report a cost no real +# workload repeats, while reporting only the steady state would hide a +# one-time cost that does exist. +# +# Both are therefore reported. The script prints an explicitly-labelled cold +# and second-call timing for `Lua.new()` before Benchee starts — taken as the +# very first thing that touches the library, so nothing has warmed the cache — +# and then Benchee measures steady state, which is what a host serving its +# second and subsequent request sees. On releases with no memoization the two +# figures converge, which is itself the interesting signal. +# +# The cold figure is an upper bound: under `mix run` it also absorbs first-time +# code loading of the stdlib modules, which a release has already done at boot. +# +# Note also that `mix run` puts the VM in `:interactive` code-loading mode. +# Releases run `:embedded`, where a memoizing implementation can skip the +# module-reload staleness check a cache hit otherwise performs — so the steady +# state measured here is, if anything, pessimistic relative to production. +# +# Compares: +# - This Lua implementation (three instantiation shapes) +# - Luerl (`:luerl.init/0`, the Erlang-based Lua 5.3 implementation) +# +# There is no C Lua row: `:luaport` instantiation means spawning an OS process +# and handshaking over a port, which measures process spawn and IPC rather than +# VM construction. The two numbers would not mean the same thing. +# +# Run modes (see benchmarks/helpers.exs): +# default — quick mode (~4 s per Benchee.run) +# LUA_BENCH_MODE=full — long windows + memory_time, for publishable numbers + +Code.require_file("helpers.exs", __DIR__) + +Application.ensure_all_started(:luerl) + +Bench.banner("VM instantiation: Lua.new/1 vs :luerl.init/0") + +# Taken before anything else touches the library, so this really is the cold +# path — on a memoizing release it is the call that populates the template. +{cold_us, _} = :timer.tc(fn -> Lua.new() end) +{second_us, _} = :timer.tc(fn -> Lua.new() end) + +IO.puts(""" +Lua.new() one-time vs repeat cost (single samples, informational): + first call on this node : #{:erlang.float_to_binary(cold_us / 1, decimals: 1)} us + second call : #{:erlang.float_to_binary(second_us / 1, decimals: 1)} us + +The first figure includes any one-time template build and first-time module +loading. Benchee's steady-state numbers below are the per-request cost after +that point. +""") + +Benchee.run( + %{ + "lua (new)" => fn -> Lua.new() end, + "lua (new, no sandbox)" => fn -> Lua.new(sandboxed: []) end, + "lua (new, custom exclude)" => fn -> Lua.new(exclude: [[:require]]) end, + "luerl (init)" => fn -> :luerl.init() end + }, + Bench.opts() +) diff --git a/website/lib/website/benchmarks.ex b/website/lib/website/benchmarks.ex new file mode 100644 index 00000000..88ee2f70 --- /dev/null +++ b/website/lib/website/benchmarks.ex @@ -0,0 +1,460 @@ +defmodule Website.Benchmarks do + @moduledoc """ + The recorded cross-version benchmark results, read from `bench_results/`. + + Every released version has a directory of committed benchmark output at + `bench_results//`, and this module turns those `summary.json` files + into the rows rendered by `/benchmarks`. + + Version directories are **discovered by glob** and ordered by + `Version.compare/2`, so recording a new release is the whole update: drop + `bench_results//summary.json` in place and the page grows a column. + The same goes for the report link, which tracks the newest + `bench_results/versions-.md`. + + What is *not* automatic is `@rows` below — the editorial choice of which + workload cases are worth showing, and under what name. A new workload only + appears once it has a row spec. A version that lacks a workload another + version has renders as `—` in that cell rather than failing. + + All file reads happen at compile time and are registered as + `@external_resource`, so editing recorded results recompiles the page. + """ + + @bench_results Path.expand("../../../bench_results", __DIR__) + + # label: what the row is called on the page + # sub: second line, chart only + # at: {workload, case, input | nil} — case/input keys as normalised below + # job: the Benchee job whose number we quote + # vs: the same-run control job the ratio is taken against + # metric: :median (time) or :memory (allocation) + # chart?: whether the row also gets a dot in the ratio plot + # warn?: the newest release is behind the control here — flagged in both views + # dagger?: the comparison is not like-for-like; the page footnotes why + @rows [ + %{ + label: "Lua.new() — steady state", + sub: "default options", + at: {"vm_new", "VM instantiation: Lua.new/1 vs :luerl.init/0", nil}, + job: "lua (new)", + vs: "luerl (init)", + metric: :median, + chart?: false, + warn?: false, + dagger?: false + }, + %{ + label: "Lua.new() — allocation", + sub: "default options", + at: {"vm_new", "VM instantiation: Lua.new/1 vs :luerl.init/0", nil}, + job: "lua (new)", + vs: "luerl (init)", + metric: :memory, + chart?: false, + warn?: false, + dagger?: false + }, + %{ + label: "fibonacci fib(30)", + sub: "recursive calls", + at: {"fibonacci", "default", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "fibonacci — allocation", + sub: "recursive calls", + at: {"fibonacci", "default", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :memory, + chart?: false, + warn?: false, + dagger?: false + }, + %{ + label: "string.format (literal-heavy)", + sub: "literal-heavy template", + at: {"string_format", "string.format: long literal-heavy format string (n=1000)", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "pcall success path (n=500)", + sub: "protected call, no raise", + at: {"pcall_varargs", "call protocol: pcall, success path (n=500)", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "pairs over hash part (n=1000)", + sub: "hash-part iteration", + at: {"table_ops", "Table Pairs (hash)", "large (n=1000)"}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "__index 3-level chain", + sub: "prototype lookup", + at: {"metamethods", "metamethods: 3-level __index chain (n=200)", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "closures", + sub: "factory + upvalue mutation", + at: {"closures", "default", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "table.sort (n=1000)", + sub: "reverse-ordered input", + at: {"table_ops", "Table Sort", "large (n=1000)"}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: false, + dagger?: false + }, + %{ + label: "varargs + multi-return (n=500)", + sub: "call protocol", + at: {"pcall_varargs", "call protocol: varargs + multiple returns (n=500)", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: true, + dagger?: false + }, + %{ + label: "pcall raise + catch (n=500)", + sub: "does strictly more work", + at: {"pcall_varargs", "call protocol: pcall, raise + catch (n=500)", nil}, + job: "lua (chunk)", + vs: "luerl", + metric: :median, + chart?: true, + warn?: true, + dagger?: true + } + ] + + summaries = Path.wildcard(Path.join(@bench_results, "*/summary.json")) + + if summaries == [] do + raise """ + no benchmark summaries found under #{@bench_results} + + /benchmarks renders the committed results in bench_results//. If \ + this is a container build, the image needs the directory: + + COPY bench_results /app/bench_results + """ + end + + for path <- summaries do + @external_resource path + end + + reports = Path.wildcard(Path.join(@bench_results, "versions-*.md")) + + # --- compile-time loading ------------------------------------------------ + + # v0.4.0's run labelled cases differently: banners carry a " (mode: full)" + # suffix, the single-case workloads say "(single case)" where later runs say + # "default", per-input results sit under "inputs" rather than "by_input", and + # input labels are prefixed "With input ". Normalise so one row spec resolves + # against every ref. + normalise_case = fn name -> + case String.replace_suffix(name, " (mode: full)", "") do + "(single case)" -> "default" + other -> other + end + end + + normalise_input = fn name -> String.replace_prefix(name, "With input ", "") end + + parse = fn + nil -> + nil + + value -> + case Regex.run(~r/^([\d.]+)\s*(\S+)$/, String.replace(value, "μ", "µ")) do + [_, number, unit] -> + scale = + case unit do + "ns" -> 1 + "µs" -> 1_000 + "ms" -> 1_000_000 + "s" -> 1_000_000_000 + "B" -> 1 + "KB" -> 1_024 + "MB" -> 1_024 * 1_024 + "GB" -> 1_024 * 1_024 * 1_024 + _ -> nil + end + + # Benchee drops the decimal point on some values ("216 µs"), so + # Float.parse rather than String.to_float. + with true <- is_integer(scale), {number, ""} <- Float.parse(number) do + number * scale + else + _ -> nil + end + + _ -> + nil + end + end + + data = + for path <- summaries, into: %{} do + version = path |> Path.dirname() |> Path.basename() |> String.trim_leading("v") + + cases = + path + |> File.read!() + |> Jason.decode!() + |> Map.new(fn {workload, body} -> + normalised = + case body do + %{"raw" => _} -> + %{} + + cases -> + Map.new(cases, fn {name, one} -> + by_input = Map.get(one, "by_input") || Map.get(one, "inputs") + + value = + if by_input, + do: Map.new(by_input, fn {k, v} -> {normalise_input.(k), v} end), + else: one + + {normalise_case.(name), value} + end) + end + + {workload, normalised} + end) + + {version, cases} + end + + @versions data |> Map.keys() |> Enum.sort(&(Version.compare(&1, &2) != :gt)) + + table_rows = + for spec <- @rows do + {workload, case_name, input} = spec.at + + values = + for version <- @versions, into: %{} do + jobs = + with %{^workload => workloads} <- data[version], + %{^case_name => one} <- workloads do + case input do + nil -> Map.get(one, "jobs") + key -> one |> Map.get(key, %{}) |> Map.get("jobs") + end + else + _ -> nil + end + + find = fn name -> + jobs && Enum.find(jobs, &(&1["name"] == name)) + end + + field = if spec.metric == :memory, do: "memory", else: "median" + mine = find.(spec.job) + control = find.(spec.vs) + + shown = mine && mine[field] + against = control && control[field] + + mine_number = parse.(shown) + against_number = parse.(against) + + ratio = + if is_number(mine_number) and is_number(against_number) and against_number > 0 do + Float.round(mine_number / against_number, 2) + end + + {version, %{value: shown, control: against, ratio: ratio, numeric: mine_number}} + end + + Map.put(spec, :values, values) + end + + @table_rows table_rows + + @report reports |> Enum.map(&Path.basename/1) |> Enum.max(fn -> nil end) + + @report_date (case @report && Regex.run(~r/(\d{4}-\d{2}-\d{2})/, @report) do + [_, date] -> Date.from_iso8601!(date) + _ -> nil + end) + + # --- public API ---------------------------------------------------------- + + @doc """ + Recorded versions, oldest first. + """ + def versions, do: @versions + + @doc """ + The newest recorded version — the one the page's headline numbers describe. + """ + def latest, do: List.last(@versions) + + @doc """ + Table rows: one per row spec, each carrying a value per version. + """ + def rows, do: @table_rows + + @doc """ + The subset of rows plotted as ratio-vs-control dots. + """ + def chart_rows, do: Enum.filter(@table_rows, & &1.chart?) + + @doc """ + Look up one row by label, for the headline tiles. + """ + def row(label), do: Enum.find(@table_rows, &(&1.label == label)) + + @doc """ + The value a row recorded for a version, or `nil` if that version lacks it. + """ + def value(row, version), do: get_in(row.values, [version, :value]) + + @doc """ + Whether a version holds the best (lowest) number in its row. Both metrics — + duration and allocation — are better when smaller. + """ + def best?(row, version) do + numbers = for {_, %{numeric: n}} <- row.values, is_number(n), do: n + mine = get_in(row.values, [version, :numeric]) + + is_number(mine) and numbers != [] and mine == Enum.min(numbers) + end + + @doc """ + The version before the newest — what the headline improvements are measured against. + """ + def previous, do: Enum.at(@versions, -2) + + @doc """ + The figures quoted in the page's headline tiles and callout, derived from the + rows so that recording a new release moves them without an edit here. + """ + def headline do + new = row("Lua.new() — steady state") + new_memory = row("Lua.new() — allocation") + fib = row("fibonacci fib(30)") + fib_memory = row("fibonacci — allocation") + raise_catch = row("pcall raise + catch (n=500)") + varargs = row("varargs + multi-return (n=500)") + + %{ + previous: previous(), + new_median: value(new, latest()), + new_median_prev: value(new, previous()), + new_speedup: improvement(new, previous(), latest()), + new_memory: value(new_memory, latest()), + new_memory_prev: value(new_memory, previous()), + new_memory_factor: improvement(new_memory, previous(), latest()), + fib_median: value(fib, latest()), + fib_median_prev: value(fib, previous()), + fib_speedup: inverse(ratio(fib, latest())), + fib_prev_ratio: ratio(fib, previous()), + fib_memory: value(fib_memory, latest()), + fib_memory_control: get_in(fib_memory.values, [latest(), :control]), + raise_ratio: ratio(raise_catch, latest()), + raise_prev_ratio: ratio(raise_catch, previous()), + varargs_ratio: ratio(varargs, latest()) + } + end + + @doc """ + A row's ratio against its same-run control for one version. + """ + def ratio(row, version), do: get_in(row.values, [version, :ratio]) + + @doc """ + How many times better `to` is than `from` in a row, as display text. + """ + def improvement(row, from, to) do + with a when is_number(a) <- get_in(row.values, [from, :numeric]), + b when is_number(b) <- get_in(row.values, [to, :numeric]), + true <- b > 0 do + format_factor(a / b) + else + _ -> nil + end + end + + @doc """ + Opacity for a version's colour, ramping oldest (faintest) to newest (solid). + """ + def version_weight(_index, count) when count < 2, do: 1.0 + def version_weight(index, count), do: Float.round(0.35 + 0.65 * (index / (count - 1)), 2) + + @doc """ + Filename of the newest campaign report, e.g. `versions-2026-07-28.md`. + """ + def report, do: @report + + @doc """ + Date of the newest campaign report, parsed from its filename. + """ + def report_date, do: @report_date + + @doc """ + Log-scale x position, as a 0..100 percentage, for a ratio in the dot plot. + """ + def plot_x(ratio) when is_number(ratio) do + :math.log(ratio / xmin()) / :math.log(xmax() / xmin()) * 100 + end + + def xmin, do: 0.15 + def xmax, do: 2.1 + + @doc """ + Gridline/tick positions for the plot axis. + """ + def ticks, do: [{0.25, "4× faster"}, {0.5, "2× faster"}, {1.0, "parity"}, {2.0, "2× slower"}] + + # A ratio below parity, restated as the speedup it represents. + defp inverse(ratio) when is_number(ratio) and ratio > 0, do: format_factor(1 / ratio) + defp inverse(_ratio), do: nil + + # Large factors read better whole ("63×"); small ones need the decimal ("1.7×"). + defp format_factor(factor) when factor >= 10, do: factor |> round() |> Integer.to_string() + defp format_factor(factor), do: :erlang.float_to_binary(factor, decimals: 1) +end diff --git a/website/lib/website_web/components/layouts.ex b/website/lib/website_web/components/layouts.ex index f711bfa3..664cbab1 100644 --- a/website/lib/website_web/components/layouts.ex +++ b/website/lib/website_web/components/layouts.ex @@ -63,6 +63,7 @@ defmodule DemoWeb.Layouts do <.nav_link href="/playground" active={@active == :playground}>Playground <.nav_link href="/tour" active={@active == :tour}>Tour <.nav_link href="/reference/opcodes" active={@active == :opcodes}>Opcodes + <.nav_link href="/benchmarks" active={@active == :benchmarks}>Benchmarks <.nav_link href="/about" active={@active == :about}>About +
  • + <.link + navigate="/benchmarks" + class={@active == :benchmarks && "active text-primary bg-primary/10"} + > + Benchmarks + +
  • <.link navigate="/about" @@ -221,6 +230,11 @@ defmodule DemoWeb.Layouts do Opcode reference
    • +
  • + <.link navigate={~p"/benchmarks"} class="text-base-content/70 hover:text-primary"> + Benchmarks + +
  • <.link navigate={~p"/about"} class="text-base-content/70 hover:text-primary"> About diff --git a/website/lib/website_web/controllers/page_controller.ex b/website/lib/website_web/controllers/page_controller.ex index 89060600..44e9c617 100644 --- a/website/lib/website_web/controllers/page_controller.ex +++ b/website/lib/website_web/controllers/page_controller.ex @@ -1,6 +1,8 @@ defmodule DemoWeb.PageController do use DemoWeb, :controller + alias Website.Benchmarks + def home(conn, _params) do %{source: fib_source} = hd(Website.LuaSandbox.home_snippets()) @@ -15,6 +17,19 @@ defmodule DemoWeb.PageController do render(conn, :about, page_title: "About") end + def benchmarks(conn, _params) do + render(conn, :benchmarks, + page_title: "Benchmarks", + versions: Benchmarks.versions(), + latest: Benchmarks.latest(), + rows: Benchmarks.rows(), + chart_rows: Benchmarks.chart_rows(), + headline: Benchmarks.headline(), + report: Benchmarks.report(), + report_date: Benchmarks.report_date() + ) + end + def health(conn, _params) do send_resp(conn, 200, "ok") end diff --git a/website/lib/website_web/controllers/page_html.ex b/website/lib/website_web/controllers/page_html.ex index 0d04475d..1ff41041 100644 --- a/website/lib/website_web/controllers/page_html.ex +++ b/website/lib/website_web/controllers/page_html.ex @@ -6,8 +6,56 @@ defmodule DemoWeb.PageHTML do """ use DemoWeb, :html + alias Website.Benchmarks + embed_templates "page_html/*" + attr :label, :string, required: true + attr :value, :string, required: true + attr :delta, :string, default: nil + slot :inner_block, required: true + + def stat_tile(assigns) do + ~H""" +
    +
    + {@label} +
    +
    {@value}
    +
    {@delta}
    +
    + {render_slot(@inner_block)} +
    +
    + """ + end + + attr :value, :float, default: nil + + def ratio(assigns) do + ~H""" + — + + {:erlang.float_to_binary(@value, decimals: 2)}× + + """ + end + + @doc """ + Hover text for a dot in the ratio plot. + """ + def dot_title(row, version) do + values = row.values[version] + + """ + #{row.label} — #{version} + chunk median: #{values.value} + same-run Luerl: #{values.control} + ratio: #{values.ratio}× #{ratio_word(values.ratio)} + """ + |> String.trim() + end + attr :icon, :string, required: true attr :title, :string, required: true attr :accent, :string, default: "primary" @@ -30,6 +78,10 @@ defmodule DemoWeb.PageHTML do """ end + defp ratio_word(ratio) when ratio < 1, do: "(faster)" + defp ratio_word(ratio) when ratio > 1, do: "(slower)" + defp ratio_word(_ratio), do: "" + defp accent_bg("primary"), do: "bg-primary/15" defp accent_bg("secondary"), do: "bg-secondary/15" defp accent_bg("accent"), do: "bg-accent/15" diff --git a/website/lib/website_web/controllers/page_html/benchmarks.html.heex b/website/lib/website_web/controllers/page_html/benchmarks.html.heex new file mode 100644 index 00000000..e1b75794 --- /dev/null +++ b/website/lib/website_web/controllers/page_html/benchmarks.html.heex @@ -0,0 +1,280 @@ + + <%!-- ============== HERO ============== --%> +
    + + +
    +

    + tv-labs/lua · full-mode benchee · {@report_date} +

    +

    + Lua on the BEAM: {Enum.join(@versions, " → ")} +

    +

    + {length(@versions)} releases of the lua + Elixir library, measured on the same machine in the same sitting, with Luerl run inside + every benchmark as a same-run control. {List.first(@versions)} was + a thin wrapper over Luerl; {@headline.previous} introduced the native VM; {@latest} is the + current release. +

    +

    + The headline: + {@latest} is the first release faster than Luerl on most workloads — and + Lua.new() + is now effectively free. +

    +
    +
    + + <%!-- ============== HEADLINE TILES ============== --%> +
    +
    + <.stat_tile + label="Lua.new() median" + value={@headline.new_median} + delta={"#{@headline.new_speedup}× faster than #{@headline.previous} (#{@headline.new_median_prev})"} + > + Steady-state, default options, under mix run. ~100× credible in an + :embedded + release; ~5× when passing custom sandbox options. + + + <.stat_tile + label="Lua.new() allocation" + value={@headline.new_memory} + delta={"#{@headline.new_memory_factor}× less than #{@headline.previous} (#{@headline.new_memory_prev})"} + > + Memoized boot-time VM template; one-time ~7 ms cold build per node. + + + <.stat_tile + label="fibonacci fib(30) median" + value={@headline.fib_median} + delta={"#{@headline.fib_speedup}× faster than same-run Luerl"} + > + Was {@headline.fib_prev_ratio}× slower + on {@headline.previous} ({@headline.fib_median_prev}). Allocation: {@headline.fib_memory} vs Luerl's {@headline.fib_memory_control}. + +
    +
    + + <%!-- ============== RATIO PLOT ============== --%> +
    +

    Runtime vs Luerl, per workload

    +

    + Each dot is a release's compiled-chunk median divided by the Luerl median from the same + run — left of the parity line is faster than Luerl. Hover a dot for the underlying + medians. Log scale. +

    + +
    +
    + + + + {version} +  (Luerl wrapper) + + + │ line = Luerl parity (1.0) +
    + +
    +
    +
    + ← faster than Luerl + slower than Luerl → +
    + +
    +
    + + {row.label} + + {row.sub} +
    + +
    +
    +
    +
    +
    + + + +
    +
    + +
    + +
    + + {label} + +
    +
    +
    +
    +
    +
    + + <%!-- ============== MEDIANS TABLE ============== --%> +
    +

    Medians across releases

    +

    + Compiled-chunk path (the production embedding path: compile once, run many). Ratio column + is {@latest} ÷ same-run Luerl; green means faster than Luerl. +

    + +
    + + + + + + + + + + + + + + + +
    Workload{version}{@latest} vs Luerl
    + + {row.label} + + + + {Benchmarks.value(row, version) || "—"} + <.ratio value={Benchmarks.ratio(row, @latest)} />
    +
    + +

    + † {@latest} builds PUC-Lua-conformant position-prefixed error messages; Luerl returns the + bare error value, doing less work per raise. +

    +
    + + <%!-- ============== WHERE WE'RE BEHIND ============== --%> +
    +
    +

    Where {@latest} is still behind Luerl

    +
    +

    + + pcall raise + catch ({@headline.raise_ratio}× slower). + + Partly apples-to-oranges: {@latest} builds position-prefixed error messages + ("<eval>:1: negative", the PUC-Lua-conformant behavior) where Luerl + returns the bare value — it does strictly more work per raise. Still, it regressed at {@headline.previous} ({@headline.raise_prev_ratio}×) and has only partially recovered. +

    +

    + + varargs + multiple returns ({@headline.varargs_ratio}× slower, ~1.9× the allocation). + + A genuine gap with no conformance excuse — the clearest optimization target for 1.1.x. +

    +

    + Host-boundary decode of large lists. + Not in the chart (its own harness): decoding a 4,096-element integer list is ~13× slower + than the {List.first(@versions)}/Luerl era; long-string lists ~25×. String-keyed maps + moved the other way (1.24× faster). Worth a look before 1.1. +

    +
    +
    +
    + + <%!-- ============== HOW TO READ ============== --%> +
    +

    How to read these numbers

    +
      +
    • + + {List.first(@versions)} is not an independent series. + + It wraps luerl 1.5.1; its own no-sandbox instantiation lands within ~1% of raw :luerl.init(). Its dots hugging the parity line is by construction, and it's + why the Luerl control column is meaningful across all {length(@versions)} runs. +
      • +
    • + Discipline: + Apple M4 · Elixir 1.20.0 / OTP 29 · LUA_BENCH_MODE=full + (10 s measure, 2 s warmup, 1 s memory) · one mix run + at a time on a quiet machine · medians quoted, not averages. C Lua (luaport) was + unavailable; Luerl is the reference. +
      • +
    • + The suite got wider for this report. + patterns, metamethods, pcall/varargs, and Lua.new + workloads are new — chosen to make the suite representative of real Lua (pattern engine, + : + method dispatch, protected calls), not just of what was optimized. +
      • +
    • + Coroutines are not benchmarked + — they are an intentional 1.0 capability exclusion in the library, not an omission from + the suite. +
      • +
    • + Sub-microsecond rows + (Lua.new() on {@latest}) show large Benchee deviation percentages due to + batching at that timescale; medians are stable across runs. +
      • +
    + +

    + Full report with per-workload analysis and reproduction instructions: + + bench_results/{@report} + + in tv-labs/lua. Raw Benchee outputs and parsed JSON are committed alongside, one directory + per released version. +

    +
    + diff --git a/website/lib/website_web/router.ex b/website/lib/website_web/router.ex index 4569c206..dadfdbd5 100644 --- a/website/lib/website_web/router.ex +++ b/website/lib/website_web/router.ex @@ -23,6 +23,7 @@ defmodule DemoWeb.Router do get "/", PageController, :home get "/about", PageController, :about + get "/benchmarks", PageController, :benchmarks live "/playground", PlaygroundLive, :index live "/playground/:example", PlaygroundLive, :example live "/tour", TourLive, :index diff --git a/website/test/website_web/controllers/page_controller_test.exs b/website/test/website_web/controllers/page_controller_test.exs index 4a52cb44..d2600a3b 100644 --- a/website/test/website_web/controllers/page_controller_test.exs +++ b/website/test/website_web/controllers/page_controller_test.exs @@ -1,10 +1,68 @@ defmodule DemoWeb.PageControllerTest do use DemoWeb.ConnCase + alias Website.Benchmarks + test "GET / renders the Lua showcase landing page", %{conn: conn} do body = conn |> get(~p"/") |> html_response(200) assert body =~ "Lua, on the" assert body =~ "Playground" assert body =~ "Tour" end + + describe "GET /benchmarks" do + test "renders a column per recorded version", %{conn: conn} do + body = conn |> get(~p"/benchmarks") |> html_response(200) + + for version <- Benchmarks.versions() do + assert body =~ version + end + end + + test "renders every row's recorded value rather than a placeholder", %{conn: conn} do + body = conn |> get(~p"/benchmarks") |> html_response(200) + + for row <- Benchmarks.rows(), version <- Benchmarks.versions() do + value = Benchmarks.value(row, version) + + assert value, "#{row.label} has no value for #{version}" + assert body =~ value, "#{row.label}/#{version} (#{value}) missing from the page" + end + end + + test "links the campaign report it was written from", %{conn: conn} do + body = conn |> get(~p"/benchmarks") |> html_response(200) + + assert body =~ "bench_results/#{Benchmarks.report()}" + end + + test "is reachable from the site navigation", %{conn: conn} do + body = conn |> get(~p"/") |> html_response(200) + + assert body =~ ~p"/benchmarks" + end + end + + describe "recorded results" do + test "every row resolves against every recorded version" do + for row <- Benchmarks.rows(), version <- Benchmarks.versions() do + assert is_binary(Benchmarks.value(row, version)), + "#{row.label} did not resolve for #{version} — check the case-name normalisation" + end + end + + test "ratios are taken against the same-run control" do + for row <- Benchmarks.rows(), version <- Benchmarks.versions() do + assert is_float(Benchmarks.ratio(row, version)), + "#{row.label}/#{version} has no ratio — control job #{row.vs} missing?" + end + end + + test "versions are ordered oldest to newest" do + assert Benchmarks.versions() == + Enum.sort(Benchmarks.versions(), &(Version.compare(&1, &2) != :gt)) + + assert Benchmarks.latest() == List.last(Benchmarks.versions()) + end + end end