Skip to content

Add benchmark suite comparing lambda-api to other Lambda frameworks (#34)#327

Merged
naorpeled merged 3 commits into
jeremydaly:mainfrom
naorpeled:feat/benchmarks-issue-34
Jun 26, 2026
Merged

Add benchmark suite comparing lambda-api to other Lambda frameworks (#34)#327
naorpeled merged 3 commits into
jeremydaly:mainfrom
naorpeled:feat/benchmarks-issue-34

Conversation

@naorpeled

Copy link
Copy Markdown
Collaborator

Closes #34.

Adds a benchmark suite comparing lambda-api against other popular AWS Lambda web frameworks.

Approach: in-process micro-benchmarks (not LocalStack)

The signal we care about is framework overhead — parse event → route → middleware → serialize — which the issue clocks at ~0.68 ms. This suite invokes each framework's compiled aws-lambda handler in-process, in the same Node VM, with identical synthetic API Gateway events, and measures it with mitata.

LocalStack / a real Lambda deploy was deliberately avoided for the primary suite: Docker + runtime bootstrap + network add tens of ms of noise (10–100× the signal) and would measure the emulator, not the framework. End-to-end deployment timing is a different question and is documented as an appendix in benchmarks/README.md.

What's included

  • benchmarks/ — an isolated package with its own package.json, so the root library keeps its zero-dependency policy. It's excluded from the npm tarball by the existing files whitelist (verified with npm pack --dry-run).
    • run.js — the harness (mitata; a correctness gate validates each cell's status/body before timing, so no framework is ever benchmarked while silently 404-ing or throwing)
    • frameworks/baseline (raw handler), lambda-api, serverless-express (Express), fastify, hono, behind a uniform { name, version, build } contract
    • lib/ — v1/v2 event builders (seeded from the repo's __tests__/sample-event-* shapes), scenarios, validator, markdown renderer, README writer
  • Scenarios: get-json, path-param, post-json, routing-50 (routing cost), not-found — each across API Gateway REST (v1) and HTTP API (v2).
  • .github/workflows/benchmark.yml — runs on release published and manual dispatch (Node 20); refreshes the README Benchmarks section, Prettier-formats it, commits back to the default branch with [skip ci], and uploads results as an artifact. Not wired into PR CI, since perf is noisy on shared runners.

README updates on every release

The README gains a marker-delimited (<!-- BENCHMARKS:START/END -->) Benchmarks section that's regenerated in place (idempotent) with a compact one-row-per-release history table appended. Move the marker block anywhere once; future runs respect its position.

How to run locally

cd benchmarks
npm install
npm run bench

Notes for review

  • The committed numbers are an example run on my machine (Apple M4 Max, Node 20) at version 0.0.0-development; the workflow refreshes both the numbers and the version (from the release tag) on each release.
  • Added a standard catch-all 404 handler to the Express setup — without it, unmatched routes hit Express's finalhandler, which on-finished can't attach to serverless-express's mock socket (it 500s). This mirrors the built-in 404 of the other frameworks.
  • benchmarks/ is added to .eslintignore/.prettierignore so the suite's dynamic import() (ESM-only hono/mitata) doesn't trip the root's ES2018 lint config.

Existing test suite (480 tests) and tsd types pass; root ESLint/Prettier are clean.

…eremydaly#34)

In-process micro-benchmarks that invoke each framework's aws-lambda handler
in the same Node VM with identical synthetic API Gateway (v1 + v2) events,
measuring framework overhead with mitata. A correctness gate validates every
cell before timing.

Frameworks: baseline (raw handler), lambda-api, @vendia/serverless-express,
@fastify/aws-lambda, hono. Scenarios: get-json, path-param, post-json,
routing-50, not-found.

Lives in an isolated benchmarks/ package so the root library keeps its
zero-dependency policy; excluded from the npm tarball by the files whitelist.

A manual/release GitHub workflow refreshes the README Benchmarks section
(marker-delimited, idempotent, with a per-release history row) on each
published release.
The @author field held the project description instead of an author name,
which read oddly in every file. The descriptive block comment is the useful
part; the boilerplate banner added nothing to these new files.
…y#34 references

- Add a middy framework (@middy/core + http-router + http-json-body-parser +
  http-error-handler) to the comparison set and README history. Middy is a
  middleware engine, so it's wired with the official router/body-parser/error
  handler and otherwise kept a thin layer; documented in the fairness notes.
- Bump the benchmarks Node floor to >=20 (Middy 6 requires it; matches the
  workflow and Fastify 5).
- Remove the redundant 'issue jeremydaly#34' mentions from the benchmark README, run.js,
  and package.json description.
@naorpeled naorpeled merged commit 557bc2d into jeremydaly:main Jun 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark tests

1 participant