Add benchmark suite comparing lambda-api to other Lambda frameworks (#34)#327
Merged
Merged
Conversation
…eremydaly#34) In-process micro-benchmarks that invoke each framework's aws-lambda handler in the same Node VM with identical synthetic API Gateway (v1 + v2) events, measuring framework overhead with mitata. A correctness gate validates every cell before timing. Frameworks: baseline (raw handler), lambda-api, @vendia/serverless-express, @fastify/aws-lambda, hono. Scenarios: get-json, path-param, post-json, routing-50, not-found. Lives in an isolated benchmarks/ package so the root library keeps its zero-dependency policy; excluded from the npm tarball by the files whitelist. A manual/release GitHub workflow refreshes the README Benchmarks section (marker-delimited, idempotent, with a per-release history row) on each published release.
The @author field held the project description instead of an author name, which read oddly in every file. The descriptive block comment is the useful part; the boilerplate banner added nothing to these new files.
…y#34 references - Add a middy framework (@middy/core + http-router + http-json-body-parser + http-error-handler) to the comparison set and README history. Middy is a middleware engine, so it's wired with the official router/body-parser/error handler and otherwise kept a thin layer; documented in the fairness notes. - Bump the benchmarks Node floor to >=20 (Middy 6 requires it; matches the workflow and Fastify 5). - Remove the redundant 'issue jeremydaly#34' mentions from the benchmark README, run.js, and package.json description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #34.
Adds a benchmark suite comparing lambda-api against other popular AWS Lambda web frameworks.
Approach: in-process micro-benchmarks (not LocalStack)
The signal we care about is framework overhead — parse event → route → middleware → serialize — which the issue clocks at ~0.68 ms. This suite invokes each framework's compiled
aws-lambdahandler in-process, in the same Node VM, with identical synthetic API Gateway events, and measures it with mitata.LocalStack / a real Lambda deploy was deliberately avoided for the primary suite: Docker + runtime bootstrap + network add tens of ms of noise (10–100× the signal) and would measure the emulator, not the framework. End-to-end deployment timing is a different question and is documented as an appendix in
benchmarks/README.md.What's included
benchmarks/— an isolated package with its ownpackage.json, so the root library keeps its zero-dependency policy. It's excluded from the npm tarball by the existingfileswhitelist (verified withnpm pack --dry-run).run.js— the harness (mitata; a correctness gate validates each cell's status/body before timing, so no framework is ever benchmarked while silently 404-ing or throwing)frameworks/—baseline(raw handler),lambda-api,serverless-express(Express),fastify,hono, behind a uniform{ name, version, build }contractlib/— v1/v2 event builders (seeded from the repo's__tests__/sample-event-*shapes), scenarios, validator, markdown renderer, README writerget-json,path-param,post-json,routing-50(routing cost),not-found— each across API Gateway REST (v1) and HTTP API (v2)..github/workflows/benchmark.yml— runs on releasepublishedand manual dispatch (Node 20); refreshes the README Benchmarks section, Prettier-formats it, commits back to the default branch with[skip ci], and uploads results as an artifact. Not wired into PR CI, since perf is noisy on shared runners.README updates on every release
The README gains a marker-delimited (
<!-- BENCHMARKS:START/END -->) Benchmarks section that's regenerated in place (idempotent) with a compact one-row-per-release history table appended. Move the marker block anywhere once; future runs respect its position.How to run locally
cd benchmarks npm install npm run benchNotes for review
0.0.0-development; the workflow refreshes both the numbers and the version (from the release tag) on each release.finalhandler, whichon-finishedcan't attach to serverless-express's mock socket (it 500s). This mirrors the built-in 404 of the other frameworks.benchmarks/is added to.eslintignore/.prettierignoreso the suite's dynamicimport()(ESM-onlyhono/mitata) doesn't trip the root's ES2018 lint config.Existing test suite (480 tests) and
tsdtypes pass; root ESLint/Prettier are clean.