Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Hi, thank you for visiting or contributing to our project, we are always looking

HTTP framework benchmark platform.

29 test profiles. 64-core dedicated hardware. Same conditions for every framework.
30 test profiles. 64-core dedicated hardware. Same conditions for every framework.

[View Leaderboard](https://www.http-arena.com/) | [Documentation](https://www.http-arena.com/#doc=) | [Add a Framework](https://www.http-arena.com/#doc=add-framework)

Expand Down Expand Up @@ -44,7 +44,7 @@ The reply states which baseline it used, and profiles the other framework does n
| Category | Profiles | Description |
|----------|----------|-------------|
| Connection | `baseline`, `pipelined` *, `limited-conn` | Mixed GET/POST with query parsing (512/4K conns), 16× batched pipelining (reference-only, shown faded, excluded from the composite score), short-lived connections that close after 10 requests |
| Workload | `json`, `json-comp`, `json-tls`, `upload`, `static` | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, 20 MB body ingestion, 20-file static asset serving |
| Workload | `json`, `json-comp`, `json-tls`, `upload`, `static`, `static-tls` | JSON serialization, gzip/brotli compression, HTTP/1.1 over TLS, 20 MB body ingestion, 20-file static asset serving (plaintext and TLS) |
| Database | `async-db`, `crud` | Async Postgres sequential scan; realistic REST API with cached reads, list, upsert, update, and optional Redis cache |
| Templates | `fortunes` * | DB query + HTML template render (TechEmpower-style Fortunes). Reference-only — measures template-engine throughput, not part of the composite score |
| Multi-endpoint | `api-4`, `api-16` | Mixed baseline + JSON + async-db at CPU-budget cliffs (4 and 16 logical CPUs, i.e. 2 and 8 full SMT cores) |
Expand Down
1 change: 1 addition & 0 deletions frameworks/genhttp-11/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
"async-db",
"crud",
"static",
"static-tls",
"echo-ws",
"echo-ws-pipeline",
"echo-ws-limited",
Expand Down
8 changes: 4 additions & 4 deletions scripts/benchmark-lite.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@
# • Threads default to nproc/2 (not 64), so a 4-core laptop gets 2
# threads instead of 64.
# • Fixed, reasonable connection counts per profile (mostly 512).
# • Smaller profile subset — skips api-4/16, json-tls, gateway-64,
# stream-grpc/stream-grpc-tls (they need either specific hardware
# topology or extra setup).
# • Smaller profile subset — skips api-4/16, json-tls, static-tls,
# gateway-64, stream-grpc/stream-grpc-tls (they need either specific
# hardware topology or extra setup).
# • --load-threads <N> override THREADS/H2THREADS/H3THREADS in one shot.
#
# The pre-refactor version lives at scripts/old/benchmark-lite-old.sh.
Expand Down Expand Up @@ -69,7 +69,7 @@ source "$SOURCE_DIR/tools/ghz.sh"
# Differences vs the full set:
# • cpu_limit is always empty (no pinning, container gets all cores)
# • conn_list is one fixed value per profile (no 256,1024 sweeps)
# • skipped profiles: api-4, api-16, json-tls, gateway-64, stream-grpc*
# • skipped profiles: api-4, api-16, json-tls, static-tls, gateway-64, stream-grpc*

unset PROFILES PROFILE_ORDER
declare -A PROFILES=(
Expand Down
2 changes: 1 addition & 1 deletion scripts/benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ FRAMEWORK="$FRAMEWORK_ARG"
# and any combination thereof.
_has_isolated_test=false
for t in baseline pipelined limited-conn json json-comp json-tls upload \
api-4 api-16 static async-db \
api-4 api-16 static static-tls async-db \
baseline-h2 static-h2 baseline-h2c json-h2c \
baseline-h3 static-h3 \
unary-grpc unary-grpc-tls stream-grpc stream-grpc-tls \
Expand Down
2 changes: 2 additions & 0 deletions scripts/gen_leaderboard_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
("json-tls", "JSON TLS", "JSON over HTTP/1.1 + TLS.", [4096], [4096], True,True),
("upload", "Upload", "Large request-body ingestion.", [32,64,256,512], [32,256], True,False),
("static", "Static", "20-file static asset serving.", [1024,4096,6800,16384],[1024,4096,6800],True,False),
("static-tls","Static TLS", "20-file static serving over TLS.", [1024,4096,6800], [1024,4096,6800],True,False),
]),
("Database", [
("async-db", "Async DB", "Async Postgres sequential scan.", [1024], [1024], True,True),
Expand Down Expand Up @@ -100,6 +101,7 @@
"json-tls": "test-profiles/h1/isolated/json-tls/implementation",
"upload": "test-profiles/h1/isolated/upload/implementation",
"static": "test-profiles/h1/isolated/static/implementation",
"static-tls": "test-profiles/h1/isolated/static-tls/implementation",
"async-db": "test-profiles/h1/isolated/async-database/implementation",
"crud": "test-profiles/h1/isolated/crud/implementation",
"fortunes": "test-profiles/h1/isolated/fortunes/implementation",
Expand Down
3 changes: 3 additions & 0 deletions scripts/lib/framework.sh
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,9 @@ framework_wait_ready() {
static)
probe_url="http://localhost:$PORT/static/reset.css"
;;
static-tls)
probe_url="https://localhost:$H1TLS_PORT/static/reset.css"
;;
json)
probe_url="http://localhost:$PORT/json/1"
;;
Expand Down
5 changes: 3 additions & 2 deletions scripts/lib/profiles.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ declare -A PROFILES=(
[api-4]="1|5|0-1,64-65|256|api-4"
[api-16]="1|5|0-7,64-71|1024|api-16"
[static]="1|200|0-31,64-95|1024,4096,6800|static"
[static-tls]="1|200|0-31,64-95|1024,4096,6800|static-tls"
[async-db]="1|0|0-31,64-95|1024|async-db"
[crud]="1|200|1-31,65-95|4096|crud"
[fortunes]="1|0|0-31,64-95|1024|fortunes"
Expand All @@ -45,7 +46,7 @@ PROFILE_ORDER=(
baseline pipelined limited-conn
json json-comp json-tls
upload api-4 api-16
static async-db crud
static static-tls async-db crud
fortunes
baseline-h2 static-h2
baseline-h2c json-h2c
Expand Down Expand Up @@ -76,7 +77,7 @@ parse_profile() {
endpoint_tool() {
case "$1" in
# wrk (lua script rotation)
static|json-tls) echo "wrk" ;;
static|static-tls|json-tls) echo "wrk" ;;
# h2load for all HTTP/2 variants (TLS via ALPN + h2c prior-knowledge)
h2|static-h2|h2c|json-h2c|gateway-64|grpc|grpc-tls|production-stack) echo "h2load" ;;
# h2load built with ngtcp2 for HTTP/3
Expand Down
5 changes: 5 additions & 0 deletions scripts/lib/tools/wrk.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ wrk_build_args() {
-s "$REQUESTS_DIR/static-rotate.lua"
"http://localhost:$PORT")
;;
static-tls)
cmd+=(-t "$THREADS" -c "$conns" -d "$duration"
-s "$REQUESTS_DIR/static-rotate.lua"
"https://localhost:$H1TLS_PORT")
;;
json-tls)
cmd+=(-t "$THREADS" -c "$conns" -d "$duration"
-s "$REQUESTS_DIR/json-tls-rotate.lua"
Expand Down
82 changes: 80 additions & 2 deletions scripts/validate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ if has_test "baseline-h2" || has_test "static-h2" || has_test "baseline-h3" || h
fi

needs_h1tls=false
if has_test "json-tls"; then
if has_test "json-tls" || has_test "static-tls"; then
needs_h1tls=true
fi

Expand All @@ -183,7 +183,7 @@ if has_test "gateway-64" || has_test "gateway-h3"; then
docker_args+=(-v "$DATA_DIR/dataset-large.json:/data/dataset-large.json:ro")
fi

if has_test "static" || has_test "static-h2" || has_test "static-h3" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack"; then
if has_test "static" || has_test "static-tls" || has_test "static-h2" || has_test "static-h3" || has_test "gateway-64" || has_test "gateway-h3" || has_test "production-stack"; then
docker_args+=(-v "$DATA_DIR/static:/data/static:ro")
fi

Expand Down Expand Up @@ -1102,6 +1102,84 @@ if has_test "static"; then
fi


# ───── Static Files TLS (GET /static/* over HTTP/1.1 + TLS on :8081) ─────

if has_test "static-tls"; then
STATICTLS_DOCS="$DOCS_BASE/h1/isolated/static-tls/validation"
echo "[test] static-tls endpoint"

# Must negotiate HTTP/1.1 (not h2) via ALPN on :8081
stls_proto=$(curl -sk --max-time 30 --http1.1 -o /dev/null -w '%{http_version}' "https://localhost:$H1TLS_PORT/static/reset.css" 2>/dev/null || echo "0")
if [ "$stls_proto" = "1.1" ]; then
echo " PASS [static-tls protocol negotiation] (HTTP/$stls_proto over TLS)"
PASS=$((PASS + 1))
else
fail_with_link "[static-tls protocol negotiation]: expected 1.1, got HTTP/$stls_proto" "$STATICTLS_DOCS"
fi

check_header "GET /static/reset.css Content-Type (TLS)" "Content-Type" "text/css" "$STATICTLS_DOCS" \
-sk "https://localhost:$H1TLS_PORT/static/reset.css"

check_header "GET /static/app.js Content-Type (TLS)" "Content-Type" "application/javascript" "$STATICTLS_DOCS" \
-sk "https://localhost:$H1TLS_PORT/static/app.js"

check_header "GET /static/manifest.json Content-Type (TLS)" "Content-Type" "application/json" "$STATICTLS_DOCS" \
-sk "https://localhost:$H1TLS_PORT/static/manifest.json"

# Verify file sizes match actual files on disk
stls_fail=false
for sf in reset.css layout.css theme.css components.css utilities.css analytics.js helpers.js app.js vendor.js router.js header.html footer.html regular.woff2 bold.woff2 logo.svg icon-sprite.svg hero.webp thumb1.webp thumb2.webp manifest.json; do
expected_size=$(wc -c < "$DATA_DIR/static/$sf" 2>/dev/null || echo "0")
actual_size=$(curl -sk --max-time 30 -o /dev/null -w '%{size_download}' "https://localhost:$H1TLS_PORT/static/$sf" || echo "0")
if [ "$actual_size" -eq "$expected_size" ] 2>/dev/null; then
true
else
fail_with_link "[static-tls/$sf size]: expected $expected_size bytes, got $actual_size" "$STATICTLS_DOCS"
stls_fail=true
fi
done
if [ "$stls_fail" = "false" ]; then
echo " PASS [static-tls file sizes] (20 files verified)"
PASS=$((PASS + 1))
fi

# Verify compression works when Accept-Encoding is sent — for each file, if server compresses, decompressed size must match original
stls_comp_fail=false
stls_comp_count=0
stls_comp_skip=0
for sf in reset.css layout.css theme.css components.css utilities.css analytics.js helpers.js app.js vendor.js router.js header.html footer.html regular.woff2 bold.woff2 logo.svg icon-sprite.svg hero.webp thumb1.webp thumb2.webp manifest.json; do
expected_size=$(wc -c < "$DATA_DIR/static/$sf" 2>/dev/null || echo "0")
_hdr_tmp=$(mktemp)
_body_tmp=$(mktemp)
curl -sk --max-time 30 --compressed -D "$_hdr_tmp" -o "$_body_tmp" "https://localhost:$H1TLS_PORT/static/$sf" || true
comp_enc=$(grep -i "^content-encoding:" "$_hdr_tmp" | sed 's/^[^:]*: *//' | tr -d '\r' | awk '{print tolower($1)}' || true)
decompressed=$(wc -c < "$_body_tmp")
rm -f "$_hdr_tmp" "$_body_tmp"
if [ -n "$comp_enc" ]; then
if [ "$decompressed" -eq "$expected_size" ] 2>/dev/null; then
stls_comp_count=$((stls_comp_count + 1))
else
fail_with_link "[static-tls/$sf compression]: Content-Encoding: $comp_enc but decompressed size $decompressed != expected $expected_size" "$STATICTLS_DOCS"
stls_comp_fail=true
fi
else
stls_comp_skip=$((stls_comp_skip + 1))
fi
done
if [ "$stls_comp_fail" = "false" ]; then
if [ "$stls_comp_count" -gt 0 ]; then
echo " PASS [static-tls compression] ($stls_comp_count files compressed, $stls_comp_skip skipped)"
PASS=$((PASS + 1))
else
echo " SKIP [static-tls compression] (server does not compress static files)"
fi
fi

check_status "GET /static/nonexistent.txt (TLS)" "404" "$STATICTLS_DOCS" \
-sk "https://localhost:$H1TLS_PORT/static/nonexistent.txt"
fi


# ───── Static Files H2 (GET /static/* over HTTP/2 + TLS) ─────

if has_test "static-h2"; then
Expand Down
1 change: 1 addition & 0 deletions site/content/docs/test-profiles/h1/isolated/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Single-endpoint benchmarks that measure framework performance on one task at a t
{{< card link="upload" title="Upload (20 MB)" subtitle="Sends a 20 MB binary payload, server returns byte count. Measures body ingestion throughput." icon="cloud-upload" >}}
{{< card link="async-database" title="Async Database (Postgres)" subtitle="Async Postgres range query over 100K rows, connection pooling, and JSON serialization. Framework-only benchmark." icon="database" >}}
{{< card link="static" title="Static Files" subtitle="Serves 20 static files - CSS, JS, HTML, fonts, images - over HTTP/1.1." icon="photograph" >}}
{{< card link="static-tls" title="Static Files over TLS" subtitle="Same 20-file static workload transported over HTTP/1.1 + TLS on port 8081 - measures the cost of encryption on bulk file serving." icon="lock-closed" >}}
{{< card link="pipelined" title="Pipelined (16x) *" subtitle="16 requests sent back-to-back per connection, testing raw I/O and pipeline batching. Reference-only - not part of the composite score." icon="fast-forward" >}}
{{< card link="crud" title="CRUD (REST API)" subtitle="Realistic REST API with paginated list, cached reads, create, and update against Postgres." icon="database" >}}
{{< card link="fortunes" title="Fortunes (Templates) *" subtitle="DB query + HTML template rendering with auto-escape. Reference-only - measures template-engine throughput, not part of the composite score." icon="document-text" >}}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: Static Files (HTTP/1.1 + TLS)
seo_title: "Static File Serving Benchmark (HTTP/1.1 + TLS)"
description: "Serves twenty static files of mixed type and size over HTTP/1.1 + TLS, approximating a realistic page load over an encrypted connection."
---

Serves 20 static files of various types and sizes over HTTP/1.1 + TLS on port 8081, simulating a realistic page load over an encrypted connection.

{{< cards >}}
{{< card link="implementation" title="Implementation Guidelines" subtitle="Endpoint specification, expected request/response format, and type-specific rules." icon="code" >}}
{{< card link="validation" title="Validation" subtitle="All checks executed by the validation script for this test profile." icon="check-circle" >}}
{{< /cards >}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: Implementation Guidelines
seo_title: "Static File Serving Benchmark (HTTP/1.1 + TLS) — Implementation Guide"
description: "Endpoint contract, request and response shapes, and the anti-cheat constraints a framework must satisfy for the HTTP/1.1 + TLS static file benchmark."
---
{{< type-rules standard="Must load files from disk on every request. No in-memory caching, no memory-mapped files, no pre-loaded file buffers. Compression must use the framework's standard middleware or built-in static file handler - no handmade compression code. Serving pre-compressed `.br`/`.gz` variants from disk **is allowed**, but only through a documented framework API (e.g. ASP.NET `MapStaticAssets`, nginx `gzip_static` / `brotli_static`, Caddy `precompressed`). No custom file-suffix lookup logic. TLS must come from a standard stack (OpenSSL, BoringSSL, rustls, SChannel, JDK JSSE, etc.) - no TLS session-ticket shortcuts that skip real handshakes." tuned="May cache files in memory at startup, use memory-mapped files, pre-rendered response headers, or any caching strategy. May serve pre-compressed files (.gz, .br) from disk via any mechanism. Free to use any compression approach and tuned TLS providers." engine="No specific rules." >}}

The Static Files over TLS profile is the [Static Files](../static/implementation/) workload transported over HTTP/1.1 + TLS on a dedicated port. It measures how much of a framework's plaintext static-serving throughput survives encryption.

**Connections:** 1,024, 4,096, 6,800

## How it works

1. The framework serves the same 20 files from `/data/static/` as the plain `static` profile
2. The framework listens on **port 8081** with HTTPS, serving HTTP/1.1 only (ALPN advertises `http/1.1`)
3. The load generator ([wrk](https://github.com/wg/wrk)) requests the 20 files in a round-robin pattern using the same Lua rotation script as `static` (`requests/static-rotate.lua`), over `https://`
4. All requests include `Accept-Encoding: br;q=1, gzip;q=0.8`

- **CSS** (5 files, 8–200 KB): `reset.css`, `layout.css`, `theme.css`, `components.css`, `utilities.css`
- **JavaScript** (5 files, 12–300 KB): `analytics.js`, `helpers.js`, `app.js`, `vendor.js`, `router.js`
- **HTML** (2 files, 55–120 KB): `header.html`, `footer.html`
- **Fonts** (2 files, 18–22 KB): `regular.woff2`, `bold.woff2`
- **SVG** (2 files, 15–70 KB): `logo.svg`, `icon-sprite.svg`
- **Images** (3 files, 6–45 KB): `hero.webp`, `thumb1.webp`, `thumb2.webp`
- **JSON** (1 file, 3 KB): `manifest.json`

Total payload: ~842 KB across 20 files (~743 KB compressible text + ~99 KB binary). Brotli-compressed total: ~219 KB.

Pre-compressed versions of all text files (`.gz` at level 9, `.br` at level 11) are available in the `data/static/` directory alongside the originals.

## Compression

Identical to the plain [`static` profile](../static/implementation/#compression): compression is **optional**, must come from standard middleware / documented pre-compressed-file APIs on **production** entries, and is unrestricted on **tuned** entries.

## Port, ALPN, and certificates

- **Port**: 8081 (distinct from 8080 plaintext and 8443 which is dedicated to HTTP/2 / HTTP/3 profiles), shared with the `json-tls` profile
- **ALPN**: advertise `http/1.1` only. HTTP/1.1-only clients (wrk) negotiate correctly and never upgrade to h2.
- **Certificates**: the same PEM files used by `json-tls` / `baseline-h2` / `static-h2`, mounted at `/certs/server.crt` and `/certs/server.key`. Frameworks typically read them via environment variables (`TLS_CERT`, `TLS_KEY`) or a hardcoded path, same pattern as the other TLS tests.

## What it measures

- Everything [Static Files](../static/implementation/#what-it-measures) measures
- **TLS handshake cost amortized over keep-alive** - connections are long-lived at 1,024–6,800 concurrent
- **Record framing overhead on large payloads** - multi-hundred-KB responses span many TLS records, unlike the small JSON bodies of `json-tls`
- **Symmetric cipher throughput** - AES-GCM / ChaCha20-Poly1305 on the hot path, dominated by bulk encryption of file bodies

## Expected request/response

```
GET /static/reset.css HTTP/1.1
Host: localhost:8081
Accept-Encoding: br;q=1, gzip;q=0.8
```

```
HTTP/1.1 200 OK
Content-Type: text/css
Content-Encoding: br

(compressed file contents)
```

Or without compression:

```
HTTP/1.1 200 OK
Content-Type: text/css

(file contents)
```

## Parameters

| Parameter | Value |
|-----------|-------|
| Endpoint | 20 URIs under `/static/*` |
| Transport | HTTP/1.1 over TLS |
| Port | 8081 |
| ALPN | `http/1.1` |
| Connections | 1,024, 4,096, 6,800 |
| Pipeline | 1 |
| Duration | 5s |
| Runs | 3 (best taken) |
| Load generator | wrk + `requests/static-rotate.lua` |
| Certificates | mounted at `/certs/server.crt` + `/certs/server.key` |
Loading