llm-serving-benchmark

The cheapest configuration in the table misses the SLO. int4-2x at 7 rps costs $0.56 per million tokens with a p99 of 3.88s against a 2s target. Backing off to 5 rps costs $0.71 — 27% more — and lands at 1.44s.

Sized on cost per token, the first row is the obvious pick. It is the wrong one.

This repo answers one question:

Which serving config meets a stated latency target at the lowest cost per million tokens?

The answer

At a 2s p99 SLO, requests per second that finish inside it:

config	load	util	p99	goodput	$/1M	quality
int4-2x	5	0.61	1.44s	4.99	0.71	0.961
int4-2x	6	0.73	2.12s	5.92	0.60	0.961
int4-2x	7	0.85	3.88s	6.33	0.56	0.961
awq-2x	4	0.59	1.64s	3.99	0.89	0.987
awq-2x	5	0.74	2.59s	4.86	0.73	0.987
fp16-4x	5	0.62	1.66s	4.98	1.43	1.000
fp16-4x	6	0.75	2.47s	5.85	1.22	1.000

Bold p99 exceeds the SLO. Full sweep across six configs: results/goodput.md.

Quality sits in the same table as the price on purpose. int4-2x is the cheapest compliant option and gives up 3.9 points of quality against fp16. That is a product decision, and separating the two tables is how it gets made by accident.

Why throughput is the wrong headline

Throughput rises monotonically with load until saturation. Goodput does not — it rises while the queue is short, turns over as latency approaches the deadline, then collapses to zero when the queue becomes unstable.

config	peak goodput	at load	at max stable load
fp16-1x	0.89	1 rps	0.89 at 1 rps
awq-1x	1.89	2 rps	1.66 at 3 rps
awq-2x	4.86	5 rps	4.73 at 6 rps
int4-2x	6.33	7 rps	6.33 at 7 rps

awq-1x is the clearest case: peak goodput at 2 rps, declining by 3 rps even though throughput is still climbing. The two curves point in opposite directions exactly where capacity decisions get made.

An unstable queue reports goodput 0.00, not a throughput figure. A server past its stability limit completes nothing on time however many tokens it emits, and reporting throughput there is the specific error this repo argues against.

How much of this is the exponential assumption?

The model is M/M/c, so service times are exponential and their coefficient of variation is exactly 1. LLM service time tracks output length, and output length is a mixture — most requests short, a minority very long — which puts the coefficient well above 1. Queueing delay scales with the square of variability, so this is not a rounding concern.

Applying the Allen-Cunneen correction, which returns exactly 1.0 at Cs = Ca = 1 and therefore treats the tables above as its first column:

config @ 5 rps	exponential	mixed-length (Cs 1.6)	heavy-tail (Cs 2.4)
int4-2x	1.44s	2.38s	4.29s
awq-2x	2.59s	4.38s	8.05s
fp16-4x	1.66s	2.56s	4.40s

int4-2x at 5 rps reports 1.44s under M/M/c and 2.38s under a realistic length mixture. The first number is comfortably inside the 2s SLO. The second is not — and the capacity plan built on the first was never real.

Compliant capacity, and what a heavy tail removes:

config	exponential	mixed	heavy-tail	capacity lost
int4-2x	5.8 rps	4.4	2.7	53%
awq-2x	4.4 rps	3.2	1.8	59%
fp16-4x	5.5 rps	4.3	3.1	44%
awq-1x	1.2 rps	0.5	0.1	92%

Single-replica configs suffer most: there is no second slot to absorb a long request, so they stop qualifying rather than merely degrading.

The winner is stable; the capacity plan is not

assumption	cheapest compliant	at load	$/1M
exponential	int4-2x	5 rps	0.71
mixed-length	int4-2x	4 rps	0.89
heavy-tail	int4-2x	2 rps	1.78

int4-2x wins under every assumption, so the recommendation is a property of the configs rather than of the tail. But its compliant load falls from 5 rps to 2 and its price rises 2.5x. Those are different claims, and the headline table only supports the first.

Second place does not survive. Under exponential service awq-2x is cheaper than fp16-4x; under a heavy tail they swap, and awq-1x and fp16-1x drop out entirely — unable to meet the SLO at any load, which is a different answer from being expensive.

Bursty arrivals compound it, because Ca and Cs enter the same correction and multiply: at the same mean rate, bursty traffic pushes the mixed-length optimum from 4 rps at $0.89 to 3 rps at $1.20.

Full tables: results/variability.md. ADR 001 records why capacity figures now name the tail assumption they were computed under.

Recommendation

• Size on goodput at your p99 target, then compare cost. Not the other way round, and never on tokens per second alone.
• Stay off the steep part of the curve. Above roughly 0.8 utilisation a small traffic increase produces a large latency increase — awq-2x goes from 1.64s at 0.59 utilisation to 5.81s at 0.88. The headroom is the product, not waste.
• Price the SLO explicitly. Here it is 27% over the cheapest non-compliant option. That number belongs in the capacity conversation up front rather than being discovered during an incident.

Limitations

• This is a queueing model, not a benchmark. No server was run and nothing was timed. Service rates, prices and quality figures are stated parameters chosen to be plausible for a mid-size open-weights model.
• M/M/c assumes exponential service times, and real LLM service time is heavier-tailed. Quantified rather than conceded: a realistic length mixture inflates p99 by roughly 1.6x and removes about half the compliant capacity. The cheapest config survives that; its capacity plan and the second-place ranking do not. The Allen-Cunneen correction used there is itself an approximation, so treat those magnitudes as indicative and the direction as reliable.
• Continuous batching is not modelled. A real server interleaves prefill and decode across requests, raising throughput and complicating the latency distribution beyond what a closed form captures.
• Quality figures are illustrative. Measuring quantisation degradation properly needs a task, a dataset and a calibrated judge — that is quantization-accuracy-curves, not this repo.

Reproduce

uv sync --extra dev
uv run python scripts/generate_results.py
uv run python scripts/generate_variability.py
uv run pytest

Closed-form arithmetic — identical output on any machine. CI regenerates the results and fails on any diff.

License

MIT