Measure what your AI inference actually costs.
Most teams running AI in production know exactly one number: the monthly cloud bill. They can't tell you cost per endpoint, per feature, or per prediction. When the bill doubles, they can't say why.
This tells you.
Two ways to use it. Pick whichever you are comfortable with.
One file, no install. Standard library only. No network code in it — grep it. It writes JSON lines to a file on your own disk and nothing leaves the machine.
https://github.com/qaisermehdi3-coder/qvunex/blob/main/qvunex_single.py
Run it with no API key and no network first, to see what the output looks like:
python3 qvunex_single.py --demo
Or the package, if you would rather:
pip install qvunexfrom qvunex import meter
@meter("checkout-classifier")
def predict(batch):
return model(batch)qvunex report
qvunex checklistThat's the whole integration.
The server already counts every request it finishes. Save its counters before and after your traffic, then compare:
curl -s localhost:8000/metrics > before.txt
# ... run your traffic ...
curl -s localhost:8000/metrics > after.txt
python3 qvunex_single.py --reconcile before.txt after.txt --gpu-rate 0.83--gpu-rate is what the server costs per hour. You get three things:
- Meter against server: requests, prompt tokens and output tokens on both sides, and the gap. Streamed calls that came back with no usage are recovered as a total from the server's own count.
- Server speed: prompt reading and answer writing per request, and output for the whole server over the window.
- GPU cost: the hourly rate over the window, idle time included, per request and per million output tokens.
The file still opens no network connection: curl saves the counters and the file reads them.
Tested live on a Colab T4, vLLM 0.27.1, Qwen2.5-0.5B, 25 September 2026:
- Meter and server agreed exactly in all 8 windows. The decode token count matched vLLM's separate inter-token gap counter, 504 = 504.
- 8 requests one at a time against 8 at once, three rounds each: per-request decode 169-170 against 122-124 tokens/s, while the whole server went 151-154 against 701-708 tokens/s.
So price by the whole-server figure. Priced from per-request speed, that load looks about 5.7x more expensive than it was. And skip the first window after a server starts: in an earlier run, 24 requests 8 at a time took 12.8 s in the first window and 1.0 to 1.6 s in later windows on the same server.
Every wake is a cold start, and the boot is billed too. So take the first
snapshot as soon as the server answers, the second just before teardown, and
pass the whole billed time of the wake (create to delete) as --window, not
the time between the snapshots:
curl -s localhost:8000/metrics > before.txt # right after the health check
# ... serve this wake's requests ...
curl -s localhost:8000/metrics > after.txt # just before teardown
python3 qvunex_single.py --reconcile before.txt after.txt --gpu-rate 1.72 --window 161Tested live on a Colab T4, vLLM 0.27.1, 26 September 2026: one cold boot, 8 requests, clock started before the server launched. The counters already exist at boot, so the first snapshot works. Meter and server agreed exactly. The server took about 155 s to answer and served all 8 requests in the few seconds after, so about 96% of the paid time was boot: priced at $1.72/hour, $0.0096 per request. Priced from the snapshots alone, the same wake would look at least 14 times cheaper. The fewer requests a wake serves, the more the boot dominates.
Only vLLM's /metrics is read today. Other servers are not covered yet.
Wrap the client and name the unit of work. Two lines.
from qvunex import wrap, task
client = wrap(anthropic.Anthropic()) # or OpenAI
with task("outbound email"):
...your existing code, unchanged...wrap matters more than it looks. Decorating your own functions only sees the
calls you wrote; wrapping the client also catches the calls your framework makes
on your behalf — which is where the spend hides when one request fans out into
six sub-agent calls.
To see the output before wiring anything up — no API key, no network, no spend:
qvunex demo --apiOn your own workload, give it a rate card (copy tools/prices.example.txt to
~/.qvunex/prices.txt) and run qvunex report:
--------------------------------------------------------------------
COST PER FINISHED TASK
--------------------------------------------------------------------
task n mean p50 p95 calls
outbound email 12 $0.13 $0.13 $0.16 42.1
support triage 5 $0.02 $0.02 $0.02 1.0
retries 1 call(s), $0.03 the same work paid for twice
--------------------------------------------------------------------
COST BY ENDPOINT
--------------------------------------------------------------------
endpoint calls tokens $ share per call
classify intent 480 $1.04 62.0% $0.0022
research 12 $0.27 16.2% $0.02
draft 13 $0.27 16.0% $0.02
--------------------------------------------------------------------
WHAT THIS SUGGESTS
--------------------------------------------------------------------
* 'classify intent' is 63.4% of token spend while being the cheapest
single call you make ($0.0022). It is the 480 runs, not the price.
* 'classify intent' wrote 408,000 tokens to the prompt cache and read
none back. A write costs more than not caching at all, so that is a
pure surcharge.
Four things there are hard to get any other way:
- Cost per finished task, not per request. Per-task cost cannot be reconstructed afterwards from per-call billing data — the id has to be attached at call time or the number is a guess.
- The spread. The mean and the p95 are different questions. An average hides the tasks that actually hurt.
- Cache writes counted apart from reads. A write costs more than not caching (1.25x input on the 5-minute TTL, 2x on the hour); a read costs 0.1x. Folded into one "input tokens" figure, a cold run and a warm run look the same.
- Retries and fallbacks as their own line. A retry is the same work paid for twice. A fallback is a model you didn't ask for answering, at its price. Both are real spend that logging the successful attempt drops on the floor.
Providers disagree about whether reasoning tokens sit inside the output count or on top of it, and neither raises an error if you assume the wrong one.
Measured against gemini-3.6-flash, three calls, using Google's own reported
total:
17 + 144 + 589 = 750
4,632 + 26 + 236 = 4,894
4,632 + 32 + 238 = 4,902
total = prompt + output + thinking, exactly, every time. So on Google, thinking
is billed on top of the output count. OpenAI is the reverse — reasoning_tokens
is a subset of completion_tokens, and adding it again inflates the bill.
Anthropic folds thinking into output_tokens and never reports it apart.
On that first call thinking was 589 of 750 tokens. Anything reading the output count alone would show a fifth of what you were charged.
So the convention is recorded per call, not assumed globally, and pricing acts
on what was recorded. Reproduce it yourself with a free key:
tools/gemini_check.py.
Where a provider reports its own total, that is the one number qvunex did not
derive. Every call carries it, the parts are added up and compared, and a
mismatch prints UNACCOUNTED along with a line saying every cost below it is
short by at least that much.
The gap is recorded, never corrected. We do not know which side is wrong, and quietly adjusting a number to make a check pass is how a meter starts lying.
A model with no entry in your rate card is reported as unpriced and left out rather than estimated.
wrap catches every call a task makes, including the ones your framework makes
for you. It does not tell you which sub-agent made each one.
If your harness doesn't say who is calling — and most SDKs don't put that on the wire — no amount of wrapping invents it. You get every call, correctly totalled, with the task it belongs to. You do not get a per-sub-agent breakdown unless you label the steps yourself.
So the honest scope:
- You own your loop — label the steps and attribution works end to end.
- Managed agent platform — per-session totals are the ceiling. Nobody can do better from outside, and a tool claiming otherwise is guessing.
A call whose owner cannot be determined is recorded as unattributed and shown on its own line. It is never folded into the parent task, because a total that looks complete and isn't is worse than a gap you can see.
====================================================================
QVUNEX METER REPORT
====================================================================
window 3600.0 s
calls 41,209
inferences 329,672
gpu rate $0.7500/hr x 1 device(s)
window cost $0.75
CPKI $0.0023 per 1,000 inferences
--------------------------------------------------------------------
COST BY ENDPOINT
--------------------------------------------------------------------
endpoint inf share cost CPKI
image-classifier 251,104 78.1% $0.59 $0.0023
text-embedder 62,336 18.6% $0.14 $0.0022
thumbnail-scorer 16,232 3.3% $0.02 $0.0015
--------------------------------------------------------------------
WASTE
--------------------------------------------------------------------
mean GPU utilisation 31.4%
time below 5% util 38.2%
cost of that idle time $0.29 <-- paid for, not used
duty cycle 44.1% time actually inside inference
batch efficiency 19.0% mean batch vs max seen (32)
Plus latency percentiles per endpoint, memory headroom, and a short list of things worth looking at — fired only on defensible thresholds, because a meter that cries wolf stops being read.
Two cost figures are only comparable if they were produced under the same conditions,
and those conditions almost never get published. qvunex checklist fills in what the
meter observed and prints everything else as MISSING, loudly:
04 WORKLOAD
------------------------------------------------------------------
4.1 batch_size / concurrency [observed]
batch 1-32, mean 6.2 (6 distinct sizes)
4.2 input_tokens / output_tokens
>> MISSING - qvunex cannot observe this. State it yourself.
Tell it what it cannot see, and those fields fill in too:
qvunex.configure(rate_usd_hour=0.35, context={
"engine": "vLLM 0.27.1",
"cuda_graphs": True,
"model_id": "Qwen/Qwen2.5-1.5B-Instruct",
"dtype": "fp16 weights, fp16 compute",
})Why this is in the tool rather than in a document: I published a benchmark showing 4-bit quantization was 24.8% more expensive than fp16, then re-ran it with CUDA graphs enabled and measured 48.3% cheaper. One unreported flag, 73 percentage points, same card, same afternoon. A checklist nobody fills in is not a standard.
Full field list and the evidence behind each one: https://gist.github.com/qaisermehdi3-coder/b00f296641681695daf90e5a500d0d23
Local only. Every measurement is appended to a JSONL file on your own disk. This package contains no network code — grep it. That isn't a limitation, it's the point: a read-only tool that never phones home gets adopted in an afternoon instead of surviving a six-month security review.
No dependencies. Pure standard library. GPU stats come from shelling out to
nvidia-smi, which exists wherever an NVIDIA GPU does. Nothing for anyone's
platform team to approve.
Never breaks the caller. Any failure inside the meter is swallowed. Your
function's exceptions propagate untouched, and a call that raises is still
recorded with ok=False. A measurement tool that can take down production is a
tool nobody installs twice.
Cheap. The hot path is a perf_counter pair, a dict, and an append to a
buffered list. No I/O per call.
Honest. Costs are attributed, not measured: we know total GPU-hours and each endpoint's share of busy time, and we divide. Under concurrency, summed call duration exceeds wall time — so duty cycle is capped and the overlap is flagged in the output rather than quietly normalised. Without a GPU price, cost is reported as unavailable, never estimated from a default.
Set your GPU price to get costs. Everything else has a working default.
import qvunex
qvunex.configure(
path="~/.qvunex/events.jsonl", # where the corpus lives
rate_usd_hour=0.75, # what you pay per GPU-hour
sample_interval=1.0, # device polling, seconds
)Or by environment, which is usually easier in a container:
QVUNEX_PATH=/data/qvunex.jsonl
QVUNEX_RATE_USD_HOUR=0.75
QVUNEX_DISABLED=1 # hard off, decorator becomes a passthrough
For code that isn't shaped like a function:
with meter.span("batch-job", batch=len(items)):
process(items)Batch size is inferred from the first argument (.shape[0], then len(),
then 1). Override it when that guess is wrong:
@meter("ranker", batch=lambda args, kwargs: len(kwargs["docs"]))
def rank(*, query, docs): ...qvunex report # default corpus
qvunex report /data/events.jsonl # a specific file
qvunex report --rate 0.75 # override the recorded price
qvunex report --json # raw analysis for your own tooling
qvunex checklist # comparability fields, gaps marked MISSING
qvunex demo # synthetic workload + report, no setupEvents are JSONL, one object per line, schema-versioned in schema.py. Three
record types: session, call, gpu.
Keep these files. The corpus is the asset — cross-workload measurement data can't be collected retroactively, and no cloud vendor can assemble it, because no cloud vendor is neutral enough to be allowed to measure its competitors.
v0.2. Measures what it claims to measure and nothing more. Precision headroom and input-triviality analysis are deliberately not here yet — those change your model, and this release only observes.
qvunex checklist exits non-zero when fields are missing, so you can wire it into
CI and fail a build that would publish an uncomparable number.
Apache-2.0.