llmp

Measures how well a local LLM setup uses its model. llmp sits between any client (a coding agent, a chat UI, a script) and any local model server, records the traffic, and reports how much time goes to re-processing prompts the server had already seen, why that happened, and what each request is made of.

$ llmp analyze session.jsonl
llmp: 9 requests over 18.3 s, 8 analysed, 7 conversation turns
time lost re-processing prompts: 411 ms of 1.5 s prefill (27%), 594 tokens
  by cause: system prompt changes 411 ms (594 tokens)
coverage: 8 of 9 records analysed (1 not chat); reuse reported on 7 of 7 turns, timing on 7
findings:
  [medium] 1 side-call on the conversation model gemma: 0 tokens lost
  [low] tool write_file offered on 7 requests, never called (60 tokens each)
  [low] tool list_dir offered on 7 requests, never called (51 tokens each)

That is a real recording (tests/fixtures/llama.cpp). The client put the date at the end of its system prompt. When the date changed, llama.cpp re-evaluated the whole prompt, and that one change cost more than a quarter of the session's prefill time.

Install

cargo install llmp

As a library:

cargo add llmp                                      # proxy + analysis
cargo add llmp --no-default-features                # analysis only: pulls in just serde

Usage

1. Record. Put the proxy in front of your server and point the client at the proxy instead:

$ llmp proxy --upstream 127.0.0.1:8081
llmp: 127.0.0.1:8080 -> 127.0.0.1:8081, recording to llmp-1789988955.jsonl

The proxy forwards everything unchanged and streams responses as they arrive. It never modifies a request or a response. Stop it with Ctrl+C; it finishes the exchanges under way and prints how many records it wrote.

Option	Default
--upstream HOST:PORT	required	the model server
--listen HOST:PORT	127.0.0.1:8080	where clients connect
--out FILE	./llmp-<unix seconds>.jsonl	the recording; appended to
--digest	off	keep hashes and sizes instead of message text
--max-body BYTES	64 MiB	bodies above this are forwarded whole but not stored

A full recording holds every prompt and reply. Use --digest for long captures, or when the text must not be kept: the analysis still works, and only estimates of edited messages get rougher.

2. Analyse.

llmp analyze FILE... [--json] [--tokenize URL] [--gap-seconds N]

• Several files are merged in time order.
• --json prints the full report instead of the summary.
• --tokenize URL asks the server's tokenize endpoint for exact token counts (llama.cpp), instead of estimating from characters.
• --gap-seconds N sets how long a pause splits two sessions (default 60).

Exit codes: 0 success, 1 runtime error, 2 usage error.

As a library

Analyse a recording:

use std::fs::File;
use std::io::BufReader;

let read = llmp::read_jsonl(BufReader::new(File::open("session.jsonl")?));
let mut report = llmp::analyze(&read.records, &llmp::Config::default());
report.coverage.bad_lines = read.bad_lines.len() as u64;

print!("{}", report.summary());
for finding in &report.findings {
    println!("{:?} on {:?}: {:?} ms lost", finding.kind, finding.model, finding.lost_ms.map(|m| m.value));
}

analyze is pure: no I/O, and the same records always give the same report. Every type is serde-serialisable.

Run the proxy inside your own program (on a tokio runtime), with the records going wherever you want. A sink can be a closure, a std::sync::mpsc::Sender<Record>, or JsonlSink:

use llmp::proxy::{BodyMode, ProxyConfig};

let config = ProxyConfig {
    listen: "127.0.0.1:8080".parse()?,
    upstream: "127.0.0.1:8081".parse()?,
    body_mode: BodyMode::Full,
    max_body_bytes: 64 << 20,
};
let (stop, stopped) = tokio::sync::oneshot::channel::<()>();
let sink = |record: llmp::Record| println!("{} {} took {} ms", record.id, record.request.path, record.t_end_ms - record.t_start_ms);
let proxy = tokio::spawn(llmp::proxy::serve(config, sink, async { let _ = stopped.await; }));

serve returns once shutdown resolves and the exchanges under way have finished; serve_blocking does the same on its own runtime until Ctrl+C.

What it measures

Everything comes from one table with one row per request, which is part of the report. Every other number is computed from it, so two outputs can never disagree about the same request.

For each conversation turn, llmp works out how much of the prompt the server could have reused from the previous turn of the same conversation, and compares it with what the server says it reused. When less was reused than could have been, the difference is time lost, and it is charged to a cause:

Cause	Meaning
system_changed, tools_changed, history_rewritten	the client changed something it had sent before (a date in the system prompt, a reordered tool list, an edited message)
side_call	another request on the same model ran in between (a title, a summary, a classifier) and evicted the conversation
other_conversation	another conversation on the same model ran in between (a subagent, a parallel task, another agent) and evicted it
other_model	a request to another model ran in between
server_empty, server_head_only, server_rollback	the server dropped its cache: all of it, all but the system prompt, or part of the history

A loss is never larger than what the server actually re-evaluated. Context compactions are recognised and not blamed on anyone.

Several agents can share one proxy. Each turn is compared with the conversation it belongs to, not with whatever request came before it, and requests are grouped into clients by user-agent, and by tool set when harnesses share a user-agent. Tools and the findings a client controls (its prompt, its tools) are reported per client.

Each request is also split into system prompt, tool schemas, history and new content. The findings are typed data (kind, evidence, affected requests, size, confidence) for a host to rank, word and act on: side-calls on the conversation model, frequent prefix breaks, repeated cache losses, eviction by other conversations or models, tools never called, oversized tool schemas and system prompts.

"Not measured" is not zero. A number the server did not report is null, and every number says where it came from: server, tokenizer, proxy (the proxy's own clock) or estimate.

Servers

Server	API	Status	What it reports
llama.cpp	/v1/chat/completions	verified (b10015)	everything, including reuse; the only one where time lost can be measured
Ollama	/api/chat	verified (0.32.14)	prompt size and prefill time, but not reuse
Ollama	/v1/chat/completions	verified (0.32.14)	prompt size only in non-streamed replies; prefill time from the proxy
vLLM	/v1/chat/completions	unverified	expected: prompt size and reuse (cached_tokens); prefill time from the proxy
LM Studio	/v1/chat/completions	unverified	expected: as vLLM

"Verified" means a real recording is checked in under tests/fixtures and tested. On a server that does not report reuse, llmp still records everything and analyses the composition and the findings that don't need counters, and it says that time lost was not measured.

Formats

The recording (JSON Lines, one exchange per line) and the report (JSON) are versioned and documented in docs/format.md. They are the contract: a tool in any language can produce or consume them without linking the crate.

Out of scope

• Launching agents, or setting environment variables for them
• Advice about particular agents, servers or config keys (findings are data; the host words them)
• Hosted APIs over HTTPS, and Anthropic-format /v1/messages clients
• Several models loaded on one server (residency, eviction between models)
• Before/after comparison of two recordings
• Editing anyone's config files
• Naming the product behind a client: a client is labelled by its user-agent, and the host maps that to a product

License

MIT