GitHub - KPST-LAB/Qml-router: Persistent KV cache sessions for LLM serving on edge and on-premises hardware: restore the KV cache instead of recomputing it, lossless long-context overflow archive, cross-engine sessions (llama.cpp, vLLM, SGLang). IEEE Access 2026.

How to run · How it works · qml-engine · Results · Docs · Engines · Paper · License

qml-router is a persistent KV cache session layer for LLM serving on edge and on-premises hardware.
qml stands for query-based machine learning; the project is not related to Qt QML.
It keeps session state outside the inference engine: a session survives process restarts,
a reconnect restores the KV cache instead of recomputing it, history past the context window
is archived without loss and retrieved when a question needs it, and a session can continue
on another engine (llama.cpp, vLLM, SGLang, qml-engine).

4.6×

faster 32K-token reconnect
_{restore 0.66 s vs recompute 3.03 s · RTX 4090, NVMe}

20.7×

faster on an edge device
_{4.2 s vs 87.9 s · Jetson AGX Orin, 30 W}

20,000

turns in one session
_{no material increase in per-turn latency · Llama-3.3-70B, H200}

_{Llama-3.2-3B-Instruct on llama.cpp, RTX 4090. Left: N users reconnect at once (latency over passing

answers only: 29/30 qml-router, 30/30 baseline; throughput not answer-checked).

Right: one user's reconnect latency versus history length.}

Note

This repository holds the source code, the benchmark programs and the paper's experiment artifact. qml-engine, the own engine, follows at a later release.

News

Date
2026‑10	The paper on qml-router is accepted by IEEE Access (Paper).
2026‑10	Source code, benchmark programs and the paper's artifact released.

Two ways to run

One session model — a token log owned by an explicit identity is the canonical state,
and the KV cache is a rebuildable acceleration derived from it — in two forms.

	qml-router	qml-engine
What it is	A layer attached to existing engines	An LLM inference engine of its own
Engines	llama.cpp, vLLM, SGLang	its own CUDA kernels (sm_90 WGMMA, sm_100 tcgen05), ggml for the other paths
Identity	`user::session`, assigned by a trusted gateway	`tenant::user::agent::session`, from verified credentials (API key, JWT, signed gateway headers)
Stored state	SQLite (session, token log, archive), LMDB (KV)	SQLite or PostgreSQL (metadata), LMDB (one file per session)
Stored KV checks	engine configuration match; otherwise token-log replay	owner, position range, model fingerprint, version and token digest, with MAC or AEAD
Long conversations	TAWT and RIG	TAWT and RIG
In the paper	yes	no
Code	after the paper's final publication	at a later release

How it works

Function	What it does
Identity-keyed persistence	Session state is stored and looked up by session identity, not by content hash.
Restore instead of recompute	A reconnect loads the stored KV payload; the token log is the fallback when the payload does not match the engine configuration.
TAWT (turn-aligned window trim)	Past the window, whole turns move to a per-session archive without loss; the system prefix stays.
RIG (Recall Injection Gate)	Archived turns are indexed with SQLite FTS5 (BM25); a confidence gate and a token budget decide what to inject on a later turn.
Cross-engine continuation	A session continues on another engine by replaying the token log, given a shared tokenizer.

qml-router layers: client and gateway, session layer, engine integration, C core, embedded store

Text version

                            QML ROUTER
┌────────────────────────────────────────────────────────────────┐
│                        Client / Gateway                        │
│ OpenAI-compatible API │ new turn + session_id (user::session)  │
├────────────────────────────────────────────────────────────────┤
│                         Session Layer                          │
│ Identity │ Token Log │ Restore │ TAWT │ RIG (FTS5, BM25)       │
├────────────────────────────────────────────────────────────────┤
│                       Engine Integration                       │
│ llama.cpp server patch + hook                                  │
│ vLLM KVConnector │ SGLang HiCache backend                      │
├────────────────────────────────────────────────────────────────┤
│                             C Core                             │
│ TAWT policy (qml_window) │ RIG policy (qml_rig)                │
├────────────────────────────────────────────────────────────────┤
│             Embedded Store (in the engine process)             │
│ SQLite: session │ token log │ archive │ FTS5 index             │
│ LMDB: KV payload                                               │
└────────────────────────────────────────────────────────────────┘

Step	One turn of a session
1	Look up the session by its identity.
2	If the stored KV matches the engine configuration, load it; otherwise replay the token log.
3	RIG searches this session's archive, gates the candidates and injects the chosen turns.
4	The engine generates the answer.
5	The turn is appended to the token log and the KV is saved.
6	Past the window, TAWT moves whole turns to the archive.

qml-engine

The same session model inside an engine of its own: identity from verified credentials,
stored KV that is verified when it comes back and recomputed from the token log when it does not match,
and prefill and decode on the engine's own CUDA kernels. Not part of the paper.

qml-engine layers: API and access, runtime, executor, CUDA kernels, GPU memory, persistent state, security, store

Text version

                            QML ENGINE
┌────────────────────────────────────────────────────────────────┐
│                          API / Access                          │
│ OpenAI API │ Session API │ Auth: key │ JWT │ gateway           │
│ Identity: tenant::user::agent::session                         │
├────────────────────────────────────────────────────────────────┤
│                            Runtime                             │
│ Scheduler (tenant quota) │ TAWT │ RIG │ Shared prefix          │
├────────────────────────────────────────────────────────────────┤
│                            Executor                            │
│ Prefill │ Decode │ Sampling │ Tools │ Vision input             │
├────────────────────────────────────────────────────────────────┤
│                          CUDA Kernels                          │
│ Attention (one definition for prefill and decode)              │
│ MoE │ INT8 matvec │ sm_90 WGMMA │ sm_100 tcgen05               │
│ ggml for the other paths                                       │
├────────────────────────────────────────────────────────────────┤
│                           GPU Memory                           │
│ KV pool (VMM) │ Identity blocks │ Prefix copy-on-write         │
├────────────────────────────────────────────────────────────────┤
│                        Persistent State                        │
│ Token log │ KV segment chain (checkpoint │ delta)              │
│ L2 host tier │ Recompute from the token log                    │
├────────────────────────────────────────────────────────────────┤
│                            Security                            │
│ MAC / AEAD per piece │ Rollback anchor │ Model fingerprint     │
│ Token digest │ Crypto-erasure │ Weight hash                    │
├────────────────────────────────────────────────────────────────┤
│                             Store                              │
│ SQLite or PostgreSQL (metadata) │ LMDB (one file per session)  │
└────────────────────────────────────────────────────────────────┘

qml-engine measured results (not part of the paper)

Measurement	Result	Device, model	Date
32K-token session restore (31.4K tokens)	58-130 ms plaintext, 161-349 ms encrypted (AES-GCM), depending on the page-cache state; recomputation 1,183 ms	B200, Qwen3-30B-A3B	2026-10-16
32K-token prompt, time to first token	Llama-3.1-8B Q8_0 0.82 s; Qwen3-30B-A3B Q3_K_M 0.92 s; gpt-oss-20b 0.97-1.17 s; Gemma 3 4B 0.85-1.04 s; Qwen3.5-35B-A3B BF16 1.53-1.56 s	B200	2026-10-15/16
20,000-turn session, 500 recall questions (questions and answers kept in the session)	500/500, median response 128 ms	H200, Qwen3-30B-A3B	2026-10-22
Same test	500/500, median response 83 ms; Qwen3.5-35B-A3B 499/500; Llama-3.1-8B 497/500; gpt-oss-20b 496/500; gpt-oss-120b 437/500	B200	2026-10-17/20
Decode on the same GGUF as llama.cpp (engine / llama.cpp)	gpt-oss-20b 302 / 321; Qwen3-30B-A3B 243 / 232; gpt-oss-120b 218 / 227 tokens/s	H200	2026-10-05
Stored-state tampering	bit flips, swapped chunks, wrong segments and old versions refused and recomputed	RTX 4090, SQLite and PostgreSQL	2026-10-07
Parser fuzzing (7 parsers)	14 defects found and fixed; afterwards no findings, no authentication bypass in 6.8M runs	RTX 4090 host	2026-10-07

GPUs used: RTX 4090 (sm_89), H200 (sm_90), B200 (sm_100). Model families run: Llama 3.x, Qwen3 and Qwen3.5 (dense and MoE), gpt-oss, Gemma 3 and 4, DeepSeek-V2-Lite and GLM-4.7-Flash (MLA); GGUF and safetensors.

Results

qml-router, Llama-3.2-3B-Instruct unless noted; every reconnect answer is checked against a value
planted in its own history. Baseline = the same engine without qml-router.

32K-token reconnect by storage and device

Environment	Recompute	Restore	Ratio
RTX 4090, storage 0.55 GB/s (container)	3.16 s	2.24 s	1.4×
Same GPU, NVMe 1.6-1.9 GB/s (host)	3.03 s	0.66 s	4.6×
Jetson AGX Orin, MAXN	23.93 s	2.26 s	10.6×
Jetson AGX Orin, 30 W	87.87 s	4.24 s	20.7×

_{Regimes where recomputation is as fast or faster (short histories on a fast GPU) are reported in the paper.}

Reconnect latency under load (llama.cpp, RTX 4090)

Condition	Baseline	qml-router	Ratio
16 users reconnect at once: slowest / median reconnect	59.9 s / 10.0 s	0.51 s / 0.49 s	—
Same run: short-request throughput	1.0 req/s	29.0 req/s	29×
One user, 49K-token history	14.5 s	1.3 s	11×
32K-token history, 10 runs (median)	5.433 s	0.760 s	7.1×
49K-token history, 10 runs (median)	14.400 s	1.329 s	10.8×

500-turn session past a 32K-token window (facts answered at turn 500)

Engine and device	Baseline	qml-router	Throughput (turns/s), baseline -> qml-router
llama.cpp, RTX 4090	0/46	42/44	0.42 -> 0.49
vLLM, RTX 4090	0/46	46/46	3.95 -> 1.49
SGLang, RTX 4090	0/46	40/46	4.23 -> 2.53
llama.cpp, Jetson AGX Orin	0/46	42/44	0.06 -> 0.13
llama.cpp, RTX 4090, Qwen3-4B	0/46	46/48	0.32 -> 0.44

On vLLM and SGLang the baseline drops the history that passed the window and stays faster; qml-router keeps the history and answers the questions about it.

Cost of adding qml-router (RTX 4090, 12 sessions per run, 3 runs)

Engine	GPU memory	Latency per request
llama.cpp	+0 MiB	225 -> 252 ms (+27)
vLLM	+0 MiB	180 -> 259 ms (+79)
SGLang	+47 MiB	166 -> 185 ms (+19)

Comparison with persistent-KV systems

✓ = supported, — = not provided, N/R = the cited work does not report the property (not taken as evidence of its absence). Cells reflect the cited publications and, for LMCache, its 2026 public documentation.

System	Canonical state	Identity-keyed session	Restart persistence	Heterogeneous engines	Lossless overflow archive	Selective history retrieval	Process isolation boundary	Encryption at rest	Correctness verification
LMCache	KV bytes (content hash)	prefix-keyed chunks; tenant salt, no session identity	✓ (standalone daemon)	vLLM, SGLang	—	—	out-of-process daemon (IPC per operation)	✓ (persistent tier)	N/R
KVGov	KV bytes (content hash, principal salt)	prefix-keyed within a principal; no session identity	—	vLLM, llama.cpp	—	—	in-process (engine cache)	N/R	N/R
CachedAttention	KV bytes	session-oriented	✓ (hierarchical store)	single engine	—	—	N/R	N/R	N/R
agent-memory	KV bytes (Q4)	prefix-keyed per agent (Q4 KV)	N/R	single runtime (MLX)	—	N/R	in-process (MLX runtime)	N/R	N/R
qml-router (llama.cpp, vLLM, SGLang)	token record; KV derived	✓ (`user::session`)	✓	✓	✓ (TAWT)	✓ (RIG)	in-process store (trust boundary = process)	— (roadmap)	✓

Models and hardware used in the experiments (including experiments that are not in the paper)

Device	Model (precision, format)	Engine	Experiments	In the paper
NVIDIA H200 (143 GB)	Llama-3.3-70B-Instruct Q8_0, GGUF	llama.cpp + qml-router	20,000-turn session (27 questions); 2,000-turn weather recall (21 questions); LongMemEval_S archive recall (reader and judge); ranker accuracy; needle recall in a continuous session (90 questions); load tests	Section V-B1, Fig. 6, Table 13 (needle recall and load tests: no)
NVIDIA H200	Llama-3.3-70B-Instruct FP8-dynamic, HF	vLLM 0.24.0, unmodified (retrieval in the client)	2,000-turn weather recall (21/21); 20,000-turn session (27/27, stock 0/27); needle recall (88/90)	No
NVIDIA H200	Llama-3.1-8B-Instruct BF16, HF and GGUF	llama.cpp, vLLM	Engine timing on identical weights; 20,000-turn session (llama.cpp 25/27, vLLM 27/27)	No
NVIDIA H200	Llama-3.1-8B-Instruct Q8_0 and Llama-3.2-3B-Instruct F16, GGUF	llama.cpp + qml-router	2,000-turn weather recall by reader size (3B, 8B, 70B)	No
NVIDIA H200	Qwen3-32B Q8_0, GGUF	llama.cpp + qml-router	Adoption cost; 500-turn context overflow	No (removed from the paper)
NVIDIA H200	bge-m3 F16, GGUF	llama.cpp (embedding)	Embedding retrieval and ranker latency	Fig. 6(a), Table 13
RTX 4090 (24 GB)	Llama-3.2-3B-Instruct F16, GGUF, Q4_0 KV	llama.cpp + qml-router	Reconnect density and crossover; adoption cost; 500-turn context overflow; repeated runs; restore/recompute boundary; persistence window; layer-versus-engine attribution; write volume; 2,000-turn weather recall checks	Fig. 1, Tables 4-7, 14, Appendix A-E (weather recall checks: no)
RTX 4090	Llama-3.2-3B-Instruct FP16, HF	vLLM 0.26.0, SGLang 0.5.16	Adoption cost; 500-turn context overflow; mixed workload; cross-engine continuation; multi-turn session load (96 turns)	Tables 4, 5, 10, Fig. 5 (load: no)
RTX 4090	Llama-3.1-8B-Instruct AWQ-INT4, HF	vLLM 0.26.0 (qml-router connector, LMCache 0.5.2), SGLang 0.5.16	Identity-keyed versus content-keyed reuse (LMCache); mixed workload	Tables 10, 11
RTX 4090	Llama-3.1-8B-Instruct Q4_K_M, GGUF	llama.cpp	Mixed workload; restart resume	Table 10 (resume: no)
RTX 4090	Llama-3.1-8B-Instruct Q8_0, GGUF	llama.cpp + qml-router	20,000-turn session (25/27; 8/9 in one continuous session); retrieval selection comparisons	No
RTX 4090	Qwen3-4B F16, GGUF, Q4_0 KV	llama.cpp + qml-router	Size-matched non-Llama check (adoption cost, 500-turn context overflow)	Section V-A2, Table 5
RTX 4090	gemma-3-4b-it, gemma-3-12b	vLLM 0.26.0, SGLang 0.5.16, llama.cpp	Early restart and resume tests; sliding-window attention checks	No
RTX 4090	bge-m3 F16, GGUF	llama.cpp (embedding)	Ranker latency on GPU and CPU; lexical versus embedding retrieval	Table 13
Jetson AGX Orin	Llama-3.2-3B-Instruct F16, GGUF, Q4_0 KV	llama.cpp + qml-router	Reconnect crossover at MAXN and 30 W; 500-turn context overflow; adoption cost	Abstract, Tables 5-7 (adoption cost: no)
NVIDIA B200 (183 GB)	Qwen3-30B-A3B (GGUF Q3_K_M, Q4_K, Q8; BF16 on SGLang 0.5.20), gpt-oss-20b, gpt-oss-120b (also on vLLM 0.30.0), Qwen3.5-35B-A3B, Llama-3.1-8B-Instruct Q8_0, Llama-3.2-3B-Instruct F16, Qwen3-32B Q8_0, a 70B Q4_K_M model; architecture checks on Qwen3.5-0.8B/4B, Gemma 3/4, DeepSeek-V2-Lite	qml-engine	20,000-turn memory stress test (500 questions); 32K restore and time to first token; 27-question weather recall	No

How to run

Item	Tested with
OS	Ubuntu 22.04 and 18.04 (x86_64); JetPack on the Jetson AGX Orin
GPU and CUDA	RTX 4090 (CUDA 11.8 to 12.x), H200 (CUDA 13.0), Jetson AGX Orin (CUDA 13.2); a CPU build also works
Engines	llama.cpp `0d47ea742` with the qml-router patch (built by the setup script), vLLM 0.26.0, SGLang 0.5.16 (`requirements-engines.txt`)
Python	the vLLM and SGLang integrations, the session layer and the benchmarks (`requirements-bench.txt`)

1. Build

sudo apt install cmake build-essential libsqlite3-dev liblmdb-dev libssl-dev

# llama.cpp at the paper's commit with the qml-router patch, in third_party/llama.cpp
# --cuda 89 = RTX 4090, 87 = Jetson AGX Orin, 90 = H200; --cpu builds without CUDA
scripts/setup_llama_cpp.sh --cuda 89

# store library for the vLLM connector and the session layer
# (build/libqml_session_store.so)
cmake -S . -B build && cmake --build build --target qml_session_store_shared -j

2. Model

# llama.cpp reads GGUF: convert the Hugging Face weights of Llama-3.2-3B-Instruct to F16
python3 third_party/llama.cpp/convert_hf_to_gguf.py <Hugging Face model directory> \
  --outtype f16 --outfile models/Llama-3.2-3B-Instruct-F16.gguf
# vLLM and SGLang load unsloth/Llama-3.2-3B-Instruct from Hugging Face directly

3. llama.cpp with qml-router

third_party/llama.cpp/build/bin/llama-server \
  -m models/Llama-3.2-3B-Instruct-F16.gguf -ngl 99 -fa on \
  -c 131072 --kv-unified -np 8 -ctk q4_0 -ctv q4_0 --cache-ram 0 --port 8397 \
  --qml-session-store store/qml.db --qml-lmdb-store store/qml_lmdb

# a request sends only the new turn with a session id;
# the server prepends the stored history
curl http://127.0.0.1:8397/completion -d '{
  "prompt": "and what was my name again?",
  "session_id": "alice::chat-1", "cache_prompt": true, "n_predict": 64}'

4. vLLM with qml-router, behind the session layer

pip install vllm==0.26.0
mkdir -p store
VLLM_PLUGINS="" PYTHONPATH=hooks/vllm python3 -m vllm.entrypoints.openai.api_server \
  --model unsloth/Llama-3.2-3B-Instruct --port 8000 \
  --kv-transfer-config '{"kv_connector": "QmlVllmConnector",
    "kv_connector_module_path": "qml_vllm_connector",
    "kv_role": "kv_both", "kv_buffer_device": "cpu",
    "kv_connector_extra_config": {"qml_sqlite_path": "store/qml.db",
      "qml_lmdb_path": "store/qml_lmdb", "qml_self_url": "http://127.0.0.1:8000",
      "qml_token_log": "off"}}'

# session layer in front of the engine:
# token log, window trim (TAWT) and archive recall (RIG)
python3 router/qml_openai_gateway.py --engine vllm --engine-url http://127.0.0.1:8000 \
  --model unsloth/Llama-3.2-3B-Instruct --tokenizer <model directory> \
  --store-db store/qml.db --store-lmdb store/qml_lmdb --window 3000 --port 8090

curl http://127.0.0.1:8090/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "session_id": "alice::chat-1", "max_tokens": 64,
  "messages": [{"role": "user", "content": "and what was my name again?"}]}'

5. SGLang with qml-router, behind the session layer

pip install sglang==0.5.16
mkdir -p store
PYTHONPATH=hooks/sglang python3 -m sglang.launch_server \
  --model-path unsloth/Llama-3.2-3B-Instruct --port 30000 --page-size 64 \
  --enable-hierarchical-cache --hicache-write-policy write_through \
  --hicache-storage-backend dynamic \
  --hicache-storage-backend-extra-config '{"module_path": "qml_sglang_storage",
    "class_name": "QmlSglangStorage", "backend_name": "qml",
    "qml_sqlite_path": "store/qml.db", "qml_lmdb_path": "store/qml_lmdb",
    "qml_token_log": "off"}'

# the same session layer, with SGLang as the engine (session id sent as extra_key)
python3 router/qml_openai_gateway.py --engine sglang --engine-url http://127.0.0.1:30000 \
  --model unsloth/Llama-3.2-3B-Instruct --tokenizer <model directory> \
  --store-db store/qml.db --store-lmdb store/qml_lmdb --window 3000 --port 8090

curl http://127.0.0.1:8090/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "session_id": "alice::chat-1", "max_tokens": 64,
  "messages": [{"role": "user", "content": "and what was my name again?"}]}'

6. Run a paper experiment

Each experiment has a folder in tests/paper/ with its run scripts and a README (model, engine, hardware, frozen log). The scripts read REPO (repository root; models under $REPO/models/) and, where they write stores, BENCH (a folder on the storage under test); they start and stop only their own servers.

# Table 7: 8K and 32K reconnect, restore versus recompute
REPO=$PWD bash tests/paper/tab07_restore_recompute_boundary/tab07_hw_nvme.sh

# Fig. 1(a)-(b): N users reconnecting at once
REPO=$PWD bash tests/paper/fig01ab_reconnect_density/fig01_density.sh

# Table 5: one engine (llama, vllm, sglang) and mode (stock, qml) per run
REPO=$PWD BENCH=/mnt/nvme/bench \
  bash tests/paper/tab05_context_overflow_500turn/tab05_500turn_4090.sh llama qml

Paper item	Script
Fig. 1(a)-(b)	`tests/paper/fig01ab_reconnect_density/fig01_density.sh`
Fig. 1(c)	`tests/paper/fig01c_reconnect_vs_history_length/fig01_crossover.sh`
Table 4	`tests/paper/tab04_adoption_cost/tab04_adoption_llama.sh`, `tab04_adoption_vllm_sglang.sh`
Table 5, Fig. 4	`tests/paper/tab05_context_overflow_500turn/tab05_500turn_4090.sh ENGINE MODE` (`llama`/`vllm`/`sglang`, `stock`/`qml`)
Table 6	`tests/paper/tab06_repeated_runs/tab06_repeat_crossover.sh`, `tab06_repeat_mixed.sh`
Table 7	`tests/paper/tab07_restore_recompute_boundary/tab07_hw_nvme.sh`
Section V-B1	`tests/paper/secVB1_session_beyond_limits/secVB1_20000turn_session.sh`, `_recall.sh`, `_latency.sh`
Section V-B2	`tests/paper/secVB2_persistence_across_restarts/secVB2_restart_recovery.sh`
Fig. 5	`tests/paper/fig05_session_portability/fig05_continuation_vllm.sh`
Fig. 6	`tests/paper/fig06_longmemeval_archive_recall/fig06_longmemeval_archive_recall.sh`
Table 11	`tests/paper/tab11_lmcache_warm_vs_warm/tab11_lmcache_mp.sh`, `tab11_qml_connector.sh`
Tables 13, 14, Section III-B	`artifact/paper/scripts/reviewer1_edit_17_`, `reviewer1_edit_18_`, `reviewer1_edit_10_*`

To check a result, compare the new log with the frozen one in artifact/paper/logs/; artifact/paper/VERSIONS.txt maps every table and figure to its log, script and data, and artifact/paper/reader.txt says where the number is in each log.

7. Unit tests (no GPU)

for t in unit_sqlite_save_tx unit_blob_fallback unit_blob_sweep unit_kv_owner; do
  cc -I include -I storage/sqlite -I storage/lmdb tests/unit/$t.c -L build \
     -lqml_session_store -llmdb -lsqlite3 -lcrypto -lpthread -o build/$t \
     && LD_LIBRARY_PATH=build build/$t /tmp
done
cc -I include -I storage/sqlite tests/unit/unit_rig.c core/qml_rig.c -o build/unit_rig \
  && build/unit_rig

Path	Contents
`core/`, `storage/`, `include/`	C session core, SQLite schema and store, LMDB payload store, public C API
`hooks/`	Engine integrations: `llama.cpp/`, `vllm/`, `sglang/`
`patches/`	The llama.cpp server patch
`router/`	Session layer for vLLM and SGLang (Python, OpenAI-compatible)
`comm/`	SQL/UDF communication layer (control-plane experiments)
`tests/`	Benchmark programs, unit tests, and the run scripts of every paper experiment (`tests/paper/`)
`artifact/`	The supplementary material published with the paper: logs, scripts, data, result-to-file map, SHA-256 manifest
`docs/`	API specification, building and running, engine integration documents

Documentation

Document	Contents
docs/API.md	API specification: C core, engine integration, session layer for vLLM and SGLang, store schema, settings
docs/RUNNING.md	Building and running with llama.cpp, vLLM and SGLang
docs/4-engines/engine-llama-cpp.md	llama.cpp integration: server patch and hook
docs/4-engines/engine-vllm.md	vLLM integration: KV connector
docs/4-engines/engine-sglang.md	SGLang integration: HiCache storage backend
docs/4-engines/engine-comparison.md	The three engines compared

Engines

Engine	Version	Interface
llama.cpp	build b10141 (upstream commit `0d47ea742`), as in the paper	qml-router: server patch plus a hook (llama.cpp has no storage extension point)
vLLM	0.26.0, as in the paper	qml-router: `KVConnector` plug-in, no engine-source modification
SGLang	0.5.16, as in the paper	qml-router: `HiCacheStorage` backend, no engine-source modification
qml-engine	own engine, not in the paper	Built in: OpenAI-compatible chat, completions and embeddings, plus session, memory and administration endpoints
LMCache (comparison only)	0.5.2	The content-keyed baseline in the paper's Section V-D

_{Assumptions and scope (qml-router). Identities are assigned by a trusted gateway, and the serving process runs inside a

trusted host boundary, because the embedded store is readable by any code in the engine process; the identity key is not an

authentication mechanism (SECURITY.md). Cross-engine results assume a shared tokenizer vocabulary and a single host.

Network filesystems and rotational media were not tested.}

Paper

"Persistent Session-State Management for LLM Serving: An Identity-Keyed Architecture
for Durable, Engine-Agnostic Sessions," IEEE Access, 2026 (accepted; manuscript ID Access-2026-49950).
Please cite the paper above. The DOI and a BibTeX entry will be added when the article is published.

License

Dual-licensed: AGPL-3.0-only (LICENSE) or a commercial license from
Korea Platform Service Technology Co., Inc. (LICENSE-COMMERCIAL.md) for use that does not follow the AGPL terms.
Third-party notices: NOTICE. Contributions require a contributor license agreement (CONTRIBUTING.md).

_{"qml" stands for query-based machine learning; it is unrelated to the Qt QML language and to quantum machine learning.

Contact: jun@kpst.co.kr — questions, commercial license, security reports}

Navigation Menu

Repository navigation

About

Repository files navigation

4.6×

20.7×

20,000

News

Two ways to run

How it works

qml-engine

Results

How to run

Documentation

Engines

Paper

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
artifact		artifact
comm		comm
core		core
docs		docs
hooks		hooks
include		include
patches		patches
router		router
scripts		scripts
storage		storage
tests		tests
third_party		third_party
CMakeLists.txt		CMakeLists.txt
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
LICENSE-COMMERCIAL.md		LICENSE-COMMERCIAL.md
NOTICE		NOTICE
README.md		README.md
SECURITY.md		SECURITY.md
requirements-bench.txt		requirements-bench.txt
requirements-engines.txt		requirements-engines.txt

Latest commit

History

Folders and files

Repository files navigation

4.6×

20.7×

20,000

News

Two ways to run

How it works

qml-engine

Results

How to run

Documentation

Engines

Paper

License

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages