Repository navigation
Requests longer than the KV pool are queued forever with no error — and --moe-cache-auto leaves only ~8k tokens of KV on a 96GB GPU #111
Description
Summary
Two related problems, one report:
- A request whose prompt needs more KV pages than
num_pagesis never scheduled and never rejected: no prefill batch, no log line, no HTTP error — the client waits until its own timeout, and the engine (which is perfectly healthy) simply serves the next request after the abort. - With
--moe-cache-autoon a 96GB RTX PRO 6000 serving MiniMax-M2.5-NVFP4, the joint budget solve yieldsmoe_cache_size=10114 num_pages=8209(page_size=1) — an effective context of ~8.2k tokens for a model advertising far more, with nothing in the logs flagging it. The pool lands just above thekv_reserve_tokens=8192floor because the solve gives nearly all remaining budget to MoE cache slots.
Together: any agent-style workload (system prompt + tools + history routinely >8k tokens) silently hangs at defaults on high-end hardware.
Environment
FreeToken v0.1.2 @ f0abe58, torch 2.11.0+cu130, CUDA 13.0.1, driver 595.91.07, RTX PRO 6000 Blackwell 96GB, EPYC 128c/503GB, nvidia/MiniMax-M2.5-NVFP4.
Reproduction
ft serve --model nvidia/MiniMax-M2.5-NVFP4 --moe-cache-auto→ log:resolved moe_cache_size=10114 num_pages=8209 (prefill_overlap=True).- Send prompts up to ~7.1k tokens → fine (single prefill batch, ~1.5k tok/s ingest).
- Send a prompt of ~8.8k tokens (> 8209 pages) → the request never returns. Server log shows no prefill batch for it at all; GPU idle; no error anywhere.
- Abort the client; send any normal request → served immediately (the engine was healthy the whole time — this cleanly discriminates it from the engine-death issue at Unhandled CUDA OOM in prefill expert workspace kills the engine; server keeps accepting requests that never return (root cause behind #20-style headless state; likely also #72) #110).
Verified allocator-independent (identical under PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True) and MoE-overlap-independent (identical with --disable-moe-prefill-overlap). The pass/hang boundary tracks token count against num_pages exactly: a prompt tokenizing to 8,011 passes; 8,760 hangs. Full experiment trail in the first comment.
Suggested fixes
- Reject over-capacity requests at admission with a clear 4xx ("prompt requires N pages, pool has M") instead of infinite queueing.
- Surface the effective KV context at startup —
num_pages=8209next to a long-context model deserves a WARN. - Reconsider the auto-solve's split (or expose a
--context-tokensknob): reserving only the floor for KV means the flagship long-context models can't hold an agent conversation at defaults; users would likely trade some expert-cache hit rate for context if the trade were visible.
Related observation (mentioning rather than filing separately — say the word and I'll split it out)
Any batch classified as prefill streams the entire expert bank set over PCIe (~4.5s / ~137GB for this model), including a 1-token extend on a full radix-cache hit — so multi-turn conversations pay a full-weights toll per turn even when KV is fully cached, and each chunk of a chunked prefill pays it again (verified: 2 chunks = 9.0s, 3 chunks = 13.5s). A token-count threshold below which prefill uses the decode path's on-demand expert loading would remove it.
Offer
Same rig and offer as #110: deterministic repros, full logs, and hardware available for validating patches.
Activity
CygnusBill commentedon Aug 23, 2026
How this was found and pinned down — the short version of a longer investigation (full trail on #110, which shares the campaign):
Discovery
Extended prompt-size sweeps (5k → 6.5k → 8k "words") on two configurations differing only in
--disable-moe-prefill-overlapboth sailed through ~7.1k tokens at increasing ingest rates (1,204 → 1,532 tok/s) and then both timed out (>300s, later >600s) on the 8k-word cell. Overlap-independence immediately ruled out the MoE movement machinery.The false lead and the tell
A replication that inserted a few seconds' gap before the big prompt passed — which briefly looked like an arrival-timing race. The token audit killed it: the gapped variant's prompt happened to tokenize to 8,011 tokens while the sweep's generator produced ~8,760 for the "same" word count. 8,011 < 8,209 (
num_pages) < 8,760. The boundary was capacity, not timing.Confirmation by prediction
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: identical infinite wait (admission logic doesn't touch the allocator) — predicted in advance.Prefill batchline; the over-capacity request logs nothing at all — it is never scheduled, not slow.Numbers for this hardware (96GB RTX PRO 6000, MiniMax-M2.5-NVFP4, defaults)
--moe-cache-auto resolved moe_cache_size=10114 num_pages=8209 (prefill_overlap=True)— the whole failure boundary in one startup line that currently draws no attention to itself.pedro-moser commentedon Aug 26, 2026
Opened #198 for the explicit-capacity side of this issue:
--num-pages/ resolved--num-tokensnow participates in the joint MoE/KV auto-budget before allocation. This is complementary to #118's scheduler admission fix and does not close the remaining default-capacity tradeoff.benwilson commentedon Aug 31, 2026
A data point for suggestions 2 and 3 from a small card, where the auto split lands the other way round:
Environment: FreeToken git
4b94bdc(HEAD 2026-08-30, no local patches), installed withpip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA toolkit 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86) on a rented vast.ai host (503 GiB RAM); imagepytorch/pytorch:2.11.0-cuda13.0-cudnn9-devel; checkpointdealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4@be794b99(standard compressed-tensors NVFP4 layout, byte-identical quant config to the RadixArk stock checkpoint — not the FTW repack).With
--moe-cache-autoand--kv-reserve-tokensset equal to the served window (--max-seq-len-override 262144 --kv-reserve-tokens 262144), the solve gavemoe_cache_size=1677 num_pages=4097(page 64 → 262,208 tokens = 6.19 GiB) and left 2.01 GiB free after graph capture; prompts to 192,587 tokens are served fine (the reserve is honoured as a floor exactly as you describe). Leave the reserve at the 8192 default and the same model on the same card would have the ~8k effective window you saw on the Pro 6000, with nothing in the log saying so. So today the only lever for context is the reserve, and it has to be set to the window by hand — a startup line likeeffective KV context: N tokens (reserve M, window W)plus a WARN when N < W would have made both of our configurations self-explaining. +1 on rejecting over-capacity prompts at admission.hypothesis: does a queued request cost decode throughput
fix(scheduler): clamp admission to the actual KV pool