Serving Qwen3.8-Flash-Next's 1M-token window on one 24 GB card #568

Before you start

• I have read the Roadmap and this is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.

Applies to

Engine

What problem does this solve

Qwen3.8-Flash-Next's model card extends its 262,144 trained positions to 1,048,576 by static YaRN. FreeToken's offload design is what makes this model fit a consumer card at all, and the same design can reach the card's full context on one 24 GB GPU, but four things stand in the way on main:

1. Rope table (Yarn support for Qwen 3.8 Next Flash #508): the cos/sin table stops at max_position_embeddings, and there is no way to pass the card's rope_parameters without editing the checkpoint.
2. K/V memory: 24 KiB of K/V per token is 24 GiB at 1M - more than the card, beside ~9 GiB of dense weights and the expert cache.
3. An unpriced prefill peak: --moe-cache-auto budgets weights and caches, but a max-length chunk's activations come on top and the allocator keeps them. At a 64k window the first 22.6k-token prompt left 430 MiB free and decode fell from 23-27 to 10-14 t/s, and stayed there.
4. A top-k cliff past a 512k window: the QSA block top-k falls back to one program per row (QSA block top-k runs a 1M-token window on one program per row #566).

Proposed solution

One change each, all measured on the hardware below:

1. feat(rotary): size the rope table for YaRN and add --hf-overrides #573 - size the rope table to the YaRN-extended context, and --hf-overrides with vLLM's semantics to pass the card's rope_parameters.
2. Host-placed K/V (--kv-placement host): decode reads the host K/V in place over PCIe, and a prefill stages its pages into a 131,072-token device window, folded by log2-sum-exp. feat(kvcache): host-RAM KV tier for sparse-attention models (--kv-host-pages) #499 reaches for the same capacity another way, so I described mine there (feat(kvcache): host-RAM KV tier for sparse-attention models (--kv-host-pages) #499 (comment)) rather than open a competing PR.
3. feat(engine): add --vram-reserve-mb to keep VRAM free at the largest prefill #574 - --vram-reserve-mb: measure a max-length prefill at start and shrink the expert cache until the reserve stays free.
4. perf(qsa): reduce block top-k candidates in levels past 16 splits #571 - reduction levels in the block top-k.

With all four, on an RTX 3090 Ti 24 GB (driver 616.92), AMD Ryzen 9 9950X3D, 256 GB DDR5, PCIe 4.0 x16, RadixArk/Qwen3.8-Flash-Next-NVFP4, one server at --num-tokens 1048576 --kv-placement host --vram-reserve-mb 2000 with the card's YaRN through --hf-overrides:

Alternatives considered

• Raising max_position_embeddings in config.json (what Yarn support for Qwen 3.8 Next Flash #508 tried): a longer table with the trained frequencies, wrong past 262,144.
• Keeping the K/V on the device with a host tier (feat(kvcache): host-RAM KV tier for sparse-attention models (--kv-host-pages) #499): the right shape when most of the context fits the card; at 1M on 24 GB it does not.
• A static margin in the cache budget instead of a measured one: the activation peak depends on the model, the chunk length and the attention backend; fix(engine): charge the attention workspace in moe-cache-auto sizing #562 charges the parts that are known statically, and the two compose.