🚧 [WIP] MoE cache, CPU experts and GGUF support #671

Tracks the refactor of the MoE expert cache and the CPU expert path and GGUF support.

Where it stands

• The GPU slot cache splits into slot pools, one per layer or one per quant (feat(moe): split the offload expert cache into slot pools #689), and each MoE layer has its own expert method and bank set (feat(moe): serve experts of several formats through per-format slot pools #700).
• The engine takes each MoE layer's own quant method, and the CPU expert executor serves a weight format per layer (feat(moe): serve experts of several formats through per-format slot pools #700).
• Several decisions made before the model is built still read one model-wide ModelConfig.expert_quant: whether fused is allowed, --moe-cpu-layers auto, the automatic offload / hybrid pick, and some family weight readers.
• GGUF checkpoints load through the quant layer and run on vendored ggml-cuda kernels (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690), TP 1, for the families under Model Support. Qwen3.5 / 3.6 / 3.8 and Gemma-4 also take image input from an mmproj (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803, see GGUF vision towers (mmproj): survey and plan #794); the other families serve text only. cpu runs every GGUF expert type through ggml-cpu (feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704); hybrid above 8 tokens per step is limited by the first known issue below.

Work items

Cache

• Slot pools: one pool per layer or one per quant (quant tag and bank layout), sized equally or from an expert-use profile. feat(moe): split the offload expert cache into slot pools #689 @jason-fxz
• Bank sets that differ per layer: the banks are the union of the formats' roles, a layer without a role gets a zero-byte placeholder, and each format gets its own slot pool. feat(moe): serve experts of several formats through per-format slot pools #700 @andy-yang-1 @jason-fxz
• Pin-capped hosts: under hybrid, OS-locked layers keep slot pools of their own and fetch part of their misses through a pinned staging buffer, so models with several slot pools (mixed-type GGUF) combine with --moe-cpu-layers; --moe-cpu-layers auto finds the pin limit by pinning until the driver refuses. feat(moe): fetch locked-layer experts through pinned staging #742 @jason-fxz

Mixed quant per layer

• Engine, loaders and FTW checkpoints take each layer's own expert method; the CPU executor gets every layer's format as (name, options) and treats all formats alike. feat(moe): serve experts of several formats through per-format slot pools #700 @andy-yang-1 @jason-fxz
• Resident (fused) experts of several formats.
• Record each layer's expert quant in ModelConfig (keep expert_quant as the most common one for compatibility) and make the decisions taken before the model is built per layer: fused eligibility, --moe-cpu-layers auto, the automatic offload / hybrid pick, and the family weight readers (qwen3_5_moe reads stacked bf16 experts only when expert_quant == "none").
• ft bench bw: record one entry per layer kind, keyed by (format, (H, I), options), e.g. ("bf16", (H, I)), ("nvfp4", (H, I)), ("gguf", (H, I), Q4_0, Q6_K), and let the engine pick hybrid and the fetch fraction per layer. Today the profile is keyed by format only and the automatic fetch cap reads the most common format's entry for every layer. (feat(moe): bench hybrid fetch fractions per moe layer kind #734)

CPU expert executor

• Split the executor from the weight formats and the GPU handshake (refactor(cpu-moe): split the executor into weight formats and gpu sync backends #670). @jason-fxz
• Kernel IO for the handshake: a publish kernel writes a step's inputs into the pinned IO block and rings the doorbell, or skips a step without CPU routes; a combine kernel reads the CPU rows back. The coordinator skips tasks without routed experts (refactor(cpu-moe): split the executor into weight formats and gpu sync backends #670). @jason-fxz
• Hybrid decode: split the routes between GPU and CPU inside the hybrid ensure kernel instead of torch ops (perf(moe): split hybrid decode routes inside the ensure kernel #685). @jason-fxz
• Batch size > 1: split the work by expert, so an expert shared by several tokens is read once.

GGUF

Kernels and loading

• GPU: llama.cpp's ggml-cuda MMQ / MMVQ kernels for every GGUF type that has an MMQ kernel, prefill and decode (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• CPU: llama.cpp's ggml-cpu row kernels (quantized, and f32 / f16 / bf16), vendored unchanged and built for AVX2 as static libraries linked into _cpu_moe; one gguf executor format per layer's (gate_up type, down type), and every type with a ggml-cpu vec_dot works. Replaces the q4_0 CPU format. feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu #704 (on top of feat(moe): serve experts of several formats through per-format slot pools #700) @jason-fxz
• Tokenizer: build GGUF tokenizers on transformers 5.18, which keeps USER_DEFINED and CONTROL tokens such as <think> whole, with llama.cpp's stop tokens (build(deps): upgrade transformers to 5.18 #698). @jason-fxz

Model Support

Every model family FreeToken registers; the backticked name is the GGUF architecture llama.cpp's converter writes.

• Qwen3.8-Flash-Next (qwen4exp) (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• Qwen3.5 / Qwen3.6, also Qwen3.8-27B (qwen35, qwen35moe). (feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints #701)
• Gemma-4 26B-A4B (gemma4) (feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels #690). @jason-fxz
• Dense Gemma-4 (gemma4). (feat(gguf): serve Gemma 4 dense GGUF checkpoints #706)
• E-series Gemma-4 (gemma4)
• Llama (llama). (feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints #722)
• Qwen2 (qwen2). (feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints #707)
• Qwen3 / Qwen3-MoE (qwen3, qwen3moe). (feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints #707)
• Qwen3-VL / Qwen3-VL-MoE (qwen3vl, qwen3vlmoe). (feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only #807)
• MiniMax-M2 (minimax-m2). (feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725)
• MiniMax-M3 (minimax-m3). (feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints #725)
• DeepSeek-V4 (deepseek4). (feat(gguf): serve DeepSeek-V4 Flash and Pro GGUF checkpoints on deepseek4 #774)
• DeepSeek-V4.1 (llama.cpp has no converter for it).
• Muse-Glimmer (muse-glimmer). (feat(gguf): serve Muse-Glimmer GGUF checkpoints #724)
• Mistral models converted before llama.cpp had mistral3 (llama): Mistral-7B, Mistral-Nemo, Mistral-Small-3.1 / 3.2, Magistral, Devstral-2507. (feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints #722)
• Mistral 3 (mistral3).
• gpt-oss (gpt-oss).
• GLM-4.5 / 4.7 (glm4moe). (feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints #731)
• GLM-5 / 5.1 / 5.2 (glm-dsa).
• GLM-5.3-Flash (glm5-next). (feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next #721)

Vision towers

• Image input from llama.cpp mmproj files, tracked in GGUF vision towers (mmproj): survey and plan #794: Qwen3.5 / 3.6 / 3.8 and Gemma-4 done (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803), GLM-5.3-Flash in review (feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj #810).

Known issues

• MoE MMQ (the GPU path above 8 tokens) hits an illegal memory access when one token routes to the same expert slot more than once. hybrid does this on every step that has CPU routes: they are parked on slot 0 with weight 0 because the GGUF MoE kernel cannot skip inactive slots. So hybrid crashes above 8 tokens per step for any quantized expert type; Q4_0 is the only one it runs today.
• Qwen3.8-Flash-Next with --ple-backend disk (wait-sync): the first CUDA graph replay of a batch size can hang, because the graph waits on a host flag that the engine sets only after replay() returns and the first replay also uploads the graph. Seen at 16 concurrent requests on the GGUF; not GGUF-specific. fix(engine): upload cuda graphs at capture so a stream wait cannot block the first launch #697 @jason-fxz
• MMQ / MMVQ use 32-bit block offsets, so a weight view of more than 2^31 blocks is not supported, e.g. a Qwen3.8-Flash-Next offload pool of Q4_0 / Q8_0 experts above 20,971 slots. fix(gguf): reject ggml weights past the 32-bit block offsets of MMQ / MMVQ #699 makes it raise instead of faulting. @jason-fxz
• IQ1_M experts have no MMQ kernel, so MoE steps above 8 tokens fail.
• NVFP4 GGUF .scale: llama.cpp's converter keeps each NVFP4 tensor's weight_scale_2 in a separate <name>.scale tensor. It is now applied on the GPU and CPU paths, and the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders read it (feat(gguf): apply the weight scales of nvfp4 gguf tensors #732). An NVFP4 token_embd is read since feat(gguf): read NVFP4 token_embd rows in the vendored get_rows #801; like llama.cpp, which cannot load such a file, one with a .scale is refused.
• NVFP4 GGUF .scale, still refused on glm4moe, qwen2, minimax-m2, minimax-m3, muse-glimmer and qwen4exp, which have no tested NVFP4 GGUF. llama.cpp's converter cannot produce a valid one for minimax-m2 and muse-glimmer.

Performance

do after basic GGUF CUDA/CPU kernel support.

• ggml CPU kernels for GGUF experts (measured on a Xeon 8480+, 12 threads): x86 quantize_row_q8_K is scalar (about 5 us per row at K=2560); IQ types are compute-bound (30-40 GB/s per expert vs about 140 GB/s for Q4_K); with several tokens on the same experts, Q4_0 / Q4_K / Q8_0 run slower than on distinct experts (cause not investigated).
• MoE MMQ: tune the tile width J of the flat grid; the automatic choice was up to 23% slower than the best J in a sweep.
• Native FP4 MMA (off unless GGML_CUDA_FP4_MMA; needs an sm_120a / sm_120f build).