Load-time MXFP8 requantization of bf16 dense, hyper-connection and lm_head weights #624

Before you start

• I have read the Roadmap and this is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.

Applies to

Engine

What problem does this solve

Checkpoints such as nvidia/Qwen3.8-Flash-Next-NVFP4 ship NVFP4 experts but bf16 dense, hyper-connection and lm_head weights: about 9 GiB of VRAM for Qwen3.8-Flash-Next text-only. That does not fit beside any expert cache on a 10–12 GB card. The FP8 per-tensor path in #389 (closed) needed sm_89 GEMMs and only covered attention/GDN.

Proposed solution

Requantize selected bf16 weights to MXFP8 at load time: e4m3 codes with one e8m0 scale per 32 elements (ceil-log2 of amax/448), computed row-chunked so the fp32 transient stays at ~32 MB. They are then served by the existing mxfp8_linear W8A16 path, which runs on any GPU. The design:

• A QuantConfig overlay (LoadTimeQuantConfig) reports MXFP8 for name patterns, so modules build MXFP8 buffers.
• The family reader quantizes the bf16 tensors before fusing shards.
• ModelSpec.load_quant_targets declares the groups per family; qwen4_exp declares dense, hc and lm_head.
• Flags --dense-quant, --hc-quant and --lm-head-quant. They are refused for families without targets, for tied heads, for FTW checkpoints and with --dtype float16.

Reference implementation: gdevenyi@92f901321cc0, gdevenyi@7e64750d52e3, plus follow-up fixes on the same branch.

Measured on Qwen3.8-Flash-Next against bf16, teacher-forced over 10,485 positions of real text:

• dense layers MXFP8: +0.0053 ± 0.0038 nats/token
• hyper-connections MXFP8: −0.0019 (noise)
• lm_head MXFP8: +0.0011
• in total about 0.5% perplexity

It frees about 4 GB of VRAM on that model. That is what lets it serve 262K context with ~1130 expert slots on an 11 GB card.

Alternatives considered

Pre-quantized block-FP8 checkpoints exist, but they need the block-FP8 GEMMs, and they cover fewer weights.