Support nvidia/Qwen3.8-Flash-Next-FP4-Local (Qwen4Exp, MIXED_PRECISION CSF NVFP4 + MXFP8 + FP4 PLE table) #817

Before you start

• I have checked the supported models and this checkpoint is not listed there. A listed checkpoint that fails is a bug; use a Bug report instead.
• I have read the Roadmap and this model is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.
• This is not a GGUF checkpoint. GGUF support is on the Roadmap; please do not open an issue for it.

Hugging Face link

https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local

Is the model architecture already supported

Yes, but this checkpoint or quantization does not load (architecture Qwen4ExpForConditionalGeneration is supported; the Qwen3.8-Flash-Next-NVFP4 release loads and serves fine).

Is the quantization already supported

Yes, but this checkpoint's weight format does not load. It is a QAD MIXED_PRECISION ModelOpt checkpoint: NVFP4 routed experts with CSF slab scale encoding, MXFP8 dense projections, and an FP4 (packed e2m1) PLE n-gram embedding table.

What happens when you load it

On v0.2.0, ft serve --model <ckpt> --moe-strategy offload --moe-cache-auto fails through a chain of gaps:

1. The 21 shards live in a tensors/ subdirectory while model.safetensors.index.json sits at the root. iter_weight_files() globs the top level only, so the dense scan silently loads 0 tensors (tqdm shows 0it).
2. Routed experts store their scales as CSF slabs (*.weight_scale.nvfp4_csf_fixed U8 + *.weight_scale.nvfp4_csf_exceptions U32). The parallel expert loader raises KeyError: 'U32' (_ST_DTYPE has no unsigned integer types), and there is no CSF decoder.
3. The expert stream then stalls at "Loading experts (parallel): 65.1G/69.6G": every input_scale tensor of the checkpoint is in the last shard, so the first finished piece waits for the last shard while ~65 GB of weights pile up unread.
4. The MXFP8 dense projections store their block scales as weight_scale, not the weight_scale_inv the ModelOpt dialect expects:

WeightLoadError: ... KeyError: 'model.layers.0.linear_attn.in_proj_qkvz.weight_scale_inv'

5. The PLE n-gram table is packed e2m1 (80-byte rows for a 160-wide head) with per-16 e4m3 scales and a global weight_scale_2 scalar; the loader and both PLE backends assume native fp8-e4m3 rows:

NotImplementedError: the disk PLE backend reads native F8_E4M3 rows; shard ... holds packed e2m1 (NVFP4 table) -- use the default pinned backend

Anything else

Other engines support this checkpoint (NVIDIA's local-inference-lab vLLM/flashinfer forks), which is where the CSF slab layout and the packed-table layout are defined.

We have a working implementation on our fork (chrisqianz/FreeToken): a CSF slab scale decoder, activation scales read out of band so the expert stream is not blocked by the last shard, U16/U32/U64 in _ST_DTYPE, scheme-aware renaming of the checkpoint's weight_scale to the model's weight_scale_inv, and a full FP4 PLE table path (CPU LUT + Triton kernel, with bit-exact cross-oracle tests). CPU test matrix: 92 passed (+12 loader) on the model's venv; end-to-end serving verified on RTX 5090 D (driver 595.84), Ubuntu 22.04.5, Intel Core Ultra 7 265K, 125 GB RAM, torch 2.14.1+cu130, transformers 5.18.0, via

ft checkpoint --model <hf-dir> --out <ftw-dir>
ft serve --model <ftw-dir> --moe-strategy offload --moe-cache-auto --ple-backend pinned ...

Two PRs to follow (loader subdirectory fix; the FP4-Local support itself, superseding #708).