Repository navigation
model: add MiniMax-M3.1 - #320
danielhanchen wants to merge 4 commits into
Conversation
…lead layers, DSpark draft, BPE NFC) - convert: MiniMax-M3.1 checkpoints (MXFP8 dense weights, NVFP4 routed experts). Experts pass through as native NVFP4 with a per-expert .scale tensor by default; --fp8-as-q8 and --outtype f32/bf16/f16/q8_0 dequantize exactly. Raw config.json is read for the sparse attention fields (transformers >= 5.15). - minimax-m3: a layer runs MSA iff it carries indexer tensors (M3.1 enables it on the dense-lead layers), NVFP4 expert scales are passed to build_moe_ffn, and the MSA batch path is chunked below 2^31 elements. - vocab: the BPE session applies tokenizer.ggml.normalizer.nfc, as the Unigram session already does. - DSpark: DSparkMiniMaxDraftModel converter, swigluoai in dflash, per-layer target taps in minimax-m3.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9a210a1ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| raise ValueError(f"MiniMax DSpark: scale {name!r} has no weight") | ||
| w = self.model_tensors[weight_name] | ||
| s = self.model_tensors.pop(name) | ||
| self.model_tensors[weight_name] = lambda w=w, s=s: dequant_mxfp8(w(), s()) |
There was a problem hiding this comment.
When converting the new DSpark checkpoint with --fp8-as-q8, this override dequantizes each MXFP8 weight but never adds weight_name to _fp8_dequantized. ModelBase.tensor_force_quant() only selects Q8_0 for names in that set, so the CLI option silently writes these weights in the inferred F16/BF16 type instead; add the same bookkeeping used by MiniMaxM3Model.dequant_model().
Useful? React with 👍 / 👎.
| try { | ||
| std::string canonical; | ||
| canonical.reserve(text.size()); | ||
| for (const uint32_t cpt : unicode_cpts_normalize_nfc(unicode_cpts_from_utf8(text))) { |
There was a problem hiding this comment.
For an NFC-enabled BPE vocabulary, malformed UTF-8 never reaches the intended catch path because unicode_cpts_from_utf8() catches invalid sequences internally and substitutes U+FFFD. The new normalizer therefore replaces raw invalid bytes before byte-level BPE processes them, whereas the previous tokenizer preserved and tokenized those bytes, so non-UTF-8 prompt data now changes tokenization despite the stated fallback.
Useful? React with 👍 / 👎.
| return self._nvfp4_pack_np(t[0], t[1]) | ||
| w = self._dequant_nvfp4_eager(t[0], t[1], t[2]) | ||
| # torch's float -> bf16 / f16 casts round to nearest even, like gguf.quants | ||
| return w.to(self._DEQUANT_RAW[self.ftype][1]).contiguous().view(torch.uint8).numpy() |
There was a problem hiding this comment.
When --bigendian is used with an explicit F32 or F16 output type, this converts the expert values to typed Torch data and then exposes them to GGUFWriter as uint8. The writer's endian conversion consequently performs a one-byte byteswap, leaving the little-endian multi-byte values unchanged while marking the GGUF as big-endian, so all routed-expert weights are corrupted on load. Preserve a typed NumPy representation or explicitly swap each 2/4-byte value before passing the raw buffer.
Useful? React with 👍 / 👎.
| if m is not None and (self._nvfp4_native or self.ftype in self._DEQUANT_RAW): | ||
| experts.setdefault((int(m.group(1)), m.group(3)), {})[int(m.group(2))] = base | ||
| continue |
There was a problem hiding this comment.
With --outtype q8_0 --fuse-gate-up-exps, Q8_0 is not in _DEQUANT_RAW, so the new packed experts fall through to the inherited conversion path, where the gate and up tensors are replaced by one ffn_gate_up_exps tensor. The MiniMax-M3 tensor loader still requires separate ffn_gate_exps and ffn_up_exps, and its graph passes no fused tensor to build_moe_ffn, so the resulting GGUF fails to load; reject this option combination or add fused-expert loader and graph support.
Useful? React with 👍 / 👎.
Upcast the norm weight to F32 before adding 1 instead of relying on the incoming dtype. prepare_tensors already turns BF16 tensors into F32 before modify_tensors, so BF16 checkpoints convert byte-identically (M3.1 and MiniMax-M3 layout fixtures, DSpark draft, all outtypes); the change keeps the add exact for F16 sources and for callers that pass a raw BF16 tensor, and matches the DSpark draft converter which already adds 1 in F32.
pr-set: pin MiniMax-M3.1 (#320)
danielhanchen
changed the title
Adds MiniMax-M3.1-Flash: https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash
NVFP4with a per-expert1/global_scaletensor (or are dequantized exactly to BF16 / F16 / F32 / Q8_0 with--outtype,--fp8-as-q8for Q8_0 dense weights); MXFP8 tensors decode exactly-ub)tokenizer.ggml.normalizer.nfclike the Unigram one already does (M3.1 declares an NFC normalizer); GGUFs without the key are unaffectedDSparkMiniMaxDraftModelconverter,swigluoaiindflash.cppand per-layer target taps1 + wnorm add runs in F32, so F16 sources no longer lose the low bits ofwTested against transformers in float64 with exact dequantization, on fixtures with the real tokenizer and storage formats: mean KL 5.8e-7 (BF16 source) and 5.6e-7 (NVFP4 + MXFP8), top-1 100%; the MiniMax-M3 layout gives 1.8e-6. The tokenizer matches HF on 3,221 documents and 605 rendered chats. On the real weights, the native NVFP4 GGUF agrees with a reference GPU implementation on 95.5% to 99.1% of top-1 tokens over its own greedy continuations (chat, math, code), and greedy decoding with the DSpark drafter is identical to plain greedy. MiniMax-M3 and MiniMax-M2 GGUFs convert byte-identically and give bitwise-identical logits on this branch. GGUFs: https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF
Run with
llama-server -m MiniMax-M3.1-Flash.gguf --jinja -fa on -np 1(sparse attention needs one slot or--no-kv-unified); for DSpark add--spec-type draft-dspark -md MiniMax-M3.1-Flash-DSpark-BF16.gguf --spec-draft-p-min 0.The branch is based on upstream
b11835, after the BPEtokenize()rewrite in ggml-org#30974.AI Usage
Used Claude and Local Models for testing, iteration and code design - manual verification of model / PR usage