GGUF vision towers (mmproj): survey and plan #794

Sub-issue of #671 (GGUF → Vision towers). It records what we found about loading llama.cpp vision towers (mmproj) for the families FreeToken serves images for. Checked against public main 546c5fe and llama.cpp master 339f3242c.

Where it stands

• feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803 (merged): image input from an mmproj for Qwen3.5 / 3.6 / 3.8 and Gemma-4 (26B-A4B, 31B, 12B unified); the behavior is under Behavior.
• feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj #810 (in review): GLM-5.3-Flash (glm5v) on the same base.
• The other families still load GGUF text only.
• llama.cpp always writes the vision tower to a separate file, mmproj-*.gguf:
  • general.architecture = clip, general.type = mmproj.
  • Tensors are named v.* / mm.*; metadata sits under clip.vision.*.
  • The projector type is in clip.projector_type or clip.vision.projector_type.
• llama.cpp loads it with --mmproj FILE. When it downloads a model with -hf, it also picks the sibling *mmproj*.gguf whose quant bits are closest to the text model.
• Old Ollama GGUFs put v.* / mm.* inside the text file. Ollama is migrating them to split files (compatmigrate). None of the HF repos we checked ship that layout.

Per family

Checked by comparing a BF16 mmproj with the HF vision weights after undoing llama.cpp's transforms (CPU only).

Notes:

• Where mmprojs are published:
  • unsloth, ggml-org, bartowski and lmstudio publish one for every family above except MiniMax-M3.
  • For MiniMax-M3, 9 of 26 GGUF repos ship one (bartowski, AesSedai, ...); unsloth does not.
• Weight types:
  • Most mmprojs are BF16, F16 or F32.
  • ggml-org also publishes Q8_0.
  • lmstudio's Muse-Glimmer mmproj is Q4_K_M.
• Image token ids: the mmproj does not store image / boi / eoi token ids. They come from the text GGUF's tokenizer, looked up by token string.
• DeepSeek-V4 specifics:
  • The text GGUF has no image tokens. The img_start / pad / end embeddings live in the mmproj.
  • ggml-org's chat template cannot render images, so the tokenizer needs an image-placeholder fallback.
• Image processors: for Qwen, Gemma-4 and GLM-5.3, an image processor built from the mmproj metadata plus the defaults above gives the same output as AutoImageProcessor on the HF release. MiniMax-M3 and Muse-Glimmer have not been compared yet. DeepSeek-V4 has its own preprocessing.

Precision

Step 1 does not quantize the vision tower. Every mmproj is dequantized to bf16 at load.

Why not quantize:

• Every FreeToken tower is plain bf16. Qwen's tower takes config.quant, but GGUFConfig maps no visual.* names.
• The FP8 / NVFP4 / MXFP8 HF releases we checked all keep vision in bf16.
• The towers are 0.1–3.7 GB, and host placement already covers memory pressure.
• vLLM and SGLang:
  • Neither has a feature that quantizes ViT weights on its own initiative; the tower follows the checkpoint.
  • SGLang hardcodes quant_config=None for the Qwen3-VL / Qwen3.5 and MiniMax-M3 towers.
  • vLLM's GGUF plugin handles mmproj for Gemma3 only, and runs it at the file's types.

Behavior

As shipped in #803, for the families with GGUF image input (Qwen3.5 / 3.6 / 3.8, Gemma-4).

• Only separate llama.cpp mmproj files are read. A text GGUF that carries v.* / mm.* tensors itself (old Ollama layout) serves text, with a warning that those tensors are not read.
• --mmproj FILE picks the file. It must be a llama.cpp vision mmproj and needs a .gguf model. A file the model cannot use (another tower type, another output width, image tokens missing from the text tokenizer) stops startup with an error.
• Without --mmproj, FreeToken looks for *mmproj*.gguf in the model file's own folder only, as llama-server does for a --models-dir entry. It does not look in parent folders, so a repo that keeps each quant in a subfolder and the mmproj at the top needs --mmproj (17 of the 200 most downloaded Qwen GGUF repos).
  • It skips a file that cannot be read, whose output width differs from the text model, or that fails the family's checks; each skip is logged.
  • When the files belong to several models, it takes the one named like the text model. If none matches, the model serves text, with a warning.
  • Among the rest it prefers BF16, then F32, then F16, then quantized files. llama.cpp -hf picks the file whose quant bits are closest to the text model instead.
• No usable mmproj: the model serves text and rejects image requests. --mm-disable vision and --text-model-only leave the mmproj unloaded the same way.
• Families without GGUF image input do not look for an mmproj, and --mmproj has no effect on them.
• Log: one line per process names the mmproj, how it was found, its projector type, its file type -> bf16, and its size. Another line lists the files not used. A quantized file gets a warning. Without an mmproj, the log says vision is disabled and why.
• Precision: the tower runs in bf16. A quantized mmproj is dequantized at load and keeps the quantization error of the file.
• Config: comes from the GGUF files alone, with no HF repo. The mmproj metadata, the text GGUF tokenizer (image token ids) and per-projector defaults supply the values. Image tokens, budgets, --image-min-tokens / --image-max-tokens and --mm-processor-kwargs work as for the HF checkpoint of the same model.
• FTW: an FTW converted from a GGUF serves text only.

Open

• FTW converted from a GGUF keeps no vision tower.
• --mmproj on a family without GGUF image input is ignored without a message.

Work items

• Shared: mmproj discovery, logging, GGUF config with the vision sections, image processor (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803)
• Qwen3.5 / 3.6 / 3.8 (qwen3vl_merger) (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803)
• GLM-5.3-Flash (glm5v): feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj #810
• Gemma-4 26B-A4B / 31B (gemma4v), 12B unified (gemma4uv, vision only) (feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj #803)
• MiniMax-M3 (minimax_m3)
• Muse-Glimmer (muse-glimmer, images only)
• DeepSeek-V4-Flash-Vision-Exp (deepseek4v) (feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj #816)
• Qwen3-VL (qwen3vl_merger with DeepStack; text archs in feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only #807)