Repository navigation
Description
Summary
Add IBM Granite 4.2 3B as a compact multilingual reasoning/tool-calling text model through Tapioca's existing llama.cpp backend.
Research and local compatibility validation date: 2026-09-02
Official sources and pinned artifact
- Official model card: https://huggingface.co/ibm-granite/granite-4.2-3b
- Official release repository: https://github.com/ibm-granite/granite-4.2-language-models
- Official IBM GGUF repository: https://huggingface.co/ibm-granite/granite-4.2-3b-GGUF
- Exact GGUF revision: https://huggingface.co/ibm-granite/granite-4.2-3b-GGUF/tree/47a3d9699d7539606c83943d717fcea7bd9f6a19
- Release date: 2026-08-25
Initial catalog artifact:
granite-4.2-3b-Q4_K_M.gguf- Size: 2,244,012,160 bytes (~2.09 GiB)
- SHA-256:
20e436143017578687f7f848225cc6c6038126c84149192229c7dff6e4e0f427 - Official, ungated IBM repository; Apache-2.0 license.
The repository also has Q2/Q3/Q4/Q5/Q6/Q8/BF16 artifacts, but this issue intentionally scopes the first entry to Q4_K_M. Additional quantizations should be added only after quality/memory testing.
Architecture and capabilities
GraniteForCausalLM, decoder-only dense transformer.- 3B parameters; 40 layers; width 2,560; 40 attention heads and 8 KV heads; SwiGLU; RoPE.
- Native 128K context with an upstream long-context extension to 512K.
- Built-in thinking, non-thinking, and low-effort modes.
- Reasoning-aware tool calling and FIM tokens.
- Twelve officially tested languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.
- The official generation recommendation is temperature 1.0 and top-p 0.95.
Pinned-runtime compatibility
Tapioca bundles llama.cpp build b10603 (c060ca974).
A local smoke test downloaded the exact Q4_K_M artifact and successfully loaded/generated with Tapioca's bundled b10603 Apple Silicon binary:
- Host: Mac mini, Apple M4 10-core, 24 GB unified memory.
- Detected model:
granite, Q4_K_M, text modality. - Short-prompt result: prompt processing 24.2 tok/s; generation 40.9 tok/s.
- The official GGUF was generated with b10622. Comparing b10603...b10622 shows no Granite architecture/model-loader changes, and the exact artifact loaded under b10603.
This verifies a plausible current-pin path, but catalog support still requires Tapioca-level tests.
Supported target path:
- llama.cpp CPU on macOS, Windows ARM64, Linux, and Windows x64.
- Metal on Apple Silicon.
- Vulkan on Tapioca's Linux and Windows x64 bundles.
Not implied or in scope:
- MLX/MLX-VLM, Diffusers, ONNX, CUDA-specific packaging, DirectML, or multimodal input.
- Granite 4.2 8B/30B and Granite hybrid/Mamba variants.
- Guaranteed 512K operation on consumer hardware.
- Automatic structured tool-call parsing until Tapioca validates the template and streaming behavior.
Hardware and context guidance
- Q4_K_M weights are ~2.09 GiB.
- Use 8 GiB minimum / 16 GiB recommended for practical local operation with moderate context and application overhead.
- The architecture's FP16 KV cache is approximately 80 KiB/token before allocator overhead: roughly 2.5 GiB at 32K and 10 GiB at 128K.
- Default Tapioca context should remain conservative (8K–16K) until memory/latency tests cover 32K and 128K.
- Reddit users report that full 128K context can require aggressive KV-cache quantization even on high-memory GPUs; this must be reproduced rather than treated as authoritative guidance.
Implementation plan
- Add a
granite-4.2text catalog entry pinned to the official IBM GGUF revision and Q4_K_M artifact. - Set the initial default to Q4_K_M with explicit file size, memory, and CPU/GPU guidance.
- Preserve the official chat template and validate thinking on/off plus low-effort controls where Tapioca exposes them.
- Validate system messages, multi-turn chat, UTF-8 multilingual input, FIM tokens, and tool-call serialization.
- Exercise Metal, Vulkan, and CPU bundle paths without changing the llama.cpp pin unless a failing platform proves a newer build is required.
- Add Q5_K_M only if the quality gain justifies its 2.61 GB artifact and platform tests pass.
Acceptance tests
- Catalog manifest pins the exact repository revision, filename, size, and SHA-256.
- Catalog schema, checksum, artifact-resolution, and adapter tests pass.
- Clean-download and offline-after-cache flows work.
- Deterministic smoke generation passes with bundled llama.cpp b10603.
- Thinking enabled, thinking disabled, and low-effort prompts terminate correctly without leaking malformed template tokens.
- Multi-turn/system-message history and Unicode prompts render correctly.
- OpenAI-compatible tool definitions produce parseable tool calls, or tool calling is explicitly excluded from advertised features.
- FIM prefix/middle/suffix behavior is tested before FIM is advertised.
- CPU, Metal, and Vulkan platform smoke tests pass; unsupported acceleration paths are clearly reported.
- Memory and throughput are measured at 8K, 32K, and 128K context; 512K remains unadvertised unless independently qualified.
- Existing text-model and catalog tests remain green.
Risks and blockers
- The official card's 512K claim is a long-context extension; native context is 128K, and consumer KV-cache cost is high.
- Thinking/tool-call templates are more complex than plain chat and need end-to-end Tapioca validation.
- Model-card benchmarks are upstream results; quality relative to Tapioca's Phi-4 Mini and Qwen 4B options should be measured before changing defaults.
- Do not confuse this dense
GraniteForCausalLMcheckpoint with the separately tracked Granite 4 hybrid/Mamba architecture. - Cross-platform load success is not established by today's Apple Silicon smoke test alone.
Reddit field evidence (non-authoritative)
- Release discussion and llama.cpp/FIM report: https://www.reddit.com/r/LocalLLaMA/comments/1vy2jz7/ibmgranitegranite4230b_hugging_face/
- 8B memory/KV-cache discussion: https://www.reddit.com/r/LocalLLaMA/comments/1vyfd81/ibm_granite_42_8b_is_52_gb_but_loads_into_27_gb/
These reports are discovery and real-hardware signals only. Tapioca's own platform, memory, template, and quality tests determine support.
Decision
Proceed with the scoped 3B Q4_K_M integration. Do not change platform defaults or advertise long-context/tool-call support until the acceptance tests pass.
Activity
pirumpi commentedon Sep 5, 2026
Implementation update — 2026-09-05: Added the exact IBM Q4_K_M artifact as built-in Granite text chat, pinned revision/size/SHA-256, preserved embedded template and an 8K model-specific default. The actual Tapioca downloader verified the artifact. Bundled b10603 generated successfully on Mac Metal and Windows CPU/Vulkan. Tapioca API tests passed system/multi-turn Spanish history, thinking enabled/disabled and low-effort termination. Tool calling, FIM and long-context operation are explicitly unadvertised. Full Go tests/vet and 92 desktop tests pass on both hosts; Electron search/filter/details/Escape smoke passes on both. v0.12.0 installer build is running. Leaving the broader memory/context and untested-platform checklist open. Measurements and sources: https://github.com/kyndlo/tapioca/blob/efe7a55/docs/guides/model-integration-qualification.md
pirumpi commentedon Sep 18, 2026
Implementation is in draft PR #41: current official Granite Q4 artifact pinned by revision and SHA-256, with full-file hash verification, bounded 8K default, Mac CPU/Metal smoke, and Tapioca serve/chat validation. The upstream Q4 artifact changed since this issue was filed, so the PR uses the current verified digest. Keeping this issue open until Windows/Linux and smaller-memory qualification passes.
pirumpi commentedon Sep 18, 2026
Status audit (2026-09-18): v0.12.0/main already ships the scoped Granite 4.2 Q4_K_M catalog entry, pinned original issue revision and SHA, 8K default context, download integrity checks, and tests. Draft #41 was superseded because it duplicated this released implementation. I am leaving this issue open for its remaining explicit acceptance criteria: cross-platform Vulkan/runtime smoke, thinking/tool/FIM validation, and measured 8K/32K/128K memory and throughput. Those are not established by a passing build alone.
pirumpi commentedon Sep 18, 2026
Additional Mac Metal context-memory smoke (2026-09-18): downloaded the exact pinned 47a3d969 GGUF and verified its 2,244,012,160-byte size and SHA-256 20e43614…f427. With bundled llama.cpp b10603, one slot, 99 GPU layers, 8 threads and a short deterministic chat that answered
4, post-load process RSS was 2,865 MiB at 8K context, 4,789 MiB at 32K, and 12,478 MiB at 128K. Load timings were ~1, ~1, ~2 seconds with a warm local file cache; do not treat these as cold-start or peak/system-memory figures. This demonstrates why 128K should not be a low-memory default. Windows/Linux inference, long-input quality, tool/FIM behavior and full peak-memory measurement remain, so the issue stays open.