Skip to content

Add Granite 4.2 3B support via llama.cpp #32

Open

Description

@pirumpi

Summary

Add IBM Granite 4.2 3B as a compact multilingual reasoning/tool-calling text model through Tapioca's existing llama.cpp backend.

Research and local compatibility validation date: 2026-09-02

Official sources and pinned artifact

Initial catalog artifact:

  • granite-4.2-3b-Q4_K_M.gguf
  • Size: 2,244,012,160 bytes (~2.09 GiB)
  • SHA-256: 20e436143017578687f7f848225cc6c6038126c84149192229c7dff6e4e0f427
  • Official, ungated IBM repository; Apache-2.0 license.

The repository also has Q2/Q3/Q4/Q5/Q6/Q8/BF16 artifacts, but this issue intentionally scopes the first entry to Q4_K_M. Additional quantizations should be added only after quality/memory testing.

Architecture and capabilities

  • GraniteForCausalLM, decoder-only dense transformer.
  • 3B parameters; 40 layers; width 2,560; 40 attention heads and 8 KV heads; SwiGLU; RoPE.
  • Native 128K context with an upstream long-context extension to 512K.
  • Built-in thinking, non-thinking, and low-effort modes.
  • Reasoning-aware tool calling and FIM tokens.
  • Twelve officially tested languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese.
  • The official generation recommendation is temperature 1.0 and top-p 0.95.

Pinned-runtime compatibility

Tapioca bundles llama.cpp build b10603 (c060ca974).

A local smoke test downloaded the exact Q4_K_M artifact and successfully loaded/generated with Tapioca's bundled b10603 Apple Silicon binary:

  • Host: Mac mini, Apple M4 10-core, 24 GB unified memory.
  • Detected model: granite, Q4_K_M, text modality.
  • Short-prompt result: prompt processing 24.2 tok/s; generation 40.9 tok/s.
  • The official GGUF was generated with b10622. Comparing b10603...b10622 shows no Granite architecture/model-loader changes, and the exact artifact loaded under b10603.

This verifies a plausible current-pin path, but catalog support still requires Tapioca-level tests.

Supported target path:

  • llama.cpp CPU on macOS, Windows ARM64, Linux, and Windows x64.
  • Metal on Apple Silicon.
  • Vulkan on Tapioca's Linux and Windows x64 bundles.

Not implied or in scope:

  • MLX/MLX-VLM, Diffusers, ONNX, CUDA-specific packaging, DirectML, or multimodal input.
  • Granite 4.2 8B/30B and Granite hybrid/Mamba variants.
  • Guaranteed 512K operation on consumer hardware.
  • Automatic structured tool-call parsing until Tapioca validates the template and streaming behavior.

Hardware and context guidance

  • Q4_K_M weights are ~2.09 GiB.
  • Use 8 GiB minimum / 16 GiB recommended for practical local operation with moderate context and application overhead.
  • The architecture's FP16 KV cache is approximately 80 KiB/token before allocator overhead: roughly 2.5 GiB at 32K and 10 GiB at 128K.
  • Default Tapioca context should remain conservative (8K–16K) until memory/latency tests cover 32K and 128K.
  • Reddit users report that full 128K context can require aggressive KV-cache quantization even on high-memory GPUs; this must be reproduced rather than treated as authoritative guidance.

Implementation plan

  1. Add a granite-4.2 text catalog entry pinned to the official IBM GGUF revision and Q4_K_M artifact.
  2. Set the initial default to Q4_K_M with explicit file size, memory, and CPU/GPU guidance.
  3. Preserve the official chat template and validate thinking on/off plus low-effort controls where Tapioca exposes them.
  4. Validate system messages, multi-turn chat, UTF-8 multilingual input, FIM tokens, and tool-call serialization.
  5. Exercise Metal, Vulkan, and CPU bundle paths without changing the llama.cpp pin unless a failing platform proves a newer build is required.
  6. Add Q5_K_M only if the quality gain justifies its 2.61 GB artifact and platform tests pass.

Acceptance tests

  • Catalog manifest pins the exact repository revision, filename, size, and SHA-256.
  • Catalog schema, checksum, artifact-resolution, and adapter tests pass.
  • Clean-download and offline-after-cache flows work.
  • Deterministic smoke generation passes with bundled llama.cpp b10603.
  • Thinking enabled, thinking disabled, and low-effort prompts terminate correctly without leaking malformed template tokens.
  • Multi-turn/system-message history and Unicode prompts render correctly.
  • OpenAI-compatible tool definitions produce parseable tool calls, or tool calling is explicitly excluded from advertised features.
  • FIM prefix/middle/suffix behavior is tested before FIM is advertised.
  • CPU, Metal, and Vulkan platform smoke tests pass; unsupported acceleration paths are clearly reported.
  • Memory and throughput are measured at 8K, 32K, and 128K context; 512K remains unadvertised unless independently qualified.
  • Existing text-model and catalog tests remain green.

Risks and blockers

  • The official card's 512K claim is a long-context extension; native context is 128K, and consumer KV-cache cost is high.
  • Thinking/tool-call templates are more complex than plain chat and need end-to-end Tapioca validation.
  • Model-card benchmarks are upstream results; quality relative to Tapioca's Phi-4 Mini and Qwen 4B options should be measured before changing defaults.
  • Do not confuse this dense GraniteForCausalLM checkpoint with the separately tracked Granite 4 hybrid/Mamba architecture.
  • Cross-platform load success is not established by today's Apple Silicon smoke test alone.

Reddit field evidence (non-authoritative)

These reports are discovery and real-hardware signals only. Tapioca's own platform, memory, template, and quality tests determine support.

Decision

Proceed with the scoped 3B Q4_K_M integration. Do not change platform defaults or advertise long-context/tool-call support until the acceptance tests pass.

Activity

  1. pirumpi commented on Sep 5, 2026

    @pirumpi
    ContributorAuthor

    Implementation update — 2026-09-05: Added the exact IBM Q4_K_M artifact as built-in Granite text chat, pinned revision/size/SHA-256, preserved embedded template and an 8K model-specific default. The actual Tapioca downloader verified the artifact. Bundled b10603 generated successfully on Mac Metal and Windows CPU/Vulkan. Tapioca API tests passed system/multi-turn Spanish history, thinking enabled/disabled and low-effort termination. Tool calling, FIM and long-context operation are explicitly unadvertised. Full Go tests/vet and 92 desktop tests pass on both hosts; Electron search/filter/details/Escape smoke passes on both. v0.12.0 installer build is running. Leaving the broader memory/context and untested-platform checklist open. Measurements and sources: https://github.com/kyndlo/tapioca/blob/efe7a55/docs/guides/model-integration-qualification.md

  2. pirumpi commented on Sep 18, 2026

    @pirumpi
    ContributorAuthor

    Implementation is in draft PR #41: current official Granite Q4 artifact pinned by revision and SHA-256, with full-file hash verification, bounded 8K default, Mac CPU/Metal smoke, and Tapioca serve/chat validation. The upstream Q4 artifact changed since this issue was filed, so the PR uses the current verified digest. Keeping this issue open until Windows/Linux and smaller-memory qualification passes.

  3. pirumpi commented on Sep 18, 2026

    @pirumpi
    ContributorAuthor

    Status audit (2026-09-18): v0.12.0/main already ships the scoped Granite 4.2 Q4_K_M catalog entry, pinned original issue revision and SHA, 8K default context, download integrity checks, and tests. Draft #41 was superseded because it duplicated this released implementation. I am leaving this issue open for its remaining explicit acceptance criteria: cross-platform Vulkan/runtime smoke, thinking/tool/FIM validation, and measured 8K/32K/128K memory and throughput. Those are not established by a passing build alone.

  4. pirumpi commented on Sep 18, 2026

    @pirumpi
    ContributorAuthor

    Additional Mac Metal context-memory smoke (2026-09-18): downloaded the exact pinned 47a3d969 GGUF and verified its 2,244,012,160-byte size and SHA-256 20e43614…f427. With bundled llama.cpp b10603, one slot, 99 GPU layers, 8 threads and a short deterministic chat that answered 4, post-load process RSS was 2,865 MiB at 8K context, 4,789 MiB at 32K, and 12,478 MiB at 128K. Load timings were ~1, ~1, ~2 seconds with a warm local file cache; do not treat these as cold-start or peak/system-memory figures. This demonstrates why 128K should not be a low-memory default. Windows/Linux inference, long-input quality, tool/FIM behavior and full peak-memory measurement remain, so the issue stays open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions