Skip to content

Add MiniCPM5-2B support via llama.cpp #37

Closed

Description

@pirumpi

Proposal

Research date: 2026-09-08. Tapioca revision: efe7a55d6e4641daecf7b35c9e04201c96c29f78.

Qualify MiniCPM5-2B Q4_K_M as an optional compact text/chat model through the existing llama.cpp backend. This is an integration candidate, not a claim of current Tapioca support. No full-weight inference or platform smoke test was performed in this review.

Identity, release and rights

  • Official model card
  • Official release changelog: released September 7, 2026; do not confuse this with earlier Hub repository creation dates.
  • Apache-2.0 license; official Hub repositories are public and ungated. Retain applicable attribution/notices; no account/token approval is needed for these downloads.
  • Dense LlamaForCausalLM, approximately 2.52B total parameters, 42 layers, 16 attention heads and 2 KV heads, head dimension 128. English/Chinese text model, not a vision model. Native context metadata is 131,072 tokens; this is not a practical desktop memory guarantee.

Exact initial artifact

Official GGUF revision

  • Repository: openbmb/MiniCPM5-2B-GGUF
  • Revision: f76b778717db10cae6f2ea6ffb0e86ce1f31d08d
  • File: MiniCPM5-2B-Q4_K_M.gguf
  • Size: 1,561,318,368 bytes (~1.45 GiB)
  • Hub-reported SHA-256: ec2d5801640099e97d8d7e8003ad4d81f336e757811f03a26173dddf386602fd
  • Exact download

The exact URL returned HTTP 206. Parsing its header/metadata verified GGUF v3, 381 tensors, general.architecture=llama, tokenizer.ggml.pre=minicpm5, and Q4_K_M file type. This was a bounded range read, not a full-file checksum verification. Verify the complete hash during qualification.

Backend and platform path

OpenBMB's deployment guide documents these GGUFs for vanilla llama.cpp, consumer GPUs and CPU/edge machines, with an 8K-context Jinja server example.

Tapioca pins b10603. Its tokenizer implementation includes minicpm5; upstream tokenizer PR #23384 merged May 27. GitHub ancestry comparison confirms the merge is included in b10603. Tapioca already launches llama-server with --jinja.

Qualification targets: CPU, Apple Silicon Metal, and the shipped Windows/Linux Vulkan bundles. Test Windows ARM64 CPU separately. These are proposed paths, not newly validated platforms. Do not bump the runtime unless a reproduced failure requires it.

Out of scope: MLX/MLX-VLM (the separate official MLX artifact is not this GGUF), Diffusers, ONNX, DirectML, CUDA-specific bundle packaging, vision/audio/video, DSpark speculative decoding, and changes to default model recommendations. Upstream CUDA availability does not mean Tapioca ships a CUDA language-model bundle.

Hardware guidance

Start qualification at 8 GiB system/unified memory, 16 GiB preferred, with 8K context and bounded generation. These are conservative test targets, not measured minimum requirements. GPU use also needs space for weights, KV and compute buffers; CPU fallback remains a qualification target.

Derived from the official architecture, FP16 K+V cache is approximately 42 × 2 × 128 × 2 × 2 = 43,008 bytes/token: ~336 MiB at 8K, 1.31 GiB at 32K, and 5.25 GiB at 128K, before allocator/runtime overhead. Measure actual resident memory; weight size alone is insufficient. Do not advertise 128K desktop operation without testing.

Implementation and acceptance

  1. Add an optional text entry with exact artifact integrity metadata in built-in and public catalogs; regenerate the manifest checksum only after tests pass.
  2. Preserve the official template and termination tokens. Initially advertise basic chat only; qualify reasoning and XML tool-call behavior separately.
  3. Test clean download, hash mismatch, offline reuse, model load, system messages, multi-turn English/Chinese, cancellation, streaming and bounded EOS termination on b10603.
  4. Test CPU/Metal/Vulkan on the intended bundles, recording hardware, memory, latency and output at 8K/32K. Keep 128K and structured tools unadvertised until independently qualified.
  5. Compare Q4 with official Q8/reference quality before recommending it over existing small models. Avoid silently enabling long reasoning for short chat requests.
  6. Run model-watch tests, live artifact checks, manifest validation/hash, catalog/adapter tests, plus affected app/server/desktop tests.

Risks and field evidence

Tokenizer recognition is necessary but does not prove generation correctness. Template compatibility, stopping behavior and quantization quality remain acceptance risks, not established upstream blockers.

Reddit evidence is non-authoritative: release discussion shows interest in small local assistants; CPU Q8 report reports about 6 tok/s with quality complaints; comparison thread reports excessive reasoning on an M4. These are not controlled benchmarks or proof of a Tapioca bug; use them to shape tests.

Duplicate review covered all open/closed issues and searches for MiniCPM, minicpm5, llama.cpp and the proposed title. #32 covers Granite, not this model. Historical #29 has no matching MiniCPM finding and was not changed.

Activity

  1. pirumpi commented on Sep 18, 2026

    @pirumpi
    ContributorAuthor

    Implementation is in draft PR #41: current official MiniCPM5 Q4 artifact pinned by revision and SHA-256, with fresh pull/offline reuse/integrity tests, bounded 8K default, Mac CPU/Metal smoke, Chinese and multi-turn chat, and Tapioca serve/chat validation. Keeping this issue open until Windows/Linux and smaller-memory qualification passes.

  2. pirumpi commented on Sep 18, 2026

    @pirumpi
    ContributorAuthor

    The current implementation is now in draft PR #43, rebased on the released v0.12.0 main branch. It supersedes #41 and preserves the newer pinned-download schema and existing Granite 4.2 support. Full SHA-256, Mac CPU/Metal chat, cached Tapioca pull and local tests passed; Windows/Linux Vulkan and low-memory qualification remain, so this issue stays open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions