gm4 — a Gemma 4 E2B-it specific CPU inference engine

A small, from-scratch C11 inference engine for one model: gemma-4-E2B-it (Q4_K_M GGUF), CPU-only. Modeled on ds4's philosophy — a narrow vertical engine (not a generic GGUF runner): tiny public API, mmap'd weights, every op gated numerically against llama.cpp on the same GGUF. It loads the model, tokenizes natively, runs the exact Gemma-4 forward graph, and generates greedily.

> gm4 -p "The capital of France is" -n 16
The capital of France is Paris.

Why a dedicated engine

Gemma-4 E2B has structural features a generic runner under-uses. gm4 implements them precisely:

• PLE (Per-Layer Embeddings): per_layer_token_embd [262144×8960] projected + mixed into every layer.
• KV-sharing: only the first 15 of 35 layers own a KV cache; the upper 20 reuse layer 15−(swa?2:1).
• Hybrid attention: 4 sliding (window 512) : 1 global, MQA (1 KV head → 8 Q heads), head_dim 256 local / 512 global.
• Dual RoPE: NEOX full rotary, θ=1e4 local / 1e6 global, with a shared rope_freqs freq-factor (division) on global layers.
• Tied, softcapped head (262144 vocab, 30·tanh(x/30)), GeGLU (gelu_pytorch_tanh), V weightless-RMSNorm.

Build & run (Windows, MSYS2 mingw64)

.\build.ps1            # builds build\bin\gm4.exe + every test exe (gcc, cygwin-free, ~7s)
.\build.ps1 test       # full suite (needs the model; set $env:GM4_TEST_GGUF or edit the default)
.\build.ps1 test-fast  # L0 smoke (no model)
$env:PATH = "C:\msys64\mingw64\bin;$env:PATH"   # runtime DLLs
.\build\bin\gm4.exe -m <model.gguf> -p "Once upon a time" -n 32
.\build\bin\gm4_info.exe <model.gguf>           # inspect hparams + 601 tensors

The Makefile is kept for Unix; on Windows use build.ps1 (the MSYS2 mkdir/rm/sh forks intermittently crash, so the driver calls gcc directly).

Correctness (how it's verified)

Every op and forward intermediate is gated against llama.cpp (the reference stack, same GGUF) via full-tensor goldens captured with a patched llama-eval-callback (-fa off -ctk f32 -ctv f32 -t 1). Tests gate on full-vector cosine + max-abs-diff (a transpose-injection test proves the gates have teeth). The tokenizer is gated byte-exact vs llama-tokenize; greedy generation is gated per-step vs llama-completion.

Suite (264+ checks, warning-clean): test_gguf (118), test_ops (49), test_forward (37), test_tokenizer (43), test_generate (23), test_common (17). Per-op intermediates match at cosine 1.0; the assembled forward gives the exact argmax as llama.cpp.

Layout

include/ public API (gm4.h) + module headers · src/ engine (gguf, model, quant, matvec, norm, rope, ffn, attn, tokenizer, forward, generate, cli, info) · tests/ one exe per concern + the golden harness · scripts/ oracle capture (Python, test-time only) · docs/ the spec, architecture, and TDD plan (both adversarial-review rounds applied).

Scope

This round: CPU · GGUF load · native BPE tokenizer · full forward matching the oracle · greedy generation · CLI. Deferred (clean seams exist): GPU/Vulkan/CUDA, HTTP server, sampling (temp/top-p), chat/reasoning/tools, disk-KV, multimodal.

Acknowledgements

ds4 (narrow-vertical philosophy) · llama.cpp / ggml (src/models/gemma4.cpp reference + the golden oracle).