gem16

Run Gemma 4 12B or 26B locally on a single 16 GB NVIDIA GPU.
Local agent server, purpose-built CUDA inference engine, and optional native desktop app.

gem16 is a local inference stack built specifically for Gemma 4 on Blackwell GPUs with about 16 GB of VRAM. Gemma 4 12B Unified and Gemma 4 26B A4B Compact Vision are the two equal, user-selectable product profiles. The primary entry is gem16-server with a local coding agent; the optional native C++ desktop app starts or attaches to the server, manages profiles and provides streamed chat. The 12B profile supports text, image, and audio; Compact Vision combines its locked Trellis35 W4A8 text Target with an FP8 E4M3FN Vision module and optional fixed-D2 Assistant.

The 12B profile loads its pinned mixed FP8/NVFP4 Safetensors checkpoint directly. Compact Vision consumes its offline-compiled, immutable GEM16 components. The former public text-only 26B NVFP4 profile remains implemented and qualified internally for regression and rollback, but is not shown as a normal Studio choice.

Important

gem16 0.2.0 is the current stable release for the supported local single-user configuration. The optimized CUDA backend targets Blackwell SM120/SM120a, and the supported model revisions are pinned deliberately. See the v0.2.0 release notes for the tested scope and known limitations.

Product profiles

Profile	Model size	Quantization / components	Download size	Capabilities
Gemma 4 12B Unified	11.95B parameters	Mixed FP8/NVFP4 Target plus Assistant	about 10.2 GB	Text, image, audio, MTP
Gemma 4 26B A4B Compact Vision	25.2B total / 3.8B active parameters	Trellis35 W4A8 Target, FP8 E4M3FN Vision, optional hybrid NVFP4/FP8/BF16 Assistant	12.8 GB without / 13.1 GB with Assistant	Text, images within context capacity, optional fixed-D2 MTP

The Compact Vision weight payloads occupy 13,060,400,408 bytes (12.16 GiB) with the Assistant. Its text Target is 12,204,692,480 bytes, Vision is 597,390,648 bytes, and the Assistant is 258,317,280 bytes.

Server and API

Build the headless server with ./scripts/build.sh --cuda --test on Linux or .\scripts\build.ps1 -Cuda -Test on Windows, then follow the server guide. Run the server independently or let Studio manage it. The server guide contains locked downloads and launch commands for both public profiles. Chat Completions and Responses support streamed text, reasoning and client-executed function tools. Compatibility is the explicit OpenAI Agent Core v1 subset; unsupported fields fail visibly. Networking defaults to local loopback. Use the tested SDK/Pi configuration for agent setup. Compact Vision everyday context: 220,000 Linux / 170,000 Windows tokens, subject to VRAM admission with 200 MiB long-context reserve.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gem16","messages":[{"role":"user","content":"Hello!"}],"max_completion_tokens":128}'

gem16-chat provides terminal chat; gem16-run, gem16-inspect and gem16-bench provide inference, checkpoint inspection and benchmarking tools.

Run the optional desktop app from source

Prerequisites

• A Blackwell NVIDIA GPU with approximately 16 GB of VRAM for the optimized inference path
• The pinned CUDA toolkit and CUTLASS submodule
• CMake 3.28 or newer, Ninja, and a C++20 compiler
• curl on PATH for resumable downloads in the native Models screen (included with current Windows)
• OpenGL/X11 or Wayland development libraries on Linux; the Windows client uses Direct3D 11
• No account or token is required for the public repositories pinned by the current Studio catalog

The exact reference environment is recorded in toolchains/blackwell16gb.lock.

Clone the repository and initialize its pinned dependencies:

git clone --recurse-submodules https://github.com/Danmoreng/gem16.git
cd gem16

Windows

From PowerShell:

.\scripts\build.ps1 -Cuda -Test
.\scripts\run-studio.ps1

The development launcher incrementally rebuilds and uses the workspace CUDA server. Pass -SkipServerBuild only when that native binary is already current; persisted settings pointing at an installed server do not override development launches.

Linux

./scripts/build.sh --cuda --test
./scripts/run-studio.sh

The Linux launcher provides the equivalent --skip-server-build opt-out.

On first launch, Studio opens Models and presents 12B Unified and 26B Compact Vision without a preselected default. Install either profile or both, then select a verified profile and verify the compiled path on Server. Studio reuses already populated Hub blobs, resumes incomplete downloads, and never copies checkpoints into the application archive. The 12B Target spans two existing upstream repositories, so Studio creates a hardlink-only runtime view inside the same Hub cache; it does not publish a mirror or duplicate the payload bytes.

Recorded performance

The 2026-10-24 final Linux text comparison directly measures the frozen day-start engine, the final accepted engine, and freshly built official llama.cpp b11866 (3f03b17724ad) on the RTX 5080 Laptop. Ranges below are the medians from both experiment orders, in tok/s.

Mode	Day-start gem16	Final gem16	Current llama.cpp b11866
ordinary	136.970–137.959	139.607–139.647	143.185–143.408
d2	193.837–194.030	197.304–197.362	198.063–198.135

The direct cumulative gain is 1.22–1.93% ordinary and 1.72–1.79% fixed-D2. Each cohort retains ten runs after three warmups, with identical 16,384 prompt IDs / 1,135 outputs / batch one / greedy sampling. This is a native-format/kernel baseline: Trellis35 W4A8 (12.20 GB) versus official QAT Q4_0 GGUF (14.44 GB), FP8 versus Q8_0 KV, and different generated streams across engines. It does not prove exact numerical or quality parity. File sizes include their format metadata; GGUF embeds tokenizer metadata while the native tokenizer is external. Requested context is 17,519; llama internally pads to 17,664. Both keep all weights on GPU without prompt-cache reuse. Dynamic Boost clocks differ; both modes' point estimates favor llama, with the reverse D2 interval spanning zero. All samples, confidence intervals, VRAM, formats and limitations are in the final comparison report.

The 2026-10-24 Linux text optimization benchmark on the RTX 5080 Laptop GPU records 139.085 tok/s ordinary decode and 196.492–196.555 tok/s fixed-D2 in the earlier FP8 projection qualification campaign. Native FP8 QKV/O row packing adds 0.42–0.58% D2 throughput against fresh controls; ordinary is unchanged within variation. All 60 retained runs reproduce the same 1,135 IDs with unchanged D2 acceptance, arenas and precision. Sampled device-total peaks are 12,455 / 12,725 MiB. The workload uses 16,384 prompt IDs, context 17,519, batch one, greedy, FP8 KV, three warmups and ten retained runs per cohort. See the qualification and raw evidence. Absolute rates vary under firmware-managed Dynamic Boost; compare paired controls. The preceding M1 GateUp and D2 attention gains remain recorded. These text tests do not extend Windows, image or release qualification; the separate final comparison above includes current llama.cpp.

The subsequent NVFP4 output-head trials produce no reproducible end-to-end gain and are not retained in production. The accepted performance figures above remain the qualified baseline. Both variants preserve exact native outputs; all numerical, memory and protected-profile checks for the Target trial pass. See the head experiment evidence.

A final T3 GateUp audit with production-matched RDC builds also rejects all three existing pipeline variants: exact outputs, but no consistent operator gain. The accepted engine implementation and performance baseline remain unchanged.

Earlier public-profile measurements on the same GPU at its firmware-managed 175 W ceiling are retained below. The public Compact Vision text Target recorded 130.906 tok/s ordinary decode and 182.526 tok/s fixed-D2 decode, with 12,522 / 12,794 MiB peak VRAM. These are retained Linux batch-one measurements with a 16,384-token prompt, checkpoint-FP8 KV, three warmups and ten runs; Ordinary and D2 produced the same 1,229 output tokens. They are text measurements, not image throughput or a new HEAD benchmark.

Performance results and reproduction retain the complete 12B Linux/Windows measurements, internal 26B NVFP4 results, competitor comparisons, revisions and semantic caveats. The benchmark contract governs new claims.

Current limits and release status

• Optimized inference targets Blackwell SM120/SM120a and batch one; no continuous batching.
• 12B supports text, image and audio. Compact Vision supports text and images within context capacity, one resident slot, and its optional fixed-D2 Assistant. The internal NVFP4 profile is text-only.
• Studio settings and non-temporary chats persist in SQLite; temporary chats are not retained.
• Full Windows/Linux API qualification, packaging parity and clean-machine onboarding remain release gates. Bounded Compact Vision P20 acceptance does not mean a release has shipped; extended QUAL01 was waived.
• Video, durable server-side conversations and Responses branching are outside the current product.

Documentation and development

Start at the documentation index for user guides, development contracts and historical evidence. Current scope lives in the product contract, active decisions and roadmap. The architecture explains the specialized C++20/CUDA engine and resident execution plans. Development is primarily assisted by AI coding agents; changes remain subject to the repository's correctness, provenance and measured-performance rules.

License

gem16 is licensed under the Apache License 2.0. Vendored dependencies retain their respective licenses. Model weights and tokenizer assets are distributed separately under their respective terms.