My local inference recipes for AMD Ryzen 5 5600X · RTX 3060 12 GiB · 32 GiB RAM: model files, engine pins, automatic launchers, agent clients and measured performance. Text only, one generation at a time. Published with sanitized evidence and reproducible validation commands.
Inventory and full-context validation: 2026-10-24. Historical comparisons from October 22–23 are retained separately. These are the selected profiles among the settings tried on this machine, not proof of a global optimum.
- Models and performance
- Hardware and scope
- Quick start
- Model recipes
- OpenAI compatible API
- OpenCode, Hermes, Codex and Claude Code
- FreeToken Desktop
- Logs and reproduction
- Repository layout and licenses
Context includes input, history, templates, reasoning and output. All five published profiles were tested with their full input budget plus 8192 actual generated tokens on 2026-10-24. The main table now reports those long requests, using the same HTTP timing protocol for every engine. Each long measurement is one run after a warmup, not a statistical average.
| Model | Engine | Total context | Input budget¹ | Output¹ | Drafter | Prefill² | Decode² | Measured workload | Quantization | Sources / config |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.6 35B A3B | FreeToken | 66560 | 58368 | 8192 | No | 1739.20 | 60.39 | 58368 + 8192 | NVFP4 ModelOpt (mixed) | Original · Weights · Config |
| Gemma 4 26B A4B | FreeToken | 66560 | 58368 | 8192 | No | 884.26 | 30.20 | 58368 + 8192 | NVFP4 ModelOpt (mixed) | Original · Weights · Config |
| GPT-OSS 20B | FreeToken | 66560 | 58368 | 8192 | No | 1444.89 | 34.93 | 58368 + 8192 | MXFP4 | Original · Weights · Config |
| Flint 39.8B A2.5B | FreeToken | 73728 | 65536 | 8192 | No | 1521.29 | 50.33 | 65536 + 8192 | NVFP4 GGUF official | Original · Weights · Config |
| NVIDIA Nemotron 3.5 Bolt E4B | llama.cpp | 65536 | 57344 | 8192 | BF16 DFlash | 1840.76 | 142.95 | 57344 + 8192 | Unsloth UD-Q8_K_XL GGUF (mixed) | Original · Weights · Config |
¹ Input includes system/template/tool overhead. Output is the normal 8192 default and was forced to that length with ignore_eos only for this throughput test. Normal use respects EOS. Counts use API token usage, including special tokens without visible text. Nemotron allocates 65792 physical slots with 256 reserved internally, leaving 65536 usable by clients.
² Effective prefill = uncached prompt tokens / time to the first nonempty content or reasoning event; decode = (completion tokens − 1) / first-content-to-terminal-usage duration, including non-rendered trailing tokens. Includes local serving overhead and token/chunk accounting effects. Prompts contain this repository’s source code with distinct synthetic prefixes. Equal methodology does not make different models’ tokenizers or workloads identical. These tests establish capacity and serving behavior, not general coding quality.
Full results, cache reuse, tool round trips and API checks · Machine-readable summary · Complete requests, SSE timelines, server logs and GPU traces. Earlier short-prompt and engine-periodic figures remain in the per-model evidence as historical series.
Ryzen 5 5600X, 6 cores / 12 threads, 32 GiB nominal RAM, RTX 3060 12 GiB (SM86), NVIDIA driver 615.78.08. FreeToken uses CPU/GPU hybrid expert offload; Nemotron fits target and drafter entirely on the 3060. Hardware · Bandwidth microbenchmarks.
An RTX 2070 8 GiB is also installed. Earlier mixed GPU experiments were affected by thermal throttling (88°C / 300 MHz), so their results are not presented as an optimized recipe. The launchers select the 3060 by UUID and expose it as device 0. NVFP4 storage on Ampere does not imply Blackwell native FP4 instructions.
Requirements: Linux x86_64, Python 3.12+, a C/C++ compiler for Triton host stubs, Git, an NVIDIA driver compatible with the selected CUDA runtime, enough disk/RAM, and uv for FreeToken. FreeToken uses pinned CPython 3.12 wheels and may need a CUDA 13 toolkit for uncached kernels. Nemotron uses the official llama.cpp b11828 CUDA 13.4 binary/runtime archives, with verified SHA-256 hashes.
gh repo clone jesusjodar/recipes
cd recipes
./recipes list
./recipes install
./recipes download gemma4
./recipes serve gemma4
# Independent engine for Nemotron; no FreeToken installation required.
./recipes install --llama-cpp
./recipes download nemotron
./recipes serve nemotronRun one server at a time, wait for LISTO, then connect a client. Stop other model servers to free VRAM. Ctrl+C cleans up the processes started by the launcher. FreeToken needs both the API port and port + 1; llama.cpp needs the API port.
./recipes serve gemma4 --port 22420
./recipes serve gemma4 --gpu GPU-YOUR-CARD-UUID
./recipes show gemma4
./recipes serve nemotron --print-commandDownloads default to ~/.freetoken/models; set RECIPES_MODEL_DIR for another library. Nemotron also reuses the existing ~/Desktop/AI/models files when present and hash verified. Its engine archives can be reused from ~/Desktop/AI/bin. All five current targets have full-file SHA-256 verification against their pinned Hub blobs; the original historical runs remain separately dated. Existing local shards are reused after the recorded integrity checks. Nemotron's drafters are also hash verified. Inventory · Runtime pins.
66560 total context, 3328 expert slots, four GDN/Mamba slots, hybrid fetch 1. The full test completed 58368 input + 8192 output. An 8192 prefill chunk ceiling caused GDN workspace OOM; the validated recipe uses 2048, retaining the context and expert cache. Start with 1024 experts and rebuild. Desktop’s stored swaTokens=2048 is unused for this recurrent model and is omitted from the CLI rebuild. Sampling: temperature 1, top-p 0.95, top-k 20. Full long-context tool and prefix-reuse results are in the validation report.
./recipes install
./recipes download qwen3.6
./recipes serve qwen3.6Vision and audio disabled. 66560 total context, 448 experts, hybrid fetch 1, prefill chunks 1024. The CLI retains its initial SWA pool of 13312; Desktop stores 1024, so those geometries are distinct. The earlier 8192 prefill ceiling OOM is preserved. The new test completed 58368 input + 8192 output with the published CLI configuration. Normal sampling: temperature 1, top-p 0.95, top-k 64. The old 1425-expert profile did not fit the larger context.
./recipes install
./recipes download gemma4
./recipes serve gemma4MXFP4, 66560 total context, 256 experts, hybrid fetch 1. The new test completed 58368 input + 8192 output. The previous 535-expert profile at 16K left insufficient space for the larger context. CLI SWA pool: 13312; Desktop: 512. The engine uses Harmony and checkpoint sampling defaults. Throughput tests request temperature 0. The validation report records the actual long tool round trip and API outcomes.
./recipes install
./recipes download gpt-oss
./recipes serve gpt-ossOfficial NVFP4 GGUF, pinned revision 6ec43e19…. 65536 input + 8192 output. Final expert cache 3456, SWA 5120, prefill chunks 4096, memory ratio 0.90, CUDA normalization and prefill D2D hits. Start with 1024 experts and rebuild after loading.
./recipes install --flint
./recipes download flint
./recipes serve flintSupport is proposed in FreeToken #797. The installer pins reviewed Python commit 6ee048e…, reusing wheel native extensions; the new long-context test was run on this exact installed commit. The table below preserves the earlier edfba5e… measurements as a historical comparison. This is not a clean CUDA source build.
| Historical workload | Effective prefill | Decode | TTFT |
|---|---|---|---|
| 8192 + 256, cold, mean of 3 | 2783.31 | 66.53 | ≈2.94 s |
| 65536 + 8192, cold, one run | 1543.59 | 49.97 | 42.46 s |
| 65536 + 256, reused prefix | 65535 tokens reused | 50.38 | 0.88 s |
Only the throughput benchmark forces ignore_eos; normal serving respects EOS. A separate >64K tool cycle completed, which does not establish general coding quality. Historical peak observed VRAM: 10.61 GiB including desktop; host experts: about 19.8 GiB. Sampling: temperature 1, top-p 0.95, top-k 20. Profile · Results · Sweep and validation logs · Clients.
A separate FutureMLS-Lab Flint ModelOpt export, revision 56f37fcf…, is also present in the research library, locally reshaped into per-layer safetensors. It was used for parity, AIME and cache validation at 8192 serving capacity, with no optimized long-context throughput or output target. It is inventoried, not advertised as another optimized recipe. Validation logs.
llama.cpp b11828 CUDA 13.4, Unsloth UD-Q8_K_XL target and official NVIDIA BF16 GGUF DFlash drafter. Draft max 7 (block size 8), full GPU offload for both, Flash Attention, Jinja, one slot, fitting disabled, and context shift explicitly disabled. Physical allocation 65792, including 256 internal DFlash slots, exposes a usable client context of 65536, tested as 57344 input + 8192 actual output. No vision projector. Temperature 1, top-p 0.95 in normal use; the fixed-length throughput test requests temperature 0.
Historical short coding tasks at 64K capacity achieved 1964.10 engine prefill / 108.04 engine decode / 106.86 API output tokens/s, with 277–411 input and 1242–1762 output tokens. Those are separate short-context metrics, not the new full-context result. Four-task 8K tests gave 117.63 API / 121.63 engine decode with BF16 versus 85.34 / 87.52 with the downloaded Unsloth Q8_0 drafter. BF16 is selected; none of these are native NVFP4 drafter results. Generated historical patches were not verified correct.
The portable endpoint is 1919. The existing Desktop/AI service uses 8000; the validation runner temporarily paused it when idle and restored it afterward.
./recipes install --llama-cpp
./recipes download nemotron
./recipes serve nemotronEvery launcher binds to 127.0.0.1, without a required API key or public exposure. Model IDs: qwen3.6, gemma4, gpt-oss, flint, nemotron.
curl --fail-with-body http://127.0.0.1:1919/v1/models
curl --fail-with-body http://127.0.0.1:1919/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"nemotron","messages":[{"role":"user","content":"Summarize this project."}],"max_tokens":8192,"stream":true}'from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:1919/v1", api_key="local-no-key")
r = client.chat.completions.create(
model="nemotron", max_tokens=8192,
messages=[{"role": "user", "content": "Summarize the diff."}],
)
print(r.choices[0].message.content)local-no-key is a public SDK placeholder. The OpenAI SDK is included in the FreeToken environment; for a llama.cpp-only installation, install it separately (uv run --with openai your_script.py). Both engines expose /v1/responses and /v1/messages; the pinned llama.cpp routes are documented here. These local adapters do not imply full OpenAI/Anthropic feature parity. FreeToken Responses is stateless: resend history and tools, without relying on stored responses, previous_response_id or background tasks. Preserve reasoning fields where required.
Start a recipe first. Examples below use Nemotron; replace the ID, client folder and context when switching models. Every model has its own files in clients/. These are configuration examples and endpoint checks, not validated full coding sessions with every client.
| Client | Protocol | Files |
|---|---|---|
| OpenCode v2.1.4 | Chat Completions | v2 JSON |
| OpenCode v1 | Chat Completions, different schema | v1 JSON |
| Hermes Agent | Custom provider, Chat Completions | YAML |
| Codex CLI 0.164.0 | Responses | Wrapper · TOML |
| Claude Code 2.1.310 | Anthropic Messages adapter | Wrapper |
Merge the matching JSON into the working project's opencode.json, keeping existing settings. v2 uses providers/package/settings; v1 uses provider/npm/options. Choose recipes/nemotron in /models, or:
opencode run --model recipes/nemotron "Summarize the project structure"The files declare text, tools, reasoning field, context/input/output budgets and max_tokens=8192.
Install through the official guide, then choose Custom endpoint with hermes model, or:
hermes config set model.provider custom
hermes config set model.default nemotron
hermes config set model.base_url http://127.0.0.1:1919/v1
hermes config set model.api_mode chat_completions
hermes config set model.context_length 65536
hermes chatDo not add obsolete model.max_tokens. The server default is 8192. Contexts: Nemotron 65536; Qwen/Gemma/GPT 66560; Flint 73728.
bash clients/nemotron/codex.shThe wrapper overrides the provider for that process, retaining global login/configuration. It uses Responses, no OpenAI authentication or WebSockets, and compacts before exhausting the input budget. The TOML shows the same options for manual integration.
bash clients/nemotron/claude-code.shUse the root URL without /v1; the client appends /v1/messages. All model aliases point to the same local model. Non-Claude models are not officially supported by Anthropic; behavior depends on adapter/model compatibility. No extra Chat Completions→Messages proxy is required for these two pinned engines. Approval controls remain enabled. Detailed client notes and official sources, in Spanish.
./recipes desktop gemma4
# Close Desktop before applying.
./recipes desktop gemma4 --applyPreview first; --apply backs up the file and merges only that model's cache profile and text-only flag. It does not install Flint support or change global settings, engine or Downloads cards. Nemotron's measured recipe uses llama.cpp and has no FreeToken Desktop profile. Captured globals and CLI/Desktop differences.
- Log manifest: recovered sources, SHA-256 hashes and sanitization. Home paths, workspace paths, GPU UUIDs and terminal colors are removed; personal conversations, credentials, voice artifacts and network details are excluded.
- Qwen, Gemma, GPT-OSS: new full server logs and per-request SSE captures, plus recovered historical output where available. Gemma's failed 8192-prefill attempt is retained.
- Nemotron: original JSON measurements, benchmark scripts and reports. Historical complete server stdout was not retained; it cannot be reconstructed faithfully from summaries.
- Flint: original sweep logs, JSON and GPU samples; separate validation.
- Method and caveats · Recovery audit · Runtime pins · Key dependency constraints.
The original temporary Gemma/GPT log files were lost. Terminal captures are explicitly labeled as recovered and can retain original truncation; fresh measurements are separate evidence, not replacements masquerading as historical originals. null means unknown. Model weights are never committed.
To reproduce, run one model with the recorded geometry and log its stdout/stderr, then use the synthetic benchmark (FreeToken environment required):
./recipes serve gemma4 > /tmp/gemma4-server.log 2>&1
# In another terminal, after LISTO:
.venv/bin/python scripts/benchmark-freetoken.py gemma4 \
--output /tmp/gemma4-new-measurement.jsonRecord engine revision, hardware, checkpoint hashes, actual token usage, cache state, EOS policy, warmups and all samples. New measurements use ignore_eos only for fixed-length benchmarking. Verify actual near-limit input/output before claiming full context performance. The installer saves a local uv pip freeze, with all 103 Python package versions pinned in constraints-validated.txt; the new isolated FreeToken environment was installed and used for these tests, including both pinned FlashInfer binary caches.
Stop other model servers to free VRAM, then run:
.venv/bin/python scripts/validate-profiles.py --output /tmp/recipes-validation
python3 -m unittest discover -s tests -vThe runner starts each published profile, builds an exact-budget synthetic prompt using server token counting, generates 8192 tokens, repeats the prefix, performs a natural tool round trip, and checks Responses/Messages. It records full requests, timestamped SSE, stdout/stderr and GPU telemetry. --models qwen3.6 selects one model. For this machine, --pause-user-service nemotron-e4b.service can pause and restore the existing idle service. Detailed outcomes, including any failed checks, are in the validation report.
profiles/ model, engine, launch arguments, context and output policy
clients/ per-model OpenCode, Hermes, Codex and Claude examples
evidence/ measurements, inventory and sanitized logs
scripts/ install, download, serve, Desktop merge and benchmarks
assets/ two original blueprint banners generated with ImageGen
docs/ methodology, recovery, Desktop, clients and artwork provenance
runtime.json engine pins, archive/wheel hashes and Flint commit
Add a profile, pinned source, real benchmark workload/logs, client examples and a table row for each new recipe. Keep reserved capacity, tested workload and intended output separate. Supporting documents are mainly in Spanish; all three READMEs contain the complete usage overview.
MIT covers original recipes/scripts; engines and models retain their own licenses. Nemotron is under OpenMDW-1.1; consult each upstream model card. Banner prompts and provenance: original ImageGen blueprint artwork, not official logos or performance certification.
Sources: FreeToken · llama.cpp b11828 · OpenCode · Hermes · OpenAI configuration · Claude gateway.