cli.md

CLI reference

ft <command> [args]

Command	Purpose
ft serve	Start the API server (OpenAI /v1/*, Anthropic /v1/messages, Responses)
ft shell	Chat with a server in the terminal
ft ctl	Query and manage a running server over HTTP
ft launch	Configure and launch a coding agent against a server
ft checkpoint	Convert an HF checkpoint to the FTW fast-load format
ft bench bw	Benchmark CPU vs PCIe bandwidth to calibrate hybrid MoE decode

ft --version prints the installed version (torch-free; nightly wheels carry a +g<sha> build stamp, tagged releases a bare version). Every command supports --help.

ft serve

ft serve --model <path-or-hf-id> [options]

--model is the only required flag — dtype, attention backend, MoE backend, MoE cache size, KV capacity, CUDA-graph sizes and the tool-call/reasoning parsers all resolve automatically from the checkpoint and the GPU.

Model

Flag	Default	Meaning
--model-path, --model	required	Local dir, HF repo id, or an FTW dir (auto-detected)
--served-model-name	basename of --model	Model id reported by /v1/models
--hf-overrides	—	JSON object applied to the checkpoint's config as vLLM's --hf-overrides: a nested config section updates key by key, any other value is replaced whole. A YaRN rope_parameters override serves original_max_position_embeddings * factor positions

Server & runtime

Flag	Default	Meaning
--host	127.0.0.1	Bind address
--port	1919	Bind port
--api-key	$FREETOKEN_API_KEY, else off	Require the key on every route except /health, as Authorization: Bearer <key> or x-api-key: <key>; other requests get a 401
--gpu	GPU 0	GPU to run on: a UUID from nvidia-smi -L or an nvidia-smi index; see below
--max-running-requests	4	Max concurrently running requests
--max-output-tokens	32768	Default output budget for requests that omit one
--max-seq-len-override	from checkpoint	Max sequence length
--max-prefill-length	8192	Chunked-prefill chunk size in tokens
--cuda-graph-max-bs, --graph	= max running requests	Max batch size captured as CUDA graphs
--decode-log-interval	40	Scheduler status line every N decode steps

Choosing a GPU

For example, a machine with an RTX 5090 and an RTX 3060 Ti:

$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-2f3a9b1c-8d7e-4a05-b6c1-0e5f9a3d7b42)
GPU 1: NVIDIA GeForce RTX 5090 (UUID: GPU-9e8d7c6b-5a49-4f13-8207-c1b0a4e6d3f5)

ft serve --model ... --gpu 1             # by nvidia-smi index -- the 5090
ft serve --model ... --gpu GPU-9e8d7c6b  # the same card by UUID (a unique prefix is enough)

KV cache & memory

Flag	Default	Meaning
--memory-ratio	0.9	Fraction of free VRAM the engine may use (weights + MoE cache + KV)
--num-pages / --num-tokens	auto	KV capacity override in pages / tokens (mutually exclusive; auto sizes from VRAM left after weights and MoE cache)
--page-size	1	KV page size; DSV4 forces 128, the TRTLLM backend needs 16/32/64, SWA models require 1
--embed-device	gpu	cpu keeps the input-embedding table in pinned host RAM; the GPU reads the looked-up rows over PCIe, and the freed VRAM goes to the KV cache (2.37 GiB on Qwen3.8-27B). Decode reads one row per token, so speed is unchanged. A table tied to the LM head stays on the GPU
--cache-type	radix	radix (prefix reuse; SWA/GDN-aware variants picked automatically) or naive
--kv-cache-dtype	bf16	bf16 or fp8: store the KV cache as e4m3 codes plus one fp32 scale per (token, kv head), roughly doubling the tokens that fit in the same VRAM; see FP8 KV cache
--attention-backend, --attn	auto	trtllm/fi/fa/triton/dsv4_sparse/dsa; prefill,decode pair allowed; auto picks per model + GPU

FP8 KV cache

ft serve --kv-cache-dtype fp8 halves the bytes per cached token (8-bit codes instead of 16), so a card that held N tokens holds close to 2N. Each (token, kv head) row keeps its own fp32 scale, which costs ~3% back at head_dim=128. Requirements and trade-offs:

• Needs the triton attention backend; --attn auto selects it (and refuses an explicit fi/fa/trtllm, which cannot be shown to apply these scales).
• Works on the plain paged, hybrid-SWA and QSA sparse (Qwen3.8-Flash-Next) KV pools. On QSA the block-selection index keys stay 16-bit; only the selected K/V rows are read back as codes. MLA/DSA latent KV, DeepSeek-V4's tiered pool and the block-sparse MiniMax-M3 pool stay 16-bit; asking for fp8 there fails at startup rather than silently ignoring the flag.
• The same bytes on every GPU FreeToken targets: the codes sit in a plain byte buffer and are decoded in software, so the cache holds identical data and produces identical numbers on any card (the fp8 type is deliberately kept out of the kernels, which is also what makes the feature work on the RTX 30 series).
• Qwen3.8 FP8-QSA uses the fast scaled16 read path by default. It keeps decoded FP8 tiles in fp16 and applies each row scale around the two dot products, substantially reducing prefill cost on FP8 KV. Set FREETOKEN_QSA_FP8_FAST_SCALE=0 to select the legacy, bit-exact reconstruction; the fast path changes final BF16 rounding order.
• Accuracy is checkpoint-dependent. Expect it to matter most on long contexts and on models with outlier key channels; keep bf16 when a run must be bit-reproducible.
• ft ctl stats / /v1/cache/status report the smaller kv_bytes_per_token, and ft ctl cache --kv N moves the same (now cheaper) pool.

MoE offload

See models.md for what each strategy does.

Flag	Default	Meaning
--moe-strategy	auto	fused/offload/cpu/hybrid; auto → offload; fused on unified-memory GPUs (GB10), see models.md. --moe-backend is the deprecated old spelling
--quant-backend	auto	Kernel per quantized layer type, layer[.kind]=name entries: linear=marlin,moe=b12x or moe.nvfp4=triton. A layer-level entry applies to every kind whose table lists the name
--nvfp4-backend	—	Deprecated: stands in for --quant-backend moe.nvfp4=<marlin|b12x|triton> (flashinfer means b12x); cannot be combined with --quant-backend
--moe-cache-size / --moe-cache-rate / --moe-cache-mem / --moe-cache-auto	auto	GPU expert-cache size as slots / fraction of all experts / memory such as 12GiB / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies)
--moe-cache-scope	unified	unified: one LRU pool of slots per expert quant (quant type and bank layout), so all MoE layers share one pool when they share a quant format; per-layer: each layer has its own pool. per-layer does not combine with --moe-strategy cpu or --moe-cpu-layers
--moe-cache-split	equal	How the pools share the slots: equal (in proportion to their layer counts), or profile:<path>: every pool keeps top_k slots and the rest go to the most used experts in the profile's per-layer expert counts. With offload, a pool smaller than the expert count lowers --max-running-requests to its slots / top_k
--kv-reserve-tokens	8192	KV token floor reserved before --moe-cache-auto fills experts
--moe-cpu-threads	physical cores	CPU worker threads for the cpu/hybrid executor
--moe-cpu-layers	all on GPU	With offload/hybrid: which MoE layers decode on CPU (3,7,11, a count, a fraction, or auto). auto is for hosts that cap CUDA pinned memory (Windows/WSL): it pins layers from the middle out until CUDA refuses one and OS-locks the rest, see models.md; every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs)
--moe-hybrid-max-fetch	auto	With hybrid: max experts fetched over PCIe per layer per step; rest computed on CPU
--moe-prefill-demand-tokens	0	Prefill: a chunk of at most this many tokens loads only the experts it routes to, through the expert cache as decode does (GPU decode only); 0 streams whole layers
--moe-locked-fetch	auto	With hybrid and locked layers: share of a locked layer's misses fetched per step into its GPU slots through pinned staging. Auto takes p / (2p + c) from the ft bench bw row of the layer's kind (at most 1 per step without a row); 0 = locked layers decode on CPU only
--moe-locked-max-fetch	8	Most experts a locked layer fetches per step; sizes the staging (that many rows of the widest expert, per bank)
--moe-collect-stats	off	Log decode MoE stats with each decode log line: hit rate, and per layer-step the misses, experts fetched and experts computed on CPU, for pinned, locked and CPU-only layers
--moe-prefill-hit-d2d	off	Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13)
--disable-moe-prefill-overlap	overlap on	Disable the two-buffer prefill copy overlap

API behaviour

Flag	Default	Meaning
--sampling-defaults	model	Fill unspecified sampling params from the checkpoint's generation_config.json (none = framework defaults)
--tool-call-parser	auto	Tool-call format; auto-inferred from the model family
--reasoning-parser	auto	Splits chain-of-thought into reasoning_content; auto-inferred; off disables
--enable-cache-report	off	Report prefix-cache hits in each response's usage block
--anthropic-inline-system	auto	Placement of late Anthropic system instructions: auto, preserve, or fold

Image input

Experimental. Needs a checkpoint whose family registers a vision encoder (models.md lists them and how each one maps the flags below); a request carrying images is rejected otherwise. Images are accepted on all three protocols (OpenAI image_url, Anthropic image blocks, Responses input_image) as an http(s) URL or base64. Images inside a tool result (an Anthropic tool_result block from Claude Code's Read, a Responses function_call_output from Codex's view_image) are moved to the user turn that follows the tool message, as vLLM does, because chat templates render tool messages as plain text. GET /v1/stats reports what the server accepts as model.input_modalities (["text"] or ["text", "image"]), so a client can gate its attachment controls without reading the checkpoint config.

Flag	Default	Meaning
--text-model-only	off	Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as --mm-disable with every encoder kind
--mm-disable	none	Encoder towers to leave unbuilt (vision, audio); every input they would serve is rejected
--mm-encoder-weights	host	Where the encoder tower's block weights live. host streams them from pinned host banks two blocks at a time behind the compute, so the GPU holds two blocks instead of the whole tower; small images pay the copy time, large ones hide it behind the compute. gpu keeps them resident. An encoder without a block stack stays resident either way. cpu (Qwen VL families) keeps the whole tower in host RAM and runs it in FP32 in a separate CPU process before a request reaches the GPUs: no VRAM at all, see CPU vision encoder
--mm-encoder-threads	one NUMA node's cores	--mm-encoder-weights cpu: torch threads of the encoder process. Defaults to the CPUs of --mm-encoder-cpus, else the physical cores of one NUMA node
--mm-encoder-cpus	not pinned	--mm-encoder-weights cpu: CPU list the encoder process is pinned to, as taskset spells it (22-43, 0-3,8)
--mm-encoder-cache-mb	256	--mm-encoder-weights cpu: host RAM for recently encoded images, keyed by content, so an image a chat client resends on the next turn is not encoded again. 0 disables it
--image-min-tokens, --image-max-tokens	processor defaults	Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units by its processor. A family with fixed budgets honors the maximum only and refuses one below its smallest budget at start-up
--mm-processor-kwargs	none	JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the token budget does not cover; applied after the budget, so an explicit key wins
--mm-embed-cache-device	cpu	Where encoded image embeddings live between prefill chunks. cpu keeps them out of the VRAM budget; cuda skips the copy back
--allowed-media-domains	any	Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain
--allowed-local-media-path	off	Directory file:// image refs may be read from; unset rejects local files

CPU vision encoder

--mm-encoder-weights cpu moves the Qwen VL vision tower out of the engine. A separate process (freetoken-vision-cpu, started with no CUDA device visible) holds the tower once in host RAM in FP32. The tokenizer sends it every request that carries images. It replaces each image's pixels with the tower's output rows and passes the request on to the scheduler. The TP ranks build no tower and load none of its weights, so serving images costs no VRAM. Running requests keep decoding while an image encodes, because the encode happens before the request reaches the GPUs. Text-only requests never pass through the encoder.

• FP32, because CPUs without AVX512-BF16 or AMX run BF16 matmuls slower than FP32, and FP32 stays closest to the reference.
• The process runs at nice 10 and sets its own torch thread count, so an inherited small OMP_NUM_THREADS does not throttle it.
• Images are capped at 1024 tokens (about 1 MP) unless --image-max-tokens says otherwise. The Qwen VL checkpoints allow 16k tokens per image, which is minutes of CPU time.
• Recently encoded images are kept by content (--mm-encoder-cache-mb), so a chat client that resends the same picture every turn pays for it once.

Measured on 2x Xeon Gold 6152 (Skylake-SP, AVX-512) with Qwen3.8-Flash-Next, 22 threads, not pinned (benchmarks/bench_cpu_vision.py):

Image tokens	Encode time	Peak RSS
256	1.1 s	2.5 GB
484	2.0 s	2.7 GB
1024	5.4 s	2.9 GB
2025	16.6 s	3.0 GB

44 threads (both sockets) was no faster than 22, and binding the process to one NUMA node was slower than leaving it unpinned. Encoding 1024-token images back to back next to a TP=4 server cost its decode about 1-3% and its time to first token about 0.2 s. Against --text-model-only on the same TP=4 server, VRAM use grew by about 20 MiB per GPU and text decode speed did not change.

ft shell

ft shell                                    # attach to a running server
ft shell --model ~/models/Qwen3.6-35B-A3B   # serve + chat in one process

• Attach mode talks to --server URL (default http://127.0.0.1:1919)
• --api-key KEY (default $FREETOKEN_API_KEY) attaches to a server started with --api-key; with --model the shell uses that server's own key
• /help inside the shell lists the commands (/think, /cache, /reset).

ft ctl

ft ctl [--base-url http://127.0.0.1:1919] [--api-key KEY] [--timeout 10] [--json] <subcommand>

--api-key defaults to $FREETOKEN_API_KEY.

Subcommand	Endpoint	Purpose
health	GET /health	Server status, model, load progress
stats	GET /v1/stats	Throughput, latency, VRAM, pool occupancy, accepted input modalities
generate [prompt] [--max-tokens N] [--ignore-eos]	POST /generate	Raw completion smoke test (no chat template)
cache	GET /v1/cache/status	Cache pool table
cache --moe N | --kv N | --mamba N | --swa N [--wait 300]	POST /v1/cache/rebuild	Live pool resizing without a restart (k/m suffixes; --kv/--swa in tokens)
requests [--since N] [--limit N]	GET /v1/requests	Recent request ring

ft launch

ft launch {claude,codex,dsh,hermes,openclaw,opencode} [options] [-- <agent args>]

Discovers the served model via /v1/models, writes the agent's provider config, installs the agent CLI if missing, then launches it. Cloud API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, …) are cleared from the child environment so the agent cannot silently fall back to a paid endpoint. When /v1/stats reports image among model.input_modalities, the written config declares the model image-capable, which Codex, OpenCode, OpenClaw and dsh require before their image tools and attachments send anything; Claude Code and Hermes need no declaration.

Flag	Meaning
--server URL	Server to point the agent at (default http://127.0.0.1:1919)
--api-key KEY	Key of a server started with --api-key (default $FREETOKEN_API_KEY). The agent gets it in its environment; Hermes and OpenClaw keep it in their config files
--dry-run	Print the planned config changes and command, touch nothing
-y, --yes	Approve install/config prompts
--config	Configure without launching
--install-only	Just install the agent CLI (needs no server)
--force-reinstall	Re-run the agent installer
-- <args>	Forwarded verbatim to the agent

ft checkpoint

ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend offload] [--quant-backend moe.nvfp4=b12x] [--shard-gib 8] [--gpu <uuid-or-index>]

Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained fast-load format; point ft serve --model at the output dir. MoE experts are always packed into expert banks, which serve the fused and the offload strategies alike (DeepSeek-V4 experts serve offload only); --moe-backend is kept for compatibility. See the FTW caveats in models.md; FTW files from older builds can be repaired with scripts/ftw_hotfix.py instead of reconverting.

ft bench bw

ft bench bw --model /path/to/model.gguf   # every MoE layer kind of a checkpoint (GGUF or HF)
ft bench bw                               # canonical geometry per expert format
ft bench bw --gpu 1                       # a specific GPU (UUID or nvidia-smi index, as for ft serve)

Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes one row per MoE layer kind, which ft serve --moe-strategy hybrid --moe-hybrid-max-fetch -1 reads to split each layer's misses.

• One profile per GPU, at ~/.cache/freetoken/benchbw/<gpu-uuid>.json; a profile from other hardware or another profile version is ignored.
• A row's key is the layer's CPU format and options, expert shape, top_k and CPU worker threads; a layer without an exact row keeps a fetch cap of 1. Bench with the same --cpu-threads as the server's --moe-cpu-threads.
• Each run replaces the rows it benched and keeps the others.
• What to measure: --model (checkpoint paths or presets), --dtype, --formats, --isa.
• --threshold (default 2.0) sets each row's recommended: hybrid when CPU bandwidth beats PCIe by that factor.

On NVLink-C2C hosts, see models.md for FREETOKEN_H2D_BLOCKS_PER_BANK if PCIe-gather sits well below the linear H2D ceiling.