Repository navigation
temperature 0 is not reproducible with thinking on (identical requests diverge on near-ties) #307
Description
Summary
With thinking on, temperature: 0 does not give reproducible output: three identical requests produced three different reasoning traces. With thinking off the same prompt is byte-identical across runs. Explicit sampling parameters are honoured (so this is not --sampling-defaults model overriding the request); it looks like a nondeterministic kernel flipping near-tied argmax picks, which content-heavy reasoning text exposes and peaked code output does not.
Environment: FreeToken git 4b94bdc (HEAD 2026-08-30, no local patches), pip install --no-build-isolation "freetoken[accel] @ git+…@4b94bdc"; torch 2.11.0+cu130, CUDA 13.0, driver 595.71.05; RTX 3090 24 GB (sm_86), PCIe 4.0 x16 (ft bench bw: H2D 23.8 GB/s), 32-core host, 503 GiB RAM; dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99 (compressed-tensors NVFP4, same layout as the RadixArk stock checkpoint).
ft serve --max-seq-len-override 262144 --kv-reserve-tokens 262144 --moe-backend offload --moe-cache-auto --nvfp4-backend triton --max-running-requests 1 --max-prefill-length 8192 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder
Repro
Prompt: Write a Python function to reverse a linked list. Output only code.
Request: {"temperature": 0.0, "max_tokens": 4096} (thinking on = template default), three times in a row, one at a time.
| run | completion tokens | reasoning sha1 (first 10) | content sha1 |
|---|---|---|---|
| 1 | 206 | 7f4e39e460 | 1c6a96da5e |
| 2 | 218 | bb6999e009 | 1c6a96da5e |
| 3 | 288 | 10921af11d | df8677eeec |
First divergence between runs 1 and 2 is at character 210 of the reasoning, on a near-tie:
A: …nal only code no explanation. Could include class? Function reverse linked list. Need maybe iterativ…
B: …nal only code no explanation. Could include class? Function to reverse linked list. Need maybe itera…
Controls, same prompt, max_tokens: 1200, chat_template_kwargs: {"enable_thinking": false}, two runs each:
| params | result |
|---|---|
temperature 0 |
IDENTICAL (91 tokens, same hash) |
temperature 0, top_k 1 |
IDENTICAL |
temperature 0, seed 7 |
IDENTICAL |
no params (--sampling-defaults model → temp 1.0/top_k 20/top_p 0.95) |
DIFFERENT — as documented |
Why it matters
Anyone using temp-0 for evals or regression tests on a thinking model gets a different trace (and sometimes a different answer: run 3 above) per run, with nothing in the response saying so. usage also carries no reasoning_tokens, so a truncated-by-cap reasoning run and a real answer look alike.
Ask
Which kernel is nondeterministic here — the offload MoE combine, the Triton NVFP4 dequant path, or the qsa_sparse attention? A --deterministic mode (accepting a throughput cost) would be ideal; failing that, a docs note that greedy ≠ reproducible with thinking on. Related: #223 (sampling-param validation), #281 (a regression harness needs to know this).
Activity
salekseev commentedon Sep 5, 2026
Corroborating this on different hardware and a different model family, plus one observation
about the tail that may be worth folding into the issue.
Setup: FreeToken
0.1.2+gaf71ba432, RTX 4080 SUPER 16 GB (sm_89), torch 2.11.0+cu130 /CUDA 13.3. Qwen3.6-35B-A3B 4-bit MoE,
--moe-backend offload --moe-cache-auto --max-running-requests 1 --sampling-defaults model --reasoning-parser qwen3 --tool-call-parser qwen3_coder. So: a different architecture (30 of 40 layers GatedDeltaNet), a different quantization, a different card, and the offload MoE path rather than
NVFP4 — and the same symptom.
Where it bit us: BFCL v4
parallel, 200 entries, thinking on (template default),temperature: 0.001(BFCL's default, effectively greedy). We ran two different quantizationsof the same base weights and got 0.935 vs 0.910 — a 5-entry difference on 200. Paired, that
is 8 both-wrong, 5 where A failed and B succeeded, 10 the other way; McNemar exact
two-sided p = 0.302.
The point for this issue: we cannot attribute that difference to the quantization, because
the harness is not reproducible at near-zero temperature with thinking on. Any A/B of model
quality on a reasoning-enabled endpoint currently has an unknown noise floor, so the only
sound reading is "not separable" — which is a real obstacle to using this engine for
evaluation work, independent of how large the underlying effect is.
Possible severity amplifier — the divergence has an unbounded tail. In the same runs, 3
of 400 entries emitted 32767 completion tokens (the full budget), against a median of 858:
parallel_71parallel_122parallel_183These are the same prompts that complete in ~800 tokens on other runs. If the mechanism here
is a nondeterministic kernel flipping near-tied argmax picks inside reasoning text, then it
does not only perturb the trace — it can occasionally steer the trace into a
non-terminating loop. On a server with
--max-running-requests 1that also blocks everyqueued request behind it (measured: three entries queued behind one runaway took ~8.5 minutes
while emitting 450/598/860 tokens each), and it cost one scored answer.
Happy to gather anything more specific if it would help — e.g. per-token logprobs around a
divergence point, or the same prompt repeated N times with thinking on/off on sm_89.