Repository navigation
Conversation
This was referenced Sep 15, 2026
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate correctness issues remain in TP sharding, MoE quantization, and FP8 weight loading.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds Qwen4Exp tensor parallelism and optional load-time FP8 dense/lm-head quantization for 5080 deployment.
Changes:
- Adds TP-aware attention, GDN, MoE, NVFP4, and dense weight paths.
- Adds dynamic FP8 quantization and FP8 linear/lm-head support.
- Adds sharding, FP8, CPU-build, and rotary-cache regression tests.
File summaries
| File | Review summary |
|---|---|
tests/models/test_nvfp4_banks_tp.py |
Tests NVFP4 expert TP partitioning. |
tests/models/qwen4_exp/test_tp_shard.py |
Tests Qwen4Exp dense projection sharding. |
tests/models/qwen4_exp/test_fp8_dense.py |
Tests FP8 kernels; nit (1 vote): add production reader/model-state-dict integration coverage. |
tests/models/qwen4_exp/test_build_cpu.py |
Tests CPU construction and rotary-cache isolation. |
python/freetoken/models/qwen4_exp/weight.py |
Adds TP and FP8 loading; critical (3 votes): shard block-FP8 scale companions; critical (3 votes): avoid re-quantizing declared block-FP8 weights; moderate (1 vote): use the installed quant config for LM-head decisions. |
python/freetoken/models/qwen4_exp/moe.py |
Combines TP MoE reductions; critical (2 votes): apply shared-expert quantization or restrict the fast path to unquantized experts. |
python/freetoken/models/qwen4_exp/model.py |
Selects the FP8 LM head; moderate (1 vote): detect incompatible bf16/FP8 FTW formats before loading. |
python/freetoken/models/qwen4_exp/gdn.py |
Adds TP-local GDN projections. |
python/freetoken/models/qwen4_exp/config.py |
Adds FP8 configuration predicates. |
python/freetoken/models/qwen4_exp/attention.py |
Adds TP and dynamic-FP8 attention projections. |
python/freetoken/models/nvfp4_banks.py |
Adds TP slicing; critical (1 vote): reject intermediate sizes not divisible by TP size. |
python/freetoken/models/config.py |
Adds FP8 environment flags. |
python/freetoken/layers/quantization/moe/nvfp4.py |
Sizes NVFP4 kernels from local intermediate widths. |
python/freetoken/layers/linear.py |
Supports explicit local merged output sizes. |
python/freetoken/layers/fp8_dynamic.py |
Implements dynamic FP8 quantization and GEMMs. |
python/freetoken/layers/embedding.py |
Adds the logits path; nit (3 votes): remove the overwritten duplicate implementation. |
python/freetoken/engine/config.py |
Increases distributed initialization timeout; moderate (1 vote): avoid globally changing Gloo/NCCL timeouts; nit (1 vote): move deployment rationale to documentation. |
Review details
Suppressed comments (5)
python/freetoken/engine/config.py:92
- The repository's contributor guidance says configuration files get no comments, so this inline deployment-specific rationale should not be added to the dataclass. Keep the timeout value here and move the explanation to PR/deployment documentation.
distributed_timeout: float = 1800.0 # ranks reach the first collective minutes apart on a 100+ GiB offload load
python/freetoken/engine/config.py:92
- This changes both Gloo and NCCL process-group timeouts in
engine.py(535-555) from 60 seconds to 30 minutes for every distributed run. A crashed or misconfigured rank will therefore make unrelated jobs wait up to 30 minutes; keep the short default and use a large-offload-specific override instead.
distributed_timeout: float = 1800.0 # ranks reach the first collective minutes apart on a 100+ GiB offload load
python/freetoken/models/qwen4_exp/model.py:140
- Enabling these flags makes the model declare FP8-only keys, but
load_weight()intentionally replays FTW tensors without runningqwen4_exp.iter_weights. A bf16 FTW therefore lacks keys such asin_proj_qkvz.weightandlm_head.weight_scaleand still fails with a bareKeyErrorinstead of a useful mismatch error. Detect the FTW format/quantization mismatch before loading, or apply the same conversion to FTW weights.
if use_fp8_lmhead(config):
# Load-time per-tensor FP8 head, for a checkpoint that ships lm_head unquantized.
# A checkpoint that declares its own scheme goes through QuantConfig instead.
from freetoken.layers.fp8_dynamic import Fp8ParallelLMHead
python/freetoken/models/qwen4_exp/weight.py:327
- The reader calls
parse_configdirectly, soconfig.quantremains its defaultNone;EngineConfigis what later attaches the installedQuantConfigto the model config. As a result,use_fp8_lmhead(config)returns true for a checkpoint whoselm_headalready has a declared quantization scheme, while the model builder returns false and expects that declared scheme. Use the installed quant config in this predicate so the reader and builder agree.
lmhead = config is not None and use_fp8_lmhead(config)
tests/models/qwen4_exp/test_fp8_dense.py:47
- These tests validate
_fp8_denseand the kernels in isolation, but do not run the productioniter_weights-> model-state-dict path with either environment flag set. That leaves the reader/builder key agreement and existing-quantization cases untested; add a synthetic checkpoint test that compares emitted keys and dtypes withQwen4ExpForCausalLM(...).state_dict().
def test_in_proj_splits_into_fp8_qkvz_and_bf16_ba_per_rank():
cfg, world = (
_cfg(),
2,
) # local: 2 k heads, 6 v heads -> qkvz = 2*2*8 + 2*6*8 = 128, ba = 12
t = torch.randn(128 + 12, 16, dtype=torch.bfloat16)
out = dict(_fp8_dense("model.layers.3.linear_attn.in_proj.weight", t, cfg, world))
- Files reviewed: 17/17 changed files
- Comments generated: 5
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+83
to
+84
| n = inter // tp_size | ||
| assert n % 16 == 0, f"NVFP4 TP shard {n} must cover whole 16-wide scale blocks" |
| and isinstance(se.down_proj, LinearRowParallel) | ||
| and isinstance(ex, OffloadMoELayer) | ||
| ): | ||
| shared = F.linear(silu_and_mul(se.gate_up_proj.forward(hidden_states)), se.down_proj.weight) |
Comment on lines
+221
to
+224
| if name.endswith(".self_attn.qkv_proj.weight"): | ||
| q = (config.num_qo_heads, 2 * config.head_dim) | ||
| kv = (config.num_kv_heads, config.head_dim) | ||
| return _shard_rows(t, [q, kv, kv], rank, world) |
Comment on lines
+339
to
+340
| if fp8 or lmhead: | ||
| yield from _fp8_dense(name, tensor, config, tp.size, dense=fp8, lm_head=lmhead) |
Comment on lines
+124
to
+127
| def _logits(self, x: torch.Tensor) -> torch.Tensor: | ||
| """The local vocab-shard GEMM; the seam a quantized head overrides.""" | ||
| module = self.tied_embedding or self | ||
| return F.linear(x, module.weight, self.bias) |
This was referenced Sep 15, 2026
gdevenyi
force-pushed
the
5080deploy
branch
from
September 15, 2026 04:00
bb37974 to
5767c99
Compare
…ics-report - the scheduler measures the prefill span itself (admission to the token sampled off the request's last prefill chunk) and ships it as a duration on that token's DetokenizeMsg: it is the one boundary a client timing SSE frames cannot see, and a duration keeps the wire free of any cross-process clock assumption - the remaining timings come off the raw ack stream, not the emitted GenEvents: a reasoning or tool-call parser holds text back, sometimes to the end of the stream, so an event-timed TTFT runs long, and acks are the one path the streaming and buffered generators share - cached_prompt_tokens reports the real prefix-cache hit whether or not --enable-cache-report is set; that flag governs the billing fields in usage, and gating the hit here would leave prefill_tokens_per_second computed over tokens that were never forwarded - decode_tokens_per_second divides by decode_tokens - 1, as vLLM's and sglang's serving benchmarks do: the first token falls out of prefill and spans no decode interval - served on /v1/chat/completions, /v1/messages and /v1/responses; /v1/completions does not go through the shared generation core and is left out Closes FlashML-org#503
Malformed or truncated FTW indexes used to surface as mmap/O_DIRECT failures deep inside the load. Parse and check the index once at open (integer ranges, required fields, duplicate keys, entry names, shard coverage, shard size), raise FTWFormatError, and reject nothing the writer produces. Shards are stat'd through symlinks because Hugging Face hub snapshots are symlink farms.
gdevenyi
force-pushed
the
5080/08-moe-stats-231
branch
from
September 18, 2026 03:06
a91e5f5 to
af3d3d1
Compare
gdevenyi
force-pushed
the
5080deploy
branch
from
September 18, 2026 03:06
5767c99 to
8adde91
Compare
- mirror_store_quant (new Triton kernel, graph-safe UVA): the write-through mirror now copies e4m3 codes + fp32 scales read from the pool AFTER quantization (the pool is the source of truth). The bf16-era mirror corrupted fp8 pages silently on rehydrate. - mirror_scales host bank + fused-copy descriptor extended with the scale banks (per-bank row sizes). - set_select_width takes max_bs: the whole batch selection must fit the physical pool by construction (fixes assert-crash on multi-row decode with small pools). - measured: 299,648 logical tokens with kv-cache-dtype fp8 + kv-host-pages, needle at 253k tokens answered correctly (~1026 t/s), quality suite identical to baseline.
Prefix reuse on hybrid GDN models needs KV pages AND a GDN state snapshot alive at the match point; the small global snapshot pool gets drained by interleaved auxiliary calls (agent title generators) and every match truncates to zero. Measured on real WorkBuddy agent turns: TTFT 20-150s recompute per message. This adds a host-resident tier: evicted GDN snapshots and KV prefix pages move to pinned RAM instead of dying. Matches rehydrate from host in ~4s (24k-token prefix) instead of a full recompute. - HybridRadixCache hooks: on_evict_node / on_match_dead_snapshot; evicted leaves become tombstones instead of cascade-freeing KV - HostPrefixTier: pinned GDN snapshot bank + cold-prefix index with its own KV slot space (logical ids get reused by the scheduler, so content is copied D2H on eviction, never referenced by id) - Admission rehydration; with KVHostOffloader active the copy is host-to-host into its mirror (no bulk H2D; lazy fetch covers residency) - Eviction copies run in 128-page slices (staging the whole prefix OOMed exactly under memory pressure, which is when eviction runs) Measured: churn test cached-token 0 -> 23,936 (16.5s -> 3.8s); total pool flush rehydrates at 24,000 cached tokens in 4.4s; quality suite identical (39/40 QA, 30/30 cloze); decode steady 33-37 t/s. All off by default (FT_GDN_HOST_TIER / FT_PREFIX_HOST envs).
…ite-pages From the Sep 15-16 debugging sessions (boolean-indexing anti-pattern family): - the indexer buffers must use the pool COMPUTE dtype (bf16), never the e4m3 store dtype of an fp8 KV pool, or CUDA-graph capture fails on tl.dot - ensure_write_pages takes token_to_req for the fixed-shape kernel path
gdevenyi
force-pushed
the
5080/08-moe-stats-231
branch
from
September 19, 2026 21:31
af3d3d1 to
e119aa4
Compare
gdevenyi
force-pushed
the
5080deploy
branch
from
September 19, 2026 21:31
8adde91 to
4bb35fc
Compare
/v1/models published `config.max_seq_len` -- the model's positional ceiling -- while the engine clamps its own `max_seq_len` to `min(config.max_seq_len, num_pages * page_size)` at startup and again in `_refresh_seq_state` after a rebuild, and the scheduler admits against that clamped value. The frontend process holds an unclamped ServerArgs copy, so the two disagreed about one quantity: on a 12 GiB card, 262144 advertised against a 178176-token pool. `ft launch` reads this number to size each agent's context window (opencode `limit.context`, codex `context_window`/`max_context_window`, CLAUDE_CODE_MAX_CONTEXT_TOKENS), so agents set their compaction threshold past what the server can hold and hit `context_length_exceeded` instead of compacting in time. Split the pool resolution out of `cache_geometry` into `kv_pool_geometry` so /v1/models and /v1/cache/status cannot drift apart, and clamp the advertised value with it. Falls back to the model ceiling while `num_pages` is still 0 (loading, or an engine build that sends no meta ack). The existing route test passed either way -- FakeState carries no pool geometry, so it fell through to the ceiling. It now asserts that fallback explicitly, alongside new cases for the clamp, a pool larger than the ceiling, and a rebuild superseding the load-time allocation. Fixes FlashML-org#448 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…idation helper CPU-resident test caches reach this helper through prefetch_prefill_layer and cannot run the Triton kernel (triton is a Linux-only dependency). The eager path clamped via Python slices; raw pointer arithmetic cannot, so refuse a buffer that does not fit instead of corrupting the heap. Assisted-by: pi
…ference on cuda Element-wise compare over three configs x 20 random trials, covering duplicate expert ids, empty slots and a nonzero slot_start, plus the bounds guard. Assisted-by: pi
gdevenyi
force-pushed
the
5080/08-moe-stats-231
branch
from
September 23, 2026 23:36
e119aa4 to
96090ae
Compare
gdevenyi
force-pushed
the
5080deploy
branch
from
September 23, 2026 23:36
4bb35fc to
2419ebc
Compare
Owner
Author
|
Follow-up: I tried extending FP8 to the remaining bf16 projections, the qwen4_exp shared expert and the hyper-connection projections (branch
So this PR's FP8 scope (dense + lm_head) stays as it is. |
…s and stream entries, host-built logprob row index, responses and truncated reasoning_tokens, logprobs null)
…d stream/non-stream parser consistency
With stop strings the detokenizer now holds a logprob entry until its text is
released, so one reply can carry several entries or none, and a per-reply
text_offset can no longer describe them. The detokenizer stamps each entry
with its own token's start instead, and UserReply.text_offset is gone;
completions reads the offset from the entry, keeps the pending carry-forward
for text-less replies, and drops stop_trimmed entries on both paths. The
non-stream chat choice keeps the always-present logprobs key, now with the
segmenter's {"content", "refusal"} object.
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
The stop-string hold block ran for every message, building and replacing empty lists even for requests that never asked for logprobs, about 0.25 us per message (6-14% of the detokenizer's per-message cost on the Qwen3.8 tokenizer). Move it into DecodeStatus.release_logprobs and call it only when the message carries an entry or entries are still held. The new test pins that held entries are still released by a message without its own entry. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
_generate_full_impl ran untrimmed() on every ack even for requests without logprobs, and its result then kept whatever raw, unsegmented entries the acks carried. Read the flag once and skip both, so the protocol-neutral result carries entries only for a request that asked for them, as the streaming path already does. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt
Fork-only shim over FlashML-org#739: FTWReader warns once and the dense, host-bank and device-bank read paths upcast fp16 NVFP4 globals to the fp32 the kernels read, so an FTW converted before FlashML-org#739 serves with its old numerics until reconverted. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…mpletions A set value of a result-changing vLLM extra (allowed_token_ids, return_tokens_as_token_ids, guided_*, ...) is a 400 instead of being dropped by extra="allow"; null/false/empty still pass. Empty prompts and more than 256 prompts x n are 400s, and an error reply's code (context_length_exceeded) now reaches the completions body. On c23d2d5 the new deny-list test fails for all 10 fields with 200 != 400. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
… logprobs Sampling is masked to the allowed ids at every position (a LogitsPlan mask like min_tokens'). A row that also asks for logprobs gets them renormalized over the set from the unprocessed logits, vLLM's processed_logprobs at temperature 0, so the combinations that mode would fold in are 400s. The -inf tail and padding are dropped from top entries, non-finite values are sent as -9999, and completions logprobs go up to 32. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
tokens and top_logprobs keys become "token_id:N", vLLM's id-keyed shape, which also keeps candidates that decode to the same text from collapsing. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
prompt as list[int] or list[list[int]] goes to the engine as is, one request per prompt x n in prompt-major order. Ids are range-checked at the API when a frontend tokenizer is up and always in tokenize(), so an id past the vocabulary never reaches the embedding. echo or suffix with one is a 400. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
The completions deny-list now covers /v1/chat/completions too, plus allowed_token_ids and return_tokens_as_token_ids, which only completions implements: a set value is a 400 instead of a 200 that ignored it. Before this, the new chat test fails for all 10 fields with 200 != 400. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
An id at or past len(tokenizer) (2**31 without a frontend tokenizer) in logit_bias or stop_token_ids is a 400 on chat and completions before anything is queued, and the tokenizer worker checks them again. Before, a huge id overflowed msgpack at send_one (a 500) and one past the vocabulary was silently ignored by the sampler. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
--logprob-read-snapshot-donate {all,boundary,none} (default all, unchanged)
applies to finished requests with max_tokens=1 and logprobs on hybrid GDN
models: boundary donates only the frozen x64 prefill snapshot, none donates
neither it nor the live state and frees the pages, since a node without a
snapshot can never be resumed. A burst of label reads then cannot evict a
long conversation's snapshots.
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…izing expert banks
…re raising in pinned_tensor
…g#700's method loader FlashML-org#700 renamed _legacy_expert_banks to _method_expert_banks and replaced load_expert_banks' dtype argument with the quant method, so the tests patched a missing name and passed an unknown keyword. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…rm offload capture test The fake graph has no CUDA graph exec for cuGraphUpload; the test fails the same way on upstream cc56b6b. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
… test host's gpu W4A4 asks the real device whether it is worth it, so on an sm_120 host auto picks w4a4 and the triton assertion fails (also on upstream cc56b6b). Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…e_start Streamed /v1/messages opened with input_tokens 0 and no cache fields (FlashML-org#718). generate_events now yields PromptAdmitted on the scheduler's admission ack and message_start waits for it, so its input/cache split matches message_delta. A keepalive or error before admission still sends message_start first, with zeros. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…starts The context check ran only in the scheduler, after the worker tokenized, so a streamed request whose prompt did not fit got HTTP 200 and an in-stream error (FlashML-org#749). Every endpoint now renders and tokenizes the prompt frontend-side before any response and rejects a prompt that leaves no room for one output token with the scheduler's "prompt is too long" text and code context_length_exceeded, against the /v1/models context. A max_tokens that does not fit is still clamped by the scheduler: agents always send a large one. The worker serves the counted ids instead of tokenizing again. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
A call cut by max_tokens inside </parameter> streamed {"city":"Lima and then
fell back to {} at close, while non-streaming returned {} (FlashML-org#710). The
qwen3_coder detector now finalizes an open call at end of stream, dropping the
partial closer and closing the JSON, and the one-shot parse replays that
finalize, so both modes return the same valid arguments with the value kept.
Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…e bank reader threads The engine allocates the host banks under torch.inference_mode(), which the reader pool's threads do not inherit, so the in-place fp32 copy raised and the server refused to start. The test now loads the banks under inference mode, as the engine does. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…ob_read predicate Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
An isolated read matches with max_len 0 (no host-tier lookup) and donates nothing. The prefill-commit branch also skips the snapshot insert for a non-donating read; the scheduler only reaches it for requests that keep decoding, never a read. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
… FREETOKEN_GRAPH_UPLOAD On Qwen3.8-Flash / RTX 5080 a decode graph uploaded with cuGraphUpload at capture (FlashML-org#697) replays with ~0.5 us instead of ~0.2 us between kernels: +0.19 ms/step (-1% decode), +0.43 ms with --moe-collect-stats. FREETOKEN_GRAPH_UPLOAD=0 keeps the driver's first-launch upload; default on. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
The prefill indexer score ran one query row per program and padded the 4 index heads to a 16-wide dot, so every row re-read all visible keys and 3/4 of the tensor-core work was padding. qsa_mqa_paged_rows gives each program 16 rows of one request (a host-built tile map cut at the cu_seqlens request boundaries) and does one [BLOCK_N, 16] dot per head, summing the heads in the one-row kernel's order, so every score is bitwise the old one. Decode (graph-captured) keeps the one-row kernel. Microbench, one 4096-row chunk at 249K context: 18.5 -> 3.2 ms per layer, no spills (134 regs). Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
… batch Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
…stay independent Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Td2gieja9Da2rPTN17Z846
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges upstream FlashML-org#389 (
feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm), headbd106c9a). FlashML-org#389 is stacked on FlashML-org#385 (Qwen3.8-Flash-Next tensor parallelism), which comes in with it. Both were closed by their author on 2026-09-14. Stacked on #16; this is the top of the stack,5080deploy.Changes
Merge of FlashML-org#389 (16 commits including FlashML-org#385). Six conflict hunks in three files, all against
main's changes since FlashML-org#389 was based:models/qwen4_exp/attention.py:mainpassesbatch.get_attn_positions()(mRoPE, feat(mm): serve image input on the Qwen families FlashML-org/FreeToken#454), while feat(qwen4_exp): tensor parallelism for Qwen3.8-Flash-Next (offload backend) FlashML-org/FreeToken#385 passesbatch.positionswith per-shard sizes. Resolved tobatch.get_attn_positions(), q.view(-1, self._local_qo_dim), k.view(-1, self._local_kv_dim).models/qwen4_exp/config.py: imports from both sides (fp8_dense_enabled,fp8_lmhead_enabled,mrope_layout_from_rope_params).models/qwen4_exp/weight.py, the only non-mechanical resolution:mainreplaced_try_fusewith the QuantConfig-driven_DenseFuser(block-FP8 dense support) and addedinclude_visionfiltering. feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm) FlashML-org/FreeToken#389's loop still called_try_fuse.main's loop (_DenseFuser.fuse/check_unfused, vision skip) and routes every tensor it yields through feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm) FlashML-org/FreeToken#389'semit(), which shards for TP and then applies_fp8_densewhen enabled.torch.cuda.empty_cache()after FP8 quantization is kept.Local fixes in the merge commit:
models/config.py(silent semantic conflict): git merged cleanly, butmainhad removedimport osand_VISION_TRUE, and feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm) FlashML-org/FreeToken#389'sfp8_dense_enabled/fp8_lmhead_enabledstill use both. Every qwen4_exp test failed withNameError: name 'os' is not defined(70 failed, 11 errors). Restoredimport osand renamed the set to_ENV_TRUE.tests/models/qwen4_exp/test_build_cpu.py(a bug in the feat(qwen4_exp): load-time per-tensor FP8 dense projections (W8A8 via _scaled_mm) FlashML-org/FreeToken#389 branch itself):_build()sets the rope device to CPU and never restores it.get_ropeisfunctools.cached, so the CPU cos/sin table leaks into later CUDA tests.test_skeleton.py::test_qsa_layer_matches_hf_denseand::test_decoder_stack_prefill_and_decodethen fail with FlashInfer'scos_sin_cache must be a CUDA tensor, but only when run aftertest_build_cpu.py; alone,test_skeleton.pypasses 10/10.rotary.get_rope.cache_clear()in afinally.Testing
Unit tests (
-m "not slow"):tests/models/qwen4_exp tests/models/test_nvfp4_banks_tp.py tests/layers tests/moe tests/engine tests/kvcache tests/kernels/test_qsa_nvfp4.py tests/kernels/test_kv_nvfp4.py: 794 passed, 53 skipped, 2 failed (the rope-cache leak). After the test fix,test_build_cpu.py+test_skeleton.py: 12 passed.Full suite on the top of the stack (2d56708),
pytest tests/ -m "not slow": 2191 passed, 191 skipped, 15 deselected, 2 failed. The failures are the same twotest_glm_dsa.pysm_120 shared-memory failures present since FlashML-org#408 (#2); nothing new.Serving with FP8 off (default). bf16 FTW, 512 slots, 4096 chunks,
--skip-prefill-warmup:Serving with
FREETOKEN_FP8_DENSE=1 FREETOKEN_FP8_LMHEAD=1. nvfp4 KV, 262,144 tokens in every run:(prefill / decode in tok/s)
FREETOKEN_FP8_DENSE=1 FREETOKEN_FP8_LMHEAD=1 ft checkpoint --model ~/models/Qwen3.8-Flash-Next-NVFP4 --out ~/models/Qwen3.8-Flash-Next-NVFP4-FTW-fp8. 70.45 GiB across 9 shards, about 5 minutes. A first attempt was killed by the host's memory watchdog; the successful run never dropped below 43.9 GB available RAM.KeyError: 'model.layers.0.linear_attn.in_proj_qkvz.weight'(FREETOKEN_FP8_DENSE=1 with a bf16 FTW checkpoint dies with KeyError in_proj_qkvz.weight #8). An FP8 FTW needs the env vars at conversion and at serve time.Deployment command (top of the stack, verified on the final restart:
/v1/modelsreports 262144, adeveloper-role request answers, and a$refinteger tool argument comes back as{"limit": 7}):Refs #2, #8.
Environment:
nvidia/Qwen3.8-Flash-Next-NVFP4Stack: the
5080deploybranch ismaine0886cc plus upstream PRs merged one at a time for a single-user, full-context (262,144-token) deployment on this card. Each PR in the stack is based on the previous one, so its diff is exactly one merge step. Merge them in order, or merge the top of the stack alone.How tests were run: with the server stopped, since a running server holds almost all VRAM and GPU tests fail spuriously. Benchmarks are streamed chat completions with 256 tokens generated with
ignore_eos, one request at a time. Prompts are salted so the radix cache cannot hit.Later merges on this branch (after 2d56708)
This PR's head is
5080deploy, so it also carries the merges below. Branch tip5767c99.Scope rule for this branch: only PRs that are generic, specific to this hardware (RTX 5080 / sm_120), or for Qwen3.8-Flash-Next. Model families other than Qwen3.8 arrive only with a newer upstream base.
2a98fa4..61128bbd125f635767c99tokenizer/tokenize.py, keeping both the developer-role mapping (FlashML-org#391) andcontinue_final_message.A merge of upstream main 3faef36 (FlashML-org#467 Gemma-4 images, FlashML-org#479 GLM-5.3 images) was briefly on this branch as
bb37974and was removed by rebuilding the branch from 2d56708. The removed file set is exactlye0886cc..3faef36. Qwen3.8-Flash-Next image input (FlashML-org#454) is part of the e0886cc base and still serves images.Testing at 5767c99 (RTX 5080, production flags from above):
pytest tests/ -m "not slow": 2240 passed, 191 skipped, 0 failed. The twotest_glm_dsa.pyfailures are gone with fix(triton): keep the float glm dsa sparse launches two-stage to fit sm_120 shared memory #21./v1/models262144.FREETOKEN 5080is read correctly. A 1920x1080 image's text box is read exactly (2,070 prompt tokens, 3.6 s).developersystem instruction is followed (German answer);$reftool argument{"limit": 7};n=2returns two choices;logit_bias-100 on the first token of the unbiased answer changes the generated tokens;/tokenizeand/detokenizeround-trip.Considered and not merged: #39 (FlashML-org#85, e4m3 rounding before the native downcast).
.to(tl.float8e4nv)already matches torch's RNE byte for byte: 0 mismatches over 4,194,304 uniform values in [-448, 448], including the clamp-then-cast shapelayers/fp8_dynamic.pyuses.round_e4m3maps tiny negative values to+0.0where torch stores-0.0(5 of the 4,194,304 values). Equal in value, but not byte-identical.Rebased 2026-09-17 onto upstream main
cac247a(v0.1.3); head is now8adde91. The stack was rebuilt merge by merge on the new base (resolutions replayed viagit rerere; the replayed stack (before the new merges) differs from the old backup tipbb37974by exactly the3faef36..cac247afile set). Upstream's 6 new commits: FlashML-org#489 release 0.1.3, FlashML-org#486 FTW/vision-encoder loading (ftw_lacks_visionrefusal,iter_vision_weights), FlashML-org#485 docs, FlashML-org#480 MiniMax-M3 images, FlashML-org#481 Muse-Glimmer images, FlashML-org#445 GB10.On top of FlashML-org#389 the branch now carries (branch
5080/09-fork-fixes-on-cac247a): fork #18-#22 replayed, #23 and #47 merged directly (theirrebased/*branches sit on 3faef36, now an ancestor),dd2ebdf(block_ends test fake getsk_scale/v_scale) andcab9be3(fork #19's FTW layout check skips a bare FTW without an HF config, which upstream FlashML-org#486'stest_load_weight_text_only_skips_the_towerwrites).New upstream merges, one stacked PR each: #48 (FlashML-org#505 hybrid checkpoint across prefill chunks), #49 (FlashML-org#466 coalesced msgpack frames), #50 (FlashML-org#414 batch-memcpy probe order), #51 (FlashML-org#118 admission clamp), #52 (FlashML-org#476
--default-thinking-mode), #53 (FlashML-org#340 dummy page in the auto KV floor), #54 (FlashML-org#198 explicit KV pages in auto-sizing), #55 (FlashML-org#459 eager FTW validation), #56 (FlashML-org#487 hoist system messages), #57 (FlashML-org#222 AbortMsg on disconnect, plus8adde91turning the request-recorder middleware into pure ASGI so a non-streaming handler can see the disconnect at all).Considered and not merged this round: FlashML-org#500 (prefill buffer invalidation) fails 3 of upstream's own
tests/moe/test_offload.pyoverlap tests at its own head (the new Triton invalidate kernel receives the tests' CPU tensors), and only matters with prefill overlap on; FlashML-org#499 (host-RAM KV tier) and FlashML-org#300 (KV/MoE ladder) are promising but large, fresh and conflict with FlashML-org#408 / the runtime-rebuild path -- left for a later round; FlashML-org#464 (incremental stop-string matching) conflicts with #47's detokenizer changes and ships no tests; FlashML-org#498, FlashML-org#484, FlashML-org#456, FlashML-org#471, FlashML-org#439, FlashML-org#405, FlashML-org#429, #31, FlashML-org#266, FlashML-org#223, FlashML-org#224, FlashML-org#477, FlashML-org#504, FlashML-org#293 and the ROCm/GLM/DSV4/Gemma PRs are out of scope or superseded.Testing (RTX 5080, production flags from above; unit suite at
69a5efa, everything live at8adde91):pytest tests/ -m "not slow"at69a5efa(the tip before8adde91, which touches onlyserver/api_server.pyand adds one test;tests/serverrerun at8adde91: 871 passed): 2412 passed, 206 skipped, 4 failed -- all environment-only: 3xtests/models/test_muse_glimmer_vision.pyfail identically on pristinecac247a(transformers 5.15.1 has nopaddingkwarg onget_vision_interpolation_indices_and_weights), andtest_async_pull_coalesced(fix(utils): tolerate coalesced msgpack frames in zmq pull queues FlashML-org/FreeToken#466) is anasync deftest needingpytest-asyncio, which the project does not declare (passes 5/5 withuv run --with pytest-asyncio)./v1/models262144; the FTW-fp8 index passes fix(checkpoint): validate FTW v1 indexes eagerly FlashML-org/FreeToken#459's eager validation.$reftool arguments,n=2,/tokenize+/detokenize,logit_bias, image OCR (FREETOKEN 5080) and a shapes image all as before.--disable-moe-prefill-overlapwas dropped from the launch script after an A/B on this tip: prefill 866 / 1407 / 1411 tok/s vs 526 / 1175 / 1203 at 1.2K / 30K / 90K, decode unchanged (38.5 / 38.9 / 38.5), 248K recall 3/3 in 182 s vs 215 s, peak VRAM 15,368 MiB of 16,650 at both 90K and 248K (the peak is the 8,192-token chunk working set, not context-dependent).Correctness re-review, 2026-09-23 (head
8975217). A full re-review of the stack found 11 bugs, fixed in #64 with a failing-then-passing test each. The worst: the GDN snapshot tier (#59, FlashML-org#525) could revive a node from a shorter prefix's snapshot (silently wrong output after a conversation diverged), and FlashML-org#338's PLE index cache let the decode CUDA graph read freed memory after ~64 distinct prompt shapes. Full suite at8975217: 2462 passed, the same 4 environment-only failures.Round of 2026-10-07 (head
ecb52158, base upstream main377a3bc). Rebuilt on upstream main (8 new commits); dropped the merge of FlashML-org#559 (now upstream as555efd8) and a net-empty merge/revert pair; added upstream FlashML-org#614 (semaphore leak) and FlashML-org#593, and a compressed-tensors expert loader for qwen4_exp. Two stacked PRs: #81 (rebuild), #82 (additions). Not merged after A/B: FlashML-org#590 (no gain), FlashML-org#585 (~1% gain, changes greedy output, one nondeterministic cold run). RedHatAI/Qwen3.8-Flash-Next-NVFP4 evaluated with a paired NLL A/B (comment on #82): not better than nvidia under W4A16, production stays on nvidia. Deployed: greedy fingerprints, decode speed and 255K peak identical to the previous tip009fa6c(branch5080/base-009fa6c).Round of 2026-10-02 (head
2224380, base upstream maind3512b4). The stack was rebuilt on upstream maind3512b4(7 new upstream commits, incl. FlashML-org#546's max_tokens fix and FlashML-org#588's eager PLE double buffer, whose graph fence is dropped because it deadlocks fork #67's filler-thread fill; see #78), then 9 upstream PRs and the generic 1080deploy fixes were added, as three stacked PRs: #75 (rebuild), #76 (upstream PRs), #77 (1080deploy fixes). Deployed. Old vs new: decode identical (32K 51.2, 128K 49.8, 255K 48.9 tok/s), cold greedy identical, quality 14/15, 255K peak 15636 MiB (was 15716), 60-min soak passed. Earlier rounds' PRs (#64-#74) were not noted here; the old tip is branch5080/base-8ba20f6.Round of 2026-09-23b (head
77d8d5c). Feature merges after a review of all open upstream PRs: #61 (FlashML-org#108GET /v1/models/{id}), #62 (FlashML-org#504 opt-in--enable-metrics, off in production), #63 (FlashML-org#224 logprobs -- on/v1/completionshere; chat logprobs are refused by design while theqwen3reasoning parser is active). Nothing else open upstream improves this deployment's speed; the evaluation is summarized in the session report. Full suite at77d8d5c: 2446 passed, the same 4 environment-only failures.Round of 2026-09-23 (head
2419ebc, base71eb1f0). Upstream moved 2 commits (FlashML-org#545 flashinfer version pin for installs, FlashML-org#469 GLM-4 router bias); the stack replayed with no conflicts and its diff to4bb35fcis exactly that delta. New merge: #60 (FlashML-org#500, now passing its own tests). Skipped: FlashML-org#531 (clamps/v1/modelscontext to the KV pool -- a no-op here, pool = 262,144), FlashML-org#491 (refactor), ROCm/GLM/GGUF updates. Full suite at2419ebc: 2417 passed, the same 4 environment-only failures. The stack branches' rebase notes on #9-#16 and #48-#59 refer to the 09-19 heads; every branch was force-pushed onto71eb1f0in the same order.Round of 2026-09-19 (head
4bb35fc, basecc1f5c2). Two new merges, one stacked PR each: #58 (FlashML-org#499 host-RAM KV tier, carried as the base of the next one; its KV-page knobs are refused on an nvfp4 pool by4bb35fcbecause the mirror does not carry block scales) and #59 (FlashML-org#525 host-resident GDN snapshot tier). The snapshot tier is now on in production (FT_GDN_HOST_TIER=1 FT_GDN_HOST_SLOTS=32, 3.7 GB pinned host RAM): a 43K-token conversation that took 30.5 s to resume after 12 aux calls of ~3.5K tokens (the 8 GPU snapshot slots were evicted; the KV pool had room) resumes in 2.76 s, facts intact, no page/slot leak over 3 rounds; 250K recall 3/3 in 183 s at a 15,502 MiB peak (+134 MB); bench unchanged (865 / 1410 / 1407 prefill, 38.1 / 38.4 / 38.3 decode).Skipped this round: FlashML-org#527 (presence/frequency penalties -- already carried through
SamplingParamsand the sampler plan by #47), FlashML-org#500 (still fails 3 of its owntest_offload.pytests at0f26fd7), FlashML-org#300 (needs--moe-cache-auto, base of Aug 30 conflicting with the #22 rebuild path, and every rung growth is a pool rebuild that flushes the cache -- the opposite of the full-context goal), FlashML-org#464 (not perf/feature, conflicts with #47, no tests). Upstream's FlashML-org#471 (greedy rows in mixed batches) arrived with the base and is unioned with #47's sampler inengine/sample.py.Full
pytest tests/ -m "not slow"at4bb35fc: 2416 passed, 206 skipped, the same 4 environment-only failures.Rebased 2026-09-19 onto upstream main
cc1f5c2(4 commits pastcac247a: FlashML-org#471 greedy sampling in mixed batches, FlashML-org#518 WeightLoadError, FlashML-org#521 tvm-ffi jit arch, FlashML-org#524 install index). Same replay as before viagit rerere; the replayed stack differs from the previous tip8adde91by exactly thecac247a..cc1f5c2file set. This PR's head is now4bb35fc.🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt