Skip to content

5080deploy: rebuild on upstream main 377a3bc (drops the upstream-merged #559 and a dead merge/revert pair) - #81

Open
gdevenyi wants to merge 107 commits into
5080/base-009fa6cfrom
5080/37-rebuild-377a3bc
Open

5080deploy: rebuild on upstream main 377a3bc (drops the upstream-merged #559 and a dead merge/revert pair)#81
gdevenyi wants to merge 107 commits into
5080/base-009fa6cfrom
5080/37-rebuild-377a3bc

Conversation

@gdevenyi

@gdevenyi gdevenyi commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Stacked on #79. The base 5080/base-009fa6c is the previous 5080deploy tip. This PR rebuilds 5080deploy on upstream main 377a3bc, which has 8 new commits:

How. The 99 first-parent commits of the old stack since d3512b4 were replayed:

  • upstream-PR steps are re-merged at their recorded heads, with rerere;
  • fork merges are cherry-picked with -m 1, so their edits outside the conflict markers survive (re-merging their old branches silently reverted a PLE test fixture);
  • plain commits are cherry-picked.

Dropped as superseded or dead:

FlashML-org#572 and FlashML-org#578 moved upstream only by pure rebases (range-diff = everywhere), so their recorded heads are kept.

Hand resolutions:

Follow-up commits:

Validated on the RTX 5080 with the production config (2100 slots, rule eviction, prefetch, host embedding), interleaved against production 009fa6c:

  • Greedy fingerprints: identical to production on 4 cold runs (a94b98014cebf515 / 6fd71cf37dc18731 / e377cb534e260935). Greedy text is identical character for character.
  • Decode: 32K 52.5-52.9, 128K 50.3-52.4, 64K 52.0-52.3 tok/s, the same as production.
  • 255K TTFT: 124.6 s.
  • 255K + mid-prefill + 3 side calls: peak 15496 MiB for both production and this build, in the same session and scenario.
  • Quality: 14/15.
  • Tests, on the stack tip (ecb52158, the next PR):
    • CPU suite: only the known test_mp_coalesced failure.
    • Full GPU suite: 3506 passed. Failures: the 4 known environment-only ones (3 in test_muse_glimmer_vision, plus test_mp_coalesced), and upstream's test_dsfp4_reference_semantics::test_gemv_path_matches_the_transcription[32|128], which fails the same way on pure upstream 377a3bc on sm_120 (DeepSeek-V4.1, out of scope).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

jason-fxz and others added 30 commits October 4, 2026 13:51
@jason-fxz
feat(moe): serve NVFP4 and MXFP4 experts fused through the expert ban…
189c8ad 
…ks (FlashML-org#601)

* feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks

Resident experts skip the state dict and load through the offload expert-bank loader (HF pieces or FTW banks) into per-layer GPU banks, so fused serves bf16, fp8_block, NVFP4 and MXFP4 alike. Unified-memory GPUs auto-select fused for these formats and ignore offload-only flags with a warning.

BREAKING CHANGE: ft checkpoint always writes MoE experts as banks; an FTW converted with --moe-backend fused or triton (dense experts) no longer loads and must be reconverted.

* docs(moe): name only GB10 / DGX Spark for unified memory
@yunwei37
fix(server): report the backend kv page size in stats (FlashML-org#599)
581fdca 
* fix(server): report the backend kv page size in stats

Use resolved backend cache metadata, retaining the frontend fallback.

Refs FlashML-org#598
Assisted-by: Codex

* test(server): cover resolved kv page size and fallback

Refs FlashML-org#598
Assisted-by: Codex
@xinze-zheng
fix(server): keep Anthropic inline system messages in place for prefi…
93b0a59 
…x reuse (FlashML-org#608)

* Implement fix for Claude Code's inline system message

* Fix for Muse
@taking-lying-flat
fix(kvcache): clone inserted radix keys to release request buffers (F…
555efd8 
…lashML-org#559)
@SuperHotDogCat
perf(moe): Improved fast_index_copy_multi_jit for NVLink-C2C hosts (F…
c1a504a 
…lashML-org#610)

* perf(moe): default fused gather to 32 blocks/bank on NVLink-C2C hosts

* build(aot): prebuild fast_index_copy_multi (1024, 32) variant

* perf(moe): make fused H2D gather grid configurable via env var

* docs: document FREETOKEN_H2D_BLOCKS_PER_BANK for NVLink-C2C hosts

* fix(moe): auto-detect H2D gather grid per device in the copy plan

* chore(docs): revised docs for NVLinkC2C
@jason-fxz @shubham3-ucb
perf(moe): spin-wait the cpu moe pool between layers (FlashML-org#619)
1c183e7 
Co-authored-by: shubham3-ucb <229030647+shubham3-ucb@users.noreply.github.com>
@jason-fxz @andy-yang-1
feat(deepseek-v41): support text and vision serving (FlashML-org#603)
127c856 
* test(deepseek-v41): add independent text and vision reference

* feat(quant): support block-32 and unrounded inference outputs

* feat(attention): add csa2 packed cache and sparse backend

* feat(deepseek-v41): serve text and images through shared runtime

* perf(mhc): tune batch-stable reduction tiles

* perf(moe): autotune fp4 decode launch geometry

* perf(moe): admit recurring misses against actual lru victims

* revert: perf(moe): admit recurring misses against actual lru victims

This reverts commit 1652d200f0ebb825379cf44c28baf52971e3a461.

* perf(mhc): derive the reduction tile from the width, not a timing

* perf(moe): replace the ds_fp4 decode autotune with a fixed two-warp launch

* fix(deepseek-v41): fence engram staging against in-flight lookups

* fix(moe): return the cpu routed sum in fp32 and round it in python

* fix(deepseek-v41): keep the vision tower and aligner unquantized

* fix(scheduler): admit bounded-replay hits by the history their resume reads

* refactor(deepseek-v41): name the attention family after the model, not csa2

* refactor(kvcache): import the dsv41 pool only for dsv41 models

* refactor(deepseek-v41): name the replay switch --swa-decoder-replay

* fix(server): keep reasoning_effort a string on the wire

* refactor(models): keep the prefill replay hooks off BaseLLMModel

* refactor(deepseek-v41): drop the fp32 routed sum and batch-invariant prefill

* fix(moe): weight ds_fp4 routes in the down epilogue again

* docs(models): anchor the v4.1 replay paragraph on the tech report

* refactor(deepseek-v41): run the decoder source's narrowed prefill on Block

* refactor(kvcache): let the dsv41 pool own the history a resume keeps live

* refactor(deepseek-v41): resume bounded-replay hits by capping the prefix match

* refactor(deepseek-v41): describe the attention group as a dsv4 variant and derive its geometry from the args

* refactor(deepseek-v41): keep the engram table locator in the model package

* refactor(deepseek-v41): drop the fp32 lm head output

- return the head's bf16 logits, as DeepSeek-V4 does
- revert ParallelLMHead(out_dtype=) and LinearKernel.apply_out_dtype, which only this call used
- remove test_head_preserves_fp32_logit_margin
- cast test_vision's prefill logits to fp32 before comparing, as the harness does

* refactor(quant): carry the w4a8 activation block on MoEConfig

- drop ActDesc and QuantScheme.act; the scheme only describes stored tensors
- Fp8BlockConfig sets MoEConfig.act_block from weight_block_size for its fp4 experts via a layer_config hook
- the ds_fp4 GPU kernel and CPU executor read the block from the layer config and require it

* refactor(deepseek-v41): size the kv pool from the args, not a geometry copy

- delete kvcache/dsv41_geometry.py; the pool and cost model read dsv41_args by attribute, as the DSV4 pool reads dsv4_args
- source / ratio aliasing reads args.roles; the duplicated source and ratio validation is gone
- the cost model owns the tier row formats, row bytes and the private window layers
- the pool keeps the format and window/ratio checks, and exposes the formats and is_private_window to the attention backend

* refactor(kvcache): move the dsv4 pool into kvcache/dsv4

* refactor(kvcache): move the dsv41 pool into kvcache/dsv4

* chore(benchmarks): drop the dsv41 selection and fp8 linear benchmarks

* test(deepseek-v41): drop tests that repeat other coverage

* test(deepseek-v41): load the tiny model's resident experts as banks

* fix(server): parse deepseek-v4.1's space-led dsml tool calls

---------

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>
@jason-fxz
ci: auto-label amd issues and prs by title (FlashML-org#620)
377a3bc
@gdevenyi
merge FlashML-org#408: nvfp4 kv quantization
e84952c 
# Conflicts:
#	docs/models.md
#	python/freetoken/attention/triton.py
#	python/freetoken/kernel/triton/attention.py
#	python/freetoken/kvcache/__init__.py
#	python/freetoken/kvcache/qsa_pool.py

# Conflicts:
#	docs/models.md
#	python/freetoken/attention/triton.py
#	python/freetoken/kernel/triton/attention.py
#	python/freetoken/kvcache/__init__.py
#	python/freetoken/kvcache/qsa_pool.py

# Conflicts:
#	docs/models.md
#	python/freetoken/attention/triton.py
#	python/freetoken/kernel/triton/attention.py
#	python/freetoken/kvcache/__init__.py
#	python/freetoken/kvcache/qsa_pool.py

# Conflicts:
#	docs/models.md
#	python/freetoken/attention/triton.py
#	python/freetoken/kernel/triton/attention.py
#	python/freetoken/kvcache/__init__.py
#	python/freetoken/kvcache/qsa_pool.py
#	tests/kernels/test_e4m3_compat.py

# Conflicts:
#	docs/cli.md
#	docs/models.md
#	python/freetoken/attention/triton.py
#	python/freetoken/kernel/triton/attention.py
#	python/freetoken/kvcache/__init__.py
#	python/freetoken/kvcache/base.py
#	python/freetoken/kvcache/qsa_pool.py
#	tests/kernels/test_e4m3_compat.py
@gdevenyi
test(kvcache): give the qsa fp8 pool fakes the mrope flag main now reads
eee9faf 
Assisted-by: Claude Opus 5
(cherry picked from commit 150c95d)
(cherry picked from commit 4db8b85)
(cherry picked from commit 495770e)
@gdevenyi
merge FlashML-org#435: resolve JSON Schema refs and union types in to…
407f3b3 
…ol arguments
@gdevenyi
merge FlashML-org#391: accept the OpenAI developer role
b57d71b 
# Conflicts:
#	tests/tokenizer/test_tokenize.py
@gdevenyi
merge FlashML-org#475: respect explicit enable_thinking=false with tools
90f42f4
@gdevenyi
merge FlashML-org#339: drop the D2H sync from the varlen GDN/KDA pref…
3cb2fc3 
…ill conv
@gdevenyi
merge FlashML-org#338: fuse the PLE n-gram row-id hash into one Trito…
939c172 
…n kernel
@gdevenyi
merge FlashML-org#169: load prefill triton kernels at startup, not mi…
ee3aa49 
…d-request

# Conflicts:
#	python/freetoken/engine/config.py

# Conflicts:
#	python/freetoken/engine/config.py

# Conflicts:
#	python/freetoken/engine/config.py

# Conflicts:
#	python/freetoken/engine/config.py
@gdevenyi
merge FlashML-org#231: --moe-collect-stats, so expert-cache behaviour…
1cbbc73 
… is measurable

# Conflicts:
#	python/freetoken/engine/engine.py

# Conflicts:
#	python/freetoken/engine/engine.py

# Conflicts:
#	python/freetoken/engine/engine.py

# Conflicts:
#	python/freetoken/engine/engine.py
@gdevenyi
merge FlashML-org#389: load-time per-tensor FP8 dense projections (st…
eca4b60 
…acked on FlashML-org#385 TP)

# Conflicts:
#	python/freetoken/models/qwen4_exp/attention.py
#	python/freetoken/models/qwen4_exp/config.py
#	python/freetoken/models/qwen4_exp/weight.py

# Conflicts:
#	python/freetoken/models/qwen4_exp/attention.py
#	python/freetoken/models/qwen4_exp/config.py
#	python/freetoken/models/qwen4_exp/weight.py

# Conflicts:
#	python/freetoken/models/qwen4_exp/attention.py
#	python/freetoken/models/qwen4_exp/config.py
#	python/freetoken/models/qwen4_exp/weight.py

# Conflicts:
#	python/freetoken/layers/quantization/moe/nvfp4.py
#	python/freetoken/models/qwen4_exp/attention.py
#	python/freetoken/models/qwen4_exp/config.py
#	python/freetoken/models/qwen4_exp/weight.py
@gdevenyi
fix(models): restore the os import and truthy set the fp8 env helpers…
fda3efd 
… read

Assisted-by: Claude Opus 5
(cherry picked from commit 82a21ae)
(cherry picked from commit 0369ed9)
(cherry picked from commit b5c7812)
@gdevenyi
test(qwen4_exp): restore the rope device and clear get_rope after the…
35d2e25 
… cpu build test

Assisted-by: Claude Opus 5
(cherry picked from commit 27ad92e)
(cherry picked from commit e152f66)
(cherry picked from commit 333f7f9)
@gdevenyi
test(kernels): give the block_ends backend fake the kv scale accessor…
7dc5db7 
…s the quantized-kv backend reads

Assisted-by: Claude Opus 5
(cherry picked from commit 600819f)
(cherry picked from commit dd2ebdf)
(cherry picked from commit 12754d4)
@gdevenyi
merge fork #18: fix(moe): report the routing top-k figure as a static…
fa609ed 
… bound, not an oracle
@gdevenyi
merge fork #19: fix(qwen4_exp): refuse an ftw checkpoint whose fp8 sp…
1133888 
…lit differs from the fp8 env flags
@gdevenyi
merge fork #20: fix(engine): let moe-cache-auto drop prefill overlap …
ed8de88 
…when its slot floor does not fit
@gdevenyi
merge fork #21: fix(triton): keep the float glm dsa sparse launches t…
02e9f04 
…wo-stage to fit sm_120 shared memory
@gdevenyi
merge fork #22: fix(engine): resize shrinking cache pools before grow…
460c011 
…ing ones in a runtime rebuild
@gdevenyi
merge fork #23 (upstream FlashML-org#17 rebased): fix(server): resolv…
d855298 
…e Hugging Face repo ids to a local checkpoint dir

# Conflicts:
#	python/freetoken/utils/hf.py
@gdevenyi
merge fork #47 (upstream FlashML-org#393 rebased): OpenAI protocol co…
4f8b0e9 
…verage

# Conflicts:
#	python/freetoken/tokenizer/tokenize.py

# Conflicts:
#	python/freetoken/engine/sample.py
#	python/freetoken/tokenizer/tokenize.py

# Conflicts:
#	python/freetoken/engine/sample.py
#	python/freetoken/tokenizer/tokenize.py

# Conflicts:
#	python/freetoken/engine/sample.py
#	python/freetoken/tokenizer/tokenize.py
@gdevenyi
fix(models): skip the ftw layout check for a bare ftw without an hf c…
718e703 
…onfig

Fork #19 asks the model family to check the stored fp8 split against the env flags; a
synthetic FTW written straight from tensors (upstream's test_ftw_weights) carries no
config.json, so there is no family to ask and the read proceeds as before.

Assisted-by: Claude Fable 5.1
(cherry picked from commit cab9be3)
(cherry picked from commit e59e544)
@gdevenyi
merge FlashML-org#505: fix(scheduler): preserve pending hybrid checkp…
73fdef7 
…oint across prefill chunks
gdevenyi added 26 commits October 7, 2026 17:32
@gdevenyi
fix(engine): count min_tokens progress from the device length
c54922a 
Under overlap scheduling the sampler prepares batch N before batch N-1's token
lands in req.input_ids, so the host count lagged by one and kept the stop ids
masked for one token past min_tokens.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 8e4891f)
@gdevenyi
test(engine): import pytest for the parametrized min_tokens test
36a8fb5 
Assisted-by: Claude Opus 5.5
@gdevenyi
fix(server): let protocol-level thinking requests win over --default-…
4f1ecf0 
…thinking-mode

returned early, so reasoning_effort / reasoning.effort / thinking.type were silently turned
off under --default-thinking-mode chat. count_tokens also skipped the default.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 8276ccd)
@gdevenyi
fix(server): type allOf-wrapped refs and untyped enum/const tool para…
58bd0ae 
…meters

FlashML-org#435 sends any parameter it cannot type to the loose JSON parser, so an untyped
string enum member such as "1" came back as the integer 1, outside the enum.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit ab2693c)
@gdevenyi
fix(server): keep /v1/models/ listing the model after the retrieve route
90a6175 
Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 7ccc433)
@gdevenyi
fix(server): open the streamed thinking block with the signature field
a33da2f 
The full response and the closing signature_delta already send signature="", but
exclude_none dropped it from content_block_start, where the SDK types it as required.
Taken from upstream FlashML-org#517 (b044276).

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 55a8e72)
@gdevenyi
test(server): exercise detector SDK and lifecycle contracts
481738d 
Upstream FlashML-org#517 tests without its docs and pyproject dev-extra: the SDK tests importorskip
anthropic/openai/httpx so the server suite still collects without them.

(cherry picked from commit b044276, tests only)

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 50f99d1)
@gdevenyi
fix(server): drop /v1/completions from the --enable-metrics-report he…
4d99e7f 
…lp, which it does not serve

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit c6b7ce7)
@gdevenyi
fix(server): keep developer ahead of system and reject images when me…
42a053b 
…rging system messages

render_messages hoisted system ahead of a leading developer message, which the
tokenizer then merged as "S\n\nD"; it now leaves developer requests to the
tokenizer's per-template mapping. Both merge sites joined str() of an image
part list into the prompt, data URL included; they now raise the template's
400 instead.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit e60b2c3)
@gdevenyi
test(server): ask for n=0 in the invalid-request sdk test, since n=2 …
ce31cd4 
…is served here

Assisted-by: Claude Opus 5.5
@gdevenyi
fix(moe): keep unpinned layers off the small-prefill path and take it…
14d1b92 
… in every layer

An unpinned layer's copy_missing is the whole-layer pageable copy, which cannot serve
the LRU slot remap, so it keeps streaming. The per-layer fallback when a layer routes
to more than half the cache is gone: the engine's num_experts slot floor already keeps
every chunk resident, and a uniform path cannot leave a later layer in an overlap
buffer that was never begun.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 37c2479)
@gdevenyi
test(moe): let the small-prefill ensure fake take the eviction policy…
b0c020d 
… kwargs

Assisted-by: Claude Opus 5.5
@gdevenyi
test(qwen4_exp): compare chunked prefill to one-shot at bf16 tolerance
71a92db 
`torch.equal` fails for all three cut points on torch 2.11.0+cu130 /
flashinfer 0.6.18 / triton 3.6.0 (sm_89): cuBLAS picks a different kernel
for M=5000 and M=5000-cut, so the indexer's q and k round differently, a
few rows flip one block at the top-k margin, and the layer output differs
by up to 2.6e-3 (a few bf16 ulps). Both paths are deterministic run to run.
Use the tolerance of the neighbouring oracle tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
(cherry picked from commit 3849e3c)

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit e700e8f)
@gdevenyi
test(qwen4_exp): compare the chunked prefill's compressed slab rows t…
f8e4396 
…o the one-shot run

At bf16 output tolerance a broken dual-source compress only swaps a block in a few rows
and passes; the slab keys are unit-scale, so a wrong straddling-group key stands out.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit cedf179)
@gdevenyi
fix(kernel): keep the triton sampling softmax finite over a fully mas…
e06b2b2 
…ked prefix

min_p (FlashML-org#393) writes -inf over most of the vocabulary; a column chunk or leading block that
is -inf throughout computed exp(-inf - -inf) = NaN and poisoned the whole row. Pascal
samples through this softmax (no flashinfer below sm_75), so min_p > 0 drew uniform noise.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit dbe82ad)
@gdevenyi
feat(engine): keep the token embedding in pinned host ram (--embed-we…
588677e 
…ights host)

The embedding is a row lookup, so a single-GPU engine can leave the table in page-locked
host RAM and gather each step's rows over UVA with the PLE row kernel (5 KiB per decode
token, graph-safe): 1.2 GiB of VRAM back for Qwen3.8-Flash-Next.

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 9f2b7a2)
@gdevenyi
fix(layers): gather fp32 host embeddings and refuse other dtypes at p…
bf064fd 
…lacement

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit ccfb2b8)
@gdevenyi
fix(kernel): compute the host gather output offset in int64
b7676a3 
No regression test: overflowing it needs an output of 2^31+ elements (4 GiB+ in bf16).

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit cc5af7a)
@gdevenyi
fix(engine): charge the host embedding's pinned bytes to the pin budget
bc1d696 
Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit a8b2f0a)
@gdevenyi
fix(engine): refuse --embed-weights host for a tied embedding or tp >…
2b71134 
… 1 at config time

Assisted-by: Claude Opus 5.5 (1M context)
(cherry picked from commit 579e03a)
@gdevenyi
test(engine): fake cached_load_hf_config with the overrides argument F…
ee3f94f 
…lashML-org#573 passes

Assisted-by: Claude Opus 5.5
@gdevenyi
test(ple): drive overlapped graph fills against an in-order fake stre…
423f205 
…am so a post-launch fence fails as a hang

Upstream FlashML-org#588's _graph_consumed fence is left out of the merge: our graph fill runs on the
filler thread before its own launch passes its WAIT, so that fence deadlocks the second step.

Assisted-by: Claude Opus 5.5
@gdevenyi
merge FlashML-org#578: perf(moe): dequant nvfp4 prefill experts by bi…
dd97213 
…t placement and load activations contiguously

# Conflicts:
#	tests/kernels/test_e4m3_compat.py

# Conflicts:
#	python/freetoken/kernel/triton/e4m3_compat.py
#	tests/kernels/test_e4m3_compat.py
@gdevenyi
fix(kernel): keep one e4m3_f32_to_u8, FlashML-org#408's copy duplicat…
28ba6af 
…es the one FlashML-org#603 added to main

Assisted-by: Claude Opus 5.5
@gdevenyi
fix(server): leave late Anthropic system messages to the inline-syste…
99c0413 
…m policy instead of hoisting them

FlashML-org#487's hoist ran before FlashML-org#608's fold on /v1/messages and moved a late system reminder into
the system head, so every reminder missed the prefix cache; the OpenAI and Responses
paths keep the hoist the qwen templates need.

Assisted-by: Claude Opus 5.5
@gdevenyi
test(server): give main's DeepSeek-V4.1 detector its contract class a…
1b6e0f0 
…nd the count fake the inline-system kwarg

Assisted-by: Claude Opus 5.5
This was referenced Oct 7, 2026
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gdevenyi @jason-fxz @yunwei37 @xinze-zheng @taking-lying-flat @SuperHotDogCat