Repository navigation
Conversation
- mirror_store_quant (new Triton kernel, graph-safe UVA): the write-through mirror now copies e4m3 codes + fp32 scales read from the pool AFTER quantization (the pool is the source of truth). The bf16-era mirror corrupted fp8 pages silently on rehydrate. - mirror_scales host bank + fused-copy descriptor extended with the scale banks (per-bank row sizes). - set_select_width takes max_bs: the whole batch selection must fit the physical pool by construction (fixes assert-crash on multi-row decode with small pools). - measured: 299,648 logical tokens with kv-cache-dtype fp8 + kv-host-pages, needle at 253k tokens answered correctly (~1026 t/s), quality suite identical to baseline.
…ite-pages From the Sep 15-16 debugging sessions (boolean-indexing anti-pattern family): - the indexer buffers must use the pool COMPUTE dtype (bf16), never the e4m3 store dtype of an fp8 KV pool, or CUDA-graph capture fails on tl.dot - ensure_write_pages takes token_to_req for the fixed-shape kernel path
…ention models (--kv-host-pages) # Conflicts: # python/freetoken/attention/qsa_sparse.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # python/freetoken/attention/qsa_sparse.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py
gdevenyi
force-pushed
the
5080/19-abort-on-disconnect-222
branch
from
September 23, 2026 23:36
202d7fe to
061981c
Compare
gdevenyi
force-pushed
the
5080/20-kv-host-tier-499
branch
from
September 23, 2026 23:36
8678ec1 to
cfe09c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #57. Merges upstream FlashML-org#499 (head
6c316e2) into5080deploy, rebuilt on upstream maincc1f5c2.Why it is in scope: Qwen3.8-Flash-Next's QSA attention re-reads only the indexer-selected pages per step, which is what makes a pinned-host KV tier viable; it is also the base FlashML-org#525 (next PR) stacks on. Opt-in (
--kv-host-pages, default 0 = no behaviour change).Merge: conflicts only with #9 (FlashML-org#408 nvfp4 KV) in
attention/qsa_sparse.py,kvcache/__init__.py,kvcache/qsa_pool.py-- all unions: the pool keeps bothkv_quantandnum_index_pages, the attention path keeps the nvfp4 kernel kwargs and gains the offloaded dispatch.Not usable with nvfp4 KV as-is: the host mirror copies codes plus the fp8 pool's per-token
_scale_buffer; the nvfp4 pool keeps per-block scales in_block_scale_buffer, which nothing mirrors, so--kv-host-pages > 0would rehydrate codes without scales.4bb35fc(in the next PR) makes both tier constructors refuse a block-scaled pool with a clear error instead. This deployment runs nvfp4 KV, so the KV tier stays off here; the PR is carried for FlashML-org#525's snapshot tier.Testing (RTX 5080 16 GB, production flags: FTW-fp8, nvfp4 KV, 262,144 tokens, 1,280 expert slots, prefill overlap on):
pytest tests/ -m "not slow"at the branch tip4bb35fc: 2416 passed, 206 skipped; the 4 failures are the known environment-only ones (3xtest_muse_glimmer_vision.pyon transformers 5.15.1,test_async_pull_coalescedneedingpytest-asyncio).🤖 Generated with Claude Code
https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt