Repository navigation
Conversation
… stages without peer access Split Flash-Next models on GPUs without peer access (Windows/WDDM) queued every stage from one host thread, which a cross-device wait blocks, so the stages took turns. Opt-in rank threads give each stage its own host thread; pipeline windows run the guessed next MTP window on the front stages from shadow states while the head verifies; decode groups step several requests as round-robin groups. The head stage now reads its MTP cells' embeddings from a mapped host copy of the table, so split models' draft chains replay their graph.
An explicit --kv-capacity of at least --max-context but below a window per sequence makes Flash-Next's sequences share one paged KV pool: one set of planes per sparse-attention layer and the MTP layer, a block table per sequence filled as it grows, pages returned at reset and taken from idle sequences when short. Prompts wait while the running requests hold the pages they need; a decode step without pages finishes at context capacity. Pipeline windows now serve whichever request decodes alone. Also: a threaded verification of several sequences hashed every part from the first one's n-gram context; rank threads run without CUDA Graphs again, since the draft/verify ordering fix removed the warm-up's illegal access.
Master pins the host image store's first pages at startup by the size of a whole-context image, which a shared KV pool refuses to give (its sequences keep no images), so a pooled server failed at startup. The pool already requires --host-state-slots 0 --host-kv-mib 0; it now runs without the store.
MSVC does not declare _cvtsh_ss, so the Windows build stopped at streamed_cpu_avx2.cpp; the conversion takes the vector F16C form, as gguf_expert_cpu_avx2.cpp does. The change is the one build/windows-release makes, line for line.
…t's last message Agent clients rewrite the last message of the previous turn's prompt (Cline-family clients drop an environment block from it), which breaks the turn closure's prefix, so every such turn of a long session prefilled its whole prompt again: an API log showed 50 turns of 214-228K tokens at 0% reuse, 2 min 10 s to each first token. A prompt of 8,192 tokens or more to its anchor now also keeps a snapshot at the last chunk boundary of its prefill before its last message, and a prompt that rewrites that message resumes from it (the long-anchor reuse path). The snapshot is taken only when the prefill started at position 0 or at another such snapshot, so it lies on the chunks a prefill from position 0 runs: taking it splits no chunk, and a prompt resumed from it computes what a prefill from scratch computes. It costs a second 74 MiB snapshot per sequence; the expert cache leaves room for both. Measured on B (int8, four pooled slots) over a 67K docs prompt: the two rewritten turns resumed 65,536 tokens and answered in 3.5 and 3.9 s instead of 1 min 31 s, and every turn's text was master's byte for byte. A snapshot at the message's own start (an earlier form of this change) split a chunk, and a 206K turn resumed from it kept a cold prefill's first 85 of 128 tokens only.
Each mode added in turn on layout A, the 206K prompt on A and B, four pooled slots beside chats, what the RTX 3060s do without the clock floor, and the message checkpoint against a cold prefill.
…ir power Master f6ec88b as pulled against this branch as the API runs it, on both layouts: 34 of 34 texts byte for byte (single prompts, agent turns that rewrite or append, requests at once), the speed of each kind of request, the GPU power and energy a token of each mode, and master's default prompt chunk.
|
Review of head 0d28854 (base 92a2b33; 8 commits, 5 files, +2290/−165; no tests). To get CI on it I pushed the head to a repository branch (run 39669204791, branch deleted since). Line numbers below are of this head. Verdict: not mergeable yet. Items 1–4 need changes; after them a re-review and Linux measurements. 1. Does not build under GCC (blocker)All three
Fix: move 2. Conflicts with master
After the rebase, 3. Modes through environment variablesEight Current defaults: rank threads and windows off (need 4. On by default without measurements
5. Rank-thread robustness
6. The pipelined-round gate
7. Docs and the PR description
8. Tests and evidenceNo tests. Needed: a CPU test of the pool arithmetic ( Minor
SuggestionSplit into four PRs with their own measurements: (a) rank threads + windows + head embedding, (b) KV pool, (c) message checkpoint, (d) decode groups. |
2 participants
What
Split Flash-Next models on GPUs without peer access (every GeForce pair under Windows/WDDM, here
on PCIe x1 risers) queued every stage from one host thread, which a cross-device wait blocks, so
the stages took turns. This branch adds three opt-in modes and two defaults for such pipelines, and
two fixes they needed on master:
NINFER_QWEN4_RANK_THREADS=1): each stage gets its own host thread, so promptchunks fill every stage at once.
NINFER_QWEN4_PIPELINE_WINDOWS=1, with rank threads): while the head stageverifies a window, the front stages run the next one, guessed by the MTP block, from shadow
states; a wrong guess is dropped. One request at a time, MTP drafts, device experts, and no
grammar or logprobs (such requests run the serial loop).
NINFER_QWEN4_DECODE_GROUPS=2..8): several decoding requests step asround-robin groups across the stages.
NINFER_QWEN4_HEAD_EMBEDDING=0turns it off): the MTP cells' embeddings come from a mapped copy of the table in pinned host
memory (322 MiB for the IQ3_S release), so the draft chain stays on the head device.
--kv-capacityof at least--max-contextbut below a window persequence makes the sequences share one paged pool (with
--host-state-slots 0 --host-kv-mib 0).snapshot at the last chunk boundary before its last message, so a turn whose client rewrote that
message (Cline-family clients drop an environment block from it) resumes there instead of
prefilling the whole prompt again. It is taken only past position 0 or another such snapshot, so
it lies on the chunks of a prefill from position 0: taking it splits no chunk, and a resumed
prompt computes exactly what a prefill from scratch computes. 74 MiB of device memory a sequence.
qwen4_exp: sequence images need per-sequence KV windows, not a shared KV pool) and now runs without the image store; MSVC doesnot declare
_cvtsh_ss(the same line asbuild/windows-release, so the two merge cleanly).Exactness against master
Master f6ec88b as pulled (only the MSVC line added), run as an API ran before these modes (MTP3,
one slot), against this branch as the API runs it (rank threads, pipeline windows, decode groups of
four, MTP2, pooled slots: four on B, two on A), both with 2,048-token chunks, greedy, no logprobs:
34 of 34 texts byte for byte, 17 on each layout. Per layout: a 206K prompt, a 33K prompt, a code
prompt and two 1,024-token chats one at a time; four turns of an agent session over a 67K docs
prompt (turns 2 and 3 drop the environment block of the user message before them, turn 4 only
appends); four chats at once; the 33K prompt beside three chats. Master's plain decode
(
--spec none) gave its MTP3 texts on B's five single prompts.Speed (same runs)
A: TITAN RTX + four RTX 3060 (
8,8,8,8,16, rk8v4). B: two RTX 3060 + TITAN RTX + RTX 3090(
8,8,17,15, int8). Windows 11, WDDM, every card on PCIe x1, the RTX 3060s held at >= 1500 MHz.On A's two slots a chat beside the 33K prompt's prefill decoded at 12.6 tok/s until that prefill
was done.
Power
One server per row, adding one mode to the row before (A) or switching one (B); the 33K prompt with
one output token, and a 1,024-token chat. nvidia-smi sampled every card each 100 ms; energy is the
layout's cards' mean power over the request times its wall time, per token.
Pipeline windows are the only mode that computes work it may throw away: 45% (A) and 47% (B) of the
guessed windows held, and the dropped ones cost 15% and 27% more energy a generated token for 43%
and 57% more speed. Rank threads run the same passes at once (38% less energy a prompt token). Four
chats at once on B: 127.8 tok/s in all with decode groups (5.62 J/token), 53.7 one after another on
master (5.41), 41.3 in the engine's batched rounds without decode groups (7.40). The decode groups'
texts were master's one-at-a-time texts; the batched rounds' were not (four of four differed).
Prompt chunk
At master's default 1,024-token chunk this branch gave master's 1,024-token texts as well (the 206K,
code and chat prompts on B); the 206K prompt prefilled at 1,457 tok/s on B and 1,557 on A, against
1,614 and 1,753 with 2,048-token chunks. Master's own 206K answer differs between the two chunk sizes.
Master itself on this rig
Measured with the same settings on master 15a52bd (the base before) and f6ec88b, this rig saw no
speedup from master's new work: the expert tier and the CPU share do not run with device experts,
and the RTX 3090's new cuBLAS gate is loaded (the server log names it) but the 27B's prefill did not
move.
One run per cell; two runs of one build differ by up to 2.5%. Texts moved between the two masters
(Flash-Next on B with logprobs, five prompts: one text the same, the others a different token after
32-192 tokens, each where one of the two had a tie or a 0.125-nat lead, one BF16 step of the
logits; chosen-token logprobs before it 0.003-0.035 nats apart on average, 0.24 at most).
Also seen on master, not changed here: the 27B (seven slots, MTP3) answered differently from its
plain decode one request at a time in 10 of 11 prompts, and seven chats at once differently from the
same chats one at a time (7 of 7), the near-ties of verify widths that
docs/serving.mddescribes.Not covered
pipeline was run.
host-expert CPU pool and the expert tier, none of these files; this tree was not rebuilt.