perf(qwen4_exp): pipelined stages without peer access, a shared kv pool and an exact message checkpoint

docs/qwen3-8-flash-next.md

@@ -451,12 +451,34 @@ one round and the streams keep their share during a long one; each round is one
 whose experts read their weights once for the whole batch, so the batch costs little more than one
 token while the experts dominate the step.
 
+An explicit `--kv-capacity` of at least `--max-context` but below `--max-concurrency` windows makes
+the sequences share one KV pool of that many positions instead: every sparse-attention layer (and
+the MTP layer) keeps one set of KV planes, and a sequence's block table maps its positions' pages
+into them, taken as it grows and returned when its sequence is reset. So four sequences of 262,144
+tokens can share 262,144 positions: one long request and short ones beside it, or several of any
+length while they fit together, each up to `--max-context`. A request takes pages for its prompt
+when it is admitted and for a few rounds ahead before each decode step; when the pool is short, a
+free sequence gives up its pages (and its cached prefix), least recently used first. A prompt that
+does not fit beside the running requests waits in the queue; a decoding request that finds no
+pages left finishes at its context capacity. A sequence still costs its own pooled keys (1.5 KiB a
+position of `--max-context`) and recurrent state: about 0.45 GiB a sequence at 262,144 tokens. The pool keeps no host images, so it needs
+`--host-state-slots 0 --host-kv-mib 0`.
+
 With the context cache on (the default), a sequence keeps its state when its request ends, and a
 snapshot of its recurrent state where the prompt's last user turn closes (or at the prompt's end
 when the template marks no turn), 74 MiB on the device: a later prompt that continues what the
 sequence holds resumes from its live state, and one that repeats the prompt up to that point (the
 next turn of a chat, which renders the previous answer without its reasoning) resumes from the
-snapshot.
+snapshot. A prompt of 8,192 tokens or more to that point keeps a second snapshot at the last prompt
+chunk boundary before its last message starts (when that lies past half of 8,192), if its prefill
+started at position 0 or at another such snapshot: agent clients rewrite that message in the next
+turn's prompt (Cline-family clients drop an environment block from it), which breaks the turn
+closure's prefix, and that prompt resumes from this snapshot instead ("long anchor" in the request
+log). The snapshot lies on the chunks of a prefill from position 0, so taking it splits no chunk
+and a prompt resumed from it computes exactly what a prefill from scratch computes; a prompt
+resumed from the live state or the turn closure matches one up to rounding (its answer tokens were
+decoded, and its chunks start elsewhere). Each costs another 74 MiB on the device, taken at the
+sequence's first such prompt.
 Either way only the new tokens are prefilled; the response's prompt summary reports how many were
 reused. A request goes to the free sequence that holds the longest such prefix of its prompt.
 `--no-prefix-reuse` (ninfer-serve) prefills every prompt from scratch.
@@ -628,6 +650,48 @@ draft-chain graph and each width its own verification graph, captured at their s
 - A pass whose layers may wait on the device for their missing experts ends synchronized: CUDA
   loads a kernel lazily at its first launch, a load may wait for the device to go idle, and a
   launch during such a wait would deadlock with the service it waits for.
+- A split model looks up its MTP cells' token embeddings on the head stage from a copy of the
+  table's payload in mapped pinned host memory (a row of the bus per cell) instead of asking stage 0
+  and copying the rows back, so the draft chain stays on one device and replays its graph there.
+  `NINFER_QWEN4_HEAD_EMBEDDING=0` keeps the round trip.
+
+### Stages without peer access (rank threads, pipeline windows)
+
+Between GPUs without peer access (every GeForce pair under Windows/WDDM), a stream that waits on
+another device's event, or a staged peer copy, blocks the host thread that queues it until the other
+device gets there. One host thread queuing every stage therefore walks each pass through the
+pipeline at device speed, and the stages take turns. Three opt-in modes, for device experts, change
+that:
+
+- **Rank threads** (`NINFER_QWEN4_RANK_THREADS=1`). The host thread prepares a pass's inputs
+  (positions, n-gram rows) into a staging entry and posts it; each stage's own thread uploads them,
+  waits on the host for its stack from the stage before (the only thread that blocks), runs its
+  segments and hands the stack on through pinned host buffers counted per boundary. Prompt chunks
+  then fill every stage, and the head stage's MTP catch-up takes the pass's embeddings from host
+  memory instead of a round trip to stage 0 that would order stage 0's next chunk after the head.
+  Calls that read results or state first wait until the threads have queued every posted pass.
+- **Pipeline windows** (`NINFER_QWEN4_PIPELINE_WINDOWS=1`, with rank threads; one request at a time,
+  MTP drafts, no grammar or logprobs). A round's verification goes to the rank threads, and before
+  it the head stage runs the MTP chain on past the round's drafts: the bonus token the window would
+  end with if it were kept whole, and the drafts after it. The front stages run that guessed window
+  at once, while the head stage verifies the current one, from shadow states: a second slot of
+  each Gated DeltaNet pool and copies of the indexer tails and PLE histories, advanced over the
+  window in flight as if it were kept whole (the fold of a commit into the shadow slot), with a
+  second set of replay records so the window in flight can still commit any prefix. When the verdict
+  keeps the window whole with the guessed bonus, the head stage completes the guessed window from
+  the front stages' stack and the next guess chain runs through its tokens; otherwise the guess is
+  dropped and the next round drafts as before. Commits run on the rank threads too. Every token is
+  still the verification's, and the weighted expert outputs are summed in fixed point, so the output
+  is the serial loop's byte for byte. `NINFER_QWEN4_PIPELINE_FORCE_MISS=1` drops every guess (the
+  shadow work still runs), a test of the shadow states.
+- **Decode groups** (`NINFER_QWEN4_DECODE_GROUPS=2..8`, with rank threads and two or more stages).
+  Several decoding requests step as up to that many groups in round robin, each group's plain step
+  posted while the others are in the pipeline, with its sampling queued on the head stage's thread;
+  a request alone keeps MTP. `NINFER_QWEN4_GROUP_STEPS` sets the steps a group runs before the
+  others get a turn (default 8).
+
+`NINFER_QWEN4_ROUND_TIMING=1` prints the host's time per MTP round phase every 100 rounds (with
+the guesses made and held), and for host-ordered verifications each stage's device time in a window.
 
 ## Measurements
 
@@ -896,6 +960,147 @@ With host experts a verification's four columns route to up to four times as man
 missing one crossing the bus, so three drafts gained about 12% here; after the October 9
 host-expert work three drafts decode 90-123 tok/s against 87-94 plain (table above).
 
+### Stages without peer access on Windows
+
+IQ3_S with the IQ4_NL table read from its file, device experts, Windows 11 (WDDM, no peer access
+between any pair, every card on PCIe x1 risers), CUDA 12.8 build for `75;86`, GPU clocks held at
+1500-2100 MHz, the built-in device profiles. A is a TITAN RTX + four RTX 3060 12 GB (`--stage-layers
+8,8,8,8,16`, the TITAN last with the head, rk8v4 KV); B is two RTX 3060 + a TITAN RTX + an RTX 3090
+(`8,8,17,15` with int8 KV, `8,8,16,16` with rk8v4). A 206,053-token prompt (the repository's docs),
+262,144-token context, 2,048-token chunks, greedy, 512 new tokens, one run per row.
+
+| Layout, KV | Mode | Prefill tok/s | Decode tok/s |
+|---|---|---:|---:|
+| A, rk8v4 | host-ordered, MTP3 | 516 | 37.1 |
+| A, rk8v4 | rank threads, MTP2 | 1,720 | 36.5 |
+| A, rk8v4 | rank threads + pipeline windows, MTP2 (26% of guesses held) | 1,720 | 45.7 |
+| A, rk8v4 | rank threads + pipeline windows, MTP3 (20% held) | 1,720 | 40.6 |
+| B, rk8v4 | host-ordered, MTP3 | 683 | 42.2 |
+| B, rk8v4 | rank threads + pipeline windows, MTP2 (32% held) | 1,640 | 60.4 |
+| B, int8 | rank threads, MTP2 | 1,590 | 46.3 |
+| B, int8 | rank threads + pipeline windows, MTP2 (31% held) | 1,590 | 62.2 |
+
+With every guess dropped (`NINFER_QWEN4_PIPELINE_FORCE_MISS=1`) B with rk8v4 and MTP2 still decoded
+at 53.4 tok/s against 43.8 host-ordered: the verifications and commits on the rank threads, which no
+longer wait for each other on the host, are about half of the gain, and the held guesses the rest.
+Every pipeline-window run generated the host-ordered run's text at the same draft count. A round of
+B (rk8v4, MTP2) took 45.5 ms host-ordered (draft 4.8, verification 39.9) and 33.9 ms with pipeline
+windows. Through `ninfer-serve` (one slot of 262,144 tokens, a 15-token chat prompt, MTP2) A decoded
+at 65.1 tok/s against 49.3 and B (int8) at 92.0 against 63.0, with the same text; a sampled request
+(temperature 0.7), a request cancelled after 3 s and three requests in a row then ran normally. The
+head stage's embedding copy alone took B's draft chain from 8.5 to 6.5 ms a round (MTP3).
+
+Four concurrent chat requests (512 tokens each, four slots of 65,536 tokens, rk8v4): B 139.7 tok/s
+in all with decode groups (`NINFER_QWEN4_DECODE_GROUPS=4`) against 45.7 with MTP3 rounds, A 112.4
+against 40.9, each request's text unchanged; one request alone kept MTP (61.0 and 49.1 tok/s).
+
+A shared KV pool of 262,144 positions with pipeline windows and decode groups: B (int8) with four
+slots of 262,144 tokens left 819 MiB free at startup against 1.06 GiB with one slot. A chat alone
+decoded at 90.7 tok/s and the 206K prompt alone at 61.7 (1.59k tok/s prefill), the one-slot server's
+text both times; the 206K request with three chats beside it decoded at 40.7 tok/s while the chats
+ran at 41.0-43.8. A (rk8v4) holds two such slots (four ran out of memory at startup): the 206K prompt
+alone at 45.4 tok/s, the one-slot text, and beside a chat at 35.9 while the chat ran at 46.5. A chat
+alone three times, one slot against the pool, same build: B 91.0/93.5/93.7 against 91.2/92.4/92.6
+tok/s (four slots), A 64.4/64.4/64.4 against 64.2/64.4/64.5 (two slots). Without decode groups four
+pooled chats at once ran at about 10.5 tok/s each.
+
+On master f6ec88bb (graph segments of at most six layers), A with rk8v4, each row adding one mode to
+the row before, a 33,009-token prompt and a 1,024-token chat, through `ninfer-serve`:
+
+| Modes | Prefill tok/s | Decode tok/s |
+|---|---:|---:|
+| none, MTP3, `NINFER_QWEN4_HEAD_EMBEDDING=0` | 573 | 49.9 |
+| the head stage's embedding copy | 703 | 51.3 |
+| rank threads | 1,644 | 51.3 |
+| pipeline windows, MTP2 | 1,641 | 67.3 |
+| decode groups, two slots sharing a 262,144-token pool | 1,648 | 67.2 |
+| the same, a build that kept each stage's segment whole | 1,646 | 67.2 |
+
+Every text was the first row's. On the 206K prompt A decoded at 33.4 tok/s (510 tok/s prefill)
+without the modes and 45.8 (1,754) with the last row's, B with int8 at 42.4 (699) and 61.8 (1,617,
+four pooled slots), the same text in each pair; B's four pooled slots ran the 206K prompt beside
+three chats at 40.7 tok/s and the chats at 40.4-42.5. Without the clock floor the RTX 3060s sank to
+210 MHz between their stages' bursts: A then decoded the chat at 9.6 tok/s, each 3060's share of a
+verification taking 60-77 ms instead of 8.
+
+Against master f6ec88bb as pulled (with only the MSVC line of `build/windows-release`), run as an API
+ran before these modes (MTP3, one slot), and this branch as the API runs it (rank threads, pipeline
+windows, decode groups of four, MTP2, four slots sharing a 262,144-token pool on B, two on A), both
+with 2,048-token chunks, greedy and without logprobs (a request with logprobs runs no pipeline
+windows): on A and on B every one of 17 texts was master's byte for byte. They were the 206K prompt
+(335 tokens on A, 422 on B), the 33K prompt, a code prompt and two 1,024-token chats, one at a time;
+four turns of an agent session over a 67K docs prompt, the second and third dropping the
+environment block of the user message before them as Cline-family clients send them, the fourth only
+appending; four chats at once; and the 33K prompt beside three chats. Master's plain decode (`--spec
+none`) gave its MTP3 texts on B's five single prompts.
+
+| A, rk8v4 | Master, MTP3 | This branch |
+|---|---:|---:|
+| 206K prompt: prefill / decode, tok/s | 531 / 35.1 | 1,750 / 46.0 |
+| 33K prompt: prefill / decode, tok/s | 569 / 40.3 | 1,600 / 56.7 |
+| code prompt, decode tok/s | 63.7 | 96.3 |
+| chats of 1,024 tokens, decode tok/s | 50.0, 58.0 | 69.5, 90.0 |
+| agent turns 1-4, time to the first token | 2 min 0 s, 2 min 0 s, 2 min 0 s, 0.56 s | 42 s, 4.5 s, 4.9 s, 0.54 s |
+| four chats: one after another (master), two at once (this branch) | 38.6 s | 25.9 s |
+
+On A's two slots a chat beside the 33K prompt's prefill decoded at 12.6 tok/s until the prompt was
+done (a prompt chunk between decode steps), against 37-46 beside another chat.
+
+What each mode costs in GPU power, one server per row adding one mode to the row before (A) or
+switching one (B): the 33K prompt with one output token (prefill) and a 1,024-token chat (decode),
+nvidia-smi sampling every card each 100 ms; the energy is the layout's cards' mean power over the
+request's wall time, times that time, per token. Master's rows are master as pulled.
+
+| A | Prefill tok/s | W | J/token | Decode tok/s | W | J/token |
+|---|---:|---:|---:|---:|---:|---:|
+| master, plain decode | 761 | 384 | 0.505 | 34.7 | 254 | 7.41 |
+| master, MTP3 | 573 | 345 | 0.603 | 48.0 | 278 | 5.89 |
+| the head stage's embedding copy | 702 | 377 | 0.539 | 49.2 | 280 | 5.79 |
+| rank threads | 1,653 | 549 | 0.333 | 49.0 | 274 | 5.69 |
+| MTP2 | 1,650 | 550 | 0.335 | 47.2 | 258 | 5.58 |
+| pipeline windows | 1,652 | 551 | 0.335 | 67.6 | 423 | 6.43 |
+| decode groups, two pooled slots | 1,653 | 553 | 0.336 | 67.7 | 424 | 6.45 |
+| the same without the RTX 3060 clock floor | 1,620 | 546 | 0.338 | 59.2 | 369 | 6.63 |
+
+| B | Prefill tok/s | W | J/token | Decode tok/s | W | J/token |
+|---|---:|---:|---:|---:|---:|---:|
+| master, MTP3 | 761 | 428 | 0.563 | 55.6 | 307 | 5.62 |
+| rank threads, decode groups, MTP2, four pooled slots | 1,570 | 548 | 0.350 | 57.6 | 301 | 5.32 |
+| the same with pipeline windows | 1,572 | 554 | 0.354 | 90.4 | 593 | 6.74 |
+
+Rank threads run the same passes, only at once: prefill 2.4 times as fast for 1.46 times the
+power, 38% less energy a prompt token. Pipeline windows are the one mode that computes what may be
+thrown away: 197 of the 437 guessed windows of A's chat held (45%), 203 of 428 on B (47%), and the
+dropped ones cost 15% (A) and 27% (B) more energy a generated token for 43% and 57% more speed.
+Four chats at once on B ran at 127.8 tok/s in all with decode groups (718 W, 5.62 J/token), against
+53.7 one after another on master (290 W, 5.41 J/token) and 41.3 in the engine's batched rounds
+without decode groups (305 W, 7.40 J/token); the decode groups' texts were the ones master gave
+one at a time, the batched rounds' were not (four of four differed). Idle, A's cards drew 64 W with
+the RTX 3060s held at 1,500 MHz and 48 W without; `--gpu-clock-floor` holds the floor only while
+requests run.
+
+At master's default prompt chunk (1,024 tokens) this branch gave master's 1,024-token texts on B's
+206K, code and chat prompts too, and prefilled the 206K prompt at 1,457 tok/s on B and 1,557 on A,
+against 1,614 and 1,753 with 2,048-token chunks (master: 665 against 700). Master's own 206K answer
+differs between the two chunk sizes; the short prompts, one chunk either way, do not.
+
+| B, int8 | Master, plain | Master, MTP3 | This branch |
+|---|---:|---:|---:|
+| 206K prompt: prefill / decode, tok/s | 760 / 37.3 | 700 / 43.8 | 1,610 / 64.7 |
+| 33K prompt: prefill / decode, tok/s | 810 / 44.2 | 751 / 50.4 | 1,540 / 70.5 |
+| code prompt, decode tok/s | 42.6 | 74.2 | 115.9 |
+| chats of 1,024 tokens, decode tok/s | 40.9, 40.9 | 56.6, 67.8 | 90.9, 118.1 |
+| agent turns 1-4, time to the first token | | 1 min 30 s, 1 min 31 s, 1 min 31 s, 0.51 s | 44 s, 3.5 s, 3.9 s, 0.44 s |
+| four chats: one after another (master), at once (this branch) | | 32.8 s | 14.0 s |
+
+Master prefilled each of the first three agent turns from scratch. This branch resumed the second
+and third from the message checkpoint at position 65,536 (2,048 tokens times 32, the boundary before
+the message's start) and the fourth from the live state. The checkpoint used to sit at the message's
+start, which split a prompt chunk: a 206K turn resumed from it answered in 1.1 s instead of 2 min
+12 s but kept a cold prefill's first 85 of 128 tokens only, then took the other side of a 0.125-nat
+tie; a turn resumed from the live state still matches a prefill from scratch only up to rounding (it
+kept all 128 tokens, with logprobs up to 0.153 nats apart), as on master.
+
 ### Hyper-connections in Q8_0
 
 The hyper-connection projections (97 down/up pairs and 96 inject matrices) are stored as ggml Q8_0,