feat(qwen4exp): 262K context for four requests in split mode, KV of some layers on the Strix

[davide221]

In split mode, Qwen3.8-Flash-Next now keeps its full 262,144-token context with four requests at once. When the R9700 runs out of memory, some attention layers keep their KV cache on the Strix Halo instead of the server cutting the context. Before this, --max-ctx auto stopped at 131,072 tokens for four slots.

Changes

1. fix(ggml-cuda): copy between two devices on the source's stream.

   • The problem: ggml_backend_cuda_buffer_cpy_tensor queued its peer copy on the per-thread stream of whichever device was current. On ROCm, a copy from the R9700 to the Strix Halo iGPU queued on the destination's stream completes without writing anything.
   • The effect: the scheduler's copies of graph inputs to the other device arrived as zeros. Those are the K/V row indices, positions and masks. With MTP, a verify step then wrote its four K/V rows into row 0, so output changed from run to run.
   • Before this PR, split serving never sent a graph input to the expert device, so nothing on main hit it.
   • The fix sets the source device first, as ggml_backend_cuda_cpy_tensor_async already does.
   • Regression test: test_cuda_copy_batch gains cross_device_tensor_copy, which needs two GPUs and skips otherwise. On an R9700 + Strix Halo it reads 0 0 0 0 without the fix and 19 20 21 22 with it.

2. feat(qwen4exp): attention layers' KV on the Strix when the R9700 is full.

   • When the fit misses, the last full-attention layers keep their K/V and indexer rows on the expert device. They run their attention there, from the input projections through the output projection, and only the layer's input and output cross between the GPUs.
   • It moves as few layers as cover the shortfall, and only while the Strix has room for them. The context shrinks only once no more layers can move.
   • Qwen4ExpCache::remote_layers keeps those rows in a second buffer on the expert device. Prefix snapshots save and restore them with the rest.
   • move_qwen4exp_attention moves those layers' attention and indexer weights.
   • Qwen4ExpLayerBuilder::on_cache_device pins the nodes a moved layer's attention adds. It covers the single-sequence, verify, pipelined-prompt and multi-slot builders.
   • The chunk plan counts only the rows left on the target. It also reports one layer's rows across all slots (layer_bytes), so the fit knows how many layers to move.
   • The QSA prefix bootstrap pools each layer's keys on the GPU that holds its rows.
   • docs/QWEN4EXP.md and the profile description cover it.

Measured

Setup: R9700 + Strix Halo (192 GB), UD-Q4_K_XL, Manage's split launcher, four slots, adaptive MTP, greedy.

	262,144 tokens, 7 of 12 layers on the Strix	131,072 tokens, every row on the R9700
Decode, a request alone (mean of 4 prompts)	45.9 tok/s	59.3 tok/s
Decode, four together	68.6 tok/s	78.3 tok/s
Outputs alone vs together	4 of 4 identical	4 of 4 identical
Recall of 3 facts hidden in a long prompt	3 of 3 at 158,927 tokens	3 of 3 at 96,307 tokens
Free on the R9700, lowest	3.9 GiB	2.6 GiB

At 262,144 tokens:

• The same prompt gives the same reply in repeated runs: short ×3 and 6K ×2, the second 6K run restored from the prefix cache.
• A short reply is the same alone and beside another request.
• A 16K prompt resent prefills in 0.25 s instead of 11.0 s, with the same reply.
• A 158,848-token prompt completes beside three short requests.

The two layouts give the same reply on 2 of 4 prompts. The other 2 diverge at a near tie, one at the first token, because those layers' attention runs on a different GPU. Both sets of replies are coherent.

The context costs speed: about 23% for a request alone and 12% for four together, from the extra GPU-to-GPU hops of the moved layers.

Tests

Suite	Result
test_server_unit	650 passed
test_cuda_copy_batch	passes, including the new test
test_qwen4exp_prefix_cache	passes
test_qwen4exp_seq_engine (exact distinct slots)	passes

Built for gfx1151 + gfx1201.

🤖 Generated with Claude Code

[davide221]

Code review

No issues found. Checked for bugs and logic errors in the changed code.

_{Reviewed cb6116c6 · 1 candidate issue checked, kept at ≥70% confidence · add the code-review label to review again}