spec(GFX1100-TG150): commit the 150 tok/s campaign spec - #4
Draft
ghazni101 wants to merge 2 commits into
Draft
Conversation
The developer set a goal on 2026-08-22: serve Qwen3.5-4B Q4_K_M on the RX 7900 XTX at 150 tok/s text generation, pure autoregressive greedy decode with no speculative path, on an fp8-e4m3 KV cache. Issue mudler#1651 records the goal and this spec turns it into a gated, staged campaign: S1 re-prices the attribution on the exact acceptance workload before any lever is chosen, S2-S5 take the kernel and dispatch levers in S1's order, S6 extends the landed KV-FP8 CPU brick to a ROCm store and paged-attention read behind the cache_dtype=fp8 surface that rocm_paged_attn.hip refuses by name today, and S7 runs the median-of-five idle-host acceptance gate. The index gains the campaign row; mudler#1586 stays scoped to the quant-GEMM bandwidth ladder. The wall-vs-GPU-busy gap at current head (17.8 tok/s wall against 4.41 ms GPU busy per token) is recorded as an inference to verify, not a finding: one rocprofv3 capture at a different prompt length priced it, and S1 owns the re-take. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:ox-alpha [omp]
2 tasks
S1-S3 complete at 96.0 tok/s median (target 150). The campaign exhausted optimization attempts within the allowed edit scope (src/vt/rocm/ and tests/vt/). The Outcome section records what was measured, what was rejected, why each default has its value, and the next traceable hypothesis: porting the CUDA-only async device mirror and executor to ROCm (requires runner.cpp changes outside the allowed scope) and a fundamentally different GEMV kernel design for >53% bandwidth efficiency. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:glm-5-2 [omp]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3 (the campaign issue; the spec is the first landing — implementation stages will land as their own row/GFX1100-TG150-* pull requests and this one stays open until S7 discharges or amends it).
What and why
The developer set a goal on 2026-08-22: serve
Qwen3.5-4B-Q4_K_M.ggufon the RX 7900 XTX (gfx1100, ROCm 7.14.0) at >= 150 tok/s text generation, pure autoregressive greedy decode, no speculative decoding, on an fp8-e4m3 KV cache (cache_dtype=fp8, the vLLM-mirrored surface). This change lands the campaign spec.agents/specs/gfx1100-tg150.mdand appends the campaign row to.agents/issue-index.md. Spec-before-code: no product source is touched.Design
Staged S1-S7: S1 re-prices attribution (wall vs GPU-busy) on the exact acceptance workload before any lever is chosen; S2 dispatch-collapse; S3 quant GEMM; S4 hipBLASLt bf16 arms; S5 GDN decode levers; S6 extends the landed KV-FP8 CPU brick to a ROCm store + paged-attention read behind
--kv-cache-dtype; S7 runs the acceptance gate (median of >= 5 reps, idle host, flock held, batch 1, ~512-token prompt, 256 generated tokens, greedy) plus docs/weights-provenance landing. Correctness policy: keep-quant integer core stays bit-exact vs CPU withtest_rocm_quant_dotunchanged; token identity vs pre-campaign outputs for every bit-exact lever; the fp8-KV arm gated distributionally per the ratified near-tie doctrine.Classification
Policy/document only:
.agents/specs/gfx1100-tg150.md(new spec),.agents/issue-index.md(append-only union row). No product, test, script, checker, CI, or generated paths touched.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:ox-alpha [omp]