Repository navigation
qwen35/27B native generation: existing Body35, frontend dispatch and mixed-IQ admission gaps #88
Description
Contribution closure status — 2026-10-26
The agreed initial contribution is plain-text dense qwen35 generation on the pinned lmstudio Q4_K_M, attached through the existing Qwen seat and Body35. Its implementation #339 and diagnostic tooling #254 are merged; the earlier #251 is closed and is not another outstanding implementation PR.
The remaining agreed validation is the maintainer-owned A6000 48 GB lane: independent reference capture from the exact file, forced-ID logits, eager/graph, and native CLI versus HTTP at short/4K prompts with the agreed session checks. The latest status posted here on 2026-10-22 still lists the A6000 results as open. This update does not claim those checks passed or that they were rerun by me. Flint GPU results are not dense-Qwen evidence.
Mixed-IQ device formats, XML binding and dense MTP are not added to my remaining contribution scope. The maintainer took the XML work back on 2026-10-19 and said no action was needed from me; IQ/MTP follow-ups remain separate from the agreed first attach. I am finishing existing contributions rather than opening new work.
Closure criterion: record the promised A6000 result (or the maintainer's explicit disposition of that validation) and confirm whether this issue closes on the agreed initial plain-text scope. Until then I leave it open; merged contributor work must not be described as missing, and pending validation must not be described as passed.
I would like to attach dense qwen35 generation to the existing native Bloomery Qwen seat, and agree the initial weight-format/GPU validation scope before a PR. The dense reader and CUDA Body35 already exist for hidden/decision workloads. On main checked at eb186d97, the Qwen3 arm of Model::serves still admits qwen3moe/qwen35moe, and a plain qwen35 backbone has the decision-head refusal.
A local frontend draft, commit a98f4ee8, connects dense qwen35 to generate_qwen3moe and, explicitly under --model qwen3, to the existing HTTP Qwen seat. It was developed/tested against main 1cba9707. Related files have changed on newer main; the snapshot is not rebased and is not a merge-ready PR.
What the draft keeps
SeatEngine, app::Session<Body35>, resident slots, checkpoint-based prefix reuse, the prompt schedule, placement and whole-fit checks remain the existing owners. No libllama/ggml inference path is added. The CUDA kernels, app session, model reader and HTTP server implementation were not modified by this draft.
The file's attention has 24 query heads / 4 KV heads at width 256: group 6 selects the existing scalar pair flash. The draft chooses mma=false through the shared shape table; the existing group-of-eight/quad cases retain their choice. Records print the actual architecture. No-word decision-head/HF-card inference and explicit head/decision-type refusals remain; Clef/other decision layouts do not become generative. MTP drafting is not enabled for the dense target.
Actual-file admission on the tested snapshot
The local file is Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf, general.architecture=qwen35, 12,120,016,960 B and 866 tensors. The native header/roles read succeeds:
metadata arch=qwen35 layers=65 trunk=64 embedding=5120 heads=24 kv_heads=4 head=256 vocab=248320 experts=0 flash_mma=false
coverage unsupported=314 unused_tensors=15
The declared next-token layer is outside the 64-layer trunk; its 15 tensors are unused. The 314 entries are 313 tensor-format/role entries plus one optional tool-call-parser entry, not 314 distinct bugs or required new kernels. token_embd.weight is IQ2_S; projection/FFN sites include IQ4_XS, IQ2_XS, IQ2_XXS, IQ3_XXS, IQ3_S, Q2_K, IQ1_M and BF16. A Q4_K output head does not make those remaining sites admitted. The host admission command exits nonzero as expected; no type gate was weakened.
Coverage-entry counts: IQ3_S 98, IQ4_XS 65, IQ3_XXS 60, BF16 48, IQ2_S 15, Q2_K 13, IQ2_XS 8, IQ2_XXS 5 and IQ1_M 1. Existing gpu::iq plane/row cores for some of these types are reuse candidates; IQ3_S there has a host-layout rule only. Reuse must match the site's activation quantization, accumulation, gains/gates and wide prompt path and get its own gates.
Checks and what is unverified
- Host model tests: 252 passed / 4 ignored.
- GPU-gates package with no GPU feature: 99 passed / 3 ignored, including two new dispatch/flash tests.
- Decision tests: 13 passed.
- Host static check and clippy of both frontends plus the admission example with the native feature set: passed, no clippy warnings. Formatting, comments, architecture/load ownership and full recipe checker: passed on the recorded snapshot.
- No
cargo oxideexecutable build, NVIDIA device/model load, generated-token gate or native serving benchmark was performed. This machine has an RX 7900 GRE, not an NVIDIA card. Host type checks do not prove an executable device bundle.
Metadata, native admission output, coverage counts, verification and scope, and the linked draft contain the reproducible command. The initial full recipe attempt and final green run are both preserved in the evidence index. No weights are uploaded.
Proposed first contribution
Rebase and validate a native text-generation clause on actual admitted Q4_K/F32 sites and an NVIDIA card that holds the whole dense model/stores. A Q4_K_M filename alone is not an admission check. Compare forced-ID logits with an independent reference on identical bits, then HTTP versus the native CLI's --last-step path at short/4K prompts, reset, prefix reuse, sampling and simultaneous slots. The existing routed Qwen serve gate assumes its routed reference files and needs a separate dense-27B clause. Only after correctness should native serving be timed; mixed-IQ kernels, tools, MTP and media can follow as separate changes.
Would that scope be useful, and which admitted GGUF/reference set and NVIDIA validation lane should it start from? The AMD operator feasibility work is tracked in #87 as a separate backend-integration question; this draft does not make native serving run on AMD.
Activity
midagedev commentedon Oct 12, 2026
Thank you for the clear scope and the draft link. We read it. The scope questions (weight formats, which GPU gates,
--model qwen3admission) are a design call for the maintainer. We will answer here in our morning (KST).midagedev commentedon Oct 12, 2026
Thank you. Yes, this scope is useful, and dense qwen35 generation should attach through the existing Qwen seat and Body35, as your draft does. No new seat.
Burntgogi commentedon Oct 14, 2026
Prepared the requested admission + frontend attach using the existing Qwen seat and CUDA Body35; no new seat or libllama model loop. It is rebased onto main
ae09980c1754dfb01fbeff6f642b9c0c53ecf30f(including the latest common-context updates). Source head:b511fb44d47ebb9823a36073c6bd93d0be916cc0.Exact initial file: HF repo
lmstudio-community/Qwen3.8-27B-GGUF, revision5a7da681f60570ab5b439a587e912d2e5eddb582, filenameQwen3.8-27B-Q4_K_M.gguf. Downloaded all 16,810,714,336 bytes and verified full SHA-256 against HF LFS:e00082f779fa385cee8c68a3ec8833a75778cc87272240b942f74e0b8243e520Native reader inventory: F32 360, Q4_K 439, Q6_K 67, 866 tensors total. Actual architecture
qwen35, 65 declared / 64 trunk layers, width 5120, FFN 17408, 24 query / 4 KV heads at width 256, vocabulary 248320, zero experts. The reader leaves 15 next-token tensors unused. Other Q4_K_M-labelled candidates contained additional quant types and were not substituted. The mixed IQ3_S file is outside this attach; no weight conversion or IQ format work.The CLI and explicit
bloomery-serve --model qwen3use native Body35, app Session, the current common context/load planner, placement, resident slots and checkpoint reuse. The planner reads architecture from its existing Split instead of an independent architecture argument. The 27B shape selects the existing scalar pair flash (mma=false); routed group-of-eight/quad shapes keep their prior MMA choice. Load/plan/listening records name the actual file architecture. No-word decision/head selection is retained, in-file decision models remain outside generation, and dense drafts are refused.Host/static results: 112 gpu-gates library tests and 333 model library tests passed; static check and warnings-denied clippy passed for both frontends, the admission example and the existing planner gate callers. The final-head full contributor T0 run passed:
DONE tier=t0 total=81 red=0 wall=253s. No CUDA executable was built with plain cargo, and no NVIDIA gate is claimed.Full coverage is still nonzero: the real file has no unsupported tensor-format/role entries, but its optional tool-call parser is missing. The admission example intentionally exits 1 with
coverage unsupported=1 unused_tensors=15anda tool-call parser for this template. No guard, parser row, band or lint was waived. This is not a full-admission or generated-token pass.Source bundle/patch, exact-file identity, final-head logs and full affected list. The bundle preserves the exact source commit/tree and requires base
ae09980c; it passesgit bundle verify. No model weights are uploaded.Publication blocker: GitHub rejected implementation-branch upload and its fork-sync endpoint because my current OAuth token lacks
workflowand the old fork predates upstream workflow changes. No workflow change is part of this attach. I requested authentication refresh; the artifact branch above uses already-published fork history, so the source/results are reviewable while the implementation PR is blocked.As agreed, the NVIDIA work is the A6000 48GB lane: cargo-oxide build; independent reference dump from this exact SHA; forced-ID logits and eager/graph comparison; HTTP vs native CLI
--last-stepat short and 4K prompts; then sampling, reset, checkpoint/prefix reuse and resident slots, alongside existing routed/decision gates. The current routedgate_qwen3_serveis not a dense reference clause. MTP, media, structured tool calls, native AMD whole-model serving and throughput vs llama.cpp/Strata remain unverified.midagedev commentedon Oct 15, 2026
Attaching admission and the frontend on top of the existing seat and Body35 is the right direction. Pinning the exact file hash and the tensor inventory made this easy to check, and leaving the missing tool-call parser as a real failure instead of waiving it is the right call.
When you open the PR, please rebase it onto the latest main. We'll run the A6000 NVIDIA checks you listed (reference dump, forced-ID logits, eager vs graph, CLI vs HTTP) on our side after the 0.3.3 release; the GPUs are busy with release gates right now.
(Comment written by the bloomery AI assistant.)
Burntgogi commentedon Oct 15, 2026
The initial native dense attach is now submitted as #251, head
d63f1bab9c7f9253338d7a1e7757c804557f9589, rebased on maine98d6c43. Latest-main/propsplacement and stage RoPE accounting are preserved. Full contributor T0 passed82 units, gpu-gates/model host libraries passed116/350 tests, and static check plus warnings-denied clippy passed. The missing tool-parser host admission still exits1; no guard is waived. The PR is correctly markedneeds-gpu.I also prepared the agreed NVIDIA work as draft #254, depending on #251:
--logit-rowsin the existing generation path, independent mainline llama.cpp forced-row writer, verified build/model/input/library identities and bounded captures.The follow-up passed118 host library tests and warnings-denied static clippy in an independent Cargo target, plus the five changed T0 units. The independent C++ harness compiled and passed six host contracts. Two CPU-only0.8B smoke captures validated row shape/alignment and provenance; they are explicitly not Bloomery/A6000 results or a substitute for27B. A shared-target stale test result was detected and excluded; published evidence uses the fresh isolated rebuild.
As agreed, actual A6000 execution remains after0.3.3. These tools prepare the captures; a registered dense reference family, measured numerical acceptance bands and the actual GPU/serving gates still need the resulting evidence. Cross-engine comparison deliberately reports diagnostics without an invented accuracy pass. No AMD whole-model or performance support is claimed.
midagedev commentedon Oct 18, 2026
Thank you. #339 and #254 are on main. The first gated scope is plain text on the pinned lmstudio Q4_K_M. We fetched that file to the box today, and the A6000 runs start now.
ggml-org's and bartowski's Q4_K_M and the Q8_0 files use the same tensor types. unsloth's UD files need IQ4_XS, IQ4_NL and IQ3_S dense kernels; that is later work on our side.
We take dense_logits_ref.cpp and dense-logits.py as the writer of a registered reference family. Your HTTP checks become clauses of our serve gate. We dump the sets and run the GPU gates on the A6000.
If you want it, a host-only PR from you: the Qwen XML tool parser bound in the qwen35 reader. MTP follows correctness.
midagedev commentedon Oct 19, 2026
Update: we will do the Qwen XML tool binding ourselves this week, because our structured-output work on this model needs it soon. No action needed from you. Thank you again for #339 and #254. We will post the A6000 gate results here.
midagedev commentedon Oct 22, 2026
Status (2026-10-22): the dense attach is on main (#339, #254,
gate-gpu-qwen38d, thelogits-qwen38dfamily). Open: the A6000 gate results we promised, the IQ device formats of the mixed-IQ file, the Qwen XML tool binding for this reader, and MTP for the dense target.midagedev commentedon Oct 27, 2026
A6000 gate results, as promised.
gate-gpu-qwen38don the RTX A6000, real tier, at main 95a483f: rc 0 in 35 s, 27 PASS, 0 FAIL (DONE total=1 red=0). It is also green on main with #781, #782 and #783 merged on top.Still open here: the IQ device formats of the mixed-IQ file, the Qwen XML tool binding for this reader, and MTP for the dense target.