diff --git a/CLAUDE.md b/CLAUDE.md index 73be925bd5..2c38f04cc1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -91,7 +91,7 @@ Task-specific instructions are split into skill files under `skills/`. You MUST | `skills/das_formatting.md` | Creating or modifying any `.das` file | | `skills/comment_style_hygiene.md` | Writing or reviewing comments, names, or local code shape in ANY language | | `skills/writing_tests.md` | Writing or editing any dastest test | -| `skills/internal/tests_in_repo.md` | Adding/moving tests **in this repo** - AOT registration, the `.das_test` gating filter, deep-engine model tests | +| `skills/internal/tests_in_repo.md` | Adding/moving tests **in this repo** - AOT registration, the `.das_test` gating filter, deep-engine model tests, the `tests/spirv` emitter suite's local `spirv-val` run | | `skills/internal/writing_cpp_tests.md` | Writing or editing C++ tests under `tests-cpp/` | | `skills/internal/documentation_rst.md` | Editing RST in `doc/source/`, `//!` doc-comments in `daslib/*.das`, tutorial RST pages | | `skills/internal/tutorials.md` | Anything that looks like a tutorial - they live under `/tutorials//`, NEVER `modules//tutorial/` | diff --git a/modules/REVIEW_SHADER_EMITTERS.md b/modules/REVIEW_SHADER_EMITTERS.md index 318b754272..2a29625751 100644 --- a/modules/REVIEW_SHADER_EMITTERS.md +++ b/modules/REVIEW_SHADER_EMITTERS.md @@ -1,14 +1,13 @@ # Shader Emitters Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `dasMetal/ARCHITECTURE.md`, `dasSpirv/ARCHITECTURE.md`. +docs: `dasMetal/ARCHITECTURE.md`, `dasSpirv/ARCHITECTURE.md`, `dasSpirv/ARCHITECTURE_COOPMAT.md`. -**Routed here by another checklist: a diff under either emitter module, or to any kernel body -or fixture either emitter compiles, applies this list together with its own folder's.** +**Routed here by another checklist: a diff under `dasMetal/` or `dasSpirv/`, or to any kernel +body or fixture either emitter compiles, applies this list together with its own folder's.** **Never put anything that cannot run on the CPU into a kernel body or into a function a kernel -calls - keep both in ordinary das.** The CPU run of the same body is what the tests compare -against. +calls - keep both in ordinary das.** The tests compare the kernel against a CPU run. **A diff that adds or changes an emitter builtin - a declaration in `daslib/shader_lingua_franca.das` or an emitter's builtin table - ships a CPU body that returns @@ -21,18 +20,17 @@ reports a compile error that names the construct.** constant.** A shape constant is any value that fixes the kernel's tiling: a tile row count, a tile column count, a cooperating-simdgroup count, a staged chunk depth. -**Never pass a matmul reduction width known only at run time into a kernel that loads its -operands with the emitter's tensor-load ops (`coopmatLoadTensor*`, `coopmatLoadTensorDecode`) -any way but through the emitter's runtime-extent descriptor - `dynamic_extent` on Metal, and on -SPIR-V a `tensorLayout2D` or `tensorLayout2DPad` whose dimension `tensorLayoutSetDimension` -sets.** The reduction width is the K dimension - the length of the loop the kernel accumulates -over; it does not fix tiling, so it is not a shape constant. A loop that stages its own tiles -and loads fragments with `coopmatLoad` takes K as a plain uniform. +**A kernel that loads its operands with the emitter's tensor-load ops (`coopmatLoadTensor*`, +`coopmatLoadTensorDecode`) receives a run-time-only matmul reduction width through the +emitter's runtime-extent descriptor and no other way - `dynamic_extent` on Metal, a +`tensorLayout2D` or `tensorLayout2DPad` whose dimension `tensorLayoutSetDimension` sets on +SPIR-V.** The reduction width is the K dimension - the length of the loop the kernel +accumulates over; it does not fix tiling, so it is not a shape constant. **A diff that makes a kernel need a shape constant known only at run time ships a -specialization path, or records in the emitter's architecture doc - `dasMetal/ARCHITECTURE.md` -for a Metal kernel, `dasSpirv/ARCHITECTURE.md` for a SPIR-V kernel - that the kernel cannot -have one.** A specialization path is one compiled variant per constant shape. +specialization path, or records in an `ARCHITECTURE*.md` at the root of the module the kernel +ships in that the kernel cannot have one.** A specialization path is one compiled variant per +constant shape. **Never check a claim about emitted shape against the das source - check it in the emitted words or text.** Emitted shape is the structure of the emitted kernel - its signature, its @@ -41,9 +39,9 @@ threadgroup sizes). **A diff that adds a kernel-model capability to one emitter adds it to the other, or records the asymmetry in the shared ledger (`dasMetal/ARCHITECTURE.md`).** A kernel-model capability -is anything a kernel author can write differently because of it - an annotation, a call form, -a declaration the emitter now accepts or refuses; a change to lowering alone is the folder's -own checklist's. +is present on an emitter when a kernel source that uses it compiles there; a diff that leaves +every kernel source compiling exactly as it did before changed lowering alone, and answers to +that emitter folder's own checklist. **A diff that puts a `daslib/shader_lingua_franca` declaration into a kernel body or fixture an emitter compiles, where that emitter does not handle it, ships, in the same change, either @@ -52,7 +50,7 @@ declaration by name.** A declaration in that module is available to both emitter **A `?:`, `&&`, or `||` in a `[spirv_kernel]` or `[compute_shader]` body, or in any `def` that body calls, whose skippable operand subscripts a global-rooted array - a module global, a -`@workgroup` array, or a `self.` resource - is written as the language reads it: never +`@workgroup` array, or a `self.` resource - is left in its short-circuit form: never rewritten around the emitter with an if/else, a clamp or an unconditional read.** The emitter lowers such an operand as a branch (`dasSpirv/ARCHITECTURE.md`, "Operand laziness follows the -language"); a branchless form where it matters waits for `select` (`plans/shader_emitter_followups.md`). +language"). diff --git a/modules/dasLLAMA/ARCHITECTURE.md b/modules/dasLLAMA/ARCHITECTURE.md index c2325cb8f2..b49302658c 100644 --- a/modules/dasLLAMA/ARCHITECTURE.md +++ b/modules/dasLLAMA/ARCHITECTURE.md @@ -48,12 +48,14 @@ re-transcoding `$LCPP/src/unicode-data.cpp`). - `ARCHITECTURE_GPU_PREFILL.md` - sec.2.2c-2.2i, 2.2u-2.2v, 2.2aa: the Metal prefill driver's GEMM form ladder, dev-W knee map, attention slab, MoE bucket rail, chunked submission, the f16 twin dual-store, the last-layer FFN tail, and the dense-KQ tensor mul_mm scaffold. -- `ARCHITECTURE_GPU_VULKAN.md` - sec.2.2j, 2.2p, 2.2ab: the Vulkan resident driver's prefill - chain and byte stores - the prefill window chain, the Q8 requant byte store, and the decode - GEMV family's grid codebook buffer. -- `ARCHITECTURE_GPU_VULKAN_GEMM.md` - sec.2.2k-2.2m, 2.2q: the cooperative-matrix tiles the - Vulkan tier's GEMMs run on - the cm2 decode lanes, the tile pick and the coopmat mode ladder, - the class-pipeline build seat, and the MoE expert chain on those tiles. +- `ARCHITECTURE_GPU_VULKAN.md` - sec.2.2j, 2.2p, 2.2ab, 2.2ac, 2.2ad: the Vulkan resident + driver's prefill chain and byte stores - the prefill window chain, the Q8 requant byte store, + the decode GEMV family's grid codebook buffer, the tile probe's shared descriptor set layout, + and the recurrent block of the prefill window. +- `ARCHITECTURE_GPU_VULKAN_GEMM.md` - sec.2.2k-2.2m, 2.2q, 2.2ae: the cooperative-matrix tiles + the Vulkan tier's GEMMs run on - the cm2 decode lanes, the tile pick and the coopmat mode + ladder, the class-pipeline build seat, the MoE expert chain on those tiles, and the KHR arm's + hand-staged kq tile. - `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` - sec.2.2n-2.2o: what a model has to fit on the card before the driver runs - the residency plan, and the GPU-slot marks swap that lets one slot serve many models. @@ -65,16 +67,18 @@ re-transcoding `$LCPP/src/unicode-data.cpp`). - `ARCHITECTURE_GPU_MTP.md` - sec.2.28-2.39: the Metal speculative round over the batch driver's same-slab verify, the box knob that sets the depth a round drafts, and the kernel argument-alignment contract enforced at every dispatch. -- `ARCHITECTURE_RUNTIME.md` - sec.2.2, 2.3, 2.3a, 2.4, 2.6-2.9, 2.11, 2.12, 2.18-2.19: kernel - shape, caches, lint policy, knobs, coverage, the GPU ramp, the hybrid worker pool, and the - MoE region split. +- `ARCHITECTURE_RUNTIME.md` - sec.2.2, 2.3, 2.3a, 2.4, 2.6-2.9, 2.11, 2.12, 2.18-2.19, 2.44: + kernel shape, caches, lint policy, knobs, coverage, the GPU ramp, the hybrid worker pool, the + MoE region split, and the job queue the engine dispatches on. - `ARCHITECTURE_MEDIA.md` - sec.2.13-2.16: the padded tower GEMM widths, the family GPU hooks, the tower weight lane, and the plain-Model ASR decoders. -- `ARCHITECTURE_MEASUREMENT.md` - sec.2.5, 2.10, 2.20, 2.21, 2.26-2.28, 2.40-2.41, 2.42a: the - benchmark rig, the tune gate, the sanctioned instrumentation rails, kernel-race fidelity, the - gemv's own tune seat, the CPU kernel bench's fixture conditions, the speculative round's ruler - record, the `[tuned]` perm precedence, the mint wall in the sidecar's provenance, and the fat - exe's first-start race. +- `ARCHITECTURE_MEASUREMENT.md` - sec.2.5, 2.10, 2.20, 2.40-2.41, 2.42a, 2.45: the benchmark + rig, the tune gate, the sanctioned instrumentation rails, the ASR board's GPU row pairs, the + `[tuned]` perm precedence, the mint wall in the sidecar's provenance, the fat exe's + first-start race, and the speculative round's ruler record. +- `ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md` - sec.2.21, 2.26-2.27: the instruments that time a + kernel away from the served graph - kernel-race fidelity, the gemv's own tune seat, and the + CPU kernel bench's fixture conditions. - `ARCHITECTURE_CPU_KERNELS.md` - sec.2.22-2.24, 2.42: the sub-block-packed k3/k6 planes, the grid formats' panel and row-group decodes, the VBMI symbol lattice, and the tier that selects on the target rather than the host. diff --git a/modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md b/modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md index 6c0b32dacd..07e8ef4e19 100644 --- a/modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md +++ b/modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md @@ -42,7 +42,7 @@ codes for the rest. Per row group and weight octet a constant two-source shuffle code bytes in its qword, `VPMULTISHIFTQB` spreads the symbols into bytes, one `vpshufb` maps them to magnitudes, and the signs ride the activation copy as a mask `(x ^ m) - m`. The lattice row shares its tile body and planes with the 512/mr16 row, so only the gemv differs - what the gemv's own seat -(`ARCHITECTURE_MEASUREMENT.md` sec.2.26) races. +(`ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md` sec.2.26) races. ### 2.42 A CPU tier selects on the TARGET, not the host {#cpu-tier-target-select} diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU.md b/modules/dasLLAMA/ARCHITECTURE_GPU.md index 714b147b38..4cae0f5f98 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU.md @@ -119,7 +119,7 @@ that a question answered for one backend has an obvious address in the other. Th **PSO lifecycle - the family shares ONE device and queue** (`metal_common_init`; the second-device question was surveyed and closed against; the tune-time race arms' transient queue is -`ARCHITECTURE_MEASUREMENT.md` sec.2.21's). The decode PSO set lives as `g_pso_*` in +`ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md` sec.2.21's). The decode PSO set lives as `g_pso_*` in `dasllama_metal_common`, is compiled by `metal_decode_init` in `dasllama_metal_kernels` and released by `metal_kernels_release` there - the kernels module owns its set's lifecycle even though the vars live with the device state. Prefill's `g_pf_pso_*` set is prefill-private end to @@ -260,10 +260,10 @@ consecutive staging runs, relaxed_precision always - are `REVIEW_GPU.md` rules a `modules/dasMetal/REVIEW.das` descriptor gate; this section keeps only the refuted shapes and why they lose. -Sections 2.2j, 2.2p and 2.2ab, the Vulkan resident driver's prefill chain and byte stores, are -`ARCHITECTURE_GPU_VULKAN.md`; its 2.2k-2.2m and 2.2q - the cooperative-matrix GEMM tiles - are -`ARCHITECTURE_GPU_VULKAN_GEMM.md`; its 2.2n-2.2o - the residency plan and the marks swap - are -`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`. +Sections 2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad, the Vulkan resident driver's prefill chain and +byte stores, are `ARCHITECTURE_GPU_VULKAN.md`; its 2.2k-2.2m, 2.2q and 2.2ae - the cooperative-matrix +GEMM tiles - are `ARCHITECTURE_GPU_VULKAN_GEMM.md`; its 2.2n-2.2o - the residency plan and the +marks swap - are `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`. ### 2.2w The tower attention routes {#tower-attn-routes} diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_QUANT_PLANES.md b/modules/dasLLAMA/ARCHITECTURE_GPU_QUANT_PLANES.md index fcf277649e..a17127c8b6 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_QUANT_PLANES.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_QUANT_PLANES.md @@ -41,4 +41,4 @@ twin exactly when that PSO is non-null, so a box with no crown never compiles it The `kq_gemv_iq3s_f4` and `kq_gemv_iq3xxs_f4` crowns are raced (`race_gemv_f4_twin`). `kq_gemv_iq2xxs_f4` cannot be settled by an isolated race at all and is minted from a serving -A/B instead - `ARCHITECTURE_MEASUREMENT.md` sec.2.21. +A/B instead - `ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md` sec.2.21. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md index cc4ebe2dc3..13b9fb08e5 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN.md @@ -1,22 +1,22 @@ # dasLLAMA Architecture - the Vulkan resident driver Companion to `ARCHITECTURE_GPU.md`; section numbers are `ARCHITECTURE.md`'s. This document -carries sections 2.2j, 2.2p and 2.2ab, the prefill chain and the byte stores of the Vulkan -resident driver: the prefill window chain, the Q8 requant byte store, and the decode GEMV -family's grid codebook buffer. The cooperative-matrix tiles the chain's GEMMs run on - the cm2 -decode spelling, the tile pick and the coopmat mode ladder, the class-pipeline build seat, and -the MoE expert chain on those tiles - are `ARCHITECTURE_GPU_VULKAN_GEMM.md`'s sections 2.2k-2.2m -and 2.2q. What a model has to fit on the card before any of this runs - the residency plan, and -the marks swap that lets one GPU slot serve many models - is -`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections 2.2n-2.2o. The decode-era mechanisms of the -per-op tier are `ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections 2.2r-2.2v. The GPU backend role -table these sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. +carries sections 2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad - the Vulkan resident driver's prefill +chain, its byte stores, and the tile probe's set layout: the prefill window chain, the Q8 +requant byte store, the decode GEMV family's grid codebook buffer, the tile probe's shared +descriptor set layout, and the recurrent block of the prefill window. The cooperative-matrix +tiles the chain's GEMMs run on - the cm2 decode spelling, the tile pick and the coopmat mode +ladder, the class-pipeline build seat, the MoE expert chain on those tiles, and the KHR arm's +hand-staged kq tile - are `ARCHITECTURE_GPU_VULKAN_GEMM.md`'s sections 2.2k-2.2m, 2.2q and +2.2ae. What a model has to fit on +the card before any of this runs - the residency plan, and the marks swap that lets one GPU +slot serve many models - is `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections 2.2n-2.2o. The +decode-era mechanisms of the per-op tier are `ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections +2.2r-2.2v. The GPU backend role table these sections build on stays in `ARCHITECTURE_GPU.md` +sec.1.5. ### 2.2j The Vulkan resident prefill window chain {#vk-prefill-window-chain} -Companion to `ARCHITECTURE_GPU.md` sec.1.5; the Metal prefill driver's own ladder is -`ARCHITECTURE_GPU_PREFILL.md`. - **A prompt longer than `PF_WINDOW` rows runs as SEQUENTIAL windows over the same activation buffers.** Every window's rope and attention address the KV mirror at ABSOLUTE positions, so window w attends everything the earlier windows stored; only the last window runs the final @@ -24,19 +24,20 @@ requant and the classifier. **The last layer's FFN runs on the window's last 32 rows only.** Nothing downstream of the final layer reads more than the last row - the classifier requantizes row `wlen - 1`, the KV -mirrors were stored before the FFN, and a later window starts from fresh embeddings - so the +mirrors are stored before the FFN, and a later window starts from fresh embeddings - so the gate, up and down GEMMs, the activation and the residual step of the last layer take a region starting 32 rows below the window's end (`fill_arena_batch_sched`'s `row0`, `ActArgs.elem0`, -`ArArgs.row0`). Thirty-two, not one, because the s tile's fast path loads a whole 32-row -column unclamped and the resident prefill's activation planes (`pf_xf`, `pf_hf`) carry no read -slack past the window - unlike the MoE chain's gathered image and hidden plane, which -`_GEMM.md` sec.2.2l sizes with 32 rows of slack past their last region. Rows below the -slice keep stale gate, up, hidden and residual values that nothing reads. The sliced GEMMs do -not split k: the split-k reduce sums partial planes from row 0, so a region starting below the -window's end would reduce the wrong rows. The slice takes the f16-fed cm2 route only -(`gu6 && dn6`); the other feeds run the full window. Only the plain residual step (`cls_ar`) -and the f16 activation honor the row base: the fused residual twins feed the NEXT layer's -projections and never run on the last layer, so they index from row 0 by design. +`ArArgs.row0`). Thirty-two, not one, because the s tile - the cm2 tile with 32-row columns +(`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l) - loads a whole 32-row column unclamped on its +fast path, and the resident prefill's activation planes (`pf_xf`, `pf_hf`) carry no read slack +past the window - unlike the MoE chain's gathered image and hidden plane, which sec.2.2l sizes +with 32 rows of slack past their last region. Rows below the slice keep stale gate, up, +hidden and residual values that nothing reads. The sliced GEMMs do not split k: the split-k +reduce sums partial planes from row 0, so a region starting below the window's end would reduce +the wrong rows. The slice takes the f16-fed cm2 route only (`gu6 && dn6`); the other feeds run +the full window. Only the plain residual step (`cls_ar`) and the f16 activation honor the row +base: the fused residual twins feed the NEXT layer's projections and never run on the last +layer, so they index from row 0 by design. **The k and v GEMMs merge into ONE dispatch when the layer's q, k and v weight planes are all q8 and the k and v planes sit adjacent in the arena.** The bump allocator places them @@ -72,27 +73,6 @@ reads it whole. The intervening attention, requant and `wo` work is what closes the copies carry a `//!` naming this section, and the placement is a driver-defect mitigation, not a chain-shape preference. -**A recurrent (deltanet) layer's window block replaces the attention head; the FFN tail is -shared.** Per window: the block's feed (f16 rows when qkv, z and out all admit the cm2 tiles, else the -q8 image), the qkv and z GEMMs into the window planes (the planes in their file formats - the loader tags a dense hybrid's deltanet planes natively when this driver will be attempted, so a Q5_K/Q6_K file rides the k5/k6 tiles; the out plane is q8, the step's o row feeds it so), the beta and alpha rows into the layer's smalls -(f32 arm: a 16-position tile GEMM over the `[beta ; alpha]` rows, its grid position tiles by 16-output groups with one output per invocation, so a layer of only `2 x nvh` rows - 64 on the 9B - still fills the card; q8 arm: two q8 GEMMs and copies), the -conv reading the layer's ring image, the sequential scan over the layer's own state slot, the o rows' feed (f16 or requant) and the out GEMM into `pf_xb2`. The -scan is the plain per-token delta rule: a four-subgroup workgroup per (head, column group), a lane keeps -16 state rows of two adjacent columns in registers (two independent chains that interleave), the token's k and q rows are staged once per workgroup in -shared and feed both columns, the tokens loop inside the kernel with two shuffle reductions per column each, the raw o rows land in the tier's workspace for the gated out-norm's one workgroup per position. -The conv history crosses windows position-major in ring image 0; the last window transposes the -tail into the decode step's per-channel layout (`dn_tail_cls`; the handoff is `_DECODE.md` -sec.2.2v's). Every window past the first carries at least the conv taps: when the rows left after -a full window would be fewer than the taps, that window gives them up so the last one holds the -taps, and only a lone first window can be shorter - its history is zero, so the tail writes the -ring's leading rows as zero (`DnTailArgs.zero_rows`). Off the f16 feed a K-quant qkv/z pair reads -the Q8_K activation form, as the attention head's kq planes do, and the q8 beta/alpha arm -re-requantizes the rows Q8_0 behind the z GEMM (one feed, two forms - the decode's rule). -Gated attention rides the batch kernels through a per-head q stride (`qhs = 2 x hs`: -the q GEMM writes `[q | gate]` per head, qk-rms and rope read q head-strided in place, the mirror -attention gates on the sigmoid of the gate half); partial rotary is the `half = rot / 2` word. -At head 256 the window takes the h256 cm2 flash stamps (Br 64, Bc 32, the h128 loop with the head-shaped tiles doubled): the gated twins load Q at the head's q stride and scale the normalized output by the sigmoid of the gate half before the store; the h128 coopmat twin stays 128-only. - **A layer's qkv feed comes out of the previous layer's FUSED add+rms twin when the fuse knob is on and the feed is not the Q8_K quant form.** The producer is layer l-1's addr_next site, the consumer is layer l's b+0 slot, and both key on one predicate (`pf_qkv_feed_fused`): where it @@ -108,12 +88,21 @@ writes the `wo` feed plane directly, so the per-layer attn-to-f16 convert never f32 instance serves the quant route. The two device converts agree bit for bit; the CPU's `float16()` rounds ties differently, so the twin's gate compares device against device. +**A hybrid's gated attention rides the batch kernels through a per-head q stride** (`qhs = 2 +x hs`, twice the head size): the q GEMM writes `[q | gate]` per head, qk-rms and rope read q +head-strided in place, and the mirror attention gates on the sigmoid of the gate half. A +partial-rope model rotates the first `rot` elements of a head, and the kernels read the count +as the `half = rot / 2` argument word. At head size 256 the window takes the h256 cm2 flash +stamps (Br 64, Bc 32, the h128 loop with the head-shaped tiles doubled): the gated twins load +q at the head's q stride and scale the normalized output by the sigmoid of the gate half +before the store; the h128 coopmat twin stays 128-only. + ### 2.2p The Q8 requant writers store one quant per byte {#q8-requant-byte-store} Every requant writer on the class rail - the prefill and decode-tail kernels that write Q8_0 or Q8_K quants - declares its output plane `array` and stores one quant per element, over SPIR-V's 8-bit storage path; the fused decode step `DnStepFused` keeps its packed-word head -requant, the one writer outside this rule. Packing four quants into a `uint` +requant, the one writer that packs instead. Packing four quants into a `uint` instead costs a shift-and-or chain per word, and in a Q8_K writer - where four co-active lanes each hold one byte of the word - two subgroup shuffles per element on top. The stored bytes are the same under either form: the amax fold, the scale and the rounding decide them, and all @@ -130,9 +119,76 @@ per 64 lanes. The tables also exist as per-index accessors over a `fixed_array` (`iq2s_grid_word` and kin), which the batch and cm2 tiles stage from, because a tile amortizes one stage over 128 rows x 64 columns. A per-row kernel cannot: the emitter lowers such an accessor to a constant composite stored into a Function variable, the driver serves the -per-lane indexed read of it serially, and the two-row GEMV workgroup paid that serial read on -every 2 x 5120 weights it walked - iq2s streamed at 84 GB/s where k4 streams at 410. The +per-lane indexed read of it serially, and a two-row GEMV workgroup pays that serial read on +every 2 x 5120 weights it walks - 84 GB/s for iq2s against k4's 410. The buffer form puts every grid GEMV in the k-format band (iq2s 388, iq2xs 407, iq2xxs 400, iq3s 415 GB/s on the reference card, `harness/vk_gemv_probe.das`). The buffer dies in the model-drop sweep with every other device buffer, and its handle zeroes there, so the next model's first kq set rebuilds it. + +### 2.2ac The tile probe's arms share one descriptor set layout {#khrx-shared-set-layout} + +`khrx` is the lever sweep of the KHR kq tile (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2ae) in +`harness/vk_gemm_probe.das` (`ARCHITECTURE_MEASUREMENT.md` sec.2.5): eight arms timed over one +shape. Two are shipped bodies - the KHR class, the reference each compared arm is checked +against, and the sdot4 kq tile. `ship` is the KHR tile copied with no lever moved, the control +the lever arms read against, and the four lever arms are that copy with one lever - the weight +stage (`dec4`, the four-wide decode callback in place of the word stage), the accumulator +width, the subgroup tiling, the workgroup size - moved back to its simpler form. +Every arm but the sdot4 one binds the same five-buffer descriptor set layout as the shipped +class, so an arm's figure differs from the shipped class's by its body alone and never by a +binding difference. The sdot4 arm reads its own Q8 activation fixture, so it binds that +class's own six-buffer set, and it is timed rather than compared. + +The `nil` arm is the ceiling arm: its weight stage writes constants, so its rate is the +tile's ceiling with the weight loads removed. It still binds the weight plane and the scale +words the shared layout declares, and its stage reads neither (`KhrPxNil`, whose `wq` field +carries `@role = "alias"`). Those two are the probe's deliberately unread bindings. + +### 2.2ad The recurrent block of the prefill window {#vk-prefill-dn-block} + +**A recurrent (deltanet) layer's window block replaces the attention head; the FFN tail is +shared.** Per window the block runs the qkv and z GEMMs into the window planes, the beta and +alpha rows into the layer's smalls, the conv over the layer's ring image, the sequential scan +over the layer's own state slot, and the out GEMM into `pf_xb2`. The smalls are the layer's +per-layer f32 plane: the layer's cold constants (conv taps, out-norm weights, the `a` and `dt` +rows), the two parity ring images, and the beta and alpha rows. The weight planes stay in their file +formats where the loader tags them natively (`ARCHITECTURE_GPU_VULKAN_DECODE.md` sec.2.2v +carries the tagging condition), so a Q5_K/Q6_K file rides the k5/k6 tiles. + +The block reads two activation feeds, and each one is decided on its own. + +- **The x feed is the layer's input rows.** The qkv and z GEMMs read it. `pf_dnx6` decides it: + f16 rows when both of those planes admit the coopmat tiles (`ARCHITECTURE_GPU_VULKAN_GEMM.md` + sec.2.2l), else the quant image. +- **The o feed is the scan's output rows.** The out GEMM reads it. `pf_dno6` decides it: f16 + rows when the out plane admits the coopmat tiles, else a requant. + +A K-quant file whose out plane is Q8_0 therefore keeps its qkv and z GEMMs on the coopmat +tiles and takes the requant on the o feed alone. + +When the x feed is not f16, a K-quant qkv/z pair reads the Q8_K activation form, as the +attention head's kq planes do, and the q8 beta/alpha arm re-requantizes the rows Q8_0 behind +the z GEMM - one feed, two forms, as the decode step does (`ARCHITECTURE_GPU_VULKAN_DECODE.md` +sec.2.2v). + +The beta and alpha rows take one of two arms. The f32 arm is a tile GEMM over the +`[beta ; alpha]` rows: one workgroup covers 16 positions by 16 output rows, one output per +invocation, and the grid runs over both group axes, so a layer of only `2 x nvh` rows (`nvh`, +the layer's value-head count) - 64 on the 9B - still fills the card. The q8 arm is two q8 GEMMs +and copies. + +The scan is the plain per-token delta rule. One four-subgroup workgroup runs per (head, column +group). A lane keeps 16 state rows of two adjacent columns in registers, so it runs two +independent chains that interleave. The token's k and q rows are staged once per workgroup in +shared memory and feed both columns. The tokens loop inside the kernel; each token costs two +shuffle reductions per column. The raw o rows land in the per-op tier's workspace, for the +gated out-norm's one workgroup per position. + +The conv history crosses windows position-major in ring image 0; the last window transposes the +tail into the decode step's per-channel layout (`dn_tail_cls`; the handoff is stated in +`ARCHITECTURE_GPU_VULKAN_DECODE.md` sec.2.2v). Every window past the first carries at least as +many rows as the conv has taps: when a full window would leave the last window fewer rows than +that, the earlier window takes fewer rows instead, so the last one still holds the taps. Only a +lone first window can be shorter - its history is zero, so the tail writes the ring's leading +rows as zero (`DnTailArgs.zero_rows`). diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md index 0fd18caa30..91a6913ce1 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_DECODE.md @@ -5,8 +5,9 @@ document carries sections 2.2r-2.2v: the decode attention block over per-layer K the streamed expert layer's GPU/CPU split, the whole-token decode span, the deltanet decode step's per-session resident state, and the whole-model driver's hybrid token command. The prefill window chain and byte stores these build on are `ARCHITECTURE_GPU_VULKAN.md` sections -2.2j, 2.2p and 2.2ab; the cm2 tiles and the MoE expert chain on them are -`ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m and 2.2q; the residency plan and the marks +2.2j, 2.2p, 2.2ab, 2.2ac and 2.2ad; the cm2 tiles, the MoE expert chain on them and the KHR +arm's kq tile are `ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m, 2.2q and 2.2ae; the +residency plan and the marks swap under them are `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` sections 2.2n-2.2o. ### 2.2r The per-op tier's decode attention block {#decode-attention-block} @@ -243,7 +244,7 @@ arm, so a gated or partial-rope model takes the fused kernel whatever the fuse g two arms need qk-norm, and a model with either but without it declines by name. **The prefill window chain carries the same three arms** (`ARCHITECTURE_GPU_VULKAN.md` -sec.2.2j): a recurrent layer's window block runs the qkv and z batch GEMMs, the beta and alpha +sec.2.2ad): a recurrent layer's window block runs the qkv and z batch GEMMs, the beta and alpha rows into the layer's own smalls, the conv, the sequential scan over the layer's own device state slot (the raw o rows in the tier's workspace), the o requant and the out GEMM into the block output, so the window command needs no host round trip per layer. The state slots are the diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md index 2a0282626d..dc7d9cfc3b 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_GEMM.md @@ -1,15 +1,17 @@ # dasLLAMA Architecture - the Vulkan tier's GEMM tile family Companion to `ARCHITECTURE_GPU_VULKAN.md`; section numbers are `ARCHITECTURE.md`'s. This -document carries sections 2.2k-2.2m and 2.2q, the cooperative-matrix tiles the Vulkan tier's -GEMMs run on: how a cm2 tile decodes its quant bytes, how a tile and the served GEMM mode are -picked, the class-pipeline build seat both shader instruments hang on, and the MoE expert chain -on those tiles. The prefill window chain that dispatches them, the Q8 requant byte store and the -decode GEMV family's grid codebook buffer are `ARCHITECTURE_GPU_VULKAN.md`'s sections 2.2j, 2.2p -and 2.2ab. What a model has to fit on the card before any of this runs is -`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections 2.2n-2.2o. The decode-era mechanisms of the -per-op tier are `ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections 2.2r-2.2v. The GPU backend role -table these sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. +document carries sections 2.2k-2.2m, 2.2q and 2.2ae, the cooperative-matrix tiles the Vulkan +tier's GEMMs run on: how a cm2 tile decodes its quant bytes, how a tile and the served GEMM +mode are picked, the class-pipeline build seat both shader instruments hang on, the MoE expert +chain on those tiles, and the KHR arm's hand-staged tile. `ARCHITECTURE_GPU_VULKAN.md` carries +the prefill window chain that dispatches them (sec.2.2j) and its recurrent block (sec.2.2ad), +the Q8 requant byte store (sec.2.2p), the decode GEMV family's grid codebook buffer +(sec.2.2ab), and the tile probe's shared descriptor set layout (sec.2.2ac). What a model has to +fit on the card before any of this runs is `ARCHITECTURE_GPU_VULKAN_RESIDENCY.md`'s sections +2.2n-2.2o. The decode-era mechanisms of the per-op tier are +`ARCHITECTURE_GPU_VULKAN_DECODE.md`'s sections 2.2r-2.2v. The GPU backend role table these +sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. ### 2.2k The cm2 decode callbacks read their quant bytes as 16-bit lanes {#cm2-decode-16bit-lanes} @@ -18,52 +20,63 @@ compiler pattern-matches only one spelling into that path: a 16-bit load (`int16 members) followed by `unpack8(w)[i & 1u]` - a byte2 lane select - with sub-fields pulled out by shift and mask. A 32-bit word with a variable shift runs slower; an `unpack8` of a 32-bit word indexed by a runtime value (a byte4 dynamic select) drops the whole kernel off the block-load -path, to about a third of the rate. Every cm2 decode - q8 and the thirteen kq superblock formats - -is spelled the 16-bit way, which is why the block structs are `int16` arrays over the same -bytes. The IQ4_XS codebook is the one runtime-indexed read a decode makes: it is staged into a -16-entry `@workgroup` f16 table ahead of the tile loop (the reference exe's shared-memory table-staging form), -never selected out of a register vector per element. - -Every kq format's four-wide twin is hand-laid (`decode_v4`, the template's `DECV4` axis): it +path, to about a third of the rate. Every cm2 decode - q8 and every kq superblock format - is +spelled the 16-bit way, which is why the block structs are `int16` arrays over the same bytes. +Every table a decode reads at a runtime index is staged into a `@workgroup` array ahead of the +tile loop, never selected out of a register vector per element: the iq4 formats' 16-entry +codebook (`kvalues_iq4nl`, shared by IQ4_XS and IQ4_NL) as f16, each grid format's codebook as +its own word array. That is the shared-memory table-staging form of the reference exe - +llama.cpp's Vulkan build, the upstream binary `benchmarks/lcpp_bench.das` measures against +(`ARCHITECTURE_MEASUREMENT.md` sec.2.5). + +Every kq format's four-wide twin is hand-written (`decode_v4`, the template's `DECV4` axis): it keeps the same spelling and shares what four consecutive elements share. A K-quant twin reads its four quant bytes as two 16-bit lanes and extracts the sub-block's scale pair once; a grid format's twin looks its grid word up once and takes the four bytes and the four sign bits from it. The synthesized twin (`DECVEC`, the axis a new format starts on) repeats the whole scalar body four times - the lane selects, the scale-plane words, the grid lookup and the sign parity -- and on the grid formats it lost to the scalar callback for exactly that reason. The twin -computes each element in the scalar's operation order, so the tile's CPU oracle holds under +- and on the grid formats it runs slower than the scalar callback for exactly that reason. The +twin computes each element in the scalar's operation order, so the tile's CPU oracle holds under either callback; which callback a box runs is the `device ready` line's `four-wide decode`. +One decode body serves the tensor load's callback, the cm2 tiles and the CPU oracle. A kernel +body can also call it on the plane element itself - `decode_v4(wq[i], ...)`, where the +element's index travels and the callee chains through the plane, so no block is copied. The +test `test_vkd_direct_decode` in `tests/test_vulkan_kernels.das` keeps that emitter capability. + ### 2.2l The cm2 tile pick and the coopmat default ladder {#cm2-tile-pick-and-default} **The l/m tile pick is a wave-efficiency comparison.** For a GEMM of width `d` over `cnt` rows -the l tile (256-row columns) and the m tile (128-row columns) each need some number of -workgroups; each grid runs in whole waves over the device's SM count, and the pick compares -occupied slots over allocated slots, cross-multiplied. The m tile wins only on a strict win; a -tie goes to l, whose bigger tile carries twice the arithmetic intensity. Three rules sit ahead -of the comparison: a region of 64 rows or fewer takes the s tile (32-row columns - the MoE -expert-bucket shape, where a 512-token window routes ~32 rows to each of 128 experts and an m -column would pad three quarters of every tile and take the edge path on all of them), a window -of 128 rows or fewer takes m (the l column would run half empty), and a device that reports no -SM count takes l and never splits k. The pick is PURE in `(d, cnt, sm_count)`, so the class the -pipeline binds and the tile rule the meta fill writes can never disagree; `cnt` is the AVERAGE -rows per active region of the dispatch, so one tile serves every region of a MoE schedule. The -narrow-n end below s is GEMV's. The s tile's fast path loads a partial 32-row column UNCLAMPED -and clamps only the store, so every f16 plane the chain feeds it - the gathered activation -image and the hidden plane - is sized with 32 rows of slack past its last region -(`ffn_cm2_chunk_rows`). - -**The split-k pick counts the dispatch group, not the GEMM.** Long K (2048 and up) on a grid -that would fill under half the SMs splits the reduction across f32 partial planes that -`SplitKReduce` sums (three chunks up to two thirds full, at most eight, each chunk 256-aligned -and a split that would strand an empty tail shed). The grid it measures is the role's own -workgroups PLUS those of the chain neighbours it runs beside - q with k and v, gate with up -(`cm2_tiles`, the same pick each neighbour's own dispatch makes) - because the hazard-mask rail -lets independent roles co-run, while every split role serializes through the one scratch plane -(`VHZ_SK`) its neighbours would also claim. Counted alone, a 512-wide k or v projection over a -512-row window fills 16 of 36 SMs and splits in two; counted beside q it runs whole, and k and v -fill the device together. Split-k is left to the lone role - wo, down, a small model's -classifier - whose grid nothing else pads. +the l tile (256-row columns) and the m tile (128-row columns) each take some number of +workgroups. Each grid runs in whole waves over the device's SM count, so a grid's wave count +times that SM count is the slots it allocates. The pick takes the tile whose workgroups fill +the larger share of its allocated slots, the two ratios compared by cross-multiplying. The m +tile wins only on a strict win; a tie goes to l, whose bigger tile carries twice the arithmetic +intensity. Three rules sit ahead of the comparison: a region of 64 rows or fewer takes the s +tile (32-row columns - the MoE expert-bucket shape, where a 512-token window routes ~32 rows to +each of 128 experts and an m column would pad three quarters of every tile and take the edge +path on all of them), a window of 128 rows or fewer takes m (the l column would run half +empty), and a device that reports no SM count takes l and never splits k. Beyond `(d, cnt, +sm_count)` the pick reads only two values fixed at init - the served mode and +`DASLLAMA_CM2_TILE` - so the class the pipeline binds and the tile rule the meta fill writes +can never disagree; `cnt` is the AVERAGE rows per active region of the dispatch, so one tile +serves every region of a MoE schedule. A region below the s tile's row count goes to the decode +GEMV family, not to a tile. The s tile's fast path loads a partial 32-row column UNCLAMPED and +clamps only the store, so every f16 plane the chain feeds it - the gathered activation image and +the hidden plane - is sized with 32 rows of slack past its last region (`ffn_cm2_chunk_rows`). + +**The split-k pick counts the dispatch group, not the GEMM.** With long K (2048 and up), a grid +that fills at most half the SMs splits its reduction across f32 partial planes that +`SplitKReduce` sums, into as many chunks as fill the device (SM count over workgroups); a grid +that fills up to two thirds splits into three. Eight chunks is the ceiling. Each chunk is +256-aligned, and a chunk count whose last chunk would then be empty drops by one. The grid it +measures is the role's own workgroups PLUS those of the chain neighbours it runs beside - q +with k and v, gate with up (`cm2_tiles`, the same pick each neighbour's own dispatch makes) - +because the hazard-mask rail lets independent roles co-run, while every split role serializes +through the one scratch plane (`VHZ_SK`) its neighbours would also claim. Counted alone, a +512-wide k or v projection over a 512-row window fills 16 of 36 SMs and splits in two; counted +beside q it runs whole, and k and v fill the device together. Split-k is left to the lone +role - wo, down, a small model's classifier - whose grid nothing else pads. **The f16 feed admits q8 and every kq superblock format** (`kq_sb`) - the set the cm2 decode callbacks cover (sec.2.2k) - and each (format, tile) pair has ONE stamped class. The @@ -79,82 +92,35 @@ NV_cooperative_matrix2, else mm where it has KHR_cooperative_matrix, else sdot4; `DASLLAMA_COOPMAT` overrides the ladder by name, and a cm2 request or force on a device without the extension lands on mm. The same resolver stamps the mode into the `.dlim` flavor configuration, so the recorded mode and the running mode cannot drift. The four-wide decode -callback is NOT in that configuration: a cm2 tile names both callbacks -(`coopmatLoadTensorDecode`'s tenth argument: the format's own `decode_v4` where the template's -`DECV4` axis is on - every kq superblock format today, sec.2.2k - else the `DECVEC` axis, -which synthesizes the twin from the scalar body and is where a new format starts, its -`cm2:` probe row deciding whether a hand-laid twin is owed), the device created with -`DASLLAMA_VK_DECVEC` and the extension decides which -one the driver runs, and neither choice shapes an image byte, so the bake identity ignores it -(the configuration's own rule: a serve-only knob is never a field). `decvec_on` is the run's arm, -announced on the `device ready` line. - -**The mm mode serves the kq formats through the same template's KHR arm.** A device with -KHR_cooperative_matrix and no NV_coopmat2 (every AMD and Intel part, the GTX and Turing lines) -has no decode-in-load tensor API, so the cm2 template carries a second body under its `KHR` -axis: the f16 weight tile is staged by hand - each of the 256 threads copies one weight row's -16-wide half of the 32-wide k step through the format's OWN decode (`decode_v4` where the format -has one, else `decode`, the same methods the tensor load names as callbacks, here called -directly), the activation half from the f16 plane as two 16-byte words (the plane aliased as -`uint4` on the same binding; sixteen scalar f16 loads ran the tile at two thirds of the rate - -the staging's global loads, not the multiply-adds, set the pace), both into `@workgroup` `uint` -arrays of f16 pairs at a stride of 20 words (16 plus 4 pad, so the fragment loads spread across -banks; the activation words land as they are, the decoded weights packed two halves to a word) - -and eight subgroups each own a 16-row weight strip across the 128 tokens as eight 16x16 f32 -accumulators, two 16-deep multiply-adds per staged step. A whole tile stores its fragments -straight into y column-major at stride `d` (y is token-major, so a fragment's (weight, token) -is `y[token * d + weight]`); an edge tile bounces each fragment through the weight staging -array - free once the k loop ends, and its 2560 words hold the eight subgroups' 256-word -fragments - and writes under the row and column bounds through a bit cast. The two staging -arrays are the whole footprint by design: a third array for the bounce cost the tile a -seventh of its rate (the probe's `slab` arm, 28.5 against 33.0 TFLOP/s on the 4B gate shape), -because 8 KB more shared memory per workgroup is one workgroup fewer per SM. The k step is 32 -(the reference exe's BK): a 64-deep step doubles the staging tiles, halves the workgroups an SM -holds, and with the 8 KB iq2s grid beside them reaches the 49152 B of workgroup memory the tier -requires of a device - its floor; a 32 KiB part is not a target, and no kernel here is sized -for one. The arm exists at ONE geometry - 128 weights -by 128 tokens, k step 32 - so in mode 3 the tile pick answers 128 and split-k never engages, and -`cm2_cls_ensure/set/enc` route to the `khr_cls_*` ladders, the same `(fmt)` key on both. The -f16 feed admits a kq format in mode 3 only on a 32-lane subgroup (`khr_kq_tile_on`): the body -indexes eight subgroups over the 128 weight rows, so a wave64 device (four subgroups per -256-thread workgroup) keeps its kq planes on the sdot4 batch tile until the wave64 twin lands. q8 never arrives here -- its mm-mode GEMM is the q8-fed mul_mm L-tile. The direct call passes the plane element -itself (`decode_v4(wq[i], ...)`): the emitter hands the callee the element's index and the -body chains through the plane, so no block is ever copied. The copy form - a `let` of the -element passed by value, which the emitter spills to a Function-storage local - ran the same -tile at well under half the rate (8.0 against 19.3 TFLOP/s on the 4B gate shape with scalar -activation loads, 29.6 with the 16-byte ones), the whole 128 to 176 B block loaded from the -plane and stored to the local on every call, four calls per thread per k step; the -plane-element form measures within noise of hand-inlined decode arithmetic. The measured -alternatives that lost: B fragments loaded straight from the plane with no staging (17.1), a -64-deep k step (15.7 - its 40 KB footprint halves the workgroups an SM holds), and the separate -bounce slab (28.7). The rows are `harness/vk_gemm_probe.das -- khrx`, best of four interleaved -rounds on the RTX 5060 Ti: the shipped class reads 32.1 on the 4B gate shape, the probe's copy -of its body 33.2, and the weight stage replaced by a constant fill 41.2 - the ceiling the -staging's global loads leave. -Either way the decode body is authored once and serves the tensor load, the KHR staging and -the CPU oracle alike. +callback is not in that configuration. A cm2 tile names both callbacks +(`coopmatLoadTensorDecode`'s tenth argument): the format's own `decode_v4` where the template's +`DECV4` axis is on - every kq superblock format, sec.2.2k - else the `DECVEC` axis, which +synthesizes the twin from the scalar body and is where a new format starts; its `cm2:` +probe row is what decides whether the format gets a hand-written twin. The device decides which +one the driver runs - it is created with `DASLLAMA_VK_DECVEC` and the extension - and neither +choice shapes an image byte, so the bake identity ignores it: a serve-only knob is not a +configuration field. `decvec_on` is the run's arm, announced on the `device ready` line. **The tile's fast path is what makes the loads unclamped.** It runs when the weight tile is whole (`m0 + 128 <= d`), the token column is whole or stamped s, and K is a whole number of BK steps; the layouts are then created clamp-Undefined and the B and output strides are masked to -a multiple of 8 f16 (`stride &= ~7`). The mask is an identity on today's shapes - `n` and `d` -are 32-multiples - and it exists to make the alignment PROVABLE to the driver's address -analysis, which is what keeps the loads on the wide path. The s column gates only the weight -tile: its partial token column loads unclamped and its store clamps. Everything else takes the -edge path with clamped layouts. +a multiple of 8 f16 (`stride &= ~7`). The mask changes nothing while `n` and `d` are +32-multiples, which every served shape is; it exists to make the alignment PROVABLE to the +driver's address analysis, which is what keeps the loads on the wide path. The s column gates +only the weight tile: its partial token column loads unclamped and its store clamps. Everything +else takes the edge path with clamped layouts. **The no-split arm keeps literal loop bounds and a literal store base.** Where `ksplit` is zero the k loop runs the literal `0 .. n` with the store at the row base rather than the general `k0`/`k1`/`ybase` form, although those values are exactly `0`, `n` and `0` on that path: the -general spelling cost 27% of prefill throughput (`benchmarks/lcpp_bench.das` pp512, 5060 Ti). +general spelling costs 27% of prefill throughput (`benchmarks/lcpp_bench.das` pp512, 5060 Ti). The split arm keeps the general form. ### 2.2m Class-pipeline creation is the Vulkan tier's one shader A/B seat {#vk-class-pipeline-build} `vkd_class_pipe` is the single place a class kernel's SPIR-V becomes a pipeline, so both shader instruments hang there and nothing else has to know about them. The four-wide decode fallback -hangs there too: when the device was created without `VK_NV_cooperative_matrix_decode_vector` +hangs there too: when the device carries no `VK_NV_cooperative_matrix_decode_vector` (`decvec_on` false) the served words go through `strip_decode_vector` (the capability, the extension and every load's `DecodeVectorFunc` operand removed, the scalar callback left to serve), after the override and before the shader module, so a dumped or overridden blob is @@ -171,7 +137,7 @@ ones. **Full subgroups are a whole-run arm, never a per-pipeline one.** `DASLLAMA_VK_FULLSG` plus a device that reports the feature sets `g_gpu.full_sg_on` once at device init, and every class pipeline is then built with `REQUIRE_FULL_SUBGROUPS`. A run never mixes pinned and plain -pipelines, so an A/B compares two whole runs. Plain is the default: pinned measured slower on +pipelines, so an A/B compares two whole runs. Plain is the default: pinned measures slower on the mm_a gate shape. ### 2.2q The MoE expert batch arm rides the cm2 tiles through a device-side f16 gather {#cm2-expert-chain} @@ -185,27 +151,75 @@ its position's row as f16 into the entry's bucket row, the inverse walk of the c same map. Gate and up then run the cm2 decode-in-load tiles over that f16 image, the act writes the hidden plane as f16, and down runs the cm2 tiles again - the resident dense chain's `pf_gemm_enc` feed, with the region records the quant form already fills. The engine asks the -tier per layer (`moe_gpu_ffn_xf_ok`): the answer is yes only in mode 4 on a coopmat2 device, for -a gate/up/down triple whose every format the f16 feed admits (sec.2.2l), with the window inside -the x plane's cap - and on yes it skips its own requant and gather, so the CPU cost of the -layer's FFN is the routing alone. The f16 form is the combined (`npos > 0`) form only: the +tier per layer (`moe_gpu_ffn_xf_ok`): the answer is yes only in cm2 mode on a coopmat2 device, +for a gate/up/down triple whose every format the f16 feed admits (sec.2.2l), with the window +inside the x plane's cap - and on yes it skips its own requant and gather, so the CPU cost of +the layer's FFN is the routing alone. The f16 form is the combined (`npos > 0`) form only: the combine is what makes the device-side gather pay, since neither the gathered image nor the bucket rows ever cross PCIe. Streamed groups take the same arm after the slot bind. **The per-op attention chain runs the same cm2 flash-attention tile the resident chain runs** -(`fa_cm2_h64` / `h128`, `ARCHITECTURE_GPU_VULKAN.md` sec.2.2j) when the device carries the coopmat2-fa trio, the fa knob is -on, the head size is 64 or 128, and the model's attention is not gated - this chain wires neither -the h256 stamps nor their gated epilogue, so gated models keep the flash-style `at_attn` pass. The tile reads -f16 K/V: the chain keeps its f32 roped-k / raw-v planes at absolute positions for the host -readback the CPU cache store consumes, and fills f16 shadows of them with the base-less -`f16cvt` over the whole attended prefix each window; the fa output lands in the same out plane -`at_attn` writes, so the requant and `wo` stages never learn which pass ran. +(`fa_cm2_h64` / `h128`, `ARCHITECTURE_GPU_VULKAN.md` sec.2.2j) when the device reports the cm2 +flash-attention features (`has_coopmat2_fa` - cooperative-matrix reductions, conversions and +per-element ops), the fa knob is on, the head size is 64 or 128, and the model's attention is +not gated - this chain wires neither the h256 stamps nor their gated epilogue, so gated models +keep the flash-style `at_attn` pass. The tile reads f16 K/V: the chain keeps its f32 roped-k / +raw-v planes at absolute positions for the host readback the CPU cache store consumes, and +fills f16 shadows of them with the base-less `f16cvt` over the whole attended prefix each +window; the fa output lands in the same out plane `at_attn` writes, so the requant and `wo` +stages never learn which pass ran. **The per-op attention chain adds a q/k/v projection bias (qwen2moe) in its prep stage.** The -layer's `[q | k | v]` row uploads to one device buffer per call and binds to `AtPrep`, whose q and -k passes add their slice (`boff` 0 and `qd`) to each projection element before the norm and the -rope - where the CPU chain adds it. v has no prep pass of its own, so a biased layer runs a third -`AtPrep` over the raw v window with the rope half 0 and the norm off, which makes the kernel a -copy plus bias in place (`boff` `qd + kv_dim`); the copy into the absolute-position v plane and -the host readback then both carry the bias. A model without the bias runs the two passes with -`hasb` 0 and never reads the binding, and the rail's `arch_ok` no longer names the bias. +layer's `[q | k | v]` row uploads to one device buffer per call and binds to `AtPrep`, whose q +and k passes add their slice (`boff` 0 and `qd`) to each projection element before the norm and +the rope - where the CPU chain adds it. v has no prep pass of its own, so a biased layer runs a +third `AtPrep` over the raw v window with the rope half 0 and the norm off, which makes the +kernel a copy plus bias in place (`boff` `qd + kv_dim`); the copy into the absolute-position v +plane and the host readback then both carry the bias. A model without the bias runs the two +passes with `hasb` 0 and never reads the binding, and the attention-quad rail's `arch_ok` test +does not name the bias. + +### 2.2ae The KHR arm's hand-staged kq tile {#khr-mm-kq-tile} + +**The mm mode serves the kq formats through the same template's KHR arm.** A device with +KHR_cooperative_matrix and no NV_coopmat2 (every AMD and Intel part, the GTX and Turing lines) +has no decode-in-load tensor API, so the cm2 template carries a second body under its `KHR` +axis, staged by hand. Each of the 256 threads writes one weight row's 16-wide half of the +32-wide k step. The weights go through the format's `khr_stage16`. It reads the row's 16 +consecutive values from the quant plane as one or two 16-byte words (`wq4`, the plane aliased +as `uint4` on its binding; the byte-granular grid formats read 32-bit words through `wqw`), +reads the row's scale words once, and writes the 16 values into the stage as f16 pairs. The +activations come straight from the f16 plane as two 16-byte words (that plane aliased as +`uint4` on its binding), stored as they arrive. Both land in `@workgroup` `uint` arrays at a +stride of 20 words (16 plus 4 pad, so the fragment loads spread across banks). q8 never arrives +here - its mm-mode GEMM is the q8-fed mul_mm L-tile. + +**The eight subgroups tile the 128 x 128 step two by four.** Each owns 64 weight rows against +32 tokens as eight 16x16 f16 accumulators (a `coopmatAcc_f16_16x16[8]` walked under +`for [unroll_full]`, so every fragment sits in registers) and loads four weight fragments and +two token fragments per 16-deep multiply-add round: 96 fragment loads per workgroup per k step +against 144 when every subgroup owned a 16-row strip across all 128 tokens. The accumulators +widen to f32 (`coopmatConvert`) before the store. A whole tile stores its fragments straight +into y (token-major, so a fragment's (weight, token) is `y[token * d + weight]`); an edge tile +bounces each widened fragment through the weight staging array - free once the k loop ends, +and its 2560 words hold the eight subgroups' 256-word fragments - and writes under the row and +column bounds through a bit cast. Moving any of the three levers back - the word stage, the f16 +accumulator width, the two-by-four tiling - costs rate: the word stage the most, the +accumulator width next, the tiling least; the probe's `khrx` arms measure them +(`ARCHITECTURE_MEASUREMENT.md` sec.2.5) and `followup_vulkan.md` item 42 keeps the figures. On +a partial token column the edge store costs nothing beyond the padded rows: per computed row a +300-token window runs at the whole-window rate. + +**The two staging arrays are the whole footprint by design.** A third array for the bounce +costs the tile a seventh of its rate, because 8 KB more shared memory per workgroup is one +workgroup fewer per SM. The k step is 32 (the reference exe's BK): a 64-deep step doubles the +staging tiles, halves the workgroups an SM holds, and with the 8 KB iq2s grid beside them +reaches the 49152 B of workgroup memory the tier requires of a device. The tile is sized to +that 49152 B floor; a device that offers less workgroup memory is not a target. + +**The arm exists at one geometry** - 128 weights by 128 tokens, k step 32 - so in mm mode the +tile pick answers 128 and split-k never engages, and `cm2_cls_ensure/set/enc` route to the +`khr_cls_*` ladders, the same `(fmt)` key on both. The f16 feed admits a kq format in mm mode +only on a 32-lane subgroup (`khr_kq_tile_on`): the body indexes eight subgroups over the tile, +so a wave64 device (four subgroups per 256-thread workgroup) keeps its kq planes on the sdot4 +batch tile. diff --git a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md index 96e5f934b1..c9d125e471 100644 --- a/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md +++ b/modules/dasLLAMA/ARCHITECTURE_GPU_VULKAN_RESIDENCY.md @@ -3,9 +3,9 @@ Companion to `ARCHITECTURE_GPU_VULKAN.md`; section numbers are `ARCHITECTURE.md`'s. This document carries sections 2.2n-2.2o: the residency plan that sizes a whole model before a byte uploads, and the marks swap that lets one GPU slot serve many models. The prefill chain and byte -stores that run once a model is resident are `ARCHITECTURE_GPU_VULKAN.md` sections 2.2j, 2.2p -and 2.2ab, and the cooperative-matrix GEMM tiles under them are -`ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m and 2.2q; the per-op tier's decode era is +stores that run once a model is resident are `ARCHITECTURE_GPU_VULKAN.md` sections 2.2j, 2.2p, +2.2ab, 2.2ac and 2.2ad, and the cooperative-matrix GEMM tiles under them are +`ARCHITECTURE_GPU_VULKAN_GEMM.md` sections 2.2k-2.2m, 2.2q and 2.2ae; the per-op tier's decode era is `ARCHITECTURE_GPU_VULKAN_DECODE.md` sections 2.2r-2.2v. The GPU backend role table these sections build on stays in `ARCHITECTURE_GPU.md` sec.1.5. @@ -48,8 +48,8 @@ at 4.35 tok/s, and with 4 GiB more the device returned all-zero logits with no e manager never demotes the idle other process; ours is the victim, so the room left is the only honest input. A pinned `VRAM_MB` takes no headroom and reads no room: the number is the user's contract, and the plan fills it. Every rate in this section is a `benchmarks/lcpp_bench.das` -tg128 reading (`-jit --for-debug-purposes`, `DASLLAMA_IMAGE=0`, the rig `plans/qwen_arc_board.md` -states) on the zen2 box's RTX 5060 Ti 16 GB, driver 616.56. +tg128 reading (`-jit --for-debug-purposes -r 5 -p 512 -n 128`, `DASLLAMA_IMAGE=0`, +`DASLLAMA_GPU=1`, 16 threads) on the zen2 box's RTX 5060 Ti 16 GB, driver 616.56. **Where no OS answers, the auto arm's headroom is the larger of 2 GiB and 27% of the tier's cap.** That share keeps a 16 GB card's plan near 10.7 GB and leaves an 8 GB card's plan where diff --git a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md index b6d925d939..7ff00b5ec0 100644 --- a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md +++ b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md @@ -1,6 +1,9 @@ # dasLLAMA Architecture - the benchmark rig and instrumentation rails -Companion to `ARCHITECTURE.md`; section numbers are that document's. +Companion to `ARCHITECTURE.md`; section numbers are that document's. The instruments that time +a kernel away from the served graph - the kernel race's fidelity conditions, the gemv's own +tune seat and the CPU kernel bench's fixture - are `ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md` +sections 2.21, 2.26 and 2.27. ### 2.5 There is ONE benchmark rig, and the records are the baseline {#one-benchmark-rig} @@ -14,8 +17,8 @@ two drivers run it: `lcpp_bench` from its loop, and dasllama-server's in-process step per tick on the model it serves. Only `lcpp_bench`'s rows become records: the server's are a self-measure the operator reads on the control page, stamped with the device, the KV codec, the exec tier and the tune state they ran under, and they enter no board, ledger or exchange. -The real `llama-bench` runs only when `--ref ` is passed; that is how the upstream -columns were produced, and they are pinned, not re-measured. +The real `llama-bench` runs only when `--ref ` is passed; the upstream columns come from +that run, and they are pinned, not re-measured. `performance/gen_bench_records.das` sweeps a board by spawning that rig once per cell, and writes `performance/records/.json`. `gen_site_records.das` merges those into the file the @@ -23,8 +26,11 @@ site renders. A stored row carries its own command, sha, version, tune stamp and a number is self-describing rather than a bare figure in a table. The command, its environment line and a sidecar's `binary` spell the home directory `~` (`tilde_home`, `daslib/fio`): a public row names no user and still reproduces on any box. The re-mint rule sanctions one edit -to a stored row - spelling its home directory `~` - since no measurement changes; an archived -tune sidecar is never edited, its sha256 being what ties a row to the exe it shipped with. +to a stored row - spelling its home directory `~` - since no measurement changes. An archived +tune sidecar's bytes are otherwise untouched: its sha256 is what ties a row to the exe it +shipped with. A sidecar archived as `records/.tune..json` is content-addressed: its +filename carries the hash of its bytes, so a re-stamp of `provenance.engine_sha` re-hashes and +renames the file, and a `records/.json` row's `tune_sha` names that file by that hash. **Regression checking inverts the same rig:** `gen_bench_records.das --oracle --legs metal` takes the store's das rows as the work list, re-measures each once, and gates one-sided against @@ -38,8 +44,9 @@ its device and its tune state are whatever the operator's box serves. **The tune stamp gates the comparison.** A manifest older than the binary fails every cell, and an untuned invocation stamps the shipped class profile (a box the profile does not cover -re-execs into the residue race) rather than measuring - so re-mint the box manifest and check -its winners against the stored rows' `tune` stamps before trusting a delta. +re-execs into the residue race - a tuner run over the families the profile leaves uncovered) +rather than measuring. A delta taken against stored rows whose `tune` stamps name other winners +compares two tune states, not two engines. **The Vulkan GEMM probe attributes prefill GEMM cost on three axes.** `harness/vk_gemm_probe.das` times one shape at a time: the serving GEMM against its alternates @@ -53,14 +60,27 @@ l and m columns with the kq batch tile as the control row, over random block byt format's device block size, and it runs the four-wide decode's two arms (the twin served, then stripped through `vkd_pipes_rebuild`) interleaved in one process, two rounds each, so a format's `DECVEC` verdict comes from one instrument. The `khrx` arm is the second axis for the KHR kq -tile: the staged k4 tile with its weight stage as the variable (a constant fill, the decode on -the plane element, the plane's words read in place), then the structural variants (unstaged B -fragments, a 64-deep k step, a separate edge-store slab), with the shipped KHR class and the -sdot4 tile as control rows. The `mmqx` arm is the first axis for the integer tile: the sdot4 -k4 tile against register-block prototypes over the same planes. Both sweeps read every -product-computing arm back against the shipped class, and time the served graph's shape: -sixteen dispatches per submit over two alternating outputs with a fresh hazard each, the arms -interleaved round by round, an arm's figure its best round. A new arm joins one of the three. +tile. It runs eight arms: a resync copy of the shipped k4 tile with no lever moved +(`khrpx_ship`), five copies each with one lever moved back - the weight stage as a constant fill +or as the four-wide callback on the plane element, f32 accumulators, a 16-row strip tiling, and +the reference exe's 128-thread geometry - and the shipped class and the sdot4 tile as controls. +The resync copy is the row a lever's arm is read against, and its bit-exact reading against the +shipped class is what says the copies still track the shipped body. The sweep runs three whole +windows and one partial window of 300 tokens, the row that takes the edge store; the copies +stage and store whole tiles, so on the partial window only the two controls run. +`khrprof:` submits one arm alone for a GPU profiler. The `mmqx` arm is the first axis for +the integer tile: the sdot4 k4 tile against register-block prototypes over the same planes. Both +sweeps time the served graph's shape first: sixteen dispatches per submit over two alternating +outputs with a fresh hazard each, the arms interleaved round by round, an arm's figure its best +round. The comparison follows the timing: each compared arm's last output is read back and +measured against the sweep's reference - the shipped class in `khrx`, the sdot4 kq tile in +`mmqx`. An arm that stages constants, or reads its own activation fixture, is timing-only and is +never read back. A khrx copy passes within a 2e-2 relative difference of the shipped class, and +reads bit-exact where the lever leaves the arithmetic alone. The shipped class itself is read +against the k4 CPU oracle on six corners of the output, at the kernel cell's bar: +`|gpu - cpu| <= 2e-2 |cpu| + 4e-3 max|y|`. The probe's exit code is non-zero on a compared arm +over its bound, a CPU-oracle miss, an unknown `khrprof` arm, or a run that produced no result +row. **A measured number proves its kernel provenance through `tune_gate()` (`performance/profile_common.das`), one arm per world it can run in.** Four worlds, because @@ -79,7 +99,7 @@ marks the rest `timing-only`; its rows never enter a record store, and a decisio confirmed by the e2e board rows. **A binary-stale sidecar still serves its `runtime` section; a foreign one serves nothing.** -The staleness rule kills measured kernel WINNERS - a rebuild can change the bodies they were +The staleness rule discards measured kernel WINNERS - a rebuild can change the bodies they were raced on - but the `runtime` knobs (lane caps, jobque shape, the `metal_tensor` crowns that turn the tensor mul_mm twins on) are properties of the box, not of the binary. The engine's no-path `apply_box_profile_runtime()` therefore takes the checked route: `stale_binary` applies the @@ -89,44 +109,19 @@ base forms - a prefill reads well under half its board cell - so `metal_decode_i warns when a profile was asked for, declined, and no crowns are set, and `lcpp_bench` stamps a cell that passed `tune_gate()` on `DASLLAMA_ALLOW_UNTUNED=1` with an `untuned:` flavor prefix. -**The retune re-exec bites scaffolding on a box its class profile does not cover, and the pin -for it is checked in.** A bare `daslang` run that requires the engine - a probe, a one-off -script, a REPL experiment - stamps the shipped class profile at compile time and runs; on a box -the profile does not cover it re-execs into the residue race when no manifest is armed, and on -either box it serves no runtime section (no Metal crowns - no profile ships those). -`performance/last_known_good_sidecar.json` exists for exactly that: a frozen copy of a complete, -noise-gated mint, tracked in git (the `*.tune.json` ignore rule deliberately does not match it). Point `DAS_TUNE_MANIFEST` at it and the framework -never retunes; on a different box the identity mismatch just serves fallbacks, and a copy minted -before the current `DASLLAMA_RELEASE` serves fallbacks on any box - the compile says which with -one `WARNING DAS_TUNE_MANIFEST` line per scope. That is the whole -contract - it suppresses the re-exec, it does not tune the box, and a number measured under it -is not a benchmark. Benches and the rig keep minting their own; refresh the copy when a -re-mint moves the crowns or `DASLLAMA_RELEASE` bumps. - -### 2.20 The ASR board's GPU row pairs {#asr-gpu-pairs} - -The das Metal ASR leg is OPT-IN per catalog row: `AsrModelSpec.metal_served` -(`performance/profile_common.das`) declares that the Metal driver serves that family end to -end - tower and decoder both. An unflagged family keeps the CPU by design, and asking for its -GPU leg trips the anti-sandbag: the `--ngl` arms assert that the tower engage counters moved, -so a family whose tower silently falls back reds its row instead of publishing a CPU wall -under a GPU heading. - -Three reference tools carry a GPU arm the board pairs against a das Metal row, each with its -own spelling: the whisper reference exe takes `-ngl`, the media-chat reference exe takes -`-ngl 99`, and the NeMo bench script takes `--device mps`. The remaining two reference legs -have no pair - the parakeet exe measures slower on the GPU, and the ONNX export is CPU-only - -so their das rows stand alone in the CPU category. - -The media-chat reference exe is built as the bench exe's sibling in one reference worktree: -`benchmarks/setup_lcpp_ref.das` builds both targets, because a bench-only build leaves the -image and audio-chat cells with no binary and the board quietly mints das-only rows. That -sibling needs the timing patch beside it (`benchmarks/asr/patches/`) - the record parser reads -its per-rep timing lines, and an unpatched sibling mints "no rep parsed" failures. The apply -is guarded on the patched marker already being in the tree, and runs three-way so it rides -pin drift. On Apple boxes `performance/setup_asr_rig.das` builds a second, Metal-ON copy of -the same patched checkout, because `-ngl` on a Metal-OFF build is inert; `mtmd_bin_metal()` -returns "" when it is absent and the GPU reference leg skips loudly. +**The retune re-exec fires on scaffolding runs on a box its class profile does not cover, and +the pin that suppresses it is checked in.** A bare `daslang` run that requires the engine - a +probe, a one-off script, a REPL experiment - stamps the shipped class profile at compile time +and runs; on a box the profile does not cover it re-execs into the residue race when no manifest +is armed, and on either box it serves no runtime section (no Metal crowns - no profile ships +those). `performance/last_known_good_sidecar.json` exists for exactly that: a frozen copy of a +complete, noise-gated mint, tracked in git (the `*.tune.json` ignore rule deliberately does not +match it). Point `DAS_TUNE_MANIFEST` at it and the framework never retunes; on a different box +the identity mismatch just serves fallbacks, and a copy minted before the current +`DASLLAMA_RELEASE` serves fallbacks on any box - the compile says which with one +`WARNING DAS_TUNE_MANIFEST` line per scope. That is the whole contract - it suppresses the +re-exec, it does not tune the box, and a number measured under it is not a benchmark. Benches +and the rig keep minting their own. ### 2.10 Sanctioned instrumentation rails @@ -141,10 +136,9 @@ where a rail entry would keep serving. Where a timed line IS the deliverable - ` `performance/`, `harness/`, and cold one-shot load/mint progress logs (image bake/map, load stages, tokenizer build) - the rails do not apply. A timing that is part of an API's answer - the facade's `TtsTimings`, the per-stage walls a synthesis returns to its caller and the -server logs per request - is a deliverable of the same kind, not instrumentation, and the -one-rail follow-up (`followup_general.md` row 72) keeps it that way while it retires the -duplicate rails. A clock whose value feeds logic is control flow, not instrumentation; it is -marked `// clock: control` so the sweep and any future lint leave it alone. +server logs per request - is a deliverable of the same kind, not instrumentation. A clock whose +value feeds logic is control flow, not instrumentation; it is marked `// clock: control` so the +sweep and any future lint leave it alone. The override-announce rule (REVIEW.md) draws its boundary here: a knob or setter whose purpose is timing still counts as an override when it moves computed numerics - two GEMM forms of the @@ -152,101 +146,29 @@ same math differ in float terms - while one that changes only WHEN work happens a CLI flag is never an override (it is the run's own command line, visible where the run is launched). -### Re-stamping inside the content-addressed archive - -A sidecar archived as `records/.tune..json` is content-addressed: its filename -carries the hash of its bytes. Re-stamping such a file's `provenance.engine_sha` to a reachable -commit (the remedy `performance/REVIEW.md` allows when the measured `modules/dasLLAMA/` tree is -byte-identical) therefore re-hashes and renames the file, and every `records/.json` row -whose `tune_sha` named the old file is repointed in the same change - a row left on the old -name points at a file that no longer exists. - -### 2.21 An isolated kernel race is only as good as the graph it imitates {#kernel-race-fidelity} - -A kernel A/B race times two spellings of one compute on a synthetic fixture. Three conditions -decide whether its winner is the winner the served graph would pick, and a race missing any of -them crowns confidently and wrongly: - -- **Overlap.** The served graph's consecutive GEMVs write DIFFERENT output buffers, so the - scheduler overlaps them. A race chaining every dispatch through one output buffer serializes - on the write-after-read hazard; on the iq2xxs f4-slab twin that one difference read +9% for - an arm the served graph rejects at -8.6%. -- **A warmed clock.** An Apple GPU's clock governor ramps under load, and `race_pair_ms` runs - base-then-twin per round, so the first side pays the ramp the second rides. About 150 ms of - GPU work before the first timed round removes the bias; back-to-back dispatches inside each - timed encoder hold the clock there. A sparse invocation without the burn reads idle-clock - times and can flip a verdict outright. -- **A real site shape.** `REVIEW_GPU.md` binds this one. - -A race arm owns a transient command queue for its timed pairs and releases it before returning - -the one exception to the family's shared device and queue (`ARCHITECTURE_GPU.md` sec.1.5) - so -the tune-time race never queues behind served work. - -Even a race meeting all three can be structurally blind. `kq_gemv_iq2xxs_f4` is the standing -case: every isolated regime crowns the twin and the served decode graph loses 8.6% with it, -because the effect is mixed-stream occupancy - it exists only when the GEMV runs beside the -rest of the token step. That crown is therefore NOT auto-raced. It is minted on the real shapes -by `harness/tune_kernels.das`'s serving confirm: two temporary manifests differing only in -`runtime.metal_tensor` membership, each served to -`benchmarks/lcpp_bench.das --for-debug-purposes --ngl 99 -p 32 -n 128 -r 3` under -`DAS_TUNE_MANIFEST`, the tg128 line the verdict. Both arms carry an EMPTY `kernels` section, so -they stamp identical fallbacks and the tg delta isolates the crown under test. The margin is -`CONFIRM_TG_MARGIN` = 1.005, best-of-3: the crown's serving win where it wins is +0.6% (m5) and -its serving loss where it loses is -8.6% (m4), so the margin only has to clear run noise, and a -spuriously minted crown costs a re-mint rather than a board row. No IQ2_XXS vehicle on the box -means no crown - the base kernel is the safe side - and the run says so with a provisioning -hint. - -The per-format isolated rig is `benchmarks/matmul/bench_metal_kq_race.das`: synthetic planes, -no model, no tuner, every arm gated against a CPU plane-dequant oracle before it is timed, and -`--burn-ms` (default 150) spent on GPU work before each cell's first timed round. Its cells -chain every dispatch through ONE shared output buffer on purpose - the serialized regime is -the instrument's probe shape, imitating the reference tool it is compared against - and its -numbers reach the engine only through a human porting decision, never a minted crown. - -### 2.26 The gemv takes its own tune seat {#gemv-seat} - -A kq family's manifest entry is its tile-best row, and the gemv gets a SECOND entry when a different -row serves the streamed decode better. Only same-mr rows can differ, because the layout companion -pins the plane's interleave; of those the two best by tile time race, the winner takes the gemv only -by the margin over the tile winner's own gemv, and the incumbent keeps a tie. Every family's perm grid -therefore carries a 256-wide `mr = 16` alternate beside its 512-wide tile crown. The seat is decided -at the engine's decode shape - a DRAM-bound plane streamed by every lane through the engine's own -splitter - because the engine's row length moves the answer (k3 on Granite Rapids: the 256 seat wins -at n=2048 and loses at 14336 - `benchmarks/matmul/kq_kernel_bench.das`, tune mode, seats pinned, d=32768). The seat fixture is a 512-row build at the ffn width tiled 320 times, -past the largest L3 a socket lends a slice of, and the seat takes the MEDIAN of seven rounds: a round -that finds the plane in L3 must not crown it. In normal mode `llvm_tune` stamps a companion from its -own manifest entry when one exists and is a perm this box can run, else from the tile's. - -### 2.27 The CPU kernel bench's fixture conditions {#cpu-kernel-bench-fixture} +### 2.20 The ASR board's GPU row pairs {#asr-gpu-pairs} -`benchmarks/matmul/kq_kernel_bench.das` times raw kernels on synthetic planes, and three fixture -properties decide whether its numbers mean anything. Every plane of one format lives in ONE arena at -fixed offsets, staggered so no two starts share their low 12 address bits: the heap places separate -arrays at run-dependent relative addresses, and planes that alias in the L1/L2 set logic make a run's -time depend on where the heap put them. Scale planes are filled with a byte that is a normal number in -every scale form, never random bytes, because denormal math runs orders of magnitude slower. Each row -is warmed before it is timed - three unmeasured rounds solo, six dispatches per row on the team arm - -because a core ramps over several rounds and one warm call is not enough. The q8 row exists in two -flavors: f32 group scales (the engine's own quantization) and `q8s16` over binary16 scales - the -wscale_f16 rail a GGUF q8_0 tensor runs, and the like-for-like row against the reference's q8_0. -Provenance for every figure in this section: `benchmarks/matmul/kq_kernel_bench.das` under -`DAS_TUNE_MODE=tune`, one thread, its default `--fmt` / `-n` / `-d` shape. +The das Metal ASR leg is OPT-IN per catalog row: `AsrModelSpec.metal_served` +(`performance/profile_common.das`) declares that the Metal driver serves that family end to +end - tower and decoder both. An unflagged family keeps the CPU by design, and asking for its +GPU leg reds the row: the `--ngl` arms assert that the tower engage counters moved, so a family +whose tower silently falls back fails instead of publishing a CPU wall under a GPU heading. -### 2.28 The speculative round's cell is a ruler record {#ruler-records} +Three reference tools carry a GPU arm the board pairs against a das Metal row, each with its +own spelling: the whisper reference exe takes `-ngl`, the media-chat reference exe takes +`-ngl 99`, and the NeMo bench script takes `--device mps`. The remaining two reference legs +have no pair - the parakeet exe measures slower on the GPU, and the ONNX export is CPU-only - +so their das rows stand alone in the CPU category. -**`performance/records/mtp/mtp__[_variant].json` is a ruler record: one file per box -and model, written only by `harness/mtp_ruler.das`.** The board (`records/.json`) has no -speculative column, because a speculative rate is not one engine's number: acceptance is a property -of the text and of the drafter both engines share, so the honest cell is the two engines on the -identical rendered prompt in one run. The ruler measures our released exe FIRST from a parent that -has loaded nothing (a parent that had just run the engine in-process read the exe's speculative arm -a fifth low), then the reference server at the ref pin, every arm settled, and writes `meta` (date, -box, `das_sha`, `das_exe`, `lcpp_server`, `lcpp_version`, the model and head with their shas, the -corpus, `ngen`, `reps`, `depths`) plus one row per engine, depth and prompt. The shape is the ruler's, -not the board's - `list_record_stores` and the records gate read `records/` one level deep and never -see the folder - and `mtp_ruler --render ` prints the table. A third-party wall lives here -only as the other half of a pair taken in the same run. +The media-chat reference exe is built as the bench exe's sibling in one reference worktree: +`benchmarks/setup_lcpp_ref.das` builds both targets, because a bench-only build leaves the +image and audio-chat cells with no binary and the board quietly mints das-only rows. That +sibling needs the timing patch beside it (`benchmarks/asr/patches/`) - the record parser reads +its per-rep timing lines, and an unpatched sibling mints "no rep parsed" failures. The apply +is guarded on the patched marker already being in the tree, and runs three-way, so it still +applies after the reference pin moves. On Apple boxes `performance/setup_asr_rig.das` builds a +second, Metal-ON copy of the same patched checkout, because `-ngl` on a Metal-OFF build is +inert; `mtmd_bin_metal()` returns "" when it is absent and the GPU reference leg skips loudly. ### 2.40 A `[tuned]` kernel's perm is decided at its own compile {#tuned-perm-precedence} @@ -257,11 +179,11 @@ profile (`performance/defaults`), the annotation's `fallback` `;`-chain, then `D `tune_kernel_pick` (llvm_tune) reads the sidecar and the profile in that order and hands back the FILE its answer came from, so a verbose compile names which of the two stamped each kernel. A box the shipped profile covers therefore compiles tuned kernels without racing anything, and a box it -does not cover falls to the fallback chain - never to another box's winners. -With no tune framework in the build the first three steps do not exist - no policy env, no sidecar, -no profile - so the ladder is the `perm=` pin, then the `fallback` chain's first UNCONDITIONAL entry -(a `suffix:requires` seat cannot be judged with no feature probe), then `DEFAULT_PERM`, and the -compile reports nothing. +does not cover falls to the fallback chain - never to another box's winners. With no tune +framework in the build the first three steps do not exist - no policy env, no sidecar, no +profile - so the ladder is the `perm=` pin, then the `fallback` chain's first UNCONDITIONAL +entry (a `suffix:requires` seat cannot be judged with no feature probe), then `DEFAULT_PERM`, +and the compile reports nothing. ### 2.41 The mint's own wall rides in the sidecar's provenance {#mint-wall-provenance} @@ -279,13 +201,13 @@ its kernels baked per CPU class and carries no tuner and no policy rail, so noth mint the sidecar's `"runtime"` section - the Metal twin crowns among its knobs, a 2-4x kernel-form gain of a tensor twin over its simdgroup kernel per twin-race row on the M5 Max (`harness/tune_kernels.das`, the metal_crowns family) - and a shipped Mac exe would run -uncrowned forever. The section needs no rebuild, so the exe -mints it itself: `dasllama_fat_start` registers `dasllama_fat_first_start` with the box-profile -apply (`set_runtime_race_hook`) from its `[init]`, and the engine umbrella -(`dasllama_transformer`) requires the module so every engine program carries the registration - -the shipped bench requires the umbrella, never the facade; `apply_box_profile_runtime_checked` fires the hook when the -sidecar is absent, another box's, or carries no runtime section, then reads the file the hook -wrote. The hook answers false outside a fat exe (`tune_fat_built()`); inside one it runs +uncrowned forever. The section needs no rebuild, so the exe mints it itself: +`dasllama_fat_start` registers `dasllama_fat_first_start` with the box-profile apply +(`set_runtime_race_hook`) from its `[init]`, and the engine umbrella (`dasllama_transformer`) +requires the module so every engine program carries the registration - the shipped bench +requires the umbrella, never the facade; `apply_box_profile_runtime_checked` fires the hook when +the sidecar is absent, another box's, or carries no runtime section, then reads the file the +hook wrote. The hook answers false outside a fat exe (`tune_fat_built()`); inside one it runs `dasllama_race_runtime_section`: the Metal twin races (`dasllama_metal_crown_race` - both halves, synthetic, no model) under the tune progress display, then `dasllama_runtime_snapshot` - the same writer the mint's kernel half ends with - merged into the app sidecar beside the exe with the @@ -296,5 +218,21 @@ keeps the crowns for the process and says so; `DAS_TUNE_MANIFEST` moves the file What a first start never does: load a model, spawn a child, or race a kernel. The tuner's confirms - the generator half's end-to-end prefill A/B, the kernel half's serving and MTP depth confirms - each spawn a daslang child on a harness script and a vehicle model, and they are the -harness's alone; under `harness/dasllama_tuner.das` on the M5 Max they were 147 of the +harness's alone; under `harness/dasllama_tuner.das` on the M5 Max the confirms take 147 s of the metal_crowns family's 161 s, the twin race itself 14 s. + +### 2.45 The speculative round's cell is a ruler record {#ruler-records} + +**`performance/records/mtp/mtp__[_variant].json` is a ruler record: one file per +box and model, written only by `harness/mtp_ruler.das`.** The board (`records/.json`) has +no speculative column, because a speculative rate is not one engine's number: acceptance is a +property of the text and of the drafter both engines share, so the honest cell is the two +engines on the identical rendered prompt in one run. The ruler measures our released exe FIRST +from a parent that has loaded nothing (a parent that had just run the engine in-process read +the exe's speculative arm a fifth low), then the reference server at the ref pin, every arm +settled, and writes `meta` (date, box, `das_sha`, `das_exe`, `lcpp_server`, `lcpp_version`, the +model and head with their shas, the corpus, `ngen`, `reps`, `depths`) plus one row per engine, +depth and prompt. The shape is the ruler's, not the board's - `list_record_stores` and the +records gate read `records/` one level deep and never see the folder - and `mtp_ruler --render +` prints the table. Every third-party wall in the file is the other half of a pair +taken in that run. diff --git a/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md new file mode 100644 index 0000000000..d254b50558 --- /dev/null +++ b/modules/dasLLAMA/ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md @@ -0,0 +1,89 @@ +# dasLLAMA Architecture - the kernel race and bench instruments + +Companion to `ARCHITECTURE_MEASUREMENT.md`; section numbers are `ARCHITECTURE.md`'s. This +document carries sections 2.21, 2.26 and 2.27, the instruments that time a kernel away from the +served graph and the conditions their verdict holds under: what makes an isolated race predict +the graph it imitates, how the gemv earns a second tune seat, and the fixture the CPU kernel +bench times on. The rig that produces recorded numbers, the tune gate, the Vulkan GEMM probe's +axes and the instrumentation rails stay in `ARCHITECTURE_MEASUREMENT.md`. + +### 2.21 An isolated kernel race is only as good as the graph it imitates {#kernel-race-fidelity} + +A kernel A/B race times two spellings of one compute on a synthetic fixture. Its winner is +minted as a crown - the family's name in the box sidecar's `runtime.metal_tensor` set, which +makes every later run on that box serve the twin in place of the base kernel +(`metal_tensor_crowned`, `dasllama/dasllama_common.das`). Three conditions decide whether its +winner is the winner the served graph would pick, and a race missing any of them mints a wrong +crown: + +- **Overlap.** The served graph's consecutive GEMVs write DIFFERENT output buffers, so the + scheduler overlaps them. A race chaining every dispatch through one output buffer serializes + on the write-after-read hazard; on the iq2xxs f4-slab twin that one difference reads +9% for + an arm the served graph rejects at -8.6%. +- **A warmed clock.** An Apple GPU's clock governor ramps under load, and `race_pair_ms` runs + base-then-twin per round, so the first side pays the ramp the second rides. About 150 ms of + GPU work before the first timed round removes the bias; back-to-back dispatches inside each + timed encoder hold the clock there. A sparse invocation without the burn reads idle-clock + times and can flip a verdict outright. +- **A real site shape.** The race sizes its operands at a real model shape. A slab small enough + to sit in cache ranks the kernels by an effect production never sees, and the race then picks + the slower kernel. + +A race arm owns a transient command queue for its timed pairs and releases it before returning - +the one exception to the family's shared device and queue (`ARCHITECTURE_GPU.md` sec.1.5) - so +the tune-time race never queues behind served work. + +Even a race meeting all three can be structurally blind. `kq_gemv_iq2xxs_f4` is the standing +case: every isolated regime crowns the twin and the served decode graph loses 8.6% with it, +because the effect is mixed-stream occupancy - it exists only when the GEMV runs beside the +rest of the token step. That crown is therefore NOT auto-raced. It is minted on the real shapes +by `harness/tune_kernels.das`'s serving confirm: two temporary manifests differing only in +`runtime.metal_tensor` membership, each served to +`benchmarks/lcpp_bench.das --for-debug-purposes --ngl 99 -p 32 -n 128 -r 3` under +`DAS_TUNE_MANIFEST`, the tg128 line the verdict. Both arms carry an EMPTY `kernels` section, so +they stamp identical fallbacks and the tg delta isolates the crown under test. The margin is +`CONFIRM_TG_MARGIN` = 1.005 over the mean of the run's three timed reps - the `tg128:` line the +bench prints: the crown's serving win where it wins is +0.6% (m5) and its serving loss where it +loses is -8.6% (m4), so the margin only has to clear run noise, and a spuriously minted crown +costs a re-mint rather than a board row. No IQ2_XXS vehicle on the box +means no crown - the base kernel is the safe side - and the run says so with a provisioning +hint. + +The per-format isolated rig is `benchmarks/matmul/bench_metal_kq_race.das`: synthetic planes, +no model, no tuner, every arm gated against a CPU plane-dequant oracle before it is timed, and +`--burn-ms` (default 150) spent on GPU work before each cell's first timed round. Its cells +chain every dispatch through ONE shared output buffer on purpose - the serialized regime is +the instrument's probe shape, imitating the reference tool it is compared against - and its +numbers reach the engine only through a human porting decision, never a minted crown. + +### 2.26 The gemv takes its own tune seat {#gemv-seat} + +A kq family's manifest entry is its tile-best row, and the gemv gets a SECOND entry when a +different row serves the streamed decode better. Only same-mr rows can differ, because the +layout companion pins the plane's interleave; of those the two best by tile time race, the +winner takes the gemv only by the margin over the tile winner's own gemv, and the incumbent +keeps a tie. Every family's perm grid therefore carries a 256-wide `mr = 16` alternate beside +its 512-wide tile crown. The seat is decided at the engine's decode shape - a DRAM-bound plane +streamed by every lane through the engine's own splitter - because the engine's row length moves +the answer (k3 on Granite Rapids: the 256 seat wins at n=2048 and loses at 14336 - +`benchmarks/matmul/kq_kernel_bench.das`, tune mode, seats pinned, d=32768). The seat fixture is +a 512-row build at the ffn width tiled 320 times, past the largest L3 a socket lends a slice of, +and the seat takes the MEDIAN of seven rounds, which discards a round that finds the plane in +L3. In normal mode `llvm_tune` stamps a companion from its own manifest entry when one exists +and is a perm this box can run, else from the tile's. + +### 2.27 The CPU kernel bench's fixture conditions {#cpu-kernel-bench-fixture} + +`benchmarks/matmul/kq_kernel_bench.das` times raw kernels on synthetic planes, and three +fixture properties decide whether its numbers mean anything. Every plane of one format lives in +ONE arena at fixed offsets, staggered so no two starts share their low 12 address bits: the heap +places separate arrays at run-dependent relative addresses, and planes that alias in the L1/L2 +set logic make a run's time depend on where the heap put them. Scale planes are filled with a +byte that is a normal number in every scale form, never random bytes, because denormal math runs +orders of magnitude slower. Each row is warmed before it is timed - three unmeasured rounds +solo, six dispatches per row on the team arm - because a core ramps over several rounds and one +warm call is not enough. The q8 row exists in two flavors: f32 group scales (the engine's own +quantization) and `q8s16` over binary16 scales - the wscale_f16 rail a GGUF q8_0 tensor runs, +and the like-for-like row against the reference's q8_0. +Provenance for every figure in this section: `benchmarks/matmul/kq_kernel_bench.das` under +`DAS_TUNE_MODE=tune`, one thread, its default `--fmt` / `-n` / `-d` shape. diff --git a/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md b/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md index 9b7516e686..526a7c9d31 100644 --- a/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md +++ b/modules/dasLLAMA/ARCHITECTURE_RUNTIME.md @@ -7,31 +7,31 @@ Companion to `ARCHITECTURE.md`; section numbers are that document's. The test is one question: *for a given compiled kernel, can this value change between dispatches?* - **Yes -> DATA.** Context depth, row counts, buffer offsets, `kv_dim`, scales, head counts. It - belongs in a uniform or a kargs struct. + belongs in a uniform or a kargs struct - the per-dispatch argument struct a kernel binds. - **No -> SHAPE.** A codec's block stride, a scale-plane stride, a lane width, an unroll factor, a - format selector. It must NOT reach the kernel as a uniform, a kargs field, or a helper parameter. + format selector. It does not reach the kernel as a uniform, a kargs field, or a helper parameter. Shape belongs to the specialization: a separate kernel class and PSO, a per-codec overload, a -monomorphized generic, or a `static_if` on a compile-time witness. Handing a shape constant over as -a value and trusting the shader compiler to fold it back is an assumption, not a guarantee, and it -is worth nothing in the kernels that matter. +generic stamped out once per shape, or a `static_if` on a compile-time witness. Handing a shape +constant over as a value and trusting the shader compiler to fold it back is an assumption, not a +guarantee. -**Verify against the EMITTED shader, never the das source.** Read the `*_msl` global or the SPIR-V -dump and confirm the constant is literal there: `blk * 34u`, not `blk * bstr`. A helper that looks -specialized in das can still lower to a runtime multiply. +**The EMITTED shader is where a shape constant is visible, not the das source.** In the `*_msl` +global or the SPIR-V dump the constant is literal - `blk * 34u`, not `blk * bstr`. A helper that +looks specialized in das can still lower to a runtime multiply. **kargs structure.** Twins of a family bind the SAME kargs type at the SAME binding, even where one -twin ignores a field. A twin that carries an extra scalar must not shift the others to different -slots, because that asymmetry propagates into the encoder as a per-form branch. Two tells that a -fold is overdue: +twin ignores a field. A twin that carries an extra scalar keeps the others in their slots; shifting +them would reach the encoder as a per-form branch. Two signs a kargs field carries a number +the dispatch already has: 1. *A value reaches the encoder twice* - a pooled scalar uniform BUFFER passed alongside the identical value as a parameter (`bd` next to `d`). The buffer is uploaded and released per step to carry a number the encoder already holds. 2. *A field is a function of the fields beside it.* An expert plane's block stride is - `kdim * ndim / blocksize`; a reciprocal scale is `1/sqrt(dim)`. Derive it in the builder - each + `kdim * ndim / blocksize`; a reciprocal scale is `1/sqrt(dim)`. The builder derives it - each one passed separately is a second place to get it wrong. Likewise, when a kernel's grid IS the - geometry it reads, take the grid off the kargs rather than re-passing the numbers. + geometry it reads, the grid carries the numbers and the kargs do not repeat them. **Nothing dispatches a kernel except its `enc_*` builder.** A hand-rolled bind list elsewhere - a tune-race harness, a benchmark, a probe - duplicates the builder and desyncs the moment the @@ -41,11 +41,11 @@ on some paths and not others; a duplicate that binds NO kargs is invisible to it ### 2.3 GPU-resident cache identity -An address-keyed entry carries its SPAN, and a hit must cover the request - a shorter first upload -must never serve a wider later one. Different upload FORMS (plain span vs concat) live in separate -tables so they can never alias; the metal `RegionEntry` rail is the model. Buffers grown out of an -entry retire to a list released only at quiesce boundaries, because unretained command buffers may -still bind them. +An address-keyed entry carries its SPAN, and a hit covers the request: a shorter first upload never +serves a wider later one. Different upload FORMS (plain span vs concat) live in separate tables so +they can never alias; the Metal `RegionEntry` rail is the model. Buffers grown out of an entry +retire to a list released only at a quiesce boundary - a point where no submitted command buffer is +still in flight - because unretained command buffers may still bind them. ### 2.3a Making weights live bumps the weights epoch {#weights-epoch-on-load} @@ -62,22 +62,24 @@ reason: flat one-call-per-item runs (a registration or release list with one lin GPU kernel bodies whose phases are coupled by barriers, cooperative-matrix ops or register residency and so cannot cross a function boundary without changing the shader. -Split only where a real seam exists - genuine duplication, a distinct phase, a self-contained arm - -and only when the extracted helper stands on its own. Two corollaries this module keeps tripping -over: **a kargs fold that grows an already-over-cap kernel body is not a reason to abandon the -fold** - unpacking N fields adds N lines; take the growth and ledger the real seam. +A split lands only where a real seam exists - genuine duplication, a distinct phase, a +self-contained arm - and only when the extracted helper stands on its own. One corollary: +**merging kargs fields into a kernel body that is already over the cap is not a reason to abandon +the merge** - unpacking N fields adds N lines, so the growth is taken and the real seam goes to +the module's follow-up ledger. ### 2.6 Capability questions and readiness questions are different questions -A predicate that mixes them cannot be reused. `prefill_decline` answers "can metal serve this -model" (capability) *and* "is this window staged" (readiness - are the rope tables built). A caller -that runs before the window is staged must ask the capability half only, or it gets "not yet" -forever and its feature silently never runs. Split such predicates rather than reordering the -caller; an optimistic capability answer is safe when the late path has a fallback, and here it does. +A predicate that mixes them cannot be reused. `prefill_decline_caps` answers "can metal serve this +model" (capability); `prefill_decline` adds "is this window staged" (readiness - are the rope +tables built) on top of it. A caller that runs before the window is staged asks the capability +half: `metal_ple_pre_gpu_gate` calls `prefill_decline_caps` directly, because asking both would +get it "not yet" forever and its feature would silently never run. An optimistic capability answer +is safe here, because the late path has a fallback. ### 2.7 A quantized activation carries its scale lattice (Vulkan) {#activation-scale-lattice} -Two activation quant forms ride the vulkan rail, and they differ in the SCALE LATTICE, not the +Two activation quant forms ride the Vulkan rail, and they differ in the SCALE LATTICE, not the int8 payload: the Q8_0 form scales per 32 values, the superblock form per 256 (with per-32 sub-scales inside). A compiled kernel indexes ONE lattice - the q8 GEMV/GEMM rail reads per-32 scales; the k-quant (k4/k5/k6/q40) kernels index the per-256 lattice. `kq_sb(fmt)` is the @@ -94,35 +96,41 @@ Three consequences the code is shaped around: change is witnessed only by parity runs over both a q8 and a k-quant model. - **The fused add-rms+requant twin exists only for the per-32 form.** The rail gates it on "every consumer of this buffer is Q8_0-scaled" (`rd_x_quants_b32`), and the profiler stamp - shape must ride the SAME gate, or profiles desync from what actually dispatched. -- **A GEMV group sharing one activation buffer must be lattice-homogeneous.** q/k/v share one - quantized x; gate/up share another. Resident arming classifies each member's consumer form - and DECLINES a mixed group rather than serving one member wrong scales. The prefill f16 feed - answers the same question one step further: one activation buffer serves every GEMM of a - group, so the f16 (cm2 decode-in-load) form engages only when EVERY member of the group is - cm2-servable - one member on the quant route pins its whole group to the quant feed. + shape rides the SAME gate; on a different gate the profiles desync from what actually + dispatched. +- **The activation group - the dispatches that read one activation buffer, never a layer - is the + decision unit: its members share one lattice, and its feed is picked once for all of them.** + q/k/v share one quantized x; gate/up share another. Resident arming classifies each member's + consumer form and DECLINES a mixed group rather than serving one member wrong scales. The + prefill f16 feed answers the same question one step further: one buffer serves every GEMM of a + group, so the f16 form engages only when EVERY member is tile-servable (the cm2 decode-in-load + tile, or the KHR kq tile) - one member on the quant route pins its whole group to the quant + feed. The recurrent head (a deltanet block) holds two groups, not one: its qkv and z GEMMs read + the block's input rows, and its out GEMM reads the scan's output rows - the raw rows the + delta-rule scan writes. So the two feeds are decided apart, and a Q8_0 out plane decides the out + feed alone: it leaves the qkv and z planes on the tiles. ### 2.8 Every program root declares its stack budget and its prefill intent `options stack` is main-module-only: it does not unify up from required modules, so no library in the forward chain can declare the depth it needs. Every program that drives the engine - each test, -harness, benchmark, and tool - must therefore declare it, and dasLLAMA's frames are deep enough +harness, benchmark, and tool - declares it instead, and dasLLAMA's frames are deep enough (by-value `Session`s, the forward/prefill chain, the generated kernel tier) that the default is never enough. -The budget is **one number in every root**, currently 524288. Per-root numbers do not survive: a -frame that grows past the smallest declared budget breaks only the program that declared least, so -the limit is discovered by crashing - and the program that crashes is whichever one is run -rarest. A measurement rig sized below a test suite is the worst case of this, because the suite -stays green while the rig dies. The cost of the uniform number is reserved address space per -context; the cost of per-root numbers is a runtime crash found by the least-covered program. +The budget is **one number in every root**, 524288. Per-root numbers do not survive: a frame that +grows past the smallest declared budget breaks only the program that declared least, so the limit +is discovered by crashing - and the program that crashes is whichever one is run rarest. A +measurement rig sized below a test suite is the worst case of this, because the suite stays green +while the rig dies. The cost of the uniform number is reserved address space per context; the cost +of per-root numbers is a runtime crash found by the least-covered program. The second declaration is prefill intent. A model-loading root declares `allow_cpu_prefill()` on the arms that hit the CPU-prefill guard - `set_metal_mode` with a runtime value declares nothing, since `MetalMode.off` leaves the guard armed - and logs, once, which configuration it -ended on. The guard panics, and a panic takes every live stream down, so an undeclared root is -a serving outage waiting on its first long prompt. Both halves of root discipline are enforced -by `tests/test_program_roots.das`. +ended on. The guard panics, and a panic takes every live stream down: an undeclared root serves +normally until its first long prompt. Both halves of root discipline are enforced by +`tests/test_program_roots.das`. ### 2.9 Environment knobs {#env-knobs} @@ -135,23 +143,19 @@ the documentation and to the registry test. The sanctioned forms beyond a plain - **Dynamic names** - a variable named by data, not by code - go through `env_is_set` / `env_value_of`; there is no field to declare because the name is not known at compile time. - **The config loads once at context init**, so `set_env_variable` mid-process is invisible to - the running config: arm a child process's environment instead. + the running config; the environment a child process inherits is the way to change it. - **A write of a foreign library's knob** (`set_env_variable` with a literal name) is allowed - only before that library first reads it, and the name must be a declared `[EnvConfig]` knob - - the registry test scans writes too, so a re-spelled name fails it. + only before that library first reads it, and its name is a declared `[EnvConfig]` knob - the + registry test scans writes too, so a re-spelled name fails it. `tests/test_env_registry.das` enforces the lot in both directions (declared <-> documented, read <-> registered, writes included). -**Override announces.** A knob that is a gate escape, a policy override, or a threshold -recalibration - one whose presence makes a run measure, mint, or emit something its defaults -would not (`DASLLAMA_ALLOW_UNTUNED`, the `DAS_TUNE_NOISE_*` pair and `DAS_TUNE_POLICY` read -through llvm_tune's accessors) - announces at the point it changes the outcome: at least one -printed or logged line naming the knob by its env spelling. Set-but-inert stays silent; -per-site repeats are correct (the same knob may legitimately announce at its `[init]` wire, at -a gate it bypasses, and in a tool banner). Ordinary behavior knobs (thread counts, rail -selection, formats) are not overrides under this contract - their state belongs in the tools' -existing config/status lines, not in per-knob announces. +**Override announces.** `DASLLAMA_ALLOW_UNTUNED`, and the `DAS_TUNE_NOISE_*` pair and +`DAS_TUNE_POLICY` that llvm_tune's accessors read, each announce at every site where they +change the outcome - the `[init]` wire, the gate they bypass, a tool banner - and stay silent +when set but inert. Thread counts, rail selection and format knobs report their state in the +tools' existing config and status lines instead. ### 2.11 The [hot_path] coverage model @@ -165,81 +169,79 @@ The tokenizer encode/decode path is sanctioned UNCOVERED by the region contracts gate is the `--tok` scaling rows, whose instrument (the size-ladder ratio) catches what the contracts cannot. -### 2.12 The post-CPU-burn GPU ramp - the residency heartbeat holds it; do not build a warm-up - -Figures in this section: M1 Max, 2026-08-23 - probes are quiet `-jit` runs with the rig's -tune manifest; cells are the released `lcpp_bench` exe (`--image --image-think -r 3 -t 8 ---ngl 99`); full tables in `history/dasLLAMA/qwen3vl_plan.md` slices M/J. - -After a CPU-only phase, the first Metal submission runs degraded - one time per window, -entirely in the kernel-side driver window (kernelStart->kernelEnd; queue hand-off and GPU -execution stay flat - split probe-verified 2026-08-23, `harness/residency_ramp_probe.das` + -`das_metal_boost`'s `metal_submit_trace`). The long-window mechanism is the OS collecting a -committed+requested `MTLResidencySet` during inactivity; the fix is the -residency HEARTBEAT - a dasMetal background thread re-requesting -residency every 5 ms for `DASLLAMA_METAL_HEARTBEAT_S` (default 180 s, 0 = A/B rail) after -the last served step, kicked from `residency_flush`. Measured on the qwen3v tower encode: -3000 ms burn drv 17.7 -> 3.0 ms. The `MTLResidencySet` pin itself -(`DASLLAMA_METAL_RESIDENCY`, on by default) holds the short-window case (-15 ms/encode on -the tower's first submission after its CPU stem); the heartbeat holds the long ones. -REFUTED by measurement (`PERF_LEDGER.md`, the heartbeat entry): every pre-payment - empty -command buffers and driver round-trips (0.107 ms - never asleep), single-dispatch kernels, -per-page touch kernels, light pulse-trains through the burn - a warm-up always pays its own -cost ON TOP of the slack it was meant to hide; do not re-attempt one. Also refuted: a -whole-map no-copy anchor buffer (page pre-wiring moves nothing) and the once-proposed -per-buffer tracking audit (resource count is not the lever) - prefill pools simply stay -tracked (untracked reds parity). The ~1-3 ms window-scaled residual is the driver/GPU -idle-state wake class, no user-space lever found. Related, same ledger: merely arming Metal -makes a CPU q8 tower encode ~1.7x slower - mechanism unnamed, deleted by a GPU-served tower. +### 2.12 The post-CPU-burn GPU ramp and the residency heartbeat +After a CPU-only phase, the first Metal submission runs degraded. The cost is paid once per idle +phase, and all of it sits in the kernel-side driver span (kernelStart->kernelEnd); the queue +hand-off and the GPU execution stay flat. A longer idle phase costs more, because the OS collects +a committed+requested `MTLResidencySet` during inactivity. + +Two mechanisms cut the ramp. The `MTLResidencySet` pin (`DASLLAMA_METAL_RESIDENCY`, on by +default) covers a short idle phase: it saves 15 ms on the tower's first submission after a +CPU-only stretch. The residency HEARTBEAT covers the long ones - a dasMetal background thread +re-requests residency every 5 ms for `DASLLAMA_METAL_HEARTBEAT_S` (default 180 s; 0 turns it off, +the A/B comparison setting) after the last served step, kicked from `residency_flush`. On the +qwen3v tower encode, a 3000 ms burn reads 17.7 ms of driver time without the heartbeat and 3.0 ms +with it. + +A warm-up does not help: work that pre-pays the cost pays it in full, on top of the delay it was +meant to remove. Prefill pools therefore stay tracked. The residual - 1 to 3 ms, and larger the +longer the idle phase - is the driver and GPU waking from an idle power state, and no user-space +call reaches it. Arming Metal at all makes a CPU q8 tower encode ~1.7x slower; the cause is +unknown, and a GPU-served tower removes the cost. `PERF_LEDGER.md`'s heartbeat entry carries the +measurements behind this section and each refuted attempt. ### 2.18 The CPU worker pool on a hybrid box {#hybrid-pool-policy} SMT siblings share the FMA and load ports, so the default pool is physical cores - 1 workers. A box with two core tiers splits on the SECOND tier's KIND: a compute tier (an M5's Super plus Performance cores) extends the pool to every core with GEMV capped to the fast tier, while an efficiency tier -(M1, M4) straggles batch barriers and gets no worker at all. A compute-grade second tier only +(M1, M4) makes batch barriers wait and gets no worker at all. A compute-grade second tier only exists beside a KNOWN fast tier - the `[init]` extension reads both, so a box reporting one without the other has an inconsistent topology, not a third policy. -Chunk-starved slow-tier workers spinning beside the fast decode lanes cost ~15% of tg on an M5 Max -(the decode-only rep probe of `benchmarks/lcpp_bench.das`), and the rank gate does not recover it - -only parking does. The pool is therefore phase-shaped: a decode step parks the slow tier -(`dispatch_phase_decode`), and batch-shaped work of 32 rows or more wakes every worker - a prefill -through `dispatch_phase_batch`, a CPU tower/ASR encode through `dispatch_phase_encode(rows)` at its -entry - while smaller work keeps the parked pool, since it cannot fill the slow tier past the -dispatch grain. +Chunk-starved slow-tier workers spinning beside the fast decode lanes cost ~15% of tg (tokens +generated per second) on an M5 Max (the decode-only rep probe of `benchmarks/lcpp_bench.das`), +and the team rank gate - which admits only as many workers to a team op as its widest stage +published chunks - does not recover it; only parking does. The pool is therefore phase-shaped: a +decode step parks the slow tier (`dispatch_phase_decode`), and batch-shaped work of 32 rows or +more wakes every worker - a prefill through `dispatch_phase_batch`, a CPU tower/ASR encode +through `dispatch_phase_encode(rows)` at its entry - while smaller work keeps the parked pool, +since it cannot fill the slow tier past the dispatch grain. Precedence, strongest first: `DAS_JOBQUE_THREADS` and an app's own cap, then the box profile's `jobque_pool` / `phase_decode_workers` / `dispatch_worker_limit` entries, then this tier-kind policy. A profile declares the shape of the NEXT queue and is inert for one that already exists when -the profile loads. A minted `gemv_lane_cap` of 0 records the old fast-tier-only "uncapped" rather -than a measured choice, so a hybrid box ignores it and keeps the fast-tier cap; a non-zero mint is -an opinion and applies. +the profile loads. A `gemv_lane_cap` of 0 in a minted profile - one the tuner wrote from a +measured race - records the fast-tier-only "uncapped" default rather than a measured choice, so a +hybrid box ignores it and keeps the fast-tier cap; a non-zero value is a measured choice and +applies. ### 2.19 The CPU MoE region list caps a region at 32 rows {#moe-region-split} The grouped MoE prefill hands its expert regions to the batch dispatcher as (weight offset, first row, count) triples. That dispatcher chunks units by COUNT, not by work, so one heavy expert in a -skewed routing draw becomes one unit that straggles the barrier - measured 620 against 3929 GFLOP/s -on a zipf k4 draw (`harness/moe_kq_probe.das`). The CPU arms therefore split a region into sub-regions of at most 32 rows: -sub-regions of one expert share its weight offset, and the per-expert bias lists repeat once per -sub-region. The GPU arms keep whole regions - their kernels chunk by work already. +skewed routing draw becomes one unit the barrier waits on - measured 620 against 3929 GFLOP/s on a +Zipf k4 draw (`harness/moe_kq_probe.das`). The CPU arms therefore split a region into sub-regions +of at most 32 rows: sub-regions of one expert share its weight offset, and the per-expert bias +lists repeat once per sub-region. The GPU arms keep whole regions - their kernels chunk by work +already. -### 2.20 The engine dispatches only on a configured job queue {#jobque-policy} +### 2.44 The engine dispatches only on a configured job queue {#jobque-policy} -A queue as `create_job_que` makes it clones and destroys a fork context per job, wakes one worker +A queue from a bare `create_job_que` clones and destroys a fork context per job, wakes one worker per push, and parks a worker the moment its job ends. The engine's fork/join dispatch issues ~160 -small dispatches per decoded token, so on that queue an E2B q8 forward step takes 3.6 s against -16 ms configured - the whole gap, not a fraction of it, and each of the four knobs in -`setup_dasllama_jobque()` removes one factor (team mode alone leaves 2.0 s; the fork pool, -batched dispatch or the spin window alone each leave ~25 ms). The pool of fork contexts is the -observable: `set_jobque_fork_pool` is per context and outlives the queue, and the setup call -always turns it on. A caller that sets the pool directly, as the dispatch probes and benches do, -has configured its queue by the same token. - -The first counted dispatch of a process therefore reads it, once: a queue without the pool panics -naming the setup call, and `DASLLAMA_ALLOW_BARE_JOBQUE=1` downgrades that to a warning for a run +small dispatches per decoded token, so on that queue a q8 forward step of gemma-4 E2B takes 3.6 s +against 16 ms configured - the whole gap, not a fraction of it. Each of the four knobs in +`setup_dasllama_jobque()` removes one factor: with team mode turned back off the step still takes +2.0 s, and with the fork pool, batched dispatch or the spin window turned back off it takes ~25 ms +each. The pool of fork contexts is the observable: `set_jobque_fork_pool` is per context and +outlives the queue, and the setup call always turns it on. A caller that sets the pool directly, +as the dispatch probes and benches do, has configured its queue too. + +The first counted dispatch of a process reads that pool, once. A queue without it panics, naming +the setup call, and `DASLLAMA_ALLOW_BARE_JOBQUE=1` downgrades the panic to a warning for a run that means to measure the bare regime. A process with no queue at all runs its arms inline and is not checked. The engine's own scoped queues - the ones `load_gguf` and `load_gguf_streaming` spin -for a caller that has none - configure themselves, so the rule reaches only queues a caller opened. +for a caller that has none - configure themselves, so the check reaches only queues a caller +opened. diff --git a/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md b/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md index 391af28fd6..6f05131f16 100644 --- a/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md +++ b/modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md @@ -220,15 +220,21 @@ device-form CPU oracle an l/m/s cell in `tests/test_vulkan_kernels.das`. Payoff on the 1B: iq4xs pp512 5161 -> 15334, k3 5174 -> 14031 (0.90x / 0.80x llama.cpp's Vulkan, from 0.30x). -The KHR instantiation rides the same decode (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l, the mm-mode -paragraph): `KhrBatch : Cm2T` with `override KHR = true`, `override BN = 128u`, the -four cm2 typedefs the uncalled tensor body still names (`BT`, `ACC`, `ACCW`, `FLO` - copy k4's), -a `[vk_dispatch(name = "kq_batch__khr_cls", ...)]`, an arm in each of `khr_cls_ensure/set/enc` +The KHR instantiation adds one method (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2ae, its first +paragraph): on the format template a `def override khr_stage16(blk, e0, sbase : uint) : void` +under `static_if (KHR)` that writes the weight row's 16 values `e0 .. e0 + 16` of block `blk` +into `khr_ao[sbase .. sbase + 8)` as f16 pairs, reading the plane as words - `wq4[...]` (`uint4`, +one or two loads where the run's bytes are 16-byte aligned: the nibble and byte formats) or +`wqw[...]` (`uint`, the byte-granular grid formats) - and the scale words once; the base declares +it `abstract`, so a stamp without one fails to compile. The k4 override is the pattern; the +decode methods' index math is the same, only read sixteen at a time. Then `KhrBatch : +Cm2T` with `override KHR = true`, `override BN = 128u`, the four cm2 typedefs the uncalled +tensor body still names (`BT`, `ACC`, `ACCW`, `FLO` - copy k4's), a +`[vk_dispatch(name = "kq_batch__khr_cls", ...)]`, an arm in each of `khr_cls_ensure/set/enc` (`dasllama_vulkan_prefill.das`), and the format's kernel cell runs its fourth arm (`ml == 3`, tile 128) wherever the device has KHR coopmat at subgroup 32 - on the 5060 Ti the same run covers the -cm2 l/m/s tiles and the KHR tile. No new decode, no new oracle: the KHR arm calls the format's -`decode`/`decode_v4` directly on the plane element (`decode_v4(wq[i], ...)` - never on a `let` -copy of it, which runs at a third of the rate) and the `f16_gemm_oracle` already holds it. +cm2 l/m/s tiles and the KHR tile. No new oracle: the `f16_gemm_oracle` already holds the KHR +arm, whose f16 accumulation sits inside the cell's 2e-2 relative bar. ## 7. Metal - `dasllama_metal_kernels.das`, `_common`, `_prefill`, `_shapes`, `dasllama_layout.das` diff --git a/modules/dasLLAMA/README.md b/modules/dasLLAMA/README.md index 6d079d5944..2551b32404 100644 --- a/modules/dasLLAMA/README.md +++ b/modules/dasLLAMA/README.md @@ -102,7 +102,7 @@ modules/dasLLAMA/ ARCHITECTURE_GPU.md # companion: the GPU backend role table, the backend asymmetries, the refuted kernel shapes ARCHITECTURE_GPU_PREFILL.md # companion: the Metal prefill driver's GEMM ladder ARCHITECTURE_GPU_VULKAN.md # companion: the Vulkan resident driver - window chain, Q8 requant store, GEMV grid codebooks - ARCHITECTURE_GPU_VULKAN_GEMM.md # companion: the Vulkan GEMM tiles - cm2 decode lanes, tile pick, class pipelines, expert chain + ARCHITECTURE_GPU_VULKAN_GEMM.md # companion: the Vulkan GEMM tiles - cm2 decode lanes, tile pick, class pipelines, expert chain, the KHR kq tile ARCHITECTURE_GPU_VULKAN_RESIDENCY.md # companion: the Vulkan residency plan and the GPU-slot marks swap ARCHITECTURE_MEDIA.md # companion: the encoder-tower, audio, ASR and vision charters ARCHITECTURE_IMAGE.md # companion: the prepared-image (.dlim) rail diff --git a/modules/dasLLAMA/REVIEW.md b/modules/dasLLAMA/REVIEW.md index be4b71483f..a8898934b4 100644 --- a/modules/dasLLAMA/REVIEW.md +++ b/modules/dasLLAMA/REVIEW.md @@ -9,11 +9,11 @@ companions belong to the routed checklists). Planned work: `followup_general.md` **A dasLLAMA `[test]` file, wherever the diff puts it, answers to this module's `tests/REVIEW.md`.** -**A timing rig - a file that times a run and reports a wall-clock time or rate as its result, -printed or returned to a caller that prints it - a kernel race - a run that times two kernel -variants (arms) against each other in one process - or a function a `benchmarks/lcpp_bench.das` -cell's timed body calls, wherever it lives, answers to this folder's `benchmarks/REVIEW.md` in -addition to its own folder's checklist.** +**A timing rig (a file that times a run and reports a wall-clock time or rate as its result, +printed or returned to a caller that prints it), a kernel race (a run that times two kernel +variants - arms - against each other in one process), or a function a +`benchmarks/lcpp_bench.das` cell's timed body calls, wherever it lives, answers to this +folder's `benchmarks/REVIEW.md` in addition to its own folder's checklist.** **A diff that writes a measured number down - into `PERF_LEDGER.md`, a checked-in doc, a code comment, checked-in data a run produced, or a PR body - or adds a serving path or moves @@ -73,17 +73,16 @@ schedules such a stream, applies `REVIEW_VISION.md`.** function does not thereby pick up the other modality's checklist. **A change to `dasllama/dasllama_tts.das`, `dasllama/dasllama_tts_types.das`, -`dasllama/dasllama_tts_blocks.das`, `dasllama/dasllama_styletts2.das`, a TTS family file - -one `dasllama/dasllama_.das` holding a single speech-synthesis family - a text -front-end file - one stage of the pass that turns text into phonemes (`dasllama/dasllama_textnorm.das`, +`dasllama/dasllama_tts_blocks.das`, `dasllama/dasllama_styletts2.das`, a TTS family file - one +`dasllama/dasllama_.das` holding a single speech-synthesis family - a text front-end +file - one stage of the pass that turns text into phonemes (`dasllama/dasllama_textnorm.das`, `dasllama/dasllama_postag.das`, `dasllama/dasllama_g2p.das`) - the front-end packs' mint (`harness/build_g2p_data.py`, `harness/train_postag.py`, `harness/mint_postag_silver.py`, `performance/build_tts_data.das`), or a call that pins the TTS weight lane (`set_tts_q8` / `set_styletts2_q8`), wherever the diff puts it, applies `REVIEW_TTS.md`.** -**A diff that adds a file under `dasllama/`, moves code between files, or lands a kernel, -codec, transform, tokenizer, tool-wire, media-IO or registration concern in a new place -applies `REVIEW_PLACEMENT.md`** - the what-lands-where rules. +**A diff that adds a file under `dasllama/`, or adds or moves a def, a `require`, or a module +global in a file under `dasllama/`, applies `REVIEW_PLACEMENT.md`** - the what-lands-where rules. **A `[test]` file that requires any `dasllama/*` module and sits under `modules/dasLLAMA/` outside `tests/` (beside this file) is a defect - move it into `tests/`.** @@ -99,10 +98,10 @@ into the class as a `@template_constant` instead.** **A function-typed global a serialized exe must re-establish lands in a `dasllama/` file with the `[init]` that establishes it at boot; landing one where `REVIEW.das`'s restore-check walk -over `dasllama/` cannot reach it, or weakening that walk, is a defect.** A serialized exe -restores globals as data, so a declaration initializer arrives null and dies at the first invoke -while every `-jit` gate stays green; a global another file's `[init]` arms (`set_runtime_race_hook`) -has no initializer, and its null default is the declared "no hook". +over `dasllama/` cannot reach it is a defect.** A serialized exe restores globals as data, so +a declaration initializer arrives null and dies at the first invoke while every `-jit` gate +stays green; a global another file's `[init]` arms (`set_runtime_race_hook`) has no +initializer, and its null default is the declared "no hook". **Never reorder or merge the float multiplies in a function that builds a RoPE angle table (`dasllama/dasllama_rope.das`).** A regrouping moves the angles in the last bits and flips @@ -135,36 +134,39 @@ kernel race report, is a defect** - instrumentation goes through the profiling rails (`profile_tag` / `profile_marker`, `prof_add`, `asr_prof_add`, the Vulkan tier's `vk_prof()`-gated ledgers), `ARCHITECTURE_MEASUREMENT.md` sec.2.10. -**A clock value that changes what the program DOES - control flow, eviction, a generated -name; not a reported wall-clock time or a best-of reduction over reported wall-clock times - -is marked `// clock: control`, in an engine file (`dasllama/`)** - unmarked, it cannot be told +**In an engine file (`dasllama/`), a clock value that changes what the program DOES - control +flow, eviction, a generated name; not a reported wall-clock time or a best-of reduction over +reported wall-clock times - is marked `// clock: control`** - unmarked, it cannot be told apart from the ad-hoc profiling an engine file may not carry. **Every new kernel or loop the runtime re-enters per token, per frame, or per prefill -quantum - one batch of prompt tokens the prefill path processes in a single pass - is COVERED -by an annotated region entry** - `[hot_path]`, any of the `[no_alloc]` / `[no_env]` / -`[no_io]` contracts, or `[cold_path]` on its only reaching entry. Covered means an annotated -entry reaches it: an annotation binds every function the entry calls, so an interior function -carries nothing of its own except a `[cold_path]` that exempts a rarely-taken branch (a guard -that logs once) from the entry's contracts; an entry no annotated entry reaches carries the -annotation itself, and a function reached only through a registered function value is reached -by none. A region -entry is the outermost such function (a kernel `*_encode` / `*_decode`, a step driver, the CPU -decoder's `forward_*` entries); a loop reached only from a load, stage, bake, or convert path is -not one. A driver that calls the `forward_*` entries and is reached only by a measurement - a -benchmark row, a rig's loop - and never by a served request carries `[cold_path]`, which covers -only the functions below it that carry no annotation of their own. - -**A renamed per-token function is not new: its annotation moves with the name in the same change.** - -**A change to `encode`/`bpe_encode`, or to a function they call, in `dasllama/dasllama_spm.das` / -`dasllama/dasllama_bpe.das` / `dasllama/dasllama_pretok.das`, ships before/after `--tok` rows -(this folder's `benchmarks/lcpp_bench.das`) for a model using the affected tokenizer.** A -change confined to the load path - a metadata default the encode reads as a value - is the -tokenizer-suite rule's below, not this one's. - -**A tokenizer wall-clock time that grows faster than linearly with input size is a defect** - -the `--tok` rows cover at least two input sizes so the growth is readable. +quantum - one batch of prompt tokens the prefill path processes in a single pass - is reached +by an annotated region entry: `[hot_path]`, any of the `[no_alloc]` / `[no_env]` / `[no_io]` +contracts, or `[cold_path]` on its only reaching entry; a renamed per-token function is not +new, and its annotation moves with the name in the same change.** An annotation binds every +function the entry calls; an unreached loop has no contract (`ARCHITECTURE_RUNTIME.md` sec.2.11). + +**The annotation sits on the region entry - the outermost function the runtime re-enters per +token, per frame, or per prefill quantum: a kernel `*_encode` / `*_decode`, a step driver, the +CPU decoder's `forward_*` entries, or a function reached only through a registered function +value - and an interior function carries an annotation only when it is a `[cold_path]` on a +rarely-taken branch (a guard that logs once).** + +**A loop reached only from a load, stage, bake, or convert path is not a region entry: it +carries no `[hot_path]` and none of the `[no_alloc]` / `[no_env]` / `[no_io]` contracts; a +`[cold_path]` may sit on it.** + +**A driver that calls the `forward_*` entries and is reached only by a measurement - a +benchmark row, a rig's loop - and never by a served request carries `[cold_path]`, which +covers only the functions below it that carry no annotation of their own.** + +**A change to `encode`/`bpe_encode`, or to a function they call, in +`dasllama/dasllama_spm.das` / `dasllama/dasllama_bpe.das` / `dasllama/dasllama_pretok.das`, +ships before/after `--tok` rows (this folder's `benchmarks/lcpp_bench.das`) at two or more +input sizes, for a model using the affected tokenizer.** A change confined to the load path - +a metadata default the encode reads as a value - does not fire this rule. + +**A tokenizer wall-clock time that grows faster than linearly with input size is a defect.** **A change to code or data in `dasllama/dasllama_tokenizer.das`, `dasllama/dasllama_spm.das`, `dasllama/dasllama_bpe.das`, or `dasllama/dasllama_pretok.das`, or to the special-token or @@ -188,9 +190,10 @@ depend on - what it calls, types, requires, or parses (facade functions, CLI fla knobs, file formats, defaults, what the installed SDK lets a program `require`) - plus the in-repo rig and tool surface: any output another tool parses. A console-only diagnostic is not user-facing. -**A diff that makes a statement in an `ARCHITECTURE_*.md` companion, a module-root document, a -`//!` docstring, or a document outside this folder whose own checklist routed this diff here, -false updates it in the same change** - a section no `[arch]` cites is the reviewer's alone. +**A diff that falsifies a statement in an `ARCHITECTURE_*.md` companion, a module-root +document, a `//!` docstring, or a document outside this folder whose own checklist routed this +diff here, updates it in the same change** - no lint checks a section no `[arch]` cites; only +the reviewer does. **Weakening `dasllama_lint` (`dasllama/dasllama_lint.das`) - the compile-time check that a consumer requires only this module's public entry modules, matched by the resolved file's @@ -220,9 +223,9 @@ edit that text asks for. What the gate enforces is read from the gate itself; ea finding text states its own rule. **A new `REVIEW.das` check ships its line on the checked file's sec.1 charter - in an -`ARCHITECTURE_*.md` companion, never `ARCHITECTURE.md` - in the same change.** The line names the check and the names it -licenses. A licensed name is one that check does not flag. When the check licenses no names, -the line says so. +`ARCHITECTURE_*.md` companion, never `ARCHITECTURE.md` - in the same change.** The line names +the check and the names it licenses. A licensed name is one that check does not flag. When the +check licenses no names, the line says so. **Checked-in text under `modules/dasLLAMA/` - docs, comments, or string data, any language - that describes a mechanism of the reference build, or names that build, its binaries or its @@ -241,30 +244,30 @@ page published beside a released model or pack - or in a ledger row naming a lic reason to adopt or reject a model, a dataset, or a dependency; anywhere else in prose it is a defect.** -**A def of a facade file, and a new OVERLOAD of one, is TAUGHT: demonstrated in runnable code -in a `tutorials/dasLLAMA/*.das` source and narrated on a -`doc/source/reference/tutorials/dasLLAMA_*.rst` page.** The facade files are -`dasllama/dasllama.das` and `dasllama/dasllama_tts.das` - a facade file's defs reach a consumer -through `require dasllama/dasllama`, and a diff that makes another file's defs reach that way -adds it here and to `check_tutorial_floor` in the same change. `REVIEW.das`'s -`check_tutorial_floor` matches def NAMES only, so an overload passes on a sibling's tutorial - -the reviewer confirms a tutorial calls the NEW signature, and a mention that only names it (a -comment, a passing reference) does not count. - -**A NEW `[EnvConfig]` area struct is rendered by `env_markdown()` in the same change.** A -struct the renderer never emits is absent from `ENVIRONMENT.md` and invisible to every test; -a struct the renderer emits but the registry does not is caught by -`tests/test_env_registry.das`. +**A def of a facade file - one whose defs reach a consumer through `require dasllama/dasllama`; +`REVIEW.das`'s `FACADE_FILES` is the list - and a new OVERLOAD of one, is TAUGHT: demonstrated +in runnable code in a `tutorials/dasLLAMA/*.das` source and narrated on a +`doc/source/reference/tutorials/dasLLAMA_*.rst` page.** +`REVIEW.das`'s `check_tutorial_floor` matches def NAMES only, so an overload passes on a +sibling's tutorial - the reviewer confirms a tutorial calls the NEW signature, and a mention +that only names it (a comment, a passing reference) does not count. + +**A diff that makes another file's defs reach a consumer through `require dasllama/dasllama` +adds that file to `REVIEW.das`'s `FACADE_FILES` in the same change.** + +**A NEW `[EnvConfig]` area struct is rendered by `env_markdown()` in the same change.** A struct +the renderer never emits is absent from `ENVIRONMENT.md` and invisible to every test; a struct +the renderer emits but the registry does not is caught by `tests/test_env_registry.das`. **Hand-editing `dasllama/dasllama_unicode.das`'s RANGES/WS tables is a defect - regenerate them by retranscoding `$LCPP/src/unicode-data.cpp` (the reference checkout) instead.** -**A diff the placement routing line above routes, or one that changes what a file owns, lands -the sec.1 edit that keeps the charters true - in an `ARCHITECTURE_*.md` companion, never -`ARCHITECTURE.md` - in the same change.** A diff that adds a file to any -folder where another file has its own sec.1 charter line lands the new file's charter line -too. A module-root doc file - a ledger, a plan - has no charter line and needs no charter -edit. +**A diff that adds a file under `dasllama/`, moves a def, a `require`, or a module global +between files there, or changes what a file owns, lands the sec.1 edit that keeps the +charters true - in an `ARCHITECTURE_*.md` companion, never `ARCHITECTURE.md` - in the same +change.** A diff that adds a file to any folder where another file has its own +sec.1 charter line lands the new file's charter line too. A module-root doc file - a ledger, a +plan - has no charter line and needs no charter edit. **A diff that adds, removes, or moves a section of an `ARCHITECTURE_*.md` companion, or adds or removes a companion, lands `ARCHITECTURE.md`'s index line and section range, the @@ -278,9 +281,9 @@ changes none of its arithmetic.** The CPU form serves every box with no driver. **A diff that writes a CPU feature name in a `[tune_perm]` `requires=` argument that `TUNE_KNOWN_FEATURES` (`modules/dasLLVM/daslib/llvm_tune.das`, repo root) does not list adds it -there in the same change.** The `features` fingerprint saved with every sidecar is this box's pass/fail over -that list, so a name outside it is never recorded and a box adopting a shipped profile re-runs -the tuning the profile was meant to save. +there in the same change.** The `features` fingerprint saved with every sidecar is this box's +pass/fail over that list, so a name outside it is never recorded and a box adopting a shipped +profile re-runs the tuning the profile was meant to save. **A value that a team-lane kernel reads - anything reachable from a `team_parallel_for` / `team_parallel_for_indexed` / `team_parallel_stages` body (`daslib/jobque_boost.das`, repo diff --git a/modules/dasLLAMA/REVIEW_GPU.md b/modules/dasLLAMA/REVIEW_GPU.md index 95d400540d..b909a9ec6c 100644 --- a/modules/dasLLAMA/REVIEW_GPU.md +++ b/modules/dasLLAMA/REVIEW_GPU.md @@ -1,7 +1,8 @@ # dasLLAMA GPU Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `ARCHITECTURE_GPU.md`, `ARCHITECTURE_GPU_VULKAN.md`. +docs: `ARCHITECTURE_GPU.md`, `ARCHITECTURE_GPU_VULKAN.md`. Planned work: `followup_metal.md` +for Metal, `followup_vulkan.md` for Vulkan. **Routed from `REVIEW.md`: a diff that checklist routes here applies this list together with it.** @@ -133,26 +134,30 @@ axis - one compile-time choice, such as single/batch, format, or single-pass/chu **A copy-pasted kernel twin - one of two kernel classes whose bodies differ on one compile-time choice - or a kernel split into hand instances where a `static_if` on a `@template_constant` -serves, is a defect - kernel twins stamp one `class template`, whatever that choice is.** Body divergence is carried by a `@template_constant`, or by an -overridden method spliced flat at emission. - -**A stamped kernel family - a class template's stamps, or a base shell's `[vk_dispatch]` / -`[metal_dispatch]` leaves - that binds a real buffer to a field a stamp's own body never reads -- a dummy bind that exists only to fill the slot - is a defect: gate the field with -`@template_gate` where a template constant decides it, and where the family shares one set -layout on purpose, name that case in `ARCHITECTURE_GPU.md` (Metal) or -`ARCHITECTURE_GPU_VULKAN.md` (Vulkan).** A field the body reads -under a run-time flag is read, and its unread arm binds a placeholder the kernel never touches. +serves, is a defect - kernel twins stamp one `class template`, whatever that choice is.** Body +divergence is carried by a `@template_constant`, or by an overridden method spliced flat at +emission. + +**A kernel-family stamp - one stamp of a class template, or one of the classes deriving from a +base shell that carry a `[vk_dispatch]` / `[metal_dispatch]` - that binds a real buffer to a +binding whose fields its compiled body, inherited code included, never reads - a dummy bind that +exists only to fill the slot - is a defect: gate the field with `@template_gate` where a +template constant decides it, and where the family shares one set layout on purpose, name that +case in `ARCHITECTURE_GPU.md` sec.1.5's ledgered kernel-binding asymmetries (Metal) or +`ARCHITECTURE_GPU_VULKAN.md` sec.2.2ac (Vulkan).** Several fields, declared in the stamp or in +the shell, may share one binding - `@role = "alias"` marks such a view - so the binding counts +as read when the compiled body reads any of them. A field the body reads under a run-time flag +is read, and its unread arm binds a placeholder the kernel never touches. **A diff that forks a kernel class out of a shared template shows that the bodies no longer differ on the compile-time choice the template carried, and names that choice in the surviving template's comment.** -**A `[metal_dispatch]` / `[vk_dispatch]` field whose memory is load-once - a model plane, or -an `upload_region` upload never written after arming - is a defect unless it carries -`@role = "weight"`, even when the kernel compiles and passes parity.** A field the kernel reads -under a run-time flag takes the role of its read arm; the placeholder its unread arm binds is -never read, so its lifetime does not decide the role. +**A `[metal_dispatch]` / `[vk_dispatch]` binding whose memory is load-once - a model plane, or +an `upload_region` upload never written after arming - is a defect unless a field at that +binding carries `@role = "weight"`, even when the kernel compiles and passes parity.** A field +the kernel reads under a run-time flag takes the role of its read arm; the placeholder its +unread arm binds is never read, so its lifetime does not decide the role. **`@role = "weight"` on per-encode data the kernel reads - a pooled buffer the host refills each encode - is a defect; a per-encode field either omits `@role` or names the access its body @@ -170,9 +175,9 @@ an arm that forces the device mode the class is gated on; a class no census mode a census model that does.** A Vulkan class never joins `CENSUS_NEVER_DISPATCHED`, which takes Metal classes only. -**Every `@ssbo` field of a new kernel class declared in `dasllama/` carries at least one of the -annotations its `[metal_dispatch]` / `[vk_dispatch]` builder reads - `@binding`, `@role`, -`@off`, `@default`.** A field carrying none of them is dropped from the bind list with no error. +**Weakening the `[metal_dispatch]` / `[vk_dispatch]` lens's refusal to compile an `@ssbo` field +with no `@binding`, or an `@ssbo` field the kernel body never accesses that declares no `@role`, +is a defect.** **Weakening `[metal_dispatch]`'s refusal to compile a `@workgroup` field with no `tgmem=` spec, or its gate `test_lens_tgmem_gate` (`tests/test_metal_misc_kernels.das`), is a defect.** @@ -225,8 +230,8 @@ backend serving the same path faster or slower is not such a change; a seat of t **A change to code that a served GPU decode or prefill path executes ships GPU-vs-CPU parity on one q8 and one kq (K-quant) model the changed path serves.** That code is anything a -served GPU decode or prefill call executes OR that selects what it -executes - a driver, a kernel class it dispatches, that class's builder, a servability gate, a +served GPU decode or prefill call executes OR that selects what it executes - a driver, a +kernel class it dispatches, that class's builder, a servability gate, a race that picks which kernel serves, a forwarder default, a weight-region or residency path, the tier forwarders and the Vulkan tier-dispatch seams (`dasllama/dasllama_vulkan_seams.das`) the call routes through; never the bake paths, never a comment. diff --git a/modules/dasLLAMA/REVIEW_GPU_RACE.md b/modules/dasLLAMA/REVIEW_GPU_RACE.md index abf8a8e7ac..681f3ca95e 100644 --- a/modules/dasLLAMA/REVIEW_GPU_RACE.md +++ b/modules/dasLLAMA/REVIEW_GPU_RACE.md @@ -1,19 +1,19 @@ # dasLLAMA GPU Race Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `ARCHITECTURE_GPU.md`, `ARCHITECTURE_GPU_VULKAN_GEMM.md`, `ARCHITECTURE_MEASUREMENT.md`. -Planned work: `followup_metal.md` for Metal, `followup_vulkan.md` for Vulkan. +docs: `ARCHITECTURE_GPU.md`, `ARCHITECTURE_MEASUREMENT.md`. Planned work: +`followup_metal.md` for Metal, `followup_vulkan.md` for Vulkan. **Routed here by another checklist: a diff that checklist routes here applies this list together with it.** -**A hand-binding arm - a race or knockout timing arm (a race times two implementations of one -computation on one queue and compares their outputs; a knockout attributes cost across stages -instead of selecting between implementations) that mirrors a class's binding order by hand - a -literal bind number, or a positional buffer and size array - instead of naming the class's -fields - that binds a field at a position the class does not declare for that field is a -defect.** A mis-bound arm dispatches, reads the wrong buffer, and its timing selects the wrong -kernel silently. +**A hand-binding arm that binds a field at a position the class does not declare for that field +is a defect.** A hand-binding arm is a race or knockout timing arm - a race times two +implementations of one computation on one queue and compares their outputs, a knockout skips a +stage to measure that stage's cost - that mirrors a class's binding order by hand, with a +literal bind number or a positional buffer and size array, instead of naming the class's fields. +A mis-bound arm dispatches, reads the wrong buffer, and its timing selects the wrong kernel +silently. **A hand-binding arm outside `dasllama/`, or one whose pipeline source or threadgroup-memory size arrives as a function parameter rather than a literal global, states in the PR that its @@ -21,15 +21,15 @@ binding order and push-constant layout were verified by hand against the class d The `REVIEW.das` gate `check_race_bind_numbers` cannot read those arms, and an arm nothing checked is where a mis-numbered bind reaches the board. -**A diff that changes anything a hand-binding arm must mirror to dispatch a kernel - binding -numbers, kargs (kernel-argument struct) layout, threadgroup memory, staging shape (the operand -tile a kernel copies into threadgroup memory before it computes), grid or threadgroup geometry - -fixes or deletes every such arm in the same change.** An arm left dispatching stale geometry -measures the wrong kernel silently. +**A diff that changes a kernel's binding numbers, kargs (kernel-argument struct) layout, +threadgroup memory, staging shape (the operand tile a kernel copies into threadgroup memory +before it computes), or grid or threadgroup geometry resyncs or deletes, in the same change, +every arm that mirrors that kernel's binding order by hand and every arm ledgered as a +retained reference in `ARCHITECTURE_GPU.md` sec.2.2b (Metal) or `ARCHITECTURE_MEASUREMENT.md` +sec.2.5 (Vulkan).** An arm left dispatching stale geometry measures the wrong kernel silently. **Race and knockout code inside the engine (`dasllama/`) sits in the file that owns the kernel -family it races, or - for a knockout - the file that owns the stage whose cost it removes, and -never in another engine file.** +family it races, or - for a knockout - the file that owns the stage whose cost it removes.** **Scaffolding that race sites in two DIFFERENT engine files share sits in `dasllama/dasllama__common.das`.** Scaffolding two races in one file share stays in that @@ -39,51 +39,55 @@ file. small enough to sit in cache ranks the kernels by an effect production never sees, and the race then picks the slower kernel. -**A timing arm whose ranking a checked-in document, box profile or sidecar records as decided - -a runtime crown (the winner the box profile records and the served graph dispatches), a -tune-sidecar row (the winner a kernel's `*.tune.json` sidecar records), an architecture-doc -entry - times its kernel at every width that kernel is dispatched at: every verify width, every -row count from 2 to `MTP_MAX_ROWS - 1` (`dasllama/dasllama_common.das`), when the batched decode -driver (`dasllama/dasllama__decode.das`) dispatches it per row count; the tile's width when -it is dispatched at one fixed width per tile; each power of two it spans when the instrument -runs on a power-of-two batch grid; the served window row counts when it is a prefill tile over a -variable window.** A ranking timed at one width alone is applied at widths it was never ranked -at. +**A timing arm whose ranking a checked-in document, box profile or sidecar records as decided +times its kernel at every fixed width it is dispatched at: every width one speculative-verify +batch runs at; every row count from 2 to `MTP_MAX_ROWS - 1` (`dasllama/dasllama_common.das`) +where the batched decode driver (`dasllama/dasllama__decode.das`) dispatches it per row +count; the tile's own width where it is dispatched at one fixed width; each power of two it +spans on a power-of-two batch grid.** A ranking timed at one width alone is applied at widths it +was never ranked at. + +**A timing arm for a prefill tile over a variable window, whose ranking a checked-in document, +box profile or sidecar records as decided, times its kernel at one window whose token count is +a whole multiple of the tile's row count and at one where it is not.** The short last tile is +what makes the tile take its partial-tile store path. **An `ARCHITECTURE_GPU.md` sec.2.2b entry for a kernel ranked on a power-of-two batch grid names that grid.** **A kernel A/B race arm whose ranking a checked-in document, box profile or sidecar records as -decided binds a DIFFERENT output buffer for consecutive dispatches of its chain, never one -shared output, and every arm of the race handles the hazard between its dispatches the same -way.** One shared output serializes the chain on its write-after-read hazard while the served -graph overlaps consecutive dispatches, so the race ranks the arms on a shape production never -runs; an arm serialized by a barrier races an arm that overlaps. +decided binds a different output buffer for consecutive dispatches of its chain, never one +shared output.** One shared output serializes the chain on its write-after-read hazard while the +served graph overlaps consecutive dispatches, so the race ranks the arms on a shape production +never runs. + +**Every arm of a kernel A/B race whose ranking a checked-in document, box profile or sidecar +records as decided handles the hazard between its dispatches the same way.** An arm serialized +by a barrier races an arm that overlaps. **Weakening the burn phase of `race_pair_ms` (`dasllama/dasllama_metal_common.das`) - the GPU work it runs on both arms before the first timed round - is a defect.** -**A race arm whose ranking a checked-in document, box profile or sidecar records as decided -and that does not go through `race_pair_ms` burns GPU work on its own arms before its first -timed round.** The arms alternate, so a first round on a cold GPU clock charges the clock's -ramp-up to whichever arm ran first. +**A race whose ranking a checked-in document, box profile or sidecar records as decided and that +does not go through `race_pair_ms` burns GPU work on both arms before the first timed round.** +The arms alternate, so a first round on a cold GPU clock charges the clock's ramp-up to +whichever arm ran first. **A timed encoder in a race arm whose ranking a checked-in document, box profile or sidecar records as decided issues its dispatches back to back.** An encoder that leaves gaps between its dispatches times an idle clock. -**A diff that ports an A/B lab's winning variant into a kernel deletes, in the same change, -that variant's class and any `*_variants.das` code that exists only for it; an arm that -survives the port dispatches the shipped kernel class's generated source, or is ledgered as a -retained reference in the architecture doc that owns the kernel's tier (`ARCHITECTURE_GPU.md` -sec.2.2b for Metal, `ARCHITECTURE_MEASUREMENT.md`'s probe axes for Vulkan).** An A/B lab is a -timing script whose output selects between implementations of the same compute; a decided arm -that outlives its decision degrades into an unmaintained duplicate of the kernel it seeded. - -**A diff that leaves an A/B lab with no undecided arm deletes the lab's driver and its -remaining arm in the same change, unless the architecture doc that owns the kernel's tier -ledgers the lab as a retained instrument.** - -**A diff that moves a shipped kernel's staging shape or its grid or threadgroup geometry -resyncs or deletes, in the same change, every arm the tier's architecture doc ledgers as a -retained reference that reimplements that kernel.** +**A diff that ports an A/B lab's winning variant - an A/B lab is a timing script that picks +between spellings of one compute - into a kernel deletes, in the same change, that variant's +class and any `*_variants.das` code that exists only for it.** + +**An arm that survives a port of the winning variant of an A/B lab - a timing script that picks +between spellings of one compute - into a kernel dispatches the shipped kernel class's generated +source, or is ledgered as a retained reference in the architecture doc that owns the kernel's +tier: `ARCHITECTURE_GPU.md` sec.2.2b for Metal, `ARCHITECTURE_MEASUREMENT.md` sec.2.5 (the +Vulkan GEMM probe's axes).** A decided arm that outlives its decision degrades into an +unmaintained duplicate of the kernel it seeded. + +**A diff that leaves an A/B lab - a timing script that picks between spellings of one compute - +with no undecided arm - every arm's ranking recorded as decided by a checked-in document, box +profile or sidecar - deletes the lab's driver and its remaining arm in the same change.** diff --git a/modules/dasLLAMA/REVIEW_GPU_VULKAN.md b/modules/dasLLAMA/REVIEW_GPU_VULKAN.md index eb49a6f61a..f10dd56eb8 100644 --- a/modules/dasLLAMA/REVIEW_GPU_VULKAN.md +++ b/modules/dasLLAMA/REVIEW_GPU_VULKAN.md @@ -1,8 +1,7 @@ # dasLLAMA Vulkan Tier Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -docs: `ARCHITECTURE_GPU_VULKAN.md`, `ARCHITECTURE_GPU_VULKAN_GEMM.md`, -`ARCHITECTURE_GPU_VULKAN_RESIDENCY.md` and `ARCHITECTURE_GPU_VULKAN_DECODE.md`. Planned work: +docs: `ARCHITECTURE_GPU_VULKAN.md`, `ARCHITECTURE_GPU_VULKAN_GEMM.md`. Planned work: `followup_vulkan.md`. **Routed from `REVIEW_GPU.md`: a diff that checklist routes here applies this list together @@ -20,14 +19,14 @@ are device-lifetime state that survives the drop and rebuilds lazily. binds it.** The bind site cannot shrink a buffer that was sized wrong. **Never cache a descriptor set or a host address - a pointer into CPU memory - across -dispatches in state that `vk_drop_model_state` does not clear** - hold it in that function's -own state in `dasllama/dasllama_vulkan_common.das`, the way the `*_ready` latches are. +dispatches in state that `vk_drop_model_state` does not clear** - hold it in +`dasllama/dasllama_vulkan_common.das` module state that `vk_drop_model_state` clears. -**Never read a `[spirv_decode]` callback's quant bytes by indexing `unpack8` of a 32-bit word -with a runtime value - read them as 16-bit lanes instead: load the lane `w` from an `int16[N]` -block member, select the byte with `unpack8(w)[i & 1u]`, and pull sub-fields out by shift and -mask.** The vendor driver's shader compiler pattern-matches only the 16-bit spelling into its -block-load path, and a runtime byte select loses that path for the whole kernel. +**Never read a weight plane's quant bytes in a kernel body by indexing `unpack8` of a 32-bit +word with a runtime value - read them as 16-bit lanes instead: load the lane `w` from an +`int16[N]` block member, select the byte with `unpack8(w)[i & 1u]`, and pull sub-fields out by +shift and mask.** The vendor driver's shader compiler pattern-matches only the 16-bit spelling +into its block-load path, and a runtime byte select loses that path for the whole kernel. **A diff that changes when `vk_rdec_prefill_ids` - the resident prefill that takes token ids rather than embeddings - accepts a call, or when the override that routes to it @@ -38,13 +37,13 @@ embed when that gate returns true, so a gate true where the prefill path decline next consumer an unfilled residual stream. **A Vulkan-tier serving gate that decides at load - a predicate or per-layer loop whose false -branch or `continue` routes work to the CPU path - logs at load how many layers or planes it -left on the CPU and the reason it left them.** A silent decline is a fallback a user finds -only by profiling. +branch or `continue` routes work to the CPU path - that does not log at load how many layers or +planes it left on the CPU and the reason it left them is a defect.** A silent decline is a +fallback a user finds only by profiling. **A Vulkan-tier serving gate that decides per call - a predicate or loop whose false branch or -`continue` routes work to the CPU path - logs the concrete reason it declined, once per reason -per armed model.** +`continue` routes work to the CPU path - that does not log the concrete reason it declined, +once per reason per armed model, is a defect.** **A prefill GEMM dispatched at a nonzero start row never asks `cm2_split_k` for a split - it encodes unsplit.** The split-k reduce sums partial planes counted from row 0, so a dispatch @@ -59,47 +58,58 @@ codec no kernel covers silently drops that codec's GPU path. **A diff that changes what a kq superblock format's cm2 tile emits - its instance set, its decode body or four-wide twin (`decode_v4`), its `DECV4` or `DECVEC` constant, or the shared -`KqCm2BatchT` body - puts that format's `cm2:` probe rows (`harness/vk_gemm_probe.das`), -both the `DASLLAMA_VK_DECVEC=1` and the `=0` rows, in the PR body, or the claim that the -format's emitted kernels are byte-identical to master's.** A cm2 tile is the -NV_cooperative_matrix2 GEMM class stamped per weight format and token-column width (the -class's `BN`) in `dasllama/dasllama_vulkan_classes.das`. +`cm2_tile` or `run` of `KqCm2BatchT` - puts that format's `cm2:` probe rows +(`harness/vk_gemm_probe.das`), both the `DASLLAMA_VK_DECVEC=1` and the `=0` rows, in the PR +body, or the claim that the format's cm2-stamped kernels are byte-identical to master's.** A +cm2 tile is the NV_cooperative_matrix2 GEMM class stamped per weight format and token-column +width (the class's `BN`) in `dasllama/dasllama_vulkan_classes.das`. + +**A diff that changes what a kq superblock format's KHR tile emits - its `khr_stage16` +override, its instance set, or the shared `khr_tile` or `run` of `KqCm2BatchT` - puts that +format's kernel cell - that format's test block in `tests/test_vulkan_kernels.das` - run on its +KHR arm in the PR body, with either the `khrx` probe rows (`harness/vk_gemm_probe.das`) or a +`tests/test_gpu_resident_hybrid.das` run on a model in that format.** A KHR tile is the +`KhrBatch` class stamped per weight format in `dasllama/dasllama_vulkan_classes.das`. **A `kq_sb` format (`dasllama/dasllama_kqformat.das`) that joins the cm2 template - a -`Cm2T` format template in -`dasllama/dasllama_vulkan_classes.das` - ships its KHR instantiation (`KhrBatch`, the -`kq_batch__khr_cls` dispatch) and its arm in each of `khr_cls_ensure`, `khr_cls_set` and -`khr_cls_enc` (`dasllama/dasllama_vulkan_prefill.das`) in the same change, and that format's -kernel cell in `tests/test_vulkan_kernels.das` runs its KHR arm.** `pf_f16_feed` admits every -`kq_sb` format in mm mode, so a format with no KHR class reaches the ladders' fall-through -`verify` - a panic on a KHR-only card that no cm2 box reproduces without `DASLLAMA_COOPMAT=mm`. +`Cm2T` format template in `dasllama/dasllama_vulkan_classes.das` - ships its KHR +instantiation (`KhrBatch`, the `kq_batch__khr_cls` dispatch) and its arm in each of +`khr_cls_ensure`, `khr_cls_set` and `khr_cls_enc` (`dasllama/dasllama_vulkan_prefill.das`), in +the same change.** `pf_f16_feed` admits every `kq_sb` format, so a format with no KHR +class panics in `khr_cls_ensure`, `khr_cls_set` or `khr_cls_enc` on a card whose +cooperative-matrix mode is KHR. + +**A `kq_sb` format that ships a KHR instantiation runs its KHR arm in that format's kernel cell, +in the same change.** + +**`khr_stage16` stays abstract on `KqCm2BatchT` (`dasllama/dasllama_vulkan_classes.das`) - a +diff that gives it a default body is a defect.** **A kernel body that calls a `[spirv_decode]` method directly passes the plane element itself (`decode(wq[i], ...)`), never a local copy of it (`let blk = wq[i]` then `decode(blk, ...)`).** Both compile: on the element the emitter passes the index and the callee chains through the -plane; on a copy it loads and spills the whole block per call, and the KHR kq tile measured -well under half its rate that way (`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l). +plane; on a copy it loads and spills the whole block per call +(`ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2k). -**A diff that puts a format's `cm2:` probe rows in the PR body whose `DASLLAMA_VK_DECVEC=1` -row is slower than its `=0` row ships one of two fixes in the same change: a hand-written `decode_v4` under -`override DECV4 = true` on that format's class (`dasllama/dasllama_vulkan_classes.das`), -re-measured so its `=1` row now beats its `=0` row; or `override DECV4 = false` and -`override DECVEC = false` together, which puts the format back on the scalar callback.** With -`DECV4 = true` the class never reads `DECVEC`, so `override DECVEC = false` alone leaves the -hand-written twin running. +**A diff that puts a format's `cm2:` probe rows in the PR body whose +`DASLLAMA_VK_DECVEC=1` row is slower than its `=0` row ships one of two fixes in the same +change: a hand-written `decode_v4` under `override DECV4 = true` on that format's class +(`dasllama/dasllama_vulkan_classes.das`), re-measured so its `=1` row now beats its `=0` row; +or `override DECV4 = false` and `override DECVEC = false` together, which puts the format back +on the scalar callback.** With `DECV4 = true` the class never reads `DECVEC`, so +`override DECVEC = false` alone leaves the hand-written twin running. **A diff that changes how many GPU timestamps the resident decode's token command records - the `pfq_ts` calls in `dasllama/dasllama_vulkan_decode.das` - updates the stamp count `rdq_sample` -expects, and with it the role-name table and accumulator of every layer kind whose count -moved: attention's `rdq_role_names` with `g_rdq_role`, recurrent's `RDQ_DN_NAMES` with -`g_rdq_dn`, both accumulators in `dasllama/dasllama_vulkan_common.das` - in the same change.** -`rdq_sample` indexes a fixed count per layer, so one extra or missing timestamp reports every -later stamp under the wrong role name. +expects and, for every layer kind whose count moved, that file's role-name table +(`rdq_role_names`, `RDQ_DN_NAMES`) and its accumulator in `dasllama/dasllama_vulkan_common.das` +(`g_rdq_role`, `g_rdq_dn`), in the same change.** `rdq_sample` indexes a fixed count per layer, +so one extra or missing timestamp reports every later stamp under the wrong role name. **A decode GEMV class - a `KqGemvBase` leaf in `dasllama/dasllama_vulkan_classes.das` - that stages a codebook into `@workgroup` memory reads it from the family's grid buffer (`gridb`, -binding 6, filled by `kq_grid_dev` at the format's `KQ_GRID_` offset), never from a -`*_grid_word` accessor.** The accessor is a constant composite the driver reads lane-serially +filled by `kq_grid_dev` at the format's `KQ_GRID_` offset), never from a `*_grid_word` +accessor.** The accessor is a constant composite the driver reads lane-serially per index, and a two-row workgroup pays that read on every row pair it walks (`ARCHITECTURE_GPU_VULKAN.md` sec.2.2ab). diff --git a/modules/dasLLAMA/REVIEW_MEASUREMENT.md b/modules/dasLLAMA/REVIEW_MEASUREMENT.md index c034331d30..ef2b6f19f5 100644 --- a/modules/dasLLAMA/REVIEW_MEASUREMENT.md +++ b/modules/dasLLAMA/REVIEW_MEASUREMENT.md @@ -3,15 +3,23 @@ **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture doc: `ARCHITECTURE_MEASUREMENT.md`. Planned work: `PERF_LEDGER.md`. -Figure rules here bind every surface the diff under review writes a figure on, wherever it -sits - code comments, checked-in docs and plans, checked-in data a run produced, the PR body - -except a page a `site*/` or `utils/` folder checklist owns (repo root), which answers there. +Figure rules here bind any text this change adds - in the repo or in its PR body - outside the +pages a `site*/` or `utils/` folder checklist owns (repo root); those pages answer to their own +checklist. The ledger rules key on `PERF_LEDGER.md`. The naming a figure rule asks for sits in +the figure's own sentence, in a table heading that covers the table's rows, in a section-level +provenance line that covers the paragraphs under it, or in a citation of the passage whose +provenance line covers it. **A `PERF_LEDGER.md` entry never states a tok/s figure or a turn wall that the `-jit` script produced - a `-jit` A/B pair enters as its ratio, with the arms' absolute rates left in the run's report.** A served turn is one whole prefill-plus-decode run; a turn wall is its wall. -The served-turn figures the ledger does state come from the released `lcpp_bench` exe -(`benchmarks/lcpp_bench.das` built by `daspkg release`) or a board cell. + +**A `PERF_LEDGER.md` entry states a served-turn figure only when the released `lcpp_bench` exe +(`benchmarks/lcpp_bench.das` built by `daspkg release`) or a board cell produced it.** + +**Only a reference cell of `performance/gen_bench_records.das` - a cell that times, over a board +workload, a binary this repository does not build - writes that binary's wall into +`PERF_LEDGER.md`.** A wall taken any other way stays in the report where it was taken. **A difference, ratio, or percentage of two measured walls, neither produced by the `-jit` script, written into `PERF_LEDGER.md` carries both raw walls in the entry.** @@ -20,27 +28,28 @@ script, written into `PERF_LEDGER.md` carries both raw walls in the entry.** **A diff that adds a `PERF_LEDGER.md` entry whose reading no board cell produced names the instrument that produced it - the script or exe whose output is that wall or rate.** A board -cell is a run `performance/gen_bench_records.das` spawns, or a manual `benchmarks/lcpp_bench.das` -cell its `PROFILE.md` section documents; its reading lands as a row of -`performance/records/.json`. A ruler record (`performance/records/mtp/*.json`, written by -`harness/mtp_ruler.das`) is not a board cell: a diff cannot mint or re-mint one in place of a -board row. +cell is a run `performance/gen_bench_records.das` spawns, or a manual +`benchmarks/lcpp_bench.das` cell its `PROFILE.md` section documents; its reading lands as a row +of `performance/records/.json`. A ruler record (`performance/records/mtp/*.json`, written +by `harness/mtp_ruler.das`) is not a board cell. + +**A ruler record's third-party row is written by the same `harness/mtp_ruler.das` run that +wrote the das row it pairs with.** A wall pasted in from another run measures a different +prompt, drafter and settle state, and the acceptance rate it implies is not the pair's. **A `PERF_LEDGER.md` entry tags its reading `direction-grade` when the reading compares across two processes or two commits, and `out-of-process` when the wall was measured from outside the benchmark process.** -**A checked-in document that states a `--for-debug-purposes` reading for a model and arm a -committed board row covers cites that row and marks the reading debug-jit; the board itself -changes only through a re-minted board cell.** A `--for-debug-purposes` row is the `-jit` -script's own output. +**A checked-in document that states a `--for-debug-purposes` reading for a model and arm that a +committed board row covers cites that row and marks the reading debug-jit.** A +`--for-debug-purposes` row is the `-jit` script's own output. -**A `PERF_LEDGER.md` entry carrying a figure from another project names the source and the -report it came from and tags it `external`.** +**A `PERF_LEDGER.md` entry carrying a figure this repository did not measure names the source +and the report it came from and tags it `external`.** **A diff never rests an adoption decision about what the engine serves on a figure from -another project - that decision rests on a self-measured board cell.** A selection between -kernel forms settles on its own instrument's run (the next rule). +another project - that decision rests on a self-measured board cell.** **A diff that adds an entry to `PERF_LEDGER.md` never records a selection timing - a timing that picks a winner between candidate kernel forms.** That timing settles its adoption @@ -48,52 +57,57 @@ decision in the report of the run that took it and in the PR that lands the kern winner enters the ledger only through a re-measured board cell. **A diff that makes the engine run end to end - serving a client, or measuring - by a route no -board cell exercises mints that cell in the same change; a route the board cannot carry names, -in the same change, the artifact that stands in for the cell - the record or gate output that -proves the route ran end to end.** A route is anything that changes which code runs a whole -prefill-plus-decode pass end to end - the format, modality, family, backend, serving lane, GPU tower, sampler class, compile tier, -cross target, or the path a run with no flags and no environment overrides takes. The fat -exe's stand-in is `tune_gate`'s fat-world report (`performance/profile_common.das`) plus the -`sanity:` lines of its `lcpp_bench` run. +board cell exercises mints that cell in the same change.** A route is anything that changes +which code runs a whole prefill-plus-decode pass end to end, including the path a run with no +flags and no environment overrides takes. + +**A change that owes a board cell for a route no leg of `performance/gen_bench_records.das` +mints a row for names instead, in the same change, the record or gate output that proves the +route ran end to end.** + +**A diff that makes the fat exe - a shipped exe carrying its tune profile +(`ARCHITECTURE_MEASUREMENT.md` sec.2.42a) - run end to end names `tune_gate`'s fat-world report +(`performance/profile_common.das`) and the `sanity:` lines of its `lcpp_bench` run.** **A diff that claims to make an already-served path faster, from an author whose box mints that path, re-mints a board row (`performance/records/.json`) that exercises that path, in the same change, and names that row in the PR body.** A box mints a path when `performance/gen_bench_records.das` mints a row for it on that box - a leg its `stored_row_leg` -(`performance/profile_common.das`) admits - rather than refusing or skipping it; a backend -route the record rig has no leg for takes the stand-in clause instead. Where no row exercises -the path, the diff mints one. The board is the module's public memory of what serving costs; a -kernel win that never lands there is invisible to the next regression check. +(`performance/profile_common.das`) admits - rather than refusing or skipping it. Where no row +exercises the path, the diff mints one. The board is the module's committed record of what +serving costs; a kernel win that never lands there is invisible to the next regression check. **A timing figure of a served turn as a whole - tok/s, latency, a whole-turn model or engine -comparison, the 512-token prefill (pp512) and 128-token decode (tg128) rates - that this module -writes down as a measurement is a defect without either a board cell behind it or a provenance -line, covering its passage or its section, naming harness, flags, box, and the exe or script -that ran it - or a citation of the passage whose provenance line covers it.** The board cell -states its quant mode and stamps box and engine provenance, so a number can never silently -describe a format nobody serves or a kernel set nobody ships. A figure labeled as a prediction -is not a reading, and this rule does not reach it. - -**A number this module writes down that is not a whole served-turn reading - any figure a run -or a build produced, timing or not - names the harness, the flags and the box that produced -it.** A figure a committed board cell or ruler record produced names the record and row -instead. The naming sits in the figure's own sentence, in a table heading that covers the table's rows, in a -section-level provenance line that covers the paragraphs under it, or in a citation of the -passage whose provenance line covers it. - -**A diff that adds a race to a shipped exe's startup races on synthetic inputs only - never -loading a model, never spawning a child process.** A shipped exe carries no vehicle model and -no harness script, so a model or a child there is a hang or a silent skip. +comparison, the 512-token prefill (pp512) and 128-token decode (tg128) rates - written down as +a measurement rather than as a prediction - is a defect without either a board cell behind it +or a provenance line naming harness, flags, box, and the exe or script that ran it.** The board +cell states its quant mode and stamps box and engine provenance, so a number can never silently +describe a format nobody serves or a kernel set nobody ships. + +**A figure that is not a whole served-turn reading and whose value depends on the box it ran +on - timing or not - names the harness, the flags and the box that produced it.** A figure a +committed board cell or ruler record produced names the record and row instead. + +**A figure that any build, fixture, or command reproduces on any box names that build, +fixture, or command.** + +**A diff that adds a race - a timed run-off between candidate implementations at startup whose +winner sets a knob - to a shipped exe's startup races on synthetic inputs only, never loading a +model, never spawning a child process.** A shipped exe carries no vehicle model - a real model +file a harness run drives - and no harness script, so a model or a child there is a hang or a +silent skip. **A shipped exe's startup race never races a `[tune]` kernel family; a GPU pso twin race, which only sets a runtime knob, is what a first start may do** (`ARCHITECTURE_MEASUREMENT.md` -sec.2.42a). A `[tune]` winner needs a recompiled clone the baked exe does not carry. +sec.2.42a). A `[tune]` winner needs a recompiled clone the shipped exe does not carry. **A diff never adds a confirm - an end-to-end A/B served on a vehicle model in a spawned child - outside `harness/`.** **A diff that pins the kernel backend - `pin_kernel_backend`, `select_kernel_backend`, or -`DASLLAMA_PIN_BACKEND` - pins it before the load it governs, and on a name the box has not -registered refuses or skips rather than running on the default.** The loader repacks weights -into the pinned backend's layout, so a pin set after the load never reaches them, and a -misspelled pin that falls through measures the default backend under the pinned one's name. +`DASLLAMA_PIN_BACKEND` - pins it before the load it governs.** The loader repacks weights into +the pinned backend's layout, so a pin set after the load never reaches them. + +**A pin naming a backend the box has not registered refuses or skips the run rather than +falling through to the default.** A misspelled pin that falls through measures the default +backend under the pinned one's name. diff --git a/modules/dasLLAMA/REVIEW_UPSTREAM.md b/modules/dasLLAMA/REVIEW_UPSTREAM.md index ce10c4994a..9912933b36 100644 --- a/modules/dasLLAMA/REVIEW_UPSTREAM.md +++ b/modules/dasLLAMA/REVIEW_UPSTREAM.md @@ -1,27 +1,27 @@ # dasLLAMA Upstream-Naming Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `ARCHITECTURE_MEASUREMENT.md`. +doc: `ARCHITECTURE_MEASUREMENT.md`. Planned work: `followup_general.md`, `followup_vulkan.md`, +`followup_metal.md`. **Routed from `REVIEW.md`: a diff that checklist routes here applies this list together with it.** The reference build is the third-party engine this module measures itself against - the -checkout `benchmarks/setup_lcpp_ref.das` pins. The text this list binds is checked-in text -under `modules/dasLLAMA/`; a repo-root `plans/` document a dasLLAMA change writes reaches it -through `plans/REVIEW.md`. +checkout `benchmarks/setup_lcpp_ref.das` pins. Reference-build work is locating a site in that +build, patching it, running it, regenerating from it, or measuring against it - planned or +performed. The text this list binds is checked-in text under `modules/dasLLAMA/`; a repo-root +`plans/` document carrying a reference-build name reaches it through `plans/REVIEW.md`. -**Text whose job is not work performed ON the reference build or measured AGAINST it - -locating a site in it, patching it, running it, regenerating from it - describes an upstream -mechanism in our own terms: no "lifted/ported verbatim from", and no name belonging to the -reference build - symbol, header, constant, binary, project or organization - write "the -reference exe" or "upstream" instead.** Mirroring the reference build's behavior in our own -code is not that job. A symbol this module's code calls or holds as a value is its own name, -not attribution. +**A sentence whose job is not reference-build work describes an upstream mechanism in our own +terms: no "lifted/ported verbatim from", and no name belonging to the reference build - symbol, +header, constant, binary, project or organization - write "the reference exe" or "upstream" +instead.** A symbol this module's code calls or holds as a value is its own name, not +attribution. -**Text whose job is work performed on the reference build or measured against it names that -build's names - symbol, header, constant, binary, project or organization - outright, and keeps -the naming inside the sentences doing that job.** The job decides, not the artifact kind; a -knob or command needed to reproduce a compared reading stays licensed inside a planned-work row. +**A figure that describes a part of the reference build rather than a whole run of it names +the kernel or symbol it was read from.** -**A paragraph that mixes a reading of the reference build with a proposal of our own keeps -them in separate sentences.** The proposal sentence names no upstream name; the reading -sentences beside it name what they read, kernel and file included. +**A sentence stating what WE will build or change next carries no reading of the reference +build and no name belonging to it - symbol, header, constant, binary, project or +organization; the reading it sits beside gets its own sentence.** A sentence stating a +reference-build run we will make is reference-build work, not such a proposal, and carries +that run's names, knobs and commands. diff --git a/modules/dasLLAMA/benchmarks/REVIEW.md b/modules/dasLLAMA/benchmarks/REVIEW.md index 444e6fc125..6e5e8df4f3 100644 --- a/modules/dasLLAMA/benchmarks/REVIEW.md +++ b/modules/dasLLAMA/benchmarks/REVIEW.md @@ -1,28 +1,28 @@ # dasLLAMA benchmarks Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `../ARCHITECTURE_MEASUREMENT.md`. Planned work: `../followup_metal.md` for anything about -the Metal backend, `../followup_vulkan.md` for anything about the Vulkan backend, -`../PERF_LEDGER.md` for any other performance followup, `../followup_general.md` for -everything else. +docs: `../ARCHITECTURE_MEASUREMENT.md`, `../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md`. Planned +work: `../followup_metal.md` for anything about the Metal backend, `../followup_vulkan.md` for +anything about the Vulkan backend, `../PERF_LEDGER.md` for any other performance followup, +`../followup_general.md` for everything else. **A GPU kernel timing arm - code that dispatches a kernel to measure it rather than to serve a call - wherever the diff puts it, applies `../REVIEW_GPU_RACE.md` too.** -**A diff that adds or changes an instrument whose timed body runs a forward pass through a -pipeline the dasLLAMA engine selected calls `tune_gate()` (`../performance/profile_common.das`) -before that instrument's first timed rep, or - where the instrument cannot require this -module's performance tree - stamps its rows with the tune state it measured on: the manifest -or class profile whose winners the run compiled against.** An instrument is a file that times -a run and reports a wall-clock time or rate as its result, printed or returned to a caller that -prints it. A pipeline the instrument compiled itself - or a reference tool's own runtime - is -not one the engine selected. Without the gate or the stamp the instrument measures fallback -kernels silently. +**A diff that adds or changes an instrument that dispatches a `[tune]` kernel - one whose body +the engine's tune selection picks, not one the instrument compiled itself and not a reference +tool's own runtime - calls `tune_gate()` (`../performance/profile_common.das`) before that +instrument's first timed rep, or - where the instrument cannot require this module's +performance tree - stamps its rows with the tune manifest (`DAS_TUNE_MANIFEST`) or the class +profile (`../performance/defaults/.tune-defaults.json`) the run compiled +against.** An instrument is a file that times a run and reports a wall-clock time or rate as +its result, printed or returned to a caller that prints it. Without the gate or the stamp the +instrument measures fallback kernels silently. **A diff that adds or changes a race alternates its arms within one process - one timed round -per arm, best-of across rounds.** A race arm times two implementations of the same computation -in one process and compares them; an A/B arm is one run of an instrument under a named lever, -compared against a paired run of the same instrument. An instrument is reviewed arm by arm. +per arm, best-of across rounds.** A race times two implementations of the same computation in +one process and compares them; an arm is one implementation's timed run. An instrument is +reviewed arm by arm. **A diff that adds or changes a race arm that computes a comparable output proves that output on its report line:** an arm whose result is bit-identical to the baseline's prints the @@ -36,33 +36,28 @@ reference.** The baseline arm is the arm running the implementation already in u reference check runs in the same process, on the same output elements the arms are judged on. Two arms can agree and both be wrong; only the reference makes the winner right. -**An arm that compares no output - a knockout, a sweep across formats or stages, an arm whose -output is not comparable - carries the literal token `timing-only` on its report line.** +**An arm that compares no output carries the literal token `timing-only` on its report line.** -**An instrument with no race arm carries the literal text `ATTRIBUTION SWEEP` in its file -header comment, on a line that also names what its arms attribute.** Without the text a reader -takes the sweep's arms for an adoption decision it never made. +**A mode of an instrument that runs no race arm carries the literal text `ATTRIBUTION SWEEP` on +its own line of the file's header comment, naming the mode and what its arms attribute.** A +mode is one selectable run of the file, chosen by its own flag or argument. Without the line a +reader takes a sweep's arms for an adoption decision it never made. **A new instrument that puts its own clock around a served turn is a defect: add a board cell -to `../performance/gen_bench_records.das`, or a `lcpp_bench.das` cell with its own -`../PROFILE.md` section, instead.** A served turn is a whole prefill-plus-decode run. A -second instrument's numbers cannot be compared to any row the board already carries. - -**An out-of-process observer never measures what the benchmark process can measure about -itself - that measurement goes inside the process instead.** An out-of-process observer is a -script that measures a benchmark process from outside. - -**Only the reference cells of `../performance/gen_bench_records.das` - the cells that time a -binary this repository does not build on a board workload - write such a binary's wall time -into `../performance/records/.json` or `../PERF_LEDGER.md`; only `../harness/mtp_ruler.das` -writes one into `../performance/records/mtp/`.** - -**A file holding a third-party wall outside `../performance/records/` and `../PERF_LEDGER.md` -- a reference leg's recovery file, a pinned reference tsv an instrument -reads back - is scratch: untracked, owned by exactly one instrument, re-derivable from a -command written where the owning instrument documents its flags, and never an input to a -board cell.** A tracked or shared copy of a third-party wall becomes a stale baseline nobody -re-derives. +instead.** A served turn is a whole prefill-plus-decode run; a board cell is a timed cell of +the published results board - one `../performance/gen_bench_records.das` spawns, or a manual +`lcpp_bench.das` cell with its own `../PROFILE.md` section. A second instrument's numbers +cannot be compared to any row the board already carries. + +**An out-of-process observer - a script watching a benchmark process from outside - never +measures what that process can measure about itself; that measurement goes inside the process +instead.** + +**A file holding a third-party wall - a wall-clock time measured for a binary this repository +does not build - outside `../performance/records/` and `../PERF_LEDGER.md` is scratch: +untracked, owned by exactly one instrument, re-derivable from a command in that instrument's +header comment, and never an input to a board cell.** A tracked or shared copy of a third-party +wall becomes a stale baseline nobody re-derives. **A diff that adds or changes an instrument that prints the difference of two walls also prints both of those walls on that report line.** A plain elapsed-time row - one clock pair, @@ -70,26 +65,24 @@ no attribution across stages - is not a difference. **A diff that changes what a board cell times ships before/after rows for each affected cell and corpus - or withdraws the affected rows and names the withdrawal and its reason in the PR -body.** A board cell is a timed cell of the published results board: one -`../performance/gen_bench_records.das` spawns, or a manual `lcpp_bench.das` cell with its own -`../PROFILE.md` section. What a cell times changes when a change inside its timed body, to its -input corpus, or to the pinned reference build (`DEFAULT_REF_SHA` in `setup_lcpp_ref.das`, or -anything else deciding which reference binary or environment the run measures) moves the -measured quantity; a change outside the timed body - a flag, a require, the submit path - does -not. The re-mint or withdrawal lands in -`../performance/records/.json`, the file the affected rows live in. - -**A diff that adds an instrument, or changes how one reports or exits, makes it exit non-zero -on a run that ends with zero result rows - wrong flags, failed load, a device that declines.** -A run that matched nothing and reported success leaves a sidecar or a record untouched, and -its caller cannot tell. - -**A diff that adds an A/B arm - one of an instrument's timed runs, distinguished by a named -lever set to a value the paired run does not use, off/on or graded - or changes how an arm -reports or exits, makes that instrument exit non-zero when the lever does not change what the -run executes; an instrument that runs that check before the arm prints a warning naming the -inert lever instead.** A lever is the flag or environment switch that names the arm; a lever -that silently no-ops prints a 1.00x row nobody can tell from a real tie. +body.** What a cell times changes when a change inside its timed body, to its input corpus, or +to the pinned reference build (`DEFAULT_REF_SHA` in `setup_lcpp_ref.das`, or anything else +deciding which reference binary or environment the run measures) moves the measured quantity; a +change outside the timed body - a flag, a require, the submit path - does not. The new rows or +the withdrawal land in `../performance/records/.json`, the file the affected rows live in. + +**A diff that adds an instrument, or changes how one reports or exits, makes every mode whose +purpose is to report result rows exit non-zero on a run that reports none - wrong flags, failed +load, a device that declines.** A result row is a row carrying a time or a rate. A run that +matched nothing and reported success leaves a sidecar or a record untouched, and its caller +cannot tell. + +**A diff that adds an A/B arm, or changes how an arm reports or exits, makes that instrument +exit non-zero when the lever does not change what the run executes - or, when the instrument +runs that check before the arm, print a warning naming the inert lever.** An A/B arm is one of +an instrument's timed runs, distinguished by a named lever - the flag or environment switch +that names the arm - set to a value the paired run does not use, off/on or graded. A lever that +silently no-ops prints a 1.00x row nobody can tell from a real tie. **A diff that adds or changes an A/B arm of an instrument over a prompt corpus makes that arm report one row per prompt, never one aggregate ratio alone.** Prompts differ in how much the diff --git a/modules/dasLLAMA/benchmarks/matmul/bench_metal_kq_race.das b/modules/dasLLAMA/benchmarks/matmul/bench_metal_kq_race.das index 68727cb9b1..89123199ce 100644 --- a/modules/dasLLAMA/benchmarks/matmul/bench_metal_kq_race.das +++ b/modules/dasLLAMA/benchmarks/matmul/bench_metal_kq_race.das @@ -295,7 +295,7 @@ def private build_oracle(fmt; n, d, nsb : int; kq, ks : array; xf : array delete wrow } -[arch(at="../../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private race_cell(queue, dev; f : KqRaceFmt; tier, shname : string; n, d : int; arms : array; args : Args; ref_tab : table) : int { let is_mm = tier == "mm" diff --git a/modules/dasLLAMA/benchmarks/matmul/kq_kernel_bench.das b/modules/dasLLAMA/benchmarks/matmul/kq_kernel_bench.das index b3f156c0eb..03b6fb44d0 100644 --- a/modules/dasLLAMA/benchmarks/matmul/kq_kernel_bench.das +++ b/modules/dasLLAMA/benchmarks/matmul/kq_kernel_bench.das @@ -203,7 +203,7 @@ def arena_base(var p : Planes) : int64 { } } -[arch(at = "../../ARCHITECTURE_MEASUREMENT.md#cpu-kernel-bench-fixture")] +[arch(at = "../../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#cpu-kernel-bench-fixture")] def fill_planes(var p : Planes; sh : PlaneShape; n, d, ntok : int64) { var end = 0l p.gemvKq = next_plane(end, d * sh.gemvW_row, 0) @@ -307,7 +307,7 @@ def team_gemv_mx4(fn : GemvMx4Fn; var yp : float?; wn, we : uint8 const?; xqp : //! the team arm's interleaved timing: the block dispatches row `row` the engine's way and every lane adds its //! call time to laneP[slot]; the row keeps the wall (best, median) and the lanes' fastest and slowest sums -[arch(at = "../../ARCHITECTURE_MEASUREMENT.md#cpu-kernel-bench-fixture")] +[arch(at = "../../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#cpu-kernel-bench-fixture")] def time_rows_team(var stats : array; rounds : int; blk : block<(row : int; var laneP : int64?) : void>) { var laneUs : array laneUs |> resize(get_total_hw_jobs() + 1) @@ -384,7 +384,7 @@ def perm_wanted(spec, suffix : string) : bool { } //! interleaved timing: round r calls every selected row once before round r + 1 starts -[arch(at = "../../ARCHITECTURE_MEASUREMENT.md#cpu-kernel-bench-fixture")] +[arch(at = "../../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#cpu-kernel-bench-fixture")] def time_rows(var stats : array; rounds : int; blk : block<(row : int) : void>) { for (row in range(length(stats))) { stats[row].us |> reserve(rounds) diff --git a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das index 9d5b924df1..eb969e1997 100644 --- a/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das +++ b/modules/dasLLAMA/dasllama/dasllama_gpu_resident.das @@ -1702,8 +1702,9 @@ def moe_gpu_upload_resident(t : Model) { // nolint:STYLE037,STYLE038 — seven } walk_say(dense_skips, "dense rail") } - // shared-expert triples — DASLLAMA_GPU_SHEXP=1 opt-in. Dense (every token, every layer, one - // fixed weight set) so residency carries no miss term; q8 only (the repack registers them fmt 0). + // shared-expert triples — on with the tier (DASLLAMA_GPU_SHEXP overrides). Dense (every token, + // every layer, one fixed weight set) so residency carries no miss term; q8 only (the repack + // registers them fmt 0). vulkan_bake_role(VkBakeRole.shexp) if (gpu_want_shexp() && t.config.n_ff_shexp > 0l && !t.config.moe_dense_shexp) { let nsh = t.config.n_ff_shexp diff --git a/modules/dasLLAMA/dasllama/dasllama_metal_kernels.das b/modules/dasLLAMA/dasllama/dasllama_metal_kernels.das index 5660cae479..986a985496 100644 --- a/modules/dasLLAMA/dasllama/dasllama_metal_kernels.das +++ b/modules/dasLLAMA/dasllama/dasllama_metal_kernels.das @@ -4852,7 +4852,7 @@ let KQ_ROWS_RACE_FORMATS <- [ ] //! needs the decode PSOs - the caller runs `metal_decode_init` first -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private race_kq_rows(queue : MetalCommandQueue?; fmt : KqFmt; qu_per_sb : int; sform : KqRaceScaleForm) : MetalTensorRaceResult { var res = MetalTensorRaceResult(family = "kq_rows_{fmt}", winner = "", base_ms = -1.0lf, twin_ms = -1.0lf, note = "") let n = 2048 @@ -4875,7 +4875,7 @@ def private race_kq_rows(queue : MetalCommandQueue?; fmt : KqFmt; qu_per_sb : in return res } -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private race_kq_k4_form(queue : MetalCommandQueue?) : MetalTensorRaceResult { var res = MetalTensorRaceResult(family = "kq_mvb2_k4_r2", winner = "", base_ms = -1.0lf, twin_ms = -1.0lf, note = "") let n = 2048 @@ -4947,7 +4947,7 @@ def private kq_race_verdict(var res : MetalTensorRaceResult; by_a, by_b : MetalB //! Decode-side tuner entry (the runtime.metal_tensor crown): the batch q8 GEMM families and the //! K-quant small-batch form per format. -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def metal_tensor_race_decode : array { var results : array results |> reserve(8 + length(KQ_ROWS_RACE_FORMATS)) @@ -5004,7 +5004,7 @@ def metal_tensor_race_decode : array { //! A GEMV f4-slab twin race over the split-scale fixture: base (constant-table, rows/4) vs //! the f4-slab twin. Winner "tensor" = the twin takes the family crown. -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private race_gemv_f4_twin(dev : MetalDevice?; queue : MetalCommandQueue?; family : string; // nolint:STYLE038 - one-shot A/B race harness; buffers + PSOs stay live to the release tail base_src, base_entry : string; base_fm : bool; twin_src, twin_entry : string; twin_fm : bool; twin_tgmem : uint64; @@ -9252,7 +9252,7 @@ class template MetalKqGemvIq2xxsFam { class MetalKqGemvIq2xxs : MetalKqGemvIq2xxsFam { } -// the F4 twin serves only under the box's serving-confirm crown (ARCHITECTURE_MEASUREMENT.md sec.2.21) +// the F4 twin serves only under the box's serving-confirm crown (ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md sec.2.21) [metal_dispatch(name = "enc_kq_iq2xxs_f4_c", pso = "g_pso_kq_iq2xxs_f4", tgmem = "MetalKqGemvIq2xxsF4_metal_kq_gemv_iq2xxs_msl_tgmem", tg = 64, grid = "rows/4", params = "rows : int64, n : int64")] class MetalKqGemvIq2xxsF4 : MetalKqGemvIq2xxsFam { override F4 = true diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das index f5544cfc90..2cedc57b42 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_classes.das @@ -617,7 +617,7 @@ class DnBaGemm { @workgroup xs : float[528] // 16 positions x 32 k, stride 33 (bank-spread) @workgroup ws : float[528] // 16 rows x 32 k, stride 33 - [spirv_kernel(local_size_x = 256, name = "dn_ba_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain")] + [spirv_kernel(local_size_x = 256, name = "dn_ba_cls_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block")] def run { let tid = gl_LocalInvocationID.x let ne = 2u * pa.nvh @@ -4682,6 +4682,8 @@ class template KqCm2BatchT : MoeCmBase { @ssbo @binding = 1 @role = "weight" ws : array // the scale plane (q8: f16 per block; kq: 5 words per superblock) @ssbo @binding = 3 @role = "alias" xf16 : array // f16 activation plane (the tensor loads' view) @ssbo @binding = 3 @role = "alias" xf16w4 : array // the same plane as 16-byte words: the KHR arm's activation stage + @ssbo @binding = 0 @role = "alias" @template_gate = KHR @readonly wq4 : array + @ssbo @binding = 0 @role = "alias" @template_gate = KHR @readonly wqw : array @ssbo @binding = 5 y : array @workgroup wg_blk0 : uint // the region's block base, staged for the decode @template_constant BN : uint = 256u // the token column @@ -4911,7 +4913,11 @@ class template KqCm2BatchT : MoeCmBase { coopmatStoreTensor(accw, y, ybase, tlo, t0, BN, m0, 128u, tv) } - def khr_tile(wblk0, row0, cnt, k0, k1, ybase, xt, wt : uint) { // nolint:STYLE037,STYLE038 — the staged k loop and its fast/edge store are one register set + //! a KHR stamp's format writes weight row `blk`'s 16 values e0 .. e0 + 16 into khr_ao[sbase .. sbase + 8) as f16 pairs, read from the plane's words; the cm2 stamps never call it + def abstract khr_stage16(blk, e0, sbase : uint) : void + + [arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#khr-mm-kq-tile")] + def khr_tile(wblk0, row0, cnt, k0, k1, ybase, xt, wt : uint) { // nolint:STYLE038 — the staged k loop and its fast/edge store are one register set static_if (KHR) { // the gated staging arrays exist on the KHR instantiations alone let t0 = row0 + xt * BN let m0 = wt * 128u @@ -4926,33 +4932,15 @@ class template KqCm2BatchT : MoeCmBase { let sbase = srow * KHR_STRIDE + khalf * KHR_KHALF_WORDS let wrow_in_d = wrow < pa.d let trow_in_cnt = trow < cnt - var acc0 : coopmatAcc_f32_16x16 - var acc1 : coopmatAcc_f32_16x16 - var acc2 : coopmatAcc_f32_16x16 - var acc3 : coopmatAcc_f32_16x16 - var acc4 : coopmatAcc_f32_16x16 - var acc5 : coopmatAcc_f32_16x16 - var acc6 : coopmatAcc_f32_16x16 - var acc7 : coopmatAcc_f32_16x16 + let wrow0 = (sg & 1u) * 64u //! the subgroup's first weight row of the tile's 128 + let trow0 = (sg >> 1u) * 32u //! the subgroup's first token row of the tile's 128 + var acc : coopmatAcc_f16_16x16[8] //! [r * 2 + c]: weight fragment r against token fragment c var k = k0 while (k < k1) { let kk = k + skh if (wrow_in_d && kk < k1) { let bcol = kk / BLKW - let e0 = kk - bcol * BLKW - static_if (DECV4) { - for (j in range(4)) { - let v4 = float4(decode_v4(wq[wblk0 + wrow * nbk + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 4u))) - khr_ao[sbase + uint(j) * 2u] = packHalf2x16(v4.xy) - khr_ao[sbase + uint(j) * 2u + 1u] = packHalf2x16(v4.zw) - } - } else { - for (j in range(int(KHR_KHALF_WORDS))) { - let lo = decode(wq[wblk0 + wrow * nbk + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 2u)) - let hi = decode(wq[wblk0 + wrow * nbk + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 2u + 1u)) - khr_ao[sbase + uint(j)] = packHalf2x16(float2(float(lo), float(hi))) - } - } + khr_stage16(wblk0 + wrow * nbk + bcol, kk - bcol * BLKW, sbase) } else { for (j in range(int(KHR_KHALF_WORDS))) { khr_ao[sbase + uint(j)] = 0u @@ -4972,74 +4960,52 @@ class template KqCm2BatchT : MoeCmBase { } } barrier() - for (ks in range(2)) { + for [unroll_full] (ks in range(2)) { let ko = uint(ks) * 8u - var a : coopmatA_f16_16x16 - coopmatLoad(a, khr_ao, int(sg * 16u * KHR_STRIDE + ko), int(KHR_STRIDE), 0) - var b : coopmatB_f16_16x16 - coopmatLoad(b, khr_b, int(ko), int(KHR_STRIDE), 1) - acc0 = coopmatMulAdd(a, b, acc0) - coopmatLoad(b, khr_b, int(16u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc1 = coopmatMulAdd(a, b, acc1) - coopmatLoad(b, khr_b, int(32u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc2 = coopmatMulAdd(a, b, acc2) - coopmatLoad(b, khr_b, int(48u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc3 = coopmatMulAdd(a, b, acc3) - coopmatLoad(b, khr_b, int(64u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc4 = coopmatMulAdd(a, b, acc4) - coopmatLoad(b, khr_b, int(80u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc5 = coopmatMulAdd(a, b, acc5) - coopmatLoad(b, khr_b, int(96u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc6 = coopmatMulAdd(a, b, acc6) - coopmatLoad(b, khr_b, int(112u * KHR_STRIDE + ko), int(KHR_STRIDE), 1) - acc7 = coopmatMulAdd(a, b, acc7) + var a : coopmatA_f16_16x16[4] + for [unroll_full] (r in range(4)) { + coopmatLoad(a[r], khr_ao, int((wrow0 + uint(r) * 16u) * KHR_STRIDE + ko), int(KHR_STRIDE), 0) + } + for [unroll_full] (c in range(2)) { + var b : coopmatB_f16_16x16 + coopmatLoad(b, khr_b, int((trow0 + uint(c) * 16u) * KHR_STRIDE + ko), int(KHR_STRIDE), 1) + for [unroll_full] (r in range(4)) { + acc[r * 2 + c] = coopmatMulAdd(a[r], b, acc[r * 2 + c]) + } + } } barrier() k += 32u } - let mrow0 = m0 + sg * 16u let tile_whole = m0 + 128u <= pa.d && xt * BN + BN <= cnt if (tile_whole) { - let yb = ybase + t0 * pa.d + mrow0 - coopmatStore(acc0, y, int(yb), int(pa.d), 1) - coopmatStore(acc1, y, int(yb + 16u * pa.d), int(pa.d), 1) - coopmatStore(acc2, y, int(yb + 32u * pa.d), int(pa.d), 1) - coopmatStore(acc3, y, int(yb + 48u * pa.d), int(pa.d), 1) - coopmatStore(acc4, y, int(yb + 64u * pa.d), int(pa.d), 1) - coopmatStore(acc5, y, int(yb + 80u * pa.d), int(pa.d), 1) - coopmatStore(acc6, y, int(yb + 96u * pa.d), int(pa.d), 1) - coopmatStore(acc7, y, int(yb + 112u * pa.d), int(pa.d), 1) + let yb = ybase + (t0 + trow0) * pa.d + m0 + wrow0 + var accf32 : coopmatAcc_f32_16x16 + for [unroll_full] (r in range(4)) { + for [unroll_full] (c in range(2)) { + coopmatConvert(accf32, acc[r * 2 + c]) + coopmatStore(accf32, y, int(yb + uint(c) * 16u * pa.d + uint(r) * 16u), int(pa.d), 1) + } + } } else { let lane = gl_SubgroupInvocationID let ob = sg * 256u - for (fi in range(8)) { - if (fi == 0) { - coopmatStore(acc0, khr_ao, int(ob), 16, 0) - } elif (fi == 1) { - coopmatStore(acc1, khr_ao, int(ob), 16, 0) - } elif (fi == 2) { - coopmatStore(acc2, khr_ao, int(ob), 16, 0) - } elif (fi == 3) { - coopmatStore(acc3, khr_ao, int(ob), 16, 0) - } elif (fi == 4) { - coopmatStore(acc4, khr_ao, int(ob), 16, 0) - } elif (fi == 5) { - coopmatStore(acc5, khr_ao, int(ob), 16, 0) - } elif (fi == 6) { - coopmatStore(acc6, khr_ao, int(ob), 16, 0) - } else { - coopmatStore(acc7, khr_ao, int(ob), 16, 0) - } - barrier() - for (e in range(8)) { - let idx = lane * 8u + uint(e) - let mr = mrow0 + idx / 16u - let tc = xt * BN + uint(fi) * 16u + idx % 16u - if (mr < pa.d && tc < cnt) { - y[ybase + (row0 + tc) * pa.d + mr] = uint_bits_to_float(khr_ao[ob + idx]) + var accf32 : coopmatAcc_f32_16x16 + for [unroll_full] (r in range(4)) { + for [unroll_full] (c in range(2)) { + coopmatConvert(accf32, acc[r * 2 + c]) + coopmatStore(accf32, khr_ao, int(ob), 16, 0) + barrier() + for (e in range(8)) { + let idx = lane * 8u + uint(e) + let mr = m0 + wrow0 + uint(r) * 16u + idx / 16u + let tc = xt * BN + trow0 + uint(c) * 16u + idx % 16u + if (mr < pa.d && tc < cnt) { + y[ybase + (row0 + tc) * pa.d + mr] = uint_bits_to_float(khr_ao[ob + idx]) + } } + barrier() } - barrier() } } } @@ -5066,6 +5032,23 @@ class template K4Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let sh = ((e0 >> 4u) & 1u) * 4u //! the low or the high nibbles of sub-block g's 16 bytes + let w4 = wq4[blk * 8u + g] + let srow = blk * 5u + let dm = unpackHalf2x16(ws[srow]) + let ds = dm.x * float((ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + let dmn = dm.y * float((ws[srow + 3u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + for [unroll_full] (c in range(4)) { + let w = w4[c] >> sh + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(w & 0xFu) - dmn, ds * float((w >> 8u) & 0xFu) - dmn)) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float((w >> 16u) & 0xFu) - dmn, ds * float((w >> 24u) & 0xFu) - dmn)) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK4Blk; bc, cib : uint2) : float16 { let g = cib.y >> 5u @@ -5103,6 +5086,31 @@ class template K6Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let bu = e0 >> 5u + let hh = (e0 >> 4u) & 1u + let lw4 = wq4[blk * 12u + bu] //! the low-nibble bytes bu * 16 + j + let hw4 = wq4[blk * 12u + 8u + (bu >> 2u) * 2u + hh] //! the high-2-bit bytes (bu >> 2) * 32 + hh * 16 + j + let ls = hh * 4u + let hs = (bu & 3u) * 2u + let srow = blk * 5u + let sidx = e0 >> 4u + let sc = int(ws[srow + (sidx >> 2u)] << ((3u - (sidx & 3u)) * 8u)) >> 24 + let ds = unpackHalf2x16(ws[srow + 4u]).x * float(sc) + for [unroll_full] (c in range(4)) { + let lw = lw4[c] >> ls + let hw = hw4[c] >> hs + let q0 = int((lw & 0xFu) | ((hw & 3u) << 4u)) - 32 + let q1 = int(((lw >> 8u) & 0xFu) | (((hw >> 8u) & 3u) << 4u)) - 32 + let q2 = int(((lw >> 16u) & 0xFu) | (((hw >> 16u) & 3u) << 4u)) - 32 + let q3 = int(((lw >> 24u) & 0xFu) | (((hw >> 24u) & 3u) << 4u)) - 32 + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(q0), ds * float(q1))) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float(q2), ds * float(q3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK6Blk; bc, cib : uint2) : float16 { let e = cib.y @@ -5153,6 +5161,30 @@ class template K5Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let h = (e0 >> 4u) & 1u + let sh = h * 4u + let w4 = wq4[blk * 10u + g] + let hw = wq4[blk * 10u + 8u + (g >> 2u)][int(g & 3u)] >> sh //! sub-block g's four 5th-bit bytes; byte c holds j = 4c .. 4c + 3 at bit (j & 3) + h * 4 + let srow = blk * 5u + let dm = unpackHalf2x16(ws[srow]) + let ds = dm.x * float((ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + let dmn = dm.y * float((ws[srow + 3u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + for [unroll_full] (c in range(4)) { + let w = w4[c] >> sh + let hb = hw >> (uint(c) * 8u) + let q0 = (w & 0xFu) | ((hb & 1u) << 4u) + let q1 = ((w >> 8u) & 0xFu) | (((hb >> 1u) & 1u) << 4u) + let q2 = ((w >> 16u) & 0xFu) | (((hb >> 2u) & 1u) << 4u) + let q3 = ((w >> 24u) & 0xFu) | (((hb >> 3u) & 1u) << 4u) + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(q0) - dmn, ds * float(q1) - dmn)) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float(q2) - dmn, ds * float(q3) - dmn)) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK5Blk; bc, cib : uint2) : float16 { let g = cib.y >> 5u @@ -5201,6 +5233,21 @@ class template Q40Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let sh = ((e0 >> 4u) & 1u) * 4u + let w4 = wq4[blk * 8u + g] + let dp = unpackHalf2x16(ws[blk * 5u + (g >> 1u)]) + let d = (g & 1u) == 0u ? dp.x : dp.y + for [unroll_full] (c in range(4)) { + let w = w4[c] >> sh + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(d * (float(w & 0xFu) - 8.0), d * (float((w >> 8u) & 0xFu) - 8.0))) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(d * (float((w >> 16u) & 0xFu) - 8.0), d * (float((w >> 24u) & 0xFu) - 8.0))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK4Blk; bc, cib : uint2) : float16 { let g = cib.y >> 5u @@ -5237,6 +5284,22 @@ class template Iq4xsCm2T : KqCm2BatchT { override IQLUT = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let sh = ((e0 >> 4u) & 1u) * 4u + let w4 = wq4[blk * 8u + g] + let srow = blk * 2u + let w0 = ws[srow] + let ds = unpackHalf2x16(w0).x * float(iq4xs_sc(w0, ws[srow + 1u], g)) + for [unroll_full] (c in range(4)) { + let w = w4[c] >> sh + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(iq4lut[int(w & 0xFu)]), ds * float(iq4lut[int((w >> 8u) & 0xFu)]))) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float(iq4lut[int((w >> 16u) & 0xFu)]), ds * float(iq4lut[int((w >> 24u) & 0xFu)]))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK4Blk; bc, cib : uint2) : float16 { let g = cib.y >> 5u @@ -5275,6 +5338,30 @@ class template K3Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let l0 = e0 & 31u + let qw4 = wq4[blk * 6u + (g >> 2u) * 2u + (l0 >> 4u)] //! the qs bytes (g >> 2) * 32 + l + let hw4 = wq4[blk * 6u + 4u + (l0 >> 4u)] //! the hmask bytes l + let qs2 = (g & 3u) * 2u + let srow = blk * 5u + let sidx = e0 >> 4u + let sc = int(ws[srow + (sidx >> 2u)] << ((3u - (sidx & 3u)) * 8u)) >> 24 + let ds = unpackHalf2x16(ws[srow + 4u]).x * float(sc) + for [unroll_full] (c in range(4)) { + let qw = qw4[c] >> qs2 + let hw = hw4[c] >> g + let q0 = int(qw & 3u) - ((hw & 1u) == 0u ? 4 : 0) + let q1 = int((qw >> 8u) & 3u) - (((hw >> 8u) & 1u) == 0u ? 4 : 0) + let q2 = int((qw >> 16u) & 3u) - (((hw >> 16u) & 1u) == 0u ? 4 : 0) + let q3 = int((qw >> 24u) & 3u) - (((hw >> 24u) & 1u) == 0u ? 4 : 0) + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(q0), ds * float(q1))) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float(q2), ds * float(q3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK3Blk; bc, cib : uint2) : float16 { let e = cib.y @@ -5321,6 +5408,24 @@ class template K2Cm2T : KqCm2BatchT { typedef ST = uint override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 4u + let qw4 = wq4[blk * 4u + (g >> 3u) * 2u + (g & 1u)] //! the qs bytes (g >> 3) * 32 + (g & 1) * 16 + l + let qs2 = ((g >> 1u) & 3u) * 2u + let srow = blk * 5u + let dm = unpackHalf2x16(ws[srow]) + let pb = (ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu + let ds = dm.x * float(pb & 15u) + let dmn = dm.y * float(pb >> 4u) + for [unroll_full] (c in range(4)) { + let qw = qw4[c] >> qs2 + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(ds * float(qw & 3u) - dmn, ds * float((qw >> 8u) & 3u) - dmn)) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(ds * float((qw >> 16u) & 3u) - dmn, ds * float((qw >> 24u) & 3u) - dmn)) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK2Blk; bc, cib : uint2) : float16 { let e = cib.y @@ -5361,6 +5466,31 @@ class template Iq3sCm2T : KqCm2BatchT { override IQ3GRID = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let r0 = e0 & 31u + let bw = blk * 26u + let qsw = wqw[bw + g * 2u + (r0 >> 4u)] //! the four grid-index bytes g * 8 + (r >> 2) + let qh = (wqw[bw + 16u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu //! the ninth-bit byte g + let sgw = wqw[bw + 18u + g] >> ((r0 >> 3u) * 8u) //! the two sign bytes g * 4 + (r >> 3) in the low half + let srow = blk * 2u + let ds = unpackHalf2x16(ws[srow]).x * float(grid_sc(ws[srow + 1u], g)) + for [unroll_full] (i in range(4)) { + let wj = (r0 >> 2u) + uint(i) + let qb = (qsw >> (uint(i) * 8u)) & 0xFFu + let gw = iq3s_gridc[qb | ((qh << (8u - wj)) & 256u)] + let sgn = sgw >> ((uint(i) >> 1u) * 8u + (uint(i) & 1u) * 4u) + let g0 = float(gw & 0xFFu) + let g1 = float((gw >> 8u) & 0xFFu) + let g2 = float((gw >> 16u) & 0xFFu) + let g3 = float(gw >> 24u) + khr_ao[sbase + uint(i) * 2u] = packHalf2x16(float2(ds * ((sgn & 1u) != 0u ? -g0 : g0), ds * (((sgn >> 1u) & 1u) != 0u ? -g1 : g1))) + khr_ao[sbase + uint(i) * 2u + 1u] = packHalf2x16(float2(ds * (((sgn >> 2u) & 1u) != 0u ? -g2 : g2), ds * (((sgn >> 3u) & 1u) != 0u ? -g3 : g3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkIq3sBlk; bc, cib : uint2) : float16 { let e = cib.y @@ -5413,6 +5543,33 @@ class template Iq3xxsCm2T : KqCm2BatchT { override IQ3XGRID = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let r0 = e0 & 31u + let bw = blk * 24u + let qsw = wqw[bw + g * 2u + (r0 >> 4u)] //! the four grid-index bytes g * 8 + (r >> 2) + let aux = wqw[bw + 16u + g] //! sub-block g's aux32: four 7-bit ksigns indices and the ls nibble + let srow = blk * 2u + let ds = unpackHalf2x16(ws[srow]).x * float(grid_sc(ws[srow + 1u], g)) + for [unroll_full] (i in range(4)) { + let l = (r0 >> 3u) + (uint(i) >> 1u) + let sidx = (aux >> (7u * l)) & 127u + var tt = sidx ^ (sidx >> 4u) + tt = tt ^ (tt >> 2u) + tt = tt ^ (tt >> 1u) + let sgn = (sidx | ((tt & 1u) << 7u)) >> ((uint(i) & 1u) * 4u) + let gw = iq3x_gridc[(qsw >> (uint(i) * 8u)) & 0xFFu] + let g0 = float(gw & 0xFFu) + let g1 = float((gw >> 8u) & 0xFFu) + let g2 = float((gw >> 16u) & 0xFFu) + let g3 = float(gw >> 24u) + khr_ao[sbase + uint(i) * 2u] = packHalf2x16(float2(ds * ((sgn & 1u) != 0u ? -g0 : g0), ds * (((sgn >> 1u) & 1u) != 0u ? -g1 : g1))) + khr_ao[sbase + uint(i) * 2u + 1u] = packHalf2x16(float2(ds * (((sgn >> 2u) & 1u) != 0u ? -g2 : g2), ds * (((sgn >> 3u) & 1u) != 0u ? -g3 : g3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkIq3xxsBlk; bc, cib : uint2) : float16 { let e = cib.y @@ -5479,6 +5636,21 @@ class template Iq4nlCm2T : KqCm2BatchT { override IQLUT = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let sh = ((e0 >> 4u) & 1u) * 4u + let w4 = wq4[blk * 8u + g] + let dp = unpackHalf2x16(ws[blk * 5u + (g >> 1u)]) + let d = (g & 1u) == 0u ? dp.x : dp.y + for [unroll_full] (c in range(4)) { + let w = w4[c] >> sh + khr_ao[sbase + uint(c) * 2u] = packHalf2x16(float2(d * float(iq4lut[int(w & 0xFu)]), d * float(iq4lut[int((w >> 8u) & 0xFu)]))) + khr_ao[sbase + uint(c) * 2u + 1u] = packHalf2x16(float2(d * float(iq4lut[int((w >> 16u) & 0xFu)]), d * float(iq4lut[int((w >> 24u) & 0xFu)]))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkK4Blk; bc, cib : uint2) : float16 { let g = cib.y >> 5u @@ -5515,6 +5687,34 @@ class template Iq2sCm2T : KqCm2BatchT { override IQ2SGRID = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let r0 = e0 & 31u + let l0 = r0 >> 3u + let bw = blk * 18u + let qsw = wqw[bw + g] >> (l0 * 8u) //! the grid-index bytes g * 4 + l, this run's two in the low half + let sgw = wqw[bw + 8u + g] >> (l0 * 8u) //! the sign bytes 32 + g * 4 + l + let qh = (wqw[bw + 16u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu //! the qh byte g + let srow = blk * 5u + let si = g * 2u + (r0 >> 4u) + let ds = unpackHalf2x16(ws[srow]).x * float((ws[srow + 1u + (si >> 2u)] >> ((si & 3u) * 8u)) & 0xFFu) + for [unroll_full] (i in range(4)) { + let li = uint(i) >> 1u + let l = l0 + li + let qb = (qsw >> (li * 8u)) & 0xFFu + let sgn = (sgw >> (li * 8u + (uint(i) & 1u) * 4u)) & 0xFu + let gw = iq2s_gridc[(qb | ((qh << (8u - 2u * l)) & 0x300u)) * 2u + (uint(i) & 1u)] + let g0 = float(gw & 0xFFu) + let g1 = float((gw >> 8u) & 0xFFu) + let g2 = float((gw >> 16u) & 0xFFu) + let g3 = float(gw >> 24u) + khr_ao[sbase + uint(i) * 2u] = packHalf2x16(float2(ds * ((sgn & 1u) != 0u ? -g0 : g0), ds * (((sgn >> 1u) & 1u) != 0u ? -g1 : g1))) + khr_ao[sbase + uint(i) * 2u + 1u] = packHalf2x16(float2(ds * (((sgn >> 2u) & 1u) != 0u ? -g2 : g2), ds * (((sgn >> 3u) & 1u) != 0u ? -g3 : g3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkIq2sBlk; bc, cib : uint2) : float16 { let e = cib.y @@ -5568,6 +5768,30 @@ class template Iq2xsCm2T : KqCm2BatchT { override IQ2XSGRID = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let qw = wqw[blk * 16u + (e0 >> 4u)] //! two u16 qs words: the run's 8-value windows + let srow = blk * 5u + let si = e0 >> 4u + let ds = unpackHalf2x16(ws[srow]).x * float((ws[srow + 1u + (si >> 2u)] >> ((si & 3u) * 8u)) & 0xFFu) + for [unroll_full] (i in range(4)) { + let w16 = (qw >> ((uint(i) >> 1u) * 16u)) & 0xFFFFu + let sidx = w16 >> 9u + var tt = sidx ^ (sidx >> 4u) + tt = tt ^ (tt >> 2u) + tt = tt ^ (tt >> 1u) + let sgn = (sidx | ((tt & 1u) << 7u)) >> ((uint(i) & 1u) * 4u) + let gw = iq2xs_gridc[(w16 & 511u) * 2u + (uint(i) & 1u)] + let g0 = float(gw & 0xFFu) + let g1 = float((gw >> 8u) & 0xFFu) + let g2 = float((gw >> 16u) & 0xFFu) + let g3 = float(gw >> 24u) + khr_ao[sbase + uint(i) * 2u] = packHalf2x16(float2(ds * ((sgn & 1u) != 0u ? -g0 : g0), ds * (((sgn >> 1u) & 1u) != 0u ? -g1 : g1))) + khr_ao[sbase + uint(i) * 2u + 1u] = packHalf2x16(float2(ds * (((sgn >> 2u) & 1u) != 0u ? -g2 : g2), ds * (((sgn >> 3u) & 1u) != 0u ? -g3 : g3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkIq2xsBlk; bc, cib : uint2) : float16 { let e = cib.y @@ -5621,6 +5845,35 @@ class template Iq2xxsCm2T : KqCm2BatchT { override IQ2XXSGRID = true override DECV4 = true + def override khr_stage16(blk, e0, sbase : uint) : void { + static_if (KHR) { + let g = e0 >> 5u + let l0 = (e0 & 31u) >> 3u + let bw = blk * 16u + let qsw = wqw[bw + g * 2u] >> (l0 * 8u) //! sub-block g's four grid-index bytes, this run's two in the low half + let aux = wqw[bw + g * 2u + 1u] //! its aux32: four 7-bit ksigns indices and the ls nibble + let srow = blk * 2u + let ds = unpackHalf2x16(ws[srow]).x * float(grid_sc(ws[srow + 1u], g)) + for [unroll_full] (i in range(4)) { + let li = uint(i) >> 1u + let l = l0 + li + let qb = (qsw >> (li * 8u)) & 0xFFu + let sidx = (aux >> (7u * l)) & 127u + var tt = sidx ^ (sidx >> 4u) + tt = tt ^ (tt >> 2u) + tt = tt ^ (tt >> 1u) + let sgn = (sidx | ((tt & 1u) << 7u)) >> ((uint(i) & 1u) * 4u) + let gw = iq2xxs_gridc[qb * 2u + (uint(i) & 1u)] + let g0 = float(gw & 0xFFu) + let g1 = float((gw >> 8u) & 0xFFu) + let g2 = float((gw >> 16u) & 0xFFu) + let g3 = float(gw >> 24u) + khr_ao[sbase + uint(i) * 2u] = packHalf2x16(float2(ds * ((sgn & 1u) != 0u ? -g0 : g0), ds * (((sgn >> 1u) & 1u) != 0u ? -g1 : g1))) + khr_ao[sbase + uint(i) * 2u + 1u] = packHalf2x16(float2(ds * (((sgn >> 2u) & 1u) != 0u ? -g2 : g2), ds * (((sgn >> 3u) & 1u) != 0u ? -g3 : g3))) + } + } + } + [spirv_decode, arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-decode-16bit-lanes")] def decode(blk : VkIq2xxsBlk; bc, cib : uint2) : float16 { let e = cib.y diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das index 3ce76d82d2..0e852a6cd1 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_common.das @@ -135,7 +135,7 @@ def cm2_splitk_env : int64 { } -[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] +[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#khr-mm-kq-tile")] def khr_kq_tile_on : bool => g_gpu.has_coopmat && g_gpu.subgroup_size == 32l [arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] @@ -240,7 +240,7 @@ let STREAM_RESERVE = 1_100_000_000l // LEGACY fallback stream-slot carve — us let DN_MAX_DS = 128l //! The sequential scan's workgroup count for a head count and state width -[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain")] +[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block")] def dn_scan_wgs(nvh, ds : int64) : int64 { let cpg = 8l * (32l / (ds / 16l)) return nvh * ((ds + cpg - 1l) / cpg) @@ -2728,6 +2728,21 @@ def public vk_kernel_coverage() : table { return <- out } +//! One kernel's dispatch count under its census name - a class kernel's is its SPIR-V name, `_spv`. +//! A name no kernel seeded panics, so a misspelt key cannot read as a zero count. +def public vk_kernel_coverage_of(kernel : string) : int64 { + if (!key_exists(g_kernel_access, kernel)) { + panic("dasLLAMA vulkan tier: no seeded kernel named '{kernel}' in the coverage census") + } + var n = 0l + for (k, c in keys(g_vk_kcov), values(g_vk_kcov)) { + if ((g_pipe_names?[k] ?? "") == kernel) { + n += c + } + } + return n +} + def public vk_kernel_coverage_reset { g_vk_kcov |> clear() } diff --git a/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das b/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das index 01027011ef..3c990a3deb 100644 --- a/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das +++ b/modules/dasLLAMA/dasllama/dasllama_vulkan_prefill.das @@ -38,8 +38,9 @@ let private PF_ROLES = 16l // quant_xb q k v rope attn quant_at wo addr_ffn qu var private @scratch g_pf_ids_stage : array +//! whether the prefill's f16 feed admits a plane of format `f` on the armed coopmat mode - the tests read the same answer the driver acts on [arch(at="../ARCHITECTURE_RUNTIME.md#activation-scale-lattice")] -def private pf_f16_feed(f : int) : bool { +def public pf_f16_feed(f : int) : bool { if (g_gpu.coopmat_mode == COOPMAT_CM2) { return f == int(KqFmt.q8) || kq_sb(f) } @@ -51,16 +52,17 @@ def private pf_qkv6(l : int64) : bool => (pf_f16_feed(g_rd.layers[l].fq) && pf_f16_feed(g_rd.layers[l].fk) && pf_f16_feed(g_rd.layers[l].fv)) [arch(at="../ARCHITECTURE_RUNTIME.md#activation-scale-lattice")] def private pf_gu6(l : int64) : bool => pf_f16_feed(g_rd.layers[l].f1) && pf_f16_feed(g_rd.layers[l].f3) -[arch(at="../ARCHITECTURE_RUNTIME.md#activation-scale-lattice")] -def private pf_dn6(l : int64) : bool => (pf_f16_feed(g_rd.layers[l].fqkv) - && pf_f16_feed(g_rd.layers[l].fz) && pf_f16_feed(g_rd.layers[l].fout)) +[arch(at="../ARCHITECTURE_RUNTIME.md#activation-scale-lattice"), arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block")] +def private pf_dnx6(l : int64) : bool => pf_f16_feed(g_rd.layers[l].fqkv) && pf_f16_feed(g_rd.layers[l].fz) +[arch(at="../ARCHITECTURE_RUNTIME.md#activation-scale-lattice"), arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block")] +def private pf_dno6(l : int64) : bool => pf_f16_feed(g_rd.layers[l].fout) [arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain")] def private pf_qkv_feed_fused(l : int64) : bool => !g_rd.layers[l].recurrent && g_env_vulkan.vk_fuse && (pf_qkv6(l) || !kq_sb(g_rd.layers[l].fq)) let private PF_CHUNK_MAX = 8l // the overlap ramp doubles 1,2,4 then holds here; the cmd ring is sized from it -[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] +[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#khr-mm-kq-tile")] def khr_cls_ensure(fmt : int) : bool { if (fmt == int(KqFmt.k4)) return ensure_kq_batch_k4_khr_cls() if (fmt == int(KqFmt.k5)) return ensure_kq_batch_k5_khr_cls() @@ -78,7 +80,7 @@ def khr_cls_ensure(fmt : int) : bool { return ensure_kq_batch_iq2xxs_khr_cls() } -[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] +[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#khr-mm-kq-tile")] def khr_cls_set(fmt : int; bufs : uint64 const[5]; sizes : int64 const[5]; gbits : uint const[5]) : VkDescriptorSet { if (fmt == int(KqFmt.k4)) return set_kq_batch_k4_khr_cls(bufs, sizes, gbits) if (fmt == int(KqFmt.k5)) return set_kq_batch_k5_khr_cls(bufs, sizes, gbits) @@ -96,7 +98,7 @@ def khr_cls_set(fmt : int; bufs : uint64 const[5]; sizes : int64 const[5]; gbits return set_kq_batch_iq2xxs_khr_cls(bufs, sizes, gbits) } -[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] +[arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#khr-mm-kq-tile")] def khr_cls_enc(fmt : int; raw : VkCommandBuffer; var h : VkHaz; var s : VkDescriptorSet; var pc : BatchArgs; groups : int64) { if (fmt == int(KqFmt.k4)) { enc_kq_batch_k4_khr_cls(raw, h, s, pc, groups) @@ -494,7 +496,7 @@ def private pf_gemm_enc(raw : VkCommandBuffer; var h : VkHaz; idx : int; fmt : i } //! one recurrent layer's prefill window sets; the o feed sets (`pf_dnof_set` / `pf_dnrq_set`) are shared by every recurrent layer -[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain")] +[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block")] def private pf_dn_layer_sets(l, dim, cd, di : int64) { var L & = unsafe(g_rd.layers[l]) let np = g_rd.pf_np @@ -520,7 +522,7 @@ def private pf_dn_layer_sets(l, dim, cd, di : int64) { fixed_array(g_rd.dn_ba_bytes, np * dim * 4l, DND_SMALLS_BYTES), fixed_array(0u, VHZ_XB, VHZ_DNSM)) } - if (pf_dn6(l)) { + if (pf_dnx6(l)) { L.pf_dn_cvt_set = set_f16cvt_cls(fixed_array(g_rd.pf_xb, g_rd.pf_xf), fixed_array(np * dim * 4l, np * dim * 2l), fixed_array(VHZ_XB, VHZ_XQ)) } @@ -537,7 +539,7 @@ def private pf_dn_zero(raw : VkCommandBuffer; var h : VkHaz) { } } -[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-window-chain"), arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#hybrid-token-command")] +[arch(at="../ARCHITECTURE_GPU_VULKAN.md#vk-prefill-dn-block"), arch(at="../ARCHITECTURE_GPU_VULKAN_DECODE.md#hybrid-token-command")] def private rd_pf_recurrent(raw : VkCommandBuffer; var h : VkHaz; l : int64; b : int; wlen : int64; last : bool) { var L & = unsafe(g_rd.layers[l]) let dim = g_rd.dim @@ -551,37 +553,37 @@ def private rd_pf_recurrent(raw : VkCommandBuffer; var h : VkHaz; l : int64; b : let taps = dconv - 1l let np = g_rd.pf_np let eps = g_rd.eps - let dn6 = pf_dn6(l) - let dnk = !dn6 && kq_sb(L.fqkv) // off the f16 feed a K-quant qkv/z pair reads the Q8_K activation form, the attention head's fqk twin - if (dn6) { + let x6 = pf_dnx6(l) + let xq8k = !x6 && kq_sb(L.fqkv) + if (x6) { var pcc0 = ActArgs(nelem = uint(wlen * dim), gelu = 0u, nblk = 0u) enc_f16cvt_cls(raw, h, L.pf_dn_cvt_set, pcc0, (wlen * dim / 4l + 255l) / 256l) } - if (dnk) { + if (xq8k) { var pc0 = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 256l)) enc_cls_q8k_rq(raw, h, g_rd.pf_sets[b + 0], pc0, (wlen * dim / 8l + 255l) / 256l) - } elif (!dn6 || !g_rd.dn_ba_f32) { + } elif (!x6 || !g_rd.dn_ba_f32) { var pc0 = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 32l)) enc_cls_dn_rq(raw, h, g_rd.pf_sets[b + 0], pc0, (wlen * dim / 4l + 255l) / 256l) } pfq_ts(raw) - let xqb = dn6 ? g_rd.pf_xf : g_rd.pf_xq - let xsb = dn6 ? g_rd.dummy : g_rd.pf_xs - let xqn = dn6 ? np * dim * 2l : np * dim - let xsn = dn6 ? 256l : np * (dim / 32l) * 4l + let xqb = x6 ? g_rd.pf_xf : g_rd.pf_xq + let xsb = x6 ? g_rd.dummy : g_rd.pf_xs + let xqn = x6 ? np * dim * 2l : np * dim + let xsn = x6 ? 256l : np * (dim / 32l) * 4l let pqkv = arena_planes(L.fqkv, L.bqkv) pf_gemm_enc(raw, h, b + 1, L.fqkv, pqkv.wq, pqkv.ws, xqb, xsb, g_rd.pf_dnqkv, - pqkv.wqb, pqkv.wsb, xqn, xsn, np * cd * 4l, VHZ_XQ, VHZ_DNP, dim, cd, L.bqkv, wlen, dn6, cm2_tiles(di, wlen)) + pqkv.wqb, pqkv.wsb, xqn, xsn, np * cd * 4l, VHZ_XQ, VHZ_DNP, dim, cd, L.bqkv, wlen, x6, cm2_tiles(di, wlen)) let pz = arena_planes(L.fz, L.bz) pf_gemm_enc(raw, h, b + 2, L.fz, pz.wq, pz.ws, xqb, xsb, g_rd.pf_dnz, - pz.wqb, pz.wsb, xqn, xsn, np * di * 4l, VHZ_XQ, VHZ_DNZ, dim, di, L.bz, wlen, dn6, cm2_tiles(cd, wlen)) + pz.wqb, pz.wsb, xqn, xsn, np * di * 4l, VHZ_XQ, VHZ_DNZ, dim, di, L.bz, wlen, x6, cm2_tiles(cd, wlen)) if (g_rd.dn_ba_f32) { var pcb = DnBaArgs(npos = uint(wlen), dim = uint(dim), nvh = uint(nvh), woff = uint(L.ba_off), xstride = uint(dim), obase_b = uint(DN_SM_BETA), obase_g = uint(DN_SM_G), ostride = uint(nvh)) enc_dn_ba_cls(raw, h, L.pf_dn_ba_set, pcb, dn_ba_wgs(wlen, nvh)) pfq_ts(raw) } else { - if (dnk) { // the qkv/z GEMMs read the rows as Q8_K; the q8 beta/alpha GEMMs read them as Q8_0 + if (xq8k) { // the qkv/z GEMMs read the rows as Q8_K; the q8 beta/alpha GEMMs read them as Q8_0 var pcq = RqArgs(inbase = 0u, nblk = uint(wlen * dim / 32l)) enc_cls_dn_rq(raw, h, g_rd.pf_sets[b + 0], pcq, (wlen * dim / 4l + 255l) / 256l) } @@ -621,7 +623,8 @@ def private rd_pf_recurrent(raw : VkCommandBuffer; var h : VkHaz; l : int64; b : cmd_copy_range(raw, g_rd.pf_dnqkv, (wlen - taps) * cd * 4l, L.dn.smalls_dev, DN_SM_HIST * 4l, taps * cd * 4l) } pfq_ts(raw) - if (dn6) { + let o6 = pf_dno6(l) + if (o6) { var pco = ActArgs(nelem = uint(wlen * di), gelu = 0u, nblk = 0u) enc_f16cvt_cls(raw, h, g_rd.pf_dnof_set, pco, (wlen * di / 4l + 255l) / 256l) } elif (kq_sb(L.fout)) { @@ -633,8 +636,8 @@ def private rd_pf_recurrent(raw : VkCommandBuffer; var h : VkHaz; l : int64; b : } pfq_ts(raw) let pout = arena_planes(L.fout, L.bout) - pf_gemm_enc(raw, h, b + 7, L.fout, pout.wq, pout.ws, dn6 ? g_rd.pf_dnof : g_gpu.hq_dev, dn6 ? g_rd.dummy : g_gpu.hs_dev, g_rd.pf_xb2, - pout.wqb, pout.wsb, dn6 ? np * di * 2l : BATCH_HQ_BYTES, dn6 ? 256l : BATCH_HS_BYTES, np * dim * 4l, VHZ_HQ, VHZ_XB2, di, dim, L.bout, wlen, dn6) + pf_gemm_enc(raw, h, b + 7, L.fout, pout.wq, pout.ws, o6 ? g_rd.pf_dnof : g_gpu.hq_dev, o6 ? g_rd.dummy : g_gpu.hs_dev, g_rd.pf_xb2, + pout.wqb, pout.wsb, o6 ? np * di * 2l : BATCH_HQ_BYTES, o6 ? 256l : BATCH_HS_BYTES, np * dim * 4l, VHZ_HQ, VHZ_XB2, di, dim, L.bout, wlen, o6) } // the prefill profile's stamps per layer: the attention head's roles (+ qkn), or the recurrent head's diff --git a/modules/dasLLAMA/followup_general.md b/modules/dasLLAMA/followup_general.md index a77738163c..b33d1121fa 100644 --- a/modules/dasLLAMA/followup_general.md +++ b/modules/dasLLAMA/followup_general.md @@ -1381,3 +1381,27 @@ read). The disk was full because the comparison bench mints an image per model and per flavor by default - 37 debug-jit images, 293 GB, in one day's walk; the bench under `--for-debug-purposes` should run `DASLLAMA_IMAGE=0` itself, since its flavor serves no rig. + +121. **`tests/CLAUDE.md` splits into a rule file and an architecture doc.** The file is 915 + lines: about fifteen binding rules (the runner rule, the iteration loop, the poison rules, the + image-rail ban, the tier and family filters, log discipline) and then some 550 lines of + present-tense per-file census under "The per-PR suites - model-free and stocked". The census + is architecture content under a CLAUDE.md name, outside LINT027's reach. Done = the census + moves to `tests/ARCHITECTURE.md` in sections (anchors where a cell cites one), `CLAUDE.md` + keeps the discipline and routes to it, and `tests/REVIEW.md`'s cites of the CLAUDE.md + headings follow the move. Ruled 2026-09-08: the next follow-up arc, not the KHR tile PR. + +122. **New MoE architectures for the files that fully fit a 16 GB card.** Found 2026-09-08 by the + MoE fit survey (every MoE GGUF at or under 14.0 GB, from the Hugging Face file listings): the + loader takes five MoE families with a fully fitting file - gpt-oss, qwen2moe, qwen3moe with + qwen3vlmoe, qwen35moe, gemma4 - and about sixteen more families fit for one new arch each. By + downloads: `deepseek2` first (DeepSeek-Coder-V2-Lite Q5_K_M 11.85 GB, Moonlight-16B-A3B, + Kimi-VL-A3B; MLA attention, two shared experts), `cohere2moe` second (North Mini Code 1.0 + Q3_K_S 13.57 GB; 128 experts top-8, sigmoid router), then `bailingmoe2` / `bailingmoe3` + (Ling-mini-2.0, Ling-3.0-tiny), `ernie4_5-moe`, `lfm2moe`, `afmoe` (Trinity-Mini), `olmoe`, + `granitemoe` / `granitehybrid`, `smallthinker`, `kimi-linear`, `laguna`; `phimoe` and Mixtral + fit only at 2-bit. The loader also takes no Q4_1, IQ1_S, IQ1_M, TQ1_0 or TQ2_0 plane, so every + `UD-IQ1_*` and `UD-TQ1_0` file is out whatever its arch. Ruled 2026-09-08: after the 0.6.4 + release, and after the Vulkan tier covers every carrier the module already serves + (`followup_vulkan.md` item 43) - no new family before either. Done = one arc per arch, + `deepseek2` first, each with its board rows. diff --git a/modules/dasLLAMA/followup_vulkan.md b/modules/dasLLAMA/followup_vulkan.md index 313e8d8cd8..50b8961c88 100644 --- a/modules/dasLLAMA/followup_vulkan.md +++ b/modules/dasLLAMA/followup_vulkan.md @@ -40,7 +40,7 @@ Ordered roughly by user-visible value; re-rank against zen2 measurements before upstream; was 11.1 on the per-op rails). The PREFILL half followed: the window chain carries the recurrent block (conv + chunked scan on the layer's device state), gated attention and partial rotary on the batch kernels, and hands the device state to the session for the decode - (`ARCHITECTURE_GPU_VULKAN.md` sec.2.2j, `_DECODE.md` sec.2.2v; gate + (`ARCHITECTURE_GPU_VULKAN.md` sec.2.2ad, `_DECODE.md` sec.2.2v; gate `tests/test_gpu_resident_hybrid.das`, one- and two-window cells). Every figure in this item: `benchmarks/lcpp_bench.das -m Qwen3.5-9B-MTP-UD-Q5_K_XL.gguf -r 3` (`-p 512 -n 128`) through `bin/Release/daslang.exe -jit` under `DASLLAMA_GPU=1 DASLLAMA_ALLOW_UNTUNED=1 @@ -703,7 +703,7 @@ module) is independent and can land any time - it is pure structure. 40. **The 2026-09-05 hang hunt's residue (the Khronos layer under GPU-assisted validation, safe mode, robustness OFF, on the 9B pp512 window).** The hang itself was the emitter's eager - `?:` (`modules/dasSpirv/ARCHITECTURE.md` sec.3, "Operand laziness follows the language"): + `?:` (`modules/dasSpirv/ARCHITECTURE.md` sec.3.4, "Operand laziness follows the language"): `qk_rms_cls` read `krows` at every q-row offset, 12.6 MB past a 4 MiB binding, and faulted the card once the overshoot left mapped VRAM (512 rows dead, 128 rows fine, the 0.8B fine, the kernel-unit cells fine - a CPU oracle cannot see an out-of-range READ). Three things the @@ -752,65 +752,148 @@ module) is independent and can land any time - it is pure structure. 42. **End of the 27B arc: the cm2-disabled sweep.** Every 27B row on the board is a cm2 row (NV_cooperative_matrix2 on the 5060 Ti). Boris (2026-09-07): measure each file with the tensor tiles off, both engines - ours `DASLLAMA_COOPMAT=mm` (KHR coopmat mul_mm) and - `DASLLAMA_COOPMAT=sdot4` + `DASLLAMA_VK_FA=0` (no cooperative matrix at all), llama.cpp - `GGML_VK_DISABLE_COOPMAT2=1` and `GGML_VK_DISABLE_COOPMAT=1` - to learn what cm2 buys, - where we stand against llama.cpp on the same arm, and whether every format's non-cm2 tile - (the `KqBatch*` sdot4 tier serves every kq format) decodes and prefills correctly - the - bench's sanity argmax and logit must match the cm2 run's. The real proof of a lower target - is a run on one (a Turing or Ampere card, an RDNA card): the knob arms exercise the kernels, - not the device-creation path a card without the extension takes. Done = the four arms on - UD-IQ4_XS, UD-Q3_K_XL and i1-IQ3_S in the PR body. The row from a non-5060 card is the - real-hardware pass, a follow-up arc after every family works here (Boris 2026-09-07): rented + `DASLLAMA_COOPMAT=sdot4` + `DASLLAMA_VK_FA=0` (no cooperative matrix at all), the + reference exe `GGML_VK_DISABLE_COOPMAT2=1` and `GGML_VK_DISABLE_COOPMAT=1` - to learn + what cm2 buys, where we stand against llama.cpp on the same arm, and whether every + format's non-cm2 tile (the `KqBatch*` sdot4 tier serves every kq format) decodes and + prefills correctly - the bench's sanity argmax and logit must match the cm2 run's. The + real proof of a lower target is a run on one (a Turing or Ampere card, an RDNA card): the + knob arms exercise the kernels, not the device-creation path a card without the extension + takes. Done = the four arms on UD-IQ4_XS, UD-Q3_K_XL and i1-IQ3_S in the PR body. The row + from a non-5060 card is the real-hardware pass, a follow-up arc after every family works + here (Boris 2026-09-07): rented boxes - AWS g4dn (T4, the KHR arm on NVIDIA's driver) and g6e/p4d (48-80 GB, the fully resident 27B Q8 and 35B MoE), an RDNA3/4 card from a GPU marketplace (AWS's AMD parts predate cooperative matrix) - each with a written plan of what to run where. FIRST PAIR (UD-IQ4_XS, 5060 Ti, pin 14000, 2026-09-07): decode is flat on every arm (ours 23.4 on all three, llama.cpp 24.1) and the sanity argmax holds (pp 13, tg 5709). Prefill: - cm2 860.8 vs 813.7; KHR coopmat 221.2 vs 677.5 (0.33x); no coopmat 215.7 vs 312.6 (0.69x). - The cause of the 0.33x: the kq formats have no KHR-coopmat tile - `DASLLAMA_COOPMAT=mm` - serves only the q8 planes on the mul_mm L-tile and every kq plane on the `KqBatch*` sdot4 - tile, so mm and sdot4 read the same on a kq-only file. A card without NV_coopmat2 (every - AMD and Intel part, the GTX line) prefilled a 27B at a third of the reference exe's rate. - The fix was the format decode on the mul_mm L-tile, the way it moved onto the cm2 template. - THE KHR kq TILE LANDED (2026-09-07 evening, `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2l): the - cm2 template's KHR arm, every kq format, 0 of 89600 off on all thirteen; the 4B Q4_K_M mm row - 1564 -> 2378 (0.56x llama.cpp's KHR 4221), the 27B UD-IQ4_XS 221 -> 395 (0.59x of 675), decode - unchanged. Its three measured steps: the decode call on the plane element (the block copy ran - the tile at a third), 16-byte activation loads, a 32-deep k step (the 64-deep one blew the - 49152 B workgroup cap on the iq2 grids). THE SLAB (the same evening): the probe's copy of the + cm2 860.8 vs 813.7; KHR coopmat 221.2 vs the reference exe's `mul_mm.comp` 677.5 (0.33x); + no coopmat 215.7 vs 312.6 (0.69x). The cause of the 0.33x: the kq formats have no + KHR-coopmat tile - `DASLLAMA_COOPMAT=mm` serves only the q8 planes on the mul_mm L-tile + and every kq plane on the `KqBatch*` sdot4 tile, so mm and sdot4 read the same on a + kq-only file. A card without NV_coopmat2 (every AMD and Intel part, the GTX line) + prefilled a 27B at a third of the reference exe's rate. The fix was the format decode on + the mul_mm L-tile, the way it moved onto the cm2 template. + THE KHR kq TILE LANDED (2026-09-07 evening, `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2ae): the + cm2 template's KHR arm, every kq format, 0 of 89600 off on all thirteen; the 4B Q4_K_M mm + row 1564 -> 2378 (0.56x of the reference exe's `mul_mm.comp` 4221), the 27B UD-IQ4_XS + 221 -> 395 (0.59x of the same tile's 675), decode unchanged. Its three measured steps: the + decode call on the plane element (the block copy ran the tile at a third), 16-byte + activation loads, a 32-deep k step (the 64-deep one blew the 49152 B workgroup cap on the + iq2 grids). THE SLAB (the same evening): the probe's copy of the tile body read 33 TFLOP/s where the shipped class read 25.6, and the difference was the 8 KB - edge-store slab the shipped class carried beside its two 10 KB staging arrays - the probe's - `slab` arm, the copy plus that one array touched on a path no dispatch takes, reads 28.5: - 8 KB more shared memory per workgroup is one workgroup fewer per SM. The staging arrays are + edge-store slab the shipped class carried beside its two 10 KB staging arrays - a probe arm + of that evening (since retired), the copy plus that one array touched on a path no dispatch + takes, read 28.5: 8 KB more shared memory per workgroup is one workgroup fewer per SM (the + two-array footprint is now the tile's own, `ARCHITECTURE_GPU_VULKAN_GEMM.md` sec.2.2ae). The staging arrays are now `uint` f16 pairs (the activation words stored as they arrive, no unpack) and the edge tile's f32 fragments bounce through the weight array once the k loop is done, the row guards hoisted out of the loop: the shipped class 25.6 -> 32.1 / 23.9 -> 31.4 / 25.6 -> 32.2 on the gate / down / q shapes, the kernel suite 108 of 108 with its edge-tile cells, the hybrid file - 10 of 10 on the KHR arm, the 4B Q4_K_M mm row 2378 -> 3051 (0.72x of llama.cpp's 4221; the - probe's khr row 32.6 / 31.8 / 32.6 on a box holding 1.3 GB for other processes). The 27B - UD-IQ4_XS row's re-measure is owed: at 1.2 GB held by other processes the resident plan's - KV room fell to 147 MB, under the 2048-position minimum, and the driver declined - the row - needs the box the board's 395.2 was taken on (about 650 MB held). - WHERE THE REST IS: llama.cpp's non-cm2 path does not - run K-quants on tensor cores at all - `quantize_y` (ggml-vulkan.cpp, needs integer dot and no - coopmat2) routes them to the integer MMQ tile (`matmul_q4_k_q8_1`: 128 threads, 128 x 128 x 32, - Q8_1 activations with a per-32 (d, sum), a 4 x 32 f32 register block per thread over - dotPacked4x8, the sub-block (d*sc, dmin*m) folded per 32-k block), measured 43-46 TFLOP/s-eq - on the 9728/4096 x 2560 shapes and 27-31 on the 80-workgroup ones (its Vulkan build under - `GGML_VK_PERF_LOGGER=1`, the 4B prefill on the 5060 Ti) against our f16 KHR tile's 32 on the - large shapes (`harness/vk_gemm_probe.das -- khrx`, same card, after the slab fix above; 25 before it). The road to parity on the KHR arm is that tile on our side - a new body with - per-format int8 word decodes (the sdot4 `KqBatch` tier's `stage_w` is the same decode at a - 32 x 32 superblock shape) - and it lifts every integer-dot card with or without coopmat. - THE FIRST PROTOTYPES (2026-09-07, `harness/vk_gemm_probe.das -- mmqx`): three register-block - shapes over the sdot4 k4 staging (64 x 32 at 4 x 8 per thread staged per superblock; the - reference exe's one-block stage at 64 x 128, 4 x 16; that stage at 4 x 4 as named scalars) - all read 8.8-9.2 TFLOP/s against the shipped tile's 12.2, and the named-scalar twin with - CONSTANTS in place of the plane reads reads 11.0 - the inner loop caps the shape, not the - memory: per block a thread issues 24 shared loads and 160 sdot4 for 16 outputs, the reference - exe's 4 x 32 block 40 loads for 1024. Two prerequisites before the next prototype: (a) the - SPIR-V emitter unrolls `for [unroll] (i in range(N))` at emission (today it emits a hinted - loop, and a fixed-array local stays a Function-storage variable indexed by the loop counter - - `plans/shader_emitter_followups.md`), so a 128-accumulator block can be written as an array - with constant indices the driver promotes to registers; (b) a hardware profile of the ceiling - twin (Nsight, not another blind bisect) to see whether the sdot4 issue rate, the shared-load - rate or the barrier stalls at two workgroups per SM hold it at 11. + 10 of 10 on the KHR arm, the 4B Q4_K_M mm row 2378 -> 3051 (0.72x of the reference exe's + `mul_mm.comp` 4221; the probe's khr row 32.6 / 31.8 / 32.6 on a box holding 1.3 GB for + other processes). The 27B UD-IQ4_XS row's re-measure is owed: at 1.2 GB held by other + processes the resident plan's KV room fell to 147 MB, under the 2048-position minimum, + and the driver declined - the row needs the box the board's 395.2 was taken on (about + 650 MB held). + WHERE THE REST IS (the 2026-09-07 reading of `quantize_y` was wrong; corrected 2026-09-08 from + `ggml_vk_load_shaders` and `ggml_vk_get_mul_mat_mat_pipeline`): on a device with KHR cooperative + matrix llama.cpp creates no integer tile - `CREATE_MMQ` sits only in the shader loader's two + scalar arms, the q8_1 pipeline set is empty, and every mat-mat product falls to the f16 + KHR-coopmat `mul_mm.comp` tile with the format decoded into shared memory (`load_a_to_shmem`), + the 4221 and 675 t/s bars included. That tile on the NVIDIA card (`warptile_mmq` for a quantized + A, the `aligned` variant on every shape of ours, split-k off): 128 threads, a 128 x 128 x 32 block, + four subgroups as 2 x 2 tiles of 64 x 64 over 16 x 16 x 16 fragments (sixteen accumulators per + subgroup, one B stage shared by all four), f16 accumulators (`coopmat_acc_f16_support` with + default precision picks the f16acc pipeline: 64 accumulator registers per lane), two 128-row + stages at a 20-word stride - our stride - plus a 2 KB store stage, 22 KB in all, and a direct + `coopMatStore` of a whole tile; the per-shape rates its perf logger (`GGML_VK_PERF_LOGGER=1`) + reads (43-46 TFLOP/s-eq on the 9728/4096 x 2560 shapes, 27-31 on the 80-workgroup ones, the 4B + prefill on the 5060 Ti) are that tile's, against our f16 KHR tile's 32 on the large shapes + (`harness/vk_gemm_probe.das -- khrx`, same card, after the slab fix above). The same hardware + path, the same shared footprint and barrier cadence. The reading of the difference (mul_mm.comp): + their four subgroups each load 4 A + 4 B fragments per 16 MMAs over one shared B stage, where + our eight each reloaded all 8 B fragments against 1 A - 144 fragment loads per workgroup per k + step against their 64 to 96 - and their accumulators are f16, 64 registers per lane against our + f32's 128. Our road to parity: the subgroup tiling and the accumulator width. The reading of the + coopmat-less arm (ggml-vulkan.cpp, `GGML_VK_DISABLE_COOPMAT=1`): its mat-mat runs the integer + MMQ tile `matmul_q4_k_q8_1` - 128 threads, 128 x 128 x 32 over four staged k blocks, Q8_1 + activations with a per-32 (d, d x sum), a 4 x 32 f32 register block per thread over + dotPacked4x8, the sub-block (d*sc, dmin*m) folded per 32-k block; no IQ format has one - whose + rates on this card are unmeasured: the perf logger names no pipeline, and + `GGML_VK_PIPELINE_STATS=q8_1` is what proves the tile was reached. Our sdot4 mode's lever is an + integer tile of that shape. + THE FIRST INTEGER-TILE PROTOTYPES (2026-09-07, `harness/vk_gemm_probe.das -- mmqx`, the sdot4 + arm's lever): three register-block shapes over the sdot4 k4 staging (64 x 32 at 4 x 8 per + thread staged per superblock; the reference exe's one-block stage at 64 x 128, 4 x 16; that + stage at 4 x 4 as named scalars) all read 8.8-9.2 TFLOP/s against the shipped tile's 12.2, and + the named-scalar twin with CONSTANTS in place of the plane reads read 11.0 - the inner loop + caps the shape, not the memory: per block a thread issues 24 shared loads and 160 sdot4 for 16 + outputs, the reference exe's `matmul_q4_k_q8_1` 4 x 32 block 40 loads for 1024. Two + prerequisites before the next prototype of either tile: (a) the SPIR-V emitter unrolls + `for [unroll_full] (i in range(N))` at emission (`plans/shader_emitter_followups.md` item 2), + so a fragment or accumulator block written as a fixed array chains constant indices the + driver promotes to registers - the coopmat tile's sixteen accumulators need it as much as + the integer tile's 128; (b) a hardware profile (Nsight) of our KHR tile beside the + reference exe's on one shape, not another blind bisect. + THE TILE (2026-09-08, `harness/vk_gemm_probe.das -- khrx` and Nsight GPU Trace on the RTX 5060 Ti): + the profile of the shipped KHR tile read the load-store pipe at 81% of its peak - the four-wide + decode callback's eight 16-bit lane loads and three scale words per 16 values - and the register + file at 99%, two workgroups of eight warps per SM; neither the f16 accumulators alone (32.7 + against 33.4 TFLOP/s on the 4B gate shape) nor the two-by-four subgroup tiling alone (35.5) + moved it, and our tile in the reference exe's geometry ran at 22 (about 245 registers per + lane, one workgroup per SM). The stage went first: every format's `khr_stage16` reads its + 16-value run as one or two words of the quant plane (37.6), then f16 accumulators (54.0), + then the two-by-four tiling (54.9) - the shipped class 53.8 / 59.8 / 57.9 on the gate / + down / q shapes against 32.6 / 31.7 / 32.5 before it and the reference exe's + `mul_mm.comp` at 43-46. THE PER-LEVER + ROWS (`-- khrx` at the arc's tip, the RTX 5060 Ti, the 4B gate / down / q shapes, TFLOP/s, + each arm the shipped tile with one lever moved back): the shipped class 54.2 / 59.5 / 57.6 and + its probe copy 58.8 / 63.9 / 60.3 (the class carries the region and split-k arithmetic the copy + omits; the copy is bit-exact against it); staging through the four-wide decode callback in + place of the words - eight 16-bit lane loads and three scale words per 16 values - 35.6 / 36.0 + / 35.2 (bit-exact); f32 accumulators 42.7 / 44.1 / 43.6 (within 0.01 of the f16 class); the + 16-row strip tiling 54.5 / 57.4 / 55.7; the reference exe's `mul_mm.comp` geometry - four subgroups of 64 x + 64 in a 128-thread workgroup, sixteen accumulators each - 53.5 / 57.1 / 57.5, no better; a + constant fill in place of the weight stage 65.6 / 66.3 / 64.6, the loop's ceiling; the sdot4 + kq tile 12.3 / 12.2 / 12.2. The shipped class holds 0 of 768 sampled outputs off the kernel + cell's bar against the k4 CPU oracle on every shape. The partial-window row (the gate shape at + 300 tokens, where the last token tile takes the edge store; only the two clamping tiles run + there): the shipped class 45.0 (54.2 at 512 - 57.6 per computed row, so the padded rows and not + the bounce are the cost), the sdot4 tile 11.6 (12.3). The kernel suite holds 108 of 108 with + its thirteen KHR cells at 0 of 89600 off, every other kernel's + SPIR-V is byte-identical, and the board's KHR rows read: the 4B Q4_K_M 3051 -> 4764 t/s + (1.13x of the reference exe's `mul_mm.comp` 4221), the 27B UD-IQ4_XS 395 -> 741 under a + 14000 MB pin (1.10x of the same tile's 675), decode unchanged (116.8 and 23.2). Beside it + the same day, the cm2 k4 l tile on the same shapes and the same un-barriered submit + (`-- k4`, the `k4lnb` rows): 59.8 / 66.0 / 62.0 - the KHR tile reads 90% of the + NV_coopmat2 tile's rate, and the 4B's end-to-end ratio (4764 against 5150) reads 92%; the + cm2 tile's own levers are the driver's (decode-in-load, its tiling), so the word stage is + not a cm2 change. THE ONE ROW UNDER PARITY (the 9B UD-Q5_K_XL at 1788 against the + reference exe's `mul_mm.comp` 2098) was a gate, not the tile: the recurrent head decided + its f16 feed per LAYER over qkv, z and out together, and + that file's out plane is Q8_0 - a format the KHR arm's f16 feed does not admit - so every + recurrent layer's K-quant qkv and z fell to the sdot4 tile (the per-role profile, + `DASLLAMA_GPU_PROF=1`, its `vk_rdpf dn` lines: qkv 72.9 ms and z 35.1 ms per window at 11 + TFLOP/s against 19.7 / 10.4 on the Q5_K_M file, whose out plane is Q5_K). The x feed (qkv, z) + and the o feed (out) are now decided apart, each by the planes that read it (`pf_dnx6` / + `pf_dno6`; the attention and FFN heads already split so): qkv 19.7 ms, z 10.5, the row 2531 / + 56.66 (1.21x / 0.98x). The witness is the hybrid parity file's mixed twin + (`Qwen3.5-0.8B-Q4_K_M-q8out.gguf`, a Q8_0 out plane beside K-quant qkv/z) holding, on a + coopmat feed, that the prefill never requantized the block input to Q8_K. Still open under this + item: the no-coopmat arm - ours `DASLLAMA_COOPMAT=sdot4 DASLLAMA_VK_FA=0` against the reference + build's coopmat-less arm (the knob in the 2026-09-07 row above), where the prototypes above are + the road - the wave64 twin of the KHR tile, and the real-hardware pass. +43. **The Vulkan tier covers every carrier the module already serves.** Ruled 2026-09-08 (Boris, + after the MoE fit survey, `followup_general.md` item 122): after the 0.6.4 release and before + any new family, every carrier dasLLAMA serves today gets its Vulkan arm at parity - the + existing families, and the vision, audio and TTS towers, whose GPU drivers are Metal today. + The known gaps on the family side are this ledger's items 3 (the quant KV codecs), 4 (batched + decode), 5 (the speculative round), 6 (the mx4 and q51 device kernels behind gpt-oss and + gemma-4-26B), the gemma4 pre/post-norm attention decline, and the fully-resident MoE chain: + the 5060 Ti holds Qwen1.5-MoE Q6_K, Qwen3-30B-A3B UD-Q3_K_XL, Qwen3.6-35B-A3B UD-IQ3_S and + gemma-4-26B-A4B UD-IQ4_XS whole (12.8 to 13.8 GB), the resident driver declines every MoE by + construction, and the per-op tier reads 0.14x prefill / 0.75x decode against the reference + engine on the one such row measured (the arc board's row 20, both engines fully resident). + Done = every family and tower row on the board with a Vulkan column at parity, and the + serving census (`coverage-vk`) with no carrier the tier declines. diff --git a/modules/dasLLAMA/harness/REVIEW.md b/modules/dasLLAMA/harness/REVIEW.md index 9471d879be..c86c775210 100644 --- a/modules/dasLLAMA/harness/REVIEW.md +++ b/modules/dasLLAMA/harness/REVIEW.md @@ -3,14 +3,13 @@ **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture doc: `../ARCHITECTURE_MEASUREMENT.md`. Planned work: `../followup_metal.md` for anything about the Metal backend, `../followup_vulkan.md` for anything about the Vulkan backend, -`../PERF_LEDGER.md` for any other performance followup, `../followup_general.md` for -everything else. +`../PERF_LEDGER.md` for any other performance followup, `../followup_general.md` for everything +else. **A diff that adds or moves a timed unit in `tune_kernels.das` keeps `dot_q8q8_laneq4x4` the -LAST row of the `benches` list and adds no CPU timing after it; the Metal crown race after the -list is the one thing allowed there.** Running that bench pins one matmul backend for the rest of -the process, so a CPU timing after it runs against the pinned backend instead of the one it would -have picked. +LAST row of the `benches` list and adds no CPU timing after it.** Running that bench pins one +matmul backend for the rest of the process, so a CPU timing after it runs against the pinned +backend instead of the one it would have picked. **A diff that points a `dashv` call in this folder at the sidecar exchange - dasllama.io's tune-sidecar service: sidecar lookup, download, or submit - is a defect; it calls the exchange @@ -18,8 +17,16 @@ through `dasllama/dasllama_exchange` (the module `../dasllama/dasllama_exchange. instead.** A `dashv` call at a reference engine's server (a llama-server on localhost) is not the exchange. -**A `[vk_dispatch]` class in this folder never silences the lens's never-accessed check with -`@role = "alias"` or `"weight"` on a binding its kernel reads through a `dasllama/` base's -body - it reads that binding in a body this folder's file declares.** The lens collects -accesses from this file's bodies only, and `@role` keeps the access out of the hazard masks, so -a base-read binding silenced here escapes the hazard rail. +**Never set `@role = "alias"` or `"weight"` on a `[vk_dispatch]` binding a file in this folder +declares that no body in that file reads and a `../dasllama/` body does - move the read into a +body the binding's own file declares.** The lens - the compile-time pass that collects a binding's accesses from the +declaring file's bodies alone - reports such a binding as never accessed, and `@role` silences +that report by leaving the binding out of the read and write binding sets the generated +dispatch builds its barriers from. + +**A diff that adds an arm to `vk_gemm_probe.das` joins it to one of the probe's three axes +(`../ARCHITECTURE_MEASUREMENT.md` sec.2.5): the kernel the engine serves a dense role shape +with, against its alternates; the shipped tile against a copy of itself with one lever moved +back; or the shipped tile against the reference engine's own GEMM shader, served in its place +through `DASLLAMA_VK_SPV_OVERRIDE`.** An arm off those axes has no alternate its number can be +ranked against. diff --git a/modules/dasLLAMA/harness/gen_tune_probe.das b/modules/dasLLAMA/harness/gen_tune_probe.das index ba7cf8d7da..1600fa8b08 100644 --- a/modules/dasLLAMA/harness/gen_tune_probe.das +++ b/modules/dasLLAMA/harness/gen_tune_probe.das @@ -454,7 +454,7 @@ def kq_tile_entry(fmt : int64) : string => fmt == 40l ? "q40q8_tile_gen" : (fmt def kq_gemv_entry(fmt : int64) : string => replace(kq_tile_entry(fmt), "_tile_gen", "_gemv_gen") //! the gemv's seat among the tile winner's same-layout rows; "" when the tile winner keeps it -[arch(at = "../ARCHITECTURE_MEASUREMENT.md#gemv-seat")] +[arch(at = "../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#gemv-seat")] def private kq_gemv_seat(fmt : int64; winner : int; names : array; vmr : array; tile, gemv : array) : string { var cand <- [for (i in range(length(names))); i; where vmr[i] == vmr[winner] && tile[i] < 1.0e29lf] cand |> sort() $(a, b) => tile[a] < tile[b] @@ -1319,7 +1319,7 @@ def kq_test_family(fmt : int64; kfxs : array) : bool { // and the gemv (streamed + hot decode shapes), print the table, and return the tile-best // suffix ("" = a gate failed). The kq gemv is nrsplit-independent — same-mr rows share the // plane and the gemv shape — so the tile bench decides the family entry. -[arch(at = "../ARCHITECTURE_MEASUREMENT.md#gemv-seat")] +[arch(at = "../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#gemv-seat")] def kq_tune_family(fmt : int64; var rows_out : table; var gemv_out : string&) : string { // nolint:STYLE037,STYLE038 — one family race: shared repacked buffers couple gate, bench and the two picks // batch shape: fat STREAMING ffn (weights > the L2 budget, multi-token-block walk) — the shape // class that carries ~90% of prefill time. The old 2048x512x64 kv-projection probe was L2-hot diff --git a/modules/dasLLAMA/harness/tune_kernels.das b/modules/dasLLAMA/harness/tune_kernels.das index a5903ccfcd..03fa7e8fd7 100644 --- a/modules/dasLLAMA/harness/tune_kernels.das +++ b/modules/dasLLAMA/harness/tune_kernels.das @@ -2343,7 +2343,7 @@ def bench_laneq4x4() : string { // nolint:STYLE038 — one bench: fixture, backe let CONFIRM_TG_MARGIN = 1.005lf -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private confirm_serving_crown(crown : string; var wins : array) { let model = confirm_decode_model() if (empty(model)) { @@ -2430,7 +2430,7 @@ def private confirm_assistant_model() : string { return empty(best) ? "" : "{md}{best}" } -[arch(at="../ARCHITECTURE_MEASUREMENT.md#kernel-race-fidelity")] +[arch(at="../ARCHITECTURE_MEASUREMENT_KERNEL_RACE.md#kernel-race-fidelity")] def private confirm_tg_run(model : string; wins : array; extra : string; bench_args : string = ""; tag : string = "tg128: ") : double { var crowns = join(wins, ",") if (!empty(extra)) { diff --git a/modules/dasLLAMA/harness/vk_gemm_probe.das b/modules/dasLLAMA/harness/vk_gemm_probe.das index 6ec8ab6a2b..de5a9d14a0 100644 --- a/modules/dasLLAMA/harness/vk_gemm_probe.das +++ b/modules/dasLLAMA/harness/vk_gemm_probe.das @@ -25,6 +25,7 @@ require dasllama/dasllama_vulkan_prefill require dasllama/dasllama_kqformat require daslib/typemacro_boost require _vk_probe_fixture // hash_word, cool_f16_pair: the synthetic planes both probes build +require ../tests/_vkd_oracles.das // k4f16_gemm_oracle: the CPU reference the khrx baseline arm is checked against let private MM_TILE = 128 let private DISPATCHES = 16 @@ -786,7 +787,7 @@ class KhrPxBase : MoeCmBase { } } - def body { + def body_f32a { let rb = sched[pa.map_off + gl_WorkGroupID.x] * 4u let wblk0 = sched[rb] let row0 = sched[rb + 1u] @@ -807,14 +808,9 @@ class KhrPxBase : MoeCmBase { let skh = (tid & 1u) * 16u let sbase = srow * KHRPX_STRIDE + skh let wrow = m0 + srow - var acc0 : coopmatAcc_f32_16x16 - var acc1 : coopmatAcc_f32_16x16 - var acc2 : coopmatAcc_f32_16x16 - var acc3 : coopmatAcc_f32_16x16 - var acc4 : coopmatAcc_f32_16x16 - var acc5 : coopmatAcc_f32_16x16 - var acc6 : coopmatAcc_f32_16x16 - var acc7 : coopmatAcc_f32_16x16 + let wrow0 = (sg & 1u) * 64u + let trow0 = (sg >> 1u) * 32u + var acc00, acc01, acc10, acc11, acc20, acc21, acc30, acc31 : coopmatAcc_f32_16x16 var k = 0u while (k < pa.n) { let kk = k + skh @@ -824,103 +820,38 @@ class KhrPxBase : MoeCmBase { barrier() for (ks in range(2)) { let ko = uint(ks) * 16u - var a : coopmatA_f16_16x16 - coopmatLoad(a, khr_a, int(sg * 16u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) - var b : coopmatB_f16_16x16 - coopmatLoad(b, khr_b, int(ko), int(KHRPX_STRIDE), 1) - acc0 = coopmatMulAdd(a, b, acc0) - coopmatLoad(b, khr_b, int(16u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc1 = coopmatMulAdd(a, b, acc1) - coopmatLoad(b, khr_b, int(32u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc2 = coopmatMulAdd(a, b, acc2) - coopmatLoad(b, khr_b, int(48u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc3 = coopmatMulAdd(a, b, acc3) - coopmatLoad(b, khr_b, int(64u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc4 = coopmatMulAdd(a, b, acc4) - coopmatLoad(b, khr_b, int(80u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc5 = coopmatMulAdd(a, b, acc5) - coopmatLoad(b, khr_b, int(96u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc6 = coopmatMulAdd(a, b, acc6) - coopmatLoad(b, khr_b, int(112u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) - acc7 = coopmatMulAdd(a, b, acc7) + var a0, a1, a2, a3 : coopmatA_f16_16x16 + coopmatLoad(a0, khr_a, int(wrow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a1, khr_a, int((wrow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a2, khr_a, int((wrow0 + 32u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a3, khr_a, int((wrow0 + 48u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + var b0, b1 : coopmatB_f16_16x16 + coopmatLoad(b0, khr_b, int(trow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + coopmatLoad(b1, khr_b, int((trow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + acc00 = coopmatMulAdd(a0, b0, acc00) + acc01 = coopmatMulAdd(a0, b1, acc01) + acc10 = coopmatMulAdd(a1, b0, acc10) + acc11 = coopmatMulAdd(a1, b1, acc11) + acc20 = coopmatMulAdd(a2, b0, acc20) + acc21 = coopmatMulAdd(a2, b1, acc21) + acc30 = coopmatMulAdd(a3, b0, acc30) + acc31 = coopmatMulAdd(a3, b1, acc31) } barrier() k += 32u } - let yb = t0 * pa.d + m0 + sg * 16u - coopmatStore(acc0, y, int(yb), int(pa.d), 1) - coopmatStore(acc1, y, int(yb + 16u * pa.d), int(pa.d), 1) - coopmatStore(acc2, y, int(yb + 32u * pa.d), int(pa.d), 1) - coopmatStore(acc3, y, int(yb + 48u * pa.d), int(pa.d), 1) - coopmatStore(acc4, y, int(yb + 64u * pa.d), int(pa.d), 1) - coopmatStore(acc5, y, int(yb + 80u * pa.d), int(pa.d), 1) - coopmatStore(acc6, y, int(yb + 96u * pa.d), int(pa.d), 1) - coopmatStore(acc7, y, int(yb + 112u * pa.d), int(pa.d), 1) - } -} - -[vk_dispatch(name = "khrpx_nil", grid = "wgs", params = "wgs : int64")] -class KhrPxNil : KhrPxBase { - @ssbo @binding = 0 @role = "alias" @readonly wq : array //! bound, never read: the ceiling stages a constant - - [spirv_kernel(local_size_x = 256, name = "khrpx_nil_spv")] - def run { - body() - } -} - -class KhrPxShipBase : KhrPxBase { - @ssbo @binding = 0 @readonly wq : array - - //! the shipped tile's weight stage: four decodes on the plane element into the row's 16 halves - def stage_decoded(wblk0, wrow, bcol, e0, sbase : uint) { - for (j in range(4)) { - let v4 = decode_v4(wq[wblk0 + wrow * (pa.n >> 8u) + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 4u)) - let o = sbase + uint(j) * 4u - khr_a[o] = v4.x - khr_a[o + 1u] = v4.y - khr_a[o + 2u] = v4.z - khr_a[o + 3u] = v4.w - } - } - - def override stage_a(wblk0, wrow, bcol, e0, sbase : uint) { - stage_decoded(wblk0, wrow, bcol, e0, sbase) - } -} - -[vk_dispatch(name = "khrpx_ship", grid = "wgs", params = "wgs : int64")] -class KhrPxShip : KhrPxShipBase { - [spirv_kernel(local_size_x = 256, name = "khrpx_ship_spv")] - def run { - body() - } -} - -//! ship plus the shipped tile's 8 KB edge-store slab, touched on a path no dispatch takes: the occupancy cost of the third workgroup array alone -[vk_dispatch(name = "khrpx_slab", grid = "wgs", params = "wgs : int64")] -class KhrPxSlab : KhrPxShipBase { - @workgroup khr_o : float[2048] - - def override stage_a(wblk0, wrow, bcol, e0, sbase : uint) { - stage_decoded(wblk0, wrow, bcol, e0, sbase) - if (pa.map_off == 0u) { - khr_o[sbase] = float(khr_a[sbase]) - khr_a[sbase] = float16(khr_o[sbase + 1u]) - } + let yb = (t0 + trow0) * pa.d + m0 + wrow0 + coopmatStore(acc00, y, int(yb), int(pa.d), 1) + coopmatStore(acc01, y, int(yb + 16u * pa.d), int(pa.d), 1) + coopmatStore(acc10, y, int(yb + 16u), int(pa.d), 1) + coopmatStore(acc11, y, int(yb + 16u * pa.d + 16u), int(pa.d), 1) + coopmatStore(acc20, y, int(yb + 32u), int(pa.d), 1) + coopmatStore(acc21, y, int(yb + 16u * pa.d + 32u), int(pa.d), 1) + coopmatStore(acc30, y, int(yb + 48u), int(pa.d), 1) + coopmatStore(acc31, y, int(yb + 16u * pa.d + 48u), int(pa.d), 1) } - [spirv_kernel(local_size_x = 256, name = "khrpx_slab_spv")] - def run { - body() - } -} - -[vk_dispatch(name = "khrpx_nob", grid = "wgs", params = "wgs : int64")] -class KhrPxNoB : KhrPxShipBase { - @ssbo @binding = 3 @role = "alias" @readonly xf16 : array //! the B fragments load straight from the plane's half view - - def override body { + def body_strip { let rb = sched[pa.map_off + gl_WorkGroupID.x] * 4u let wblk0 = sched[rb] let row0 = sched[rb + 1u] @@ -941,90 +872,61 @@ class KhrPxNoB : KhrPxShipBase { let skh = (tid & 1u) * 16u let sbase = srow * KHRPX_STRIDE + skh let wrow = m0 + srow - var acc0 : coopmatAcc_f32_16x16 - var acc1 : coopmatAcc_f32_16x16 - var acc2 : coopmatAcc_f32_16x16 - var acc3 : coopmatAcc_f32_16x16 - var acc4 : coopmatAcc_f32_16x16 - var acc5 : coopmatAcc_f32_16x16 - var acc6 : coopmatAcc_f32_16x16 - var acc7 : coopmatAcc_f32_16x16 + var acc0, acc1, acc2, acc3, acc4, acc5, acc6, acc7 : coopmatAcc_f16_16x16 var k = 0u while (k < pa.n) { let kk = k + skh let bcol = kk >> 8u stage_a(wblk0, wrow, bcol, kk - bcol * 256u, sbase) + stage_b((t0 + srow) * pa.n + kk, sbase) barrier() for (ks in range(2)) { let ko = uint(ks) * 16u var a : coopmatA_f16_16x16 coopmatLoad(a, khr_a, int(sg * 16u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) - let bb = t0 * pa.n + k + ko var b : coopmatB_f16_16x16 - coopmatLoad(b, xf16, int(bb), int(pa.n), 1) + coopmatLoad(b, khr_b, int(ko), int(KHRPX_STRIDE), 1) acc0 = coopmatMulAdd(a, b, acc0) - coopmatLoad(b, xf16, int(bb + 16u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(16u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc1 = coopmatMulAdd(a, b, acc1) - coopmatLoad(b, xf16, int(bb + 32u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(32u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc2 = coopmatMulAdd(a, b, acc2) - coopmatLoad(b, xf16, int(bb + 48u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(48u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc3 = coopmatMulAdd(a, b, acc3) - coopmatLoad(b, xf16, int(bb + 64u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(64u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc4 = coopmatMulAdd(a, b, acc4) - coopmatLoad(b, xf16, int(bb + 80u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(80u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc5 = coopmatMulAdd(a, b, acc5) - coopmatLoad(b, xf16, int(bb + 96u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(96u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc6 = coopmatMulAdd(a, b, acc6) - coopmatLoad(b, xf16, int(bb + 112u * pa.n), int(pa.n), 1) + coopmatLoad(b, khr_b, int(112u * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) acc7 = coopmatMulAdd(a, b, acc7) } barrier() k += 32u } let yb = t0 * pa.d + m0 + sg * 16u - coopmatStore(acc0, y, int(yb), int(pa.d), 1) - coopmatStore(acc1, y, int(yb + 16u * pa.d), int(pa.d), 1) - coopmatStore(acc2, y, int(yb + 32u * pa.d), int(pa.d), 1) - coopmatStore(acc3, y, int(yb + 48u * pa.d), int(pa.d), 1) - coopmatStore(acc4, y, int(yb + 64u * pa.d), int(pa.d), 1) - coopmatStore(acc5, y, int(yb + 80u * pa.d), int(pa.d), 1) - coopmatStore(acc6, y, int(yb + 96u * pa.d), int(pa.d), 1) - coopmatStore(acc7, y, int(yb + 112u * pa.d), int(pa.d), 1) - } - - [spirv_kernel(local_size_x = 256, name = "khrpx_nob_spv")] - def run { - body() - } -} - -[vk_dispatch(name = "khrpx_k64", grid = "wgs", params = "wgs : int64")] -class KhrPxK64 : MoeCmBase { //! its own base: the 32-step staging arrays would count against its footprint - @ssbo @binding = 0 @readonly wq : array - @ssbo @binding = 1 @readonly ws : array - @ssbo @binding = 3 @readonly xf16 : array - @ssbo @binding = 5 y : array - @workgroup wg_blk0 : uint - @workgroup khr_a64 : float16[9216] - @workgroup khr_b64 : float16[9216] - - [spirv_decode] - def decode_v4(blk : VkK4Blk; bc, cib : uint2) : half4 { - let g = cib.y >> 5u - let e = cib.y & 31u - let li = int((g * 16u + (e & 15u)) >> 1u) - let l0 = uint(int(blk.qs[li])) & 0xFFFFu - let l1 = uint(int(blk.qs[li + 1])) & 0xFFFFu - let sh = (e >> 4u) * 4u - let srow = (wg_blk0 + bc.x * (pa.n >> 8u) + bc.y) * 5u - let dm = unpackHalf2x16(ws[srow]) - let ds = dm.x * float((ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) - let dmn = dm.y * float((ws[srow + 3u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) - return half4(float4(ds * float((l0 >> sh) & 0xFu) - dmn, ds * float((l0 >> (sh + 8u)) & 0xFu) - dmn, - ds * float((l1 >> sh) & 0xFu) - dmn, ds * float((l1 >> (sh + 8u)) & 0xFu) - dmn)) - } - - def body { // nolint:STYLE038 — the whole-tile path at the 64-deep step, one register set + var accf32 : coopmatAcc_f32_16x16 + coopmatConvert(accf32, acc0) + coopmatStore(accf32, y, int(yb), int(pa.d), 1) + coopmatConvert(accf32, acc1) + coopmatStore(accf32, y, int(yb + 16u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc2) + coopmatStore(accf32, y, int(yb + 32u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc3) + coopmatStore(accf32, y, int(yb + 48u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc4) + coopmatStore(accf32, y, int(yb + 64u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc5) + coopmatStore(accf32, y, int(yb + 80u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc6) + coopmatStore(accf32, y, int(yb + 96u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc7) + coopmatStore(accf32, y, int(yb + 112u * pa.d), int(pa.d), 1) + } + + [arch(at="../ARCHITECTURE_GPU_VULKAN_GEMM.md#cm2-tile-pick-and-default")] + def body { let rb = sched[pa.map_off + gl_WorkGroupID.x] * 4u let wblk0 = sched[rb] let row0 = sched[rb + 1u] @@ -1042,112 +944,258 @@ class KhrPxK64 : MoeCmBase { //! its own base: the 32-step staging arrays woul let tid = gl_LocalInvocationID.x let sg = gl_SubgroupID let srow = tid >> 1u - let skh = (tid & 1u) * 32u - let sbase = srow * 72u + skh + let skh = (tid & 1u) * 16u + let sbase = srow * KHRPX_STRIDE + skh let wrow = m0 + srow - let nbk = pa.n >> 8u - var acc0 : coopmatAcc_f32_16x16 - var acc1 : coopmatAcc_f32_16x16 - var acc2 : coopmatAcc_f32_16x16 - var acc3 : coopmatAcc_f32_16x16 - var acc4 : coopmatAcc_f32_16x16 - var acc5 : coopmatAcc_f32_16x16 - var acc6 : coopmatAcc_f32_16x16 - var acc7 : coopmatAcc_f32_16x16 + let wrow0 = (sg & 1u) * 64u + let trow0 = (sg >> 1u) * 32u + var acc00, acc01, acc10, acc11, acc20, acc21, acc30, acc31 : coopmatAcc_f16_16x16 var k = 0u while (k < pa.n) { let kk = k + skh let bcol = kk >> 8u - let e0 = kk - bcol * 256u - for (j in range(8)) { - let v4 = decode_v4(wq[wblk0 + wrow * nbk + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 4u)) - let o = sbase + uint(j) * 4u - khr_a64[o] = v4.x - khr_a64[o + 1u] = v4.y - khr_a64[o + 2u] = v4.z - khr_a64[o + 3u] = v4.w - } - let xb = (t0 + srow) * pa.n + kk - for (j in range(32)) { - khr_b64[sbase + uint(j)] = xf16[xb + uint(j)] - } + stage_a(wblk0, wrow, bcol, kk - bcol * 256u, sbase) + stage_b((t0 + srow) * pa.n + kk, sbase) barrier() - for (ks in range(4)) { + for (ks in range(2)) { let ko = uint(ks) * 16u - var a : coopmatA_f16_16x16 - coopmatLoad(a, khr_a64, int(sg * 16u * 72u + ko), 72, 0) - var b : coopmatB_f16_16x16 - coopmatLoad(b, khr_b64, int(ko), 72, 1) - acc0 = coopmatMulAdd(a, b, acc0) - coopmatLoad(b, khr_b64, int(16u * 72u + ko), 72, 1) - acc1 = coopmatMulAdd(a, b, acc1) - coopmatLoad(b, khr_b64, int(32u * 72u + ko), 72, 1) - acc2 = coopmatMulAdd(a, b, acc2) - coopmatLoad(b, khr_b64, int(48u * 72u + ko), 72, 1) - acc3 = coopmatMulAdd(a, b, acc3) - coopmatLoad(b, khr_b64, int(64u * 72u + ko), 72, 1) - acc4 = coopmatMulAdd(a, b, acc4) - coopmatLoad(b, khr_b64, int(80u * 72u + ko), 72, 1) - acc5 = coopmatMulAdd(a, b, acc5) - coopmatLoad(b, khr_b64, int(96u * 72u + ko), 72, 1) - acc6 = coopmatMulAdd(a, b, acc6) - coopmatLoad(b, khr_b64, int(112u * 72u + ko), 72, 1) - acc7 = coopmatMulAdd(a, b, acc7) + var a0, a1, a2, a3 : coopmatA_f16_16x16 + coopmatLoad(a0, khr_a, int(wrow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a1, khr_a, int((wrow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a2, khr_a, int((wrow0 + 32u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a3, khr_a, int((wrow0 + 48u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + var b0, b1 : coopmatB_f16_16x16 + coopmatLoad(b0, khr_b, int(trow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + coopmatLoad(b1, khr_b, int((trow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + acc00 = coopmatMulAdd(a0, b0, acc00) + acc01 = coopmatMulAdd(a0, b1, acc01) + acc10 = coopmatMulAdd(a1, b0, acc10) + acc11 = coopmatMulAdd(a1, b1, acc11) + acc20 = coopmatMulAdd(a2, b0, acc20) + acc21 = coopmatMulAdd(a2, b1, acc21) + acc30 = coopmatMulAdd(a3, b0, acc30) + acc31 = coopmatMulAdd(a3, b1, acc31) } barrier() - k += 64u + k += 32u } - let yb = t0 * pa.d + m0 + sg * 16u - coopmatStore(acc0, y, int(yb), int(pa.d), 1) - coopmatStore(acc1, y, int(yb + 16u * pa.d), int(pa.d), 1) - coopmatStore(acc2, y, int(yb + 32u * pa.d), int(pa.d), 1) - coopmatStore(acc3, y, int(yb + 48u * pa.d), int(pa.d), 1) - coopmatStore(acc4, y, int(yb + 64u * pa.d), int(pa.d), 1) - coopmatStore(acc5, y, int(yb + 80u * pa.d), int(pa.d), 1) - coopmatStore(acc6, y, int(yb + 96u * pa.d), int(pa.d), 1) - coopmatStore(acc7, y, int(yb + 112u * pa.d), int(pa.d), 1) + let yb = (t0 + trow0) * pa.d + m0 + wrow0 + var accf32 : coopmatAcc_f32_16x16 + coopmatConvert(accf32, acc00) + coopmatStore(accf32, y, int(yb), int(pa.d), 1) + coopmatConvert(accf32, acc01) + coopmatStore(accf32, y, int(yb + 16u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc10) + coopmatStore(accf32, y, int(yb + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc11) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc20) + coopmatStore(accf32, y, int(yb + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc21) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc30) + coopmatStore(accf32, y, int(yb + 48u), int(pa.d), 1) + coopmatConvert(accf32, acc31) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 48u), int(pa.d), 1) } +} - [spirv_kernel(local_size_x = 256, name = "khrpx_k64_spv")] +[vk_dispatch(name = "khrpx_nil", grid = "wgs", params = "wgs : int64")] +class KhrPxNil : KhrPxBase { + @ssbo @binding = 0 @role = "alias" @readonly wq : array + + [spirv_kernel(local_size_x = 256, name = "khrpx_nil_spv"), arch(at="../ARCHITECTURE_GPU_VULKAN.md#khrx-shared-set-layout")] def run { body() } } -[vk_dispatch(name = "khrpx_inline", grid = "wgs", params = "wgs : int64")] -class KhrPxInline : KhrPxBase { - @ssbo @binding = 0 @readonly wq32 : array //! the k4 plane as words: one block = 64 lanes = 32 words +[vk_dispatch(name = "khrpx_dec4", grid = "wgs", params = "wgs : int64")] +class KhrPxDec4 : KhrPxBase { + @ssbo @binding = 0 @readonly wq : array def override stage_a(wblk0, wrow, bcol, e0, sbase : uint) { - let bw = (wblk0 + wrow * (pa.n >> 8u) + bcol) * 32u - let srow = (wg_blk0 + wrow * (pa.n >> 8u) + bcol) * 5u - let dm = unpackHalf2x16(ws[srow]) for (j in range(4)) { - let e = e0 + uint(j) * 4u - let g = e >> 5u - let ee = e & 31u - let w = wq32[bw + ((g * 16u + (ee & 15u)) >> 2u)] //! lanes li, li + 1 share one word (li even) - let l0 = w & 0xFFFFu - let l1 = w >> 16u - let sh = (ee >> 4u) * 4u - let ds = dm.x * float((ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) - let dmn = dm.y * float((ws[srow + 3u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + let v4 = decode_v4(wq[wblk0 + wrow * (pa.n >> 8u) + bcol], uint2(wrow, bcol), uint2(0u, e0 + uint(j) * 4u)) let o = sbase + uint(j) * 4u - khr_a[o] = float16(ds * float((l0 >> sh) & 0xFu) - dmn) - khr_a[o + 1u] = float16(ds * float((l0 >> (sh + 8u)) & 0xFu) - dmn) - khr_a[o + 2u] = float16(ds * float((l1 >> sh) & 0xFu) - dmn) - khr_a[o + 3u] = float16(ds * float((l1 >> (sh + 8u)) & 0xFu) - dmn) + khr_a[o] = v4.x + khr_a[o + 1u] = v4.y + khr_a[o + 2u] = v4.z + khr_a[o + 3u] = v4.w + } + } + + [spirv_kernel(local_size_x = 256, name = "khrpx_dec4_spv")] + def run { + body() + } +} + +//! the shipped tile's weight stage, copied: every lever arm below inherits it unmoved +class KhrPxW4Base : KhrPxBase { + @ssbo @binding = 0 @readonly wq4 : array + + def override stage_a(wblk0, wrow, bcol, e0, sbase : uint) { + let g = e0 >> 5u + let sh = ((e0 >> 4u) & 1u) * 4u + let w4 = wq4[(wblk0 + wrow * (pa.n >> 8u) + bcol) * 8u + g] + let srow = (wg_blk0 + wrow * (pa.n >> 8u) + bcol) * 5u + let dm = unpackHalf2x16(ws[srow]) + let ds = dm.x * float((ws[srow + 1u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + let dmn = dm.y * float((ws[srow + 3u + (g >> 2u)] >> ((g & 3u) * 8u)) & 0xFFu) + for [unroll_full] (c in range(4)) { + let w = w4[c] + let o = sbase + uint(c) * 4u + khr_a[o] = float16(ds * float((w >> sh) & 0xFu) - dmn) + khr_a[o + 1u] = float16(ds * float((w >> (sh + 8u)) & 0xFu) - dmn) + khr_a[o + 2u] = float16(ds * float((w >> (sh + 16u)) & 0xFu) - dmn) + khr_a[o + 3u] = float16(ds * float((w >> (sh + 24u)) & 0xFu) - dmn) } } +} - [spirv_kernel(local_size_x = 256, name = "khrpx_inline_spv")] +//! the shipped tile, copied: the control row every lever's arm is timed against +[vk_dispatch(name = "khrpx_ship", grid = "wgs", params = "wgs : int64")] +class KhrPxShip : KhrPxW4Base { + [spirv_kernel(local_size_x = 256, name = "khrpx_ship_spv")] def run { body() } } +[vk_dispatch(name = "khrpx_f32a", grid = "wgs", params = "wgs : int64")] +class KhrPxF32a : KhrPxW4Base { + [spirv_kernel(local_size_x = 256, name = "khrpx_f32a_spv")] + def run { + body_f32a() + } +} + +[vk_dispatch(name = "khrpx_strip", grid = "wgs", params = "wgs : int64")] +class KhrPxStrip : KhrPxW4Base { + [spirv_kernel(local_size_x = 256, name = "khrpx_strip_spv")] + def run { + body_strip() + } +} + +[vk_dispatch(name = "khrpx_wg128", grid = "wgs", params = "wgs : int64")] +class KhrPxWg128 : KhrPxW4Base { + [spirv_kernel(local_size_x = 128, name = "khrpx_wg128_spv")] + def run { // nolint:STYLE038 — one measurement body, its sixteen accumulators named + let rb = sched[pa.map_off + gl_WorkGroupID.x] * 4u + let wblk0 = sched[rb] + let row0 = sched[rb + 1u] + let cnt = sched[rb + 2u] + let ttiles = (cnt + 127u) / 128u + let tix = gl_WorkGroupID.x - sched[rb + 3u] + let xt = tix % ttiles + let wt = tix / ttiles + if (gl_LocalInvocationID.x == 0u) { + wg_blk0 = wblk0 + } + barrier() + let t0 = row0 + xt * 128u + let m0 = wt * 128u + let srow = gl_LocalInvocationID.x + let sg = gl_SubgroupID + let sbase = srow * KHRPX_STRIDE + let wrow = m0 + srow + let wrow0 = (sg & 1u) * 64u + let trow0 = (sg >> 1u) * 64u + var acc00, acc01, acc02, acc03, acc10, acc11, acc12, acc13 : coopmatAcc_f16_16x16 + var acc20, acc21, acc22, acc23, acc30, acc31, acc32, acc33 : coopmatAcc_f16_16x16 + var k = 0u + while (k < pa.n) { + let bcol = k >> 8u + let e0 = k - bcol * 256u + stage_a(wblk0, wrow, bcol, e0, sbase) + stage_a(wblk0, wrow, bcol, e0 + 16u, sbase + 16u) + let xb = (t0 + srow) * pa.n + k + stage_b(xb, sbase) + stage_b(xb + 16u, sbase + 16u) + barrier() + for (ks in range(2)) { + let ko = uint(ks) * 16u + var a0, a1, a2, a3 : coopmatA_f16_16x16 + coopmatLoad(a0, khr_a, int(wrow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a1, khr_a, int((wrow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a2, khr_a, int((wrow0 + 32u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + coopmatLoad(a3, khr_a, int((wrow0 + 48u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 0) + var b0, b1, b2, b3 : coopmatB_f16_16x16 + coopmatLoad(b0, khr_b, int(trow0 * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + coopmatLoad(b1, khr_b, int((trow0 + 16u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + coopmatLoad(b2, khr_b, int((trow0 + 32u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + coopmatLoad(b3, khr_b, int((trow0 + 48u) * KHRPX_STRIDE + ko), int(KHRPX_STRIDE), 1) + acc00 = coopmatMulAdd(a0, b0, acc00) + acc01 = coopmatMulAdd(a0, b1, acc01) + acc02 = coopmatMulAdd(a0, b2, acc02) + acc03 = coopmatMulAdd(a0, b3, acc03) + acc10 = coopmatMulAdd(a1, b0, acc10) + acc11 = coopmatMulAdd(a1, b1, acc11) + acc12 = coopmatMulAdd(a1, b2, acc12) + acc13 = coopmatMulAdd(a1, b3, acc13) + acc20 = coopmatMulAdd(a2, b0, acc20) + acc21 = coopmatMulAdd(a2, b1, acc21) + acc22 = coopmatMulAdd(a2, b2, acc22) + acc23 = coopmatMulAdd(a2, b3, acc23) + acc30 = coopmatMulAdd(a3, b0, acc30) + acc31 = coopmatMulAdd(a3, b1, acc31) + acc32 = coopmatMulAdd(a3, b2, acc32) + acc33 = coopmatMulAdd(a3, b3, acc33) + } + barrier() + k += 32u + } + let yb = (t0 + trow0) * pa.d + m0 + wrow0 + var accf32 : coopmatAcc_f32_16x16 + coopmatConvert(accf32, acc00) + coopmatStore(accf32, y, int(yb), int(pa.d), 1) + coopmatConvert(accf32, acc01) + coopmatStore(accf32, y, int(yb + 16u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc02) + coopmatStore(accf32, y, int(yb + 32u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc03) + coopmatStore(accf32, y, int(yb + 48u * pa.d), int(pa.d), 1) + coopmatConvert(accf32, acc10) + coopmatStore(accf32, y, int(yb + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc11) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc12) + coopmatStore(accf32, y, int(yb + 32u * pa.d + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc13) + coopmatStore(accf32, y, int(yb + 48u * pa.d + 16u), int(pa.d), 1) + coopmatConvert(accf32, acc20) + coopmatStore(accf32, y, int(yb + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc21) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc22) + coopmatStore(accf32, y, int(yb + 32u * pa.d + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc23) + coopmatStore(accf32, y, int(yb + 48u * pa.d + 32u), int(pa.d), 1) + coopmatConvert(accf32, acc30) + coopmatStore(accf32, y, int(yb + 48u), int(pa.d), 1) + coopmatConvert(accf32, acc31) + coopmatStore(accf32, y, int(yb + 16u * pa.d + 48u), int(pa.d), 1) + coopmatConvert(accf32, acc32) + coopmatStore(accf32, y, int(yb + 32u * pa.d + 48u), int(pa.d), 1) + coopmatConvert(accf32, acc33) + coopmatStore(accf32, y, int(yb + 48u * pa.d + 48u), int(pa.d), 1) + } +} + let private KHR_ARMS = 8 +let private KHR_ARM_NAMES = fixed_array("nil ", "ship ", "dec4 ", "khr ", "kq ", "f32a ", "strip ", "wg128 ") +let private KHR_ARM_KHR = 3 //! the shipped class: the compared arms' reference and the CPU check's subject +let private KHR_ARM_KQ = 4 //! the sdot4 kq tile, 32 x 32 where every other arm tiles 128 x 128 +let private KHR_ARM_COMPARED = fixed_array(false, true, true, false, false, true, true, true) //! khr is the reference; nil stages constants and kq reads its own Q8 activation fixture: timing-only +let private KHR_ARM_REL_BOUND = 2e-2lf //! a compared arm passes within this relative difference of the shipped class; the copies read 0, the f32-accumulator arm 0.005-0.009 + +//! the bisect arms stage and store whole 128-row tiles - only the two shipped tiles clamp token rows to the window's count, so only they run on a partial window +def private khr_arm_clamps(v : int) : bool => v == KHR_ARM_KHR || v == KHR_ARM_KQ let private MMQ_ARMS = 5 +var private g_prof_arm = "" +var private g_probe_exit = 0 //! the process exit code: a decline, a compared arm over its bound, or a profiler run that dispatched nothing //! the interleaved timing of one sweep's arms: after a two-submit warm-up each, every round times //! SUBMITS / ROUNDS submits of every arm in turn, and an arm's figure is its best round @@ -1156,11 +1204,17 @@ def private sweep_best_us(var raws : array) : array { best |> resize(length(raws)) for (raw, bv in raws, best) { bv = 1.0e30lf + if (raw == null) { + continue + } submit_wait(raw) submit_wait(raw) } for (_r in range(ROUNDS)) { for (raw, bv in raws, best) { + if (raw == null) { + continue + } let t0 = ref_time_ticks() for (_s in range(SUBMITS / ROUNDS)) { submit_wait(raw) @@ -1213,16 +1267,16 @@ def private khr_arm_set(v, fmt : int; wqd, wsd, scd, xfd, xqd, axsd, yd : uint64 } elif (v == 1) { return set_khrpx_ship(bufs, sizes, gbits) } elif (v == 2) { - return set_khrpx_inline(bufs, sizes, gbits) + return set_khrpx_dec4(bufs, sizes, gbits) } elif (v == 3) { return set_kq_batch_k4_khr_cls(bufs, sizes, gbits) } elif (v == 5) { - return set_khrpx_nob(bufs, sizes, gbits) + return set_khrpx_f32a(bufs, sizes, gbits) } elif (v == 6) { - return set_khrpx_k64(bufs, sizes, gbits) + return set_khrpx_strip(bufs, sizes, gbits) } verify(v == 7, "khrx: the arm table has eight arms") - return set_khrpx_slab(bufs, sizes, gbits) + return set_khrpx_wg128(bufs, sizes, gbits) } def private khr_arm_enc(v, fmt : int; raw : VkCommandBuffer; var hz : VkHaz; var s : VkDescriptorSet; var pc : BatchArgs; wgs : int64) { @@ -1231,23 +1285,105 @@ def private khr_arm_enc(v, fmt : int; raw : VkCommandBuffer; var hz : VkHaz; var } elif (v == 1) { enc_khrpx_ship(raw, hz, s, pc, wgs) } elif (v == 2) { - enc_khrpx_inline(raw, hz, s, pc, wgs) + enc_khrpx_dec4(raw, hz, s, pc, wgs) } elif (v == 3) { enc_kq_batch_k4_khr_cls(raw, hz, s, pc, wgs) } elif (v == 4) { kq_batch_cls_enc_for(fmt, false, raw, hz, s, pc, wgs) } elif (v == 5) { - enc_khrpx_nob(raw, hz, s, pc, wgs) + enc_khrpx_f32a(raw, hz, s, pc, wgs) } elif (v == 6) { - enc_khrpx_k64(raw, hz, s, pc, wgs) + enc_khrpx_strip(raw, hz, s, pc, wgs) } else { verify(v == 7, "khrx: the arm table has eight arms") - enc_khrpx_slab(raw, hz, s, pc, wgs) + enc_khrpx_wg128(raw, hz, s, pc, wgs) + } +} + +//! the baseline arm against the CPU oracle over six corners of the output - eight weight rows by sixteen tokens at the +//! first, middle and last rows, at the first and last tokens - on the kernel cell's bar: |gpu - cpu| <= 2e-2 |cpu| + 4e-3 max|y| +def private khr_cpu_check(wqh, wsuh, xfh : array; y_gpu : array; d, n, cnt : int) : tuple { + let nsb = n / 256 + var ymax = 0.0 + for (v in y_gpu) { + ymax = max(ymax, abs(v)) + } + var samples = 0 + var off = 0 + var max_abs = 0.0lf + for (r0 in [0, (d / 2) & ~7, d - 8]) { + for (t0 in [0, cnt - 16]) { + let recs = [uint(r0 * nsb), uint(t0), 16u, 0u] + var ys : array + ys |> resize((t0 + 16) * 8) + k4f16_gemm_oracle(wqh, wsuh, xfh, recs, 1, n, 8, ys) + for (r in range(16)) { + for (c in range(8)) { + let cpu = ys[(t0 + r) * 8 + c] + let gpu = y_gpu[(t0 + r) * d + r0 + c] + let diff = abs(gpu - cpu) + samples++ + if (diff > 2e-2 * abs(cpu) + 4e-3 * ymax) { + off++ + } + max_abs = max(max_abs, double(diff)) + } + } + delete ys + } + } + return (samples = samples, off = off, max_abs = max_abs) +} + +//! one line per arm: its best rate, and for a compared arm its distance from the shipped class - bit-exact, within the bound, +//! or OVER it (the exit code); the shipped class's output stays in `y_ref` for the CPU check +def private khr_report(name : string; var raws : array; best : array; last_out : uint64; y_bytes : int64; flop : double; var y_ref : array) { + read_back_y(raws[KHR_ARM_KHR], last_out, y_bytes, y_ref) + var y_arm : array + for (v in range(KHR_ARMS)) { + if (raws[v] == null) { + print("{name} {KHR_ARM_NAMES[v]}: not run - a whole-tile arm on a partial window\n") + continue + } + let tf = flop / (best[v] * 1000000.0lf) + var tail = "timing-only" + if (KHR_ARM_COMPARED[v]) { + read_back_y(raws[v], last_out, y_bytes, y_arm) + let mrd = max_rel_diff(y_arm, y_ref) + if (mrd == 0.0lf) { + tail = "bit-exact vs khr" + } else { + let within = mrd <= KHR_ARM_REL_BOUND + tail = "max rel diff {mrd:.5f} vs khr ({within ? "within" : "OVER"} {KHR_ARM_REL_BOUND})" + if (!within) { + g_probe_exit = 1 + } + } + } + print("{name} {KHR_ARM_NAMES[v]}: {best[v] / 1000.0lf} ms/dispatch {tf} TFLOP/s {tail}\n") + } + delete y_arm +} + +//! the khrprof mode: one submit per arm - or the one arm named - so a GPU profiler sees a single kernel's dispatches +def private khr_prof_submit(name : string; var raws : array) { + var submitted = 0 + for (v in range(KHR_ARMS)) { + if (raws[v] != null && (g_prof_arm == "" || strip(KHR_ARM_NAMES[v]) == g_prof_arm)) { + submit_wait(raws[v]) + submitted++ + } + } + let which = g_prof_arm == "" ? "all {KHR_ARMS}" : g_prof_arm + print("{name}: one submit per arm ({which}), {DISPATCHES} dispatches each\n") + if (submitted == 0) { + print("khrprof: no arm named '{g_prof_arm}' - nothing was dispatched\n") + g_probe_exit = 1 } } [arch(at="../ARCHITECTURE_MEASUREMENT.md#one-benchmark-rig")] -def private run_khr_shape(name : string; d, n, cnt : int) { // nolint:STYLE038 — one interleaved measurement sweep over eight arms +def private run_khr_shape(name : string; d, n, cnt : int; prof : bool = false) { // nolint:STYLE038 — one interleaved measurement sweep over eight arms, its compare and its CPU check let fmt = int(KqFmt.k4) let nsb = n / 256 let totsb = d * nsb @@ -1302,12 +1438,16 @@ def private run_khr_shape(name : string; d, n, cnt : int) { // nolint:STYLE038 upload_region_at(xqd, 0l, addr(xqh[0]), xq_bytes) upload_region_at(axsd, 0l, addr(axsh[0]), axs_bytes) let yd2 = make_device_buf(y_bytes) //! consecutive dispatches alternate outputs, as the served graph overlaps them - let vnames = fixed_array("nil ", "ship ", "inline", "khr ", "kq ", "nob ", "k64 ", "slab ") - let compared = fixed_array(false, true, true, false, false, false, true, true) //! khr is the reference; nil stages constants, nob no B, and kq reads its own Q8 activation fixture: timing-only + let partial = cnt % MM_TILE != 0 var raws : array raws |> reserve(KHR_ARMS) for (v in range(KHR_ARMS)) { - let tile = v == 4 ? 32 : 128 + if (partial && !khr_arm_clamps(v)) { + var none : VkCommandBuffer + raws |> push(none) + continue + } + let tile = v == KHR_ARM_KQ ? 32 : 128 let wgs = ((cnt + tile - 1) / tile) * ((d + tile - 1) / tile) var sched : array sched |> resize(4 + wgs) @@ -1333,23 +1473,21 @@ def private run_khr_shape(name : string; d, n, cnt : int) { // nolint:STYLE038 vk_check(vkEndCommandBuffer(raw), null) raws |> push(raw) } - var best <- sweep_best_us(raws) - let last_out = DISPATCHES % 2 == 0 ? yd2 : yd - var y_ref : array - read_back_y(raws[3], last_out, y_bytes, y_ref) - var y_arm : array - for (v in range(KHR_ARMS)) { - let tf = flop / (best[v] * 1000000.0lf) - var tail = "timing-only" - if (compared[v]) { - read_back_y(raws[v], last_out, y_bytes, y_arm) - tail = "max rel diff {max_rel_diff(y_arm, y_ref):.5f} vs khr" + if (prof) { + khr_prof_submit(name, raws) + } else { + var best <- sweep_best_us(raws) + let last_out = DISPATCHES % 2 == 0 ? yd2 : yd + var y_ref : array + khr_report(name, raws, best, last_out, y_bytes, flop, y_ref) + let cpu = khr_cpu_check(wqh, wsuh, xfh, y_ref, d, n, cnt) + print("{name} khr vs CPU oracle: {cpu.off} of {cpu.samples} sampled outputs off the cell bar, max |diff| {cpu.max_abs:.5f}\n") + if (cpu.off != 0) { + g_probe_exit = 1 } - print("{name} {vnames[v]}: {best[v] / 1000.0lf} ms/dispatch {tf} TFLOP/s {tail}\n") + delete y_ref + delete best } - delete y_ref - delete y_arm - delete best delete raws } delete wqh @@ -2548,6 +2686,7 @@ def private run_ref_gemm(d, n, cnt : int) { // nolint:STYLE038 — one linear [arch(at="../ARCHITECTURE_MEASUREMENT.md#one-benchmark-rig")] def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape dispatcher + g_probe_exit = 1 //! every early return below is a run with no result row; the sweeps clear it if (!ensure_q8_batch_cls()) { print("no Vulkan device\n") return @@ -2569,14 +2708,28 @@ def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape var only = "" var cm2_fmt = -1 for (a in get_command_line_arguments()) { - if (a == "gate" || a == "down" || a == "q" || a == "kv" || a == "kvm" || a == "qkv" || a == "cm2x" || a == "ref" || a == "tl" || a == "k4" || a == "k6" || a == "k6x" || a == "khrx" || a == "mmqx") { + if (a == "gate" || a == "down" || a == "q" || a == "kv" || a == "kvm" || a == "qkv" || a == "cm2x" || a == "ref" || a == "tl" || a == "k4" || a == "k6" || a == "k6x" || a == "khrx" || a == "khrprof" || a == "mmqx") { only = a } elif (a |> starts_with("cm2:")) { only = "cm2:" var pf : KqFmt cm2_fmt = kq_fmt_of_name(slice(a, 4), pf) ? int(pf) : -1 + } elif (a |> starts_with("khrprof:")) { + only = "khrprof" + g_prof_arm = slice(a, 8) } } + if (only == "khrprof" && g_prof_arm != "") { + var known = false + for (vn in KHR_ARM_NAMES) { + known = known || strip(vn) == g_prof_arm + } + if (!known) { + print("khrprof: no arm named '{g_prof_arm}' - the arms are {KHR_ARM_NAMES}\n") + return + } + } + g_probe_exit = 0 if (only == "cm2:") { if (!g_gpu.has_coopmat2) { print("no coopmat2 on this device\n") @@ -2669,12 +2822,20 @@ def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape return } if (only == "khrx") { - verify(ensure_khrpx_nil() && ensure_khrpx_ship() && ensure_khrpx_inline() && ensure_kq_batch_k4_khr_cls() - && ensure_khrpx_nob() && ensure_khrpx_k64() && ensure_khrpx_slab() + verify(ensure_khrpx_nil() && ensure_khrpx_ship() && ensure_khrpx_dec4() && ensure_kq_batch_k4_khr_cls() + && ensure_khrpx_f32a() && ensure_khrpx_strip() && ensure_khrpx_wg128() && kq_batch_cls_ensure(int(KqFmt.k4), false), "KHR probe pipelines must engage") run_khr_shape("q4k gate", 9728, 2560, 512) run_khr_shape("q4k down", 2560, 9728, 512) run_khr_shape("q4k q/wo", 4096, 2560, 512) + run_khr_shape("q4k gate 300", 9728, 2560, 300) //! a partial window: the last token tile takes the edge store + return + } + if (only == "khrprof") { + verify(ensure_khrpx_nil() && ensure_khrpx_ship() && ensure_khrpx_dec4() && ensure_kq_batch_k4_khr_cls() + && ensure_khrpx_f32a() && ensure_khrpx_strip() && ensure_khrpx_wg128() + && kq_batch_cls_ensure(int(KqFmt.k4), false), "KHR probe pipelines must engage") + run_khr_shape("q4k gate", 9728, 2560, 512, true) return } if (only == "cm2x") { @@ -2709,6 +2870,7 @@ def private run_probe { // nolint:STYLE037,STYLE038 — the flat per-arg shape } [export] -def main { +def main : int { run_probe() + return g_probe_exit } diff --git a/modules/dasLLAMA/performance/REVIEW.md b/modules/dasLLAMA/performance/REVIEW.md index 4b7f8f50d0..1571345ee5 100644 --- a/modules/dasLLAMA/performance/REVIEW.md +++ b/modules/dasLLAMA/performance/REVIEW.md @@ -5,19 +5,19 @@ docs: `../ARCHITECTURE.md`, `../ARCHITECTURE_ENGINE.md`, `../ARCHITECTURE_MEASUR Planned work: `../followup_general.md`. **Never add a second validator for exchange submissions (record stores and tune sidecars) - -validate through `../dasllama/dasllama_exchange_schema.das` instead.** The engine-free half (no -`dasllama/` require beyond the lint macro module) is `REVIEW.das`'s to enforce; weakening -that gate is a defect. +validate through `../dasllama/dasllama_exchange_schema.das` instead.** -**Weakening any check in `REVIEW.das` - the conditions it fires on - is a defect; the fix for a -red is a re-mint on a quiet, session-free box, never an edit.** +**A diff that narrows any `REVIEW.das` check - the files it walks, the names it does not flag - +ledgers the excluded scope in `../ARCHITECTURE_ENGINE.md` or `../ARCHITECTURE_MEASUREMENT.md` +in the same change.** -**Narrowing the scope of any `REVIEW.das` check - the files it walks, the names it does not -flag - is a defect unless `../ARCHITECTURE_ENGINE.md` or `../ARCHITECTURE_MEASUREMENT.md` -ledgers the excluded scope in the same change.** What each check enforces is -read from the gate itself; each check's finding text states its own rule. The -single-exchange-client check walks the engine (`../dasllama/`); a measurement harness talking -HTTP to a reference server is the ledgered exclusion. +**A diff that weakens any `REVIEW.das` check in any other way - the conditions it fires on - is +a defect.** + +**A diff that answers a `REVIEW.das` red on the conditions a `records/` row, an archived +sidecar, a `defaults/` profile, or `last_known_good_sidecar.json` was measured under - box +noise, a remote-desktop session, the release it was minted at - by hand-editing that artifact +is a defect: re-mint it on a quiet, session-free box instead.** **A diff that writes a commit stamp anywhere under this folder naming a commit the branch under review cannot reach is a defect - re-mint, or re-stamp to a reachable commit whose @@ -31,41 +31,52 @@ renames the file and repoints every `records/.json` row whose `tune_sha` na name, in the same change.** The archive is content-addressed; a row left on the old name points at a file that no longer exists. -**A diff that writes a reference-engine row to `records/` whose `sha` names anything but the -standing ref pin (`DEFAULT_REF_SHA`, `../benchmarks/setup_lcpp_ref.das`) is a defect - -re-mint.** A reference row that carries no `sha` is pinned by the builder its record names, -and that builder is the ref pin's checkout - the cli tools and the reference server by the -checkout the record's provenance spells, the python legs by -`../benchmarks/asr/requirements-*.txt`. +**A diff that writes a reference-engine row - a run row whose `engine` is not `das` - to +`records/` whose `sha` names anything but the standing ref pin (`DEFAULT_REF_SHA`, +`../benchmarks/setup_lcpp_ref.das`) is a defect - re-mint.** + +**A reference-engine row that carries no `sha` names, in its provenance, the checkout that +built the binary it timed; a python leg names `../benchmarks/asr/requirements-*.txt` instead.** **A diff that writes a records row, sidecar archive, or `defaults/` profile under this -folder whose `provenance.dasllama_version` differs from, or is absent where, -`DASLLAMA_RELEASE` (`../dasllama/dasllama_version.das`) is a defect - re-mint.** For a -sidecar with an `engine_sha`, read the value at that commit; a `defaults/` profile compares -against the branch under review. A ruler record pins its engines through `meta.das_sha`, and -through `meta.lcpp_version` when a reference arm ran (`-` when none did). +folder whose version pin is missing, or differs from `DASLLAMA_RELEASE` +(`../dasllama/dasllama_version.das`), is a defect - re-mint.** The pin is a records row's +`dasllama_version`, and `provenance.dasllama_version` in a sidecar archive or a `defaults/` +profile. For a sidecar with an `engine_sha`, read the value at that commit; a `defaults/` +profile compares against the branch under review. **A diff that writes a row to `records/.json` mints that row from a board cell.** A board cell is one `gen_bench_records.das` spawns, or a manual `../benchmarks/lcpp_bench.das` cell -its `../PROFILE.md` section documents. A timing taken any other way stays out of `records/` -and settles its own decision in the report where it was taken. +its `../PROFILE.md` section documents. A timing taken any other way settles its own decision +in the report where it was taken. + +**Only the reference cells of `gen_bench_records.das` - the cells that time, over a board +workload, a program this repository does not build - write a reference-engine row into +`records/.json`.** **A file under `records/mtp/` is written only by `../harness/mtp_ruler.das`, never by hand; a diff that adds one names the ruler command line in the PR body.** The ruler record is the -speculative round's cell (`../ARCHITECTURE_MEASUREMENT.md` sec.2.28); its shape is the ruler's, +speculative round's cell (`../ARCHITECTURE_MEASUREMENT.md` sec.2.45); its shape is the ruler's, and the board walkers (`list_record_stores`) do not read it. -**A `records/mtp/` file that carries a reference-engine row with no our-engine row from the same -run is a defect - re-mint the pair.** A file with our-engine rows alone conforms. +**A `records/mtp/` file names its engines in `meta.das_sha`, and in `meta.lcpp_version` when +a reference arm ran (`-` when none did).** -**A `records/mtp/` file whose `meta.settle` is below the ruler's default names the reason in the -PR body, and a diff that adds one names its rows `direction-grade` wherever it cites them.** +**A `records/mtp/` file that carries a reference-engine row with no `das` row from the same run +is a defect - re-mint the pair.** -**A diff that writes a `das` row to `records/.json` times that row with the released -`lcpp_bench` exe.** That exe is `../benchmarks/lcpp_bench.das` built by `daspkg release`. +**A `records/mtp/` file whose `meta.settle` is below the ruler's default names the reason in +the PR body.** -**A diff that writes a reference-engine row to `records/.json` times that row with the -reference exe the ref pin builds.** +**A diff that adds a `records/mtp/` file whose `meta.settle` is below the ruler's default names +its rows `direction-grade` wherever it cites them.** + +**A diff that writes a `das` row - a run row whose `engine` is `das` - to `records/.json` +times that row with the released `lcpp_bench` exe.** That exe is +`../benchmarks/lcpp_bench.das` built by `daspkg release`. + +**A diff that writes a `llama.cpp` row - a run row whose `engine` is `llama.cpp` - to +`records/.json` times that row with the reference exe the ref pin builds.** **A field added to what `write_bench_records` (`profile_common.das`) writes is added to `../dasllama/dasllama_exchange_schema.das`'s run validation in the same change** - the @@ -80,23 +91,26 @@ second tool's record carrying the wrong engine, and looks real. board membership, provenance, or parity fixtures is a defect - write it as a view over those two functions.** Board membership is which models the site results board shows. -**A view over `model_specs()` or `asr_catalog()` recomputes from them on every call, selects -rows by one field whose value on the row states membership, and stores no -`url`/`bytes`/`sha256` of its own; a view that matches a field against a list of literal -values - file names, name prefixes, recipe constants - is a defect.** A literal list is a -second catalog that drifts from the first. +**A view over `model_specs()` or `asr_catalog()` recomputes from them on every call and stores +no `url`, `bytes`, or `sha256` of its own.** + +**A view over `model_specs()` or `asr_catalog()` selects its rows by one field whose value on +the row states membership; a view that matches a field against a list of literal values - file +names, name prefixes, recipe constants - is a defect.** A literal list is a second catalog that +drifts from the first. -**A diff that makes a recorded row or manifest under this folder pin a model file keeps that -file's provenance on its own row.** The row is a row of `model_specs()` (`model_specs.das`) or -of `asr_catalog()` (`profile_common.das`). The row carries the `url` + `bytes` + `sha256` -itself, or a `recipe` a reader can run. One named accessor call may stand in for the row: a -function in `model_specs.das` whose own body carries those three fields. A second hop does not -count - an accessor forwarding to another accessor, or an unnamed table lookup. +**A diff that makes a records row, a sidecar archive or a `defaults/` profile under this +folder pin a model file keeps that file's provenance on its own row.** The row is a row of +`model_specs()` (`model_specs.das`) or of `asr_catalog()` (`profile_common.das`). The row +carries the `url` + `bytes` + `sha256` itself, or a `recipe` a reader can run. One named +accessor call may stand in for the row: a function in `model_specs.das` whose own body carries +those three fields. A second hop does not count - an accessor forwarding to another accessor, +or an unnamed table lookup. **A diff that adds a companion artifact - a file fetched or verified with a model and consumed beside it: a projector, a draft head, an assistant sidecar, an image fixture - puts it in the -`companions` of the row that pins its carrier, and names it from every other row that consumes -it.** Uniqueness itself is `../tests/test_model_specs.das`'s to enforce. +`companions` of the row that pins its carrier, and names it from every other row that +consumes it.** **A diff that changes what any `serve_*` function in `model_specs.das` returns - a `serve_*` field on a row, the function's body, or a `companions` entry with a `url` on a row a `serve_*` @@ -105,8 +119,8 @@ change** - the serving catalog is a view over these rows, so its gates red on a ships. **A convert, a bench, or a tune-state write reached from `fetch_models.das --fetch` is a -defect - `--fetch` downloads only.** Each has its own home: a conversion recipe runs under -`--convert`, a timing runs in a board cell (`gen_bench_records.das` or a +defect - `--fetch` downloads only.** Each has its own home: a conversion recipe runs +under `--convert`, a timing runs in a board cell (`gen_bench_records.das` or a `../benchmarks/lcpp_bench.das` cell), and a tune sidecar is written under a `--tune` run. **A diff that adds a row or a `companions` entry, or changes a `bytes`, `sha256` or `recipe` @@ -116,10 +130,10 @@ row's model file, ending `0 pending, 0 failed` with the row reported `ok`, plus `fetch_models.das --` run in which no row the diff touched is `pending`.** A box stocks only some of the rows, so an unscoped run's `pending` count is the box's. -**A diff that changes the `recipe` of a row carrying no `sha256` records, in the PR -description, the conversion command as run and the produced file's identity - its byte size, -or a `file_identity` hash.** `fetch_models` reports such a row `ok` on presence alone, so its -run cannot tell a re-mint from the stale file the old recipe made. +**A diff that adds a `recipe` row carrying no `sha256`, or changes such a row's `recipe`, +records, in the PR description, the conversion command as run and the produced file's +identity - its byte size, or a `file_identity` hash.** `fetch_models` reports such a row `ok` +on presence alone, so its run cannot tell a re-mint from the stale file the old recipe made. **A diff that changes `fetch_models.das` beyond its comments records its settling evidence in the PR description: one unscoped `fetch_models.das --` run ending `0 failed`.** diff --git a/modules/dasLLAMA/performance/model_specs.das b/modules/dasLLAMA/performance/model_specs.das index a55e9bf3c0..e71bc4588e 100644 --- a/modules/dasLLAMA/performance/model_specs.das +++ b/modules/dasLLAMA/performance/model_specs.das @@ -308,6 +308,9 @@ def model_specs() : array { // nolint:STYLE038 — flat model-set t // out plane q4_K - the step leaves the o row f32 for the superblock out plane's requant ModelSpec(file = "Qwen3.5-0.8B-Q4_K_M.gguf", recipe = "llama-quantize --allow-requantize Qwen3.5-0.8B-Q8_0.gguf Qwen3.5-0.8B-Q4_K_M.gguf Q4_K_M"), + // the mixed twin: the deltanet out plane at Q8_0 (the unsloth UD shape) - the one carrier whose x and o feeds part ways + ModelSpec(file = "Qwen3.5-0.8B-Q4_K_M-q8out.gguf", + recipe = "llama-quantize --allow-requantize --tensor-type ssm_out=q8_0 Qwen3.5-0.8B-Q8_0.gguf Qwen3.5-0.8B-Q4_K_M-q8out.gguf Q4_K_M"), // the 9B hybrid with its in-file MTP block, the whole-model driver's parity carrier // (head 256 gated attention, partial rope, Q5_K/Q6_K deltanet planes) and the vulkan // serving census's head-256 row; the local name carries the MTP tag the release's does not diff --git a/modules/dasLLAMA/tests/CLAUDE.md b/modules/dasLLAMA/tests/CLAUDE.md index 1c42b85471..55ee304818 100644 --- a/modules/dasLLAMA/tests/CLAUDE.md +++ b/modules/dasLLAMA/tests/CLAUDE.md @@ -10,7 +10,7 @@ one-arm fix into an afternoon. ./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --arm [--suite decode|mtp|prefill|matrix|kernels|image|image-vulkan|coverage|all] [--family llama] ./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --suite model-free # the per-PR gate that needs no models - runs the same on a bare box, no --arm ./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --suite stocked # the per-PR gate on a box with models: the model-gated files, no --arm -./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --suite stocked --exclude test_ple_modes # the iteration form - drops the ~10 min PLE file +./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --suite stocked --exclude test_ple_modes # the iteration form - drops the PLE file ./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --changed [--base origin/master] # after an edit: the areas the changed files reach ./bin/daslang -jit modules/dasLLAMA/tests/run.das -- --area audio # one area: audio | vision | tts | llm | infra (comma list) ``` @@ -44,7 +44,7 @@ every child runs with `DAS_TUNE_POLICY=reference`, so the `[tune]` families and hints fall to their reference bodies - the portable tier's arm of the gate. That arm also runs with `DASLLAMA_IMAGE=0`: the reference policy is a different box identity, and a `.dlim` minted under it would GC-purge the box's tuned images, so the runner refuses `--no-tune` with the -`image` and `image-vulkan` suites, and the one image-reading cell outside them +`image` and `image-vulkan` suites, and the image-reading cell outside them (`test_audio_embedder`'s direct-route cell) skips on the knob and keeps its coverage on the tuned arm. The runner redirects the COMPLETE output to a log file, and prints that path on the DONE line. It owns the dastest @@ -134,7 +134,8 @@ so at npos % 32 == 2 the 576-wide sites must refuse while exactly the hidden-kdi equality is not a valid instrument on a 135M); span = the non-causal media eval shape, head + embd span, per codec; span-fused = the image turn's ONE-eval shape - causal head + media rows + causal tail through the per-query mask, one GPU prefill, with a same-backend fused-vs-splice logits -witness on poison-calibrated per-codec bars; span-mrope = the qwen grid-roped turn - one GPU +witness on poison-calibrated per-codec bars (a poison is a value added to the expected result +that must red the bar); span-mrope = the qwen grid-roped turn - one GPU prefill via the per-row-table capability, token-exact vs the all-CPU control, plus a same-backend grid-vs-sequential prefill-logits witness; span-ds = the deepstack turn - one GPU prefill via the slice-add capability, token parity plus the add-CONTRIBUTION witness: @@ -172,8 +173,9 @@ gemma4uv` selects it too - arm filters match by substring); `mtower` is the whis tower-blocks gate, Apple builds only - whisper tiny + large-v3-turbo transcript-exact and qwen3a f32-rail transcript equality, CPU vs GPU, with geometry-derived counter deltas, the twin-W legs - `wblob`, the halfword copy of a tower's GEMM weights baked beside the f32 blob and -read only where a tensor crown compiled the half GEMMs (`../ARCHITECTURE_IMAGE.md` sec 2.1i) -- on whisper f16 + qwen3a bf16, each engaging by the route counter, plus whisper's own +read only where a tensor crown - the pin that compiles a family's half-precision GEMM twins +(`set_metal_tensor_crowns`) - compiled the half GEMMs - on whisper f16 + qwen3a bf16, each +engaging by the route counter, plus whisper's own twin-knob freeze and whisper's own wblob-ONLY poison that must CHANGE the GPU transcript (both legs are whisper's alone; qwen3a carries neither), the gemma4a Metal Conformer cell (f32-lane transcript equality CPU vs GPU + encode rel-rms + counter deltas - @@ -182,7 +184,7 @@ same discipline over the rel-pos XL block loop; decoder = the q8_0 serving artif the tower q8-decline - a q8 whisper encoder never dispatches and records the `quant_mode` decline, and the whisper serving default IS q8 unless `set_asr_fp32` / `set_asr_tower_fp32` asks for f32 (whisper carries no lane policy). Canary and gemma4a do: un-pinned, their lane -follows whether the Metal tower would serve (`../ARCHITECTURE_MEDIA.md` sec 2.15). +follows whether the Metal tower would serve. Then the required-mode panic and Conformer-absence (parakeet) cells; the arm's DECODER half is the `test_whisper_metal_cross_kv` cell in `test_model_image.das` - GPU cross-KV on the q8 serving default, transcript-exact against the CPU chain with window/step counter deltas and the knob and quant_mode declines, @@ -226,8 +228,8 @@ the device-free rail unit; the serving vulkan census runs on the PC box. ## Metal kernel gates The `kernels` suite (test_metal_{prefill,decode,rope,gemv,misc,attn,gemm}_kernels - model-less -per-class CPU-oracle units covering the FULL metal kernel census, ~2-3 min) has no arms; -remember it exists (the hand-bound-gate sync obligation is `REVIEW_KERNEL_CELLS.md`'s). The misc file also +per-class CPU-oracle units covering the FULL metal kernel census, ~2-3 min) has no arms. The +hand-bound-gate sync obligation is `REVIEW_KERNEL_CELLS.md`'s. The misc file also carries `test_lens_tgmem_gate` - not a CPU-oracle unit: it spawns two `daslang -compile-only` child builds (up to 120 s each) proving the lens refuses a `[metal_dispatch]` class with `@workgroup` members and no `tgmem=`, twin fixture as the must-compile control. Shared fixtures @@ -266,8 +268,10 @@ through its registry. device-side f16 gather, the streamed-group slot hand-off, and the streamed split's async head. `test_vulkan_kernels.das` - model-free (a Vulkan device, else skips): the per-class CPU-oracle units of the Vulkan kernel census (`_vkd_oracles.das` runs the class methods on the CPU as the -oracle; `_vkd_toy.das` is the `[vk_dispatch]` bring-up fixture). The thirteen per-format tile -cells (`test_vkd__cm2_batch`) run four arms: the cm2 l/m/s tiles in mode 4 on an +oracle; `_vkd_toy.das` is the `[vk_dispatch]` bring-up fixture). The per-format tile cells +(`test_vkd__cm2_batch`, one per `kq_sb` format; q8's cm2 tiles ride their own fmt-0 cells +`test_vkd_cm2l_batch` / `test_vkd_cm2m_batch` / `test_vkd_cm2s_batch`, which carry no KHR arm, +and q51 carries no tile cell) run four arms: the cm2 l/m/s tiles in mode 4 on an NV_coopmat2 device and the KHR 128x128 tile wherever the device has KHR coopmat at subgroup 32 - the cell skips only when the device has neither, so a KHR-only card still runs its arm; `test_vkd_direct_decode` @@ -329,26 +333,53 @@ from a Config or a synthetic Model shell (`resident_unserved_features`, shape is named in the text a user reads, a served one yields ""; plus the KV mirror's binding cap (`resident_binding_ctx`) on a hybrid shell whose layer 0 is recurrent, its dense twin, and a shell with no attention layer. -`test_gpu_resident_hybrid.das` - stocked suite; the whole-model resident driver on a deltanet -hybrid (Qwen3.5-0.8B-Q8_0, `DASLLAMA_GPU=1`): the resident window chain prefills (recurrent -layers through conv + chunked scan on device state, gated partial-rope attention over the -device K/V mirror) and the resident decode advances the recurrent layers on device; forced-feed -logits within the 4% deltanet bar of the all-CPU chain (the model dropped off the device) after -the prefill and at every step, the one-step-off compare against the CPU chain's next step as -the must-EXCEED control, and counter witnesses that the resident tier armed and that the -prefill served on the device; one-window (a 40-row prompt, inside one window), two-window (a -prompt one window plus 88 rows long), one-past-the-window (one row past one window: the chain -shortens the preceding window so the last one still carries the conv taps) and two-token (a -prompt shorter than the conv taps: the conv history ring's leading rows are zero) cells; skips -without the model or the armed tier. The K-quant twin (`Qwen3.5-0.8B-Q4_K_M.gguf`, minted from -the Q8_0 by the recipe its `../performance/model_specs.das` row carries) runs the same one-window -and two-window cells with the deltanet qkv (q6_K), z (q4_K) and out (q4_K) planes in their file -formats on the driver, asserts the loader kept them so, and holds a 6% bar (the K-quant chain's device-vs-CPU -noise runs near double the Q8 file's, flat across steps). Both files make their sessions on the -mirror codec the box arms, so under `DASLLAMA_VK_KV32=1` the Q8 cells run on the f32 mirrors and -the K-quant cells skip (their bar is calibrated on the f16 mirror). Run under `DASLLAMA_COOPMAT=mm` the -same file is the KHR arm's end-to-end gate: the K-quant twin's planes then prefill on the KHR kq -tile (mode 3), and the bars hold there too. + +`test_gpu_resident_hybrid.das` - stocked suite, `-jit` only; the whole-model resident driver on a +deltanet hybrid under `DASLLAMA_GPU=1`. Each fixture is a row in `../performance/model_specs.das`: +the Q8 carrier, the K-quant twin, the mixed twin. + +The Q8 carrier is `Qwen3.5-0.8B-Q8_0.gguf`. The resident window chain prefills it - the recurrent +layers through conv + chunked scan on device state, the gated partial-rope attention over the +device K/V mirror - and the resident decode advances the recurrent layers on device. The gate is +the forced-feed logits-tolerance form: the logits sit within the 4% deltanet bar of the all-CPU +chain (the model dropped off the device) after the prefill and at every step. The control is the +one-step-off compare - each step's resident logits against the CPU chain's PREVIOUS step - which +must EXCEED the bar. Counter witnesses hold that the resident tier armed and that the prefill +served on the device. The cells: one window (a 40-row prompt), two windows (a prompt one window +plus 88 rows long), one row past one window (the chain shortens the preceding window, so the last +one still carries the conv taps) and two tokens (a prompt shorter than the conv taps: the conv +history ring's leading rows are zero). The file skips without the model or the armed tier. + +The K-quant twin is `Qwen3.5-0.8B-Q4_K_M.gguf`, minted from the Q8_0 by the recipe its row +carries. It runs the one-window and two-window cells with the deltanet qkv (q6_K), z (q4_K) and +out (q4_K) planes in their file formats on the driver, and asserts the loader kept them off Q8_0. It +holds a 6% bar - 10% where the prefill takes the quant feed. The quant feed is the case where no +coopmat tile serves the planes, so the prefill's activations ride Q8_K. The K-quant chain's +device-vs-CPU noise runs near double the Q8 file's, flat across steps. + +The mixed twin is `Qwen3.5-0.8B-Q4_K_M-q8out.gguf`, the same mint with the deltanet out plane at +Q8_0 - the shape the published Unsloth dynamic quants take. Its recurrent layers decide two feeds +separately - the x feed (the rows the qkv and z GEMMs read) from the K-quant qkv/z planes, the o +feed (the rows the out GEMM reads) from the Q8_0 out plane - so on the KHR arm x rides the f16 +feed while o falls to the Q8_0 requant. It runs the same one-window and two-window cells, on the same +bars and with the same `DASLLAMA_VK_KV32=1` skip. Every K-quant cell - the twin's two and the +mixed twin's two - carries the routing witness: the prefill's Q8_K requant census count, read +through `vk_kernel_coverage_of("cls_q8k_rq_spv")`, below the recurrent-layer count on the f16 +feed and at or above it off the feed. The mixed twin is the fixture that discriminates: a +per-layer feed decision would send qkv/z to the sdot4 tile for the out plane's sake, and the +witness reds it there. The witness reads `RQ_NO_CENSUS` and stands down in a build with no vulkan +module. + +Every fixture makes its session on the mirror codec the box arms. Under `DASLLAMA_VK_KV32=1` the +Q8 cells run on the f32 mirrors and the K-quant cells skip - their bar is calibrated on the f16 +mirror. Under `DASLLAMA_COOPMAT=mm` this file is the KHR arm's end-to-end gate: the K-quant +planes prefill on the KHR kq tile (mode 3), and the 6% bar holds there too. +`DASLLAMA_COOPMAT=sdot4` names the integer dot tile and forces the quant feed, so the 10% bar +applies there. + +One cell is model-free: `test_kernel_census_by_name` holds that the census accessor panics on a +kernel name nothing seeded, so a misspelt key cannot read as a zero count. + `test_gpu_resident_qwen2.das` - stocked suite; the whole-model resident driver on a qwen2 (Qwen2.5-0.5B-Instruct-Q8_0, `DASLLAMA_GPU=1`): the q/k/v projection bias folded into the rope stage on the device - the hybrid file's forced-feed logits-tolerance form (its K-quant 6% bar, @@ -440,7 +471,7 @@ lossless; reports SKIPPED where the vocab is not stocked. One cell is model-free reading `add_bos == false` where that fixture is stocked. `test_exe_smoke.das` - stocked suite; model-gated (SmolLM2-135M, small tier): the standalone-exe context gate. Builds `_exe_smoke_root.das` with `-jit -exe` and runs the -artifact - the one rail where globals restore as DATA, so a function-typed global with no +artifact - the rail where globals restore as DATA, so a function-typed global with no boot-restore `[init]` dies on its first invoke while every `-jit` suite stays green. ~90 s. `test_gen_records_args.das` - model-free: the measurement orchestrator's pure seams - the pybench args builder's per-tool arms (onnx carries the `--out` recovery file and never a @@ -623,7 +654,7 @@ the sentence must not render an empty code), the WAV container, the codec's malf astral arms, kitten's dropped-symbol rule, and the `rtf` guard; model-gated (`kitten-nano.gguf` + the front-end packs): the streaming form's chunks concatenate to the buffered synthesis sample for sample, one synthesis at one lane and the other at the box's lanes, the phonemizer on the corpus -rail, the model-keyed chunker's Kitten arm (the driver's comma - the one arm no model-free cell +rail, the model-keyed chunker's Kitten arm (the driver's comma - the arm no model-free cell reaches), and the language form of `tts_phonemize` - the declared language reads as the bare form does, an undeclared one panics at the call site. `test_tts_blocks.das` - model-free: the block home's two layouts against each other - every @@ -651,7 +682,7 @@ each against an in-test reference. unit of their own, each against an in-test fp64 reference over a procedural seeded-LCG fixture and each bar carrying its own must-EXCEED poison arm - the padded-width GEMM wrappers (`mm_blob_b` on an owned blob and on a borrowed plane, `mm_bf16_b` on bf16-exact operands, -`mm_plane_b`'s per-tensor routing, and the sec.2.13 claim that a zero-padded served width is +`mm_plane_b`'s per-tensor routing, and the claim that a zero-padded served width is bit-identical to the unpadded GEMM), `layernorm` and both `layernorm_batch` overloads at a dim off every lane width, `add_bias_rows` and `add_inplace_rows` against the exact scalar loop at lengths off the float4 block, `gelu_erf_batch` against an fp64 erf series and against its own @@ -703,8 +734,7 @@ remainder strips and the pure 4x16 tile, each against an fp64 GEMM that starts f pre-initialized C (so the accumulate contract is part of the claim) and against each other bit-for-bit. Every tolerance bar in the file ships its control in the same cell: the expected value offset by an added 0.01, which must land outside the bar. -`test_q8q8_family.das` - model-free: the q8q8 kernel family end to end, promoted from the -hand-run `harness/gen_parity_probe.das` / `gen_slot_parity_probe.das`. Six widths with tails +`test_q8q8_family.das` - model-free: the q8q8 kernel family end to end. Six widths with tails (64, 96, 512, 1024, 1056, 3072) across five cells, each judged by an in-test fp64 dequant reference (int8 products summed exactly, both block scales applied in double) whose bar is the per-block envelope times the block count, and each bar carrying a poison leg - 0.25 ADDED to @@ -732,10 +762,10 @@ added-value poison that must exceed it. The `_tab` forms also ride a tight twin un-tabled forms (loose only by the cross-compilation-unit cos/sin ulp drift), the `_part` forms are bit-exact against a full apply over the gathered rotated prefix with the un-rotated dims proven to pass through, and `rope_apply` is bit-exact against the leaf its `neox` flag names, -including the `use_ff = false` p-RoPE arm. These kernels were previously only the Apple-only -Metal rope tests' oracle; this file makes them a subject on every platform. -`test_prefill_cpu_kernels.das` - model-free: the prefill and KV CPU kernels no suite gated -before, on q4_K / q6_K / q4_0 synthetic disk planes built in-file. `matmul_kq_batch` and +including the `use_ff = false` p-RoPE arm. The Apple-only Metal rope tests use these kernels as +their oracle; this file gates them on every platform. +`test_prefill_cpu_kernels.das` - model-free: the prefill and KV CPU kernels, on q4_K / q6_K / +q4_0 synthetic disk planes built in-file. `matmul_kq_batch` and `matmul_kq_batch_groupn` - the per-position and per-expert GEMV routes a tier with no kq batch slot runs, bit-matched against per-(token,row) and per-(region,token) disk dots, with no skip on any tier; where a kq-carrying backend can be pinned (restored on exit) the native batched @@ -856,7 +886,7 @@ instead - the forced stream plus the GPU's greedy would-be picks, or both next-t `cached_ids`/`cached_vals` pin a CPU trajectory into `.ref..tsv` beside the model. FREEFORM-prompt caches sit on sub-noise near-ties, so any numerics-adjacent master merge can legitimately move the CURRENT CPU trajectory off the cached one - the parity assert then fails -with the GPU side actually CORRECT (it matches today's CPU). Before declaring a fam-row red a +with the GPU side actually CORRECT (it matches the current CPU chain). Before declaring a fam-row red a regression: (1) stash + clean-tree rerun (same red => not your diff), (2) `mv` the cell's `.ref` tsv aside and rerun - a fresh-truth green means stale cache, keep the refreshed tsv. Counting caches are tie-proof by construction and should NOT move; a counting-cache mismatch is a real diff --git a/modules/dasLLAMA/tests/REVIEW.md b/modules/dasLLAMA/tests/REVIEW.md index 0833f6c51e..d44d07bbc9 100644 --- a/modules/dasLLAMA/tests/REVIEW.md +++ b/modules/dasLLAMA/tests/REVIEW.md @@ -6,8 +6,9 @@ doc: `CLAUDE.md`. Planned work: `../followup_general.md`, `../followup_vulkan.md **A kernel-unit cell - a model-less cell that dispatches one kernel class and asserts on its output - or a gate that hand-dispatches or hand-binds a kernel, wherever the diff puts it, and -a diff that changes a `[metal_dispatch]` or `[vk_dispatch]` class's dispatch geometry, kargs, -or kargs fields, apply `REVIEW_KERNEL_CELLS.md` (beside this file) together with this list.** +a diff that changes a `[metal_dispatch]` or `[vk_dispatch]` class's dispatch geometry, its +kernel-argument struct (`kargs`) or that struct's fields, apply `REVIEW_KERNEL_CELLS.md` +(beside this file) together with this list.** **Every PR runs `run.das -- --suite model-free` and `run.das -- --suite stocked` on a box with the models stocked, plus every test here the change reaches - never the whole directory.** A @@ -19,18 +20,20 @@ asserts on; a comment-only edit reaches none. the iteration form between PRs; a PR that ships on it never ran the PLE coverage. **A test file - a `.das` in this folder that dastest runs: one carrying at least one `[test]` -function, or one whose `cant_`, `failed_` or `invalid_` prefix makes its compile the assertion -- whose cells cannot hold under `DASLLAMA_CPU_PREFILL=1` says so in its header and sits in no -`run.das` suite; every other test file in this folder sits in one.** `DASLLAMA_CPU_PREFILL=1` -is what the runner arms for every suite. +function, or one whose `cant_`, `failed_` or `invalid_` prefix makes its compile the +assertion - whose cells cannot hold under `DASLLAMA_CPU_PREFILL=1` says so in its header and +joins the exempt list of `test_run_suites.das`'s suite-membership gate in the same change; +weakening that gate is a defect.** `DASLLAMA_CPU_PREFILL=1` is what the runner arms for every +suite. -**Invoking dastest directly on a test file in a `run.das` model suite (every suite but -`model-free` and `stocked`) is a defect - run it through `run.das`.** +**Invoking dastest directly on a test file in a `run.das` suite other than `model-free` and +`stocked` is a defect - run it through `run.das`.** -**`run.das` runs nothing on require - no `[init]`, and no global whose initializer spawns, -logs, writes the environment or touches the filesystem; a diff that adds one is a defect.** -`test_run_suites.das` and `test_run_summary.das` require `run` by bare same-dir name, so -anything that fires on require fires inside every one of those test processes. +**`run.das` declares no global whose initializer spawns, logs, writes the environment or +touches the filesystem; a diff that adds one is a defect, and weakening `test_run_suites.das`'s +no-`[init]` check is a defect.** `test_run_suites.das` and `test_run_summary.das` require `run` +by bare same-dir name, so anything that fires on require fires inside every one of those test +processes. **A cell asserting a chat template's INSTRUCT wire - a closed empty thought block and no thinking gate - calls `set_thinking(c, false)` on its `ChatSession` before the first turn.** @@ -45,15 +48,15 @@ a run of skips is not the coverage the suite owes. `run.das` suite listing is the only registration these files get.** **A diff that adds, removes or moves a gate - one test cell, or a file between suites - updates -the `CLAUDE.md` census entry of the file it lands in, in the same change.** A brace list or a -suite roster that only names the file carries nothing to correct. +the `CLAUDE.md` census entry of the file it lands in, in the same change.** A `{a,b}` shorthand +naming several files at once, or a suite roster, carries nothing to correct. **A diff that changes the contract a gate pins - what its asserts hold fixed, an axis gained or lost - updates that gate's entry in this checklist's pinned set in the same change.** -**A diff that changes a gate's skip condition - the model, fixture, device or arm it needs - -updates the test file's own header and the `CLAUDE.md` clause that states it, in the same -change.** +**A diff that adds, changes, or drops a gate's skip condition - the model, fixture, device or +arm it needs - updates the test file's own header in the same change, and adds or corrects the +skip clause in that file's `CLAUDE.md` entry where `CLAUDE.md` carries one.** **A diff that adds, moves, or removes a `[test]` file outside `modules/dasLLAMA/` that carries a `require dasllama/...` line of its own adds, corrects, or drops its row, with the reason it @@ -93,7 +96,8 @@ wire-key pin read out of `../dasllama/dasllama_tune_scope.das`) and `test_scheduler.das`'s media-stream bypass check (no cached hit at `prefix_attach`, no donated pages at `donate_stream`); `test_vulkan_kernels.das`'s tile-pick cell (which tile the Vulkan matmul picks for a given width, row count and coopmat mode, and whether that dispatch splits its -reduction across partial planes); `utils/dasllama-server/test_worker_dispatch.das` (repo root: worker-local fork pools, shared queue policy). +reduction across partial planes); `utils/dasllama-server/test_worker_dispatch.das` (repo root) - +worker-local fork pools, shared queue policy. **A diff that adds a gate whose failure means a documented contract changed, rather than a kernel regressing, adds it to the pinned set in the same change** - as a file when every @@ -125,11 +129,8 @@ and their total size is under `LARGE_TIER_BYTES` unless `DASLLAMA_PARITY_FULL=1` other stocked fixture gates on its own presence. **A test - or a program a test builds or spawns - whose subject is not the `.dlim` image rail -never mints or maps an image: it either runs with `DASLLAMA_IMAGE=0` in its environment, or -calls no `load_model`, `load_model_cached`, or `load_model_image` and loads each carrier through -that carrier's own loader.** Decoders: `load_model_` (`../dasllama/dasllama_load.das`); other -carriers: `load__tower` / `load__encoder` / `load__model`; TTS: -`load_tts_model` or `load_styletts2`. +never mints or maps a MODEL image: it either runs with `DASLLAMA_IMAGE=0` in its environment, +or calls no `load_model`, `load_model_cached`, or `load_model_image`.** **A predicate whose value the BOX decides (a device capability, a policy default) and that therefore cannot differ between two runs on one machine is never tested through its own @@ -157,12 +158,13 @@ number.** A cap, a resize, or a counter showing the path ran is not evidence tha reached. **A freeform token-parity cell whose two sides can round differently - different lanes, -backends, batch shapes or kernel forms - is a defect, and a token-exact freeform compare -states in the cell what makes its two sides one code path: the shared entry point, or an -assert pinning the lane.** Freeform coverage across a pair that rounds differently uses the -forced-feed logits-tolerance form - the same fixed tokens fed to both sides, logits compared -within a bar. Counting cells - those whose prompt forces a continuation that cannot tie, so -greedy tokens are fixed - stay token-exact. +backends, batch shapes or kernel forms - is a defect: use the forced-feed logits-tolerance +form, the same fixed tokens fed to both sides and logits compared within a bar.** A counting +cell - one whose prompt forces a continuation that cannot tie, so greedy tokens are fixed - +stays token-exact. + +**A token-exact freeform compare states in the cell what makes its two sides one code path - +the shared entry point, or an assert pinning the lane.** **An ASR family with no token-for-token oracle cell is a defect** - the cell compares a transcript against a reference leg, external dump or CPU control alike. @@ -179,10 +181,11 @@ beside the dumps under `models_dir()`, named by the test that loads the dump. the backend, the flash-attention setting, and the mmproj precision the dump came from - is a defect.** -**A cell that does not establish every process-wide driver setter and serving-lane knob its -claim depends on, and leave every family pin unset on return - whether or not this cell set -one - and every driver setter it touched at its default, is a defect.** This holds even when -the claim needs the knob at its DEFAULT value. `reset__q8` is the restore; why a hook +**A cell sets every process-wide driver setter and serving-lane knob its claim depends on, even +when the claim needs the knob at its DEFAULT value.** + +**A cell returns with every family pin unset - whether or not this cell set one - and every +driver setter it touched back at its default; `reset__q8` is the restore.** Why a hook left set changes what the next cell measures is `CLAUDE.md`'s "Metal fixtures". **A cell claiming a family serving lane that does not pin it through the family's own lane @@ -194,10 +197,11 @@ whichever lane the box's policy picked. **A cell that loads a media carrier under a lane pin - a `set__q8`-class knob, or a `set_metal_tensor_crowns` / `pin_metal_tensor_crowns` pin - and whose subject is not that lane knob itself mints in memory through the family's `stage_*` + `mint_*` pair, never through a -`.dlim`-baking loader (`load__tower` / `load__encoder` / `load__model` / -`load_styletts2` / `load_model` / `load_model_cached` / `load_model_image`).** A disk bake under a pinned lane -GC-purges the serving lane's `.dlim` beside the model, and the next direct-image load in -another suite panics on the wrong identity. +`.dlim`-baking loader (`load__tower` / `load__encoder` / +`load__model` / `load_tts_model` / `load_styletts2` / `load_model` / +`load_model_cached` / `load_model_image`).** A disk bake under a pinned lane GC-purges the +serving lane's `.dlim` beside the model, and the next direct-image load in another suite panics +on the wrong identity. **A cell whose subject IS the lane knob (`load_asr_model` under `set_asr_tower_fp32`) loads through the `.dlim`-baking loader, never around it.** The pin is part of what the image @@ -209,10 +213,10 @@ one CPU inference reads) and the stages a decode override selects on that model' (`blob_twin(t, path, seq_cap)`, `test_metal_decode_parity.das`), in one session.** The planar model and its blob twin share one shape, so one session serves both. -**A diff that adds a model-loading block to a `run.das` MODEL suite (`decode`, `mtp`, `prefill`, -`matrix`, `image`, `image-vulkan`, `coverage` - not `model-free` or `stocked`) tags it with its -family.** The family tag is the token passed to `family_on(t, name)` (`_model_tier.das`). An -untagged block silently joins every family's gate. +**A diff that adds a model-loading block to a `run.das` suite other than `model-free`, +`stocked` and `kernels` tags it with its family.** The family tag is the token passed to +`family_on(t, name)` (`_model_tier.das`). An untagged block silently joins every family's +gate. **No CPU-control batch parity runs against `Llama-3.3-70B-Instruct-Q4_K_M.gguf`.** The batched code paths get their parity on small models, through pins. @@ -222,10 +226,9 @@ starts is a defect - set it before that process starts.** That process is a chil spawns, or the runner's own. An in-cell set is invisible to the running config, which is read once at context init. -**A cell that cannot set an environment-read knob before its reader starts, and whose text -that prints with a red - the cell label or the assert - does not name the value it asserts -under, is a defect.** An environment-read knob is one the running config reads once, at -context init. +**A cell that cannot set an environment-read knob before its reader starts names that knob's +value in the text a red prints - the cell label or the assert.** An environment-read knob is +one the running config reads once, at context init. **A cell asserting the UNPINNED default lane never compares against a hardcoded lane - it compares against the predicates the lane policy itself consults, `float_batch_override_active()` @@ -246,8 +249,9 @@ nobody else can play makes a red unreadable. A newly stocked clip joins this lis change. **A media fixture an embedder-parity cell regenerates in-test and compares against an oracle -dump, with no exact-value generator, is a defect.** A generator running libm -transcendentals is not exact-value: it is not float-portable. +dump, with no exact-value generator - one whose values are exactly representable floats, so +every box produces the same bytes - is a defect.** A generator running libm transcendentals is +not exact-value: it is not float-portable. **An embedding-parity cell that does not name its fixture, or does not log the measured maxdiff on green as well as red, is a defect.** @@ -257,9 +261,7 @@ outside the new bar.** A bar nothing has ever exceeded is not known to discrimin **A family that gains a live thinking or tool format ships its recognition tests in the same change** - the wire-shape pins, the render pins, and a live server case gated on the family's -smallest GGUF that sits under `LARGE_TIER_BYTES` (`_model_tier.das`) (the file homes are -`CLAUDE.md`'s "The per-PR suites - model-free and stocked" and "Out-of-folder test files" notes). A family -whose vocab carries no thinking or tool markers has no format to test. +smallest GGUF that sits under `LARGE_TIER_BYTES` (`_model_tier.das`). **A poison control on a tower the Metal driver serves - a run of the gate with the tower's weights zeroed, which must RED - zeroes every weight buffer the served route reads.** Which diff --git a/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md b/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md index 41d5b9392b..cdcbb4be6e 100644 --- a/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md +++ b/modules/dasLLAMA/tests/REVIEW_KERNEL_CELLS.md @@ -7,27 +7,26 @@ doc: `CLAUDE.md`. Planned work: `../followup_general.md`, `../followup_vulkan.md **Routed from `REVIEW.md` (beside this file): a diff that checklist routes here applies this list together with it.** -**A diff that changes any dispatch geometry a gate itself encodes - a grid divisor, a -threadgroup size, a threadgroup-memory length the gate sets - updates every gate that -hand-dispatches that kernel, in the same change.** A hand-dispatched gate encodes the geometry -itself, so a moved divisor leaves the gate dispatching the wrong shape with no error; a -`@workgroup` array's size is compiled into the kernel and no gate carries it. +**A diff that changes a kernel's dispatch geometry - a grid divisor, a threadgroup size, a +threadgroup-memory length the dispatch sets - updates every gate (a cell or probe that +dispatches a kernel) that hand-dispatches that kernel, in the same change.** A moved divisor +leaves the gate dispatching the wrong shape with no error; a `@workgroup` array's size is +compiled into the kernel, so no gate carries it. **A diff that changes a kernel's kargs - the kernel-argument struct, or any buffer binding - updates every gate that hand-binds that kernel, in the same change.** A stale hand bind reads the wrong buffer and passes on garbage that happens to compare. -**A kernel that gains a kargs field whose non-default value changes what it computes or which -elements it reads or writes - a branch selector, a row or element base, a stride - ships, in -the same change, a kernel-unit cell that sets that field to a non-default value.** At the -default the new field has no visible effect: a CPU oracle that ignores it and the kernel that -honors it agree. A model-level cell that reaches the field does not discharge this: it binds the -layer's own row, so a base stays at its default there. - **A kernel-unit cell - a model-less cell that dispatches one kernel class and asserts on its output - missing a compare against a CPU oracle that can witness the cell's property is a defect.** +**A kernel that gains a kargs field whose non-default value changes what it computes or which +elements it reads or writes - a branch selector, a row or element base, a stride - ships a +model-less kernel-unit cell that sets that field to a non-default value, in the same change.** +At the default the new field has no visible effect: a CPU oracle that ignores it and the kernel +that honors it agree. + **A kernel-unit cell fills a GPU output buffer with a sentinel before every dispatch whose output it then reads.** An unprefilled output can pass by staying stale - the previous dispatch's values, or garbage that happens to sit inside the tolerance bar. @@ -45,9 +44,11 @@ dispatched before ships a control that reds it in the same change.** A control i the same gate that must RED - a poisoned input, a poisoned expectation, a disconnected mechanism, or a second independent lane; a gate's own reference is never its control. -**A kernel-unit cell whose kernel reads f16 operands and whose oracle is wider-precision -feeds inputs that are exact in f16.** Otherwise the compare measures input rounding, and the -bar has to be loosened until it no longer discriminates. +**A kernel-unit cell whose kernel computes at a narrower precision than its oracle at any +step - operands or accumulator - bounds that step's error by construction - f16-exact inputs, +magnitude-bounded fixtures - or states in the cell how its new bar follows from that step's +error.** A bar moved without that derivation is a loosening: the compare then measures the +narrowing until it no longer discriminates. **A gate for a kernel that attends inside a restricted horizon - a window, a sliding span, a block-diagonal range - writes its CPU oracle to attend strictly inside that horizon.** A leak diff --git a/modules/dasLLAMA/tests/test_gpu_resident_hybrid.das b/modules/dasLLAMA/tests/test_gpu_resident_hybrid.das index b671da8f78..16e1fbdaf4 100644 --- a/modules/dasLLAMA/tests/test_gpu_resident_hybrid.das +++ b/modules/dasLLAMA/tests/test_gpu_resident_hybrid.das @@ -16,24 +16,38 @@ require daslib/fio require dasllama/dasllama_math // setup_dasllama_jobque_ (the engine-standard jobque config) require dasllama/dasllama_gpu_tier // moe_gpu_tier_installed / gpu_want_auto: is the tier armed require dasllama/dasllama_gpu_resident // moe_gpu_resident_active: did the whole-model driver take the hybrid -require ?vulkan dasllama/dasllama_vulkan_common // PF_WINDOW and the resolved coopmat mode, where the module exists +require ?vulkan dasllama/dasllama_vulkan_common // PF_WINDOW, the resolved coopmat mode and the kernel census, where the module exists +require ?vulkan dasllama/dasllama_vulkan_prefill // pf_f16_feed: the feed admission the driver acts on require _model_tier // models_dir() + model_available() presence/size-tier gate -// The whole-model resident driver on a deltanet hybrid (Qwen3.5-0.8B-Q8_0 under DASLLAMA_GPU=1): -// the resident window chain prefills the prompt (the recurrent layers through the conv + chunked -// scan over their device state, the gated partial-rope attention over the mirror), then the -// resident decode steps every layer - the recurrent ones through the fused deltanet step - as one -// submit per token. The GPU chains and the CPU chain are not bit-identical (GPU exp / f32 -// accumulation order), so the gate is the forced-feed logits-tolerance form: the same fixed tokens -// fed to both arms, the logits within a bar of the CPU chain after the prefill and at every step, -// and the one-step-off control that must exceed it. Four prompt lengths: one window, two windows +// The whole-model resident driver on a deltanet hybrid under DASLLAMA_GPU=1: the resident window +// chain prefills the prompt (the recurrent layers through the conv + chunked scan over their device +// state, the gated partial-rope attention over the mirror), then the resident decode steps every +// layer - the recurrent ones through the fused deltanet step - as one submit per token. The GPU +// chains and the CPU chain are not bit-identical (GPU exp / f32 accumulation order), so the gate is +// the forced-feed logits-tolerance form: the same fixed tokens fed to both arms, the logits within +// a bar of the CPU chain after the prefill and at every step, and the one-step-off control that +// must exceed it. Three fixtures: the Q8_0 file over four prompt lengths - one window, two windows // (the conv tail and the state carry across the window seam), one past the window, and two tokens -// (both conv-tail edges: the last window shorter than the taps). +// (both conv-tail edges: the last window shorter than the taps); the K-quant twin (Q4_K_M) over one +// and two windows; and the mixed twin (Q4_K_M with the deltanet out plane at Q8_0) over the same +// two, whose recurrent layers take a K-quant x feed and a Q8_0 o feed - the routing witness below +// holds that the x feed rode the coopmat tiles. let private DN_LOGIT_BAR_REL = 0.04 // of the step's max |logit| - the deltanet bar test_deltanet.das holds too -//! the K-quant bar on the f16 mirror: CPU-vs-device rounding sits flat at 0.39-0.57 of a ~12.5 max logit at every step (the Q8 file 0.19-0.36), with no growth and the one-step-off control well past it -let private DN_LOGIT_BAR_REL_KQ = 0.06 +//! the K-quant bar on the f16 feed and mirror: CPU-vs-device rounding sits flat at 0.39-0.57 of a ~12.5 max logit at every step (the Q8 file 0.19-0.36), with no growth and the one-step-off control well past it +let private DN_LOGIT_BAR_REL_KQ_F16_FEED = 0.06 +//! the K-quant bar on the quant feed (no coopmat tile serves the planes, the prefill's activations ride Q8_K): the mixed twin's two-window cell under `DASLLAMA_COOPMAT=sdot4` on the RTX 5060 Ti reads 0.084 of an 11.6 max logit at one step, argmax unchanged, the rest under 0.05 +let private DN_LOGIT_BAR_REL_KQ_QUANT_FEED = 0.10 let private PF_WINDOW_ROWS = 512l //! the engine's PF_WINDOW, asserted equal by the two-window cells: a prompt past it runs as sequential windows +let private RQ_NO_CENSUS = -1l //! the census count with no vulkan module: the routing witness stands down + +//! the plane formats a fixture's deltanet layers carry: all Q8_0, all K-quant, or K-quant qkv/z beside a Q8_0 out plane +enum HybridPlanes { + q8 + kq + kq_q8out +} //! the served tile arm, so a red names which prefill tile it came from def private tile_arm_tag() : string { @@ -61,23 +75,50 @@ def private argmax_of(v : array) : int64 { return bi } +//! the coverage census's count of Q8_K requant dispatches so far; RQ_NO_CENSUS with no vulkan module +def private q8k_requants() : int64 { + static_if (typeinfo builtin_module_exists(vulkan)) { + return vk_kernel_coverage_of("cls_q8k_rq_spv") + } else { + return RQ_NO_CENSUS + } +} + +//! the driver's own answer: does the prefill's f16 feed admit a K-quant plane on the armed coopmat mode +def private f16_feed_admits_kq() : bool { + static_if (typeinfo builtin_module_exists(vulkan)) { + return pf_f16_feed(int(KqFmt.k4)) + } else { + return false + } +} + +struct FedLogits { + rows : array> //! after the prefill, then after each fed step + prefill_q8k_rq : int64 //! Q8_K requants the prefill alone dispatched (RQ_NO_CENSUS with no vulkan module) +} + // prefill `prompt`, then feed `toks` one per step; the logits after the prefill and after each step -def private feed_logits(tr : Model; var s : Session; prompt : array; toks : array) : array> { +def private feed_logits(tr : Model; var s : Session; prompt : array; toks : array) : FedLogits { let n = long_length(prompt) - var out : array> - out |> reserve(long_length(toks) + 1l) + var out : FedLogits + out.rows |> reserve(long_length(toks) + 1l) + let rq0 = q8k_requants() forward_prefill(tr, s, prompt, n, 0l) - out |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + out.prefill_q8k_rq = rq0 == RQ_NO_CENSUS ? RQ_NO_CENSUS : q8k_requants() - rq0 + out.rows |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] for (st in range64(long_length(toks))) { forward(tr, s, toks[st], n + st) - out |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] + out.rows |> emplace <| [for (i in range64(tr.config.vocab_size)); s.logits[i]] } return <- out } // prefill `n` tokens + 8 fed steps on the resident driver, then the same on the all-CPU chain; the // prefill logits and every step's within the deltanet bar, the one-step-off control past it. -def private hybrid_forced_feed(t : T?; n : int64; file : string; kq : bool) { +def private hybrid_forced_feed(t : T?; n : int64; file : string; planes : HybridPlanes) { + let kq = planes != HybridPlanes.q8 + let q8out = planes == HybridPlanes.kq_q8out if (!jit_enabled()) { t |> skip("interpreted (dasLLAMA model tests are JIT-only)") return @@ -94,15 +135,19 @@ def private hybrid_forced_feed(t : T?; n : int64; file : string; kq : bool) { } var tr <- load_model_(path, QuantMode.q8) t |> success(moe_gpu_resident_active(), "the whole-model driver armed on the hybrid (the load log names any decline)") + var nrec = 0l if (kq) { var native = 0l for (l in range64(tr.config.n_layers)) { - if (layer_is_recurrent(tr.config, l) && fmt_at(tr.dnqkv_fmt, l) != KqFmt.q8 && fmt_at(tr.dngate_fmt, l) != KqFmt.q8 - && fmt_at(tr.dnout_fmt, l) != KqFmt.q8) { + if (!layer_is_recurrent(tr.config, l)) continue + nrec++ + if (fmt_at(tr.dnqkv_fmt, l) != KqFmt.q8 && fmt_at(tr.dngate_fmt, l) != KqFmt.q8 + && ((fmt_at(tr.dnout_fmt, l) == KqFmt.q8) == q8out)) { native++ } } - t |> success(native > 0l, "the loader kept the deltanet qkv/z/out planes in their K-quant file formats ({native} recurrent layers)") + let out_form = q8out ? "the out plane Q8_0" : "the out plane too" + t |> success(native > 0l, "the loader kept the deltanet qkv/z planes in their K-quant file formats, {out_form} ({native} recurrent layers)") } with_job_que() { setup_dasllama_jobque_() @@ -113,22 +158,31 @@ def private hybrid_forced_feed(t : T?; n : int64; file : string; kq : bool) { var gpu <- feed_logits(tr, sg, prompt, toks) t |> success(moe_gpu_resident_active(), "the whole-model driver stayed armed through the run") t |> equal(moe_gpu_resident_prefills(), served0 + 1l, "the resident window chain served the {n}-token prompt") + if (kq && gpu.prefill_q8k_rq != RQ_NO_CENSUS) { + //! off the f16 feed the x route requants once per recurrent layer per window - the threshold both arms read + to_log(LOG_INFO, "prefill ({n} tokens) [{tile_arm_tag()}]: {gpu.prefill_q8k_rq} Q8_K requants over {nrec} recurrent layers, f16 feed admits kq {f16_feed_admits_kq()}\n") + if (f16_feed_admits_kq()) { + t |> success(gpu.prefill_q8k_rq < nrec, "on the f16 feed the prefill never requantized the block input to Q8_K for the qkv/z pair ({gpu.prefill_q8k_rq} Q8_K requants, {nrec} recurrent layers) [{tile_arm_tag()}]") + } else { + t |> success(gpu.prefill_q8k_rq >= nrec, "off the f16 feed the K-quant qkv/z pair reads the Q8_K form ({gpu.prefill_q8k_rq} Q8_K requants, {nrec} recurrent layers) [{tile_arm_tag()}]") + } + } delete sg moe_gpu_drop_model() // every later step is the CPU chain var sc <- create_session(tr, mirror_kv_dtype()) var cpu <- feed_logits(tr, sc, prompt, toks) delete sc - let rel = kq ? DN_LOGIT_BAR_REL_KQ : DN_LOGIT_BAR_REL - for (st in range64(long_length(cpu))) { - let diff = logits_maxdiff(gpu[st], cpu[st]) - let bar = rel * logits_maxabs(cpu[st]) - let ag = argmax_of(gpu[st]) - let ac = argmax_of(cpu[st]) + let rel = kq ? (f16_feed_admits_kq() ? DN_LOGIT_BAR_REL_KQ_F16_FEED : DN_LOGIT_BAR_REL_KQ_QUANT_FEED) : DN_LOGIT_BAR_REL + for (st in range64(long_length(cpu.rows))) { + let diff = logits_maxdiff(gpu.rows[st], cpu.rows[st]) + let bar = rel * logits_maxabs(cpu.rows[st]) + let ag = argmax_of(gpu.rows[st]) + let ac = argmax_of(cpu.rows[st]) let what = (st == 0l ? "prefill ({n} tokens)" : "step {st - 1l} (fed {toks[st - 1l]})") + " [{tile_arm_tag()}]" - to_log(LOG_INFO, "{what}: argmax cpu {ac} ({cpu[st][ac]}), resident {ag} ({gpu[st][ag]}); logits maxdiff {diff} (bar {bar})\n") + to_log(LOG_INFO, "{what}: argmax cpu {ac} ({cpu.rows[st][ac]}), resident {ag} ({gpu.rows[st][ag]}); logits maxdiff {diff} (bar {bar})\n") t |> success(diff <= bar, "{what} logits: resident within {bar} of the CPU chain (maxdiff {diff})") if (st > 0l) { // the control: the previous step's CPU logits are a disconnected mechanism the bar must RED - let off = logits_maxdiff(gpu[st], cpu[st - 1l]) + let off = logits_maxdiff(gpu.rows[st], cpu.rows[st - 1l]) t |> success(off > bar, "{what} control: the bar discriminates a one-step-off compare (maxdiff {off} vs bar {bar})") } } @@ -143,14 +197,14 @@ def private hybrid_forced_feed(t : T?; n : int64; file : string; kq : bool) { [test] def test_gpu_resident_hybrid_decode(t : T?) { t |> run("Qwen3.5-0.8B: resident prefill + decode with deltanet layers on device vs the CPU chain, one window") @(t : T?) { - hybrid_forced_feed(t, 40l, "Qwen3.5-0.8B-Q8_0.gguf", false) + hybrid_forced_feed(t, 40l, "Qwen3.5-0.8B-Q8_0.gguf", HybridPlanes.q8) } } [test] def test_gpu_resident_hybrid_two_windows(t : T?) { t |> run("Qwen3.5-0.8B: the resident prefill across two windows (the conv tail and state carry the seam)") @(t : T?) { - hybrid_forced_feed(t, PF_WINDOW_ROWS + 88l, "Qwen3.5-0.8B-Q8_0.gguf", false) + hybrid_forced_feed(t, PF_WINDOW_ROWS + 88l, "Qwen3.5-0.8B-Q8_0.gguf", HybridPlanes.q8) } } @@ -160,20 +214,50 @@ def test_gpu_resident_hybrid_two_windows(t : T?) { [test] def test_gpu_resident_hybrid_short_windows(t : T?) { t |> run("Qwen3.5-0.8B: a prompt one past the window - the last window still carries the conv taps") @(t : T?) { - hybrid_forced_feed(t, PF_WINDOW_ROWS + 1l, "Qwen3.5-0.8B-Q8_0.gguf", false) + hybrid_forced_feed(t, PF_WINDOW_ROWS + 1l, "Qwen3.5-0.8B-Q8_0.gguf", HybridPlanes.q8) } t |> run("Qwen3.5-0.8B: a two-token prompt - one window shorter than the conv taps") @(t : T?) { - hybrid_forced_feed(t, 2l, "Qwen3.5-0.8B-Q8_0.gguf", false) + hybrid_forced_feed(t, 2l, "Qwen3.5-0.8B-Q8_0.gguf", HybridPlanes.q8) } } [test] def test_gpu_resident_hybrid_kq(t : T?) { t |> run("Qwen3.5-0.8B-Q4_K_M: K-quant deltanet planes on the resident driver vs the CPU chain, one window") @(t : T?) { - hybrid_forced_feed(t, 40l, "Qwen3.5-0.8B-Q4_K_M.gguf", true) + hybrid_forced_feed(t, 40l, "Qwen3.5-0.8B-Q4_K_M.gguf", HybridPlanes.kq) } t |> run("Qwen3.5-0.8B-Q4_K_M: the same across two windows") @(t : T?) { window_pin(t) - hybrid_forced_feed(t, PF_WINDOW_ROWS + 88l, "Qwen3.5-0.8B-Q4_K_M.gguf", true) + hybrid_forced_feed(t, PF_WINDOW_ROWS + 88l, "Qwen3.5-0.8B-Q4_K_M.gguf", HybridPlanes.kq) + } +} + +[test] +def test_gpu_resident_hybrid_kq_q8out(t : T?) { + t |> run("Qwen3.5-0.8B-Q4_K_M-q8out: K-quant qkv/z with a Q8_0 out plane on the resident driver vs the CPU chain, one window") @(t : T?) { + hybrid_forced_feed(t, 40l, "Qwen3.5-0.8B-Q4_K_M-q8out.gguf", HybridPlanes.kq_q8out) + } + t |> run("Qwen3.5-0.8B-Q4_K_M-q8out: the same across two windows") @(t : T?) { + window_pin(t) + hybrid_forced_feed(t, PF_WINDOW_ROWS + 88l, "Qwen3.5-0.8B-Q4_K_M-q8out.gguf", HybridPlanes.kq_q8out) + } +} + +// model-free: the routing witness stands on the census answering by name - a name nothing seeded panics instead of reading a zero count +[test] +def test_kernel_census_by_name(t : T?) { + t |> run("the kernel census refuses a name no kernel seeded") @(t : T?) { + static_if (typeinfo builtin_module_exists(vulkan)) { + var tripped = false + try { + vk_kernel_coverage_of("no_such_kernel_spv") + } recover { + tripped = true + } + t |> success(tripped, "an unseeded census name panics, so a misspelt key cannot read as a zero count") + t |> equal(vk_kernel_coverage_of("cls_q8k_rq_spv") >= 0l, true, "a seeded class kernel answers by its SPIR-V name, device or not") + } else { + t |> skip("no vulkan module in this build") + } } } diff --git a/modules/dasMetal/ARCHITECTURE.md b/modules/dasMetal/ARCHITECTURE.md index fe99109fe0..bfc29f6e46 100644 --- a/modules/dasMetal/ARCHITECTURE.md +++ b/modules/dasMetal/ARCHITECTURE.md @@ -173,7 +173,7 @@ than a second GPU. Cross-GPU parity of one source is secondary. Current entries: - **A kernel body calling a `[spirv_decode]` method directly is Vulkan-only; pending, not deliberate.** dasSpirv emits the call as an ordinary function - on a plane element the callee takes the element's index and chains through the plane, on a copy the struct - parameter spills to a local (`dasSpirv/ARCHITECTURE.md` sec.3.5). An MSL function takes a + parameter spills to a local (`dasSpirv/ARCHITECTURE_COOPMAT.md` sec.3.5). An MSL function takes a struct by value with its members addressable as written, so Metal needs no spill; the direct-call form itself has no MSL fixture yet, and a Metal kernel wanting one decode body for a staged tile is what would land it. diff --git a/modules/dasSpirv/ARCHITECTURE.md b/modules/dasSpirv/ARCHITECTURE.md index 04c08e56a9..dbae00ca1c 100644 --- a/modules/dasSpirv/ARCHITECTURE.md +++ b/modules/dasSpirv/ARCHITECTURE.md @@ -2,8 +2,8 @@ **Read `ARCHITECTURE_COMMON.md` (repo root) first - its contract binds this document.** The checklists that bind a diff here are `REVIEW.md` (this folder) and -`modules/REVIEW_SHADER_EMITTERS.md`. The plan sections and the implementation log this -document grew from are archived at `history/dasSpirv/MASTERPLAN_LOG.md`. +`modules/REVIEW_SHADER_EMITTERS.md`. Sections 3.2, 3.3, 3.5, 3.6 and 3.7 - cooperative matrix and +emission-time unrolling - are in `ARCHITECTURE_COOPMAT.md` beside this file. ## 1. Why @@ -38,17 +38,15 @@ compute test as a ready-made end-to-end gate. 2. **Fresh frontend** - its own shader annotations (sec.3). dasSpirv has no code dependency on dasGlsl or dasOpenGL; it reuses only the generic AST tooling, none of which lives in dasGlsl. -3. The opcode census declares the supported opcode set and is checked against the fixtures in - both directions; LCOV covers the runtime-reached files beside it (sec.4). 4. **SPIR-V 1.3 is the default header version; a feature that needs more raises it.** 1.3 gives StorageBuffer storage class + `Block` (not the deprecated 1.0 BufferBlock+Uniform path), lavapipe advertises >= 1.2, and at `<= 1.3` the entry-point interface lists only Input/Output globals (1.4 requires *all* globals, which would churn every stage's interface). The version is a per-module field, `SpirvModule.version`: a mesh or task stage raises it to 1.4 because `SPV_EXT_mesh_shader` requires it, a call that re-types a - Block-laid-out struct with `OpCopyLogical` (sec.3.5) raises it to 1.4 because that opcode - requires it, a few subgroup ops raise it to 1.5, and the cooperative-matrix ops raise it to - 1.6. Every other stage stays at 1.3. + Block-laid-out struct with `OpCopyLogical` (`ARCHITECTURE_COOPMAT.md` sec.3.5) raises it to + 1.4 because that opcode requires it, a few subgroup ops raise it to 1.5, and the + cooperative-matrix ops raise it to 1.6. Every other stage stays at 1.3. ## 3. Files and emission mechanism {#files-and-emission} @@ -60,7 +58,7 @@ consumes it via `require spirv/...` and feeds the emitted `array` (SPIR-V | File | Gen/Hand | Purpose | |---|---|---| | `spirv/spirv_grammar.das` | **generated** | Opcode + GLSL.std.450 enums + StorageClass/Decoration/BuiltIn/ExecutionMode/Capability token constants, generated from the vendored, pinned SPIRV-Headers grammar JSON. Single source of numeric truth (+ opcode->name table for the disassembler). | -| `spirv/spirv_builder.das` | hand | `SpirvModule` (per-section append buffers), monotonic id allocator, type/constant dedup pools, `emit(section, opcode, ...)`, `get_type_id`, `get_const_id`, `finalize() : array`. AST-agnostic. | +| `spirv/spirv_builder.das` | hand | `SpirvModule` (per-section append buffers), monotonic id allocator, type/constant dedup pools (`type_pool` / `const_pool`, behind the `type_*` and `const_*` helpers), `emit(section, opcode, ...)`, and `module_words() : array`, which prepends the five-word header and concatenates the sections. `finalize` frees the buffers and the pools. AST-agnostic. | | `spirv/spirv_types.das` | hand | daslang `TypeDecl` -> SPIR-V type-id (scalars, vec/mat, fixed + runtime arrays, structs, pointers) with layout decorations. Highest-detail correctness file. | | `spirv/spirv_builtins.das` | hand | The builtin surface the emitter recognizes by name: builtin globals (`gl_*`), the opaque resource marker structs (`sampler2D`/`image2D`/`sampler2DShadow`/...), and the intrinsic stubs (`texture`, `imageLoad`, `barrier`, the atomics, the derivatives, ...). Every stub is `[sideeffects]` so no const-fold or DCE pass can reach it before the annotation runs. | | `spirv/spirv_emit.das` | hand | `SpirvEmit : AstVisitor` codegen visitor + the `[macro_function] generate_spirv(...)` entry point. Every construct the emitter cannot lower has a rejection override or hits the `value_of`/`ptr_of` backstop, so it becomes a clean compile error rather than a bad blob. | @@ -68,17 +66,19 @@ consumes it via `require spirv/...` and feeds the emitted `array` (SPIR-V | `spirv/spirv_shader.das` | hand | The shader annotations, each a `SpirvShader : AstFunctionAnnotation` subclass carrying its stage: `[compute_shader]`, `[spirv_kernel]` (class-method authoring), `[vertex_shader]`, `[fragment_shader]`, `[mesh_shader]`, `[task_shader]`, `[raygen_shader]`, `[miss_shader]`, `[closest_hit_shader]`; plus the `[spirv_decode]` / `[spirv_combine]` / `[spirv_per_element]` callback annotations. `apply` reserves the blob global and its `_reflect` companion; `fixup` runs dependency collection, calls `generate_spirv`, and sets both inits. | | `spirv/spirv_dis.das` | hand | Minimal disassembler + opcode-census helper (self-delimiting walk: word0 = `(wordCount<<16)\|opcode`). Symbolic via `spirv_grammar`'s opcode->name table. | | `generator/gen_spirv_grammar.das` | hand | The mini-generator: reads vendored grammar JSON -> emits `spirv/spirv_grammar.das`. | -| `spirv_headers/*.json` | vendored | Pinned `spirv.core.grammar.json` + `extinst.glsl.std.450.grammar.json`; license in `SPIRV_HEADERS.LICENSE`, provenance in `history/dasSpirv/MASTERPLAN_LOG.md`. | +| `spirv_headers/*.json` | vendored | The pinned Khronos SPIRV-Headers grammars: `spirv.core.grammar.json` at SPIR-V 1.6 revision 7, and `extinst.glsl.std.450.grammar.json` at version 100 revision 2. License in `SPIRV_HEADERS.LICENSE`. | | `CMakeLists.txt` | hand | `ADD_MODULE_DAS_FROM_DESCRIPTOR(spirv spirv)` + install rule, modeled on `modules/dasGlsl/CMakeLists.txt`. | -**SSA backend (llvm_jit template).** `SpirvEmit` carries `e2v : table` -(Expression->result-id), `v2v : table` (Variable->pointer-id), -`ite2blocks`/`loop_stack` for control flow, and a hand-maintained `cur_block_terminated` -(SPIR-V has no builder to query - set on every terminator, cleared on `OpLabel`; guards -mirror llvm_jit's `current_block_terminates()`). Section buffers concatenate at `finalize` -in SPIR-V's mandated order: header(5) -> capabilities -> extensions -> ext-imports -> -memory-model -> entry-points -> exec-modes -> debug -> decorations -> types+constants+global-vars -(interleaved, define-before-use, deduplicated) -> functions. +**SSA backend (llvm_jit template).** `SpirvEmit` carries `e2id` (intptr(Expression) -> rvalue +result-id), `e2ptr`/`e2pty` (intptr(Expression) -> lvalue pointer-id + pointee type-id), +`ctx.local_vars` (intptr(Variable) -> the Function-storage `OpVariable` backing a local), +`ite_ids`/`loop_ids` and `ctx.loop_stack` for control flow, and a hand-maintained +`ctx.terminated` (SPIR-V has no builder to query - set on every terminator, cleared on +`OpLabel`; guards mirror llvm_jit's `current_block_terminates()`). `module_words` +concatenates the section buffers behind the five-word header, in SPIR-V's mandated order: +capabilities -> extensions -> ext-imports -> memory-model -> entry-points -> exec-modes -> +debug -> decorations -> types+constants+global-vars (interleaved, define-before-use, +deduplicated) -> functions. **Capture mechanism (dasGlsl analog).** dasGlsl's `fixup` sets `glob.init = new ExprConstString(value := text)`; ours builds an `ExprMakeArray` of @@ -91,20 +91,14 @@ without macro plumbing. **`[spirv_decode]` method form.** The decode callback's SPIR-V signature is a rigid three parameters. The method form erases the das-level `self` from it, so the decode body still reads its class members - a separate scale plane, push constants, `@workgroup` staging. The four-wide -twin of that callback is section 3.3; a kernel body calling the method directly is section 3.5. - -**Cooperative-matrix element loops carry `Unroll`.** `coopmatClamp` walks a coopmat local -element by element through a hand-emitted structured loop bounded by -`OpCooperativeMatrixLengthKHR`, and its `OpLoopMerge` sets loop control `Unroll` - the control -glslang emits for `[[unroll]]`. Rolled, the dynamic per-element `OpAccessChain` index demotes -the accumulator out of tensor-register form into addressable storage for the whole kernel, not -only for the loop. On an RTX 5060 Ti (driver 610.74) the cm2 l-tile min-kernel runs 34.2 TFLOP/s -rolled and 57.8 unrolled. +twin of that callback is `ARCHITECTURE_COOPMAT.md` section 3.3; a kernel body calling the method +directly is that document's section 3.5. ### 3.1 The 8/16-bit small-integer surface {#small-int-surface} A shader reads AND writes `int8`/`uint8`/`int16`/`uint16`/`float16` SSBO elements and struct -members, and the write direction costs the emitter no arm of its own. A narrowing daslang cast +members, and the write direction costs the emitter no *arm* of its own. An **arm** is one branch +of the emitter that handles one construct. A narrowing daslang cast (`int8(v)`, `uint16(u)`) is one of the conversion opcodes `convert_op` already picks for any narrowing pair, and a store reaches its element through the same width-aware std430 access chain a load reads - so `ensure_member_storage_caps`, pulling `StorageBuffer8BitAccess` / @@ -119,56 +113,6 @@ needs no emitter change at all. The `byte4`/`ubyte4` type factory pulls the `Int and widening an unpacked lane (`int4(b4)`) is a same-class `OpSConvert`, which gives sign extension for free. -### 3.2 A cm2 tile shape is one struct declaration {#cm2-tile-markers} - -The workgroup-scope cooperative-matrix tiles are marker structs in `spirv_builtins.das` whose -NAMES carry their geometry - `coopmatWg{A|B|Acc}_{f16|f32|s8|s32}_{R}x{C}` - and -`coopmat_wg_info` parses that name instead of looking the struct up in a table. Adding a tile -shape is therefore one struct declaration plus the overload that types the das call -(`coopmatMulAdd` for a multiply tile, `coopmatConvert` for an accumulator-only tile): no -emitter arm changes, because every cm2 arm reads rows, columns and component width out of the -parse. The tile markers are empty structs with no storage, so every builtin over them - the -tensor loads and stores, the decode forms, `coopmatMulAdd`, `coopmatConvert`, `coopmatClamp`, -the reductions - has an inert CPU body that cannot compute what the emitted form computes: a -coopmat kernel's device test takes a plain CPU reference as its oracle, the one sanctioned -exception class `modules/REVIEW_SHADER_EMITTERS.md` admits. A reduction width -known only at run time reaches a SPIR-V kernel through a `tensorLayout2D` or -`tensorLayout2DPad` whose dimension `tensorLayoutSetDimension` sets - that layout is this -emitter's runtime-extent descriptor. - -### 3.3 The four-wide decode twin {#cm2-decode-vector} - -`SPV_NV_cooperative_matrix_decode_vector` adds no opcode: one capability and the -`DecodeVectorFunc` bit of the tensor-load's addressing mask, whose operand names a second decode -function returning the 4-lane vector of the tile component type, called for four consecutive -elements along the block's last dimension from a multiple of four. The scalar `DecodeFunc` stays -mandatory beside it and the driver picks per call site, so one module serves devices with and -without the feature. `coopmatLoadTensorDecode`'s tenth argument selects the twin: `true` has the -emitter synthesize it as four `OpFunctionCall`s of the scalar body at `coordInBlock.y + 0..3` -composed into the vector (the driver's compiler inlines the calls and merges the loads they -share), `false` is the scalar-only load, and a `[spirv_decode]` function returning `half4` -over a `float16` decode, with the scalar contract's parameters, is a hand-laid twin (daslang -spells no int8 or 32-bit four-vector the tiles would take, so the hand-laid form is f16-only; -the synthesized twin covers every scalar type a decode may return). Twins register in their own -table keyed by the function they came from, so one scalar body yields one twin however many loads -name it, and emit after the scalar bodies they call. - -The extension stands apart from the cm2 base: a module that uses the twin declares the -`CooperativeMatrixDecodeVectorNV` capability and the `SPV_NV_cooperative_matrix_decode_vector` -extension name, and the device must have its own -`VkPhysicalDeviceCooperativeMatrixDecodeVectorFeaturesNV` bit enabled. - -The tensor-addressing operands follow the mask word in bit order - `TensorView`, then -`DecodeFunc`, then `DecodeVectorFunc` - after the load's fixed prefix: result type, result, -pointer, object, layout, memory-access mask and that mask's own extra words. The emitter appends -them in that order and the strip counts forward to the operand it removes by the same rule. - -A finished module is downgraded rather than recompiled. `strip_decode_vector` walks the word -stream and removes the capability, the extension declaration, and each -`OpCooperativeMatrixLoadTensorNV`'s `DecodeVectorFunc` bit together with its operand word; the -twin's `OpFunction` stays in the module, unreferenced, and the mandatory scalar `DecodeFunc` -serves the load. One emitted blob therefore runs on a device without the feature. - ### 3.4 Operand laziness follows the language {#operand-laziness} `cond ? a : b`, `&&` and `||` lower to `OpSelect` / `OpLogicalAnd` / `OpLogicalOr` while both @@ -176,52 +120,14 @@ operands are pure - branchless, both evaluated. An operand that indexes a global ssbo, a block field, a `@workgroup` array) lowers as a branch instead: `SpirvTempAlloc` hoists a Function-storage temp per such node (`ctx.lazy_temps`), the operand the condition admits stores through it, and the merge block reloads it as the value. For `&&` and `||` the left operand's -answer stores before the branch, so the edge the left operand settles goes straight to the merge; -the branch labels are allocated after the condition is visited, in the visitor's if/else hook -order (the left- and right-operand pre-visit hooks). `OpSelect` evaluates both operands, and -the operand a condition rules out is exactly the one whose index the condition guards - on the -device an out-of-range load is a fault that surfaces only when the overshoot leaves mapped memory, -so it tracks allocation layout, not the kernel's inputs. A local fixed array stays eager: its -index is register arithmetic, not a device address. - -### 3.5 A kernel calls a decode method directly {#direct-decode-call} - -A kernel body may CALL a `[spirv_decode]` method directly: that is an ordinary user function - -its own OpFunction, registered beside the callback form and pulling no cooperative-matrix -capability - which is what lets one decode body serve a tensor load on a cm2 device and a -hand-staged tile on a KHR one. The block argument takes one of two forms, fixed per method at -discovery (a method called both ways is refused). Called on the plane element itself -(`decode(wq[i], bc, cib)`), the block parameter is emitted as the element's `uint` INDEX and the -function's entry chains `OpAccessChain plane, 0, index` once, binding the parameter as a memory -local in the plane's own storage class, so the body's member reads chain through the plane -exactly as the callback form's chain through its block pointer - the element is never loaded as -a value. A pointer could not travel instead: under logical addressing an SSBO pointer is not a -legal OpFunctionCall argument without the VariablePointersStorageBuffer capability, and the -index needs no capability at all. Called on a copy (`let blk = wq[i]; decode(blk, bc, cib)`), -the block goes by value and lands in the spill local of the next paragraph - the whole block -loaded and stored per call, which the KHR tile measured at a third of the element form's rate. +answer stores before the branch, so the edge that skips the right operand runs straight to the +merge. The branch labels are allocated after the condition is visited, in the visitor's if/else +hook order (the left- and right-operand pre-visit hooks). -**A struct value parameter with an aggregate member is a memory local.** A read-only parameter -of a plain data struct that carries a fixed-array or nested-struct member (not a coopmat tile, a -tensor object, a ray query or a sampler marker) binds its SSA OpFunctionParameter and is stored -at entry into a Function-storage OpVariable. The variable is declared first in the entry block, -ahead of the body's locals and call temps; the store that fills it follows the block's last -OpVariable, because SPIR-V requires every OpVariable of a block to lead the block. The parameter -is thereafter that local: its members access-chain like a `var` struct local's. The reason is a -fixed-array member indexed at run time (`blk.qs[cib.y >> 1]`): an SSA composite offers no -pointer, and `OpCompositeExtract` takes literal indices only. A scalar-only struct parameter -keeps the value path (its members extract), and a `let` copy of a block element in a body stays -a composite (scalar and vector members extract; its array and struct members are still refused), -because reading it whole and then storing it would double every element load; the parameter -form pays the store once per call. The Metal emitter needs no arm for this: an MSL function -takes a struct by value and its members are addressable as written. - -**A struct loaded out of a `Block` re-types at the call.** A struct loaded out of a `Block` - -an ssbo element - carries the Block's laid-out `OpTypeStruct`, a different type id from the -plain struct a parameter takes, so a call passing one re-types the value with `OpCopyLogical` -first, and the module's version floor rises to SPIR-V 1.4, which that opcode requires. The -emitter remembers the struct type each loaded id was read as and copies only when the two ids -differ, so a struct already at the plain type passes through untouched. +`OpSelect` evaluates both operands. The operand a condition rules out is the one whose index that +condition guards. On the device an out-of-range load faults only when the overshoot leaves mapped +memory, so whether it faults follows the allocation layout, not the kernel's inputs. A local fixed +array stays eager: its index is register arithmetic, not a device address. ## 4. Test architecture - "every emitted instruction has a test" @@ -264,17 +170,5 @@ real-driver layer, which lives in dasVulkan): ## 5. Cross-backend parity - the kernel-model asymmetry ledger -`modules/REVIEW_SHADER_EMITTERS.md` requires a kernel-model capability added to one emitter to -be added to the other or recorded as an asymmetry. That ledger is shared - one list for both -backends, not one per backend - and lives in `modules/dasMetal/ARCHITECTURE.md` sec.5. - -## 6. Verification - -- **Standing per-change gate (main tree):** - `daslang dastest/dastest.das -- --test tests/spirv --cov-path spirv.lcov --isolated-mode` - -> all green, opcode census == declared set both directions, every blob spirv-val-clean, no - `GC APP LEAK`. -- **Disassembly check:** an emitted module is dumped via `spirv_dis` (symbolic) and diffed - against external `spirv-dis` as ground truth. -- **Real-driver gate (dasVulkan):** the integration suite under lavapipe and the local real - GPU. +`modules/REVIEW_SHADER_EMITTERS.md` requires one kernel-model asymmetry ledger for both emitters. +That list lives in `modules/dasMetal/ARCHITECTURE.md` sec.5. diff --git a/modules/dasSpirv/ARCHITECTURE_COOPMAT.md b/modules/dasSpirv/ARCHITECTURE_COOPMAT.md new file mode 100644 index 0000000000..5fa9127d9c --- /dev/null +++ b/modules/dasSpirv/ARCHITECTURE_COOPMAT.md @@ -0,0 +1,151 @@ +# dasSpirv - cooperative matrix and emission-time unrolling + +Companion to `ARCHITECTURE.md`; section numbers are `ARCHITECTURE.md`'s. Two +cooperative-matrix arms run through these sections: **KHR** is `SPV_KHR_cooperative_matrix`, +the vendor-neutral subgroup-scope tile, and **cm2** is NVIDIA's `SPV_NV_cooperative_matrix2` +with `SPV_NV_tensor_addressing`, which adds workgroup-scope tiles and tensor-addressed block +loads that decode a quantized plane through a callback function. This document carries +sections 3.2 (the cm2 tile markers), 3.3 (the four-wide decode twin), 3.5 (a kernel calling a +decode method directly), 3.6 (`for [unroll_full]`) and 3.7 (the coopmat element loop's `Unroll`). +**Read `ARCHITECTURE_COMMON.md` (repo root) first - its contract binds this document.** The +checklists that bind a diff here are `REVIEW.md` (this folder) and +`modules/REVIEW_SHADER_EMITTERS.md`. + +### 3.2 A cm2 tile shape is one struct declaration {#cm2-tile-markers} + +The workgroup-scope cooperative-matrix tiles are marker structs in `spirv_builtins.das` whose +NAMES carry their geometry - `coopmatWg{A|B|Acc}_{f16|f32|s8|s32}_{R}x{C}` - and +`coopmat_wg_info` parses that name instead of looking the struct up in a table. Adding a tile +shape is therefore one struct declaration plus the overload that types the das call +(`coopmatMulAdd` for a multiply tile, `coopmatConvert` for an accumulator-only tile): no branch of +the emitter changes, because every cm2 branch reads rows, columns and component width out of the +parse. The tile markers are empty structs with no storage, so every builtin over them - the tensor +loads and stores, the decode forms, `coopmatMulAdd`, `coopmatConvert`, `coopmatClamp`, the +reductions - has an inert CPU body that cannot compute what the emitted form computes. A coopmat +kernel's device test therefore judges its result against a plain CPU reference of the same +arithmetic - one of the oracle forms `REVIEW.md` (this folder) lists - because running the +kernel body itself on the CPU computes nothing. A reduction width known only at run time reaches +a SPIR-V kernel through a `tensorLayout2D` or `tensorLayout2DPad` whose dimension +`tensorLayoutSetDimension` sets - that layout is this emitter's runtime-extent descriptor. + +### 3.3 The four-wide decode twin {#cm2-decode-vector} + +`SPV_NV_cooperative_matrix_decode_vector` adds no opcode. It adds one capability and the +`DecodeVectorFunc` bit of the tensor-load's addressing mask. That bit's operand names a second +decode function, which returns the 4-lane vector of the tile component type. The driver calls it +for four consecutive elements along the block's last dimension, starting at a coordinate that is +a multiple of four. The scalar `DecodeFunc` stays mandatory beside it and the driver picks per +call site, so one module serves devices with and without the feature. + +`coopmatLoadTensorDecode`'s tenth argument selects the twin. `true` has the emitter synthesize +it: four `OpFunctionCall`s of the scalar body at `coordInBlock.y + 0..3`, composed into the vector +(the driver's compiler inlines the calls and merges the loads they share). `false` is the +scalar-only load. A `[spirv_decode]` function returning `half4` over a `float16` decode, with the +scalar contract's parameters, is a hand-laid twin, named in that same tenth slot as `@@` +or `self.`. daslang spells no int8 or 32-bit four-vector the tiles would take, so the +hand-laid form is f16-only; the synthesized twin covers every scalar type a decode may return. +Twins register in their own table keyed by the function they came from, so one scalar body yields +one twin however many loads name it, and they emit after the scalar bodies they call. + +The extension stands apart from the cm2 base: a module that uses the twin declares the +`CooperativeMatrixDecodeVectorNV` capability and the `SPV_NV_cooperative_matrix_decode_vector` +extension name, and the twin runs only where the device has its own +`VkPhysicalDeviceCooperativeMatrixDecodeVectorFeaturesNV` bit enabled. + +The tensor-addressing operands follow the mask word in bit order - `TensorView`, then +`DecodeFunc`, then `DecodeVectorFunc` - after the load's fixed prefix: result type, result, +pointer, object, layout, memory-access mask and that mask's own extra words. The emitter appends +them in that order, and `strip_decode_vector` (below) counts forward by the same rule to the +operand it removes. + +A finished module is downgraded rather than recompiled. `strip_decode_vector` walks the word +stream and removes the capability, the extension declaration, and each +`OpCooperativeMatrixLoadTensorNV`'s `DecodeVectorFunc` bit together with its operand word. The +twin's `OpFunction` stays in the module, unreferenced, and the mandatory scalar `DecodeFunc` +serves the load. One emitted blob therefore runs on a device without the feature. + +### 3.5 A kernel calls a decode method directly {#direct-decode-call} + +A kernel body calling a `[spirv_decode]` method directly makes an ordinary user function call: +the method has its own OpFunction, registered beside the callback form and pulling no +cooperative-matrix capability. One decode body therefore serves a tensor load on a cm2 device and +a hand-staged tile on a KHR one. The block argument takes one of two forms, fixed per method at +discovery; a method called both ways is refused. + +Called on the plane element itself (`decode(wq[i], bc, cib)`), the block parameter is emitted as +the element's `uint` INDEX. The function's entry chains `OpAccessChain plane, 0, index` once, +which binds the parameter as a memory local in the plane's own storage class. The body's member +reads then chain through the plane exactly as the callback form's chain through its block +pointer, and the element is never loaded as a value. A pointer could not travel instead: under +logical addressing an SSBO pointer is not a legal OpFunctionCall argument without the +VariablePointersStorageBuffer capability, and the index needs no capability at all. Called on a +copy (`let blk = wq[i]; decode(blk, bc, cib)`), the block goes by value and lands in the spill +local of the next paragraph - the whole block loaded and stored per call, which the KHR tile +measures at a third of the element form's rate. + +**A struct value parameter with an aggregate member is a memory local.** A read-only parameter of +a plain data struct that carries a fixed-array or nested-struct member (not a coopmat tile, a +tensor object, a ray query or a sampler marker) binds its SSA OpFunctionParameter and is stored at +entry into a Function-storage OpVariable. The variable is declared first in the entry block, ahead +of the body's locals and call temps; the store that fills it follows the block's last OpVariable, +because SPIR-V requires every OpVariable of a block to lead the block. The parameter is thereafter +that local: its members access-chain like a `var` struct local's. The reason is a fixed-array +member indexed at run time (`blk.qs[cib.y >> 1]`): an SSA composite offers no pointer, and +`OpCompositeExtract` takes literal indices only. A scalar-only struct parameter keeps the value +path, and its members extract. A `let` copy of a block element in a body stays a composite - its +scalar and vector members extract, and its array and struct members are refused - because reading +it whole and then storing it would double every element load, while the parameter form pays the +store once per call. The Metal emitter needs no arm for this: an MSL function takes a struct by +value and its members are addressable as written. + +**A struct loaded out of a `Block` re-types at the call.** A struct loaded out of a `Block` - an +ssbo element - carries the Block's laid-out `OpTypeStruct`, a different type id from the plain +struct a parameter takes. A call passing one re-types the value with `OpCopyLogical` first, which +raises the module's version floor to 1.4 (`ARCHITECTURE.md` sec.2). The emitter remembers the +struct type each loaded id was read as and copies only when the two ids differ, so a struct +already at the plain type passes through untouched. + +### 3.6 `for [unroll_full]` unrolls at emission {#unroll-full} + +A `for [unroll_full] (i in range(lo, hi))` over literal bounds emits `hi - lo` copies of its body +and no loop construct. The emitter walks the body once per copy, with the induction variable bound +to that copy's `OpConstant`. + +The visitor framework's own walk of the body runs after those copies and emits one more. The +emitter therefore records the lengths of the function and decoration sections after the last copy +it emits itself, and at the loop's visit truncates both sections back to those lengths - the +discarded walk's words go. The error count and the terminated flag rewind with them. The induction +binding outlives the copies, so the discarded walk still resolves the variable. + +Scalar integer arithmetic on the induction constant folds at emission. Each side is a literal, the +induction constant, or an earlier fold on either, and at least one side comes from the unroll; the +folded forms are `+ - * / % << >> & | ^`, unary `-` and `~`, and a 32-bit `int`/`uint` cast. The +fold fires only inside an unrolled body, so a loop written without `[unroll_full]` emits its +arithmetic as ordinary instructions. `acc[t * 16 + c]` therefore chains an `OpConstant` index - +the shape a driver promotes to registers. A fixed array of cooperative-matrix tiles +(`coopmatAcc_f16_16x16[16]`) is one Function-storage `OpTypeArray` over the tile type, each +element reached through a constant `OpAccessChain`, so a 2-D subgroup tiling's accumulators stay +in registers where a rolled loop or the driver hint `[unroll]` leaves them indexed by the counter. + +The induction variable gets no `OpVariable`: it is a constant in every copy, so `collect_locals` +skips it. The body's other locals are declared once by that same pre-scan and shared across the +copies. + +A bound the compiler folds counts as a literal, so a named module constant is a legal bound (lint +and LSP compiles run with optimizations off, so `range(N)` and `int(KHR_KHALF_WORDS)` arrive +unfolded and the emitter asks for the fold). Refused: a bound the emitter cannot fold, `break`, +`continue`, `return`, `[unroll_full]` on a `while`, and any other loop hint written beside +`[unroll_full]`. An assignment to the induction variable never reaches the emitter - a loop +variable is a constant value, and the front end refuses the write. An error the body raises is +reported once, not once per copy. + +dasMetal lowers the hint to `#pragma clang loop unroll(full)`, and the JIT to +`llvm.loop.unroll.full`. The interpreter ignores it: the CPU run is the same body rolled. + +### 3.7 A cooperative-matrix element loop carries `Unroll` + +`coopmatClamp` walks a coopmat local element by element through a hand-emitted structured loop +bounded by `OpCooperativeMatrixLengthKHR`, and its `OpLoopMerge` sets loop control `Unroll` - the +control glslang emits for `[[unroll]]`. Rolled, the dynamic per-element `OpAccessChain` index +moves the accumulator out of the registers holding the tile into addressable storage for the +whole kernel, not only for the loop. diff --git a/modules/dasSpirv/REVIEW.md b/modules/dasSpirv/REVIEW.md index 7ee967eb39..06f81f6cdd 100644 --- a/modules/dasSpirv/REVIEW.md +++ b/modules/dasSpirv/REVIEW.md @@ -1,7 +1,8 @@ # dasSpirv Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `ARCHITECTURE.md`. Shared emitter rules: `modules/REVIEW_SHADER_EMITTERS.md` - apply that +docs: `ARCHITECTURE.md`, `ARCHITECTURE_COOPMAT.md` (cooperative matrix, emission-time +unrolling). Shared emitter rules: `modules/REVIEW_SHADER_EMITTERS.md` - apply that list with this one. A SPIR-V fixture - a file that compiles a shader and asserts on its emitted words - answers to `tests/spirv/REVIEW.md` (repo root), wherever the diff puts it. @@ -11,26 +12,35 @@ under `tests/spirv/` (repo root) that exercises the change and asserts on the em Emitted words no fixture asserts are produced by nothing the suite runs. The fixture forms are `ARCHITECTURE.md` section 4. -**A diff that adds a rejection path also adds its fixture under `tests/spirv/_fail_closed/` -(repo root) and asserts that fixture's error text in `tests/spirv/test_fail_closed.das`, in -the same change.** A rejection path is emitter code that refuses a construct with a compile -error. +**A diff that adds emitter code refusing a construct a `.das` program can compile to also adds +its fixture under `tests/spirv/_fail_closed/` (repo root) and asserts that fixture's error text +in `tests/spirv/test_fail_closed.das`, in the same change.** A guard on a construct the front +end refuses first is reachable from no `.das` program, so no fixture can drive it. **A diff that adds an emitter capability - a name the emitter recognizes, an opcode it emits, -or a type it accepts - that a downstream consumer uses also adds, in the same change, a test in -that consumer's device suite - `modules/dasVulkan/tests/integration/` or -`modules/dasLLAMA/tests/test_vulkan_kernels.das` - that runs a kernel using it on a device -against an independent CPU computation of the same result: the kernel body run on the CPU, the -CPU body that returns what the builtin's emitted form returns, or a plain CPU reference of the -same arithmetic, never an expectation re-spelled inline in the test.** A fixture asserts words; -only a device run shows the words compute. +or a type it accepts - that a downstream consumer uses leaves a device cell covering that +capability after the change.** A device cell runs a kernel using +the capability on a device, in `modules/dasVulkan/tests/integration/` or +`modules/dasLLAMA/tests/test_vulkan_kernels.das`. A fixture asserts words; only a device run +shows the words compute. + +**A diff under this folder that adds or changes a device cell judges that cell's result against +a CPU result computed independently of the emitter - the kernel body run on the CPU, a CPU body that returns what the +builtin's emitted form returns, or a plain CPU reference of the same arithmetic - never an +expectation re-spelled inline in the test.** An inline expectation is read off the emitter's own +output, so it passes whatever the emitter does. **A diff under `modules/dasSpirv` that edits a file under `modules/dasGlsl` or `modules/dasOpenGL` is a defect** - dasSpirv copies dasGlsl's design, not its code. **A diff that adds an `OpVariable` emit to an emitted function's entry block puts it ahead of every non-`OpVariable` instruction of that block, and an instruction it emits into that block -goes after the block's last `OpVariable` - in `emit_user_function`, after `collect_locals` and -`alloc_call_temps`.** SPIR-V requires every `OpVariable` of a block to lead the block -(`ARCHITECTURE.md` section 3.5), and CI runs no `spirv-val` to catch an invalid module -(section 4). +goes after the block's last `OpVariable`.** SPIR-V requires every `OpVariable` of a block to +lead the block (`ARCHITECTURE_COOPMAT.md` section 3.5), and CI runs no `spirv-val` to catch an +invalid module (`ARCHITECTURE.md` section 4). + +**A diff under `modules/dasSpirv`, or to a fixture under `tests/spirv` (repo root), runs +`tests/spirv` locally on a box that resolves `spirv-val` and names the run in the PR.** CI +resolves no `spirv-val`, so a module the validator rejects reds nowhere but on that box. The +run's command and what green means: `skills/internal/tests_in_repo.md`, the emitter suite +section. diff --git a/modules/dasSpirv/spirv/spirv_builder.das b/modules/dasSpirv/spirv/spirv_builder.das index c3d83d7b22..414c5c88a4 100644 --- a/modules/dasSpirv/spirv/spirv_builder.das +++ b/modules/dasSpirv/spirv/spirv_builder.das @@ -253,7 +253,7 @@ def public ensure_coopmat_conversions(var m : SpirvModule) { m.type_pool |> insert("cap_cm2_convert", 1u) } -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def public ensure_coopmat_decode_vector(var m : SpirvModule) { ensure_cooperative_matrix2(m) return if ((m.type_pool?["cap_cm2_decvec"] ?? 0u) != 0u) diff --git a/modules/dasSpirv/spirv/spirv_builtins.das b/modules/dasSpirv/spirv/spirv_builtins.das index c7e7209d57..8aa27de194 100644 --- a/modules/dasSpirv/spirv/spirv_builtins.das +++ b/modules/dasSpirv/spirv/spirv_builtins.das @@ -447,7 +447,7 @@ struct coopmatWgB_f16_32x32 {} struct coopmatWgB_f16_32x128 {} struct coopmatWgB_f16_64x64 {} struct coopmatWgB_f16_128x32 {} -[arch(at="../ARCHITECTURE.md#cm2-tile-markers")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-tile-markers")] struct coopmatWgAcc_f16_64x128 {} struct coopmatWgAcc_f32_64x128 {} // the h256 flash tiles (Br=64, Bc=32): Q & O 64x256, Qf16 as A, K^T as B 256x32, V as B 32x256 diff --git a/modules/dasSpirv/spirv/spirv_dis.das b/modules/dasSpirv/spirv/spirv_dis.das index 31b18f8131..0c3a6bfc5b 100644 --- a/modules/dasSpirv/spirv/spirv_dis.das +++ b/modules/dasSpirv/spirv/spirv_dis.das @@ -153,7 +153,7 @@ def private memory_access_extra_words(mask : uint) : int { //! Downgrades a finished module for a device without the four-wide decode: the scalar callback //! serves every load afterwards. True when anything was removed. -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def public strip_decode_vector(var words : array) : bool { var out : array out |> reserve(length(words)) diff --git a/modules/dasSpirv/spirv/spirv_emit.das b/modules/dasSpirv/spirv/spirv_emit.das index 473e4e10a5..4bb97ad7d0 100644 --- a/modules/dasSpirv/spirv/spirv_emit.das +++ b/modules/dasSpirv/spirv/spirv_emit.das @@ -220,6 +220,17 @@ struct LoopTargets { cont : uint } +//! the targets a `for [unroll_full]` body runs under: no merge and no continue block exist +def private unrolled_loop_targets : LoopTargets => LoopTargets(merge = 0u, cont = 0u) +def private is_unrolled_body(lt : LoopTargets) : bool => lt.merge == 0u + +struct UnrollRewind { + funcs_len : int + decor_len : int + errs_len : int + terminated : bool +} + // ===== per-control-flow-node label ids (visitor side-maps; allocated to match the hand-walker's // linear id order so the emitted blob stays byte-identical) ===== struct IteIds { @@ -534,7 +545,7 @@ def private is_subpass_input_type(t : TypeDecl?) : bool { } // The caller already verified the "coopmatWg" prefix. One peek_data pass; `good` fails the parse closed. -[arch(at="../ARCHITECTURE.md#cm2-tile-markers")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-tile-markers")] def private coopmat_wg_info(n : string) : tuple { var ok = false var isf = false @@ -3009,7 +3020,8 @@ def private emit_mul(var m : SpirvModule; op2 : ExprOp2?; e : ExpressionPtr; l, // SPIR-V requires every Function-storage OpVariable to be among the first instructions of the // function's first block. So we pre-scan the whole body (one walk, all scopes — SPIR-V locals are // function-scoped, not block-scoped) BEFORE emitting any body code, declaring an OpVariable for each -// mutable local (`var`) and loop-induction variable. Reads/writes then resolve to OpLoad/OpStore. +// mutable local (`var`) and the induction variable of a rolled loop. Reads/writes then resolve to OpLoad/OpStore. +[arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] def private declare_local_var(var m : SpirvModule; var ctx : EmitCtx; v : VariablePtr; var errors : array) { // A REFERENCE local aliases storage that already has an address, so it gets no OpVariable of its // own: visitExprLetVariable binds it to the pointer its initializer lowered to. Declaring one @@ -3059,6 +3071,15 @@ def private declare_local_var(var m : SpirvModule; var ctx : EmitCtx; v : Variab ctx.local_vars |> insert(intptr(v), LocalVar(ptr_id = cmid, value_type = cmt)) return } + if (bt == Type.tFixedArray && v._type.firstType != null && coopmat_info(v._type.firstType).ok) { + let elt = coopmat_type_of(m, v._type.firstType) + let arrt = type_array(m, elt, v._type.fixedDim) + let arrpt = type_pointer(m, SpvStorageClass.Function, arrt) + let arrid = alloc_id(m) + emit(m, SEC_FUNCS, SpvOp.Variable, arrpt, arrid, uint(SpvStorageClass.Function)) + ctx.local_vars |> insert(intptr(v), LocalVar(ptr_id = arrid, value_type = arrt)) + return + } // a matrix is a tHandle, so numeric_info does not see it -- emit_type lowers it to OpTypeMatrix, // which is a perfectly good Function-storage pointee (glslang gives a local `mat4` an OpVariable too) if (!is_emittable_value_type(v._type) && bt != Type.tFixedArray && bt != Type.tStructure && @@ -3081,6 +3102,7 @@ def private declare_local_var(var m : SpirvModule; var ctx : EmitCtx; v : Variab ctx.local_vars |> insert(intptr(v), LocalVar(ptr_id = pid, value_type = vt)) } +[arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] def private collect_locals(var m : SpirvModule; var ctx : EmitCtx; e : ExpressionPtr; var errors : array) { if (e == null) return let blk = e ?as ExprBlock @@ -3114,8 +3136,10 @@ def private collect_locals(var m : SpirvModule; var ctx : EmitCtx; e : Expressio } let fr = e ?as ExprFor if (fr != null) { - for (fv in fr.iteratorVariables) { - declare_local_var(m, ctx, fv, errors) + if (!unrolls_at_emission(fr)) { + for (fv in fr.iteratorVariables) { + declare_local_var(m, ctx, fv, errors) + } } collect_locals(m, ctx, fr.body, errors) return @@ -3157,6 +3181,116 @@ def private const_int_for(var m : SpirvModule; cls : int; v : int) : uint { return const_int(m, v) } +def private has_loop_hint(args : AnnotationArgumentList; name : string) : bool { + for (a in args) { + if (a.name == name) return true + } + return false +} + +[arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] +def private literal_int_of(e : ExpressionPtr) : tuple { + let ci = e ?as ExprConstInt + if (ci != null) return (ok = true, value = ci.value) + let cu = e ?as ExprConstUInt + if (cu != null) return (ok = true, value = int(cu.value)) + let ce = get_const_expr(compiling_program(), e) + let fi = ce ?as ExprConstInt + if (fi != null) return (ok = true, value = fi.value) + let fu = ce ?as ExprConstUInt + if (fu != null) return (ok = true, value = int(fu.value)) + return (ok = false, value = 0) +} + +//! the induction variable of such a loop is a constant in every copy: it gets no OpVariable +def private unrolls_at_emission(fr : ExprFor?) : bool => has_loop_hint(fr.annotations, "unroll_full") && length(fr.sources) == 1 && literal_range_of(fr.sources[0]).ok + +[arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] +def private literal_range_of(src : ExpressionPtr) : tuple { + let cr = src ?as ExprConstRange + if (cr != null) { + let ft = unsafe(reinterpret(cr.value)) + return (ok = true, lo = ft.x, hi = ft.y) + } + let cur = src ?as ExprConstURange + if (cur != null) { + let ft = unsafe(reinterpret(cur.value)) + return (ok = true, lo = int(ft.x), hi = int(ft.y)) + } + let call = src ?as ExprCall + if (call == null || ("{call.name}" != "range" && "{call.name}" != "urange")) return (ok = false, lo = 0, hi = 0) + let na = length(call.arguments) + if (na != 1 && na != 2) return (ok = false, lo = 0, hi = 0) + var lo = 0 + if (na == 2) { + let lv = literal_int_of(call.arguments[0]) + if (!lv.ok) return (ok = false, lo = 0, hi = 0) + lo = lv.value + } + let hv = literal_int_of(call.arguments[na - 1]) + if (!hv.ok) return (ok = false, lo = 0, hi = 0) + return (ok = true, lo = lo, hi = hv.value) +} + +def private fold_uint_op(op : string; ua, ub : uint) : tuple { + var r = 0u + if (op == "+") { + r = ua + ub + } elif (op == "-") { + r = ua - ub + } elif (op == "*") { + r = ua * ub + } elif (op == "/" || op == "%") { + if (ub == 0u) return (ok = false, value = 0l) + r = op == "/" ? ua / ub : ua % ub + } elif (op == "<<" || op == ">>") { + if (ub >= 32u) return (ok = false, value = 0l) + r = op == "<<" ? ua << ub : ua >> ub + } elif (op == "&") { + r = ua & ub + } elif (op == "|") { + r = ua | ub + } elif (op == "^") { + r = ua ^ ub + } else { + return (ok = false, value = 0l) + } + return (ok = true, value = int64(r)) +} + +let private INT32_MIN = -2147483647 - 1 + +def private fold_sint_op(op : string; ia, ib : int) : tuple { + var r = 0 + if (op == "+") { + r = ia + ib + } elif (op == "-") { + r = ia - ib + } elif (op == "*") { + r = ia * ib + } elif (op == "/" || op == "%") { + if (ib == 0 || (ib == -1 && ia == INT32_MIN)) return (ok = false, value = 0l) + r = op == "/" ? ia / ib : ia % ib + } elif (op == "<<" || op == ">>") { + if (ib < 0 || ib >= 32) return (ok = false, value = 0l) + r = op == "<<" ? ia << ib : ia >> ib + } elif (op == "&") { + r = ia & ib + } elif (op == "|") { + r = ia | ib + } elif (op == "^") { + r = ia ^ ib + } else { + return (ok = false, value = 0l) + } + return (ok = true, value = int64(r)) +} + +def private fold_int_op(op : string; cls : int; a, b : int64) : tuple { + if (cls == 1) return fold_uint_op(op, uint(a), uint(b)) + return fold_sint_op(op, int(a), int(b)) +} + // ===== SpirvEmit : AstVisitor ===== // The body-codegen visitor (modeled on llvm_jit). generate_spirv builds the module scaffold (caps / // entry-point / OpFunction / trailing return) and drives this visitor over fn.body for all codegen. @@ -3180,6 +3314,10 @@ class SpirvEmit : AstVisitor { lazy_ids : table //!< intptr(ExprOp3 / ExprOp2 &&,||) lowered as a branch (ctx.lazy_temps) -> its labels loop_ids : table // intptr(ExprWhile/ExprFor) -> its 5 loop-skeleton labels for_src : table // for-source range() call ptrs to intercept in visitExprCall (6.3) + unroll_binds : table //!< intptr(Variable) of an unrolled induction variable -> the OpConstant of the copy being emitted + unroll_consts : table //!< OpConstant ids an unrolled copy derives (the induction constant, the folds on it) -> value + lit_consts : table //!< every 32-bit integer literal's OpConstant id -> value: the operand a fold may take beside a derived one + unroll_rewinds : table //!< intptr(ExprFor) unrolled at emission -> the emitter state the visitor's own walk of its body rewinds to copy_dst : uint64 // intptr(ExprCopy.left) while that destination subtree is walked suppress_bounds_arg : bool = false // next node is a `.[]` call's __context__/__lineinfo__ tail — elide suppress_fn_addr : bool = false // next node is coopmatLoadTensorDecode's trailing @@decode — read structurally by the cm2 load arm, not lowered @@ -3528,12 +3666,16 @@ class SpirvEmit : AstVisitor { // ----- constants ----- def override visitExprConstInt(var expr : ExprConstInt?) : ExpressionPtr { var bm & = unsafe(*m) - e2id[intptr(expr)] = const_int(bm, expr.value) + let cid = const_int(bm, expr.value) + lit_consts[cid] = int64(expr.value) + e2id[intptr(expr)] = cid return expr } def override visitExprConstUInt(var expr : ExprConstUInt?) : ExpressionPtr { var bm & = unsafe(*m) - e2id[intptr(expr)] = const_uint(bm, expr.value) + let cid = const_uint(bm, expr.value) + lit_consts[cid] = int64(expr.value) + e2id[intptr(expr)] = cid return expr } def override visitExprConstFloat(var expr : ExprConstFloat?) : ExpressionPtr { @@ -3647,6 +3789,10 @@ class SpirvEmit : AstVisitor { if (ctx.self_arg != 0ul && vk == ctx.self_arg) { return expr // the kernel class's self: every use resolves via member_var_of at the access site } + if (key_exists(unroll_binds, vk)) { + e2id[k] = unroll_binds?[vk] ?? 0u + return expr + } if (key_exists(ctx.local_vars, vk)) { let lv = ctx.local_vars?[vk] ?? LocalVar() e2ptr[k] = lv.ptr_id @@ -3782,7 +3928,8 @@ class SpirvEmit : AstVisitor { } let idx = value_of(index_expr) if (idx == 0u) return // index failed to lower (error already pushed) — no 0-operand access chain - let pointee = emit_type(bm, result_expr._type) + let cmt = coopmat_type_of(bm, result_expr._type) + let pointee = cmt != 0u ? cmt : emit_type(bm, result_expr._type) let ptr_t = type_pointer(bm, SpvStorageClass.Function, pointee) let res = alloc_id(bm) emit(bm, SEC_FUNCS, SpvOp.AccessChain, ptr_t, res, lv.ptr_id, idx) @@ -3939,6 +4086,17 @@ class SpirvEmit : AstVisitor { } let v = value_of(expr.subexpr) if (v == 0u) return expr + if (!empty(unroll_binds) && key_exists(unroll_consts, v) && (op == "-" || op == "~")) { + let ucls = scalar_class(expr.subexpr._type.baseType) + if ((ucls == 0 || ucls == 1) && arith_lanes(expr.subexpr._type.baseType) == 1) { + let bits = int(unroll_consts?[v] ?? 0l) + let r = op == "-" ? (ucls == 1 ? int64(0u - uint(bits)) : int64(-bits)) : (ucls == 1 ? int64(~uint(bits)) : int64(~bits)) + let cid = const_int_for(bm, ucls, int(r)) + unroll_consts[cid] = r + e2id[intptr(expr)] = cid + return expr + } + } if (op == "!") { let bt = type_bool(bm) let res = alloc_id(bm) @@ -3958,6 +4116,25 @@ class SpirvEmit : AstVisitor { return expr } + [arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] + def fold_unrolled_op2(op : string; cls : int; lbt : Type; l, r : uint; k : uint64) : bool { + if (empty(unroll_binds) || (cls != 0 && cls != 1) || arith_lanes(lbt) != 1) return false + let l_derived = key_exists(unroll_consts, l) + let r_derived = key_exists(unroll_consts, r) + let l_known = l_derived || key_exists(lit_consts, l) + let r_known = r_derived || key_exists(lit_consts, r) + if (!(l_derived || r_derived) || !l_known || !r_known) return false + let a = l_derived ? (unroll_consts?[l] ?? 0l) : (lit_consts?[l] ?? 0l) + let b = r_derived ? (unroll_consts?[r] ?? 0l) : (lit_consts?[r] ?? 0l) + let fv = fold_int_op(op, cls, a, b) + if (!fv.ok) return false + var bm & = unsafe(*m) + let cid = const_int_for(bm, cls, int(fv.value)) + unroll_consts[cid] = fv.value + e2id[k] = cid + return true + } + [arch(at="../ARCHITECTURE.md#operand-laziness")] def override preVisitExprOp2Right(var expr : ExprOp2?; right : ExpressionPtr) : void { let k = intptr(expr) @@ -4026,7 +4203,7 @@ class SpirvEmit : AstVisitor { let cls = arith_class(lbt) let l = value_of(expr.left) let r = value_of(expr.right) - if (l == 0u || r == 0u) return expr + if (l == 0u || r == 0u || fold_unrolled_op2(op, cls, lbt, l, r, intptr(expr))) return expr if (op == "*") { let mr = emit_mul(bm, expr, expr, l, r, errs) if (mr.handled) { @@ -4286,6 +4463,10 @@ class SpirvEmit : AstVisitor { // block with OpUnreachable so the blob stays structurally valid (one terminator per block). def override visitExprReturn(var expr : ExprReturn?) : ExpressionPtr { var bm & = unsafe(*m) + if (!empty(ctx.loop_stack) && is_unrolled_body(ctx.loop_stack[length(ctx.loop_stack) - 1])) { + errs |> push("return inside a `for [unroll_full]` body: the loop is unrolled at emission and a return in one copy leaves the copies after it unreachable - restructure the body or take [unroll]") + return expr + } if (expr.subexpr != null) { let v = value_of(expr.subexpr) if (v != 0u) { @@ -4508,7 +4689,7 @@ class SpirvEmit : AstVisitor { // remaining arguments sit AHEAD of f.arguments — a method invoke is [Cls.method field, // receiver, args...] against f.arguments [self, params...], so skip=2 / fshift=1 pairs // args[i] with f.arguments[i-1] (and the callee's self was dropped at registration). - [arch(at="../ARCHITECTURE.md#direct-decode-call")] + [arch(at="../ARCHITECTURE_COOPMAT.md#direct-decode-call")] def emit_call_core(var bm : SpirvModule; uf : UserFunc; arguments : dasvector`ptr`Expression; skip, fshift : int; resKey : uint64) { let f = uf.fn let res = alloc_id(bm) @@ -4618,7 +4799,7 @@ class SpirvEmit : AstVisitor { // ----- calls: texture / vector constructors / dot / GLSL.std.450 math (args pre-visited) ----- // Mirrors emit_call but reads operand ids via value_of (children already visited). for-source // range()/urange() calls are consumed by preVisitExprForBody and skipped here. - [arch(at="../ARCHITECTURE.md#files-and-emission"), arch(at="../ARCHITECTURE.md#small-int-surface"), arch(at="../ARCHITECTURE.md#cm2-decode-vector")] + [arch(at="../ARCHITECTURE.md#files-and-emission"), arch(at="../ARCHITECTURE.md#small-int-surface"), arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def override visitExprCall(var expr : ExprCall?) : ExpressionPtr { if (key_exists(for_src, intptr(expr))) return expr var bm & = unsafe(*m) @@ -6043,6 +6224,13 @@ class SpirvEmit : AstVisitor { if (aid == 0u) return expr let src_cls = sn.cls let src_w = sn.width + if (!empty(unroll_binds) && key_exists(unroll_consts, aid) && dst.width == 32 && (dst.cls == 0 || dst.cls == 1) && (src_cls == 0 || src_cls == 1) && src_w == 32) { + let bits = int(unroll_consts?[aid] ?? 0l) + let cid = const_int_for(bm, dst.cls, bits) + unroll_consts[cid] = dst.cls == 1 ? int64(uint(bits)) : int64(bits) + e2id[k] = cid + return expr + } if (src_cls == dst.cls && src_w == dst.width) { // same class+width no-op, e.g. float(someFloat) e2id[k] = aid return expr @@ -6456,6 +6644,8 @@ class SpirvEmit : AstVisitor { for (a in args) { if (a.name == "unroll") { mask |= uint(SpvLoopControl.Unroll) + } elif (a.name == "unroll_full") { + errs |> push("loop hint [unroll_full] applies to a `for (i in range(lo, hi))` with literal bounds, which unrolls at emission; a while loop takes [unroll], the driver's hint") } elif (a.name == "dont_unroll") { mask |= uint(SpvLoopControl.DontUnroll) } elif (a.name == "dependency_infinite") { @@ -6579,16 +6769,20 @@ class SpirvEmit : AstVisitor { return } let iv = expr.iteratorVariables[0] - let lvr = ctx.local_vars?[intptr(iv)] ?? LocalVar() - if (lvr.ptr_id == 0u) { - errs |> push("for: induction variable '{iv.name}' was not declared") - return - } let cls = scalar_class(iv._type.baseType) if (cls != 0 && cls != 1) { errs |> push("for: induction variable '{iv.name}' must be int or uint (got {iv._type.baseType})") return } + if (has_loop_hint(expr.annotations, "unroll_full")) { + unroll_at_emission(expr, iv, cls) + return + } + let lvr = ctx.local_vars?[intptr(iv)] ?? LocalVar() + if (lvr.ptr_id == 0u) { + errs |> push("for: induction variable '{iv.name}' was not declared") + return + } var lo = 0u var hi = 0u let cr = expr.sources[0] ?as ExprConstRange @@ -6637,7 +6831,21 @@ class SpirvEmit : AstVisitor { ctx.loop_stack |> push(LoopTargets(merge = ids.merge, cont = ids.cont_l)) } + [arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] def override visitExprFor(var expr : ExprFor?) : ExpressionPtr { + if (key_exists(unroll_rewinds, intptr(expr))) { + let um = unroll_rewinds?[intptr(expr)] ?? UnrollRewind() + var bm & = unsafe(*m) + resize(bm.sections[SEC_FUNCS], um.funcs_len) + resize(bm.sections[SEC_DECOR], um.decor_len) + if (length(errs) > um.errs_len) { + resize(errs, um.errs_len) + } + ctx.terminated = um.terminated + if (!empty(ctx.loop_stack)) pop(ctx.loop_stack) + unroll_binds |> erase(intptr(expr.iteratorVariables[0])) + return expr + } if (!key_exists(loop_ids, intptr(expr))) return expr // setup failed; errors already pushed let ids = loop_ids?[intptr(expr)] ?? LoopIds() var bm & = unsafe(*m) @@ -6659,6 +6867,67 @@ class SpirvEmit : AstVisitor { return expr } + [arch(at="../ARCHITECTURE_COOPMAT.md#unroll-full")] + def unroll_at_emission(var expr : ExprFor?; iv : VariablePtr; cls : int) { + var bm & = unsafe(*m) + if (!unroll_hints_ok(expr.annotations)) return + let rg = literal_range_of(expr.sources[0]) + if (!rg.ok) { + errs |> push("for [unroll_full]: the range must have literal bounds - range(N) or range(lo, hi) - to unroll at emission; a runtime bound takes [unroll], the driver's hint") + return + } + let vk = intptr(iv) + ctx.loop_stack |> push(unrolled_loop_targets()) + for (kv in range(rg.lo, rg.hi)) { + let cid = const_int_for(bm, cls, kv) + unroll_consts[cid] = cls == 1 ? int64(uint(kv)) : int64(kv) + unroll_binds[vk] = cid + let errs_before = length(errs) + visit(expr.body, adapter) + drop_repeated_errors(errs_before) + } + unroll_binds[vk] = const_int_for(bm, cls, rg.lo) //!< the binding outlives the copies: the discarded walk resolves the induction variable too + unroll_rewinds[intptr(expr)] = UnrollRewind(funcs_len = length(bm.sections[SEC_FUNCS]), decor_len = length(bm.sections[SEC_DECOR]), + errs_len = length(errs), terminated = ctx.terminated) + } + + //! an error a copy raises is reported once: a copy's error already on the list from an earlier copy goes, a new one stays + def drop_repeated_errors(start : int) { + var kept : array + for (i in range(start, length(errs))) { + var seen = false + for (j in range(start)) { + if (errs[j] == errs[i]) { + seen = true + break + } + } + if (!seen) { + kept |> push(errs[i]) + } + } + resize(errs, start) + errs |> push_from(kept) + } + + //! the hints beside [unroll_full]: an unknown name, or a driver loop control that has no loop construct to attach to, is refused + def unroll_hints_ok(hints : AnnotationArgumentList) : bool { + var ok = true + for (a in hints) { + let n = "{a.name}" + if (n == "unroll_full") continue + if (n == "unroll" || n == "dont_unroll" || n == "dependency_infinite" || n == "dependency_length" || n == "min_iterations" + || n == "max_iterations" || n == "iteration_multiple" || n == "peel_count" || n == "partial_count") { + errs |> push("loop hint [{n}] beside [unroll_full]: the loop is unrolled at emission and emits no loop construct to carry it - drop it, or take [unroll] alone for the driver's hint") + ok = false + } else { + errs |> push("unknown vulkan loop hint '{n}'") + ok = false + } + } + return ok + } + // ----- break / continue -> branch to the innermost loop's merge / continue block ----- def override preVisitExprBreak(var expr : ExprBreak?) : void { var bm & = unsafe(*m) @@ -6666,7 +6935,12 @@ class SpirvEmit : AstVisitor { errs |> push("break outside of a loop") return } - emit(bm, SEC_FUNCS, SpvOp.Branch, ctx.loop_stack[length(ctx.loop_stack) - 1].merge) + let lt = ctx.loop_stack[length(ctx.loop_stack) - 1] + if (is_unrolled_body(lt)) { + errs |> push("break inside a `for [unroll_full]` body: the loop is unrolled at emission and has no merge block to branch to - restructure the body or take [unroll]") + return + } + emit(bm, SEC_FUNCS, SpvOp.Branch, lt.merge) ctx.terminated = true } @@ -6676,7 +6950,12 @@ class SpirvEmit : AstVisitor { errs |> push("continue outside of a loop") return } - emit(bm, SEC_FUNCS, SpvOp.Branch, ctx.loop_stack[length(ctx.loop_stack) - 1].cont) + let lt = ctx.loop_stack[length(ctx.loop_stack) - 1] + if (is_unrolled_body(lt)) { + errs |> push("continue inside a `for [unroll_full]` body: the loop is unrolled at emission and has no continue block to branch to - restructure the body or take [unroll]") + return + } + emit(bm, SEC_FUNCS, SpvOp.Branch, lt.cont) ctx.terminated = true } @@ -7067,7 +7346,7 @@ def private register_decode_func(var m : SpirvModule; var ctx : EmitCtx; f : Fun //! Returns the twin's id; 0 is both the `false` form and a rejection - the caller tells them apart //! by the error count. -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def private resolve_decode_vector_arg(var m : SpirvModule; var ctx : EmitCtx; va0 : ExpressionPtr; sf : Function const?; var errors : array) : uint { let r2v = va0 ?as ExprRef2Value let va = r2v != null ? r2v.subexpr : va0 @@ -7107,7 +7386,7 @@ def private resolve_decode_vector_arg(var m : SpirvModule; var ctx : EmitCtx; va return df.valid ? df.fn_id : 0u } -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def private register_decode_vector_synth(var m : SpirvModule; var ctx : EmitCtx; sf : Function const?) : uint { if (key_exists(ctx.decode_vec_funcs, intptr(sf))) { let ex = ctx.decode_vec_funcs?[intptr(sf)] ?? DecodeFn() @@ -7130,7 +7409,7 @@ def private register_decode_vector_synth(var m : SpirvModule; var ctx : EmitCtx; return df.valid ? df.fn_id : 0u } -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def private validate_decode_func(var m : SpirvModule; ctx : EmitCtx; f : Function const?; vector_twin : bool; var errors : array) : DecodeFn { var df = DecodeFn(fn = f, valid = true) if (vector_twin) { @@ -7264,7 +7543,7 @@ def private emit_decode_function(var m : SpirvModule; var ctx : EmitCtx; df : De ctx.self_arg = saved_self } -[arch(at="../ARCHITECTURE.md#cm2-decode-vector")] +[arch(at="../ARCHITECTURE_COOPMAT.md#cm2-decode-vector")] def private emit_decode_vector_synth_body(var m : SpirvModule; ctx : EmitCtx; df : DecodeFn; p0 : uint; coord_pids : uint[2]) { let sdf = ctx.decode_funcs?[df.synth_of] ?? DecodeFn() let ut = type_uint(m) @@ -7483,7 +7762,7 @@ class private SpirvMethodScan : AstVisitor { ctx.invoke_fields |> insert(intptr(da0)) found_decode |> push(mfn) } - [arch(at="../ARCHITECTURE.md#direct-decode-call")] + [arch(at="../ARCHITECTURE_COOPMAT.md#direct-decode-call")] def note_decode_elem_arg(mfn : FunctionPtr; expr : ExprInvoke const?) : void { if (!function_has_annotation(mfn, "spirv_decode") || length(expr.arguments) < 3) return let a0 = expr.arguments[2] @@ -7757,7 +8036,7 @@ def private alloc_call_temps(var m : SpirvModule; var ctx : EmitCtx; f : Functio } } -[arch(at="../ARCHITECTURE.md#direct-decode-call")] +[arch(at="../ARCHITECTURE_COOPMAT.md#direct-decode-call")] def private spills_struct_param(var m : SpirvModule; at : TypeDecl?) : bool { if (at == null || at.baseType != Type.tStructure || at.structType == null || coopmat_info(at).ok || sampler_info(at).ok || "{at.structType.name}" == "rayQueryEXT" @@ -7776,7 +8055,7 @@ def private spills_struct_param(var m : SpirvModule; at : TypeDecl?) : bool { // SSA id, a struct value param additionally spills into a Function local; ref params bind a Function // pointer), the entry block (locals + ref-arg temps + the spill stores), the body, and the trailing // return. The result id / type were pre-allocated by register_user_func. -[arch(at="../ARCHITECTURE.md#direct-decode-call")] +[arch(at="../ARCHITECTURE_COOPMAT.md#direct-decode-call")] def private emit_user_function(var m : SpirvModule; var ctx : EmitCtx; uf : UserFunc; var errors : array) { let f = unsafe(reinterpret(uf.fn)) // read-only emission; strip pointee const emit(m, SEC_FUNCS, SpvOp.Function, uf.ret_type, uf.fn_id, 0u, uf.type_id) diff --git a/plans/REVIEW.md b/plans/REVIEW.md index 6e71d32c1c..5aa056533f 100644 --- a/plans/REVIEW.md +++ b/plans/REVIEW.md @@ -2,6 +2,9 @@ **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** -**A document here that a dasLLAMA change writes - a model board, a kernel census, a ledger of -that module's work - applies `modules/dasLLAMA/REVIEW_UPSTREAM.md` too.** Its figure and -reference-build rules bind the rows and readings the document carries. +**A diff that writes a measured number a dasLLAMA run or its reference build produced into a +document here applies `modules/dasLLAMA/REVIEW_MEASUREMENT.md` too.** The reference build is the +third-party engine dasLLAMA measures itself against. + +**A diff that writes a reference-build name, or a description of one of its mechanisms, into a +document here applies `modules/dasLLAMA/REVIEW_UPSTREAM.md` too.** diff --git a/plans/qwen_arc_board.md b/plans/qwen_arc_board.md index 39f0c51ec9..323a6c1d23 100644 --- a/plans/qwen_arc_board.md +++ b/plans/qwen_arc_board.md @@ -30,32 +30,32 @@ final PR body of the arc quotes it. | # | model | GB | arch | status | ours cm2 pp / tg | llama.cpp cm2 pp / tg | ratio | ours KHR pp / tg | llama.cpp KHR pp / tg | |---|---|---|---|---|---|---|---|---|---| | 1 | Qwen2.5-0.5B-Instruct-Q8_0 | 0.53 | qwen2 | works (fixed: the q/k/v bias arm) | 42355 / 411.8 | 32338 / 381.9 | 1.31 / 1.08 | 33375 / 411.6 | 25437 / 365.9 | -| 2 | Qwen3.5-0.8B-Q4_K_M (local requant, the hybrid test twin) | 0.53 | qwen35 hybrid | works | 16651 / 314.2 | 16752 / 280.0 | 0.99 / 1.12 | 8580 / 314.8 | 14760 / 280.3 | +| 2 | Qwen3.5-0.8B-Q4_K_M (local requant, the hybrid test twin) | 0.53 | qwen35 hybrid | works; KHR re-measured on the word-stage kq tile (8580 -> 16052) | 16651 / 314.2 | 16752 / 280.0 | 0.99 / 1.12 | 16052 / 313.3 | 14760 / 280.3 | | 3 | Qwen3-0.6B-Q8_0 | 0.64 | qwen3 | works; tg 0.95x = the dispatch count per token (see notes) | 29065 / 318.8 | 22736 / 335.9 | 1.28 / 0.95 | 21187 / 319.8 | 20919 / 330.6 | -| 4 | Qwen2.5-1.5B-Instruct-IQ3_XS | 0.73 | qwen2 (bias) + iq2s/iq3xxs planes | works; pp 0.91x = the grid formats' cm2 decode callbacks | 12517 / 264.7 | 13830 / 252.0 | 0.91 / 1.05 | 4482 / 264.5 | 10809 / 245.9 | +| 4 | Qwen2.5-1.5B-Instruct-IQ3_XS | 0.73 | qwen2 (bias) + iq2s/iq3xxs planes | works; pp 0.91x = the grid formats' cm2 decode callbacks; KHR re-measured on the word-stage kq tile (4482 -> 12058) | 12517 / 264.7 | 13830 / 252.0 | 0.91 / 1.05 | 12058 / 263.8 | 10809 / 245.9 | | 5 | Qwen3.5-0.8B-Q8_0 | 0.81 | qwen35 hybrid | works | 19222 / 272.5 | 16624 / 243.6 | 1.16 / 1.12 | 17007 / 274.0 | 14978 / 242.6 | | 6 | Qwen2.5-1.5B-Instruct-Q8_0 | 1.65 | qwen2 | works; tg 0.93x (see notes) | 14080 / 180.1 | 14184 / 193.1 | 0.99 / 0.93 | 11309 / 181.2 | 10813 / 189.0 | -| 7 | Qwen3-4B-Instruct-2507-Q4_K_M | 2.50 | qwen3 | works; tg 0.93x (see notes); KHR re-measured after the kq tile and its slab fix (`followup_vulkan.md` item 42) | 5150 / 117.6 | 5142 / 126.3 | 1.00 / 0.93 | 3051 / 117.4 | 4221 / 125.7 | -| 8 | Qwen3-4B-Instruct-2507-Q5_K_M | 2.89 | qwen3 | works; tg 0.93x (see notes) | 4700 / 104.8 | 4965 / 113.2 | 0.95 / 0.93 | 1489 / 104.8 | 4015 / 111.3 | -| 9 | Qwen3-4B-Instruct-2507-Q6_K | 3.31 | qwen3 | works | 4566 / 95.88 | 4762 / 97.50 | 0.96 / 0.98 | 1581 / 95.79 | 3911 / 96.81 | +| 7 | Qwen3-4B-Instruct-2507-Q4_K_M | 2.50 | qwen3 | works; tg 0.93x (see notes); KHR re-measured on the word-stage kq tile (`followup_vulkan.md` item 42): 1564 -> 3051 -> 4764 | 5150 / 117.6 | 5142 / 126.3 | 1.00 / 0.93 | 4764 / 116.8 | 4221 / 125.7 | +| 8 | Qwen3-4B-Instruct-2507-Q5_K_M | 2.89 | qwen3 | works; tg 0.93x (see notes); KHR re-measured on the word-stage kq tile (1489 -> 4422) | 4700 / 104.8 | 4965 / 113.2 | 0.95 / 0.93 | 4422 / 102.9 | 4015 / 111.3 | +| 9 | Qwen3-4B-Instruct-2507-Q6_K | 3.31 | qwen3 | works; KHR re-measured on the word-stage kq tile (1581 -> 4284) | 4566 / 95.88 | 4762 / 97.50 | 0.96 / 0.98 | 4284 / 95.69 | 3911 / 96.81 | | 10 | Qwen3-4B-Instruct-2507-Q8_0 | 4.28 | qwen3 | works; tg 0.94x (see notes) | 5907 / 79.84 | 5064 / 85.03 | 1.17 / 0.94 | 5304 / 79.70 | 4614 / 84.14 | | 11 | Qwen3.5-4B-Q8_0 | 4.48 | qwen35 hybrid | works (fixed: the mirror's binding cap on a hybrid) | 5087 / 74.37 | 4120 / 74.78 | 1.23 / 0.99 | 4562 / 74.74 | 3657 / 74.46 | -| 12 | Qwen3.5-9B-IQ4_XS | 5.47 | qwen35 hybrid | works (re-measured no pin, ctx 262143) | 2700 / 62.94 | 2794 / 66.82 | 0.97 / 0.94 | 738 / 63.17 | 2199 / 66.26 | -| 13 | Qwen3.5-9B-MTP-Q5_K_M | 6.64 | qwen35 hybrid | works | 2743 / 57.29 | 2752 / 59.67 | 1.00 / 0.96 | 728 / 57.64 | 2102 / 59.41 | -| 14 | Qwen3.5-9B-MTP-UD-Q5_K_XL | 6.87 | qwen35 hybrid | works (re-measured no pin; pp at 10 reps - the 5-rep run drifted, see notes) | 2698 / 56.40 | 2783 / 58.12 | 0.97 / 0.97 | 767 / 56.72 | 2098 / 57.86 | +| 12 | Qwen3.5-9B-IQ4_XS | 5.47 | qwen35 hybrid | works (re-measured no pin, ctx 262143); KHR re-measured on the word-stage kq tile (738 -> 2560) | 2700 / 62.94 | 2794 / 66.82 | 0.97 / 0.94 | 2560 / 62.70 | 2199 / 66.26 | +| 13 | Qwen3.5-9B-MTP-Q5_K_M | 6.64 | qwen35 hybrid | works; KHR re-measured on the word-stage kq tile (728 -> 2525) | 2743 / 57.29 | 2752 / 59.67 | 1.00 / 0.96 | 2525 / 57.33 | 2102 / 59.41 | +| 14 | Qwen3.5-9B-MTP-UD-Q5_K_XL | 6.87 | qwen35 hybrid | works (re-measured no pin; pp at 10 reps - the 5-rep run drifted, see notes); KHR re-measured on the word-stage kq tile (767 -> 1788 -> 2531): the 1788 was a feed gate, not the tile - the recurrent head decided its f16 feed per layer and this file's Q8_0 out plane sent the K-quant qkv/z to the sdot4 tile (see notes 14-KHR) | 2698 / 56.40 | 2783 / 58.12 | 0.97 / 0.97 | 2530.6 / 56.66 | 2098 / 57.86 | | 15 | Qwen3.5-9B-Q8_0 | 9.53 | qwen35 hybrid | works (re-measured no pin, ctx 174167) | 3227 / 43.82 | 2799 / 43.99 | 1.15 / 1.00 | 2140 / 43.58 | 1871 / 43.95 | | 16 | Qwen3.5-9B-MTP-Q8_0 | 9.79 | qwen35 hybrid | works | 3233 / 43.82 | 2797 / 44.03 | 1.16 / 1.00 | 2150 / 43.62 | 1873 / 44.00 | -| 17 | Qwen3.8-27B.i1-IQ3_S | 12.60 | qwen35 hybrid | works (re-measured no pin, ctx 30623; the pinned run read 752 / 24.95) | 802.9 / 27.51 | 807.9 / 24.67 | 0.99 / 1.12 | 238.9 / 27.39 | 650.6 / 24.65 | -| 18 | Qwen3.8-27B-UD-Q3_K_XL | 13.15 | qwen35 hybrid | works (re-measured no pin, ctx 24651; the 5-rep run caught the stall at 803.9 +-33.8 / 25.29 - the row carries the flat 20-rep profiled run, see notes) | 840.0 / 24.92 | 812.6 / 24.88 | 1.03 / 1.00 | 228.2 / 25.15 | 675.4 / 24.80 | -| 19 | Qwen3.8-27B-UD-IQ4_XS | 14.25 | qwen35 hybrid | works, no pin; KHR re-measured after the kq tile; the slab fix's re-measure is owed - with 1.2 GB held by other processes the plan's KV room fell under the 2048 minimum and the driver declined (`followup_vulkan.md` item 42) | 865.4 / 23.51 | 814.6 / 24.13 | 1.06 / 0.97 | 395.2 / 23.51 | 675.2 / 24.07 | +| 17 | Qwen3.8-27B.i1-IQ3_S | 12.60 | qwen35 hybrid | works (re-measured no pin, ctx 30623; the pinned run read 752 / 24.95); KHR re-measured on the word-stage kq tile under the 14000 MB pin (238.9 -> 736.7) | 802.9 / 27.51 | 807.9 / 24.67 | 0.99 / 1.12 | 736.7 / 27.35 | 650.6 / 24.65 | +| 18 | Qwen3.8-27B-UD-Q3_K_XL | 13.15 | qwen35 hybrid | works (re-measured no pin, ctx 24651; the 5-rep run caught the stall at 803.9 +-33.8 / 25.29 - the row carries the flat 20-rep profiled run, see notes); KHR re-measured on the word-stage kq tile under the 14000 MB pin (228.2 -> 729.2) | 840.0 / 24.92 | 812.6 / 24.88 | 1.03 / 1.00 | 729.2 / 25.04 | 675.4 / 24.80 | +| 19 | Qwen3.8-27B-UD-IQ4_XS | 14.25 | qwen35 hybrid | works, no pin (cm2 columns); KHR re-measured on the word-stage kq tile under `DASLLAMA_GPU_VRAM_MB=14000` - the desktop holds 1.1 GB, which leaves the no-pin plan's KV room under the 2048-position minimum (`followup_vulkan.md` item 42): 221 -> 395 -> 741 | 865.4 / 23.51 | 814.6 / 24.13 | 1.06 / 0.97 | 740.7 / 23.24 | 675.2 / 24.07 | | 20 | Qwen1.5-MoE-A2.7B-Chat.Q8_0 | 15.23 | qwen2moe | works (fixed: the per-op attention chain's bias arm + a 2048-wide kv cap); LAGS on the per-op MoE prefill structure and a span without a shared-expert arm (see notes; first run 390.2 / 29.31 = 0.11 / 0.56) | 506.1 / 39.04 | 3507 / 52.15 | 0.14 / 0.75 | 525.5 / 39.94 | 2185 / 51.84 | -| 21 | Qwen3-30B-A3B-Instruct-2507-Q4_K_M | 18.56 | qwen3moe | works on the per-op tier: experts of layers [13..48) resident, [0..13) streamed; llama.cpp -ngl 99 OOMs, its cells are the same-split offload (experts 0-12 on the CPU; -ngl 36 in the notes) | 749.8 / 66.45 | 612.5 / 42.24 | 1.22 / 1.57 | 499.9 / 64.39 | 588.6 / 42.49 | -| 22 | Qwen3-Coder-30B-A3B-Instruct-Q4_K_M | 18.56 | qwen3moe | | | | | | | +| 21 | Qwen3-30B-A3B-Instruct-2507-Q4_K_M | 18.56 | qwen3moe | works on the per-op tier: experts of layers [13..48) resident, [0..13) streamed; llama.cpp -ngl 99 OOMs, its cells are the same-split offload (experts 0-12 on the CPU; -ngl 36 in the notes); KHR re-measured on the word-stage kq tile (499.9 -> 712.0, the run's split [14..48) resident) | 749.8 / 66.45 | 612.5 / 42.24 | 1.22 / 1.57 | 712.0 / 64.53 | 588.6 / 42.49 | +| 22 | Qwen3-Coder-30B-A3B-Instruct-Q4_K_M | 18.56 | qwen3moe | works on the per-op tier like row 21 (see notes 22 and 24 for its cm2 and first KHR rows); KHR re-measured on the word-stage kq tile (494.3 -> 764.1 +-13.4, then 721.8 +-6.2 on the re-run after the reboot, the row's figure); the first run's decode read 33.61 against 63.60 before - the re-run's 64.23 beside row 21's 64.53 says that reading was the box, not the tile (the expert split was [14..48) resident both times) | 745.3 / 65.50 | 608.8 / 40.65 | 1.22 / 1.61 | 721.8 / 64.23 | 572.6 / 41.81 | | 23 | Qwen3.8-27B-Q4_K_M | 18.97 | qwen35 hybrid | does not fit 16 GB on either engine (llama.cpp OOM) | | | | | | -| 24 | Qwen3.6-35B-A3B-MTP-UD-Q4_K_M | 22.13 | qwen3moe hybrid | works on the per-op tier: deltanet triples of 30 layers, attention quads of 10, experts of [16..40) resident, [0..16) streamed; llama.cpp -ngl 99 OOMs, its cells = the same-split offload (experts 0-15 on the CPU; -ngl 24 in the notes) | 657.6 / 36.39 | 414.1 / 36.76 | 1.59 / 0.99 | 422.3 / 37.03 | 393.4 / 37.06 | +| 24 | Qwen3.6-35B-A3B-MTP-UD-Q4_K_M | 22.13 | qwen3moe hybrid | works on the per-op tier: deltanet triples of 30 layers, attention quads of 10, experts of [16..40) resident, [0..16) streamed; llama.cpp -ngl 99 OOMs, its cells = the same-split offload (experts 0-15 on the CPU; -ngl 24 in the notes); KHR re-measured on the word-stage kq tile (422.3 -> 665.4; that run's split [17..40) resident) | 657.6 / 36.39 | 414.1 / 36.76 | 1.59 / 0.99 | 665.4 / 36.86 | 393.4 / 37.06 | | 25 | Qwen3-Coder-30B-A3B-Instruct-Q8_0 | 32.48 | qwen3moe | works on the per-op tier: attention quads of 48 layers, experts of [29..48) resident, [0..29) streamed; llama.cpp cells = the same-split offload (experts 0-28 on the CPU; -ngl 20 read 201.0 / 17.38) | 279.1 / 30.64 | 210.5 / 22.27 | 1.33 / 1.38 | 276.8 / 29.82 | 202.7 / 22.65 | | 26 | Qwen3.6-35B-A3B-Q8_0 | 36.90 | qwen3moe hybrid | works on the per-op tier: 30 deltanet triples, 10 attention quads, experts of [27..40) resident, [0..27) streamed; llama.cpp cells = the same-split offload (-ngl 15 read 173.5 / 14.66); the KHR arm's prefill argmax is a near-tie flip (198 at 8.333 against 220 at 8.355), decode argmax and logits match | 221.4 / 28.61 | 177.8 / 25.84 | 1.25 / 1.11 | 239.0 / 28.90 | 176.3 / 26.16 | -| 27 | Qwen3-Coder-Next-Q4_K_M | 48.53 | qwen3next | works on the per-op tier: 132 dense planes, 48 shared experts, 36 deltanet triples, 12 attention quads resident, experts of [37..48) resident, [0..37) streamed; the FIRST run died out of host heap in the image mint (see notes); llama.cpp cells = the same-split offload (-ngl 13 read 115.4 / 13.88); both arms' prefill argmax are whitespace near-ties (198 vs 197), decode argmax matches | 381.3 / 27.76 | 118.1 / 22.14 | 3.23 / 1.25 | 155.8 / 27.62 | 116.4 / 22.50 | +| 27 | Qwen3-Coder-Next-Q4_K_M | 48.53 | qwen3next | works on the per-op tier: 132 dense planes, 48 shared experts, 36 deltanet triples, 12 attention quads resident, experts of [37..48) resident, [0..37) streamed; the FIRST run died out of host heap in the image mint (see notes); llama.cpp cells = the same-split offload (-ngl 13 read 115.4 / 13.88); both arms' prefill argmax are whitespace near-ties (198 vs 197), decode argmax matches; KHR re-measured on the word-stage kq tile (155.8 -> 319.9; that run's split [38..48) resident) | 381.3 / 27.76 | 118.1 / 22.14 | 3.23 / 1.25 | 319.9 / 27.05 | 116.4 / 22.50 | The 9B rows 12/14/15 and the 27B rows 17/18 were measured 2026-09-06 with the grid-format GEMV fix and the memory-priority flip already in; the 27B rows 17 and 18 ran under a VRAM pin below @@ -113,6 +113,20 @@ interfering GPU client, or the memory manager taking the idle gap. From row 20 o clocks, utilization and memory.used at 100 ms beside every run of ours, so the next occurrence is caught. Rows 14 and 18 carry their flat repeat runs. +### 14-KHR. The one KHR row under parity, and why +On the word-stage kq tile every K-quant row went past llama.cpp's KHR arm except this file: 1788 +against 2098 while its Q5_K_M sibling (row 13) read 2525. The per-role profile (`DASLLAMA_GPU_PROF=1`, +`vk_rdpf dn`) put the difference in two roles: the recurrent layers' qkv 72.9 ms and z 35.1 ms per +512-row window against 19.7 / 10.4 on the Q5_K_M file, the same Q6_K format and the same shape, and +the rates (11 TFLOP/s) were the sdot4 tile's. The cause was the feed gate: the recurrent head decided +its f16 feed per LAYER over the qkv, z and out planes together, and this file's out plane is Q8_0 +(unsloth's UD mixture; the Q5_K_M's is Q5_K), a format the KHR arm's f16 feed does not admit - so +every recurrent layer's K-quant qkv and z pair fell to the Q8_K quant route. The x feed (qkv, z) +and the o feed (out) are now decided apart, each by the planes that read it, as the attention and +FFN heads already were: qkv 19.7 ms, z 10.5, the row 2530.6 / 56.66 (1.21x / 0.98x). Under cm2 the +gate never showed (its feed admits q8). The witness is the hybrid parity file's mixed twin +(`Qwen3.5-0.8B-Q4_K_M-q8out.gguf`, `followup_vulkan.md` item 42). + ### 11. Qwen3.5-4B-Q8_0 First run: the resident driver declined at the device prepare - "KV mirror (8 x 262144 x 1024) exceeds maxStorageBufferRange 4294967295" - and the per-op rails served with the dense FFN of 32 @@ -199,4 +213,5 @@ per-op logger: GEMVs 37.7 vs 37.0 ms/token (gate+up faster than theirs, the beta ms slower - the long-rows-few-outputs GEMV shape), small ops equal; the residual add is fused into their down/out mat-vecs. Levers left: the beta/alpha GEMV shape, the fused residual add (~1% each). On the KHR arm prefill was 0.33x before the kq formats had a KHR-coopmat tile, 0.59x -with it, and the row above is the tile after its slab fix (`followup_vulkan.md` item 42). +with it, 0.73x after its slab fix, and 1.10x on the word-stage tile with f16 accumulators the row +above carries (`followup_vulkan.md` item 42). diff --git a/plans/shader_emitter_followups.md b/plans/shader_emitter_followups.md index 1dda56dd49..2753ac7654 100644 --- a/plans/shader_emitter_followups.md +++ b/plans/shader_emitter_followups.md @@ -35,21 +35,38 @@ the binding rules in `modules/REVIEW_SHADER_EMITTERS.md`. This file holds what i Done = the operators and `select` documented in `skills/daslang/`, the emitter heuristic gone, the census sites converted with the two measurements beside them. -2. **`for [unroll]` unrolls at emission; a fixed-array local with constant indices becomes +2. **`for [unroll_full]` unrolls at emission; a fixed-array local with constant indices becomes registers.** Found 2026-09-07 by the integer GEMM tile prototypes (`modules/dasLLAMA/harness/vk_gemm_probe.das -- mmqx`, `modules/dasLLAMA/followup_vulkan.md` - item 42): dasSpirv emits `for [unroll] (i in range(N))` as a loop carrying the `Unroll` loop - control and a `var acc : float[64]` local as a Function-storage `OpVariable` indexed by the - loop counter - a register block written as an array runs through local memory unless the - driver both unrolls and scalarizes, and the 5060 Ti's measured the same rate with the block - as named scalars, which says the shape, not the array, capped that kernel, but the array form - is what a 128-accumulator block (the reference exe's 4 x 32 register block) needs to be - writable at all. The plan: dasSpirv clones the body N times for a constant `range(N)` with the - counter bound to `OpConstant`, so every `arr[expr(i)]` chains a constant index (SROA-friendly in - every driver), and reports the unroll it performed in the same note channel item 1 gives the - branched operators; dasMetal needs nothing - MSL's `#pragma unroll` and the Metal compiler's - scalarization already do this, which is why the Metal GEMV twins carry `sumf : float[NR]` - arrays. Gate: a `tests/spirv` fixture pinning zero `OpLoopMerge` under an unrolled body and a - constant-index `OpAccessChain` per element, the kernel suite byte-identical elsewhere, and the - `mmqx` probe's ceiling twin re-measured with the block as an array. - Done = the fixture, the note, and the probe row. + item 42): a hinted loop leaves a `var acc : float[64]` local a Function-storage `OpVariable` + indexed by the loop counter unless the driver both unrolls and scalarizes, and a 128-accumulator + block - or a coopmat tile's sixteen accumulator fragments - is writable only as an array. + LANDED 2026-09-08 (`modules/dasSpirv/ARCHITECTURE_COOPMAT.md` section 3.6): `[unroll_full]` with + literal bounds emits the body once per copy with the induction variable an `OpConstant`, folds + the integer arithmetic on it so `acc[t * 16 + c]` chains a literal index, and refuses `break`, + `continue` and a runtime bound; `[unroll]` stays the driver's `Unroll` hint. The spelling is the + one the JIT (`llvm.loop.unroll.full`) and dasMetal (`#pragma clang loop unroll(full)`) already + lower, so a kernel source reads the same on every tier, and no shipped SPIR-V kernel changed (the + golden set is byte-identical). Fixture `ufor` in `tests/spirv` (zero `OpLoopMerge`, 23 + constant-index chains, the fold assertions) and two fail-closed fixtures. Still owed: the note + channel - the unroll's copy count and item 1's per-operator note share one channel once item 1 + lands (a `to_log` at `LOG_DEBUG` prints on every kernel compile, so nothing is reported today); + and the first kernel written on it, the KHR coopmat tile's accumulator block (item 42), whose + device cells are the runtime gate a fixture cannot be. + Done = the note, and the tile's cells green on the array form. + - Measured 2026-09-08 (moved here from `modules/dasSpirv/ARCHITECTURE.md`): the `Unroll` loop + control on `coopmatClamp`'s hand-emitted element walk is worth 34.2 TFLOP/s rolled against + 57.8 unrolled, on the cm2 l-tile min-kernel, RTX 5060 Ti, driver 610.74. + +3. **Add the missing fixtures to the `tests/spirv` census roster.** Found 2026-09-08 by the KHR + tile round: 111 `*_words` fixture defs against 87 `add_set` rows in `test_census.das`, so + about 25 fixtures' opcodes are outside the census sum, and an opcode nothing declares can be + emitted unnoticed. `tests/spirv/REVIEW.md` binds a diff that adds a fixture (ruled 2026-09-08: + the rule lands in the KHR tile PR, the backlog is the follow-up PR); the gap itself is the + `REVIEW.das` candidate - a gate that reads the fixture defs and the roster and fails on the + difference. + Same PR: the suite's skip line `spirv-val not found locally; skipping (CI enforces)` claims a + CI check no lane makes - the run is local by rule (`modules/dasSpirv/REVIEW.md`), so the feint + text says so instead. + Done = the unrostered fixtures rostered, the suite green, the gate in place so the roster + cannot drift again, and the skip line true. diff --git a/plans/vulkan_hybrid_ladder.md b/plans/vulkan_hybrid_ladder.md index cc15c230b3..ef31d99daf 100644 --- a/plans/vulkan_hybrid_ladder.md +++ b/plans/vulkan_hybrid_ladder.md @@ -169,7 +169,7 @@ rebuilt binary aged the sidecar; Boris ruled no re-mint until Vulkan is fully fu - Gate: `test_gpu_resident_hybrid` gains the K-quant fixture (Qwen3.5-0.8B-Q4_K_M, minted from the Q8_0 with llama-quantize), the declines test names the out-plane case, the 9B UD pair against a same-day llama-bench control, the profiled window's dn GEMM roles. - - Docs: 2.2j / 2.2v say the planes ride their file formats; the loader's tag comment follows. + - Docs: 2.2ad / 2.2v say the planes ride their file formats; the loader's tag comment follows. ## Measurement diff --git a/skills/internal/tests_in_repo.md b/skills/internal/tests_in_repo.md index 76e7271e0d..13232eaf1a 100644 --- a/skills/internal/tests_in_repo.md +++ b/skills/internal/tests_in_repo.md @@ -116,3 +116,22 @@ should keep the fixture in its own directory rather than reaching across the tre - All tests: `bin/Release/daslang.exe dastest/dastest.das -- --test tests/` Sweep the whole `tests` root for AOT/JIT validation, per the root-path caveat above. + +## The dasSpirv emitter suite (`tests/spirv`) + +`tests/spirv` is the SPIR-V emitter's own gate: every fixture compiles a shader and asserts on +the emitted words, `test_census.das` holds the emitted opcode set equal to the declared set in +both directions, and every blob is validated with `spirv-val` where the box resolves one - a cell +prints `spirv-val not found locally; skipping` otherwise, and no CI lane resolves it. The run +`modules/dasSpirv/REVIEW.md` owes, on a box with the Vulkan SDK on `PATH`: + +``` +bin/Release/daslang.exe dastest/dastest.das -- --test tests/spirv --cov-path spirv.lcov --isolated-mode +``` + +Green is: every cell passing, the census equal both ways, every blob `spirv-val`-clean, no +`GC APP LEAK`; `spirv.lcov` is the emitter's dispatch coverage. Two checks stand outside the suite. +A disassembly diff: dump an emitted module through `spirv_dis` (the emitter's symbolic +disassembler, `modules/dasSpirv/spirv/spirv_dis.das`) and diff it against the external +`spirv-dis` as ground truth. The real-driver gate: `modules/dasVulkan`'s integration suite under +lavapipe and on a local GPU, where the words are shown to compute. diff --git a/skills/review_md.md b/skills/review_md.md index cba87c90cd..7aad95221a 100644 --- a/skills/review_md.md +++ b/skills/review_md.md @@ -20,19 +20,22 @@ block, verbatim except for the module name and the architecture-doc path: A checklist whose folder has a follow-up ledger appends `` Planned work: ``. `` to the same line - `` Planned work: `` (sec. ). `` when the ledger is one section -of a larger document, so the pointer still finds it. A checklist whose folder's architecture doc has split into companions writes -the plural - `` Architecture docs: ``, ``. `` - listing every doc its rules cite. A checklist with routed companions - a shared concern file, subfolder -checklists, a skill reviewing a construct family - carries their routing lines in or right -after the opening, each keyed by KIND ("a `[test]` file, wherever the diff puts it, answers to -the `tests/` subfolder's checklist"), because the folder walk finds only files named -`REVIEW.md` - a companion is reachable solely through its routing line. An opening that -deviates from this block - or a checklist that restates contract text instead of pointing - is -a self-review finding, fixed like any other. +of a larger document, so the pointer still finds it. A checklist whose folder's architecture +doc has split into companions writes the plural - `` Architecture docs: ``, ``. `` - +listing every doc its rules cite. A checklist whose folder has no rationale document - a +folder of transient documents, each deleted when its work lands, so no rule there can cite a +reason - writes no `Architecture doc:` slot. A checklist with routed companions - a shared +concern file, subfolder checklists, a skill reviewing a construct family - carries their +routing lines in or right after the opening, each keyed by KIND ("a `[test]` file, wherever +the diff puts it, answers to the `tests/` subfolder's checklist"), because the folder walk +finds only files named `REVIEW.md` - a companion is reachable solely through its routing +line. An opening that deviates from this block - or a checklist that restates contract text +instead of pointing - is a self-review finding, fixed like any other. `` is the module's own design document - its `ARCHITECTURE.md` when it has one, otherwise its `CLAUDE.md`; a `README.md` that carries the module's charter and mechanisms -fills the slot too (`site/README.md` is the ruled precedent). Name it concretely; a module -with no rationale home needs one before its rules can cite a reason. +fills the slot too (`site/README.md` is one). Name it concretely; a module with no rationale +home needs one before its rules can cite a reason. (repo-only) ## The executable half - REVIEW.das @@ -55,6 +58,7 @@ review) is the REVIEW audit row of `skills/internal/make_pr.md`; the tree-wide w - A separable concern gets its own file, routed to from the opening - `modules/REVIEW_SHADER_EMITTERS.md` and `modules/dasLLAMA/REVIEW_GPU.md` are the pattern. + (repo-only) - The `placement-auditor` agent checks a placement block's file list against the folder's actual contents. - Test rules add the specifics `REVIEW_COMMON.md`'s "New functionality ships with tests" @@ -134,4 +138,4 @@ questions, not its limits - that file is the law. Then apply the self-review rul and run the followability classes above over the touched rules. The `dragon` agent runs this audit on any modified rule document; the `placement-auditor` agent audits a placement block against its folder. The canonical conforming set is `modules/dasLLAMA/REVIEW.md` and its -routed companions. +routed companions. (repo-only) diff --git a/tests/spirv/REVIEW.md b/tests/spirv/REVIEW.md index 5fa128e4a3..f95637e31c 100644 --- a/tests/spirv/REVIEW.md +++ b/tests/spirv/REVIEW.md @@ -1,17 +1,25 @@ # tests/spirv Code Review Checklist **Read `REVIEW_COMMON.md` (repo root) first - its contract binds this checklist.** Architecture -doc: `modules/dasSpirv/ARCHITECTURE.md`. Shared emitter rules: `modules/REVIEW_SHADER_EMITTERS.md` -- apply that list with this one. +docs: `modules/dasSpirv/ARCHITECTURE.md`, `modules/dasSpirv/ARCHITECTURE_COOPMAT.md`. Shared +emitter rules: `modules/REVIEW_SHADER_EMITTERS.md` - apply that list with this one. -**Weakening `test_census.das` is a defect** - it holds every fixture opcode declared and every -declared opcode emitted. +**A diff that adds a `*_words` fixture adds it to the roster in `test_census.das` - the +per-fixture `add_set` list the census sums.** A fixture outside the roster is outside the +census, so an opcode nothing declares can be emitted unnoticed. + +**Removing a fixture from the `test_census.das` roster, or weakening either direction of the +census assert - every emitted opcode is in the declared set, every declared opcode is emitted - +is a defect.** **A diff that adds or changes a fixture asserting a claim `validate_spirv` would check also asserts that claim on the emitted words.** `validate_spirv` skips where the tool is absent or predates an extension, so a spirv-val-only assertion can check nothing. **A fixture cell that validates its words asserts the module version on `words[1]` in the same -cell, and passes a `target_env` - directly or through the file's `validate` wrapper - that -admits that version: `vulkan1.2` for 1.4 and 1.5, `vulkan1.3` for 1.6.** The default, -`vulkan1.1`, admits SPIR-V 1.3 at most. +cell.** `validate_spirv` checks the words against the target env it was given, not against the +version the fixture expects. + +**A fixture cell that validates its words passes a `target_env` - directly or through the +file's own validate wrapper - that admits the version it asserts: `vulkan1.1` admits SPIR-V 1.3 +and below, `vulkan1.2` up to 1.5, `vulkan1.3` up to 1.6.** `vulkan1.1` is the default. diff --git a/tests/spirv/_fail_closed/_fc_unroll_break.das b/tests/spirv/_fail_closed/_fc_unroll_break.das new file mode 100644 index 0000000000..67078ca7c2 --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_break.das @@ -0,0 +1,21 @@ +// Fail-closed fixture: break inside a [unroll_full] body. The unrolled copies form no loop construct, so a +// break has no merge block to branch to; it is refused rather than lowered into an invalid structured CFG. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_break_spv")] +def fc_unroll_break { + let i = gl_GlobalInvocationID.x + var acc = 0u + for [unroll_full] (j in range(8)) { + if (j == 4) break + acc += uint(j) + } + data[i] = acc +} diff --git a/tests/spirv/_fail_closed/_fc_unroll_continue.das b/tests/spirv/_fail_closed/_fc_unroll_continue.das new file mode 100644 index 0000000000..3fa556c0a7 --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_continue.das @@ -0,0 +1,21 @@ +// Fail-closed fixture: continue inside a [unroll_full] body. The unrolled copies form no loop construct, so a +// continue has no continue block to branch to; it is refused rather than lowered into an invalid structured CFG. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_continue_spv")] +def fc_unroll_continue { + let i = gl_GlobalInvocationID.x + var acc = 0u + for [unroll_full] (j in range(8)) { + if (j == 4) continue + acc += uint(j) + } + data[i] = acc +} diff --git a/tests/spirv/_fail_closed/_fc_unroll_hint_pair.das b/tests/spirv/_fail_closed/_fc_unroll_hint_pair.das new file mode 100644 index 0000000000..705c67dbbe --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_hint_pair.das @@ -0,0 +1,20 @@ +// Fail-closed fixture: a driver loop control beside [unroll_full]. The unrolled copies emit no loop construct +// to carry [dont_unroll], so the pair is refused rather than the second hint dropped without a word. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_hint_pair_spv")] +def fc_unroll_hint_pair { + let i = gl_GlobalInvocationID.x + var acc = 0u + for [unroll_full, dont_unroll] (j in range(8)) { + acc += uint(j) + } + data[i] = acc +} diff --git a/tests/spirv/_fail_closed/_fc_unroll_return.das b/tests/spirv/_fail_closed/_fc_unroll_return.das new file mode 100644 index 0000000000..6c52295bb7 --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_return.das @@ -0,0 +1,21 @@ +// Fail-closed fixture: return inside a [unroll_full] body. A return in one copy leaves the copies after it +// unreachable, so the emitter refuses it instead of emitting instructions after a block terminator. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_return_spv")] +def fc_unroll_return { + let i = gl_GlobalInvocationID.x + var acc = 0u + for [unroll_full] (j in range(8)) { + acc += uint(j) + if (acc > 6u) return + } + data[i] = acc +} diff --git a/tests/spirv/_fail_closed/_fc_unroll_runtime.das b/tests/spirv/_fail_closed/_fc_unroll_runtime.das new file mode 100644 index 0000000000..0ff991d365 --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_runtime.das @@ -0,0 +1,21 @@ +// Fail-closed fixture: [unroll_full] over a runtime bound. The emitter unrolls the loop at emission, so both +// bounds must be literals; a runtime bound is refused rather than silently downgraded to the driver's hint. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_runtime_spv")] +def fc_unroll_runtime { + let i = gl_GlobalInvocationID.x + let n = data[i] + var acc = 0u + for [unroll_full] (j in urange(0u, n)) { + acc += j + } + data[i] = acc +} diff --git a/tests/spirv/_fail_closed/_fc_unroll_while.das b/tests/spirv/_fail_closed/_fc_unroll_while.das new file mode 100644 index 0000000000..90053e5377 --- /dev/null +++ b/tests/spirv/_fail_closed/_fc_unroll_while.das @@ -0,0 +1,22 @@ +// Fail-closed fixture: [unroll_full] on a while loop. Only a `for` over literal bounds unrolls at emission; +// a while loop has no copy count, so the hint is refused and the driver's [unroll] named instead. +expect 50501 + +options gen2 + +require spirv/spirv_shader +require spirv/spirv_builtins + +var @ssbo @binding = 0 data : array + +[compute_shader(local_size_x=64, name="fc_unroll_while_spv")] +def fc_unroll_while { + let i = gl_GlobalInvocationID.x + var acc = 0u + var k = 0u + while [unroll_full] (k < 4u) { + acc += k + k++ + } + data[i] = acc +} diff --git a/tests/spirv/_gen_golden.das b/tests/spirv/_gen_golden.das index 2b675ddc37..9958f0335c 100644 --- a/tests/spirv/_gen_golden.das +++ b/tests/spirv/_gen_golden.das @@ -40,6 +40,7 @@ def main { // nolint:STYLE038 — one emit per golden fixture: a flat list, on emit_golden(dir, "echain", echain_words()) emit_golden(dir, "eret", eret_words()) emit_golden(dir, "rfor", rfor_words()) + emit_golden(dir, "ufor", ufor_words()) emit_golden(dir, "tri_vert", tri_vert_words()) emit_golden(dir, "tri_frag", tri_frag_words()) emit_golden(dir, "point_size", point_size_words()) diff --git a/tests/spirv/_golden/ufor.txt b/tests/spirv/_golden/ufor.txt new file mode 100644 index 0000000000..82e589f700 --- /dev/null +++ b/tests/spirv/_golden/ufor.txt @@ -0,0 +1,174 @@ +; SPIR-V module: 723 words, id-bound 0xb8 +OpCapability 0x1 +OpMemoryModel 0x0 0x1 +OpEntryPoint 0x5 0xc 0x6e69616d 0x0 0x9 +OpExecutionMode 0xc 0x11 0x40 0x1 0x1 +OpDecorate 0x2 0x6 0x4 +OpDecorate 0x3 0x2 +OpMemberDecorate 0x3 0x0 0x23 0x0 +OpDecorate 0x5 0x22 0x0 +OpDecorate 0x5 0x21 0x0 +OpDecorate 0x9 0xb 0x1c +OpTypeFloat 0x1 0x20 +OpTypeRuntimeArray 0x2 0x1 +OpTypeStruct 0x3 0x2 +OpTypePointer 0x4 0xc 0x3 +OpVariable 0x4 0x5 0xc +OpTypeInt 0x6 0x20 0x0 +OpTypeVector 0x7 0x6 0x3 +OpTypePointer 0x8 0x1 0x7 +OpVariable 0x8 0x9 0x1 +OpTypeVoid 0xa +OpTypeFunction 0xb 0xa +OpConstant 0x6 0xe 0x8 +OpTypeArray 0xf 0x1 0xe +OpTypePointer 0x10 0x7 0xf +OpTypePointer 0x12 0x7 0x1 +OpTypePointer 0x14 0x1 0x6 +OpConstant 0x6 0x16 0x0 +OpConstantNull 0xf 0x18 +OpTypeInt 0x19 0x20 0x1 +OpConstant 0x19 0x1a 0x0 +OpConstant 0x19 0x1d 0x1 +OpConstant 0x19 0x20 0x2 +OpConstant 0x19 0x23 0x3 +OpConstant 0x19 0x26 0x4 +OpConstant 0x19 0x29 0x5 +OpConstant 0x19 0x2c 0x6 +OpConstant 0x19 0x2f 0x7 +OpTypePointer 0x37 0xc 0x1 +OpConstant 0x6 0x3c 0x1 +OpConstant 0x6 0x43 0x2 +OpConstant 0x6 0x4a 0x3 +OpConstant 0x6 0x5d 0x9 +OpConstant 0x6 0x64 0xa +OpConstant 0x6 0x6b 0xb +OpConstant 0x1 0x95 0x0 +OpFunction 0xa 0xc 0x0 0xb +OpLabel 0xd +OpVariable 0x10 0x11 0x7 +OpVariable 0x12 0x13 0x7 +OpAccessChain 0x14 0x15 0x9 0x16 +OpLoad 0x6 0x17 0x15 +OpStore 0x11 0x18 +OpAccessChain 0x12 0x1b 0x11 0x1a +OpConvertSToF 0x1 0x1c 0x1a +OpStore 0x1b 0x1c +OpAccessChain 0x12 0x1e 0x11 0x1d +OpConvertSToF 0x1 0x1f 0x1d +OpStore 0x1e 0x1f +OpAccessChain 0x12 0x21 0x11 0x20 +OpConvertSToF 0x1 0x22 0x20 +OpStore 0x21 0x22 +OpAccessChain 0x12 0x24 0x11 0x23 +OpConvertSToF 0x1 0x25 0x23 +OpStore 0x24 0x25 +OpAccessChain 0x12 0x27 0x11 0x26 +OpConvertSToF 0x1 0x28 0x26 +OpStore 0x27 0x28 +OpAccessChain 0x12 0x2a 0x11 0x29 +OpConvertSToF 0x1 0x2b 0x29 +OpStore 0x2a 0x2b +OpAccessChain 0x12 0x2d 0x11 0x2c +OpConvertSToF 0x1 0x2e 0x2c +OpStore 0x2d 0x2e +OpAccessChain 0x12 0x30 0x11 0x2f +OpConvertSToF 0x1 0x31 0x2f +OpStore 0x30 0x31 +OpAccessChain 0x12 0x34 0x11 0x1a +OpIAdd 0x6 0x35 0x17 0x16 +OpAccessChain 0x37 0x36 0x5 0x16 0x35 +OpLoad 0x1 0x38 0x36 +OpLoad 0x1 0x39 0x34 +OpFAdd 0x1 0x3a 0x39 0x38 +OpStore 0x34 0x3a +OpAccessChain 0x12 0x3b 0x11 0x1d +OpIAdd 0x6 0x3d 0x17 0x3c +OpAccessChain 0x37 0x3e 0x5 0x16 0x3d +OpLoad 0x1 0x3f 0x3e +OpLoad 0x1 0x40 0x3b +OpFAdd 0x1 0x41 0x40 0x3f +OpStore 0x3b 0x41 +OpAccessChain 0x12 0x42 0x11 0x20 +OpIAdd 0x6 0x44 0x17 0x43 +OpAccessChain 0x37 0x45 0x5 0x16 0x44 +OpLoad 0x1 0x46 0x45 +OpLoad 0x1 0x47 0x42 +OpFAdd 0x1 0x48 0x47 0x46 +OpStore 0x42 0x48 +OpAccessChain 0x12 0x49 0x11 0x23 +OpIAdd 0x6 0x4b 0x17 0x4a +OpAccessChain 0x37 0x4c 0x5 0x16 0x4b +OpLoad 0x1 0x4d 0x4c +OpLoad 0x1 0x4e 0x49 +OpFAdd 0x1 0x4f 0x4e 0x4d +OpStore 0x49 0x4f +OpAccessChain 0x12 0x56 0x11 0x26 +OpIAdd 0x6 0x57 0x17 0xe +OpAccessChain 0x37 0x58 0x5 0x16 0x57 +OpLoad 0x1 0x59 0x58 +OpLoad 0x1 0x5a 0x56 +OpFAdd 0x1 0x5b 0x5a 0x59 +OpStore 0x56 0x5b +OpAccessChain 0x12 0x5c 0x11 0x29 +OpIAdd 0x6 0x5e 0x17 0x5d +OpAccessChain 0x37 0x5f 0x5 0x16 0x5e +OpLoad 0x1 0x60 0x5f +OpLoad 0x1 0x61 0x5c +OpFAdd 0x1 0x62 0x61 0x60 +OpStore 0x5c 0x62 +OpAccessChain 0x12 0x63 0x11 0x2c +OpIAdd 0x6 0x65 0x17 0x64 +OpAccessChain 0x37 0x66 0x5 0x16 0x65 +OpLoad 0x1 0x67 0x66 +OpLoad 0x1 0x68 0x63 +OpFAdd 0x1 0x69 0x68 0x67 +OpStore 0x63 0x69 +OpAccessChain 0x12 0x6a 0x11 0x2f +OpIAdd 0x6 0x6c 0x17 0x6b +OpAccessChain 0x37 0x6d 0x5 0x16 0x6c +OpLoad 0x1 0x6e 0x6d +OpLoad 0x1 0x6f 0x6a +OpFAdd 0x1 0x70 0x6f 0x6e +OpStore 0x6a 0x70 +OpStore 0x13 0x95 +OpAccessChain 0x12 0x96 0x11 0x1d +OpLoad 0x1 0x97 0x96 +OpLoad 0x1 0x98 0x13 +OpFAdd 0x1 0x99 0x98 0x97 +OpStore 0x13 0x99 +OpAccessChain 0x12 0x9a 0x11 0x20 +OpLoad 0x1 0x9b 0x9a +OpLoad 0x1 0x9c 0x13 +OpFAdd 0x1 0x9d 0x9c 0x9b +OpStore 0x13 0x9d +OpAccessChain 0x12 0x9e 0x11 0x23 +OpLoad 0x1 0x9f 0x9e +OpLoad 0x1 0xa0 0x13 +OpFAdd 0x1 0xa1 0xa0 0x9f +OpStore 0x13 0xa1 +OpAccessChain 0x12 0xa2 0x11 0x26 +OpLoad 0x1 0xa3 0xa2 +OpLoad 0x1 0xa4 0x13 +OpFAdd 0x1 0xa5 0xa4 0xa3 +OpStore 0x13 0xa5 +OpAccessChain 0x12 0xa6 0x11 0x29 +OpLoad 0x1 0xa7 0xa6 +OpLoad 0x1 0xa8 0x13 +OpFAdd 0x1 0xa9 0xa8 0xa7 +OpStore 0x13 0xa9 +OpAccessChain 0x12 0xaa 0x11 0x2c +OpLoad 0x1 0xab 0xaa +OpLoad 0x1 0xac 0x13 +OpFAdd 0x1 0xad 0xac 0xab +OpStore 0x13 0xad +OpAccessChain 0x12 0xae 0x11 0x2f +OpLoad 0x1 0xaf 0xae +OpLoad 0x1 0xb0 0x13 +OpFAdd 0x1 0xb1 0xb0 0xaf +OpStore 0x13 0xb1 +OpAccessChain 0x37 0xb6 0x5 0x16 0x17 +OpLoad 0x1 0xb7 0x13 +OpStore 0xb6 0xb7 +OpReturn +OpFunctionEnd diff --git a/tests/spirv/_spirv_common.das b/tests/spirv/_spirv_common.das index 70812b5b0c..8547558f06 100644 --- a/tests/spirv/_spirv_common.das +++ b/tests/spirv/_spirv_common.das @@ -12,6 +12,7 @@ module _spirv_common shared public require spirv/spirv_shader require spirv/spirv_grammar +require spirv/spirv_dis require spirv/spirv_builtins public require daslib/fio require math @@ -297,6 +298,58 @@ def public rfor_words : array { return clone_to_move(rfor_spv) } +//! three shapes, three folds: the bare induction constant, a nested pair through a `uint()` re-type, a non-zero lower bound. +[compute_shader(local_size_x=64, name="ufor_spv"), marker(no_coverage)] +def ufor { + let gi = gl_GlobalInvocationID.x + var acc : float[8] + for [unroll_full] (i in range(8)) { + acc[i] = float(i) + } + for [unroll_full] (t in range(2)) { + for [unroll_full] (c in range(4)) { + acc[t * 4 + c] += fdata[gi + (uint(t) * 8u + uint(c))] + } + } + var s = 0.0 + for [unroll_full] (i in range(1, 8)) { + s += acc[i] + } + fdata[gi] = s +} + +def public ufor_words : array { + return clone_to_move(ufor_spv) +} + +let private UFOLD_N = 4 //! a named module constant as the bound: the lint compile hands it over unfolded + +//! every integer fold the unroll carries, over one induction variable: unary - and ~, the binary - / % << >> & | ^; +//! then the bound shapes a literal range() cannot spell: a literal urange, a zero-trip range and a one-trip range +[compute_shader(local_size_x=64, name="ufold_spv"), marker(no_coverage)] +def ufold { + let gi = gl_GlobalInvocationID.x + var s = 0.0 + for [unroll_full] (i in range(UFOLD_N)) { + let u = uint(i) + s += float(-i) + float(~u) + float(i - 1) + float(i / 2) + float(i % 3) + float(i << 1) + float(i >> 1) + float(i & 1) + float(i | 2) + float(i ^ 1) + } + for [unroll_full] (k in urange(2u, 5u)) { + s += float(k) + } + for [unroll_full] (_z in range(0)) { + s += 100.0 + } + for [unroll_full] (_o in range(1)) { + s++ + } + fdata[gi] = s +} + +def public ufold_words : array { + return clone_to_move(ufold_spv) +} + // ===== Phase-3 vertex/fragment fixtures: stage I/O (@in/@out + Location), rasterizer builtins // (gl_Position/gl_VertexIndex/gl_InstanceIndex/gl_FragCoord), vector constructors // (OpCompositeConstruct), dot (OpDot), and GLSL.std.450 math (OpExtInst). Each emitted blob is @@ -2305,6 +2358,56 @@ def public coopmat_f16acc_words : array { return clone_to_move(coopmat_f16acc_spv) } +var @ssbo @binding = 0 cma_a : array +var @ssbo @binding = 1 cma_b : array +var @ssbo @binding = 2 cma_c : array + +[compute_shader(local_size_x=32, name="coopmat_arr_spv"), marker(no_coverage)] +def coopmat_arr_kernel { + var a : coopmatA_f16_16x16 + var b : coopmatB_f16_16x16 + var acc : coopmatAcc_f16_16x16[4] + coopmatLoad(a, cma_a, 0, 16, 0) + for [unroll_full] (c in range(4)) { + coopmatLoad(b, cma_b, c * 256, 16, 1) + acc[c] = coopmatMulAdd(a, b, acc[c]) + } + var accw : coopmatAcc_f32_16x16 + for [unroll_full] (c in range(4)) { + coopmatConvert(accw, acc[c]) + coopmatStore(accw, cma_c, c * 256, 16, 1) + } +} + +def public coopmat_arr_words : array { + return clone_to_move(coopmat_arr_spv) +} + +def public count_function_storage_chains(words : array) : tuple { + var consts : table + var fvars : table + foreach_inst(words) $(opcode : uint; w : array; s, n : int) { + if (opcode == uint(SpvOp.Constant) && n >= 2) { + consts |> insert(w[s + 1]) + } elif (opcode == uint(SpvOp.Variable) && n >= 3 && w[s + 2] == uint(SpvStorageClass.Function)) { + fvars |> insert(w[s + 1]) + } + } + var chains = 0 + var dynamic_indices = 0 + foreach_inst(words) $(opcode : uint; w : array; s, n : int) { + if (opcode == uint(SpvOp.AccessChain) && n >= 4 && key_exists(fvars, w[s + 2])) { + chains++ + for (i in range(3, n)) { + if (!key_exists(consts, w[s + i])) { + dynamic_indices++ + } + } + } + } + return (chains = chains, dynamic_indices = dynamic_indices) +} + // ===== packed integer dot product: sdot4 -> OpSDot with the PackedVectorFormat4x8Bit literal. // The DotProduct / DotProductInput4x8BitPacked caps + SPV_KHR_integer_dot_product ride the // declared Capability/Extension opcodes. ===== diff --git a/tests/spirv/test_census.das b/tests/spirv/test_census.das index e1064e2964..fc82d60cf1 100644 --- a/tests/spirv/test_census.das +++ b/tests/spirv/test_census.das @@ -41,6 +41,9 @@ def test_opcode_census(t : T?) { // nolint:STYLE038 — a flat one-add_set-per-f add_set(present, echain_words()) add_set(present, eret_words()) add_set(present, rfor_words()) + add_set(present, ufor_words()) + add_set(present, ufold_words()) + add_set(present, coopmat_arr_words()) add_set(present, tri_vert_words()) add_set(present, tri_frag_words()) add_set(present, point_size_words()) diff --git a/tests/spirv/test_coopmat.das b/tests/spirv/test_coopmat.das index ace4b03ec8..c39fac3491 100644 --- a/tests/spirv/test_coopmat.das +++ b/tests/spirv/test_coopmat.das @@ -41,7 +41,7 @@ def test_coopmat(t : T?) { // nolint:STYLE038 - flat list of opcode/capability a if (r.ran) { t |> success(r.ok, "spirv-val (vulkan1.3): {r.msg}") } else { - feint("spirv-val not found locally; skipping (CI enforces)\n") + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") } delete words } @@ -64,7 +64,7 @@ def test_coopmat(t : T?) { // nolint:STYLE038 - flat list of opcode/capability a if (r.ran) { t |> success(r.ok, "spirv-val (vulkan1.3): {r.msg}") } else { - feint("spirv-val not found locally; skipping (CI enforces)\n") + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") } delete words } @@ -82,7 +82,7 @@ def test_coopmat(t : T?) { // nolint:STYLE038 - flat list of opcode/capability a if (r.ran) { t |> success(r.ok, "spirv-val (vulkan1.3): {r.msg}") } else { - feint("spirv-val not found locally; skipping (CI enforces)\n") + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") } delete words } @@ -96,7 +96,26 @@ def test_coopmat(t : T?) { // nolint:STYLE038 - flat list of opcode/capability a if (r.ran) { t |> success(r.ok, "spirv-val (vulkan1.3): {r.msg}") } else { - feint("spirv-val not found locally; skipping (CI enforces)\n") + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") + } + delete words + } + t |> run("an ARRAY of accumulator tiles under [unroll_full]: one OpTypeArray, every element chain a constant") <| @(t : T?) { + var words <- coopmat_arr_words() + t |> equal(count_op(words, SpvOp.TypeArray), 1, "one OpTypeArray (the tile array; the ssbos are runtime arrays)") + t |> equal(count_op(words, SpvOp.LoopMerge), 0, "no loop construct survives the unroll") + t |> equal(count_op(words, SpvOp.CooperativeMatrixMulAddKHR), 4, "four OpCooperativeMatrixMulAddKHR, one per copy") + t |> equal(count_op(words, SpvOp.FConvert), 4, "four OpFConvert widenings") + t |> equal(count_op(words, SpvOp.CooperativeMatrixStoreKHR), 4, "four stores") + let fc = count_function_storage_chains(words) + t |> equal(fc.chains, 12, "twelve element chains into the tile array (read, write and convert per copy)") + t |> equal(fc.dynamic_indices, 0, "every chain into the tile array indexes with an OpConstant") + t |> success(length(words) >= 2 && words[1] == SPV_VERSION_1_6, "SPIR-V header version 1.6") + let r = validate_spirv(words, "vulkan1.3") + if (r.ran) { + t |> success(r.ok, "spirv-val (vulkan1.3): {r.msg}") + } else { + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") } delete words } diff --git a/tests/spirv/test_fail_closed.das b/tests/spirv/test_fail_closed.das index be60ebdcea..a538bc99b5 100644 --- a/tests/spirv/test_fail_closed.das +++ b/tests/spirv/test_fail_closed.das @@ -41,6 +41,21 @@ def private check_rejects(t : T?; fixture, needle : string) { } } +//! the refusal a fixture raises in every unrolled copy reaches the issues ONCE: the unroll drops a copy's repeat of an earlier copy's error +def private check_rejects_once(t : T?; fixture, needle : string) { + let r = compile_issues("{get_das_root()}/tests/spirv/_fail_closed/{fixture}.das") + var hits = 0 + var at = find(r.issues, needle) + while (at >= 0) { + hits++ + at = find(r.issues, needle, at + 1) + } + t |> equal(hits, 1, "{fixture}: '{needle}' is reported once, not once per unrolled copy") + if (hits != 1) { + to_log(LOG_ERROR, "FAIL-CLOSED {fixture}: {hits} hits, issues=\n{r.issues}\n") + } +} + [test] def test_fail_closed_rejections(t : T?) { // nolint:STYLE038 — flat one-check-per-fixture list t |> run("emitter rejects shader-illegal constructs cleanly") <| @(t : T?) { @@ -54,6 +69,13 @@ def test_fail_closed_rejections(t : T?) { // nolint:STYLE038 — flat one-chec check_rejects(t, "_fc_localtype", "type tInt64 is not supported") check_rejects(t, "_fc_stmt", "memzero is not supported in SPIR-V shaders") check_rejects(t, "_fc_deadcode", "unreachable code after a block terminator") + check_rejects(t, "_fc_unroll_runtime", "the range must have literal bounds") + check_rejects(t, "_fc_unroll_break", "break inside a `for [unroll_full]` body") + check_rejects_once(t, "_fc_unroll_break", "break inside a `for [unroll_full]` body") + check_rejects(t, "_fc_unroll_continue", "continue inside a `for [unroll_full]` body") + check_rejects(t, "_fc_unroll_return", "return inside a `for [unroll_full]` body") + check_rejects(t, "_fc_unroll_while", "loop hint [unroll_full] applies to a `for (i in range(lo, hi))`") + check_rejects(t, "_fc_unroll_hint_pair", "loop hint [dont_unroll] beside [unroll_full]") // a write-swizzle chains a component pointer off its base, so the base needs an OpVariable; a // swizzle of a swizzle has only a loaded value, and there is nothing to store through check_rejects(t, "_fc_wswz", "write-swizzle target must be addressable") diff --git a/tests/spirv/test_golden.das b/tests/spirv/test_golden.das index d447b3edfb..67e08758e2 100644 --- a/tests/spirv/test_golden.das +++ b/tests/spirv/test_golden.das @@ -121,6 +121,7 @@ def test_golden_disassembly(t : T?) { // nolint:STYLE038 - flat list of golden f check(t, "echain", echain_words()) check(t, "eret", eret_words()) check(t, "rfor", rfor_words()) + check(t, "ufor", ufor_words()) // resource / composite fixtures: ids legitimately renumber (call/composite result allocated // before operands in the hand-walker, after them in the post-order visitor) -> id-isomorphic. check_iso(t, "tri_vert", tri_vert_words()) diff --git a/tests/spirv/test_loops.das b/tests/spirv/test_loops.das index dd4bc72d98..99e5edc12c 100644 --- a/tests/spirv/test_loops.das +++ b/tests/spirv/test_loops.das @@ -22,7 +22,7 @@ def private validate(t : T?; words : array; lbl : string) { if (r.ran) { t |> success(r.ok, "{lbl}: spirv-val: {r.msg}") } else { - feint("spirv-val not found locally; skipping (CI enforces)\n") + feint("spirv-val not found locally; skipping - run on a box with the Vulkan SDK on PATH\n") } } @@ -91,6 +91,34 @@ def test_early_return_in_loop(t : T?) { } } +[test] +def test_unroll_full(t : T?) { + t |> run("for [unroll_full] with literal bounds: emitted as copies, indices folded to constants") <| @(t : T?) { + var words <- ufor_words() + t |> equal(count_op(words, SpvOp.LoopMerge), 0, "ufor: no loop construct survives the unroll") + t |> equal(count_op(words, SpvOp.IMul), 0, "ufor: every index multiply folded at emission") + t |> equal(count_op(words, SpvOp.IAdd), 8, "ufor: one add per copy takes the invocation id; the index adds folded") + let fc = count_function_storage_chains(words) + t |> equal(fc.chains, 23, "ufor: 8 + 8 + 7 element chains into the register block") + t |> equal(fc.dynamic_indices, 0, "ufor: every chain into the block indexes with an OpConstant") + t |> success(length(words) >= 2 && words[1] == SPV_VERSION_1_3, "ufor: SPIR-V header version stays 1.3") + validate(t, words, "ufor") + delete words + } + t |> run("for [unroll_full] over a module constant: every integer fold lands, no integer op survives") <| @(t : T?) { + var words <- ufold_words() + t |> equal(count_op(words, SpvOp.LoopMerge), 0, "ufold: the named-constant bound unrolls like a literal") + t |> equal(count_op(words, SpvOp.FAdd), 44, "ufold: 4 copies x 10 terms over the named-constant bound, 3 over urange(2u, 5u), 0 over range(0), 1 over range(1)") + for (op in [SpvOp.SNegate, SpvOp.Not, SpvOp.ISub, SpvOp.SDiv, SpvOp.SRem, SpvOp.ShiftLeftLogical, + SpvOp.ShiftRightArithmetic, SpvOp.BitwiseAnd, SpvOp.BitwiseOr, SpvOp.BitwiseXor, SpvOp.IMul, SpvOp.IAdd]) { + t |> equal(count_op(words, op), 0, "ufold: no {op} survives - every integer op on the induction constant folds at emission") + } + t |> success(length(words) >= 2 && words[1] == SPV_VERSION_1_3, "ufold: SPIR-V header version stays 1.3") + validate(t, words, "ufold") + delete words + } +} + [test] def test_runtime_for_and_bool_local(t : T?) { t |> run("runtime urange(0u, n) bound + bool Function OpVariable") <| @(t : T?) { diff --git a/tutorials/integration/cpp/class_adapters_module.das.inc b/tutorials/integration/cpp/class_adapters_module.das.inc index 908c829486..b79fa630fd 100644 --- a/tutorials/integration/cpp/class_adapters_module.das.inc +++ b/tutorials/integration/cpp/class_adapters_module.das.inc @@ -1,53 +1,53 @@ -static unsigned char class_adapters_module_das[] = { -0x6f,0x70,0x74,0x69,0x6f,0x6e,0x73,0x20, -0x67,0x65,0x6e,0x32,0x0a, -0x6f,0x70,0x74,0x69,0x6f,0x6e,0x73,0x20, -0x72,0x65,0x6d,0x6f,0x76,0x65,0x5f,0x75, -0x6e,0x75,0x73,0x65,0x64,0x5f,0x73,0x79, -0x6d,0x62,0x6f,0x6c,0x73,0x20,0x3d,0x20, -0x66,0x61,0x6c,0x73,0x65,0x0a, -0x0a, -0x2f,0x2f,0x20,0x41,0x62,0x73,0x74,0x72, -0x61,0x63,0x74,0x20,0x62,0x61,0x73,0x65, -0x20,0x63,0x6c,0x61,0x73,0x73,0x20,0xe2, -0x80,0x94,0x20,0x64,0x61,0x73,0x6c,0x61, -0x6e,0x67,0x20,0x73,0x69,0x64,0x65,0x2e, -0x0a, -0x2f,0x2f,0x20,0x43,0x2b,0x2b,0x20,0x77, -0x69,0x6c,0x6c,0x20,0x69,0x6e,0x63,0x6c, -0x75,0x64,0x65,0x20,0x74,0x68,0x69,0x73, -0x20,0x76,0x69,0x61,0x20,0x63,0x6f,0x6d, -0x70,0x69,0x6c,0x65,0x42,0x75,0x69,0x6c, -0x74,0x69,0x6e,0x4d,0x6f,0x64,0x75,0x6c, -0x65,0x20,0x61,0x6e,0x64,0x20,0x74,0x68, -0x65,0x20,0x58,0x44,0x44,0x20,0x2e,0x64, -0x61,0x73,0x2e,0x69,0x6e,0x63,0x20,0x66, -0x69,0x6c,0x65,0x2e,0x0a, -0x2f,0x2f,0x20,0x64,0x61,0x73,0x6c,0x61, -0x6e,0x67,0x20,0x63,0x6c,0x61,0x73,0x73, -0x65,0x73,0x20,0x63,0x61,0x6e,0x20,0x64, -0x65,0x72,0x69,0x76,0x65,0x20,0x66,0x72, -0x6f,0x6d,0x20,0x74,0x68,0x69,0x73,0x20, -0x61,0x6e,0x64,0x20,0x6f,0x76,0x65,0x72, -0x72,0x69,0x64,0x65,0x20,0x74,0x68,0x65, -0x20,0x76,0x69,0x72,0x74,0x75,0x61,0x6c, -0x20,0x6d,0x65,0x74,0x68,0x6f,0x64,0x73, -0x2e,0x0a, -0x0a, -0x63,0x6c,0x61,0x73,0x73,0x20,0x54,0x75, -0x74,0x6f,0x72,0x69,0x61,0x6c,0x42,0x61, -0x73,0x65,0x43,0x6c,0x61,0x73,0x73,0x20, -0x7b,0x0a, -0x20,0x20,0x20,0x20,0x64,0x65,0x66,0x20, -0x61,0x62,0x73,0x74,0x72,0x61,0x63,0x74, -0x20,0x75,0x70,0x64,0x61,0x74,0x65,0x28, -0x64,0x74,0x20,0x3a,0x20,0x66,0x6c,0x6f, -0x61,0x74,0x29,0x20,0x3a,0x20,0x76,0x6f, -0x69,0x64,0x0a, -0x20,0x20,0x20,0x20,0x64,0x65,0x66,0x20, -0x61,0x62,0x73,0x74,0x72,0x61,0x63,0x74, -0x20,0x67,0x65,0x74,0x5f,0x70,0x6f,0x73, -0x69,0x74,0x69,0x6f,0x6e,0x20,0x3a,0x20, -0x66,0x6c,0x6f,0x61,0x74,0x33,0x0a, -0x7d,0x0a, -}; +static unsigned char class_adapters_module_das[] = { +0x6f,0x70,0x74,0x69,0x6f,0x6e,0x73,0x20, +0x67,0x65,0x6e,0x32,0x0a, +0x6f,0x70,0x74,0x69,0x6f,0x6e,0x73,0x20, +0x72,0x65,0x6d,0x6f,0x76,0x65,0x5f,0x75, +0x6e,0x75,0x73,0x65,0x64,0x5f,0x73,0x79, +0x6d,0x62,0x6f,0x6c,0x73,0x20,0x3d,0x20, +0x66,0x61,0x6c,0x73,0x65,0x0a, +0x0a, +0x2f,0x2f,0x20,0x41,0x62,0x73,0x74,0x72, +0x61,0x63,0x74,0x20,0x62,0x61,0x73,0x65, +0x20,0x63,0x6c,0x61,0x73,0x73,0x20,0xe2, +0x80,0x94,0x20,0x64,0x61,0x73,0x6c,0x61, +0x6e,0x67,0x20,0x73,0x69,0x64,0x65,0x2e, +0x0a, +0x2f,0x2f,0x20,0x43,0x2b,0x2b,0x20,0x77, +0x69,0x6c,0x6c,0x20,0x69,0x6e,0x63,0x6c, +0x75,0x64,0x65,0x20,0x74,0x68,0x69,0x73, +0x20,0x76,0x69,0x61,0x20,0x63,0x6f,0x6d, +0x70,0x69,0x6c,0x65,0x42,0x75,0x69,0x6c, +0x74,0x69,0x6e,0x4d,0x6f,0x64,0x75,0x6c, +0x65,0x20,0x61,0x6e,0x64,0x20,0x74,0x68, +0x65,0x20,0x58,0x44,0x44,0x20,0x2e,0x64, +0x61,0x73,0x2e,0x69,0x6e,0x63,0x20,0x66, +0x69,0x6c,0x65,0x2e,0x0a, +0x2f,0x2f,0x20,0x64,0x61,0x73,0x6c,0x61, +0x6e,0x67,0x20,0x63,0x6c,0x61,0x73,0x73, +0x65,0x73,0x20,0x63,0x61,0x6e,0x20,0x64, +0x65,0x72,0x69,0x76,0x65,0x20,0x66,0x72, +0x6f,0x6d,0x20,0x74,0x68,0x69,0x73,0x20, +0x61,0x6e,0x64,0x20,0x6f,0x76,0x65,0x72, +0x72,0x69,0x64,0x65,0x20,0x74,0x68,0x65, +0x20,0x76,0x69,0x72,0x74,0x75,0x61,0x6c, +0x20,0x6d,0x65,0x74,0x68,0x6f,0x64,0x73, +0x2e,0x0a, +0x0a, +0x63,0x6c,0x61,0x73,0x73,0x20,0x54,0x75, +0x74,0x6f,0x72,0x69,0x61,0x6c,0x42,0x61, +0x73,0x65,0x43,0x6c,0x61,0x73,0x73,0x20, +0x7b,0x0a, +0x20,0x20,0x20,0x20,0x64,0x65,0x66,0x20, +0x61,0x62,0x73,0x74,0x72,0x61,0x63,0x74, +0x20,0x75,0x70,0x64,0x61,0x74,0x65,0x28, +0x64,0x74,0x20,0x3a,0x20,0x66,0x6c,0x6f, +0x61,0x74,0x29,0x20,0x3a,0x20,0x76,0x6f, +0x69,0x64,0x0a, +0x20,0x20,0x20,0x20,0x64,0x65,0x66,0x20, +0x61,0x62,0x73,0x74,0x72,0x61,0x63,0x74, +0x20,0x67,0x65,0x74,0x5f,0x70,0x6f,0x73, +0x69,0x74,0x69,0x6f,0x6e,0x20,0x3a,0x20, +0x66,0x6c,0x6f,0x61,0x74,0x33,0x0a, +0x7d,0x0a, +};