feat(inference): prove Gemma 4 production Q4_K parity (ST6) - #64
Merged
Conversation
LARQL-INFERENCE-TRUST-001A-ST6: production Q4_K/Q6_K semantic parity
against the ST5-proven F32 reference for Gemma 4 E2B.
Decision: RED — the committed Q4_K quality gate is not met, though
first-token top-1 is exact for all four ST3 prompts and teacher-forced
top-1 agreement is 100%. The earliest budget breach is post_layer@9
(NRMSE 0.158 > 0.15) on the arithmetic prompt; top-10 overlap is 6-7/10
for the two short prompts (raw_completion, chat). The tied f16 lm-head
is near-lossless (C-vs-B NRMSE ~5e-6); the Q4_K body dominates the
error. No writer/loader/dequantizer defect was found — the breaches are
expected Q4_K quantization noise marginally exceeding the committed
coarse-NRMSE gate at mid layers.
Framework (extends larql-inference::parity rather than a new system):
- capture.rs: write_larql_q4k_trace / capture_prompt_q4k run the
production route (predict_kquant_hidden_hooked + traced_tail_from_hidden),
recording per-layer boundaries + shared-KV consumer markers.
- compare.rs: Policy::st6_default three-tier budget (coarse 0.15/0.98,
hidden 0.10/0.99, logits 0.05/0.995) + Diagnostic tier for the
FFN-block-internal stages (post_ffn/post_ple/layer_input/embedding),
per ST6 §5's "post-attention and post-layer residual" gating. Richer
ranking diagnostics (margins, softmax KL, max prob diff).
- larql-compute: traced_tail_from_hidden / traced_tail_with_lm_head
shared tail math (final-norm + lm-head + logits transform) for the
F32 and Q4_K capture paths + the §8 lm-head decomposition.
Tests (CI-runnable, no artifact required):
- parity_diagnostics_q4k.rs: 10 §10 self-tests.
- gemma4_q4k_shared_kv.rs: 5 §7 shared-KV/attention topology proofs
(source layers 13/14, local-windowed/global-full, proportional RoPE)
+ 1 env-gated F32/Q4_K topology-agreement test.
- gemma4_q4k_f32_semantic_parity.rs: env-gated official test
(§1 provenance, §4 trace capture, §5 comparison, §6 teacher-forced,
§7 shared-KV, §8 lm-head decomposition, §11 reports).
Evidence:
- bench/baselines/gemma4-e2b-q4k-f32-semantic-parity-2026-07-12.{json,md}
- ST5 report metadata corrected (head SHA, merge commit, CI status).
Scope: CPU full-recompute Q4_K only. No KV-cached decode, GPU, or
sampling. Recommended next slice: ST6A (narrow correction) or ST7
(cached prefill/decode parity).
The ST6 parity framework added traced_tail_from_hidden and traced_tail_with_lm_head to forward/predict/raw.rs, but they were only exercised by larql-inference's parity tests — dropping raw.rs below the 90% per-file coverage floor (88.14%) and failing the compute coverage gate. Add two unit tests in raw.rs: - traced_tail_from_hidden_matches_forward_raw_logits: the shared tail must agree with forward_raw_logits' inline pre-final-norm / final-norm / final-logits at the last token. - traced_tail_with_lm_head_uses_explicit_matrix: passing the model's own lm-head matches the inline path; a zeroed lm-head yields zero logits.
larql-kv's `_inplace_matches_owned_concat` decode-parity tests compare two mathematically-equivalent code paths that accumulate FP ops in different orders. Under multi-threaded BLAS the accumulation order is nondeterministic, so the comparison flakes (divergences up to ~4× the 1e-2 tolerance). The larql-inference workflow already pins OPENBLAS/OMP_NUM_THREADS=1 for the same reason; larql-kv was missing the pin even though it depends on larql-inference and runs BLAS-heavy forward passes. Add OPENBLAS_NUM_THREADS=1 + OMP_NUM_THREADS=1 to the kv workflow env, matching the inference workflow. Verified locally: 3/3 full kv lib runs pass (765 tests) with the pin vs ~50% flake rate without.
The BLAS-thread pin fixed the numerical flake in the `_inplace_matches_owned_concat` decode-parity tests (765 lib tests now pass), but the full `cargo test -p larql-kv` still SIGSEGV'd during teardown — a background thread in the kv-dispatch decode path races the test-harness exit under parallel execution. Same memory-safety pattern that larql-compute-metal and larql-server already mitigate with `--test-threads=1`. Add `--test-threads=1` to the kv test step. Verified locally: full `cargo test -p larql-kv -- --test-threads=1` passes 797 tests cleanly with no SIGSEGV (vs intermittent SIGSEGV + numerical flake under parallel execution).
The kv coverage job runs `cargo llvm-cov` which executes the test binary in parallel by default, hitting the same decode-parity flake (BLAS FP nondeterminism) the test step was just fixed for. Add `-- --test-threads=1` to the llvm-cov invocation, matching the larql-server coverage step's pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LARQL-INFERENCE-TRUST-001A-ST6 — Production Q4_K Semantic Parity
Decision: RED — the committed Q4_K quality gate is not met, though first-token top-1 is exact for all four ST3 prompts and teacher-forced top-1 agreement is 100%.
What this proves
Compares LARQL's production CPU Q4_K/Q6_K route (
predict_kquant_hidden_hooked+traced_tail_from_hidden) against the ST5-proven F32 reference for Gemma 4 E2B, on the four ST3 prompts: per-layer residual boundaries, final hidden, final logits, a 4-step teacher-forced continuation, the quantized-lm-head error decomposition, and shared-KV topology.Key findings (RED)
post_layer@9(arithmetic), NRMSE 0.158 > 0.15 (cosine passes).Canonical artifact
Built from the pinned source (
9dbdf8a…, safetensors2db5482b…) vialarql extract … --level all --quant q4k. 7.87 GB; attn Q/K/O→Q4_K, V→Q6_K; FFN gate/up→Q4_K, down→Q6_K; lm-head tied to f16 embeddings; norms→F32. Provenance (hashes, format assignments, 35 layers, PLE, no F32 duplicate) verified in the env-gated test.Framework (extends
larql-inference::parity)capture.rs: Q4_K production-route trace capture.compare.rs:st6_defaultthree-tier budget + Diagnostic tier for FFN-internal stages (ST6 §5 gates post-attention + post-layer residual); richer ranking diagnostics.larql-compute: sharedtraced_tail_from_hidden/traced_tail_with_lm_headtail math.Tests
parity_diagnostics_q4k.rs— 10 §10 self-tests (CI, no artifact).gemma4_q4k_shared_kv.rs— 5 §7 shared-KV/attention topology proofs + 1 env-gated (CI + gated).gemma4_q4k_f32_semantic_parity.rs— env-gated official test (soft-skips in CI).Validation
cargo fmt --all -- --check✓ ·cargo clippy(5 crates,-D warnings) ✓ ·cargo test(models 32, compute 760, inference 1301, kv 765, vindex 1154) ✓ ·cargo build -p larql-cli --release✓ · ST4/ST4A/ST5 regressions ✓Scope exclusions
KV-cached E2B decode, direct-matvec decode, CUDA/Vulkan/Metal, sampling, performance optimization, Q5/Q8/F16 to obtain GREEN, multimodal, tools/thinking, long-form generation quality.
Recommended next slice
ST6A (narrow correction targeting the recorded first budget breach) or ST7 (Q4_K cached prefill/decode parity).