diff --git a/README.md b/README.md index cf47027..9d28d06 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,7 @@ Grouped by purpose. | Path | Contents | |---|---| | `skill/references/evidence.md` | Index from each decision step to the tables behind it | -| `skill/references/findings.md` | The generalisable lessons (32), each with its sources | +| `skill/references/findings.md` | The generalisable lessons (35), each with its sources | | `skill/references/CORRECTIONS.md` | Where findings disagreed with the data; corrections win | | `skill/references/tables/` | One table per measurement set, extracted from the DATA-tagged sources | | `skill/references/hardware-tiers.md` | Every hardware class from 8 GB Intel Arc to 512 GB Mac Studio and DGX (sourced, not measured here) | diff --git a/data/INVENTORY.md b/data/INVENTORY.md index 793e705..419bcf5 100644 --- a/data/INVENTORY.md +++ b/data/INVENTORY.md @@ -270,6 +270,52 @@ Changes (model-tuning-relevant headings): --- +## state/evals/2026-10-04 (promoted 2026-10-14) + +### state/evals/2026-10-04/strata-pr783-depth/RESULTS.md: Strata #783 depth A/B, alternating engine-only change +- **Measured:** base vs +#783 vs +#783+#796, 2 rounds each, 119K/238K decode at depth + recall; #783 +14.2% (119K) / +9.5% (238K) decode, prefill unchanged, recall 4/4 all arms. +- **Tag:** FINDING (engine version is a tuning variable) + DATA (tabulated in `skill/references/tables/prefill-decode-speed.md` T9). + +### state/evals/2026-10-04/strata-calib/RESULTS.md: short-prompt calibration vs depth +- **Measured:** calibrate.py short prompts picked pcie-frac 0.20 + spec-min-p 0.70; depth harness shows spec-min-p 0.70 costs decode at depth, learned-from-short-prompts profile -20% at 119K; live is pcie-frac 0.20 + spec-min-p 0.5. +- **Tag:** FINDING (calibration caveat) — short-prompt calibration does not transfer to depth. + +### state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md: NInfer-27B vs Strata IQ2_XS head-to-head, quick tier + needle +- **Measured:** n=2-3 per model; review tie (12,10 vs 12,10,9), needle 9/9 both to 200K; Strata ~35% faster decode, 2.1x faster 200K read; NInfer one-shot builds ~1.7x larger/richer. +- **Tag:** DATA (tabulated in prefill-decode-speed T12, quality-bench T16) + FINDING (report t/s AND quality; preliminary, n=2-3). + +### state/evals/2026-10-04/r15-real-tasks/RESULTS.md: real-ticket graded stage 2 +- **Measured:** 3 real tickets, Claude-graded 5 axes; NInfer 112/150 (2/3 finished in cap), Strata 86/150 (1/3). Quick-tier tie broke only on real tasks. +- **Tag:** FINDING (tie-break by real tasks; preliminary, n=1 task set) + DATA (quality-bench T17). + +### state/evals/2026-10-04/strata-0139-512k/RESULTS.md: engine-upgrade RAM pinning (already tabulated as vram-fit T7) +- **Measured:** 0.1.39 pinned 17.24→19.20 GiB, min free 3.36→1.41 GiB; headroom 6 restores 3.53 GiB. +- **Tag:** FINDING (already finding 31; numbers now also in SKILL.md Step 4 trap). + +## state/evals/2026-10-10/2026-10-11 (promoted 2026-10-14) + +### state/evals/2026-10-11/r17-strata-0143/result.md: Strata 0.1.43 vs 0.1.39-based deploy +- **Measured:** same 262K config, 3 runs per variant; agent-session decode +8.3%, prefill +6-8% at 16-32K, peak VRAM unchanged (23,778 vs 23,776 MiB). Deployed B. +- **Tag:** DATA (prefill-decode-speed T10) + FINDING (engine-version A/B, quality held). + +### state/evals/2026-10-11/v3-switch/result.md: NInfer v3 vs v2 +- **Measured:** prefill +1-2%, decode -0.8% (156.3 vs 157.5 t/s), acceptance 88.7% vs 86.7%, PPL +0.007% vs A16, greedy parity identical. +- **Tag:** DATA (prefill-decode-speed T11, speculative-mtp T7). + +### state/evals/2026-10-11/r15-agentfast/result.md + fit.jsonl: template/sampler token efficiency + fit gate +- **Measured:** n=1 per variant; Sharp temp 0.6 same 10/12 in -18% tokens; presence 1.5 cuts MTP accept 89→84.2%, decode 308→254 t/s. Q4_K_XL ruled out by fit gate before tuning ran. +- **Tag:** DATA (speculative-mtp T7, vram-fit T8) — sampler rows preliminary, needs repeats. + +### state/evals/2026-10-10/v3-agent/RESULT.md: chat-template effort trap +- **Measured:** v3 400 `invalid_prompt` because jinja refuses effort `high` (what Claude Code sends); fix maps high/max→xhigh, minimal→low. +- **Tag:** FINDING (effort sets differ per template/engine; SKILL.md Step 7 trap). + +## state/evals/2026-10-14 (promoted 2026-10-14) + +### state/evals/2026-10-14/gemma4-ab/side-*.log + repro JSON: Gemma4-26B-A4B-QAT first point +- **Measured:** NInfer engine, sm_89 native; 14.6 GiB weights, kv_capacity 131072 with kv_headroom_bytes=0; decode 330-400 t/s, MTP 72-86%, tool-call ok. Review 8-9 (preliminary, n=1 per arm). +- **Tag:** DATA (prefill-decode-speed T13, speculative-mtp T7, vram-fit T8, quality-bench T18). First non-Qwen family point. + ## External verification sources (cited in `references/`) These are web sources verified live (2026-10-08) and cited in the new reference files. None are measurements from this repo; all are externally sourced and must be verified on the user's own hardware. diff --git a/skill/SKILL.md b/skill/SKILL.md index 8005699..b24eef4 100644 --- a/skill/SKILL.md +++ b/skill/SKILL.md @@ -123,7 +123,9 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri desktop compositor to the iGPU is a real lever on a shared card. **Re-measure host RAM headroom after every engine upgrade**: defaults that pin memory can change: an upgrade grew the default page-locked copy from 17.24 to 19.20 GiB and cut min free RAM at a 477K prompt - from 3.36 to 1.41 GiB on the same box and config, with no change in VRAM. + from 3.36 to 1.41 GiB on the same box and config, with no change in VRAM. A headroom + setting of 6 GiB (`STRATA_RESIDENT_HEADROOM_GIB`) restored 3.53 GiB and kept most of the + long-prompt gain. Engine-specific numbers. - **Verify:** peak host RAM and peak VRAM per run, plus the engine's slot/pinned figures, before and after any engine upgrade. @@ -151,7 +153,9 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri - **Evidence:** `references/tables/speculative-mtp.md` T1–T5; findings 2 (corrected), 11, 12, 13. - **Trap:** the draft context is a separate allocation that can OOM even when the target model fits. That is the measured fact. Do not claim it grows during generation; the measured runtime - growth is the compute buffers (finding 5). MTP compatibility depends + growth is the compute buffers (finding 5). Short-prompt calibration does not transfer to depth: + a calibrator's pick of spec-min-p 0.70 cost decode at 119K/238K and a profile learned from short + prompts cost ~20% at 119K; verify any calibrated spec setting at depth before promoting (finding 34). MTP compatibility depends on exact builder/commit: flag names changed between builds (`--draft-max` removed → `--spec-draft-n-max`), so pin versions. Auto-fit and manual offload are mutually exclusive: `common_fit_params` aborts when `-ngl` is already set. @@ -179,7 +183,10 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri length cap and produced nothing reviewable). - **Trap:** on models whose template implements a `reasoning_effort` switch, `medium` is a silent mode that injects nothing; only the default (xhigh) injects reasoning instructions, and an - invalid value hard-fails the template render. `--jinja` is required for template-driven kwargs + invalid value hard-fails the template render. Effort value sets differ per template/engine: one + engine's jinja refused effort `high` (HTTP 400, client hung with no retry); the fix mapped + high/max to xhigh and minimal to low. Check the effort values against the template in use. + `--jinja` is required for template-driven kwargs to work at all. None of this applies to a model that has no such switch. - **Verify:** finish_reason per run. `finish=length` with no content is a budget failure, not a model failure; several models scored 0/8 in round 1 for exactly that. diff --git a/skill/references/findings.md b/skill/references/findings.md index 5468690..bb20127 100644 --- a/skill/references/findings.md +++ b/skill/references/findings.md @@ -34,3 +34,6 @@ The most generalisable lessons from the evidence inventory. Each line is one les 30. Quality-per-second, not raw quality, should drive role choice. A blind-ranking tie was broken by 69 s vs 209 s at equal quality (the 280 s figure in the source is the later 27B-Research run, not the tied pair). (corrected, see `CORRECTIONS.md`; `state/evals/2026-09-25/think-ranking.md`, `state/evals/2026-09-25/r2.log`) 31. Re-measure host RAM headroom after every engine upgrade: defaults that pin memory can change. An upgrade that grew the default page-locked copy (17.24 -> 19.20 GiB) cut min free RAM at a 477K prompt from 3.36 to 1.41 GiB on a 31 GB box; a headroom setting (6 GiB) restored 3.53 GiB and kept most of the speed. Engine-specific numbers. (`state/evals/2026-10-04/strata-0139-512k/RESULTS.md`) 32. CPU offload trades speed for headroom. Measured: higher offload = more headroom, less speed (68–124 t/s across `--n-cpu-moe` settings on one MoE entry, vs 234–245 t/s fully on GPU). `--n-cpu-moe` is the correct overflow lever, never `-ngl`; offloading the draft layer trades the MTP win away. (`reference/LOCAL_MODELS.md`, `reference/llama-swap/TUNING-4090.md`) +33. Engine version is a tuning variable: A/B it with quality held constant. Strata #783 gained +14.2% decode at 119K and +9.5% at 238K with recall 4/4 and prefill unchanged (alternating arms, 2 rounds); Strata 0.1.43 gained +8.3% agent-session decode and +6-8% prefill at 16-32K at identical VRAM; NInfer v3 gained +1-2% prefill at -0.8% decode with greedy parity identical and PPL +0.007% vs A16. Engine-specific numbers. (`state/evals/2026-10-04/strata-pr783-depth/RESULTS.md`, `state/evals/2026-10-11/r17-strata-0143/result.md`, `state/evals/2026-10-11/v3-switch/result.md`) +34. Short-prompt calibration does not transfer to depth: verify at depth before promoting. A calibrator's short-prompt pick (spec-min-p 0.70) cost decode at 119K/238K, and a profile learned from short prompts cost ~20% at 119K; the deployed setting kept spec-min-p 0.5 with pcie-frac 0.20. Single-run depth check — re-verify with repeats. (`state/evals/2026-10-04/strata-calib/RESULTS.md`) +35. Report t/s AND quality: a quick-tier tie breaks only on real tasks. Strata decoded ~35% faster and read 200K 2.1x faster with review tied (12,10 vs 12,10,9, n=2-3) and richer NInfer one-shot output (~1.7x rules); graded real tickets then split them (NInfer 112/150 finishing 2/3 in cap, Strata 86/150 finishing 1/3). Preliminary (n=2-3 quick tier, n=1 task set); do not harden into a rule. (`state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`, `state/evals/2026-10-04/r15-real-tasks/RESULTS.md`) diff --git a/skill/references/tables/prefill-decode-speed.md b/skill/references/tables/prefill-decode-speed.md index af4476d..3ec2ae0 100644 --- a/skill/references/tables/prefill-decode-speed.md +++ b/skill/references/tables/prefill-decode-speed.md @@ -143,3 +143,65 @@ Run-to-run spread inside each arm is +-20 t/s, so short-prompt decode is a wash; difference is the one that held across depths. Source: `state/evals/2026-10-03/strata-ctx/RESULTS.md`, `state/evals/2026-10-03/strata-512k-live/RESULTS.md`, `state/evals/2026-10-04/strata-0139-512k/RESULTS.md`. + +## Table 9: Strata #783 depth A/B, alternating engine-only change (IQ2_XS, live 262K q4_0, 2 rounds per arm, recall held) + +| Build | 119K prefill t/s | 119K decode t/s | 238K prefill t/s | 238K decode t/s | Recall | +|---|---|---|---|---|---| +| base (v0.1.39 + #700) | 4048 / 4044 | 106.2 / 106.6 | 3898 / 3898 | 126.7 / 127.3 | 4/4 | +| + #783 | 4041 / 4031 | 121.8 / 121.5 | 3897 / 3889 | 138.9 / 139.3 | 4/4 | +| + #783 + #796 | 4046 / 4046 | 121.6 / 121.5 | 3896 / 3904 | 139.4 / 138.6 | 4/4 | + +#783: decode at depth +14.2% at 119K, +9.5% at 238K; prompt reading unchanged. #796: no speed change (startup-safety value only). Engine-specific (Strata). + +Source: `state/evals/2026-10-04/strata-pr783-depth/RESULTS.md`. + +## Table 10: Strata 0.1.43 vs 0.1.39-based deploy, same 262K config (r17, 3 runs per variant, engine-specific) + +| Variant | Chat decode | Agent-session decode | 16K prompt prefill | 32K prompt prefill | Peak VRAM | +|---|---|---|---|---|---| +| A: deploy (0.1.39-based) | 133.6 t/s | 144.7 t/s | 3,757 t/s | 3,760 t/s | 23,776 MiB | +| B: v0.1.43, same config | 134.8 (+0.9%) | 156.7 (+8.3%) | 3,978 (+5.9%) | 4,054 (+7.8%) | 23,778 MiB | +| C: v0.1.43, no --pcie-frac | 141.8 (+6.1%) | 152.3 (+5.3%) | 3,957 | 4,033 | 23,778 MiB | + +B vs C within noise (C's measured PCIe share settled at 0.15-0.20, near the pinned 0.20). Deployed B. Engine-specific (Strata). + +Source: `state/evals/2026-10-11/r17-strata-0143/result.md`. + +## Table 11: NInfer v3 vs v2, same flags (INT8 prefill, rk4v4-e8 KV, MTP3, engine-specific) + +| Metric | v2 (deploy 46645ada) | v3 (port 322a7f51) | v3/v2 | +|---|---|---|---| +| prefill 2048 | 3403.7 t/s | 3462.5 t/s | 1.02 | +| prefill 8192 | 3730.7 t/s | 3789.3 t/s | 1.02 | +| prefill 32768 | 3501.1 t/s | 3550.2 t/s | 1.01 | +| prefill 65536 | 3106.7 t/s | 3145.6 t/s | 1.01 | +| decode (code, greedy, median of 3) | 157.5 t/s | 156.3 t/s | 0.99 | + +Parity identical (greedy); perplexity +0.007% vs A16 (4.343314 vs 4.343023); draft acceptance 88.7% vs 86.7%. Engine-specific (NInfer). + +Source: `state/evals/2026-10-11/v3-switch/result.md`. + +## Table 12: r14 head-to-head prefill at depth + decode (same box, production configs, n=2-3) + +| Size | NInfer-Qwen3.8-27B prefill | Strata IQ2_XS prefill | NInfer decode | Strata decode | +|---|---|---|---|---| +| 60K (55,586 tok) | 3,037 t/s (18.3 s prefill; 23.3 s logged incl. answer) | 3,494 t/s (17.3 s) | 127-140 t/s | 173-185 t/s | +| 120K (117,153 tok) | 2,391 t/s (51.2 s) | 3,904 t/s (31.3 s) | | | +| 200K (~199.5K tok) | 1,857 t/s (110.2 s) | 3,833 t/s (53.7 s) | | | + +Strata decodes ~35% faster and reads a 200K prompt 2.1x faster on this box. Needle recall 9/9 both (see quality-bench T16). Logged seconds are whole-request wall time (needle.py times the full chat call, answer generation included); prefill-only time is tokens / t/s - at 60K the NInfer engine log records TTFT 18.3 s vs total 23.3 s. Engine-specific. + +Source: `state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`. + +## Table 13: Gemma4-26B-A4B-QAT speeds (NInfer engine, sm_89 native, 2026-10-14, n=2 builds) + +| Request | Prefill t/s | Decode t/s | MTP accept | +|---|---|---|---| +| build turn1 (prompt 619) | 4.28-4.29k | 330.8-357.4 | 72.0-77.2% | +| build turn1 tool-call (prompt 176, output 4.7-6.1K) | 2.88-2.90k | 391.4-399.1 | 84.2-86.1% | +| review probe (prompt 5.0-6.5K, output ~30) | 5.51-5.59k | 290.7-324.7 | 64.3-79.2% | + +Weights 14.6 GiB; kv_capacity 131072 with kv_headroom_bytes=0; tool-call ok. First non-Qwen family point. Engine-specific (NInfer). + +Source: `state/evals/2026-10-14/gemma4-ab/side-B.log`, `side-C.log`. diff --git a/skill/references/tables/quality-bench.md b/skill/references/tables/quality-bench.md index 23d3512..07a12df 100644 --- a/skill/references/tables/quality-bench.md +++ b/skill/references/tables/quality-bench.md @@ -228,3 +228,42 @@ Source: role-suite results JSON at the `state/evals/` root, dated 2026-09-25 (th The same models failed a real multi-turn build (see Table 4: Coder-Pruned 0/8; GAIN hit the length cap with no build; Mini skipped the tool call), the suite could not see it. Source: `reference/llama-swap/config.yaml` (header comment, 2026-09-23). + +## Table 16: r14 head-to-head quality (same harness, same prompts, n=2-3 per model — preliminary) + +| Run | Model | Review /12 | Build CSS rules / props | Decode t/s | +|---|---|---|---|---| +| r14n-a | NInfer-Qwen3.8-27B | 12/12 | 181 / 379 | 140 | +| r14n-b | NInfer-Qwen3.8-27B | 10/12 | 190 / 405 | 127 | +| r14s-a | Strata IQ2_XS | 12/12 | 113 / 271 | 173 | +| r14s-b | Strata IQ2_XS | 10/12 | 108 / 241 | 182 | +| r14s-n | Strata IQ2_XS | 9/12 | 106 / 250 | 185 | + +Review: tie (both 12 then 10; Strata third run 9/12 within ±2 noise). Needle recall 9/9 both to 200K. NInfer one-shot builds ~1.7x larger/richer (181-190 vs 106-113 rules); size is a proxy, not a judged score. + +Source: `state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`. + +## Table 17: r15 real-task graded outcomes (3 real tickets, Claude-graded 0-10 x5 axes, same tools both engines) + +| Task | Strata IQ2_XS | NInfer-Qwen3.8-27B | +|---|---|---| +| T1 homelab Python | 42 (done, committed, unreported at cap) | 35 (done, 72 min) | +| T2 Aphotic QML | 0 (nothing written at 150-min cap) | 34 (complete at 80 min, died before commit) | +| T3 llm-tune evidence | 44 (done, 120 min) | 43 (done, 62 min) | +| Total /150 | 86 | 112 | + +Quick-tier tie (T16) broke only on real tasks: NInfer finished 2/3 within cap, Strata 1/3. n=1 task set — preliminary. + +Source: `state/evals/2026-10-04/r15-real-tasks/RESULTS.md`. + +## Table 18: Gemma4-26B-A4B-QAT review (planted-bug suite, NInfer engine, 2026-10-14 — preliminary) + +| Run | Review found (raw count, NOT /12 — non-comparable to T16) | Tool-call | +|---|---|---| +| gemma4-review (earlier pass) | 8 | n/a | +| gemma4-ab sampler arm | 9 | ok (tool_calls finish) | +| gemma4-ab thinkmax arm | 8 | ok (tool_calls finish) | + +First non-Qwen family quality point; raw counts are non-comparable to the /12 tables (T4–T17) — harness denominator unknown from the source as logged, do not compare across tables. n=1 per arm — preliminary. + +Source: `state/evals/2026-10-14/gemma4-review/review2/Gemma4-26B-A4B-QAT.md`, `state/evals/2026-10-14/gemma4-ab/review2/*.md`. diff --git a/skill/references/tables/speculative-mtp.md b/skill/references/tables/speculative-mtp.md index 51af06f..b4e6514 100644 --- a/skill/references/tables/speculative-mtp.md +++ b/skill/references/tables/speculative-mtp.md @@ -118,4 +118,21 @@ Source: `state/evals/2026-10-03/ninfer-sweep1/results.jsonl`. Source: `state/evals/2026-10-03/strata-k8v4/sweep.jsonl` (`engine_mem_lines`). +## Table 7: post-v0.2 acceptance datapoints (engine-specific, raw) + +| Source | Arm | Acceptance | Decode | Note | +|---|---|---|---|---| +| NInfer v3-switch | v2 | 86.7% | 157.5 t/s | greedy parity identical | +| NInfer v3-switch | v3 | 88.7% | 156.3 t/s (-0.8%) | prefill +1-2%, PPL +0.007% vs A16 | +| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | build turn1 | 72.0-77.2% | 330.8-357.4 t/s | weights 14.6 GiB | +| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | build tool-call turn | 84.2-86.1% | 391.4-399.1 t/s | tool-call ok | +| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | review probe | 64.3-79.2% | 290.7-324.7 t/s | short output | +| r15-agentfast AF-base (n=1, preliminary) | temp 0.6, qwen3.6 template | 89.0% | 308 t/s | review 10/12 | +| r15-agentfast AF-sharp (n=1, preliminary) | temp 0.6, Sharp template | 88.5% | 309 t/s | same 10/12 in -18% tokens | +| r15-agentfast AF-sharp-general (n=1, preliminary) | temp 1.0, presence 1.5 | 84.2% | 254 t/s | build thinned (62 vs 116 rules), not for this entry | + +Sampler/acceptance rows are n=1 — preliminary, needs repeats before promotion. + +Source: `state/evals/2026-10-11/v3-switch/result.md`, `state/evals/2026-10-14/gemma4-ab/side-*.log`, `state/evals/2026-10-11/r15-agentfast/result.md`. + Reference datapoint (model card, not measured here): Q4_K_S regular ~75 t/s vs MTP >90 t/s at 60% acceptance, 2 tokens, roughly +20% at 60% acceptance; the ratio, not the absolute t/s, is the transferable part. Gate: below ~50% acceptance, regular quants run faster. Source: `state/evals/2026-09-25/gain-research.md` (Note 3), `runbooks/llama-swap.md` (bench gate >= 0.4). diff --git a/skill/references/tables/vram-fit.md b/skill/references/tables/vram-fit.md index fa7174f..f05f3f8 100644 --- a/skill/references/tables/vram-fit.md +++ b/skill/references/tables/vram-fit.md @@ -127,3 +127,13 @@ Headroom 6 restores 3.53 GiB and keeps most of the long-prompt gain (-12% read a default, +10% decode at depth). Recall held at all four depths on every arm. Source: `state/evals/2026-10-04/strata-0139-512k/RESULTS.md`. + +## Table 8: post-v0.2 fit datapoints (raw, engine-specific) + +| Entry | Weights / peak | ctx | Result | +|---|---|---|---| +| Gemma4-26B-A4B-QAT (NInfer, sm_89 native) | 14.6 GiB weights | 131072 | loads; kv_capacity 131072 with kv_headroom_bytes=0 (zero headroom); tool-call ok | +| Qwen3.6-35B-A3B UD-Q4_K_XL (llama.cpp, r15-agentfast fit probe) | load failed at ctx 200000/131072/65536+ with n_cpu_moe 0 | 131072+ | ruled out by the fit gate before tuning ran; UD-IQ4_XS (peak 22,572 MiB) used instead | +| Qwen3.6-35B-A3B UD-Q4_K_XL | peak 23,924 MiB at ub1024 / 23,782 at ub512 | default probe ctx (no ctx field logged) | loads only with n_cpu_moe 6-7 in the probe ladder; still ruled out for the run (fit-fragile) | + +Source: `state/evals/2026-10-14/gemma4-ab/side-*.log`, `state/evals/2026-10-11/r15-agentfast/fit.jsonl`.