Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ Grouped by purpose.
| Path | Contents |
|---|---|
| `skill/references/evidence.md` | Index from each decision step to the tables behind it |
| `skill/references/findings.md` | The generalisable lessons (32), each with its sources |
| `skill/references/findings.md` | The generalisable lessons (35), each with its sources |
| `skill/references/CORRECTIONS.md` | Where findings disagreed with the data; corrections win |
| `skill/references/tables/` | One table per measurement set, extracted from the DATA-tagged sources |
| `skill/references/hardware-tiers.md` | Every hardware class from 8 GB Intel Arc to 512 GB Mac Studio and DGX (sourced, not measured here) |
Expand Down
46 changes: 46 additions & 0 deletions data/INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,52 @@ Changes (model-tuning-relevant headings):

---

## state/evals/2026-10-04 (promoted 2026-10-14)

### state/evals/2026-10-04/strata-pr783-depth/RESULTS.md: Strata #783 depth A/B, alternating engine-only change
- **Measured:** base vs +#783 vs +#783+#796, 2 rounds each, 119K/238K decode at depth + recall; #783 +14.2% (119K) / +9.5% (238K) decode, prefill unchanged, recall 4/4 all arms.
- **Tag:** FINDING (engine version is a tuning variable) + DATA (tabulated in `skill/references/tables/prefill-decode-speed.md` T9).

### state/evals/2026-10-04/strata-calib/RESULTS.md: short-prompt calibration vs depth
- **Measured:** calibrate.py short prompts picked pcie-frac 0.20 + spec-min-p 0.70; depth harness shows spec-min-p 0.70 costs decode at depth, learned-from-short-prompts profile -20% at 119K; live is pcie-frac 0.20 + spec-min-p 0.5.
- **Tag:** FINDING (calibration caveat) — short-prompt calibration does not transfer to depth.

### state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md: NInfer-27B vs Strata IQ2_XS head-to-head, quick tier + needle
- **Measured:** n=2-3 per model; review tie (12,10 vs 12,10,9), needle 9/9 both to 200K; Strata ~35% faster decode, 2.1x faster 200K read; NInfer one-shot builds ~1.7x larger/richer.
- **Tag:** DATA (tabulated in prefill-decode-speed T12, quality-bench T16) + FINDING (report t/s AND quality; preliminary, n=2-3).

### state/evals/2026-10-04/r15-real-tasks/RESULTS.md: real-ticket graded stage 2
- **Measured:** 3 real tickets, Claude-graded 5 axes; NInfer 112/150 (2/3 finished in cap), Strata 86/150 (1/3). Quick-tier tie broke only on real tasks.
- **Tag:** FINDING (tie-break by real tasks; preliminary, n=1 task set) + DATA (quality-bench T17).

### state/evals/2026-10-04/strata-0139-512k/RESULTS.md: engine-upgrade RAM pinning (already tabulated as vram-fit T7)
- **Measured:** 0.1.39 pinned 17.24→19.20 GiB, min free 3.36→1.41 GiB; headroom 6 restores 3.53 GiB.
- **Tag:** FINDING (already finding 31; numbers now also in SKILL.md Step 4 trap).

## state/evals/2026-10-10/2026-10-11 (promoted 2026-10-14)

### state/evals/2026-10-11/r17-strata-0143/result.md: Strata 0.1.43 vs 0.1.39-based deploy
- **Measured:** same 262K config, 3 runs per variant; agent-session decode +8.3%, prefill +6-8% at 16-32K, peak VRAM unchanged (23,778 vs 23,776 MiB). Deployed B.
- **Tag:** DATA (prefill-decode-speed T10) + FINDING (engine-version A/B, quality held).

### state/evals/2026-10-11/v3-switch/result.md: NInfer v3 vs v2
- **Measured:** prefill +1-2%, decode -0.8% (156.3 vs 157.5 t/s), acceptance 88.7% vs 86.7%, PPL +0.007% vs A16, greedy parity identical.
- **Tag:** DATA (prefill-decode-speed T11, speculative-mtp T7).

### state/evals/2026-10-11/r15-agentfast/result.md + fit.jsonl: template/sampler token efficiency + fit gate
- **Measured:** n=1 per variant; Sharp temp 0.6 same 10/12 in -18% tokens; presence 1.5 cuts MTP accept 89→84.2%, decode 308→254 t/s. Q4_K_XL ruled out by fit gate before tuning ran.
- **Tag:** DATA (speculative-mtp T7, vram-fit T8) — sampler rows preliminary, needs repeats.

### state/evals/2026-10-10/v3-agent/RESULT.md: chat-template effort trap
- **Measured:** v3 400 `invalid_prompt` because jinja refuses effort `high` (what Claude Code sends); fix maps high/max→xhigh, minimal→low.
- **Tag:** FINDING (effort sets differ per template/engine; SKILL.md Step 7 trap).

## state/evals/2026-10-14 (promoted 2026-10-14)

### state/evals/2026-10-14/gemma4-ab/side-*.log + repro JSON: Gemma4-26B-A4B-QAT first point
- **Measured:** NInfer engine, sm_89 native; 14.6 GiB weights, kv_capacity 131072 with kv_headroom_bytes=0; decode 330-400 t/s, MTP 72-86%, tool-call ok. Review 8-9 (preliminary, n=1 per arm).
- **Tag:** DATA (prefill-decode-speed T13, speculative-mtp T7, vram-fit T8, quality-bench T18). First non-Qwen family point.

## External verification sources (cited in `references/`)

These are web sources verified live (2026-10-08) and cited in the new reference files. None are measurements from this repo; all are externally sourced and must be verified on the user's own hardware.
Expand Down
13 changes: 10 additions & 3 deletions skill/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,9 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri
desktop compositor to the iGPU is a real lever on a shared card. **Re-measure host RAM
headroom after every engine upgrade**: defaults that pin memory can change: an upgrade grew
the default page-locked copy from 17.24 to 19.20 GiB and cut min free RAM at a 477K prompt
from 3.36 to 1.41 GiB on the same box and config, with no change in VRAM.
from 3.36 to 1.41 GiB on the same box and config, with no change in VRAM. A headroom
setting of 6 GiB (`STRATA_RESIDENT_HEADROOM_GIB`) restored 3.53 GiB and kept most of the
long-prompt gain. Engine-specific numbers.
- **Verify:** peak host RAM and peak VRAM per run, plus the engine's slot/pinned figures, before
and after any engine upgrade.

Expand Down Expand Up @@ -151,7 +153,9 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri
- **Evidence:** `references/tables/speculative-mtp.md` T1–T5; findings 2 (corrected), 11, 12, 13.
- **Trap:** the draft context is a separate allocation that can OOM even when the target model
fits. That is the measured fact. Do not claim it grows during generation; the measured runtime
growth is the compute buffers (finding 5). MTP compatibility depends
growth is the compute buffers (finding 5). Short-prompt calibration does not transfer to depth:
a calibrator's pick of spec-min-p 0.70 cost decode at 119K/238K and a profile learned from short
prompts cost ~20% at 119K; verify any calibrated spec setting at depth before promoting (finding 34). MTP compatibility depends
on exact builder/commit: flag names changed between builds (`--draft-max` removed →
`--spec-draft-n-max`), so pin versions. Auto-fit and manual offload are mutually exclusive:
`common_fit_params` aborts when `-ngl` is already set.
Expand Down Expand Up @@ -179,7 +183,10 @@ Run the steps in order. Each step: the rule, the evidence, the trap, how to veri
length cap and produced nothing reviewable).
- **Trap:** on models whose template implements a `reasoning_effort` switch, `medium` is a silent
mode that injects nothing; only the default (xhigh) injects reasoning instructions, and an
invalid value hard-fails the template render. `--jinja` is required for template-driven kwargs
invalid value hard-fails the template render. Effort value sets differ per template/engine: one
engine's jinja refused effort `high` (HTTP 400, client hung with no retry); the fix mapped
high/max to xhigh and minimal to low. Check the effort values against the template in use.
`--jinja` is required for template-driven kwargs
to work at all. None of this applies to a model that has no such switch.
- **Verify:** finish_reason per run. `finish=length` with no content is a budget failure, not a
model failure; several models scored 0/8 in round 1 for exactly that.
Expand Down
3 changes: 3 additions & 0 deletions skill/references/findings.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,6 @@ The most generalisable lessons from the evidence inventory. Each line is one les
30. Quality-per-second, not raw quality, should drive role choice. A blind-ranking tie was broken by 69 s vs 209 s at equal quality (the 280 s figure in the source is the later 27B-Research run, not the tied pair). (corrected, see `CORRECTIONS.md`; `state/evals/2026-09-25/think-ranking.md`, `state/evals/2026-09-25/r2.log`)
31. Re-measure host RAM headroom after every engine upgrade: defaults that pin memory can change. An upgrade that grew the default page-locked copy (17.24 -> 19.20 GiB) cut min free RAM at a 477K prompt from 3.36 to 1.41 GiB on a 31 GB box; a headroom setting (6 GiB) restored 3.53 GiB and kept most of the speed. Engine-specific numbers. (`state/evals/2026-10-04/strata-0139-512k/RESULTS.md`)
32. CPU offload trades speed for headroom. Measured: higher offload = more headroom, less speed (68–124 t/s across `--n-cpu-moe` settings on one MoE entry, vs 234–245 t/s fully on GPU). `--n-cpu-moe` is the correct overflow lever, never `-ngl`; offloading the draft layer trades the MTP win away. (`reference/LOCAL_MODELS.md`, `reference/llama-swap/TUNING-4090.md`)
33. Engine version is a tuning variable: A/B it with quality held constant. Strata #783 gained +14.2% decode at 119K and +9.5% at 238K with recall 4/4 and prefill unchanged (alternating arms, 2 rounds); Strata 0.1.43 gained +8.3% agent-session decode and +6-8% prefill at 16-32K at identical VRAM; NInfer v3 gained +1-2% prefill at -0.8% decode with greedy parity identical and PPL +0.007% vs A16. Engine-specific numbers. (`state/evals/2026-10-04/strata-pr783-depth/RESULTS.md`, `state/evals/2026-10-11/r17-strata-0143/result.md`, `state/evals/2026-10-11/v3-switch/result.md`)
34. Short-prompt calibration does not transfer to depth: verify at depth before promoting. A calibrator's short-prompt pick (spec-min-p 0.70) cost decode at 119K/238K, and a profile learned from short prompts cost ~20% at 119K; the deployed setting kept spec-min-p 0.5 with pcie-frac 0.20. Single-run depth check — re-verify with repeats. (`state/evals/2026-10-04/strata-calib/RESULTS.md`)
35. Report t/s AND quality: a quick-tier tie breaks only on real tasks. Strata decoded ~35% faster and read 200K 2.1x faster with review tied (12,10 vs 12,10,9, n=2-3) and richer NInfer one-shot output (~1.7x rules); graded real tickets then split them (NInfer 112/150 finishing 2/3 in cap, Strata 86/150 finishing 1/3). Preliminary (n=2-3 quick tier, n=1 task set); do not harden into a rule. (`state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`, `state/evals/2026-10-04/r15-real-tasks/RESULTS.md`)
62 changes: 62 additions & 0 deletions skill/references/tables/prefill-decode-speed.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,3 +143,65 @@ Run-to-run spread inside each arm is +-20 t/s, so short-prompt decode is a wash;
difference is the one that held across depths.

Source: `state/evals/2026-10-03/strata-ctx/RESULTS.md`, `state/evals/2026-10-03/strata-512k-live/RESULTS.md`, `state/evals/2026-10-04/strata-0139-512k/RESULTS.md`.

## Table 9: Strata #783 depth A/B, alternating engine-only change (IQ2_XS, live 262K q4_0, 2 rounds per arm, recall held)

| Build | 119K prefill t/s | 119K decode t/s | 238K prefill t/s | 238K decode t/s | Recall |
|---|---|---|---|---|---|
| base (v0.1.39 + #700) | 4048 / 4044 | 106.2 / 106.6 | 3898 / 3898 | 126.7 / 127.3 | 4/4 |
| + #783 | 4041 / 4031 | 121.8 / 121.5 | 3897 / 3889 | 138.9 / 139.3 | 4/4 |
| + #783 + #796 | 4046 / 4046 | 121.6 / 121.5 | 3896 / 3904 | 139.4 / 138.6 | 4/4 |

#783: decode at depth +14.2% at 119K, +9.5% at 238K; prompt reading unchanged. #796: no speed change (startup-safety value only). Engine-specific (Strata).

Source: `state/evals/2026-10-04/strata-pr783-depth/RESULTS.md`.

## Table 10: Strata 0.1.43 vs 0.1.39-based deploy, same 262K config (r17, 3 runs per variant, engine-specific)

| Variant | Chat decode | Agent-session decode | 16K prompt prefill | 32K prompt prefill | Peak VRAM |
|---|---|---|---|---|---|
| A: deploy (0.1.39-based) | 133.6 t/s | 144.7 t/s | 3,757 t/s | 3,760 t/s | 23,776 MiB |
| B: v0.1.43, same config | 134.8 (+0.9%) | 156.7 (+8.3%) | 3,978 (+5.9%) | 4,054 (+7.8%) | 23,778 MiB |
| C: v0.1.43, no --pcie-frac | 141.8 (+6.1%) | 152.3 (+5.3%) | 3,957 | 4,033 | 23,778 MiB |

B vs C within noise (C's measured PCIe share settled at 0.15-0.20, near the pinned 0.20). Deployed B. Engine-specific (Strata).

Source: `state/evals/2026-10-11/r17-strata-0143/result.md`.

## Table 11: NInfer v3 vs v2, same flags (INT8 prefill, rk4v4-e8 KV, MTP3, engine-specific)

| Metric | v2 (deploy 46645ada) | v3 (port 322a7f51) | v3/v2 |
|---|---|---|---|
| prefill 2048 | 3403.7 t/s | 3462.5 t/s | 1.02 |
| prefill 8192 | 3730.7 t/s | 3789.3 t/s | 1.02 |
| prefill 32768 | 3501.1 t/s | 3550.2 t/s | 1.01 |
| prefill 65536 | 3106.7 t/s | 3145.6 t/s | 1.01 |
| decode (code, greedy, median of 3) | 157.5 t/s | 156.3 t/s | 0.99 |

Parity identical (greedy); perplexity +0.007% vs A16 (4.343314 vs 4.343023); draft acceptance 88.7% vs 86.7%. Engine-specific (NInfer).

Source: `state/evals/2026-10-11/v3-switch/result.md`.

## Table 12: r14 head-to-head prefill at depth + decode (same box, production configs, n=2-3)

| Size | NInfer-Qwen3.8-27B prefill | Strata IQ2_XS prefill | NInfer decode | Strata decode |
|---|---|---|---|---|
| 60K (55,586 tok) | 3,037 t/s (18.3 s prefill; 23.3 s logged incl. answer) | 3,494 t/s (17.3 s) | 127-140 t/s | 173-185 t/s |
| 120K (117,153 tok) | 2,391 t/s (51.2 s) | 3,904 t/s (31.3 s) | | |
| 200K (~199.5K tok) | 1,857 t/s (110.2 s) | 3,833 t/s (53.7 s) | | |

Strata decodes ~35% faster and reads a 200K prompt 2.1x faster on this box. Needle recall 9/9 both (see quality-bench T16). Logged seconds are whole-request wall time (needle.py times the full chat call, answer generation included); prefill-only time is tokens / t/s - at 60K the NInfer engine log records TTFT 18.3 s vs total 23.3 s. Engine-specific.

Source: `state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`.

## Table 13: Gemma4-26B-A4B-QAT speeds (NInfer engine, sm_89 native, 2026-10-14, n=2 builds)

| Request | Prefill t/s | Decode t/s | MTP accept |
|---|---|---|---|
| build turn1 (prompt 619) | 4.28-4.29k | 330.8-357.4 | 72.0-77.2% |
| build turn1 tool-call (prompt 176, output 4.7-6.1K) | 2.88-2.90k | 391.4-399.1 | 84.2-86.1% |
| review probe (prompt 5.0-6.5K, output ~30) | 5.51-5.59k | 290.7-324.7 | 64.3-79.2% |

Weights 14.6 GiB; kv_capacity 131072 with kv_headroom_bytes=0; tool-call ok. First non-Qwen family point. Engine-specific (NInfer).

Source: `state/evals/2026-10-14/gemma4-ab/side-B.log`, `side-C.log`.
39 changes: 39 additions & 0 deletions skill/references/tables/quality-bench.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,42 @@ Source: role-suite results JSON at the `state/evals/` root, dated 2026-09-25 (th
The same models failed a real multi-turn build (see Table 4: Coder-Pruned 0/8; GAIN hit the length cap with no build; Mini skipped the tool call), the suite could not see it.

Source: `reference/llama-swap/config.yaml` (header comment, 2026-09-23).

## Table 16: r14 head-to-head quality (same harness, same prompts, n=2-3 per model — preliminary)

| Run | Model | Review /12 | Build CSS rules / props | Decode t/s |
|---|---|---|---|---|
| r14n-a | NInfer-Qwen3.8-27B | 12/12 | 181 / 379 | 140 |
| r14n-b | NInfer-Qwen3.8-27B | 10/12 | 190 / 405 | 127 |
| r14s-a | Strata IQ2_XS | 12/12 | 113 / 271 | 173 |
| r14s-b | Strata IQ2_XS | 10/12 | 108 / 241 | 182 |
| r14s-n | Strata IQ2_XS | 9/12 | 106 / 250 | 185 |

Review: tie (both 12 then 10; Strata third run 9/12 within ±2 noise). Needle recall 9/9 both to 200K. NInfer one-shot builds ~1.7x larger/richer (181-190 vs 106-113 rules); size is a proxy, not a judged score.

Source: `state/evals/2026-10-04/r14-ninfer-vs-strata/RESULTS.md`.

## Table 17: r15 real-task graded outcomes (3 real tickets, Claude-graded 0-10 x5 axes, same tools both engines)

| Task | Strata IQ2_XS | NInfer-Qwen3.8-27B |
|---|---|---|
| T1 homelab Python | 42 (done, committed, unreported at cap) | 35 (done, 72 min) |
| T2 Aphotic QML | 0 (nothing written at 150-min cap) | 34 (complete at 80 min, died before commit) |
| T3 llm-tune evidence | 44 (done, 120 min) | 43 (done, 62 min) |
| Total /150 | 86 | 112 |

Quick-tier tie (T16) broke only on real tasks: NInfer finished 2/3 within cap, Strata 1/3. n=1 task set — preliminary.

Source: `state/evals/2026-10-04/r15-real-tasks/RESULTS.md`.

## Table 18: Gemma4-26B-A4B-QAT review (planted-bug suite, NInfer engine, 2026-10-14 — preliminary)

| Run | Review found (raw count, NOT /12 — non-comparable to T16) | Tool-call |
|---|---|---|
| gemma4-review (earlier pass) | 8 | n/a |
| gemma4-ab sampler arm | 9 | ok (tool_calls finish) |
| gemma4-ab thinkmax arm | 8 | ok (tool_calls finish) |

First non-Qwen family quality point; raw counts are non-comparable to the /12 tables (T4–T17) — harness denominator unknown from the source as logged, do not compare across tables. n=1 per arm — preliminary.

Source: `state/evals/2026-10-14/gemma4-review/review2/Gemma4-26B-A4B-QAT.md`, `state/evals/2026-10-14/gemma4-ab/review2/*.md`.
17 changes: 17 additions & 0 deletions skill/references/tables/speculative-mtp.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,4 +118,21 @@ Source: `state/evals/2026-10-03/ninfer-sweep1/results.jsonl`.

Source: `state/evals/2026-10-03/strata-k8v4/sweep.jsonl` (`engine_mem_lines`).

## Table 7: post-v0.2 acceptance datapoints (engine-specific, raw)

| Source | Arm | Acceptance | Decode | Note |
|---|---|---|---|---|
| NInfer v3-switch | v2 | 86.7% | 157.5 t/s | greedy parity identical |
| NInfer v3-switch | v3 | 88.7% | 156.3 t/s (-0.8%) | prefill +1-2%, PPL +0.007% vs A16 |
| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | build turn1 | 72.0-77.2% | 330.8-357.4 t/s | weights 14.6 GiB |
| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | build tool-call turn | 84.2-86.1% | 391.4-399.1 t/s | tool-call ok |
| Gemma4-26B-A4B-QAT (NInfer, 2026-10-14) | review probe | 64.3-79.2% | 290.7-324.7 t/s | short output |
| r15-agentfast AF-base (n=1, preliminary) | temp 0.6, qwen3.6 template | 89.0% | 308 t/s | review 10/12 |
| r15-agentfast AF-sharp (n=1, preliminary) | temp 0.6, Sharp template | 88.5% | 309 t/s | same 10/12 in -18% tokens |
| r15-agentfast AF-sharp-general (n=1, preliminary) | temp 1.0, presence 1.5 | 84.2% | 254 t/s | build thinned (62 vs 116 rules), not for this entry |

Sampler/acceptance rows are n=1 — preliminary, needs repeats before promotion.

Source: `state/evals/2026-10-11/v3-switch/result.md`, `state/evals/2026-10-14/gemma4-ab/side-*.log`, `state/evals/2026-10-11/r15-agentfast/result.md`.

Reference datapoint (model card, not measured here): Q4_K_S regular ~75 t/s vs MTP >90 t/s at 60% acceptance, 2 tokens, roughly +20% at 60% acceptance; the ratio, not the absolute t/s, is the transferable part. Gate: below ~50% acceptance, regular quants run faster. Source: `state/evals/2026-09-25/gain-research.md` (Note 3), `runbooks/llama-swap.md` (bench gate >= 0.4).
10 changes: 10 additions & 0 deletions skill/references/tables/vram-fit.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,3 +127,13 @@ Headroom 6 restores 3.53 GiB and keeps most of the long-prompt gain (-12% read a
default, +10% decode at depth). Recall held at all four depths on every arm.

Source: `state/evals/2026-10-04/strata-0139-512k/RESULTS.md`.

## Table 8: post-v0.2 fit datapoints (raw, engine-specific)

| Entry | Weights / peak | ctx | Result |
|---|---|---|---|
| Gemma4-26B-A4B-QAT (NInfer, sm_89 native) | 14.6 GiB weights | 131072 | loads; kv_capacity 131072 with kv_headroom_bytes=0 (zero headroom); tool-call ok |
| Qwen3.6-35B-A3B UD-Q4_K_XL (llama.cpp, r15-agentfast fit probe) | load failed at ctx 200000/131072/65536+ with n_cpu_moe 0 | 131072+ | ruled out by the fit gate before tuning ran; UD-IQ4_XS (peak 22,572 MiB) used instead |
| Qwen3.6-35B-A3B UD-Q4_K_XL | peak 23,924 MiB at ub1024 / 23,782 at ub512 | default probe ctx (no ctx field logged) | loads only with n_cpu_moe 6-7 in the probe ladder; still ruled out for the run (fit-fragile) |

Source: `state/evals/2026-10-14/gemma4-ab/side-*.log`, `state/evals/2026-10-11/r15-agentfast/fit.jsonl`.
Loading