Skip to content

Promote 8 proven post-v0.2 deltas (tables, findings, traps) - #1

Merged
T-Crypt merged 3 commits into
mainfrom
review/2026-10-14-llm-tune-proven-deltas
Oct 14, 2026
Merged

T-Crypt merged 3 commits into
mainfrom
review/2026-10-14-llm-tune-proven-deltas

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Oct 14, 2026

Copy link
Copy Markdown
Owner

What changed

Promotes only the post-v0.2 homelab results proven with repeats or quality-held-constant (main @ a02634f, v0.2 2026-10-08) into llm-tune's tables, findings, and traps. 7 files, +187/-3, SKILL.md steps still 1-9, no renumbering.

8 deltas promoted

  1. Engine version is a tuning variable (alternating-arm A/B, quality held): Strata #783 +14.2% decode at 119K, +9.5% at 238K, recall 4/4; 0.1.43 agent-session +8.3%, prefill +6-8% at 16-32K; NInfer v3 prefill +1-2%, decode -0.8%, PPL +0.007% vs A16.
  2. RAM-pinning trap numbers: 0.1.39 pinned 17.24->19.20 GiB, min free 3.36->1.41 GiB; headroom 6 restores 3.53 GiB (SKILL.md Step 4).
  3. Short-prompt calibration vs depth (new finding 34): spec-min-p 0.70 costs decode at depth; short-prompt profile -20% at 119K; live is pcie-frac 0.20 + spec-min-p 0.5.
  4. r14 head-to-head (finding 35): Strata ~35% faster decode, 2.1x faster 200K read, NInfer 1.7x richer output; review tie, recall 9/9 both. Report t/s AND quality.
  5. Template/sampler token efficiency (n=1, preliminary): Sharp temp 0.6 same 10/12 in -18% tokens; presence 1.5 cuts MTP accept 89->84.2%, decode 308->254.
  6. Chat-template effort trap (Step 7): v3 400s because jinja refuses effort high; fix maps high/max->xhigh, minimal->low.
  7. Gemma4-26B-A4B-QAT DATA (2026-10-14, sm_89 native): 14.6 GiB weights, kv_headroom 0 @131072, decode 330-400 t/s, MTP 72-86%, tool-call ok. First non-Qwen family point.
  8. Q4_K_XL fit-gate rejection: ruled out by fit gate before tuning ran (vram-fit T8).

Files (7)

  • skill/references/tables/prefill-decode-speed.md (T9-T13)
  • skill/references/tables/quality-bench.md (T16-T18)
  • skill/references/tables/speculative-mtp.md (T7)
  • skill/references/tables/vram-fit.md (T8)
  • skill/references/findings.md (findings 33-35)
  • skill/SKILL.md (traps only: Steps 4, 5/6, 7)
  • data/INVENTORY.md (dated entries per source)

What the stages proved

  • Tests: python -m py_compile skill/bench/*.py -> OK (all 3 scripts compile). pytest tests/ has no runnable tests (both subdirs are data: scenario answers, bench jsonl/txt) — nothing to break.
  • Reviews clean: review.md found 2 minor nits (finding 34 untraced parenthetical; T18 missing denominator) + 2 cosmetic notes; review2.md verifies both nits FIXED (parenthetical removed, T18 header now raw-count non-comparable to T16). Table numbering sequential (prefill T1-T13, quality T1-T18, mtp T1-T7, vram T1-T8), all SKILL.md reference links resolve, CORRECTIONS.md untouched, engine tags on every new row, small-n items labeled preliminary.
  • Arithmetic spot-checks (review stage, recomputed): T9 +14.3%/+9.5%, T10 +8.3%/+5.9%/+7.8%, T12 200K ~2.05x, v3 PPL +0.0067% — all match claims.
  • Limitation (stated plainly): state/evals/ sources live in homelab, not this repo — figures checked for internal consistency + INVENTORY match, not against source logs.

Deliberately NOT promoted

  • ninfer-pr15-norm/: kernel norm internals; no skill-level lesson identified.
  • router-fast-tier/results.json, omp-pr2-ab.jsonl: no verdict read; needs a RESULTS.md first.
  • r16-agentfast-parallel/: script only, never ran (no par.jsonl). Method reuse only.

Note on .router/

.router/2026-10-14-llm-tune-review/ (plan.md, review.md, review2.md) is untracked plan material and is NOT part of this PR. Repo convention treats planning docs as untracked (see commits ignoring docs/dev/), so it stays out unless the owner says to commit it.

Never merge without owner review — small-n items (r15 n=1, r14 n=2-3) are labeled preliminary and must not harden into rules without repeats.

- Engine-version A/B: Strata #783 (+14.2% decode at 119K), 0.1.43
  (+8.3% agent-session), NInfer v3 (prefill +1-2%, decode -0.8%)
- RAM-pinning trap numbers (0.1.39: 17.24->19.20 GiB pinned)
- Short-calibration vs depth caveat (spec-min-p 0.70, -20% at 119K)
- r14 head-to-head: Strata ~35% faster decode, tie on quality
- Sampler token efficiency (n=1, preliminary) + Q4_K_XL fit rejection
- Effort-value trap (high/max->xhigh) in Step 7
- Gemma4-26B-A4B-QAT first non-Qwen DATA point
- Findings 33-35; SKILL.md traps only, steps still 1-9
@T-Crypt
T-Crypt marked this pull request as ready for review October 14, 2026 15:39
@T-Crypt

T-Crypt commented Oct 14, 2026

Copy link
Copy Markdown
Owner Author

Independent review, 2026-10-14 (fresh pass; verdict: merge-with-nits)

Safety fact: the diff is exactly the 7 claimed .md files, zero non-md files, zero table-row deletions, SKILL.md steps still 1-9. Merging changes no executable behavior; the only risk channel is the numbers agents read, and the internal-consistency checks bound it.

Findings and their disposition:

  1. FIXED this pass. T12 60K NInfer cell said "3,037 t/s (23.3 s)" but 55,586 / 3,037 = 18.3 s. Resolved against the run's own engine log (side-NInfer-Qwen3.8-27B.log, req#1): TTFT 18.3 s, total 23.3 s, prefill 3.04k tok/s. The logged seconds are whole-request wall time (answer generation included); the cell now says so and a note explains the semantics for the table.
  2. FIXED this pass. README layout table said "lessons (32)"; findings.md now has 35. Updated to the counted value.
  3. Noted, left as-is. T9 claims +14.2% at 119K; row values recompute to +14.33%. Within 0.1pp, consistent with 0.1 t/s input rounding; no rule derives from it.
  4. Open nit. quality-bench T18 "gemma4-review (earlier pass)" row has no data/INVENTORY.md entry.
  5. Open nit. INVENTORY.md section header lumps two dates.

Residual risk, stated plainly: source logs live in the operator's homelab, not this repo. Figures were verified for internal consistency and cross-checked against the engine log where it was in scope (T12), not re-derived from raw logs wholesale. Small-n rows (r15 n=1, r14 n=2-3) remain labeled preliminary and must not harden into rules without repeats.

@T-Crypt
T-Crypt merged commit 8c26fa9 into main Oct 14, 2026
2 checks passed
@T-Crypt
T-Crypt deleted the review/2026-10-14-llm-tune-proven-deltas branch October 14, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant