feat(BACKEND-TENSTORRENT): tt_clock_state sampler/judge + busy-slice refold (#2005); #2003 re-adjudicated clock-attributed

.agents/benchmark-record.md

@@ -28751,3 +28751,79 @@ the repository: `docs/bench-evidence/tower-skip-rss-qwen3vl-thor-20260824.log`
 (the harness report verbatim) and
 `docs/bench-evidence/tower-skip-rss-qwen3vl-thor-20260824.legs.log` (five
 `/usr/bin/time -v` records, four server logs, the cmake configure).
+
+## TT P150 #2003 RE-ADJUDICATED CLOCK-ATTRIBUTED: the inversion stands at governor parity — every busy sample of both arms at the 1350 MHz cap; tt_clock_state lands as the TT sibling of gpu_clock_state (#2005) (2026-08-26, `bench/tt-clock-state`, P150 `thalia`)
+
+Same binary (`21fe11cf1` bench build), workload, order-alternation, and lock
+discipline as the same-day unattributed entry above, PLUS a
+`tools/bench/tt_clock_state.py sample --leg-pid <pid>` window per arm
+(interval 1 s, duration 75 s, 53 samples each). Evidence:
+[`../docs/bench-evidence/tt-p150-clock-attributed-20260826.log`](../docs/bench-evidence/tt-p150-clock-attributed-20260826.log)
+(raw windows, judge verdicts, busy census).
+
+**Throughput (12 warm samples per arm):** default host-free eager
+**median 10.880** tok/s (mean 10.838) vs `VT_TT_HOST_FREE_DECODE=0`
+**median 13.645** (mean 13.520) — **ratio 1.254 median / 1.247 mean**,
+confirming the morning's 10.822-vs-13.369 unattributed result.
+
+**The clock finding that upgrades attribution.** The Blackhole P150 AICLK
+governor is TWO-STATE on this telemetry: 800 idle, pegged 1350 under load.
+Raw windows contain an idle pre-open head, so all six refuse within-run
+spread (40.74%) — correctly under rules written for quasi-continuous clocks.
+But the busy column was recorded LIVE per interval from pid-held
+/dev/tenstorrent fds, a criterion independent of outcome values: refolding
+to busy-only slices gives **six windows whose distinct AICLK set is exactly
+{1350} — min=median=max, spread 0.00%, cross-arm offsets 0/0% — and the
+pair judge returns PASS with zero reasons**
+(`tools/bench/tt_refold_busy.py`, rc=0). Both arms ran compute at identical,
+cap-pegged clocks. The inverted ratio is real at clock parity, not a clock
+excursion: **#2003 stays open as a genuine hybrid-path performance win owed
+a per-op delta explanation**, with the polarity question sharper than ever.
+
+Knowns recorded deliberately: claimed-max 1350 carries provenance
+"UNVERIFIED pin owed" in every summary; raw-window spread refusals are kept
+in the evidence log rather than hidden; the throttle-unobservability caveat
+ships inside every judged window until TT exposes a live bitmap.
+
+## Qwen3.5-0.8B on TT P150 — FIRST speed numbers: warm decode ~0.089 tok/s (~11.2 s/token), both host-free arms identical; the hybrid-GDN family runs ~100x slower than a 7B dense on the SAME board and the wall is neither polarity nor kernel-mode (2026-08-26, `bench/tt-clock-state`, P150 `thalia`)
+
+Row [`BACKEND-TENSTORRENT-QWEN35`](https://github.com/mudler/vllm.cpp/issues/1715)
+context, base `21fe11cf1`, production entry point (`vllm-cli --device auto`
+→ W2a allow-list dispatch, tt_cluster UMD lines in evidence). Every leg a
+fresh process under one `$HOME/gpu.lock`; probes follow a JIT-caching cold
+proc (110.9 s for 8 tokens, discarded). Evidence:
+[`../docs/bench-evidence/tt-qwen35-first-speed-20260826.log`](../docs/bench-evidence/tt-qwen35-first-speed-20260826.log).
+
+**Numbers (greedy b1):** default arm 12-token runs **0.080 / 0.089 / 0.089
+tok/s** — warm steady ≈ 11.2 s/token, reproduced exactly; opt-out arm
+(`VT_TT_HOST_FREE_DECODE=0`) **0.080 / 0.090 / 0.089 — IDENTICAL**, so the
+#2003/#1604 polarity question does not even arise for this family. One
+96-token generation at 0.084 tok/s (1148 s) is consistent with steady state,
+not a JIT artifact. Output coherent ("The hum of the machine's internal …"),
+finish_reason=length everywhere.
+
+**The finding.** Same board, same day: Mistral-7B bf16 decodes at 9.817
+tok/s and Qwen3-0.6B at 10.8–13.6; Qwen3.5-**0.8B** decodes at 0.089 —
+roughly **two orders of magnitude below a model 9x its size** running the
+identical backend build. The W2a paged-engine gate proves mixed-execution
+CORRECTNESS for this family (golden pairs committed), so device work is
+happening somewhere; nothing until today measured how much of the step runs
+off-device. Ranked hypotheses for the wall, unattributed by intent: (1) a
+per-layer host/reference fallback inside the hybrid-GDN stack dominates each
+step — plausible candidates are reference-matmul projections or attention
+paths lacking kTENSTORRENT providers, since 11.2 s/token over ~28 layers
+≈ 400 ms/layer-token, microseconds-scale per GDN kernel; (2) the host-staged
+KV path of `kPagedAttention`/`kReshapeAndCache` (`tenstorrent_ops.cpp:14`)
+paying per-step upload/download — but that lane carries Mistral to 9.8 tok/s,
+so it alone cannot explain 3 orders; (3) NOT the host-free polarity (A/B
+identical above) and NOT GDN kernel mode (composed vs chunked differ by
+milliseconds against an 11,000 ms step).
+
+Clock attribution: these figures are UNATTRIBUTED (no sampler window on the
+probes); given ~100% duty cycles over minutes-long steps both arms were near
+certainly cap-pegged throughout, which only sharpens the two-order gap.
+
+Owns the next lever: profile ONE eager decode step of this family on TT and
+name the dominant op before anyone registers more kernels blindly
+([#1715](https://github.com/mudler/vllm.cpp/issues/1715); captured tracing
+stays blocked behind #1625, so the first pass is eager-side).