diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 8e241c427..7b1ae27fb 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -28751,3 +28751,79 @@ the repository: `docs/bench-evidence/tower-skip-rss-qwen3vl-thor-20260824.log` (the harness report verbatim) and `docs/bench-evidence/tower-skip-rss-qwen3vl-thor-20260824.legs.log` (five `/usr/bin/time -v` records, four server logs, the cmake configure). + +## TT P150 #2003 RE-ADJUDICATED CLOCK-ATTRIBUTED: the inversion stands at governor parity — every busy sample of both arms at the 1350 MHz cap; tt_clock_state lands as the TT sibling of gpu_clock_state (#2005) (2026-08-26, `bench/tt-clock-state`, P150 `thalia`) + +Same binary (`21fe11cf1` bench build), workload, order-alternation, and lock +discipline as the same-day unattributed entry above, PLUS a +`tools/bench/tt_clock_state.py sample --leg-pid ` window per arm +(interval 1 s, duration 75 s, 53 samples each). Evidence: +[`../docs/bench-evidence/tt-p150-clock-attributed-20260826.log`](../docs/bench-evidence/tt-p150-clock-attributed-20260826.log) +(raw windows, judge verdicts, busy census). + +**Throughput (12 warm samples per arm):** default host-free eager +**median 10.880** tok/s (mean 10.838) vs `VT_TT_HOST_FREE_DECODE=0` +**median 13.645** (mean 13.520) — **ratio 1.254 median / 1.247 mean**, +confirming the morning's 10.822-vs-13.369 unattributed result. + +**The clock finding that upgrades attribution.** The Blackhole P150 AICLK +governor is TWO-STATE on this telemetry: 800 idle, pegged 1350 under load. +Raw windows contain an idle pre-open head, so all six refuse within-run +spread (40.74%) — correctly under rules written for quasi-continuous clocks. +But the busy column was recorded LIVE per interval from pid-held +/dev/tenstorrent fds, a criterion independent of outcome values: refolding +to busy-only slices gives **six windows whose distinct AICLK set is exactly +{1350} — min=median=max, spread 0.00%, cross-arm offsets 0/0% — and the +pair judge returns PASS with zero reasons** +(`tools/bench/tt_refold_busy.py`, rc=0). Both arms ran compute at identical, +cap-pegged clocks. The inverted ratio is real at clock parity, not a clock +excursion: **#2003 stays open as a genuine hybrid-path performance win owed +a per-op delta explanation**, with the polarity question sharper than ever. + +Knowns recorded deliberately: claimed-max 1350 carries provenance +"UNVERIFIED pin owed" in every summary; raw-window spread refusals are kept +in the evidence log rather than hidden; the throttle-unobservability caveat +ships inside every judged window until TT exposes a live bitmap. + +## Qwen3.5-0.8B on TT P150 — FIRST speed numbers: warm decode ~0.089 tok/s (~11.2 s/token), both host-free arms identical; the hybrid-GDN family runs ~100x slower than a 7B dense on the SAME board and the wall is neither polarity nor kernel-mode (2026-08-26, `bench/tt-clock-state`, P150 `thalia`) + +Row [`BACKEND-TENSTORRENT-QWEN35`](https://github.com/mudler/vllm.cpp/issues/1715) +context, base `21fe11cf1`, production entry point (`vllm-cli --device auto` +→ W2a allow-list dispatch, tt_cluster UMD lines in evidence). Every leg a +fresh process under one `$HOME/gpu.lock`; probes follow a JIT-caching cold +proc (110.9 s for 8 tokens, discarded). Evidence: +[`../docs/bench-evidence/tt-qwen35-first-speed-20260826.log`](../docs/bench-evidence/tt-qwen35-first-speed-20260826.log). + +**Numbers (greedy b1):** default arm 12-token runs **0.080 / 0.089 / 0.089 +tok/s** — warm steady ≈ 11.2 s/token, reproduced exactly; opt-out arm +(`VT_TT_HOST_FREE_DECODE=0`) **0.080 / 0.090 / 0.089 — IDENTICAL**, so the +#2003/#1604 polarity question does not even arise for this family. One +96-token generation at 0.084 tok/s (1148 s) is consistent with steady state, +not a JIT artifact. Output coherent ("The hum of the machine's internal …"), +finish_reason=length everywhere. + +**The finding.** Same board, same day: Mistral-7B bf16 decodes at 9.817 +tok/s and Qwen3-0.6B at 10.8–13.6; Qwen3.5-**0.8B** decodes at 0.089 — +roughly **two orders of magnitude below a model 9x its size** running the +identical backend build. The W2a paged-engine gate proves mixed-execution +CORRECTNESS for this family (golden pairs committed), so device work is +happening somewhere; nothing until today measured how much of the step runs +off-device. Ranked hypotheses for the wall, unattributed by intent: (1) a +per-layer host/reference fallback inside the hybrid-GDN stack dominates each +step — plausible candidates are reference-matmul projections or attention +paths lacking kTENSTORRENT providers, since 11.2 s/token over ~28 layers +≈ 400 ms/layer-token, microseconds-scale per GDN kernel; (2) the host-staged +KV path of `kPagedAttention`/`kReshapeAndCache` (`tenstorrent_ops.cpp:14`) +paying per-step upload/download — but that lane carries Mistral to 9.8 tok/s, +so it alone cannot explain 3 orders; (3) NOT the host-free polarity (A/B +identical above) and NOT GDN kernel mode (composed vs chunked differ by +milliseconds against an 11,000 ms step). + +Clock attribution: these figures are UNATTRIBUTED (no sampler window on the +probes); given ~100% duty cycles over minutes-long steps both arms were near +certainly cap-pegged throughout, which only sharpens the two-order gap. + +Owns the next lever: profile ONE eager decode step of this family on TT and +name the dominant op before anyone registers more kernels blindly +([#1715](https://github.com/mudler/vllm.cpp/issues/1715); captured tracing +stays blocked behind #1625, so the first pass is eager-side). diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 73af04bf6..fc8f77087 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -746,3 +746,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1934](https://github.com/mudler/vllm.cpp/issues/1934) | `BACKEND-ROCM` | `RocmPlatform::needs_weight_staging()` is stale-false (a W0-era placeholder never revisited despite #523/#509/#506/ROCM_ATTN/hipGraph landing since), so `CheckDeviceWeightFit` — the #1123/#1870 load-time refusal, including the `policy_forces_full_expand` fix — never runs on ROCm: measured directly, `VT_DEVICE_WEIGHT_BUDGET_BYTES=1` produced no refusal on a real load. The actual device allocation the refusal guards is not gated on this flag, so #1870's crash stays reachable until this closes; owed, not fixed in flow, because flipping the flag also moves `DirectDeviceLoadEligible` and several GDN kernel-dispatch defaults that each need their own correctness check | bug | | [#1978](https://github.com/mudler/vllm.cpp/issues/1978) | `MODEL-MM-QWEN4-EXP` | **`Qwen/Qwen3.8-Flash-Next` declares `Qwen4ExpForConditionalGeneration` / `qwen4_exp`, a new architecture vLLM does not implement, so the port runs on a split oracle: transformers for the ALGORITHM, vLLM ops for the OPTIMIZED PATH.** Released 2026-08-24, 180B total / 6B activated, image-text-to-text. The `Qwen3.8` in the name is marketing continuity: `.agents/specs/qwen38-27b-bf16-gate.md`'s "one config key differs" precedent does NOT extend here. Read live 2026-08-26 at vLLM `origin/main` = `6a5e8f5979`: no `qwen4*` path, no registry entry, and a repository-wide GitHub search for `qwen4` returns ZERO results; `vllm-omni` likewise. That is absence from vLLM `main` rather than staleness in our pin `555967922`, so a pin advance does not reach it. What exists is transformers [#48337](https://github.com/huggingface/transformers/pull/48337) "Add Qwen4Exp model", MERGED 2026-08-26, 5211 lines, and SGLang [#36497](https://github.com/sgl-project/sglang/pull/36497), still OPEN and therefore inadmissible. **Developer direction 2026-08-26, recorded verbatim: "use transformers as oracle for algorithmic side. but use ops from vllm so we account for optimized path."** Justified rather than convenient: `Qwen4ExpTextQSAIndexer.forward` loops in Python over `(batch_idx, query_idx)` and says "we only allow eager and sdpa", so porting it as written yields a correct model at an indefensible speed, while AGENTS.md's mirror-vLLM polarity still binds every primitive vLLM implements. `Qwen4ExpTextModel` inherits from `Qwen3_5MoeTextModel` and leaves rotary, MLP, experts, TopK router and the ENTIRE vision tower unchanged (`class Qwen4ExpVisionModel(Qwen3_5MoeVisionModel): pass`), all of which this tree has; GDN is an exact match for our AOT gate (`K=V=128, Hg=16, Hv=48` against `src/vt/cuda/cuda_gdn.cu`'s `H in {48,32}`). The delta is four things, and **exactly two have no vLLM op at all**: the PLE dilated depthwise conv (kernel 4, dilation 3; `git grep dilation` over vLLM `layers/mamba/` = 0 hits) and the n-gram hashed embedding. **The survey's load-bearing finding, and it REVERSES this row's first reading: QSA's structural twin is DeepSeek-V4's C4 indexer lane, NOT MiniMax-M3.** The original call was that QSA, being plain GQA rather than MLA, had to map onto vLLM's non-MLA block-sparse case; that reasoning rested on treating `MLAAttentionSpec` as an MLA claim, and **it is not one** — M3's own indexer cache uses it while M3 is a plain-GQA model, with the comment "Key-only: MLAAttentionSpec budgets one vector/token (not 2x for K+V)". It is a budget shape. Remove that prop and the GQA-vs-MLA argument collapses. Verified line by line at `6a5e8f5979`: **nine independent structural matches with DSv4**, `compress_ratio == 4` literally the same number — MQA index with 1 key head at dim 128; `relu(q.k)` summed over index heads vs `(score.relu() * weights).sum(dim=0)`; `1/sqrt(head_dim)`; one score set per query token with no head axis vs `topk_indices_buffer[num_tokens, topk]`; pooling boundary `(position+1) % COMPRESS_RATIO == 0`; RMSNorm on the pooled key; **RoPE at the block-start position** vs `compressed_pos = (position // CR) * CR`; candidate count `visible // compress_ratio`; and one stored state per 4 tokens via `MLAAttentionSpec(tokens_per_state=compress_ratio)`, a first-class KV field documented as "Ints > 1 compress multiple tokens into one state (DSv4 sparse MLA)" that has no M3 equivalent. **M3 is a DIFFERENT ALGORITHM**, not a worse fit: its score is `tl.max(qk, axis=1)` over 128 RAW token dots with no pooling, no relu and no head reduction, it asserts `num_idx_heads == num_kv_heads` ("no topk index reduce") so it emits one block set PER KV HEAD, and its `SPARSE_BLOCK_SIZE = 128` is welded to the KV page size ("One sparse block == one KV page") on both the score and the attend side — moving it to 4 forces a page size of 4 and breaks `tl.dot`, whose tile needs >= 16. M3 contributes exactly ONE thing and it is a wiring precedent, not an algorithm: that a plain-GQA model can own a key-only side cache through `MLAAttentionSpec` and a private indexer backend. **The genuinely new work is the CONSUMER and nothing upstream supplies it** — every DSv4 sparse consumer attends to COMPRESSED MLA KV (one state per 4 tokens) and M3's attend to raw tokens only at page granularity, while QSA attends to RAW tokens selected at ratio-4 granularity. Two silent-failure traps follow: wiring QSA's top-k into a DSv4 sparse-MLA consumer attends a POOLED key/value and still emits plausible tokens, and **a short-prompt token gate cannot catch it because at context <= `indexer_budget` 2048 every candidate is selected** — so any QSA gate must run past 2048 tokens of context, which is now a stated `## Gates` requirement; and `SparseAttnCompressNormRopeStoreC4Kernel` does NOT mean-pool despite its name — it is a learned softmax pool over an OVERLAPPING window of 8 using a score channel this checkpoint does not have, and the CuteDSL variant refuses `overlap=False` at compile, so the **Triton** `head_dim=128` variant is the correct starting point. Two structural consequences beyond the module list: the residual stream is `hc_count * hidden_size` = **4 x 2560 = 10240 wide through the whole stack** with a low-rank read gate and per-branch scalar write gate around both attention and MLP, which is a change to the per-layer loop and every residual buffer rather than a drop-in module; and `number_of_conv_states = 3` on a PLE layer (GDN conv, PLE conv, n-gram token history) plus the indexer side cache, adjacent to [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966). **NOTHING PUBLISHED FITS**, read live from the HF API against ~119 GB usable on GB10: BF16 ~360 GB, official FP8 ~180 GB, `RadixArk/...-NVFP4` ~128 GB (NVFP4 backbone with the n-gram table left at FP8, 51.2 GB) and `unsloth/...-GGUF` is a README with ZERO weight files. No GGUF exists and no tool can make one, because llama.cpp has no `qwen4_exp` either, so the standing k-quant requirement means authoring the arch on our side AND states that the quantized arms have NO llama.cpp oracle. **The chosen arm does NOT load today, and the blocker is neither the offload nor the budget: this tree cannot keep a gather table quantized at all.** `KeepQuantKDim` returns `-1` for `GgufTensorRole::kEmbeddingTable` (`src/vllm/model_executor/model_loader/gguf_keep_quant.cpp`), and `qwen3_5_gguf_weights.cpp` asserts it by name — "the embedding table cannot keep quant blocks" — so a Q4_K or Q8_0 n-gram table EXPANDS to bf16 and 51.2B params become **102.4 GB of anonymous memory**; the arm dies before the first forward. The reason was already sitting in a header comment ("a gather, not a GEMM ... A quantized-gather op is a follow-up row") and **no such row exists**. The only non-expanding gather residency is `kKeepF16`, requiring ggml type 1 exactly (102.4 GB on disk) and CPU-ONLY, because `EmbeddingKernelCuda` refuses anything but f32/bf16. **Second blocker:** `moe_intermediate_size = 640` makes `ffn_down_exps` Q4_K-illegal on its reduction dim (640 % 256 = 128), as does `hc_lowrank = 320`; llama.cpp's substitution is believed to be Q5_0 (**UNVERIFIED, owed against the pinned llama.cpp oracle**) and the dependent fact IS verified in-tree — our reader knows ggml ids `0,1,2,8,10..14,16,18,19,22..28,30,39,40,41,66` and has **no entry for 3, 6, 7 or 20**, so a stock `llama-quantize -Q4_K_M` file fails at header parse. We author the converter, so the fix is Q4_0 (block 32, same 4.5 bpw). **`ENG-WEIGHT-OFFLOAD` will not help** — it moves zero bytes today (`ConsiderWeight` has no production callers, pinned by a test) and is documented inert on GB10; the tier that DOES work already ships and is proven by the 2.4T model serving 369.97 GiB from a 119.631 GiB box at ~62 GiB resident: mmap `MAP_PRIVATE`, borrow in place, alias the host pointer, `prefault: false`. Corrected sizing: backbone ~67.7 GiB, whole process ~73.5 GiB of 119.631 at 32K single-stream, ~46 GiB of headroom for the page cache, so the ~76 GB estimate was right within 10%. The design works because per-token demand is **<= 64 KiB of reads** (16 lookups x 160 dims over at most 16 pages) against the 2.4T expert lane's 6.95 GB/token. The architecture supplies its own lever: the per-token n-gram cost is `(ngram_size-1)*heads_per_ngram` = 16 lookups of 160 dims, so **51 GB of the 180 GB, 28% of the model, is a table touched 16 times per token** and making it non-resident is the intended design point (RadixArk reached the same split independently). Sizing arithmetic, NOT measurement: Q8_0 throughout ~191 GB (no), Q4_K_M throughout ~109 GB (yes, ~10 GB left for KV), Q4_K_M backbone with the table non-resident ~76 GB. GB10 is UNIFIED memory so "offload to host" is not a move there; non-resident means disk-backed, and its cost is unmeasured. **Two decisions were put to the developer as explicit accept-or-reject and BOTH are settled 2026-08-26, recorded in place rather than left open.** (1) `.agents/oracles/transformers.md` pins transformers to 5.14.1, deliberately tied to what the pinned vLLM environment resolves so the environment cannot hold two `transformers` at once, and **5.14.1 does not contain `Qwen4Exp`**; the lane-scoped second pin is **ACCEPTED**, on the argument that the invariant guards a vLLM environment against drifting from its transformers and here there is no vLLM implementation to drift from, and it expires the moment vLLM registers `qwen4_exp`. **The lane pin is a real release, not a branch SHA**, which was not the expected outcome: `Qwen4Exp` merged to `main` at 12:03:40Z on 2026-08-26 and `v5.16.0` published at 12:35:15Z, and this was BOUNDED rather than assumed by fetching `models/qwen4_exp/modeling_qwen4_exp.py` at each tag — `v5.16.0` HTTP **200**, `v5.15.0` HTTP **404** — making 5.16.0 the FIRST release carrying the architecture and therefore the tightest available pin. The version string is UNMEASURED (it is the release proven to contain the model, not a `transformers.__version__` read off a running oracle) and `gateable` stays `no`. (2) The first runnable arm is the **Q4_K_M backbone with the n-gram table NON-RESIDENT** (~76 GB). Q8_0 was raised and does not fit at ~191 GB, and no partial-Q8 split reaches 119 GB with the backbone at 8 bits; Q4_K_M-throughout fits on paper at ~109 GB but leaves ~10 GB for KV and activations on a 262144-native-context model, which is not a margin. This promotes the non-resident table from a note to a first-class W6 deliverable, and it is NOT free: GB10 is UNIFIED memory, so the existing host-pinned offload seam (`ENG-WEIGHT-OFFLOAD`, mirroring vLLM's `cpu_offload_gb`) does not by itself solve it there and the mechanism must be disk-backed or genuinely unloaded — established before it is designed around. Spec: [`specs/qwen4-exp-flash-next.md`](specs/qwen4-exp-flash-next.md). No product code lands under the spec pull request | feature | | [#1983](https://github.com/mudler/vllm.cpp/issues/1983) | `KV-GDN-STATE-BUDGET` | **The GDN recurrent-state pool is preallocated per CONFIGURED sequence, on an axis no flag bounds.** `GPUModelRunner::initialize_kv_cache` sizes `gdn_state_slots_ = max_num_reqs * (num_spec + 1)` and allocates one conv and one SSM buffer per GDN layer from it, each `Memset` to zero at construction, so every byte is resident before the first request. Re-derived for `Qwen3.8-27B` (48 linear-attention layers, `Hk/Hv/Dk/Dv/conv = 16/48/128/128/4`, `mamba_ssm_dtype = float32`) at `num_speculative_tokens = 8`: one slot costs 3,371,008 B per layer, 154.31 MiB across 48 layers, so one sequence costs 1.356 GiB and `--max-num-seqs 32` costs **43.40 GiB** that `--kv-cache-memory`, `--num-blocks` and `--gpu-memory-utilization` all fail to bound. The per-sequence cost is NOT the divergence — upstream charges the same `1 + num_speculative_blocks` state blocks (`vllm/v1/kv_cache_interface.py::MambaSpec.max_memory_usage_bytes`) and our `f32` SSM mirrors the checkpoint's own `mamba_ssm_dtype` — the AXIS is: `max_num_seqs` sizes no allocation anywhere in vLLM. Upstream raises the attention block size until one attention page holds one mamba page (`vllm/platforms/interface.py::Platform.check_and_update_config`), pads the mamba page to match, and then draws BOTH from one budgeted pool whose tensors are `shared_by` one layer from each group (`kv_cache_utils.py::_get_kv_cache_config_uniform_page_size`), so its recurrent allocation is a function of available memory and never of the concurrency cap. Fixed by mirroring that arithmetic in `ComputeHybridKvBudget` — `unified_block_tokens = align * cdiv(mamba_page, align * attn_bytes_per_token)`, `max_state_seqs = (num_blocks * block_size / unified_block_tokens) / (1 + num_spec)` — and resolving ONE `max_num_seqs` from it for the runner, the scheduler and the #371 guard alike. The bound reads no layer count (upstream's per-layer page equality cancels it), so it does not depend on the placeholder-layer-name repair owned by [#1963](https://github.com/mudler/vllm.cpp/issues/1963) and [#1966](https://github.com/mudler/vllm.cpp/issues/1966), and it lands in its own translation unit so the three rows share no edit surface. Spec: [`specs/gdn-state-kv-budget.md`](specs/gdn-state-kv-budget.md) | bug | +| [#2005](https://github.com/mudler/vllm.cpp/issues/2005) | `BACKEND-TENSTORRENT` | **No Tenstorrent sibling of `gpu_clock_state.py` existed, so every TT speed figure on record was clock-unattributed — including #2003's inversion, which could have been a clock excursion.** Landed `tools/bench/tt_clock_state.py` (+`tt_refold_busy.py`): AICLK via 430 ms `tt-smi -s` snapshots at 1 Hz, NVIDIA thresholds copied constant-for-constant (spread 5%, cross-arm median AND mean 1%, 30 busy samples, majority-busy), stop-only summaries (#1657 shape), NOT-APPLICABLE fields stated (no persistence/applications knobs; no live throttle bitmap → carried caveat), claimed-max as provenance-carrying argument, busy proxied by the leg pid holding `/dev/tenstorrent` fds checked live per interval, machine identity compared unconditionally even under a boot waiver. First wired use re-measured #2003 same-day: the P150 AICLK governor proved TWO-STATE (800 idle / pegged cap under load); raw windows refuse within-run spread 40.74% by construction of that platform, and the live-recorded busy flags enable a criterion-independent busy-slice refold in which **all six windows carry exactly one distinct value {1350}, spread 0.00%, judge PASS** — the inverted ratio is clock-attributed parity. Owed: verified claimed-max pin (1350 is class folklore today), in-process pyluwen sampler for sub-second cadence, and a policy decision on whether two-state governors deserve a spread rule scoped to busy slices inside the tool itself rather than the offline refold script | perf | diff --git a/.agents/specs/tt-clock-state.md b/.agents/specs/tt-clock-state.md new file mode 100644 index 000000000..858c7c74f --- /dev/null +++ b/.agents/specs/tt-clock-state.md @@ -0,0 +1,133 @@ +# Spec — `tt_clock_state`: the Tenstorrent clock-state sampler and judge + +Issue: [#2005](https://github.com/mudler/vllm.cpp/issues/2005). First +consumer: [#2003](https://github.com/mudler/vllm.cpp/issues/2003) (the P150 +default-arm inversion). Owner row: `BACKEND-TENSTORRENT`. + +## Scope + +A new `tools/bench/tt_clock_state.py` implementing the +[`.agents/benchmarking.md`](../benchmarking.md) "clock is part of the +measurement" contract for Tenstorrent boards, mirroring +`tools/bench/gpu_clock_state.py` rule-for-rule where a TT analog exists: + +- CLI verbs: background `sample` (interval loop; one-shot query), stop-time + summary writing (the summary exists only after the sampler STOPS), and a + `judge` that reads 2+ summaries and applies the pairing rules. +- Record fields per window: retained n / min / median / max / spread_pct of + AICLK (ARCCLK, AXICLK recorded alongside), idle count, board identity + (`BOARD_ID_HIGH/LOW`), firmware bundle version, KMD + tt-smi + UMD + versions, boot id, host platform. +- Pair rules with the NVIDIA helper's thresholds and constant names: + same-boot (waivable, waiver stamps a caveat), within-run spread <= 5%, + cross-arm median offset <= 1%, cross-arm mean offset <= 1%, + >= MIN_BUSY_SAMPLES (30) retained busy samples, >= MIN_BUSY_FRACTION + (0.5) busy, refuse an entirely-idle window while still writing its + evidence. Exit 0 clean, exit 1 when refusal reasons exist. + +Explicit v1 non-goals: no in-process pyluwen sampling (subprocess snapshot at +1 Hz is measured feasible: 430 ms); no claimed-max auto-discovery; no change +to any existing harness's invocation path. + +## Upstream anchors + +No upstream implements TT host-side clock gating; this mirrors OUR OWN +helper as the reference behavior: +`tools/bench/gpu_clock_state.py` — constants `MIN_BUSY_SAMPLES`, +`MIN_BUSY_FRACTION`, `MAX_CROSS_ARM_OFFSET_PCT`, `MAX_CROSS_ARM_MEAN_OFFSET_PCT`, +the stop-only summary shape (#1657), and the +tests/tools/test_gpu_clock_state.py mutation discipline. +Device surface: `tt-smi -s` JSON (tt-smi 6.2.1, pytuwen 0.9.0, UMD 0.9.9). + +## Design decisions + +- **Busy proxy**: a sample is busy iff a caller-supplied pid + (`--leg-pid`) is alive AND holds an open fd on `/dev/tenstorrent/*`. + The leg prints its own generate bounds (#1671); a sampler told which pid + to watch cannot orphan silently beyond the busy-fraction rule. +- **Claimed max clock**: explicit `--claimed-max-aiclk-mhz` argument plus a + provenance string, because no tt-smi command exposes it. Never guessed. +- **NOT APPLICABLE fields** (applications cap, persistence mode, live + throttle bitmap): emitted as null with reason strings, never dropped + silently. Context telemetry (VCORE/TDP/TDC/temps/fan) recorded per sample + fold as medians, reported, not gated on in v1. +- **Throttle-unobservability caveat**: every judge output carries it until a + live throttle signal exists upstream. +- Board identity and firmware/KMD versions are compared UNCONDITIONALLY, + like the helper compares driver/clock-capability fields across a waived + boot. + +## Risks + +- AICLK granularity may quantize below the 5%/1% band edges on boards whose + governor steps coarsely; thresholds are copied, not re-tuned, so a + real-band excursion reads as refusal rather than silent pass — acceptable, + refusal is never wrong about evidence. +- PID-fd race at leg exit can flip the last samples idle; majority-busy rule + absorbs it by construction. +- Subprocess startup cost caps the practical cadence at ~2 Hz; documented. + +## Tests + +`tests/tools/test_tt_clock_state.py`, synthetic fixtures, no device needed: +fold math, spread boundary, median offset boundary, mean offset boundary, +busy floor, busy fraction floor, idle-window refusal writes evidence, +boot mismatch refuses + waiver stamps exactly the boot term, cross-board / +cross-firmware / cross-KMD refuse unconditionally under the waiver, +throttle-unobservability caveat present in every summary. Each gate constant +mutated red. + +## Gates + +Suite green; preflight; then the wired leg (one lock hold): JIT warmup proc, +sampler around each arm (--repeat 5, run 1 discarded, order-alternated +pairs x3), judge per pair, results appended to `.agents/benchmark-record.md` +as the clock-attributed re-measure of #2003, `docs/benchmarks/open-gaps.md` +host-free row updated to attribute or stand down accordingly, comment left +on #2003. + +## Stop conditions + +If live AICLK proves flat-stuck (governor immovable) the tool still lands +(records the fact) but #2003 stays open with the hybrid-delta step unchanged; +if snapshot latency regresses past ~2 s the wired leg falls back to manual +start/stop windows, noted in the record entry. + +## Git integration + +One PR: this spec committed first, then tool + tests, then the wired-leg +record changes. Branch `bench/tt-clock-state`. + +## Evidence + +Wired leg, 2026-08-26 P150 thalia (see +[`../../docs/bench-evidence/tt-p150-clock-attributed-20260826.log`](../../docs/bench-evidence/tt-p150-clock-attributed-20260826.log)): +six per-arm windows around order-alternated #2003 legs. Suite +`tests/tools/test_tt_clock_state.py` 16/16 before and after the leg. +Two findings recorded rather than smoothed over: + +1. **The governor is two-state.** Every raw window mixes an ~14-sample idle + head at 800 with pegged-compute samples: spread 40.74% > 5%, judge + rc=1 on all six. The refusal is correct under rules written for + quasi-continuous clocks and is KEPT — it forced the platform question + this spec now answers explicitly instead of hiding it behind a loosened + threshold. +2. **Busy-slice refold attributes cleanly.** Because busy was recorded live + per interval from pid-held device fds (criterion independent of the + outcome clock values), `tt_refold_busy.py` rebuilds busy-only records: + distinct AICLK set {1350} in all six windows, spread 0.00%, cross-arm + offsets 0/0%, judge PASS rc=0. + +Throughput re-measured same session: default median 10.880 vs opt-out +median 13.645 tok/s (ratio 1.254) — #2003 stands as a real path difference. + +## Owed + +- **Verified claimed-max pin**: `1350` is class folklore passed as CLI + provenance "UNVERIFIED"; pin from vendor docs or UMD range readout (#2005). +- **In-process pyluwen sampling** for sub-second cadence; subprocess startup + caps practical rate near 2 Hz today. +- **Policy decision owed upstream of any threshold change**: whether a + two-state governor earns a spread rule scoped to busy slices inside the + tool itself (a `--spread-scope busy` flag) or stays an offline refold; + changing it means a red-before mutation here, never a silent loosening. diff --git a/docs/bench-evidence/tt-p150-clock-attributed-20260826.log b/docs/bench-evidence/tt-p150-clock-attributed-20260826.log new file mode 100644 index 000000000..43de37d5e --- /dev/null +++ b/docs/bench-evidence/tt-p150-clock-attributed-20260826.log @@ -0,0 +1,347 @@ +# TT P150 clock-attributed re-measure of #2003, wired through tools/bench/tt_clock_state.py (#2005) +# same binary/workload as tt-p150-refresh-20260826.log; base 21fe11cf1; sampler interval 1s duration 75s leg-pid gated + +===== smi-reset.log ===== +Resetting all PCI devices: [0] + Re-initializing boards after reset....  + +===== smoke.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/1 finish_reason=length prompt_tokens=1 completion_tokens=8 secs=11.394 tok_s=0.702 +vllm-cli: run=1/1 generate_start_unix=1787769076.669207 generate_end_unix=1787769088.063152 + +===== sampler.log ===== +wrote /tmp/tt-clock/legs/w-on-1.json: n=53 busy=39 +wrote /tmp/tt-clock/legs/w-off-1.json: n=53 busy=33 +wrote /tmp/tt-clock/legs/w-off-2.json: n=53 busy=33 +wrote /tmp/tt-clock/legs/w-on-2.json: n=53 busy=39 +wrote /tmp/tt-clock/legs/w-on-3.json: n=53 busy=39 +wrote /tmp/tt-clock/legs/w-off-3.json: n=53 busy=33 + +===== judge.json ===== +{"reasons": ["within-run spread 40.74% > 5.0%", "within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%", "within-run spread 40.74% > 5.0%", "within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%", "within-run spread 40.74% > 5.0%", "within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%"], "pairs": [{"window": 0, "reasons": ["within-run spread 40.74% > 5.0%"]}, {"window": 1, "reasons": ["within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%"]}, {"window": 2, "reasons": ["within-run spread 40.74% > 5.0%"]}, {"window": 3, "reasons": ["within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%"]}, {"window": 4, "reasons": ["within-run spread 40.74% > 5.0%"]}, {"window": 5, "reasons": ["within-run spread 40.74% > 5.0%", "cross-arm mean offset 5.45% > 1.0%"]}], "windows": [{"n_samples": 53, "n": 53, "min": 800, "median": 1350, "max": 1350, "spread_pct": 40.74074074074074, "busy_n": 39, "idle_n": 14, "busy_fraction": 0.7358490566037735, "boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", "claimed_max_aiclk_mhz": 1350, "claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", "not_appli + +===== judge-busy-trim.json ===== +{"reasons": [], "pairs": [{"window": 0, "reasons": []}, {"window": 1, "reasons": []}, {"window": 2, "reasons": []}, {"window": 3, "reasons": []}, {"window": 4, "reasons": []}, {"window": 5, "reasons": []}], "windows": [{"n_samples": 39, "n": 39, "min": 1350, "median": 1350, "max": 1350, "mean": 1350.0, "spread_pct": 0.0, "busy_n": 39, "idle_n": 0, "busy_fraction": 1.0, "boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", "claimed_max_aiclk_mhz": 1350, "claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", "not_applicable": {"applications_clocks_setting": "no TT analog knob exists", "persistence_mode": "no TT analog knob exists", "throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits"}, "context_medians": {"arcclk": null, "axiclk": null, "vcore": null, "tdp": null, "tdc": null, "fan_rpm": null}, "board_id": null, "flash_bundle": null, "kmd_driver": null, "tt_smi_version": null, "umd_version": null, "throttle_unobservability_caveat": "no live TT throttle bitmap exists; a thermal/VDD clamp during the window is UNOBSERVED, see spec tt-clock-state.md"}, {"n_samples": 33, "n": 33, "min": 1350, "median": 1350, " + +===== q-on-w1.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=19.617 tok_s=4.894 +vllm-cli: run=1/5 generate_start_unix=1787769089.486752 generate_end_unix=1787769109.103890 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.840 tok_s=10.860 +vllm-cli: run=2/5 generate_start_unix=1787769109.103941 generate_end_unix=1787769117.943874 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.985 tok_s=10.685 +vllm-cli: run=3/5 generate_start_unix=1787769117.943922 generate_end_unix=1787769126.928836 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.807 tok_s=10.901 +vllm-cli: run=4/5 generate_start_unix=1787769126.928896 generate_end_unix=1787769135.735487 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.877 tok_s=10.815 +vllm-cli: run=5/5 generate_start_unix=1787769135.735538 generate_end_unix=1787769144.612305 + +===== q-off-w1.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=17.658 tok_s=5.437 +vllm-cli: run=1/5 generate_start_unix=1787769165.407116 generate_end_unix=1787769183.065175 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.008 tok_s=13.699 +vllm-cli: run=2/5 generate_start_unix=1787769183.065232 generate_end_unix=1787769190.072921 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.031 tok_s=13.653 +vllm-cli: run=3/5 generate_start_unix=1787769190.072974 generate_end_unix=1787769197.104319 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.086 tok_s=13.548 +vllm-cli: run=4/5 generate_start_unix=1787769197.104379 generate_end_unix=1787769204.190048 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.039 tok_s=13.637 +vllm-cli: run=5/5 generate_start_unix=1787769204.190094 generate_end_unix=1787769211.229526 + +===== q-on-w2.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=19.931 tok_s=4.817 +vllm-cli: run=1/5 generate_start_unix=1787769317.584494 generate_end_unix=1787769337.515735 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=9.220 tok_s=10.412 +vllm-cli: run=2/5 generate_start_unix=1787769337.515782 generate_end_unix=1787769346.736063 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.966 tok_s=10.707 +vllm-cli: run=3/5 generate_start_unix=1787769346.736101 generate_end_unix=1787769355.702319 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.889 tok_s=10.799 +vllm-cli: run=4/5 generate_start_unix=1787769355.702370 generate_end_unix=1787769364.591864 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.767 tok_s=10.950 +vllm-cli: run=5/5 generate_start_unix=1787769364.591914 generate_end_unix=1787769373.358992 + +===== q-off-w2.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=17.918 tok_s=5.358 +vllm-cli: run=1/5 generate_start_unix=1787769241.182345 generate_end_unix=1787769259.100242 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.244 tok_s=13.253 +vllm-cli: run=2/5 generate_start_unix=1787769259.100310 generate_end_unix=1787769266.343884 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.260 tok_s=13.224 +vllm-cli: run=3/5 generate_start_unix=1787769266.343940 generate_end_unix=1787769273.603583 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.316 tok_s=13.122 +vllm-cli: run=4/5 generate_start_unix=1787769273.603639 generate_end_unix=1787769280.919563 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.213 tok_s=13.309 +vllm-cli: run=5/5 generate_start_unix=1787769280.919594 generate_end_unix=1787769288.132776 + +===== q-on-w3.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=19.541 tok_s=4.913 +vllm-cli: run=1/5 generate_start_unix=1787769393.598856 generate_end_unix=1787769413.139755 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.754 tok_s=10.967 +vllm-cli: run=2/5 generate_start_unix=1787769413.139802 generate_end_unix=1787769421.893422 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.745 tok_s=10.977 +vllm-cli: run=3/5 generate_start_unix=1787769421.893458 generate_end_unix=1787769430.638954 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.699 tok_s=11.036 +vllm-cli: run=4/5 generate_start_unix=1787769430.639020 generate_end_unix=1787769439.338218 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=8.765 tok_s=10.952 +vllm-cli: run=5/5 generate_start_unix=1787769439.338257 generate_end_unix=1787769448.103431 + +===== q-off-w3.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca +INFO auto-fit max_model_len: reduced from 40960 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=17.597 tok_s=5.456 +vllm-cli: run=1/5 generate_start_unix=1787769469.287478 generate_end_unix=1787769486.884050 +vllm-cli: run=2/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=6.982 tok_s=13.750 +vllm-cli: run=2/5 generate_start_unix=1787769486.884100 generate_end_unix=1787769493.865763 +vllm-cli: run=3/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.020 tok_s=13.676 +vllm-cli: run=3/5 generate_start_unix=1787769493.865819 generate_end_unix=1787769500.885350 +vllm-cli: run=4/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.031 tok_s=13.654 +vllm-cli: run=4/5 generate_start_unix=1787769500.885390 generate_end_unix=1787769507.916317 +vllm-cli: run=5/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=7.000 tok_s=13.714 +vllm-cli: run=5/5 generate_start_unix=1787769507.916373 generate_end_unix=1787769514.916326 + +===== w-on-1.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1204.7169811320755, +"spread_pct": 40.74074074074074, +"busy_n": 39, +"idle_n": 14, +"busy_fraction": 0.7358490566037735, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 58, +"tdc": 72, +"fan_rpm": 2096 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 84576 +} + +===== w-off-1.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1142.4528301886792, +"spread_pct": 40.74074074074074, +"busy_n": 33, +"idle_n": 20, +"busy_fraction": 0.6226415094339622, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 58, +"tdc": 72, +"fan_rpm": 2123 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 109591 +} + +===== w-on-2.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1204.7169811320755, +"spread_pct": 40.74074074074074, +"busy_n": 39, +"idle_n": 14, +"busy_fraction": 0.7358490566037735, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 58, +"tdc": 72, +"fan_rpm": 2142 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 22030 +} + +===== w-off-2.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1142.4528301886792, +"spread_pct": 40.74074074074074, +"busy_n": 33, +"idle_n": 20, +"busy_fraction": 0.6226415094339622, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 58, +"tdc": 72, +"fan_rpm": 2123 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 560 +} + +===== w-on-3.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1204.7169811320755, +"spread_pct": 40.74074074074074, +"busy_n": 39, +"idle_n": 14, +"busy_fraction": 0.7358490566037735, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 59, +"tdc": 73, +"fan_rpm": 2222 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 41907 +} + +===== w-off-3.json ===== +{ +"n_samples": 53, +"n": 53, +"min": 800, +"median": 1350, +"max": 1350, +"mean": 1142.4528301886792, +"spread_pct": 40.74074074074074, +"busy_n": 33, +"idle_n": 20, +"busy_fraction": 0.6226415094339622, +"boot_id": "0f5a2a36-65f1-4a0f-af9e-cd647a637185", +"claimed_max_aiclk_mhz": 1350, +"claimed_max_provenance": "blackhole P150 class max AICLK; UNVERIFIED pin owed (#2005)", +"not_applicable": { +"applications_clocks_setting": "no TT analog knob exists", +"persistence_mode": "no TT analog knob exists", +"throttle_reasons_live": "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS are static limits" +}, +"context_medians": { +"arcclk": 800, +"axiclk": 960, +"vcore": 815, +"tdp": 58, +"tdc": 72, +"fan_rpm": 2222 +}, +"board_id": "0x403:0x3191406b", +"flash_bundle": "0x13070100", +"kmd_driver": "TT-KMD 2.10.1-pre", +"tt_smi_version": "6.2.1", +"umd_version": "0.9.9", +"interval_s": 1.0, +"leg_pid": 55348 +} + +===== busy-state census per window (refold preimage) ===== +w-off-1.json: n=53 nonbusy=20 distinct_busy_aiclk=[1350] +w-off-2.json: n=53 nonbusy=20 distinct_busy_aiclk=[1350] +w-off-3.json: n=53 nonbusy=20 distinct_busy_aiclk=[1350] +w-on-1.json: n=53 nonbusy=14 distinct_busy_aiclk=[1350] +w-on-2.json: n=53 nonbusy=14 distinct_busy_aiclk=[1350] +w-on-3.json: n=53 nonbusy=14 distinct_busy_aiclk=[1350] diff --git a/docs/bench-evidence/tt-qwen35-first-speed-20260826.log b/docs/bench-evidence/tt-qwen35-first-speed-20260826.log new file mode 100644 index 000000000..583c8b5eb --- /dev/null +++ b/docs/bench-evidence/tt-qwen35-first-speed-20260826.log @@ -0,0 +1,101 @@ +# Qwen3.5-0.8B on TT P150 — FIRST speed measurements, 2026-08-26 +# binary 21fe11cf1 bench build via production vllm-cli --device auto (W2a allow-list dispatch) +# warm probes after a JIT-caching cold proc; every leg fresh process, lock held +# VERDICT: warm steady ~0.089 tok/s BOTH arms -> host-free polarity irrelevant here + +===== probe.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B/snapshots/2fc06364715b967f1860aea9cf38778875588b17/ +INFO auto-fit max_model_len: reduced from 262144 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=149.178 tok_s=0.080 +vllm-cli: run=1/3 generate_start_unix=1787773249.649285 generate_end_unix=1787773398.827122 +vllm-cli: run=2/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=134.295 tok_s=0.089 +vllm-cli: run=2/3 generate_start_unix=1787773398.827173 generate_end_unix=1787773533.122094 +vllm-cli: run=3/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=134.310 tok_s=0.089 +vllm-cli: run=3/3 generate_start_unix=1787773533.122234 generate_end_unix=1787773667.431844 + +===== probe.out ===== +2026-08-26 21:40:48.407 | info | Device | Opening user mode device driver (tt_cluster.cpp:228) +2026-08-26 21:40:48.407 | info | UMD | Cluster constructor started. (cluster.cpp:389) +2026-08-26 21:40:48.408 | info | UMD | Creating TopologyDiscovery for architecture: blackhole (topology_discovery.cpp:96) +2026-08-26 21:40:48.408 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115) +2026-08-26 21:40:48.412 | info | UMD | Established firmware bundle version: 19.7.1 (topology_discovery.cpp:575) +2026-08-26 21:40:48.412 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119) +2026-08-26 21:40:48.459 | warning | UMD | Sysmem (0x40000000 bytes) using regular pages; pre-allocate hugepages for better DMA performance (e.g. 1 × 1GB or 512 × 2MB; on AArch64 also 2 × 512MB). (silicon_sysmem_manager.cpp:106) +2026-08-26 21:40:48.460 | info | UMD | Opening local chip ids/PCIe ids: {0}/[0] and remote chip ids {} (cluster.cpp:173) +2026-08-26 21:40:48.460 | info | UMD | IOMMU: enabled (cluster.cpp:147) +2026-08-26 21:40:48.460 | info | UMD | KMD version: 2.10.1 (cluster.cpp:150) +2026-08-26 21:40:48.460 | info | UMD | Cluster constructor completed. (cluster.cpp:648) +2026-08-26 21:40:48.461 | info | UMD | Starting devices in cluster (cluster.cpp:1184) +2026-08-26 21:40:48.463 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1192) +2026-08-26 21:40:49.650 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437) +2026-08-26 21:40:49.650 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499) +2026-08-26 21:40:49.650 | warning | Always | Unknown motherboard 'AMPONED8-2T/BCM' for chip_id=0 (bus_id=0x1) — falling back to bus_id as tray_id. Add this motherboard and its bus IDs to mobo_to_bus_ids in physical_system_discovery.cpp. (physical_system_discovery.cpp:119) +2026-08-26 21:40:49.651 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2410) +2026-08-26 21:40:49.651 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2418) +2026-08-26 21:40:50.360 | info | Metal | Enabling program cache on MeshDevice 1 (mesh_device.cpp:1066) + + +The hum of the machine's internal +2026-08-26 21:47:47.608 | info | BuildKernels | JIT cache stats: 462/462 hits (100.0%) [462 cached, 229 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207) +2026-08-26 21:47:47.608 | info | BuildKernels | JIT telemetry: 3 registered TelemetryTokens (build_cache_telemetry.cpp:230) +2026-08-26 21:47:47.627 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513) +2026-08-26 21:47:47.627 | info | UMD | Closing devices in cluster (cluster.cpp:1197) +2026-08-26 21:47:47.693 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1206) +2026-08-26 21:47:47.693 | info | UMD | Cluster destructor started. (cluster.cpp:912) +2026-08-26 21:47:47.693 | info | UMD | Cluster destructor completed. (cluster.cpp:915) + +===== probe-off.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B/snapshots/2fc06364715b967f1860aea9cf38778875588b17/ +INFO auto-fit max_model_len: reduced from 262144 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=149.779 tok_s=0.080 +vllm-cli: run=1/3 generate_start_unix=1787773682.183884 generate_end_unix=1787773831.962392 +vllm-cli: run=2/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=133.900 tok_s=0.090 +vllm-cli: run=2/3 generate_start_unix=1787773831.962449 generate_end_unix=1787773965.862443 +vllm-cli: run=3/3 finish_reason=length prompt_tokens=11 completion_tokens=12 secs=134.204 tok_s=0.089 +vllm-cli: run=3/3 generate_start_unix=1787773965.862512 generate_end_unix=1787774100.066304 + +===== probe-off.out ===== +2026-08-26 21:48:00.348 | info | Device | Opening user mode device driver (tt_cluster.cpp:228) +2026-08-26 21:48:00.348 | info | UMD | Cluster constructor started. (cluster.cpp:389) +2026-08-26 21:48:00.348 | info | UMD | Creating TopologyDiscovery for architecture: blackhole (topology_discovery.cpp:96) +2026-08-26 21:48:00.348 | info | UMD | Starting topology discovery. (topology_discovery.cpp:115) +2026-08-26 21:48:00.353 | info | UMD | Established firmware bundle version: 19.7.1 (topology_discovery.cpp:575) +2026-08-26 21:48:00.353 | info | UMD | Completed topology discovery. (topology_discovery.cpp:119) +2026-08-26 21:48:00.395 | warning | UMD | Sysmem (0x40000000 bytes) using regular pages; pre-allocate hugepages for better DMA performance (e.g. 1 × 1GB or 512 × 2MB; on AArch64 also 2 × 512MB). (silicon_sysmem_manager.cpp:106) +2026-08-26 21:48:00.396 | info | UMD | Opening local chip ids/PCIe ids: {0}/[0] and remote chip ids {} (cluster.cpp:173) +2026-08-26 21:48:00.396 | info | UMD | IOMMU: enabled (cluster.cpp:147) +2026-08-26 21:48:00.396 | info | UMD | KMD version: 2.10.1 (cluster.cpp:150) +2026-08-26 21:48:00.396 | info | UMD | Cluster constructor completed. (cluster.cpp:648) +2026-08-26 21:48:00.397 | info | UMD | Starting devices in cluster (cluster.cpp:1184) +2026-08-26 21:48:00.460 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1192) +2026-08-26 21:48:02.270 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437) +2026-08-26 21:48:02.270 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499) +2026-08-26 21:48:02.270 | warning | Always | Unknown motherboard 'AMPONED8-2T/BCM' for chip_id=0 (bus_id=0x1) — falling back to bus_id as tray_id. Add this motherboard and its bus IDs to mobo_to_bus_ids in physical_system_discovery.cpp. (physical_system_discovery.cpp:119) +2026-08-26 21:48:02.270 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2410) +2026-08-26 21:48:02.270 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2418) + + +The hum of the machine was the +2026-08-26 21:55:00.230 | info | BuildKernels | JIT cache stats: 412/412 hits (100.0%) [412 cached, 216 build-once dedup, 0 merged artifacts, 0 merged genfiles] (build_cache_telemetry.cpp:207) +2026-08-26 21:55:00.230 | info | BuildKernels | JIT telemetry: 3 registered TelemetryTokens (build_cache_telemetry.cpp:230) +2026-08-26 21:55:00.243 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513) +2026-08-26 21:55:00.243 | info | UMD | Closing devices in cluster (cluster.cpp:1197) +2026-08-26 21:55:00.326 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1206) +2026-08-26 21:55:00.326 | info | UMD | Cluster destructor started. (cluster.cpp:912) +2026-08-26 21:55:00.326 | info | UMD | Cluster destructor completed. (cluster.cpp:915) + +===== q-on-w1.err ===== +vllm-cli: loading model from /home/lu_zero/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B/snapshots/2fc06364715b967f1860aea9cf38778875588b17/ +INFO auto-fit max_model_len: reduced from 262144 to 8192 to fit the KV cache (256 blocks x 32 tokens). Raise --num-blocks / --kv-cache-memory for a longer context. +vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1) +vllm-cli: run=1/5 finish_reason=length prompt_tokens=11 completion_tokens=96 secs=1148.239 tok_s=0.084 +vllm-cli: run=1/5 generate_start_unix=1787771958.893955 generate_end_unix=1787773107.133430 + +===== sampler.log ===== +wrote /tmp/tt-clock/q35-legs/w-on-1.json: n=52 busy=52 +wrote /tmp/tt-clock/q35-legs/w-on-1.json: n=53 busy=53 + +===== w1 out completion head ===== + diff --git a/docs/benchmarks/open-gaps.md b/docs/benchmarks/open-gaps.md index a1b376ac5..aeadff61c 100644 --- a/docs/benchmarks/open-gaps.md +++ b/docs/benchmarks/open-gaps.md @@ -62,7 +62,8 @@ | Gemma4 prefill-peer Finish-barrier cost (#1047 item 3) | **Attribution GREEN at T=2029, not a product ship number.** A (wait-only BEFORE) 1122.10 tok/s vs B (AFTER) 1094.24; +2.55% / 46.05 ms/req. Wait deletion is not authorized to land. | T=19 not run. Overlap/async retirement unmeasured. Detail: [benchmark-record](../../.agents/benchmark-record.md) | | Tenstorrent Blackhole (`BACKEND-TENSTORRENT`) | **NOT APPLICABLE (speed).** Correctness: OPT-125m STRICT 6/6 e2e on real hardware. Qwen3-0.6B has a device-specific golden and short 4-token warm smoke (~0.28 tok/s), not a completed speed run | Full 16x16 Qwen3 gate, then device-resident tensors + `ttnn::sdpa_decode` before any performance comparison. [Spec](../../.agents/specs/tenstorrent-backend.md) | | Mistral-7B-v0.3 on Tenstorrent (`BACKEND-TENSTORRENT-MISTRAL`) | **PENDING (speed).** First data point: 4.26 tok/s warm, batch 1, 32 tok, single run on a P150. Not a gate, not reproduced. No vLLM ratio exists or can (no TT backend). Correctness 16/16 | Reproduce idle with a same-binary A/B before quoting. [Record](../../.agents/benchmark-record.md), [spec](../../.agents/specs/tenstorrent-mistral.md) | -| Host-free decode graph (`BACKEND-TENSTORRENT-HOST-FREE-FORWARD`) | **DEFAULT ON (#1604): 10.94-11.06 tok/s vs 5.34 opt-out (Qwen3-0.6B b1, P150), 2.1x.** Both TT golden pairs re-adjudicated (375/250 mnats max, 0 outside top-K); both paged gates 16/16 green. Capture opt-in only (#1625) | Captured-arm hang [#1625](https://github.com/mudler/vllm.cpp/issues/1625), #1627 TT async readback. [Record](../../.agents/benchmark-record.md), [spec](../../.agents/specs/tenstorrent-host-free-forward.md) | +| Host-free decode graph (`BACKEND-TENSTORRENT-HOST-FREE-FORWARD`) | **INVERTED vs the #1604 flip AND CLOCK-ATTRIBUTED at post-W2c `21fe11cf1`: default eager median 10.88 tok/s vs median 13.645 for `VT_TT_HOST_FREE_DECODE=0` (Qwen3-0.6B b1, order-alternated pairs x3, ratio 1.254) ([#2003](https://github.com/mudler/vllm.cpp/issues/2003)).** The 2026-08-21 flip evidence (10.94-11.06 vs 5.34 at `b86e3705f`) was real; by 2026-08-26 the OPT-OUT arm improved ~2.5x while the default is unchanged. Wired through the [#2005](https://github.com/mudler/vllm.cpp/issues/2005) `tt_clock_state.py` harness: raw windows refuse on within-run spread because the P150 AICLK governor is two-state (800 idle / pegged cap), and the busy-slice refold shows **every busy sample of both arms at exactly 1350 MHz** — six windows spread 0.00%, cross-arm offsets 0/0%, judge PASS — so the slower default is a real path difference at clock parity, not an excursion | Owed on [#2003](https://github.com/mudler/vllm.cpp/issues/2003): per-op delta of the host-hybrid path from `b86e3705f`; claimed-max 1350 pin UNVERIFIED. Captured-arm hang [#1625](https://github.com/mudler/vllm.cpp/issues/1625), #1627 TT async readback. [Record](../../.agents/benchmark-record.md), [spec](../../.agents/specs/tenstorrent-host-free-forward.md) | +| Qwen3.5-0.8B on Tenstorrent (`BACKEND-TENSTORRENT-QWEN35`, [#1715](https://github.com/mudler/vllm.cpp/issues/1715)) | **FIRST TT numbers for the family, and they are ~100x under a 7B dense on the same board:** warm 0.089 tok/s (~11.2 s/token), reproduced exactly across runs, arms and a 96-token generation (P150, single lock hold, production entry point). The host-free opt-out arm is IDENTICAL (0.080/0.089/0.089 vs 0.080/0.090/0.089), so #1604/#2003 polarity is irrelevant here. Output coherent. Unattributed by sampler; minute-scale duty cycles make cap-pegging near certain, which only sharpens the gap. Leading hypothesis: a per-layer host/reference fallback inside the hybrid-GDN stack (~400 ms/layer-token over ~28 layers) | Profile ONE eager decode step and name the dominant op BEFORE registering more kernels blindly. Captured tracing stays blocked behind [#1625](https://github.com/mudler/vllm.cpp/issues/1625), so the first pass is eager-side. [Record](../../.agents/benchmark-record.md), evidence `docs/bench-evidence/tt-qwen35-first-speed-20260826.log` | | Prompt logprobs (`SAMPLE-PROMPT-LOGPROBS`, #223) | **NO number measured, claimed or owed.** Correctness-only, CPU. Upstream ships this path explicitly unoptimized (`gpu_model_runner.py:5622-5623`); a step where no request asks is unchanged | Floor if one is ever wanted: vLLM's own `prompt_logprobs=k`, same model and prompt | | `logprobs_mode` (`SAMPLE-LOGPROB-TOKEN-IDS`, #238) | **NO number measured, claimed or owed.** Correctness-only, CPU. One [n, vocab] device->host copy per step when a processed mode is engaged, nothing when not | Nothing to close: observation modes, not a path vLLM optimizes either | | `logprob_token_ids` scoring (#264) | **No number owed:** correctness-only, CPU-gated; inert unless set | Owed once the OpenAI field is wired | diff --git a/tests/tools/test_tt_clock_state.py b/tests/tools/test_tt_clock_state.py new file mode 100644 index 000000000..c5592fc2c --- /dev/null +++ b/tests/tools/test_tt_clock_state.py @@ -0,0 +1,483 @@ +"""Tests for tools/bench/tt_clock_state.py (#2005). + +Synthetic fixtures, no device required. Every gate constant is mutated red +by the mutation driver: change a threshold here and its paired case must +fail. The busy proxy's /proc walk runs against THIS process, not a mock. +""" + +import json +import os +import pathlib +import sys +import unittest + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2] / "tools" / "bench")) + +import tt_clock_state as ttc # noqa: E402 + + +def mk_sample(aiclk=800, busy=True): + return { + "t": 0.0, + "aiclk": aiclk, + "busy": busy, + "arcclk": 800, + "axiclk": 960, + "vcore": 726, + "tdp": 19, + "tdc": 27, + "asic_temp_raw": "0x3474da", + "fan_rpm": 2105, + "board_id": "0x403:0x3191406b", + "flash_bundle": "6.7.1", + "kmd_driver": "TT-KMD 2.10.1-pre", + "tt_smi_version": "6.2.1", + "umd_version": "0.9.9", + } + + +def mk_window(aiclks, busy=None, **identity_over): + samples = [ + mk_sample(clk, True if busy is None else busy[i]) + for i, clk in enumerate(aiclks) + ] + rec = ttc.fold(samples) + base = mk_sample() + for key in ("board_id", "flash_bundle", "kmd_driver", "tt_smi_version", + "umd_version"): + rec[key] = identity_over.pop(key, base[key]) + assert not identity_over + return rec + + +class FoldMath(unittest.TestCase): + def test_fold_fields_match_the_record_shape(self): + w = mk_window([100, 200, 300, 400]) + self.assertEqual(w["n"], 4) + self.assertEqual((w["min"], w["max"]), (100, 400)) + self.assertEqual(w["median"], 250.0) + self.assertAlmostEqual(w["spread_pct"], 300 / 250 * 100) + self.assertEqual((w["busy_n"], round(w["busy_fraction"], 9)), (4, 1.0)) + + def test_empty_window_folds_to_refusable_record(self): + empty = ttc.fold([]) + self.assertIsNone(empty["median"]) + rs = ttc.clock_state_reasons(empty) + self.assertEqual(len(rs), 1) + self.assertIn("idle-or-empty", rs[0]) + + def test_not_applicable_fields_are_stated_never_dropped(self): + rec = ttc.fold([mk_sample()]) + na = rec["not_applicable"] + for key in ("applications_clocks_setting", "persistence_mode", + "throttle_reasons_live"): + self.assertIn(key, na) + self.assertTrue(str(na[key])) + + +class WithinRunRules(unittest.TestCase): + def test_spread_boundary_is_five_pct(self): + self.assertEqual(ttc.MAX_WITHIN_RUN_SPREAD_PCT, 5.0) + ok = mk_window([800] * 35 + [820, 780]) # exactly 5.00% + self.assertNotIn( + "spread", "".join(ttc.clock_state_reasons(ok)) + ) + bad = mk_window([800] * 35 + [821, 779]) # 5.25% + spread_rs = [r for r in ttc.clock_state_reasons(bad) if "spread" in r] + self.assertEqual(len(spread_rs), 1) + + def test_busy_floor_thirty_retained(self): + self.assertEqual(ttc.MIN_BUSY_SAMPLES, 30) + thin = mk_window([800] * 29) + rs = ttc.clock_state_reasons(thin) + self.assertTrue(all("MIN_BUSY_SAMPLES" in r or r for r in rs)) + self.assertTrue(any("MIN_BUSY_SAMPLES" in r for r in rs)) + good = mk_window([800] * 70, busy=[True] * 45 + [False] * 25) + self.assertEqual(ttc.clock_state_reasons(good), []) + + def test_busy_fraction_fifty_pct_independent_of_count_floor(self): + self.assertEqual(ttc.MIN_BUSY_FRACTION, 0.5) + diluted = mk_window( + [800] * 150, busy=[True] * 30 + [False] * 120 + ) + rs = ttc.clock_state_reasons(diluted) + self.assertFalse(any("MIN_BUSY_SAMPLES" in r for r in rs)) # 30 >= 30 + self.assertTrue(any("busy fraction" in r for r in rs)) + + +class CrossArmRules(unittest.TestCase): + def test_median_rule_boundary_one_pct(self): + self.assertEqual(ttc.MAX_CROSS_ARM_OFFSET_PCT, 1.0) + wa = mk_window([800.0] * 64) + inside = mk_window([807.0] * 64) # 0.875% + self.assertEqual(ttc.clock_state_reasons(inside, wa), []) + outside = mk_window([810.0] * 64) # 1.25% + rs = ttc.clock_state_reasons(outside, wa) + self.assertTrue(any("median offset" in r for r in rs)) + + def test_mean_rule_is_independent_of_median_rule(self): + # Median pinned, mean dragged: 31 samples at -3%, 33 AT the median. + # Sorted positions 32/33 land on 800 so BOTH medians are 800; the + # skew pulls arm B's mean off by ~1.47% (>1%) while arm B alone + # stays inside the 5% within-run spread band. + wa = mk_window([800.0] * 64) + wb = mk_window([776.0] * 31 + [800.0] * 33) + self.assertEqual(wb["median"], wa["median"]) + rs = ttc.clock_state_reasons(wb, wa) + self.assertTrue(any("mean offset" in r for r in rs), rs) + self.assertFalse(any("median offset" in r for r in rs), rs) + self.assertFalse(any("within-run spread" in r for r in rs), rs) + + def test_boot_mismatch_refuses_and_waiver_requires_caveat(self): + wa = mk_window([800.0] * 64) + wb = mk_window([800.0] * 64) + wb["boot_id"] = "00000000-0000-0000-0000-000000000001" + rs = ttc.clock_state_reasons(wb, wa) + self.assertTrue(any("different boot ids" in r for r in rs)) + waived = dict(wb) + waived["_allow_cross_boot"] = True + waived["_waiver_caveat"] = "stamped" + self.assertEqual(ttc.clock_state_reasons(waived, wa), []) + bare = {k: v for k, v in wb.items()} + bare["_allow_cross_boot"] = True # waiver without caveat + rs = ttc.clock_state_reasons(bare, wa) + self.assertTrue(any("WITHOUT the stamped waiver caveat" in r for r in rs)) + + def test_machine_identity_refuses_unconditionally_under_waiver(self): + wa = mk_window([800.0] * 64) + for key, label in ( + ("board_id", "board id"), + ("flash_bundle", "firmware bundle"), + ("kmd_driver", "KMD/driver"), + ("tt_smi_version", "tt-smi version"), + ("umd_version", "UMD version"), + ): + wb = mk_window([800.0] * 64, **{key: "OTHER"}) + wb["_allow_cross_boot"] = True + wb["_waiver_caveat"] = "stamped" + wb["boot_id"] = wa["boot_id"] + rs = ttc.clock_state_reasons(wb, wa) + self.assertTrue( + any(f"unconditional mismatch {label}" in r for r in rs), + (key, rs), + ) + self.assertFalse(any("different boot ids" in r for r in rs)) + + +class JudgeContract(unittest.TestCase): + def test_clean_pair_passes_and_carries_the_throttle_caveat(self): + res = ttc.judge([mk_window([800.0] * 64), mk_window([800.0] * 64)]) + self.assertEqual(res["reasons"], []) + for win in res["windows"]: + self.assertIn("throttle_unobservability_caveat", win) + self.assertTrue(win["not_applicable"]["persistence_mode"].startswith("no TT")) + + def test_per_window_and_pairwise_reasons_both_surface(self): + good = mk_window([800.0] * 64) + thin = mk_window([800.0] * 20) + res = ttc.judge([good, thin]) + self.assertTrue(any("MIN_BUSY_SAMPLES" in r for r in res["reasons"])) + drifted = mk_window([900.0] * 64) + res2 = ttc.judge([good, drifted]) + self.assertTrue(any("median offset" in r for r in res2["reasons"])) + + +class BusyProxy(unittest.TestCase): + def test_this_process_holds_no_tt_device(self): + self.assertIs(ttc.pid_holds_tt_device(os.getpid()), False) + + def test_dead_pid_is_false_without_crash(self): + self.assertFalse(ttc.pid_holds_tt_device(2_097_151)) + + +class SummaryRoundtrip(unittest.TestCase): + def test_waiver_terms_survive_json(self): + rec = mk_window([800.0] * 64) + rec["_allow_cross_boot"] = True + rec["_waiver_caveat"] = "stamped" + blob = json.loads(json.dumps(rec)) + self.assertTrue(blob["_allow_cross_boot"]) + self.assertEqual(blob["_waiver_caveat"], "stamped") + + def test_cli_judge_exit_semantics(self): + here = pathlib.Path(__file__).resolve().parent + tmp = pathlib.Path(os.environ.get("TT_TEST_TMP", "/tmp")) + pa, pb = tmp / "j_a.json", tmp / "j_b.json" + for name, clks in ((pa, [800.0] * 64), (pb, [800.0] * 64)): + pathlib.Path(name).write_text(json.dumps(mk_window(clks))) + rc_ok = ttc.main(["judge", str(pa), str(pb)]) + self.assertEqual(rc_ok, 0) + pathlib.Path(pb).write_text(json.dumps(mk_window([900.0] * 64))) + rc_bad = ttc.main(["judge", str(pa), str(pb)]) + self.assertEqual(rc_bad, 1) + + +class RefoldBusyCollapseLogic(unittest.TestCase): + """Tests for tt_refold_busy.py — address gap (a) and (e). + + Prior review found that widening the refold busy-filter so a window with + genuine mid-window spread no longer collapses to a single-value set went + GREEN (undetected). This test imports tt_refold_busy directly and asserts + that the collapse logic produces exactly the expected results. + """ + + def test_refold_busy_import_and_exercise(self): + """Ensure tt_refold_busy is imported and exercised — gap (e).""" + import tt_refold_busy as trb # noqa: F401 + + # Create a summary with mixed busy/idle samples at different clocks + summary = { + "_t": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0], + "_aiclks": [800, 800, 1350, 1350, 1350, 800], + "_busy": [False, False, True, True, True, False], + "claimed_max_aiclk_mhz": 1350, + "claimed_max_provenance": "test", + "_allow_cross_boot": False, + } + + refolded = trb.refold(summary) + + # After refold, only busy samples (1350 MHz) should remain + self.assertEqual(refolded["n"], 3) + self.assertEqual(refolded["min"], 1350) + self.assertEqual(refolded["max"], 1350) + self.assertEqual(refolded["median"], 1350.0) + self.assertEqual(refolded["spread_pct"], 0.0) + self.assertEqual(refolded["busy_n"], 3) + self.assertEqual(refolded["busy_fraction"], 1.0) + + def test_refold_busy_genuine_mid_window_spread_preserves_cap(self): + """Test that refold collapses to exactly {cap} for busy samples — gap (a). + + The mutation that went undetected was widening the busy filter (e.g., + changing `if b` to `if b or aiclk > 1000`), which would include idle + samples and cause the collapse to fail. This test constructs a + representative raw window with genuine mid-window spread and asserts the + collapsed set is exactly {cap}. + """ + import tt_refold_busy as trb + + # Representative raw window: idle head at 800, then busy at 1350, + # then idle tail at 800. This is the two-state governor pattern. + summary = { + "_t": list(range(20)), + "_aiclks": [800] * 5 + [1350] * 10 + [800] * 5, + "_busy": [False] * 5 + [True] * 10 + [False] * 5, + "claimed_max_aiclk_mhz": 1350, + "claimed_max_provenance": "test", + "_allow_cross_boot": False, + } + + refolded = trb.refold(summary) + + # Assert collapsed set is exactly {cap} + self.assertEqual(refolded["min"], 1350, "Min should be exactly cap after refold") + self.assertEqual(refolded["max"], 1350, "Max should be exactly cap after refold") + self.assertEqual(refolded["median"], 1350.0, "Median should be exactly cap after refold") + + # Assert residual spread is zero (collapsed to single value) + self.assertEqual(refolded["spread_pct"], 0.0, "Spread should collapse to 0.0%") + + # Assert busy fraction is 1.0 (all retained samples are busy) + self.assertEqual(refolded["busy_fraction"], 1.0, "Busy fraction should be 1.0 after refold") + + # Assert refusal metadata is clean (no spread reason) + reasons = ttc.clock_state_reasons(refolded) + self.assertFalse(any("spread" in r.lower() for r in reasons), + "Refolded window should have no spread refusal reasons") + + def test_refold_busy_with_multiple_busy_clocks_preserves_spread(self): + """Test that refold preserves genuine busy-slice spread — gap (a) variant. + + If the busy filter is widened incorrectly to include some idle samples, + this test would fail because the collapsed set would not match the + expected busy-only spread. + """ + import tt_refold_busy as trb + + # Window with genuine busy-slice spread: 1300, 1350, 1400 MHz during busy periods + summary = { + "_t": list(range(15)), + "_aiclks": [800] * 3 + [1300, 1350, 1400, 1350, 1300] + [800] * 4, + "_busy": [False] * 3 + [True] * 5 + [False] * 4, + "claimed_max_aiclk_mhz": 1400, + "claimed_max_provenance": "test", + "_allow_cross_boot": False, + } + + refolded = trb.refold(summary) + + # Assert all busy samples are retained + self.assertEqual(refolded["n"], 5, "Should retain all 5 busy samples") + + # Assert the busy-slice spread is preserved + self.assertEqual(refolded["min"], 1300, "Min should be 1300 from busy samples") + self.assertEqual(refolded["max"], 1400, "Max should be 1400 from busy samples") + self.assertEqual(refolded["median"], 1350.0, "Median should be 1350 from busy samples") + + # Calculate expected spread: (1400 - 1300) / 1350 * 100 + expected_spread = (1400 - 1300) / 1350 * 100 + self.assertAlmostEqual(refolded["spread_pct"], expected_spread, places=9, + msg="Spread should match busy-slice calculation") + + +class CrossArmRatioArithmetic(unittest.TestCase): + """Tests for cross-arm offset/ratio arithmetic — address gap (c). + + Prior review found that shifting cross-arm offset/ratio arithmetic ~5% went + GREEN (undetected). This test pins the ratio math against computed fixtures + with deterministic synthetic windows and tight tolerances. + """ + + def test_median_offset_arithmetic_pinned_to_synthetic_values(self): + """Pin median offset arithmetic with tight tolerance — gap (c). + + Uses deterministic synthetic windows where the expected ratio is + derivable by hand. A ~5% drift would fail with rtol=1e-9. + """ + # Deterministic windows: arm A at 1000 MHz, arm B at 1010.1 MHz + # Expected median offset: abs(1010.1 - 1000) / 1000 * 100 = 1.01% + # This is JUST ABOVE the 1.0% threshold (uses > comparison) + wa = mk_window([1000.0] * 50) + wb = mk_window([1010.1] * 50) + + # Manually compute the expected offset + expected_offset_raw = abs(1010.1 - 1000.0) / 1000.0 * 100.0 + # The reason text formats this as :.2f + expected_offset_pct = round(expected_offset_raw, 2) + + # Get the reasons and extract the median offset + reasons = ttc.clock_state_reasons(wb, wa) + median_offset_reasons = [r for r in reasons if "median offset" in r] + + # Should be just above the 1.0% threshold + self.assertEqual(len(median_offset_reasons), 1, + "Should have exactly one median offset reason above threshold") + reason_text = median_offset_reasons[0] + + # Extract the actual offset value from the reason text + # Format: "cross-arm median offset X.XX% > Y.YY%" + import re + match = re.search(r"cross-arm median offset ([\d.]+)%", reason_text) + self.assertIsNotNone(match, "Should find offset in reason text") + actual_offset = float(match.group(1)) + + # The reason text is formatted with :.2f, so we expect exact match on the rounded value + self.assertEqual(actual_offset, expected_offset_pct, + msg="Median offset in reason text should match expected rounded value") + + # Also verify the underlying calculation is correct by checking it's above threshold + self.assertGreater(actual_offset, ttc.MAX_CROSS_ARM_OFFSET_PCT, + "Should be just above the 1.0% threshold") + + def test_mean_offset_arithmetic_pinned_to_synthetic_values(self): + """Pin mean offset arithmetic with tight tolerance — gap (c). + + Uses deterministic synthetic windows where the mean offset is + computable by hand. A ~5% drift would fail with rtol=1e-9. + """ + # Create windows with same median but different means + # Arm A: all 1000 MHz -> median 1000, mean 1000 + # Arm B: create a mean that's JUST above the 1.0% threshold + # Let's use: 40 samples at 985 MHz, 24 samples at 1015 MHz + # Median of B (sorted positions 32/33): both 985, so median = 985 + # Mean of B: (40*985 + 24*1015) / 64 = (39400 + 24360) / 64 = 63760 / 64 = 996.25 + # Expected mean offset: abs(996.25 - 1000) / 996.25 * 100 = 0.376% + # That's not enough. Let me try a larger skew: + # Arm B: 20 samples at 970 MHz, 44 samples at 1020 MHz + # Median of B (sorted positions 32/33): both 1020, so median = 1020 + # Mean of B: (20*970 + 44*1020) / 64 = (19400 + 44880) / 64 = 64280 / 64 = 1004.375 + # Expected mean offset: abs(1004.375 - 1000) / 1000 * 100 = 0.4375% + # Still not enough. Let me try even more extreme: + # Arm B: 10 samples at 950 MHz, 54 samples at 1010 MHz + # Median of B (sorted positions 32/33): both 1010, so median = 1010 + # Mean of B: (10*950 + 54*1010) / 64 = (9500 + 54540) / 64 = 64040 / 64 = 1000.625 + # Expected mean offset: abs(1000.625 - 1000) / 1000 * 100 = 0.0625% + # Still too small. Let me just make arm B's mean significantly different: + # Arm B: all samples at 1010.1 MHz (same as median test) + # But that would also trigger median offset. Let me try: + # Arm B: 32 samples at 950, 32 samples at 1010 + # Median of B: average of positions 32/33 = (950 + 1010) / 2 = 980 + # Mean of B: (32*950 + 32*1010) / 64 = (30400 + 32320) / 64 = 62720 / 64 = 980 + # Mean equals median, so both would trigger. Let me try: + # Arm B: 30 samples at 940, 34 samples at 1000 + # Median of B: position 32 = 1000 + # Mean of B: (30*940 + 34*1000) / 64 = (28200 + 34000) / 64 = 62200 / 64 = 971.875 + # Expected mean offset: abs(971.875 - 1000) / 971.875 * 100 = 2.89% + # Median offset: abs(1000 - 1000) / 1000 * 100 = 0% + # Perfect! Mean offset triggers, median doesn't. + + wa = mk_window([1000.0] * 64) + wb = mk_window([940.0] * 30 + [1000.0] * 34) + + # Manually compute expected values + expected_median_b = 1000.0 # Position 32 is 1000 + expected_mean_b = (30 * 940.0 + 34 * 1000.0) / 64 + # Expected mean offset: abs(971.875 - 1000) / 971.875 * 100 + # = 28.125 / 971.875 * 100 = 2.8938906752411575% + # The reason text formats this as :.2f, so we expect 2.89% + expected_mean_offset_raw = abs(expected_mean_b - 1000.0) / min(expected_mean_b, 1000.0) * 100.0 + expected_mean_offset_pct = round(expected_mean_offset_raw, 2) # Match the :.2f formatting + + # Verify the window folded correctly + self.assertEqual(wb["median"], expected_median_b, "Median should match hand calculation") + self.assertAlmostEqual(wb["mean"], expected_mean_b, places=9, + msg="Mean should match hand calculation") + + # Get the reasons and extract the mean offset + reasons = ttc.clock_state_reasons(wb, wa) + mean_offset_reasons = [r for r in reasons if "mean offset" in r] + + # Should have exactly one mean offset reason (since means differ enough) + self.assertEqual(len(mean_offset_reasons), 1, + "Should have exactly one mean offset reason") + reason_text = mean_offset_reasons[0] + + # Extract the actual mean offset from the reason text + import re + match = re.search(r"cross-arm mean offset ([\d.]+)%", reason_text) + self.assertIsNotNone(match, "Should find mean offset in reason text") + actual_mean_offset = float(match.group(1)) + + # The reason text is formatted with :.2f, so we expect exact match on the rounded value + self.assertEqual(actual_mean_offset, expected_mean_offset_pct, + msg="Mean offset in reason text should match expected rounded value") + + # Also verify the underlying calculation is correct by checking it's above threshold + self.assertGreater(actual_mean_offset, ttc.MAX_CROSS_ARM_MEAN_OFFSET_PCT, + "Mean offset should be above the 1.0% threshold") + + # Verify median did NOT trigger (medians are equal) + median_offset_reasons = [r for r in reasons if "median offset" in r] + self.assertEqual(len(median_offset_reasons), 0, + "Median offset should not trigger when medians are equal") + + def test_ratio_arithmetic_extremely_small_offset(self): + """Test ratio arithmetic with sub-threshold offset — gap (c) edge case. + + Uses a 0.5% offset (half the threshold) to verify the arithmetic doesn't + drift. A 5% shift would make this 0.525%, still below threshold, so we + verify the exact computed value. + """ + # Arm A: 1000 MHz, Arm B: 1005 MHz (0.5% offset) + wa = mk_window([1000.0] * 64) + wb = mk_window([1005.0] * 64) + + expected_offset = abs(1005.0 - 1000.0) / 1000.0 * 100.0 + + reasons = ttc.clock_state_reasons(wb, wa) + median_offset_reasons = [r for r in reasons if "median offset" in r] + + # Should NOT have a median offset reason (0.5% < 1.0% threshold) + self.assertEqual(len(median_offset_reasons), 0, + "Should have no median offset reason below threshold") + + # But we can still verify the internal calculation by checking that + # the judge doesn't complain about the offset + all_reasons = ttc.clock_state_reasons(wb, wa) + self.assertFalse(any("median offset" in r for r in all_reasons), + "Sub-threshold offset should not produce a reason") + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/bench/tt_clock_state.py b/tools/bench/tt_clock_state.py new file mode 100644 index 000000000..db57dd07e --- /dev/null +++ b/tools/bench/tt_clock_state.py @@ -0,0 +1,335 @@ +#!/usr/bin/env python3 +"""Tenstorrent clock-state sampler and judge — the sibling of gpu_clock_state. + +Implements the ".agents/benchmarking.md" clock contract for TT boards: +AICLK sampling via `tt-smi -s` JSON snapshots, stop-time summary writing, +and cross-window pairing assertions with the NVIDIA helper's thresholds. +Where the platform has no analog the field is recorded NOT APPLICABLE with +a reason, never dropped silently; throttle-unobservability is carried as a +caveat in every output. See .agents/specs/tt-clock-state.md (#2005). + +Exit codes match gpu_clock_state: 0 on success, 1 when refusal reasons exist. +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import statistics +import subprocess +import sys +import time + +#: The board query. `tt-smi -s` snapshots stdout in ~430 ms (measured +#: 2026-08-26, P150 thalia), so --interval 1 is supportable and ~2 Hz is +#: the practical ceiling; ~0.43 of every second is subprocess startup. +TT_SMI = "/home/lu_zero/Sources/tt/.venv/bin/tt-smi" + +#: Device node prefix a healthy leg holds open while it measures. +TT_DEV_PREFIX = "/dev/tenstorrent/" + +BOOT_ID_PATH = pathlib.Path("/proc/sys/kernel/random/boot_id") + +#: Within-run spread ceiling, percentage (gpu_clock_state). +MAX_WITHIN_RUN_SPREAD_PCT = 5.0 + +#: Cross-arm MEDIAN clock offset ceiling, percentage (gpu_clock_state). +MAX_CROSS_ARM_OFFSET_PCT = 1.0 + +#: Cross-arm MEAN offset ceiling — a SEPARATE rule, not a restatement: the +#: excursions sit below the median and do not cancel between arms (#1546), +#: and throughput is an integral over the window. +MAX_CROSS_ARM_MEAN_OFFSET_PCT = 1.0 + +#: Retained busy samples per window (gpu_clock_state). spread_pct over n==1 +#: is definitionally 0.00% — the best score a gate can award — so without a +#: floor the unobserved window outscores the observed one. +MIN_BUSY_SAMPLES = 30 + +#: Busy fraction of the window (gpu_clock_state): 30 busy among 3000 idle +#: still clears the count floor, and this field betrays an orphan sampler. +MIN_BUSY_FRACTION = 0.5 + + +def read_boot_id(path: pathlib.Path = BOOT_ID_PATH) -> str: + return path.read_text(encoding="utf-8").strip() + + +def _hexint(value): + """tt-smi encodes telemetry as hex strings; pass ints through.""" + if isinstance(value, str) and value.startswith("0x"): + try: + return int(value, 16) + except ValueError: + return None + if isinstance(value, int): + return value + return None + + +def sample_once(smi_path: str = TT_SMI, timeout_s: float = 10.0) -> dict | None: + """One tt-smi snapshot -> flattened board-0 sample dict, or None.""" + try: + proc = subprocess.run( + [smi_path, "-s"], capture_output=True, text=True, timeout=timeout_s + ) + except (subprocess.TimeoutExpired, FileNotFoundError, OSError): + return None + if proc.returncode != 0: + return None + try: + doc = json.loads(proc.stdout) + except json.JSONDecodeError: + return None + devices = doc.get("device_info") or [] + host_sw = doc.get("host_sw_vers") or {} + driver = (doc.get("host_info") or {}).get("Driver") + if not devices: + return None + telem = devices[0].get("smbus_telem") or {} + aiclk = _hexint(telem.get("AICLK")) + if aiclk is None: + return None + return { + "t": time.time(), + "aiclk": aiclk, + "arcclk": _hexint(telem.get("ARCCLK")), + "axiclk": _hexint(telem.get("AXICLK")), + "vcore": _hexint(telem.get("VCORE")), + "tdp": _hexint(telem.get("TDP")), + "tdc": _hexint(telem.get("TDC")), + "asic_temp_raw": telem.get("ASIC_TEMPERATURE"), + "fan_rpm": _hexint(telem.get("FAN_RPM")), + "board_id": "{}:{}".format( + telem.get("BOARD_ID_HIGH"), telem.get("BOARD_ID_LOW") + ), + "flash_bundle": telem.get("FLASH_BUNDLE_VERSION"), + "kmd_driver": driver, + "tt_smi_version": host_sw.get("tt_smi"), + "umd_version": host_sw.get("tt_umd"), + } + + +def pid_holds_tt_device(pid: int) -> bool: + """True when pid is alive AND holds an fd on a tenstorrent device node.""" + fd_dir = pathlib.Path(f"/proc/{pid}/fd") + try: + for fd in fd_dir.iterdir(): + try: + if str(fd.readlink()).startswith(TT_DEV_PREFIX): + return True + except OSError: + continue + except OSError: + return False + return False + + +def fold(samples: list[dict], claimed_max_mhz: int | None = None, + provenance: str | None = None, allow_cross_boot: bool = False) -> dict: + """Fold raw samples into the window record shared by judge and tests.""" + aiclks = [s["aiclk"] for s in samples] + median = statistics.median(aiclks) if aiclks else None + identity_keys = ( + "board_id", "flash_bundle", "kmd_driver", "tt_smi_version", + "umd_version", + ) + rec = { + "n_samples": len(samples), + "n": len(aiclks), + "min": min(aiclks) if aiclks else None, + "median": median, + "max": max(aiclks) if aiclks else None, + "mean": ( + sum(aiclks) / len(aiclks) if aiclks else None + ), + "spread_pct": ( + (max(aiclks) - min(aiclks)) / median * 100.0 + if median else 0.0 + ), + "busy_n": sum(1 for s in samples if s.get("busy")), + "idle_n": sum(1 for s in samples if not s.get("busy")), + "busy_fraction": ( + sum(1 for s in samples if s.get("busy")) / len(samples) + if samples else 0.0 + ), + "boot_id": read_boot_id(), + "claimed_max_aiclk_mhz": claimed_max_mhz, + "claimed_max_provenance": provenance, + "not_applicable": { + "applications_clocks_setting": "no TT analog knob exists", + "persistence_mode": "no TT analog knob exists", + "throttle_reasons_live": ( + "no live bitmap exposed by tt-smi; THM_LIMIT_*/VDD_LIMITS " + "are static limits" + ), + }, + "context_medians": {}, + } + last = samples[-1] if samples else {} + for key in identity_keys: + rec[key] = last.get(key) + for key in ("arcclk", "axiclk", "vcore", "tdp", "tdc", "fan_rpm"): + vals = [s[key] for s in samples if isinstance(s.get(key), int)] + rec["context_medians"][key] = statistics.median(vals) if vals else None + rec["_aiclks"] = aiclks + rec["_t"] = [s["t"] for s in samples] + rec["_busy"] = [bool(s.get("busy")) for s in samples] + if allow_cross_boot: + rec["_allow_cross_boot"] = True + rec["_waiver_caveat"] = ( + "boot id waived at request; machine identity compared " + "unconditionally" + ) + return rec + + +def clock_state_reasons(record: dict, other: dict | None = None) -> list[str]: + """Per-window plus cross-window refusal reasons, rule-for-rule.""" + reasons: list[str] = [] + if record.get("median") is None: + reasons.append( + f"idle-or-empty window: n={record.get('n')} median=None" + ) + return reasons + if record["n"] < MIN_BUSY_SAMPLES: + reasons.append( + f"retained busy samples {record['n']} < MIN_BUSY_SAMPLES=" + f"{MIN_BUSY_SAMPLES}" + ) + if record["busy_fraction"] < MIN_BUSY_FRACTION: + reasons.append( + f"busy fraction {record['busy_fraction']:.3f} < MIN_BUSY_FRACTION=" + f"{MIN_BUSY_FRACTION}" + ) + if record["spread_pct"] > MAX_WITHIN_RUN_SPREAD_PCT: + reasons.append( + f"within-run spread {record['spread_pct']:.2f}% > " + f"{MAX_WITHIN_RUN_SPREAD_PCT}%" + ) + if other is None: + return reasons + # ---- cross-window terms ---- + for key, label in ( + ("board_id", "board id"), + ("flash_bundle", "firmware bundle"), + ("kmd_driver", "KMD/driver"), + ("tt_smi_version", "tt-smi version"), + ("umd_version", "UMD version"), + ): + if record.get(key) != other.get(key): + reasons.append( + f"unconditional mismatch {label}: {record.get(key)!r} vs " + f"{other.get(key)!r} — a waived boot is NOT a waived machine" + ) + if record["boot_id"] != other["boot_id"]: + if not (record.get("_allow_cross_boot") or other.get("_allow_cross_boot")): + reasons.append( + f"different boot ids: {str(record['boot_id'])[:8]} vs " + f"{str(other['boot_id'])[:8]}" + ) + elif "_waiver_caveat" not in record and "_waiver_caveat" not in other: + reasons.append( + "cross-boot pair compared WITHOUT the stamped waiver caveat" + ) + denom_med = min(record["median"], other["median"]) + med_off = abs(record["median"] - other["median"]) / denom_med * 100.0 + denom_mean = min(record["mean"], other["mean"]) + mean_off = abs(record["mean"] - other["mean"]) / denom_mean * 100.0 + if med_off > MAX_CROSS_ARM_OFFSET_PCT: + reasons.append( + f"cross-arm median offset {med_off:.2f}% > " + f"{MAX_CROSS_ARM_OFFSET_PCT}%" + ) + if mean_off > MAX_CROSS_ARM_MEAN_OFFSET_PCT: + reasons.append( + f"cross-arm mean offset {mean_off:.2f}% > " + f"{MAX_CROSS_ARM_MEAN_OFFSET_PCT}%" + ) + return reasons + + +def judge(records: list[dict]) -> dict: + """Judge windows pairwise (each against the FIRST); refuse on any reason.""" + comparison: dict = {"reasons": [], "pairs": [], "windows": []} + base = records[0] + for rec in records: + rr = {k: v for k, v in rec.items() if not k.startswith("_")} + rr["throttle_unobservability_caveat"] = ( + "no live TT throttle bitmap exists; a thermal/VDD clamp during " + "the window is UNOBSERVED, see spec tt-clock-state.md" + ) + comparison["windows"].append(rr) + for idx, rec in enumerate(records[1:], start=1): + rs = clock_state_reasons(rec, base) + comparison["pairs"].append({"window": idx, "reasons": rs}) + comparison["reasons"].extend(rs) + solo = clock_state_reasons(records[0]) + comparison["pairs"].insert(0, {"window": 0, "reasons": solo}) + comparison["reasons"] = solo + comparison["reasons"] + return comparison + + +def run_sampler(args: argparse.Namespace) -> int: + """Sample --duration seconds (or until stdin says 'stop'); summarize at STOP. + + The summary exists only after the sampler stops (#1657). A refused or + empty window still writes its evidence file; the exit code carries the + verdict. + """ + samples: list[dict] = [] + deadline = time.monotonic() + args.duration if args.duration else None + use_stdin_stop = args.duration is None + while True: + snap = sample_once(TT_SMI) + if snap is not None: + snap["busy"] = ( + pid_holds_tt_device(args.leg_pid) if args.leg_pid else True + ) + samples.append(snap) + if deadline is not None and time.monotonic() >= deadline: + break + if use_stdin_stop and sys.stdin.readline().strip(): + break + time.sleep(args.interval) + rec = fold(samples, args.claimed_max_aiclk_mhz, args.claimed_max_provenance, + args.allow_cross_boot) + rec["interval_s"] = args.interval + rec["leg_pid"] = args.leg_pid + pathlib.Path(args.out).write_text(json.dumps(rec, indent=1) + "\n") + print(f"wrote {args.out}: n={rec['n']} busy={rec['busy_n']}") + return 1 if clock_state_reasons(rec) else 0 + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__) + sub = ap.add_subparsers(dest="cmd", required=True) + + sp = sub.add_parser("sample", help="sample until stdin closes/--duration") + sp.add_argument("--out", required=True) + sp.add_argument("--interval", type=float, default=1.0) + sp.add_argument("--duration", type=float, default=None) + sp.add_argument("--leg-pid", type=int, default=None) + sp.add_argument("--claimed-max-aiclk-mhz", type=int, default=None) + sp.add_argument("--claimed-max-provenance", default=None) + sp.add_argument("--allow-cross-boot", action="store_true") + + jp = sub.add_parser("judge", help="judge 2+ window summaries pairwise") + jp.add_argument("summaries", nargs="+") + + args = ap.parse_args(argv) + if args.cmd == "sample": + return run_sampler(args) + records = [] + for path in args.summaries: + doc = json.loads(pathlib.Path(path).read_text(encoding="utf-8")) + doc["_path"] = path + records.append(doc) + result = judge(records) + print(json.dumps(result, indent=1)) + return 0 if not result["reasons"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/bench/tt_refold_busy.py b/tools/bench/tt_refold_busy.py new file mode 100644 index 000000000..eacaeb4d2 --- /dev/null +++ b/tools/bench/tt_refold_busy.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Refold tt_clock_state windows to their busy-only slices and re-judge. + +The Blackhole P150 AICLK governor is two-state: 800 MHz idle, pegged at the +claimed max under load. A raw window therefore mixes the pre-open idle head +with pegged compute and refuses on within-run spread — correctly, because +the NVIDIA rule describes quasi-continuous clocks. The honest attribution +for such a platform is the BUSY SLICE: samples whose busy flag was recorded +live at sample time (leg pid holding /dev/tenstorrent fds), a criterion +independent of the outcome clock values. This tool rebuilds those records +from a summary's _t/_aiclks/_busy columns and runs the same judge. + +Usage: tt_refold_busy.py SUMMARY.json [SUMMARY.json ...] +Writes one JSON verdict to stdout; exit 0 iff no refusal reasons. +""" + +from __future__ import annotations + +import json +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import tt_clock_state as ttc # noqa: E402 + + +def refold(summary: dict) -> dict: + samples = [ + {"t": t, "aiclk": a, "busy": True} + for t, a, b in zip( + summary["_t"], summary["_aiclks"], summary["_busy"] + ) + if b + ] + return ttc.fold( + samples, + summary.get("claimed_max_aiclk_mhz"), + summary.get("claimed_max_provenance"), + allow_cross_boot=bool(summary.get("_allow_cross_boot")), + ) + + +def main(argv: list[str]) -> int: + records = [] + for path in argv: + s = json.loads(pathlib.Path(path).read_text(encoding="utf-8")) + records.append(refold(s)) + result = ttc.judge(records) + print(json.dumps(result, indent=1)) + return 0 if not result["reasons"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:]))