feat(BACKEND-TENSTORRENT): tt_clock_state sampler/judge + busy-slice refold (#2005); #2003 re-adjudicated clock-attributed - #2006
Merged
Conversation
…ck_state (mudler#2005) Every Tenstorrent speed figure on record, including mudler#2003's inversion, is clock-unattributed because the NVIDIA-only helper has no TT twin and nothing else samples a board clock during a leg. This spec commits FIRST (spec-first rule) and defines the contract: AICLK sampling via 430 ms `tt-smi -s` snapshots at 1 Hz, stop-time summaries (mudler#1657 shape), the NVIDIA thresholds copied constant-for-constant with their names, explicit NOT APPLICABLE fields where the platform has no analog, throttle-unobservability stated, busy proxied by the measured pid holding /dev/tenstorrent fds. First consumer is mudler#2003: per-arm windows around order-alternated legs before any hybrid-path per-op delta effort is spent. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…sted suite (mudler#2005) The TT twin of tools/bench/gpu_clock_state.py, contract mirrored rule-for-rule with the NVIDIA thresholds copied constant-for-constant and by name: MAX_WITHIN_RUN_SPREAD_PCT=5, MAX_CROSS_ARM_OFFSET_PCT=1, MAX_CROSS_ARM_MEAN_OFFSET_PCT=1 (separate rule, mudler#1546), MIN_BUSY_SAMPLES=30, MIN_BUSY_FRACTION=0.5. AICLK comes from 430 ms `tt-smi -s` JSON snapshots at 1 Hz; the summary exists only when the sampler STOPS (mudler#1657 shape) and an empty or refused window still writes its evidence while exiting nonzero. Where the platform has no analog the field states itself rather than disappearing: applications/persistence knobs are NOT APPLICABLE strings, the live throttle bitmap does not exist so every judged window carries an explicit throttle-unobservability caveat, and claimed max clock is an argument plus provenance because no tt-smi command exposes it. Board id, firmware bundle, KMD and smi/umd versions are compared UNCONDITIONALLY even under a stamped boot waiver — a waived boot is not a waived machine. Busy is proxied by the measured leg pid holding /dev/tenstorrent fds, checked live per interval; orphan samplers therefore decay toward the busy-fraction refusal instead of passing silently. tests/tools/test_tt_clock_state.py pins every threshold and both exit semantics on synthetic fixtures; fixing its first red run caught fold() missing the median key outright. CLI judge verified live: a 3 s n=3 sanity window refuses with MIN_BUSY_SAMPLES exactly as an unobserved window should. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…ted — inversion stands at governor parity; tt_clock_state first wired use (mudler#2005) The tool landed same-day and its first consumer was the morning's unattributed result. Same binary, workload, alternation and lock discipline; one sampler window per arm around six order-alternated legs. Throughput: default host-free eager median 10.880 tok/s vs opt-out 13.645 (ratio 1.254) — reproduces the morning's 10.822 vs 13.369. Attribution finding: the Blackhole P150 AICLK governor is two-state on this telemetry, 800 idle vs pegged under load, so all six raw windows refuse within-run spread at 40.74% — kept verbatim in the evidence log, because loosening the NVIDIA rule to fit one board is exactly the failure mode the protocol forbids. Attribution instead rides the live-recorded busy column: pid-held /dev/tenstorrent fds per interval, recorded before any outcome value existed. tt_refold_busy.py rebuilds those slices and every window shows ONE distinct AICLK {1350}, spread 0.00%, cross-arm offsets 0/0, judge PASS rc=0. Both arms computed at identical cap clocks; the inverted ratio is a real path difference and mudler#2003 stays open for the per-op delta. Also on this branch: the consolidated host-free gap row (supersedes either single-session wording regardless of merge order against mudler#2004), the mudler#2005 index row naming its owed items — verified claimed-max pin, in-process pyluwen sampling, and the spread-scope policy decision — and the spec's Evidence/Owed sections carrying both findings and the explicit never-loosen-a-threshold note. Evidence verbatim: docs/bench-evidence/tt-p150-clock-attributed-20260826.log FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…0.8B decodes at 0.089 tok/s warm, ~100x under a 7B dense on the same board (mudler#1715) Nobody had measured throughput for this family on Tenstorrent: W2a delivered correctness (allow-list, golden pairs, 16/16 gate) and the speed question sat unasked behind it. Production entry point, fresh-process probes under one lock hold after a JIT-caching discard proc. Warm steady state: 0.089 tok/s (~11.2 s/token), reproduced exactly across runs, arms, and a 96-token generation (0.084). The opt-out arm is IDENTICAL to the default (0.080/0.089/0.089 vs 0.080/0.090/0.089), so the mudler#1604/mudler#2003 polarity is irrelevant for this family — the wall lives deeper than the decode-graph machinery. Output coherent. Ranked in the record with reasoning: a per-layer host/reference fallback inside the hybrid-GDN stack (~400 ms/layer-token over ~28 layers) leads; the host-staged KV lane alone cannot explain three orders of magnitude (Mistral serves 9.8 tok/s through it); GDN kernel-mode differences are milliseconds against an 11-second step. Figures unattributed by sampler (minute-scale duty cycles make cap-pegging near certain, which only sharpens the gap). Owns its next lever inside mudler#1715: profile ONE eager decode step and name the dominant op before registering more kernels blindly. Captured tracing stays blocked behind mudler#1625, so the first pass is eager-side. Evidence verbatim: docs/bench-evidence/tt-qwen35-first-speed-20260826.log FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…eed axis (mudler#1715) docs/benchmarks/open-gaps.md carried Tenstorrent rows for the backend floor, Mistral-7B and the host-free inversion, but nothing for the Qwen3.5-0.8B measurement cb82003 recorded only in benchmark-record.md — the public gap surface could not answer "how fast is the family on TT". The new row states the measured number with its reproduction span (0.089 tok/s warm, arms identical), names the polarity irrelevance against mudler#1604/mudler#2003 so nobody re-runs that A/B expecting separation, ranks the leading hypothesis (per-layer host/reference fallback in the hybrid-GDN stack) instead of asserting it, and owns the next lever inside mudler#1715: profile ONE eager decode step, captured tracing blocked behind mudler#1625. Record edit riding the pull request whose change produced the fact, not a records-only branch. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
lu-zero
force-pushed
the
bench/tt-clock-state
branch
from
August 26, 2026 20:54
f2467b6 to
8adcbd4
Compare
) Prior review found three mutations the existing test suite failed to detect: (a) Widening the refold busy-filter so a window with genuine mid-window spread no longer collapses to a single-value set went GREEN (undetected). Added test_refold_busy_genuine_mid_window_spread_preserves_cap that constructs representative raw-window input with idle head (800 MHz) and busy compute (1350 MHz), feeds through tt_refold_busy's collapse logic, and asserts the collapsed set is exactly {1350} with zero spread. RED under mutation `if b` → `if b or a >= 800`: AssertionError: 800 != 1350. (c) Shifting cross-arm offset/ratio arithmetic ~5% went GREEN. Added test_median_offset_arithmetic_pinned_to_synthetic_values and test_mean_offset_arithmetic_pinned_to_synthetic_values that pin the ratio math against deterministic synthetic windows where the expected ratio is derivable by hand with tight tolerance. RED under mutation `* 100.0` → `* 100.0 * 1.05`: AssertionError: 1.06 != 1.01 (median) and 3.04 != 2.89 (mean). (e) tt_refold_busy.py had ZERO references from the test file. Added test_refold_busy_import_and_exercise that imports and exercises the real entry point, satisfying gap (e). All new tests follow red-first discipline: written against current code (confirming green baseline), then mutations hand-applied and observed RED, then byte-for-byte restore confirmed (SHA256 e09a86f5... and e8afbe33... match before/after). Final green: 22/22 tests pass. Closes the three red-first gaps the first fresh-review pass found on PR mudler#2006: mutations (a) refold-collapse and (c) ... FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai-glm-5.3-flash [maki]
Collaborator
Author
|
Rebased onto origin/main
Stacking note: this PR supersedes parts of #2004's morning-unattributed wording (host-free row, benchmark-record entries) with the clock-attributed re-adjudication. Mechanical consequence, verified by merge simulation in both directions in review: git conflicts on those record lines whichever lands first. Intended order: #2004 first (self-consistent morning snapshot), then this PR rebases over it taking the target rows and reapplying the scoped clock-attributed edits, per the keyed-record rule. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
record(BACKEND-TENSTORRENT): #2003 re-adjudicated clock-attributed — inversion stands at governor parity; tt_clock_state first wired use (#2005)
The tool landed same-day and its first consumer was the morning's
unattributed result. Same binary, workload, alternation and lock discipline;
one sampler window per arm around six order-alternated legs.
Throughput: default host-free eager median 10.880 tok/s vs opt-out 13.645
(ratio 1.254) — reproduces the morning's 10.822 vs 13.369.
Attribution finding: the Blackhole P150 AICLK governor is two-state on this
telemetry, 800 idle vs pegged under load, so all six raw windows refuse
within-run spread at 40.74% — kept verbatim in the evidence log, because
loosening the NVIDIA rule to fit one board is exactly the failure mode the
protocol forbids. Attribution instead rides the live-recorded busy column:
pid-held /dev/tenstorrent fds per interval, recorded before any outcome
value existed. tt_refold_busy.py rebuilds those slices and every window
shows ONE distinct AICLK {1350}, spread 0.00%, cross-arm offsets 0/0,
judge PASS rc=0. Both arms computed at identical cap clocks; the inverted
ratio is a real path difference and #2003 stays open for the per-op delta.
Also on this branch: the consolidated host-free gap row (supersedes either
single-session wording regardless of merge order against #2004), the #2005
index row naming its owed items — verified claimed-max pin, in-process
pyluwen sampling, and the spread-scope policy decision — and the spec's
Evidence/Owed sections carrying both findings and the explicit
never-loosen-a-threshold note.
Evidence verbatim: docs/bench-evidence/tt-p150-clock-attributed-20260826.log
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]