Skip to content

feat(BACKEND-TENSTORRENT): tt_clock_state sampler/judge + busy-slice refold (#2005); #2003 re-adjudicated clock-attributed - #2006

Merged
lu-zero merged 6 commits into
mudler:mainfrom
lu-zero:bench/tt-clock-state
Aug 26, 2026
Merged

feat(BACKEND-TENSTORRENT): tt_clock_state sampler/judge + busy-slice refold (#2005); #2003 re-adjudicated clock-attributed#2006
lu-zero merged 6 commits into
mudler:mainfrom
lu-zero:bench/tt-clock-state

Conversation

@lu-zero

@lu-zero lu-zero commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

record(BACKEND-TENSTORRENT): #2003 re-adjudicated clock-attributed — inversion stands at governor parity; tt_clock_state first wired use (#2005)

The tool landed same-day and its first consumer was the morning's
unattributed result. Same binary, workload, alternation and lock discipline;
one sampler window per arm around six order-alternated legs.

Throughput: default host-free eager median 10.880 tok/s vs opt-out 13.645
(ratio 1.254) — reproduces the morning's 10.822 vs 13.369.

Attribution finding: the Blackhole P150 AICLK governor is two-state on this
telemetry, 800 idle vs pegged under load, so all six raw windows refuse
within-run spread at 40.74% — kept verbatim in the evidence log, because
loosening the NVIDIA rule to fit one board is exactly the failure mode the
protocol forbids. Attribution instead rides the live-recorded busy column:
pid-held /dev/tenstorrent fds per interval, recorded before any outcome
value existed. tt_refold_busy.py rebuilds those slices and every window
shows ONE distinct AICLK {1350}, spread 0.00%, cross-arm offsets 0/0,
judge PASS rc=0. Both arms computed at identical cap clocks; the inverted
ratio is a real path difference and #2003 stays open for the per-op delta.

Also on this branch: the consolidated host-free gap row (supersedes either
single-session wording regardless of merge order against #2004), the #2005
index row naming its owed items — verified claimed-max pin, in-process
pyluwen sampling, and the spread-scope policy decision — and the spec's
Evidence/Owed sections carrying both findings and the explicit
never-loosen-a-threshold note.

Evidence verbatim: docs/bench-evidence/tt-p150-clock-attributed-20260826.log

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]

…ck_state (mudler#2005)

Every Tenstorrent speed figure on record, including mudler#2003's inversion, is
clock-unattributed because the NVIDIA-only helper has no TT twin and nothing
else samples a board clock during a leg. This spec commits FIRST (spec-first
rule) and defines the contract: AICLK sampling via 430 ms `tt-smi -s`
snapshots at 1 Hz, stop-time summaries (mudler#1657 shape), the NVIDIA thresholds
copied constant-for-constant with their names, explicit NOT APPLICABLE
fields where the platform has no analog, throttle-unobservability stated,
busy proxied by the measured pid holding /dev/tenstorrent fds.

First consumer is mudler#2003: per-arm windows around order-alternated legs before
any hybrid-path per-op delta effort is spent.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…sted suite (mudler#2005)

The TT twin of tools/bench/gpu_clock_state.py, contract mirrored
rule-for-rule with the NVIDIA thresholds copied constant-for-constant and by
name: MAX_WITHIN_RUN_SPREAD_PCT=5, MAX_CROSS_ARM_OFFSET_PCT=1,
MAX_CROSS_ARM_MEAN_OFFSET_PCT=1 (separate rule, mudler#1546),
MIN_BUSY_SAMPLES=30, MIN_BUSY_FRACTION=0.5. AICLK comes from 430 ms
`tt-smi -s` JSON snapshots at 1 Hz; the summary exists only when the
sampler STOPS (mudler#1657 shape) and an empty or refused window still writes its
evidence while exiting nonzero.

Where the platform has no analog the field states itself rather than
disappearing: applications/persistence knobs are NOT APPLICABLE strings,
the live throttle bitmap does not exist so every judged window carries an
explicit throttle-unobservability caveat, and claimed max clock is an
argument plus provenance because no tt-smi command exposes it. Board id,
firmware bundle, KMD and smi/umd versions are compared UNCONDITIONALLY even
under a stamped boot waiver — a waived boot is not a waived machine.

Busy is proxied by the measured leg pid holding /dev/tenstorrent fds,
checked live per interval; orphan samplers therefore decay toward the
busy-fraction refusal instead of passing silently.

tests/tools/test_tt_clock_state.py pins every threshold and both exit
semantics on synthetic fixtures; fixing its first red run caught fold()
missing the median key outright. CLI judge verified live: a 3 s n=3
sanity window refuses with MIN_BUSY_SAMPLES exactly as an unobserved
window should.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…ted — inversion stands at governor parity; tt_clock_state first wired use (mudler#2005)

The tool landed same-day and its first consumer was the morning's
unattributed result. Same binary, workload, alternation and lock discipline;
one sampler window per arm around six order-alternated legs.

Throughput: default host-free eager median 10.880 tok/s vs opt-out 13.645
(ratio 1.254) — reproduces the morning's 10.822 vs 13.369.

Attribution finding: the Blackhole P150 AICLK governor is two-state on this
telemetry, 800 idle vs pegged under load, so all six raw windows refuse
within-run spread at 40.74% — kept verbatim in the evidence log, because
loosening the NVIDIA rule to fit one board is exactly the failure mode the
protocol forbids. Attribution instead rides the live-recorded busy column:
pid-held /dev/tenstorrent fds per interval, recorded before any outcome
value existed. tt_refold_busy.py rebuilds those slices and every window
shows ONE distinct AICLK {1350}, spread 0.00%, cross-arm offsets 0/0,
judge PASS rc=0. Both arms computed at identical cap clocks; the inverted
ratio is a real path difference and mudler#2003 stays open for the per-op delta.

Also on this branch: the consolidated host-free gap row (supersedes either
single-session wording regardless of merge order against mudler#2004), the mudler#2005
index row naming its owed items — verified claimed-max pin, in-process
pyluwen sampling, and the spread-scope policy decision — and the spec's
Evidence/Owed sections carrying both findings and the explicit
never-loosen-a-threshold note.

Evidence verbatim: docs/bench-evidence/tt-p150-clock-attributed-20260826.log

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…0.8B decodes at 0.089 tok/s warm, ~100x under a 7B dense on the same board (mudler#1715)

Nobody had measured throughput for this family on Tenstorrent: W2a delivered
correctness (allow-list, golden pairs, 16/16 gate) and the speed question sat
unasked behind it. Production entry point, fresh-process probes under one
lock hold after a JIT-caching discard proc.

Warm steady state: 0.089 tok/s (~11.2 s/token), reproduced exactly across
runs, arms, and a 96-token generation (0.084). The opt-out arm is IDENTICAL
to the default (0.080/0.089/0.089 vs 0.080/0.090/0.089), so the mudler#1604/mudler#2003
polarity is irrelevant for this family — the wall lives deeper than the
decode-graph machinery. Output coherent.

Ranked in the record with reasoning: a per-layer host/reference fallback
inside the hybrid-GDN stack (~400 ms/layer-token over ~28 layers) leads;
the host-staged KV lane alone cannot explain three orders of magnitude
(Mistral serves 9.8 tok/s through it); GDN kernel-mode differences are
milliseconds against an 11-second step. Figures unattributed by sampler
(minute-scale duty cycles make cap-pegging near certain, which only
sharpens the gap).

Owns its next lever inside mudler#1715: profile ONE eager decode step and name
the dominant op before registering more kernels blindly. Captured tracing
stays blocked behind mudler#1625, so the first pass is eager-side.

Evidence verbatim: docs/bench-evidence/tt-qwen35-first-speed-20260826.log

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
…eed axis (mudler#1715)

docs/benchmarks/open-gaps.md carried Tenstorrent rows for the backend floor,
Mistral-7B and the host-free inversion, but nothing for the Qwen3.5-0.8B
measurement cb82003 recorded only in benchmark-record.md — the public gap
surface could not answer "how fast is the family on TT". The new row states
the measured number with its reproduction span (0.089 tok/s warm, arms
identical), names the polarity irrelevance against mudler#1604/mudler#2003 so nobody re-runs
that A/B expecting separation, ranks the leading hypothesis (per-layer
host/reference fallback in the hybrid-GDN stack) instead of asserting it, and
owns the next lever inside mudler#1715: profile ONE eager decode step, captured
tracing blocked behind mudler#1625.

Record edit riding the pull request whose change produced the fact, not a
records-only branch.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
@lu-zero
lu-zero force-pushed the bench/tt-clock-state branch from f2467b6 to 8adcbd4 Compare August 26, 2026 20:54
)

Prior review found three mutations the existing test suite failed to detect:

(a) Widening the refold busy-filter so a window with genuine mid-window
    spread no longer collapses to a single-value set went GREEN (undetected).
    Added test_refold_busy_genuine_mid_window_spread_preserves_cap that
    constructs representative raw-window input with idle head (800 MHz) and
    busy compute (1350 MHz), feeds through tt_refold_busy's collapse logic,
    and asserts the collapsed set is exactly {1350} with zero spread. RED
    under mutation `if b` → `if b or a >= 800`: AssertionError: 800 != 1350.

(c) Shifting cross-arm offset/ratio arithmetic ~5% went GREEN.
    Added test_median_offset_arithmetic_pinned_to_synthetic_values and
    test_mean_offset_arithmetic_pinned_to_synthetic_values that pin the
    ratio math against deterministic synthetic windows where the expected
    ratio is derivable by hand with tight tolerance. RED under mutation
    `* 100.0` → `* 100.0 * 1.05`: AssertionError: 1.06 != 1.01 (median)
    and 3.04 != 2.89 (mean).

(e) tt_refold_busy.py had ZERO references from the test file.
    Added test_refold_busy_import_and_exercise that imports and exercises
    the real entry point, satisfying gap (e).

All new tests follow red-first discipline: written against current code
(confirming green baseline), then mutations hand-applied and observed RED,
then byte-for-byte restore confirmed (SHA256 e09a86f5... and e8afbe33...
match before/after). Final green: 22/22 tests pass.

Closes the three red-first gaps the first fresh-review pass found on PR mudler#2006: mutations (a) refold-collapse and (c) ...

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai-glm-5.3-flash [maki]
@lu-zero

lu-zero commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto origin/main 2a42cb369 (KV-GDN-STATE-BUDGET), clean. Head now b3182637b:

Stacking note: this PR supersedes parts of #2004's morning-unattributed wording (host-free row, benchmark-record entries) with the clock-attributed re-adjudication. Mechanical consequence, verified by merge simulation in both directions in review: git conflicts on those record lines whichever lands first. Intended order: #2004 first (self-consistent morning snapshot), then this PR rebases over it taking the target rows and reapplying the scoped clock-attributed edits, per the keyed-record rule.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant