27B throughput measured, and the collapse alarm can now attribute it (#440/#441) - #2341
Merged
Conversation
…441) MEASURED 2026-08-20, 6 samples of 150 tok on the live M5 lane: 4.74 / 5.25 / 6.55 / 6.57 / 6.66 / 6.67 tok/s, median 6.56, at a 22,528 PER-SLOT window with `busy_slots = 2` of 4 on EVERY sample. NOT written into `tokens_per_second`, deliberately. That field is what #441's collapse alarm reads as the SINGLE-STREAM expectation, and 6.56 is a contended rate — `ThroughputBaseline`'s own doc says "single sequence". Putting it there would drop the alarm's floor from ~4.3 to ~1.6 tok/s and let a genuine collapse pass silently. The pinned 17.2 stays. Both numbers are true; they measure different machines. THE GAP IS THE POINT: the sentinel cannot currently tell "contended" from "degraded" because nothing records concurrency at measurement time. That is the sibling of the window axis landed in #2339, and it is #441's remaining half. MTP held 83.9-84.8% draft acceptance across all six, and 80.3-84.3% across three earlier samples on a different prompt. Acceptance is FLAT while t/s moves 1.4x, so the draft head is not the variance source. Also corrects my own earlier reading: I took `-c 89280` off the lane argv and called it the live window. It is the TOTAL across 4 slots; per-slot is 22,528 against the catalog's 19,712 — 1.14x, not the 4.5x I claimed. The window was never the explanation. cargo check exit 0; model_registry tests exit 0 (both read by exit code). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…n attribute it (#441) The alarm compared a live decode rate against a SINGLE-STREAM expectation and named three suspects — CPU fallback, pager thrash, GPU contention — with no evidence for any of them. Measured 2026-08-20 on the 27B: 6.56 t/s against a 17.2 pinned expectation, ratio 0.38, with TWO model calls in flight. That is real contention, and a reader handed only the ratio would have gone hunting a CPU fallback that does not exist. Now the warning carries `inflight_model_calls` and says what it means: >1 in flight and the expectation is not like-for-like; at 1 in flight the original suspects stand. ANNOTATE, NOT GATE — deliberately. This alarm's own contract lists "contended GPU" as something it EXISTS to catch, so suppressing on concurrency would defeat its purpose and weaken a guard. The firing decision is untouched; only the evidence attached to it changed. REUSES the existing gauge rather than counting again: `resource_admission::inflight_model_calls()` is already public, already maintained by an RAII guard, and its doc says it "reflects exactly the model-call window (lane-queue + prefill + decode)" — precisely the axis needed. The concurrency sibling of the window axis landed in #2339. cargo check 0; throughput + resource_admission tests 0 (read by exit code). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two halves of the same finding. Supersedes #2340 (its commit is included here).
Measured (#440)
Six samples on the live M5 lane: median 6.56 tok/s, 22,528 per-slot window,
busy_slots = 2of 4 every time. MTP held 83.9–84.8% draft acceptance throughout — flat while t/s moved 1.4×, so the draft head is not the variance source.Deliberately not written into
tokens_per_second: that field is the single-stream expectation the alarm reads, and 6.56 is contended. Recording it would drop the collapse floor from ~4.3 to ~1.6 t/s and let a real collapse pass silently. The pinned 17.2 stays.Attributable (#441)
The alarm compared that rate against the single-stream expectation and named three suspects — CPU fallback, pager thrash, GPU contention — with evidence for none. Ratio 0.38 with 2 calls in flight is real contention, and a reader handed only the ratio goes hunting a defect that isn't there.
Now the warning carries
inflight_model_calls: >1 in flight → not like-for-like; 1 in flight → the original suspects stand.Annotate, not gate. The alarm's own contract lists "contended GPU" as something it exists to catch, so suppressing on concurrency would defeat it. Firing decision untouched.
Reuses the existing gauge —
resource_admission::inflight_model_calls()is already public and RAII-maintained, doc says it "reflects exactly the model-call window (lane-queue + prefill + decode)". No second counter. Concurrency sibling of the window axis in #2339.Correction
-c 89280is the total across 4 slots, not the live window. Per-slot is 22,528 vs the catalog's 19,712 — 1.14×, not the 4.5× I claimed earlier. The window was never the explanation; recorded in the code comment where the next reader hits it.cargo check 0; throughput + resource_admission + model_registry tests 0 (all by exit code).
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo