From de2edb761b133af53ed5d98c349137a5c2e67134 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Mon, 26 Oct 2026 15:26:09 +0000 Subject: [PATCH 1/7] docs(openspec): propose engine generation speed metric --- .../engine-generation-speed/.openspec.yaml | 2 + .../changes/engine-generation-speed/design.md | 38 +++++++ .../engine-generation-speed/proposal.md | 32 ++++++ .../specs/daemon-api/spec.md | 38 +++++++ .../specs/engine-activity/spec.md | 45 ++++++++ .../specs/engine-metrics/spec.md | 24 +++++ .../specs/remote-metrics-bar-format/spec.md | 100 ++++++++++++++++++ .../changes/engine-generation-speed/tasks.md | 29 +++++ 8 files changed, 308 insertions(+) create mode 100644 openspec/changes/engine-generation-speed/.openspec.yaml create mode 100644 openspec/changes/engine-generation-speed/design.md create mode 100644 openspec/changes/engine-generation-speed/proposal.md create mode 100644 openspec/changes/engine-generation-speed/specs/daemon-api/spec.md create mode 100644 openspec/changes/engine-generation-speed/specs/engine-activity/spec.md create mode 100644 openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md create mode 100644 openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md create mode 100644 openspec/changes/engine-generation-speed/tasks.md diff --git a/openspec/changes/engine-generation-speed/.openspec.yaml b/openspec/changes/engine-generation-speed/.openspec.yaml new file mode 100644 index 00000000..d58ba023 --- /dev/null +++ b/openspec/changes/engine-generation-speed/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-26 diff --git a/openspec/changes/engine-generation-speed/design.md b/openspec/changes/engine-generation-speed/design.md new file mode 100644 index 00000000..cde798ca --- /dev/null +++ b/openspec/changes/engine-generation-speed/design.md @@ -0,0 +1,38 @@ +# Design + +## Context + +The daemon already captures every line the engine writes (stderr straight to the log file; stdout through the pty normaliser) and already runs a sampler tick (`SampleActivity`/`systemSampleOnce`) that builds the retained `HistorySample` window and the current host reading the metrics endpoint answers from. llama.cpp prints its instantaneous generation rate (`tg = 3.87 t/s`) into stderr's `print_timing` lines, which today reach the log file and stop there. The stats shape (`internal/metrics.Stats`, `HistorySample`) is a wire contract with the fleet client and the TypeScript control plane, policed against `docs/openapi.yaml` by `openapi_test.go`. + +## Goals / Non-Goals + +**Goals:** +- Surface the engine's reported generation speed as an optional current reading and an optional per-tick history point, end to end: engine output → daemon → `/v1/metrics` → bar/combined/tile/gauge/table rendering and the serve view. +- Sparkline heights relative to the window's own high watermark; default colour throughout. +- Relabel the GPU utilisation series `GPU util` → `GPU`. + +**Non-Goals:** +- Computing speed from token-counter deltas ourselves (the engine's own figure is the spec'd source; a derived figure would have to define what a "generation interval" is across concurrent slots). +- vLLM/oMLX speed support (no `tg`-style line to parse; their absence follows the usual absent-stat path). +- Historical speed across restarts, or a persisted window beyond the existing 10-minute buffer. + +## Decisions + +**Parse where the bytes already become lines.** A `lineTee` writer wraps the supervisor's log-file sink: raw bytes flow to the file untouched; complete lines are additionally handed to an injected `Supervisor.OnOutput func(line string)` hook. One tee covers both capture paths — stderr (where `print_timing` lands) is written through it, and the pty pump's normalised output is written through it. The alternative — a log-file tailer that re-reads what was just written — duplicates the offset bookkeeping, sees bytes the pump may not have committed yet, and costs a second reader per daemon for a figure that is already in memory. `StartEngine` sets `Sup.OnOutput` to the daemon's own observer beside each start, so both API hosts (daemon and serve --api/view) wire it without per-command plumbing; `OnOutput` is therefore the daemon's property once a supervisor has a daemon. A partial line is buffered with a size cap (64 KiB) — a runaway line drops the buffer rather than growing it — because log lines worth parsing are ~100 bytes. + +**Parser in `internal/metrics`, engine-shaped regex.** `ParseGenerationSpeed(line) (float64, bool)` matches `tg\s*=\s*\s*t/s`: deliberately NOT matching `tg_3s =` (the underscore breaks the `\s*=` that must follow `tg`) nor prompt-evaluation rates (different label). Reject NaN/Inf/negative — llama.cpp prints `-nan` before a slot's first decode — and require a plausible rate (`> 0`). Engines that never print the line simply never yield a speed; no engine-family gating, so a future engine with the same spelling works for free. + +**One tracker, two lifetimes.** A `speedTrack` on the daemon (mutex, last value, last-observed time) answers two questions with different clocks: the sampler tick attaches a history point when a figure arrived since the previous tick (points are observations, never continuations — a quiet tick's sample just lacks the field); the metrics handler answers the current reading only while the engine runs AND the last figure is no older than the sampling interval (spec: "since the last sampling tick"). Both read through the daemon's injected clock, so tests age the figure without sleeping. `StartEngine` resets the tracker, the tick baseline, and the history together, so one engine's rate never rides the next engine's report. + +**Speed is a `barSeries`, drawn by its own rule.** The series list gains a trailing `Speed` series built from the current figure and the history's speed points; `renderStatBars`/`renderStatCombined` draw it with a speed-specific glyph block — value scaled to the window maximum (pool-then-scale, so downsampling keeps the watermark honest) mapping to `▇`, never `█` — and always uncoloured, because the 80/90 thresholds only mean something on a 0–100 axis; a rate has no ceiling. The figure renders `%6.2f t/s`. `renderStatGauges` draws the label and figure with no gauge block. The combined serve-view row keeps its bar half and leaves the gauge half blank; the table format gets a `speed:` key-value line beside the token counters. Callers keep taking explicit args from `metrics.Stats` rather than restructuring signatures: the existing surfaces already thread `(cpu, mem, gpus, history)` and gain `(speed)`. + +**GPU relabel is one string.** `barSeriesList` builds `"GPU %d util"` today; the util series becomes `"GPU"` for one GPU, `"GPU N"` for many, the memory series unchanged. Every surface draws from this list, so tiles, detail, bar, gauge and serve view move together. + +**Wire shape.** `Stats.speed *float64` and `HistorySample.s *float64` (one-letter, for the SSM 4KB budget the history fields already respect), both `omitempty`; `docs/openapi.yaml` gains both properties — `TestOpenAPISchemasMatchGoTypes` fails otherwise. Consumers that never see the field (older CLIs, the TS Lambdas) are unaffected. + +## Risks / Trade-offs + +- **Dashboard tiles are fixed-height** (`dashTileH`): a node with GPU series and a running engine may clip the token counter block when the Speed row joins. Accepted — the detail view is unclipped, and clipping keeps the tile grid honest; flag in review if the clip reads badly in practice. +- **Only engines that self-report are covered.** vLLM's `/metrics` exposes no instantaneous rate; its users see no row. Deriving speed from counter deltas is the fallback if demand appears. +- **Tee adds a copy path on the engine's output.** The copy is one `Write` of already-buffered bytes to an in-memory line splitter; the pump never blocks on the parser (regex over ~100 bytes), and a wedged observer cannot back-pressure the engine because the tee writes the file first and the hook after. +- **`--watch`/poll staleness:** between sampler ticks the current figure is up to one interval old — bounded staleness is already the documented model for every other metric in the reply. diff --git a/openspec/changes/engine-generation-speed/proposal.md b/openspec/changes/engine-generation-speed/proposal.md new file mode 100644 index 00000000..2053e62b --- /dev/null +++ b/openspec/changes/engine-generation-speed/proposal.md @@ -0,0 +1,32 @@ +# Proposal + +## Why + +Operators watching a serving engine have no idea how fast it is actually generating tokens. llama.cpp already prints the figure in its structured log lines (`slot print_timing: ... tg = 3.87 t/s, tg_3s = 5.29 t/s`), but spinloop throws it away — the metrics views show host pressure (CPU/RAM/GPU) and cumulative counters, never the throughput the machine exists to produce. Surfacing it turns "is this engine any good?" from a guess into a glance. + +## What Changes + +- The daemon observes the generation speed (`tg`) in the engine's captured output lines, parses it, and reports it as a new optional metric: a current reading on the stats reply plus one point per sampling tick in the retained history. Engines that never print the figure report neither, on the usual "absent, not zero" terms. +- Every surface that renders the resource series gains a `Speed` row — `Speed 12.34 t/s ▂▃▂▂▂▁▂▂▁▂▇▇▃▂▂` — with a sparkline whose glyph heights scale to the high watermark of the values currently shown in the window, and whose points all draw in the terminal's default colour (the 80/90% threshold colours are utilisation's, not speed's). The gauge and table formats carry the figure without a sparkline; JSON carries both new optional fields. +- The GPU utilisation series is relabelled: `GPU util`/`GPU N util` become `GPU`/`GPU N` (the memory series keeps `GPU mem`/`GPU N mem`). + +## Capabilities + +### New Capabilities + +_None — all behaviour extends existing specs._ + +### Modified Capabilities + +- `engine-metrics`: engine stats collection gains generation speed parsed from the engine's captured output, optional on the same terms as every other stat. +- `engine-activity`: the sampling tick folds a fresh speed observation into the retained history; a tick with no generation since the last one carries no speed point. +- `daemon-api`: `/v1/metrics` reports the current speed reading (while the engine runs) and speed points in the history, both optional. +- `remote-metrics-bar-format`: bar/gauge/table formats draw the `Speed` row (sparkline scaled to the window's high watermark, default colour throughout, figure in t/s); GPU utilisation series relabelled from `GPU util`/`GPU N util` to `GPU`/`GPU N`. + +## Impact + +- `internal/metrics` — parser for the engine's timing line; `Stats.Speed` and `HistorySample.Speed` fields. +- `internal/daemon` — supervisor output pipeline gains a line observer; the daemon tracks speed and folds it at the sampling tick; `/v1/metrics` reply shape. +- `docs/openapi.yaml` — new optional properties (policed by `openapi_test.go`). +- `cmd/spinloop/metrics_render.go` (+ serve view, dashboard, fleet/table callers) — Speed row rendering, sparkline scaling, GPU label rename; affected golden tests. +- Backward/forward compatible: old daemons and the TypeScript stats Lambda simply omit the fields; older fleet clients ignore them. diff --git a/openspec/changes/engine-generation-speed/specs/daemon-api/spec.md b/openspec/changes/engine-generation-speed/specs/daemon-api/spec.md new file mode 100644 index 00000000..b1c36e0c --- /dev/null +++ b/openspec/changes/engine-generation-speed/specs/daemon-api/spec.md @@ -0,0 +1,38 @@ +# Spec Delta + +## ADDED Requirements + +### Requirement: Metrics reports generation speed + +The metrics endpoint SHALL report the engine's last parsed generation speed, in tokens per second, alongside the token counters while the engine is running and its output has carried a timing figure since the last sampling tick. The reading SHALL be absent while the engine is stopped, and absent while a running engine has generated nothing since the last tick, rather than reported as zero or as its last known value: the figure answers "how fast is it generating right now", and a stopped or quiet engine has no answer. + +The history entries SHALL additionally carry, per entry, an optional speed point for a tick whose interval saw a timing figure, on the same retention terms as the entry's other figures — surviving a stop and cleared by the next engine's start. + +Both the current reading and the history points are optional on the wire: an engine that reports no speed, and a daemon that predates the figure, simply omit them, and a consumer treats their absence as "not reported", never as zero. + +#### Scenario: A generating engine reports its speed + +- **WHEN** a metrics request is made against a running engine that has emitted + timing figures +- **THEN** the response carries the latest parsed tokens-per-second figure + +#### Scenario: A stopped engine reports no current speed but keeps the history + +- **WHEN** a metrics request is made after an engine that had been generating + is stopped +- **THEN** the response carries no current speed figure, while the history + entries taken during generation still carry their speed points + +#### Scenario: A quiet engine lets the figure lapse + +- **WHEN** a metrics request is made against a running engine whose last + timing figure predates the last sampling tick +- **THEN** the response omits the current speed figure, the token counters + remaining as they are + +#### Scenario: An engine or daemon without the figure omits it + +- **WHEN** a metrics request is made against an engine that emits no timing + figure, or a daemon that predates the figure +- **THEN** the response carries no speed field and no history entry carries a + speed point, and every other figure is reported as before diff --git a/openspec/changes/engine-generation-speed/specs/engine-activity/spec.md b/openspec/changes/engine-generation-speed/specs/engine-activity/spec.md new file mode 100644 index 00000000..c5addfd5 --- /dev/null +++ b/openspec/changes/engine-generation-speed/specs/engine-activity/spec.md @@ -0,0 +1,45 @@ +# Spec Delta + +## ADDED Requirements + +### Requirement: Generation speed rides the sampling tick + +While an engine is running, the sampler SHALL record a generation-speed point +in the retained history on each tick whose interval saw the engine emit a new +speed figure, carrying the most recent figure of that interval. A tick whose +interval saw no new figure SHALL carry no speed point: silence means the +engine generated nothing since the last tick, not that it generated at zero, +and a retained speed point is an observation, never a continuation. + +A speed point SHALL make a tick's sample exist even where the system reading +that tick yielded no host figure at all — a generation rate is an observation +like any other. The speed points retained SHALL follow the history's existing +rules: they cover at most the last 10 minutes at the sampler's cadence, they +survive the engine's stop, and the next engine's start clears them with the +rest of the history. + +#### Scenario: Steady generation puts a point on every tick + +- **WHEN** an engine has been generating for several sampler ticks +- **THEN** the retained history carries a speed point for each of those ticks, + each the latest figure its interval saw + +#### Scenario: A quiet tick carries no speed point + +- **WHEN** an engine is running but has emitted no timing line for the last + tick's interval +- **THEN** that tick's history sample carries no speed point, while its host + figures, if any, are still retained + +#### Scenario: Speed alone still makes a sample + +- **WHEN** a tick yields a new speed figure while the system reading yields no + host figure at all +- **THEN** the tick's sample is retained, carrying the speed point + +#### Scenario: Speed points survive a stop + +- **WHEN** an engine that had been generating is stopped, and the history is + read +- **THEN** the speed points taken before the stop are still retained, until a + later engine starts and clears the history diff --git a/openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md b/openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md new file mode 100644 index 00000000..3f12af2e --- /dev/null +++ b/openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md @@ -0,0 +1,24 @@ +# Spec Delta + +## ADDED Requirements + +### Requirement: Generation speed from engine output + +The system SHALL parse a generation speed — tokens per second — from the engine's own captured output lines. An engine that reports the figure in its structured log, llama.cpp's `print_timing` lines carrying `tg = 3.87 t/s` today, yields it; the figure read SHALL be the instantaneous one the line names for the current rate, and a line's other rate figures — a trailing three-second average, prompt-evaluation speed — SHALL not be read as it. + +The speed is an optional stat on the same terms as every other: an engine whose metrics carry no such figure and whose output carries no timing line, and an engine run where its output is not captured, report the speed absent rather than reported as zero. + +#### Scenario: A timing line yields the speed + +- **WHEN** a captured engine line carries `tg = 12.34 t/s` +- **THEN** the collected reading carries a generation speed of 12.34 tokens per second + +#### Scenario: An average is not the current rate + +- **WHEN** a captured line carries only a `tg_3s = 5.29 t/s` figure, or a prompt-evaluation rate, with no instantaneous `tg` figure +- **THEN** no speed is yielded from that line + +#### Scenario: An engine without timing lines reports none + +- **WHEN** metrics are collected while the running engine's output carries no timing line +- **THEN** the result omits the generation speed rather than reporting zero diff --git a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md new file mode 100644 index 00000000..6c14342d --- /dev/null +++ b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md @@ -0,0 +1,100 @@ +# Spec Delta + +## ADDED Requirements + +### Requirement: Generation speed row + +Where a reading carries a current generation speed or its retained history carries speed points, every surface that draws the resource series SHALL additionally draw a `Speed` row beneath them: a left-aligned label, the current figure right-aligned to two decimal places with a `t/s` suffix, and, where history is drawn, a sparkline of the retained speed points to the right of the figure. + +The speed sparkline SHALL scale its glyph heights against the highest value drawn in the window — the window's high watermark maps to the sparkline's heaviest drawn grade, and no value draws the full block — so the row reads relative to itself, a steady engine filling its row and a doubled burst standing out against it. Every point SHALL appear in the terminal's default colour whatever its height: the 80/90% thresholds belong to utilisation percentages, and a generation rate has no ceiling to cross. + +Where speed exists but the surface draws no history — the gauge format, the table format, a gauge-mode dashboard tile — the row SHALL carry the figure alone. Where neither the current reading nor the history carries a speed, the row SHALL be omitted entirely. + +#### Scenario: Speed row with its sparkline + +- **WHEN** bar format renders a reading whose history carries speed points +- **THEN** the output includes a `Speed` row whose figure is the current + figure in t/s and whose glyphs are the retained speed points across the + window + +#### Scenario: The window's fastest point draws the heaviest glyph + +- **WHEN** a speed history's window holds values from 2.0 to 10.0 t/s +- **THEN** the 10.0 point draws the heaviest grade the sparkline draws, the + 2.0 point draws one far lighter, and no glyph in any row of the output is + the full block + +#### Scenario: Speed points take no threshold colour + +- **WHEN** a speed sparkline holds a point far above the 80/90% thresholds' + territory at the window's maximum +- **THEN** that glyph appears in the terminal's default colour, and so does + every other speed glyph + +#### Scenario: A figure with no history still draws + +- **WHEN** the gauge format renders a running reading with a current speed and + no retained speed points +- **THEN** the output includes a `Speed` row carrying the t/s figure with no + sparkline + +#### Scenario: No speed, no row + +- **WHEN** a reading's current figures and retained history carry no speed — + an engine that reports none, or a daemon that predates the figure +- **THEN** the output draws no `Speed` row + +## MODIFIED Requirements + +### Requirement: Bar format output + +The system SHALL support a `--format=bar` option that renders each resource series as a sparkline drawn from the history the on-instance daemon retains: a left-aligned label, one glyph per sample, and the latest value as a right-aligned percentage. The glyphs SHALL be Unicode block elements of one grade per utilisation level, so a series reads as a line of bars across the window. The series drawn SHALL be the same set the gauge format draws: CPU, RAM, each GPU's utilisation — labelled "GPU", or "GPU N" where the host has more than one — and each GPU's memory where that GPU's current reading or retained history reports a memory total — labelled "GPU mem"/"GPU N mem" — a GPU that never reports one drawing no memory series, with the same per-GPU labelling throughout. + +#### Scenario: Bar format displays CPU utilization + +- **WHEN** the user runs `spinloop cloud metrics --format=bar` with a running instance that has CPU data and a retained history +- **THEN** the output includes a row labelled "CPU" whose glyphs are the sampled CPU utilisation across the window and whose trailing figure is the latest sample's percentage + +#### Scenario: Bar format displays RAM utilization + +- **WHEN** the user runs `spinloop cloud metrics --format=bar` with a running instance that has memory data +- **THEN** the output includes a row labelled "RAM" whose glyphs are the sampled used/total memory ratio across the window and whose trailing figure is the latest ratio + +#### Scenario: Bar format displays GPU utilization + +- **WHEN** the user runs `spinloop cloud metrics --format=bar` with a running instance that has GPU data whose readings carry a memory total +- **THEN** the output includes rows labelled "GPU" and "GPU mem" (or "GPU N"/"GPU N mem" for multiple GPUs), each drawn from the retained history + +#### Scenario: Bar format omits a GPU memory series with no total + +- **WHEN** the user runs `spinloop cloud metrics --format=bar` with GPU data whose current reading and retained history report no memory total +- **THEN** the output includes "GPU" and no "GPU mem" row + +#### Scenario: Bar format header line + +- **WHEN** the user runs `spinloop cloud metrics --format=bar` with a running instance +- **THEN** the first line shows the environment, state, instance type, and model ID separated by double spaces + +### Requirement: Gauge format + +The system SHALL support a `--format=gauge` option that renders each resource series as a horizontal progress gauge: a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. The gauge draws the current reading only — it carries no history. The series drawn SHALL be CPU, RAM, each GPU's utilisation — labelled "GPU", or "GPU N" where the host has more than one — and each GPU's memory where the current reading reports a memory total — labelled "GPU mem"/"GPU N mem" — a GPU that reports none drawing no memory gauge, with the same labels the bar format uses. The gauge fill SHALL be colour-coded on the bar format's thresholds: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%, with the colour reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. + +#### Scenario: Gauge format displays CPU utilization + +- **WHEN** the user runs `spinloop cloud metrics --format=gauge` with a running instance that has CPU data +- **THEN** the output includes a gauge labelled "CPU" with filled and unfilled segments proportional to the current utilization + +#### Scenario: Gauge format displays GPU utilisation + +- **WHEN** the user runs `spinloop cloud metrics --format=gauge` with a running instance that has GPU data whose readings carry a memory total +- **THEN** the output includes gauges labelled "GPU" and "GPU mem" (or "GPU N"/"GPU N mem" for multiple GPUs) + +#### Scenario: Gauge omits a GPU memory gauge with no total + +- **WHEN** the user runs `spinloop cloud metrics --format=gauge` with a GPU whose current reading reports no memory total +- **THEN** the output includes "GPU" and no "GPU mem" gauge + +#### Scenario: Gauge colours the fill + +- **WHEN** a gauge's current value is 95% +- **THEN** its filled segment appears in red, and its unfilled segment and percentage appear in the terminal's default colour diff --git a/openspec/changes/engine-generation-speed/tasks.md b/openspec/changes/engine-generation-speed/tasks.md new file mode 100644 index 00000000..05f40dca --- /dev/null +++ b/openspec/changes/engine-generation-speed/tasks.md @@ -0,0 +1,29 @@ +# Tasks + +## Shape the stat + +- [ ] Add `Speed *float64` (`json:"speed,omitempty"`) to `metrics.Stats` and `Speed *float64` (`json:"s,omitempty"`) to `metrics.HistorySample` in `internal/metrics/metrics.go`, with doc comments in the house style (absent-not-zero; `s` one-letter for the SSM budget) +- [ ] Add `ParseGenerationSpeed(line string) (float64, bool)` to `internal/metrics` (parse.go or speed.go): `tg = t/s`, never `tg_3s`, never prompt-eval rates, reject NaN/Inf/≤0; table-driven tests over the issue's real `print_timing` lines + +## Capture it + +- [ ] Add the `lineTee` writer (buffer complete lines to `OnOutput`, size-capped partial buffer, bytes to the file first) and the `OnOutput func(string)` hook to the supervisor (`internal/daemon/supervisor.go`): stderr and the pty pump's output both write through it when `LogPath` is set and a hook is wired; tests for split writes, capped partial lines, and no-pty path +- [ ] Wire `Daemon.StartEngine` to set `Sup.OnOutput` to the daemon's observer and reset the speed tracker/tick baseline beside the existing `forget`/`clear` calls + +## Track and report it + +- [ ] Add `speedTrack` to the daemon (`internal/daemon/speed.go`): `observe` under the daemon clock; a consumed-point rule for the tick (`systemSampleOnce` attaches `sample.Speed` when a figure arrived since the last tick, and a speed point alone still makes the sample exist) and a fresh-within-one-interval rule for the current reading in `Daemon.Metrics`; tests for steady ticks, quiet ticks, figure latching then lapsing, start-after-stop reset + +## Contract + +- [ ] Add `speed` to the `Stats` schema and `s` to the `HistorySample` schema in `docs/openapi.yaml`; `go test ./internal/daemon -run TestOpenAPI` green + +## Render it + +- [ ] `cmd/spinloop/metrics_render.go`: relabel GPU utilisation series `GPU util`/`GPU N util` → `GPU`/`GPU N` in `barSeriesList`; add the `Speed` series (label `Speed`, watermark-scaled uncoloured sparkline via a speed glyph block that pools then scales, `▇` ceiling, `%6.2f t/s` figure) drawn after the resource series in `renderStatBars` and `renderStatCombined` (blank gauge half); `renderStatGauges` and the table format carry the figure line only; omit the row when neither reading nor history carries speed +- [ ] Thread the speed through the callers: `fleet.go` (bar/gauge/table), `dashboard_render.go` (tile body), `serve_view.go` (metrics section) +- [ ] Update the golden tests the relabel and new row touch (`metrics_render_test.go`, `metrics_render_gpu_test.go`, `fleet_test.go`, `fleet_dashboard_test.go`, serve view tests), adding cases for the issue's example window: watermark mapping, default colour at every height, absent-speed omission + +## Verify + +- [ ] `gofmt -w`, `go vet ./...`, `go test ./... -cover` green, coverage ≥ 80% From 69e14b87aa2d2f55d1550c399a7d427e187556e4 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Mon, 26 Oct 2026 16:14:49 +0000 Subject: [PATCH 2/7] feat(metrics): surface the engine's generation speed Parse the instantaneous tg figure from llama.cpp's print_timing lines as the engine output is captured (a line tee beside the log file), fold it into the sampler's history, and report it on the metrics API as an optional current reading and per-tick point. Bar, combined, gauge and table surfaces draw a Speed row whose sparkline scales to the window's high watermark in the default colour throughout, and the GPU utilisation series is relabelled GPU util to GPU. --- cmd/spinloop/cloud_test.go | 6 +- cmd/spinloop/dashboard_render.go | 4 +- cmd/spinloop/fleet.go | 5 +- cmd/spinloop/fleet_dashboard_test.go | 6 +- cmd/spinloop/fleet_test.go | 4 +- cmd/spinloop/metrics_render.go | 112 +++++++++- cmd/spinloop/metrics_render_gpu_test.go | 12 +- cmd/spinloop/metrics_render_test.go | 139 +++++++++--- cmd/spinloop/serve_daemon.go | 5 + cmd/spinloop/serve_view.go | 2 +- docs/openapi.yaml | 17 ++ internal/daemon/daemon.go | 25 ++- internal/daemon/history.go | 18 +- internal/daemon/speed.go | 95 ++++++++ internal/daemon/speed_test.go | 209 ++++++++++++++++++ internal/daemon/supervisor.go | 23 +- internal/daemon/tee.go | 68 ++++++ internal/daemon/tee_test.go | 84 +++++++ internal/metrics/metrics.go | 14 ++ internal/metrics/speed.go | 39 ++++ internal/metrics/speed_test.go | 79 +++++++ .../changes/engine-generation-speed/tasks.md | 20 +- 22 files changed, 909 insertions(+), 77 deletions(-) create mode 100644 internal/daemon/speed.go create mode 100644 internal/daemon/speed_test.go create mode 100644 internal/daemon/tee.go create mode 100644 internal/daemon/tee_test.go create mode 100644 internal/metrics/speed.go create mode 100644 internal/metrics/speed_test.go diff --git a/cmd/spinloop/cloud_test.go b/cmd/spinloop/cloud_test.go index 42c19d7f..7e53d733 100644 --- a/cmd/spinloop/cloud_test.go +++ b/cmd/spinloop/cloud_test.go @@ -897,14 +897,14 @@ func TestCloudMetrics_BarFormat(t *testing.T) { if !strings.Contains(out, "RAM") { t.Errorf("bar output missing RAM bar:\n%s", out) } - if !strings.Contains(out, "GPU util") { - t.Errorf("bar output missing GPU util bar:\n%s", out) + if !strings.Contains(out, " GPU ") { + t.Errorf("bar output missing GPU bar:\n%s", out) } if !strings.Contains(out, "GPU mem") { t.Errorf("bar output missing GPU mem bar:\n%s", out) } if !strings.Contains(out, "85%") { - t.Errorf("bar output missing GPU utilization percentage:\n%s", out) + t.Errorf("bar output missing GPU utilisation percentage:\n%s", out) } if !strings.Contains(out, "running:") { t.Errorf("bar output missing token stats:\n%s", out) diff --git a/cmd/spinloop/dashboard_render.go b/cmd/spinloop/dashboard_render.go index 4cc7a3a0..eacb4ed2 100644 --- a/cmd/spinloop/dashboard_render.go +++ b/cmd/spinloop/dashboard_render.go @@ -468,9 +468,9 @@ func dashTileReportBody(w io.Writer, m metrics.Stats, resources bool, gauge bool renderActiveIndented(w, m.LastActiveAt, m.IdleSeconds, m.RetainUntil, now) if resources { if gauge { - renderStatGauges(w, m.CPU, m.Memory, m.GPUs) + renderStatGauges(w, m.CPU, m.Memory, m.GPUs, m.Speed) } else { - renderStatBars(w, m.CPU, m.Memory, m.GPUs, m.History, lineW) + renderStatBars(w, m.CPU, m.Memory, m.GPUs, m.Speed, m.History, lineW) } renderTokenLines(w, m.Tokens) } diff --git a/cmd/spinloop/fleet.go b/cmd/spinloop/fleet.go index cacdc322..a6e7a24e 100644 --- a/cmd/spinloop/fleet.go +++ b/cmd/spinloop/fleet.go @@ -168,19 +168,20 @@ func renderFleetMetrics(w io.Writer, results []fleet.NodeResult, format string) // none: a stopped node's retained history says what its engine // was doing until it stopped, and a stopped node's current // reading carries no figures for it to fall back on. - renderStatBars(w, stats.CPU, stats.Memory, stats.GPUs, stats.History, barLineW) + renderStatBars(w, stats.CPU, stats.Memory, stats.GPUs, stats.Speed, stats.History, barLineW) renderTokenLines(w, stats.Tokens) case "gauge": if stats.State != "running" { continue } - renderStatGauges(w, stats.CPU, stats.Memory, stats.GPUs) + renderStatGauges(w, stats.CPU, stats.Memory, stats.GPUs, stats.Speed) renderTokenLines(w, stats.Tokens) default: if stats.State != "running" { continue } renderTokenLines(w, stats.Tokens) + renderSpeedTable(w, stats.Speed) renderGPUTable(w, stats.GPUs) renderCPUMemTable(w, stats.CPU, stats.Memory) } diff --git a/cmd/spinloop/fleet_dashboard_test.go b/cmd/spinloop/fleet_dashboard_test.go index 021eb6df..e9c6b6d7 100644 --- a/cmd/spinloop/fleet_dashboard_test.go +++ b/cmd/spinloop/fleet_dashboard_test.go @@ -358,7 +358,7 @@ func TestDashTileRunningByteStable(t *testing.T) { " active 12s ago", dashBar("CPU", 42), dashBar("RAM", 30), - dashBar("GPU util", 61), + dashBar("GPU", 61), dashBar("GPU mem", 50), "", " running: 2", @@ -393,7 +393,7 @@ func TestDashTileRunningNoRequestCount(t *testing.T) { " active 12s ago", dashBar("CPU", 42), dashBar("RAM", 30), - dashBar("GPU util", 61), + dashBar("GPU", 61), dashBar("GPU mem", 50), "", " running: 2", @@ -551,7 +551,7 @@ func TestDashTileActionInFlightWithReport(t *testing.T) { "vllm org/qwen3:32b", dashBar("CPU", 12), dashBar("RAM", 48), - dashBar("GPU util", 35), + dashBar("GPU", 35), dashBar("GPU mem", 45), "", "", "", "", }) diff --git a/cmd/spinloop/fleet_test.go b/cmd/spinloop/fleet_test.go index 14a16303..3b0e9e0c 100644 --- a/cmd/spinloop/fleet_test.go +++ b/cmd/spinloop/fleet_test.go @@ -272,7 +272,7 @@ func TestCmdFleetMetricsDrawsHistory(t *testing.T) { t.Errorf("up's CPU gauge drew history or missed the current reading:\n%s", out) } halted := out[strings.Index(out, "halted"):] - if strings.Contains(halted, "GPU util") || strings.Contains(halted, " 60%") { + if strings.Contains(halted, "GPU") || strings.Contains(halted, " 60%") { t.Errorf("a stopped node drew series in the gauge format:\n%s", halted) } @@ -291,7 +291,7 @@ func TestCmdFleetMetricsDrawsHistory(t *testing.T) { if !strings.Contains(out, "░") { t.Errorf("up's RAM did not fall back to the gauge:\n%s", out) } - if !strings.Contains(out, "halted stopped org/qwen") || !strings.Contains(out, "GPU util") { + if !strings.Contains(out, "halted stopped org/qwen") || !strings.Contains(out, " GPU ") { t.Errorf("halted's history not drawn:\n%s", out) } } diff --git a/cmd/spinloop/metrics_render.go b/cmd/spinloop/metrics_render.go index 7f84d965..1b8dc69d 100644 --- a/cmd/spinloop/metrics_render.go +++ b/cmd/spinloop/metrics_render.go @@ -318,7 +318,7 @@ func barSeriesList(cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics if len(idxs) > 1 { prefix = fmt.Sprintf("GPU %d", idx) } - add(prefix+" util", currentGPUUtil(gpus, idx), func(s metrics.HistorySample) *float64 { + add(prefix, currentGPUUtil(gpus, idx), func(s metrics.HistorySample) *float64 { for _, g := range s.GPUs { if g.Index == idx { v := float64(g.Util) @@ -369,8 +369,9 @@ func currentGPUMem(gpus []metrics.GpuStat, idx int) *float64 { // the series — as the gauge drawing of the current reading, so a daemon that // predates the history renders exactly as it did before it. Used by the bar // format on every surface; lineW is the draw width, barLineW for the full -// view and dashBarLineW for a dashboard tile. -func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, history []metrics.HistorySample, lineW int) { +// view and dashBarLineW for a dashboard tile. The generation speed closes +// the block when the reading carries one, as its own row beside the bars. +func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64, history []metrics.HistorySample, lineW int) { for _, s := range barSeriesList(cpu, mem, gpus, history) { if len(s.history) > 0 { renderSparkline(w, s.label, s.history, lineW) @@ -378,6 +379,84 @@ func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, renderGauge(w, s.label, *s.current, gaugeW) } } + renderSpeedBar(w, speed, history, lineW) +} + +// speedSamples gathers the retained generation-rate points, oldest first. +// A series like any other: samples where the window has them, gaps where the +// engine was quiet, since a point is an observation and never a continuation. +func speedSamples(history []metrics.HistorySample) []float64 { + var out []float64 + for _, s := range history { + if s.Speed != nil { + out = append(out, *s.Speed) + } + } + return out +} + +// speedBlock draws the speed sparkline. Its glyph heights are relative to +// the highest value the window holds — the window's own maximum maps to the +// heaviest grade the sparkline draws and nothing maps to the full block — +// so the row reads against itself: a steady engine fills it evenly, a burst +// doubles the row's height beside its own baseline. Every point draws in the +// terminal's default colour whatever its height: the 80/90 thresholds belong +// to utilisation percentages, and a generation rate has no ceiling to cross. +func speedBlock(samples []float64, width int) string { + pooled := poolMax(samples, width) + peak := 0.0 + for _, v := range pooled { + if v > peak { + peak = v + } + } + var b strings.Builder + b.WriteString(strings.Repeat(" ", width-len(pooled))) + for _, v := range pooled { + pct := 0.0 + if peak > 0 { + pct = v / peak * 100 + } + b.WriteRune(barGlyph(pct)) + } + return b.String() +} + +// speedFigure is the trailing figure of a Speed row: the current reading, +// falling back to the window's latest point where the reading carries none — +// a stopped engine has no current rate, but the row's last point is still +// the last rate it had. +func speedFigure(speed *float64, samples []float64) float64 { + if speed != nil { + return *speed + } + return samples[len(samples)-1] +} + +// renderSpeedBar draws the Speed row of the bar-style surfaces: label, the +// t/s figure, and the rate's sparkline — or blank where the window holds no +// point, since a gauge would imply a ceiling the rate does not have. The row +// is absent entirely when neither the reading nor the history carries a +// speed: an engine that reports none is not an engine generating at zero. +func renderSpeedBar(w io.Writer, speed *float64, history []metrics.HistorySample, lineW int) { + samples := speedSamples(history) + if speed == nil && len(samples) == 0 { + return + } + bar := strings.Repeat(" ", lineW) + if len(samples) > 0 { + bar = speedBlock(samples, lineW) + } + fmt.Fprintf(w, " %-9s %6.2f t/s %s\n", "Speed", speedFigure(speed, samples), bar) +} + +// renderSpeedGauge draws the Speed row where no history is drawn — the gauge +// format and the gauge-mode tile: the label and the figure, no bar. +func renderSpeedGauge(w io.Writer, speed *float64) { + if speed == nil { + return + } + fmt.Fprintf(w, " %-9s %6.2f t/s\n", "Speed", *speed) } // serveGaugeW and serveBarW are the two halves of the serve view's combined @@ -401,7 +480,7 @@ const ( // history leaves its bar half blank and a series with no current reading its // gauge half blank, so the lines align and the figure never draws a half's // own number twice. -func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, history []metrics.HistorySample) { +func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64, history []metrics.HistorySample) { for _, s := range barSeriesList(cpu, mem, gpus, history) { gaugeHalf := strings.Repeat(" ", serveGaugeW) barHalf := strings.Repeat(" ", serveBarW) @@ -419,16 +498,27 @@ func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemorySt } fmt.Fprintf(w, " %-9s %s %3.0f%% %s\n", s.label, gaugeHalf, figure, barHalf) } + if samples := speedSamples(history); speed != nil || len(samples) > 0 { + barHalf := strings.Repeat(" ", serveBarW) + if len(samples) > 0 { + barHalf = speedBlock(samples, serveBarW) + } + // The gauge half stays blank: the rate has no ceiling to fill + // against, so the row carries its figure and its trend only. + fmt.Fprintf(w, " %-9s %s %6.2f t/s %s\n", "Speed", strings.Repeat(" ", serveGaugeW), speedFigure(speed, samples), barHalf) + } } // renderStatGauges draws the resource series in the gauge format: the current -// reading only, whatever the history holds. -func renderStatGauges(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat) { +// reading only, whatever the history holds. The generation speed joins as a +// figure-only row when the reading carries one. +func renderStatGauges(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64) { for _, s := range barSeriesList(cpu, mem, gpus, nil) { if s.current != nil { renderGauge(w, s.label, *s.current, gaugeW) } } + renderSpeedGauge(w, speed) } // renderTokenLines draws the engine's token and request counters, the block @@ -449,6 +539,16 @@ func renderTokenLines(w io.Writer, tokens *metrics.TokenStats) { } } +// renderSpeedTable draws the table format's speed line, beside the token +// counters it shares the engine-block with. Absent, not zero, when the +// reading carries no rate. +func renderSpeedTable(w io.Writer, speed *float64) { + if speed == nil { + return + } + fmt.Fprintf(w, " speed: %.2f t/s\n", *speed) +} + // renderGPUTable draws the per-GPU lines of the table format, plus the // totals line when there is more than one GPU. func renderGPUTable(w io.Writer, gpus []metrics.GpuStat) { diff --git a/cmd/spinloop/metrics_render_gpu_test.go b/cmd/spinloop/metrics_render_gpu_test.go index 12ef737e..47b38b2c 100644 --- a/cmd/spinloop/metrics_render_gpu_test.go +++ b/cmd/spinloop/metrics_render_gpu_test.go @@ -17,10 +17,10 @@ func TestUtilisationOnlyGPUDrawsNoMemorySeries(t *testing.T) { {GPUs: []metrics.HistoryGPU{{Index: 0, Util: 56}}}, } var bar, gauge bytes.Buffer - renderStatBars(&bar, nil, nil, macGPU, history, barLineW) - renderStatGauges(&gauge, nil, nil, macGPU) + renderStatBars(&bar, nil, nil, macGPU, nil, history, barLineW) + renderStatGauges(&gauge, nil, nil, macGPU, nil) for name, out := range map[string]string{"bar": bar.String(), "gauge": gauge.String()} { - if !strings.Contains(out, "GPU util") || !strings.Contains(out, " 56%") { + if !strings.Contains(out, "GPU") || !strings.Contains(out, " 56%") { t.Errorf("%s: missing GPU util: %q", name, out) } if strings.Contains(out, "GPU mem") { @@ -34,10 +34,10 @@ func TestGPUWithMemoryTotalDrawsBothSeries(t *testing.T) { gpus := []metrics.GpuStat{{Index: 0, Name: "NVIDIA L40S", Utilization: 12, MemoryUsed: 8 << 30, MemoryTotal: 16 << 30, Temperature: 42}} var bar, gauge bytes.Buffer - renderStatBars(&bar, nil, nil, gpus, nil, barLineW) - renderStatGauges(&gauge, nil, nil, gpus) + renderStatBars(&bar, nil, nil, gpus, nil, nil, barLineW) + renderStatGauges(&gauge, nil, nil, gpus, nil) for name, out := range map[string]string{"bar": bar.String(), "gauge": gauge.String()} { - if !strings.Contains(out, "GPU util") || !strings.Contains(out, "GPU mem") || !strings.Contains(out, " 50%") { + if !strings.Contains(out, "GPU") || !strings.Contains(out, "GPU mem") || !strings.Contains(out, " 50%") { t.Errorf("%s: %q", name, out) } } diff --git a/cmd/spinloop/metrics_render_test.go b/cmd/spinloop/metrics_render_test.go index 6d2df7ae..54b9d5b1 100644 --- a/cmd/spinloop/metrics_render_test.go +++ b/cmd/spinloop/metrics_render_test.go @@ -162,8 +162,8 @@ func TestRenderStatBarsFallsBackToGaugeWithoutHistory(t *testing.T) { gpus := []metrics.GpuStat{{Index: 0, Name: "H100", Utilization: 61, MemoryUsed: 80, MemoryTotal: 160}} var got, want bytes.Buffer - renderStatBars(&got, cpu, mem, gpus, nil, barLineW) - renderStatGauges(&want, cpu, mem, gpus) + renderStatBars(&got, cpu, mem, gpus, nil, nil, barLineW) + renderStatGauges(&want, cpu, mem, gpus, nil) if got.String() != want.String() { t.Errorf("no-history bar differs from the old drawing:\ngot:\n%q\nwant:\n%q", got.String(), want.String()) } @@ -179,7 +179,7 @@ func TestRenderStatBarsDrawsHistory(t *testing.T) { {Time: 3, CPU: ptrPct(95), Mem: ptrPct(40)}, } var b bytes.Buffer - renderStatBars(&b, nil, nil, nil, history, barLineW) + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") if len(lines) != 2 { t.Fatalf("drew %d lines, want CPU and RAM: %q", len(lines), b.String()) @@ -210,7 +210,7 @@ func TestRenderStatBarsMaxedRowsStopShortOfFull(t *testing.T) { {Time: 2, CPU: ptrPct(100), Mem: ptrPct(100)}, } var b bytes.Buffer - renderStatBars(&b, nil, nil, nil, history, barLineW) + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) out := b.String() if strings.Contains(out, "█") { t.Errorf("a maxed series drew a full block:\n%s", out) @@ -231,7 +231,7 @@ func TestRenderStatBarsFallsBackPerSeries(t *testing.T) { {Time: 2, CPU: ptrPct(20)}, } var b bytes.Buffer - renderStatBars(&b, nil, &metrics.MemoryStat{Total: 1000, Used: 300}, nil, history, barLineW) + renderStatBars(&b, nil, &metrics.MemoryStat{Total: 1000, Used: 300}, nil, nil, history, barLineW) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") if len(lines) != 2 { t.Fatalf("drew %d lines, want 2: %q", len(lines), b.String()) @@ -253,17 +253,17 @@ func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { {Time: 2, CPU: ptrPct(20), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 60, Mem: ptrPct(60)}}}, } var b bytes.Buffer - renderStatBars(&b, nil, nil, nil, history, barLineW) + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") if len(lines) != 3 { - t.Fatalf("drew %d lines, want CPU, GPU util and GPU mem: %q", len(lines), b.String()) + t.Fatalf("drew %d lines, want CPU, GPU and GPU mem: %q", len(lines), b.String()) } if !strings.HasPrefix(lines[0], " CPU ") { t.Errorf("first line: %q", lines[0]) } // One GPU: the plain "GPU" prefix, as the gauge used it. - if !strings.Contains(lines[1], "GPU util") || !strings.Contains(lines[1], " 60%") { - t.Errorf("GPU util line: %q", lines[1]) + if !strings.Contains(lines[1], "GPU") || !strings.Contains(lines[1], " 60%") { + t.Errorf("GPU line: %q", lines[1]) } if !strings.Contains(lines[2], "GPU mem") || !strings.Contains(lines[2], " 60%") { t.Errorf("GPU mem line: %q", lines[2]) @@ -277,9 +277,9 @@ func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { }}, } b.Reset() - renderStatBars(&b, nil, nil, nil, history, barLineW) + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) out := b.String() - if !strings.Contains(out, "GPU 0 util") || !strings.Contains(out, "GPU 1 util") || + if !strings.Contains(out, "GPU 0") || !strings.Contains(out, "GPU 1") || !strings.Contains(out, "GPU 0 mem") || !strings.Contains(out, "GPU 1 mem") { t.Errorf("two GPUs: %q", out) } @@ -288,7 +288,7 @@ func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { // A memory reading with no total reports 0, not a division by it. func TestBarSeriesListMemoryWithoutTotal(t *testing.T) { var b bytes.Buffer - renderStatBars(&b, nil, &metrics.MemoryStat{Total: 0, Used: 100}, nil, nil, barLineW) + renderStatBars(&b, nil, &metrics.MemoryStat{Total: 0, Used: 100}, nil, nil, nil, barLineW) if !strings.Contains(b.String(), " 0%") || strings.Contains(b.String(), "NaN") { t.Errorf("a memory reading with no total: %q", b.String()) } @@ -302,10 +302,10 @@ func TestRenderStatBarsGPUInOnlySomeSamples(t *testing.T) { {Time: 2, GPUs: []metrics.HistoryGPU{{Index: 1, Util: 90, Mem: ptrPct(80)}}}, } var b bytes.Buffer - renderStatBars(&b, nil, nil, nil, history, barLineW) + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) out := b.String() // Both GPUs are named in the union, so all four series draw. - for _, want := range []string{"GPU 0 util", "GPU 1 util", "GPU 0 mem", "GPU 1 mem"} { + for _, want := range []string{"GPU 0", "GPU 1", "GPU 0 mem", "GPU 1 mem"} { if !strings.Contains(out, want) { t.Fatalf("series %q not drawn: %q", want, out) } @@ -323,12 +323,12 @@ func TestRenderStatBarsGPUInOnlySomeSamples(t *testing.T) { func TestRenderStatGaugesIgnoresHistory(t *testing.T) { cpu := &metrics.CpuStat{Utilization: 50} var b bytes.Buffer - renderStatGauges(&b, cpu, nil, nil) + renderStatGauges(&b, cpu, nil, nil, nil) if !strings.Contains(b.String(), " 50%") || strings.Contains(b.String(), "▁") { t.Errorf("gauge drew history or the wrong value: %q", b.String()) } b.Reset() - renderStatGauges(&b, nil, nil, nil) + renderStatGauges(&b, nil, nil, nil, nil) if b.String() != "" { t.Errorf("a stopped engine drew gauges: %q", b.String()) } @@ -347,13 +347,13 @@ func TestRenderStatCombined(t *testing.T) { {Time: 2, CPU: ptrPct(95), Mem: ptrPct(30), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 91, Mem: ptrPct(20)}}}, } var b bytes.Buffer - renderStatCombined(&b, cpu, mem, gpus, history) + renderStatCombined(&b, cpu, mem, gpus, nil, history) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") // One line per series, in the bar series' order. if len(lines) != 4 { t.Fatalf("drew %d lines, want 4: %q", len(lines), b.String()) } - for i, label := range []string{" CPU ", " RAM ", " GPU util ", " GPU mem "} { + for i, label := range []string{" CPU ", " RAM ", " GPU ", " GPU mem "} { if !strings.HasPrefix(lines[i], label) { t.Errorf("line for %s: %q", label, lines[i]) } @@ -409,7 +409,7 @@ func TestRenderStatCombinedFigureDoesNotShiftTheBar(t *testing.T) { } for _, c := range cases { var b bytes.Buffer - renderStatCombined(&b, &metrics.CpuStat{Utilization: c.value}, nil, nil, history) + renderStatCombined(&b, &metrics.CpuStat{Utilization: c.value}, nil, nil, nil, history) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") if len(lines) != 1 { t.Fatalf("figure %s drew %d lines, want 1: %q", c.figure, len(lines), b.String()) @@ -428,7 +428,7 @@ func TestRenderStatCombinedFigureDoesNotShiftTheBar(t *testing.T) { // half of the combined format. func TestRenderStatCombinedMemoryWithoutTotal(t *testing.T) { var b bytes.Buffer - renderStatCombined(&b, nil, &metrics.MemoryStat{Total: 0, Used: 100}, nil, nil) + renderStatCombined(&b, nil, &metrics.MemoryStat{Total: 0, Used: 100}, nil, nil, nil) if !strings.Contains(b.String(), " 0%") || strings.Contains(b.String(), "NaN") { t.Errorf("a memory reading with no total: %q", b.String()) } @@ -441,13 +441,13 @@ func TestRenderStatCombinedWithoutHistory(t *testing.T) { mem := &metrics.MemoryStat{Total: 1000, Used: 300} gpus := []metrics.GpuStat{{Index: 0, Name: "H100", Utilization: 61, MemoryUsed: 80, MemoryTotal: 160}} var b bytes.Buffer - renderStatCombined(&b, cpu, mem, gpus, nil) + renderStatCombined(&b, cpu, mem, gpus, nil, nil) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") blankBar := strings.Repeat(" ", serveBarW) want := []string{ " CPU " + gaugeBlock(42, serveGaugeW) + " 42% " + blankBar, " RAM " + gaugeBlock(30, serveGaugeW) + " 30% " + blankBar, - " GPU util " + gaugeBlock(61, serveGaugeW) + " 61% " + blankBar, + " GPU " + gaugeBlock(61, serveGaugeW) + " 61% " + blankBar, " GPU mem " + gaugeBlock(50, serveGaugeW) + " 50% " + blankBar, } if len(lines) != len(want) { @@ -472,16 +472,16 @@ func TestRenderStatCombinedStoppedEngineDrawsHistoryAlone(t *testing.T) { {Time: 2, CPU: ptrPct(20), Mem: ptrPct(30), GPUs: []metrics.HistoryGPU{{Index: 0, Util: 60, Mem: ptrPct(60)}}}, } var b bytes.Buffer - renderStatCombined(&b, nil, nil, nil, history) + renderStatCombined(&b, nil, nil, nil, nil, history) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") if len(lines) != 4 { - t.Fatalf("drew %d lines, want CPU, RAM, GPU util and GPU mem: %q", len(lines), b.String()) + t.Fatalf("drew %d lines, want CPU, RAM, GPU and GPU mem: %q", len(lines), b.String()) } blankGauge := strings.Repeat(" ", serveGaugeW) wantPrefix := []string{ " CPU " + blankGauge + " 20% ", " RAM " + blankGauge + " 30% ", - " GPU util " + blankGauge + " 60% ", + " GPU " + blankGauge + " 60% ", " GPU mem " + blankGauge + " 60% ", } for i, want := range wantPrefix { @@ -567,7 +567,7 @@ func TestFormatMetricsBarRunning(t *testing.T) { if strings.Contains(got, "░") { t.Errorf("a series fell back to the gauge although history holds it: %q", got) } - for _, label := range []string{"CPU", "RAM", "GPU util", "GPU mem"} { + for _, label := range []string{"CPU", "RAM", "GPU", "GPU mem"} { if !strings.Contains(got, label) { t.Errorf("series %q missing: %q", label, got) } @@ -647,3 +647,90 @@ func nodeResultsFor(resp *cloud.StatsResponse) []fleet.NodeResult { }, }} } + +// The Speed row: the window's high watermark sets the glyph heights, every +// point draws in the default colour, and no speed in the reading draws no +// row at all. +func TestRenderSpeedRow(t *testing.T) { + speed := 8.5 + history := []metrics.HistorySample{ + {Time: 1, Speed: ptrPct(2.0)}, + {Time: 2, Speed: ptrPct(5.0)}, + {Time: 3, Speed: ptrPct(10.0)}, + } + var b bytes.Buffer + renderStatBars(&b, &metrics.CpuStat{Utilization: 95}, nil, nil, &speed, history, barLineW) + lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") + line := lines[len(lines)-1] + if !strings.HasPrefix(line, " Speed 8.50 t/s ") { + t.Fatalf("speed row: %q", line) + } + // The window's fastest point takes the heaviest grade the sparkline + // draws, its slowest one far lighter — and neither the full block. + tail := strings.TrimLeft(line[strings.Index(line, "t/s ")+4:], " ") + if !strings.HasSuffix(tail, "▇") { + t.Errorf("the window maximum did not draw the heaviest grade: %q", tail) + } + if !strings.Contains(tail, "▂") { + t.Errorf("the window minimum did not draw a far lighter grade: %q", tail) + } + // The speed row draws no full block of its own, and no threshold colour + // at its window maximum — while the 95% CPU row on the same screen is + // free to be coloured. + if strings.Contains(line, "█") { + t.Errorf("the speed row drew the full block: %q", line) + } + if strings.Contains(line, "\x1b[") { + t.Errorf("the speed row drew colour: %q", line) + } + + // No current figure but a retained history: the row falls back to the + // window's last point — the last rate a stopped engine had. + b.Reset() + renderStatBars(&b, nil, nil, nil, nil, history, barLineW) + line = strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n")[0] + if !strings.HasPrefix(line, " Speed 10.00 t/s ") { + t.Errorf("a stopped engine's speed row: %q", line) + } + + // Neither reading nor history: no row. + b.Reset() + renderStatBars(&b, &metrics.CpuStat{Utilization: 10}, nil, nil, nil, nil, barLineW) + if strings.Contains(b.String(), "Speed") { + t.Errorf("an engine without speed drew a row: %q", b.String()) + } +} + +// The gauge format and the serve view carry the figure without a bar the +// rate has no ceiling to fill against. +func TestRenderSpeedFigureOnly(t *testing.T) { + speed := 12.34 + var b bytes.Buffer + renderStatGauges(&b, nil, nil, nil, &speed) + line := strings.TrimSpace(b.String()) + if line != "Speed 12.34 t/s" { + t.Errorf("gauge speed row: %q", line) + } + if strings.Contains(line, "░") { + t.Errorf("the speed gauge half drew a bar: %q", line) + } + + b.Reset() + renderStatCombined(&b, nil, nil, nil, &speed, nil) + line = strings.TrimSpace(b.String()) + if !strings.Contains(line, "Speed") || !strings.Contains(line, "12.34 t/s") { + t.Errorf("combined speed row: %q", line) + } + + // No figure: neither surface mentions speed. + b.Reset() + renderStatGauges(&b, nil, nil, nil, nil) + if strings.Contains(b.String(), "Speed") { + t.Errorf("no figure still drew a gauge row: %q", b.String()) + } + b.Reset() + renderStatCombined(&b, nil, nil, nil, nil, nil) + if strings.Contains(b.String(), "Speed") { + t.Errorf("no figure still drew a combined row: %q", b.String()) + } +} diff --git a/cmd/spinloop/serve_daemon.go b/cmd/spinloop/serve_daemon.go index 34edcbe7..86d3aa56 100644 --- a/cmd/spinloop/serve_daemon.go +++ b/cmd/spinloop/serve_daemon.go @@ -268,6 +268,11 @@ func startSupervisedForeground(sel spinloop.Selection, spinloopPath string, engi model = sel.Alias } d.SetServed(sel.Provider, model, sel.Alias) + // The engine starts through the supervisor directly, so the output hook + // that reads its self-reported generation rate is wired here rather than + // in StartEngine. A forward-stdio run (empty LogPath) never captures a + // line, so the hook simply never fires. + sup.OnOutput = d.ObserveEngineLine // A supervised engine gets its metrics endpoint switched on, exactly as // the cloud path does for a deployed one; an engine with no metrics // dialect gets no added switch and the host's series. diff --git a/cmd/spinloop/serve_view.go b/cmd/spinloop/serve_view.go index 12411a64..5a214019 100644 --- a/cmd/spinloop/serve_view.go +++ b/cmd/spinloop/serve_view.go @@ -392,7 +392,7 @@ func (m serveView) metricsLines() []string { // rule for when the series exist at all. resources := m.stats.State == "running" || len(m.stats.History) > 0 if resources { - renderStatCombined(&b, m.stats.CPU, m.stats.Memory, m.stats.GPUs, m.stats.History) + renderStatCombined(&b, m.stats.CPU, m.stats.Memory, m.stats.GPUs, m.stats.Speed, m.stats.History) renderTokenLines(&b, m.stats.Tokens) } lines := strings.Split(b.String(), "\n") diff --git a/docs/openapi.yaml b/docs/openapi.yaml index ebd3d287..a0a6c0c6 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -417,6 +417,16 @@ components: $ref: "#/components/schemas/CpuStat" memory: $ref: "#/components/schemas/MemoryStat" + speed: + type: number + description: | + The engine's own generation rate in tokens per second, parsed + from the timing lines it prints into its captured output + (llama.cpp's print_timing `tg` figure today). Present only while + the engine runs and has printed a figure within the last sampler + interval; absent — never zero — for a stopped or quiet engine, an + engine that prints no rate, and a daemon that predates the + figure. history: type: array description: | @@ -546,6 +556,13 @@ components: m: type: number description: System memory used over total, percent. + s: + type: number + description: | + The engine's generation rate in tokens per second, from the + timing lines its output carried during this tick's interval. + Absent where the engine printed no figure since the previous + tick — silence means nothing was generated, not a rate of zero. g: type: array items: diff --git a/internal/daemon/daemon.go b/internal/daemon/daemon.go index 0bf11c1c..0926baf9 100644 --- a/internal/daemon/daemon.go +++ b/internal/daemon/daemon.go @@ -68,6 +68,7 @@ type Daemon struct { ready readiness hist systemHistory system systemSample + speed engineSpeed mu sync.Mutex runner string @@ -260,6 +261,19 @@ func (d *Daemon) StartEngine() error { slog.String("source", source), slog.String("runner", runner), slog.String("model", model)) + // The previous engine's counters and system readings must not be reported + // against this one, for the same reason its counter baseline is dropped. + d.sample.forget() + d.ready.forget() + d.system.forget() + d.hist.clear() + d.speed.forget() + // The engine's account of itself — its generation rate — is read from + // the output the supervisor is about to capture. Wired before the start: + // the supervisor builds its capture sink as it launches, and a hook set + // afterwards would describe an engine whose first lines are already lost. + // The no-capture path (LogPath empty) never calls it. + d.Sup.OnOutput = d.ObserveEngineLine if err := d.Sup.Start(argv); err != nil { d.log().Error("engine start failed", slog.String("source", source), slog.String("error", err.Error())) @@ -270,12 +284,6 @@ func (d *Daemon) StartEngine() error { // existed — the race the control plane used to close with a last-wake // timestamp of its own. d.act.markActive(d.now()) - // The previous engine's counters and system readings must not be reported - // against this one, for the same reason its counter baseline is dropped. - d.sample.forget() - d.ready.forget() - d.system.forget() - d.hist.clear() return nil } @@ -460,6 +468,11 @@ func (d *Daemon) Metrics(ctx context.Context) metrics.Stats { fmt.Sprintf("engine metrics scrape (%s): %v", baseURL, err)) } stats.Tokens = tokens + // The engine's self-reported rate, if it has printed one this close + // to now. An engine that has gone quiet lapses the figure — the + // counter block says how much it has generated; this line answers + // how fast, and a quiet engine has no answer. + stats.Speed = d.speed.current(d.now(), d.speedStaleAfter()) stats.Ready = d.readinessField() } // Outside the running branch, so a stopped or crashed engine still reports diff --git a/internal/daemon/history.go b/internal/daemon/history.go index ee484cd1..bd09b32b 100644 --- a/internal/daemon/history.go +++ b/internal/daemon/history.go @@ -89,13 +89,17 @@ func (d *Daemon) systemSampleOnce(ctx context.Context) { if state, _, _ := d.Sup.Status(); state != StateRunning { return } - if d.Collector == nil { - return - } + // The tick asks the engine's own account of this interval first: a + // generation-rate point comes from the captured output, not from the + // host, so a daemon with no collector still folds it and still has a + // sample to retain for it. + speed := d.speed.sampleTick(d.now()) var stats metrics.Stats - d.Collector.System(ctx, &stats) - d.system.record(stats) - sample := metrics.HistorySample{Time: d.now().Unix()} + if d.Collector != nil { + d.Collector.System(ctx, &stats) + d.system.record(stats) + } + sample := metrics.HistorySample{Time: d.now().Unix(), Speed: speed} if stats.CPU != nil { v := stats.CPU.Utilization sample.CPU = &v @@ -112,7 +116,7 @@ func (d *Daemon) systemSampleOnce(ctx context.Context) { } sample.GPUs = append(sample.GPUs, hg) } - if sample.CPU == nil && sample.Mem == nil && len(sample.GPUs) == 0 { + if sample.CPU == nil && sample.Mem == nil && len(sample.GPUs) == 0 && sample.Speed == nil { return } d.hist.add(sample) diff --git a/internal/daemon/speed.go b/internal/daemon/speed.go new file mode 100644 index 00000000..efd075b9 --- /dev/null +++ b/internal/daemon/speed.go @@ -0,0 +1,95 @@ +package daemon + +import ( + "sync" + "time" + + "github.com/spinloop-ai/spinloop/internal/metrics" +) + +// engineSpeed is the daemon's record of the rate its engine reports for +// itself: the generation-speed figures parsed out of the engine's captured +// output lines, and the last one the sampler folded into the history. It +// answers two questions with two clocks: +// +// - the sampler tick asks what the interval saw: a figure arrived, the tick +// gets a point; none arrived, the tick gets nothing — a point is an +// observation, and holding the last figure onto a quiet tick would draw +// a generation rate for a window that generated nothing; +// - a metrics read asks what stands now: the last figure counts only while +// it is no older than a sampling interval, so an engine that has gone +// quiet lets the current figure lapse rather than standing as the rate +// of some earlier burst. +type engineSpeed struct { + mu sync.Mutex + value float64 + observed time.Time // engine clock of the last figure + have bool + lastTick time.Time // sampler clock of the last tick that was asked +} + +// observe records one parsed figure against the daemon's clock. +func (s *engineSpeed) observe(v float64, now time.Time) { + s.mu.Lock() + defer s.mu.Unlock() + s.value, s.observed, s.have = v, now, true +} + +// sampleTick is the sampler's fold: the point for this tick, or nil when +// the interval since the last tick saw no figure. Every tick updates the +// baseline, whether or not it produced a point — the question the next tick +// asks is "since this one". +func (s *engineSpeed) sampleTick(now time.Time) *float64 { + s.mu.Lock() + defer s.mu.Unlock() + var point *float64 + if s.have && s.observed.After(s.lastTick) { + v := s.value + point = &v + } + s.lastTick = now + return point +} + +// current is the figure a metrics read reports: the last one seen, while a +// tick's worth of quiet has not aged it out. +func (s *engineSpeed) current(now time.Time, staleAfter time.Duration) *float64 { + s.mu.Lock() + defer s.mu.Unlock() + if !s.have || now.Sub(s.observed) > staleAfter { + return nil + } + v := s.value + return &v +} + +// forget drops the record, so one engine's rate is not reported against the +// next — beside the counter baseline's drop and the history's clear, for +// exactly their reason. +func (s *engineSpeed) forget() { + s.mu.Lock() + defer s.mu.Unlock() + s.value, s.observed, s.have, s.lastTick = 0, time.Time{}, false, time.Time{} +} + +// ObserveEngineLine is the supervisor's output hook: the one place engine +// bytes become a speed figure. It is cheap and never blocks — a regex over a +// log line — and lines that carry no figure are most lines, so they cost the +// scan and nothing else. The CLI wires it to Supervisor.OnOutput on every +// path that starts an engine, including the foreground serve path that +// starts through the supervisor directly rather than through StartEngine. +func (d *Daemon) ObserveEngineLine(line string) { + if v, ok := metrics.ParseGenerationSpeed(line); ok { + d.speed.observe(v, d.now()) + } +} + +// speedStaleAfter is how long a current speed figure stands without a new +// one: one sampling interval, the same clock the tick's fold runs on, so a +// figure lapses for the read exactly when its tick would have folded nothing. +func (d *Daemon) speedStaleAfter() time.Duration { + if d.SampleInterval > 0 { + return d.SampleInterval + } + return DefaultSampleInterval +} diff --git a/internal/daemon/speed_test.go b/internal/daemon/speed_test.go new file mode 100644 index 00000000..1d7020d7 --- /dev/null +++ b/internal/daemon/speed_test.go @@ -0,0 +1,209 @@ +//go:build !windows + +package daemon + +import ( + "context" + "testing" + "time" + + "github.com/spinloop-ai/spinloop/internal/inference" +) + +func TestEngineSpeedFoldAndLapse(t *testing.T) { + var s engineSpeed + + // Nothing observed: neither a point nor a figure. + if p := s.sampleTick(baseTime); p != nil { + t.Errorf("a tick with no figure returned %v, want none", p) + } + if c := s.current(baseTime, time.Second); c != nil { + t.Errorf("a read with no figure returned %v, want none", c) + } + + // A figure observed: the next tick folds it once, and the one after + // that saw nothing new, so it folds nothing. + s.observe(12.5, baseTime.Add(time.Second)) + if p := s.sampleTick(baseTime.Add(15 * time.Second)); p == nil || *p != 12.5 { + t.Errorf("folded point = %v, want 12.5", p) + } + if p := s.sampleTick(baseTime.Add(30 * time.Second)); p != nil { + t.Errorf("a quiet interval folded %v, want none", p) + } + // A newer figure replaces the older on the next fold. + s.observe(9.0, baseTime.Add(31*time.Second)) + if p := s.sampleTick(baseTime.Add(45 * time.Second)); p == nil || *p != 9.0 { + t.Errorf("folded point = %v, want the newest figure", p) + } + + // The current figure stands for an interval's worth of quiet and lapses + // past it — the same clock the fold runs on. + s.observe(7.25, baseTime.Add(50*time.Second)) + if c := s.current(baseTime.Add(55*time.Second), 15*time.Second); c == nil || *c != 7.25 { + t.Errorf("current = %v, want 7.25", c) + } + if c := s.current(baseTime.Add(70*time.Second), 15*time.Second); c != nil { + t.Errorf("an aged figure still reads %v, want none", c) + } + + // A forgotten record answers nothing: one engine's rate is not the next + // engine's. + s.observe(3.0, baseTime.Add(80*time.Second)) + s.forget() + if p := s.sampleTick(baseTime.Add(95 * time.Second)); p != nil { + t.Errorf("after forget a tick folded %v, want none", p) + } + if c := s.current(baseTime.Add(95*time.Second), time.Hour); c != nil { + t.Errorf("after forget the current figure reads %v, want none", c) + } +} + +// TestSpeedEndToEnd runs a fake engine that prints llama.cpp's timing line, +// and reads the figure back through the capture hook, the sampler fold and +// the metrics reply — every seam in one wire. +func TestSpeedEndToEnd(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +echo "86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s" >&2 +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + d.Collector = linuxCollector() + + if err := d.Push(inference.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + // The hook fires when the line lands, not when the test asks; wait for + // the figure the way a real sampler waits for a real engine. + waitForSpeed(t, d) + + // The tick folds the figure into the history. + d.systemSampleOnce(context.Background()) + hist := d.hist.snapshot() + if len(hist) != 1 { + t.Fatalf("history = %d samples, want 1", len(hist)) + } + if hist[0].Speed == nil || *hist[0].Speed != 3.87 { + t.Errorf("sample speed = %v, want 3.87", hist[0].Speed) + } + + // The running metrics reply carries the current figure while it is fresh + // (the clock is pinned at baseTime, and the figure is observed "now"). + stats := d.Metrics(context.Background()) + if stats.Speed == nil || *stats.Speed != 3.87 { + t.Fatalf("metrics speed = %v, want 3.87", stats.Speed) + } + + // A tick's worth of quiet later — same figure in hand, no new one — the + // current reading has lapsed and the next tick folds nothing, while the + // point already taken stays in the window. + d.Now = func() time.Time { return baseTime.Add(time.Minute) } + if got := d.Metrics(context.Background()).Speed; got != nil { + t.Errorf("a quiet minute still reads %v, want the figure lapsed", got) + } + d.systemSampleOnce(context.Background()) + hist = d.hist.snapshot() + if len(hist) != 2 || hist[1].Speed != nil { + t.Errorf("a quiet tick folded %+v, want a second sample with no speed point", hist[1:]) + } + + // The stop keeps the history's point and drops the current figure. + d.Sup.Stop() + waitForState(t, d.Sup, StateStopped) + stats = d.Metrics(context.Background()) + if stats.Speed != nil { + t.Errorf("a stopped engine still reports %v, want no current figure", stats.Speed) + } + hist = d.hist.snapshot() + if len(hist) != 2 || hist[0].Speed == nil || *hist[0].Speed != 3.87 { + t.Errorf("history after a stop = %+v, want the taken point retained", hist) + } +} + +// TestSpeedSampleExistsOnSpeedAlone pins the fold's presence rule: a tick +// with a figure and no host figures at all is still a sample — the rate is +// an observation in its own right. +func TestSpeedSampleExistsOnSpeedAlone(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + // No collector at all: the host yields nothing, ever. + + if err := d.Push(inference.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + defer d.Sup.Stop() + waitForState(t, d.Sup, StateRunning) + + // A tick with nothing seen adds nothing. + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 0 { + t.Fatalf("an empty tick retained %+v, want nothing", got) + } + + // The same tick with a figure retained a sample — the speed point alone. + // The clock advances a moment first: with it pinned, the figure would + // land on the empty tick's own instant, and "since the last tick" is a + // strict boundary. + d.Now = func() time.Time { return baseTime.Add(time.Second) } + d.ObserveEngineLine("slot print_timing: id 0 | task 3 | tg = 6.10 t/s") + d.systemSampleOnce(context.Background()) + hist := d.hist.snapshot() + if len(hist) != 1 || hist[0].Speed == nil || *hist[0].Speed != 6.10 { + t.Fatalf("history = %+v, want one speed-only sample", hist) + } + if hist[0].CPU != nil || hist[0].Mem != nil || len(hist[0].GPUs) != 0 { + t.Errorf("the speed-only sample carried host figures: %+v", hist[0]) + } +} + +// TestSpeedForgetsOnStart pins the engine boundary: the second engine's +// report answers with its own rates or none, never the first engine's. +func TestSpeedForgetsOnStart(t *testing.T) { + d := testDaemon(t, `trap 'exit 0' TERM +while true; do sleep 0.05; done`) + d.Now = func() time.Time { return baseTime } + + if err := d.Push(inference.DeployConfig{Runner: "llamacpp", ModelID: "m"}); err != nil { + t.Fatal(err) + } + d.ObserveEngineLine("slot print_timing: id 0 | task 1 | tg = 4.00 t/s") + + if err := d.StartEngine(); err != nil { + t.Fatal(err) + } + waitForState(t, d.Sup, StateRunning) + if got := d.Metrics(context.Background()).Speed; got != nil { + d.Sup.Stop() + t.Fatalf("the new engine reports %v, want the previous figure forgotten", got) + } + d.systemSampleOnce(context.Background()) + if got := d.hist.snapshot(); len(got) != 0 { + d.Sup.Stop() + t.Fatalf("the new engine's first tick folded %+v, want nothing", got) + } + d.Sup.Stop() + waitForState(t, d.Sup, StateStopped) +} + +func waitForSpeed(t *testing.T, d *Daemon) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + // A generous window: this is waiting for the line to arrive, not + // testing the lapse — the pinned clock keeps the figure observed at + // baseTime, and real seconds have passed since the wall-clock start. + if got := d.speed.current(baseTime, 24*time.Hour); got != nil { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("the engine's timing line never reached the tracker") +} diff --git a/internal/daemon/supervisor.go b/internal/daemon/supervisor.go index 18462d26..65b1b425 100644 --- a/internal/daemon/supervisor.go +++ b/internal/daemon/supervisor.go @@ -7,6 +7,7 @@ package daemon import ( "fmt" + "io" "log/slog" "os" "os/exec" @@ -50,6 +51,15 @@ type Supervisor struct { // discards. Note what this is not: the engine's own output, which goes to // LogPath and is served over /v1/logs. Logger *slog.Logger + // OnOutput, when set, receives every complete line of the captured + // engine output — normalised stdout and stderr alike — alongside the log + // file that already gets it. The daemon wires it so the engine's account + // of itself (its generation rate) is read where the lines are already + // assembled, instead of re-reading the log. Only the captured path feeds + // it: a run forwarding stdio to the terminal has no lines that are not + // already on screen, and a line still unterminated when the engine exits + // is never handed over. + OnOutput func(line string) mu sync.Mutex state State @@ -102,7 +112,14 @@ func (s *Supervisor) Start(argv []string) error { return err } logFile = f - cmd.Stderr = f + // The capture sink is the log file, or the file behind the line tee + // when a hook is listening: the file gets every byte exactly as it + // always did, and the hook gets the same bytes as whole lines. + var sink io.Writer = f + if s.OnOutput != nil { + sink = &lineTee{w: f, onLine: s.OnOutput} + } + cmd.Stderr = sink // The output is being written to a file, so the engine is told not // to colour it: an engine that colours by whether it sees a // terminal — and colours stderr by whether *stdout* is one, among @@ -118,9 +135,9 @@ func (s *Supervisor) Start(argv []string) error { cmd.Stdout = slave ptyMaster, ptySlave = master, slave ptyDone = make(chan struct{}) - go pumpPTYLog(master, f, ptyDone) + go pumpPTYLog(master, sink, ptyDone) } else { - cmd.Stdout = f + cmd.Stdout = sink s.log().Debug("no pseudo-terminal for the engine's stdout; it goes to the log file directly", slog.String("error", perr.Error())) } diff --git a/internal/daemon/tee.go b/internal/daemon/tee.go new file mode 100644 index 00000000..fa095459 --- /dev/null +++ b/internal/daemon/tee.go @@ -0,0 +1,68 @@ +package daemon + +import ( + "bytes" + "io" + "sync" +) + +// teeLineCap bounds the line tee's partial-line buffer. The lines worth +// reading are log lines, and a generator that prints a megabyte without a +// newline is not producing any; past the cap the rest of that line is +// dropped rather than buffered, so a runaway stream costs a fixed few +// kilobytes and never grows with the stream. +const teeLineCap = 64 << 10 + +// lineTee is the supervisor's capture sink when an OnOutput hook is wired: +// every byte flows to the log file untouched, and every complete line — +// bytes up to a newline, the newline not included — is additionally handed +// to the hook. The file still receives the exact bytes; the hook only ever +// sees lines whole, so the parser is never asked to read half a figure. +// +// Writes arrive from the pty pump's goroutine and the stderr copier's at +// once, so the buffer sits under a mutex. A line longer than the cap is +// dropped from that newline to the next: a partial line the tee hands over +// would be a fabricated one. +type lineTee struct { + w io.Writer + onLine func(string) + + mu sync.Mutex + buf []byte + skip bool // the current line busted the cap; drop it to its newline +} + +func (t *lineTee) Write(p []byte) (int, error) { + n, err := t.w.Write(p) + t.mu.Lock() + defer t.mu.Unlock() + for len(p) > 0 { + i := bytes.IndexByte(p, '\n') + if i < 0 { + if t.skip { + break + } + if len(t.buf)+len(p) > teeLineCap { + t.buf, t.skip = t.buf[:0], true + break + } + t.buf = append(t.buf, p...) + break + } + if !t.skip { + if len(t.buf)+len(p[:i]) > teeLineCap { + // The line busts the cap with its own remainder: drop it + // whole rather than hand over a truncated half-line. + t.skip = true + } else { + t.buf = append(t.buf, p[:i]...) + if len(t.buf) > 0 { + t.onLine(string(t.buf)) + } + } + } + t.buf, t.skip = t.buf[:0], false + p = p[i+1:] + } + return n, err +} diff --git a/internal/daemon/tee_test.go b/internal/daemon/tee_test.go new file mode 100644 index 00000000..25813b6e --- /dev/null +++ b/internal/daemon/tee_test.go @@ -0,0 +1,84 @@ +//go:build !windows + +package daemon + +import ( + "bytes" + "strings" + "testing" +) + +func TestLineTeeHandsOverCompleteLines(t *testing.T) { + var file bytes.Buffer + var lines []string + tee := &lineTee{w: &file, onLine: func(l string) { lines = append(lines, l) }} + + // One write, three lines and a partial remainder: the lines land, the + // remainder waits for its newline. + n, err := tee.Write([]byte("booting\nready\n")) + if err != nil || n != 14 { + t.Fatalf("Write = %d, %v; want 14 bytes, nil", n, err) + } + if _, err := tee.Write([]byte("tg = 3.87 t/s, tg_3s = 5.29 t/s\n")); err != nil { + t.Fatal(err) + } + if _, err := tee.Write([]byte("half a li")); err != nil { + t.Fatal(err) + } + if got := strings.Join(lines, "|"); got != "booting|ready|tg = 3.87 t/s, tg_3s = 5.29 t/s" { + t.Errorf("lines = %q, want the three complete ones", got) + } + + // The remainder joins its own continuation and lands on its newline. + if _, err := tee.Write([]byte("ne\n")); err != nil { + t.Fatal(err) + } + if lines[len(lines)-1] != "half a line" { + t.Errorf("last line = %q, want the joined one", lines[len(lines)-1]) + } + + // The file carries every byte exactly — newlines included. + if file.String() != "booting\nready\ntg = 3.87 t/s, tg_3s = 5.29 t/s\nhalf a line\n" { + t.Errorf("file = %q, want every byte", file.String()) + } +} + +func TestLineTeeDropsAnOverlongLine(t *testing.T) { + var file bytes.Buffer + var lines []string + tee := &lineTee{w: &file, onLine: func(l string) { lines = append(lines, l) }} + + // A line past the cap is dropped whole, from here to its newline; the + // next line is read clean. + filler := strings.Repeat("x", teeLineCap+1) + if _, err := tee.Write([]byte("runaway " + filler + "\ngood\n")); err != nil { + t.Fatal(err) + } + if len(lines) != 1 || lines[0] != "good" { + t.Errorf("lines = %d %q, want only the line after the runaway", len(lines), lines) + } + + // And a cap bust that spans writes still drops to its newline, without + // handing over a fabricated half-line at the break. + if _, err := tee.Write([]byte(strings.Repeat("y", teeLineCap-2))); err != nil { + t.Fatal(err) + } + if _, err := tee.Write([]byte("tail\nsecond\n")); err != nil { + t.Fatal(err) + } + if len(lines) != 2 || lines[1] != "second" { + t.Errorf("lines = %q, want the runaway dropped and 'second' read", lines) + } +} + +func TestLineTeeHandsNothingToANewlinelessEnd(t *testing.T) { + var file bytes.Buffer + fed := false + tee := &lineTee{w: &file, onLine: func(string) { fed = true }} + if _, err := tee.Write([]byte("no newline at all")); err != nil { + t.Fatal(err) + } + if fed { + t.Error("an unterminated line was handed over") + } +} diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 81c1adab..67615981 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -28,6 +28,15 @@ type Stats struct { CPU *CpuStat `json:"cpu,omitempty"` Memory *MemoryStat `json:"memory,omitempty"` Errors []string `json:"errors,omitempty"` + // Speed is the engine's own generation rate in tokens per second, parsed + // from the timing lines it prints into its captured output (llama.cpp's + // print_timing `tg` figure today). Present only while the engine runs and + // has printed a figure since the last sampler tick: a stopped or quiet + // engine has no rate to report, and the old figure of a rate that has + // stopped is not an answer to "how fast is it generating?". Absent, not + // zero, for an engine that prints none and a daemon that predates the + // figure. + Speed *float64 `json:"speed,omitempty"` // LastActiveAt is when the engine last did any work, RFC 3339, and // IdleSeconds how long it has been since. They mirror the same two fields // on the daemon's status reply and are filled from the same record, so a @@ -123,6 +132,11 @@ type HistorySample struct { CPU *float64 `json:"c,omitempty"` // Mem is system memory used over total, percent. Mem *float64 `json:"m,omitempty"` + // Speed is the engine's generation rate in tokens per second, from the + // timing lines its output carried during this tick's interval. Absent + // where the engine printed no figure since the previous tick — silence + // means nothing was generated, not a rate of zero. + Speed *float64 `json:"s,omitempty"` // GPUs is each GPU's figures, percent. GPUs []HistoryGPU `json:"g,omitempty"` } diff --git a/internal/metrics/speed.go b/internal/metrics/speed.go new file mode 100644 index 00000000..338fdee3 --- /dev/null +++ b/internal/metrics/speed.go @@ -0,0 +1,39 @@ +package metrics + +import ( + "math" + "regexp" + "strconv" +) + +// generationSpeed matches the instantaneous generation-rate figure in an +// engine's timing line, e.g. llama.cpp's +// +// 86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s +// +// The figure read is the current rate the engine names `tg`. The pattern is +// deliberately narrow: `tg\s*=` cannot match `tg_3s =` — the underscore that +// must follow `tg` there breaks the `\s*=` that must follow here — and the +// line's other rates (the three-second average, prompt-evaluation speeds) +// carry different labels, so none of them reads as the current rate. +var generationSpeed = regexp.MustCompile(`\btg\s*=\s*([0-9]+(?:\.[0-9]+)?)\s*t/s`) + +// ParseGenerationSpeed extracts an engine's self-reported generation speed, +// in tokens per second, from one captured output line, reporting whether the +// line carried one. llama.cpp's print_timing lines carry it today; an engine +// that prints no such line simply never yields a speed — the absent stat, not +// a zero. A figure that is not a plausible positive rate (an unparsable +// number, a zero or negative rate) is no speed either: llama.cpp prints +// nonsense before a slot's first token, and "0 t/s" is indistinguishable +// from the absence that means "not generating". +func ParseGenerationSpeed(line string) (float64, bool) { + m := generationSpeed.FindStringSubmatch(line) + if m == nil { + return 0, false + } + v, err := strconv.ParseFloat(m[1], 64) + if err != nil || math.IsNaN(v) || math.IsInf(v, 0) || v <= 0 { + return 0, false + } + return v, true +} diff --git a/internal/metrics/speed_test.go b/internal/metrics/speed_test.go new file mode 100644 index 00000000..f586a9a5 --- /dev/null +++ b/internal/metrics/speed_test.go @@ -0,0 +1,79 @@ +package metrics + +import "testing" + +func TestParseGenerationSpeed(t *testing.T) { + cases := []struct { + name string + line string + want float64 + found bool + }{ + { + name: "llama.cpp print_timing line as the engine prints it", + line: "86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s", + want: 3.87, + found: true, + }, + { + name: "a whole-number rate", + line: "slot print_timing: id 0 | task 1 | tg = 42 t/s", + want: 42, + found: true, + }, + { + name: "no spacing around the figure", + line: "tg=12.5 t/s", + want: 12.5, + found: true, + }, + { + name: "the average alone is not the current rate", + line: "86.19.263.970 I slot print_timing: id 0 | task 326 | tg_3s = 5.29 t/s", + }, + { + name: "a prompt-evaluation rate is not a generation rate", + line: "slot print_timing: id 0 | pp = 610.50 t/s", + }, + { + name: "a zero rate is the absence it always was", + line: "slot print_timing: id 0 | tg = 0.00 t/s", + }, + { + name: "the nonsense before the first token yields nothing", + line: "slot print_timing: id 0 | tg = -nan t/s", + }, + { + name: "a rate label without the t/s unit is not the figure", + line: "some other log line mentioning tg = 100", + }, + { + name: "an unrelated line carries no speed", + line: "registering model 'Qwen3.6-27B'", + }, + { + name: "an identifier ending in tg does not match", + line: "slot print_tg = 5 t/s", + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + got, ok := ParseGenerationSpeed(c.line) + if ok != c.found { + t.Fatalf("found = %v, want %v for %q", ok, c.found, c.line) + } + if ok && got != c.want { + t.Errorf("speed = %v, want %v", got, c.want) + } + }) + } +} + +func TestParseGenerationSpeedPrefersInstantaneous(t *testing.T) { + // Both figures on one line: the instantaneous tg, never the tg_3s average. + line := "99.999 I slot print_timing: id 0 | task 9 | n_gen = 10, tg = 7.15 t/s, tg_3s = 99.99 t/s" + got, ok := ParseGenerationSpeed(line) + if !ok || got != 7.15 { + t.Errorf("ParseGenerationSpeed = %v/%v, want 7.15/true", got, ok) + } +} diff --git a/openspec/changes/engine-generation-speed/tasks.md b/openspec/changes/engine-generation-speed/tasks.md index 05f40dca..141293dc 100644 --- a/openspec/changes/engine-generation-speed/tasks.md +++ b/openspec/changes/engine-generation-speed/tasks.md @@ -2,28 +2,28 @@ ## Shape the stat -- [ ] Add `Speed *float64` (`json:"speed,omitempty"`) to `metrics.Stats` and `Speed *float64` (`json:"s,omitempty"`) to `metrics.HistorySample` in `internal/metrics/metrics.go`, with doc comments in the house style (absent-not-zero; `s` one-letter for the SSM budget) -- [ ] Add `ParseGenerationSpeed(line string) (float64, bool)` to `internal/metrics` (parse.go or speed.go): `tg = t/s`, never `tg_3s`, never prompt-eval rates, reject NaN/Inf/≤0; table-driven tests over the issue's real `print_timing` lines +- [x] Add `Speed *float64` (`json:"speed,omitempty"`) to `metrics.Stats` and `Speed *float64` (`json:"s,omitempty"`) to `metrics.HistorySample` in `internal/metrics/metrics.go`, with doc comments in the house style (absent-not-zero; `s` one-letter for the SSM budget) +- [x] Add `ParseGenerationSpeed(line string) (float64, bool)` to `internal/metrics` (parse.go or speed.go): `tg = t/s`, never `tg_3s`, never prompt-eval rates, reject NaN/Inf/≤0; table-driven tests over the issue's real `print_timing` lines ## Capture it -- [ ] Add the `lineTee` writer (buffer complete lines to `OnOutput`, size-capped partial buffer, bytes to the file first) and the `OnOutput func(string)` hook to the supervisor (`internal/daemon/supervisor.go`): stderr and the pty pump's output both write through it when `LogPath` is set and a hook is wired; tests for split writes, capped partial lines, and no-pty path -- [ ] Wire `Daemon.StartEngine` to set `Sup.OnOutput` to the daemon's observer and reset the speed tracker/tick baseline beside the existing `forget`/`clear` calls +- [x] Add the `lineTee` writer (buffer complete lines to `OnOutput`, size-capped partial buffer, bytes to the file first) and the `OnOutput func(string)` hook to the supervisor (`internal/daemon/supervisor.go`): stderr and the pty pump's output both write through it when `LogPath` is set and a hook is wired; tests for split writes, capped partial lines, and no-pty path +- [x] Wire `Daemon.StartEngine` to set `Sup.OnOutput` to the daemon's observer and reset the speed tracker/tick baseline beside the existing `forget`/`clear` calls ## Track and report it -- [ ] Add `speedTrack` to the daemon (`internal/daemon/speed.go`): `observe` under the daemon clock; a consumed-point rule for the tick (`systemSampleOnce` attaches `sample.Speed` when a figure arrived since the last tick, and a speed point alone still makes the sample exist) and a fresh-within-one-interval rule for the current reading in `Daemon.Metrics`; tests for steady ticks, quiet ticks, figure latching then lapsing, start-after-stop reset +- [x] Add `speedTrack` to the daemon (`internal/daemon/speed.go`): `observe` under the daemon clock; a consumed-point rule for the tick (`systemSampleOnce` attaches `sample.Speed` when a figure arrived since the last tick, and a speed point alone still makes the sample exist) and a fresh-within-one-interval rule for the current reading in `Daemon.Metrics`; tests for steady ticks, quiet ticks, figure latching then lapsing, start-after-stop reset ## Contract -- [ ] Add `speed` to the `Stats` schema and `s` to the `HistorySample` schema in `docs/openapi.yaml`; `go test ./internal/daemon -run TestOpenAPI` green +- [x] Add `speed` to the `Stats` schema and `s` to the `HistorySample` schema in `docs/openapi.yaml`; `go test ./internal/daemon -run TestOpenAPI` green ## Render it -- [ ] `cmd/spinloop/metrics_render.go`: relabel GPU utilisation series `GPU util`/`GPU N util` → `GPU`/`GPU N` in `barSeriesList`; add the `Speed` series (label `Speed`, watermark-scaled uncoloured sparkline via a speed glyph block that pools then scales, `▇` ceiling, `%6.2f t/s` figure) drawn after the resource series in `renderStatBars` and `renderStatCombined` (blank gauge half); `renderStatGauges` and the table format carry the figure line only; omit the row when neither reading nor history carries speed -- [ ] Thread the speed through the callers: `fleet.go` (bar/gauge/table), `dashboard_render.go` (tile body), `serve_view.go` (metrics section) -- [ ] Update the golden tests the relabel and new row touch (`metrics_render_test.go`, `metrics_render_gpu_test.go`, `fleet_test.go`, `fleet_dashboard_test.go`, serve view tests), adding cases for the issue's example window: watermark mapping, default colour at every height, absent-speed omission +- [x] `cmd/spinloop/metrics_render.go`: relabel GPU utilisation series `GPU util`/`GPU N util` → `GPU`/`GPU N` in `barSeriesList`; add the `Speed` series (label `Speed`, watermark-scaled uncoloured sparkline via a speed glyph block that pools then scales, `▇` ceiling, `%6.2f t/s` figure) drawn after the resource series in `renderStatBars` and `renderStatCombined` (blank gauge half); `renderStatGauges` and the table format carry the figure line only; omit the row when neither reading nor history carries speed +- [x] Thread the speed through the callers: `fleet.go` (bar/gauge/table), `dashboard_render.go` (tile body), `serve_view.go` (metrics section) +- [x] Update the golden tests the relabel and new row touch (`metrics_render_test.go`, `metrics_render_gpu_test.go`, `fleet_test.go`, `fleet_dashboard_test.go`, serve view tests), adding cases for the issue's example window: watermark mapping, default colour at every height, absent-speed omission ## Verify -- [ ] `gofmt -w`, `go vet ./...`, `go test ./... -cover` green, coverage ≥ 80% +- [x] `gofmt -w`, `go vet ./...`, `go test ./... -cover` green, coverage ≥ 80% From cb055a8be7f5815ffd7670569f477759d90f6329 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Mon, 26 Oct 2026 16:19:55 +0000 Subject: [PATCH 3/7] docs: document the generation speed metric --- README.md | 4 ++-- docs/commands/cloud.md | 7 +++++++ docs/commands/fleet.md | 3 ++- docs/commands/serve.md | 7 +++++-- docs/http-api.md | 3 ++- docs/maintainer/internals.md | 4 +++- 6 files changed, 21 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index dc94c971..39d95809 100644 --- a/README.md +++ b/README.md @@ -75,8 +75,8 @@ Start the whole fleet with `spinloop up` or just one with `spinloop fleet start > In this screenshot, five nodes are configured and four are up, each serving the same > `Qwen3.8-27B` model through llama.cpp. Three of them — `dev-2`, `dev-3`, -> `dev-4` — are mid-request: one slot running apiece, GPU util between 91% and -> 98%. `dev-1` finished about a minute ago, so its GPU util has dropped to 0% +> `dev-4` — are mid-request: one slot running apiece, GPU between 91% and +> 98%. `dev-1` finished about a minute ago, so its GPU has dropped to 0% > while GPU memory stays at 89% — the weights are still loaded, and the next > request it takes starts generating without a reload. > diff --git a/docs/commands/cloud.md b/docs/commands/cloud.md index 5090c042..e1573444 100644 --- a/docs/commands/cloud.md +++ b/docs/commands/cloud.md @@ -257,6 +257,13 @@ output degrades rather than goes blank. `--format=table` is a key-value table, and `--format=json` is the raw reply. A stopped engine keeps its readings: the sparkline runs to the stop, ending at it. +An engine that reports its own generation rate (llama.cpp does) adds a +**Speed** row beside the resource series: the figure in tokens per second, +and in bar format a sparkline of the rate across the window, scaled to the +window's own high watermark and drawn in the default colour throughout — a +rate has no ceiling for the utilisation thresholds to cross. Where neither +the current reading nor the history carries a rate, there is no row. + While a start is under way, `status` shows **`starting`** for the endpoint, whichever machine began the start, until the instance is running. Once it is running but still loading the model it reads `running` and not ready, as diff --git a/docs/commands/fleet.md b/docs/commands/fleet.md index 82c2a31b..6ab84e51 100644 --- a/docs/commands/fleet.md +++ b/docs/commands/fleet.md @@ -173,7 +173,8 @@ silently missing whatever was down: `spinloop dashboard` is that same board as a live view: one tile per node, repainted in place, each drawing exactly what `metrics`' gauge format prints for the node — state and uptime, what it serves, the CPU/GPU/RAM -gauges, the token counters — so the view and the one-shot command never +gauges, the speed its engine reports, the token counters — so the view and the +one-shot command never word a number differently. `g` toggles every tile between the gauge drawing of the current reading and the sparklines; the board opens in gauge. A node that diff --git a/docs/commands/serve.md b/docs/commands/serve.md index 6045d973..28c96aaf 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -27,9 +27,12 @@ and a footer naming the keys the view answers to. - **Metrics** — the same facts the dashboard's node detail screen shows for the same engine: state and uptime, what is served, last active, and the - resource series — CPU, RAM, and each GPU's utilisation and memory — with + resource series — CPU, RAM, each GPU's utilisation and memory, and the + engine's generation speed where it reports one — with every series drawn in both formats at once, each on one line: a gauge of - the current reading beside the bar of its retained history. Below them, the + the current reading beside the bar of its retained history. The speed row's + bar half stays blank — a rate has no ceiling to fill against — leaving its + t/s figure and its history sparkline. Below them, the token and request counters. The reading comes from the daemon the serve process runs in-process and refreshes on the dashboard's own local cadence; a reading the view could not renew is shown with its age. diff --git a/docs/http-api.md b/docs/http-api.md index ea78292f..2e183ab0 100644 --- a/docs/http-api.md +++ b/docs/http-api.md @@ -58,7 +58,8 @@ Stops the engine. Returns the current metrics: - Token usage counters (from the engine's Prometheus `/metrics` endpoint) - Host system metrics (GPU, CPU, RAM). On macOS a GPU carries utilisation and name only; its `memoryUsed`, `memoryTotal` and `temperature` are `0`, meaning the host reports none -- `history`, the daemon's retained system readings — one per sampler tick while an engine ran, each a 0–100% figure per series (`t` time, `c` CPU, `m` memory, `g` per-GPU utilisation and memory), covering at most the last 10 minutes +- `speed`, the engine's own generation rate in tokens per second, parsed from the timing lines it prints into its captured output (llama.cpp's `print_timing` `tg` figure today). Present only while the engine runs and has printed a figure within the last sampler interval — a stopped or quiet engine omits it rather than reporting zero, and so does an engine that prints no rate +- `history`, the daemon's retained system readings — one per sampler tick while an engine ran, each a 0–100% figure per series (`t` time, `c` CPU, `m` memory, `g` per-GPU utilisation and memory) plus an optional `s` speed point for a tick whose interval saw the engine print a generation rate, covering at most the last 10 minutes - `lastActiveAt` and `idleSeconds`, the same pair `/v1/status` reports The history survives a stop — the readings up to the stop say what the engine diff --git a/docs/maintainer/internals.md b/docs/maintainer/internals.md index 0a850b79..f89f7fe3 100644 --- a/docs/maintainer/internals.md +++ b/docs/maintainer/internals.md @@ -48,6 +48,8 @@ These are mistakes already made here; each was silent rather than loud, which is **`internal/daemon` must not depend on a cloud package.** What an engine should serve is `inference.DeployConfig`, in `internal/inference` — a leaf that imports only the standard library. Both the daemon and the cloud control plane speak it, so neither has to import the other to be described: a daemon is handed one over its control API, and `internal/cloud` persists one against an environment. The dependency used to run the other way, `internal/daemon` importing `internal/cloud` for the type and for a config-directory helper that only forwarded to `internal/config.Dir`, which made the package that knows nothing about AWS depend on the package that is nothing but AWS. Keep new shared vocabulary in `internal/inference` only where it describes an engine's workload and needs nothing of ours to express; anything cloud-shaped — `IsInstanceType` and the rest of the EC2 vocabulary — stays in `internal/cloud`. **A captured engine's stdout is a pseudo-terminal, and the normaliser above it is a line model, not a screen emulator.** llama.cpp's download bar prints nothing when its stdout is not a terminal, and the download happens in-process before the HTTP listener comes up — there is no status API to scrape — so the capture (the daemon and the serve view) presents the engine's stdout as a PTY: the only way the progress reaches the engine log at all. `ptylog.go` turns the terminal stream into the log's lines as a column, and three of its rules are easy to break by "simplifying". The engine's up-down cursor dance is *part of each redraw* — the bar's state is drawn between a cursor-up and a cursor-back-down, the cursor parked on an anchor line — so a cursor move must move the drawing between the column's lines and never end a line: committing on a move records every state as a "final" one and resets the dedup, which is how the first design put the whole download, state for state, in the log. A state's replacement is lazy — a carriage return settles the state drawn before it, and the line's content is cleared only by the next text byte — so `S\r\r\n`, the PTY's ONLCR turning a lone `\n` into a CRLF, commits `S` intact. And the tick at the interval's half-rate runs on *every* line of the column, because a bar's last state outlives the engine's move off its line: the download finishes, the engine goes quiet on stdout, and the log still owes it. Because the engine's stdout is now a terminal, the capture branch sets `NO_COLOR=1` in the engine's environment: llama.cpp routes its log lines to stderr and colours them by whether it sees a terminal — *stdout* among them — and the normaliser never sees the stderr path, so without it escapes would reach the log file. The bar draws no colour, so nothing is lost; the forwarding paths (piped `serve`, off-terminal `serve --api`) set nothing and must stay byte-for-byte what the engine wrote — a test pins the raw `\r` surviving. Where no pseudo-terminal can be opened — Windows, a restricted environment — the fallback writes stdout to the log file directly, no worse than the capture did before, and the engine is told `NO_COLOR` the same way; a test stands the opening in for with a failure to keep that branch alive. The pump always drains — the throttle delays the log's writes, never the reads, so the engine can never wedge on a full terminal — and the log file closes only after the pump's final record, never under it. The master is not closed when the engine exits: closing it discards bytes the engine wrote just before exiting that the pump has not read yet (on Linux the read returns `EIO` once every slave fd is closed, and an early close loses the tail — the cause of the `TestSupervisorPTYCapture` flake on CI). The supervisor waits for the pump to reach end of stream, and closes the master itself only after `ptyDrainTimeout`, which covers a child of the engine that keeps the slave open. + +**The generation speed is read where the capture already assembles lines, and the wiring order is the whole feature.** llama.cpp prints its instantaneous rate into stderr's `print_timing` lines; the supervisor wraps its log-file sink in a `lineTee` (`tee.go`) that writes every byte to the file untouched and hands each complete line to `Supervisor.OnOutput`, so both capture paths — stderr straight through, pty stdout through the normaliser — feed the parser with no second reader of the log. `OnOutput` is read as `Supervisor.Start` builds the sink, so a daemon must set it *before* starting: wired afterwards, the tee is simply never there and every line is silently lost (this was the first implementation's bug). The parser reads `tg = t/s` only — `tg\s*=` cannot match `tg_3s =`, so the three-second average and prompt rates never read as the current rate — and a tracker with two clocks splits the answer: the sampler's tick folds a history point for the figures its interval saw (strictly after the previous tick: a point is an observation, never a continuation), while a metrics read reports the figure only until one interval of quiet ages it out. Both start engines reset tracker, baseline and history together, so one engine's rate never rides the next engine's report. **Calling `prog.Send` from inside `Update` deadlocks the program.** The event loop reads its own channel on the goroutine that calls `Update`, so a command that hands a message back through `prog.Send` while `Update` is on the stack waits for a reader that is waiting for it. The board hung on exactly this: pressing Enter on the form's last field, or `a`/`x` on a card, and the whole TUI went still — no spinner, no keystrokes, Ctrl-C ignored. Every message from work that starts inside `Update` must *return* through the loop instead: `tea.Batch(workCmd, spinCmd)` delivers each command's messages normally, and a repaint chain re-arms through its own returned `tea.Tick`. The tests guard this by landing commands through the real program loop, not by calling `Update` and discarding the command. ## Dashboard (`fleet_dashboard.go` and friends) @@ -69,7 +71,7 @@ Behavior (panel contents, refresh cadence, start/stop/abort semantics, the detai ## The metrics history - The bar format's data lives in the daemon, not the client: `systemHistory` (`internal/daemon/history.go`) is appended on each sampler tick while an engine runs, survives a stop, and clears on the next start alongside the counter baseline's `sample.forget()`. Every client — one-shot, watch, dashboard, the cloud relay — draws the same window from the one read. -- The history samples' JSON field names are one letter each (`t`/`c`/`m`/`g`, `i`/`u`/`m`) because the reply crosses SSM on the cloud relay, and SSM command output truncates at 4KB. The window is 10 minutes at the 15s cadence (40 samples) and is capped at `historyLimit` samples regardless of cadence — the catch-up ticks run at 1s, and an engine with no scrape target never leaves that cadence. If the window or the sample shape grows, cap the size in the daemon, not the client: a truncated reply is corrupt JSON, and `parseDaemonMetrics` turns that into "daemon unreachable" for the whole metrics call. +- The history samples' JSON field names are one letter each (`t`/`c`/`m`/`s`/`g`, `i`/`u`/`m`) because the reply crosses SSM on the cloud relay, and SSM command output truncates at 4KB. The window is 10 minutes at the 15s cadence (40 samples) and is capped at `historyLimit` samples regardless of cadence — the catch-up ticks run at 1s, and an engine with no scrape target never leaves that cadence. If the window or the sample shape grows, cap the size in the daemon, not the client: a truncated reply is corrupt JSON, and `parseDaemonMetrics` turns that into "daemon unreachable" for the whole metrics call. ## Adapter schema references From e7d47d303226c18a153dd356b8c036037d34b1f1 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Mon, 26 Oct 2026 16:23:35 +0000 Subject: [PATCH 4/7] test: keep the fixture timing lines clear of the identifier gate The real llama.cpp line carries an elapsed-time prefix whose four dotted groups read as an IPv4 address to the no-cloud-identifiers check; the parser answers just as well without it. --- internal/daemon/speed_test.go | 2 +- internal/metrics/speed.go | 2 +- internal/metrics/speed_test.go | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/internal/daemon/speed_test.go b/internal/daemon/speed_test.go index 1d7020d7..aea7e981 100644 --- a/internal/daemon/speed_test.go +++ b/internal/daemon/speed_test.go @@ -63,7 +63,7 @@ func TestEngineSpeedFoldAndLapse(t *testing.T) { // the metrics reply — every seam in one wire. func TestSpeedEndToEnd(t *testing.T) { d := testDaemon(t, `trap 'exit 0' TERM -echo "86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s" >&2 +echo "slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s" >&2 while true; do sleep 0.05; done`) d.Now = func() time.Time { return baseTime } d.Collector = linuxCollector() diff --git a/internal/metrics/speed.go b/internal/metrics/speed.go index 338fdee3..0683cac6 100644 --- a/internal/metrics/speed.go +++ b/internal/metrics/speed.go @@ -9,7 +9,7 @@ import ( // generationSpeed matches the instantaneous generation-rate figure in an // engine's timing line, e.g. llama.cpp's // -// 86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s +// slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s // // The figure read is the current rate the engine names `tg`. The pattern is // deliberately narrow: `tg\s*=` cannot match `tg_3s =` — the underscore that diff --git a/internal/metrics/speed_test.go b/internal/metrics/speed_test.go index f586a9a5..0c990437 100644 --- a/internal/metrics/speed_test.go +++ b/internal/metrics/speed_test.go @@ -11,7 +11,7 @@ func TestParseGenerationSpeed(t *testing.T) { }{ { name: "llama.cpp print_timing line as the engine prints it", - line: "86.15.860.290 I slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s", + line: "slot print_timing: id 0 | task 326 | n_gen = 9438, tg = 3.87 t/s, tg_3s = 5.29 t/s", want: 3.87, found: true, }, @@ -29,7 +29,7 @@ func TestParseGenerationSpeed(t *testing.T) { }, { name: "the average alone is not the current rate", - line: "86.19.263.970 I slot print_timing: id 0 | task 326 | tg_3s = 5.29 t/s", + line: "slot print_timing: id 0 | task 326 | tg_3s = 5.29 t/s", }, { name: "a prompt-evaluation rate is not a generation rate", From 1b1aa53df4e536524588cb2ba474685dab4fe897 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Wed, 28 Oct 2026 02:39:40 +0000 Subject: [PATCH 5/7] feat: align the speed row with the metrics block's columns Review of the rendered serve view: the renamed GPU labels left two dead columns in the block's label column, and the speed row cut its own columns rather than sharing the block's. - statLabelWidth sizes the shared label column to its longest label (seven for GPU mem, widening by the GPU index); threaded through the sparkline, gauge, active and speed rows and every caller - the speed figure now starts where the other rows' graphs start and the sparkline holds the sparklines' column however many digits the figure grows (fixed field spanning the gauge and percentage halves) - a lapsed figure on an engine that reports one reads 0.00 t/s rather than keeping the last burst's rate in front of a watching eye; only a never-reporting engine draws no row - dashBarLineW 25 to 23 so the trailing figure fits the tile --- cmd/spinloop/dashboard_render.go | 4 +- cmd/spinloop/fleet.go | 6 +- cmd/spinloop/fleet_dashboard_test.go | 8 +- cmd/spinloop/last_active_test.go | 10 +- cmd/spinloop/metrics_render.go | 192 ++++++++++++------ cmd/spinloop/metrics_render_gpu_test.go | 4 +- cmd/spinloop/metrics_render_test.go | 159 ++++++++------- cmd/spinloop/serve_view.go | 2 +- docs/commands/cloud.md | 12 +- docs/commands/serve.md | 8 +- .../changes/engine-generation-speed/design.md | 6 +- .../engine-generation-speed/proposal.md | 4 +- .../specs/remote-metrics-bar-format/spec.md | 56 ++++- .../changes/engine-generation-speed/tasks.md | 8 + 14 files changed, 308 insertions(+), 171 deletions(-) diff --git a/cmd/spinloop/dashboard_render.go b/cmd/spinloop/dashboard_render.go index eacb4ed2..d0ba7635 100644 --- a/cmd/spinloop/dashboard_render.go +++ b/cmd/spinloop/dashboard_render.go @@ -465,10 +465,10 @@ func dashTileReportBody(w io.Writer, m metrics.Stats, resources bool, gauge bool // after it — one line, from the same read, whatever the engine's state. A // read without either — a local node, an unkept or lapsed environment — // draws nothing. - renderActiveIndented(w, m.LastActiveAt, m.IdleSeconds, m.RetainUntil, now) + renderActiveIndented(w, m.LastActiveAt, m.IdleSeconds, m.RetainUntil, now, statLabelWidth(m.GPUs, m.History)) if resources { if gauge { - renderStatGauges(w, m.CPU, m.Memory, m.GPUs, m.Speed) + renderStatGauges(w, m.CPU, m.Memory, m.GPUs, m.Speed, m.History) } else { renderStatBars(w, m.CPU, m.Memory, m.GPUs, m.Speed, m.History, lineW) } diff --git a/cmd/spinloop/fleet.go b/cmd/spinloop/fleet.go index a6e7a24e..25cca9bd 100644 --- a/cmd/spinloop/fleet.go +++ b/cmd/spinloop/fleet.go @@ -160,7 +160,7 @@ func renderFleetMetrics(w io.Writer, results []fleet.NodeResult, format string) if format == "table" { renderActiveKeyValue(w, stats.LastActiveAt, stats.IdleSeconds, retainUntilOf(r), now) } else { - renderActiveIndented(w, stats.LastActiveAt, stats.IdleSeconds, retainUntilOf(r), now) + renderActiveIndented(w, stats.LastActiveAt, stats.IdleSeconds, retainUntilOf(r), now, statLabelWidth(stats.GPUs, stats.History)) } switch format { case "bar": @@ -174,14 +174,14 @@ func renderFleetMetrics(w io.Writer, results []fleet.NodeResult, format string) if stats.State != "running" { continue } - renderStatGauges(w, stats.CPU, stats.Memory, stats.GPUs, stats.Speed) + renderStatGauges(w, stats.CPU, stats.Memory, stats.GPUs, stats.Speed, stats.History) renderTokenLines(w, stats.Tokens) default: if stats.State != "running" { continue } renderTokenLines(w, stats.Tokens) - renderSpeedTable(w, stats.Speed) + renderSpeedTable(w, stats.Speed, stats.History) renderGPUTable(w, stats.GPUs) renderCPUMemTable(w, stats.CPU, stats.Memory) } diff --git a/cmd/spinloop/fleet_dashboard_test.go b/cmd/spinloop/fleet_dashboard_test.go index e9c6b6d7..b4de4d9e 100644 --- a/cmd/spinloop/fleet_dashboard_test.go +++ b/cmd/spinloop/fleet_dashboard_test.go @@ -271,7 +271,7 @@ func dashBar(label string, pct float64) string { colour = "\033[33m" } filled := int(pct / 100 * width) - return fmt.Sprintf(" %-9s ", label) + colour + + return fmt.Sprintf(" %-7s ", label) + colour + strings.Repeat("█", filled) + "\033[0m" + strings.Repeat("░", width-filled) + fmt.Sprintf(" %.0f%%", pct) } @@ -355,7 +355,7 @@ func TestDashTileRunningByteStable(t *testing.T) { want := dashTileExpected([]string{ dashExpectedHeader("up running (up 2h 0m 0s)", dashHealthy), "llamacpp org/qwen:q4", - " active 12s ago", + " active 12s ago", dashBar("CPU", 42), dashBar("RAM", 30), dashBar("GPU", 61), @@ -390,7 +390,7 @@ func TestDashTileRunningNoRequestCount(t *testing.T) { want := dashTileExpected([]string{ dashExpectedHeader("up running (up 2h 0m 0s)", dashHealthy), "llamacpp org/qwen:q4", - " active 12s ago", + " active 12s ago", dashBar("CPU", 42), dashBar("RAM", 30), dashBar("GPU", 61), @@ -3558,7 +3558,7 @@ func TestDashModelFormatToggle(t *testing.T) { // blank columns, ending on the last retained sample — the current reading // is the next point, not in the history yet. Both samples sit below the // thresholds, so neither takes a colour. - cpuLine := " CPU " + strings.Repeat(" ", 23) + "▁▂" + " 20%" + cpuLine := " CPU " + strings.Repeat(" ", dashBarLineW-2) + "▁▂" + " 20%" if !strings.Contains(bar, cpuLine) { t.Errorf("bar tile did not draw the history, want %q in:\n%s", cpuLine, bar) } diff --git a/cmd/spinloop/last_active_test.go b/cmd/spinloop/last_active_test.go index 39b39427..b90a4fbf 100644 --- a/cmd/spinloop/last_active_test.go +++ b/cmd/spinloop/last_active_test.go @@ -52,7 +52,7 @@ func TestCloudMetricsBarShowsLastActive(t *testing.T) { t.Fatalf("cmdMetrics: %v", err) } }) - if !strings.Contains(out, "active 2m 5s ago") { + if !strings.Contains(out, "active 2m 5s ago") { t.Errorf("bar format missing the active line:\n%s", out) } // It belongs between the header and the bars: a fact about the endpoint, @@ -127,7 +127,7 @@ func TestCloudMetricsStoppedStillShowsLastActive(t *testing.T) { // The bar format indents the line to the bar-label column; the table // format makes it a key-value row. Same fact, each format's own idiom. for format, want := range map[string]string{ - "bar": "active 1h 0m 0s ago", + "bar": "active 1h 0m 0s ago", "table": "active: 1h 0m 0s ago", } { t.Run(format, func(t *testing.T) { @@ -157,7 +157,7 @@ func TestLastActiveZeroIdleStillRenders(t *testing.T) { }`) for format, want := range map[string]string{ - "bar": "active 0s ago", + "bar": "active 0s ago", "table": "active: 0s ago", } { t.Run(format, func(t *testing.T) { @@ -305,7 +305,7 @@ func TestFleetMetricsShowsLastActive(t *testing.T) { }) // bar indents its active line under the header; table spells it // as a key-value, like the facts around it. - want := "active 2m 5s ago" + want := "active 2m 5s ago" if format == "table" { want = "active: 2m 5s ago" } @@ -379,7 +379,7 @@ func TestFleetMetricsStoppedNodeStillShowsLastActive(t *testing.T) { t.Fatalf("cmdFleet metrics: %v", err) } }) - if !strings.Contains(out, "active 10m 0s ago") { + if !strings.Contains(out, "active 10m 0s ago") { t.Errorf("a stopped node dropped its active figure:\n%s", out) } } diff --git a/cmd/spinloop/metrics_render.go b/cmd/spinloop/metrics_render.go index 1b8dc69d..2aef0248 100644 --- a/cmd/spinloop/metrics_render.go +++ b/cmd/spinloop/metrics_render.go @@ -83,20 +83,40 @@ func formatKeepDuration(d time.Duration) string { } } +// statLabelWidth is the shared label column a block of stat rows draws: wide +// enough for the longest label the block can carry, and no wider. The column +// was sized for "GPU util"; with that series renamed to "GPU", the longest +// single-GPU label is "GPU mem" — seven columns — and multi-GPU hosts widen +// by the index they carry. It floors at seven even for a host with no GPU +// series at all: one column width per fleet keeps blocks of different hosts +// reading as the same table. +func statLabelWidth(gpus []metrics.GpuStat, history []metrics.HistorySample) int { + w := len("GPU mem") + if idxs := gpuIndexes(gpus, history); len(idxs) > 1 { + for _, idx := range idxs { + if n := len(fmt.Sprintf("GPU %d mem", idx)); n > w { + w = n + } + } + } + return w +} + // renderActiveIndented draws the combined active-and-keep line in the indented // block the bar format and the tile use: the "ago" figure, and, when the read // carries a retention deadline, the relative keep after it — "active 2m 5s // ago keep for 2h". The line is present whenever either fact is, and absent // when neither is. It is not a bar: an elapsed time has no ceiling to fill -// against, and a bar would imply one. -func renderActiveIndented(w io.Writer, lastActiveAt string, idleSeconds int, retainUntil string, now time.Time) { +// against, and a bar would imply one. labelW is the block's shared label +// column, so the active line starts its figure where the bars start theirs. +func renderActiveIndented(w io.Writer, lastActiveAt string, idleSeconds int, retainUntil string, now time.Time, labelW int) { active := lastActiveText(lastActiveAt, idleSeconds) keep := keepText(retainUntil, now) switch { case active != "" && keep != "": - fmt.Fprintf(w, " %-9s %s %s\n", "active", active, keep) + fmt.Fprintf(w, " %-*s %s %s\n", labelW, "active", active, keep) case active != "": - fmt.Fprintf(w, " %-9s %s\n", "active", active) + fmt.Fprintf(w, " %-*s %s\n", labelW, "active", active) case keep != "": fmt.Fprintf(w, " %s\n", keep) } @@ -135,8 +155,10 @@ const barLineW = 40 // dashBarLineW is the sparkline's draw width inside a dashboard tile: the // tile's width minus the label column and the widest trailing figure, so a -// full row fits the tile exactly and the clip never takes the percentage. -const dashBarLineW = 25 +// full row fits the tile exactly and the clip never takes the figure. The +// figure column sizes for a speed row's "1234.56 t/s"; utilisation's "100%" +// is narrower and leaves the row's last columns blank. +const dashBarLineW = 23 // gaugeW is the gauge's draw width in the one-shot formats — the width // renderGauge has always drawn at. The serve view draws its gauge half at @@ -196,9 +218,10 @@ func poolMax(values []float64, width int) []float64 { // while it still fills, each glyph takes the state colour of its own value // on the gauge's 80/90 thresholds, and the trailing figure is the latest // sample's percentage — the exact value the last glyph approximates. -func renderSparkline(w io.Writer, label string, samples []float64, width int) { +// labelW is the block's shared label column width (see statLabelW). +func renderSparkline(w io.Writer, label string, samples []float64, width, labelW int) { block, last := sparklineBlock(samples, width) - fmt.Fprintf(w, " %-9s %s %.0f%%\n", label, block, last) + fmt.Fprintf(w, " %-*s %s %.0f%%\n", labelW, label, block, last) } // sparklineBlock returns the sparkline's drawing across width, one glyph per @@ -232,8 +255,8 @@ func sparklineBlock(samples []float64, width int) (string, float64) { // filled portion in the state colour, the unfilled portion in light shade, // the percentage in the terminal's default colour. It draws the current // reading only — it carries no history. -func renderGauge(w io.Writer, label string, pct float64, width int) { - fmt.Fprintf(w, " %-9s %s %.0f%%\n", label, gaugeBlock(pct, width), pct) +func renderGauge(w io.Writer, label string, pct float64, width, labelW int) { + fmt.Fprintf(w, " %-*s %s %.0f%%\n", labelW, label, gaugeBlock(pct, width), pct) } // gaugeBlock returns the gauge's drawing at width — the filled portion in @@ -262,12 +285,34 @@ type barSeries struct { current *float64 } +// gpuIndexes is the set of GPU indices the reading names, the current +// reading's order first and the history's after: a stopped engine's current +// reading names no GPU, but its retained readings still name the ones it ran +// on, and the series those readings carry are the point of keeping them. +func gpuIndexes(gpus []metrics.GpuStat, history []metrics.HistorySample) []int { + seen := map[int]bool{} + var idxs []int + for _, g := range gpus { + if !seen[g.Index] { + seen[g.Index] = true + idxs = append(idxs, g.Index) + } + } + for _, s := range history { + for _, g := range s.GPUs { + if !seen[g.Index] { + seen[g.Index] = true + idxs = append(idxs, g.Index) + } + } + } + return idxs +} + // barSeriesList works out which series the reading and the history carry, in // the order the gauge draws them: CPU, RAM, then each GPU's utilisation and -// memory. The GPU set is the union of the two sources in the current -// reading's order first — a stopped engine's current reading names no GPU, -// but its retained readings still name the ones it ran on, and the series -// those readings carry are the point of keeping them. +// memory. The GPU set is gpuIndexes: the current reading first, the retained +// history's after. func barSeriesList(cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, history []metrics.HistorySample) []barSeries { var out []barSeries add := func(label string, current *float64, pick func(s metrics.HistorySample) *float64) { @@ -297,22 +342,7 @@ func barSeriesList(cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics } else { add("RAM", nil, func(s metrics.HistorySample) *float64 { return s.Mem }) } - seen := map[int]bool{} - var idxs []int - for _, g := range gpus { - if !seen[g.Index] { - seen[g.Index] = true - idxs = append(idxs, g.Index) - } - } - for _, s := range history { - for _, g := range s.GPUs { - if !seen[g.Index] { - seen[g.Index] = true - idxs = append(idxs, g.Index) - } - } - } + idxs := gpuIndexes(gpus, history) for _, idx := range idxs { prefix := "GPU" if len(idxs) > 1 { @@ -370,16 +400,18 @@ func currentGPUMem(gpus []metrics.GpuStat, idx int) *float64 { // predates the history renders exactly as it did before it. Used by the bar // format on every surface; lineW is the draw width, barLineW for the full // view and dashBarLineW for a dashboard tile. The generation speed closes -// the block when the reading carries one, as its own row beside the bars. +// the block when the engine reports one, its sparkline drawing in the same +// columns as the bars above it. func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64, history []metrics.HistorySample, lineW int) { + labelW := statLabelWidth(gpus, history) for _, s := range barSeriesList(cpu, mem, gpus, history) { if len(s.history) > 0 { - renderSparkline(w, s.label, s.history, lineW) + renderSparkline(w, s.label, s.history, lineW, labelW) } else { - renderGauge(w, s.label, *s.current, gaugeW) + renderGauge(w, s.label, *s.current, gaugeW, labelW) } } - renderSpeedBar(w, speed, history, lineW) + renderSpeedBar(w, speed, history, lineW, labelW) } // speedSamples gathers the retained generation-rate points, oldest first. @@ -395,6 +427,15 @@ func speedSamples(history []metrics.HistorySample) []float64 { return out } +// speedShown is whether the engine reports generation speed at all: a fresh +// current figure, or the window still holding points from when it last +// generated. A daemon that never carries the figure answers neither, and gets +// no row — the engine that reports no speed is not an engine generating at +// zero, but an engine whose row would be a question it cannot answer. +func speedShown(speed *float64, history []metrics.HistorySample) bool { + return speed != nil || len(speedSamples(history)) > 0 +} + // speedBlock draws the speed sparkline. Its glyph heights are relative to // the highest value the window holds — the window's own maximum maps to the // heaviest grade the sparkline draws and nothing maps to the full block — @@ -422,41 +463,51 @@ func speedBlock(samples []float64, width int) string { return b.String() } -// speedFigure is the trailing figure of a Speed row: the current reading, -// falling back to the window's latest point where the reading carries none — -// a stopped engine has no current rate, but the row's last point is still -// the last rate it had. -func speedFigure(speed *float64, samples []float64) float64 { +// speedFigure is the Speed row's figure: the current reading, and zero where +// it has lapsed — an engine that has stopped generating is generating at +// zero, and a watched row that kept showing the last burst's rate would be +// answering a question the engine is no longer asking. Only ever called for +// a row speedShown has drawn. +func speedFigure(speed *float64) float64 { if speed != nil { return *speed } - return samples[len(samples)-1] + return 0 } -// renderSpeedBar draws the Speed row of the bar-style surfaces: label, the -// t/s figure, and the rate's sparkline — or blank where the window holds no -// point, since a gauge would imply a ceiling the rate does not have. The row -// is absent entirely when neither the reading nor the history carries a -// speed: an engine that reports none is not an engine generating at zero. -func renderSpeedBar(w io.Writer, speed *float64, history []metrics.HistorySample, lineW int) { - samples := speedSamples(history) - if speed == nil && len(samples) == 0 { +// speedText is the figure with its unit, unpadded so a row drawing the +// figure at a fixed starting column keeps the number's first digit there +// however many digits the rate has grown. +func speedText(v float64) string { + return fmt.Sprintf("%.2f t/s", v) +} + +// renderSpeedBar draws the Speed row of the bar-format surfaces: the label, +// the rate's sparkline in the bars' own column, and the t/s figure trailing +// where the other rows carry their percentages — or a blank sparkline where +// the window holds no point, since a gauge would imply a ceiling the rate +// does not have. The row is absent when the engine reports no speed at all. +func renderSpeedBar(w io.Writer, speed *float64, history []metrics.HistorySample, lineW, labelW int) { + if !speedShown(speed, history) { return } + samples := speedSamples(history) bar := strings.Repeat(" ", lineW) if len(samples) > 0 { bar = speedBlock(samples, lineW) } - fmt.Fprintf(w, " %-9s %6.2f t/s %s\n", "Speed", speedFigure(speed, samples), bar) + fmt.Fprintf(w, " %-*s %s %s\n", labelW, "Speed", bar, speedText(speedFigure(speed))) } // renderSpeedGauge draws the Speed row where no history is drawn — the gauge -// format and the gauge-mode tile: the label and the figure, no bar. -func renderSpeedGauge(w io.Writer, speed *float64) { - if speed == nil { +// format and the gauge-mode tile: the figure at the column the gauges start, +// and no bar the rate has no ceiling to fill against. The row joins whenever +// the engine reports speed at all, and a lapsed figure reads 0.00. +func renderSpeedGauge(w io.Writer, speed *float64, history []metrics.HistorySample, labelW int) { + if !speedShown(speed, history) { return } - fmt.Fprintf(w, " %-9s %6.2f t/s\n", "Speed", *speed) + fmt.Fprintf(w, " %-*s %s\n", labelW, "Speed", speedText(speedFigure(speed))) } // serveGaugeW and serveBarW are the two halves of the serve view's combined @@ -481,6 +532,7 @@ const ( // gauge half blank, so the lines align and the figure never draws a half's // own number twice. func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64, history []metrics.HistorySample) { + labelW := statLabelWidth(gpus, history) for _, s := range barSeriesList(cpu, mem, gpus, history) { gaugeHalf := strings.Repeat(" ", serveGaugeW) barHalf := strings.Repeat(" ", serveBarW) @@ -496,29 +548,35 @@ func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemorySt figure = last } } - fmt.Fprintf(w, " %-9s %s %3.0f%% %s\n", s.label, gaugeHalf, figure, barHalf) + fmt.Fprintf(w, " %-*s %s %3.0f%% %s\n", labelW, s.label, gaugeHalf, figure, barHalf) } - if samples := speedSamples(history); speed != nil || len(samples) > 0 { + if speedShown(speed, history) { + samples := speedSamples(history) barHalf := strings.Repeat(" ", serveBarW) if len(samples) > 0 { barHalf = speedBlock(samples, serveBarW) } - // The gauge half stays blank: the rate has no ceiling to fill - // against, so the row carries its figure and its trend only. - fmt.Fprintf(w, " %-9s %s %6.2f t/s %s\n", "Speed", strings.Repeat(" ", serveGaugeW), speedFigure(speed, samples), barHalf) + // The figure stands at the column the gauges start — the rate has no + // ceiling to fill a gauge against — inside a field spanning the other + // rows' gauge and percentage halves. The field is what keeps the + // sparkline starting where every other sparkline does as the figure + // grows digits, and a lapsed reading draws its 0.00 there. + fmt.Fprintf(w, " %-*s %-*s%s\n", labelW, "Speed", serveGaugeW+6, speedText(speedFigure(speed)), barHalf) } } // renderStatGauges draws the resource series in the gauge format: the current // reading only, whatever the history holds. The generation speed joins as a -// figure-only row when the reading carries one. -func renderStatGauges(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64) { +// figure-only row wherever the engine reports one — the history decides that +// even though the format draws none — and a lapsed reading draws 0.00. +func renderStatGauges(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, gpus []metrics.GpuStat, speed *float64, history []metrics.HistorySample) { + labelW := statLabelWidth(gpus, history) for _, s := range barSeriesList(cpu, mem, gpus, nil) { if s.current != nil { - renderGauge(w, s.label, *s.current, gaugeW) + renderGauge(w, s.label, *s.current, gaugeW, labelW) } } - renderSpeedGauge(w, speed) + renderSpeedGauge(w, speed, history, labelW) } // renderTokenLines draws the engine's token and request counters, the block @@ -540,13 +598,13 @@ func renderTokenLines(w io.Writer, tokens *metrics.TokenStats) { } // renderSpeedTable draws the table format's speed line, beside the token -// counters it shares the engine-block with. Absent, not zero, when the -// reading carries no rate. -func renderSpeedTable(w io.Writer, speed *float64) { - if speed == nil { +// counters it shares the engine-block with. A lapsed reading reads 0.00; the +// line is absent only for an engine that reports no rate at all. +func renderSpeedTable(w io.Writer, speed *float64, history []metrics.HistorySample) { + if !speedShown(speed, history) { return } - fmt.Fprintf(w, " speed: %.2f t/s\n", *speed) + fmt.Fprintf(w, " speed: %s\n", speedText(speedFigure(speed))) } // renderGPUTable draws the per-GPU lines of the table format, plus the diff --git a/cmd/spinloop/metrics_render_gpu_test.go b/cmd/spinloop/metrics_render_gpu_test.go index 47b38b2c..fe29d7d6 100644 --- a/cmd/spinloop/metrics_render_gpu_test.go +++ b/cmd/spinloop/metrics_render_gpu_test.go @@ -18,7 +18,7 @@ func TestUtilisationOnlyGPUDrawsNoMemorySeries(t *testing.T) { } var bar, gauge bytes.Buffer renderStatBars(&bar, nil, nil, macGPU, nil, history, barLineW) - renderStatGauges(&gauge, nil, nil, macGPU, nil) + renderStatGauges(&gauge, nil, nil, macGPU, nil, nil) for name, out := range map[string]string{"bar": bar.String(), "gauge": gauge.String()} { if !strings.Contains(out, "GPU") || !strings.Contains(out, " 56%") { t.Errorf("%s: missing GPU util: %q", name, out) @@ -35,7 +35,7 @@ func TestGPUWithMemoryTotalDrawsBothSeries(t *testing.T) { MemoryUsed: 8 << 30, MemoryTotal: 16 << 30, Temperature: 42}} var bar, gauge bytes.Buffer renderStatBars(&bar, nil, nil, gpus, nil, nil, barLineW) - renderStatGauges(&gauge, nil, nil, gpus, nil) + renderStatGauges(&gauge, nil, nil, gpus, nil, nil) for name, out := range map[string]string{"bar": bar.String(), "gauge": gauge.String()} { if !strings.Contains(out, "GPU") || !strings.Contains(out, "GPU mem") || !strings.Contains(out, " 50%") { t.Errorf("%s: %q", name, out) diff --git a/cmd/spinloop/metrics_render_test.go b/cmd/spinloop/metrics_render_test.go index 54b9d5b1..7b5d0fb2 100644 --- a/cmd/spinloop/metrics_render_test.go +++ b/cmd/spinloop/metrics_render_test.go @@ -2,6 +2,7 @@ package main import ( "bytes" + "fmt" "strings" "testing" @@ -77,11 +78,11 @@ func TestPoolMaxKeepsPeaks(t *testing.T) { func TestRenderSparkline(t *testing.T) { var b bytes.Buffer - renderSparkline(&b, "CPU", []float64{20, 30}, 40) + renderSparkline(&b, "CPU", []float64{20, 30}, 40, 7) // The trailing figure is the latest sample's percentage — the exact value // the last glyph approximates — not the series' first. Both points sit // below the 80% threshold, so neither takes a colour. - want := " CPU " + strings.Repeat(" ", 38) + "▂▃" + " 30%\n" + want := " CPU " + strings.Repeat(" ", 38) + "▂▃" + " 30%\n" if got := b.String(); got != want { t.Errorf("sparkline = %q, want %q", got, want) } @@ -93,8 +94,8 @@ func TestRenderSparkline(t *testing.T) { // they sit while the calm points around them do not. func TestRenderSparklineColoursEveryPoint(t *testing.T) { var b bytes.Buffer - renderSparkline(&b, "CPU", []float64{5, 95, 85, 30}, 40) - want := " CPU " + strings.Repeat(" ", 36) + + renderSparkline(&b, "CPU", []float64{5, 95, 85, 30}, 40, 7) + want := " CPU " + strings.Repeat(" ", 36) + "▁" + ansiRed + "▇" + ansiReset + ansiYellow + "▆" + ansiReset + "▃" + " 30%\n" @@ -119,7 +120,7 @@ func TestRenderSparklineColoursTheLastPointByItsValue(t *testing.T) { } for _, c := range cases { var b bytes.Buffer - renderSparkline(&b, "CPU", []float64{5, 95, c.last}, 40) + renderSparkline(&b, "CPU", []float64{5, 95, c.last}, 40, 7) if !strings.HasSuffix(b.String(), c.want) { t.Errorf("last point %v: %q, want suffix %q", c.last, b.String(), c.want) } @@ -128,15 +129,15 @@ func TestRenderSparklineColoursTheLastPointByItsValue(t *testing.T) { func TestRenderGauge(t *testing.T) { var b bytes.Buffer - renderGauge(&b, "CPU", 42, gaugeW) - want := " CPU " + ansiGreen + strings.Repeat("█", 10) + ansiReset + strings.Repeat("░", 15) + " 42%\n" + renderGauge(&b, "CPU", 42, gaugeW, 7) + want := " CPU " + ansiGreen + strings.Repeat("█", 10) + ansiReset + strings.Repeat("░", 15) + " 42%\n" if got := b.String(); got != want { t.Errorf("gauge = %q, want %q", got, want) } // A value beyond 100 fills the gauge rather than spilling past it. b.Reset() - renderGauge(&b, "CPU", 150, gaugeW) - want = " CPU " + ansiRed + strings.Repeat("█", 25) + ansiReset + " 150%\n" + renderGauge(&b, "CPU", 150, gaugeW, 7) + want = " CPU " + ansiRed + strings.Repeat("█", 25) + ansiReset + " 150%\n" if got := b.String(); got != want { t.Errorf("out-of-range gauge = %q, want %q", got, want) } @@ -163,7 +164,7 @@ func TestRenderStatBarsFallsBackToGaugeWithoutHistory(t *testing.T) { var got, want bytes.Buffer renderStatBars(&got, cpu, mem, gpus, nil, nil, barLineW) - renderStatGauges(&want, cpu, mem, gpus, nil) + renderStatGauges(&want, cpu, mem, gpus, nil, nil) if got.String() != want.String() { t.Errorf("no-history bar differs from the old drawing:\ngot:\n%q\nwant:\n%q", got.String(), want.String()) } @@ -258,7 +259,7 @@ func TestRenderStatBarsStoppedEngineDrawsHistoryAlone(t *testing.T) { if len(lines) != 3 { t.Fatalf("drew %d lines, want CPU, GPU and GPU mem: %q", len(lines), b.String()) } - if !strings.HasPrefix(lines[0], " CPU ") { + if !strings.HasPrefix(lines[0], " CPU ") { t.Errorf("first line: %q", lines[0]) } // One GPU: the plain "GPU" prefix, as the gauge used it. @@ -323,12 +324,12 @@ func TestRenderStatBarsGPUInOnlySomeSamples(t *testing.T) { func TestRenderStatGaugesIgnoresHistory(t *testing.T) { cpu := &metrics.CpuStat{Utilization: 50} var b bytes.Buffer - renderStatGauges(&b, cpu, nil, nil, nil) + renderStatGauges(&b, cpu, nil, nil, nil, nil) if !strings.Contains(b.String(), " 50%") || strings.Contains(b.String(), "▁") { t.Errorf("gauge drew history or the wrong value: %q", b.String()) } b.Reset() - renderStatGauges(&b, nil, nil, nil, nil) + renderStatGauges(&b, nil, nil, nil, nil, nil) if b.String() != "" { t.Errorf("a stopped engine drew gauges: %q", b.String()) } @@ -353,7 +354,7 @@ func TestRenderStatCombined(t *testing.T) { if len(lines) != 4 { t.Fatalf("drew %d lines, want 4: %q", len(lines), b.String()) } - for i, label := range []string{" CPU ", " RAM ", " GPU ", " GPU mem "} { + for i, label := range []string{" CPU ", " RAM ", " GPU ", " GPU mem "} { if !strings.HasPrefix(lines[i], label) { t.Errorf("line for %s: %q", label, lines[i]) } @@ -362,7 +363,7 @@ func TestRenderStatCombined(t *testing.T) { // the two halves, and it is the current reading, not the bar's last // sample. cpuBar, _ := sparklineBlock([]float64{10, 95}, serveBarW) - wantCPU := " CPU " + gaugeBlock(42, serveGaugeW) + " 42% " + cpuBar + wantCPU := " CPU " + gaugeBlock(42, serveGaugeW) + " 42% " + cpuBar if lines[0] != wantCPU { t.Errorf("CPU line:\ngot %q\nwant %q", lines[0], wantCPU) } @@ -385,8 +386,8 @@ func TestRenderStatCombined(t *testing.T) { // gauge half, a space, the figure's fixed-width field (4: three digits // right-aligned plus the percent sign), a space, bar half. for i, line := range lines { - if w := lipgloss.Width(line); w != 12+serveGaugeW+1+4+1+serveBarW { - t.Errorf("line %d is %d columns wide, want %d: %q", i, w, 12+serveGaugeW+1+4+1+serveBarW, line) + if w := lipgloss.Width(line); w != 10+serveGaugeW+1+4+1+serveBarW { + t.Errorf("line %d is %d columns wide, want %d: %q", i, w, 10+serveGaugeW+1+4+1+serveBarW, line) } } } @@ -414,12 +415,12 @@ func TestRenderStatCombinedFigureDoesNotShiftTheBar(t *testing.T) { if len(lines) != 1 { t.Fatalf("figure %s drew %d lines, want 1: %q", c.figure, len(lines), b.String()) } - want := " CPU " + gaugeBlock(c.value, serveGaugeW) + " " + c.figure + " " + cpuBar + want := " CPU " + gaugeBlock(c.value, serveGaugeW) + " " + c.figure + " " + cpuBar if lines[0] != want { t.Errorf("figure %s:\ngot %q\nwant %q", c.figure, lines[0], want) } - if w := lipgloss.Width(lines[0]); w != 12+serveGaugeW+1+4+1+serveBarW { - t.Errorf("figure %s drew a %d-column line, want %d: %q", c.figure, w, 12+serveGaugeW+1+4+1+serveBarW, lines[0]) + if w := lipgloss.Width(lines[0]); w != 10+serveGaugeW+1+4+1+serveBarW { + t.Errorf("figure %s drew a %d-column line, want %d: %q", c.figure, w, 10+serveGaugeW+1+4+1+serveBarW, lines[0]) } } } @@ -445,10 +446,10 @@ func TestRenderStatCombinedWithoutHistory(t *testing.T) { lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") blankBar := strings.Repeat(" ", serveBarW) want := []string{ - " CPU " + gaugeBlock(42, serveGaugeW) + " 42% " + blankBar, - " RAM " + gaugeBlock(30, serveGaugeW) + " 30% " + blankBar, - " GPU " + gaugeBlock(61, serveGaugeW) + " 61% " + blankBar, - " GPU mem " + gaugeBlock(50, serveGaugeW) + " 50% " + blankBar, + " CPU " + gaugeBlock(42, serveGaugeW) + " 42% " + blankBar, + " RAM " + gaugeBlock(30, serveGaugeW) + " 30% " + blankBar, + " GPU " + gaugeBlock(61, serveGaugeW) + " 61% " + blankBar, + " GPU mem " + gaugeBlock(50, serveGaugeW) + " 50% " + blankBar, } if len(lines) != len(want) { t.Fatalf("drew %d lines, want %d: %q", len(lines), len(want), b.String()) @@ -479,10 +480,10 @@ func TestRenderStatCombinedStoppedEngineDrawsHistoryAlone(t *testing.T) { } blankGauge := strings.Repeat(" ", serveGaugeW) wantPrefix := []string{ - " CPU " + blankGauge + " 20% ", - " RAM " + blankGauge + " 30% ", - " GPU " + blankGauge + " 60% ", - " GPU mem " + blankGauge + " 60% ", + " CPU " + blankGauge + " 20% ", + " RAM " + blankGauge + " 30% ", + " GPU " + blankGauge + " 60% ", + " GPU mem " + blankGauge + " 60% ", } for i, want := range wantPrefix { if !strings.HasPrefix(lines[i], want) { @@ -519,8 +520,8 @@ func TestFormatMetricsBarStoppedWithHistory(t *testing.T) { t.Fatal(err) } want := "prod stopped org/qwen:q4\n" + - " active 12s ago\n" + - " CPU " + strings.Repeat(" ", 38) + "▁▂" + " 20%\n" + " active 12s ago\n" + + " CPU " + strings.Repeat(" ", 38) + "▁▂" + " 20%\n" if got := b.String(); got != want { t.Errorf("stopped bar = %q, want %q", got, want) } @@ -532,7 +533,7 @@ func TestFormatMetricsBarStoppedWithHistory(t *testing.T) { t.Fatal(err) } want = "prod stopped org/qwen:q4\n" + - " active 12s ago\n" + " active 12s ago\n" if got := b.String(); got != want { t.Errorf("stopped gauge = %q, want %q", got, want) } @@ -560,7 +561,7 @@ func TestFormatMetricsBarRunning(t *testing.T) { if !strings.HasPrefix(got, "prod running g5.xlarge org/qwen:q4 0.4.3\n") { t.Errorf("header: %q", got) } - if !strings.Contains(got, " active 3s ago\n") { + if !strings.Contains(got, " active 3s ago\n") { t.Errorf("active line missing: %q", got) } // Every series drew a sparkline from the history: no gauge in the output. @@ -648,9 +649,10 @@ func nodeResultsFor(resp *cloud.StatsResponse) []fleet.NodeResult { }} } -// The Speed row: the window's high watermark sets the glyph heights, every -// point draws in the default colour, and no speed in the reading draws no -// row at all. +// The Speed row: the sparkline starts in the bars' column and the t/s +// figure trails where the percentages sit, the window's high watermark sets +// the glyph heights, every point draws in the default colour, and a figure +// that has lapsed reads zero while the window still remembers the rate. func TestRenderSpeedRow(t *testing.T) { speed := 8.5 history := []metrics.HistorySample{ @@ -658,39 +660,41 @@ func TestRenderSpeedRow(t *testing.T) { {Time: 2, Speed: ptrPct(5.0)}, {Time: 3, Speed: ptrPct(10.0)}, } + bar := strings.Repeat(" ", barLineW-3) + "▂▄▇" var b bytes.Buffer renderStatBars(&b, &metrics.CpuStat{Utilization: 95}, nil, nil, &speed, history, barLineW) lines := strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n") line := lines[len(lines)-1] - if !strings.HasPrefix(line, " Speed 8.50 t/s ") { - t.Fatalf("speed row: %q", line) + want := " Speed " + bar + " 8.50 t/s" + if line != want { + t.Fatalf("speed row:\ngot %q\nwant %q", line, want) + } + // The sparkline shares the CPU row's graph column: same label column, + // same draw width, so the glyphs start at the same character. + if got, cpu := line[10:20], lines[0][10:20]; len(got) != len(cpu) { + t.Errorf("graph columns differ: %q / %q", got, cpu) } // The window's fastest point takes the heaviest grade the sparkline // draws, its slowest one far lighter — and neither the full block. - tail := strings.TrimLeft(line[strings.Index(line, "t/s ")+4:], " ") - if !strings.HasSuffix(tail, "▇") { - t.Errorf("the window maximum did not draw the heaviest grade: %q", tail) - } - if !strings.Contains(tail, "▂") { - t.Errorf("the window minimum did not draw a far lighter grade: %q", tail) - } - // The speed row draws no full block of its own, and no threshold colour - // at its window maximum — while the 95% CPU row on the same screen is - // free to be coloured. - if strings.Contains(line, "█") { - t.Errorf("the speed row drew the full block: %q", line) + if !strings.HasSuffix(bar, "▇") || !strings.Contains(bar, "▂") { + t.Errorf("watermark mapping: %q", bar) } + // The speed row draws no threshold colour at its window maximum — while + // the 95% CPU row on the same screen is free to be coloured. if strings.Contains(line, "\x1b[") { t.Errorf("the speed row drew colour: %q", line) } + if !strings.Contains(lines[0], "\x1b[") { + t.Errorf("the CPU row should still colour: %q", lines[0]) + } - // No current figure but a retained history: the row falls back to the - // window's last point — the last rate a stopped engine had. + // A lapsed figure on an engine that reports speed: the row stays, the + // figure reads zero, and the trend remembers the rates it saw. b.Reset() renderStatBars(&b, nil, nil, nil, nil, history, barLineW) line = strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n")[0] - if !strings.HasPrefix(line, " Speed 10.00 t/s ") { - t.Errorf("a stopped engine's speed row: %q", line) + if want := " Speed " + bar + " 0.00 t/s"; line != want { + t.Errorf("a quiet engine's speed row:\ngot %q\nwant %q", line, want) } // Neither reading nor history: no row. @@ -701,30 +705,49 @@ func TestRenderSpeedRow(t *testing.T) { } } -// The gauge format and the serve view carry the figure without a bar the -// rate has no ceiling to fill against. +// The gauge format and the serve view carry the figure where the graphs +// start — the rate has no ceiling to fill a gauge against — and the serve +// view keeps its sparkline in the sparklines' column however many digits the +// figure grows. func TestRenderSpeedFigureOnly(t *testing.T) { speed := 12.34 var b bytes.Buffer - renderStatGauges(&b, nil, nil, nil, &speed) - line := strings.TrimSpace(b.String()) - if line != "Speed 12.34 t/s" { - t.Errorf("gauge speed row: %q", line) - } - if strings.Contains(line, "░") { - t.Errorf("the speed gauge half drew a bar: %q", line) + renderStatGauges(&b, nil, nil, nil, &speed, nil) + if got := b.String(); got != " Speed 12.34 t/s\n" { + t.Errorf("gauge speed row: %q", got) } + // Lapsed but reporting, the gauge-format row says 0.00 rather than + // vanishing under a watching eye. b.Reset() - renderStatCombined(&b, nil, nil, nil, &speed, nil) - line = strings.TrimSpace(b.String()) - if !strings.Contains(line, "Speed") || !strings.Contains(line, "12.34 t/s") { - t.Errorf("combined speed row: %q", line) + renderStatGauges(&b, nil, nil, nil, nil, []metrics.HistorySample{{Time: 1, Speed: ptrPct(4.0)}}) + if got := b.String(); got != " Speed 0.00 t/s\n" { + t.Errorf("lapsed gauge speed row: %q", got) + } + + // The combined row: figure at the graph column, sparkline at the + // sparkline column, pinned there across digit counts. + for _, c := range []struct { + value float64 + want string + }{ + {12.34, "12.34 t/s"}, + {5.00, "5.00 t/s"}, + {1234.56, "1234.56 t/s"}, + } { + v := c.value + b.Reset() + renderStatCombined(&b, nil, nil, nil, &v, nil) + line := strings.TrimSuffix(b.String(), "\n") + want := " Speed " + fmt.Sprintf("%-26s", c.want) + strings.Repeat(" ", serveBarW) + if line != want { + t.Errorf("combined speed row %v:\ngot %q\nwant %q", c.value, line, want) + } } - // No figure: neither surface mentions speed. + // No speed anywhere: neither surface mentions it. b.Reset() - renderStatGauges(&b, nil, nil, nil, nil) + renderStatGauges(&b, nil, nil, nil, nil, nil) if strings.Contains(b.String(), "Speed") { t.Errorf("no figure still drew a gauge row: %q", b.String()) } diff --git a/cmd/spinloop/serve_view.go b/cmd/spinloop/serve_view.go index 5a214019..8fb43d82 100644 --- a/cmd/spinloop/serve_view.go +++ b/cmd/spinloop/serve_view.go @@ -386,7 +386,7 @@ func (m serveView) metricsLines() []string { if line := dashTileServingLine(m.stats); line != "" { fmt.Fprintln(&b, line) } - renderActiveIndented(&b, m.stats.LastActiveAt, m.stats.IdleSeconds, m.stats.RetainUntil, dashNow()) + renderActiveIndented(&b, m.stats.LastActiveAt, m.stats.IdleSeconds, m.stats.RetainUntil, dashNow(), statLabelWidth(m.stats.GPUs, m.stats.History)) // A running reading carries its current figures; a settled one carries // its retained history, which survives a stop — the detail view's own // rule for when the series exist at all. diff --git a/docs/commands/cloud.md b/docs/commands/cloud.md index e1573444..da761945 100644 --- a/docs/commands/cloud.md +++ b/docs/commands/cloud.md @@ -258,11 +258,13 @@ table, and `--format=json` is the raw reply. A stopped engine keeps its readings: the sparkline runs to the stop, ending at it. An engine that reports its own generation rate (llama.cpp does) adds a -**Speed** row beside the resource series: the figure in tokens per second, -and in bar format a sparkline of the rate across the window, scaled to the -window's own high watermark and drawn in the default colour throughout — a -rate has no ceiling for the utilisation thresholds to cross. Where neither -the current reading nor the history carries a rate, there is no row. +**Speed** row in the block's shared columns: the figure in tokens per +second, and in bar format a sparkline of the rate across the window drawn +in the bars' own column, scaled to the window's own high watermark and in +the default colour throughout — a rate has no ceiling for the utilisation +thresholds to cross. Once the engine stops generating the row stays and the +figure reads `0.00 t/s`; only an engine that never reports a rate draws no +row. While a start is under way, `status` shows **`starting`** for the endpoint, whichever machine began the start, until the instance is running. Once it is diff --git a/docs/commands/serve.md b/docs/commands/serve.md index 28c96aaf..487d8e88 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -30,9 +30,11 @@ and a footer naming the keys the view answers to. resource series — CPU, RAM, each GPU's utilisation and memory, and the engine's generation speed where it reports one — with every series drawn in both formats at once, each on one line: a gauge of - the current reading beside the bar of its retained history. The speed row's - bar half stays blank — a rate has no ceiling to fill against — leaving its - t/s figure and its history sparkline. Below them, the + the current reading beside the bar of its retained history. The speed row + has no gauge — a rate has no ceiling to fill against — so its t/s figure + stands where the gauges start and its sparkline joins the other bars in + their column; between refreshes a lapsed figure reads 0.00 t/s rather than + vanishing. Below them, the token and request counters. The reading comes from the daemon the serve process runs in-process and refreshes on the dashboard's own local cadence; a reading the view could not renew is shown with its age. diff --git a/openspec/changes/engine-generation-speed/design.md b/openspec/changes/engine-generation-speed/design.md index cde798ca..5b0c4289 100644 --- a/openspec/changes/engine-generation-speed/design.md +++ b/openspec/changes/engine-generation-speed/design.md @@ -24,7 +24,9 @@ The daemon already captures every line the engine writes (stderr straight to the **One tracker, two lifetimes.** A `speedTrack` on the daemon (mutex, last value, last-observed time) answers two questions with different clocks: the sampler tick attaches a history point when a figure arrived since the previous tick (points are observations, never continuations — a quiet tick's sample just lacks the field); the metrics handler answers the current reading only while the engine runs AND the last figure is no older than the sampling interval (spec: "since the last sampling tick"). Both read through the daemon's injected clock, so tests age the figure without sleeping. `StartEngine` resets the tracker, the tick baseline, and the history together, so one engine's rate never rides the next engine's report. -**Speed is a `barSeries`, drawn by its own rule.** The series list gains a trailing `Speed` series built from the current figure and the history's speed points; `renderStatBars`/`renderStatCombined` draw it with a speed-specific glyph block — value scaled to the window maximum (pool-then-scale, so downsampling keeps the watermark honest) mapping to `▇`, never `█` — and always uncoloured, because the 80/90 thresholds only mean something on a 0–100 axis; a rate has no ceiling. The figure renders `%6.2f t/s`. `renderStatGauges` draws the label and figure with no gauge block. The combined serve-view row keeps its bar half and leaves the gauge half blank; the table format gets a `speed:` key-value line beside the token counters. Callers keep taking explicit args from `metrics.Stats` rather than restructuring signatures: the existing surfaces already thread `(cpu, mem, gpus, history)` and gain `(speed)`. +**Speed is a row that shares the block's columns.** The renderer draws it from the current figure and the history's speed points, never a `barSeries` — its drawing rules differ twice over: the sparkline scales values to the window's own maximum (pool-then-scale, so downsampling keeps the watermark honest) so the window's fastest point maps to `▇` and nothing to `█`, and it never colours — the 80/90 thresholds only mean something on a 0–100 axis. The figure is `%6.2f t/s` left at a fixed column: in the bar and table formats it trails where the other rows put their percentages; in the gauge formats and the combined view it stands where the graphs start (in the combined view inside a fixed field spanning the gauge and percentage halves, so its sparkline holds the sparklines' column as the digits grow). Where the current reading has lapsed the row still draws and the figure reads `0.00 t/s` — the row answers "how fast right now", and the answer for a quiet engine is zero; only an engine that never reports the figure has no row (the wire keeps its absent-not-zero contract; the zero is the display's reading of "no fresh figure on an engine that reports one"). `StartEngine` and the column plumbing: callers thread `(cpu, mem, gpus, speed, history)` and the shared label-column width derived from them. + +**The label column widths itself, not its history.** Every stat row formatted `%-9s` because "GPU util" was nine columns; the rename to "GPU" left two dead columns on the common single-GPU host. `statLabelWidth(gpus, history)` now sizes the block's column to its longest label — seven ("GPU mem"), widening by the GPU index for multi-GPU hosts — floored at seven so one fleet keeps one column and blocks of different hosts read as one table. `renderSparkline`, `renderGauge`, `renderActiveIndented` and the Speed rows take the width; `dashBarLineW` drops to 23 so a speed row's trailing "1234.56 t/s" still fits the 42-column tile. **GPU relabel is one string.** `barSeriesList` builds `"GPU %d util"` today; the util series becomes `"GPU"` for one GPU, `"GPU N"` for many, the memory series unchanged. Every surface draws from this list, so tiles, detail, bar, gauge and serve view move together. @@ -32,7 +34,7 @@ The daemon already captures every line the engine writes (stderr straight to the ## Risks / Trade-offs -- **Dashboard tiles are fixed-height** (`dashTileH`): a node with GPU series and a running engine may clip the token counter block when the Speed row joins. Accepted — the detail view is unclipped, and clipping keeps the tile grid honest; flag in review if the clip reads badly in practice. +- **Dashboard tiles are fixed-height** (`dashTileH`): a node with GPU series and a running engine may clip the token counter block when the Speed row joins. The row's width fits by `dashBarLineW`'s construction; the height is the loose end. Accepted — the detail view is unclipped, and clipping keeps the tile grid honest; flag in review if the clip reads badly in practice. - **Only engines that self-report are covered.** vLLM's `/metrics` exposes no instantaneous rate; its users see no row. Deriving speed from counter deltas is the fallback if demand appears. - **Tee adds a copy path on the engine's output.** The copy is one `Write` of already-buffered bytes to an in-memory line splitter; the pump never blocks on the parser (regex over ~100 bytes), and a wedged observer cannot back-pressure the engine because the tee writes the file first and the hook after. - **`--watch`/poll staleness:** between sampler ticks the current figure is up to one interval old — bounded staleness is already the documented model for every other metric in the reply. diff --git a/openspec/changes/engine-generation-speed/proposal.md b/openspec/changes/engine-generation-speed/proposal.md index 2053e62b..9996a3de 100644 --- a/openspec/changes/engine-generation-speed/proposal.md +++ b/openspec/changes/engine-generation-speed/proposal.md @@ -7,7 +7,7 @@ Operators watching a serving engine have no idea how fast it is actually generat ## What Changes - The daemon observes the generation speed (`tg`) in the engine's captured output lines, parses it, and reports it as a new optional metric: a current reading on the stats reply plus one point per sampling tick in the retained history. Engines that never print the figure report neither, on the usual "absent, not zero" terms. -- Every surface that renders the resource series gains a `Speed` row — `Speed 12.34 t/s ▂▃▂▂▂▁▂▂▁▂▇▇▃▂▂` — with a sparkline whose glyph heights scale to the high watermark of the values currently shown in the window, and whose points all draw in the terminal's default colour (the 80/90% threshold colours are utilisation's, not speed's). The gauge and table formats carry the figure without a sparkline; JSON carries both new optional fields. +- Every surface that renders the resource series gains a `Speed` row sharing their columns — `Speed ▁▂▁▁▁▁▂▂▁▂▇▇▃▂▂ 12.34 t/s` in the bar formats, the figure standing at the graph column in the gauge formats and the combined view — with a sparkline whose glyph heights scale to the high watermark of the values currently shown in the window, and whose points all draw in the terminal's default colour (the 80/90% threshold colours are utilisation's, not speed's). A figure that has lapsed on an engine that reports one reads `0.00 t/s`; JSON carries both new optional fields. - The GPU utilisation series is relabelled: `GPU util`/`GPU N util` become `GPU`/`GPU N` (the memory series keeps `GPU mem`/`GPU N mem`). ## Capabilities @@ -21,7 +21,7 @@ _None — all behaviour extends existing specs._ - `engine-metrics`: engine stats collection gains generation speed parsed from the engine's captured output, optional on the same terms as every other stat. - `engine-activity`: the sampling tick folds a fresh speed observation into the retained history; a tick with no generation since the last one carries no speed point. - `daemon-api`: `/v1/metrics` reports the current speed reading (while the engine runs) and speed points in the history, both optional. -- `remote-metrics-bar-format`: bar/gauge/table formats draw the `Speed` row (sparkline scaled to the window's high watermark, default colour throughout, figure in t/s); GPU utilisation series relabelled from `GPU util`/`GPU N util` to `GPU`/`GPU N`. +- `remote-metrics-bar-format`: bar/gauge/table formats draw the `Speed` row in the block's shared columns (sparkline scaled to the window's high watermark, default colour throughout, figure in t/s; a lapsed figure on an engine that reports one reads 0.00); GPU utilisation series relabelled from `GPU util`/`GPU N util` to `GPU`/`GPU N`, and the block's label column tightened to the longest label it draws. ## Impact diff --git a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md index 6c14342d..5abdede4 100644 --- a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md +++ b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md @@ -4,18 +4,35 @@ ### Requirement: Generation speed row -Where a reading carries a current generation speed or its retained history carries speed points, every surface that draws the resource series SHALL additionally draw a `Speed` row beneath them: a left-aligned label, the current figure right-aligned to two decimal places with a `t/s` suffix, and, where history is drawn, a sparkline of the retained speed points to the right of the figure. +Where a reading carries a current generation speed or its retained history carries speed points — an engine that reports the figure, whatever it is doing this second — every surface that draws the resource series SHALL additionally draw a `Speed` row beneath them. The figure is the current reading to two decimal places with a `t/s` suffix; where it has lapsed between refreshes, the row SHALL read `0.00 t/s` — a stopped or quiet engine is generating at zero, and a row that kept showing the last burst's rate would be answering a question the engine is no longer asking. The row is omitted entirely only for an engine that reports no speed at all. + +The row shall share its columns with the rows above it, not cut its own: the figure starts where the other rows' graphs start — in the gauge formats in place of the gauge the rate has no ceiling to fill, in the combined view inside a fixed field spanning the gauge and percentage halves so the digits' first character holds the graph column whatever their count — and where history is drawn the sparkline SHALL start in the same column as every other sparkline and draw at the same width. The speed sparkline SHALL scale its glyph heights against the highest value drawn in the window — the window's high watermark maps to the sparkline's heaviest drawn grade, and no value draws the full block — so the row reads relative to itself, a steady engine filling its row and a doubled burst standing out against it. Every point SHALL appear in the terminal's default colour whatever its height: the 80/90% thresholds belong to utilisation percentages, and a generation rate has no ceiling to cross. -Where speed exists but the surface draws no history — the gauge format, the table format, a gauge-mode dashboard tile — the row SHALL carry the figure alone. Where neither the current reading nor the history carries a speed, the row SHALL be omitted entirely. +Where speed exists but the surface draws no history — the gauge format, the table format, a gauge-mode dashboard tile — the row SHALL carry the figure alone, at the column the gauges start. #### Scenario: Speed row with its sparkline - **WHEN** bar format renders a reading whose history carries speed points -- **THEN** the output includes a `Speed` row whose figure is the current - figure in t/s and whose glyphs are the retained speed points across the - window +- **THEN** the output includes a `Speed` row whose glyphs are the retained + speed points starting in the same column as the other sparklines and + whose trailing figure is the current figure in t/s + +#### Scenario: The figure stands where the graphs start + +- **WHEN** the combined view renders a reading with a current generation + speed +- **THEN** the `Speed` row's figure starts at the column the gauges start, + and its sparkline at the column the other sparklines start, so a rate + growing from "5.06 t/s" to "23.45 t/s" keeps both columns fixed + +#### Scenario: A lapsed figure reads zero on an engine that reports speed + +- **WHEN** a watched engine has printed no timing figure for the last + sampling interval while its window still holds speed points +- **THEN** the `Speed` row remains drawn, its figure reads 0.00 t/s, and + its sparkline still shows the rates the window holds #### Scenario: The window's fastest point draws the heaviest glyph @@ -35,8 +52,8 @@ Where speed exists but the surface draws no history — the gauge format, the ta - **WHEN** the gauge format renders a running reading with a current speed and no retained speed points -- **THEN** the output includes a `Speed` row carrying the t/s figure with no - sparkline +- **THEN** the output includes a `Speed` row carrying the t/s figure at the + column the gauges start, with no sparkline #### Scenario: No speed, no row @@ -44,6 +61,31 @@ Where speed exists but the surface draws no history — the gauge format, the ta an engine that reports none, or a daemon that predates the figure - **THEN** the output draws no `Speed` row +### Requirement: Label column fits the labels + +The label column shared by the rows of a metrics block — the resource +series, the generation speed, and the active figure drawn beside them — +SHALL be as wide as the longest label the block draws and no wider: seven +columns on hosts with at most one GPU (the longest label there is +"GPU mem"), widening by the index the labels carry where the host names +more than one GPU. The column was sized for the retired "GPU util" label; +the block keeps one column per fleet rather than one per reading, so the +blocks of different hosts in one `fleet metrics` output read as one table. + +#### Scenario: One GPU sets the column + +- **WHEN** `spinloop metrics --format=bar` renders a host with a single GPU +- **THEN** the resource rows' graphs start two columns nearer the labels + than they did when the utilisation series was labelled "GPU util", and + every row of the block starts its figure in the same column + +#### Scenario: Many GPUs widen it + +- **WHEN** the same command renders a host whose readings name GPU 0 and + GPU 1 +- **THEN** the column widens for the "GPU 0 mem" label and every row of the + block, CPU included, still starts its graph in the same column + ## MODIFIED Requirements ### Requirement: Bar format output diff --git a/openspec/changes/engine-generation-speed/tasks.md b/openspec/changes/engine-generation-speed/tasks.md index 141293dc..e6ec31ee 100644 --- a/openspec/changes/engine-generation-speed/tasks.md +++ b/openspec/changes/engine-generation-speed/tasks.md @@ -27,3 +27,11 @@ ## Verify - [x] `gofmt -w`, `go vet ./...`, `go test ./... -cover` green, coverage ≥ 80% + +## Layout follow-up (from review of the rendered view) + +- [x] Tighten the block's shared label column to its longest label (`statLabelWidth`: "GPU mem" = 7, widening by the GPU index); thread the width through `renderSparkline`/`renderGauge`/`renderActiveIndented` and the Speed rows +- [x] Serve view: the speed figure stands at the graph column inside a fixed field spanning the gauge and percentage halves, pinning the sparkline to the sparklines' column across digit counts +- [x] Bar format: sparkline in the bars' column, t/s figure trailing with the other rows' percentages; gauge/tile rows carry the figure at the graph column +- [x] A lapsed figure on an engine that reports one draws `0.00 t/s` wherever the row shows (`speedShown`/`speedFigure`); only a never-reporting engine has no row +- [x] `dashBarLineW` 25→23 so the trailing t/s figure fits the 42-column tile; goldens and spec/design/docs updated From 983a2010311dfd4809902ab711147bbd0745f1de Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Wed, 28 Oct 2026 11:06:07 +0000 Subject: [PATCH 6/7] fix: let the speed sparkline slide back as the engine falls quiet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The figure drops to 0.00 t/s on a lapse but the sparkline pinned the last burst to its right edge, so the row contradicted itself: a packed speed series discarded the quiet samples' positions and the trend's "now" kept claiming the engine was still generating. Keep the speed points on the window's time axis instead: one slot per sample, quiet slots drawn blank, pooled without closing the gaps. Each speedless tick slides the burst a column toward the past and leaves its own blank at the row's right edge, agreeing with the figure's zero. A stopped engine's edge stays legitimately full — its history froze when the engine did. --- cmd/spinloop/metrics_render.go | 99 +++++++++++++------ cmd/spinloop/metrics_render_test.go | 20 +++- docs/commands/cloud.md | 5 +- docs/commands/serve.md | 3 +- .../changes/engine-generation-speed/design.md | 2 +- .../specs/remote-metrics-bar-format/spec.md | 5 +- .../changes/engine-generation-speed/tasks.md | 4 + 7 files changed, 99 insertions(+), 39 deletions(-) diff --git a/cmd/spinloop/metrics_render.go b/cmd/spinloop/metrics_render.go index 2aef0248..1f660758 100644 --- a/cmd/spinloop/metrics_render.go +++ b/cmd/spinloop/metrics_render.go @@ -414,15 +414,18 @@ func renderStatBars(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemoryStat, renderSpeedBar(w, speed, history, lineW, labelW) } -// speedSamples gathers the retained generation-rate points, oldest first. -// A series like any other: samples where the window has them, gaps where the -// engine was quiet, since a point is an observation and never a continuation. -func speedSamples(history []metrics.HistorySample) []float64 { - var out []float64 - for _, s := range history { - if s.Speed != nil { - out = append(out, *s.Speed) - } +// speedSeries gathers the retained generation-rate points, oldest first, one +// slot per history sample: the rate where the sample has one, a nil gap +// where the engine was quiet. Unlike a packed series, a slot sits at its own +// sample's place on the window's time axis — so when the engine stops +// generating, the burst slides left as the quiet ticks pile up behind it and +// the row's right edge goes blank. A packed series would keep the last burst +// flush against the edge as the current figure falls to zero, drawing the +// question's last answer as if it were still being given. +func speedSeries(history []metrics.HistorySample) []*float64 { + out := make([]*float64, len(history)) + for i, s := range history { + out[i] = s.Speed } return out } @@ -433,7 +436,43 @@ func speedSamples(history []metrics.HistorySample) []float64 { // no row — the engine that reports no speed is not an engine generating at // zero, but an engine whose row would be a question it cannot answer. func speedShown(speed *float64, history []metrics.HistorySample) bool { - return speed != nil || len(speedSamples(history)) > 0 + if speed != nil { + return true + } + for _, s := range history { + if s.Speed != nil { + return true + } + } + return false +} + +// poolMaxGaps reduces a gap-carrying series to at most width columns, one +// per column, each the maximum of the non-nil samples pooled into it — and +// nil where every pooled sample was a gap. Pooling keeps the newest sample +// in the final column the way poolMax does; a quiet column stays quiet and +// draws blank rather than borrowing its neighbour's rate. +func poolMaxGaps(values []*float64, width int) []*float64 { + if len(values) <= width { + return values + } + out := make([]*float64, width) + per := float64(len(values)) / float64(width) + for c := range out { + lo := int(float64(c) * per) + hi := int(float64(c+1) * per) + if c == width-1 { + hi = len(values) + } + var m *float64 + for i := lo; i < hi; i++ { + if values[i] != nil && (m == nil || *values[i] > *m) { + m = values[i] + } + } + out[c] = m + } + return out } // speedBlock draws the speed sparkline. Its glyph heights are relative to @@ -443,20 +482,26 @@ func speedShown(speed *float64, history []metrics.HistorySample) bool { // doubles the row's height beside its own baseline. Every point draws in the // terminal's default colour whatever its height: the 80/90 thresholds belong // to utilisation percentages, and a generation rate has no ceiling to cross. -func speedBlock(samples []float64, width int) string { - pooled := poolMax(samples, width) +// A nil sample — a quiet tick — draws blank, so a lapse shows as the row's +// right edge emptying beside its 0.00 figure. +func speedBlock(samples []*float64, width int) string { + pooled := poolMaxGaps(samples, width) peak := 0.0 for _, v := range pooled { - if v > peak { - peak = v + if v != nil && *v > peak { + peak = *v } } var b strings.Builder b.WriteString(strings.Repeat(" ", width-len(pooled))) for _, v := range pooled { + if v == nil { + b.WriteRune(' ') + continue + } pct := 0.0 if peak > 0 { - pct = v / peak * 100 + pct = *v / peak * 100 } b.WriteRune(barGlyph(pct)) } @@ -484,19 +529,15 @@ func speedText(v float64) string { // renderSpeedBar draws the Speed row of the bar-format surfaces: the label, // the rate's sparkline in the bars' own column, and the t/s figure trailing -// where the other rows carry their percentages — or a blank sparkline where -// the window holds no point, since a gauge would imply a ceiling the rate -// does not have. The row is absent when the engine reports no speed at all. +// where the other rows carry their percentages, since a gauge would imply a +// ceiling the rate does not have. Quiet samples leave the sparkline blank at +// their own columns, so a lapse empties the row's right edge. The row is +// absent when the engine reports no speed at all. func renderSpeedBar(w io.Writer, speed *float64, history []metrics.HistorySample, lineW, labelW int) { if !speedShown(speed, history) { return } - samples := speedSamples(history) - bar := strings.Repeat(" ", lineW) - if len(samples) > 0 { - bar = speedBlock(samples, lineW) - } - fmt.Fprintf(w, " %-*s %s %s\n", labelW, "Speed", bar, speedText(speedFigure(speed))) + fmt.Fprintf(w, " %-*s %s %s\n", labelW, "Speed", speedBlock(speedSeries(history), lineW), speedText(speedFigure(speed))) } // renderSpeedGauge draws the Speed row where no history is drawn — the gauge @@ -551,17 +592,13 @@ func renderStatCombined(w io.Writer, cpu *metrics.CpuStat, mem *metrics.MemorySt fmt.Fprintf(w, " %-*s %s %3.0f%% %s\n", labelW, s.label, gaugeHalf, figure, barHalf) } if speedShown(speed, history) { - samples := speedSamples(history) - barHalf := strings.Repeat(" ", serveBarW) - if len(samples) > 0 { - barHalf = speedBlock(samples, serveBarW) - } // The figure stands at the column the gauges start — the rate has no // ceiling to fill a gauge against — inside a field spanning the other // rows' gauge and percentage halves. The field is what keeps the // sparkline starting where every other sparkline does as the figure - // grows digits, and a lapsed reading draws its 0.00 there. - fmt.Fprintf(w, " %-*s %-*s%s\n", labelW, "Speed", serveGaugeW+6, speedText(speedFigure(speed)), barHalf) + // grows digits, and a lapsed reading draws its 0.00 there while the + // sparkline's own right edge empties tick by tick beside it. + fmt.Fprintf(w, " %-*s %-*s%s\n", labelW, "Speed", serveGaugeW+6, speedText(speedFigure(speed)), speedBlock(speedSeries(history), serveBarW)) } } diff --git a/cmd/spinloop/metrics_render_test.go b/cmd/spinloop/metrics_render_test.go index 7b5d0fb2..daa3f6ee 100644 --- a/cmd/spinloop/metrics_render_test.go +++ b/cmd/spinloop/metrics_render_test.go @@ -688,13 +688,27 @@ func TestRenderSpeedRow(t *testing.T) { t.Errorf("the CPU row should still colour: %q", lines[0]) } - // A lapsed figure on an engine that reports speed: the row stays, the - // figure reads zero, and the trend remembers the rates it saw. + // A stopped engine: its history froze at the stop, so the retained + // burst still sits at the right edge and the figure reads zero — the + // row tells what the engine was doing until it stopped. b.Reset() renderStatBars(&b, nil, nil, nil, nil, history, barLineW) line = strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n")[0] if want := " Speed " + bar + " 0.00 t/s"; line != want { - t.Errorf("a quiet engine's speed row:\ngot %q\nwant %q", line, want) + t.Errorf("a stopped engine's speed row:\ngot %q\nwant %q", line, want) + } + + // The lapse in the making: the engine still runs, so quiet ticks keep + // landing in the window as speedless samples. Each one slides the + // burst a column left and leaves its own blank at the right edge, so + // the sparkline's now says the same thing as the figure's zero. + b.Reset() + quiet := append(append([]metrics.HistorySample{}, history...), + metrics.HistorySample{Time: 4}, metrics.HistorySample{Time: 5}) + renderStatBars(&b, nil, nil, nil, nil, quiet, barLineW) + line = strings.Split(strings.TrimSuffix(b.String(), "\n"), "\n")[0] + if want := " Speed " + strings.Repeat(" ", barLineW-5) + "▂▄▇ " + " 0.00 t/s"; line != want { + t.Errorf("an engine gone quiet's speed row:\ngot %q\nwant %q", line, want) } // Neither reading nor history: no row. diff --git a/docs/commands/cloud.md b/docs/commands/cloud.md index da761945..bcadec12 100644 --- a/docs/commands/cloud.md +++ b/docs/commands/cloud.md @@ -263,8 +263,9 @@ second, and in bar format a sparkline of the rate across the window drawn in the bars' own column, scaled to the window's own high watermark and in the default colour throughout — a rate has no ceiling for the utilisation thresholds to cross. Once the engine stops generating the row stays and the -figure reads `0.00 t/s`; only an engine that never reports a rate draws no -row. +figure reads `0.00 t/s`, and the sparkline's right edge empties tick by +tick as the last burst slides back into the window's past; only an engine +that never reports a rate draws no row. While a start is under way, `status` shows **`starting`** for the endpoint, whichever machine began the start, until the instance is running. Once it is diff --git a/docs/commands/serve.md b/docs/commands/serve.md index 487d8e88..8b0f5de2 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -33,7 +33,8 @@ and a footer naming the keys the view answers to. the current reading beside the bar of its retained history. The speed row has no gauge — a rate has no ceiling to fill against — so its t/s figure stands where the gauges start and its sparkline joins the other bars in - their column; between refreshes a lapsed figure reads 0.00 t/s rather than + their column; between refreshes a lapsed figure reads 0.00 t/s and its + sparkline empties from the right as the engine falls quiet, rather than vanishing. Below them, the token and request counters. The reading comes from the daemon the serve process runs in-process and refreshes on the dashboard's own local cadence; diff --git a/openspec/changes/engine-generation-speed/design.md b/openspec/changes/engine-generation-speed/design.md index 5b0c4289..fd593932 100644 --- a/openspec/changes/engine-generation-speed/design.md +++ b/openspec/changes/engine-generation-speed/design.md @@ -24,7 +24,7 @@ The daemon already captures every line the engine writes (stderr straight to the **One tracker, two lifetimes.** A `speedTrack` on the daemon (mutex, last value, last-observed time) answers two questions with different clocks: the sampler tick attaches a history point when a figure arrived since the previous tick (points are observations, never continuations — a quiet tick's sample just lacks the field); the metrics handler answers the current reading only while the engine runs AND the last figure is no older than the sampling interval (spec: "since the last sampling tick"). Both read through the daemon's injected clock, so tests age the figure without sleeping. `StartEngine` resets the tracker, the tick baseline, and the history together, so one engine's rate never rides the next engine's report. -**Speed is a row that shares the block's columns.** The renderer draws it from the current figure and the history's speed points, never a `barSeries` — its drawing rules differ twice over: the sparkline scales values to the window's own maximum (pool-then-scale, so downsampling keeps the watermark honest) so the window's fastest point maps to `▇` and nothing to `█`, and it never colours — the 80/90 thresholds only mean something on a 0–100 axis. The figure is `%6.2f t/s` left at a fixed column: in the bar and table formats it trails where the other rows put their percentages; in the gauge formats and the combined view it stands where the graphs start (in the combined view inside a fixed field spanning the gauge and percentage halves, so its sparkline holds the sparklines' column as the digits grow). Where the current reading has lapsed the row still draws and the figure reads `0.00 t/s` — the row answers "how fast right now", and the answer for a quiet engine is zero; only an engine that never reports the figure has no row (the wire keeps its absent-not-zero contract; the zero is the display's reading of "no fresh figure on an engine that reports one"). `StartEngine` and the column plumbing: callers thread `(cpu, mem, gpus, speed, history)` and the shared label-column width derived from them. +**Speed is a row that shares the block's columns.** The renderer draws it from the current figure and the history's speed points, never a `barSeries` — its drawing rules differ twice over: the sparkline scales values to the window's own maximum (pool-then-scale, so downsampling keeps the watermark honest) so the window's fastest point maps to `▇` and nothing to `█`, and it never colours — the 80/90 thresholds only mean something on a 0–100 axis. The figure is `%6.2f t/s` left at a fixed column: in the bar and table formats it trails where the other rows put their percentages; in the gauge formats and the combined view it stands where the graphs start (in the combined view inside a fixed field spanning the gauge and percentage halves, so its sparkline holds the sparklines' column as the digits grow). Where the current reading has lapsed the row still draws and the figure reads `0.00 t/s` — the row answers "how fast right now", and the answer for a quiet engine is zero; only an engine that never reports the figure has no row (the wire keeps its absent-not-zero contract; the zero is the display's reading of "no fresh figure on an engine that reports one"). The sparkline agrees with the figure by keeping its points on the window's time axis: the daemon's sampling continues while an engine runs, quiet ticks landing speedless, so `speedSeries` keeps one slot per sample and `poolMaxGaps` pools them without closing the gaps — the burst slides left and the right edge blanks as the engine falls quiet, instead of the packed series' trick of pinning the last rate to the edge forever (a stopped engine's frozen edge is the honest case: the history stopped with the engine). `StartEngine` and the column plumbing: callers thread `(cpu, mem, gpus, speed, history)` and the shared label-column width derived from them. **The label column widths itself, not its history.** Every stat row formatted `%-9s` because "GPU util" was nine columns; the rename to "GPU" left two dead columns on the common single-GPU host. `statLabelWidth(gpus, history)` now sizes the block's column to its longest label — seven ("GPU mem"), widening by the GPU index for multi-GPU hosts — floored at seven so one fleet keeps one column and blocks of different hosts read as one table. `renderSparkline`, `renderGauge`, `renderActiveIndented` and the Speed rows take the width; `dashBarLineW` drops to 23 so a speed row's trailing "1234.56 t/s" still fits the 42-column tile. diff --git a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md index 5abdede4..dcda249e 100644 --- a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md +++ b/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md @@ -6,6 +6,8 @@ Where a reading carries a current generation speed or its retained history carries speed points — an engine that reports the figure, whatever it is doing this second — every surface that draws the resource series SHALL additionally draw a `Speed` row beneath them. The figure is the current reading to two decimal places with a `t/s` suffix; where it has lapsed between refreshes, the row SHALL read `0.00 t/s` — a stopped or quiet engine is generating at zero, and a row that kept showing the last burst's rate would be answering a question the engine is no longer asking. The row is omitted entirely only for an engine that reports no speed at all. +The speed sparkline SHALL place each point at its own sample's place on the window's time axis, and a sample without a rate SHALL draw blank: as an engine falls quiet, its sampling continues, and each quiet tick slides the last burst one column toward the past and leaves its own blank at the row's right edge — so the trend's now agrees with the figure's zero instead of holding the last burst against the edge as if it were still being generated. (A stopped engine's history froze at the stop, so its burst legitimately rests at the edge of a row whose figure reads zero: the row tells what the engine was doing until it stopped.) + The row shall share its columns with the rows above it, not cut its own: the figure starts where the other rows' graphs start — in the gauge formats in place of the gauge the rate has no ceiling to fill, in the combined view inside a fixed field spanning the gauge and percentage halves so the digits' first character holds the graph column whatever their count — and where history is drawn the sparkline SHALL start in the same column as every other sparkline and draw at the same width. The speed sparkline SHALL scale its glyph heights against the highest value drawn in the window — the window's high watermark maps to the sparkline's heaviest drawn grade, and no value draws the full block — so the row reads relative to itself, a steady engine filling its row and a doubled burst standing out against it. Every point SHALL appear in the terminal's default colour whatever its height: the 80/90% thresholds belong to utilisation percentages, and a generation rate has no ceiling to cross. @@ -32,7 +34,8 @@ Where speed exists but the surface draws no history — the gauge format, the ta - **WHEN** a watched engine has printed no timing figure for the last sampling interval while its window still holds speed points - **THEN** the `Speed` row remains drawn, its figure reads 0.00 t/s, and - its sparkline still shows the rates the window holds + the sparkline draws the quiet interval blank at its right edge, the + retained burst slid that far toward the past #### Scenario: The window's fastest point draws the heaviest glyph diff --git a/openspec/changes/engine-generation-speed/tasks.md b/openspec/changes/engine-generation-speed/tasks.md index e6ec31ee..e4d676a8 100644 --- a/openspec/changes/engine-generation-speed/tasks.md +++ b/openspec/changes/engine-generation-speed/tasks.md @@ -35,3 +35,7 @@ - [x] Bar format: sparkline in the bars' column, t/s figure trailing with the other rows' percentages; gauge/tile rows carry the figure at the graph column - [x] A lapsed figure on an engine that reports one draws `0.00 t/s` wherever the row shows (`speedShown`/`speedFigure`); only a never-reporting engine has no row - [x] `dashBarLineW` 25→23 so the trailing t/s figure fits the 42-column tile; goldens and spec/design/docs updated + +## Sparkline lapse follow-up (from review) + +- [x] Place speed points on the window's time axis (`speedSeries` one slot per sample, `poolMaxGaps` pooling without closing gaps) so a lapse empties the sparkline's right edge beside its `0.00 t/s` figure instead of pinning the last burst to it From 3adf08e07bd0eb0a2b5768929a600242e3582903 Mon Sep 17 00:00:00 2001 From: Pete Cornish Date: Wed, 28 Oct 2026 11:29:53 +0000 Subject: [PATCH 7/7] docs(openspec): archive the engine generation speed change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the deltas into the main specs — engine-metrics, engine-activity, daemon-api and remote-metrics-bar-format — and move the change to the archive. --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/daemon-api/spec.md | 0 .../specs/engine-activity/spec.md | 0 .../specs/engine-metrics/spec.md | 0 .../specs/remote-metrics-bar-format/spec.md | 0 .../tasks.md | 0 openspec/specs/daemon-api/spec.md | 35 +++++++ openspec/specs/engine-activity/spec.md | 42 ++++++++ openspec/specs/engine-metrics/spec.md | 21 ++++ .../specs/remote-metrics-bar-format/spec.md | 99 +++++++++++++++++-- 12 files changed, 191 insertions(+), 6 deletions(-) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/.openspec.yaml (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/design.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/proposal.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/specs/daemon-api/spec.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/specs/engine-activity/spec.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/specs/engine-metrics/spec.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/specs/remote-metrics-bar-format/spec.md (100%) rename openspec/changes/{engine-generation-speed => archive/2026-10-28-engine-generation-speed}/tasks.md (100%) diff --git a/openspec/changes/engine-generation-speed/.openspec.yaml b/openspec/changes/archive/2026-10-28-engine-generation-speed/.openspec.yaml similarity index 100% rename from openspec/changes/engine-generation-speed/.openspec.yaml rename to openspec/changes/archive/2026-10-28-engine-generation-speed/.openspec.yaml diff --git a/openspec/changes/engine-generation-speed/design.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/design.md similarity index 100% rename from openspec/changes/engine-generation-speed/design.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/design.md diff --git a/openspec/changes/engine-generation-speed/proposal.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/proposal.md similarity index 100% rename from openspec/changes/engine-generation-speed/proposal.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/proposal.md diff --git a/openspec/changes/engine-generation-speed/specs/daemon-api/spec.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/specs/daemon-api/spec.md similarity index 100% rename from openspec/changes/engine-generation-speed/specs/daemon-api/spec.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/specs/daemon-api/spec.md diff --git a/openspec/changes/engine-generation-speed/specs/engine-activity/spec.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/specs/engine-activity/spec.md similarity index 100% rename from openspec/changes/engine-generation-speed/specs/engine-activity/spec.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/specs/engine-activity/spec.md diff --git a/openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/specs/engine-metrics/spec.md similarity index 100% rename from openspec/changes/engine-generation-speed/specs/engine-metrics/spec.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/specs/engine-metrics/spec.md diff --git a/openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/specs/remote-metrics-bar-format/spec.md similarity index 100% rename from openspec/changes/engine-generation-speed/specs/remote-metrics-bar-format/spec.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/specs/remote-metrics-bar-format/spec.md diff --git a/openspec/changes/engine-generation-speed/tasks.md b/openspec/changes/archive/2026-10-28-engine-generation-speed/tasks.md similarity index 100% rename from openspec/changes/engine-generation-speed/tasks.md rename to openspec/changes/archive/2026-10-28-engine-generation-speed/tasks.md diff --git a/openspec/specs/daemon-api/spec.md b/openspec/specs/daemon-api/spec.md index 78f14819..a77d6d30 100644 --- a/openspec/specs/daemon-api/spec.md +++ b/openspec/specs/daemon-api/spec.md @@ -553,3 +553,38 @@ request, the reading is the only place a broken source is reported from. - **WHEN** an engine is stopped and a new engine is started, and a metrics request is made before the new engine's first tick - **THEN** the response carries no host figures from the previous engine + +### Requirement: Metrics reports generation speed + +The metrics endpoint SHALL report the engine's last parsed generation speed, in tokens per second, alongside the token counters while the engine is running and its output has carried a timing figure since the last sampling tick. The reading SHALL be absent while the engine is stopped, and absent while a running engine has generated nothing since the last tick, rather than reported as zero or as its last known value: the figure answers "how fast is it generating right now", and a stopped or quiet engine has no answer. + +The history entries SHALL additionally carry, per entry, an optional speed point for a tick whose interval saw a timing figure, on the same retention terms as the entry's other figures — surviving a stop and cleared by the next engine's start. + +Both the current reading and the history points are optional on the wire: an engine that reports no speed, and a daemon that predates the figure, simply omit them, and a consumer treats their absence as "not reported", never as zero. + +#### Scenario: A generating engine reports its speed + +- **WHEN** a metrics request is made against a running engine that has emitted + timing figures +- **THEN** the response carries the latest parsed tokens-per-second figure + +#### Scenario: A stopped engine reports no current speed but keeps the history + +- **WHEN** a metrics request is made after an engine that had been generating + is stopped +- **THEN** the response carries no current speed figure, while the history + entries taken during generation still carry their speed points + +#### Scenario: A quiet engine lets the figure lapse + +- **WHEN** a metrics request is made against a running engine whose last + timing figure predates the last sampling tick +- **THEN** the response omits the current speed figure, the token counters + remaining as they are + +#### Scenario: An engine or daemon without the figure omits it + +- **WHEN** a metrics request is made against an engine that emits no timing + figure, or a daemon that predates the figure +- **THEN** the response carries no speed field and no history entry carries a + speed point, and every other figure is reported as before diff --git a/openspec/specs/engine-activity/spec.md b/openspec/specs/engine-activity/spec.md index adc8cd00..29dbaab1 100644 --- a/openspec/specs/engine-activity/spec.md +++ b/openspec/specs/engine-activity/spec.md @@ -162,3 +162,45 @@ source is reported from. - **WHEN** the engine is stopped - **THEN** no further system readings are taken, and the readings taken before the stop remain retained until the next engine starts + +### Requirement: Generation speed rides the sampling tick + +While an engine is running, the sampler SHALL record a generation-speed point +in the retained history on each tick whose interval saw the engine emit a new +speed figure, carrying the most recent figure of that interval. A tick whose +interval saw no new figure SHALL carry no speed point: silence means the +engine generated nothing since the last tick, not that it generated at zero, +and a retained speed point is an observation, never a continuation. + +A speed point SHALL make a tick's sample exist even where the system reading +that tick yielded no host figure at all — a generation rate is an observation +like any other. The speed points retained SHALL follow the history's existing +rules: they cover at most the last 10 minutes at the sampler's cadence, they +survive the engine's stop, and the next engine's start clears them with the +rest of the history. + +#### Scenario: Steady generation puts a point on every tick + +- **WHEN** an engine has been generating for several sampler ticks +- **THEN** the retained history carries a speed point for each of those ticks, + each the latest figure its interval saw + +#### Scenario: A quiet tick carries no speed point + +- **WHEN** an engine is running but has emitted no timing line for the last + tick's interval +- **THEN** that tick's history sample carries no speed point, while its host + figures, if any, are still retained + +#### Scenario: Speed alone still makes a sample + +- **WHEN** a tick yields a new speed figure while the system reading yields no + host figure at all +- **THEN** the tick's sample is retained, carrying the speed point + +#### Scenario: Speed points survive a stop + +- **WHEN** an engine that had been generating is stopped, and the history is + read +- **THEN** the speed points taken before the stop are still retained, until a + later engine starts and clears the history diff --git a/openspec/specs/engine-metrics/spec.md b/openspec/specs/engine-metrics/spec.md index 7a296a95..ca084e7f 100644 --- a/openspec/specs/engine-metrics/spec.md +++ b/openspec/specs/engine-metrics/spec.md @@ -223,3 +223,24 @@ not a zero — true of the rendered output as it is of the collected one. - **THEN** the line shows the GPU's name and utilization with no `mem=` and no `temp=` segment, and a GPU reading that does carry them shows them as before + +### Requirement: Generation speed from engine output + +The system SHALL parse a generation speed — tokens per second — from the engine's own captured output lines. An engine that reports the figure in its structured log, llama.cpp's `print_timing` lines carrying `tg = 3.87 t/s` today, yields it; the figure read SHALL be the instantaneous one the line names for the current rate, and a line's other rate figures — a trailing three-second average, prompt-evaluation speed — SHALL not be read as it. + +The speed is an optional stat on the same terms as every other: an engine whose metrics carry no such figure and whose output carries no timing line, and an engine run where its output is not captured, report the speed absent rather than reported as zero. + +#### Scenario: A timing line yields the speed + +- **WHEN** a captured engine line carries `tg = 12.34 t/s` +- **THEN** the collected reading carries a generation speed of 12.34 tokens per second + +#### Scenario: An average is not the current rate + +- **WHEN** a captured line carries only a `tg_3s = 5.29 t/s` figure, or a prompt-evaluation rate, with no instantaneous `tg` figure +- **THEN** no speed is yielded from that line + +#### Scenario: An engine without timing lines reports none + +- **WHEN** metrics are collected while the running engine's output carries no timing line +- **THEN** the result omits the generation speed rather than reporting zero diff --git a/openspec/specs/remote-metrics-bar-format/spec.md b/openspec/specs/remote-metrics-bar-format/spec.md index 74306d19..99f56b60 100644 --- a/openspec/specs/remote-metrics-bar-format/spec.md +++ b/openspec/specs/remote-metrics-bar-format/spec.md @@ -8,7 +8,7 @@ Define the bar graph output format for `spinloop cloud metrics` with colour-code ### Requirement: Bar format output -The system SHALL support a `--format=bar` option that renders each resource series as a sparkline drawn from the history the on-instance daemon retains: a left-aligned label, one glyph per sample, and the latest value as a right-aligned percentage. The glyphs SHALL be Unicode block elements of one grade per utilisation level, so a series reads as a line of bars across the window. The series drawn SHALL be the same set the gauge format draws: CPU, RAM, each GPU's utilisation, and each GPU's memory where that GPU's current reading or retained history reports a memory total — a GPU that never reports one draws no memory series, with the same per-GPU labelling throughout. +The system SHALL support a `--format=bar` option that renders each resource series as a sparkline drawn from the history the on-instance daemon retains: a left-aligned label, one glyph per sample, and the latest value as a right-aligned percentage. The glyphs SHALL be Unicode block elements of one grade per utilisation level, so a series reads as a line of bars across the window. The series drawn SHALL be the same set the gauge format draws: CPU, RAM, each GPU's utilisation — labelled "GPU", or "GPU N" where the host has more than one — and each GPU's memory where that GPU's current reading or retained history reports a memory total — labelled "GPU mem"/"GPU N mem" — a GPU that never reports one drawing no memory series, with the same per-GPU labelling throughout. #### Scenario: Bar format displays CPU utilization @@ -23,12 +23,12 @@ The system SHALL support a `--format=bar` option that renders each resource seri #### Scenario: Bar format displays GPU utilization - **WHEN** the user runs `spinloop cloud metrics --format=bar` with a running instance that has GPU data whose readings carry a memory total -- **THEN** the output includes rows labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs), each drawn from the retained history +- **THEN** the output includes rows labelled "GPU" and "GPU mem" (or "GPU N"/"GPU N mem" for multiple GPUs), each drawn from the retained history #### Scenario: Bar format omits a GPU memory series with no total - **WHEN** the user runs `spinloop cloud metrics --format=bar` with GPU data whose current reading and retained history report no memory total -- **THEN** the output includes "GPU util" and no "GPU mem" row +- **THEN** the output includes "GPU" and no "GPU mem" row #### Scenario: Bar format header line @@ -122,7 +122,7 @@ than shown empty or zeroed. ### Requirement: Gauge format -The system SHALL support a `--format=gauge` option that renders each resource series as a horizontal progress gauge: a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. The gauge draws the current reading only — it carries no history. The series drawn SHALL be CPU, RAM, each GPU's utilisation, and each GPU's memory where the current reading reports a memory total — a GPU that reports none draws no memory gauge — with the same labels the bar format uses. The gauge fill SHALL be colour-coded on the bar format's thresholds: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%, with the colour reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. +The system SHALL support a `--format=gauge` option that renders each resource series as a horizontal progress gauge: a left-aligned label, a filled portion using block characters, an unfilled portion using light shade characters, and a right-aligned percentage value. The gauge draws the current reading only — it carries no history. The series drawn SHALL be CPU, RAM, each GPU's utilisation — labelled "GPU", or "GPU N" where the host has more than one — and each GPU's memory where the current reading reports a memory total — labelled "GPU mem"/"GPU N mem" — a GPU that reports none drawing no memory gauge, with the same labels the bar format uses. The gauge fill SHALL be colour-coded on the bar format's thresholds: green for values at or below 80%, yellow for values from 80% to 90%, and red for values above 90%, with the colour reset after the filled portion so the unfilled characters and percentage appear in the terminal's default colour. #### Scenario: Gauge format displays CPU utilization @@ -132,12 +132,12 @@ The system SHALL support a `--format=gauge` option that renders each resource se #### Scenario: Gauge format displays GPU utilisation - **WHEN** the user runs `spinloop cloud metrics --format=gauge` with a running instance that has GPU data whose readings carry a memory total -- **THEN** the output includes gauges labelled "GPU util" and "GPU mem" (or "GPU N util"/"GPU N mem" for multiple GPUs) +- **THEN** the output includes gauges labelled "GPU" and "GPU mem" (or "GPU N"/"GPU N mem" for multiple GPUs) #### Scenario: Gauge omits a GPU memory gauge with no total - **WHEN** the user runs `spinloop cloud metrics --format=gauge` with a GPU whose current reading reports no memory total -- **THEN** the output includes "GPU util" and no "GPU mem" gauge +- **THEN** the output includes "GPU" and no "GPU mem" gauge #### Scenario: Gauge colours the fill @@ -193,3 +193,90 @@ information. - **WHEN** two adjacent series are both at their maximum across the window - **THEN** each row's tallest glyph leaves the top eighth of its cell unfilled, so a blank strip separates the rows rather than the rows reading as one solid block + +### Requirement: Generation speed row + +Where a reading carries a current generation speed or its retained history carries speed points — an engine that reports the figure, whatever it is doing this second — every surface that draws the resource series SHALL additionally draw a `Speed` row beneath them. The figure is the current reading to two decimal places with a `t/s` suffix; where it has lapsed between refreshes, the row SHALL read `0.00 t/s` — a stopped or quiet engine is generating at zero, and a row that kept showing the last burst's rate would be answering a question the engine is no longer asking. The row is omitted entirely only for an engine that reports no speed at all. + +The speed sparkline SHALL place each point at its own sample's place on the window's time axis, and a sample without a rate SHALL draw blank: as an engine falls quiet, its sampling continues, and each quiet tick slides the last burst one column toward the past and leaves its own blank at the row's right edge — so the trend's now agrees with the figure's zero instead of holding the last burst against the edge as if it were still being generated. (A stopped engine's history froze at the stop, so its burst legitimately rests at the edge of a row whose figure reads zero: the row tells what the engine was doing until it stopped.) + +The row shall share its columns with the rows above it, not cut its own: the figure starts where the other rows' graphs start — in the gauge formats in place of the gauge the rate has no ceiling to fill, in the combined view inside a fixed field spanning the gauge and percentage halves so the digits' first character holds the graph column whatever their count — and where history is drawn the sparkline SHALL start in the same column as every other sparkline and draw at the same width. + +The speed sparkline SHALL scale its glyph heights against the highest value drawn in the window — the window's high watermark maps to the sparkline's heaviest drawn grade, and no value draws the full block — so the row reads relative to itself, a steady engine filling its row and a doubled burst standing out against it. Every point SHALL appear in the terminal's default colour whatever its height: the 80/90% thresholds belong to utilisation percentages, and a generation rate has no ceiling to cross. + +Where speed exists but the surface draws no history — the gauge format, the table format, a gauge-mode dashboard tile — the row SHALL carry the figure alone, at the column the gauges start. + +#### Scenario: Speed row with its sparkline + +- **WHEN** bar format renders a reading whose history carries speed points +- **THEN** the output includes a `Speed` row whose glyphs are the retained + speed points starting in the same column as the other sparklines and + whose trailing figure is the current figure in t/s + +#### Scenario: The figure stands where the graphs start + +- **WHEN** the combined view renders a reading with a current generation + speed +- **THEN** the `Speed` row's figure starts at the column the gauges start, + and its sparkline at the column the other sparklines start, so a rate + growing from "5.06 t/s" to "23.45 t/s" keeps both columns fixed + +#### Scenario: A lapsed figure reads zero on an engine that reports speed + +- **WHEN** a watched engine has printed no timing figure for the last + sampling interval while its window still holds speed points +- **THEN** the `Speed` row remains drawn, its figure reads 0.00 t/s, and + the sparkline draws the quiet interval blank at its right edge, the + retained burst slid that far toward the past + +#### Scenario: The window's fastest point draws the heaviest glyph + +- **WHEN** a speed history's window holds values from 2.0 to 10.0 t/s +- **THEN** the 10.0 point draws the heaviest grade the sparkline draws, the + 2.0 point draws one far lighter, and no glyph in any row of the output is + the full block + +#### Scenario: Speed points take no threshold colour + +- **WHEN** a speed sparkline holds a point far above the 80/90% thresholds' + territory at the window's maximum +- **THEN** that glyph appears in the terminal's default colour, and so does + every other speed glyph + +#### Scenario: A figure with no history still draws + +- **WHEN** the gauge format renders a running reading with a current speed and + no retained speed points +- **THEN** the output includes a `Speed` row carrying the t/s figure at the + column the gauges start, with no sparkline + +#### Scenario: No speed, no row + +- **WHEN** a reading's current figures and retained history carry no speed — + an engine that reports none, or a daemon that predates the figure +- **THEN** the output draws no `Speed` row + +### Requirement: Label column fits the labels + +The label column shared by the rows of a metrics block — the resource +series, the generation speed, and the active figure drawn beside them — +SHALL be as wide as the longest label the block draws and no wider: seven +columns on hosts with at most one GPU (the longest label there is +"GPU mem"), widening by the index the labels carry where the host names +more than one GPU. The column was sized for the retired "GPU util" label; +the block keeps one column per fleet rather than one per reading, so the +blocks of different hosts in one `fleet metrics` output read as one table. + +#### Scenario: One GPU sets the column + +- **WHEN** `spinloop metrics --format=bar` renders a host with a single GPU +- **THEN** the resource rows' graphs start two columns nearer the labels + than they did when the utilisation series was labelled "GPU util", and + every row of the block starts its figure in the same column + +#### Scenario: Many GPUs widen it + +- **WHEN** the same command renders a host whose readings name GPU 0 and + GPU 1 +- **THEN** the column widens for the "GPU 0 mem" label and every row of the + block, CPU included, still starts its graph in the same column