Visualize dstack metrics as sparklines

[peterschmidt85]

Before

dstack metrics printed a line of raw numbers per GPU, repeated on every poll under --watch:

gpu=0 mem=70.65GB/80GB util=0%
gpu=1 mem=70.65GB/80GB util=0%
... (six more)

Eight GPUs meant eight near-identical lines, and none of them showed whether a number was steady, climbing, or a momentary spike. The shape of a run was only visible by sitting and watching the command.

After

        UTILIZATION                                          MEMORY
cpu     ▂▂▃▂▂▂▃▂▂▂▂▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂ 34% of 208             ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅ 589GB/1480GB

gpu=0   ▆▆▇▆▆▆▇▆▁▁▁▆▆▇▆▆▆▇▆▆▆▇▆▆▆▇▆▆▆ 88%                    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 74GB/80GB
gpu=1   ▆▇▆▆▆▇▆▆▁▁▁▆▇▆▆▆▇▆▆▆▇▆▆▆▇▆▆▆▇ 86%                    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 74GB/80GB

        4 Aug 10:15 ───────────────── now                    4 Aug 10:15 ───────────────── now

Each series is drawn over the metrics the server retains — about an hour for a running job.

Design notes

Height is the peak, colour is the mean. A cell covers a few minutes of samples. Using the mean for height would let a card that touched 100% for ten seconds disappear entirely; using the peak alone can't distinguish that card from one that held 100% for the whole bucket. Both are drawn, so both questions are answerable.

Colour is never the only carrier. The latest value is printed beside every sparkline, and the fixed 0–100% scale means glyph height always means the same thing as the number next to it — no autoscaling, so a row at 19% of capacity is not drawn nearly full. The number itself is left unstyled: colouring it just repeats what the sparkline already says.

Glyphs stop at ▇, not █. A full block fills its cell to the top edge, so a column of them in consecutive rows fuses into a single mass and the rows stop reading as separate series.

Device rows come from the offer rather than from the metrics, so the shape of a run is visible before any samples arrive. The sparklines take whatever width the terminal has (10–80 cells, clamped), and fall back to ASCII glyphs where the console encoding can't carry block characters.

Behaviour change

Like dstack logs, this now shows a single job: --replica and --job select one, both defaulting to 0. Previously every job of a service was printed at once, so comparing replicas now means one invocation each.

Why api/server/_metrics.py changes too

The metrics endpoint takes a limit on how many samples to return, declared as limit: int = 1 rather than Optional[int]. The Python client had no limit parameter at all, so every request it made fell through to that default of 1.

That was correct until now — the old output only ever printed the latest reading, so one sample was exactly what it needed. Sparklines need the series, so the client has to start asking for it. Without this the command still renders, but on a single sample: one glyph, however long the run has been up, which looks plausible rather than broken.

The client now passes limit (plus after and before, which the endpoint already accepts), and the CLI requests 1000 samples — the same number the UI's metrics page asks for. That is a sample count rather than a time window, so it outruns the hour of history a running job retains and a young run is never under-filled.

Backward compatibility

Output format changes, and only one job is printed. Anything parsing dstack metrics output will break — it was a NAME STATUS CPU MEMORY GPU table and is now charts. Comparing replicas means one invocation per replica. --json support for scripted use will follow in a separate PR.

No CLI flags removed. The command took only -w/--watch; --replica and --job are additive and default to 0.

Older servers are fine. limit has been on the endpoint since 0.18.44, and unknown query params are ignored rather than rejected, so the worst case against an older server is a single sample rather than an error.

MetricsAPIClient.get_job_metrics stays source-compatible. after, before and limit are appended after the existing parameters and default to None, and are left out of the request when unset — an existing call sends the same request it sent before.

🤖 Generated with Claude Code