Visualize dstack metrics as sparklines

mkdocs/blog/posts/agentic-orchestration.md

@@ -250,8 +250,12 @@ $ dstack event --within-run train-qwen
 ```shell
 $ dstack metrics train-qwen
 
- NAME        STATUS   CPU  MEMORY       GPU
- train-qwen  running  92%  118GB/200GB  gpu=0 mem=71GB/80GB util=97%
+        UTILIZATION                            MEMORY
+ cpu    ▅▄▄▆▆▆▆▆▆▆▆▆▆▆▆▆▆▅▆▆▆▆▆▆▆▆▆ 91% of 32  ▃▃▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ 116GB/200GB
+
+ gpu=0  ▁▂▃▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 92%        ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 70GB/80GB
+
+        4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now            4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now
 ```
 
 </div>

mkdocs/blog/posts/dstack-metrics.md

@@ -16,7 +16,23 @@ While it's possible to use third-party monitoring tools with `dstack`, it is oft
 track metrics out of the box. That's why, with the latest release, `dstack` introduced [`dstack stats`](../../docs/reference/cli/dstack/metrics.md), a new CLI (and API)
 for monitoring container metrics, including GPU usage for `NVIDIA`, `AMD`, and other accelerators.
 
-<img src="https://dstack.ai/static-assets/static-assets/images/dstack-stats-v2.png" width="725"/>
+<div class="termy">
+
+```shell
+$ dstack metrics llama-70b-sft
+
+        UTILIZATION                            MEMORY
+ cpu    ▅▄▄▄▃▃▃▃▃▃▃▃▃▃▃▃▅▅▄▂▃▃▃▃▃▃▃ 39% of 64  ▃▃▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ 297GB/480GB
+
+ gpu=0  ▁▂▃▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 89%        ▄▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 67GB/80GB
+ gpu=1  ▁▂▆▆▆▅▆▆▆▆▆▆▆▆▆▆▁▁▅▆▆▅▆▆▆▆▆ 84%        ▄▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 67GB/80GB
+ gpu=2  ▁▂▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆▆ 87%        ▄▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 67GB/80GB
+ gpu=3  ▂▃▆▅▅▅▅▅▅▆▅▅▆▆▆▆▁▅▅▅▅▅▆▅▅▅▅ 82%        ▄▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 67GB/80GB
+
+        4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now            4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now
+```
+
+</div>
 
 <!-- more -->
 
@@ -31,6 +47,9 @@ difference is that `dstack stats` includes GPU VRAM usage and GPU utilization pe
 Similar to `kubectl top`, if a run consists of multiple jobs (such as distributed training or an auto-scalable service),
 `dstack stats` will display metrics per job.
 
+> Note, `dstack metrics` now shows one job at a time, like `dstack logs`. Use `--replica` and `--job` to
+> choose it; both default to `0`.
+
 !!! info "HTTP API"
     In addition to the `dstack stats` CLI commands, metrics can be obtained via the
     [`/api/project/{project_name}/metrics/job/{run_name}`](../../docs/reference/http/metrics.md) HTTP endpoint.

mkdocs/docs/concepts/metrics.md

@@ -18,27 +18,36 @@ This tab displays key CPU, memory, and GPU metrics collected during the last hou
 
 ## CLI
 
-As an alternative to the UI, you can track real-time essential metrics via the CLI.
-The `dstack metrics` command displays the most recently tracked CPU, memory, and GPU metrics.
+As an alternative to the UI, you can track essential metrics via the CLI.
+The `dstack metrics` command charts CPU, memory, and GPU utilization over the last hour of the
+job, with the latest value beside each chart.
 
 <div class="termy">
 
 ```shell
 dstack metrics gentle-mayfly-1
 
- NAME             STATUS  CPU  MEMORY          GPU
- gentle-mayfly-1  done    0%   16.27GB/2000GB  gpu=0 mem=72.48GB/80GB util=0%
-                                               gpu=1 mem=64.99GB/80GB util=0%
-                                               gpu=2 mem=580MB/80GB util=0%
-                                               gpu=3 mem=4MB/80GB util=0%
-                                               gpu=4 mem=4MB/80GB util=0%
-                                               gpu=5 mem=4MB/80GB util=0%
-                                               gpu=6 mem=4MB/80GB util=0%
-                                               gpu=7 mem=292MB/80GB util=0%
+        UTILIZATION                             MEMORY
+ cpu    ▅▄▄▄▃▃▃▃▃▃▃▃▃▃▃▃▅▅▄▃▃▃▃▃▃▃▃ 41% of 128  ▃▃▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ 581GB/960GB
+
+ gpu=0  ▁▂▃▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 89%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=1  ▁▂▆▆▆▅▆▆▆▆▆▆▆▆▆▆▁▁▅▆▆▅▆▆▆▆▆ 84%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=2  ▁▂▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆▆ 87%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=3  ▂▃▆▅▅▅▅▅▅▆▅▅▆▆▆▆▁▅▅▅▅▅▆▅▅▅▅ 82%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=4  ▂▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▁▆▆▆▆▆▆▆▆▆▆ 90%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=5  ▂▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆▆▆ 85%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=6  ▃▆▆▅▅▆▆▆▆▆▆▆▆▆▆▁▅▅▅▅▆▆▆▅▅▅▅ 83%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+ gpu=7  ▃▆▆▆▆▆▆▆▆▆▆▆▆▆▁▁▆▆▆▆▆▆▆▆▆▆▆ 88%         ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+
+        4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now             4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now
 ```
 
 </div>
 
+Like [`dstack logs`](../reference/cli/dstack/logs.md), the command shows a single job.
+Use `--replica` and `--job` to select one; both default to `0`.
+Pass `-w` to keep the charts updating.
+
 ## Prometheus
 
 To enable exporting metrics to Prometheus, set the

mkdocs/docs/guides/migration/slurm.md

@@ -1475,8 +1475,13 @@ Check real-time metrics:
 
 ```shell
 $ dstack metrics training-job
-  NAME             STATUS  CPU  MEMORY          GPU
-  training-job     running 45%  16.27GB/200GB   gpu=0 mem=72.48GB/80GB util=95%
+
+        UTILIZATION                            MEMORY
+ cpu    ▅▄▄▄▃▃▃▃▃▃▃▃▃▃▃▃▅▅▄▃▃▃▃▃▃▃▃ 45% of 32  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 16GB/200GB
+
+ gpu=0  ▁▂▃▆▆▆▆▆▆▆▆▆▆▆▆▆▆▁▆▆▆▆▆▆▆▆▆ 90%        ▄▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ 71GB/80GB
+
+        4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now            4 Aug 14:10 ┄┄┄┄┄┄┄┄┄┄┄ now
 ```
 
 </div>

src/dstack/_internal/cli/commands/metrics.py

@@ -1,21 +1,22 @@
 import argparse
 import time
-from typing import Any, List, Optional
 
 from rich.live import Live
-from rich.table import Table
 
 from dstack._internal.cli.commands import APIBaseCommand
 from dstack._internal.cli.services.completion import RunNameCompleter
 from dstack._internal.cli.utils.common import (
-    LIVE_TABLE_PROVISION_INTERVAL_SECS,
     LIVE_TABLE_REFRESH_RATE_PER_SEC,
-    add_row_from_dict,
     console,
 )
+from dstack._internal.cli.utils.metrics import (
+    MAX_SAMPLES,
+    WATCH_INTERVAL_SECONDS,
+    get_metrics_table,
+)
 from dstack._internal.core.errors import CLIError
-from dstack._internal.core.models.instances import Resources
 from dstack._internal.core.models.metrics import JobMetrics
+from dstack._internal.core.models.runs import Job
 from dstack.api._public import Client
 from dstack.api._public.runs import Run
 
@@ -33,121 +34,61 @@ def _register(self):
             help="Watch run metrics in realtime",
             action="store_true",
         )
+        self._parser.add_argument(
+            "--replica",
+            help="The replica number. Defaults to 0.",
+            type=int,
+            default=0,
+        )
+        self._parser.add_argument(
+            "--job",
+            help="The job number inside the replica. Defaults to 0.",
+            type=int,
+            default=0,
+        )
 
     def _command(self, args: argparse.Namespace):
         super()._command(args)
-        run = self.api.runs.get(run_name=args.run_name)
-        if run is None:
-            raise CLIError(f"Run {args.run_name} not found")
-        metrics = _get_run_jobs_metrics(api=self.api, run=run)
+        job, metrics = self._fetch(args)
 
         if not args.watch:
-            console.print(_get_metrics_table(run, metrics))
+            console.print(get_metrics_table(job, metrics))
             return
 
         try:
             with Live(console=console, refresh_per_second=LIVE_TABLE_REFRESH_RATE_PER_SEC) as live:
                 while True:
-                    live.update(_get_metrics_table(run, metrics))
-                    time.sleep(LIVE_TABLE_PROVISION_INTERVAL_SECS)
-                    run = self.api.runs.get(run_name=args.run_name)
-                    if run is None:
-                        raise CLIError(f"Run {args.run_name} not found")
-                    metrics = _get_run_jobs_metrics(api=self.api, run=run)
+                    live.update(get_metrics_table(job, metrics))
+                    time.sleep(WATCH_INTERVAL_SECONDS)
+                    job, metrics = self._fetch(args)
         except KeyboardInterrupt:
             pass
 
+    def _fetch(self, args: argparse.Namespace) -> tuple[Job, JobMetrics]:
+        run = self.api.runs.get(run_name=args.run_name)
+        if run is None:
+            raise CLIError(f"Run {args.run_name} not found")
+        job = _get_job(run, args.replica, args.job)
+        return job, _get_job_metrics(self.api, run, job)
 
-def _get_run_jobs_metrics(api: Client, run: Run) -> List[JobMetrics]:
-    metrics = []
-    for job in run._run.jobs:
-        job_metrics = api.client.metrics.get_job_metrics(
-            project_name=api.project,
-            run_name=run.name,
-            replica_num=job.job_spec.replica_num,
-            job_num=job.job_spec.job_num,
-        )
-        metrics.append(job_metrics)
-    return metrics
-
-
-def _get_metrics_table(run: Run, metrics: List[JobMetrics]) -> Table:
-    table = Table(box=None)
-    table.add_column("NAME", style="bold", no_wrap=True)
-    table.add_column("STATUS")
-    table.add_column("CPU")
-    table.add_column("MEMORY")
-    table.add_column("GPU")
-
-    run_row = {"NAME": run.name, "STATUS": run.status.value}
-    if len(run._run.jobs) != 1:
-        add_row_from_dict(table, run_row)
-
-    for job, job_metrics in zip(run._run.jobs, metrics):
-        jrd = job.job_submissions[-1].job_runtime_data
-        jpd = job.job_submissions[-1].job_provisioning_data
-        resources: Optional[Resources] = None
-        if jrd is not None and jrd.offer is not None:
-            resources = jrd.offer.instance.resources
-        elif jpd is not None:
-            resources = jpd.instance_type.resources
-        cpu_usage = _get_metric_value(job_metrics, "cpu_usage_percent")
-        if cpu_usage is not None:
-            if resources is not None:
-                cpu_usage = cpu_usage / resources.cpus
-            cpu_usage = f"{cpu_usage:.0f}%"
-        memory_usage = _get_metric_value(job_metrics, "memory_working_set_bytes")
-        if memory_usage is not None:
-            memory_usage = _format_memory(memory_usage, 2)
-            if resources is not None:
-                memory_usage += f"/{_format_memory(resources.memory_mib * 1024 * 1024, 2)}"
-        gpu_metrics = ""
-        gpus_detected_num = _get_metric_value(job_metrics, "gpus_detected_num")
-        if gpus_detected_num is not None:
-            for i in range(gpus_detected_num):
-                gpu_memory_usage = _get_metric_value(job_metrics, f"gpu_memory_usage_bytes_gpu{i}")
-                gpu_util_percent = _get_metric_value(job_metrics, f"gpu_util_percent_gpu{i}")
-                if gpu_memory_usage is not None:
-                    if i != 0:
-                        gpu_metrics += "\n"
-                    gpu_metrics += f"gpu={i} mem={_format_memory(gpu_memory_usage, 2)}"
-                    if resources is not None:
-                        gpu_metrics += (
-                            f"/{_format_memory(resources.gpus[i].memory_mib * 1024 * 1024, 2)}"
-                        )
-                    gpu_metrics += f" util={gpu_util_percent}%"
-
-        job_row = {
-            "NAME": f"  replica={job.job_spec.replica_num} job={job.job_spec.job_num}",
-            "STATUS": job.job_submissions[-1].status.value,
-            "CPU": cpu_usage or "-",
-            "MEMORY": memory_usage or "-",
-            "GPU": gpu_metrics or "-",
-        }
-        if len(run._run.jobs) == 1:
-            job_row.update(run_row)
-        add_row_from_dict(table, job_row)
-
-    return table
-
-
-def _get_metric_value(job_metrics: JobMetrics, name: str) -> Optional[Any]:
-    for metric in job_metrics.metrics:
-        if metric.name == name:
-            return metric.values[-1]
-    return None
-
-
-def _format_memory(memory_bytes: int, decimal_places: int) -> str:
-    """See test_format_memory in tests/_internal/cli/commands/test_metrics.py for examples."""
-    memory_mb = memory_bytes / 1024 / 1024
-    if memory_mb >= 1024:
-        value = memory_mb / 1024
-        unit = "GB"
-    else:
-        value = memory_mb
-        unit = "MB"
 
-    if decimal_places == 0:
-        return f"{round(value)}{unit}"
-    return f"{value:.{decimal_places}f}".rstrip("0").rstrip(".") + unit
+def _get_job(run: Run, replica_num: int, job_num: int) -> Job:
+    for job in run._run.jobs:
+        if job.job_spec.replica_num == replica_num and job.job_spec.job_num == job_num:
+            return job
+    raise CLIError(
+        f"Run {run.name} has no replica={replica_num} job={job_num}."
+        " Use --replica and --job to select one."
+    )
+
+
+def _get_job_metrics(api: Client, run: Run, job: Job) -> JobMetrics:
+    """`limit` must be sent explicitly: the endpoint declares it `limit: int = 1`, not
+    Optional, so omitting it caps the response at one sample."""
+    return api.client.metrics.get_job_metrics(
+        project_name=api.project,
+        run_name=run.name,
+        replica_num=job.job_spec.replica_num,
+        job_num=job.job_spec.job_num,
+        limit=MAX_SAMPLES,
+    )