feat: fleet storage tiers and model placement view

CHANGELOG.md

@@ -7,6 +7,15 @@ Format: version sections are listed newest first.
 
 ---
 
+## [Unreleased]
+
+### Added
+- **Fleet model storage tiers** — a dedicated **Storage** view (fleet-wide) plus per-Spark storage cards and a model-placement table
+- **Tier classification** — each mounted disk is labeled **Hot** (root NVMe), **Warm** (other local disks), or **Cold** (NAS `/mnt/modelshelf` and `cifs`/`smb`/`nfs` mounts), driven by `tierPaths` overrides, filesystem type, and mount prefix
+- **Model inventory** — scans configured model directories (`.safetensors` / `.gguf` / `.bin` / `.pt` / `.pth` / `.ckpt` weight files) per tier, local and over SSH, with a new `models` metrics domain (`POLL_INTERVAL_MODELS`, default 30 s). Each scanned dir is inspected **one level deep**, so `modelDirs` should point at the directory whose direct children are models (e.g. the outer family dir for HF-style nests)
+- **Dual placement** — a model is shown as **resident** on the Sparks holding a local copy, and as served **over the CX7/ConnectX fabric** on a peer that has it loaded via its LLM probe without a local copy
+- Optional per-Spark `modelDirs` / `tierPaths` in `config/sparks.json` to override which directories map to which tier
+
 ## [1.6.0] — 2026-08-07
 
 ### Added

README.md

@@ -79,6 +79,7 @@ Full history: [CHANGELOG.md](./CHANGELOG.md)
 | **Secrets** | SSH passwords AES-256-GCM encrypted; never in `sparks.json` or API responses |
 | **Docker-first** | Single privileged container for host metrics; prod and dev Compose files |
 | **Hot config** | Add / edit / remove / reorder Sparks from the UI with no process restart |
+| **Storage tiers** | Fleet **Storage** view — disks classified **Hot / Warm / Cold** (NAS/nfs/cifs vs local), model inventory, and resident-vs-CX7-fabric placement |
 
 ---
 
@@ -292,6 +293,7 @@ Copy `.env.example` to `.env` if needed:
 | `POLL_INTERVAL_LLM` | `2000` | LLM probe poll (ms) |
 | `POLL_INTERVAL_BANDWIDTH` | `2000` | Memory bandwidth / dmon poll (ms) |
 | `POLL_INTERVAL_LIVENESS` | `5000` | Online/SSH liveness check (ms) |
+| `POLL_INTERVAL_MODELS` | `30000` | Model inventory / tier scan (ms) |
 | `SPARKDASH_SECRETS_KEY` | _(auto)_ | Passphrase or 64-char hex for secret encryption |
 | `HOST_PROC_PATH` | `/host/proc` | Host proc mount inside container |
 | `HOST_SYS_PATH` | `/host/sys` | Host sys mount |
@@ -387,6 +389,14 @@ Each configured LLM port gets its own `LlmProbe` instance running in parallel. P
 
 Rates are derived from per-probe cumulative counter diffs (or SGLang sticky throughput while it moves). Multiple ports can be added or removed at runtime without restarting the monitor.
 
+### Storage tiers & model placement
+
+Each mounted disk is assigned a tier — **Hot** (root NVMe), **Warm** (other local disks), or **Cold** (NAS: `/mnt/modelshelf`, `/media`, `/Volumes`, `/mnt`, or any `cifs`/`smb`/`nfs` mount). Per-Spark `tierPaths` in `config/sparks.json` override the heuristic per mount. The collector scans the configured model directories (grouped by tier via `modelDirs`) for weight files (`.safetensors`, `.gguf`, `.bin`, `.pt`, `.pth`, `.ckpt`) and reports each model by name, size, and tier.
+
+The scan looks **one directory level deep**: each immediate child of a `modelDirs` entry counts as a model if it is a loose weight file or a subdirectory whose immediate children include a weight file. Point `modelDirs` at the directory whose *direct* children are your models. HF-hub-style nesting (`models--org--name/snapshots/<ref>/*.safetensors`, or `~/models/hf/<family>/*.safetensors`) is two levels deep, so use the outer family dir (e.g. `~/models/hf`), not the parent, as the `modelDirs` root.
+
+A model is **resident** on the Sparks holding a local copy. Because a peer with the model loaded in its LLM probe (matched by name) serves it to the fleet over the CX7/ConnectX fabric, such a Spark is shown as placing the model **over the fabric** even with no local copy. The fleet **Storage** view rolls these up per tier and lists every model with its resident and fabric placement.
+
 ---
 
 ## Contributing

package.json

@@ -9,7 +9,7 @@
     "dev:client": "vite",
     "build": "vite build",
     "typecheck": "tsc --noEmit",
-    "test": "node --test server/collectors/__tests__/*.test.js server/sparks/__tests__/*.test.js",
+    "test": "node --experimental-test-module-mocks --test server/collectors/__tests__/*.test.js server/sparks/__tests__/*.test.js",
     "preview": "vite preview",
     "start": "node server/index.js",
     "docker:up": "docker compose up -d",

server/collectors/SystemCollector.js

@@ -1,9 +1,72 @@
 import fs from "fs";
 import path from "path";
-import { HOST_PATHS, GPU_MEMORY_JSON_PATH, DGX_SPARK, HARDWARE_DEFAULTS } from "../config.js";
+import {
+  HOST_PATHS,
+  GPU_MEMORY_JSON_PATH,
+  DGX_SPARK,
+  HARDWARE_DEFAULTS,
+  TIER_DEFAULTS,
+  WEIGHT_EXTENSIONS,
+} from "../config.js";
 import { normalizeMac, WOL_INTERFACE } from "../wol.js";
 import { sshExec } from "./ssh.js";
 
+/**
+ * Classify a storage device into a tier: "hot" | "warm" | "cold".
+ * Pure (no `this`) so it is unit-testable in isolation.
+ *
+ * @param {object} params
+ * @param {string} params.device  lsblk/df device name (e.g. "nvme0n1p2", "sdb1")
+ * @param {string} params.mount   mount path (host-style, e.g. "/", "/mnt/modelshelf")
+ * @param {string} [params.fstype] filesystem type (may be "")
+ * @param {object} [params.tierPaths] per-Spark override { hot?, warm?, cold?: string[] }
+ * @returns {"hot"|"warm"|"cold"}
+ */
+export function classifyTier(device, mount, fstype = "", tierPaths = {}) {
+  const overrides = tierPaths || {};
+  // Explicit per-Spark overrides win — match by mount-prefix or device name.
+  for (const tier of ["hot", "warm", "cold"]) {
+    const paths = overrides[tier];
+    if (Array.isArray(paths)) {
+      for (const p of paths) {
+        if (!p) continue;
+        if (mount === p || mount.startsWith(p + "/") || device === p) return tier;
+      }
+    }
+  }
+  // Cold: NAS/library share — by fstype or well-known mount prefix.
+  const ft = (fstype || "").toLowerCase();
+  if (TIER_DEFAULTS.COLD_FSTYPES.includes(ft)) return "cold";
+  if (
+    mount &&
+    TIER_DEFAULTS.COLD_MOUNT_PREFIXES.some((p) => mount === p || mount.startsWith(p + "/"))
+  ) {
+    return "cold";
+  }
+  // Hot: the primary internal disk — root mount or the GB10 root partition.
+  if (mount === "/" || device === "nvme0n1p2") return "hot";
+  // Everything else that is a real local mount is warm.
+  return "warm";
+}
+
+/** True when a file name carries a model-weight extension. */
+export function isWeightFile(name) {
+  const dot = name.lastIndexOf(".");
+  if (dot < 0) return false;
+  return WEIGHT_EXTENSIONS.has(name.slice(dot).toLowerCase());
+}
+
+/** Strip the weight extension from a file name (e.g. "model.gguf" -> "model"). */
+export function stripWeightExt(name) {
+  const dot = name.lastIndexOf(".");
+  return dot > 0 ? name.slice(0, dot) : name;
+}
+
+/** find -name args for weight extensions (used by the remote scan). */
+export const WEIGHT_ARGS = Array.from(WEIGHT_EXTENSIONS)
+  .map((ext) => `-name '*${ext}'`)
+  .join(" ");
+
 /**
  * SystemCollector — collects hardware metrics for a Spark.
  * In Phase 2, this is the LOCAL path only (no SSH).
@@ -96,6 +159,130 @@ export class SystemCollector {
     }
   }
 
+  /**
+   * Collect the Spark's resident model inventory, grouped by storage tier.
+   * Which dirs are scanned come from the per-Spark `modelDirs` config
+   * ({ hot?, warm?, cold?: string[] }); a tier with no configured dir
+   * contributes nothing. With no `modelDirs` set, returns [].
+   */
+  async collectModels() {
+    try {
+      const dirs = this.spark.modelDirs || {};
+      const models = [];
+      const seen = new Set();
+      for (const tier of ["hot", "warm", "cold"]) {
+        const roots = dirs[tier];
+        if (!Array.isArray(roots) || roots.length === 0) continue;
+        for (const root of roots) {
+          if (!root) continue;
+          const found = this.spark.isLocal
+            ? await this._scanLocalRoot(root)
+            : await this._scanRemoteRoot(root);
+          for (const m of found) {
+            if (!m?.name || !m?.sizeBytes || seen.has(m.name)) continue;
+            seen.add(m.name);
+            models.push({ name: m.name, sizeBytes: m.sizeBytes, tier });
+          }
+        }
+      }
+      return models;
+    } catch (err) {
+      console.error(`[SystemCollector] Models error for ${this.spark.id}:`, err.message);
+      return [];
+    }
+  }
+
+  /**
+   * Scan one model dir locally. A top-level entry is a model if it is a loose
+   * weight file, or a subdirectory whose immediate children include a weight
+   * file. Size is on-disk bytes (dir size recursed; loose-file size direct).
+   * @returns {Promise<Array<{name:string,sizeBytes:number}>>}
+   */
+  async _scanLocalRoot(root) {
+    let entries;
+    try {
+      entries = await fs.promises.readdir(root, { withFileTypes: true });
+    } catch {
+      return [];
+    }
+    const out = [];
+    for (const entry of entries) {
+      const full = path.join(root, entry.name);
+      try {
+        if (entry.isFile() && isWeightFile(entry.name)) {
+          const st = await fs.promises.stat(full);
+          out.push({ name: stripWeightExt(entry.name), sizeBytes: st.size });
+        } else if (entry.isDirectory() && (await this._dirHasWeights(full))) {
+          out.push({ name: entry.name, sizeBytes: await this._dirSizeBytes(full) });
+        }
+      } catch {
+        // Unreadable entry — skip (matches the storage collector's per-mount catch)
+      }
+    }
+    return out;
+  }
+
+  /** True when the immediate children of `dir` include a weight file. */
+  async _dirHasWeights(dir) {
+    try {
+      const names = await fs.promises.readdir(dir);
+      return names.some((n) => isWeightFile(n));
+    } catch {
+      return false;
+    }
+  }
+
+  /** Recursive on-disk size of a directory (bytes). */
+  async _dirSizeBytes(dir) {
+    let total = 0;
+    const stack = [dir];
+    while (stack.length > 0) {
+      const cur = stack.pop();
+      let names;
+      try {
+        names = await fs.promises.readdir(cur, { withFileTypes: true });
+      } catch {
+        continue;
+      }
+      for (const e of names) {
+        const full = path.join(cur, e.name);
+        if (e.isDirectory()) stack.push(full);
+        else if (e.isFile()) {
+          try {
+            total += (await fs.promises.stat(full)).size;
+          } catch {}
+        }
+      }
+    }
+    return total;
+  }
+
+  /**
+   * Scan one model dir over SSH (single call per root). Prints one
+   * "name<TAB>bytes" line per model so parsing stays trivial; a subdir only
+   * qualifies when it immediately contains a weight file; loose weight files
+   * are reported by file stem.
+   * @returns {Promise<Array<{name:string,sizeBytes:number}>>}
+   */
+  async _scanRemoteRoot(root) {
+    const cmd =
+      `find '${root}' -maxdepth 1 -type f \( ${WEIGHT_ARGS} \) -printf '%f\t%s\n' 2>/dev/null; ` +
+      `for d in '${root}'/*/; do [ -d "$d" ] || continue; ` +
+      `if find "$d" -maxdepth 1 -type f \( ${WEIGHT_ARGS} \) -print -quit 2>/dev/null | grep -q .; then ` +
+      `echo "$(basename "$d")\t$(du -sb "$d" 2>/dev/null | cut -f1)"; fi; done`;
+    const output = await sshExec(this.spark, cmd, { timeoutMs: 15000 });
+    const models = [];
+    for (const line of output.split("\n")) {
+      const m = line.match(/^(.+?)\t(\d+)$/);
+      if (!m) continue;
+      const name = String(m[1]).replace(/^\//, "");
+      const sizeBytes = parseInt(m[2], 10) || 0;
+      if (!name || sizeBytes <= 0) continue;
+      models.push({ name, sizeBytes });
+    }
+    return models;
+  }
+
   /** Collect network metrics (interfaces, speeds). */
   async collectNetwork() {
     if (!this.spark.isLocal) return this._getRemoteNetwork();
@@ -596,6 +783,7 @@ export class SystemCollector {
           readSpeed: io.readSpeed,
           writeSpeed: io.writeSpeed,
           disabled: isDisabled,
+          tier: classifyTier(name, displayMount, fstype, this.spark.tierPaths),
         });
       } catch (err) {
         console.warn(
@@ -897,6 +1085,22 @@ export class SystemCollector {
       }
     } catch {}
 
+    // NV_ERR_NO_MEMORY kernel error count from the NVRM driver (host journal).
+    let nvErrNoMemory = 0;
+    try {
+      const js = "journalctl -k --no-pager 2>/dev/null | grep -c \"NV_ERR_NO_MEMORY\" || true";
+      const out = this._hasHostProc() ? await this._execOnHost(js) : await this._exec(js);
+      const n = Number(String(out).trim());
+      if (!Number.isNaN(n) && n > 0) nvErrNoMemory = Math.round(n);
+    } catch {
+      try {
+        const dmesg = "dmesg 2>/dev/null | grep -c \"NV_ERR_NO_MEMORY\" || true";
+        const out = this._hasHostProc() ? await this._execOnHost(dmesg) : await this._exec(dmesg);
+        const n = Number(String(out).trim());
+        if (!Number.isNaN(n) && n > 0) nvErrNoMemory = Math.round(n);
+      } catch {}
+    }
+
     return {
       total: totalMB,
       gpuUsed: gpuUsedMB,
@@ -906,6 +1110,7 @@ export class SystemCollector {
       percentage,
       oomRisk,
       bandwidth,
+      nvErrNoMemory,
     };
   }
 
@@ -1084,6 +1289,7 @@ export class SystemCollector {
           readSpeed: 0,
           writeSpeed: 0,
           disabled: isDisabled,
+          tier: classifyTier(device, mount, type, this.spark.tierPaths),
         });
       }
 
@@ -1248,6 +1454,17 @@ export class SystemCollector {
       const percentage = totalMB > 0 ? Math.round((usedMB / totalMB) * 100) : 0;
       const oomRisk = percentage > 85 ? "high" : percentage > 60 ? "medium" : "low";
 
+      // NV_ERR_NO_MEMORY kernel error count from the NVRM driver (remote host).
+      let nvErrNoMemory = 0;
+      try {
+        const countRaw = await sshExec(
+          this.spark,
+          "journalctl -k --no-pager 2>/dev/null | grep -c \"NV_ERR_NO_MEMORY\" || true"
+        );
+        const n = Number(String(countRaw).trim());
+        if (!Number.isNaN(n) && n > 0) nvErrNoMemory = Math.round(n);
+      } catch {}
+
       return {
         total: totalMB,
         gpuUsed: gpuUsedMB,
@@ -1257,6 +1474,7 @@ export class SystemCollector {
         percentage,
         oomRisk,
         bandwidth: { current: 0, peak: 400 },
+        nvErrNoMemory,
       };
     } catch (err) {
       console.error(`[SystemCollector] Remote Unified Memory error for ${this.spark.id}:`, err.message);
@@ -1411,6 +1629,7 @@ export class SystemCollector {
       percentage: 0,
       oomRisk: "low",
       bandwidth: { current: 0, peak: 0 },
+      nvErrNoMemory: 0,
     };
   }