LLM Systems Manager is a self-hosted control plane for the machines that run your models. It watches every inference host in real time, raises alarms before small problems become outages, and gives operators a single dashboard to start, stop, upgrade, and tune the stack — without SSH-ing into a fleet of boxes.
- Live observability — per-host metrics (CPU, memory, GPU, throughput, inference state) streamed to a real-time dashboard over SSE.
- Alarm engine — threshold, rate-of-change, z-score, moving-average and percentile rules with auto-resolve, quiet hours, and multi-channel notifications.
- Fleet control — manage inference backends (llama.cpp, LM Studio) and system services from one place: status, restarts, model loads, and guided upgrades.
- Agent-based — a lightweight agent on each host reports telemetry and executes authorized control actions; the manager keeps the source of truth.
- History that lasts — metrics persist to InfluxDB with tiered downsampling, so a 30-day chart is as fast as the last hour.
| Component | Role |
|---|---|
| Manager | Flask backend + dashboard. Auth, fleet state, control actions, live streams. |
| Alarm Engine | Independent service that evaluates rules against the metric stream and dispatches notifications. |
| Agent | Runs on each inference host. Collects metrics, reports state, performs authorized control. |
Metrics flow Agent → Manager → InfluxDB; rules run in the Alarm Engine; operators drive everything from the dashboard.
- llm-systems-manager — manager, alarm engine, and agent for the full observability & control stack.
Built for operators who run their own models. ·
llmsyscore