feat(monitoring): settle the scrape path for smaug, the first scraped host

.github/dependabot.yml

@@ -76,13 +76,17 @@ updates:
       wazuh:
         patterns: ["wazuh/*"]
 
-  # The media stack on smaug. One image, so no groups. Present before the pool
-  # is, for the reason the entries above give — but this one was absent while
-  # three documents said it was here: ADR-0040 keeps stacks/media in this
-  # repository rather than making it a TrueNAS catalogue app precisely so
-  # "Dependabot, the digest pins and make validate keep reaching it", and
-  # build-the-nas.md §6 and the stack's own README repeat the claim. The pins
-  # and make validate did reach it. This is the third.
+  # The media stack on smaug. Present before the pool is, for the reason the
+  # entries above give — but this one was absent while three documents said it
+  # was here: ADR-0040 keeps stacks/media in this repository rather than making
+  # it a TrueNAS catalogue app precisely so "Dependabot, the digest pins and
+  # make validate keep reaching it", and build-the-nas.md §6 and the stack's own
+  # README repeat the claim. The pins and make validate did reach it. This is
+  # the third.
+  #
+  # Two images now, not one: #256 added a node_exporter beside Jellyfin, because
+  # this host is scraped rather than running an agent. They share nothing, so no
+  # groups — each bump is its own PR.
   - package-ecosystem: docker-compose
     directory: /stacks/media
     schedule:

README.md

@@ -63,7 +63,7 @@ documents for different readers.
   can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
   identically everywhere. [How](docs/architecture.md#observability-data-flow)
 - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and
-  100 alert rules — 82 metric-based in Prometheus, 18 log-based in Loki — sharing
+  101 alert rules — 83 metric-based in Prometheus, 18 log-based in Loki — sharing
   one Alertmanager routing tree. No dashboard exists only in a database.
 - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
   decrypted at deploy time into gitignored paths, with `git log` showing which
@@ -186,7 +186,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
 .
 ├── stacks/observability/     # the deployed stack — one compose file, eight services
 │   ├── compose.yaml
-│   ├── prometheus/           # config, file_sd targets, 82 alert rules
+│   ├── prometheus/           # config, file_sd targets, 83 alert rules
 │   ├── alertmanager/         # routing and inhibition
 │   ├── loki/                 # single-binary config + 18 LogQL rules
 │   ├── alloy/                # the agent config directory, shipped to every host

docs/architecture.md

@@ -191,7 +191,7 @@ hole from the monitoring VLAN into the monitored one.
 | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | **Not built yet** — the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) |
 | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
 | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
-| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Runs no agent of its own: this host is scraped and never pushes, which makes it the estate's first scraped host and is why [#256](https://github.com/Gerrrt/HomeLab/issues/256) exists — that issue now also has a fork in it, because TrueNAS ships its own metrics endpoint where the issue specifies `node_exporter` |
+| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live; the target itself stays commented until there is a pool to run the exporter from |
 | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
 | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
 | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |

docs/network.md

@@ -320,6 +320,14 @@ Televisions and consoles. Internet only.
   to CasaBonita* on their interfaces. Everything else on every other segment is
   still refused, and the televisions need no rule at all because they share this
   broadcast domain with the server.
+- **What answers on `9100` is `node_exporter`**, which makes this the one host
+  in the estate that Prometheus *scrapes* rather than is pushed to
+  ([#256](https://github.com/Gerrrt/HomeLab/issues/256),
+  [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)).
+  Nothing answers there yet — the exporter is a container and the container
+  needs the pool — so `prometheus/targets/node.yaml` carries the target
+  commented out rather than pointing a live check at a port with nothing behind
+  it. Port `22` is inert for a different reason: TrueNAS ships SSH disabled.
 
 [^OLEDTV]: [LG OLED TV](https://www.lg.com/us/tvs/oled)
 [^PS5]: [PlayStation 5](https://www.playstation.com/en-us/ps5/)

docs/observability.md

@@ -37,6 +37,7 @@ it is not closed by anything in this document.
 | Source | Via | Interval | Examples |
 | --- | --- | --- | --- |
 | Linux hosts | Alloy → `node_exporter` | 60s | CPU, memory, filesystem, network, load, clock offset |
+| `smaug`, the NAS | Prometheus → `node_exporter` | 60s | The same, minus network — the one host that is SCRAPED and not pushed to, and the one that ships no logs ([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#255](https://github.com/Gerrrt/HomeLab/issues/255)) |
 | Docker containers | Alloy → cAdvisor | 60s | Per-container CPU, memory, network, restarts, OOM |
 | Container logs | Alloy → Docker socket | stream | stdout/stderr per container |
 | systemd journal | Alloy | stream | unit, boot ID, transport, priority. Delivery is watched by `JournalSourceStopped` |
@@ -185,12 +186,23 @@ it should report hundreds. Both were invisible for hours because the only view
 of the collection path was `up{job="alloy"}`, which stayed `1` throughout.
 
 `up` is a poor liveness signal for half of what this stack collects, and the
-dashboard says so rather than papering over it. Prometheus scrapes nine jobs
+dashboard says so rather than papering over it. Prometheus scrapes twelve jobs
 directly; the rest arrive by remote_write — one `<host>-metrics` and one
 `<host>-alloy` per agent, plus `integrations/cadvisor` wherever there is
 Docker. **A directly scraped target that dies sets `up` to 0. A remote-writing
 agent that dies just stops pushing, so its `up` goes stale and ages out instead
 of falling** — and `InstanceDown` is `up == 0`, so it cannot see that at all.
+
+Eleven of those twelve jobs are containers on the compose network or devices
+behind an exporter. The twelfth is `node`, and it is a MACHINE — `smaug`, which
+[ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) put
+on a segment that may not initiate upward, so Prometheus reaches in and scrapes
+it rather than being pushed to ([#256](https://github.com/Gerrrt/HomeLab/issues/256)).
+It is the estate's first scraped host, it needs no new down-detection because
+`InstanceDown` has no job matcher, and it is deliberately **not** called
+`smaug-metrics`: that name would enrol a pulled job in `RemoteWriteJobStale`,
+whose notification says an Alloy agent has stopped pushing. This host has no
+Alloy agent, by decision.
 The *Sample staleness by job* panel is what covers the pushed jobs on the
 dashboard, and the *Every target* table puts `Staleness` next to `Up` for the
 same reason.
@@ -326,7 +338,7 @@ separates a quiet stream from a stopped one.
 
 ## Alerting
 
-100 rules in total: 82 metric-based in `prometheus/rules/`, and 18 log-based in
+101 rules in total: 83 metric-based in `prometheus/rules/`, and 18 log-based in
 `loki/rules/`.
 
 ### Log-based (Loki ruler)
@@ -433,15 +445,15 @@ argument and for what to do when it exits 1.
 
 ### Metric-based (Prometheus)
 
-82 rules across eleven files in `prometheus/rules/`:
+83 rules across eleven files in `prometheus/rules/`:
 
 | File | Covers |
 | --- | --- |
 | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots — and, for the two laptops, whether the shelf is on mains and whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap); and whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)) |
 | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) |
 | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
 | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
-| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and remote-writing agents that stop pushing — the case `up == 0` structurally cannot see. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault |
+| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` is written into `targets/node.yaml` and left disabled until the NAS has a pool to run the exporter from. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault |
 | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
 | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) |
 | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) |
@@ -456,12 +468,13 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
 `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
 feed a rule synthetic series and assert it fires — paired with a case asserting
 it stays quiet, because a test that only ever expects silence would have passed
-against the broken rule too. Coverage is sixty-two rules of 82 so far — the five
+against the broken rule too. Coverage is sixty-three rules of 83 so far — the five
 in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
 `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
 `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
 [#76](https://github.com/Gerrrt/HomeLab/issues/76), all six in
 `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`,
+`ScrapeTargetDisappeared`,
 `SuricataStopped`, the two gateway rules from
 [#353](https://github.com/Gerrrt/HomeLab/issues/353), and all twenty-two in
 `host.rules.yaml` —
@@ -686,6 +699,11 @@ short:
 - **A Linux host:** run Alloy with `LOKI_URL` and
   `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. Nothing on the
   monitoring host changes.
+- **A Linux host that may not push:** a firewall pass first, then
+  `node_exporter` in that host's own compose stack, then a target in
+  `prometheus/targets/node.yaml` with `instance` set to the hostname. The
+  direction reverses when the segment demands it, and the tool reverses with
+  it — `smaug` is the only one today.
 - **An SNMP device:** append a target to
   `prometheus/targets/snmp.yaml` and a module plus auth to
   `snmp-exporter/generator.yaml`. file_sd picks the target up within five