From 36c4eb40a7c97899f08918ea902c69bbd8090dec Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:52:21 +0000 Subject: [PATCH] feat(monitoring): settle the scrape path for smaug, the first scraped host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every host in this estate pushes: Alloy remote_writes to 10.0.99.20 and Prometheus is never told the host exists. ADR-0016 reversed that for the NAS, because CasaBonita is terminal-outward and nothing on it may initiate upward — so Prometheus reaches in and scrapes instead, making smaug the estate's first scraped HOST rather than a container or a device behind an exporter. ADR-0040 then put TrueNAS on that host and opened a fork this issue had to settle: node_exporter on 9100, as the issue specified, or TrueNAS's own metrics endpoint on another port. THE ANSWER IS node_exporter, run as a digest-pinned container out of stacks/media. The 99 -> 40:9100 pass already exists and is verified in position; host-overview.json and seven rules in host.rules.yaml are built entirely on the node_* namespace; and a container here stays inside Dependabot, the digest pins and make validate, which is ADR-0040 decision 2's argument for this stack living in the repository at all. Said plainly in targets/node.yaml, because it matters: the alternative was declined on the pass that exists and the dashboard that exists, and NOT on a comparison. This repository records nothing about what TrueNAS natively exposes — four documents restate the same unsourced sentence. The target ships COMMENTED OUT. TrueNAS has no package manager and an immutable root, so every route to an exporter there runs through Docker and therefore through the apps pool, which waits on the drives. A live target would mean up == 0 and InstanceDown paging urgent every four hours forever — ADR-0017's refusal, and the call blackbox-dns.yaml and the lab's Windows job both already make. ADR-0040's "the scrape, then the stack" is right about authoring order and not achievable for liveness; build-the-nas.md 6.1 is the one-line uncomment that finishes it, with the check that catches a wrong --path.rootfs first. On the alerting question the issue asked: InstanceDown is `up == 0` with no job matcher, so it covers a scraped host for free and is already unit tested. The real gap was the other half — a target that stops being a target. An emptied or unparseable targets/node.yaml makes the series vanish rather than fall to 0, so up == 0 matches nothing, and RemoteWriteJobStale excludes directly scraped jobs by design. That hole was exactly the width of the one host Prometheus has to be told about. ScrapeTargetDisappeared closes it, mirroring RemoteWriteJobStale's shape and bound. Measured on the pinned image rather than assumed, as stacks/media asks: it declares user nobody, ships wget and no curl, has NO /-/healthy (404 — the four siblings healthchecked that way all serve it), boots read-only as 65534 with cap_drop ALL, and idles at 3.4 MiB. And the one that changed a decision: a bridged container reads its own netns, so node_network_* named the container's eth0 on a host with real NICs — which passes host-overview's device filter and would have been charted as this NAS's throughput. Those collectors are disabled; reporting nothing is honest, reporting a believable wrong number is not. The four test cases were mutation-tested. `for: 15m -> 0m` SURVIVED the first round, because a firing assertion at 60m cannot tell it from `for: 0m`; the eval_time: 40m pairing is what kills it. Recorded in the test file rather than quietly fixed. The job is called `node` and not `smaug-metrics`: that name would enrol a pulled job in RemoteWriteJobStale, whose notification text says an Alloy agent stopped pushing — false twice over about the one host with no Alloy agent by decision, and quietly so, because the InstanceDown inhibit would suppress the duplicate. The cost is one dashboard expression. Also folded in, because this change makes them wrong or found them wrong: security.md still said three CasaBonita passes and "neither is created yet" (four, created 2026-09-16); observability.md said Prometheus scrapes nine jobs (eleven before this, twelve after); dependabot.yml had no /stacks/media entry, so Jellyfin's pin was already rotting unnoticed. Refs #256 Co-Authored-By: Claude Opus 5 --- .github/dependabot.yml | 17 +++ README.md | 4 +- docs/architecture.md | 2 +- docs/network.md | 12 +- docs/observability.md | 28 +++- docs/roadmap.md | 13 +- docs/runbooks/add-monitored-device.md | 111 ++++++++++++++- docs/runbooks/build-the-nas.md | 41 +++++- docs/runbooks/deploy-stack.md | 4 +- docs/security.md | 18 ++- stacks/media/.env.example | 14 ++ stacks/media/README.md | 43 +++++- stacks/media/compose.yaml | 131 ++++++++++++++++++ stacks/observability/README.md | 2 +- .../grafana/dashboards/host-overview.json | 2 +- .../observability/prometheus/prometheus.yaml | 51 +++++++ .../prometheus/rules/stack.rules.yaml | 65 +++++++++ .../prometheus/targets/node.yaml | 103 ++++++++++++++ .../prometheus/tests/host.test.yaml | 40 ++++++ .../prometheus/tests/stack.test.yaml | 104 ++++++++++++++ 20 files changed, 772 insertions(+), 33 deletions(-) create mode 100644 stacks/observability/prometheus/targets/node.yaml diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 635bc031..7d282cbc 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -76,6 +76,23 @@ updates: wazuh: patterns: ["wazuh/*"] + # The media tier, on smaug. Two images that share nothing with each other — + # Jellyfin and a node_exporter — so no groups; each bump is its own PR. Added + # with the exporter (#256) rather than with the stack, because until then this + # directory held one pinned image and no entry, and it had already been + # rotting for the reason every comment above gives: pins rot from the day they + # are written, and a stack nobody has deployed yet is exactly where that goes + # unnoticed. + - package-ecosystem: docker-compose + directory: /stacks/media + schedule: + interval: weekly + day: sunday + open-pull-requests-limit: 5 + commit-message: + prefix: "chore(deps)" + labels: ["dependencies", "media"] + - package-ecosystem: github-actions directory: / schedule: diff --git a/README.md b/README.md index 43880117..261822fe 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and - 100 alert rules — 82 metric-based in Prometheus, 18 log-based in Loki — sharing + 101 alert rules — 83 metric-based in Prometheus, 18 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -186,7 +186,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, eight services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 82 alert rules +│ ├── prometheus/ # config, file_sd targets, 83 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 18 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/architecture.md b/docs/architecture.md index c4604870..2bf49f78 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -190,7 +190,7 @@ hole from the monitoring VLAN into the monitored one. | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Runs no agent of its own: this host is scraped and never pushes, which makes it the estate's first scraped host and is why [#256](https://github.com/Gerrrt/HomeLab/issues/256) exists — that issue now also has a fork in it, because TrueNAS ships its own metrics endpoint where the issue specifies `node_exporter` | +| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live; the target itself stays commented until there is a pool to run the exporter from | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/network.md b/docs/network.md index 0822fa96..3195968c 100644 --- a/docs/network.md +++ b/docs/network.md @@ -300,8 +300,8 @@ Televisions and consoles. Internet only. and running TrueNAS by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) ([#413](https://github.com/Gerrrt/HomeLab/issues/413)). Its ZFS mirror does - not exist yet and neither does `stacks/media`; what exists is a host on its - address. **It does not change the *Reaches* column**, and that is the point + not exist yet and `stacks/media` is authored but not deployed; what exists is + a host on its address. **It does not change the *Reaches* column**, and that is the point ADR-0016 made in advance: nothing on this segment initiates anywhere, and the four rules created that day all let a more trusted segment reach **in**. That is the direction this row records, and it is the one that is unchanged. @@ -311,6 +311,14 @@ Televisions and consoles. Internet only. to CasaBonita* on their interfaces. Everything else on every other segment is still refused, and the televisions need no rule at all because they share this broadcast domain with the server. +- **What answers on `9100` is `node_exporter`**, which makes this the one host + in the estate that Prometheus *scrapes* rather than is pushed to + ([#256](https://github.com/Gerrrt/HomeLab/issues/256), + [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md)). + Nothing answers there yet — the exporter is a container and the container + needs the pool — so `prometheus/targets/node.yaml` carries the target + commented out rather than pointing a live check at a port with nothing behind + it. Port `22` is inert for a different reason: TrueNAS ships SSH disabled. [^OLEDTV]: [LG OLED TV](https://www.lg.com/us/tvs/oled) [^PS5]: [PlayStation 5](https://www.playstation.com/en-us/ps5/) diff --git a/docs/observability.md b/docs/observability.md index 630277bb..a9271b42 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -37,6 +37,7 @@ it is not closed by anything in this document. | Source | Via | Interval | Examples | | --- | --- | --- | --- | | Linux hosts | Alloy → `node_exporter` | 60s | CPU, memory, filesystem, network, load, clock offset | +| `smaug`, the NAS | Prometheus → `node_exporter` | 60s | The same, minus network — the one host that is SCRAPED and not pushed to, and the one that ships no logs ([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#255](https://github.com/Gerrrt/HomeLab/issues/255)) | | Docker containers | Alloy → cAdvisor | 60s | Per-container CPU, memory, network, restarts, OOM | | Container logs | Alloy → Docker socket | stream | stdout/stderr per container | | systemd journal | Alloy | stream | unit, boot ID, transport, priority. Delivery is watched by `JournalSourceStopped` | @@ -185,12 +186,23 @@ it should report hundreds. Both were invisible for hours because the only view of the collection path was `up{job="alloy"}`, which stayed `1` throughout. `up` is a poor liveness signal for half of what this stack collects, and the -dashboard says so rather than papering over it. Prometheus scrapes nine jobs +dashboard says so rather than papering over it. Prometheus scrapes twelve jobs directly; the rest arrive by remote_write — one `-metrics` and one `-alloy` per agent, plus `integrations/cadvisor` wherever there is Docker. **A directly scraped target that dies sets `up` to 0. A remote-writing agent that dies just stops pushing, so its `up` goes stale and ages out instead of falling** — and `InstanceDown` is `up == 0`, so it cannot see that at all. + +Eleven of those twelve jobs are containers on the compose network or devices +behind an exporter. The twelfth is `node`, and it is a MACHINE — `smaug`, which +[ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) put +on a segment that may not initiate upward, so Prometheus reaches in and scrapes +it rather than being pushed to ([#256](https://github.com/Gerrrt/HomeLab/issues/256)). +It is the estate's first scraped host, it needs no new down-detection because +`InstanceDown` has no job matcher, and it is deliberately **not** called +`smaug-metrics`: that name would enrol a pulled job in `RemoteWriteJobStale`, +whose notification says an Alloy agent has stopped pushing. This host has no +Alloy agent, by decision. The *Sample staleness by job* panel is what covers the pushed jobs on the dashboard, and the *Every target* table puts `Staleness` next to `Up` for the same reason. @@ -326,7 +338,7 @@ separates a quiet stream from a stopped one. ## Alerting -100 rules in total: 82 metric-based in `prometheus/rules/`, and 18 log-based in +101 rules in total: 83 metric-based in `prometheus/rules/`, and 18 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -433,7 +445,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -82 rules across eleven files in `prometheus/rules/`: +83 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -441,7 +453,7 @@ argument and for what to do when it exits 1. | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | -| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and remote-writing agents that stop pushing — the case `up == 0` structurally cannot see. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | +| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` is written into `targets/node.yaml` and left disabled until the NAS has a pool to run the exporter from. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) | @@ -456,12 +468,13 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is sixty-two rules of 82 so far — the five +against the broken rule too. Coverage is sixty-three rules of 83 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from [#76](https://github.com/Gerrrt/HomeLab/issues/76), all six in `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, +`ScrapeTargetDisappeared`, `SuricataStopped`, the two gateway rules from [#353](https://github.com/Gerrrt/HomeLab/issues/353), and all twenty-two in `host.rules.yaml` — @@ -686,6 +699,11 @@ short: - **A Linux host:** run Alloy with `LOKI_URL` and `PROMETHEUS_REMOTE_WRITE_URL` pointed at `10.0.99.20`. Nothing on the monitoring host changes. +- **A Linux host that may not push:** a firewall pass first, then + `node_exporter` in that host's own compose stack, then a target in + `prometheus/targets/node.yaml` with `instance` set to the hostname. The + direction reverses when the segment demands it, and the tool reverses with + it — `smaug` is the only one today. - **An SNMP device:** append a target to `prometheus/targets/snmp.yaml` and a module plus auth to `snmp-exporter/generator.yaml`. file_sd picks the target up within five diff --git a/docs/roadmap.md b/docs/roadmap.md index 90c8307f..2ba96112 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -789,11 +789,14 @@ what left this one unfireable for months. the `22` that ADR-0016's table names — that port assumed a box administered over SSH, which was an operating-system decision inside a firewall table, and ADR-0040 carries the correction against its own text. Port 22 survives on the - Winterfell rule and is **inert**, because TrueNAS ships SSH disabled. Whether - the scrape target is `node_exporter` on `9100` or TrueNAS's own endpoint is - open, and it is not a free choice — - [#256](https://github.com/Gerrrt/HomeLab/issues/256) settles it, and the - second answer costs a fifth rule. Terminal survives + Winterfell rule and is **inert**, because TrueNAS ships SSH disabled. The scrape target is + **`node_exporter` on `9100`**, settled by + [#256](https://github.com/Gerrrt/HomeLab/issues/256) on 2026-09-17 — the pass + that exists and a dashboard built entirely on `node_*` series, against an + alternative the repository had never measured and which would have cost a + fifth rule. The job and the target file are written; the target is commented + until the pool exists, because on TrueNAS every route to an exporter runs + through it. Terminal survives in the direction that carries it — CasaBonita stopped being terminal inbound on 2026-09-16 and stays terminal outbound, with #223's tripwire untouched and reading zero packets. Capacity buys a diff --git a/docs/runbooks/add-monitored-device.md b/docs/runbooks/add-monitored-device.md index af97c232..8619859e 100644 --- a/docs/runbooks/add-monitored-device.md +++ b/docs/runbooks/add-monitored-device.md @@ -1,8 +1,8 @@ # Runbook: Add a monitored device -Four paths: a host that can run an agent, a device that cannot, an endpoint -with a URL, and a resolver — which is asked what it answered rather than -whether it can be reached. +Five paths: a host that can run an agent, a host that may not, a device that +cannot, an endpoint with a URL, and a resolver — which is asked what it +answered rather than whether it can be reached. --- @@ -157,6 +157,111 @@ which is what a missing rule looks like. --- +## A Linux host that may not push + +Read the path above first. This one exists because its opening sentence is +false for some hosts, and the difference is a decision about the network rather +than about the machine. + +**When this path applies.** The host sits on a segment that may not initiate +upward — a terminal segment, in the sense +[ADR-0013](../adr/0013-read-the-firewall-before-writing-about-it.md) uses — +or it cannot run an agent at all. `smaug` is the estate's only one today: the +NAS is on CasaBonita, and +[ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) +would not let it reach up to the monitoring host, so Prometheus reaches down and +scrapes it instead. The convention that every host runs Alloy exists to serve a +direction; where the direction reverses, so does the tool. + +The cost is stated where the decision is: Loki has no pull, so a host monitored +this way gets metrics and **no logs** +([#255](https://github.com/Gerrrt/HomeLab/issues/255)). That is not a gap to +work around here — it is the price of the placement, and it belongs in +`docs/security.md` as a residual before the host is monitored at all. + +### 1. The firewall pass first, and verified in position + +Nothing below is testable until the monitoring host can reach the port, and a +pass appended where new rules naturally land will sit *below* an existing deny +and match nothing — which presents as "the scrape is broken" rather than as a +misplaced rule. Create it host-scoped and port-scoped, then **read the ruleset +on `morpheus` rather than the UI** to confirm the position. `build-the-nas.md` +§0.5 and §0.6 are the worked example. + +ADR-0016 declined to create its passes until the host answered, for the reason +that a `pass` to an address with nothing behind it is a rule nobody can test. +The order is hardware → rule → target, and each step is only checkable after the +one before it. + +### 2. An exporter on the host, in that host's own stack + +`node_exporter`, because it is what the `host-overview` dashboard and seven +rules in `host.rules.yaml` are written against — see +`prometheus/targets/node.yaml`, which carries that argument in full. + +It belongs in the host's own compose stack, digest-pinned like everything else +([ADR-0004](../adr/0004-one-compose-stack-per-host.md); ADR-0040 decision 3 +declined to make `smaug` an exception to it). `stacks/media/compose.yaml`'s +`node-exporter` service is the model, including the two things that are not +obvious from upstream's documentation: it has no `/-/healthy`, and a bridged +container reads its own veth for `node_network_*`. + +### 3. Add the target — `prometheus/targets/node.yaml` + +```yaml +- targets: ["
:9100"] + labels: + instance: + role: + vlan: "" +``` + +**`instance` is the one that matters.** A direct scrape defaults it to the +address, and the dashboards template their host picker on it — so without this +line the host appears as an IP and a port beside a list of hostnames, and +`InstanceDown` pages reading the same. Set it to what `hostname` would say, so a +scraped host sorts with the pushed ones. + +**Never set a `job` label.** `honor_labels` is false, so it arrives as +`exported_job` while `job` keeps the value from `prometheus.yaml` — and every +rule matching on `job` then matches nothing, with the target showing UP. + +The file is a directory mount, so Prometheus re-reads it within five minutes +with no restart and no deploy. + +### 4. Nothing else, and that is the point + +`InstanceDown` is `up == 0` with no job matcher, so it covers this host from its +first scrape — `tests/host.test.yaml` asserts exactly that against `smaug`. +`ScrapeTargetDisappeared` covers the other half, which is this target going +missing from the file rather than the host going down. +`RemoteWriteJobStale` deliberately does **not** apply: it is for jobs that push, +and giving a scraped job a `-metrics` name would quietly enrol it in a +rule whose notification text is false for it. + +### Verify + +```promql +up{job="node"} +``` + +Then read the metrics themselves rather than the target state, because a wrong +`--path.rootfs` produces a healthy target describing a container: + +```bash +curl -s http://
:9100/metrics | grep -c '^node_filesystem_avail_bytes' +``` + +Zero means every disk rule on that host is blind and nothing will say so. + +### 5. Document it + +A row in `docs/network.md`, the host's row in `docs/architecture.md` — phrased +"runs no Alloy", which `scripts/check_docs.py` reads to keep the agent count +honest — and the residual in `docs/security.md` if the host ships no logs. + +--- + ## An SNMP device Three steps, no restart. They touch five files: diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 86a8887f..d4e22c62 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -323,7 +323,7 @@ Create a dedicated TrueNAS user for it rather than sharing the admin account. The admin credential is the one that guards everything on this box, and an SMB share is mounted by televisions. -## §6 — Jellyfin +## §6 — The stack, and the scrape Deploy the stack from the repository, per [ADR-0040](../adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md): @@ -332,7 +332,38 @@ catalogue app. That is what keeps Dependabot, the digest pins and `make validate` reaching it. Jellyfin binds `8096`, reads `erebor/media`, and writes its state to -`erebor/apps`. +`erebor/apps`. `node-exporter` binds `9100` and is the whole of how this host +is monitored — see §6.1. + +### §6.1 — Turn the scrape on, and prove it before you do + +The `node-exporter` service comes up with the stack. Nothing was scraping it +until now: [#256](https://github.com/Gerrrt/HomeLab/issues/256) wrote the `node` +job and `prometheus/targets/node.yaml` with **the target commented out**, +because a scrape aimed at a port with nothing behind it means `up == 0` and +`InstanceDown` paging `urgent` every four hours until the drives arrive — +[ADR-0017](../adr/0017-buy-ifrit-and-give-it-no-monitoring.md)'s refusal. + +From the monitoring host, which can reach `9100` and nothing else on this +segment: + +```bash +curl -s http://10.0.40.30:9100/metrics | grep -c '^node_filesystem_avail_bytes' +``` + +**It must be greater than zero, and it must count `erebor`.** A wrong +`--path.rootfs` produces a container's filesystems, or none, while `up` still +reads 1 and the target shows green — so a zero here means every disk rule on +this host is blind and nothing else will tell you. + +Then uncomment the four lines at the end of `prometheus/targets/node.yaml` and +commit. That directory is a bind mount, so Prometheus re-reads it within five +minutes: no restart, no deploy, no `--force-recreate`. + +Expect **no** `node_network_*` series from this host. Those collectors are +disabled on purpose, because a bridged container reads its own veth and would +chart it as this NAS's throughput; `stacks/media/compose.yaml` carries the +measurement. > **The check ADR-0040 named as its reopen condition belongs here.** Confirm > the iGPU reaches the container: @@ -354,7 +385,8 @@ Jellyfin binds `8096`, reads `erebor/media`, and writes its state to - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved - A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096` -- The monitoring host reaches `9100` and **nothing else** +- The monitoring host reaches `9100` and **nothing else** — and `up{job="node"}` + is `1`, labelled `instance="smaug"` rather than an address - The `igc0.40` tripwire counter is **still zero** - Port 15 on `neo` reads PVID **40**, untagged, with `smaug`'s MAC learned on it in VLAN 40 — read in the switch UI, and **not** inferred from the host having @@ -364,9 +396,6 @@ Jellyfin binds `8096`, reads `erebor/media`, and writes its state to ## §8 — What this leaves open -- **[#256](https://github.com/Gerrrt/HomeLab/issues/256), the scrape path.** It - specifies `node_exporter` on `9100`, and TrueNAS ships its own metrics - endpoint — so the target shape is a fork that issue now has to settle. - **[#255](https://github.com/Gerrrt/HomeLab/issues/255)**, the residual saying this host ships no logs, which is true the day it exists. - **[ADR-0027](../adr/0027-defer-proxmox-backup-server-until-there-is-somewhere-to-send-it.md)'s diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 0a53badc..0d0e9ba4 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,8 +118,8 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 82 rules loaded, none in error. The - page counts 83: the extra one is `homelab_suricata_expected_interface`, +2. **Prometheus → Status → Rules.** 83 rules loaded, none in error. The + page counts 84: the extra one is `homelab_suricata_expected_interface`, the stack's only recording rule. Everything counted in this repository is alert rules, so the two numbers differ by one and always have. 3. **Grafana → Dashboards → HomeLab.** Seven dashboards, populated. diff --git a/docs/security.md b/docs/security.md index 8df1e4cb..fc49eeca 100644 --- a/docs/security.md +++ b/docs/security.md @@ -17,7 +17,7 @@ What this network is actually built to survive: | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 100 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 101 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — 94 % of design on `prometheus`, 72 % on `oracle` ([#454](https://github.com/Gerrrt/HomeLab/issues/454)) — see below | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | @@ -255,8 +255,20 @@ than carrying a port list, and the `22` is inert — TrueNAS ships SSH disabled `10.0.99.40 → 10.0.20.104:80,443/tcp` — Home Assistant to the Hue bridge, the one device on that segment with a local API — still waits above the block that has stood between 99 and 20 since the segments existed, on the host that would -use it ([#134](https://github.com/Gerrrt/HomeLab/issues/134)). The row ADR-0008 -wrote as `99 → 20` is narrower than it read: one host to one device on two +use it ([#134](https://github.com/Gerrrt/HomeLab/issues/134)). + +One of CasaBonita's four is a monitoring pass, and it carries a residual worth +naming. `10.0.99.20 → 10.0.40.30:9100` lets Prometheus scrape `node_exporter` +on the NAS ([#256](https://github.com/Gerrrt/HomeLab/issues/256)), and that +endpoint is **unauthenticated**. The firewall rule is what stops other segments +reading it; nothing stops CasaBonita itself, because the televisions and the +game consoles share that broadcast domain and the firewall never sees those +packets — the same property that lets them reach Jellyfin, working the other +way. node_exporter has no write API, so the exposure is disclosure of the +host's shape: filesystems, uptime, load. Accepted, and the same class as the +unauthenticated ports the observability stack publishes. + +The row ADR-0008wrote as `99 → 20` is narrower than it read: one host to one device on two ports, with the twenty other devices on Skids still unreachable from anywhere, and both segments still initiating nothing. The tripwire below is the check that the second half holds — waiting on Skids, and since 2026-09-16 actually diff --git a/stacks/media/.env.example b/stacks/media/.env.example index e7fad851..54165af2 100644 --- a/stacks/media/.env.example +++ b/stacks/media/.env.example @@ -54,3 +54,17 @@ MEDIA_PATH=/mnt/erebor/media # Hicks workstation reaches it through the rule in build-the-nas.md §0.5, and # default deny sees to everything else. JELLYFIN_BIND_ADDR=0.0.0.0 + +# The address node_exporter's 9100 is published on. +# +# Same reasoning as JELLYFIN_BIND_ADDR above — smaug is single-homed, so 0.0.0.0 +# means 10.0.40.30 and there is nothing else it could mean. The two are separate +# keys rather than one shared one because they answer different questions: 8096 +# is published FOR this segment, and 9100 is published for one host on another +# one. If those ever need to differ, they differ here rather than in a rewrite. +# +# Who may actually reach 9100 across a segment boundary is one firewall rule — +# `10.0.99.20 → 10.0.40.30:9100`, build-the-nas.md §0.5 — and not this line. +# Everything already on CasaBonita reaches it regardless; docs/security.md +# records that as the residual it is. +NODE_EXPORTER_BIND_ADDR=0.0.0.0 diff --git a/stacks/media/README.md b/stacks/media/README.md index e261851e..e73a8487 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -14,8 +14,12 @@ make up STACK=media | Service | Image | Port | Purpose | | --- | --- | --- | --- | | jellyfin | `jellyfin/jellyfin` | 8096 (http), on the segment | The media server the televisions reach directly, with Quick Sync hardware transcoding on the E3-1225 v6's HD P630 ([#138], [ADR-0016]) | +| node-exporter | `prom/node-exporter` | 9100 (http), to `10.0.99.20` only | How this host is monitored at all — Prometheus scrapes it, because nothing on this segment may push ([#256], [ADR-0016]) | -One service, and that is a decision rather than a starting point. [ADR-0016] +Two services, and only one of them is the tier. `node-exporter` is here +because of the section below; everything else in this file is about Jellyfin. + +One media service, and that is a decision rather than a starting point. [ADR-0016] builds **Jellyfin alone** and adds Plex only if a screen on 40 turns out to have no working Jellyfin client — the LG OLED, which is the primary screen, has one. Plex authenticates its clients through `plex.tv` even on a local @@ -47,10 +51,43 @@ every television would have to trust, and a second thing to be down. | --- | --- | | Televisions on CasaBonita | Natively, same broadcast domain — the firewall never sees the packet | | A Hicks workstation | The one rule in [`build-the-nas.md`] §0.5, `50 → 10.0.40.30:443,8096` | +| Prometheus, on `9100` | The rule in [`build-the-nas.md`] §0.5, `10.0.99.20 → 10.0.40.30:9100` | | Everything else on the estate | Not at all — default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and -here there are two: every screen in the house, and one workstation. +here there are three: every screen in the house, one workstation, and the +monitoring host. + +Neither published port is private to its consumer, and the reason is the same +for both: everything already on CasaBonita shares this broadcast domain and +reaches them without the firewall seeing a packet. For 8096 that is the whole +point. For 9100 it is a residual — an unauthenticated read of this host's +filesystems, uptime and load, by the televisions — and `docs/security.md` +records it rather than the firewall rule being mistaken for a boundary it is +not. + +## Why this host is scraped, and runs no agent + +Every other machine in the estate runs Alloy and **pushes** metrics and logs to +`10.0.99.20`. This one may not. [ADR-0016] put the NAS on a segment that is +terminal outward — nothing on CasaBonita initiates anywhere — so Prometheus +reaches in over `99 → 40:9100` and scrapes instead. The convention that every +host runs Alloy exists to serve a direction; here the direction reverses, so +the tool does too. `smaug` is the estate's first scraped host. + +[#256] settled what shape that target takes, because [ADR-0040] opened a fork +in it: TrueNAS ships a metrics endpoint of its own. The answer is +`node_exporter`, for three reasons written out in full in +`stacks/observability/prometheus/targets/node.yaml` — the firewall pass already +exists for `9100`, the `host-overview` dashboard and seven rules in +`host.rules.yaml` are built on the `node_*` namespace, and a container in this +repository stays inside Dependabot, the digest pins and `make validate`, which +is [ADR-0040] decision 2's argument for this stack being here at all. + +What this does **not** buy is logs. Loki has no pull, and its ingest is +unauthenticated by [ADR-0012], so centralising this host's logs would mean a +`40 → 99:3100` rule that lets anything reaching the NAS write to the log store. +The NAS gets metrics and no logs; [#255] is where that residual lives. ## The backup split @@ -111,4 +148,6 @@ library exists**, because moving a populated library is a weekend. [#138]: https://github.com/Gerrrt/HomeLab/issues/138 [#140]: https://github.com/Gerrrt/HomeLab/issues/140 [#141]: https://github.com/Gerrrt/HomeLab/issues/141 +[#255]: https://github.com/Gerrrt/HomeLab/issues/255 +[#256]: https://github.com/Gerrrt/HomeLab/issues/256 [#413]: https://github.com/Gerrrt/HomeLab/issues/413 diff --git a/stacks/media/compose.yaml b/stacks/media/compose.yaml index 66bc929e..7db40925 100644 --- a/stacks/media/compose.yaml +++ b/stacks/media/compose.yaml @@ -168,6 +168,137 @@ services: # figure covers a start that has one to read. start_period: 60s + # --------------------------------------------------------------------------- + # node_exporter — how this host is monitored at all (#256, ADR-0016) + # --------------------------------------------------------------------------- + # THIS IS THE HALF OF THE ESTATE'S MONITORING THAT RUNS BACKWARDS. Every other + # host runs Alloy and pushes; CasaBonita is terminal-outward, so nothing here + # may initiate upward and Prometheus reaches in and scrapes instead. ADR-0016 + # chose `node_exporter` on 9100 for that, #256 confirmed it against TrueNAS's + # own endpoint, and the argument is written out in the observability stack's + # prometheus/targets/node.yaml rather than repeated here. + # + # ADR-0016 assumed a distribution package. TrueNAS has no package manager and + # an immutable root, so on this host the exporter is a container — which is + # what keeps it inside Dependabot, the digest pins and `make validate`, the + # argument ADR-0040 decision 2 made for keeping this stack in the repository. + # + # MEASURED 2026-09-17 on the pinned image, and each is a line someone would + # otherwise guess at: + # + # 1. The image DOES declare a user — `nobody` — which is the opposite of + # Jellyfin above. The `user:` line below is still written out, because + # an image's default is a thing that can change under a Dependabot bump + # and every other service in this estate names its uid. + # + # 2. `wget`, `nc` and `busybox` are present; `curl` is NOT — again the + # opposite of Jellyfin, which decides the healthcheck the other way. + # + # 3. `/-/healthy` DOES NOT EXIST HERE. It answers **404**: + # + # docker run --rm -d --name ne + # docker exec ne wget --spider -q http://localhost:9100/-/healthy + # + # Prometheus, Alertmanager, snmp-exporter and blackbox-exporter all + # serve it and are all healthchecked that way in stacks/observability. + # node_exporter is not one of them, and a healthcheck copied from a + # sibling file would mark this container unhealthy forever on + # `restart: unless-stopped` — a restart loop, for a typo in a path. + # + # 4. It boots read-only as uid 65534 with `cap_drop: [ALL]` and no tmpfs at + # all: it writes nothing. + # + # 5. Idle RSS: **3.4 MiB**, rising to 5.3 MiB once it has served a scrape. + node-exporter: + <<: *service-defaults + image: prom/node-exporter:v1.12.1@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0 + container_name: media-node-exporter + user: "65534:65534" + command: + # The host's filesystem, process table and sysfs, read through the mount + # below rather than the container's own. Without these three the metrics + # describe the container and `up` still reads 1 — the failure that looks + # exactly like success. targets/node.yaml's header carries the one-line + # check that catches it. + - --path.rootfs=/host + - --path.procfs=/host/proc + - --path.sysfs=/host/sys + - --web.listen-address=:9100 + # NO NETWORK METRICS FROM THIS HOST, deliberately, and this is the one + # line in the file worth reading twice. + # + # `/proc/net` is a symlink to `/proc/self/net` and resolves in the + # READING process's network namespace. A bridged container reading a + # bind-mounted host `/proc` therefore reports its own veth: measured + # 2026-09-17, `node_network_receive_bytes_total` named `eth0` and `lo` + # and nothing else, on a host with real NICs. + # + # That is worse than a gap. The Host Overview's throughput panels filter + # `device!~"lo|veth.*|docker.*|br-.*"`, which `eth0` passes — so the + # container's loopback-quiet veth would be charted as this NAS's network + # throughput, plausibly and silently, on the one host in the estate whose + # job is moving bytes over a network. Reporting nothing is honest; + # reporting a believable wrong number is not. + # + # The alternative is `network_mode: host`, and it costs more than it + # buys: the anchor above carries `networks:`, which compose refuses + # alongside `network_mode`, so this service would drop out of + # x-service-defaults and lose `restart`, `logging`, `init` and + # `pids_limit`; and `ports:` is silently ignored under host networking, + # so the bind address below — ADR-0012's discipline — would stop meaning + # anything while still reading as though it did. + - --no-collector.netdev + - --no-collector.netclass + - --no-collector.netstat + - --no-collector.arp + - --no-collector.sockstat + # A ceiling to catch a leak, not a measurement — the same position Jellyfin + # above and stacks/sensitive take. The number it is a ceiling OVER is + # 3.4 MiB idle, so this is two orders of magnitude of headroom; a + # node_exporter that reaches it has stopped being a node_exporter. + mem_limit: 128m + memswap_limit: 128m + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + read_only: true + volumes: + # The whole root, read-only. This is what node_filesystem_* reads, which + # is how the estate learns `erebor` is filling — HostDiskCritical and + # HostDiskWillFillIn24h in the observability stack are written against + # those series and cover this host the day it answers, with nothing added. + # + # The trade is real and belongs here rather than in a commit message: + # every path on this host that uid 65534 can read is readable from inside + # this container. It earns that by being the only way the one host with no + # logs (ADR-0016, #255) says anything about itself at all. + # + # rslave so datasets mounted after the container starts — which is every + # dataset on erebor, on every boot — are visible without a restart. + - /:/host:ro,rslave + # PUBLISHED, with a named off-host consumer as ADR-0012 asks: Prometheus at + # 10.0.99.20, through the `99 → 40:9100` pass created and verified in + # position on 2026-09-16 (build-the-nas.md §0.5). That pass is what makes + # this the estate's first scraped host. + # + # The bind address does NOT make this private. Everything on CasaBonita — + # the televisions, the game consoles — shares this broadcast domain and + # reaches 9100 without the firewall seeing a packet, exactly as they reach + # Jellyfin's 8096 above. node_exporter has no write API, so the residual is + # disclosure of this host's shape; docs/security.md records it. + ports: + - "${NODE_EXPORTER_BIND_ADDR:?}:9100:9100" + healthcheck: + # `/` and not `/-/healthy`, which 404s here — see MEASURED (3). And not + # `/metrics` either: that path runs every enabled collector, so polling it + # every 30 seconds would spend real work proving a listener is up. `/` is + # the landing page and answers 200 without collecting anything. + test: ["CMD", "wget", "--spider", "-q", "http://localhost:9100/"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + networks: media: driver: bridge diff --git a/stacks/observability/README.md b/stacks/observability/README.md index ec45dea1..70e0adb6 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 82 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 83 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/grafana/dashboards/host-overview.json b/stacks/observability/grafana/dashboards/host-overview.json index 4d8b1fe4..b376f67a 100644 --- a/stacks/observability/grafana/dashboards/host-overview.json +++ b/stacks/observability/grafana/dashboards/host-overview.json @@ -343,7 +343,7 @@ "uid": "prometheus" }, "editorMode": "code", - "expr": "count(up{job=~\".*-metrics\"} == 1)", + "expr": "count(up{job=~\".*-metrics|node\"} == 1)", "legendFormat": "__auto", "range": false, "instant": true, diff --git a/stacks/observability/prometheus/prometheus.yaml b/stacks/observability/prometheus/prometheus.yaml index cfcdd621..e47be864 100644 --- a/stacks/observability/prometheus/prometheus.yaml +++ b/stacks/observability/prometheus/prometheus.yaml @@ -199,3 +199,54 @@ scrape_configs: action: labeldrop - target_label: __address__ replacement: blackbox-exporter:9115 + + # --------------------------------------------------------------------------- + # The one host that is pulled. + # + # Every other machine in this estate PUSHES: Alloy remote_writes metrics here + # and pushes logs to :3100, and Prometheus is never told the host exists. + # `smaug` cannot. ADR-0016 put the NAS on CasaBonita, which is terminal + # outward — nothing on 40 may initiate upward — so the direction reverses and + # Prometheus reaches in over the `99 → 40:9100` pass instead. That makes this + # the estate's first scraped HOST, as opposed to a container on this compose + # network or a device behind an exporter, and it is why #256 was more than a + # line in a YAML file. + # + # WHY THE JOB IS CALLED `node` AND NOT `smaug-metrics`, which is the name the + # convention would suggest. RemoteWriteJobStale in rules/stack.rules.yaml + # selects `up{job=~"integrations/.+|.+-metrics|.+-alloy"}` and says in its own + # comment that "every directly scraped job is deliberately excluded: those + # have `up == 0` and InstanceDown". A job named `smaug-metrics` would join + # that selector by its NAME, and be covered by the rule written for the case + # `up == 0` structurally cannot see — telling whoever is paged that "the Alloy + # agent on smaug is down", which is false twice over about the one host with + # no Alloy agent by decision. Alertmanager's InstanceDown inhibit would then + # suppress the duplicate, so it would be quietly wrong rather than loudly so. + # + # The cost of the honest name is one expression: the Host Overview's "hosts + # reporting" stat counted `job=~".*-metrics"` and now counts `node` too. + # + # WHY file_sd RATHER THAN static targets, for the same reason the SNMP job + # above gives, plus one this job feels harder: prometheus.yaml is a + # SINGLE-FILE bind mount, so editing it needs the container recreated, while + # targets/ is a DIRECTORY mount and a file dropped in it is visible at once. + # Enabling this host is therefore an uncomment and a 5-minute wait, not a + # deploy — which matters because the target is written before it can answer. + # + # WHY NO relabel_configs. The snmp and blackbox jobs have them because their + # targets are arguments to an exporter and have to be turned into query + # parameters. There is no exporter in front of this one: the address in the + # file is the address Prometheus connects to, and the labels in the file are + # the labels on the metrics. targets/node.yaml is where they are argued. + # --------------------------------------------------------------------------- + - job_name: node + # 60s, not the 15s global. config.alloy collects every pushed host at 60s, + # so a finer interval here would make the one host reached across a segment + # boundary four times denser than the estate it is compared against, for no + # signal that any rule or panel asks for. + scrape_interval: 60s + scrape_timeout: 10s + file_sd_configs: + - files: + - /etc/prometheus/targets/node.yaml + refresh_interval: 5m diff --git a/stacks/observability/prometheus/rules/stack.rules.yaml b/stacks/observability/prometheus/rules/stack.rules.yaml index 6e5c70a8..5d7e4fd0 100644 --- a/stacks/observability/prometheus/rules/stack.rules.yaml +++ b/stacks/observability/prometheus/rules/stack.rules.yaml @@ -186,6 +186,71 @@ groups: the Alloy agent on {{ $labels.instance }} is down or it cannot reach this host. Metrics from it are being lost, not queued. + - alert: ScrapeTargetDisappeared + # THE MIRROR OF RemoteWriteJobStale, for the one host that is pulled. + # + # Between them those two rules were supposed to cover every collected + # target. They did not, and #256 is where the hole became visible: + # RemoteWriteJobStale catches a pushed job that stops pushing, and + # InstanceDown catches a scraped target that is PRESENT and failing. A + # scraped target that stops being a target is neither. + # + # An emptied targets/node.yaml, a YAML error that makes file_sd drop the + # file, a bad merge that deletes the block: `up{job="node"}` ceases to + # exist, so `up == 0` matches nothing and InstanceDown stays silent. + # RemoteWriteJobStale's selector cannot help — it excludes directly + # scraped jobs deliberately, and says so where it is written. So the one + # host in the estate that Prometheus has to be TOLD about was also the + # one host that could be forgotten in silence, which is the #62 and #63 + # class exactly: not a rule that misfires, a gap nothing reports. + # + # The shape is RemoteWriteJobStale's, for its reason: an instant + # selector goes absent along with the series, so "which targets were + # here and are not here now" has to be asked over a range. + # `count_over_time` reads the range and sees through staleness. + # + # THE WINDOW IS THE SAME BOUND AND THE SAME HONEST COST. 24h is how long + # this can still remember a target existed; a target restored inside a + # day resolves truthfully, and one gone longer resolves *falsely* at 24h + # having notified at least twice by then. + # + # SCOPED TO job="node" RATHER THAN EVERY SCRAPED JOB, and that is a + # narrowing worth naming. The same hole exists for targets/snmp.yaml and + # the three blackbox files. It is smaller there — those jobs have many + # targets, so one vanishing leaves siblings behind and the dashboards + # thin out visibly — and covering them means deciding what a partial + # disappearance should say, which is more than #256 needs. `node` has + # exactly one target: it disappears completely or not at all. + # + # WARNING AND NOT CRITICAL, deliberately, because of what it means. A + # host that is down produces up == 0, and InstanceDown pages `urgent` + # for it. This fires when the REPOSITORY lost the target while the host + # is very likely fine — a configuration fault, found in daylight, fixed + # by an editor. + # + # Do not put {{ $value }} in the annotations: `count by` returns the + # number of samples in the window, a number like 1440 that means nothing + # to whoever reads the notification. The same warning RemoteWriteJobStale + # carries, for the same reason. + expr: | + count by (job, instance) (count_over_time(up{job="node"}[24h])) + unless + count by (job, instance) (up{job="node"}) + for: 15m + labels: + component: stack + severity: warning + category: availability + annotations: + summary: "scraped target {{ $labels.instance }} has disappeared from service discovery" + description: >- + Prometheus was scraping {{ $labels.instance }} within the last day + and no longer has it as a target at all. This is not a host that is + down — that is InstanceDown's, and it would need the target to still + exist. Check that prometheus/targets/node.yaml still lists it and + still parses: Status → Service Discovery, and + prometheus_sd_file_read_errors_total. + - alert: LogEntriesDropped # Alloy accepted a log line and Loki refused it. There is no retry # behind a rejection, unlike the network errors diff --git a/stacks/observability/prometheus/targets/node.yaml b/stacks/observability/prometheus/targets/node.yaml new file mode 100644 index 00000000..b39b24c6 --- /dev/null +++ b/stacks/observability/prometheus/targets/node.yaml @@ -0,0 +1,103 @@ +--- +# Scraped host targets, consumed by the `node` job via file_sd_configs. +# Hot-reloaded every 5 minutes — no Prometheus restart required. +# +# THE FORK THIS FILE SETTLES (#256). ADR-0016 specified `node_exporter` on 9100 +# for the NAS. ADR-0040 then put TrueNAS on that host and reopened the question, +# because TrueNAS ships a metrics endpoint of its own — so the target shape was +# a choice nobody had made. THE ANSWER IS node_exporter, run as a digest-pinned +# container out of stacks/media/compose.yaml. Three reasons, in order of weight: +# +# 1. host-overview.json is built entirely on node_exporter's namespace. Its +# only template variable is `label_values(node_uname_info, instance)` and +# every panel matches `instance=~"$instance"`. So are seven rules in +# rules/host.rules.yaml — InstanceDown, HostDiskWillFillIn24h, +# HostDiskCritical, HostMemoryPressure, HostHighLoad, HostClockSkew and +# HostRebooted all cover this host the day it answers, with nothing added. +# The other branch means a parallel dashboard and parallel rules, for one +# host. +# 2. The `10.0.99.20 → 10.0.40.30:9100` pass already exists, created and +# verified in position above *Block access to CasaBonita* on 2026-09-16. +# The other branch is a fifth inbound rule, with all of build-the-nas.md +# §0.5 and §0.6's ordering care, verified from `morpheus` and not the UI. +# 3. A container in this repository keeps the exporter inside Dependabot, the +# digest pins and `make validate` — ADR-0040 decision 2's argument for +# keeping that stack here at all. ADR-0016 assumed a distribution package; +# TrueNAS has no package manager and an immutable root, so "install +# node_exporter" is a container on this host or it is clicks. +# +# SAY WHAT THIS WAS DECIDED ON, because it was not a comparison. This repository +# records nothing about what TrueNAS natively exposes — no port, no protocol, no +# metric names; four documents restate the same unsourced sentence. The branch +# above was declined on the pass that exists and the dashboard that exists, and +# NOT on having measured the alternative. If someone measures it later and it is +# better, that is a new decision with new evidence and not a reversal of this +# one. +# +# THE LABELS, and every one of them is load-bearing: +# +# `instance` is the WHOLE CONTRACT of this file. A direct scrape defaults it to +# the address, so without this line the Host Overview's host picker +# would list `10.0.40.30:9100` beside five hostnames and InstanceDown +# would page reading "node target 10.0.40.30:9100 is down" — one more +# lookup than a person mid-incident should have to do, which is the +# argument stacks/lab/prometheus/prometheus.yaml already makes for +# the Windows guests it scrapes. Set it to the hostname Alloy would +# have sent, so this host sorts with the others. +# `role` as elsewhere: what the thing is for. +# `vlan` which segment it is on, as targets/snmp.yaml carries it. On this +# host it is not decoration — it is the reason the scrape runs +# backwards, and the label a reader needs to understand why. +# +# NO `job` LABEL, ever. honor_labels is false, so one set here arrives as +# `exported_job` while `job` stays whatever prometheus.yaml said — and every +# rule matching on `job` would then match nothing, with the target showing UP +# and healthy the whole time. +# +# NO `host` LABEL, deliberately. Nothing reads it for scraped metrics: the +# host-keyed rules all join on textfile-collector series that carry their own, +# the Logs dashboard templates on it and this host ships no logs (#255), and +# `instance` already carries the name. A label nothing reads is a label that +# rots. + +# --- smaug --------------------------------------------------------------- +# +# WRITTEN OUT AND NOT ENABLED. The host holds 10.0.40.30 and the firewall pass +# is verified, but nothing answers on 9100 yet: the exporter is a container, the +# container needs Docker, and Docker on TrueNAS needs the apps pool — which is +# on the ZFS mirror that does not exist. Every route to node_exporter on this +# host runs through that pool, so this cannot be live before the drives land no +# matter where the exporter is put. ADR-0040's "the scrape, then the stack" is +# right about the order these are AUTHORED in and is not achievable for +# liveness. +# +# Enabling it today would mean `up == 0` from the moment the file loaded, and +# InstanceDown is `up == 0` at severity critical — routed to `urgent`, repeating +# every 4 hours, forever. ADR-0017 refuses a check that is permanently red for a +# known reason by name, and targets/blackbox-dns.yaml and the commented Windows +# job in stacks/lab make the same call. +# +# When the pool exists and stacks/media is deployed, verify FROM THIS HOST — +# which can reach 9100 and may not reach anything else on 40 — BEFORE +# uncommenting: +# +# nc -z -w3 10.0.40.30 9100 +# curl -s http://10.0.40.30:9100/metrics | grep -E '^node_(uname_info|boot_time_seconds)' +# curl -s http://10.0.40.30:9100/metrics | grep -c '^node_filesystem_avail_bytes' +# +# The third is the one that matters. `--path.rootfs` pointed at the wrong place +# produces a container's filesystems, or none, while `up` still reads 1 and the +# target shows green — so a count of 0 means every disk rule on this host is +# blind and nothing will say so. It must be the number of mounted filesystems on +# smaug, and after §3 that includes erebor. +# +# Expect NO node_network_* series at all: stacks/media/compose.yaml disables +# those collectors on purpose, because a bridged container reads its own veth +# and would chart it as this NAS's throughput. That file carries the +# measurement and the argument. + +# - targets: ["10.0.40.30:9100"] +# labels: +# instance: smaug +# role: nas +# vlan: "40" diff --git a/stacks/observability/prometheus/tests/host.test.yaml b/stacks/observability/prometheus/tests/host.test.yaml index 0ebc2f37..57e65943 100644 --- a/stacks/observability/prometheus/tests/host.test.yaml +++ b/stacks/observability/prometheus/tests/host.test.yaml @@ -176,6 +176,46 @@ tests: alertname: InstanceDown exp_alerts: [] + # --- InstanceDown: fires for a SCRAPED HOST, naming it rather than its port - + # `smaug` is the estate's first scraped host (#256, ADR-0016): CasaBonita is + # terminal-outward, so the NAS cannot push and Prometheus reaches in over + # 99 → 40:9100 instead. This case is the executable form of the claim that + # issue rests on — that a scraped host needs no new down-detection, because + # `up == 0` above has no job matcher and covers it the day it answers. + # + # It also pins the OTHER half of that claim, which lives in a different file: + # the summary reads "smaug" and not "10.0.40.30:9100" only because + # prometheus/targets/node.yaml sets `instance` explicitly. Delete that label + # and a direct scrape defaults it back to the address — the rule still fires, + # the dashboards still work, and the page quietly becomes an IP nobody can + # place at 3 a.m. This assertion is what notices. + - interval: 1m + input_series: + - series: 'up{instance="smaug",job="node"}' + values: "0x20" + alert_rule_test: + - eval_time: 10m + alertname: InstanceDown + exp_alerts: + - exp_labels: + alertname: InstanceDown + instance: smaug + job: node + component: host + severity: critical + category: availability + exp_annotations: + summary: "node target smaug is down" + description: >- + Prometheus has failed to scrape smaug + (node) for 5 minutes. + + # The pair, for the reason every pair in this file exists: a scrape that + # misses once must not page. Same fixture, inside `for: 5m`. + - eval_time: 3m + alertname: InstanceDown + exp_alerts: [] + # --- HostDiskCritical: fires below 10% free ------------------------------- - interval: 5m input_series: diff --git a/stacks/observability/prometheus/tests/stack.test.yaml b/stacks/observability/prometheus/tests/stack.test.yaml index e20a7aa9..1bbf00ab 100644 --- a/stacks/observability/prometheus/tests/stack.test.yaml +++ b/stacks/observability/prometheus/tests/stack.test.yaml @@ -26,6 +26,24 @@ # firing case here is the proof that the form actually shipped does fire, and # the two quiet cases are the proof that it has not simply been made to fire # for everything. +# +# ScrapeTargetDisappeared was added by #256 and is the mirror of that rule for +# the one host this estate PULLS. Its four cases were mutation-tested on +# 2026-09-17, against a copy of the rules, each mutation asserted to apply +# before being run — #189's trap is a mutation that silently fails and looks +# exactly like a rule that cannot be broken. All four were killed: +# +# ScrapeTargetDisappeared unless -> and; [24h] -> [5m]; +# drop the job="node" selector; for: 15m -> 0m +# +# The last one SURVIVED on the first attempt, and that is worth recording rather +# than quietly fixing. Three cases and a firing assertion at 60m cannot tell +# `for: 15m` from `for: 0m` — by 60m the alert has been firing for 25 minutes +# either way. The `eval_time: 40m` pairing under the firing case is what kills +# it: the series ends at 30m, ages out of the instant selector at 35m, and is +# pending until 50m. A green run without that assertion was proving less than it +# appeared to. + rule_files: - ../rules/stack.rules.yaml @@ -180,6 +198,92 @@ tests: alertname: RemoteWriteJobStale exp_alerts: [] + # --- ScrapeTargetDisappeared: fires when the target stops being a target -- + # The mirror of the RemoteWriteJobStale case above, and the same fixture + # shape for the same reason: the series arrives every minute and then stops. + # What differs is the cause, and it is the whole point of the rule — a pushed + # job stops because the agent died, a scraped job stops because the TARGET + # WAS REMOVED. From the series there is no telling those apart, which is why + # the selector and not the shape is what separates the two rules. + - interval: 1m + input_series: + - series: 'up{job="node",instance="smaug"}' + values: "1x30" + alert_rule_test: + - eval_time: 60m + alertname: ScrapeTargetDisappeared + exp_alerts: + - exp_labels: + alertname: ScrapeTargetDisappeared + instance: smaug + job: node + component: stack + severity: warning + category: availability + exp_annotations: + summary: "scraped target smaug has disappeared from service discovery" + description: >- + Prometheus was scraping smaug within the last day and no longer + has it as a target at all. This is not a host that is down — + that is InstanceDown's, and it would need the target to still + exist. Check that prometheus/targets/node.yaml still lists it + and still parses: Status → Service Discovery, and + prometheus_sd_file_read_errors_total. + + # Gone, but not for long enough yet. The last sample is at 30m and the + # instant selector stops resolving it five minutes later, so the alert is + # pending from 35m and firing from 50m. This is the case that makes + # `for: 15m` load-bearing: without it the rule fires here, and mutating + # the `for` to 0m would otherwise have left every test in this file green. + - eval_time: 40m + alertname: ScrapeTargetDisappeared + exp_alerts: [] + + # --- ScrapeTargetDisappeared: quiet while the target is being scraped ----- + # The production state, once smaug answers. Without this case the rule would + # pass its test while firing continuously for a target that is perfectly fine. + - interval: 1m + input_series: + - series: 'up{job="node",instance="smaug"}' + values: "1x70" + alert_rule_test: + - eval_time: 60m + alertname: ScrapeTargetDisappeared + exp_alerts: [] + + # --- ScrapeTargetDisappeared: quiet for a target that is present and down -- + # smaug powered off, or the firewall pass moved below the deny. The target + # still EXISTS, so up is 0 rather than absent — this rule must stay silent and + # InstanceDown must be the one that fires. The two partition the failure and + # do not overlap, which is the same claim the cAdvisor case above makes for + # the pushed half. Held at 0 for the whole range rather than written as a gap, + # because it is the presence of the series and not its value that this rule + # keys on. + - interval: 1m + input_series: + - series: 'up{job="node",instance="smaug"}' + values: "0x70" + alert_rule_test: + - eval_time: 60m + alertname: ScrapeTargetDisappeared + exp_alerts: [] + + # --- ScrapeTargetDisappeared: quiet for a PUSHED job that goes silent ----- + # This is the case that pins the selector rather than the expression. The + # fixture is byte-for-byte the firing case of RemoteWriteJobStale above, and + # that rule is expected to fire on it — this one must not, because widening + # `job="node"` to every job would cover remote_write jobs twice and page for + # the same outage under two names. Without this case the selector could be + # dropped entirely and every other test here would still pass. + - interval: 1m + input_series: + - series: 'up{job="oracle-metrics",instance="oracle"}' + values: "1x30" + alert_rule_test: + - eval_time: 60m + alertname: ScrapeTargetDisappeared + exp_alerts: [] + # --- LogEntriesDropped: fires on loss that keeps happening --------------- # A counter climbing steadily for well over the hour. That is loss that is # still going on, which is the only thing this rule is for.