feat(monitoring): settle the scrape path for smaug, the first scraped host

[Gerrrt]

What changed

Settles the fork #256 inherited from ADR-0040 and builds the scrape path for smaug — the estate's first host that is pulled rather than pushing.

The fork, settled: node_exporter on 9100, as a digest-pinned container in stacks/media. The argument lives in prometheus/targets/node.yaml's header, where the issue asked for it:

1. host-overview.json is built entirely on the node_* namespace — its only template variable is label_values(node_uname_info, instance) — and seven rules in host.rules.yaml come free with it. The other branch means a parallel dashboard and parallel rules, for one host.
2. The 10.0.99.20 → 10.0.40.30:9100 pass already exists and is verified in position. The other branch is a fifth inbound rule with all of §0.5/§0.6's ordering care.
3. A container here stays inside Dependabot, the digest pins and make validate — ADR-0040 decision 2's argument for this stack being in the repo at all. ADR-0016 assumed a distribution package; TrueNAS has no package manager and an immutable root.

Stated honestly in the file: the TrueNAS branch was declined on the pass that exists and the dashboard that exists, not on a comparison. This repository records nothing about what TrueNAS natively exposes — no port, no protocol, no metric names; four documents restate the same unsourced sentence. If someone measures it later and it is better, that is a new decision with new evidence.

The target ships commented out, and the issue's premise that it "doesn't wait on the ZFS pool" turns out to be wrong: every route to an exporter on TrueNAS runs through Docker and therefore through the apps pool. A live target means up == 0 and InstanceDown paging urgent every 4 hours forever — ADR-0017's refusal, and the call blackbox-dns.yaml and the lab's Windows job already make. build-the-nas.md §6.1 is the one-line uncomment that finishes it.

Why

On the alerting question the issue asked to check: InstanceDown is up == 0 with no job matcher, so a scraped host is covered for free — and host.test.yaml already tested it against a node_exporter-shaped series. The real gap was the other half: a target that stops being a target. An emptied or unparseable targets/node.yaml makes the series vanish rather than fall to 0, so up == 0 matches nothing, and RemoteWriteJobStale excludes directly scraped jobs by design. That hole was exactly the width of the one host Prometheus has to be told about — the #62/#63 class. ScrapeTargetDisappeared closes it.

The job is node, not smaug-metrics. The convention would suggest the latter, but it would enrol a pulled job in RemoteWriteJobStale, whose notification says an Alloy agent stopped pushing — false twice over about the one host with no Alloy agent by decision, and quietly so, since the InstanceDown inhibit would suppress the duplicate. Cost of the honest name: one dashboard expression.

Two measurements changed decisions rather than confirming them:

• /-/healthy returns 404 on node_exporter. Prometheus, Alertmanager, snmp-exporter and blackbox-exporter all serve it and are all healthchecked that way here. A healthcheck copied from a sibling file would have marked the container unhealthy forever on restart: unless-stopped — a restart loop, for a path.
• A bridged container reads its own netns. node_network_receive_bytes_total named eth0 and lo on a host with real NICs. eth0 passes host-overview's device!~"lo|veth.*|docker.*|br-.*" filter, so the container's quiet veth would have been charted as this NAS's throughput — on the one host whose job is moving bytes. Those collectors are disabled: reporting nothing is honest, reporting a believable wrong number is not. (This is likely already true of every Alloy host in the estate; filed separately, not fixed here.)

Mutation testing caught a weak test. for: 15m -> 0m survived the first round — a firing assertion at 60m cannot tell it from for: 0m. The eval_time: 40m pairing is what kills it. Recorded in the test file rather than quietly fixed.

Also folded in, because this change makes them wrong or found them wrong: observability.md said Prometheus scrapes nine jobs (eleven before this, twelve after), and dependabot.yml had no /stacks/media entry, so Jellyfin's pin was already rotting unnoticed.

Blast radius

• No change to network segmentation or firewall rules — the 9100 pass already existed and is used, not created
• No new port published to a VLAN that could not already reach the service — 9100 is published on a host that is not deployed; when it is, the segment could already reach it, which is recorded in docs/security.md as a residual
• Nothing deploys from this PR. stacks/media is undeployed and the scrape target is commented out, so the running stack gains a node job with zero targets
• prometheus.yaml is a single-file bind mount — applying this on the monitoring host needs make up, which runs check_mounted_config.py --fix

Verification

• make validate passes (3 skips, all "not the deployment checkout")
• promtool test rules — 11 files, all SUCCESS, including 4 new ScrapeTargetDisappeared cases and a smaug InstanceDown case
• All four mutations killed: unless→and, [24h]→[5m], drop the job="node" selector, for: 15m→0m
• amtool config routes test — 8 assertions, unchanged; warning+availability was already in use
• check_compose_health.py --probe execs wget inside the pinned image
• check_docs.py clean for every count this PR touches (83 rules, 63 tested, 141 panels)
• Docs updated

Note

An earlier push of this branch showed a red Validate configs. That was inherited from main, which was failing at 95b7621 because two PRs each landed an ADR numbered 0041 and the README.md ADR/runbook counts went stale in the same merges. Fixed on main by #503, and this branch has been rebased onto 107256f, where check_docs.py is clean. Nothing in this PR was ever implicated.

Uses Refs rather than Closes: all seven of the issue's checkboxes are done, but the scrape is not live until the pool exists and the four lines are uncommented.

🤖 Generated with Claude Code