Add the scrape path for a host that pushes nothing

[Gerrrt]

Every host in this estate is monitored by pushing. prometheus, oracle
and Saruman all run Alloy agents that remote_write to 10.0.99.20:9090 and
push to :3100; the only scraped things are the six containers on the compose
network, plus SNMP devices and blackbox endpoints behind their exporters.

ADR-0016
breaks that for the NAS on CasaBonita (40), and had to: a host on a terminal
segment must not initiate upward, so Prometheus reaches in and scrapes it over
99→40:9100 instead. That makes smaug the estate's first scraped host — the
first target that is a machine rather than a container or a device behind an
exporter.

That is more than a line in a YAML file, which is why it is not just a checkbox
on #102.

Note

Updated 2026-09-17. This is no longer blocked, and it grew a fork.
The host is smaug, not zion
(ADR-0038),
and it runs TrueNAS, not Ubuntu Server
(ADR-0040).
#102 is closed and the tracker is
#413. Moved to Wave 0.

The fork, which is the first thing to settle

This issue specifies node_exporter on 9100. TrueNAS ships its own metrics
endpoint, so the target shape is a choice nobody has made, and
build-the-nas.md
§8 names it as this issue's to settle. It is not a free choice in either
direction:

• The 99 → 40:9100 pass already exists — created and verified in position
  on 2026-09-16. If the answer is node_exporter, the firewall work is done.
• If the answer is TrueNAS's own endpoint on another port, that is a fifth
  inbound rule, which means the ordering care §0.5 and §0.6 document — above
  Block access to CasaBonita, verified from morpheus rather than from the
  UI — applies again.
• The two do not carry the same metric names, and the host-overview
  dashboard is built on node-exporter metrics.

What it needs

• Settle the fork above, and say which in the file's header.
• A node job in prometheus.yaml with file_sd_configs, following the
  shape the snmp and blackbox jobs already use — the comment above the
  SNMP job explains why file_sd rather than static targets, and the same
  reasoning applies.
• prometheus/targets/node.yaml, with a header in the house style: what
  each label means and who reads it. blackbox.yaml's header is the model.
• Decide the label set. snmp.yaml carries device, role and vlan;
  whatever this file carries has to work with the existing host-overview
  dashboard, which is built on node-exporter metrics and currently sees
  only what Alloy pushes.
• Alert rules. A pushed host that dies stops writing and existing rules
  notice; a scraped host that dies produces up == 0, which is a different
  signal and may not be covered. Check whether anything currently fires on
  up == 0 for a non-container target — if it does not, this host would
  go dark silently, which is the exact class of fault
  #63 and
  #62 were.
• A promtool test rules case for whatever rule that turns out to be. The
  repo has unit tests for exactly one rule today, and the reason it has them
  is that syntax-checking cannot tell you an expression can never be true.
• Consider whether docs/runbooks/add-monitored-device.md grows a second
  path, or whether a scraped host is different enough to want its own.

The ordering trap

The scrape target cannot be written before the firewall rule exists, and the
firewall rule should not be created before the host answers on that address —
ADR-0016 declined to create it for exactly that reason: a pass to an address
with nothing behind it is a rule nobody can test. So the sequence is hardware →
rule → target → alert, and each step is testable only after the one before it.

The first two steps are done. smaug has held 10.0.40.30 since
2026-09-16 and the passes are created and verified, so the target and the alert
are what is left — and neither waits on the ZFS pool, which is why this left
Wave 3. Writing the YAML early was the risk; the risk now is only writing it
against the wrong endpoint, which is what the fork above is for.

Found writing ADR-0016 for #95.

[Gerrrt]

Two of the checkboxes here are answerable without the hardware, and both answers
reduce the work. Recording them rather than writing the target file, because
this issue is explicit that doing that early is the trap: "writing the YAML
early gets a target that has never been scraped, which is how blackbox.yaml
first shipped probing the wiki over the wrong scheme."

"Check whether anything currently fires on up == 0 for a non-container target"

It does, unconditionally. InstanceDown in host.rules.yaml is:

expr: up == 0
for: 5m

No job selector, no instance selector. It covers every scraped target that
exists and every one that ever will, including a node job for zion. So the
worry this checkbox raises — "this host would go dark silently, which is the
exact class of fault #63 and #62 were" — does not apply.

It is also already unit-tested, and the fixture happens to use
instance="oracle:9100", which is the scraped convention rather than the pushed
one. So the next checkbox — "a promtool test rules case for whatever rule
that turns out to be" — is satisfied by a test that already exists.

The partition is clean and worth stating, because it is not obvious from either
rule alone: scraped targets are covered by InstanceDown (up == 0),
pushed ones by RemoteWriteJobStale (the series ageing out), and
stack.rules.yaml's header explains why up == 0 structurally cannot see a
pushed agent die.

"Decide the label set"

The constraint is narrower than it looks, and it is the host-overview
dashboard rather than the rules. Its host picker is:

label_values(node_uname_info, instance)

and every panel filters instance=~"$instance". So instance is the only label
that has to be right.

What the estate does today:

pushed  instance=oracle       job=oracle-metrics
        instance=Saruman      job=Saruman-metrics
        instance=prometheus   job=prometheus-metrics

snmp    instance=10.0.99.1    device=morpheus  role=firewall        vlan=99
        instance=10.0.99.10   device=mjolnir   role=ups             vlan=99
        instance=10.0.30.10   device=shiva     role=hypervisor-bmc  vlan=30
        instance=10.7.7.2     device=neo       role=switch          vlan=lan

Alloy sets instance to the bare hostname. A node job left to itself sets
it to zion:9100, so the dashboard's host list would read
oracle, prometheus, Saruman, zion:9100 — the odd one out, in every panel
title, forever.

So: relabel instance to the bare hostname, and carry device, role and
vlan in targets/node.yaml the way snmp.yaml does. That satisfies the
dashboard and matches both existing conventions rather than inventing a third.

What is still genuinely blocked

The target file and the firewall rule, in that order and after the hardware —
the sequence this issue already sets out. Nothing above changes that; it just
means the person who picks this up after #102 is choosing between
already-answered options instead of rediscovering them.

🤖 Generated with Claude Code

[Gerrrt]

Still blocked on the hardware, and the 2026-09-06 comment above has the two
big answers right (InstanceDown is unscoped and already tested; relabel
instance to the bare hostname). Three things found on 2026-09-08 that the
person who picks this up should know, none of which writes the YAML early.

The host-overview dashboard has a tile that will lie

The host picker is label_values(node_uname_info, instance) and every panel
filters on $instance, so a scraped zion will appear there once
instance is relabelled. But the Hosts reporting stat is

count(up{job=~".*-metrics"} == 1)

— it counts pushed jobs by name. A node job matches nothing in it, so the
dashboard will list zion in the picker and every panel while the tile says
one host fewer than the picker does. Either the job is named to match
(node-metrics?) or the tile's regex grows a branch; decide it with the job
name, not after.

InstanceDown pages urgent, and it will page for zion

The rule is unscoped, critical, for: 5m, and Alertmanager routes critical to
the urgent receiver. A scraped NAS on a low-trust segment inherits a
phone-buzzing page for a reboot by default. That may be the right call — it is
the only host that is scraped, so it is the only one this rule can see — but
it is a choice, and the alternative (a node-job-scoped warning with a longer
for, and InstanceDown excluding the job) is one rule and one test.
stack.rules.yaml's header already explains the pushed/scraped partition; the
choice belongs beside it.

The shape already exists in the sibling stack

stacks/lab/prometheus/prometheus.yaml carries a commented-out windows
scrape job with explicit per-target instance and role labels, and states
why it lands commented: a job pointed at machines that do not exist yields
up == 0 on every one, and InstanceDown fires permanently from the moment
the file deploys. That is this issue's ordering trap from the other side, and
the file is the template — including the sentence about why instance is set
explicitly ("10.0.30.53:9182 has fallen back to its own CMOS clock" is one
more lookup than a person mid-incident should need).

Also: targets/blackbox-dns.yaml is the file_sd precedent for "every target
commented out until #102 builds the box", so a targets/node.yaml with a
header and no live target is within house style if it is ever wanted before
the hardware — but the sequence in the body (hardware → rule → target →
alert) still holds, and nothing here changes it.

🤖 Generated with Claude Code

[Gerrrt]

Unblocked, and moved to Wave 0. smaug has held 10.0.40.30 since
2026-09-16 and the 99 → 40:9100 pass is created and verified in position —
see the build comment on #413.
The ordering trap this issue describes has had its first two steps satisfied;
the target and the alert are what is left, and neither waits on the ZFS pool.

Body updated with the fork ADR-0040 created: node_exporter on 9100 against
TrueNAS's own metrics endpoint. Settle it with
#483, which is the same host,
the same rule and the same question asked about SMART instead of about load
averages — two independent decisions about one scrape path is how they end up
inconsistent.

[Gerrrt]

Delivered, and closing on it.

• The fork is settled: node_exporter, run as a digest-pinned container out of stacks/media/compose.yaml, with the three reasons in the header of stacks/observability/prometheus/targets/node.yaml — and the admission that it was decided on the dashboard that exists, not on having measured TrueNAS's endpoint. feat(monitoring): settle the scrape path for smaug, the first scraped host #508.
• The node job with file_sd_configs at 60 s is in prometheus.yaml; the target went live 2026-09-19 after the three checks from the monitoring host (36 filesystems, not 0 — the --path.rootfs trap). feat(nas): the drives are in, erebor exists, the stack runs and the scrape is on #522.
• InstanceDown and the scraped-target-missing rule are unit-tested against smaug in tests/host.test.yaml and tests/stack.test.yaml.
• add-monitored-device.md grew the second path, including the job-name trap raised above.

What this issue handed to #483 — SMART and patch state for a host the exporter cannot push from — is recorded there, with the constraint this decision created: the exporter is unprivileged, read-only and cap-dropped, so SMART is not free off the back of it.

Two sentences in docs/roadmap.md still say the target is commented and the trays are empty; the docs PR from this pass fixes them.