Roadmap

Open work, extracted from the per-VLAN task lists that used to live inside the inventory. Ordered roughly by how much it matters.

The tracking lives in Issues. This file is the shape of the work — what is outstanding, and why it is in this order. Whether a thing is started, blocked or done is on its issue. Two places holding the same checkbox is how a checkbox stops being true, and this file has already been wrong about the switch answering SNMP and about the history purge.

Detail that used to live here — the trap in the MokerLink walk, the exact order to bring the UPS back, the isolation question ifrit defers — moved to the issues intact. Nothing was summarised away.

Everything still to buy

The one list. It exists because purchases kept appearing one at a time in issues, ADRs and runbooks, and the person paying for them found out about each by surprise — the sensitive tier's host had no line anywhere until one box had been bought for two jobs (ADR-0034). Compiled on 2026-09-09 from every ADR, runbook, open issue and document in this repository. The rule from here: nothing enters the first list without a decision the operator made, and a PR that implies a purchase edits this section in the same commit.

Revised 2026-09-10 against a costed shopping list, which is what the rule above is for: it found four things this section had wrong or missing — smaug's boot disk was never listed, two deferred items had been decided, ADR-0017's 32 GB is not what the candidate machines ship with, and the Never line forbade a battery it was never written about.

2026-09-11 emptied the hardware half of this list. The two NAS drives, the boot SSD, the bracket and the double-sided tape that carry it in the TS150's 5.25" optical bay, and the I226 card the firewall rehearsal needs were all bought that day — the table goes from seven rows to four. Nothing left on it is a part for a machine: two licences, a switch, a drive kept at another address and a battery. So neither #413 nor the rehearsal in #404 is blocked on money any more — each is waiting on parts landing and a build. The tape was never a row and does not get one now; it is part of the bracket purchase, recorded in hardware.md with it, because the rule is that a purchase edits this section and not that only listed purchases do.

2026-09-13: the switch is bought. A used MikroTik CRS326-24G-2S+RM, recorded in hardware.md — bought for the TLS management interface, which is the one argument the correction on #444 left standing, with #84 riding along. The table goes from four rows to three, and nothing on it now touches the rack: two licences, a drive kept at another address and a battery. What closes #84 and ADR-0018's residual is a cabling window, not money — neo carries every VLAN, so the swap waits for a rack visit it can share.

2026-09-13, later the same day: the battery is bought. An A1437 cell for prometheus, new, recorded in hardware.md — the consumable the exception to Never below was written for, and the one row on this list whose case was safety before availability (#454). The table goes from three rows to two, and neither of them is a part: two licences and a drive kept at another address. What closes #454 is not the cell landing but the fit and the test after it — charge_full at or near design, a low cycle count, and mains pulled with the host staying up — which replace-the-laptop-cell.md names. This paragraph pointed at fit-the-ups-battery.md until 2026-09-18 — the rack pack in mjolnir, not this cell — which is the same wrong pointer #506 corrected in the alert's own description. oracle's cell, the one that measures worse, stays unbought and second in line, as that issue ranked it.

2026-09-18: the cell is fitted, and the issue stays open. The A1437 went into prometheus the day it was delivered; the host was down from 14:53 to about 18:12 UTC. The new pack reads charge_full 6.889 Ah of 6.8 Ah design at one cycle, and both of its design figures differ from the old pack's — an aftermarket cell reports its own, which is the only proof of a changed part on a machine that exports no serial number. The buy table does not move: the cell left it on 2026-09-13, and fitting is not buying. What is still owed is the mains pull on the charged pack and the runtime it measures, so #454 stays open and this gets no Done entry.

Buy these, and the estate as decided is fully bought:

Item	For	Decided by	When it is needed
Two Windows 11 Pro keys	The lab domain's two endpoints; the four servers are free evaluations	ADR-0029, #414	When the domain build reaches the endpoints, not before
An external drive kept at another address	ADR-0023's off-estate copy of the household's photographs and documents. Buying it is the decision that ADR was waiting on	ADR-0023, #455	Before ADR-0022's first trigger, so the decision is not made under pressure

One more, later, and it is the last: ifrit, the range host — a quiet SFF box, NVMe, two socketed DIMM slots with 32 GB fitted, one NIC (ADR-0017, #421). Gated on the domain being built; it is the last purchase on this list, not the next.

32 GB is a spec the candidate machines do not meet as shipped, which was found by pricing them rather than by reading the ADR. The SFF boxes in the class ADR-0017 describes — ThinkCentre Tiny, EliteDesk Mini and their relatives — ship with 16 GB in two slots at this price. So a SO-DIMM kit is part of that purchase and not a later contingency, and the bullet below that used to say otherwise has been corrected. The model, the CPU and the disk capacity are still chosen at the till and recorded in hardware.md afterwards, per the ADR — nothing is named here before it is bought.

Already paid for and on hand, as of 2026-09-17: the ProDesk 600 G4 — trinity, the tier's host and the firewall's spare hardware (#404); the TS150 NAS and its boot SSD with the bracket and tape that mount it in the optical bay (#413); the 2 TB USB drive that becomes the photo library's disk on trinity; two SM863a SSDs for Saruman (#418); and the removable medium the second age recipient lives on.

Already paid for and still moving: the I226 card on its M.2 adapter that a restore onto trinity needs (#404) and the pfSense installer stick — between them, the whole of what the rehearsal now waits on; the two Exos X20 18 TB drives for smaug's mirror (#413); the CRS326 that replaces neo (#444). The A1437 cell for prometheus was on this list until 2026-09-18 and is on neither now: it is fitted (#454). The two SFF drive trays for Saruman left it the same day, and took the SM863a pair off the paragraph above with them — see the 2026-09-18 note below. These two paragraphs track money that has not yet become a working part, and that has.

2026-09-17: the trays were bought on 2026-09-11 and are only now written down. They were paid for in the same sitting as the I226 card, the Exos pair and the boot disk's bracket, and unlike every one of those they got no entry in hardware.md and no line here. This section's rule is that a purchase is recorded when the money is spent; six days of silence is that rule failing, and it was caught by reading a delivery notice for a different issue rather than by anything this repository does. Nothing here checks the list against what was actually bought, which is the gap the omission found.

Four things crossed from the second paragraph to the first: the TS150, the ProDesk, the boot SSD, and the bracket and tape that mount it. All four were recorded here as 2026-09-15; the ProDesk's is now 2026-09-14, from the carrier's own notice, and the other three are unaudited. They were written in the same commit, that evening, and dated by UTC after it had turned over — so the same day may be out by one for each of them. Checking is #413's, not this section's, and nothing here depends on which day it was.

2026-09-17: the two SM863a SSDs are on hand. They were delivered 2026-09-11 and carried above as still moving for six days. Delivery is recorded against the order, which was one order for the pair; that both drives were in it is not separately proved and is #418's to confirm at the bays.

2026-09-18: the trays arrived and both SSDs are in the bays. Confirmed at the bays, as the paragraph above asked: the iLO reports two new drives at Port 1I Box 1 Bay 3 and Bay 4 from 20:11 UTC, with different serials, so the pair was a pair. Fitted is not finished — the drives are in no logical drive, nothing has moved to them, and the measurement the purchase exists for has not been taken; #418 carries that. Neither the SSDs nor the trays appear in either paragraph above now: they are parts, and hardware.md has them.

No row on the table above moves — arriving is not buying, and this section tracks money. Nothing was racked, fitted or built by it either. What it changed is one sentence: the two builds stopped waiting on a van for their machines, and wait only for the parts named above.

What those boxes actually are was read off them on 2026-09-15 and 2026-09-16, which this paragraph listed as still owed: the TS150's machine type-model, serial, CPU and the single DIMM in four slots; the boot SSD's model and serial and its smartctl -a taken before the install rather than after; and trinity's 512 GB SSD, which is an M.2 stick, so there is no 2.5" drive carrier and the second M.2 slot is free for the I226 card that was the thing this sentence was worried about (#404). hardware.md carries each of them as a reading now rather than as a debt.

Only if a decision is taken, and none is pending — these are not on the list, and each names what would put it there:

A dedicated firewall cold spare: deferred by ADR-0034 until the tier holding real data makes an hour of firewall downtime unacceptable. There is one ProDesk, not two — trinity is the tier's host and the firewall's spare hardware, which is what ADR-0034 decided and what this bullet is the deferral of. The reserved name zion belongs to the box this bullet would buy, and no host answers to it (ADR-0038).
A Zigbee or Z-Wave coordinator for Home Assistant: only if a device needs one, and nothing on Skids does today — Ring is cloud, Hue has its own bridge, the assistants are Wi-Fi (#134).
More memory for ifrit than ADR-0017's 32 GB: only if 32 GB proves short. Reaching 32 GB is above, in the purchase itself — this bullet used to read "a memory kit for the second slot, only if 32 GB proves short", which assumed the box arrives with 32 GB fitted and none of them do.
A Coral TPU and RTSP cameras: declined with Frigate (ADR-0032).
Plex Pass: ADR-0016 builds Jellyfin alone and adds Plex only if a client on CasaBonita turns out to need it — a test nobody has run. Its headline feature is hardware transcoding, and hardware.md already records that the TS150's Quick Sync is what #138 needs, so Jellyfin has it for nothing.

Two things left this list because the decision got taken: the MokerLink replacement, now #444, and off-estate storage, now #455. Both moved into the first table above; the switch has since left it by being bought, on 2026-09-13, and the drive is still there. Neither is deleted from the record — moving up is what taking the decision looks like, and leaving is what buying looks like.

The switch's row was narrower than the bullet it replaced, and deliberately. That bullet named #84, the switch half of #85 and ADR-0018's residual as three things one purchase would close; ADR-0036 has since removed the middle one — the switch answers v3 on the wire and was never the blocker. A purchase justified by three residuals when one of them has gone is the shape this section exists to prevent, so it was bought for the TLS management interface, and #84 rides along.

Never, and the documents say so: anything to make prometheus or oracle faster or bigger, and any disk or memory on account of Wazuh — ADR-0030 sizes it to what Saruman has. A 2012 MacBook running the whole observability stack is the point, not a problem to spend money on.

The one exception, and it narrows this line rather than reversing it: a consumable whose failure is a safety or availability event is not an upgrade. prometheus's battery is the worked example (#454) — the estate's mains-cut path depends on it, host.rules.yaml has measured it since 2026-09-12 — 94 % of design after 108 cycles, on the cell that was there then — and the failure mode of a thirteen-year-old cell is a fire on a shelf. That cell was bought 2026-09-13 and fitted 2026-09-18, so this is the exception being exercised rather than restated; the new pack reads 101 % of its design capacity at one cycle, and the runtime it was bought for is still unmeasured. oracle has the same kind of cell and it measures worse, 72 %, so HostBatteryHealthLow fires for it first; it stays second in line only because prometheus is the host whose death is the estate going blind. The cell for prometheus was bought on 2026-09-13; nothing else about either machine is.

Security

#229 The switch LAN still carries pfSense's stock Default allow LAN to any. 10.7.7.0/24 reaches every VLAN; network.md said "Nothing". Bounded by that segment holding only the switch — which is also the device whose SNMP agent serves GETBULK to any short community (#84) and stays on v2c (#85). Lower risk than #228: getting it wrong costs SNMP polling of neo, which is monitored.
#84 Retire the MokerLink switch's previous SNMP community. The rows were overwritten and the switch rebooted on 2026-09-12, and the measurement that followed changed the issue: neo serves GETBULK to any community of sixteen characters or fewer without checking the table, which is what every earlier sighting of the stock public and private answering had been — and every STILL ACCEPTED for the old one. Over GET, which it does check, both stock strings and a junk string are refused; the previous community's row is unverified rather than retired, because the string was not to hand in the window and is not recoverable — it was the shared value purged from history. So this closes by the hardware leaving, not by a measurement. snmp-verify.sh probes with GET since that date and sends junk strings over both PDUs weekly, WARN for the switch. Accepted residual, larger than before and recorded in SECURITY.md; what closes it is the replacement switch — bought 2026-09-13, delivery estimated 2026-09-23 — and the window that racks it, not another window on this one. Nothing further is owed on the MokerLink: the overwrite §2.5 prescribes was done on 2026-09-12 and persisted. → runbook
#85 Move to SNMPv3 authPriv where the hardware supports it. Decided by ADR-0036: per poll, by where the poll travels, and mixed on purpose. The iLO first — its poll is delivered into the lab segment, layer-2 adjacent to the attack VM, so its community is the one an adversary is meant to be able to try for — then the UPS card on the same procedure. The firewall stays on v2c because bsnmpd is the only daemon that serves the pf MIB and pfSense writes no v3 user for it; that was the issue's "three can", checked on the box on 2026-09-09, and the switch was never what blocked it. The switch stays on v2c and its UI gets checked once for a v3 user page — its agent answers v3 on the wire, which ADR-0018 did not know. The tooling is done: a device's version and key names come from its auth block in generator.yaml, and snmp-verify.sh speaks v3. What is left is the device side, one at a time, device first — runbook §4.
#182 Authenticate the Prometheus and Loki ingest ports. Both are published and unauthenticated, so anything that can route to 10.0.99.20 can read every metric and log line, inject metrics and delete log ranges. They stay published because oracle's agent pushes to them and has no other path, which is why #70 could close Alertmanager and not these. Firewall default-deny is the whole control. Accepted residual, recorded in SECURITY.md.
#103 Give ADR-0008's SSO deferral an expiry. Answered by ADR-0022: the deferral ends on a state rather than a date — the first real secret, photo or document in the sensitive tier, any reachability from outside the house, or a third account holder, whichever comes first. At the first of those a decision gets recorded. Re-accepting is allowed; arriving at the same place by never looking is what the expiry removes.

Writing it turned up that ADR-0008's substitute for SSO does not exist for half the tier. Per-application TOTP is available on Vaultwarden, Paperless-ngx and Home Assistant, and on none of Grafana, Immich or AdGuard Home — Grafana OSS has no MFA in any edition, Immich's upstream has declined it and points at OAuth, and AdGuard has one password-only admin. Grafana is the only one of the six deployed, so the thing ADR-0008 offered in place of SSO has never been available here, and for those three an identity provider is the only route to a second factor rather than a heavier alternative to one. What is outstanding belongs to #102: TOTP enrolled at first login on the three that can carry it, and the mini PC's disk encryption decided at build time rather than inherited from prometheus — a vault behind one factor on an unencrypted disk is not the bet SECURITY.md accepted for a metrics dashboard.
#122 Settle who reaches the family's credentials, photos and documents when the estate is down. Answered by ADR-0023, and the answer is that no sensitive-tier service has to stay reachable — one mini PC cannot be made highly available, and every way of pretending otherwise adds components that can take it down. The constraint is on the path instead: nothing the household needs in an emergency may have the estate on its only route. Four classes, falling due on ADR-0022's triggers — the household's own credentials recoverable without Vaultwarden and opened once from the other person's device; an encrypted off-estate copy of Immich and Paperless-ngx whose staleness is visible; nothing on the break-glass card depending on a certificate this estate issues; and nothing physical operable only through Home Assistant.

Writing it turned up that off-host is not off-estate. oracle holds the firewall export precisely so it is not on the machine it protects (ADR-0015, #92) — and it is on the same VLAN, rack, power feed and room as the mini PC would be, so every failure this issue is about reaches both in one event. Two smaller findings went the other way: ADR-0010 already keeps name resolution alive when the mini PC dies, which is a dependency of every out-of-estate path and is met by accident of a decision made for other reasons; and ADR-0011's break-glass card already says "where credentials are", a sentence that goes false the day Vaultwarden holds anything real. Nothing here is built — the preconditions land on #131, #132, #133 and #134.

Monitoring

#114 Set memory limits on the six services. Done 2026-09-04, on seven — blackbox-exporter joined the stack after the issue was written. The gate this entry named was explaining the 2026-08-29 hour in which loki and alloy both peaked, and that is what unblocked it: Loki's RSS tracks the ingest rate through it minute for minute — ~100 MiB at 3 lines/s until 06:15, 461 MiB the minute 5,501 lines/s arrived, 1,015 MiB at 06:29, and back to 107 MiB nine minutes after the flow stopped. The source is #286: make backup pipes each volume's gzip stream to a container's stdout, and Alloy tails every container on the socket, so the archives come back through the log pipeline. The compactor cycling and the 400s in the logs are a Loki shedding load, not the cause — which is the reading that has to be got right, because it is the difference between a ceiling sized for a self-healing blip and one sized for a recurring bug. It recurs: 577 MiB on 09-04. The other half was the method. 3x every peak did not fit the machine because it was being read off the working set, which includes reclaimable page cache: alloy's 512 MiB peak is 221 MiB of anonymous memory and the rest cache from walking /rootfs, and mem_limit bounds a cgroup the kernel reclaims cache from before it kills anything. Sized from container_memory_rss instead, with the multiplier chosen per service rather than flat, the seven limits sum to 5120 MiB against 7816 MiB of RAM. loki gets the loosest ratio and the largest number for the reason above; the issue had listed it among the low-risk services to start with, on a 152 MiB peak from a six-hour window. memswap_limit equals mem_limit everywhere, so the stack cannot page into the unencrypted /swap.img — left unset it defaults to twice mem_limit, which would have made that exposure worse rather than better. Enforcement and detection stay uncoupled, which is the whole reason #63 is a separate issue: ContainerHighMemory still measures against machine_memory_bytes, and the limit-relative ContainerNearMemoryLimit is a complement with its own promtool test rules cases — including one that fails if the numerator is ever "simplified" from RSS to working set, and one that fails if the divide-by-zero guard is dropped. #71 had already taken the half that was sizeable without any of this: pids_limit, because tens of threads against a 10,000-thread abort is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which is at a measurable steady state at day 28 of 30.
#286 Alloy tails the backup archiver's tar stream into Loki. Done 2026-09-04. Found while sizing #114, and it is the reason loki's ceiling is 1536M rather than about 512M: backup-volumes.sh writes each volume's gzip stream to a container's stdout and discovery.docker tails every container on the socket, so one make backup put 765 MB of binary through the log pipeline in three minutes — roughly three days of the estate's real logs, into a loki-data volume that is not encrypted where the archives deliberately are.

The fix this entry proposed would have broken log collection. Filtering on __meta_docker_container_label_com_docker_compose_project and dropping what does not carry it reads as the tidy answer, but that label is empty for every container on oracle — wiki, db and the agent itself are all plain docker run — so it would have silently stopped collecting logs for the one service in this estate anybody uses. Checked against container_last_seen across both hosts before writing any of it. What landed instead is --log-driver none on the archiver runs, so Docker discards the stream at source, plus an opt-out label (homelab.logs=off) that Alloy drops on before opening a stream at all.

Opt-out is a weaker guarantee than the allow-list this entry wanted, and the gap is named rather than papered over: a future throwaway container that sets neither flag is still tailed. Bounding that is a per-stream ingestion limit in Loki, which trades a flood for silently dropped lines and is its own decision. loki's mem_limit is unchanged until a fortnight without the flood exists to re-derive from — 2026-09-18.
#249 Scrape the UPS self-test schedule. #93 left mjolnir testing itself every fortnight and nothing able to see that it does. The schedule, the last-test date and the last result are all PowerNet OIDs and the apc_ups module walks the standard UPS-MIB only, so a card that reverts to never(5) produces no alert and no changed metric — upsTestResultsSummary holds 1 (donePass) forever. The missing pack was visible in a MIB already walked; the missing schedule is not, which is the same failure one level up. The real cost is not the two rules but a pinning decision for APC's MIB in scripts/snmp-mibs.sh, which has no first-party git ref to point at. → runbook
#102 Turn on the AdGuard probe once the machine exists. The failure ADR-0010 chose on purpose: filtering fails open, so losing it costs advertisements rather than connectivity and nobody in the house reports it. That is #90's shape one service over — a filter that is dead and one that is merely quiet look identical from outside — and the mechanism is the blackbox exporter that now exists.

The detection is written, and it is the target that is blocked, not the work. #126 — the same failure, filed twice — landed the two dns modules, AdGuardNotAnswering, AdGuardNotFiltering and their unit tests on 2026-09-04, with all four states measured against a real AdGuard Home first. The entry above was right that a probe cannot go through morpheus, right that "answering" and "filtering" are two questions, and wrong about only one thing: what could not be written today was the target, not the rules. It sits commented out in targets/blackbox-dns.yaml under its own scrape job, because #102 has not bought the machine and Unbound is not forwarding to it — a target written before then is red from the moment the file loads, which is the check targets/blackbox.yaml has already agreed not to ship. Uncommenting it is two lines and a verify, and it belongs to #102.

This entry was headed #123 until 2026-09-04, and is re-headed rather than deleted because the work outlived the issue. #123 asked whether moving DNS off the gateway adds a quiet way to lose the internet; ADR-0010 and the verification appended to it answered that, and #123 closed. #126 was the same detection filed separately and closed with it. What survived both is the paragraph above, which is #102's to land — so it is filed under #102 here rather than left pointing at a closed issue, which is the failure this file keeps finding in itself.

What is left outside this repository: one line for the family runbook in Gerrrt/Lemmiwinks covering filtering is down and the internet is fine, which presents as advertisements returning rather than as an outage. Held until the machine exists, for the same reason the target is — today there is no filtering to lose, so it would describe a symptom that cannot occur. Two edits there come due in the same sitting: the "there isn't a Pi-hole" bullet in runbooks/dns_is_broken stops being true as written, and that page's source note deserves a line saying this was the one thing that did need writing. A note to whoever maintains those pages, not a tenth step in the walk.
#309 Check shiva's iLO firmware version too. #292 scoped check_versions.py to the Compute table in hardware.md, which is the list of hosts rather than of devices. shiva is not in it — correctly, being the BMC on the same physical box as Saruman — but since #292 it is the only device outside that table with both halves of a checkable claim: network.md gives it iLO 2.82, and its sysDescr is already scraped, answering Integrated Lights-Out 4 2.82 Feb 06 2023. So the only thing keeping it unchecked is a scoping line. It is not a one-line change, because the generic parser misreads both sides — os_key takes the first word and the first number, giving ('integrated', '4') against the documented ('ilo', '2.82'), and that 4 is the iLO generation, not the firmware. It needs a per-device extraction rule, the way morpheus already has one. Worth doing because a BMC that can power-cycle the hypervisor and mount virtual media, on the segment ADR-0014 exists to contain, is the first thing anyone checks against an advisory.
#311 Collect the Proxmox VE version from Saruman. The other host #292 left uncovered, and a different problem from #309: that one is parsing, this one is collection. Nothing on the wire carries the number hardware.md claims. node_os_info reports Debian GNU/Linux 13 (trixie) — the base PVE 9 is built on, unrelated numbering — and node_uname_info reports 7.0.14-12-pve, which confirms Proxmox and gives the kernel version. So Proxmox VE 9 is currently unfalsifiable, which is the condition #292 existed to remove. The documents are not wrong: recording Debian 13 would not tell a reader what the box is. The collection is the easy half — the textfile collector is already enabled in config.alloy and deploy-agent.sh already creates its directory on agent hosts, so a .prom file reaches Prometheus through the agent already there. The awkward half is where the thing that writes it lives: the monitoring host cannot reach VLAN 30 — TCP/22 to 10.0.30.110 does not open — so this cannot be another entry in the JOBS table, and wants a local timer on a host this repository can only reach through a person.
#12 Capture dashboard screenshots. make screenshots does five of the seven; the Logs and Security dashboards are deliberately excluded. → images/README.md

The stack does not watch itself

Found while verifying #12, and new since this file was last honest:

#67 No dead man's switch on the notification path — a 200 into a dead topic is a successful notification. The awkward half of it — the watcher has to be somewhere other than this host — is answered: ADR-0015 puts it on oracle. What that buys is bounded, and the ADR says so: it catches a silently dead notification path, and it cannot report a mains cut, because the switch between the two laptops has no battery (#110).

That last clause has been stale since 2026-09-08 and is corrected rather than swapped. The switch was racked in U4 on UPS power by #110, and each laptop rides a cut on its own cell — measured since 2026-09-12 and, on prometheus, replaced on 2026-09-18 (#454). So the off-host check can report a mains cut, for exactly as long as prometheus stays up on its cell, which is the one thing still unmeasured.

Two collection faults of the same kind were fixed in #62: the agent was answering to the name of the server, and cAdvisor could only see its own cgroup. Both ran healthy and produced nothing.

#63 was the same fault one layer up. ContainerHighMemory divided by a memory limit no service sets and guarded on it being non-zero, so it could not fire for any input while showing as loaded and healthy. It now measures against the host total instead. promtool check rules had passed it the entire time — it parses PromQL and never asks whether an expression can be true — so the fix came with the first promtool test rules unit tests in the repo, which fail if the rule stops being able to fire. They cover that one rule. The other 33 are still syntax-checked only, so the same class of fault could be sitting in any of them and would look just as healthy.

It was, in the rule directly above it. #305 is ContainerRestartLoop, which counted changes() of container_start_time_seconds — a metric that reports the container's creation time and therefore does not move when Docker restarts one in place, which is what restart: unless-stopped does to every service here. Where a start time would genuinely differ, a recreate, it is a new container id and so a new series carrying a different constant, and changes() is per-series. Measured against a deliberately crash-looping container: 38 restarts, and the expression returned six series every one of which was 0, against a threshold of 3. Nothing else cAdvisor exports moves either — the cgroup scope is reused across restarts, so the CPU counter does not reset. What moves is the series identity, because cAdvisor synthesises a restartcount label and emits it only while non-zero, so the fix counts generations rather than changes. Found while measuring for #114 rather than by looking, which is the uncomfortable part: two of the four rules in that file could not fire, and both were found by accident. Setting the memory limits themselves is #114, deliberately separate: a limit enforces, a rule detects, and making the second depend on the first is what left this one unfireable for months.

Infrastructure

#92 Get the firewall backup off prometheus, and buy a spare ProDesk. Half done. Since 2026-09-03 make backup-firewall copies every export to oracle — ciphertext only, the key stays here — and exits non-zero if it cannot, so the nightly job's metric says "stopped leaving this host" rather than "fine". Off-host, not offsite: both laptops share a shelf and a roof, and nothing copies anywhere a fire would not reach. The copy needs a one-time key exchange between the two laptops before its first run can succeed, and fails on purpose until then. That copy shipped defaulting to the wrong account — robo@10.0.99.30, where the login is atropos — so every nightly run would have failed on Permission denied from the first one, the local export written and verified and the copy step dead. It was invisible for a day only because the checkout the timer runs from was behind the commit that added it, which is its own lesson: a default nobody has executed is a guess. Retention landed with the fix, because until then both sides kept every export ever taken, nightly, forever; FW_KEEP (default thirty, and deliberately not KEEP, which make backup already owns in the shared environment file) bounds each side, never evicts the newest, never touches a file the script did not write, and clears the .part fragments a died copy leaves on oracle. Both sides converge in one run from any divergence. The far side's login shell is zsh, where an unmatched glob is fatal rather than literal, so the prune deletes by explicit basename and sends no pattern over the wire at all. What remains is the rehearsal, which is what turns restore-the-firewall.md from a hypothesis into a runbook; it now carries the bench procedure to follow and what to record. The box to rehearse on is bought, 2026-09-08 — the same ProDesk 600 G4 model, i5-8500T, 32 GB — and ADR-0034 decides it is not a cold spare on a shelf: it is rehearsed on first, then wiped and built as the sensitive tier's host (#404), and it is the firewall's spare hardware in a disaster at the cost of the tier being down until a replacement arrives. The powered-off shelf spare is deferred to the day that cost is unacceptable. The box is in hand, and proving it has a deadline the rehearsal does not: it was sold refurbished with a one-year warranty and a thirty-day return, and that return closes 2026-10-08. The spec, the serial and the NIC count are still the listing's word rather than the machine's, and the step that would settle them — installing pfSense over the Windows 11 Pro it arrived with — is also the step that ends the return. So the machine gets proved before 2026-10-08 whether or not the card has landed by then, which is the one part of this issue that is not waiting on a package. Writing that procedure found the runbook's own decrypt command had never been run: it passed --input-type binary, which sops rejects on the first byte of a real export, so a restore following the runbook would have stopped at step one. Fixed, and it is the kind of thing the rehearsal exists to find. Preparing for it found two more on 2026-09-09, by reading morpheus rather than the documents: the runbook said the spare needs a USB NIC, and the shopping list had one on it, but the box has none — its second interface is an Intel I226-V on an M.2 adapter, igc0, carrying the management LAN and every VLAN, and a USB adapter would have come up under another name and put the restore into the interface-assignment dialogue the same-model rule exists to avoid. And the version the verify prints is the config schema (24.6), not the release (pfSense CE 2.9.0); the runbook told the reader to match it to an installer, which cannot be done. Both fixed, the release recorded in hardware.md, and the shopping list names the card. The volume sets make backup writes still sit on the host they protect. Unlike the firewall, they have been restored — the whole stack was brought up on a restored set on 2026-08-29 and verified — but nothing copies them anywhere. Where they go is no longer open: ADR-0015 sends them to oracle alongside the firewall exports, which fits — a set is 867 MB of age ciphertext against 67 GB free — and leaves only the copying to build.
#251 Put the wiki on oracle into the repository, and back up its database. ADR-0015 ratified a host whose main service is not described anywhere here: wiki and its Postgres were created by hand in November, the content volume is anonymous, nothing copies either volume anywhere, and /etc/wiki/.db-secret is mode 664. The pages survive a disk failure because Wiki.js syncs from the Lemmiwinks repository; the accounts, history and configuration do not.
#102 Build ADR-0008's sensitive tier on VLAN 99. One box running Vaultwarden, Immich, Paperless-ngx and Home Assistant behind Caddy and step-ca, with AdGuard Home, ntfy and Homepage alongside. The box is bought and nothing is built. It is the ProDesk 600 G4 of 2026-09-08, by ADR-0034 — the one purchase this tier needed, made under #92's name because the tier's host had no issue and no place in the shopping sentence after this issue split; #404 is the tracker it lost. The placement is not the outstanding part — ADR-0008 settled it, and ADR-0010 has since been decided on top of it. What is outstanding is the build under #404, the rest of a stack, and four firewall rules the ADR counted as two. stacks/sensitive/ exists since 2026-09-09 with its foundation — Caddy as the only published port, step-ca as a certificate authority of the tier's own issuing to Caddy over ACME — authored ahead of the hardware the way stacks/lab was, and checked by everything make validate runs, caddy validate included. The foundation first claimed step-ca would be an intermediate beneath the lab CA; that root carries pathlen:0 and cannot have one, which ADR-0037 measured before deciding, along with where the root key lives (the monitoring host, never trinity) and why the leaves are seven days. The ACME path was proved on the monitoring host under a throwaway project; what remains for #130 is the mint and install on the real host, under #404, and a TLS-expiry rule sized for seven-day leaves (#426). AdGuard Home joined it the same day (#135), in the shape ADR-0010 decided and not the one the service assumes: the blocklist set is a tracked file copied in on every start, the admin hash is in SOPS, port 53 is published on the host's own address and answered for the firewall and the blackbox prober only, and the default 20 qps per-client rate limit — which would have throttled the whole house through its one client — is off. The half on morpheus waits for the host: forward-dns-to-adguard.md is the forwarding-mode change below, its verification, and the deliberate-failure test ADR-0010 asks for. So did Vaultwarden (#131) — the service whose restore path mattered more than its deployment: make backup and make restore learned the tier's volumes and now encrypt to the stack's own recipients rather than the first key in .sops.yaml, which is the pair of defects #428 names, and the round trip was rehearsed on the monitoring host with a seeded vault before the host exists (restore-the-sensitive-tier.md). Immich followed the same day as well (#132): four containers behind Caddy, pinned by digest, each under a memory limit, booted once on the monitoring host to find where the images write. What that landing turned up is that the off-estate copy ADR-0023 makes the precondition on the first real photo still has no destination — #132 stays open for it. Paperless-ngx followed (#133): the document archive behind Caddy, with a Postgres and a Valkey of its own, running as the operator with every capability dropped and a CPU ceiling — the estate's first — and its volumes in backup-volumes.sh's sentinel table, alongside entries for the foundation's, Home Assistant's, AdGuard's and Immich's, which had none; #428's recipient half landed with Vaultwarden the same day. Booted from the pinned images before the file was written, on the monitoring host, since the tier's is not built; the limits are stated as unmeasured on the hardware they are for.

ADR-0010 costs more to implement than it reads, measured 2026-09-04. Unbound on morpheus is recursive and DNSSEC-validating with zero forward-zone blocks, so "put AdGuard first in the forwarder list" is a resolution-mode change rather than an edit to a list that already exists — and it hands a query stream that currently reaches no third party to AdGuard, Cloudflare and Google. Worth accepting deliberately when this build happens, not by ticking Enable Forwarding Mode. The ADR carries the detail.

All four are insertions above a deny, and none of them is an addition. ADR-0008 named 50→40 and 99→20 and said the estate's count "rises from three to five"; ADR-0013 retired the count precisely because a number cannot say where in the order a rule goes, and when these land they belong in that ADR's list rather than in a new total. ADR-0016 then read the ruleset and turned 50→40 into three — one on the Hicks tab and two on Winterfell's, each above a Block access to CasaBonita — and #95 carries those with the hardware, because a pass to an address with no NAS behind it is a rule nobody can test.

99→20 was the row nobody had read. It is read now. pfctl -sr on morpheus, 2026-09-04: Winterfell blocks every other VLAN explicitly above its egress pass, Block access to Skids (10.0.99.0/24 → 10.0.20.0/24) among them, so Home Assistant's rule is the fourth insertion and not an append. It goes beside the two SNMP passes that already sit above that stack. Skids as a source is untouched — it blocks all five other VLANs, carries #223's tripwire, then egresses — and the return traffic for a session Home Assistant opens is carried by state and never reaches the ruleset. So Skids stops being terminal inbound and stays terminal outbound, the same trade ADR-0016 made for CasaBonita and with the same test: the tripwire's counter, zero today, must not move.

What the firewall cannot tell you is how wide the rule should be. The other three are host- and port-scoped. This one has a source that does not exist yet, and a destination that is a whole segment unless the IoT devices are given statics — which would be a segment-wide grant of the kind #228 exists to close, this time out of Winterfell and into the VLAN whose stated assumption is that everything on it is already compromised. Worth settling in the same sitting: Home Assistant discovers devices over mDNS, which is link-local and does not cross a VLAN boundary, so nothing on 20 appears by itself however the pass is written.

Settled 2026-09-09, and narrower than the row read. ADR-0035 read the Skids inventory for what Home Assistant would actually open a connection to and found one device: the Hue bridge. Ring, the Echos, the HomePods, the litter robot and the white-noise machine are all reached through a vendor's cloud or not at all. So the pass is 10.0.99.40 → 10.0.20.104:80,443/tcp — the two ports the aiohue code uses, 80 once at pairing and 443 after — above Block access to Skids, and it waits on two things: trinity, and a Kea reservation for the bridge, because Skids has none and its pool holds every address on the segment. Home Assistant itself is authored (#134): the Container flavour, no Supervisor and no add-ons, as an ordinary member of the tier's network behind Caddy — not network_mode: host, which exists for discovery that cannot cross a VLAN anyway — booted read-only with every capability dropped against the pinned image before it was committed. Its credentials are the one place the tier steps outside SOPS, and the ADR says why. No USB radio, so where the box sits is not this service's concern.

Two of the three things said to be waiting on this tier are not waiting on it. #67's watcher went to oracle under ADR-0015 and needs no self-hosted ntfy; #97's host override was never downstream of AdGuard, which ADR-0018 says outright and ADR-0010 is the reason for. #98 is the one that stands — there is no eero integration until there is a Home Assistant. What moving ntfy in-house does change is the alert path: the heartbeat's whole value is that it leaves the house, and an endpoint on a network with no external exposure cannot reach a phone that is not on it. Whether the in-house ntfy replaces the external topics or sits beside them is a decision this build makes, not a detail of it.

One of the four services arrives with monitoring already written for it. #126 shipped the DNS probes and rules that catch a silently-dead AdGuard, with the two targets disabled in stacks/observability/prometheus/targets/blackbox-dns.yaml because there is nothing to probe yet. Uncommenting them is part of this build, not a separate task: the file carries the address assumption to correct — 10.0.99.40, the next free reservation — and the two commands to verify the probes against the running exporter, including the negative one, since a filtering probe that cannot go red is measuring nothing. Blocking mode has to be Default/Null IP, which the module pins deliberately.

Two purchases where the plan assumed zero, and this is the first. oracle cannot host it — ADR-0015 measured 2549 MiB available behind a 5400 rpm disk and a 100 Mb/s NIC — and ADR-0007 keeps household services off the lab hypervisor. The SSO this box deliberately does not get is #103, and as of ADR-0022 it does not get it until this box holds real data rather than indefinitely — which puts two things on this build: TOTP enrolled on the three services that can carry it, and a disk encryption decision made here rather than inherited.
#95 Plan and build the NAS on VLAN 40. ADR-0016 answers the four questions #95 raised. The box is bought, 2026-09-09, tracked under #413 since #95 closed on the decision: a Lenovo ThinkServer TS150 — Xeon E3-1225 v6, 8 GB ECC, four 3.5" bays, no drives, no OS. Not the "quiet N100-class" chassis the ADRs pictured, and the difference is worth stating rather than smoothing: a 73 W desktop Xeon in a tower against a 6 W part in a shoebox, so it draws and makes more, and it will not rack. What it has that an N100 box does not is ECC memory under a ZFS mirror, four real bays, and a Kaby Lake iGPU whose Quick Sync is what #138's transcoding needs. The drives were bought on 2026-09-11 — two Seagate Exos X20 18 TB, chosen on cost per terabyte per the ADR — and the boot disk with them, an Intel DC S3520 240 GB, with the bracket and the tape that carry it in the optical bay bought the same day. This paragraph said a bracket was the only thing left to buy, which had not been true since 2026-09-11. The box landed 2026-09-15, and the boot disk, the bracket and the tape with it. The OS is decided, and it is TrueNAS, not Ubuntu Server — this line said the opposite until 2026-09-15, when the operator said aloud what was about to be installed and it turned out ADR-0016 had decided against the only OS ever intended for this box (ADR-0040). The media stack stays in this repository either way, as a compose file TrueNAS launches rather than catalogue apps, so Dependabot and the image-pin check keep reaching it; what leaves CI's reach is the pool and share layout. The host is built and the pool is not — this paragraph said "nothing is configured" until 2026-09-16, and on that day build-the-nas.md §0 was completed end to end: the BIOS flashed, AMT found on Intel's factory-default credential and unprovisioned, the optical drive swapped for the boot SSD and its SMART read before the install, TrueNAS 25.10 installed, the static and the Kea reservation both set, and the inbound rules created and verified. What is left is the two Exos drives and everything downstream of them: the mirror erebor, its two datasets, the household share, the stack, and the one test that decides whether the stack stays here at all — whether Quick Sync reaches a container, which is ADR-0040's reopen condition and is still unrun. Reading the enforced ruleset first changed two of the answers, and both were borne out when the rules were created. 50→40 is not simply a rule to add: Hicks and Winterfell each carry an explicit Block access to CasaBonita above their catch-all, so the pass has to be ordered in front of a deny, and one appended where new rules naturally land would match nothing — the same fault ADR-0013 found in Allow Hicks access to ImaginationLAN. That is why position was verified from morpheus with pfctl rather than from the web UI, where an appended rule looks present while matching nothing. And one rule is not enough: every other host in the estate is monitored and backed up by pushing, so a NAS built like the others would have its Alloy agent initiating 40→99, the first upward path in the estate and the end of the property ADR-0008 claims to keep. The ADR reverses the direction instead — scraped rather than Alloy pushing, the metadata backup pulled by prometheus rather than sent — which costs the NAS its logs, because Loki has no pull and its ingest is unauthenticated, and costs it its SMART and its patch state for the same reason (#255, #483). ADR-0016 wrote down three rules, all inbound, all host- and port-scoped, and deliberately did not create them: a pass to an address with nothing behind it is a rule nobody can test. Four exist since 2026-09-16, because the Hicks pass is split into two rather than carrying a port list, and Hicks reaches 443 and not the 22 that ADR-0016's table names — that port assumed a box administered over SSH, which was an operating-system decision inside a firewall table, and ADR-0040 carries the correction against its own text. Port 22 survives on the Winterfell rule and is inert, because TrueNAS ships SSH disabled. The scrape target is node_exporter on 9100, settled by #256 on 2026-09-17 — the pass that exists and a dashboard built entirely on node_* series, against an alternative the repository had never measured and which would have cost a fifth rule. The job and the target file are written; the target is commented until the pool exists, because on TrueNAS every route to an exporter runs through it. Terminal survives in the direction that carries it — CasaBonita stopped being terminal inbound on 2026-09-16 and stays terminal outbound, with #223's tripwire untouched and reading zero packets. Capacity buys a four-bay chassis with two bays filled, because the bay count is the half that cannot be changed later and the library's size is a number nobody has; both trays are still empty.
#101 Build ADR-0007's defended estate on Saruman — a Windows domain, Wazuh, Velociraptor, PBS and a second observability stack. The umbrella. The observability half is built and the security half is decided but not yet racked; the work is split seven ways, which is what moved it out of Decided but not built below. ADR-0020 answered the two questions ADR-0007 left open, and both of them blocked the first line of work. stacks/lab/ runs in a guest, not on the hypervisor: Saruman is the one host in the estate that must not run Docker, because ADR-0014 leans on its own firewall and Docker rewrites iptables — which is why #88 deployed the native .deb there rather than a container. And the stack carries its own Prometheus: ADR-0007 named three services while saying in the same sentence that config.alloy is reused with only the two *_URL variables changed, and that file has two sinks, so a lab without a Prometheus points the second one at 10.0.99.20 and inverts the isolation the ADR exists for. #264 is built and running: stacks/lab/ is deployed on alexander (#262, 2026-09-05), the guest on Saruman that ADR-0020 called for. Four services, its own age key, its own leaf from the estate's CA, and three Loki log sources confirmed on the host — the check the Ubuntu-over-Debian decision exists for. → runbook Building it made the tooling stack-aware (render-config.sh derives its required keys per stack rather than demanding the estate's ten, reload-config.sh skips services a stack does not declare, bootstrap.sh refuses to give one age key both stacks) and gave .sops.yaml the lab rule ADR-0020 asked for. #263 followed it: scripts/stacks.sh is now the single definition of what a stack is, and validate.sh, ci.yml, pin-digests.sh and the Python checkers all read it instead of carrying stacks/observability. Both stacks are checked, each line says which, and a directory under stacks/ with no compose.yaml fails rather than being skipped — a stack nothing checks being the defect the list exists to prevent. Two guards got stronger on the way: rules without promtool unit tests are now a failure rather than an absence nobody measured (#63), and the reload/ABSENT_BINARIES cross-checks gained a cross-stack mode, because "not in this compose file" stopped meaning "in no stack at all" the moment there were two. #265 decided the domain and closed on the decision; #414 is the build, and it is what everything else is pointed at — not built as of 2026-09-09: six VMs, the runbook's eleven sections, evenings at Saruman from a Hicks workstation, nothing to buy until the endpoints. It is sized by ADR-0029 rather than by taste: six guests on the .50 decade, because NTLM relay needs a destination that is not the origin and, since Windows 11 24H2 requires inbound SMB signing where Server 2025 does not, the only relayable host in a DC-plus-workstations domain is the DC itself. The servers run continuously and the endpoints per session, because a 7.2K mirror serves about ninety random write IOPS and six idle Windows guests would be most of them — the number #418's SSDs were bought against, and the one their fit re-derives. The drives went into bays 3 and 4 on 2026-09-18 and are still unassigned, and an unassigned drive changes no duty cycle either — the sentence above holds until the runbook's step 8 measures both arrays. Three things that ADR left explicit because they fail quietly: the DC takes its clock from the gateway, not time.windows.com — ADR-0014 named that failure and did not fix it, and the alert reads the sync source rather than the offset, because the offset reads zero for exactly the case where w32time has fallen back to the CMOS clock; the domain answers its own names inward only, with no delegation on Unbound, which is the one documented exception to ADR-0010's "clients receive the gateway as their only resolver"; and the domain is scraped, not published, reversing four comments in stacks/lab that expected the ports: block to open here — an unauthenticated remote-write receiver on the segment that exists to hold attackers hands a delete-series API to the thing it is meant to be recording. #266 Wazuh and #267 Velociraptor are one decision, and ADR-0030 makes it: both in stacks/soc/ on odin, a second guest, because alexander is 8 GiB and Wazuh's stated minimums are six before the dashboard — and because a second host means a second directory under stacks/, which is what ADR-0004 says rather than something it forbids. It also corrects #266's own premise. Six agents make about 21 GB of alerts a quarter, which the disks do not notice; what runs out is heap-per-shard, at OpenSearch's twenty-five-shards-per-GiB against one daily index each, so retention is thirty days because that is what a 2 GiB heap buys. stacks/soc/ is not yet authored, and nothing stops it being written and CI-validated ahead of odin the way stacks/lab was ahead of alexander; what it cannot do before #414 is say anything, because an agentless Wazuh has nothing to report. #268 PBS was decided and deferred by ADR-0027: a hypervisor backing up its own guests to itself is not a backup, smaug did not exist, and PVE already does the snapshots the local-only answer needs — so PBS would add a service for a capability that exists. That deferral named its own trigger — "PBS follows the NAS answering on 10.0.40.30, not the box arriving" — and the trigger has fired: smaug has held that address since 2026-09-16. This paragraph said the trigger had moved from a purchase to a build, which was right, and then the build happened. What has not cleared is the blocker: the pool does not exist, so there is still nowhere to send anything, and ADR-0027's sync job was designed against a Linux host rather than TrueNAS — on TrueNAS it is PBS in a VM or a change to an NFS/SMB datastore, which are not the same answer. #485 carries the re-read, because a fired trigger whose tracker closed on the decision is how an accepted ADR quietly becomes a rejected one. The lab has revert and not backup until it lands, and the ADR names what gets backed up when it does — which is also why ADR-0029 gives PBS no disk on this pool: there is nothing to give it yet, and that stops being true on the day this is built. Liveness stays where it already was, with #257: ADR-0020 decides only that no Alertmanager goes inside the stack, and the lab is otherwise being built to go quiet.
#421 Buy ifrit and build the range — only after the main network is finished. This entry was keyed to #96 until 2026-09-17, and #96 closed on the decision and the runbook on 2026-09-04: the purchase it ended by naming had no tracker at all for five days, and this file went on pointing at the closed issue for another eight after #421 was opened — the same shape as #102 → #404 and #95 → #413. The isolation mechanism ADR-0007 deferred is settled by ADR-0014: ifrit is single-homed on ImaginationLAN, the targets sit on a bridge with no physical port on a subnet the firewall does not route, the attack VM does not forward, and the hypervisor management planes close at the host. ADR-0017 settles the rest — buy for IOPS and quiet rather than for threads, because the range's whole operation is snapshot-and-revert and Saruman's complaint is already spindles; socketed RAM, because prometheus's is soldered; 172.30.30.0/24 on the isolated bridge with no gateway anywhere on it; and no backups, no monitoring and no patching for the guests, so the least important part of the lab joins none of the estate's loops. "After the main network is finished" named three issues when that sentence was written, and two of them have closed since: #234's tripwire was armed and verified on 2026-09-08, and #235 was decided the same day — the iLO stays, ADR-0033 — with the hardening that decision owed done on 2026-09-09. Both are recorded in full under Done below, which is why this paragraph does not restate them. What is left of the gate is #101, and it is an umbrella rather than a thing anyone builds: what it means here is the domain (#414) and the SOC stack (#266 Wazuh and #267 Velociraptor), because an attack VM pointed at an uninstrumented estate teaches nothing. This paragraph ended "what is left is the purchase itself and the build" until 2026-09-17, which read as though the money were the next thing to spend. The purchase is gated on #414 being built, and it is the last purchase on the estate's list, not the next — which is what Everything still to buy at the top of this file has said since #421 was opened, and what this entry now agrees with rather than contradicts. → runbook
#418 Fit the two SSDs in Saruman. Two Samsung SM863a 960 GB SATA enterprise drives, bought 2026-09-09, delivered 2026-09-11, fitted 2026-09-18 and in no logical drive yet. ADR-0007's constraint — "the fleet is sized against spindles, not RAM" — became a number in ADR-0029, about ninety random write IOPS for the whole machine, and that number sized the lab domain's duty cycle and #266's indexer. The SSDs raise the ceiling.

Two of the three questions the issue left to the fit are now answered, and the one that matters is not. The layout is decided: the SSDs become a second logical drive on the P440ar, RAID 1, Smart Array managed, with the 7.2K mirror keeping Proxmox, the ISOs and the backups. That is the smaller change, it keeps a spindle for the things that do not need IOPS, and it leaves IloDrivePredictiveFailure, IloDriveSmartUnreadable and the replaceDriveSSDWearOut(4) state reading cpqida.mib exactly as they do today — so #351's decision to skip smart-state on this host stays right rather than needing re-checking. #76's cache is read and handed over, not changed: modify cacheratio= is controller-wide and would land on the array holding every guest, so the runbook takes the ssacli reading nobody has ever taken on this machine and #76 owns what to do about it.

The trays blocked it for a day, and the fit stopped at the bays. A Gen9 bay holds a drive only in a SmartDrive carrier; two 651687-001 were bought 2026-09-11, had not arrived on the morning of 2026-09-17, and were in the chassis with a drive in each by 20:11 UTC on 2026-09-18 — both drives present, Bay 3 and Bay 4, different serials, solid-state, SMART ok, notConfigured, and nothing alerted. That is the runbook's step 4 and no step after it: no ssacli on the host yet, no logical drive, no thin pool, alexander still on the HDD mirror. Three predictions were wrong on the day and are corrected in the runbook — the iLO names the drive SAMSUNG and not by part number, the walk did not get slower, and the serials were read back through the iLO rather than off the labels first, which the runbook now says can still be put right while the drives are unassigned and not after. One reading is worse than hoped: the wear and endurance columns are blank on both SSDs, as they are on the HDDs, and if they stay blank once the drives are in a logical drive, wear monitoring on the newest drives in the estate needs smartctl through the hpsa path, which is an issue the runbook's closing paragraph already describes.

What is still open after that is the only thing the purchase was for. ADR-0029's ninety is derived, not measured — seek plus half a rotation at 7200 rpm — so the runbook measures both arrays at the parameters that derivation implies, 4 KiB at queue depth 1, and three times: the HDD mirror loaded, the SSD array idle, and the HDD mirror idle once the guest has moved off it. Until that reading exists, ADR-0029, ADR-0007, ADR-0017 and the #414 paragraph above all stand as written. The measurement makes them stale on the day it lands, not the day the drives did, and it gets a dated note on each rather than a silent edit — ADR-0001 makes them immutable. → runbook

Automation

#98 Device joins as events. Answered by ADR-0019, which keeps the issue's landing site and changes its source. The events belong in Loki and in security.rules.yaml — that part was right. But "the eero API" is a cloud API: there is no local one, the integration everyone means is a HACS component polling api-user.e2ro.com every 120 seconds, and it cannot log in with an Amazon-linked account. Routing a question about this network's own wire through Amazon makes the answer late and makes it disappear whenever the WAN does. morpheus already knows. The eeros are bridged, Kea serves every segment, and 1,200 lease lines a day are one pfSense checkbox from the 1514 listener that already carries filterlog. Three rules land in the dhcp group: first lease on Hicks in seven days (warning), the same on Winterfell (critical, and zero in 13 days of logs), and DhcpLeaseLogsStopped, because the other two fail silently. Measured cost on Hicks: about one alert every three days, and every one of the five in the sample was worth a look — two of them an OUI the inventory places on Skids. No longer blocked behind ADR-0008's sensitive tier, and leaves are dropped rather than deferred: 9 releases against 4,732 allocations in four days, and a departure is not a security event. What is left is ticking DHCP Events on morpheus — before the rules deploy, or DhcpLeaseLogsStopped fires truthfully — and reading the first week, which is one alert per device and therefore an inventory check. → runbook
#99 Move deployment from make up over SSH to something pull-based, so the host converges on the repo rather than being pushed to. Answered by ADR-0021: an hourly timer running scripts/converge.sh, on #77's existing wrapper and alert machinery. The issue's stated blocker — an age key on a host that pulls from a public repository — turned out not to be one, because the key was already on that host and public means readable. The real question was unattended execution, and the answer is a pinned signing fingerprint plus a record of every revision deployed. What is deliberately left out: oracle and saruman are still pushed to with deploy-agent.sh, and nothing tracks unifying that.
**#100 Automate the Grafana
#436 Build a deployment jumpbox on ImaginationLAN, and decide where the CA lives. Decided by ADR-0043; the guest is not built. The host is phoenix, 10.0.30.70, a guest on Saruman that holds the estate's first Proxmox API credential and the SSH key the toolchain will inject into what it builds — the prerequisite for the Packer, OpenTofu and Ansible issues, which today have nowhere to run from. The CA question the issue carried was answered by reading the root first: the issue's premise — that the tier's step-ca already sits beneath the estate's CA — is the sentence ADR-0037 retracted, and the estate's root is pathlen:0, so the only question was where one key file sits. It stays on prometheus, because the host that holds credentials for every other host must not also hold the key every other host trusts, and because VLAN 30 is the segment ADR-0014 built to hold attackers. Two things the issue did not count: the Proxmox firewall on Saruman admits 8006 from Hicks only, so the build widens ADR-0014's rule by one address on one port, recorded as a marked amendment there and on ADR-0039; and certificates/ca-key.pem has no backup or custody story at all, unlike the age key and the tier's root — found, named in the ADR with what the answer is not, and carried by #496. The same host is where ADR-0042 terminates the remote path; that ADR left the 8006 question to this issue, and it is taken here. → runbook

Decided but not built

Accepted ADRs with no work behind them. Recorded here because an accepted ADR with nothing tracking it is indistinguishable from a rejected one after six months.

#102 ADR-0008's sensitive tier — its host (#404, the ProDesk of ADR-0034), its nine services and the four firewall rules. Under Infrastructure above, because it has a shape now rather than only a decision.
#103 The SSO deferral ADR-0008 takes knowingly, given an expiry by ADR-0022. Under Security above, because it has a condition now rather than only a decision.
#85 SNMPv3 on the iLO and the UPS card, decided by ADR-0036 with the repository side built and the device side not yet done. Under Security above, because it has a procedure now rather than only a decision.
#444 The CRS326's operating system and management posture, decided by ADR-0041 ahead of the cabling window rather than during it — RouterOS over SwOS, the name neo and 10.7.7.2 inherited, www-ssl from the estate's CA with plain www off, an SNMPv3 authPriv user, and mirroring left disabled per ADR-0006. Nothing is built: the switch is still in transit and neo is still the MokerLink. Under Security above, because it has a procedure now — swap-the-switch.md — rather than only a decision.

Considered and declined

The mirror of the section above, and recorded for the same reason turned around: a service rejected for good reasons with nothing written down is indistinguishable from one nobody thought of. It gets proposed again, evaluated again, and can be deployed on the second pass because the first pass left no trace (#150).

These are declines, not bans. Each records what was weighed, so a later proposal argues with the reasoning rather than restarting from nothing — and several of them name the condition that would change the answer.

Nextcloud — the obvious "one app for everything" answer, and declined because it overlaps three services already chosen: Immich (#132) for photos, Paperless-ngx (#133) for documents, and file sync. It is more surface and more upkeep than all three together, and its app ecosystem is a second, unpinned supply chain operating outside compose.yaml — the same objection that rules out Home Assistant add-ons in #134. If the want is file sync rather than a suite, Syncthing does that with no server-side application at all, and would be the thing to evaluate instead.
The *arr stack — Sonarr, Radarr, Lidarr and the indexer and subtitle services around them. Five or more services, each holding indexer credentials and each with a standing outbound appetite, added to the tier ADR-0008 deliberately defined as low consequence. The media tier's whole justification is that its compromise costs a film night; this raises what is at stake there while adding the most moving parts of anything on the list. Declined for now rather than permanently — but it should be its own decision with its own reasoning, not a footnote to the NAS build.
YunoHost-class installers — YunoHost, Tipi, HomelabOS, StartOS and similar. They own the compose file, the update path and often the reverse proxy, which conflicts with essentially everything this repository does on purpose: one compose stack per host (ADR-0004), every image pinned by tag and digest, configuration validated in CI, secrets rendered from SOPS at deploy time. Adopting one trades the properties that make this estate reproducible for a faster first install. Wrong trade here — and the trade, not the software, is the reason.
Guacamole — a clientless browser-reachable RDP/VNC gateway. Convenient, and a larger concession on the management segment than SSH already is: it turns any browser session on Hicks into a potential path to every console in the estate, and it stores connection credentials to do it. The estate already has a KVM in U6 for physical console access. Declined. The remote-access question it gestured at is answered differently by ADR-0042: WireGuard to the lab jumpbox, terminating on ImaginationLAN and reaching the lab only. That is not a softening of this decline — it stores no connection credentials at a gateway, it never touches Winterfell, and the boundary is the firewall's rather than an application's. Guacamole's objection was about the management segment, and nothing about ADR-0042 goes near it.
Frigate — locally-processed object detection on camera streams, and the one service on the shortlist that would change the network's shape rather than its population: continuous RTSP from every camera through the 99 → 20 rule ADR-0008 authorised for Home Assistant's occasional control traffic, a clip archive that is the most sensitive data store in the house on the management segment, and a Coral or GPU the sensitive tier does not have. Declined by ADR-0032 on a fact that comes before all four: every camera on Skids is a Ring device, and Ring exposes no local stream, so Frigate has nothing to consume. Adopting it is a camera replacement first, which is its own decision. Reopened by RTSP cameras, an accelerator, and a separate row in ADR-0008's table for the continuous rule — all three, each decided on its own (#149).
Proxmox clustering — joining Saruman and ifrit into one cluster once #421 makes them two Proxmox hosts on VLAN 30: one pane of glass, guest migration, shared storage. The first entry here that is a capability rather than a service, and declined by ADR-0039 because a cluster is one /etc/pve, one realm and one quorum across exactly the boundary ADR-0007 draws — root on the attacker's host becomes root on the estate's — and cannot be built without the second NIC or VLAN-aware bridge ADR-0014 names as grounds to reopen it, on a host that is off between sessions by design and would take the estate's hypervisor's quorum down with it. Reopened by a third host that is neither attacker nor defended estate, a live-migration need that snapshot-and-rebuild does not serve, or ifrit ceasing to hold attack tooling — each its own decision (#443).
Authelia / Authentik — already decided, and listed only so the next shortlist does not present it as new. ADR-0008 defers SSO knowingly for two users with no external access; ADR-0022 gives that deferral an expiry and #103 tracks it. An identity provider is also the only route to a second factor for Grafana, Immich and AdGuard Home, none of which can carry one themselves — so this decline has a known end, unlike the others here.

Done

#441 Alerted on a sensor that stops logging, and closed the Zeek half by deciding it elsewhere. 2026-09-17. SuricataLogsStopped landed on 2026-09-12 in loki/rules/security.rules.yaml: absent_over_time({app="suricata"}[9h]), aggregate across both interfaces, for: 0s like the two absence rules it copies. The window was read rather than picked — 22 days of the stream, in which the longest silence across both interfaces was 75 minutes — which is what the issue asked for, because the thirty minutes it opened with would have paged about twice a day.

**Re-measured on 2026-09-17 over 27.7 days and it holds.** Worst aggregate
silence 80 minutes, six gaps past an hour, none past two, so nine hours
keeps nearly seven times the headroom `DhcpLeaseLogsStopped` settled on.
The aggregate design earned its keep in the interval: `igc0.10` alone went
quiet for more than nine hours **seven times**, worst 47.5 hours, so a
per-interface rule would have paged seven times in a fortnight. The
short-gap counts are not comparable between the two measurements — 135
over thirty minutes here against 54 on 2026-09-12, a difference in method,
not in Suricata — and the rule's comment says so rather than presenting
one series.

**The Zeek half was not built, and will not be as the issue described it.**
`ZeekLogsStopped` was to be the same shape against the lab's Loki once
[#437](https://github.com/Gerrrt/HomeLab/issues/437) existed. Three things
make that the wrong instrument. #437 puts Zeek's logs on `alexander` and
never on `10.0.99.20`, which ADR-0007 requires;
[ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)
gives that stack no Alertmanager on purpose, and `stacks/lab/loki/` ships
no ruler and no `rules/` because *"a ruler with nowhere to deliver
evaluates rules and discards the result, which reads as coverage and is
not"*; and
[ADR-0028](adr/0028-let-guest-liveness-cross-but-not-guest-telemetry.md)
revisited that exact boundary and kept it. #437 already carries the right
mechanism — a `homelab_zeek_mirror_active` textfile gauge read on the
hypervisor, crossing as guest state under ADR-0028 — so Zeek's liveness is
a metric question answered where Zeek is built, and this issue closes
rather than holding a slot behind a Wave 2 dependency that itself waits on
[#414](https://github.com/Gerrrt/HomeLab/issues/414).

What this proves and what it does not, unchanged from the merge: Loki
rules still have no unit-test harness, so `check_loki_rules.sh` shows the
rule parses and the ruler evaluates it, not that it fires. The live ruler
reports it `health=ok` and `state=inactive`, which is the good state and
not evidence of detection.

#470 The wiki's drift check is watched. 2026-09-14. Gerrrt/Lemmiwinks/.claude/tools/drift-check reads the wiki's machine-checkable claims against the machine and files a wiki issue when they disagree; since the morning of 2026-09-14 it ran from atropos's crontab on oracle with no metric, which is #400's shape — a check that stops looks like a wiki with nothing wrong. Shipped as the fifth agent collector: scripts/collect-drift-check.sh runs the checker as the wiki maintainer's user from a root unit that exists to write the textfile directory, records when it ran, how it exited and the four claim counts, and DriftCheckStopped fires on a timestamp more than a day old. The checker stays in the wiki's repository, because its claims are that wiki's sentences; this repository watches that it runs.

#468 The backup verifier mistook every Loki archive for a Paperless one. 2026-09-14. Both backup-volumes and verify-backups exited 2 from 2026-09-13, and every loki-data archive on the host — five sets that had verified clean the day before — failed the same line: this archive carries the sentinel of paperless-data. #133 gave paperless-data the sentinel ./index; a Loki data volume has a top-level ./index of its own, listed as a loki-data companion two tables down in the same script, and the change reached the host with the 2026-09-12 revision. The archives were intact; no manifest was written after 2026-09-06 and nothing was pruned.

Found from the wiki side, by reading the units' journal out of Loki
(`Gerrrt/Lemmiwinks#279`). The fix keeps `./index` — it is the measured
marker — and makes `verify()` treat a foreign sentinel that is one of the
volume's own companions as evidence only when the volume's own sentinel
is also missing, in `--hot` as well; a Paperless archive mislabelled
`loki-data` still fails on both counts. `load_inventory()` now refuses a
table in which two volumes share a sentinel outright, the case
`verify()` cannot recover from. Applied by hand on the host after merge,
since converge is report-only; the next Sunday set is the proof.

#110 Racked the shelf switch in U4, on UPS power. 2026-09-08. The 1U vented shelf, the TP-Link that prometheus and oracle hang off moved onto it with its uplink back on port 3 of neo, and its power onto a UPS-fed outlet — the step that actually closes the gap, since a relocated switch on a wall socket is tidier and no better protected. The two laptops now keep their network on a mains cut as well as their batteries, which is what #93's pack was always half of. Bought with that pack on 2026-08-27 and closed the same day by a commit message that quoted "closes #110" — twice, the second time by the commit documenting the first — while every document said the shelf was on hand and not racked; reopened 2026-09-08 and done the same afternoon.
#234 Armed the lab tripwire on ImaginationLAN. 2026-09-08. The firewall rule arrived on 2026-09-06 with the untagged-LAN blocks, pointed at Internal_Segments — which names the lab's own subnet — and logged 1,239 DNS queries to the lab gateway in three days and nothing else. Moved to a House_Segments alias (every segment but 30) on 2026-09-08: sixteen minutes later the rule showed 26 evaluations and 0 packets, the lookups still reaching it and no longer matching. LabSegmentReachedInternalNetwork reads VLAN 30 as a source and excludes it as a destination, so it was right through the noisy days and fired on none of them; the restore runbook expects four tripwires and checks the lab's alias by name; the security dashboard charts the lab boundary beside the terminal one. Loki rules still have no unit-test harness, so the rule is proven to parse and evaluate, not to fire — the 1,239 lines were, accidentally, the proof for its source half.
#235 Decided: the iLO stays on the lab segment. 2026-09-08. ADR-0033. shiva is the BMC of the box being attacked, and a BMC compromise in the lab costs the lab; that is accepted and recorded in SECURITY.md. Two facts the issue predated settled it: since ADR-0031 a workstation reaches Winterfell on a named list, so a BMC there would need three more Hicks passes for its web UI and console — the issue's "console access unchanged" no longer held — and the management segment should not accumulate a device whose firmware line has ended. What follows is hardening on the iLO itself, by hand, and one firewall follow-up: the 10.0.30.10 → 10.0.99.20/udp "return path" rule is redundant with pf state and is the BMC's only path to Alloy's syslog listener, so it goes. Both done 2026-09-09: the rule deleted with the scrape watched through it — one failed scrape at the reload, clean since, zero packets ever matched — and the BMC hardened.
#228 Decided: Hicks reaches management on a named list, and reaches the lab entire. 2026-09-08. ADR-0031. Option 3 with the enumeration done: ten host- and port-scoped passes into Winterfell — SSH, the pfSense UI, DNS, NTP, ping, the wiki, Grafana, the UPS card — above a logged Block access to Winterfell, applied on morpheus 2026-09-02 and read back for the ADR on 2026-09-08 (28 packets dropped by the block, about two million passed by SSH alone). 50 → 30 stays wholesale by a rule on the Hicks interface, because ADR-0014 consumes it and the list that would narrow it does not exist until the lab does. Two deviations recorded rather than tidied: the lab pass is TCP-only, and the DNS/NTP passes to 10.0.99.1 carry nothing. The Winterfell half had been on the firewall for six days before any document called it a decision — architecture.md still described the catch-all — which is the gap this entry closes.

#153 Decided: the documents stay hand-written, and are checked where the truth is. 2026-09-06. ADR-0026.

**The answer was already built and never recorded.** #153 offered three
ways to close — adopt a tool, write a CI cross-check, or accept
hand-maintenance and say why — and `check_docs.py` landed on 2026-08-26,
the same day the issue was filed, following the pattern
`snmp-targets.sh --check` set on 2026-08-17. It now carries seven
assertions. So the middle option won by default and nothing said so, which
is the state #153 warned about in its own last line: *"leaving it
undecided is what produced the list at the top."*

No discovery tool. NetBox is, as the issue says itself, "a source of truth
you maintain, not a discovery tool" — it moves the hand-maintenance rather
than removing it, and brings a Postgres-backed Django app for four SNMP
devices. Scanopy is closer to the want and worse for this estate: a
discovery tool that can see every VLAN is by construction a device that
violates the segmentation model.

**The residual is prose about the firewall, and this session produced two
instances of it in one day** — ADR-0013's stale claim
([#344](https://github.com/Gerrrt/HomeLab/issues/344)) and the correction
for it asserting something that had been false for four days
([#229](https://github.com/Gerrrt/HomeLab/issues/229)). Both passed
`check_docs.py` correctly: it has nothing to compare them against, because
`config.xml` carries rule bodies, the WAN address and password hashes, and
`backup-firewall.sh` explains at length why that stays out of the
repository. Committing it so CI could read it would trade a documentation
defect for *"one age-key compromise hands over the complete blueprint"*.

So the firewall wants a deploy-time check rather than a CI one, in the
family this estate already has — `check_loki_coverage.py`,
`check_alert_channels.py --live`, `check_versions.py`,
`check_mounted_config.py`. Tracked as
[#363](https://github.com/Gerrrt/HomeLab/issues/363).

#229 The switch LAN's default-allow was replaced four days before anyone wrote it down. 2026-09-06. Read off pfctl -sr rather than taken from the issue: the interface carries six logged blocks — one per VLAN — above an egress rule renamed Allow internet, with DNS and NTP to the gateway the only passes above them. Exactly the shape #229 proposed, done on 2026-09-02, datable because pfSense rule identifiers are creation timestamps.

**This corrects a claim I introduced an hour earlier.** ADR-0025 and three
places in `network.md` said the switch LAN "still reaches every segment
outbound". That was ADR-0013's claim carried forward without being checked,
restated with a "still" that turned a quotation into a fresh assertion —
which is the exact failure mode ADR-0025 exists to fix, committed while
fixing it. `security.md` said the same. All four corrected, and ADR-0025
carries the correction rather than a quiet edit, per ADR-0001.

**One residual, latent rather than live.** The stock *Default allow LAN
IPv6 to any* rule is still there with no IPv6 blocks above it, while every
other interface carries paired `inet`/`inet6` blocks — the 2026-09-02 work
was IPv4 only. `igc0` has only a link-local address, which does not route,
so nothing can use it today. Worth closing when IPv6 is decided
([#353](https://github.com/Gerrrt/HomeLab/issues/353)).

#152 The estate knows how far behind its own packages are — on one host. 2026-09-06. Images are tagged, digest-pinned, CI-enforced and Dependabot-bumped; the kernel underneath was patched when somebody remembered.

**Cheapest thing that closes the gap, which is what the issue asked for:**
*"the point of the issue is the gap, not the tool. A four-host estate
probably wants the exporter, not another web UI."* No new service, no new
image, and **no root** — `/usr/lib/update-notifier/apt-check` runs
unprivileged, `/var/run/reboot-required` is a world-readable flag, and the
textfile directory is already owned by the user the timers run as. That
matters because every richer option wanted privilege the job table does not
have (#339, #351).

Two rules. `SecurityUpdatesPending` is **security** updates only, not all
pending upgrades — a host three ordinary packages behind is not a finding
and alerting on it is how this becomes noise. Its `for: 7d` is load-bearing
rather than cautious: unattended-upgrades applies security updates on its
own, so anything still pending after a week is held, phased or waiting on a
reboot. `RebootRequired` waits three days, because rebooting this host
blinds the estate and the alert is a reminder rather than an instruction.

**A test fixture taught me something about the rules I had just written.**
The first firing case used samples an hour apart and failed with `got:[]`
against a correct rule: an instant vector only looks back 5 minutes, so an
hourly fixture is stale for 55 minutes of every hour and the `for` clock
resets each time. Production is unaffected — the collector writes daily but
the textfile is scraped every 60s — and the fixture has to imitate the
SCRAPE interval, not the collection interval.

**It covers this host alone, and that is written down rather than implied.**
`oracle` is Ubuntu and would need it shipped by `deploy-agent.sh`;
`Saruman` is Proxmox and the same; `morpheus` is FreeBSD with no apt at
all. Tracked in [#360](https://github.com/Gerrrt/HomeLab/issues/360).

#193 Alloy reaches the Docker API through a read-only proxy. 2026-09-06. The socket is no longer mounted into Alloy at all.

`:ro` on a socket mount is close to decorative — it applies to the socket
FILE, not the API behind it, and anything that can talk to that API can
`POST /containers/create` with `/` bound read-write, which is root on this
host and the age key with it. #188 took Alloy's capabilities and closed the
direct read off `/rootfs`; this closes the larger half.

**Proven, not assumed.** Through the proxy,
`POST /containers/create` returns `403 Forbidden` and
`GET /containers/json` works.

**The allowlist was wrong on the first attempt, and only measurement found
it.** `discovery.docker` calls `/networks` to compute the network labels it
puts on every target, so with `NETWORKS: 0` the component failed with a
403, container log collection stopped entirely, and *the agent stayed
healthy while cAdvisor carried on unaffected* —
`loki_source_docker_target_entries_total` sat at 0 and nothing else looked
wrong. That is exactly the #62/#63 shape #193 predicted for a too-narrow
allowlist and the reason it insisted on before/after numbers. With
`NETWORKS: 1` the flow resumed: 864 entries streamed, 450 lines in two
minutes.

Before and after on this host, cAdvisor unaffected throughout:
`container_last_seen` 10 → 11 (the proxy is a container),
`cadvisor_version_info` 1 → 1, and the named set unchanged apart from the
proxy arriving and two of my own throwaway test containers leaving.

**One container is deliberately not `read_only`,** which is the exception
#186 asked to have recorded rather than skipped. The image generates
`haproxy.cfg` from its environment at every start, and a tmpfs over that
directory shadows the template it ships — both observed by running it, not
predicted. It still drops every capability, takes `no-new-privileges`,
holds no secret, and publishes nothing.

Said plainly in `SECURITY.md` and `docs/security.md`: this **moves** the
trust boundary rather than removing it. The proxy holds the socket now.
What it buys is that Alloy — network listener, rootfs mount, largest
surface in the stack — has no path to POST.

`oracle` still mounts the socket directly. `docker.alloy` reads
`DOCKER_API` and falls back to the socket when unset, so one config file
still deploys to every host and that one keeps working until it gets a
proxy of its own.

#344 ADR-0013's title became half false; ADR-0025 supersedes it. 2026-09-06. Gerrrt/Lemmiwinks#177 added a logged block from Winterfell to 10.7.7.0/24, so default deny now holds there with SNMP as the one pass above it — the exact thing ADR-0013 said the switch LAN lacked.

**The issue declined to fix it because "superseding versus amending is a
call for whoever owns the decision record". ADR-0001 already makes that
call:** *"ADRs are immutable once accepted. A decision that changes gets a
new ADR that supersedes the old one, and the old one is marked Superseded
rather than edited."* ADR-0002 → ADR-0013 is the precedent, and its
Superseded note is careful to say which claim fell — ADR-0013's now does
the same.

A note would have been the wrong instrument. ADR-0013 already carries one
for a table row added later, which is right for a table gaining an entry;
a *title* that has become false is not that.

Read off the firewall rather than taken from the issue: `pfctl -sr` shows
the block as rule 174 with the interface catch-all at 175, and the SNMP
pass above at 159.

**That reading also found a hole the issue only suspected.** The
`10.0.99.20 → 10.7.7.2:80/tcp` pass, added to keep the `switch-ui` blackbox
probes alive while the block landed, is still on the firewall — and those
probes were removed in
[#343](https://github.com/Gerrrt/HomeLab/pull/343). Verified there is no
consumer: nothing probes `10.7.7.2`, and `targets/blackbox.yaml` names it
zero times. Removing it is a firewall change on the Lemmiwinks side and is
recorded in ADR-0025's consequences rather than silently left.

#355 A deploy no longer reports success over a stale config. 2026-09-06. Found the same day, when #166 deployed clean — make converge fine, make up fine, reload-config.sh reporting reloaded prometheus, check_container_health.py reporting prometheus healthy — and its three latency targets never appeared. Prometheus was running the previous config.

`compose.yaml` bind-mounts four config files individually, and a
single-file bind mount is pinned to the inode. `git merge` writes a
temporary file and renames it over the target, so the container keeps the
old inode and `POST /-/reload` returns 200 having faithfully re-read the
pre-merge bytes. `docker compose up -d` recreates a container only when its
service definition changes, so a config-only commit recreates nothing and
the stale mount survives — which is most changes here. It went unnoticed
until #166 only because the deploys before it happened to change
`compose.yaml` too (#187, #330, #186) and recreated everything.

`reload-config.sh` already knew this shape: it records that
`render-config.sh` truncates with `>` to keep the inode, and that
write-temp-then-mv "would leave the mount pointing at the old inode". That
covers the files this repository writes. It never covered the files git
rewrites, which is every committed config.

**Bytes, not inodes**, which is a change from what the issue first
proposed. Comparing inodes detects this one mechanism and cries wolf on
another: a file rewritten with identical content has a new inode and
nothing wrong with it. Verified — after restoring the original bytes
through a fresh inode, the content check correctly reported a match where
an inode check would have reported staleness. Comparing bytes also works
on `loki`, whose distroless image has no shell at all, because `docker cp`
needs neither a shell nor `/proc`.

`make up` runs it with `--fix`, before the reload rather than after,
because the reload is not what is broken. It recreates only the services
that actually diverged and then asserts the recreate worked, so a
force-recreate that rebound nothing cannot report success.

Reproduced end to end rather than reasoned about: `blackbox.yaml` rewritten
the way git does it, the check failing and naming the service, `--fix`
recreating it, and the new bytes confirmed inside the container. The live
stack was restored afterwards and its eleven probes re-verified.

#166 Measure latency, and say where it is. 2026-09-06. Three targets, a blackbox-latency job and two rules, so the estate can answer the question #166 opened over — "is the internet bad right now, and is it us or the ISP?"

**`tcp_connect`, not ICMP.** #166 called `NET_RAW` "the one wrinkle". Since
it was written, blackbox-exporter gained `cap_drop: [ALL]`, uid 65534 and a
read-only root filesystem (#187, #330, #186) — so an `icmp` module would
hand exactly one container back a capability the rest of the stack just
gave up. A TCP connect is also the better measurement, not merely the
cheaper one: ISPs routinely deprioritise ICMP, so a ping time is not what a
game or a call experiences.

**Thresholds measured, not chosen.** Twelve samples each from `10.0.99.20`:

| target | min | median | max |
| --- | --- | --- | --- |
| `10.0.99.1:53` | 0.40 | 0.52 | 0.66 ms |
| `1.1.1.1:443` | 8.01 | 13.66 | 16.13 ms |
| `8.8.8.8:53` | 7.72 | 13.71 | 28.72 ms |

10 ms for the gateway is twenty times its median; 100 ms for the anchors is
seven times theirs and well clear of that 28.72 outlier — which is exactly
what an instantaneous threshold would have fired on. Both rules average
over ten minutes and then wait ten more, because #166 asked for precisely
that: *"a probe that alarms on one bad RTT will alarm constantly and be
muted within a week"*. A unit test holds a single 500 ms sample and
requires silence.

Two anchors on different networks, and two rules rather than one, because
the diagnosis is the point: gateway slow means the house, anchors slow with
the gateway fine means beyond it, one anchor slow means that provider.

**A separate job**, like `blackbox-dns` and for a related reason:
`EndpointUnreachable` is critical and routes to `urgent`, and an ISP blip
reaching `1.1.1.1` is not a 2 a.m. page.

**The second half of #166 is answered rather than built.** It asked whether
pfSense's own `dpinger` was absent from Loki because monitoring was off,
logging was off, or the syslog selector excluded it. Asked the firewall
directly: `dpinger` is running, `WAN_DHCP` reports 12.486 ms / 0.0% loss to
`1.1.1.1` — which independently corroborates the 13.66 ms baseline measured
from this host — and it logs to syslog only on a state change, so a stable
gateway writes nothing. Nothing was broken; there was nothing to ship.

That same command found `WAN_DHCP6` **down at 100% loss**, which nothing in
the estate knew. Filed as
[#353](https://github.com/Gerrrt/HomeLab/issues/353), deliberately without
a fix: nothing in the documents mentions IPv6 at all, so whether it is
wanted has to be decided before it is repaired.

#151 SMART on the drives that matter, with no new collection. 2026-09-06. The issue asked to choose between Scrutiny and smartctl_exporter. Working it turned up that the disks it said mattered most were already being scraped.

`Saruman`'s two SAS drives sit behind an HPE Smart Array, and the `ilo`
module already walks `1.3.6.1.4.1.232.2`. `cpqDaPhyDrvSmartStatus` was
there all along, reading `ok(2)` for both. It had no rule.

The distinction is the whole issue. `IloHardwareDegraded` already reads
`cpqDaPhyDrvCondition > 2` — the drive's CURRENT condition, which moves
once redundancy is spent. `cpqDaPhyDrvSmartStatus` is SMART's PREDICTION,
and it moves while the array still says everything is fine. On a RAID 1
mirror that gap is exactly what #151 was worried about: the mirror keeps
serving reads through a dying disk, and the failure only becomes visible
when the second one goes.

Enumeration read out of `cpqida.mib` rather than assumed, in HPE's own
words: `replaceDrive(3)` is *"a S.M.A.R.T predictive failure error"*,
`replaceDriveSSDWearOut(4)` is approaching the write limit, and `other(1)`
is the agent being unable to determine anything.

Two rules, because "replace this drive" and "I can no longer tell" want
different actions. `IloDrivePredictiveFailure` is `> 2` and **warning, not
critical** — `IloHardwareDegraded` is critical because redundancy is
already spent, this fires before that, and paging at the same level for
both would make the critical one mean less. `IloDriveSmartUnreadable` is
`== 1` with `for: 1h`, so a controller initialising does not page, and so
the predictive check cannot go blind quietly — the #63 shape.

Five test cases, including `ok(2)` staying quiet (both drives read 2
today, so without it the rule would pass its test while firing for a
healthy array) and `other(1)` staying quiet for the predictive rule, which
is what stops someone widening it to `!= 2`.

**The rest of the estate is still unwatched and that is
[#351](https://github.com/Gerrrt/HomeLab/issues/351).** Neither option
#151 proposed fits any more: Scrutiny is a service with its own datastore,
which ADR-0004 argues against, and `smartctl_exporter` as a container needs
raw device access — it would be the one container reversing `cap_drop`,
non-root and `read_only` all at once, to read something the host reads for
free. The fit is node_exporter's textfile collector, and the catch is that
it needs root while every timer in the `JOBS` table runs as `robo`.

#214 Catch a broken notification path without using it. 2026-09-06. scripts/check_alert_channels.py, in three parts, because the question has three different homes.

The failure it exists for: 471 of 493 notifications failed over ten and a
half hours on 2026-08-31, every receiver at once, because all four read
their URL from the same directory and it was unreadable inside the
container. `AlertmanagerNotificationsFailing` fired correctly and could not
be delivered — the alert about the broken delivery path travelled the
broken delivery path. `IloBatteryCondition` was firing and undeliverable
through the whole window.

**The static half now runs in CI**, which is the part that was missing.
`render-config.sh` already asserted that every `url_file` has a matching
`AM_CHANNELS` entry, but only at render time on the monitoring host — so
"added a receiver, forgot the renderer" failed a deploy rather than a pull
request. That half needs no secret and no host, so it is pure text and it
gates a PR. It checks both directions: a `url_file` nothing renders is the
#214 failure waiting to happen, and a rendered file nothing reads is a
secret written for no reason.

`--files` adds **non-empty**, which the existing assertion did not. A SOPS
key that is present but blank renders zero bytes, passes an `-f` test, and
makes Alertmanager POST to the empty string.

`--live` is the one that would actually have caught #214, and it is the
reason this is a check rather than an alert: it asks what the CONTAINER
can open, because there the files were present on this host and absent
inside the container. `make up` runs it after every deploy.

Deliberately does not ask Alertmanager whether it is healthy, and
deliberately sends no test notification. A check that depends on the
delivery path inherits the blind spot that made this last ten hours. Every
assertion reads a file. The dead man's switch is the other half of the
answer; it was armed and then tripped on purpose on 2026-09-09 — #288.

All three failure paths were exercised rather than assumed: a `url_file`
with no renderer, a rendered-but-empty file, and a container that cannot
see one. The `wc -c` probe returns 1 on a missing file, so the checker
reports it rather than crashing on empty output — checked, because that
branch is the one that runs on the bad day.

#341 Loki is not losing log data. 2026-09-06. The issue — which I filed — said Loki was discarding ~185,000 entries a week. Both halves of that were wrong.

**The arithmetic.** It summed `max_over_time` of a *cumulative* counter
over 7 days, which includes everything accumulated before the window.
113,630 of the 146,867 predated it. The true in-window increase was 33,237,
in a single event.

**The conclusion.** Every discard was an Alloy restart replaying history,
in three flavours of one event: `greater_than_max_sample_age` is the docker
source re-reading a container's log from the start — oracle's `db` has been
up nine months, so the replay carried an entry stamped 2026-08-11 against a
168h limit; `too_far_behind` is the journal source replaying its 24h
`max_age` against a stream already current; `rate_limited` is the same
replay arriving faster than Loki's default ingestion limit.

The timings settle it. oracle's agent started 02:37:07 on 09-04 and Loki
logged its rejection at 02:38:21; this host's started 02:54:07 and its
rejection landed at 02:54:15. The replayed entries are duplicates already
in the store, so nothing was lost — Loki refusing them is the system
working. Nothing has moved since: `greater_than_max_sample_age` flat for
two days, `rate_limited` zero over 15m, 1h and 3h.

`LogEntriesDropped` is reshaped around **duration rather than reason**. It
previously excluded `too_far_behind` by name, which was the wrong axis: it
silenced one flavour of a benign event, left the other two to page after
every deploy, and would have hidden a genuine persistently-behind stream.
A restart burst keeps `rate()` positive for the drain plus fifteen minutes;
`for: 1h` cannot be satisfied by that and is satisfied by loss that keeps
happening, so every reason stays in scope.

The regression test took two attempts to be worth anything. A single-step
fixture passes under both forms — the rate from one step is positive for
exactly the window length, one minute short of `for: 15m`. The committed
fixture climbs over six minutes, which is what a real replay does, and it
fails at `for: 15m` and passes at `for: 1h`.

#249 Watch the UPS self-test schedule. 2026-09-06. #93 set mjolnir to test itself every fortnight and nothing in the stack could see that setting, so the control keeping upsTestResultsSummary meaningful was itself unmonitored.

**The vendor MIB turned out not to be the cost the issue expected.** #249
framed adding PowerNet as "a pinning decision, not a URL" — Schneider
distribute it as a versioned download rather than a git ref, so the
options looked like vendoring 2.2 MB into the tree or trusting a moving
vendor path. Neither was needed: the observium commit this repository
**already pins** for UPS-MIB carries `mibs/apc/PowerNet-MIB` too. The line
added is a path against an existing pin, inheriting its argument unchanged.

One subtree, `1.3.6.1.4.1.318.1.1.1.7.2`, never the enterprise root — the
`pfTablesAddrTable` lesson one vendor along. Measured against the live
card: 7 rows, 2 GETBULKs, 0.07s to walk, and end to end the scrape goes
from 3 packets / 51 PDUs / 56 series to 4 / 58 / 63.

Two rules, and they are a pair on purpose. `UpsSelfTestScheduleOff` is the
fast signal, true the moment the card reads `never(5)`;
`UpsSelfTestStale` is the backstop for a card that says it is scheduled
and is not. Either alone leaves half the failure.

The enumeration was read out of the MIB rather than assumed, because an
older PowerNet enumeration stopping at `twelveWeeks(7)` is in circulation
and would read this card's `8` as out of range — the real definition runs
to `fiftytwoWeeks(12)`. A unit test pins `8` as quiet for exactly that
reason: a rule that fired for the estate's correct setting would be worse
than no rule.

`UpsSelfTestStale` counts distinct series over 21 days rather than using
`for:`, because `upsAdvTestLastDiagnosticsDate` is a DisplayString and
snmp_exporter renders it as a value-1 gauge carrying the date as a label —
so "the date changed" is "a second series appeared", and no date parsing
is needed. It is gated on `up{job="snmp"}` because a dead exporter
produces no new series either and would otherwise read as a card that
stopped testing.

**`tests/ups.test.yaml` did not exist**; nine rules were syntax-only,
which is the standing #63 objected to. It does now, with seven cases
covering both new rules — including the dead-exporter trap and the
long-schedule case that stops someone tidying the expression into
`<= 5 or >= 10`.

**A false positive shipped with it and was caught on the live stack twenty
minutes later.** `UpsSelfTestStale` fired immediately on deploy: a metric
that has just appeared has exactly one distinct value by construction, so
"one date in 21 days" was true with one sample in the window. The rule is
now also gated on `present_over_time(...[1d] offset 20d)`, so the count
only means what it claims once there are twenty days of history, with a
regression test. The cost is stated: it says nothing for the first twenty
days after the metric appears or after a Prometheus wipe, which is correct
because before then there is genuinely no evidence.

Still outstanding and not a monitoring question: the proof the schedule
*runs* rather than merely being set is the date advancing with nobody at
the card, due around 2026-09-11. As of 2026-09-06 it still reads
`08/28/2026`. `UpsSelfTestStale` is now what will say so if it does not.

#194 The journal is not under-delivered; the measurement was. 2026-09-06. The issue reported Alloy shipping ~1.5% of the host journal. It ships all of it.

The 1.5% came from `{job="/var/log/journal"}`, and the stream carries
`job="loki.source.journal.journal"` — Alloy overrides `job` with its own
component name, so the configured value survives only on `component`. The
query counted one label set and missed the other. #194's own last "worth
checking" bullet asked whether the count compared like with like; it did
not.

Compared like with like — `journalctl --output=json` entries against Loki
entries over identical windows — delivery is **100%** (6/6 over 10m, 59/59
over 1h, 2581/2581 over 6h) and was **98.8%** on 2026-08-31, the day the
issue measured 1.5%: 28,801 of 29,154 entries. An hourly reconstruction of
that day matches the host almost exactly once the `count_over_time[1h]`
one-hour stamp offset is accounted for.

**What the search did find**, and the reason the issue was worth working
rather than closing: Loki had been discarding around 185,000 entries a
week and nothing said so —
`greater_than_max_sample_age` 146,867, `too_far_behind` 26,398,
`rate_limited` 11,704. Tracked as
[#341](https://github.com/Gerrrt/HomeLab/issues/341); the root cause wants
splitting by source before any limit is changed.

#194's second acceptance box — "if the journal is genuinely
under-collected, something detects it" — is closed by two rules in
`stack.rules.yaml`, both with paired firing and quiet unit tests.
`LogEntriesDropped` fires on any rejection, with no tolerance band,
because a dropped line is evidence that no longer exists — except
`too_far_behind`, which is excluded and which is the difference between a
usable alert and one that fires on every deploy. An Alloy with no position
file replays up to 24h of journal on start and Loki rejects nearly all of
it against a stream that is already current: +19,490 discards from one
agent start, measured while testing #186. Those entries are duplicates the
previous agent already delivered. The cost of the exclusion, stated: a
stream persistently behind rather than briefly replaying is real and this
will not see it — that is the other half of #341.
`JournalSourceStopped` asserts `== 0` rather than a tuned threshold, and
that is only honest because the quietest host was measured: `Saruman`
reads 3.1 entries an hour at its slowest over 24h, against 37.6 here and
70.2 on `oracle`. The quiet test fixture is deliberately as slow as
`Saruman` really is, so a rule written as `< 5/hour` would fail it.

#309 Check shiva's iLO firmware against the documents. 2026-09-06. check_versions.py grows an OUT_OF_BAND table for devices that have a checkable version but are not hosts, and shiva is its only entry.

A table rather than a third bespoke comparison, which is what #309 asked
for and the reason is arithmetic: `morpheus` already needs its own
extraction because pfSense packs two versions into one string, this is the
second, and a fourth is how a script ends up unreadable.

The generic parser genuinely cannot do it. `os_key()` takes the first word
as the family and the first number as the version, so
`'Integrated Lights-Out 4 2.82 Feb 06 2023'` becomes `('integrated', '4')`
against the document's `('ilo', '2.82')` — both halves disagree, and the
running side is wrong in the way that matters, because the `4` is the iLO
generation and `2.82` is the firmware.

Proved it can fail, not just pass: with `network.md` edited to `iLO 2.79`
it exits 1 with *"says shiva runs 'iLO 2.79'; sysDescr reports 2.82"*, and
with an unreadable cell it says so rather than passing quietly.

`neo` stays out, and the reason changed underneath the issue: #309 said the
switch answers no `sysDescr`, and #310 gave it one. The answer is the
literal string `"Switch"` — no version — so there is still nothing to
compare, and a row would produce a permanent SKIP.

#339 Fail when a declared timer is not actually installed. 2026-09-06. make check-timers now asks systemctl is-enabled for every job in the JOBS table and fails naming the ones that are not.

Found while landing #335, and the finding was not the new job:
**`check-versions` had been declared since #292, had a correct unit file,
and had never been installed.** The weekly documented-versions check
simply never ran, for two weeks, and nothing said so.

Nothing *could* say so. `ScheduledJobNeverRan` is exactly the rule for
"declared but never ran" — `max_age unless on(homelab_job) last_success` —
and `max_age` is written by `--install`. A job added to the table and
never installed has NEITHER series, both sides of the `unless` are empty,
and there is nothing to alert on. The alerting was keyed on the installed
state while the table is what a pull request reviews, so a row merged
green and the job did not exist. Checks 1-5 compare the table against the
`.timer` files, which were present and correct throughout.

Read-only — `systemctl is-enabled` queries and changes nothing — so it is
safe inside `make validate`. It can only mean something on the deployment
checkout, so elsewhere it goes through `skip_offhost()` for the same
reason check 5 does: a CI runner failing because it has not installed the
monitoring host's timers would be nonsense. The header comment claiming
"there is exactly one" such skip was corrected in the same commit.

#335 Run the Loki coverage check on a schedule. 2026-09-06. homelab-loki-coverage.{service,timer}, daily at 07:45, plus a row in the JOBS table in install-timers.sh. No alert rule was edited: the staleness rules join against homelab_job_max_age_seconds, so a timer is a row in that table and nothing else — the property #99 tested and this confirms a second time.

**Daily, not the weekly the issue proposed, and the window went 7d -> 24h.**
#335 worried that a 7-day window against a weekly interval barely overlaps,
so a gap opening and closing inside a week could be missed. That framing
was wrong twice. The window is not a sensitivity dial — both sides of the
comparison use it, so a host that goes quiet leaves the denominator as well
as the numerator and the check goes vacuous rather than wrong. What it
actually sets is DETECTION LAG: `reach` counts lines over the window, so a
selector blinded an hour ago still looks reached until the last
pre-breakage line ages out, and 7d hides a new gap for a week. And the
overlap worry does not apply to the defect class at all — a rule going
blind is configuration, it persists until somebody fixes it, so tiling 24h
windows daily is enough.

24h is as short as the estate allows, measured rather than picked. Lines
per host: `Saruman` 15/hour and 186/day against `morpheus` 89,340/day, so
the quietest host is comfortably present. The cost, stated rather than
glossed: a host whose *subject* lines are rare — `Saruman` produced two SSH
accepts in seven days — has none inside 24h, so a real gap there reports as
a latent WARN instead of a live FAIL. Checked both ways against the
pre-#261 rules, where 24h still exits 1 on `morpheus`.

Verified end to end through the wrapper the timer actually calls:
`run-scheduled.sh --job loki-coverage` exits 0 in 2s and writes the four
`homelab_job_*` series, and a forced failure advances `last_run` while
leaving `last_success` behind — which is what lets the staleness rule fire
on repeated failures rather than only on a stopped timer.

#327 Fail when a Loki rule is blind to a host. 2026-09-06. scripts/check_loki_coverage.py, against the live Loki, outside make validate — check_loki_rules.sh boots the pinned image against a throwaway config with no data and asks whether the rules parse, which is the right question for CI and a different question from this one.

**No expectation table**, which #327 called the hard part and it is: a
table of which rule should see which host drifts, and a drifting table is
the defect this exists to catch. Two derived questions instead. Do the
rule's own stream selectors, unioned across its `or` branches, reach every
host shipping logs? And for a host they miss, do lines matching what the
rule *hunts* exist there anyway? The second grades the first — blind to a
host producing those lines is a live hole, blind to a quiet one is latent
— and it makes `useradd` never matching on FreeBSD `morpheus` answer
itself rather than need a row.

The subject query keeps the rule's positive filters (`|~`, `|=`) and drops
its negative ones (`!=`, `!~`), and that is load-bearing rather than
tidy. The negatives are policy — `SshLoginFromUnexpectedSubnet` excludes
`10.0.50.` and `10.0.99.` — and with them the rule matches nothing
anywhere, which is indistinguishable from being blind. Keeping them made
an earlier draft pass over #261.

Verified both ways against the live store. Against the rules as they stood
before #261 was fixed it exits 1 on `SshLoginFromUnexpectedSubnet`,
naming `Saruman` (journal, 2 lines) and `morpheus` (syslog, 46) and
warning that the four sibling rules cannot reach those hosts either.
Against the rules as they are now, all eight host-scoped rules reach all
four hosts.

The candidate selector is `{log_type=~".+", log_type!="docker"}` and
#327's trap is why. Loki logs its own query text, so `{host=~".+"}` makes
a rule match the string of its own line filter: 43,913 lines for
`SudoFailure`'s pattern over seven days against 2 real ones, 30,365 of the
difference being `service_name="loki"`. The label is required to be
*present* rather than only non-docker because 11,721 more are container
logs from before a labelling change that carry no `log_type` at all, and
`!=` matches a stream where the label is absent.

#310 Ask the switch what it is. 2026-09-06. sysDescr joins the mokerlink module; ifType was walked and deliberately left out. The issue proposed both and named its own failure condition for the second — "26 identical values would answer nothing" — which is exactly what came back: .1.3.6.1.2.1.2.2.1.3.N = 117 on all 26 ports, IANAifType gigabitEthernet, uplinks included. It does not distinguish the SFP cages, so it costs 26 varbinds a scrape and carries no information, and on this switch scrape volume is a safety property. Recorded in generator.yaml so nobody walks it a third time; the question it was for is answered by looking at the rack.

`sysDescr` went in, and the honest note is that it says nothing either:
`.1.3.6.1.2.1.1.1.0` is the literal string `"Switch"` with a trailing NUL.
No model, no firmware, no hardware revision. It is there because the
switch was the only SNMP target reporting no `sysDescr` at all, and
because it lights up on its own if a firmware bump ever populates it.

Measured rather than argued, twice. `snmp-walk.sh` first at the module's
request shape, per the note that file carries — 1 row, 2 requests, 0.53s,
the overshoot being `sysObjectID` answering `.1.3.6.1.4.1.27282` rather
than the zero-length OBJECT IDENTIFIER that makes this switch's
`ifSpecific` poison a whole response. Then end to end, because a walk
probe is not a scrape: the pinned exporter run against the live switch
with the regenerated config, beside a control container on the committed
one, three interleaved rounds each and identical every time — 31 packets,
131 PDUs, 1.46s and 136 series with it, against 30, 130, 1.45s and 135
without.

#292 Detect pfSense version drift from the box. 2026-09-04. sysDescr joins the pfsense SNMP module and scripts/check_versions.py compares what the documents claim against what the hosts report, weekly. The drift that prompted it — morpheus recorded as pfSense CE 2.8.1 on FreeBSD 15 in eleven places while running 2.9.0-RELEASE on FreeBSD 16.0-CURRENT — was corrected by hand in #283 after a human noticed, which is the standing this file says is not good enough.

**pfSense sets `sysDescr` explicitly**, so one string carries both
versions: `pfSense morpheus.matrix.elysium 2.9.0-RELEASE FreeBSD
16.0-CURRENT amd64`. It was absent only because the OID was not in the
module. Probed with `snmp-walk.sh` first, per the mokerlink note — 1 row,
1 request, 0.47s — and the generator resolves it to a GET of `.0`, so no
GETBULK is issued at scrape time. Regeneration added exactly one metric.

The check is deliberately out of `check_docs.py` and out of `make
validate`: that script compares documents to repo files and runs in CI,
which has no route to the stack. It compares release lines rather than
full strings, because the tables record `Ubuntu 24.04 LTS` where the hosts
report `24.04.4` and a check that fires the day after every update is one
that gets switched off. It also covers the single live version claim in
prose — `enable-suricata.md`'s "morpheus now runs 2.9.0-RELEASE" — which
#283 introduced while fixing this very drift.

**Deploying it found a bug that no stub could have.** `sysDescr` is not
`morpheus`-only: the `ilo` module walks it too, so the query returns two
series and `shiva` sorts first. The runbook check took the first series
carrying a three-part version, which is correct only because the iLO
firmware is `2.82` — two parts. An iLO numbered `2.82.1` and it would have
reported the BMC's firmware under the firewall's name: a confident,
specific, wrong finding, which is worse than no check.
[#306](https://github.com/Gerrrt/HomeLab/pull/306) selects by device
instead. Verified against the running stack after convergence: `morpheus`
moved from SKIP to PASS on both table rows and the prose claim.

What it does not cover is tracked rather than assumed:
[#309](https://github.com/Gerrrt/HomeLab/issues/309) for `shiva` and
[#311](https://github.com/Gerrrt/HomeLab/issues/311) for `Saruman`.

#105 Confirm the unconfigured Snort package actually went. 2026-09-04. It did. pkg info on morpheus lists pfSense-pkg-suricata and suricata and no Snort of any kind, so ADR-0006's line 49 was describing a fact and the runbook prerequisite asking for the removal was describing a job already done. The prerequisite is gone from enable-suricata.md §0; the ADR stands as written.

**What "removed" left behind is worth knowing before the next package is
uninstalled.** pfSense removed the package but honoured
`forcekeepsettings`, so `config.xml` kept a `<snortglobal>` stanza — and
inside it `snort_alerts:col2:open`, a widget record pointing at a
`snort_alerts` widget no longer on disk; only `suricata_alerts.widget.php`
is there. `/var/log/snort/` also survived, holding one 111-byte
rules-update log from 2025-10-30. The `snort`-named keys under
`<suricata>` — `snortcommunityrules`, `enable_snort_custom_url` — are not
residue at all: they are Suricata's own names for the Snort Community
ruleset options, both `off`, and were left alone.

**Both are now cleared.** `config_del_path()` and `write_config()` over
SSH for the stanza, `rm -rf` for the log directory, with an encrypted
off-host export taken either side. Not because the residue was dangerous
— nothing ran, updated or listened, and the live dashboard reads
`<widgets><sequence>`, which never referenced `snort_alerts`, so nothing
was even visibly broken. It went because of the one line in the stanza
that was not inert: `<forcekeepsettings>on</forcekeepsettings>` is what a
future `pkg install pfSense-pkg-snort` would have read its settings back
out of, so leaving it meant a reinstall resurrecting a half-configured
Snort rather than starting clean — the 1am mistake this issue was opened
about, deferred rather than closed.

Verified after the write: 88 user-defined rules, the same count the
pre-change export recorded; `<widgets><sequence>` byte-identical;
Suricata still on `igc0.20` and `igc0.10` under the same PIDs, never
restarted; web UI answering 200. `write_config()` leaves its own audit
line in the config revision log, so the word `snortglobal` still appears
once in `config.xml` — as the description of the change that removed it.

#97 Work out DNS for the MokerLink management UI so it is not reached by IP. 2026-09-04. Answered by ADR-0018, which splits the issue in two and grants one half. The name is a host override like any other — neo → 10.7.7.2 — and was never blocked behind ADR-0008, because ADR-0010 keeps the overrides on Unbound whatever AdGuard does. The certificate half is closed as unavailable rather than pending: the switch has no TLS listener and no way to import one, checked against the device on 2026-09-04. That is its third firmware limit after #84 and #85, and the argument for replacing it — where TLS management belongs in the selection criteria next to SNMPv3.
```
Both halves were already settled when this file still listed the granted
one as outstanding. Confirmed while verifying
[#123](https://github.com/Gerrrt/HomeLab/issues/123): `neo` → `10.7.7.2`
is one of six host overrides read off `morpheus` on 2026-09-04, and the
`via: dns` twin probes green from the running exporter (`probe_success 1`,
`probe_http_status_code 200`, lookup 0.9 ms). The entry outlived the work
by a day, which is the failure this section exists to prevent.
```

#100 Automate the Grafana dashboard export step. 2026-09-04. make dashboards-export pulls every dashboard back by uid and writes it over the file, so the loop is edit → one command → git diff rather than a hand copy out of the JSON Model panel — manual, and therefore skipped under pressure, which is what this file said about it.

**The issue's design could not work as written, and finding out why is
most of what this was.** The plan was to pull each dashboard from the API
and write it back. But `allowUiUpdates` was `false`, and that does not
mean what the issue assumed it meant: Grafana does not discard a UI edit
at the next restart, it refuses to *store* one at all —
`POST /api/dashboards/db` answers `400 Cannot save provisioned
dashboard`. So the API could only ever return the file it was provisioned
from. Every export would have been a clean no-op over the very edit it
existed to capture, exiting zero and writing nothing while `git diff`
reported no change to something plainly different on the screen. Measured
against the running stack before anything was built: the API's copy of
`homelab-docker` was identical to the committed file in every field but
`id` and `version`.

So `allowUiUpdates` is now `true`. The JSON stays the source of truth —
a file change re-provisions over Grafana's copy — but an edit now survives
long enough to be exported. What `false` bought for free was that the
running dashboard and the committed one could not disagree, and that is
bought back rather than dropped: `ARGS=--check` writes nothing and exits
non-zero when Grafana holds an edit git does not, and the daily
`dashboards-drift` timer runs it, so forgetting to export ages into a
stale job with an alert behind it. The rules in `backup.rules.yaml` join
against the `JOBS` table rather than naming jobs, so it needed no rule.

Two things found on the way, both of which would have made the diffs
unreadable. Grafana serialises keys **alphabetically** at every level,
while these files put `uid`, `title` and `description` first — a naive
write-back would have reordered every key in all seven and buried the one
line that changed, so the committed order is preserved and only new keys
are appended. And Grafana persists whatever the browser was showing at
save time, so the time picker's range and each variable's selection are
read back out of the file rather than taken from the API: without that,
one person's afternoon of debugging silently becomes everyone's default
time range. Those fields are the one thing the round trip will not write,
and the dashboards README says so.

The round-trip check the issue asked for boots the pinned Grafana image,
provisions the committed JSON into it and reads it back, which is what
keeps an export from arriving as noise. It also asserts that a save to a
provisioned dashboard is still accepted — the condition the whole feature
depends on and the one it cannot detect for itself, since flipping the
flag back breaks the export silently. Verified in both directions: with
`allowUiUpdates: false` the check fails and names the setting.
→ [`grafana/dashboards/README.md`](../stacks/observability/grafana/dashboards/README.md)

#94 Decide what oracle is for. 2026-09-03, ADR-0015. The issue's four options were answered by first checking the machine, which made one of them impossible: oracle has run the Lemmiwinks wiki and its Postgres since 2025-11-12, ADR-0011 depends on it, and blackbox has been probing it twice on host: oracle all along. Three places in the repository said it had no role at the same time — the host table in architecture.md, the VLAN 99 notes in network.md, and backup-firewall.sh's header ("It has no role (#94)"). check_docs.py reads that architecture row for the word "Alloy" and for a stacks/ path and never for what it claims the host does, which is how the wrong sentence sat next to a checked one.

**Decided:** it stays powered, and its role is the small off-host jobs —
work whose value is that it is not on the monitoring host. The wiki and
the firewall export copy it already has; the volume backup sets (#92) and
the dead man's switch watcher (#67) are added, decided here and built
under their own issues. **Rejected:** a second age recipient *on `oracle`*,
because a private key there would put the backups and the means to open
them on one disk and retire the property the off-host copy exists to
have — narrowed by
[ADR-0024](adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md),
which keeps that rejection and makes a second recipient held *off* the
estate the design (#106); its holder was decided on 2026-09-08 under
[#294](https://github.com/Gerrrt/HomeLab/issues/294). Also rejected: the ADR-0007 stack, the
ADR-0008 tier, bringing `wlp22s0` up to give the watcher an independent
path — that dual-homes a VLAN 99 host onto an untrusted segment — and
switching the machine off, which was never really on offer: the wiki had
been running there for nine months when the issue was filed.

Measured rather than quoted, 2026-09-03: 3785 MiB of RAM with 2549
available under the wiki, its database and Alloy; a 465.8 GB 5400 rpm
disk carrying one 100 GB LV with 67 GB free and 362 GB unallocated; and a
NIC that advertises 10/100 only, so the link is 100 Mb/s and no cable
will change that. A 867 MB backup set is 75 seconds of wire time there;
seven of them are 6.1 GB, and they are already `age` ciphertext before
they leave this host.

#93 Replace the UPS battery, delete the silence, put the card under scheduled test. 2026-09-03. The first two steps were done on 2026-08-28: an APCRBC115 into mjolnir, upsTestResultsSummary 4 (aborted) → 1 (donePass) at 22:45 UTC, and the UpsSelfTestFailed silence 54f1715c-e57b-4322-8a6d-5435bc8e1bd8 deleted at 23:14 rather than left to lapse on 2026-09-20 — nine minutes after the proving reading instead of before it, which is the inversion the runbook exists to prevent and which cost nothing only because the test passed.

The third step turned out to need checking rather than doing. The card
reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), with
`upsAdvTestDiagnosticsResults` `1` (ok) and
`upsAdvTestLastDiagnosticsDate` `08/28/2026`. Six files had been asserting
the opposite since the fit; they now say what the device says. Whether the
schedule was set at the rack or has been the default all along, the NMC
will not say after the fact.

**Nothing watches it.** All three are PowerNet OIDs and the `apc_ups`
module walks the standard UPS-MIB only, so a card that reverts to
`never(5)` produces no alert and no changed metric — `upsTestResultsSummary`
would simply hold `1` forever. Closing that means adding APC's MIB to
`scripts/snmp-mibs.sh`, which is a new vendor source with its own pinning
decision; it is filed as
[#249](https://github.com/Gerrrt/HomeLab/issues/249) rather than done
here. Until it is, the check is
`scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`, and
what proves the schedule *runs* rather than merely being set is that date
advancing unattended, due around 2026-09-11.

Two smaller findings. `upsBasicBatteryLastReplaceDate` still reads
`08/15/2026` for a pack fitted on the 28th, so the card's battery-age
accounting is keyed to a date on which its own self-test was still
aborting over an empty bay. And runtime is a poor proof of a real pack on
this UPS: since the fit it has sat on exactly `63` — the fabricated
value — for 744 of 764 samples. Voltage moving across `540`-`549` is the
comparison that actually discriminates.
→ [runbook](runbooks/fit-the-ups-battery.md)

#91 Probe the services this was filed for, and probe TLS expiry. 2026-09-03. Seven named, five probed, plus the one the sentence about "four devices" implied: Grafana by name and by address through a new http_2xx_lab_ca module that verifies the chain against certificates/ca.pem (now mounted into blackbox-exporter, CA only); Prometheus and Loki at the address the agents push to, so a mis-set BIND_ADDR fails the probe while every up stays green; Alertmanager on the compose network, the only network it is on; the switch UI, which is plain http and drops 443; and the APC card, https-only with a self-signed certificate, through http_2xx_self_signed. TlsCertificateExpiringSoon at 30 days and TlsCertificateExpiryImminent at 7 read probe_ssl_earliest_cert_expiry off the handshake, aggregated per certificate so Grafana's two URLs raise one alert, the critical inhibiting the warning. Every enabled target was probed through the new modules from a throwaway exporter on the compose network before landing, and the lab-CA module was checked to refuse the APC card's certificate.
```
Two of the seven are written into `targets/blackbox.yaml` and disabled.
From `10.0.99.20` the iLO and the pfSense UI both time out, and neither
is a fault: VLAN 99 → 30 passes SNMP and nothing else, and "Block HTTPS to
pfSense" on igc0.99 is an explicit rule. Each needs one pass on the
Winterfell interface, written out beside the target, and each is a
segmentation decision to record in ADR-0013's table when made — the
pfSense one hands a host with two unauthenticated push ports a path to
the firewall's login page. #235 kept the iLO on the lab segment
(ADR-0033), so the iLO probe still needs its `99 → 30:443` pass.
```

#126 Notice when the house stops filtering DNS. 2026-09-04. Also the monitoring half of #123, which describes the same failure and whose entry under Monitoring has been corrected: the rules were writable today, only the target was not. ADR-0010 left the public upstreams in Unbound's forwarder list so that a dead AdGuard costs filtering rather than connectivity, and that trade converts a loud failure into a silent one: the only symptom is advertisements reappearing, which nobody reports. Two dns modules in blackbox/blackbox.yaml and two rules in a new dns.rules.yaml. AdGuardNotAnswering asks whether it answers at all; AdGuardNotFiltering asks whether it is still blocking, which is the quieter failure — a blocklist that silently stopped updating leaves a perfectly healthy service filtering nothing. Both warning, so they route to default rather than urgent: nothing is down and nobody is blocked.

**The whole design is in where the probe is aimed.** A query through the
normal resolver path always passes, because the fallback is doing its
job, so both probes go straight at port 53 on the filter under their own
scrape job — separate from `blackbox` because `EndpointUnreachable` is
critical and routes to `urgent`, and because `EndpointNameNotResolving`
joins on `name` in a way that means nothing when the target is itself a
resolver. `AdGuardNotFiltering` carries an `and on (name)` join so a
stopped AdGuard raises one alert and not two, and `tests/dns.test.yaml`
asserts that silence — the #63 hazard, since a join that stops matching
passes `promtool check rules`.

All four states were measured against a real AdGuard Home running the
default filter before either rule was written: healthy, blocklists not
loaded, blocking mode changed, and stopped — plus the trap itself,
Unbound at `10.0.99.1`, which answers the liveness probe and fails the
filtering one. The canary is `doubleclick.net`, a real blocklist entry
rather than a rule of our own, because a custom rule would keep working
while every downloaded list failed to load.

**The targets are written out and disabled**, the same call #91 made for
the iLO and pfSense UIs: AdGuard does not exist until #102 buys the mini
PC, and a probe enabled today would report a service down from the moment
the file loaded. The scrape job ships anyway, so turning them on is a
file_sd append with no restart. Enabling them belongs to #102.

#90 Detect Suricata being dead. 2026-09-03. The roadmap line said the SNMP module does not expose a heartbeat; the firewall's agent already did. bsnmpd on morpheus loads snmp_hostres.so, so HOST-RESOURCES-MIB hrSWRunTable is served — 94 rows, 0.04s for a full walk — with one suricata row per interface and the interface in hrSWRunParameters (-i igc0.20 …), indexed by pid+1 and renewed on every rule update. The pfsense module now fetches those rows and nothing else from the table, through a dynamic filter on hrSWRunName, with a DisplayString override because both string columns are InternationalDisplayString and would otherwise arrive as hex. SuricataStopped in prometheus/rules/ids.rules.yaml is "declared but not running" per interface, gated on the scrape being up, unit-tested against the rows as morpheus read them. It proves the process is alive, not that it detects; the runbook's test alert still owns that.
```
Proved live the same day. The Degens instance was stopped at 04:49:59
UTC; `SuricataStopped` for `igc0.10` alone went firing at 05:00:49
(10m50s) and reached the `security` receiver, Skids stayed quiet. Started
again at 05:01:12, resolved at 05:03:00 with the row back under index
`90324` for pid `90323`. A full package restart at 05:03:23 — the shape
of the daily rule update — never reached firing. Scrape cost for the
module went from nothing measurable to 0.26s.
```

#89 Extend Suricata to Degens (VLAN 10). 2026-09-02. Second interface, twelve days after the first, alert-only on both. Its alerts are told apart from Skids' by the syslog facility — the one per-interface setting pfSense puts on the wire — which syslog.alloy maps to interface; both Loki rules and the dashboard split on it. Proven per interface with the §4 test rules: the first real alert arrived two seconds after the engine started, labelled, and was the same stream-timestamp signature that is 79% of Skids.

Two things the pipeline test found on the way. *Restart* on the
interface left the process stopped until started by hand, and loading
the ruleset then took 35 seconds — a test visited in that window fires
nothing and looks like a broken pipeline. And a guest iPhone with
iCloud Private Relay produced no plaintext DNS or HTTP at all, so neither
test rule could fire from it; the engine's own `alerts.log` on the box is
what separates "never fired" from "fired and lost". Both are in the
runbook now. The fortnight comparison of that shared signature is dated
2026-09-16 there. → [runbook](runbooks/enable-suricata.md)

#76 Replace shiva's Smart Storage Battery. 2026-09-02. Spare 815983-001 fitted; Saruman was off 22:30–22:58 UTC. The first scrape after it came back read chassis 0 battery 1 at cpqHeSysBatteryCondition 2 (ok) and cpqHeSysBatteryStatus 1 (noError), down from 4 and 13, with serial 6EZBN0FB2431YM where the failed pack was 6EZBN0CB29N3YZ — a different part being read, not the old one reading differently. The Smart Array re-enabled its cache on the same scrape: cpqDaAccelStatus 5 → 3 (enabled), cpqDaAccelBackupPowerSource 1 → 4 (smartbattery), cpqDaAccelBattery 6 → 2, and the three controller rollups that had read 3 (degraded) since 2026-08-18 back to 2. cpqDaAccelBadData stayed 2: nothing dirty was lost at either end.

The silence `bfdfff66-d9c3-4df4-9495-f1f38ebf93c1` was deleted at 23:11
UTC rather than left to expire on 2026-10-01, so `IloBatteryCondition`
and `IloWriteCacheDisabled` are live again over the new part. Deleted
nine minutes *after* the proving reading, not before — the UPS
inversion again. It cost nothing because both rules carry a `for:`
longer than nine minutes, which is the rule shape being kind, not the
procedure working.

**Two readings did not move, and neither is alerted on.**
`cpqDaAccelWriteCachePercent` / `ReadCachePercent` still read `0` / `0`
ten minutes in, where a P440ar normally reports a ratio; the controller
says the cache is enabled and reports no split. And
`cpqDaAccelFailedBatteries` still reads `1`, unchanged from the failed
pack. Both are named in the runbook with the `ssacli` check on `Saruman`
that resolves the first; the metrics before 2026-09-02 show the failed
pack and a write-through array, which any range crossing that date will
include.
→ [runbook](runbooks/replace-the-smart-storage-battery.md)

#88 Deploy Alloy to Saruman and oracle. 2026-09-02. One script, scripts/deploy-agent.sh, for both: the compose service written out as docker run for a Docker host, and the .deb matching the compose tag for a host that should not run Docker, which a Proxmox hypervisor is. The agent config became a directory of three files so the native package loads no Docker components and opens no syslog port on the hypervisor's real interface.

`oracle` turned out to have been the actual gap. It was deployed by hand
on 2026-08-30 and by 2026-09-01 was running a version behind compose, as
`--privileged`, with the config it was copied with — no self-scrape, so
`oracle-alloy` never existed and nothing said so — and no volume for its
WAL. The runbook's verification could not fail: Alloy logs to stderr and
the grep read stdout. Redeployed with the script; three jobs now.

`Saruman` needed a decision, not a deploy: ADR-0007 said it does not
remote-write to Winterfell and ADR-0012 assumed it does. Resolved for the
hypervisor's own telemetry only, over one unlogged pass above the
ADR-0014 tripwire; both ADRs carry a note. The rule and the run are the
operator's, from Hicks, since 99 → 30 is closed.

Two more things found on the way, neither fixed here. Since #188 every
Docker-host agent logs cAdvisor's `rootDiskErr` every few minutes —
root without `DAC_READ_SEARCH` cannot always size overlay layers —
partial, nothing charted depends on it, and nobody wrote it down; the
deploy gate ignores exactly that line. And a throwaway `alloy run` on the monitoring host resolves
`prometheus` through the host's DNS and pushes into the live stores:
four `instance="smoke"` series and a silenced `RemoteWriteJobStale`,
because there is no admin API to delete them.

#87 Add ifXTable (64-bit counters) to the mokerlink module. Swapped in rather than added: the 64-bit ifHCInOctets/ifHCOutOctets replaced the 32-bit pair, so the walk stayed at five columns and the load on a switch that has wedged under polling stayed where it was. Each column was fetched first with the new scripts/snmp-walk.sh, at the exporter's own request shape, and the low 32 bits matched the live counters on every busy port.
```
Two things found on the way went with it. `SwitchCounterWrapSuspected`
could never fire — `rate()` never goes negative — so it was deleted, not
re-pointed. And no switch metric had ever carried an `ifDescr` label: the
generator comment said "label lookup" but no lookup existed, so every
port name in the dashboard and the `SwitchInterfaceDown` summary rendered
blank. One `lookups` stanza fixed both.
```
#86 Decide whether the lab VLAN needs egress filtering. No. ADR-0014. The question and #96's deferred isolation mechanism were one decision, and the fact that decided it is that the techniques the estate exists to detect are layer 2 — poisoning, spoofing, rogue DHCP — and do not cross a router. So the attacker shares the segment with the Windows domain, and the vulnerable targets get no route at all: a bridge inside ifrit with no physical port, on a subnet the firewall does not know. A port allowlist would have passed C2 on 443 and broken Kerberos by blocking NTP; a range VLAN would have routed every scan through the box that runs the house and hidden the layer-2 techniques from the estate.
```
Two things it found and did not decide: the iLO of the defended estate
is now on the attackers' segment (#235), and the segment gets a log-only
tripwire like the three terminal ones (#234).
```

#104 A superseding ADR for 0002's rule count. ADR-0013. The issue asked for the third rule and the current total; reading the enforced ruleset instead of recounting the prose showed the total was the wrong thing to ask for.

Default deny holds for 99, 30, 40, 20 and 10. It does not hold for Hicks,
which blocks 40/20/10 and then passes to `any` — so it reaches all of
Winterfell and all of ImaginationLAN, wholesale, which no rule grants and
no rule denies. Nor for the switch LAN, which still carries pfSense's stock
*Default allow LAN to any* and reaches every segment while `network.md`
said "Nothing". One explicit rule — *Allow Hicks access to ImaginationLAN*
— sits on the ImaginationLAN interface, where Hicks traffic never arrives,
and matches nothing.

Four documents held four different counts. They now hold a list.
`network.md` had Hicks right the whole time.

#223 TerminalSegmentReachedInternalNetwork could not fire. The firewall logged blocks only, and the rule matches action="pass", so the alert the segmentation design depends on could not fire for any input — the #63 shape, found while building #82's dashboard.

Armed with three tripwire rules rather than by logging passes broadly.
The obvious fix does not work: every inter-VLAN pass rule is sourced from
an internal segment, so none can carry a terminal-VLAN packet. Logging
the terminal `→ any` egress rules does work and costs ~12.6M lines a day,
about 142× current volume. The tripwires — `pass` + `log` for
`<terminal net> → Internal_Segments` on `igc0.10/20/40`, below the blocks
and above `→ any` — cost nothing while segmentation holds and match only
if the blocks are removed or reordered.

The logging path was proven rather than assumed: 49 of 49 pass lines put
the source where the alert's regex reads it, and the destination half
already matched 2058 block lines.

#82 A dashboard for the Suricata and firewall-log labels. homelab-security, 21 panels, all from labels config.alloy was already extracting and five Loki rules were already firing on. It charts blocks per second by interface and direction, top blocked sources — parsed out of the line at query time, because ADR-0003 keeps addresses out of the index — Suricata by classification and priority, and terminal-segment violations, which is the rule the segmentation design exists to enforce and which had no view but the alert.

It also closed a gap it walked into: dashboard PromQL had been parsed by
promtool since #78, and dashboard LogQL had been parsed by nothing, so a
typo in a Loki panel rendered an empty panel and read as quiet traffic.
`check_dashboards.py --emit-logql` now feeds every panel query to the
Loki that `check_loki_rules.sh` already boots — 28 expressions, including
the eleven in `homelab-logs` that had been unchecked since it landed.

Not captured by `make screenshots`, and never will be, for the reason
`homelab-logs` is not: three of its panels exist to show real addresses.

#81 A dashboard for the observability stack itself. homelab-stack, 33 panels, all from metrics already collected. The argument in the issue was that two of the three faults found while verifying #12 would have been visible on it immediately, and the panels that would have shown them are the two the dashboard is really built around: Samples returned per scrape, where cAdvisor collapsing from hundreds of series to one is a step change while up stays 1, and the Alloy remote-write lag, where a stale address shows as a climbing line rather than as nothing at all. Three things came with it because the dashboard could not be honest without them. The five rules watching the stack's own components were carrying component: containers and so were filed under Container alerts on the Docker dashboard; they are now stack.rules.yaml on component: stack, a relabel with the expressions untouched. Each Alloy agent now scrapes itself and remote-writes the result, because prometheus.yaml could only ever reach the agent on this host — oracle publishes Alloy's port on loopback and there is no address to point at. And check_docs.py was matching spelled counts in lowercase only, so "Five dashboards are provisioned" was unguarded while "There are five dashboards" two files away was checked; both were stale together. One thing the dashboard made obvious was that up is not a liveness signal for the jobs that arrive by remote_write: a pushing agent that dies stops pushing, so its series ages out rather than falling to 0, and InstanceDown is up == 0. RemoteWriteJobStale closes that, and is worth reading for how it is written rather than what it covers. The obvious form, a threshold on the staleness the dashboard graphs, is unfireable for the same reason #63 was — an instant selector stops returning a sample after the lookback delta, so the difference never reaches the threshold. Both that form and the version with the guard dropped were run against the tests and both fail them. What ships asks which jobs were reporting in the last 24 hours and are not reporting now. It was then watched working rather than argued: the agent on the monitoring host was stopped for six minutes with the rule silenced, and the alert went pending at t+282s having returned nothing at all for the four and a half minutes before that — the lookback delta, and the real blind window on any remote-written target. The residual is in the 24-hour window: an agent away longer than a day resolves the alert falsely, having notified at least twice first. That is the price of matching on the job-name convention instead of a list, and the list is what would silently miss Saruman when it arrives (#88). Two panels were added afterwards, which is what closed the issue. The Alloy row charted throughput, lag, component health and forwarded lines but not the WAL, which the issue had asked for by name — WAL size and append rate and WAL replay and corruption now do, and the replay one earns its place by dating an agent restart to the minute, the context that is missing when the lag panel jumps and nothing says why. Adding them also found that the screenshot this dashboard has been waiting for could never have worked. homelab-stack renders 4582px tall against a BROWSER_MAX_HEIGHT of 3000, so a capture would have come back cropped at the Alertmanager row — with the Alloy panels, the reason to shoot it at all, off the bottom — and reported success. The ceiling is raised in compose.yaml and in the script together, and the dashboard is in DASHBOARDS; it is still unshot, because the window wants a clean day behind it rather than the hour after a deploy.
#77 Schedule something. Four systemd timers are written to run make backup weekly, make backup ARGS='--verify-only --all' and make backup-firewall nightly, and make snmp-verify weekly; make check-digests runs weekly in GitHub Actions, which is the only one of the five that is genuinely off-host. Installing them on the monitoring host is a separate step (make install-timers) and it was missed — for the first days of this entry's life the sentence above was in the present tense and simply untrue, no unit was installed, and no scheduled job had ever run (#215). make validate passed throughout, because the check it ran compared two copies of the schedule that both live in git. It now also asks the host. Every run records its outcome as a metric, so five rules in backup.rules.yaml alert on a job having stopped being run rather than only on one that failed — which was the actual ask. make secrets-verify-backup deliberately has no timer: it needs a human to mount removable media, so it gets a ninety-day deadline and an alert instead. What this does not solve is that the host still verifies its own backups — that is #92 and #99, both still open. → runbook
Enable Suricata on morpheus. Running on Skids (VLAN 20) alert-only since 2026-08-21; alerts reach Loki with classification and priority parsed into labels, verified against real traffic. First tuning decision made from measurement rather than prediction: sid 2200121 "Ethertype unknown" was 100% of alerts and turned out to be LLDP from neo, suppressed by signature rather than by disabling the whole decoder-events category. → runbook · ADR-0006
Turn on remote logging on morpheus. The firewall now ships filterlog to Loki, so TerminalSegmentReachedInternalNetwork and IoTAttemptedLateralMovement have input for the first time. pfSense saved the settings without restarting syslogd and sent nothing until the page was saved a second time — the runbook now says so, and says to confirm on the wire with tcpdump before believing an empty query. → runbook
Settle the 10.0.30.10 question. It is the iLO BMC on its dedicated port; the Proxmox host is Saruman at 10.0.30.110. The SNMP target's hypervisor-bmc role label was correct all along — the inventory was not
Bridge mode on the ISP gateway
Lock down guest VLAN firewall rules
Move IoT devices onto their own SSID and VLAN
Stand up Prometheus, Grafana, Loki, snmp-exporter and Alloy
Consolidate five broken compose files into one working stack
Provision Grafana datasources and dashboards from files
Add alerting (34 rules) and Alertmanager routing
Move secrets to SOPS + age
Add CI: lint, config validation, secret scanning
Pin every image by digest, not just tag, with drift detection in CI
Add SECURITY.md with a disclosure policy and known-exposure summary
Loki alerting rules (13) for auth, SSH brute force and disk/OOM events, validated in CI by booting the pinned Loki image against them
Replace the CA and leaf certificates that leaked, and add tooling so issuing one is a command rather than a research project
Serve Grafana over TLS with that CA, verified end to end — Prometheus scrapes it with ca_file and server_name rather than insecure_skip_verify
Point Alertmanager at a real receiver. The webhook was the ntfy.example.invalid placeholder for the entire life of the stack, so no alert had ever been delivered
Surface firing alerts on the dashboards. Forty rules and one routing tree existed with nothing showing them; four dashboards now carry a table of their own component's alerts
Stop the UPS dashboard reporting a battery that is not there — the management card fabricates charge, runtime and status
Purge the shared SNMP community, the inline Grafana password and the TLS private keys under certificates/ from git history, and delete the .gitleaksignore that acknowledged them
Give every SNMP device its own community and rotate all four on the hardware, confirming pfSense, the APC NMC and iLO each refuse the old shared string. The switch accepts its new one but also still holds its previous community — an accepted residual, recorded in SECURITY.md

FilesExpand file tree

roadmap.md

Latest commit

History