From 11a9c4879d65f2361551558f2ef430981a44608d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 18 Sep 2026 22:15:15 +0000 Subject: [PATCH 1/4] =?UTF-8?q?docs(runbook):=20=C2=A71=20gains=20the=20pi?= =?UTF-8?q?n-3=20trap,=20and=20the=20connectors=20are=20named=20(#413)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Exos pair landed on 2026-09-18 and the first question off the bench was what the connectors are, because a helium drive held PCB-up does not look like a SATA drive. They are SATA — 15-pin power, 7-pin data, and Seagate's four-pin jumper block, which stays empty. The paragraph that matters is the one the runbook never carried: enterprise SATA drives read power pin 3 as Power Disable, and a desktop supply that puts 3.3 V on it holds the drive off. It never spins and never shows in lsblk, and the boot SSD working is no evidence either way, because the S3520 predates the feature. Five wires with an orange one is the tell; Kapton over pin 3 or a Molex-to-SATA adapter is the fix. Refs #413 Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-nas.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index b19ee424..c738fd05 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -291,6 +291,22 @@ stay free. **Do not disturb the bay fan on `AUX1_FAN`.** It is the airflow over these two drives, and two 7200 rpm Exos under a scrub will want it. +**Look at the power lead before the case closes.** The Exos, like every +enterprise SATA drive since about 2016, reads **pin 3 of its power connector as +Power Disable**: a supply that puts 3.3 V on that pin holds the drive off, so it +never spins and never appears in `lsblk`. A desktop supply's SATA lead with +**five wires, one of them orange**, carries that 3.3 V; four wires and no orange +does not. If the orange wire is there, put Kapton tape over pin 3 on each +drive's power receptacle, or feed the bays through a Molex-to-SATA adapter, +which has no 3.3 V to offer. The boot disk booting proves nothing here — the +S3520 predates the feature and ignores the pin. A drive missing from §2's +`lsblk` is this before it is a cable or a port. + +The connectors themselves are ordinary SATA, and it is worth saying because +the drive held PCB-up does not look like it: the wide receptacle is the 15-pin +power, the narrow one the 7-pin data, and the small four-pin block beside them +is Seagate's jumper header, which stays empty. + ## §2 — Read the drives before trusting them From **option 8, Open Linux Shell**, at the console — **not over SSH**. From a76846f4a7f41dfcb34171ba7c12f921db615cbd Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 19 Sep 2026 03:53:04 +0000 Subject: [PATCH 2/4] feat(nas): the drives are in, erebor exists, the stack runs and the scrape is on (#413) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Exos pair landed on 2026-09-18. What was read rather than assumed: both drives PASSED at 0 power-on hours by SMART and — the reading §2 did not know to ask for — 0 hours and 0 spindle hours by the FARM log, which a SMART reset does not touch. One arrived carrying a Windows quick format and a short self-test at hour 0, which is a seller's bench and nothing more. Serials and counters go into hardware.md, and the Compute table's Storage column reads the mirror. The pool is a mirror with encryption off by decision, written into §3 with the reason. The datasets are on the SMB and Apps presets; the share is `media` and its user is `bilbo`, SMB-only. The one ACL entry that had to be added was Jellyfin's, not bilbo's: everyone@ Read, because the container reads the library as nobody and the preset grants only builtin_users. §6 as written could not run on this box. `make up` renders secrets this stack does not have, on a host without make or sops, so the section now says what was done: give Apps the pool, fetch the two files from main into /mnt/erebor/apps/stack, and `docker compose up -d`. It also records that Jellyfin's named volume lives on erebor/ix-apps and not on erebor/apps, which #484 has to know before it decides the backup. The scrape target in targets/node.yaml is uncommented, after the three checks its header demands passed from the monitoring host: 9100 open, 443 and 8096 refused, 36 filesystem series including erebor and both datasets, and no node_network_* series. Inside the container the render node is present and the process carries GID 107; the transcode itself is the half of ADR-0040's condition still to run. Refs #413 #138 #484 #256 Co-Authored-By: Claude Fable 5.1 --- docs/architecture.md | 2 +- docs/hardware.md | 41 +++-- docs/network.md | 6 +- docs/observability.md | 2 +- docs/roadmap.md | 12 +- docs/runbooks/build-the-nas.md | 142 +++++++++++++++++- stacks/media/README.md | 22 ++- .../prometheus/targets/node.yaml | 47 +++--- 8 files changed, 215 insertions(+), 59 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index c36f83df..0520f94e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -191,7 +191,7 @@ hole from the monitoring VLAN into the monitored one. | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | **Not built yet** — the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live; the target itself stays commented until there is a pool to run the exporter from | +| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/hardware.md b/docs/hardware.md index 667ae263..dfca29d8 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -43,7 +43,7 @@ quietly swapped. | `Saruman` | HPE ProLiant DL360 Gen9 | 2× Xeon E5-2680 v3 (48 threads) | 128 GB | 2× 1 TB SAS HDD, RAID 1 | Proxmox VE 9 | | `prometheus` | Apple MacBook Pro (2012, Retina 13") | i5/i7 | 8 GB | 256 GB SSD | Ubuntu Server 24.04 LTS | | `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04 LTS | -| `smaug` | Lenovo ThinkServer TS150 | Xeon E3-1225 v6 (4 cores) | 8 GB ECC | 240 GB SATA SSD (boot) | TrueNAS 25.10 | +| `smaug` | Lenovo ThinkServer TS150 | Xeon E3-1225 v6 (4 cores) | 8 GB ECC | 240 GB SATA SSD (boot) + 2× 18 TB ZFS mirror `erebor` | TrueNAS 25.10 | The observability stack runs on a thirteen-year-old MacBook. It handles four SNMP devices at a 60-second interval, four Alloy agents, and 30 days of metric @@ -194,10 +194,11 @@ revisions of this repository treated `shiva` as the hypervisor itself. has created would be the kind of claim this table exists to not make. The boot disk the TrueNAS install wants ([ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md)) and the bracket that carries it in the optical bay are the entries below and - landed with it; the two drives for the mirror, bought 2026-09-11, have not. Nothing + landed with it; the two drives for the mirror, bought 2026-09-11, landed on + 2026-09-18 and were in the bays that evening. Nothing for this machine is outstanding on the roadmap's - [list](roadmap.md#everything-still-to-buy) any more; what is left is those - two drives landing, and a build. + [list](roadmap.md#everything-still-to-buy) any more, and the Storage column + reads the mirror since 2026-09-19. **Read off the machine on 2026-09-15**, where everything above it came off a listing: `ThinkServer TS150`, machine type-model `70UB000AUX`, serial `MJ05N4NK`. Xeon E3-1225 v6 at 3.30 GHz, four cores, and `Active Video: IGD` @@ -248,7 +249,8 @@ revisions of this repository treated `shiva` as the hypervisor itself. access ZFS wants and the thing [#418](https://github.com/Gerrrt/HomeLab/issues/418) is the cautionary tale for, and `CSM [Disabled]`, so it boots UEFI as TrueNAS wants. - Two 3.5" trays, both empty — exactly the mirror and no spare. + Two 3.5" trays, filled by the Exos pair on 2026-09-18 — exactly the mirror + and no spare. **The 5.25" bay was not empty**: a PLDS `DVD-RW DU8AESH` answered on SATA5. A photograph of the open case had been read here as an empty cage and was wrong; the BIOS summary is what caught it. The optical drive came out on @@ -257,15 +259,30 @@ revisions of this repository treated `shiva` as the hypervisor itself. that fan is **not optional**: it is the airflow over the drive bays, and two 7200 rpm Exos under a scrub will want it. Reconnected after the swap and reading `Aux Fan: Operating`. -- 2× Seagate Exos X20 18 TB (`ST18000NM003D`), 3.5" SATA — bought - 2026-09-11, in transit. `smaug`'s ZFS mirror +- 2× Seagate Exos X20 18 TB (`ST18000NM003D`, firmware `SN03`), 3.5" SATA — + bought 2026-09-11, **in hand since 2026-09-18**, a day after the carrier's + window lapsed. `smaug`'s ZFS mirror ([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#413](https://github.com/Gerrrt/HomeLab/issues/413)). A mirror of two is - one drive's capacity, so this is 18 TB usable, not 36. The listing's - **zero power-on hours is a claim, not a fact** — read it back with - `smartctl -a` on arrival and record what the drives actually report here, - before the mirror is built on them. Serials go here when they land. They - enter the Compute table with the NAS, which is not built. + one drive's capacity, so this is 18 TB usable, not 36. + **Read off both drives with `smartctl -a` at the console on 2026-09-18, + before the pool existed**: serials `ZVTBS4NL` and `ZVTBSDL3`, both `PASSED`, + both at **0 power-on hours**, 0 reallocated, 0 pending, 0 uncorrectable, no + errors logged, 26 °C in the bays against a lifetime range of 24–26. **The + zero hours is a fact, not a claim, and it was checked the one way that + settles it.** SMART's hours counter can be reset, and used Exos drives with + it zeroed were sold as new through 2025; Seagate's FARM log keeps a second + counter that the reset does not touch, and `smartctl -l farm` on this host + read **0 power-on hours and 0 spindle hours on both drives**. What the + resettable counters add is consistent with new drives bench-checked by the + seller and with nothing more: `ZVTBSDL3` arrived carrying a Windows quick + format — a 16 MB Microsoft reserved partition and an NTFS volume labelled + `New Volume` filling the rest — with 718,258 LBAs written, about 370 MB + and the size of that format, three power cycles, and one short self-test + logged at lifetime hour 0; `ZVTBS4NL` arrived blank, two power cycles, + nothing written. Both spun up and enumerated on the first power-up + (`build-the-nas.md` §1 records the pin-3 trap they could have hit). They + enter the Compute table with the pool. - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot disk, carrying TrueNAS and the media stack it launches diff --git a/docs/network.md b/docs/network.md index 551b3fef..416122df 100644 --- a/docs/network.md +++ b/docs/network.md @@ -299,9 +299,9 @@ Televisions and consoles. Internet only. [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) - ([#413](https://github.com/Gerrrt/HomeLab/issues/413)). Its ZFS mirror does - not exist yet, and `stacks/media` is authored and undeployed against that - absence; what exists is a host on its address. **It does not change the + ([#413](https://github.com/Gerrrt/HomeLab/issues/413)). Its ZFS mirror + `erebor` exists since 2026-09-18 and `stacks/media` runs on it since + 2026-09-19, publishing `8096` to this segment. **It does not change the *Reaches* column**, and that is the point ADR-0016 made in advance: nothing on this segment initiates anywhere, and the rules created that day all let a more trusted segment reach **in**. That is the direction this row records, diff --git a/docs/observability.md b/docs/observability.md index e6931280..b91e02c2 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -453,7 +453,7 @@ argument and for what to do when it exits 1. | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | -| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` is written into `targets/node.yaml` and left disabled until the NAS has a pool to run the exporter from. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | +| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) | diff --git a/docs/roadmap.md b/docs/roadmap.md index d74a4414..c9f93e73 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -789,11 +789,13 @@ what left this one unfireable for months. the BIOS flashed, AMT found on Intel's factory-default credential and unprovisioned, the optical drive swapped for the boot SSD and its SMART read **before** the install, TrueNAS 25.10 installed, the static and the Kea - reservation both set, and the inbound rules created and verified. What is - left is the two Exos drives and everything downstream of them: the mirror - `erebor`, its two datasets, the household share, the stack, and the one test - that decides whether the stack stays here at all — whether Quick Sync reaches - a container, which is ADR-0040's reopen condition and is still unrun. + reservation both set, and the inbound rules created and verified. **The + drives landed 2026-09-18**, both at zero hours by the FARM log and not only + by SMART, and the rest followed: the mirror `erebor`, its two datasets, the + household share `media`, and the stack running under TrueNAS's Docker with + the scrape live on 2026-09-19. Of ADR-0040's reopen condition, the render + node reaches the container and the process carries the render group; the + transcode itself is the half still unrun. Reading the enforced ruleset first changed two of the answers, and both were borne out when the rules were created. **50→40 is not simply a rule to add**: Hicks and Winterfell each carry an explicit *Block access to diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index c738fd05..75957531 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -10,6 +10,23 @@ SSH is off — `neo`'s web UI at `http://10.7.7.2` and a yellow Cat6 lead for §0.2b, the pfSense UI on `morpheus`, a shell on the monitoring host for §0.6, and the two Exos X20 drives for §1 onward. +> **Status — 2026-09-19: the drives are in, the pool exists, the stack is +> deployed and the scrape is on.** The Exos pair landed on 2026-09-18, a day +> after the carrier's window lapsed, and §1–§6 were done that evening and the +> next morning. Both drives read **0 power-on hours in the FARM log** as well +> as in SMART, which is the reading that settles the listing's claim (§2). +> `erebor` is a mirror with encryption off by decision (§3); `erebor/media` +> and `erebor/apps` exist (§4); the share is `media` and its user is `bilbo` +> (§5); the stack runs from `/mnt/erebor/apps/stack` under TrueNAS's Docker, +> and **§6 as previously written could not run on this box** — it is rewritten +> below. `node_exporter` answered the three §6.1 checks from the monitoring +> host and the target in `targets/node.yaml` is live. Inside the container the +> render node is present and the process carries GID 107 — the two halves of +> ADR-0040's condition that a shell can check. **Still open in §7:** the +> transcode itself, a television playing something, the two extended +> self-tests (about 28 hours each, started 2026-09-18), and the tripwire and +> port-15 re-reads. +> > **Status — 2026-09-16: §0 is the work that can be done before the drives > land, and it is the whole of what is blocking.** > @@ -45,7 +62,7 @@ and the two Exos X20 drives for §1 onward. > from `-vv`. So match on the rule descriptions, which do not depend on how > the ruleset is being printed. > -> What is left is the drives. The pool does not exist and nothing is deployed. +> What was left on 2026-09-16 was the drives. They landed two days later. This builds what [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) placed and [ADR-0040](../adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) @@ -309,6 +326,20 @@ is Seagate's jumper header, which stays empty. ## §2 — Read the drives before trusting them +> **Done 2026-09-18.** `sda` was `ZVTBS4NL` and `sdb` was `ZVTBSDL3`, both +> `PASSED`, 0 reallocated, 0 pending, 0 power-on hours by SMART — and, the +> reading this section did not know to ask for, **0 power-on hours and 0 +> spindle hours by `smartctl -l farm`** on both. Seagate's FARM log keeps an +> hours counter that a SMART reset does not touch, and it is what caught the +> used Exos drives sold as new through 2025; the `smartctl` on TrueNAS 25.10 +> reads it. Run it, and treat SMART's zero as a claim until it agrees. +> `ZVTBSDL3` arrived carrying a Windows quick format — a 16 MB reserved +> partition and an NTFS volume labelled `New Volume` — with about 370 MB +> written and one short self-test at hour 0, which is a seller's bench check +> and nothing more; the pool creation wiped it. Extended tests started on +> both the same evening, at about 28 hours each. Full readings are in +> [`hardware.md`](../hardware.md). + From **option 8, Open Linux Shell**, at the console — **not over SSH**. TrueNAS ships SSH disabled, and §0.5's port-22 pass is inert until someone turns it on. Enabling it here to save a walk to the machine widens this host's @@ -347,6 +378,9 @@ smartctl -t long /dev/sdc ## §3 — Create the mirror +> **Done 2026-09-18.** Mirror of the two Exos, encryption unchecked, with the +> extended self-tests still running underneath it. + **Storage → Create Pool.** | Setting | Value | @@ -354,6 +388,18 @@ smartctl -t long /dev/sdc | Name | `erebor` | | Layout | **Mirror** | | Disks | the two Exos X20 | +| Encryption | **off** | + +**Encryption is off by decision, not by default.** ADR-0008 ruled the library +replaceable and `erebor/apps` is watch history; encryption at rest is what the +sensitive tier gets, and nothing asks it of this one. Both key modes cost +something here: a key file auto-unlocks from a boot pool that lives on a used +SSD with reallocated sectors, and a passphrase leaves the pool locked after +every reboot on a box whose remote console was disabled on purpose (§0.2). +Either is a new secret with a handover obligation, guarding films. ZFS +encryption is per dataset, so a future dataset that holds something that +matters can be created encrypted on this plain pool; pool-level encryption +cannot be removed later without a rebuild. Saying no keeps the option. **A mirror of two is one drive's capacity — 18 TB usable, not 36.** ADR-0016 chose availability, not capacity: a dead disk becomes a drive swap instead of a @@ -365,6 +411,22 @@ the optical bay exists for. ## §4 — Datasets +> **Done 2026-09-18.** `erebor/media` on the **SMB** preset — case-insensitive +> with NFSv4 ACLs, which is what televisions and Windows clients expect and +> cannot be changed after creation — and `erebor/apps` on the **Apps** preset. +> The Add Dataset dialog calls these *Dataset Presets*; the record size and +> atime are under its advanced options. +> +> **One row of the table below is not yet true, and it is the one that says +> "backed up".** Jellyfin's `/config` is a Docker named volume, and TrueNAS +> keeps named volumes on the pool it was given for Apps, in a dataset of its +> own — `erebor/ix-apps/docker`, not `erebor/apps`. So `erebor/apps` holds the +> compose file and its `.env` (§6) and nothing Jellyfin writes. Whether the +> stack binds `/config` to `erebor/apps` instead, or the backup reads the +> `ix-apps` dataset, is [#484](https://github.com/Gerrrt/HomeLab/issues/484)'s +> to decide with the rest of the backup mechanism; it is recorded here so the +> table is not read as describing what exists. + **Storage → `erebor` → Add Dataset.** Two of them, and the split is the backup decision made deliberately rather than drifted into. @@ -385,6 +447,19 @@ compression stays on and costs nothing on already-compressed media. ## §5 — The household share +> **Done 2026-09-18.** The share is **`media`**, reached as +> `\\10.0.40.30\media`, created from the dataset dialog's *Create SMB Share* +> box rather than from the Shares page — same result. The user is **`bilbo`**: +> SMB on, and TrueNAS access, shell, SSH and sudo all off, so the credential +> that lives on televisions can mount one share and do nothing else anywhere. +> It needed no ACL entry of its own: an SMB user joins `builtin_users` on +> creation, and the SMB preset's default ACL already grants that group +> Modify, so `bilbo` reads and writes. What the ACL was missing was +> **Jellyfin's** read path — the container reads the library as uid 65534, +> which is nobody's group and not `builtin_users` — so one entry was added: +> `everyone@`, Allow, Basic Read, Inherit. The list now has five entries, the +> four the preset wrote and that one. + **Shares → Windows (SMB) → Add**, pointed at `erebor/media`. Create a dedicated TrueNAS user for it rather than sharing the admin account. @@ -399,16 +474,53 @@ a compose file this repository owns, run under TrueNAS's app runtime, **not** a catalogue app. That is what keeps Dependabot, the digest pins and `make validate` reaching it. -Copy `stacks/media/.env.example` to `.env`, confirm `RENDER_GID` still matches -what this host reports, and bring it up: +> **Done 2026-09-19, and not the way this section said.** It read +> `make up STACK=media` until then. That target renders config first, the +> render decrypts `secrets/.sops.yaml`, and this stack has no secrets +> file because it needs no secrets — so the render dies on the missing file, +> on a box that ships neither `make` nor `sops` in any case. What `make up` +> does underneath is `docker compose up`, and that is what runs here. The +> steps below are what was done. + +**First, give Apps a pool.** Docker does not exist on TrueNAS until it has +one: **Apps → Configuration → Choose Pool → `erebor`**, and wait for Apps to +report running. That creates `erebor/ix-apps`, where Docker's images and named +volumes live from then on. + +**Then fetch the two files the stack is.** The repository is public and this +host has egress, so they come straight from `main`. The `.env` is copied +as-is, because every value in it is a plain host fact: ```bash -make up STACK=media +mkdir -p /mnt/erebor/apps/stack && cd /mnt/erebor/apps/stack \ + && curl -fsSLO https://raw.githubusercontent.com/Gerrrt/HomeLab/main/stacks/media/compose.yaml \ + && curl -fsSL https://raw.githubusercontent.com/Gerrrt/HomeLab/main/stacks/media/.env.example -o .env ``` -Jellyfin binds `8096`, reads `erebor/media`, and writes its state to -`erebor/apps`. `node-exporter` binds `9100` and is the whole of how this host -is monitored — see §6.1. +The folder is under `erebor/apps` because that is the dataset §4 set aside +for application state; its name is arbitrary, since the compose file sets its +own project name. **Do not call it `media`** — that is the library's name one +level up, and the collision confused the first person to do this. + +Confirm `RENDER_GID` still matches what this host reports — it is hard-coded +in `.env`, and `107 render` was re-read on 2026-09-19 — then bring it up: + +```bash +stat -c '%g %G' /dev/dri/renderD128 +docker compose up -d && docker compose ps +``` + +Jellyfin binds `8096` and reads `erebor/media`. `node-exporter` binds `9100` +and is the whole of how this host is monitored — see §6.1. Jellyfin's state +goes to a named volume, which is on `erebor/ix-apps` and not on `erebor/apps`; +§4 says why that matters. + +**Updating the stack is the same two `curl` lines and `docker compose up -d` +again.** Nothing on this host pulls from `main` on its own: there is no +converge timer here, so a Dependabot bump that merges is not deployed until +someone does this. That is a residual of ADR-0040's shape and not a defect in +it, and it wants a line in `stacks/media/README.md` rather than an issue until +it bites. ### §6.1 — Turn the scrape on, and prove it before you do @@ -435,6 +547,16 @@ Then uncomment the four lines at the end of `prometheus/targets/node.yaml` and commit. That directory is a bind mount, so Prometheus re-reads it within five minutes: no restart, no deploy, no `--force-recreate`. +> **Done 2026-09-19.** From the monitoring host: `9100` open, `443` and +> `8096` refused, `node_uname_info` and `node_boot_time_seconds` present, +> **36** `node_filesystem_avail_bytes` series including `erebor`, +> `erebor/media` and `erebor/apps`, and **0** `node_network_*` series. The +> target was uncommented the same morning. Inside the container: +> `renderD128` listed as `root 107` and `id` read +> `uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup),107` — both +> halves of the check below that a shell can make. The transcode is the one +> it cannot, and is still to run. + Expect **no** `node_network_*` series from this host. Those collectors are disabled on purpose, because a bridged container reads its own veth and would chart it as this NAS's throughput; `stacks/media/compose.yaml` carries the @@ -469,6 +591,12 @@ measurement. ## §7 — Verify +> **As of 2026-09-19:** the monitoring-host line holds in both halves, the +> device checks hold, and `erebor` is mounted with its datasets. **Not yet +> read:** a television playing, the QSV transcode, `zpool status`, the two +> extended self-tests, and the post-deploy re-reads of the tripwire and port +> 15. + - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved - A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096` diff --git a/stacks/media/README.md b/stacks/media/README.md index 51e83449..47e76c0b 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -2,13 +2,17 @@ [ADR-0008]'s media tier — the household's media server — on `smaug` (`10.0.40.30`, CasaBonita / VLAN 40), the ThinkServer TS150 that [#413] bought -and [ADR-0040] gave TrueNAS. **The pool is not built yet**; -[`build-the-nas.md`] §6 deploys this stack, and this directory is authored -ahead of the storage the way `stacks/sensitive` was authored ahead of -`trinity`. +and [ADR-0040] gave TrueNAS. **Deployed 2026-09-19**, from a copy of this +directory's two files on the pool, under TrueNAS's own Docker; +[`build-the-nas.md`] §6 is the procedure. `make up STACK=media` does **not** +work on that host — it renders secrets this stack does not have, on a box +without `make` or `sops` — so a change here reaches `smaug` by re-fetching +`compose.yaml` and `.env.example` from `main` into `/mnt/erebor/apps/stack` +and running `docker compose up -d` there. Nothing pulls from `main` on that +host on its own: a Dependabot bump is merged here and deployed there by hand. ```bash -make up STACK=media +cd /mnt/erebor/apps/stack && docker compose up -d ``` | Service | Image | Port | Purpose | @@ -130,7 +134,7 @@ Every non-obvious line in `compose.yaml` came off the pinned image on is a restart loop. - **65 MiB idle RSS**, which is what the 2 GiB ceiling is a ceiling over. -## The check that is not done yet +## The check that is half done [ADR-0040] keeps the media stack in this repository on the strength of Quick Sync working, and names its own reopen condition: **the iGPU reaching a @@ -140,6 +144,12 @@ different claims. `devices: /dev/dri` and `RENDER_GID` are where the second claim is made; [`build-the-nas.md`] §6 is where it gets tested, **before the library exists**, because moving a populated library is a weekend. +**On 2026-09-19 the shell half passed**: `renderD128` is listed inside the +container as `root 107`, and `id` there reads +`groups=65534(nogroup),107`. What that does not prove is a transcode, which +is a claim only Jellyfin's playback settings and a played file can make, and +that is the half still open. + [ADR-0008]: ../../docs/adr/0008-place-services-by-data-trust.md [ADR-0012]: ../../docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md [ADR-0016]: ../../docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md diff --git a/stacks/observability/prometheus/targets/node.yaml b/stacks/observability/prometheus/targets/node.yaml index b39b24c6..67f5e84e 100644 --- a/stacks/observability/prometheus/targets/node.yaml +++ b/stacks/observability/prometheus/targets/node.yaml @@ -62,24 +62,20 @@ # --- smaug --------------------------------------------------------------- # -# WRITTEN OUT AND NOT ENABLED. The host holds 10.0.40.30 and the firewall pass -# is verified, but nothing answers on 9100 yet: the exporter is a container, the -# container needs Docker, and Docker on TrueNAS needs the apps pool — which is -# on the ZFS mirror that does not exist. Every route to node_exporter on this -# host runs through that pool, so this cannot be live before the drives land no -# matter where the exporter is put. ADR-0040's "the scrape, then the stack" is -# right about the order these are AUTHORED in and is not achievable for -# liveness. +# ENABLED 2026-09-19, after the three checks below passed from this host. It +# was written out and commented from 2026-09-17 until then, because every +# route to node_exporter on this host runs through the ZFS pool — the exporter +# is a container, the container needs Docker, and Docker on TrueNAS needs the +# apps pool — and enabling it before the pool existed would have meant +# `up == 0` from the moment the file loaded, with InstanceDown at severity +# critical paging `urgent` every 4 hours, forever. ADR-0017 refuses a check +# that is permanently red for a known reason by name, and +# targets/blackbox-dns.yaml and the commented Windows job in stacks/lab make +# the same call. # -# Enabling it today would mean `up == 0` from the moment the file loaded, and -# InstanceDown is `up == 0` at severity critical — routed to `urgent`, repeating -# every 4 hours, forever. ADR-0017 refuses a check that is permanently red for a -# known reason by name, and targets/blackbox-dns.yaml and the commented Windows -# job in stacks/lab make the same call. -# -# When the pool exists and stacks/media is deployed, verify FROM THIS HOST — -# which can reach 9100 and may not reach anything else on 40 — BEFORE -# uncommenting: +# What was verified FROM THIS HOST — which can reach 9100 and may not reach +# anything else on 40 — before the four lines below were uncommented, and what +# to re-run if the exporter is ever rebuilt: # # nc -z -w3 10.0.40.30 9100 # curl -s http://10.0.40.30:9100/metrics | grep -E '^node_(uname_info|boot_time_seconds)' @@ -89,15 +85,18 @@ # produces a container's filesystems, or none, while `up` still reads 1 and the # target shows green — so a count of 0 means every disk rule on this host is # blind and nothing will say so. It must be the number of mounted filesystems on -# smaug, and after §3 that includes erebor. +# smaug, and after build-the-nas.md §3 that includes erebor. On 2026-09-19 it +# read 36: the boot pool's datasets, `erebor`, `erebor/media`, `erebor/apps`, +# and the `.system` and `ix-apps` datasets TrueNAS creates for itself. 443 and +# 8096 were refused from here in the same minute, as §0.5's rules say. # # Expect NO node_network_* series at all: stacks/media/compose.yaml disables # those collectors on purpose, because a bridged container reads its own veth # and would chart it as this NAS's throughput. That file carries the -# measurement and the argument. +# measurement and the argument. The count was 0 on the day. -# - targets: ["10.0.40.30:9100"] -# labels: -# instance: smaug -# role: nas -# vlan: "40" +- targets: ["10.0.40.30:9100"] + labels: + instance: smaug + role: nas + vlan: "40" From e016b71d6c3e7d79c70a8ac813799a03e3eaa879 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 19 Sep 2026 03:57:15 +0000 Subject: [PATCH 3/4] docs: the TS150's power lead has no 3.3 V wire, so pin 3 is moot on this box (#413) Refs #413 Co-Authored-By: Claude Fable 5.1 --- docs/hardware.md | 8 +++++--- docs/runbooks/build-the-nas.md | 5 +++++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/hardware.md b/docs/hardware.md index d2a6e6ab..3d14cd42 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -280,9 +280,11 @@ revisions of this repository treated `shiva` as the hypervisor itself. `New Volume` filling the rest — with 718,258 LBAs written, about 370 MB and the size of that format, three power cycles, and one short self-test logged at lifetime hour 0; `ZVTBS4NL` arrived blank, two power cycles, - nothing written. Both spun up and enumerated on the first power-up - (`build-the-nas.md` §1 records the pin-3 trap they could have hit). They - enter the Compute table with the pool. + nothing written. Both spun up and enumerated on the first power-up. **The + TS150's SATA power lead has four wires and no orange one**, read on + 2026-09-18, so this supply puts nothing on pin 3 and the Power Disable trap + `build-the-nas.md` §1 records does not apply on this box — it would on a + supply that does. They entered the Compute table with the pool. - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss protection — bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot disk, carrying TrueNAS and the media stack it launches diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 75957531..645ca1b0 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -298,6 +298,11 @@ anything in this runbook. ## §1 — Fit the drives +> **Done 2026-09-18.** Both trays, `SATA2` and `SATA3`, the bay fan left +> alone. **The power lead had four wires and no orange one**, so the pin-3 +> trap below did not fire on this supply; both drives spun up and appeared in +> `lsblk` first time. + Power down, unplug, hold the power button five seconds, ground yourself. Both 3.5" trays are already in the bays and empty. Screw a drive into each, From aa2365b85a2eb6f81c86168f388088f0ee5cff3d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sat, 19 Sep 2026 04:05:44 +0000 Subject: [PATCH 4/4] docs(runbook): erebor is ONLINE with no errors (#413) Refs #413 Co-Authored-By: Claude Fable 5.1 --- docs/runbooks/build-the-nas.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 645ca1b0..94917fcb 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -384,7 +384,9 @@ smartctl -t long /dev/sdc ## §3 — Create the mirror > **Done 2026-09-18.** Mirror of the two Exos, encryption unchecked, with the -> extended self-tests still running underneath it. +> extended self-tests still running underneath it. `zpool status erebor` on +> 2026-09-19: `ONLINE`, one `mirror-0` of two members, 0 read, 0 write, 0 +> checksum errors, no known data errors. **Storage → Create Pool.** @@ -597,10 +599,9 @@ measurement. ## §7 — Verify > **As of 2026-09-19:** the monitoring-host line holds in both halves, the -> device checks hold, and `erebor` is mounted with its datasets. **Not yet -> read:** a television playing, the QSV transcode, `zpool status`, the two -> extended self-tests, and the post-deploy re-reads of the tripwire and port -> 15. +> device checks hold, and `zpool status erebor` is `ONLINE` with no errors. +> **Not yet read:** a television playing, the QSV transcode, the two extended +> self-tests, and the post-deploy re-reads of the tripwire and port 15. - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved