diff --git a/docs/architecture.md b/docs/architecture.md index c36f83df..0520f94e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -191,7 +191,7 @@ hole from the monitoring VLAN into the monitored one. | `phoenix` (10.0.30.70) | ๐ŸŸข 30 | *(none โ€” a toolchain host, no Docker)* | **Not built yet** โ€” the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | ๐Ÿ”ด 99 | *(none โ€” hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs โ€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | ๐Ÿ”ด 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** โ€” ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich โ€” four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) โ€” Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | ๐ŸŸก 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position โ€” but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy โ€” the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 โ†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live; the target itself stays commented until there is a pool to run the exporter from | +| `smaug` (10.0.40.30) | ๐ŸŸก 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 โ€” from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) ยง6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy โ€” the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 โ†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool | | `bahamut` (10.0.30.50) | ๐ŸŸข 30 | *(none โ€” Windows)* | **Not built yet** โ€” Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` โ€” Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | ๐ŸŸข 30 | *(none โ€” Windows)* | **Not built yet** โ€” Windows Server 2025 second domain controller and DNS โ€” Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | ๐ŸŸข 30 | *(none โ€” Windows)* | **Not built yet** โ€” Windows Server 2025 file and member server โ€” the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound โ€” Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/hardware.md b/docs/hardware.md index e528b60e..3d14cd42 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -43,7 +43,7 @@ quietly swapped. | `Saruman` | HPE ProLiant DL360 Gen9 | 2ร— Xeon E5-2680 v3 (48 threads) | 128 GB | 2ร— 1 TB SAS HDD, RAID 1; 2ร— 960 GB SATA SSD, unassigned | Proxmox VE 9 | | `prometheus` | Apple MacBook Pro (2012, Retina 13") | i5/i7 | 8 GB | 256 GB SSD | Ubuntu Server 24.04 LTS | | `oracle` | Dell Inspiron 15-3565 | AMD A6-9200 (2 cores) | 4 GB | 500 GB HDD | Ubuntu Server 24.04 LTS | -| `smaug` | Lenovo ThinkServer TS150 | Xeon E3-1225 v6 (4 cores) | 8 GB ECC | 240 GB SATA SSD (boot) | TrueNAS 25.10 | +| `smaug` | Lenovo ThinkServer TS150 | Xeon E3-1225 v6 (4 cores) | 8 GB ECC | 240 GB SATA SSD (boot) + 2ร— 18 TB ZFS mirror `erebor` | TrueNAS 25.10 | The observability stack runs on a thirteen-year-old MacBook. It handles four SNMP devices at a 60-second interval, four Alloy agents, and 30 days of metric @@ -194,10 +194,11 @@ revisions of this repository treated `shiva` as the hypervisor itself. has created would be the kind of claim this table exists to not make. The boot disk the TrueNAS install wants ([ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md)) and the bracket that carries it in the optical bay are the entries below and - landed with it; the two drives for the mirror, bought 2026-09-11, have not. Nothing + landed with it; the two drives for the mirror, bought 2026-09-11, landed on + 2026-09-18 and were in the bays that evening. Nothing for this machine is outstanding on the roadmap's - [list](roadmap.md#everything-still-to-buy) any more; what is left is those - two drives landing, and a build. + [list](roadmap.md#everything-still-to-buy) any more, and the Storage column + reads the mirror since 2026-09-19. **Read off the machine on 2026-09-15**, where everything above it came off a listing: `ThinkServer TS150`, machine type-model `70UB000AUX`, serial `MJ05N4NK`. Xeon E3-1225 v6 at 3.30 GHz, four cores, and `Active Video: IGD` @@ -248,7 +249,8 @@ revisions of this repository treated `shiva` as the hypervisor itself. access ZFS wants and the thing [#418](https://github.com/Gerrrt/HomeLab/issues/418) is the cautionary tale for, and `CSM [Disabled]`, so it boots UEFI as TrueNAS wants. - Two 3.5" trays, both empty โ€” exactly the mirror and no spare. + Two 3.5" trays, filled by the Exos pair on 2026-09-18 โ€” exactly the mirror + and no spare. **The 5.25" bay was not empty**: a PLDS `DVD-RW DU8AESH` answered on SATA5. A photograph of the open case had been read here as an empty cage and was wrong; the BIOS summary is what caught it. The optical drive came out on @@ -257,15 +259,32 @@ revisions of this repository treated `shiva` as the hypervisor itself. that fan is **not optional**: it is the airflow over the drive bays, and two 7200 rpm Exos under a scrub will want it. Reconnected after the swap and reading `Aux Fan: Operating`. -- 2ร— Seagate Exos X20 18 TB (`ST18000NM003D`), 3.5" SATA โ€” bought - 2026-09-11, in transit. `smaug`'s ZFS mirror +- 2ร— Seagate Exos X20 18 TB (`ST18000NM003D`, firmware `SN03`), 3.5" SATA โ€” + bought 2026-09-11, **in hand since 2026-09-18**, a day after the carrier's + window lapsed. `smaug`'s ZFS mirror ([ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md), [#413](https://github.com/Gerrrt/HomeLab/issues/413)). A mirror of two is - one drive's capacity, so this is 18 TB usable, not 36. The listing's - **zero power-on hours is a claim, not a fact** โ€” read it back with - `smartctl -a` on arrival and record what the drives actually report here, - before the mirror is built on them. Serials go here when they land. They - enter the Compute table with the NAS, which is not built. + one drive's capacity, so this is 18 TB usable, not 36. + **Read off both drives with `smartctl -a` at the console on 2026-09-18, + before the pool existed**: serials `ZVTBS4NL` and `ZVTBSDL3`, both `PASSED`, + both at **0 power-on hours**, 0 reallocated, 0 pending, 0 uncorrectable, no + errors logged, 26 ยฐC in the bays against a lifetime range of 24โ€“26. **The + zero hours is a fact, not a claim, and it was checked the one way that + settles it.** SMART's hours counter can be reset, and used Exos drives with + it zeroed were sold as new through 2025; Seagate's FARM log keeps a second + counter that the reset does not touch, and `smartctl -l farm` on this host + read **0 power-on hours and 0 spindle hours on both drives**. What the + resettable counters add is consistent with new drives bench-checked by the + seller and with nothing more: `ZVTBSDL3` arrived carrying a Windows quick + format โ€” a 16 MB Microsoft reserved partition and an NTFS volume labelled + `New Volume` filling the rest โ€” with 718,258 LBAs written, about 370 MB + and the size of that format, three power cycles, and one short self-test + logged at lifetime hour 0; `ZVTBS4NL` arrived blank, two power cycles, + nothing written. Both spun up and enumerated on the first power-up. **The + TS150's SATA power lead has four wires and no orange one**, read on + 2026-09-18, so this supply puts nothing on pin 3 and the Power Disable trap + `build-the-nas.md` ยง1 records does not apply on this box โ€” it would on a + supply that does. They entered the Compute table with the pool. - Intel DC S3520 240 GB, 2.5" SATA 6 Gb/s enterprise SSD with power-loss protection โ€” bought 2026-09-11, **in hand since 2026-09-15**. `smaug`'s boot disk, carrying TrueNAS and the media stack it launches diff --git a/docs/network.md b/docs/network.md index 551b3fef..416122df 100644 --- a/docs/network.md +++ b/docs/network.md @@ -299,9 +299,9 @@ Televisions and consoles. Internet only. [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) - ([#413](https://github.com/Gerrrt/HomeLab/issues/413)). Its ZFS mirror does - not exist yet, and `stacks/media` is authored and undeployed against that - absence; what exists is a host on its address. **It does not change the + ([#413](https://github.com/Gerrrt/HomeLab/issues/413)). Its ZFS mirror + `erebor` exists since 2026-09-18 and `stacks/media` runs on it since + 2026-09-19, publishing `8096` to this segment. **It does not change the *Reaches* column**, and that is the point ADR-0016 made in advance: nothing on this segment initiates anywhere, and the rules created that day all let a more trusted segment reach **in**. That is the direction this row records, diff --git a/docs/observability.md b/docs/observability.md index e6931280..b91e02c2 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -453,7 +453,7 @@ argument and for what to do when it exits 1. | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack โ€” `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values โ€” see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | -| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see โ€” a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` is written into `targets/node.yaml` and left disabled until the NAS has a pool to run the exporter from. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | +| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see โ€” a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left โ€” Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense โ€” a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) | diff --git a/docs/roadmap.md b/docs/roadmap.md index 34587a5b..1601790a 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -799,11 +799,13 @@ what left this one unfireable for months. the BIOS flashed, AMT found on Intel's factory-default credential and unprovisioned, the optical drive swapped for the boot SSD and its SMART read **before** the install, TrueNAS 25.10 installed, the static and the Kea - reservation both set, and the inbound rules created and verified. What is - left is the two Exos drives and everything downstream of them: the mirror - `erebor`, its two datasets, the household share, the stack, and the one test - that decides whether the stack stays here at all โ€” whether Quick Sync reaches - a container, which is ADR-0040's reopen condition and is still unrun. + reservation both set, and the inbound rules created and verified. **The + drives landed 2026-09-18**, both at zero hours by the FARM log and not only + by SMART, and the rest followed: the mirror `erebor`, its two datasets, the + household share `media`, and the stack running under TrueNAS's Docker with + the scrape live on 2026-09-19. Of ADR-0040's reopen condition, the render + node reaches the container and the process carries the render group; the + transcode itself is the half still unrun. Reading the enforced ruleset first changed two of the answers, and both were borne out when the rules were created. **50โ†’40 is not simply a rule to add**: Hicks and Winterfell each carry an explicit *Block access to diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index b19ee424..94917fcb 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -10,6 +10,23 @@ SSH is off โ€” `neo`'s web UI at `http://10.7.7.2` and a yellow Cat6 lead for ยง0.2b, the pfSense UI on `morpheus`, a shell on the monitoring host for ยง0.6, and the two Exos X20 drives for ยง1 onward. +> **Status โ€” 2026-09-19: the drives are in, the pool exists, the stack is +> deployed and the scrape is on.** The Exos pair landed on 2026-09-18, a day +> after the carrier's window lapsed, and ยง1โ€“ยง6 were done that evening and the +> next morning. Both drives read **0 power-on hours in the FARM log** as well +> as in SMART, which is the reading that settles the listing's claim (ยง2). +> `erebor` is a mirror with encryption off by decision (ยง3); `erebor/media` +> and `erebor/apps` exist (ยง4); the share is `media` and its user is `bilbo` +> (ยง5); the stack runs from `/mnt/erebor/apps/stack` under TrueNAS's Docker, +> and **ยง6 as previously written could not run on this box** โ€” it is rewritten +> below. `node_exporter` answered the three ยง6.1 checks from the monitoring +> host and the target in `targets/node.yaml` is live. Inside the container the +> render node is present and the process carries GID 107 โ€” the two halves of +> ADR-0040's condition that a shell can check. **Still open in ยง7:** the +> transcode itself, a television playing something, the two extended +> self-tests (about 28 hours each, started 2026-09-18), and the tripwire and +> port-15 re-reads. +> > **Status โ€” 2026-09-16: ยง0 is the work that can be done before the drives > land, and it is the whole of what is blocking.** > @@ -45,7 +62,7 @@ and the two Exos X20 drives for ยง1 onward. > from `-vv`. So match on the rule descriptions, which do not depend on how > the ruleset is being printed. > -> What is left is the drives. The pool does not exist and nothing is deployed. +> What was left on 2026-09-16 was the drives. They landed two days later. This builds what [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) placed and [ADR-0040](../adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) @@ -281,6 +298,11 @@ anything in this runbook. ## ยง1 โ€” Fit the drives +> **Done 2026-09-18.** Both trays, `SATA2` and `SATA3`, the bay fan left +> alone. **The power lead had four wires and no orange one**, so the pin-3 +> trap below did not fire on this supply; both drives spun up and appeared in +> `lsblk` first time. + Power down, unplug, hold the power button five seconds, ground yourself. Both 3.5" trays are already in the bays and empty. Screw a drive into each, @@ -291,8 +313,38 @@ stay free. **Do not disturb the bay fan on `AUX1_FAN`.** It is the airflow over these two drives, and two 7200 rpm Exos under a scrub will want it. +**Look at the power lead before the case closes.** The Exos, like every +enterprise SATA drive since about 2016, reads **pin 3 of its power connector as +Power Disable**: a supply that puts 3.3 V on that pin holds the drive off, so it +never spins and never appears in `lsblk`. A desktop supply's SATA lead with +**five wires, one of them orange**, carries that 3.3 V; four wires and no orange +does not. If the orange wire is there, put Kapton tape over pin 3 on each +drive's power receptacle, or feed the bays through a Molex-to-SATA adapter, +which has no 3.3 V to offer. The boot disk booting proves nothing here โ€” the +S3520 predates the feature and ignores the pin. A drive missing from ยง2's +`lsblk` is this before it is a cable or a port. + +The connectors themselves are ordinary SATA, and it is worth saying because +the drive held PCB-up does not look like it: the wide receptacle is the 15-pin +power, the narrow one the 7-pin data, and the small four-pin block beside them +is Seagate's jumper header, which stays empty. + ## ยง2 โ€” Read the drives before trusting them +> **Done 2026-09-18.** `sda` was `ZVTBS4NL` and `sdb` was `ZVTBSDL3`, both +> `PASSED`, 0 reallocated, 0 pending, 0 power-on hours by SMART โ€” and, the +> reading this section did not know to ask for, **0 power-on hours and 0 +> spindle hours by `smartctl -l farm`** on both. Seagate's FARM log keeps an +> hours counter that a SMART reset does not touch, and it is what caught the +> used Exos drives sold as new through 2025; the `smartctl` on TrueNAS 25.10 +> reads it. Run it, and treat SMART's zero as a claim until it agrees. +> `ZVTBSDL3` arrived carrying a Windows quick format โ€” a 16 MB reserved +> partition and an NTFS volume labelled `New Volume` โ€” with about 370 MB +> written and one short self-test at hour 0, which is a seller's bench check +> and nothing more; the pool creation wiped it. Extended tests started on +> both the same evening, at about 28 hours each. Full readings are in +> [`hardware.md`](../hardware.md). + From **option 8, Open Linux Shell**, at the console โ€” **not over SSH**. TrueNAS ships SSH disabled, and ยง0.5's port-22 pass is inert until someone turns it on. Enabling it here to save a walk to the machine widens this host's @@ -331,6 +383,11 @@ smartctl -t long /dev/sdc ## ยง3 โ€” Create the mirror +> **Done 2026-09-18.** Mirror of the two Exos, encryption unchecked, with the +> extended self-tests still running underneath it. `zpool status erebor` on +> 2026-09-19: `ONLINE`, one `mirror-0` of two members, 0 read, 0 write, 0 +> checksum errors, no known data errors. + **Storage โ†’ Create Pool.** | Setting | Value | @@ -338,6 +395,18 @@ smartctl -t long /dev/sdc | Name | `erebor` | | Layout | **Mirror** | | Disks | the two Exos X20 | +| Encryption | **off** | + +**Encryption is off by decision, not by default.** ADR-0008 ruled the library +replaceable and `erebor/apps` is watch history; encryption at rest is what the +sensitive tier gets, and nothing asks it of this one. Both key modes cost +something here: a key file auto-unlocks from a boot pool that lives on a used +SSD with reallocated sectors, and a passphrase leaves the pool locked after +every reboot on a box whose remote console was disabled on purpose (ยง0.2). +Either is a new secret with a handover obligation, guarding films. ZFS +encryption is per dataset, so a future dataset that holds something that +matters can be created encrypted on this plain pool; pool-level encryption +cannot be removed later without a rebuild. Saying no keeps the option. **A mirror of two is one drive's capacity โ€” 18 TB usable, not 36.** ADR-0016 chose availability, not capacity: a dead disk becomes a drive swap instead of a @@ -349,6 +418,22 @@ the optical bay exists for. ## ยง4 โ€” Datasets +> **Done 2026-09-18.** `erebor/media` on the **SMB** preset โ€” case-insensitive +> with NFSv4 ACLs, which is what televisions and Windows clients expect and +> cannot be changed after creation โ€” and `erebor/apps` on the **Apps** preset. +> The Add Dataset dialog calls these *Dataset Presets*; the record size and +> atime are under its advanced options. +> +> **One row of the table below is not yet true, and it is the one that says +> "backed up".** Jellyfin's `/config` is a Docker named volume, and TrueNAS +> keeps named volumes on the pool it was given for Apps, in a dataset of its +> own โ€” `erebor/ix-apps/docker`, not `erebor/apps`. So `erebor/apps` holds the +> compose file and its `.env` (ยง6) and nothing Jellyfin writes. Whether the +> stack binds `/config` to `erebor/apps` instead, or the backup reads the +> `ix-apps` dataset, is [#484](https://github.com/Gerrrt/HomeLab/issues/484)'s +> to decide with the rest of the backup mechanism; it is recorded here so the +> table is not read as describing what exists. + **Storage โ†’ `erebor` โ†’ Add Dataset.** Two of them, and the split is the backup decision made deliberately rather than drifted into. @@ -369,6 +454,19 @@ compression stays on and costs nothing on already-compressed media. ## ยง5 โ€” The household share +> **Done 2026-09-18.** The share is **`media`**, reached as +> `\\10.0.40.30\media`, created from the dataset dialog's *Create SMB Share* +> box rather than from the Shares page โ€” same result. The user is **`bilbo`**: +> SMB on, and TrueNAS access, shell, SSH and sudo all off, so the credential +> that lives on televisions can mount one share and do nothing else anywhere. +> It needed no ACL entry of its own: an SMB user joins `builtin_users` on +> creation, and the SMB preset's default ACL already grants that group +> Modify, so `bilbo` reads and writes. What the ACL was missing was +> **Jellyfin's** read path โ€” the container reads the library as uid 65534, +> which is nobody's group and not `builtin_users` โ€” so one entry was added: +> `everyone@`, Allow, Basic Read, Inherit. The list now has five entries, the +> four the preset wrote and that one. + **Shares โ†’ Windows (SMB) โ†’ Add**, pointed at `erebor/media`. Create a dedicated TrueNAS user for it rather than sharing the admin account. @@ -383,16 +481,53 @@ a compose file this repository owns, run under TrueNAS's app runtime, **not** a catalogue app. That is what keeps Dependabot, the digest pins and `make validate` reaching it. -Copy `stacks/media/.env.example` to `.env`, confirm `RENDER_GID` still matches -what this host reports, and bring it up: +> **Done 2026-09-19, and not the way this section said.** It read +> `make up STACK=media` until then. That target renders config first, the +> render decrypts `secrets/.sops.yaml`, and this stack has no secrets +> file because it needs no secrets โ€” so the render dies on the missing file, +> on a box that ships neither `make` nor `sops` in any case. What `make up` +> does underneath is `docker compose up`, and that is what runs here. The +> steps below are what was done. + +**First, give Apps a pool.** Docker does not exist on TrueNAS until it has +one: **Apps โ†’ Configuration โ†’ Choose Pool โ†’ `erebor`**, and wait for Apps to +report running. That creates `erebor/ix-apps`, where Docker's images and named +volumes live from then on. + +**Then fetch the two files the stack is.** The repository is public and this +host has egress, so they come straight from `main`. The `.env` is copied +as-is, because every value in it is a plain host fact: ```bash -make up STACK=media +mkdir -p /mnt/erebor/apps/stack && cd /mnt/erebor/apps/stack \ + && curl -fsSLO https://raw.githubusercontent.com/Gerrrt/HomeLab/main/stacks/media/compose.yaml \ + && curl -fsSL https://raw.githubusercontent.com/Gerrrt/HomeLab/main/stacks/media/.env.example -o .env ``` -Jellyfin binds `8096`, reads `erebor/media`, and writes its state to -`erebor/apps`. `node-exporter` binds `9100` and is the whole of how this host -is monitored โ€” see ยง6.1. +The folder is under `erebor/apps` because that is the dataset ยง4 set aside +for application state; its name is arbitrary, since the compose file sets its +own project name. **Do not call it `media`** โ€” that is the library's name one +level up, and the collision confused the first person to do this. + +Confirm `RENDER_GID` still matches what this host reports โ€” it is hard-coded +in `.env`, and `107 render` was re-read on 2026-09-19 โ€” then bring it up: + +```bash +stat -c '%g %G' /dev/dri/renderD128 +docker compose up -d && docker compose ps +``` + +Jellyfin binds `8096` and reads `erebor/media`. `node-exporter` binds `9100` +and is the whole of how this host is monitored โ€” see ยง6.1. Jellyfin's state +goes to a named volume, which is on `erebor/ix-apps` and not on `erebor/apps`; +ยง4 says why that matters. + +**Updating the stack is the same two `curl` lines and `docker compose up -d` +again.** Nothing on this host pulls from `main` on its own: there is no +converge timer here, so a Dependabot bump that merges is not deployed until +someone does this. That is a residual of ADR-0040's shape and not a defect in +it, and it wants a line in `stacks/media/README.md` rather than an issue until +it bites. ### ยง6.1 โ€” Turn the scrape on, and prove it before you do @@ -419,6 +554,16 @@ Then uncomment the four lines at the end of `prometheus/targets/node.yaml` and commit. That directory is a bind mount, so Prometheus re-reads it within five minutes: no restart, no deploy, no `--force-recreate`. +> **Done 2026-09-19.** From the monitoring host: `9100` open, `443` and +> `8096` refused, `node_uname_info` and `node_boot_time_seconds` present, +> **36** `node_filesystem_avail_bytes` series including `erebor`, +> `erebor/media` and `erebor/apps`, and **0** `node_network_*` series. The +> target was uncommented the same morning. Inside the container: +> `renderD128` listed as `root 107` and `id` read +> `uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup),107` โ€” both +> halves of the check below that a shell can make. The transcode is the one +> it cannot, and is still to run. + Expect **no** `node_network_*` series from this host. Those collectors are disabled on purpose, because a bridged container reads its own veth and would chart it as this NAS's throughput; `stacks/media/compose.yaml` carries the @@ -453,6 +598,11 @@ measurement. ## ยง7 โ€” Verify +> **As of 2026-09-19:** the monitoring-host line holds in both halves, the +> device checks hold, and `zpool status erebor` is `ONLINE` with no errors. +> **Not yet read:** a television playing, the QSV transcode, the two extended +> self-tests, and the post-deploy re-reads of the tripwire and port 15. + - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved - A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096` diff --git a/stacks/media/README.md b/stacks/media/README.md index 51e83449..47e76c0b 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -2,13 +2,17 @@ [ADR-0008]'s media tier โ€” the household's media server โ€” on `smaug` (`10.0.40.30`, CasaBonita / VLAN 40), the ThinkServer TS150 that [#413] bought -and [ADR-0040] gave TrueNAS. **The pool is not built yet**; -[`build-the-nas.md`] ยง6 deploys this stack, and this directory is authored -ahead of the storage the way `stacks/sensitive` was authored ahead of -`trinity`. +and [ADR-0040] gave TrueNAS. **Deployed 2026-09-19**, from a copy of this +directory's two files on the pool, under TrueNAS's own Docker; +[`build-the-nas.md`] ยง6 is the procedure. `make up STACK=media` does **not** +work on that host โ€” it renders secrets this stack does not have, on a box +without `make` or `sops` โ€” so a change here reaches `smaug` by re-fetching +`compose.yaml` and `.env.example` from `main` into `/mnt/erebor/apps/stack` +and running `docker compose up -d` there. Nothing pulls from `main` on that +host on its own: a Dependabot bump is merged here and deployed there by hand. ```bash -make up STACK=media +cd /mnt/erebor/apps/stack && docker compose up -d ``` | Service | Image | Port | Purpose | @@ -130,7 +134,7 @@ Every non-obvious line in `compose.yaml` came off the pinned image on is a restart loop. - **65 MiB idle RSS**, which is what the 2 GiB ceiling is a ceiling over. -## The check that is not done yet +## The check that is half done [ADR-0040] keeps the media stack in this repository on the strength of Quick Sync working, and names its own reopen condition: **the iGPU reaching a @@ -140,6 +144,12 @@ different claims. `devices: /dev/dri` and `RENDER_GID` are where the second claim is made; [`build-the-nas.md`] ยง6 is where it gets tested, **before the library exists**, because moving a populated library is a weekend. +**On 2026-09-19 the shell half passed**: `renderD128` is listed inside the +container as `root 107`, and `id` there reads +`groups=65534(nogroup),107`. What that does not prove is a transcode, which +is a claim only Jellyfin's playback settings and a played file can make, and +that is the half still open. + [ADR-0008]: ../../docs/adr/0008-place-services-by-data-trust.md [ADR-0012]: ../../docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md [ADR-0016]: ../../docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md diff --git a/stacks/observability/prometheus/targets/node.yaml b/stacks/observability/prometheus/targets/node.yaml index b39b24c6..67f5e84e 100644 --- a/stacks/observability/prometheus/targets/node.yaml +++ b/stacks/observability/prometheus/targets/node.yaml @@ -62,24 +62,20 @@ # --- smaug --------------------------------------------------------------- # -# WRITTEN OUT AND NOT ENABLED. The host holds 10.0.40.30 and the firewall pass -# is verified, but nothing answers on 9100 yet: the exporter is a container, the -# container needs Docker, and Docker on TrueNAS needs the apps pool โ€” which is -# on the ZFS mirror that does not exist. Every route to node_exporter on this -# host runs through that pool, so this cannot be live before the drives land no -# matter where the exporter is put. ADR-0040's "the scrape, then the stack" is -# right about the order these are AUTHORED in and is not achievable for -# liveness. +# ENABLED 2026-09-19, after the three checks below passed from this host. It +# was written out and commented from 2026-09-17 until then, because every +# route to node_exporter on this host runs through the ZFS pool โ€” the exporter +# is a container, the container needs Docker, and Docker on TrueNAS needs the +# apps pool โ€” and enabling it before the pool existed would have meant +# `up == 0` from the moment the file loaded, with InstanceDown at severity +# critical paging `urgent` every 4 hours, forever. ADR-0017 refuses a check +# that is permanently red for a known reason by name, and +# targets/blackbox-dns.yaml and the commented Windows job in stacks/lab make +# the same call. # -# Enabling it today would mean `up == 0` from the moment the file loaded, and -# InstanceDown is `up == 0` at severity critical โ€” routed to `urgent`, repeating -# every 4 hours, forever. ADR-0017 refuses a check that is permanently red for a -# known reason by name, and targets/blackbox-dns.yaml and the commented Windows -# job in stacks/lab make the same call. -# -# When the pool exists and stacks/media is deployed, verify FROM THIS HOST โ€” -# which can reach 9100 and may not reach anything else on 40 โ€” BEFORE -# uncommenting: +# What was verified FROM THIS HOST โ€” which can reach 9100 and may not reach +# anything else on 40 โ€” before the four lines below were uncommented, and what +# to re-run if the exporter is ever rebuilt: # # nc -z -w3 10.0.40.30 9100 # curl -s http://10.0.40.30:9100/metrics | grep -E '^node_(uname_info|boot_time_seconds)' @@ -89,15 +85,18 @@ # produces a container's filesystems, or none, while `up` still reads 1 and the # target shows green โ€” so a count of 0 means every disk rule on this host is # blind and nothing will say so. It must be the number of mounted filesystems on -# smaug, and after ยง3 that includes erebor. +# smaug, and after build-the-nas.md ยง3 that includes erebor. On 2026-09-19 it +# read 36: the boot pool's datasets, `erebor`, `erebor/media`, `erebor/apps`, +# and the `.system` and `ix-apps` datasets TrueNAS creates for itself. 443 and +# 8096 were refused from here in the same minute, as ยง0.5's rules say. # # Expect NO node_network_* series at all: stacks/media/compose.yaml disables # those collectors on purpose, because a bridged container reads its own veth # and would chart it as this NAS's throughput. That file carries the -# measurement and the argument. +# measurement and the argument. The count was 0 on the day. -# - targets: ["10.0.40.30:9100"] -# labels: -# instance: smaug -# role: nas -# vlan: "40" +- targets: ["10.0.40.30:9100"] + labels: + instance: smaug + role: nas + vlan: "40"