diff --git a/README.md b/README.md index ca0b8865..e73ee90b 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,8 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 42 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 28 +- **Documented decisions and runbooks.** 43 ADRs covering what was chosen + and what was rejected — including the costs accepted knowingly; 30 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. @@ -207,7 +207,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 41 architecture decision records +│ ├── adr/ # 43 architecture decision records │ └── runbooks/ # successor handover (start here), deploy, converge, │ # add device, rotate creds, certs, key backup, │ # purge, restore the firewall, restore the stack, diff --git a/docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md b/docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md index 5899912b..5acde4b6 100644 --- a/docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md +++ b/docs/adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md @@ -23,6 +23,18 @@ > a second source subnet — `172.31.0.0/24`, the WireGuard peers — which is why > it also widens the tripwire this ADR decided. `172.30.` is the range; > `172.31.` is the tunnel. Nothing here is amended. +> +> --- +> +> **One rule is superseded, 2026-09.** The management-plane rule below admits +> `8006`, `8007` and `22` from `10.0.50.0/24` only. Since +> [ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +> it also admits one address on VLAN 30, `10.0.30.70`, to `8006` alone — the +> deployment host, driving the API it exists to drive, and the host the +> tunnel above terminates on. That is a change to a decision rather than a +> premise, which is why it is a supersession of that one rule and not a +> note; every other bullet below stands. The text is left as written, per +> ADR-0001. ## Context diff --git a/docs/adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md b/docs/adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md index 4c1fcce4..377d6270 100644 --- a/docs/adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md +++ b/docs/adr/0039-decline-proxmox-clustering-while-ifrit-is-the-range.md @@ -2,6 +2,15 @@ **Status:** Accepted · 2026-09 +> [!NOTE] +> **One consequence is amended, 2026-09.** The first bullet under +> Consequences says nothing on VLAN 30 gains a path to `8006` on either +> hypervisor. Since +> [ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +> one address does — `10.0.30.70`, the deployment host, to `Saruman` alone. +> The decision this ADR records is unchanged: neither hypervisor reaches the +> other. The text below is left as written, per ADR-0001. + ## Context [#443](https://github.com/Gerrrt/HomeLab/issues/443) asked for a decision diff --git a/docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md b/docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md index 096a1e05..d0610e95 100644 --- a/docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md +++ b/docs/adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md @@ -2,6 +2,16 @@ **Status:** Accepted · 2026-09 +> [!NOTE] +> **One of the conditions under *What would reopen this* has fired, 2026-09.** +> The Proxmox `8006` rule widens to admit the jumpbox, by +> [ADR-0043](0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +> — the decision this ADR's consequences left to #436, and the one that names +> the jumpbox `phoenix` at `10.0.30.70`. Nothing here is amended: routed mode, +> decision 3, is what that widening relies on to admit one host and not every +> peer, and ADR-0014 carries the supersession of its rule. The text below is +> left as written, per ADR-0001. + ## Context The estate has no remote access. Every path in is a workstation physically on diff --git a/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md new file mode 100644 index 00000000..d939d386 --- /dev/null +++ b/docs/adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md @@ -0,0 +1,326 @@ +# ADR-0043: Keep the CA on `prometheus`, and build `phoenix` as the deployment host + +**Status:** Accepted · 2026-09 · supersedes the management-plane rule of +[ADR-0014](0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md) +and amends the first consequence of +[ADR-0039](0039-decline-proxmox-clustering-while-ifrit-is-the-range.md) + +## Context + +[#436](https://github.com/Gerrrt/HomeLab/issues/436) is the prerequisite for +the Packer, OpenTofu and Ansible work that follows it. The estate has +thirty-odd ADRs and a convergence loop that refuses to deploy without a pinned +signature, and it builds virtual machines by clicking in a browser from a +Hicks workstation. The toolchain that would change that wants a host: one that +holds a Proxmox API token, an SSH key and a checkout, and stays up. The issue +places it — a guest on `Saruman`, VLAN 30, a dedicated Proxmox user rather +than `root@pam` — and then carries one question it declines to answer by +default: where the estate's certificate authority should live, given that the +root key sits today on the monitoring host's unencrypted disk. + +The issue asked for that to be decided rather than inherited, and for the key +not to move as a side effect of the build. Both are honoured here. Deciding it +meant reading the root first, and the reading changed the question. + +### The issue's premise, corrected + +The issue says the tree is already shaped for a move either way, because +"`stacks/sensitive`'s step-ca is already an intermediate beneath that root". +It is not. The estate's root is minted with +`basicConstraints=critical,CA:TRUE,pathlen:0` +([`scripts/gen-certs.sh`](../../scripts/gen-certs.sh)), which forbids any CA +beneath it, and +[ADR-0037](0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md) +gave the tier a root of its own for exactly that reason — after measuring the +failure, `path length constraint exceeded`, in the Go `crypto/x509` every +client here is built on. "An intermediate beneath the lab CA" is the sentence +ADR-0037 was written to retract, and the issue was written before it. + +Two things follow. The CA question is not a tree-shape question; there is one +root, it can have nothing beneath it, and the only thing to decide is where +one key file sits. And losing that file does not mean "cannot reissue" — it +means a new root: `--ca --force`, two leaves, one copy through the Mac, and a +re-trust everywhere the old `ca.pem` was imported. That path is already +written in [`successor-handover.md`](../runbooks/successor-handover.md), and +ADR-0037 counted its cost. It is bounded. What is *not* written is the list of +places `ca.pem` was ever trusted, which ADR-0037 admits is "the Mac's system +store and Firefox's separate one, at least, and whatever else nobody wrote +down". + +### What `phoenix` is, and what it holds + +A deployment host is not a server that serves anything. It is the one machine +in the estate whose purpose is to hold **credentials for other machines**: a +Proxmox API token that can create and destroy guests on the hypervisor whose +own firewall ADR-0014 relies on, an SSH key that the toolchain will inject into +every guest it builds, and a checkout of this repository. It is the estate's +first hypervisor API credential — nothing in this repository calls `pveum` — +and it is a pet: the thing that rebuilds everything else, and that nothing +rebuilds. Since +[ADR-0042](0042-terminate-the-remote-path-on-the-lab-and-route-it.md) it is +also the host the estate's first inbound path from the internet terminates +on: the WireGuard peers of `172.31.0.0/24` arrive on `phoenix` and are routed +into the lab from it. + +That is the property that decides the CA question, and it decides it before +any argument about disks. The estate's signing key on the same host as the +estate's lateral-movement credentials means one compromise yields both "I can +reach every box" and "I can mint a certificate any of them will trust". +Keeping those two things on different machines is not a refinement; it is the +whole reason to have a key that never leaves one host. A machine that is both +the tunnel's end and the holder of the hypervisor's token is the last place in +the estate for its signing key. + +### Where the key could go, counted + +**On `prometheus`, where it is.** A 2012 laptop with an unencrypted disk and +swap, recorded as an accepted residual in [`SECURITY.md`](../../SECURITY.md) +on a threat model that excludes physical access to the rack, and already the +custodian of the estate's two other root secrets: the age key +([ADR-0005](0005-secrets-with-sops-and-age.md)) and the tier's cold root +(ADR-0037 §2). `gen-certs.sh` prices its passphrase-less key against exactly +this: "the exposure this time is bounded by the key never leaving this host". + +**On `phoenix`, on VLAN 30.** The issue frames this as trading one unencrypted +disk for another. The disks are equivalent; the segments are not. ADR-0014 +calls ImaginationLAN "the segment that holds attackers" in its own decision, +scopes `ifrit`'s attack VM to reach `10.0.30.0/24` on purpose, and has it take +a lease from the pool like any other guest. `10.0.30.70` is inside the range +the estate's own Kali is configured to scan. +[`build-the-lab-guest.md`](../runbooks/build-the-lab-guest.md) already says +what that means for this file: "a copy of it on a machine that sits on the +segment built to hold attackers is a different class of problem from a leaked +leaf". And the transfer the move is meant to remove does not go away — it +inverts and grows. `99 → 30` is closed, so today one leaf, `alexander`'s, is +carried through the Mac with `scp -3`. With the CA on `phoenix`, the Grafana +leaf on `prometheus` acquires that hop, `alexander`'s keeps it, and both hops +now carry a private key rather than a certificate. + +**Cold, on the offline medium that holds the age key's second copy.** This is +the tier's model, and it works for the tier because an intermediate does the +daily issuing while the root sleeps. The estate's root has `pathlen:0`; there +is no intermediate and cannot be one. "Cold" here means every issuance mounts +removable media, which is the interactive-issuance failure `gen-certs.sh` +rejects in as many words: "how a lab ends up with one long-lived certificate +nobody dares reissue". + +**On `trinity`, beside the tier's root.** The tier's root is not on `trinity`. +ADR-0037 §2 keeps it cold on `prometheus` and ships `trinity` a bundle with the +key left out — so "beside the tier root" already means `prometheus`. `trinity` +is also the host ADR-0037 describes as carrying the largest attack surface in +the estate, in a section titled *What `trinity` could forge*, and it is not +built ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). + +**Re-mint at `pathlen:1`, and issue from an intermediate on `phoenix`.** The +one option that gives the deployment host issuing power without the root, and +the one that has to be answered rather than dismissed. ADR-0037 costed this +exact re-mint and declined it for a stronger warrant — so that the household's +devices could trust one root — with the test "not the size of the cost but who +pays it and for what". A deployment host is a weaker warrant than the +household. And it re-creates precisely what ADR-0037 closed: an intermediate +on VLAN 30 can mint `grafana.matrix.elysium`, and Prometheus, blackbox and the +operator's browser will accept it. + +There is one PKI role a deployment host can hold without touching any of +this: carrying the *public* root, `ca.pem`, which `gen-certs.sh` prints as +"safe to distribute", to the guests it builds. That is noted and not decided. + +### Three declines this has to answer + +[ADR-0002](0002-vlan-segmentation-strategy.md) and +[ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md) both say a +bastion "is not worth it at this scale", and a host called a jumpbox reads as +one. Both declines are about a specific thing: an SSH chokepoint a compromised +Hicks workstation would have to traverse to reach Winterfell. `phoenix` is not +that and structurally cannot be. It is a guest on VLAN 30, VLAN 30 has no pass +into 99, and "guests get no such rule" is +[ADR-0007](0007-defensive-estate-and-offensive-range.md)'s sentence, restated +three times in [`network.md`](../network.md). A machine that cannot reach the +segment a bastion exists to protect is not that bastion. + +[ADR-0021](0021-converge-on-a-timer-instead-of-deploying-over-ssh.md) chose +pull over push, and rejected `ansible-pull` as "a wrapper around `make up`". +It decided how a host **that already exists and runs a stack** gets its +configuration. It decided nothing about how a host comes into existence, +because there is no host-creation plane in the estate at all: eight guests +are planned, six of them Windows, every one built by hand from a runbook. +ADR-0021 also kept a push plane on purpose and said so — "`oracle` and +`saruman` are pushed to by `deploy-agent.sh`. That is not a transitional +state" — and `phoenix` inherits that half, not the half ADR-0021 decided +against. + +### The rule the issue did not count + +The issue says the host "needs no new rule", and on `morpheus` that is true: +[ADR-0031](0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md) +already carries Hicks to all of VLAN 30, and everything `phoenix` pushes to is +on its own segment. But the Proxmox API is not behind `morpheus`. ADR-0014 +closes it on the hypervisor itself: the Proxmox firewall on `Saruman` "admits +`8006`, `8007` and `22` from `10.0.50.0/24` only", written as the one control +against "an attacker sharing a broadcast domain with the estate's hypervisor". +A guest on VLAN 30 that drives the API has to be admitted through it, and +today nothing on VLAN 30 is — ADR-0039 lists that as a consequence it relies +on. + +ADR-0042 saw the same collision and left it here: "whichever issue lands +first owns it, and routed mode is what stops a widened `8006` rule from +silently admitting every VPN peer." This is that decision. + +So building what #436 asks widens that rule by exactly one source address on +exactly one port, on the hypervisor's own firewall and nowhere on `morpheus`. +The alternatives are a host on Hicks, which has no hardware to be a guest of, +and a host on Winterfell, which the issue rejects for ADR-0008's reason and +which would need a rule on `morpheus` instead. The widening is accepted, with +its cost stated below rather than found later. + +## Decision + +**The estate's root key stays on `prometheus`. `phoenix` is built as a guest +on `Saruman` at `10.0.30.70`, holds the hypervisor's first API credential and +no key that signs anything, and gets one pass to `8006` on `Saruman` and no +pass anywhere else.** Four parts. + +**1. `certificates/ca-key.pem` does not move, and this ADR is the reason the +next proposal argues with.** The arguments are the ones counted above, in +order of weight: the host that holds credentials for every other host must +not also hold the key every other host trusts; VLAN 30 is the segment ADR-0014 +built to hold attackers, at an address the range is scoped to reach; the +through-the-Mac transfer inverts and doubles rather than disappearing; and +custodianship of root secrets is already a property of `prometheus` decided +twice, so moving one of three creates a second place to remember rather than +less concentration. Nothing in `gen-certs.sh` binds the CA to a host — the +path is relative to whichever checkout runs it — so this invariant is a +sentence in this document and a comment beside the `pathlen:0` line, and +those are what hold it. + +**2. `phoenix`.** A VM, not an LXC, on `Saruman`, VMID `170`, `10.0.30.70/24` +with a Kea reservation, Ubuntu Server LTS, and a Final Fantasy summon like +every other named host on the segment — `shiva`, `ifrit`, `alexander`, `odin` +and ADR-0029's six. That is a segment-local pattern and is claimed as one; +[ADR-0038](0038-name-the-nas-smaug-and-reserve-zion-for-the-box-that-does-not-exist.md) +holds that the estate has no naming scheme, and it is right. It runs no +compose stack and no Docker. Its Alloy is the native package, deployed by +`scripts/deploy-agent.sh` from the Mac, and pushes to the lab's stores on +`alexander` — never to `10.0.99.20`, per ADR-0007 — which makes it the first +use of that script's `--monitoring-host` flag and the second client the lab's +Prometheus and Loki open their ports for, or the first, whichever of it and +`odin` is built first. + +Three sentences bound what it is, and each is a control only because it is +written down: + +- **It gets no pass out of VLAN 30.** It is a guest, and ADR-0007's "guests + get no such rule" covers it exactly as it covers `alexander` and `odin`. A + rule from `10.0.30.70` into Winterfell would make it the bastion ADR-0002 + and ADR-0012 twice declined; that is a new ADR, not a runbook step, and + [`firewall-claims.yaml`](../firewall-claims.yaml) will not catch it, because + it records wholesale posture and "deliberately NOT a list of what each + segment can reach". +- **It is a deployment host, not a transit host into any other segment.** + What crosses it is ADR-0042's tunnel, routed into the lab and nowhere else, + and that is the whole of it: no interactive SSH to another machine goes + through it, `ProxyJump` through `phoenix` is not a supported path, and the + day it becomes one is the day ADR-0002's decline is reopened rather than + worked around. +- **It holds no age key and runs no `make up` against another host.** The + toolchain builds machines. Once a machine exists and runs a stack, ADR-0021 + owns how it converges, and a `phoenix` that pushed compose stacks over SSH + would be the `ansible-pull` rejection re-litigated with a different tool. + +**3. The hypervisor credential, and the one rule that admits it.** A dedicated +Proxmox user, `phoenix@pve`, with a role of its own scoped to guests and the +storage and bridge they need — `VM.*`, `Datastore.AllocateSpace`, +`Datastore.AllocateTemplate`, `SDN.Use` on `/vms`, the two storages and +`vmbr0` — plus read-only audit on the node, and nothing under `/nodes` beyond +that. Never `PVEAdmin`, never `root@pam`: the token that can create guests +must not be able to touch the host firewall ADR-0014 depends on. The token is +issued with privilege separation off, as the issue asks, so it carries the +user's permissions and no second set to keep in step. + +The credential lives on `phoenix` at mode 600, outside this repository. That +is a stated deviation from the issue's "credential into `secrets/`", and it is +forced rather than chosen: `scripts/check_sops_rules.py` derives the paths it +proves each rule against from the stack directories, so a `secrets/phoenix` +rule matches nothing and fails CI until a `stacks/phoenix` exists, and +`bootstrap.sh` needs an example file and a placeholder rule to run at all. The +encrypted-in-repo form arrives with the toolchain issue that consumes the +token, which is what defines the file's shape. Until then it is +[ADR-0015](0015-give-oracle-the-off-host-jobs.md)'s posture in reverse: one +file, one host, nothing in git. + +ADR-0014's rule on `Saruman` gains one line: `10.0.30.70` to `8006`, TCP, +unlogged. Not `22` — the API is the whole interface, and `qm` over SSH is the +clicking this host exists to replace. The cost, stated: the rule is by source +address on a segment where the attack VM shares the wire, so it can be spoofed +by a guest that takes `.70` while `phoenix` is off. What that buys an attacker +is reachability to an authenticated API from a less trusted segment than +Hicks — Proxmox's login surface, not its privileges. The token is the control; +the rule is the door. ADR-0042's routed mode is what keeps the door that +narrow: a tunnel peer arrives with its own `172.31.` source and does not +inherit a pass written for `10.0.30.70`, which under NAT every peer would. `firewall-claims.yaml` cannot see this either, because +it lives in `/etc/pve` and not in pf, and the marked amendment on ADR-0014 and +ADR-0039 is the only place the widening is recorded. + +**4. The custody of `ca-key.pem` is a found gap, named here and fixed under +[#496](https://github.com/Gerrrt/HomeLab/issues/496).** The age key has +[ADR-0024](0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) +and [`back-up-the-age-key.md`](../runbooks/back-up-the-age-key.md): a second +recipient off-host and off-estate, proved on demand by decrypting the real +ciphertext, nagged at ninety days. The tier's root goes to the same offline +medium by [`build-the-tier-ca.md`](../runbooks/build-the-tier-ca.md). The +estate's root has nothing — not a copy, not a sentence. The loss is bounded, +as counted above, and the reason it gets the age key's backup and not, yet, +the age key's alerting is one sentence: an age key has no re-mint path, and a +CA does. Two things are decided so the issue is specifiable rather than +aspirational. It is not SOPS-in-git — ADR-0037 rejected that for the tier's +root, and this key is one of the ones purged from history. And it is not a +passphrase, which reopens the trade `gen-certs.sh` closes deliberately. The +proof, when it is built, needs no decryption: the public half of the backup +compared with the public half of `ca.pem`, a sibling of +`verify-key-backup.sh` with the same refusal to run against the live key. + +## Consequences + +- **The lab's Prometheus and Loki open to a first off-host client, and the + runbooks stop assuming which one.** `stacks/lab/compose.yaml` ships both + `ports:` blocks commented, and + [`build-the-soc-guest.md`](../runbooks/build-the-soc-guest.md) §7 was the + step that uncomments them for `odin`. `phoenix` is the same kind of client + and may be built first, so its runbook carries the same step conditionally + and the comments in `stacks/lab` name both. Being scraped by `alexander` + instead, the way ADR-0029's six are, was considered and loses on one fact: a + deployment host's value in an incident is its logs — what it did, to which + guest, when — and a scrape carries nothing to Loki. +- **One address on VLAN 30 now reaches the hypervisor's API**, and two ADRs + said none did. ADR-0014's rule is superseded by this one and ADR-0039's + first consequence is amended, each with a marked note per + [ADR-0001](0001-record-architecture-decisions.md), and the text of both is + left as written. ADR-0042 listed this widening under what would reopen it, + and it has happened; it is discharged rather than reopened, because what + that ADR asked of the widening — routed mode, so the pass admits one host + and not every peer — is what part 3 relies on, and it carries a note saying + so. +- **Its telemetry lands where no house alert looks, and ADR-0042 already says + so.** The Alloy on the host that terminates the internet's only path in + pushes to `alexander`, per ADR-0007, and nothing on Winterfell reads that + store. This ADR does not change that; it notes that the deployment host and + the tunnel host are the same machine, so the gap ADR-0042 records is this + host's. +- **The estate holds its first hypervisor API credential**, and its scope is + the sentence to argue with when the toolchain wants more. The Packer and + OpenTofu issues will find out which privileges the role is missing; they add + privileges to the role, not the role to `/`. +- **No purchase.** A guest on hardware already owned. Every other new host in + the roadmap has arrived with a row in the buy table, and this one does not. +- **`gen-certs.sh`'s `pathlen:0` comment gains a line** saying the key does + not move to the deployment host and why, the way ADR-0037 amended the same + comment. That comment is where the next person will look. +- **`docs/hardware.md`'s Alloy agent count is wrong on the day `phoenix` is + built**, not before: a `**Not built yet**` row does not count, and dropping + the marker fails `check_docs.py` until the sentence says the new number. + The runbook's last section lists that with the other build-day edits. +- **Reopened by:** ADR-0037's own condition — Grafana renewing from step-ca, + at which point the estate's CA can be argued out of existence and this + question with it; `phoenix` needing any pass out of VLAN 30, or `22` on + `Saruman`, either of which is the bastion this ADR says it is not; or a + client appearing that must trust both roots. diff --git a/docs/architecture.md b/docs/architecture.md index c4604870..9e3d96af 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -185,9 +185,10 @@ hole from the monitoring VLAN into the monitored one. | Host | VLAN | Stack | Contents | | --- | --- | --- | --- | | `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, docker-socket-proxy, Alloy | -| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running one guest — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)) — and seven more planned below: `odin` for the security tooling (ADR-0030) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | +| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, running one guest — `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)) — and eight more planned below: `odin` for the security tooling (ADR-0030), `phoenix` for the deployment toolchain (ADR-0043) and ADR-0029's six-machine domain; Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest | | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | +| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | **Not built yet** — the deployment host: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) once that is built; the one thing it reaches that no other guest does is `8006` on `Saruman`, by a single host-firewall rule. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copy of the firewall export (`make backup-firewall`). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **The host is built and the pool is not**: TrueNAS is installed on its boot SSD, it holds the static above since 2026-09-16, and the four inbound rules are created and verified in position — but the ZFS mirror waits on two Exos X20 drives and nothing is deployed on it. The stack is authored and CI-validated ahead of the storage, the way `stacks/sensitive` was ahead of `trinity`: Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Runs no agent of its own: this host is scraped and never pushes, which makes it the estate's first scraped host and is why [#256](https://github.com/Gerrrt/HomeLab/issues/256) exists — that issue now also has a fork in it, because TrueNAS ships its own metrics endpoint where the issue specifies `node_exporter` | diff --git a/docs/network.md b/docs/network.md index 8ee1201c..bb5a5892 100644 --- a/docs/network.md +++ b/docs/network.md @@ -370,6 +370,26 @@ Where things get broken on purpose. pushes to `alexander` and not to Winterfell — "guests get no such rule" covers it — and ADR-0029's six machines report to it as agents. Every path it needs is intra-segment, so it adds no firewall rule. +- A third guest, `phoenix`, is planned at `10.0.30.70` — the next decade — as + the deployment host: the Proxmox API token, the SSH key and the checkout + that the Packer, OpenTofu and Ansible work after + [#436](https://github.com/Gerrrt/HomeLab/issues/436) runs from, placed by + [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md). + **Not built**; it enters the table above when it is + ([`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md)). It runs no stack + and holds no key that signs anything — the estate's CA stays on + `prometheus`, and that ADR says why. It is also where + [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)'s + WireGuard tunnel terminates once the endpoint it records as blocking exists; + the peers reach the lab through it and nothing else. Its Alloy pushes to + `alexander` and + not to Winterfell, and it gets no pass out of this segment: a rule from it + into 99 would make it the bastion ADR-0002 and ADR-0012 declined. It adds + nothing on `morpheus`. What it adds is on `Saruman`: one line in the + hypervisor's own firewall admitting `10.0.30.70` to `8006`, the first + address on this segment with a path to the Proxmox API, which + `firewall-claims.yaml` cannot see because it lives in `/etc/pve` and not in + pf. - `Saruman` runs an Alloy agent and is the one host on this segment with a path into Winterfell: a single pass, `10.0.30.110 → 10.0.99.20` on 9090 and 3100 TCP, unlogged and above the ADR-0014 tripwire. The hypervisor's own telemetry diff --git a/docs/roadmap.md b/docs/roadmap.md index d94292dc..59065de3 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1028,6 +1028,32 @@ what left this one unfireable for months. unifying that. - **[#100](https://github.com/Gerrrt/HomeLab/issues/100) Automate the Grafana +- **[#436](https://github.com/Gerrrt/HomeLab/issues/436) Build a deployment + jumpbox on ImaginationLAN, and decide where the CA lives.** Decided by + [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md); + the guest is not built. The host is `phoenix`, `10.0.30.70`, a guest on + `Saruman` that holds the estate's first Proxmox API credential and the SSH + key the toolchain will inject into what it builds — the prerequisite for + the Packer, OpenTofu and Ansible issues, which today have nowhere to run + from. The CA question the issue carried was answered by reading the root + first: the issue's premise — that the tier's step-ca already sits beneath + the estate's CA — is the sentence ADR-0037 retracted, and the estate's + root is `pathlen:0`, so the only question was where one key file sits. It + stays on `prometheus`, because the host that holds credentials for every + other host must not also hold the key every other host trusts, and + because VLAN 30 is the segment ADR-0014 built to hold attackers. Two things + the issue did not count: the Proxmox firewall on `Saruman` admits `8006` + from Hicks only, so the build widens ADR-0014's rule by one address on one + port, recorded as a marked amendment there and on ADR-0039; and + `certificates/ca-key.pem` has no backup or custody story at all, unlike + the age key and the tier's root — found, named in the ADR with what the + answer is not, and carried by + [#496](https://github.com/Gerrrt/HomeLab/issues/496). The same host is + where [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) + terminates the remote path; that ADR left the `8006` question to this issue, + and it is taken here. + → [runbook](runbooks/build-the-jumpbox.md) + ## Decided but not built Accepted ADRs with no work behind them. Recorded here because an accepted ADR diff --git a/docs/runbooks/build-the-jumpbox.md b/docs/runbooks/build-the-jumpbox.md new file mode 100644 index 00000000..e050329d --- /dev/null +++ b/docs/runbooks/build-the-jumpbox.md @@ -0,0 +1,327 @@ +# Runbook: Build `phoenix`, the deployment host + +**Target:** `phoenix` — a guest on `Saruman`, ImaginationLAN (VLAN 30) +**Time:** about an hour, most of it the OS installer +**You will need:** the Proxmox web UI on `Saruman` (or a shell on it through +the KVM or `shiva`), an Ubuntu Server ISO, the pfSense UI on `morpheus` for one +reservation, a shell on `alexander` for §5 if the lab's ports are still shut, +and the Mac for §6 — it is the only machine that reaches this segment with a +checkout in hand +**After this:** [`open-the-remote-path.md`](open-the-remote-path.md), if the +remote path is wanted — +[ADR-0042](../adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md) +terminates the WireGuard tunnel on this host, and its runbook begins where +this one ends + +This builds the host [ADR-0043](../adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md) +decided: the one machine whose purpose is to hold credentials for other +machines — a Proxmox API token, an SSH key, a checkout — so that the Packer, +OpenTofu and Ansible issues that follow +[#436](https://github.com/Gerrrt/HomeLab/issues/436) have somewhere to run +from. It is the shape of [`build-the-lab-guest.md`](build-the-lab-guest.md) +and leans on it: where a step is identical, this says so and points there +rather than carrying a second copy that drifts. Where this guest differs, it +differs in what it does **not** get: no compose stack, no Docker, no age key, +and no certificate authority. + +--- + +## 0. What is decided, and why + +| | Decision | Why this and not the obvious alternative | +| --- | --- | --- | +| Name | `phoenix` | Every named host on this segment is a Final Fantasy summon — `shiva`, `ifrit`, `alexander`, `odin`, ADR-0029's six — and this continues it. The one whose meaning fits the job: the thing that rebuilds everything else | +| Address | `10.0.30.70/24` | Static below `.100`, the next decade after `odin`'s `.60`. `.20` is reserved for `Saruman`'s own move ([`build-the-playground.md`](build-the-playground.md) §3) | +| VMID | `170` | Last octet legible from `qm list`, as `alexander` is `140` and `odin` is `160` | +| Kind | **VM, not LXC** | The toolchain will want nested virtualisation off and a kernel of its own for Packer's boot ISOs; an LXC shares the host's. Same answer as the other two guests, for a different reason | +| OS | **Ubuntu Server LTS** | Same reason as `alexander`: `config.alloy` tails `/var/log/auth.log` and `/var/log/syslog`, and a journald-only install collects nothing from either while reporting healthy. §7 is the check | +| vCPU | 2 | Packer waits on other machines; OpenTofu plans. Neither is compute | +| RAM | 4 GiB | Enough for Packer to hold a boot ISO in the page cache while a build runs. Bounded, not measured | +| Disk | 32 GB | A checkout, a few ISOs, the toolchain's binaries. `discard=on` so a deleted ISO returns its space to `local-lvm` | +| Stack | **none** | It runs no compose stack and no Docker at all. The Alloy agent is the native package, deployed by `scripts/deploy-agent.sh` the way `Saruman`'s is | +| Pushes to | `alexander`, `10.0.30.40` | Never `10.0.99.20`: guests get no pass into Winterfell (ADR-0007), and everything this host needs is on its own segment | +| Remote path | **Not here** | ADR-0042 terminates the WireGuard tunnel on this host, and [`open-the-remote-path.md`](open-the-remote-path.md) builds it — after this runbook, and only once the endpoint that ADR records as blocking exists | + +> [!CAUTION] +> **`certificates/ca-key.pem` does not come here, and ADR-0043 is why.** This +> is the host that holds credentials for every other host on the segment, and +> — once ADR-0042's tunnel is built — the host the estate's only inbound path +> from the internet terminates on. The +> key every other host trusts stays on `prometheus`, the same rule +> [`build-the-lab-guest.md`](build-the-lab-guest.md) §5 gives for `alexander`. +> If the toolchain ever needs a certificate for a guest it built, the leaf is +> issued on the monitoring host and carried here the way that runbook carries +> it — never the key, and never `make certs ARGS=--ca` on this machine. + +## 1. Create the VM + +On `Saruman`, as for `alexander` (§1 there explains every flag; nothing about +them changes except the numbers): + +```bash +qm create 170 \ + --name phoenix \ + --ostype l26 \ + --cpu host --cores 2 --sockets 1 \ + --memory 4096 --balloon 0 \ + --scsihw virtio-scsi-single \ + --scsi0 local-lvm:32,discard=on,iothread=1 \ + --net0 virtio,bridge=vmbr0 \ + --agent enabled=1 \ + --onboot 1 \ + --ide2 local:iso/ubuntu-26.04-live-server-amd64.iso,media=cdrom \ + --boot order='scsi0;ide2' +``` + +The ISO name is whatever `pvesm list local --content iso` prints; `alexander` +runs 26.04 and the same image serves. `--onboot 1` is for the day `Saruman` +reboots under a maintenance window: a deployment host that stays down is a +toolchain nobody can run, and nothing else brings it back. + +## 2. Install Ubuntu Server + +As `alexander`'s §2, with these values. **Hostname `phoenix`** — +`deploy-agent.sh` takes the host label from `hostname` on the target, and it +becomes `instance` and `host` on every series and line this guest ships. + +| | | +| --- | --- | +| Address | `10.0.30.70/24` | +| Gateway | `10.0.30.1` | +| DNS | `10.0.30.1` — Unbound on the gateway ([ADR-0010](../adr/0010-keep-the-resolver-on-the-gateway.md)) | + +Install OpenSSH. **Install no Docker** — not the snap the installer offers, +not the repository package afterwards. There is nothing here for it to run, +and `deploy-agent.sh` reads the absence as its cue to install the native +package. + +## 3. The reservation on `morpheus` + +Read the guest's MAC from the hypervisor: + +```bash +qm config 170 | grep net0 +``` + +Then one reservation on `morpheus`, under *Services → DHCP Server → +ImaginationLAN*, mapping it to `10.0.30.70`. + +> [!IMPORTANT] +> **The reservation is not what protects an address below `.100`.** The pool +> is `.100–.200` and this address sits outside it, so nothing was going to +> lease it anyway. The reservation is there so the address is recorded where +> a reader looks for it — the static is set on the host, the reservation on +> the server, and both are done because either alone is a single point of +> drift. [`build-the-lab-domain.md`](build-the-lab-domain.md) §2 says the same +> for the domain's six. + +## 4. The Proxmox user, the role, the token — and the door + +**On `Saruman`.** This is the estate's first hypervisor API credential, and +the role is its own so that what it can do is a list rather than +`PVEAdmin`: + +```bash +pveum role add PhoenixBuilder --privs \ + "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit \ + VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network \ + VM.Config.Options VM.Console VM.Monitor VM.PowerMgmt VM.Audit VM.Snapshot \ + Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use" +pveum user add phoenix@pve --comment "deployment host, ADR-0043" +pveum acl modify /vms --users phoenix@pve --roles PhoenixBuilder +pveum acl modify /storage/local --users phoenix@pve --roles PhoenixBuilder +pveum acl modify /storage/local-lvm --users phoenix@pve --roles PhoenixBuilder +pveum acl modify /sdn/zones/localnetwork/vmbr0 --users phoenix@pve --roles PhoenixBuilder +pveum acl modify /nodes/saruman --users phoenix@pve --roles PVEAuditor +pveum user token add phoenix@pve builder --privsep 0 +``` + +Four of those are worth knowing rather than copying: + +- **`/vms`, the two storages and the bridge, and nothing under `/nodes` but + audit.** The token that can create guests must not be able to touch the + host firewall ADR-0014 depends on, and `PVEAuditor` on the node is + read-only. When Packer or OpenTofu fail with `Permission check failed`, + the fix is a privilege added to `PhoenixBuilder`, not the role granted at + `/`. +- **`/nodes/saruman`** is the node name as `pvesh get /nodes` prints it, which + is lower-case whatever the documents call the box. +- **`--privsep 0`** is the issue's choice: the token carries the user's + permissions and there is no second set to keep in step. The trade is that + it is exactly as powerful as the user, which is why the user is this narrow. +- **The secret prints once.** Copy it now; it cannot be shown again, only + regenerated. + +**Then the door.** ADR-0014 closes `8006` on `Saruman` to `10.0.50.0/24` and +this guest is not on it. ADR-0043 admits one address, on this port and no +other. In `/etc/pve/nodes/saruman/host.fw`, beneath the three rules ADR-0014 +wrote: + +```ini +IN ACCEPT -source 10.0.30.70 -p tcp -dport 8006 -log nolog +``` + +> [!CAUTION] +> If that file does not exist yet, ADR-0014's rules were never applied on +> `Saruman` and the firewall is off. Do not turn it on as a side effect of +> this line — read [`build-the-playground.md`](build-the-playground.md) §4 +> first, with the KVM console to hand, because a `DROP` input policy with the +> rules unrendered locks you out of a machine whose console is a switch away. + +**On `phoenix`**, the credential and the key: + +```bash +install -d -m 700 ~/.config/proxmox +umask 077 +cat > ~/.config/proxmox/phoenix.env <<'EOT' +PROXMOX_URL=https://10.0.30.110:8006/api2/json +PROXMOX_TOKEN_ID=phoenix@pve!builder +PROXMOX_TOKEN_SECRET= +EOT +ssh-keygen -t ed25519 -C phoenix -f ~/.ssh/id_ed25519 +git clone https://github.com/Gerrrt/HomeLab.git ~/HomeLab +``` + +That file is mode 600, on this host, and **not in the repository** — a +stated deviation from #436's "credential into `secrets/`", and ADR-0043 +records why it is forced: `check_sops_rules.py` proves every `.sops.yaml` +rule against the stack directories, so a `phoenix` rule fails CI until a +`stacks/phoenix` exists. The toolchain issue that consumes the token defines +the encrypted file; this runbook does not guess its shape. The SSH key is the +one the toolchain will inject into every guest it builds. It is generated +here and goes nowhere else. The checkout holds **no age key**: `make render` +fails here by design, and this host converges nothing — ADR-0021 owns what +happens to a machine after it exists. + +Prove the door and the token together, from `phoenix`: + +```bash +set -a; . ~/.config/proxmox/phoenix.env; set +a +curl -sk -H "Authorization: PVEAPIToken=${PROXMOX_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" \ + "${PROXMOX_URL}/nodes/saruman/qemu" | python3 -m json.tool | grep '"name"' +``` + +`alexander` and `phoenix` at minimum. A connection timeout is the door — the +`host.fw` line is missing or the firewall was not reloaded (`pve-firewall +compile` shows what it thinks the rules are). A `401` is the token. An empty +list with a `200` is the ACL: the user can reach the node and see no guests. + +## 5. Open the lab's doors, if they are still shut + +This guest's Alloy pushes to the lab's Prometheus and Loki. The two `ports:` +blocks in `stacks/lab/compose.yaml` and the two port lines in +`stacks/lab/.env.example` landed commented, waiting for a client with no +scrape alternative — [`build-the-soc-guest.md`](build-the-soc-guest.md) §7 is +the step that opens them for `odin`, and whichever of the two guests is built +first does it. Check before editing anything, from `phoenix`: + +```bash +nc -zv -w 3 10.0.30.40 9090 3100 +``` + +Both open: skip to §6. Either refused: do §7 of the SOC runbook now, **on +`alexander`**, and commit and push from there, for the reason it gives. +Nothing on `morpheus` is involved either way — the path is intra-segment. + +Being scraped by `alexander` instead, the way ADR-0029's six Windows machines +are, would have needed no `ports:` change at all. It loses because a +deployment host's value in an incident is its logs — what it did, to which +guest, when — and a scrape carries nothing to Loki. That is why this step +exists rather than a `scrape_configs` entry. + +## 6. The Alloy agent, from the Mac + +The script's native path needs root or passwordless sudo on the target +(`deploy-agent.sh` refuses otherwise, and says so). On `phoenix`, for the +installer's first user: + +```bash +echo "$USER ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/deploy >/dev/null +sudo chmod 440 /etc/sudoers.d/deploy +``` + +That is a trade, and it is the same one every host `deploy-agent.sh` ships to +natively makes: a redeploy after an image bump is one command from a laptop +rather than an evening, and the account it runs as already holds a key that +can build every guest on the segment. + +Then from a checkout **on the Mac** — Hicks reaches this segment and +`prometheus` does not, and the script's header says it is safe from macOS: + +```bash +./scripts/deploy-agent.sh --runtime native --monitoring-host 10.0.30.40 @10.0.30.70 +``` + +Two flags, both load-bearing: + +- **`--monitoring-host 10.0.30.40`** is the lab's stack on `alexander`. The + flag defaults to `10.0.99.20`, which from this segment is the wrong stack + *and* an unreachable address, and omitting it fails in a way that reads as + a firewall problem. This is the first use of the flag in the estate. +- **`--runtime native`** is a statement, not an autodetect: there is no Docker + here to detect, and if one day there is, this line still says which was + meant. + +The script's own arrival check queries `10.0.30.40` afterwards. If it warns +*"is the pass to 10.0.30.40:9090 in place?"* the answer is that there is no +pass to place — go back to §5, because that message was written for hosts on +another VLAN and this one is not. + +## 7. Verify — including the thing that fails quietly + +```bash +make validate +``` + +From the Mac's checkout; it does not need the guest. Then the things it +cannot see, in the **lab's** Grafana at `https://10.0.30.40:3000/` — not the +estate's, which this host never reaches. Explore, Prometheus datasource: + +```promql +up{instance="phoenix"} +``` + +Two jobs, `phoenix-metrics` and `phoenix-alloy`, both `1`. Three on a Docker +host is the count `alexander` shows; two is right here, because there is no +cAdvisor to scrape and nothing for it to report on. Loki datasource: + +```logql +count by (job) (count_over_time({host="phoenix"} [15m])) +``` + +Three jobs at minimum: `/var/log/auth.log`, `/var/log/syslog` and +`/var/log/journal`. **Two is the failure this section exists to catch**, not a +quiet host — a journald-only install, and `sudo apt-get install -y rsyslog` +on `phoenix` fixes it. Check it on the host too: + +```bash +ls -l /var/log/auth.log /var/log/syslog +``` + +And the door, once more, from `phoenix` — §4's `curl` — because §6 may have +rebooted nothing but §5 may have restarted the lab stack, and an API that +answered an hour ago is the one thing in this runbook with a firewall between +it and this host. + +## 8. Write it down + +The guest is not built until the documents say so, and `make check-docs` +walks you through the first three: + +- `docs/network.md` — a row for `phoenix` in the ImaginationLAN table, and the + planned-guest note in that section becomes a description. Say in it that + this is the one address on the segment with a path to `8006`, because + `firewall-claims.yaml` cannot say it for you. +- `docs/architecture.md` — drop `**Not built yet**` from the `phoenix` row. + The marker is load-bearing: while it stands, `check_docs.py` requires the + host to be **absent** from `network.md`, so the row and the marker cannot + coexist. +- `docs/hardware.md` — removing the marker raises the Alloy agent count, and + the sentence there that states it fails until it says the new number. +- `docs/network.md`'s WAN note, which says the jumpbox does not exist — it + does now, and the endpoint question is the part that is still open. +- `docs/observability.md` — the sentence that says what pushes to `alexander` + gains a host, and the one that says the lab's ports are shut stops being + true, if §5 was yours. +- [#436](https://github.com/Gerrrt/HomeLab/issues/436) closes on the day this + section is done — not on the day the guest comes up. diff --git a/docs/runbooks/build-the-soc-guest.md b/docs/runbooks/build-the-soc-guest.md index e86ae2cd..11861de9 100644 --- a/docs/runbooks/build-the-soc-guest.md +++ b/docs/runbooks/build-the-soc-guest.md @@ -193,7 +193,9 @@ stays in that gitignored directory and nowhere else. and until now nothing off `alexander` did — the two `ports:` blocks in `stacks/lab/compose.yaml` and the two port lines in `stacks/lab/.env.example` have been commented since the day that stack landed, waiting for a client with -no scrape alternative. This is that client (ADR-0030). +no scrape alternative. This is that client (ADR-0030) — unless `phoenix` was +built first, in which case [`build-the-jumpbox.md`](build-the-jumpbox.md) §5 +already did this and the four are open; check before editing. Uncomment all four — `PROMETHEUS_PORT` and `LOKI_PORT` in `.env.example`, and the `ports:` block under `prometheus` and under `loki` in `compose.yaml` — and diff --git a/scripts/gen-certs.sh b/scripts/gen-certs.sh index ef542dff..dc7f0fb1 100755 --- a/scripts/gen-certs.sh +++ b/scripts/gen-certs.sh @@ -138,6 +138,12 @@ that holds it. Pass --force only if that is what you mean." # here — which is why the sensitive tier's step-ca has a root of its own # rather than sitting beneath this one (ADR-0037, measured before deciding). # Lifting it means re-minting this root and re-trusting it everywhere. + # + # And the key stays on the monitoring host. ADR-0043 decided it does not + # follow the deployment toolchain to `phoenix` on VLAN 30: that host holds + # credentials for every other host, and the key every other host trusts + # must not sit beside them. Nothing below enforces that — the path is + # relative to whichever checkout runs this — so this comment is the guard. openssl req -x509 -newkey rsa:4096 -sha256 -nodes \ -keyout "${CA_KEY}" -out "${CA_CRT}" -days "${CA_DAYS}" \ -subj "${CA_SUBJECT}" \ diff --git a/stacks/lab/.env.example b/stacks/lab/.env.example index efb649d1..a1720cda 100644 --- a/stacks/lab/.env.example +++ b/stacks/lab/.env.example @@ -15,10 +15,11 @@ BIND_ADDR=0.0.0.0 # The one published port: Grafana, opened from Hicks over the 50 -> 30 rule # that already exists (ADR-0007). Nothing else in this stack is published, and # #265 did not change that — ADR-0029 scrapes the domain rather than letting it -# push. `odin` (stacks/soc, ADR-0030) is the first thing with no scrape -# alternative — its Alloy pushes here — and build-the-soc-guest.md §7 -# uncomments the two ports below, and the `ports:` blocks in compose.yaml, on -# the day it comes up. +# push. `odin` (stacks/soc, ADR-0030) and `phoenix` (ADR-0043) are the things +# with no scrape alternative — their Alloy pushes here — and whichever is +# built first uncomments the two ports below, and the `ports:` blocks in +# compose.yaml, on the day it comes up: build-the-soc-guest.md §7 or +# build-the-jumpbox.md §5. GRAFANA_PORT=3000 # Alloy's debug UI, pinned to 127.0.0.1 in compose.yaml. diff --git a/stacks/lab/compose.yaml b/stacks/lab/compose.yaml index 55d8aee2..eabf896c 100644 --- a/stacks/lab/compose.yaml +++ b/stacks/lab/compose.yaml @@ -134,11 +134,13 @@ services: # direction the connection travels, because that is the control. # # ADR-0030 made that argument, and stacks/soc is the client: `odin`'s Alloy - # pushes here, and there is no scrape alternative for it. The block still - # lands COMMENTED, and build-the-soc-guest.md §7 uncomments it — here and - # on loki below — in the same change that brings odin up. A receiver - # published with no client yet is exactly the ADR-0012 case, on the - # segment where it matters most. + # pushes here, and there is no scrape alternative for it. ADR-0043 adds a + # second, `phoenix`, the deployment host, whose worth in an incident is its + # logs. The block still lands COMMENTED, and whichever guest is built first + # uncomments it — build-the-soc-guest.md §7 or build-the-jumpbox.md §5, + # here and on loki below — in the same change that brings that guest up. A + # receiver published with no client yet is exactly the ADR-0012 case, on + # the segment where it matters most. # ports: # - "${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090" expose: @@ -178,8 +180,9 @@ services: # answered the rest the other way round: Wazuh's alerts stay in its own # indexer and are never copied here, and what odin's Alloy ships to this # Loki is that GUEST's own logs — journal, auth.log, containers — the way - # every other host's agent does. That is the client this opens for, on the - # same day and by the same runbook step as prometheus above: + # every other host's agent does — as `phoenix`'s will (ADR-0043). Those are + # the clients this opens for, on the same day and by the same runbook step + # as prometheus above: # ports: # - "${BIND_ADDR:-0.0.0.0}:${LOKI_PORT:-3100}:3100" expose: diff --git a/stacks/observability/loki/rules/security.rules.yaml b/stacks/observability/loki/rules/security.rules.yaml index f0374607..edaa7426 100644 --- a/stacks/observability/loki/rules/security.rules.yaml +++ b/stacks/observability/loki/rules/security.rules.yaml @@ -352,7 +352,7 @@ groups: # 10.7.7.0/24 is the untagged switch-management LAN, in both aliases # since 2026-09-06; a logged block above the tripwire catches it first. # - # 172.31.0.0/24 is the WireGuard peer subnet (ADR-0041). Routed, not + # 172.31.0.0/24 is the WireGuard peer subnet (ADR-0042). Routed, not # masqueraded, so a peer's own address is what arrives on igc0.30 and # what a filterlog line carries — which is the point of routing rather # than translating, and the reason this source class has two members @@ -382,7 +382,7 @@ groups: tooling has reached the house. Either a block rule was removed or reordered, or the tripwire's alias gained the wrong subnet. A 10.0.30.x source is the lab itself; a 172.31.x source is a remote - peer, which also means the ADR-0041 blocks are missing or ordered + peer, which also means the ADR-0042 blocks are missing or ordered below the catch-all. Treat as a live segmentation failure. - alert: IoTAttemptedLateralMovement