Build phoenix, the deployment host on ImaginationLAN (ADR-0043)

[Gerrrt]

Decided; now build it

The half of this issue that was a question is answered: ADR-0043 (#505, 2026-09-18) keeps the CA on prometheus and builds phoenix as the deployment host. The body as filed assumed step-ca sits beneath the estate root; it does not — the root is pathlen:0 (ADR-0037), which is why the tier has its own. The CA-key custody gap the ADR found is #496.

docs/runbooks/build-the-jumpbox.md is the procedure: phoenix, 10.0.30.70, VMID 170, a VM not an LXC, 2 vCPU / 4 GiB / 32 GB, no stack, no Docker, no age key, no CA — with a CAUTION that certificates/ca-key.pem does not come here. One cost the issue did not count and the ADR did: the Proxmox firewall on Saruman admits 8006 from Hicks only, so this widens ADR-0014's rule by one address on one port.

Why it moved

It is the root of the automation chain — #440, #445 and #448 hang on it, and only #449–#451 hang on #414 — and it has no predecessor: it needs Saruman, which exists. A milestone described as a strict chain should not hold the one item in it that is not blocked. Moved to the Saruman milestone beside #527.

Done when

phoenix is in network.md and hardware.md, architecture.md no longer says Not built yet, check_docs.py's absence marker is removed, and a make deploy from it has landed on one host.

---

Body as filed, superseded 2026-09-19

This estate has 34 ADRs, 20 runbooks, 95 alert rules and a convergence loop
that refuses to deploy without a pinned GPG signature — and it builds virtual
machines by clicking. The Packer, OpenTofu and Ansible issues that follow this
one close that gap. This is their prerequisite: there is nowhere for the
toolchain to live.

Today Proxmox is driven from a Hicks workstation by hand. Packer, OpenTofu and
Ansible want a host that holds an API token, an SSH key and a checkout, and that
host is a pet — it is the thing that rebuilds everything else.

What is decided by placement

VLAN 30, not 99. It drives the lab, and ADR-0031
already grants Hicks all of ImaginationLAN, so it needs no new rule. Putting it
on Winterfell would widen the management segment for exactly the reason
ADR-0008 argues against —
"Winterfell becomes busier and less special."

A dedicated Proxmox user and API token, not root@pam, per the same instinct
that gave the lab guest its own age recipient under
ADR-0020:
the token that can create VMs should not be the token that can do everything.

The open question this issue carries

The estate's CA currently lives on prometheus. scripts/gen-certs.sh
deliberately leaves the CA key without a passphrase — the trade is written down,
that interactive issuance leads to certificates nobody dares reissue — and
SECURITY.md records that host's disk and swap as unencrypted. So the root of
trust for the estate sits on the monitoring box, protected by file permissions.

Moving it to the jumpbox is not obviously right; it trades one unencrypted disk
for another. But it is worth deciding rather than inheriting. stacks/sensitive's
step-ca is already an intermediate beneath that root (#130),
so the tree is shaped for it either way. Decide here, record it, and do not
move the key as a side effect of this build.

Order of work

1. Guest on Saruman, VLAN 30, static below .100 with a Kea reservation.
2. Proxmox user and API token, privilege separation off, credential into
   secrets/.
3. Alloy agent via scripts/deploy-agent.sh — it is a lab host, so it pushes to
   the lab's stack on alexander, never to 10.0.99.20
   (ADR-0007).
4. Decide the CA question. Record it.

Purchases this needs

None.

[Gerrrt]

Build checklist, 2026-09-19

The repo half is split in two because scripts/check_docs.py will not let the **Not built yet** marker and a network.md row coexist, and the lab's ports must be open before the agent first pushes:

• feat(lab): prometheus and loki publish 9090 and 3100 on alexander for phoenix's agent (#436) #543 opens the lab's doors (stacks/lab publishes 9090 and 3100). Merged 2026-09-19; apply it on alexander in §5.
• docs: phoenix is built at 10.0.30.70 on Saruman, the lab has its first off-host client, and Saruman admits it to 8006 (#436) #544 (draft) carries §8's docs (network.md row, marker gone, "five Alloy agents", Saruman's guest count, the WAN note, observability.md, the roadmap, a §0 status block in the runbook). It merges only after §7 is green, and it is the one that closes this issue.

Everything below is build-the-jumpbox.md; the runbook has the commands. What each step needs reported back is so the docs can be exact.

1. §1–§2, on Saruman — qm create 170 … (ISO name from pvesm list local --content iso), Ubuntu Server with hostname phoenix, 10.0.30.70/24, gateway and DNS 10.0.30.1, OpenSSH, no Docker. Report: build date, ISO filename, the OUI from qm config 170 | grep net0 (the first three octets only; the row masks the rest).
2. §3, on morpheus — the Kea reservation for that MAC → 10.0.30.70, under Services → DHCP Server → ImaginationLAN.
3. §4, on Saruman and phoenix — the pveum block, then the one host.fw line beneath ADR-0014's three (read the CAUTION first: if /etc/pve/nodes/saruman/host.fw does not exist, the firewall was never on). On phoenix: ~/.config/proxmox/phoenix.env at mode 600, ssh-keygen -t ed25519 -C phoenix, git clone the repo. Run the curl proof; report which of timeout / 401 / empty 200 / a list it gives. The token secret stays on that host and never enters a session transcript.
4. §5, on alexander — feat(lab): prometheus and loki publish 9090 and 3100 on alexander for phoenix's agent (#436) #543 is merged: git pull && make up STACK=lab, then ss -ltn '( sport = :9090 or sport = :3100 )'. Report both lines.
5. §6, from the Mac — the sudoers line on phoenix, then ./scripts/deploy-agent.sh --runtime native --monitoring-host 10.0.30.40 <user>@10.0.30.70. Report the script's arrival check.
6. §7 — make validate in phoenix's checkout; in the lab Grafana, up{instance="phoenix"} (expect phoenix-metrics and phoenix-alloy, both 1) and count by (job) (count_over_time({host="phoenix"} [15m])) (three jobs or more; two means rsyslog is missing). Report the three results.

Per the issue's "Done when", the runbook's §6 is the deploy proof: the agent lands on phoenix by make deploy-agent from the Mac, and phoenix's own checkout passes make validate. No SSH trust from phoenix to any other host is added.

VLAN 30 is unreachable from the monitoring host and the lab Prometheus refuses morpheus, so none of §1–§7 can be driven or verified from a session here; the reports above are what turns the draft PR into the merge that closes this.