Rehearse the firewall restore on trinity

[Gerrrt]

What is left: the rehearsal

The two sections this issue opened with are done or withdrawn. The firewall backup's off-host copy to oracle exists and was verified (2026-09-03). "Buy a spare ProDesk" was withdrawn by ADR-0034 — there is one ProDesk, trinity, the tier's host and the firewall's spare hardware — and it was bought and landed 2026-09-14 (#492). ADR-0034 and restore-the-firewall.md deliberately keep the rehearsal here and the build in #404: step 6 of the runbook hands the box to #404 afterwards.

restore-the-firewall.md: "Nobody has done this yet." The hypothesis sentence is still in the runbook, unresolved.

What it waits on

The I226 card on its M.2 adapter and the pfSense installer stick, both in transit (hardware.md). trinity's SSD is M.2 and the second M.2 slot is free (#474), so the card has a home.

Order of work

• Before 2026-10-08: read spec, serial, product number and NIC count off the case — the return window, and the one step not blocked on a package (hardware.md; restore-the-firewall.md calls it "the only entry on this list with a deadline"). Shared with Build the sensitive-tier host on the ProDesk (ADR-0034) #404 step 0.
• Card in, stick in, restore-the-firewall.md §1–§5 on the bench.
• Write the result where the runbook says — the hypothesis sentence replaced by what happened, in the runbook and in roadmap.md under this issue.
• Card out, box to Build the sensitive-tier host on the ProDesk (ADR-0034) #404.

Done when

A restore from the oracle copy boots on trinity with the I226 card and serves the estate's VLANs on the bench, and the runbook no longer says nobody has done it.

---

Body as filed, superseded 2026-09-19

make backup-firewall exports and encrypts morpheus's config, and
restore-the-firewall.md
documents the restore. Two things are still missing, and they are the two that
matter.

The backup is on the machine it protects

backup-firewall.sh:125-126:

This is on the same host as everything else it protects. Copy it to the backup
target and offsite.

Nothing does that copy. A backup on the same shelf as the thing it protects is
not a backup.

The runbook has never been executed

restore-the-firewall.md:168:

Until that has been done once, this runbook is a hypothesis.

Doing it needs a spare, and the spare should be the same ProDesk model:
pfSense stores interface assignments by device name, so identical hardware
restores straight through and anything else drops you into the console
interface-assignment dialogue — at exactly the moment you least want to be
answering questions.

The two halves are one purchase apart. Buying the spare unblocks the rehearsal,
and the rehearsal is what turns the runbook into something you can rely on at
1am.

Where the spare goes — #110

It racks on the 1U shelf from
#110, beside the switch, and is
left powered off.

Powered off is the point, not an omission. A spare that is plugged in and on the
network is exposed to whatever took the primary — and the primary is the device
that terminates every VLAN, so "whatever took it" is the case this spare exists
for.

It cannot go beside morpheus: that sits at U5 in a purpose-built mount holding
exactly one ProDesk Mini, which was checked at the rack rather than assumed. The
1U shelf has room for the switch and a second small box; the U5 bracket by design
does not. So the shelf is not only where the spare can live, it is the only
place in the rack it does.

Order of operations, if both are bought together: the shelf goes in first (U4 is
the one free slot), then the spare has somewhere to land.

The other half is still unowned

Racking the spare does not address the first problem above. The encrypted export
still lands on prometheus and nothing copies it anywhere — not to the spare,
not offsite. That copy needs a destination decided;
#94 ("decide what oracle is
for") and #77 ("nothing schedules
any backup") are the two it most likely wants solving with.

Tracked in docs/roadmap.md; filed as an issue so it has a place to be discussed and closed.

[Gerrrt]

Status check, because this issue's body no longer matches the code.

The copy exists. It landed 2026-09-03 in 896da41 and 41444c2: make backup-firewall copies every export to oracle as the last step of the same run and exits non-zero if it cannot, so ScheduledJobFailed doubles as "the config has stopped leaving this host". oracle holds ciphertext only.

It did not work as shipped. The default was robo@10.0.99.30; the login there is atropos, and there is no /etc/default/homelab-timers on this host to override it. Every nightly run would have failed on Permission denied. It stayed hidden for a day only because the checkout the timer runs from was behind the commit that added the copy — so the first real run was going to be the first proof, and it would have been a failure. Fixed in #250, along with the five robo@10.0.99.30 references in restore-the-firewall.md (including the ssh-copy-id a reader would have run against the wrong account).

Retention came with it. Both sides were keeping every export ever taken, nightly, forever. FW_KEEP (default 30) now bounds each side. Not a disk-space emergency — an export is ~220 KB against 67 GB free — but the sibling backups/volumes has been bounded since #64 and this was not.

Verified end to end against the real morpheus and oracle, including the remote prune on a scratch directory there, and through the unit's own ExecStart: exit 0, homelab_job_last_exit_code 0.

What is still open here

• The spare ProDesk, same model, racked on the Rack the shelf switch in U4 — buy it with the UPS pack #110 shelf and left powered off. Blocked on the purchase.
• The rehearsal. restore-the-firewall.md now carries the bench procedure and what to record, but it has not been executed, so the runbook is still a hypothesis.
• Offsite. Unchanged and still unowned: prometheus and oracle share a shelf, a mains circuit and a roof, so a fire takes both. Off-host is not offsite, and this needs a destination decided — Decide what oracle is for #94 and Nothing schedules any backup, digest check or key verification #77 territory.

The volume sets make backup writes also still sit on the host they protect.

[Gerrrt]

The first half is done. The title now overstates what is outstanding, so re-scoping rather than closing.

"The backup is on the machine it protects" — closed

896da41 feat(backup): copy every firewall export to oracle, and fail the job if it cannot (#92) (2026-09-03) implemented it, and it is running:

$ ls -1 ~/backups/firewall/          # on oracle, 10.0.99.30
...
config-20260904T044214Z.sops.yaml
config-20260904T140058Z.sops.yaml
count=13

Thirteen encrypted exports off-host, newest from today. The design answers the objection the description raised — the copy is part of make backup-firewall rather than a second target, so a run whose copy fails exits non-zero even though the local file was written, and FW_KEEP bounds retention on both sides so a nightly job cannot grow without bound. The age private key is not copied there and must not be, which is what makes oracle an acceptable destination.

This is a second location on the same shelf, not offsite. That is a real improvement over "the same host" and it is what the issue asked for, but it does not survive the shelf.

"The runbook has never been executed" — still open, and now the whole issue

docs/runbooks/restore-the-firewall.md still ends on:

Until that has been done once, this runbook is a hypothesis.

Nothing here changes that, and nothing can until the spare exists. The two halves were "one purchase apart" when this was filed; they are still one purchase apart, and it is now the only thing this issue is about:

• Buy the same ProDesk model — pfSense stores interface assignments by device name, so identical hardware restores straight through and anything else drops you into the console assignment dialogue at 1am.
• Rack it on the Rack the shelf switch in U4 — buy it with the UPS pack #110 shelf, left powered off, for the reason the description gives: a spare that is on the network is exposed to whatever took the primary.
• Rehearse the restore, which is what turns the runbook into something you can rely on.

Suggest retitling to something like "Buy a spare ProDesk and rehearse the firewall restore" so the open part is what the title describes.

Same shape as the residual noted on #120: the mechanism is built and verifiable, the drill is what tells you it works, and only one of those two has happened.

[Gerrrt]

Retitled from "Get the firewall backup off prometheus, and buy a spare ProDesk to rehearse the restore" now that the first half is done and verified.

The description above still opens on the completed half, which is worth leaving as the record of why this was filed — but for anyone landing here, the off-host copy is implemented, running nightly, and bounded by FW_KEEP. Only the purchase and the drill remain.

[Gerrrt]

The purchase happened: spare ordered 2026-09-08, shipping UPS Ground.

Model match, checked against morpheus

	morpheus (hardware.md)	The spare
Model	HP ProDesk 600 G4 Mini	HP ProDesk 600 G4 Micro — same machine, HP's two names for it
CPU	i5-8500T	i5-8500T
RAM	32 GB	32 GB
Disk	1 TB SSD	512 GB SSD
Ships with	—	Windows 11 Pro, which the pfSense installer wipes

Same model is the property this issue asked for: pfSense stores interface
assignments by device name, so the onboard NIC comes up under the same name and
the restore goes straight through. The smaller SSD changes nothing — a config
export is about 220 KB and pfSense itself is a few GB.

Two things it does not come with

• A USB NIC. morpheus reaches the switch-management LAN over one, and
  restore-the-firewall.md §3 says the spare needs one too — ideally the same
  chipset, so that interface also keeps its device name. Worth buying before
  the box arrives, not after the rehearsal stalls on it.
• The installer. hardware.md lists the pfSense USB stick as in transit;
  it belongs in the rack. The rehearsal wants the same pfSense version the
  newest export names (make backup-firewall ARGS=--verify-only prints it).

When it arrives, in this order

1. Bench, not rack — the runbook's Rehearse the restore on the spare
   section, exactly as written: laptop cabled to the onboard NIC, nothing else
   connected, the copy from oracle rather than from prometheus, and the
   answers written back into §3 (what the fresh install assigns to the onboard
   NIC, whether the USB NIC comes back under the same name, how long it takes).
2. Shred the plaintext, power it off, and rack it on the U4 shelf — Rack the shelf switch in U4 — buy it with the UPS pack #110 is
   closed, so the shelf is there — powered off, for the reason the
   description gives.
3. hardware.md gets its row then, and the runbook loses its "this runbook is
   a hypothesis" line. Until step 1 has been done once, it keeps it.

Roadmap and README updated to say "rehearse" rather than "buy" in
#401. Nothing else in the
repository changes on an order; the hardware page describes what is racked.

🤖 Generated with Claude Code

[Gerrrt]

Correction to my comment above, and a re-scope of this issue.

The box is not a cold spare. I recorded the ProDesk bought on 2026-09-08 as
the spare because it matched this issue's spec and morpheus exactly, without
asking what it was bought for. It was bought to be the sensitive tier's host
as well — and the tier's host, which ADR-0008 lists as a separate purchase,
had no issue and no place in the README's shopping sentence after #102 split,
so "buy the spare ProDesk" was the only instruction anyone ever saw.

ADR-0034 (#405) decides it:

1. The firewall restore is rehearsed on this box first, on the bench, before
   it holds anything — the half of this issue that turns the runbook from a
   hypothesis into something rehearsed. That half stays here.
2. Then it is wiped and built as the sensitive tier's host under
   #404.
3. In a disaster it is the spare hardware: same model, so the restore goes
   straight through, at the cost of the tier being down until a replacement
   ProDesk arrives. ADR-0023 already accepts that the tier can be down.
4. The powered-off shelf spare is deferred, with a stated reopen condition,
   rather than bought twice.

What this issue still owns: the rehearsal, and the date it was done
written into restore-the-firewall.md. The "racked on the #110 shelf, powered
off" half is withdrawn by the ADR. The USB NIC and the installer stick are
still needed for the rehearsal. #401 is closed unmerged; #405 carries the
roadmap and README text instead.

🤖 Generated with Claude Code

[Gerrrt]

Preparing the rehearsal so it does not stall the day the box arrives, and a correction to my 2026-09-08 comment above, which said the spare needs a USB NIC "ideally the same chipset" as morpheus's.

morpheus has no USB NIC. Read off the box on 2026-09-09 rather than from the documents:

Device	Hardware	Carries
em0	onboard Intel I219-LM	WAN
igc0	Intel I226-V 2.5 GbE card on an M.2 B+M-key adapter, in the G4's second M.2 slot	untagged switch-management LAN, and all six VLANs

pciconf shows the I226 on PCIe root port 21; usbconfig lists only the Wi-Fi module's Bluetooth half. The label came from network.md's footnote, which called the M.2 card a "USB NIC adapter" — the link itself sells the M.2 card. It then reached the runbook, my comment here, the newest comment on #404 and the roadmap's shopping list.

It matters because of the property this issue is about: a USB adapter comes up as ure0 or axge0, a Realtek card as re0, and either way igc0 does not exist at boot, so pfSense stops at the console to ask which interface is which. The card is the part that makes the restore go straight through; the same-model box only has to have a slot for it. Any I225/I226 (igc driver) card will do.

The version the verify prints is not the release. make backup-firewall prints config version 24.6. That is pfSense's config schema, not the release, and no installer is labelled with it. The release is /etc/version on the box: pfSense CE 2.9.0-RELEASE, build 20260817-1836. 2.9.0's own schema is 24.6, so it is the floor for the current export; worth checking what is on the stick in transit.

#425 carries both corrections into the runbook, hardware.md (which now records the release), network.md and the shopping list, plus a Before the box arrives list in the runbook and a WAN MAC caveat — nothing in the config pins one, so the spare presents its own to the ISP. Also checked while here: the nightly copy is healthy, 29 exports on oracle, the newest byte-identical, last run exit 0.

Still blocked on the box for the rehearsal itself; nothing else changes here.

🤖 Generated with Claude Code