From b6a965c80a400793ba13ee7d5c411dfac1852555 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:02:24 +0000 Subject: [PATCH] =?UTF-8?q?docs(adr):=20ADR-0041=20=E2=80=94=20the=20CRS32?= =?UTF-8?q?6=20runs=20RouterOS,=20keeps=20neo,=20and=20serves=20TLS=20(#44?= =?UTF-8?q?4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #444 asked for a switch that can do TLS. The purchase already happened — a used MikroTik CRS326-24G-2S+RM, 2026-09-13, recorded by #463 — and the issue's own plan says nothing changes until a cabling window. This is the decision half of that plan, written now rather than at the rack. ADR-0018's note said the successor ADR would be written when the switch was racked. That is the wrong time: `neo` carries every VLAN, so the swap is a house-wide outage, and an outage is not when to weigh a dual-boot device's two operating systems or to decide whether a management LAN survives. The note is amended to say so. Nothing else about it changes, and nothing in the estate does either — `neo` is still the MokerLink, still on plain HTTP. What ADR-0041 decides: RouterOS over SwOS, because SwOS serves HTTP only and speaks v1/v2c only, which reproduces the limit the purchase exists to escape on hardware that is capable of better. The name `neo` and `10.7.7.2` are inherited, and the switch LAN stays — for a reason it did not originally have. network.md explains that LAN as a workaround for a UI that would not bind to a tagged interface; that is the MokerLink's limit and it leaves with it. It survives because an untagged point-to-point link to morpheus is the way back in after a bad bridge VLAN-filtering commit, on the one device that carries every VLAN. `www-ssl` serves a leaf from the estate's CA, not the tier's — a switch cannot ask for an ACME renewal, and blackbox already verifies estate-CA leaves by ca_file, so the switch-ui probe can move to https and actually check the certificate. An SNMPv3 authPriv user, v2c off. Port mirroring stays disabled. It was in #444's selection criteria, and ADR-0006 decided the sensor belongs on morpheus with the switch's mirroring off and available on demand; buying a device that can mirror was never a decision to mirror. swap-the-switch.md is the procedure, in three phases, because only the middle one needs the house offline. Two things in it are not in #444's list. The first is a firewall rule: network.md blocks the switch LAN apart from `10.7.7.2:80`, so disabling plain www before widening that pass to 443 leaves a correctly configured switch that Hicks is not allowed to reach — widen, prove, then close 80. The second is that the MokerLink's SNMP community leaked into a session transcript on 2026-08-20 and SECURITY.md records it as deliberately not rotated, because rotating it meant repeating the residual. That reasoning expires with the hardware only if the value does too, so commissioning derives nothing from it. Also: no per-port VLAN map exists in this repository, and the runbook makes capturing one off the MokerLink the first bench step — 26 cables and no map is how the house comes back wrong. And the port census is the one thing here that could still imply a purchase: 26 copper ports replaced by 24 plus two SFP+, so if more than 24 are populated, an SFP+ copper module edits *Everything still to buy* in the same commit. The buy table is unchanged by this commit; #463 already took the switch's row off it. README's ADR count goes 40 to 41 and its runbook count 25 to 26, both required by check_docs.py; the tree comment at line 210 was separately stale at 39 and is corrected. roadmap.md gains a *Decided but not built* entry, which is what an accepted ADR with no work behind it is. #444 and #84 both stay open until the window. Verified: python3 scripts/check_docs.py green at 10 assertions, ./scripts/lint.sh --require-all green. Refs #444 #84 #85 Co-Authored-By: Claude Opus 5 --- README.md | 6 +- ...e-switch-and-leave-its-ui-on-plain-http.md | 30 +- ...outeros-and-keep-neo-and-its-switch-lan.md | 196 +++++++++++++ docs/hardware.md | 12 +- docs/roadmap.md | 10 + docs/runbooks/swap-the-switch.md | 262 ++++++++++++++++++ 6 files changed, 504 insertions(+), 12 deletions(-) create mode 100644 docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md create mode 100644 docs/runbooks/swap-the-switch.md diff --git a/README.md b/README.md index 31c85056..cfb49517 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,8 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 40 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 25 +- **Documented decisions and runbooks.** 41 ADRs covering what was chosen + and what was rejected — including the costs accepted knowingly; 26 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. @@ -207,7 +207,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 39 architecture decision records +│ ├── adr/ # 41 architecture decision records │ └── runbooks/ # successor handover (start here), deploy, converge, │ # add device, rotate creds, certs, key backup, │ # purge, restore the firewall, restore the stack, diff --git a/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md b/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md index d9e636f4..2f1c1c93 100644 --- a/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md +++ b/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md @@ -3,13 +3,29 @@ **Status:** Accepted · 2026-09 > [!NOTE] -> The replacement the last consequence below asks for was bought on -> 2026-09-13: a MikroTik CRS326-24G-2S+RM, chosen for the TLS management -> interface ([`hardware.md`](../hardware.md), -> [#444](https://github.com/Gerrrt/HomeLab/issues/444)). Nothing here changes -> until it is racked — `neo` is still the switch, still on plain HTTP. The ADR -> that names the new switch and decides its OS and management posture is -> written then. The text here is left as written, per ADR-0001. +> **One clause of this ADR is superseded. The text below is left as written, +> per ADR-0001.** +> +> **The purchase.** The replacement the last consequence below asks for was +> bought on 2026-09-13: a MikroTik CRS326-24G-2S+RM, chosen for the TLS +> management interface ([`hardware.md`](../hardware.md), +> [#444](https://github.com/Gerrrt/HomeLab/issues/444)). +> +> **The UI.** Decision 2 — that the UI stays plain HTTP, and that #97's +> certificate goal is closed as not achievable — is superseded by +> [ADR-0041](0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md), +> which runs the CRS326 on RouterOS and serves the UI from `www-ssl` with a +> leaf from the estate's CA. Decisions 1 and 3 are unchanged, and so is the +> reasoning behind them: the switch keeps the name `neo` and the address +> `10.7.7.2`, and both stay written down because the name depends on +> `morpheus`. The rejection of renumbering onto `10.0.99.x` also stands. +> +> This note previously said that ADR would be written when the switch was +> racked. It was written before instead: a house-wide outage is not when to +> choose between a dual-boot device's two operating systems. **Nothing here +> changes until it is racked** — `neo` is still the MokerLink, still on plain +> HTTP, until the window in +> [`swap-the-switch.md`](../runbooks/swap-the-switch.md). ## Context diff --git a/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md b/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md new file mode 100644 index 00000000..594d9790 --- /dev/null +++ b/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md @@ -0,0 +1,196 @@ +# ADR-0041: Run the CRS326 on RouterOS, and keep neo and its switch LAN + +**Status:** Accepted · 2026-09 · supersedes the plain-HTTP clause of +[ADR-0018](0018-name-the-switch-and-leave-its-ui-on-plain-http.md) + +## Context + +[ADR-0018](0018-name-the-switch-and-leave-its-ui-on-plain-http.md) closed #97's +certificate goal as *not achievable on this hardware* and said the only thing +that would reopen it was replacing the switch. A replacement was bought on +2026-09-13 — a used MikroTik CRS326-24G-2S+RM, recorded in +[`hardware.md`](../hardware.md) — and +[#444](https://github.com/Gerrrt/HomeLab/issues/444) is where it was decided. + +It was bought for **one property, a TLS management interface**. The issue +originally named three residuals one purchase would close; +[ADR-0036](0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) +had already removed the middle one, because `neo`'s agent answers SNMPv3 on the +wire and the switch was never what blocked +[#85](https://github.com/Gerrrt/HomeLab/issues/85). What stands is the plain +HTTP — no TLS listener, no certificate import, checked against the live device +on 2026-09-04 — and with it the fact that the switch admin credential, which is +read-write, crosses the wire in clear *through the switch it protects*. +[#84](https://github.com/Gerrrt/HomeLab/issues/84)'s un-deletable community +rides along, because it leaves with the firmware that will not persist its +deletion. + +**Why this is written before the switch is racked, and not during.** ADR-0018's +own note said the successor ADR would be written then. That is the wrong time. +`neo` carries every VLAN, so the swap is a house-wide outage sharing a rack +visit, and an outage is not when to weigh a dual-boot device's two operating +systems or to decide whether a management LAN survives. This record decides the +half that cannot be changed later; what the device turns out to be — serial, +management MAC, the OS and version it arrives on, the factory reset — is learned +at the bench and written to `hardware.md`, which is the split +[ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md) already +uses for a purchase. Nothing in the estate changes until the cabling window; +[`swap-the-switch.md`](../runbooks/swap-the-switch.md) is the procedure. + +### The operating system is not really a choice + +The CRS326 dual-boots RouterOS and SwOS, and SwOS is the tempting one: a single +web page, no routing stack, nothing to misconfigure on a device that should only +switch. It cannot do the job. SwOS serves its web UI over plain HTTP with no +certificate import, and its SNMP agent answers v1 and v2c only. Choosing it +would reproduce, exactly, both of the firmware limits the purchase exists to +escape — and would do so on hardware that is capable, which is worse than the +MokerLink's honest incapacity. RouterOS has a `www-ssl` service that binds an +imported certificate, and an SNMP implementation with v3 authentication, +encryption and address-based access control. + +This is recorded rather than left obvious because the device is dual-boot: the +reset button offers the other OS, and a future operator troubleshooting a +RouterOS bridge at 1am will be tempted by the simpler page. The simpler page +gives the credential back to the wire. + +### The switch LAN outlives the reason it was built for + +[`network.md`](../network.md) says `10.7.7.0/24` exists "solely to reach the +switch's management UI, which will not bind to a tagged interface". That is a +MokerLink limitation, and it leaves with the MokerLink. Read narrowly, the LAN +should go: RouterOS will happily put management on a tagged VLAN, and ADR-0018 +recorded that `10.7.7.2` is the one address in a `10.0.x` estate that does not +match the convention. + +It should stay, for a reason it did not originally have. The way an operator +loses a RouterOS switch is a bridge VLAN-filtering commit — the config that +decides which ports carry which tags is the same config the management path +depends on, and a wrong `frame-types` or a forgotten tagged member on the +management VLAN locks the door from the inside. On the device that carries every +VLAN in the house, that is not a hypothetical. An untagged point-to-point link +from `morpheus`'s `igc0` to port 1 is unaffected by any of it: it is the way back +in, and it is a cable rather than a configuration. ADR-0018's other argument +survives untouched — the name depends on Unbound on `morpheus`, so +`neo.matrix.elysium` is gone at the moment the switch matters most, and +`10.7.7.2` is the break-glass form. Both stay written down. + +Keeping the **name** is the cheaper half of the same decision. `device: neo` is +a label on the SNMP scrape target, a series in two Grafana dashboards, a subject +in eleven runbooks and a node in `architecture.md`'s diagram. The switch is a +role before it is a box, and the role does not change. The alternative buys a +tidier history — one hostname, one physical device — at the cost of touching +every one of those during the window that is already the longest outage this +estate plans for. + +### The certificate comes from the estate's CA, not the tier's + +There are two certificate authorities here and +[`generate-certificates.md`](../runbooks/generate-certificates.md) opens by +saying which is which. The sensitive tier's CA issues over ACME from step-ca, +with seven-day leaves, for what Caddy serves on `trinity` +([ADR-0037](0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)). +A switch cannot ask for an ACME renewal, so that root is out on mechanism alone. + +The estate's CA is the right one for a second reason, not just by elimination: +Prometheus and blackbox-exporter already verify leaves beneath `ca.pem` by +`ca_file`. That means the `switch-ui` probe can move from `http` to `https` and +actually *verify* the certificate, rather than reaching for +`insecure_skip_verify` and monitoring a TLS listener without checking it is the +right one — which would be a strange end for a purchase made to stop trusting +the wire. + +### What does not change + +Port mirroring was in #444's selection criteria, and the CRS326 has it. It stays +off. [ADR-0006](0006-detect-at-the-chokepoint.md) decided at length that the +sensor belongs on `morpheus` at the chokepoint and that mirroring on `neo` stays +disabled, available on demand as a deliberate, temporary action; +[ADR-0039](0039-decline-proxmox-clustering-while-ifrit-is-the-range.md) restated +it. Buying a device that *can* mirror is not a decision to mirror, and the +criterion was about not foreclosing the option. + +Renumbering onto `10.0.99.x` also stays rejected. ADR-0018 rejected it rather +than deferring it, and every cost it named — the firewall's LAN interface, the +hard-coded targets, and the narrow passes that are all +[ADR-0025](0025-close-the-switch-lan-to-winterfell.md) left reaching +`10.7.7.0/24` — is still there. + +## Decision + +**RouterOS, the same name and address, and TLS on the management UI.** + +1. **The CRS326 runs RouterOS.** SwOS is rejected, not deferred: it serves + HTTP only, which reproduces exactly the limit the purchase exists to escape, + and it speaks SNMP v1 and v2c only, which gives back the authPriv the + replacement would otherwise gain. +2. **It inherits `neo` and `10.7.7.2`, and the switch LAN stays.** ADR-0018's + host override and its decision to keep the address written down beside the + name both survive as written. The LAN's justification is replaced, not + removed: it is the out-of-band path that survives a bad bridge + VLAN-filtering commit, not a workaround for a UI that would not bind to a + tag. +3. **`www-ssl` serves the UI, and plain `www` is disabled** — not merely + unused. The certificate is a leaf from the estate's CA, + `make certs ARGS="--host neo.matrix.elysium --ip 10.7.7.2"`, carrying both + the name and the break-glass address. +4. **An SNMPv3 authPriv user, by §4 of + [`rotate-snmp-community.md`](../runbooks/rotate-snmp-community.md), and v2c + off.** ADR-0036 left `neo` on v2c because the device could not be trusted to + persist the change; the replacement can. #85's switch half closes here, and + #84's community leaves with the hardware rather than being deleted from it. +5. **Port mirroring stays disabled**, per ADR-0006, and the capability is + recorded as available on demand. + +What the device turns out to be is not decided here. The serial, the management +MAC, the OS and version it ships with, and the factory reset go to +`hardware.md` at the bench. + +## Consequences + +- **Two accepted residuals close together, and they are the ones `neo` still + carried.** `SECURITY.md` and [`security.md`](../security.md) hold the plain + HTTP and #84's un-deletable community as knowingly accepted; both change at + the window, not before, and #84 closes on `snmp-verify.sh` clean over GET + *and* GETBULK against the new device. +- **The certificate is an 825-day leaf that the switch cannot renew.** The tier + CA's seven-day ACME leaves renew themselves; this one does not, and a switch + UI that has stopped being trusted by the browser is how an operator learns to + click through warnings. It needs a diary entry, and + [`schedule-maintenance.md`](../runbooks/schedule-maintenance.md) is where that + lives. +- **`switch-ui` and its `via: dns` twin move to `https` with a `ca_file`.** + [`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml) + uses `switch-ui` as its worked example of an endpoint with no dns twin; that + comment stopped being true under ADR-0018 and now needs a different subject + entirely. +- **The SNMP module is renamed and the port mapping is not assumed.** The + `mokerlink` module in `generator.yaml`, `auth_mokerlink`, the scrape target + and the switch block in `network.rules.yaml` all name a device that is gone. + More quietly: this is a 24 + 2 device replacing a 26-port one, so `ifIndex` + and `ifName` change, and any dashboard panel or rule that names a port must be + re-checked rather than carried across. +- **A firewall rule names the port, and it is the one change that can lock the + operator out.** [`network.md`](../network.md) records the switch LAN as + "blocked apart from `10.7.7.2:80`, the switch's own web UI". Turning plain + `www` off without widening that pass to `443` first leaves a reachable switch + that nothing is allowed to reach — from Hicks, which is the workstation the + window is being run from. Pass `443`, prove the new UI, then close `80`: that + ordering is in the runbook and it is not optional. +- **`snmp-walk.sh` may stop earning its keep.** It exists because the MokerLink + locks up under normal polling. If the CRS326 does not, the script becomes a + workaround for a device nobody has — which is a deletion, on evidence, not a + tidy-up. +- **The switch LAN is now documented as deliberate, which it was not.** + `network.md`'s note currently explains it as a consequence of the MokerLink's + firmware. Left as written, the next reader would correctly conclude the LAN + could go, and would be removing the way back in. +- **Nothing is deployed by this record.** It is a decision and a runbook; the + estate still runs on the MokerLink, still on plain HTTP, until a cabling + window outside working hours takes the house offline. +- **Reopened by:** the CRS326 arriving dead or misdescribed, which puts the + selection criteria back in play rather than this posture; a certificate + automation path the estate can actually serve a switch over, which would + retire the 825-day leaf and its diary entry; or a decision to give the estate + a second managed switch, at + which point whether `neo` is a role or a box has to be answered properly. diff --git a/docs/hardware.md b/docs/hardware.md index 29b119f8..c2fecd6c 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -304,7 +304,9 @@ revisions of this repository treated `shiva` as the hypervisor itself. not before — that issue also names the ADRs whose arithmetic the fit makes stale - MikroTik CRS326-24G-2S+RM[^CRS326] — 24 × 1 GbE, 2 × SFP+, 1U, dual-boot - RouterOS / SwOS — bought used 2026-09-13, in transit. The replacement for + RouterOS / SwOS — bought used 2026-09-13; in transit, delivery estimated + 2026-09-23, moved out from the 09-16 to 09-21 window quoted at purchase. The + replacement for `neo` that [ADR-0018](adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md) asked for in its last consequence and @@ -315,7 +317,13 @@ revisions of this repository treated `shiva` as the hypervisor itself. [#84](https://github.com/Gerrrt/HomeLab/issues/84)'s GETBULK residual rides along; SNMPv3 is not the argument, because [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) - found the switch in the rack already answers v3 on the wire. It enters the + found the switch in the rack already answers v3 on the wire. **How it is + configured was decided before the window rather than during it** — + [ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) + runs it on RouterOS, keeps the name `neo` and the address `10.7.7.2`, serves + `www-ssl` from a leaf off the estate's CA with plain `www` off, and gives it + an SNMPv3 authPriv user; the procedure is + [`swap-the-switch.md`](runbooks/swap-the-switch.md). It enters the Rack table at U9 and [`network.md`](network.md) when it is racked — `neo` carries every VLAN, so the swap is a house-wide outage and shares a rack visit rather than getting its own — and until then `neo` is the switch in diff --git a/docs/roadmap.md b/docs/roadmap.md index 6de3c814..2659b2af 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -964,6 +964,16 @@ months. with the repository side built and the device side not yet done. Under **Security** above, because it has a procedure now rather than only a decision. +- **[#444](https://github.com/Gerrrt/HomeLab/issues/444)** The CRS326's + operating system and management posture, decided by + [ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) + ahead of the cabling window rather than during it — RouterOS over SwOS, the + name `neo` and `10.7.7.2` inherited, `www-ssl` from the estate's CA with plain + `www` off, an SNMPv3 authPriv user, and mirroring left disabled per ADR-0006. + Nothing is built: the switch is still in transit and `neo` is still the + MokerLink. Under **Security** above, because it has a procedure now — + [`swap-the-switch.md`](runbooks/swap-the-switch.md) — rather than only a + decision. ## Considered and declined diff --git a/docs/runbooks/swap-the-switch.md b/docs/runbooks/swap-the-switch.md new file mode 100644 index 00000000..c3e72a25 --- /dev/null +++ b/docs/runbooks/swap-the-switch.md @@ -0,0 +1,262 @@ +# Runbook: Swap neo for the CRS326 + +**Target:** `neo` at `10.7.7.2`, Rack U9 — the MokerLink 26-port managed switch, +replaced by the MikroTik CRS326-24G-2S+RM +**Time:** about two hours at the bench, then a 30–60 minute window with the house +offline, then a pass over the documents +**You will need:** physical access to the rack, a workstation on Hicks, the age +key on the monitoring host, and the estate CA at `certificates/ca.pem` + +[ADR-0041](../adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) +decided what the new switch is and how it is configured; this is how it gets +there. The switch is replaced for one property — a TLS management interface, so +the read-write admin credential stops crossing the wire in clear through the +device it protects — and +[#84](https://github.com/Gerrrt/HomeLab/issues/84)'s un-deletable SNMP community +closes with it, because it leaves with the firmware. + +**`neo` carries every VLAN.** Phase 2 takes the whole house offline: no +internet, no wireless, no televisions, no cameras. Schedule it outside working +hours and share it with another rack visit rather than giving it its own. + +**Do Phase 1 completely before booking the window.** Everything except the +cabling can be done on a bench with the switch on a desk, and the entire point +of the split is that the window is cabling and verification, not configuration. + +--- + +## Read this part first + +**Three ways this locks you out. Two of them are silent.** + +1. **The firewall pass names port 80.** [`network.md`](../network.md) records + the switch LAN as blocked apart from `10.7.7.2:80`. Disable plain `www` on + the switch before widening that rule to `443` and you have a switch that is + up, correctly configured and unreachable from the workstation you are + standing at. **Widen first, prove the new UI, close `80` last.** +2. **Bridge VLAN filtering is one commit from a dead trunk.** On RouterOS the + config that decides which ports carry which tags is the config the management + path rides on. This is why ADR-0041 keeps the untagged `10.7.7.0/24` link + from `morpheus`'s `igc0` to port 1: it is a cable, not a configuration, and it + is the way back in. Do not put management on a tagged VLAN "while you are in + there". +3. **A used RouterOS device arrives with its last owner's configuration**, + users and all — the same lesson `SECURITY.md` records for `smaug`'s Intel + AMT, where a second-hand server arrived with the factory default still set on + a management plane nobody thought to look at. Reset it before it touches the + network, not after. + +**Silence Alertmanager before Phase 2.** Pulling the core switch is +indistinguishable from the estate burning down, and the alerts are real ones +that page a real person. + +--- + +## Phase 1 — at the bench, before any window + +Nothing here touches the running estate. The switch is on a desk, on its own +cable to a laptop. + +### 1.1 Capture the MokerLink first, while it is still running + +This is the step that makes the window recoverable, and it is the one most +easily skipped because the new switch is the interesting object. + +**Export the MokerLink's running configuration and write down its full port map +— every port, its VLAN membership, tagged or untagged, and what is plugged into +it.** No document in this repository records it. `architecture.md` has the +topology and `network.md` has the addresses; neither has a per-port map, so if +you unplug 26 cables without one, the house comes back wrong and you will be +diagnosing it at midnight with no switch to compare against. + +**Count the copper.** The MokerLink has 26 copper ports; the CRS326 has 24 × +RJ45 plus 2 × SFP+, which do not take an RJ45 patch lead. If more than 24 copper +ports are actually populated, **an SFP+ copper module is a purchase**, and by +[`roadmap.md`](../roadmap.md)'s own rule the PR that buys it edits *Everything +still to buy* in the same commit. Establish this now, not at the rack. + +Nothing in the estate uses PoE, so there is no powered-port shortfall to plan +around. + +### 1.2 Intake + +Confirm the OS and version it booted, the serial and the management MAC: + +```text +/system/resource/print +/system/routerboard/print +/interface/print +``` + +Check the box for rack ears and a power supply — it is a used listing. **These +facts go into [`hardware.md`](../hardware.md)**, replacing the "in transit" +line, and that edit can land on its own before the window. + +### 1.3 Reset, then RouterOS + +Boot RouterOS, not SwOS. ADR-0041 records why at length; briefly, SwOS serves +HTTP only and speaks SNMP v1 and v2c only, which is both of the firmware limits +this purchase exists to escape. + +Wipe whatever the last owner left: + +```text +/system/reset-configuration no-defaults=yes skip-backup=yes +``` + +Then set a unique admin password, from the password manager, and record where it +lives in [`successor-handover.md`](successor-handover.md) alongside the others. + +### 1.4 The certificate + +Issue the leaf from **the estate's CA**, not the sensitive tier's — the tier's +root issues seven-day leaves over ACME for `trinity` and a switch cannot ask for +a renewal ([ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md), +[`generate-certificates.md`](generate-certificates.md)). Run it **from the main +checkout on the monitoring host**, not from a worktree, so it lands in the +`certificates/` the live stack uses: + +```bash +make certs ARGS="--host neo.matrix.elysium --ip 10.7.7.2" +``` + +Both SANs matter: the name for normal use, the address because it is the +break-glass form when `morpheus` is the thing you are diagnosing. That produces +`certificates/neo.matrix.elysium.pem` and `certificates/neo.matrix.elysium-key.pem`. + +Upload both to the switch and import them, then bind `www-ssl`. **Leave plain +`www` enabled for now** — it is closed in Phase 2, after the firewall rule moves +and the new UI is proven: + +```text +/certificate/import file-name=neo.matrix.elysium.pem +/certificate/import file-name=neo.matrix.elysium-key.pem +/ip/service/set www-ssl certificate=neo.matrix.elysium disabled=no +``` + +Confirm the browser trusts it without a warning. If it does not, the leaf is +wrong or the CA is not installed on the workstation — fix that here, where there +is no outage running. + +### 1.5 SNMPv3, and no v2c + +Follow §4 of [`rotate-snmp-community.md`](rotate-snmp-community.md) for the +passphrases and for the five repository files that change. RouterOS configures +v3 users under `/snmp/community` with `security=private`, which reads oddly if +you are expecting a separate user table. + +**Nothing the MokerLink held is carried across.** `SECURITY.md` records that the +MokerLink's SNMP community leaked into a local session transcript on 2026-08-20 +and was *deliberately not rotated*, because rotating it meant repeating the +residual — the firmware would not persist the deletion of the old one. That +reasoning expires with the hardware, but only if the value does too. Generate +the v3 passphrases fresh (`make gen-secret ARGS=--snmp` for the shape, §4.1 of +[`rotate-snmp-community.md`](rotate-snmp-community.md) for the procedure) and +derive nothing from the community the old switch used. A leaked credential that +migrates onto its replacement has survived the swap that was meant to retire it. + +Restrict the agent to the scraper at `10.0.99.20` — the switch has address-based +access control and the MokerLink did not, so this is a control the estate gains +rather than one it carries across. **Configure no v2c community at all**, not +even a fresh one: ADR-0041 decision 4 turns v2c off, and the defaults `public` +and `private` go with it. Prove it with `snmp-verify.sh` pointed at the bench address before +the switch is ever racked; that proof is what closes #84, and doing it here +means the window does not have to. + +### 1.6 The port map, and mirroring stays off + +Build the bridge and its VLAN membership from the map captured in §1.1. Keep the +management path untagged on port 1. + +**Do not configure port mirroring.** The capability is why the CRS326 met #444's +criteria, and [ADR-0006](../adr/0006-detect-at-the-chokepoint.md) decided the +sensor belongs on `morpheus` at the chokepoint with the switch's mirroring +disabled and available on demand. Buying a device that can mirror was never a +decision to mirror. + +--- + +## Phase 2 — the window + +House offline. Alertmanager silenced. The MokerLink stays on the bench, cabled +and powered, until Phase 3 passes. + +1. **Widen the firewall pass to `443`** — the rule in `network.md` that admits + `10.7.7.2:80`. Both ports open for the duration of the window. +2. Rack the CRS326 at U9. Cat6 from `morpheus`'s `igc0` to **port 1**, the + trunk. +3. Move the patch leads, following the §1.1 map. +4. Verify **from the address before the name**: `https://10.7.7.2/` reaches the + UI and the certificate validates. `neo.matrix.elysium` depends on Unbound on + `morpheus`, which depends on the switch you have just replaced. +5. Walk the VLANs: internet, wireless, a camera on Skids, a host on VLAN 99, the + lab on VLAN 30. +6. Confirm the SNMP scrape is up and the `switch-ui` probe is green. +7. **Only now**, disable plain `www` on the switch and narrow the firewall rule + from `80` to `443`. Prove the UI again afterwards. + +If any of 4–6 fails and is not fixed within the window's budget, roll back: the +MokerLink returns to U9, the patch leads go back by the same map, and the +firewall rule goes back to `80`. A failed swap that is reverted is a short +outage; a failed swap that is debugged live is not. + +--- + +## Phase 3 — the documents and the configuration + +This is where the estate stops describing a MokerLink. None of it is urgent, and +all of it is the point. + +- [`network.md`](../network.md) — the LAN row, the `10.7.7.2` notes, the + `[^MokerLink]` footnote, and the Hicks rule that now names `443`. **Rewrite + the note explaining why the switch LAN exists**: it currently says the UI + "will not bind to a tagged interface", which was the MokerLink's limit and is + not the CRS326's. Left as written, the next reader correctly concludes the LAN + can go, and removes the way back in. ADR-0041 has the replacement reasoning. +- [`hardware.md`](../hardware.md) — the Rack table at U9, and the CRS326 entry + moves from *still moving* to on-hand with its serial and MAC. +- [`architecture.md`](../architecture.md) — the mermaid node label. +- `generator.yaml`'s `mokerlink` module and `auth_mokerlink`, + `prometheus/targets/snmp.yaml`, and the switch block in + `network.rules.yaml`. **Do not carry the port references across unchecked**: + this is a 24 + 2 device replacing a 26-port one, so `ifIndex` and `ifName` + change, and any rule or dashboard panel naming a port needs re-deriving. +- The `switch-ui` blackbox target and its `via: dns` twin, `http` → `https`, + with a `ca_file` rather than `insecure_skip_verify` — the estate CA is already + how blackbox verifies Grafana. + [`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml) + uses `switch-ui` as its worked example of an endpoint with no dns twin; that + needs a different subject. +- `SNMP_COMMUNITY_MOKERLINK` in `secrets/observability.sops.yaml` has no + consumer once v2c is off, and its name is a misnomer the moment `neo` is a + MikroTik. Retiring it touches `observability.example.yaml`, + `render-config.sh`, `verify-key-backup.sh` and `generator.yaml`. **Run + `make render` and `make reload` from the main checkout**, never a worktree — + render writes into the tree it runs from, and a worktree render produces a + file no container mounts. +- `snmp-verify.sh`'s GETBULK `WARN` becomes a `FAIL` again, now that the device + it excused is gone. +- `snmp-walk.sh` exists because the MokerLink locks up under normal polling. If + the CRS326 does not, the script is a workaround for a device nobody owns — + delete it on that evidence, not on the assumption. +- `SECURITY.md` and [`security.md`](../security.md) — the GETBULK and plain-HTTP + residuals close. Both are the accepted-residual record, so they close with the + date and the proof, not by deletion. +- [`roadmap.md`](../roadmap.md) — #84 and #444 move to *Done*. + +Close [#84](https://github.com/Gerrrt/HomeLab/issues/84) on `snmp-verify.sh` +clean over GET **and** GETBULK, against the new device and against the old +community. Then close +[#444](https://github.com/Gerrrt/HomeLab/issues/444). + +--- + +## Also required + +- The MokerLink leaves the estate carrying a community that cannot be deleted, + that is known to have leaked, and an admin password that has crossed the wire + in clear for its whole life. Wipe what can be wiped and do not re-home it onto + anything that matters. +- The leaf is good for 825 days and the switch cannot renew it. Put the expiry + in the diary — [`schedule-maintenance.md`](schedule-maintenance.md) is where + the estate's dated obligations live.