diff --git a/README.md b/README.md index 31c85056..cfb49517 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,8 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 40 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 25 +- **Documented decisions and runbooks.** 41 ADRs covering what was chosen + and what was rejected — including the costs accepted knowingly; 26 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. @@ -207,7 +207,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). ├── docs/ │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md -│ ├── adr/ # 39 architecture decision records +│ ├── adr/ # 41 architecture decision records │ └── runbooks/ # successor handover (start here), deploy, converge, │ # add device, rotate creds, certs, key backup, │ # purge, restore the firewall, restore the stack, diff --git a/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md b/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md index d9e636f4..2f1c1c93 100644 --- a/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md +++ b/docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md @@ -3,13 +3,29 @@ **Status:** Accepted · 2026-09 > [!NOTE] -> The replacement the last consequence below asks for was bought on -> 2026-09-13: a MikroTik CRS326-24G-2S+RM, chosen for the TLS management -> interface ([`hardware.md`](../hardware.md), -> [#444](https://github.com/Gerrrt/HomeLab/issues/444)). Nothing here changes -> until it is racked — `neo` is still the switch, still on plain HTTP. The ADR -> that names the new switch and decides its OS and management posture is -> written then. The text here is left as written, per ADR-0001. +> **One clause of this ADR is superseded. The text below is left as written, +> per ADR-0001.** +> +> **The purchase.** The replacement the last consequence below asks for was +> bought on 2026-09-13: a MikroTik CRS326-24G-2S+RM, chosen for the TLS +> management interface ([`hardware.md`](../hardware.md), +> [#444](https://github.com/Gerrrt/HomeLab/issues/444)). +> +> **The UI.** Decision 2 — that the UI stays plain HTTP, and that #97's +> certificate goal is closed as not achievable — is superseded by +> [ADR-0041](0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md), +> which runs the CRS326 on RouterOS and serves the UI from `www-ssl` with a +> leaf from the estate's CA. Decisions 1 and 3 are unchanged, and so is the +> reasoning behind them: the switch keeps the name `neo` and the address +> `10.7.7.2`, and both stay written down because the name depends on +> `morpheus`. The rejection of renumbering onto `10.0.99.x` also stands. +> +> This note previously said that ADR would be written when the switch was +> racked. It was written before instead: a house-wide outage is not when to +> choose between a dual-boot device's two operating systems. **Nothing here +> changes until it is racked** — `neo` is still the MokerLink, still on plain +> HTTP, until the window in +> [`swap-the-switch.md`](../runbooks/swap-the-switch.md). ## Context diff --git a/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md b/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md new file mode 100644 index 00000000..594d9790 --- /dev/null +++ b/docs/adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md @@ -0,0 +1,196 @@ +# ADR-0041: Run the CRS326 on RouterOS, and keep neo and its switch LAN + +**Status:** Accepted · 2026-09 · supersedes the plain-HTTP clause of +[ADR-0018](0018-name-the-switch-and-leave-its-ui-on-plain-http.md) + +## Context + +[ADR-0018](0018-name-the-switch-and-leave-its-ui-on-plain-http.md) closed #97's +certificate goal as *not achievable on this hardware* and said the only thing +that would reopen it was replacing the switch. A replacement was bought on +2026-09-13 — a used MikroTik CRS326-24G-2S+RM, recorded in +[`hardware.md`](../hardware.md) — and +[#444](https://github.com/Gerrrt/HomeLab/issues/444) is where it was decided. + +It was bought for **one property, a TLS management interface**. The issue +originally named three residuals one purchase would close; +[ADR-0036](0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) +had already removed the middle one, because `neo`'s agent answers SNMPv3 on the +wire and the switch was never what blocked +[#85](https://github.com/Gerrrt/HomeLab/issues/85). What stands is the plain +HTTP — no TLS listener, no certificate import, checked against the live device +on 2026-09-04 — and with it the fact that the switch admin credential, which is +read-write, crosses the wire in clear *through the switch it protects*. +[#84](https://github.com/Gerrrt/HomeLab/issues/84)'s un-deletable community +rides along, because it leaves with the firmware that will not persist its +deletion. + +**Why this is written before the switch is racked, and not during.** ADR-0018's +own note said the successor ADR would be written then. That is the wrong time. +`neo` carries every VLAN, so the swap is a house-wide outage sharing a rack +visit, and an outage is not when to weigh a dual-boot device's two operating +systems or to decide whether a management LAN survives. This record decides the +half that cannot be changed later; what the device turns out to be — serial, +management MAC, the OS and version it arrives on, the factory reset — is learned +at the bench and written to `hardware.md`, which is the split +[ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md) already +uses for a purchase. Nothing in the estate changes until the cabling window; +[`swap-the-switch.md`](../runbooks/swap-the-switch.md) is the procedure. + +### The operating system is not really a choice + +The CRS326 dual-boots RouterOS and SwOS, and SwOS is the tempting one: a single +web page, no routing stack, nothing to misconfigure on a device that should only +switch. It cannot do the job. SwOS serves its web UI over plain HTTP with no +certificate import, and its SNMP agent answers v1 and v2c only. Choosing it +would reproduce, exactly, both of the firmware limits the purchase exists to +escape — and would do so on hardware that is capable, which is worse than the +MokerLink's honest incapacity. RouterOS has a `www-ssl` service that binds an +imported certificate, and an SNMP implementation with v3 authentication, +encryption and address-based access control. + +This is recorded rather than left obvious because the device is dual-boot: the +reset button offers the other OS, and a future operator troubleshooting a +RouterOS bridge at 1am will be tempted by the simpler page. The simpler page +gives the credential back to the wire. + +### The switch LAN outlives the reason it was built for + +[`network.md`](../network.md) says `10.7.7.0/24` exists "solely to reach the +switch's management UI, which will not bind to a tagged interface". That is a +MokerLink limitation, and it leaves with the MokerLink. Read narrowly, the LAN +should go: RouterOS will happily put management on a tagged VLAN, and ADR-0018 +recorded that `10.7.7.2` is the one address in a `10.0.x` estate that does not +match the convention. + +It should stay, for a reason it did not originally have. The way an operator +loses a RouterOS switch is a bridge VLAN-filtering commit — the config that +decides which ports carry which tags is the same config the management path +depends on, and a wrong `frame-types` or a forgotten tagged member on the +management VLAN locks the door from the inside. On the device that carries every +VLAN in the house, that is not a hypothetical. An untagged point-to-point link +from `morpheus`'s `igc0` to port 1 is unaffected by any of it: it is the way back +in, and it is a cable rather than a configuration. ADR-0018's other argument +survives untouched — the name depends on Unbound on `morpheus`, so +`neo.matrix.elysium` is gone at the moment the switch matters most, and +`10.7.7.2` is the break-glass form. Both stay written down. + +Keeping the **name** is the cheaper half of the same decision. `device: neo` is +a label on the SNMP scrape target, a series in two Grafana dashboards, a subject +in eleven runbooks and a node in `architecture.md`'s diagram. The switch is a +role before it is a box, and the role does not change. The alternative buys a +tidier history — one hostname, one physical device — at the cost of touching +every one of those during the window that is already the longest outage this +estate plans for. + +### The certificate comes from the estate's CA, not the tier's + +There are two certificate authorities here and +[`generate-certificates.md`](../runbooks/generate-certificates.md) opens by +saying which is which. The sensitive tier's CA issues over ACME from step-ca, +with seven-day leaves, for what Caddy serves on `trinity` +([ADR-0037](0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)). +A switch cannot ask for an ACME renewal, so that root is out on mechanism alone. + +The estate's CA is the right one for a second reason, not just by elimination: +Prometheus and blackbox-exporter already verify leaves beneath `ca.pem` by +`ca_file`. That means the `switch-ui` probe can move from `http` to `https` and +actually *verify* the certificate, rather than reaching for +`insecure_skip_verify` and monitoring a TLS listener without checking it is the +right one — which would be a strange end for a purchase made to stop trusting +the wire. + +### What does not change + +Port mirroring was in #444's selection criteria, and the CRS326 has it. It stays +off. [ADR-0006](0006-detect-at-the-chokepoint.md) decided at length that the +sensor belongs on `morpheus` at the chokepoint and that mirroring on `neo` stays +disabled, available on demand as a deliberate, temporary action; +[ADR-0039](0039-decline-proxmox-clustering-while-ifrit-is-the-range.md) restated +it. Buying a device that *can* mirror is not a decision to mirror, and the +criterion was about not foreclosing the option. + +Renumbering onto `10.0.99.x` also stays rejected. ADR-0018 rejected it rather +than deferring it, and every cost it named — the firewall's LAN interface, the +hard-coded targets, and the narrow passes that are all +[ADR-0025](0025-close-the-switch-lan-to-winterfell.md) left reaching +`10.7.7.0/24` — is still there. + +## Decision + +**RouterOS, the same name and address, and TLS on the management UI.** + +1. **The CRS326 runs RouterOS.** SwOS is rejected, not deferred: it serves + HTTP only, which reproduces exactly the limit the purchase exists to escape, + and it speaks SNMP v1 and v2c only, which gives back the authPriv the + replacement would otherwise gain. +2. **It inherits `neo` and `10.7.7.2`, and the switch LAN stays.** ADR-0018's + host override and its decision to keep the address written down beside the + name both survive as written. The LAN's justification is replaced, not + removed: it is the out-of-band path that survives a bad bridge + VLAN-filtering commit, not a workaround for a UI that would not bind to a + tag. +3. **`www-ssl` serves the UI, and plain `www` is disabled** — not merely + unused. The certificate is a leaf from the estate's CA, + `make certs ARGS="--host neo.matrix.elysium --ip 10.7.7.2"`, carrying both + the name and the break-glass address. +4. **An SNMPv3 authPriv user, by §4 of + [`rotate-snmp-community.md`](../runbooks/rotate-snmp-community.md), and v2c + off.** ADR-0036 left `neo` on v2c because the device could not be trusted to + persist the change; the replacement can. #85's switch half closes here, and + #84's community leaves with the hardware rather than being deleted from it. +5. **Port mirroring stays disabled**, per ADR-0006, and the capability is + recorded as available on demand. + +What the device turns out to be is not decided here. The serial, the management +MAC, the OS and version it ships with, and the factory reset go to +`hardware.md` at the bench. + +## Consequences + +- **Two accepted residuals close together, and they are the ones `neo` still + carried.** `SECURITY.md` and [`security.md`](../security.md) hold the plain + HTTP and #84's un-deletable community as knowingly accepted; both change at + the window, not before, and #84 closes on `snmp-verify.sh` clean over GET + *and* GETBULK against the new device. +- **The certificate is an 825-day leaf that the switch cannot renew.** The tier + CA's seven-day ACME leaves renew themselves; this one does not, and a switch + UI that has stopped being trusted by the browser is how an operator learns to + click through warnings. It needs a diary entry, and + [`schedule-maintenance.md`](../runbooks/schedule-maintenance.md) is where that + lives. +- **`switch-ui` and its `via: dns` twin move to `https` with a `ca_file`.** + [`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml) + uses `switch-ui` as its worked example of an endpoint with no dns twin; that + comment stopped being true under ADR-0018 and now needs a different subject + entirely. +- **The SNMP module is renamed and the port mapping is not assumed.** The + `mokerlink` module in `generator.yaml`, `auth_mokerlink`, the scrape target + and the switch block in `network.rules.yaml` all name a device that is gone. + More quietly: this is a 24 + 2 device replacing a 26-port one, so `ifIndex` + and `ifName` change, and any dashboard panel or rule that names a port must be + re-checked rather than carried across. +- **A firewall rule names the port, and it is the one change that can lock the + operator out.** [`network.md`](../network.md) records the switch LAN as + "blocked apart from `10.7.7.2:80`, the switch's own web UI". Turning plain + `www` off without widening that pass to `443` first leaves a reachable switch + that nothing is allowed to reach — from Hicks, which is the workstation the + window is being run from. Pass `443`, prove the new UI, then close `80`: that + ordering is in the runbook and it is not optional. +- **`snmp-walk.sh` may stop earning its keep.** It exists because the MokerLink + locks up under normal polling. If the CRS326 does not, the script becomes a + workaround for a device nobody has — which is a deletion, on evidence, not a + tidy-up. +- **The switch LAN is now documented as deliberate, which it was not.** + `network.md`'s note currently explains it as a consequence of the MokerLink's + firmware. Left as written, the next reader would correctly conclude the LAN + could go, and would be removing the way back in. +- **Nothing is deployed by this record.** It is a decision and a runbook; the + estate still runs on the MokerLink, still on plain HTTP, until a cabling + window outside working hours takes the house offline. +- **Reopened by:** the CRS326 arriving dead or misdescribed, which puts the + selection criteria back in play rather than this posture; a certificate + automation path the estate can actually serve a switch over, which would + retire the 825-day leaf and its diary entry; or a decision to give the estate + a second managed switch, at + which point whether `neo` is a role or a box has to be answered properly. diff --git a/docs/hardware.md b/docs/hardware.md index 29b119f8..c2fecd6c 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -304,7 +304,9 @@ revisions of this repository treated `shiva` as the hypervisor itself. not before — that issue also names the ADRs whose arithmetic the fit makes stale - MikroTik CRS326-24G-2S+RM[^CRS326] — 24 × 1 GbE, 2 × SFP+, 1U, dual-boot - RouterOS / SwOS — bought used 2026-09-13, in transit. The replacement for + RouterOS / SwOS — bought used 2026-09-13; in transit, delivery estimated + 2026-09-23, moved out from the 09-16 to 09-21 window quoted at purchase. The + replacement for `neo` that [ADR-0018](adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md) asked for in its last consequence and @@ -315,7 +317,13 @@ revisions of this repository treated `shiva` as the hypervisor itself. [#84](https://github.com/Gerrrt/HomeLab/issues/84)'s GETBULK residual rides along; SNMPv3 is not the argument, because [ADR-0036](adr/0036-poll-the-ilo-and-the-ups-card-over-snmpv3-and-keep-the-firewall-on-bsnmpd.md) - found the switch in the rack already answers v3 on the wire. It enters the + found the switch in the rack already answers v3 on the wire. **How it is + configured was decided before the window rather than during it** — + [ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) + runs it on RouterOS, keeps the name `neo` and the address `10.7.7.2`, serves + `www-ssl` from a leaf off the estate's CA with plain `www` off, and gives it + an SNMPv3 authPriv user; the procedure is + [`swap-the-switch.md`](runbooks/swap-the-switch.md). It enters the Rack table at U9 and [`network.md`](network.md) when it is racked — `neo` carries every VLAN, so the swap is a house-wide outage and shares a rack visit rather than getting its own — and until then `neo` is the switch in diff --git a/docs/roadmap.md b/docs/roadmap.md index 6de3c814..2659b2af 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -964,6 +964,16 @@ months. with the repository side built and the device side not yet done. Under **Security** above, because it has a procedure now rather than only a decision. +- **[#444](https://github.com/Gerrrt/HomeLab/issues/444)** The CRS326's + operating system and management posture, decided by + [ADR-0041](adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) + ahead of the cabling window rather than during it — RouterOS over SwOS, the + name `neo` and `10.7.7.2` inherited, `www-ssl` from the estate's CA with plain + `www` off, an SNMPv3 authPriv user, and mirroring left disabled per ADR-0006. + Nothing is built: the switch is still in transit and `neo` is still the + MokerLink. Under **Security** above, because it has a procedure now — + [`swap-the-switch.md`](runbooks/swap-the-switch.md) — rather than only a + decision. ## Considered and declined diff --git a/docs/runbooks/swap-the-switch.md b/docs/runbooks/swap-the-switch.md new file mode 100644 index 00000000..c3e72a25 --- /dev/null +++ b/docs/runbooks/swap-the-switch.md @@ -0,0 +1,262 @@ +# Runbook: Swap neo for the CRS326 + +**Target:** `neo` at `10.7.7.2`, Rack U9 — the MokerLink 26-port managed switch, +replaced by the MikroTik CRS326-24G-2S+RM +**Time:** about two hours at the bench, then a 30–60 minute window with the house +offline, then a pass over the documents +**You will need:** physical access to the rack, a workstation on Hicks, the age +key on the monitoring host, and the estate CA at `certificates/ca.pem` + +[ADR-0041](../adr/0041-run-the-crs326-on-routeros-and-keep-neo-and-its-switch-lan.md) +decided what the new switch is and how it is configured; this is how it gets +there. The switch is replaced for one property — a TLS management interface, so +the read-write admin credential stops crossing the wire in clear through the +device it protects — and +[#84](https://github.com/Gerrrt/HomeLab/issues/84)'s un-deletable SNMP community +closes with it, because it leaves with the firmware. + +**`neo` carries every VLAN.** Phase 2 takes the whole house offline: no +internet, no wireless, no televisions, no cameras. Schedule it outside working +hours and share it with another rack visit rather than giving it its own. + +**Do Phase 1 completely before booking the window.** Everything except the +cabling can be done on a bench with the switch on a desk, and the entire point +of the split is that the window is cabling and verification, not configuration. + +--- + +## Read this part first + +**Three ways this locks you out. Two of them are silent.** + +1. **The firewall pass names port 80.** [`network.md`](../network.md) records + the switch LAN as blocked apart from `10.7.7.2:80`. Disable plain `www` on + the switch before widening that rule to `443` and you have a switch that is + up, correctly configured and unreachable from the workstation you are + standing at. **Widen first, prove the new UI, close `80` last.** +2. **Bridge VLAN filtering is one commit from a dead trunk.** On RouterOS the + config that decides which ports carry which tags is the config the management + path rides on. This is why ADR-0041 keeps the untagged `10.7.7.0/24` link + from `morpheus`'s `igc0` to port 1: it is a cable, not a configuration, and it + is the way back in. Do not put management on a tagged VLAN "while you are in + there". +3. **A used RouterOS device arrives with its last owner's configuration**, + users and all — the same lesson `SECURITY.md` records for `smaug`'s Intel + AMT, where a second-hand server arrived with the factory default still set on + a management plane nobody thought to look at. Reset it before it touches the + network, not after. + +**Silence Alertmanager before Phase 2.** Pulling the core switch is +indistinguishable from the estate burning down, and the alerts are real ones +that page a real person. + +--- + +## Phase 1 — at the bench, before any window + +Nothing here touches the running estate. The switch is on a desk, on its own +cable to a laptop. + +### 1.1 Capture the MokerLink first, while it is still running + +This is the step that makes the window recoverable, and it is the one most +easily skipped because the new switch is the interesting object. + +**Export the MokerLink's running configuration and write down its full port map +— every port, its VLAN membership, tagged or untagged, and what is plugged into +it.** No document in this repository records it. `architecture.md` has the +topology and `network.md` has the addresses; neither has a per-port map, so if +you unplug 26 cables without one, the house comes back wrong and you will be +diagnosing it at midnight with no switch to compare against. + +**Count the copper.** The MokerLink has 26 copper ports; the CRS326 has 24 × +RJ45 plus 2 × SFP+, which do not take an RJ45 patch lead. If more than 24 copper +ports are actually populated, **an SFP+ copper module is a purchase**, and by +[`roadmap.md`](../roadmap.md)'s own rule the PR that buys it edits *Everything +still to buy* in the same commit. Establish this now, not at the rack. + +Nothing in the estate uses PoE, so there is no powered-port shortfall to plan +around. + +### 1.2 Intake + +Confirm the OS and version it booted, the serial and the management MAC: + +```text +/system/resource/print +/system/routerboard/print +/interface/print +``` + +Check the box for rack ears and a power supply — it is a used listing. **These +facts go into [`hardware.md`](../hardware.md)**, replacing the "in transit" +line, and that edit can land on its own before the window. + +### 1.3 Reset, then RouterOS + +Boot RouterOS, not SwOS. ADR-0041 records why at length; briefly, SwOS serves +HTTP only and speaks SNMP v1 and v2c only, which is both of the firmware limits +this purchase exists to escape. + +Wipe whatever the last owner left: + +```text +/system/reset-configuration no-defaults=yes skip-backup=yes +``` + +Then set a unique admin password, from the password manager, and record where it +lives in [`successor-handover.md`](successor-handover.md) alongside the others. + +### 1.4 The certificate + +Issue the leaf from **the estate's CA**, not the sensitive tier's — the tier's +root issues seven-day leaves over ACME for `trinity` and a switch cannot ask for +a renewal ([ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md), +[`generate-certificates.md`](generate-certificates.md)). Run it **from the main +checkout on the monitoring host**, not from a worktree, so it lands in the +`certificates/` the live stack uses: + +```bash +make certs ARGS="--host neo.matrix.elysium --ip 10.7.7.2" +``` + +Both SANs matter: the name for normal use, the address because it is the +break-glass form when `morpheus` is the thing you are diagnosing. That produces +`certificates/neo.matrix.elysium.pem` and `certificates/neo.matrix.elysium-key.pem`. + +Upload both to the switch and import them, then bind `www-ssl`. **Leave plain +`www` enabled for now** — it is closed in Phase 2, after the firewall rule moves +and the new UI is proven: + +```text +/certificate/import file-name=neo.matrix.elysium.pem +/certificate/import file-name=neo.matrix.elysium-key.pem +/ip/service/set www-ssl certificate=neo.matrix.elysium disabled=no +``` + +Confirm the browser trusts it without a warning. If it does not, the leaf is +wrong or the CA is not installed on the workstation — fix that here, where there +is no outage running. + +### 1.5 SNMPv3, and no v2c + +Follow §4 of [`rotate-snmp-community.md`](rotate-snmp-community.md) for the +passphrases and for the five repository files that change. RouterOS configures +v3 users under `/snmp/community` with `security=private`, which reads oddly if +you are expecting a separate user table. + +**Nothing the MokerLink held is carried across.** `SECURITY.md` records that the +MokerLink's SNMP community leaked into a local session transcript on 2026-08-20 +and was *deliberately not rotated*, because rotating it meant repeating the +residual — the firmware would not persist the deletion of the old one. That +reasoning expires with the hardware, but only if the value does too. Generate +the v3 passphrases fresh (`make gen-secret ARGS=--snmp` for the shape, §4.1 of +[`rotate-snmp-community.md`](rotate-snmp-community.md) for the procedure) and +derive nothing from the community the old switch used. A leaked credential that +migrates onto its replacement has survived the swap that was meant to retire it. + +Restrict the agent to the scraper at `10.0.99.20` — the switch has address-based +access control and the MokerLink did not, so this is a control the estate gains +rather than one it carries across. **Configure no v2c community at all**, not +even a fresh one: ADR-0041 decision 4 turns v2c off, and the defaults `public` +and `private` go with it. Prove it with `snmp-verify.sh` pointed at the bench address before +the switch is ever racked; that proof is what closes #84, and doing it here +means the window does not have to. + +### 1.6 The port map, and mirroring stays off + +Build the bridge and its VLAN membership from the map captured in §1.1. Keep the +management path untagged on port 1. + +**Do not configure port mirroring.** The capability is why the CRS326 met #444's +criteria, and [ADR-0006](../adr/0006-detect-at-the-chokepoint.md) decided the +sensor belongs on `morpheus` at the chokepoint with the switch's mirroring +disabled and available on demand. Buying a device that can mirror was never a +decision to mirror. + +--- + +## Phase 2 — the window + +House offline. Alertmanager silenced. The MokerLink stays on the bench, cabled +and powered, until Phase 3 passes. + +1. **Widen the firewall pass to `443`** — the rule in `network.md` that admits + `10.7.7.2:80`. Both ports open for the duration of the window. +2. Rack the CRS326 at U9. Cat6 from `morpheus`'s `igc0` to **port 1**, the + trunk. +3. Move the patch leads, following the §1.1 map. +4. Verify **from the address before the name**: `https://10.7.7.2/` reaches the + UI and the certificate validates. `neo.matrix.elysium` depends on Unbound on + `morpheus`, which depends on the switch you have just replaced. +5. Walk the VLANs: internet, wireless, a camera on Skids, a host on VLAN 99, the + lab on VLAN 30. +6. Confirm the SNMP scrape is up and the `switch-ui` probe is green. +7. **Only now**, disable plain `www` on the switch and narrow the firewall rule + from `80` to `443`. Prove the UI again afterwards. + +If any of 4–6 fails and is not fixed within the window's budget, roll back: the +MokerLink returns to U9, the patch leads go back by the same map, and the +firewall rule goes back to `80`. A failed swap that is reverted is a short +outage; a failed swap that is debugged live is not. + +--- + +## Phase 3 — the documents and the configuration + +This is where the estate stops describing a MokerLink. None of it is urgent, and +all of it is the point. + +- [`network.md`](../network.md) — the LAN row, the `10.7.7.2` notes, the + `[^MokerLink]` footnote, and the Hicks rule that now names `443`. **Rewrite + the note explaining why the switch LAN exists**: it currently says the UI + "will not bind to a tagged interface", which was the MokerLink's limit and is + not the CRS326's. Left as written, the next reader correctly concludes the LAN + can go, and removes the way back in. ADR-0041 has the replacement reasoning. +- [`hardware.md`](../hardware.md) — the Rack table at U9, and the CRS326 entry + moves from *still moving* to on-hand with its serial and MAC. +- [`architecture.md`](../architecture.md) — the mermaid node label. +- `generator.yaml`'s `mokerlink` module and `auth_mokerlink`, + `prometheus/targets/snmp.yaml`, and the switch block in + `network.rules.yaml`. **Do not carry the port references across unchecked**: + this is a 24 + 2 device replacing a 26-port one, so `ifIndex` and `ifName` + change, and any rule or dashboard panel naming a port needs re-deriving. +- The `switch-ui` blackbox target and its `via: dns` twin, `http` → `https`, + with a `ca_file` rather than `insecure_skip_verify` — the estate CA is already + how blackbox verifies Grafana. + [`blackbox.test.yaml`](../../stacks/observability/prometheus/tests/blackbox.test.yaml) + uses `switch-ui` as its worked example of an endpoint with no dns twin; that + needs a different subject. +- `SNMP_COMMUNITY_MOKERLINK` in `secrets/observability.sops.yaml` has no + consumer once v2c is off, and its name is a misnomer the moment `neo` is a + MikroTik. Retiring it touches `observability.example.yaml`, + `render-config.sh`, `verify-key-backup.sh` and `generator.yaml`. **Run + `make render` and `make reload` from the main checkout**, never a worktree — + render writes into the tree it runs from, and a worktree render produces a + file no container mounts. +- `snmp-verify.sh`'s GETBULK `WARN` becomes a `FAIL` again, now that the device + it excused is gone. +- `snmp-walk.sh` exists because the MokerLink locks up under normal polling. If + the CRS326 does not, the script is a workaround for a device nobody owns — + delete it on that evidence, not on the assumption. +- `SECURITY.md` and [`security.md`](../security.md) — the GETBULK and plain-HTTP + residuals close. Both are the accepted-residual record, so they close with the + date and the proof, not by deletion. +- [`roadmap.md`](../roadmap.md) — #84 and #444 move to *Done*. + +Close [#84](https://github.com/Gerrrt/HomeLab/issues/84) on `snmp-verify.sh` +clean over GET **and** GETBULK, against the new device and against the old +community. Then close +[#444](https://github.com/Gerrrt/HomeLab/issues/444). + +--- + +## Also required + +- The MokerLink leaves the estate carrying a community that cannot be deleted, + that is known to have leaked, and an admin password that has crossed the wire + in clear for its whole life. Wipe what can be wiped and do not re-home it onto + anything that matters. +- The leaf is good for 825 days and the switch cannot renew it. Put the expiry + in the diary — [`schedule-maintenance.md`](schedule-maintenance.md) is where + the estate's dated obligations live.