From 08760eed2d3d1664d980283402fb7d61468032ad Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:08:55 +0000 Subject: [PATCH] docs(security): #84 ends with the hardware, not another window #84 was holding a reboot window open for an overwrite that had already been carried out on 2026-09-12, and it was holding it on the strength of STILL ACCEPTED verdicts that the same day's measurement withdrew. neo does not check the community on GETBULK at or below sixteen characters, so every earlier sighting of the old string and of stock public/private was that bypass rather than a table row. SECURITY.md, docs/security.md and docs/roadmap.md were brought in line at the time; the issue was not, so it was the last artefact still directing a reader to overwrite a community row and reboot the switch the whole house runs through. Rewritten, and it stays open. docs/security.md carried a live contradiction. Its credentials-in-history table still asserted that neo "still accepts its previous community, and the stock public and private besides - measured 2026-09-06 and 2026-09-09", which are precisely the withdrawn GETBULK artefacts, while SECURITY.md recorded the row as unverified. The rotation runbook names this file as the one that must move with SECURITY.md and warns that leaving it stale "makes the two disagree, which is worse than either being stale alone". It had been disagreeing since 2026-09-12. SECURITY.md's transcript-leak row justified not rotating the switch community on the grounds that rotating it "means the neo residual above all over again". That reasoning is spent: the overwrite it feared was done and persisted, and per #444 the CRS326 is commissioned with v2c off entirely, so the leaked value is retired along with the PDU that served it rather than replaced by a fresh one. Recorded, so commissioning cannot quietly carry it across. Also sharpened "the string was not to hand in the window", which read as not yet. It is not recoverable, so the previous community's row retires with the hardware rather than by measurement - and noted that measuring it was never the expensive part, since --old is a single GET and the agent outage the issue warned about belonged to the deletion attempts. Delivery of the replacement is recorded as estimated 2026-09-23, a courier estimate rather than an arrival, and the swap is a house-wide outage sharing a rack visit - so it is the earliest a window could be picked, not a date this closes on. No device was touched: no render, no reload, no rotation, no reboot. docs/hardware.md, README.md and ADR-0018/0041 are deliberately untouched; a concurrent session owns those and was sent the delivery date and the commissioning requirement directly. make validate and python3 scripts/check_docs.py both pass. Refs #84 Co-Authored-By: Claude Opus 5 --- SECURITY.md | 19 +++++++++++++------ docs/roadmap.md | 13 ++++++++----- docs/security.md | 11 +++++++---- 3 files changed, 28 insertions(+), 15 deletions(-) diff --git a/SECURITY.md b/SECURITY.md index 299bef27..df138bae 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -43,7 +43,7 @@ is a very different thing from an overlooked one. Full detail in | Grafana `admin`/`admin` with anonymous Admin access enabled | Fixed — anonymous auth off, password from SOPS | | Passphrase-encrypted TLS private keys under `certificates/` | Removed from `HEAD` and purged from history. A new CA and leaf have been generated with [`scripts/gen-certs.sh`](scripts/gen-certs.sh); the old keys are superseded and should be treated as compromised wherever they were ever trusted. | | Decrypted secrets in editor undo files, written by `make secrets-edit` | Found 2026-08-20: three files under `~/.local/state/nvim/undodir/` holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664. Shredded. `make secrets-edit` now hardens `$EDITOR` before handing it plaintext, so it cannot recur. Never committed and never left the host, so those three communities were not rotated on that basis. | -| Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. | +| Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. **That reasoning is spent, and this value now retires with the hardware.** The overwrite it was afraid of was done on 2026-09-12 and persisted, and the MokerLink is being replaced by a switch commissioned with SNMP v2c off entirely — so what retires this community is not a fresh community but the absence of one. The thing that must not be carried across is any value the MokerLink held, passphrases included. | | Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. | | Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. | | Prometheus and Loki published on `0.0.0.0` with no authentication | **Accepted residual, not a fix in progress.** Anything that can route to `10.0.99.20:9090` or `10.0.99.20:3100` can read every metric and log line, inject metrics through Prometheus' remote-write receiver, and delete log ranges through Loki's delete API. Both stay published because `oracle`'s Alloy agent remote-writes to 9090 and pushes to 3100 — it is not a scrape target, so those ports are its only path. Firewall default-deny is the whole control, and since 2026-09-02 it is narrower than it was: the ingest ports are reachable from Winterfell (99) itself and from `10.0.30.110` on ImaginationLAN, which has an explicit pass for `Saruman`'s Alloy agent. Hicks (50) reaches `10.0.99.20` on `3000` only — a logged *Block access to Winterfell* drops the rest — and no untrusted segment reaches it at all. `docs/network.md` lists what Hicks may reach. Closing it properly means authentication in front of the ingest ports and a credential on every agent, which is a separate piece of work — see below. | @@ -76,7 +76,11 @@ Over GET, after the stale rows were overwritten and the switch rebooted on 2026-09-12, `public`, `private` and a junk string are refused. The previous community's row is the one thing still unmeasured over GET, because the string — the shared value purged from history — was not to hand in the -window; it is recorded as unverified rather than as retired. +window, and is not expected to become available: it is recorded as unverified +rather than as retired, and it retires with the hardware rather than by +measurement. If it ever does turn up, `./scripts/snmp-verify.sh --old` settles +it for the cost of a single GET — no SET, no deletion, and none of the agent +outage that the deletion attempts cost. `scripts/snmp-verify.sh` probes with GET since that date, because GET is what the switch authenticates. It also sends a short and a long junk string over @@ -97,10 +101,13 @@ exposed beyond those. What would close it is a switch whose firmware checks what it serves, which is the replacement [ADR-0018](docs/adr/0018-name-the-switch-and-leave-its-ui-on-plain-http.md) names — a MikroTik CRS326, bought 2026-09-13 and recorded in -[`docs/hardware.md`](docs/hardware.md); this residual closes when that switch -is racked, not before. SET was not tested, because -a SET is a change to the device. The overwrite procedure stays at [§2.5, -*The MokerLink switch: overwrite +[`docs/hardware.md`](docs/hardware.md), delivery estimated 2026-09-23; this +residual closes when the MokerLink hardware leaves the rack, not before. That +date is a courier estimate and not an arrival, and the swap is a house-wide +outage that shares a rack visit rather than getting its own — so it is not the +date this closes on, only the earliest a window could be picked. SET was not +tested, because a SET is a change to the device. The overwrite procedure stays +at [§2.5, *The MokerLink switch: overwrite the row*](docs/runbooks/rotate-snmp-community.md#the-mokerlink-switch-overwrite-the-row) for the rows that are real. diff --git a/docs/roadmap.md b/docs/roadmap.md index 6de3c814..d34979fb 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -203,11 +203,14 @@ either machine is. ACCEPTED` for the old one. Over GET, which it does check, both stock strings and a junk string are refused; the previous community's row is unverified rather than retired, because the string was not to hand in the - window. `snmp-verify.sh` probes with GET since that date and sends junk - strings over both PDUs weekly, `WARN` for the switch. Accepted residual, - larger than before and recorded in `SECURITY.md`; what closes it is the - replacement switch — bought 2026-09-13 and in transit — and the window - that racks it, not another window on this one. → + window and is not recoverable — it was the shared value purged from history. + So this closes by the hardware leaving, not by a measurement. `snmp-verify.sh` + probes with GET since that date and sends junk strings over both PDUs weekly, + `WARN` for the switch. Accepted residual, larger than before and recorded in + `SECURITY.md`; what closes it is the replacement switch — bought 2026-09-13, + delivery estimated 2026-09-23 — and the window that racks it, not another + window on this one. Nothing further is owed on the MokerLink: the overwrite + §2.5 prescribes was done on 2026-09-12 and persisted. → [runbook](runbooks/rotate-snmp-community.md#the-mokerlink-switch-overwrite-the-row) - **[#85](https://github.com/Gerrrt/HomeLab/issues/85) Move to SNMPv3 authPriv where the hardware supports it.** Decided by diff --git a/docs/security.md b/docs/security.md index 9709cb7e..b785dd87 100644 --- a/docs/security.md +++ b/docs/security.md @@ -318,10 +318,10 @@ repository must be treated as compromised: | What | Where | Status | | --- | --- | --- | -| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` (now rewritten) | Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. `morpheus`, `mjolnir` and `shiva` verified answering the new community and refusing the old; `neo` answers the new one but still accepts its previous community, and the stock `public` and `private` besides — measured 2026-09-06 and 2026-09-09, the other three refuse both — accepted risk, see [`SECURITY.md`](../SECURITY.md) and the [runbook](runbooks/rotate-snmp-community.md) | +| SNMP community shared across all four devices | `snmp.yaml`, from commit `ee3d443` (now rewritten) | Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. `morpheus`, `mjolnir` and `shiva` verified answering the new community and refusing the old. `neo` answers the new one; whether it still holds the previous one is **unverified**, not confirmed. Its agent serves GETBULK to any community of sixteen characters or fewer without consulting the table (measured 2026-09-12), which is what the 2026-09-06 and 2026-09-09 sightings of the old string and the stock `public` and `private` actually were; those measurements are withdrawn. Over GET, which it does check, both stock strings and a junk string are refused — the previous community's row is the one thing still unmeasured, because that string was purged from history and is not to hand. Accepted risk, see [`SECURITY.md`](../SECURITY.md) and the [runbook](runbooks/rotate-snmp-community.md) | | Grafana `admin` / `admin` with anonymous Admin access | compose file | Fixed: password from SOPS, anonymous auth disabled | | Decrypted secrets in editor undo files | `~/.local/state/nvim/undodir/`, written by `make secrets-edit` | Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. `make secrets-edit` now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis | -| Alertmanager webhook URL and the MokerLink SNMP community | a local Claude Code session transcript under `~/.claude/projects/` | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the `neo` residual below all over again | +| Alertmanager webhook URL and the MokerLink SNMP community | a local Claude Code session transcript under `~/.claude/projects/` | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the `neo` residual below all over again. That reasoning is spent — the overwrite it was afraid of was done on 2026-09-12 and persisted — and the value retires with the MokerLink hardware, which is replaced by a switch commissioned with v2c off entirely; no value the MokerLink held is carried across | | Passphrase-encrypted TLS private keys | `certificates/`, added in `efb2632`, deleted in `647d90a` | Purged from history, and the CA replaced — see [runbook](runbooks/generate-certificates.md). Anything that trusted the old CA must be re-pointed at the new one | CI scans both the working tree and the full history, with no ignore file. Both @@ -501,8 +501,11 @@ may or may not have (ADR-0036). A TLS management interface belongs in the selection criteria whenever this switch is replaced — and the replacement that has one, a MikroTik CRS326, was bought 2026-09-13 ([`hardware.md`](hardware.md), -[#444](https://github.com/Gerrrt/HomeLab/issues/444)). This section changes -when it is racked, not before. +[#444](https://github.com/Gerrrt/HomeLab/issues/444)), with delivery estimated +2026-09-23. This section changes when it is racked, not before — that date is a +courier estimate rather than an arrival, and the swap takes the whole house +offline and shares a rack visit, so it buys a window to pick rather than a date +this closes on. ## Hardening applied to the stack