diff --git a/docs/roadmap.md b/docs/roadmap.md index 6de3c81..3edb934 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1050,6 +1050,51 @@ them name the condition that would change the answer. ## Done +- [x] **[#441](https://github.com/Gerrrt/HomeLab/issues/441) Alerted on a sensor + that stops logging, and closed the Zeek half by deciding it elsewhere.** + 2026-09-17. `SuricataLogsStopped` landed on 2026-09-12 in + `loki/rules/security.rules.yaml`: `absent_over_time({app="suricata"}[9h])`, + aggregate across both interfaces, `for: 0s` like the two absence rules it + copies. The window was read rather than picked — 22 days of the stream, in + which the longest silence across both interfaces was 75 minutes — which is + what the issue asked for, because the thirty minutes it opened with would + have paged about twice a day. + + **Re-measured on 2026-09-17 over 27.7 days and it holds.** Worst aggregate + silence 80 minutes, six gaps past an hour, none past two, so nine hours + keeps nearly seven times the headroom `DhcpLeaseLogsStopped` settled on. + The aggregate design earned its keep in the interval: `igc0.10` alone went + quiet for more than nine hours **seven times**, worst 47.5 hours, so a + per-interface rule would have paged seven times in a fortnight. The + short-gap counts are not comparable between the two measurements — 135 + over thirty minutes here against 54 on 2026-09-12, a difference in method, + not in Suricata — and the rule's comment says so rather than presenting + one series. + + **The Zeek half was not built, and will not be as the issue described it.** + `ZeekLogsStopped` was to be the same shape against the lab's Loki once + [#437](https://github.com/Gerrrt/HomeLab/issues/437) existed. Three things + make that the wrong instrument. #437 puts Zeek's logs on `alexander` and + never on `10.0.99.20`, which ADR-0007 requires; + [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) + gives that stack no Alertmanager on purpose, and `stacks/lab/loki/` ships + no ruler and no `rules/` because *"a ruler with nowhere to deliver + evaluates rules and discards the result, which reads as coverage and is + not"*; and + [ADR-0028](adr/0028-let-guest-liveness-cross-but-not-guest-telemetry.md) + revisited that exact boundary and kept it. #437 already carries the right + mechanism — a `homelab_zeek_mirror_active` textfile gauge read on the + hypervisor, crossing as guest state under ADR-0028 — so Zeek's liveness is + a metric question answered where Zeek is built, and this issue closes + rather than holding a slot behind a Wave 2 dependency that itself waits on + [#414](https://github.com/Gerrrt/HomeLab/issues/414). + + What this proves and what it does not, unchanged from the merge: Loki + rules still have no unit-test harness, so `check_loki_rules.sh` shows the + rule parses and the ruler evaluates it, not that it fires. The live ruler + reports it `health=ok` and `state=inactive`, which is the good state and + not evidence of detection. + - [x] **[#470](https://github.com/Gerrrt/HomeLab/issues/470) The wiki's drift check is watched.** 2026-09-14. `Gerrrt/Lemmiwinks/.claude/tools/drift-check` reads the wiki's machine-checkable claims against the machine and files a diff --git a/docs/security.md b/docs/security.md index 9709cb7..b1d7a90 100644 --- a/docs/security.md +++ b/docs/security.md @@ -101,8 +101,26 @@ process is alive, not that it is inspecting anything. `SuricataLogsStopped` in once both interfaces have been silent for nine hours, a window read from 22 days of the stream whose longest silence was 75 minutes, and it cannot see one interface going quiet on its own — the guest segment is silent for days at a -time ([#441](https://github.com/Gerrrt/HomeLab/issues/441)). The runbook's test -alert is still the only proof that it detects. +time ([#441](https://github.com/Gerrrt/HomeLab/issues/441)). Re-measured on +2026-09-17 over 27.7 days, the worst aggregate silence was 80 minutes and the +guest segment alone crossed nine hours seven times, so the window holds and the +decision to aggregate is what keeps the rule quiet. The runbook's test alert is +still the only proof that it detects. + +**A limit none of the three names, and no rule here closes: Suricata is not a +protocol logger, and the lab is out of reach.** SNI, JA3 and certificate metadata — the +ground the plaintext limit gives up — are Zeek's, and east-west traffic between +the lab's domain guests crosses no router, so `morpheus` never sees a packet of +it ([#437](https://github.com/Gerrrt/HomeLab/issues/437), +[ADR-0006](adr/0006-detect-at-the-chokepoint.md)). When that sensor is built its +logs stay on `alexander` and never reach `10.0.99.20`, which ADR-0007 requires +and [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) +enforces by giving the lab no Alertmanager for a ruler to deliver to. So the +absence rule this section describes has no Zeek equivalent and will not get one: +whether that sensor is still running is answered on the hypervisor, as guest +state crossing under +[ADR-0028](adr/0028-let-guest-liveness-cross-but-not-guest-telemetry.md), by the +`homelab_zeek_mirror_active` gauge #437 builds alongside the mirror. **Device joins are detected as of 2026-09-04**, from the DHCP server rather than from the wireless. `morpheus` ships Kea's lease log to Loki, and the first diff --git a/stacks/observability/loki/rules/security.rules.yaml b/stacks/observability/loki/rules/security.rules.yaml index 4923756..8231803 100644 --- a/stacks/observability/loki/rules/security.rules.yaml +++ b/stacks/observability/loki/rules/security.rules.yaml @@ -616,6 +616,23 @@ groups: # SuricataLogsStopped fires only when BOTH interfaces have been silent, and a # Degens-only silence is invisible to it by design. # + # RE-VERIFIED 2026-09-17, five weeks on and over a retention window that has + # rolled forward: 27.7 days ending 2026-09-17 22:25 UTC, same five-minute + # buckets. The worst aggregate silence is 80 minutes — five minutes longer + # than the 2026-09-12 reading — six gaps exceeded an hour, none exceeded two, + # and nine hours is still nearly seven times the worst gap. The case for + # aggregating got stronger rather than weaker: igc0.10 alone crossed nine + # hours SEVEN times, worst 47.5 hours, so a per-interface rule would have + # paged seven times in a fortnight. + # + # The short-gap counts are NOT comparable across the two measurements — this + # one counts 135 gaps over thirty minutes where 2026-09-12 counted 54, which + # is a difference in method (bucket alignment, and which streams were summed) + # rather than in Suricata. The long tail, which is the half that sets the + # window, agrees. Note also that `interface` is only stamped from 2026-09-02, + # when syslog.alloy began mapping facility local1/local2, so the per-interface + # figures cover 13.7 days against the aggregate's 27.7. + # # Two rules, two questions. SuricataStopped in prometheus/rules/ids.rules.yaml # reads the firewall's process table over SNMP (#90) — bsnmpd serves # HOST-RESOURCES-MIB hrSWRunTable, the pfsense module fetches the rows named