Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -1050,6 +1050,51 @@ them name the condition that would change the answer.

## Done

- [x] **[#441](https://github.com/Gerrrt/HomeLab/issues/441) Alerted on a sensor
that stops logging, and closed the Zeek half by deciding it elsewhere.**
2026-09-17. `SuricataLogsStopped` landed on 2026-09-12 in
`loki/rules/security.rules.yaml`: `absent_over_time({app="suricata"}[9h])`,
aggregate across both interfaces, `for: 0s` like the two absence rules it
copies. The window was read rather than picked — 22 days of the stream, in
which the longest silence across both interfaces was 75 minutes — which is
what the issue asked for, because the thirty minutes it opened with would
have paged about twice a day.

**Re-measured on 2026-09-17 over 27.7 days and it holds.** Worst aggregate
silence 80 minutes, six gaps past an hour, none past two, so nine hours
keeps nearly seven times the headroom `DhcpLeaseLogsStopped` settled on.
The aggregate design earned its keep in the interval: `igc0.10` alone went
quiet for more than nine hours **seven times**, worst 47.5 hours, so a
per-interface rule would have paged seven times in a fortnight. The
short-gap counts are not comparable between the two measurements — 135
over thirty minutes here against 54 on 2026-09-12, a difference in method,
not in Suricata — and the rule's comment says so rather than presenting
one series.

**The Zeek half was not built, and will not be as the issue described it.**
`ZeekLogsStopped` was to be the same shape against the lab's Loki once
[#437](https://github.com/Gerrrt/HomeLab/issues/437) existed. Three things
make that the wrong instrument. #437 puts Zeek's logs on `alexander` and
never on `10.0.99.20`, which ADR-0007 requires;
[ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)
gives that stack no Alertmanager on purpose, and `stacks/lab/loki/` ships
no ruler and no `rules/` because *"a ruler with nowhere to deliver
evaluates rules and discards the result, which reads as coverage and is
not"*; and
[ADR-0028](adr/0028-let-guest-liveness-cross-but-not-guest-telemetry.md)
revisited that exact boundary and kept it. #437 already carries the right
mechanism — a `homelab_zeek_mirror_active` textfile gauge read on the
hypervisor, crossing as guest state under ADR-0028 — so Zeek's liveness is
a metric question answered where Zeek is built, and this issue closes
rather than holding a slot behind a Wave 2 dependency that itself waits on
[#414](https://github.com/Gerrrt/HomeLab/issues/414).

What this proves and what it does not, unchanged from the merge: Loki
rules still have no unit-test harness, so `check_loki_rules.sh` shows the
rule parses and the ruler evaluates it, not that it fires. The live ruler
reports it `health=ok` and `state=inactive`, which is the good state and
not evidence of detection.

- [x] **[#470](https://github.com/Gerrrt/HomeLab/issues/470) The wiki's drift
check is watched.** 2026-09-14. `Gerrrt/Lemmiwinks/.claude/tools/drift-check`
reads the wiki's machine-checkable claims against the machine and files a
Expand Down
22 changes: 20 additions & 2 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,26 @@ process is alive, not that it is inspecting anything. `SuricataLogsStopped` in
once both interfaces have been silent for nine hours, a window read from 22 days
of the stream whose longest silence was 75 minutes, and it cannot see one
interface going quiet on its own — the guest segment is silent for days at a
time ([#441](https://github.com/Gerrrt/HomeLab/issues/441)). The runbook's test
alert is still the only proof that it detects.
time ([#441](https://github.com/Gerrrt/HomeLab/issues/441)). Re-measured on
2026-09-17 over 27.7 days, the worst aggregate silence was 80 minutes and the
guest segment alone crossed nine hours seven times, so the window holds and the
decision to aggregate is what keeps the rule quiet. The runbook's test alert is
still the only proof that it detects.

**A limit none of the three names, and no rule here closes: Suricata is not a
protocol logger, and the lab is out of reach.** SNI, JA3 and certificate metadata — the
ground the plaintext limit gives up — are Zeek's, and east-west traffic between
the lab's domain guests crosses no router, so `morpheus` never sees a packet of
it ([#437](https://github.com/Gerrrt/HomeLab/issues/437),
[ADR-0006](adr/0006-detect-at-the-chokepoint.md)). When that sensor is built its
logs stay on `alexander` and never reach `10.0.99.20`, which ADR-0007 requires
and [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)
enforces by giving the lab no Alertmanager for a ruler to deliver to. So the
absence rule this section describes has no Zeek equivalent and will not get one:
whether that sensor is still running is answered on the hypervisor, as guest
state crossing under
[ADR-0028](adr/0028-let-guest-liveness-cross-but-not-guest-telemetry.md), by the
`homelab_zeek_mirror_active` gauge #437 builds alongside the mirror.

**Device joins are detected as of 2026-09-04**, from the DHCP server rather
than from the wireless. `morpheus` ships Kea's lease log to Loki, and the first
Expand Down
17 changes: 17 additions & 0 deletions stacks/observability/loki/rules/security.rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -616,6 +616,23 @@ groups:
# SuricataLogsStopped fires only when BOTH interfaces have been silent, and a
# Degens-only silence is invisible to it by design.
#
# RE-VERIFIED 2026-09-17, five weeks on and over a retention window that has
# rolled forward: 27.7 days ending 2026-09-17 22:25 UTC, same five-minute
# buckets. The worst aggregate silence is 80 minutes — five minutes longer
# than the 2026-09-12 reading — six gaps exceeded an hour, none exceeded two,
# and nine hours is still nearly seven times the worst gap. The case for
# aggregating got stronger rather than weaker: igc0.10 alone crossed nine
# hours SEVEN times, worst 47.5 hours, so a per-interface rule would have
# paged seven times in a fortnight.
#
# The short-gap counts are NOT comparable across the two measurements — this
# one counts 135 gaps over thirty minutes where 2026-09-12 counted 54, which
# is a difference in method (bucket alignment, and which streams were summed)
# rather than in Suricata. The long tail, which is the half that sets the
# window, agrees. Note also that `interface` is only stamped from 2026-09-02,
# when syslog.alloy began mapping facility local1/local2, so the per-interface
# figures cover 13.7 days against the aggregate's 27.7.
#
# Two rules, two questions. SuricataStopped in prometheus/rules/ids.rules.yaml
# reads the firewall's process table over SNMP (#90) — bsnmpd serves
# HOST-RESOURCES-MIB hrSWRunTable, the pfsense module fetches the rows named
Expand Down
Loading