Alert on a sensor that stops logging: absence rules for Suricata and Zeek

[Gerrrt]

SuricataStopped reads the firewall's process table over SNMP, and
docs/security.md is careful about what that buys: "It proves the process is
alive, not that it is inspecting anything."

There is a gap under that sentence. A Suricata that is running but has stopped
emitting — bound to the wrong interface, a hung capture thread, a rule reload
that produced an empty set — reads identically to a healthy one from SNMP, and
no Loki rule watches for its silence either.

The estate already has this shape solved twice, in FirewallLogsStopped and
DhcpLeaseLogsStopped, both absent_over_time on the same syslog pipe. The IDS
stream is the one that was missed.

What to add

• SuricataLogsStopped — absent_over_time({app="suricata"}[<window>]). This
  half can land now, independently of anything else.
• ZeekLogsStopped — the same shape against the lab's Loki, once
  #437 exists.

The window needs measuring, not choosing

Suricata on a quiet segment is legitimately silent for long stretches — Skids and
Degens are terminal, and a quiet night is normal. Thirty minutes is a starting
guess, and the honest way to set it is to read the actual inter-alert gap over
the retention already in Loki before committing to a number.

Otherwise this becomes an alert that fires at 3 a.m. because nobody was
browsing, and gets muted within a week — which is the outcome
#166 wrote down and designed
around.

What this can and cannot prove

Unit tests for both, per the standing rule that a stack with rules and no
promtool tests is a CI failure. But Loki rules still have no unit-test harness
— #234 records this — so these
are proven to parse and evaluate, not to fire. Worth saying in the PR rather
than implying otherwise.

Purchases this needs

None.

[Gerrrt]

Closing this. Both halves are settled — one built, one decided to belong elsewhere.

The Suricata half is done, and the window was re-read today

SuricataLogsStopped landed on 2026-09-12 in #459: absent_over_time({app="suricata"}[9h]), aggregate across both interfaces. The live ruler reports it health=ok, state=inactive.

The window was measured rather than chosen, which is what this issue asked for. Re-measured today over the full current retention — 27.7 days to 2026-09-17 22:25 UTC, five-minute buckets — it holds:

stream	span	gaps >1h	gaps >9h	longest
both interfaces (what the rule sees)	27.7 d	6	0	1.33 h
igc0.20 (Skids)	15.4 d	3	0	1.25 h
igc0.10 (Degens)	13.7 d	30	7	47.5 h

The worst aggregate silence grew from 75 to 80 minutes in five weeks, so nine hours keeps nearly seven times the headroom DhcpLeaseLogsStopped settled on. And the aggregate design earned its keep in the interval: igc0.10 alone crossed nine hours seven times, so the thirty-minute per-interface rule this issue opened with would have paged seven times in a fortnight — the outcome #166 wrote down and this issue asked to avoid.

One number is deliberately not compared: 135 gaps over thirty minutes here against 54 on 2026-09-12. That is bucket alignment and which streams were summed, not a change in Suricata; the long tail, which is the half that sets the window, agrees. Worth stating rather than presenting a single series.

The Zeek half is #437's, and it is a metric, not a Loki rule

This issue says ZeekLogsStopped is "the same shape against the lab's Loki, once #437 exists". #437 existing is not sufficient, and would not make it the right instrument:

• Run Zeek on a mirror port, and see the VLAN 30 traffic the firewall never will #437 puts Zeek's logs on alexander and never on 10.0.99.20 — its own body says so, ADR-0007 requires it, and ADR-0039 reaffirms it by keeping the mirror on Saruman's own bridge rather than a SPAN from neo.
• The lab's Loki has no ruler and no Alertmanager, on purpose. ADR-0020 decided "No Alertmanager inside the stack… nothing in the lab pages", and stacks/lab/loki/loki-config.yaml spells out the consequence: "a ruler with nowhere to deliver evaluates rules and discards the result, which reads as coverage and is not." There is no stacks/lab/loki/rules/ directory at all.
• ADR-0028 revisited exactly this boundary and kept it — guest liveness crosses, guest telemetry does not, and it names what stays uncovered.

A ZeekLogsStopped written against {app="zeek"} would also be absent_over_time over a stream that has never existed: permanently firing from the moment it loads, which is the #63/#305 defect class.

#437 already carries the right mechanism. Its body specifies a homelab_zeek_mirror_active textfile gauge, read on the hypervisor by the agent already there, crossing as hypervisor state under ADR-0028 — precisely because a mirror port does not survive a reboot. That is absence-detection for Zeek, built from metrics, inside the issue that builds Zeek. Commented on #437 to say it now owns this.

So this issue ends here rather than holding a slot behind a Wave 2 dependency that itself waits on #414.

Recorded in #489

docs/security.md now states the limit next to the three it already states; the ids block comment carries both measurements; and docs/roadmap.md gets a Done entry, which #441 had none of.

What this proves and what it does not: Loki rules still have no unit-test harness, so check_loki_rules.sh shows the rule parses and the ruler evaluates it — not that it fires. The runbook's plaintext test alert is still the only proof of detection.