Skip to content

fix(anomaly): skip underflowed memory samples in memory-overhead detector - #372

Open
jamby77 wants to merge 1 commit into
masterfrom
bugfix/369-memory-overhead-underflow-guard
Open

fix(anomaly): skip underflowed memory samples in memory-overhead detector#372
jamby77 wants to merge 1 commit into
masterfrom
bugfix/369-memory-overhead-underflow-guard

Conversation

@jamby77

@jamby77 jamby77 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Hardens the memory-overhead detector against the valkey-io/valkey#1373 reporting artifact: INFO memory accounting is sampled non-atomically across threads, so a sample can momentarily report used_memory_overhead > used_memory, wrapping used_memory_dataset toward 2^64 and pushing used_memory_dataset_perc far past 100%. Feeding such a sample into the detector produced bogus "huge non-dataset overhead" anomalies, most visibly on replicas.

Changes

  • Skip a sample when any of: used_memory_overhead > used_memory, used_memory_dataset > used_memory, or used_memory_dataset_perc > 100. A skipped sample emits nothing and — critically — does not touch hysteresis state, so a mid-glitch "low" reading cannot re-arm an already-acknowledged alert.
  • Track consecutive skipped samples; a run of 10+ polls surfaces once as a low-severity INFO note ("memory accounting looks inconsistent", pointing at valkey#1373) rather than a false overhead spike. Upstream describes the artifact as gone by the next INFO read, so a run that long is genuinely worth a look.
  • Service maps the new info finding level to AnomalySeverity.INFO (already rendered by the web dashboard) and logs it as an advisory, not an anomaly.
  • Test fixture now models physically consistent INFO samples (used_memory = dataset + overhead); the old defaults were accidentally inconsistent and would themselves trip the guard.

Test plan

  • 7 new unit tests: each skip condition in isolation, normal-sample passthrough, hysteresis protection across a glitch, the persistent-run INFO note (emit-retry + acknowledge contract), and streak reset on a healthy sample.
  • Full anomaly-detection suite: 505/505 green; tsc --noEmit clean.

Closes #369


Note

Low Risk
Detector-only change with conservative skip rules and INFO-level escalation for persistent inconsistency; no auth, persistence, or API surface changes.

Overview
Hardens the memory-overhead detector against the valkey#1373 INFO artifact: non-atomic memory accounting can briefly report used_memory_overhead > used_memory, wrapping used_memory_dataset and inflating used_memory_dataset_perc, which previously produced false “huge overhead” anomalies (often on replicas).

Skipped samples when overhead exceeds total memory, dataset exceeds total memory, or dataset percent is above 100. Those polls emit nothing and do not touch hysteresis, so a glitch cannot re-arm an already-acknowledged warning.

After 10 consecutive invalid polls, a single INFO advisory notes persistent inconsistency (with valkey#1373 context). anomaly.service maps that level to AnomalySeverity.INFO and logs it as an advisory rather than a warn-level anomaly.

Test fixtures now default to physically consistent INFO (used_memory = dataset + overhead); seven new unit tests cover skip conditions, hysteresis across glitches, the INFO note contract, and streak reset on a healthy sample.

Reviewed by Cursor Bugbot for commit dbd434c. Bugbot is set up for automated code reviews on this repo. Configure here.

…ctor

- Guard against used_memory_overhead > used_memory, dataset values
  wrapped toward 2^64 and used_memory_dataset_perc > 100 — the transient
  non-atomic INFO accounting artifact from valkey#1373
- Leave hysteresis state untouched on skipped samples so a glitch cannot
  re-arm an acknowledged alert
- Surface a persistent run of invalid samples (10+ consecutive polls) as
  a single low-severity INFO note instead of a false overhead spike

Closes #369
@jamby77

jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

@BugBot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit dbd434c. Configure here.

@jamby77
jamby77 requested a review from KIvanow August 11, 2026 05:39

@KIvanow KIvanow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall - LGTM. A few nits for future PRs:

  • Diff noise: roughly a third of the anomaly.service.ts churn is unrelated Prettier reformatting that could be split out to keep the fix focused.
  • Event shape: the INFO advisory reuses the overhead event with anomalyType: SPIKE and value/threshold: 0, which reads a little oddly - a dedicated advisory type would be more honest.
  • Guard placement: the underflow guard runs before the pure detector's maxmemory <= 0 early-return, which is moot today since the service gates on maxmemory first, but it's a latent coupling worth a comment.
  • Test gap: the new info → AnomalySeverity.INFO mapping and the logger.log advisory branch in the service aren't covered - the detector is, but that wiring isn't.
  • Scope note: the canonical valkey#1373 case is a maxmemory:0 replica, which the service already short-circuits before the detector, so the real-world blast radius here is the narrower maxmemory > 0 instances that hit the same sampling race -worth setting expectations accordingly.

None of these block merge or need to be explicitly fixed (it's just a nice to have)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guard memory-overhead detector against used_memory_dataset underflow

2 participants