fix(anomaly): skip underflowed memory samples in memory-overhead detector - #372
Open
jamby77 wants to merge 1 commit into
Open
fix(anomaly): skip underflowed memory samples in memory-overhead detector#372jamby77 wants to merge 1 commit into
jamby77 wants to merge 1 commit into
Conversation
…ctor - Guard against used_memory_overhead > used_memory, dataset values wrapped toward 2^64 and used_memory_dataset_perc > 100 — the transient non-atomic INFO accounting artifact from valkey#1373 - Leave hysteresis state untouched on skipped samples so a glitch cannot re-arm an acknowledged alert - Surface a persistent run of invalid samples (10+ consecutive polls) as a single low-severity INFO note instead of a false overhead spike Closes #369
Collaborator
Author
|
@BugBot review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit dbd434c. Configure here.
KIvanow
approved these changes
Aug 12, 2026
KIvanow
left a comment
Member
There was a problem hiding this comment.
Overall - LGTM. A few nits for future PRs:
- Diff noise: roughly a third of the anomaly.service.ts churn is unrelated Prettier reformatting that could be split out to keep the fix focused.
- Event shape: the INFO advisory reuses the overhead event with anomalyType: SPIKE and value/threshold: 0, which reads a little oddly - a dedicated advisory type would be more honest.
- Guard placement: the underflow guard runs before the pure detector's maxmemory <= 0 early-return, which is moot today since the service gates on maxmemory first, but it's a latent coupling worth a comment.
- Test gap: the new info → AnomalySeverity.INFO mapping and the logger.log advisory branch in the service aren't covered - the detector is, but that wiring isn't.
- Scope note: the canonical valkey#1373 case is a maxmemory:0 replica, which the service already short-circuits before the detector, so the real-world blast radius here is the narrower maxmemory > 0 instances that hit the same sampling race -worth setting expectations accordingly.
None of these block merge or need to be explicitly fixed (it's just a nice to have)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the
memory-overheaddetector against the valkey-io/valkey#1373 reporting artifact: INFO memory accounting is sampled non-atomically across threads, so a sample can momentarily reportused_memory_overhead > used_memory, wrappingused_memory_datasettoward 2^64 and pushingused_memory_dataset_percfar past 100%. Feeding such a sample into the detector produced bogus "huge non-dataset overhead" anomalies, most visibly on replicas.Changes
used_memory_overhead > used_memory,used_memory_dataset > used_memory, orused_memory_dataset_perc > 100. A skipped sample emits nothing and — critically — does not touch hysteresis state, so a mid-glitch "low" reading cannot re-arm an already-acknowledged alert.infofinding level toAnomalySeverity.INFO(already rendered by the web dashboard) and logs it as an advisory, not an anomaly.used_memory = dataset + overhead); the old defaults were accidentally inconsistent and would themselves trip the guard.Test plan
tsc --noEmitclean.Closes #369
Note
Low Risk
Detector-only change with conservative skip rules and INFO-level escalation for persistent inconsistency; no auth, persistence, or API surface changes.
Overview
Hardens the memory-overhead detector against the valkey#1373 INFO artifact: non-atomic memory accounting can briefly report
used_memory_overhead > used_memory, wrappingused_memory_datasetand inflatingused_memory_dataset_perc, which previously produced false “huge overhead” anomalies (often on replicas).Skipped samples when overhead exceeds total memory, dataset exceeds total memory, or dataset percent is above 100. Those polls emit nothing and do not touch hysteresis, so a glitch cannot re-arm an already-acknowledged warning.
After 10 consecutive invalid polls, a single INFO advisory notes persistent inconsistency (with valkey#1373 context).
anomaly.servicemaps that level toAnomalySeverity.INFOand logs it as an advisory rather than a warn-level anomaly.Test fixtures now default to physically consistent INFO (
used_memory = dataset + overhead); seven new unit tests cover skip conditions, hysteresis across glitches, the INFO note contract, and streak reset on a healthy sample.Reviewed by Cursor Bugbot for commit dbd434c. Bugbot is set up for automated code reviews on this repo. Configure here.