Skip to content

feat(anomaly): extend config-drift to encoding-threshold configs (valkey#3479) - #374

Merged
jamby77 merged 2 commits into
masterfrom
feature/370-config-drift-encoding-thresholds
Aug 13, 2026
Merged

feat(anomaly): extend config-drift to encoding-threshold configs (valkey#3479)#374
jamby77 merged 2 commits into
masterfrom
feature/370-config-drift-encoding-thresholds

Conversation

@jamby77

@jamby77 jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Extends the existing config-drift detector (valkey#1193) to the encoding-threshold configs from valkey-io/valkey#3479. When these diverge across a replication group, the same data silently re-encodes on the divergent node after an RDB reload, restore, or clone (e.g. listpack promoted to hashtable/skiplist), occupying far more memory than the source needed — upstream's "retain encodings during RDB reload" knob is unshipped, but the config divergence that triggers the bloat is observable today.

Per the issue's guardrails this is an extension, not a parallel detector: it reuses the per-node CONFIG snapshot, grouping, signature dedupe, and partial-fetch retention logic unchanged.

Changes

  • New labeled subgroup ENCODING_THRESHOLD_CONFIG_DRIFT_KEYS (8 keys): hash-max-listpack-entries/-value, list-max-listpack-size, set-max-intset-entries, set-max-listpack-entries/-value, zset-max-listpack-entries/-value — kept separate from DEFAULT_CONFIG_DRIFT_KEYS so the advisory can explain the encoding-specific consequence.
  • isEncodingThresholdKey() classifier; the service's drift advisory branches on it: encoding keys get the memory-re-encoding rationale (valkey#3479), everything else keeps the existing valkey#1193 reconcile message.
  • The service now fetches and compares the combined list. Partial CONFIG fetches retain prior snapshot values exactly as before (merge logic untouched).

Test plan

  • 5 new unit tests: divergence on an encoding key fires and names the key, matched thresholds stay silent, a node that failed to report a key is excluded rather than treated as drifted, classifier covers exactly the subgroup (and none of the default keys), and the subgroup spans all four collection types.
  • Full anomaly-detection suite: 503/503 green; tsc --noEmit clean.

Closes #370


Note

Medium Risk
Changes cross-node config monitoring and alert copy; throttling could delay detection of newly changed keys for up to one recheck window, though replid changes bypass the cache.

Overview
Extends config drift detection with eight encoding-threshold keys (valkey#3479) and adds throttled CONFIG reads for the drift allowlist while keeping per-poll drift evaluation.

Encoding thresholds: New ENCODING_THRESHOLD_CONFIG_DRIFT_KEYS and isEncodingThresholdKey(); the anomaly service fetches and compares them alongside the existing default keys. Drift advisories for those keys use a dedicated message about silent re-encoding and memory bloat after RDB reload/restore/clone, instead of the generic valkey#1193 reconcile text.

Snapshot refresh: Config snapshot updates move into refreshConfigSnapshot() with a per-connection configDriftRecheck countdown (CONFIG_DRIFT_RECHECK_POLLS = 60) so the full allowlist is not CONFIG GET on every poll. Drift is still scanned every poll from cached snapshots; a replid change forces an immediate re-read. Connection teardown clears configDriftRecheck.

Tests: Unit coverage for encoding-threshold drift/classification; service tests for throttle behavior, peer mismatch on cached snapshot, failover replid bypass, and expireConfigRecheck for scenarios that need a fresh read within the recheck window.

Reviewed by Cursor Bugbot for commit d046591. Bugbot is set up for automated code reviews on this repo. Configure here.

@jamby77

jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

@claude @cursoragent review

@jamby77

jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

@BugBot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit c6275f6. Configure here.

@jamby77
jamby77 requested a review from KIvanow August 11, 2026 09:42

@KIvanow KIvanow left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doubles the per-poll CONFIG GET volume (7→15 commands per connection, every poll) with no throttle — for config that basically never changes. Could fold it into a single multi-arg CONFIG GET a b c … (Valkey 7.0+), or add a slow-recheck countdown like the large-reply/COB threshold path already does. Won't that work?

…key#3479)

- Add the eight hash/list/set/zset listpack/intset threshold configs as
  a labeled encoding-threshold subgroup of the drift allowlist
- Divergence advisory explains the specific harm: the same data
  re-encodes when loaded under different thresholds, so an RDB reload,
  restore or clone onto the divergent node silently bloats memory
- Non-encoding keys keep the existing valkey#1193 reconcile message
- Cache each connection's curated config snapshot and re-read it only
  every CONFIG_DRIFT_RECHECK_POLLS polls, matching the COB-limit and
  large-reply threshold caches, so the allowlist no longer costs a
  CONFIG GET per key on every poll
- Force an immediate re-read when master_replid changes, so groupKey is
  never rewritten without a matching fresh config read
- Evaluate drift on every poll regardless, so a newly registered peer
  still surfaces an existing mismatch on its first poll
@jamby77
jamby77 force-pushed the feature/370-config-drift-encoding-thresholds branch from c6275f6 to d046591 Compare August 13, 2026 06:30
@jamby77

jamby77 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Good catch — throttled it rather than folding into a multi-arg CONFIG GET, since the recheck-countdown path already exists here (cobLimitRecheck / largeReplyThresholdRecheck) and works on any server version, whereas multi-arg CONFIG GET would need a 7.0+ capability check plus a per-key fallback for older nodes.

refreshConfigSnapshot now caches each connection's curated snapshot and re-reads it only every CONFIG_DRIFT_RECHECK_POLLS (60), so amortised cost is 15 CONFIG GETs per 60 polls — below the pre-PR baseline of 7 per poll.

Two things worth flagging:

  • Drift is still evaluated every poll; only the fetch is throttled. A newly registered peer surfaces an existing mismatch on its first poll rather than waiting out someone else's window.
  • A master_replid change forces an immediate re-read, so groupKey is never rewritten without a matching fresh config read — otherwise a failover could park a node in a new group for up to a window while serving cached config, which is exactly the re-fire the fetchedAny guard exists to prevent.

Three tests added for the new behaviour (window reuse, evaluation-while-throttled, replid-change refetch); the two existing tests that simulate observing a later config value now expire the countdown explicitly, since adjacent polls in a test stand in for reads minutes apart in the field.

@jamby77
jamby77 merged commit d192a2e into master Aug 13, 2026
3 checks passed
@jamby77
jamby77 deleted the feature/370-config-drift-encoding-thresholds branch August 13, 2026 07:31
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend config-drift to encoding-threshold configs

2 participants