feat(anomaly): extend config-drift to encoding-threshold configs (valkey#3479) - #374
Conversation
|
@claude @cursoragent review |
|
@BugBot review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c6275f6. Configure here.
KIvanow
left a comment
There was a problem hiding this comment.
this doubles the per-poll CONFIG GET volume (7→15 commands per connection, every poll) with no throttle — for config that basically never changes. Could fold it into a single multi-arg CONFIG GET a b c … (Valkey 7.0+), or add a slow-recheck countdown like the large-reply/COB threshold path already does. Won't that work?
…key#3479) - Add the eight hash/list/set/zset listpack/intset threshold configs as a labeled encoding-threshold subgroup of the drift allowlist - Divergence advisory explains the specific harm: the same data re-encodes when loaded under different thresholds, so an RDB reload, restore or clone onto the divergent node silently bloats memory - Non-encoding keys keep the existing valkey#1193 reconcile message
- Cache each connection's curated config snapshot and re-read it only every CONFIG_DRIFT_RECHECK_POLLS polls, matching the COB-limit and large-reply threshold caches, so the allowlist no longer costs a CONFIG GET per key on every poll - Force an immediate re-read when master_replid changes, so groupKey is never rewritten without a matching fresh config read - Evaluate drift on every poll regardless, so a newly registered peer still surfaces an existing mismatch on its first poll
c6275f6 to
d046591
Compare
|
Good catch — throttled it rather than folding into a multi-arg
Two things worth flagging:
Three tests added for the new behaviour (window reuse, evaluation-while-throttled, replid-change refetch); the two existing tests that simulate observing a later config value now expire the countdown explicitly, since adjacent polls in a test stand in for reads minutes apart in the field. |
Summary
Extends the existing
config-driftdetector (valkey#1193) to the encoding-threshold configs from valkey-io/valkey#3479. When these diverge across a replication group, the same data silently re-encodes on the divergent node after an RDB reload, restore, or clone (e.g. listpack promoted to hashtable/skiplist), occupying far more memory than the source needed — upstream's "retain encodings during RDB reload" knob is unshipped, but the config divergence that triggers the bloat is observable today.Per the issue's guardrails this is an extension, not a parallel detector: it reuses the per-node CONFIG snapshot, grouping, signature dedupe, and partial-fetch retention logic unchanged.
Changes
ENCODING_THRESHOLD_CONFIG_DRIFT_KEYS(8 keys):hash-max-listpack-entries/-value,list-max-listpack-size,set-max-intset-entries,set-max-listpack-entries/-value,zset-max-listpack-entries/-value— kept separate fromDEFAULT_CONFIG_DRIFT_KEYSso the advisory can explain the encoding-specific consequence.isEncodingThresholdKey()classifier; the service's drift advisory branches on it: encoding keys get the memory-re-encoding rationale (valkey#3479), everything else keeps the existing valkey#1193 reconcile message.Test plan
tsc --noEmitclean.Closes #370
Note
Medium Risk
Changes cross-node config monitoring and alert copy; throttling could delay detection of newly changed keys for up to one recheck window, though replid changes bypass the cache.
Overview
Extends config drift detection with eight encoding-threshold keys (valkey#3479) and adds throttled CONFIG reads for the drift allowlist while keeping per-poll drift evaluation.
Encoding thresholds: New
ENCODING_THRESHOLD_CONFIG_DRIFT_KEYSandisEncodingThresholdKey(); the anomaly service fetches and compares them alongside the existing default keys. Drift advisories for those keys use a dedicated message about silent re-encoding and memory bloat after RDB reload/restore/clone, instead of the generic valkey#1193 reconcile text.Snapshot refresh: Config snapshot updates move into
refreshConfigSnapshot()with a per-connectionconfigDriftRecheckcountdown (CONFIG_DRIFT_RECHECK_POLLS = 60) so the full allowlist is not CONFIG GET on every poll. Drift is still scanned every poll from cached snapshots; a replid change forces an immediate re-read. Connection teardown clearsconfigDriftRecheck.Tests: Unit coverage for encoding-threshold drift/classification; service tests for throttle behavior, peer mismatch on cached snapshot, failover replid bypass, and
expireConfigRecheckfor scenarios that need a fresh read within the recheck window.Reviewed by Cursor Bugbot for commit d046591. Bugbot is set up for automated code reviews on this repo. Configure here.