feat(anomaly): detect replica full-resync failure loop (valkey#1836)

[jamby77]

Summary

Extends the stuck-replica module with the full-resync failure-loop signature from valkey-io/valkey#1836: a replica that connects, starts a full sync, fails to load the transferred RDB (disk errors, corruption, permissions, out-of-space, a filesystem/AV layer mangling the transfer), disconnects and retries forever. master_link_status never leaves down, so the replica serves stale data indefinitely while looking "merely syncing" at any single glance.

Unlike the existing CLUSTER NODES snapshot detector in the module, this evaluator folds INFO replication/stats fields from the polled replica across polls, so it works in standalone and cluster mode alike.

Detection discipline (per the issue's guardrails)

• Fires only after the link has been continuously down for 5 minutes AND ≥ 2 failed full-sync cycles were observed — a normal first sync of a large dataset is one attempt still progressing and stays silent.
• Failed cycles are counted from sync_full deltas, falling back to master_sync_in_progress toggling off while the link stays down (for replicas whose own counter never moves).
• A drop in master_link_down_since_seconds means the link DID reach up between polls (a sync completed) — the window restarts rather than firing.
• Recovery (link up / promotion / non-replica) clears all state, so a later loop earns the window afresh and alerts again.
• Same emit-retry contract as the module's other stateful detectors: the finding is re-produced until the emit succeeds, acknowledged only after.

Wiring

• New RESYNC_LOOP metric type (state-based, added to METRICS_HANDLED_OUTSIDE_EXTRACTOR), WARNING severity, distinct message pointing at replica-side disk/permissions/RDB-integrity checks.
• Called from the poll loop next to the COB-pressure detector (its replica-side complement); per-connection state cleaned up on connection removal.
• Dashboard metric label: "Replica Resync Loop".

Test plan

• 8 new unit tests covering the issue's acceptance criteria: healthy first sync of a large dataset (silent), genuine loop via sync_full (fires past window + cycles), loop via the in-progress toggle, silent before the window despite cycles, recovery clears + re-alerts on a fresh loop, emit-retry/acknowledge contract, non-replica reset, and the down-counter-drop window restart.
• Full anomaly-detection suite: 506/506 green; tsc --noEmit clean for api and web.

Closes #371

---

Note

Medium Risk
New replication anomaly path on every poll for replica connections; logic is gated and well-tested, but false positives during flaky syncs could add noise until the window and cycle thresholds are met.

Overview
Adds state-based detection for replicas stuck in a full-resync failure loop (valkey#1836): master_link_status stays down while full-sync attempts start and fail repeatedly, so the replica can keep serving stale data.

The new evaluateResyncLoop logic in stuck-replica-detector tracks INFO replication across polls (standalone or cluster). It only fires after a 5-minute continuous link-down window and ≥2 failed cycles (sync in-progress ends while the link is still down). A long first sync that never drops in-progress stays silent; a drop in master_link_down_since_seconds resets the window. Recovery or promotion clears state so a later loop can alert again, with the same emit-retry / acknowledge pattern as other stateful detectors.

AnomalyService wires detectResyncLoop on each poll (beside COB pressure), registers MetricType.RESYNC_LOOP outside the z-score extractor, and cleans per-connection state on disconnect. The anomaly dashboard labels the metric Replica Resync Loop.

Eight unit tests cover healthy long sync, genuine loops, timing gates, recovery/re-alert, acknowledge dedupe, non-replica reset, and down-counter restart. Remaining diff in the dashboard file is mostly formatting.

^{Reviewed by Cursor Bugbot for commit 1ec3e30. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jamby77]

@BugBot review

[jamby77]

@claude review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1ec3e30. Configure here.}

[KIvanow]

Looks good! Most/some of the nitpicks from #372 are applicable here as well, so I won't repeat them