Do not replace the master until a quorum of replicas is ready (#23)

[usiegj00]

Fixes #23.

In operator-managed failover mode a rolling update replaced the master before the
replacement replica had synced, leaving the cluster with no master for tens of
seconds on any routine pod-spec change.

UpdateRedisesPods gated the replacement on a loop over GetRedisesIPs, which
only reports pods whose phase is already Running. While the replacement replica
is terminating or being recreated it is absent from that list, so the loop had
nothing to iterate and "no replica reported unready" was trivially true. The gate
passed on an empty set and the master was deleted straight after.

Sentinel deployments never saw this: CheckSentinelSlavesNumberQuorumInMemory
gates the master delete on sentinel's own view of the replicas, which lags pod
creation and does block. Operator-managed mode skips that gate, so the replica
count has to be checked here instead.

The fix

Count ready replicas and require a quorum before replacing any pod. A replica
that is missing now blocks the replacement instead of silently satisfying it.
Quorum rather than the full count mirrors the sentinel gate, so one permanently
unavailable replica cannot block a rollout forever.

Scoped to OperatorManagedFailover(), so sentinel behaviour is unchanged.

Measured before and after

The rollout test previously only checked the end state, which converges either
way. It now samples master availability every 500ms for the duration of the
rollout and records the longest run with no master at all.

The first commit adds only that measurement, so CI recorded the broken behaviour
before the fix landed:

k8s	before (b90165eb)	after (fd323578)
1.35.8	0 masterless — pass	554 samples, 0 masterless (0s)
1.36.4	0 masterless — pass	544 samples, 0 masterless (0s)
1.37.0	164 samples, 118 masterless (59s) — fail	525 samples, 0 masterless (0s)

It reproduced on 1 of 3 versions, which is expected for a race between the master
delete and the replacement replica appearing in GetRedisesIPs. The 59s measured
here matches a 28-83s window observed on a production cluster.

No-op for sentinel

The unit test covers both modes from an identical fixture: with replicas missing
entirely, operator-managed mode now keeps the master, while sentinel mode is
unchanged and still reaches its own quorum gate.

An earlier attempt applied the gate unconditionally and broke ~20 tests, every one
of them a sentinel test (generateRF enables sentinel). After scoping, all of
them pass untouched.

Notes for reviewers

• Rollouts are slower now, by design: replicas are replaced one at a time, each
  syncing before the next pod is touched, as described in
  upgrade redisfailover cluster spotahome/redis-operator#637. The rollout test's wait went from 5 to 10 minutes
  to accommodate it, and its timeout error now reports how many pods were
  recreated so a slow rollout is distinguishable from a wedged one.
• Three operator-managed unit fixtures reported only the master IP and expected it
  to be replaced regardless. They described a cluster with nothing left to
  promote, so they now model the master plus its ready replicas.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
operator/redisfailover/checker.go	84.61%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!