Skip to content

Adopt upstream's settled check in place of our replica quorum - #28

Open
usiegj00 wants to merge 1 commit into
mainfrom
adopt-upstream-settled-check
Open

usiegj00 wants to merge 1 commit into
mainfrom
adopt-upstream-settled-check

Conversation

@usiegj00

Copy link
Copy Markdown
Collaborator

Preparing the upstream PR (Saremox#201) turned up that upstream already fixed half of #23, differently and better.

Their redisPodsSettled:

if len(pods.Items) < int(rf.Spec.Redis.Replicas) { return wait(...) }
if pod.DeletionTimestamp != nil { return wait("terminating") }
if pod.Labels[revision] == updateRevision && !util.PodIsReady(pod) { return wait("not ready") }

That closes the same hole our #24 quorum gate closed, by counting pods against the expected replica count rather than inferring from whichever replicas happen to answer. Their test table even has the cases by name: a pod is being deleted, the deleted pod is not recreated yet, the recreated pod is not ready yet.

Carrying two implementations of one guard is how a fork drifts, so this takes theirs.

Changes

  • Port operator/redisfailover/util/pod.go (PodIsReady) and redisPodsSettled verbatim.
  • Drop our ready-replica quorum gate and its readyReplicas counter.
  • Replace TestUpdateRedisesPodsWaitsForReplicaQuorum with upstream's TestUpdateRedisesPodsWaitsForTheLastReplacement, plus their settledK8sServices/redisPod helpers. Adapted only where the master rows differ — a stale master is promoted away, not deleted.

What stays ours

The handover itself (#26): promote before replacing the master pod. Upstream still does DeletePod(master) unconditionally and elects afterwards. That is proposed as Saremox#201; if it lands, this file converges with upstream entirely.

Verification

Full suite green, gofmt and go vet clean. The adopted check is load-bearing: removing the redisPodsSettled call fails all pods up and a pod is being deleted.

Note we are still 24 commits behind upstream overall, including three worth having — Saremox#196 (fsGroup on the pod not the container, the warning on every helm upgrade), Saremox#182 (disconnect a demoted master's clients), Saremox#193 (don't promote while a ready pod doesn't answer). A full sync is a separate piece of work.

#24 added a ready-replica quorum gate because the readiness loop only
iterates pods that are already Running, so a replica being recreated was
invisible and the gate passed on an empty set.

Upstream solved the same problem in redisPodsSettled, and solved it better:
it counts pods against spec.redis.replicas and waits while any is
terminating, rather than inferring from whichever replicas happen to answer.

Carrying two implementations of one guard is how a fork drifts, so take
theirs. Port util.PodIsReady and redisPodsSettled verbatim, drop our quorum
gate, and replace our bespoke test with upstream's table test - adapted only
where the master rows differ, since a stale master is now promoted away
rather than deleted.

What remains ours is the handover itself (#26), which is proposed upstream
as Saremox#201. If that lands, this file converges.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.26087% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
operator/redisfailover/util/pod.go 0.00% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants