KAFKA-20914: Clear append and fetch purgatories when a leader resigns or becomes unattached - #23118
Open
anjy7 wants to merge 2 commits into
Open
KAFKA-20914: Clear append and fetch purgatories when a leader resigns or becomes unattached#23118anjy7 wants to merge 2 commits into
anjy7 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a
KafkaRaftClientleader stops being the leader, its parked requests can no longer complete normally: pending appends can't gather the acknowledgments to commit at the old epoch, and held fetches should be failed so callers retry against the new leader. Both purgatories should therefore be completed exceptionally on any leader → non-leader transition. OnlyonBecomeFollowerdid this; the other two leader-exit paths did not.So a leader that resigns, or drops to unattached (e.g. on seeing a higher epoch with no known leader), left pending appends. And, for unattached, pending fetches, sitting in purgatory until their request timeout expired.
Impact
The parked futures eventually expire via
request.timeout.ms. But until then, callers wait the full timeout for a failure that is already certain, and each pending append future keeps itsCompletedBatch(and backingByteBuffer) pinned in memory for that duration.Testing
testResignWillCompleteAppendPurgatoryandtestTransitionToUnattachedWillCompleteAppendPurgatory(new): park an uncommitted append and assert the append purgatory goes from non-empty to empty
across the transition (via a new test-only
appendPurgatoryNumWaiting()accessor).testTransitionToUnattachedWillCompleteFetchPurgatory(new): asserts a held fetch is completedwith
NOT_LEADER_OR_FOLLOWERon the unattached path (mirrors the existingtestResignWillCompleteFetchPurgatory).