Skip to content

KAFKA-20914: Clear append and fetch purgatories when a leader resigns or becomes unattached - #23118

Open
anjy7 wants to merge 2 commits into
apache:trunkfrom
anjy7:KAFKA-3152
Open

KAFKA-20914: Clear append and fetch purgatories when a leader resigns or becomes unattached#23118
anjy7 wants to merge 2 commits into
apache:trunkfrom
anjy7:KAFKA-3152

Conversation

@anjy7

@anjy7 anjy7 commented Aug 10, 2026

Copy link
Copy Markdown

Problem

When a KafkaRaftClient leader stops being the leader, its parked requests can no longer complete normally: pending appends can't gather the acknowledgments to commit at the old epoch, and held fetches should be failed so callers retry against the new leader. Both purgatories should therefore be completed exceptionally on any leader → non-leader transition. Only onBecomeFollower did this; the other two leader-exit paths did not.

So a leader that resigns, or drops to unattached (e.g. on seeing a higher epoch with no known leader), left pending appends. And, for unattached, pending fetches, sitting in purgatory until their request timeout expired.

Impact

The parked futures eventually expire via request.timeout.ms. But until then, callers wait the full timeout for a failure that is already certain, and each pending append future keeps its CompletedBatch (and backing ByteBuffer) pinned in memory for that duration.

Testing

  • testResignWillCompleteAppendPurgatory and testTransitionToUnattachedWillCompleteAppendPurgatory
    (new): park an uncommitted append and assert the append purgatory goes from non-empty to empty
    across the transition (via a new test-only appendPurgatoryNumWaiting() accessor).
  • testTransitionToUnattachedWillCompleteFetchPurgatory (new): asserts a held fetch is completed
    with NOT_LEADER_OR_FOLLOWER on the unattached path (mirrors the existing
    testResignWillCompleteFetchPurgatory).

@github-actions github-actions Bot added triage PRs from the community kraft labels Aug 10, 2026
@anjy7 anjy7 changed the title KMETA-3152: Clear append and fetch purgatories when a leader resigns or becomes unattached KAFKA-3152: Clear append and fetch purgatories when a leader resigns or becomes unattached Aug 10, 2026
@anjy7 anjy7 changed the title KAFKA-3152: Clear append and fetch purgatories when a leader resigns or becomes unattached KAFKA-20914: Clear append and fetch purgatories when a leader resigns or becomes unattached Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kraft triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant