KAFKA-20599: Improve retry logic in KafkaStatusBackingStore with exponential backoff by akhil-sabu-ibm · Pull Request #22324 · apache/kafka

akhil-sabu-ibm · 2026-05-19T09:24:45Z

This PR improves the retry logic in KafkaStatusBackingStore to address
the TODO comment at line 283.

Problem

Previously, the code retried indefinitely without backoff when
encountering RetriableException, which could:

Exhaust system resources during prolonged outages
Overwhelm Kafka brokers with rapid retry attempts
Lack visibility into retry behavior

Solution

Implemented graceful retry mechanism with:

Exponential backoff: 300ms → 600ms → 1.2s → 2.4s → ... → 60s (max)
Retry limit: Maximum 10 attempts before giving up
Enhanced logging: Tracks retry attempts, backoff times, and final
outcomes
Resource protection: Prevents infinite loops while maintaining
resilience

Changes

Added retry configuration constants (MAX_RETRY_ATTEMPTS=10,
INITIAL_RETRY_BACKOFF_MS=300, MAX_RETRY_BACKOFF_MS=60000)
Implemented sendWithRetry() methods with exponential backoff for
both topic status and general status updates
Implemented calculateBackoff() using formula: min(INITIAL_BACKOFF * 2^attempt, MAX_BACKOFF)
Updated sendTopicStatus() and send() methods to use new retry
logic
Added comprehensive JavaDoc documentation
Preserved existing safe-write checks and generation validation

Testing

All 24 existing tests pass without modification
Compilation successful with Java 21
Checkstyle and SpotBugs checks passed
Maintains backward compatibility
No breaking changes

Benefits

Prevents resource exhaustion during Kafka broker outages
Reduces load on brokers with exponential backoff
Improves observability with detailed retry logging
Aligns with best practices used in other Kafka Connect components
(e.g., RetryUtil, DistributedHerder)

JIRA: https://issues.apache.org/jira/browse/KAFKA-20599

This is an improvement to system reliability with no breaking changes.

Reviewers: Mickael Maison mickael.maison@gmail.com

github-actions · 2026-05-27T04:35:31Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

mimaison

Thanks for the PR. I left a few comments

mimaison · 2026-06-18T12:46:38Z

+                    sendRetryExecutor.submit(() -> {
+                        time.sleep(backoffMs);


Shouldn't we use a ScheduledExecutor instead of doing sleep()?

mimaison · 2026-06-18T12:48:23Z

+                    long backoffMs = calculateBackoff(attemptNumber);
+                    log.warn("Failed to write status update for key {} (attempt {}/{}). " +
+                            "Retrying after {}ms. Reason: {}",
+                            key, attemptNumber + 1, MAX_RETRY_ATTEMPTS + 1, backoffMs, exception.getMessage());


The message does not seem correct. The first attempt will print 1/11 and the last one will print 10/11, shouldn't it be 1/10 to 10/10?

mimaison · 2026-06-18T12:49:47Z

+                    if (attemptNumber >= MAX_RETRY_ATTEMPTS) {
+                        log.error("Failed to write status update for key {} after {} attempts. Giving up.",
+                                key, attemptNumber + 1, exception);
+                        return;
+                    }


I'm uneasy with completely giving up and only logging it, especially on a RetriableException

1. Use ScheduledExecutorService instead of ExecutorService with sleep() - Changed sendRetryExecutor to ScheduledExecutorService - Use schedule() method for delayed execution instead of blocking with sleep() - More efficient and cleaner implementation 2. Fix confusing retry attempt log messages - Simplified messages to show 'attempt X' instead of 'X/Y' - Clear message after MAX_RETRY_ATTEMPTS indicating continued retries 3. Continue retrying indefinitely on RetriableException - Removed logic that gave up after MAX_RETRY_ATTEMPTS - RetriableException indicates temporary failures that should eventually succeed - Backoff is capped at MAX_RETRY_BACKOFF_MS after MAX_RETRY_ATTEMPTS - Existing safeguards (stale checks, generation checks) prevent issues

akhil-sabu-ibm · 2026-06-22T10:28:58Z

@mimaison The PR has been updated with the changes. Could you please review?

mimaison

Thanks for the updates. I made another pass and left another question. I think it would also be nice to have tests for the new logic.

mimaison · 2026-07-06T13:23:54Z

                if (exception instanceof RetriableException) {
-                    sendRetryExecutor.submit(() -> kafkaLog.send(key, value, this));
+                    long backoffMs = calculateBackoff(attemptNumber);
+                    if (attemptNumber < MAX_RETRY_ATTEMPTS) {


Is MAX_RETRY_ATTEMPTS correctly named? It seems this keeps retrying even after MAX_RETRY_ATTEMPTS.

@mimaison The PR has been updated with the changes. Could you please review?

mimaison

Thanks for the updates! I took another look and left a few more comments.

Also you need to rebase your branch on trunk to resolve the CI issues.

mimaison · 2026-07-28T15:37:00Z

+    // Retry configuration constants
+    // After this many attempts, backoff will be capped at MAX_RETRY_BACKOFF_MS
+    private static final int BACKOFF_ESCALATION_THRESHOLD = 10;
+    private static final long INITIAL_RETRY_BACKOFF_MS = 300;
+    private static final long MAX_RETRY_BACKOFF_MS = 60000; // 60 seconds


Rather than rolling a custom exponential backoff, could we reuse org.apache.kafka.common.utils.internals.ExponentialBackoff? For example it's already used in DistributedHerder.

mimaison · 2026-07-28T15:39:19Z

+                    if (attemptNumber < BACKOFF_ESCALATION_THRESHOLD) {
+                        log.warn("Failed to write status update for key {} (attempt {}). " +
+                                "Retrying after {}ms. Reason: {}",
+                                key, attemptNumber + 1, backoffMs, exception.getMessage());
+                    } else {
+                        log.warn("Failed to write status update for key {} after {} attempts. " +
+                                "Will continue retrying with {}ms backoff. Reason: {}",
+                                key, attemptNumber + 1, backoffMs, exception.getMessage());
+                    }


I'm not sure having different messages is worth it.

@mimaison The PR has been updated with the changes. Could you please review?

mimaison · 2026-08-07T13:15:04Z

I think you need to rebase your branch on trunk to resolve the CI issues.

1. Use ScheduledExecutorService instead of ExecutorService with sleep() - Changed sendRetryExecutor to ScheduledExecutorService - Use schedule() method for delayed execution instead of blocking with sleep() - More efficient and cleaner implementation 2. Fix confusing retry attempt log messages - Simplified messages to show 'attempt X' instead of 'X/Y' - Clear message after MAX_RETRY_ATTEMPTS indicating continued retries 3. Continue retrying indefinitely on RetriableException - Removed logic that gave up after MAX_RETRY_ATTEMPTS - RetriableException indicates temporary failures that should eventually succeed - Backoff is capped at MAX_RETRY_BACKOFF_MS after MAX_RETRY_ATTEMPTS - Existing safeguards (stale checks, generation checks) prevent issues

…nential backoff - Implemented exponential backoff for retry attempts (300ms initial, 60s max) - Added maximum retry attempt limit (10 attempts) - Added proper logging for retry attempts and failures - Prevents infinite retry loops during prolonged outages - Improved observability with structured logging The retry logic now uses exponential backoff starting at 300ms and doubling with each attempt up to a maximum of 60 seconds. After 10 failed attempts, the system will give up and log an error. This prevents resource exhaustion during prolonged Kafka broker outages while still providing resilience for transient failures.

1. Use ScheduledExecutorService instead of ExecutorService with sleep() - Changed sendRetryExecutor to ScheduledExecutorService - Use schedule() method for delayed execution instead of blocking with sleep() - More efficient and cleaner implementation 2. Fix confusing retry attempt log messages - Simplified messages to show 'attempt X' instead of 'X/Y' - Clear message after MAX_RETRY_ATTEMPTS indicating continued retries 3. Continue retrying indefinitely on RetriableException - Removed logic that gave up after MAX_RETRY_ATTEMPTS - RetriableException indicates temporary failures that should eventually succeed - Backoff is capped at MAX_RETRY_BACKOFF_MS after MAX_RETRY_ATTEMPTS - Existing safeguards (stale checks, generation checks) prevent issues

The constant name was misleading since retries continue indefinitely after this threshold. The new name better reflects its actual purpose: controlling when the exponential backoff reaches its maximum value. After BACKOFF_ESCALATION_THRESHOLD attempts, the backoff is capped at MAX_RETRY_BACKOFF_MS, but retries continue indefinitely for RetriableException.

- Replace custom calculateBackoff() + three constants with the existing ExponentialBackoff utility class (already used in DistributedHerder), as suggested by reviewer mimaison - Merge the two separate log.warn branches (below/above threshold) into a single unified message, as suggested by reviewer mimaison

akhil-sabu-ibm · 2026-08-11T14:55:24Z

@mimaison I have rebased the branch

github-actions Bot added connect triage PRs from the community labels May 19, 2026

sjhajharia added the ci-approved label May 19, 2026

github-actions Bot added the needs-attention label May 27, 2026

mimaison reviewed Jun 18, 2026

View reviewed changes

github-actions Bot removed needs-attention triage PRs from the community labels Jun 19, 2026

mimaison reviewed Jul 6, 2026

View reviewed changes

mimaison reviewed Jul 28, 2026

View reviewed changes

github-actions Bot added the small Small PRs label Aug 5, 2026

akhil-sabu-ibm and others added 4 commits August 8, 2026 01:33

akhil-sabu-ibm force-pushed the KAFKA-20599-improve-status-store-retry-logic branch from da6fb0e to 651f588 Compare August 7, 2026 20:04

akhil-sabu-ibm closed this Aug 11, 2026

akhil-sabu-ibm reopened this Aug 11, 2026

github-actions Bot added triage PRs from the community and removed triage PRs from the community labels Aug 11, 2026

Uh oh!

Conversation

akhil-sabu-ibm commented May 19, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Changes

Testing

Benefits

Uh oh!

github-actions Bot commented May 27, 2026

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akhil-sabu-ibm commented Jun 22, 2026

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimaison commented Aug 7, 2026

Uh oh!

akhil-sabu-ibm commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

akhil-sabu-ibm commented May 19, 2026 •

edited by github-actions Bot

Loading