Skip to content

Prevent CacheListener accept-error busy loops - #233

Merged
enteriaza merged 2 commits into
masterfrom
187-bug-cache-listener-accept-error-busy-loop
Oct 21, 2026
Merged

enteriaza merged 2 commits into
masterfrom
187-bug-cache-listener-accept-error-busy-loop

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Summary

Fixes #187 by preventing repeated socket-accept failures from creating a tight retry loop in CacheListenerService.AcceptLoopAsync.

Previously, SocketException was caught and immediately retried while the service was running. A persistent accept failure could consume a CPU core indefinitely without logging or transitioning the listener into a failed state.

Changes

  • Cancellation-aware backoff: Wait 250 ms before retrying an accept failure, preventing immediate retries while allowing prompt shutdown.
  • Bound consecutive failures: Stop the accept loop after five consecutive socket failures rather than retrying indefinitely.
  • Fail through the existing lifecycle: Log the terminal failure and throw an InvalidOperationException with the original SocketException as its inner exception, allowing the existing Execution fault path to handle unexpected termination.
  • Rate-limited logging: Log the first failure, periodically report persistent failures every five seconds, and record the terminal failure.
  • Recovery after successful accepts: Reset the consecutive-failure counter and log recovery after an accept succeeds.
  • Disposal after failure: Ensure disposal completes cleanup when the accept task has already faulted, avoiding a listener stuck in the Retiring state.
  • Regression tests: Add deterministic tests using narrow accept and delay overrides without changing production defaults.

Retry Policy

Setting Value
Retry delay 250 ms
Maximum consecutive failures 5
Repeated-failure log interval 5 seconds

All socket errors currently use the same retry policy. Error-specific classification and exponential backoff are outside the scope of this change.

Validation

  • Focused CacheListenerServiceTests: 29 passed
  • Full BackFillerService test suite, Release x64: 1,320 passed
  • Release x64 solution build: 0 warnings, 0 errors
  • dotnet format --verify-no-changes: passed
  • Local CodeQL analysis: completed successfully, with no new findings on the changed files

Closes #187.

@enteriaza enteriaza self-assigned this Oct 21, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 21, 2026
@enteriaza
enteriaza merged commit 7411e6a into master Oct 21, 2026
7 checks passed
@enteriaza
enteriaza deleted the 187-bug-cache-listener-accept-error-busy-loop branch October 21, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug: Listener] Accept failures are retried with no delay

2 participants