Skip to content

Fail over RabbitMQ connections that remain blocked - #237

Merged
enteriaza merged 3 commits into
masterfrom
192-bug-rabbitmq-blocked-connection-failover
Oct 21, 2026
Merged

enteriaza merged 3 commits into
masterfrom
192-bug-rabbitmq-blocked-connection-failover

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Summary

Fixes #192.

Detect RabbitMQ connections that remain broker-blocked beyond the configured timeout and trigger recovery through the existing connection-generation and multi-host reconnect path, even when the AMQP connection remains open and heartbeats continue.

Root cause

The client logged connection.blocked and connection.unblocked events but did not initiate recovery when a block persisted. RabbitMqService relied on ConnectionLost, normally triggered by connection shutdown, to start reconnection.

Additionally, ConnectionBlockedTimeoutSeconds was assigned to RequestedConnectionTimeout, which controls connection establishment rather than the duration a live connection may remain blocked.

Consequently, a broker experiencing a disk or memory alarm could remain connected indefinitely while publishers were unable to make progress.

Changes

  • Add blocked-duration timeout handling: Track continuous broker-blocked periods using a cancellation-aware timer backed by TimeProvider.
  • Trigger recovery on persistent blocking: When the timeout expires, request recovery through the existing connection-generation and reconnect machinery, preserving configured multi-host failover behaviour.
  • Cancel on unblock: An unblock before timeout cancels the pending recovery timer. A subsequent blocked episode receives a fresh timeout.
  • Prevent duplicate recovery: Repeated blocked notifications do not create competing timers or duplicate recovery transitions for the same generation.
  • Protect connection generations: Validate connection identity and timer ownership so stale callbacks and timeout work cannot affect a replacement connection.
  • Handle lifecycle races: Cancel pending timeout work during connection loss, retirement, publication of a replacement connection, and service shutdown.
  • Separate timeout responsibilities: ConnectionBlockedTimeoutSeconds controls the maximum continuous blocked duration. SocketTimeoutSeconds controls socket read/write timeouts and RabbitMQ RequestedConnectionTimeout.
  • Update configuration and diagnostics: Align runtime options, validation, documentation, and structured logging with the new behaviour.

Configuration

Setting Default Valid range / constraint Purpose
SocketTimeoutSeconds 30 5–600 Socket read/write and connection-establishment timeout
ConnectionBlockedTimeoutSeconds 30 5–3600; must be at least RpcTimeoutSeconds Maximum continuous broker-blocked duration before recovery

Both settings default to 30 seconds. Deployments that previously relied on a larger ConnectionBlockedTimeoutSeconds value to control connection-establishment timeout should review their SocketTimeoutSeconds configuration.

Validation

  • Focused RabbitMQ tests: 38 BackFillerService tests and 47 Common tests passed
  • VectorNNTPD.BackFillerService.Tests (Release x64): 1,344 passed
  • VectorNNTPD.Common.Tests (Release x64): 1,139 passed
  • Release x64 solution build: succeeded, 0 warnings
  • dotnet format VectorNNTP.slnx --verify-no-changes: passed
  • tools/Invoke-CodeQlAnalysis.ps1: completed successfully; no findings on the changed RabbitMQ files. Six pre-existing findings remain in NntpSessionPool.cs.

Correctness guarantees

  • A connection that remains blocked beyond its configured timeout triggers at most one effective recovery transition for that connection generation.
  • An unblock that wins the synchronized state transition prevents the pending timeout from initiating recovery.
  • Once timeout expiry has claimed recovery, a later unblock does not reverse that decision.
  • Stale callbacks cannot alter the active connection generation's blocked-timeout state.
  • Service cancellation suppresses post-stop recovery.
  • Normal multi-host reconnect and existing message-disposition behaviour are preserved.

@enteriaza enteriaza self-assigned this Oct 21, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 21, 2026
@enteriaza
enteriaza merged commit 3e12264 into master Oct 21, 2026
7 checks passed
@enteriaza
enteriaza deleted the 192-bug-rabbitmq-blocked-connection-failover branch October 21, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blocked RabbitMQ node is not failed over while the connection stays open

2 participants