Skip to content

Harden Article Work channel recovery after RabbitMQ failures - #238

Merged
enteriaza merged 2 commits into
masterfrom
193-bug-article-work-channel-failure-recovery
Oct 21, 2026
Merged

enteriaza merged 2 commits into
masterfrom
193-bug-article-work-channel-failure-recovery

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Summary

Fixes #193 by recovering Article Work consumers and response publishers when their RabbitMQ channels fail independently of the underlying AMQP connection.

Previously, a consumer could remain marked Running after its channel closed or the broker cancelled its consumer. Similarly, a failed response-publisher channel could remain installed indefinitely, causing subsequent publishes to fail repeatedly.

Changes

Consumer recovery

  • Surface unexpected channel shutdown and broker-side consumer cancellation through IRabbitMqManualAckChannel.
  • Retire failed consumer sessions and trigger reconciliation to create replacements on the current connection.
  • Validate channel openness during reconciliation rather than relying solely on the session's Running state.
  • Suppress expected cancellation during normal retirement and application shutdown.
  • Protect against stale callbacks and close the fault-registration race by establishing session identity before subscribing and checking channel openness.

Publisher recovery

  • Detect publisher-channel failures and rebuild the channel without waiting for a new AMQP connection generation.
  • Coordinate concurrent rebuilds through a shared, chained recovery task.
  • Protect against stale callbacks and races between channel recovery, connection replacement, and disposal.
  • Ensure shutdown waits for pending rebuild work before disposing synchronization resources.
  • Preserve publish failure propagation so a failed operation is never reported as successful.

Tests and reliability

  • Add regression coverage for consumer cancellation, channel shutdown, publisher replacement, stale callbacks, concurrent rebuilds, and shutdown races.
  • Verify that unsettled deliveries remain eligible for broker redelivery.
  • Add tests for chained channel and connection replacement and recovery during shutdown.

Validation

  • Channel recovery tests: 17 passed
  • RabbitMQ Common tests: 47 passed
  • BackFillerService.Tests: 1,361 passed
  • Common.Tests: 1,139 passed on retry after an initial unrelated ACME parallel-test flake
  • Release x64 solution build: passed, 0 warnings and 0 errors
  • dotnet format --verify-no-changes: passed
  • CodeQL: completed successfully; reported findings were existing catch-of-all-exceptions findings with suppressions

Recovery behaviour

Channel-level failures are handled independently of connection-level failures. A healthy AMQP connection is retained, while genuine connection failures continue through the existing connection-recovery mechanism.

@enteriaza enteriaza self-assigned this Oct 21, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 21, 2026
@enteriaza
enteriaza merged commit cf17085 into master Oct 21, 2026
7 checks passed
@enteriaza
enteriaza deleted the 193-bug-article-work-channel-failure-recovery branch October 21, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Article Work consumer and publisher are not rebuilt when the channel dies

2 participants