Skip to content

Isolate Article Work topology declaration failures - #239

Merged
enteriaza merged 1 commit into
masterfrom
194-bug-isolate-article-work-topology-declaration-failures
Oct 21, 2026
Merged

enteriaza merged 1 commit into
masterfrom
194-bug-isolate-article-work-topology-declaration-failures

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Summary

Fixes #194 by isolating Article Work topology declaration failures so that one incompatible backbone queue cannot prevent other backbones from becoming available.

Previously, EnsureProviderTopologyAsync declared all missing backbones using a single manual-ack channel. If a quorum queue declaration failed with PRECONDITION_FAILED, RabbitMQ closed the channel. Subsequent declarations continued on the unusable channel, preventing later backbones from being declared on each reconciliation pass.

Changes

Channel failure isolation

  • Reuse the manual-ack channel while it remains open, allowing successful declarations to share one channel.
  • Dispose and clear the channel after a declaration failure before attempting the next backbone.
  • Open a replacement channel against the current connection generation when further declarations are required.
  • Preserve successful declarations and allow independent backbones to become ready despite another backbone's topology failure.

Improved diagnostics

  • Surface structured RabbitMQ shutdown details when available, including reply code, reply text, and initiator.
  • Make the original PRECONDITION_FAILED incompatibility visible rather than reporting only cascading channel-closed failures.
  • Preserve the existing reconciliation retry and backoff behaviour.

Regression tests

  • Verify that one or multiple failed declarations do not block later healthy backbones.
  • Ensure closed channels are never reused.
  • Verify structured 406 PRECONDITION_FAILED diagnostics.
  • Confirm failed declarations can be retried on a later reconciliation pass.
  • Ensure healthy running consumers are retained.
  • Verify channel disposal and cancellation behaviour.
  • Preserve channel sharing when multiple declarations succeed.

No queues are deleted, purged, or recreated automatically. Existing incompatible queues still require an operator to resolve the underlying topology mismatch.

Validation

  • Topology and provider tests: 15 passed
  • BackFiller RabbitMQ tests: 69 passed
  • Common RabbitMQ tests: 47 passed
  • BackFillerService.Tests: 1,369 passed
  • Common.Tests: 1,139 passed
  • Release x64 solution build: passed, 0 warnings and 0 errors
  • dotnet format VectorNNTPD.slnx --verify-no-changes: passed
  • CodeQL: completed successfully; no new findings on the topology declaration path

Outcome

A failed topology declaration for one backbone no longer blocks subsequent independent backbones. The existing reconciliation mechanism can retry failures while already-running consumers remain undisturbed.

@enteriaza enteriaza self-assigned this Oct 21, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 21, 2026
@enteriaza
enteriaza merged commit 087744c into master Oct 21, 2026
7 checks passed
@enteriaza
enteriaza deleted the 194-bug-isolate-article-work-topology-declaration-failures branch October 21, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One failed quorum declare closes the shared channel and blocks later backbones

1 participant