Skip to content

Make E2E stress queue cleanup safe for foreign messages - #251

Merged
enteriaza merged 1 commit into
masterfrom
stress-queue-cleanup
Oct 26, 2026
Merged

enteriaza merged 1 commit into
masterfrom
stress-queue-cleanup

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Summary

Eliminates the message-loss window in BackFillerService E2E stress queue cleanup by keeping foreign RabbitMQ deliveries unacknowledged until the cleanup channel closes.

Changes

  • Discard only messages whose StressRunId exactly matches the current stress run.
  • Leave foreign, missing, malformed, and unsupported headers unacknowledged.
  • Remove the foreign-message buffering and republishing path.
  • Rely on RabbitMQ to requeue unacknowledged deliveries when the cleanup channel closes.
  • Introduce a testable scan abstraction and add coverage for mixed messages, ownership matching, scan limits, and failure cleanup.

Verification

  • Cleanup tests: 9 passed.
  • SilentConsumerMonitor tests: 56 passed.
  • Full BackFillerService test suite: 1,600 passed.
  • Release x64 solution build: succeeded, zero warnings.
  • Formatting verification for changed files: passed.
  • Verified BasicGet progression and broker requeue behaviour using an isolated exclusive queue.

Limitations

  • Requeued foreign messages may change relative order when concurrent publishers are active.
  • The bounded scan may leave owned messages behind a sufficiently large foreign-message backlog.
  • Foreign deliveries remain unacknowledged for the duration of the scan.

No production monitoring or recovery code is changed.

- Keep foreign deliveries unacknowledged during cleanup
- Discard only messages matching the current StressRunId
- Rely on channel disposal to requeue foreign deliveries
- Add deterministic tests for mixed messages and scan limits
@enteriaza enteriaza self-assigned this Oct 26, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 26, 2026
@enteriaza
enteriaza merged commit 969c5bb into master Oct 26, 2026
7 checks passed
@enteriaza
enteriaza deleted the stress-queue-cleanup branch October 26, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant