Skip to content

Fix orphaned RabbitMQ messages in BackFillerService E2E stress tests - #250

Merged
enteriaza merged 1 commit into
masterfrom
e2e-test-fix
Oct 26, 2026
Merged

enteriaza merged 1 commit into
masterfrom
e2e-test-fix

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Description

Summary

Fixes orphaned work messages left in shared RabbitMQ queues by BackFillerService E2E stress tests when requests time out without an active consumer.

Changes

  • Add StressRunId-scoped cleanup to remove work messages belonging to the current stress run.
  • Preserve foreign messages encountered during cleanup rather than discarding messages belonging to other runs or workloads.
  • Ensure cleanup runs during harness teardown, including failure paths.
  • Bound the post-wait broker-outstanding drain period to five seconds.
  • Add unit tests covering message ownership, missing or malformed headers, and AMQP property handling.

Verification

  • SilentConsumerMonitor tests: 56 passed.
  • Cleanup tests: 5 passed.
  • Full BackFillerService test suite: 1,596 passed.
  • RabbitMQ ready-message depth: 0 → 0 after verification.
  • Formatting verification for new cleanup files: passed.

Scope

Test-harness changes only. No production monitoring or recovery behaviour is modified.

Known limitations

Foreign messages are temporarily acknowledged while being preserved and subsequently republished. A process failure during this interval could lose a parked message, and concurrent publishers may affect ordering. Cleanup also has a bounded scan limit, so a sufficiently large foreign-message backlog may prevent all owned messages from being reached in one pass.

- Clean up work messages by StressRunId during harness teardown
- Preserve foreign messages while scanning shared queues
- Bound post-test drain waits
- Add tests for owned-message matching and header handling
@enteriaza enteriaza self-assigned this Oct 26, 2026
@enteriaza enteriaza added the bug Something isn't working label Oct 26, 2026
@enteriaza
enteriaza merged commit a12ea5d into master Oct 26, 2026
7 checks passed
@enteriaza
enteriaza deleted the e2e-test-fix branch October 26, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant