fix(finance): retry RabbitMQ connect at startup and self-heal on drop - #167
Merged
Merged
Conversation
setupRabbitMQ() previously dialed RabbitMQ exactly once at startup with no retry, so any pod that raced RabbitMQ's readiness during a rolling restart permanently lost its publisher and every dependent RPC (RM cost recalc, cost sheet export, Oracle sync trigger) failed forever with "message queue unavailable: RabbitMQ not connected" until the pod was manually restarted. Connection now retries on startup (NewConnectionWithRetry) and self-heals via a mutex-guarded Supervise loop that swaps in a fresh connection and reruns registered OnReconnect callbacks after a drop. ErrPublisherUnavailable keeps the "RabbitMQ not connected" substring so existing error-based tests/checks still match.
ilramdhan
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes an issue where
setupRabbitMQ()dialed RabbitMQ only once at startup with no retry logic. During rolling restarts or startup races with RabbitMQ's readiness, pods permanently lost connection, causing all dependent RPCs (such as RM cost recalculation, cost sheet export, and Oracle sync triggers) to fail continuously with"message queue unavailable: RabbitMQ not connected".Connection initialization now retries on startup via
NewConnectionWithRetryand self-heals after connection drops through a mutex-guardedSuperviseloop that swaps in a fresh connection and reruns registeredOnReconnectcallbacks.Type of Change
Service(s) Affected
Changes Made
setupRabbitMQ()withNewConnectionWithRetryfor startup resilience.Superviseloop to automatically detect connection drops, establish a fresh connection, and re-triggerOnReconnectcallbacks."RabbitMQ not connected"error substring inErrPublisherUnavailableto maintain backward compatibility with existing tests and error handling.Related Issues
Fixes #
Related to #
API Changes (if applicable)
Proto Changes
N/A
Breaking Changes
None. Error substrings remain unchanged to preserve existing error-handling behavior.
Testing Performed
Unit Tests
Integration Tests
Manual Testing
Lint & Build
golangci-lint run ./...passesgo build ./...succeedsgo test -race ./...passesDatabase (if applicable)
Documentation
Rollback Plan
Revert this PR if unexpected reconnect behavior occurs. The system will fall back to single-attempt dialing at startup.
Screenshots/Logs (if applicable)
N/A
Pre-merge Checklist
Reviewer Notes
None