Skip to content

Add SilentConsumerMonitor with per-session stall detection and multi-instance monitoring - #249

Merged
enteriaza merged 3 commits into
masterfrom
silentconsumermonitor
Oct 26, 2026
Merged

enteriaza merged 3 commits into
masterfrom
silentconsumermonitor

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Description

Summary

Introduces SilentConsumerMonitor into VectorNNTPD.BackFillerService to improve detection and observability of silent consumer stalls, consumer loss, and queue-processing issues that existing monitoring may not identify promptly.

Key Changes

  • Per-session stall detection: Distinguishes individual consumer sessions, preventing healthy sibling sessions on the same backbone from masking a stalled session.
  • Dispatch-stall detection: Identifies sessions with queued in-flight deliveries that have not progressed into active processing.
  • Consumer-loss detection: Observes local consumer availability and shared queue state against expected active capacity.
  • Multi-instance monitoring: Combines instance-local session observations with RabbitMQ Management API queue observations.
  • Management API resilience: Handles unavailable observations without incorrectly resolving existing alerts; monitoring resumes evaluation when valid observations return.
  • Lifecycle integration: Registers the monitor within the existing BackFillerService hosting and dependency-injection lifecycle.
  • Structured diagnostics: Adds condition-specific logging with per-observation rate limiting.
  • Configuration and architecture documentation: Documents the monitor's configuration, observation scopes, and operational behaviour.
  • XML documentation: Improves XML documentation throughout the affected production and test files.

Behavioural Guarantees

  • Monitoring remains observational and does not initiate recovery.
  • Existing reconciliation and recovery mechanisms remain authoritative.
  • Existing queue topology and message acknowledgement behaviour are preserved.
  • Shared queue observations remain aggregate; they do not claim to identify an individual RabbitMQ consumer.
  • Missing or unavailable Management API observations do not automatically resolve open shared alerts.

Testing and Validation

  • Adds and extends tests covering state transitions, observation failures, session isolation, target lifecycle, configuration validation, and dispatch-stall lifecycle scenarios.
  • Release solution build completed successfully with zero warnings and zero errors.
  • In-scope formatting validation completed successfully.
  • Full solution formatting verification reported a pre-existing whitespace issue in BackFillerE2EStressHarness.cs, outside this change set.
  • Tests were not run as part of the final documentation-only pass.

Scope

Changes are limited to the SilentConsumerMonitor implementation, its BackFillerService integration, related tests, and associated documentation.

Detect stalled article processing and recover through the existing
reconciliation lifecycle. Ensure recovery cancels hung work even when
normal retirement has already started, and prevent queued admissions
from masking stalled handlers.

Improve quorum and topology failure observability, and disable
blocking on full asynchronous file logging queues to prevent logging
backpressure from stalling article processing.

Add regression tests for retirement races, cancellation failures,
delayed cancellation, and concurrent recovery. Preserve existing
daily log rotation, compression, and retention policies.

Release x64 build and all applicable regression suites pass.
@enteriaza enteriaza self-assigned this Oct 26, 2026
@enteriaza enteriaza added the enhancement New feature or request label Oct 26, 2026
- Correct HttpResponseMessage ownership in test handler
- Simplify healthy-session and consumer-loss counts using LINQ
- Preserve existing locking and monitoring semantics
@enteriaza
enteriaza merged commit ea8a19d into master Oct 26, 2026
7 checks passed
@enteriaza
enteriaza deleted the silentconsumermonitor branch October 26, 2026 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants