Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/common/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Top-level `RabbitMQ` section. The Common sample is `src/VectorNNTPD.Common/Vecto

NNTPD and StorageServer are not projects in this repository, so their build output is not part of the contract here. BackFiller ships connectivity in `src/VectorNNTPD.BackFillerService/VectorNNTPD.RabbitMQ.json`. Its primary file, `VectorNNTPD.BackFillerService.json`, still must not repeat a top-level `RabbitMQ` section. `BackFiller:Logging:RabbitMQ` is the log sink, not broker connectivity.

`RabbitMqService` owns one process-wide connection, verifies that it is open, and replaces it on connectivity loss while incrementing a monotonic generation. Client automatic recovery is disabled. The AMQP client name comes from `IRabbitMqConnectionNameProvider`. Callers obtain the current connection with `TryGetCurrent` and do not dispose it.
`RabbitMqService` owns one process-wide connection, verifies that it is open, and replaces it on connectivity loss or when a broker block lasts longer than `ConnectionBlockedTimeoutSeconds`, while incrementing a monotonic generation. Client automatic recovery is disabled. The AMQP client name comes from `IRabbitMqConnectionNameProvider`. Callers obtain the current connection with `TryGetCurrent` and do not dispose it.

There is no application-level RabbitMQ connection pool. `MinConnections` and `MaxConnections` are validated and are not enforced as a pool.

Expand All @@ -33,10 +33,10 @@ There is no application-level RabbitMQ connection pool. `MinConnections` and `Ma
| `VirtualHost` | string | `/` | **yes** | RabbitMQ virtual host |
| `EnableSsl` | bool | `true` | **yes** | Whether the connection uses TLS (TLS 1.2 and 1.3) |
| `RequestedHeartbeatSeconds` | int | `60` | **yes** | AMQP heartbeat (`0–3600`; `0` disables) |
| `SocketTimeoutSeconds` | int | `30` | **yes** | Socket read/write timeout (`5–600`) |
| `SocketTimeoutSeconds` | int | `30` | **yes** | Socket read/write timeout and AMQP connection establishment (`RequestedConnectionTimeout`) (`5–600`) |
| `RequestedChannelMax` | int | `2047` | **yes** | Requested channel limit per connection (`1–65535`) |
| `RpcTimeoutSeconds` | int | `30` | **yes** | Client continuation/handshake timeout (`1–3600`) |
| `ConnectionBlockedTimeoutSeconds` | int | `30` | **yes** | Client connection timeout (`5–3600`; must be ≥ `RpcTimeoutSeconds`) |
| `ConnectionBlockedTimeoutSeconds` | int | `30` | **yes** | Maximum continuous broker-blocked duration before the connection generation is retired and multi-host reconnect runs (`5–3600`; must be ≥ `RpcTimeoutSeconds`). Not the AMQP dial timeout. |
| `NetworkRecoveryIntervalSeconds` | int | `5` | **yes** | Client recovery-interval setting (`1–3600`; unused while automatic recovery is disabled) |
| `PoolReconnectBaseDelayMs` | int | `250` | **yes** | Application reconnect base delay (`50–60000`) |
| `PoolReconnectMaxDelayMs` | int | `30000` | **yes** | Application reconnect max delay (`50–300000`; ≥ base) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,24 @@ internal interface IRabbitMqConnection : IAsyncDisposable
/// </remarks>
event EventHandler<RabbitMqConnectionLostEventArgs>? ConnectionLost;

/// <summary>
/// Raised when the broker blocks the connection (for example under resource alarm).
/// </summary>
/// <remarks>
/// <see cref="RabbitMqService"/> is the only production subscriber. The connection remains
/// open; the service owns the blocked-duration timeout and recovery decision.
/// </remarks>
event EventHandler<RabbitMqConnectionBlockedEventArgs>? ConnectionBlocked;

/// <summary>
/// Raised when the broker clears a prior block on this connection.
/// </summary>
/// <remarks>
/// <see cref="RabbitMqService"/> is the only production subscriber. Cancels a pending
/// blocked-duration timeout for the current generation when the sender is still current.
/// </remarks>
event EventHandler? ConnectionUnblocked;

/// <summary>
/// Opens a short-lived channel that can declare exchanges, queues, and bindings.
/// </summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,12 @@ internal RabbitMqClientConnection(IConnection connection, string virtualHost, IL
/// <inheritdoc />
public event EventHandler<RabbitMqConnectionLostEventArgs>? ConnectionLost;

/// <inheritdoc />
public event EventHandler<RabbitMqConnectionBlockedEventArgs>? ConnectionBlocked;

/// <inheritdoc />
public event EventHandler? ConnectionUnblocked;

/// <inheritdoc />
public async Task<IRabbitMqTopologyChannel> CreateTopologyChannelAsync(CancellationToken cancellationToken)
{
Expand Down Expand Up @@ -222,23 +228,32 @@ private Task OnCallbackExceptionAsync(object sender, CallbackExceptionEventArgs
return Task.CompletedTask;
}

/// <summary>Logs the broker block reason. Does not pause channels or request recovery.</summary>
/// <summary>
/// Logs the broker block reason and raises <see cref="ConnectionBlocked"/>.
/// Does not pause channels or request recovery; <see cref="RabbitMqService"/> owns the timeout.
/// </summary>
/// <param name="sender">Unused.</param>
/// <param name="eventArgs">Broker block reason.</param>
/// <returns>A completed task.</returns>
private Task OnBlockedAsync(object sender, ConnectionBlockedEventArgs eventArgs)
{
RabbitMqLogMessages.ConnectionBlocked(_logger, eventArgs.Reason);
var reason = eventArgs.Reason ?? string.Empty;
RabbitMqLogMessages.ConnectionBlocked(_logger, reason);
ConnectionBlocked?.Invoke(this, new RabbitMqConnectionBlockedEventArgs(reason));
return Task.CompletedTask;
}

/// <summary>Logs that the broker unblocked the connection. <paramref name="eventArgs"/> is unused.</summary>
/// <summary>
/// Logs that the broker unblocked the connection and raises <see cref="ConnectionUnblocked"/>.
/// <paramref name="eventArgs"/> is unused.
/// </summary>
/// <param name="sender">Unused.</param>
/// <param name="eventArgs">Unused client event args.</param>
/// <returns>A completed task.</returns>
private Task OnUnblockedAsync(object sender, AsyncEventArgs eventArgs)
{
RabbitMqLogMessages.ConnectionUnblocked(_logger);
ConnectionUnblocked?.Invoke(this, EventArgs.Empty);
return Task.CompletedTask;
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ internal static ConnectionFactory CreateClientFactory(
TopologyRecoveryEnabled = false,
NetworkRecoveryInterval = TimeSpan.FromSeconds(options.NetworkRecoveryIntervalSeconds),
RequestedHeartbeat = TimeSpan.FromSeconds(options.RequestedHeartbeatSeconds),
RequestedConnectionTimeout = TimeSpan.FromSeconds(options.ConnectionBlockedTimeoutSeconds),
// Handshake/dial timeout is independent of ConnectionBlockedTimeoutSeconds, which
// RabbitMqService uses as the maximum continuous broker-blocked duration.
RequestedConnectionTimeout = TimeSpan.FromSeconds(options.SocketTimeoutSeconds),
ContinuationTimeout = TimeSpan.FromSeconds(options.RpcTimeoutSeconds),
HandshakeContinuationTimeout = TimeSpan.FromSeconds(options.RpcTimeoutSeconds),
SocketReadTimeout = TimeSpan.FromSeconds(options.SocketTimeoutSeconds),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
namespace VectorNNTPD.Common.Messaging.RabbitMq
{
/// <summary>Broker block details for a single AMQP connection.</summary>
/// <param name="Reason">Block reason supplied by the broker.</param>
internal sealed record RabbitMqConnectionBlockedEventArgs(string Reason);
}
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,24 @@ internal static partial void ConnectionLost(
Message = "RabbitMQ broker unblocked the connection")]
internal static partial void ConnectionUnblocked(ILogger logger);

/// <summary>
/// Logs event 2807 (warning): a connection remained broker-blocked through
/// <c>ConnectionBlockedTimeoutSeconds</c>; recovery is requested for that generation.
/// </summary>
/// <param name="logger">Logger that receives the event.</param>
/// <param name="Generation">Connection generation that stayed blocked.</param>
/// <param name="TimeoutSeconds">Configured blocked-duration timeout that elapsed.</param>
/// <param name="Reason">Block reason from the start of this blocked episode.</param>
[LoggerMessage(
EventId = 2807,
Level = LogLevel.Warning,
Message = "RabbitMQ connection generation {Generation} remained blocked for {TimeoutSeconds}s ({Reason}); requesting recovery")]
internal static partial void ConnectionBlockedTimeout(
ILogger logger,
long Generation,
int TimeoutSeconds,
string Reason);

/// <summary>Logs event 2808 (information): a reconnect attempt is about to wait.</summary>
/// <param name="logger">Logger that receives the event.</param>
/// <param name="Attempt">One-based attempt number for the current recovery.</param>
Expand Down Expand Up @@ -211,5 +229,21 @@ internal static partial void ConnectionReplacedHandlerFailed(
Exception exception,
long Generation,
string Subscriber);

/// <summary>
/// Logs event 2821 (debug): blocked-timeout CTS cancel or dispose raced with another cleanup path.
/// Cleanup continues; this is not a recovery failure.
/// </summary>
/// <param name="logger">Logger that receives the event.</param>
/// <param name="exception">The <see cref="ObjectDisposedException"/> from the raced CTS.</param>
/// <param name="Operation"><c>Cancel</c> or <c>Dispose</c>.</param>
[LoggerMessage(
EventId = 2821,
Level = LogLevel.Debug,
Message = "RabbitMQ blocked-timeout CTS was already disposed during {Operation}")]
internal static partial void BlockedTimeoutCtsAlreadyDisposed(
ILogger logger,
Exception exception,
string Operation);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,9 @@ public sealed class RabbitMqOptions
public int? RpcTimeoutSeconds { get; set; } = 30;

/// <summary>
/// Maximum duration, in seconds, that a broker-blocked RabbitMQ connection may remain
/// blocked before recovery evaluation. Also used as the client connection timeout.
/// Maximum continuous duration, in seconds, that a broker-blocked RabbitMQ connection may
/// remain blocked before <see cref="RabbitMqService"/> retires that generation and reconnects.
/// Not the AMQP handshake timeout; handshake uses <see cref="SocketTimeoutSeconds"/>.
/// </summary>
public int? ConnectionBlockedTimeoutSeconds { get; set; } = 30;

Expand Down Expand Up @@ -197,7 +198,8 @@ public sealed class RabbitMqOptions
public int? RequestedHeartbeatSeconds { get; set; } = 60;

/// <summary>
/// RabbitMQ socket operation timeout, in seconds, for low-level network I/O.
/// RabbitMQ socket read/write timeout and AMQP connection-establishment timeout
/// (<c>RequestedConnectionTimeout</c>), in seconds.
/// </summary>
public int? SocketTimeoutSeconds { get; set; } = 30;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ namespace VectorNNTPD.Common.Messaging.RabbitMq
/// <param name="EnableSsl">Whether TLS is enabled for RabbitMQ connectivity.</param>
/// <param name="ChannelLeaseTimeoutSeconds">Maximum channel lease timeout in seconds.</param>
/// <param name="RpcTimeoutSeconds">Maximum RabbitMQ RPC timeout in seconds.</param>
/// <param name="ConnectionBlockedTimeoutSeconds">Maximum blocked-connection timeout in seconds.</param>
/// <param name="ConnectionBlockedTimeoutSeconds">Maximum continuous broker-blocked duration before generation retirement, in seconds.</param>
/// <param name="ChannelPoolSize">Configured RabbitMQ channel pool size.</param>
/// <param name="MinConnections">Configured minimum pooled RabbitMQ connections.</param>
/// <param name="MaxConnections">Configured maximum pooled RabbitMQ connections.</param>
Expand All @@ -27,7 +27,7 @@ namespace VectorNNTPD.Common.Messaging.RabbitMq
/// <param name="DegradedThreshold">Degraded health threshold ratio.</param>
/// <param name="UnhealthyThreshold">Consecutive unhealthy evaluations threshold.</param>
/// <param name="RequestedHeartbeatSeconds">Requested AMQP heartbeat in seconds.</param>
/// <param name="SocketTimeoutSeconds">Socket read/write timeout in seconds.</param>
/// <param name="SocketTimeoutSeconds">Socket read/write and AMQP connection-establishment timeout in seconds.</param>
/// <param name="RequestedChannelMax">Requested channel max for AMQP negotiation.</param>
/// <param name="ConsumerPrefetchCount">Optional RabbitMQ consumer prefetch count.</param>
/// <param name="DiagnosticPayloadCorrelationId">Optional AMQP CorrelationId used to gate temporary payload diagnostics.</param>
Expand Down
Loading
Loading