Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Consequences:

Monitor HTTP never enters Article Work dispatch, settlement, NNTP, or ACK/NACK paths. Failed/missing metrics are `Unknown` (never invented as zero). The monitor does not change reconcile, topology, or logging retention.

Provider queue type is derived from the configured unique `RabbitMQ:Hosts` count (never from reachable hosts or live cluster members). Exactly one host declares a durable classic queue (`x-queue-type=classic`) without quorum arguments or Management quorum health. Two or more hosts declare a durable quorum queue with `x-quorum-initial-group-size` and readiness membership floor equal to that host count, plus configured `x-delivery-limit`. All configured hosts must belong to the same RabbitMQ cluster; the client host list does not establish membership. Successful AMQP quorum declaration alone does not mark a backbone ready: readiness also requires Management HTTP verification of members, online members, and leader. Pre-existing queues with inequivalent type or immutable arguments are not deleted or recreated; operators must migrate or grow membership explicitly. Changing `Hosts` from one entry to multiple (or the reverse) changes the desired queue type and is not a safe in-place conversion. Management outages and timeouts leave multi-host backbones unready and follow the same reconcile backoff.
Provider queue type is derived from the configured unique `RabbitMQ:Hosts` count (never from reachable hosts or live cluster members). Exactly one host declares a durable classic queue (`x-queue-type=classic`) without quorum arguments or Management quorum health. Two or more hosts declare a durable quorum queue with `x-quorum-initial-group-size` and readiness membership floor equal to that host count, plus configured `x-delivery-limit` (default `5` via `ArticleWorkDeliveryLimit`). Both classic and quorum declarations include configured `x-max-length` (AMQP `long`, default `10000` via `ArticleWorkQueueMaxLength`) and `x-message-ttl` (AMQP `long`, default `60000` via `ArticleWorkQueueMessageTtlMilliseconds`) so redeclaration matches existing broker policy. Queue TTL is distinct from per-message expiration on Article Work response publications. All configured hosts must belong to the same RabbitMQ cluster; the client host list does not establish membership. Successful AMQP quorum declaration alone does not mark a backbone ready: readiness also requires Management HTTP verification of members, online members, and leader. Pre-existing queues with inequivalent type or immutable arguments are not deleted or recreated; operators must migrate or grow membership explicitly. Changing `Hosts` from one entry to multiple (or the reverse) changes the desired queue type and is not a safe in-place conversion. Management outages and timeouts leave multi-host backbones unready and follow the same reconcile backoff.

## Article Work settlement

Expand Down
4 changes: 3 additions & 1 deletion docs/common/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ There is no application-level RabbitMQ connection pool. `MinConnections` and `Ma
| `PoolReconnectMaxDelayMs` | int | `30000` | **yes** | Application reconnect max delay (`50–300000`; ≥ base) |
| `ChannelLeaseTimeoutSeconds` | int | `60` | **yes** | Validated; reserved for later channel work (`1–3600`; ≥ `RpcTimeoutSeconds`) |
| `WorkRequestMaxPayloadBytes` | int | `1024` | **yes** | Maximum admitted Article Work request payload bytes (`1–4096`) |
| `ArticleWorkDeliveryLimit` | int | `10` | **yes** | Declared as `x-delivery-limit` on quorum Article Work queues and used as the application requeue cap when broker delivery-count metadata is available (`1–1000`). Not declared on classic single-host queues (broker does not accept that argument there). With count `0..(limit-1)` the app may requeue; at `limit` it rejects without requeue. No DLX is declared; exhausted messages are dropped. Confirmed mandatory-unroutable replies use the same discard class (NACK without requeue) with structured `ReplyUnroutableTerminal` logging — not a dead-letter |
| `ArticleWorkDeliveryLimit` | int | `5` | **yes** | Declared as `x-delivery-limit` on quorum Article Work queues and used as the application requeue cap when broker delivery-count metadata is available (`1–1000`). Not declared on classic single-host queues (broker does not accept that argument there). With count `0..(limit-1)` the app may requeue; at `limit` it rejects without requeue. No DLX is declared; exhausted messages are dropped. Confirmed mandatory-unroutable replies use the same discard class (NACK without requeue) with structured `ReplyUnroutableTerminal` logging — not a dead-letter. Must match any value already configured on the queue |
| `ArticleWorkQueueMaxLength` | int | `10000` | **yes** | Declared as `x-max-length` (AMQP `long`) on classic and quorum Article Work provider queues (`1–10000000`). Caps ready messages retained on the queue. Must match any value already configured on the queue; inequivalent values yield `PRECONDITION_FAILED` and are not rewritten in place |
| `ArticleWorkQueueMessageTtlMilliseconds` | int | `60000` | **yes** | Declared as `x-message-ttl` (AMQP `long`) on classic and quorum Article Work provider queues (`1–86400000`). Ready-message lifetime in milliseconds. Distinct from per-message expiration on Article Work response publications and from `BackFiller:ArticleRetention:RetentionTtlSeconds`. Must match any value already configured on the queue; inequivalent values yield `PRECONDITION_FAILED` and are not rewritten in place |
| `ChannelPoolSize` | int | `512` | **yes** | Validated; reserved for later consumer buffering (`1–8192`) |
| `MinConnections` | int | `4` | **yes** | Validated; reserved (`1–512`; ≤ `MaxConnections`) |
| `MaxConnections` | int | `16` | **yes** | Validated; reserved (`1–512`) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ internal static class ArticleWorkResponseWireProtocol
/// Applied when BackFiller publishes a terminal reply to the caller's exclusive ReplyTo queue.
/// Leftover replies should not linger if the caller has already abandoned the correlation.
/// This is RabbitMQ message TTL on the reply path, not a processing deadline for a work
/// delivery already handed to a consumer, and not a TTL for work <em>requests</em> waiting
/// in <c>backfiller.{backbone}</c>. Work requests must remain eligible for broker buffering
/// until a consumer delivers them; they must not carry this expiration. Queue declarations
/// must not set <c>x-message-ttl</c> that silently drops ready work.
/// delivery already handed to a consumer, and not the provider-queue ready-message TTL
/// declared as <c>x-message-ttl</c> via <c>RabbitMQ:ArticleWorkQueueMessageTtlMilliseconds</c>.
/// Work-request publishes must not carry this reply-path expiration; queue-level TTL is a
/// separate broker policy argument on <c>backfiller.{backbone}</c>.
/// </remarks>
internal const string ExpirationMilliseconds = "1000";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ namespace VectorNNTPD.BackFillerService.ArticleWork.SilentConsumerMonitor
/// </summary>
/// <remarks>
/// <see cref="ConditionSignal.Unknown"/> neither opens nor resolves: an already-open alert stays open
/// so a Management outage is not treated as recovery. Sustained <see cref="ConditionSignal.Active"/>
/// past the configured open window opens; sustained <see cref="ConditionSignal.Inactive"/> past the
/// resolve window resolves. Persistence clocks reset when the opposite conclusive signal arrives.
/// so a Management outage is not treated as recovery. Unknown also <em>freezes</em> Active/Inactive
/// persistence clocks (shifts clock origins by the Unknown gap) so outage wall time cannot complete an
/// open or resolve window. Sustained <see cref="ConditionSignal.Active"/> past the configured open
/// window opens; sustained <see cref="ConditionSignal.Inactive"/> past the resolve window resolves.
/// Persistence clocks reset when the opposite conclusive signal arrives.
/// </remarks>
internal sealed class ObservationStateMachine
{
Expand All @@ -29,6 +31,11 @@ internal sealed class ObservationStateMachine
/// <summary>UTC start of the current Inactive persistence clock while open; null otherwise.</summary>
private DateTimeOffset? _inactiveSinceUtc;

/// <summary>
/// UTC of the last <see cref="Apply"/> used to freeze persistence across <see cref="ConditionSignal.Unknown"/> gaps.
/// </summary>
private DateTimeOffset? _lastApplyUtc;

/// <summary>Creates a machine for one observation identity.</summary>
/// <param name="key">Stable identity (instance session key, <c>local</c>, or shared queue name).</param>
/// <param name="openWindow">Minimum sustained Active duration before <see cref="ObservationTransition.Opened"/>.</param>
Expand Down Expand Up @@ -57,11 +64,14 @@ public ObservationResult Apply(ConditionSignal signal, DateTimeOffset atUtc, str
switch (signal)
{
case ConditionSignal.Unknown:
FreezePersistenceClocks(atUtc);
_lastApplyUtc = atUtc;
return Result(signal, ObservationTransition.None, CurrentPersistence(atUtc), detail);

case ConditionSignal.Active:
_inactiveSinceUtc = null;
_activeSinceUtc ??= atUtc;
_lastApplyUtc = atUtc;
var activeFor = atUtc - _activeSinceUtc.Value;
if (_phase == ObservationPhase.Clear)
{
Expand All @@ -81,10 +91,12 @@ public ObservationResult Apply(ConditionSignal signal, DateTimeOffset atUtc, str
if (_phase == ObservationPhase.Clear)
{
_inactiveSinceUtc = null;
_lastApplyUtc = atUtc;
return Result(signal, ObservationTransition.None, TimeSpan.Zero, detail);
}

_inactiveSinceUtc ??= atUtc;
_lastApplyUtc = atUtc;
var inactiveFor = atUtc - _inactiveSinceUtc.Value;
if (inactiveFor >= _resolveWindow)
{
Expand All @@ -96,10 +108,40 @@ public ObservationResult Apply(ConditionSignal signal, DateTimeOffset atUtc, str
return Result(signal, ObservationTransition.Continued, inactiveFor, detail);

default:
_lastApplyUtc = atUtc;
return Result(ConditionSignal.Unknown, ObservationTransition.None, TimeSpan.Zero, "unhandled");
}
}

/// <summary>
/// Shifts Active/Inactive clock origins by the gap since <see cref="_lastApplyUtc"/> so Unknown
/// wall time does not advance open or resolve persistence.
/// </summary>
/// <param name="atUtc">Current evaluation instant.</param>
private void FreezePersistenceClocks(DateTimeOffset atUtc)
{
if (_lastApplyUtc is not { } last)
{
return;
}

var gap = atUtc - last;
if (gap <= TimeSpan.Zero)
{
return;
}

if (_activeSinceUtc is { } activeSince)
{
_activeSinceUtc = activeSince + gap;
}

if (_inactiveSinceUtc is { } inactiveSince)
{
_inactiveSinceUtc = inactiveSince + gap;
}
}

/// <summary>Active or Inactive persistence relative to <paramref name="atUtc"/>, or zero when neither clock is set.</summary>
/// <param name="atUtc">Evaluation instant.</param>
/// <returns>Elapsed persistence for the active clock, inactive clock, or <see cref="TimeSpan.Zero"/>.</returns>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -177,8 +177,9 @@ public IReadOnlyList<ObservationResult> ObserveInstance(
/// <exception cref="ArgumentNullException">Thrown when <paramref name="poll"/> is null.</exception>
/// <remarks>
/// Non-success polls append an invalid sample so shared evaluators return <see cref="ConditionSignal.Unknown"/>.
/// Unknown neither opens nor resolves; an already-open shared alert remains open until a conclusive
/// Inactive signal, catalog prune (target gone — not broker recovery), or process restart.
/// Unknown neither opens nor resolves and freezes open/resolve persistence clocks; an already-open
/// shared alert remains open until a conclusive Inactive signal, catalog prune (target gone — not
/// broker recovery), or process restart.
/// </remarks>
public IReadOnlyList<ObservationResult> ObserveSharedQueue(
string queueName,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -205,10 +205,18 @@ internal sealed record NntpDbRuntimeOptions(
/// Declared <c>x-delivery-limit</c> on quorum queues and application requeue cap when broker delivery-count
/// metadata is available.
/// </param>
/// <param name="QueueMaxLength">
/// Declared <c>x-max-length</c> on classic and quorum Article Work provider queues (ready-message cap).
/// </param>
/// <param name="QueueMessageTtlMilliseconds">
/// Declared <c>x-message-ttl</c> on classic and quorum Article Work provider queues (ready-message lifetime).
/// </param>
internal sealed record BackFillerRabbitMqRuntimeOptions(
int WorkRequestMaxPayloadBytes,
int PublishConfirmTimeoutSeconds,
ushort? ConsumerPrefetchCount,
VectorNNTPD.Common.Messaging.RabbitMq.ArticleWorkQueueTopology QueueTopology,
int DeliveryLimit);
int DeliveryLimit,
int QueueMaxLength,
int QueueMessageTtlMilliseconds);
}
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,9 @@ internal static BackFillerRuntimeOptions Create(
PublishConfirmTimeoutSeconds: rabbitMq.PublishConfirmTimeoutSeconds ?? 10,
ConsumerPrefetchCount: rabbitMq.ConsumerPrefetchCount,
QueueTopology: ArticleWorkQueueTopology.FromConfiguredHosts(rabbitMq.Hosts),
DeliveryLimit: rabbitMq.ArticleWorkDeliveryLimit ?? 10),
DeliveryLimit: rabbitMq.ArticleWorkDeliveryLimit ?? 5,
QueueMaxLength: rabbitMq.ArticleWorkQueueMaxLength ?? 10000,
QueueMessageTtlMilliseconds: rabbitMq.ArticleWorkQueueMessageTtlMilliseconds ?? 60000),
NntpDb: new NntpDbRuntimeOptions(
nntpDb.ConnectionString.Trim(),
nntpDbBuilder.Server ?? string.Empty,
Expand Down
Loading
Loading