Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The worker consumes RabbitMQ Article Work, retrieves articles from upstream NNTP

After start, non-null `IApplicationService.Execution` tasks are watched. A fault, a successful completion, or cancellation of that execution raises `UnexpectedServiceTermination`. `BackFillerApplicationHostedService` marks `BackFillerApplicationHealth` unhealthy and calls `IHostApplicationLifetime.StopApplication`.

`SystemdLifecycleNotifier` sends `READY=1` at most once after `IHostApplicationLifetime.ApplicationStarted`. Startup failure does not report ready. If watchdog health is not yet true at that moment, or if a handled transient notify failure occurs, readiness is retried until it succeeds, shutdown begins, or the notifier is disposed. Once a successful `READY=1` is sent, or once shutdown begins, further readiness notifications are suppressed. `SystemdWatchdogService` sends keep-alives only while `IApplicationHealth.IsHealthyForWatchdog` is true. That flag is also false while the published TLS certificate is missing, unreadable, expired, or still inside the renewal window after a renewal attempt that did not replace it. A later successful publication of a usable certificate clears that condition. Unexpected supervised-service termination stays sticky. Readiness is not inferred from DI registration order.
`SystemdLifecycleNotifier` sends `READY=1` at most once after `IHostApplicationLifetime.ApplicationStarted`. Startup failure does not report ready. If watchdog health is not yet true at that moment, or if a handled transient notify failure occurs, readiness is retried until it succeeds, shutdown begins, or the notifier is disposed. Once a successful `READY=1` is sent, or once shutdown begins, further readiness notifications are suppressed. `SystemdWatchdogService` sends keep-alives only while `IApplicationHealth.IsHealthyForWatchdog` is true. That flag is also false while the published TLS certificate is missing, unreadable, or expired. Being inside the renewal window alone does not block keep-alives while the current material remains usable and unexpired; a recoverable renewal failure keeps serving that material. Each health read compares the recorded certificate not-after to the process clock so expiry during a prolonged ACME outage fails closed without waiting for a new observation. A later successful publication of a usable certificate clears the block. Unexpected supervised-service termination stays sticky. Readiness is not inferred from DI registration order.

Article Work shutdown uses `BackFillerShutdownOptions`:

Expand Down
50 changes: 39 additions & 11 deletions src/VectorNNTPD.BackFillerService/Acme/AcmeCertificateWatchdog.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,54 +14,82 @@ internal sealed class AcmeCertificateWatchdog : IAcmeCertificateObservation
/// <summary>Logger for the block and restore events.</summary>
private readonly ILogger<AcmeCertificateWatchdog> _logger;

/// <summary>Clock used when re-checking material not-after against a possibly stale status snapshot.</summary>
private readonly TimeProvider _time;

/// <summary>One while keep-alives are blocked for the certificate. Used so the log is written only on a change.</summary>
private int _blocked;

/// <summary>Retains the health flag and logger.</summary>
/// <summary>Retains the health flag, logger, and clock.</summary>
/// <param name="health">Watchdog health updated by each observation.</param>
/// <param name="logger">Logger for events 1924 and 1925.</param>
public AcmeCertificateWatchdog(BackFillerApplicationHealth health, ILogger<AcmeCertificateWatchdog> logger)
/// <param name="time">
/// Clock for not-after checks. Null uses <see cref="TimeProvider.System"/>.
/// Should match the clock injected into <paramref name="health"/>.
/// </param>
public AcmeCertificateWatchdog(
BackFillerApplicationHealth health,
ILogger<AcmeCertificateWatchdog> logger,
TimeProvider? time = null)
{
ArgumentNullException.ThrowIfNull(health);
ArgumentNullException.ThrowIfNull(logger);
_health = health;
_logger = logger;
_time = time ?? TimeProvider.System;
}

/// <summary>
/// True when watchdog keep-alives must stop for this certificate snapshot.
/// A usable certificate outside the renewal window does not block. A usable certificate inside the window
/// blocks only when this attempt did not replace it. Missing, unreadable, and expired material always blocks.
/// A usable, unexpired certificate never blocks, including when renewal is due and the last attempt
/// did not replace it. Missing, unreadable, and expired material always blocks. Expiry is evaluated
/// against <paramref name="utcNow"/> (or the system clock) so a status whose <see cref="CertificateStatus.Usable"/>
/// was computed earlier cannot keep keep-alives alive past <see cref="CertificateMaterial.NotAfter"/>.
/// </summary>
/// <param name="status">Fresh validation snapshot.</param>
/// <param name="renewalReplacedCertificate">True when this attempt persisted and published a replacement.</param>
/// <param name="status">Validation snapshot.</param>
/// <param name="renewalReplacedCertificate">
/// True when this attempt persisted and published a replacement. Unused while <paramref name="status"/> is usable;
/// retained for callers that already report replacement outcomes.
/// </param>
/// <param name="utcNow">
/// Instant used for the not-after check. Null uses <see cref="DateTimeOffset.UtcNow"/>.
/// </param>
/// <returns>True when keep-alives must stop.</returns>
internal static bool CertificateRequiresWatchdogBlock(CertificateStatus status, bool renewalReplacedCertificate)
internal static bool CertificateRequiresWatchdogBlock(
CertificateStatus status,
bool renewalReplacedCertificate,
DateTimeOffset? utcNow = null)
{
ArgumentNullException.ThrowIfNull(status);
_ = renewalReplacedCertificate;
if (!status.Usable || status.Material is null)
{
return true;
}

return status.DueForRenewal && !renewalReplacedCertificate;
var now = utcNow ?? DateTimeOffset.UtcNow;
return now >= status.Material.NotAfter;
}

/// <inheritdoc />
public void OnObserved(CertificateStatus status, bool renewalReplacedCertificate)
{
if (CertificateRequiresWatchdogBlock(status, renewalReplacedCertificate))
var now = _time.GetUtcNow();
if (CertificateRequiresWatchdogBlock(status, renewalReplacedCertificate, now))
{
_health.MarkCertificateUnavailable();
if (Interlocked.Exchange(ref _blocked, 1) == 0)
{
AcmeCertificateWatchdogLogMessages.CertificateWatchdogBlocked(_logger, status.Reason);
var reason = status.Usable && status.Material is not null && now >= status.Material.NotAfter
? "expired"
: status.Reason;
AcmeCertificateWatchdogLogMessages.CertificateWatchdogBlocked(_logger, reason);
}

return;
}

_health.MarkCertificateAvailable();
_health.MarkCertificateAvailable(status.Material!.NotAfter);
if (Interlocked.Exchange(ref _blocked, 0) == 1)
{
AcmeCertificateWatchdogLogMessages.CertificateWatchdogRestored(_logger);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,8 @@ internal static HostApplicationBuilder AddBackFillerHosting(this HostApplication
builder.Services.AddSingleton<VectorNNTPD.Common.Acme.IAcmeCertificateObservation>(static provider =>
new AcmeCertificateWatchdog(
provider.GetRequiredService<BackFillerApplicationHealth>(),
provider.GetRequiredService<ILogger<AcmeCertificateWatchdog>>()));
provider.GetRequiredService<ILogger<AcmeCertificateWatchdog>>(),
provider.GetRequiredService<TimeProvider>()));
VectorNNTPD.Common.Hosting.AcmeCloudflareServiceCollectionExtensions.AddAcmeCloudflareInfrastructure(
builder.Services);
VectorNNTPD.Common.Hosting.NntpDbServiceCollectionExtensions.AddNntpDbOptions(builder.Services);
Expand All @@ -159,7 +160,8 @@ internal static HostApplicationBuilder AddBackFillerHosting(this HostApplication

// Register systemd lifecycle notifier before other hosted services so it is
// constructed early and observes ApplicationStarted/Stopping for READY/STOPPING.
builder.Services.AddSingleton<BackFillerApplicationHealth>();
builder.Services.AddSingleton(static provider =>
new BackFillerApplicationHealth(provider.GetRequiredService<TimeProvider>()));
builder.Services.AddSingleton<IApplicationHealth>(static provider =>
provider.GetRequiredService<BackFillerApplicationHealth>());
builder.Services.TryAddSingleton<ISystemdNotifyBridge, SystemdNotifyBridge>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,15 @@ namespace VectorNNTPD.BackFillerService.Hosting.Systemd
/// has not begun, no unexpected supervised-service termination has been recorded, and the published TLS
/// certificate is not blocking keep-alives. Start, shutdown, and unexpected termination are sticky.
/// <see cref="MarkCertificateUnavailable"/> is cleared by <see cref="MarkCertificateAvailable"/>.
/// When a not-after is recorded with <see cref="MarkCertificateAvailable"/>, each
/// <see cref="IsHealthyForWatchdog"/> read compares it to the injected clock so expiry is not sticky
/// across a stale observation snapshot.
/// </remarks>
internal sealed class BackFillerApplicationHealth : IApplicationHealth
{
/// <summary>Clock used to evaluate recorded certificate not-after against current time.</summary>
private readonly TimeProvider _time;

/// <summary>Non-zero after <see cref="MarkStarted"/>.</summary>
private int _started;

Expand All @@ -21,17 +27,35 @@ internal sealed class BackFillerApplicationHealth : IApplicationHealth
private int _unexpectedTermination;

/// <summary>
/// Non-zero while the published certificate is missing, unusable, expired, or inside the renewal window
/// after a renewal that did not replace it. Cleared by <see cref="MarkCertificateAvailable"/>.
/// Non-zero while the published certificate is missing, unusable, or explicitly marked unavailable.
/// Cleared by <see cref="MarkCertificateAvailable"/>. Expiry after a recorded not-after is checked
/// separately via <see cref="_certificateNotAfterUtcTicks"/>.
/// </summary>
private int _certificateUnavailable;

/// <summary>
/// UTC ticks of the published certificate's not-after when keep-alives were last cleared.
/// Zero means no expiry bound (tests that mark available without a not-after, or after unavailable).
/// </summary>
private long _certificateNotAfterUtcTicks;

/// <summary>Creates health that evaluates certificate expiry with <paramref name="time"/>.</summary>
/// <param name="time">
/// Clock for not-after checks. Null uses <see cref="TimeProvider.System"/>.
/// Production registers <see cref="TimeProvider.System"/>; tests may inject a manual clock.
/// </param>
public BackFillerApplicationHealth(TimeProvider? time = null)
{
_time = time ?? TimeProvider.System;
}

/// <inheritdoc />
public bool IsHealthyForWatchdog =>
Volatile.Read(ref _started) == 1
&& Volatile.Read(ref _stopping) == 0
&& Volatile.Read(ref _unexpectedTermination) == 0
&& Volatile.Read(ref _certificateUnavailable) == 0;
&& Volatile.Read(ref _certificateUnavailable) == 0
&& CertificateNotExpired();

/// <summary>
/// Raised after <see cref="MarkCertificateAvailable"/> clears the certificate block.
Expand All @@ -40,16 +64,33 @@ internal sealed class BackFillerApplicationHealth : IApplicationHealth
internal event Action? WatchdogHealthMayHaveImproved;

/// <summary>Stops watchdog keep-alives until <see cref="MarkCertificateAvailable"/>.</summary>
/// <remarks>Reversible. Does not mark unexpected termination and does not stop the host by itself.</remarks>
internal void MarkCertificateUnavailable() => Interlocked.Exchange(ref _certificateUnavailable, 1);
/// <remarks>
/// Reversible. Does not mark unexpected termination and does not stop the host by itself.
/// Sets the unavailable flag before clearing the recorded not-after so a concurrent
/// <see cref="IsHealthyForWatchdog"/> read cannot observe a cleared expiry bound while still available
/// (which would briefly treat an expired certificate as healthy).
/// </remarks>
internal void MarkCertificateUnavailable()
{
Interlocked.Exchange(ref _certificateUnavailable, 1);
Volatile.Write(ref _certificateNotAfterUtcTicks, 0);
}

/// <summary>Allows watchdog keep-alives again after a usable certificate is published.</summary>
/// <param name="notAfter">
/// Exclusive UTC end of validity for the published material. Null leaves no expiry bound on this flag
/// (callers that only exercise start/stop health). Production certificate observations always pass the
/// material not-after so keep-alives stop when the clock reaches expiry without a new observation.
/// </param>
/// <remarks>
/// Does not clear <see cref="_stopping"/> or <see cref="_unexpectedTermination"/>.
/// Writes the not-after before clearing the unavailable flag so a concurrent health read cannot
/// observe available without the replacement expiry bound.
/// Raises <see cref="WatchdogHealthMayHaveImproved"/> after clearing the certificate block.
/// </remarks>
internal void MarkCertificateAvailable()
internal void MarkCertificateAvailable(DateTimeOffset? notAfter = null)
{
Volatile.Write(ref _certificateNotAfterUtcTicks, notAfter?.UtcTicks ?? 0);
Interlocked.Exchange(ref _certificateUnavailable, 0);
WatchdogHealthMayHaveImproved?.Invoke();
}
Expand All @@ -67,5 +108,19 @@ internal void MarkCertificateAvailable()
/// Sticky. Callers do not distinguish a faulted execution from one that completed or was canceled.
/// </remarks>
internal void MarkUnexpectedTermination() => Interlocked.Exchange(ref _unexpectedTermination, 1);

/// <summary>
/// True when no not-after was recorded, or the clock is still before that instant.
/// </summary>
private bool CertificateNotExpired()
{
var notAfterTicks = Volatile.Read(ref _certificateNotAfterUtcTicks);
if (notAfterTicks == 0)
{
return true;
}

return _time.GetUtcNow().UtcTicks < notAfterTicks;
}
}
}
3 changes: 2 additions & 1 deletion src/VectorNNTPD.Common/Acme/AcmeLogMessages.cs
Original file line number Diff line number Diff line change
Expand Up @@ -176,12 +176,13 @@ internal static partial void AuthoritativeTxtLookupFailed(

/// <summary>
/// Written when the on-disk certificate is still usable but <see cref="CertificateStatus.DueForRenewal"/> is true, before replacement issuance.
/// Startup and the renewal loop keep the existing certificate when that issuance fails.
/// </summary>
/// <param name="logger">Logger that receives the event.</param>
[LoggerMessage(
EventId = 1915,
Level = LogLevel.Information,
Message = "Existing certificate due for renewal; issuing replacement")]
Message = "Existing certificate due for renewal; attempting replacement")]
internal static partial void ExistingCertificateDueForRenewal(ILogger logger);

/// <summary>
Expand Down
Loading
Loading