Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
266 changes: 191 additions & 75 deletions .github/workflows/build.yml

Large diffs are not rendered by default.

25 changes: 20 additions & 5 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
<!--
Production executable copyright. GenerateAssemblyInfo emits AssemblyCopyrightAttribute from $(Copyright).
-->
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService'">
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService'">
<Copyright>©2026 Chris Knipe &lt;cknipe@opticnetworks.net&gt;</Copyright>
</PropertyGroup>

Expand All @@ -22,7 +22,7 @@
-->
<Target Name="ApplyDateTimeApplicationVersion"
BeforeTargets="GetAssemblyVersion"
Condition="'$(DesignTimeBuild)' != 'true' AND '$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService'">
Condition="'$(DesignTimeBuild)' != 'true' AND ('$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService')">
<PropertyGroup>
<_ApplicationVersionNow>$([System.DateTime]::Now.Ticks)</_ApplicationVersionNow>
<_ApplicationVersionBuild>$([System.DateTime]::new($([System.Int64]::Parse($(_ApplicationVersionNow)))).Subtract($([System.DateTime]::new(2000,1,1))).Days)</_ApplicationVersionBuild>
Expand All @@ -38,21 +38,36 @@
Production publish is self-contained Native AOT for win-x64 and linux-x64 only.
A normal build does not set RuntimeIdentifier, so libraries, tests, and tools stay portable.
-->
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService'">
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService'">
<RuntimeIdentifiers>win-x64;linux-x64</RuntimeIdentifiers>
<SelfContained Condition="'$(RuntimeIdentifier)' == 'win-x64' OR '$(RuntimeIdentifier)' == 'linux-x64'">true</SelfContained>
<PublishAot Condition="'$(RuntimeIdentifier)' == 'win-x64' OR '$(RuntimeIdentifier)' == 'linux-x64'">true</PublishAot>
</PropertyGroup>

<!--
StorageService still uses Serilog.Settings.Configuration for MinimumLevel/enrichment while
sinks are registered explicitly via StorageServerConfiguredSinks. Native AOT/single-file
analysis then fails publish on Serilog.Settings.Configuration:
IL3000/IL3002 (Assembly.Location / DependencyContext in single-file) and IL2104/IL3050
(assembly trim/AOT analysis). Scope is StorageService + PublishAot only. BackFiller does
not need this because it never calls ReadFrom.Configuration.
-->
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.StorageService' AND '$(PublishAot)' == 'true'">
<NoWarn>$(NoWarn);IL2104;IL3050;IL3000;IL3002</NoWarn>
<WarningsNotAsErrors>$(WarningsNotAsErrors);IL2104;IL3050</WarningsNotAsErrors>
<ILLinkTreatWarningsAsErrors>false</ILLinkTreatWarningsAsErrors>
<IlcTreatWarningsAsErrors>false</IlcTreatWarningsAsErrors>
</PropertyGroup>

<Target Name="RejectNonX64Platform"
BeforeTargets="Restore;Build;Publish">
<Error Condition="'$(Platform)' != 'x64'"
Text="BackFiller builds only for Platform x64. Platform '$(Platform)' is not supported." />
Text="VectorNNTPD builds only for Platform x64. Platform '$(Platform)' is not supported." />
</Target>

<Target Name="RejectUnsupportedProductionRuntimeIdentifier"
BeforeTargets="Restore;Build;Publish"
Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' AND '$(RuntimeIdentifier)' != '' AND '$(RuntimeIdentifier)' != 'win-x64' AND '$(RuntimeIdentifier)' != 'linux-x64'">
Condition="('$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService') AND '$(RuntimeIdentifier)' != '' AND '$(RuntimeIdentifier)' != 'win-x64' AND '$(RuntimeIdentifier)' != 'linux-x64'">
<Error Text="Production applications publish only win-x64 and linux-x64. RuntimeIdentifier '$(RuntimeIdentifier)' is not supported." />
</Target>

Expand Down
38 changes: 24 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# BackFiller (Standalone)
# VectorNNTPD

Standalone migration of `VectorNNTPD.BackFillerService` into an independent repository.
Standalone VectorNNTPD hosts extracted from VectorNNTP.NNTPD. This repository currently ships BackFiller and StorageService, plus shared `VectorNNTPD.Common`.

## Structure

- `src/VectorNNTPD.BackFillerService/VectorNNTPD.BackFillerService.csproj` - standalone BackFiller worker app (net10.0)
- `src/VectorNNTPD.BackFillerService/` - production code organized by responsibility (for example `Acme`, `Articles`, `Configuration`, `Core`, `Hosting`, `Nntp`, `Transport`)
- `tests/VectorNNTPD.BackFillerService.Tests/VectorNNTPD.BackFillerService.Tests.csproj` - xUnit test suite, namespace `VectorNNTPD.BackFillerService.Tests`
- `VectorNNTPD.slnx` - solution including app + tests
- `src/VectorNNTPD.BackFillerService/VectorNNTPD.BackFillerService.csproj` - BackFiller worker app (net10.0)
- `src/VectorNNTPD.StorageService/VectorNNTPD.StorageService.csproj` - StorageService / VATP cache host (net10.0)
- `src/VectorNNTPD.Common/VectorNNTPD.Common.csproj` - shared library
- `tests/VectorNNTPD.BackFillerService.Tests/` / `tests/VectorNNTPD.StorageService.Tests/` / `tests/VectorNNTPD.Common.Tests/` - xUnit suites
- `VectorNNTPD.slnx` - solution including apps + tests
- `tools/Publish-Release.ps1` - Native AOT release publish for both production hosts
- `tools/Invoke-CodeQlAnalysis.ps1` - local traced C# CodeQL analysis

## Build
Expand All @@ -24,21 +26,27 @@ dotnet test .\tests\VectorNNTPD.BackFillerService.Tests\VectorNNTPD.BackFillerSe

## Publish

Windows x64:
Preferred release path (both hosts into one flat directory):

```powershell
dotnet publish .\src\VectorNNTPD.BackFillerService\VectorNNTPD.BackFillerService.csproj -c Release -p:Platform=x64 -r win-x64 --self-contained true -p:PublishAot=true -p:SelfContained=true
.\tools\Publish-Release.ps1 -Platform win-64
.\tools\Publish-Release.ps1 -Platform linux-64
```

Linux x64:
Per-project Native AOT publish still works. Example for StorageService:

```powershell
dotnet publish .\src\VectorNNTPD.BackFillerService\VectorNNTPD.BackFillerService.csproj -c Release -p:Platform=x64 -r linux-x64 --self-contained true -p:PublishAot=true -p:SelfContained=true
dotnet publish .\src\VectorNNTPD.StorageService\VectorNNTPD.StorageService.csproj -c Release -p:Platform=x64 -r win-x64 --self-contained true -p:PublishAot=true -p:SelfContained=true
```

The production project references the `StuDev.AotAnywhere` 1.0.4 MSBuild SDK. On Windows that SDK links `linux-x64` with the Zig toolchain restored by NuGet. A globalization-enabled Linux host still needs `libicu`.
Production hosts reference the `StuDev.AotAnywhere` 1.0.4 MSBuild SDK. On Windows that SDK links `linux-x64` with the Zig toolchain restored by NuGet. A globalization-enabled Linux host still needs `libicu`.

Native AOT publish keeps trim and AOT analysis enabled. CI then runs the published executable with `VECTORNNTP_BACKFILLER_STARTUP_SMOKE=1` and `VECTORNNTP_BACKFILLER_STARTUP_SMOKE_CONFIG` pointing at an isolated JSON file. The process must print `BackFiller startup smoke: configuration serverId=1 fqdn=backfiller01.usenet.ninja` and exit 0. That checkpoint is configuration binding and host construction. It does not open a listener or contact a database, broker, DNS provider, or certificate authority.
Native AOT publish keeps trim and AOT analysis enabled. CI runs each published executable against an isolated JSON fixture:

- BackFiller: `VECTORNNTP_BACKFILLER_STARTUP_SMOKE=1` / `VECTORNNTP_BACKFILLER_STARTUP_SMOKE_CONFIG` → checkpoint `BackFiller startup smoke: configuration serverId=1 fqdn=backfiller01.usenet.ninja`
- StorageService: `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE=1` / `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE_CONFIG` → checkpoint `StorageService startup smoke: configuration serverId=1 fqdn=cache01.usenet.ninja`

Those checkpoints prove configuration binding and host construction (and StorageService logging initialization). They do not open a listener or contact a database, broker, DNS provider, or certificate authority. StorageService AOT publish still scopes Serilog.Settings.Configuration IL suppressions (`IL2104`/`IL3050`/`IL3000`/`IL3002`) because MinimumLevel is applied via `ReadFrom.Configuration` while sinks are registered explicitly.

BackFiller references `VectorNNTPD.Common` for `IApplicationService`, NntpDB, ACME certificate management, the Cloudflare DNS API client, authoritative DNS queries used for DNS-01, RabbitMQ connectivity, and article parsing and validation. It keeps its own `ApplicationServiceManager` and does not use Common's `ApplicationLifecycle`. BackFiller still owns certificate watchdog health, the decision to publish listener-bound A/AAAA records only after the cache listener binds, Article Work topology, the RabbitMQ hosted adapter, and the retained article image that dot-stuffs body lines after Common's parser accepts the article.

Expand All @@ -48,9 +56,11 @@ BackFiller references `VectorNNTPD.Common` for `IApplicationService`, NntpDB, AC

## Linux service

`tools/Publish-Release.ps1 -Platform linux-64` writes `release/linux-64/`. The Linux CI job writes `.artifacts/linux-x64/` and uploads that directory. Both contain `VectorNNTPD.BackFillerService`, `VectorNNTPD.BackFillerService.json`, `VectorNNTPD.RabbitMQ.json`, `README.md`, `vectornntp-backfiller.service`, and `backfiller.env.example`.
`tools/Publish-Release.ps1 -Platform linux-64` writes `release/linux-64/`. The Linux CI job writes `.artifacts/linux-x64/` and uploads that directory. Both contain the flat Native AOT outputs for `VectorNNTPD.BackFillerService` and `VectorNNTPD.StorageService`, their primary JSON files, shared `VectorNNTPD.RabbitMQ.json`, `README.md`, and the BackFiller and StorageService systemd units plus environment-file examples.

Install BackFiller at `/opt/vectornntp/backfiller`. `deploy/systemd/vectornntp-backfiller.service` uses that path for `ExecStart`, `WorkingDirectory`, and `Documentation=`. `ReadWritePaths` lists only `logs/` and `certs/`. Under `ProtectSystem=strict` that re-opens those two directories for writing. It does not change Unix ownership or mode. The shipped JSON writes file logs under `logs/` and ACME state under `certs/`, and startup creates files inside directories that must already exist.

Install that directory at `/opt/vectornntp/backfiller`. `deploy/systemd/vectornntp-backfiller.service` uses that path for `ExecStart`, `WorkingDirectory`, and `Documentation=`. `ReadWritePaths` lists only `logs/` and `certs/`. Under `ProtectSystem=strict` that re-opens those two directories for writing. It does not change Unix ownership or mode. The shipped JSON writes file logs under `logs/` and ACME state under `certs/`, and startup creates files inside directories that must already exist.
Install StorageService at `/opt/vectornntp/storageservice` using `VectorNNTPD.StorageService`, `VectorNNTPD.StorageService.json`, `VectorNNTPD.RabbitMQ.json`, `vectornntp-storageservice.service`, and `storageservice.env.example` from the same release directory. That unit’s `ReadWritePaths` also include `db/` and `spool/` for the article index and segment-pool roots.

Create the account and install the release as root-owned files. This step copies the placeholder environment file and reloads systemd. It does not start the service.

Expand Down
1 change: 1 addition & 0 deletions VectorNNTPD.sln.DotSettings
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
<s:Boolean x:Key="/Default/UserDictionary/Words/=codeql/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=NNTPD/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=nntpsharedconfig/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=VATP/@EntryIndexedValue">True</s:Boolean></wpf:ResourceDictionary>
7 changes: 7 additions & 0 deletions VectorNNTPD.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
<Project Path="src/VectorNNTPD.Common/VectorNNTPD.Common.csproj">
<Platform Project="x64" />
</Project>
<Project Path="src/VectorNNTPD.StorageService/VectorNNTPD.StorageService.csproj">
<Platform Project="x64" />
</Project>
</Folder>
<Folder Name="/tests/">
<Project Path="tests/VectorNNTPD.BackFillerService.Tests/VectorNNTPD.BackFillerService.Tests.csproj">
Expand All @@ -17,5 +20,9 @@
<Project Path="tests/VectorNNTPD.Common.Tests/VectorNNTPD.Common.Tests.csproj">
<Platform Project="x64" />
</Project>
<Project Path="tests/VectorNNTPD.StorageService.Tests/VectorNNTPD.StorageService.Tests.csproj">
<Platform Project="x64" />
</Project>
</Folder>
</Solution>

16 changes: 16 additions & 0 deletions deploy/systemd/storageservice.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# systemd EnvironmentFile for vectornntp-storageservice.service.
# One KEY=VALUE per line. A line whose first non-space character is # is a comment.
# systemd parses this file itself. source does not reproduce that parse.
# An unquoted backslash escapes the next character. \\ is one backslash.
# Confirm a value that contains a backslash, a quote, or leading or trailing whitespace on the target host.
# Copy to /etc/vectornntp/storageservice.env as root:root mode 0600, then replace the placeholder with sudoedit.
# Do not commit the installed file.
ConnectionStrings__NntpDB=Server=REPLACE_WITH_MYSQL_HOST;User ID=REPLACE_WITH_MYSQL_USER;Password=REPLACE_WITH_MYSQL_PASSWORD;Database=REPLACE_WITH_DATABASE_NAME
# These shared names are not required in /etc/vectornntp/storageservice.env.
# StorageService does not take them from this file. nntpsharedconfig supplies them after the connection string works.
# VECTOR__RABBITMQ__USERNAME
# VECTOR__RABBITMQ__PASSWORD
# VECTOR__CLOUDFLAREAPIKEY
# VECTOR__CLOUDFLAREZONEID
# VECTOR__ACMEACCOUNT
# VECTOR__ACMECERTIFICATEPASSWORD
77 changes: 77 additions & 0 deletions deploy/systemd/vectornntp-storageservice.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
[Unit]
Description=VectorNNTP StorageService
# Install the flat linux-x64 release directory contents needed for this service
# at this path. The shared release also contains BackFiller binaries and units.
# This unit uses only VectorNNTPD.StorageService, VectorNNTPD.StorageService.json,
# VectorNNTPD.RabbitMQ.json, README.md, vectornntp-storageservice.service, and
# storageservice.env.example.
# Copy the unit to /etc/systemd/system/. Create user and group vectornntp,
# create logs/, certs/, db/, and spool/ owned by that account, and install
# /etc/vectornntp/storageservice.env before enable.
Documentation=file:///opt/vectornntp/storageservice/README.md
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main

# Dedicated non-root service account (create before enabling the unit).
# Same account as BackFiller is acceptable when both share /opt/vectornntp.
User=vectornntp
Group=vectornntp

# The executable directory is the configuration root. The process loads
# VectorNNTPD.StorageService.json and VectorNNTPD.RabbitMQ.json from that directory,
# not appsettings.json, RabbitMq.json, or the process working directory.
# Relative logs (logs/), ACME state (certs/), article index (db/), and spool
# roots also resolve there.
WorkingDirectory=/opt/vectornntp/storageservice
ExecStart=/opt/vectornntp/storageservice/VectorNNTPD.StorageService

# Host GracefulShutdownTimeout follows StorageServer:GracefulShutdownTimeout.
# The shipped value is 00:00:30. TimeoutStopSec must stay above that setting
# or systemd SIGKILLs an in-progress drain.
TimeoutStartSec=180
TimeoutStopSec=60

Restart=on-failure
RestartSec=5

# systemd reads this file as root before exec. The leading '-' suppresses errors
# reading the file and still starts the process. Current systemd ignores a missing,
# unreadable, or invalid file. That does not insert ConnectionStrings__NntpDB.
# Startup validation still fails when the variable is absent. The file is KEY=VALUE,
# not a shell script. An unquoted backslash is an escape. The connection string has
# no VECTOR__ prefix. RabbitMQ, ACME, and Cloudflare secrets come from
# nntpsharedconfig; this unit does not set them.
EnvironmentFile=-/etc/vectornntp/storageservice.env
# DOTNET_ENVIRONMENT selects the optional VectorNNTPD.StorageService.Production.json overlay.
Environment=DOTNET_ENVIRONMENT=Production

# Watchdog heartbeats run only when EnableWatchdog is true (the shipped default)
# and this unit sets WatchdogSec, which supplies WATCHDOG_USEC.
# Leave WatchdogSec unset until readiness has been checked on the target host.
# Example (enable only after validation):
# WatchdogSec=30
# Restart=on-watchdog

# File logging is the shipped destination (StorageServer:LogDir=logs).
# The shipped VectorNNTPD.StorageService.json also enables the Console sink, so
# Serilog Information+ lines reach stdout and the journal under this unit.
# Hardening. StorageService binds TLS and opens outbound RabbitMQ, MySQL, and Cloudflare.
# ProtectSystem=strict mounts the filesystem read-only. ReadWritePaths re-opens only
# logs/, certs/, db/, and spool/. It does not grant Unix ownership; those directories
# must already exist and be writable by the vectornntp account. The executable and
# JSON stay read-only.
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/vectornntp/storageservice/logs /opt/vectornntp/storageservice/certs /opt/vectornntp/storageservice/db /opt/vectornntp/storageservice/spool

KillSignal=SIGTERM
SendSIGKILL=yes

[Install]
WantedBy=multi-user.target
2 changes: 1 addition & 1 deletion docs/common/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ NNTPD, BackFiller, and StorageServer publish the ACME account email, ACME certif

Top-level `RabbitMQ` section. The Common sample is `src/VectorNNTPD.Common/VectorNNTPD.RabbitMQ.json`. `ApplicationJsonConfiguration.AddSharedRabbitMqJsonFile` loads only the filename the caller passes. BackFiller passes `VectorNNTPD.RabbitMQ.json`. The file is inserted after the entry-assembly JSON files and before environment variables and command-line arguments. Options types and connection validation live in `VectorNNTPD.Common.Messaging.RabbitMq`. Applications bind the section and call `AddRabbitMqInfrastructure()`. Nested `Management` is validated only when `BaseUrl` is set.

NNTPD and StorageServer are not projects in this repository, so their build output is not part of the contract here. BackFiller ships connectivity in `src/VectorNNTPD.BackFillerService/VectorNNTPD.RabbitMQ.json`. Its primary file, `VectorNNTPD.BackFillerService.json`, still must not repeat a top-level `RabbitMQ` section. `BackFiller:Logging:RabbitMQ` is the log sink, not broker connectivity.
NNTPD is not a project in this repository, so its build output is not part of the contract here. BackFiller and StorageService ship connectivity as `VectorNNTPD.RabbitMQ.json` beside each host project. Primary host files (`VectorNNTPD.BackFillerService.json`, `VectorNNTPD.StorageService.json`) must not repeat a top-level `RabbitMQ` section. `BackFiller:Logging:RabbitMQ` is the log sink, not broker connectivity.

`RabbitMqService` owns one process-wide connection, verifies that it is open, and replaces it on connectivity loss or when a broker block lasts longer than `ConnectionBlockedTimeoutSeconds`, while incrementing a monotonic generation. Client automatic recovery is disabled. The AMQP client name comes from `IRabbitMqConnectionNameProvider`. Callers obtain the current connection with `TryGetCurrent` and do not dispose it.

Expand Down
4 changes: 2 additions & 2 deletions src/VectorNNTPD.Common/VectorNNTPD.Common.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
<InternalsVisibleTo Include="VectorNNTP.NNTPD.Tests" />
<InternalsVisibleTo Include="VectorNNTPD.BackFillerService" />
<InternalsVisibleTo Include="VectorNNTPD.BackFillerService.Tests" />
<InternalsVisibleTo Include="VectorNNTP.StorageServer" />
<InternalsVisibleTo Include="VectorNNTP.StorageServer.Tests" />
<InternalsVisibleTo Include="VectorNNTPD.StorageService" />
<InternalsVisibleTo Include="VectorNNTPD.StorageService.Tests" />
<InternalsVisibleTo Include="VectorNNTP.StorageServer.PersistenceBench" />
<InternalsVisibleTo Include="VectorNNTP.StorageQualification" />
<!-- Storage smoke builds a pre-ingestion record with internal TryCreate(..., ArticlePathMode.Normalize).
Expand Down
Loading
Loading