Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions .github/codeql/codeql-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ name: BackFiller CodeQL configuration
queries:
- uses: ./.github/codeql/csharp-with-alert-suppression.qls

# paths-ignore documents intent for non-traced languages / documentation.
# For traced C# builds, compiled obj/bin (including source-generators) still appear in
# SARIF; tools/Filter-CodeQlSarif.ps1 and .github/workflows/codeql.yml strip those
# results before upload. See tools/Invoke-CodeQlAnalysis.ps1.
paths-ignore:
- '**/bin/**'
- '**/obj/**'
380 changes: 306 additions & 74 deletions .github/workflows/build.yml

Large diffs are not rendered by default.

34 changes: 33 additions & 1 deletion .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,40 @@ jobs:
- name: Restore solution
run: dotnet restore VectorNNTPD.slnx -p:Platform=x64

# --no-incremental ensures CoreCompile/csc runs under the CodeQL tracer for every
# project. An up-to-date incremental skip can leave only .csproj entries in the DB.
- name: Build solution
run: dotnet build VectorNNTPD.slnx --configuration Release -p:Platform=x64 --no-restore
run: dotnet build VectorNNTPD.slnx --configuration Release -p:Platform=x64 --no-restore --no-incremental

# upload: never — traced C# SARIF includes **/obj/** source-generator noise that
# paths-ignore cannot drop. Filter first, then upload-sarif.
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
with:
upload: never
output: codeql-sarif

- name: Filter generated bin/obj findings from SARIF
shell: pwsh
run: |
$inputs = Get-ChildItem -Path codeql-sarif -Filter *.sarif -File -ErrorAction Stop
if ($inputs.Count -lt 1) {
throw "No SARIF files found under codeql-sarif/"
}
New-Item -ItemType Directory -Force -Path codeql-sarif-filtered | Out-Null
foreach ($sarif in $inputs) {
$out = Join-Path codeql-sarif-filtered $sarif.Name
# StripAcceptedSuppressions: in-source // codeql[...] findings stay justified in
# the repo; uploading them as results still produced "new alerts" on PR #256
# because CodeQL's {state:accepted} was not treated as GitHub dismissals.
& ./tools/Filter-CodeQlSarif.ps1 -InputPath $sarif.FullName -OutputPath $out -StripAcceptedSuppressions
}

# Keep the historical analysis category so GitHub's required CodeQL check continues
# to match refs/heads/master (category "csharp" yielded NEUTRAL
# "1 configuration not found" on PR #256 check run 123524079410).
- name: Upload filtered CodeQL SARIF
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: codeql-sarif-filtered
category: ".github/workflows/codeql.yml:analyze"
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Concise routing for Cursor agents (including JetBrains Rider ACP). This file is an entry point; it does not prove that Rider ACP automatically loads it.

This repository is standalone **VectorNNTPD.BackFillerService**. It was extracted from VectorNNTP.NNTPD. NNTPD names that remain in code comments or shared configuration are provenance or shared-domain context. The solution, projects, and verification commands are VectorNNTPD's.
This repository is **VectorNNTPD**: BackFillerService, StorageService, NNTP ReaderService, NNTP TransitService, and `VectorNNTPD.Common` (Reader/Transit embed duplicated NNTP server code; there is no `VectorNNTPD.NntpServer` library). It was extracted from VectorNNTP.NNTPD. NNTPD names that remain in code comments or shared configuration are provenance or shared-domain context. Reader/Transit hosting is documented in [`docs/nntpd-reader-transit.md`](docs/nntpd-reader-transit.md).

## Before substantive work

Expand Down
26 changes: 17 additions & 9 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,22 @@
<Copyright>©2026 Chris Knipe &lt;cknipe@opticnetworks.net&gt;</Copyright>
</PropertyGroup>

<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.ReaderService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.TransitService'">
<Copyright>© 2026 Chris Knipe &lt;cknipe@opticnetworks.net&gt;</Copyright>
</PropertyGroup>

<!--
Production executable version: 1.0.<days since 2000-01-01>.<seconds since local midnight / 2>.
One DateTime.Now tick count feeds both components. Deterministic compilation stays enabled.
Libraries and tools are left on the SDK default version.
-->
<PropertyGroup>
<_IsProductionExecutable Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.ReaderService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.TransitService'">true</_IsProductionExecutable>
</PropertyGroup>

<Target Name="ApplyDateTimeApplicationVersion"
BeforeTargets="GetAssemblyVersion"
Condition="'$(DesignTimeBuild)' != 'true' AND ('$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService')">
Condition="'$(DesignTimeBuild)' != 'true' AND '$(_IsProductionExecutable)' == 'true'">
<PropertyGroup>
<_ApplicationVersionNow>$([System.DateTime]::Now.Ticks)</_ApplicationVersionNow>
<_ApplicationVersionBuild>$([System.DateTime]::new($([System.Int64]::Parse($(_ApplicationVersionNow)))).Subtract($([System.DateTime]::new(2000,1,1))).Days)</_ApplicationVersionBuild>
Expand All @@ -38,23 +46,23 @@
Production publish is self-contained Native AOT for win-x64 and linux-x64 only.
A normal build does not set RuntimeIdentifier, so libraries, tests, and tools stay portable.
-->
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService'">
<PropertyGroup Condition="'$(_IsProductionExecutable)' == 'true'">
<RuntimeIdentifiers>win-x64;linux-x64</RuntimeIdentifiers>
<SelfContained Condition="'$(RuntimeIdentifier)' == 'win-x64' OR '$(RuntimeIdentifier)' == 'linux-x64'">true</SelfContained>
<PublishAot Condition="'$(RuntimeIdentifier)' == 'win-x64' OR '$(RuntimeIdentifier)' == 'linux-x64'">true</PublishAot>
</PropertyGroup>

<!--
StorageService still uses Serilog.Settings.Configuration for MinimumLevel/enrichment while
sinks are registered explicitly via StorageServerConfiguredSinks. Native AOT/single-file
StorageService, ReaderService, and TransitService use Serilog.Settings.Configuration for
MinimumLevel/enrichment while sinks are registered explicitly. Native AOT/single-file
analysis then fails publish on Serilog.Settings.Configuration:
IL3000/IL3002 (Assembly.Location / DependencyContext in single-file) and IL2104/IL3050
(assembly trim/AOT analysis). Scope is StorageService + PublishAot only. BackFiller does
(assembly trim/AOT analysis). Scope is those hosts + PublishAot only. BackFiller does
not need this because it never calls ReadFrom.Configuration.
-->
<PropertyGroup Condition="'$(MSBuildProjectName)' == 'VectorNNTPD.StorageService' AND '$(PublishAot)' == 'true'">
<NoWarn>$(NoWarn);IL2104;IL3050;IL3000;IL3002</NoWarn>
<WarningsNotAsErrors>$(WarningsNotAsErrors);IL2104;IL3050</WarningsNotAsErrors>
<PropertyGroup Condition="('$(MSBuildProjectName)' == 'VectorNNTPD.StorageService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.ReaderService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.TransitService') AND '$(PublishAot)' == 'true'">
<NoWarn>$(NoWarn);IL2026;IL2104;IL3050;IL3000;IL3002</NoWarn>
<WarningsNotAsErrors>$(WarningsNotAsErrors);IL2026;IL2104;IL3050</WarningsNotAsErrors>
<ILLinkTreatWarningsAsErrors>false</ILLinkTreatWarningsAsErrors>
<IlcTreatWarningsAsErrors>false</IlcTreatWarningsAsErrors>
</PropertyGroup>
Expand All @@ -67,7 +75,7 @@

<Target Name="RejectUnsupportedProductionRuntimeIdentifier"
BeforeTargets="Restore;Build;Publish"
Condition="('$(MSBuildProjectName)' == 'VectorNNTPD.BackFillerService' OR '$(MSBuildProjectName)' == 'VectorNNTPD.StorageService') AND '$(RuntimeIdentifier)' != '' AND '$(RuntimeIdentifier)' != 'win-x64' AND '$(RuntimeIdentifier)' != 'linux-x64'">
Condition="'$(_IsProductionExecutable)' == 'true' AND '$(RuntimeIdentifier)' != '' AND '$(RuntimeIdentifier)' != 'win-x64' AND '$(RuntimeIdentifier)' != 'linux-x64'">
<Error Text="Production applications publish only win-x64 and linux-x64. RuntimeIdentifier '$(RuntimeIdentifier)' is not supported." />
</Target>

Expand Down
40 changes: 29 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
# VectorNNTPD

Standalone VectorNNTPD hosts extracted from VectorNNTP.NNTPD. This repository currently ships BackFiller and StorageService, plus shared `VectorNNTPD.Common`.
Standalone VectorNNTPD hosts extracted from VectorNNTP.NNTPD. This repository ships BackFiller, StorageService, NNTP ReaderService, NNTP TransitService, and shared `VectorNNTPD.Common`. Reader and Transit each embed their own NNTP server implementation (no shared NntpServer library).

## Structure

- `src/VectorNNTPD.BackFillerService/VectorNNTPD.BackFillerService.csproj` - BackFiller worker app (net10.0)
- `src/VectorNNTPD.StorageService/VectorNNTPD.StorageService.csproj` - StorageService / NNTP TAKETHIS feed host (net10.0)
- `src/VectorNNTPD.Common/VectorNNTPD.Common.csproj` - shared library
- `tests/VectorNNTPD.BackFillerService.Tests/` / `tests/VectorNNTPD.StorageService.Tests/` / `tests/VectorNNTPD.Common.Tests/` - xUnit suites
- `src/VectorNNTPD.BackFillerService/` - BackFiller worker app (net10.0)
- `src/VectorNNTPD.StorageService/` - StorageService / NNTP TAKETHIS feed host (net10.0)
- `src/VectorNNTPD.ReaderService/` - NNTP reader-server host (`AddNntpReaderHosting`)
- `src/VectorNNTPD.TransitService/` - NNTP transit-server host (`AddNntpTransitHosting`)
- `src/VectorNNTPD.Common/` - shared library
- `tests/VectorNNTPD.BackFillerService.Tests/` / `tests/VectorNNTPD.StorageService.Tests/` / `tests/VectorNNTPD.ReaderService.Tests/` / `tests/VectorNNTPD.TransitService.Tests/` / `tests/VectorNNTPD.Common.Tests/` - xUnit suites
- `docs/nntpd-reader-transit.md` - Reader/Transit architecture and deployment notes
- `VectorNNTPD.slnx` - solution including apps + tests
- `tools/Publish-Release.ps1` - Native AOT release publish for both production hosts
- `tools/Publish-Release.ps1` - Native AOT release publish for BackFiller, StorageService, ReaderService, and TransitService
- `tools/Invoke-CodeQlAnalysis.ps1` - local traced C# CodeQL analysis

## Build
Expand All @@ -26,13 +29,24 @@ dotnet test .\tests\VectorNNTPD.BackFillerService.Tests\VectorNNTPD.BackFillerSe

## Publish

Preferred release path (both hosts into one flat directory):
Preferred release path (one independent package directory per service):

```powershell
.\tools\Publish-Release.ps1 -Platform win-64
.\tools\Publish-Release.ps1 -Platform linux-64
```

Output layout:

```text
release/<platform>/backfiller/
release/<platform>/storage/
release/<platform>/reader/
release/<platform>/transit/
```

Each package contains that host’s Native AOT executable, its primary JSON, and `VectorNNTPD.RabbitMQ.json`. Linux packages also include that role’s systemd unit, environment example, and `README.md`. Reader and Transit never share a package directory. CI writes the same package names under `.artifacts/win-x64/` and `.artifacts/linux-x64/`.

Per-project Native AOT publish still works. Example for StorageService:

```powershell
Expand All @@ -45,6 +59,8 @@ Native AOT publish keeps trim and AOT analysis enabled. CI runs each published e

- BackFiller: `VECTORNNTP_BACKFILLER_STARTUP_SMOKE=1` / `VECTORNNTP_BACKFILLER_STARTUP_SMOKE_CONFIG` → checkpoint `BackFiller startup smoke: configuration serverId=1 fqdn=backfiller01.usenet.ninja`
- StorageService: `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE=1` / `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE_CONFIG` → checkpoint `StorageService startup smoke: configuration serverId=1 fqdn=cache01.usenet.ninja`
- ReaderService: `VECTORNNTP_READER_STARTUP_SMOKE=1` / `VECTORNNTP_READER_STARTUP_SMOKE_CONFIG` → checkpoint `ReaderService startup smoke: configuration serverId=1 fqdn=reader01.usenet.ninja`
- TransitService: `VECTORNNTP_TRANSIT_STARTUP_SMOKE=1` / `VECTORNNTP_TRANSIT_STARTUP_SMOKE_CONFIG` → checkpoint `TransitService startup smoke: configuration serverId=1 fqdn=transit01.usenet.ninja`

Those checkpoints prove configuration binding and host construction (and StorageService logging initialization). They do not open a listener or contact a database, broker, DNS provider, or certificate authority. StorageService AOT publish still scopes Serilog.Settings.Configuration IL suppressions (`IL2104`/`IL3050`/`IL3000`/`IL3002`) because MinimumLevel is applied via `ReadFrom.Configuration` while sinks are registered explicitly.

Expand All @@ -56,13 +72,15 @@ BackFiller references `VectorNNTPD.Common` for `IApplicationService`, NntpDB, AC

## Linux service

`tools/Publish-Release.ps1 -Platform linux-64` writes `release/linux-64/`. The Linux CI job writes `.artifacts/linux-x64/` and uploads that directory. Both contain the flat Native AOT outputs for `VectorNNTPD.BackFillerService` and `VectorNNTPD.StorageService`, their primary JSON files, shared `VectorNNTPD.RabbitMQ.json`, `README.md`, and the BackFiller and StorageService systemd units plus environment-file examples.
`tools/Publish-Release.ps1 -Platform linux-64` writes per-service packages under `release/linux-64/{backfiller,storage,reader,transit}/`. The Linux CI job writes the same package names under `.artifacts/linux-x64/` and uploads that tree. Each package is independently installable (executable, host JSON, RabbitMQ JSON, that role’s systemd unit and environment example, and `README.md`).

Install BackFiller from `backfiller/` at `/opt/vectornntp/backfiller`. `deploy/systemd/vectornntp-backfiller.service` uses that path for `ExecStart`, `WorkingDirectory`, and `Documentation=`. `ReadWritePaths` lists only `logs/` and `certs/`. Under `ProtectSystem=strict` that re-opens those two directories for writing. It does not change Unix ownership or mode. The shipped JSON writes file logs under `logs/` and ACME state under `certs/`, and startup creates files inside directories that must already exist.

Install BackFiller at `/opt/vectornntp/backfiller`. `deploy/systemd/vectornntp-backfiller.service` uses that path for `ExecStart`, `WorkingDirectory`, and `Documentation=`. `ReadWritePaths` lists only `logs/` and `certs/`. Under `ProtectSystem=strict` that re-opens those two directories for writing. It does not change Unix ownership or mode. The shipped JSON writes file logs under `logs/` and ACME state under `certs/`, and startup creates files inside directories that must already exist.
Install StorageService from `storage/` at `/opt/vectornntp/storageservice` using `VectorNNTPD.StorageService`, `VectorNNTPD.StorageService.json`, `VectorNNTPD.RabbitMQ.json`, `vectornntp-storageservice.service`, and `storageservice.env.example`. That unit’s `ReadWritePaths` also include `db/` and `spool/` for the article index and segment-pool roots.

Install StorageService at `/opt/vectornntp/storageservice` using `VectorNNTPD.StorageService`, `VectorNNTPD.StorageService.json`, `VectorNNTPD.RabbitMQ.json`, `vectornntp-storageservice.service`, and `storageservice.env.example` from the same release directory. That unit’s `ReadWritePaths` also include `db/` and `spool/` for the article index and segment-pool roots.
Install ReaderService from `reader/` at `/opt/vectornntp/reader` and TransitService from `transit/` at `/opt/vectornntp/transit`. See `docs/nntpd-reader-transit.md` for the Reader/Transit deployment handoff (environment files, schema, peers, rollback).

Create the account and install the release as root-owned files. This step copies the placeholder environment file and reloads systemd. It does not start the service.
Create the account and install the release as root-owned files. Run the following from the `backfiller/` package directory. This step copies the placeholder environment file and reloads systemd. It does not start the service.

```bash
sudo groupadd --system vectornntp || true
Expand Down
12 changes: 12 additions & 0 deletions VectorNNTPD.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,15 @@
<Project Path="src/VectorNNTPD.Common/VectorNNTPD.Common.csproj">
<Platform Project="x64" />
</Project>
<Project Path="src/VectorNNTPD.ReaderService/VectorNNTPD.ReaderService.csproj">
<Platform Project="x64" />
</Project>
<Project Path="src/VectorNNTPD.StorageService/VectorNNTPD.StorageService.csproj">
<Platform Project="x64" />
</Project>
<Project Path="src/VectorNNTPD.TransitService/VectorNNTPD.TransitService.csproj">
<Platform Project="x64" />
</Project>
</Folder>
<Folder Name="/tests/">
<Project Path="tests/VectorNNTPD.BackFillerService.Tests/VectorNNTPD.BackFillerService.Tests.csproj">
Expand All @@ -20,8 +26,14 @@
<Project Path="tests/VectorNNTPD.Common.Tests/VectorNNTPD.Common.Tests.csproj">
<Platform Project="x64" />
</Project>
<Project Path="tests/VectorNNTPD.ReaderService.Tests/VectorNNTPD.ReaderService.Tests.csproj">
<Platform Project="x64" />
</Project>
<Project Path="tests/VectorNNTPD.StorageService.Tests/VectorNNTPD.StorageService.Tests.csproj">
<Platform Project="x64" />
</Project>
<Project Path="tests/VectorNNTPD.TransitService.Tests/VectorNNTPD.TransitService.Tests.csproj">
<Platform Project="x64" />
</Project>
</Folder>
</Solution>
13 changes: 13 additions & 0 deletions deploy/systemd/reader.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Installed as /etc/vectornntp/reader.env (root:root mode 0600).
# KEY=VALUE only. Unquoted backslash is an escape.
#
# Required database connection (no VECTOR__ prefix):
ConnectionStrings__NntpDB=REPLACE_WITH_MYSQL_CONNECTION_STRING
#
# Required AES-256 XTrace key (64 hex or Base64). Never commit the real value.
NNTPD__XTRACEKEY=REPLACE_WITH_32_BYTE_KEY_AS_64_HEX_OR_BASE64
#
# Site-specific Nntpd overrides use Nntpd__* (section name Nntpd is preserved).
# Do not invent bind addresses or zone ids here; set them for the deployment.
# Example: Nntpd__ServerId=1
# Example: Nntpd__DnsSuffix=example.test
13 changes: 13 additions & 0 deletions deploy/systemd/transit.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Installed as /etc/vectornntp/transit.env (root:root mode 0600).
# KEY=VALUE only. Unquoted backslash is an escape.
#
# Required database connection (no VECTOR__ prefix):
ConnectionStrings__NntpDB=REPLACE_WITH_MYSQL_CONNECTION_STRING
#
# Required AES-256 XTrace key (64 hex or Base64). Never commit the real value.
NNTPD__XTRACEKEY=REPLACE_WITH_32_BYTE_KEY_AS_64_HEX_OR_BASE64
#
# Site-specific Nntpd overrides use Nntpd__* (section name Nntpd is preserved).
# Do not invent bind addresses, zone ids, or peer catalogues here; set them for the deployment.
# Example: Nntpd__ServerId=1
# Example: Nntpd__DnsSuffix=example.test
48 changes: 48 additions & 0 deletions deploy/systemd/vectornntp-reader.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
[Unit]
Description=VectorNNTP ReaderService
# Install VectorNNTPD.ReaderService, VectorNNTPD.ReaderService.json,
# VectorNNTPD.RabbitMQ.json, and this unit. Create user/group vectornntp,
# create logs/ and certs/ owned by that account, and install
# /etc/vectornntp/reader.env before enable.
Documentation=file:///opt/vectornntp/reader/README.md
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=main

User=vectornntp
Group=vectornntp

# Configuration root is the executable directory (not the process CWD alone).
WorkingDirectory=/opt/vectornntp/reader
ExecStart=/opt/vectornntp/reader/VectorNNTPD.ReaderService

# Align TimeoutStopSec above Nntpd graceful shutdown (shipped default 30s).
TimeoutStartSec=180
TimeoutStopSec=60

Restart=on-failure
RestartSec=5

# Requires ConnectionStrings__NntpDB. RabbitMQ/ACME/Cloudflare secrets come from
# nntpsharedconfig when configured that way; this unit does not set them.
EnvironmentFile=-/etc/vectornntp/reader.env
Environment=DOTNET_ENVIRONMENT=Production

# WatchdogSec example (enable only after readiness is verified):
# WatchdogSec=30
# Restart=on-watchdog

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/vectornntp/reader/logs /opt/vectornntp/reader/certs

KillSignal=SIGTERM
SendSIGKILL=yes

[Install]
WantedBy=multi-user.target
Loading
Loading