Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,23 @@ Standalone VectorNNTPD hosts extracted from VectorNNTP.NNTPD. This repository cu
- `tools/Publish-Release.ps1` - Native AOT release publish for both production hosts
- `tools/Invoke-CodeQlAnalysis.ps1` - local traced C# CodeQL analysis

## Documentation

Permanent docs live under [`docs/`](docs/). Start with [`docs/README.md`](docs/README.md).

| Document | Contents |
|---|---|
| [docs/architecture.md](docs/architecture.md) | BackFiller host lifecycle, Article Work, byte-oriented data plane |
| [docs/configuration.md](docs/configuration.md) | BackFiller options, RabbitMQ, NntpDB, secrets (BackFiller-focused) |
| [docs/storage.md](docs/storage.md) | StorageService architecture and configuration |
| [docs/storage-nntp-feed.md](docs/storage-nntp-feed.md) | Storage NNTP feed (ARTICLE / TAKETHIS / QUIT) |
| [docs/logging.md](docs/logging.md) | Serilog for BackFiller and StorageService |
| [docs/systemd.md](docs/systemd.md) | systemd notify/watchdog for both hosts |
| [docs/common/](docs/common/) | Shared library contracts (ArticleRecord, VATP, Common config) |
| [docs/standards/](docs/standards/) | Local RFC and protocol standards library |
| [docs/schema/](docs/schema/) | Upstream Transit/PostFilter SQL references only — not applied by these hosts |
| [docs/reference/upstream-nntpd/](docs/reference/upstream-nntpd/) | Labeled upstream NNTPD notes (not VectorNNTPD behaviour) |

## Build

```powershell
Expand Down
68 changes: 68 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# VectorNNTPD documentation

This directory is the permanent documentation for the **VectorNNTPD** solution currently in `VectorNNTPD.slnx`: BackFillerService, StorageService, and Common (plus their tests).

ReaderService and TransitService may appear on a development machine as untracked working-tree directories; they are **not** part of the solution on `master` and are not documented here as shipping hosts.

## Start here

| Document | Audience | Contents |
|---|---|---|
| [../README.md](../README.md) | Operators / developers | Build, test, publish, Linux install overview |
| [architecture.md](architecture.md) | Developers | BackFiller host lifecycle, Article Work, NNTP client/listener roles, silent-consumer monitoring, byte-oriented data plane |
| [configuration.md](configuration.md) | Operators / developers | **BackFiller only** — options, JSON locations, RabbitMQ, NntpDB, secrets |
| [storage.md](storage.md) | Operators / developers | **StorageService** architecture, configuration, persistence, messaging, ops |
| [storage-nntp-feed.md](storage-nntp-feed.md) | Developers / integrators | Storage NNTP feed: ARTICLE / TAKETHIS / QUIT only |
| [logging.md](logging.md) | Operators / developers | Serilog-only logging for BackFiller and StorageService |
| [systemd.md](systemd.md) | Operators | systemd notify/watchdog install and operations for both hosts |

Redis / PostFilter / Transit DDL under [schema/](schema/) are **not** Storage or BackFiller startup requirements.

## Shared library

| Document | Contents |
|---|---|
| [common/README.md](common/README.md) | What Common owns vs application hosts |
| [common/architecture.md](common/architecture.md) | Common boundaries |
| [common/configuration.md](common/configuration.md) | Shared configuration helpers |
| [common/ArticleRecord.md](common/ArticleRecord.md) | Canonical article envelope |
| [common/vatp.md](common/vatp.md) | Vector Article Transfer Protocol |
| [common/systemd-host-abstraction.md](common/systemd-host-abstraction.md) | **Proposal only** — comparing Storage vs BackFiller systemd stacks |

## Standards and schema

| Path | Contents |
|---|---|
| [standards/rfcs/](standards/rfcs/) | Local NNTP / Usenet RFC text library (authoritative for protocol work in this repo) |
| [standards/spamd/](standards/spamd/) | SPAMC/SPAMD protocol text — **upstream PostFilter interoperability**; not a VectorNNTPD host feature |
| [standards/haproxy/](standards/haproxy/) | PROXY protocol text — standards library only; not an implementation claim for BackFiller/Storage |
| [standards/pgpverify/](standards/pgpverify/) | INN pgpverify format notes — interoperability reference; BackFiller does not implement PGPVERIFY |
| [schema/](schema/) | **Upstream-only** Transit / PostFilter / `account_art_type` SQL — see [schema/README.md](schema/README.md) |

## Upstream NNTPD reference (not hosts in this solution)

Material from `VectorNNTP.NNTPD` for engineers who need command or PostFilter context when
working beside an NNTPD deployment. Paths such as `src/VectorNNTP.NNTPD/...` refer to the
**upstream** repository. They are **not** an implementation guarantee for
`VectorNNTPD.BackFillerService` or `VectorNNTPD.StorageService`.

For what this solution actually speaks on the wire, use [architecture.md](architecture.md)
(BackFiller upstream client + cache listener) and [storage-nntp-feed.md](storage-nntp-feed.md)
(Storage ARTICLE / TAKETHIS / QUIT). Do not treat the upstream command checklist as Storage or
BackFiller CAPABILITIES.

Upstream Redis and PostFilter configuration are **not** required to operate VectorNNTPD hosts.

| Document | Topic |
|---|---|
| [reference/upstream-nntpd/commands.md](reference/upstream-nntpd/commands.md) | Upstream NNTPD command inventory |
| [reference/upstream-nntpd/postfilter.md](reference/upstream-nntpd/postfilter.md) | Upstream PostFilter operator guide |

## Agent and contribution docs (repository root)

| Document | Contents |
|---|---|
| [../AGENTS.md](../AGENTS.md) | Cursor agent entry point |
| [../CONTRIBUTING.md](../CONTRIBUTING.md) | Contribution guidance |
| [../SECURITY.md](../SECURITY.md) | Security reporting |
| [../CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Conduct |
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ The listener command path is byte-oriented, as is the upstream NNTP client. They

## Source-generated structured logging

Application logs use `LoggerMessageAttribute` on partial methods, usually in a `*LogMessages.cs` companion. Serilog is the host logging stack. The rolling gzip file sink, compressor, active-file writer, and periodic flush wrapper live in `VectorNNTPD.Common.Logging`. BackFiller selects targets, levels, retention, syslog, and RabbitMQ publishing. Message templates keep structured property names. Do not preformat those properties into the message string.
Application logs use `LoggerMessageAttribute` on partial methods, usually in a `*LogMessages.cs` companion. Serilog is the host logging stack. The rolling gzip file sink, compressor, active-file writer, and periodic flush wrapper live in `VectorNNTPD.Common.Logging`. BackFiller selects targets, levels, retention, syslog, and RabbitMQ publishing. Message templates keep structured property names. Do not preformat those properties into the message string. Operator-facing sink and systemd details: [logging.md](logging.md), [systemd.md](systemd.md).

Do not log Cloudflare API keys, ACME passwords, RabbitMQ passwords, `AUTHINFO` secrets, or article payloads. NNTP wire debug logs redact `AUTHINFO USER` and `AUTHINFO PASS`. Guard expensive trace or debug arguments with `IsEnabled` when building them has a cost.

Expand Down
139 changes: 139 additions & 0 deletions docs/logging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# VectorNNTPD — Logging (Serilog)

Serilog is the **only** logging implementation for both production hosts in this solution
(`VectorNNTPD.BackFillerService` and `VectorNNTPD.StorageService`). Application code uses
`Microsoft.Extensions.Logging.ILogger<T>`; those resolve to Serilog sinks.

This document describes **current** BackFiller and StorageService behaviour. It does **not**
document INN `news` / Path-survey (`inpaths`) streams; those belong to upstream
`VectorNNTP.NNTPD` and are not hosts in `VectorNNTPD.slnx`.

Shared rolling-file primitives live in `VectorNNTPD.Common.Logging`
(`GzipRollingFileSink`, compressor, active-file writer, periodic flush). Prefer
`LoggerMessageAttribute` + partial methods for application messages; keep structured
property names and do not preformat them into the message string. See
[architecture.md](architecture.md#source-generated-structured-logging).

## Abstractions vs implementation

| Layer | Role |
|-------|------|
| `ILogger<T>` / `ILoggerFactory` | Application and framework logging API |
| `SerilogLoggerFactory` | Exclusive MEL factory registered by `AddSerilog` |
| Serilog sinks | Destinations (file, optional console / syslog / RabbitMQ depending on host) |

Do not re-add `AddConsole`, `AddDebug`, or `AddEventLog` as MEL providers. Both hosts clear
MEL providers and remove the default `ILoggerFactory` before registering Serilog.

`AddSystemd()` remains for systemd lifetime and notify support only. Any Microsoft console
formatter registration it may add is stripped afterward; Serilog owns console formatting when
a console sink is enabled.

## BackFillerService

### Configuration surface

BackFiller does **not** read a `Serilog` JSON section. The only bindable surface is
`BackFiller:Logging` in `VectorNNTPD.BackFillerService.json` (see
[configuration.md](configuration.md)).

| Setting | Default | Meaning |
|---------|---------|---------|
| `LogLevel` | `Information` | Serilog minimum applied to enabled targets (`Verbose`…`Fatal`) |
| `LogRetentionDays` | `14` | Daily file retention when the File target is enabled (range 1–3650) |
| `Json` | `false` | When true, enabled text-capable sinks use JSON formatting |
| `File:Enabled` | `true` | Rolling gzip file sink |
| `File:LogDir` | `logs` | Directory; required when File is enabled. Relative paths use Common `ApplicationLocalPath` against the application binary directory |
| `Syslog:Enabled` | `false` | Optional UDP/TCP syslog |
| `RabbitMQ:Enabled` | `false` | Optional publish via existing `IRabbitMqService` (`Exchange` / `RoutingKey`; broker credentials stay in `nntpsharedconfig` / RabbitMQ connectivity JSON) |

Sink graph construction is code-owned (`BackFillerFileLogging`) for Native AOT / single-file safety.
It does not call `ReadFrom.Configuration`.

### Console

The console sink is **off** unless the process is started with `--console`. Under the shipped
systemd unit, journald still receives stdout/stderr from the process, but Serilog does not
write application events to the console sink by default. See the unit comments in
`deploy/systemd/vectornntp-backfiller.service`.

### File sink behaviour

- Rolling path: `{LogDir}/{entry assembly name}-yyyyMMdd.log` (Serilog day token `-.log`).
- Daily roll; `rollOnFileSizeLimit` is false (no size-based roll).
- Completed days are gzip-compressed (`CompressionLevel.Fastest`) via Common
`GzipRollingFileSink`; the active file is never compressed or deleted.
- Async wrapper: buffer size `50000`, **`blockWhenFull: false`** so a stalled file sink drops
excess events instead of blocking article-path loggers. Critical diagnostics may be lost under
sustained backpressure; prefer ops alerts on log-volume disk pressure.
- File sink is buffered with about a one-second flush interval.

### Bootstrap

`Program` assigns a bootstrap logger before the host is built so configuration and construction
failures are recorded, then flushes on shutdown (`Log.CloseAndFlushAsync`). After `host.Build()`,
an Information “logging initialized” event proves the Serilog factory is live (no secrets).

## StorageService

Host architecture and non-logging options: [storage.md](storage.md).

### Configuration surface

StorageService uses a `Serilog` section in `VectorNNTPD.StorageService.json` for minimum levels,
enrichment, and `WriteTo` arguments, plus `StorageServer:LogDir` for the directory.

| Setting | Source of truth |
|---------|-----------------|
| Console / File / Async arguments | `Serilog:WriteTo` |
| Log directory | `StorageServer:LogDir` (shipped `logs`; code default constant is `/logs` if the key is blank) |
| Path rewrite | `StorageServerFileLogging.BindResolvedFilePath` creates the directory and overwrites the File `path` so the JSON placeholder is never the runtime path |

Native AOT cannot rely on reflection discovery of sink extension methods for every `WriteTo`
entry. Storage applies MinimumLevel / enrichment via `ReadFrom.Configuration` while registering
Console / Async / File sinks explicitly (`StorageServerConfiguredSinks`).

### Console and journald

The shipped JSON enables the Console sink (Debug+ restricted level on that sink; global default
Information with `VectorNNTPD.StorageService` overridden to Debug). Under systemd, Information+
lines reach stdout and journald. See `deploy/systemd/vectornntp-storageservice.service`.

### File sink behaviour

- Same daily `{entry assembly name}-yyyyMMdd.log` naming and Common gzip rotation as BackFiller.
- Shipped Async args: `bufferSize` 50000, **`blockWhenFull: true`** (logging calls block when the
async queue is full — deliberate difference from BackFiller).
- `retainedFileCountLimit` 14 in the shipped JSON.
- `buffered` true; `flushToDiskInterval` `00:00:01`.

### Bootstrap

Same pattern as BackFiller: bootstrap logger, `ConfigureStorageServerLogging`, then
`WriteLoggingInitialized` after the host is built.

## Host comparison

| Topic | BackFiller | StorageService |
|-------|------------|----------------|
| Config keys | `BackFiller:Logging` only | `Serilog` + `StorageServer:LogDir` |
| Console by default | Off (`--console` to enable) | On (shipped JSON) |
| Async `blockWhenFull` | `false` (drop under backpressure) | `true` (block) |
| Optional syslog / RabbitMQ log targets | Yes | No (not on this host’s logging options) |
| Shared gzip file implementation | Common `GzipRollingFileSink` | Same |

## Under systemd

See [systemd.md](systemd.md). File logs remain under each install tree’s `logs/` (unit
`ReadWritePaths`). Journal queries:

```bash
sudo journalctl -u vectornntp-backfiller.service -f
sudo journalctl -u vectornntp-storageservice.service -f
```

## What this repository does not implement

- INN-compatible `news` disposition log and Path-survey `inpaths` files (upstream NNTPD).
- Operator-configurable INN field order via `outputTemplate` for those streams.
- Microsoft MEL console / debug / event-log providers alongside Serilog.
Loading
Loading