From cfb7527084ea962e42d564dc84f13e7119be934b Mon Sep 17 00:00:00 2001 From: "savage@savage.za.org" Date: Thu, 29 Oct 2026 18:11:38 +0200 Subject: [PATCH] Migrate and update VectorNNTPD documentation - Add StorageService architecture and configuration guide - Document NNTP feed command and response behaviour - Migrate logging, systemd, schema, and standards references - Clarify upstream-only documentation and database schemas - Update documentation index and cross-references --- README.md | 17 + docs/README.md | 68 ++ docs/architecture.md | 2 +- docs/logging.md | 139 +++ docs/reference/upstream-nntpd/commands.md | 153 +++ docs/reference/upstream-nntpd/postfilter.md | 586 +++++++++++ docs/schema/nntptransit.sql | 436 ++++++++ docs/schema/nntpusers-account-art-type.sql | 27 + docs/schema/postfilter.sql | 200 ++++ docs/standards/haproxy/README.md | 40 + docs/standards/haproxy/proxy-protocol.txt | 1053 +++++++++++++++++++ docs/standards/pgpverify/FORMAT | 251 +++++ docs/standards/pgpverify/README.md | 62 ++ docs/standards/spamd/PROTOCOL.md | 210 ++++ docs/standards/spamd/README.md | 26 + docs/storage-nntp-feed.md | 127 +++ docs/storage.md | 246 +++++ docs/systemd.md | 211 ++++ 18 files changed, 3853 insertions(+), 1 deletion(-) create mode 100644 docs/README.md create mode 100644 docs/logging.md create mode 100644 docs/reference/upstream-nntpd/commands.md create mode 100644 docs/reference/upstream-nntpd/postfilter.md create mode 100644 docs/schema/nntptransit.sql create mode 100644 docs/schema/nntpusers-account-art-type.sql create mode 100644 docs/schema/postfilter.sql create mode 100644 docs/standards/haproxy/README.md create mode 100644 docs/standards/haproxy/proxy-protocol.txt create mode 100644 docs/standards/pgpverify/FORMAT create mode 100644 docs/standards/pgpverify/README.md create mode 100644 docs/standards/spamd/PROTOCOL.md create mode 100644 docs/standards/spamd/README.md create mode 100644 docs/storage-nntp-feed.md create mode 100644 docs/storage.md create mode 100644 docs/systemd.md diff --git a/README.md b/README.md index a73b419a..050a4c12 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,23 @@ Standalone VectorNNTPD hosts extracted from VectorNNTP.NNTPD. This repository cu - `tools/Publish-Release.ps1` - Native AOT release publish for both production hosts - `tools/Invoke-CodeQlAnalysis.ps1` - local traced C# CodeQL analysis +## Documentation + +Permanent docs live under [`docs/`](docs/). Start with [`docs/README.md`](docs/README.md). + +| Document | Contents | +|---|---| +| [docs/architecture.md](docs/architecture.md) | BackFiller host lifecycle, Article Work, byte-oriented data plane | +| [docs/configuration.md](docs/configuration.md) | BackFiller options, RabbitMQ, NntpDB, secrets (BackFiller-focused) | +| [docs/storage.md](docs/storage.md) | StorageService architecture and configuration | +| [docs/storage-nntp-feed.md](docs/storage-nntp-feed.md) | Storage NNTP feed (ARTICLE / TAKETHIS / QUIT) | +| [docs/logging.md](docs/logging.md) | Serilog for BackFiller and StorageService | +| [docs/systemd.md](docs/systemd.md) | systemd notify/watchdog for both hosts | +| [docs/common/](docs/common/) | Shared library contracts (ArticleRecord, VATP, Common config) | +| [docs/standards/](docs/standards/) | Local RFC and protocol standards library | +| [docs/schema/](docs/schema/) | Upstream Transit/PostFilter SQL references only — not applied by these hosts | +| [docs/reference/upstream-nntpd/](docs/reference/upstream-nntpd/) | Labeled upstream NNTPD notes (not VectorNNTPD behaviour) | + ## Build ```powershell diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..5ccfc6eb --- /dev/null +++ b/docs/README.md @@ -0,0 +1,68 @@ +# VectorNNTPD documentation + +This directory is the permanent documentation for the **VectorNNTPD** solution currently in `VectorNNTPD.slnx`: BackFillerService, StorageService, and Common (plus their tests). + +ReaderService and TransitService may appear on a development machine as untracked working-tree directories; they are **not** part of the solution on `master` and are not documented here as shipping hosts. + +## Start here + +| Document | Audience | Contents | +|---|---|---| +| [../README.md](../README.md) | Operators / developers | Build, test, publish, Linux install overview | +| [architecture.md](architecture.md) | Developers | BackFiller host lifecycle, Article Work, NNTP client/listener roles, silent-consumer monitoring, byte-oriented data plane | +| [configuration.md](configuration.md) | Operators / developers | **BackFiller only** — options, JSON locations, RabbitMQ, NntpDB, secrets | +| [storage.md](storage.md) | Operators / developers | **StorageService** architecture, configuration, persistence, messaging, ops | +| [storage-nntp-feed.md](storage-nntp-feed.md) | Developers / integrators | Storage NNTP feed: ARTICLE / TAKETHIS / QUIT only | +| [logging.md](logging.md) | Operators / developers | Serilog-only logging for BackFiller and StorageService | +| [systemd.md](systemd.md) | Operators | systemd notify/watchdog install and operations for both hosts | + +Redis / PostFilter / Transit DDL under [schema/](schema/) are **not** Storage or BackFiller startup requirements. + +## Shared library + +| Document | Contents | +|---|---| +| [common/README.md](common/README.md) | What Common owns vs application hosts | +| [common/architecture.md](common/architecture.md) | Common boundaries | +| [common/configuration.md](common/configuration.md) | Shared configuration helpers | +| [common/ArticleRecord.md](common/ArticleRecord.md) | Canonical article envelope | +| [common/vatp.md](common/vatp.md) | Vector Article Transfer Protocol | +| [common/systemd-host-abstraction.md](common/systemd-host-abstraction.md) | **Proposal only** — comparing Storage vs BackFiller systemd stacks | + +## Standards and schema + +| Path | Contents | +|---|---| +| [standards/rfcs/](standards/rfcs/) | Local NNTP / Usenet RFC text library (authoritative for protocol work in this repo) | +| [standards/spamd/](standards/spamd/) | SPAMC/SPAMD protocol text — **upstream PostFilter interoperability**; not a VectorNNTPD host feature | +| [standards/haproxy/](standards/haproxy/) | PROXY protocol text — standards library only; not an implementation claim for BackFiller/Storage | +| [standards/pgpverify/](standards/pgpverify/) | INN pgpverify format notes — interoperability reference; BackFiller does not implement PGPVERIFY | +| [schema/](schema/) | **Upstream-only** Transit / PostFilter / `account_art_type` SQL — see [schema/README.md](schema/README.md) | + +## Upstream NNTPD reference (not hosts in this solution) + +Material from `VectorNNTP.NNTPD` for engineers who need command or PostFilter context when +working beside an NNTPD deployment. Paths such as `src/VectorNNTP.NNTPD/...` refer to the +**upstream** repository. They are **not** an implementation guarantee for +`VectorNNTPD.BackFillerService` or `VectorNNTPD.StorageService`. + +For what this solution actually speaks on the wire, use [architecture.md](architecture.md) +(BackFiller upstream client + cache listener) and [storage-nntp-feed.md](storage-nntp-feed.md) +(Storage ARTICLE / TAKETHIS / QUIT). Do not treat the upstream command checklist as Storage or +BackFiller CAPABILITIES. + +Upstream Redis and PostFilter configuration are **not** required to operate VectorNNTPD hosts. + +| Document | Topic | +|---|---| +| [reference/upstream-nntpd/commands.md](reference/upstream-nntpd/commands.md) | Upstream NNTPD command inventory | +| [reference/upstream-nntpd/postfilter.md](reference/upstream-nntpd/postfilter.md) | Upstream PostFilter operator guide | + +## Agent and contribution docs (repository root) + +| Document | Contents | +|---|---| +| [../AGENTS.md](../AGENTS.md) | Cursor agent entry point | +| [../CONTRIBUTING.md](../CONTRIBUTING.md) | Contribution guidance | +| [../SECURITY.md](../SECURITY.md) | Security reporting | +| [../CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) | Conduct | diff --git a/docs/architecture.md b/docs/architecture.md index 20617b67..303d5615 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -177,7 +177,7 @@ The listener command path is byte-oriented, as is the upstream NNTP client. They ## Source-generated structured logging -Application logs use `LoggerMessageAttribute` on partial methods, usually in a `*LogMessages.cs` companion. Serilog is the host logging stack. The rolling gzip file sink, compressor, active-file writer, and periodic flush wrapper live in `VectorNNTPD.Common.Logging`. BackFiller selects targets, levels, retention, syslog, and RabbitMQ publishing. Message templates keep structured property names. Do not preformat those properties into the message string. +Application logs use `LoggerMessageAttribute` on partial methods, usually in a `*LogMessages.cs` companion. Serilog is the host logging stack. The rolling gzip file sink, compressor, active-file writer, and periodic flush wrapper live in `VectorNNTPD.Common.Logging`. BackFiller selects targets, levels, retention, syslog, and RabbitMQ publishing. Message templates keep structured property names. Do not preformat those properties into the message string. Operator-facing sink and systemd details: [logging.md](logging.md), [systemd.md](systemd.md). Do not log Cloudflare API keys, ACME passwords, RabbitMQ passwords, `AUTHINFO` secrets, or article payloads. NNTP wire debug logs redact `AUTHINFO USER` and `AUTHINFO PASS`. Guard expensive trace or debug arguments with `IsEnabled` when building them has a cost. diff --git a/docs/logging.md b/docs/logging.md new file mode 100644 index 00000000..c1b30236 --- /dev/null +++ b/docs/logging.md @@ -0,0 +1,139 @@ +# VectorNNTPD — Logging (Serilog) + +Serilog is the **only** logging implementation for both production hosts in this solution +(`VectorNNTPD.BackFillerService` and `VectorNNTPD.StorageService`). Application code uses +`Microsoft.Extensions.Logging.ILogger`; those resolve to Serilog sinks. + +This document describes **current** BackFiller and StorageService behaviour. It does **not** +document INN `news` / Path-survey (`inpaths`) streams; those belong to upstream +`VectorNNTP.NNTPD` and are not hosts in `VectorNNTPD.slnx`. + +Shared rolling-file primitives live in `VectorNNTPD.Common.Logging` +(`GzipRollingFileSink`, compressor, active-file writer, periodic flush). Prefer +`LoggerMessageAttribute` + partial methods for application messages; keep structured +property names and do not preformat them into the message string. See +[architecture.md](architecture.md#source-generated-structured-logging). + +## Abstractions vs implementation + +| Layer | Role | +|-------|------| +| `ILogger` / `ILoggerFactory` | Application and framework logging API | +| `SerilogLoggerFactory` | Exclusive MEL factory registered by `AddSerilog` | +| Serilog sinks | Destinations (file, optional console / syslog / RabbitMQ depending on host) | + +Do not re-add `AddConsole`, `AddDebug`, or `AddEventLog` as MEL providers. Both hosts clear +MEL providers and remove the default `ILoggerFactory` before registering Serilog. + +`AddSystemd()` remains for systemd lifetime and notify support only. Any Microsoft console +formatter registration it may add is stripped afterward; Serilog owns console formatting when +a console sink is enabled. + +## BackFillerService + +### Configuration surface + +BackFiller does **not** read a `Serilog` JSON section. The only bindable surface is +`BackFiller:Logging` in `VectorNNTPD.BackFillerService.json` (see +[configuration.md](configuration.md)). + +| Setting | Default | Meaning | +|---------|---------|---------| +| `LogLevel` | `Information` | Serilog minimum applied to enabled targets (`Verbose`…`Fatal`) | +| `LogRetentionDays` | `14` | Daily file retention when the File target is enabled (range 1–3650) | +| `Json` | `false` | When true, enabled text-capable sinks use JSON formatting | +| `File:Enabled` | `true` | Rolling gzip file sink | +| `File:LogDir` | `logs` | Directory; required when File is enabled. Relative paths use Common `ApplicationLocalPath` against the application binary directory | +| `Syslog:Enabled` | `false` | Optional UDP/TCP syslog | +| `RabbitMQ:Enabled` | `false` | Optional publish via existing `IRabbitMqService` (`Exchange` / `RoutingKey`; broker credentials stay in `nntpsharedconfig` / RabbitMQ connectivity JSON) | + +Sink graph construction is code-owned (`BackFillerFileLogging`) for Native AOT / single-file safety. +It does not call `ReadFrom.Configuration`. + +### Console + +The console sink is **off** unless the process is started with `--console`. Under the shipped +systemd unit, journald still receives stdout/stderr from the process, but Serilog does not +write application events to the console sink by default. See the unit comments in +`deploy/systemd/vectornntp-backfiller.service`. + +### File sink behaviour + +- Rolling path: `{LogDir}/{entry assembly name}-yyyyMMdd.log` (Serilog day token `-.log`). +- Daily roll; `rollOnFileSizeLimit` is false (no size-based roll). +- Completed days are gzip-compressed (`CompressionLevel.Fastest`) via Common + `GzipRollingFileSink`; the active file is never compressed or deleted. +- Async wrapper: buffer size `50000`, **`blockWhenFull: false`** so a stalled file sink drops + excess events instead of blocking article-path loggers. Critical diagnostics may be lost under + sustained backpressure; prefer ops alerts on log-volume disk pressure. +- File sink is buffered with about a one-second flush interval. + +### Bootstrap + +`Program` assigns a bootstrap logger before the host is built so configuration and construction +failures are recorded, then flushes on shutdown (`Log.CloseAndFlushAsync`). After `host.Build()`, +an Information “logging initialized” event proves the Serilog factory is live (no secrets). + +## StorageService + +Host architecture and non-logging options: [storage.md](storage.md). + +### Configuration surface + +StorageService uses a `Serilog` section in `VectorNNTPD.StorageService.json` for minimum levels, +enrichment, and `WriteTo` arguments, plus `StorageServer:LogDir` for the directory. + +| Setting | Source of truth | +|---------|-----------------| +| Console / File / Async arguments | `Serilog:WriteTo` | +| Log directory | `StorageServer:LogDir` (shipped `logs`; code default constant is `/logs` if the key is blank) | +| Path rewrite | `StorageServerFileLogging.BindResolvedFilePath` creates the directory and overwrites the File `path` so the JSON placeholder is never the runtime path | + +Native AOT cannot rely on reflection discovery of sink extension methods for every `WriteTo` +entry. Storage applies MinimumLevel / enrichment via `ReadFrom.Configuration` while registering +Console / Async / File sinks explicitly (`StorageServerConfiguredSinks`). + +### Console and journald + +The shipped JSON enables the Console sink (Debug+ restricted level on that sink; global default +Information with `VectorNNTPD.StorageService` overridden to Debug). Under systemd, Information+ +lines reach stdout and journald. See `deploy/systemd/vectornntp-storageservice.service`. + +### File sink behaviour + +- Same daily `{entry assembly name}-yyyyMMdd.log` naming and Common gzip rotation as BackFiller. +- Shipped Async args: `bufferSize` 50000, **`blockWhenFull: true`** (logging calls block when the + async queue is full — deliberate difference from BackFiller). +- `retainedFileCountLimit` 14 in the shipped JSON. +- `buffered` true; `flushToDiskInterval` `00:00:01`. + +### Bootstrap + +Same pattern as BackFiller: bootstrap logger, `ConfigureStorageServerLogging`, then +`WriteLoggingInitialized` after the host is built. + +## Host comparison + +| Topic | BackFiller | StorageService | +|-------|------------|----------------| +| Config keys | `BackFiller:Logging` only | `Serilog` + `StorageServer:LogDir` | +| Console by default | Off (`--console` to enable) | On (shipped JSON) | +| Async `blockWhenFull` | `false` (drop under backpressure) | `true` (block) | +| Optional syslog / RabbitMQ log targets | Yes | No (not on this host’s logging options) | +| Shared gzip file implementation | Common `GzipRollingFileSink` | Same | + +## Under systemd + +See [systemd.md](systemd.md). File logs remain under each install tree’s `logs/` (unit +`ReadWritePaths`). Journal queries: + +```bash +sudo journalctl -u vectornntp-backfiller.service -f +sudo journalctl -u vectornntp-storageservice.service -f +``` + +## What this repository does not implement + +- INN-compatible `news` disposition log and Path-survey `inpaths` files (upstream NNTPD). +- Operator-configurable INN field order via `outputTemplate` for those streams. +- Microsoft MEL console / debug / event-log providers alongside Serilog. diff --git a/docs/reference/upstream-nntpd/commands.md b/docs/reference/upstream-nntpd/commands.md new file mode 100644 index 00000000..6221e9fc --- /dev/null +++ b/docs/reference/upstream-nntpd/commands.md @@ -0,0 +1,153 @@ +# NNTP command inventory + +> **Upstream reference only.** This file describes `VectorNNTP.NNTPD` in the Usenet.ninja +> VectorNNTP repository. It is **not** a claim about commands implemented by +> `VectorNNTPD.BackFillerService` or `VectorNNTPD.StorageService`. Paths such as +> `src/VectorNNTP.NNTPD/...` refer to that upstream tree. Kept here so VectorNNTPD engineers +> can consult the historical NNTP command inventory without opening the source repo. +> +> VectorNNTPD wire behaviour: see [`../../architecture.md`](../../architecture.md) (BackFiller +> client + cache listener) and [`../../storage-nntp-feed.md`](../../storage-nntp-feed.md) +> (Storage ARTICLE / TAKETHIS / QUIT). A `[x]` below does **not** mean Storage or BackFiller implements it. + +Tracker for VectorNNTP.NNTPD command implementations under `src/VectorNNTP.NNTPD/Session/Commands/`. +Command-processing infrastructure (dispatch, catalog, execution, response writer, logging) lives under `src/VectorNNTP.NNTPD/Session/CommandProcessor/`. + +- `[x]` = real protocol behavior present (not merely a stub class). +- `[ ]` = registered placeholder; returns not-implemented (or equivalent) until the RFC-compliant handler is written. + +Do **not** mark `[x]` solely because a `.cs` file exists, or because a LIST keyword is recognized by the parser. Recognition is not implementation. + +## Implemented + +```text +[x] CAPABILITIES +[x] AUTHINFO USER +[x] AUTHINFO PASS +[x] AUTHINFO SASL +[x] MODE READER +[x] HELP +[x] DATE +[x] MODE STREAM +[x] QUIT +[x] STARTTLS +[x] COMPRESS DEFLATE +[x] TAKETHIS +[x] CHECK +[x] IHAVE +[x] SPEEDTEST +[x] POST +[x] LIST +[x] LIST ACTIVE +[x] LIST COUNTS +[x] LIST NEWSGROUPS +[x] LIST OVERVIEW.FMT +[x] LIST HEADERS +[x] GROUP +[x] ARTICLE +[x] HEAD +[x] BODY +[x] STAT +``` + +AUTHINFO USER/PASS (RFC 4643 §2.3) caches a username then authenticates. AUTHINFO SASL (RFC 4643 §2.4) implements PLAIN, LOGIN, CRAM-MD5, and SCRAM-SHA-256 only. DIGEST-MD5, SCRAM-SHA-1, SCRAM-SHA-256-PLUS, EXTERNAL, and GSSAPI are not implemented and are not advertised. Challenges use `383` plus Base64 (`383 =` for an empty PLAIN challenge; LOGIN uses Base64 `Username:` / `Password:`; CRAM-MD5 Base64-encodes the RFC 2195 `` challenge, where `Fqdn` is the generated Nntpd identity). Success is `281`, or `283` plus Base64 when the mechanism returns additional SASL data (SCRAM server-final). A syntactically valid SCRAM-SHA-256 client-first always receives `383` server-first; unknown, disabled, and non-SCRAM accounts use a process-local dummy verifier and still fail with `481`. Dummy material cannot authenticate. Reader identity comes only from a successful AUTHINFO/SASL exchange in Reader authority (`MODE READER` or unspecified). `MODE STREAM` AUTHINFO uses Transit peer credentials only; there is no Transit ↔ MySQL fallback. MySQL `nntpusers` is loaded through the existing NntpDB pool. CAPABILITIES advertises `AUTHINFO USER SASL` and `SASL PLAIN LOGIN SCRAM-SHA-256 CRAM-MD5` when SASL is registered and cleartext AUTHINFO is permitted. + +CHECK (RFC 4644 §2.4) uses HistoryDB `PeekAsync` (local/Redis hit is `438`, or `431` when the connection's Receive `DeferOnDuplicate` is true; `238` double miss without recording the identifier; `431` Redis unavailable regardless of that flag). Consecutive transit-authorized CHECK commands may overlap Redis lookups in a per-session window of **16** (`CheckPipeline.Depth`; architectural constant, not configurable). Responses are emitted in send order. Every other command is a serial barrier: outstanding CHECK replies are drained first. See `docs/architecture.md` (Redis and HistoryDB). + +LIST / LIST ACTIVE / LIST COUNTS / LIST NEWSGROUPS (RFC 3977 §7.6 / RFC 6048 §2.2) and GROUP (RFC 3977 §6.1.1) read one captured in-memory `NewsgroupSnapshot`. The NNTP command path does not query MySQL. LIST ACTIVE, LIST COUNTS, and LIST NEWSGROUPS write precomputed UTF-8 lines; a wildmat filters by group name only and an empty match is still `215`. LIST ACTIVE and LIST COUNTS status is exactly the stored octet `y` / `n` / `m` / `x` / `j` (RFC 3977 §7.6.3 plus RFC 6048 §3.1 single-letter values). The RFC 6048 `=` status form is not supported; this is not complete RFC 6048 LIST ACTIVE status support. LIST COUNTS lines are `group high low estimated status` with a single ASCII space between fields. The estimate is the same GROUP watermark estimate already stored on the snapshot (`high − low + 1` when `high >= low`, except the empty `0 0` case; `0` when `high < low`). It is not an exact stored-article count. ARTICLE / HEAD / BODY / STAT (RFC 3977 §6.2) message-id form resolves through ArticleWork RPC then VATP to a CanonicalV1 `ArticleRecord`, then emits `220` / `221` / `222` / `223` (article number `0` for message-id selection). Local ingest uses `TryAdmit` with `InboundArticleProducer.BackFiller` and does not gate the NNTP response. ArticleWork not-found → `430`; VATP/transfer failures → `400` without closing. Article-number form remains `412` without a selected group and `423` after GROUP; the omitted current-article form is `412` / `420`. GROUP does not invent a current article number from water marks. Unsuccessful lookups do not change the selected group or current article. Error replies are single-line (no multiline terminator). LISTGROUP remains a registered placeholder (`500`) until an article-number source exists; it does not consult the catalogue and does not invent article numbers from `count_low`/`count_high`. LIST OVERVIEW.FMT and LIST HEADERS are static field-list descriptions (RFC 3977 §8.4 / §8.6), not article-data operations. Both write an immortal precomputed `215` multiline response and do not read the catalogue, NntpDb, or articles. OVERVIEW.FMT uses the RFC 3977 §8.4.2 compatibility names `Bytes:` and `Lines:` (not `:bytes` / `:lines`) and does not invent extra fields. LIST HEADERS, LIST HEADERS MSGID, and LIST HEADERS RANGE return the same static field list because HDR forms are not distinguished; metadata items use the RFC HEADERS names `:bytes` and `:lines`. This is not an implementation of OVER, HDR, or article overview storage. LIST MOTD remains recognized without stored information (`503 Data item not stored`). Unknown LIST keywords (ACTIVE.TIMES, DISTRIB.PATS, DISTRIBUTIONS, MODERATORS, SUBSCRIPTIONS, and any unregistered token) remain `501 Unknown command variant`. Extra arguments on MOTD/OVERVIEW.FMT, invalid HEADERS arguments, extra LIST tokens, and invalid wildmats remain `501`. CAPABILITIES advertises `LIST ACTIVE COUNTS HEADERS NEWSGROUPS OVERVIEW.FMT` with READER. OVER and HDR are not advertised. + +POST (RFC 3977 §6.3.1) is not pipelined. `440` is returned when the session is not permitted to post and no article is read. After `340`, the article is streamed under command-work accounting (`Nntpd:MaxArticleSize` destuffed). Client headers that are accepted are written through. `From`, `Newsgroups`, and `Subject` are required. A valid client `Date` is preserved; when `Date` is absent, POST writes one generated from the injection time at the header/body boundary and does not replace a valid client `Date`. A valid client `Message-ID` is preserved; when `Message-ID` is absent, POST writes a generated one. A client `Path` is discarded. Server-owned headers (`Path: .POSTED`, `Injection-Date`, `Injection-Info` without a plaintext peer IP, and authenticated-encrypted `X-Trace`) are generated at the header/body boundary. `NNTP-Posting-Date` and `NNTP-Posting-Host` are discarded and are not generated. Client `Control` is rejected (`441`) unless the session has `ControlCancelPermitted` (newsmaster AUTHINFO only) and the header is exactly `cancel ` per RFC 5536 / RFC 5537 §5.3. Ordinary authenticated users cannot inject `Control`. The protected `X-Trace` payload includes peer IP, peer port, injection timestamp, trace id, and the AUTHINFO username captured at POST admission (empty when the session is unauthenticated). The body is copied with stuffing preserved into one owned stuffed buffer (IHAVE/TAKETHIS queue representation) and admitted with `TryAdmit`. History Peek stays after the terminator and before ArticleRecord / PostFilter. After a successful `ArticleRecord` create, POST first applies the session `account_art_type` capability when an AUTHINFO policy is present (`441` if the article flags are not a subset of the allowed mask; unauthenticated/newsmaster remain unrestricted), then evaluates the immutable PostFilter snapshot loaded from NntpDB (`nntppostfilterpolicy`; seed `gate=Disabled`: no quota, no SPAMD). A PostFilter reject is `441 Posting failed`; the internal reason is not on the wire. Rejection evidence is enqueued asynchronously and cannot change the wire code. Unapproved catalogue `m` submission, IHAVE, and TAKETHIS do not run PostFilter. TAKETHIS enforces the same account capability after `ArticleRecord` creation (`439`). IHAVE still queues stuffed wire and classifies downstream of `235`, so IHAVE cannot enforce ArtType at the command boundary. ARTICLE/HEAD/BODY/STAT still have no stored ArtType. See `docs/postfilter.md`. `240` is sent only after queue admission; POST does not await spool or worker processing. + +RFC 3977 does not expose the article `Newsgroups:` list before `340`, so group-policy failures after the article (or at the header/body boundary during receive) are `441 Posting failed`, not `440`. Every `Newsgroups:` target is parsed by the existing POST header parser, then checked against one captured `NewsgroupCatalogue.Current` snapshot using the same case-insensitive `TryGet` as GROUP/LIST. + +| `posting_status` | Local POST | +|------------------|------------| +| `y` | Ordinary local posting (when every other target is also `y`, or `y` plus authorized/forwarded `m`) | +| `n` | Rejected — local posting prohibited | +| `m` | Moderated. No `Approved:` → proto-article is offered to `IModerationSubmissionService` for the leftmost moderated group (RFC 5537 §3.5.1) and is not injected. `Approved:` → ordinary injection only when the authenticated AUTHINFO principal is authorized by the `nntpmoderators` catalogue for every moderated target and the header identities match those routes. | +| `x` | Rejected — closed: local posting and peer articles are prohibited | +| `j` | Rejected — peer-only: local posting is not accepted | +| unknown name | Rejected — the group is not in the captured snapshot | + +`Approved:` is an assertion (RFC 5536 mailbox-list), not a boolean and not trust by itself. Header-name comparison is case-insensitive. Mailbox identities compare ASCII case-insensitively. Conflicting identities are rejected; identical duplicates collapse. `Approved: 1` / `true` are malformed. `Control:PgpAuthorities` is not a moderator catalogue. + +`nntpmoderators.moderator_address` is a routing destination (static mailbox or INN `%s` template: matched newsgroup name with `.` → `-`). That address is where an unapproved proto-article would be submitted. It does not authenticate a client. `account_name` is the AUTHINFO principal required for local reinjection. The INN `samples/moderators` URL in `Moderation:Source` is provenance only. + +Moderator reinjection is a normal POST: `AUTHINFO USER/PASS`, then `POST` with `Approved: moderator@example.com`. There is no `MODERATE` command. The authenticated session identity plus a configured `Username` mapping is the authorization. + +Unapproved moderated forwarding happens after Message-ID/Date are present and before Injection-Info/Injection-Date (RFC 5537 §3.5 step 7). The original Message-ID is preserved. The production `IModerationSubmissionService` composes an RFC 5537 `application/news-transmission; usage=moderate` email and calls `IEmailService.SendAsync`. `240` means the complete moderation email was durably written to the local filesystem spool (RFC 3977: received, possibly following further processing). It is not proof of remote SMTP or moderator receipt. SMTP delivery is asynchronous. When email is disabled, the spool write fails, or encoding fails, POST returns `441`. POST does not perform SMTP. Successful submission does not Peek, `TryAdmit`, or Remember. + +Group-policy and unauthorized-approval rejection drain the article. History Peek / `TryAdmit` / Remember happen only after injection authorization succeeds. Malformed or empty `Newsgroups:` remain syntax failures (`441`) and do not consult the snapshot. + +## Recognized LIST keywords (information not stored) + +These parse as valid LIST keywords and return `503 Data item not stored`. That is the RFC 3977 §7.6.1.2 response for a recognized keyword whose information is not maintained. It is not a `500` unimplemented-command placeholder, and it is not a successful LIST implementation. + +```text +LIST MOTD +``` + +## Placeholder (registered, not implemented) + +```text +[ ] LISTGROUP +[ ] LAST +[ ] NEXT +[ ] OVER +[ ] HDR +``` + +## File map + +| File | Commands | +|------|----------| +| `Capabilities.cs` | CAPABILITIES | +| `AuthInfo.cs` | AUTHINFO USER, PASS, SASL | +| `Mode.cs` | MODE READER, MODE STREAM | +| `Help.cs` | HELP | +| `Date.cs` | DATE | +| `Quit.cs` | QUIT | +| `StartTls.cs` | STARTTLS | +| `Compress.cs` | COMPRESS DEFLATE | +| `List.cs` | LIST, LIST ACTIVE, LIST COUNTS, LIST NEWSGROUPS, LIST OVERVIEW.FMT, LIST HEADERS; MOTD returns 503 | +| `Group.cs` | GROUP | +| `ListGroup.cs` | LISTGROUP | +| `Newsgroups/` | Immutable catalogue snapshot + five-minute refresh | +| `Article.cs` | ARTICLE, HEAD, BODY, STAT | +| `Last.cs` | LAST | +| `Next.cs` | NEXT | +| `Over.cs` | OVER | +| `Hdr.cs` | HDR | +| `Post.cs` | POST | +| `IHave.cs` | IHAVE | +| `Check.cs` | CHECK | +| `Session/CheckPipeline.cs` | Per-session CHECK overlap window (not a command) | +| `TakeThis.cs` | TAKETHIS | +| `SpeedTest.cs` | SPEEDTEST (VectorNNTP diagnostic extension) | +| `Session/SpeedTest/` | SPEEDTEST coordinator, payload, concurrency | + +IHAVE (RFC 3977 §6.3.2) is a serial, non-pipelined transit ingest: HistoryDB peek → non-blocking Transit queue probe → `335`/`435`/`436` → raw article receive (frame `CRLF . CRLF`, stuffed wire) → destuff + `ArticleRecordFactory` → CanonicalV1 `ArticleRecord` → non-blocking `TryAdmit` → `235`/`436`/`437`. A History hit is `435`, or the pre-article `436 Transfer not possible; try again later` when the connection's Receive `DeferOnDuplicate` is true; neither reads an article. IHAVE never waits for queue memory; temporary inability to accept is `436`. Articles that cannot become CanonicalV1 are `437` after `335`. TAKETHIS production receive/framing is unchanged. See `docs/architecture.md` (IHAVE article ingestion). + +`SPEEDTEST ` is a VectorNNTP private diagnostic extension (not an RFC command). `` is the configured Transit peer identifier (exact ordinal match, one NNTP token). `PeerName` is the human-readable administrative name and is not a command argument. The command is transit-authorized and measures TX on the **current** NNTP session with a synthetic payload. It does not open outbound Transit sockets, does not use article ingestion, HistoryDB, or Redis, and never reports measurements that were not performed. Outbound initiator support is not implemented (`OUTBOUND=NOT-AVAILABLE`). See `docs/architecture.md`. + +Article ingestion (TAKETHIS / POST / IHAVE → CanonicalV1 `ArticleRecord` → byte-budgeted queue → `IncomingSpoolWriterService` / `IngestionWorkerPool` → protobuf OverviewDB handoff on `overviewdb.queue` → dedicated Serilog `news` file → dedicated Serilog Path-survey `inpaths` file → /dev/null no-op sink) lives under `ArticleIngestion/`. Accepted `+`/`j` lines are generated after a confirmed OverviewDB publish. The Path-survey line is `Path: ` plus canonical `ArticleRecord.Path` from that same record; it is not added to news. Deliberate rejections write `-` at the protocol decision because those articles never enter the queue. Successful moderated POST writes `m` at that decision. Cancel (`c`) is not implemented. The news line format is application-owned and not configurable (innd `ARTlog` field order: inbound Transit identifier, Message-ID, size, then outbound `?` only on accepted `+`; `j`/`-`/`m` omit outbound sites; response codes are not rendered). The Path-survey format is likewise application-owned (`Path: ` plus LF) and is not configurable. Serilog owns path, rolling, retention, compression, and flush for both files. Completed `inpaths` files are streamed by the internal ninpaths processor when `Nntpd:Top1000` has recipients. + +Registration lives in `Session/CommandProcessor/DefaultNntpCommandCatalog.cs` (descriptors + authorization metadata only). + +## NNTPCancelMessage + +`src/VectorNNTP.NNTPCancelMessage` is a separate newsmaster client. It is not part of the NNTP server runtime. + +```text +NNTPCancelMessage --host nntpd01.usenet.ninja --port 563 --tls \ + --username newsmaster --password 'secret' + +NNTPCancelMessage --host nntpd01.usenet.ninja --port 563 --tls \ + --username newsmaster --password 'secret' --cancel +``` + +`--cancel` / `-cancel` is the destructive confirmation. Command-line values override `NNTPCancelMessage.json`. `--password` can appear in OS process listings and shell history; prefer environment or secrets. + +Flow: connect → AUTHINFO USER/PASS when credentials are supplied (mandatory for `--cancel`) → `HEAD ` → decrypt original `X-Trace` → if `--cancel`, canonicalize PGPVERIFY signed fields, sign, then POST an RFC 5536/5537 cancel article that copies the original `Newsgroups` and uses `Control: cancel ` (not the obsolete RFC 1036 `cmsg` Subject convention). `--cancel` cannot skip signing. RFC 1036 is retained under `docs/standards/rfcs/` for historical reference only. RFC 5536 and RFC 5537 govern the CANCEL article. PGPVERIFY is a de-facto interoperability convention (see `docs/standards/pgpverify/`); it is not an RFC-defined standard. The signed header is `X-PGP-Sig` as FORMAT specifies, not PGP/MIME and not `X-PGP-Signature`. Server-owned Path / Injection-Date / Injection-Info / X-Trace remain unsigned and are generated by POST. + +**HEAD note:** VectorNNTP.NNTPD `HEAD ` retrieves CanonicalV1 headers through ArticleWork + VATP when BackFiller can supply the article. Numeric `HEAD` forms remain lookup-failure only. diff --git a/docs/reference/upstream-nntpd/postfilter.md b/docs/reference/upstream-nntpd/postfilter.md new file mode 100644 index 00000000..98b811eb --- /dev/null +++ b/docs/reference/upstream-nntpd/postfilter.md @@ -0,0 +1,586 @@ +# PostFilter v1 — operator guide + +> **Upstream reference only.** This guide describes PostFilter as implemented in +> `VectorNNTP.NNTPD` (`src/VectorNNTP.NNTPD/PostFilter/`). It is **not** part of the +> VectorNNTPD.BackFillerService or VectorNNTPD.StorageService hosts. Schema DDL under +> [`../../schema/`](../../schema/) is upstream reference only (see that README). Applying or +> operating PostFilter, Redis quota, or SPAMD remains an upstream NNTPD concern. +> **VectorNNTPD hosts do not require Redis or PostFilter to start or run.** + +Authoritative operator reference for NNTPD PostFilter as implemented in `src/VectorNNTP.NNTPD/PostFilter/`. + +**NntpDB is the only authoritative cluster policy.** Every NNTPD instance reads the published revision from `nntppostfiltercurrent`, loads that revision’s scalar row and collections, compiles an immutable local snapshot, and evaluates POST against that snapshot. There is no `Nntpd:PostFilter` options binding. A leftover `Nntpd:PostFilter` section fails startup. Redis holds quota **runtime state** only, never policy. + +Protocol replies stay `240 Article received OK` or `441 Posting failed`; PostFilter reasons are never sent on the NNTP wire. + +| Concern | Owner | +|---|---| +| Authoritative policy | MySQL `nntppostfiltercurrent` + revision-keyed tables | +| Repository | `MySqlPostFilterPolicyRepository` | +| Snapshot / refresh | `PostFilterPolicyCompiler`, `PostFilterPolicyService` (60s) | +| POST evaluation | `PostFilterEvaluator` (snapshot only; no SQL) | +| Redis quota | `PostFilter/Quota/` | +| SPAMD client | `SpamdCheckClient`, `SpamdScanArticleBuilder` | +| Settings table | Upstream VectorNNTP `docs/configuration.md` (NNTPD Redis / PostFilter keys — **not** migrated; VectorNNTPD [configuration.md](../../configuration.md) is BackFiller-only) | +| POST protocol | [commands.md](commands.md) | +| Validation matrix | `tests/VectorNNTP.NNTPD.Tests/PostFilter/PostFilterProductionPolicyMatrixTests.cs` | + +NNTPD does not create or migrate these tables. Apply the canonical provisioning script [`docs/schema/postfilter.sql`](../../schema/postfilter.sql) once per NntpDB. Seed `Gate=Disabled` so the cluster stays inert until operators publish a new revision. + +Account ArtType capability is a separate additive change on existing `nntpusers`. Apply [`docs/schema/nntpusers-account-art-type.sql`](../../schema/nntpusers-account-art-type.sql) to the account database **before** deploying an NNTPD that `SELECT`s `account_art_type`. A missing column is AUTHINFO `503`, not `481` and not unrestricted access. + +--- + +## 1. What PostFilter does + +PostFilter is a **POST-only** accept-path policy that runs after a destuffed article has been parsed into an immutable `ArticleRecord`. When the gate is `Active`, it can: + +1. Reject a POST before queue admission (deny lists, ArtType policy, unauthenticated session, quota, SpamAssassin). +2. Reserve distributed accept-quota in Redis for the authenticated account. +3. Optionally send a **disposable** email-like scan representation to SpamAssassin SPAMD `CHECK`. +4. After `TryAdmit` succeeds, commit the reservation (or leave it until TTL if COMMIT cannot run). + +PostFilter does **not** mutate `ArticleRecord`, does **not** create a second canonical article, and does **not** change History, spool, or worker processing. + +--- + +## 2. Where it runs + +PostFilter is invoked only from the ordinary POST injection path, after History Peek and `ArticleRecordIngress.TryCreateFromDestuffed`, and before `CreateQueued` / `TryAdmit`. + +```text +POST + → posting-permitted? (else 440; no article is read) + → 340 + → StreamingPostArticleReader + → catalogue `m` unapproved proto-article? + yes → IModerationSubmissionService (no PostFilter, no Peek, no TryAdmit) + → History Peek (Seen / Unavailable → 441; no PostFilter) + → ArticleRecordIngress.TryCreateFromDestuffed + → ArticleRecord (immutable canonical bytes) + → account ArtType capability (AUTHINFO flags; unrestricted when no policy) + → PostFilter.EvaluateAsync (one captured policy snapshot) + Gate Disabled → accept, no Redis, no SPAMD + Gate Closed → 441 + deny / ArtType / unauthenticated → 441 (no Redis) + Redis RESERVE + optional SPAMD CHECK + → CreateQueued + → TryAdmit + fail → Redis RELEASE + enqueue admission evidence + 441 + → Redis COMMIT (never RELEASE on COMMIT failure) + → History Remember + → 240 Article received OK +``` + +Every PostFilter `Reject` (and post-accept `TryAdmit` failure) enqueues one evidence row onto a bounded background queue. The 441 is written regardless of whether the enqueue or later MySQL insert succeeds. Acceptance never writes evidence. + +Complete unstuffed article bytes (`ArticleRecord.ArtData`) are available at every PostFilter decision in the current POST path. The evidence payload is therefore those bytes when present, or NULL only if a future path rejects without a complete article. + +One POST captures `PostFilterPolicyService.Current` once. That snapshot is used for the entire request. POST never queries NntpDB. Mid-request database edits do not affect that evaluation. + +--- + +## 3. What it does not filter + +PostFilter does **not** run for: + +| Path | Why | +|------|-----| +| IHAVE | Transit ingest; no `IPostFilter` call | +| TAKETHIS | Transit ingest; no `IPostFilter` call | +| BackFiller | Separate process; no PostFilter | +| Catalogue status `m` unapproved submission | `SubmitForModerationAsync` returns before ArticleRecord / PostFilter | +| Incoming spool / queue workers | Filter already decided at POST admission | +| History duplicate / History unavailable | Peek rejects before PostFilter | +| Receive/parse failures (`Nntpd:MaxArticleSize`, malformed article) | Rejected before ArticleRecord | + +Authorized moderator reinjection (`Approved:` + catalogue authorization) is ordinary POST and **does** run PostFilter. + +--- + +## 4. Default / safe posture + +| Gate | Meaning | +|------|---------| +| `Disabled` (default) | Skip remaining stages. No quota reservation. No SPAMD call. Ordinary POST continues. | +| `Active` | Evaluate deny, ArtType, quotas, and SpamAssassin. | +| `Closed` | Reject every POST that reaches PostFilter (`441`). No Redis. No SPAMD. | + +`Disabled` is the recommended seed for `nntppostfilterpolicy.gate`. Existing POST behaviour continues until an operator updates NntpDB. There is no per-node Gate in appsettings. + +When `Gate` is `Active`, an unauthenticated session that passes deny/ArtType is rejected at the quota stage (`reason=unauthenticated`) without RESERVE. AUTHINFO is therefore required for any POST that reaches an `Active` filter. + +--- + +## 5. Authoritative NntpDB policy + +```text +NntpDB nntppostfiltercurrent.revision + → SELECT nntppostfilterpolicy WHERE revision = current + → SELECT collections WHERE revision = current + → PostFilterPolicyCompiler + → immutable PostFilterPolicySnapshot + → Volatile publish + → POST reads Current only +``` + +NNTPD follows the same catalogue contract as `nntpgroups` / `nntpmoderators`: + +- Initial load during `PostFilterPolicyService.StartAsync`. Failure prevents `RUNNING`. There is **no** appsettings fallback. +- Refresh every **60 seconds**. A new published revision compiles a snapshot and publishes it with `Volatile.Write`. +- The same published revision is re-read and recompiled for validity, but does not replace the live snapshot. +- Refresh failure (MySQL down, missing current row, compile error) keeps last-known-good. PostFilter is not disabled and does not revert to local JSON. +- Empty/malformed/duplicate collection entries fail compile of that refresh (or startup). + +`nntppostfiltercurrent.revision` is the operator-visible published revision. Logs emit `revision={Revision}`. NNTPD never reads collection rows from a different revision than the current pointer. + +--- + +## 6. NntpDB Schema and Administration + +Existing NntpDB tables (`nntpgroups`, `nntpusers`, `nntpmoderators`) have no checked-in `CREATE TABLE`. NNTPD only `SELECT`s them. PostFilter is the first NntpDB surface with product DDL. + +**Canonical provisioning script:** [`docs/schema/postfilter.sql`](../../schema/postfilter.sql) + +Apply that script once per NntpDB. NNTPD does not create or migrate tables. If an earlier singleton (`nntppostfilterpolicy.policy_id = 1` without revision-keyed collections) was applied, drop those objects and apply this script; there is no automated migrator. + +InnoDB + `utf8mb4` / `utf8mb4_unicode_ci` match typical MySQL 8 NntpDB deployments. Confirm `SHOW TABLE STATUS LIKE 'nntpmoderators'` if your server uses another collation. + +### 6.1 Control-plane model + +| Object | Role | +|--------|------| +| `nntppostfilterpolicy` | Append-only scalar revisions. Primary key is `revision`. | +| `nntppostfiltercurrent` | Singleton `policy_id = 1` pointing at the published revision. | +| `nntppostfilteraccounts` | Deny/allow **MD5 hex** AUTH identities (`CHAR(32)`). Real usernames are not stored. | +| `nntppostfiltercidrs` | Deny/allow CIDRs for **that revision**. | +| `nntppostfilterarttypes` | `reject` / `sa_exclude` **one** classifier name per row (`ENUM` of `ArticleType` names). | +| `nntppostfiltersahosts` | SPAMD hosts (`host_order`) for **that revision**. | +| `nntppostfilterrejections` | Append-only PostFilter rejection evidence (not SA-specific). | +| `trg_nntppostfiltercurrent_revision_forward` | Published revision must increase. | + +`nntpusers.account_art_type` is **not** created by `postfilter.sql`. Apply [`nntpusers-account-art-type.sql`](../../schema/nntpusers-account-art-type.sql) on the existing account table (`INT UNSIGNED NOT NULL DEFAULT 65535` = `ArticleTypeCapabilities.All`). That is the AUTHINFO session capability, not a PostFilter policy ENUM. + +Three related but different ArtType representations: + +| Surface | Representation | Meaning | +|---------|----------------|---------| +| `nntppostfilterarttypes.art_type` | MySQL `ENUM` of public `ArticleType` names | One type per policy row (`reject` / `sa_exclude`) | +| `ArticleRecord.ArtType` | `[Flags]` classifier bitmask | What this article is | +| `nntpusers.account_art_type` | `INT UNSIGNED` of the same flags | Types the authenticated account may post/read | + +Matching: policy lists trim then MD5 (UTF-8, lowercase hex). The evaluator hashes the session username **without** introducing a new trim. Case remains ordinal. + +`nntppostfilterrejections` is append-oriented. NNTPD has no evidence retention sweeper; deletion is an operator decision. + +Evidence writes are asynchronous (`PostFilterRejectionEvidenceQueue`, capacity 128, drop-newest-on-full). Queue-full or MySQL failure is logged and counted. It never turns 441 into 240 or the reverse. + +The article payload column is `LONGBLOB` so it can hold `Nntpd:MaxArticleSize` up to the configured 100 MiB ceiling. Stored bytes are the unstuffed complete article, not an `ArticleRecord` and not a restuffed wire copy. `source_ip` is `VARBINARY(16)` after the same IPv4-mapped normalization as session identity (`::ffff:a.b.c.d` stores as IPv4). Display with `INET6_NTOA(source_ip)`. + +Collection rows belong to a revision because `revision` is part of every primary key and a foreign key to `nntppostfilterpolicy.revision`. A loader that binds `@revision` from `nntppostfiltercurrent` cannot assemble accounts from revision 11 with scalars from revision 12 unless the database itself is deliberately broken (missing FK). + +Historical revisions remain after publish. Rollback is **not** pointing current backward (the trigger rejects that). Rollback is inserting a **new higher** revision that copies the previous content, then publishing it. + +### 6.2 Atomic update + +Every publish is one transaction. Insert the new revision and **all** of its collections, then update `nntppostfiltercurrent` last. + +```sql +START TRANSACTION; + +SELECT revision FROM nntppostfiltercurrent WHERE policy_id = 1 FOR UPDATE; +-- @old = that value; @new = @old + 1 + +INSERT INTO nntppostfilterpolicy (revision, updated_utc, gate, ...) VALUES (@new, UTC_TIMESTAMP(3), ...); +INSERT INTO nntppostfilteraccounts (revision, list_kind, account_name) + VALUES (@new, 'deny', LOWER(MD5('spammer'))); +INSERT INTO nntppostfiltercidrs ... +INSERT INTO nntppostfilterarttypes ... +INSERT INTO nntppostfiltersahosts ... + +UPDATE nntppostfiltercurrent SET revision = @new WHERE policy_id = 1 AND revision = @old; + +COMMIT; +``` + +`FOR UPDATE` serializes two operators. A duplicate `@new` hits the policy primary key. An uncommitted transaction is invisible to NNTPD. `ROLLBACK` leaves the previous published revision intact. + +Do **not** `UPDATE` an already-published revision in place. NNTPD ignores same-revision reloads for snapshot publication. + +### 6.3 Atomic read + +`MySqlNntpDbConnection.QueryPostFilterPolicyAsync` starts a `REPEATABLE READ` transaction, reads `nntppostfiltercurrent.revision`, then selects policy and collections with `WHERE revision = @revision`. Missing `policy_id = 1` fails startup. A current pointer with no policy row fails the load (FK should prevent that). + +Maximum cluster propagation delay after `COMMIT` is one refresh interval: **60 seconds**. + +### 6.4 Seed / what a new NntpDB must contain + +Before NNTPD can start, these objects and rows must exist: + +1. The six tables and the forward-revision trigger from [`docs/schema/postfilter.sql`](../../schema/postfilter.sql). +2. Policy revision `1` (Disabled, 1 day / 10 minutes, all ceilings `0`, SA disabled, no invented SPAMD host). +3. `nntppostfilterarttypes (1, 'sa_exclude', 'YEncoded')`. +4. `nntppostfiltercurrent (1, 1)`. + +No host rows. `sa_on_failure` is NULL while disabled. Missing current row prevents startup. NNTPD will not invent a Disabled policy. + +Inspect: + +```sql +SELECT c.revision, p.gate, p.updated_utc +FROM nntppostfiltercurrent c +JOIN nntppostfilterpolicy p ON p.revision = c.revision +WHERE c.policy_id = 1; + +SELECT list_kind, account_name FROM nntppostfilteraccounts WHERE revision = ( + SELECT revision FROM nntppostfiltercurrent WHERE policy_id = 1); +``` + +### 6.5 Operator change cases + +| Change | Procedure | +|--------|-----------| +| A–F. Any scalar and/or collection change | Insert `@old+1` with the **complete** new policy (copy unchanged lists), publish current. | +| G. Transaction never committed | Invisible. Nodes keep last published revision. | +| H. `ROLLBACK` | Previous published revision remains. | +| I. Duplicate collection entry | Primary key / unique key rejects the `INSERT`. | +| J. Invalid CIDR syntax | Database accepts the string; compiler fails the refresh; last-good retained. | +| K. Invalid ArtType name | Same as J. | +| L. Publish a lower revision | Trigger rejects `UPDATE nntppostfiltercurrent`. | +| M. Two operators | `FOR UPDATE` or duplicate revision PK; one commit wins. | + +When NntpDB is unavailable after a successful start: last-known-good snapshot remains. PostFilter stays enabled. There is no appsettings fallback. + +### 6.6 Runtime property → column map + +| Runtime | Table.column | SQL | Null | Seed | Validation | +|---------|--------------|-----|------|------|------------| +| Snapshot.Revision | `nntppostfiltercurrent.revision` / `nntppostfilterpolicy.revision` | `BIGINT UNSIGNED` | no | `1` | `>= 1`; published value must increase | +| Record.UpdatedUtc | `nntppostfilterpolicy.updated_utc` | `DATETIME(3)` | no | `UTC_TIMESTAMP(3)` | set on insert | +| Gate | `gate` | `VARCHAR(16)` | no | `Disabled` | `Disabled`/`Active`/`Closed` | +| Quota.LongWindow | `long_window_ms` | `BIGINT UNSIGNED` | no | `86400000` | `>= 1` | +| Quota.ShortWindow | `short_window_ms` | `BIGINT UNSIGNED` | no | `600000` | `>= 1` | +| Quota.MaxMessagesLong | `max_messages_long` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| Quota.MaxBytesLong | `max_bytes_long` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| Quota.MaxIdenticalLong | `max_identical_long` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| Quota.MaxMessagesShort | `max_messages_short` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| Quota.MaxBytesShort | `max_bytes_short` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| Quota.MaxIdenticalShort | `max_identical_short` | `BIGINT UNSIGNED` | no | `0` | `0` disables | +| DeniedAccounts | `nntppostfilteraccounts` `deny` | `VARCHAR(255)` | n/a | empty | unique per revision; compiler rejects blank/duplicate | +| AllowlistedAccounts | `nntppostfilteraccounts` `allow` | `VARCHAR(255)` | n/a | empty | same | +| DeniedCidrs | `nntppostfiltercidrs` `deny` | `VARCHAR(64)` | n/a | empty | unique; compiler parses CIDR | +| AllowlistedCidrs | `nntppostfiltercidrs` `allow` | `VARCHAR(64)` | n/a | empty | same | +| RejectArtTypes | `nntppostfilterarttypes` `reject` | `VARCHAR(32)` | n/a | empty | compiler `ArticleType` | +| SA.Enabled | `sa_enabled` | `CHAR(1)` | no | `N` | `Y`/`N` | +| SA.OnFailure | `sa_on_failure` | `VARCHAR(16)` | yes when disabled | `NULL` | required `Reject`/`Accept` when `Y` | +| SA.MaxArticleSize | `sa_max_article_size` | `INT` | no | `131072` | `>= 0`; `0` disables size gate | +| SA.ExcludeArtTypes | `nntppostfilterarttypes` `sa_exclude` | `VARCHAR(32)` | n/a | `YEncoded` | compiler `ArticleType` | +| SA.Hosts | `nntppostfiltersahosts.host` | `VARCHAR(255)` | n/a | empty | unique host/order; required when SA enabled | +| SA.Port | `sa_port` | `INT UNSIGNED` | no | `783` | `1–65535` | +| SA.ProtocolVersion | `sa_protocol_version` | `VARCHAR(16)` | no | `1.5` | required when SA enabled | +| SA.MaxConnections | `sa_max_connections` | `INT UNSIGNED` | no | `4` | `1–32` | +| SA.HostSelection | `sa_host_selection` | `VARCHAR(16)` | no | `RoundRobin` | `RoundRobin`/`Failover` | +| SA.ConnectTimeout | `sa_connect_timeout_ms` | `INT UNSIGNED` | no | `5000` | `>= 1` and `<=` operation | +| SA.OperationTimeout | `sa_operation_timeout_ms` | `INT UNSIGNED` | no | `30000` | `>= 1`; compiler `1s–2m` when SA enabled | + +Reservation TTL is compiled, not stored. `ServerId` and `Redis:*` are node-local infrastructure, not PostFilter policy. + +### 6.7 Former `Nntpd:PostFilter` mapping + +| Former key | NntpDB | +|------------|--------| +| `Gate` | `nntppostfilterpolicy.gate` | +| `DeniedAccounts` / `AllowlistedAccounts` | `nntppostfilteraccounts` (`deny` / `allow`) | +| `DeniedCidrs` / `AllowlistedCidrs` | `nntppostfiltercidrs` (`deny` / `allow`) | +| `RejectArtTypes` | `nntppostfilterarttypes` (`reject`) | +| `Quota:LongWindow` / `ShortWindow` | `long_window_ms` / `short_window_ms` | +| `Quota:MaxMessages*` / `MaxBytes*` / `MaxIdentical*` | matching columns | +| `SpamAssassin:Enabled` | `sa_enabled` (`Y`/`N`) | +| `SpamAssassin:OnFailure` | `sa_on_failure` (`Reject`/`Accept`, NULL when disabled) | +| `SpamAssassin:MaxArticleSize` | `sa_max_article_size` (exclusive; `0` disables) | +| `SpamAssassin:ExcludeArtTypes` | `nntppostfilterarttypes` (`sa_exclude`) | +| `SpamAssassin:Hosts` | `nntppostfiltersahosts` (`host_order`, `host`) | +| `SpamAssassin:Port` / `ProtocolVersion` / `MaxConnections` / `HostSelection` | `sa_port`, `sa_protocol_version`, `sa_max_connections`, `sa_host_selection` | +| `SpamAssassin:ConnectTimeout` / `OperationTimeout` | `sa_connect_timeout_ms` / `sa_operation_timeout_ms` | + +Do not automatically copy leftover appsettings values. Operators apply the seed, then edit NntpDB. + +SPAMD endpoints are **cluster policy** (same Hosts/Port on every node). Reservation identity still uses node-local `ServerId` (already NNTPD identity, not PostFilter policy). Redis connection settings remain the top-level `Redis` section. + +`ArticleType` names: `Default`, `Control`, `Cancel`, `Mime`, `Binary`, `UuEncode`, `Base64`, `YEncoded`, `BommaNews`, `UniData`, `Multipart`, `Html`, `PostScript`, `BinHex`, `Partial`, `PgpMessage`. + +Account names compare **ordinal** after trim. Empty or duplicate collection rows fail compile. Ceiling `0` disables that quota dimension. Reservation TTL is not a database column (see Redis). + +--- + +## 7. Policy semantics + +Compiled snapshot fields include gate, deny/allow sets, reject ArtType mask, quota windows/ceilings, SpamAssassin enablement, `OnFailure`, eligibility, hosts, port, protocol, pool size, host selection, timeouts, and reservation hold. + +Evaluation order when `Gate` is `Active`: + +1. Deny (account or CIDR) → `441`, no Redis. +2. `RejectArtTypes` if any listed flag is set on `ArtType` → `441`, no Redis. +3. Missing AUTH username → `441` (`unauthenticated`), no Redis. +4. Redis `RESERVE` (fail-closed on Unavailable). +5. SpamAssassin CHECK only when enabled, not allowlisted, and `ShouldScan` is true. +6. Accept with a live lease. + +Allowlist skips step 5 only. + +--- + +## 8. Redis + +Redis holds **distributed runtime quota state**, not policy. Policy is the local immutable snapshot. The same account on every NNTPD node shares the same Redis hashes. + +| Item | Current behaviour | +|------|-------------------| +| Identity | Authenticated AUTH username (UTF-8 SHA-256, lowercase hex). Not client IP. | +| Quota HASH | `nntpd:pf:q:{sha256hex}` — committed counters + live reservations | +| Multipost HASH | `nntpd:pf:m:{sha256hex}` — committed identical-body counts | +| Reservation field | `r:{nodeId}:{incarnation}:{generation}` | +| Node id | `nntpd{ServerId:00}` host label (same pattern as session ownership) | +| Model | Two fixed-epoch windows (L + S). Bucket id = `nowMs / windowMs`. | +| Hold / crash TTL | Floor **10s**. When SpamAssassin is enabled: `max(10s, OperationTimeout + 1s)`. Not a v1 operator knob. | +| Idle key TTL | `max(L,S) + hold + 1s` — garbage collection only; does not change bucket maths. | + +`RESERVE` is atomic Lua EVAL. A deny writes **nothing**. A live reservation counts toward the ceiling immediately. `COMMIT` moves reserved units into the current buckets and deletes the reservation. `RELEASE` deletes a matching live reservation and **never** decrements committed usage. + +These conservative outcomes are intentional: + +| Condition | Client | Queue | Quota | +|-----------|--------|-------|-------| +| `RESERVE` Unavailable | `441` | not queued | no reservation | +| `COMMIT` Unavailable after `TryAdmit` | `240` | article remains queued | reservation **not** released; committed usage may temporarily under-count | +| `RELEASE` Unavailable | reject/cancel path already decided | not queued (if reject) | reservation may remain until hold TTL expires | +| Process crash after `RESERVE`, before `COMMIT`/`RELEASE` | no `240` | not admitted | reservation expires at hold TTL | + +Redis itself remains a required NNTPD dependency (startup connect/PING). PostFilter adds the keys above on the same multiplexer. Redis binding lives in upstream VectorNNTP `docs/configuration.md` (not migrated). VectorNNTPD BackFiller [configuration.md](../../configuration.md) has no Redis section. + +--- + +## 9. Quotas + +Dimensions (each independently disableable with ceiling `0`): + +| Dimension | What is counted | +|-----------|-----------------| +| Messages | `1` per reserved POST | +| Bytes | `ArticleRecord.ArtSize` | +| Identical-message / multipost | `1` when a body identity is computed | + +Identical-message identity is **XXH3-64 of the canonical body octets after the first `CRLF CRLF`**, lowercase hex. It is **not** `ArticleRecord.ArtHash`. Articles whose `ArtType` includes `YEncoded`, `Binary`, `UuEncode`, `Base64`, or `BinHex` do not participate (no multipost units). + +Accounts are isolated: the same body on two usernames does not share a ceiling. Cross-node POSTs for one username share Redis state. + +Deterministic reserve outcomes (log `reason`, never on the wire): + +| Status | Meaning | +|--------|---------| +| `Accepted` | Live reservation counts toward the ceiling | +| `DeniedMessagesLong` / `DeniedMessagesShort` | Message ceiling | +| `DeniedBytesLong` / `DeniedBytesShort` | Byte ceiling | +| `DeniedIdenticalLong` / `DeniedIdenticalShort` | Identical-body ceiling | +| `Conflict` | Same token, different payload (should not occur in production) | +| `Unavailable` | Redis/EVAL unavailable — fail closed | + +--- + +## 10. SpamAssassin + +SPAMD does **not** receive `ArticleRecord` / `ArtData` directly. + +```text +ArticleRecord (read-only) + → SpamdScanArticleBuilder (disposable buffer) + → SPAMD CHECK SPAMC/1.5 +``` + +`ArtData` is not mutated. The scan buffer is independent and discarded after CHECK. + +### 10.1 Scan representation (current builder only) + +Synthetic headers prepended: + +- `Received: from [] by with NNTP [id ]; ` +- `To: usenet@` +- `X-Usenet-Newsgroups: ` when present +- `Date:` only when the original article has no Date header + +Stripped (not copied into the scan): `Path`, `Xref`, `Injection-Info`, `X-Trace`, `X-Complaints-To`, `NNTP-Posting-Host`. + +All other original header blocks are copied. The body after the first `CRLF CRLF` is copied verbatim. A missing/unsafe FQDN becomes `localhost`. + +### 10.2 Transport + +- Protocol: SPAMC/SPAMD `CHECK` with configured `ProtocolVersion` (default `1.5`). +- Persistent connection pool sized by `MaxConnections`. +- Each connection **serializes** CHECKs. No pipelining. No multiplexing (SPAMD has no request IDs). +- `RoundRobin`: rotate start host. `Failover`: always start at `Hosts[0]`. +- Connect-time failures fail over to the next host. +- After the request is written, the CHECK is **not** retried on another host. +- Stale idle socket (`closed` / `connection` on a reused connection): retry **once on the same host**. +- Bytes after the response `\r\n\r\n` are `Failed leftover`; the connection is evicted. + +### 10.3 Eligibility (`ShouldScan`) + +CHECK runs only when all of the following are true: + +- `Gate` is `Active` (or the filter otherwise reached the SA stage) +- `SpamAssassin:Enabled` is true +- the session is not allowlisted +- `MaxArticleSize` is `0` **or** `ArtSize < MaxArticleSize` (exclusive) +- `ArtType` has no bits in `ExcludeArtTypes` + +A skip does **not** contact SPAMD and is **not** an `OnFailure` event. Quota reservation already exists and the rest of POST continues. + +Default eligibility matches the historical small-article boundary: `ArtSize < 131072` and not `YEncoded`. A 768 KiB article is not sent to SPAMD under that default. + +### 10.4 Results + +| Scanner outcome | Filter | Reservation | +|-----------------|--------|-------------| +| HAM (`Spam: False`) | Accept | keep lease | +| SPAM (`Spam: True`) | `441` | RELEASE | +| Failed + `OnFailure=Reject` | `441` | RELEASE | +| Failed + `OnFailure=Accept` | Accept | keep lease | +| Caller cancellation | throw | RELEASE (`CancellationToken.None`) | + +--- + +## 11. Failure matrix + +Client rejects after `340` are `441 Posting failed`. Success is `240 Article received OK`. + +| Condition | Redis | SPAMD | Queue | Client | Result | +|-----------|-------|-------|-------|--------|--------| +| `Gate=Disabled` | none | none | admitted | `240` | Normal POST; no reserve | +| `Gate=Closed` | none | none | no | `441` | No reserve, no CHECK | +| CIDR / account deny | none | none | no | `441` | Before quota | +| ArtType deny | none | none | no | `441` | Before quota | +| Unauthenticated + `Active` | none | none | no | `441` | `reason=unauthenticated` | +| Quota denial | deny writes nothing | none | no | `441` | Live reservations of others still count | +| SPAMD spam | RELEASE | CHECK ran | no | `441` | Not remembered | +| SPAMD failure `Reject` | RELEASE | failed | no | `441` | Not an eligibility skip | +| SPAMD failure `Accept` | COMMIT on admit | failed | admitted | `240` | Reservation kept | +| Redis unavailable at `RESERVE` | fail-closed | none | no | `441` | Intentional | +| `TryAdmit` failure | RELEASE | already done | no | `441` | Article not queued | +| `COMMIT` unavailable | reservation left | — | **admitted** | `240` | Under-count until TTL; not a rollback | +| Successful POST | COMMIT | HAM or skip | admitted | `240` | Remember | +| Cancel after `RESERVE` | RELEASE (`None`) | — | no | no completion | No `240` | + +Eligibility skips (size / excluded ArtType / allowlist / SA disabled) are **not** dependency failures: SPAMD is never contacted. + +--- + +## 12. Observability + +Filter reasons stay off the wire. Use logs. + +| EventId | Level | When | +|---------|-------|------| +| 2800 | Information | Reject (`stage`, `reason`, `account`, `artId`) | +| 2801 | Debug | Accept (`Gate` disabled or complete; `reserved`) | +| 2802 | Warning | `COMMIT` NOOP after successful admit | +| 2803 | Warning | `COMMIT` unavailable after admit (no RELEASE) | +| 2804 | Debug | `RELEASE` status | +| 2805 | Warning | Policy refresh failed; last-good retained | +| 2806 | Warning | SpamAssassin failure (`action`, `detail`) | +| 2807 | Information | SPAMD transport stopped (connects/checks/reuses/reconnects/evictions) | +| 2808 | Information | Policy published (`revision`, gate, SA flags, hosts, pool) | +| 2809 | Error | Initial NntpDB policy load failed | +| 2810 | Information | Policy revision changed (`from` → `to`) | +| 2811 | Error | Policy repository failed | + +`artId` in these logs is `ArticleRecord.ArtHash` hex, not the identical-body XXH3-64. + +In-process counters (not a metrics HTTP endpoint): + +- `PostFilterMetrics.CommitNoop` / `CommitUnavailable` — incremented only after successful `TryAdmit` +- `SpamdTransportMetrics` — summarized at transport stop (EventId 2807) + +**Current limitations (do not add noisy logging solely to close these):** + +- EventIds 2800–2808 **overlap** RabbitMQ lifecycle EventIds. Filter on message text (`PostFilter …`) or logger category, not EventId alone. +- Successful 60s refresh of an unchanged published revision is not logged and does not replace the snapshot. +- `RESERVE` Unavailable appears as EventId 2800 `stage=Quota reason=Unavailable`, not a dedicated Redis-down event. + +POST also emits existing `PostLogMessages` Accepted/Rejected lines (`441` category `PolicyRejected` plus the internal reason). + +--- + +## 13. Safe deployment / activation + +1. Apply [`docs/schema/postfilter.sql`](../../schema/postfilter.sql) (tables, trigger, Disabled seed). Do not add `Nntpd:PostFilter` to appsettings. +2. Confirm `nntppostfiltercurrent.policy_id = 1` exists. Initial load is required for RUNNING. +3. Confirm Redis (`Redis:Host` / startup PING). PostFilter uses the same multiplexer for quota state. +4. If SpamAssassin will be used, confirm SPAMD from every NNTPD host, then publish a new revision with `sa_enabled='Y'` and host rows. +5. Change policy only by inserting a complete new revision and updating `nntppostfiltercurrent` in one transaction. Keep `gate='Disabled'` while rehearsing. +6. Start with **conservative** ceilings (or `0` to disable a dimension). There is no universal production number. +7. Watch EventId 2808 (`revision=`) on each node after a new publish. Failed compile/refresh keeps last-good (2805 / 2809). +8. Publish `gate='Active'` as a new revision. Within 60s both nodes should log 2810 and the new revision. +9. POST a known HAM from an authenticated account that is not denied. Expect `240` and a Redis COMMIT. +10. POST enough volume to hit a test ceiling. Expect `441` and EventId 2800 `stage=Quota`. +11. Confirm clients see only `240` / `441`. Confirm both nodes report the same revision. +12. Inspect Redis keys `nntpd:pf:q:*` / `nntpd:pf:m:*` (runtime state, not policy). + +To disable quickly: insert a new revision that copies the current scalars with `gate='Disabled'` (or `Closed`), publish `nntppostfiltercurrent`, and wait up to 60s. Do not `UPDATE` the live revision in place. + +--- + +## 14. Troubleshooting + +| Symptom | Distinguish | What to check | +|---------|-------------|---------------| +| Every POST is `441` | Closed gate vs deny vs quota vs SA vs receive/parse | EventId 2800 `stage`/`reason`. `Closed` / `denied` / `arttype` do not touch Redis. `Nntpd:MaxArticleSize` and catalogue policy reject **before** PostFilter (POST rejected logs, no 2800). | +| Redis unavailable | `reason=Unavailable` at Quota | Redis process, `Redis:Host`, multiplexer health. Fail-closed is expected. | +| Quota rejects unexpectedly | Ceiling includes **live reservations** | Other in-flight POSTs, short window still open, shared cluster state, case-sensitive account names, `ArtSize` vs assumed size. | +| SPAMD unavailable | EventId 2806 `action=Reject` or `Accept` | Hosts/port/firewall. `OnFailure=Accept` still yields `240`. | +| SPAMD rejects HAM | `reason=spam` | Disposable scan headers (Received/To/X-Usenet-Newsgroups). Tune SA rules, not ArticleRecord. | +| SPAMD never contacted | Eligibility skip vs Disabled vs allowlist vs Closed | Debug 2801 `reserved=false` (Disabled). Size: `ArtSize >= MaxArticleSize`. `ExcludeArtTypes`. Allowlist. `Enabled=false`. | +| Large articles skip SA | Exclusive size gate | Default `131072`. `0` disables the size gate. | +| yEnc / listed ArtType skip SA | Default `ExcludeArtTypes=["YEncoded"]` | Empty array disables type exclusion. | +| Allowlisted account still quota-denied | Allowlist skips SA only | Deny/ArtType/quotas still apply. | +| `240` but COMMIT warning | EventId 2802 / 2803 | Article is queued. Do not RELEASE. Committed count may lag until TTL. | +| Reservation fields linger | RELEASE Unavailable or crash | Hold TTL (10s, or `OperationTimeout+1s` when SA is enabled). Expected. | + +--- + +## 15. Current limitations + +Absent from this implementation (not implied as a roadmap): + +- DNSBL, URIBL, TOR, rDNS / domain identity +- Dictionary / bad-word / regex filtering +- Reject-streak logic +- Anonymous / IP quotas (quota identity is AUTH username only) +- Moderation as a PostFilter outcome (`Moderate` is not implemented; catalogue `m` submission is a separate path) +- Header/body mutation of the canonical article +- Generic filter framework / plugin stages beyond gate, deny, ArtType, quota, and SpamAssassin +- Per-POST metrics export or a dedicated Redis-down EventId +- Configurable reservation TTL +- IHAVE ArtType capability enforcement (classification is after the `235` boundary) +- ARTICLE/HEAD/BODY/STAT ArtType capability enforcement (no article repository exposes persisted `ArticleType`) + +--- + +## 16. Test / validation evidence + +POST-boundary coverage lives in `PostFilterProductionPolicyMatrixTests` (test-local snapshots; production `Gate` stays `Disabled`): + +- Gate `Disabled` / `Closed` +- Account and CIDR deny; ArtType deny; allowlist (SA skip only) +- Message, byte, and identical-body quotas +- Account isolation; shared in-memory quota across two reservation identities +- SPAMD HAM / SPAM; size and ArtType skip; allowlisted skip +- `OnFailure=Reject` / `OnFailure=Accept` +- Redis unavailable at RESERVE; `TryAdmit` failure; cancel after RESERVE +- COMMIT unavailable after admit (`240`, queued, no RELEASE) +- Complete reserve → scan → admit → COMMIT → Remember +- `ArticleRecord` immutability and disposable SPAMD representation + +Additional suites: `PostFilterEvaluatorTests`, `PostFilterPostCommandTests`, quota Lua/engine/live-cluster tests, SPAMD transport/eligibility/builder tests. + +Ordinary CI talks to a loopback SPAMC listener (`LocalSpamdServer`) through `SpamdCheckClient`. It does not dial a SpamAssassin host. Compatibility with a real `spamd` is opt-in: set `VECTORNNTP_SPAMD_INTEGRATION=host:port` before running `PostFilterSpamdLiveTests` methods marked `SpamdIntegrationFact`. When the variable is absent those tests are skipped. When it is present they use that endpoint only and fail if it is unreachable. There is no built-in address. diff --git a/docs/schema/nntptransit.sql b/docs/schema/nntptransit.sql new file mode 100644 index 00000000..e5598cee --- /dev/null +++ b/docs/schema/nntptransit.sql @@ -0,0 +1,436 @@ +-- NntpDB Transit control-plane schema. +-- This file documents the catalogue installed in the nntp database. +-- Do not re-apply it against that catalogue. NNTPD only SELECTs. +-- One stable peer identity has a Receive plane and a Send plane. +-- Receive and Send revisions are independent. nntptransitpublication names one of each, +-- plus one global revision. nntptransitcurrent points at exactly one publication. +-- A publication is valid only when those two peer revisions contain the same identifiers. +-- Published revision rows are immutable. Change a plane by inserting a new revision, then a new publication. +-- There is no application migration runner. +-- Conventions match existing NntpDB catalogues: InnoDB, snake_case, CHAR(1) Y/N, DATETIME(3), utf8mb4_unicode_ci. +-- Peer identifiers are ascii/ascii_bin so they are not case-folded. +-- +-- Connection limits are the enable switches. There is no Enabled flag. +-- max_inbound = 0 closes receive. max_outbound = 0 closes send. Zero is not unlimited. +-- article_types is the ArticleType / ArticleTypeCapabilities mask (0..65535). +-- 0 permits no classified article. 1 is ArticleType.Default (text only). 65535 is unrestricted. +-- patterns is one NewsfeedsPattern expression per plane. '*' is all groups. +-- Empty and whitespace-only patterns are rejected here. Wildmat grammar is NewsfeedsPattern.TryParse. +-- +-- Not stored here (process-local): +-- Nntpd:Transit:StreamOutstandingArticleDepth +-- Nntpd:TransitQueueMemoryLimit +-- Nntpd:ArticleIngestion:MaxArticleBytes +-- +-- nntptransitsend.max_article_bytes: NULL is unlimited. A non-NULL value is an explicit +-- maximum article size in bytes (1–2147483647). 0 is not unlimited. +-- nntptransitreceive.max_article_bytes stays NOT NULL in that same positive range. +-- Send revision 1 was seeded from nntpsharedconfig.maxartsize (5242880), not from each peer's receive size. + +CREATE TABLE nntptransitpeer ( + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + peer_name VARCHAR(256) NOT NULL, + PRIMARY KEY (identifier), + CONSTRAINT chk_nntptransitpeer_identifier CHECK ( + CHAR_LENGTH(identifier) BETWEEN 1 AND 256 + AND identifier REGEXP '^[!-~]+$' + ), + CONSTRAINT chk_nntptransitpeer_name CHECK ( + CHAR_LENGTH(peer_name) BETWEEN 1 AND 256 + AND CHAR_LENGTH(TRIM(peer_name)) >= 1 + AND peer_name NOT REGEXP '[[:cntrl:]]' + ) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Stable Transit peer identity. peer_name is display only'; + +CREATE TABLE nntptransitreceiverevision ( + revision BIGINT UNSIGNED NOT NULL, + published_utc DATETIME(3) NOT NULL, + PRIMARY KEY (revision), + CONSTRAINT chk_nntptransitreceiverevision_revision CHECK (revision >= 1) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Immutable header for one complete Receive catalogue revision'; + +CREATE TABLE nntptransitreceive ( + revision BIGINT UNSIGNED NOT NULL, + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + max_inbound INT UNSIGNED NOT NULL, + username VARCHAR(255) NOT NULL, + password VARCHAR(255) NOT NULL, + defer_on_duplicate CHAR(1) NOT NULL, + max_article_bytes INT UNSIGNED NOT NULL, + article_types INT UNSIGNED NOT NULL, + patterns VARCHAR(4096) NOT NULL, + PRIMARY KEY (revision, identifier), + CONSTRAINT fk_nntptransitreceive_revision + FOREIGN KEY (revision) REFERENCES nntptransitreceiverevision (revision), + CONSTRAINT fk_nntptransitreceive_peer + FOREIGN KEY (identifier) REFERENCES nntptransitpeer (identifier), + CONSTRAINT chk_nntptransitreceive_inbound CHECK (max_inbound <= 4096), + CONSTRAINT chk_nntptransitreceive_credentials CHECK ( + (CHAR_LENGTH(username) = 0 AND CHAR_LENGTH(password) = 0) + OR (CHAR_LENGTH(username) > 0 AND CHAR_LENGTH(password) > 0) + ), + CONSTRAINT chk_nntptransitreceive_defer CHECK (defer_on_duplicate IN ('Y', 'N')), + CONSTRAINT chk_nntptransitreceive_size CHECK (max_article_bytes BETWEEN 1 AND 2147483647), + CONSTRAINT chk_nntptransitreceive_types CHECK (article_types <= 65535), + CONSTRAINT chk_nntptransitreceive_patterns CHECK ( + CHAR_LENGTH(patterns) BETWEEN 1 AND 4096 + AND CHAR_LENGTH(TRIM(patterns)) >= 1 + ) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Receive plane: what this peer may send into us. max_inbound 0 closes receive'; + +CREATE TABLE nntptransitallowfrom ( + revision BIGINT UNSIGNED NOT NULL, + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + ordinal INT UNSIGNED NOT NULL, + entry VARCHAR(255) NOT NULL, + PRIMARY KEY (revision, identifier, ordinal), + CONSTRAINT fk_nntptransitallowfrom_receive + FOREIGN KEY (revision, identifier) REFERENCES nntptransitreceive (revision, identifier), + CONSTRAINT chk_nntptransitallowfrom_entry CHECK (CHAR_LENGTH(TRIM(entry)) BETWEEN 1 AND 255) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Receive AllowFrom sources in configuration order. Zero rows means the peer never matches'; + +CREATE TABLE nntptransitsendrevision ( + revision BIGINT UNSIGNED NOT NULL, + published_utc DATETIME(3) NOT NULL, + PRIMARY KEY (revision), + CONSTRAINT chk_nntptransitsendrevision_revision CHECK (revision >= 1) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Immutable header for one complete Send catalogue revision'; + +CREATE TABLE nntptransitsend ( + revision BIGINT UNSIGNED NOT NULL, + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + max_outbound INT UNSIGNED NOT NULL, + ssl_mode VARCHAR(16) NOT NULL, + username VARCHAR(255) NOT NULL, + password VARCHAR(255) NOT NULL, + max_article_bytes INT UNSIGNED NULL COMMENT 'NULL means unlimited. A non-NULL value is the maximum article size in bytes', + article_types INT UNSIGNED NOT NULL, + patterns VARCHAR(4096) NOT NULL, + path_token VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (revision, identifier), + CONSTRAINT fk_nntptransitsend_revision + FOREIGN KEY (revision) REFERENCES nntptransitsendrevision (revision), + CONSTRAINT fk_nntptransitsend_peer + FOREIGN KEY (identifier) REFERENCES nntptransitpeer (identifier), + CONSTRAINT chk_nntptransitsend_outbound CHECK (max_outbound <= 4096), + CONSTRAINT chk_nntptransitsend_ssl CHECK (ssl_mode IN ('None', 'Tls', 'StartTls')), + CONSTRAINT chk_nntptransitsend_credentials CHECK ( + (CHAR_LENGTH(username) = 0 AND CHAR_LENGTH(password) = 0) + OR (CHAR_LENGTH(username) > 0 AND CHAR_LENGTH(password) > 0) + ), + CONSTRAINT chk_nntptransitsend_size CHECK ( + max_article_bytes IS NULL OR max_article_bytes BETWEEN 1 AND 2147483647 + ), + CONSTRAINT chk_nntptransitsend_types CHECK (article_types <= 65535), + CONSTRAINT chk_nntptransitsend_patterns CHECK ( + CHAR_LENGTH(patterns) BETWEEN 1 AND 4096 + AND CHAR_LENGTH(TRIM(patterns)) >= 1 + ), + CONSTRAINT chk_nntptransitsend_path CHECK ( + CHAR_LENGTH(path_token) <= 255 + AND path_token NOT REGEXP '[[:cntrl:]]' + ) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Send plane: what we are willing to send to this peer. max_outbound 0 closes send'; + +CREATE TABLE nntptransitendpoint ( + revision BIGINT UNSIGNED NOT NULL, + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + ordinal INT UNSIGNED NOT NULL, + host VARCHAR(255) NOT NULL, + port INT UNSIGNED NOT NULL, + PRIMARY KEY (revision, identifier, ordinal), + UNIQUE KEY uq_nntptransitendpoint_host (revision, identifier, host, port), + CONSTRAINT fk_nntptransitendpoint_send + FOREIGN KEY (revision, identifier) REFERENCES nntptransitsend (revision, identifier), + CONSTRAINT chk_nntptransitendpoint_host CHECK (CHAR_LENGTH(TRIM(host)) BETWEEN 1 AND 255), + CONSTRAINT chk_nntptransitendpoint_port CHECK (port BETWEEN 1 AND 65535) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Send ConnectTo endpoints. Host is unbracketed. Zero rows is allowed'; + +CREATE TABLE nntptransitsendpathexclude ( + revision BIGINT UNSIGNED NOT NULL, + identifier VARCHAR(256) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, + ordinal INT UNSIGNED NOT NULL, + path_token VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, + PRIMARY KEY (revision, identifier, ordinal), + UNIQUE KEY uq_nntptransitsendpathexclude_token (revision, identifier, path_token), + CONSTRAINT fk_nntptransitsendpathexclude_send + FOREIGN KEY (revision, identifier) REFERENCES nntptransitsend (revision, identifier), + CONSTRAINT chk_nntptransitsendpathexclude_token CHECK ( + CHAR_LENGTH(path_token) BETWEEN 1 AND 255 + AND path_token NOT REGEXP '[[:cntrl:]]' + ) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Additional Path hops that suppress sending to this peer. Distinct from path_token and from identifier'; + +CREATE TABLE nntptransitglobalrevision ( + revision BIGINT UNSIGNED NOT NULL, + published_utc DATETIME(3) NOT NULL, + want_trash CHAR(1) NOT NULL, + log_trash CHAR(1) NOT NULL, + PRIMARY KEY (revision), + CONSTRAINT chk_nntptransitglobalrevision_revision CHECK (revision >= 1), + CONSTRAINT chk_nntptransitglobalrevision_want CHECK (want_trash IN ('Y', 'N')), + CONSTRAINT chk_nntptransitglobalrevision_log CHECK (log_trash IN ('Y', 'N')) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Site-wide Transit junk policy. Not per peer'; + +CREATE TABLE nntptransitpublication ( + publication_id BIGINT UNSIGNED NOT NULL, + receive_revision BIGINT UNSIGNED NOT NULL, + send_revision BIGINT UNSIGNED NOT NULL, + global_revision BIGINT UNSIGNED NOT NULL, + published_utc DATETIME(3) NOT NULL, + PRIMARY KEY (publication_id), + CONSTRAINT fk_nntptransitpublication_receive + FOREIGN KEY (receive_revision) REFERENCES nntptransitreceiverevision (revision), + CONSTRAINT fk_nntptransitpublication_send + FOREIGN KEY (send_revision) REFERENCES nntptransitsendrevision (revision), + CONSTRAINT fk_nntptransitpublication_global + FOREIGN KEY (global_revision) REFERENCES nntptransitglobalrevision (revision), + CONSTRAINT chk_nntptransitpublication_id CHECK (publication_id >= 1) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='One atomic Transit snapshot: receive revision + send revision + global revision'; + +CREATE TABLE nntptransitcurrent ( + policy_id TINYINT UNSIGNED NOT NULL, + publication_id BIGINT UNSIGNED NOT NULL, + PRIMARY KEY (policy_id), + CONSTRAINT fk_nntptransitcurrent_publication + FOREIGN KEY (publication_id) REFERENCES nntptransitpublication (publication_id), + CONSTRAINT chk_nntptransitcurrent_policy_id CHECK (policy_id = 1) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Singleton published Transit snapshot (policy_id=1)'; + +DELIMITER $$ + +CREATE TRIGGER trg_nntptransitpublication_peer_set +BEFORE INSERT ON nntptransitpublication +FOR EACH ROW +BEGIN + IF (SELECT COUNT(*) FROM nntptransitreceive WHERE revision = NEW.receive_revision) + <> (SELECT COUNT(*) FROM nntptransitsend WHERE revision = NEW.send_revision) + OR EXISTS ( + SELECT 1 + FROM nntptransitreceive r + LEFT JOIN nntptransitsend s + ON s.revision = NEW.send_revision AND s.identifier = r.identifier + WHERE r.revision = NEW.receive_revision AND s.identifier IS NULL + ) + THEN + SIGNAL SQLSTATE '45000' + SET MESSAGE_TEXT = 'Transit receive and send revisions must contain the same peer identifiers'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitcurrent_publication_forward +BEFORE UPDATE ON nntptransitcurrent +FOR EACH ROW +BEGIN + IF NEW.publication_id <= OLD.publication_id THEN + SIGNAL SQLSTATE '45000' + SET MESSAGE_TEXT = 'nntptransitcurrent.publication_id must increase'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitreceive_insert_frozen +BEFORE INSERT ON nntptransitreceive +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = NEW.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitreceive_update_frozen +BEFORE UPDATE ON nntptransitreceive +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitreceive_delete_frozen +BEFORE DELETE ON nntptransitreceive +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitallowfrom_insert_frozen +BEFORE INSERT ON nntptransitallowfrom +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = NEW.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitallowfrom_update_frozen +BEFORE UPDATE ON nntptransitallowfrom +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitallowfrom_delete_frozen +BEFORE DELETE ON nntptransitallowfrom +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE receive_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit receive revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsend_insert_frozen +BEFORE INSERT ON nntptransitsend +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = NEW.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsend_update_frozen +BEFORE UPDATE ON nntptransitsend +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsend_delete_frozen +BEFORE DELETE ON nntptransitsend +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitendpoint_insert_frozen +BEFORE INSERT ON nntptransitendpoint +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = NEW.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitendpoint_update_frozen +BEFORE UPDATE ON nntptransitendpoint +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitendpoint_delete_frozen +BEFORE DELETE ON nntptransitendpoint +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsendpathexclude_insert_frozen +BEFORE INSERT ON nntptransitsendpathexclude +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = NEW.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsendpathexclude_update_frozen +BEFORE UPDATE ON nntptransitsendpathexclude +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitsendpathexclude_delete_frozen +BEFORE DELETE ON nntptransitsendpathexclude +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE send_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit send revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitglobalrevision_update_frozen +BEFORE UPDATE ON nntptransitglobalrevision +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE global_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit global revision is immutable'; + END IF; +END$$ + +CREATE TRIGGER trg_nntptransitglobalrevision_delete_frozen +BEFORE DELETE ON nntptransitglobalrevision +FOR EACH ROW +BEGIN + IF EXISTS (SELECT 1 FROM nntptransitpublication WHERE global_revision = OLD.revision) THEN + SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Published Transit global revision is immutable'; + END IF; +END$$ + +DELIMITER ; + +INSERT INTO nntptransitpeer (identifier, peer_name) VALUES + ('blueworld-hosting', 'Blueworld Hosting'), + ('giganews', 'Giganews, Inc.'), + ('usenet-ninja', 'Usenet Ninja'); + +INSERT INTO nntptransitreceiverevision (revision, published_utc) VALUES (1, UTC_TIMESTAMP(3)); +INSERT INTO nntptransitsendrevision (revision, published_utc) VALUES (1, UTC_TIMESTAMP(3)); +INSERT INTO nntptransitglobalrevision (revision, published_utc, want_trash, log_trash) +VALUES (1, UTC_TIMESTAMP(3), 'Y', 'Y'); + +INSERT INTO nntptransitreceive ( + revision, identifier, max_inbound, username, password, defer_on_duplicate, + max_article_bytes, article_types, patterns +) VALUES + (1, 'blueworld-hosting', 10, '', '', 'Y', 1048576, 65535, '*'), + (1, 'giganews', 80, '', '', 'Y', 5242880, 65535, '*'), + (1, 'usenet-ninja', 10, '', '', 'Y', 5242880, 65535, '*'); + +INSERT INTO nntptransitallowfrom (revision, identifier, ordinal, entry) VALUES + (1, 'blueworld-hosting', 0, 'usenet.blueworldhosting.com'), + (1, 'giganews', 0, 'news-out.nntp.giganews.com'), + (1, 'usenet-ninja', 0, '198.18.0.0/15'); + +INSERT INTO nntptransitsend ( + revision, identifier, max_outbound, ssl_mode, username, password, + max_article_bytes, article_types, patterns, path_token +) VALUES + (1, 'blueworld-hosting', 10, 'None', '', '', 5242880, 65535, '*,!unidata.*,!control.*,!junk', 'usenet.blueworldhosting.com'), + (1, 'giganews', 80, 'None', '', '', 5242880, 65535, '*,!unidata.*', 'nntp.giganews.com'), + (1, 'usenet-ninja', 10, 'None', '', '', 5242880, 65535, '*', 'nntp.usenet.ninja'); + +INSERT INTO nntptransitendpoint (revision, identifier, ordinal, host, port) VALUES + (1, 'blueworld-hosting', 0, 'usenet.blueworldhosting.com', 119), + (1, 'giganews', 0, 'opticnetworks-in.nntp.ord.giganews.com', 119); + +INSERT INTO nntptransitpublication ( + publication_id, receive_revision, send_revision, global_revision, published_utc +) VALUES (1, 1, 1, 1, UTC_TIMESTAMP(3)); + +INSERT INTO nntptransitcurrent (policy_id, publication_id) VALUES (1, 1); diff --git a/docs/schema/nntpusers-account-art-type.sql b/docs/schema/nntpusers-account-art-type.sql new file mode 100644 index 00000000..9737d1e6 --- /dev/null +++ b/docs/schema/nntpusers-account-art-type.sql @@ -0,0 +1,27 @@ +-- Additive nntpusers.account_art_type for AUTHINFO ArticleType capability. +-- Apply to the existing account database before deploying an NNTPD that +-- SELECTs account_art_type. Idempotent. Does not drop nntpusers. +-- Default 65535 is ArticleTypeCapabilities.All (unrestricted for existing users). +-- NNTPD does not run this script. + +DROP PROCEDURE IF EXISTS nntpusers_add_account_art_type; + +DELIMITER $$ +CREATE PROCEDURE nntpusers_add_account_art_type() +BEGIN + IF NOT EXISTS ( + SELECT 1 + FROM information_schema.columns + WHERE table_schema = DATABASE() + AND table_name = 'nntpusers' + AND column_name = 'account_art_type' + ) THEN + ALTER TABLE nntpusers + ADD COLUMN account_art_type INT UNSIGNED NOT NULL DEFAULT 65535 + COMMENT 'ArticleType flags; 65535=ArticleTypeCapabilities.All (unrestricted)'; + END IF; +END$$ +DELIMITER ; + +CALL nntpusers_add_account_art_type(); +DROP PROCEDURE nntpusers_add_account_art_type; diff --git a/docs/schema/postfilter.sql b/docs/schema/postfilter.sql new file mode 100644 index 00000000..12a881c0 --- /dev/null +++ b/docs/schema/postfilter.sql @@ -0,0 +1,200 @@ +-- NntpDB PostFilter control-plane schema. +-- Apply this script to an empty NntpDB before VectorNNTP.NNTPD starts. +-- PostFilter policy is authoritative in NntpDB, not appsettings. +-- Policy rows are versioned and append-only. +-- nntppostfiltercurrent identifies the published revision. +-- Collection rows are keyed by revision so a load cannot mix revisions. +-- Seed revision 1 is Gate=Disabled (no filtering until an operator publishes). +-- Change policy by inserting a complete new revision, then publishing it. +-- Do not modify a published revision in place. +-- NNTPD only SELECTs. There is no application migration runner. +-- Conventions match existing NntpDB catalogues: InnoDB, snake_case, +-- CHAR(1) Y/N, DATETIME(3), utf8mb4_unicode_ci. + +CREATE TABLE nntppostfilterpolicy ( + revision BIGINT UNSIGNED NOT NULL, + updated_utc DATETIME(3) NOT NULL, + gate VARCHAR(16) NOT NULL, + long_window_ms BIGINT UNSIGNED NOT NULL, + short_window_ms BIGINT UNSIGNED NOT NULL, + max_messages_long BIGINT UNSIGNED NOT NULL, + max_bytes_long BIGINT UNSIGNED NOT NULL, + max_identical_long BIGINT UNSIGNED NOT NULL, + max_messages_short BIGINT UNSIGNED NOT NULL, + max_bytes_short BIGINT UNSIGNED NOT NULL, + max_identical_short BIGINT UNSIGNED NOT NULL, + sa_enabled CHAR(1) NOT NULL, + sa_on_failure VARCHAR(16) NULL, + sa_max_article_size INT NOT NULL, + sa_port INT UNSIGNED NOT NULL, + sa_protocol_version VARCHAR(16) NOT NULL, + sa_max_connections INT UNSIGNED NOT NULL, + sa_host_selection VARCHAR(16) NOT NULL, + sa_connect_timeout_ms INT UNSIGNED NOT NULL, + sa_operation_timeout_ms INT UNSIGNED NOT NULL, + PRIMARY KEY (revision), + CONSTRAINT chk_nntppostfilterpolicy_revision CHECK (revision >= 1), + CONSTRAINT chk_nntppostfilterpolicy_gate CHECK (gate IN ('Disabled', 'Active', 'Closed')), + CONSTRAINT chk_nntppostfilterpolicy_windows CHECK (long_window_ms >= 1 AND short_window_ms >= 1), + CONSTRAINT chk_nntppostfilterpolicy_sa_enabled CHECK (sa_enabled IN ('Y', 'N')), + CONSTRAINT chk_nntppostfilterpolicy_sa_on_failure CHECK ( + sa_on_failure IS NULL OR sa_on_failure IN ('Reject', 'Accept') + ), + CONSTRAINT chk_nntppostfilterpolicy_sa_required CHECK ( + sa_enabled <> 'Y' OR sa_on_failure IN ('Reject', 'Accept') + ), + CONSTRAINT chk_nntppostfilterpolicy_sa_size CHECK (sa_max_article_size >= 0), + CONSTRAINT chk_nntppostfilterpolicy_sa_port CHECK (sa_port BETWEEN 1 AND 65535), + CONSTRAINT chk_nntppostfilterpolicy_sa_pool CHECK (sa_max_connections BETWEEN 1 AND 32), + CONSTRAINT chk_nntppostfilterpolicy_sa_hosts CHECK (sa_host_selection IN ('RoundRobin', 'Failover')), + CONSTRAINT chk_nntppostfilterpolicy_sa_timeouts CHECK ( + sa_connect_timeout_ms >= 1 + AND sa_operation_timeout_ms >= 1 + AND sa_connect_timeout_ms <= sa_operation_timeout_ms + ) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Append-only PostFilter scalar revisions'; + +CREATE TABLE nntppostfiltercurrent ( + policy_id TINYINT UNSIGNED NOT NULL, + revision BIGINT UNSIGNED NOT NULL, + PRIMARY KEY (policy_id), + CONSTRAINT chk_nntppostfiltercurrent_policy_id CHECK (policy_id = 1), + CONSTRAINT fk_nntppostfiltercurrent_revision + FOREIGN KEY (revision) REFERENCES nntppostfilterpolicy (revision) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Singleton published PostFilter revision (policy_id=1)'; + +CREATE TABLE nntppostfilteraccounts ( + revision BIGINT UNSIGNED NOT NULL, + list_kind VARCHAR(8) NOT NULL, + account_name CHAR(32) NOT NULL, + PRIMARY KEY (revision, list_kind, account_name), + CONSTRAINT chk_nntppostfilteraccounts_kind CHECK (list_kind IN ('deny', 'allow')), + CONSTRAINT chk_nntppostfilteraccounts_name CHECK (account_name REGEXP '^[0-9a-f]{32}$'), + CONSTRAINT fk_nntppostfilteraccounts_revision + FOREIGN KEY (revision) REFERENCES nntppostfilterpolicy (revision) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='PostFilter deny/allow MD5(hex) AUTH identities; real usernames are not stored'; + +CREATE TABLE nntppostfiltercidrs ( + revision BIGINT UNSIGNED NOT NULL, + list_kind VARCHAR(8) NOT NULL, + cidr VARCHAR(64) NOT NULL, + PRIMARY KEY (revision, list_kind, cidr), + CONSTRAINT chk_nntppostfiltercidrs_kind CHECK (list_kind IN ('deny', 'allow')), + CONSTRAINT chk_nntppostfiltercidrs_value CHECK (CHAR_LENGTH(TRIM(cidr)) > 0), + CONSTRAINT fk_nntppostfiltercidrs_revision + FOREIGN KEY (revision) REFERENCES nntppostfilterpolicy (revision) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='PostFilter deny/allow client CIDRs for one revision'; + +CREATE TABLE nntppostfilterarttypes ( + revision BIGINT UNSIGNED NOT NULL, + list_kind VARCHAR(16) NOT NULL, + art_type ENUM( + 'Default', + 'Control', + 'Cancel', + 'Mime', + 'Binary', + 'UuEncode', + 'Base64', + 'YEncoded', + 'BommaNews', + 'UniData', + 'Multipart', + 'Html', + 'PostScript', + 'BinHex', + 'Partial', + 'PgpMessage' + ) NOT NULL, + PRIMARY KEY (revision, list_kind, art_type), + CONSTRAINT chk_nntppostfilterarttypes_kind CHECK (list_kind IN ('reject', 'sa_exclude')), + CONSTRAINT fk_nntppostfilterarttypes_revision + FOREIGN KEY (revision) REFERENCES nntppostfilterpolicy (revision) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='PostFilter reject and SA-exclude ArtTypes; one classifier name per row'; + +CREATE TABLE nntppostfiltersahosts ( + revision BIGINT UNSIGNED NOT NULL, + host_order INT UNSIGNED NOT NULL, + host VARCHAR(255) NOT NULL, + PRIMARY KEY (revision, host_order), + UNIQUE KEY uq_nntppostfiltersahosts_host (revision, host), + CONSTRAINT chk_nntppostfiltersahosts_host CHECK (CHAR_LENGTH(TRIM(host)) > 0), + CONSTRAINT fk_nntppostfiltersahosts_revision + FOREIGN KEY (revision) REFERENCES nntppostfilterpolicy (revision) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='PostFilter SPAMD hosts in compiled order for one revision'; + +CREATE TABLE nntppostfilterrejections ( + rejection_id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, + rejected_utc DATETIME(3) NOT NULL, + revision BIGINT UNSIGNED NOT NULL, + account_name CHAR(32) NULL, + source_ip VARBINARY(16) NOT NULL, + art_type INT UNSIGNED NOT NULL, + message_id VARCHAR(250) NULL, + article_size INT UNSIGNED NOT NULL, + stage VARCHAR(32) NOT NULL, + reason VARCHAR(64) NOT NULL, + sa_status VARCHAR(16) NULL, + sa_score DECIMAL(8,3) NULL, + sa_threshold DECIMAL(8,3) NULL, + article_payload LONGBLOB NULL, + PRIMARY KEY (rejection_id), + KEY ix_nntppostfilterrejections_rejected_utc (rejected_utc), + KEY ix_nntppostfilterrejections_account (account_name), + KEY ix_nntppostfilterrejections_revision (revision), + CONSTRAINT chk_nntppostfilterrejections_source_ip CHECK (OCTET_LENGTH(source_ip) IN (4, 16)), + CONSTRAINT chk_nntppostfilterrejections_account CHECK ( + account_name IS NULL OR account_name REGEXP '^[0-9a-f]{32}$' + ), + CONSTRAINT chk_nntppostfilterrejections_stage CHECK (CHAR_LENGTH(TRIM(stage)) > 0), + CONSTRAINT chk_nntppostfilterrejections_reason CHECK (CHAR_LENGTH(TRIM(reason)) > 0) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci + COMMENT='Append-only PostFilter rejection evidence; payload is unstuffed article bytes or NULL'; + +DELIMITER $$ +CREATE TRIGGER trg_nntppostfiltercurrent_revision_forward +BEFORE UPDATE ON nntppostfiltercurrent +FOR EACH ROW +BEGIN + IF NEW.revision <= OLD.revision THEN + SIGNAL SQLSTATE '45000' + SET MESSAGE_TEXT = 'nntppostfiltercurrent.revision must increase'; + END IF; +END$$ +DELIMITER ; + +INSERT INTO nntppostfilterpolicy ( + revision, updated_utc, gate, + long_window_ms, short_window_ms, + max_messages_long, max_bytes_long, max_identical_long, + max_messages_short, max_bytes_short, max_identical_short, + sa_enabled, sa_on_failure, sa_max_article_size, sa_port, + sa_protocol_version, sa_max_connections, sa_host_selection, + sa_connect_timeout_ms, sa_operation_timeout_ms +) VALUES ( + 1, UTC_TIMESTAMP(3), 'Disabled', + 86400000, 600000, + 0, 0, 0, 0, 0, 0, + 'N', NULL, 131072, 783, + '1.5', 4, 'RoundRobin', + 5000, 30000 +); + +INSERT INTO nntppostfilterarttypes (revision, list_kind, art_type) +VALUES (1, 'sa_exclude', 'YEncoded'); + +INSERT INTO nntppostfiltercurrent (policy_id, revision) VALUES (1, 1); + +-- nntppostfilterrejections is append-oriented evidence. This script does not +-- delete rows. Retention is an explicit operator decision; NNTPD has no +-- PostFilter evidence sweeper. +-- +-- Authenticated account ArtType capability is not created here. +-- Apply docs/schema/nntpusers-account-art-type.sql to the existing account +-- database before deploying the AUTHINFO SELECT that reads account_art_type. diff --git a/docs/standards/haproxy/README.md b/docs/standards/haproxy/README.md new file mode 100644 index 00000000..b6e69c5a --- /dev/null +++ b/docs/standards/haproxy/README.md @@ -0,0 +1,40 @@ +# HAProxy PROXY protocol (normative reference) + +## Document + +| Field | Value | +| --------------- | ------------------------------------------------------------------------------- | +| Title | The PROXY protocol — Versions 1 & 2 | +| Author / org | Willy Tarreau / HAProxy Technologies | +| Document date | **2026/04/27** | +| Local file | [`proxy-protocol.txt`](proxy-protocol.txt) | +| Source (raw) | https://raw.githubusercontent.com/haproxy/haproxy/master/doc/proxy-protocol.txt | +| Source (browse) | https://github.com/haproxy/haproxy/blob/master/doc/proxy-protocol.txt | +| SHA-256 | `6b8b5b2dbdcbca8870ae8dca37dc5d2d68ebb6c7638e2309f8a5af753cd66b30` | + +The local file is a **verbatim** copy of the raw document above. Do not edit +it for style or summaries; replace it only by re-downloading and updating the +hash when the upstream document changes. + +## Role in upstream VectorNNTP / pyNNTPD + +Historical note from the standards bundle. VectorNNTPD keeps this file as a protocol +reference; do not infer that PROXY protocol is implemented by every host in this solution. + +- IETF RFCs (`docs/rfcs/`) define NNTP and related **protocol** behaviour. +- This HAProxy document is an **external normative interoperability standard** + for transporting original connection metadata across TCP proxies. +- pyNNTPD implements the **receiver** side needed for a trusted HAProxy + backend: validate and consume PROXY v1/v2 **before** TLS/NNTP processing + when `PYNNTPD_PROXY_TRUSTED_SOURCES` is non-empty. + +## Explicit non-goals + +Optional PROXY SSL / client-certificate **TLVs** described in the document do +**not** imply that pyNNTPD authenticates TLS clients with certificates. +pyNNTPD does not request, inspect, or validate client certificates. + +A pyNNTPD/VectorNNTP PROXY compliance matrix lived beside this file in the upstream +repository (`docs/standards/haproxy-proxy-protocol-compliance.md`). It was not +migrated into VectorNNTPD because this solution does not claim PROXY receiver +behaviour for BackFiller or StorageService. diff --git a/docs/standards/haproxy/proxy-protocol.txt b/docs/standards/haproxy/proxy-protocol.txt new file mode 100644 index 00000000..a7ec9219 --- /dev/null +++ b/docs/standards/haproxy/proxy-protocol.txt @@ -0,0 +1,1053 @@ +2026/04/27 Willy Tarreau + HAProxy Technologies + The PROXY protocol + Versions 1 & 2 + +Abstract + + The PROXY protocol provides a convenient way to safely transport connection + information such as a client's address across multiple layers of NAT or TCP + proxies. It is designed to require little changes to existing components and + to limit the performance impact caused by the processing of the transported + information. + +Revision history + + 2010/10/29 - first version + 2011/03/20 - update: implementation and security considerations + 2012/06/21 - add support for binary format + 2012/11/19 - final review and fixes + 2014/05/18 - modify and extend PROXY protocol version 2 + 2014/06/11 - fix example code to consider ver+cmd merge + 2014/06/14 - fix v2 header check in example code, and update Forwarded spec + 2014/07/12 - update list of implementations (add Squid) + 2015/05/02 - update list of implementations and format of the TLV add-ons + 2017/03/10 - added the checksum, noop and more SSL-related TLV types, + reserved TLV type ranges, added TLV documentation, clarified + string encoding. With contributions from Andriy Palamarchuk + (Amazon.com). + 2020/03/05 - added the unique ID TLV type (Tim Düsterhus) + 2025/09/09 - added SSL-related TLVs for key exchange group and signature + scheme (Steven Collison) + 2026/01/15 - added SSL client certificate TLV (Simon Ser) + 2026/04/27 - clarified UDP usage (Valaphee) + +1. Background + +Relaying TCP connections through proxies generally involves a loss of the +original TCP connection parameters such as source and destination addresses, +ports, and so on. Some protocols make it a little bit easier to transfer such +information. For SMTP, Postfix authors have proposed the XCLIENT protocol [1] +which received broad adoption and is particularly suited to mail exchanges. +For HTTP, there is the "Forwarded" extension [2], which aims at replacing the +omnipresent "X-Forwarded-For" header which carries information about the +original source address, and the less common X-Original-To which carries +information about the destination address. + +However, both mechanisms require a knowledge of the underlying protocol to be +implemented in intermediaries. + +Then comes a new class of products which we'll call "dumb proxies", not because +they don't do anything, but because they're processing protocol-agnostic data. +Both Stunnel[3] and Stud[4] are examples of such "dumb proxies". They talk raw +TCP on one side, and raw SSL on the other one, and do that reliably, without +any knowledge of what protocol is transported on top of the connection. HAProxy +running in pure TCP mode obviously falls into that category as well. + +The problem with such a proxy when it is combined with another one such as +haproxy, is to adapt it to talk the higher level protocol. A patch is available +for Stunnel to make it capable of inserting an X-Forwarded-For header in the +first HTTP request of each incoming connection. HAProxy is able not to add +another one when the connection comes from Stunnel, so that it's possible to +hide it from the servers. + +The typical architecture becomes the following one : + + +--------+ HTTP :80 +----------+ + | client | --------------------------------> | | + | | | haproxy, | + +--------+ +---------+ | 1 or 2 | + / / HTTPS | stunnel | HTTP :81 | listening| + <________/ ---------> | (server | ---------> | ports | + | mode) | | | + +---------+ +----------+ + +The problem appears when haproxy runs with keep-alive on the side towards the +client. The Stunnel patch will only add the X-Forwarded-For header to the first +request of each connection and all subsequent requests will not have it. One +solution could be to improve the patch to make it support keep-alive and parse +all forwarded data, whether they're announced with a Content-Length or with a +Transfer-Encoding, taking care of special methods such as HEAD which announce +data without transferring them, etc... In fact, it would require implementing a +full HTTP stack in Stunnel. It would then become a lot more complex, a lot less +reliable and would not anymore be the "dumb proxy" that fits every purposes. + +In practice, we don't need to add a header for each request because we'll emit +the exact same information every time : the information related to the client +side connection. We could then cache that information in haproxy and use it for +every other request. But that becomes dangerous and is still limited to HTTP +only. + +Another approach consists in prepending each connection with a header reporting +the characteristics of the other side's connection. This method is simpler to +implement, does not require any protocol-specific knowledge on either side, and +completely fits the purpose since what is desired precisely is to know the +other side's connection endpoints. It is easy to perform for the sender (just +send a short header once the connection is established) and to parse for the +receiver (simply perform one read() on the incoming connection to fill in +addresses after an accept). The protocol used to carry connection information +across proxies was thus called the PROXY protocol. + +2. The PROXY protocol header + +This document uses a few terms that are worth explaining here : + - "connection initiator" is the party requesting a new connection + - "connection target" is the party accepting a connection request + - "client" is the party for which a connection was requested + - "server" is the party to which the client desired to connect + - "proxy" is the party intercepting and relaying the connection + from the client to the server. + - "sender" is the party sending data over a connection. + - "receiver" is the party receiving data from the sender. + - "header" or "PROXY protocol header" is the block of connection information + the connection initiator prepends at the beginning of a connection, which + makes it the sender from the protocol point of view. + +The PROXY protocol's goal is to fill the server's internal structures with the +information collected by the proxy that the server would have been able to get +by itself if the client was connecting directly to the server instead of via a +proxy. The information carried by the protocol are the ones the server would +get using getsockname() and getpeername() : + - address family (AF_INET for IPv4, AF_INET6 for IPv6, AF_UNIX) + - socket protocol (SOCK_STREAM for TCP, SOCK_DGRAM for UDP) + - layer 3 source and destination addresses + - layer 4 source and destination ports if any + +Unlike the XCLIENT protocol, the PROXY protocol was designed with limited +extensibility in order to help the receiver parse it very fast. Version 1 was +focused on keeping it human-readable for better debugging possibilities, which +is always desirable for early adoption when few implementations exist. Version +2 adds support for a binary encoding of the header which is much more efficient +to produce and to parse, especially when dealing with IPv6 addresses that are +expensive to emit in ASCII form and to parse. + +In both cases, the protocol simply consists in an easily parsable header placed +by the connection initiator at the beginning of each connection. The protocol +is intentionally stateless in that it does not expect the sender to wait for +the receiver before sending the header, nor the receiver to send anything back. + +This specification supports two header formats, a human-readable format which +is the only format supported in version 1 of the protocol, and a binary format +which is only supported in version 2. Both formats were designed to ensure that +the header cannot be confused with common higher level protocols such as HTTP, +SSL/TLS, FTP or SMTP, and that both formats are easily distinguishable one from +each other for the receiver. + +Version 1 senders MAY only produce the human-readable header format. Version 2 +senders MAY only produce the binary header format. Version 1 receivers MUST at +least implement the human-readable header format. Version 2 receivers MUST at +least implement the binary header format, and it is recommended that they also +implement the human-readable header format for better interoperability and ease +of upgrade when facing version 1 senders. + +Both formats are designed to fit in the smallest TCP segment that any TCP/IP +host is required to support (576 - 40 = 536 bytes). This ensures that the whole +header will always be delivered at once when the socket buffers are still empty +at the beginning of a connection. The sender must always ensure that the header +is sent at once, so that the transport layer maintains atomicity along the path +to the receiver. The receiver may be tolerant to partial headers or may simply +drop the connection when receiving a partial header. Recommendation is to be +tolerant, but implementation constraints may not always easily permit this. It +is important to note that nothing forces any intermediary to forward the whole +header at once, because TCP is a streaming protocol which may be processed one +byte at a time if desired, causing the header to be fragmented when reaching +the receiver. But due to the places where such a protocol is used, the above +simplification generally is acceptable because the risk of crossing such a +device handling one byte at a time is close to zero. + +The receiver MUST NOT start processing the connection before it receives a +complete and valid PROXY protocol header. This is particularly important for +protocols where the receiver is expected to speak first (eg: SMTP, FTP or SSH). +The receiver may apply a short timeout and decide to abort the connection if +the protocol header is not seen within a few seconds (at least 3 seconds to +cover a TCP retransmit). + +For UDP, the PROXY protocol header and the proxied UDP payload MUST be sent in +the same datagram. The sender MUST NOT split the PROXY protocol header across +multiple UDP datagrams, and the receiver MUST parse the header independently +for each received datagram. + +The receiver MUST be configured to only receive the protocol described in this +specification and MUST not try to guess whether the protocol header is present +or not. This means that the protocol explicitly prevents port sharing between +public and private access. Otherwise it would open a major security breach by +allowing untrusted parties to spoof their connection addresses. The receiver +SHOULD ensure proper access filtering so that only trusted proxies are allowed +to use this protocol. + +Some proxies are smart enough to understand transported protocols and to reuse +idle server connections for multiple messages. This typically happens in HTTP +where requests from multiple clients may be sent over the same connection. Such +proxies MUST NOT implement this protocol on multiplexed connections because the +receiver would use the address advertised in the PROXY header as the address of +all forwarded requests's senders. In fact, such proxies are not dumb proxies, +and since they do have a complete understanding of the transported protocol, +they MUST use the facilities provided by this protocol to present the client's +address. + +2.1. Human-readable header format (Version 1) + +This is the format specified in version 1 of the protocol. It consists in one +line of US-ASCII text matching exactly the following block, sent immediately +and at once upon the connection establishment and prepended before any data +flowing from the sender to the receiver : + + - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 ) + Seeing this string indicates that this is version 1 of the protocol. + + - exactly one space : " " ( \x20 ) + + - a string indicating the proxied INET protocol and family. As of version 1, + only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6" + ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Other, unsupported, + or unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E + \x4B \x4E \x4F \x57 \x4E ). For "UNKNOWN", the rest of the line before the + CRLF may be omitted by the sender, and the receiver must ignore anything + presented before the CRLF is found. Note that an earlier version of this + specification suggested to use this when sending health checks, but this + causes issues with servers that reject the "UNKNOWN" keyword. Thus is it + now recommended not to send "UNKNOWN" when the connection is expected to + be accepted, but only when it is not possible to correctly fill the PROXY + line. + + - exactly one space : " " ( \x20 ) + + - the layer 3 source address in its canonical format. IPv4 addresses must be + indicated as a series of exactly 4 integers in the range [0..255] inclusive + written in decimal representation separated by exactly one dot between each + other. Heading zeroes are not permitted in front of numbers in order to + avoid any possible confusion with octal numbers. IPv6 addresses must be + indicated as series of sets of 4 hexadecimal digits (upper or lower case) + delimited by colons between each other, with the acceptance of one double + colon sequence to replace the largest acceptable range of consecutive + zeroes. The total number of decoded bits must exactly be 128. The + advertised protocol family dictates what format to use. + + - exactly one space : " " ( \x20 ) + + - the layer 3 destination address in its canonical format. It is the same + format as the layer 3 source address and matches the same family. + + - exactly one space : " " ( \x20 ) + + - the TCP source port represented as a decimal integer in the range + [0..65535] inclusive. Heading zeroes are not permitted in front of numbers + in order to avoid any possible confusion with octal numbers. + + - exactly one space : " " ( \x20 ) + + - the TCP destination port represented as a decimal integer in the range + [0..65535] inclusive. Heading zeroes are not permitted in front of numbers + in order to avoid any possible confusion with octal numbers. + + - the CRLF sequence ( \x0D \x0A ) + +The maximum line lengths the receiver must support including the CRLF are : + - TCP/IPv4 : + "PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n" + => 5 + 1 + 4 + 1 + 15 + 1 + 15 + 1 + 5 + 1 + 5 + 2 = 56 chars + + - TCP/IPv6 : + "PROXY TCP6 ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n" + => 5 + 1 + 4 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 104 chars + + - unknown connection (short form) : + "PROXY UNKNOWN\r\n" + => 5 + 1 + 7 + 2 = 15 chars + + - worst case (optional fields set to 0xff) : + "PROXY UNKNOWN ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n" + => 5 + 1 + 7 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 107 chars + +So a 108-byte buffer is always enough to store all the line and a trailing zero +for string processing. + +The receiver must wait for the CRLF sequence before starting to decode the +addresses in order to ensure they are complete and properly parsed. If the CRLF +sequence is not found in the first 107 characters, the receiver should declare +the line invalid. A receiver may reject an incomplete line which does not +contain the CRLF sequence in the first atomic read operation. The receiver must +not tolerate a single CR or LF character to end the line when a complete CRLF +sequence is expected. + +Any sequence which does not exactly match the protocol must be discarded and +cause the receiver to abort the connection. It is recommended to abort the +connection as soon as possible so that the sender gets a chance to notice the +anomaly and log it. + +If the announced transport protocol is "UNKNOWN", then the receiver knows that +the sender speaks the correct PROXY protocol with the appropriate version, and +SHOULD accept the connection and use the real connection's parameters as if +there were no PROXY protocol header on the wire. However, senders SHOULD not +use the "UNKNOWN" protocol when they are the initiators of outgoing connections +because some receivers may reject them. When a load balancing proxy has to send +health checks to a server, it SHOULD build a valid PROXY line which it will +fill with a getsockname()/getpeername() pair indicating the addresses used. It +is important to understand that doing so is not appropriate when some source +address translation is performed between the sender and the receiver. + +An example of such a line before an HTTP request would look like this (CR +marked as "\r" and LF marked as "\n") : + + PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n + GET / HTTP/1.1\r\n + Host: 192.168.0.11\r\n + \r\n + +For the sender, the header line is easy to put into the output buffers once the +connection is established. Note that since the line is always shorter than an +MSS, the sender is guaranteed to always be able to emit it at once and should +not even bother handling partial sends. For the receiver, once the header is +parsed, it is easy to skip it from the input buffers. Please consult section 9 +for implementation suggestions. + +2.2. Binary header format (version 2) + +Producing human-readable IPv6 addresses and parsing them is very inefficient, +due to the multiple possible representation formats and the handling of compact +address format. It was also not possible to specify address families outside +IPv4/IPv6 nor non-TCP protocols. Another drawback of the human-readable format +is the fact that implementations need to parse all characters to find the +trailing CRLF, which makes it harder to read only the exact bytes count. Last, +the UNKNOWN address type has not always been accepted by servers as a valid +protocol because of its imprecise meaning. + +Version 2 of the protocol thus introduces a new binary format which remains +distinguishable from version 1 and from other commonly used protocols. It was +specially designed in order to be incompatible with a wide range of protocols +and to be rejected by a number of common implementations of these protocols +when unexpectedly presented (please see section 7). Also for better processing +efficiency, IPv4 and IPv6 addresses are respectively aligned on 4 and 16 bytes +boundaries. + +The binary header format starts with a constant 12 bytes block containing the +protocol signature : + + \x0D \x0A \x0D \x0A \x00 \x0D \x0A \x51 \x55 \x49 \x54 \x0A + +Note that this block contains a null byte at the 5th position, so it must not +be handled as a null-terminated string. + +The next byte (the 13th one) is the protocol version and command. + +The highest four bits contains the version. As of this specification, it must +always be sent as \x2 and the receiver must only accept this value. + +The lowest four bits represents the command : + - \x0 : LOCAL : the connection was established on purpose by the proxy + without being relayed. The connection endpoints are the sender and the + receiver. Such connections exist when the proxy sends health-checks to the + server. The receiver must accept this connection as valid and must use the + real connection endpoints and discard the protocol block including the + family which is ignored. + + - \x1 : PROXY : the connection was established on behalf of another node, + and reflects the original connection endpoints. The receiver must then use + the information provided in the protocol block to get original the address. + + - other values are unassigned and must not be emitted by senders. Receivers + must drop connections presenting unexpected values here. + +The 14th byte contains the transport protocol and address family. The highest 4 +bits contain the address family, the lowest 4 bits contain the protocol. + +The address family maps to the original socket family without necessarily +matching the values internally used by the system. It may be one of : + + - 0x0 : AF_UNSPEC : the connection is forwarded for an unknown, unspecified + or unsupported protocol. The sender should use this family when sending + LOCAL commands or when dealing with unsupported protocol families. The + receiver is free to accept the connection anyway and use the real endpoint + addresses or to reject it. The receiver should ignore address information. + + - 0x1 : AF_INET : the forwarded connection uses the AF_INET address family + (IPv4). The addresses are exactly 4 bytes each in network byte order, + followed by transport protocol information (typically ports). + + - 0x2 : AF_INET6 : the forwarded connection uses the AF_INET6 address family + (IPv6). The addresses are exactly 16 bytes each in network byte order, + followed by transport protocol information (typically ports). + + - 0x3 : AF_UNIX : the forwarded connection uses the AF_UNIX address family + (UNIX). The addresses are exactly 108 bytes each. + + - other values are unspecified and must not be emitted in version 2 of this + protocol and must be rejected as invalid by receivers. + +The transport protocol is specified in the lowest 4 bits of the 14th byte : + + - 0x0 : UNSPEC : the connection is forwarded for an unknown, unspecified + or unsupported protocol. The sender should use this family when sending + LOCAL commands or when dealing with unsupported protocol families. The + receiver is free to accept the connection anyway and use the real endpoint + addresses or to reject it. The receiver should ignore address information. + + - 0x1 : STREAM : the forwarded connection uses a SOCK_STREAM protocol (eg: + TCP or UNIX_STREAM). When used with AF_INET/AF_INET6 (TCP), the addresses + are followed by the source and destination ports represented on 2 bytes + each in network byte order. + + - 0x2 : DGRAM : the forwarded connection uses a SOCK_DGRAM protocol (eg: + UDP or UNIX_DGRAM). When used with AF_INET/AF_INET6 (UDP), the addresses + are followed by the source and destination ports represented on 2 bytes + each in network byte order. + + - other values are unspecified and must not be emitted in version 2 of this + protocol and must be rejected as invalid by receivers. + +In practice, the following protocol bytes are expected : + + - \x00 : UNSPEC : the connection is forwarded for an unknown, unspecified + or unsupported protocol. The sender should use this family when sending + LOCAL commands or when dealing with unsupported protocol families. When + used with a LOCAL command, the receiver must accept the connection and + ignore any address information. For other commands, the receiver is free + to accept the connection anyway and use the real endpoints addresses or to + reject the connection. The receiver should ignore address information. + + - \x11 : TCP over IPv4 : the forwarded connection uses TCP over the AF_INET + protocol family. Address length is 2*4 + 2*2 = 12 bytes. + + - \x12 : UDP over IPv4 : the forwarded connection uses UDP over the AF_INET + protocol family. Address length is 2*4 + 2*2 = 12 bytes. + + - \x21 : TCP over IPv6 : the forwarded connection uses TCP over the AF_INET6 + protocol family. Address length is 2*16 + 2*2 = 36 bytes. + + - \x22 : UDP over IPv6 : the forwarded connection uses UDP over the AF_INET6 + protocol family. Address length is 2*16 + 2*2 = 36 bytes. + + - \x31 : UNIX stream : the forwarded connection uses SOCK_STREAM over the + AF_UNIX protocol family. Address length is 2*108 = 216 bytes. + + - \x32 : UNIX datagram : the forwarded connection uses SOCK_DGRAM over the + AF_UNIX protocol family. Address length is 2*108 = 216 bytes. + +Only the UNSPEC protocol byte (\x00) is mandatory to implement on the receiver. +A receiver is not required to implement other ones, provided that it +automatically falls back to the UNSPEC mode for the valid combinations above +that it does not support. + +The 15th and 16th bytes is the address length in bytes in network endian order. +It is used so that the receiver knows how many address bytes to skip even when +it does not implement the presented protocol. Thus the length of the protocol +header in bytes is always exactly 16 + this value. When a sender presents a +LOCAL connection, it should not present any address so it sets this field to +zero. Receivers MUST always consider this field to skip the appropriate number +of bytes and must not assume zero is presented for LOCAL connections. When a +receiver accepts an incoming connection showing an UNSPEC address family or +protocol, it may or may not decide to log the address information if present. + +So the 16-byte version 2 header can be described this way : + + struct proxy_hdr_v2 { + uint8_t sig[12]; /* hex 0D 0A 0D 0A 00 0D 0A 51 55 49 54 0A */ + uint8_t ver_cmd; /* protocol version and command */ + uint8_t fam; /* protocol family and address */ + uint16_t len; /* number of following bytes part of the header */ + }; + +Starting from the 17th byte, addresses are presented in network byte order. +The address order is always the same : + - source layer 3 address in network byte order + - destination layer 3 address in network byte order + - source layer 4 address if any, in network byte order (port) + - destination layer 4 address if any, in network byte order (port) + +The address block may directly be sent from or received into the following +union which makes it easy to cast from/to the relevant socket native structs +depending on the address type : + + union proxy_addr { + struct { /* for TCP/UDP over IPv4, len = 12 */ + uint32_t src_addr; + uint32_t dst_addr; + uint16_t src_port; + uint16_t dst_port; + } ipv4_addr; + struct { /* for TCP/UDP over IPv6, len = 36 */ + uint8_t src_addr[16]; + uint8_t dst_addr[16]; + uint16_t src_port; + uint16_t dst_port; + } ipv6_addr; + struct { /* for AF_UNIX sockets, len = 216 */ + uint8_t src_addr[108]; + uint8_t dst_addr[108]; + } unix_addr; + }; + +The sender must ensure that all the protocol header is sent at once. This block +is always smaller than an MSS, so there is no reason for it to be segmented at +the beginning of the connection. The receiver should also process the header +at once. The receiver must not start to parse an address before the whole +address block is received. The receiver must also reject incoming connections +containing partial protocol headers. + +A receiver may be configured to support both version 1 and version 2 of the +protocol. Identifying the protocol version is easy : + + - if the incoming byte count is 16 or above and the 13 first bytes match + the protocol signature block followed by the protocol version 2 : + + \x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A\x20 + + - otherwise, if the incoming byte count is 8 or above, and the 5 first + characters match the US-ASCII representation of "PROXY" then the protocol + must be parsed as version 1 : + + \x50\x52\x4F\x58\x59 + + - otherwise the protocol is not covered by this specification and the + connection must be dropped. + +If the length specified in the PROXY protocol header indicates that additional +bytes are part of the header beyond the address information, a receiver may +choose to skip over and ignore those bytes, or attempt to interpret those +bytes. + +The information in those bytes will be arranged in Type-Length-Value (TLV +vectors) in the following format. The first byte is the Type of the vector. +The second two bytes represent the length in bytes of the value (not included +the Type and Length bytes), and following the length field is the number of +bytes specified by the length. + + struct pp2_tlv { + uint8_t type; + uint8_t length_hi; + uint8_t length_lo; + uint8_t value[0]; + }; + +A receiver may choose to skip over and ignore the TLVs it is not interested in +or it does not understand. Senders can generate the TLVs only for +the information they choose to publish. + +The following types have already been registered for the field : + + #define PP2_TYPE_ALPN 0x01 + #define PP2_TYPE_AUTHORITY 0x02 + #define PP2_TYPE_CRC32C 0x03 + #define PP2_TYPE_NOOP 0x04 + #define PP2_TYPE_UNIQUE_ID 0x05 + #define PP2_TYPE_SSL 0x20 + #define PP2_SUBTYPE_SSL_VERSION 0x21 + #define PP2_SUBTYPE_SSL_CN 0x22 + #define PP2_SUBTYPE_SSL_CIPHER 0x23 + #define PP2_SUBTYPE_SSL_SIG_ALG 0x24 + #define PP2_SUBTYPE_SSL_KEY_ALG 0x25 + #define PP2_SUBTYPE_SSL_GROUP 0x26 + #define PP2_SUBTYPE_SSL_SIG_SCHEME 0x27 + #define PP2_SUBTYPE_SSL_CLIENT_CERT 0x28 + #define PP2_TYPE_NETNS 0x30 + +2.2.1 PP2_TYPE_ALPN + +Application-Layer Protocol Negotiation (ALPN). It is a byte sequence defining +the upper layer protocol in use over the connection. The most common use case +will be to pass the exact copy of the ALPN extension of the Transport Layer +Security (TLS) protocol as defined by RFC7301 [9]. + +2.2.2 PP2_TYPE_AUTHORITY + +Contains the host name value passed by the client, as an UTF8-encoded string. +In case of TLS being used on the client connection, this is the exact copy of +the "server_name" extension as defined by RFC3546 [10], section 3.1, often +referred to as "SNI". There are probably other situations where an authority +can be mentioned on a connection without TLS being involved at all. + +2.2.3. PP2_TYPE_CRC32C + +The value of the type PP2_TYPE_CRC32C is a 32-bit number storing the CRC32c +checksum of the PROXY protocol header. + +When the checksum is supported by the sender after constructing the header +the sender MUST: + + - initialize the checksum field to '0's. + + - calculate the CRC32c checksum of the PROXY header as described in RFC4960, + Appendix B [8]. + + - put the resultant value into the checksum field, and leave the rest of + the bits unchanged. + +If the checksum is provided as part of the PROXY header and the checksum +functionality is supported by the receiver, the receiver MUST: + + - store the received CRC32c checksum value aside. + + - replace the 32 bits of the checksum field in the received PROXY header with + all '0's and calculate a CRC32c checksum value of the whole PROXY header. + + - verify that the calculated CRC32c checksum is the same as the received + CRC32c checksum. If it is not, the receiver MUST treat the TCP connection + providing the header as invalid. + +The default procedure for handling an invalid TCP connection is to abort it. + +2.2.4. PP2_TYPE_NOOP + +The TLV of this type should be ignored when parsed. The value is zero or more +bytes. Can be used for data padding or alignment. Note that it can be used +to align only by 3 or more bytes because a TLV can not be smaller than that. + +2.2.5. PP2_TYPE_UNIQUE_ID + +The value of the type PP2_TYPE_UNIQUE_ID is an opaque byte sequence of up to +128 bytes generated by the upstream proxy that uniquely identifies the +connection. + +The unique ID can be used to easily correlate connections across multiple +layers of proxies, without needing to look up IP addresses and port numbers. + +2.2.6. The PP2_TYPE_SSL type and subtypes + +For the type PP2_TYPE_SSL, the value is itself a defined like this : + + struct pp2_tlv_ssl { + uint8_t client; + uint32_t verify; + struct pp2_tlv sub_tlv[0]; + } __attribute__((packed)); + +Note the "packed" attribute which indicates that each field starts immediately +after the previous one (i.e. without type-specific alignment nor padding). + +The field will be zero if the client presented a certificate +and it was successfully verified, and non-zero otherwise. + +The field is made of a bit field from the following values, +indicating which element is present : + + #define PP2_CLIENT_SSL 0x01 + #define PP2_CLIENT_CERT_CONN 0x02 + #define PP2_CLIENT_CERT_SESS 0x04 + +Note, that each of these elements may lead to extra data being appended to +this TLV using a second level of TLV encapsulation. It is thus possible to +find multiple TLV values after this field. The total length of the pp2_tlv_ssl +TLV will reflect this. + +The PP2_CLIENT_SSL flag indicates that the client connected over SSL/TLS. When +this field is present, the US-ASCII string representation of the TLS version is +appended at the end of the field in the TLV format using the type +PP2_SUBTYPE_SSL_VERSION. + +PP2_CLIENT_CERT_CONN indicates that the client provided a certificate over the +current connection. PP2_CLIENT_CERT_SESS indicates that the client provided a +certificate at least once over the TLS session this connection belongs to. + +The second level TLV PP2_SUBTYPE_SSL_CIPHER provides the US-ASCII string name +of the used cipher, for example "ECDHE-RSA-AES128-GCM-SHA256". + +The second level TLV PP2_SUBTYPE_SSL_SIG_ALG provides the US-ASCII string name +of the algorithm used to sign the certificate presented by the frontend when +the incoming connection was made over an SSL/TLS transport layer, for example +"RSA-SHA256". + +The second level TLV PP2_SUBTYPE_SSL_KEY_ALG provides the US-ASCII string name +of the algorithm used to generate the key of the certificate presented by the +frontend when the incoming connection was made over an SSL/TLS transport layer, +for example "RSA2048". + +The second level TLV PP2_SUBTYPE_SSL_GROUP provides the US-ASCII string name of +the key exchange algorithm used for the frontend TLS connection, for example +"secp256r1". + +The second level TLV PP2_SUBTYPE_SSL_SIG_SCHEME provides the US-ASCII string +name of the algorithm the frontend used to sign the ServerKeyExchange or +CertificateVerify message, for example "rsa_pss_rsae_sha256". + +The optional second level TLV PP2_SUBTYPE_SSL_CLIENT_CERT provides the raw +X.509 client certificate encoded in ASN.1 DER. The frontend may choose to omit +this TLV depending on configuration. + +In all cases, the string representation (in UTF8) of the Common Name field +(OID: 2.5.4.3) of the client certificate's Distinguished Name, is appended +using the TLV format and the type PP2_SUBTYPE_SSL_CN. E.g. "example.com". + +2.2.7. The PP2_TYPE_NETNS type + +The type PP2_TYPE_NETNS defines the value as the US-ASCII string representation +of the namespace's name. + +2.2.8. Reserved type ranges + +The following range of 16 type values is reserved for application-specific +data and will be never used by the PROXY Protocol. If you need more values +consider extending the range with a type field in your TLVs. + + #define PP2_TYPE_MIN_CUSTOM 0xE0 + #define PP2_TYPE_MAX_CUSTOM 0xEF + +This range of 8 values is reserved for temporary experimental use by +application developers and protocol designers. The values from the range will +never be used by the PROXY protocol and should not be used by production +functionality. + + #define PP2_TYPE_MIN_EXPERIMENT 0xF0 + #define PP2_TYPE_MAX_EXPERIMENT 0xF7 + +The following range of 8 values is reserved for future use, potentially to +extend the protocol with multibyte type values. + + #define PP2_TYPE_MIN_FUTURE 0xF8 + #define PP2_TYPE_MAX_FUTURE 0xFF + +3. Implementations + +HAProxy 1.5 implements version 1 of the PROXY protocol on both sides : + - the listening sockets accept the protocol when the "accept-proxy" setting + is passed to the "bind" keyword. Connections accepted on such listeners + will behave just as if the source really was the one advertised in the + protocol. This is true for logging, ACLs, content filtering, transparent + proxying, etc... + + - the protocol may be used to connect to servers if the "send-proxy" setting + is present on the "server" line. It is enabled on a per-server basis, so it + is possible to have it enabled for remote servers only and still have local + ones behave differently. If the incoming connection was accepted with the + "accept-proxy", then the relayed information is the one advertised in this + connection's PROXY line. + + - HAProxy 1.5 also implements version 2 of the PROXY protocol as a sender. In + addition, a TLV with limited, optional, SSL information has been added. + +Stunnel added support for version 1 of the protocol for outgoing connections in +version 4.45. + +Stud added support for version 1 of the protocol for outgoing connections on +2011/06/29. + +Postfix added support for version 1 of the protocol for incoming connections +in smtpd and postscreen in version 2.10. + +A patch is available for Stud[5] to implement version 1 of the protocol on +incoming connections. + +Support for versions 1 and 2 of the protocol was added to Varnish 4.1 [6]. + +Exim added support for version 1 and version 2 of the protocol for incoming +connections on 2014/05/13, and will be released as part of version 4.83. + +Squid added support for versions 1 and 2 of the protocol in version 3.5 [7]. + +Jetty 9.3.0 supports protocol version 1. + +lighttpd added support for versions 1 and 2 of the protocol for incoming +connections in version 1.4.46 [11]. + +The protocol is simple enough that it is expected that other implementations +will appear, especially in environments such as SMTP, IMAP, FTP, RDP where the +client's address is an important piece of information for the server and some +intermediaries. In fact, several proprietary deployments have already done so +on FTP and SMTP servers. + +Proxy developers are encouraged to implement this protocol, because it will +make their products much more transparent in complex infrastructures, and will +get rid of a number of issues related to logging and access control. + +4. Architectural benefits +4.1. Multiple layers + +Using the PROXY protocol instead of transparent proxy provides several benefits +in multiple-layer infrastructures. The first immediate benefit is that it +becomes possible to chain multiple layers of proxies and always present the +original IP address. for instance, let's consider the following 2-layer proxy +architecture : + + Internet + ,---. | client to PX1: + ( X ) | native protocol + `---' | + | V + +--+--+ +-----+ + | FW1 |------| PX1 | + +--+--+ +-----+ | PX1 to PX2: PROXY + native + | V + +--+--+ +-----+ + | FW2 |------| PX2 | + +--+--+ +-----+ | PX2 to SRV: PROXY + native + | V + +--+--+ + | SRV | + +-----+ + +Firewall FW1 receives traffic from internet-based clients and forwards it to +reverse-proxy PX1. PX1 adds a PROXY header then forwards to PX2 via FW2. PX2 +is configured to read the PROXY header and to emit it on output. It then joins +the origin server SRV and presents the original client's address there. Since +all TCP connections endpoints are real machines and are not spoofed, there is +no issue for the return traffic to pass via the firewalls and reverse proxies. +Using transparent proxy, this would be quite difficult because the firewalls +would have to deal with the client's address coming from the proxies in the DMZ +and would have to correctly route the return traffic there instead of using the +default route. + +4.2. IPv4 and IPv6 integration + +The protocol also eases IPv4 and IPv6 integration : if only the first layer +(FW1 and PX1) is IPv6-capable, it is still possible to present the original +client's IPv6 address to the target server even though the whole chain is only +connected via IPv4. + +4.3. Multiple return paths + +When transparent proxy is used, it is not possible to run multiple proxies +because the return traffic would follow the default route instead of finding +the proper proxy. Some tricks are sometimes possible using multiple server +addresses and policy routing but these are very limited. + +Using the PROXY protocol, this problem disappears as the servers don't need +to route to the client, just to the proxy that forwarded the connection. So +it is perfectly possible to run a proxy farm in front of a very large server +farm and have it working effortless, even when dealing with multiple sites. + +This is particularly important in Cloud-like environments where there is little +choice of binding to random addresses and where the lower processing power per +node generally requires multiple front nodes. + +The example below illustrates the following case : virtualized infrastructures +are deployed in 3 datacenters (DC1..DC3). Each DC uses its own VIP which is +handled by the hosting provider's layer 3 load balancer. This load balancer +routes the traffic to a farm of layer 7 SSL/cache offloaders which load balance +among their local servers. The VIPs are advertised by geolocalised DNS so that +clients generally stick to a given DC. Since clients are not guaranteed to +stick to one DC, the L7 load balancing proxies have to know the other DCs' +servers that may be reached via the hosting provider's LAN or via the internet. +The L7 proxies use the PROXY protocol to join the servers behind them, so that +even inter-DC traffic can forward the original client's address and the return +path is unambiguous. This would not be possible using transparent proxy because +most often the L7 proxies would not be able to spoof an address, and this would +never work between datacenters. + + Internet + + DC1 DC2 DC3 + ,---. ,---. ,---. + ( X ) ( X ) ( X ) + `---' `---' `---' + | +-------+ | +-------+ | +-------+ + +----| L3 LB | +----| L3 LB | +----| L3 LB | + | +-------+ | +-------+ | +-------+ + ------+------- ~ ~ ~ ------+------- ~ ~ ~ ------+------- + ||||| |||| ||||| |||| ||||| |||| + 50 SRV 4 PX 50 SRV 4 PX 50 SRV 4 PX + +5. Security considerations + +Version 1 of the protocol header (the human-readable format) was designed so as +to be distinguishable from HTTP. It will not parse as a valid HTTP request and +an HTTP request will not parse as a valid proxy request. Version 2 add to use a +non-parsable binary signature to make many products fail on this block. The +signature was designed to cause immediate failure on HTTP, SSL/TLS, SMTP, FTP, +and POP. It also causes aborts on LDAP and RDP servers (see section 6). That +makes it easier to enforce its use under certain connections and at the same +time, it ensures that improperly configured servers are quickly detected. + +Implementers should be very careful about not trying to automatically detect +whether they have to decode the header or not, but rather they must only rely +on a configuration parameter. Indeed, if the opportunity is left to a normal +client to use the protocol, it will be able to hide its activities or make them +appear as coming from somewhere else. However, accepting the header only from a +number of known sources should be safe. + +6. Validation + +The version 2 protocol signature has been sent to a wide variety of protocols +and implementations including old ones. The following protocol and products +have been tested to ensure the best possible behavior when the signature was +presented, even with minimal implementations : + + - HTTP : + - Apache 1.3.33 : connection abort => pass/optimal + - Nginx 0.7.69 : 400 Bad Request + abort => pass/optimal + - lighttpd 1.4.20 : 400 Bad Request + abort => pass/optimal + - thttpd 2.20c : 400 Bad Request + abort => pass/optimal + - mini-httpd-1.19 : 400 Bad Request + abort => pass/optimal + - haproxy 1.4.21 : 400 Bad Request + abort => pass/optimal + - Squid 3 : 400 Bad Request + abort => pass/optimal + - SSL : + - stud 0.3.47 : connection abort => pass/optimal + - stunnel 4.45 : connection abort => pass/optimal + - nginx 0.7.69 : 400 Bad Request + abort => pass/optimal + - FTP : + - Pure-ftpd 1.0.20 : 3*500 then 221 Goodbye => pass/optimal + - vsftpd 2.0.1 : 3*530 then 221 Goodbye => pass/optimal + - SMTP : + - postfix 2.3 : 3*500 + 221 Bye => pass/optimal + - exim 4.69 : 554 + connection abort => pass/optimal + - POP : + - dovecot 1.0.10 : 3*ERR + Logout => pass/optimal + - IMAP : + - dovecot 1.0.10 : 5*ERR + hang => pass/non-optimal + - LDAP : + - openldap 2.3 : abort => pass/optimal + - SSH : + - openssh 3.9p1 : abort => pass/optimal + - RDP : + - Windows XP SP3 : abort => pass/optimal + +This means that most protocols and implementations will not be confused by an +incoming connection exhibiting the protocol signature, which avoids issues when +facing misconfigurations. + +7. Future developments + +It is possible that the protocol may slightly evolve to present other +information such as the incoming network interface, or the origin addresses in +case of network address translation happening before the first proxy, but this +is not identified as a requirement right now. Some deep thinking has been spent +on this and it appears that trying to add a few more information open a Pandora +box with many information from MAC addresses to SSL client certificates, which +would make the protocol much more complex. So at this point it is not planned. +Suggestions on improvements are welcome. + +8. Contacts and links + +Please use w@1wt.eu to send any comments to the author. + +The following links were referenced in the document. + +[1] http://www.postfix.org/XCLIENT_README.html +[2] http://tools.ietf.org/html/rfc7239 +[3] http://www.stunnel.org/ +[4] https://github.com/bumptech/stud +[5] https://github.com/bumptech/stud/pull/81 +[6] https://www.varnish-cache.org/docs/trunk/phk/ssl_again.html +[7] http://wiki.squid-cache.org/Squid-3.5 +[8] https://tools.ietf.org/html/rfc4960#appendix-B +[9] https://tools.ietf.org/rfc/rfc7301.txt +[10] https://www.ietf.org/rfc/rfc3546.txt +[11] https://redmine.lighttpd.net/issues/2804 + +9. Sample code + +The code below is an example of how a receiver may deal with both versions of +the protocol header for TCP over IPv4 or IPv6. The function is supposed to be +called upon a read event. Addresses may be directly copied into their final +memory location since they're transported in network byte order. The sending +side is even simpler and can easily be deduced from this sample code. + + struct sockaddr_storage from; /* already filled by accept() */ + struct sockaddr_storage to; /* already filled by getsockname() */ + const char v2sig[12] = "\x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A"; + + /* returns 0 if needs to poll, <0 upon error or >0 if it did the job */ + int read_evt(int fd) + { + union { + struct { + char line[108]; + } v1; + struct { + uint8_t sig[12]; + uint8_t ver_cmd; + uint8_t fam; + uint16_t len; + union { + struct { /* for TCP/UDP over IPv4, len = 12 */ + uint32_t src_addr; + uint32_t dst_addr; + uint16_t src_port; + uint16_t dst_port; + } ip4; + struct { /* for TCP/UDP over IPv6, len = 36 */ + uint8_t src_addr[16]; + uint8_t dst_addr[16]; + uint16_t src_port; + uint16_t dst_port; + } ip6; + struct { /* for AF_UNIX sockets, len = 216 */ + uint8_t src_addr[108]; + uint8_t dst_addr[108]; + } unx; + } addr; + } v2; + } hdr; + + int size, ret; + + do { + ret = recv(fd, &hdr, sizeof(hdr), MSG_PEEK); + } while (ret == -1 && errno == EINTR); + + if (ret == -1) + return (errno == EAGAIN) ? 0 : -1; + + if (ret >= 16 && memcmp(&hdr.v2, v2sig, 12) == 0 && + (hdr.v2.ver_cmd & 0xF0) == 0x20) { + size = 16 + ntohs(hdr.v2.len); + if (ret < size) + return -1; /* truncated or too large header */ + + switch (hdr.v2.ver_cmd & 0xF) { + case 0x01: /* PROXY command */ + switch (hdr.v2.fam) { + case 0x11: /* TCPv4 */ + ((struct sockaddr_in *)&from)->sin_family = AF_INET; + ((struct sockaddr_in *)&from)->sin_addr.s_addr = + hdr.v2.addr.ip4.src_addr; + ((struct sockaddr_in *)&from)->sin_port = + hdr.v2.addr.ip4.src_port; + ((struct sockaddr_in *)&to)->sin_family = AF_INET; + ((struct sockaddr_in *)&to)->sin_addr.s_addr = + hdr.v2.addr.ip4.dst_addr; + ((struct sockaddr_in *)&to)->sin_port = + hdr.v2.addr.ip4.dst_port; + goto done; + case 0x21: /* TCPv6 */ + ((struct sockaddr_in6 *)&from)->sin6_family = AF_INET6; + memcpy(&((struct sockaddr_in6 *)&from)->sin6_addr, + hdr.v2.addr.ip6.src_addr, 16); + ((struct sockaddr_in6 *)&from)->sin6_port = + hdr.v2.addr.ip6.src_port; + ((struct sockaddr_in6 *)&to)->sin6_family = AF_INET6; + memcpy(&((struct sockaddr_in6 *)&to)->sin6_addr, + hdr.v2.addr.ip6.dst_addr, 16); + ((struct sockaddr_in6 *)&to)->sin6_port = + hdr.v2.addr.ip6.dst_port; + goto done; + } + /* unsupported protocol, keep local connection address */ + break; + case 0x00: /* LOCAL command */ + /* keep local connection address for LOCAL */ + break; + default: + return -1; /* not a supported command */ + } + } + else if (ret >= 8 && memcmp(hdr.v1.line, "PROXY", 5) == 0) { + char *end = memchr(hdr.v1.line, '\r', ret - 1); + if (!end || end[1] != '\n') + return -1; /* partial or invalid header */ + *end = '\0'; /* terminate the string to ease parsing */ + size = end + 2 - hdr.v1.line; /* skip header + CRLF */ + /* parse the V1 header using favorite address parsers like inet_pton. + * return -1 upon error, or simply fall through to accept. + */ + } + else { + /* Wrong protocol */ + return -1; + } + + done: + /* we need to consume the appropriate amount of data from the socket */ + do { + ret = recv(fd, &hdr, size, 0); + } while (ret == -1 && errno == EINTR); + return (ret >= 0) ? 1 : -1; + } diff --git a/docs/standards/pgpverify/FORMAT b/docs/standards/pgpverify/FORMAT new file mode 100644 index 00000000..bc80c9be --- /dev/null +++ b/docs/standards/pgpverify/FORMAT @@ -0,0 +1,251 @@ +From: tale@isc.org (David C Lawrence) + +This file describes the format of the input to PGP for both signing a +control message and subsequently verifying it. It is meant for people +who are implementing their own signing/verifying code because they +cannot (or for some reason do not want to) use signcontrol/pgpverify. + +Please forgive the rough format of this documentation. It is a +combination of two messages I wrote in reply to email queries and I +have not found the time to refine it properly. + + ++++++++++++++++++++++++ + SIGNING CONTROL MESSAGES + ++++++++++++++++++++++++ + +The input for PGP to sign when generating a control message looks +something like this: + +================ cut here +X-Signed-Headers: From,Subject,Control,Message-ID,Date,From,Sender +Subject: cmsg newgroup GROUP +Control: newgroup GROUP +Message-ID: +Date: DATE +From: ADDRESS +Sender: ADDRESS + +BODY TEXT +================ cut here + +Capital letters above are all variable text. There is a very special +reason each element is signed: + + * Subject: some very old servers use this header to implement control + messages (based on a "cmsg" keyword, ie, "cmsg newgroup GROUP"). + * Control: the vast majority of servers figure out what to do based on this. + * Message-ID: used by servers to avoid accepting articles it already took. + * Date: also used by servers to avoid accepting articles it already took. + * From: used to identify who sent the message, which is used by some + servers to decide whether to process the control message. + * Sender: used like From: when present + * BODY TEXT: newsgroups file information is taken from the body. + +If group is moderated, the above Subject and Control headers would be +appended with the string " moderated". + +The X-Signed-Headers pseudoheader will not be in the final output, but +rather have its value put into X-PGP-Sig. Obviously, it specifies all +of the headers that are signed. More importantly, it also specifies +the order in which they appear in the signed data (even if they are +rearranged as they propagate around Usenet), and the character case +used in each signed header (because some gateways will unfortunately +change header case, while preserving all of the other information just +fine). + +Note that Message-ID and Date have some strict rules about how they +can be formatted, or else news servers will not even accept the article. +These rules can be found in RFC 822. + +For Message-ID, I simply use: + + Message-ID: ... eg ... + Message-ID: <907576376.25529@isc.org> + +where TIME is the an integer number of seconds since a fixed point in +history (on Unix machines, Jan 1, 1970), PROCESSID is the process +number on my Unix machine, and HOST is my domain. The important part +is that each message-id be UNIQUE ... whatever method you can use to +ensure a unique lefthand part is fine. + +For Date, I simply use: + + Date: Day, DD Mon YYYY HH:MM:SS ZONE ... eg ... + Date: Mon, 05 Oct 1998 08:32:56 -0000 + +My control messages do not have sender headers, so the Sender: header +goes into the file to be signed but without any value for the address; +that is, "Sender: " followed by the end of line, and the single space +after the colon is necessary. _Every_ header must have a space +after the colon. + +The signcontrol script also does some other things to its input. +First, it ensures that the group name meets Usenet standards: + + # "component MUST contain at least one letter" + # "[component] MUST not contain uppercase letters" + # "[component] MUST begin with a letter or digit" + # "[component] MUST not be longer than 14 characters" + # "sequences 'all' and 'ctl' MUST not be used as components" + # "first component MUST begin with a letter" + # and enforcing "subsequent components SHOULD begin with a letter" as MUST + # and enforcing at least a 2nd level group + +The 14 character limit is expected to be lifted in the next revision +of the standard for Usenet articles, but it still exists. + +In the body text, signcontrol insists that newgroups have a first line +that reads "GROUP is a(n) (un)moderated newsgroup", where GROUP must +match what is in the Control header and "an unmoderated" or "a moderated" +is also checked against the presence of the " moderated" keyword in +the Subject/Control headers. + +Signcontrol also ensures that a "For your newsgroups file:" line is in +the body, followed immediately by a properly formatted newsgroups file +line. + +Here's what I try to keep for group descriptions: + +group.namedescription.[ (Moderated)] + +There should be one or more hard tabs (assume 8 column tabstops) to +get to a description; if the group.name is more than 24 characters, +use just one tab. The description should start with a capital and end +in a period and not be more than 56 characters (80 - 24) long. If the +group is moderated, it should have " (Moderated)" following the +period, not counted as part of the length of the description. The +goal is to keep the total line under 80 columns, so if the group name +is 25 characters long you'd have eight less description characters to +work with. + +Some over-long descriptions could be made to easily fit the length by +dropping "puff" phrases like "Discussion of" which don't meaningfully +contribute to the description. Others are usually pretty easy to get +to no more than column eighty, except when the group names start +getting really long. Hopefully then the group name itself contains +quite a bit of description. + +The signcontrol script also used to ensure the format of lines which +specified submission and contact addresses for moderated newsgroups. +I no longer recommend including them in control messages, because some +sites were erroneously trying to use them to maintain their own +complete set of submission aliases, and getting rapidly, hopelessly +out of sync with the real master forwarding list. + +Now, that covers the format of the article to be signed. signcontrol +simply invokes pgp with the options "-fast -u SIGNER +verbose=0" and +then parses the output. It generates an X-PGP-Sig header that starts +with the pgp version number from the signed output, then has the +signed headers separated by commas (no spaces), and finally has the +signature. Thus: + +X-PGP-Sig: 2.6.2 Subject,Control,Message-ID,Date,From,Sender + iQCVAwUBNhiHnsJdOtO4janBAQGFFwP/SCsq1Isgw8DXHDaRkr0cdkZidVH41N6d + IO+AuE4aRUUQN8Eym5bBzpdRnODVKQjp/npubNowSqv93IDlf+veoemc8yB9QTX9 + PUFbRXn3r5DCJAOsH6M4oHbU6PCMKZRPZZENglcLhsYl2fM55l3Bhnxpu/GVOuDX + qKVwKgi+srA= + +There is another way to do this, rather than parsing the output of +pgp, which you might find easier to implement. pgp generates +"detached" signatures if you use the -b flag, which cuts down on some +parsing. + +Now, to generate the final article, suitable for distribution around +the network without any further modification, you just need to add +these headers to the original set you signed: + * X-PGP-Sig as above + * "Path: bounce-back" (the "bounce-back" phrase is to combat very + old mail agents which try to reply to the Path header; it is not + important for the purposes of verifying the control message) + * "Approved: YOUR-ADDRESS" + * "Newsgroups: GROUP" + +For that last, note that the control message should be posted only to +the group you are trying to create or remove. This is the right thing +to do with regard to article propagation issues. + +signcontrol also adds an X-Info header to direct people to more +information about pgp signed messages. You can also use it to direct +people to more information about the hierarchy or group. + +The best way to transfer the final article is to avoid "inews" or +NNTP'S "POST" command or anyting designed for the usual case of a user +posting a new article. You have a fully formed article which can be +injected directly into the news system by using "rnews", NNTP's +"ihave", or what have you. When doing so you needn't worry about any +of your signed headers being altered or any other validity checks that +the news system might attempt to perform but which would be incorrect +for your message. + + ++++++++++++++++++++++++++ + VERIFYING CONTROL MESSAGES + ++++++++++++++++++++++++++ + +To verify a signed control message, you need to recreate the original +output that resulted when PGP signed the input file created above. +That output looks something like this: + +================================ cut here +-----BEGIN PGP SIGNED MESSAGE----- + +X-Signed-Headers: From,Subject,Control,Foo,Bar,Blech +From: from-content +Subject: subject-content +Control: control-content +Foo: foo-content +Bar: bar-content +Blech: blech-content + +Message Body + +-----BEGIN PGP SIGNATURE----- +Version: version + +signature +-----BEGIN PGP SIGNATURE----- +================================ cut here + +Empty lines (a line end followed immediately by another line end) in +the above are significant. + +Using this header, from a message I signed a long time ago: + +X-PGP-Sig: 2.6.2 Subject,Control,Message-ID,Date,From,Sender + iQCVAwUBM8QJNsJdOtO4janBAQGkUAP6AlzO065jDQFrG20/b3/SaOm4WGQBly5D + pXlVJdYBqPAG3HvxVqAdKM7y6ixM7Mml4OdfK0JeVCH03nqeGuBc51sTDIZ6kyAx + +YHlNSnp/JJnpDuJCfXZjwNl4kWImucGgwI5BxrQco8re949Cg5m5TFXiwYMiR/+ + AjKZCTtmV1Y= + +You substitute: + * "2.6.2" into the Version field + * "Subject,Control,Message-ID,Date,From,Sender" + where I have "From,Subject,Control,Foo,Bar,Blech" above + * the remaining gibberish, without the leading whitespace + on each line, for "signature". + +It is important that the headers you write into the temporary file are +in the very same order and with the same case as listed in the +X-PGP-Sig header, and that any header which is not included in the +actual article still be in the file to verify. For example, none of +my control messages have a Sender header, but the file that gets +verified still has a "Sender: ", followed by the end of line, in the +order determined by X-PGP-Sig. Note that the presence of a space +after the colon is necessary even for an empty header. + +In the body, you have to "ASCII armor" (PGP's term) text lines that +begin with a dash character. Any line starting with "-" must be +changed to "- -". + +There is a way to do this with detached signatures that I heard about +after implementing pgpverify that I understand does not require doing +the ASCII armoring. Using this method, the signed message (the part +between the "BEGIN PGP SIGNED MESSAGE" and "BEGIN PGP SIGNATURE" +above) are placed in one file, and the signature ("BEGIN PGP +SIGNATURE" through "END PGP SIGNATURE") in a separate file which has +the same name as the first but with a ".asc" appended. Since I was +already satisfied with pgpverify program working the way it was +written, I never bothered to explore using detached signatures. You +might find it easier to do it that way, though. + +You can use ftp://ftp.isc.org/pub/pgpcontrol/sample.control to test +your verifying program. diff --git a/docs/standards/pgpverify/README.md b/docs/standards/pgpverify/README.md new file mode 100644 index 00000000..cb65201c --- /dev/null +++ b/docs/standards/pgpverify/README.md @@ -0,0 +1,62 @@ +# PGPVERIFY (de-facto control-message authentication) + +PGPVERIFY is a **de-facto Netnews control-message authentication convention**. +It is **not** standardized by RFC 5537. + +RFC 5537 §5.1 states that there is no standardized means of authenticating +control-message senders, and cites `[PGPVERIFY]` as an unstandardized +mechanism in common use. VectorNNTP implements PGPVERIFY so CANCEL articles +posted by `NNTPCancelMessage` can be verified by INN `pgpverify` and +other Netnews infrastructure that already understands this convention. + +Do not describe PGPVERIFY as RFC-compliant. The CANCEL article itself follows +RFC 5536 / RFC 5537. PGPVERIFY is an interoperability extension on top of that +article. + +This is **not** PGP/MIME (RFC 3156), S/MIME, or a VectorNNTP-specific header. + +## Local copies + +| File | Role | Retrieved from | +|------|------|----------------| +| [`FORMAT`](FORMAT) | Signing and verification input (David Lawrence, “Signing Control Messages”) | https://www.eyrie.org/~eagle/usefor/other/pgpverify (Usefor copy of `ftp://ftp.isc.org/pub/pgpcontrol/FORMAT`, the URL cited by RFC 5537) | +| ISC operator README (HTML not vendored here) | “Authentication of Usenet Group Changes” — use the upstream URL; the large `README.html` copy was omitted from this migration | https://ftp.isc.org/pub/pgpcontrol/README.html (also cited by RFC 5536) | + +The original ISC FTP URL `ftp://ftp.isc.org/pub/pgpcontrol/FORMAT` returned HTTP 500 at retrieval time. The Usefor copy is the same David Lawrence (`tale@isc.org`) document RFC 5537 points at. + +## FORMAT vs deployed INN vs VectorNNTP + +PGPVERIFY FORMAT and INN `pgpverify` detached verification diverge on trailing +horizontal whitespace, including the empty signed Sender header. FORMAT is not +amended by INN. VectorNNTP follows INN so generated CANCEL controls verify on +deployed servers. + +| Source | Empty signed Sender | Trailing whitespace | +|--------|---------------------|---------------------| +| Historical/documented FORMAT | `Sender: \n` (colon + space required, including empty headers) | FORMAT does not strip SP/HT before LF | +| Deployed INN `pgpverify` 1.23–1.31 | Reconstructs `Sender: \n`, then hashes `Sender:\n` | `$message =~ s/[ \t]+\n/\n/g` after reconstruction (INN documents this as compatibility with historical attached signatures) | +| VectorNNTP | `Sender:\n` in the hashed bytes; Sender omitted on the wire (INN `parse_header` rejects empty on-wire values) | Same SP/HT-immediately-before-LF rule as INN detached verification | + +## VectorNNTP mapping + +| FORMAT rule | VectorNNTP CANCEL | +|-------------|-------------------| +| Signature header | `X-PGP-Sig` (not `X-PGP-Signature`, not PGP/MIME) | +| First tokens | ` ` | +| Signature body | tab-folded radix64 from an ASCII-armored detached OpenPGP signature | +| Signed headers (X-PGP-Sig example list) | `Subject,Control,Message-ID,Date,From,Sender` | +| Signed data prefix | `X-Signed-Headers: ` + that list (pseudoheader; not on the wire) | +| Header syntax | FORMAT construction is `Name: ` (colon + space), including missing headers; INN then strips trailing SP/HT before LF, so empty Sender hashes as `Sender:\n` | +| Sender | empty signed header present in canonical data as `Sender:\n`; omitted from the article (INN `parse_header` rejects empty on-wire values) | +| Body | signed; CRLF normalized to LF; trailing SP/HT immediately before LF stripped (INN rule) | +| Line endings in signed data | Unix LF (not `Environment.NewLine`) | +| Encoding | UTF-8 bytes of that LF text | +| Added after signing | `Newsgroups` (FORMAT), then server Path / Injection-Date / Injection-Info / X-Trace | +| OpenPGP | detached binary-document signature, SHA-256, RSA; package `BouncyCastle.Cryptography` 2.7.0 | + +## SHA-256 + +| File | SHA-256 | +|------|---------| +| `FORMAT` | `10556ca2d8c3b53918ae60b2baa7061596c0f9e4c9979f0c51560ac20dfb69ca` | +| `README.html` | `3d6dbdccda36df4d930a1ed4db13842e6025cde632d487ad637773b44d87b629` | diff --git a/docs/standards/spamd/PROTOCOL.md b/docs/standards/spamd/PROTOCOL.md new file mode 100644 index 00000000..f4693b19 --- /dev/null +++ b/docs/standards/spamd/PROTOCOL.md @@ -0,0 +1,210 @@ +> **VectorNNTP provenance.** This file is a bundled protocol reference for the +> SPAMC/SPAMD integration. The specification is from Apache SpamAssassin. +> Source: https://github.com/apache/spamassassin/blob/trunk/spamd/PROTOCOL +> +> The protocol specification below is the upstream document. It is not rewritten +> or reinterpreted. + +The SpamAssassin Network Protocol +================================= + +(This document is perpetually in draft status) + +Synopsis +-------- + +The protocol for communication between spamc/spamd is somewhat HTTP-like. The +conversation looks like: + + spamc --> PROCESS SPAMC/1.2\r\n + spamc --> Content-length: \r\n + (optional) spamc --> User: \r\n + spamc --> [optional \r\n-delimited headers...] + spamc --> \r\n [blank line] + spamc --> --message sent here-- + + spamd --> SPAMD/1.1 0 EX_OK\r\n + spamd --> Content-length: \r\n + spamc --> [optional \r\n-delimited headers...] + spamd --> \r\n [blank line] + spamd --> --processed message sent here-- + +After each side is done writing, it shuts down its side of the connection. + +The first line from spamc is the command for spamd to execute (PROCESS a +message is the command in protocol<=1.5) followed by the protocol version. + +There may be additional headers following the command, which are as yet +undefined. Servers should ignore these, and keep looking for headers which +they do support, or the "\r\n\r\n" end-of-headers marker. + +The first line of the response from spamd is the protocol version (note this is +SPAMD here, where it was SPAMC on the other side) followed by a response code +from sysexits.h followed by a response message string which describes the error +if there was one. If the response code is not 0, then the processed message +will not be sent, and the socket will be closed after the first line is sent. + +Again, there may be additional headers following the response line, which are +as yet undefined. Clients should ignore these, and keep looking for headers +which they do support, or the "\r\n\r\n" end-of-headers marker. + + +Commands +-------- + +The following commands are defined as of protocol 1.5: + +CHECK -- Just check if the passed message is spam or not and reply as + described below + +SYMBOLS -- Check if message is spam or not, and return score plus list + of symbols hit + +REPORT -- Check if message is spam or not, and return score plus report + +REPORT_IFSPAM -- Check if message is spam or not, and return score plus report + if the message is spam + +SKIP -- Ignore this message -- client opened connection then changed + its mind + +PING -- Return a confirmation that spamd is alive. + +PROCESS -- Process this message as described above and return modified + message + +TELL -- Tell what type of we are to process and what should be done + with that message. This includes setting or removing a local + or a remote database (learning, reporting, forgetting, revoking). + +HEADERS -- Same as PROCESS, but return only modified headers, not body + (new in protocol 1.4) + + +CHECK command returns just a header (terminated by "\r\n\r\n") with the first +line as for PROCESS (ie a response code and message), and then a header called +"Spam:" with value of either "True" or "False", then a semi-colon, and then the +score for this message, " / " then the threshold. So the entire response looks +like either: + + SPAMD/1.1 0 EX_OK\r\n + Spam: True ; 15 / 5\r\n + +or + + SPAMD/1.1 0 EX_OK\r\n + Spam: False ; 2 / 5\r\n + +There may be additional headers following the "Spam:" header, which +are as yet undefined. Clients should ignore these, and keep looking for +headers which they do support, or the "\r\n\r\n" end-of-headers marker. + + +SYMBOLS command returns the same as CHECK, followed by a line listing all the +rule names, separated by commas. Note that some versions of the protocol +terminate this line with "\r\n", and some do not, due to an oversight; so +clients should be flexible on whether or not a CR-LF pair follows +the symbol text, and how many CR-LFs there are. Protocol version 1.3 +onwards will always not terminate the line with "\r\n". + + +REPORT command returns the same as CHECK, followed immediately by the report +generated by spamd: + + SPAMD/1.1 0 EX_OK\r\n + Spam: False ; 2 / 5\r\n + \r\n + This mail is probably spam. The original message has been attached + along with this report, so you can recognize or block similar unwanted + mail in future. See https://spamassassin.apache.org/tag/ for more details. + [...] + +Note that the superfluous-score/threshold-line bug that appeared in +SpamAssassin 2.5x is fixed. + +Clients should be flexible on whether or not a CR-LF pair follows +the report text, and how many CR-LFs there are. + + +REPORT_IFSPAM returns the same as REPORT if the message is spam, or nothing at +all if the message is non-spam. + + +The PING command does not actually trigger any spam checking, and (as with +SKIP) no additional headers are expected. It returns a simple confirmation +response, like this: + + SPAMD/1.5 0 PONG\r\n + +This facility may be useful for monitoring programs which wish to check that +the daemon is alive and providing at least a basic response within a reasonable +time frame. + +Note that since the protocol version 1.5, a client sending a PING command +is required to follow the command (and a null header) with an empty line, +for consistency with other commands (fixes bug 6187). + + +TELL accepts three new headers, Message-class, Set and Remove and will return +two possible headers, DidSet and DidRemove which indicate which action was +taken. It is up to the caller to determine if the proper action happened. +Here are some examples: + +To learn a message as spam: +TELL SPAMC/1.3\r\n +Message-class: spam\r\n +Set: local\r\n + +To forget a learned message: +TELL SPAMC/1.3\r\n +Remove: local\r\n + +To report a spam message: +TELL SPAMC/1.3\r\n +Message-class: spam\r\n +Set: local, remote\r\n + +To revoke a ham message: +TELL SPAMC/1.3\r\n +Message-class: ham\r\n +Set: local\r\n +Remove: remote\r\n + +HEADERS returns the same as PROCESS, up to and including the double-newline +separator between message headers and body -- but stops there. It was +added in SpamAssassin 3.2.0. Note that this requires protocol version +1.4. + + + +Headers +------- + +The following optional headers are defined as of protocol 1.4: + +Content-length + + Length of a request or response body, in bytes (generally a requirement + as of protocol version 1.2 onwards) + +Spam + + See above; used in server responses to the CHECK command. + +User + + Username of the user on whose behalf this scan is being performed. The + meaning of this is up to the server; format is that of a traditional UNIX + username ([-A-Za-z0-9_]+). + +Compress + + An optional header, sent by the client to the server, whose value may + consist of the string "zlib", indicating that the message body transmitted + by the client is compressed using Zlib compression. (This is new in + SpamAssassin 3.2.0.) + +As-yet-undefined headers should not be treated as errors, and instead +should be ignored. Multiple headers can appear in requests and responses +(this was not clearly defined until protocol version 1.3). + diff --git a/docs/standards/spamd/README.md b/docs/standards/spamd/README.md new file mode 100644 index 00000000..e7fbbded --- /dev/null +++ b/docs/standards/spamd/README.md @@ -0,0 +1,26 @@ +# Apache SpamAssassin SPAMD network protocol + +## Document + +| Field | Value | +| --------------- | --------------------------------------------------------------------- | +| Title | The SpamAssassin Network Protocol | +| Author / org | Apache SpamAssassin | +| Local file | [`PROTOCOL.md`](PROTOCOL.md) | +| Source (browse) | https://github.com/apache/spamassassin/blob/trunk/spamd/PROTOCOL | +| Source (raw) | https://raw.githubusercontent.com/apache/spamassassin/trunk/spamd/PROTOCOL | +| Upstream SHA-256 | `ddd66ba900080cb510244bcb6e855f13ea469febe24373a84cff323caaaf0af5` | + +This directory holds a bundled copy of the Apache SpamAssassin `spamc`/`spamd` +network protocol specification used by **upstream** VectorNNTP.NNTPD PostFilter. +VectorNNTPD.BackFillerService and VectorNNTPD.StorageService do not implement a SPAMD client. + +`PROTOCOL.md` begins with a short VectorNNTP provenance note. The protocol +specification text after that note is a **verbatim** copy of the raw upstream +document. Do not edit the specification for style or summaries; replace it only +by re-downloading the upstream file and updating the hash when that document +changes. + +The upstream document states that it is perpetually in draft status. This +library does not assign a protocol version or revision date beyond what the +upstream text itself contains. diff --git a/docs/storage-nntp-feed.md b/docs/storage-nntp-feed.md new file mode 100644 index 00000000..fe50894c --- /dev/null +++ b/docs/storage-nntp-feed.md @@ -0,0 +1,127 @@ +# StorageService NNTP feed protocol + +Wire behaviour implemented by `VectorNNTPD.StorageService.Listener.StorageNntpFeedSession` +(and TLS accept path in `StorageNntpListenerService`). This is **VectorNNTPD StorageService** +behaviour only. It is not a full NNTP server profile and not upstream `VectorNNTP.NNTPD`. + +Standards text for general NNTP framing: [standards/rfcs/](standards/rfcs/) +(RFC 3977 article/command framing; RFC 4644 streaming/`TAKETHIS` context). Where this host +diverges, the **code wins**. + +Architecture and configuration: [storage.md](storage.md). +Do not use [reference/upstream-nntpd/commands.md](reference/upstream-nntpd/commands.md) as a +capability list for this host. + +## Scope + +| Supported | Not supported (receive `500 Unknown command` or no implementation) | +|-----------|---------------------------------------------------------------------| +| Greeting, then `ARTICLE`, `TAKETHIS`, `QUIT` | `MODE STREAM` / `MODE READER`, `CAPABILITIES`, `AUTHINFO`, `IHAVE`, `CHECK`, `POST`, `LIST`, group navigation, compression, STARTTLS (listener is already TLS) | + +No mode negotiation is required before `ARTICLE` or `TAKETHIS`. There is no cleartext listener; +TLS 1.2/1.3 is established before the session runs. + +## Connection lifecycle + +1. TCP accept → TLS server handshake (timeout: `Listener:TlsHandshakeTimeoutSeconds`, default 30). +2. Session writes greeting: `200 VectorNNTP StorageServer ready` + CRLF. + (Code uses status **200**, not `201`. Text is fixed; not an RFC capabilities advertisement.) +3. Command loop: read one command line (max **512** octets including CRLF). Overlong → `501 Syntax error`, connection stays open. +4. Keywords are ASCII case-insensitive. Arguments: Message-ID must pass Common + `NntpMessageIdValidation` (RFC 5536-oriented grammar used by this codebase). +5. Session ends on: successful `QUIT`, peer disconnect, cancellation, or handlers that return + “close” after writing `400 Temporary failure` (see TAKETHIS). + +Per-connection I/O uses `IoProgressTimeoutSeconds` (default 60). While TAKETHIS waits for queue +capacity, read-progress timing is suspended, then armed again. Overall TAKETHIS body receive uses +`ArticleReceiveTimeoutSeconds` (default 300), which does **not** reset on drip bytes. + +Path identity for article parse/normalize is the runtime FQDN (`cache{ServerId:00}.{DnsSuffix}`). + +## QUIT + +| Request | Response | Session | +|---------|----------|---------| +| `QUIT` with no argument | `205 Connection closing` | Closes after write | +| `QUIT` with any argument | `501 Syntax error` | Stays open | + +## ARTICLE + +Serves an indexed, **published** CanonicalV1 spool file for `ARTICLE `. + +| Condition | Response | +|-----------|----------| +| Missing / invalid Message-ID / extra tokens | `501 Syntax error` | +| Not in index / path resolve empty / file or directory not found | `430 No Such Article Found` | +| I/O / access / truncated read while loading file | `400 Temporary failure` | +| Empty payload, size > 5 MiB (`ArticleResourceLimits.MaxArticleBytes`, enforced on file length / buffer before `220`), or not frameable | `403 Article not available` | +| Success | `220 0 article follows` + CRLF, article bytes (header lines beginning with `.` may be stuffed for the socket), then `.\r\n` | + +Important implementation details: + +- A read lease is taken before path resolve so eviction cannot delete between lookup and open. +- The full article is read and framing analyzed **before** any `220` is written, so a read failure + never injects a second status into a partial body. +- Write failure after `220` propagates and ends the session without another NNTP status. +- Temporary `.writing-*` files are never opened for retrieval. +- Best-effort last-access touch after a successful write does not change the already-sent `220`. + +`ARTICLE` with no Message-ID (or an article-number-only form) is not implemented as group-current +article semantics; invalid/empty arguments yield `501`. + +## TAKETHIS + +Streaming-style transfer: command line carries the Message-ID; body follows as a multiline +article ending in `.\r\n` (dot-stuffing undone while reading). + +On many reject paths the body is **drained** (discarded) before the status is written so the next +command is not contaminated. + +### Success + +1. Destuff and assemble body (overall receive timeout; size capped by + `ArticleResourceLimits.MaxArticleBytes` = **5 MiB** during receive/drain and again at + `ArticleRecordFactory` / parse). +2. Reject if already present in the article index (`439`). +3. `ArticleRecordFactory.TryCreate` with path normalize; reject if not accepted or ArtId ≠ request Message-ID (`439`). +4. Reserve/commit into `IAcceptedArticleQueue` (may wait; I/O progress suspended during wait). +5. On successful commit, write **`239 `** + CRLF. + +`239` means the article entered the **accepted queue**, not that spool write, SQLite index update, +NVMe/SATA flush, or any replication finished. See [storage.md](storage.md) durability note. + +### Rejection and temporary failure + +| Condition | Typical response | Connection | +|-----------|------------------|------------| +| Invalid Message-ID on command (body drained when applicable) | `501 Syntax error` | Open | +| Capacity gate denies admission (critical/unknown) | `439 ` | Open | +| No assembly admission permit | `439 ` | Open | +| Duplicate ArtId already indexed | `439 ` | Open | +| Parse/materialize not accepted | `439 ` | Open | +| Message-ID ≠ canonical ArtId | `439 ` | Open | +| Article larger than queue `MaxQueuedBytes` | `439 ` | Open | +| Queue reserve fails / completed queue | `439` or `400 Temporary failure` | `400` paths close | +| Commit fails; receive timeout; oversized/unrecoverable framing (`InvalidOperationException`) | `400 Temporary failure` | **Closes** (`HandleTakeThisAsync` returns false) | + +`439` status lines include the request Message-ID (same scratch format as `239`). This host uses +`439` broadly for “not accepted into the queue” conditions above; it does not implement a separate +RFC 4644 `CHECK`/`TAKETHIS` negotiation dialogue beyond this command. + +Unknown verbs → `500 Unknown command` (connection stays open). + +## Differences from “full” NNTP expectations + +- Greeting `200` with Storage-specific text; no capability advertisement. +- No authentication; TLS is the only access control at this layer. +- No `MODE STREAM` requirement despite offering `TAKETHIS`. +- Cache-style `ARTICLE` by Message-ID from local spool/index, not newsgroup article numbers. +- `239` is queue admission, not proven durable store. +- Upstream NNTPD command inventories and PostFilter/Redis behaviour do not apply here. + +## Evidence pointers + +- Session commands and status bytes: `StorageNntpFeedSession.cs` +- TLS listener / FQDN path identity: `StorageNntpListenerService.cs` +- Size ceiling: `VectorNNTPD.Common.Articles.ArticleResourceLimits.MaxArticleBytes` +- Queue / persistence semantics: `SpoolPersistenceOptions` remarks; `AcceptedArticleQueue` diff --git a/docs/storage.md b/docs/storage.md new file mode 100644 index 00000000..a686ca3d --- /dev/null +++ b/docs/storage.md @@ -0,0 +1,246 @@ +# VectorNNTPD.StorageService — architecture and configuration + +Operator and developer contract for the StorageService host in `VectorNNTPD.slnx`. +Claims below are verified against `src/VectorNNTPD.StorageService/` and related Common types +on the current tree. This is **not** BackFiller documentation and **not** upstream NNTPD. + +Related docs (do not duplicate): [logging.md](logging.md), [systemd.md](systemd.md), +[common/configuration.md](common/configuration.md), [common/vatp.md](common/vatp.md), +[common/ArticleRecord.md](common/ArticleRecord.md), feed wire behaviour +[storage-nntp-feed.md](storage-nntp-feed.md), root [../README.md](../README.md). + +## Responsibilities + +StorageService is a **TLS-only NNTP feed / cache host**: + +- Accepts inbound TLS NNTP sessions and speaks a small command set (`ARTICLE`, `TAKETHIS`, `QUIT`). +- Parses accepted TAKETHIS bodies into CanonicalV1 `ArticleRecord` values, queues them, and + persists them into configured segment-pool spool directories. +- Indexes articles in a **local SQLite** database under `StorageServer:DbDir`. +- Publishes capacity advertisements to RabbitMQ exchange `cache.broadcast` (connectivity only; + this host does not register Article Work consumers). +- Loads ACME / Cloudflare / RabbitMQ credentials from MySQL `nntpsharedconfig` (selected columns). + +It does **not** implement NNTPD Reader/Transit command sets, PostFilter, Redis quota, or +BackFiller Article Work settlement. Upstream SQL under [schema/](schema/) is not applied by this host. + +## Architectural boundaries + +| Concern | Owner | +|---------|--------| +| Options bind / validate | `StorageServerOptions`, `StorageServerOptionsValidator` | +| Host entry / JSON | `Program.cs`, `ApplicationJsonConfiguration` | +| Application service order | `StorageServerServiceCollectionExtensions.AddStorageServerHosting` | +| Lifecycle / READY | Common `ApplicationLifecycle` + Storage `SystemdLifecycleNotifier` | +| NNTP TLS listener | `StorageNntpListenerService` | +| Per-connection session | `StorageNntpFeedSession` | +| In-memory admit queue | `AcceptedArticleQueue` / `AcceptedArticleWorker` | +| Spool write / LFU | `SpoolArticlePersistenceProcessor`, segment-pool / eviction services | +| Article index | `ArticleIndexService` / `ArticleIndexDatabase` (SQLite) | +| Shared MySQL row | `StorageServerSharedConfigurationService` | +| Capacity ads | `StorageServerAdvertisementPublisherService` → `cache.broadcast` | + +### Startup order (application services) + +Documented in code remarks and registration order: + +1. NntpDB +2. `nntpsharedconfig` +3. Cloudflare DNS reconciliation +4. RabbitMQ (connectivity) +5. ACME certificates +6. Segment-pool validation +7. Spool shard initialization +8. Article index +9. Block-device monitor +10. Spool eviction monitors +11. Accepted-article worker +12. NNTP feed listener +13. Advertisement publisher + +Shutdown is the reverse: advertisement stops first; the listener drains before the worker stops. +NntpDB does not migrate schema. Segment-pool validation and shard initialization throw before the +worker, listener, or advertisement publisher start. + +## Configuration sources and precedence + +Content root is `AppContext.BaseDirectory` (executable directory), not the process working directory. + +| Source | Role | +|--------|------| +| `VectorNNTPD.StorageService.json` | Primary host JSON (`{entry assembly name}.json`) | +| Optional `VectorNNTPD.StorageService.{Environment}.json` | Environment overlay when present | +| `VectorNNTPD.RabbitMQ.json` | Shared RabbitMQ connectivity (inserted after entry-assembly JSON) | +| Environment variables | Generic Host default environment source (for example `StorageServer__ServerId`, `ConnectionStrings__NntpDB`) plus `AddVectorEnvironmentVariables()` for the shared `VECTOR__` prefix (RabbitMQ/ACME/Cloudflare names documented in [common/configuration.md](common/configuration.md)). Later sources override earlier JSON for the same keys. Bound RabbitMQ user/password and shared ACME/Cloudflare secrets from env are still discarded/projected as described above. | +| Command-line args | Generic Host args after env | +| `/etc/vectornntp/storageservice.env` | systemd `EnvironmentFile` (typically `ConnectionStrings__NntpDB`) | +| MySQL `nntpsharedconfig` | ACME directory URL, renewal days, zone id, DNS suffix, ACME account/password, Cloudflare API key, RabbitMQ username/password | + +Bound RabbitMQ username/password from JSON/env are cleared at PostConfigure; live credentials come from +`nntpsharedconfig` after that snapshot is published. Do not log complete options objects. + +Section name: **`StorageServer`**. RabbitMQ uses top-level section **`RabbitMQ`** (not nested). + +## Required versus optional + +### Required for a normal production start + +| Item | Evidence | +|------|----------| +| `StorageServer:ServerId` | Integer 1–255; missing ≠ `0` | +| `StorageServer:BindPortTls` | Integer 1–65535; TLS-only — no cleartext fallback | +| `ConnectionStrings:NntpDB` / `ConnectionStrings__NntpDB` | Startup validation / NntpDB | +| Reachable singleton `nntpsharedconfig` with columns Storage selects | `StorageServerSharedConfigurationReader` | +| Writable `DbDir`, `LogDir`, `AcmeStateDir` / certs, and each `SegmentPools:Directories` entry | Validation + start-time services | +| Usable ACME certificate before listener bind | Listener refuses to start without TLS material | + +### Optional / deferred from JSON + +| Item | Notes | +|------|-------| +| `BindAddress` | Empty/omitted → all interfaces (`*` after PostConfigure). Explicit addresses must be local. | +| `BindPort` | Default `0`; never listened on | +| `DnsSuffix`, `CloudFlareZoneId`, `AcmeDirectoryUrl`, `AcmeRenewalThresholdDays` | Published from `nntpsharedconfig`, not trusted from this JSON section after projection | +| `SpoolPersistence` | Omitted uses type defaults (still validated when present as a section) | +| `StartupTimeout` | Optional lifecycle bound | +| systemd `WatchdogSec` | Unit ships commented out; app `EnableWatchdog` default true | + +Storage’s `nntpsharedconfig` reader does **not** select `maxartsize`, `sitename`, or `prometheusurl`. +Article size for this host uses Common `ArticleResourceLimits.MaxArticleBytes` (5 MiB) at parse/receive, +not a MySQL maxartsize column. + +## Configuration keys (bindable) + +Defaults below are code defaults on `StorageServerOptions` / nested types. The shipped +`VectorNNTPD.StorageService.json` may override some (for example `LogDir: logs`, explicit +`BindAddress`, `BindPortTls: 1191`). + +| Key | Default (code) | Notes | +|-----|----------------|-------| +| `ServerId` | none | Required 1–255 | +| `BindAddress` | `*` after PostConfigure | Wildcards `*`, `0.0.0.0`, `::` | +| `BindPort` | `0` | Unused cleartext | +| `BindPortTls` | none | Required 1–65535 | +| `AcmeStateDir` | `certs` | Relative → `ApplicationLocalPath` | +| `CertificateDirectory` | `certs` | | +| `LogDir` | `/logs` (constant); shipped JSON uses `logs` | File sink directory | +| `DbDir` | `db` | SQLite index; must not coincide with a spool root | +| `GracefulShutdownTimeout` | `00:00:30` | Also drives host `ShutdownTimeout` | +| `StopHostOnUnexpectedServiceTermination` | `true` | | +| `ApplicationName` | entry assembly name | Lifecycle display; log file names use entry assembly name | +| `Listener:ParserAccumulationMaxBytes` | `262144` | | +| `Listener:TlsHandshakeTimeoutSeconds` | `30` | | +| `Listener:IoProgressTimeoutSeconds` | `60` | Suspended while waiting for queue capacity | +| `Listener:ArticleReceiveTimeoutSeconds` | `300` | Overall TAKETHIS receive deadline | +| `Listener:MaxActiveConnections` | `1024` | | +| `SegmentPools:Directories` | `["spool/cache"]` | Each dir dedicated mount before traffic | +| `SegmentPools:LowPercent` / `HighPercent` / `CriticalPercent` | `75` / `79` / `85` | Per-spool LFU / admission; Critical excludes placement | +| `SpoolPersistence:MaxQueuedBytes` | 1 GiB | Range 1 MiB–8 GiB | +| `SpoolPersistence:MinWorkers` / `MaxWorkers` | `1` / `8` | | +| `SpoolPersistence:MaxInFlightPerSpool` | `2` | | +| `SpoolPersistence` scale/control intervals | see `SpoolPersistenceOptions` | Bound later disk work; **do not** make TAKETHIS `239` durable | +| `Systemd:*` | ready/stop/status/watchdog flags; fraction `0.5` | See [systemd.md](systemd.md) | + +FQDN is generated: `cache{ServerId:00}.{DnsSuffix}` after `DnsSuffix` is published. It is not a bindable key. + +## Persistence and paths + +| Path | Role | +|------|------| +| `DbDir` | SQLite `ArticleIndex` / `ArticleIndexMeta` (+ WAL). Created at startup. Schema created/migrated in-process — **not** the MySQL scripts under `docs/schema/`. | +| `SegmentPools:Directories` | Canonical article spool roots; placement and LFU eviction | +| `LogDir` | Serilog rolling gzip files | +| `AcmeStateDir` / `CertificateDirectory` | ACME/TLS material | + +Under systemd, unit `ReadWritePaths` include `logs/`, `certs/`, `db/`, and `spool/` under +`/opt/vectornntp/storageservice`. See [systemd.md](systemd.md) and root README. + +**Durability note (code contract):** TAKETHIS `239` is written after the article is committed into +`AcceptedArticleQueue`. SpoolPersistence settings bound later disk work. They do **not** make that +`239` mean the article is already durable on disk. + +## Ingestion flow (summary) + +1. Peer completes TLS; session greets and accepts commands ([storage-nntp-feed.md](storage-nntp-feed.md)). +2. Successful TAKETHIS → parse/normalize → queue commit → `239`. +3. `AcceptedArticleWorker` persists to an eligible spool and updates the article index. +4. `ARTICLE ` serves indexed published CanonicalV1 files (not `.writing-*` temps). +5. Advertisement publisher emits capacity to `cache.broadcast` about once per second while running. + +## Messaging + +- Shared `RabbitMqService` connection (Common). Username/password from `nntpsharedconfig`. +- Host registers **connectivity** and the advertisement publisher. Topology/queue/consumer registration + for Article Work is **not** done in `AddStorageServerHosting` (code remark). +- This host does **not** register a `cache.requests` consumer in the current hosting setup (that + fanout name exists in Common `CacheFleetTopology` for NNTPD↔Storage fleet lookup contracts). + +### `cache.broadcast` advertisements + +Publisher: `StorageServerAdvertisementPublisherService`. Declares durable **fanout** exchange +`cache.broadcast` so Storage can start before an NNTPD consumer exists. + +| Behaviour | Verified value | +|-----------|----------------| +| Cadence | `CacheFleetTopology.AdvertisementInterval` = **1 s** | +| AMQP expiration | `AdvertisementExpirationMilliseconds` = **`"3000"`** (3 s message TTL) | +| Persistent | `false` | +| Routing key | Exchange name (fanout ignores key; empty keys are rejected by the publisher helper) | +| Content-Type | `application/json` | +| AppId | Runtime FQDN | +| Starts after | Advertisement service start (after NNTP listener is in the start order) | + +**Capacity advertisement v1** JSON (`StorageServerAdvertisementWireProtocol`): `version`, `serverId`, +`fqdn`, `totalBytes`, `usedBytes`, `availableBytes`, `timestamp` (UTC ISO-8601), optional +`vatpPort` (1–65535). Storage fills `vatpPort` from `BindPortTls` (wire name remains `vatpPort`). + +**Current capacity source:** DI registers `NoSpoolCapacityReader`, which always returns +`(TotalBytes, UsedBytes, AvailableBytes) = (0, 0, 0)`. Segment-pool Critical/High/Low percents do +**not** rewrite those advertisement fields. Treat published byte counts as **placeholder zeros** +until a real `IStorageCapacityReader` is registered. Identity/`vatpPort`/timestamp still publish. + +**Draining announcement** (separate JSON, still on `cache.broadcast`): on stop, after the publish +loop ends, one lifecycle body with `state: "Draining"` plus `version`, `serverId`, `fqdn`, +`timestamp`, `vatpPort` (`StorageServerLifecycleWireProtocol`). Best-effort within a 2 s timeout; +failure does not block shutdown. Periodic advertisements remain the liveness/readiness signal for +peers (NNTPD registry semantics live outside this host). + +## Logging and troubleshooting + +Serilog-only. Shipped JSON enables Console + Async File; `StorageServer:LogDir` rewrites the file path. +See [logging.md](logging.md). Under systemd, journald collects console output. + +Common failures: + +| Symptom | Check | +|---------|--------| +| Exit 1 / validation | `ServerId`, `BindPortTls`, paths, `ConnectionStrings__NntpDB` | +| Never READY | Startup order stuck (MySQL, ACME, segment pools); `TimeoutStartSec` | +| Listener will not start | ACME certificate not ready (TLS-only) | +| TAKETHIS always `439` | Capacity critical / assembly admission / duplicate / parse reject / queue budget — see feed doc | +| `239` but article missing later | Persistence backlog, spool eligibility, index — `239` is not durability | + +Do not paste connection strings or `nntpsharedconfig` secrets into tickets. + +## systemd, readiness, shutdown + +- Ready: `ApplicationLifecycle` → `Running` (single-shot). Notify failures **rethrow**. +- Stopping: lifecycle STOPPING / host shutdown; `GracefulShutdownTimeout` vs unit `TimeoutStopSec`. +- Details and install paths: [systemd.md](systemd.md), root README Storage install notes. + +## Build and deploy + +- Solution: `VectorNNTPD.slnx`; project `src/VectorNNTPD.StorageService/`. +- Publish: `tools/Publish-Release.ps1 -Platform linux-64` (flat layout with BackFiller). +- Startup smoke: `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE=1` and + `VECTORNNTP_STORAGESERVICE_STARTUP_SMOKE_CONFIG` (path to isolated JSON) — builds host without + listener/DB; see root README. + +## Active databases (not `docs/schema/`) + +| Store | Applied by | +|-------|------------| +| MySQL `nntpsharedconfig` | External / shared ops; Storage only SELECTs listed columns | +| SQLite under `DbDir` | Storage `ArticleIndexDatabase` at runtime | + +Upstream Transit/PostFilter SQL remains under [schema/](schema/) as **reference only**. diff --git a/docs/systemd.md b/docs/systemd.md new file mode 100644 index 00000000..79d2381e --- /dev/null +++ b/docs/systemd.md @@ -0,0 +1,211 @@ +# VectorNNTPD — systemd deployment guide + +Linux systemd installation and operations for the two production hosts in +`VectorNNTPD.slnx`: **BackFillerService** and **StorageService**. + +This document covers Type=notify readiness, STOPPING, optional watchdog, journald, +and unit hardening. It does **not** replace the full install ceremony in +[../README.md](../README.md) (account creation, `install` modes, environment-file +secrets handling). It does **not** cover NNTP listeners, RabbitMQ topology, Article Work, +or VATP details — see [architecture.md](architecture.md) and [common/vatp.md](common/vatp.md). + +Both hosts share the same *operational* systemd contract (Type=notify, READY/STOPPING, +optional WatchdogSec, journald) but keep **distinct** readiness and notify-failure +semantics. Do not assume identical internal state machines. A design-only comparison of +extracting a shared kit is in +[common/systemd-host-abstraction.md](common/systemd-host-abstraction.md) (**proposal only**). + +Upstream `VectorNNTP.NNTPD` units and install paths are **not** documented here. + +--- + +## Shared facts + +| Item | Value | +|------|-------| +| Publish | `pwsh -File tools/Publish-Release.ps1 -Platform linux-64` → `release/linux-64/` (flat Native AOT layout for both hosts + shared `VectorNNTPD.RabbitMQ.json` + unit/env examples) | +| Service account | `vectornntp` / group `vectornntp` (may be shared by both trees under `/opt/vectornntp`) | +| Unit type | `Type=notify`, `NotifyAccess=main` | +| Watchdog in shipped units | `WatchdogSec` **commented out** until readiness is validated on the host | +| Secrets | `ConnectionStrings__NntpDB` in `/etc/vectornntp/*.env`; RabbitMQ/ACME/Cloudflare from `nntpsharedconfig` (not invented in the unit) | + +Detection limitations (both hosts): + +- Non-Linux: systemd notify/watchdog inactive; Windows Service registration still applies on Windows. +- Interactive Linux without `NOTIFY_SOCKET`: ordinary `dotnet run` is unaffected. +- `NOTIFY_SOCKET` is consumed by `Microsoft.Extensions.Hosting.Systemd` when the notifier is constructed; application code uses the host notify bridge, not a later re-read of the environment variable. +- Application configuration never overrides the systemd-supplied watchdog deadline (`WATCHDOG_USEC`). + +--- + +## BackFillerService + +### Paths and unit + +| Item | Value | +|------|-------| +| Install root | `/opt/vectornntp/backfiller` | +| Executable | `VectorNNTPD.BackFillerService` | +| Primary JSON | `VectorNNTPD.BackFillerService.json` | +| Unit | `deploy/systemd/vectornntp-backfiller.service` → `/etc/systemd/system/` | +| Environment file | `EnvironmentFile=-/etc/vectornntp/backfiller.env` | +| `ReadWritePaths` | `logs/`, `certs/` | +| `TimeoutStartSec` / `TimeoutStopSec` | 180 / 60 (shipped) | + +Follow [../README.md](../README.md#linux-service) for the authoritative `install` sequence, +environment-file permissions (`root:root` `0600`), and secret-handling rules. Do not +`chown -R` the install tree to `vectornntp`. + +### Application options (`BackFiller:Systemd`) + +| Option | Default | Meaning | +|--------|---------|---------| +| `EnableWatchdog` | `true` | Allow keep-alives when systemd set `WatchdogSec=` / `WATCHDOG_USEC` | +| `ReportLifecycleStatus` | `true` | Send `STATUS=` on Generic Host lifetime transitions | +| `NotifyReadyOnApplicationRunning` | `true` | Send `READY=1` after hosted services have started (see below) | +| `NotifyStoppingOnApplicationShutdown` | `true` | Send `STOPPING=1` when graceful shutdown begins | +| `WatchdogIntervalFraction` | `0.5` | Heartbeat interval as a fraction of the systemd deadline (validated ~0.05–0.9) | + +Align `TimeoutStopSec` with `BackFiller:Shutdown:GracePeriodSeconds` (shipped grace 30; maximum 600). + +### Readiness and notify failure + +- Ready boundary: Generic Host `ApplicationStarted` (all `IHostedService.StartAsync` succeeded, including work owned through `ApplicationServiceManager`). Implementation: `SystemdLifecycleNotifier` under `Hosting/Systemd/`. +- READY does **not** wait for every external provider NNTP session to be connected. +- Startup failure never reports ready; the process exits non-zero. +- Ready retries on unhealthy or handled notify failure until success, shutdown, or dispose. +- Notify failures are **logged and swallowed** so a broken notify socket does not crash the process. Missing `READY=1` / keep-alives can still trip systemd timeouts when configured. +- Watchdog keep-alives run only while `IApplicationHealth` reports healthy for watchdog (host started, shutdown not begun, no unexpected supervised `IApplicationService` termination / related sticky unhealthy conditions). + +### Logging under systemd + +Shipped file destination: `BackFiller:Logging:File:LogDir=logs`. Console Serilog sink is off unless `--console`. See [logging.md](logging.md). + +```bash +sudo journalctl -u vectornntp-backfiller.service -f +sudo systemctl status vectornntp-backfiller.service --no-pager +``` + +--- + +## StorageService + +Architecture, configuration keys, and persistence: [storage.md](storage.md). +NNTP feed commands: [storage-nntp-feed.md](storage-nntp-feed.md). + +### Paths and unit + +| Item | Value | +|------|-------| +| Install root | `/opt/vectornntp/storageservice` | +| Executable | `VectorNNTPD.StorageService` | +| Primary JSON | `VectorNNTPD.StorageService.json` | +| Unit | `deploy/systemd/vectornntp-storageservice.service` → `/etc/systemd/system/` | +| Environment file | `EnvironmentFile=-/etc/vectornntp/storageservice.env` | +| `ReadWritePaths` | `logs/`, `certs/`, `db/`, `spool/` | +| `TimeoutStartSec` / `TimeoutStopSec` | 180 / 60 (shipped) | + +Mirror the BackFiller install pattern: root-owned binary and JSON; `vectornntp`-owned writable directories; `storageservice.env.example` → `/etc/vectornntp/storageservice.env` mode `0600`. Create `db/` and `spool/` before enable. + +Example skeleton (adjust if README Storage steps are extended later): + +```bash +sudo install -d -o root -g root -m 0755 /opt/vectornntp /opt/vectornntp/storageservice +sudo install -o root -g root -m 0755 VectorNNTPD.StorageService /opt/vectornntp/storageservice/VectorNNTPD.StorageService +sudo install -o root -g root -m 0644 VectorNNTPD.StorageService.json VectorNNTPD.RabbitMQ.json README.md /opt/vectornntp/storageservice/ +sudo install -d -o vectornntp -g vectornntp -m 0750 \ + /opt/vectornntp/storageservice/logs \ + /opt/vectornntp/storageservice/certs \ + /opt/vectornntp/storageservice/db \ + /opt/vectornntp/storageservice/spool +sudo install -o root -g root -m 0600 storageservice.env.example /etc/vectornntp/storageservice.env +sudo install -o root -g root -m 0644 vectornntp-storageservice.service /etc/systemd/system/vectornntp-storageservice.service +sudo systemctl daemon-reload +sudoedit /etc/vectornntp/storageservice.env # replace REPLACE_WITH_ placeholders +sudo systemd-analyze verify vectornntp-storageservice.service +sudo systemctl enable --now vectornntp-storageservice.service +``` + +### Application options (`StorageServer:Systemd`) + +| Option | Default | Meaning | +|--------|---------|---------| +| `EnableWatchdog` | `true` | Allow keep-alives when systemd configured a watchdog | +| `NotifyReadyOnApplicationRunning` | `true` | Send `READY=1` when lifecycle reaches `Running` | +| `WatchdogIntervalFraction` | `0.5` | Heartbeat fraction of systemd deadline (same validation band as BackFiller) | + +Align `TimeoutStopSec` with `StorageServer:GracefulShutdownTimeout` (shipped `00:00:30`). + +### Readiness and notify failure + +- Ready boundary: Common `ApplicationLifecycle` transition to `ApplicationState.Running` after successful application-service initialization. Implementation: Storage `SystemdLifecycleNotifier`. +- Single-shot ready (no BackFiller-style retry loop). +- Startup failure never reports ready. +- Notify failures are **logged and rethrown** (broken notify fails closed). +- Watchdog keep-alives require lifecycle `Running`, no unexpected termination, shutdown not requested. + +### Logging under systemd + +Shipped `StorageServer:LogDir=logs`. Console sink enabled in shipped JSON → journald. See [logging.md](logging.md). + +```bash +sudo journalctl -u vectornntp-storageservice.service -f +``` + +--- + +## Deliberate differences + +| Topic | BackFiller | StorageService | +|-------|------------|----------------| +| Ready trigger | `IHostApplicationLifetime.ApplicationStarted` | `ApplicationLifecycle` → `Running` | +| Ready retry | Yes (until healthy + notify succeeds) | No (one attempt) | +| Notify failure | Log + swallow | Log + rethrow | +| Lifecycle owner | `ApplicationServiceManager` + hosted services | `ApplicationLifecycle` + application services | +| Writable paths | `logs`, `certs` | `logs`, `certs`, `db`, `spool` | +| Install path | `/opt/vectornntp/backfiller` | `/opt/vectornntp/storageservice` | + +--- + +## Troubleshooting + +### Failure to start + +```bash +sudo systemctl status vectornntp-backfiller.service --full +sudo journalctl -u vectornntp-backfiller.service -b --no-pager +# same pattern for vectornntp-storageservice.service +``` + +Check publish layout, execute bit, options validation, missing `ConnectionStrings__NntpDB`, and writable directory ownership under `ProtectSystem=strict`. + +### Readiness notification timeouts + +Symptoms: `TimeoutStartSec` exceeded, unit never becomes active. + +- Confirm `Type=notify` and that the process is the main PID. +- Confirm initialization reaches the host-specific ready boundary (BackFiller: hosted services started; Storage: lifecycle `Running`). +- ACME / RabbitMQ / listener bind can legitimately need a higher `TimeoutStartSec` (shipped 180). + +### Watchdog-triggered restarts + +Leave `WatchdogSec` unset until ready has been proven. Then confirm `EnableWatchdog=true` and that health remains true. Unexpected supervised-service termination stops keep-alives by design. + +### Shutdown timeouts + +Raise `TimeoutStopSec` when raising BackFiller `GracePeriodSeconds` or Storage `GracefulShutdownTimeout`. + +### Permission problems + +Confirm `User=`/`Group=`, that `ReadWritePaths` directories exist and are writable by `vectornntp`, and that binaries/JSON remain root-owned and not world-writable. + +### Running outside systemd + +Interactive `dotnet run` must not send watchdog keep-alives. If `NOTIFY_SOCKET` is accidentally set in a developer shell, remove it. + +## Security notes + +- Do not run as root. +- Prefer environment files with restrictive permissions over embedding secrets in the unit. +- Do not paste connection strings or `nntpsharedconfig` secrets into tickets or logs. +- Do not run `systemctl show -p Environment` when diagnosing production hosts.