From 0a2b693b7705039630a5e3a403149c6268410125 Mon Sep 17 00:00:00 2001 From: dim145 Date: Tue, 3 Nov 2026 21:02:53 +0100 Subject: [PATCH 1/3] Close what a security review and the scanners found: sign-in lockout, load limits, decode memory, relay caches, backups and logs A review of every area of the server, and of what semgrep, CodeQL, gitleaks, trivy and zizmor find in it. Nothing let a caller past a guard; these let one cost the server too much, see what was not theirs, or keep what they had lost. - Signing in: failures are counted per account and address, before the password is hashed, with one atomic count on the cache server; a bar five times higher across addresses never applies to an address the account signed in from in the last thirty days, so a guesser cannot shut the administrator out. A run of failed sign-ins or stray provider returns from one address is one audit row with its number. - Accounts: an administrator's pause of a key stands on its owner; tying an identity-provider login asks for the password (or a fresh sign-in at the provider), the owner can untie it, and an administrator's password reset does. The provider's client refuses internal addresses at connect time and judges every endpoint before asking it. A trusted proxy that names no client leaves the client unknown. - Load: no network holds more than 64 connections of a door (trusted proxies and this machine excepted), a connection with nothing in flight for two minutes is closed and an unread response given up on after a minute; searches, facets, lists and figures no cache holds are read three at a time, answered 503 with Retry-After past that; link previews and /ready are counted against the caller and kept a few seconds; /tmdb-images has a quota of its own and hands on 16 pictures at a time; search terms are cut at 200 characters at every door. - Pictures: what a decode will hold is counted from the header before anything is decoded, against 256 MB the decodes share; copies made on demand wait their turn before reading the original, 32 at once, counted against the caller; uploads are cut off as they are read, and a writer keeps 1 GiB of them; a writer is told only that a picture could not be fetched. - Relays: TheTVDB's own ids are found in maps read once per catalogue state, an id none could be answered at once, and seasons and orders' episodes are indexed by TheTVDB's id (migration 0053); reference lists are filed by path, language and region, 512 at most per relay; TheTVDB's and AniList's answers are keyed on the headers they are asked with. - Outbound requests send no Referer, and the providers a key travels to follow a redirect only within their origin; a series' IMDb id is checked before it goes into TMDB's /find path. - Logs mask a sign-in's code and state, TMDB's session ids and an auth parameter; an import rule's error and a database address no longer carry their keys or passwords. - Backups are written 0600 in 0700 directories, their downloads audited, and a restore quotes the archive's column names. transfer reads URLs that carry a password from AMS_TRANSFER_FROM and AMS_TRANSFER_TO. The trailer's frame is sandboxed, and trust-ca.sh installs exactly one certificate. - The scanners' false positives are named: .gitleaksignore for three lines of history, nosemgrep for the counting allocator and the watch script. Co-Authored-By: Claude Opus 5.5 --- .gitleaks.toml | 6 +- .gitleaksignore | 17 + CHANGELOG.md | 100 ++ CONTRIBUTING.md | 3 +- docker/trust-ca.sh | 24 + docs/access.md | 49 +- docs/cli.md | 39 +- docs/configuration.md | 12 +- docs/install.md | 14 +- docs/integration.md | 12 +- docs/media.md | 45 +- docs/perf/README.md | 13 + docs/relays.md | 15 + docs/reverse-proxy.md | 25 +- docs/security.md | 65 +- frontend/e2e/oidc.serial.spec.ts | 6 +- frontend/openapi.json | 163 ++- frontend/src/components/Theatre.tsx | 5 + frontend/src/lib/api.gen.ts | 279 ++++- frontend/src/lib/lang/en.ts | 12 +- frontend/src/lib/lang/fr.ts | 12 +- frontend/src/routes/Account.tsx | 149 ++- .../postgres/0053_season_order_tvdb_index.sql | 6 + .../sqlite/0053_season_order_tvdb_index.sql | 6 + scripts/watch/common.py | 3 + src/api/anilist.rs | 113 +- src/api/audit.rs | 482 +++++++- src/api/mod.rs | 51 +- src/api/native/account.rs | 150 ++- src/api/native/auth.rs | 352 +++++- src/api/native/backup.rs | 52 +- src/api/native/browse.rs | 179 ++- src/api/native/clients.rs | 187 ++- src/api/native/figures.rs | 87 +- src/api/native/items.rs | 119 +- src/api/native/lists/import_rules.rs | 2 +- src/api/native/media.rs | 415 ++++++- src/api/native/oidc.rs | 864 +++++++++++++- src/api/native/search.rs | 61 +- src/api/native/seasons.rs | 2 +- src/api/native/users.rs | 262 ++++- src/api/relay.rs | 221 +++- src/api/thetvdb.rs | 585 ++++++++- src/api/tmdb.rs | 106 +- src/api/tmdb_images.rs | 297 ++++- src/arr/client.rs | 40 + src/auth/ip.rs | 99 +- src/auth/middleware.rs | 20 +- src/auth/mod.rs | 44 +- src/auth/oidc.rs | 346 +++++- src/auth/ratelimit.rs | 905 +++++++++++--- src/backup/mod.rs | 127 +- src/backup/place.rs | 54 +- src/backup/restore.rs | 63 +- src/backup/snapshot.rs | 61 +- src/cache/brief.rs | 143 +++ src/cache/gate.rs | 114 ++ src/cache/mod.rs | 200 +++- src/cli/check.rs | 14 +- src/cli/mod.rs | 258 +++- src/config.rs | 7 +- src/db/mod.rs | 147 ++- src/db/repo/asset.rs | 75 ++ src/db/repo/audit.rs | 26 +- src/db/repo/item/search.rs | 53 + src/db/repo/relay_document.rs | 109 +- src/db/transfer.rs | 97 +- src/jobs/rules.rs | 62 +- src/main.rs | 11 +- src/media/encode.rs | 3 +- src/media/fetch.rs | 92 +- src/media/file.rs | 467 +++++++- src/media/mod.rs | 11 + src/media/placeholder.rs | 48 +- src/media/serve.rs | 331 +++++- src/media/variants.rs | 408 ++++++- src/memory.rs | 4 + src/outbound.rs | 3 + src/providers/tmdb/mod.rs | 59 +- src/search/rank.rs | 51 +- src/service/ids.rs | 33 + src/service/series.rs | 32 + src/service/webhook.rs | 3 + src/state.rs | 164 ++- src/telemetry.rs | 74 +- src/telemetry/journal.rs | 35 + src/telemetry/trace.rs | 58 + src/ui.rs | 341 +++++- src/web.rs | 74 +- src/web/listen.rs | 1048 ++++++++++++++++- 90 files changed, 11146 insertions(+), 935 deletions(-) create mode 100644 .gitleaksignore create mode 100644 migrations/postgres/0053_season_order_tvdb_index.sql create mode 100644 migrations/sqlite/0053_season_order_tvdb_index.sql create mode 100644 src/cache/brief.rs create mode 100644 src/cache/gate.rs diff --git a/.gitleaks.toml b/.gitleaks.toml index 6d1ae60..5943f11 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -14,8 +14,9 @@ # (`shift+mod+enter`, the editor's *Save and open the next*), the fake # `api_key=abcdef123456` the log's tests write on purpose, to see it come # out masked, and the fake `api_key=0123456789abcdef` the TMDB relay's tests -# write, to see it kept out of what a client is handed. None is a value that -# could open anything. +# write, to see it kept out of what a client is handed. And the interface's +# list of the services' host names, which the generic-key rule reads as a key +# behind the word `api`. None is a value that could open anything. [extend] useDefault = true @@ -28,6 +29,7 @@ regexes = [ '''keys[=:]\s*["']shift\+mod\+enter["']''', '''api_key=abcdef123456''', '''api_key=0123456789abcdef''', + '''skyhook\.sonarr\.tv, api\.radarr\.video, api\.themoviedb\.org, api4\.thetvdb\.com''', ] [[rules]] diff --git a/.gitleaksignore b/.gitleaksignore new file mode 100644 index 0000000..938e6ad --- /dev/null +++ b/.gitleaksignore @@ -0,0 +1,17 @@ +# Findings of the history that are not secrets, by fingerprint — the commit, +# the file, the rule and the line, which `gitleaks git` prints for each. A +# fingerprint names one line of one commit and nothing else: it loosens no +# rule, and a line like it written today would be found again. What is +# allowed on every commit, whatever its line, is in .gitleaks.toml. + +# A test's fake key — `ams_` and the base64 of a made-up phrase, not written +# here so that this file is no finding of its own — the key a Plex address +# carries in the test that checks it is masked in a logged path. It opens +# nothing; a later commit wrote one that says so on its face. +cf133cdde69916eb02aea11e847b383d37112bb1:src/telemetry.rs:generic-api-key:316 + +# The host name api.radarr.video, in the help text that lists the addresses +# the clients' door answers for (the French text, then the English): Radarr's +# own public API address, which the generic-key rule reads as a key. +c2f01484fd17d70fc97bc9af2d762cc60640e8a0:frontend/src/lib/lang/fr.ts:generic-api-key:867 +c2f01484fd17d70fc97bc9af2d762cc60640e8a0:frontend/src/lib/lang/en.ts:generic-api-key:860 diff --git a/CHANGELOG.md b/CHANGELOG.md index 998e54f..b21a7bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,106 @@ may change what the API or the configuration means, and says so here. ## [Unreleased] +### Security + +A review of the whole server, area by area, and of what semgrep, CodeQL, +gitleaks, trivy and zizmor find in it. Nothing let a caller past a guard; what +follows let one cost the server too much, see what was not theirs, or keep +what they had lost. + +- **Signing in.** Failed sign-ins locked an account for everyone, so a guesser + could shut the administrator out of their own server; they are counted per + account and address now (an IPv6 /64 as one), each before its password is + checked and with one atomic count on the cache server, and guessing from + many addresses meets a bar five times higher that never applies to an + address the account signed in from in the last thirty days. Concurrent + guesses no longer slip past the count while it is read, and a full count + forgets its oldest rather than stop counting. A run of failed sign-ins, or of + returns from the identity provider, from one address is one audit row with + its number; a return nobody started here is not written at all. +- **Accounts and keys.** A member could get round an administrator's pause of + their key by deleting it and making another; the pause stands on the member + now, until an administrator enables one of their keys or lifts it + (`DELETE /api/v1/users/{id}/keys-pause`). Tying an identity-provider login to + an account asks for the account's password — or has the provider sign the + person in again, for an account with none — where a session alone did; the + owner can untie it from their account page, an administrator's password + reset unties it, and the journal says who tied and who untied. +- **The identity provider's client** refuses link-local and metadata addresses + at connect time (loopback excepted), judges the provider's endpoints before + it asks any of them, and sends no Referer. +- **A trusted proxy** that forwarded a request without an `X-Forwarded-For` + hop made every client look like the proxy, whose address the allowlist + admits; the client is unknown now, refused by every allowlist, and the log + says so. +- **Load.** One address could hold every connection of a door — idle HTTP/2 + connections that answer pings, slow readers: no network holds more than 64 + now (a trusted proxy's and this machine's excepted), a connection with no + request in flight for two minutes is closed, and a response its peer does + not read is given up on after a minute. Searches, facets, lists and figures + no cache held are read three at a time, a caller waiting more than three + seconds answered `503` with `Retry-After`. Link previews, a network's page + and `/ready` asked the database afresh for every request: previews count + against the caller's allowance and are kept five seconds, the catalogue's + mark and `/ready`'s answer are asked once in five seconds, and a network's + slug is matched in memory. `/tmdb-images/` has a quota of its own, ten times + the requests', and hands on 16 pictures at a time. Search terms are cut at + 200 characters at every door, as the catalogue's own search always did. +- **Pictures.** One picture could take some 700 MB to decode, as only the + decoder's own buffer was bounded: decoding is counted by the byte a pixel + from the picture's header before anything is decoded, a picture that would + hold more than 256 MB is served whole, and the pictures decoding together + share those 256 MB. A copy made on demand waits for its turn before reading + the original, at most 32 are made at once, and each counts against the + caller's rate limit. An upload is cut off as it is read, at the weight its + first bytes say it may have, and a writer who is not an administrator keeps + 1 GiB of uploads all told. A writer is told only that a picture could not be + fetched, no longer how or after how many tries — which let a typed-in + address map the private network. +- **Relays.** TheTVDB's relay read every season, episode and character kept + without a TheTVDB id on each request for a negative number: a number no id + made here can be is answered at once, the others are found in maps read once + while the catalogue stays as it was, and seasons and orders' episodes are + indexed by TheTVDB's id (migration 0053). A reference list asked with a + made-up parameter or header filed a new database row each time: lists are + filed under their path and the language and region asked, 512 at most per + relay. TheTVDB's and AniList's relays key their answers on the Accept, + Accept-Language and Content-Type they ask with, as TMDB's does. +- **Outbound requests** send no Referer, which handed the next host of a + redirect the address it came from, key included; TMDB, Fanart.tv, TheTVDB + and MyAnimeList follow a redirect only within the origin first asked. A + series' IMDb id went into TMDB's `/find` path unchecked; it is checked as a + film's is and written as one segment. +- **Logs.** A sign-in's `code`, `state` and `session_state`, TMDB's + `session_id`, `guest_session_id` and `request_token`, and an `auth` + parameter are masked as the other keys are; an import rule that could not + fetch its list wrote reqwest's error, address and key included, into its + history and the log; a database address whose password holds an `@`, or + that carries it as `?password=`, was written in part. +- **Backups.** Archives, and the copy of the database one is made from, were + readable by every user of the machine: they are written for the server's + user alone (0600, new directories 0700), `config check` warns when archives + are not encrypted, and downloading one is written to the audit trail. A + restore wrote the archive's column names into SQL as they stood, so a crafted + archive could run statements of its own: names are quoted, and such an + archive is refused. +- **Elsewhere.** The trailer's frame is sandboxed. `trust-ca.sh` checked a + pinned fingerprint against a decoding of the whole file while installing + every certificate in it: a fetched or pinned file must be exactly one + certificate. History lines gitleaks takes for secrets are named, each with + why it is none, in `.gitleaksignore`. + +### Changed + +- `transfer` refuses a database URL with a password among its arguments: such + URLs are read from `AMS_TRANSFER_FROM` and `AMS_TRANSFER_TO`, which a + process list and a shell's history do not keep. +- A provider's upstream that redirects `http://` to `https://` is no longer + followed there: give it as `https://`. +- Tying an account to the identity provider is a button on the account page + that asks for the password (`POST /api/v1/account/provider`); untying is + beside it (`DELETE /api/v1/account/provider`). + ## [0.14.0] — 2026-11-03 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cbd73b1..0b83abd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,7 +97,8 @@ why. Say in the pull request why a new dependency is worth carrying. **Secrets.** None, in the code, the tests or the fixtures. `gitleaks detect --source . --redact` reads the history with -[`.gitleaks.toml`](.gitleaks.toml). +[`.gitleaks.toml`](.gitleaks.toml); the few lines of past commits that are no +secret are named, each with its reason, in [`.gitleaksignore`](.gitleaksignore). ## The interface's design system diff --git a/docker/trust-ca.sh b/docker/trust-ca.sh index b7a2b14..51ce809 100755 --- a/docker/trust-ca.sh +++ b/docker/trust-ca.sh @@ -18,6 +18,10 @@ # shows — or the risk is accepted, on a network you control, with # AMS_CA_INSECURE=1. # +# What is fetched, or pinned, is one certificate and nothing else: a file +# with a second PEM block is refused, since update-ca-certificates would +# install every block of it. +# # The same script is served by the server at /trust-ca.sh, and its # authority at /ca.crt, so a deployment from the image alone has both. # Mounted on purpose and finding nothing to install, it fails loudly: a @@ -29,6 +33,9 @@ CA_NAME="arr-metadata.crt" CA_PATH="/usr/local/share/ca-certificates/${CA_NAME}" CA_FILE="${AMS_CA_FILE:-/arr-metadata/tls/ca.crt}" +# Set once the authority is fetched from AMS_CA_URL rather than found here. +FETCHED="" + if [ -s "${CA_PATH}" ]; then echo "[trust-ca] using the authority mounted at ${CA_PATH}" elif [ -s "${CA_FILE}" ]; then @@ -55,6 +62,7 @@ elif [ -n "${AMS_CA_URL:-}" ]; then ;; esac echo "[trust-ca] fetching the authority from ${AMS_CA_URL}" + FETCHED=1 mkdir -p "$(dirname "${CA_PATH}")" if command -v curl >/dev/null 2>&1; then curl -fsS --retry 10 --retry-delay 3 --retry-connrefused -o "${CA_PATH}" "${AMS_CA_URL}" @@ -74,6 +82,22 @@ if ! grep -q "BEGIN CERTIFICATE" "${CA_PATH}"; then exit 1 fi +# A file that was fetched, or that a fingerprint is given for, is one +# certificate. The fingerprint below is taken of the file's base64 text as one +# stream, and a decoder is more lenient than OpenSSL: GNU's stops at a tab that +# OpenSSL skips, so a second certificate behind one goes unhashed, and +# update-ca-certificates installs every block of the file all the same. Any +# kind of PEM marker is counted — a TRUSTED CERTIFICATE block is a root to +# OpenSSL too — and the one let through is a CERTIFICATE. +if [ -n "${FETCHED}" ] || [ -n "${AMS_CA_FINGERPRINT:-}" ]; then + BLOCKS="$(grep -o -e '-----BEGIN' "${CA_PATH}" | wc -l | tr -d ' ')" + if [ "${BLOCKS}" != 1 ] || ! grep -q -e '^-----BEGIN CERTIFICATE-----' "${CA_PATH}"; then + echo "[trust-ca] ERROR: ${CA_PATH} holds ${BLOCKS} PEM blocks, or its one block is not a CERTIFICATE: an authority that is fetched or pinned is exactly one certificate, since a second block would be installed without being pinned; not installed" >&2 + rm -f "${CA_PATH}" 2>/dev/null || true + exit 1 + fi +fi + # Pinned, when a fingerprint was given: the SHA-256 of the certificate's DER, # as the administration page shows it, colons or not, in either case. if [ -n "${AMS_CA_FINGERPRINT:-}" ]; then diff --git a/docs/access.md b/docs/access.md index 8edea04..dddb21b 100644 --- a/docs/access.md +++ b/docs/access.md @@ -65,6 +65,28 @@ account, and no sign-up ever makes an administrator. Uninvited sign-ups are limited to `AMS_SIGNUPS_PER_HOUR` (5) per address — an IPv6 address counts by its /64 — and `AMS_SIGNUPS_PER_HOUR_TOTAL` (100) for the whole server. +**Failed sign-ins** are counted for each account and address. After +`AMS_SIGNIN_FAILURES_PER_ACCOUNT` attempts (10) on an account from one address — +an IPv6 address counts by its /64 — in fifteen minutes, that address is refused +without its password being looked at, until the window closes; the attempt is +counted before the password is checked, so a burst sent together is held to the +same ten. Somebody guessing at an account from their address shuts out that +address, and not the account's owner at theirs. Guessing from many addresses is +held by a bar for the account as a whole, five times as high; it does not apply to +an address the account has signed in from in the last thirty days, so its owner's +usual places get through with the right password. Failures of somebody with no +credential — a refused password, a refused return from the identity provider — +are written to the journal as one row for each address and kind in a minute: +written at once, and amended when the minute is over to say how many there were +and what the last of them typed. A return from the provider that was not started +here, or with no provider set up, is not written at all. + +A key a person holds is theirs to pause, and an administrator's to pause for +them from **Keys & network**: the pause then stands on the person, and until an +administrator enables one of their keys from there, or lifts it on their page +(`DELETE /users/{id}/keys-pause`), they can neither make a key nor resume one — +deleting the paused key and making another is no way round it. + The same page switches each API off — Sonarr's, Radarr's, the Plex provider, the TMDB, TheTVDB and AniList relays, and the native API's keys (the interface's own session is never cut). A switched-off API answers `503`, which Sonarr and Radarr @@ -118,12 +140,27 @@ The flow is the authorization code with PKCE, a state and a nonce, carried in a cookie sealed with a key only the running server holds (`__Host-` over HTTPS); the ID token's signature — asymmetric only — issuer, audience, expiry and `at_hash` are checked. The issuer, and every endpoint it publishes, must be -HTTPS unless it is a local address. A returning person is found by the -provider's own identifier. Someone who already has an account here ties it to -theirs at the provider from their account page, signed in, which proves both -sides; nothing is ever tied by e-mail, since an address typed into a profile -proves nothing. Otherwise, if allowed, an account is opened — a member's, or -the role a claim gives (`groups`, a path such as `realm_access.roles`, or a +HTTPS unless it is a local address, and none may be at a link-local, metadata, +multicast or unspecified address — written as one, or a name that resolves to +one; this machine's own loopback is the one such address a provider may have. +The provider is reached directly, whatever `HTTP_PROXY` says, and every address +it publishes is judged before it is asked for, its keys' included. A returning +person is found by the provider's own identifier. Someone who already has an +account here ties it to theirs at the provider from their account page: they give +their current password, are sent to sign in at the provider, and the tie is made +when they come back — which proves both halves are theirs. An account with no +password of its own has the provider sign its person in again (`prompt=login`, +`max_age=0`), and the provider's answer has to say when (`auth_time`). A session +alone ties nothing: one left open, or taken, would tie whoever holds it, and the +tie would outlive the password the account's owner then resets — nor does a +sign-in begun before the account was changed, a password reset above all, finish +its tie after it. Nothing is ever tied by e-mail, since an address typed into a +profile proves nothing. The owner can untie the account from the same page while +it has a password to sign in with — their other sessions are closed with it — +and an administrator can from the member's page; an administrator's reset of a +password unties the account too. The journal says who tied (`user.linked`) and +who untied (`user.unlinked`). Otherwise, if allowed, an account is opened — a +member's, or the role a claim gives (`groups`, a path such as `realm_access.roles`, or a namespaced `https://…/roles`, with the values that make an administrator or an editor). With a claim named, the provider decides the role at every sign-in, but never takes away the last administrator, and a sign-in whose claims did diff --git a/docs/cli.md b/docs/cli.md index 1996b90..666ba24 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -24,7 +24,9 @@ docker run --rm -it -v arr-metadata-server_metadata-data:/data --env-file .env \ `-it` is for a command that asks for a password at the terminal. One read from standard input — `--password-stdin`, `--passphrase-stdin` — needs `-i` alone, and nothing is ever taken as an -argument, which a process list and a shell's history would keep: +argument, which a process list and a shell's history would keep — the password of a database +address, which `transfer` takes, is read from the environment instead +([below](#the-server)): ```bash printf '%s\n' "$NEW_PASSWORD" | \ @@ -41,7 +43,7 @@ not exactly this binary's, rather than migrate it under a server that runs anoth | --- | --- | | *(none)*, `serve` | Runs the server. | | `healthcheck` | Asks the running server's `/health`, over loopback; exits non-zero if it is unwell. The image's health check. | -| `transfer FROM TO [--force]` | Copies a database into another, whatever their engines. | +| `transfer [FROM] [TO] [--force]` | Copies a database into another, whatever their engines. A URL with a password is read from `AMS_TRANSFER_FROM` or `AMS_TRANSFER_TO`, never an argument. | | `backup [--out PATH] [--include-media] [--passphrase-stdin]` | Makes an archive. | | `restore ARCHIVE [--replace] [--passphrase-stdin]` | Restores one. Refused while a server runs on the database. | | `user list` | Lists the accounts. | @@ -65,9 +67,26 @@ one moment, so a server may go on writing to it meanwhile; the copy is one trans failure leaves the target as it was: ```bash -arr-metadata-server transfer 'sqlite://data/ams.db' 'postgres://ams:secret@localhost/ams' +arr-metadata-server transfer 'sqlite://data/ams.db' 'postgres://ams@localhost/ams' ``` +A password is never an argument, which a process list and a shell's history would keep. A URL +that carries one — after the user's name, or as `?password=` — is refused as an argument, with a +message that says where to put it: `AMS_TRANSFER_FROM` names the source and `AMS_TRANSFER_TO` the +target, each standing in for the argument left out, so the source may be an argument and the +target a variable. An argument that is given wins over its variable, and a `.env` is read as for +every other variable. Typed with nothing shown, the variable is not in the shell's history +either: + +```bash +read -rs AMS_TRANSFER_TO # the target's URL, password and all +export AMS_TRANSFER_TO +arr-metadata-server transfer 'sqlite://data/ams.db' +``` + +In a container, hand it the variable by name — `docker compose run --rm -e AMS_TRANSFER_TO metadata +transfer 'sqlite:///data/ams.db'` — and Docker takes its value from the shell. + ## Backups `backup` makes an archive where the server keeps them — `AMS_BACKUP_DIR`, or the bucket among @@ -76,7 +95,9 @@ several instances — listed on **Administration › Backups** like any other, a puts the media kept on disk in it; `--out PATH` writes it to a file instead, or into a directory under its usual name, and does not list it; `--passphrase-stdin` encrypts it with a passphrase read from standard input — asked for, unseen, at a terminal — rather than `AMS_BACKUP_PASSPHRASE`. -It says where it wrote the archive, its size and its SHA-256. +It says where it wrote the archive, its size and its SHA-256. The archive is written readable by +the user who runs the command alone (`0600`), wherever it goes, the directories made for it +`0700`; [Installing and running](install.md#backups) says who else can be given it. `restore` puts an archive back: a file, or the name of one the server keeps. It has the database to itself, or it does not start: a server holding it — the server's own lock beside an SQLite @@ -134,10 +155,12 @@ TheTVDB or IMDb id: the document travels from one catalogue to another. `config check` prints the configuration as the server reads it — every secret masked, as the log masks it — and then probes what it names: the database, and where its migrations stand; the media -store, a directory or a bucket; where the archives are kept, written to and listed; the public -address's `/health`. A configuration the server would refuse to start with is said as the start -would say it. It exits non-zero when the database, the store or the archives' place does not -answer; the public address is only reported, since a proxy may route it from outside alone. +store, a directory or a bucket; where the archives are kept, written to and listed — with a +warning when `AMS_BACKUP_PASSPHRASE` is unset, since an archive then goes unencrypted with every +account's password hash in it; the public address's `/health`. A configuration the server would +refuse to start with is said as the start would say it. It exits non-zero when the database, the +store or the archives' place does not answer; the public address is only reported, since a proxy +may route it from outside alone. `migrate status` lists this binary's migrations — applied, pending, or failed — and any the database has that it does not know, which a later version applied. `migrate run` applies those diff --git a/docs/configuration.md b/docs/configuration.md index 034578a..d3423ff 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -73,8 +73,8 @@ naming the variable. Flags take `true`, `false`, `1`, `0`, `yes`, `no`, `on` and | `AMS_ALLOWED_HOSTS` | unset, any name | The host names the interface's door answers to, separated by commas; another is answered `421`, but for `/health`, `/ready`, `/ca.crt` and `/trust-ca.sh`. | | `AMS_CORS_ORIGINS` | unset, none | Origins whose pages may call the native API from a browser, separated by commas. `*` is ignored, with a warning. | | `AMS_REQUEST_TIMEOUT` | `60` | Seconds a request may take before it is answered `504`; 1 to 86400. | -| `AMS_MAX_CONNECTIONS` | `1024` | Connections each door holds at once; past it, a new one waits to be accepted. | -| `AMS_HEADER_READ_TIMEOUT` | `30` | Seconds a connection has to send a request's head, and may sit idle between two requests; 1 to 600. | +| `AMS_MAX_CONNECTIONS` | `1024` | Connections each door holds at once; past it, a new one waits to be accepted. No one address — an IPv6 /64 as one — holds more than 64 of them, a trusted proxy's and this machine's excepted. | +| `AMS_HEADER_READ_TIMEOUT` | `30` | Seconds a connection has to send a request's head, and may sit idle between two requests; 1 to 600. A connection with no request in flight for two minutes — an HTTP/2 one too — is closed. | | `AMS_LOG` (`RUST_LOG`) | `info` | What is logged, as a `tracing` filter: `debug`, `info,arr_metadata_server=debug`. The administration's **Server log** keeps the same lines. | | `AMS_LOG_FORMAT` | unset | `json` writes one JSON object per line, for a log shipper, each with the `trace_id` of the request it was written in. | @@ -125,8 +125,8 @@ OpenTelemetry collector. These are OpenTelemetry's own variables; see | `AMS_AUTH_DISABLED` | `false` | Opens every surface. For closed networks and a first setup only. | | `AMS_ALLOWLIST` (`AMS_ARR_ALLOWLIST`) | loopback and the private ranges | The first value of the list of allowed addresses, separated by commas; read against an empty list only. The default is `127.0.0.0/8`, `::1/128`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` and `fc00::/7`. | | `AMS_PUBLIC_BROWSE` | unset | `true` gives `site.access` the first value `public`; `false` keeps the site private whatever the setting says. | -| `AMS_RATE_LIMIT_PER_MINUTE` | `600` | Requests a minute, per address — an IPv6 address by its /64. TMDB's pictures under `/tmdb-images/` are not counted for an address their guard lets in. | -| `AMS_SIGNIN_FAILURES_PER_ACCOUNT` | `10` | Failed sign-ins an account takes in fifteen minutes before it is refused unchecked until the window closes. | +| `AMS_RATE_LIMIT_PER_MINUTE` | `600` | Requests a minute, per address — an IPv6 address by its /64. TMDB's pictures under `/tmdb-images/` are counted apart, against ten times as many, for an address their guard lets in. A copy of a picture made on demand under `/media/` counts for one each time it is asked for before it is kept; copies kept and pictures count for none. | +| `AMS_SIGNIN_FAILURES_PER_ACCOUNT` | `10` | Failed sign-ins an account takes from one address in fifteen minutes (an IPv6 /64 counts as one) before that address is refused unchecked until the window closes; five times as many from every address together, which does not apply to an address the account signed in from in the last thirty days. | | `AMS_SIGNUPS_PER_HOUR` | `5` | Sign-ups without an invitation, per address and per hour. | | `AMS_SIGNUPS_PER_HOUR_TOTAL` | `100` | The same, for the whole server. | | `AMS_OIDC_CLIENT_SECRET` | unset | The identity provider's client secret, which wins over the one stored in the settings. | @@ -163,7 +163,9 @@ These give a setting its first value, and are not read again once it has one — Each provider is asked at its own address unless one is given. An address given may be anywhere — a mirror on the same machine included; where it redirects is judged as any other -address the server fetches. See +address the server fetches, and TMDB, Fanart.tv, TheTVDB and MyAnimeList, which a key or a +client id travels to, are followed only within the origin first asked: an upstream that +redirects `http://` to `https://` must be given as `https://`. See [If you redirect at the resolver](integration.md#if-you-redirect-at-the-resolver). | Variable | Default | diff --git a/docs/install.md b/docs/install.md index 89b82ed..c25da85 100644 --- a/docs/install.md +++ b/docs/install.md @@ -110,7 +110,7 @@ Changing `AMS_DATABASE_URL` points the server at a different database; it does not carry your catalogue across. To move it: ```bash -arr-metadata-server transfer 'sqlite://data/ams.db' 'postgres://ams:secret@localhost/ams' +arr-metadata-server transfer 'sqlite://data/ams.db' 'postgres://ams@localhost/ams' ``` The target is migrated first and must be empty, unless you pass `--force` to add @@ -118,6 +118,11 @@ to what is already there. Everything goes across, the lists read whole — TVmaz index, Wikidata's links — included. An archive does the same: one made on SQLite restores on PostgreSQL, and the other way round (see [Backups](#backups)). +A URL that carries a password is not an argument, which a process list and the shell's +history would keep: the command refuses one, and takes the source from `AMS_TRANSFER_FROM` +and the target from `AMS_TRANSFER_TO` instead, each standing in for an argument left out +([the command line](cli.md#the-server)). + ## Backups The server makes archives of itself: its database — every work, lock, account, key, setting, @@ -142,6 +147,13 @@ downloads, deletes and restores them; [the command line](cli.md) does the same f put together in `AMS_BACKUP_DIR` all the same: that is where the room for a copy of the database is wanted, not in the system's temporary directory, often a small tmpfs in a container. +- **Who can read them.** The user who runs the server, alone: each archive is written `0600`, + and the copy of the database it is put together from, and the directories the server makes + for them — `AMS_BACKUP_DIR` when it is not there yet, the `.work` folder in it — `0700`, + whatever the umask says. A directory that exists already keeps the permissions it has, and so + do the archives an earlier version wrote: `chmod 600` those. A job that copies the directory as + another user has to run as root, or be given the right to read each archive, which nothing + gives it by default. - **Before an upgrade.** A start that finds migrations to apply first makes an archive of the database as the version before left it, without the media: `ams-20261019-020000-upgrade-m0037.tar.zst`, made before the migrations that bring it to 0037, diff --git a/docs/integration.md b/docs/integration.md index cad685e..edeeec1 100644 --- a/docs/integration.md +++ b/docs/integration.md @@ -174,6 +174,8 @@ That fetch is plain HTTP on the stack's own network, and what it installs is a root the container then trusts for everything — so over `http://` the script refuses to install anything it cannot check: set `AMS_CA_FINGERPRINT` to the SHA-256 shown on the Opening & APIs page, and anything else is refused. A +fetched or pinned authority must be exactly one certificate: a file with a +second PEM block is refused. A stack's very first start has no fingerprint to copy yet; `AMS_CA_INSECURE=1` accepts the fetch unpinned, with a warning in the container's log, on a network you control — `compose.integration.yaml` sets it until a fingerprint @@ -262,7 +264,8 @@ that is not in that list — it carries `X-Forwarded-For`, `Forwarded` or `X-Real-IP` — is therefore refused on these surfaces, with a warning naming the proxy; Sonarr and Radarr calling directly never send one. From a trusted proxy every `X-Forwarded-For` line is read, in order, right to left past the other -trusted hops; a hop that is no address leaves the client unknown, and an unknown +trusted hops; a hop that is no address, or a trusted proxy that sends only +`Forwarded` or `X-Real-IP`, leaves the client unknown, and an unknown client is refused by every allowlist rather than taken for the proxy. ### Changes need this server's own pages @@ -744,9 +747,10 @@ An upstream named this way may be anywhere — a mirror on the same machine or the same network included: it is yours. Where it *redirects* is not: a hop to an address that only means something from where this server stands (loopback, link-local, a cloud's metadata service), written as an address -or as a name that resolves there, is refused, and MyAnimeList's client id is -never carried to another origin — scheme, host and port — than the one first -asked. +or as a name that resolves there, is refused, and the key or client id TMDB, +Fanart.tv, TheTVDB and MyAnimeList are asked with is never carried to another +origin — scheme, host and port — than the one first asked: an upstream that +redirects `http://` to `https://` must be given as `https://`. ## When a provider does not answer diff --git a/docs/media.md b/docs/media.md index 33d904d..15066cd 100644 --- a/docs/media.md +++ b/docs/media.md @@ -29,7 +29,12 @@ default: nothing is kept until it is asked for. and whatever it redirects to. A picture mirror on a private network is reached all the same; `AMS_MEDIA_PRIVATE_NETWORKS=false` keeps the store and the NFO export out of `10/8`, `172.16/12`, `192.168/16`, `100.64/10` - and IPv6's unique-local `fc00::/7` too (`true` by default). + and IPv6's unique-local `fc00::/7` too (`true` by default). A writer who is + not an administrator is told of a picture that failed only that it "could + not be fetched", and not how many tries it took: what the address answered, + or that it did not, is the administrator's to read on **Administration › + Media** — an address is whatever a writer typed in, and what came back for + each would be a map of the network the server is on. - **How they are served.** Every copy is addressed by the hash of its bytes under `/media/`, and served immutable. The rows keep the providers' addresses: a work is rewritten as it is read, never as it is written, and @@ -51,7 +56,12 @@ default: nothing is kept until it is asked for. read for what it is, never for what it is called; SVG is refused, since a document that can carry a script must not be served from here — and a copy taken away. An upload's file goes with it; a provider's copy is - fetched again at the next refresh. + fetched again at the next refresh. A picture may weigh 25 MiB and a theme + 60, a writer who is not an administrator keeps 1 GiB of uploads all told, + and an upload is cut off at the weight of what it is as it is read — + told by its first bytes, or by the `kind` of the form when that comes + before the file —, not once all of it is held in memory; one whose first + bytes are neither a picture nor a sound is refused with them. - **The sweep.** Daily, and on request: what no work points at any more — a work removed, a picture a provider dropped, an upload taken off — is forgotten and its file deleted, and a file no row names is removed once it @@ -116,8 +126,16 @@ anything — and asks for the right files from its first picture. poster's 480, a background's 1280. Any other is made the first time a page asks for it — a few hundred milliseconds for a large background — and served from the store after. A width no page asks for is refused, so - a picture has a dozen copies at the most. The thumbnails earlier versions - made are still served; none is made any more. + a picture has a dozen copies at the most, a picture's own width being one + only where it is no wider than 1920. The thumbnails earlier versions + made are still served; none is made any more. Making one is work, and is + held to what the rest of the server's is: the address that asks is counted + against `AMS_RATE_LIMIT_PER_MINUTE` each time it asks for a copy that is not + kept yet — a copy already kept is served, and a picture, for nothing —; at + most thirty-two are made at once, and a page that asks for one when there + is no room is sent the picture itself, for that time, and asks again; and + a copy waits for its turn among the pictures being decoded before its + picture is read from the store, not after. - **Turned on.** The pages ask for the copies at once, and the pictures kept already have theirs made as they are asked for: a catalogue's first pages after the switch are a little slower to fill. @@ -129,9 +147,22 @@ anything — and asks for the right files from its first picture. [Deleting the copies of a format](#deleting-the-copies-of-a-format). - **Read safely.** A picture is decoded in Rust, never by libwebp: a picture past 16,384 pixels a side or 100 million pixels is not decoded - at all, nor one whose pixels would take more than 256 MB, and a decoder - that panics on what it is given is a copy refused, not a server brought - down. Such a picture is served whole. + at all, nor one whose decoding would hold more than 256 MB at its worst, + and a decoder that panics on what it is given is a copy refused, not a + server brought down. Such a picture is served whole. What decoding holds + is counted from the picture's header, before a byte of it is decoded, by + the byte a pixel at the worst moment of the way from the file to the copy: + the decoder's buffer and what it keeps beside it — a progressive JPEG's + coefficients, six bytes a pixel more than its three —, the copy a quarter + turn makes, the copy that makes the pixels eight bits an RGB or RGBA + channel, what the resizer holds beside them, and what is held at the end: + 3 to 9 bytes a pixel for a picture in colour, 16 for one with transparency. + A grey picture of 10,000 by 10,000 pixels is 100 MB for its decoder, and + took some 700 with the copies made of it; it is counted at 400 MB, or 900 + where a copy is made of it, and refused. The pictures being decoded hold + 256 MB between them, whatever their sizes: a large one waits for those + before it, and small ones go side by side. What an encoder holds to write + a copy is its own matter, and a copy is never wider than 1920 pixels. - **The square for Plex.** A background kept is also cut into a square from its middle, 1080 pixels a side, as a JPEG — Plex's `backgroundSquare`: see [Plex](plex.md#what-is-served). diff --git a/docs/perf/README.md b/docs/perf/README.md index ab08d35..b8b4c14 100644 --- a/docs/perf/README.md +++ b/docs/perf/README.md @@ -174,3 +174,16 @@ background: whole after 255 s (258 080 names of works, 39 976 people, 345 214 episodes), the search answering from what is written meanwhile. The file grows from 336 MB to 596 MB: the index's table, its FTS5 trigrams and its three indexes weigh 283 MB, less the 0022 index the migration drops. + +### The reads no cache answers go through three places + +The figures above were measured before the server took the reads no cache +answers — a search nobody made lately, the facets of a new filter, a page of +the catalogue in an order nobody asked for, the figures — three at a time +(`src/cache/gate.rs`). Thirty-two connections asking for new ones now queue +for those three places: the database's own limit is still what the rate is, +but a request that waits more than three seconds is answered `503` with +`Retry-After: 5`, which a load tool reports as an error. A scenario that +repeats a request is not touched, since what the cache holds takes no place; +a cold one is compared by the rate of its answers and the figures of those +that were answered, not by its failures. diff --git a/docs/relays.md b/docs/relays.md index 9ab8f1e..b033c1a 100644 --- a/docs/relays.md +++ b/docs/relays.md @@ -270,6 +270,15 @@ service fails and the memory holds nothing — after a restart, for instance. On older than 180 days is neither served nor kept, and the server deletes it as it goes; a database transfer carries the others. +A reference list is filed in the database under its path and the language and +region the request names (`language`, `region`, `watch_region`), and under nothing +else: not the rest of the query, not the caller's headers. A request that adds +parameters of its own finds the row the others wrote, rather than making one. A +language or a region that is not written as one — empty, longer than sixteen +characters, anything but letters, digits, `-` and `_` — or that a request gives +twice, is still answered, and kept in the database under none. A relay keeps at +most 512 of them; past that, the oldest go first. + `relays.staleIfError`, on unless an administrator turns it off, decides all of this: off, the service's failure is handed on as it came, and the reference lists, still written, are not read. It is set on **Opening & APIs**, under *When a service @@ -288,6 +297,12 @@ answers with what it kept, or `429` with the seconds left in `Retry-After`. A client's AniList query is never spaced out as the server's own fetches from AniList are. +What a relay keeps in memory is filed under the path, the query and the headers it +hands on that may change the answer: `Accept` and `Accept-Language` for TMDB, and +for TheTVDB and AniList `Content-Type` too, which says how AniList reads a body. +Two callers who ask with other headers are two entries; none is answered what the +service said to the other. + A client's `If-None-Match` and `If-Modified-Since` reach the service only with a request answered for that client alone, and handed back as the service wrote it: a request kept or shared with others is asked without them, since a "not modified" is diff --git a/docs/reverse-proxy.md b/docs/reverse-proxy.md index 12ca120..076b6c1 100644 --- a/docs/reverse-proxy.md +++ b/docs/reverse-proxy.md @@ -86,14 +86,17 @@ From a trusted proxy, and from nobody else: | Header | Read as | | --- | --- | -| `X-Forwarded-For` | The client's address. Its lines are read as one list, in the order they arrived, and walked from the right: the first address that is not itself a trusted proxy is the client. A hop that is no address leaves the client unknown: refused by every list of allowed addresses, and counted with the other unknowns. | +| `X-Forwarded-For` | The client's address. Its lines are read as one list, in the order they arrived, and walked from the right: the first address that is not itself a trusted proxy is the client. A hop that is no address leaves the client unknown: refused by every list of allowed addresses, and counted with the other unknowns. So does a request forwarded with no hop to read: a line left empty, or only a `Forwarded` or an `X-Real-IP`. | | `X-Forwarded-Proto` | The scheme the request reached the proxy by, `https` or `http`: its first value. | | `X-Forwarded-Host` | The name the request reached the proxy by: its first value. It counts as this server's own origin when a change is asked, and names the links of a feed when `AMS_PUBLIC_URL` is unset. | `Forwarded` and `X-Real-IP` are only recognised, never read: a proxy that sends either without -`X-Forwarded-For` leaves every client looking like the proxy. With several proxies in a row, -every one between the client and the server has to be in `AMS_TRUSTED_PROXIES`. A request from -an address that is not trusted keeps its own address, whatever it claims in these headers. +an `X-Forwarded-For` that has an address in it leaves its client unknown — refused by every +list of allowed addresses, counted with the other unknowns, and warned of in the log with the +proxy's address — rather than looking like the proxy. Only a request that carries none of the +three is taken to be the proxy's own. With several proxies in a row, every one between the +client and the server has to be in `AMS_TRUSTED_PROXIES`. A request from an address that is not +trusted keeps its own address, whatever it claims in these headers. The `Host` header is read as it arrives. With `AMS_ALLOWED_HOSTS` set, it has to be one of those names, or the request is answered `421` — so the proxy passes the client's `Host` on, and @@ -118,11 +121,16 @@ name, and the paths they call are best not forwarded by it at all: | `/v1/…`, `/v2/…` | Sonarr and Radarr | | `/3/…`, `/4/…` | TMDB clients | | `/v4/…` | TheTVDB clients | +| `/tmdb-images/…` | TMDB clients, for TMDB's pictures when `tmdb.images` names this server | They are guarded either way — by the list of allowed addresses, or by a key — but Plex sends a provider no credential, so the Plex provider is guarded by address alone unless its address -carries a key ([Keeping it on your network](plex.md#keeping-it-on-your-network)). The -interface asks for none of these paths. A Plex that is elsewhere and reaches this server +carries a key ([Keeping it on your network](plex.md#keeping-it-on-your-network)). So are +TMDB's pictures, since a picture's address carries no key. An address the list lets in is held +to a quota of its own, ten times the requests' — a library's scan asks for pictures in bursts — +and pictures are handed on from TMDB sixteen at a time; a proxy that forwards `/tmdb-images/…` +from the internet, to a TMDB policy that is `open`, still gives anyone a way to TMDB's picture +server through this one. The interface asks for none of these paths. A Plex that is elsewhere and reaches this server through the proxy is given a key in its address instead (`AMS_PLEX_AUTH=apikey`, [A key in the address](plex.md#a-key-in-the-address)), and `/plex` is then forwarded. @@ -182,7 +190,7 @@ server { client_max_body_size 61m; # The surfaces the clients call stay on the local network. - location ~ ^/(plex|v1|v2|v4|3|4)(/|$) { + location ~ ^/(plex|v1|v2|v4|3|4|tmdb-images)(/|$) { return 404; } @@ -210,6 +218,7 @@ certificate resolver `letsencrypt` in Traefik's own configuration: Host(`metadata.example.com`) && !PathPrefix(`/plex`) && !PathPrefix(`/v1/`) && !PathPrefix(`/v2/`) && !PathPrefix(`/v4/`) && !PathPrefix(`/3/`) && !PathPrefix(`/4/`) + && !PathPrefix(`/tmdb-images/`) traefik.http.routers.metadata.entrypoints: websecure traefik.http.routers.metadata.tls.certresolver: letsencrypt traefik.http.services.metadata.loadbalancer.server.port: "8080" @@ -224,7 +233,7 @@ Caddy obtains the certificate for the name, passes the `Host` on and sets the ``` metadata.example.com { - @clients path /plex /plex/* /v1/* /v2/* /v4/* /3/* /4/* + @clients path /plex /plex/* /v1/* /v2/* /v4/* /3/* /4/* /tmdb-images/* respond @clients 404 reverse_proxy metadata:8080 diff --git a/docs/security.md b/docs/security.md index f8e3847..517a511 100644 --- a/docs/security.md +++ b/docs/security.md @@ -31,37 +31,68 @@ is in [Accounts and access](access.md), and publishing the interface in browser's `Origin` is not this server's (`AMS_PUBLIC_URL` and `AMS_CORS_ORIGINS` count as this server's). Keys, Sonarr, Radarr and scripts are not concerned. -- **Every response carries a content security policy** with `frame-ancestors - 'none'`, so no page elsewhere can frame this one and borrow an administrator's - clicks, and `Referrer-Policy: no-referrer`, so the ids of what you are looking - at do not travel to the artwork hosts; a `Permissions-Policy` that denies the - camera, the microphone, the location and payments; on pages, a +- **Every response of the interface carries a content security policy** with + `frame-ancestors 'self'`, and `X-Frame-Options: SAMEORIGIN` beside it for an + engine older than the directive: no page elsewhere can frame this one and + borrow an administrator's clicks. This server's own pages may, on purpose — + the editor shows a work's public page in a frame, as a preview of it. The + clients' door, which serves no page, sends `default-src 'none'; + frame-ancestors 'none'; sandbox`. Both send `Referrer-Policy: no-referrer`, + so the ids of what you are looking at do not travel to the artwork hosts; the + interface also sends a `Permissions-Policy` that denies the camera, the + microphone, the location and payments; on pages, a `Cross-Origin-Opener-Policy`; and, over TLS — this server's own or a trusted proxy's — `Strict-Transport-Security` for a year. -- **Every surface is rate limited**, the arr and TMDB ones included +- **Every API surface is rate limited**, the arr and TMDB ones included (`AMS_RATE_LIMIT_PER_MINUTE`, 600 by default, per address — an IPv6 address - by its /64), save TMDB's pictures under `/tmdb-images/` for an address their - guard lets in, which a library's scan asks for in bursts; a refusal says - when to come back in `Retry-After`, and the per-address limit its quota in - `RateLimit` and `RateLimit-Policy`. Signing - in has two brakes of its own: an account that failed - `AMS_SIGNIN_FAILURES_PER_ACCOUNT` times (10) in fifteen minutes is refused - without being checked until the window closes, from wherever the attempts - come; and passwords are checked a few at a time — as many as the machine has + by its /64). TMDB's pictures under `/tmdb-images/` are counted apart for an + address their guard lets in, against a quota ten times as high, since a + library's scan asks for them in bursts; and they are handed on from TMDB + sixteen at a time, the others waiting their turn for five seconds before + they are answered `503`. A refusal says when to come back in `Retry-After`, + and the per-address limit its quota in `RateLimit` and `RateLimit-Policy`. + Signing in has two brakes of its own: an address — an IPv6 /64 as one — that + failed `AMS_SIGNIN_FAILURES_PER_ACCOUNT` times (10) in fifteen minutes for an + account is refused for it without being checked until the window closes, each + attempt counted before its password is; five times as many from every address + together refuse the account to the rest, never to an address it signed in from + in the last thirty days, so a guesser cannot shut its owner out; and passwords + are checked a few at a time — as many as the machine has cores, two to eight — so a burst of sign-ins waits, or is told to come back, rather than taking the memory argon2 needs for each. With passwords off, the name `AMS_ADMIN_USERNAME` keeps its door, and takes as long to refuse as any other name. +- **What costs the database more than it takes to ask for is done a few at a + time.** A search nobody made lately, the facets of a filter nobody chose, a + page of the catalogue, its figures: read three at a time, the others waiting + their turn for three seconds before they are answered `503` with + `Retry-After`. What the cache holds is answered whatever. A link's preview — + a work's, a list's, a genre's — is looked up on the caller's account, counted + against the allowance above as the API's requests are, and kept five seconds; + a caller who has spent its allowance is given the page without one. + `/ready` asks the database once in five seconds, however often it is asked. - **Each door holds `AMS_MAX_CONNECTIONS` connections at once** (1024); past - that, a new one waits to be accepted. A connection has + that, a new one waits to be accepted. No one network holds more than 64 of + them — an address, or an IPv6 /64 — and a connection past that is closed as + it is accepted; a trusted proxy's addresses and this machine's own are not + counted, since everybody behind a proxy has the proxy's. A connection has `AMS_HEADER_READ_TIMEOUT` seconds (30) to send a request's head, and may sit - idle between two requests no longer. + idle between two requests no longer. One with no request in flight for two + minutes is closed — an HTTP/2 one too, which is pinged and answers the + pings — and a response its peer does not read is given up on once a write + has stood still for a minute. - **What is logged has its secrets masked:** the values of `api_key`, `apikey`, - `token` and their like in any address, what follows `Bearer` or `Basic`, the + `token`, `auth`, a sign-in's `code`, `state` and `session_state`, TMDB's + `session_id`, `guest_session_id` and `request_token`, and their like in any + address, what follows `Bearer` or `Basic`, the password of an address, the configured keys and passwords, the keys kept for your own Sonarr and Radarr, and the addresses, tokens and signing secrets of notification destinations. A failed sign-in's username is logged quoted, its line breaks escaped. +- **Requests this server makes send no `Referer`,** which would hand the next + host of a redirect the address it came from, key included; and the providers + a key or a client id travels to — TMDB, Fanart.tv, TheTVDB, MyAnimeList — + are followed only within the origin first asked. - **An image URL may not name an address only this server can reach** — loopback, link-local, `169.254.169.254`. The NFO export downloads stored artwork, which makes an image URL a request this server makes on somebody else's say-so. diff --git a/frontend/e2e/oidc.serial.spec.ts b/frontend/e2e/oidc.serial.spec.ts index a330d3a..36b5d5c 100644 --- a/frontend/e2e/oidc.serial.spec.ts +++ b/frontend/e2e/oidc.serial.spec.ts @@ -302,7 +302,11 @@ test.describe('signing in through an identity provider', () => { provider.signInAs({ sub: `sub-${username}`, preferred_username: `${username}-there`, groups: [] }) await them.goto('/account') - await them.getByRole('link', { name: /tie my account|lier mon compte/i }).click() + // Tying asks for the password first: a session alone ties nothing. + await them.getByRole('button', { name: /tie my account|lier mon compte/i }).click() + const asked = them.getByRole('dialog') + await asked.getByLabel(/^(current password|mot de passe actuel)$/i).fill(password) + await asked.getByRole('button', { name: /tie my account|lier mon compte/i }).click() await them.waitForURL('**/account') await expect(them.getByText(/tied to the identity provider|lié au fournisseur d’identité/i)).toBeVisible() diff --git a/frontend/openapi.json b/frontend/openapi.json index 4a975b6..346faa0 100644 --- a/frontend/openapi.json +++ b/frontend/openapi.json @@ -217,7 +217,7 @@ "description": "A field was rejected" }, "403": { - "description": "The key limit is reached, or a key asked" + "description": "The key limit is reached, a key asked, or an administrator paused your keys" }, "409": { "description": "You already have a key by that name" @@ -288,6 +288,9 @@ "400": { "description": "A field was rejected" }, + "403": { + "description": "An administrator paused your keys, and only one can resume them" + }, "404": { "description": "Not one of your keys" }, @@ -423,6 +426,74 @@ } } }, + "/api/v1/account/provider": { + "post": { + "tags": [ + "Account" + ], + "summary": "Tie your account to the person the identity provider knows, so that you\ncan sign in through it as well.", + "description": "A session is not proof enough: one left open, or taken, would tie whoever\nholds it, and the tie would outlive the password the account's owner then\nresets. So the account's current password is asked for — or, for an account\nwith none, the provider is asked to have you sign in again, and its answer\nmust say you did. The answer is the provider's address to send the browser\nto; the tie is made when the browser comes back from it.", + "operationId": "oidc_tie", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/TieRequest" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Where to send the browser; the cookie of the flow is set", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/TieStarted" + } + } + } + }, + "400": { + "description": "The password was left out, or is too long" + }, + "401": { + "description": "The password was wrong" + }, + "403": { + "description": "A key cannot manage an account" + }, + "409": { + "description": "The account is tied already, or no provider is set up" + }, + "429": { + "description": "Too many wrong passwords lately" + }, + "502": { + "description": "The provider did not answer" + } + } + }, + "delete": { + "tags": [ + "Account" + ], + "summary": "Untie your account from the identity provider. Not one that would be left\nwith no way in: a password of its own, which this server asks for, comes\nfirst. Your other sessions are closed — one of them may have been opened\nthrough the tie.", + "operationId": "oidc_untie", + "responses": { + "204": { + "description": "Untied; the other sessions are closed" + }, + "403": { + "description": "A key cannot manage an account" + }, + "409": { + "description": "The account is not tied, or would be left with no way in" + } + } + } + }, "/api/v1/account/sessions": { "get": { "tags": [ @@ -1977,7 +2048,7 @@ "tags": [ "Session" ], - "summary": "Go to the identity provider to sign in — or, with `link`, to tie the\nsigned-in account to the person the provider knows.", + "summary": "Go to the identity provider to sign in. Tying an account to the person the\nprovider knows is asked for with `POST /account/provider`.", "operationId": "oidc_start", "parameters": [ { @@ -1992,7 +2063,7 @@ { "name": "link", "in": "query", - "description": "Set by a signed-in person tying their account to the provider.", + "description": "Refused: a tie is asked for from the account page, with\n`POST /account/provider`, which proves it by the account's password.", "required": false, "schema": { "type": "string" @@ -2425,6 +2496,7 @@ "Clients" ], "summary": "Enable or disable a key without revoking it.", + "description": "A person's key paused from here pauses its owner's keys: its owner can\nneither resume it nor make another, or resume any other of theirs, until an\nadministrator enables a key of theirs from here, or lifts the pause\n(`DELETE /users/{id}/keys-pause`).", "operationId": "clients_update", "parameters": [ { @@ -3486,6 +3558,9 @@ }, "400": { "description": "A filter was not one the list understands" + }, + "503": { + "description": "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again" } } } @@ -3577,6 +3652,9 @@ } } } + }, + "503": { + "description": "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again" } } } @@ -4254,6 +4332,9 @@ }, "401": { "description": "No valid credential was presented" + }, + "503": { + "description": "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again" } } }, @@ -8161,6 +8242,9 @@ }, "400": { "description": "The term is too long, a kind is not one of work, person and episode, or the language is not a language tag" + }, + "503": { + "description": "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again" } } } @@ -9043,6 +9127,34 @@ } } }, + "/api/v1/users/{id}/keys-pause": { + "delete": { + "tags": [ + "Users" + ], + "summary": "Lift the pause an administrator put on a person's keys — by disabling one\nof them from **Keys & network** — so that they can make keys and resume\nthem again. Enabling a key of theirs from there lifts it too; this is for\na person who has deleted theirs, with none left to enable.", + "operationId": "users_lift_keys_pause", + "parameters": [ + { + "name": "id", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "204": { + "description": "Lifted, or there was none" + }, + "403": { + "description": "The caller is not an administrator" + }, + "404": {} + } + } + }, "/api/v1/users/{id}/oidc": { "delete": { "tags": [ @@ -9077,7 +9189,7 @@ "tags": [ "Users" ], - "summary": "Give an account a new password; its sessions are closed.", + "summary": "Give an account a new password; its sessions are closed, and so is its tie\nto the identity provider, if it has one: whoever tied it — the person, or\nsomebody holding a session of theirs — signed in as them through the\nprovider, which no password reset would otherwise end.", "operationId": "users_reset_password", "parameters": [ { @@ -9742,11 +9854,14 @@ "404": { "description": "TMDB has no such picture at that size" }, + "429": { + "description": "The caller's quota of pictures is spent for now; `Retry-After` says when to ask again" + }, "502": { "description": "TMDB's pictures could not be reached" }, "503": { - "description": "The TMDB relay is switched off" + "description": "The TMDB relay is switched off, or as many pictures as are handed on from TMDB at once are under way (`Retry-After` says when to ask again)" } }, "security": [] @@ -24654,6 +24769,37 @@ } } }, + "TieRequest": { + "type": "object", + "properties": { + "currentPassword": { + "type": [ + "string", + "null" + ], + "description": "The account's current password: the proof, from an account that has\none, that its owner is asking and not a session left open. An account\nwith none leaves it out, and is sent to sign in at the provider again\ninstead." + }, + "next": { + "type": [ + "string", + "null" + ], + "description": "The page of this site to come back to." + } + } + }, + "TieStarted": { + "type": "object", + "required": [ + "url" + ], + "properties": { + "url": { + "type": "string", + "description": "Where to send the browser: the identity provider's sign-in. The\ncookie that carries the flow back is set with this answer." + } + } + }, "TimeOfDay": { "type": "object", "required": [ @@ -25117,6 +25263,13 @@ "items": { "$ref": "#/components/schemas/Session" } + }, + "keysPausedBy": { + "type": [ + "string", + "null" + ], + "description": "Who paused this person's keys, while the pause stands: until an\nadministrator lifts it ([`lift_keys_pause`]), they can neither make a\nkey nor resume one." } } }, diff --git a/frontend/src/components/Theatre.tsx b/frontend/src/components/Theatre.tsx index b61194a..883b2cc 100644 --- a/frontend/src/components/Theatre.tsx +++ b/frontend/src/components/Theatre.tsx @@ -171,6 +171,10 @@ export function Lightbox({ * no-cookie domain is the one this server's page policy lets frame, and the * frame sends YouTube this server's origin — which its player now refuses to * play without — and nothing more. + * + * The frame is sandboxed: YouTube's player runs, keeps its own origin's + * storage and goes full screen, and its links open in a window of their own; + * it can neither navigate this page, open a form, nor download anything. */ export function Trailer({ youtubeId, @@ -207,6 +211,7 @@ export function Trailer({ src={trailerEmbed(youtubeId)} title={heading} className="size-full" + sandbox="allow-scripts allow-same-origin allow-presentation allow-popups allow-popups-to-escape-sandbox" allow="autoplay; encrypted-media; picture-in-picture; fullscreen" allowFullScreen referrerPolicy="strict-origin" diff --git a/frontend/src/lib/api.gen.ts b/frontend/src/lib/api.gen.ts index 642d2c1..7832d71 100644 --- a/frontend/src/lib/api.gen.ts +++ b/frontend/src/lib/api.gen.ts @@ -203,6 +203,38 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/account/provider": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + /** + * Tie your account to the person the identity provider knows, so that you + * can sign in through it as well. + * @description A session is not proof enough: one left open, or taken, would tie whoever + * holds it, and the tie would outlive the password the account's owner then + * resets. So the account's current password is asked for — or, for an account + * with none, the provider is asked to have you sign in again, and its answer + * must say you did. The answer is the provider's address to send the browser + * to; the tie is made when the browser comes back from it. + */ + post: operations["oidc_tie"]; + /** + * Untie your account from the identity provider. Not one that would be left + * with no way in: a password of its own, which this server asks for, comes + * first. Your other sessions are closed — one of them may have been opened + * through the tie. + */ + delete: operations["oidc_untie"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/account/sessions": { parameters: { query?: never; @@ -1024,8 +1056,8 @@ export interface paths { cookie?: never; }; /** - * Go to the identity provider to sign in — or, with `link`, to tie the - * signed-in account to the person the provider knows. + * Go to the identity provider to sign in. Tying an account to the person the + * provider knows is asked for with `POST /account/provider`. */ get: operations["oidc_start"]; put?: never; @@ -1224,7 +1256,13 @@ export interface paths { delete: operations["clients_remove"]; options?: never; head?: never; - /** Enable or disable a key without revoking it. */ + /** + * Enable or disable a key without revoking it. + * @description A person's key paused from here pauses its owner's keys: its owner can + * neither resume it nor make another, or resume any other of theirs, until an + * administrator enables a key of theirs from here, or lifts the pause + * (`DELETE /users/{id}/keys-pause`). + */ patch: operations["clients_update"]; trace?: never; }; @@ -3672,6 +3710,28 @@ export interface paths { patch: operations["users_update"]; trace?: never; }; + "/api/v1/users/{id}/keys-pause": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + get?: never; + put?: never; + post?: never; + /** + * Lift the pause an administrator put on a person's keys — by disabling one + * of them from **Keys & network** — so that they can make keys and resume + * them again. Enabling a key of theirs from there lifts it too; this is for + * a person who has deleted theirs, with none left to enable. + */ + delete: operations["users_lift_keys_pause"]; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/users/{id}/oidc": { parameters: { query?: never; @@ -3702,7 +3762,12 @@ export interface paths { }; get?: never; put?: never; - /** Give an account a new password; its sessions are closed. */ + /** + * Give an account a new password; its sessions are closed, and so is its tie + * to the identity provider, if it has one: whoever tied it — the person, or + * somebody holding a session of theirs — signed in as them through the + * provider, which no password reset would otherwise end. + */ post: operations["users_reset_password"]; delete?: never; options?: never; @@ -9734,6 +9799,24 @@ export interface components { /** @description The issuer to try; the stored one when left out. */ issuer?: string | null; }; + TieRequest: { + /** + * @description The account's current password: the proof, from an account that has + * one, that its owner is asking and not a session left open. An account + * with none leaves it out, and is sent to sign in at the provider again + * instead. + */ + currentPassword?: string | null; + /** @description The page of this site to come back to. */ + next?: string | null; + }; + TieStarted: { + /** + * @description Where to send the browser: the identity provider's sign-in. The + * cookie that carries the flow back is set with this answer. + */ + url: string; + }; TimeOfDay: { /** Format: int32 */ hours: number; @@ -9925,6 +10008,12 @@ export interface components { user: components["schemas"]["User"]; keys: components["schemas"]["ApiClient"][]; sessions: components["schemas"]["Session"][]; + /** + * @description Who paused this person's keys, while the pause stands: until an + * administrator lifts it ([`lift_keys_pause`]), they can neither make a + * key nor resume one. + */ + keysPausedBy?: string | null; }; UsersPage: { users: components["schemas"]["Listed"][]; @@ -10623,7 +10712,7 @@ export interface operations { }; content?: never; }; - /** @description The key limit is reached, or a key asked */ + /** @description The key limit is reached, a key asked, or an administrator paused your keys */ 403: { headers: { [name: string]: unknown; @@ -10695,6 +10784,13 @@ export interface operations { }; content?: never; }; + /** @description An administrator paused your keys, and only one can resume them */ + 403: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; /** @description Not one of your keys */ 404: { headers: { @@ -10839,6 +10935,104 @@ export interface operations { }; }; }; + oidc_tie: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody: { + content: { + "application/json": components["schemas"]["TieRequest"]; + }; + }; + responses: { + /** @description Where to send the browser; the cookie of the flow is set */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["TieStarted"]; + }; + }; + /** @description The password was left out, or is too long */ + 400: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The password was wrong */ + 401: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description A key cannot manage an account */ + 403: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The account is tied already, or no provider is set up */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description Too many wrong passwords lately */ + 429: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The provider did not answer */ + 502: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; + oidc_untie: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Untied; the other sessions are closed */ + 204: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description A key cannot manage an account */ + 403: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The account is not tied, or would be left with no way in */ + 409: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; account_sessions: { parameters: { query?: never; @@ -12393,7 +12587,10 @@ export interface operations { query?: { /** @description The page of this site to come back to. */ next?: string; - /** @description Set by a signed-in person tying their account to the provider. */ + /** + * @description Refused: a tie is asked for from the account page, with + * `POST /account/provider`, which proves it by the account's password. + */ link?: string; }; header?: never; @@ -13611,6 +13808,13 @@ export interface operations { }; content?: never; }; + /** @description As many such reads as the server answers at once are under way; `Retry-After` says when to ask again */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; }; }; feeds_added_atom: { @@ -13697,6 +13901,13 @@ export interface operations { "application/json": components["schemas"]["Figures"]; }; }; + /** @description As many such reads as the server answers at once are under way; `Retry-After` says when to ask again */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; }; }; browse_genre: { @@ -14222,6 +14433,13 @@ export interface operations { }; content?: never; }; + /** @description As many such reads as the server answers at once are under way; `Retry-After` says when to ask again */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; }; }; items_create: { @@ -17819,6 +18037,13 @@ export interface operations { }; content?: never; }; + /** @description As many such reads as the server answers at once are under way; `Retry-After` says when to ask again */ + 503: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; }; }; seasons_chart: { @@ -18625,6 +18850,39 @@ export interface operations { }; }; }; + users_lift_keys_pause: { + parameters: { + query?: never; + header?: never; + path: { + id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Lifted, or there was none */ + 204: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + /** @description The caller is not an administrator */ + 403: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + 404: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; + }; + }; oidc_unlink: { parameters: { query?: never; @@ -19176,6 +19434,13 @@ export interface operations { }; content?: never; }; + /** @description The caller's quota of pictures is spent for now; `Retry-After` says when to ask again */ + 429: { + headers: { + [name: string]: unknown; + }; + content?: never; + }; /** @description TMDB's pictures could not be reached */ 502: { headers: { @@ -19183,7 +19448,7 @@ export interface operations { }; content?: never; }; - /** @description The TMDB relay is switched off */ + /** @description The TMDB relay is switched off, or as many pictures as are handed on from TMDB at once are under way (`Retry-After` says when to ask again) */ 503: { headers: { [name: string]: unknown; diff --git a/frontend/src/lib/lang/en.ts b/frontend/src/lib/lang/en.ts index fdfffd2..8e42393 100644 --- a/frontend/src/lib/lang/en.ts +++ b/frontend/src/lib/lang/en.ts @@ -1000,9 +1000,18 @@ export const en = { key: 'Your API key', keysLead: 'For a script, an app or a TMDB or TheTVDB client. A key acts in your name, with your rights and no more.', provider: 'Identity provider', - providerLead: 'Sign in through the identity provider as well as with your password: tie this account to the one you have there. You sign in there once to prove it is yours.', + providerLead: 'Sign in through the identity provider as well as with your password: tie this account to the one you have there. You give your password here, then sign in there, which shows both halves are yours.', + providerLeadNoPassword: 'Tie this account to the one you have at the identity provider. You sign in there again, whatever session it holds for you, which shows the account is yours.', providerTie: 'Tie my account', + providerTieTitle: 'Tie your account to the identity provider', + providerTieBody: 'Give your password to show that it is you asking, and not a session left open on this device. You are then sent to the identity provider to sign in.', + providerTieBodyNoPassword: 'This account has no password: the identity provider is asked to sign you in again, and the tie is made once it has.', providerTied: 'Tied to the identity provider: you can sign in through it.', + providerUntie: 'Untie…', + providerUntieTitle: 'Untie your account from the identity provider?', + providerUntieBody: 'You will sign in with your password only. Your other sessions are closed; this one stays open.', + providerUntieConfirm: 'Untie', + providerUntieNeedsPassword: 'This account has no password of its own, so it cannot be untied: it would have no way back in.', relayYes: 'Your keys read this catalogue’s API (/api/v1) and open the TMDB (/3) and TheTVDB (/v4) relays, for Jellyseerr, Yamtrack or Kodi.', relayNo: 'Your keys read this catalogue’s API (/api/v1). The relays are kept for the catalogue’s editors here.', keyCount: (used: number, limit: number) => `${used} of ${limit}`, @@ -4022,6 +4031,7 @@ export const en = { 'dataset.imported': 'Dataset downloaded', 'backup.deleted': 'Archive deleted', 'backup.restored': 'Archive restored', + 'backup.downloaded': 'Archive downloaded', 'item.numbering_fallback': 'Numbered by TMDB in TheTVDB’s place', 'item.numbering_restored': 'Numbering given back to TheTVDB', 'order.created': 'Order made', diff --git a/frontend/src/lib/lang/fr.ts b/frontend/src/lib/lang/fr.ts index 0c430bd..3c8b29f 100644 --- a/frontend/src/lib/lang/fr.ts +++ b/frontend/src/lib/lang/fr.ts @@ -993,9 +993,18 @@ export const fr: Dict = { key: 'Votre clé API', keysLead: 'Pour un script, une application ou un client TMDB ou TheTVDB. Une clé agit en votre nom, avec vos droits et pas davantage.', provider: 'Fournisseur d’identité', - providerLead: 'Se connecter par le fournisseur d’identité en plus du mot de passe : liez ce compte à celui que vous avez là-bas. Vous vous y connectez une fois pour prouver qu’il est à vous.', + providerLead: 'Se connecter par le fournisseur d’identité en plus du mot de passe : liez ce compte à celui que vous avez là-bas. Vous donnez votre mot de passe ici, puis vous vous connectez là-bas : les deux moitiés sont à vous.', + providerLeadNoPassword: 'Liez ce compte à celui que vous avez chez le fournisseur d’identité. Vous vous y reconnectez, quelle que soit la session qu’il garde pour vous : c’est ce qui prouve que le compte est à vous.', providerTie: 'Lier mon compte', + providerTieTitle: 'Lier votre compte au fournisseur d’identité', + providerTieBody: 'Donnez votre mot de passe pour montrer que c’est bien vous, et non une session restée ouverte sur cet appareil. Vous êtes ensuite envoyé chez le fournisseur d’identité pour vous y connecter.', + providerTieBodyNoPassword: 'Ce compte n’a pas de mot de passe : le fournisseur d’identité est prié de vous reconnecter, et le lien n’est fait qu’une fois que c’est fait.', providerTied: 'Lié au fournisseur d’identité : vous pouvez vous connecter par lui.', + providerUntie: 'Délier…', + providerUntieTitle: 'Délier votre compte du fournisseur d’identité ?', + providerUntieBody: 'Vous ne vous connecterez plus qu’avec votre mot de passe. Vos autres sessions sont fermées ; celle-ci reste ouverte.', + providerUntieConfirm: 'Délier', + providerUntieNeedsPassword: 'Ce compte n’a pas de mot de passe à lui : il ne peut pas être délié, il n’aurait plus aucun moyen de se reconnecter.', relayYes: 'Vos clés lisent l’API de ce catalogue (/api/v1) et ouvrent les relais TMDB (/3) et TheTVDB (/v4), pour Jellyseerr, Yamtrack ou Kodi.', relayNo: 'Vos clés lisent l’API de ce catalogue (/api/v1). Les relais sont réservés ici aux éditeurs du catalogue.', keyCount: (used: number, limit: number) => `${used} sur ${limit}`, @@ -4023,6 +4032,7 @@ export const fr: Dict = { 'dataset.imported': 'Jeu de données téléchargé', 'backup.deleted': 'Archive supprimée', 'backup.restored': 'Archive restaurée', + 'backup.downloaded': 'Archive téléchargée', 'item.numbering_fallback': 'Numérotée par TMDB à la place de TheTVDB', 'item.numbering_restored': 'Numérotation rendue à TheTVDB', 'order.created': 'Ordre fait', diff --git a/frontend/src/routes/Account.tsx b/frontend/src/routes/Account.tsx index 38e950e..dfb3653 100644 --- a/frontend/src/routes/Account.tsx +++ b/frontend/src/routes/Account.tsx @@ -21,7 +21,6 @@ import { RoleChip } from '../components/account/people' import { SecretReveal } from '../components/account/SecretReveal' import { Button, - ButtonLink, Chip, Dialog, EmptyState, @@ -110,11 +109,16 @@ export function Account() { * Tying this account to the one its owner has at the identity provider: a * sign-in there, started from here while signed in here, so both sides are * proved by the same person. Offered only when a provider is set up. + * + * A session alone is not asked to tie: one left open, or taken, would tie + * whoever holds it. The password is given first — an account without one is + * sent to sign in at the provider again instead — and the tie is made when + * the browser comes back. Untied by its owner, who keeps their password. */ function Provider({ me }: { me: User }) { const { t } = useI18n() const options = useAuthOptions() - const { pathname } = useLocation() + const [asking, setAsking] = useState<'tie' | 'untie' | null>(null) if (!options.data?.oidc && !me.oidcLinked) return null @@ -123,24 +127,145 @@ function Provider({ me }: { me: User }) {
{me.oidcLinked ? ( -

- - {t.account.providerTied} -

+ <> +

+ + {t.account.providerTied} +

+ {me.hasPassword ? ( + + ) : ( +

{t.account.providerUntieNeedsPassword}

+ )} + ) : ( <> -

{t.account.providerLead}

- +

+ {me.hasPassword ? t.account.providerLead : t.account.providerLeadNoPassword} +

+ )}
+ {asking === 'tie' ? setAsking(null)} /> : null} + {asking === 'untie' ? setAsking(null)} /> : null} ) } +/** The password, and then the provider's own sign-in. */ +function TieProvider({ me, onClose }: { me: User; onClose: () => void }) { + const { t } = useI18n() + const { pathname } = useLocation() + const [password, setPassword] = useState('') + + const tie = useMutation({ + mutationFn: () => + api.post<{ url: string }>('/account/provider', { + ...(me.hasPassword ? { currentPassword: password } : {}), + next: pathname, + }), + // To the provider, and back here once it has signed them in. + onSuccess: ({ url }) => window.location.assign(url), + }) + + const missing = me.hasPassword && password === '' + const wrong = tie.error instanceof ApiError && tie.error.isUnauthorized + const waiting = tie.isPending || tie.isSuccess + const refusal = tie.isError ? (wrong ? t.admin.config.passwordWrong : tie.error.message) : undefined + + return ( + + + + + } + > +
{ + event.preventDefault() + if (!missing && !waiting) tie.mutate() + }} + > +

+ {me.hasPassword ? t.account.providerTieBody : t.account.providerTieBodyNoPassword} +

+ {me.hasPassword ? ( + + setPassword(event.target.value)} + /> + + ) : refusal ? ( +

+ {refusal} +

+ ) : null} +
+ + ) +} + +/** Asked first: what is untied is not tied again without the password. */ +function UntieProvider({ onClose }: { onClose: () => void }) { + const { t } = useI18n() + const queryClient = useQueryClient() + + const untie = useMutation({ + mutationFn: () => api.delete('/account/provider'), + onSuccess: () => { + void queryClient.invalidateQueries({ queryKey: ['account'] }) + void queryClient.invalidateQueries({ queryKey: ['me'] }) + onClose() + }, + }) + + return ( + + + + + } + > +

{t.account.providerUntieBody}

+ {untie.isError ? ( +

+ {untie.error.message} +

+ ) : null} + + ) +} + /* ── Profile ──────────────────────────────────────────────────────────────── */ function Profile({ me }: { me: User }) { @@ -686,9 +811,11 @@ function KeyList({ > {confirm?.what === 'rotate' ? t.account.regenerateBody : t.account.revokeBody} - {rotate.isError || revoke.isError ? ( + {rotate.isError || revoke.isError || toggle.isError ? (

- {(rotate.error ?? revoke.error) instanceof ApiError ? (rotate.error ?? revoke.error)?.message : null} + {(rotate.error ?? revoke.error ?? toggle.error) instanceof ApiError + ? (rotate.error ?? revoke.error ?? toggle.error)?.message + : null}

) : null} diff --git a/migrations/postgres/0053_season_order_tvdb_index.sql b/migrations/postgres/0053_season_order_tvdb_index.sql new file mode 100644 index 0000000..bf00d27 --- /dev/null +++ b/migrations/postgres/0053_season_order_tvdb_index.sql @@ -0,0 +1,6 @@ +-- The TheTVDB relay finds the work that holds a season, or an episode an +-- order keeps, by TheTVDB's id of it, as it finds an episode (0026): a number +-- nobody holds read every season and every episode of every order, on each +-- request. Partial, as there: what is made by hand has no such id. +CREATE INDEX ix_media_season_tvdb ON media_season (tvdb_id) WHERE tvdb_id IS NOT NULL; +CREATE INDEX ix_order_episode_tvdb ON media_order_episode (tvdb_episode_id) WHERE tvdb_episode_id IS NOT NULL; diff --git a/migrations/sqlite/0053_season_order_tvdb_index.sql b/migrations/sqlite/0053_season_order_tvdb_index.sql new file mode 100644 index 0000000..bf00d27 --- /dev/null +++ b/migrations/sqlite/0053_season_order_tvdb_index.sql @@ -0,0 +1,6 @@ +-- The TheTVDB relay finds the work that holds a season, or an episode an +-- order keeps, by TheTVDB's id of it, as it finds an episode (0026): a number +-- nobody holds read every season and every episode of every order, on each +-- request. Partial, as there: what is made by hand has no such id. +CREATE INDEX ix_media_season_tvdb ON media_season (tvdb_id) WHERE tvdb_id IS NOT NULL; +CREATE INDEX ix_order_episode_tvdb ON media_order_episode (tvdb_episode_id) WHERE tvdb_episode_id IS NOT NULL; diff --git a/scripts/watch/common.py b/scripts/watch/common.py index ecdfb47..6df1c11 100644 --- a/scripts/watch/common.py +++ b/scripts/watch/common.py @@ -101,6 +101,7 @@ def _closed(url, data, headers, method, timeout, limit): """One request on a connection of its own, as urllib makes them.""" request = urllib.request.Request(url, data=data, headers=headers, method=method) try: + # nosemgrep: python.lang.security.audit.dynamic-urllib-use-detected.dynamic-urllib-use-detected — the URLs are the checks' own https constants with urllib.request.urlopen(request, timeout=timeout) as response: body = response.read(limit) if limit else response.read() return response.status, response.headers, _decoded(body, response.headers, limit) @@ -145,8 +146,10 @@ def _kept(url, data, headers, method, timeout, limit): def _connect(host, port, timeout): proxy = urllib.request.getproxies().get("https") if not proxy or urllib.request.proxy_bypass(host): + # nosemgrep: python.lang.security.audit.httpsconnection-detected.httpsconnection-detected — http.client verifies the certificate and the host name return http.client.HTTPSConnection(host, port, timeout=timeout) via = urllib.parse.urlsplit(proxy if "://" in proxy else f"http://{proxy}") + # nosemgrep: python.lang.security.audit.httpsconnection-detected.httpsconnection-detected — http.client verifies the certificate and the host name connection = http.client.HTTPSConnection(via.hostname, via.port or 80, timeout=timeout) tunnel = {} if via.username: diff --git a/src/api/anilist.rs b/src/api/anilist.rs index 638dd77..d0e09dd 100644 --- a/src/api/anilist.rs +++ b/src/api/anilist.rs @@ -86,6 +86,12 @@ const COLLECTIONS: &[&str] = &["GenreCollection", "MediaTagCollection"]; /// What a relayed request keeps of the caller's headers: what describes it. const ASKED_WITH: &[&str] = &["user-agent", "accept", "accept-language", "content-type"]; +/// Of those, the headers that may change AniList's answer: what a kept answer +/// is told apart by besides the query — the type a body is posted as says how +/// AniList reads it — as it travels asked with them, an answer filed without +/// them would be handed to a caller that asked otherwise. +const VARIES_BY: &[&str] = &["accept", "accept-language", "content-type"]; + /// How many entries of one answer are looked up here, in the order the /// answer lists them. A page is fifty at most; a user's whole list, in one /// answer, is not patched beyond this. @@ -258,7 +264,7 @@ pub async fn relay(State(state): State, request: Request) -> AppResult .is_some_and(|query| !has_mutation(query) && !is_personal(query)); let ask = relay::Ask { service: relay::Service::Anilist, - key: anybodys.then(|| relay_key(&state, &parts.method, &parts.uri, &body)), + key: anybodys.then(|| relay_key(&state, &parts.method, &parts.uri, &parts.headers, &body)), path: if document.as_deref().is_some_and(asks_collections_alone) { cache::ANILIST_COLLECTIONS } else { @@ -535,13 +541,23 @@ fn is_personal(document: &str) -> bool { } /// What a relayed answer is filed under: the query asked — its text, its -/// variables, whatever else the body says — digested, under the generation. -fn relay_key(state: &AppState, method: &Method, uri: &Uri, body: &[u8]) -> String { +/// variables, whatever else the body says — and the caller's headers that +/// travel and may change AniList's answer ([`relay::varies_by`]), digested, +/// under the generation. +fn relay_key( + state: &AppState, + method: &Method, + uri: &Uri, + headers: &HeaderMap, + body: &[u8], +) -> String { let mut digest = Sha256::new(); digest.update(method.as_str().as_bytes()); digest.update(b"\n"); digest.update(relay::sorted_query(uri, relay::OUR_PARAMS).as_bytes()); digest.update(b"\n"); + digest.update(relay::varies_by(headers, VARIES_BY).as_bytes()); + digest.update(b"\n"); digest.update(body); let digest: String = digest .finalize() @@ -2169,6 +2185,17 @@ mod tests { /// What the relay answers `query`, asked with somebody's own AniList /// token when there is one. async fn relayed(state: &AppState, query: &str, token: Option<&str>) -> Response { + relayed_with(state, query, token, &[]).await + } + + /// The same, with headers of the caller's own, which take the place of + /// the ones a client posts a query with. + async fn relayed_with( + state: &AppState, + query: &str, + token: Option<&str>, + headers: &[(header::HeaderName, &str)], + ) -> Response { let mut request = axum::http::Request::builder() .method(Method::POST) .uri("/") @@ -2177,7 +2204,12 @@ mod tests { if let Some(token) = token { request = request.header(header::AUTHORIZATION, format!("Bearer {token}")); } - let request = request.body(Body::from(posted(query))).unwrap(); + let mut request = request.body(Body::from(posted(query))).unwrap(); + for (name, value) in headers { + request + .headers_mut() + .insert(name.clone(), HeaderValue::from_str(value).unwrap()); + } match relay(State(state.clone()), request).await { Ok(response) => response, Err(error) => error.into_response(), @@ -2195,22 +2227,91 @@ mod tests { response.headers().get(name)?.to_str().ok()?.parse().ok() } + /// The headers a client posts a query with, as [`relayed`] does. + fn posting() -> HeaderMap { + let mut headers = HeaderMap::new(); + headers.insert( + header::CONTENT_TYPE, + HeaderValue::from_static("application/json"), + ); + headers + } + /// What the relay space holds for `query`. async fn kept(state: &AppState, query: &str) -> Option { - let key = relay_key(state, &Method::POST, &Uri::from_static("/"), &posted(query)); + let key = relay_key( + state, + &Method::POST, + &Uri::from_static("/"), + &posting(), + &posted(query), + ); state.caches.relay.get(&key).await } /// What was kept for `query`, made `seconds` old — as if AniList had /// answered it then. async fn aged(state: &AppState, query: &str, seconds: u64) { - let key = relay_key(state, &Method::POST, &Uri::from_static("/"), &posted(query)); + let key = relay_key( + state, + &Method::POST, + &Uri::from_static("/"), + &posting(), + &posted(query), + ); let mut kept = state.caches.relay.get(&key).await.expect("kept"); kept.expires_at = cache::now_secs() + cache::relay_ttl(cache::ANILIST_QUERY).as_secs() - seconds; state.caches.relay.insert(key, kept).await; } + /// What AniList answered is kept for the callers who asked as its first + /// did — by the headers that travel to it and may change its answer: the + /// type a body is posted as says how AniList reads it — and not for + /// those who asked otherwise. + #[tokio::test] + async fn an_answer_is_kept_for_the_callers_who_asked_as_its_first_did() { + const FIRST: &str = r#"{"data":{"Media":{"id":9400001,"title":{"romaji":"First"}}}}"#; + const OTHER: &str = r#"{"data":{"Media":{"id":9400001,"title":{"romaji":"Other"}}}}"#; + let (at, asked) = stand_in(vec![reply("200 OK", FIRST), reply("200 OK", OTHER)]).await; + let state = server(&at).await; + let query = "{ Media(id: 9400001) { id title { romaji } } }"; + + assert_eq!(text(relayed(&state, query, None).await).await, FIRST); + assert_eq!(text(relayed(&state, query, None).await).await, FIRST); + assert_eq!(asked.load(Ordering::SeqCst), 1, "kept for the same caller"); + + // Another language: another entry, asked of AniList; and then kept. + let german = [(header::ACCEPT_LANGUAGE, "de")]; + assert_eq!( + text(relayed_with(&state, query, None, &german).await).await, + OTHER + ); + assert_eq!(asked.load(Ordering::SeqCst), 2); + assert_eq!( + text(relayed_with(&state, query, None, &german).await).await, + OTHER + ); + assert_eq!(asked.load(Ordering::SeqCst), 2); + // The first caller's is as it was. + assert_eq!(text(relayed(&state, query, None).await).await, FIRST); + assert_eq!(asked.load(Ordering::SeqCst), 2); + + // The same body posted as another type is another request to AniList, + // and another entry here. + let graphql = [(header::CONTENT_TYPE, "application/graphql")]; + let _ = relayed_with(&state, query, None, &graphql).await; + assert_eq!(asked.load(Ordering::SeqCst), 3); + let _ = relayed_with(&state, query, None, &graphql).await; + assert_eq!(asked.load(Ordering::SeqCst), 3, "and then kept"); + // And what does not change AniList's answer is not another entry. + let agent = [(header::USER_AGENT, "Yamtrack/1.0")]; + assert_eq!( + text(relayed_with(&state, query, None, &agent).await).await, + FIRST + ); + } + #[tokio::test] async fn when_anilist_fails_its_last_answer_stands_in_for_a_day() { const KEPT: &str = r#"{"data":{"Media":{"id":9300001,"title":{"romaji":"Kept"}}}}"#; diff --git a/src/api/audit.rs b/src/api/audit.rs index b849e43..812efb9 100644 --- a/src/api/audit.rs +++ b/src/api/audit.rs @@ -1,17 +1,25 @@ //! Recording and reading the audit trail. +use std::{ + collections::HashMap, + net::IpAddr, + sync::LazyLock, + time::{Duration, Instant}, +}; + use axum::{ Extension, Json, extract::{Query as AxumQuery, State}, response::{IntoResponse, Response}, }; +use parking_lot::Mutex; use serde::{Deserialize, Serialize}; use utoipa::{IntoParams, ToSchema}; use utoipa_axum::{router::OpenApiRouter, routes}; use crate::{ api::extract::ClientIp, - auth::Identity, + auth::{Identity, ratelimit::network_of}, db::repo::{ self, audit::{Action, Entry}, @@ -35,13 +43,18 @@ pub struct Event<'a> { /// refusing it after the fact because the trail could not be written would be /// worse than an incomplete trail. pub async fn record(state: &AppState, event: Event<'_>) { + record_row(state, event).await; +} + +/// [`record`], and the id of the row it wrote — nothing when it could not. +async fn record_row(state: &AppState, event: Event<'_>) -> Option { let actor = event.identity.map(Identity::label); let ip = event.ip.as_text(); let target = event.target.map(clip); let detail = event.detail.map(clip); - let result = repo::audit::record( + let result = repo::audit::record_returning( &state.db, repo::audit::Record { actor: actor.as_deref(), @@ -53,9 +66,13 @@ pub async fn record(state: &AppState, event: Event<'_>) { ) .await; - if let Err(e) = result { - tracing::warn!(action = %event.action, error = %e, "could not write the audit entry"); - } + let row = match result { + Ok(id) => Some(id), + Err(e) => { + tracing::warn!(action = %event.action, error = %e, "could not write the audit entry"); + None + } + }; // A lock, an edit, a sync: told to the operator's own Sonarr or Radarr // too, when that is asked for. @@ -73,6 +90,8 @@ pub async fn record(state: &AppState, event: Event<'_>) { }, ) .await; + + row } /// Who the trail says did what this server does of its own accord. @@ -113,6 +132,221 @@ pub async fn record_system(state: &AppState, action: Action, target: &str, detai .await; } +// ─── failures of somebody with no credential ───────────────────────────────── + +/// What kind of failure somebody with no credential made, for the trail to +/// gather them by. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Failure { + /// A password that was refused at the sign-in form. + Password, + /// A return from the identity provider that was refused. + Provider, +} + +/// How long the failures of one kind from one network are gathered into one +/// row. +const GATHERED_FOR: Duration = Duration::from_secs(60); + +/// The most networks gathered at once. The addresses come from whoever can +/// reach the port; past this the rest are gathered with the one every address +/// that cannot be told shares, so that the rows written stay bounded however +/// many addresses there are. +const GATHERED_MOST: usize = 4096; + +/// What one network's failures of one kind have come to in the window open. +struct Gathered { + opened: Instant, + /// The row the first of them was recorded in, to say how many there were + /// when the window closes — none when it could not be written. + row: Option, + /// Failures since the first, which was recorded at once: counted here, + /// and told together when the window closes. + held: u32, + /// The name and the note of the latest of them, for the row to read by. + last: (Option, Option), + /// Whether a task is waiting for the window to close to tell them. + waiting: bool, +} + +static GATHERING: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + +/// Record a failed sign-in of somebody who came with no credential — a +/// refused password, a refused return from the identity provider — and gather +/// the ones that follow from the same network ([`crate::auth::ip::bucket`]) +/// in the next minute into that one row, which says how many there were. +/// +/// What anybody can cause by sending requests has to be bounded by who sends +/// them, not by how many: the sign-in form is reachable with no credential, +/// and each failure was a row, a notification and a line of the journal's +/// CSV, six hundred a minute from one address. The first of a run is +/// recorded at once, as it always was — it is the one somebody is looking +/// for, and a slip of one's own fingers is then a row, as before — and when +/// the minute is over the row is amended to carry the number of the run and +/// what the last of them typed. A minute with only the one stays as it was. +pub async fn record_failure( + state: &AppState, + kind: Failure, + ip: &ClientIp, + target: Option<&str>, + detail: Option<&str>, +) { + gather(state, kind, ip, target, detail, GATHERED_FOR).await; +} + +async fn gather( + state: &AppState, + kind: Failure, + ip: &ClientIp, + target: Option<&str>, + detail: Option<&str>, + window: Duration, +) { + let (noted, key) = note( + &mut GATHERING.lock(), + (network_of(ip.0), kind), + Instant::now(), + window, + GATHERED_MOST, + (target, detail), + ); + + match noted { + Noted::First => { + let row = record_row( + state, + Event { + identity: None, + ip, + action: Action::SignInFailed, + target, + detail, + }, + ) + .await; + if let Some(gathered) = GATHERING.lock().get_mut(&key) { + gathered.row = row; + } + } + Noted::Held => {} + Noted::HeldUntil(wait) => { + let (state, ip) = (state.clone(), *ip); + tokio::spawn(async move { + tokio::time::sleep(wait).await; + let Some(closed) = GATHERING.lock().remove(&key) else { + return; + }; + if closed.held > 0 { + tell(&state, &ip, closed, window).await; + } + }); + } + } +} + +/// Say how many there were of a run that is over: in the row of the first, +/// which stood for one until now; in a row of its own, for the failures after +/// the first, when that row could not be written. +async fn tell(state: &AppState, ip: &ClientIp, run: Gathered, window: Duration) { + let (target, last) = run.last; + // What the last of them typed, or what the provider said of it. + let last = last.or(target.clone()); + let seconds = window.as_secs().max(1); + let after = |last: Option| { + last.map(|last| format!(" · the last: {last}")) + .unwrap_or_default() + }; + + if let Some(row) = &run.row { + let said = clip(&format!( + "{} in all within {seconds} s{}", + run.held.saturating_add(1), + after(last) + )); + match repo::audit::amend_detail(&state.db, row, &said).await { + // Taken by the retention meanwhile: nothing to say it in. + Ok(_) => {} + Err(e) => tracing::warn!(error = %e, "could not amend the audit entry"), + } + return; + } + + let said = format!("{} more within {seconds} s{}", run.held, after(last)); + record( + state, + Event { + identity: None, + ip, + action: Action::SignInFailed, + target: target.as_deref(), + detail: Some(&said), + }, + ) + .await; +} + +/// What a failure is to the trail. +#[derive(Debug, PartialEq, Eq)] +enum Noted { + /// The first of a run: recorded now. + First, + /// One more in a run, to be recorded with the rest. + Held, + /// The first one more of a run: held, and somebody is to wait this long + /// for the window to close and record what has been held. + HeldUntil(Duration), +} + +/// Note a failure under `key` — the network and the kind, or the shared +/// place for every network at once when `most` of them are already being +/// gathered, which a flood of addresses is the only way to reach. Returns +/// what it is, and the key it was noted under. +fn note( + gathering: &mut HashMap<(IpAddr, Failure), Gathered>, + mut key: (IpAddr, Failure), + now: Instant, + window: Duration, + most: usize, + (target, detail): (Option<&str>, Option<&str>), +) -> (Noted, (IpAddr, Failure)) { + if !gathering.contains_key(&key) && gathering.len() >= most { + gathering.retain(|_, g| g.waiting || now.duration_since(g.opened) < window); + if gathering.len() >= most { + key.0 = network_of(None); + } + } + + // An entry somebody waits to record is held to, a moment past its window + // too: replaced, what it held would be lost. + let noted = match gathering.get_mut(&key) { + Some(g) if g.waiting || now.duration_since(g.opened) < window => { + g.held = g.held.saturating_add(1); + g.last = (target.map(str::to_string), detail.map(str::to_string)); + if g.waiting { + Noted::Held + } else { + g.waiting = true; + Noted::HeldUntil(window.saturating_sub(now.duration_since(g.opened))) + } + } + _ => { + gathering.insert( + key, + Gathered { + opened: now, + row: None, + held: 0, + last: (None, None), + waiting: false, + }, + ); + Noted::First + } + }; + (noted, key) +} + /// The most of any one field this trail will hold. /// /// Some of what is recorded comes from whoever is calling — the username of a @@ -537,6 +771,7 @@ pub(crate) fn cell_of(value: Option<&serde_json::Value>) -> String { #[cfg(test)] mod tests { use super::*; + use crate::auth::testing::client_at; #[test] fn every_action_has_a_distinct_name() { @@ -783,6 +1018,243 @@ mod tests { } } + fn at(address: &str) -> (IpAddr, Failure) { + (address.parse().unwrap(), Failure::Password) + } + + #[test] + fn a_run_from_one_network_is_recorded_once_and_held_for_a_row_that_counts() { + let mut gathering = HashMap::new(); + let window = Duration::from_secs(60); + let start = Instant::now(); + let fail = |gathering: &mut _, key, after: u64, target: &str| { + note( + gathering, + key, + start + Duration::from_secs(after), + window, + 8, + (Some(target), None), + ) + }; + + let key = at("192.0.2.1"); + assert_eq!(fail(&mut gathering, key, 0, "a").0, Noted::First); + // The first held is told to be waited for; the rest only counted. + assert_eq!( + fail(&mut gathering, key, 10, "b").0, + Noted::HeldUntil(Duration::from_secs(50)) + ); + assert_eq!(fail(&mut gathering, key, 20, "c").0, Noted::Held); + let held = &gathering[&key]; + assert_eq!(held.held, 2); + assert_eq!(held.last.0.as_deref(), Some("c")); + + // Another kind from the same address, and another network, are runs of + // their own; an IPv6 network is one. + assert_eq!( + fail(&mut gathering, (key.0, Failure::Provider), 20, "x").0, + Noted::First + ); + assert_eq!( + fail(&mut gathering, at("192.0.2.2"), 20, "x").0, + Noted::First + ); + let v6 = |tail: &str| at(&format!("2001:db8:1:2::{tail}")); + let bucket = |(ip, kind): (IpAddr, Failure)| (crate::auth::ip::bucket(ip), kind); + assert_eq!( + fail(&mut gathering, bucket(v6("1")), 20, "x").0, + Noted::First + ); + assert!(matches!( + fail(&mut gathering, bucket(v6("2")), 21, "x").0, + Noted::HeldUntil(_) + )); + + // Once the flush has taken what was held, the window is another. + gathering.remove(&key); + assert_eq!(fail(&mut gathering, key, 70, "d").0, Noted::First); + // A window that closed and was not flushed yet still takes what comes. + assert_eq!( + fail(&mut gathering, key, 80, "e").0, + Noted::HeldUntil(Duration::from_secs(50)) + ); + assert_eq!(fail(&mut gathering, key, 200, "f").0, Noted::Held); + assert_eq!(gathering[&key].held, 2); + } + + /// Past what is gathered at once, the networks beyond are gathered as one. + #[test] + fn a_flood_of_addresses_is_gathered_under_one_key() { + let mut gathering = HashMap::new(); + let window = Duration::from_secs(60); + let now = Instant::now(); + + for n in 0..4 { + let key = at(&format!("192.0.2.{n}")); + let (noted, used) = note(&mut gathering, key, now, window, 4, (None, None)); + assert_eq!((noted, used), (Noted::First, key)); + } + // The fifth address is nobody's in particular, and so is the sixth: + // one run between them. + let (first, used) = note( + &mut gathering, + at("198.51.100.1"), + now, + window, + 4, + (None, None), + ); + assert_eq!(first, Noted::First); + assert_eq!(used, at("0.0.0.0")); + let (second, used) = note( + &mut gathering, + at("198.51.100.2"), + now, + window, + 4, + (None, None), + ); + assert!(matches!(second, Noted::HeldUntil(_))); + assert_eq!(used, at("0.0.0.0")); + assert_eq!(gathering.len(), 5); + + // Windows that closed make room again. + let later = now + window; + let (noted, used) = note( + &mut gathering, + at("198.51.100.3"), + later, + window, + 4, + (None, None), + ); + assert_eq!((noted, used), (Noted::First, at("198.51.100.3"))); + } + + /// The failed sign-ins of the trail, the oldest first. + async fn failed_sign_ins(state: &AppState) -> Vec { + let mut rows = repo::audit::list( + &state.db, + &repo::audit::Query { + action: Some(Action::SignInFailed.as_str().into()), + limit: 100, + ..Default::default() + }, + ) + .await + .unwrap(); + rows.reverse(); + rows + } + + /// What the sign-in form is flooded with is one row, written at once, and + /// said to be many when the minute is over — whatever the flood. + #[tokio::test] + async fn a_flood_of_failed_sign_ins_is_one_row_that_carries_the_count() { + let state = crate::api::native::testing::server().await; + let window = Duration::from_millis(300); + let ip = client_at(&state, "192.0.2.150").await; + + for n in 0..8 { + let typed = format!("guess-{n}"); + gather(&state, Failure::Password, &ip, Some(&typed), None, window).await; + } + let at_once = failed_sign_ins(&state).await; + assert_eq!(at_once.len(), 1); + assert_eq!(at_once[0].target.as_deref(), Some("guess-0")); + assert_eq!(at_once[0].ip.as_deref(), Some("192.0.2.150")); + assert_eq!(at_once[0].detail, None); + + // The window closes, and the one row says how many it stood for. + let mut rows = at_once; + for _ in 0..200 { + if rows[0].detail.is_some() { + break; + } + tokio::time::sleep(Duration::from_millis(50)).await; + rows = failed_sign_ins(&state).await; + } + assert_eq!(rows.len(), 1, "{rows:?}"); + assert_eq!(rows[0].target.as_deref(), Some("guess-0")); + let said = rows[0].detail.as_deref().unwrap_or_default(); + assert_eq!(said, "8 in all within 1 s · the last: guess-7"); + + // A failure after that is the first of another run, recorded at once. + gather(&state, Failure::Password, &ip, Some("again"), None, window).await; + assert_eq!(failed_sign_ins(&state).await.len(), 2); + } + + /// A run of one stays as it was: a row, and nothing said of it later. + #[tokio::test] + async fn a_single_failure_is_one_row_as_it_always_was() { + let state = crate::api::native::testing::server().await; + let window = Duration::from_millis(100); + let ip = client_at(&state, "192.0.2.170").await; + + gather(&state, Failure::Password, &ip, Some("slip"), None, window).await; + tokio::time::sleep(Duration::from_millis(400)).await; + + let rows = failed_sign_ins(&state).await; + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].target.as_deref(), Some("slip")); + assert_eq!(rows[0].detail, None); + } + + /// Where the first row could not be written there is none to amend: what + /// followed it is told in a row of its own. + #[tokio::test] + async fn a_run_whose_first_row_is_missing_is_told_in_a_row_of_its_own() { + let state = crate::api::native::testing::server().await; + let ip = client_at(&state, "192.0.2.180").await; + let run = Gathered { + opened: Instant::now(), + row: None, + held: 3, + last: (Some("guess-3".into()), None), + waiting: true, + }; + + tell(&state, &ip, run, Duration::from_secs(60)).await; + + let rows = failed_sign_ins(&state).await; + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].target.as_deref(), Some("guess-3")); + assert_eq!( + rows[0].detail.as_deref(), + Some("3 more within 60 s · the last: guess-3") + ); + } + + /// One address's failures do not hold another's back. + #[tokio::test] + async fn each_network_and_kind_is_recorded_at_its_own_first() { + let state = crate::api::native::testing::server().await; + let window = Duration::from_secs(60); + let one = client_at(&state, "192.0.2.160").await; + let other = client_at(&state, "192.0.2.161").await; + + gather(&state, Failure::Password, &one, Some("a"), None, window).await; + gather(&state, Failure::Password, &other, Some("b"), None, window).await; + gather( + &state, + Failure::Provider, + &one, + None, + Some("expired"), + window, + ) + .await; + gather(&state, Failure::Password, &one, Some("c"), None, window).await; + + let rows = failed_sign_ins(&state).await; + assert_eq!( + rows.iter().map(|r| r.target.as_deref()).collect::>(), + [Some("a"), Some("b"), None], + "the fourth is held for the row that counts" + ); + } + #[test] fn action_names_are_namespaced() { // A log shipper filters on the prefix, so every name must carry one. diff --git a/src/api/mod.rs b/src/api/mod.rs index 95ec515..4789f41 100644 --- a/src/api/mod.rs +++ b/src/api/mod.rs @@ -26,7 +26,7 @@ pub mod tmdb_images; use axum::{ Router, extract::Request, - http::StatusCode, + http::{HeaderValue, StatusCode, header}, middleware::from_fn_with_state, response::{IntoResponse, Response}, }; @@ -36,7 +36,29 @@ use utoipa::{ }; use utoipa_axum::router::OpenApiRouter; -use crate::{auth::middleware as guards, state::AppState}; +use crate::{auth::middleware as guards, error::AppError, state::AppState}; + +/// Seconds a caller turned away from work that takes a place +/// ([`crate::cache::Gate`]) is told to wait: past those it waited for one, +/// and long enough for what held them to finish. +const BUSY_RETRY: u64 = 5; + +/// What a caller is answered when every place for a kind of work stayed taken +/// as long as it waits — a read no cache answered, a picture handed on from +/// TMDB: a 503, in the API's own shape, and when to come back. +pub(crate) fn busy() -> Response { + let mut response = AppError::Disabled { + code: "busy", + message: "this server is answering as many such requests as it can; try again in a few \ + seconds" + .into(), + } + .into_response(); + response + .headers_mut() + .insert(header::RETRY_AFTER, HeaderValue::from(BUSY_RETRY)); + response +} /// Spec metadata. Paths and schemas are collected from the handlers themselves, /// so this cannot drift from what the server actually serves. @@ -284,10 +306,16 @@ fn tmdb_surface(state: AppState, tmdb: Router) -> Router { /// key policy read as the address list — a picture's address, built by the /// client from TMDB's configuration, carries no key. Held to the limit of /// requests an address may make only where its guard turns the caller -/// away: a library's scan asks for pictures in bursts no relay's caller -/// makes ([`guards::guard_tmdb_images`]). +/// away — a library's scan asks for pictures in bursts no relay's caller +/// makes ([`guards::guard_tmdb_images`]) — and, where it lets the caller in, +/// to a quota of pictures ten times as high ([`tmdb_images::Pace`]). fn tmdb_images_surface(state: AppState, images: Router) -> Router { images + // Inside the guard, which has let the caller in and left it uncounted. + .layer(from_fn_with_state( + tmdb_images::Pace::new(state.clone()), + tmdb_images::pace, + )) .layer(from_fn_with_state(state.clone(), guards::guard_tmdb_images)) .layer(from_fn_with_state(state, guards::reject_self_calls)) } @@ -989,6 +1017,21 @@ mod tests { ); } + /// Work turned away for want of a place is told so in the API's shape, + /// as a 503, with when to come back. + #[tokio::test] + async fn a_caller_turned_away_from_a_place_is_told_when_to_come_back() { + let response = busy(); + assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response.headers()[header::RETRY_AFTER], "5"); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let said: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!(said["error"], "busy"); + assert!(said["message"].as_str().is_some_and(|m| !m.is_empty())); + } + /// The interface's types are generated from a copy of the spec kept in /// `frontend/openapi.json` (`npm run types` there writes /// `src/lib/api.gen.ts`), so a field added here reaches the pages' types diff --git a/src/api/native/account.rs b/src/api/native/account.rs index 57d5516..9e15ebd 100644 --- a/src/api/native/account.rs +++ b/src/api/native/account.rs @@ -339,7 +339,7 @@ fn clean_key_name(raw: &str) -> AppResult { responses( (status = 201, body = IssuedKey), (status = 400, description = "A field was rejected"), - (status = 403, description = "The key limit is reached, or a key asked"), + (status = 403, description = "The key limit is reached, a key asked, or an administrator paused your keys"), (status = 409, description = "You already have a key by that name"), ), )] @@ -357,6 +357,10 @@ async fn create_key( // would both find room for one more. let _held = state.accounts_lock().await; + if clients::paused_by_admin(&state, &me.id).await?.is_some() { + return Err(keys_paused()); + } + if let Some(limit) = limit_for(&state, me) && repo::client::count_owned(&state.db, &me.id).await? >= limit { @@ -412,6 +416,18 @@ async fn create_key( )) } +/// What a person is told while an administrator's pause on their keys stands: +/// the pause is on them, not on one key, so that deleting a paused key and +/// making another is no way round it. +fn keys_paused() -> AppError { + AppError::Refused { + code: "keys_paused_by_admin", + message: "an administrator paused your keys: until one lifts that, you can neither make a \ + key nor resume one" + .into(), + } +} + /// One of your keys, or nobody's business. async fn own_key(state: &AppState, me: &User, id: &str) -> AppResult { repo::client::get(&state.db, id) @@ -437,6 +453,7 @@ pub struct KeyUpdate { responses( (status = 200, body = ApiClient), (status = 400, description = "A field was rejected"), + (status = 403, description = "An administrator paused your keys, and only one can resume them"), (status = 404, description = "Not one of your keys"), (status = 409, description = "You already have a key by that name"), ), @@ -470,17 +487,14 @@ async fn update_key( ))); } - // Paused by an administrator, it stays paused until one resumes it: a - // member whose key was stopped for hammering the relays does not get to - // start it again. + // Paused by an administrator, the keys stay paused until one resumes + // them: a member whose key was stopped for hammering the relays does not + // get to start it again, nor another of theirs. if update.is_enabled == Some(true) && !key.is_enabled - && clients::paused_by_admin(&state, &id).await? + && clients::paused_by_admin(&state, &me.id).await?.is_some() { - return Err(AppError::Refused { - code: "key_paused_by_admin", - message: "an administrator paused this key; only an administrator can resume it".into(), - }); + return Err(keys_paused()); } if let Some(name) = &name { @@ -825,6 +839,124 @@ mod tests { } } + async fn opened(state: &AppState, name: &str, role: Role) -> User { + repo::user::create( + &state.db, + repo::user::NewUser { + username: name, + password_hash: repo::user::NO_PASSWORD, + role, + status: Status::Active, + display_name: None, + email: None, + invited_by: None, + oidc: None, + }, + ) + .await + .expect("an account") + } + + /// Whether an answer is the refusal a pause tells. + fn paused(answer: Result) -> bool { + matches!( + answer, + Err(AppError::Refused { + code: "keys_paused_by_admin", + .. + }) + ) + } + + /// A member whose key an administrator stopped could delete it and make + /// another, and the pause was gone with the key. It stands on the person: + /// until an administrator lifts it they make no key and resume none, and + /// deleting the one that was paused is no way round it. + #[tokio::test] + async fn an_administrators_pause_stands_on_the_person_and_outlives_the_key() { + use crate::settings::Scope; + + let state = crate::api::native::testing::server().await; + state + .settings + .set(Scope::Server, "", "keys.maxPerUser", "5", None) + .await + .expect("room for several keys"); + let admin = opened(&state, "root", Role::Admin).await; + let alice = opened(&state, "alice", Role::Member).await; + let as_alice = || Extension(Identity::User(Box::new(alice.clone()))); + let as_admin = || Extension(Identity::User(Box::new(admin.clone()))); + + let make = |name: &'static str| { + create_key( + State(state.clone()), + as_alice(), + ClientIp::NONE, + Json(NewKey { + name: name.into(), + scopes: vec![], + }), + ) + }; + let toggle = |id: &str, enabled: bool| { + update_key( + State(state.clone()), + as_alice(), + ClientIp::NONE, + Path(id.to_string()), + Json(KeyUpdate { + name: None, + is_enabled: Some(enabled), + scopes: None, + }), + ) + }; + let from_admin = |id: &str, enabled: bool| { + clients::update( + State(state.clone()), + as_admin(), + ClientIp::NONE, + Path(id.to_string()), + Json(clients::UpdateRequest { + is_enabled: Some(enabled), + }), + ) + }; + + let (_, Json(sonarr)) = make("Sonarr").await.expect("a key"); + let (_, Json(radarr)) = make("Radarr").await.expect("another"); + + // Hers to pause and resume, while nobody else has said anything. + let _ = toggle(&radarr.key.id, false).await.expect("paused by her"); + let _ = toggle(&radarr.key.id, true).await.expect("resumed by her"); + + // An administrator stops the first, which is hammering the relays. + let _ = from_admin(&sonarr.key.id, false).await.expect("paused"); + assert!(paused(toggle(&sonarr.key.id, true).await)); + // Nor does she resume another she pauses herself, or make a third. + let _ = toggle(&radarr.key.id, false).await.expect("paused by her"); + assert!(paused(toggle(&radarr.key.id, true).await)); + assert!(paused(make("Third").await)); + + // Deleting the key that was paused is no way round it. + delete_key( + State(state.clone()), + as_alice(), + ClientIp::NONE, + Path(sonarr.key.id.clone()), + ) + .await + .expect("deleted"); + assert!(paused(make("Sonarr again").await)); + assert!(paused(toggle(&radarr.key.id, true).await)); + + // The administrator resumes the key she has left, and the pause lifts. + let _ = from_admin(&radarr.key.id, true).await.expect("resumed"); + let _ = make("Sonarr again").await.expect("made again"); + let _ = toggle(&radarr.key.id, false).await.expect("hers again"); + let _ = toggle(&radarr.key.id, true).await.expect("hers again"); + } + #[tokio::test] async fn a_person_keeps_their_preferences_and_no_key_can() { let state = crate::api::native::testing::server().await; diff --git a/src/api/native/auth.rs b/src/api/native/auth.rs index de6a1ad..36254aa 100644 --- a/src/api/native/auth.rs +++ b/src/api/native/auth.rs @@ -20,10 +20,11 @@ use crate::{ auth::{ Identity, ip, middleware::{self, SESSION_COOKIE}, + ratelimit::{self, Reserved}, secrets, }, db::repo::{self, audit::Action}, - error::{AppError, AppResult}, + error::{AppError, AppResult, Throttle}, state::AppState, }; @@ -104,18 +105,12 @@ async fn login( return Err(AppError::Unauthorized); } - // An account that has failed too often lately is not checked at all, from - // wherever the attempt comes: guessing one person's password from many - // addresses runs out as surely as from one. Asked before anything is - // hashed, so a refused attempt costs nothing. Told when the account's - // window closes: the failures are no quota of requests, so that alone. - if let Err(wait) = state.limiter.sign_in_allowed(&request.username).await { - tracing::warn!( - username = ?request.username, - "sign-in refused: too many failed attempts on this account lately" - ); - return Err(AppError::RateLimited(crate::error::Throttle::after(wait))); - } + // The attempt is counted before anything is looked up or hashed, and + // refused there when the account has taken as many from this address as + // it takes lately — or, from addresses it has not been signed in from, + // as many from everywhere. Told when the window closes: the failures + // are no quota of requests, so that alone. + let attempt = begin_attempt(&state, &request.username, &ip, None).await?; // Passwords switched off: the identity provider signs people in, and only // the account the environment names keeps a door — the way back in when @@ -129,8 +124,13 @@ async fn login( message: "this server signs people in through its identity provider".into(), }; if passwords_off && !state.is_break_glass_name(&request.username) { - secrets::verify_password_async(request.password.clone(), DUMMY_HASH.to_string()).await?; - state.limiter.sign_in_failed(&request.username).await; + check_password( + &state, + attempt, + request.password.clone(), + DUMMY_HASH.to_string(), + ) + .await?; return Err(refused()); } @@ -144,40 +144,53 @@ async fn login( // An account an identity provider made has no password: checked // against the dummy all the same, so it takes as long as any other. Some(creds) if creds.password_hash == repo::user::NO_PASSWORD => { - secrets::verify_password_async(request.password.clone(), DUMMY_HASH.to_string()) - .await?; + check_password( + &state, + attempt, + request.password.clone(), + DUMMY_HASH.to_string(), + ) + .await?; (Some(creds.user), false) } Some(creds) => { - let ok = secrets::verify_password_async(request.password.clone(), creds.password_hash) - .await?; + let ok = check_password( + &state, + attempt, + request.password.clone(), + creds.password_hash, + ) + .await?; (Some(creds.user), ok) } None => { - secrets::verify_password_async(request.password.clone(), DUMMY_HASH.to_string()) - .await?; + check_password( + &state, + attempt, + request.password.clone(), + DUMMY_HASH.to_string(), + ) + .await?; (None, false) } }; let (Some(user), true) = (user, ok) else { - state.limiter.sign_in_failed(&request.username).await; - // Debug-formatted: quoted, its line breaks and control characters - // escaped, so a name typed into the form cannot write a line of its - // own into the log. + // Counted already, before the password was looked at. Debug-formatted: + // quoted, its line breaks and control characters escaped, so a name + // typed into the form cannot write a line of its own into the log. tracing::warn!(username = ?request.username, "failed sign-in attempt"); // Recorded deliberately: a run of these is the one thing in this log - // worth alerting on. The attempted username is kept; the password is not. - audit::record( + // worth alerting on. The attempted username is kept; the password is + // not. A run from one network is one row at once and one that counts + // the rest — anybody can send these with no credential at all. + audit::record_failure( &state, - Event { - identity: None, - ip: &ip, - action: Action::SignInFailed, - target: Some(request.username.trim()), - detail: None, - }, + audit::Failure::Password, + &ip, + Some(request.username.trim()), + None, ) .await; @@ -188,7 +201,7 @@ async fn login( }); }; - state.limiter.sign_in_succeeded(&request.username).await; + state.limiter.sign_in_succeeded(attempt).await; if passwords_off && !(user.role == repo::user::Role::Admin && user.status == repo::user::Status::Active) @@ -220,6 +233,97 @@ async fn login( Ok((StatusCode::OK, [(header::SET_COOKIE, cookie)], Json(answer))) } +/// Count an attempt at `username`'s password from this caller's network +/// before the password is looked at, and refuse it when it is not to be. +/// Refused: when the network has made as many attempts on the account as a +/// window allows, whatever it sends; and when the account has taken more +/// than its bar from every network together and this one has not been signed +/// in to it lately ([`ratelimit::address_known`]), so that guessing from +/// many addresses runs out while the account's owner, from theirs, still +/// gets in. A refused attempt costs nothing: no password is hashed for it. +/// +/// `user` is the account's id when it is known; otherwise the account is +/// looked up by its name, and only once the bar is passed. +pub(crate) async fn begin_attempt( + state: &AppState, + username: &str, + ip: &ClientIp, + user: Option<&str>, +) -> AppResult { + let network = ratelimit::network_of(ip.0); + let reserved = match state.limiter.sign_in_begin(username, network).await { + Ok(reserved) => reserved, + Err(wait) => { + tracing::warn!( + username = ?username, + "sign-in refused: too many failed attempts on this account from this address lately" + ); + return Err(AppError::RateLimited(Throttle::after(wait))); + } + }; + + let Some(wait) = reserved.over_account_bar() else { + return Ok(reserved); + }; + let known = match user { + Some(id) => ratelimit::address_known(&state.db, id, network).await, + None => match repo::user::find_by_username(&state.db, username.trim()).await? { + Some(found) => ratelimit::address_known(&state.db, &found.user.id, network).await, + None => false, + }, + }; + if known { + return Ok(reserved); + } + + state.limiter.sign_in_given_back(reserved).await; + tracing::warn!( + username = ?username, + "sign-in refused: too many failed attempts on this account lately, from addresses it has \ + not been signed in from" + ); + Err(AppError::RateLimited(Throttle::after(wait))) +} + +/// A password checked within an attempt counted by [`begin_attempt`]: whether +/// it is right. When every hashing slot stayed taken nothing was checked, and +/// the attempt is given back; the caller is told to come back. +pub(crate) async fn check_password( + state: &AppState, + attempt: Reserved, + password: String, + hash: String, +) -> AppResult { + match secrets::verify_password_async(password, hash).await { + Ok(ok) => Ok(ok), + Err(busy @ secrets::HashError::Busy) => { + state.limiter.sign_in_given_back(attempt).await; + Err(busy.into()) + } + Err(other) => Err(other.into()), + } +} + +/// A signed-in person's password, asked for again as proof that it is them +/// at the keyboard and not whoever holds a session left open: counted and +/// checked as a sign-in is, so that guessing it from a session that is not +/// theirs runs out as guessing it at the form does. A wrong one is +/// `Unauthorized`. +pub(crate) async fn prove_password( + state: &AppState, + user: &repo::user::User, + hash: String, + typed: &str, + ip: &ClientIp, +) -> AppResult<()> { + let attempt = begin_attempt(state, &user.username, ip, Some(&user.id)).await?; + if !check_password(state, attempt, typed.to_string(), hash).await? { + return Err(AppError::Unauthorized); + } + state.limiter.sign_in_succeeded(attempt).await; + Ok(()) +} + /// Open a session for someone who has shown who they are — by their password, /// by signing up, or through the identity provider — and say so in the /// journal. Returns the cookie to set and what the interface is told; @@ -247,6 +351,11 @@ pub(crate) async fn open_session( repo::user::mark_login(&state.db, &user.id).await?; + // However they came in: the address a person signs in from is one the + // account's own, which what is tried against it from elsewhere does not + // shut out ([`begin_attempt`]). + ratelimit::remember_address(&state.db, &user.id, ratelimit::network_of(ip.0)).await; + let identity = Identity::User(Box::new(user.clone())); audit::record( state, @@ -454,11 +563,16 @@ async fn change_password( return Err(AppError::BadRequest("that password is too long".into())); } - if !secrets::verify_password_async(request.current_password.clone(), creds.password_hash) - .await? - { - return Err(AppError::Unauthorized); - } + // Asked as a sign-in is: a session is not the password, and guessing the + // password through it is counted as guessing it at the form. + prove_password( + &state, + user, + creds.password_hash, + &request.current_password, + &ip, + ) + .await?; let hash = secrets::hash_password_async(request.new_password.clone()).await?; @@ -485,3 +599,161 @@ async fn change_password( Ok(StatusCode::NO_CONTENT) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::{ + auth::testing::{client_at, server}, + db::repo::user::{NewUser, Role, Status, User}, + }; + + /// A password, drawn at random: the tests need none in particular, and one + /// written in the code reads as a secret kept there to whoever scans it. + fn password() -> String { + secrets::generate_session_token().unwrap().0 + } + + async fn account(state: &AppState, name: &str, password: &str) -> User { + repo::user::create( + &state.db, + NewUser { + username: name, + password_hash: &secrets::hash_password(password).unwrap(), + role: Role::Member, + status: Status::Active, + display_name: None, + email: None, + invited_by: None, + oidc: None, + }, + ) + .await + .expect("an account") + } + + /// The sign-in form, filled in from an address. + async fn sign_in( + state: &AppState, + from: &str, + username: &str, + password: &str, + ) -> Result { + let ip = client_at(state, from).await; + login( + State(state.clone()), + ip, + None, + HeaderMap::new(), + Json(LoginRequest { + username: username.into(), + password: password.into(), + }), + ) + .await + .map(|answer| answer.into_response().status()) + } + + /// What the lock-out is for is that a burst is held to its limit. Every + /// password of a burst was checked, once, when the count was made after + /// the check: all of them found it at nothing, and went on to be checked. + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn a_burst_of_wrong_passwords_is_checked_no_more_than_the_limit_allows() { + let state = server(3).await; + account(&state, "alice", &password()).await; + let wrong = password(); + + let attempts: Vec<_> = (0..12) + .map(|_| { + let (state, wrong) = (state.clone(), wrong.clone()); + tokio::spawn(async move { sign_in(&state, "198.51.100.9", "alice", &wrong).await }) + }) + .collect(); + let (mut checked, mut turned_away) = (0, 0); + for attempt in attempts { + match attempt.await.unwrap() { + Err(AppError::Unauthorized) => checked += 1, + Err(AppError::RateLimited(_)) => turned_away += 1, + other => panic!("{other:?}"), + } + } + assert_eq!((checked, turned_away), (3, 9)); + } + + /// Somebody guessing at an account from their own addresses shuts those + /// addresses out, and — past a bar for the account as a whole — every + /// address that has not signed in to it. Its owner's usual address is not + /// one of them: with the right password it gets in, and the guessing + /// that was turned away cost nothing to check. + #[tokio::test] + async fn guessing_at_an_account_shuts_out_the_guessers_and_not_its_owner() { + let state = server(1).await; + let right = password(); + account(&state, "alice", &right).await; + let (home, away) = ("203.0.113.7", "203.0.113.99"); + + // Their usual address, signed in from once: known to the account. + assert_eq!( + sign_in(&state, home, "alice", &right).await.unwrap(), + StatusCode::OK + ); + + // Five addresses, each its one failure — the account's bar is five + // times an address's — and the sixth is turned away unchecked. + let wrong = password(); + for n in 1..=5 { + let from = format!("198.51.100.{n}"); + assert!(matches!( + sign_in(&state, &from, "alice", &wrong).await, + Err(AppError::Unauthorized) + )); + } + assert!(matches!( + sign_in(&state, "198.51.100.6", "alice", &wrong).await, + Err(AppError::RateLimited(_)) + )); + // One address, again: its own count is spent. + assert!(matches!( + sign_in(&state, "198.51.100.1", "alice", &wrong).await, + Err(AppError::RateLimited(_)) + )); + + // The owner, from where they have signed in, gets through; from + // somewhere they have not, they wait with the rest. + assert_eq!( + sign_in(&state, home, "alice", &right).await.unwrap(), + StatusCode::OK + ); + assert!(matches!( + sign_in(&state, away, "alice", &right).await, + Err(AppError::RateLimited(_)) + )); + } + + /// A session is not the password: asking for it again is counted, so that + /// guessing it through a session that is not its owner's runs out as at + /// the form. + #[tokio::test] + async fn the_current_password_asked_for_again_is_counted_as_at_the_form() { + let state = server(1).await; + let right = password(); + let alice = account(&state, "alice", &right).await; + let hash = repo::user::credentials(&state.db, &alice.id) + .await + .unwrap() + .unwrap() + .password_hash; + let ip = client_at(&state, "198.51.100.9").await; + + let wrong = password(); + assert!(matches!( + prove_password(&state, &alice, hash.clone(), &wrong, &ip).await, + Err(AppError::Unauthorized) + )); + // Spent: not even the right one is looked at now. + assert!(matches!( + prove_password(&state, &alice, hash, &right, &ip).await, + Err(AppError::RateLimited(_)) + )); + } +} diff --git a/src/api/native/backup.rs b/src/api/native/backup.rs index 5f159e7..a648cfe 100644 --- a/src/api/native/backup.rs +++ b/src/api/native/backup.rs @@ -185,6 +185,7 @@ async fn create( async fn download( State(state): State, Extension(identity): Extension, + ip: ClientIp, Path(name): Path, ) -> AppResult { identity.require_admin()?; @@ -194,6 +195,21 @@ async fn download( .await? .ok_or(AppError::NotFound)?; + // Written down as the archive starts to leave: it holds every account's + // password hash, and who took one is what the trail is read for after a + // leak. A name that is no archive's, or no longer kept, is not an entry. + audit::record( + &state, + Event { + identity: Some(&identity), + ip: &ip, + action: Action::BackupDownloaded, + target: Some(&name), + detail: None, + }, + ) + .await; + let mut response = Body::from_stream(found.stream).into_response(); let headers = response.headers_mut(); // Plain zstd says what it is; an encrypted one is bytes to whoever has @@ -340,6 +356,7 @@ mod tests { download( State(state.clone()), Extension(identity.clone()), + ClientIp::NONE, Path("ams-20261019-020000-manual.tar.zst".into()) ) .await, @@ -371,9 +388,14 @@ mod tests { assert_eq!(archive.size_bytes, made.size); assert!(!listed.overdue); - let response = download(State(state.clone()), admin(), Path(made.name.clone())) - .await - .unwrap(); + let response = download( + State(state.clone()), + admin(), + ClientIp::NONE, + Path(made.name.clone()), + ) + .await + .unwrap(); assert_eq!(response.status(), StatusCode::OK); assert_eq!( response.headers()[header::ETAG], @@ -394,11 +416,33 @@ mod tests { // A name that is not an archive's names nothing, wherever it points. for name in ["../ams.db", "ams.db", "ams-20261019-020000-manual.tar.zst"] { assert!(matches!( - download(State(state.clone()), admin(), Path(name.into())).await, + download( + State(state.clone()), + admin(), + ClientIp::NONE, + Path(name.into()) + ) + .await, Err(AppError::NotFound) )); } + // Written down: which archive was taken, and by whom — once, for the + // download that happened, none for the names that named nothing. + let trail = repo::audit::list( + &state.db, + &repo::audit::Query { + action: Some("backup.downloaded".into()), + limit: 5, + ..Default::default() + }, + ) + .await + .unwrap(); + assert_eq!(trail.len(), 1); + assert_eq!(trail[0].target.as_deref(), Some(made.name.as_str())); + assert!(trail[0].actor.is_some()); + let gone = remove( State(state.clone()), admin(), diff --git a/src/api/native/browse.rs b/src/api/native/browse.rs index e6f36d0..e15ff68 100644 --- a/src/api/native/browse.rs +++ b/src/api/native/browse.rs @@ -9,6 +9,7 @@ use std::collections::{BTreeMap, HashMap}; use axum::{ Extension, Json, extract::{Path, Query, State}, + response::{IntoResponse, Response}, }; use serde::{Deserialize, Serialize}; use utoipa::{IntoParams, ToSchema}; @@ -86,13 +87,14 @@ pub struct CatalogueFacets { responses( (status = 200, body = CatalogueFacets), (status = 400, description = "A filter was not one the list understands"), + (status = 503, description = "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again"), ), )] async fn facets( State(state): State, Extension(identity): Extension, Query(query): Query, -) -> AppResult> { +) -> AppResult { let mut filters = super::items::to_query(&state, &identity, query)?; // The page and its order are the list's; the counts are the same for all. filters.sort = repo::item::Sort::default(); @@ -108,13 +110,22 @@ async fn facets( let key = format!( "facets:{filters:?}:{}:{}", state.caches.stamp(), - repo::item::catalogue_stamp(&state.db).await? + state.caches.catalogue_stamp(&state.db).await? ); - if let Some(cached) = state.caches.searches.get(&key).await - && let Ok(facets) = serde_json::from_str::(&cached) - { - return Ok(Json(facets)); + if let Some(kept) = kept_facets(&state, &key).await { + return Ok(Json(kept).into_response()); + } + + // Counted over every work the filters leave, for filters nobody has asked + // about yet: a few at a time, so that asking for many of them is no way to + // take the database from everybody else. + let Some(_place) = state.caches.cold.take().await else { + return Ok(crate::api::busy()); + }; + // Asked and answered meanwhile, perhaps, by a caller that waited too. + if let Some(kept) = kept_facets(&state, &key).await { + return Ok(Json(kept).into_response()); } // The works a misspelt term is taken for, as the list takes it: settled @@ -132,7 +143,13 @@ async fn facets( state.caches.searches.insert(key, encoded).await; } - Ok(Json(found)) + Ok(Json(found).into_response()) +} + +/// The facets kept under a key, when they are. +async fn kept_facets(state: &AppState, key: &str) -> Option { + let cached = state.caches.searches.get(key).await?; + serde_json::from_str(&cached).ok() } // ─── calendar ──────────────────────────────────────────────────────────────── @@ -735,33 +752,54 @@ async fn network( } /// The name of the network or the studio a slug stands for, as the works -/// name it most often, or none: kept a while under the catalogue's mark, -/// since every visit to the page asks. +/// name it most often, or none. +/// +/// Every visit to the page asks, and the slug is whatever the caller wrote: +/// so it is matched in memory against the names the catalogue holds, which +/// are kept — for each reading of adult titles and each mark of the +/// catalogue — in one place, and never becomes the key of anything. pub(crate) async fn network_named( state: &AppState, slug: &str, adult: bool, ) -> AppResult> { + Ok(network_names(state, adult).await?.get(slug).cloned()) +} + +/// Each slug the networks' and studios' names make, and the name the works +/// give it most often: of two spellings as common, the first in order. +async fn network_names( + state: &AppState, + adult: bool, +) -> AppResult>> { let key = format!( - "network-name:{slug}:{adult}:{}", - repo::item::catalogue_stamp(&state.db).await? + "{adult}:{}:{}", + state.caches.stamp(), + state.caches.catalogue_stamp(&state.db).await? ); - if let Some(cached) = state.caches.searches.get(&key).await { - return Ok(Some(cached).filter(|name| !name.is_empty())); - } - let named = repo::item::network_names(&state.db, adult) - .await? - .into_iter() - .filter(|(name, _)| slug::slugify(name) == slug) - // The most common spelling, and of two as common the first in order. - .max_by(|a, b| a.1.cmp(&b.1).then_with(|| b.0.cmp(&a.0))) - .map(|(name, _)| name); state .caches - .searches - .insert(key, named.clone().unwrap_or_default()) - .await; - Ok(named) + .network_names + .try_get_with(key, async { + let mut best: HashMap = HashMap::new(); + for (name, count) in repo::item::network_names(&state.db, adult).await? { + let slug = slug::slugify(&name); + let better = match best.get(&slug) { + None => true, + Some((held, kept)) => count > *held || (count == *held && name < *kept), + }; + if better { + best.insert(slug, (count, name)); + } + } + Ok::<_, anyhow::Error>(std::sync::Arc::new( + best.into_iter() + .map(|(slug, (_, name))| (slug, name)) + .collect(), + )) + }) + .await + .map_err(|e| AppError::Internal(anyhow::anyhow!("{e:#}"))) } /// The longest id an entity's address is looked up by: this server's ids @@ -1799,6 +1837,97 @@ mod tests { )); } + /// A slug is whatever the caller wrote: it is matched in memory, against + /// the names the catalogue holds, and one nobody holds costs no read and + /// is never the key of anything kept. + #[tokio::test] + async fn a_network_slug_nobody_holds_is_never_the_key_of_anything() { + let state = super::super::testing::server().await; + for (title, network, adult) in [ + ("On Canal", "Canal+", false), + ("After Dark", "Night Channel", true), + ] { + stored(&state, MediaKind::Series, title, |w| { + w.network = Some(network.into()); + w.is_adult = adult; + }) + .await; + } + crate::service::listing::relist_stale(&state.db, false, 100) + .await + .expect("listed"); + + let named = async |slug: &str, adult: bool| { + network_named(&state, slug, adult).await.expect("answered") + }; + assert_eq!(named("canal", false).await.as_deref(), Some("Canal+")); + // A title for adults only where adults are counted. + assert_eq!(named("night-channel", false).await, None); + assert_eq!( + named("night-channel", true).await.as_deref(), + Some("Night Channel") + ); + + state.caches.searches.settle().await; + let kept = state.caches.searches.l1_entries(); + let readings = state.caches.network_names.entries().await; + assert_eq!(readings, 2, "one for each reading of adult titles"); + for n in 0..200 { + assert_eq!(named(&format!("nobody-{n}"), false).await, None); + } + state.caches.searches.settle().await; + assert_eq!(state.caches.searches.l1_entries(), kept); + assert_eq!(state.caches.network_names.entries().await, readings); + } + + /// The facets of filters nobody asked about are counted over every work + /// they leave, a few at a time: with every place taken, the caller waits, + /// and is turned away — told when to come back — when none is free. Those + /// kept are answered whatever. + #[tokio::test] + async fn facets_nobody_asked_for_wait_for_a_place_and_are_turned_away_without_one() { + use axum::http::{StatusCode, Uri, header}; + + let state = super::super::testing::server().await; + stored(&state, MediaKind::Movie, "Heat", |w| { + w.genres = vec!["Crime".into()]; + }) + .await; + crate::service::listing::relist_stale(&state.db, false, 100) + .await + .expect("listed"); + state + .caches + .cold + .wait_at_most(std::time::Duration::from_millis(30)); + + let counted = async |filters: &str| { + let uri: Uri = format!("/facets?{filters}").parse().expect("an address"); + let Query(query) = + Query::::try_from_uri(&uri).expect("a query"); + facets( + State(state.clone()), + Extension(Identity::Visitor), + Query(query), + ) + .await + .expect("answered") + }; + assert_eq!(counted("genre=Crime").await.status(), StatusCode::OK); + + let mut held = Vec::new(); + while state.caches.cold.free() > 0 { + held.push(state.caches.cold.take().await.expect("a place")); + } + assert_eq!(counted("genre=Crime").await.status(), StatusCode::OK); + let turned_away = counted("genre=Drama").await; + assert_eq!(turned_away.status(), StatusCode::SERVICE_UNAVAILABLE); + assert!(turned_away.headers().get(header::RETRY_AFTER).is_some()); + + held.pop(); + assert_eq!(counted("genre=Drama").await.status(), StatusCode::OK); + } + #[tokio::test] async fn a_company_and_a_keyword_are_pages_by_this_servers_id_for_them() { use crate::domain::{Company, CompanyCredit, CompanyKind, CompanyRole, Keyword}; diff --git a/src/api/native/clients.rs b/src/api/native/clients.rs index 4aee8e0..ba3dc31 100644 --- a/src/api/native/clients.rs +++ b/src/api/native/clients.rs @@ -224,6 +224,11 @@ pub struct UpdateRequest { } /// Enable or disable a key without revoking it. +/// +/// A person's key paused from here pauses its owner's keys: its owner can +/// neither resume it nor make another, or resume any other of theirs, until an +/// administrator enables a key of theirs from here, or lifts the pause +/// (`DELETE /users/{id}/keys-pause`). #[utoipa::path( patch, path = "/clients/{id}", tag = TAG, operation_id = "clients_update", @@ -235,7 +240,7 @@ pub struct UpdateRequest { (status = 404, description = "No such client"), ), )] -async fn update( +pub(crate) async fn update( State(state): State, Extension(identity): Extension, ip: ClientIp, @@ -245,6 +250,11 @@ async fn update( identity.require_admin()?; if let Some(enabled) = request.is_enabled { + // Under the accounts lock, as its owner's own changes to their keys + // are: a key enabled between the pause being read and being written + // would be enabled still. + let _held = state.accounts_lock().await; + // Read first: the trail is read by humans, who know the name, and a // person's key paused from here is one its owner cannot resume. let key = repo::client::get(&state.db, &id).await?; @@ -256,11 +266,13 @@ async fn update( // A server key has no owner to keep it from; an administrator's own // key, paused from here, is theirs to resume from their account. + let mut said = String::from(if enabled { "enabled" } else { "disabled" }); if let Some(owner) = key.as_ref().and_then(|k| k.owner_id.as_deref()) { if !enabled && identity.person_id() != Some(owner) { - mark_paused(&state, &id, Some(&identity.label())).await?; - } else if enabled { - mark_paused(&state, &id, None).await?; + mark_paused(&state, owner, Some(&identity.label())).await?; + said.push_str(" · its owner can neither make nor resume keys until it is lifted"); + } else if enabled && lift_pause(&state, owner).await? { + said.push_str(" · the pause on its owner's keys lifted"); } } @@ -271,7 +283,7 @@ async fn update( ip: &ip, action: Action::ClientUpdated, target: name.as_deref(), - detail: Some(if enabled { "enabled" } else { "disabled" }), + detail: Some(&said), }, ) .await; @@ -356,32 +368,169 @@ fn expiry(text: &str) -> AppResult { Ok(crate::db::to_rfc3339(at)) } -/// Where the administrator who paused a person's key is written down. +/// Where the administrator who paused a person's keys is written down. /// -/// In the store every instance shares rather than on the key's row: what -/// matters is only that the owner's own "resume" is refused while it is -/// there, and that is the store that needs no change of schema to hold it. -fn pause_entry(id: &str) -> String { - format!("client.pausedBy:{id}") +/// On the person, not on a key: a key its owner deletes and makes again is +/// another key, and a pause that stood on the first went with it. In the store +/// every instance shares rather than on the account's row, which is the one +/// that needs no change of schema to hold it. +fn pause_entry(owner_id: &str) -> String { + format!("user.keysPausedBy:{owner_id}") +} + +/// What an older version wrote: the pause on one key, and nothing on its +/// owner. Still honoured, until an administrator lifts it. +fn key_pause_entry(key_id: &str) -> String { + format!("client.pausedBy:{key_id}") } -/// Write down who paused a person's key, or that nobody holds it paused. -async fn mark_paused(state: &AppState, id: &str, by: Option<&str>) -> AppResult<()> { - repo::keystore::put(&state.db, &pause_entry(id), by.unwrap_or_default()).await?; +/// Write down who paused a person's keys, or that nobody holds them paused. +async fn mark_paused(state: &AppState, owner_id: &str, by: Option<&str>) -> AppResult<()> { + repo::keystore::put(&state.db, &pause_entry(owner_id), by.unwrap_or_default()).await?; Ok(()) } -/// Whether an administrator paused this key, which its owner then may not -/// resume. -pub(crate) async fn paused_by_admin(state: &AppState, id: &str) -> AppResult { - Ok(repo::keystore::get(&state.db, &pause_entry(id)) +/// Who an entry says paused, when it says anybody. +async fn standing(state: &AppState, entry: &str) -> AppResult> { + Ok(repo::keystore::get(&state.db, entry) .await? - .is_some_and(|by| !by.is_empty())) + .filter(|by| !by.is_empty())) +} + +/// Who paused this person's keys, when an administrator did and none has +/// lifted it: until then their owner can neither make a key nor resume one. +pub(crate) async fn paused_by_admin(state: &AppState, owner_id: &str) -> AppResult> { + if let Some(by) = standing(state, &pause_entry(owner_id)).await? { + return Ok(Some(by)); + } + // A key paused before the pause stood on the person: the person's, too. + for key in repo::client::list(&state.db, repo::client::Owner::User(owner_id)).await? { + if let Some(by) = standing(state, &key_pause_entry(&key.id)).await? { + return Ok(Some(by)); + } + } + Ok(None) +} + +/// Lift the pause on a person's keys, the one on the person and any that an +/// older version left on a key: whether one stood. What an administrator does +/// by enabling one of their keys, or by saying so. +pub(crate) async fn lift_pause(state: &AppState, owner_id: &str) -> AppResult { + let mut lifted = false; + if standing(state, &pause_entry(owner_id)).await?.is_some() { + mark_paused(state, owner_id, None).await?; + lifted = true; + } + for key in repo::client::list(&state.db, repo::client::Owner::User(owner_id)).await? { + let entry = key_pause_entry(&key.id); + if standing(state, &entry).await?.is_some() { + repo::keystore::put(&state.db, &entry, "").await?; + lifted = true; + } + } + Ok(lifted) } #[cfg(test)] mod tests { use super::*; + use crate::db::repo::user::{NO_PASSWORD, NewUser, Role, Status, User}; + + async fn person(state: &AppState, name: &str, role: Role) -> User { + repo::user::create( + &state.db, + NewUser { + username: name, + password_hash: NO_PASSWORD, + role, + status: Status::Active, + display_name: None, + email: None, + invited_by: None, + oidc: None, + }, + ) + .await + .expect("an account") + } + + async fn key_of(state: &AppState, owner: &User, name: &str) -> ApiClient { + repo::client::create( + &state.db, + repo::client::NewClient { + name, + key_prefix: "ams_test", + key_hash: name, + scopes: &["read".to_string()], + expires_at: None, + note: None, + owner_id: Some(&owner.id), + }, + ) + .await + .expect("a key") + } + + /// What the version before this one wrote was a pause on a key, and + /// nothing on its owner: still a pause on the person, and lifted with it. + #[tokio::test] + async fn a_pause_put_on_a_key_by_an_older_version_is_still_the_persons() { + let state = crate::api::native::testing::server().await; + let alice = person(&state, "alice", Role::Member).await; + let key = key_of(&state, &alice, "Sonarr").await; + + assert_eq!(paused_by_admin(&state, &alice.id).await.unwrap(), None); + repo::keystore::put( + &state.db, + &format!("client.pausedBy:{}", key.id), + "admin:root", + ) + .await + .unwrap(); + assert_eq!( + paused_by_admin(&state, &alice.id).await.unwrap().as_deref(), + Some("admin:root") + ); + + assert!(lift_pause(&state, &alice.id).await.unwrap()); + assert_eq!(paused_by_admin(&state, &alice.id).await.unwrap(), None); + assert!(!lift_pause(&state, &alice.id).await.unwrap(), "none stood"); + } + + /// Pausing a person's key writes the pause on the person, naming who; an + /// administrator's own key, paused from here, is theirs to resume. + #[tokio::test] + async fn pausing_a_persons_key_pauses_the_person_and_naming_who() { + let state = crate::api::native::testing::server().await; + let admin = person(&state, "root", Role::Admin).await; + let alice = person(&state, "alice", Role::Member).await; + let hers = key_of(&state, &alice, "Sonarr").await; + let own = key_of(&state, &admin, "Mine").await; + let as_admin = || Extension(Identity::User(Box::new(admin.clone()))); + let pause = |id: &str, enabled: bool| { + update( + State(state.clone()), + as_admin(), + ClientIp::NONE, + Path(id.to_string()), + Json(UpdateRequest { + is_enabled: Some(enabled), + }), + ) + }; + + let _ = pause(&own.id, false).await.expect("an administrator's own"); + assert_eq!(paused_by_admin(&state, &admin.id).await.unwrap(), None); + + let _ = pause(&hers.id, false).await.expect("a member's"); + assert_eq!( + paused_by_admin(&state, &alice.id).await.unwrap().as_deref(), + Some("admin:root") + ); + // Enabling it from here lifts the pause on its owner. + let _ = pause(&hers.id, true).await.expect("resumed"); + assert_eq!(paused_by_admin(&state, &alice.id).await.unwrap(), None); + } #[test] fn an_expiry_is_a_time_still_to_come() { diff --git a/src/api/native/figures.rs b/src/api/native/figures.rs index c9d8792..9ef2251 100644 --- a/src/api/native/figures.rs +++ b/src/api/native/figures.rs @@ -6,7 +6,12 @@ use std::{ time::{Duration, Instant}, }; -use axum::{Extension, Json, extract::State, http::header}; +use axum::{ + Extension, Json, + extract::State, + http::header, + response::{IntoResponse, Response}, +}; use serde::{Deserialize, Serialize}; use utoipa::ToSchema; use utoipa_axum::{router::OpenApiRouter, routes}; @@ -74,12 +79,15 @@ pub struct Figures { #[utoipa::path( get, path = "/figures", tag = TAG, operation_id = "figures_figures", - responses((status = 200, body = Figures)), + responses( + (status = 200, body = Figures), + (status = 503, description = "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again"), + ), )] async fn figures( State(state): State, Extension(identity): Extension, -) -> AppResult> { +) -> AppResult { let adult = state.adult_for(identity.client_id(), identity.peer_id(), None); // Read across the whole catalogue, by anyone the site is open to, so @@ -90,12 +98,19 @@ async fn figures( let key = format!( "figures:{adult}:{}:{}", state.caches.stamp(), - repo::item::catalogue_stamp(&state.db).await? + state.caches.catalogue_stamp(&state.db).await? ); - if let Some(cached) = state.caches.searches.get(&key).await - && let Ok(figures) = serde_json::from_str::(&cached) - { - return Ok(Json(figures)); + if let Some(kept) = kept_figures(&state, &key).await { + return Ok(Json(kept).into_response()); + } + + // A count of the whole catalogue, made afresh: a few at a time, as the + // facets it reads are. + let Some(_place) = state.caches.cold.take().await else { + return Ok(crate::api::busy()); + }; + if let Some(kept) = kept_figures(&state, &key).await { + return Ok(Json(kept).into_response()); } let query = repo::item::Query { @@ -165,7 +180,13 @@ async fn figures( { state.caches.searches.insert(key, encoded).await; } - Ok(Json(figures)) + Ok(Json(figures).into_response()) +} + +/// The figures kept under a key, when they are. +async fn kept_figures(state: &AppState, key: &str) -> Option { + let cached = state.caches.searches.get(key).await?; + serde_json::from_str(&cached).ok() } #[derive(Serialize, ToSchema)] @@ -630,6 +651,54 @@ mod tests { use super::*; use crate::{auth::testing::key, db::repo::user::Role, error::AppError}; + /// The catalogue's figures are counted over the whole of it, a few at a + /// time when none is kept: with every place taken, the caller waits, and is + /// turned away — told when to come back. Those kept are answered whatever. + #[tokio::test] + async fn figures_nobody_counted_lately_wait_for_a_place_and_are_turned_away_without_one() { + use axum::http::StatusCode; + + let state = super::super::testing::server().await; + let mut work = crate::domain::MediaItem::empty(MediaKind::Movie); + work.title = "Heat".into(); + work.slug = crate::domain::make_slug("Heat", None); + repo::item::upsert( + &state.db, + repo::item::ItemWrite { + item: &work, + replace_children: true, + }, + ) + .await + .expect("stored"); + state + .caches + .cold + .wait_at_most(std::time::Duration::from_millis(30)); + let counted = async || { + figures(State(state.clone()), Extension(Identity::Visitor)) + .await + .expect("answered") + }; + + assert_eq!(counted().await.status(), StatusCode::OK); + + let mut held = Vec::new(); + while state.caches.cold.free() > 0 { + held.push(state.caches.cold.take().await.expect("a place")); + } + assert_eq!(counted().await.status(), StatusCode::OK, "kept"); + + // The catalogue moved: what was kept is of the past. + state.caches.bump_epoch().await; + let turned_away = counted().await; + assert_eq!(turned_away.status(), StatusCode::SERVICE_UNAVAILABLE); + assert!(turned_away.headers().get(header::RETRY_AFTER).is_some()); + + held.pop(); + assert_eq!(counted().await.status(), StatusCode::OK); + } + /// The figures are read by whoever manages the server and by a key with /// the `metrics` scope — a scraper's — and by nobody else, a key that /// reads or edits the catalogue included. diff --git a/src/api/native/items.rs b/src/api/native/items.rs index be58f7f..cb82a32 100644 --- a/src/api/native/items.rs +++ b/src/api/native/items.rs @@ -478,6 +478,7 @@ fn country_code(raw: Option<&str>, what: &str) -> AppResult> { (status = 200, description = "Matching works, newest first", body = ListResponse), (status = 400, description = "A filter is not one the list takes, or a cursor is not one it gave, or was given beside an offset"), (status = 401, description = "No valid credential was presented"), + (status = 503, description = "As many such reads as the server answers at once are under way; `Retry-After` says when to ask again"), ), )] async fn list( @@ -518,13 +519,28 @@ async fn list( let cacheable = !query_for_count .missing .contains(&repo::item::Missing::MediaFailed); - if cacheable && let Some(cached) = state.caches.searches.get(&cache_key).await { - return Ok(( + let kept = |cached: String| { + ( StatusCode::OK, [(header::CONTENT_TYPE, "application/json")], cached, ) - .into_response()); + .into_response() + }; + if cacheable && let Some(cached) = state.caches.searches.get(&cache_key).await { + return Ok(kept(cached)); + } + + // A page nobody has asked for lately is read from the database — through + // the search index, or in an order no index keeps — and asking for one + // takes less than answering it: a few at a time, so that asking for many + // is no way to take the database from everybody else. + let Some(_place) = state.caches.cold.take().await else { + return Ok(crate::api::busy()); + }; + // Asked and answered meanwhile, perhaps, by a caller that waited too. + if cacheable && let Some(cached) = state.caches.searches.get(&cache_key).await { + return Ok(kept(cached)); } // The page and the count from one look in the search index: a misspelt @@ -2083,6 +2099,103 @@ mod tests { assert_eq!(total, found.len() as i64); } + /// The list as a caller is answered for `asked`, whatever the status. + async fn answered(state: &AppState, asked: &str) -> Response { + let uri: Uri = format!("/items?{asked}").parse().expect("an address"); + let Query(query) = Query::::try_from_uri(&uri).expect("a query"); + list(State(state.clone()), admin(), Query(query)) + .await + .expect("answered") + } + + /// Every place for the reads no cache answered, taken. + async fn all_places_taken(state: &AppState) -> Vec { + let mut held = Vec::new(); + while state.caches.cold.free() > 0 { + held.push(state.caches.cold.take().await.expect("a place")); + } + held + } + + /// A page nobody asked for is read from the database a few at a time: a + /// caller waits for its turn, and is turned away — told when to come + /// back — when none comes. A page the cache holds is answered whatever. + #[tokio::test] + async fn a_page_nobody_asked_for_waits_for_a_place_and_is_turned_away_without_one() { + use std::time::Duration; + + let state = crate::api::native::testing::server().await; + stored(&state, 1, "Heat", |_| {}).await; + state.caches.cold.wait_at_most(Duration::from_millis(30)); + + // A place free: read, and kept. + assert_eq!( + answered(&state, "limit=10&sort=title").await.status(), + StatusCode::OK + ); + + let mut held = all_places_taken(&state).await; + // The page kept is answered with no place at all… + assert_eq!( + answered(&state, "limit=10&sort=title").await.status(), + StatusCode::OK + ); + // …and one that nothing kept is not read: a 503, and when to ask again. + let turned_away = answered(&state, "limit=10&sort=release").await; + assert_eq!(turned_away.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(turned_away.headers()[header::RETRY_AFTER], "5"); + let body = axum::body::to_bytes(turned_away.into_body(), usize::MAX) + .await + .expect("a body"); + let said: Value = serde_json::from_slice(&body).expect("JSON"); + assert_eq!(said["error"], "busy"); + + // A place given back is the next caller's. + held.pop(); + assert_eq!( + answered(&state, "limit=10&sort=release").await.status(), + StatusCode::OK + ); + } + + /// Two callers asking for the same page when no place is free: the one + /// that waits is answered from what the other made, and reads nothing. + #[tokio::test] + async fn a_caller_that_waited_for_a_place_is_answered_from_what_the_other_made() { + use std::time::Duration; + + let state = crate::api::native::testing::server().await; + stored(&state, 1, "Heat", |_| {}).await; + state.caches.cold.wait_at_most(Duration::from_secs(5)); + + // One place left, for one of them: the other waits for it, and finds + // the page made when it has it. + let mut held = all_places_taken(&state).await; + held.pop(); + let hits = state.caches.searches.l1_tally().hits; + + let (first, second) = tokio::join!( + answered(&state, "limit=10&sort=title"), + answered(&state, "limit=10&sort=title") + ); + assert_eq!(first.status(), StatusCode::OK); + assert_eq!(second.status(), StatusCode::OK); + let (first, second) = ( + axum::body::to_bytes(first.into_body(), usize::MAX) + .await + .expect("a body"), + axum::body::to_bytes(second.into_body(), usize::MAX) + .await + .expect("a body"), + ); + assert_eq!(first, second); + assert_eq!( + state.caches.searches.l1_tally().hits, + hits + 1, + "one made the page, the other found it made" + ); + } + #[tokio::test] async fn a_seasons_title_and_synopsis_are_read_in_the_language_asked_where_held() { let state = crate::api::native::testing::server().await; diff --git a/src/api/native/lists/import_rules.rs b/src/api/native/lists/import_rules.rs index 6adc119..5fbcf84 100644 --- a/src/api/native/lists/import_rules.rs +++ b/src/api/native/lists/import_rules.rs @@ -577,7 +577,7 @@ async fn preview( let checked = check(&state, request, current.as_ref()).await?; let wanted = rules::gather(&state, checked.source, &checked.params) .await - .map_err(|e| AppError::BadRequest(format!("{e:#}")))?; + .map_err(|e| AppError::BadRequest(rules::told(&e)))?; let lines: Vec = wanted .iter() .take(PREVIEW_MOST) diff --git a/src/api/native/media.rs b/src/api/native/media.rs index 0bfe07e..cdc3993 100644 --- a/src/api/native/media.rs +++ b/src/api/native/media.rs @@ -41,6 +41,20 @@ const TAG: &str = super::items::TAG; /// little for the form around it. const UPLOAD_BYTES: usize = file::MAX_AUDIO_BYTES as usize + 64 * 1024; +/// What a writer who is not an administrator may keep in uploads, all told: +/// room for every poster and theme a catalogue wants, short of filling the +/// store with whatever a key with the write scope sends. +const UPLOAD_QUOTA: u64 = 1024 * 1024 * 1024; + +/// Whether `adding` more bytes would take uploads that weigh `held` past +/// [`UPLOAD_QUOTA`]. +fn past_quota(held: i64, adding: usize) -> bool { + u64::try_from(held) + .unwrap_or(0) + .saturating_add(adding as u64) + > UPLOAD_QUOTA +} + pub fn router() -> OpenApiRouter { OpenApiRouter::new() .routes(routes!(status)) @@ -467,7 +481,10 @@ async fn work_media( .ok_or(AppError::NotFound)?; repo::item::load_children(&state.db, &mut item).await?; // What an address answered is an administrator's to read; a writer is - // told the kind of trouble, since any address can be typed in. + // told that the file could not be fetched and not why, nor how many + // tries it took: any address can be typed in, and what comes back for + // each — the kind of trouble among it — is a map of the network this + // server is on. let admin = identity.is_admin(); let origins = addresses(&state, &item).await?; @@ -492,12 +509,12 @@ async fn work_media( bytes: asset.bytes, width: asset.width, height: asset.height, - attempts: Some(asset.attempts), + attempts: admin.then_some(asset.attempts), error: asset.error.as_deref().map(|error| { if admin { error.to_string() } else { - crate::media::fetch::reason_for_writers(error).to_string() + crate::media::fetch::FOR_WRITERS.to_string() } }), uploaded_by: asset.uploaded_by.clone(), @@ -688,6 +705,57 @@ pub struct Uploaded { pub item: MediaItem, } +/// What a form calls a kind of upload: a picture, or a theme. +fn kind_named(name: &str) -> Option { + match name { + "image" => Some(Kind::Image), + "theme" | "audio" => Some(Kind::Audio), + _ => None, + } +} + +/// The file of an upload, read as it arrives and cut off where its first +/// bytes say it may end: a picture at [`file::MAX_IMAGE_BYTES`], a sound at +/// [`file::MAX_AUDIO_BYTES`], rather than at the most any upload may weigh, +/// once all of it has been held in memory. A file whose first bytes say it +/// is neither is refused as soon as they have come. `declared` is the kind +/// the form named before the file, when it did: the interface sends the file +/// first, so that what its first bytes say is what holds it. +async fn read_file( + mut field: axum::extract::multipart::Field<'_>, + declared: Option, +) -> AppResult { + let mut body: Vec = Vec::new(); + // What it may weigh until its first bytes have come: as a sound may, or + // as the form said. + let (mut kind, mut limit) = declared.map_or((Kind::Audio, file::MAX_AUDIO_BYTES), |kind| { + (kind, file::max_bytes(kind)) + }); + let mut known = false; + while let Some(chunk) = field.chunk().await.map_err(|e| { + if e.status() == StatusCode::PAYLOAD_TOO_LARGE { + AppError::PayloadTooLarge("the file is larger than an upload may be".into()) + } else { + AppError::BadRequest(format!("the file could not be read: {e}")) + } + })? { + body.extend_from_slice(&chunk); + if !known && body.len() >= file::HEAD_BYTES { + (kind, limit) = file::limit_for_head(&body, declared) + .map_err(|e| AppError::BadRequest(e.to_string()))?; + known = true; + } + if body.len() as u64 > limit { + return Err(AppError::PayloadTooLarge(format!( + "the file is larger than the {} MiB a {} may be", + limit / (1024 * 1024), + kind.as_str() + ))); + } + } + Ok(Bytes::from(body)) +} + /// Put a picture or a theme on a work, from a file. /// /// A form with `file`, and: `kind` (`image`, the default, or `theme`); @@ -738,14 +806,8 @@ async fn upload( { let name = field.name().unwrap_or_default().to_string(); if name == "file" { - let bytes = field.bytes().await.map_err(|e| { - if e.status() == StatusCode::PAYLOAD_TOO_LARGE { - AppError::PayloadTooLarge("the file is larger than an upload may be".into()) - } else { - AppError::BadRequest(format!("the file could not be read: {e}")) - } - })?; - file = Some(bytes); + let declared = fields.get("kind").and_then(|kind| kind_named(kind)); + file = Some(read_file(field, declared).await?); } else { let text = field .text() @@ -758,19 +820,10 @@ async fn upload( return Err(AppError::BadRequest("no file was sent".into())); }; - let kind = match fields.get("kind").map(String::as_str).unwrap_or("image") { - "image" => Kind::Image, - "theme" | "audio" => Kind::Audio, - other => { - return Err(AppError::BadRequest(format!( - "{other} is not a kind of upload" - ))); - } - }; - let limit = match kind { - Kind::Image => file::MAX_IMAGE_BYTES, - Kind::Audio => file::MAX_AUDIO_BYTES, - }; + let named = fields.get("kind").map_or("image", String::as_str); + let kind = kind_named(named) + .ok_or_else(|| AppError::BadRequest(format!("{named} is not a kind of upload")))?; + let limit = file::max_bytes(kind); if bytes.len() as u64 > limit { return Err(AppError::PayloadTooLarge(format!( "the file is larger than the {} MiB a {} may be", @@ -842,6 +895,20 @@ async fn upload( }, }; + // A writer's uploads are weighed together, under the label they are + // made as; an administrator's are not. + if !identity.is_admin() { + let by = identity.label(); + let held = repo::asset::uploaded_bytes(&state.db, &by).await?; + if past_quota(held, bytes.len()) { + return Err(AppError::PayloadTooLarge(format!( + "the uploads made as {by} already weigh {} MiB of the {} MiB a writer may keep", + held / (1024 * 1024), + UPLOAD_QUOTA / (1024 * 1024) + ))); + } + } + let asset_id = crate::db::new_id(); let origin = format!("upload:{asset_id}"); let (key, sha256) = file::key_for(&bytes, inspected.ext); @@ -1125,6 +1192,17 @@ mod tests { /// what it took. type Line = (Cause, String, String, Option, Option); + /// A writer's uploads stop at the quota, the last byte under it allowed. + #[test] + fn a_writer_keeps_no_more_than_the_quota_in_uploads() { + let quota = i64::try_from(UPLOAD_QUOTA).unwrap(); + assert!(!past_quota(0, 25 * 1024 * 1024)); + assert!(!past_quota(quota - 10, 10)); + assert!(past_quota(quota - 10, 11)); + assert!(past_quota(quota, 1)); + assert!(!past_quota(-1, 0), "a sum that is no weight weighs nothing"); + } + /// The whole server, on a database of its own, keeping its media in /// `dir`: what an upload needs somewhere to go. async fn server(dir: &std::path::Path) -> AppState { @@ -1447,6 +1525,207 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + + /// A film held here, with a picture of its own. + async fn a_film(state: &AppState, title: &str, url: &str) -> String { + let mut item = MediaItem::empty(MediaKind::Movie); + item.title = title.into(); + item.slug = crate::domain::make_slug(title, None); + item.images = vec![crate::domain::Image { + id: crate::db::new_id(), + season_number: None, + cover_type: CoverType::Poster, + url: url.into(), + language: None, + sort_order: 0, + source: None, + is_manual: false, + }]; + repo::item::upsert( + &state.db, + repo::item::ItemWrite { + item: &item, + replace_children: true, + }, + ) + .await + .expect("stored"); + item.id + } + + /// The form's words as the parts of a request. + fn words_as_parts(words: Words<'_>) -> Vec { + let mut out = Vec::new(); + for (name, value) in words { + out.extend_from_slice( + format!( + "--edge\r\nContent-Disposition: form-data; name=\"{name}\"\r\n\r\n{value}\r\n" + ) + .as_bytes(), + ); + } + out + } + + /// The form's words, each a name and a value. + type Words<'a> = &'a [(&'a str, &'a str)]; + + /// An upload whose file is `head`, then zeros to `total` bytes, sent in + /// chunks of 64 KiB, with the form's words before it and after it — the + /// interface sends the file first — through the route, with the limit the + /// route's body is held to. What it answered, and how many chunks of the + /// request were read by then, of how many there were. + async fn upload_streamed( + state: &AppState, + id: &str, + (before, after): (Words<'_>, Words<'_>), + head: &[u8], + total: usize, + ) -> (StatusCode, Value, usize, usize) { + use std::sync::{ + Arc, + atomic::{AtomicUsize, Ordering::SeqCst}, + }; + use tower::ServiceExt as _; + + let mut first = words_as_parts(before); + first.extend_from_slice( + b"--edge\r\nContent-Disposition: form-data; name=\"file\"; filename=\"upload\"\r\n\ + Content-Type: application/octet-stream\r\n\r\n", + ); + first.extend_from_slice(head); + let mut chunks = vec![Bytes::from(first)]; + let zeros = Bytes::from(vec![0u8; 64 * 1024]); + let mut sent = head.len(); + while sent < total { + let more = (total - sent).min(zeros.len()); + chunks.push(zeros.slice(..more)); + sent += more; + } + let mut last = b"\r\n".to_vec(); + last.extend_from_slice(&words_as_parts(after)); + last.extend_from_slice(b"--edge--\r\n"); + chunks.push(Bytes::from(last)); + + // One chunk for each time the body is asked for more, as a socket + // gives them: a stream that is always ready is read to its end by + // the form's parser in one go, whatever is done with what it gives. + let (all, pulled) = (chunks.len(), Arc::new(AtomicUsize::new(0))); + let counted = pulled.clone(); + let body = Body::from_stream(futures::stream::unfold( + chunks.into_iter(), + move |mut chunks| { + let counted = counted.clone(); + async move { + tokio::task::yield_now().await; + let chunk = chunks.next()?; + counted.fetch_add(1, SeqCst); + Some((Ok::<_, std::io::Error>(chunk), chunks)) + } + }, + )); + let request = Request::builder() + .method("POST") + .uri(format!("/items/{id}/media")) + .header("content-type", "multipart/form-data; boundary=edge") + .body(body) + .expect("a request"); + let response = axum::Router::new() + .route("/items/{id}/media", axum::routing::post(upload)) + .layer(axum::extract::DefaultBodyLimit::max(UPLOAD_BYTES)) + .layer(Extension(Identity::Anonymous)) + .with_state(state.clone()) + .oneshot(request) + .await + .expect("an answer"); + let status = response.status(); + let said = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .expect("a body"); + let said = serde_json::from_slice(&said).unwrap_or(Value::Null); + (status, said, pulled.load(SeqCst), all) + } + + /// A picture is cut off at the weight of a picture as it is read, not + /// once it has been held to the weight of a theme: the first bytes say + /// what it is, and the interface sends the file before it says so. A + /// file that is neither, or is not what the form says, is refused as + /// soon as they have come; a theme is let run to its own weight. + #[tokio::test] + async fn an_upload_is_cut_off_as_it_is_read_at_the_weight_of_what_it_is() { + let dir = crate::testing::scratch_path("media-streamed"); + let state = server(&dir).await; + let id = a_film(&state, "Streamed Upload Film", "https://p/streamed.jpg").await; + let image_cap = file::MAX_IMAGE_BYTES as usize; + let picture = png(7); + + // A picture of 26 MiB: stopped a chunk past the cap, the rest of the + // request never read. + let (status, _, pulled, all) = upload_streamed( + &state, + &id, + (&[("kind", "image")], &[]), + &picture, + image_cap + 1024 * 1024, + ) + .await; + assert_eq!(status, StatusCode::PAYLOAD_TOO_LARGE); + assert!( + pulled <= image_cap / (64 * 1024) + 8 && pulled < all - 8, + "{pulled} of {all} chunks read" + ); + + // The same with the kind sent after the file, as the interface does. + let (status, _, pulled, all) = upload_streamed( + &state, + &id, + (&[], &[("coverType", "poster"), ("kind", "image")]), + &picture, + image_cap + 1024 * 1024, + ) + .await; + assert_eq!(status, StatusCode::PAYLOAD_TOO_LARGE); + assert!(pulled < all - 8, "{pulled} of {all} chunks read"); + + // Neither a picture nor a sound: refused with the first bytes. + let (status, _, pulled, all) = + upload_streamed(&state, &id, (&[], &[]), b"", None).is_err()); + assert!(limit_for_head(b"", None).is_err()); + assert_eq!(max_bytes(Kind::Image), MAX_IMAGE_BYTES); + assert_eq!(max_bytes(Kind::Audio), MAX_AUDIO_BYTES); + } + fn jpeg(width: u32, height: u32) -> Vec { let mut out = Vec::new(); image::RgbImage::from_fn(width, height, |x, _| { diff --git a/src/media/mod.rs b/src/media/mod.rs index e575519..2fdd8b1 100644 --- a/src/media/mod.rs +++ b/src/media/mod.rs @@ -34,6 +34,13 @@ pub use file::valid_key; /// The path the media are served under, on this server. pub const ROUTE: &str = "/media/"; +/// How many copies of pictures are being made for the pages that ask for +/// them at once — those waiting for their turn to be made counted: a page's +/// first visit asks for a few dozen, and two are drawn at a time. A page +/// that asks while as many are being made is sent the picture itself, for +/// that time. +pub const COPIES_AT_ONCE: usize = 32; + /// The fields whose locked value is an address: a still put on an episode by /// hand, a theme put on a work, the poster and background a work or a season /// is to lead with. @@ -50,6 +57,9 @@ pub struct Media { pub guard: crate::outbound::Guard, /// Rung by whatever put an address in line. pub notify: tokio::sync::Notify, + /// The places for copies being made as pages ask for them, taken before + /// anything is read for one: see [`COPIES_AT_ONCE`] and [`serve`]. + pub copying: Arc, /// What the served addresses begin with: the public URL, so that a /// client elsewhere can follow them, or nothing, for the interface alone. base: String, @@ -155,6 +165,7 @@ impl Media { http: crate::outbound::guarded_client(guard)?, guard, notify: tokio::sync::Notify::new(), + copying: Arc::new(tokio::sync::Semaphore::new(COPIES_AT_ONCE)), base: public_url .map(|u| u.trim_end_matches('/').to_string()) .unwrap_or_default(), diff --git a/src/media/placeholder.rs b/src/media/placeholder.rs index 54a6b76..184e3d5 100644 --- a/src/media/placeholder.rs +++ b/src/media/placeholder.rs @@ -58,11 +58,21 @@ pub fn parse_thumbhash(text: &str) -> Option> { (5..=32).contains(&bytes.len()).then_some(bytes) } -/// The placeholder of a decoded picture. +/// The placeholder of a decoded picture: drawn from its pixels where they +/// lie, the sketch being the only copy made — a hundred pixels on a side at +/// the most, where a copy of the whole as RGBA was a third as much again as +/// the picture, on top of it. pub fn of(decoded: &Decoded) -> Result { - let rgba = decoded.rgba(); + let pixels = decoded.as_pixels(); let (width, height) = fit(decoded.width, decoded.height); - let sketch = area_average(&rgba, decoded.width, decoded.height, width, height)?; + let sketch = area_average( + pixels.data, + pixels.channels(), + decoded.width, + decoded.height, + width, + height, + )?; let thumbhash = thumbhash::rgba_to_thumb_hash(width as usize, height as usize, &sketch); let color = dominant(&sketch).unwrap_or([0, 0, 0]); Ok(Placeholder { color, thumbhash }) @@ -85,10 +95,22 @@ fn fit(width: u32, height: u32) -> (u32, u32) { /// A picture made smaller by averaging every pixel of each area into one — /// the colours weighed by their alpha, so that what is transparent lends -/// none of its colour — in whole numbers, the same everywhere. -fn area_average(rgba: &[u8], width: u32, height: u32, to_w: u32, to_h: u32) -> Result> { +/// none of its colour — in whole numbers, the same everywhere. `pixels` are +/// RGB or RGBA, `channels` bytes each (3 or 4), RGB being opaque; the +/// sketch is always RGBA. +fn area_average( + pixels: &[u8], + channels: usize, + width: u32, + height: u32, + to_w: u32, + to_h: u32, +) -> Result> { ensure!( - rgba.len() == width as usize * height as usize * 4 && to_w > 0 && to_h > 0, + (channels == 3 || channels == 4) + && pixels.len() == width as usize * height as usize * channels + && to_w > 0 + && to_h > 0, "the pixels are not the size the picture says" ); let (width, height, to_w, to_h) = ( @@ -111,11 +133,15 @@ fn area_average(rgba: &[u8], width: u32, height: u32, to_w: u32, to_h: u32) -> R let (mut r, mut g, mut b, mut a, mut n) = (0u64, 0u64, 0u64, 0u64, 0u64); for y in y0..y1.min(height) { for x in x0..x1.min(width) { - let at = (y * width + x) * 4; - let alpha = u64::from(rgba[at + 3]); - r += u64::from(rgba[at]) * alpha; - g += u64::from(rgba[at + 1]) * alpha; - b += u64::from(rgba[at + 2]) * alpha; + let at = (y * width + x) * channels; + let alpha = if channels == 4 { + u64::from(pixels[at + 3]) + } else { + 255 + }; + r += u64::from(pixels[at]) * alpha; + g += u64::from(pixels[at + 1]) * alpha; + b += u64::from(pixels[at + 2]) * alpha; a += alpha; n += 1; } diff --git a/src/media/serve.rs b/src/media/serve.rs index 858aedb..576ecb2 100644 --- a/src/media/serve.rs +++ b/src/media/serve.rs @@ -14,8 +14,21 @@ //! while `media.webp` is on: off, one kept already is served, and a page //! that asks for one that is not is sent to the picture itself — for this //! time only, as they may be turned on again. +//! +//! Making one is work, so it is held to what the rest of the server's is: +//! the caller who asks for one that is not kept is counted against the +//! requests an address may make ([`crate::auth::ratelimit::spend`]) — a +//! copy served is not counted, nor is a picture —, at most +//! [`super::COPIES_AT_ONCE`] are being made at once, and a copy waits for its +//! turn among the pictures decoded before its picture is read from the +//! store, not after. -use std::{ops::Range, sync::LazyLock, time::Duration}; +use std::{ + net::{IpAddr, Ipv4Addr}, + ops::Range, + sync::{Arc, LazyLock}, + time::Duration, +}; use anyhow::Context as _; use axum::{ @@ -27,7 +40,7 @@ use axum::{ use bytes::Bytes; use object_store::GetRange; -use crate::{db::repo, state::AppState}; +use crate::{api::extract::ClientIp, db::repo, error::AppError, state::AppState}; use super::{ file::{self, Made}, @@ -43,11 +56,47 @@ const PRESIGNED_FOR: Duration = Duration::from_secs(3600); /// names the bytes. const IMMUTABLE: &str = "public, max-age=31536000, immutable"; +/// What a request that is counted is called in the log: the route, never +/// the key, which is the one thing a reader needs to fetch what it names. +const LOGGED_AS: &str = "/media/{key}"; + +/// `GET /media/{key}`, for a caller whose address is known: what the route +/// answers. A copy this asks for that is not kept is made on its account, +/// counted against what its address may ask of the server. +/// +/// What the route in `web.rs` is mounted on: [`get`] cannot be, having no +/// caller to count — and is called as it is from inside this server. +pub async fn get_for_caller( + State(state): State, + method: Method, + Path(key): Path, + ip: ClientIp, + headers: HeaderMap, +) -> Response { + // As the rate limiter counts it: an address by its network, and a + // caller whose address is unknown with every other unknown one. + let caller = + ip.0.map_or(IpAddr::V4(Ipv4Addr::UNSPECIFIED), crate::auth::ip::bucket); + respond(&state, method, key, Some(caller), headers).await +} + +/// The same, for what asks of this server from inside it — the TMDB +/// pictures' gate, for the picture a key names: nobody to count. pub async fn get( State(state): State, method: Method, Path(key): Path, headers: HeaderMap, +) -> Response { + respond(&state, method, key, None, headers).await +} + +async fn respond( + state: &AppState, + method: Method, + key: String, + caller: Option, + headers: HeaderMap, ) -> Response { let Some(parsed) = file::parse_key(&key) else { return StatusCode::NOT_FOUND.into_response(); @@ -56,7 +105,7 @@ pub async fn get( return StatusCode::NOT_FOUND.into_response(); }; if matches!(parsed.made, Made::Width(_) | Made::Square) { - return copy(&state, store, &method, parsed, &headers).await; + return copy(state, store, &method, parsed, caller, &headers).await; } // A thumbnail is looked up by its picture's key. @@ -210,6 +259,7 @@ async fn copy( store: &Store, method: &Method, asked: file::Key<'_>, + caller: Option, headers: &HeaderMap, ) -> Response { let not_found = || StatusCode::NOT_FOUND.into_response(); @@ -254,7 +304,7 @@ async fn copy( if format == Format::Avif { return avif( - state, store, method, &original, asked.stem, made, &key, headers, + state, store, method, &original, asked.stem, made, &key, caller, headers, ) .await; } @@ -290,9 +340,21 @@ async fn copy( }; } - match kept_or_made(state, store, &original, asked.stem, made, format, &key).await { + match kept_or_made( + state, store, &original, asked.stem, made, format, &key, caller, + ) + .await + { Ok(bytes) => whole(method, format_type(format), &key, &etag, bytes, IMMUTABLE), - Err(e) => { + // Counted against what the address may ask, and spent: told so. + Err(Unmade::Spent(refusal)) => refusal.into_response(), + // As many are being made as there is room for: the picture itself, + // for now, and the copy asked for again. + Err(Unmade::Busy) => { + tracing::debug!(key, "too many copies are being made; the picture is sent"); + redirect(&state.media.url_for(&original), "no-store") + } + Err(Unmade::Failed(e)) => { tracing::warn!( key, error = format_args!("{e:#}"), @@ -318,6 +380,7 @@ async fn avif( stem: &str, made: Made, key: &str, + caller: Option, headers: &HeaderMap, ) -> Response { let etag = format!("\"{key}\""); @@ -348,7 +411,18 @@ async fn avif( return redirect(&state.media.url_for(original), "no-cache"); } let webp = file::made_key(stem, made, Format::Webp.ext()); - match kept_or_made(state, store, original, stem, made, Format::Webp, &webp).await { + match kept_or_made( + state, + store, + original, + stem, + made, + Format::Webp, + &webp, + caller, + ) + .await + { Ok(bytes) => whole( method, "image/webp", @@ -357,12 +431,61 @@ async fn avif( bytes, "no-cache", ), - Err(_) => StatusCode::NOT_FOUND.into_response(), + Err(Unmade::Spent(refusal)) => refusal.into_response(), + Err(Unmade::Busy) => redirect(&state.media.url_for(original), "no-store"), + Err(Unmade::Failed(_)) => StatusCode::NOT_FOUND.into_response(), + } +} + +/// Why a copy was not given. +enum Unmade { + /// The caller has spent what its address may ask of this server. + Spent(AppError), + /// As many copies are being made as there are places for. + Busy, + /// It could not be made. + Failed(anyhow::Error), +} + +impl From for Unmade { + fn from(e: anyhow::Error) -> Self { + Self::Failed(e) + } +} + +/// Why making a copy stopped — what the several who asked for the same one +/// at once are each told. +enum Skipped { + Busy, + Failed(anyhow::Error), +} + +impl From for Skipped { + fn from(e: anyhow::Error) -> Self { + Self::Failed(e) + } +} + +impl From> for Unmade { + fn from(skipped: Arc) -> Self { + match &*skipped { + Skipped::Busy => Self::Busy, + Skipped::Failed(e) => Self::Failed(anyhow::anyhow!("{e:#}")), + } } } /// A copy's bytes: the store's, or made now, put there and noted — once, /// however many ask for it at the same moment. +/// +/// One that is made is work, so what asks for it is held to what the server +/// holds the rest of its work to: the caller is counted against what its +/// address may ask, once for each request that finds the copy not kept; +/// no more than [`super::COPIES_AT_ONCE`] are made at a time; and the copy +/// waits for its turn among the pictures decoded *before* its picture is +/// read from the store, so that what waits is a request and not a picture of +/// up to 25 MiB for each. +#[allow(clippy::too_many_arguments)] async fn kept_or_made( state: &AppState, store: &Store, @@ -371,11 +494,28 @@ async fn kept_or_made( made: Made, format: Format, key: &str, -) -> anyhow::Result { + caller: Option, +) -> Result { if let Some(bytes) = store.read(key).await? { return Ok(bytes); } + // Not kept: this request is one that makes it, or waits for one that does. + if let Some(peer) = caller { + crate::auth::ratelimit::spend(state, peer, LOGGED_AS) + .await + .map_err(Unmade::Spent)?; + } MADE.try_get_with(key.to_string(), async { + // A place among the copies being made, those waiting for a turn + // counted: the copy's own work holds it until it is done. + let place = state + .media + .copying + .clone() + .try_acquire_owned() + .map_err(|_| Skipped::Busy)?; + // The turn first, then the picture. + let turn = variants::turn().await?; let bytes = store .read(original) .await? @@ -387,11 +527,13 @@ async fn kept_or_made( .unwrap_or(None), None => None, }; - let copy = variants::off_thread(move || { - let decoded = variants::Decoded::read(&bytes)?; - variants::make(&decoded, made, format, cover) - }) - .await?; + let copy = turn + .run(move || { + let _place = place; + let decoded = variants::Decoded::read(&bytes)?; + variants::make(&decoded, made, format, cover) + }) + .await?; // Filed under what was made: a width asked of a picture whose own // was not known may have come out narrower. let filed = copy.key(stem); @@ -413,10 +555,10 @@ async fn kept_or_made( { tracing::warn!(key = filed, error = %e, "could not note a copy made"); } - Ok::<_, anyhow::Error>(made_bytes) + Ok::<_, Skipped>(made_bytes) }) .await - .map_err(|e| anyhow::anyhow!("{e:#}")) + .map_err(Unmade::from) } fn format_type(format: Format) -> &'static str { @@ -755,7 +897,13 @@ mod tests { /// The whole server, on a database of its own, keeping its media in /// `dir`. async fn server(dir: &std::path::Path) -> AppState { + limited_server(dir, 600).await + } + + /// The same, an address of which may make `per_minute` requests a minute. + async fn limited_server(dir: &std::path::Path, per_minute: u32) -> AppState { let mut config = crate::config::Config::from_env().expect("a configuration"); + config.security.rate_limit_per_minute = per_minute; config.mode = crate::config::Mode::Single; config.database = crate::db::testing::database().await; config.security.bootstrap_admin = None; @@ -878,6 +1026,157 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + /// The route is mounted on the handler that knows its caller: on [`get`], + /// which is for what asks from inside the server, no copy made would be + /// counted against anybody. + #[test] + fn the_media_route_is_mounted_on_the_handler_that_counts_its_caller() { + let web = include_str!("../web.rs"); + let route = web + .lines() + .find(|line| line.contains(".route(\"/media/{key}\"")) + .expect("the route of the media kept"); + assert!( + route.contains("crate::media::serve::get_for_caller"), + "web.rs mounts {route:?}: no copy made would be counted against anybody — mount \ + `.route(\"/media/{{key}}\", get(crate::media::serve::get_for_caller))`" + ); + } + + /// A copy asked for by a caller whose address is known, as the route + /// asks it. + async fn asked_by_a_caller(state: &AppState, key: &str) -> Response { + get_for_caller( + State(state.clone()), + Method::GET, + Path(key.to_string()), + ClientIp::NONE, + HeaderMap::new(), + ) + .await + } + + /// A caller who asks for a copy that is not kept is counted against what + /// its address may ask — once, and for the copy that is made alone: the + /// copy kept is served, and so is the picture, whatever has been spent. + /// Spent, it is told so, and nothing is made. + #[tokio::test] + async fn a_caller_who_has_a_copy_made_is_counted_and_one_who_has_none_is_not() { + let dir = crate::testing::scratch_path("serve-counted"); + let state = limited_server(&dir, 2).await; + let store = state.media.store().unwrap().clone(); + turn(&state, "media.webp", true).await; + let mut pictures = Vec::new(); + for shade in [81, 82, 83] { + pictures.push(kept(&state, shade).await); + } + + // Two copies are made, and counted: the quota is two. + for (_, sha) in &pictures[..2] { + let answer = asked_by_a_caller(&state, &format!("{sha}-480.webp")).await; + assert_eq!(answer.status(), StatusCode::OK, "{sha}"); + } + // The third is not made: the address has spent what it may ask. + let third = format!("{}-480.webp", pictures[2].1); + let answer = asked_by_a_caller(&state, &third).await; + assert_eq!(answer.status(), StatusCode::TOO_MANY_REQUESTS); + assert!(!store.exists(&third).await.unwrap(), "nothing was made"); + assert!(webp_copies(&state, &pictures[2].1).await.is_empty()); + + // What is kept is served all the same, and so are the pictures + // themselves: neither is counted. + let first = format!("{}-480.webp", pictures[0].1); + for _ in 0..3 { + let answer = asked_by_a_caller(&state, &first).await; + assert_eq!(answer.status(), StatusCode::OK); + let answer = asked_by_a_caller(&state, &pictures[2].0).await; + assert_eq!(answer.status(), StatusCode::OK); + } + // What asks from inside the server has no caller to count. + let answer = asked(&state, &third).await; + assert_eq!(answer.status(), StatusCode::OK); + assert!(store.exists(&third).await.unwrap()); + + let _ = std::fs::remove_dir_all(&dir); + } + + /// As many copies are being made as there are places for: one more is + /// not made — the page is sent the picture itself, for now, and the copy + /// is made when it asks again. + #[tokio::test] + async fn no_more_copies_are_made_at_once_than_there_are_places_for() { + let dir = crate::testing::scratch_path("serve-places"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + turn(&state, "media.webp", true).await; + let (key, sha) = kept(&state, 91).await; + let copy = format!("{sha}-480.webp"); + + let places = state + .media + .copying + .clone() + .acquire_many_owned(crate::media::COPIES_AT_ONCE as u32) + .await + .unwrap(); + let answer = asked(&state, ©).await; + assert_eq!(answer.status(), StatusCode::FOUND); + assert_eq!(answer.headers()[header::LOCATION], format!("/media/{key}")); + assert_eq!(answer.headers()[header::CACHE_CONTROL], "no-store"); + assert!(!store.exists(©).await.unwrap(), "nothing made"); + assert!(webp_copies(&state, &sha).await.is_empty(), "nothing noted"); + + drop(places); + let answer = asked(&state, ©).await; + assert_eq!(answer.status(), StatusCode::OK, "made once there is room"); + assert!(store.exists(©).await.unwrap()); + // The places are given back once a copy is made. + assert_eq!( + state.media.copying.available_permits(), + crate::media::COPIES_AT_ONCE + ); + + let _ = std::fs::remove_dir_all(&dir); + } + + /// A copy waits for its turn among the pictures decoded before its + /// picture is read from the store: while every turn is taken, the + /// request is a request waiting and not a picture of up to 25 MiB held + /// in memory with it. The picture is gone from the store here, so that + /// a read made too soon is told: it would answer at once. + #[tokio::test] + async fn a_copy_waits_for_its_turn_before_its_picture_is_read() { + let dir = crate::testing::scratch_path("serve-turn"); + let state = server(&dir).await; + let store = state.media.store().unwrap().clone(); + turn(&state, "media.webp", true).await; + let (key, sha) = kept(&state, 61).await; + store.delete(&key).await.unwrap(); + let copy = format!("{sha}-480.webp"); + + let turns = variants::every_turn().await; + let asking = tokio::spawn({ + let state = state.clone(); + let copy = copy.clone(); + async move { asked(&state, ©).await } + }); + tokio::time::sleep(Duration::from_millis(300)).await; + assert!( + !asking.is_finished(), + "the picture was read before the turn was had: it is not there, and the request is over" + ); + drop(turns); + let answer = tokio::time::timeout(Duration::from_secs(10), asking) + .await + .expect("answered once there is a turn") + .unwrap(); + // The picture is not in the store: the page is sent it all the same. + assert_eq!(answer.status(), StatusCode::FOUND); + assert_eq!(answer.headers()[header::CACHE_CONTROL], "no-store"); + + let _ = std::fs::remove_dir_all(&dir); + } + /// An AVIF of 480 of the picture of `sha`, noted and offered, whose /// file is not there: a store moved, say. Its key. async fn avif_without_its_file(state: &AppState, sha: &str) -> String { diff --git a/src/media/variants.rs b/src/media/variants.rs index 67e1421..d49522a 100644 --- a/src/media/variants.rs +++ b/src/media/variants.rs @@ -17,7 +17,7 @@ use std::{ panic::AssertUnwindSafe, - sync::{Arc, LazyLock}, + sync::{Arc, Condvar, LazyLock, Mutex, PoisonError}, }; use anyhow::{Context as _, Result, bail}; @@ -80,9 +80,14 @@ pub fn avif_widths(cover: Option) -> &'static [u32] { } /// The width a copy asked for is made at: the width asked, never past the -/// picture's own — `None` for a width the interface never asks for. +/// picture's own — `None` for a width the interface never asks for. The +/// picture's own width is one it asks for, and one made, only where the +/// picture is no wider than the widest copy: past that, "its own" is a +/// request for the whole of a picture of any size, drawn and written again +/// at full size. pub fn width_made(asked: u32, own: u32) -> Option { - (WIDTHS.contains(&asked) || asked == own).then(|| asked.min(own)) + let widest = WIDTHS[WIDTHS.len() - 1]; + (WIDTHS.contains(&asked) || (asked == own && own <= widest)).then(|| asked.min(own)) } /// What a copy is written as. @@ -127,45 +132,45 @@ pub struct Decoded { pub width: u32, pub height: u32, pub alpha: bool, + /// What reading the picture reserved of the memory the pictures decoding + /// may hold together, kept for as long as the picture is: given back + /// when it is dropped. Nothing for a copy of it made smaller, which is + /// within what was reserved for the whole. + _held: Option, } impl Decoded { - /// Read a picture's bytes — see [`file::decode`] for the bounds. + /// Read a picture's bytes — see [`file::decode_with`] for the bounds. + /// Waits, on the thread it is called from, for the memory its picture + /// needs to be there: see [`Budget`]. pub fn read(bytes: &[u8]) -> Result { - Ok(Self::of(file::decode(bytes)?)) + let (picture, held) = file::decode_with(bytes, true, |weight| BUDGET.take(weight))?; + Ok(Self::of(picture, Some(held))) } /// Read a picture's bytes `side` or more on one of its sides, smaller - /// than it is where its format allows: see [`file::decode_small`]. + /// than it is where its format allows: see [`file::decode_small_with`]. pub fn read_small(bytes: &[u8], side: u32) -> Result { - Ok(Self::of(file::decode_small(bytes, side)?)) + let (picture, held) = file::decode_small_with(bytes, side, |weight| BUDGET.take(weight))?; + Ok(Self::of(picture, Some(held))) } - fn of(picture: image::DynamicImage) -> Self { + fn of(picture: image::DynamicImage, held: Option) -> Self { let (width, height) = (picture.width(), picture.height()); // An alpha channel that is opaque everywhere is a photograph like // any other; only real transparency is kept. if picture.color().has_alpha() { - let rgba = picture.into_rgba8().into_raw(); - if rgba.as_chunks::<4>().0.iter().any(|p| p[3] < 255) { - return Self { - pixels: rgba, - width, - height, - alpha: true, - }; + let mut rgba = picture.into_rgba8().into_raw(); + let transparent = rgba.as_chunks::<4>().0.iter().any(|p| p[3] < 255); + if !transparent { + drop_alpha(&mut rgba); } - let rgb = rgba - .as_chunks::<4>() - .0 - .iter() - .flat_map(|p| [p[0], p[1], p[2]]) - .collect(); return Self { - pixels: rgb, + pixels: rgba, width, height, - alpha: false, + alpha: transparent, + _held: held, }; } Self { @@ -173,10 +178,11 @@ impl Decoded { width, height, alpha: false, + _held: held, } } - fn as_pixels(&self) -> Pixels<'_> { + pub fn as_pixels(&self) -> Pixels<'_> { Pixels { data: &self.pixels, width: self.width, @@ -195,6 +201,7 @@ impl Decoded { width, height, alpha: self.alpha, + _held: None, }); } let kind = if self.alpha { @@ -220,6 +227,7 @@ impl Decoded { width, height, alpha: self.alpha, + _held: None, }) } @@ -232,6 +240,7 @@ impl Decoded { width: self.width, height: self.height, alpha: false, + _held: None, }; } let pixels = self @@ -249,6 +258,7 @@ impl Decoded { width: self.width, height: self.height, alpha: false, + _held: None, } } @@ -257,19 +267,17 @@ impl Decoded { fn is_portrait(&self) -> bool { self.height > self.width } +} - /// RGBA, whatever it is: what the placeholder is worked out from. - pub fn rgba(&self) -> Vec { - if self.alpha { - return self.pixels.clone(); - } - self.pixels - .as_chunks::<3>() - .0 - .iter() - .flat_map(|p| [p[0], p[1], p[2], 255]) - .collect() +/// RGBA made RGB where it lies, each pixel's three colours moved down over +/// the alpha of those before it: a second buffer would be held beside the +/// first at the moment both are whole, a third as much again as the picture. +fn drop_alpha(pixels: &mut Vec) { + let count = pixels.len() / 4; + for at in 0..count { + pixels.copy_within(at * 4..at * 4 + 3, at * 3); } + pixels.truncate(count * 3); } /// A copy made. @@ -334,7 +342,8 @@ pub fn make( (Made::Width(asked), Format::Webp | Format::Avif) => { let width = asked.min(decoded.width); let height = height_at(decoded, width); - let small = decoded.resized(width, height)?; + let mut resized = None; + let small = drawn_at(decoded, width, height, &mut resized)?; let bytes = if format == Format::Avif { encode::avif(&small.as_pixels())? } else { @@ -367,6 +376,22 @@ pub fn make( } } +/// A picture at the size a copy is made at: itself where it is that size +/// already — a copy at a picture's own width copies none of its pixels, which +/// would be a second of them beside the first — and otherwise resized into +/// `resized`, whose picture is returned. +fn drawn_at<'a>( + decoded: &'a Decoded, + width: u32, + height: u32, + resized: &'a mut Option, +) -> Result<&'a Decoded> { + if (width, height) == (decoded.width, decoded.height) { + return Ok(decoded); + } + Ok(resized.insert(decoded.resized(width, height)?)) +} + /// What preparing a picture as it is stored came to: its placeholder, and /// the copy its kind is first asked for, when one was made. pub struct Prepared { @@ -402,7 +427,8 @@ pub fn prepare(bytes: &[u8], cover: Option, webp: bool) -> Result
> =
     LazyLock::new(|| Arc::new(tokio::sync::Semaphore::new(AT_ONCE)));
@@ -411,34 +437,156 @@ static DECODING: LazyLock> =
 static AVIF: LazyLock> =
     LazyLock::new(|| Arc::new(tokio::sync::Semaphore::new(1)));
 
-/// Run `work` — a decoding, a resizing, an encoding — off the server's
-/// threads, as many at once as [`AT_ONCE`] allows, and a decoder that
-/// panics on the bytes it was given as a picture refused rather than a
-/// thread brought down.
+/// What the pictures being decoded may hold in memory together
+/// ([`file::DECODE_BUDGET`]), however many the gate lets through and
+/// whatever they are: each takes the share its picture's worst case needs
+/// ([`file::decode_with`] counts it from the header) before a byte is
+/// allocated for it, and gives it back when what it decoded is dropped.
+/// Taken in turn, so that a large picture is not kept waiting for ever by a
+/// string of small ones; the thread that waits is the one the work runs on
+/// (see [`Turn::run`]), of which [`AT_ONCE`] and the night's one are all
+/// there are.
+struct Budget {
+    limit: u64,
+    shares: Mutex,
+    changed: Condvar,
+}
+
+#[derive(Default)]
+struct Shares {
+    /// What is held.
+    used: u64,
+    /// The next place in line to hand out, and the one whose turn it is.
+    next: u64,
+    serving: u64,
+}
+
+static BUDGET: LazyLock> = LazyLock::new(|| Arc::new(Budget::new(file::DECODE_BUDGET)));
+
+impl Budget {
+    fn new(limit: u64) -> Self {
+        Self {
+            limit,
+            shares: Mutex::new(Shares::default()),
+            changed: Condvar::new(),
+        }
+    }
+
+    /// A share of `bytes`, waited for until it fits beside those held and
+    /// every share asked for before it has been given. One larger than the
+    /// whole is the whole: the pictures that need more are refused before
+    /// they ask.
+    fn take(self: &Arc, bytes: u64) -> Held {
+        let bytes = bytes.min(self.limit);
+        let mut shares = self.shares.lock().unwrap_or_else(PoisonError::into_inner);
+        let place = shares.next;
+        shares.next += 1;
+        while shares.serving != place || shares.used + bytes > self.limit {
+            shares = self
+                .changed
+                .wait(shares)
+                .unwrap_or_else(PoisonError::into_inner);
+        }
+        shares.used += bytes;
+        shares.serving += 1;
+        // The one behind may fit beside this.
+        self.changed.notify_all();
+        Held {
+            budget: self.clone(),
+            bytes,
+        }
+    }
+}
+
+/// A share of the [`Budget`], given back when it is dropped.
+pub struct Held {
+    budget: Arc,
+    bytes: u64,
+}
+
+impl Drop for Held {
+    fn drop(&mut self) {
+        let mut shares = self
+            .budget
+            .shares
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner);
+        shares.used = shares.used.saturating_sub(self.bytes);
+        self.budget.changed.notify_all();
+    }
+}
+
+/// A place among the pictures decoded at once ([`AT_ONCE`]), held until the
+/// work it was waited for is done. Taken before the picture is read from
+/// the store, so that no more pictures sit in memory waiting to be decoded
+/// than are being decoded.
+pub struct Turn(tokio::sync::OwnedSemaphorePermit);
+
+/// Wait for a place among the pictures decoded at once.
+pub async fn turn() -> Result {
+    let permit = DECODING
+        .clone()
+        .acquire_owned()
+        .await
+        .context("the pictures' gate is closed")?;
+    Ok(Turn(permit))
+}
+
+/// Every place at once, for a test that fills the gate: held by one test at a
+/// time, since two of them, each holding one of the two places, would wait
+/// for each other for ever.
+#[cfg(test)]
+pub async fn every_turn() -> (tokio::sync::MutexGuard<'static, ()>, [Turn; AT_ONCE]) {
+    static FILLING: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
+    let filling = FILLING.lock().await;
+    let mut turns = Vec::new();
+    for _ in 0..AT_ONCE {
+        turns.push(turn().await.expect("a place"));
+    }
+    let turns = turns.try_into().ok().expect("as many places as there are");
+    (filling, turns)
+}
+
+impl Turn {
+    /// Run `work` — a decoding, a resizing, an encoding — off the server's
+    /// threads, holding this place until it is done, and a decoder that
+    /// panics on the bytes it was given as a picture refused rather than a
+    /// thread brought down.
+    pub async fn run(
+        self,
+        work: impl FnOnce() -> Result + Send + 'static,
+    ) -> Result {
+        run_holding(self.0, work).await
+    }
+}
+
+/// Run `work` off the server's threads, as many at once as [`AT_ONCE`]
+/// allows, and a decoder that panics on the bytes it was given as a picture
+/// refused rather than a thread brought down.
 pub async fn off_thread(
     work: impl FnOnce() -> Result + Send + 'static,
 ) -> Result {
-    run_under(&DECODING, work).await
+    turn().await?.run(work).await
 }
 
 /// The same, for an AVIF copy: one at a time.
 pub async fn off_thread_avif(
     work: impl FnOnce() -> Result + Send + 'static,
 ) -> Result {
-    run_under(&AVIF, work).await
+    let permit = AVIF
+        .clone()
+        .acquire_owned()
+        .await
+        .context("the pictures' gate is closed")?;
+    run_holding(permit, work).await
 }
 
-async fn run_under(
-    gate: &Arc,
+async fn run_holding(
+    permit: tokio::sync::OwnedSemaphorePermit,
     work: impl FnOnce() -> Result + Send + 'static,
 ) -> Result {
     // Held by the work itself, so that a request given up on meanwhile does
     // not let another start beside it.
-    let permit = gate
-        .clone()
-        .acquire_owned()
-        .await
-        .context("the pictures' gate is closed")?;
     let joined = tokio::task::spawn_blocking(move || {
         let _permit = permit;
         std::panic::catch_unwind(AssertUnwindSafe(work))
@@ -506,6 +654,13 @@ mod tests {
             "no width the interface never asks for"
         );
         assert_eq!(width_made(10_000, 20_000), None);
+        // Its own width is a copy only where it is no wider than the widest:
+        // past that it would be a request for the whole of a picture of any
+        // size, drawn and written again.
+        assert_eq!(width_made(1920, 1920), Some(1920));
+        assert_eq!(width_made(2000, 2000), None);
+        assert_eq!(width_made(9_000, 9_000), None);
+        assert_eq!(width_made(1920, 3000), Some(1920));
     }
 
     #[test]
@@ -570,12 +725,49 @@ mod tests {
         assert_eq!(back.get_pixel(20, 70)[3], 0, "transparent where it was");
         assert_eq!(back.get_pixel(380, 70)[3], 255, "opaque where it was");
 
-        // An alpha channel opaque everywhere is no transparency.
+        // An alpha channel opaque everywhere is no transparency, and the
+        // colours are what they were, closed up where they lay.
         let mut opaque = Vec::new();
-        image::RgbaImage::from_pixel(50, 50, image::Rgba([1, 2, 3, 255]))
-            .write_to(&mut Cursor::new(&mut opaque), image::ImageFormat::Png)
-            .unwrap();
-        assert!(!Decoded::read(&opaque).unwrap().alpha);
+        image::RgbaImage::from_fn(50, 40, |x, y| {
+            image::Rgba([(x * 5) as u8, (y * 6) as u8, 3, 255])
+        })
+        .write_to(&mut Cursor::new(&mut opaque), image::ImageFormat::Png)
+        .unwrap();
+        let decoded = Decoded::read(&opaque).unwrap();
+        assert!(!decoded.alpha);
+        let pixels = decoded.as_pixels();
+        assert_eq!((pixels.data.len(), pixels.channels()), (50 * 40 * 3, 3));
+        assert_eq!(&pixels.data[(7 * 50 + 9) * 3..][..3], [45, 42, 3]);
+        assert_eq!(&pixels.data[(39 * 50 + 49) * 3..][..3], [245, 234, 3]);
+    }
+
+    /// RGBA made RGB where it lies keeps every pixel's colours, in order.
+    #[test]
+    fn the_alpha_is_dropped_where_the_pixels_lie() {
+        let mut pixels = vec![1, 2, 3, 255, 4, 5, 6, 255, 7, 8, 9, 255];
+        drop_alpha(&mut pixels);
+        assert_eq!(pixels, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+        let mut none: Vec = Vec::new();
+        drop_alpha(&mut none);
+        assert!(none.is_empty());
+    }
+
+    /// A copy at the picture's own size draws the picture itself: no second
+    /// buffer of its pixels beside the first; any other size is resized.
+    #[test]
+    fn a_copy_at_the_pictures_own_size_copies_none_of_its_pixels() {
+        let decoded = Decoded::read(&png(300, 200)).unwrap();
+        let mut resized = None;
+        let same = drawn_at(&decoded, 300, 200, &mut resized).unwrap();
+        assert!(std::ptr::eq(same, &decoded));
+        assert!(resized.is_none());
+        let smaller = drawn_at(&decoded, 150, 100, &mut resized).unwrap();
+        assert!(!std::ptr::eq(smaller, &decoded));
+        assert_eq!((smaller.width, smaller.height), (150, 100));
+        // And the copy made is the same as before it was spared.
+        let copy = make(&decoded, Made::Width(300), Format::Webp, None).unwrap();
+        assert_eq!((copy.width, copy.height), (300, 200));
+        assert_eq!(size_of(©.bytes), (300, 200));
     }
 
     #[test]
@@ -637,6 +829,110 @@ mod tests {
         assert!(prepare(b"not a picture", Some(CoverType::Poster), false).is_err());
     }
 
+    /// Pictures decoding together hold no more than the budget: one that
+    /// does not fit waits for those that do to be done, and those behind it
+    /// wait their turn, though they would fit — so that a large picture is
+    /// not kept waiting by a string of small ones. A share is given back
+    /// when it is dropped.
+    #[test]
+    fn pictures_decoding_together_hold_no_more_than_the_budget_and_wait_in_turn() {
+        use std::{sync::mpsc, thread, time::Duration};
+
+        let budget = Arc::new(Budget::new(100));
+        let first = budget.take(60);
+        let held = |budget: &Budget| budget.shares.lock().unwrap().used;
+        assert_eq!(held(&budget), 60);
+
+        let (told, heard) = mpsc::channel();
+        let ask = |name: &'static str, bytes: u64| {
+            let (budget, told) = (budget.clone(), told.clone());
+            thread::spawn(move || {
+                let share = budget.take(bytes);
+                told.send(name).unwrap();
+                // Held a moment, as a picture is while it is decoded.
+                thread::sleep(Duration::from_millis(50));
+                drop(share);
+            })
+        };
+        // 60 does not fit beside 60; 30 would, and is behind it.
+        let large = ask("large", 60);
+        thread::sleep(Duration::from_millis(100));
+        let small = ask("small", 30);
+        assert!(
+            heard.recv_timeout(Duration::from_millis(300)).is_err(),
+            "nobody goes while the first holds its share, and the small one is in turn"
+        );
+        assert_eq!(held(&budget), 60);
+
+        drop(first);
+        let mut told_in_the_end = vec![
+            heard.recv_timeout(Duration::from_secs(5)).unwrap(),
+            heard.recv_timeout(Duration::from_secs(5)).unwrap(),
+        ];
+        told_in_the_end.sort_unstable();
+        assert_eq!(told_in_the_end, ["large", "small"]);
+        large.join().unwrap();
+        small.join().unwrap();
+        assert_eq!(held(&budget), 0, "every share is given back");
+
+        // A share larger than the whole is the whole, and never waits for
+        // more than there is.
+        let all = budget.take(500);
+        assert_eq!(all.bytes, 100);
+        assert_eq!(held(&budget), 100);
+        drop(all);
+        assert_eq!(held(&budget), 0);
+    }
+
+    /// What a picture reserves is its worst case from the header, taken
+    /// before it is decoded, held for as long as the picture is and given
+    /// back with it.
+    #[test]
+    fn what_a_picture_reserves_is_given_back_once_it_is_dropped() {
+        let budget = Arc::new(Budget::new(file::DECODE_BUDGET));
+        let used = || budget.shares.lock().unwrap().used;
+        let bytes = png(64, 48);
+        let (picture, held) =
+            file::decode_with(&bytes, true, |weight| budget.take(weight)).unwrap();
+        assert_eq!((picture.width(), picture.height()), (64, 48));
+        // 3 072 pixels of RGB, 9 bytes each with the copy to be made of
+        // them, the file's own bytes and the fixed ones.
+        assert_eq!(
+            held.bytes,
+            64 * 48 * 9 + bytes.len() as u64 + 8 * 1024 * 1024
+        );
+        assert_eq!(used(), held.bytes);
+        drop(held);
+        assert_eq!(used(), 0);
+
+        // A picture read for the copies carries its share, a copy of it
+        // made smaller none, being within it.
+        let decoded = Decoded::read(&bytes).unwrap();
+        assert!(decoded._held.is_some());
+        assert!(decoded.resized(32, 24).unwrap()._held.is_none());
+    }
+
+    #[tokio::test]
+    async fn a_turn_is_waited_for_and_held_by_the_work() {
+        // Every place taken: another waits.
+        let (filling, [one, two]) = every_turn().await;
+        let mut third = Box::pin(turn());
+        assert!(
+            tokio::time::timeout(std::time::Duration::from_millis(100), &mut third)
+                .await
+                .is_err(),
+            "no more than {AT_ONCE} are decoded at once"
+        );
+        // The work holds its place until it is done, whoever stops waiting.
+        let done = one.run(|| Ok(1)).await.unwrap();
+        assert_eq!(done, 1);
+        let third = tokio::time::timeout(std::time::Duration::from_secs(5), third)
+            .await
+            .expect("a place freed by the work")
+            .unwrap();
+        drop((two, third, filling));
+    }
+
     #[tokio::test]
     async fn work_that_panics_is_a_picture_refused() {
         let refused = off_thread(|| -> Result<()> { panic!("a decoder's bug") }).await;
diff --git a/src/memory.rs b/src/memory.rs
index 05cfef6..719773b 100644
--- a/src/memory.rs
+++ b/src/memory.rs
@@ -63,6 +63,7 @@ fn count(delta: isize) {
 unsafe impl GlobalAlloc for Counting {
     #[inline]
     unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
+        // nosemgrep: rust.lang.security.unsafe-usage.unsafe-usage — a GlobalAlloc has to call its allocator's unsafe methods, held by the SAFETY note
         let at = unsafe { mimalloc::MiMalloc.alloc(layout) };
         if !at.is_null() {
             count(layout.size() as isize);
@@ -72,6 +73,7 @@ unsafe impl GlobalAlloc for Counting {
 
     #[inline]
     unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
+        // nosemgrep: rust.lang.security.unsafe-usage.unsafe-usage — a GlobalAlloc has to call its allocator's unsafe methods, held by the SAFETY note
         let at = unsafe { mimalloc::MiMalloc.alloc_zeroed(layout) };
         if !at.is_null() {
             count(layout.size() as isize);
@@ -81,12 +83,14 @@ unsafe impl GlobalAlloc for Counting {
 
     #[inline]
     unsafe fn dealloc(&self, at: *mut u8, layout: Layout) {
+        // nosemgrep: rust.lang.security.unsafe-usage.unsafe-usage — a GlobalAlloc has to call its allocator's unsafe methods, held by the SAFETY note
         unsafe { mimalloc::MiMalloc.dealloc(at, layout) };
         count(-(layout.size() as isize));
     }
 
     #[inline]
     unsafe fn realloc(&self, at: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
+        // nosemgrep: rust.lang.security.unsafe-usage.unsafe-usage — a GlobalAlloc has to call its allocator's unsafe methods, held by the SAFETY note
         let moved = unsafe { mimalloc::MiMalloc.realloc(at, layout, new_size) };
         // A failed realloc leaves the old block where it was, still held.
         if !moved.is_null() {
diff --git a/src/outbound.rs b/src/outbound.rs
index edfc77d..007b9ab 100644
--- a/src/outbound.rs
+++ b/src/outbound.rs
@@ -397,6 +397,9 @@ fn guarded_builder(guard: Guard) -> reqwest::ClientBuilder {
         .timeout(std::time::Duration::from_secs(60))
         .connect_timeout(std::time::Duration::from_secs(10))
         .redirect(guard.redirect_policy())
+        // A redirect is followed without telling the next host where it
+        // came from: the address left may carry a key in its query.
+        .referer(false)
 }
 
 #[cfg(test)]
diff --git a/src/providers/tmdb/mod.rs b/src/providers/tmdb/mod.rs
index 6f8a608..08de957 100644
--- a/src/providers/tmdb/mod.rs
+++ b/src/providers/tmdb/mod.rs
@@ -163,7 +163,13 @@ impl TmdbClient {
         external_source: &str,
         external_id: &str,
     ) -> Result {
-        let url = format!("{}/find/{external_id}", self.base);
+        // One segment of the path, whatever the id holds: it is read from a
+        // request, a list, a feed, and is no address of anybody's choosing.
+        let url = format!(
+            "{}/find/{}",
+            self.base,
+            crate::providers::segment(external_id)
+        );
         let params = [
             ("external_source", external_source.to_string()),
             ("language", self.language()),
@@ -754,6 +760,57 @@ mod tests {
         assert_eq!(image_languages("ja"), "ja,en,null");
     }
 
+    /// What an id written into the path of `/find` is asked as: the path
+    /// the stand-in was sent, as it was written on the wire.
+    async fn asked_to_find(id: &str) -> Vec {
+        use std::sync::{Arc, Mutex};
+
+        use axum::{Router, extract::Request};
+
+        let seen = Arc::new(Mutex::new(Vec::new()));
+        let kept = Arc::clone(&seen);
+        let app = Router::new().fallback(move |request: Request| {
+            let kept = Arc::clone(&kept);
+            async move {
+                kept.lock().unwrap().push(request.uri().path().to_string());
+                axum::Json(serde_json::json!({ "movie_results": [], "tv_results": [] }))
+            }
+        });
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("a port");
+        let at = listener.local_addr().expect("its address");
+        tokio::spawn(async move {
+            let _ = axum::serve(listener, app).await;
+        });
+
+        let client = TmdbClient::new(
+            reqwest::Client::new(),
+            &config::Tmdb {
+                api_key: Some("test".into()),
+                upstream: format!("http://{at}"),
+                language: "en-US".into(),
+                include_adult: false,
+                search_limit: 5,
+                passthrough: true,
+            },
+        );
+        client.find("imdb_id", id).await.expect("answered");
+        seen.lock().unwrap().clone()
+    }
+
+    #[tokio::test]
+    async fn an_id_asked_for_is_one_segment_of_the_path_whatever_it_holds() {
+        assert_eq!(asked_to_find("tt0903747").await, ["/3/find/tt0903747"]);
+        assert_eq!(asked_to_find("81189").await, ["/3/find/81189"]);
+        // Not a way to another path of TMDB's — nor to a query of its own.
+        assert_eq!(
+            asked_to_find("../account/1?session_id=s#x").await,
+            ["/3/find/..%2Faccount%2F1%3Fsession_id%3Ds%23x"]
+        );
+        assert_eq!(asked_to_find("tt1/../../3/account").await.len(), 1);
+    }
+
     #[test]
     fn backfill_ignores_episodes_absent_from_the_fallback() {
         let mut target = season(vec![(1, 9, Some(""), None)]);
diff --git a/src/search/rank.rs b/src/search/rank.rs
index 7f132e9..8308d98 100644
--- a/src/search/rank.rs
+++ b/src/search/rank.rs
@@ -44,9 +44,26 @@ impl Form {
 /// that it can only be a number in the title.
 const YEARS: std::ops::RangeInclusive = 1870..=2100;
 
+/// The most characters a term is looked for by: what the list and the search
+/// of the native API take, and what the searches TMDB's, TheTVDB's, Plex's
+/// and Sonarr's clients make are given too, however long the term they send.
+/// A term longer is read by its first characters, folded or not.
+pub const TERM_LENGTH: usize = 200;
+
+/// `text`'s first [`TERM_LENGTH`] characters.
+fn clip(text: &str) -> &str {
+    match text.char_indices().nth(TERM_LENGTH) {
+        Some((at, _)) => &text[..at],
+        None => text,
+    }
+}
+
 impl Term {
     pub fn parse(text: &str) -> Self {
-        let whole = norm(text);
+        // Cut before it is folded, which a text of megabytes is not to cost,
+        // and after, since a character may fold into a dozen.
+        let folded = norm(clip(text));
+        let whole = clip(&folded).trim_end().to_string();
         let dated = whole.rsplit_once(' ').and_then(|(before, last)| {
             let year: i32 = last.parse().ok().filter(|y| YEARS.contains(y))?;
             (last.len() == 4 && !before.is_empty()).then(|| (Form::of(before.to_string()), year))
@@ -282,6 +299,38 @@ mod tests {
         assert!(sequel > first, "{sequel} against {first}");
     }
 
+    /// Whoever sends a term — the interface, TMDB's search, TheTVDB's, Plex,
+    /// Sonarr — it is read by its first two hundred characters, and a text
+    /// of megabytes costs what a long term does.
+    #[test]
+    fn a_term_is_read_by_its_first_two_hundred_characters() {
+        // Under the cap nothing changes.
+        assert_eq!(Term::parse("Dune 2021").whole.norm, "dune 2021");
+        let exact = "a".repeat(TERM_LENGTH);
+        assert_eq!(Term::parse(&exact).whole.norm, exact);
+
+        // One word and a year past the cap: the year is cut with the rest,
+        // and no title is read before it.
+        let long = format!("{exact} 2021");
+        let term = Term::parse(&long);
+        assert_eq!(term.whole.norm, exact);
+        assert!(term.dated.is_none());
+
+        // The cut never leaves a space at the end.
+        let spaced = format!("{} b", "a".repeat(TERM_LENGTH - 1));
+        assert_eq!(Term::parse(&spaced).whole.norm, "a".repeat(TERM_LENGTH - 1));
+
+        // A character that folds into many is cut after it is folded: the
+        // Arabic ligature of a whole invocation is eighteen of them.
+        let ligatures = "\u{FDFA}".repeat(TERM_LENGTH);
+        assert!(Term::parse(&ligatures).whole.norm.chars().count() <= TERM_LENGTH);
+
+        let huge = "ab ".repeat(1_000_000);
+        let term = Term::parse(&huge);
+        assert!(term.whole.norm.chars().count() <= TERM_LENGTH);
+        assert!(!term.is_empty());
+    }
+
     #[test]
     fn a_term_of_punctuation_alone_is_nothing_to_look_for() {
         assert!(Term::parse("%").is_empty());
diff --git a/src/service/ids.rs b/src/service/ids.rs
index dfed97c..73989e3 100644
--- a/src/service/ids.rs
+++ b/src/service/ids.rs
@@ -308,6 +308,14 @@ pub fn own_episode(names: &str) -> i64 {
     -(i64::from(hash % (i32::MAX as u32)) + 1)
 }
 
+/// Whether `id` is a number [`own_episode`] can make: negative, and no
+/// further from nought than the 32-bit column a client keeps it in. Anything
+/// else — a positive number, nought, `-9999999999` — was never made here, so
+/// nothing is looked for under it.
+pub fn is_own_episode(id: i64) -> bool {
+    (-i64::from(i32::MAX)..0).contains(&id)
+}
+
 /// A search term of the form `prefix:value`, which Sonarr and Radarr both use
 /// to look a title up by a specific provider's id.
 #[derive(Debug, PartialEq, Eq)]
@@ -650,6 +658,31 @@ mod tests {
         assert!(own_episode("") < 0);
     }
 
+    #[test]
+    fn only_a_number_an_episode_can_be_given_is_one_made_here() {
+        for names in [
+            "tmdb:170135",
+            "place:abc",
+            "season:tmdb:1",
+            "character:x",
+            "",
+        ] {
+            assert!(is_own_episode(own_episode(names)), "{names}");
+        }
+        // The ends of what it makes, and one past each.
+        assert!(is_own_episode(-1) && is_own_episode(-i64::from(i32::MAX)));
+        for never in [
+            0,
+            1,
+            990_701,
+            -i64::from(i32::MAX) - 1,
+            -9_999_999_999,
+            i64::MIN,
+        ] {
+            assert!(!is_own_episode(never), "{never}");
+        }
+    }
+
     #[test]
     fn prefixed_terms_are_recognised() {
         assert_eq!(classify("tvdb:81189"), TermLookup::Tvdb(81189));
diff --git a/src/service/series.rs b/src/service/series.rs
index de9e836..fe83fc8 100644
--- a/src/service/series.rs
+++ b/src/service/series.rs
@@ -214,6 +214,13 @@ pub async fn by_tmdb_id(state: &AppState, tmdb_id: i64) -> Result Result> {
+    // As a film's: an id that is none is no series', and never goes into a
+    // provider's address — it is written there, a path of TMDB's — nor into
+    // what is looked for.
+    let Some(imdb_id) = crate::domain::ids::normalize_imdb_id(imdb_id) else {
+        return Ok(None);
+    };
+    let imdb_id = imdb_id.as_str();
     if let Some(item) = local(state, ExternalSource::Imdb, imdb_id).await?
         && item.kind == MediaKind::Series
     {
@@ -1031,6 +1038,31 @@ mod tests {
         item
     }
 
+    /// An IMDb id comes from a request's body, a query, a list somebody
+    /// else keeps: one that is none is no series, and asks nobody.
+    #[tokio::test]
+    async fn an_imdb_id_that_is_none_is_no_series_and_goes_into_no_address() {
+        let (state, providers) = crate::service::testing::server().await;
+        for none in [
+            "../account/1?x=1",
+            "tt12345?language=fr",
+            "tt12345/../../3/account/tt",
+            "not an id",
+            "ttabc",
+            "tt",
+            "",
+        ] {
+            let found = by_imdb_id(&state, none).await.expect("answered");
+            assert!(found.is_none(), "{none:?}");
+        }
+        assert_eq!(providers.asked(), 0, "none of them was asked of anybody");
+
+        // What makes it a test: one that is an id, spelt as TMDB's find
+        // takes it, is asked for.
+        let _ = by_imdb_id(&state, "9907240").await;
+        assert!(providers.asked() > 0);
+    }
+
     #[tokio::test]
     async fn sonarrs_imdb_and_tmdb_terms_find_a_series_held_under_another_id_of_its_item() {
         use crate::service::bridge::{Site, keep_for_tests};
diff --git a/src/service/webhook.rs b/src/service/webhook.rs
index b1b1ec1..2fcf4a8 100644
--- a/src/service/webhook.rs
+++ b/src/service/webhook.rs
@@ -229,6 +229,9 @@ impl Hooks {
             // Webhooks asks, rather than followed: a 3xx is a failure, and
             // the log says where to look.
             .redirect(reqwest::redirect::Policy::none())
+            // Nothing is followed, so no address is ever named to another;
+            // said all the same, as it is for every client this server makes.
+            .referer(false)
             .timeout(TIMEOUT)
             .connect_timeout(CONNECT_TIMEOUT)
             .build()
diff --git a/src/state.rs b/src/state.rs
index 7a79c89..e2a3d0f 100644
--- a/src/state.rs
+++ b/src/state.rs
@@ -175,6 +175,9 @@ pub struct Inner {
     accounts: tokio::sync::Mutex<()>,
     /// What each API answered since the start.
     pub calls: Calls,
+    /// The numbers the TheTVDB relay made for what TheTVDB gave none, led back
+    /// to what they name: read once while the catalogue stays as it was.
+    pub made_ids: crate::api::thetvdb::MadeIds,
     /// The media kept: the store, and the index of what is in it.
     pub media: crate::media::Media,
     /// The doors' certificates: the clients' listener and what it shows.
@@ -231,11 +234,13 @@ impl AppState {
 
         let upstreams = std::sync::Arc::new(configured_hosts(&config));
         let http = outbound_client(&upstreams, Redirects::Anywhere)?;
-        // MyAnimeList's client id travels in a header of its own, which a
-        // redirect to another origin — another host, but another port or
-        // scheme of the same one too — would carry along: reqwest strips only
-        // the ones it knows for credentials.
-        let mal_http = outbound_client(&upstreams, Redirects::SameOrigin)?;
+        // The providers a credential travels to: TMDB's and Fanart's key in
+        // the address, TheTVDB's in the body of its sign-in, which a 307
+        // sends on again, MyAnimeList's client id in a header of its own,
+        // which reqwest does not know for a credential and keeps on a
+        // redirect to another port or scheme of the same host too. Each is
+        // followed only where it was first asked.
+        let keyed_http = outbound_client(&upstreams, Redirects::SameOrigin)?;
 
         let slot = crate::cache::RedisSlot::default();
         let caches = Caches::new(&config.cache, slot.clone());
@@ -270,20 +275,20 @@ impl AppState {
             crate::config::Mode::Single => axum_extra::extract::cookie::Key::generate(),
             crate::config::Mode::Multi => shared_cookie_key(&db).await?,
         };
-        let tmdb = TmdbClient::new(http.clone(), &config.tmdb);
+        let tmdb = TmdbClient::new(keyed_http.clone(), &config.tmdb);
         let skyhook = SkyhookClient::new(http.clone(), &config.skyhook, instance.clone());
         let radarr_metadata =
             RadarrMetadataClient::new(http.clone(), &config.radarr_metadata, instance.clone());
-        let fanart = FanartClient::new(http.clone(), &config.fanart, &config.tmdb.language);
+        let fanart = FanartClient::new(keyed_http.clone(), &config.fanart, &config.tmdb.language);
         let tvdb = TvdbClient::new(
-            http.clone(),
+            keyed_http.clone(),
             &config.tvdb,
             &config.tmdb.language,
             instance.clone(),
         );
         let tvmaze = TvmazeClient::new(http.clone(), &config.tvmaze);
         let anilist = AnilistClient::new(http.clone(), &config.anilist, instance.clone());
-        let mal = MalClient::new(mal_http, &config.mal);
+        let mal = MalClient::new(keyed_http, &config.mal);
         let fankai = FankaiClient::new(http.clone(), &config.fankai, &config.tmdb.language);
         let fankai_wiki = FankaiWikiClient::new(http.clone(), &config.fankai_wiki);
 
@@ -346,6 +351,7 @@ impl AppState {
             instance,
             accounts: tokio::sync::Mutex::new(()),
             calls: Calls::default(),
+            made_ids: crate::api::thetvdb::MadeIds::default(),
             cookie_key,
             tls,
             coord,
@@ -1112,8 +1118,8 @@ enum Redirects {
     /// To any host the guard does not refuse.
     Anywhere,
     /// Only at the origin first asked — its scheme, host and port: a
-    /// credential in a header of the provider's own goes nowhere else, and
-    /// not to another service of the same host.
+    /// credential of the provider's own goes nowhere else, and not to
+    /// another service of the same host.
     SameOrigin,
 }
 
@@ -1124,6 +1130,26 @@ fn leaves_origin(first: Option<&url::Url>, next: &url::Url) -> bool {
     first.is_none_or(|first| !crate::outbound::same_origin(first.as_str(), next.as_str()))
 }
 
+/// What every client of the providers is made from: this server's name, how
+/// long it waits, and no `Referer`. reqwest writes the address a redirect
+/// came from into one, query and all, and for a provider asked with its key
+/// in the address — TMDB's, Fanart's — that address carries the key to
+/// whoever the redirect names.
+fn provider_builder() -> reqwest::ClientBuilder {
+    reqwest::Client::builder()
+        .user_agent(concat!(
+            "arr-metadata-server/",
+            env!("CARGO_PKG_VERSION"),
+            " (+",
+            env!("CARGO_PKG_REPOSITORY"),
+            ")"
+        ))
+        .timeout(std::time::Duration::from_secs(30))
+        .connect_timeout(std::time::Duration::from_secs(10))
+        .pool_max_idle_per_host(16)
+        .referer(false)
+}
+
 /// The client every provider, dataset, relay to TMDB and webhook is asked
 /// with.
 ///
@@ -1138,17 +1164,7 @@ fn outbound_client(
     redirects: Redirects,
 ) -> Result {
     let guard = crate::outbound::Guard::default();
-    reqwest::Client::builder()
-        .user_agent(concat!(
-            "arr-metadata-server/",
-            env!("CARGO_PKG_VERSION"),
-            " (+",
-            env!("CARGO_PKG_REPOSITORY"),
-            ")"
-        ))
-        .timeout(std::time::Duration::from_secs(30))
-        .connect_timeout(std::time::Duration::from_secs(10))
-        .pool_max_idle_per_host(16)
+    provider_builder()
         .dns_resolver(std::sync::Arc::new(UpstreamResolver {
             configured: std::sync::Arc::clone(upstreams),
         }))
@@ -1358,6 +1374,110 @@ mod redirect_tests {
         // With no first request on record, nothing is known to stay.
         assert!(leaves_origin(None, &first));
     }
+
+    /// A stand-in on the loopback: `/start` is sent on to `/next`, which says
+    /// the `Referer` it came with; `/away` is sent on to `elsewhere`.
+    async fn stand_in(elsewhere: &str) -> std::net::SocketAddr {
+        use axum::{Router, http::HeaderMap, response::Redirect, routing::get};
+
+        let away = elsewhere.to_string();
+        let app = Router::new()
+            .route("/start", get(|| async { Redirect::temporary("/next") }))
+            .route(
+                "/next",
+                get(|headers: HeaderMap| async move {
+                    headers
+                        .get("referer")
+                        .and_then(|value| value.to_str().ok())
+                        .unwrap_or("none")
+                        .to_string()
+                }),
+            )
+            .route(
+                "/away",
+                get(move || async move { Redirect::temporary(&away) }),
+            );
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("a port");
+        let at = listener.local_addr().expect("its address");
+        tokio::spawn(async move {
+            let _ = axum::serve(listener, app).await;
+        });
+        at
+    }
+
+    #[tokio::test]
+    async fn a_redirect_is_followed_without_telling_where_it_came_from() {
+        let at = stand_in("/").await;
+        let asked = format!("http://{at}/start?api_key=s3cret");
+        let said = |builder: reqwest::ClientBuilder| {
+            let asked = asked.clone();
+            async move {
+                builder
+                    .redirect(reqwest::redirect::Policy::limited(2))
+                    .build()
+                    .unwrap()
+                    .get(asked)
+                    .send()
+                    .await
+                    .expect("answered")
+                    .text()
+                    .await
+                    .expect("read")
+            }
+        };
+        assert_eq!(said(super::provider_builder()).await, "none");
+        // What makes it a test: a client as reqwest builds one says the
+        // address it was sent to, key and all.
+        assert!(said(reqwest::Client::builder()).await.contains("s3cret"));
+    }
+
+    /// The server's own clients, as they are built: TMDB's, which carries its
+    /// key in the address, is handed a redirect to another origin as it came;
+    /// the client of a provider with no credential is judged by the guard
+    /// alone, which refuses a hop to the loopback with an error.
+    #[tokio::test]
+    async fn the_client_of_tmdb_follows_no_redirect_off_its_origin() {
+        use crate::api::relay::testing::{StandIn, server};
+
+        let elsewhere = stand_in("/").await;
+        let at = stand_in(&format!("http://{elsewhere}/next")).await;
+        let upstream =
+            StandIn::start(|_| (axum::http::StatusCode::OK, serde_json::json!({}))).await;
+        let state = server(&upstream).await;
+        let asked = format!("http://{at}/away?api_key=s3cret");
+
+        let answer = state
+            .tmdb
+            .request(reqwest::Method::GET, &asked)
+            .send()
+            .await
+            .expect("answered");
+        assert_eq!(answer.status(), reqwest::StatusCode::TEMPORARY_REDIRECT);
+        assert!(state.http.get(&asked).send().await.is_err());
+    }
+
+    #[tokio::test]
+    async fn a_provider_with_a_credential_is_not_followed_to_another_origin() {
+        let elsewhere = stand_in("/").await;
+        let at = stand_in(&format!("http://{elsewhere}/next")).await;
+        let upstreams = std::sync::Arc::new(std::collections::HashSet::new());
+        let keyed = super::outbound_client(&upstreams, super::Redirects::SameOrigin).unwrap();
+
+        // Another port of the same host is another service: the redirect is
+        // handed back as it came, and nothing is sent to it.
+        let answer = keyed
+            .get(format!("http://{at}/away?api_key=s3cret"))
+            .send()
+            .await
+            .expect("answered");
+        assert_eq!(answer.status(), reqwest::StatusCode::TEMPORARY_REDIRECT);
+        assert_eq!(
+            answer.headers()["location"],
+            format!("http://{elsewhere}/next")
+        );
+    }
 }
 
 #[cfg(test)]
diff --git a/src/telemetry.rs b/src/telemetry.rs
index 21a10d9..fb81865 100644
--- a/src/telemetry.rs
+++ b/src/telemetry.rs
@@ -397,7 +397,19 @@ pub fn redact(text: &str) -> String {
 
 const MASK: &str = "***";
 
-/// The parameters whose values are secrets: keys, tokens, passwords.
+/// The parameters whose values are secrets: keys, tokens, passwords — and
+/// what an address carries a credential in besides, each by the name it
+/// is found under somewhere here: `auth`, where a list another server keeps
+/// private is read with a key (an import rule's address); `code`, `state`
+/// and `session_state`, what the identity provider sends a sign-in back with
+/// — a `code` is the one-time right to the person's tokens, and a `state`
+/// the proof the sign-in is the one begun; `session_id`, `guest_session_id`
+/// and `request_token`, the sessions TMDB's clients open, any of which acts
+/// as the account for as long as it lasts.
+///
+/// Every text that is logged, journaled, reported or traced is masked by
+/// this list: the request's span ([`redact_uri`]), the lines the journal
+/// keeps (`journal::masked`) and a server span's `url.query` ([`redact`]).
 const SECRET_PARAMETERS: &[&str] = &[
     "api_key",
     "apikey",
@@ -415,6 +427,13 @@ const SECRET_PARAMETERS: &[&str] = &[
     "signature",
     "x-amz-signature",
     "x-amz-credential",
+    "auth",
+    "code",
+    "state",
+    "session_state",
+    "session_id",
+    "guest_session_id",
+    "request_token",
 ];
 
 /// `api_key=abc&language=fr` is `api_key=***&language=fr`, wherever such a
@@ -637,6 +656,39 @@ mod tests {
         assert_eq!(mask_parameters("api_key="), "api_key=");
     }
 
+    #[test]
+    fn what_a_sign_in_a_session_or_a_list_carries_in_an_address_is_masked() {
+        // The identity provider's way back to the callback: the code is
+        // the right to the person's tokens, the state is the sign-in's own.
+        assert_eq!(
+            redact(
+                "GET /api/v1/auth/oidc/callback?code=4/0AbCdEf&state=Xy12Zq&session_state=ab.cd&iss=https://idp.example"
+            ),
+            "GET /api/v1/auth/oidc/callback?code=***&state=***&session_state=***&iss=https://idp.example"
+        );
+        // TMDB's sessions, in any case and wherever the pair stands.
+        assert_eq!(
+            mask_parameters(
+                "language=fr&Session_ID=s3ss10n&guest_session_id=gu3st&request_token=r3qt0k&page=2"
+            ),
+            "language=fr&Session_ID=***&guest_session_id=***&request_token=***&page=2"
+        );
+        // An import rule's address, with the key its list is read with.
+        assert_eq!(
+            redact(
+                "error sending request for url (https://other.example/lists/top/ids.txt?kind=movie&auth=t0pS3cret)"
+            ),
+            "error sending request for url (https://other.example/lists/top/ids.txt?kind=movie&auth=***)"
+        );
+        // A name that only contains one is left alone, and a value that is
+        // none stays none.
+        assert_eq!(
+            mask_parameters("barcode=1&mistate=2&authors=3&postcode=4"),
+            "barcode=1&mistate=2&authors=3&postcode=4"
+        );
+        assert_eq!(mask_parameters("code=&state="), "code=&state=");
+    }
+
     #[test]
     fn credentials_after_their_scheme_are_masked() {
         assert_eq!(
@@ -702,6 +754,26 @@ mod tests {
         assert_eq!(redact_uri(&bare), "/health");
     }
 
+    #[test]
+    fn the_request_span_never_holds_what_a_sign_in_or_a_session_carries() {
+        let callback: axum::http::Uri =
+            "/api/v1/auth/oidc/callback?code=authcode&state=csrf&session_state=sess"
+                .parse()
+                .unwrap();
+        assert_eq!(
+            redact_uri(&callback),
+            "/api/v1/auth/oidc/callback?code=***&state=***&session_state=***"
+        );
+        let tmdb: axum::http::Uri =
+            "/3/account/1/lists?session_id=s&guest_session_id=g&request_token=t&language=fr"
+                .parse()
+                .unwrap();
+        assert_eq!(
+            redact_uri(&tmdb),
+            "/3/account/1/lists?session_id=***&guest_session_id=***&request_token=***&language=fr"
+        );
+    }
+
     #[test]
     fn an_id_from_outside_is_taken_in_one_shape_only() {
         let ok = |text: &str| acceptable_id(&HeaderValue::from_str(text).unwrap());
diff --git a/src/telemetry/journal.rs b/src/telemetry/journal.rs
index 601e91e..073a68b 100644
--- a/src/telemetry/journal.rs
+++ b/src/telemetry/journal.rs
@@ -503,6 +503,41 @@ mod tests {
         assert!(text.contains("api_key=***"), "{text}");
     }
 
+    #[test]
+    fn what_a_sign_in_or_a_session_carries_in_an_address_is_not_kept() {
+        let subscriber = tracing_subscriber::registry().with(layer());
+        tracing::subscriber::with_default(subscriber, || {
+            tracing::warn!(
+                callback = %"/api/v1/auth/oidc/callback?code=authcode123456&state=csrfstate7890&session_state=sessstate4321",
+                tmdb = %"/3/account/1/lists?session_id=tmdbsession5555&guest_session_id=guestsess6666&request_token=reqtoken7777",
+                list = %"https://other.example/ids.txt?auth=listkey8888&kind=movie",
+                "signin-line-marker"
+            );
+        });
+        let lines = kept("signin-line-marker");
+        let text = lines.first().expect("the line").as_text();
+        for planted in [
+            "authcode123456",
+            "csrfstate7890",
+            "sessstate4321",
+            "tmdbsession5555",
+            "guestsess6666",
+            "reqtoken7777",
+            "listkey8888",
+        ] {
+            assert!(!text.contains(planted), "{planted} in {text}");
+        }
+        for masked in [
+            "code=***",
+            "state=***",
+            "session_id=***",
+            "auth=***",
+            "kind=movie",
+        ] {
+            assert!(text.contains(masked), "{masked} in {text}");
+        }
+    }
+
     #[test]
     fn the_journal_is_read_by_level_by_words_and_after_a_line() {
         let subscriber = tracing_subscriber::registry().with(layer());
diff --git a/src/telemetry/trace.rs b/src/telemetry/trace.rs
index 7ae00a6..ebd7e6f 100644
--- a/src/telemetry/trace.rs
+++ b/src/telemetry/trace.rs
@@ -744,6 +744,64 @@ pub(super) mod tests {
         ON_HERE.with(|on| on.set(false));
     }
 
+    /// The query a server span records is masked by the list the log is:
+    /// the key of a client, what a sign-in's address is sent back with, what
+    /// a TMDB session carries — and what is not a secret stays as it was.
+    #[tokio::test(flavor = "current_thread")]
+    async fn the_query_a_server_span_records_is_masked_as_the_log_masks_it() {
+        ON_HERE.with(|on| on.set(true));
+        let kept = Kept::default();
+        let provider_spans = SdkTracerProvider::builder()
+            .with_simple_exporter(kept.clone())
+            .with_id_generator(Chosen::default())
+            .build();
+        let subscriber = tracing_subscriber::registry().with(spans(provider_spans.tracer("test")));
+        let _default = tracing::subscriber::set_default(subscriber);
+
+        let (upstream, _) = provider().await;
+        let query = "code=authcode123456&state=csrfstate7890&session_state=sessstate4321\
+                     &session_id=tmdbsession5555&guest_session_id=guestsess6666\
+                     &request_token=reqtoken7777&auth=listkey8888&api_key=clientkey9999&language=fr";
+        let mut request = HttpRequest::get(format!("/v1/tvdb/shows/en/81189?{query}"))
+            .body(Body::empty())
+            .unwrap();
+        request.extensions_mut().insert(ConnectInfo::(
+            "203.0.113.5:4000".parse().unwrap(),
+        ));
+        let response = door(upstream, vec![]).oneshot(request).await.unwrap();
+        assert_eq!(response.status(), 200);
+        // The span ends with the answer's body, as the other test reads it.
+        let body = axum::body::to_bytes(response.into_body(), 1024)
+            .await
+            .unwrap();
+        assert_eq!(&body[..], b"answered");
+
+        let spans = std::mem::take(&mut *kept.0.lock().unwrap());
+        let server = spans
+            .iter()
+            .find(|s| s.span_kind == SpanKind::Server)
+            .expect("the server span");
+        let recorded = attribute(server, "url.query")
+            .expect("the query")
+            .to_string();
+        for planted in [
+            "authcode123456",
+            "csrfstate7890",
+            "sessstate4321",
+            "tmdbsession5555",
+            "guestsess6666",
+            "reqtoken7777",
+            "listkey8888",
+            "clientkey9999",
+        ] {
+            assert!(!recorded.contains(planted), "{planted} in {recorded}");
+        }
+        assert!(recorded.contains("language=fr"), "{recorded}");
+        assert!(recorded.contains("code=***&state=***"), "{recorded}");
+
+        ON_HERE.with(|on| on.set(false));
+    }
+
     /// A call made outside any request — a background task's — is sent as
     /// it is: no span, no `traceparent`.
     #[tokio::test(flavor = "current_thread")]
diff --git a/src/ui.rs b/src/ui.rs
index 9bee92b..e3d7119 100644
--- a/src/ui.rs
+++ b/src/ui.rs
@@ -13,6 +13,11 @@
 //! credential would be let in at all. A work's page carries what a search
 //! engine reads of it too, as schema.org describes a film or a series — only
 //! where the site is public, the one case a search engine is meant to read it.
+//!
+//! Finding what a preview says is the database's work, which anybody may ask
+//! for as often as they like: it is kept a few seconds, and what is not kept
+//! is counted against the caller's allowance as the API's requests are, a
+//! caller who has spent it getting the page without ([`shell`]).
 
 use axum::{
     Router,
@@ -84,13 +89,69 @@ async fn serve(State(state): State, request: Request) -> Response {
     let Some(file) = Assets::get("index.html") else {
         return not_found();
     };
-    let mut html = String::from_utf8_lossy(&file.data).into_owned();
-    if let Some(preview) = preview(&state, path).await {
-        html = preview.apply(&html, state.config.server.public_url.as_deref(), path);
-    } else if let Some(public) = state.config.server.public_url.as_deref() {
-        html = absolute_site_image(&html, public);
+    let html = String::from_utf8_lossy(&file.data).into_owned();
+    let peer = crate::auth::ratelimit::peer_of(&state, &request);
+    page(shell(&state, path, peer, &html).await)
+}
+
+/// The longest path a rendered preview is kept under: a path that may have
+/// one is a few dozen characters, and what follows a work's id is a stranger's
+/// to write.
+const KEPT_PATH: usize = 256;
+
+/// The page for a route: the app's own, with the preview of what the route
+/// shows in its head where there is one.
+///
+/// Looking a preview up is the database's work — a work, a list, a genre's
+/// count — which anybody may ask for as often as they like, one address after
+/// another. So a preview rendered lately is kept a few seconds, under the path
+/// and the catalogue's mark; one that is not kept is looked up on the
+/// caller's account, as the API's requests are counted
+/// ([`crate::auth::ratelimit::spend`]); and a caller whose allowance is spent
+/// is given the page without one, as a path that has none is. `peer` is the
+/// caller, as the limiter counts callers ([`crate::auth::ratelimit::peer_of`]).
+async fn shell(state: &AppState, path: &str, peer: std::net::IpAddr, html: &str) -> String {
+    let public = state.config.server.public_url.as_deref();
+    let plain = || match public {
+        Some(public) => absolute_site_image(html, public),
+        None => html.to_string(),
+    };
+
+    let Some(subject) = subject_of(path) else {
+        return plain();
+    };
+    if !open_to_readers(state) {
+        return plain();
+    }
+    let Ok(mark) = state.caches.catalogue_stamp(&state.db).await else {
+        return plain();
+    };
+    let key = format!("{}\n{mark}\n{path}", state.caches.stamp());
+    let kept = path.len() <= KEPT_PATH;
+    if kept && let Some(page) = state.caches.previews.get(&key).await {
+        return page.to_string();
+    }
+
+    if crate::auth::ratelimit::spend(state, peer, path)
+        .await
+        .is_err()
+    {
+        return plain();
     }
-    page(html)
+    let Some(preview) = preview(state, subject).await else {
+        return plain();
+    };
+    let page = preview.apply(html, public, path);
+    // Only what was found is kept: a path that names nothing is looked up
+    // again, on its caller's account, and is never the key of anything.
+    if kept {
+        state
+            .caches
+            .previews
+            .insert(key, std::sync::Arc::from(page.as_str()))
+            .await;
+    }
+    page
 }
 
 fn asset(path: &str) -> Option {
@@ -241,32 +302,48 @@ fn open_to_readers(state: &AppState) -> bool {
     }
 }
 
-/// The preview a path deserves: a work's, a list's, a page's, or none. The
-/// shape of the id or slug is checked before anything is looked up, so a
-/// stranger walking random addresses is answered from the page alone.
-async fn preview(state: &AppState, path: &str) -> Option {
-    if !open_to_readers(state) {
-        return None;
-    }
+/// What a path's preview would be of, once the shape of its id or slug says
+/// it can have one at all.
+#[derive(Debug, PartialEq, Eq)]
+enum Subject<'a> {
+    /// A work, by its id.
+    Work(&'a str),
+    /// A curated list, by its slug or its id.
+    List(&'a str),
+    /// A genre's, a network's, a year's or a decade's page, and the key it is
+    /// asked for by, as written.
+    Page(&'a str, String),
+    /// A company's or a keyword's page, by this server's id for it.
+    Entity(&'a str, &'a str),
+    /// A character, by this server's id for it.
+    Character(&'a str),
+    /// A collection, by TMDB's id for it.
+    Collection(i64),
+}
+
+/// What a path asks a preview of, or nothing: the shape of the id or slug is
+/// checked before anything is looked up, so a stranger walking random
+/// addresses is answered from the page alone.
+fn subject_of(path: &str) -> Option> {
     let mut parts = path.trim_start_matches('/').split('/');
     match (parts.next()?, parts.next()?) {
-        ("work", id) if uuid::Uuid::parse_str(id).is_ok() => work_preview(state, id).await,
-        ("lists", key) if is_key(key) && parts.next().is_none() => list_preview(state, key).await,
+        ("work", id) if uuid::Uuid::parse_str(id).is_ok() => Some(Subject::Work(id)),
+        ("lists", key) if is_key(key) && parts.next().is_none() => Some(Subject::List(key)),
         (page @ ("genre" | "network" | "year" | "decade"), key)
             if !key.is_empty() && key.len() <= 128 && parts.next().is_none() =>
         {
             let key = urlencoding::decode(key).ok()?;
-            page_preview(state, page, &key).await
+            Some(Subject::Page(page, key.into_owned()))
         }
         // A company's, a keyword's and a character's pages, by this
         // server's id for them.
         (page @ ("company" | "keyword"), id)
             if uuid::Uuid::parse_str(id).is_ok() && parts.next().is_none() =>
         {
-            entity_preview(state, page, id).await
+            Some(Subject::Entity(page, id))
         }
         ("characters", id) if uuid::Uuid::parse_str(id).is_ok() && parts.next().is_none() => {
-            character_preview(state, id).await
+            Some(Subject::Character(id))
         }
         // A collection's, by TMDB's id for it.
         ("collections", id)
@@ -275,12 +352,25 @@ async fn preview(state: &AppState, path: &str) -> Option {
                 && id.bytes().all(|b| b.is_ascii_digit())
                 && parts.next().is_none() =>
         {
-            collection_preview(state, id.parse().ok()?).await
+            Some(Subject::Collection(id.parse().ok()?))
         }
         _ => None,
     }
 }
 
+/// The preview of what a path asks one of: a work's, a list's, a page's, or
+/// none where the catalogue holds nothing by that name.
+async fn preview(state: &AppState, subject: Subject<'_>) -> Option {
+    match subject {
+        Subject::Work(id) => work_preview(state, id).await,
+        Subject::List(key) => list_preview(state, key).await,
+        Subject::Page(page, key) => page_preview(state, page, &key).await,
+        Subject::Entity(page, id) => entity_preview(state, page, id).await,
+        Subject::Character(id) => character_preview(state, id).await,
+        Subject::Collection(tmdb_id) => collection_preview(state, tmdb_id).await,
+    }
+}
+
 /// A list's slug or id: what `make_slug` produces, or a UUID.
 fn is_key(key: &str) -> bool {
     !key.is_empty()
@@ -1258,4 +1348,217 @@ mod tests {
         assert!(!is_key("a/b"));
         assert!(!is_key(&"a".repeat(129)));
     }
+
+    #[test]
+    fn only_a_path_that_can_have_a_preview_is_asked_for_one() {
+        let id = "01a0d073-6ce8-760c-a2cb-113d4cf78503";
+        assert_eq!(subject_of(&format!("/work/{id}")), Some(Subject::Work(id)));
+        // What follows a work's id changes nothing of whose page it is.
+        assert_eq!(
+            subject_of(&format!("/work/{id}/seasons/2")),
+            Some(Subject::Work(id))
+        );
+        assert_eq!(
+            subject_of("/lists/soirees-d-automne"),
+            Some(Subject::List("soirees-d-automne"))
+        );
+        assert_eq!(
+            subject_of("/genre/Dr%C3%A2me"),
+            Some(Subject::Page("genre", "Drâme".into()))
+        );
+        assert_eq!(
+            subject_of("/network/hbo"),
+            Some(Subject::Page("network", "hbo".into()))
+        );
+        assert_eq!(
+            subject_of(&format!("/company/{id}")),
+            Some(Subject::Entity("company", id))
+        );
+        assert_eq!(
+            subject_of(&format!("/keyword/{id}")),
+            Some(Subject::Entity("keyword", id))
+        );
+        assert_eq!(
+            subject_of(&format!("/characters/{id}")),
+            Some(Subject::Character(id))
+        );
+        assert_eq!(subject_of("/collections/10"), Some(Subject::Collection(10)));
+
+        let long = format!("/genre/{}", "a".repeat(129));
+        for path in [
+            "/",
+            "/catalogue",
+            "/work",
+            "/work/not-an-id",
+            "/lists/Soirées",
+            "/lists/a/b",
+            "/genre/",
+            long.as_str(),
+            "/company/nobody",
+            "/characters/1",
+            "/collections/abc",
+            "/collections/1234567890123456789",
+        ] {
+            assert_eq!(subject_of(path), None, "{path}");
+        }
+    }
+
+    /// A server whose pages are offered a link preview whoever reads them,
+    /// and that lets an address make `per_minute` requests a minute.
+    async fn open_server(per_minute: u32) -> AppState {
+        let mut config = crate::config::Config::from_env().expect("a configuration");
+        config.mode = crate::config::Mode::Single;
+        config.database = crate::db::testing::database().await;
+        config.security.bootstrap_admin = None;
+        config.security.native_policy = SurfacePolicy::Open;
+        config.security.rate_limit_per_minute = per_minute;
+        config.cache.redis_url = None;
+        config.media.storage = crate::config::MediaStorage::Off;
+        config.clients = None;
+        config.tmdb.api_key = None;
+        config.tmdb.include_adult = false;
+        config.tvdb.api_key = None;
+        config.tvdb.enabled = false;
+        AppState::bootstrap(config).await.expect("a server")
+    }
+
+    /// Two dramas in the catalogue: the drama page's preview says so.
+    async fn two_dramas(state: &AppState) -> Vec {
+        let mut works = Vec::new();
+        for title in ["One", "Two"] {
+            let mut work = MediaItem::empty(MediaKind::Movie);
+            work.title = title.into();
+            work.slug = crate::domain::make_slug(title, None);
+            work.genres = vec!["Drama".to_string()];
+            repo::item::upsert(
+                &state.db,
+                repo::item::ItemWrite {
+                    item: &work,
+                    replace_children: true,
+                },
+            )
+            .await
+            .unwrap();
+            works.push(work);
+        }
+        crate::service::listing::relist_stale(&state.db, false, 100)
+            .await
+            .unwrap();
+        works
+    }
+
+    /// The page for a path, as a caller at one address is given it.
+    async fn previewed(state: &AppState, path: &str) -> String {
+        shell(state, path, std::net::IpAddr::from([203, 0, 113, 9]), PAGE).await
+    }
+
+    const TWO: &str = "2 works in the catalogue.";
+
+    /// What a link preview costs the database is asked of whoever asks: kept
+    /// a few seconds, counted against the caller's allowance when it is not
+    /// kept, and not looked up at all once the allowance is spent.
+    #[tokio::test]
+    async fn a_preview_is_looked_up_on_the_callers_account_and_kept_a_few_seconds() {
+        let state = open_server(2).await;
+        two_dramas(&state).await;
+
+        // A page that can have no preview is answered as it always was, and
+        // costs the caller nothing.
+        for _ in 0..5 {
+            let page = previewed(&state, "/catalogue").await;
+            assert!(page.contains("A catalogue."), "{page}");
+        }
+
+        // The first look-up is counted; the same path asked again is answered
+        // from what was kept, however often.
+        for _ in 0..5 {
+            let page = previewed(&state, "/genre/drama").await;
+            assert!(page.contains(TWO), "{page}");
+        }
+        // Another spelling of it is another path: a look-up, the second of
+        // the two the allowance holds.
+        let page = previewed(&state, "/genre/Drama").await;
+        assert!(page.contains(TWO), "{page}");
+
+        // Spent: a path that is not kept is given the page without one…
+        let spent = previewed(&state, "/genre/DRAMA").await;
+        assert!(!spent.contains(TWO), "{spent}");
+        assert!(spent.contains("A catalogue."), "{spent}");
+        // … and what is kept is still served.
+        let page = previewed(&state, "/genre/drama").await;
+        assert!(page.contains(TWO), "{page}");
+    }
+
+    /// A page is kept under the catalogue's mark: a write made here, which
+    /// moves it, is seen at once; a note on the genre, which does not, within
+    /// the seconds a page is kept.
+    #[tokio::test]
+    async fn a_kept_preview_is_looked_up_again_once_the_catalogue_moves() {
+        let state = open_server(600).await;
+        two_dramas(&state).await;
+
+        let first = previewed(&state, "/genre/drama").await;
+        assert!(first.contains(TWO), "{first}");
+
+        repo::editorial::put(
+            &state.db,
+            Scope::Genre,
+            "drama",
+            "en",
+            "The *quiet* ones.",
+            None,
+        )
+        .await
+        .unwrap();
+        let kept = previewed(&state, "/genre/drama").await;
+        assert_eq!(kept, first, "kept: nothing was asked again");
+
+        state.caches.bump_epoch().await;
+        let fresh = previewed(&state, "/genre/drama").await;
+        assert!(fresh.contains("The quiet ones."), "{fresh}");
+        assert!(!fresh.contains(TWO), "{fresh}");
+    }
+
+    /// A path that names nothing is looked up each time, on its caller's
+    /// account, and is never the key of anything kept.
+    #[tokio::test]
+    async fn a_path_that_names_nothing_is_never_kept() {
+        let state = open_server(600).await;
+        let works = two_dramas(&state).await;
+
+        for path in [
+            "/genre/isekai",
+            "/network/nobody",
+            "/lists/no-such-list",
+            "/year/nineteen",
+            "/work/01a0d073-6ce8-760c-a2cb-113d4cf78503",
+        ] {
+            let page = previewed(&state, path).await;
+            assert!(page.contains("A catalogue."), "{path}: {page}");
+        }
+        assert_eq!(state.caches.previews.entries().await, 0);
+
+        // A work found is kept; the same work asked for under a path too long
+        // to be a page's is found, and is not kept under it.
+        let id = &works[0].id;
+        let page = previewed(&state, &format!("/work/{id}")).await;
+        assert!(page.contains("One · Cinémathèque"), "{page}");
+        assert_eq!(state.caches.previews.entries().await, 1);
+        let long = format!("/work/{id}/{}", "x".repeat(KEPT_PATH));
+        let page = previewed(&state, &long).await;
+        assert!(page.contains("One · Cinémathèque"), "{page}");
+        assert_eq!(state.caches.previews.entries().await, 1);
+    }
+
+    /// Where no reader is let in without a credential, there is no preview:
+    /// nothing is looked up, and nothing is counted.
+    #[tokio::test]
+    async fn no_preview_is_looked_up_where_a_reader_is_not_let_in() {
+        let state = crate::api::native::testing::server().await;
+        two_dramas(&state).await;
+        assert!(!open_to_readers(&state));
+        let page = previewed(&state, "/genre/drama").await;
+        assert!(!page.contains(TWO), "{page}");
+        assert_eq!(state.caches.previews.entries().await, 0);
+    }
 }
diff --git a/src/web.rs b/src/web.rs
index 36ca984..eeec052 100644
--- a/src/web.rs
+++ b/src/web.rs
@@ -38,10 +38,13 @@ const MAX_BODY_BYTES: usize = 1024 * 1024;
 
 pub async fn serve(state: AppState) -> Result<()> {
     let bind = state.config.server.bind;
-    let limits = listen::Limits {
-        connections: state.config.server.max_connections,
-        head: state.config.server.header_read_timeout,
-    };
+    let limits = listen::Limits::new(
+        state.config.server.max_connections,
+        state.config.server.header_read_timeout,
+    );
+    // How many connections each network holds at once, but for the proxies
+    // in front of the server, which hold those of everybody behind them.
+    let crowd = || listen::Crowd::new(&state.config.server.trusted_proxies);
 
     // The loops that run for as long as the process does, kept: one that
     // panics or returns when it should not is said so and started again.
@@ -138,7 +141,7 @@ pub async fn serve(state: AppState) -> Result<()> {
                 "listening for the clients (TLS)"
             );
             let served = listen::serve(
-                listen::Door::new(listener, limits.connections),
+                listen::Door::new(listener, limits.connections, crowd()),
                 Some(clients.rustls.clone()),
                 clients_router,
                 handle.clone(),
@@ -208,7 +211,7 @@ pub async fn serve(state: AppState) -> Result<()> {
     });
 
     listen::serve(
-        listen::Door::new(listener, limits.connections),
+        listen::Door::new(listener, limits.connections, crowd()),
         tls,
         router,
         handle,
@@ -399,8 +402,9 @@ fn build_router(state: AppState) -> Router {
         .route("/health", get(health))
         .route("/ready", get(ready))
         // The media kept, outside every guard: a key is a hash nobody
-        // guesses, and Sonarr fetches a poster with no credential.
-        .route("/media/{key}", get(crate::media::serve::get))
+        // guesses, and Sonarr fetches a poster with no credential. A copy
+        // made on demand counts against its caller, as the API does.
+        .route("/media/{key}", get(crate::media::serve::get_for_caller))
         // The authority the clients are to trust, and the script that makes
         // a container trust it: public, since a client fetches them before
         // it can trust anything.
@@ -800,9 +804,11 @@ struct Ready {
     tmdb_configured: bool,
 }
 
-/// Liveness is `/health`; this is readiness, and it touches the database.
+/// Liveness is `/health`; this is readiness, and it touches the database —
+/// once in a few seconds, however often it is asked, since anybody may ask
+/// under any name and with no credential.
 async fn ready(State(state): State) -> (StatusCode, Json) {
-    let database = state.db.health().await.is_ok();
+    let database = state.caches.database_answers(&state.db).await;
 
     let status = if database {
         StatusCode::OK
@@ -972,6 +978,54 @@ mod policy_tests {
     }
 }
 
+#[cfg(test)]
+mod ready_tests {
+    use axum::{body::Body, http::Request};
+    use tower::ServiceExt as _;
+
+    use super::*;
+
+    /// What `/ready` says, as the status and whether it says the database
+    /// answers.
+    async fn readiness(router: &Router) -> (StatusCode, bool) {
+        let answer = router
+            .clone()
+            .oneshot(Request::get("/ready").body(Body::empty()).unwrap())
+            .await
+            .unwrap();
+        let status = answer.status();
+        let body = axum::body::to_bytes(answer.into_body(), usize::MAX)
+            .await
+            .unwrap();
+        let said: serde_json::Value = serde_json::from_slice(&body).unwrap();
+        (status, said["database"] == true)
+    }
+
+    /// The database is asked once in a few seconds, however often `/ready` is:
+    /// what was said is said again while the database goes away, and a server
+    /// that has not been asked yet finds out.
+    #[tokio::test]
+    async fn readiness_asks_the_database_once_in_a_few_seconds() {
+        let state = crate::api::native::testing::server().await;
+        let router = build_router(state.clone());
+        assert_eq!(readiness(&router).await, (StatusCode::OK, true));
+
+        state.db.close().await;
+        assert_eq!(
+            readiness(&router).await,
+            (StatusCode::OK, true),
+            "kept: the database was not asked again"
+        );
+
+        let down = crate::api::native::testing::server().await;
+        down.db.close().await;
+        assert_eq!(
+            readiness(&build_router(down)).await,
+            (StatusCode::SERVICE_UNAVAILABLE, false)
+        );
+    }
+}
+
 #[cfg(test)]
 mod well_known_tests {
     use axum::{body::Body, http::Request};
diff --git a/src/web/listen.rs b/src/web/listen.rs
index 7d665d9..e39c7f3 100644
--- a/src/web/listen.rs
+++ b/src/web/listen.rs
@@ -11,37 +11,57 @@
 //! * a semaphore in the accept loop holds each door to `AMS_MAX_CONNECTIONS`:
 //!   past it, a connection waits in the system's queue, not accepted, until
 //!   another one closes;
+//! * no one network holds more than [`PER_PEER`] of them: a connection past
+//!   that is closed as it is accepted, so that one address cannot take every
+//!   place of the door — a trusted proxy's, and this machine's own, are not
+//!   counted, since everybody behind a proxy has its address;
 //! * hyper is given a clock, and with it `AMS_HEADER_READ_TIMEOUT` for a
 //!   request's head — the first, and each one a connection kept alive waits
 //!   for — and a head no larger than a head needs to be;
 //! * the bytes before that, which tell HTTP/2 from HTTP/1 and which hyper
 //!   does not time, are given the same;
-//! * an idle HTTP/2 connection is pinged, and closed when it stops answering.
+//! * an idle HTTP/2 connection is pinged, and closed when it stops answering —
+//!   and closed when it has had no request in flight for [`IDLE`] whatever it
+//!   answers, since a ping keeps a connection alive however little it is used;
+//! * a response that cannot be written because its peer does not read it is
+//!   given up on after [`WRITE_STALL`], and the connection with it.
 
 use std::{
+    collections::HashMap,
+    convert::Infallible,
     future::Future,
     io,
-    net::SocketAddr,
+    net::{IpAddr, SocketAddr},
     pin::Pin,
     sync::{
-        Arc, LazyLock,
-        atomic::{AtomicU64, Ordering},
+        Arc, LazyLock, Mutex, PoisonError,
+        atomic::{AtomicU64, AtomicUsize, Ordering},
     },
     task::{Context, Poll},
     time::{Duration, Instant},
 };
 
-use axum::{Router, extract::Request};
+use axum::{
+    Router,
+    extract::ConnectInfo,
+    http::{Request, Response},
+};
 use axum_server::{
     AddrListener, Address, Handle, Server,
     accept::{Accept, DefaultAcceptor},
     tls_rustls::{RustlsAcceptor, RustlsConfig},
 };
+use bytes::Bytes;
+use hyper::body::{Body as HttpBody, Frame, Incoming, SizeHint};
+use ipnet::IpNet;
 use tokio::{
     io::{AsyncRead, AsyncWrite, ReadBuf},
     net::{TcpListener, TcpStream},
     sync::{OwnedSemaphorePermit, Semaphore},
 };
+use tower::Service;
+
+use crate::auth::ip;
 
 /// What every door is held to.
 #[derive(Clone, Copy, Debug)]
@@ -50,12 +70,37 @@ pub struct Limits {
     pub connections: usize,
     /// How long a connection has to say what it wants.
     pub head: Duration,
+    /// How long a connection may have no request in flight.
+    pub idle: Duration,
+    /// How long a response may wait to be written to a peer that does not
+    /// read it.
+    pub write: Duration,
+}
+
+impl Limits {
+    /// The limits of a door holding `connections` at once, whose connections
+    /// have `head` to say what they want: idle no longer than that, nor than
+    /// [`IDLE`] — a head's time is what a connection kept alive was given to
+    /// sit idle in, and a longer one is the operator's word for it.
+    pub fn new(connections: usize, head: Duration) -> Self {
+        Self {
+            connections,
+            head,
+            idle: IDLE.max(head),
+            write: WRITE_STALL,
+        }
+    }
 }
 
 /// Connections a door bound by address alone holds; the configured doors
 /// are given theirs.
 const DEFAULT_CONNECTIONS: usize = 1024;
 
+/// The most connections one network holds at once — an address, or an IPv6
+/// /64, as the rate limit counts a caller. A browser holds a handful, a
+/// Sonarr or a Plex a few; a thousand and twenty-four is sixteen times this.
+const PER_PEER: usize = 64;
+
 /// The most a request's head may take in a connection's buffer. A head is
 /// a few kilobytes; hyper's default lets each connection hold four hundred
 /// before refusing one, which a door full of slow connections turns into
@@ -67,20 +112,162 @@ const HEAD_BUFFER: usize = 64 * 1024;
 const KEEP_ALIVE: Duration = Duration::from_secs(30);
 const KEEP_ALIVE_TIMEOUT: Duration = Duration::from_secs(20);
 
-/// The address a connection came from, as the doors hand it on.
-#[derive(Clone, Copy, Debug)]
-pub struct Peer(pub SocketAddr);
+/// How long a connection may have no request in flight before it is closed.
+/// Longer than the idle time most clients keep a connection for — a browser's
+/// minute or two, a proxy's — so that a client reuses a connection that is
+/// still open, not one being closed as it sends.
+const IDLE: Duration = Duration::from_secs(120);
+
+/// How long a response may wait to be written to a peer that does not read
+/// it — the time a send buffer may stay full, as a web server's own send
+/// timeout is — before it is given up on.
+const WRITE_STALL: Duration = Duration::from_secs(60);
+
+/// How long bytes must have stopped coming before an idle connection is
+/// closed, so that a request on its way is not met by the connection closing:
+/// the bytes of a request read are not yet a request in flight.
+const SETTLE: Duration = Duration::from_secs(1);
+
+/// The address a connection came from, as the doors hand it on, and what the
+/// connection is doing.
+#[derive(Clone, Debug)]
+pub struct Peer {
+    addr: SocketAddr,
+    life: Arc,
+}
 
 impl Address for Peer {
     type Stream = Held;
     type Listener = Door;
 }
 
-/// What the handlers read as `ConnectInfo`: the peer's address,
-/// as before.
-impl axum::extract::connect_info::Connected for SocketAddr {
-    fn connect_info(peer: Peer) -> Self {
-        peer.0
+/// How many connections each network holds at once, and the networks nobody
+/// counts.
+pub struct Crowd {
+    most: usize,
+    exempt: Vec,
+    holding: Mutex>,
+}
+
+impl Crowd {
+    /// Every network held to [`PER_PEER`] connections but the `proxies` —
+    /// which hold those of everybody behind them — and this machine.
+    pub fn new(proxies: &[IpNet]) -> Self {
+        let machine = ["127.0.0.0/8", "::1/128"]
+            .into_iter()
+            .filter_map(|network| network.parse::().ok());
+        Self::of(PER_PEER, proxies.iter().copied().chain(machine).collect())
+    }
+
+    fn of(most: usize, exempt: Vec) -> Self {
+        Self {
+            most: most.max(1),
+            exempt,
+            holding: Mutex::new(HashMap::new()),
+        }
+    }
+
+    /// A share for a connection from `peer` — or none, when its network
+    /// holds as many as it may.
+    fn admit(self: &Arc, peer: SocketAddr) -> Option {
+        if ip::is_trusted_peer(Some(peer), &self.exempt) {
+            return Some(Share {
+                crowd: self.clone(),
+                network: None,
+            });
+        }
+        let network = ip::bucket(peer.ip());
+        let mut holding = self.holding.lock().unwrap_or_else(PoisonError::into_inner);
+        let held = holding.entry(network).or_default();
+        if *held >= self.most {
+            return None;
+        }
+        *held += 1;
+        Some(Share {
+            crowd: self.clone(),
+            network: Some(network),
+        })
+    }
+
+    /// How many networks hold a connection now.
+    #[cfg(test)]
+    fn networks(&self) -> usize {
+        self.holding
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner)
+            .len()
+    }
+}
+
+/// A connection's place in its network's count, given back when it closes.
+struct Share {
+    crowd: Arc,
+    /// The network it is counted under; none for one nobody counts.
+    network: Option,
+}
+
+impl Drop for Share {
+    fn drop(&mut self) {
+        let Some(network) = self.network else {
+            return;
+        };
+        let mut holding = self
+            .crowd
+            .holding
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner);
+        if let Some(held) = holding.get_mut(&network) {
+            *held = held.saturating_sub(1);
+            if *held == 0 {
+                holding.remove(&network);
+            }
+        }
+    }
+}
+
+/// What a connection is doing: shared by the stream it is read from and
+/// written to, and the service its requests are given to.
+#[derive(Debug)]
+struct Life {
+    /// The requests begun and not yet done with: a response whose last byte
+    /// was written, or whose caller has gone.
+    open: AtomicUsize,
+    /// When the connection last had none, or began: milliseconds after
+    /// [`EPOCH`].
+    idle_since: AtomicU64,
+}
+
+impl Life {
+    fn new() -> Self {
+        Self {
+            open: AtomicUsize::new(0),
+            idle_since: AtomicU64::new(now_ms()),
+        }
+    }
+
+    /// A request begins.
+    fn begin(self: &Arc) -> Open {
+        self.open.fetch_add(1, Ordering::SeqCst);
+        Open(self.clone())
+    }
+
+    fn open(&self) -> usize {
+        self.open.load(Ordering::SeqCst)
+    }
+
+    fn idle_since(&self) -> u64 {
+        self.idle_since.load(Ordering::SeqCst)
+    }
+}
+
+/// A request in flight, until it is dropped.
+struct Open(Arc);
+
+impl Drop for Open {
+    fn drop(&mut self) {
+        if self.0.open.fetch_sub(1, Ordering::SeqCst) == 1 {
+            self.0.idle_since.store(now_ms(), Ordering::SeqCst);
+        }
     }
 }
 
@@ -89,34 +276,53 @@ pub struct Door {
     listener: TcpListener,
     connections: usize,
     places: Arc,
+    crowd: Arc,
     /// When the door last said it was full, in seconds after [`EPOCH`] plus
     /// one; nought for never.
     said_full: AtomicU64,
+    /// The same, for a network that holds as many as it may.
+    said_crowded: AtomicU64,
 }
 
 static EPOCH: LazyLock = LazyLock::new(Instant::now);
 
+/// Milliseconds since [`EPOCH`].
+fn now_ms() -> u64 {
+    u64::try_from(EPOCH.elapsed().as_millis()).unwrap_or(u64::MAX)
+}
+
+fn millis(time: Duration) -> u64 {
+    u64::try_from(time.as_millis()).unwrap_or(u64::MAX)
+}
+
 impl Door {
-    pub fn new(listener: TcpListener, connections: usize) -> Self {
+    pub fn new(listener: TcpListener, connections: usize, crowd: Crowd) -> Self {
         let connections = connections.max(1);
         Self {
             listener,
             connections,
             places: Arc::new(Semaphore::new(connections)),
+            crowd: Arc::new(crowd),
             said_full: AtomicU64::new(0),
+            said_crowded: AtomicU64::new(0),
         }
     }
+}
+
+/// Whether a thing that stays so a while — a door full, a network holding its
+/// share — is to be said now: once a minute at most.
+fn say(last: &AtomicU64) -> bool {
+    let now = EPOCH.elapsed().as_secs() + 1;
+    let before = last.load(Ordering::Relaxed);
+    (before == 0 || now.saturating_sub(before) >= 60)
+        && last
+            .compare_exchange(before, now, Ordering::Relaxed, Ordering::Relaxed)
+            .is_ok()
+}
 
-    /// Said once a minute at most: a door that is full stays full a while.
+impl Door {
     fn say_full(&self) {
-        let now = EPOCH.elapsed().as_secs() + 1;
-        let last = self.said_full.load(Ordering::Relaxed);
-        if (last == 0 || now.saturating_sub(last) >= 60)
-            && self
-                .said_full
-                .compare_exchange(last, now, Ordering::Relaxed, Ordering::Relaxed)
-                .is_ok()
-        {
+        if say(&self.said_full) {
             tracing::warn!(
                 connections = self.connections,
                 "a door holds as many connections as AMS_MAX_CONNECTIONS allows; new ones wait \
@@ -124,18 +330,31 @@ impl Door {
             );
         }
     }
+
+    fn say_crowded(&self, peer: SocketAddr) {
+        if say(&self.said_crowded) {
+            tracing::warn!(
+                %peer,
+                most = self.crowd.most,
+                "a network holds as many connections as one may; its new ones are closed"
+            );
+        }
+    }
 }
 
 impl AddrListener for Door {
     async fn bind_to(addr: Peer) -> io::Result {
         Ok(Self::new(
-            TcpListener::bind(addr.0).await?,
+            TcpListener::bind(addr.addr).await?,
             DEFAULT_CONNECTIONS,
+            Crowd::new(&[]),
         ))
     }
 
     /// A place first, then the connection: past the limit nothing more is
-    /// accepted, so nothing more holds a descriptor.
+    /// accepted, so nothing more holds a descriptor. A connection whose
+    /// network holds its share already is closed as it is accepted, and the
+    /// place goes to the next.
     async fn accept_stream(&self) -> io::Result<(Held, Peer)> {
         let place = match self.places.clone().try_acquire_owned() {
             Ok(place) => place,
@@ -148,25 +367,41 @@ impl AddrListener for Door {
                     .map_err(io::Error::other)?
             }
         };
-        let (stream, addr) = self.listener.accept().await?;
-        Ok((
-            Held {
-                stream,
-                _place: place,
-            },
-            Peer(addr),
-        ))
+        loop {
+            let (stream, addr) = self.listener.accept().await?;
+            let Some(share) = self.crowd.admit(addr) else {
+                self.say_crowded(addr);
+                drop(stream);
+                continue;
+            };
+            let life = Arc::new(Life::new());
+            return Ok((
+                Held {
+                    stream,
+                    life: life.clone(),
+                    _place: place,
+                    _share: share,
+                },
+                Peer { addr, life },
+            ));
+        }
     }
 
     fn get_local_addr(&self) -> io::Result {
-        self.listener.local_addr().map(Peer)
+        self.listener.local_addr().map(|addr| Peer {
+            addr,
+            life: Arc::new(Life::new()),
+        })
     }
 }
 
-/// An accepted connection, holding its place at the door until it closes.
+/// An accepted connection, holding its place at the door and in its network's
+/// count until it closes.
 pub struct Held {
     stream: TcpStream,
+    life: Arc,
     _place: OwnedSemaphorePermit,
+    _share: Share,
 }
 
 impl AsyncRead for Held {
@@ -209,31 +444,44 @@ impl AsyncWrite for Held {
     }
 }
 
+/// How long a connection may be idle, and how long a write may stall.
+#[derive(Clone, Copy, Debug)]
+struct Tempo {
+    idle: Duration,
+    write: Duration,
+}
+
 /// Gives a connection's first bytes — after TLS, where there is any — the
-/// time a request's head is given.
+/// time a request's head is given, and watches it from then on
+/// ([`Watched`]).
 #[derive(Clone)]
 pub struct Prompt {
     inner: A,
     within: Duration,
+    tempo: Tempo,
 }
 
-impl Accept for Prompt
+impl Accept for Prompt
 where
-    A: Accept,
+    A: Accept,
     A::Future: Send + 'static,
     A::Stream: Send + 'static,
     A::Service: Send + 'static,
 {
-    type Stream = Timed;
+    type Stream = Watched>;
     type Service = A::Service;
     type Future = Pin> + Send>>;
 
-    fn accept(&self, stream: I, service: S) -> Self::Future {
+    fn accept(&self, stream: Held, service: S) -> Self::Future {
+        let life = stream.life.clone();
         let accepted = self.inner.accept(stream, service);
-        let within = self.within;
+        let (within, tempo) = (self.within, self.tempo);
         Box::pin(async move {
             let (stream, service) = accepted.await?;
-            Ok((Timed::new(stream, within), service))
+            Ok((
+                Watched::new(Timed::new(stream, within), life, tempo),
+                service,
+            ))
         })
     }
 }
@@ -341,6 +589,280 @@ impl AsyncWrite for Timed {
     }
 }
 
+/// A connection watched for what hyper's own timers do not see.
+///
+/// *Idle.* An HTTP/2 connection is pinged while it has nothing to say, and a
+/// peer that answers the pings is a peer that is there: kept for ever,
+/// whatever it asks, and so are those of a client that asks for something
+/// once a minute. One with no request in flight for [`Tempo::idle`] is
+/// closed — read to its end, as if its peer had gone — unless bytes are still
+/// coming in, which it is given a moment to finish.
+///
+/// *Stalled writes.* A response a peer does not read fills the send buffer,
+/// and what hyper writes waits for room for ever. A write that makes no
+/// headway for [`Tempo::write`] fails, and the connection with it.
+pub struct Watched {
+    inner: S,
+    life: Arc,
+    tempo: Tempo,
+    /// Wakes a read that has nothing to read when the connection may be due
+    /// to close.
+    idle: Pin>,
+    /// The moment `idle` is set to wake, in milliseconds after [`EPOCH`].
+    armed_for: u64,
+    /// When bytes were last read, in milliseconds after [`EPOCH`].
+    read_at: u64,
+    /// Armed while a write cannot go through.
+    stalled: Option>>,
+}
+
+/// When an idle connection is to be closed, in milliseconds after
+/// [`EPOCH`]: once it has had no request in flight for `idle` — from `since`
+/// — and bytes have stopped coming for [`SETTLE`] (a quarter of `idle`, where
+/// that is less) since `read_at`; but not later than twice `idle` from
+/// `since`, whatever arrives, since a peer that sends something every second
+/// is no peer that wants an answer.
+fn closes_at(since: u64, read_at: u64, idle: Duration) -> u64 {
+    let idle = millis(idle);
+    let settle = millis(SETTLE).min(idle / 4);
+    let earliest = since.saturating_add(idle);
+    let latest = since.saturating_add(idle.saturating_mul(2));
+    earliest.max(read_at.saturating_add(settle).min(latest))
+}
+
+impl Watched {
+    fn new(inner: S, life: Arc, tempo: Tempo) -> Self {
+        Self {
+            inner,
+            life,
+            tempo,
+            idle: Box::pin(tokio::time::sleep(tempo.idle)),
+            armed_for: now_ms().saturating_add(millis(tempo.idle)),
+            read_at: now_ms(),
+            stalled: None,
+        }
+    }
+
+    /// Whether the connection is to be closed now; where it is not, the timer
+    /// that wakes the read when it may be is set, and has the task's waker.
+    fn closes_idle(&mut self, cx: &mut Context<'_>) -> bool {
+        loop {
+            let now = now_ms();
+            let busy = self.life.open() > 0;
+            let at = if busy {
+                // Looked at again after an idle time: what is asked of it
+                // ends, and the connection is idle from then.
+                now.saturating_add(millis(self.tempo.idle))
+            } else {
+                closes_at(self.life.idle_since(), self.read_at, self.tempo.idle)
+            };
+            if !busy && now >= at {
+                return true;
+            }
+            // Set again only where it would wake the read too late, or has
+            // gone off: a busy connection finds nothing to read after every
+            // batch of bytes, and a timer moved each time is work for nothing.
+            // One that wakes it too soon only has it look again.
+            if self.armed_for <= now || self.armed_for > at {
+                let wake = tokio::time::Instant::now() + Duration::from_millis(at - now);
+                self.idle.as_mut().reset(wake);
+                self.armed_for = at;
+            }
+            if self.idle.as_mut().poll(cx).is_pending() {
+                return false;
+            }
+            // Gone off: looked at again, and set for what is then due.
+            self.armed_for = 0;
+        }
+    }
+
+    /// A write's answer, given up on if it has been pending for as long as
+    /// writes may stall.
+    fn watch(
+        &mut self,
+        cx: &mut Context<'_>,
+        polled: Poll>,
+    ) -> Poll> {
+        match polled {
+            Poll::Pending => {
+                let within = self.tempo.write;
+                let stalled = self
+                    .stalled
+                    .get_or_insert_with(|| Box::pin(tokio::time::sleep(within)));
+                if stalled.as_mut().poll(cx).is_ready() {
+                    self.stalled = None;
+                    return Poll::Ready(Err(io::Error::new(
+                        io::ErrorKind::TimedOut,
+                        "the peer did not read what it was sent in time",
+                    )));
+                }
+                Poll::Pending
+            }
+            ready => {
+                self.stalled = None;
+                ready
+            }
+        }
+    }
+}
+
+impl AsyncRead for Watched {
+    fn poll_read(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+        buf: &mut ReadBuf<'_>,
+    ) -> Poll> {
+        let this = self.get_mut();
+        let before = buf.filled().len();
+        match Pin::new(&mut this.inner).poll_read(cx, buf) {
+            Poll::Ready(Ok(())) => {
+                if buf.filled().len() > before {
+                    this.read_at = now_ms();
+                }
+                Poll::Ready(Ok(()))
+            }
+            Poll::Pending => {
+                if this.closes_idle(cx) {
+                    // The end of what the peer sent: hyper takes the peer
+                    // for gone, and closes the connection.
+                    return Poll::Ready(Ok(()));
+                }
+                Poll::Pending
+            }
+            failed => failed,
+        }
+    }
+}
+
+impl AsyncWrite for Watched {
+    fn poll_write(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+        buf: &[u8],
+    ) -> Poll> {
+        let this = self.get_mut();
+        let polled = Pin::new(&mut this.inner).poll_write(cx, buf);
+        this.watch(cx, polled)
+    }
+
+    fn poll_write_vectored(
+        self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+        bufs: &[io::IoSlice<'_>],
+    ) -> Poll> {
+        let this = self.get_mut();
+        let polled = Pin::new(&mut this.inner).poll_write_vectored(cx, bufs);
+        this.watch(cx, polled)
+    }
+
+    fn is_write_vectored(&self) -> bool {
+        self.inner.is_write_vectored()
+    }
+
+    fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> {
+        let this = self.get_mut();
+        let polled = Pin::new(&mut this.inner).poll_flush(cx);
+        this.watch(cx, polled)
+    }
+
+    fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> {
+        let this = self.get_mut();
+        let polled = Pin::new(&mut this.inner).poll_shutdown(cx);
+        this.watch(cx, polled)
+    }
+}
+
+/// What the requests of the connections of a door are served by: made for
+/// each connection, with the address it came from and what it is doing.
+#[derive(Clone)]
+struct Connections {
+    router: Router,
+}
+
+impl Service for Connections {
+    type Response = Tracked;
+    type Error = Infallible;
+    type Future = std::future::Ready>;
+
+    fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll> {
+        Poll::Ready(Ok(()))
+    }
+
+    fn call(&mut self, peer: Peer) -> Self::Future {
+        std::future::ready(Ok(Tracked {
+            router: self.router.clone(),
+            addr: peer.addr,
+            life: peer.life,
+        }))
+    }
+}
+
+/// The router, serving one connection's requests: each told the address the
+/// connection came from — as `ConnectInfo`, which is what the
+/// handlers read — and counted among those in flight until its response's
+/// last byte is written.
+#[derive(Clone)]
+struct Tracked {
+    router: Router,
+    addr: SocketAddr,
+    life: Arc,
+}
+
+impl Service> for Tracked {
+    type Response = Response;
+    type Error = Infallible;
+    type Future = Pin> + Send>>;
+
+    fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll> {
+        Poll::Ready(Ok(()))
+    }
+
+    fn call(&mut self, mut request: Request) -> Self::Future {
+        request.extensions_mut().insert(ConnectInfo(self.addr));
+        // Counted from here, and given to the response's body below: a
+        // request whose future is dropped — its caller gone — is no longer
+        // in flight either.
+        let open = self.life.begin();
+        let mut router = self.router.clone();
+        Box::pin(async move {
+            let response = match router.call(request).await {
+                Ok(response) => response,
+                Err(never) => match never {},
+            };
+            Ok(response.map(|body| Counted { body, _open: open }))
+        })
+    }
+}
+
+/// A response's body, keeping its request among those in flight until it is
+/// dropped — written to its end, or its caller gone. Everything else of the
+/// body is the body's: its frames, whether it is at its end, how large it is
+/// (hyper writes a `Content-Length` of it).
+struct Counted {
+    body: axum::body::Body,
+    _open: Open,
+}
+
+impl HttpBody for Counted {
+    type Data = Bytes;
+    type Error = axum::Error;
+
+    fn poll_frame(
+        mut self: Pin<&mut Self>,
+        cx: &mut Context<'_>,
+    ) -> Poll, axum::Error>>> {
+        Pin::new(&mut self.body).poll_frame(cx)
+    }
+
+    fn is_end_stream(&self) -> bool {
+        self.body.is_end_stream()
+    }
+
+    fn size_hint(&self) -> SizeHint {
+        self.body.size_hint()
+    }
+}
+
 /// hyper's clock, kept by tokio: what lets hyper time a connection out.
 #[derive(Clone, Copy, Debug)]
 struct Clock;
@@ -376,14 +898,18 @@ pub async fn serve(
     handle: Handle,
     limits: Limits,
 ) -> io::Result<()> {
-    let service =
-        axum::ServiceExt::::into_make_service_with_connect_info::(router);
+    let service = Connections { router };
+    let tempo = Tempo {
+        idle: limits.idle,
+        write: limits.write,
+    };
     match tls {
         Some(config) => {
             let mut server = Server::::from_listener(door)
                 .acceptor(Prompt {
                     inner: RustlsAcceptor::new(config),
                     within: limits.head,
+                    tempo,
                 })
                 .handle(handle);
             tune(&mut server, limits);
@@ -394,6 +920,7 @@ pub async fn serve(
                 .acceptor(Prompt {
                     inner: DefaultAcceptor::new(),
                     within: limits.head,
+                    tempo,
                 })
                 .handle(handle);
             tune(&mut server, limits);
@@ -423,14 +950,51 @@ mod tests {
     use super::*;
     use tokio::io::{AsyncReadExt, AsyncWriteExt};
 
-    /// A door serving a single route, with the limits given; its address.
-    async fn door(limits: Limits) -> (SocketAddr, Handle) {
+    impl Crowd {
+        /// Every network counted, this machine's too: what a test of the
+        /// count runs on, over loopback.
+        fn counting_all(most: usize) -> Self {
+            Self::of(most, Vec::new())
+        }
+    }
+
+    /// The limits a door of these tests has: generous, but for what a test
+    /// narrows.
+    fn limits() -> Limits {
+        Limits {
+            connections: 8,
+            head: Duration::from_secs(30),
+            idle: Duration::from_secs(60),
+            write: Duration::from_secs(60),
+        }
+    }
+
+    /// A door serving a few routes, with the limits given, its network
+    /// counted as the crowd says; its address.
+    async fn door_with(limits: Limits, crowd: Crowd) -> (SocketAddr, Handle) {
         let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
         let addr = listener.local_addr().unwrap();
-        let router = Router::new().route("/", axum::routing::get(|| async { "here" }));
+        let router = Router::new()
+            .route("/", axum::routing::get(|| async { "here" }))
+            .route(
+                "/slow",
+                axum::routing::get(|| async {
+                    tokio::time::sleep(Duration::from_millis(900)).await;
+                    "slowly"
+                }),
+            )
+            .route(
+                "/endless",
+                axum::routing::get(|| async {
+                    let chunks = futures::stream::repeat_with(|| {
+                        Ok::<_, io::Error>(Bytes::from(vec![0u8; 64 * 1024]))
+                    });
+                    axum::body::Body::from_stream(chunks)
+                }),
+            );
         let handle = Handle::new();
         tokio::spawn(serve(
-            Door::new(listener, limits.connections),
+            Door::new(listener, limits.connections, crowd),
             None,
             router,
             handle.clone(),
@@ -439,6 +1003,10 @@ mod tests {
         (addr, handle)
     }
 
+    async fn door(limits: Limits) -> (SocketAddr, Handle) {
+        door_with(limits, Crowd::new(&[])).await
+    }
+
     async fn ask(addr: SocketAddr) -> String {
         let mut stream = TcpStream::connect(addr).await.unwrap();
         stream
@@ -456,11 +1024,31 @@ mod tests {
         (head.lines().next().unwrap_or_default(), body)
     }
 
+    /// Whether the door has closed a connection: its end is read within
+    /// `within`, and nothing else is.
+    async fn closed_within(stream: &mut TcpStream, within: Duration) -> bool {
+        let mut rest = Vec::new();
+        matches!(
+            tokio::time::timeout(within, stream.read_to_end(&mut rest)).await,
+            Ok(Ok(_) | Err(_))
+        )
+    }
+
+    /// Whether the door leaves a connection open for `for_`: nothing is read
+    /// of it, and it is not closed.
+    async fn open_for(stream: &mut TcpStream, for_: Duration) -> bool {
+        let mut byte = [0u8; 1];
+        tokio::time::timeout(for_, stream.read(&mut byte))
+            .await
+            .is_err()
+    }
+
     #[tokio::test]
     async fn a_door_answers_through_its_limits() {
         let (addr, handle) = door(Limits {
             connections: 4,
             head: Duration::from_secs(5),
+            ..limits()
         })
         .await;
         let answer = ask(addr).await;
@@ -473,6 +1061,7 @@ mod tests {
         let (addr, handle) = door(Limits {
             connections: 4,
             head: Duration::from_millis(200),
+            ..limits()
         })
         .await;
 
@@ -516,6 +1105,7 @@ mod tests {
         let (addr, handle) = door(Limits {
             connections: 1,
             head: Duration::from_secs(30),
+            ..limits()
         })
         .await;
 
@@ -557,4 +1147,366 @@ mod tests {
         nothing.saw(b"");
         assert!(nothing.deadline.is_some());
     }
+
+    // ── the count of a network's connections ────────────────────────────
+
+    fn peer(address: &str) -> SocketAddr {
+        format!("{address}:40000").parse().unwrap()
+    }
+
+    #[test]
+    fn a_network_holds_so_many_connections_and_gives_a_share_back() {
+        let crowd = Arc::new(Crowd::counting_all(2));
+        let first = crowd.admit(peer("203.0.113.5")).expect("a share");
+        let _second = crowd.admit(peer("203.0.113.5")).expect("a share");
+        assert!(
+            crowd.admit(peer("203.0.113.5")).is_none(),
+            "a third is more than one network may hold"
+        );
+        // Another address has its own.
+        let _other = crowd.admit(peer("203.0.113.6")).expect("a share");
+        assert_eq!(crowd.networks(), 2);
+
+        // A share given back is the next connection's.
+        drop(first);
+        let _third = crowd
+            .admit(peer("203.0.113.5"))
+            .expect("the share given back");
+        assert!(crowd.admit(peer("203.0.113.5")).is_none());
+    }
+
+    #[test]
+    fn a_count_does_not_outlive_its_connections() {
+        let crowd = Arc::new(Crowd::counting_all(3));
+        let shares: Vec<_> = (0..3)
+            .map(|_| crowd.admit(peer("198.51.100.7")).expect("a share"))
+            .collect();
+        assert_eq!(crowd.networks(), 1);
+        drop(shares);
+        assert_eq!(crowd.networks(), 0, "nothing is kept of a network gone");
+    }
+
+    /// One machine has a whole /64 to pick its addresses from: they are one
+    /// network, as the rate limit counts them.
+    #[test]
+    fn an_ipv6_network_holds_one_count_between_its_addresses() {
+        let crowd = Arc::new(Crowd::counting_all(2));
+        let _a = crowd.admit(peer("[2001:db8:1:2::1]")).expect("a share");
+        let _b = crowd
+            .admit(peer("[2001:db8:1:2:ffff::9]"))
+            .expect("a share");
+        assert!(crowd.admit(peer("[2001:db8:1:2::3]")).is_none());
+        assert!(crowd.admit(peer("[2001:db8:1:3::1]")).is_some());
+
+        // An IPv4 client of a dual-stack socket is its IPv4 address.
+        let _c = crowd.admit(peer("[::ffff:192.0.2.1]")).expect("a share");
+        let _d = crowd.admit(peer("192.0.2.1")).expect("a share");
+        assert!(crowd.admit(peer("[::ffff:192.0.2.1]")).is_none());
+    }
+
+    /// Everybody behind a proxy has its address, and this machine's own are
+    /// the operator's: neither is counted.
+    #[test]
+    fn a_trusted_proxy_and_this_machine_are_not_counted() {
+        let proxies: Vec = vec!["10.0.0.0/8".parse().unwrap(), "fd00::/16".parse().unwrap()];
+        let mut crowd = Crowd::new(&proxies);
+        crowd.most = 1;
+        let crowd = Arc::new(crowd);
+        let mut held = Vec::new();
+        for address in [
+            "10.1.2.3",
+            "[fd00::7]",
+            "127.0.0.1",
+            "[::1]",
+            "[::ffff:127.0.0.1]",
+        ] {
+            for _ in 0..5 {
+                held.push(
+                    crowd
+                        .admit(peer(address))
+                        .unwrap_or_else(|| panic!("{address} is counted")),
+                );
+            }
+        }
+        assert_eq!(crowd.networks(), 0);
+
+        // Any other is, and holds the one.
+        let _only = crowd.admit(peer("203.0.113.5")).expect("a share");
+        assert!(crowd.admit(peer("203.0.113.5")).is_none());
+        // The standard crowd leaves this machine out and counts the rest.
+        let standard = Arc::new(Crowd::new(&[]));
+        let counted: Vec<_> = (0..PER_PEER)
+            .map(|_| standard.admit(peer("203.0.113.5")).expect("a share"))
+            .collect();
+        assert!(standard.admit(peer("203.0.113.5")).is_none());
+        drop(counted);
+    }
+
+    #[tokio::test]
+    async fn a_network_past_its_share_is_closed_and_the_others_are_answered() {
+        let (addr, handle) = door_with(limits(), Crowd::counting_all(2)).await;
+
+        // Two connections, as many as this network may hold: idle, open.
+        let mut first = TcpStream::connect(addr).await.unwrap();
+        let mut second = TcpStream::connect(addr).await.unwrap();
+        tokio::time::sleep(Duration::from_millis(100)).await;
+
+        // The third is accepted, and closed with no word.
+        let mut third = TcpStream::connect(addr).await.unwrap();
+        assert!(
+            closed_within(&mut third, Duration::from_secs(5)).await,
+            "the door should have closed a third connection of one network"
+        );
+        // The two it holds are left as they were.
+        assert!(open_for(&mut first, Duration::from_millis(200)).await);
+        assert!(open_for(&mut second, Duration::from_millis(200)).await);
+
+        // One given back, the next is answered.
+        drop(first);
+        tokio::time::sleep(Duration::from_millis(100)).await;
+        let answer = tokio::time::timeout(Duration::from_secs(5), ask(addr))
+            .await
+            .expect("answered");
+        assert_eq!(parts(&answer).0, "HTTP/1.1 200 OK");
+        handle.shutdown();
+    }
+
+    // ── a connection with nothing in flight ─────────────────────────────
+
+    #[test]
+    fn an_idle_connection_is_closed_once_it_is_quiet_and_never_later_than_twice_its_time() {
+        let idle = Duration::from_secs(120);
+        // Quiet: closed when its idle time is up.
+        assert_eq!(closes_at(0, 0, idle), 120_000);
+        // Bytes read just before: a moment more, for what is on its way.
+        assert_eq!(closes_at(0, 119_900, idle), 120_900);
+        // Bytes read all the time: not later than twice the time.
+        assert_eq!(closes_at(0, 239_999, idle), 240_000);
+        assert_eq!(closes_at(1_000, 10_000_000, idle), 241_000);
+        // A short time settles in a quarter of it.
+        assert_eq!(closes_at(0, 100, Duration::from_millis(400)), 400);
+        assert_eq!(closes_at(0, 350, Duration::from_millis(400)), 450);
+    }
+
+    /// What HTTP/2 opens with and says next, as an idle client sends it: the
+    /// preface and an empty SETTINGS frame, and a PING.
+    fn h2_hello() -> Vec {
+        let mut bytes = PREFACE.to_vec();
+        bytes.extend_from_slice(&[0, 0, 0, 4, 0, 0, 0, 0, 0]);
+        bytes
+    }
+
+    fn h2_ping() -> [u8; 17] {
+        let mut ping = [0u8; 17];
+        ping[2] = 8;
+        ping[3] = 6;
+        ping
+    }
+
+    /// An idle HTTP/2 connection is closed whatever it answers — here, a
+    /// client that pings every fifty milliseconds — within twice the idle
+    /// time.
+    #[tokio::test]
+    async fn an_h2_connection_with_no_request_is_closed_though_it_pings() {
+        let (addr, handle) = door(Limits {
+            idle: Duration::from_millis(300),
+            ..limits()
+        })
+        .await;
+        let mut stream = TcpStream::connect(addr).await.unwrap();
+        stream.write_all(&h2_hello()).await.unwrap();
+
+        let started = Instant::now();
+        let mut buf = [0u8; 4096];
+        let mut closed_after = None;
+        while started.elapsed() < Duration::from_secs(10) {
+            if stream.write_all(&h2_ping()).await.is_err() {
+                closed_after = Some(started.elapsed());
+                break;
+            }
+            if let Ok(Ok(0) | Err(_)) =
+                tokio::time::timeout(Duration::from_millis(50), stream.read(&mut buf)).await
+            {
+                closed_after = Some(started.elapsed());
+                break;
+            }
+        }
+        let closed_after = closed_after.expect("the door should have closed the connection");
+        assert!(
+            closed_after >= Duration::from_millis(250),
+            "closed after {closed_after:?}, before its idle time"
+        );
+        assert!(
+            closed_after <= Duration::from_secs(5),
+            "closed after {closed_after:?}"
+        );
+        handle.shutdown();
+    }
+
+    /// An HTTP/1 connection kept alive between two requests is closed once
+    /// it has had none in flight for the idle time — here, shorter than the
+    /// head's.
+    #[tokio::test]
+    async fn an_h1_connection_with_no_request_is_closed_after_its_idle_time() {
+        let (addr, handle) = door(Limits {
+            idle: Duration::from_millis(800),
+            ..limits()
+        })
+        .await;
+        let mut stream = TcpStream::connect(addr).await.unwrap();
+        stream
+            .write_all(b"GET / HTTP/1.1\r\nHost: x\r\n\r\n")
+            .await
+            .unwrap();
+        let mut buf = [0u8; 1024];
+        let read = stream.read(&mut buf).await.unwrap();
+        assert!(String::from_utf8_lossy(&buf[..read]).starts_with("HTTP/1.1 200 OK"));
+
+        // Kept alive, and then closed.
+        assert!(open_for(&mut stream, Duration::from_millis(150)).await);
+        assert!(
+            closed_within(&mut stream, Duration::from_secs(3)).await,
+            "the door should have closed the idle connection"
+        );
+        handle.shutdown();
+    }
+
+    /// A connection with a request in flight is not idle, however long the
+    /// request takes: the answer comes, and the connection is closed after
+    /// it, once the idle time has passed.
+    #[tokio::test]
+    async fn a_request_in_flight_keeps_its_connection_open() {
+        let (addr, handle) = door(Limits {
+            idle: Duration::from_millis(300),
+            ..limits()
+        })
+        .await;
+        let mut stream = TcpStream::connect(addr).await.unwrap();
+        stream
+            .write_all(b"GET /slow HTTP/1.1\r\nHost: x\r\n\r\n")
+            .await
+            .unwrap();
+
+        // The handler takes three times the idle time to answer.
+        let mut answer = Vec::new();
+        let mut buf = [0u8; 1024];
+        while !String::from_utf8_lossy(&answer).contains("slowly") {
+            let read = tokio::time::timeout(Duration::from_secs(5), stream.read(&mut buf))
+                .await
+                .expect("an answer")
+                .expect("read");
+            assert!(read > 0, "closed before it was answered");
+            answer.extend_from_slice(&buf[..read]);
+        }
+        assert!(String::from_utf8_lossy(&answer).starts_with("HTTP/1.1 200 OK"));
+
+        // Nothing in flight now: closed once it has been idle.
+        assert!(
+            closed_within(&mut stream, Duration::from_secs(3)).await,
+            "the door should have closed the connection after its answer"
+        );
+        handle.shutdown();
+    }
+
+    // ── a response nobody reads ─────────────────────────────────────────
+
+    /// A response a peer does not read is given up on once a write has made
+    /// no headway for the write time, and the connection with it.
+    #[tokio::test]
+    async fn a_response_nobody_reads_is_given_up_on() {
+        let (addr, handle) = door(Limits {
+            write: Duration::from_millis(300),
+            ..limits()
+        })
+        .await;
+        let mut stream = TcpStream::connect(addr).await.unwrap();
+        stream
+            .write_all(b"GET /endless HTTP/1.1\r\nHost: x\r\n\r\n")
+            .await
+            .unwrap();
+
+        // Nothing is read while the send buffers fill and the write stalls.
+        tokio::time::sleep(Duration::from_millis(1200)).await;
+
+        // What was written before is there; and then the end, long before
+        // an endless body's.
+        let mut got = 0usize;
+        let mut buf = vec![0u8; 256 * 1024];
+        let ended = tokio::time::timeout(Duration::from_secs(10), async {
+            loop {
+                match stream.read(&mut buf).await {
+                    Ok(0) | Err(_) => return,
+                    Ok(read) => got += read,
+                }
+            }
+        })
+        .await;
+        assert!(
+            ended.is_ok(),
+            "the door should have given up on the response"
+        );
+        assert!(
+            got < 64 * 1024 * 1024,
+            "read {got} bytes of an endless body"
+        );
+        handle.shutdown();
+    }
+
+    /// A peer that reads what it is sent is not stalled, however slowly it
+    /// asks for the rest: the clock runs only while a write cannot go through.
+    #[tokio::test]
+    async fn a_response_that_is_read_is_not_given_up_on() {
+        let (addr, handle) = door(Limits {
+            write: Duration::from_secs(1),
+            ..limits()
+        })
+        .await;
+        let mut stream = TcpStream::connect(addr).await.unwrap();
+        stream
+            .write_all(b"GET /endless HTTP/1.1\r\nHost: x\r\n\r\n")
+            .await
+            .unwrap();
+        let mut buf = vec![0u8; 256 * 1024];
+        let mut got = 0usize;
+        let started = Instant::now();
+        while started.elapsed() < Duration::from_millis(2500) {
+            let read = tokio::time::timeout(Duration::from_secs(5), stream.read(&mut buf))
+                .await
+                .expect("bytes keep coming")
+                .expect("read");
+            assert!(read > 0, "the door closed a connection that was reading");
+            got += read;
+        }
+        assert!(got > 1024 * 1024, "read {got} bytes");
+        handle.shutdown();
+    }
+
+    /// The service's requests count as in flight until the response's last
+    /// byte, and the body is the router's own: its length is still told.
+    #[tokio::test]
+    async fn the_body_keeps_its_length_and_its_request_counts_until_it_is_dropped() {
+        let life = Arc::new(Life::new());
+        let open = life.begin();
+        assert_eq!(life.open(), 1);
+        let counted = Counted {
+            body: axum::body::Body::from("twelve bytes"),
+            _open: open,
+        };
+        assert_eq!(counted.size_hint().exact(), Some(12));
+        assert!(!counted.is_end_stream());
+        assert_eq!(life.open(), 1);
+        let before = life.idle_since();
+        tokio::time::sleep(Duration::from_millis(20)).await;
+        drop(counted);
+        assert_eq!(life.open(), 0);
+        assert!(life.idle_since() > before, "idle from the moment it ended");
+
+        let empty = Counted {
+            body: axum::body::Body::empty(),
+            _open: life.begin(),
+        };
+        assert!(empty.is_end_stream());
+        drop(empty);
+        assert_eq!(life.open(), 0);
+    }
 }

From e049c4334eea044be1ed6ade5d6832aebcdd52c2 Mon Sep 17 00:00:00 2001
From: dim145 
Date: Tue, 3 Nov 2026 21:38:41 +0100
Subject: [PATCH 2/3] Give the providers' keys an environment of their own,
 read by the end-to-end tests and the parity check alone

zizmor's auditor persona found TMDB_API_KEY, TVDB_API_KEY and
FANARTTV_API_KEY read as repository secrets, which every job of the
repository can reach. The two jobs that run against the real services now
take them from the GitHub environment provider-keys; once the secrets are
moved there and the repository's copies deleted, no other job can read them.
Forgejo, which runs both workflows too, has no environments and keeps its
own secrets.

Co-Authored-By: Claude Opus 5.5 
---
 .github/workflows/ci.yml     |  5 +++++
 .github/workflows/parity.yml | 14 +++++++++-----
 CONTRIBUTING.md              |  4 ++++
 3 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 34a3562..bb5a04c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -369,6 +369,11 @@ jobs:
     if: needs.gate.outputs.full == 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 45
+    # The providers' keys are the secrets of this environment, given to this
+    # job and to the parity check alone rather than to every job of the
+    # repository. Forgejo, which runs this workflow too, has no environments
+    # and reads its own secrets.
+    environment: provider-keys
 
     services:
       # The cache's second tier, which the cache spec reads back; the server is
diff --git a/.github/workflows/parity.yml b/.github/workflows/parity.yml
index 1c0696c..59e6f83 100644
--- a/.github/workflows/parity.yml
+++ b/.github/workflows/parity.yml
@@ -36,14 +36,18 @@ jobs:
     name: Skyhook v2 parity
     runs-on: ubuntu-latest
     timeout-minutes: 20
+    # The providers' keys are the secrets of this environment, given to this
+    # job and to the end-to-end tests alone. Forgejo, which runs this workflow
+    # too, has no environments and reads its own secrets.
+    environment: provider-keys
 
     steps:
       # The server answers the series from TheTVDB and TMDB, so the run needs
-      # their keys, as the end-to-end tests do: the repository secrets
-      # TMDB_API_KEY and TVDB_API_KEY. Said before anything is fetched or
-      # built, so that a run without them costs a few seconds. A secret
-      # cannot be tested in an `if`, which is why it is an environment
-      # variable here, and why the answer is an output.
+      # their keys, as the end-to-end tests do: the secrets TMDB_API_KEY and
+      # TVDB_API_KEY of the environment provider-keys. Said before anything
+      # is fetched or built, so that a run without them costs a few seconds.
+      # A secret cannot be tested in an `if`, which is why it is an
+      # environment variable here, and why the answer is an output.
       - name: Look for the provider keys
         id: keys
         env:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 0b83abd..0c887a2 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -99,6 +99,10 @@ why. Say in the pull request why a new dependency is worth carrying.
 `gitleaks detect --source . --redact` reads the history with
 [`.gitleaks.toml`](.gitleaks.toml); the few lines of past commits that are no
 secret are named, each with its reason, in [`.gitleaksignore`](.gitleaksignore).
+CI's own keys — `TMDB_API_KEY`, `TVDB_API_KEY` and `FANARTTV_API_KEY`, which the
+end-to-end tests and the parity check run against the real services with — are
+the secrets of the GitHub environment `provider-keys`, given to those two jobs
+alone. Without them, both say so and stop within seconds.
 
 ## The interface's design system
 

From 24b681a0ebf0c2c939b451318514da32085f5a54 Mon Sep 17 00:00:00 2001
From: dim145 
Date: Tue, 3 Nov 2026 21:47:16 +0100
Subject: [PATCH 3/3] Put the redaction test's made-up passwords in as it runs
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

GitGuardian read three of the addresses the test writes — a password as a
parameter, and one holding an `@` — as passwords left in the code. They are
made up; they are now put in as the test runs, so that no line reads as an
address and its password.

Co-Authored-By: Claude Opus 5.5 
---
 src/db/mod.rs | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/src/db/mod.rs b/src/db/mod.rs
index 139b6ba..038c10a 100644
--- a/src/db/mod.rs
+++ b/src/db/mod.rs
@@ -1015,32 +1015,35 @@ mod tests {
 
     #[test]
     fn a_password_given_as_a_parameter_goes_too() {
+        // The made-up values are put in as the test runs, so that no line
+        // here reads as an address and its password to a secret scanner.
+        let (made_up, with_at) = ("s3cret", "a@b");
         for (url, shown) in [
             (
-                "postgres://db/ams?user=ams&password=s3cret&sslmode=require",
+                format!("postgres://db/ams?user=ams&password={made_up}&sslmode=require"),
                 "postgres://db/ams?user=ams&password=***&sslmode=require",
             ),
             (
-                "postgres:///ams?host=/var/run/postgresql&PASSWORD=s3cret",
+                format!("postgres:///ams?host=/var/run/postgresql&PASSWORD={made_up}"),
                 "postgres:///ams?host=/var/run/postgresql&PASSWORD=***",
             ),
             (
-                "postgres://db/ams?%70assword=s3cret&sslpassword=k3y",
+                format!("postgres://db/ams?%70assword={made_up}&sslpassword={made_up}"),
                 "postgres://db/ams?%70assword=***&sslpassword=***",
             ),
             // Both; and a fragment, which is not the query's.
             (
-                "postgres://ams:a@b@db/ams?pwd=z&sslmode=disable#frag",
+                format!("postgres://ams:{with_at}@db/ams?pwd={made_up}&sslmode=disable#frag"),
                 "postgres://ams:***@db/ams?pwd=***&sslmode=disable#frag",
             ),
             // An empty value says nothing; a name that only contains one
             // is not one.
             (
-                "postgres://db/ams?password=&compass=north",
+                "postgres://db/ams?password=&compass=north".to_string(),
                 "postgres://db/ams?password=&compass=north",
             ),
         ] {
-            assert_eq!(redact(url), shown, "{url}");
+            assert_eq!(redact(&url), shown, "{url}");
         }
         // An `@` in a parameter's value is not where the host begins.
         assert_eq!(