diff --git a/README.md b/README.md index 38aa286b..851d05b2 100644 --- a/README.md +++ b/README.md @@ -70,10 +70,10 @@ Humans and AI agents are both first-class citizens, with different powers by des | D1 | changes, issues, events, audit, costs | **required** | | KV | project/workspace identity, session state | **required** | | Queues | imports, events, webhook delivery | recommended | - | Durable Objects | merge queue, repo hot index, rate limiting | recommended | + | Durable Objects | merge queue, repo hot index, rate limiting, usage meter | recommended | | R2 | backups; backups no-op when unbound | recommended | | Analytics Engine | request analytics | optional | - | Workers AI | the LLM evaluator | optional | + | Workers AI | the LLM evaluator (unless projects bring their own provider key) | optional | | Sandboxes | sandbox evaluator, post-merge smoke tests | **beta, off by default** | `wrangler.toml` declares every binding above **except `[[sandboxes]]`, which is commented @@ -81,11 +81,13 @@ Humans and AI agents are both first-class citizens, with different powers by des self-host has it. Uncomment it (and add it to each `[env.*]` block you deploy) only once your account has Sandboxes access. - Missing bindings do not crash the Worker, but the two evaluator bindings **fail closed** - rather than degrade: with no `SANDBOX` binding a `sandbox` evaluator is replaced by one - that returns score 0 / failed, and the same is true of `llm` with no `AI` binding. Naming - either in `merge.requiredEvaluators` while its binding is absent blocks **every** merge in - that project. `merge.postMergeCommand` is the exception — with no `SANDBOX` binding it is + Missing bindings do not crash the Worker, but the evaluators that need one **fail + closed** rather than degrade: with no `SANDBOX` binding a `sandbox` evaluator is + replaced by one that returns score 0 / failed, and an `llm` evaluator does the same + with no `AI` binding — *unless* it names a provider from `LLM_PROVIDERS`, which runs + on the project's own key and needs no `AI` binding at all. Naming an evaluator in + `merge.requiredEvaluators` while the binding it actually uses is absent blocks + **every** merge in that project. `merge.postMergeCommand` is the exception — with no `SANDBOX` binding it is skipped with a warning. ### Installation @@ -218,7 +220,7 @@ into an editor's config, and **Settings → Connected applications** revokes acc immediately. Headless callers can present a `stratum_user_` or `stratum_agent_` token directly instead. -That exposes the whole eval-gated change flow as eighteen MCP tools, so Claude Code, +That exposes the whole eval-gated change flow as nineteen MCP tools, so Claude Code, Cursor, Zed, Copilot, or a custom agent can drive Stratum without a bespoke integration. See the [MCP guide](docs/user-guide/mcp.md). @@ -307,10 +309,10 @@ commit log, and tags. **The evaluation gate** Secret scanner (always on and blocking; 25+ credential patterns plus entropy detection) · -diff analysis · webhook for external CI · LLM review via the Workers AI binding · sandboxed -test execution (Sandboxes beta; see [Prerequisites](#prerequisites)) · per-evaluator -evidence and estimated resource costs (LLM tokens, sandbox time, git ops) · branch -protection · provenance recorded per merged commit. +diff analysis · webhook for external CI · LLM review, on the Workers AI binding or on the +project's own provider key · sandboxed test execution (Sandboxes beta; see +[Prerequisites](#prerequisites)) · per-evaluator evidence and metered resource costs (LLM +tokens, sandbox time, git ops) · branch protection · provenance recorded per merged commit. **Post-merge deployments** A `deploys:` block in `.stratum/policy.yaml` publishes the merged tree to Cloudflare diff --git a/ROADMAP.md b/ROADMAP.md index dcc9f9da..1bd0df04 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -44,8 +44,13 @@ under [Deployments](#deployments). server-side `changes/merge-batch` endpoint already does this for an explicit batch; this item is the queue doing it automatically.) - [ ] **SSO/SAML** — enterprise sign-in alongside magic-link, GitHub OAuth, and Google OAuth. -- [ ] **Multi-tenancy and billing** — tenant isolation, usage metering, billing. Per-change - cost tracking already exists and provides the metering foundation. +- [ ] **Multi-tenancy and billing** — usage metering, entitlements and enforcement have + shipped: attributed cost records, an owner-scoped `usage_periods` aggregate, a + `UsageMeter` Durable Object, an inert-by-default entitlements seam, observe-only + enforcement, BYOK for the `llm` evaluator, and per-account usage visibility. What + remains is the paid half — plan definitions and checkout live outside this repository + — plus org/seat billing, retention and storage limits, and a per-user cap on org + creation. - [ ] **Monitoring dashboard UI** — a UI over the existing `/api/admin/metrics` (queue depth, evaluation latency, error rates, event outbox lag). - [x] **Backup strategy for D1 and Artifacts** — daily and on-demand backups to R2 with a diff --git a/changelog.d/370-metered-allowances-byok.md b/changelog.d/370-metered-allowances-byok.md new file mode 100644 index 00000000..3664ce4f --- /dev/null +++ b/changelog.d/370-metered-allowances-byok.md @@ -0,0 +1,162 @@ +### Added + +- **Bring your own model for the merge gate.** The `llm` evaluator has only ever + been able to call the instance's Workers AI binding, so the reviewer model was + the operator's choice and the tokens were the operator's bill. A project can + now run that gate on its own account: the operator lists named providers in a + new `LLM_PROVIDERS` variable (`anthropic`, or anything OpenAI-compatible), the + project stores its key in the same encrypted per-project secret store deploys + already use, and `.stratum/policy.yaml` names the provider it wants: + + ```yaml + evaluators: + - type: llm + provider: anthropic + model: claude-sonnet-4-5 + threshold: 0.7 + ``` + + Self-hosting, this is yours to configure and costs you nothing to ignore: + with `LLM_PROVIDERS` unset — the default — the binding is the only option and + every path below is inert. + + Three things are worth knowing before you switch. **A policy may never name a + URL**, only select a provider the operator configured, because a base URL + chosen by repository content turns the Worker into a request-forgery + primitive pointed at a host of the author's choosing — the `llm` entry is a + whitelist of `provider`, `model`, `threshold` and `maxDiffChars`, and + `model` is required whenever `provider` is set. **Everything fails closed**: + an unconfigured provider name, a missing or undecryptable key, an instance + with no `DEPLOY_SECRET_KEY`, or a provider that answers with a redirect fails + the gate with a reason saying which, and never quietly falls back to the + operator's binding. And **any project writer can spend the owner's credit** by + opening changes; the hourly evaluation ceiling still applies, and your own key + does not lift it. Rotating `DEPLOY_SECRET_KEY` — already documented as making + every stored deploy secret undecryptable — now also blocks the gate on BYOK + projects until each re-enters its key. The threat model is + `docs/adr/008-llm-provider-byok-threat-model.md`. + +- **See what you are spending.** A new `/settings/usage` page shows this + account's consumption against its allowances for the current period, what is + left, and when it resets; `GET /api/users/me/usage` and a nineteenth MCP tool, + `stratum_get_usage`, return the same figures, so an agent that can see 12% of + its allowance left can batch its changes or ask its human instead of + discovering the wall by hitting it. Crossing 80% of a meter raises a banner and + sends one email per crossing. Spend on a project's own provider key is reported + separately and never counted against an allowance. The billing surface is + **read-only everywhere** — no endpoint and no tool can raise a limit, buy + capacity, or store a provider key, for the same reason agent tokens cannot + submit review verdicts. Self-hosted, the page reads "unlimited" and no banner + or email is ever sent. + +- **Metered allowances — inert unless you connect a billing service.** Stratum + can now check usage against a plan: an entitlements seam behind + `BILLING_SERVICE_URL` + `BILLING_SERVICE_SECRET`, a `UsageMeter` Durable Object + holding the monthly counters and a bucketed sliding `evaluations_per_hour` + window, and checks at the LLM evaluator, the deploy consumer, the request + limiter and private-project creation. **With those two variables unset — every + self-hosted deployment — none of it does anything**: every allowance reads as + unlimited, the meter binding is never touched, and nothing can be refused. Even + configured, every decision is observe-only (recorded, then admitted) unless + `ENTITLEMENTS_ENFORCE=1`. Plan definitions and payment live outside this + repository. One design note that outlives the billing service: a limit is + checked against the **person who ran the evaluation**, not the project's owner, + so an allowance follows you across your own and your organizations' projects, + and creating another organization does not hand you a fresh one. + +### Changed + +- **Every cost record now names who pays for it, and LLM token counts are the + real ones.** `cost_records` gains `owner_id`, `owner_type` and `source` + (`platform` or `byok`), and a new `usage_periods` table keeps the owner-scoped + monthly totals those roll up into — migrations 048 and 049, applied like any + other. Attribution is **always on**, self-hosted included, because it is a + ledger rather than a paywall: expect a small number of additional D1 writes per + evaluated change, and a `USAGE_METER` Durable Object binding in `wrangler.toml` + that is created on deploy and never contacted unless a billing service is + configured. Token accounting stopped guessing at the same time: Workers AI + reports `prompt_tokens`/`completion_tokens` and those are now recorded as-is, + where every LLM cost record used to be a `~4 chars/token` estimate. The + estimate survives only as a per-response fallback for a provider that omits the + counts, and a record built that way still says `estimated`. + +### Fixed + +- **A policy file is bounded, and an unusable `llm` entry says why.** + `evaluators:` now takes at most 16 entries, the cap `deploys:` has always had + and for the same reason — one merge must not amplify into an unbounded number + of external calls. An entry that names a provider must also name a `model`, + since the default is a Workers AI model id that fails every call against any + other provider. The merge-blocking reason now names the provider (or the field) + that made the entry unusable instead of only counting entries. A verdict + truncated at the provider's token cap records the tokens the provider reported + for it, rather than recording a charged call as free. +- **A deploy refusal can no longer overwrite a deployment that is running.** The + paths that write a terminal status without claiming the row first — a failed + guard, a superseded commit, and now an exhausted allowance — run before + `claimDeployment`, and `running` is not a terminal status, so a redelivery of + the same message could land `failed` on a row another delivery was mid-deploy + on and clear its lease. Those writes are now refused against a claimed row, + and the `deployment.failed` event they emit is withheld when the write does + not land: an event announcing a failure for a deploy that is about to succeed + is the one a webhook consumer acts on. +- **The commented `LLM_PROVIDERS` example carries the API version path.** The + provider appends `/messages` (or `/chat/completions`) to `baseUrl`, so the + `https://api.anthropic.com` shown in `wrangler.toml` would have 404'd on every + call; it is `https://api.anthropic.com/v1` now. A `baseUrl` is also stored + normalized, so a trailing `#` or `?` — both of which pass the "no fragment, no + query" check — cannot swallow the path the provider appends. + +### Security + +- **A project secret can no longer escape through an error message.** A stored + value containing CR, LF or NUL made the outbound provider request's header + construction throw a `TypeError` that quotes the offending value; that message + became an evaluation reason, which is persisted on the change and rendered — + world-readably, for a public project. Both ends are closed: the store rejects + control characters in a value, and the provider builds its headers where a + failure maps to a constant that interpolates nothing. +- **The LLM provider request no longer follows redirects.** The allowlist + validates the host a request is *sent* to; a 3xx from an allowlisted host used + to move the prompt (the diff and the policy) to an unvalidated one, re-sending + the body on 307/308 with the project's `x-api-key` attached, which the Fetch + spec does not strip cross-origin. A redirect is now a failed evaluation. +- **Credential stripping covers the whole policy, recursively.** Fields whose + name says "credential" were removed only from the top level of each evaluator + entry, so one written at the top of `.stratum/policy.yaml`, or nested inside an + entry of an unmodelled type, reached the review model and the body POSTed to a + policy-supplied webhook URL. The word list also missed common names — + `auth`, plurals such as `tokens` and `apiKeys`, `pwd`, `pat`, `hmac`, `sig` — + while keeping innocuous ones like `keystone`. +- **One host filter, not two — which also closed a pre-existing SSRF hole in the + webhook evaluator.** The LLM provider allowlist had its own copy of the + private-address check, and it had drifted: CGNAT (`100.64/10`), the + `.internal`/`.local` suffixes (GCP's metadata endpoint is + `metadata.google.internal`), everything in `fe80::/10` not spelled `fe80`, and + IPv4-compatible IPv6 (`[::127.0.0.1]`, which the URL parser rewrites to + `[::7f00:1]`) all got through. Both callers now share one filter, which + expands an IPv6 literal rather than matching its spelling. The webhook + evaluator's URL check — which has shipped for far longer, and accepts a URL + straight from `.stratum/policy.yaml` — had the last two gaps too, so a + `webhook` entry pointed at `[::127.0.0.1]` or at `feb0::…` used to reach the + loopback and link-local addresses it was written to refuse. Both are refused + now. The filter also refuses the non-global ranges that are neither RFC 1918 + nor obviously local — IPv4 multicast, `240.0.0.0/4` (nominally reserved, and + used for real internal addressing inside more than one cloud provider), the + RFC 2544 benchmark block, `192.0.0.0/24` and the TEST-NETs; IPv6 multicast + (`ff02::1` is all-nodes on the local link), site-local, and `2001:db8::/32`. + The bare-label rule moved into that shared filter as well: `metadata` is + the metadata endpoint's short name on both AWS and GCP, DNS is what makes it + an address, and the webhook check had the rule while the provider allowlist + did not. +- **A policy declaring two `llm` entries is refused.** The BYOK provider was + resolved from the first entry while an evaluator was built for every entry, so + `[{llm}, {llm, provider: …}]` silently ran twice on the operator's Workers AI + bill — the fail-open this work exists to prevent. Both the parser and the + provider resolution now refuse rather than choosing one. +- The shared outbound-host filter now rejects the non-global IPv6 ranges it was + still accepting: discard-only (`100::/64`), IETF protocol assignments + (`2001::/23`, which covers benchmarking and Teredo), 6to4 (`2002::/16`) and + the NAT64 prefixes (`64:ff9b::/96`, `64:ff9b:1::/48`). The IPv4 half already + refused its equivalents, so webhook delivery and BYOK provider URLs were + applying two different rules through one filter. diff --git a/docs/CURRENT_CAPABILITIES.md b/docs/CURRENT_CAPABILITIES.md index 6edd2274..4291b895 100644 --- a/docs/CURRENT_CAPABILITIES.md +++ b/docs/CURRENT_CAPABILITIES.md @@ -8,15 +8,16 @@ the Git LFS and submodule limitations recorded below. > **Read the caveats, not just the bullets.** Two capabilities below exist in > code but cannot be used on a stock instance, because the Cloudflare Sandboxes > binding they need is a gated beta and is **commented out in `wrangler.toml`** -> (`# [[sandboxes]]`, `wrangler.toml:156` — the only occurrence in the file, and +> (`# [[sandboxes]]`, `wrangler.toml:170` — the only occurrence in the file, and > named environments do not inherit top-level bindings). See > [Sandbox-dependent features](#sandbox-dependent-features-unavailable-by-default). ## Core platform - Cloudflare Worker (Hono) on Artifacts, KV, D1, Queues, R2 (backups), Workers - AI, Analytics Engine, Email, and three Durable Objects — `MergeQueue`, - `RepoDO` and `MagicLinkRateLimiter` (`src/index.ts:52`, `wrangler.toml:78`). + AI, Analytics Engine, Email, and four Durable Objects — `MergeQueue`, + `RepoDO`, `MagicLinkRateLimiter` and `UsageMeter` (`src/index.ts:58-61`, + `wrangler.toml:87-105`). - Project create/import (GitHub/GitLab/Bitbucket), workspace fork/commit/delete, change creation with synchronous evaluation, evaluation-gated merge, provenance (records the agent, the model and prompt hash snapshotted at change creation, and @@ -28,10 +29,15 @@ the Git LFS and submodule limitations recorded below. ## Evaluation & merge pipeline -- Evaluators: secret scan (always on, blocking), diff, webhook, LLM (AI binding), - sandbox (Sandboxes binding — **unavailable by default**, see below). Per-change - evaluator evidence and estimated resource costs (LLM tokens, sandbox time, git - ops). +- Evaluators: secret scan (always on, blocking), diff, webhook, LLM, sandbox + (Sandboxes binding — **unavailable by default**, see below; fails closed when + absent). The LLM evaluator runs on the Workers AI binding by default, or on the + project's own credential against an operator-configured provider (`anthropic` + or an OpenAI-compatible endpoint) — see BYOK below. Per-change evaluator + evidence and resource costs (LLM tokens, sandbox time, git ops); token counts + are the ones the provider reported, and fall back to a `~4 chars/token` + estimate — marked `estimated` on the cost record — only for a response that + omits them. - Branch protection in `.stratum/policy.yaml` (`merge:`): required evaluators (latest run per type), required human approvals, force-merge control (**deny-by-default** — force is only allowed when the policy sets @@ -64,28 +70,57 @@ the Git LFS and submodule limitations recorded below. ### Sandbox-dependent features (unavailable by default) `[[sandboxes]]` is a gated Cloudflare beta and is commented out in -`wrangler.toml` (`wrangler.toml:155-157`; it appears nowhere else, and named +`wrangler.toml` (`wrangler.toml:169-171`; it appears nowhere else, and named environments do not inherit top-level bindings). Unless an operator has Sandboxes access and uncomments it, `env.SANDBOX` is absent on **every** environment, including the maintainer's hosted instance, with two consequences: - **The `sandbox` evaluator is not skipped — it fails closed.** `buildEvaluators` substitutes an `UnavailableEvaluator` that returns score 0 - / failed (`src/services/change-flow.ts:137-152`), so naming `sandbox` in + / failed (`src/services/change-flow.ts:242-256`), so naming `sandbox` in `merge.requiredEvaluators` **blocks every merge in that project** until the binding is enabled or the evaluator is removed from the policy. - **`merge.postMergeCommand` is skipped with a warning**: `post-merge.ts` logs `Post-merge command configured but SANDBOX binding is absent` and returns `{ status: "skipped", reason: "Sandbox binding is not configured" }` - (`src/merge/post-merge.ts:39-43`). Nothing runs and nothing is reverted, but + (`src/merge/post-merge.ts:39-44`). Nothing runs and nothing is reverted, but the skip is reported — it is in the logs and in the merge's post-merge status, not swallowed. This repo's own `.stratum/policy.yaml` uses only `diff` and `llm` and sets no `postMergeCommand`, so it avoids both. The `llm` evaluator has the same fail-closed shape when the AI binding is absent -(`src/services/change-flow.ts:129-136`), but `[ai]` **is** bound in every -environment (`wrangler.toml:75`), so it works out of the box. +(`src/services/change-flow.ts:235-241`), but `[ai]` **is** bound in every +environment (`wrangler.toml:84-85`), so it works out of the box. + +## Usage metering, entitlements, and BYOK + +- Every cost record names who pays for it (`owner_id`, `owner_type`) and whether + it was `platform` or `byok` spend, and rolls up into `usage_periods`, an + owner-scoped monthly aggregate keyed `(owner_id, period, meter, source)`. + Metering is **always on**, including self-hosted: it is a ledger, not a + paywall. +- BYOK for the `llm` evaluator: an operator allowlist (`LLM_PROVIDERS`) of named + providers (`anthropic`, `openai-compatible`), selected by name from + `.stratum/policy.yaml`, with the project's own credential read from + `project_secrets`. A policy can never supply a `baseUrl`. Every failure path + fails the gate closed and none falls back to the operator's `AI` binding. + See `adr/008-llm-provider-byok-threat-model.md`. +- A `UsageMeter` Durable Object holds the monthly reserve/settle counters and a + bucketed sliding `evaluations_per_hour` window (a rate ceiling BYOK does not + lift). +- An entitlements seam (`BILLING_SERVICE_URL` + `BILLING_SERVICE_SECRET`) that + is **inert when unset** — every allowance reads as unlimited, no meter binding + is touched, and nothing can be refused. Plan definitions and payment live + outside this repository. Enforcement is additionally observe-only unless + `ENTITLEMENTS_ENFORCE=1`: a decision is evaluated and recorded, and admits. +- Limits are checked against the **acting user** (an agent resolves to its + owner), not the project's owner, so an allowance follows the person; recording + still names the true owner, so the ledger is unaffected. +- Visibility: `/settings/usage`, an 80%-of-a-meter banner and one email per + crossing, `stratum_get_usage` over MCP, and `GET /api/users/me/usage`. The + billing surface is read-only everywhere — no tool or endpoint can raise a + limit, buy capacity, or set a provider key. ## Events & integrations @@ -132,8 +167,9 @@ environment (`wrangler.toml:75`), so it works out of the box. syntax-highlighted file viewer (dependency-free lexer), commit log, changes with diff viewer + evaluator evidence + costs + reviews + comments, issues, activity, webhooks management, deployments (history, log tail for writers, Approve and - Retry buttons), deploy-secret management in project settings, settings. Open - changes poll via meta refresh. + Retry buttons), deploy-secret management in project settings, settings, and a + per-account usage page (`/settings/usage`). Open changes poll via meta + refresh. ## Tooling @@ -163,7 +199,11 @@ environment (`wrangler.toml:75`), so it works out of the box. - Deployments have no build step, no preview environment and no rollback, and the deploy DLQ has no consumer. - Phase 4 operational items remain: load testing at 1000+ concurrent workspaces, - D1 hot/cold rotation, SSO/SAML, multi-tenancy/billing for Stratum Cloud. + D1 hot/cold rotation, SSO/SAML, and the rest of multi-tenancy/billing for + Stratum Cloud — the metering, entitlement and enforcement machinery above is + in the tree, but plan definitions, checkout and subscription state are not, + and there is no org billing UI, no seat model, and no retention or storage + limit. - Durability is covered: D1 and KV identity back up to R2 daily and on demand, along with the reachable history of a rotating slice of repos (coverage rotates across runs under a per-run cap), with a tested restore path diff --git a/docs/README.md b/docs/README.md index f21e159d..54098b51 100644 --- a/docs/README.md +++ b/docs/README.md @@ -52,10 +52,12 @@ docs/ │ ├── 004-high-frequency-agent-commits.md │ ├── 005-git-smart-http-proxy.md │ ├── 006-ssh-transport.md -│ └── 007-sandbox-evaluator-threat-model.md +│ ├── 007-sandbox-evaluator-threat-model.md +│ └── 008-llm-provider-byok-threat-model.md ├── research/ # Research notes │ ├── github-alternatives-pain-points.md │ ├── master-plan-alignment.md +│ ├── monetization-and-byok.md │ └── option-b-warm-repo-do-spike.md ├── runbooks/ # Operational runbooks │ ├── artifacts-scaling.md @@ -129,6 +131,7 @@ docs/ - [ADR 005: Native `git push` via a Smart-HTTP Proxy](adr/005-git-smart-http-proxy.md) - [ADR 006: SSH Transport for Git](adr/006-ssh-transport.md) - [ADR 007: Sandbox Evaluator Threat Model and Time Budget](adr/007-sandbox-evaluator-threat-model.md) +- [ADR 008: LLM Provider Seam and BYOK Threat Model](adr/008-llm-provider-byok-threat-model.md) **Historical Reference:** - [Archived Documents](archive/README.md) - Code reviews, audits, etc. @@ -167,7 +170,7 @@ short stub that needs to be fleshed out. | Developer - Database | 🚧 Outline | High | 2026-08-25 | | Developer - Queues | 🚧 Outline | Medium | 2026-08-25 | | Developer - Testing | 🚧 Outline | Medium | 2026-08-25 | -| ADRs (001-007) | ✅ Complete | Low | 2026-08-31 | +| ADRs (001-008) | ✅ Complete | Low | 2026-09-06 | **Legend:** ✅ Complete | 🚧 Outline / In Progress | 📋 Planned diff --git a/docs/REMAINING_WORK.md b/docs/REMAINING_WORK.md index bf47d975..a61db5e1 100644 --- a/docs/REMAINING_WORK.md +++ b/docs/REMAINING_WORK.md @@ -39,9 +39,62 @@ OAuth options. Required for most paid team adoption. ### Multi-tenancy and billing -Tenant isolation, usage metering, and billing integration for Stratum Cloud. -Per-change cost tracking (LLM tokens, sandbox time, git ops) already exists -and provides the metering foundation. +**Delivered.** Cost records name their payer (`owner_id`, `owner_type`) and +their source (`platform` or `byok`) and roll up into `usage_periods`, an +owner-scoped monthly aggregate. A `UsageMeter` Durable Object holds the monthly +reserve/settle counters and a bucketed sliding `evaluations_per_hour` window. An +entitlements seam fetches a subject's allowances from a billing service and is +inert with `BILLING_SERVICE_URL` unset; enforcement consults it at the LLM +evaluator, the deploy consumer, the request limiter and private-project +creation, observe-only unless `ENTITLEMENTS_ENFORCE=1`. Limits are checked +against the acting user rather than the project's owner, so an allowance follows +the person — except for an org the billing service positively reports as +pooling, which is the subject for its own projects. `.stratum/policy.yaml` can select an operator-configured LLM +provider and run the merge gate on the project's own key. Usage is visible at +`/settings/usage`, in an 80% banner and email, over MCP (`stratum_get_usage`) +and at `GET /api/users/me/usage`. + +**Still open.** + +- **The paid half lives outside this repository**, by design: plan definitions, + Stripe checkout, subscription webhooks and invoicing sit behind + `BILLING_SERVICE_URL`. Nothing in the tree knows what a plan costs, and the + entitlements payload is the whole contract. +- **No org billing.** There is no org UI at all, so there is no seat model, no + per-seat pricing, and no way for an org to see or manage its own usage: an + org's threshold notice reaches the person who ran the evaluation, by email, + because there is no org page to show it on. +- **Hard tenant isolation is still the access model, not an infrastructure + boundary.** Every tenant shares one D1 database, one KV namespace and one + Worker; separation is enforced by authorization checks and by scoping every + query, which is what this item originally meant by "tenant isolation". +- **No retention or storage limits.** The metered flows are LLM tokens, sandbox + milliseconds and deploys, plus a private-project gauge. Repository storage, + event/audit retention and artifact size are unmetered and unbounded. +- **Org creation is still subject to no per-user quota.** Charging the actor closed the allowance + *reset* (a new org no longer means a new allowance), but `POST /api/orgs` + itself has no per-user cap, and the multi-account variant of the same trick is + accepted rather than solved. The escalation ladder — rate-limit org creation, + require OAuth beyond the first org, a card to raise limits — is designed and + unbuilt. +- **Enforcement has never bound anything in production.** It ships observe-only + so a month of recorded decisions can be read before any of them blocks a merge. +- **Entitlements fail open**, so a billing outage permits unmetered usage for up + to one cache TTL, reconcilable from the ledger afterwards; an upgrade likewise + takes effect up to one TTL late. +- **BYOK couples the merge gate to `DEPLOY_SECRET_KEY`.** Rotating that key + already makes every stored deploy secret undecryptable, and now blocks the + gate on every BYOK project until each re-enters its provider key. +- **The `webhook` evaluator's `secret` is a credential in a committed file.** + Unlike `deploys:`, which takes secret *names* and resolves the values from + D1, `evaluators: [{ type: webhook, secret: ... }]` takes the HMAC key + literally, so it lives in `.stratum/policy.yaml` — in git, in every clone, and + in any policy export. `sanitizePolicy` strips it before the policy reaches a + model or a webhook body, which is the codebase already treating it as a + credential, and the storage half is simply missing. Closing it means the + `deploys:` shape — a name resolved from `project_secrets` — plus a migration + path for projects using the literal form today. Predates the metering work and + was left alone by it. ### Monitoring dashboard UI diff --git a/docs/adr/008-llm-provider-byok-threat-model.md b/docs/adr/008-llm-provider-byok-threat-model.md new file mode 100644 index 00000000..fb3d959a --- /dev/null +++ b/docs/adr/008-llm-provider-byok-threat-model.md @@ -0,0 +1,293 @@ +# ADR 008: LLM Provider Seam and BYOK Threat Model + +## Status + +Accepted. Records the threat model for the `llm` evaluator's provider seam and +for bring-your-own-key (BYOK), shipped alongside usage metering and +entitlements. + +It lives here rather than in the design document it came from, for the reason +ADR 007 exists: a threat model buried in a product document dies with that +document, and the guards below only stay correct while the reasoning behind them +is readable by whoever next edits `policy-loader.ts` or `llm-providers.ts`. + +## Context + +Until this change the `llm` evaluator had exactly one place to send a diff: the +instance's Workers AI binding. The endpoint was not configurable, so no part of +`.stratum/policy.yaml` could influence which host the Worker talked to. + +Two things changed that. + +**A provider seam.** `LlmProvider` now has three implementations — +`WorkersAiProvider` (the default, byte-identical to the previous behavior), +`AnthropicProvider`, and `OpenAiCompatibleProvider`. The last two speak HTTP to +a URL. + +**BYOK.** A project may run the merge gate on its own credential instead of the +operator's account, by naming a provider in its policy file and storing a key in +`project_secrets`. That is a feature the operator's own cost model wants — it is +how a project that needs a better reviewer model stops charging the operator for +it — and it is also the point at which repository content starts influencing an +outbound request. + +Everything below follows from that last sentence. **The policy file is +repository content**: it is authored in a workspace, by a human or by an agent, +and it is exactly the kind of input this platform exists to gate. + +### Who the attacker actually is + +Getting this wrong produces guards against the wrong thing, so it is worth +stating precisely. + +An earlier draft of this model said "anyone who can open a change can edit the +policy". Two facts in the tree contradict it: + +1. **Policy loads from the project's DEFAULT BRANCH.** `runChangeEvaluation` + calls `loadPolicy(project.remote, …, projectDefaultBranch(project), …)` + (`src/services/change-flow.ts`), not from the workspace under evaluation. A + proposed edit to `.stratum/policy.yaml` therefore **never governs its own + evaluation**. It takes effect only after it merges, behind whatever approvals + the current policy requires, and `.stratum/policy.yaml` is in + `PROTECTED_CONFIG_FILES`, so a change touching it is flagged + (`touches_protected_config`) and cannot be force-merged. +2. **Opening a change requires `canWriteProject`** (`src/routes/changes.ts`). + The actor is already a collaborator trusted with write access, not an + anonymous passer-by. + +So the reachable threats are: + +- **(a) A self-serve project OWNER**, on a multi-tenant instance, using the + operator's Worker as an SSRF or exfiltration proxy against a host of their + choosing. They need no privilege escalation at all — they own the project and + can merge whatever they like into their own default branch. On the hosted + instance, signing up is the entire cost of becoming this attacker. +- **(b) A compromised or careless writer** whose policy edit lands at merge — + the ordinary insider case, bounded by review rather than by the loader. + +Both are bounded by the same control, but a junior reader who believes the +threat is "the change being evaluated right now" will build guards in the wrong +place and leave the real one open. + +**The writer-only assumption is load-bearing, and it expires.** Everything above +rests on `canWriteProject` gating change creation. The day Stratum supports open +contribution — a fork-and-propose flow from someone with no write access — an +untrusted party can propose a policy file, and the second control below (a +policy may not name an endpoint) stops being defense in depth and becomes the +only thing standing between an anonymous submission and the operator's Worker. +Flagged here so it is a known dependency and not a surprise. + +## Decision + +### 1. A policy may SELECT a provider by name. It may never supply a URL. + +The endpoint set is closed and operator-owned. `LLM_PROVIDERS` is an environment +variable holding a list of `{ name, kind, baseUrl }` entries; +`.stratum/policy.yaml` carries `provider: ` and nothing else. `baseUrl` is +not read from the policy file at any level of validation — it is not a +restricted field, it is not a field at all. + +This is the control the whole model rests on. A `baseUrl` chosen by repository +content turns the Worker into a request-forgery primitive aimed at whatever host +the policy names, and the request it would make carries the diff and the policy +in its body. No amount of URL validation makes "the attacker picks the +destination" safe, because a legitimate destination is indistinguishable from an +attacker-controlled one that answers the same protocol. + +An unrecognized provider name is a **rejection, never a fallback**. See control +4 for why it blocks rather than degrades. + +### 2. Every URL rule is checked once, at configuration parse time + +`llmProviderCatalog` validates each `LLM_PROVIDERS` entry when it is parsed, not +per request: `https` only, no embedded credentials, no query string or fragment, +a name matching `^[a-z][a-z0-9-]{0,31}$`, at most 16 entries, and a host that +passes the private-address filter. The stored `baseUrl` is **normalized** to +origin-plus-path, because `https://host#` and `https://host?` both pass an +emptiness check on `hash`/`search` while burying the provider's appended +`/messages` path inside a fragment or a query. + +Parse-time is deliberate. Re-validating an immutable value on the hot path costs +something and buys nothing, and worse, a per-request check means a bad entry is +discovered by the first project unlucky enough to select it rather than by the +operator who wrote it. + +### 3. One private-host filter, shared with the webhook evaluator + +Host filtering is `privateHostReason` in `src/utils/validation.ts`, and both the +provider allowlist and `validateWebhookUrl` call it. It rejects loopback, +link-local, RFC1918, CGNAT (`100.64/10`), the `.internal` and `.local` suffixes +(GCP's metadata endpoint is `metadata.google.internal`), and IPv6 equivalents, +classifying bracketed literals by mask rather than by their spelling. + +This began as two implementations and they drifted, which is what a duplicated +security filter always does. The provider copy missed CGNAT, `.internal`, +everything in `fe80::/10` not literally spelled `fe80`, and the IPv4-compatible +IPv6 form `[::127.0.0.1]` — which the URL parser hands on as `[::7f00:1]`. The +webhook copy had the last two as well, so unifying them **closed a pre-existing +SSRF gap in the webhook evaluator**, which is worth recording: the fix belongs +to a path nobody set out to change. + +### 4. `sanitizeLlmConfig` is a whitelist, and it fails the file closed + +`sanitizeLlmConfig` (`src/evaluation/policy-loader.ts`) replaced a copy-through. +It keeps four keys — `provider`, `model`, `threshold`, `maxDiffChars` — and +resolves `provider` against the operator's catalog at parse time. Anything else +in the entry, `baseUrl` included, is dropped with a warning. A policy may +declare **at most one** `llm` entry, and `model` is **required** whenever +`provider` is set. + +Rejecting an entry fails the **whole policy file** closed and blocks merges. +That is the opposite of what a rejected `deploys:` entry does two files away +(`src/deploy/config.ts`, whose comment says "do not fix this"), and the asymmetry +is deliberate: a deploy runs *after* the merge, so a bad entry costs a failed +deployment row, whereas an `llm` entry is a **gate**, and a gate that cannot run +must not pass. An unresolvable provider name is precisely a gate that cannot +run. + +Two of these rules are fail-open closures rather than tidiness: + +- **One `llm` entry.** `LLMEvaluator` read the *first* `llm` entry while + `buildEvaluators` constructed an evaluator for *every* one, so + `[{llm}, {llm, provider: …}]` ran twice on the operator's Workers AI account + with no error and no log — a policy that asked to run on its own key silently + running on the operator's. +- **`model` required with `provider`.** The default model id is a Workers AI + slug, and posting it to Anthropic or an OpenAI-compatible endpoint fails every + call. Inventing a per-kind default would silently choose a model — and a price + — on the project's behalf. + +### 5. Credential handling fails closed, with no `env.AI` fallback + +Credentials live in `project_secrets` (AES-GCM, `(project_id, name)` as AAD, no +route-reachable read path). The secret name is **derived** from the provider +name (`anthropic` → `ANTHROPIC_API_KEY`) rather than configured, so an +`LLM_PROVIDERS` entry cannot point at a credential the project stored for +something else. + +Every failure — `DEPLOY_SECRET_KEY` unset, the secret missing, the row +undecryptable, the provider name unresolvable, a redirect — fails the gate with +its own distinct reason. **None of them falls back to the Workers AI binding.** +Falling back would put the spend back on the operator's bill, which is the hole +BYOK closes, and would be a gate that silently stops gating. + +The credential load is skipped entirely when the policy names no provider, which +is the overwhelmingly common case and matters because the load is a D1 read plus +a 100k-iteration PBKDF2 derivation on the change-creation path. + +### 6. Redirects are refused + +The provider fetch uses `redirect: "manual"` and treats any 3xx as a failed +evaluation. The allowlist binds the host a request is **sent** to; a redirect +from an allowlisted host moves the prompt — the diff and the policy — to a host +nobody validated, and on 307/308 the body goes with it. `x-api-key` is not a +header the Fetch spec strips cross-origin, so the project's own credential +follows. The webhook evaluator in the same directory already refused to follow +redirects, for exactly this reason. + +### 7. Prompt sanitization is recursive over the whole policy + +The serialized policy is part of the prompt (`llm-evaluator.ts`), so anything in +`.stratum/policy.yaml` is sent to the provider verbatim. `sanitizePolicy` used +to strip `webhook.secret` alone, then one level inside each evaluator entry — +while `parsePolicyContent` spreads unknown root keys onto the policy. A +credential written at the top of the file, or nested inside an entry of an +unmodelled type, reached both the review model and the body POSTed to a +policy-supplied webhook URL, which is an attacker-chosen host. + +It now strips recursively over the whole policy, with depth and size bounds, on +field **names** (`secret`, `token`, `auth`, `apiKey`, `hmac`, `sig`, `pat`, …, +with plurals stemmed) rather than on values — there is no reliable way to +recognise a secret by its bytes. `keystone` and `monkeys` survive; `tokens` and +`apiKeys` do not. + +Two boundaries on this, stated so they are not mistaken for guarantees: + +- **`redactSecrets` is the backstop, not the control.** Its own doc comment says + literal-substring redaction is defeated by a base64 or JSON-escaped echo. The + actual control is structural and already in the file: **provider output never + reaches `EvalResult.reason`.** +- **An oversize policy fails the gate closed.** `policyContext` is capped at + `MAX_POLICY_CONTEXT_CHARS` (8,000), so adding a provider block to an already + large policy can block merges on a project that worked yesterday. That is the + safe direction, and it is a real operational edge, not a theoretical one. + +### 8. The provider block must live in a protected config file + +Extending `PROTECTED_CONFIG_FILES` was considered and cut: the check is +whole-file, and `.stratum/policy.yaml` is already covered in full, provider +block included. The requirement is the inverse and it is a constraint on future +work — **the provider block must never move to a third file the protected-config +flag does not cover.** + +Its actual shape is worth repeating from ADR 007, because this ADR leans on it: +`diffTouchesProtectedConfig` is a substring match for +`diff --git a/.stratum/policy.yaml b/.stratum/policy.yaml`. A rename, non-default +diff prefixes, or a path requiring git quoting would defeat it. + +## What is explicitly NOT defended + +- **DNS rebinding.** The host in `baseUrl` is resolved by the runtime at request + time, after the allowlist validated the *name*. A name that passes the private + address filter and then resolves to a private address defeats the filter. There + is no resolve-then-pin primitive available to a Worker's `fetch`, so the filter + is a check on the configured name, not on the packet's destination. It is + mitigated in practice by the endpoint set being operator-configured rather than + attacker-supplied — which is control 1 doing the work again. +- **A malicious operator.** Everything here protects the operator from the + projects on their instance, not the projects from their operator. An operator + configures the endpoint list, holds `DEPLOY_SECRET_KEY`, and runs the Worker + that decrypts and uses a project's provider credential. A project trusting an + instance with a key is trusting whoever runs it; nothing in this design changes + that, and pretending otherwise would be worse than saying it. +- **The provider itself.** The diff and the policy are sent to the endpoint the + operator configured. Its logging, retention and jurisdiction are the operator's + choice and the project's risk to accept. +- **A project owner spending their own credit.** Any project *writer* can drain + the owner's provider credit by opening changes. It is bounded by + `evaluations_per_hour`, deliberately not lifted by BYOK, and by nothing else; + BYOK adds no approval step in front of the spend. Documented at the point of + configuration (`docs/user-guide/getting-started.md`), where a reader who assumes + "my own key means no limits" would otherwise file the ceiling as a bug. + +## Residual risks accepted + +- **`LLM_PROVIDERS` is trusted input.** It is operator configuration and is + validated, but a misconfigured entry — a proxy that fans out, an endpoint that + logs prompts — is not something the parser can detect. Staging exists to + exercise a new entry before production, where a bad one shows up only as blocked + merges on the projects selecting it. +- **BYOK couples the merge gate to `DEPLOY_SECRET_KEY` rotation.** Rotating that + key already makes every stored deploy secret undecryptable + (`wrangler.toml`); it now also blocks the gate on every BYOK project until each + re-enters its provider key. This widens the blast radius of a documented + operation rather than introducing an undocumented one. +- **Token accounting is per response, not per provider.** An OpenAI-compatible + endpoint that omits `usage` puts that response back on the `~4 chars/token` + estimate, marked `estimated` on the cost record. A provider that reports + *wrong* counts is believed. +- **There is no MCP surface for writing a provider key, and there must not be.** + `stratum_set_provider_key` looks like an obvious gap to anyone extending the + tool list; it is ruled out because a credential must never pass through a + model's context window, and this model rests on credentials not being + selectable by the wrong party. The write surface is the web UI or the CLI — + the same line that keeps agent tokens from submitting review verdicts. + +## Alternatives considered + +**Allow `baseUrl` in the policy, validated per request.** Rejected. It is the +threat, not a configuration convenience; see control 1. Every validation scheme +proposed for it reduces to "the attacker picks the destination and we check that +the destination looks normal". + +**Make the secret name configurable per provider entry.** Rejected: it would let +one provider entry read a credential a project stored for an unrelated purpose. +Derivation from the provider name makes the mapping total and unguessable-free. + +**Fall back to Workers AI when BYOK cannot run.** Rejected as the defining +fail-open of this feature. It restores the operator's bill, and it converts a +gate that cannot run into a gate that passes. + +**Treat an unresolvable provider as a dropped evaluator rather than a policy +error.** Rejected for the reason in control 4: a merge gate that quietly stops +existing is the top risk of the whole change. diff --git a/docs/api/endpoints/users.md b/docs/api/endpoints/users.md index b096f7bb..96caae4c 100644 --- a/docs/api/endpoints/users.md +++ b/docs/api/endpoints/users.md @@ -8,6 +8,26 @@ credential — in [Authentication](../authentication.md). Returns the authenticated user's profile. +## Get Current Usage + +`GET /api/users/me/usage` + +The caller's metered usage against their plan for the current billing period: +each meter's consumption and limit, the rate ceilings, the period, and when it +resets. A limit of `-1` is unlimited and `0` is a plan that forbids that meter +outright, so a client renders words rather than computing a percentage against +either. + +Read-only, and deliberately the **only** billing surface a token can reach: +there is no endpoint here to raise a limit, buy capacity, or attach a payment +method. An **agent token** reports its owner's allowance, because that is the +account its evaluations are charged to. + +Figures cover platform-billed usage only. Spend on a project's own provider key +(BYOK) is reported separately and never counted against an allowance. On an +instance with no billing service configured — every self-hosted deployment — +`metered` is `false` and every limit is unlimited. + ## API tokens Listing, creating, revoking, and disabling the legacy token accept the diff --git a/docs/api/openapi.yml b/docs/api/openapi.yml index 4ec88a79..4d726760 100644 --- a/docs/api/openapi.yml +++ b/docs/api/openapi.yml @@ -3005,6 +3005,105 @@ paths: "500": $ref: "#/components/responses/InternalError" + /api/users/me/usage: + get: + operationId: getMyUsage + tags: [users] + summary: Get the caller's metered usage for the current period + description: | + Read-only, and deliberately the only billing surface a token can reach: + nothing here raises a limit, buys capacity, or attaches a payment method. + + An **agent token** reports its OWNER's allowance, because that is the + account its evaluations are charged to. + + `used` counts platform-billed consumption only. Spend on a project's own + provider key is reported separately in `byok` and never counted against + an allowance. On an instance with no billing service configured — every + self-hosted deployment — `metered` is `false` and every limit is `-1`. + + A limit of `-1` is unlimited and `0` is a plan that forbids the meter + outright, so render words rather than computing a percentage against + either; `percentUsed` and `remaining` are `null` in both cases. + responses: + "200": + description: Usage for the current billing period + content: + application/json: + schema: + type: object + required: + [subject, usedSource, plan, metered, period, resetsAt, meters, rates] + properties: + subject: + type: object + description: Whose allowance this is — the enforcement subject. + required: [ownerId, ownerType] + properties: + ownerId: { type: string } + ownerType: { type: string, enum: [user, org] } + usedSource: + type: string + enum: [meter, ledger] + description: | + `meter` is the live enforcement counter a limit is compared + against. `ledger` is the recorded aggregate alone — every + self-hoster, and any instance whose meter binding is + missing or unreadable — which omits work done in an org + namespace, where the org is the recorded owner. + plan: + type: string + description: Opaque plan name; `unlimited` when there is none. + metered: + type: boolean + description: Whether any allowance is in force at all. + period: { type: string, description: "'YYYY-MM' UTC." } + resetsAt: + type: string + format: date-time + description: When the monthly meters roll over. + meters: + type: array + items: + type: object + required: + [meter, used, byok, limit, unlimited, blocked, percentUsed, remaining] + properties: + meter: + type: string + enum: [llm_tokens_month, sandbox_ms_month, deploys_month] + used: + type: integer + description: Platform-billed consumption — the only kind a limit binds. + byok: + type: integer + description: Spend on a project's own provider key; never counted against the limit. + limit: { type: integer, description: "`-1` unlimited, `0` a hard block." } + unlimited: { type: boolean } + blocked: { type: boolean } + percentUsed: + type: number + nullable: true + description: Null when there is no finite limit. + remaining: + type: integer + nullable: true + description: Floored at zero; null when unlimited. + rates: + type: array + description: Burst ceilings. Reported as a limit only. + items: + type: object + required: [rate, limit, unlimited] + properties: + rate: { type: string, enum: [evaluations_per_hour] } + limit: { type: integer } + unlimited: { type: boolean } + "401": + $ref: "#/components/responses/Unauthorized" + "500": + $ref: "#/components/responses/InternalError" + /api/users/me/rotate-token: post: operationId: rotateToken diff --git a/docs/developer/architecture.md b/docs/developer/architecture.md index 45fa6708..beb3b422 100644 --- a/docs/developer/architecture.md +++ b/docs/developer/architecture.md @@ -501,6 +501,40 @@ export class MergeQueue extends DurableObject { - Auto-rebase for clean merges - Batch merging for non-conflicting changes +## Usage Meter + +Durable Object holding one billing subject's counters: the monthly meters +(`llm_tokens_month`, `sandbox_ms_month`, `deploys_month`) under a single record +keyed by period, and the rolling rate windows (`evaluations_per_hour`) under a +second one. Two records, because their lifetimes are unrelated — an hourly +window straddling midnight on the 1st must not be reset by the month rolling. + +`reserve` takes capacity before the work runs and `settle` reconciles it against +what was actually spent, so N concurrent evaluations cannot each read the same +pre-spend total. A `reserve` with no `settle` is exactly a rate check, which is +what the hourly window uses. + +It is a Durable Object rather than KV for the reason `MagicLinkRateLimiter` +gives, plus one more. KV reads come from a per-colo edge cache about a minute +stale, which the request limiter tolerates only because its bucket also rolls +every minute — window and staleness are the same size. Stretch the window to an +hour and an N/hour limit admits roughly N x colos x 60 to a distributed caller. +The hourly window is summed over 5-minute buckets rather than held in one hourly +bucket, so a fresh hour cannot admit the whole allowance in its first second; +that bounds the burst at up to 2x the limit across a boundary rather than +eliminating it. + +The object is subject-keyed on the **enforcement** subject (the acting user, or +an org positively known to pool on a paid plan), which is not necessarily the +subject a cost record names. It is never touched at all when +`BILLING_SERVICE_URL` is unset. + +Deleting a **project** deliberately does *not* purge it, unlike `RepoDO` and +`MergeQueue`: the meter is keyed on a subject rather than a project, and purging +it on project deletion would refund the month by deleting a project — the same +refund keeping `usage_periods` out of the project-scoped tables prevents. The +**account** cascade erases it, and nothing else does. + ## Queue Processing Background jobs processed by Cloudflare Queues. @@ -596,6 +630,12 @@ src/ │ └── run-backup.ts # Backup orchestration ├── beta/ │ └── gate.ts # Beta access gating +├── billing/ +│ ├── enforcement.ts # Consulting a limit: the one place it decides +│ ├── entitlements.ts # What a plan allows; inert without a billing service +│ ├── usage-banner.ts # The 80% banner's stored receipt +│ ├── usage-notifications.ts # Edge-triggered threshold email +│ └── usage-report.ts # One account's usage, for the page and MCP ├── email/ │ └── templates.ts # Email templates ├── evaluation/ @@ -617,7 +657,7 @@ src/ │ ├── dispatch.ts # Runs tool calls against the real routers, in-process │ ├── protocol.ts # JSON-RPC 2.0 / MCP message layer │ ├── schema.ts # Tool arg schemas: JSON Schema + validator -│ └── tools.ts # The eighteen tools +│ └── tools.ts # The nineteen tools ├── merge/ │ ├── post-merge.ts # Post-merge actions │ └── protection.ts # Merge protection rules @@ -641,6 +681,7 @@ src/ │ ├── merge-queue.ts # Merge queue durable object │ ├── repo-do.ts # Repository durable object │ ├── ttl-sweep.ts # TTL cleanup +│ ├── usage-meter.ts # Usage meter durable object │ └── webhook-delivery.ts # Outbound webhook delivery ├── routes/ │ ├── agents.ts # Agent management @@ -707,6 +748,7 @@ src/ │ ├── state.ts # KV state management │ ├── sync.ts # Sync status tracking │ ├── teams.ts +│ ├── usage.ts # usage_periods, the owner-scoped monthly aggregate │ ├── users.ts │ └── webhooks.ts ├── templates/ diff --git a/docs/developer/database.md b/docs/developer/database.md index bfdc898a..6e9bad70 100644 --- a/docs/developer/database.md +++ b/docs/developer/database.md @@ -20,6 +20,19 @@ ### api_tokens - id, user_id, name, token_hash, token_prefix, scope, expires_at, revoked_at +### usage_periods + +- owner_id, owner_type, period ('YYYY-MM' UTC), meter, source, quantity +- PK is `(owner_id, period, meter, source)`. `source` is in the key on purpose: + `platform` and `byok` spend accumulate as separate rows, and an enforcement + read sums the `platform` rows alone — a sum that omitted the filter would + charge a project's own provider spend against its hosted allowance. +- Subject-keyed rather than project-keyed, so it survives the project-deletion + cascade that hard-deletes `cost_records`; otherwise a month's usage would be + refundable by deleting the project that incurred it. It joins the *account* + deletion cascades instead, and is backed up in its own right because it is + not recoverable by replaying `cost_records`. + ### oauth_clients - id (the client_id), client_secret_hash, client_name, redirect_uris, scope, token_endpoint_auth_method diff --git a/docs/research/monetization-and-byok.md b/docs/research/monetization-and-byok.md new file mode 100644 index 00000000..00e0b94c --- /dev/null +++ b/docs/research/monetization-and-byok.md @@ -0,0 +1,408 @@ +# Monetization and BYOK + +**Status:** research / proposal, now **partly superseded by what shipped**. The +metering, entitlement, enforcement and BYOK machinery described below is in the +tree — see `docs/CURRENT_CAPABILITIES.md` for what exists, +`docs/adr/008-llm-provider-byok-threat-model.md` for the threat model as +implemented, and `docs/REMAINING_WORK.md` for what is still open. Where this +document and those disagree, they are right and this is the earlier thinking. +Pricing, plan definitions and payment remain unimplemented, and a self-hoster's +behavior is still unchanged: every allowance is unlimited unless a billing +service is configured. + +This document scopes how Stratum charges for hosted usage without withholding +code from the open-source repository, what the free tier costs the operator, and +how a project supplies its own model credentials (BYOK) instead of consuming a +hosted allowance. + +It exists because [`ROADMAP.md`](../../ROADMAP.md) carries **Multi-tenancy and +billing** as an open operational item with "per-change cost tracking already +exists and provides the metering foundation", and that foundation needs a design +before it grows a payment integration around it. + +--- + +## 1. The constraint the license already set + +[`LICENSING.md`](../../LICENSING.md) settled two things that decide most of this +design, and re-litigating them here would be wasted effort. + +The server is **AGPL-3.0-or-later**. Anyone may fork it, delete a paywall, and +run it — commercially. §13 only obliges them to publish the modified source. +That is not a loophole to close; it is the deal the project made deliberately at +v0.3.0. Any monetization plan that depends on a self-hoster being unable to +remove a check is already broken, so no plan below depends on that. + +There is a **CLA** ([`CLA.md`](../../CLA.md)), so dual licensing is available, +and `LICENSING.md` already advertises commercial terms. That is a second revenue +line that requires no code split at all, and it is worth more than it looks: +a large share of enterprises run an OSS policy that flatly refuses AGPL +dependencies. For those buyers the product is not a feature — it is the license. + +What follows from that: **sell operations and terms, not code.** The hosted +instance sells the thing a fork cannot copy — running it, on someone else's +Cloudflare bill, with backups tested and an SLA. The commercial license sells +the thing the AGPL cannot give. Neither requires a proprietary `ee/` directory, +and both survive a fork. + +### The one gap to close first + +Trademark is the only enforcement lever the AGPL leaves against a hosted +competitor, and the repository does not currently exercise it. There is no +`TRADEMARK.md`, and "Stratum" is not reserved anywhere in the licensing docs. +A fork may run the code; it may not call itself Stratum or use the mark to sell +against you. Registering the mark and adding a short trademark policy is cheap, +takes a week, and should land before any of the engineering below. + +--- + +## 2. What actually costs money + +The honest basis for a paywall is marginal cost to the operator plus scarcity of +a shared resource. Sorted by both, using what the code already meters and bounds: + +**Unbounded per-change spend.** The LLM evaluator (`src/evaluation/llm-evaluator.ts`) +runs on the Workers AI binding — the operator's neurons, not the author's. It is +invoked from `runEvaluation` in `src/services/change-flow.ts`, **synchronously at +change creation**, with a per-call input bound of `MAX_DIFF_CHARS_CEILING` +(100,000 chars) and no bound at all on how many times a user may create a change. +This is the single largest exposure and the obvious first thing to meter. + +**Shared scarce capacity.** Deployments are worse than expensive — they are +serialized. `MAX_DEPLOY_CONCURRENCY = 1` in `src/deploy/limits.ts`, and the +comment there explains it cannot be raised without first cutting the per-deploy +memory peak. One free user's deploy queue is therefore in front of a paying +customer's. Same shape, less severe, for the import queue and the merge queue +Durable Object. + +**Sandbox milliseconds.** Bounded per run by `MAX_TOTAL_BUDGET_MS` +(`src/evaluation/limits.ts`), unbounded in count. Currently theoretical: the +`[[sandboxes]]` binding is commented out at `wrangler.toml:156`, so nothing +executes code on any deploy today. It becomes the second-largest exposure the +day that binding is enabled. + +**Bytes that never go away.** Git objects on Artifacts and R2, backup blobs +(`BACKUP_RETENTION`, default 14 runs), and D1 rows — `events`, `audit_log`, +`provenance`, `cost_records`. The roadmap's "D1 hot/cold rotation" item is the +same problem seen from the operations side; retention tiers are the same lever +seen from the pricing side, and they should be designed together rather than +twice. + +**Near zero.** Everything else. The change flow itself, reviews, issues, orgs and +teams, the CLI, `/mcp`, git clone and push, webhooks, API tokens, and the +deterministic evaluators (`diff-evaluator`, `secret-scanner`) are Worker CPU on +requests that are already rate-limited. Gating any of these recovers no money +and costs the project its reputation. They stay free, permanently, for everyone. + +That last paragraph is the load-bearing one. The credible version of this +paywall meters four things — model tokens, sandbox time, deploy slots, stored +bytes — and leaves the product alone. + +--- + +## 3. Tier shape + +Concrete allowances are deliberately omitted; §7 says to measure before setting +them. What matters now is which axis each tier moves along. + +**Free.** Unlimited public projects and unlimited collaborators on them. A small +number of private projects. A monthly platform-LLM allowance sized to a few dozen +real changes. Sandbox off. One deploy target. A storage cap. Short retention on +events and audit entries. Community support. **BYOK available** — see §5. + +**Pro**, per seat. Private projects unlimited, a real platform-LLM allowance, +sandbox on, deploys, 90-day retention, a higher rate-limit bucket, priority in +the deploy and import queues, email support. + +**Team**, per seat. Orgs, teams, per-project permission grants (a roadmap item +today), SSO, one-year audit retention with export, self-serve backup restore, +higher concurrency, an SLA. + +**Enterprise self-managed.** A commercial license per `LICENSING.md`, support, +and no usage metering whatsoever, because it runs on their infrastructure. + +Two notes on the boundaries. Do not put SSO alone behind the top tier as the +only reason to reach it — the "SSO tax" is the most reliably resented pattern in +this market. It belongs in Team alongside SCIM, audit export, and permission +grants, as one item in a coherent administrative bundle rather than the toll. +And free-tier caps must be **hard**, with an explicit top-up, not silent +overage: Workers AI and queue time are real spend and a loop is one script away. +Paid tiers get soft overage with alerts, because a blocked merge on a paying +customer's release day is worth more than the overage. + +--- + +## 4. Enforcement architecture + +The pattern to copy is already in the repository. `src/beta/gate.ts` is a +cloud-only hook that lives in the AGPL tree, calls out to a service that is not +in this repository, and is **completely inert when its environment variables are +unset** — which is the default, so self-hosters are unaffected and the code that +does the gating is still readable by everyone. Generalize exactly that. + +### 4.1 Entitlements + +A new `src/billing/entitlements.ts` defines the interface and ships a default +implementation that grants everything: + +```ts +export type LimitKey = + | "llm_tokens_month" + | "sandbox_ms_month" + | "deploys_month" + | "storage_bytes" + | "private_projects" + | "requests_per_minute" + | "retention_days"; + +export interface Entitlements { + plan: string; // "self-hosted" by default + limits: Record; // -1 = unlimited +} + +export interface EntitlementsProvider { + forOwner(ownerId: string, kind: "user" | "org"): Promise; +} +``` + +`UnlimitedEntitlements` is the default and returns `-1` for every key. A +`RemoteEntitlements` provider activates only when `BILLING_SERVICE_URL` and +`BILLING_SERVICE_SECRET` are set, mirroring `betaGateEnabled`. Plan definitions, +Stripe, invoices, dunning, and the customer portal all live behind that URL, in +the same private cloud layer that already hosts the referral service. None of it +belongs in this repository, and none of it needs to be. + +**Fail direction, and why it differs from the beta gate.** `validateInviteCode` +fails *closed* — a service outage must not let ungated users through a signup +wall. Entitlements must fail *open*, to the last cached value or to the free +tier, because a billing-service outage that blocks a paying customer's merge is +a strictly worse failure than a few minutes of unmetered usage. Cache in +`env.STATE` with a short TTL and a stale-while-error read. Write this asymmetry +into the module's doc comment; it will otherwise be "fixed" by someone applying +the beta gate's convention. + +### 4.2 Metering + +There are two counters with different jobs, and conflating them is the classic +way to build a billing system that is both slow and wrong. + +`cost_records` (migration 021) stays the **durable ledger**. It is already +written by `recordCosts` from `runEvaluation`, already carries +`llm_tokens | sandbox_ms | git_ops` and an `estimated` flag, and it is what an +invoice or a dispute is reconciled against. It needs two additive migrations: + +- **`owner_id`** — the table records `project`, `project_id`, `change_id`, and + `workspace`, but nothing identifying who pays. Every per-customer aggregate is + impossible until this exists, and backfilling it later means joining through + namespaces for historical rows. Add it before the volume grows. +- **`source`** (`'platform' | 'byok'`) — §5 needs to attribute usage that must + not be charged. Without it, BYOK tokens are indistinguishable from hosted ones + in the ledger. + +The **hot counter** that gates a request is separate. It must not lose +increments under concurrency, which rules out the read-modify-write on KV that +`src/middleware/rate-limit.ts` currently performs — that race is acceptable for a +per-minute request limiter and is not acceptable for anything that gates spend. +The codebase already has the right precedent: `MagicLinkRateLimiter`, described +in `src/types.ts` as "serialized magic-link send counters". Add a `UsageMeter` +Durable Object per owner on the same model, reconciled against `cost_records` by +the existing scheduled handler. + +The rule to state once and follow: **Durable Object for anything that gates +spend, KV for anything that gates request rate.** + +### 4.3 Where the checks go + +Four call sites, and one design decision inside the first. + +*Before the LLM evaluator runs*, in `buildEvaluators` or at the top of +`LLMEvaluator.evaluate`. The decision is what an exhausted allowance means for a +merge *gate*. Silently skipping the evaluator is a governance hole — a policy +that requires AI review would stop requiring it the moment someone ran out of +credit, which is the worst possible failure for the product's entire premise. +Instead, mirror the existing behavior in that file for an oversize policy +context: return `ok()` with `passed: false` and a reason naming the cause and +both remedies (upgrade, or add your own key). That is already the established +shape at `llm-evaluator.ts` and it converts honestly. + +*Before claiming a deploy message*, in the queue consumer, so a free-tier deploy +never occupies the single concurrency slot ahead of a paid one. + +*At import and at `git-receive-pack`*, for storage bytes. Pushes already flow +through the rate limiter deliberately (`isExemptGitRead` exempts only the read +side), so the hook point exists. + +*At private-project creation*, for the count. + +And `rateLimitMiddleware`'s `defaultLimit` becomes entitlement-derived rather +than the current binary `isAuthenticated ? 1000 : 60`. + +--- + +## 5. BYOK + +BYOK is the feature that makes this a capacity limit rather than a paywall on +the open-source project, and it should ship *before* any cap is enforced. A free +user who supplies a key gets unlimited AI review, bounded only by request rate. +That framing is both true and the honest thing to say publicly. + +### 5.1 The agent already does it + +`agent/src/llm.ts` calls the Anthropic API directly with the user's own +`ANTHROPIC_API_KEY`, from the user's own machine, under Apache-2.0. The most +expensive model work in the whole system — actually writing the code — already +never touches the server and never will. Say so in the pricing page; it removes +the first objection before it is raised. + +### 5.2 What is missing: server-side evaluator BYOK + +`LLMEvaluator` takes an `AiBinding` in its constructor and is hard-wired to +Workers AI. Introducing a provider seam is a small refactor that is worth doing +for its own sake, independent of billing — it is what lets a self-hoster use a +frontier model, or a local vLLM, for their merge gate. + +```ts +export interface LlmProvider { + run(model: string, messages: Message[]): Promise<{ text: string; usage?: Usage }>; +} +``` + +Three implementations: `workers-ai` wrapping the existing binding and remaining +the default so nothing changes for anyone; `anthropic`; and `openai-compatible`, +which covers OpenAI, OpenRouter, Groq, Together, and self-hosted vLLM in one. +A real provider also returns real token counts, which retires the +`~4 chars/token` estimate and the `estimated: true` flag on those cost samples. + +### 5.3 Key storage — already built + +`project_secrets` (migration 047) is the right store and needs no redesign. It +is AES-GCM with `(project_id, name)` bound as additional authenticated data, so +a row copied to another project fails to decrypt rather than authorizing against +the wrong account; it is write-only by construction with no read path reachable +from any route; and `SECRET_NAME_PATTERN` already accepts `ANTHROPIC_API_KEY` +and `OPENAI_API_KEY` unchanged. The evaluator gets a `loadSecretValues` path +alongside the deploy runner's, held to the same rule: not reachable from a +route. + +The one extension needed is scope. `SecretScope` in `src/utils/crypto.ts` is +project-shaped, and a team will want one key for an org rather than one per +project. Add an org-scoped variant of the AAD; do not widen the existing one. + +### 5.4 The security problem, which is the real work here + +`.stratum/policy.yaml` is **repository content**. Anyone who can open a change +can propose an edit to it. Adding a `provider` and `baseUrl` to the evaluator +config therefore hands an attacker a request forgery primitive on the Worker, +and the body of that request is the diff — the customer's source code, plus +whatever the secret scanner was about to catch. + +Three mitigations, all mandatory: + +1. **`baseUrl` is never taken from the policy file.** The operator sets an + allowlist in the environment (`LLM_BYOK_ALLOWED_HOSTS`), and the policy may + only *select* a configured provider by name. This is not negotiable and is + the reason the config surface is `provider: anthropic` rather than a URL. +2. **Sanitize like `deploys:` already does.** `sanitizeDeploys` in + `src/deploy/config.ts` is the pattern: build a fresh object, never pass a + slice of parsed user input downstream, and surface every rejection as a + visible `deployRejections`-style entry rather than a dropped `logger.warn`. +3. **Tie into the existing protected-config signal.** Migration 039 added + `changes.touches_protected_config`. A change that edits the evaluator's + provider block is exactly what that flag is for, and it should require review + at a level above an ordinary change. + +Note also that the key itself is a redaction surface: `src/deploy/redact.ts` +already exists for provider output on `deployments.log_tail`, and provider error +bodies from an LLM call need the same treatment before they reach an +`EvalResult.reason`. + +### 5.5 Billing consequence + +BYOK usage is recorded with `source = 'byok'` and does not decrement the +`llm_tokens_month` allowance. It is not free to the operator, though — it still +costs a subrequest and Worker wall time — so BYOK moves a project from a token +quota onto the request-rate quota rather than off all quotas. State that plainly +in the docs; a user who discovers it at a 429 will assume they were misled. + +### 5.6 The pattern is already the house style + +Deploy credentials are BYO (`project_secrets` holds the Vercel and Cloudflare +tokens), and BYO-R2 for backups is a natural next one. "Stratum orchestrates, +you own the accounts it spends against" is a coherent product position, not a +concession, and it is materially easier to sell to a security team than a +platform that proxies everything. + +--- + +## 6. Abuse + +*Written while signup was gated by invite codes (`src/beta/gate.ts`), which was +doing the anti-abuse work and hiding the problem. That gate has since lifted — +`app.usestratum.dev` has open signup (`docs/user-guide/faq.md`) — so the +condition this section anticipated is the one in force now, not a future one. The +gate code remains, inert unless an operator configures it.* + +A free LLM allowance on an open signup is a target. + +The design has an unusual advantage here: the evaluator is not a chat endpoint. +`SYSTEM_PROMPT` is fixed in the source, the output is parsed as a strict verdict +object, and anything else fails closed. It is close to useless as a general +model proxy, which removes the main motive. What remains is ordinary +free-tier farming, handled by keying quota on the **owner** rather than the +project — otherwise forking a project resets the counter — and by keeping the +existing verified-identity signup paths (GitHub, Google, magic link). + +--- + +## 7. Sequencing + +Nothing here should ship in the order it was written. The order that matters: + +**0. Trademark policy and registration.** Independent of all engineering, and +the only real lever against a hosted fork. Do it first because it has a lead +time. + +**1. Measure.** `cost_records` is already being written and +`getProjectCostSummary` in `src/storage/costs.ts` is currently **dead code** — +defined, exported, never called. Add `owner_id` and `source`, then build the +per-owner usage view on top of that existing function. Run it for a month +against real traffic on `app.usestratum.dev` before choosing a single number in +§3. Every allowance in this document is deliberately unspecified for that reason. + +**2. Entitlements interface**, with `UnlimitedEntitlements` wired in as the +default. Zero behavior change on merge, which makes it reviewable on its own. + +**3. `UsageMeter` Durable Object** plus reconciliation against `cost_records`. +Still no enforcement. + +**4. BYOK.** Ships as a feature to self-hosters — pick your own model for the +merge gate — and lands before any cap so the escape hatch predates the wall. + +**5. The usage UI.** Non-negotiable prerequisite for step 6. If the first thing +a user learns about a quota is a blocked merge, the quota was a bug report. + +**6. Cloud entitlements provider and Stripe**, in the private layer. + +**7. Turn on free-tier caps**, announced at least 30 days ahead, with the usage +page live throughout. + +--- + +## 8. What not to do + +Stated explicitly, because each of these is the default mistake: + +Do not add a license-key check to the AGPL server, or obfuscate any part of it. +It is removable in one commit, it reads as bad faith, and it buys nothing. + +Do not move a shipped feature behind the wall. Everything in +`docs/CURRENT_CAPABILITIES.md` shipped under MIT or AGPL. Taking any of it back +costs more in credibility than it can return in revenue. Only *new* hosted +capacity limits, and self-host stays unlimited by default. + +Do not degrade the open-source build to make the hosted one look better. The +self-hosted path is the top of the funnel for the commercial license, which is +the higher-margin product of the two. + +Do not gate collaborators or public projects. Free multiplayer on public work is +how the product gets seen at all. diff --git a/docs/user-guide/ci-integration.md b/docs/user-guide/ci-integration.md index 854d7d9d..123e50b2 100644 --- a/docs/user-guide/ci-integration.md +++ b/docs/user-guide/ci-integration.md @@ -325,10 +325,12 @@ To set expectations, Stratum has none of the following today: - Build artifacts (upload/download/retention) - Dependency/build caching - Scheduled jobs (cron workflows) -- A secrets store *for evaluators*. Stratum does have an encrypted per-project - secret store, but it is **deploy-only**: the deploy runner is the sole reader, - and nothing interpolates it into a policy file. The webhook evaluator's - `secret` still lives literally in `.stratum/policy.yaml`. +- A general secrets store *for evaluators*. Stratum does have an encrypted + per-project secret store, but only two things read it: the deploy runner, and + the `llm` evaluator when a policy names a provider to bring its own key to + ([Getting started](getting-started.md#bringing-your-own-model-key)). Nothing + interpolates it into a policy file — the webhook evaluator's `secret` still + lives literally in `.stratum/policy.yaml`. - Deployment *environments* — no staging/production separation, no per-environment variables, no environment protection rules. What exists is a flat list of named deploys, each optionally gated by a single approval diff --git a/docs/user-guide/deployments.md b/docs/user-guide/deployments.md index aa8c88d7..5732bd86 100644 --- a/docs/user-guide/deployments.md +++ b/docs/user-guide/deployments.md @@ -378,8 +378,9 @@ happens to be stored beside your credentials, not as a credential. Secrets are per project, encrypted at rest (AES-GCM, with the project id and secret name bound as additional authenticated data), and there is **no read -path** — no API, UI or CLI surface returns a stored value. Only the deploy -runner decrypts them. +path** — no API, UI or CLI surface returns a stored value. Two things decrypt +them: the deploy runner, and the `llm` evaluator when a project brings its own +model key ([Getting started](getting-started.md#bringing-your-own-model-key)). - **In the UI:** *Project → Settings → Deploy secrets*. Add by name and value; delete by name. Values are never rendered back. @@ -521,7 +522,10 @@ Self-hosters need three things beyond the default deployment: **Rotating it makes every stored secret undecryptable.** There is no re-encryption path: after a rotation, deploys fail with `Could not decrypt - project secret…` and every project must re-enter its values. + project secret…` and every project must re-enter its values. That now reaches + further than deploys — a project whose policy names an LLM provider reads its + provider key from the same store, so a rotation blocks the **merge gate** on + those projects until the key is re-entered, not just their deployments. 2. **The queues must exist**, and the `DEPLOY_QUEUE` binding must be configured. Without the binding, merges simply never enqueue a deploy, and diff --git a/docs/user-guide/faq.md b/docs/user-guide/faq.md index c23c06f5..b519fe21 100644 --- a/docs/user-guide/faq.md +++ b/docs/user-guide/faq.md @@ -57,9 +57,14 @@ running GitHub Actions on the promoted PRs. See [CI Integration](ci-integration.md). Two things on that list have since arrived in a narrow form. Stratum now has an -**encrypted per-project secret store** — but it is deploy-only: the deploy -runner is its sole reader, and the webhook evaluator's `secret` still lives -literally in the policy file. And it can **deploy the merged tree** to +**encrypted per-project secret store**, read by the deploy runner and, since +BYOK, by the `llm` evaluator for a project's own provider key. It is not a +general-purpose store: nothing else reads it, and the webhook evaluator's +`secret` is still taken literally from the policy file rather than by name from +the store — so it sits in `.stratum/policy.yaml`, in git, and in every clone. +Treat it as published to anyone who can read the repository, and prefer a +receiver that verifies something other than a shared secret until that field +resolves from storage the way `deploys:` already does. And it can **deploy the merged tree** to Cloudflare or Vercel from a `deploys:` block, with an optional approval gate and a retry. That is not deployment environments: there is no staging/production separation, no per-environment variables, no build step, no @@ -124,13 +129,24 @@ The software is free software (see the licensing question below) and costs nothing; self-hosted, you pay only for the Cloudflare resources you use. The full binding list is in the [README](https://github.com/stratum-eng/stratum#prerequisites) — Workers, Artifacts, D1, KV, Queues, Durable Objects, R2 and Analytics Engine, -plus Workers AI for the `llm` evaluator and Sandboxes for the `sandbox` -evaluator **and** `merge.postMergeCommand`. Stratum meters estimated +plus Workers AI for the `llm` evaluator (or an `LLM_PROVIDERS` allowlist, so +projects can bring their own model key instead) and Sandboxes for the `sandbox` +evaluator **and** `merge.postMergeCommand`. Pointing the `llm` evaluator at +your own provider through `LLM_PROVIDERS` adds that provider's own charges, +billed to you by them and not visible on your Cloudflare bill. Stratum meters resource usage per change — LLM tokens, sandbox execution milliseconds, and git operations — and shows it alongside the evaluation evidence, so you can see what -each (agent) change cost you. The hosted instance has open signup and is free -while billing and multi-tenancy for a managed offering ("Stratum Cloud") are -planned but not built. +each (agent) change cost you. LLM tokens are the counts the provider reported, +falling back to a `~4 chars/token` estimate (marked as estimated) only when a response +omits them. Those per-change records roll up into a monthly per-account total +you can read at `/settings/usage`. + +Nothing in a self-hosted instance ever refuses you on the strength of those +numbers. Allowances, and the enforcement that consults them, are switched on by +pointing the instance at a billing service; unconfigured — the default — every +allowance reads as unlimited. The hosted instance has open signup and is free: +plan definitions, checkout and subscription state for a managed offering +("Stratum Cloud") are still not built. ## What are the current limitations? @@ -190,13 +206,14 @@ SSH transport is not supported (Workers have no raw TCP listener). Node.js 22.13+ and your own Cloudflare account with Workers, **Artifacts (which is in beta — you need access to it)**, D1, KV, Queues, and Durable Objects. -Optional: the Workers AI binding for the LLM evaluator, R2 for backups, and -Cloudflare Email for magic links. Sandboxes — for the `sandbox` evaluator and -`merge.postMergeCommand` — is a gated beta whose `[[sandboxes]]` binding is -commented out in `wrangler.toml`; uncomment it in every `[env.*]` block you -deploy once your account has access, and until then keep both features out of -your policy, since the evaluator fails closed (and a `requiredEvaluators` entry -for it blocks all merges). +Optional: the Workers AI binding for the LLM evaluator (or, instead, an +`LLM_PROVIDERS` allowlist so projects can bring their own model key), R2 for +backups, and Cloudflare Email for magic links. Sandboxes — for the `sandbox` +evaluator and `merge.postMergeCommand` — is a gated beta whose `[[sandboxes]]` +binding is commented out in `wrangler.toml`; uncomment it in every `[env.*]` +block you deploy once your account has access, and until then keep both features +out of your policy, since the evaluator fails closed (and a `requiredEvaluators` +entry for it blocks all merges). The [README Quick Start](../../README.md#quick-start) covers secrets, migrations, and deployment; keep production and staging in separate Artifacts namespaces. diff --git a/docs/user-guide/getting-started.md b/docs/user-guide/getting-started.md index 7b57c776..87f6d0a4 100644 --- a/docs/user-guide/getting-started.md +++ b/docs/user-guide/getting-started.md @@ -142,11 +142,15 @@ evaluators: # totalBudgetMs: 150000 # allowInstallScripts: false - # AI review of the diff via the Workers AI binding, scored 0.0-1.0. - # Omitting `model` uses the default (@cf/meta/llama-3.1-8b-instruct); - # a model id Workers AI doesn't serve makes this evaluator fail closed. + # AI review of the diff, scored 0.0-1.0. With no `provider` this runs on + # the instance's Workers AI binding, and omitting `model` uses the default + # (@cf/meta/llama-3.1-8b-instruct); a model id Workers AI doesn't serve makes + # this evaluator fail closed. Naming a `provider` runs the review on your own + # key instead (see below) and then `model` is required. - type: llm threshold: 0.7 + # provider: anthropic + # model: claude-sonnet-4-5 merge: requiredApprovals: 1 @@ -216,18 +220,118 @@ requiring a human approval and refusing force-merge. when you need it is *not* a failing install, but a native module that installs unbuilt and then fails when the test command loads it. - **`llm`** — sends the diff to an LLM for review against your criteria. - `model` picks the reviewer and must be a model the **Workers AI binding** - serves (default: `@cf/meta/llama-3.1-8b-instruct`); `threshold` is the - minimum passing score (0.0–1.0); `maxDiffChars` bounds how much diff is - sent (default 24,000, max 100,000). An unavailable model or unparseable - verdict fails closed. Token usage is recorded on the change as a cost - record. + `model` picks the reviewer (default: `@cf/meta/llama-3.1-8b-instruct`, which + the **Workers AI binding** serves); `threshold` is the minimum passing score + (0.0–1.0); `maxDiffChars` bounds how much diff is sent (default 24,000, max + 100,000); `provider` runs the review on your own key instead of the + instance's binding (see [Bringing your own model key](#bringing-your-own-model-key)). + An unavailable model or unparseable verdict fails closed. Token usage is + recorded on the change as a cost record — the counts the provider reports, + or an estimate marked as estimated when a response omits them. + **A policy may declare at most one `llm` entry** — a second one is a + merge-blocking policy error, because the two entries cannot both be the + configuration in force. + + Those four keys are the *only* keys this entry accepts. It is a whitelist: + anything else you write here — including `baseUrl` — is ignored with a + warning in the instance's logs and never reaches the reviewer. There is + deliberately no way for a policy file to name an endpoint; a project may + only select a provider the operator has already configured. + +A policy may declare **at most 16 `evaluators:` entries**; more than that is a +policy error and blocks merges, the same as any other unusable entry. (The +`deploys:` list has the same cap, for the same reason: one merge must not turn +into an unbounded number of external calls.) Two top-level knobs sit alongside `evaluators`: `requireAll` (default `true`) makes the aggregate verdict demand every evaluator pass — set it to `false` to pass when any one does — and `minScore` (default `0.7`, clamped to 0–1) is the per-evaluator pass threshold the `diff` and `sandbox` evaluators score against. +### Bringing your own model key + +By default the `llm` evaluator runs on the instance's Workers AI binding, and +the operator pays for the tokens. A project can instead run the merge gate on +its own account — its own provider, its own model, its own bill. + +It takes two things, and both are deliberate: + +1. **The operator configures the provider.** A named allowlist lives in the + instance's `LLM_PROVIDERS` variable; each entry has a `name`, a `kind` + (`anthropic` or `openai-compatible`), and the `baseUrl` for that endpoint. + Self-hosting? That variable is yours to set. Unset — the default — the + binding is the only option and everything below is inert. +2. **The project stores the key.** The credential goes in the project's secret + store, the same one deploys use (*Project → Settings → Deploy secrets*, or + `PUT /api/projects/{namespace}/{slug}/secrets/{name}`), under the name + derived from the provider: provider `anthropic` reads `ANTHROPIC_API_KEY`, + provider `my-gateway` reads `MY_GATEWAY_API_KEY`. Only a project admin who + is not an agent may write it, and no route, log line or evaluation reason + ever reads it back. + +Then name the provider in the policy: + +```yaml +evaluators: + - type: llm + provider: anthropic + model: claude-sonnet-4-5 + threshold: 0.7 +``` + +**`model` is required whenever `provider` is set.** The default model id is a +Workers AI one and would fail every call against any other endpoint, so a +provider entry that does not name its model is rejected rather than guessed at. + +**Everything here fails closed, and never falls back to the instance's +binding.** A provider name the instance has not configured, a missing or +undecryptable key, an instance with no `DEPLOY_SECRET_KEY`, a provider that +answers with a redirect — each one fails the `llm` gate with a reason naming +which of them it was, and none of them quietly moves your review back onto the +operator's account. The corollary is worth planning for: a policy that names a provider +this instance does not have is a **policy error, and policy errors block +merges**. + +Two more consequences to know before you switch: + +- **Any project *writer* can spend your credit** by opening changes. BYOK moves + the token bill to your provider account; it does not add an approval step in + front of it. +- **On a hosted instance, your own key lifts the token allowance and nothing + else.** The evaluation rate ceiling below is independent of who is billed for + tokens: it bounds evaluation and Worker capacity, which is the operator's + whoever owns the model account. + +### Usage limits on a hosted instance + +Self-hosted, there are no limits: the whole metering path is inert unless the +operator has configured a billing service, and every allowance reads as +unlimited. On a hosted instance the shape is: + +- **A monthly token allowance**, plus monthly sandbox time and deploy counts. + Only platform-billed usage counts against it — spend on your own provider key + is billed by that provider and is tracked separately. +- **An hourly ceiling on evaluations**, which bounds burst rather than spend. + **Bringing your own key does not lift it.** +- **An allowance follows the person, not the project.** By default it is checked + against whoever ran the evaluation (an agent spends its owner's), so it is the + same allowance whether you work in your own namespace or an organization's, and + creating another organization does not hand you a fresh one. The one exception + is an organization on a plan that pools: there the organization is the subject + and its members draw on its allowance rather than their own. An organization + the billing service has never heard of is not that — pooling has to be + positively granted, or creating an organization would be the reset this rule + exists to close. + +You can see all of it — consumption, allowance, and when the period resets — at +`/settings/usage`, or over MCP with `stratum_get_usage`. Crossing 80% of a meter +raises a banner and sends one email. Where an operator has switched enforcement +on, an exhausted allowance surfaces as a **failing gate** on the change that hit +it — never as a skipped one — naming what ran out, when it resets, and both ways +out. An operator who has not switched it on records the same decisions and +admits every one of them, which is how a new limit gets measured before it +blocks anything. + ### The merge protections Everything under `merge:` is branch protection, enforced at the merge step: diff --git a/docs/user-guide/mcp.md b/docs/user-guide/mcp.md index 7ab17d80..5b520587 100644 --- a/docs/user-guide/mcp.md +++ b/docs/user-guide/mcp.md @@ -158,7 +158,7 @@ vetted by anyone. A name you do not recognise should be disconnected. ## The tools -Eighteen tools cover the full contribution loop. Project arguments take a +Nineteen tools cover the full contribution loop. Project arguments take a `namespace/slug` reference (`@acme/api` and `acme/api` both work). **Identity and reading** @@ -166,6 +166,7 @@ Eighteen tools cover the full contribution loop. Project arguments take a | Tool | What it does | |---|---| | `stratum_whoami` | Identify the authenticated **user** (calls `GET /api/users/me`, which refuses agent tokens). | +| `stratum_get_usage` | This account's metered usage against its plan limits for the current period — per-meter consumption, the limit on each, the rate ceilings, and when the period resets. An agent token reports its **owner's** allowance, because that is the account its evaluations are charged to. | | `stratum_list_projects` | List projects in the caller's **own namespace** — org-namespace projects are not included, and an agent token gets an empty list. | | `stratum_get_project` | Project metadata — id, namespace, visibility, git remote. | | `stratum_list_files` | File paths at the HEAD of the default branch. | @@ -211,6 +212,14 @@ anyway. | `stratum_list_issues` | List issues, optionally by status. | | `stratum_update_issue` | Update status, title, or body by issue number. | +**The billing surface is read-only, and stays that way.** `stratum_get_usage` is +the only tool that touches it: there is no tool to raise a limit, buy capacity, +attach a payment method, or store a provider key. A spend limit an agent can +lift is not a limit — the same line that keeps agent tokens from submitting +review verdicts — and a provider credential must never pass through a model's +context window. Those actions live in the web UI, where a human consents to them +in a browser. + ## What an agent token can — and cannot — do The server accepts either token kind, but the API behind it does not treat diff --git a/docs/user-guide/troubleshooting.md b/docs/user-guide/troubleshooting.md index 0e6368e8..3cda905f 100644 --- a/docs/user-guide/troubleshooting.md +++ b/docs/user-guide/troubleshooting.md @@ -151,6 +151,44 @@ and then fails at load. If your build genuinely needs lifecycle scripts Re-evaluating runs under the current defaults, and the default install timeout dropped from 120s to 90s. Re-check against `totalBudgetMs` above. +### The `llm` gate fails with a reason about a provider + +These are all the same shape: the gate **ran and failed**, rather than being +skipped, and the reason names its own cause. That is deliberate — a policy that +requires AI review must not stop requiring it because the configuration is +wrong, so none of these fall back to the operator's model. + +- **`provider "x" is not configured`** — `.stratum/policy.yaml` names a provider + this instance's operator has not put in `LLM_PROVIDERS`. Self-hosting, add it; + on someone else's instance, ask them which names they offer, or drop + `provider:` to use the instance's own binding. +- **A missing or undecryptable key** — the project has no stored credential for + that provider, or the instance has no `DEPLOY_SECRET_KEY`. Re-enter the key in + the project's secret settings. +- **After a `DEPLOY_SECRET_KEY` rotation** — rotating that key makes every stored + project secret undecryptable, deploy secrets and provider keys alike, so BYOK + projects block until each re-enters its key. This is the documented cost of a + rotation, not a fault. +- **A redirect from the provider** — a 3xx is refused rather than followed, + because the allowlist validates the host a request is *sent* to and a redirect + would move the prompt and your API key somewhere unvalidated. Check the + `baseUrl` the operator configured; it usually means a missing `/v1`. +- **Two `llm` entries** — a policy declaring the evaluator twice is refused + outright rather than one being picked. Keep one. + +### `... allowance is used up` or an evaluation rate limit + +Only reachable on an instance whose operator has configured a billing service +**and** switched enforcement on. The reason names what ran out, when it resets, +and both ways forward: bring your own provider key, or raise the plan. Your own +key lifts the **token** allowance only. The hourly evaluation ceiling still +applies whichever account owns the model key, because what it bounds is +evaluation and Worker capacity on the instance rather than model spend. + +Check `/settings/usage` (or `GET /api/users/me/usage`, or `stratum_get_usage` +over MCP) for what is left and when the period rolls over. Self-hosted, none of +this is reachable: every allowance reads as unlimited and nothing is refused. + ## Merging ### `STALE_BASE` / `STALE_WORKSPACE` / `WORKSPACE_UNVERIFIABLE` diff --git a/migrations/048_cost_attribution.sql b/migrations/048_cost_attribution.sql new file mode 100644 index 00000000..3c344658 --- /dev/null +++ b/migrations/048_cost_attribution.sql @@ -0,0 +1,48 @@ +-- Migration 048: name who pays for each cost sample. +-- +-- `cost_records` (021, given `project_id` by 025) has recorded what a change +-- consumed since it existed, but never who owes for it. A project name is not a +-- billing subject: names are unique only per-namespace, and projects are +-- transferred and deleted. These columns make every new row attributable at the +-- moment the spend is incurred, which is the only moment the payer is known for +-- certain. + +-- NULLABLE, and permanently so. Every historical row predates attribution, and +-- the mapping needed to backfill it lives in KV (the ProjectEntry's owner), not +-- in D1 — the same reason 025 backfilled no `project_id`. Live code writes NULL +-- too: `resolveBillingSubject` (src/storage/costs.ts) returns null rather than +-- throwing when an owner cannot be named, because `recordCosts` is best-effort +-- inside change creation, merge and deploy. So a NULL owner is a row nobody can +-- be billed for, which is honest, where an owner guessed from a project name +-- would be a row billed to possibly the wrong account. +ALTER TABLE cost_records ADD COLUMN owner_id TEXT; + +-- CHECK'd rather than free text: an owner_type outside this set is unreachable +-- from `resolveBillingSubject`, which maps ProjectEntry's third kind ('agent') +-- onto the user that owns the agent instead of passing it through. A row +-- claiming a type no billing subject can have would aggregate under an account +-- that will never be invoiced. +ALTER TABLE cost_records ADD COLUMN owner_type TEXT CHECK(owner_type IN ('user','org')); + +-- NOT NULL DEFAULT because the backfill is correct by construction, not merely +-- convenient: BYOK does not exist yet, so every row already in this table was +-- spend on the operator's own provider account, and the default states that +-- truth in place. Precedent for both ALTER shapes: a CHECK on an added column in +-- 037, NOT NULL DEFAULT in 032/034/041. No table rebuild — unlike `kind`, whose +-- closed CHECK would need one to widen. +ALTER TABLE cost_records ADD COLUMN source TEXT NOT NULL DEFAULT 'platform' + CHECK(source IN ('platform','byok')); + +-- Serves the owner-scoped usage read this work is being built toward +-- (`WHERE owner_id = ? ORDER BY created_at DESC`); no caller issues it yet. +-- +-- Sorting on `created_at` is safe HERE specifically. Application code writes it +-- only through `recordCosts`, as `new Date().toISOString()`, so every row this +-- table gets in production is ISO 8601 with a 'T' separator. The hazard +-- 042/044/047 warn about is a column mixing that with `datetime('now')`'s space +-- separator, which sorts differently as TEXT (' ' 0x20 < 'T' 0x54) and silently +-- breaks every range comparison. A single-format column has no such problem. +-- +-- Rows with a NULL owner are still indexed (SQLite indexes NULLs) but are never +-- selected by that predicate, which is the intent. +CREATE INDEX IF NOT EXISTS idx_costs_owner ON cost_records(owner_id, created_at DESC); diff --git a/migrations/049_usage_periods.sql b/migrations/049_usage_periods.sql new file mode 100644 index 00000000..e35babd5 --- /dev/null +++ b/migrations/049_usage_periods.sql @@ -0,0 +1,94 @@ +-- Migration 049: the owner-scoped usage aggregate a limit is compared against. +-- +-- `cost_records` cannot be that aggregate, and this is the whole reason for a +-- second table. It is in PROJECT_SCOPED_TABLES (src/storage/deletion.ts) and +-- hard-deleted by the project cascade, so any allowance summed from it is +-- refunded by deleting a project: burn the month's quota, delete the project, +-- get the quota back. This table keys on the billing subject instead and joins +-- only the ACCOUNT cascades, so the one thing that clears usage is erasing the +-- account that incurred it — at which point there is no subject left to bill. +-- +-- `cost_records` remains the per-change evidence of what was spent; this is the +-- running total. Both are written by `recordCosts` (src/storage/costs.ts) from +-- the same timestamp, so a cost row and the period it counts toward can never +-- disagree. +-- +-- This total is NOT recoverable from that ledger, which is the uncomfortable +-- half and the reason it is backed up in its own right (see BACKUP_TABLES in +-- src/storage/d1-backup.ts). The cascade that hard-deletes cost rows leaves +-- these standing, so replaying the ledger would rebuild usage only for projects +-- that still exist and hand every other owner their month back — the same +-- refund this table exists to prevent, arriving by a different route. A lost +-- upsert is therefore a permanent under-count, accepted because the alternative +-- is failing a change over an accounting write. +-- +-- Timestamps are ISO 8601 strings written by the application, NOT +-- `datetime('now')`, for the reason 042/044/047 give: the two formats sort +-- differently as TEXT (' ' 0x20 < 'T' 0x54), and a column mixing them silently +-- breaks every comparison over it. +-- +-- Rollback: drop the table. Writes are best-effort by contract and reads are +-- Result-typed, so an absent table degrades to logged failures rather than a +-- broken change flow — but every accumulated total is gone with it. + +CREATE TABLE IF NOT EXISTS usage_periods ( + -- The subject `resolveBillingSubject` named: a user or an org, never a + -- project and never an agent. NOT NULL, unlike `cost_records.owner_id`, + -- which 048 deliberately left nullable so an unattributable sample is still + -- recorded. There is no such thing here: a quantity nobody can be billed for + -- cannot be enforced against anyone, so `recordCosts` drops those samples + -- rather than accumulating a bucket no limit could ever apply to. + owner_id TEXT NOT NULL, + -- CHECK'd, and closed, for the same reason as `cost_records.owner_type`: an + -- owner_type outside this set is unreachable from `resolveBillingSubject`, + -- which maps ProjectEntry's third kind ('agent') onto the user that owns the + -- agent instead of passing it through. Unlike `meter` below, this set does + -- not grow — it is exactly what `BillingSubject` admits. + owner_type TEXT NOT NULL CHECK(owner_type IN ('user','org')), + -- 'YYYY-MM', UTC (PRD Open Question 1). Zero-padded, so lexical order is + -- chronological order and a period range is a plain TEXT comparison. + period TEXT NOT NULL, + -- Deliberately NOT CHECK'd — the one place this table departs from 048's + -- shape, and a decision rather than an omission. `cost_records.kind` carries + -- a closed CHECK from 021, and the bill for it is already recorded in this + -- feature's own scope: `storage_bytes` is a Non-Goal partly because adding a + -- kind would force a 043-style table rebuild. The meter set is expected to + -- grow (`deploys_month` has no recording site yet), so a CHECK here would buy + -- validation that the `MeterKey` union already gives at the only writer, at a + -- cost that has demonstrably bitten once. An unrecognised meter is an inert + -- row nothing reads; an unwritable one is a failed write on a hot path. + meter TEXT NOT NULL, + -- Which provider account paid, carried into the aggregate rather than summed + -- away. An allowance is a limit on what the OPERATOR spends, so enforcement + -- reads the 'platform' rows alone: a project running the gate on its own key + -- costs the operator nothing and must not be charged against a hosted + -- allowance. Collapsing the two here would be a one-way door — this table is + -- not rebuildable from `cost_records` (see above), so any month accumulated + -- without the dimension could never have it recovered, and the usage page + -- could never show a project what its own key paid for. + source TEXT NOT NULL DEFAULT 'platform' CHECK(source IN ('platform','byok')), + -- REAL, matching `cost_records.quantity`, which is what accumulates into it: + -- sandbox milliseconds and character-derived token estimates are not + -- guaranteed integral, and rounding at the aggregate would drift. + -- CHECK'd non-negative, because the UPSERT below adds rather than replaces + -- and nothing else stops a negative delta from REFUNDING the month — through + -- the one statement this whole table exists to make non-refundable. It is + -- reachable without malice: sandbox_ms is wall-clock arithmetic + -- (`Date.now() - startedAt` in post-merge.ts and sandbox-evaluator.ts), so a + -- backwards clock step yields one. Failing the write is correct here; the + -- caller treats accumulation as best-effort and an under-count is the safe + -- direction, where a refund is not. + quantity REAL NOT NULL DEFAULT 0 CHECK(quantity >= 0), + updated_at TEXT NOT NULL, + -- `owner_type` is deliberately outside the key: it describes the owner, it is + -- not part of its identity. In the key it would let one account hold two rows + -- for the same meter and month, and a limit compared against either alone + -- would be wrong. `source` IS in the key for the opposite reason — platform + -- and BYOK spend are two separately-answerable questions, and one limit + -- applies to only one of them. The key doubles as the UPSERT's conflict + -- target (src/storage/usage.ts), and SQLite's implicit index on it already + -- serves the owner+period lookup, so no separate index is added. (It is not a + -- *covering* index — `quantity` and `updated_at` sit outside the key — but the + -- search is on the key, which is what matters here.) + PRIMARY KEY (owner_id, period, meter, source) +); diff --git a/package-lock.json b/package-lock.json index 5cdf67f7..3dd28338 100644 --- a/package-lock.json +++ b/package-lock.json @@ -7,7 +7,7 @@ "": { "name": "stratum", "version": "0.2.0", - "license": "MIT", + "license": "AGPL-3.0-or-later", "dependencies": { "@isomorphic-git/lightning-fs": "^4.6.2", "diff": "^9.0.0", diff --git a/src/billing/enforcement.ts b/src/billing/enforcement.ts new file mode 100644 index 00000000..864a0fd2 --- /dev/null +++ b/src/billing/enforcement.ts @@ -0,0 +1,620 @@ +/** + * Consulting entitlements: the one place a limit turns into a decision. + * + * `entitlements.ts` defines and fetches allowances and enforces nothing. This + * module is the other half — it evaluates a decision, records it, and hands the + * call site an answer. Every metered call site goes through {@link checkMeter} + * or {@link checkGauge} rather than reading a limit itself, so "what does the + * plan allow", "what has been used", "who is this checked against" and "does + * the answer bind" are decided once instead of five times. + * + * ## Everything here ships observe-only + * + * A decision is always *evaluated* and *recorded*; it is *binding* only when + * `ENTITLEMENTS_ENFORCE=1`. Under observe-only a refusal still reserves — the + * meter is told to count what it refuses ({@link checkMeter}) — still logs, and + * still admits. That is the month of measurement PRD §8 asks for before anything + * blocks, and the counting half is what makes it worth a month: a counter that + * froze at the limit could report THAT people would be blocked and never BY HOW + * MUCH, which is the number the whole exercise exists to produce. `admitted` is + * what a caller acts on; `refused` is what the decision was, and the two differ + * exactly when enforcement is off. + * + * ## Inert with `BILLING_SERVICE_URL` unset + * + * The first line of every entry point is `entitlementsEnabled`. With the billing + * vars unset — every self-hoster — nothing here reads entitlements, nothing + * touches the `USAGE_METER` binding, and no refusal can be produced. + * `tests/enforcement.test.ts` asserts the binding is never touched in that + * configuration, because "inert by default" is a claim that decays silently. + * + * ## The subject (PRD §4a) + * + * The subject a limit is CHECKED against is the acting user, not the subject the + * spend is RECORDED against. See {@link resolveEnforcementSubject}. + */ +import { + type MeterReserveOutcome, + type RateMeterKey, + type UsageMeterKey, + rateWindowSeconds, + usageMeterName, +} from "../queue/usage-meter"; +import { resolveBillingSubject } from "../storage/costs"; +import { type MeterKey, getOwnerMeterTotals, isMeterKey, usagePeriod } from "../storage/usage"; +import type { Env } from "../types"; +import type { Logger } from "../utils/logger"; +import { + type CountKey, + type Entitlements, + RemoteEntitlements, + UNLIMITED, + UnlimitedEntitlements, + entitlementsEnabled, + warmEntitlements, +} from "./entitlements"; + +/** Who a limit is checked against. Not necessarily who the spend is billed to. */ +export interface EnforcementSubject { + ownerId: string; + ownerType: "user" | "org"; + /** + * True only when this subject is an org that is *positively known* to pool + * its members' usage on a paid plan. Never inferred; see + * {@link resolveEnforcementSubject}. + */ + pooled: boolean; +} + +/** What one check decided, and whether that decision binds. */ +export interface EnforcementDecision { + /** + * Whether the caller may proceed. Under observe-only this is `true` even for + * a refusal — gate on this, and report {@link EnforcementDecision.refused}. + */ + admitted: boolean; + /** What the limit says, regardless of whether it binds. */ + refused: boolean; + /** Whether `ENTITLEMENTS_ENFORCE=1` made this decision binding. */ + enforcing: boolean; + /** Whether a limit was consulted at all (false when the seam is inert). */ + checked: boolean; + /** Whether a reservation was taken and therefore needs settling. */ + reserved: boolean; + /** The limit consulted: `-1` unlimited, `0` a hard block, else a ceiling. */ + limit: number; + /** The subject's counter after this check, as the meter reported it. */ + count: number; + /** + * Present when refused: the user- and agent-facing copy naming what is + * exhausted, when it resets and both remedies (PRD §4c). Never a bare error + * string — this is the highest-intent moment in the funnel. + */ + reason?: string; +} + +/** Every entry point short-circuits to this when the seam is off. */ +function inert(): EnforcementDecision { + return { + admitted: true, + refused: false, + enforcing: false, + checked: false, + reserved: false, + limit: UNLIMITED, + count: 0, + }; +} + +/** + * Whether a decision, once evaluated, is allowed to refuse anything. + * + * Two switches, not one (PRD Goal 3): metering is always on, enforcement is + * opt-in on top of a configured billing service. `ENTITLEMENTS_ENFORCE=1` with + * no service is a misconfiguration `entitlementsConfigError` reports; here it + * simply cannot bind, because there is no limit to bind to. + */ +export function enforcementBinding(env: Env): boolean { + return entitlementsEnabled(env) && env.ENTITLEMENTS_ENFORCE === "1"; +} + +/** + * The subject a limit is checked against (PRD §4a). + * + * **The acting user, not the project's owner.** A project in an org namespace is + * recorded against the org, and an org costs nothing to create and needs only + * *write* access to put a project in — so an allowance keyed on the recorded + * owner is one an exhausted user resets by making an org, or by being added to + * someone else's. Keying the check on the person who ran the evaluation makes + * the allowance follow the person: ten orgs do not help, and nobody can drain an + * allowance that is not theirs. + * + * **An org pools only when positively known to be paid**, which is the one place + * this PRD does not fail open, and the direction matters. `forOwner` is a cached + * read that never fetches and nothing warms an org's entitlements before the + * first project of that org is evaluated, so an org's plan is ALWAYS a miss on + * first contact. Reading "unknown" as "own subject" would therefore make every + * org its own permanently-unlimited allowance — the hole, reopened by the + * safety mechanism. Unknown or uncached means free, and the actor is charged. + * The cost of that direction is that during a billing outage a paying org's + * members fall back to their personal limits: a degradation, and recoverable, + * where the alternative is permanent. + * + * The org's entitlements are warmed here via `waitUntil` — the auth middleware + * knows the caller but not the project, so it cannot do it — which is why the + * first check for a given org resolves to the actor and a later one can pool. + * + * **With no actor at all** — the queue consumers — the recorded subject is the + * only nameable one, and it is usable ONLY when it is a person. An org that is + * not positively known to be paid must never become its own subject here: it + * would get a counter of its own AND, since nothing warms an org's plan on that + * path, a permanent cache miss resolving to `UnlimitedEntitlements` — a fresh, + * permanently unlimited allowance per org, which is verbatim the hole §4a + * exists to close. There is no safe substitute either: charging `orgs.owner_id` + * is the aggregation §4a examined and rejected, because it hands an exhausted + * member a fresh root and drains one person's allowance for a whole org. So the + * check is reported UNAVAILABLE (`null`) and logged, rather than answered + * wrongly. The cost is stated rather than hidden: an org-owned operation that + * names nobody — today only a merge-triggered deploy — is not volume-checked + * until either the org's plan is known to pool or the call site threads an + * actor. + * + * @param opts.actorUserId - The user who ran the operation, when the site knows one + * @param opts.owner - The project's recorded owner (`agent` walks to its user) + * @param opts.waitUntil - Used to warm an org's plan off the response path + * @returns The subject, or `null` when nobody can be named + */ +export async function resolveEnforcementSubject( + env: Env, + logger: Logger, + opts: { + actorUserId?: string; + owner?: { ownerId?: string; ownerType?: "user" | "org" | "agent" }; + waitUntil?: (promise: Promise) => void; + }, +): Promise { + const { actorUserId, owner, waitUntil } = opts; + + if (owner?.ownerType === "org" && owner.ownerId) { + const orgId = owner.ownerId; + if (entitlementsEnabled(env)) { + const resolved = await new RemoteEntitlements(env, logger).forOwner(orgId, "org"); + // `source: "default"` is "we have never heard about this org", not "this + // org is on the default plan" — pooling on it would pool on ignorance. + const pooled = + resolved.success && resolved.data.source !== "default" && resolved.data.entitlements.pooled; + if (pooled) { + return { ownerId: orgId, ownerType: "org", pooled: true }; + } + // Warmed for NEXT time, never awaited: a paid org pools from its second + // evaluation onward, and the first one is charged to the actor. + warmEntitlements(env, waitUntil, { ownerId: orgId, ownerType: "org" }, logger); + } + } + + if (actorUserId) return { ownerId: actorUserId, ownerType: "user", pooled: false }; + + // No actor: a queue consumer, or a path that does not carry one. The recorded + // subject is then the only nameable one, and only when it names a PERSON. + if (!owner?.ownerId) return null; + // Nothing below this line can produce a decision when the seam is off, so the + // D1 hop the agent walk costs is not one a self-hoster should pay for. + if (!entitlementsEnabled(env)) return null; + // `resolveBillingSubject` also performs the agent -> owner walk, so an + // agent-owned project resolves to the person who owns the agent. + const recorded = await resolveBillingSubject(env.DB, logger, owner); + if (!recorded) return null; + if (recorded.ownerType === "user") return { ...recorded, pooled: false }; + + // An org, with nobody to charge and no positive signal that it pools. Never + // inferred into a subject of its own (see this function's doc comment): the + // check is simply unavailable, and saying so is what keeps it from looking + // like an allowance that was checked and passed. + logger.warn("Entitlement check unavailable: an org-owned operation named no actor", { + ownerId: recorded.ownerId, + ownerType: recorded.ownerType, + }); + return null; +} + +/** + * The subject's allowances, from the cached read only. + * + * Never fetches: this runs on the request path, where a cold billing service + * would put a five-second dependency in front of a merge. A miss resolves to + * `UnlimitedEntitlements`, which admits — the fail-open rule of the module it + * reads from. + */ +async function entitlementsFor( + env: Env, + logger: Logger, + subject: EnforcementSubject, +): Promise { + const resolved = await new RemoteEntitlements(env, logger).forOwner( + subject.ownerId, + subject.ownerType, + ); + if (resolved.success) return resolved.data.entitlements; + // Fails open, but never silently: admitting because the limits could not be + // read is a different fact from admitting because the plan allows it, and a + // month of observation that cannot tell them apart measures nothing. + logger.warn("Entitlements unreadable; admitting against unlimited defaults", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + error: resolved.error.message, + }); + return UnlimitedEntitlements; +} + +/** + * Subjects whose D1 aggregate is being, or has been, reconciled into their + * counter in THIS isolate, keyed `subjectType:subjectId:period`. + * + * The counter is authoritative for enforcement but it is not the ledger: a + * subject that spent before the meter existed, or whose object was purged, or + * whose spend was recorded by a path that never reserved, has a `usage_periods` + * total the object has never seen. `setFloor` raises the counter to it (never + * lowers — see the object's doc comment), and doing that once per isolate keeps + * the D1 read off every single evaluation. + * + * The value is the in-flight PROMISE, not a bare marker, and that is the whole + * correctness of it: a synchronous "remembered" flag lets a second request for + * the same subject sail past and reserve against a counter the floor has not + * landed in yet. Concurrent callers await the same reconcile instead. + */ +const reconciled = new Map>(); + +/** + * How many subject-periods one isolate remembers having reconciled. Bounded + * because an isolate is long-lived and this map is otherwise append-only: a + * busy instance would keep an entry per subject for the life of the process. + * Oldest out first (a `Map` iterates in insertion order); the only cost of + * forgetting one is a repeat of a read that raises nothing. + */ +const RECONCILE_MEMO_LIMIT = 1_000; + +/** + * Raise the subject's counters to the D1 aggregate, at most once per isolate — + * and only once it has actually SUCCEEDED. + * + * A failed reconcile is forgotten rather than remembered: memoizing before the + * read returned meant one D1 blip disabled the floor for that subject for the + * life of the isolate, so a counter behind the ledger stayed behind it and the + * subject spent the difference twice. + * + * `source = "platform"` is not optional here and not a detail: migration 049 + * keys usage on `(owner_id, period, meter, source)` exactly so a project paying + * its own provider bill is not charged against the hosted allowance, and a sum + * that omitted the filter would hand that distinction straight back (Goal 5). + */ +async function reconcileFloor( + env: Env, + logger: Logger, + meterNamespace: NonNullable, + subject: EnforcementSubject, + period: string, +): Promise { + const key = `${subject.ownerType}:${subject.ownerId}:${period}`; + const inFlight = reconciled.get(key); + if (inFlight) { + await inFlight; + return; + } + const run = applyFloor(env, logger, meterNamespace, subject, period); + reconciled.set(key, run); + if (reconciled.size > RECONCILE_MEMO_LIMIT) { + const oldest = reconciled.keys().next(); + if (!oldest.done) reconciled.delete(oldest.value); + } + if (!(await run)) reconciled.delete(key); +} + +/** + * The reconcile itself. Never rejects: a floor that could not be applied leaves + * an allowance read too low, which is a billing problem, where a rejection here + * would be a merge that failed because of metering. + * + * @returns Whether the floor landed, so the caller knows whether to remember it + */ +async function applyFloor( + env: Env, + logger: Logger, + meterNamespace: NonNullable, + subject: EnforcementSubject, + period: string, +): Promise { + if (!env.DB) return false; + try { + const totals = await getOwnerMeterTotals(env.DB, logger, subject.ownerId, period, "platform"); + if (!totals.success) return false; + const stub = meterNamespace.get( + meterNamespace.idFromName(usageMeterName(subject.ownerType, subject.ownerId)), + ); + for (const [meter, quantity] of Object.entries(totals.data)) { + // The key set is validated by the read, so this is a narrowing and not a + // trust decision; anything else in the table is not a meter this build + // has a limit for, and must not be written into the counter. + if (isMeterKey(meter) && typeof quantity === "number") { + await stub.setFloor(meter, quantity, period); + } + } + return true; + } catch (error) { + logger.warn("Usage floor could not be reconciled; the counter may lag the ledger", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + period, + error: error instanceof Error ? error.message : String(error), + }); + return false; + } +} + +export interface MeterCheckOptions { + subject: EnforcementSubject; + meter: UsageMeterKey; + /** Upper bound on what this operation will consume. Settled afterwards. */ + estimate: number; + /** Caller's clock, so one operation's counters cannot straddle a boundary. */ + nowMs?: number; + /** 'YYYY-MM' UTC; defaults to the month `nowMs` falls in. */ + period?: string; + /** What is being gated, for the refusal copy (e.g. "AI review"). */ + what: string; +} + +/** + * Evaluate one metered limit for one subject: reserve an upper bound, record the + * decision, and admit unless enforcement is on and the bound does not fit. + * + * The reservation is the point. An LLM call's true cost is known only from its + * response, so checking a counter and then spending would be check-then-act + * against a shared total; `reserve` admits or refuses the whole bound + * atomically, and {@link settleMeter} hands back the difference. + * + * A BINDING refusal takes no reservation — the meter does not count what it + * actually stops — so `reserved` is what the meter reports it counted, not what + * the limit said. Under observe-only a refusal IS counted, because the spend + * goes ahead regardless and an uncounted one is a measurement thrown away. + */ +export async function checkMeter( + env: Env, + logger: Logger, + opts: MeterCheckOptions, +): Promise { + if (!entitlementsEnabled(env)) return inert(); + const meterNamespace = env.USAGE_METER; + if (!meterNamespace) { + // Configured to enforce with nothing to count against. Logged rather than + // refusing: `entitlementsConfigError` is where this gets fixed, and a + // missing binding must not become a merge outage. + logger.warn("Entitlement check skipped: USAGE_METER binding is not configured", { + meter: opts.meter, + }); + return inert(); + } + + const nowMs = opts.nowMs ?? Date.now(); + const period = opts.period ?? usagePeriod(new Date(nowMs)); + const entitlements = await entitlementsFor(env, logger, opts.subject); + const limit = limitFor(entitlements, opts.meter); + + if (isMonthlyMeter(opts.meter)) { + await reconcileFloor(env, logger, meterNamespace, opts.subject, period); + } + + const stub = meterNamespace.get( + meterNamespace.idFromName(usageMeterName(opts.subject.ownerType, opts.subject.ownerId)), + ); + const enforcing = enforcementBinding(env); + // Observe-only counts what it refuses. The refusal does not bind, so the + // spend it did not stop still happens; a counter that froze at the limit + // would leave the measurement period unable to answer the one question it + // exists for — by how much would this subject have gone over? + let outcome: MeterReserveOutcome; + try { + outcome = await stub.reserve(opts.meter, opts.estimate, limit, period, nowMs, { + countRefused: !enforcing, + }); + } catch (error) { + // Fails open, for the reason the whole seam does: a meter that cannot be + // reached is a billing outage, and a billing outage that blocks a paying + // customer's merge is a worse failure than a spend that went uncounted. + // Reported, never silent — `checked: false` keeps an unmeasured admission + // out of the observe-only numbers rather than passing it off as a fit. + logger.warn("Usage meter unreachable; admitting without a reservation", { + subjectId: opts.subject.ownerId, + subjectType: opts.subject.ownerType, + meter: opts.meter, + error: error instanceof Error ? error.message : String(error), + }); + return inert(); + } + const refused = !outcome.admitted; + + const decision: EnforcementDecision = { + admitted: outcome.admitted || !enforcing, + refused, + enforcing, + checked: true, + reserved: outcome.counted, + limit, + count: outcome.count, + }; + if (refused) { + decision.reason = meterRefusalReason(opts.what, opts.meter, limit, period, nowMs); + } + record(logger, opts.subject, opts.meter, decision, entitlements.plan); + return decision; +} + +/** + * Apply the true cost of a spend a {@link checkMeter} reservation covered. + * + * Call it in a `finally`. A provider call that throws has still consumed the + * reservation, and leaving it standing would charge the subject the upper bound + * for a call that never produced a token — a few of those and the allowance is + * gone with nothing to show for it. + */ +export async function settleMeter( + env: Env, + logger: Logger, + opts: { + subject: EnforcementSubject; + meter: UsageMeterKey; + /** Signed difference between the true cost and the reserved bound. */ + delta: number; + nowMs?: number; + period?: string; + }, +): Promise { + if (!entitlementsEnabled(env)) return; + const meterNamespace = env.USAGE_METER; + if (!meterNamespace) return; + if (!Number.isFinite(opts.delta) || opts.delta === 0) return; + const nowMs = opts.nowMs ?? Date.now(); + const period = opts.period ?? usagePeriod(new Date(nowMs)); + const stub = meterNamespace.get( + meterNamespace.idFromName(usageMeterName(opts.subject.ownerType, opts.subject.ownerId)), + ); + try { + await stub.settle(opts.meter, opts.delta, period, nowMs); + } catch (error) { + // Callers settle in a `finally`, so a throw here would replace whatever the + // block was returning — including a successful evaluation — with a metering + // error. The reservation stands uncorrected instead, which overcharges by + // the difference and is the cheaper of the two failures. + logger.warn("Usage meter unreachable; reservation left unsettled", { + subjectId: opts.subject.ownerId, + subjectType: opts.subject.ownerType, + meter: opts.meter, + delta: opts.delta, + error: error instanceof Error ? error.message : String(error), + }); + } +} + +/** + * Evaluate a counted gauge — a thing that exists rather than a thing that is + * spent, so there is nothing to reserve and nothing to settle. + * + * @param opts.current - How many the subject already has, counted by the caller + */ +export async function checkGauge( + env: Env, + logger: Logger, + opts: { subject: EnforcementSubject; count: CountKey; current: number; what: string }, +): Promise { + if (!entitlementsEnabled(env)) return inert(); + const entitlements = await entitlementsFor(env, logger, opts.subject); + const limit = entitlements.counts[opts.count]; + const enforcing = enforcementBinding(env); + // Same reading of a limit as the meter: `-1` unlimited, `0` a hard block, + // anything not a usable limit admits rather than blocking on a bad payload. + const usable = Number.isInteger(limit) && limit >= 0; + const refused = usable && opts.current + 1 > limit; + + const decision: EnforcementDecision = { + admitted: !refused || !enforcing, + refused, + enforcing, + checked: true, + reserved: false, + limit, + count: opts.current, + }; + if (refused) decision.reason = gaugeRefusalReason(opts.what, limit); + record(logger, opts.subject, opts.count, decision, entitlements.plan); + return decision; +} + +/** + * Every decision is recorded, admitted or not. + * + * This is what makes observe-only worth a month: a log line per decision, with + * the plan and whether the decision bound, is the difference between measuring + * limits and guessing at them. + */ +function record( + logger: Logger, + subject: EnforcementSubject, + meter: string, + decision: EnforcementDecision, + plan: string, +): void { + logger.info("Entitlement decision", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + pooled: subject.pooled, + plan, + meter, + limit: decision.limit, + count: decision.count, + refused: decision.refused, + enforcing: decision.enforcing, + admitted: decision.admitted, + }); +} + +function isMonthlyMeter(meter: UsageMeterKey): meter is MeterKey { + return meter.endsWith("_month"); +} + +function limitFor(entitlements: Entitlements, meter: UsageMeterKey): number { + return isMonthlyMeter(meter) + ? entitlements.meters[meter] + : entitlements.rates[meter as RateMeterKey]; +} + +/** First instant of the month after `period`, ISO 8601 — when a monthly meter frees up. */ +export function periodResetsAt(period: string): string { + const match = /^(\d{4})-(\d{2})$/.exec(period); + if (!match) return "the start of next month"; + const year = Number(match[1]); + const month = Number(match[2]); + return new Date(Date.UTC(year, month, 1)).toISOString(); +} + +/** + * The copy a blocked caller reads (PRD §4c, §8). + * + * It is product copy, not an error message: this is the moment somebody learns + * a limit exists, and an agent over `/mcp` gets only this string. So it names + * what is exhausted, when it resets, and BOTH ways out — and for the rate meter + * it says plainly that a project's own key is not one of them, because that is + * the remedy an agent would otherwise reach for first. + */ +function meterRefusalReason( + what: string, + meter: UsageMeterKey, + limit: number, + period: string, + nowMs: number, +): string { + if (meter === "evaluations_per_hour") { + const resetsAt = new Date(nowMs + rateWindowSeconds(meter) * 1000).toISOString(); + return `${what} could not run: this account has used its hourly evaluation allowance (${limit} per hour). Capacity frees up as the window rolls, and in full by ${resetsAt}. Two ways forward: wait for the window, or raise this account's plan limits. Bringing your own provider key does NOT lift this one — it bounds how often an evaluation runs, not whose tokens it spends.`; + } + const remedy = + meter === "llm_tokens_month" + ? "set `provider:` on the llm evaluator in .stratum/policy.yaml to run reviews on this project's own API key" + : "reduce this month's usage"; + return `${what} could not run: this account's ${meterLabel(meter)} for ${period} is used up (limit ${limit}). It resets at ${periodResetsAt(period)}. Two ways forward: ${remedy}, or raise this account's plan limits.`; +} + +function gaugeRefusalReason(what: string, limit: number): string { + return `${what} is not allowed: this account's plan allows ${limit} of them. Two ways forward: remove or make public one you already have, or raise this account's plan limits.`; +} + +function meterLabel(meter: UsageMeterKey): string { + switch (meter) { + case "llm_tokens_month": + return "monthly LLM token allowance"; + case "sandbox_ms_month": + return "monthly sandbox time allowance"; + case "deploys_month": + return "monthly deploy allowance"; + default: + return meter; + } +} diff --git a/src/billing/entitlements.ts b/src/billing/entitlements.ts new file mode 100644 index 00000000..65de6879 --- /dev/null +++ b/src/billing/entitlements.ts @@ -0,0 +1,496 @@ +/** + * Entitlements — what one billing subject's plan allows (Stratum Cloud only). + * + * This is a cloud-only hook living in the AGPL tree, in the shape `src/beta/gate.ts` + * established: the plan service itself lives in the cloud layer, core just asks it. + * When `BILLING_SERVICE_URL` / `BILLING_SERVICE_SECRET` are unset — the default for + * every OSS self-hoster — every function here is inert and every owner resolves to + * `UnlimitedEntitlements`, which is exactly today's behavior. + * + * Three key sets, deliberately not one union, because they are not one kind of + * thing and do not live in one place: a `MeterKey` is a monthly *flow* accumulated + * in `usage_periods`, a `CountKey` is a *gauge* counted on demand, and a `RateKey` + * is a *rate* over a short window, enforced out of KV or off the usage meter but + * never out of the monthly aggregate. Collapsing them invites a limit to be + * looked up from the wrong store. + * + * ## Fail OPEN — deliberately the opposite of the beta gate + * + * `validateInviteCode` (`src/beta/gate.ts`) fails CLOSED on timeout, non-OK or a + * parse failure: a signup wall that opens under load is not a wall. Entitlements + * do the reverse. On any failure to reach or understand the billing service we + * serve the cached value, and with no cache `UnlimitedEntitlements`. A billing + * outage that blocks a paying customer's merge is strictly worse than minutes of + * unmetered usage; the hole is bounded by the cache TTL and is an accepted cost. + * + * Do not "correct" this to match the gate. The two hooks look alike and want + * opposite failure modes, which is why both say so in their own doc comments. + * + * ## Reading a limit + * + * `-1` means unlimited. **`0` means a hard block** — not "unset", not "unlimited"; + * an owner whose limit is `0` may not consume that resource at all. Both readings + * of `0` exist in the wild, so this file fixes one and callers can rely on it. + * Any other value that is not a non-negative integer — a string, `NaN`, `1.5`, + * `-7` — is DISCARDED in favor of the cached or default value for that field, and + * never coerced: a limit derived from garbage is worse than a limit we admit we + * do not have. + * + * NOTHING here enforces anything. Defining and fetching entitlements is separate + * from consulting them (PRD §8) precisely so the fetch can be observed for a month + * before it decides anything. + */ +import { METER_KEYS, type MeterKey } from "../storage/usage"; +import type { Env } from "../types"; +import { type AppError, ValidationError } from "../utils/errors"; +import type { Logger } from "../utils/logger"; +import { type Result, err, ok } from "../utils/result"; + +/** A gauge counted on demand, not a monthly flow — there is no `private_projects` meter. */ +export type CountKey = "private_projects"; + +/** + * A bound on how often something may happen, rather than on how much it spends. + * + * The two members are enforced in different places on purpose, and the split is + * the one PRD §4b argues: `requests_per_minute` stays on the KV request limiter, + * where the bucket rolls as fast as KV goes stale, and `evaluations_per_hour` + * lives on the `UsageMeter` Durable Object, because an hour-long window read + * through a per-colo cache that is a minute stale is not a limit at all. + */ +export type RateKey = "requests_per_minute" | "evaluations_per_hour"; + +/** + * Every meter a limit may be set on. Re-exported from the storage layer that + * defines `MeterKey` rather than restated here: two lists that could drift are + * two answers to "which meters exist", and `parseEntitlements` reads this one. + */ +export { METER_KEYS }; + +export const COUNT_KEYS: readonly CountKey[] = ["private_projects"]; + +export const RATE_KEYS: readonly RateKey[] = ["requests_per_minute", "evaluations_per_hour"]; + +/** + * One billing subject's allowances. + * + * Every limit is `-1` (unlimited), `0` (hard block) or a positive integer. The + * record is total over each key set on purpose: a missing key would make "no + * limit configured" and "limit of zero" indistinguishable at the call site. + */ +export interface Entitlements { + /** Opaque plan identifier as the billing service names it; for display and logs. */ + plan: string; + /** + * Whether an ORG on this plan pools its members' usage onto the org itself + * (PRD §4a). + * + * Explicit rather than derived from {@link Entitlements.plan}, which is an + * opaque display string: pooling decides WHOSE allowance a spend is checked + * against, and inferring that from a name the billing service is free to + * change is how a rename becomes a free-usage hole. + * + * It defaults to `false`, and that default is load bearing — it is the one + * place this file does not fail open. `forOwner` never fetches and org + * entitlements are never warmed by the auth path, so an org's plan is always + * a cache miss on FIRST contact; reading an unknown plan as "pooled" would + * make every org its own unlimited subject permanently, which is exactly the + * hole the enforcement subject exists to close. Unknown therefore means free, + * and the acting user is charged instead. + */ + pooled: boolean; + meters: Record; + counts: Record; + rates: Record; +} + +/** + * Where a resolved value came from. + * + * Carried because observe-only telemetry that cannot tell "unlimited because that + * is the plan" from "unlimited because the billing service timed out" cannot + * measure anything: a month of measurement would be indistinguishable from a + * month of outage. + */ +export type EntitlementsSource = "remote" | "cache" | "default"; + +export interface EntitlementsResolution { + entitlements: Entitlements; + source: EntitlementsSource; +} + +export interface EntitlementsProvider { + /** + * The allowances in force for one billing subject. + * + * A `Result` rather than a throw per AGENTS.md ("errors are values"). Because + * resolution fails open, the error arm is reserved for a caller mistake — an + * unidentified subject — which must not silently resolve to "unlimited". + */ + forOwner( + ownerId: string, + ownerType: "user" | "org", + ): Promise>; +} + +/** The value of a limit that is not a limit. */ +export const UNLIMITED = -1; + +/** The plan name reported when nobody has told us a real one. */ +const DEFAULT_PLAN = "unlimited"; + +/** + * The default in every sense: what a self-hoster always gets, and what a cloud + * request falls back to when the billing service cannot be reached and nothing + * is cached. Frozen because it is shared by every such resolution. + */ +export const UnlimitedEntitlements: Entitlements = Object.freeze({ + plan: DEFAULT_PLAN, + meters: Object.freeze({ + llm_tokens_month: UNLIMITED, + sandbox_ms_month: UNLIMITED, + deploys_month: UNLIMITED, + }), + counts: Object.freeze({ private_projects: UNLIMITED }), + rates: Object.freeze({ requests_per_minute: UNLIMITED, evaluations_per_hour: UNLIMITED }), + // Never pooled by default: "we do not know this org's plan" must not read as + // "this org pools", or an unknown org becomes a fresh allowance. See the + // field's own doc comment. + pooled: false, +}); + +/** + * True only when the service is configured AND we hold a credential for it — + * mirroring `betaGateEnabled`. A URL without a secret is a misconfiguration that + * would produce a request the service rejects on every single hit, so it counts + * as off rather than as half-on. + */ +export function entitlementsEnabled(env: Env): boolean { + return !!env.BILLING_SERVICE_URL && !!env.BILLING_SERVICE_SECRET; +} + +/** + * Cap how long a warm can wait on the billing service. This runs off the response + * path via `waitUntil`, so the timeout is not protecting a user's latency — it is + * bounding how long a Worker invocation is kept alive by a hung dependency. + */ +const BILLING_TIMEOUT_MS = 5000; + +/** Positive cache lifetime (PRD §4). Consequence: an upgrade takes effect up to this late. */ +const CACHE_TTL_SECONDS = 300; + +/** + * Negative-cache and lock lifetime. Both are "as short as possible": KV's minimum + * `expirationTtl` is 60 seconds, so this is the floor, not a tuned value. The + * negative entry keeps a hard outage from being re-fetched once per request; the + * lock keeps a cold cache from stampeding the service. + */ +const FAILURE_TTL_SECONDS = 60; + +/** + * Versioned so a shape change cannot be read back as the old shape. + * + * Bumped to v2 by the enforcement work: `pooled` and `evaluations_per_hour` + * joined the payload, and a v1 entry read back through the new parser would + * silently take the defaults for both — which for `pooled` is the safe answer + * and for `evaluations_per_hour` is "unlimited". Neither is a value anybody + * chose, so the old entries are abandoned rather than reinterpreted. The + * billing service's response contract changes with it (private cloud layer). + */ +const KEY_PREFIX = "entitlements:v2:"; + +function cacheKey(ownerType: string, ownerId: string): string { + return `${KEY_PREFIX}${ownerType}:${ownerId}`; +} + +function negativeKey(ownerType: string, ownerId: string): string { + return `${KEY_PREFIX}neg:${ownerType}:${ownerId}`; +} + +function lockKey(ownerType: string, ownerId: string): string { + return `${KEY_PREFIX}lock:${ownerType}:${ownerId}`; +} + +function serviceUrl(env: Env, path: string): string { + return `${(env.BILLING_SERVICE_URL ?? "").replace(/\/$/, "")}${path}`; +} + +/** + * One limit, or `null` when the service sent something that is not one. + * + * `Number.isInteger` rejects strings, `NaN` and both infinities in one check, so + * a JSON `"100"` is discarded rather than becoming 100 — a limit we invented is + * indistinguishable, downstream, from one the customer is paying for. + */ +function parseLimit(value: unknown): number | null { + if (!Number.isInteger(value)) return null; + const limit = value as number; + if (limit < 0 && limit !== UNLIMITED) return null; + return limit; +} + +/** + * Build entitlements field by field, keeping `fallback`'s value for anything the + * payload does not supply validly. + * + * Field by field rather than all-or-nothing so one bad key does not throw away + * the rest of a customer's plan — and never partially: every key is present in + * the result because it was present in `fallback`. + */ +function parseEntitlements(payload: unknown, fallback: Entitlements): Entitlements { + const record = (typeof payload === "object" && payload !== null ? payload : {}) as Record< + string, + unknown + >; + const readGroup = ( + group: unknown, + keys: readonly K[], + defaults: Record, + ): Record => { + const source = (typeof group === "object" && group !== null ? group : {}) as Record< + string, + unknown + >; + const out = {} as Record; + for (const key of keys) { + out[key] = parseLimit(source[key]) ?? defaults[key]; + } + return out; + }; + + const plan = typeof record.plan === "string" && record.plan.trim() ? record.plan.trim() : null; + return { + plan: plan ?? fallback.plan, + // Only a real boolean pools. A truthy string or a 1 is a payload we do not + // understand, and the safe reading of that is "not positively known". + pooled: typeof record.pooled === "boolean" ? record.pooled : fallback.pooled, + meters: readGroup(record.meters, METER_KEYS, fallback.meters), + counts: readGroup(record.counts, COUNT_KEYS, fallback.counts), + rates: readGroup(record.rates, RATE_KEYS, fallback.rates), + }; +} + +/** + * Entitlements from the cloud billing service, cached in KV. + * + * `forOwner` is a CACHED READ and never touches the network: it is called on the + * request path, and a cold cache there would put a 5-second dependency in front + * of a merge. The network fetch happens in `refresh`, which `warmEntitlements` + * runs off the response path after auth — that is what makes the cached read hit. + */ +export class RemoteEntitlements implements EntitlementsProvider { + constructor( + private readonly env: Env, + private readonly logger: Logger, + ) {} + + async forOwner( + ownerId: string, + ownerType: "user" | "org", + ): Promise> { + if (!ownerId) { + // Not a fail-open case: an unidentified subject is a bug in the caller, and + // answering "unlimited" would hide it behind exactly the behavior it breaks. + return err(new ValidationError("Entitlements requested without an owner id", { ownerType })); + } + if (!entitlementsEnabled(this.env)) { + return ok({ entitlements: UnlimitedEntitlements, source: "default" }); + } + const cached = await this.readCache(ownerId, ownerType); + if (cached) return ok({ entitlements: cached, source: "cache" }); + return ok({ entitlements: UnlimitedEntitlements, source: "default" }); + } + + /** + * Ask the billing service and populate the cache. + * + * Fails open (see the module doc comment): a timeout, a non-OK status or an + * unparseable body all serve the cached value, or `UnlimitedEntitlements` when + * there is none, and record a negative-cache entry so an outage is not + * re-fetched once per request. + */ + async refresh( + ownerId: string, + ownerType: "user" | "org", + ): Promise> { + if (!ownerId) { + return err(new ValidationError("Entitlements refresh without an owner id", { ownerType })); + } + if (!entitlementsEnabled(this.env)) { + return ok({ entitlements: UnlimitedEntitlements, source: "default" }); + } + + const cached = await this.readCache(ownerId, ownerType); + const fallback: EntitlementsResolution = cached + ? { entitlements: cached, source: "cache" } + : { entitlements: UnlimitedEntitlements, source: "default" }; + + try { + const url = `${serviceUrl(this.env, "/api/billing/entitlements")}?ownerId=${encodeURIComponent( + ownerId, + )}&ownerType=${encodeURIComponent(ownerType)}`; + const res = await fetch(url, { + headers: { Authorization: `Bearer ${this.env.BILLING_SERVICE_SECRET ?? ""}` }, + signal: AbortSignal.timeout(BILLING_TIMEOUT_MS), + }); + if (!res.ok) { + this.logger.warn("Entitlements lookup returned non-OK; failing open", { + status: res.status, + ownerId, + source: fallback.source, + }); + await this.markFailure(ownerId, ownerType); + return ok(fallback); + } + const entitlements = parseEntitlements(await res.json(), fallback.entitlements); + await this.writeCache(ownerId, ownerType, entitlements); + return ok({ entitlements, source: "remote" }); + } catch (error) { + // Never swallowed: an outage that is invisible is one nobody fixes, and the + // free usage it hands out is only acceptable while it is being watched. + this.logger.error( + "Entitlements lookup failed; failing open", + error instanceof Error ? error : undefined, + { ownerId, ownerType, source: fallback.source }, + ); + await this.markFailure(ownerId, ownerType); + return ok(fallback); + } + } + + private async readCache(ownerId: string, ownerType: string): Promise { + if (!this.env.STATE) return null; + try { + const raw = await this.env.STATE.get(cacheKey(ownerType, ownerId)); + if (!raw) return null; + // Re-validated on the way out, not trusted because we wrote it: the cache + // outlives deploys, so an entry written by an older shape must degrade to + // the default rather than be cast into a limit nobody validated. + return parseEntitlements(JSON.parse(raw), UnlimitedEntitlements); + } catch (error) { + this.logger.warn("Entitlements cache read failed; treating as a miss", { + ownerId, + error: error instanceof Error ? error.message : String(error), + }); + return null; + } + } + + private async writeCache( + ownerId: string, + ownerType: string, + entitlements: Entitlements, + ): Promise { + if (!this.env.STATE) return; + try { + await this.env.STATE.put(cacheKey(ownerType, ownerId), JSON.stringify(entitlements), { + expirationTtl: CACHE_TTL_SECONDS, + }); + await this.env.STATE.delete(negativeKey(ownerType, ownerId)); + } catch (error) { + this.logger.warn("Entitlements cache write failed", { + ownerId, + error: error instanceof Error ? error.message : String(error), + }); + } + } + + /** + * Record that the service just failed, in its OWN key: a failure must never + * overwrite a good cached plan, or one blip would demote a paying customer to + * the default for a full TTL. + */ + private async markFailure(ownerId: string, ownerType: string): Promise { + if (!this.env.STATE) return; + try { + await this.env.STATE.put(negativeKey(ownerType, ownerId), String(Date.now()), { + expirationTtl: FAILURE_TTL_SECONDS, + }); + } catch { + // A failure to record a failure changes nothing about the answer we already + // have; the fetch failure itself was logged above. + } + } +} + +/** + * Refreshes in flight in THIS isolate, keyed by cache key. + * + * Two layers of single-flight because there are two stampedes. KV has no + * compare-and-set, so its lock is best-effort and cross-isolate only — two + * concurrent requests in one isolate can both read a null lock before either + * writes it. This map closes that window exactly where it is widest: one isolate + * handling a burst of requests for the same owner. + */ +const inFlightWarms = new Map>(); + +/** + * Warm one owner's entitlements cache off the response path. + * + * Called after auth resolves an identity, so `forOwner`'s cached read hits on + * subsequent requests — without this the cached read would be permanently cold + * and every enforcement site would see `source: "default"` forever. + * + * Fire-and-forget by construction: it schedules on `waitUntil` and returns + * nothing. With no `ExecutionContext` (unit tests, non-Workers runtimes) it does + * NOT fall back to awaiting inline — unlike the token `last_used` touches nearby, + * which are cheap local writes. This one is a cross-network call, and putting it + * on the request path is the exact latency this design avoids. Skipping it is + * safe: readers fail open. + */ +export function warmEntitlements( + env: Env, + waitUntil: ((promise: Promise) => void) | undefined, + subject: { ownerId: string; ownerType: "user" | "org" }, + logger: Logger, +): void { + if (!waitUntil) return; + if (!subject.ownerId) return; + if (!entitlementsEnabled(env)) return; + if (!env.STATE) return; + + const key = cacheKey(subject.ownerType, subject.ownerId); + const existing = inFlightWarms.get(key); + if (existing) { + waitUntil(existing); + return; + } + const run = warmOnce(env, subject, logger).finally(() => { + inFlightWarms.delete(key); + }); + inFlightWarms.set(key, run); + waitUntil(run); +} + +async function warmOnce( + env: Env, + subject: { ownerId: string; ownerType: "user" | "org" }, + logger: Logger, +): Promise { + const { ownerId, ownerType } = subject; + const kv = env.STATE; + try { + if (await kv.get(cacheKey(ownerType, ownerId))) return; + // A recent failure is respected rather than retried: re-fetching per request + // during an outage is how a fail-open dependency turns into a load generator. + if (await kv.get(negativeKey(ownerType, ownerId))) return; + if (await kv.get(lockKey(ownerType, ownerId))) return; + await kv.put(lockKey(ownerType, ownerId), "1", { expirationTtl: FAILURE_TTL_SECONDS }); + } catch (error) { + // The lock is an optimization. Losing it must not lose the warm, exactly as + // the backup runner reasons about its own KV lock. + logger.warn("Entitlements warm lock unavailable; proceeding without single-flight", { + ownerId, + error: error instanceof Error ? error.message : String(error), + }); + } + + await new RemoteEntitlements(env, logger).refresh(ownerId, ownerType); + + try { + // Released as soon as the answer is cached; the TTL is only the crash path. + await kv.delete(lockKey(ownerType, ownerId)); + } catch { + // Held locks expire on their own. A stuck one costs one TTL of staleness. + } +} diff --git a/src/billing/usage-banner.ts b/src/billing/usage-banner.ts new file mode 100644 index 00000000..0037feb6 --- /dev/null +++ b/src/billing/usage-banner.ts @@ -0,0 +1,215 @@ +/** + * The 80% banner: the same crossing the email announces, shown on the pages a + * signed-in user actually visits (PRD §8, Task 9.3). + * + * ## Why it is a stored receipt and not a computed check + * + * "Are you near a limit?" is a question about D1 (`usage_periods`) and the + * billing service, and asking it on every page render would put a database + * read and an entitlements lookup in front of every dashboard, project and + * change page — for a banner that is absent for almost everyone, almost + * always. So nothing is computed here. `usage-notifications.ts` already + * detects the crossing exactly once, on the write that causes it, from the + * totals `upsertUsage` returns; {@link recordUsageBanner} stores what it + * found, and rendering costs **one KV `get`** and no D1 at all. + * + * The cost is bounded further: {@link loadUsageBanner} returns immediately — + * with no KV read whatsoever — when there is no signed-in user or when + * `entitlementsEnabled` is false, so a self-hoster's pages are byte-for-byte + * as cheap as before. + * + * ## Users only + * + * There is no org UI in this application, so an org's threshold reaches its + * actor by email alone (PRD §8). A banner is therefore written only when the + * enforcement subject IS a user, and is keyed on that user. + * + * ## One entry per (user, period), holding one notice per meter + * + * A key per meter would cost one KV read per meter on every page render, for a + * banner that is absent for almost everyone; a single notice per period silently + * lost the first crossing when a second meter crossed. So the entry holds a + * notice per meter and the reader renders the one nearest its limit — still one + * `get`, and no crossing is overwritten by another. + * + * ## What it costs in freshness, stated rather than discovered + * + * The stored notice is a SNAPSHOT of the crossing, not a live figure: at 80% of + * 1,000 it says "800 of 1,000", and it still says that at 999. So it understates + * how little headroom is left for the rest of the period — and it overstates the + * usage if the account's limit is RAISED mid-period, until the period rolls. + * Both are why the banner reports a threshold that was crossed rather than a + * balance, and why it links to `/settings/usage`, which reads live figures. The + * alternative — re-reading entitlements and usage per page — is the cost this + * design exists to avoid. + */ +import { type MeterKey, isMeterKey } from "../storage/usage"; +import type { Env } from "../types"; +import type { Logger } from "../utils/logger"; +import { enforcementBinding } from "./enforcement"; +import { entitlementsEnabled } from "./entitlements"; + +/** One crossing, as it is stored. Every field is display data. */ +export interface StoredUsageBanner { + meter: MeterKey; + /** Hosted consumption at the moment of the crossing. */ + used: number; + /** The limit in force then. See the freshness note in the module doc. */ + limit: number; + /** The threshold that was crossed, as a percentage (80). */ + percent: number; + /** 'YYYY-MM' UTC — the period the figures belong to. */ + period: string; +} + +/** What a rendered banner says: a stored crossing plus what it means right now. */ +export interface UsageBannerNotice extends StoredUsageBanner { + /** + * Whether limits actually bind on this instance right now + * (`enforcementBinding`). Decided at READ time and never stored: the copy has + * to say "is refused" or "nothing is refused yet", and during the observe-only + * month the second one is the true sentence. A stored flag would keep + * asserting whichever mode was on when the crossing happened. + */ + enforcing: boolean; +} + +/** + * Versioned like the entitlements cache: a shape change must not read back as + * the old shape. Bumped to v2 when the entry became one notice per METER — a v1 + * entry is a bare notice and would parse as an empty set of them. + */ +const BANNER_PREFIX = "usage-banner:v2:"; + +/** + * A month plus the grace the meter itself allows. The period is IN the key, so + * a new month silently reads as "no banner" without anything sweeping the old + * one; the TTL is only what stops KV accumulating them forever. + */ +const BANNER_TTL_SECONDS = 40 * 24 * 60 * 60; + +function bannerKey(userId: string, period: string): string { + return `${BANNER_PREFIX}${userId}:${period}`; +} + +/** + * Store the crossing so the next page render can show it, keeping any crossing + * already recorded for another meter this period. + * + * Read-modify-write rather than a blind put, because the entry is per period and + * a second meter's crossing used to overwrite the first — the user was warned + * about tokens and never about deploys. The race a read-modify-write admits is + * bounded to nothing worse than that same lost banner: crossings are + * edge-triggered and deduped by their own receipt, so at most one write per + * (meter, limit) exists to be lost, and the 100% refusal still speaks. + * + * Best-effort by contract, like everything on the notification path: a banner + * that cannot be written must never fail the merge whose usage produced it. + */ +export async function recordUsageBanner( + env: Env, + logger: Logger, + userId: string, + notice: StoredUsageBanner, +): Promise { + if (!env.STATE || !userId) return; + const key = bannerKey(userId, notice.period); + try { + const existing = parseEntry(await readJson(env, key), notice.period); + const merged = existing.filter((entry) => entry.meter !== notice.meter); + merged.push(notice); + await env.STATE.put(key, JSON.stringify({ notices: merged }), { + expirationTtl: BANNER_TTL_SECONDS, + }); + } catch (error) { + logger.warn("Usage banner could not be stored", { + userId, + error: error instanceof Error ? error.message : String(error), + }); + } +} + +/** + * The banner to render for this user right now, or `null`. + * + * One KV read, and none at all when the seam is off or nobody is signed in. + * Every failure and every unrecognised value resolves to `null`: a page is not + * worth failing for a notice, and a half-parsed one would render nonsense. + * + * When several meters have crossed, the one FURTHEST through its allowance is + * shown: one banner is the shared chrome's budget, and the meter nearest the + * wall is the one worth the reader's attention. + * + * @param userId - The signed-in user; agents are not shown pages + * @param period - 'YYYY-MM' UTC; the caller passes the period it is rendering + */ +export async function loadUsageBanner( + env: Env, + logger: Logger, + userId: string | undefined, + period: string, +): Promise { + if (!userId) return null; + if (!entitlementsEnabled(env)) return null; + if (!env.STATE) return null; + try { + const notices = parseEntry(await readJson(env, bannerKey(userId, period)), period); + let worst: StoredUsageBanner | null = null; + for (const notice of notices) { + if (!worst || notice.used / notice.limit > worst.used / worst.limit) worst = notice; + } + return worst === null ? null : { ...worst, enforcing: enforcementBinding(env) }; + } catch (error) { + logger.debug("Usage banner unreadable; rendering without it", { + userId, + error: error instanceof Error ? error.message : String(error), + }); + return null; + } +} + +/** The stored entry, or `null`. Throws nothing a caller has not already caught. */ +async function readJson(env: Env, key: string): Promise { + const raw = await env.STATE?.get(key); + return raw ? JSON.parse(raw) : null; +} + +/** + * Validated on the way out, not trusted because we wrote it: the entry outlives + * deploys, and a notice from an older shape must vanish rather than render as + * "NaN% of undefined". An unrecognised entry reads as "no crossings", which is + * also what a v1 entry does — the prefix is versioned, so it cannot be read at + * all, and this is the belt to that brace. + */ +function parseEntry(value: unknown, period: string): StoredUsageBanner[] { + if (typeof value !== "object" || value === null) return []; + const { notices } = value as { notices?: unknown }; + if (!Array.isArray(notices)) return []; + const parsed: StoredUsageBanner[] = []; + for (const entry of notices) { + const notice = parseNotice(entry, period); + if (notice) parsed.push(notice); + } + return parsed; +} + +function parseNotice(value: unknown, period: string): StoredUsageBanner | null { + if (typeof value !== "object" || value === null) return null; + const record = value as Record; + const { meter, used, limit, percent } = record; + // Validated against the known set, not cast: `meterTitle` and the page both + // key off it, and a meter this build does not have a name for renders as a + // raw column value in the shared chrome of every page. + if (!isMeterKey(meter)) return null; + if (!Number.isFinite(used) || !Number.isFinite(limit) || !Number.isFinite(percent)) return null; + if ((limit as number) <= 0) return null; + return { + meter, + used: used as number, + limit: limit as number, + percent: percent as number, + // The key already pins the period; echoing the caller's keeps a mismatched + // stored value from being displayed against the month it was not about. + period, + }; +} diff --git a/src/billing/usage-notifications.ts b/src/billing/usage-notifications.ts new file mode 100644 index 00000000..8cbf1967 --- /dev/null +++ b/src/billing/usage-notifications.ts @@ -0,0 +1,307 @@ +/** + * "You are approaching a limit" — the missing middle between a usage page + * nobody visits and a merge that suddenly blocks (PRD §8, Goal 10). + * + * ## Edge-triggered, not state-triggered + * + * The notice fires on the CROSSING of 80%, which is why `upsertUsage` returns + * the post-write total: the totals either side of one write are what make "just + * went past" distinguishable from "is past", without a second read and without + * a job that scans everyone. + * + * ## The receipt is in KV, and the limit is part of its key + * + * A column on `usage_periods` would be at the wrong grain (per `source`) and on + * the wrong subject (the recorded one), and a new table would pull in + * `tests/backup-coverage.test.ts`, which requires every migration-created table + * to be classified — a notification receipt is neither billing data nor worth + * restoring. The limit is IN the key on purpose: raising someone's limit + * mid-period drops them back under 80%, and a receipt keyed without the limit + * would silence the warning for the second crossing, which is the case the + * dedupe was supposed to make safe. + * + * ## Not on the events queue + * + * `events.project` is NOT NULL and a usage threshold is owner-scoped, and every + * domain event fans out through `webhookHandler` — which would publish a + * customer's billing state to whatever third-party endpoint a project happens to + * have configured. If this ever goes on the queue it must be an internal event + * type excluded from fan-out. + * + * Gated on `entitlementsEnabled` throughout: a self-hoster must never be nagged + * about a limit they do not have. + */ +import { getUsageThresholdEmail } from "../email/templates"; +import type { BillingSubject } from "../storage/costs"; +import type { UsageWriteTotal } from "../storage/usage"; +import { getUser } from "../storage/users"; +import type { Env } from "../types"; +import type { Logger } from "../utils/logger"; +import { enforcementBinding, periodResetsAt, resolveEnforcementSubject } from "./enforcement"; +import { RemoteEntitlements, UnlimitedEntitlements, entitlementsEnabled } from "./entitlements"; +import { recordUsageBanner } from "./usage-banner"; + +/** Fraction of a limit that earns a warning. One threshold, so one receipt shape. */ +export const USAGE_NOTICE_THRESHOLD = 0.8; + +/** Percent form, used in the receipt key and the copy. */ +const THRESHOLD_PERCENT = Math.round(USAGE_NOTICE_THRESHOLD * 100); + +/** Versioned like the entitlements cache, so a key-shape change cannot be misread. */ +const RECEIPT_PREFIX = "usage-notice:v1:"; + +/** + * Long enough to outlive the period the crossing belongs to (a month plus the + * grace the meter itself uses), short enough that receipts do not accumulate + * forever. KV expires them; nothing sweeps. + */ +const RECEIPT_TTL_SECONDS = 40 * 24 * 60 * 60; + +/** What a recording site hands over so a crossing can be noticed. */ +export interface UsageNoticeInput { + /** Whose `usage_periods` rows were just written. */ + recorded: BillingSubject; + /** The user who ran the operation, when the site knows one (PRD §4a). */ + actorUserId?: string; + period: string; + /** Post-write totals from `upsertUsage`, including what this write added. */ + totals: readonly UsageWriteTotal[]; + /** Present on a request path; absent in the queue consumers, which run inline. */ + waitUntil?: (promise: Promise) => void; +} + +/** + * Notice any 80% crossing in a just-written batch of usage and email the actor. + * + * Best-effort by contract, like the recording it hangs off: every failure is + * logged and swallowed, because a notification must never fail a merge. + * + * Delivery is `waitUntil` where a request context exists and inline where it + * does not — the deploy runner and the post-merge check are queue consumers + * with no request to hang work off. + */ +export function noticeUsageThresholds(env: Env, logger: Logger, input: UsageNoticeInput): void { + if (!entitlementsEnabled(env)) return; + if (input.totals.length === 0) return; + const run = deliver(env, logger, input).catch((error) => { + logger.warn("Usage threshold notification failed", { + ownerId: input.recorded.ownerId, + error: error instanceof Error ? error.message : String(error), + }); + }); + if (input.waitUntil) input.waitUntil(run); + // Deliberately NOT awaited when there is no `waitUntil`: the two queue + // consumers that land here are mid-deploy and mid-post-merge, and an email + // send is not something either should block a terminal write on. The promise + // is best-effort and its failure path is above. +} + +async function deliver(env: Env, logger: Logger, input: UsageNoticeInput): Promise { + // BYOK spend is not under the allowance at all (migration 049, Goal 5), so a + // `byok` row can never approach a hosted limit and must not warn about one. + const platform = input.totals.filter((total) => total.source === "platform" && total.added > 0); + if (platform.length === 0) return; + + const subject = await resolveEnforcementSubject(env, logger, { + ...(input.actorUserId !== undefined ? { actorUserId: input.actorUserId } : {}), + owner: input.recorded, + }); + if (!subject) return; + + // The totals in hand are the RECORDED subject's rows. When the enforcement + // subject is somebody else — an actor charged for a spend recorded against a + // free org (§4a) — those totals are the wrong denominator for their limit, and + // warning off them would be a number about someone else's month. That case is + // left to the 100% refusal, which names the limit at the moment it binds. + if (subject.ownerId !== input.recorded.ownerId) { + logger.debug("Usage threshold check skipped: recorded subject is not the enforcement subject", { + recordedId: input.recorded.ownerId, + subjectId: subject.ownerId, + }); + return; + } + + const resolved = await new RemoteEntitlements(env, logger).forOwner( + subject.ownerId, + subject.ownerType, + ); + if (!resolved.success) { + // Fails open like every other reader — unlimited has no 80%, so this warns + // about nothing — but never silently: a warning nobody got because the + // billing service was unreachable is the failure Goal 10 exists to prevent, + // and it must be visible in the log rather than inferred from its absence. + logger.warn("Usage threshold check fell back to unlimited: entitlements unreadable", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + error: resolved.error.message, + }); + } + const entitlements = resolved.success ? resolved.data.entitlements : UnlimitedEntitlements; + + for (const total of platform) { + const limit = entitlements.meters[total.meter]; + // Unlimited (-1) has no 80%, and a hard block (0) is not something anyone + // can approach — they hit it on the first unit, where the refusal speaks. + if (!Number.isInteger(limit) || limit <= 0) continue; + const threshold = limit * USAGE_NOTICE_THRESHOLD; + // AT or above, not the crossing itself. An edge trigger + // (`before < threshold && quantity >= threshold`) fires exactly once per + // period, which is correct only while the send succeeds: a delivery that + // failed left the receipt unwritten to be retried, but every later write + // already has `before >= threshold`, so it was skipped here — before the + // receipt was ever consulted — and the notice was lost for the period after + // all. The receipt below is what makes this once-per-period; this line only + // decides whether the subject is over the line. Costs one KV read per meter + // per recording once a subject is above 80%, which is the price of the + // warning actually arriving. + if (total.quantity < threshold) continue; + + const key = + `${RECEIPT_PREFIX}${subject.ownerType}:${subject.ownerId}:${input.period}:` + + `${total.meter}:${THRESHOLD_PERCENT}:${limit}`; + if (await alreadySent(env, logger, key)) continue; + + // The in-app banner (Task 9.3) is written from the crossing this loop has + // already detected, so a page render costs one KV read and no D1. Users + // only: there is no org UI to render it in, so a pooled org's threshold + // reaches its actor by email alone (PRD §8). + if (subject.ownerType === "user") { + await recordUsageBanner(env, logger, subject.ownerId, { + meter: total.meter, + used: total.quantity, + limit, + percent: THRESHOLD_PERCENT, + period: input.period, + }); + } + + // Per meter, because the crossing is edge-triggered on `markSent`: a mail + // provider that rejects one meter's notice would otherwise throw out of the + // loop, and a second meter that crossed in the same batch would lose both + // its email and its banner for the rest of the period — one delivery + // failure silencing a warning nobody gets a second chance at. + try { + // Only on a delivery that happened. `send` returns early — successfully, + // without throwing — when there is no recipient, when EMAIL / + // EMAIL_FROM_ADDRESS / DB is unconfigured, or when the user row cannot be + // read. Marking the receipt on those would burn the once-per-period slot + // on a notice nobody received, and the operator who then fixes the + // configuration would still hear nothing for the rest of the month. This + // is the same failure as the crossing-gate one above, one layer down: the + // gate fix caught the throwing path and left the silent ones. + if ( + await send(env, logger, { + subject, + actorUserId: input.actorUserId, + meter: total.meter, + used: total.quantity, + limit, + period: input.period, + }) + ) { + await markSent(env, logger, key); + } + } catch (error) { + // Deliberately NOT marked sent: the receipt is what makes this + // once-per-crossing, so leaving it unwritten is what lets the next + // evaluation in this period try again. + logger.warn("Usage threshold notice failed for one meter", { + meter: total.meter, + subjectId: subject.ownerId, + subjectType: subject.ownerType, + error: error instanceof Error ? error.message : String(error), + }); + } + } +} + +async function alreadySent(env: Env, logger: Logger, key: string): Promise { + if (!env.STATE) return true; + try { + return (await env.STATE.get(key)) !== null; + } catch (error) { + // A receipt we cannot read is treated as SENT: a duplicate warning is a + // worse failure than a missing one, and the 100% refusal still speaks. + logger.warn("Usage threshold receipt unreadable; treating as already sent", { + error: error instanceof Error ? error.message : String(error), + }); + return true; + } +} + +async function markSent(env: Env, logger: Logger, key: string): Promise { + if (!env.STATE) return; + try { + await env.STATE.put(key, String(Date.now()), { expirationTtl: RECEIPT_TTL_SECONDS }); + } catch (error) { + logger.warn("Usage threshold receipt could not be written", { + error: error instanceof Error ? error.message : String(error), + }); + } +} + +/** + * Mail the person whose allowance it is. + * + * An org has no email column (`migrations/003_orgs.sql`) and there is no org UI + * to render a banner in, so under §4a the addressee is the actor — the person + * whose allowance is being spent. `users.email` is `UNIQUE NOT NULL` and there + * is no email-change route in the tree, so a user can only ever mail themselves: + * this is not an enumeration or spam surface. + */ +async function send( + env: Env, + logger: Logger, + opts: { + subject: { ownerId: string; ownerType: "user" | "org" }; + actorUserId?: string; + meter: string; + used: number; + limit: number; + period: string; + }, +): Promise { + const recipientId = + opts.actorUserId ?? (opts.subject.ownerType === "user" ? opts.subject.ownerId : undefined); + if (!recipientId) return false; + const fromAddress = env.EMAIL_FROM_ADDRESS; + // Both halves are required and one of them was declared for staging only until + // this work; without either, the send path returns early and the feature would + // ship silently never sending. See wrangler.toml's [env.production.vars]. + if (!env.EMAIL || !fromAddress || !env.DB) return false; + + const user = await getUser(env.DB, recipientId, logger); + if (!user.success) { + logger.warn("Usage threshold notice not sent: recipient could not be resolved", { + userId: recipientId, + }); + return false; + } + + const content = getUsageThresholdEmail({ + meter: opts.meter, + used: opts.used, + limit: opts.limit, + percent: THRESHOLD_PERCENT, + period: opts.period, + resetsAt: periodResetsAt(opts.period), + // What happens at 100% is a question about whether limits BIND, not about + // whether they exist: during the observe-only month nothing is refused, and + // a mail that says otherwise is teaching people to disbelieve it later. + enforcing: enforcementBinding(env), + }); + await env.EMAIL.send({ + to: user.data.email, + from: { email: fromAddress, name: "Stratum" }, + subject: content.subject, + text: content.text, + html: content.html, + }); + logger.info("Usage threshold notice sent", { + userId: recipientId, + meter: opts.meter, + limit: opts.limit, + period: opts.period, + }); + return true; +} diff --git a/src/billing/usage-report.ts b/src/billing/usage-report.ts new file mode 100644 index 00000000..7a8cab05 --- /dev/null +++ b/src/billing/usage-report.ts @@ -0,0 +1,289 @@ +/** + * What one account has consumed against what its plan allows. + * + * The single read behind both visibility surfaces of PRD §8 and §4c — the + * server-rendered `/settings/usage` page and the `stratum_get_usage` MCP tool — + * so a human and an agent are never shown two different answers to the same + * question. + * + * Three properties are load bearing, and all three are easy to get wrong: + * + * - **The subject is the §4a ENFORCEMENT subject**, resolved through + * {@link resolveEnforcementSubject}, not the recorded billing subject. A + * limit is checked against the acting person, so a page that reported the + * recorded owner's totals would show a number no limit is ever compared to. + * That is not the same as reading the actor's `usage_periods` rows, and the + * difference is the reason {@link UsageReport.usedSource} exists: for a + * project in an org namespace the LEDGER records the ORG (deliberately — a + * ledger says what happened) while the CHECK charges the actor's counter. A + * user working only in org namespaces has no rows of their own at all, so a + * page summing D1 alone showed them `0 of 10,000` while the counter refusing + * them was full. So `used` is the counter where there is one, floored by the + * ledger — which is exactly the pair `checkMeter` compares against a limit, + * since `reconcileFloor` raises the counter to the same filtered D1 sum. + * - **Consumption is `source = 'platform'` only.** Migration 049 keys usage on + * `(owner_id, period, meter, source)` precisely so BYOK spend accumulates + * apart, and Goal 5 says it is not under the hosted allowance. Summing both + * would tell someone their own provider bill had eaten an allowance it never + * touched. BYOK is reported as its OWN figure, for information, never folded + * into `used`. + * - **`-1` is unlimited and says so.** A self-hoster's every limit is `-1` + * (`UnlimitedEntitlements`), so a renderer that turned a limit into a + * percentage would show them an empty or a divide-by-zero bar on every row + * and look broken. {@link UsageMeterReport.unlimited} is the flag a caller + * branches on, and `percentUsed` is `null` rather than a made-up number. + * + * Read-only throughout: nothing here reserves, settles, records or mutates. + */ +import { usageMeterName } from "../queue/usage-meter"; +import { type MeterKey, getOwnerMeterTotals, isMeterKey, usagePeriod } from "../storage/usage"; +import type { Env } from "../types"; +import { type AppError, ValidationError } from "../utils/errors"; +import type { Logger } from "../utils/logger"; +import { type Result, err, ok } from "../utils/result"; +import { periodResetsAt, resolveEnforcementSubject } from "./enforcement"; +import { + METER_KEYS, + RATE_KEYS, + type RateKey, + RemoteEntitlements, + UNLIMITED, + UnlimitedEntitlements, + entitlementsEnabled, +} from "./entitlements"; + +/** One meter's consumption this period, and the allowance it is measured against. */ +export interface UsageMeterReport { + meter: MeterKey; + /** Hosted consumption — `source = 'platform'`, the only kind a limit binds. */ + used: number; + /** + * Bring-your-own-key consumption, reported apart and never added to + * {@link UsageMeterReport.used}: it is spent on the project's own provider + * account and no allowance applies to it (Goal 5). + */ + byok: number; + /** `-1` unlimited, `0` a hard block, otherwise the ceiling. */ + limit: number; + /** True when `limit` is `-1`. Callers render words, not an empty bar. */ + unlimited: boolean; + /** True when `limit` is `0` — not "unset", a plan that forbids this outright. */ + blocked: boolean; + /** `used / limit` as a percentage, or `null` when there is no finite limit. */ + percentUsed: number | null; + /** What is left of a finite allowance, floored at zero; `null` when unlimited. */ + remaining: number | null; +} + +/** A rate ceiling. Reported as a limit only — its window lives on the meter object, not in D1. */ +export interface UsageRateReport { + rate: RateKey; + limit: number; + unlimited: boolean; +} + +export interface UsageReport { + /** Whose allowance this is — the §4a enforcement subject. */ + subject: { ownerId: string; ownerType: "user" | "org" }; + /** + * Where the `used` figures came from, so a surface can say which it is + * showing instead of implying a precision it does not have. + * + * `"meter"` is the live enforcement counter, the number a limit is actually + * compared against, floored by the ledger. `"ledger"` is `usage_periods` + * alone — every self-hoster, and any instance whose meter binding is missing + * or unreadable — which counts spend RECORDED against this subject and so + * omits work done in an org namespace, where the org is the recorded owner. + */ + usedSource: "meter" | "ledger"; + /** Opaque plan name from the billing service; `"unlimited"` when there is none. */ + plan: string; + /** + * Whether any allowance is in force at all. False for every self-hoster (no + * `BILLING_SERVICE_URL`), which is why the page must read as "unlimited" + * rather than as "we could not load your usage". + */ + metered: boolean; + /** 'YYYY-MM' UTC. */ + period: string; + /** ISO 8601 instant the monthly meters roll over. */ + resetsAt: string; + meters: UsageMeterReport[]; + rates: UsageRateReport[]; +} + +/** + * A short display name per meter, for a column heading or a banner sentence. + * + * Deliberately not `enforcement.ts`'s `meterLabel`, which builds the fragment + * "monthly LLM token allowance" for the middle of a refusal sentence. This one + * is a heading; sharing one string would make one of the two read badly. + */ +export function meterTitle(meter: MeterKey): string { + switch (meter) { + case "llm_tokens_month": + return "LLM tokens"; + case "sandbox_ms_month": + return "Sandbox time"; + case "deploys_month": + return "Deploys"; + default: + return meter; + } +} + +function meterEntry( + meter: MeterKey, + limit: number, + platform: Partial>, + byok: Partial>, + counter: Partial> | null, +): UsageMeterReport { + // The higher of the two, which is what a limit check sees: the counter leads + // the ledger (a reservation is taken before the spend, and it holds the + // actor's org-namespace work, which the ledger records against the org), + // while `reconcileFloor` raises the counter to the ledger whenever the ledger + // is ahead. Neither alone is the number that refuses somebody. + const used = Math.max(platform[meter] ?? 0, counter?.[meter] ?? 0); + // A limit the billing service never sent, or sent as garbage, has already + // been discarded by `parseEntitlements`; anything still non-integer here is + // treated as "no usable limit" for the same reason `checkGauge` does — a + // limit derived from garbage must not render as a bar someone acts on. + const usable = Number.isInteger(limit); + const unlimited = !usable || limit === UNLIMITED; + const finite = usable && limit > 0; + return { + meter, + used, + byok: byok[meter] ?? 0, + limit, + unlimited, + blocked: usable && limit === 0, + percentUsed: finite ? Math.round((used / limit) * 100) : null, + remaining: finite ? Math.max(0, limit - used) : null, + }; +} + +/** + * Build one account's usage report for a period. + * + * @param opts.actorUserId - The signed-in user, or an agent's owner (§4a) + * @param opts.at - The moment whose period to report; defaults to now + * @param opts.includeByok - Read the `byok` totals too. The page shows them as + * a separate labelled figure; the MCP tool does not ask for them, and the + * flag keeps that read off a path that would only discard it. + * @returns The report, or the error that stopped it. A caller mistake (no + * actor) is an error rather than an empty report — "you have used nothing" + * is a claim, and it must not be made about nobody. + */ +export async function buildUsageReport( + env: Env, + logger: Logger, + opts: { actorUserId: string; at?: Date; includeByok?: boolean }, +): Promise> { + if (!opts.actorUserId) { + return err(new ValidationError("Usage requested without an actor")); + } + const at = opts.at ?? new Date(); + const period = usagePeriod(at); + + // No project in hand, so this resolves to the actor itself and performs no + // I/O — which is exactly §4a's answer for "whose allowance is this page + // about": the person reading it, never an org they happen to belong to. + const subject = await resolveEnforcementSubject(env, logger, { actorUserId: opts.actorUserId }); + if (!subject) { + return err(new ValidationError("Usage subject could not be resolved")); + } + + const resolved = await new RemoteEntitlements(env, logger).forOwner( + subject.ownerId, + subject.ownerType, + ); + if (!resolved.success) { + // Fails open exactly as every other reader does: an unreachable billing + // service shows unlimited rather than a page full of errors. Logged, not + // dropped — a page telling somebody they have no limits because the limits + // could not be read must be diagnosable from the log (AGENTS.md). + logger.warn("Usage report fell back to unlimited: entitlements unreadable", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + error: resolved.error.message, + }); + } + const entitlements = resolved.success ? resolved.data.entitlements : UnlimitedEntitlements; + + // The same read `reconcileFloor` performs before a limit check, with the same + // `source` filter. Anything else would be a second definition of "used". It + // is the FLOOR under the figure reported, not the figure itself — see below. + const platform = await getOwnerMeterTotals(env.DB, logger, subject.ownerId, period, "platform"); + if (!platform.success) return err(platform.error); + + const counter = await readMeterCounts(env, logger, subject, period); + + let byok: Partial> = {}; + if (opts.includeByok) { + const read = await getOwnerMeterTotals(env.DB, logger, subject.ownerId, period, "byok"); + // A failed BYOK read is not worth failing the page over: it is an + // informational figure beside the allowance, not the allowance itself. + if (read.success) byok = read.data; + } + + return ok({ + subject: { ownerId: subject.ownerId, ownerType: subject.ownerType }, + usedSource: counter === null ? "ledger" : "meter", + plan: entitlements.plan, + metered: entitlementsEnabled(env), + period, + resetsAt: periodResetsAt(period), + meters: METER_KEYS.map((meter) => + meterEntry(meter, entitlements.meters[meter], platform.data, byok, counter), + ), + rates: RATE_KEYS.map((rate) => { + const limit = entitlements.rates[rate]; + return { rate, limit, unlimited: !Number.isInteger(limit) || limit === UNLIMITED }; + }), + }); +} + +/** + * This subject's live enforcement counters for the period, or `null` when there + * are none to read. + * + * `null` is not a failure to hide: it is every self-hosted instance (no billing + * service, so nothing ever reserved), an instance whose `USAGE_METER` binding is + * missing, and an object that could not be reached. The caller reports which of + * the two numbers it is showing rather than silently presenting one as the + * other. + * + * Read-only: this is `read`, never `reserve`, so looking at the usage page + * cannot consume any part of an allowance. + */ +async function readMeterCounts( + env: Env, + logger: Logger, + subject: { ownerId: string; ownerType: "user" | "org" }, + period: string, +): Promise> | null> { + if (!entitlementsEnabled(env)) return null; + const namespace = env.USAGE_METER; + if (!namespace) return null; + try { + const stub = namespace.get( + namespace.idFromName(usageMeterName(subject.ownerType, subject.ownerId)), + ); + const snapshot = await stub.read(period); + const counts: Partial> = {}; + for (const [meter, quantity] of Object.entries(snapshot.counts)) { + if (isMeterKey(meter) && typeof quantity === "number") counts[meter] = quantity; + } + return counts; + } catch (error) { + logger.warn("Usage counters unreadable; reporting the ledger instead", { + subjectId: subject.ownerId, + subjectType: subject.ownerType, + period, + error: error instanceof Error ? error.message : String(error), + }); + return null; + } +} diff --git a/src/deploy/runner.ts b/src/deploy/runner.ts index df4dcf64..55f83cc4 100644 --- a/src/deploy/runner.ts +++ b/src/deploy/runner.ts @@ -29,11 +29,13 @@ * injects its own seams: without them, testing any of the above needs a real git * remote, a real clock, and a real provider account. */ +import { checkMeter, resolveEnforcementSubject, settleMeter } from "../billing/enforcement"; +import { type LlmProviderCatalog, llmProviderCatalog } from "../evaluation/llm-providers"; import { parsePolicyContent } from "../evaluation/policy-loader"; import type { DeployConfig, DeployRejection } from "../evaluation/types"; import { type StratumEvent, emitEvent } from "../queue/events"; import { getChange } from "../storage/changes"; -import { recordCosts } from "../storage/costs"; +import { recordCosts, resolveBillingSubject } from "../storage/costs"; import { isTargetDeleting } from "../storage/deletion"; import { type Deployment, @@ -353,7 +355,7 @@ async function runMergeMessage( } const files = treeResult.data; - const policy = policyFromTree(files, logger); + const policy = policyFromTree(files, logger, llmProviderCatalog(env, logger)); // A rejected entry becomes a visible failed row, never a dropped one: a deploy // the author wrote and that never runs means production silently stopped @@ -488,7 +490,7 @@ async function runDeploymentMessage( // Config comes from the pinned commit here too, so an approval granted days // later still deploys with the configuration the commit was reviewed under. - const policy = policyFromTree(treeResult.data, logger); + const policy = policyFromTree(treeResult.data, logger, llmProviderCatalog(env, logger)); const config = policy.deploys.find((entry) => entry.name === deployment.name); if (!config) { const rejection = policy.rejections.find((entry) => entry.name === deployment.name); @@ -572,6 +574,17 @@ async function runOneDeployment( }); } + // Volume, and only volume (PRD §8). The queue is FIFO with + // `max_concurrency = 1` and no priority, so this cannot make anything fairer; + // what it bounds is how many deploys a month one subject may run. It goes + // BEFORE the claim so a refusal never takes the lease it would then have to + // release, and a refusal writes a persisted `failed` row with a named reason + // and acks — the message is not left to burn its two retries into the DLQ + // over a limit that redelivery cannot change, and it is never silently acked, + // because a deploy that did not happen must be visible. + const volume = await checkDeployVolume(ctx, deployment); + if (volume.refusal !== null) return await failWithoutRunning(ctx, deployment, volume.refusal); + const claim = await claimDeployment(env.DB, logger, { projectId: project.id, deploymentId: deployment.id, @@ -582,6 +595,9 @@ async function runOneDeployment( deploymentId: deployment.id, projectId: project.id, }); + // This message will be redelivered and will reserve again; the unit taken + // above belongs to no deploy, so it goes back. + await volume.release(); return { deploymentId: deployment.id, name: deployment.name, status: deployment.status }; } if (!claim.data.claimed) { @@ -593,8 +609,14 @@ async function runOneDeployment( projectId: project.id, why: claim.data.reason, }); + // Whoever does hold the lease reserved its own unit; this attempt's is not + // paying for anything. + await volume.release(); return { deploymentId: deployment.id, name: deployment.name, status: deployment.status }; } + // Past this point the attempt owns the row and the `finally` below always + // writes it terminal, so the reservation is spent on a deploy that happened — + // successful or failed — and is never released. const startedAt = ctx.now(); /** @@ -693,6 +715,76 @@ async function runOneDeployment( return record; } +/** A volume decision, plus the means to hand back what it reserved. */ +interface DeployVolumeCheck { + /** Why this deploy may not run, or `null` when it may. */ + refusal: string | null; + /** + * Gives back the unit this check reserved. Called on every path that does NOT + * go on to run the deploy, because `reserve` is a charge and only a deploy + * that actually happens should carry one. + */ + release: () => Promise; +} + +/** Nothing was checked, so there is nothing to refuse and nothing to give back. */ +const NO_VOLUME_CHECK: DeployVolumeCheck = { refusal: null, release: async () => {} }; + +/** + * Whether this deploy fits the subject's monthly allowance. + * + * The enforcement subject is the deploy's requester when a human asked for one + * (an approve or a retry), and otherwise the project's own billing subject — + * but only when that names a PERSON. A merge-triggered deploy of an ORG-owned + * project names no actor, and `resolveEnforcementSubject` reports the check + * unavailable there rather than making the org its own subject, which would + * hand every org a fresh, permanently unlimited deploy allowance (PRD §4a). So + * that one case is deliberately unmetered, and logged where it happens. + * + * Inert with the billing service unconfigured — `resolveEnforcementSubject` + * names nobody and `checkMeter` returns before it reads anything — so a + * self-hoster's deploys cost neither the D1 walk nor the check. + */ +async function checkDeployVolume( + ctx: RunContext, + deployment: Deployment, +): Promise { + const { env, logger, project } = ctx; + const actorUserId = deployment.requestedByType === "user" ? deployment.requestedById : undefined; + const subject = await resolveEnforcementSubject(env, logger, { + ...(actorUserId !== undefined ? { actorUserId } : {}), + owner: { ownerId: project.ownerId, ownerType: project.ownerType }, + }); + if (!subject) return NO_VOLUME_CHECK; + const nowMs = ctx.now(); + const decision = await checkMeter(env, logger, { + subject, + meter: "deploys_month", + estimate: 1, + nowMs, + what: "This deployment", + }); + // A deploy meter has no natural settle — the true cost of a deploy is one + // deploy — so the reservation is released explicitly by whoever decides this + // attempt is not the one that runs. Without it, every attempt that stops + // before the claim (a lease held elsewhere, a D1 failure, a redelivered + // message) burned another unit, and one deploy could cost two or three. + const release = decision.reserved + ? async () => { + await settleMeter(env, logger, { + subject, + meter: "deploys_month", + delta: -1, + nowMs, + }); + } + : NO_VOLUME_CHECK.release; + return { + refusal: decision.admitted ? null : (decision.reason ?? "Deploy allowance exhausted"), + release, + }; +} + /** * Resolve secrets and hand the tree to the provider. Returns what should be * written; never writes anything itself, so its caller's `finally` owns the row. @@ -852,6 +944,7 @@ async function readTree( projectDefaultBranch(project), ); + const subject = await resolveBillingSubject(env.DB, logger, project); // Recorded whether or not the read succeeded: a clone that failed still cost // the round trip. await recordCosts( @@ -861,6 +954,12 @@ async function readTree( project: project.name, projectId: project.id, ...(changeId !== undefined ? { changeId } : {}), + ...(subject ?? {}), + // Best-effort inline, with no `waitUntil`: this is a queue consumer and + // there is no request to schedule against (PRD §8). `git_ops` maps to no + // meter today, so this notices nothing until `deploys_month` gains a D1 + // writer — wired anyway so that writer does not have to remember. + notify: { env }, }, [{ kind: "git_ops", quantity: 1 }], ); @@ -877,14 +976,23 @@ async function readTree( * effect. A malformed file yields one named rejection rather than silence, * because a single YAML typo must not quietly stop production updating. */ -function policyFromTree(files: ReadonlyMap, logger: Logger): TreePolicy { +function policyFromTree( + files: ReadonlyMap, + logger: Logger, + /** The same allowlist the evaluation path parses with. Passed even though + * only `deploys:` is read here: a policy naming a provider the parser cannot + * resolve is *malformed*, and reading it without the catalog would report + * "no deploy configuration could be read" for a file that is perfectly + * valid. */ + providers: LlmProviderCatalog, +): TreePolicy { const decoder = new TextDecoder(); for (const { path, format } of POLICY_FILES) { const bytes = files.get(path); if (bytes === undefined) continue; - const parsed = parsePolicyContent(decoder.decode(bytes), format, logger); + const parsed = parsePolicyContent(decoder.decode(bytes), format, logger, providers); if (parsed.status === "malformed") { return { deploys: [], @@ -966,6 +1074,15 @@ async function createDeployment( * Write a terminal `failed` on a row that was never claimed — the change was * reverted, the project is being deleted, the tree would not read, or the deploy * is no longer declared. Leaving it `queued` would strand it forever. + * + * `onlyIfUnclaimed` because this runs BEFORE `claimDeployment`: a redelivery of + * the same message may already hold the lease and be deploying, and `running` is + * not a terminal status, so the write would otherwise land on a live row and + * release its lease. When it does not land — that case, or a row that reached a + * terminal status first — the row belongs to someone else, so no + * `deployment.failed` is emitted and the row's own status is reported back. An + * event announcing a failure for a deploy that is about to succeed is worse than + * no event: it is the one a webhook consumer acts on. */ async function failWithoutRunning( ctx: RunContext, @@ -980,6 +1097,7 @@ async function failWithoutRunning( status: "failed", reason: truncated, completedAt: ctx.iso(), + onlyIfUnclaimed: true, }); if (!completed.success) { ctx.logger.error("Could not fail the deployment", completed.error, { @@ -987,6 +1105,14 @@ async function failWithoutRunning( projectId: ctx.project.id, }); } + if (!completed.success || !completed.data) { + ctx.logger.warn("Deployment was not failed; leaving it to its owner", { + deploymentId: deployment.id, + projectId: ctx.project.id, + reason: truncated, + }); + return { deploymentId: deployment.id, name: deployment.name, status: deployment.status }; + } await emitDeploymentEvent(ctx, deployment, "deployment.failed", { reason: truncated }); return { @@ -1022,6 +1148,7 @@ async function supersedeWithoutRunning( status: "superseded", reason: truncated, completedAt: ctx.iso(), + onlyIfUnclaimed: true, }); if (!completed.success) { ctx.logger.error("Could not supersede the deployment", completed.error, { @@ -1029,6 +1156,9 @@ async function supersedeWithoutRunning( projectId: ctx.project.id, }); } + if (!completed.success || !completed.data) { + return { deploymentId: deployment.id, name: deployment.name, status: deployment.status }; + } return { deploymentId: deployment.id, diff --git a/src/email/templates.ts b/src/email/templates.ts index bb8cdca4..d1a9f7cd 100644 --- a/src/email/templates.ts +++ b/src/email/templates.ts @@ -294,6 +294,101 @@ Your code management platform`; return { subject, text, html: wrapEmail({ title: subject, body }) }; } +export interface UsageThresholdEmailData { + /** The meter key, e.g. `llm_tokens_month`. */ + meter: string; + used: number; + limit: number; + /** The threshold that was crossed, as a percentage. */ + percent: number; + /** 'YYYY-MM' UTC. */ + period: string; + /** ISO 8601 instant the period rolls over and the allowance refills. */ + resetsAt: string; + /** + * Whether limits BIND on this instance (`enforcementBinding`), as opposed to + * merely existing. Through the observe-only month nothing is refused, and a + * mail saying evaluations "stop passing" would be describing a wall that is + * not there — so the consequence sentence is chosen from this, never assumed. + */ + enforcing: boolean; +} + +/** Human wording for a meter key. Falls back to the key so a new meter still sends. */ +function meterTitle(meter: string): string { + switch (meter) { + case "llm_tokens_month": + return "AI review tokens"; + case "sandbox_ms_month": + return "sandbox time"; + case "deploys_month": + return "deployments"; + default: + return meter; + } +} + +/** + * "You have used 80% of X" — sent once per (subject, period, meter, threshold, + * limit) when a usage meter crosses its warning threshold. + * + * Written as a heads-up rather than a warning, because it arrives while nothing + * is blocked yet: it names what is running out, when it refills, and the two + * ways to avoid meeting the wall. Sent only when the cloud entitlements service + * is configured — a self-hoster has no limits to be warned about — and what it + * promises will happen at 100% depends on {@link UsageThresholdEmailData.enforcing}, + * because through the observe-only month the answer is "nothing". + */ +export function getUsageThresholdEmail(data: UsageThresholdEmailData): { + subject: string; + text: string; + html: string; +} { + const { used, limit, percent, period, resetsAt, enforcing } = data; + const title = meterTitle(data.meter); + const subject = `You've used ${percent}% of your Stratum ${title}`; + // Two true sentences rather than one convenient one; see `enforcing`. + const consequence = enforcing + ? "Nothing is blocked yet. When the allowance runs out, evaluations that need it stop\npassing until it refills." + : "Nothing is refused: this account's allowances are being measured, not enforced,\nand you will be told before that changes."; + + const text = `Heads up — this month's ${title} on your Stratum account is ${percent}% used. + +Used: ${used.toLocaleString("en-US")} of ${limit.toLocaleString("en-US")} (${period}) +Refills: ${resetsAt} + +${consequence} If you expect to need more this month you can either +raise your plan limits or point the llm evaluator at your own provider key with +\`provider:\` in .stratum/policy.yaml. + +-- +Stratum +Your code management platform`; + + const body = ` +
+ ${escapeHtml(String(percent))}% of your ${escapeHtml(title)} is used +
+
+ You've used ${escapeHtml(used.toLocaleString("en-US"))} + of ${escapeHtml(limit.toLocaleString("en-US"))} + for ${escapeHtml(period)}. The allowance refills at ${escapeHtml(resetsAt)}. +
+
+ ${ + enforcing + ? "Nothing is blocked yet. When the allowance runs out, evaluations that need it stop passing until it refills." + : "Nothing is refused: this account's allowances are being measured, not enforced, and you will be told before that changes." + } + If you expect to need more this month, raise your plan + limits, or point the llm evaluator at your own + provider key with provider: in + .stratum/policy.yaml. +
`; + + return { subject, text, html: wrapEmail({ title: subject, body }) }; +} + /** * Generic email template wrapper with Stratum branding * TODO: Use this for future email templates (notifications, invites, etc.) diff --git a/src/evaluation/index.ts b/src/evaluation/index.ts index 5f61b7a5..de25ec90 100644 --- a/src/evaluation/index.ts +++ b/src/evaluation/index.ts @@ -4,5 +4,8 @@ export * from "./webhook-evaluator"; export * from "./composite-evaluator"; export * from "./policy-loader"; export * from "./sandbox-evaluator"; +export * from "./llm-provider"; +export * from "./llm-providers"; +export * from "./llm-byok"; export * from "./llm-evaluator"; export * from "./secret-scanner"; diff --git a/src/evaluation/llm-byok.ts b/src/evaluation/llm-byok.ts new file mode 100644 index 00000000..12ceebfb --- /dev/null +++ b/src/evaluation/llm-byok.ts @@ -0,0 +1,166 @@ +/** + * Resolving the LLM provider an evaluation runs on: the operator's Workers AI + * binding, or the project's own credential against an operator-configured + * endpoint (BYOK). + * + * Two rules shape everything here, and both are PRD §6/§7: + * + * 1. **Never fall back to `env.AI`.** A BYOK misconfiguration that quietly ran + * on Workers AI would put the spend back on the operator's bill — the exact + * hole BYOK closes — and would be a gate that silently stops gating. Every + * failure below is fail-closed with a distinct reason instead. + * 2. **Skip the credential load entirely when no BYOK provider is named.** + * That is the overwhelmingly common case, and the load is a D1 read plus a + * 100k-iteration PBKDF2 derivation on the change-creation path, which + * already does two clones and a diff. + */ +import { DEPLOY_SECRET_KEY_MISSING, loadSecretValues } from "../storage/project-secrets"; +import type { Env, ProjectEntry } from "../types"; +import type { Logger } from "../utils/logger"; +import { AnthropicProvider, type LlmProvider, OpenAiCompatibleProvider } from "./llm-provider"; +import { llmProviderCatalog, providerSecretName } from "./llm-providers"; +import type { EvalPolicy, LlmEvaluatorConfig } from "./types"; + +/** Env slice the resolution needs. Named rather than `Env` so the credential + * path's dependencies are visible at a glance. */ +export type LlmByokEnv = Pick; + +export type LlmProviderSelection = + /** No BYOK provider named — the caller uses Workers AI, as it always has. */ + | { status: "platform" } + /** Run on the project's credential; cost samples from this run are `byok`. */ + | { status: "byok"; provider: LlmProvider; providerName: string } + /** The named provider cannot be used. The gate fails closed on this reason, + * and never on `env.AI`. */ + | { status: "unavailable"; reason: string }; + +/** + * Every `llm` entry in the policy. + * + * `filter`, not `find`, and the difference is a fail-open. `LLMEvaluator` reads + * the FIRST `llm` entry, while `buildEvaluators` constructs an evaluator for + * EVERY one — so resolving a provider from the first entry alone meant + * `[{llm}, {llm, provider: anthropic}]` produced two evaluators on `env.AI`, + * with no error and no log: a policy that asked to run on its own key silently + * running on the operator's. `sanitizeLlmConfig` now rejects a second entry at + * parse time, and {@link resolveLlmProvider} refuses to choose between them for + * any policy that reaches it another way (a KV-cached one, say). + */ +export function llmConfigsOf(policy: EvalPolicy): LlmEvaluatorConfig[] { + return policy.evaluators.filter((entry): entry is LlmEvaluatorConfig => entry.type === "llm"); +} + +/** + * Resolve the provider for this policy, loading the project's credential only + * when the policy names one. + * + * The plaintext credential is used in exactly one place: an `Authorization` (or + * `x-api-key`) header on the provider request. `loadSecretValues` is the only + * read path for a secret value in the codebase and is held to the same rule as + * the deploy runner — no route returns it, renders it, or logs it. That an + * HTTP request *triggers* this resolution is not the same thing as the value + * being route-reachable. + * + * "Used in one place" is not the same as "reaches one place", and the + * difference was a live leak: `new Headers()` throws a `TypeError` that quotes + * the offending value, so a stored credential containing CR, LF or NUL became + * an error message, then an `EvalResult.reason`, then a rendered line on a + * world-readable change page. Both ends are now closed — the store rejects + * control characters (`storage/project-secrets.ts`) and the provider builds its + * headers outside the try, mapping a failure to a constant that interpolates + * nothing (`llm-provider.ts`). + * + * **There is deliberately no MCP surface for writing the credential this + * reads** (PRD §4c). `stratum_set_provider_key` looks like an obvious gap and + * is ruled out on purpose: a provider key must never pass through a model's + * context window, and §7's threat model rests on credentials and endpoints not + * being selectable by the wrong party. The write surface is the web UI or the + * CLI — the same rule that keeps agent tokens from submitting review verdicts. + */ +export async function resolveLlmProvider( + env: LlmByokEnv, + project: ProjectEntry, + policy: EvalPolicy, + logger: Logger, +): Promise { + const configs = llmConfigsOf(policy); + if (configs.length > 1) { + // Never "pick the first one". Which entry wins decides whose bill the run + // lands on, and a policy that cannot say is a policy this must refuse. + return { + status: "unavailable", + reason: `policy declares ${configs.length} "llm" evaluator entries; at most one is allowed, so no provider could be resolved`, + }; + } + + const providerName = configs[0]?.provider; + // The skip. Nothing below this line runs for a project that never opted in. + if (providerName === undefined) return { status: "platform" }; + + const configured = llmProviderCatalog(env, logger).get(providerName); + if (!configured) { + // Unreachable through `parsePolicyContent`, which rejects an unknown name + // and fails the whole file closed. Kept because this function must be safe + // to call with a policy from anywhere — a KV-cached one written before the + // operator's allowlist changed, say — and the fallback must be a refusal. + return { + status: "unavailable", + reason: `policy names LLM provider "${providerName}", which is not configured in LLM_PROVIDERS`, + }; + } + + const secretName = providerSecretName(providerName); + const loaded = await loadSecretValues(env.DB, logger, env, { + projectId: project.id, + names: [secretName], + }); + + if (!loaded.success) { + // `DEPLOY_SECRET_KEY` unset is an operator misconfiguration with a specific + // remedy, and it is the coupling PRD Open Question 4 accepted: BYOK reuses + // the deploy secret key, so rotating that key blocks BYOK merges until the + // provider keys are re-entered. Named explicitly so the reader is not left + // guessing which of the two subsystems broke. + if (loaded.error.code === DEPLOY_SECRET_KEY_MISSING) { + return { + status: "unavailable", + reason: `project secret ${secretName} cannot be decrypted because the operator has not configured DEPLOY_SECRET_KEY`, + }; + } + return { + status: "unavailable", + reason: `project secret ${secretName} could not be read (${loaded.error.code})`, + }; + } + + if (loaded.data.missing.includes(secretName)) { + return { + status: "unavailable", + reason: `project secret ${secretName} is not set for this project; add it before selecting the "${providerName}" provider`, + }; + } + if (loaded.data.undecryptable.includes(secretName)) { + return { + status: "unavailable", + reason: `project secret ${secretName} could not be decrypted; DEPLOY_SECRET_KEY was rotated or the value was restored from another project, so it must be re-entered`, + }; + } + + const apiKey = loaded.data.values.get(secretName); + if (apiKey === undefined) { + // Belt and braces: `loadSecretValues` reports every unresolved name in one + // of the two lists above, so this is not reachable — but the alternative to + // checking is constructing a provider with `undefined` for a credential. + return { + status: "unavailable", + reason: `project secret ${secretName} did not resolve to a value`, + }; + } + + const provider = + configured.kind === "anthropic" + ? new AnthropicProvider(configured.baseUrl, apiKey) + : new OpenAiCompatibleProvider(configured.baseUrl, apiKey); + + return { status: "byok", provider, providerName }; +} diff --git a/src/evaluation/llm-evaluator.ts b/src/evaluation/llm-evaluator.ts index 71bc941d..5a685063 100644 --- a/src/evaluation/llm-evaluator.ts +++ b/src/evaluation/llm-evaluator.ts @@ -1,11 +1,26 @@ -import type { AiBinding } from "../types"; +import { + type EnforcementSubject, + checkMeter, + resolveEnforcementSubject, + settleMeter, +} from "../billing/enforcement"; +import { usagePeriod } from "../storage/usage"; +import type { Env } from "../types"; import type { AppError } from "../utils/errors"; import { ExternalServiceError } from "../utils/errors"; import type { Logger } from "../utils/logger"; import type { Result } from "../utils/result"; import { err, ok } from "../utils/result"; +import type { LlmProvider } from "./llm-provider"; +import { LlmProviderResponseError } from "./llm-provider"; import { sanitizePolicy } from "./sanitize-policy"; -import type { EvalPolicy, EvalResult, Evaluator } from "./types"; +import type { + EvalPolicy, + EvalResult, + EvaluationContext, + Evaluator, + LlmEvaluatorConfig, +} from "./types"; const DEFAULT_MODEL = "@cf/meta/llama-3.1-8b-instruct"; const DEFAULT_THRESHOLD = 0.7; @@ -19,6 +34,23 @@ const MAX_DIFF_CHARS_FLOOR = 1_000; // pathological policy must not blow the context (or starve the diff), so an // oversize one fails closed before any model call. const MAX_POLICY_CONTEXT_CHARS = 8_000; +/** + * Upper bound on the verdict this evaluator asks for, in tokens. + * + * Reserved alongside the input estimate because a reservation has to cover the + * whole spend before any of it happens, and the response is part of it. The + * verdict is a small JSON object — a score, a boolean, one sentence, a short + * issue list — so this is generous by a wide margin on purpose: over-reserving + * is corrected by the settle a few seconds later, while under-reserving is a + * limit that admits more than it says. + */ +const MAX_OUTPUT_TOKENS = 1_000; + +/** + * What the refusals call the thing they are refusing. One string so the two + * metered gates below cannot describe the same evaluator differently. + */ +const GATE_LABEL = "AI review"; const SYSTEM_PROMPT = [ "You are a rigorous code reviewer acting as an automated merge gate.", @@ -30,19 +62,51 @@ const SYSTEM_PROMPT = [ ].join(" "); export class LLMEvaluator implements Evaluator { - constructor(private ai: AiBinding) {} + /** + * @param provider Who runs the model. Workers AI by default; an operator- + * configured HTTP provider on the project's own credential under BYOK. + * @param costSource Who paid for the tokens this run reports. `"byok"` marks + * every cost sample so a project running on its own key does not decrement + * the hosted allowance — the whole point of the field, and something no + * later layer can reconstruct, since the recorded sample is all it sees. + * Defaults to `"platform"`, which the sample omits entirely (absent means + * the operator paid). + * @param enforcement What this evaluator needs to consult an allowance: the + * `Env` holding the entitlements configuration and the `USAGE_METER` + * binding, and a `waitUntil` for warming an org's plan off the response + * path (PRD §4a). Omitted where a caller has no environment to give — + * every check is then skipped, exactly as it is for a self-hoster with no + * billing service configured. + */ + constructor( + private provider: LlmProvider, + private costSource: "platform" | "byok" = "platform", + private enforcement?: { + env: Env; + waitUntil?: (promise: Promise) => void; + }, + ) {} async evaluate( diff: string, policy: EvalPolicy, logger: Logger, + context?: EvaluationContext, ): Promise> { logger.debug("Starting LLM evaluation"); + const nowMs = Date.now(); + const period = usagePeriod(new Date(nowMs)); + /** Set once a reservation is taken, so the `finally` knows what to settle. */ + let subject: EnforcementSubject | null = null; + let tokensReserved: number | null = null; + let rateReserved = false; + let reachedProvider = false; + /** True token cost of the run, once known; settled against the reservation. */ + let tokensSpent = 0; + try { - const config = policy.evaluators.find((e) => e.type === "llm") as - | { type: "llm"; model?: string; threshold?: number; maxDiffChars?: number } - | undefined; + const config = policy.evaluators.find((e): e is LlmEvaluatorConfig => e.type === "llm"); const model = config?.model ?? DEFAULT_MODEL; const threshold = config?.threshold ?? DEFAULT_THRESHOLD; const maxDiffChars = @@ -81,23 +145,158 @@ export class LLMEvaluator implements Evaluator { ]; const promptChars = messages.reduce((sum, m) => sum + m.content.length, 0); - const raw = await this.ai.run(model, { messages }); - if (raw instanceof ReadableStream) { - logger.error("LLM evaluation failed: unexpected stream response"); - return ok({ - score: 0, - passed: false, - reason: "LLM evaluator error: unexpected stream response", + // Allowances are consulted HERE, between building the request and making + // it, and a refusal below returns a FAILING gate rather than an absent + // one. That is the whole shape of this check: a policy that requires AI + // review must not stop requiring it because someone ran out of credit, + // which is what dropping the evaluator (or passing it) would do. It + // mirrors the oversize-policy branch above — `ok()` with `passed: false` + // and a reason a human and an agent can both act on. + if (this.enforcement && context?.billing) { + const { env, waitUntil } = this.enforcement; + subject = await resolveEnforcementSubject(env, logger, { + ...(context.billing.actorUserId !== undefined + ? { actorUserId: context.billing.actorUserId } + : {}), + owner: { ownerId: context.billing.ownerId, ownerType: context.billing.ownerType }, + ...(waitUntil !== undefined ? { waitUntil } : {}), }); + + if (subject) { + // The burst bound first, and it applies to BYOK too (PRD §4b): the + // operator still pays for the subrequest and the Worker wall time, + // whoever owns the tokens. One evaluation, no settle — a rate check + // is a reserve that is never given back. + const rate = await checkMeter(env, logger, { + subject, + meter: "evaluations_per_hour", + estimate: 1, + nowMs, + period, + what: GATE_LABEL, + }); + rateReserved = rate.reserved; + if (!rate.admitted) { + return ok({ + score: 0, + passed: false, + reason: rate.reason ?? `${GATE_LABEL} could not run: evaluation rate limit reached`, + }); + } + + // The token allowance, and the one thing BYOK does lift: a project + // paying its own provider bill is not spending the hosted allowance, + // so charging it against one would be charging twice. + if (this.costSource === "platform") { + // The PROMPT that is about to be sent, not the window it is allowed + // to fill. Reserving `maxDiffChars` charged a three-line diff ~6,000 + // tokens up front, so someone with 5,000 left was refused for a call + // that would have cost 300 — a refusal produced by the estimate + // rather than by the spend. It is still an upper bound: ~4 chars per + // token is the conservative direction for English and code, and the + // response is covered by MAX_OUTPUT_TOKENS on top. + const estimate = Math.ceil(promptChars / 4) + MAX_OUTPUT_TOKENS; + const tokens = await checkMeter(env, logger, { + subject, + meter: "llm_tokens_month", + estimate, + nowMs, + period, + what: GATE_LABEL, + }); + if (tokens.reserved) tokensReserved = estimate; + if (!tokens.admitted) { + return ok({ + score: 0, + passed: false, + reason: tokens.reason ?? `${GATE_LABEL} could not run: token allowance is used up`, + }); + } + } + } } - const responseText = raw.response; - // Workers AI does not report token usage; ~4 chars/token is the standard estimate. - const estimatedTokens = Math.ceil((promptChars + (responseText?.length ?? 0)) / 4); - const costs: EvalResult["costs"] = [ - { kind: "llm_tokens", quantity: estimatedTokens, estimated: true }, - ]; + // Written only for `byok`: an absent `source` already means "platform", + // so always emitting it would add a field that says nothing to every + // sample the operator pays for. + const source = this.costSource === "byok" ? { source: "byok" as const } : {}; + + reachedProvider = true; + const runResult = await this.provider.run(model, messages); + + if (!runResult.success) { + // A response that arrived but cannot be used fails the gate closed with + // a readable reason; a transport failure stays an error the caller + // surfaces. Either way the provider's own message is metadata only and + // becomes user-visible: this block interpolates it into the reason + // below, and `runEvaluation` copies the message of the error the other + // branch returns into a reason of its own. + if (runResult.error instanceof LlmProviderResponseError) { + // The `cause` is logged, not described. It is the only record of what + // the provider actually sent — a `SyntaxError` from `JSON.parse` + // quotes the bytes that failed — and attaching it would be pointless + // if nothing ever read it. It goes to the OPERATOR's log and never + // into the reason below, which is rendered to anyone who can read the + // change; the bound keeps a large body out of a log line. + const cause = runResult.error.cause; + logger.error( + `LLM evaluation failed: ${runResult.error.message}`, + runResult.error, + cause === undefined + ? undefined + : { cause: String(cause instanceof Error ? cause.message : cause).slice(0, 200) }, + ); + // A verdict truncated at the provider's token cap was generated, and + // billed. `failClosed` further down records the cost of a response it + // could not read, and this is the same case one layer up: without it a + // charged call records zero. Only counts the provider actually + // REPORTED are recorded — a failure that produced no tokens (a 401, a + // refused redirect) carries none and bills nothing. + const billed = runResult.error.usage; + if (billed) tokensSpent = billed.inputTokens + billed.outputTokens; + return ok({ + score: 0, + passed: false, + reason: `LLM evaluator error: ${runResult.error.message}`, + ...(billed + ? { + costs: [ + { + kind: "llm_tokens" as const, + quantity: billed.inputTokens + billed.outputTokens, + ...source, + }, + ], + } + : {}), + }); + } + logger.error("LLM evaluation failed", runResult.error); + return err(runResult.error); + } + + const responseText = runResult.data.text; + const usage = runResult.data.usage; + // Real counts when the provider reports them, otherwise ~4 chars/token — + // the distinction is per response, not per source. Every provider behind + // this seam reports counts when it has them, Workers AI included; a + // response that omits them, or reports ones the provider would not trust, + // is what keeps a run on the estimate. + const costs: EvalResult["costs"] = usage + ? [{ kind: "llm_tokens", quantity: usage.inputTokens + usage.outputTokens, ...source }] + : [ + { + kind: "llm_tokens", + quantity: Math.ceil((promptChars + responseText.length) / 4), + estimated: true, + ...source, + }, + ]; + // What the settle in the `finally` corrects the reservation by. Read off + // the same sample the ledger records, so the counter and the ledger + // cannot disagree about what one run cost. + tokensSpent = costs[0]?.quantity ?? 0; // A gate whose verdict can't be read has not passed. Never infer a score // from prose ("LGTM") — that let unparseable output half-approve a merge. @@ -110,7 +309,7 @@ export class LLMEvaluator implements Evaluator { passed: false, reason: `LLM evaluator failed closed: ${why}`, issues: [ - `Model response (${responseText?.length ?? 0} chars) was not a valid verdict object`, + `Model response (${responseText.length} chars) was not a valid verdict object`, ...(truncationNote ? [truncationNote] : []), ], costs, @@ -119,7 +318,7 @@ export class LLMEvaluator implements Evaluator { let parsed: { score: unknown; passed: unknown; reason: unknown; issues?: unknown }; try { - parsed = JSON.parse(responseText ?? "") as { + parsed = JSON.parse(responseText) as { score: unknown; passed: unknown; reason: unknown; @@ -165,6 +364,47 @@ export class LLMEvaluator implements Evaluator { error instanceof Error ? error : undefined, ) as AppError, ); + } finally { + // In a `finally` because a provider call that throws has still consumed + // the reservation. Leaving it standing would charge the subject the whole + // upper bound for a run that produced nothing, and a handful of those + // walls somebody off at a fraction of an allowance they never spent. + const settleSubject = subject; + if (settleSubject && this.enforcement) { + const { env } = this.enforcement; + // Contained deliberately, exactly as `SandboxEvaluator` contains its + // teardown: an `await` that rejects inside `finally` REPLACES the + // returned Result with a rejection, which rejects the `Promise.all` in + // `runEvaluation` and fails change creation outright. A Durable Object + // RPC that cannot be reached is an operational problem to log — a + // correction to a counter, not a merge outage. It is the same failure + // the meter's malformed-input deviation exists to prevent, and it must + // not be reintroduced by the settle path. + const settle = async ( + meter: "llm_tokens_month" | "evaluations_per_hour", + delta: number, + ) => { + try { + await settleMeter(env, logger, { subject: settleSubject, meter, delta, nowMs, period }); + } catch (error) { + logger.error( + "Usage meter settle failed; the reservation stands until the period rolls", + error instanceof Error ? error : new Error(String(error)), + { meter, delta }, + ); + } + }; + if (tokensReserved !== null) { + await settle("llm_tokens_month", tokensSpent - tokensReserved); + } + // A rate slot is given back only when the evaluation never reached the + // provider — a token refusal after the rate check admitted. Once the + // request is out the door the operator has paid for the subrequest and + // the wall time it bounds, so a failed run still costs its slot. + if (rateReserved && !reachedProvider) { + await settle("evaluations_per_hour", -1); + } + } } } } diff --git a/src/evaluation/llm-provider.ts b/src/evaluation/llm-provider.ts new file mode 100644 index 00000000..fb1fae20 --- /dev/null +++ b/src/evaluation/llm-provider.ts @@ -0,0 +1,467 @@ +import type { AiBinding } from "../types"; +import { AppError, ExternalServiceError } from "../utils/errors"; +import type { Result } from "../utils/result"; +import { err, ok } from "../utils/result"; + +/** + * One turn of the conversation handed to a provider. Structurally identical to + * the shape `AiBinding.run` already takes, so `WorkersAiProvider` forwards it + * unchanged. + */ +export interface Message { + role: string; + content: string; +} + +/** Token counts a provider actually reported. Never an estimate — see `LlmResponse.usage`. */ +export interface LlmUsage { + inputTokens: number; + outputTokens: number; +} + +export interface LlmResponse { + /** The assistant's text. Empty string when the provider returned a well-formed + * response carrying no text, which the evaluator then fails closed on. */ + text: string; + /** + * Present only when the provider reported real token counts. Its absence is + * load-bearing: the evaluator falls back to the `~4 chars/token` estimate and + * marks the cost sample `estimated`. That distinction is **per response, not + * per source**: every provider here reports counts when it has them (Workers + * AI included), and any response that omits `usage` — or reports one this + * file will not trust, see `usageOrNothing` — is estimated. + */ + usage?: LlmUsage; +} + +/** + * The seam between the LLM evaluator and whoever actually runs the model. + * + * Errors are values: a provider never throws across this boundary. Two error + * shapes are distinguished, because they mean different things to the gate: + * `LlmProviderResponseError` for a response that arrived but cannot be used + * (which fails the evaluation closed with a readable reason), and any other + * `AppError` for a transport failure. + */ +export interface LlmProvider { + run(model: string, messages: Message[]): Promise>; +} + +/** + * The provider answered, but the answer is unusable — a non-2xx status, a body + * that is not JSON, a JSON body of the wrong shape, a verdict truncated at the + * token cap, or a stream where a completion was expected. + * + * **The message is metadata only.** It reaches the user: `LLMEvaluator` + * intercepts every error of this class and interpolates the message into the + * `EvalResult.reason` it fails the gate closed with. A provider's error body + * can quote the request — which contains the diff — and some providers echo the + * credential that failed, so nothing the provider sent is ever interpolated + * into it: these messages carry an HTTP status, a fixed description of the + * shape problem, or a constant this file chose, and nothing else. + */ +export class LlmProviderResponseError extends AppError { + /** + * Token counts the provider reported for a response that cannot be used. + * + * Present only where the provider *billed* the call anyway — a verdict + * truncated at the token cap is a completed, charged generation. The + * evaluator records it as a cost sample, so a run the provider charged for is + * not recorded as zero. Absent for every failure that produced no tokens (a + * non-2xx status, an unreadable body), which must not be billed. + */ + readonly usage?: LlmUsage; + + constructor(message: string, cause?: unknown, usage?: LlmUsage) { + super(message, "LLM_PROVIDER_RESPONSE", 502); + this.name = "LlmProviderResponseError"; + // Attached rather than described: a `SyntaxError` from `JSON.parse` quotes + // the bytes that failed, which are the provider's body — exactly what the + // message must not carry. Dropping it entirely would swallow the only + // record of what went wrong, and `LLMEvaluator` logs it (operator-visible) + // without ever putting it in the user-visible reason. + if (cause !== undefined) this.cause = cause; + if (usage !== undefined) this.usage = usage; + } +} + +function transportError(error: unknown): AppError { + const message = error instanceof Error ? error.message : String(error); + return new ExternalServiceError( + "LLM", + message, + error instanceof Error ? error : undefined, + ) as AppError; +} + +/** How long any provider gets before the gate gives up on it. */ +const DEFAULT_TIMEOUT_MS = 30_000; + +/** + * The two constant messages the request path can fail with. Constants rather + * than interpolations because both are reached with attacker-influenced values + * in hand — a redirect `Location`, or a header value that could not be built — + * and every message from this file becomes a user-visible evaluation reason. + */ +const REDIRECT_REFUSED = "provider redirected the request, which is not followed"; +const HEADER_CONSTRUCTION_FAILED = "provider request headers could not be constructed"; + +export interface ProviderOptions { + /** Give up on the request after this many milliseconds. */ + timeoutMs?: number; +} + +/** + * Bound a call that cannot be cancelled. + * + * `AiBinding.run` takes no `AbortSignal`, so unlike the HTTP providers there is + * nothing here to abort: the inference keeps running (and is still billed) + * after this rejects. What it bounds is how long the *gate* waits, which is the + * failure that matters — a binding that never settles otherwise holds the + * evaluation open for as long as the Worker's own limits allow. + */ +function withTimeout(work: Promise, timeoutMs: number, what: string): Promise { + let timer: ReturnType | undefined; + const expiry = new Promise((_resolve, reject) => { + timer = setTimeout( + () => reject(new Error(`${what} did not respond within ${timeoutMs}ms`)), + timeoutMs, + ); + }); + return Promise.race([work, expiry]).finally(() => { + if (timer !== undefined) clearTimeout(timer); + }); +} + +/** + * The Workers AI binding, and the default. Behaviour here is deliberately + * identical to what `LLMEvaluator` did inline before the seam existed: + * a `ReadableStream` is rejected rather than consumed, and a response object + * with no usable `response` field yields `""` so the evaluator fails it closed + * as unparseable output rather than treating it as a distinct condition. + * + * It does report token counts — Cloudflare documents the synchronous output of + * a text-generation model as `response`, `tool_calls` and `usage`, and its own + * generated types name the counts `usage.prompt_tokens` / + * `usage.completion_tokens` (`AiTextGenerationOutput` in + * `@cloudflare/workers-types`). They go through the same `usageOrNothing` + * validation as the HTTP providers, so this path falls back to the + * `~4 chars/token` estimate only when a response actually omits them. + */ +export class WorkersAiProvider implements LlmProvider { + constructor( + private ai: AiBinding, + private options: ProviderOptions = {}, + ) {} + + async run(model: string, messages: Message[]): Promise> { + let raw: Awaited>; + try { + raw = await withTimeout( + this.ai.run(model, { messages }), + this.options.timeoutMs ?? DEFAULT_TIMEOUT_MS, + "Workers AI", + ); + } catch (error) { + return err(transportError(error)); + } + + if (raw instanceof ReadableStream) { + return err(new LlmProviderResponseError("unexpected stream response")); + } + + // Checked, not asserted: the binding's own types make `response` optional, + // and the evaluator dereferences `.length` on whatever comes back. + const text = typeof raw.response === "string" ? raw.response : ""; + return ok({ + text, + ...usageOrNothing(raw.usage?.prompt_tokens, raw.usage?.completion_tokens), + }); + } +} + +/** + * Credentials and endpoints for both HTTP providers are **constructor + * arguments supplied by the caller** — never read from `env` here, and never + * taken from `.stratum/policy.yaml`. + * + * The omission is deliberate, not a gap. A base URL a repository's own policy + * file can choose turns this Worker into a request-forgery primitive aimed at + * whatever host the policy names, and the prompt it would receive contains the + * diff (PRD §7). Resolving a provider name against an operator-configured + * allowlist is Task 7's job; these classes only take what they are handed. + */ +abstract class HttpLlmProvider implements LlmProvider { + constructor( + protected readonly baseUrl: string, + protected readonly apiKey: string, + protected readonly options: ProviderOptions = {}, + ) {} + + abstract run(model: string, messages: Message[]): Promise>; + + /** + * POST a JSON body and parse a JSON response, converting every failure into + * a `Result`. Returns the decoded body as `unknown`: each provider validates + * its own shape, because a body that parsed is not a body that can be read. + */ + protected async post(path: string, headers: Record, body: unknown) { + // Built HERE, outside the try, and never from inside it. `new Headers` throws + // a `TypeError` whose message QUOTES the offending value, and the value here + // is the project's API key: inside the try that message would become a + // transport error, then an `ExternalServiceError`, then `EvalResult.reason`, + // which is persisted on the change and rendered on a page that is + // world-readable for a public project. The store rejects control characters + // in a secret value (`storage/project-secrets.ts`) so this should be + // unreachable; it is the second of the two ends, and it maps to a constant + // that interpolates nothing. + let requestHeaders: Headers; + try { + requestHeaders = new Headers({ "content-type": "application/json", ...headers }); + } catch { + return err(new LlmProviderResponseError(HEADER_CONSTRUCTION_FAILED) as AppError); + } + + const controller = new AbortController(); + const timer = setTimeout( + () => controller.abort(), + this.options.timeoutMs ?? DEFAULT_TIMEOUT_MS, + ); + + try { + const response = await fetch(`${this.baseUrl.replace(/\/+$/, "")}${path}`, { + method: "POST", + headers: requestHeaders, + body: JSON.stringify(body), + signal: controller.signal, + // Never follow a redirect: the allowlist binds the host this request was + // *sent* to, and a followed 3xx would re-send the prompt (the diff and + // the policy) to a host nobody validated — with `x-api-key` still + // attached, since the Fetch spec strips only `Authorization` + // cross-origin. The sibling `webhook-evaluator.ts` does the same. + redirect: "manual", + }); + + if (response.status >= 300 && response.status < 400) { + return err(new LlmProviderResponseError(REDIRECT_REFUSED) as AppError); + } + + if (!response.ok) { + // The status, never the body: an error body can echo the API key that + // failed, and this message becomes a user-visible evaluation reason. + return err( + new LlmProviderResponseError(`provider returned HTTP ${response.status}`) as AppError, + ); + } + + // Read the bytes first, parse them second. `response.json()` rejects both + // when the transfer drops mid-body and when a body that arrived whole is + // not JSON, and those mean opposite things to the gate: the first is a + // transport failure the caller surfaces (it falls to the outer catch), + // the second fails the evaluation closed with a user-visible reason. + const raw = await response.text(); + try { + return ok(JSON.parse(raw) as unknown); + } catch (error) { + return err( + new LlmProviderResponseError("provider response was not valid JSON", error) as AppError, + ); + } + } catch (error) { + return err(transportError(error)); + } finally { + clearTimeout(timer); + } + } +} + +/** + * The output budget for one verdict, shared by both HTTP providers so the gate + * is bounded by the same number whichever one runs it. The evaluator asks for a + * single small JSON object; a response that needs more than this has stopped + * answering the question it was asked. + */ +const VERDICT_TOKEN_BUDGET = 1024; + +/** + * Anthropic's Messages API: `POST {baseUrl}/messages`, `x-api-key` plus the + * required `anthropic-version` header, assistant text in `content[]` blocks of + * type `text`, real token counts in `usage.input_tokens` / `usage.output_tokens`. + * + * Docs: https://platform.claude.com/docs/en/api/messages/create and + * https://platform.claude.com/docs/en/api/overview (headers table). + */ +export class AnthropicProvider extends HttpLlmProvider { + /** Pinned rather than configurable: the API is versioned by this header, and + * a value the caller could vary is a shape this code has not been written for. */ + static readonly API_VERSION = "2023-06-01"; + /** The evaluator asks for one small JSON verdict; `max_tokens` is required. */ + static readonly MAX_TOKENS = VERDICT_TOKEN_BUDGET; + + async run(model: string, messages: Message[]): Promise> { + // Anthropic takes the system prompt as a top-level parameter, not as a + // message with role "system" — a system-role message is rejected. + const system = messages + .filter((m) => m.role === "system") + .map((m) => m.content) + .join("\n\n"); + const turns = messages.filter((m) => m.role !== "system"); + + const posted = await this.post( + "/messages", + { + "x-api-key": this.apiKey, + "anthropic-version": AnthropicProvider.API_VERSION, + }, + { + model, + max_tokens: AnthropicProvider.MAX_TOKENS, + ...(system.length > 0 ? { system } : {}), + messages: turns, + }, + ); + if (!posted.success) return posted; + + const body = posted.data as { + content?: unknown; + stop_reason?: unknown; + usage?: { input_tokens?: unknown; output_tokens?: unknown }; + } | null; + + if (body === null || typeof body !== "object" || !Array.isArray(body.content)) { + return err( + new LlmProviderResponseError("provider response had no content array") as AppError, + ); + } + + // A verdict cut off at our own cap is well-formed JSON containing half a + // JSON object, so without this the evaluator fails closed blaming the + // model's formatting for a limit this file chose. Anthropic says so + // explicitly: `stop_reason` is `"max_tokens"` in exactly that case. + if (body.stop_reason === "max_tokens") { + // With the counts: the generation ran to the cap and the provider billed + // it, so an unusable verdict here still costs real tokens. + return err( + new LlmProviderResponseError( + `provider response was truncated at the ${AnthropicProvider.MAX_TOKENS}-token limit`, + undefined, + usageOrNothing(body.usage?.input_tokens, body.usage?.output_tokens).usage, + ) as AppError, + ); + } + + const text = body.content + .filter( + (block): block is { type: "text"; text: string } => + typeof block === "object" && + block !== null && + (block as { type?: unknown }).type === "text" && + typeof (block as { text?: unknown }).text === "string", + ) + .map((block) => block.text) + .join(""); + + return ok({ text, ...usageOrNothing(body.usage?.input_tokens, body.usage?.output_tokens) }); + } +} + +/** + * The OpenAI Chat Completions shape, which OpenAI, OpenRouter, Groq, Together + * and a self-hosted vLLM server all speak: `POST {baseUrl}/chat/completions` + * with `Authorization: Bearer`, assistant text at `choices[0].message.content`, + * token counts at `usage.prompt_tokens` / `usage.completion_tokens`. + * + * Docs: https://developers.openai.com/api/reference/chat-completions/overview, + * https://openrouter.ai/docs/api-reference/chat-completion, + * https://console.groq.com/docs/api-reference, + * https://docs.vllm.ai/en/latest/serving/openai_compatible_server/. + */ +export class OpenAiCompatibleProvider extends HttpLlmProvider { + /** The same budget the Anthropic sibling sends; see `VERDICT_TOKEN_BUDGET`. */ + static readonly MAX_COMPLETION_TOKENS = VERDICT_TOKEN_BUDGET; + + async run(model: string, messages: Message[]): Promise> { + // `max_completion_tokens`, never `max_tokens`: OpenAI documents `max_tokens` + // as deprecated in its favour and "not compatible with o-series models", + // Groq documents the same deprecation, and OpenRouter and vLLM take either. + // Together AI documents only `max_tokens` and its schema does not forbid + // extra fields, so there the bound is still the timeout — an unenforced + // field on one vendor beats a 400 on the ones that reject `max_tokens`. + const posted = await this.post( + "/chat/completions", + { authorization: `Bearer ${this.apiKey}` }, + { + model, + messages, + max_completion_tokens: OpenAiCompatibleProvider.MAX_COMPLETION_TOKENS, + }, + ); + if (!posted.success) return posted; + + const body = posted.data as { + choices?: unknown; + usage?: { prompt_tokens?: unknown; completion_tokens?: unknown }; + } | null; + + if (body === null || typeof body !== "object" || !Array.isArray(body.choices)) { + return err( + new LlmProviderResponseError("provider response had no choices array") as AppError, + ); + } + + const choice = body.choices[0] as + | { message?: { content?: unknown }; finish_reason?: unknown } + | undefined; + + // The same truncation the Anthropic path reports, under this shape's name + // for it: `finish_reason: "length"` means the cap above ended the verdict, + // not the model, and the half-object left behind must not be reported as + // the model's own formatting failure. + if (choice?.finish_reason === "length") { + // Billed, like the Anthropic sibling — see the note there. + return err( + new LlmProviderResponseError( + `provider response was truncated at the ${OpenAiCompatibleProvider.MAX_COMPLETION_TOKENS}-token limit`, + undefined, + usageOrNothing(body.usage?.prompt_tokens, body.usage?.completion_tokens).usage, + ) as AppError, + ); + } + + const content = choice?.message?.content; + if (typeof content !== "string") { + return err( + new LlmProviderResponseError("provider response had no assistant message text") as AppError, + ); + } + + return ok({ + text: content, + ...usageOrNothing(body.usage?.prompt_tokens, body.usage?.completion_tokens), + }); + } +} + +/** + * A count that could actually be a number of tokens: a non-negative integer. + * + * `Number.isInteger` already excludes `NaN` and both infinities, so the whole + * test is integrality plus sign. Both halves matter downstream — a negative + * quantity becomes a ledger row for negative spend, and a fractional one a + * fractional token — and neither is a count any provider can legitimately + * report, so a value failing this is a malformed `usage`, not a small error. + */ +function isTokenCount(value: unknown): value is number { + return typeof value === "number" && Number.isInteger(value) && value >= 0; +} + +/** + * Report usage only when both counts are usable. A partial or malformed + * `usage` object is treated as absent rather than half-trusted: a wrong token + * count recorded as exact is worse than an estimate labelled as one — the + * estimate at least carries `estimated: true` into `cost_records`. + */ +function usageOrNothing(input: unknown, output: unknown): { usage?: LlmUsage } { + if (!isTokenCount(input) || !isTokenCount(output)) return {}; + return { usage: { inputTokens: input, outputTokens: output } }; +} diff --git a/src/evaluation/llm-providers.ts b/src/evaluation/llm-providers.ts new file mode 100644 index 00000000..aa3766ec --- /dev/null +++ b/src/evaluation/llm-providers.ts @@ -0,0 +1,273 @@ +/** + * The operator's LLM provider allowlist, parsed from the `LLM_PROVIDERS` var. + * + * This is the **only** place a provider endpoint may be named. A project's + * `.stratum/policy.yaml` may SELECT one of these by name and can never supply a + * `baseUrl` (PRD §7): a base URL a repository's own policy chose would turn the + * operator's Worker into a request-forgery primitive aimed at whatever host the + * policy names, and the prompt it would receive contains the diff. + * + * Every URL rule below is checked once, here, at parse time — not per request. + * Per-request checking would re-validate an immutable value on the hot path and, + * worse, would mean a bad entry is discovered by the first project unlucky + * enough to select it rather than by the operator who wrote it. + */ +import type { Env } from "../types"; +import type { Logger } from "../utils/logger"; +import { privateHostReason } from "../utils/validation"; + +/** + * Wire protocols the evaluator can speak, one per HTTP implementation in + * `llm-provider.ts`. A third kind needs a class before it can be a value here. + */ +export const LLM_PROVIDER_KINDS = ["anthropic", "openai-compatible"] as const; +export type LlmProviderKind = (typeof LLM_PROVIDER_KINDS)[number]; + +const KIND_SET = new Set(LLM_PROVIDER_KINDS); + +/** One operator-configured provider. Credentials are deliberately absent: they + * live per project in `project_secrets`, never in an environment variable that + * every project's policy can select. */ +export interface ConfiguredLlmProvider { + name: string; + kind: LlmProviderKind; + baseUrl: string; +} + +/** Provider name to definition. Empty means "Workers AI only" — the default. */ +export type LlmProviderCatalog = ReadonlyMap; + +/** Parsing either yields a catalog or says why the operator's config is unusable. */ +export type LlmProvidersParse = + | { status: "ok"; providers: LlmProviderCatalog } + | { status: "invalid"; reason: string }; + +/** Fields an `LLM_PROVIDERS` entry may carry. */ +const ENTRY_KEYS = new Set(["name", "kind", "baseUrl"]); + +/** + * Provider names are as narrow as deploy names, and for the same reason: a + * policy file selects one by name, and the name is echoed into log lines, + * evaluation reasons, and the derived secret name. + */ +const PROVIDER_NAME_PATTERN = /^[a-z][a-z0-9-]{0,31}$/; + +/** Enough for every provider one instance realistically fronts; bounds the parse. */ +const MAX_PROVIDERS = 16; + +const EMPTY_CATALOG: LlmProviderCatalog = new Map(); + +/** + * The `project_secrets` name holding this provider's credential. + * + * Derived rather than configured so an entry cannot name an arbitrary secret: + * `LLM_PROVIDERS` is operator config, but making the secret *name* configurable + * would let one provider entry read a credential a project stored for something + * else entirely. `PROVIDER_NAME_PATTERN` guarantees the result matches + * `SECRET_NAME_PATTERN` (`^[A-Z][A-Z0-9_]{0,63}$`) — an uppercased 32-char + * name plus the suffix is 40. + */ +export function providerSecretName(providerName: string): string { + return `${providerName.toUpperCase().replace(/-/g, "_")}_API_KEY`; +} + +/** + * Why this host must not be fetched, or null when it is fine. + * + * A thin alias for `privateHostReason` (`utils/validation.ts`), which is the + * one host filter in the codebase and is also what `validateWebhookUrl` uses. + * This used to be a second implementation, and the two drifted exactly as a + * duplicated security filter always does: this copy missed CGNAT, the + * `.internal` suffix (GCP's metadata endpoint is `metadata.google.internal`), + * everything in `fe80::/10` not literally spelled `fe80`, and the + * IPv4-compatible IPv6 form `[::127.0.0.1]`, which the URL parser hands on as + * `[::7f00:1]`. + * + * Kept as a named export because the reason is interpolated into the operator's + * config error, and because the allowlist is where this rule is *enforced*: + * every URL rule is checked once, at parse time, not per request. + */ +export function blockedHostReason(hostname: string): string | null { + return privateHostReason(hostname); +} + +/** A validated `baseUrl`, normalized, or why the entry cannot be used. */ +type BaseUrlCheck = { ok: true; baseUrl: string } | { ok: false; reason: string }; + +function checkBaseUrl(name: string, raw: unknown): BaseUrlCheck { + if (typeof raw !== "string" || raw.length === 0) { + return { ok: false, reason: `provider "${name}": "baseUrl" is required and must be a string` }; + } + + let url: URL; + try { + url = new URL(raw); + } catch { + return { ok: false, reason: `provider "${name}": "baseUrl" is not a valid absolute URL` }; + } + + if (url.protocol !== "https:") { + // Not tidiness: the request carries the project's API key and the diff. + return { + ok: false, + reason: `provider "${name}": "baseUrl" must use https, not ${url.protocol.replace(":", "")}`, + }; + } + if (url.username !== "" || url.password !== "") { + return { ok: false, reason: `provider "${name}": "baseUrl" must not embed credentials` }; + } + if (url.search !== "" || url.hash !== "") { + return { + ok: false, + reason: `provider "${name}": "baseUrl" must not carry a query string or fragment`, + }; + } + const blocked = blockedHostReason(url.hostname); + if (blocked) { + return { + ok: false, + reason: `provider "${name}": "baseUrl" host is not reachable from an allowlist — ${blocked}`, + }; + } + + // Normalized, never the raw string. `https://host#` and `https://host?` both + // pass the two emptiness checks above — the parser reports an empty hash and + // an empty search for them — while the bytes the operator wrote would put the + // provider's `/messages` path inside a fragment or a query. Origin plus path + // is what `href` would give minus exactly those two, and the trailing slash + // goes because the provider classes append an absolute path. + return { ok: true, baseUrl: `${url.origin}${url.pathname}`.replace(/\/+$/, "") }; +} + +/** + * Parse `LLM_PROVIDERS`, rejecting the whole document on the first problem. + * + * All-or-nothing rather than "keep the entries that parsed", because a + * half-applied allowlist is the failure this is meant to prevent: an operator + * who typo'd one entry would get merges blocked on the projects selecting it + * with nothing anywhere saying why. One rejection with a reason is louder, and + * `llmProvidersConfigError` puts it in the logs on the first request after a + * bad deploy. + * + * An unset (or blank) value is not an error — it is the default, and it means + * Workers AI only. + */ +export function parseLlmProviders(raw: string | undefined): LlmProvidersParse { + if (raw === undefined || raw.trim() === "") return { status: "ok", providers: EMPTY_CATALOG }; + + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (error) { + return { + status: "invalid", + reason: `LLM_PROVIDERS is not valid JSON (${error instanceof Error ? error.message : "parse error"})`, + }; + } + + if (!Array.isArray(parsed)) { + return { status: "invalid", reason: "LLM_PROVIDERS must be a JSON array of provider objects" }; + } + if (parsed.length > MAX_PROVIDERS) { + return { + status: "invalid", + reason: `LLM_PROVIDERS declares ${parsed.length} providers; at most ${MAX_PROVIDERS} are allowed`, + }; + } + + const providers = new Map(); + + for (const [index, entry] of parsed.entries()) { + const at = `LLM_PROVIDERS[${index}]`; + if (typeof entry !== "object" || entry === null || Array.isArray(entry)) { + return { status: "invalid", reason: `${at} must be an object` }; + } + const source = entry as Record; + + const unknownKeys = Object.keys(source).filter((key) => !ENTRY_KEYS.has(key)); + if (unknownKeys.length > 0) { + // Rejected rather than ignored: a misspelled key here is silently + // dropped configuration on a security boundary, and `baseURL` for + // `baseUrl` is exactly the typo an operator would not notice. + return { + status: "invalid", + reason: `${at} has unrecognized field(s) ${unknownKeys.map((k) => `"${k}"`).join(", ")}`, + }; + } + + if (typeof source.name !== "string" || !PROVIDER_NAME_PATTERN.test(source.name)) { + return { + status: "invalid", + reason: `${at}: "name" must match ${PROVIDER_NAME_PATTERN.source} (lowercase letters, digits and dashes, max 32 chars)`, + }; + } + const name = source.name; + if (providers.has(name)) { + return { status: "invalid", reason: `${at}: duplicate provider name "${name}"` }; + } + + if (typeof source.kind !== "string" || !KIND_SET.has(source.kind)) { + return { + status: "invalid", + reason: `provider "${name}": "kind" must be one of ${LLM_PROVIDER_KINDS.join(", ")}`, + }; + } + + const url = checkBaseUrl(name, source.baseUrl); + if (!url.ok) return { status: "invalid", reason: url.reason }; + + providers.set(name, { + name, + kind: source.kind as LlmProviderKind, + baseUrl: url.baseUrl, + }); + } + + return { status: "ok", providers }; +} + +/** + * The parse memoized for the isolate's lifetime, keyed on the raw value so a + * test (or a rebind) that changes it is not served a stale catalog. + */ +let cached: { raw: string | undefined; parse: LlmProvidersParse } | null = null; + +/** Drop the memoized parse. Exists for tests; production rebinds per isolate. */ +export function resetLlmProviderCache(): void { + cached = null; +} + +function parseCached(raw: string | undefined): LlmProvidersParse { + if (cached === null || cached.raw !== raw) cached = { raw, parse: parseLlmProviders(raw) }; + return cached.parse; +} + +/** + * The catalog for this environment, or an empty one when the config is unusable. + * + * Empty-on-invalid is the fail-closed direction, not a silent disable: an empty + * catalog means every policy naming a provider is rejected by + * `sanitizeLlmConfig`, which fails the policy file closed and blocks merges with + * a reason that NAMES the unconfigured provider — the rejection it records is + * carried into `parsePolicyContent`'s malformed reason, so the person fixing the + * file is not left with a bare count of unusable entries. Projects on Workers AI + * — which is every project that never opted into BYOK — are untouched by an + * operator's typo. The typo itself is reported by `llmProvidersConfigError` and + * logged here on the isolate's first parse. + */ +export function llmProviderCatalog( + env: Pick, + logger?: Logger, +): LlmProviderCatalog { + const hadCache = cached !== null && cached.raw === env.LLM_PROVIDERS; + const parse = parseCached(env.LLM_PROVIDERS); + if (parse.status === "invalid") { + if (!hadCache) { + logger?.error("LLM_PROVIDERS is invalid; BYOK providers are unavailable", undefined, { + reason: parse.reason, + }); + } + return EMPTY_CATALOG; + } + return parse.providers; +} diff --git a/src/evaluation/policy-loader.ts b/src/evaluation/policy-loader.ts index 206638ee..6eadb47b 100644 --- a/src/evaluation/policy-loader.ts +++ b/src/evaluation/policy-loader.ts @@ -9,7 +9,14 @@ import { MIN_PHASE_TIMEOUT_MS, MIN_TOTAL_BUDGET_MS, } from "./limits"; -import type { EvalPolicy, EvaluatorConfig, MergePolicy, SandboxEvaluatorConfig } from "./types"; +import type { LlmProviderCatalog } from "./llm-providers"; +import type { + EvalPolicy, + EvaluatorConfig, + LlmEvaluatorConfig, + MergePolicy, + SandboxEvaluatorConfig, +} from "./types"; const DEFAULT_POLICY: EvalPolicy = { evaluators: [{ type: "diff" }], @@ -25,6 +32,42 @@ const DEFAULT_POLICY: EvalPolicy = { */ export const PROTECTED_CONFIG_FILES = [".stratum/policy.yaml", "stratum.config.json"] as const; +/** + * Most `evaluators:` entries one policy file may declare. + * + * The `deploys:` sibling has had `MAX_DEPLOY_ENTRIES` from the start and for + * the same reason, which applies at least as strongly here: every entry runs + * inside one Worker invocation, sharing its CPU, memory and subrequest budget, + * and an `llm` or `webhook` entry is an outbound call. Without a cap a policy + * file amplifies one merge into an unbounded number of provider requests. + * + * Unlike a rejected deploy entry this fails the whole file closed, the way + * every other unusable evaluator entry does: an evaluator is a gate, and + * quietly running the first sixteen of a hundred gates is not "most of the + * policy", it is a policy nobody wrote. + */ +export const MAX_EVALUATOR_ENTRIES = 16; + +/** + * Most `llm` entries one policy file may declare. + * + * One, because two cannot be resolved: `LLMEvaluator` reads the first entry's + * model and threshold while `buildEvaluators` builds an evaluator for every + * entry, so `[{llm}, {llm, provider: x}]` used to mean two runs on the + * operator's Workers AI bill from a policy that asked for one on its own key. + * Rejecting the file is the only answer that does not silently pick one. + */ +const MAX_LLM_ENTRIES = 1; + +/** + * The operator's provider allowlist, as the policy parser sees it. + * + * Defaulted to empty rather than made required so a caller that forgets to pass + * one blocks BYOK policies instead of admitting them: an empty catalog resolves + * no provider name, and an unresolved name fails the file closed. + */ +const NO_PROVIDERS: LlmProviderCatalog = new Map(); + /** Does a unified diff (git-style `diff --git a/… b/…` headers) modify a * protected merge-protection config file? */ export function diffTouchesProtectedConfig(diff: string): boolean { @@ -44,6 +87,9 @@ export async function loadPolicy( token: string, logger: Logger, branch = "main", + /** Operator-configured LLM providers a policy may select from. Omitted means + * none are configured, so any policy naming one fails closed. */ + providers: LlmProviderCatalog = NO_PROVIDERS, ): Promise { const yaml = await readAndParsePolicy( remote, @@ -52,6 +98,7 @@ export async function loadPolicy( "yaml", logger, branch, + providers, ); if (yaml.status === "ok") return yaml.policy; if (yaml.status === "malformed") @@ -64,6 +111,7 @@ export async function loadPolicy( "json", logger, branch, + providers, ); if (json.status === "ok") return json.policy; if (json.status === "malformed") @@ -115,6 +163,7 @@ export function parsePolicyContent( content: string, format: "json" | "yaml", logger: Logger = defaultLogger, + providers: LlmProviderCatalog = NO_PROVIDERS, ): PolicyParse { let parsed: unknown; try { @@ -161,8 +210,20 @@ export function parsePolicyContent( // the parsed input (or, via `DEFAULT_POLICY`'s own array, into the module // default) by mutating what it was handed. const declared = raw.evaluators as unknown[]; + if (declared.length > MAX_EVALUATOR_ENTRIES) { + return { + status: "malformed", + reason: `'evaluators' declares ${declared.length} entries; at most ${MAX_EVALUATOR_ENTRIES} are allowed`, + }; + } + + // Why each dropped entry was dropped, in the file's own terms. Collected + // rather than only logged because the count alone ("2 unusable entries") + // is what the merge gate shows the person who has to fix the file, and a + // count does not name the provider that is not configured. + const rejections: string[] = []; const evaluators = declared - .map((entry) => sanitizeEvaluator(entry, logger)) + .map((entry, index) => sanitizeEvaluator(entry, logger, providers, rejections, index)) .filter((entry): entry is EvaluatorConfig => entry !== null); // Any dropped entry fails the gate closed — not just the case where every @@ -178,11 +239,24 @@ export function parsePolicyContent( // `buildEvaluators`, so a policy naming a future evaluator type does not // trip this. if (evaluators.length < declared.length) { + const dropped = declared.length - evaluators.length; + return { + status: "malformed", + reason: `${dropped} unusable entr${dropped === 1 ? "y" : "ies"} in 'evaluators'${ + rejections.length > 0 ? `: ${rejections.join("; ")}` : "" + }`, + }; + } + + // One `llm` entry, checked here rather than inside `sanitizeLlmConfig` + // because it is a property of the LIST, not of an entry — and it fails the + // file closed rather than dropping the extra, so nothing has to decide + // which of two entries the project meant. + const llmEntries = evaluators.filter((entry) => entry.type === "llm").length; + if (llmEntries > MAX_LLM_ENTRIES) { return { status: "malformed", - reason: `${declared.length - evaluators.length} unusable entr${ - declared.length - evaluators.length === 1 ? "y" : "ies" - } in 'evaluators'`, + reason: `'evaluators' declares ${llmEntries} 'llm' entries; at most ${MAX_LLM_ENTRIES} is allowed, because a second one would run a second model call on a configuration nothing can choose between`, }; } policy.evaluators = evaluators; @@ -223,6 +297,7 @@ async function readAndParsePolicy( format: "json" | "yaml", logger: Logger, branch = "main", + providers: LlmProviderCatalog = NO_PROVIDERS, ): Promise { try { const contentResult = await readFileFromRepo(remote, token, path, logger, branch); @@ -231,7 +306,7 @@ async function readAndParsePolicy( const content = contentResult.data; if (content === null || content === undefined) return { status: "absent" }; - return parsePolicyContent(content, format, logger); + return parsePolicyContent(content, format, logger, providers); } catch (e) { // A transient repo-read blip — treat as absent so it doesn't block every // merge. Sanitization failures cannot reach here; `parsePolicyContent` @@ -404,6 +479,126 @@ function sanitizeSandboxConfig( return config; } +/** Fields an `llm` evaluator entry may carry. Everything else is ignored. */ +const LLM_CONFIG_KEYS = new Set(["type", "provider", "model", "threshold", "maxDiffChars"]); + +/** + * Longest `model` a policy may name. Generous for every provider's identifiers + * (`@cf/meta/llama-3.1-8b-instruct`, `anthropic/claude-sonnet-4-5`) and short + * enough that the value cannot become a payload in its own right. + */ +const MAX_MODEL_LENGTH = 128; + +/** Model identifiers across the three providers: slugs with `/`, `.`, `:`, `@`, `-`, `_`. */ +const MODEL_PATTERN = /^[A-Za-z0-9._:@/-]+$/; + +/** + * Keep only whitelisted, well-typed fields from a user-supplied `llm` entry, + * resolving `provider` against the operator's allowlist. + * + * Returning null fails the **whole policy file** closed and blocks merges (see + * the dropped-entry check in `parsePolicyContent`). That is the opposite of + * what a rejected `deploys:` entry does two files away — `deploy/config.ts` + * says in as many words "do not fix this" — and the asymmetry is deliberate: + * a deploy runs *after* the merge, so a bad entry costs a failed deployment + * row, whereas an `llm` entry is a gate. A gate that cannot run must not pass. + * An unresolvable provider name is precisely a gate that cannot run, so it + * blocks rather than degrading to Workers AI on the operator's bill. + * + * `baseUrl` is never read here, at any level of validation. The endpoint set is + * closed and lives in `LLM_PROVIDERS`; a policy may only *select* from it by + * name. A `baseUrl` in the file is warned about and dropped like any other + * unrecognized key — see `llm-providers.ts` for why the field cannot exist. + * + * @param rejections Collects why an entry was dropped, so the merge-blocking + * reason can name the provider instead of only counting entries. Optional: + * a caller that only wants the config need not care. + */ +export function sanitizeLlmConfig( + source: Record, + logger: Logger, + providers: LlmProviderCatalog, + rejections?: string[], +): LlmEvaluatorConfig | null { + const config: LlmEvaluatorConfig = { type: "llm" }; + const reject = (reason: string): null => { + rejections?.push(reason); + return null; + }; + + for (const key of Object.keys(source)) { + if (!LLM_CONFIG_KEYS.has(key)) { + logger.warn("Ignoring unrecognized llm evaluator field", { field: key }); + } + } + + if (source.provider !== undefined) { + if (typeof source.provider !== "string") { + logger.error("Dropping llm evaluator entry whose provider is not a string", undefined, { + submitted: forLog(source.provider), + }); + return reject("llm evaluator: 'provider' must be a string naming a configured provider"); + } + if (!providers.has(source.provider)) { + logger.error( + "Dropping llm evaluator entry naming an unconfigured provider — failing the policy closed", + undefined, + { provider: forLog(source.provider) }, + ); + return reject( + `llm evaluator names provider "${forLog(source.provider)}", which this instance has not configured in LLM_PROVIDERS`, + ); + } + config.provider = source.provider; + } + + if (source.model !== undefined) { + const model = typeof source.model === "string" ? source.model.trim() : ""; + if (model.length === 0 || model.length > MAX_MODEL_LENGTH || !MODEL_PATTERN.test(model)) { + logger.error("Dropping llm evaluator entry with an unusable model", undefined, { + submitted: forLog(source.model), + }); + return reject("llm evaluator: 'model' is not a usable model identifier"); + } + config.model = model; + } + + // A BYOK entry must name its model. The default (`@cf/meta/llama-3.1-8b-instruct`) + // is a Workers AI model id, and posting it to Anthropic or an OpenAI-compatible + // endpoint fails every single call — a gate that never runs, discovered one + // merge at a time. Requiring the field is clearer than inventing a default per + // provider kind, which would silently pick a model (and a price) for the + // project. Workers AI keeps the default: there the id is the right one. + if (config.provider !== undefined && config.model === undefined) { + logger.error("Dropping llm evaluator entry that names a provider but no model", undefined, { + provider: config.provider, + }); + return reject( + `llm evaluator names provider "${config.provider}" but no "model"; a provider entry must name the model to run, since the default is a Workers AI model id`, + ); + } + + // Clamped, not rejected, for the reason `clampScore` gives: a threshold + // outside [0,1] is a bounded mistake, and 1.5 (nothing passes) is the safe + // direction anyway. -5 is not — it would make every model verdict pass — so + // the clamp is load-bearing rather than cosmetic. + const threshold = clampScore(source.threshold, logger); + if (threshold !== undefined) config.threshold = threshold; + + if (typeof source.maxDiffChars === "number" && Number.isFinite(source.maxDiffChars)) { + // Left unclamped here on purpose: `LLMEvaluator` clamps it against its own + // floor and ceiling, which are the numbers that actually bound the prompt. + config.maxDiffChars = source.maxDiffChars; + } else if (source.maxDiffChars !== undefined) { + logger.warn("Ignoring policy field that is not a finite number", { + field: "llm.maxDiffChars", + submitted: forLog(source.maxDiffChars), + }); + } + + return config; +} + /** * Validate one entry of the user-supplied `evaluators` array. * @@ -417,19 +612,38 @@ function sanitizeSandboxConfig( * passed through by reference, so a returned policy never shares object * identity with the parsed input or with `DEFAULT_POLICY`. */ -function sanitizeEvaluator(raw: unknown, logger: Logger): EvaluatorConfig | null { +function sanitizeEvaluator( + raw: unknown, + logger: Logger, + providers: LlmProviderCatalog, + rejections: string[], + index: number, +): EvaluatorConfig | null { + const at = `evaluators[${index}]`; if (typeof raw !== "object" || raw === null || Array.isArray(raw)) { logger.warn("Dropping non-object evaluator entry", { submitted: forLog(raw) }); + rejections.push(`${at} is not an evaluator object`); return null; } const source = raw as Record; if (typeof source.type !== "string") { logger.warn("Dropping evaluator entry without a string type", { submitted: forLog(raw) }); + rejections.push(`${at} has no string 'type'`); return null; } if (source.type === "sandbox") return sanitizeSandboxConfig(source, logger); + // Replaces the copy-through below for `llm`, which used to hand the parsed + // object's fields to the evaluator unvalidated — including any `baseUrl` or + // inline credential the file carried. + if (source.type === "llm") { + const entryRejections: string[] = []; + const llm = sanitizeLlmConfig(source, logger, providers, entryRejections); + for (const reason of entryRejections) rejections.push(`${at}: ${reason}`); + return llm; + } + if (source.type === "webhook") { // `url` is required by the type, so check it rather than asserting it. The // evaluator does fail closed on a missing URL, but telling the compiler a @@ -439,6 +653,7 @@ function sanitizeEvaluator(raw: unknown, logger: Logger): EvaluatorConfig | null logger.warn("Dropping webhook evaluator entry without a url", { submitted: forLog(raw), }); + rejections.push(`${at}: webhook evaluator has no 'url'`); return null; } // The same unbounded-timeout defect the sandbox entry had, one array diff --git a/src/evaluation/sanitize-policy.ts b/src/evaluation/sanitize-policy.ts index c0e78886..90eaea5b 100644 --- a/src/evaluation/sanitize-policy.ts +++ b/src/evaluation/sanitize-policy.ts @@ -1,19 +1,157 @@ import type { EvalPolicy, EvaluatorConfig } from "./types"; +/** + * Fields whose *name* says the value is a credential. + * + * Matched on the key rather than the value because there is no reliable way to + * recognise a secret by its bytes, and this runs on repository content: whatever + * a user wrote in the policy file, under whatever key they chose. + * + * Stems, not spellings. Plurals are normalized by {@link isCredentialWord}, so + * `token` covers `tokens` and `apiKey` covers `apiKeys` and `access_tokens` + * without either having to be listed. `auth` is here on its own because it is + * one of the most common credential key names there is, and listing only + * `authorization` let it through. + */ +const CREDENTIAL_WORDS = new Set([ + "secret", + "token", + "password", + "passwd", + "pwd", + "passphrase", + "credential", + "cred", + "auth", + "authorization", + "authorisation", + "bearer", + "key", + "apikey", + "signature", + "sig", + "hmac", + "pat", + "otp", +]); + +/** + * Split a field name into words on camelCase and separator boundaries, so + * `apiKey`, `api_key` and `API-KEY` are all recognised while `monkeys` — which + * merely contains "key" — is not. + */ +function words(key: string): string[] { + return key + .split(/[_\-.\s]+|(?<=[a-z0-9])(?=[A-Z])/) + .map((word) => word.toLowerCase()) + .filter((word) => word.length > 0); +} + +/** + * Whether one word of a field name names a credential. + * + * The trailing-`s` rule is what makes `tokens`, `apiKeys` and `passwords` + * behave like their singulars. It is deliberately only a *stem* test and not a + * substring one, because the property the strip list must keep is that + * `keystone` and `monkeys` survive: `monkeys` stems to `monkey`, which is not a + * credential word, while a substring match would strip both. + */ +function isCredentialWord(word: string): boolean { + if (CREDENTIAL_WORDS.has(word)) return true; + return word.endsWith("s") && CREDENTIAL_WORDS.has(word.slice(0, -1)); +} + +/** Keys stripped wherever they appear in the policy, regardless of shape. */ +function isStrippedKey(key: string): boolean { + // The provider *name* is not a credential, but it names the operator's + // infrastructure to a third party the operator did not choose — and the model + // has no use for it. It costs nothing to withhold, and withholding it is what + // makes adding a provider to a policy free against + // `MAX_POLICY_CONTEXT_CHARS`: the serialized policy does not grow, so a + // project that fits today cannot start failing its gate closed by opting into + // BYOK. + if (key === "provider") return true; + return words(key).some(isCredentialWord); +} + +/** + * Bounds on the walk, so a hostile policy file cannot make sanitization + * expensive (or produce an output larger than the input). + * + * The depth bound doubles as the cycle guard: a YAML alias cycle + * (`a: &x {b: *x}`) is a perfectly well-formed file and an infinite structure, + * and the walk below would otherwise not terminate on one. + */ +const MAX_DEPTH = 8; +const MAX_NODES = 2_000; +const MAX_ARRAY_ITEMS = 256; +const DEPTH_ELISION = "[omitted: policy nesting limit]"; +const SIZE_ELISION = "[omitted: policy size limit]"; + +interface Budget { + left: number; +} + +/** Recursively rebuild `value` without any credential-shaped key. */ +function stripValue(value: unknown, depth: number, budget: Budget): unknown { + if (budget.left <= 0) return SIZE_ELISION; + budget.left -= 1; + if (value === null || typeof value !== "object") return value; + if (depth >= MAX_DEPTH) return DEPTH_ELISION; + + if (Array.isArray(value)) { + const kept: unknown[] = value + .slice(0, MAX_ARRAY_ITEMS) + .map((item) => stripValue(item, depth + 1, budget)); + if (value.length > MAX_ARRAY_ITEMS) kept.push(SIZE_ELISION); + return kept; + } + + const out: Record = {}; + for (const [key, nested] of Object.entries(value)) { + if (isStrippedKey(key)) continue; + if (budget.left <= 0) break; + out[key] = stripValue(nested, depth + 1, budget); + } + return out; +} + /** * Strip credentials from a policy before it leaves the Worker (serialized into - * an LLM prompt or a webhook request body). The webhook secret is used to sign - * outgoing requests; it must never appear in a payload itself. + * an LLM prompt, or a webhook request body). + * + * This is the last thing that runs before the policy is handed to a third + * party, so it strips by key shape rather than by known field, **recursively, + * over the whole policy object** — not just over the top level of each + * evaluator entry, which is all it used to do. Both of the holes that left are + * real: `parsePolicyContent` spreads unknown ROOT keys onto the policy, so an + * `apiKey:` written at the top of `.stratum/policy.yaml` reached the model + * prompt and the policy-supplied webhook URL (a host the policy file chose); + * and `sanitizeEvaluator` copies an entry of an unmodelled `type` through + * whole, so a credential nested one level inside such an entry — an object or + * an array, exactly the copy-through case the strip list exists for — was + * untouched. + * + * `webhook.secret` is the field this was originally written for: it signs + * outgoing requests and must never appear in a payload itself. + * + * Note what this is *not*: a defence against the model seeing secrets in the + * diff. The diff is the thing being reviewed, and `redactSecrets` + * (`deploy/redact.ts`) is explicit that literal-substring redaction is a + * backstop. The structural control is upstream — provider output never reaches + * `EvalResult.reason` (`llm-evaluator.ts`). */ export function sanitizePolicy(policy: EvalPolicy): EvalPolicy { + const stripped = stripValue(policy, 0, { left: MAX_NODES }); + const base = + typeof stripped === "object" && stripped !== null && !Array.isArray(stripped) + ? (stripped as Record) + : {}; return { - ...policy, - evaluators: policy.evaluators.map((cfg: EvaluatorConfig) => { - if (cfg.type === "webhook") { - const { secret: _secret, ...rest } = cfg; - return rest; - } - return cfg; - }), - }; + ...base, + // Restated rather than trusted: `evaluators` is typed as an array and both + // callers serialize the result, so a policy large enough to exhaust the + // node budget must still hand back the shape the type promises. + evaluators: Array.isArray(base.evaluators) ? (base.evaluators as EvaluatorConfig[]) : [], + } as EvalPolicy; } diff --git a/src/evaluation/types.ts b/src/evaluation/types.ts index d22f04b5..e313f786 100644 --- a/src/evaluation/types.ts +++ b/src/evaluation/types.ts @@ -7,15 +7,75 @@ export interface EvalResult { passed: boolean; reason: string; issues?: string[]; - /** Resource usage the evaluator incurred, recorded for cost tracking. */ - costs?: Array<{ kind: "llm_tokens" | "sandbox_ms"; quantity: number; estimated?: boolean }>; + /** + * Resource usage the evaluator incurred, recorded for cost tracking. + * + * Structurally a subset of `CostSample` (src/storage/costs.ts) rather than an + * import of it, so the evaluation layer does not depend on storage. `source` + * is what lets an evaluator running on the project's own provider credential + * say so: every recording site flattens this array straight into + * `recordCosts`, so a field missing here is a distinction that cannot be made + * at all. + * Omitted means `"platform"` — the operator paid. + */ + costs?: Array<{ + kind: "llm_tokens" | "sandbox_ms"; + quantity: number; + estimated?: boolean; + source?: "platform" | "byok"; + }>; } /** - * What the diff is a diff *of*. A diff alone does not identify the tree it - * applies to, so an evaluator that reproduces the change out-of-process (the - * webhook evaluator) cannot tell which base it should apply the hunks to - * (#274). + * Who pays for the metered resources an evaluation consumes. + * + * The LLM evaluator spends the operator's Workers AI budget on every call, and + * before this existed nothing at the evaluator layer could say whose change + * caused it. Carrying the subject on the evaluation context is what makes that + * spend attributable, and later meterable, without every evaluator having to + * grow a project-shaped constructor argument. + * + * `ownerType` is narrower than `ProjectEntry.ownerType`, which also admits + * `"agent"`: an agent is not a billing subject, it resolves to the user that + * owns it. `billingContextFor` yields nothing rather than performing that + * resolution. + */ +export interface BillingContext { + /** The paying user or org. Never an agent id — see `ownerType`. */ + ownerId: string; + ownerType: "user" | "org"; + /** The project whose policy is being enforced. */ + projectId: string; + /** + * The user who ran this evaluation, when the call site knows one. + * + * Not the payer, and deliberately a separate field from `ownerId`: PRD §4a + * separates the subject a spend is RECORDED against from the subject a LIMIT + * is checked against, because an org-owned project bills the org and an org + * costs nothing to create. Recording still names the owner above; enforcement + * uses this. Absent where no acting user exists (a queue consumer), which is + * not the same as free — it means the check falls back to the recorded + * subject. + * + * Like every other field here, it must never be forwarded off-box: the + * webhook evaluator names the fields it sends for exactly this reason. + */ + actorUserId?: string; +} + +/** + * What an evaluation is *of*, and who pays for it: the tree the diff applies + * to, plus the billing subject. A diff alone does not identify the tree, so an + * evaluator that reproduces the change out-of-process (the webhook evaluator) + * cannot tell which base it should apply the hunks to (#274). + * + * **Nothing on this type may be forwarded off-box wholesale.** The webhook + * evaluator POSTs to a URL taken from `.stratum/policy.yaml` — repository + * content — and it stays safe only because it names the fields it sends + * (`diff`, sanitized policy, `baseSha`) instead of spreading this object. A + * `...context` there would ship `billing.ownerId` to an arbitrary endpoint the + * policy file chose. Add a field here and that exclusion is one careless + * refactor away; `tests/evaluator-billing-context.test.ts` pins it. */ export interface EvaluationContext { /** @@ -28,6 +88,15 @@ export interface EvaluationContext { * commit would report a verdict for a combination the change never proposed. */ baseSha?: string; + /** + * Who pays for whatever this evaluation spends. + * + * Optional so evaluators that consume nothing metered — every evaluator + * today except `llm` — can ignore it entirely. Absent where no billing + * subject can be named, which is deliberately not the same as free: an + * agent-owned project has a payer, it just has to be resolved first. + */ + billing?: BillingContext; } export interface Evaluator { @@ -152,4 +221,31 @@ export type EvaluatorConfig = } | { type: "webhook"; url: string; secret?: string; timeoutMs?: number } | SandboxEvaluatorConfig - | { type: "llm"; model?: string; threshold?: number; maxDiffChars?: number }; + | LlmEvaluatorConfig; + +/** + * The `llm` evaluator's slice of `.stratum/policy.yaml`. + * + * Named rather than inlined for the reason `SandboxEvaluatorConfig` is: two + * places narrow this shape out of a policy (`LLMEvaluator`, and the BYOK + * provider resolution), and an inline re-declaration in either would drift. + * + * Produced only by `sanitizeLlmConfig` (`policy-loader.ts`), which is a + * whitelist: these four fields are the whole surface a repository's policy file + * may set. There is deliberately **no `baseUrl`** — see `llm-providers.ts`. + */ +export interface LlmEvaluatorConfig { + type: "llm"; + /** + * Selects one of the operator's `LLM_PROVIDERS` entries by name, running the + * evaluation on the project's own credential (BYOK). Absent means Workers AI, + * on the operator's bill. A name the operator has not configured is not a + * fallback: it fails the policy file closed. + */ + provider?: string; + model?: string; + /** Score at or above which the model's verdict is allowed to pass. */ + threshold?: number; + /** Diff characters sent to the model; clamped again by the evaluator. */ + maxDiffChars?: number; +} diff --git a/src/index.ts b/src/index.ts index a660b1b9..ab06fc4a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -58,6 +58,7 @@ import { createLogger } from "./utils/logger"; export { MergeQueue } from "./queue/merge-queue"; export { MagicLinkRateLimiter } from "./queue/magic-link-limiter"; export { RepoDO } from "./queue/repo-do"; +export { UsageMeter } from "./queue/usage-meter"; const app = new Hono<{ Bindings: Env }>(); diff --git a/src/mcp/client.ts b/src/mcp/client.ts index 305f9cd6..bfe70a65 100644 --- a/src/mcp/client.ts +++ b/src/mcp/client.ts @@ -260,4 +260,15 @@ export class StratumClient { async me() { return this.request<{ id: string; email: string }>("GET", "/api/users/me"); } + + /** + * The caller's metered usage against their plan limits. + * + * Read-only by design and by surface: there is no billing WRITE anywhere on + * this client, because an agent that can raise its own limit does not have + * one (PRD §4c). + */ + async getUsage() { + return this.request("GET", "/api/users/me/usage"); + } } diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 6f70bc5d..e84d897a 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -1,5 +1,5 @@ /** - * The eighteen tools the MCP server exposes. + * The nineteen tools the MCP server exposes. * * The full agent contribution loop: read a project, fork a workspace, commit, * open an evaluation-gated change, follow it through review to merge, and track @@ -12,6 +12,16 @@ * Tool DESCRIPTIONS are the only documentation a model gets, so they state the * refusals as well as the capabilities: a model that knows `stratum_review_change` * will reject its agent token asks a human instead of retrying nineteen times. + * + * The billing surface is READ-ONLY and stays that way. `stratum_get_usage` + * exists because an agent that can see 12% of its allowance left batches its + * changes, moves the project to its own provider key, or stops and asks its + * human — where an agent that cannot see it discovers the limit by hitting a + * wall. There is deliberately no upgrade, top-up or payment-method tool, and no + * `stratum_set_provider_key`: a limit an agent can raise is not a limit, and a + * credential must never pass through a model's context window (PRD §4c). Both + * refusals are stated in `stratum_get_usage`'s own description so a model reads + * them where it would otherwise go looking. */ import type { StratumClient } from "./client"; import { InvalidArgumentError, parseProjectRef } from "./client"; @@ -115,6 +125,12 @@ export function buildTools(client: StratumClient): ToolDef[] { {}, () => client.me(), ), + tool( + "stratum_get_usage", + 'Read this account\'s metered usage against its plan limits for the current billing period: per-meter consumption (LLM tokens, sandbox time, deploys), the limit on each, the rate ceilings, the period, and when it resets. A limit of -1 means unlimited, 0 means the plan forbids that meter outright; when a limit is unlimited there is no percentage to compute. Figures cover platform-billed usage only — spend on a project\'s own provider key (BYOK) is billed by that provider and is never counted against these allowances. `usedSource` says where each `used` figure comes from: "meter" is the live counter a limit is actually compared against, covering work in any namespace; "ledger" means those counters were unavailable and the figures are the usage recorded against this account alone, which excludes work recorded against an organization. An agent token reports its OWNER\'s allowance, because that is the account its evaluations are charged to. Read-only, and the only billing tool there is: nothing here can raise a limit, buy capacity, add a payment method or set a provider key, for the same reason agent tokens cannot submit review verdicts — a spend limit an agent can lift is not a limit. Do not look for one; hand the account owner https://docs.usestratum.dev and let a human do it in a browser.', + {}, + () => client.getUsage(), + ), // ── Projects ────────────────────────────────────────────────────────── tool( diff --git a/src/merge/post-merge.ts b/src/merge/post-merge.ts index 21adc24f..7d60f3ac 100644 --- a/src/merge/post-merge.ts +++ b/src/merge/post-merge.ts @@ -2,7 +2,7 @@ import { materializeTree } from "../evaluation/sandbox-evaluator"; import type { EvalPolicy } from "../evaluation/types"; import { emitEvent } from "../queue/events"; import { updateChangeStatus } from "../storage/changes"; -import { recordCosts } from "../storage/costs"; +import { recordCosts, resolveBillingSubject } from "../storage/costs"; import { freshRepoToken, getCommitParent, readRepoFiles, revertToCommit } from "../storage/git-ops"; import type { Env, ProjectEntry } from "../types"; import { projectDefaultBranch } from "../types"; @@ -85,10 +85,21 @@ export async function runPostMergeCheck( const run = await sandbox.run(command, { timeout: merge?.postMergeTimeoutMs ?? DEFAULT_POST_MERGE_TIMEOUT_MS, }); + const subject = await resolveBillingSubject(env.DB, logger, project); await recordCosts( env.DB, logger, - { project: project.name, projectId: project.id, changeId: opts.changeId }, + { + project: project.name, + projectId: project.id, + changeId: opts.changeId, + ...(subject ?? {}), + // No `waitUntil`: this runs in the merge queue consumer, with no + // request to hang background work off, so delivery is best-effort + // inline (PRD §8). No actor either — a queue message names no user, + // so the check falls back to the recorded subject. + notify: { env }, + }, [ { kind: "sandbox_ms", quantity: Date.now() - runStartedAt }, { kind: "git_ops", quantity: 1 }, diff --git a/src/middleware/auth.ts b/src/middleware/auth.ts index ace96837..6bcb0d1c 100644 --- a/src/middleware/auth.ts +++ b/src/middleware/auth.ts @@ -1,5 +1,6 @@ import type { MiddlewareHandler } from "hono"; import { getCookie } from "hono/cookie"; +import { warmEntitlements } from "../billing/entitlements"; import { isGitHttpPath } from "../routes/git-http"; import { isOAuthClientEndpoint } from "../routes/mcp-oauth"; import { getAgentByToken } from "../storage/agents"; @@ -95,6 +96,22 @@ export const authMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, ne c.set("logger", logger); + /** + * Pre-fetch the caller's plan limits for LATER requests (PRD §4). + * + * Called at every point below where an identity is established, because this + * is the only place that knows who is paying before the routes run. It is + * inert unless the cloud billing vars are set, never awaited (it schedules on + * `waitUntil` and is skipped outright when there is none), and nothing in this + * request reads the result — a warm that fails costs a later reader nothing + * but a fail-open default. + * + * An agent warms its OWNER, not itself: agents are not billing subjects. + */ + const warmOwner = (ownerId: string): void => { + warmEntitlements(c.env, getWaitUntil(c), { ownerId, ownerType: "user" }, logger); + }; + // The git smart-HTTP router authenticates over HTTP Basic itself; let it own // the challenge instead of rejecting the non-Bearer header here. if (isGitHttpPath(c.req.path)) { @@ -266,6 +283,8 @@ export const authMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, ne else await touch; } + warmOwner(user.id); + logger.debug("Auth success - user", { userId: user.id, username: user.username, scope }); await next(); return; @@ -307,6 +326,8 @@ export const authMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, ne // choice governs it — otherwise opting out could be defeated by routing // traffic through an agent. The owner row is already in hand. c.set("telemetryOptOut", ownerResult.data.telemetryOptOut === true); + warmOwner(agentResult.data.ownerId); + logger.debug("Auth success - agent", { agentId: agentResult.data.id, ownerId: agentResult.data.ownerId, @@ -395,6 +416,8 @@ export const authMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, ne if (waitUntil) waitUntil(touch); else await touch; + warmOwner(grant.data.user.id); + logger.debug("Auth success - OAuth grant", { userId: grant.data.user.id, clientId: grant.data.clientId, @@ -450,6 +473,8 @@ export const authMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, ne userResult.data.username || (userResult.data.email.split("@")[0] ?? "").toLowerCase().replace(/[^a-z0-9]/g, ""); c.set("username", username); + warmOwner(sessionResult.data.userId); + logger.debug("Auth success - session", { userId: sessionResult.data.userId, username }); } } else { diff --git a/src/middleware/config-guard.ts b/src/middleware/config-guard.ts index e3e6d8de..7e5748cb 100644 --- a/src/middleware/config-guard.ts +++ b/src/middleware/config-guard.ts @@ -1,4 +1,5 @@ import type { MiddlewareHandler } from "hono"; +import { parseLlmProviders } from "../evaluation/llm-providers"; import type { Env } from "../types"; import { createLogger } from "../utils/logger"; @@ -26,21 +27,96 @@ export function repoDoConfigError( return null; } -// Log the config problem at most once per isolate rather than on every request. +/** + * Returns a human-readable error when entitlement enforcement is switched on + * without the service it enforces against, else null. + * + * Structurally the same footgun as `REPO_DO_ENABLED` above, one step quieter: + * `ENTITLEMENTS_ENFORCE=1` with no `BILLING_SERVICE_URL` throws nothing and + * breaks nothing — every owner resolves to `UnlimitedEntitlements` and every + * decision admits. That is enforcement that looks on in the dashboard and does + * nothing in production, which is worse than being off, because nobody goes + * looking for a limit they believe is already applied. + * + * `BILLING_SERVICE_SECRET` is not checked here: it is a Wrangler secret rather + * than a var, so a deploy that has the URL and not the secret is an operational + * state to fix, not a config file to reread — and `entitlementsEnabled` already + * treats it as off. + */ +export function entitlementsConfigError( + env: Pick, +): string | null { + if (env.ENTITLEMENTS_ENFORCE !== "1") return null; + // BOTH, because `entitlementsEnabled` requires both: with the URL set and the + // secret missing the seam reports itself off, every check returns `inert()`, + // and an operator who believes they switched enforcement on gets silence. + // Naming only the URL — as this did — meant the one misconfiguration hardest + // to notice from the outside was the one the guard did not report. + const missing: string[] = []; + if (!env.BILLING_SERVICE_URL) missing.push("BILLING_SERVICE_URL"); + if (!env.BILLING_SERVICE_SECRET) missing.push("BILLING_SERVICE_SECRET"); + if (missing.length > 0) { + // Joined rather than concatenated: mixing a template literal with `+` trips + // biome's useTemplate, and the autofix for it is one unreadable line. + return [ + `ENTITLEMENTS_ENFORCE is '1' but ${missing.join(" and ")}`, + `${missing.length === 1 ? "is" : "are"} not set — no plan limits can be`, + "fetched, so every owner resolves to unlimited and every enforcement point", + "admits. Set BILLING_SERVICE_URL as a var and BILLING_SERVICE_SECRET as a", + "secret for the [env.] block, or unset ENTITLEMENTS_ENFORCE.", + ].join(" "); + } + return null; +} + +/** + * Returns a human-readable error when `LLM_PROVIDERS` is set but unusable, else + * null. + * + * Loud on purpose, and the reason Open Question 3 settled on a typed parse: a + * malformed allowlist silently disables BYOK, and the projects that opted in + * discover it as blocked merges naming a provider the operator believes is + * configured. Unset is not a problem — it is the default, and it means Workers + * AI only. + */ +export function llmProvidersConfigError(env: Pick): string | null { + const parse = parseLlmProviders(env.LLM_PROVIDERS); + if (parse.status === "invalid") { + // One template literal rather than concatenation: `useTemplate` rejects + // mixing an interpolated piece with `+`, and the sibling above can use plain + // strings only because it interpolates nothing. + return `LLM_PROVIDERS is set but could not be parsed (${parse.reason}) — no BYOK provider is available, so every policy selecting one blocks merges until this is fixed. Correct the value or unset it to run on Workers AI.`; + } + return null; +} + +// Log the config problems at most once per isolate rather than on every request. let hasLoggedConfigError = false; /** - * Surfaces an incoherent RepoDO/R2 configuration loudly in Workers Logs on the - * first request after a bad deploy, instead of only when a later merge throws. - * Non-fatal: reads and the UI still work without the bucket, so we log rather - * than reject the request. + * Surfaces an incoherent RepoDO/R2, entitlements or LLM-provider configuration + * loudly in Workers Logs on the first request after a bad deploy, instead of + * only when a later merge throws — or, for entitlements, never. Non-fatal: + * reads and the UI still work in every case, so we log rather than reject the + * request. */ export const configGuardMiddleware: MiddlewareHandler<{ Bindings: Env }> = async (c, next) => { if (!hasLoggedConfigError) { - const problem = repoDoConfigError(c.env); - if (problem) { - hasLoggedConfigError = true; - createLogger({ component: "config-guard" }).error(problem); + // Set before the checks, not inside the `problems.length` branch: a + // correctly configured deploy has no problems to log, so the flag would + // never be set and every request would re-run all three — including + // `llmProvidersConfigError`, which `JSON.parse`s `LLM_PROVIDERS`. The + // config cannot change within an isolate, so one evaluation answers for + // its whole life either way. + hasLoggedConfigError = true; + const problems = [ + repoDoConfigError(c.env), + entitlementsConfigError(c.env), + llmProvidersConfigError(c.env), + ].filter((problem): problem is string => problem !== null); + if (problems.length > 0) { + const logger = createLogger({ component: "config-guard" }); + for (const problem of problems) logger.error(problem); } } await next(); diff --git a/src/middleware/rate-limit.ts b/src/middleware/rate-limit.ts index 5ce8d700..58f4880b 100644 --- a/src/middleware/rate-limit.ts +++ b/src/middleware/rate-limit.ts @@ -1,4 +1,6 @@ import type { MiddlewareHandler } from "hono"; +import { enforcementBinding } from "../billing/enforcement"; +import { RemoteEntitlements, UNLIMITED, entitlementsEnabled } from "../billing/entitlements"; import { isGitHttpPath } from "../routes/git-http"; import { isPostHogProxyPath, isPostHogSdkPath } from "../routes/posthog-proxy"; import type { Env } from "../types"; @@ -75,10 +77,40 @@ export function rateLimitMiddleware(opts?: RateLimitOptions): MiddlewareHandler< const isAuthenticated = Boolean(userId ?? agentId); const defaultLimit = isAuthenticated ? 1000 : 60; + // A plan's own request rate, when we hold one. Deliberately narrow: + // + // - **Users only.** Today's limiter keys on `userId ?? agentId ?? IP` while + // entitlements key on a billing subject, and resolving an agent to its + // owner needs a D1 hop inside this middleware. Folding a user's agents + // into one bucket is also a user-visible TIGHTENING (a user plus five + // agents get 6x1000 rpm today), so agents keep their own bucket at + // today's numbers and that change gets its own CHANGELOG entry. + // - **Cached read only.** `forOwner` never fetches; the auth middleware's + // warm is what makes it hit. A miss falls back to today's numbers rather + // than blocking the request on a billing round trip. + // - **`-1` is not a number to enforce.** Unlimited is the FAIL-OPEN default + // of the entitlements layer, so honouring it literally would let one + // uncached miss (or one billing outage) uncap a user's request rate. The + // default cap stands instead. + // - **Read only where it is used.** Analytics ingest and any route passing + // its own `requestsPerMinute` (`POST /projects` sets 20) discard this + // value, and paying a KV round trip on the request path for a number + // thrown away is a cost with no decision behind it. + // + // Observe-only never tightens: with `ENTITLEMENTS_ENFORCE` off a plan may + // raise this ceiling but not lower it, so the month of measurement PRD §8 + // asks for cannot start 429-ing anybody. + const subjectLimit = async (): Promise => { + const planLimit = await entitledRequestsPerMinute(c.env, userId); + if (planLimit === null) return defaultLimit; + return enforcementBinding(c.env) ? planLimit : Math.max(planLimit, defaultLimit); + }; // Far above what a real session produces and far below anything worth // relaying through someone else's Worker. const analyticsLimit = 600; - const limit = isAnalyticsIngest ? analyticsLimit : (opts?.requestsPerMinute ?? defaultLimit); + const limit = isAnalyticsIngest + ? analyticsLimit + : (opts?.requestsPerMinute ?? (await subjectLimit())); const identifier = userId ?? agentId ?? c.req.header("CF-Connecting-IP") ?? "anonymous"; const minuteBucket = Math.floor(Date.now() / 60000); @@ -144,6 +176,30 @@ export function rateLimitMiddleware(opts?: RateLimitOptions): MiddlewareHandler< }; } +/** + * The user's own `requests_per_minute`, or `null` when there is none to use. + * + * `null` covers every case the caller must fall back to today's numbers for: the + * billing service is unconfigured (every self-hoster), the caller is not a user, + * the cache is cold, or the plan says "unlimited" — which here means "we do not + * know", because unlimited is what this layer fails open to. + */ +async function entitledRequestsPerMinute( + env: Env, + userId: string | undefined, +): Promise { + if (!userId) return null; + if (!entitlementsEnabled(env)) return null; + const resolved = await new RemoteEntitlements(env, logger).forOwner(userId, "user"); + if (!resolved.success) return null; + // "default" is a cache miss, not a plan. Only a value somebody actually sent + // us gets to move a limit. + if (resolved.data.source === "default") return null; + const limit = resolved.data.entitlements.rates.requests_per_minute; + if (!Number.isInteger(limit) || limit < 0 || limit === UNLIMITED) return null; + return limit; +} + /** * Rate limiting middleware specifically for import endpoints. * Provides both per-user and per-project rate limiting to prevent resource exhaustion. diff --git a/src/queue/usage-meter.ts b/src/queue/usage-meter.ts new file mode 100644 index 00000000..53d83464 --- /dev/null +++ b/src/queue/usage-meter.ts @@ -0,0 +1,786 @@ +import { DurableObject } from "cloudflare:workers"; +import { UNLIMITED } from "../billing/entitlements"; +import type { MeterKey } from "../storage/usage"; +import type { Env } from "../types"; +import { type Logger, createLogger } from "../utils/logger"; + +/** + * The monthly counters, all of them, under one key: `{ period, counts }`. + * + * One key per meter would make a rollover something to sweep; one key holding + * the period makes it free, exactly as `MagicLinkRateLimiter`'s bucket does — + * the first write of a new month overwrites the record and the previous month's + * counts are gone without a delete. + */ +const PERIOD_KEY = "period"; + +/** + * The windowed rate counters, kept apart from {@link PERIOD_KEY} because their + * lifetimes are unrelated: an hourly window that straddles midnight on the 1st + * must not be reset by the month rolling, and a month's counter must not be + * dropped when the hour it happened to be written in expires. + */ +const RATE_KEY = "rate"; + +/** + * How long after a window closes the object waits before erasing what belongs + * to it. Longer than the precedent's minute because the window here is a + * calendar month: a late `settle` for a spend that started before midnight on + * the last day is ordinary, and erasing the record out from under it would + * discard a correction rather than a stale count. + */ +const CLEANUP_GRACE_MS = 60 * 60 * 1000; + +/** + * A rate meter: a bound on burst over a rolling window, not a bound on the + * month's spend. `reserve` with no `settle` is exactly that check. + * + * This lives here rather than in KV (PRD §4b) for one reason. KV reads come + * from a per-colo edge cache whose staleness is about a minute, which is + * tolerable for `rateLimitMiddleware` only because its bucket also rolls every + * minute — staleness and window are the same size. Stretch the window to an + * hour and staleness is 1/60th of it, so an N/hour limit admits roughly + * N x colos x 60 to a distributed caller: not a loose limit, no limit at all, + * against precisely the distributed burst this control exists to stop. + */ +export type RateMeterKey = "evaluations_per_hour"; + +/** + * Everything this object counts: the monthly flows of `MeterKey`, plus the + * rate meters above. + * + * The two are counted differently and the difference is deliberate. A monthly + * meter is keyed on the caller's `period` string and reconciled against + * `usage_periods`; a rate meter is keyed on wall-clock buckets and reconciled + * against nothing, because no D1 aggregate of it exists. + */ +export type UsageMeterKey = MeterKey | RateMeterKey; + +interface RateWindow { + /** How far back the limit looks. */ + windowSeconds: number; + /** + * The granularity the window is summed at. SHORT on purpose: one bucket per + * window is a fixed window, and a fixed window admits the whole allowance in + * the first second of every new one — twice the limit across the boundary, + * which is the burst the meter exists to bound. + * + * Twelve buckets do not eliminate that, they bound it: this is the ordinary + * bucketed approximation of a sliding window, so a caller spending the whole + * allowance at the very end of one bucket gets it back one bucket-width + * before the hour is out — up to 2x the limit inside 55 minutes rather than + * inside a second. Halving `bucketSeconds` halves the error and doubles the + * stored keys. Say the bound out loud rather than claiming an exactness the + * arithmetic does not have. + */ + bucketSeconds: number; +} + +const RATE_WINDOWS: Record = { + evaluations_per_hour: { windowSeconds: 60 * 60, bucketSeconds: 5 * 60 }, +}; + +/** + * How far back a rate meter's window looks, in seconds. + * + * Exported so the refusal copy can tell a caller when its burst allowance frees + * up without restating the window — a message promising a reset the object does + * not honour is worse than no message (PRD §4c). + * + * @param meter - The rate meter to describe + * @returns The window length in seconds + */ +export function rateWindowSeconds(meter: RateMeterKey): number { + return RATE_WINDOWS[meter].windowSeconds; +} + +/** Bucket id (`floor(epochSeconds / bucketSeconds)`) -> quantity in that bucket. */ +type RateBuckets = Record; + +interface PeriodRecord { + /** 'YYYY-MM' UTC, from `usagePeriod`. The counts below belong to this month and no other. */ + period: string; + counts: Partial>; +} + +interface RateRecord { + meters: Partial>; +} + +export interface MeterReserveOutcome { + admitted: boolean; + /** + * Whether the estimate was actually added to the counter, and therefore needs + * settling. + * + * Not the same as {@link MeterReserveOutcome.admitted}, and the difference is + * the whole point of `countRefused`: under observe-only a refused reservation + * is still counted, so the month of measurement can say by HOW MUCH a subject + * would have been blocked rather than only that it would have been. Callers + * gate on `admitted` and settle on `counted`. + */ + counted: boolean; + /** The counter after this call. Reported for logging and telemetry; callers gate on `admitted`. */ + count: number; +} + +export interface MeterSnapshot { + /** Echoed back so a caller cannot mistake a rolled-over read for a live one. */ + period: string; + /** Empty when this object holds nothing for `period` — "never used" and "used nothing" are one fact. */ + counts: Partial>; +} + +/** + * Which instance an enforcement subject maps to. + * + * The type prefix is not decoration. This object is keyed on the ENFORCEMENT + * subject (see the class doc comment), so a user and an org can both name one, + * and the deletion cascade must be able to erase one without touching the + * other: deleting a single org may not purge a counter that belongs to a + * person. Distinct prefixes make that a property of the naming rather than of + * whichever id generator happens not to collide today. + * + * @param subjectType - Whether the enforcement subject is a user or an org + * @param subjectId - The subject's id + * @returns The Durable Object name for that subject's counters + */ +export function usageMeterName(subjectType: "user" | "org", subjectId: string): string { + return `${subjectType}:${subjectId}`; +} + +/** + * Serialized usage counters, one instance per ENFORCEMENT subject. + * + * ## What this is for + * + * A Workers KV read-modify-write lets N concurrent requests read the same total + * and each write back "+1", so the counter advances by one and the cap bounds + * sequential traffic only — the race `MagicLinkRateLimiter`'s doc comment + * describes, and the one `rateLimitMiddleware` still has. A Durable Object + * closes it: the runtime's input gate holds incoming events for an instance + * while one of its storage operations is in flight, so the read and the write + * below cannot interleave with another request against the same subject. + * Serialization is per instance, so different subjects still run in parallel. + * + * ## Reserve, then settle + * + * An LLM call's true cost is known only from the response, so a plain increment + * would leave enforcement doing check-then-act — the exact property the object + * is bought for. `reserve` takes an UPPER BOUND before the spend and admits or + * refuses it atomically; `settle` applies the signed difference once the real + * cost is known. `MagicLinkRateLimiter.refund` is the model for the second half. + * + * ## The subject is the ENFORCEMENT subject, and this object does not choose it + * + * PRD §4a: the subject a limit is CHECKED against is not the subject a spend is + * RECORDED against. Recording keeps naming the project's owner, because a + * ledger must say what happened; a limit is checked against the acting user (or + * an org positively known to hold a paid plan), because an allowance keyed on + * the recorded owner is resettable — create an org, create a project in it, get + * a fresh `usage_periods` subject and a fresh allowance. + * + * That resolution is a decision about plans and actors, and it is NOT made + * here. The caller passes an already-resolved subject id and this object counts + * against it. Keep it that way: a DO that resolved its own subject would need + * the entitlements cache and the request's actor, and the one property it + * exists to provide — that everything for one subject is serialized — would + * then depend on a lookup that can be stale. + * + * ## Limits are caller-supplied + * + * As in the precedent, so the policy numbers stay next to the enforcement point + * rather than being baked into storage. That is safe only because DO RPC is + * reachable from Worker code alone: no request can name its own limit. **Do not + * expose this class over `fetch`** — a `fetch` handler would make `limit` an + * attacker-controlled input. + * + * ## Malformed input ADMITS — a deliberate deviation from the precedent + * + * `MagicLinkRateLimiter.reserve` returns `{admitted:false}` on a malformed + * limit: fail closed, correct for a magic-link cap, where refusing a send is a + * minor inconvenience and an unlimited endpoint is an abuse vector. It is wrong + * here. The limits reaching this object come from a billing payload over the + * network, and a `NaN` limit that refuses would block EVERY merge on the + * instance for as long as the billing service kept sending it — an outage + * caused by the metering, in a system whose entitlements layer fails open on + * purpose for exactly this reason (`src/billing/entitlements.ts`). So a + * malformed limit, estimate, period or clock admits, and says so in the log. + * + * Well-formed limits are NOT this branch: `-1` is unlimited (admit and count), + * and `0` is a hard block (refuse). Only a value that is not a limit at all — + * `NaN`, `1.5`, `-7`, an infinity — takes the admitting path. + */ +export class UsageMeter extends DurableObject { + /** + * Atomically admits or refuses an upper bound on a spend, and counts it. + * + * The whole bound has to fit: a reservation is admitted only when + * `count + estimate` is within the limit, because the caller cannot know yet + * that the spend will come in cheaper. `settle` gives back the difference. + * + * A rate meter (see {@link RateMeterKey}) takes the windowed path instead, + * where `period` is not consulted at all — its window is wall-clock and does + * not reset when the month does. + * + * @param meter - Which counter to charge + * @param estimate - Upper bound on this spend; a non-usable value counts nothing and still admits + * @param limit - The subject's allowance: `-1` unlimited, `0` a hard block, otherwise a ceiling + * @param period - 'YYYY-MM' UTC, from `usagePeriod`; ignored by rate meters + * @param nowMs - Caller's clock, so one request's counters cannot straddle a window boundary + * @param opts.countRefused - Count the estimate even when the limit refuses it. + * Passed by an observe-only caller, whose refusals do not bind: a counter + * that stopped moving at the limit would freeze exactly when the measurement + * gets interesting, and `tokensReserved` would stay null so the true cost + * was never settled either. It never widens what is ADMITTED — `admitted` + * still reports what the limit says. + * @returns Whether the reservation was admitted, whether it was counted, and + * the counter after this call + */ + async reserve( + meter: UsageMeterKey, + estimate: number, + limit: number, + period: string, + nowMs: number, + opts: { countRefused?: boolean } = {}, + ): Promise { + const usable = usableQuantity(estimate); + if (usable === null) { + // Counting nothing is the safe half of the deviation: a poisoned counter + // (NaN never compares below a limit again) would block the subject for + // the rest of the month, which is the outcome the deviation exists to + // avoid. The limit still applies to what is already counted. + warn("Usage meter estimate is not a usable quantity; reserving zero", { + meter, + estimate: String(estimate), + }); + } + const quantity = usable ?? 0; + const countRefused = opts.countRefused === true; + if (isRateMeter(meter)) return this.reserveRate(meter, quantity, limit, nowMs, countRefused); + return this.reservePeriod(meter, quantity, limit, period, countRefused); + } + + /** + * Applies the real cost of a spend that was already reserved. + * + * **A negative delta is legitimate here**, and that reads as a contradiction + * next to `upsertUsage`, which drops one, and migration 049, whose CHECK + * forbids one. Both are right, because they hold different things. A + * `usage_periods` row is the month's accumulated total, and a negative there + * REFUNDS THE MONTH through the one statement that exists to make the month + * non-refundable. This counter is the in-flight reservation ledger for a + * single period: the negative it takes is the unused remainder of a bound + * this same request reserved moments ago — 4,000 tokens reserved, 900 spent, + * 3,100 handed back — and refusing it would leave every over-estimate + * charged in full, which walls a subject off at a fraction of its allowance. + * The floor at zero is what keeps the difference safe: a settle can never + * hand back more than the counter holds. + * + * A settle for a period this object is no longer holding is dropped, as the + * precedent's `refund` drops one whose window has rolled. The reservation it + * corrects went with the rollover, so there is nothing to correct. + * + * @param meter - The counter the matching `reserve` charged + * @param delta - Signed difference between the true cost and the reserved bound + * @param period - The period the matching `reserve` used + * @param nowMs - Caller's clock; used by rate meters to find the live bucket + */ + async settle(meter: UsageMeterKey, delta: number, period: string, nowMs: number): Promise { + if (!Number.isFinite(delta)) { + warn("Usage meter settle ignored: delta is not a finite number", { meter, delta }); + return; + } + if (isRateMeter(meter)) { + await this.settleRate(meter, delta, nowMs); + return; + } + const record = await this.ctx.storage.get(PERIOD_KEY); + if (record?.period !== period) { + // The period rolled between reserve and settle, so the charge being + // refunded is not in the live record. Logged rather than dropped: a + // reservation that outlives its month leaves the old counter high, and + // that has to be diagnosable from Workers Logs. + warn("Usage meter settle ignored: period has rolled since the reservation", { + meter, + settlePeriod: period, + livePeriod: record?.period ?? null, + }); + return; + } + const current = record.counts[meter] ?? 0; + await this.ctx.storage.put(PERIOD_KEY, { + period, + counts: { ...record.counts, [meter]: Math.max(0, current + delta) }, + }); + } + + /** + * Raises a counter to match the D1 aggregate. The reconcile write path. + * + * **Monotonic within a live period: it may raise, never lower** (PRD §3). The + * counter is ahead of `usage_periods` by design — a reservation is taken + * before the spend and the aggregate is written after it — so a reconcile + * that treated D1 as authoritative would hand back every in-flight + * reservation on the instance, once per reconcile. Lowering is a correction + * to a CLOSED period, and it belongs in D1, not in a live counter. + * + * A floor for a period older than the one held is dropped for the same + * reason: 'YYYY-MM' sorts chronologically, so a stale reconcile cannot + * clobber the live month with last month's total. + * + * Rate meters are not reconcilable and are ignored here: there is no D1 + * aggregate of `evaluations_per_hour` to reconcile against, by design — an + * hourly window is not something a monthly table can carry. + * + * WHICH rows the caller summed is a decision this object cannot make or + * check, and getting it wrong is silent. It must be the ENFORCEMENT + * subject's rows — the actor's, not the project owner's, since that is what + * this instance counts — and `source = 'platform'` alone: migration 049 put + * `source` in the primary key so that a project paying its own provider bill + * is not charged against the hosted allowance, and a sum that omits the + * filter hands that distinction straight back. + * + * @param meter - The monthly counter to raise + * @param quantity - The aggregate's total for this subject, meter and period + * @param period - 'YYYY-MM' UTC the quantity was summed over + */ + async setFloor(meter: UsageMeterKey, quantity: number, period: string): Promise { + if (isRateMeter(meter)) { + warn("Usage meter floor ignored: rate meters have no aggregate to reconcile", { meter }); + return; + } + const floor = usableQuantity(quantity); + if (floor === null) { + warn("Usage meter floor ignored: quantity is not a usable total", { meter, quantity }); + return; + } + const endsAt = periodEndMs(period); + if (endsAt === null) { + warn("Usage meter floor ignored: unparseable period", { meter, period }); + return; + } + const record = await this.ctx.storage.get(PERIOD_KEY); + if (record && record.period > period) return; + const counts = record?.period === period ? record.counts : {}; + const current = counts[meter] ?? 0; + if (record?.period === period && current >= floor) return; + await this.ctx.storage.put(PERIOD_KEY, { + period, + counts: { ...counts, [meter]: Math.max(current, floor) }, + }); + await this.armCleanup(endsAt + CLEANUP_GRACE_MS); + } + + /** + * This subject's monthly counters for one period. + * + * The period is an argument rather than "now" for the reason `nowMs` is one + * on `reserve`: the caller decides which window it is asking about, so a read + * cannot straddle a boundary and answer for half of each. + * + * Rate meters are deliberately absent. Which of their buckets are live + * depends on a clock this method is not given, and a rate total answered + * against the wrong instant is worse than no answer; `reserve` returns the + * live count to the one caller that needs it. + * + * @param period - 'YYYY-MM' UTC, from `usagePeriod` + * @returns The counters held for that period, empty when none are + */ + async read(period: string): Promise { + const record = await this.ctx.storage.get(PERIOD_KEY); + return { period, counts: record?.period === period ? record.counts : {} }; + } + + /** + * Deletion-cascade RPC: DO storage is only reachable from inside the class, + * so the ACCOUNT cascade calls this to drop the counters of a subject that no + * longer exists. Deliberately not called by the project cascade — an + * allowance a user can reset by deleting a project is not an allowance + * (`src/storage/deletion.ts`, and migration 049's header). + */ + async purge(): Promise { + // Clears the cleanup alarm along with the counters: `deleteAll` deletes an + // object's alarm from compatibility date 2026-02-24, and this Worker is on + // 2026-04-29, so a purged subject is left with nothing to wake it. + await this.ctx.storage.deleteAll(); + } + + /** + * Erases what has expired, so a subject seen once does not leave durable + * state behind forever. Without it a per-owner monthly meter accumulates a + * record per subject for the life of the deployment; this is what + * `expirationTtl` did for the KV counters the precedent replaced. + * + * Selective rather than the precedent's flat `deleteAll` because two + * independent lifetimes share this object: a flat wipe on the month's alarm + * would erase a live hourly window, handing a caller a fresh burst allowance + * every time a month ended. Uses `Date.now()` rather than a caller's clock — + * nothing is being admitted or refused here, so there is no decision a + * straddled boundary could corrupt. + */ + override async alarm(): Promise { + const nowMs = Date.now(); + let nextAlarm: number | null = null; + + const record = await this.ctx.storage.get(PERIOD_KEY); + if (record) { + const endsAt = periodEndMs(record.period); + const expiresAt = endsAt === null ? null : endsAt + CLEANUP_GRACE_MS; + // An unparseable period cannot expire on its own, so it is swept now + // rather than pinning storage forever. + if (expiresAt === null || expiresAt <= nowMs) await this.ctx.storage.delete(PERIOD_KEY); + else nextAlarm = expiresAt; + } + + const rates = await this.ctx.storage.get(RATE_KEY); + if (rates) { + const pruned = pruneRates(rates, nowMs); + if (pruned === null) { + await this.ctx.storage.delete(RATE_KEY); + } else { + await this.ctx.storage.put(RATE_KEY, pruned.record); + nextAlarm = nextAlarm === null ? pruned.expiresAt : Math.max(nextAlarm, pruned.expiresAt); + } + } + + // Nothing left to expire: erase whatever remains (a key written by an + // older shape of this class, say), which drops the alarm with it, so the + // object costs nothing until it is next used. Re-arming goes through + // `armCleanup` rather than `setAlarm` so the horizon clamp applies here + // too — a stored period far enough in the future is exactly the input that + // would otherwise strand this object's storage behind an alarm centuries + // out. + if (nextAlarm === null) await this.ctx.storage.deleteAll(); + else await this.armCleanup(nextAlarm); + } + + private async reservePeriod( + meter: UsageMeterKey, + quantity: number, + limit: number, + period: string, + countRefused: boolean, + ): Promise { + const endsAt = periodEndMs(period); + if (endsAt === null) { + warn("Usage meter reserve admitted: unparseable period", { meter, period }); + return { admitted: true, counted: false, count: 0 }; + } + const record = await this.ctx.storage.get(PERIOD_KEY); + // A record from an earlier month reads as zero rather than being deleted: + // the `put` below overwrites it, so the stale record costs nothing. + const counts = record?.period === period ? record.counts : {}; + const current = counts[meter] ?? 0; + const decision = limitAdmits(current, quantity, limit); + if (decision.malformed) { + warn("Usage meter reserve admitted: limit is not a usable limit", { + meter, + limit: String(limit), + }); + } + if (!decision.admitted && !countRefused) { + return { admitted: false, counted: false, count: current }; + } + const next = current + quantity; + await this.ctx.storage.put(PERIOD_KEY, { + period, + counts: { ...counts, [meter]: next }, + }); + // Re-armed on every count so the erase always trails the live period. + await this.armCleanup(endsAt + CLEANUP_GRACE_MS); + return { admitted: decision.admitted, counted: true, count: next }; + } + + private async reserveRate( + meter: RateMeterKey, + quantity: number, + limit: number, + nowMs: number, + countRefused: boolean, + ): Promise { + if (!isUsableClock(nowMs)) { + // Admit and count nothing, as with every other malformed input. Counting + // it is what the earlier version did, and a bucket ~1000x in the future + // (microseconds mistaken for milliseconds) then counted against every + // window from here on, refusing the subject permanently. + warn("Usage meter reserve admitted: clock is not a usable timestamp", { + meter, + nowMs: String(nowMs), + }); + return { admitted: true, counted: false, count: 0 }; + } + const window = RATE_WINDOWS[meter]; + const bucket = bucketId(nowMs, window.bucketSeconds); + const record = await this.ctx.storage.get(RATE_KEY); + const live = liveBuckets(record?.meters[meter], bucket, window); + const total = sum(live); + const decision = limitAdmits(total, quantity, limit); + if (decision.malformed) { + warn("Usage meter reserve admitted: limit is not a usable limit", { + meter, + limit: String(limit), + }); + } + if (!decision.admitted && !countRefused) { + return { admitted: false, counted: false, count: total }; + } + const key = String(bucket); + live[key] = (live[key] ?? 0) + quantity; + await this.ctx.storage.put(RATE_KEY, { + meters: { ...record?.meters, [meter]: live }, + }); + await this.armCleanup(rateExpiresAt(bucket, window)); + return { admitted: decision.admitted, counted: true, count: total + quantity }; + } + + /** + * Rate meters are reserve-with-no-settle by convention, so this exists only + * for the caller that reserved an evaluation which then never ran. + * + * A refund DRAWS DOWN the live buckets rather than landing on the current + * one. The earlier version added the delta to the bucket `nowMs` falls in, + * which is empty whenever the settle outlives the reservation's bucket — the + * normal case, since buckets are minutes and an LLM gate is seconds to + * minutes. `Math.max(0, …)` then clamped the refund away and the original + * charge sat in its own bucket for the rest of the window, so an evaluation + * that failed three minutes after it was reserved still burned its slot for + * the next hour. + * + * Oldest bucket first: the reservation is by definition older than its + * settle, so that is where the charge most likely sits, and it is the choice + * that cannot free capacity a later reservation is still holding. Per-bucket + * floors of zero keep a refund from inventing headroom that was never taken. + */ + private async settleRate(meter: RateMeterKey, delta: number, nowMs: number): Promise { + if (!isUsableClock(nowMs)) { + warn("Usage meter settle ignored: clock is not a usable timestamp", { + meter, + nowMs: String(nowMs), + }); + return; + } + const window = RATE_WINDOWS[meter]; + const bucket = bucketId(nowMs, window.bucketSeconds); + const record = await this.ctx.storage.get(RATE_KEY); + if (!record) { + warn("Usage meter rate settle ignored: no record for this subject", { meter, delta }); + return; + } + const live = liveBuckets(record.meters[meter], bucket, window); + + if (delta >= 0) { + const key = String(bucket); + live[key] = (live[key] ?? 0) + delta; + } else { + let owed = -delta; + for (const key of Object.keys(live).sort((a, b) => Number(a) - Number(b))) { + if (owed <= 0) break; + const taken = Math.min(owed, live[key] ?? 0); + live[key] = (live[key] ?? 0) - taken; + owed -= taken; + } + if (owed > 0) { + // More was given back than the window is holding — a settle without a + // matching reserve, or one whose reservation has already aged out. + // Refusing to go negative is right; saying so is what makes it + // diagnosable rather than a counter that quietly disagrees with D1. + warn("Usage meter rate refund exceeded the window's charge", { + meter, + delta, + unapplied: owed, + }); + } + } + + await this.ctx.storage.put(RATE_KEY, { + meters: { ...record.meters, [meter]: live }, + }); + } + + /** + * Arms the cleanup alarm for `atMs` unless a later one is already set. A DO + * has one alarm and two lifetimes to sweep, so the later time wins and + * `alarm()` re-arms for whatever it did not erase. + */ + private async armCleanup(atMs: number): Promise { + // Capped, because `armCleanup` only ever RAISES the alarm: one bad input + // that armed it centuries out could never be lowered by a later correct + // call, and the subject's storage would never be swept. + const ceiling = Date.now() + MAX_ALARM_HORIZON_MS; + const at = Math.min(atMs, ceiling); + if (at !== atMs) { + warn("Usage meter cleanup alarm clamped to the horizon", { requestedAtMs: atMs, atMs: at }); + } + const existing = await this.ctx.storage.getAlarm(); + if (existing === null || existing < at) await this.ctx.storage.setAlarm(at); + } +} + +/** + * The band a caller's `nowMs` must fall in to be a timestamp at all. + * + * Deliberately an ABSOLUTE range rather than a tolerance around this object's + * own clock. What this rejects is the wrong UNIT or the wrong epoch — seconds + * (which land in 1970), microseconds or nanoseconds (which land tens of + * thousands of years out) — not skew, which is real and must keep working. A + * tolerance around `Date.now()` would also make the check depend on wall time, + * so a test with a fixed synthetic clock would pass or fail by the calendar. + * + * The failure this closes: a microsecond value passes `Number.isFinite`, lands + * a bucket ~1000x into the future where it counts against every later window + * forever, and re-arms the cleanup alarm past the year 58000. The subject's + * evaluations are refused permanently and its storage is stranded — the same + * "metering causes the outage" failure the malformed-input deviation exists to + * prevent, arriving through a value that looked finite. + */ +const MIN_PLAUSIBLE_MS = Date.UTC(2020, 0, 1); +const MAX_PLAUSIBLE_MS = Date.UTC(2200, 0, 1); + +/** + * The furthest ahead a cleanup alarm may ever be set. Longest month plus slack: + * nothing this object sweeps has a lifetime beyond one period, so an alarm + * further out than this is arithmetic on a bad input, and `armCleanup` only + * ever RAISES — so one bad value could never be lowered by a later good one. + */ +const MAX_ALARM_HORIZON_MS = 40 * 24 * 60 * 60 * 1000; + +/** Is `nowMs` a timestamp this object can do window arithmetic with? */ +function isUsableClock(nowMs: number): boolean { + return Number.isFinite(nowMs) && nowMs >= MIN_PLAUSIBLE_MS && nowMs <= MAX_PLAUSIBLE_MS; +} + +function isRateMeter(meter: UsageMeterKey): meter is RateMeterKey { + return meter in RATE_WINDOWS; +} + +/** + * A quantity that can safely be added to a counter, or null. + * + * `NaN` or an infinity would poison the counter permanently — no later + * comparison against a limit could be true again — and a negative would refund + * the period, which is what `reserve` is here to stop. `settle` is the only + * signed path, and it clamps at zero. + */ +function usableQuantity(value: number): number | null { + return Number.isFinite(value) && value >= 0 ? value : null; +} + +/** + * Whether a reservation fits, and whether the limit was a limit at all. + * + * `malformed` is reported rather than folded into `admitted` so the caller's + * log can tell "admitted under the allowance" from "admitted because the + * billing payload was garbage" — the distinction `EntitlementsSource` exists + * for one layer up, and a month of observation that cannot make it is a month + * of unreadable data. + */ +function limitAdmits( + current: number, + quantity: number, + limit: number, +): { admitted: boolean; malformed: boolean } { + if (!Number.isInteger(limit) || limit < UNLIMITED) return { admitted: true, malformed: true }; + if (limit === UNLIMITED) return { admitted: true, malformed: false }; + // `0` is a hard block, not "unset" (see `src/billing/entitlements.ts`): a + // subject limited to zero may not consume the resource at all, so a + // zero-quantity reservation must not sneak past on the arithmetic below. + if (limit === 0) return { admitted: false, malformed: false }; + return { admitted: current + quantity <= limit, malformed: false }; +} + +/** The instant a 'YYYY-MM' UTC period ends, or null when it is not one. */ +function periodEndMs(period: string): number | null { + const match = /^(\d{4})-(\d{2})$/.exec(period); + if (!match) return null; + const year = Number(match[1]); + const month = Number(match[2]); + if (month < 1 || month > 12) return null; + // Date.UTC's month is 0-based, so passing the 1-based month lands on the + // first instant of the NEXT month — the end of this one. + return Date.UTC(year, month, 1); +} + +function bucketId(nowMs: number, bucketSeconds: number): number { + return Math.floor(nowMs / 1000 / bucketSeconds); +} + +function bucketsPerWindow(window: RateWindow): number { + return Math.max(1, Math.ceil(window.windowSeconds / window.bucketSeconds)); +} + +/** The buckets still inside the window ending at `bucket`, as a fresh object. */ +function liveBuckets( + buckets: RateBuckets | undefined, + bucket: number, + window: RateWindow, +): RateBuckets { + const oldest = bucket - bucketsPerWindow(window) + 1; + const live: RateBuckets = {}; + for (const [id, quantity] of Object.entries(buckets ?? {})) { + const parsed = Number(id); + // A bucket in the FUTURE is kept: a caller's clock that ran ahead once must + // not have its reservation dropped by the next caller's slower clock. + if (Number.isFinite(parsed) && parsed >= oldest) live[id] = quantity; + } + return live; +} + +function sum(buckets: RateBuckets): number { + let total = 0; + for (const quantity of Object.values(buckets)) total += quantity; + return total; +} + +/** When a bucket stops counting toward any window, plus the cleanup grace. */ +function rateExpiresAt(bucket: number, window: RateWindow): number { + const bucketMs = window.bucketSeconds * 1000; + return (bucket + bucketsPerWindow(window)) * bucketMs + CLEANUP_GRACE_MS; +} + +/** + * Drop every bucket that has fallen out of its window. Returns null when + * nothing is left to keep, so the caller can erase the key rather than store an + * empty shell. + */ +function pruneRates( + record: RateRecord, + nowMs: number, +): { record: RateRecord; expiresAt: number } | null { + const meters: Partial> = {}; + let expiresAt = 0; + for (const [name, buckets] of Object.entries(record.meters ?? {})) { + if (!isRateMeter(name as UsageMeterKey)) continue; + const meter = name as RateMeterKey; + const window = RATE_WINDOWS[meter]; + const live = liveBuckets(buckets, bucketId(nowMs, window.bucketSeconds), window); + const ids = Object.keys(live); + if (ids.length === 0) continue; + meters[meter] = live; + const newest = Math.max(...ids.map(Number)); + expiresAt = Math.max(expiresAt, rateExpiresAt(newest, window)); + } + return expiresAt === 0 ? null : { record: { meters }, expiresAt }; +} + +// One logger per isolate rather than one per call: these paths are rare (a +// malformed input) but sit on the merge path, and `createLogger` builds a pino +// instance every time it is called. +let meterLogger: Logger | undefined; + +/** + * Every path that admits without counting, and every settle that cannot be + * applied, logs. Each one silently ADMITS or silently keeps a charge (see the + * class doc comment), and AGENTS.md forbids swallowing a failure. A limit that + * stopped limiting, or a refund that never landed, must be visible in Workers + * Logs rather than inferred from a bill. + * + * Values that may be `NaN` or an infinity are stringified by the caller: JSON + * collapses all three to `null`, which would make the log unable to report the + * exact value it exists to report. + */ +function warn(message: string, meta: Record): void { + meterLogger ??= createLogger({ component: "usage-meter" }); + meterLogger.warn(message, meta); +} diff --git a/src/routes/changes.ts b/src/routes/changes.ts index caca75af..72b83744 100644 --- a/src/routes/changes.ts +++ b/src/routes/changes.ts @@ -1,5 +1,6 @@ import { type Context, Hono } from "hono"; import { diffTouchesProtectedConfig, loadPolicy } from "../evaluation"; +import { llmProviderCatalog } from "../evaluation/llm-providers"; import type { EvalPolicy } from "../evaluation/types"; import { GitHubClient } from "../github/client"; import { buildEvaluationReport, reportEvaluationToGitHub } from "../github/sync"; @@ -9,6 +10,7 @@ import { enqueueMergeDeploy } from "../queue/deploy-queue"; import { emitEvent } from "../queue/events"; import type { MergeOutcome } from "../queue/merge-queue"; import { + billingContextFor, buildEvaluators, createChangeWithEvaluation, resolveProjectHead, @@ -24,7 +26,12 @@ import { mergeTransitionOpts, updateChangeStatus, } from "../storage/changes"; -import { type CostSample, getChangeCostSummary, recordCosts } from "../storage/costs"; +import { + type CostSample, + getChangeCostSummary, + recordCosts, + resolveBillingSubject, +} from "../storage/costs"; import { isTargetDeleting } from "../storage/deletion"; import { listEvalRuns, recordEvalRuns } from "../storage/eval-runs"; import { @@ -129,6 +136,7 @@ async function loadMergePolicyCached( tok.data, logger, projectDefaultBranch(project), + llmProviderCatalog(env, logger), ); if (cacheable) { policyCache.set(project.id, { policy: loaded, expires: Date.now() + POLICY_CACHE_TTL_MS }); @@ -800,6 +808,7 @@ app.post("/changes/:id/merge", async (c) => { }); } + const mergeSubject = await resolveBillingSubject(c.env.DB, logger, project); await recordCosts( c.env.DB, logger, @@ -809,6 +818,9 @@ app.post("/changes/:id/merge", async (c) => { projectId: change.projectId ?? project.id, changeId: id, workspace: change.workspace, + // The project's owner now, not the author of the change: this is the + // account the clone and push were spent against. + ...(mergeSubject ?? {}), }, [{ kind: "git_ops", quantity: 2 }], ); @@ -1373,7 +1385,13 @@ app.post("/changes/:id/evaluate", async (c) => { if (!workspaceReadToken.success) return internalError(workspaceReadToken.error.message); const branch = projectDefaultBranch(project); - const policy = await loadPolicy(project.remote, projectReadToken.data, logger, branch); + const policy = await loadPolicy( + project.remote, + projectReadToken.data, + logger, + branch, + llmProviderCatalog(c.env, logger), + ); const diffResult = await getDiffBetweenRepos( project.remote, @@ -1397,16 +1415,30 @@ app.post("/changes/:id/evaluate", async (c) => { baseOid, } = diffResult.data; - const evaluators = buildEvaluators(c.env, policy, change.project, logger, { - remote: workspace.remote, - token: workspaceReadToken.data, - ref: evaluatedSha, - }); + const evaluators = await buildEvaluators( + c.env, + policy, + project, + logger, + { + remote: workspace.remote, + token: workspaceReadToken.data, + ref: evaluatedSha, + }, + getWaitUntil(c), + ); // The base this re-evaluation's diff was built against — not `change.baseSha`, // which records the base at creation and is exactly the value that has gone // stale by the time a change is re-evaluated (#274). + // One resolution for both the meters and the ledger below (see + // `billingContextFor`): an agent-owned project's payer is a D1 walk, and + // resolving it twice would pay for it twice and could name two payers. + const evaluateSubject = await resolveBillingSubject(c.env.DB, logger, project); const { evalRuns, evalResult } = await runEvaluation(evaluators, diff, policy, logger, { baseSha: baseOid, + // This route is user-credentialed (agent tokens are rejected above), so the + // acting user is the caller — the subject PRD §4a checks a limit against. + billing: billingContextFor(evaluateSubject, project.id, userId), }); const recordResult = await recordEvalRuns(c.env.DB, logger, id, evalRuns); @@ -1427,6 +1459,8 @@ app.post("/changes/:id/evaluate", async (c) => { projectId: change.projectId ?? project.id, changeId: id, workspace: change.workspace, + ...(evaluateSubject ?? {}), + notify: { env: c.env, actorUserId: userId, waitUntil: getWaitUntil(c) }, }, evaluateCostSamples, ); diff --git a/src/routes/deployments.ts b/src/routes/deployments.ts index 7c4f3a4b..c6234189 100644 --- a/src/routes/deployments.ts +++ b/src/routes/deployments.ts @@ -12,9 +12,13 @@ * path. `canReadProject` returns true unconditionally for a public project, * and a provider error payload can echo request context back at us. * - * No route on either router returns a secret value; there is no read path for - * one anywhere in the codebase (`loadSecretValues` is the deploy runner's, and - * is not reachable from here). + * No route on either router returns a secret value. `loadSecretValues` is the + * only function that yields one, and it is not reachable from this router at + * all — but it is no longer route-unreachable in general: BYOK resolves a + * project's provider credential on the change-creation path + * (`evaluation/llm-byok.ts`). That value never comes back out either; it goes + * straight into the provider request header. The invariant these routes keep is + * the narrower, checkable one: nothing here reads a value. * * The browser talks to these same routes. An HTML form can only issue GET or * POST and cannot read a JSON body, so the secret writes have form-friendly diff --git a/src/routes/projects.ts b/src/routes/projects.ts index d6fd49b9..f1fbfe00 100644 --- a/src/routes/projects.ts +++ b/src/routes/projects.ts @@ -1,5 +1,7 @@ import { Hono } from "hono"; import type { Context } from "hono"; +import { checkGauge, resolveEnforcementSubject } from "../billing/enforcement"; +import { entitlementsEnabled } from "../billing/entitlements"; import { importRateLimitMiddleware, rateLimitMiddleware, @@ -65,6 +67,7 @@ import { filterReadableProjects, isDirectOwner, } from "../utils/authz"; +import { getWaitUntil } from "../utils/execution-ctx"; import { createLogger } from "../utils/logger"; import type { Logger } from "../utils/logger"; import { readJsonWithLimit } from "../utils/request-body"; @@ -181,6 +184,54 @@ app.post("/", async (c) => { visibility = body.visibility; } + // The private-project gauge (PRD §8): a thing that exists rather than a thing + // that is spent, so it is counted on demand and there is nothing to reserve. + // + // **The counted set and the limit belong to the same subject**, which is the + // only reading that is not a hole in one direction or the other. Counting the + // TARGET namespace against the ACTOR's limit — what this did first — meant an + // actor at their limit created a free org and got a count of zero, the "ten + // orgs" reset §4a closes everywhere else; and the inverse bit just as hard, + // since being added to a busy org exhausted your personal gauge against + // projects you do not own. So the namespace listed is the subject's own: the + // actor's personal namespace for the actor (§4a's usual answer), and the org's + // for an org positively known to pool. KV has no index by owner, and a + // namespace listing is exactly "everything this subject owns" without one. + // + // Gated on `entitlementsEnabled` FIRST: the listing is a KV list of a whole + // namespace, and a self-hoster must not pay for it to feed a check that + // `checkGauge` would return `inert()` from anyway. + if (visibility === "private" && entitlementsEnabled(c.env)) { + const gaugeSubject = await resolveEnforcementSubject(c.env, logger, { + actorUserId: userId, + owner: { ownerId: owner.id, ownerType: owner.type }, + waitUntil: getWaitUntil(c), + }); + // An org subject can only be the pooled one (§4a), and its namespace is the + // one being created into; anything else is the actor, whose namespace is + // their own — never the target's. + const gaugeNamespace = + gaugeSubject?.ownerType === "org" ? namespace : getUserNamespace(username); + if (gaugeSubject) { + const existing = await listProjectsByNamespace(c.env.STATE, gaugeNamespace, logger); + // A failed listing does not block a creation: the gauge is a limit, not a + // correctness rule, and this layer fails open by design. + if (existing.success) { + const decision = await checkGauge(c.env, logger, { + subject: gaugeSubject, + count: "private_projects", + current: existing.data.filter( + (entry) => entry.visibility === "private" && entry.ownerId === gaugeSubject.ownerId, + ).length, + what: "Creating another private project", + }); + if (!decision.admitted) { + return forbidden(decision.reason ?? "Private project limit reached"); + } + } + } + } + const seed = body.seed === "true" || body.seed === true || c.req.query("seed") === "true"; const files = body.files !== undefined diff --git a/src/routes/sync-management.ts b/src/routes/sync-management.ts index de472a2a..646ccd1d 100644 --- a/src/routes/sync-management.ts +++ b/src/routes/sync-management.ts @@ -1,11 +1,13 @@ import { Hono } from "hono"; +import { llmProviderCatalog } from "../evaluation/llm-providers"; import { loadPolicy } from "../evaluation/policy-loader"; import { scanContentForSecrets } from "../evaluation/secret-scanner"; import { checkResolutionMergeProtection } from "../merge/protection"; import { authMiddleware } from "../middleware/auth"; -import { buildEvaluators, runEvaluation } from "../services/change-flow"; +import { billingContextFor, buildEvaluators, runEvaluation } from "../services/change-flow"; import { recordAudit } from "../storage/audit"; import { getChange } from "../storage/changes"; +import { recordCosts, resolveBillingSubject } from "../storage/costs"; import { MAX_FILE_BYTES, buildManualResolutionDiff, @@ -22,8 +24,9 @@ import { updateProjectAfterSync, } from "../storage/sync"; import type { Env } from "../types"; -import { projectDefaultBranch } from "../types"; +import { projectDefaultBranch, projectDisplayName } from "../types"; import { canWriteProject } from "../utils/authz"; +import { getWaitUntil } from "../utils/execution-ctx"; import { createLogger } from "../utils/logger"; import { readJsonWithLimit } from "../utils/request-body"; import { notFound, ok } from "../utils/response"; @@ -619,6 +622,12 @@ app.post("/projects/conflicts/:id/resolve", async (c) => { if (!projectToken.success) return c.json({ error: projectToken.error.message }, 502); if (!workspaceToken.success) return c.json({ error: workspaceToken.error.message }, 502); + // One resolution for every billing site on this route. Both recording sites + // and the manual path's meters attribute the same project, and for an + // agent-owned one this is a `getAgent` D1 read — paid for once, and unable to + // name two different payers for the same request. + const billingSubject = await resolveBillingSubject(c.env.DB, logger, project); + // Route a manual resolution through the same merge gate a normal Change goes // through: the project's configured evaluator suite, then merge protection // (required evaluators + required approvals), both run against the exact @@ -655,20 +664,58 @@ app.post("/projects/conflicts/:id/resolve", async (c) => { } const { diff, baseSha } = diffResult.data; - const policy = await loadPolicy(project.remote, projectToken.data, logger, branch); - const projectName = `${conflictCtx.namespace}/${conflictCtx.slug}`; + const policy = await loadPolicy( + project.remote, + projectToken.data, + logger, + branch, + llmProviderCatalog(c.env, logger), + ); // No workspace repo access is passed for the sandbox evaluator: the content // being judged here has no commit of its own yet — that's the point, it must // pass BEFORE resolveConflict creates one — so there is no ref a sandbox // could check out. A policy naming `sandbox` fails closed via // UnavailableEvaluator, same as any other missing prerequisite. - const evaluators = buildEvaluators(c.env, policy, projectName, logger); + const evaluators = await buildEvaluators( + c.env, + policy, + project, + logger, + undefined, + getWaitUntil(c), + ); // `buildManualResolutionDiff` resolved this base from the clone it built the // diff on, so it names the tree the resolution actually applies to (#274). + // This path runs the LLM evaluator too, so it needs a payer as much as + // change creation does. const { evalRuns, evalResult } = await runEvaluation(evaluators, diff, policy, logger, { baseSha, + // The resolver is the actor: this route is user-credentialed, and PRD §4a + // checks the allowance against the person who ran the suite. + billing: billingContextFor(billingSubject, project.id, userId), }); + // Recorded here, BEFORE the verdict is acted on, because the spend is + // already incurred: a resolution the suite rejects burned exactly the same + // model tokens as one it accepts, and the early return below would drop + // them. Mirrors the recording POST /changes/:id/evaluate does, with one + // git_op rather than its pair — `buildManualResolutionDiff` cloned once. + await recordCosts( + c.env.DB, + logger, + { + project: projectDisplayName(project), + projectId: project.id, + // The Change whose merge attempt produced this conflict, when the + // conflict context named one — a resolution is part of landing it. + ...(conflictCtx.changeId ? { changeId: conflictCtx.changeId } : {}), + workspace: conflictCtx.workspaceName, + ...(billingSubject ?? {}), + notify: { env: c.env, actorUserId: userId, waitUntil: getWaitUntil(c) }, + }, + [{ kind: "git_ops", quantity: 1 }, ...evalRuns.flatMap(({ result }) => result.costs ?? [])], + ); + if (!evalResult.passed) { logger.warn("Manual conflict resolution blocked by evaluator suite", { conflictId, @@ -759,6 +806,36 @@ app.post("/projects/conflicts/:id/resolve", async (c) => { logger, ); + // `resolveConflict` runs for EVERY strategy, so this is recorded here rather + // than in the `manual` branch above, which only ever sees the evaluator + // suite's spend. Without it, accept-project and accept-workspace resolutions + // record nothing at all, and manual under-records the git work it paid for. + // + // Recorded before the failure branch below, for the same reason the manual + // branch records before its 422: a resolution that clones and then fails to + // push cost the operator what a successful one did, and a failure must not be + // the cheap way to use the platform. + // + // The count is what the strategy WOULD do, not a measurement — `resolveConflict` + // reports only a commit sha, so there is nothing to measure. It therefore + // over-bills a resolution that dies before its first clone (a rejected token) + // and is exact otherwise. Tolerable while `git_ops` maps to no meter + // (`meterForCostKind`); if it ever gains one, `resolveConflict` has to report + // the operations it actually completed rather than being guessed at here. + const resolveGitOps = strategy === "accept-workspace" ? 3 : 2; + await recordCosts( + c.env.DB, + logger, + { + project: projectDisplayName(project), + projectId: project.id, + ...(conflictCtx.changeId ? { changeId: conflictCtx.changeId } : {}), + workspace: conflictCtx.workspaceName, + ...(billingSubject ?? {}), + }, + [{ kind: "git_ops", quantity: resolveGitOps }], + ); + if (!resolveResult.success) { const status = resolveResult.error.statusCode === 401 ? 401 : 422; return c.json({ error: resolveResult.error.message, code: resolveResult.error.code }, status); diff --git a/src/routes/ui.tsx b/src/routes/ui.tsx index 4ee339b6..c3a982b1 100644 --- a/src/routes/ui.tsx +++ b/src/routes/ui.tsx @@ -1,6 +1,8 @@ import { Hono } from "hono"; import type { Context } from "hono"; import { fetchInviteCodes, referralServiceConfigured } from "../beta/gate"; +import { type UsageBannerNotice, loadUsageBanner } from "../billing/usage-banner"; +import { buildUsageReport } from "../billing/usage-report"; import { cannotMintLegacyCredential } from "../middleware/auth"; import { NO_ANALYTICS_HEADER } from "../middleware/web-analytics"; import { createAgent, deleteAgent, getAgent, listAgents } from "../storage/agents"; @@ -48,6 +50,7 @@ import { listWorkspaces, } from "../storage/state"; import { getProjectSourceUrl, getSyncStatus } from "../storage/sync"; +import { usagePeriod } from "../storage/usage"; import { disableLegacyToken, getUser, @@ -81,6 +84,7 @@ import { } from "../ui/pages/settings"; import { SyncPage } from "../ui/pages/sync"; import { TagsPage } from "../ui/pages/tags"; +import { UsagePage } from "../ui/pages/usage"; import { WebhooksPage } from "../ui/pages/webhooks"; import { WorkspacesPage } from "../ui/pages/workspaces"; import { canReadProject, canWriteProject, filterMemberProjects } from "../utils/authz"; @@ -145,6 +149,23 @@ async function getCurrentUser( }; } +/** + * The 80% usage warning for the signed-in browser user, or `null`. + * + * Threaded into the pages a user actually visits — dashboard, settings, the + * changes list and a change's detail — rather than only onto `/settings/usage`, + * because a page nobody opens warns nobody (PRD §8). It costs **one KV read** + * and no D1: the crossing was already detected and stored by + * `usage-notifications.ts` at the moment the usage was written. Signed-out + * visitors and self-hosted instances take no read at all. + */ +async function usageBannerFor( + c: Context<{ Bindings: Env }>, + logger: ReturnType, +): Promise { + return loadUsageBanner(c.env, logger, c.get("userId"), usagePeriod()); +} + /** * Resolve issue author ids to display names ("@username" for users, the agent's * name for agents). Unresolvable authors fall back to their author type. @@ -215,7 +236,9 @@ app.get("/", async (c) => { })); logger.debug("Rendering home page", { projectCount: view.length }); - return c.html(); + return c.html( + , + ); }); // GET /new — New project form @@ -409,7 +432,7 @@ async function renderSettings( logger: ReturnType, extras: { freshToken?: FreshCredential; notice?: SettingsNotice } = {}, ) { - const [agents, tokens, grants, usernameChange, invites] = await Promise.all([ + const [agents, tokens, grants, usernameChange, invites, usageNotice] = await Promise.all([ loadAgentSummaries(c.env.DB, user.id, logger), loadApiTokens(c.env.DB, user.id, logger), loadOAuthGrants(c.env.DB, user.id, logger), @@ -420,6 +443,7 @@ async function renderSettings( referralServiceConfigured(c.env) ? fetchInviteCodes(c.env, user.id, logger) : Promise.resolve(undefined), + usageBannerFor(c, logger), ]); // A notice about the action just taken outranks a listing's own failure — // the caller needs to know what their POST did first. @@ -439,6 +463,7 @@ async function renderSettings( // is what keeps them from rendering identically — see `loadOAuthGrants`. oauthGrantsUnavailable={grants.notice !== undefined} telemetryOptOut={telemetryOptOut} + usageNotice={usageNotice} // Threaded here rather than at each call site: this helper is the single // render path for the settings page, so the CSP nonce main added cannot // be forgotten by a future caller. @@ -497,6 +522,39 @@ app.get("/settings", async (c) => { return c.html(page); }); +/** + * GET /settings/usage — this account's consumption against its allowances. + * + * A page of its own, not a section of `/settings`: that route renders one + * hash-anchored document (which `/profile` redirects into), and usage is a + * different question with a different read behind it. + * + * Users only, and a browser session only, like every other account page. There + * is deliberately no organization view — this application has no org UI, and + * under PRD §4a an allowance is checked against the person anyway, so what is + * shown here is exactly what every limit is compared to. + */ +app.get("/settings/usage", async (c) => { + const logger = createLogger({ path: c.req.path, userId: c.get("userId") }); + const access = await requireAccountSession(c, logger); + if ("response" in access) return access.response; + + const [report, usageNotice] = await Promise.all([ + buildUsageReport(c.env, logger, { actorUserId: access.user.id, includeByok: true }), + usageBannerFor(c, logger), + ]); + if (!report.success) { + // The allowances are still worth showing when only the totals failed to + // read, but this build cannot tell those apart, so the page says plainly + // that the figures are incomplete rather than rendering zeroes as fact. + logger.error("Failed to build usage report", report.error); + const message = "Your usage could not be loaded. Please try again."; + return c.html(errorPage(500, message, access.user), 500); + } + + return c.html(); +}); + // POST /settings/account — Set or clear the display name app.post("/settings/account", async (c) => { const logger = createLogger({ path: c.req.path, userId: c.get("userId") }); @@ -1218,6 +1276,7 @@ app.get("/p/:name/changes", async (c) => { changes={view} canWrite={canWrite} user={userResult} + usageNotice={await usageBannerFor(c, logger)} />, ); }); @@ -1344,6 +1403,7 @@ app.get("/changes/:id", async (c) => { canReview={canReview} projectRef={projectRef(projectResult.data)} user={userResult} + usageNotice={await usageBannerFor(c, logger)} />, ); }); @@ -1449,6 +1509,7 @@ app.get("/:namespace/:slug/changes", async (c) => { changes={changes} canWrite={canWrite} user={userResult} + usageNotice={await usageBannerFor(c, logger)} />, ); }); diff --git a/src/routes/users.ts b/src/routes/users.ts index d5ab3e8e..a903f0c7 100644 --- a/src/routes/users.ts +++ b/src/routes/users.ts @@ -1,5 +1,6 @@ import { Hono } from "hono"; import type { Context } from "hono"; +import { buildUsageReport } from "../billing/usage-report"; import { cannotMintLegacyCredential } from "../middleware/auth"; import { MAX_TOKEN_EXPIRY_DAYS, @@ -62,6 +63,41 @@ app.get("/me", async (c) => { return ok({ id: user.id, email: user.email, createdAt: user.createdAt }); }); +/** + * GET /api/users/me/usage — the caller's metered usage against their plan. + * + * Read-only, and deliberately the ONLY billing surface reachable from a token: + * there is no endpoint here to raise a limit, buy capacity or attach a payment + * method (PRD §4c). An agent that can lift its own ceiling does not have one, + * which is the same line this codebase already draws at review verdicts. + * + * An AGENT token resolves to its owner (`agentOwnerId`), because that is the + * §4a enforcement subject its evaluations are actually charged to — an agent + * shown its own empty allowance would be reading a number no check uses. + */ +app.get("/me/usage", async (c) => { + const logger = createLogger({ + requestId: crypto.randomUUID(), + path: c.req.path, + method: c.req.method, + userId: c.get("userId"), + }); + + // The acting user, or the person an agent acts for. Never a project's org: + // limits follow the person (PRD §4a). + const actorUserId = c.get("userId") ?? c.get("agentOwnerId"); + if (!actorUserId) { + return c.json({ error: "Unauthorized" }, 401); + } + + const report = await buildUsageReport(c.env, logger, { actorUserId }); + if (!report.success) { + logger.error("Failed to build usage report", report.error); + return internalError("Usage could not be read"); + } + return ok(report.data); +}); + // POST /api/users/me/rotate-token - Replace the caller's API key app.post("/me/rotate-token", async (c) => { const logger = createLogger({ diff --git a/src/services/change-flow.ts b/src/services/change-flow.ts index de199e6d..e07fab3e 100644 --- a/src/services/change-flow.ts +++ b/src/services/change-flow.ts @@ -5,16 +5,30 @@ import { SandboxEvaluator, SecretScanEvaluator, WebhookEvaluator, + WorkersAiProvider, diffTouchesProtectedConfig, loadPolicy, } from "../evaluation"; +import { resolveLlmProvider } from "../evaluation/llm-byok"; +import { llmProviderCatalog } from "../evaluation/llm-providers"; import type { SandboxRepoAccess } from "../evaluation/sandbox-evaluator"; -import type { EvalPolicy, EvalResult, EvaluationContext, Evaluator } from "../evaluation/types"; +import type { + BillingContext, + EvalPolicy, + EvalResult, + EvaluationContext, + Evaluator, +} from "../evaluation/types"; import { buildEvaluationReport, reportEvaluationToGitHub } from "../github/sync"; import { type EventActor, emitEvent } from "../queue/events"; import { getAgent } from "../storage/agents"; import { createChange, updateChangeStatus } from "../storage/changes"; -import { type CostSample, recordCosts } from "../storage/costs"; +import { + type BillingSubject, + type CostSample, + recordCosts, + resolveBillingSubject, +} from "../storage/costs"; import { recordEvalRuns } from "../storage/eval-runs"; import { artifactsRepoNameFromRemote, @@ -30,6 +44,7 @@ import { type ProjectEntry, getArtifactsRepoName, projectDefaultBranch, + projectDisplayName, } from "../types"; import { AppError } from "../utils/errors"; import type { Logger } from "../utils/logger"; @@ -96,11 +111,58 @@ export class UnavailableEvaluator implements Evaluator { } } +/** + * The evaluation-time view of an already-resolved billing subject, or + * `undefined` when none could be named. + * + * Takes the subject rather than the `ProjectEntry` it came from, and that is the + * whole point of the signature. `ownerType` admits `"agent"` and an agent is not + * a payer — it belongs to one — so naming the payer means walking + * `agents.owner_id`, which is D1 and therefore async. This function used to + * refuse an agent-owned project instead, which left `LLMEvaluator` with no + * billing context and so no meter check at all: the walk existed in + * `resolveEnforcementSubject` and nothing ever reached it. Every call site now + * resolves once with `resolveBillingSubject` and passes the result to both this + * and `recordCosts`, so the allowance and the ledger cannot disagree about who + * pays, and neither pays for two D1 reads of the same fact. + * + * `projectId` is checked because KV entries are cast without validation + * (`src/storage/state.ts`) and legacy rows can lack fields the type promises. A + * context keyed on `""` would be worse than none: spend that *looks* attributed + * aggregates under an empty key no account will ever reconcile. + * + * @param subject - From `resolveBillingSubject`; `null` when nobody can be named + * @param projectId - The project whose policy is being enforced + * @param actorUserId - The user who ran the evaluation, when the caller knows + * one. Carried alongside the payer rather than instead of it: PRD §4a checks a + * limit against the actor while the ledger keeps recording the owner. For an + * agent-authored change this is the agent's OWNER. + */ +export function billingContextFor( + subject: BillingSubject | null, + projectId: string | undefined, + actorUserId?: string, +): BillingContext | undefined { + if (!subject || !projectId) return undefined; + return { + ownerId: subject.ownerId, + ownerType: subject.ownerType, + projectId, + ...(actorUserId ? { actorUserId } : {}), + }; +} + /** * Build the evaluator set for a policy: the always-on blocking secret scan plus * whatever the policy configures. Evaluators whose binding is missing become * UnavailableEvaluator (score 0, fail) rather than silently vanishing. * + * Takes the whole `ProjectEntry` rather than a display name because the payer + * has to be nameable from it. Nothing built here receives the project: the + * billing subject reaches evaluators on the `EvaluationContext` that + * `runEvaluation` forwards, and inside this function the entry is used for the + * log line below and nothing else. + * * `workspaceRepo` is read access to the workspace being evaluated (remote + * read token + the pinned evaluated commit); the sandbox evaluator needs it to * materialize the full tree it runs the test command against. Without it — or @@ -108,13 +170,33 @@ export class UnavailableEvaluator implements Evaluator { * ops decision) — a policy naming `sandbox` fails closed with a reason that * says exactly which prerequisite is missing. */ -export function buildEvaluators( +export async function buildEvaluators( env: Env, policy: EvalPolicy, - projectName: string, + project: ProjectEntry, logger: Logger, workspaceRepo?: SandboxRepoAccess, -): Array<{ type: string; evaluator: Evaluator }> { + /** + * Cloudflare Workers `ExecutionContext.waitUntil`, when the caller has one. + * + * Used by the metered evaluator for one thing: warming an ORG's entitlements + * off the response path (PRD §4a). Auth middleware warms the caller, but it + * knows the caller and not the project, so nothing before this point can warm + * the org a project belongs to. Absent means no warm — the check then resolves + * to the actor, which is the safe direction and the documented first-contact + * behaviour, never a blocking fetch on the merge path. + */ + waitUntil?: (promise: Promise) => void, +): Promise> { + // Async only because of this: a policy that selects a BYOK provider needs the + // project's credential resolved before the evaluator exists. It is a no-op — + // no D1 read, no key derivation — for every policy that names no provider, + // which is every policy that has not opted in. + const llmProvider = await resolveLlmProvider(env, project, policy, logger); + + // Guarded, because a legacy entry can carry no namespace — see + // `projectDisplayName`, and this file's own `resolveProjectHead`. + const projectName = projectDisplayName(project); const evaluators: Array<{ type: string; evaluator: Evaluator }> = [ { type: "secret_scan", evaluator: new SecretScanEvaluator() }, ]; @@ -126,14 +208,40 @@ export function buildEvaluators( return [{ type: "diff", evaluator: new DiffEvaluator() }]; case "webhook": return [{ type: "webhook", evaluator: new WebhookEvaluator() }]; - case "llm": - if (env.AI) return [{ type: "llm", evaluator: new LLMEvaluator(env.AI) }]; + case "llm": { + // Every failure to resolve a project-supplied credential lands here, + // and NONE of them falls back to `env.AI`. Falling back would move + // the spend to the operator's bill — the hole BYOK closes — and would + // turn a misconfigured gate into one that quietly passes on someone + // else's budget. Fail closed with the reason instead. + if (llmProvider.status === "unavailable") { + return [ + { type: "llm", evaluator: new UnavailableEvaluator("llm", llmProvider.reason) }, + ]; + } + const enforcement = { env, ...(waitUntil !== undefined ? { waitUntil } : {}) }; + if (llmProvider.status === "byok") { + return [ + { + type: "llm", + evaluator: new LLMEvaluator(llmProvider.provider, "byok", enforcement), + }, + ]; + } + if (env.AI) + return [ + { + type: "llm", + evaluator: new LLMEvaluator(new WorkersAiProvider(env.AI), "platform", enforcement), + }, + ]; return [ { type: "llm", evaluator: new UnavailableEvaluator("llm", "AI binding is not configured"), }, ]; + } case "sandbox": if (!env.SANDBOX) { // Fail closed with an actionable reason: the [[sandboxes]] binding @@ -191,21 +299,47 @@ export async function runEvaluation( diff: string, policy: EvalPolicy, logger: Logger, - /** What the diff is a diff of. Forwarded to every evaluator so one that - * reproduces the change out-of-process can pin the base it applies to (#274). - * Callers pass the base resolved from the clone the diff came from, never a - * fresh read of the project head. */ + /** What the diff is a diff of, and who pays for evaluating it. Forwarded to + * every evaluator so one that reproduces the change out-of-process can pin + * the base it applies to (#274). Callers pass the base resolved from the + * clone the diff came from, never a fresh read of the project head, and the + * billing subject from `billingContextFor`. + * + * One object is shared by every evaluator in the `Promise.all` below, so it + * must stay immutable data. Hanging a mutable balance or counter off it would + * be a race across concurrently running evaluators, not a shared budget. */ context?: EvaluationContext, ): Promise<{ evalRuns: EvaluationRun[]; evalResult: EvalResult }> { const evalRuns = await Promise.all( evaluators.map(async ({ type, evaluator }) => { - const result = await evaluator.evaluate(diff, policy, logger, context); - return { - evaluatorType: type, - result: result.success - ? result.data - : { score: 0, passed: false, reason: result.error.message }, - }; + // `evaluate` returns a Result by contract, and this catch is what keeps + // that contract from being a promise the caller cannot rely on. One + // evaluator that REJECTS — a Durable Object RPC that could not be + // reached, a runtime fault — would otherwise reject this `Promise.all` + // and take change creation down with it, discarding every other + // evaluator's verdict on the way. A gate that could not run is a gate + // that did not pass; it is reported as a failing verdict, exactly as a + // returned error already is, and never as a missing one. + try { + const result = await evaluator.evaluate(diff, policy, logger, context); + return { + evaluatorType: type, + result: result.success + ? result.data + : { score: 0, passed: false, reason: result.error.message }, + }; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + logger.error( + `Evaluator "${type}" threw instead of returning a result`, + error instanceof Error ? error : new Error(message), + { evaluatorType: type }, + ); + return { + evaluatorType: type, + result: { score: 0, passed: false, reason: `${type} evaluator failed: ${message}` }, + }; + } }), ); @@ -367,7 +501,13 @@ export async function createChangeWithEvaluation( } const branch = projectDefaultBranch(project); - const policy = await loadPolicy(project.remote, projectReadToken.data, logger, branch); + const policy = await loadPolicy( + project.remote, + projectReadToken.data, + logger, + branch, + llmProviderCatalog(env, logger), + ); const diffResult = await getDiffBetweenRepos( project.remote, @@ -393,16 +533,30 @@ export async function createChangeWithEvaluation( baseOid, } = diffResult.data; - const evaluators = buildEvaluators(env, policy, projectName, logger, { - remote: workspaceRemote, - token: workspaceReadToken.data, - ref: evaluatedSha, - }); + const evaluators = await buildEvaluators( + env, + policy, + project, + logger, + { + remote: workspaceRemote, + token: workspaceReadToken.data, + ref: evaluatedSha, + }, + waitUntil, + ); // `baseOid`, not the `baseSha` resolved further up: that one is read before // the diff clone, so the default branch can advance in between. The evaluated // base must be the one the diff was actually built against (#274). + // Once per request, and before the evaluation rather than after it: the same + // subject gates the meters below and is recorded in the ledger, so resolving + // it twice would pay for two agent walks and could name two different payers. + const createSubject = await resolveBillingSubject(env.DB, logger, project); const { evalRuns, evalResult } = await runEvaluation(evaluators, diff, policy, logger, { baseSha: baseOid, + // The actor is the human behind the change — the agent's owner for an + // agent-authored one — which is who an allowance follows under PRD §4a. + billing: billingContextFor(createSubject, project.id, createdByUserId), }); const newStatus: Change["status"] = evalResult.passed ? "accepted" : "needs_changes"; @@ -423,7 +577,18 @@ export async function createChangeWithEvaluation( await recordCosts( env.DB, logger, - { project: projectName, projectId: project.id, changeId: change.id, workspace: workspaceName }, + { + project: projectName, + projectId: project.id, + changeId: change.id, + workspace: workspaceName, + ...(createSubject ?? {}), + notify: { + env, + ...(createdByUserId !== undefined ? { actorUserId: createdByUserId } : {}), + ...(waitUntil !== undefined ? { waitUntil } : {}), + }, + }, createCostSamples, ); diff --git a/src/storage/costs.ts b/src/storage/costs.ts index fc3f7ca1..76f5673c 100644 --- a/src/storage/costs.ts +++ b/src/storage/costs.ts @@ -1,15 +1,46 @@ +import { noticeUsageThresholds } from "../billing/usage-notifications"; +import type { Env } from "../types"; import { AppError } from "../utils/errors"; import { newId } from "../utils/ids"; import type { Logger } from "../utils/logger"; import { type Result, err, ok } from "../utils/result"; +import { getAgent } from "./agents"; +import { + type UsageDelta, + type UsageWriteTotal, + meterForCostKind, + upsertUsage, + usagePeriod, +} from "./usage"; export type CostKind = "llm_tokens" | "sandbox_ms" | "git_ops"; +/** + * Whose provider account paid for a sample: the operator's (`platform`) or the + * project's own credential (`byok`). Recorded per sample rather than per change + * because one evaluation can mix the two — a BYOK `llm` evaluator alongside + * git operations that are always the operator's cost. + */ +export type CostSource = "platform" | "byok"; + export interface CostSample { kind: CostKind; quantity: number; /** True when the quantity is an estimate (e.g. character-based token counts). */ estimated?: boolean; + /** Defaults to `"platform"`: an evaluator that says nothing spent our money. */ + source?: CostSource; +} + +/** + * An account a cost row can actually be billed to. + * + * Narrower than `ProjectEntry.ownerType`, which also admits `"agent"` — see + * `resolveBillingSubject`, which is the only thing allowed to make that mapping. + */ +export interface BillingSubject { + ownerId: string; + ownerType: "user" | "org"; } export interface CostSummaryEntry { @@ -24,25 +55,183 @@ interface SummaryRow { any_estimated: number; } +/** + * The account that owes for spend incurred on behalf of `owner`, or `null` when + * no account can be named. + * + * The single home for the `ownerType -> billing subject` mapping: every + * recording site resolves through here so the agent walk and the + * cannot-be-attributed rule exist once rather than at each of them. Accepts a whole + * `ProjectEntry` (structurally) or a bare owner pair. + * + * An **agent is not a payer** — it belongs to one — so `"agent"` walks + * `agents.owner_id` (migrations/001_core.sql) to the user that owns it. + * `billingContextFor` (src/services/change-flow.ts) deliberately does not do + * this: it is synchronous and KV-only, while this needs D1. No creation path + * writes `ownerType: "agent"` today, so that branch is defensive against + * restored backups and future ownership transfer, not a live code path. + * + * **Never throws.** `recordCosts` is best-effort by contract and runs inside + * change creation, merge and deploy; an unresolvable owner must cost the + * attribution, never the change. Every failure returns `null` and logs — a row + * with a NULL owner is honest about being unattributed, which is why migration + * 048 leaves those columns nullable. + * + * @param owner - A `ProjectEntry`, or any object carrying its owner fields + * @returns The billing subject, or `null` when none can be named + */ +export async function resolveBillingSubject( + db: D1Database, + logger: Logger, + // Spelled out rather than `ProjectEntry["ownerType"]` on purpose: if a fourth + // owner kind is ever added, this must fail to compile so someone decides who + // pays, instead of silently falling through to the agent walk. + owner: { ownerId?: string; ownerType?: "user" | "org" | "agent" }, +): Promise { + const { ownerId, ownerType } = owner; + // KV entries are cast without shape validation and legacy rows genuinely lack + // fields, so the type's promise is not a runtime guarantee. + if (!ownerId || !ownerType) { + logger.warn("Cost attribution skipped: project names no owner", { ownerId, ownerType }); + return null; + } + if (ownerType === "user" || ownerType === "org") return { ownerId, ownerType }; + + const agentResult = await getAgent(db, ownerId, logger); + if (!agentResult.success) { + logger.warn("Cost attribution skipped: agent owner could not be resolved to a user", { + agentId: ownerId, + error: agentResult.error.message, + }); + return null; + } + // No emptiness check on the resolved id, unlike the owner fields above: + // `agents.owner_id` is NOT NULL and REFERENCES users(id), and D1 enforces + // foreign keys, so a row that exists names a user that exists. + return { ownerId: agentResult.data.ownerId, ownerType: "user" }; +} + +/** + * Accumulate the metered half of a batch into `usage_periods`. + * + * Split out of `recordCosts` so every recording site gets it without any of + * them knowing it exists — the aggregate must never be something a call site + * can forget, because a site that forgets is an owner whose spend does not + * count against their allowance. + * + * Samples with no billing subject are skipped. `cost_records` keeps them with a + * NULL owner because the spend still happened, but a quantity nobody can be + * billed for cannot be enforced against anyone, so there is no row here to + * write it into — `usage_periods.owner_id` is NOT NULL for exactly that reason. + * + * The period comes from the same `createdAt` the ledger rows carry, so a cost + * row and the month it counts toward cannot land on opposite sides of a + * boundary. + */ +async function accumulateUsage( + db: D1Database, + logger: Logger, + opts: { ownerId?: string; ownerType?: BillingSubject["ownerType"] }, + samples: CostSample[], + createdAt: string, +): Promise<{ subject: BillingSubject; period: string; totals: UsageWriteTotal[] } | null> { + if (!opts.ownerId || !opts.ownerType) return null; + const deltas: UsageDelta[] = []; + for (const sample of samples) { + // `git_ops` maps to no meter; see `meterForCostKind`. + const meter = meterForCostKind(sample.kind); + if (meter) { + deltas.push({ meter, quantity: sample.quantity, source: sample.source ?? "platform" }); + } + } + if (deltas.length === 0) return null; + // The Result is deliberately not folded into `recordCosts`'s own: the ledger + // rows above ARE written, and reporting an error would tell the caller the + // spend went unrecorded when it did not. `upsertUsage` logs its own failure. + // + // Be clear about what that costs, because it is not free and there is no job + // that repairs it: this aggregate cannot be rebuilt from `cost_records` (the + // project cascade deletes those rows and leaves these standing — see + // migration 049), so a failed upsert is a permanent under-count of that + // owner's month. Accepted because the alternative is failing a merge over an + // accounting write, which is the worse trade in both directions. + const subject: BillingSubject = { ownerId: opts.ownerId, ownerType: opts.ownerType }; + const period = usagePeriod(new Date(createdAt)); + const written = await upsertUsage(db, logger, subject, period, deltas); + // The post-write totals travel back up so the caller can notice an 80% + // crossing without a second read; a failed write reports nothing to notice. + return written.success ? { subject, period, totals: written.data } : null; +} + /** * Record cost samples for a change. Best-effort: failures are logged and * reported, but callers treat cost recording as non-blocking. + * + * `ownerId`/`ownerType` name who pays, and come from `resolveBillingSubject` — + * omitting them records the spend unattributed rather than refusing to record + * it, because a sample nobody can be billed for is still a sample that happened. */ export async function recordCosts( db: D1Database, logger: Logger, - opts: { project: string; projectId?: string; changeId?: string; workspace?: string }, + opts: { + project: string; + projectId?: string; + changeId?: string; + workspace?: string; + ownerId?: string; + ownerType?: BillingSubject["ownerType"]; + /** + * Everything the threshold notification needs, when the call site can + * supply it. Absent means the write is recorded and nobody is told they are + * approaching a limit — which is the correct behaviour for a site with no + * `Env` in hand, and the reason this is optional rather than a second + * required argument. Inert unless the cloud billing service is configured. + */ + notify?: { + env: Env; + /** The acting user, for the §4a enforcement subject and the addressee. */ + actorUserId?: string; + /** Present on a request path; the queue consumers pass nothing. */ + waitUntil?: (promise: Promise) => void; + }; + }, samples: CostSample[], ): Promise> { if (samples.length === 0) return ok(undefined); + + // Dropped BEFORE the batch, not inside the aggregate that also guards them. + // `db.batch` is one transaction and `cost_records.quantity` is NOT NULL, + // while SQLite stores NaN as NULL — so a single unusable sample does not cost + // itself, it aborts the write and loses every other sample in the same + // evaluation. Filtering here keeps one bad number from taking the ledger with + // it, and is why `upsertUsage`'s own guard is a backstop rather than the + // first line of defence. + // + // Negative quantities are dropped for a different reason: they write fine. + // A row for negative spend is never right — it silently refunds an owner's + // month through `accumulateUsage` — and no meter this records can go + // backwards. The providers validate their own token counts (`usageOrNothing` + // in `evaluation/llm-provider.ts`); this is the defence that does not depend + // on every future sample source remembering to. + const usable = samples.filter( + (sample) => Number.isFinite(sample.quantity) && sample.quantity >= 0, + ); + if (usable.length !== samples.length) { + logger.warn("Cost samples dropped: quantity is not a finite, non-negative number", { + project: opts.project, + dropped: samples.length - usable.length, + }); + } + if (usable.length === 0) return ok(undefined); const createdAt = new Date().toISOString(); try { const stmt = db.prepare( - "INSERT INTO cost_records (id, project, project_id, change_id, workspace, kind, quantity, estimated, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + "INSERT INTO cost_records (id, project, project_id, change_id, workspace, kind, quantity, estimated, created_at, owner_id, owner_type, source) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", ); await db.batch( - samples.map((sample) => + usable.map((sample) => stmt.bind( newId("cost"), opts.project, @@ -53,6 +242,11 @@ export async function recordCosts( sample.quantity, sample.estimated ? 1 : 0, createdAt, + opts.ownerId ?? null, + opts.ownerType ?? null, + // Bound explicitly rather than left to the column DEFAULT: the column + // is NOT NULL, so an unset source must resolve here, not in SQLite. + sample.source ?? "platform", ), ), ); @@ -61,6 +255,18 @@ export async function recordCosts( changeId: opts.changeId, count: samples.length, }); + // Only after the ledger write succeeded: the aggregate derives from those + // rows, so it must never count spend the ledger does not record. + const accumulated = await accumulateUsage(db, logger, opts, usable, createdAt); + if (accumulated && opts.notify) { + noticeUsageThresholds(opts.notify.env, logger, { + recorded: accumulated.subject, + ...(opts.notify.actorUserId !== undefined ? { actorUserId: opts.notify.actorUserId } : {}), + period: accumulated.period, + totals: accumulated.totals, + ...(opts.notify.waitUntil !== undefined ? { waitUntil: opts.notify.waitUntil } : {}), + }); + } return ok(undefined); } catch (error) { const appError = @@ -110,37 +316,3 @@ export async function getChangeCostSummary( return err(appError); } } - -export async function getProjectCostSummary( - db: D1Database, - logger: Logger, - project: string, -): Promise> { - try { - const result = await db - .prepare( - "SELECT kind, SUM(quantity) AS total, MAX(estimated) AS any_estimated FROM cost_records WHERE project = ? GROUP BY kind", - ) - .bind(project) - .all(); - return ok( - result.results.map((row) => ({ - kind: row.kind as CostKind, - total: row.total, - estimated: row.any_estimated === 1, - })), - ); - } catch (error) { - const appError = - error instanceof AppError - ? error - : new AppError( - error instanceof Error ? error.message : "Failed to summarize costs", - "DATABASE_ERROR", - 500, - { operation: "getProjectCostSummary", project }, - ); - logger.error("Failed to get project cost summary", appError, { project }); - return err(appError); - } -} diff --git a/src/storage/d1-backup.ts b/src/storage/d1-backup.ts index 5b594151..71fadf6a 100644 --- a/src/storage/d1-backup.ts +++ b/src/storage/d1-backup.ts @@ -43,6 +43,13 @@ export const BACKUP_TABLES: readonly string[] = [ "import_metrics", "failed_imports", "cost_records", + // The owner-scoped usage aggregate (migration 049). Backed up rather than + // excluded, and NOT recoverable from `cost_records` above: the project + // cascade hard-deletes cost rows while this table survives, so replaying the + // ledger after a restore would rebuild only the usage of projects that still + // exist and hand everybody else their month back. Declares no foreign keys, + // so its position here is only to sit beside the ledger it derives from. + "usage_periods", "events", "sync_history", "issues", diff --git a/src/storage/deletion.ts b/src/storage/deletion.ts index 21536e0a..8dcc4247 100644 --- a/src/storage/deletion.ts +++ b/src/storage/deletion.ts @@ -1,3 +1,4 @@ +import { usageMeterName } from "../queue/usage-meter"; import type { Env, ProjectEntry, WorkspaceEntry } from "../types"; import { type AppError, toAppError } from "../utils/errors"; import type { Logger } from "../utils/logger"; @@ -382,9 +383,17 @@ async function deleteArtifactsRepo(env: Env, name: string, logger: Logger): Prom return false; } -/** Purge a Durable Object's storage via its `purge()` RPC; false on failure. */ -async function purgeDurableObject( - namespace: DurableObjectNamespace | undefined, +/** + * Purge a Durable Object's storage via its `purge()` RPC; false on failure. + * + * Generic over the namespace's class because the bindings differ: MERGE_QUEUE + * and REPO_DO are untyped namespaces while USAGE_METER is parameterised by its + * class (`src/types.ts`), and the two are not assignable to one another. The + * `purge()` call is asserted below either way, since a typed stub is not enough + * to prove the method exists on an untyped one. + */ +async function purgeDurableObject( + namespace: DurableObjectNamespace | undefined, key: string, label: string, logger: Logger, @@ -497,6 +506,13 @@ export async function deleteProjectCascade( residuals.push(`do:MergeQueue:${key}`); } } + // USAGE_METER is deliberately absent from this cascade, and its absence is + // the point of the aggregate it counts against: the meter is keyed on an + // ENFORCEMENT subject (a user, or a paid org) and never on a project, so a + // project deletion must not touch it. Purging it here would refund the + // month by deleting a project — exactly what keeping `usage_periods` out of + // PROJECT_SCOPED_TABLES prevents (migration 049). The ACCOUNT cascade + // erases it, and nothing else does. // 5) KV last, project entry very last: as long as the entry exists, the // project is still resolvable and a re-drive can recapture everything. @@ -673,6 +689,24 @@ const IDENTITY_COLUMNS: readonly [table: string, column: string][] = [ // agent's id under requested_by_type = 'agent' and is left alone. ["deployments", "requested_by_id"], ["deployments", "approved_by"], + // Cost rows outlive an erasure by two routes: the project is org-owned, or it + // is agent-owned and so never cascaded at all (only `ownerType === "user"` + // projects are). In the agent case the row was stamped with the *user* id by + // `resolveBillingSubject`'s agent walk, and step 3 of the cascade then deletes + // the agent — destroying the only link that could re-derive the attribution + // and leaving a bare, unresolvable account id behind. `cost_records` has no + // foreign key to `users`, so nothing else catches it. Only `owner_id` is + // rewritten: `quantity` and `kind` are what the spend was, not who incurred + // it, and the row must stay readable as spend that happened. + ["cost_records", "owner_id"], + // `usage_periods.owner_id` is deliberately NOT here, and its absence is a + // decision rather than an oversight. Rewriting it to the shared sentinel + // would collide two erased users' rows on PRIMARY KEY (owner_id, period, + // meter) and throw mid-erasure; and there is nothing worth preserving. A cost + // row is evidence that spend happened and reads perfectly well with the payer + // anonymized, whereas an aggregate exists only to be compared against a limit + // for a subject that no longer exists. `deleteAccountCascade` below deletes + // those rows outright instead. ]; /** @@ -775,6 +809,32 @@ async function resolveOrgOwnership( .bind(org.id) .run(); await db.prepare("DELETE FROM teams WHERE org_id = ?").bind(org.id).run(); + // The org's usage aggregate goes with the org, and ONLY on this branch: where + // a successor was promoted above, the org survives and so must its + // consumption — a change of owner is not a fresh monthly allowance. + await db.prepare("DELETE FROM usage_periods WHERE owner_id = ?").bind(org.id).run(); + // ...and so does the org's enforcement counter, on this branch alone. The DO + // is named `org:`, never a bare id, so deleting one org can only reach + // that org's own counter: a person who spent inside it was charged against + // `user:` (PRD §4a — the enforcement subject is the actor unless the org + // is positively known to be paid), which is a different object this cannot + // name. Purging a shared counter here would erase a stranger's allowance. + const meterName = usageMeterName("org", org.id); + if (!(await purgeDurableObject(env.USAGE_METER, meterName, "UsageMeter", logger))) { + // The orgs row is what a re-drive finds this org by (step 4 selects on + // `owner_id`), so dropping it now would strand the counter permanently: + // nothing would ever name `org:` again, and the residual would name a + // purge no run can retry. Retained instead, exactly as the users row is — + // the org has no members and no projects left, and the residual keeps the + // erasure job out of `completed` until the purge lands. + residuals.push(`do:UsageMeter:${meterName}`); + residuals.push(`org:${org.id}:row-retained-pending-residuals`); + logger.warn("Org meter purge failed; retaining the org row for re-drive", { + orgId: org.id, + meterName, + }); + return; + } await db.prepare("DELETE FROM orgs WHERE id = ?").bind(org.id).run(); } @@ -849,6 +909,23 @@ export async function deleteAccountCascade( // erasure request is itself a retention bug. await db.prepare("DELETE FROM agents WHERE owner_id = ?").bind(userId).run(); await db.prepare("DELETE FROM api_tokens WHERE user_id = ?").bind(userId).run(); + // The owner-scoped usage aggregate (migration 049). Erasing the account is + // the ONLY event that may clear it: `usage_periods` is deliberately outside + // `PROJECT_SCOPED_TABLES` because an allowance a user can reset by deleting + // the project that burned it is not an allowance. Here the subject itself is + // going, so there is nothing left to bill and nothing left to enforce. + await db.prepare("DELETE FROM usage_periods WHERE owner_id = ?").bind(userId).run(); + // The same subject's live counters. The DO is keyed on the ENFORCEMENT + // subject (PRD §4a), which for a person is the person: everything this user + // ran — including inside orgs they are merely a member of — was checked + // against `user:`, so erasing the account erases all of it while those + // orgs keep their own rows and their own counters. A failure becomes a + // residual, which retains the users row for a re-drive rather than + // stranding a counter under a deleted account. + const userMeterName = usageMeterName("user", userId); + if (!(await purgeDurableObject(env.USAGE_METER, userMeterName, "UsageMeter", logger))) { + residuals.push(`do:UsageMeter:${userMeterName}`); + } // Same reasoning, same trap: `oauth_tokens.user_id` and // `oauth_auth_codes.user_id` both REFERENCE users(id) (#349). An OAuth // grant is a live credential handed to an editor, so an erasure that left diff --git a/src/storage/deployments.ts b/src/storage/deployments.ts index 615fb1b5..e4b1a7d5 100644 --- a/src/storage/deployments.ts +++ b/src/storage/deployments.ts @@ -618,8 +618,16 @@ export async function claimDeployment( * `lease_expires_at` is always cleared: a finished deployment holding a lease * would look reclaimable to the next consumer that read it. * + * `onlyIfUnclaimed` narrows that to rows nobody is running. The paths that + * write a terminal status WITHOUT claiming first — a refused guard, an + * exhausted allowance, a superseded row — run before `claimDeployment`, so + * another delivery of the same message may already hold the lease and be + * mid-deploy; `running` is not terminal, so without this they would overwrite + * its row and clear its lease, and the deploy would keep going with nothing + * left to record it. Callers that own the row through a claim must NOT set it. + * * @returns `true` when a row was written, `false` when the row was missing, - * in another project, or already terminal. + * in another project, already terminal, or (with `onlyIfUnclaimed`) running. */ export async function completeDeployment( db: D1Database, @@ -633,6 +641,8 @@ export async function completeDeployment( logTail?: string | null; durationMs?: number | null; completedAt?: string; + /** Refuse a row another delivery has claimed and is running. */ + onlyIfUnclaimed?: boolean; }, ): Promise> { const completedAtResult = resolveTimestamp(opts.completedAt, "completedAt"); @@ -640,11 +650,12 @@ export async function completeDeployment( const completedAt = completedAtResult.data; const terminalList = TERMINAL_DEPLOYMENT_STATUSES.map(() => "?").join(", "); + const unclaimed = opts.onlyIfUnclaimed ? " AND status != 'running'" : ""; try { const result = await db .prepare( - `UPDATE deployments SET status = ?, reason = ?, url = ?, log_tail = ?, duration_ms = ?, completed_at = ?, lease_expires_at = NULL WHERE id = ? AND project_id = ? AND status NOT IN (${terminalList})`, + `UPDATE deployments SET status = ?, reason = ?, url = ?, log_tail = ?, duration_ms = ?, completed_at = ?, lease_expires_at = NULL WHERE id = ? AND project_id = ? AND status NOT IN (${terminalList})${unclaimed}`, ) .bind( opts.status, diff --git a/src/storage/project-secrets.ts b/src/storage/project-secrets.ts index f9df1f56..616c9333 100644 --- a/src/storage/project-secrets.ts +++ b/src/storage/project-secrets.ts @@ -1,9 +1,16 @@ /** - * Per-project encrypted secret store, backing deploy provider credentials. + * Per-project encrypted secret store, backing deploy provider credentials and + * the per-project LLM provider keys BYOK runs on. * - * Write-only by construction: the only function that yields a plaintext value is - * {@link loadSecretValues}, which exists for the deploy runner and is deliberately - * not reachable from any route. Everything else returns names and metadata. + * Write-only by construction *to a caller*: the only function that yields a + * plaintext value is {@link loadSecretValues}, and no route returns, renders or + * logs what it resolves. It is no longer unreachable from a route, though — + * BYOK resolves a provider credential on the change-creation path, so the + * routes that create or re-evaluate a change (`routes/changes.ts`, + * `routes/git-http.ts`, `routes/sync-management.ts`) reach it through + * `evaluation/llm-byok.ts`. What keeps that safe is the rule below, not the + * call graph: the value it returns goes into one outbound request header and + * nowhere else. Everything else here returns names and metadata. */ import type { Env } from "../types"; import { type SecretScope, decryptSecret, deriveSecretKey, encryptSecret } from "../utils/crypto"; @@ -101,7 +108,43 @@ function validateName(name: string): ValidationError | null { ); } +/** + * Characters no credential contains and that a header cannot carry: C0, DEL, + * and NUL among them. + * + * Rejected at the WRITE, which is the only place the value is ever attacker- + * chosen. A stored value ends up in an outbound header, and `new Headers()` + * throws a `TypeError` whose message QUOTES the offending value — so a CR, LF + * or NUL in a stored secret turns into an error message carrying the secret, + * which then becomes an evaluation reason persisted on the change and rendered + * on a page that is world-readable for a public project. The provider closes + * the other end (`evaluation/llm-provider.ts` builds its headers outside the + * try and maps a failure to a constant), so neither end alone is load-bearing. + */ +/** + * Does `value` contain a C0 control character or DEL? + * + * A char-code scan rather than a regex: matching control characters is the + * whole point here, which Biome's `noControlCharactersInRegex` flags, and a + * suppression for a rule this deliberately trips reads worse than the loop. + */ +function hasControlCharacter(value: string): boolean { + for (let i = 0; i < value.length; i++) { + const code = value.charCodeAt(i); + if (code <= 0x1f || code === 0x7f) return true; + } + return false; +} + function validateValue(value: string): ValidationError | null { + if (hasControlCharacter(value)) { + // Deliberately says nothing about the value — not its length, not the + // offending character, and above all not the value itself: this message is + // rendered back to the caller. + return new ValidationError( + "Secret value must not contain control characters (including CR, LF, tab and NUL)", + ); + } const bytes = new TextEncoder().encode(value).length; if (bytes <= MAX_SECRET_VALUE_BYTES) return null; return new ValidationError(`Secret value exceeds ${MAX_SECRET_VALUE_BYTES} bytes`, { @@ -127,12 +170,29 @@ async function resolveKey(env: SecretKeyEnv): Promise } | null = null; + /** * Creates or replaces a project's secret, returning only its metadata. * @@ -278,8 +338,16 @@ export async function deleteSecret( } /** - * Resolves plaintext secret values for the deploy runner. **The only read path - * for a value anywhere in the codebase** — do not call it from a route. + * Resolves plaintext secret values. **The only read path for a value anywhere + * in the codebase**, and the rule for a caller is that the plaintext goes + * straight into the one outbound request that needs it and is never returned, + * rendered, logged, or interpolated into an error. + * + * Two callers today: the deploy runner (`deploy/runner.ts`), and BYOK provider + * resolution (`evaluation/llm-byok.ts`), which the change-creation and + * re-evaluation routes reach synchronously. The comment that used to stand here + * said no route may call it; that stopped being true when BYOK landed, and a + * rule nobody can follow is worse than the honest one above. * * Derives the AES key once for the whole batch: PBKDF2 at 100k iterations is CPU * work and the deploy consumer runs under a CPU limit, so per-secret derivation diff --git a/src/storage/usage.ts b/src/storage/usage.ts new file mode 100644 index 00000000..14c94c7f --- /dev/null +++ b/src/storage/usage.ts @@ -0,0 +1,363 @@ +import { AppError } from "../utils/errors"; +import type { Logger } from "../utils/logger"; +import { type Result, err, ok } from "../utils/result"; +import type { BillingSubject, CostKind, CostSource } from "./costs"; + +/** + * A metered quantity an allowance can be set against, over a calendar month. + * + * The `_month` suffix is redundant with the row's `period` column and is kept + * anyway: these are the strings an entitlement's limits are keyed on, so a + * limit lookup is `limits[row.meter]` rather than a second mapping table that + * could disagree with this one. + * + * `deploys_month` has no writer yet, deliberately — see `meterForCostKind`. + */ +export type MeterKey = "llm_tokens_month" | "sandbox_ms_month" | "deploys_month"; + +/** + * Every {@link MeterKey}, as values. The single list: `entitlements.ts` + * re-exports it so a limit lookup and a meter total cannot disagree about which + * meters exist. + */ +export const METER_KEYS: readonly MeterKey[] = [ + "llm_tokens_month", + "sandbox_ms_month", + "deploys_month", +]; + +/** + * Whether a string read out of D1 (or out of KV, or off the wire) is a meter + * this build knows. + * + * The `meter` column carries no CHECK on purpose (migration 049), so the type + * is a claim about what we write, never a guarantee about what we read. A cast + * where that matters is how a row written by a future meter — or by a rollback + * to an older build — reaches `setFloor` and gets written into a counter under + * a key nothing can ever read back. + */ +export function isMeterKey(value: unknown): value is MeterKey { + return typeof value === "string" && (METER_KEYS as readonly string[]).includes(value); +} + +/** One meter's increment. Quantities are added, never replaced. */ +export interface UsageDelta { + meter: MeterKey; + quantity: number; + /** + * Whose provider account paid. Omitted means `"platform"` — the operator's, + * and so the only kind an allowance limits. Kept as a separate total rather + * than summed in, because a limit applies to one of the two and not the + * other; see migration 049. + */ + source?: CostSource; +} + +/** + * One meter's total after an {@link upsertUsage} write, as the UPSERT itself + * reported it. + * + * Returned rather than re-read because the 80% threshold notification is + * edge-triggered (PRD §8): it fires on the CROSSING, which needs the totals + * either side of this write. A second `SELECT` per evaluated change would cost + * a D1 round trip on the change-creation path and still race a concurrent + * recorder — `RETURNING` reports the value the statement that did the adding + * actually wrote. + */ +export interface UsageWriteTotal { + meter: MeterKey; + source: CostSource; + /** The meter's total for this owner and period AFTER this write. */ + quantity: number; + /** What this write added, so the caller can reconstruct the total before it. */ + added: number; +} + +export interface UsageSummaryEntry { + meter: MeterKey; + source: CostSource; + quantity: number; + /** ISO 8601, app-written. When this owner last consumed this meter. */ + updatedAt: string; +} + +interface UsageRow { + meter: string; + source: string; + quantity: number; + updated_at: string; +} + +/** + * The meter a cost sample accumulates into, or `null` when it accumulates into + * none. + * + * `CostKind` and `MeterKey` are not the same vocabulary and are not meant to + * be. A cost kind names a *resource consumed by one change*; a meter names a + * *thing an allowance is set on*. Two align exactly: + * + * - `llm_tokens` -> `llm_tokens_month` + * - `sandbox_ms` -> `sandbox_ms_month` + * + * `git_ops` maps to nothing. It is the count of clones and pushes an operation + * performed — several per change — and there is no `git_ops_month` allowance + * in the PRD's meter set. The tempting misreading is to route it to + * `deploys_month`, which would be wrong in both directions: `post-merge.ts` + * records `git_ops` for a merge that deploys nothing, and `deploy/runner.ts` + * records one for a config read that may precede a deploy that never happens. + * `deploys_month` counts `deployments` rows, so its writer is the deploy + * consumer and not this function; the key exists in `MeterKey` now only so + * that adding that writer needs no migration. + */ +export function meterForCostKind(kind: CostKind): MeterKey | null { + switch (kind) { + case "llm_tokens": + return "llm_tokens_month"; + case "sandbox_ms": + return "sandbox_ms_month"; + case "git_ops": + return null; + } +} + +/** + * The 'YYYY-MM' UTC period a moment falls in (PRD Open Question 1). + * + * Derived from the ISO string rather than `getMonth()` so it is UTC by + * construction: a local-time month would put a spend just after midnight UTC + * into the previous period for half the world, and two counters keyed on + * different months are two allowances. + * + * @param at - The moment to classify; defaults to now + * @returns The calendar month in 'YYYY-MM' form, UTC + */ +export function usagePeriod(at: Date = new Date()): string { + return at.toISOString().slice(0, 7); +} + +/** + * Add to an owner's meters for one period, creating the rows if they do not + * exist yet. + * + * Each meter is a single SQLite UPSERT, which is where the correctness lives: + * `ON CONFLICT ... DO UPDATE SET quantity = quantity + excluded.quantity` reads + * and writes inside one statement, so concurrent recorders sum. A + * read-then-write would let N of them all read the same total and each write + * back "+1", losing N-1 increments — precisely the race + * `MagicLinkRateLimiter`'s doc comment describes for KV counters. + * + * Deliberately not `db.batch`: the meters are independent, so cross-meter + * atomicity buys nothing, and one statement per meter keeps the atomicity claim + * at the statement where it is actually true. However many deltas arrive, only + * as many statements run as there are distinct (meter, source) pairs. + * + * Best-effort by contract, like `recordCosts` which drives it: this returns a + * Result and logs, and never throws into change creation, merge or deploy. + * + * @param subject - The account the usage is billed to + * @param period - 'YYYY-MM' UTC, from `usagePeriod` + * @param deltas - Increments to apply; same-meter entries are summed first + * @returns The post-write total for each (meter, source) touched, or the + * database error that stopped the accumulation + */ +export async function upsertUsage( + db: D1Database, + logger: Logger, + subject: BillingSubject, + period: string, + deltas: readonly UsageDelta[], +): Promise> { + // Keyed on both dimensions the row is keyed on, so platform and BYOK spend + // for one meter accumulate as two totals rather than one. + const merged = new Map(); + for (const delta of deltas) { + // A NaN or Infinity would poison the running total permanently: SQLite has + // no way back from it and every later comparison against a limit would be + // false. A NEGATIVE would do something worse than poison it — it would + // refund the month, through the single statement that exists to stop the + // month being refundable. Wall-clock arithmetic makes that reachable + // without malice (`Date.now() - startedAt` under a backwards clock step). + // Drop either and keep the meter usable; migration 049's CHECK is the + // backstop if one ever gets past here. + if (!Number.isFinite(delta.quantity) || delta.quantity < 0) { + logger.warn("Usage delta skipped: quantity is not a usable increment", { + ownerId: subject.ownerId, + meter: delta.meter, + quantity: delta.quantity, + }); + continue; + } + const source: CostSource = delta.source ?? "platform"; + const key = `${delta.meter}\u0000${source}`; + const existing = merged.get(key); + merged.set(key, { + meter: delta.meter, + source, + quantity: (existing?.quantity ?? 0) + delta.quantity, + }); + } + if (merged.size === 0) return ok([]); + + const updatedAt = new Date().toISOString(); + try { + const totals: UsageWriteTotal[] = []; + for (const { meter, source, quantity } of merged.values()) { + // `RETURNING quantity` on the same statement that does the adding: the + // post-write total is what makes the threshold crossing detectable + // without a second read, and reading it back separately would race + // another recorder and report a total this write did not produce. + const row = await db + .prepare( + "INSERT INTO usage_periods (owner_id, owner_type, period, meter, source, quantity, updated_at) " + + "VALUES (?, ?, ?, ?, ?, ?, ?) " + + "ON CONFLICT(owner_id, period, meter, source) DO UPDATE SET " + + "quantity = quantity + excluded.quantity, updated_at = excluded.updated_at " + + "RETURNING quantity", + ) + .bind(subject.ownerId, subject.ownerType, period, meter, source, quantity, updatedAt) + .first<{ quantity: number }>(); + // A driver that does not hand RETURNING back is not an error: the row is + // written either way, and the only thing lost is the notification edge. + if (row && Number.isFinite(row.quantity)) { + totals.push({ meter, source, quantity: row.quantity, added: quantity }); + } + } + logger.debug("Usage aggregate updated", { + ownerId: subject.ownerId, + period, + meters: merged.size, + }); + return ok(totals); + } catch (error) { + const appError = + error instanceof AppError + ? error + : new AppError( + error instanceof Error ? error.message : "Failed to accumulate usage", + "DATABASE_ERROR", + 500, + { operation: "upsertUsage", ownerId: subject.ownerId, period }, + ); + logger.error("Failed to accumulate usage", appError, { ownerId: subject.ownerId, period }); + return err(appError); + } +} + +/** + * What one account consumed in one period, one entry per meter it touched. + * + * The period is a required argument rather than defaulted to the current month + * for the reason the PRD gives for `UsageMeter`'s `nowMs`: the caller decides + * which window it is asking about, so one request cannot straddle a boundary + * and read half of each. Meters with no rows are absent rather than zero — + * "never used" and "used nothing" are the same fact, and the caller knows the + * meter set it wants to display. + * + * @param ownerId - The billing subject's id + * @param period - 'YYYY-MM' UTC, from `usagePeriod` + * @returns The per-meter totals, or the database error + */ +export async function getOwnerUsageSummary( + db: D1Database, + logger: Logger, + ownerId: string, + period: string, +): Promise> { + try { + const result = await db + .prepare( + "SELECT meter, source, quantity, updated_at FROM usage_periods WHERE owner_id = ? AND period = ? ORDER BY meter, source", + ) + .bind(ownerId, period) + .all(); + return ok( + result.results.map((row) => ({ + // Cast, not validated: the column carries no CHECK on purpose (see + // migration 049), so a row written by a future meter this build does + // not know about reads through rather than failing the whole summary. + meter: row.meter as MeterKey, + source: row.source as CostSource, + quantity: row.quantity, + updatedAt: row.updated_at, + })), + ); + } catch (error) { + const appError = + error instanceof AppError + ? error + : new AppError( + error instanceof Error ? error.message : "Failed to summarize usage", + "DATABASE_ERROR", + 500, + { operation: "getOwnerUsageSummary", ownerId, period }, + ); + logger.error("Failed to get owner usage summary", appError, { ownerId, period }); + return err(appError); + } +} + +/** + * One owner's totals for a period, **filtered to one source**, as a lookup by + * meter. + * + * Separate from {@link getOwnerUsageSummary}, which reports every source so a + * usage page can show what BYOK covered, because an enforcement read must not. + * Migration 049 put `source` in the primary key precisely so hosted and + * bring-your-own spend accumulate apart; a limit check that summed both would + * charge a project's own provider bill against the operator's allowance and + * contradict Goal 5. Callers checking a limit pass `"platform"`, always. + * + * Meters with no rows are absent rather than zero, as in the summary: "never + * used" and "used nothing" are one fact, and the caller knows its meter set. + * + * @param ownerId - The subject whose rows to sum — the ENFORCEMENT subject for + * a limit check (PRD §4a), which is not always the recorded billing subject + * @param period - 'YYYY-MM' UTC, from `usagePeriod` + * @param source - Which provider account's spend to count + * @returns Totals by meter, or the database error + */ +export async function getOwnerMeterTotals( + db: D1Database, + logger: Logger, + ownerId: string, + period: string, + source: CostSource, +): Promise>, AppError>> { + try { + const result = await db + .prepare( + "SELECT meter, SUM(quantity) AS quantity FROM usage_periods " + + "WHERE owner_id = ? AND period = ? AND source = ? GROUP BY meter", + ) + .bind(ownerId, period, source) + .all<{ meter: string; quantity: number }>(); + const totals: Partial> = {}; + for (const row of result.results) { + // Validated, not cast: this total is what `reconcileFloor` writes into the + // usage meter, and an unrecognised meter string would be stored under a + // key no limit lookup can reach — counted forever, checked never. + if (!isMeterKey(row.meter)) { + logger.warn("Usage total skipped: unknown meter in usage_periods", { + ownerId, + period, + meter: String(row.meter), + }); + continue; + } + if (Number.isFinite(row.quantity)) totals[row.meter] = row.quantity; + } + return ok(totals); + } catch (error) { + const appError = + error instanceof AppError + ? error + : new AppError( + error instanceof Error ? error.message : "Failed to total usage", + "DATABASE_ERROR", + 500, + { operation: "getOwnerMeterTotals", ownerId, period }, + ); + logger.error("Failed to total owner usage", appError, { ownerId, period, source }); + return err(appError); + } +} diff --git a/src/types.ts b/src/types.ts index e3f4c102..9a1e811a 100644 --- a/src/types.ts +++ b/src/types.ts @@ -2,6 +2,7 @@ // runner's message union cannot create a runtime import cycle. import type { DeployQueueMessage } from "./deploy/runner"; import type { MagicLinkRateLimiter } from "./queue/magic-link-limiter"; +import type { UsageMeter } from "./queue/usage-meter"; import type { LoggerContext } from "./utils/logger"; export type { LoggerContext }; @@ -72,7 +73,23 @@ export interface AiBinding { messages?: Array<{ role: string; content: string }>; prompt?: string; }, - ): Promise<{ response?: string } | ReadableStream>; + ): Promise< + | { + response?: string; + /** + * Token counts Workers AI actually reported. Declared because it is + * genuinely there and this repo was throwing it away: Cloudflare's + * generated types give a text-generation output as `response`, + * `tool_calls` and `usage`, with the counts named `prompt_tokens` / + * `completion_tokens` (`AiTextGenerationOutput` and `UsageTags` in + * `@cloudflare/workers-types`). Optional field by field, unlike those + * types, because a binding that omits them must be a cost sample marked + * `estimated`, not a type error — `usageOrNothing` decides which. + */ + usage?: { prompt_tokens?: number; completion_tokens?: number }; + } + | ReadableStream + >; } export interface SandboxBinding { @@ -154,6 +171,28 @@ export interface Env { BETA_GATE?: string; REFERRAL_SERVICE_URL?: string; REFERRAL_SERVICE_SECRET?: string; + /** Plan/entitlements service (Stratum Cloud only; OSS self-hosters leave these + * unset -> every owner resolves to `UnlimitedEntitlements` and nothing is + * fetched). BOTH are required for the hook to be live — see + * `entitlementsEnabled` in src/billing/entitlements.ts. */ + BILLING_SERVICE_URL?: string; + BILLING_SERVICE_SECRET?: string; + /** "1" makes entitlement decisions REFUSE rather than only record. Off by + * default: the limits are measured for a period before they gate anything. + * Setting it without BILLING_SERVICE_URL is enforcement that looks on and does + * nothing — `entitlementsConfigError` flags exactly that. */ + ENTITLEMENTS_ENFORCE?: string; + /** JSON array of the operator's LLM providers, e.g. + * `[{"name":"anthropic","kind":"anthropic","baseUrl":"https://api.anthropic.com/v1"}]`. + * The ONLY place a provider endpoint may be named: a project's policy file + * selects one of these by name and can never supply a `baseUrl`. Unset (the + * default) means Workers AI only. Credentials are NOT here — they live per + * project in `project_secrets`. Parsed by `parseLlmProviders`, which rejects + * a non-https, credential-bearing, loopback, link-local or private-range + * `baseUrl` at parse time and stores it normalized. Include the API version + * path: the provider appends `/messages` or `/chat/completions` to whatever + * this names, so a `baseUrl` missing the `/v1` 404s on every call. */ + LLM_PROVIDERS?: string; ANALYTICS?: AnalyticsEngineDataset; SANDBOX?: SandboxBinding; AI?: AiBinding; @@ -168,6 +207,19 @@ export interface Env { * is type-only, so the cycle with magic-link-limiter.ts (which imports `Env` * from here) is erased. */ MAGIC_LINK_LIMITER?: DurableObjectNamespace; + /** Serialized usage counters, one instance per ENFORCEMENT subject (PRD §4a: + * the acting user, or an org positively known to be paid — NOT the recorded + * billing subject, which is resettable by creating an org). Optional only so + * tests and minimal deployments can omit it; every deploy from this repo's + * wrangler.toml binds it, and nothing here enforces anything until the + * enforcement call sites exist. + * + * Parameterised by the class for the same reason as the limiter above: the + * `.get()` stub is typed, so a change to `reserve`/`settle`/`setFloor` is a + * compile error at the call site rather than something a cast absorbs. The + * import is type-only, so the cycle with usage-meter.ts (which imports `Env` + * from here) is erased. */ + USAGE_METER?: DurableObjectNamespace; /** Content-addressed git object plane (ADR 004 Phase 2). */ REPO_OBJECTS?: R2Bucket; /** Durable backup store (D1 dumps, KV identity, repo packs). Optional: when @@ -329,6 +381,25 @@ export function projectDefaultBranch(project: { return project.sourceDefaultBranch || project.githubDefaultBranch || "main"; } +/** + * A project's `@namespace/slug` label, falling back to its bare name. + * + * The fallback is the point. A legacy entry can carry no namespace — KV records + * are cast without shape validation, and this file's own `projectDefaultBranch` + * exists for the same class of drift — so interpolating the pair unguarded + * yields the string `"undefined/my-repo"`. That is merely ugly in a log line + * and actively wrong in `cost_records.project`, where it becomes a persisted + * identifier that matches no project during the deletion cascade's name-form + * lookup. + */ +export function projectDisplayName(project: { + namespace?: string; + slug?: string; + name: string; +}): string { + return project.namespace && project.slug ? `${project.namespace}/${project.slug}` : project.name; +} + export interface WorkspaceEntry { name: string; remote: string; diff --git a/src/ui/layout.tsx b/src/ui/layout.tsx index bc9e30bd..981e8ca6 100644 --- a/src/ui/layout.tsx +++ b/src/ui/layout.tsx @@ -1,4 +1,6 @@ import type { FC } from "hono/jsx"; +import type { UsageBannerNotice } from "../billing/usage-banner"; +import { meterTitle } from "../billing/usage-report"; import { SourceFooter } from "./components/source-footer"; /** @@ -21,11 +23,58 @@ interface LayoutProps { * see the URL, and a page under /new/import is still "new" to the reader. */ active?: NavItem; + /** + * The 80% usage warning, when this account has crossed one (PRD §8). + * + * A prop rather than something the layout fetches: the layout is a pure + * component with no request context, and the route that renders a page is + * the only place that can decide whether one KV read is worth taking. Pages + * that do not thread it simply do not warn. + */ + usageNotice?: UsageBannerNotice | null | undefined; children?: unknown; } +/** + * "You are near a limit", shown where a user is already looking. + * + * A page nobody visits warns nobody, which is why this lives in the shared + * chrome and not on `/settings/usage` alone. It has no dismiss control on + * purpose: dismissal is state, state without script means a cookie or a POST, + * and the notice is true until the period rolls over anyway. + * + * The last sentence turns on whether limits actually BIND + * (`enforcementBinding`), not on whether billing is configured. Through the + * observe-only month the instance has allowances and refuses nothing, and a + * banner promising a refusal that cannot happen teaches people to disbelieve + * the one that eventually does. + */ +const UsageBanner: FC<{ notice: UsageBannerNotice }> = ({ notice }) => ( + // `` rather than a div with role="status": it is the semantic element + // for exactly this, which is what Biome's useSemanticElements asks for, and it + // carries the same implicit ARIA role to a screen reader. + + + You have used {notice.percent}% of this account's monthly{" "} + {meterTitle(notice.meter).toLowerCase()} allowance for {notice.period} ( + {notice.used.toLocaleString("en-US")} of {notice.limit.toLocaleString("en-US")}).{" "} + {notice.enforcing + ? "Work that needs it is refused once it runs out." + : "Nothing is refused yet — this instance is measuring usage, not enforcing it."} + + View usage + +); + /** The page chrome shared by every server-rendered page: header nav, main column, footer, and the CSP-nonced scripts. */ -export const Layout: FC = ({ title, user, refreshSeconds, active, children }) => { +export const Layout: FC = ({ + title, + user, + refreshSeconds, + active, + usageNotice, + children, +}) => { const current = (item: NavItem) => (active === item ? "page" : undefined); return ( @@ -106,7 +155,10 @@ export const Layout: FC = ({ title, user, refreshSeconds, active, c )} -
{children}
+
+ {usageNotice && } + {children} +
diff --git a/src/ui/pages/change-detail.tsx b/src/ui/pages/change-detail.tsx index 886d4df3..b5466151 100644 --- a/src/ui/pages/change-detail.tsx +++ b/src/ui/pages/change-detail.tsx @@ -1,4 +1,5 @@ import type { FC } from "hono/jsx"; +import type { UsageBannerNotice } from "../../billing/usage-banner"; import type { ChangeComment, ChangeReview } from "../../storage/change-reviews"; import type { CostSummaryEntry } from "../../storage/costs"; import { type DiffFile, DiffView, LineCommentThreads } from "../components/diff-view"; @@ -44,6 +45,8 @@ interface ChangeDetailProps { /** Namespaced project ref for the header; legacy changes may lack one. */ projectRef?: { name: string; namespace: string; slug: string; visibility?: string } | null; user?: { id: string; email: string; username: string } | null; + /** The 80% warning, when this account has one. See `Layout`. */ + usageNotice?: UsageBannerNotice | null; } function statusBadgeClass(status: string): string { @@ -101,6 +104,7 @@ export const ChangeDetailPage: FC = ({ canReview = false, projectRef = null, user, + usageNotice, }) => { // Line-anchored comments (and their replies) render as threads beneath the // diff; anchorless comments stay in the change-level discussion below. @@ -116,6 +120,7 @@ export const ChangeDetailPage: FC = ({ {projectRef && } diff --git a/src/ui/pages/changes.tsx b/src/ui/pages/changes.tsx index 20f1ee1d..7cf39db1 100644 --- a/src/ui/pages/changes.tsx +++ b/src/ui/pages/changes.tsx @@ -1,4 +1,5 @@ import type { FC } from "hono/jsx"; +import type { UsageBannerNotice } from "../../billing/usage-banner"; import { ProjectHeader } from "../components/project-header"; import { Layout } from "../layout"; @@ -19,6 +20,8 @@ interface ChangesProps { createdAt: string; }>; user?: { id: string; email: string; username: string } | null; + /** The 80% warning, when this account has one. See `Layout`. */ + usageNotice?: UsageBannerNotice | null; } function statusBadgeClass(status: string): string { @@ -40,9 +43,15 @@ function statusBadgeClass(status: string): string { } } -export const ChangesPage: FC = ({ project, changes, canWrite, user }) => { +export const ChangesPage: FC = ({ + project, + changes, + canWrite, + user, + usageNotice, +}) => { return ( - + {changes.length === 0 ? ( diff --git a/src/ui/pages/home.tsx b/src/ui/pages/home.tsx index b7f72737..97c6964f 100644 --- a/src/ui/pages/home.tsx +++ b/src/ui/pages/home.tsx @@ -1,4 +1,5 @@ import type { FC } from "hono/jsx"; +import type { UsageBannerNotice } from "../../billing/usage-banner"; import { Layout } from "../layout"; interface HomeProps { @@ -11,11 +12,13 @@ interface HomeProps { visibility?: string; }>; user?: { id: string; email: string; username: string } | null; + /** The 80% warning, when this account has one. See `Layout`. */ + usageNotice?: UsageBannerNotice | null; } -export const HomePage: FC = ({ projects, user }) => { +export const HomePage: FC = ({ projects, user, usageNotice }) => { return ( - +

Dashboard

{user && ( diff --git a/src/ui/pages/settings.tsx b/src/ui/pages/settings.tsx index 9867ac4e..db6f5baa 100644 --- a/src/ui/pages/settings.tsx +++ b/src/ui/pages/settings.tsx @@ -1,5 +1,6 @@ import type { FC } from "hono/jsx"; import type { InviteCodesResult } from "../../beta/gate"; +import type { UsageBannerNotice } from "../../billing/usage-banner"; import { type ApiTokenSummary, MAX_ACTIVE_TOKENS_PER_USER, @@ -80,6 +81,8 @@ interface SettingsPageProps { nonce?: string; /** True when the user has opted out of product analytics (#257). */ telemetryOptOut: boolean; + /** The 80% warning, when this account has one. See `Layout`. */ + usageNotice?: UsageBannerNotice | null; } const SCOPE_LABEL: Record = { @@ -305,6 +308,7 @@ export const SettingsPage: FC = ({ notice, nonce, telemetryOptOut, + usageNotice, }) => { const now = Date.now(); // Mirrors the server-side cap in `createApiToken`: an expired row occupies no @@ -314,9 +318,14 @@ export const SettingsPage: FC = ({ (token) => token.revokedAt === undefined && !isExpired(token.expiresAt ?? null), ).length; return ( - +

Settings

+
+ + Usage + +