feat: meter research runs against account credits (route-0 billing) - #531
Conversation
User-facing billing for research runs, per the teacher-approved direction
(platform credits settle the author; LLM gateway comes later):
- domain/entitlements/research_credits.py: authorize/refund one research
run against the account balance. Same strict-opt-in master switch as
backtest metering (CREDITS_METERING_ENABLED); the per-run price lives in
RESEARCH_RUN_CREDIT_COST (env, default 10) because Deep Research spend
varies by agent and provider tier in a way backtest granularity never
did. Store errors fail open like authorize_llm_run.
- POST /api/v1/research/agents/{id}/runs: debit at accept, before the
agent service is called; 402 with an account-shaped message when the
balance can't cover it; refund only when the service never accepted the
run — refund_credits is unconditional at the store level, so the route
gates on outcome.charged to never mint free credits.
- test_research_credits.py: the billing decision table (off/armed/empty/
anonymous/cost-env/refund pairing), on the same temp-store seams as
test_credit_metering.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…edit) Reworks route-0 billing per confirmation: research runs now reserve and settle on the CreditsStore ($1 = 1 credit, micro-denominated) — the same real-money rail platform-credit backtests settle on — instead of the legacy integer admin quota, which real-money credit holders never carried. - submit: credits_service.reserve_llm_credits holds a per-run usage ceiling (RESEARCH_RUN_ESTIMATE_USD_CENTS, default $1); the store's InsufficientCreditsError maps to 402. The minted reservation id is persisted on the run row (new column, ALTER-migrated for existing installs). - completion: settle at the reserved estimate (route-0 interim); contract v1.1's usage reporting upgrades this to settle-at-actual. - discovered failure: release the reservation. - the route-0 integer-quota module (research_credits.py) and its tests are deleted; the legacy entitlements flat module (credits.py) is annotated NOT-wired so nobody wires it by mistake.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19fb28b81b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| credits_service.release_llm_credits( | ||
| run["reservation_id"], reason="research run failed", | ||
| ) |
There was a problem hiding this comment.
The reservation is released when the status response says failed, but not when a completed run's /result request raises an HTTP error: that branch marks the run failed, after which _maybe_complete_run returns early on every poll. Because an open reservation is deducted from available Credits, even a transient result-endpoint 5xx permanently strands the user's held balance; keep that failure retryable or otherwise close the reservation before making the run terminal.
Useful? React with 👍 / 👎.
| def _estimate_micro_from_run(run: Dict[str, Any]) -> int: | ||
| """Route-0 settle amount: the reserved estimate, from env-tunable cents.""" | ||
| return credits_micro_for_cents(_estimate_usd_cents()) |
There was a problem hiding this comment.
This recomputes the settlement from the current environment instead of using the amount held for this run. If RESEARCH_RUN_ESTIMATE_USD_CENTS changes during a deployment while a run is in progress, a run reserved at 1 Credit can later settle at 2 Credits, consuming supplementary balance or recording an overage and restricting the account; persist the submitted estimate or retrieve the reservation's reserved_micro for route-0 settlement.
Useful? React with 👍 / 👎.
| try: | ||
| with _connect() as conn: | ||
| conn.execute( | ||
| "ALTER TABLE research_runs ADD COLUMN reservation_id TEXT" |
There was a problem hiding this comment.
On an existing installation this migration leaves reservation_id NULL for every pre-billing row. If a queued or running legacy run completes after deployment, _maybe_complete_run passes that NULL value to settle_llm_credits (or to release_llm_credits on failure), which raises because reservation IDs must be non-empty; completed legacy reports therefore remain stuck as running and cannot be opened. Legacy rows need an explicit no-billing completion path or a valid backfill.
Useful? React with 👍 / 👎.
| except httpx.HTTPError as exc: | ||
| credits_service.release_llm_credits( | ||
| reservation_id, reason="submit failed; no Deep Research invocation" | ||
| ) |
There was a problem hiding this comment.
This cleanup only runs for httpx.HTTPError, while response.json() raises JSONDecodeError/ValueError for a 2xx response with malformed JSON. In that scenario the newly created reservation remains open, no local run is persisted to settle or release it later, and each client retry can strand another portion of the user's balance; the post-reservation submission path must close the hold for response-decoding and shape failures too.
Useful? React with 👍 / 👎.
| except InsufficientCreditsError as exc: | ||
| raise HTTPException(status_code=402, detail=str(exc)) from None |
There was a problem hiding this comment.
The credit store also raises CreditAccountRestrictedStoreError when an account is paused for refund review or unpaid overage, but this handler catches only insufficient balance. Any restricted user submitting research therefore receives an uncaught 500 instead of the intentional account-restricted refusal used by the other Credits execution path; catch and translate that store error to an appropriate 403/402 response.
Useful? React with 👍 / 👎.
- Settle at the amount reserved for THIS run (persisted estimate_micro at submit) instead of re-reading the env at completion — a mid-flight env change could settle a different price than was reserved. - Result-fetch failures are retryable, not terminal: the run completed on the service, so a transient /result 5xx must not strand the reservation behind a failed terminal status. - Release the hold on any post-reserve submit failure, including a 2xx response whose JSON body is malformed (ValueError path), which previously left the hold open with no run row to settle it later. - Map CreditAccountRestrictedStoreError to a 403 refusal instead of an uncaught 500, matching the other Credits surfaces. - Legacy pre-billing runs (reservation_id NULL after the migration) complete without touching Credits instead of raising on a NULL id.
This branch was successfully deployed
2 participants
What this does
User-facing billing for research runs on the real-money Credits rail ($1 = 1 credit) — the same system Stripe purchases fund and platform-credit backtests settle on (CreditsStore, micro-denominated). Reworked from the first draft per confirmation: the earlier route-0 draft debited the legacy integer admin quota, which real-money credit holders never carried.
Design
credits_service.reserve_llm_creditsholds a per-run usage ceiling (RESEARCH_RUN_ESTIMATE_USD_CENTS, default $1) from the caller's balance. Store'sInsufficientCreditsError→ 402.Changes
domain/entitlements/research_credits.py— authorize/refund one research run against the account balance. Same strict opt-in master switch as backtest metering (CREDITS_METERING_ENABLED); the per-run price lives inRESEARCH_RUN_CREDIT_COST(env, default 10) because Deep Research spend varies by agent and provider tier in a way backtest granularity never did. Store errors fail open likeauthorize_llm_run.api/routers/research.py—create_research_rundebits at accept, passes through the service's 422s, refunds on submission failure when charged.domain/entitlements/research_credits.pydeleted — superseded by this CreditsStore wiring. Legacy credits.py annotated NOT-wired (flat per-run debit; the live backtest billing is the reserve–settle system).