Skip to content

feat(cost): import reference prices from models.dev - #693

Open
FradSer wants to merge 2 commits into
langgenius:mainfrom
FradSer:feat/runtime-models-dev
Open

FradSer wants to merge 2 commits into
langgenius:mainfrom
FradSer:feat/runtime-models-dev

Conversation

@FradSer

@FradSer FradSer commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Replace the hand-maintained reference price table with one checked-in models.dev prices-only subset in the API: 317 models, approximately 41 KB. pkgs/runtime-catalog and the web bundle remain unchanged.
  • Use one normalized provider/model lookup for imported prices and a short reviewed override for the missing gpt-5.3 preset. Missing cache rates use the applicable input rate; explicit zero rates remain free. Reasoning is already included in output tokens, and audio rates are ignored.
  • Keep full rate precision and structured context tiers with strict > thresholds. Normalized input/cache buckets count once. Existing usage costs and price snapshots remain stored; there is no snapshot history, dated schedule, Run-time price selection, or historical repricing.
  • Add the 62-line just models-refresh script, which fetches api.json, selects mapped-provider prices and context tiers, and writes the file for git-diff review.

Why

Reference prices drifted from upstream while maintained by hand. This addresses the narrowed pricing-only scope of #691 and the six requests in #693 (review).

Closes #691.

Verification

Current head: 0e6420589f172880ee2115117f44e1666266ae09. The approved optional follow-up removes the gpt-5.3 cache-write override, so the existing input-price fallback applies (1.75 USD/million tokens, previously 2.1875). Input, output and cache-read rates are unchanged. Current pricing-only diff: 10 files, +2,744/-631.

Fresh verification on this exact head in an independent clean Linux checkout (Bun 1.4.0): pricing 53/53, persisted usage 6/6, idempotency 1/1 passed. No new pricing code change was needed during the subsequent maintenance check.

The hosted run for this head remains failed at the unchanged Driver backpressure fixture. Investigation reproduced the old fixture's premature burst-completion assertion: it could drain admitted work before all 900 notifications had arrived. The separate Driver #136 repair observes the final notification and peer write callback; it passed 40/40 repeated instances. This PR does not consume that branch or change its Driver gitlink. Rerunning the existing failed hosted job requires repository admin rights; no empty commit or branch change is used to retrigger it.

The results below are the earlier full verification on 969ed1a5; they are retained with their original source attribution.

Source: 969ed1a54076a76bdf28f8fa845647e5f783371e, rebuilt on upstream 83b7472121905c7657b26970da8f825371c6e768. Final diff: 10 pricing-related files, +2,741/-631 (2,126 added lines are JSON data).

  • Same-Mac Linux container, Bun 1.4.0: just test-package @mosoo/api passed 1,215 tests, 1 optional native Claude/Host checkpoint skip, 0 failures. This includes all 52 pricing regressions and the usage/session identity integration tests.
  • Formatting, doc links, lint, all workspace typechecks, GraphQL freshness, Public API OpenAPI freshness/backward compatibility, commit policy and whitespace checks passed. An independent read-only review found no confirmed issue against the six maintainer requests.
  • The local full just check gate is not green. The first Driver run had six failures. After completing the locked OpenCode dependency setup, its four contract tests passed; the OpenAI client file passed 55/55 on serial retest. The ACP backend file still passed 56/57: fails the turn before its terminal when a provider ignores a committed file report times out after 2,000 ms waiting for harness.lifecycleFailure.
  • A fresh, clean upstream 83b74721 checkout with the same upstream Driver pin 49df36e9735f261829ea90b6570382bd33a67775, runtime and dependency versions reproduces the same ACP file result (56 pass / 1 fail). Selecting that test alone passes on both upstream and this revision. Driver is unchanged; this file-context-sensitive failure is not introduced by the pricing diff. New hosted checks remain separate from these local results.
  • Browser/live-provider acceptance and production deployment were not run for this pricing-only revision.

Impact

  • User/API/contract changes: current reference-price estimates and normalized lookup only; no model admission, protocol, display-name or runtime capability change.
  • Generated files / GraphQL / DB / lockfile: new reviewed API-local price JSON; no generated schema, migration or lockfile changes.
  • Env or config changes: only the manual just models-refresh recipe; no credentials, refresh cadence, deployment configuration, or new provider.
  • Risk and rollback: imported estimates depend on upstream reference prices. Review refreshes as git diffs; use Git history for rollback. Recorded usage retains its stored cost/price snapshot.

Review

  • Closest review areas: price projection/provider mapping, normalized lookup/override precedence, cache fallback and zero semantics, precision and strict context tier boundaries.
  • All previous runtime-catalog metadata, snapshot-history, prek, Vite, Gemini harness, Preview, ACP/Pi directory and Driver gitlink changes are removed from this PR and retained in a recoverable local backup. No companion Driver PR is required by this revision.

Design (UI changes only, otherwise N/A)

N/A — no UI changes.

@Yevanchen Yevanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on, @FradSer. The pricing problem in #691 is real. Against today's official pricing pages, main's hand-maintained table is wrong for Claude Sonnet 5, Opus 4.5–4.7, Haiku 4.5, and DeepSeek V4, and models.dev has all of them right. I'd like to land that part, but much smaller than this PR.

What #691 needs today. Prices in cost-pricing.ts are the only model metadata Mosoo maintains by hand. Presets don't store capabilities, limits, or lifecycle status, and a display name is one reviewed line per preset. The first step is therefore narrow: reference prices come from a checked-in models.dev subset instead of the hand-written table. Everything else can wait until something consumes it. I've narrowed #691's scope to match.

Requested changes

  1. Pricing only.
    • Drop the display-name import, protocol resolution from npm, the catalog metadata fields, canonical_model_id, and the opencode-go provider (that belongs with #684).
    • Leave pkgs/runtime-catalog unchanged so the web bundle doesn't carry the snapshot. Its minified bundle grows from 12.9 KB to 229 KB here.
  2. One snapshot, no history.
    • Usage events already persist their computed cost and price snapshot, so a refresh doesn't reprice recorded usage. OpenCode works the same way: it computes cost once at step end from the current catalog and stores it (packages/core/src/session/runner/step.ts).
    • Please remove the append-only history, Run-creation-time price selection (pricingAtMs), and the dated schedules. Git history covers provenance and rollback.
  3. A small refresh script.
    • Fetch api.json, keep input, output, cache_read, cache_write, and context tiers for the mapped providers, and write the file. Review is the git diff.
    • A script a maintainer runs by hand doesn't need the lock file, atomic rename, bounded body reader, 575-line validator, or a weekly/monthly cadence. A prices-only subset is about 45 KB.
  4. Don't abstain when a rate is missing or extra.
    • Bill reasoning tokens at the output rate and ignore audio rates, as OpenCode does (packages/core/src/session/usage.ts). DeepSeek V4's reasoning rate equals its output rate, and Gemini 3.5 Flash's input_audio rate equals its input rate. Output tokens already include reasoning tokens, and audio isn't metered. As written, DeepSeek V4 Pro/Flash and Gemini 3.5 Flash (both gemini and opencode) become unknown.
    • When cache_read or cache_write is omitted, bill those tokens at the input rate, as if they weren't cached. OpenCode treats an omitted cache rate as 0, but normalizeUsageTokens carves cache tokens out of input, so 0 would make them free.
    • As written, missingCachePrice turns any call with cache-creation tokens into unknown. That affects 10 more presets that main prices today: gpt-5.5, gpt-5.4, gpt-5.4-mini, gpt-5.2, kimi-k2.6, kimi-k2.7-code, MiniMax-M3, and OpenCode's deepseek-v4-pro, glm-5.2, and minimax-m2.7.
  5. Keep one lookup path.
    • findImportedPricing returns null on a miss, so the existing ID normalization never runs for current Runs. kimi/kimi-k2.6, zai/glm-4.6, minimax/minimax-m3, opencode/qwen3.6-plus, and kimi-k2.7-code-2026-06-25 are priced on main but null here.
    • The existing guard tests ("has pricing for cost-managed preset models" and the normalization test) only pass because they were moved to an atMs before the snapshot date. Please run them at the current time again.
    • Put a short list of reviewed overrides on top of the snapshot. For example, gpt-5.3 is unknown here because models.dev doesn't list it. OpenCode layers its config over models.dev the same way.
  6. Split out the unrelated changes into their own PRs: prek default_stages, Vite allowedHosts, the Gemini e2e harness, the Preview locators, and the ACP/Pi additionalDirectories branch. Also drop the driver gitlink bump; it points at an unmerged fork commit (langgenius/mosoo-agent-driver#136).

Already right: using cost.tiers with the exact threshold (input + cache read + cache write, strict >) and ignoring the legacy context_over_200k matches both models.dev's guidance and OpenCode.

Capabilities and limits are worth importing once something consumes them. For example, limit and modalities could feed the typed model connection from #671, so Pi and custom OpenCode providers stop falling back to defaults. That deserves its own issue.

@FradSer
FradSer force-pushed the feat/runtime-models-dev branch from 30b18fb to 969ed1a Compare October 16, 2026 01:42
@FradSer FradSer changed the title feat(runtime): source model metadata from models.dev feat(cost): import reference prices from models.dev Oct 16, 2026
@FradSer

FradSer commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@Yevanchen addressed all six requests in 969ed1a5, rebuilt on current upstream 83b74721:

  1. Pricing only: runtime-catalog is unchanged; metadata/display-name/protocol imports and opencode-go are removed.
  2. One API-local snapshot: no history, dated schedules or pricingAtMs; recorded usage keeps its stored cost and price snapshot.
  3. A 62-line manual refresh script selects only prices and context tiers, producing a ~41 KB subset for git-diff review.
  4. Reasoning uses output once; audio rates are ignored. Missing cache rates use the applicable input rate, preserving explicit zero.
  5. One normalized lookup plus a short gpt-5.3 override; current-time guards and the listed legacy IDs now pass. Precision and strict tier/cache boundaries remain covered.
  6. All unrelated changes and the Driver gitlink bump are removed and backed up separately.

Validation: API 1,215 pass / 1 optional native checkpoint skip / 0 fail; formatting, lint, all typechecks, GraphQL/OpenAPI checks and commit policy pass. An independent review found no confirmed issue.

The local full gate is not green: the unchanged Driver ACP file has a 2,000 ms lifecycle timeout in fails the turn before its terminal when a provider ignores a committed file report. A fresh clean upstream checkout reproduces the same 56 pass / 1 fail; the isolated test passes on both revisions. OpenCode/OpenAI serial retests pass. I left Driver unchanged to keep this PR pricing-only.

Hosted follow-up: PR Check for 969ed1a5 passed, including native Host checkpoint recovery. All current checks are terminal: 5 successful, 2 skipped. Local baseline observations and hosted results are recorded separately.

@Yevanchen Yevanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @FradSer. This addresses all six requests.

What I checked on 969ed1a5:

  • Running the refresh script's selection on today's api.json reproduces models-dev.json byte for byte.
  • Every cost-managed preset is priced at the current time, including calls with cache reads, cache writes, and long contexts. The OpenCode-style IDs (kimi/kimi-k2.6 and the others) resolve again.
  • PR Check passed.

Optional nit, fine as a follow-up: the gpt-5.3 override keeps main's 1.25× cache_write (2.1875). models.dev lists no cache_write for its same-priced siblings gpt-5.2 and gpt-5.3-codex, so they bill cache writes at the input rate. Dropping the field would match them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(runtime): source model metadata from models.dev

2 participants