Catalog and pricing: finish collapsing model facts and price tables into one authority #6396

Founder reconfirmed the complete MODELS_DEV_REPAIR_CHECKLIST.md as required for v0.10.2 on 2026-10-08. All 14 implementation items and the installed-candidate acceptance below are release requirements, not a post-release deferral. Source artifact: /Volumes/VIXinSSD/CW/artifacts/ratatui-pet-main-20261008/followup/space-recovery-20261009/MODELS_DEV_REPAIR_CHECKLIST.md. Continue this existing issue and ownership; do not create a second queue.

Required 0.10.2 work: repair the existing models.dev integration

Founder-requested implementation checklist, added 2026-10-08. Continue the existing 0.10.2 work and current ownership. Repair the existing integration; do not create a new crate, repository, database, service or second catalog. This section defines the current model-catalog repair and supersedes older statements below that defer these correctness defects wholesale to 0.11. The broader consolidation issue retains its follow-up scope; private billing migrations and invoice reconciliation are not prerequisites for this repair.

The read-only audit at Engine c54be26ec8ddcacb0d054831209025b531e605f9 confirmed four remaining defects: sparse provider rows shadow rich metadata; live models.dev rates are explicitly rejected; the parser misses current identity/pricing fields; and different consumers use competing readers. Remote fetching, bounded caching and generated offline seed support already exist. Recheck drift and continue rather than restarting that investigation. Preserve the active implementer's claimed paths and progress below. This checklist is requested work, not evidence of implementation or acceptance.

Implementation checklist

• Establish a reproducible baseline and claim the existing work.

  Work in canonical main. Coordinate dirty files before editing. Compare relevant stranded changes by content against main before recovering anything; preserve backups and inventory, and never use an old checkout as a new development lane. Use the Core execution plan and Catalog and pricing: finish collapsing model facts and price tables into one authority #6396 for ownership/evidence. Capture one fetched catalog with timestamp/hash and the matching Codewhale raw JSON/list/picker/runtime outputs. Record the actual provider/endpoint/auth mode and active config overrides without secrets. This gives an exact before/after comparison without depending on a changing upstream website.

• Define which facts each source owns.

  Models.dev supplies public model facts and provider-offering specifications/prices. The selected provider/account supplies actual access, route identity and any account-specific metadata. Explicit Codewhale configuration supplies user overrides and effective operating settings. Preserve these distinctions within one resolver: a catalog listing is not proof the account can call the model, and a published price is not an invoice. Keep supported, configured, available, deprecated and unknown states distinct. Do not automatically switch the selected model, account or billing route when a catalog refreshes.

• Parse the generated upstream schema completely for the features Codewhale uses.

  Read the combined catalog, including canonical models and provider offerings. Consume canonical_model_id from generated JSON, retaining legacy base_model compatibility where needed and rejecting conflicting identity claims rather than guessing. Preserve display names/families, distinct context/input/output limits, capability booleans, modalities, reasoning controls, interleaving, lifecycle status, provider/model transport hints and pricing structure. Preserve descriptive metadata needed by existing views; unknown optional fields must not break all ingestion. The generated provider row already incorporates upstream inheritance and overrides: do not reconstruct a different result or refill intentionally removed fields. Invalid required values must produce a bounded visible diagnostic rather than silently erasing the entire previous good catalog.

  Owner: crates/config/src/models_dev.rs and its projection in crates/config/src/catalog.rs. Several fields parse today but are lost in downstream projection, so follow each required field to the final consumer. References: current upstream schema and API/identity/inheritance documentation.

• Correct model identity and availability joins.

  Keep canonical model ID separate from the exact provider wire ID sent on a request. Normalize provider aliases consistently and preserve distinct endpoint/account/auth scopes. Use explicit upstream canonical relationships instead of name prefixes, family guesses or flattening gateway IDs. Same-named models on direct Anthropic, a gateway and a custom endpoint can have different limits and rates. Metadata-only entries may be shown in discovery without inventing a callable route. Check pagination and completeness of provider discovery before absence in a response is treated as removal; preserve configured custom models through existing explicit policy.

• Replace whole-row shadowing with field-aware resolution.

  An ID-only provider response states that a model exists; it does not state that context, capabilities and prices became unknown. Enrich that row from the exact compatible models.dev provider offering. Preserve explicit provider values, false, zero and empty values with meaningful upstream semantics; fill only genuine absence. Scope signed corrections and user overrides to the facts/routes they actually name. Track where each resolved fact came from so a provider roster cannot relabel a models.dev price as provider-confirmed.

  Repair both merged-list and exact-route paths in crates/tui/src/provider_lake.rs, plus any used CatalogCompiler::compile path in crates/config/src/catalog.rs. Both currently insert whole rows. Keep the source of ID-only offerings in crates/tui/src/client.rs honest; do not replace missing provider facts with invented defaults there. A single resolver should own this merge rule wherever consumed.

• Accept live models.dev rates as published-price estimates.

  Remove the blanket capabilities-only rejection in crates/tui/src/pricing.rs. Its current verified_catalog_offering path replaces live models.dev pricing with bundled rates or strips it. Update existing assertions that demand that obsolete behavior. Live catalog rates should refresh displayed estimates; Codewhale-managed billable rates, provider-reported charge receipts, enterprise terms and subscription quotas retain their own explicitly identified basis. Missing price is unknown, not free. An unpriced portion must not disappear from a total presented as complete.

• Preserve and evaluate the actual price structure.

  Support input/output, cache read/write, context tiers and relevant separately priced reasoning/audio token classes. Normalize the legacy context_over_200k representation with tiers when encountered, without counting both. Apply the tier threshold and usage definitions for the actual serving route. Do not silently flatten an unsupported tier/mode into the cheapest base rate; make the unsupported estimate explicit. Preserve provider-specific schedules and pricing modes already supported, instead of losing them in the merge. Avoid double-counting cached input or reasoning tokens that are subsets of total output.

  Reuse crates/config/src/pricing.rs for a shared quote calculation, with precise amounts/rounding and explicit currency/units; retire duplicate TUI calculations as their last consumers migrate. Four optional f64 fields in ModelsDevCost cannot express the upstream pricing structure. Do not introduce another parallel quote engine. Preserve sufficiently small rates without rounding them to zero.

• Make price freshness and history truthful.

  Distinguish source fetch time, upstream update metadata, any actual rate effective time and cache expiry. A fresh download alone is not an invoice verification or a provider's declared effective date. Show whether a current estimate is live, cached, stale or unavailable. Keep offline facts available under an explicit fallback policy; do not silently advertise an expired fallback price as current. Persist the quote used for each dispatch in the existing receipt path. Never apply today's refreshed rate retroactively to old usage or discard authoritative historical amounts.

• Fix context/output budgeting through the entire runtime.

  Use resolved limits for the picker, request validation, output ceiling, compaction and context percentage. Preserve advertised context, separate max input/output, configured output budget and effective usable context as distinct values. Respect explicit user/route overrides and any applicable endpoint constraint. When effective capacity is smaller than the catalog maximum, show the reason. Remove stale hardcoded fallbacks where an actual fact exists. For genuinely unknown limits, preserve a labeled conservative policy rather than presenting a guess as a published maximum. Do not set every Claude model to 1M: the dated snapshot includes both 1M models and Haiku 4.5 at 200K.

  Trace crates/tui/src/config.rs, crates/models/src/lib.rs, crates/config/src/catalog/reviewed.rs, route-budget/compaction consumers and the picker/footer. A correct picker number with an old runtime denominator does not complete the repair.

• Fix capability labels and request settings together.

  Distinguish reasoning supported, unsupported and unknown from the current reasoning selection. Preserve supported toggle/effort/token-budget options and their actual values; do not collapse every provider into one low/medium/high set. Display unknown as unknown, not “off”. Make supported tools, image/file/audio/video input, structured output and temperature controls agree with the resolved offering and Codewhale's implemented transport. Only send parameters the selected route supports. Preserve prior user settings when valid, explaining a now-invalid setting rather than silently changing route or account.

• Make every relevant consumer use the same resolved facts.

  Cover both codewhale models and codewhale model list, model resolve/set, the TUI picker/detail views, defaults/fallback selection, runtime planning/compaction, parent and delegated model selection, status/usage/cost views, and Engine API/client projections. The singular CLI command still constructs ModelRegistry::default(), while other paths consume the provider lake. Remove separate live-ignorant decisions in agent ModelInfo, bundled intrinsic helpers, constants and generated web facts as their consumers migrate. Keep exact user-selected model IDs and existing persisted config/session compatibility. Do not build a resolver with one caller and leave the rest on the old system.

• Complete refresh, persistence and offline behavior without adding another fetcher.

  Reuse crates/tui/src/models_dev_live.rs: it already supplies background fetching, TTL, bounded responses, disk caching and atomic writes. Ensure all catalog entry points initialize/use it consistently; successful refresh invalidates every derived snapshot and visible projection. Provide a working explicit refresh/retry action through existing controls. Keep input responsive, avoid per-card network requests and repeated parsing, and preserve the last good snapshot on timeout/parse failure. Handle old cache formats through the existing migration/invalidating path; do not tell the user to delete all configuration or credentials. Check source override isolation, empty/partial responses and restart persistence. Pin each active request to coherent facts while allowing later requests to see new metadata.

• Regenerate the fallback and remove catalog drift.

  Use the existing generated offline seed pipeline from models.dev; do not paste new model values into Rust match arms. Ensure its documented coverage meets supported cold-start behavior and does not accidentally freeze an obsolete six-model list. Keep any seed-size subset intentional and visible. Shrink catalog_corrections.json and other local overrides to current, sourced exceptions. Data-only corrections belong upstream when practical; transport/account policy belongs in Codewhale. Unknown new providers may be discoverable without claiming an unimplemented transport is callable. Existing supported providers should gain new model facts through data refresh, without another binary release.

• Make mismatches explainable and verify the finished behavior.

  Through existing model detail/diagnostic output, expose the selected provider and exact wire ID, canonical ID, relevant source/fetch time, config override, effective limit and price basis. Keep the ordinary picker compact. Produce a bounded comparison of one upstream snapshot against resolved Codewhale output; every difference must be an explicit route/config/policy difference or an identified defect. This is a direct verification artifact, not a new permanent audit service.

  Required checks: Anthropic ID-only refresh retaining its metadata; current 1M rows and a 200K control; reasoning unknown versus disabled; cache/tier-edge estimates; a gateway override and custom endpoint with the same model spelling; a subscription route with no fabricated per-token bill; refresh during use; offline first start; stale/corrupt cache; historical cost stability; consistent values across both CLIs, picker, context usage and Engine API. Verify that model changes reach the installed candidate and that responsiveness/idle CPU do not regress. Run existing focused checks, update obsolete expectations and run required repository gates. Add no new unit/integration tests or code comments without an explicit task request. Report real nonzero counts; direct execution, tests, provider acceptance and installed behavior are separate evidence.

0.10.2 acceptance and handoff

• Capture one catalog snapshot with timestamp/hash and compare the same provider/endpoint/model against Codewhale before and after. Explain every remaining mismatch.
• Demonstrate that an Anthropic ID-only refresh retains prices, context, output limits and capabilities; both CLI model commands, picker, runtime context usage and Engine API agree.
• Demonstrate tier/cache pricing, supported versus enabled reasoning, route/account isolation, offline/stale/corrupt-cache behavior and frozen historical costs.
• Qualify the installed candidate with exact source identity, installed version/hash, actual nonzero existing-check counts and responsiveness evidence. Do not infer provider/invoice/physical-terminal acceptance from unit checks.
• Update existing help/feature/release documentation and this issue's evidence; clearly identify the completed 0.10.2 slice and broader consolidation work still open.

Use canonical codewhale on main under the current workspace instructions; preserve peer changes and reconcile older checkout work before editing. Do not add new unit/integration tests or code comments without explicit task authorization; run existing checks and update obsolete expectations. Config pricing is currently floating-point: reuse its provenance/quote ownership while correcting precision, rather than assuming an exact-money implementation already exists.

The related ChatGPT authorization repair #6925, xAI authentication inspection #6926, responsiveness and every-shell inspection work remain separate active requirements; this checklist does not replace them. Release visibility: 0.10.2 planning tracker #6094.

Earlier scope, implementation history and active ownership

Status at 0.10.1 (re-triaged 2026-10-08): partly done. Filed against 0.10.0 (filed 2026-09-22, the day v0.10.0 was tagged; body cites 0.10.0 commit 2dca7fc).

What remains

• Pricing: fold crates/tui/src/pricing.rs (5,888 lines, f64) into the provenance model in crates/config/src/pricing.rs - integer micros, sourceRef/checkedAt/expiresAt per rate, one quote function, golden vectors for DeepSeek time-of-day and tier edges
• Catalog: move aliases and supports_tools out of crates/agent/src/lib.rs ModelInfo into the catalog; retire the remaining match tables in crates/models/src/lib.rs and the consts in crates/config/src/provider_defaults.rs
• Reconcile the three price authorities (TUI pricing.rs, platform rate cards, cwc_usage_rate_cards) and diff rates against real provider invoices
• Upstream thin vendors (AICraft, MiMo, SenseNova, StepFun, DashScope, command-code) to models.dev and shrink crates/config/assets/catalog_corrections.json accordingly

Checked in the 0.10.1 source

• Slice A done: crates/config/src/catalog.rs:448-487 at v0.10.1 enumerates namespaced top-level catalog.models keys as offerings after provider rows (comment cites Catalog and pricing: finish collapsing model facts and price tables into one authority #6396)
• Slice B done: commits cbfbcb6 'upstream-keyed canonical entries join offline (Catalog and pricing: finish collapsing model facts and price tables into one authority #6396 slice B)', 6227825 'generate the offline seed from a reviewed spec and a lock', cd5e643 'regenerate the offline seed from Models.dev'; CHANGELOG.md:834-841 'The offline model list is now generated from Models.dev instead of edited by hand'
• Bundled seed now carries upstream-shaped keys: crates/config/assets/models_dev.bundled.json at v0.10.1 has models keys xiaomi/mimo-v2.6-pro and xiaomi/mimo-v2.6-flash plus _meta/_reviewed
• Slice C partly done: git grep -w ApiProvider v0.10.1 -- crates matches only a test-local enum (crates/tui/src/config/tests.rs:16519) and git grep KIND_LOOKUP v0.10.1 is empty; crates/models/assets/model_catalog.bundled.json no longer exists; crates/tui/src/config/models.rs is down to 34 lines / 6 consts
• Slice C still open: crates/agent/src/lib.rs:29-35 still owns ModelInfo aliases (21 supports_tools hits); crates/tui/src/pricing.rs is still 5,888 lines; crates/config/src/provider_defaults.rs and web/lib/facts.generated.ts still exist
• Pricing partly done: unknown-is-a-value exists as UnpricedReason (crates/tui/src/pricing.rs:1197) and provenance-carrying rows in crates/config/src/pricing.rs:1-19, but pricing.rs still has 56 f64 lines and there are two Rust pricing modules; CHANGELOG.md:823-833 and :842-845 record the 0.10.1 price corrections

Supersedes #4173, which was opened from a 2026-07 architecture doc with counts that have since drifted (it says 81 models / 31 providers; both are now far larger). Its acceptance criterion was "there is an inventory of which registries are product contracts vs generated/catalog data vs internal implementation." That inventory now exists and is below, so the old issue is closed in favour of this scoped one.

Root cause, verified

offerings_from_models_dev (crates/config/src/catalog.rs:344) enumerates catalog offerings from catalog.providers[*].models only.

models.dev deliberately splits facts about a model from facts about a provider serving it. We implemented one half. A model that exists only in the top-level models map therefore produces no offering and can never be listed — so we are structurally late for every vendor, not just the one that surfaced it.

Two precisions worth recording, because they change the fix:

1. The top-level map is not entirely unread — crates/config/src/models_dev.rs:86 reads it for reasoning-support enrichment. The defect is in enumeration, not deserialization. The fix is "enumerate from the union", not "read a field nobody reads".
2. ModelsDevModel (models_dev.rs:199) carries no provider field — id, name, family, capabilities, limit, modalities, and nothing naming who serves it. The join must therefore come from the map key (xiaomi/mimo-v2.6-pro). That works against live upstream data, but our bundled snapshot keys models bare (its only two entries are deepseek-v4-pro, deepseek-v4-flash). So the ingestion fix alone does not repair offline/first-boot until crates/config/assets/models_dev.bundled.json is regenerated in upstream shape.

Consequence for 0.10.0: commit 2dca7fcdda (hand-seeded MiMo 2.6 rows) is a human doing the join the code refuses to do, and it stays load-bearing for cold offline start. Retire it when the seed is regenerated, not when the ingestion fix lands.

Inventory: seven independent declarations of the same model facts

Plus two parallel provider enums — ProviderKind (52) and ApiProvider (53) — bridged by a fixed-size KIND_LOOKUP: [_; 53] in crates/tui/src/config.rs.

Adding one provider today costs ~14 hand edits across provider_kind.rs, provider.rs, lib.rs, the TUI enum and that fixed-size array. That cost is why vendors go missing: adding one is a code change, so it competes with feature work and loses.

Slices

A — model-only ingestion. Enumerate offerings from the union of providers[*].models and namespaced top-level models keys. ~30 lines plus a test. Stops every vendor being late. Land first, independent of everything else.

B — regenerate the offline seed in upstream key shape, so A also fixes cold start, then retire the hand-seeded rows.

C — collapse to one authority. Migrate the last consumer of each of the seven, deleting as you go; move aliases + supports_tools out of crates/agent into the catalog (their only home today); collapse ApiProvider into ProviderKind so KIND_LOOKUP dies. Large blast radius — exhaustive matches across two crates — so it is deliberately not a release-window change.

D — upstream the thin vendors to models.dev (AICraft, MiMo, SenseNova, StepFun, DashScope, command-code). It is a public repo; the live refresh then picks them up with no infrastructure, and every other tool benefits. Keep a local overlay only for what models.dev structurally cannot carry: transport facts and prices we bill against.

Pricing: hold it to billing standard before billing exists

Display-only today, but PRODUCT_PRD.md:220 commits to "an evidence-backed model catalog". A wrong cost number is a false product claim regardless of whether money moves.

There are currently three pricing authorities: pricing.rs (display), the platform rate cards (CWC_MANAGED_MODEL_RATE_CARDS_JSON), and cwc_usage_rate_cards (Postgres). When they disagree we show one number and would charge another. Managed inference is gated off today, so fixing the divergence now costs nothing; after it is enabled it means migrating historical charges.

The rate-card shape in wrangler.jsonc is already the professional one — revision, sourceRef, pricingPolicy, checkedAt, expiresAt, integer micros, customer vs provider rates, CWC_RATE_CARD_MAX_AGE_DAYS. The work is propagating it, not inventing it:

1. Integer micros end to end, never f64.
2. sourceRef / checkedAt / expiresAt on every rate; past expiry reads unpriced, never stale-priced.
3. Unknown is a value, not zero — Quote::Unpriced { reason } as a type, so no surface can render a guess as $0.00. The MiMo seed already gets this right in data; make it unrepresentable to get wrong.
4. Separate rate data (catalog rows) from billing-surface detection (credential logic, stays local) from rate rules (time-of-day, context tiers) as a few typed variants over data rather than 287 arms.
5. Freeze the quote at dispatch and persist it in the receipt; a later refresh must never reprice history. Already correct on the cloud-facts path — generalize it.
6. One quote function, one stated rounding rule, an audit trail per number.
7. Golden vectors: (route, usage, timestamp) → exact amount, covering the DeepSeek time-of-day windows and the 512k/272k/200k tier edges.
8. Reconcile against real invoices. The evidence file records invoice reconciliation as "unqualified". 287 arms nobody has diffed against a bill are 287 guesses, however tidy. This is the only item that needs something we do not already have, and it is the one that separates professional from merely organized.

Items 1–7 are engineering. Item 8 is ground truth, and no amount of 1–7 substitutes for it.

2026-10-08 Codex /root takeover (local 0.10.2 candidate, no push): claimed crates/config/src/{models_dev.rs,catalog.rs,pricing.rs}, existing related tests/literals, crates/tui/src/{provider_lake.rs,provider_catalog_live.rs,pricing.rs,tui/model_picker.rs}, and crates/cli/src/lib.rs for current models.dev schema, sparse-roster completion, scoped/frozen estimates, and CLI agreement. Source confirms config pricing is currently floating-point; the earlier handoff's claim of an existing exact-money implementation was inaccurate. Broader invoice and managed-billing consolidation remains open. No new test cases or code comments under current operator instructions; existing cases will be adapted where behavior changes.