Skip to content

ori: @preset/<slug> model ids launch with wrong client-side metadata in embedded pi (wrong contextWindow + pricing) #186

Description

@eins78

ori: @preset/<slug> model ids launch with wrong client-side metadata (embedded pi)

Filed here as the OpenRouter team's public tracker since no dedicated ori issue tracker is documented — happy to move it if there is a better home. Findings verified empirically on ori 0.14.3.

Summary

ori <harness> --model @preset/<slug> passes the preset slug through to the harness, where it resolves as an unknown model id — because none of OpenRouter's catalog endpoints (GET /api/v1/models, GET /api/v1/models/user) list presets. In the embedded pi runtime this triggers the "custom model id" fallback, which inherits the full metadata of the provider's default model (moonshotai/kimi-k2.6), so the session runs with wrong context window and pricing:

inherited (wrong) actual (z-ai/glm-5.3-flash via preset)
contextWindow 262,144 1,310,720 (5×)
pricing in/out 0.95 / 4.00 $/M 0.075 / 0.25 $/M (~13×)
maxTokens ~218k 131,072

Concrete harms observed:

  1. Early compaction — pi believes the context is 262k and starts summarizing ~1M tokens before the real window ends.
  2. Wrong cost display — session stats show ~13× the real cost.
  3. max_tokens reservation mismatches — pi requests max_tokens derived from the inherited metadata; on a budget-limited API key this can produce hard 402s when the reservation is priced at the inherited model's rates (observed: You requested up to 218110 tokens, but can only afford 111458 on a kimi-priced reservation, while the same request through the preset's flash pricing succeeds).

The routing itself is fine — OpenRouter applies the preset server-side regardless. This is purely a client-side metadata gap.

Reproduction

# preset exists and works; metadata resolution is the problem
ori pi --model @preset/glm-sonnet -p "reply with just: ok"
# → Warning: Model "@preset/glm-sonnet" not found for provider "openrouter". Using custom model id.
# → ok   (request succeeds, routed through the preset)

# the runtime sees the raw slug with no metadata:
ori pi --list-models preset   # → no @preset entries

# catalogs never list presets:
curl -s https://openrouter.ai/api/v1/models/user -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  | jq '[.data[].id | select(startswith("@preset"))] | length'    # → 0

Levers tried (all client-side, all clobbered under ori)

lever result
~/.pi/agent/models.json (pi's user model registration) never read — ori embeds the pi SDK and drives it directly; the file is a CLI-startup feature
-e extension calling pi.registerProvider("openrouter", …) extension loads, but the registration is replaced by ori's own provider registration / startup model refresh
ORI_PI_OPENROUTER_CATALOG env (the extension's catalog-file override) ori unconditionally re-sets it to its own catalog path in-process
GET /api/v1/models/user refresh 431 models, presets never included

Only a direct pi launch (outside ori) honors models.json and gets correct metadata.

Suggested fix

When a model id starts with @preset/, resolve it once at launch (and on refresh):

  1. GET /api/v1/presets/<slug>/versions — the designated version's config.models array exposes the underlying model chain (e.g. ["z-ai/glm-5.3-flash", "qwen/qwen3.8-flash"]).
  2. Look up the primary model in the catalog ori already maintains and use its metadata (contextWindow, maxTokens, pricing, api, compat) for the client-side model object, while keeping @preset/<slug> as the wire model id.
  3. Fall back to "unknown metadata, don't guess" (e.g. conservative window) rather than the provider default's metadata — inheriting a different model's limits/pricing turns cosmetic warnings into wrong behavior.

This would benefit every harness ori launches, not just pi — any harness that computes context budgets, compaction thresholds, or max_tokens from client-side metadata will misbehave on preset ids.

Environment

  • ori 0.14.3 (macOS, arm64, bun binary)
  • embedded @earendil-works/pi-coding-agent runtime (pi CLI standalone: 0.85.1)
  • OpenRouter default workspace, OAuth

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions