diff --git a/AGENTS.md b/AGENTS.md index 8e9dbd0e..4274c472 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,6 +80,37 @@ This applies to `AGENTS.md`-following agents as much as to humans. If a task asks you to write up a security finding, put the write-up in scratch space and say where it is; do not add it to `devlog/`, `structure/`, or `docs-site/`. +## Native Codex + OpenCodex coexistence invariant + +OpenCodex extends native Codex; it does not replace the native OpenAI identity or create a second +authority for native model metadata. + +- On the normal loopback install, keep Codex's built-in `openai` provider identity and the user's + ordinary ChatGPT/Codex login. Route it through OCX with the managed `openai_base_url` override; + do not re-tag native threads to an OCX provider. +- While OCX owns active routing, `$CODEX_HOME/opencodex-catalog.json` is the only active merged + catalog. Never create or point Codex at parallel merge files such as + `~/.codex/model-catalogs/native-plus-ocx.json`. +- Managed sync/build must target that canonical catalog **before** config injection. Never use the + current root `model_catalog_json` as an OCX write target: it may still be the user's pre-OCX + catalog. Cache invalidation must consume the exact catalog path written by the same sync. +- Restore/eject may restore a user-owned catalog pointer from the journal, but catalog cleanup must + still target only the canonical OCX-managed catalog. Never strip routed rows from that restored + user catalog. +- Bare native OpenAI rows must come from authoritative native discovery for the installed Codex + client/account and retain upstream capability metadata. In particular, do not drop or synthesize + `context_window`, reasoning ladders, `supports_search_tool`, `tool_mode`, or + `use_responses_lite` for newly rolled-out models. +- If the OCX merged catalog cannot be materialized, or it does not contain the currently selected + bare native GPT/Codex model, prefer Codex's native catalog over generic fallback metadata or a + stale/custom root `model_catalog_json`. The injection journal preserves the user's prior + config/catalog pointer and restores it on stop/eject. +- A user-owned external `model_provider` or root `openai_base_url` remains an ownership boundary; + OCX must not silently take it over. + +Any change to injection, catalog sync, native discovery, install/start/ensure, or restore must keep +these invariants covered by focused regression tests. + ## Commands ```bash @@ -197,4 +228,4 @@ Use the canonical labels `needs-triage`, `needs-info`, `ready-for-agent`, `ready This repository uses a single-context domain layout with root `CONTEXT.md` (when present) and `docs/adr/`. See `docs/agents/domain.md`. -Compound Engineering overlay: `.compound-engineering/` (tracked `config.yaml`, gitignored `config.local.yaml`). Artifact root `.compound-engineering/artifacts/`. Portable skills `~/.agents/skills/ce-*`; native Cursor plugin is fallback only when this overlay is absent. +Compound Engineering overlay: `.compound-engineering/` (tracked `config.yaml`, gitignored `config.local.yaml`). Artifact root `.compound-engineering/artifacts/`. Portable skills `~/.agents/skills/ce-*`; native Cursor plugin is fallback only when this overlay is absent. \ No newline at end of file diff --git a/docs-site/src/content/docs/guides/codex-integration.md b/docs-site/src/content/docs/guides/codex-integration.md index d2d7ac93..d2b31208 100644 --- a/docs-site/src/content/docs/guides/codex-integration.md +++ b/docs-site/src/content/docs/guides/codex-integration.md @@ -31,6 +31,28 @@ The proxy listens on port `10100` by default and serves `POST /v1/responses`, `POST /v1/responses/compact`, `POST /v1/images/generations`, `POST /v1/images/edits`, `GET /v1/models`, `GET /healthz`, and the `/api/*` management surface. +### Native Codex and OpenCodex together + +The loopback setup is intentionally **native-first**. You stay signed in to Codex with your normal +ChatGPT/Codex account, threads keep the native `openai` provider id, and native OpenAI models remain +bare ids such as `gpt-6.1-sol`. OpenCodex adds routing and extra providers around that native path; +it does not require a second Codex identity. + +While OpenCodex owns that routing, `$CODEX_HOME/opencodex-catalog.json` is the active merged catalog. +`ocx start`, `ocx ensure`, and `ocx sync` enforce that pointer. If `config.toml` previously +pointed at another custom merge file, that value is journaled for restore but is not kept as the +active catalog while OCX is running. This prevents a stale custom catalog from downgrading a newly +rolled-out native model to Codex's generic fallback metadata. + +Native rows are discovered from ChatGPT's Codex model endpoint with the installed Codex client +version and the effective Codex account. Their upstream capability fields are authoritative, +including context window, reasoning levels, search/deferred-tool support, Responses-lite behavior, +modalities, and visibility. If the OCX merged catalog cannot be built, OpenCodex prefers Codex's +native catalog rather than leaving an unrelated custom root `model_catalog_json` active. + +Do not manually create a second `native-plus-ocx.json` style catalog to combine the two. The +canonical OCX catalog is already the native-plus-routed merge. + ### Built-in image generation (`image_gen`) Codex's built-in `image_gen` tool does not go through `/v1/responses` — the codex-rs extension @@ -135,9 +157,9 @@ mode leaves this profile untouched. :::caution Root keys such as `openai_base_url`, `model_provider`, and `model_catalog_json` **must** sit before the -first `[table]` header. The injector guarantees that placement, removes its own stale/duplicate -copies, and never overwrites a user-owned root `openai_base_url`; if one exists, sync updates the -catalog but reports that routing was not injected. +first `[table]` header. The injector guarantees that placement and removes its own stale/duplicate +copies. A user-owned root `openai_base_url` is an ownership boundary: OpenCodex leaves both routing +and the root catalog pointer untouched rather than managing only half of the configuration. ::: ## Shared model catalog @@ -184,13 +206,16 @@ start and on `ocx sync`, opencodex: 1. **Backs up** the pristine catalog once to `~/.opencodex/catalog-backup.json` (so featuring is reversible). -2. **Fetches** eligible providers' live model catalogs (cached ~5 min; falls back to the last good - list, then configured `models[]`). Forward auth has no model endpoint, and Cursor uses its - `GetUsableModels` RPC rather than `/models`. -3. **Merges** routed models in as namespaced entries (`provider/model`), cloned from a native Codex - catalog template so Codex's strict parser accepts them. -4. **Filters** `config.disabledModels` and each provider's non-empty `selectedModels` allowlist. -5. **Re-ranks** so featured models sort first (see below), then writes the merged catalog back. +2. **Discovers native OpenAI rows live** from the Codex model endpoint using the installed client + version and the effective ChatGPT/Codex account. New native slugs are accepted without waiting + for an OpenCodex release, and their upstream metadata is preserved. +3. **Fetches** eligible routed providers' live model catalogs (cached ~5 min; falls back to the last + good list, then configured `models[]`). Forward auth has no generic provider model endpoint, and + Cursor uses its `GetUsableModels` RPC rather than `/models`. +4. **Merges** routed models in as namespaced entries (`provider/model`), cloned from a native Codex + catalog template so Codex's strict parser accepts them without mutating authoritative native rows. +5. **Filters** `config.disabledModels` and each provider's non-empty `selectedModels` allowlist. +6. **Re-ranks** so featured models sort first (see below), then writes the merged catalog back. Routed catalog entries also get their GPT-5 identity rewritten to the real upstream model name. Reasoning controls come from provider/model metadata across Codex's `low | medium | high | xhigh | @@ -246,38 +271,33 @@ rows below them. ### Catalog troubleshooting If a model is missing from Codex, or the catalog order/visibility looks wrong, check in order: - -1. **`selectedModels`** on the provider — a non-empty allowlist exposes only those ids to Codex; +\n2. **Active catalog ownership** — while OCX owns routing, the root `model_catalog_json` should point + to `$CODEX_HOME/opencodex-catalog.json`. A parallel `native-plus-ocx.json` or other merged file + is drift. Run `ocx sync` (or `ocx ensure`) to repair the managed pointer; `ocx stop` restores + the pre-OCX user value from the journal.\n2. **`selectedModels`** on the provider — a non-empty allowlist exposes only those ids to Codex; empty or omitted exposes all discovered models. An id not in the allowlist never reaches the - catalog. -2. **`disabledModels`** (top level) — hides models from both the catalog and `/v1/models`, and flips - bare native GPT slugs to `visibility: "hide"`. -3. **`liveModels: false` with empty `models`** — when live discovery is off and `models` is empty or - omitted, opencodex exposes no routed models for that provider. -4. **Cursor `GetUsableModels`** — the Cursor adapter discovers models through its protobuf + catalog.\n3. **`disabledModels`** (top level) — hides models from both the catalog and `/v1/models`, and flips + bare native GPT slugs to `visibility: "hide"`.\n4. **`liveModels: false` with empty `models`** — when live discovery is off and `models` is empty or + omitted, opencodex exposes no routed models for that provider.\n5. **Cursor `GetUsableModels`** — the Cursor adapter discovers models through its protobuf `GetUsableModels` RPC, not `/models`, so a Cursor-side change can alter which ids are visible - independently of other providers. -5. **Cache and `ocx sync`** — live catalogs are cached for about five minutes (`modelCacheTtlMs`, - default `300000`). Run `ocx sync` to force a fresh fetch and rewrite the catalog immediately. -6. **Running Codex `app-server`** — rewriting the on-disk catalog is not enough while a long-lived + independently of other providers.\n6. **Cache and `ocx sync`** — live catalogs are cached for about five minutes (`modelCacheTtlMs`, + default `300000`). Run `ocx sync` to force a fresh fetch and rewrite the catalog immediately.\n7. **Running Codex `app-server`** — rewriting the on-disk catalog is not enough while a long-lived Codex `app-server` (Desktop / CLI background host) keeps the previous list in memory. `ocx sync` and `ocx sync-cache` warn when those processes are detected. Restart them with `ocx sync --restart-codex` (or stop the matching `app-server` processes yourself), then let Codex - recreate them so the new list appears. -7. **`hideUnavailableModels`** — when enabled, a provider that is dead (all accounts need reauth, or + recreate them so the new list appears.\n8. **`hideUnavailableModels`** — when enabled, a provider that is dead (all accounts need reauth, or discovery fails N times) drops from `/v1/models` and the new-session picker while the admin Models tab still shows last-good rows with a reason. Codex and Cursor cache their pickers; start a **new session** (or restart the client / run `ocx sync --restart-codex`) before expecting the filtered list. Existing sessions keep routing to last-good models. :::caution[Other local writers] -Catalog writes (`opencodex-catalog.json`, `config.toml`) are atomic **inside** opencodex, which only -prevents half-written files when two opencodex-owned writers race. That does **not** stop another -local process, file watcher, or sync agent from rewriting catalog visibility or order after opencodex -has written. Codex keeps its separate `models_cache.json` and can refresh it independently, changing -the visible list without rewriting `opencodex-catalog.json`. If models flip unexpectedly while the -proxy is running, stop or reconfigure the competing writers, then run `ocx sync` — this is an -external-writer hazard, not a confirmed opencodex defect. +Catalog writes (`opencodex-catalog.json`, `config.toml`) are atomic **inside** opencodex. Another +local process can still rewrite them afterwards. While OCX owns routing, changing the root +`model_catalog_json` to a competing merged catalog is configuration drift; the next `ocx start`, +`ocx ensure`, or `ocx sync` repairs the managed pointer. Codex also keeps +`models_cache.json` and can retain a stale in-memory catalog in a long-lived app-server, so restart +that client after a catalog repair when needed. ::: ## Proxy connection errors @@ -341,4 +361,4 @@ ocx restore back # point plain Codex at the running proxy again When opencodex runs as a managed [background service](/reference/cli/#ocx-service), it sets `OCX_SERVICE=1` so a service-driven restart does **not** thrash the Codex config — only an explicit -`ocx stop` / `ocx service stop` restores native Codex. +`ocx stop` / `ocx service stop` restores native Codex. \ No newline at end of file diff --git a/src/AGENTS.md b/src/AGENTS.md index ab7a5fef..c478db12 100644 --- a/src/AGENTS.md +++ b/src/AGENTS.md @@ -19,10 +19,26 @@ This file applies to `src/` and inherits the repository-wide rules in `/AGENTS.m - Adapter changes must preserve the internal event contract, streaming behavior, tool calls, cancellation, error mapping, and image handling relevant to that adapter. - Authentication, OAuth, token, credential, management API, and CORS changes are security-boundary changes. +## Codex native coexistence + +For Codex integration work, the loopback path is additive: keep the built-in `openai` provider +identity and ordinary ChatGPT/Codex auth, and let OCX own only the managed proxy transport plus the +canonical merged catalog at `$CODEX_HOME/opencodex-catalog.json`. + +Do not introduce alternate "native + OCX" catalog files or preserve a competing root +`model_catalog_json` while OCX owns routing. Managed sync must write the canonical OCX catalog +directly before injection and must never use the currently configured user catalog as an +intermediate write target. Restore cleanup likewise targets only the managed catalog after the +journal restores user config. Native bare OpenAI rows are authoritative live rows; +preserve their capability fields unchanged so newly rolled-out models do not fall back to generic +Codex metadata. When no managed catalog is available, or the selected bare native GPT/Codex slug is +absent from it, remove the managed root catalog override and let native Codex metadata win. +Restore/eject must recover the user's pre-OCX config through the journal. + ## Tests and validation - Place focused regression coverage near the existing tests for the affected subsystem. - For focused behavior, run the relevant `bun test tests/.test.ts` and `bun run typecheck`. - For shared routing, adapters, config, OAuth, or server behavior, also run `bun run test`. - For logging, requests, credentials, account data, or fixtures, also run `bun run privacy:scan`. -- Update `docs-site/` when the change affects user-visible behavior or configuration. +- Update `docs-site/` when the change affects user-visible behavior or configuration. \ No newline at end of file diff --git a/src/codex/catalog/metadata.ts b/src/codex/catalog/metadata.ts index 32fed5e7..2be0aa93 100644 --- a/src/codex/catalog/metadata.ts +++ b/src/codex/catalog/metadata.ts @@ -137,10 +137,18 @@ export function nativeModelRows(config: Pick): Arra }); } -export function applyNativeVisibility(entries: RawEntry[], disabledNative: Set): RawEntry[] { +export function applyNativeVisibility( + entries: RawEntry[], + disabledNative: Set, + authoritativeNativeSlugs: ReadonlySet = new Set(), +): RawEntry[] { for (const entry of entries) { const slug = typeof entry.slug === "string" ? entry.slug : ""; - if (!slug || slug.includes("/") || !SUPPORTED_NATIVE_OPENAI_SLUGS.has(slug)) continue; + if ( + !slug + || slug.includes("/") + || (!SUPPORTED_NATIVE_OPENAI_SLUGS.has(slug) && !authoritativeNativeSlugs.has(slug)) + ) continue; entry.visibility = disabledNative.has(slug) ? "hide" : "list"; } return entries; diff --git a/src/codex/catalog/native-discovery.ts b/src/codex/catalog/native-discovery.ts new file mode 100644 index 00000000..ef598e9d --- /dev/null +++ b/src/codex/catalog/native-discovery.ts @@ -0,0 +1,251 @@ +import type { OcxConfig } from "../../types"; +import { + MODEL_DISCOVERY_MAX_MODEL_ID_LENGTH, + MODEL_DISCOVERY_MAX_MODELS, + MODEL_DISCOVERY_MAX_RESPONSE_BYTES, + extractModelEnvelopeRows, + readBoundedDiscoveryJson, + type BoundedDiscoveryJsonResult, +} from "../../providers/model-discovery"; +import { isSelectableCodexPoolAccount, MAIN_CODEX_ACCOUNT_ID } from "../account-id"; +import { getValidCodexToken } from "../account-store"; +import { getMainAccountToken } from "../main-account"; +import { getEffectiveActiveCodexAccountId } from "../routing"; +import { resolveCodexRuntime } from "../runtime"; +import type { RawEntry } from "./parsing"; + +const NATIVE_MODELS_ENDPOINT = "https://chatgpt.com/backend-api/codex/models"; + +function hasNativeModelIdControlChars(value: string): boolean { + for (const char of value) { + const code = char.codePointAt(0); + if (code === undefined) continue; + if ( + code <= 0x1f + || (code >= 0x7f && code <= 0x9f) + || code === 0x2028 + || code === 0x2029 + ) { + return true; + } + } + return false; +} + +type NativeCredential = { + accessToken: string; + chatgptAccountId: string; +}; + +type NativePoolToken = NativeCredential & { + generation: number; +}; + +export interface NativeOpenAiCatalogDiscovery { + models: RawEntry[]; + clientVersion: string | null; +} + +export interface NativeOpenAiCatalogDiscoveryDeps { + fetch?: typeof fetch; + getEffectiveActiveCodexAccountId?: (config: OcxConfig) => string | undefined; + getMainAccountToken?: () => NativeCredential | null; + getValidCodexToken?: (id: string) => Promise; + resolveClientVersion?: () => string | null; +} + +type CredentialCandidate = + | { kind: "main" } + | { kind: "pool"; id: string }; + +function isRecord(value: unknown): value is Record { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function defaultClientVersion(): string | null { + return resolveCodexRuntime({ discoverAlternatives: false }).runtime.version; +} + +function credentialCandidates( + config: OcxConfig, + selectedId: string | undefined, +): CredentialCandidate[] { + const paused = new Set(config.pausedCodexAccountIds ?? []); + const poolIds = config.codexAccountPools === false + ? [] + : (config.codexAccounts ?? []) + .filter(isSelectableCodexPoolAccount) + .map(account => account.id) + .filter(id => !paused.has(id)); + + const selectedPool = selectedId + && selectedId !== MAIN_CODEX_ACCOUNT_ID + && poolIds.includes(selectedId) + ? selectedId + : undefined; + + const out: CredentialCandidate[] = []; + if (selectedPool) out.push({ kind: "pool", id: selectedPool }); + if (!paused.has(MAIN_CODEX_ACCOUNT_ID)) out.push({ kind: "main" }); + for (const id of poolIds) { + if (id !== selectedPool) out.push({ kind: "pool", id }); + } + return out; +} + +async function resolveCredential( + candidate: CredentialCandidate, + deps: Required>, +): Promise { + if (candidate.kind === "main") return deps.getMainAccountToken(); + try { + const token = await deps.getValidCodexToken(candidate.id); + return { + accessToken: token.accessToken, + chatgptAccountId: token.chatgptAccountId, + }; + } catch { + return null; + } +} + +function validatedNativeModels(value: unknown): RawEntry[] | null { + const envelope = extractModelEnvelopeRows(value, MODEL_DISCOVERY_MAX_MODELS, ["models"]); + if (!envelope.ok) return null; + + const models: RawEntry[] = []; + const seen = new Set(); + for (const raw of envelope.rows) { + if (!isRecord(raw)) return null; + const slug = raw.slug; + if ( + typeof slug !== "string" + || !slug + || slug !== slug.trim() + || slug.length > MODEL_DISCOVERY_MAX_MODEL_ID_LENGTH + || hasNativeModelIdControlChars(slug) + ) { + return null; + } + if (seen.has(slug)) continue; + seen.add(slug); + const clone: RawEntry = { ...raw }; + // The request is already version-filtered for the installed Codex runtime. Keeping this + // field would make a proxy serving another compatible Codex build hide an otherwise usable row. + delete clone.minimal_client_version; + models.push(clone); + } + return models; +} + +/** + * Fetch the native Codex model catalog with the credential OCX is actually routing through. + * + * This intentionally never rewrites ~/.codex/auth.json. A selected pool account is tried first; + * otherwise the physical Desktop login is tried first. Authentication failures and invalid + * responses fall through to the remaining pool credentials, and total failure leaves the + * snapshot-backed catalog path untouched. + */ +export async function discoverNativeOpenAiCatalog( + config: OcxConfig, + injected: NativeOpenAiCatalogDiscoveryDeps = {}, +): Promise { + const deps = { + fetch: injected.fetch ?? fetch, + getEffectiveActiveCodexAccountId: + injected.getEffectiveActiveCodexAccountId ?? getEffectiveActiveCodexAccountId, + getMainAccountToken: injected.getMainAccountToken ?? getMainAccountToken, + getValidCodexToken: injected.getValidCodexToken ?? getValidCodexToken, + resolveClientVersion: injected.resolveClientVersion ?? defaultClientVersion, + }; + + const clientVersion = deps.resolveClientVersion(); + if (!clientVersion) return { models: [], clientVersion: null }; + + const url = new URL(NATIVE_MODELS_ENDPOINT); + url.searchParams.set("client_version", clientVersion); + + const candidates = credentialCandidates( + config, + deps.getEffectiveActiveCodexAccountId(config), + ); + // One wall-clock budget for the entire account fallback sequence. A dead upstream must not + // multiply the discovery delay by the number of configured pool accounts. + const requestSignal = AbortSignal.timeout(8_000); + for (const candidate of candidates) { + const credential = await resolveCredential(candidate, deps); + if (!credential?.accessToken || !credential.chatgptAccountId) continue; + + let response: Response; + try { + response = await deps.fetch(url, { + method: "GET", + headers: { + authorization: `Bearer ${credential.accessToken}`, + "chatgpt-account-id": credential.chatgptAccountId, + originator: "codex_cli_rs", + version: clientVersion, + }, + signal: requestSignal, + }); + } catch { + continue; + } + + if (!response.ok) { + try { + void response.body?.cancel().catch(() => undefined); + } catch { + // Best-effort body cleanup only. + } + continue; + } + + let parsed: BoundedDiscoveryJsonResult; + try { + parsed = await readBoundedDiscoveryJson( + response, + MODEL_DISCOVERY_MAX_RESPONSE_BYTES, + ); + } catch { + continue; + } + if (!parsed.ok) continue; + const models = validatedNativeModels(parsed.value); + if (!models) continue; + return { models, clientVersion }; + } + + return { models: [], clientVersion }; +} + +/** Replace same-slug native rows with live authoritative rows and append newly rolled-out ones. */ +export function mergeDiscoveredNativeCatalogRows( + catalogModels: RawEntry[], + discoveredModels: RawEntry[], +): RawEntry[] { + if (discoveredModels.length === 0) return catalogModels; + + const bySlug = new Map(); + for (const model of discoveredModels) { + if (typeof model.slug === "string" && !model.slug.includes("/")) { + bySlug.set(model.slug, model); + } + } + if (bySlug.size === 0) return catalogModels; + + const merged = catalogModels.map(model => { + const slug = typeof model.slug === "string" && !model.slug.includes("/") + ? model.slug + : undefined; + if (!slug) return model; + const replacement = bySlug.get(slug); + if (!replacement) return model; + bySlug.delete(slug); + return replacement; + }); + return [...merged, ...bySlug.values()]; +} diff --git a/src/codex/catalog/sync.ts b/src/codex/catalog/sync.ts index 3931f2bd..6746b09b 100644 --- a/src/codex/catalog/sync.ts +++ b/src/codex/catalog/sync.ts @@ -33,6 +33,7 @@ import upstreamModelsSnapshot from "../data/upstream-models.json"; import { activeCodexModelsCachePath, applyJawcodeCatalogMetadata, applyMultiAgentMode, applyNativeOpenAiContextOverride, catalogModelSlug, ensureCatalogBackup, ensureStrictCatalogFields, findNativeTemplate, isRoutedModelCompatibilityExcluded, normalizeRoutedCatalogEntry, normalizeServiceTiers, readCatalog, readCatalogBackup, readCodexCatalogPath, readNativeBaseline } from "./parsing"; import type { CatalogModel, MultiAgentMode, RawEntry } from "./parsing"; +import { discoverNativeOpenAiCatalog, mergeDiscoveredNativeCatalogRows } from "./native-discovery"; import { applyNativeVisibility, disabledNativeSlugs, isUnsupportedOpenAiNativeSlug, nativeOpenAiSlugs, shouldUpgradeToUpstreamEntry, upstreamNativeEntry } from "./metadata"; import { loadCatalogForSync, resetBundledCatalogCacheForTests } from "./bundled"; import { applyCatalogModelMetadata, applyReasoningLevels, catalogEntryEfforts, clampCatalogModelsToCodexSupport, ensureGpt56ReasoningLevels, ensureUltraReasoningLevel, isGpt56NativeSlug } from "./effort"; @@ -132,6 +133,17 @@ export function finishUpstreamNativeEntry(clone: RawEntry, priority: number): Ra return ensureStrictCatalogFields(normalizeServiceTiers(clone)); } +/** + * Normalize a live native Codex row without synthesizing reasoning tiers. + * The upstream response is already filtered for the requesting Codex version. + */ +export function finishAuthoritativeNativeEntry(entry: RawEntry, priority: number): RawEntry { + const clone = structuredClone(entry); + if (priority !== 9) clone.priority = priority; + applyNativeOpenAiContextOverride(clone); + return ensureStrictCatalogFields(normalizeServiceTiers(clone)); +} + export function isExactComboCatalogModel( model: CatalogModel | undefined, exactComboSlugs: ReadonlySet, @@ -235,6 +247,7 @@ export function buildCatalogEntries( wsEnabled = false, multiAgentMode: MultiAgentMode = "default", exactComboSlugs: ReadonlySet = new Set(), + authoritativeNativeEntries: ReadonlyMap = new Map(), ): RawEntry[] { // Codex's models-manager sorts by `priority` ASC and advertises the first 5 picker-visible // models to spawn_agent (sort_by_key(priority) + MAX_MODEL_OVERRIDES_IN_SPAWN_AGENT=5). Catalog @@ -247,7 +260,10 @@ export function buildCatalogEntries( .filter(model => model.provider === COMBO_NAMESPACE) .map(catalogModelSlug)); for (const slug of gptSlugs) { - const e = deriveEntry(template, slug, "OpenAI native model (Codex OAuth passthrough).", 9); + const authoritative = authoritativeNativeEntries.get(slug); + const e = authoritative + ? finishAuthoritativeNativeEntry(authoritative, 9) + : deriveEntry(template, slug, "OpenAI native model (Codex OAuth passthrough).", 9); if (rank.has(slug)) e.priority = rank.get(slug)!; out.push(e); } @@ -332,6 +348,7 @@ export function mergeCatalogEntriesForSync( exactComboSlugs: ReadonlySet = new Set(), hasPhysicalComboProvider = false, includeNativeOpenAi = true, + authoritativeNativeSlugs: ReadonlySet = new Set(), ): RawEntry[] { const rank = new Map(featured.map((slug, i) => [slug, i] as const)); const native = includeNativeOpenAi @@ -340,7 +357,10 @@ export function mergeCatalogEntriesForSync( && !(m.slug as string).includes("/") && m.owned_by !== COMBO_NAMESPACE && !goIds.has(m.slug as string) - && !isUnsupportedOpenAiNativeSlug(m.slug as string)) + && ( + !isUnsupportedOpenAiNativeSlug(m.slug as string) + || authoritativeNativeSlugs.has(m.slug as string) + )) .map(m => { const slug = m.slug as string; // Featured models rank first (rank order); non-featured natives are pushed below the featured @@ -369,7 +389,9 @@ export function mergeCatalogEntriesForSync( const preserved = normalizeServiceTiers({ ...m, priority }); // Older natives kept from disk still need the mock top tiers (max + ultra always // for subagent max spawns; wire-clamped to the model's real top rung). - if (!isGpt56NativeSlug(slug)) ensureUltraReasoningLevel(preserved); + if (!authoritativeNativeSlugs.has(slug) && !isGpt56NativeSlug(slug)) { + ensureUltraReasoningLevel(preserved); + } return preserved; }) : []; @@ -432,6 +454,8 @@ export function mergeCatalogEntriesForSync( const normalized = normalizeServiceTiers(m); applyNativeOpenAiContextOverride(normalized); const exactCombo = typeof m.slug === "string" && exactComboSlugs.has(m.slug); + const authoritativeNative = typeof m.slug === "string" + && authoritativeNativeSlugs.has(m.slug); const e = ensureStrictCatalogFields(normalized, { preserveExactInputModalities: exactCombo, isRouted: finalRoutedEntries.includes(m), @@ -439,7 +463,7 @@ export function mergeCatalogEntriesForSync( // Mock-max universality (260709): preserved routed entries from disk may predate // the max rung — ensure it here so subagent max spawns validate on every // reasoning-capable entry. max only: 5.6 exact ladders (luna: no ultra) stay intact. - if (!exactCombo) { + if (!exactCombo && !authoritativeNative) { const levels = Array.isArray(e.supported_reasoning_levels) ? e.supported_reasoning_levels as Array<{ effort?: string }> : []; @@ -459,16 +483,27 @@ export function mergeCatalogEntriesForSync( }); // Native enable/disable (single choke point: bare slugs in `disabledModels`). Runs as the // LAST pass so the upstream-upgrade branch above can never clobber a hide flag back to list. - return applyMultiAgentMode(applyNativeVisibility(mergedEntries, disabledNative), multiAgentMode); + return applyMultiAgentMode( + applyNativeVisibility(mergedEntries, disabledNative, authoritativeNativeSlugs), + multiAgentMode, + ); } -export async function syncCatalogModels(config: OcxConfig): Promise<{ +export interface SyncCatalogModelsOptions { + /** Explicit build target. Managed lifecycle callers use the canonical OCX catalog. */ + catalogPath?: string; +} + +export async function syncCatalogModels( + config: OcxConfig, + options: SyncCatalogModelsOptions = {}, +): Promise<{ added: number; path: string; catalogWritten: boolean; comboOmissions: ComboCatalogOmission[]; }> { - const catalogPath = readCodexCatalogPath(); + const catalogPath = options.catalogPath ?? readCodexCatalogPath(); const catalog = loadCatalogForSync(catalogPath); if (!catalog) return { added: 0, path: catalogPath, catalogWritten: false, comboOmissions: [] }; @@ -516,15 +551,43 @@ export async function syncCatalogModels(config: OcxConfig): Promise<{ // bare gpt-* rows that hard-404 via NoEnabledOpenAiProviderError. Keep natives when no // providers are configured yet (fresh install / catalog bootstrap tests). const includeNativeOpenAi = enabledProviders.length === 0 || hasCanonicalOpenai; - catalog.models = mergeCatalogEntriesForSync(catalog.models ?? [], goEntries, baseline, featured, wsEnabled, goIds, template, disabledNativeSlugs(config), gatheredProviderNames, multiAgentMode, exactComboSlugs, hasPhysicalComboProvider, includeNativeOpenAi); + const liveNative = includeNativeOpenAi + ? await discoverNativeOpenAiCatalog(config) + : { models: [], clientVersion: null }; + const authoritativeNativeSlugs = new Set( + liveNative.models.flatMap(model => + typeof model.slug === "string" && !model.slug.includes("/") ? [model.slug] : [] + ), + ); + const catalogModels = mergeDiscoveredNativeCatalogRows( + catalog.models ?? [], + liveNative.models, + ); + catalog.models = mergeCatalogEntriesForSync( + catalogModels, + goEntries, + baseline, + featured, + wsEnabled, + goIds, + template, + disabledNativeSlugs(config), + gatheredProviderNames, + multiAgentMode, + exactComboSlugs, + hasPhysicalComboProvider, + includeNativeOpenAi, + authoritativeNativeSlugs, + ); clampCatalogModelsToCodexSupport(catalog.models); atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); return { added: goEntries.length, path: catalogPath, catalogWritten: true, comboOmissions }; } -export function restoreCodexCatalog(): { removed: number; kept: number; path: string } { - const catalogPath = readCodexCatalogPath(); +export function restoreCodexCatalog( + catalogPath: string = readCodexCatalogPath(), +): { removed: number; kept: number; path: string } { const catalog = readCatalog(catalogPath); if (!catalog || !Array.isArray(catalog.models)) return { removed: 0, kept: 0, path: catalogPath }; const backup = readCatalogBackup(catalogPath); @@ -552,9 +615,10 @@ export function restoreCodexCatalog(): { removed: number; kept: number; path: st } /** Force Codex's models_cache stale from the on-disk catalog. Returns whether a cache write occurred. */ -export function invalidateCodexModelsCache(): boolean { +export function invalidateCodexModelsCache( + catalogPath: string = readCodexCatalogPath(), +): boolean { try { - const catalogPath = readCodexCatalogPath(); if (!existsSync(catalogPath)) return false; const catalog = JSON.parse(readFileSync(catalogPath, "utf8")); const models = catalog.models ?? catalog; diff --git a/src/codex/inject.ts b/src/codex/inject.ts index 0e5b09fd..c74be49e 100644 --- a/src/codex/inject.ts +++ b/src/codex/inject.ts @@ -37,6 +37,22 @@ export function currentExternalCodexModelProvider(): string | null { return externalCodexModelProvider(readFileSync(CODEX_CONFIG_PATH, "utf8")); } +/** A root base URL without the OCX ownership marker belongs to another config manager or the user. */ +export function hasUserOwnedRootOpenaiBaseUrl(content: string): boolean { + return rootTomlString(content, "openai_base_url") !== null + && !hasInjectedOpenaiBaseUrl(content); +} + +/* Read-only ownership probe used before sync so catalog writes cannot precede routing checks. */ +export function currentUserOwnedRootOpenaiBaseUrl(): boolean { + if (!existsSync(CODEX_CONFIG_PATH)) return false; + try { + return hasUserOwnedRootOpenaiBaseUrl(readFileSync(CODEX_CONFIG_PATH, "utf8")); + } catch { + return false; + } +} + /** * Detect the file's dominant line ending. Every transform in this module is LF-pure * (split("\n") + hard "\n" joins), so CRLF configs (Windows-edited config.toml) are @@ -354,24 +370,23 @@ function setRootModelProvider(content: string): string { return lines.join("\n"); } -function readRootModelCatalogPath(content: string): string | null { - return readRootTomlString(content, "model_catalog_json"); -} - +/** + * While OpenCodex owns active Codex routing, it also owns the active root model catalog pointer. + * + * A pre-existing user catalog is preserved by the injection journal and restored on stop/eject, + * but it must not remain the runtime source of truth while the proxy is active. Leaving a second + * merged catalog in place can silently strip newly rolled-out native OpenAI metadata and force + * Codex onto generic fallback model capabilities. + */ export function setRootModelCatalogPath(content: string, catalogPath: string): string { const lines = content.split("\n"); const firstTable = lines.findIndex(l => /^\s*\[/.test(l)); const key = `model_catalog_json = ${tomlString(catalogPath)}`; const rootEnd = firstTable === -1 ? lines.length : firstTable; for (let i = 0; i < rootEnd; i++) { - const m = lines[i].match(/^\s*model_catalog_json\s*=\s*("(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*')\s*$/); - if (!m) continue; - const existing = parseTomlString(m[1]); - if (isOpencodexCatalogPath(existing)) { - lines[i] = key; - return lines.join("\n"); - } - return content; + if (!/^\s*model_catalog_json\s*=/.test(lines[i])) continue; + lines[i] = key; + return lines.join("\n"); } if (firstTable === -1) { return content.replace(/\n+$/, "") + "\n" + key + "\n"; @@ -382,6 +397,16 @@ export function setRootModelCatalogPath(content: string, catalogPath: string): s return lines.join("\n"); } +/** Remove any root catalog override while OCX is active without touching table-scoped values. */ +export function stripRootModelCatalogPath(content: string): string { + const lines = content.split("\n"); + const firstTable = lines.findIndex(l => /^\s*\[/.test(l)); + const rootEnd = firstTable === -1 ? lines.length : firstTable; + return lines + .filter((line, index) => index >= rootEnd || !/^\s*model_catalog_json\s*=/.test(line)) + .join("\n"); +} + function removeProfileSection(content: string): string { const lines = content.split("\n"); const filtered: string[] = []; @@ -470,16 +495,48 @@ export function buildProfileFile(port: number, catalogPath?: string | null, supp return lines.join("\n"); } -export function chooseCatalogPathForInjection(content: string, requested?: string | null): string | null { - if (requested !== undefined) return requested; +function isBareNativeCodexModelId(model: string | null): model is string { + return model !== null + && !model.includes("/") + && /^(?:gpt-|codex-)/i.test(model); +} - const existing = readRootModelCatalogPath(content); - if (existing) { - const resolved = resolveCodexConfigPath(existing); - if (!isOpencodexCatalogPath(resolved) || existsSync(resolved)) return existing; +function catalogContainsModelSlug(catalogPath: string, slug: string): boolean { + try { + const parsed = JSON.parse( + readFileSync(resolveCodexConfigPath(catalogPath), "utf8"), + ) as { models?: unknown }; + return Array.isArray(parsed.models) + && parsed.models.some(model => + model !== null + && typeof model === "object" + && (model as { slug?: unknown }).slug === slug + ); + } catch { + return false; } +} - return existsSync(DEFAULT_CATALOG_PATH) ? DEFAULT_CATALOG_PATH : null; +/** + * Choose the one active OCX catalog, but never let that catalog make a selected native model + * disappear. A missing native row would make Codex synthesize generic fallback metadata (including + * the wrong context/tool capabilities); leaving model_catalog_json unset lets native Codex own the + * model metadata instead. + */ +export function chooseCatalogPathForInjection(content: string, requested?: string | null): string | null { + const candidate = requested !== undefined + ? requested + : (existsSync(DEFAULT_CATALOG_PATH) ? DEFAULT_CATALOG_PATH : null); + if (!candidate) return null; + + const selectedModel = readRootTomlString(content, "model"); + if ( + isBareNativeCodexModelId(selectedModel) + && !catalogContainsModelSlug(candidate, selectedModel) + ) { + return null; + } + return candidate; } export interface CodexInjectResult { @@ -513,6 +570,27 @@ export async function injectCodexConfig(port: number, config?: OcxConfig, option }; } + if (hasUserOwnedRootOpenaiBaseUrl(rawContent)) { + // A user/external manager owns the routing surface. Do not half-own the same config. Keep an + // existing journal: injected-state hashes prevent replay over user edits while retaining the + // original pre-OCX baseline for reversible cleanup of still-owned state. + return { + success: true, + message: `⚠️ Codex routing and catalog NOT injected: config.toml has a user-owned root openai_base_url.\n` + + ` OpenCodex leaves both routing and model_catalog_json untouched to avoid split ownership.\n` + + ` To let OpenCodex manage native Codex plus routed providers together, remove that root override and rerun 'ocx start'.`, + }; + } + + if (options.catalogPath && !isOpencodexCatalogPath(options.catalogPath)) { + return { + success: false, + message: `Codex config injection refused: OpenCodex owns the active merged catalog while routing is managed, ` + + `but the requested catalog is not the canonical opencodex-catalog.json: ${options.catalogPath}. ` + + `No files were changed; rebuild the catalog through 'ocx sync'.`, + }; + } + // Marker-owned native defaults are OpenCodex residue, never part of the // user's journal baseline. Clean them before either snapshotting or adding a // root routing key: inserting that key ahead of a marker-owned first table @@ -554,8 +632,17 @@ export async function injectCodexConfig(port: number, config?: OcxConfig, option content = normalizeServiceTier(content); content = ensureFastModeFeature(content); + const candidateCatalogPath = options.catalogPath !== undefined + ? options.catalogPath + : (existsSync(DEFAULT_CATALOG_PATH) ? DEFAULT_CATALOG_PATH : null); + const selectedModel = readRootTomlString(content, "model"); const catalogPath = chooseCatalogPathForInjection(content, options.catalogPath); - content = catalogPath ? setRootModelCatalogPath(content, catalogPath) : stripOpencodexCatalogPath(content); + const nativeCatalogFallback = candidateCatalogPath !== null + && catalogPath === null + && isBareNativeCodexModelId(selectedModel); + // No safe OCX catalog means "native catalog", never "whatever custom root catalog happened to be there". + // The original user value remains in the journal and is restored on stop/eject. + content = catalogPath ? setRootModelCatalogPath(content, catalogPath) : stripRootModelCatalogPath(content); const legacyMode = shouldInjectApiAuthHeader(config); let keptUserBaseUrl = false; @@ -614,7 +701,9 @@ export async function injectCodexConfig(port: number, config?: OcxConfig, option const catalogMessage = catalogPath ? ` Codex model catalog: ${catalogPath}\n` - : ` Codex model catalog not injected because no opencodex catalog file exists yet.\n`; + : nativeCatalogFallback + ? ` ⚠️ Codex model catalog: native fallback because the managed OCX catalog does not contain selected native model ${tomlString(selectedModel!)}.\n` + : ` Codex model catalog not injected because no safe opencodex catalog file exists yet.\n`; const migratedRows = (history.rows ?? 0) + ("ejectedRows" in history ? history.ejectedRows ?? 0 : 0); const historyMessage = config?.syncResumeHistory === false ? ` Codex resume history: left unchanged (syncResumeHistory=false).\n` @@ -781,7 +870,9 @@ export function restoreNativeCodex(): { success: boolean; message: string } { const cfg = journal.configRestored ? { success: true, message: "Codex config restored from opencodex journal." } : removeCodexConfig({ preserveProfile: journal.profileRestored || journal.profileChanged }); - const cat = restoreCodexCatalog(); + // Never follow a catalog pointer that the journal just restored. Only the canonical OCX + // catalog is ours to rewrite; a restored user catalog is immutable here. + const cat = restoreCodexCatalog(DEFAULT_CATALOG_PATH); // Design B (loopback) steady state: threads are already tagged openai, so prove the // no-op with a readonly probe instead of write-opening a DB the Codex app may hold // (Windows: WAL writer lock -> seconds of stalling + a false warning on every stop). @@ -806,4 +897,4 @@ export function restoreNativeCodex(): { success: boolean; message: string } { export function getCodexConfigPath(): string { return CODEX_CONFIG_PATH; -} +} \ No newline at end of file diff --git a/src/codex/refresh.ts b/src/codex/refresh.ts index 1cb582e8..46c012e8 100644 --- a/src/codex/refresh.ts +++ b/src/codex/refresh.ts @@ -26,6 +26,11 @@ const defaultDeps: RefreshDeps = { existsSync, }; +export interface CodexCatalogRefreshOptions { + /** Managed lifecycle callers pass the canonical OCX catalog path explicitly. */ + catalogPath?: string; +} + export function syncCodexModelsCacheFromCatalog(catalogPath: string): void { const content = readFileSync(catalogPath, "utf8"); atomicWriteFile(CODEX_MODELS_CACHE_PATH, content); @@ -40,14 +45,15 @@ export function syncCodexModelsCacheFromCatalog(catalogPath: string): void { export async function refreshCodexModelCatalog( config: OcxConfig, deps: RefreshDeps = defaultDeps, + options: CodexCatalogRefreshOptions = {}, ): Promise { - const result = await deps.syncCatalogModels(config); + const result = await deps.syncCatalogModels(config, options); const catalogExists = deps.existsSync(result.path); const catalogWritten = result.catalogWritten === true; const comboOmissions = result.comboOmissions ?? []; if (!catalogExists) { return { ...result, catalogExists, catalogWritten: false, cacheSynced: false, comboOmissions }; } - const cacheSynced = deps.invalidateCodexModelsCache(); + const cacheSynced = deps.invalidateCodexModelsCache(result.path); return { ...result, catalogExists, catalogWritten, cacheSynced, comboOmissions }; } diff --git a/src/codex/sync.ts b/src/codex/sync.ts index 2ae208b1..2cca6120 100644 --- a/src/codex/sync.ts +++ b/src/codex/sync.ts @@ -1,4 +1,8 @@ -import { currentExternalCodexModelProvider, injectCodexConfig } from "./inject"; +import { + currentExternalCodexModelProvider, + currentUserOwnedRootOpenaiBaseUrl, + injectCodexConfig, +} from "./inject"; import { printProjectCodexConfigWarnings, groupProjectCodexConfigWarningsByPath, type ProjectCodexConfigWarning } from "./project-config-warnings"; import { refreshCodexModelCatalog } from "./refresh"; import { applyProxyEnv, loadConfig } from "../config"; @@ -6,6 +10,7 @@ import type { OcxConfig } from "../types"; import { collectOrcaCodexHomeDiagnostic } from "./home"; import { summarizeComboCatalogOmissions, type ComboCatalogOmission } from "./catalog/aggregation"; import { syncExternalOcxCatalog } from "./external-ocx-catalog"; +import { activeDefaultCatalogPath } from "./catalog/parsing"; export interface CodexSyncResult { ok: boolean; @@ -26,6 +31,7 @@ interface CodexSyncDeps { refreshCodexModelCatalog: typeof refreshCodexModelCatalog; injectCodexConfig: typeof injectCodexConfig; currentExternalCodexModelProvider?: typeof currentExternalCodexModelProvider; + currentUserOwnedRootOpenaiBaseUrl?: typeof currentUserOwnedRootOpenaiBaseUrl; collectCodexHomeDiagnostic?: typeof collectOrcaCodexHomeDiagnostic; syncExternalOcxCatalog?: typeof syncExternalOcxCatalog; } @@ -101,6 +107,25 @@ export async function syncModelsToCodex( }; } + const userOwnedBaseUrl = + (deps.currentUserOwnedRootOpenaiBaseUrl ?? currentUserOwnedRootOpenaiBaseUrl)(); + if (userOwnedBaseUrl) { + // Routing and catalog ownership are atomic. Never refresh a catalog before this check. + const result = await deps.injectCodexConfig(p, config, {}); + log?.log(result.message); + reportCodexHomeTarget(log, deps.collectCodexHomeDiagnostic ?? collectOrcaCodexHomeDiagnostic); + return { + ok: result.success, + added: 0, + catalogPath: null, + catalogExists: false, + catalogWritten: false, + cacheSynced: false, + message: result.message, + ...(result.nativeSubagentDefaultsWarning ? { nativeSubagentDefaultsWarning: result.nativeSubagentDefaultsWarning } : {}), + }; + } + applyProxyEnv(config); // `ocx ensure`/`ocx sync` fetch provider models outside the server process let added = 0; let catalogPath: string | null = null; @@ -110,21 +135,30 @@ export async function syncModelsToCodex( let cacheSynced = false; let warning: string | undefined; let comboOmissions: ComboCatalogOmission[] = []; + const canonicalCatalogPath = activeDefaultCatalogPath(); try { - const cat = await deps.refreshCodexModelCatalog(config); + const cat = await deps.refreshCodexModelCatalog( + config, + undefined, + { catalogPath: canonicalCatalogPath }, + ); added = cat.added; catalogExists = cat.catalogExists; catalogWritten = cat.catalogWritten; cacheSynced = cat.cacheSynced; - catalogPathForInjection = cat.catalogExists ? cat.path : null; + // Existence alone is not authority. Only advertise a catalog materialized by this sync. + catalogPathForInjection = cat.catalogExists && cat.catalogWritten ? cat.path : null; catalogPath = catalogPathForInjection; comboOmissions = cat.comboOmissions ?? []; - if (cat.added > 0) { + if (cat.added > 0 && cat.catalogWritten) { log?.log(` + ${cat.added} models appended to Codex catalog (${cat.path})`); } else if (!cat.catalogExists) { warning = "catalog sync skipped: no Codex catalog source found; keeping Codex's native catalog."; log?.error(warning); + } else if (!cat.catalogWritten) { + warning = "catalog sync did not materialize the managed OCX catalog; keeping Codex's native catalog."; + log?.error(warning); } if (comboOmissions.length > 0) { // Individual omission lines already went through console.warn during gather; @@ -135,6 +169,8 @@ export async function syncModelsToCodex( } } catch (e) { warning = `catalog sync skipped: ${e instanceof Error ? e.message : String(e)}`; + // Explicit null prevents inject from reusing a stale managed catalog after refresh failed. + catalogPathForInjection = null; log?.error(warning); } diff --git a/src/providers/free-directory.ts b/src/providers/free-directory.ts index ee3651bd..1db10b4b 100644 --- a/src/providers/free-directory.ts +++ b/src/providers/free-directory.ts @@ -202,13 +202,13 @@ const CONNECTABLE: Record = { }, ), cohere: openAi( - "https://api.cohere.com/compatibility/v1", + "https://api.cohere.ai/compatibility/v1", "https://dashboard.cohere.com/api-keys", { supportLevel: "supported", verification: "official", documentationUrl: "https://docs.cohere.com/reference/list-models", - modelsUrl: "https://api.cohere.com/compatibility/v1/models", + modelsUrl: "https://api.cohere.ai/compatibility/v1/models", }, ), friendliai: openAi( @@ -677,7 +677,7 @@ const CONNECTABLE: Record = { verification: "official", }), nebius: openAi( - "https://api.tokenfactory.nebius.com/v1", + "https://api.studio.nebius.com/v1", "https://studio.nebius.com", { verification: "official" }, ), diff --git a/src/providers/registry.ts b/src/providers/registry.ts index 7080a288..260c43cc 100644 --- a/src/providers/registry.ts +++ b/src/providers/registry.ts @@ -1439,6 +1439,70 @@ export const PROVIDER_REGISTRY: readonly ProviderRegistryEntry[] = [ dashboardUrl: "https://cloud.cerebras.ai/platform/apikeys", defaultModel: "gpt-oss-120b", }, + // Cohere's OpenAI Compatibility API supports the OpenAI SDK at this base URL. + // Trial/evaluation API keys are free but rate-limited. + // Evidence: https://docs.cohere.com/docs/compatibility-api + // https://docs.cohere.com/v2/docs/rate-limits + { + id: "cohere", + label: "Cohere", + baseUrl: "https://api.cohere.ai/compatibility/v1", + adapter: "openai-chat", + authKind: "key", + dashboardUrl: "https://dashboard.cohere.com/api-keys", + freeTier: true, + liveModels: true, + preserveCustomDestination: true, + note: "OpenAI Compatibility API. Free evaluation keys are rate-limited and intended for evaluation/prototyping.", + }, + // Friendli Model API is OpenAI-compatible and publishes GET /models on the same serverless base. + // Evidence: https://learn.friendli.ai/articles/4213111023-q10-1-how-do-i-connect-friendliai-with-litellm-or-other-openai-compatible-client + { + id: "friendliai", + label: "FriendliAI", + baseUrl: "https://api.friendli.ai/serverless/v1", + adapter: "openai-chat", + authKind: "key", + dashboardUrl: "https://suite.friendli.ai", + liveModels: true, + preserveCustomDestination: true, + }, + // SambaCloud documents an OpenAI-compatible API at api.sambanova.ai/v1. + // Evidence: https://cloud.sambanova.ai/dashboard + { + id: "sambanova", + label: "SambaNova", + baseUrl: "https://api.sambanova.ai/v1", + adapter: "openai-chat", + authKind: "key", + dashboardUrl: "https://cloud.sambanova.ai/apis", + liveModels: true, + preserveCustomDestination: true, + }, + // Nebius AI Studio's current OpenAI-compatible OAS publishes chat, responses and GET /v1/models. + // Evidence: https://api.studio.nebius.com/docs + { + id: "nebius", + label: "Nebius AI Studio", + baseUrl: "https://api.studio.nebius.com/v1", + adapter: "openai-chat", + authKind: "key", + dashboardUrl: "https://studio.nebius.com", + liveModels: true, + preserveCustomDestination: true, + }, + // Novita's current serverless docs use the OpenAI-compatible /openai/v1 path. + // Evidence: https://blogs.novita.ai/glm-5-1-api-novita-ai/ + { + id: "novita", + label: "Novita AI", + baseUrl: "https://api.novita.ai/openai/v1", + adapter: "openai-chat", + authKind: "key", + dashboardUrl: "https://novita.ai/settings/key-management", + liveModels: true, + preserveCustomDestination: true, + }, // FREEZE 2026-07-10: exact serverless ids remain auth-gated/unverified. Evidence: devlog/_plan/260710_provider_hardening/003_research_aggregators.md. { id: "together", diff --git a/src/server/index.ts b/src/server/index.ts index 4a766268..a35625d4 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -835,8 +835,27 @@ export function startServer(port?: number) { return jsonResponse({ data }, 200, req, config); } if (url.searchParams.has("client_version")) { - // Codex client → Codex catalog shape: native gpt + namespaced routed models, - // cloned from a native template so required fields (base_instructions, etc.) are present. + // Codex client → Codex catalog shape. Ask the same upstream catalog the client would + // use, but authenticate with OCX's effective account so a stale Desktop main login + // cannot hide models available to the selected pool account. + const { discoverNativeOpenAiCatalog } = + await import("../codex/catalog/native-discovery"); + const requestedClientVersion = url.searchParams.get("client_version")?.trim() || null; + const liveNative = await discoverNativeOpenAiCatalog(config, { + resolveClientVersion: () => requestedClientVersion, + }); + const authoritativeNativeEntries = + new Map(); + for (const model of liveNative.models) { + if (typeof model.slug === "string" && !model.slug.includes("/")) { + authoritativeNativeEntries.set(model.slug, model); + } + } + const authoritativeNativeSlugs = new Set(authoritativeNativeEntries.keys()); + const codexNativeSlugs = [ + ...new Set([...nativeSlugs, ...authoritativeNativeSlugs]), + ]; + // Pass the subagent picks so featured models lead by priority (matches the on-disk file). // Disabled natives stay in the catalog shape with visibility "hide" (mirrors the // on-disk sync; codex-rs keeps them out of the picker itself). @@ -846,18 +865,20 @@ export function startServer(port?: number) { : "default"; const entries = buildCatalogEntries( loadCatalogTemplate(), - nativeSlugs, + codexNativeSlugs, goOrdered, config.subagentModels, websocketsEnabled(config), maMode as "v1" | "default" | "v2", exactComboCatalogSlugs(config), + authoritativeNativeEntries, ); return jsonResponse( { models: applyNativeVisibility( entries, disabledNativeSlugs(config), + authoritativeNativeSlugs, ), }, 200, diff --git a/structure/02_config-and-codex-home.md b/structure/02_config-and-codex-home.md index 09432b7c..f99adab9 100644 --- a/structure/02_config-and-codex-home.md +++ b/structure/02_config-and-codex-home.md @@ -55,21 +55,42 @@ are consumed incrementally and at most 512 stale files are attempted per process ## Config injection -`src/codex/inject.ts` inserts root-level keys and an opencodex provider table: +The default loopback install is an additive native-Codex integration. OpenCodex keeps the built-in +`openai` provider identity and the user's ordinary ChatGPT/Codex login, then points that native +provider at the local proxy with the supported root override: ```toml -model_provider = "opencodex" +openai_base_url = "http://127.0.0.1:10100/v1" model_catalog_json = "/absolute/path/to/opencodex-catalog.json" - -[model_providers.opencodex] -name = "OpenCodex Proxy" -base_url = "http://127.0.0.1:10100/v1" -wire_api = "responses" -requires_openai_auth = true ``` +The merged catalog is `$CODEX_HOME/opencodex-catalog.json`. While OpenCodex owns active routing, +that path is the only supported root `model_catalog_json`: a pre-existing custom catalog pointer is +journaled for restore but replaced for the active OCX session. Catalog refresh writes the canonical +path directly **before** injection changes the root pointer, so a user catalog is never mutated as an +intermediate OCX build target. Cache invalidation reads the exact path written by that refresh. +Restore cleanup also targets only the canonical OCX catalog after journal restore, never the restored +user catalog. Parallel merge artifacts such as `~/.codex/model-catalogs/native-plus-ocx.json` are +configuration drift and must not remain active. + +If the managed catalog cannot be materialized, injection removes the active root catalog override +instead of leaving an unrelated or stale custom catalog in control. That deliberately falls back to +Codex's native model metadata. The pre-OCX config is still preserved in the journal and is restored +on stop/eject. + +Native bare OpenAI rows in the managed catalog are account/client-authoritative: they come from the +native Codex model-discovery endpoint for the installed client and effective ChatGPT/Codex account. +Do not synthesize newly rolled-out native rows from a static template when a live row is available, +and do not drop native capability fields such as `context_window`, reasoning ladders, +`supports_search_tool`, `tool_mode`, or `use_responses_lite`. + +Non-loopback binds still use the explicit `model_providers.opencodex` table because Codex's built-in +provider cannot carry the required admission-token headers. That transport exception does not make +OCX a second authority for native model metadata. + Root TOML keys must be written before the first `[table]`. Re-injection strips stale opencodex -blocks, stale root context-window overrides, and stale opencodex catalog paths before rewriting. +blocks, stale root context-window overrides, and competing root catalog pointers before rewriting the +managed state. Native Codex sub-agent defaults are a separate, explicit opt-in. When `syncCodexSubagentDefaults` is true and `injectionModel` is set, injection writes marker-owned @@ -81,8 +102,8 @@ restore must preserve later user edits while stripping those managed values. If the root config selects a provider other than `openai` or `opencodex`, injection must leave the config byte-for-byte unchanged and skip profile creation/updates and history migration. External provider managers own that routing configuration, and replacing their provider id can hide -otherwise intact Codex sessions. This ownership check must run before catalog/cache refresh, -journal creation, and the background history migration guardian. +otherwise intact Codex sessions. A user-owned root `openai_base_url` is the same kind of ownership +boundary for loopback routing. `supports_websockets = true` is appended only when `websocketsEnabled(config)` returns true. @@ -117,4 +138,4 @@ uninstall with their exact paths. Legacy nonempty config directories are deliberately not retroactively claimed. If either ownership file is missing, malformed, or bound to another root, uninstall refuses config deletion and reports -the residual directory for manual review; there is no recursive-delete fallback. +the residual directory for manual review; there is no recursive-delete fallback. \ No newline at end of file diff --git a/structure/03_catalog-and-subagents.md b/structure/03_catalog-and-subagents.md index e422d0d5..860acd52 100644 --- a/structure/03_catalog-and-subagents.md +++ b/structure/03_catalog-and-subagents.md @@ -4,8 +4,12 @@ `src/codex/catalog.ts` builds a shared Codex-shaped catalog for CLI, TUI, App, and SDK. It: -- preserves native OpenAI entries from the live catalog or static fallback, and emits - gpt-5.6 natives from the pinned upstream models.json snapshot +- preserves native OpenAI entries from authoritative live Codex discovery for the installed + client/effective ChatGPT account, falling back only when live discovery is unavailable; newly + rolled-out bare native slugs are admitted without waiting for a static OpenCodex whitelist; +- preserves authoritative native capability metadata unchanged, including context-window, + reasoning, search/tool-exposure, Responses-lite, modalities, and visibility fields; +- keeps the pinned upstream models.json snapshot only as fallback metadata for known static natives (`src/codex/data/upstream-models.json` — exact per-slug ladders: luna has no ultra); - clones a native template for routed `provider/model` entries; - forces strict Codex catalog fields required by the current parser; @@ -35,13 +39,24 @@ display name use `provider/model`. ## Native passthrough -Native bare OpenAI entries form one `openai` group. The provider's Pool(default)/Direct option -changes account selection without changing those ids; `openai-apikey/` creates the separate -API-key identity. The API GPT-5.6 rows use 1,050,000 context / 922,000 max input; their `*-pro` virtual rows -rewrite to the base upstream model with `reasoning.mode: "pro"` while public state keeps the virtual -slug. Native OpenAI entries remain available for ChatGPT passthrough. Routed non-OpenAI models must not -inherit native-only service tier or WebSocket metadata unless the user explicitly enables that -capability. Detailed invariants live in [`08_openai-provider-tiers.md`](08_openai-provider-tiers.md). +Native bare OpenAI entries form one `openai` group and remain native even while OCX is active. On +loopback installs Codex keeps its built-in `openai` provider id and ordinary ChatGPT/Codex login; +OCX is the transport/routing layer, not a replacement identity. The provider's Pool(default)/Direct +option changes account selection without changing those ids; `openai-apikey/` creates the +separate API-key identity. + +The active merged catalog is `$CODEX_HOME/opencodex-catalog.json`. Do not create a second +"native + OCX" merge file or point active Codex routing at a competing root `model_catalog_json`. +Live native discovery is the authority for bare OpenAI rows, including future model slugs. If a live +row says that a model supports deferred/search-tool exposure, Responses-lite, a larger context +window, or a specific reasoning ladder, those fields must survive catalog assembly unchanged; a +static template must not downgrade them to generic Codex fallback metadata. + +The API GPT-5.6 rows use 1,050,000 context / 922,000 max input; their `*-pro` virtual rows rewrite +to the base upstream model with `reasoning.mode: "pro"` while public state keeps the virtual slug. +Routed non-OpenAI models must not inherit native-only service tier or WebSocket metadata unless the +user explicitly enables that capability. Detailed invariants live in +[`08_openai-provider-tiers.md`](08_openai-provider-tiers.md). ## Multi-agent surface mode (3-state) @@ -116,4 +131,4 @@ identity-resolved `claude-*` or `anthropic-*` model while native passthrough is claims and `nativePassthrough:false` restore the guard. The guard avoids creating oversized skill messages before the proxy can intervene; inbound elision remains the fallback if a client still sends a blocked bundle. An explicit empty list disables both routed-model -behaviors. +behaviors. \ No newline at end of file diff --git a/tests/claude-models-discovery.test.ts b/tests/claude-models-discovery.test.ts index cf7958a4..6fc5501e 100644 --- a/tests/claude-models-discovery.test.ts +++ b/tests/claude-models-discovery.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, expect, setDefaultTimeout, test } from "bun:test"; -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../src/config"; @@ -49,6 +49,23 @@ function configWithStaticModels(claudeCode?: OcxConfig["claudeCode"]): OcxConfig } as OcxConfig; } +function configWithNativeOpenAi(): OcxConfig { + return { + port: 0, + defaultProvider: "openai", + openaiProviderTierVersion: 2, + codexAccountPools: false, + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", + codexAccountMode: "direct", + }, + }, + }; +} + test("anthropic-version header flips /v1/models to the discovery contract", async () => { saveConfig(configWithStaticModels()); const server = startServer(0); @@ -158,3 +175,75 @@ test("OpenAI list shape and Codex catalog shape stay unchanged", async () => { server.stop(true); } }); + +test("Codex client catalog includes live native metadata for its client version", async () => { + if (!isolatedCodexHome) throw new Error("isolated Codex home not installed"); + writeFileSync( + join(isolatedCodexHome.path, "auth.json"), + JSON.stringify({ + tokens: { + access_token: "native-test-access", + account_id: "native-test-account", + }, + }), + "utf8", + ); + saveConfig(configWithNativeOpenAi()); + + const server = startServer(0); + const originalFetch = globalThis.fetch; + let upstreamRequest: { url: string; headers: Headers } | null = null; + const mockFetch: typeof fetch = async (input, init) => { + const target = String(input); + if (target.startsWith("https://chatgpt.com/backend-api/codex/models")) { + upstreamRequest = { url: target, headers: new Headers(init?.headers) }; + return new Response(JSON.stringify({ + models: [{ + slug: "gpt-6.1-sol", + display_name: "GPT-6.1 Sol", + description: "Live native model", + base_instructions: "Live native instructions.", + shell_type: "shell_command", + visibility: "list", + priority: 1, + supported_in_api: true, + default_reasoning_level: "high", + supported_reasoning_levels: [ + { effort: "high", description: "High reasoning" }, + ], + context_window: 400_000, + input_modalities: ["text", "image"], + }], + }), { status: 200, headers: { "content-type": "application/json" } }); + } + return originalFetch(input, init); + }; + globalThis.fetch = mockFetch; + + try { + const response = await originalFetch( + new URL("/v1/models?client_version=9.9.9", server.url), + ); + expect(response.status).toBe(200); + const json: { models?: Array> } = await response.json(); + const live = json.models?.find(model => model.slug === "gpt-6.1-sol"); + expect(live?.display_name).toBe("GPT-6.1 Sol"); + expect(live?.context_window).toBe(400_000); + expect(live?.supported_reasoning_levels).toEqual([ + { effort: "high", description: "High reasoning" }, + ]); + + if (!upstreamRequest) throw new Error("expected native upstream catalog request"); + const upstreamUrl = new URL(upstreamRequest.url); + expect(upstreamUrl.searchParams.get("client_version")).toBe("9.9.9"); + expect(upstreamRequest.headers.get("authorization")).toBe( + "Bearer native-test-access", + ); + expect(upstreamRequest.headers.get("chatgpt-account-id")).toBe( + "native-test-account", + ); + } finally { + globalThis.fetch = originalFetch; + server.stop(true); + } +}); diff --git a/tests/codex-inject.test.ts b/tests/codex-inject.test.ts index 3f65cdef..86891545 100644 --- a/tests/codex-inject.test.ts +++ b/tests/codex-inject.test.ts @@ -1,4 +1,7 @@ import { describe, expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { applyEol, buildOpenaiBaseUrlLine, @@ -6,11 +9,14 @@ import { buildProviderTableBlock, chooseCatalogPathForInjection, dominantEol, + hasUserOwnedRootOpenaiBaseUrl, + setRootModelCatalogPath, setRootOpenaiBaseUrl, shouldInjectApiAuthHeader, stripInjectedOpenaiBaseUrl, stripOpencodexConfig, stripRootContextWindowOverrides, + stripRootModelCatalogPath, } from "../src/codex/inject"; import { MANAGED_AGENTS_TABLE_MARKER, @@ -290,6 +296,18 @@ describe("Codex config injection", () => { } }); + test("treats an unmarked root openai_base_url as an external ownership boundary", () => { + expect(hasUserOwnedRootOpenaiBaseUrl( + 'openai_base_url = "https://my-own-gateway.example/v1"\n', + )).toBe(true); + + const managed = setRootOpenaiBaseUrl( + 'model = "gpt-6.1-sol"\n', + 10100, + ).content; + expect(hasUserOwnedRootOpenaiBaseUrl(managed)).toBe(false); + }); + test("honors an explicit unavailable catalog decision", () => { const path = chooseCatalogPathForInjection( 'model_catalog_json = "/tmp/opencodex-catalog.json"\n', @@ -299,6 +317,88 @@ describe("Codex config injection", () => { expect(path).toBeNull(); }); + test("falls back to native metadata when the managed catalog omits the selected native model", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-native-coverage-")); + const catalogPath = join(dir, "opencodex-catalog.json"); + try { + writeFileSync(catalogPath, JSON.stringify({ + models: [{ slug: "gpt-5.6-sol", context_window: 372000 }], + })); + + const config = [ + 'model = "gpt-6.1-sol"', + 'model_catalog_json = "/stale/native-plus-ocx.json"', + "", + ].join("\n"); + expect(chooseCatalogPathForInjection(config, catalogPath)).toBeNull(); + + writeFileSync(catalogPath, JSON.stringify({ + models: [{ + slug: "gpt-6.1-sol", + context_window: 400000, + supports_search_tool: true, + tool_mode: "code_mode_only", + }], + })); + expect(chooseCatalogPathForInjection(config, catalogPath)).toBe(catalogPath); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("does not misclassify a bare third-party selector as a native coverage guard", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-bare-routed-")); + const catalogPath = join(dir, "opencodex-catalog.json"); + try { + writeFileSync(catalogPath, JSON.stringify({ models: [] })); + expect( + chooseCatalogPathForInjection('model = "glm-5.2-fast-preview"\n', catalogPath), + ).toBe(catalogPath); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("managed routing replaces a pre-existing merged catalog with the canonical OCX catalog", () => { + const original = [ + 'model = "gpt-6.1-sol"', + 'model_catalog_json = "/home/joep/.codex/model-catalogs/native-plus-ocx.json"', + "", + "[features]", + "fast_mode = true", + "", + ].join("\n"); + + const injected = setRootModelCatalogPath( + original, + "/home/joep/.codex/opencodex-catalog.json", + ); + + expect(injected).toContain( + 'model_catalog_json = "/home/joep/.codex/opencodex-catalog.json"', + ); + expect(injected).not.toContain("native-plus-ocx.json"); + expect(injected.match(/model_catalog_json/g)?.length).toBe(1); + expect(injected).toContain('model = "gpt-6.1-sol"'); + }); + + test("managed routing removes any root catalog override when the OCX catalog is unavailable", () => { + const original = [ + 'model = "gpt-6.1-sol"', + 'model_catalog_json = "/home/joep/.codex/model-catalogs/native-plus-ocx.json"', + "", + "[profiles.work]", + 'model_catalog_json = "/tmp/profile-only.json"', + "", + ].join("\n"); + + const injected = stripRootModelCatalogPath(original); + + expect(injected).not.toContain("native-plus-ocx.json"); + expect(injected).toContain('model_catalog_json = "/tmp/profile-only.json"'); + expect(injected).toContain('model = "gpt-6.1-sol"'); + }); + test("strips injected TOML sections without swallowing later indented tables", () => { const stripped = stripOpencodexConfig( [ @@ -510,4 +610,4 @@ describe("EOL boundary helpers (Windows CRLF configs)", () => { // Idempotent on already-normalized input. expect(applyEol(crlf, "\r\n")).toBe(crlf); }); -}); +}); \ No newline at end of file diff --git a/tests/codex-journal.test.ts b/tests/codex-journal.test.ts index cee39e6f..6975cd95 100644 --- a/tests/codex-journal.test.ts +++ b/tests/codex-journal.test.ts @@ -234,6 +234,82 @@ describe("codex-journal", () => { expect(existsSync(join(testDir, "opencodex-journal.json"))).toBe(false); }); + test("restoreNativeCodex never rewrites a restored user-owned catalog", () => { + const userCatalogPath = join(testDir, "native-plus-ocx.json"); + const userCatalog = JSON.stringify({ + models: [ + { slug: "gpt-5.5", display_name: "GPT-5.5" }, + { slug: "user-provider/custom-model", display_name: "User custom route" }, + ], + }, null, 2) + "\n"; + const original = [ + 'model = "gpt-5.5"', + `model_catalog_json = ${JSON.stringify(userCatalogPath)}`, + "", + ].join("\n"); + writeFileSync(join(testDir, "config.toml"), original, "utf8"); + writeFileSync(userCatalogPath, userCatalog, "utf8"); + + const r = runScript(testDir, ` + const { injectCodexConfig, restoreNativeCodex } = require("./src/codex/inject"); + (async () => { + const injected = await injectCodexConfig( + 10100, + { port: 10100, providers: {}, defaultProvider: "openai" }, + { catalogPath: null }, + ); + if (!injected.success) throw new Error(injected.message); + console.log(JSON.stringify(restoreNativeCodex())); + })(); + `); + + expect(r.status).toBe(0); + expect(JSON.parse(r.stdout).success).toBe(true); + expect(readFileSync(join(testDir, "config.toml"), "utf8")).toBe(original); + expect(readFileSync(userCatalogPath, "utf8")).toBe(userCatalog); + }); + + test("user-owned openai_base_url takeover keeps the pre-OCX restore journal", () => { + const original = '# original config\nmodel = "gpt-5.5"\n'; + writeFileSync(join(testDir, "config.toml"), original, "utf8"); + + const r = runScript(testDir, ` + const fs = require("fs"); + const path = require("path"); + const { injectCodexConfig } = require("./src/codex/inject"); + (async () => { + const first = await injectCodexConfig( + 10100, + { port: 10100, providers: {}, defaultProvider: "openai" }, + { catalogPath: null }, + ); + if (!first.success) throw new Error(first.message); + const configPath = path.join(process.env.CODEX_HOME, "config.toml"); + fs.writeFileSync( + configPath, + 'openai_base_url = "https://user-gateway.example/v1"\nmodel = "gpt-5.5"\n', + "utf8", + ); + const second = await injectCodexConfig( + 10100, + { port: 10100, providers: {}, defaultProvider: "openai" }, + { catalogPath: null }, + ); + console.log(JSON.stringify({ + success: second.success, + journalExists: fs.existsSync(path.join(process.env.CODEX_HOME, "opencodex-journal.json")), + config: fs.readFileSync(configPath, "utf8"), + })); + })(); + `); + + expect(r.status).toBe(0); + const result = JSON.parse(r.stdout) as { success: boolean; journalExists: boolean; config: string }; + expect(result.success).toBe(true); + expect(result.journalExists).toBe(true); + expect(result.config).toContain('openai_base_url = "https://user-gateway.example/v1"'); + }); + test("restoreNativeCodex reports damaged managed-default cleanup during fallback restore", () => { const original = '# original config\nmodel_provider = "openai"\n'; writeFileSync(join(testDir, "config.toml"), original, "utf8"); diff --git a/tests/codex-models-cache-invalidate.test.ts b/tests/codex-models-cache-invalidate.test.ts index e013ac22..9d62f48e 100644 --- a/tests/codex-models-cache-invalidate.test.ts +++ b/tests/codex-models-cache-invalidate.test.ts @@ -54,6 +54,27 @@ describe("invalidateCodexModelsCache write gate (#476 / #518)", () => { expect(cache.models).toEqual([{ slug: "gpt-5.5" }]); }); + test("explicit cache source never follows a user model_catalog_json pointer", () => { + const userCatalogPath = join(codexHome, "native-plus-ocx.json"); + const managedCatalogPath = join(codexHome, "opencodex-catalog.json"); + const userCatalog = JSON.stringify({ + models: [{ slug: "user-provider/custom-model" }], + }, null, 2) + "\n"; + writeFileSync(join(codexHome, "config.toml"), 'model_catalog_json = "native-plus-ocx.json"\n', "utf8"); + writeFileSync(userCatalogPath, userCatalog, "utf8"); + writeFileSync(managedCatalogPath, JSON.stringify({ + models: [{ slug: "gpt-6.1-sol", context_window: 400000 }], + }, null, 2) + "\n", "utf8"); + + expect(invalidateCodexModelsCache(managedCatalogPath)).toBe(true); + expect(readFileSync(userCatalogPath, "utf8")).toBe(userCatalog); + + const cache = JSON.parse(readFileSync(join(codexHome, "models_cache.json"), "utf8")) as { + models: Array<{ slug: string; context_window?: number }>; + }; + expect(cache.models).toEqual([{ slug: "gpt-6.1-sol", context_window: 400000 }]); + }); + test("returns false for a missing catalog and does not warn/restart app-servers", () => { const errors: string[] = []; const logs: string[] = []; @@ -111,46 +132,37 @@ describe("invalidateCodexModelsCache write gate (#476 / #518)", () => { expect(logs).toEqual([]); }); - test("ocx sync --restart-codex neither warns nor restarts when catalog exists but is unreadable", async () => { - // Non-default catalog path that exists on disk but cannot be read or rewritten as JSON. - // (A directory at the catalog path: existsSync true, load/write both fail.) - writeFileSync(join(codexHome, "config.toml"), 'model_catalog_json = "broken.json"\n', "utf8"); - mkdirSync(join(codexHome, "broken.json")); - - const syncResult = await syncModelsToCodex(10100, emptyConfig, null, { - refreshCodexModelCatalog, - injectCodexConfig: async () => ({ success: true, message: "injected" }), - currentExternalCodexModelProvider: () => null, - }); - - expect(syncResult.catalogExists).toBe(true); - expect(syncResult.catalogWritten).toBe(false); - expect(syncResult.cacheSynced).toBe(false); - - const errors: string[] = []; - const logs: string[] = []; - let listed = 0; - - // Mirrors `ocx sync --restart-codex`: only handle app-servers after a real write. - if (syncResult.catalogWritten || syncResult.cacheSynced) { - afterCatalogWriteHandleAppServers({ - restart: true, - log: { log: line => logs.push(String(line)), error: line => errors.push(String(line)) }, - io: { - listSnapshots: () => { - listed += 1; - return [{ pid: 7, commandLine: "codex app-server" }]; - }, - kill: () => {}, - isAlive: () => false, - waitExit: () => true, - }, - }); - } - - expect(listed).toBe(0); - expect(errors).toEqual([]); - expect(logs).toEqual([]); + test("refresh forwards one explicit managed path to build and cache invalidation", async () => { + const managedCatalogPath = join(codexHome, "opencodex-catalog.json"); + let syncTarget: string | undefined; + let invalidatedPath: string | undefined; + + const result = await refreshCodexModelCatalog(emptyConfig, { + syncCatalogModels: async (_config, options) => { + syncTarget = options?.catalogPath; + writeFileSync( + managedCatalogPath, + JSON.stringify({ models: [{ slug: "gpt-6.1-sol" }] }), + "utf8", + ); + return { + added: 0, + path: managedCatalogPath, + catalogWritten: true, + comboOmissions: [], + }; + }, + invalidateCodexModelsCache: (path) => { + invalidatedPath = path; + return true; + }, + existsSync, + }, { catalogPath: managedCatalogPath }); + + expect(syncTarget).toBe(managedCatalogPath); + expect(invalidatedPath).toBe(managedCatalogPath); + expect(result.catalogWritten).toBe(true); + expect(result.cacheSynced).toBe(true); }); test("ocx sync --restart-codex neither warns nor restarts when catalog JSON is malformed", async () => { diff --git a/tests/codex-native-model-discovery.test.ts b/tests/codex-native-model-discovery.test.ts new file mode 100644 index 00000000..8ac722d4 --- /dev/null +++ b/tests/codex-native-model-discovery.test.ts @@ -0,0 +1,258 @@ +import { describe, expect, test } from "bun:test"; +import { discoverNativeOpenAiCatalog } from "../src/codex/catalog/native-discovery"; +import { buildCatalogEntries, mergeCatalogEntriesForSync } from "../src/codex/catalog/sync"; +import type { OcxConfig } from "../src/types"; + +const liveNative = { + slug: "gpt-6.1-sol", + display_name: "GPT-6.1 Sol", + description: "Live native model", + supported_reasoning_levels: [ + { effort: "high", description: "High reasoning" }, + ], + visibility: "list", + priority: 1, + supported_in_api: true, + supports_search_tool: true, + tool_mode: "code_mode_only", + use_responses_lite: true, + context_window: 400_000, + input_modalities: ["text", "image"], +}; + +function config( + overrides: Pick< + OcxConfig, + "codexAccounts" | "activeCodexAccountId" | "codexAccountPools" + > = {}, +): OcxConfig { + return { + port: 10100, + providers: {}, + defaultProvider: "openai", + ...overrides, + }; +} + +describe("live native OpenAI catalog discovery", () => { + test("uses the explicitly selected pool account before the physical main account", async () => { + let mainReads = 0; + const requests: Array<{ url: string; headers: Headers }> = []; + + const result = await discoverNativeOpenAiCatalog(config({ + codexAccounts: [{ id: "pool-a", email: "a@example.test", isMain: false }], + activeCodexAccountId: "pool-a", + }), { + getEffectiveActiveCodexAccountId: () => "pool-a", + getMainAccountToken: () => { + mainReads += 1; + return { accessToken: "stale-main", chatgptAccountId: "stale-main-account" }; + }, + getValidCodexToken: async (id: string) => { + expect(id).toBe("pool-a"); + return { + accessToken: "pool-access", + chatgptAccountId: "pool-chatgpt-account", + generation: 1, + }; + }, + resolveClientVersion: () => "0.160.0", + fetch: async (input: RequestInfo | URL, init?: RequestInit) => { + requests.push({ url: String(input), headers: new Headers(init?.headers) }); + return new Response(JSON.stringify({ models: [liveNative] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }, + }); + + expect(mainReads).toBe(0); + expect(requests).toHaveLength(1); + const request = requests[0]; + if (!request) throw new Error("expected one native catalog request"); + expect(request.url).toBe( + "https://chatgpt.com/backend-api/codex/models?client_version=0.160.0", + ); + expect(request.headers.get("authorization")).toBe("Bearer pool-access"); + expect(request.headers.get("chatgpt-account-id")).toBe("pool-chatgpt-account"); + expect(request.headers.get("originator")).toBe("codex_cli_rs"); + expect(request.headers.get("version")).toBe("0.160.0"); + expect(result.models.map(model => model.slug)).toEqual(["gpt-6.1-sol"]); + }); + + test("falls back from a dead physical main account to a usable pool credential", async () => { + const seenAuth: string[] = []; + const result = await discoverNativeOpenAiCatalog(config({ + codexAccounts: [{ id: "pool-b", email: "b@example.test", isMain: false }], + }), { + getEffectiveActiveCodexAccountId: () => undefined, + getMainAccountToken: () => ({ + accessToken: "dead-main", + chatgptAccountId: "dead-main-account", + }), + getValidCodexToken: async () => ({ + accessToken: "pool-good", + chatgptAccountId: "pool-good-account", + generation: 1, + }), + resolveClientVersion: () => "0.160.0", + fetch: async (_input: RequestInfo | URL, init?: RequestInit) => { + const auth = new Headers(init?.headers).get("authorization") ?? ""; + seenAuth.push(auth); + if (auth === "Bearer dead-main") return new Response("", { status: 401 }); + return new Response(JSON.stringify({ models: [liveNative] }), { status: 200 }); + }, + }); + + expect(seenAuth).toEqual(["Bearer dead-main", "Bearer pool-good"]); + expect(result.models.map(model => model.slug)).toEqual(["gpt-6.1-sol"]); + }); + + test("falls back when a successful response body fails while reading", async () => { + let calls = 0; + const result = await discoverNativeOpenAiCatalog(config({ + codexAccounts: [{ id: "pool-stream", email: "stream@example.test", isMain: false }], + }), { + getEffectiveActiveCodexAccountId: () => undefined, + getMainAccountToken: () => ({ + accessToken: "main-stream-fails", + chatgptAccountId: "main-stream-account", + }), + getValidCodexToken: async () => ({ + accessToken: "pool-stream-good", + chatgptAccountId: "pool-stream-account", + generation: 1, + }), + resolveClientVersion: () => "0.160.0", + fetch: async () => { + calls += 1; + if (calls === 1) { + return new Response(new ReadableStream({ + start(controller) { + controller.error(new Error("upstream body failed")); + }, + }), { status: 200 }); + } + return new Response(JSON.stringify({ models: [liveNative] }), { status: 200 }); + }, + }); + + expect(calls).toBe(2); + expect(result.models.map(model => model.slug)).toEqual(["gpt-6.1-sol"]); + }); + + test("does not consult pool credentials when account pools are disabled", async () => { + let poolReads = 0; + const result = await discoverNativeOpenAiCatalog(config({ + codexAccountPools: false, + codexAccounts: [{ id: "pool-disabled", email: "off@example.test", isMain: false }], + activeCodexAccountId: "pool-disabled", + }), { + getEffectiveActiveCodexAccountId: () => "pool-disabled", + getMainAccountToken: () => ({ + accessToken: "dead-main", + chatgptAccountId: "dead-main-account", + }), + getValidCodexToken: async () => { + poolReads += 1; + return { + accessToken: "must-not-be-read", + chatgptAccountId: "must-not-be-read", + generation: 1, + }; + }, + resolveClientVersion: () => "0.160.0", + fetch: async () => new Response("", { status: 401 }), + }); + + expect(poolReads).toBe(0); + expect(result.models).toEqual([]); + }); + + test("shares one total request deadline across account fallbacks", async () => { + const signals: AbortSignal[] = []; + let calls = 0; + const result = await discoverNativeOpenAiCatalog(config({ + codexAccounts: [{ id: "pool-c", email: "c@example.test", isMain: false }], + }), { + getEffectiveActiveCodexAccountId: () => undefined, + getMainAccountToken: () => ({ + accessToken: "dead-main", + chatgptAccountId: "dead-main-account", + }), + getValidCodexToken: async () => ({ + accessToken: "pool-good", + chatgptAccountId: "pool-good-account", + generation: 1, + }), + resolveClientVersion: () => "0.160.0", + fetch: async (_input: RequestInfo | URL, init?: RequestInit) => { + if (!(init?.signal instanceof AbortSignal)) { + throw new Error("expected native catalog request deadline"); + } + signals.push(init.signal); + calls += 1; + if (calls === 1) return new Response("", { status: 401 }); + return new Response(JSON.stringify({ models: [liveNative] }), { status: 200 }); + }, + }); + + expect(signals).toHaveLength(2); + expect(signals[1]).toBe(signals[0]); + expect(result.models.map(model => model.slug)).toEqual(["gpt-6.1-sol"]); + }); + + test("catalog builder preserves authoritative metadata for a newly rolled-out native slug", () => { + const entries = buildCatalogEntries( + null, + ["gpt-6.1-sol"], + [], + undefined, + false, + "default", + new Set(), + new Map([["gpt-6.1-sol", liveNative]]), + ); + + const row = entries.find(entry => entry.slug === "gpt-6.1-sol"); + expect(row?.display_name).toBe("GPT-6.1 Sol"); + expect(row?.context_window).toBe(400_000); + expect(row?.supported_reasoning_levels).toEqual([ + { effort: "high", description: "High reasoning" }, + ]); + expect(row?.supports_search_tool).toBe(true); + expect(row?.tool_mode).toBe("code_mode_only"); + expect(row?.use_responses_lite).toBe(true); + }); + + test("authoritative live native rows survive the static native whitelist unchanged", () => { + const args: Parameters = [ + [liveNative], + [], + new Map(), + [], + false, + new Set(), + null, + new Set(), + new Set(), + "default", + new Set(), + false, + true, + new Set(["gpt-6.1-sol"]), + ]; + const result = mergeCatalogEntriesForSync(...args); + + const row = result.find(entry => entry.slug === "gpt-6.1-sol"); + expect(row).toBeDefined(); + expect(row?.display_name).toBe("GPT-6.1 Sol"); + expect(row?.supported_reasoning_levels).toEqual([ + { effort: "high", description: "High reasoning" }, + ]); + expect(row?.context_window).toBe(400_000); + expect(row?.supports_search_tool).toBe(true); + expect(row?.tool_mode).toBe("code_mode_only"); + expect(row?.use_responses_lite).toBe(true); + }); +}); \ No newline at end of file diff --git a/tests/codex-sync-api.test.ts b/tests/codex-sync-api.test.ts index 85064b58..ab0f35c1 100644 --- a/tests/codex-sync-api.test.ts +++ b/tests/codex-sync-api.test.ts @@ -159,12 +159,104 @@ describe("GUI/CLI Codex sync backend", () => { currentExternalCodexModelProvider: () => null, }); - expect(injectedCatalogPath).toBeUndefined(); + expect(injectedCatalogPath).toBeNull(); expect(result.ok).toBe(true); expect(result.catalogPath).toBeNull(); expect(result.warning).toContain("catalog boom"); }); + test("forces the canonical OCX catalog build target before injection", async () => { + writeFileSync( + join(TEST_CODEX_HOME, "config.toml"), + 'model = "gpt-5.5"\nmodel_catalog_json = "native-plus-ocx.json"\n', + "utf8", + ); + let refreshedCatalogPath: string | undefined; + let injectedCatalogPath: string | null | undefined; + + const result = await syncModelsToCodex(10100, config, null, { + refreshCodexModelCatalog: async (_config, _deps, options) => { + const catalogPath = options?.catalogPath; + if (!catalogPath) throw new Error("expected canonical catalog target"); + refreshedCatalogPath = catalogPath; + return { + added: 0, + path: catalogPath, + catalogExists: true, + catalogWritten: true, + cacheSynced: true, + comboOmissions: [], + }; + }, + injectCodexConfig: async (_port, _config, options) => { + injectedCatalogPath = options.catalogPath; + return { success: true, message: "injected canonical" }; + }, + currentExternalCodexModelProvider: () => null, + currentUserOwnedRootOpenaiBaseUrl: () => false, + }); + + const expected = join(TEST_CODEX_HOME, "opencodex-catalog.json"); + expect(refreshedCatalogPath).toBe(expected); + expect(injectedCatalogPath).toBe(expected); + expect(result.catalogPath).toBe(expected); + expect(result.catalogWritten).toBe(true); + }); + + test("does not refresh any catalog when a user owns root openai_base_url", async () => { + let refreshed = false; + let injectedCatalogPath: string | null | undefined = "unset"; + + const result = await syncModelsToCodex(10100, config, null, { + refreshCodexModelCatalog: async () => { + refreshed = true; + throw new Error("must not refresh"); + }, + injectCodexConfig: async (_port, _config, options) => { + injectedCatalogPath = options.catalogPath; + return { success: true, message: "user routing preserved" }; + }, + currentExternalCodexModelProvider: () => null, + currentUserOwnedRootOpenaiBaseUrl: () => true, + }); + + expect(refreshed).toBe(false); + expect(injectedCatalogPath).toBeUndefined(); + expect(result).toEqual({ + ok: true, + added: 0, + catalogPath: null, + catalogExists: false, + catalogWritten: false, + cacheSynced: false, + message: "user routing preserved", + }); + }); + + test("existing but unwritten managed catalog is not injected", async () => { + let injectedCatalogPath: string | null | undefined = "unset"; + const result = await syncModelsToCodex(10100, config, null, { + refreshCodexModelCatalog: async (_config, _deps, options) => ({ + added: 0, + path: options?.catalogPath ?? "missing-canonical-catalog", + catalogExists: true, + catalogWritten: false, + cacheSynced: false, + comboOmissions: [], + }), + injectCodexConfig: async (_port, _config, options) => { + injectedCatalogPath = options.catalogPath; + return { success: true, message: "native fallback" }; + }, + currentExternalCodexModelProvider: () => null, + currentUserOwnedRootOpenaiBaseUrl: () => false, + }); + + expect(injectedCatalogPath).toBeNull(); + expect(result.catalogWritten).toBe(false); + expect(result.warning).toContain("did not materialize"); + }); + test("returns native subagent default conflicts as structured warnings", async () => { const result = await syncModelsToCodex(10100, config, null, { refreshCodexModelCatalog: async () => ({ diff --git a/tests/provider-registry-parity.test.ts b/tests/provider-registry-parity.test.ts index 50829c0a..5c177fb2 100644 --- a/tests/provider-registry-parity.test.ts +++ b/tests/provider-registry-parity.test.ts @@ -51,6 +51,11 @@ const EXPECTED_KEY_PROVIDER_IDS = [ "azure-openai", "deepseek", "cerebras", + "cohere", + "friendliai", + "sambanova", + "nebius", + "novita", "together", "fireworks", "firepass", @@ -743,6 +748,7 @@ describe("provider registry parity", () => { expect(nvidia?.keyOptional).toBeUndefined(); expect(freeTierProviders).toEqual([ "tokenharbor", + "cohere", "nvidia", "cloudflare-workers-ai", "omniroute",