[bug] base_instructions stub replaces Codex's own system prompt on every routed session

Symptom

Switchyard's Codex model card advertises base_instructions: "You are Codex, a coding agent.", and Codex adopts it in place of its own system prompt. Every routed session therefore runs with a six-word system prompt instead of the full Codex instructions, which changes agent behaviour on every turn and quietly invalidates any A/B comparison made through Switchyard.

Cause

crates/switchyard-server/src/lib.rs, in codex_model_entry_json:

// Required `ModelInfo` string. Unlike the launcher, the server cannot read
// Codex's bundled prompt, so it sends a minimal stub.
"base_instructions": "You are Codex, a coding agent.",

The comment is accurate about the constraint but the consequence looks unintended: this is not a fallback Codex ignores, it is a value Codex prefers over its own.

Evidence

Codex records the provenance of the instructions it used. Routed through Switchyard:

"base_instructions": {
  "text": "You are Codex, a coding agent.",
  "provenance": {"type": "model", "model": "sy-solo"}
}

The same client on the same task with no proxy records its real multi-paragraph prompt ("You are a coding agent running in the Codex CLI, a terminal-based coding assistant…").

Corroboration that the string travels over the wire rather than being a client-side default: it is absent from the codex 0.150.1 binary and present in the Switchyard binary.

Expected vs. actual

Expected: a proxy that routes requests does not replace the client's system prompt.
Actual: it does, on every route, silently.

Impact

Any benchmark or routing evaluation that compares "through Switchyard" against "direct" is measuring two different agents, not two different routes. That is a measurement-integrity problem rather than a crash, which is why it is easy to miss.

Two neighbouring fields on the same entry have the same shape of problem: default_reasoning_level is forced to "xhigh" for any reasoning-capable route, and truncation_policy to {"mode": "tokens", "limit": 10000}, regardless of client or target.

Suggested direction

Omit base_instructions if ModelInfo permits it, so Codex keeps its own; otherwise make it configurable per route with "leave the client's prompt alone" as the default. Happy to send a PR if you can say which of those you would accept.

Environment

Commit: 7f3b2fe9 (main)
Client: codex-cli 0.150.1

feat(client): responses_tool_images — re-home a tool-returned image w…

@ardada2468

⛔ RETRACTED — the conclusion below is wrong. Omitting the field breaks catalog
decoding entirely; Codex only appears to tolerate it because the failure is a warning.
See the correction
and the resulting recommendation.
Kept unedited below for the record.

Reporter here, with the Codex-side measurement #631 is missing.

In #631 @ardada2468 notes he could not verify whether base_instructions is a
required ModelInfo field, because Codex is not installed in his environment. It is
installed here, and this deployment has been serving a catalog without the field
since 2026-08-30.

How. A local patch on top of main adds a SWITCHYARD_BASE_INSTRUCTIONS env var:
unset → today's stub, a string → that string verbatim, empty → the key is removed
from the entry (object.remove("base_instructions") after codex_model_entry_json
builds it, so it is absent rather than "" or null). The unit runs it empty. That is
a different mechanism from #631, but the served shape is the same, and the served
shape is what Codex sees.

What Codex is served. GET /v1/models → the models[] array, read just now. All
six routes report base_instructions absent. One entry verbatim:

{
 "slug": "sy-solo",
 "display_name": "sy-solo",
 "description": "Switchyard-routed model.",
 "default_reasoning_level": "xhigh",
 "shell_type": "shell_command",
 "visibility": "list",
 "supported_in_api": true,
 "apply_patch_tool_type": "freeform",
 "web_search_tool_type": "text",
 "truncation_policy": {"mode": "tokens", "limit": 10000},
 "supports_parallel_tool_calls": true,
 "context_window": 131072,
 "max_context_window": 131072,
 "input_modalities": ["text", "image"]
}

Every sibling field is present; base_instructions is simply not there.

What Codex did with it. 504 streamed requests handled through that catalog on
2026-08-30 and 08-31 (226 + 278), wire_api = "responses", driven by real Codex
sessions rather than curl. Zero catalog-parse failures, and the string
base_instructions appears nowhere in the server journal for that period.

⇒ base_instructions is not a required ModelInfo field. Codex parses an entry
that omits it and runs normally — it does not fail loudly at catalog parse. So
omission is viable, and nothing forces the per-route option as the fallback.

Scope, so this is not over-read. That traffic was Codex 0.149.1. This host runs
0.153.4 now, but the upgrade landed 2026-09-05 and no Codex session has crossed this
catalog since, so I cannot speak for 0.153.x from measurement — only that the field was
optional through 0.149.1. Happy to run a 0.153.4 session against it and report back if
that would help settle the decision. I have also not built #631's branch: the evidence
above is that Codex tolerates an absent key, not a review of that diff.

@ardada2468

Correction to my previous comment. I had this backwards: omitting the field does not
work, and my evidence was measuring the wrong thing.

I checked the Codex source, which I should have done first. base_instructions is not a
field of ModelInfo at all — it is handled by a legacy compatibility shim in
codex-rs/protocol/src/openai_models.rs:

#[derive(Deserialize)]
struct ModelInfoWithLegacyBaseInstructions {
    #[serde(default)]
    base_instructions: Option<String>,
    #[serde(flatten)]
    model: ModelInfo,
}

The #[serde(default)] is what misled me: optional at the field level. But
deserialize_model_infos_with_legacy_base, wired into ModelsResponse through
deserialize_with, promotes that value into model_messages.instructions_template and
then hard-errors when neither is present:

return Err(D::Error::custom(format!(
    "model `{model_slug}` is missing both `base_instructions` and \
     `model_messages.instructions_template`"
)));

Identical code in rust-v0.149.1 and rust-v0.153.4.

Measured. Codex 0.153.4 against a static local server replaying this deployment's
real /v1/models bytes, with a control arm:

catalog served	result
`base_instructions` absent (what we serve today)	`ERROR codex_models_manager::manager: failed to refresh available models: failed to decode models response: model` sy-capability`is missing both`base_instructions`and`model_messages.instructions_template``, followed by `warning: Model metadata for` sy-solo` not found. Defaulting to fallback metadata; this can degrade performance and cause issues.`
identical catalog, field restored	silent — no error, no warning

Why my earlier evidence looked like success. The failure is a warning, not a fatal
error. Codex logs it to stderr and continues on fallback metadata, so sessions run and
turns complete — which is exactly the "504 requests, zero failures" I reported. What I
never checked was whether the catalog was being used. It was not.

The consequence is worse than the bug. ModelsResponse.models decodes as a single
Vec, so one offending entry discards the entire catalog. Everything Switchyard
advertises is then thrown away — context_window, reasoning levels, truncation_policy,
supports_parallel_tool_calls, and input_modalities. That last one is the field #567
added so Codex stops dropping images (#563). Omitting base_instructions would silently
undo #567 for every Codex user, and the only symptom would be a stderr warning.

⇒ #631 as written will not do what it intends. @ardada2468 — your reviewer note
guessed right: Codex does fail at catalog decode, so the per-route option from this
issue's second proposal is the shape that works. The other viable option is to emit
model_messages.instructions_template instead, which is where Codex puts the value
anyway, and which would let the client keep its own prompt without breaking the catalog.

Please read my previous comment as retracted.

One fix to my own correction, because it changes the recommendation.

I wrote that emitting model_messages.instructions_template would "let the client keep
its own prompt without breaking the catalog". That is wrong.
ModelInfo::get_model_instructions returns the template verbatim:

if let Some(model_messages) = &self.model_messages
    && let Some(template) = &model_messages.instructions_template
{
    if model_messages.instructions_variables.is_none() {
        return template.clone();
    }

So a served template is adopted exactly like the legacy field. It is the same value under
a different name, not an escape from it.

⇒ The structural point: a proxy cannot serve a valid Codex catalog entry that says
nothing about instructions. The decoder requires one of the two fields, and whichever is
supplied is adopted. Within this catalog format there is no "leave the client's own prompt
alone" option to reach for.

That leaves two fixes that actually work:

Per-route configuration — this issue's second proposal. The operator supplies the
instructions and can paste Codex's own prompt for parity; Switchyard stops inventing a
six-word one. This is the only one actionable in this repo.
Upstream in Codex — make the template genuinely optional and fall back to the
bundled prompt models-manager already carries
(pub const BASE_INSTRUCTIONS: &str = include_str!("../prompt.md")). Then omission
would behave the way fix(server): configure Codex base instructions per route, and cap the windowed judge payload #631 intends, and every OpenAI-compatible proxy would benefit
rather than just this one.

One more measurement that makes option 1 concrete. The prompt Codex would otherwise
use is a static file in its own tree, codex-rs/models-manager/prompt.md — 20,903 bytes,
and byte-identical between rust-v0.149.1 and rust-v0.153.4 (cmp clean). So a
per-route option is not "the operator invents a system prompt": an operator who wants
parity with a direct session can serve that file verbatim and get it, and the catalog
still decodes. That also gives #631 a testable acceptance criterion — a routed session
and a direct session should produce the same base instructions, rather than the routed one
simply producing something.

(Version coupling is the obvious caveat: the file is stable across these two releases but
nothing guarantees it stays so, which is an argument for the Codex-side fallback in option
2 being the durable fix.)

Navigation Menu

Repository navigation

[bug] base_instructions stub replaces Codex's own system prompt on every routed session #565

Symptom

Cause

Evidence

Expected vs. actual

Impact

Suggested direction

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[bug] base_instructions stub replaces Codex's own system prompt on every routed session #565

Description

Symptom

Cause

Evidence

Expected vs. actual

Impact

Suggested direction

Environment

Activity

mrPronin commented on Sep 6, 2026

mrPronin commented on Sep 6, 2026

mrPronin commented on Sep 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions