fix: reject non-leading system messages at /v1/responses admission

[zmanian]

Fixes #1025.

Reported by an external team (Trace Commons) hitting this through a proxy in front of the gateway, with the Codex CLI as the client. Happy to adjust to whatever you prefer here.

Revised twice after review, both times narrowing what the PR refuses. It now changes exactly one thing — it folds a leading system-level input message — and refuses nothing.

The defect

A POST /v1/responses carrying both instructions and a developer-role input message is accepted with HTTP 200, the SSE stream is opened, and only then does it fail:

event: response.failed
{"error":{"message":"Provider failed for model 'Qwen/Qwen3.6-35B-A3B-FP8': System message must be at the beginning.",
          "type":"invalid_request_error"},"status_code":400}

The body already carries "status_code":400, but it is delivered inside a stream committed with a 200. Clients cannot tell it apart from a mid-stream disconnect: the Codex CLI retried five times and each attempt failed identically. Reproduced on Qwen/Qwen3.6-35B-A3B-FP8 and Qwen/Qwen3.8-27B; the same request without a system-level input message succeeds, while the identical message list is accepted by four models across DeepSeek and GLM (transcript below).

The triggering shape is the Codex CLI's: a ~17,000-character instructions string plus a developer-role message as the first input item (input roles: developer, user, user).

Root cause: the gateway builds the second system message itself

The second system message is not something the client sends and not something the provider invents — this code makes it:

• Service::load_conversation_context (crates/services/src/responses/service.rs) pushes a system message derived from instructions (merged with the language instruction and time context) first, before any input item, then forwards each input message's role verbatim.
• Service::to_chat_messages (crates/services/src/completions/mod.rs:1468) maps "system" | "developer" => MessageRole::System.

So developer becomes a system message at index 1, behind the one built from instructions. Captured from the mock provider with the fix disabled — this is the payload that goes upstream, abridged for reading (each content is really Some(String(...)), and the name / tool_call_id / tool_calls fields are omitted):

[ChatMessage { role: System, content: "You are a coding agent.\n\nAlways respond in the exact same language..." },
 ChatMessage { role: System, content: "Repository guidelines." },
 ChatMessage { role: User,   content: "Fix the build." },
 ChatMessage { role: User,   content: "Then run the tests." }]

Two system messages, the second not at the beginning — exactly what Qwen's chat template objects to.

The fix

The gateway now builds one leading system message, coalescing every system-level source it has: the organization prompt (when configured), the request context (instructions, the language instruction, the time context), and a system-level input message that precedes all other content. They keep the order they had as separate messages, so the model is told the same content in the same sequence; the provider sees the single leading system message its template requires; and the Codex request works rather than merely failing legibly.

Coalescing the organization prompt came from review — @hanakannzashi pointed out that folding only the input message left system(org prompt), system(instructions + developer), user(...), still a system message at index 1, so the reported failure stayed reachable for every organization with a prompt configured.

That is the whole change. There is no new rejection.

Why nothing is refused at admission

The issue asks for a 400 at admission, and an earlier revision of this PR did that. It was wrong, and the review caught the first half of why: CreateResponseRequest::validate runs before model and provider resolution, so any check it makes is a pure role-and-position test applied to every backend. /v1/responses serves external and non-Qwen models, and the shapes in question work there — the transcript below shows the identical message list returning 200 on four models across DeepSeek and GLM while Qwen returns 400.

Note the asymmetry. The fold is provider-agnostic because it preserves what the model is told: it is correct regardless of which backend receives it. A rejection is not — it is provider-specific reasoning applied provider-agnostically, generalizing one chat template's constraint into a gateway-wide admission rule. So the three shapes that cannot be folded without reordering — a system-level message after other content, one behind replayed conversation history, one whose content is not plain text — are forwarded unchanged, exactly as today. Whether such a payload is acceptable is the provider's judgement. Nothing that works today breaks; Qwen behaves on those shapes exactly as it does now; and the value of the PR is untouched, because it comes from the fold.

The fold guard tests the messages actually built rather than the request fields, so pass-through needed no separate code path. That also corrected an assumption: previous_response_id alone replays no history — it only selects a branch within a conversation — so such a request is folded, and the test covering replayed history sets up a conversation.

Tests

Unit (responses::models::tests): pins that validate accepts the leading developer shape, each of the three un-foldable shapes, user/assistant, and an unmapped spelling ("Developer") anywhere — to_chat_messages matches exactly, so that becomes a user message and never produces a system message at all.

E2E (crates/api/tests/e2e_all/responses_system_message_position.rs, modelled on first_stream_event.rs). The first three assert the payload the provider actually receives, via mock.last_chat_params():

• the Codex-shaped request returns 200 and reaches the provider as exactly one system message, first, starting with the instructions text and containing the developer text, with both user turns intact in order;
• a system message after user content returns 200 and arrives as [System, User, System] — still in third place, exactly where the caller put it;
• a developer message behind replayed conversation history returns 200 and arrives as its own system message after that history, not folded;
• with an organization prompt configured, the Codex-shaped request produces exactly one system message carrying all three sources in order — organization prompt, instructions, folded developer text;
• a user-only request still returns 200.

The fold fails without the change: with it disabled the first test sees the two-system-message payload quoted above.

Everything .github/workflows/test.yml runs, locally (PostgreSQL 15 in Docker; cargo test rather than cargo nextest):

$ cargo fmt --all -- --check                                  # clean
$ cargo clippy --all-targets --all-features -- -D warnings    # clean
$ cargo test --lib --bins
test result: ok. 647 passed; 0 failed; 1 ignored
test result: ok. 435 passed; 0 failed; 1 ignored
(+ 6 further crates, all ok, 0 failed)
$ cargo test --test integration_tests
test result: ok. 8 passed; 0 failed; 1 ignored
$ cargo test --test e2e_all -- --test-threads=8
test result: ok. 738 passed; 0 failed; 10 ignored; 0 measured; 0 filtered out; finished in 46.37s

Verified live against the model

Confirmed on 2026-09-06 against Qwen/Qwen3.6-35B-A3B-FP8 via https://cloud-api.near.ai/v1/chat/completions, max_tokens: 16. Two requests differing only in whether the two system messages are folded.

A — the unfolded shape (two system messages, the second not at index 0):

messages: [ {role: system, content: "You are a coding agent."},
            {role: system, content: "Repository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 400

{"error":{"message":"Provider failed for model 'Qwen/Qwen3.6-35B-A3B-FP8': System message must be at the beginning.","type":"invalid_request_error","param":null,"code":null}}

B — the folded shape this PR produces:

messages: [ {role: system, content: "You are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 200

{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","reasoning_content":"Here's a thinking process:\n\n1.  **Analyze User Input:**"}}],
 "id":"a4035f807e104471a02fbf01a5604c5f","model":"Qwen/Qwen3.6-35B-A3B-FP8","object":"chat.completion",
 "created":1788726154,"usage":{"prompt_tokens":28,"completion_tokens":16}}

The error string in A is character-for-character the one in #1025. Folding is what makes the difference between a rejection and a completion.

The same payload A on other model families

Same gateway and method, max_tokens: 8, also 2026-09-06. The request is byte-identical to payload A — the two-system-message shape Qwen refuses. Four models, two families, all HTTP 200:

• deepseek-ai/DeepSeek-V4-Flash — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK."}}],"id":"520dead424fa4813b2e808f31c8d54e0","created":1788727428}
• deepseek/deepseek-v3.2 — {"id":"c3173b1436a54795bdf9cd875c213964","created":1788727439,"choices":[{"index":0,"message":{"role":"assistant","content":"OK."}}]}
• zai-org/GLM-5.1-FP8 — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK"}}],"id":"9173ec431fb84a1d970388dd9e970974","created":1788727429}
• z-ai/glm-5.2 — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK."}}],"id":"1645a61c377f445aa67f2a95fea7f4b2","created":1788727441}

This is the whole argument for refusing nothing, and it is now demonstrated rather than asserted: one payload, a 400 on Qwen and a 200 on four models elsewhere. A gateway-wide admission check on that shape would have broken all four.

The organization-prompt configuration

The configuration @hanakannzashi identified, run against the model rather than reasoned about. Same gateway and method, max_tokens: 16, 2026-09-07. The two requests carry the same three pieces of content and differ only in whether they are coalesced.

Coalesced — what this PR now produces:

messages: [ {role: system, content: "Follow the house style.\n\nYou are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 200

{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","reasoning_content":"Here's a thinking process:\n\n1.  **Analyze User Input:**"}}],
 "id":"8a1a2400e2fc4060a5f892d15815ed6c","model":"Qwen/Qwen3.6-35B-A3B-FP8","created":1788790715,"object":"chat.completion"}

Split — the shape the review describes, an organization prompt emitted on its own:

messages: [ {role: system, content: "Follow the house style."},
            {role: system, content: "You are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 400

{"error":{"message":"Provider failed for model 'Qwen/Qwen3.6-35B-A3B-FP8': System message must be at the beginning.","type":"invalid_request_error","param":null,"code":null}}

This is the first direct evidence that an organization prompt specifically triggers the failure — the earlier Qwen run used the instructions-plus-developer shape. Coalescing is confirmed to fix it against the real template rather than against a reading of it.

What this does and does not cover. All three runs exercise the provider-facing chat payload — the messages the gateway builds and sends upstream — which is the level the fold operates at, and it is the claim the PR rests on. None was driven end-to-end through /v1/responses against a deployed build of this branch. The joined system messages here are the two or three sources concatenated, without the language instruction and time context the real path also merges in; what is verified is the shape, a single leading system message, not those exact strings. Four models across two families is what was tested — it does not establish how every model the gateway serves treats a non-leading system message.

One thing left alone

Stored system-role history items. Conversation items stored with a system role are replayed with that role intact. Folding or filtering those is a call about what stored history means, and not one to make provider-agnostically without knowing you want it.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-06T19:20:06.076184Z	8ae1494	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ae1494ce2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Scope the rejection to providers that require it

This admission check applies before model/provider resolution, so it rejects every Responses request containing a system or developer input item, including requests routed to providers that accept or normalize multiple system-level messages. For example, the repository's Anthropic conversion path explicitly extracts MessageRole::System entries into Anthropic's top-level system field rather than returning a position error, while OpenAI-compatible providers may accept these messages directly. Such requests previously reached the provider and could succeed; they now unconditionally return 400. Move this check after provider resolution and apply it only to affected providers/models, or normalize their system messages safely.

Useful? React with 👍 / 👎.

[zmanian]

Agreed, and fixed in de3f4a7 — though by folding rather than by scoping. Confirmed the break is real: /v1/responses serves external models too (openai_tiered_pricing.rs drives it with openai/*). One correction to the mechanism: developer never reaches any provider as developer — to_chat_messages (completions/mod.rs:1468) maps "system" | "developer" => MessageRole::System for every backend, Anthropic conversion included. So the second system message is built here, not sent by the client, and the same payload is what OpenAI accepts today.

Rather than resolving the provider first, a system-level input message that precedes all other content (on a request replaying no history) is now folded into the leading system message the service already prepends. Nothing crosses it, so the ordering is unchanged, it stays provider-agnostic, and the Codex request works everywhere instead of failing cleanly on some backends. Only genuinely unfoldable shapes are refused at admission.

[claude]

Review — fail-fast for non-leading system messages

Thanks for the unusually thorough write-up; the diagnosis in build_messages is correct — crates/services/src/responses/service.rs:2283 unconditionally pushes a leading system message in both the Some(instructions) and else branches, so a system/developer item inside input can never be first in the provider payload. Turning a 200-then-response.failed into an admission-time 400 is the right instinct. Two issues below need resolving before merge; no prior review threads on this PR to build on (only the Codex bot summary).

⚠️ 1. The check is model-agnostic, so it rejects traffic that works today on external providers

CreateResponseRequest::validate() (crates/services/src/responses/models.rs:1161) has no access to the resolved provider — it runs before InferenceProviderPool selection. But /v1/responses routes to external providers too (provider_type: "external" → OpenAI / Anthropic / Gemini, see inference_provider_pool/mod.rs:1029 load_external_providers, and crates/api/tests/e2e_all/external_providers.rs). OpenAI's Responses API defines developer as a first-class input role and accepts it at any position; Anthropic and Gemini backends don't enforce leading-system either.

So the PR premise — "rejects exactly the set of requests that already fail today" — holds for vLLM chat templates that enforce leading-system, but not for the external path. Concretely: POST /v1/responses with model: "openai/gpt-4o", instructions: "...", and input[0].role = "developer" succeeds on main and returns 400 after this change. That's a hard break for any client hitting external models with the Codex-shaped payload.

Two ways out, either is fine:

• Preferred — fix instead of reject. Fold system-level input items into the leading system block in build_messages. For the reported Codex shape (developer at index 0, immediately after the synthesized instructions system message) this is order-preserving — the developer text lands exactly where it already sits relative to every other message — and it makes the reported request work rather than fail cleanly. Non-leading system items (index > 0, e.g. after a user turn) can't be hoisted without changing semantics; reject those.
• Or scope the check. Move it to where the provider/model is resolved and only apply it to backends that enforce leading-system.

⚠️ 2. The same failure is still reachable through two stored-item paths

The admission check only covers items in this request's input. Both replay paths forward role verbatim into the provider payload with no equivalent guard:

• service.rs:2451 — previous_response_id replay of stored response items.
• service.rs:2644 — stored conversation items (the POST /v1/conversations/{id}/items route accepts a system-role item; crates/api/tests/e2e_all/conversations.rs:1271 exercises exactly that).

Net effect: the request this PR 400s can be laundered through a conversation item and still produce 200 + mid-stream response.failed, which is the exact defect being fixed. You flagged the conversations case in the description — previous_response_id is the same class and worth naming too. This isn't a blocker on its own, but it does mean the bug isn't closed, and it argues for the hoist-at-build_messages approach in (1), which sits on the path all three sources converge on.

Minor

• Error text is inaccurate when instructions is absent. The message asserts the item "is placed after the system message derived from instructions", but with no instructions the leading system message comes from the else branch (language instruction + time context). Suggest phrasing it as "after the system message the service prepends" so it's true in both branches.
• Case sensitivity. SYSTEM_LEVEL_INPUT_ROLES.contains(&role) is exact-match, so "System" / "Developer" slip past admission and still fail mid-stream — the same UX this PR removes. role.eq_ignore_ascii_case(...) closes that.

What's good

• Validation placed on the existing create_response admission path that already returns 400 invalid_request_error, so no new error-shape surface.
• e2e test asserts content-type: application/json and the error envelope, not just the status — that's the property that actually distinguishes this from a mid-stream disconnect.
• Negative control (user_only_input_is_still_accepted) and the user/assistant unit case guard against over-rejection.
• No customer content in any log statement added.

⚠️

[ironloopai]

Review · Status

🟩 Completed

IronLoop completed the review and posted it to GitHub.

Result

Open submitted review →

Run details

• Run: d3c13e75-6d01-40b6-b917-339407738413
• Base: main at 5f68657
• Head: fix/responses-non-leading-system-message at 8ae1494
• Created: 2026-09-06 19:22 UTC
• Updated: 2026-09-06 19:26 UTC

_{Automatic trigger · attempt 1 of 3 · completed in 3m 24s}

[ironloopai]

Review · Summary

🟢 No actionable findings

No additional actionable findings were identified in the reviewed change.

Validation

• ✅ Change inspection — Reviewed all changed files and traced the new validation through response creation and provider-message conversion.
• ✅ Diff hygiene — No whitespace errors were found in the change.

Review details

• Run: d3c13e75-6d01-40b6-b917-339407738413
• Attempts: 1

[github-actions]

🔍 OpenCodeReview found 2 issue(s) in this PR.

• ✅ 2 posted as inline comment(s)
• 📝 0 posted as summary

[github-actions]

The role field in ResponseInputItem::Message is a raw String deserialized without any case normalization, and the role is forwarded verbatim to the provider (role: role.clone() in build_messages). The SYSTEM_LEVEL_INPUT_ROLES.contains(&role) check is case-sensitive, so a client sending "System", "Developer", or "SYSTEM" bypasses this admission-time validation entirely. If the downstream provider normalizes role case internally, the request would still hit the mid-stream "System message must be at the beginning" failure that this check was specifically designed to prevent. The check should compare case-insensitively.

Suggestion:

Suggested change

	if SYSTEM_LEVEL_INPUT_ROLES.contains(&role) {
	if SYSTEM_LEVEL_INPUT_ROLES.contains(&role.to_ascii_lowercase().as_str()) {

[zmanian]

Deliberately not taken, and now commented in the code with a unit test pinning it. The role does not reach the provider as a raw string: to_chat_messages (completions/mod.rs:1468) matches "system" | "developer" exactly and everything else falls to _ => MessageRole::User. So "System" or "Developer" becomes a user message, produces no system message at all, and cannot hit the mid-stream failure — comparing case-insensitively here would refuse a request that currently works.

If you would rather make that mapping itself case-insensitive, I am happy to; both places should then move together, and the pinning test flipped.

[github-actions]

The error message hardcodes "the system message derived from instructions", but build_messages (service.rs lines 2293–2301) also prepends a system message when instructions is None — synthesized from the default language instruction and time context. In that case the message is inaccurate and will confuse a client that never sent instructions. The suggestion to use the instructions field is still valid, but the diagnostic portion should account for the no-instructions path (e.g. "the system message derived from instructions or the default context"). All three new unit tests and both E2E tests set instructions, so this scenario is untested.

Suggestion:

Suggested change

	return Err(format!(
	"System message must be at the beginning. Input item at index {index} \
	has role '{role}', which is placed after the system message derived \
	from `instructions`. Send system-level content in the top-level \
	`instructions` field instead."
	));
	return Err(format!(
	"System message must be at the beginning. Input item at index {index} \
	has role '{role}', which is placed after the leading system message \
	(derived from `instructions` or the default context). Send system-level \
	content in the top-level `instructions` field instead."
	));

[zmanian]

Fixed in de3f4a7 — the wording is now "the system message the service prepends", which holds whether or not instructions is set. Thanks, this was accurate.

[zmanian]

Thanks — point 1 was right and I have reworked the PR around it. Pushed as de3f4a7.

On the break. Confirmed: /v1/responses does serve external models (openai_tiered_pricing.rs drives it with openai/*), so a blanket admission 400 would have refused traffic that works. One correction to the mechanism, which turns out to matter: developer never reaches OpenAI as developer. to_chat_messages (crates/services/src/completions/mod.rs:1468) maps "system" | "developer" => MessageRole::System for every backend, so the Codex payload reaches OpenAI as two system messages and is accepted there. That also means the second system message is built by this repo, not sent by the client — captured from the mock with the fix disabled:

[System("You are a coding agent.\n\nAlways respond in the exact same language..."),
 System("Repository guidelines."),
 User("Fix the build."),
 User("Then run the tests.")]

Now folds. A system-level input message that precedes all other content, on a request replaying no conversation / previous_response_id history, is appended to the system message load_conversation_context already prepends. Order-preserving, and the Codex request now works instead of failing cleanly. Only the unfoldable shapes are refused at admission: after other content, ahead-of-replayed-history, or carrying image content. The e2e test asserts the folded payload through mock.last_chat_params() — exactly one system message, first, instructions still leading, developer text present, both user turns intact.

Point 2, stored items. The history cases are now refused at admission rather than reaching the provider misplaced, so the laundering route you describe returns a 400. What remains is a replayed item that was itself stored with a system role: still forwarded as-is. Folding or filtering those is a call about what stored history means, and it seemed wrong to make it unasked — happy to take it in this PR if you want it, otherwise flagging it as still open.

On case sensitivity — I deliberately did not take this one, and added a comment saying why. Since to_chat_messages matches exactly, "Developer" falls through to MessageRole::User and produces no system message to misplace; eq_ignore_ascii_case here would refuse a request that currently works. There is a unit test pinning that. Glad to switch if you would rather make the mapping itself case-insensitive first, in which case both should move together.

Error text reworded to "the system message the service prepends", which is true whether or not instructions is set.

Full local run against PostgreSQL 15 in Docker: fmt and clippy clean, --lib --bins 647 + 435 + 6 crates all green, integration_tests 8 passed, e2e_all 737 passed, 0 failed, 10 ignored.

[zmanian]

Narrowed again in b685496 — the PR now refuses nothing.

Reviewing my own previous revision, the objection about admission-time checks not knowing the provider reaches one step further than my fix did. CreateResponseRequest::validate still returned a 400 for three shapes on every model, and that check was a pure role-and-position test with no provider scoping. The constraint it encoded belongs to one chat template: "System message must be at the beginning." is a raise_exception in Qwen 3.6's official Jinja template, and the identical Codex request succeeds against DeepSeek and GLM adapters. So a request that works today on those families would have started returning 400.

The asymmetry is what settled it: the fold is provider-agnostic because it preserves what the model is told, so it is correct on any backend. A rejection is provider-specific reasoning applied provider-agnostically. Those three shapes are now forwarded unchanged — nothing that works today breaks, Qwen behaves exactly as it does now, and the Codex fix is untouched because it comes from the fold.

One correction to something I said earlier: previous_response_id alone replays no history — it only selects a branch within a conversation. A request carrying just it is folded, and the test covering replayed history now sets up a conversation. The fold guard tests the messages actually built rather than the request fields, so pass-through needed no new code path.

Tests that asserted a 400 now assert pass-through at the provider payload rather than only at validate: a system message after user content arrives as [System, User, System], still third; a developer message behind replayed history arrives as its own system message after that history, unfolded.

One thing worth your attention before merge. Everything I have is from the mock provider and reading the code — I could not get NEAR AI credentials in this environment, so "the folded payload satisfies the real Qwen template" is inference from the error text and the public template, not an observation. It is the PR's load-bearing claim and you can settle it in one request: send the two-system-message payload to Qwen/Qwen3.6-35B-A3B-FP8 and confirm it fails, then the folded single-system-message payload and confirm it succeeds. Flagging rather than implying I checked.

Full local run against PostgreSQL 15 in Docker: fmt and clippy clean, --lib --bins 647 + 435 + 6 crates green, integration_tests 8 passed, e2e_all 738 passed, 0 failed, 10 ignored.

[zmanian]

Settled — the verification I asked you to do before merge is done, so please disregard that ask. The fold is confirmed against the real model, and the PR body now carries the transcript.

Run on 2026-09-06 against Qwen/Qwen3.6-35B-A3B-FP8 via https://cloud-api.near.ai/v1/chat/completions, max_tokens: 16, two requests differing only in whether the two system messages are folded.

A — the unfolded shape the gateway builds today:

messages: [ {role: system, content: "You are a coding agent."},
            {role: system, content: "Repository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 400

{"error":{"message":"Provider failed for model 'Qwen/Qwen3.6-35B-A3B-FP8': System message must be at the beginning.","type":"invalid_request_error","param":null,"code":null}}

B — the folded shape this PR produces:

messages: [ {role: system, content: "You are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 200

{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","reasoning_content":"Here's a thinking process:\n\n1.  **Analyze User Input:**"}}],
 "id":"a4035f807e104471a02fbf01a5604c5f","model":"Qwen/Qwen3.6-35B-A3B-FP8","object":"chat.completion",
 "created":1788726154,"usage":{"prompt_tokens":28,"completion_tokens":16}}

The error string in A is character-for-character the one in #1025.

Being precise about the scope: this exercises the provider-facing chat payload — the messages the gateway builds and sends upstream, which is the level the fold operates at — not an end-to-end /v1/responses call against a deployed build of this branch. The folded system message here is just the two contents joined, without the language instruction and time context the real path also merges in, so what is verified is the shape (a single leading system message), not that exact string.

One consequence worth your attention. Result A rejects a system message at index 1, not merely one trailing user content — so the template admits a system message only at position 0. That turns the organization-system-prompt case from a guess into demonstrated behavior: when an org prompt is configured, load_conversation_context emits it as its own system message ahead of the instructions one, so the Codex shape becomes three system messages and folds to two — payload A exactly, and the same 400. I have left it out as beyond the reported defect, but folding the org prompt into the same leading message would close it and I am glad to add it here if you would rather have one change than two.

No code change since b685496; CI is green on it.

[zmanian]

The DeepSeek/GLM half is now verified too, so both sides of the argument are transcripts rather than assertions. PR body updated; no code change, still b685496.

Same gateway and method as the Qwen run, max_tokens: 8, 2026-09-06. The request is byte-identical to payload A — the two-system-message shape, the second not at index 0, which Qwen refuses with a 400:

messages: [ {role: system, content: "You are a coding agent."},
            {role: system, content: "Repository guidelines."},
            {role: user,   content: "Say OK."} ]

Four models, two families, all HTTP 200:

• deepseek-ai/DeepSeek-V4-Flash — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK."}}],"id":"520dead424fa4813b2e808f31c8d54e0","created":1788727428}
• deepseek/deepseek-v3.2 — {"id":"c3173b1436a54795bdf9cd875c213964","created":1788727439,"choices":[{"index":0,"message":{"role":"assistant","content":"OK."}}]}
• zai-org/GLM-5.1-FP8 — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK"}}],"id":"9173ec431fb84a1d970388dd9e970974","created":1788727429}
• z-ai/glm-5.2 — {"choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"OK."}}],"id":"1645a61c377f445aa67f2a95fea7f4b2","created":1788727441}

One payload, a 400 on Qwen and a 200 on four models elsewhere. That is the case for the PR refusing nothing: a gateway-wide admission check on this shape — which is what my first revision did — would have broken all four.

Same limits as before, stated rather than glossed: both runs exercise the provider-facing chat payload, not an end-to-end /v1/responses call against a deployed build of this branch, and four models across two families is what was tested — it says nothing about how every model the gateway serves treats a non-leading system message.

[hanakannzashi]

The fold still leaves the reported failure reachable for organizations with a configured system prompt.

load_conversation_context first emits system(org prompt) and then emits the request/default system context. The new code appends a leading developer message only to the latter, so the provider still receives:

system(org prompt), system(instructions + developer), user(...)

The live Qwen result in this PR establishes that a system message at index 1 is rejected, so this supported configuration still produces the same 200-then-response.failed behavior. Could we coalesce the organization prompt, generated request context, and foldable leading system/developer input into one leading system message here, with an E2E test? I think this should be fixed before merging the PR.

[zmanian]

@hanakannzashi you are right, and thank you — this is fixed in a325aab.

Your reading of the code was exactly correct, including the payload. Reverting the fix locally and re-running the new test reproduces the shape you described, character for character:

[ChatMessage { role: System, content: Some(String("Follow the house style.")) },
 ChatMessage { role: System, content: Some(String("You are a coding agent.\n\nAlways respond in the exact same language...\n\nCurrent UTC time: ...\n\nRepository guidelines.")) },
 ChatMessage { role: User,   content: Some(String("Fix the build.")) }]

The fold appended the developer message to the second system message, exactly as you said, leaving the organization prompt at index 0 and the request context at index 1.

What changed. The organization prompt, the request context (instructions, language instruction, time context) and any foldable leading system-level input message are now coalesced into one leading system message. The pieces keep the order they had as separate messages — organization prompt, then instructions, then language instruction, then time context, then the folded input — so the model is told the same things in the same sequence. That order-preservation is what makes the fold safe on backends that accept either shape, and coalescing a third source does not change it.

Nothing new is refused. A shape that still cannot be folded — a system-level input message behind replayed conversation history, say — is forwarded unchanged, as before.

The E2E test you asked for is org_prompt_and_leading_developer_message_coalesce_into_one_system_message, asserting the upstream payload via mock.last_chat_params(): an organization prompt configured plus a Codex-shaped request with a leading developer message must produce exactly one system message, first, with all three sources present in that order. The transcript above is that test failing with the coalescing reverted, so it is not asserting something already true.

Full local run against PostgreSQL 15 in Docker: fmt and clippy clean, --lib --bins 647 + 435 + 6 crates green, integration_tests 8 passed, e2e_all 739 passed, 0 failed, 10 ignored.

One note on the evidence, since you reasoned from it: the Qwen result in this PR was captured at the chat-payload level rather than end-to-end through /v1/responses, so "a system message at index 1 is rejected" is established for the payload the gateway sends upstream, which is the level this fix operates at. If you would like the coalesced organization-prompt shape confirmed against Qwen/Qwen3.6-35B-A3B-FP8 directly, we are glad to run it and post the transcript.

[zmanian]

@hanakannzashi following up with the live confirmation — your configuration is now reproduced against the model, not just against our reading of the code. Both directions, Qwen/Qwen3.6-35B-A3B-FP8 via https://cloud-api.near.ai/v1/chat/completions, max_tokens: 16, 2026-09-07. Added to the PR body under "The organization-prompt configuration".

Coalesced — what the PR now produces:

messages: [ {role: system, content: "Follow the house style.\n\nYou are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 200

{"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","reasoning_content":"Here's a thinking process:\n\n1.  **Analyze User Input:**"}}],
 "id":"8a1a2400e2fc4060a5f892d15815ed6c","model":"Qwen/Qwen3.6-35B-A3B-FP8","created":1788790715,"object":"chat.completion"}

Split — the shape you described, same content emitted as two system messages:

messages: [ {role: system, content: "Follow the house style."},
            {role: system, content: "You are a coding agent.\n\nRepository guidelines."},
            {role: user,   content: "Say OK."} ]

HTTP 400

{"error":{"message":"Provider failed for model 'Qwen/Qwen3.6-35B-A3B-FP8': System message must be at the beginning.","type":"invalid_request_error","param":null,"code":null}}

Worth noting this is the first direct evidence that an organization prompt specifically triggers the failure — the earlier Qwen transcript in this PR used the instructions-plus-developer shape, and you inferred the organization-prompt case from it. The inference was correct.

Credit where it is due: you found this by reading the code without running it, your description of the resulting payload matched our reproduction character for character, and you got there by reasoning from a transcript we had attached for a different purpose. That is a better catch than the one this PR started as.

Same limits as the other transcripts, stated rather than glossed: this exercises the provider-facing chat payload, not an end-to-end /v1/responses call against a deployed build of the branch, and the joined content here is the three sources concatenated — not the exact production string, which also carries the language instruction and time context between them. What is verified is the shape: a single leading system message is accepted where a split one is not.