Repository navigation
fix: strip reasoning toggles from upstream requests (thinking variant exhausted max_tokens) - #5
Merged
Merged
Conversation
…requests Production forensics (2026-10-04 frame capture): requests carrying reasoning_effort route the CNB gateway to its thinking variant, which counts the reasoning phase against max_tokens — once reasoning exhausts the budget the stream ends with finish_reason=length, content empty (1024 frames, all reasoning_only), and the answer is never generated. The upstream offers no correct knob to configure, so the gateway now strips reasoning_effort / reasoning / enable_thinking / thinking at the single outbound choke point in forward() (covers both the OpenAI passthrough path and the Anthropic /v1/messages adapter), the same policy the adapter already applies by dropping thinking blocks. Tests: outbound-strip assertions against a body-capturing mock upstream, thinking-param non-leakage through the Anthropic adapter, and lock-in tests for the reasoning→thinking block ordering plus the degenerate reasoning-only-truncated stream shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (production forensics, 2026-10-04)
Raw upstream frame capture of a failing request:
Requests carrying
reasoning_effortroute the CNB gateway to a thinking variant whose reasoning phase is counted againstmax_tokens. On hard prompts reasoning consumes the entire budget before any answer text is produced, so the stream ends withfinish_reason=lengthandcontent=[]— the answer is never generated. Reproduced deterministically at both 2048 and 8192 max_tokens (hard prompt), and confirmed the fix condition: withoutreasoning_effortthe same prompt returns a normal content-only answer.The OpenAI passthrough path forwarded client
reasoning_effortverbatim ({ ...payload, stream: true }), which is how real clients (CPA panel / cpa-cnb) hit this every time. The Anthropic/v1/messagesadapter was unaffected by construction (it never emits thinking params) — the same "no thinking outbound" policy this PR now applies to both paths.Fix
Single choke point in
forward()(covers OpenAI passthrough + Anthropic adapter): stripreasoning_effort,reasoning,enable_thinking,thinkingfrom the outbound body. Table-driven so new gateway thinking toggles are a one-line addition.Response side intentionally unchanged: if the thinking variant is ever hit,
reasoning_content→thinkingblock conversion andfinish_reason=length → max_tokensstop-reason mapping remain faithful (lock-in tests added; no fabricated answer text).Tests
reasoning_effort: 'medium'leaking through)thinking/reasoning_effort/reasoning/enable_thinking) never leak outboundreasoning_content→ thinking block ordering before text block; degenerate reasoning-only truncated stream yields a single thinking block closed withstop_reason: max_tokens, no fabricated text