fix(local): disable thinking on Ollama's OpenAI-compat endpoint (hybrid reasoning models) - #358
Open
alekseikondratenko wants to merge 2 commits into
Conversation
Hybrid reasoning models (Qwen3.5 family) emit a full thinking stream by default on Ollama's /v1/chat/completions, and Ollama has no model-level or server-level way to turn it off. Each describe_frame call then burns 500-900 reasoning tokens before the 1-2 sentence answer, taking a batch from ~6 minutes to 15+ minutes on Apple Silicon - the analysis queue never catches up. Send reasoning_effort: "none" when the engine is Ollama. Non-thinking models ignore the field; nil is never encoded for LM Studio or custom endpoints, which have their own thinking controls and in some cases reject "none". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified LM Studio 0.4.21 honors reasoning_effort: "none" for hybrid models (Qwen3.5-9B MLX, 26s/70 tokens vs 120s/1800+ thinking tokens), while its template kwargs are not honored via the server API - so the same guard applies to both local engines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Updated: extended the same guard to the LM Studio engine after verifying LM Studio 0.4.21 also honors |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Local-engine analysis silently degrades ~5-10x when the configured Ollama model is a hybrid reasoning model (any of the Qwen3.5 family, and other recent releases that ship with thinking enabled).
Dayflow talks to Ollama through the OpenAI-compatible
/v1/chat/completionsendpoint, and on that endpoint Ollama enables the thinking stream by default for these models. There is no model-level or server-level way to turn it off on the Ollama side:PARAMETER think falsein a Modelfile is not supported (Feature request: PARAMETER think false in Modelfile for Qwen 3.5 / thinking models ollama/ollama#14809, How to completely disable the thinking mode by configuring the modelfile in Qwen3.5 ollama/ollama#14617)/no_thinksoft switch is ignored by Qwen3.5Since
ChatRequestis a fixed struct, users have no workaround short of switching models.Measured on an M4 MacBook Air (24 GB),
qwen3.5:9b, real Dayflow screenshots:reasoning_effort: "none"At 15 sampled frames per batch, that is the difference between a batch finishing in ~6 minutes and a batch taking longer than the 15-minute window it covers - the analysis queue falls behind permanently and the machine stays hot.
Fix
Send
reasoning_effort: "none"when (and only when) the local engine is Ollama. Ollama's OpenAI-compat layer honors this field (verified against Ollama 0.32.14); non-thinking models such as the recommendedqwen3-vl:4bpreset simply ignore it.The field is an optional on
ChatRequestthat staysnilfor LM Studio and custom endpoints, andJSONEncodernever encodes it in that case - so those request bodies are byte-identical to today's. LM Studio has its own per-model thinking controls, and some OpenAI-compatible providers reject"none", which is why the field is gated to the Ollama engine rather than sent unconditionally.Testing
reasoning_effort: "none"disables the reasoning stream and answer quality is unchanged, via direct/v1/chat/completionscalls against Ollama 0.32.14 withqwen3.5:9bon real Dayflow screenshot batches (numbers above).nil, the encoded body is unchanged from the current one.makeChatURLRequest. Happy to adjust if CI or a maintainer build turns anything up.🤖 Generated with Claude Code