fix: thinking not actually disabled for DeepSeek models (bench reads ~35% low) - #54
Open
y1618 wants to merge 1 commit into
Open
fix: thinking not actually disabled for DeepSeek models (bench reads ~35% low)#54y1618 wants to merge 1 commit into
y1618 wants to merge 1 commit into
Conversation
The decode bench and showcase disable thinking via
`chat_template_kwargs.enable_thinking`, but DeepSeek V4 templates only
honor `thinking` and silently ignore `enable_thinking`. With a server
default of `{"thinking":true,"reasoning_effort":"max"}` the whole bench
window is spent inside `<think>`, where MTP acceptance is much lower —
on a 2x DGX Spark DeepSeek-V4-Flash stack the c=1 decode bench read
44.1 tok/s; with thinking actually off it reads 70.4 tok/s.
Map deepseek model ids to `thinking` in applyThinkingFlags (keeping
enable_thinking for everyone else), strip it in stripThinkingFlags for
the 400-retry path, and add unit tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NRSGQ8VRdFgB76tz9JbtMX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
applyThinkingFlagsdisables thinking viachat_template_kwargs.enable_thinking, but DeepSeek V4 chat templates only honorthinkingand silently ignoreenable_thinking.When the server is launched with a thinking-on default (the DSpark stacks use
--default-chat-template-kwargs '{"thinking":true,"reasoning_effort":"max"}'), the decode bench's entire fixed-length window is spent inside<think>. Reasoning text has much lower MTP/speculative acceptance, so the bench under-reads sustained decode throughput. The recorded bench results confirm it:reasoningChunks: 175on a 512-token c=1 stream that was supposed to have thinking off.Measured on a 2x DGX Spark DeepSeek-V4-Flash-0731 stack (c=1, 512 tokens, temp 0):
enable_thinking:false, ignored)thinking:false)Fix
applyThinkingFlags: model ids containingdeepseekalso getchat_template_kwargs.thinking(everyone else keepsenable_thinking, mirroring the existing MiniMaxthinking_modespecial case)stripThinkingFlags: also removesthinkingso the HTTP-400 retry path stays cleanLlmStreaming.thinking.test.js, 5 cases)node --test 'server/collectors/__tests__/*.test.js': 110 pass, 1 pre-existing unrelated failure (ComfyProbe.test.jsneeds thewspackage outside the container).🤖 Generated with Claude Code
https://claude.ai/code/session_01NRSGQ8VRdFgB76tz9JbtMX