feat(server): add --default-thinking-mode flag for server-wide thinking default - #476
Open
rafaeldrincon wants to merge 1 commit into
Open
rafaeldrincon wants to merge 1 commit into
rafaeldrincon wants to merge 1 commit into
Conversation
…ng default
OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI
SDK openai-compatible provider, llama-swap, LiteLLM proxies) currently get
the model's template default, which for reasoning models like Qwen3.6 means
thinking is always on — the token budget goes to reasoning_content and
content comes back empty.
--default-thinking-mode {auto,chat,thinking} lets the operator pick the
server-wide default:
- auto (default): current behavior, template decides
- chat: fills enable_thinking=False into every request that does not set
any explicit thinking control
- thinking: fills enable_thinking=True the same way
An explicit per-request value (enable_thinking / thinking / thinking_mode
in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always
wins over the server default.
Applied to all three frontends: OpenAI chat completions, Anthropic
messages, and Responses API.
Closes FlashML-org#472
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
OpenAI-compatible clients that never send
chat_template_kwargs(Vercel AI SDKopenai-compatibleprovider, llama-swap, LiteLLM proxies) get the model's template default, which for reasoning models like Qwen3.6 means thinking is always on. The token budget goes toreasoning_contentandcontentcomes back empty.Measured on real hardware: a 200-token
max_tokensbudget produced 199 reasoning tokens and 0 content tokens; the client saw an empty answer.Current workaround (
--reasoning-parser off) just dumps thinking intocontent— still wasting the budget.Solution
--default-thinking-mode {auto,chat,thinking}server flag:auto(default): current behavior, the template decideschat: fillsenable_thinking=Falseinto every request that does not set any explicit thinking controlthinking: fillsenable_thinking=Truethe same wayAn explicit per-request value (
enable_thinking/thinking/thinking_modeinchat_template_kwargs, or the DeepSeekthinking.typewire toggle) always wins over the server default — the flag only fills what the request left unset.This mirrors llama.cpp's
--jinjatemplate behavior where the server owns the default and clients can override per-request.Implementation
ServerArgs.default_thinking_mode+ argparse entry (server/args.py)apply_default_thinking_mode()merge helper inserver/openai_api.pytests/server/test_default_thinking_mode.py(verified all pass)Closes #472
Test environment
With
--default-thinking-mode chatthe same Understory agent workload that previously returned emptycontentproduces correct answers with no client changes.