Skip to content

feat(server): add --default-thinking-mode flag for server-wide thinking default - #476

Open
rafaeldrincon wants to merge 1 commit into
FlashML-org:mainfrom
rafaeldrincon:feat/default-thinking-mode
Open

rafaeldrincon wants to merge 1 commit into
FlashML-org:mainfrom
rafaeldrincon:feat/default-thinking-mode

Conversation

@rafaeldrincon

Copy link
Copy Markdown

Problem

OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI SDK openai-compatible provider, llama-swap, LiteLLM proxies) get the model's template default, which for reasoning models like Qwen3.6 means thinking is always on. The token budget goes to reasoning_content and content comes back empty.

Measured on real hardware: a 200-token max_tokens budget produced 199 reasoning tokens and 0 content tokens; the client saw an empty answer.

Current workaround (--reasoning-parser off) just dumps thinking into content — still wasting the budget.

Solution

--default-thinking-mode {auto,chat,thinking} server flag:

  • auto (default): current behavior, the template decides
  • chat: fills enable_thinking=False into every request that does not set any explicit thinking control
  • thinking: fills enable_thinking=True the same way

An explicit per-request value (enable_thinking / thinking / thinking_mode in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always wins over the server default — the flag only fills what the request left unset.

This mirrors llama.cpp's --jinja template behavior where the server owns the default and clients can override per-request.

Implementation

  • ServerArgs.default_thinking_mode + argparse entry (server/args.py)
  • apply_default_thinking_mode() merge helper in server/openai_api.py
  • Applied at all three frontends: OpenAI chat completions, Anthropic messages, Responses API
  • 6 unit tests in tests/server/test_default_thinking_mode.py (verified all pass)

Closes #472

Test environment

ft version: 0.1.2
model: nvidia/Qwen3.6-35B-A3B-NVFP4
gpu: RTX 3080 Ti Laptop (16GB, sm_86)
cuda: 13.1, driver: 580

With --default-thinking-mode chat the same Understory agent workload that previously returned empty content produces correct answers with no client changes.

…ng default

OpenAI-compatible clients that never send chat_template_kwargs (Vercel AI
SDK openai-compatible provider, llama-swap, LiteLLM proxies) currently get
the model's template default, which for reasoning models like Qwen3.6 means
thinking is always on — the token budget goes to reasoning_content and
content comes back empty.

--default-thinking-mode {auto,chat,thinking} lets the operator pick the
server-wide default:

- auto (default): current behavior, template decides
- chat: fills enable_thinking=False into every request that does not set
  any explicit thinking control
- thinking: fills enable_thinking=True the same way

An explicit per-request value (enable_thinking / thinking / thinking_mode
in chat_template_kwargs, or the DeepSeek thinking.type wire toggle) always
wins over the server default.

Applied to all three frontends: OpenAI chat completions, Anthropic
messages, and Responses API.

Closes FlashML-org#472
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(server): server-wide default thinking mode flag (--default-thinking-mode)

1 participant