Repository navigation
Conversation
…aptive-only models Running Artemis with claude-sonnet-5-5 failed mid-task with: 400 messages.3.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation. Sonnet 5.5 (like Opus 5.5, Haiku 5.5 and Fable 5.1) binds each thinking block to the exact history that produced it. The transcript ledger scrubs old UI lists and screenshots from earlier turns, so the next request replays a thinking block whose prefix changed and the API rejects it. The Anthropic request builder also sent shapes these models reject: temperature 0.0 (non-default sampling is a 400), thinking.budget_tokens from thinking_budget / reasoning_effort (removed, 400), and forced tool_choice from with_structured_output (400). langchain-anthropic also falls back to max_tokens=4096 for models it has no profile for, which thinking can exhaust before the tool call. For adaptive-only Claude models (Opus 4.7+, Sonnet 5+, Haiku 5+, Fable, Mythos): - omit temperature and set max_tokens to 16000 - map reasoning_effort / thinking_budget to output_config.effort with adaptive thinking - on models that bind thinking blocks, send thinking.block_binding.prefix_mismatch_behavior="drop_block" with the thinking-binding-controls-2026-08-01 beta, so blocks invalidated by history scrubbing are dropped instead of failing the request - default with_structured_output to method="json_schema" Older Claude ids (claude-3-7-sonnet, Sonnet/Opus 4.6 and earlier) keep the existing request shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
vvinogra
marked this pull request as draft
November 1, 2026 10:16
With claude-sonnet-5-5, a multi-turn Flash task looped from turn 3 on:
every request returned HTTP 200 with stop_reason "refusal" and no
content, and the runner nudged and re-sent the same prompt each turn.
Sonnet 5.5 returns its between-tool narration as thinking blocks, so the
ledger sees every turn as silent and appends REASONING_REMINDER. Its
wording ("internal thinking is not shown ... write ...") trips the
reasoning_extraction safety classifier (reproduced 1/1 live; reworded
prompt: 4/4 normal tool calls, with visible text).
- Reword REASONING_REMINDER as a request for a short progress note.
- Raise LLMPermanentError with a new REFUSAL failure category when a
response has stop_reason "refusal", so the fallback endpoint is tried
and a repeated refusal fails the task instead of looping.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running Artemis with
claude-sonnet-5-5fails partway through a task:The first few turns succeed. The task dies on the first request after the transcript ledger has edited an earlier turn.
Root cause
Claude Sonnet 5.5 (and Opus 5.5, Haiku 5.5, Fable 5.1) uses preserved thinking: each thinking block is bound to the exact
system/tools/ earliermessagesthat produced it. Replaying a block after any of that changes is a 400.TranscriptLedgerchanges earlier turns on purpose. It strips old UI lists at depth 1 and screenshots at depth K. So once one observation has been scrubbed (messages.2.content.4above), the thinking block in the next assistant turn no longer matches, and the request fails.The same request builder (
ModelFactory.create_model, Anthropic branch) also sends three other shapes these models reject. Users have been patching around them in config (setting"temperature": 1.0on every node and never settingthinking_budget/reasoning_effort):temperature: 0.0(theModelEndpointdefault)thinking: {type: "enabled", budget_tokens: N}fromthinking_budget/reasoning_effortbudget_tokensremoved → 400with_structured_output(...), which defaults to forcedtool_choice(checker, planner validation)tool_choice→ 400 on Sonnet 5.5 / Opus 5.5 / Fable 5.1max_tokens, so langchain-anthropic 1.5.6 falls back to 4096 (it has no profile for the 5.5 models)max_tokens, so a turn can run out before it emits the tool callFix
A new
artemis/llm/claude.pyparses Claude model ids, following the pattern ofartemis/llm/google/provider.py. It answers two questions:For adaptive-only models, the router now:
temperaturereasoning_effort/thinking_budgettooutput_config.effortwiththinking: {type: "adaptive"}(budget buckets 2048 / 8192 / 32768 map to low / medium / high, the same thresholds as the existing budget path)max_tokenstoendpoint.max_tokensor 16000. That leaves room for thinking and stays under the SDK's non-streaming limit, since structured-output calls don't stream.thinking.block_binding.prefix_mismatch_behavior: "drop_block"with thethinking-binding-controls-2026-08-01beta header. The API then drops thinking blocks that history scrubbing made stale, and the request goes through. This is the documented approach for agents that prune screenshots client-side. It keeps the ledger's token-saving scrubbing; making the ledger append-only would be a much bigger change.RobustChatModelWrapper.with_structured_outputdefaults tomethod="json_schema"(native structured outputs) for adaptive-only Claude models. An explicitmethodis still respected.Legacy ids (
claude-3-7-sonnet-*, Sonnet/Opus 4.6 and earlier) keep the existing request shape, includingbudget_tokensandtemperature=1.0.Second issue: refusal loop on the reasoning reminder
With the request shape fixed, a multi-turn Flash run on Sonnet 5.5 looped from turn 3 on. Every request returned HTTP 200 with
stop_reason: "refusal"and no content, and the runner nudged and re-sent the same prompt each turn (about 15k input tokens a turn).thinkingblocks, which are empty by default. So the ledger sees every turn as silent and appendsREASONING_REMINDER.reasoning_extractionsafety classifier. Reproduced live: original wording 1/1 refused; the new wording 4/4 normal tool calls, with visible text.Fix:
REASONING_REMINDERnow asks for a short progress note instead of the model's reasoning.RobustChatModelWrapperraisesLLMPermanentErrorwith a newFailureCategory.REFUSAL(not retryable, may fall back) when a response hasstop_reason: "refusal". The fallback endpoint is tried, and a repeated refusal fails the task with the reason instead of looping.Testing
tests/unit/test_claude_models.py(29 tests):httpx.MockTransport. For Sonnet 5.5 it shows notemperature,thinking.block_bindingintact after SDK serialization, the beta header present, and the prior thinking block replayed unchanged.main(missingGOOGLE_API_KEY, mocked providers), and 2442 passed vs 2413 onmain.ruff check,ruff format --check,pyright --project pyright-core.json: clean.400 Invalid signature in thinking block … bound to a different conversation(the production error)input_transformations: [{type: thinking_dropped, reason: prefix_binding_mismatch}]400 temperature is deprecated for this model🤖 Generated with Claude Code