Skip to content

fix: make Artemis work with Claude Sonnet 5.5 (request shape, preserved thinking, refusal loop) - #231

Draft
vvinogra wants to merge 2 commits into
google:mainfrom
vvinogra:fix/claude-5-5-compat
Draft

vvinogra wants to merge 2 commits into
google:mainfrom
vvinogra:fix/claude-5-5-compat

Conversation

@vvinogra

@vvinogra vvinogra commented Oct 31, 2026 •

Copy link
Copy Markdown

Problem

Running Artemis with claude-sonnet-5-5 fails partway through a task:

LLMPermanentError: 400 invalid_request_error: messages.3.content.0: Invalid `signature` in `thinking` block.
The block is bound to a different conversation. ... Content that preceded this block when it was created
is missing from this request, starting at `messages.2.content.4`.

The first few turns succeed. The task dies on the first request after the transcript ledger has edited an earlier turn.

Root cause

Claude Sonnet 5.5 (and Opus 5.5, Haiku 5.5, Fable 5.1) uses preserved thinking: each thinking block is bound to the exact system / tools / earlier messages that produced it. Replaying a block after any of that changes is a 400.

TranscriptLedger changes earlier turns on purpose. It strips old UI lists at depth 1 and screenshots at depth K. So once one observation has been scrubbed (messages.2.content.4 above), the thinking block in the next assistant turn no longer matches, and the request fails.

The same request builder (ModelFactory.create_model, Anthropic branch) also sends three other shapes these models reject. Users have been patching around them in config (setting "temperature": 1.0 on every node and never setting thinking_budget / reasoning_effort):

Sent today On Sonnet 5.5 / Opus 4.7+
temperature: 0.0 (the ModelEndpoint default) non-default sampling → 400
thinking: {type: "enabled", budget_tokens: N} from thinking_budget / reasoning_effort budget_tokens removed → 400
with_structured_output(...), which defaults to forced tool_choice (checker, planner validation) forced tool_choice → 400 on Sonnet 5.5 / Opus 5.5 / Fable 5.1
no max_tokens, so langchain-anthropic 1.5.6 falls back to 4096 (it has no profile for the 5.5 models) adaptive thinking counts toward max_tokens, so a turn can run out before it emits the tool call

Fix

A new artemis/llm/claude.py parses Claude model ids, following the pattern of artemis/llm/google/provider.py. It answers two questions:

  • is the model adaptive-only (Opus 4.7+, Sonnet 5+, Haiku 5+, Fable, Mythos)?
  • does it bind thinking blocks to their history (Sonnet 5.5, Opus 5.5, Haiku 5.5, Fable/Mythos 5.1+)?

For adaptive-only models, the router now:

  • omits temperature
  • maps reasoning_effort / thinking_budget to output_config.effort with thinking: {type: "adaptive"} (budget buckets 2048 / 8192 / 32768 map to low / medium / high, the same thresholds as the existing budget path)
  • sets max_tokens to endpoint.max_tokens or 16000. That leaves room for thinking and stays under the SDK's non-streaming limit, since structured-output calls don't stream.
  • on models that bind thinking blocks, sends thinking.block_binding.prefix_mismatch_behavior: "drop_block" with the thinking-binding-controls-2026-08-01 beta header. The API then drops thinking blocks that history scrubbing made stale, and the request goes through. This is the documented approach for agents that prune screenshots client-side. It keeps the ledger's token-saving scrubbing; making the ledger append-only would be a much bigger change.

RobustChatModelWrapper.with_structured_output defaults to method="json_schema" (native structured outputs) for adaptive-only Claude models. An explicit method is still respected.

Legacy ids (claude-3-7-sonnet-*, Sonnet/Opus 4.6 and earlier) keep the existing request shape, including budget_tokens and temperature=1.0.

Second issue: refusal loop on the reasoning reminder

With the request shape fixed, a multi-turn Flash run on Sonnet 5.5 looped from turn 3 on. Every request returned HTTP 200 with stop_reason: "refusal" and no content, and the runner nudged and re-sent the same prompt each turn (about 15k input tokens a turn).

  • Sonnet 5.5 returns its between-tool narration as thinking blocks, which are empty by default. So the ledger sees every turn as silent and appends REASONING_REMINDER.
  • The reminder's wording ("internal thinking is not shown … write …") trips the reasoning_extraction safety classifier. Reproduced live: original wording 1/1 refused; the new wording 4/4 normal tool calls, with visible text.

Fix:

  • REASONING_REMINDER now asks for a short progress note instead of the model's reasoning.
  • RobustChatModelWrapper raises LLMPermanentError with a new FailureCategory.REFUSAL (not retryable, may fall back) when a response has stop_reason: "refusal". The fallback endpoint is tried, and a repeated refusal fails the task with the reason instead of looping.

Testing

  • New tests/unit/test_claude_models.py (29 tests):
    • id parsing (Bedrock prefix, date snapshots, legacy order)
    • the capability table
    • effort mapping
    • legacy path unchanged
    • structured-output method selection
    • the real HTTP request captured through an httpx.MockTransport. For Sonnet 5.5 it shows no temperature, thinking.block_binding intact after SDK serialization, the beta header present, and the prior thinking block replayed unchanged.
  • Full unit suite: same 80 environment-dependent failures as main (missing GOOGLE_API_KEY, mocked providers), and 2442 passed vs 2413 on main.
  • ruff check, ruff format --check, pyright --project pyright-core.json: clean.
  • Live API, Sonnet 5.5, a thinking block replayed after the earlier observation was scrubbed:
    • old request shape: 400 Invalid signature in thinking block … bound to a different conversation (the production error)
    • fixed shape: 200, input_transformations: [{type: thinking_dropped, reason: prefix_binding_mismatch}]
    • old shape also gets 400 temperature is deprecated for this model
  • Live Artemis Flash runs on an Android emulator with an all-Claude config (Sonnet 5.5, Haiku 5.5 fallback): see the PR comments for the end-to-end result.

🤖 Generated with Claude Code

…aptive-only models

Running Artemis with claude-sonnet-5-5 failed mid-task with:

  400 messages.3.content.0: Invalid `signature` in `thinking` block. The
  block is bound to a different conversation.

Sonnet 5.5 (like Opus 5.5, Haiku 5.5 and Fable 5.1) binds each thinking
block to the exact history that produced it. The transcript ledger scrubs
old UI lists and screenshots from earlier turns, so the next request
replays a thinking block whose prefix changed and the API rejects it.

The Anthropic request builder also sent shapes these models reject:
temperature 0.0 (non-default sampling is a 400), thinking.budget_tokens
from thinking_budget / reasoning_effort (removed, 400), and forced
tool_choice from with_structured_output (400). langchain-anthropic also
falls back to max_tokens=4096 for models it has no profile for, which
thinking can exhaust before the tool call.

For adaptive-only Claude models (Opus 4.7+, Sonnet 5+, Haiku 5+, Fable,
Mythos):
- omit temperature and set max_tokens to 16000
- map reasoning_effort / thinking_budget to output_config.effort with
  adaptive thinking
- on models that bind thinking blocks, send
  thinking.block_binding.prefix_mismatch_behavior="drop_block" with the
  thinking-binding-controls-2026-08-01 beta, so blocks invalidated by
  history scrubbing are dropped instead of failing the request
- default with_structured_output to method="json_schema"

Older Claude ids (claude-3-7-sonnet, Sonnet/Opus 4.6 and earlier) keep the
existing request shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@google-cla

google-cla Bot commented Oct 31, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@vvinogra
vvinogra marked this pull request as draft November 1, 2026 10:16
With claude-sonnet-5-5, a multi-turn Flash task looped from turn 3 on:
every request returned HTTP 200 with stop_reason "refusal" and no
content, and the runner nudged and re-sent the same prompt each turn.

Sonnet 5.5 returns its between-tool narration as thinking blocks, so the
ledger sees every turn as silent and appends REASONING_REMINDER. Its
wording ("internal thinking is not shown ... write ...") trips the
reasoning_extraction safety classifier (reproduced 1/1 live; reworded
prompt: 4/4 normal tool calls, with visible text).

- Reword REASONING_REMINDER as a request for a short progress note.
- Raise LLMPermanentError with a new REFUSAL failure category when a
  response has stop_reason "refusal", so the fallback endpoint is tried
  and a repeated refusal fails the task instead of looping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vvinogra vvinogra changed the title fix: make Anthropic requests valid for Claude Sonnet 5.5 and other adaptive-only models fix: make Artemis work with Claude Sonnet 5.5 (request shape, preserved thinking, refusal loop) Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant