Skip to content

A request a safety classifier declined goes to the bot's fallback model, that request alone - #349

Merged
lydakis merged 4 commits into
mainfrom
claude/project-thread-filai4
Oct 29, 2026
Merged

lydakis merged 4 commits into
mainfrom
claude/project-thread-filai4

Conversation

@lydakis

@lydakis lydakis commented Oct 29, 2026

Copy link
Copy Markdown
Owner

What

A bot created with --fallback PROVIDER/MODEL (a known provider's model in its family, inherited by forks) resends a request that ended with stop_reason: "refusal" to that model, on the same body. Only that request. The next request goes straight back to the bot's own model, with the fallback's answer in its history. A refusal the fallback repeats, or a refusal on a bot with no fallback, ends the turn with provider_refusal as before.

Triggered only by the typed stop_reason; nothing matches message text.

Details

  • Parser: the refusal's stop_details.category and fallback_credit_token are kept as the error's facts. The credit is taken out before the error is recorded.
  • Credit: on the Claude API, a retry on the bot's own provider redeems the credit (fallback-credit-2026-07-01), so the fallback's cache writes are priced as if it had run there all along. A 400 to the credit sends the same retry without it. Bedrock requests send no credit: Mantle answers betas it does not take with a 400, and this one has not been tried there.
  • Events: the retry is a retry event with fallback: "provider/model"; the answer fires the existing model_fallback event, and its usage names the fallback's provider and model in models, so it is priced at that model.
  • No server-side fallback: Anthropic's fallbacks: "default" is no longer asked for, since its routing sticks to the fallback model for about an hour. --fallbacks is replaced by --fallback, and the store's fallbacks column by fallback TEXT (schema 60). The server-side beta stays on the Claude API so stored answers it served still replay.
  • Thinking drops: only prefix_binding_mismatch entries in input_transformations count as binding drops. A fallback's thinking blocks that the API drops on the bot's model (model_binding_mismatch) are expected.
  • Config: [fallbacks] in ~/.agent/config.toml names a fallback per model. The CLI, Sonny's leads and its tasks give it to a new bot that names none:
    [fallbacks]
    "anthropic/claude-opus-5-5" = "anthropic/claude-opus-4-8"
  • Harbor: the adapter takes --ak fallback=PROVIDER/MODEL instead of always asking for server-side fallbacks, and bot_settings records each bot's fallback model.

Tests

  • Rust: parser facts (category, credit, null category), credit in the request body and never on Bedrock, store and fork round trip, config parsing, project creation picking the fallback for the kept model.
  • Runtime (synthetic Anthropic endpoint): a refused request is answered by the fallback with the credit, then the next request goes back to the bot's model; a refusal the fallback repeats fails the turn; a bot with no fallback fails at once; the CLI applies [fallbacks] and --fallback wins.
  • Not tested live: a real classifier refusal was not reproduced, on either the Claude API or Bedrock.

A follow-up will send a refusal that stands to later requests as a short note naming the category, in place of the refused turn.


Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T05:58:40.831609Z b997b99 New commits
🔒 Security Review ✅ Completed 2026-10-29T05:59:48.743590Z b997b99 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: daf9ed4b93

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/server/turn.rs Resolved
Comment thread src/server/turn.rs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93b3d2c370

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/server/turn.rs Resolved
@lydakis
lydakis force-pushed the claude/project-thread-filai4 branch from a46dc16 to 7de1ff4 Compare October 29, 2026 05:23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7de1ff466b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/server/turn.rs Resolved
…el, that request alone

A bot created with `--fallback PROVIDER/MODEL` (a known provider's model in
its family, inherited by forks) resends a request that ended with
`stop_reason: "refusal"` to that model, on the same body. The next request
goes back to the bot's own model, with the fallback's answer in its history.
A refusal the fallback repeats, or one on a bot with no fallback, ends the
turn with `provider_refusal` as before.

- The refusal's `stop_details` category and fallback credit are kept as the
  error's facts. On the Claude API, a retry on the bot's own provider
  redeems the credit (`fallback-credit-2026-07-01`); a 400 to it sends the
  same retry without it. Bedrock requests send no credit.
- The retry is a `retry` event naming the fallback; the answer fires
  `model_fallback` and its usage is priced at the fallback's model.
- Anthropic's server-side `fallbacks` is no longer asked for: its routing
  sticks to the fallback model for about an hour. `--fallbacks` is replaced
  by `--fallback`; the store's `fallbacks` column by `fallback` (schema 60).
  Its beta stays on the Claude API so stored answers it served replay.
- Only `prefix_binding_mismatch` thinking drops count as binding drops; a
  fallback's blocks dropped on the bot's model (`model_binding_mismatch`)
  are expected.
- `[fallbacks]` in ~/.agent/config.toml names a fallback per model; the CLI,
  Sonny's leads and its tasks give it to a new bot that names none.
- The Harbor adapter takes `fallback` instead of always asking for
  server-side fallbacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGCoLcHQMmGw188SP8o
…point that refused

A call parked on the fallback model's pool kept its attempts but not its
fallback, so it resumed on the bot's model and was refused again. The park
now keeps the fallback and its credit.

The credit went with the retry only when the fallback named the bot's own
provider; a turn run on another binding's model compared the wrong one. It
now goes when the fallback is on the endpoint that refused, and the
`model_fallback` event names the model the call asked for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGCoLcHQMmGw188SP8o
…sent to

A provider name rebound to another family across a restart would have taken
the bot's history in the wrong encoding; the refusal stands instead, for a
new fallback and one a parked call resumes alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGCoLcHQMmGw188SP8o
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jkdGCoLcHQMmGw188SP8o
@lydakis
lydakis force-pushed the claude/project-thread-filai4 branch from 7de1ff4 to b997b99 Compare October 29, 2026 05:56
@lydakis
lydakis merged commit 925cedc into main Oct 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants