Skip to content

fix(openai): move to the next account when the stripped retry fails for the account's own reason - #249

Merged
HTExplicit merged 2 commits into
mainfrom
fix/recovery-retry-account-failure
Oct 5, 2026
Merged

HTExplicit merged 2 commits into
mainfrom
fix/recovery-retry-account-failure

Conversation

@HTExplicit

Copy link
Copy Markdown
Owner

What

Reasoning recovery strips unverifiable ciphertext and re-sends once on the same account. When that re-send failed for a reason that belongs to the account (credit exhausted, rate limit, auth, 5xx, an upstream failure inside the stream), the attempt ended as a recovery terminal and the client received the failure, although another account could have answered the unchanged request.

Observed in production after v0.2.13-codexrip.9: the stripped retry on one account ended twice with credit_balance_exhausted and the request stopped there.

Change

Such a failure is now returned as an ordinary account failover carrying a mark (UpstreamFailoverError.RecoveryRetrySpent).

  • Scope: POST /v1/responses only (native, compact, passthrough). Chat Completions → Responses is unchanged.
  • Condition: the stripped retry was dispatched, nothing reached the client, and the failure is one a first send would move to another account. Validation rejections, refusal outcomes, transport errors, header timeouts and unclassified errors still end on the account.
  • Handler: no same-account retry (the one stripped retry is spent), the retry-exhausted cooldown applies, and the recovery attempt is reset so the next account receives the original request and has its own retry.
  • Classification of HTTP answers is that of a first send: a request-budget rejection stays request-scoped, model_not_supported stays scoped to the account/model pair, a budget_exceeded 429 stays the account's, other pool-retryable statuses stay outside the API-key health breaker. This exit writes no forward-layer account state.
  • Ops continuation diagnostics mark the attempt with recovery.account_failure.

The recovery policy (codexruntime/recovery), nativeapi and the decision order are untouched.

Tests

  • Service: state-level hand-off rows (including redaction of the body), native and passthrough HTTP exits (hand-off, non-failover answer stays terminal, pool exemption, first-send classification for request-budget rejection / model_not_supported / budget_exceeded).
  • Handler end to end with a fake upstream: in-stream credit exhaustion, HTTP 402, OAuth 5xx all move to the next account, which gets the original request; an account failure is not treated as a mismatch; the failed account is cooled.
  • Each new branch was checked with an overlay mutant that the tests kill.
  • No real model calls. Locally TestOllamaProbeCallback_StaleLongDoesNotOverrideNewShort fails on this Windows machine on main as well; it is unrelated to this change.

🤖 Generated with Claude Code

Codex and others added 2 commits October 5, 2026 19:20
…or the account's own reason

Reasoning recovery strips unverifiable ciphertext and re-sends once on the
same account. When that re-send failed for a reason that belongs to the
account (credit exhausted, rate limit, auth, 5xx, an upstream failure inside
the stream), the attempt ended as a recovery terminal and the client got the
failure, although any other account could have answered the unchanged
request. Seen in production on 2026-10-05: the stripped retry on one account
ended twice with credit_balance_exhausted and the request stopped there.

Such a failure is now returned as an ordinary account failover carrying a
mark (RecoveryRetrySpent):

- Only on POST /v1/responses (native, compact, passthrough), only when the
  retry was dispatched, nothing reached the client, and the failure is one a
  first send would move to another account. Validation rejections, refusal
  outcomes, transport errors, header timeouts and unclassified errors still
  end on the account.
- The handler does not retry the same account (its one stripped retry is
  spent), applies the retry-exhausted cooldown, and resets the recovery
  attempt so the next account gets the original request and its own retry.
- HTTP answers keep the classification of a first send: a request-budget
  rejection stays request-scoped, model_not_supported stays scoped to the
  account/model pair, a budget_exceeded 429 stays the account's, and other
  pool-retryable statuses stay outside the API-key health breaker. No
  forward-layer account state is written on this exit.
- Ops continuation diagnostics mark the attempt with recovery.account_failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments and handbook wording only; no behaviour change.

- The handler's retry-exhausted cooldown does nothing for an OpenAI API-key
  account, so the comment, the test comment and the handbook no longer say
  such an account is cooled after an HTTP answer to its stripped retry.
- A non-failover answer to the stripped retry ends the request unless it is a
  ciphertext rejection, which StopError still hands on as a mismatch.
- A recovery terminal after committed output needs a ciphertext rejection.
- The handbook names per path which answers move to the next account: the
  passthrough rule is narrower than the native one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HTExplicit
HTExplicit merged commit 11ea5ab into main Oct 5, 2026
18 checks passed
@HTExplicit
HTExplicit deleted the fix/recovery-retry-account-failure branch October 5, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant