From 298e4572efd28d7e02cbc70483d2847ae3f3038d Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 5 Oct 2026 19:20:48 +0800 Subject: [PATCH 1/2] fix(openai): move to the next account when the stripped retry fails for the account's own reason Reasoning recovery strips unverifiable ciphertext and re-sends once on the same account. When that re-send failed for a reason that belongs to the account (credit exhausted, rate limit, auth, 5xx, an upstream failure inside the stream), the attempt ended as a recovery terminal and the client got the failure, although any other account could have answered the unchanged request. Seen in production on 2026-10-05: the stripped retry on one account ended twice with credit_balance_exhausted and the request stopped there. Such a failure is now returned as an ordinary account failover carrying a mark (RecoveryRetrySpent): - Only on POST /v1/responses (native, compact, passthrough), only when the retry was dispatched, nothing reached the client, and the failure is one a first send would move to another account. Validation rejections, refusal outcomes, transport errors, header timeouts and unclassified errors still end on the account. - The handler does not retry the same account (its one stripped retry is spent), applies the retry-exhausted cooldown, and resets the recovery attempt so the next account gets the original request and its own retry. - HTTP answers keep the classification of a first send: a request-budget rejection stays request-scoped, model_not_supported stays scoped to the account/model pair, a budget_exceeded 429 stays the account's, and other pool-retryable statuses stay outside the API-key health breaker. No forward-layer account state is written on this exit. - Ops continuation diagnostics mark the attempt with recovery.account_failure. Co-Authored-By: Claude Opus 5.5 --- .downstream/gpt6-models-and-prompts.md | 2 +- ...enai_continuation_account_mismatch_test.go | 117 +++++++++- .../handler/openai_gateway_handler.go | 25 ++- ...easoning_recovery_terminal_handler_test.go | 18 +- backend/internal/service/gateway_service.go | 1 + .../service/openai_continuation_diagnostic.go | 4 + .../service/openai_gateway_forward.go | 12 +- .../service/openai_gateway_passthrough.go | 12 +- .../service/openai_gateway_upstream_errors.go | 50 +++++ .../service/openai_reasoning_recovery.go | 37 ++- ...openai_reasoning_recovery_terminal_test.go | 2 +- .../service/openai_reasoning_recovery_test.go | 210 +++++++++++++++++- 12 files changed, 460 insertions(+), 30 deletions(-) diff --git a/.downstream/gpt6-models-and-prompts.md b/.downstream/gpt6-models-and-prompts.md index 7c9366103fbf..504a23ba6ccb 100644 --- a/.downstream/gpt6-models-and-prompts.md +++ b/.downstream/gpt6-models-and-prompts.md @@ -66,7 +66,7 @@ Claude OAuth 伪装系统块只由上游设置决定:设置 → 网关的 `ena 侧栏“扩展功能 → 推理恢复”(`/admin/reasoning-recovery`):一个全局开关和保存按钮,默认开启,没有账号级选项;页面打开时读取一次,读取失败时显示错误、不显示开关。存为 `settings.reasoning_recovery_config` 一行 JSON(`{"enabled": }`),没有该行即开启;`GET/PUT /api/v1/admin/reasoning-recovery` 以标准响应封装读写 `{"enabled": }`,管理员认证、无二次验证,保存记入操作审计,请求体不是严格的 `{"enabled": }` 时返回 400 且不保存。请求路径只读内存中的布尔值(保存即替换,60 秒后台刷新以同步多实例),不查询数据库,每次转发开始时读取一次。启动时读取失败或存储值不是严格的 `{"enabled": }` 则停止启动;运行中刷新失败保留当前值。 -开启时,OpenAI 平台的 API Key、OAuth 和 Setup Token 账号(含自动透传)在原生 Responses(含 compact)与 Chat Completions 转 Responses 的 HTTP 请求上生效:上游以结构化错误码 `invalid_encrypted_content` 或 `thinking_signature_invalid` 拒绝、且尚无语义输出时,去掉推理项的 `encrypted_content`,在同一账号重发一次:上游用 `input[N]` 指明条目时只去掉该项,否则去掉请求里全部推理项的密文。上游未用 `input[N]` 指明条目、而请求里还有其他密文载体(压缩项、多智能体 `agent_message` 的正文段)时,只有上游错误消息按条目 id 点名的是带密文的推理项才恢复,否则不恢复;这条点名规则只在推理项 id 随请求发出的路径上成立(API Key 账号与自动透传),OAuth / Setup Token 的非透传请求发出前会去掉推理项 id,遇到其他密文载体仍不恢复。其他载体的密文任何情况下都不改动。被去掉的密文按来源在 Redis 记忆 24 小时,期间同源请求发送前即去掉;记忆按每批 64 条读写,读与写共用每个请求 100 毫秒的 I/O 预算。本账号修不好时换号:在 `POST /v1/responses`(含 compact 与自动透传)上,上游报告请求里的密文无法解密或校验(结构化错误码、同类错误消息,或 `could not be decrypted` 的措辞),而本账号无法修复(不能恢复,或去掉推理密文后的重发、发送前已去掉密文的请求仍得到这类拒绝),且客户端尚未收到本次尝试的任何输出时,请求原样交给分组里的下一个账号——密文只能由签发它的上游组织解开,别的账号可能直接接受。每个请求最多因此换号 3 次(同时计入正常换号预算),下一个账号各自按上述规则恢复一次;被换下的账号不计健康惩罚,也不做同账号重试。换号上限或可选账号用尽时,客户端收到第一个报告密文问题的账号的那个错误;途中某个账号给出自己的终态(例如读得懂密文但请求本身无效、恢复重发遇到 5xx)或网关侧容量限制时,返回该结果。没有账号给出回答时,被这次换号挪到后续账号上的粘性绑定回到第一个报告密文问题的账号,调度器保留在别处的绑定不动;有账号接受或已开始输出时,会话留在该账号。普通的请求校验错误、带 `previous_response_id` 或 `conversation` 的请求、上游状态为 401/402/403/407/429 时不换号;先试过透传账号再换到非透传账号时,沿用既有规则整块去掉加密推理项。经原生 Responses 路径转发的其他平台账号同样适用;Chat Completions 转 Responses 不换号。Ops 续接诊断用 `recovery.account_mismatch` 标出这样结束的尝试。Messages 桥接与 WebSocket 不恢复。关闭时不重发也不预先去除,签名拒绝仍作为请求级终态返回,不切换账号。 +开启时,OpenAI 平台的 API Key、OAuth 和 Setup Token 账号(含自动透传)在原生 Responses(含 compact)与 Chat Completions 转 Responses 的 HTTP 请求上生效:上游以结构化错误码 `invalid_encrypted_content` 或 `thinking_signature_invalid` 拒绝、且尚无语义输出时,去掉推理项的 `encrypted_content`,在同一账号重发一次:上游用 `input[N]` 指明条目时只去掉该项,否则去掉请求里全部推理项的密文。上游未用 `input[N]` 指明条目、而请求里还有其他密文载体(压缩项、多智能体 `agent_message` 的正文段)时,只有上游错误消息按条目 id 点名的是带密文的推理项才恢复,否则不恢复;这条点名规则只在推理项 id 随请求发出的路径上成立(API Key 账号与自动透传),OAuth / Setup Token 的非透传请求发出前会去掉推理项 id,遇到其他密文载体仍不恢复。其他载体的密文任何情况下都不改动。被去掉的密文按来源在 Redis 记忆 24 小时,期间同源请求发送前即去掉;记忆按每批 64 条读写,读与写共用每个请求 100 毫秒的 I/O 预算。本账号修不好时换号:在 `POST /v1/responses`(含 compact 与自动透传)上,上游报告请求里的密文无法解密或校验(结构化错误码、同类错误消息,或 `could not be decrypted` 的措辞),而本账号无法修复(不能恢复,或去掉推理密文后的重发、发送前已去掉密文的请求仍得到这类拒绝),且客户端尚未收到本次尝试的任何输出时,请求原样交给分组里的下一个账号——密文只能由签发它的上游组织解开,别的账号可能直接接受。每个请求最多因此换号 3 次(同时计入正常换号预算),下一个账号各自按上述规则恢复一次;被换下的账号不计健康惩罚,也不做同账号重试。换号上限或可选账号用尽时,客户端收到第一个报告密文问题的账号的那个错误;途中某个账号给出自己的终态(例如读得懂密文但请求本身无效)或网关侧容量限制时,返回该结果。没有账号给出回答时,被这次换号挪到后续账号上的粘性绑定回到第一个报告密文问题的账号,调度器保留在别处的绑定不动;有账号接受或已开始输出时,会话留在该账号。普通的请求校验错误、带 `previous_response_id` 或 `conversation` 的请求、上游状态为 401/402/403/407/429 时不换号;先试过透传账号再换到非透传账号时,沿用既有规则整块去掉加密推理项。经原生 Responses 路径转发的其他平台账号同样适用;Chat Completions 转 Responses 不换号。Ops 续接诊断用 `recovery.account_mismatch` 标出因密文读不了而交给下一个账号的尝试。恢复重发因账号自身的原因失败时也换号(同样只在 `POST /v1/responses`,含 compact 与自动透传;Chat Completions 转 Responses 不换号):重发已发出、失败属于首次发送时就会换号的那一类(额度用尽、限流、鉴权、5xx、流内的上游失败),且客户端尚未收到本次尝试的任何输出时,请求按普通账号失败交给下一个账号——下一个账号收到原样请求,本账号不再重试,换号预算同普通换号,账号健康按首次发送时的分类记录(调度失败上报、同账号重试用尽后的冷却):请求预算被拒(402/403 `balance_insufficient`)仍只算本次请求,结构化的 `model_not_supported` 仍只算该账号与模型,`budget_exceeded` 429 仍算账号,其余池模式可重试的状态仍不计入 API Key 健康熔断。Ops 续接诊断用 `recovery.account_failure` 标出这样交给下一个账号的尝试。重发的 HTTP 错误应答不执行转发层在首次发送时的账号状态写入(标记限流、置为错误、停用、预算用尽标记),也不开启 OAuth 429 的同账号重试窗口,这些由之后的首次发送处理;首次发送因同时停用账号而把池模式可重试状态计入账号的情形,这里无从得知,按不计入处理;处理器侧的冷却照常执行,其中 OAuth 账号遇到硬额度 429 时会写入额度状态。重发遇到请求校验错误、拒绝类结果、传输错误、响应头超时或未归类为可换号的错误时,仍在本账号结束。Messages 桥接与 WebSocket 不恢复。关闭时不重发也不预先去除,签名拒绝仍作为请求级终态返回,不切换账号。 迁移 `267_purge_account_reasoning_options.sql` 一次性删除各账号 `extra` 中原有的两个账号级推理选项键(键名见该文件);此后没有代码读写它们,已存的账号级取值不带入全局开关。Chat Completions 转 Responses 的请求只由客户端消息转换而来,网关不保存响应中的推理项。 diff --git a/backend/internal/handler/openai_continuation_account_mismatch_test.go b/backend/internal/handler/openai_continuation_account_mismatch_test.go index 5f751a42c463..09fdaf301c7a 100644 --- a/backend/internal/handler/openai_continuation_account_mismatch_test.go +++ b/backend/internal/handler/openai_continuation_account_mismatch_test.go @@ -26,7 +26,10 @@ type continuationMismatchUpstream struct { bodies [][]byte accepting map[int64]bool replies map[int64]continuationMismatchReply - rejection continuationMismatchReply + // sequence scripts an account's answers call by call; when it runs out the + // account falls back to replies and rejection. + sequence map[int64][]continuationMismatchReply + rejection continuationMismatchReply } type continuationMismatchReply struct { @@ -44,12 +47,18 @@ func (u *continuationMismatchUpstream) Do(req *http.Request, _ string, accountID u.mu.Lock() u.accountIDs = append(u.accountIDs, accountID) u.bodies = append(u.bodies, body) + var next *continuationMismatchReply + if pending := u.sequence[accountID]; len(pending) > 0 { + next, u.sequence[accountID] = &pending[0], pending[1:] + } u.mu.Unlock() - if u.accepting[accountID] { + if next == nil && u.accepting[accountID] { return openAIOfficialHTTPSuccess(openAIOfficialHTTPAttempt{accountID: accountID, path: req.URL.Path, body: body}), nil } reply, scripted := u.replies[accountID] - if !scripted { + if next != nil { + reply = *next + } else if !scripted { reply = u.rejection } if reply.stream != "" { @@ -125,9 +134,14 @@ func newContinuationMismatchHandler(t *testing.T, upstream service.HTTPUpstream, } func serveContinuationMismatch(t *testing.T, h *OpenAIGatewayHandler, body string) *responseRecorderResult { + t.Helper() + return serveContinuationMismatchSession(t, h, body, "one-conversation") +} + +func serveContinuationMismatchSession(t *testing.T, h *OpenAIGatewayHandler, body, session string) *responseRecorderResult { t.Helper() c, recorder := newReplayableEndpointContext(t, openAIOfficialHTTPGroupID, http.MethodPost, "/v1/responses", []byte(body)) - c.Request.Header.Set("session_id", "one-conversation") + c.Request.Header.Set("session_id", session) h.Responses(c) return &responseRecorderResult{code: recorder.Code, body: recorder.Body.String()} } @@ -155,6 +169,10 @@ func continuationMismatchBoundAccount(t *testing.T, cache *openAIOfficialHTTPSti // whose body is ciphertext the gateway cannot remove. const continuationMismatchBody = `{"model":"gpt-5.1","stream":false,"input":[{"type":"reasoning","id":"rs_a","encrypted_content":"cipher-reasoning","summary":[]},{"type":"agent_message","id":"amsg_b","author":"/root","recipient":"/root/worker","content":[{"type":"input_text","text":"Payload:"},{"type":"encrypted_content","encrypted_content":"cipher-agent"}]}]}` +// A history whose only ciphertext is reasoning: any rejection of it can be +// repaired by the stripped retry, also where reasoning ids are not sent. +const continuationMismatchReasoningOnlyBody = `{"model":"gpt-5.1","stream":false,"input":[{"type":"reasoning","id":"rs_a","encrypted_content":"cipher-reasoning","summary":[]},{"type":"message","role":"user","content":[{"type":"input_text","text":"continue"}]}]}` + func TestOpenAIGatewayHandler_ContinuationCiphertextMismatchMovesToAnAccountThatAccepts(t *testing.T) { gin.SetMode(gin.TestMode) for _, test := range []struct { @@ -277,3 +295,94 @@ func TestOpenAIGatewayHandler_ContinuationCiphertextMismatchKeepsTheSessionOnAnA require.Contains(t, response.body, "partial-answer") require.Equal(t, int64(2), continuationMismatchBoundAccount(t, cache)) } + +// The stripped retry can fail for a reason of the account's own. The answer is +// then the same as on a first send: the request goes to the next account, which +// receives it unchanged, and the first account is not asked a third time. +func TestOpenAIGatewayHandler_ContinuationStrippedRetryAccountFailureMovesToTheNextAccount(t *testing.T) { + gin.SetMode(gin.TestMode) + for _, test := range []struct { + name string + stream bool + accountType string + body string + first continuationMismatchReply + retry continuationMismatchReply + }{ + // Production, 2026-10-05: the retry is accepted at HTTP level and fails in the stream. + {"credit_exhausted_in_stream", true, service.AccountTypeAPIKey, continuationMismatchBody, continuationMismatchNamedReasoning, + continuationMismatchReply{status: http.StatusOK, stream: "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"id\":\"resp_failed\",\"status\":\"failed\",\"error\":{\"code\":\"credit_balance_exhausted\",\"message\":\"You have no credits remaining. do-not-leak-billing\"}}}\n\n"}}, + {"payment_required", false, service.AccountTypeAPIKey, continuationMismatchBody, continuationMismatchNamedReasoning, + continuationMismatchReply{status: http.StatusPaymentRequired, payload: `{"error":{"type":"insufficient_quota","code":"insufficient_quota","message":"do-not-leak-billing"}}`}}, + // An OAuth account would otherwise get a same-account retry for a 5xx. + {"server_error_on_oauth", true, service.AccountTypeOAuth, continuationMismatchReasoningOnlyBody, continuationMismatchUnrecoverable, + continuationMismatchReply{status: http.StatusServiceUnavailable, payload: `{"error":{"type":"server_error","message":"do-not-leak-provider"}}`}}, + } { + t.Run(test.name, func(t *testing.T) { + upstream := &continuationMismatchUpstream{ + accepting: map[int64]bool{2: true}, + sequence: map[int64][]continuationMismatchReply{1: {test.first, test.retry}}, + rejection: continuationMismatchUnrecoverable, + } + handler, cache := newContinuationMismatchHandler(t, upstream, continuationMismatchAccounts(2, test.accountType)) + body := test.body + if test.stream { + body = strings.Replace(body, `"stream":false`, `"stream":true`, 1) + } + response := serveContinuationMismatch(t, handler, body) + + require.Equal(t, []int64{1, 1, 2}, upstream.accountIDs, "first send, stripped retry, then the next account") + require.False(t, gjson.GetBytes(upstream.bodies[1], "input.0.encrypted_content").Exists(), "the retry on the first account is the stripped one") + require.Equal(t, gjson.GetBytes(upstream.bodies[0], "input").Raw, gjson.GetBytes(upstream.bodies[2], "input").Raw, "the next account receives the request unchanged") + require.Equal(t, http.StatusOK, response.code, response.body) + require.NotContains(t, response.body, "do-not-leak") + require.NotContains(t, response.body, "response.failed") + if test.stream { + require.Contains(t, response.body, "response.completed") + } else { + require.Contains(t, response.body, "healthy answer") + } + require.Equal(t, int64(2), continuationMismatchBoundAccount(t, cache)) + }) + } +} + +// An account failure of the stripped retry is an ordinary account failure, not +// a mismatch: when nobody answers, the last account's failure is the answer and +// the session stays where selection put it. +func TestOpenAIGatewayHandler_ContinuationStrippedRetryAccountFailureIsNotAMismatch(t *testing.T) { + gin.SetMode(gin.TestMode) + upstream := &continuationMismatchUpstream{ + sequence: map[int64][]continuationMismatchReply{1: {continuationMismatchNamedReasoning, + {status: http.StatusPaymentRequired, payload: `{"error":{"type":"insufficient_quota","code":"insufficient_quota","message":"do-not-leak-billing"}}`}}}, + replies: map[int64]continuationMismatchReply{2: {status: http.StatusServiceUnavailable, payload: `{"error":{"type":"server_error","message":"do-not-leak-provider"}}`}}, + } + handler, cache := newContinuationMismatchHandler(t, upstream, continuationMismatchAccounts(2, service.AccountTypeAPIKey)) + response := serveContinuationMismatch(t, handler, continuationMismatchBody) + + require.Equal(t, []int64{1, 1, 2}, upstream.accountIDs) + require.GreaterOrEqual(t, response.code, http.StatusInternalServerError, response.body) + require.NotEqual(t, service.OpenAIContinuationStateUnavailableCode, gjson.Get(response.body, "error.code").String()) + require.NotContains(t, response.body, "do-not-leak") + require.Equal(t, int64(2), continuationMismatchBoundAccount(t, cache), "no mismatch, so no binding is moved back") +} + +// The account that failed on its stripped retry gets the cooldown of an account +// whose same-account attempts are used up: the next request is not sent to it. +func TestOpenAIGatewayHandler_ContinuationStrippedRetryAccountFailureCoolsTheAccount(t *testing.T) { + gin.SetMode(gin.TestMode) + upstream := &continuationMismatchUpstream{ + accepting: map[int64]bool{1: true, 2: true}, + // An authentication failure blocks the whole account for a while, which + // the next selection can be seen to respect. + sequence: map[int64][]continuationMismatchReply{1: {continuationMismatchUnrecoverable, + {status: http.StatusUnauthorized, payload: `{"error":{"type":"authentication_error","message":"credentials rejected"}}`}}}, + } + handler, _ := newContinuationMismatchHandler(t, upstream, continuationMismatchAccounts(2, service.AccountTypeOAuth)) + body := strings.Replace(continuationMismatchReasoningOnlyBody, `"stream":false`, `"stream":true`, 1) + serveContinuationMismatch(t, handler, body) + require.Equal(t, []int64{1, 1, 2}, upstream.accountIDs) + + serveContinuationMismatchSession(t, handler, body, "another-conversation") + require.Equal(t, []int64{1, 1, 2, 2}, upstream.accountIDs, "account 1 would accept again, but is cooling") +} diff --git a/backend/internal/handler/openai_gateway_handler.go b/backend/internal/handler/openai_gateway_handler.go index 91faa00b4280..63123da85669 100644 --- a/backend/internal/handler/openai_gateway_handler.go +++ b/backend/internal/handler/openai_gateway_handler.go @@ -1046,10 +1046,17 @@ func (h *OpenAIGatewayHandler) Responses(c *gin.Context) { h.handleFailoverExhausted(c, failoverErr, streamStarted) return } - // An account mismatch says nothing against the account: no - // same-account retry and no cooldown, the next account is tried. retryAction := openAIFailoverRetrySwitchAccount - if !failoverErr.IsOpenAICiphertextAccountMismatch() { + switch { + case failoverErr.IsOpenAICiphertextAccountMismatch(): + // A mismatch says nothing against the account: no + // same-account retry and no cooldown, the next account is tried. + case failoverErr.IsOpenAIRecoveryRetrySpent(): + // The account failed on its stripped retry. It is not asked + // again, and gets the cooldown of an account whose + // same-account attempts are used up. + h.gatewayService.CooldownOpenAIRetryExhausted(c.Request.Context(), account, account.GetMappedModel(routingModel), failoverErr) + default: retryAction = retryState.HandleHTTP( c.Request.Context(), h.gatewayService, @@ -1099,6 +1106,9 @@ func (h *OpenAIGatewayHandler) Responses(c *gin.Context) { ciphertextSwitchCount++ service.ResetOpenAIReasoningRecoveryAttempt(c) } + if failoverErr.IsOpenAIRecoveryRetrySpent() { + service.ResetOpenAIReasoningRecoveryAttempt(c) + } if switchCount >= maxAccountSwitches { h.handleFailoverExhausted(c, exhaustedFailoverErr(failoverErr), streamStarted) return @@ -4071,9 +4081,12 @@ func (h *OpenAIGatewayHandler) handleOpenAINoAccountError(c *gin.Context, classi h.handleStreamingAwareErrorWithCode(c, classification.Status, classification.ErrType, code, classification.Message, streamStarted, false) } -// A completed reasoning-recovery attempt is a terminal, not another scheduler -// candidate. Preserve its classification without allowing refusal, account, or -// transport retries to re-enter the loop. +// A recovery terminal ends the request: it is any attempt that spent its +// stripped retry, or had output committed, and was not handed on. Preserve its +// classification without allowing refusal, account, or transport retries to +// re-enter the loop. An account mismatch and an account failure of the +// stripped retry, both possible only before anything reached the client, are +// returned as failovers instead and never reach this function. func (h *OpenAIGatewayHandler) handleReasoningRecoveryTerminal(c *gin.Context, err error, selection *service.AccountSelectionResult, account *service.Account, model string, streamStarted bool, reporter openAIFailoverSelectionReporter) bool { var terminal *service.OpenAIReasoningRecoveryTerminalError if !errors.As(err, &terminal) { diff --git a/backend/internal/handler/openai_reasoning_recovery_terminal_handler_test.go b/backend/internal/handler/openai_reasoning_recovery_terminal_handler_test.go index 281572a4420b..cddf1dbdc7db 100644 --- a/backend/internal/handler/openai_reasoning_recovery_terminal_handler_test.go +++ b/backend/internal/handler/openai_reasoning_recovery_terminal_handler_test.go @@ -80,7 +80,10 @@ func TestOpenAIGatewayHandler_ReasoningFailureAlreadyForwardedIsNotDuplicated(t } } -func TestOpenAIGatewayHandler_ReasoningRecoveryTerminalNeverReentersScheduler(t *testing.T) { +// After the stripped retry, a request rejection ends the request on that +// account; a provider failure is the account's own and the request goes to the +// next account, never back to the first. +func TestOpenAIGatewayHandler_ReasoningRecoveryOutcomeAfterTheStrippedRetry(t *testing.T) { gin.SetMode(gin.TestMode) for _, passthrough := range []bool{false, true} { mode := "native" @@ -92,9 +95,10 @@ func TestOpenAIGatewayHandler_ReasoningRecoveryTerminalNeverReentersScheduler(t status int clientStatus int healthEvents int + accountIDs []int64 }{ - {"request_rejected", http.StatusBadRequest, http.StatusBadRequest, 0}, - {"provider_error", http.StatusInternalServerError, http.StatusBadGateway, 1}, + {"request_rejected", http.StatusBadRequest, http.StatusBadRequest, 0, []int64{1, 1}}, + {"provider_error", http.StatusInternalServerError, http.StatusBadGateway, 1, []int64{1, 1, 2}}, } { t.Run(mode+"/"+failure.name, func(t *testing.T) { setupOpsErrorLogTestQueue(t, 4) @@ -104,7 +108,7 @@ func TestOpenAIGatewayHandler_ReasoningRecoveryTerminalNeverReentersScheduler(t Status: service.StatusActive, Schedulable: true, Credentials: map[string]any{"api_key": "fixture", "base_url": "https://api.example.test", "pool_mode": true, "pool_mode_retry_count": 5}, Extra: map[string]any{"use_responses_api": true, "openai_passthrough": passthrough}}, - {ID: 2, Name: "must-not-replay", Platform: service.PlatformOpenAI, Type: service.AccountTypeAPIKey, + {ID: 2, Name: "next-account", Platform: service.PlatformOpenAI, Type: service.AccountTypeAPIKey, Status: service.StatusActive, Schedulable: true, Priority: 1, Credentials: map[string]any{"api_key": "other-fixture", "base_url": "https://other.example.test"}, Extra: map[string]any{"use_responses_api": true}}, @@ -126,7 +130,7 @@ func TestOpenAIGatewayHandler_ReasoningRecoveryTerminalNeverReentersScheduler(t request.Header.Set("Content-Type", "application/json") router.ServeHTTP(recorder, request) - require.Equal(t, []int64{1, 1}, upstream.accountIDs, "no third POST, same-account scheduler retry, or second account") + require.Equal(t, failure.accountIDs, upstream.accountIDs, "no third POST or same-account scheduler retry on the first account") require.True(t, gjson.GetBytes(upstream.bodies[0], "input.0.encrypted_content").Exists()) require.False(t, gjson.GetBytes(upstream.bodies[1], "input.0.encrypted_content").Exists()) require.Equal(t, gjson.GetBytes(upstream.bodies[0], "input.1").Raw, gjson.GetBytes(upstream.bodies[1], "input.1").Raw) @@ -143,8 +147,10 @@ func TestOpenAIGatewayHandler_ReasoningRecoveryTerminalNeverReentersScheduler(t require.NotNil(t, job.entry.UpstreamStatusCode) require.Equal(t, failure.status, *job.entry.UpstreamStatusCode) // The minimal router fixture intentionally disables the optional - // advanced scheduler. Exercise the same terminal finalizer with an + // advanced scheduler. Exercise the terminal finalizer with an // observable reporter instead of treating inert metrics as evidence. + // A provider failure still ends as a terminal when the attempt did + // not classify it, so both classifications are probed directly. reporter := &recordingFailoverSelectionReporter{} classified := &service.UpstreamFailoverError{StatusCode: failure.status} if failure.clientStatus == http.StatusBadRequest { diff --git a/backend/internal/service/gateway_service.go b/backend/internal/service/gateway_service.go index 155f68bb743f..70132eafad00 100644 --- a/backend/internal/service/gateway_service.go +++ b/backend/internal/service/gateway_service.go @@ -746,6 +746,7 @@ type UpstreamFailoverError struct { ClientMessage string SuppressAccountHealthPenalty bool CiphertextAccountMismatch bool // 本账号无法校验请求里的密文且无法就地修复;原样请求可能被签发该密文的组织下的其他账号接受 + RecoveryRetrySpent bool // 本账号的剥离重发已发出并因账号自身的原因失败;换下一个账号,不得在本账号再试 } func (e *UpstreamFailoverError) Error() string { diff --git a/backend/internal/service/openai_continuation_diagnostic.go b/backend/internal/service/openai_continuation_diagnostic.go index ee07248b5b38..2c209010031d 100644 --- a/backend/internal/service/openai_continuation_diagnostic.go +++ b/backend/internal/service/openai_continuation_diagnostic.go @@ -51,6 +51,10 @@ type openAIContinuationRecoveryShape struct { // ciphertext of the request and could not repair it. Such a request is // handed to another account when one is left to try. AccountMismatch bool `json:"account_mismatch,omitempty"` + // AccountFailure marks an attempt whose stripped retry was sent and failed + // for a reason of the account's own. Such a request is handed to another + // account when one is left to try. + AccountFailure bool `json:"account_failure,omitempty"` } // Bytes, Characters and SHA256 always describe the complete original value. diff --git a/backend/internal/service/openai_gateway_forward.go b/backend/internal/service/openai_gateway_forward.go index 81a776e73d73..bc3d268c9583 100644 --- a/backend/internal/service/openai_gateway_forward.go +++ b/backend/internal/service/openai_gateway_forward.go @@ -1086,7 +1086,17 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco continue } if reasoningRecovery.RecoveryAttempt() { - return nil, reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, respBody) + failure := reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, respBody) + // Only an answer that a first send would move to another account + // is the account's failure; any other ends the request here. + retryMsg := sanitizeUpstreamErrorMessage(strings.TrimSpace(extractUpstreamErrorMessage(respBody))) + if !s.shouldFailoverOpenAIUpstreamResponse(account, resp.StatusCode, retryMsg, respBody) { + failure.NextAccountAction = NextAccountStop + } else if failure.ShouldRetryNextAccount() { + failure = s.openAIRecoveryRetryHTTPFailure(account, resp.StatusCode, resp.Header, respBody, retryMsg, + openAIHTTPPoolRetryable(ctx, account, resp.StatusCode, retryMsg, respBody, false)) + } + return nil, failure } if _, rejected := parseOpenAIReasoningRejection(respBody); rejected { return nil, NewOpenAIContinuationStateUnavailableError(resp.StatusCode, resp.Header, respBody) diff --git a/backend/internal/service/openai_gateway_passthrough.go b/backend/internal/service/openai_gateway_passthrough.go index 2f2b00f606d0..c29dca044fe2 100644 --- a/backend/internal/service/openai_gateway_passthrough.go +++ b/backend/internal/service/openai_gateway_passthrough.go @@ -437,7 +437,17 @@ retryUpstream: continue } if reasoningRecovery.RecoveryAttempt() { - return nil, reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, probeBody) + failure := reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, probeBody) + // Only an answer that a first send would move to another account is + // the account's failure; any other ends the request here. + if !shouldFailoverOpenAIPassthroughResponse(account, resp.StatusCode, probeBody) { + failure.NextAccountAction = NextAccountStop + } else if failure.ShouldRetryNextAccount() { + retryMsg := sanitizeUpstreamErrorMessage(strings.TrimSpace(extractUpstreamErrorMessage(probeBody))) + failure = s.openAIRecoveryRetryHTTPFailure(account, resp.StatusCode, resp.Header, probeBody, retryMsg, + account != nil && account.IsPoolMode() && account.IsPoolModeRetryableStatus(resp.StatusCode)) + } + return nil, failure } if _, rejected := parseOpenAIReasoningRejection(probeBody); rejected { return nil, NewOpenAIContinuationStateUnavailableError(resp.StatusCode, resp.Header, probeBody) diff --git a/backend/internal/service/openai_gateway_upstream_errors.go b/backend/internal/service/openai_gateway_upstream_errors.go index 6617b8d94b1f..6f614ac60a1b 100644 --- a/backend/internal/service/openai_gateway_upstream_errors.go +++ b/backend/internal/service/openai_gateway_upstream_errors.go @@ -678,6 +678,56 @@ func (e *UpstreamFailoverError) IsOpenAICiphertextAccountMismatch() bool { return e != nil && e.CiphertextAccountMismatch } +// openAIRecoveryRetryAccountFailure marks the failure of a stripped retry that +// is the account's own, such as exhausted credit or a provider error. The +// request goes to the next account as for any account failure. The mark +// changes nothing else: classification, body and the fields account health +// reads stay as the attempt built them. The mark alone tells the handler that +// this account, whose one stripped retry is spent, is not asked again. +func openAIRecoveryRetryAccountFailure(failure *UpstreamFailoverError) *UpstreamFailoverError { + marked := *failure + marked.ResponseHeaders = failure.ResponseHeaders.Clone() + marked.RecoveryRetrySpent = true + return &marked +} + +// openAIRecoveryRetryHTTPFailure classifies an HTTP answer to the stripped +// retry that a first send would move to another account, as that first send +// classifies it: a request-budget rejection stays the request's, an +// unsupported model stays the account/model pair's, a budget 429 stays the +// account's, and any other status the pool retries stays the pool's. +// +// It writes no account state and opens no same-account retry window. Whether +// a first send would also have disabled the account, which makes it count a +// pool-retryable status against the account, is not known here. +func (s *OpenAIGatewayService) openAIRecoveryRetryHTTPFailure( + account *Account, + statusCode int, + headers http.Header, + body []byte, + upstreamMsg string, + poolRetryable bool, +) *UpstreamFailoverError { + body = bytes.Clone(body) + switch { + case account != nil && account.IsOpenAICompatible() && isOpenAIBudgetExceededResponse(statusCode, body): + failure := newOpenAIUpstreamFailoverError(statusCode, headers, body, openAIBudgetExceededMessage(body), false) + failure.Scope = GatewayFailureScopeAccount + failure.NextAccountAction = NextAccountRetry + return failure + case isOpenAIRequestBudgetRejection(account, statusCode, body), + account != nil && account.IsOpenAICompatible() && isOpenAIModelNotSupportedError(statusCode, upstreamMsg, body): + return s.newOpenAIAccountFailoverError(account, statusCode, headers, body, upstreamMsg, false, false) + } + return newOpenAIUpstreamFailoverError(statusCode, headers, body, upstreamMsg, poolRetryable) +} + +// IsOpenAIRecoveryRetrySpent reports an account failure that ended an attempt +// whose stripped retry was already sent. +func (e *UpstreamFailoverError) IsOpenAIRecoveryRetrySpent() bool { + return e != nil && e.RecoveryRetrySpent +} + // openAIUpstreamReportsUndecryptableCiphertext reads an upstream failure for a // statement that ciphertext in the request could not be decrypted or verified. // Besides the continuation-state classification it accepts the wording used diff --git a/backend/internal/service/openai_reasoning_recovery.go b/backend/internal/service/openai_reasoning_recovery.go index 33733ceb376e..acb4b67d5fc7 100644 --- a/backend/internal/service/openai_reasoning_recovery.go +++ b/backend/internal/service/openai_reasoning_recovery.go @@ -84,6 +84,9 @@ type openAIReasoningRecoveryState struct { switchAccounts bool // accountMismatch records that this attempt ended as an account mismatch. accountMismatch bool + // accountFailure records that the stripped retry failed for a reason of + // the account's own. + accountFailure bool } func (s *OpenAIGatewayService) newOpenAIReasoningRecoveryState(ctx context.Context, c *gin.Context, account *Account, token string) *openAIReasoningRecoveryState { @@ -611,8 +614,10 @@ func openAIReasoningRecoveryProtectedStatus(status int) bool { // and account-health accounting, but deliberately does not implement Unwrap. // A caller must handle this terminal explicitly, never feed Failure back into // scheduling: the single same-source recovery budget has already been spent. -// An attempt that ended as an account mismatch is not a terminal; StopError -// returns its failure marked by openAICiphertextAccountMismatch instead. +// Two outcomes are not terminals, and StopError returns the attempt's failure +// marked for the next account instead: an account mismatch +// (openAICiphertextAccountMismatch) and a stripped retry that failed for a +// reason of the account's own (openAIRecoveryRetryAccountFailure). type OpenAIReasoningRecoveryTerminalError struct { Failure *UpstreamFailoverError // FailureTerminalForwarded is explicit parser/write evidence, not a claim @@ -676,6 +681,10 @@ func (r *openAIReasoningRecoveryState) StopError(err error) error { if errors.As(err, &stopped) { return err } + // returned is the failure the attempt itself produced, if any; failure may + // be replaced below by one built from the observed payload. + var returned *UpstreamFailoverError + errors.As(err, &returned) var failure *UpstreamFailoverError var signal *openAIReasoningRecoverySignalError if errors.As(err, &signal) { @@ -690,6 +699,7 @@ func (r *openAIReasoningRecoveryState) StopError(err error) error { } cacheSkipRejected := r.cacheSkippedItems > 0 && failure != nil && failure.IsOpenAIRequestRejected() r.accountMismatch = r.anotherAccountMayAccept(failure, signatureRejected) + r.accountFailure = !r.accountMismatch && r.accountFailedAfterRetry(returned) if !r.retryUsed && !signatureRejected && !cacheSkipRejected && !r.accountMismatch { return err } @@ -713,6 +723,13 @@ func (r *openAIReasoningRecoveryState) StopError(err error) error { if r.accountMismatch { return openAICiphertextAccountMismatch(failure) } + if r.accountFailure { + handOff := openAIRecoveryRetryAccountFailure(returned) + if r.redactPayload != nil && len(handOff.ResponseBody) > 0 { + handOff.ResponseBody = r.redactPayload(handOff.ResponseBody) + } + return handOff + } if !r.retryUsed && !r.semanticCommitted { return failure } @@ -749,6 +766,21 @@ func (r *openAIReasoningRecoveryState) anotherAccountMayAccept(failure *Upstream } } +// accountFailedAfterRetry reports that the stripped retry was sent and the +// attempt then ended, before anything reached the client, with a failure of +// the account itself: one the attempt returned as a failover to the next +// account, which is how the same answer is treated on a first send. A request +// rejection, continuation state and a refusal are not the account's, and an +// error the attempt did not classify as a failover ends the request as before. +func (r *openAIReasoningRecoveryState) accountFailedAfterRetry(returned *UpstreamFailoverError) bool { + if returned == nil || !r.switchAccounts || !r.retryUsed || !r.retryDispatched || + r.semanticCommitted || r.failureTerminalForwarded || r.ctx.Err() != nil { + return false + } + return returned.ShouldRetryNextAccount() && !returned.IsOpenAIRequestRejected() && + !returned.IsOpenAIContinuationStateUnavailable() && !returned.IsOpenAIRefusalRecovery() +} + // ResetOpenAIReasoningRecoveryAttempt detaches the state of a rejected attempt // before the request moves to another account, so that a forwarder which // creates no state of its own does not read this one. The response header that @@ -784,6 +816,7 @@ func (r *openAIReasoningRecoveryState) continuationDiagnosticRecovery(payload [] RetryAttempted: r.retryDispatched, Disposition: r.diagnosticState, AccountMismatch: r.accountMismatch, + AccountFailure: r.accountFailure, } if shape.Disposition == "" { shape.Disposition = "not_attempted" diff --git a/backend/internal/service/openai_reasoning_recovery_terminal_test.go b/backend/internal/service/openai_reasoning_recovery_terminal_test.go index 14a7e9460d1b..754b752aa3d7 100644 --- a/backend/internal/service/openai_reasoning_recovery_terminal_test.go +++ b/backend/internal/service/openai_reasoning_recovery_terminal_test.go @@ -118,7 +118,7 @@ func TestOpenAIReasoningRecoveryOutcomesPreserveClassificationAndDiagnostic(t *t var stopped *OpenAIReasoningRecoveryTerminalError if retried && !mismatch { require.ErrorAs(t, err, &stopped) - require.False(t, errors.As(err, &failure), "spent recovery must not reenter failover") + require.False(t, errors.As(err, &failure), "a validation rejection of the stripped retry is a terminal") failure = stopped.Failure } else { require.False(t, errors.As(err, &stopped), "an account mismatch is not a terminal") diff --git a/backend/internal/service/openai_reasoning_recovery_test.go b/backend/internal/service/openai_reasoning_recovery_test.go index aff197419510..cc94b7cf1abb 100644 --- a/backend/internal/service/openai_reasoning_recovery_test.go +++ b/backend/internal/service/openai_reasoning_recovery_test.go @@ -168,6 +168,74 @@ func TestOpenAIReasoningRecoveryStopErrorMarksAccountMismatch(t *testing.T) { } } +func TestOpenAIReasoningRecoveryStopErrorHandsAnAccountFailureOfTheStrippedRetryOn(t *testing.T) { + provider := func() *UpstreamFailoverError { + return &UpstreamFailoverError{StatusCode: http.StatusBadGateway, ResponseBody: []byte(`{"error":{"code":"credit_balance_exhausted","message":"no credits"}}`), RetryableOnSameAccount: true, SameAccountRetryMax: 2} + } + tests := []struct { + name string + attempt func() error + arrange func(*openAIReasoningRecoveryState) + want bool + }{ + {"provider_failure", func() error { return provider() }, nil, true}, + {"request_rejected", func() error { return NewOpenAIRequestRejectedError(http.StatusBadRequest, nil) }, nil, false}, + {"refusal", func() error { return NewOpenAIRefusalRecoveryFailoverError(nil) }, nil, false}, + {"failure_that_stops", func() error { f := provider(); f.NextAccountAction = NextAccountStop; return f }, nil, false}, + {"unclassified_error", func() error { return errors.New("stream read error") }, nil, false}, + {"retry_not_sent", func() error { return provider() }, func(r *openAIReasoningRecoveryState) { r.retryDispatched = false }, false}, + {"output_already_sent", func() error { return provider() }, func(r *openAIReasoningRecoveryState) { r.semanticCommitted = true }, false}, + {"failure_already_forwarded", func() error { return provider() }, func(r *openAIReasoningRecoveryState) { + // A forwarded terminal always follows an observed failure payload. + r.ObserveFailure([]byte(`{"type":"response.failed","response":{"status":"failed","error":{"code":"credit_balance_exhausted"}}}`), false) + r.failureTerminalForwarded = true + }, false}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + state, req, _ := newReasoningRecoveryTestState(t, context.Background(), reasoningRecoveryFixture) + state.BindDiagnosticRequest([]byte(reasoningRecoveryFixture), req) + _, retry := state.TryRecover(http.StatusBadRequest, nil, []byte(`{"error":{"code":"invalid_encrypted_content"}}`), false) + require.True(t, retry) + // The stripped retry is sent; its failure is observed on a new attempt. + state.BindDiagnosticRequest([]byte(reasoningRecoveryFixture), req) + state.MarkAttemptDispatched() + state.redactPayload = func([]byte) []byte { return []byte(`{"error":{"message":"redacted"}}`) } + if test.arrange != nil { + test.arrange(state) + } + attempt := test.attempt() + stopped := state.StopError(attempt) + var failure *UpstreamFailoverError + var terminal *OpenAIReasoningRecoveryTerminalError + if errors.As(stopped, &terminal) { + require.False(t, test.want, "an account failure of the stripped retry is not a terminal") + require.False(t, terminal.Failure.ShouldRetryNextAccount()) + return + } + require.ErrorAs(t, stopped, &failure) + require.Equal(t, test.want, failure.IsOpenAIRecoveryRetrySpent()) + if !test.want { + return + } + require.True(t, failure.ShouldRetryNextAccount()) + require.Equal(t, http.StatusBadGateway, failure.StatusCode) + // Everything account health reads is what a first send carries. + require.True(t, failure.RetryableOnSameAccount) + require.Equal(t, 2, failure.SameAccountRetryMax) + require.JSONEq(t, `{"error":{"message":"redacted"}}`, string(failure.ResponseBody), "the body leaves the attempt redacted") + var original *UpstreamFailoverError + require.ErrorAs(t, attempt, &original) + require.False(t, original.IsOpenAIRecoveryRetrySpent(), "the attempt's own error value is not modified") + require.JSONEq(t, string(provider().ResponseBody), string(original.ResponseBody)) + events, _ := state.c.Get(OpsUpstreamErrorsKey) + recorded := events.([]*OpsUpstreamErrorEvent) + require.Equal(t, "recovery_failed", recorded[len(recorded)-1].Message) + require.True(t, recorded[len(recorded)-1].ContinuationDiagnostic.Recovery.AccountFailure) + }) + } +} + func TestResetOpenAIReasoningRecoveryAttemptDetachesTheState(t *testing.T) { state, _, rec := newReasoningRecoveryTestState(t, context.Background(), reasoningRecoveryFixture) _, retry := state.TryRecover(http.StatusBadRequest, nil, []byte(`{"error":{"code":"invalid_encrypted_content"}}`), false) @@ -400,6 +468,128 @@ func TestOpenAIReasoningRecoveryIdentityWithoutReasoningStillIsolatesSources(t * require.Equal(t, one, two, "cache scope uses semantic JSON identity, not reasoning key order") } +// A status the pool retries on the same account is the pool's condition, not +// the account's: the API-key health breaker does not count it on a first send, +// and must not count it when it answers the stripped retry either. +func TestOpenAIReasoningRecoveryRetryAccountFailureKeepsThePoolExemption(t *testing.T) { + for _, passthrough := range []bool{false, true} { + t.Run(map[bool]string{true: "passthrough", false: "native"}[passthrough], func(t *testing.T) { + first := newJSONResponse(400, `{"error":{"code":"thinking_signature_invalid","param":"input[1].encrypted_content"}}`) + first.Header.Set("Content-Type", "application/json") + second := newJSONResponse(http.StatusTooManyRequests, `{"error":{"type":"rate_limit_error","code":"rate_limit_exceeded","message":"slow down"}}`) + second.Header.Set("Content-Type", "application/json") + upstream := &httpUpstreamRecorder{responses: []*http.Response{first, second}} + svc := newOpenAIImageGenerationControlTestService(upstream) + c, _ := newOpenAIImageGenerationControlTestContext(false, "test-client") + account := newOpenAIImageGenerationControlTestAccount() + account.Credentials["pool_mode"] = true + account.Extra = map[string]any{"openai_passthrough": passthrough, "responses_api_supported": true} + require.True(t, account.IsPoolMode() && account.IsPoolModeRetryableStatus(http.StatusTooManyRequests), "fixture: 429 is pool-retryable") + _, err := svc.Forward(context.Background(), c, account, []byte(reasoningRecoveryFixture)) + var failover *UpstreamFailoverError + require.ErrorAs(t, err, &failover) + require.True(t, failover.IsOpenAIRecoveryRetrySpent()) + require.True(t, failover.RetryableOnSameAccount) + _, _, counted := classifyOpenAIAPIKeyHealthFailure(err) + require.False(t, counted, "the breaker exemption of a first send is kept") + }) + } +} + +// An answer a first send classifies by more than its status keeps that +// classification when it answers the stripped retry, so account health sees +// the same failure either way. +func TestOpenAIReasoningRecoveryRetryAccountFailureKeepsTheFirstSendClassification(t *testing.T) { + tests := []struct { + name string + status int + body string + pool bool + check func(*testing.T, *UpstreamFailoverError, error) + }{ + { + name: "request_budget_rejected", + status: http.StatusForbidden, + body: `{"error":{"type":"balance_insufficient_error","code":"balance_insufficient","message":"private balance"}}`, + check: func(t *testing.T, failure *UpstreamFailoverError, _ error) { + require.Equal(t, GatewayFailureScopeRequest, failure.Scope) + require.True(t, failure.SuppressAccountHealthPenalty) + require.Equal(t, "upstream_request_budget_rejected", failure.ClientErrorCode) + }, + }, + { + name: "model_not_supported", + status: http.StatusBadRequest, + body: `{"error":{"type":"model_not_supported","code":"400","message":"The requested model is not supported"}}`, + check: func(t *testing.T, failure *UpstreamFailoverError, _ error) { + require.Equal(t, openAIModelNotSupportedReason, failure.Reason) + require.True(t, failure.SuppressAccountHealthPenalty) + }, + }, + { + name: "budget_exceeded_on_a_pool_account", + status: http.StatusTooManyRequests, + body: `{"error":{"type":"budget_exceeded","message":"budget spent"}}`, + pool: true, + check: func(t *testing.T, failure *UpstreamFailoverError, err error) { + require.Equal(t, GatewayFailureScopeAccount, failure.Scope) + require.False(t, failure.RetryableOnSameAccount, "a spent budget is the account's, whatever the pool retries") + _, _, counted := classifyOpenAIAPIKeyHealthFailure(err) + require.True(t, counted) + }, + }, + } + for _, test := range tests { + for _, passthrough := range []bool{false, true} { + t.Run(test.name+map[bool]string{true: "/passthrough", false: "/native"}[passthrough], func(t *testing.T) { + first := newJSONResponse(400, `{"error":{"code":"thinking_signature_invalid","param":"input[1].encrypted_content"}}`) + first.Header.Set("Content-Type", "application/json") + second := newJSONResponse(test.status, test.body) + second.Header.Set("Content-Type", "application/json") + upstream := &httpUpstreamRecorder{responses: []*http.Response{first, second}} + svc := newOpenAIImageGenerationControlTestService(upstream) + c, _ := newOpenAIImageGenerationControlTestContext(false, "test-client") + account := newOpenAIImageGenerationControlTestAccount() + account.Credentials["pool_mode"] = test.pool + account.Extra = map[string]any{"openai_passthrough": passthrough, "responses_api_supported": true} + _, err := svc.Forward(context.Background(), c, account, []byte(reasoningRecoveryFixture)) + var failover *UpstreamFailoverError + require.ErrorAs(t, err, &failover) + require.Len(t, upstream.requests, 2) + require.True(t, failover.IsOpenAIRecoveryRetrySpent()) + require.True(t, failover.ShouldRetryNextAccount()) + require.Equal(t, test.status, failover.StatusCode) + test.check(t, failover, err) + }) + } + } +} + +// An answer to the stripped retry that a first send would not move to another +// account is not the account's failure: the request ends on this account. +func TestOpenAIReasoningRecoveryRetryAnswerThatIsNoFailoverEndsTheRequest(t *testing.T) { + for _, passthrough := range []bool{false, true} { + t.Run(map[bool]string{true: "passthrough", false: "native"}[passthrough], func(t *testing.T) { + first := newJSONResponse(400, `{"error":{"code":"thinking_signature_invalid","param":"input[1].encrypted_content"}}`) + first.Header.Set("Content-Type", "application/json") + second := newJSONResponse(http.StatusNotFound, `{"error":{"code":"not_found","message":"private upstream detail"}}`) + second.Header.Set("Content-Type", "application/json") + upstream := &httpUpstreamRecorder{responses: []*http.Response{first, second}} + svc := newOpenAIImageGenerationControlTestService(upstream) + c, _ := newOpenAIImageGenerationControlTestContext(false, "test-client") + account := newOpenAIImageGenerationControlTestAccount() + account.Extra = map[string]any{"openai_passthrough": passthrough, "responses_api_supported": true} + _, err := svc.Forward(context.Background(), c, account, []byte(reasoningRecoveryFixture)) + require.Len(t, upstream.requests, 2) + var stopped *OpenAIReasoningRecoveryTerminalError + require.ErrorAs(t, err, &stopped) + require.False(t, stopped.Failure.IsOpenAIRecoveryRetrySpent()) + require.False(t, stopped.Failure.ShouldRetryNextAccount()) + require.Equal(t, http.StatusNotFound, stopped.Failure.StatusCode) + }) + } +} + func TestOpenAIReasoningRecoveryForwardHTTPAndPassthroughSingleRetry(t *testing.T) { for _, passthrough := range []bool{false, true} { for _, secondOK := range []bool{false, true} { @@ -430,15 +620,19 @@ func TestOpenAIReasoningRecoveryForwardHTTPAndPassthroughSingleRetry(t *testing. require.Contains(t, rec.Body.String(), "resp_final") } else { require.Error(t, err) - var failover *UpstreamFailoverError - require.False(t, errors.As(err, &failover)) + // A provider failure of the stripped retry is the account's own: + // the request is handed to the next account, as on a first send, + // and never back to this one. var stopped *OpenAIReasoningRecoveryTerminalError - require.ErrorAs(t, err, &stopped) - require.Equal(t, http.StatusServiceUnavailable, stopped.Failure.StatusCode) - require.False(t, stopped.Failure.IsOpenAIRequestRejected()) - require.False(t, stopped.Failure.SuppressAccountHealthPenalty, "a genuine provider failure is not fabricated as a request validation error") - require.True(t, stopped.Failure.ShouldReportAccountScheduleFailure()) - require.False(t, stopped.Failure.ShouldRetryNextAccount()) + require.False(t, errors.As(err, &stopped)) + var failover *UpstreamFailoverError + require.ErrorAs(t, err, &failover) + require.True(t, failover.IsOpenAIRecoveryRetrySpent()) + require.Equal(t, http.StatusServiceUnavailable, failover.StatusCode) + require.False(t, failover.IsOpenAIRequestRejected()) + require.False(t, failover.SuppressAccountHealthPenalty, "a genuine provider failure is not fabricated as a request validation error") + require.True(t, failover.ShouldReportAccountScheduleFailure()) + require.True(t, failover.ShouldRetryNextAccount()) require.NotContains(t, err.Error(), "private upstream failure") } }) From 5077a00dc9e85e9883418c383f494b523c51dffa Mon Sep 17 00:00:00 2001 From: Codex Date: Mon, 5 Oct 2026 19:45:05 +0800 Subject: [PATCH 2/2] docs(openai): state the stripped-retry hand-off as the code behaves Comments and handbook wording only; no behaviour change. - The handler's retry-exhausted cooldown does nothing for an OpenAI API-key account, so the comment, the test comment and the handbook no longer say such an account is cooled after an HTTP answer to its stripped retry. - A non-failover answer to the stripped retry ends the request unless it is a ciphertext rejection, which StopError still hands on as a mismatch. - A recovery terminal after committed output needs a ciphertext rejection. - The handbook names per path which answers move to the next account: the passthrough rule is narrower than the native one. Co-Authored-By: Claude Opus 5.5 --- .downstream/gpt6-models-and-prompts.md | 2 +- ...enai_continuation_account_mismatch_test.go | 5 +++-- .../handler/openai_gateway_handler.go | 21 +++++++++++-------- .../service/openai_gateway_forward.go | 3 ++- .../service/openai_gateway_passthrough.go | 3 ++- .../service/openai_reasoning_recovery_test.go | 3 ++- 6 files changed, 22 insertions(+), 15 deletions(-) diff --git a/.downstream/gpt6-models-and-prompts.md b/.downstream/gpt6-models-and-prompts.md index 504a23ba6ccb..8919caceaa79 100644 --- a/.downstream/gpt6-models-and-prompts.md +++ b/.downstream/gpt6-models-and-prompts.md @@ -66,7 +66,7 @@ Claude OAuth 伪装系统块只由上游设置决定:设置 → 网关的 `ena 侧栏“扩展功能 → 推理恢复”(`/admin/reasoning-recovery`):一个全局开关和保存按钮,默认开启,没有账号级选项;页面打开时读取一次,读取失败时显示错误、不显示开关。存为 `settings.reasoning_recovery_config` 一行 JSON(`{"enabled": }`),没有该行即开启;`GET/PUT /api/v1/admin/reasoning-recovery` 以标准响应封装读写 `{"enabled": }`,管理员认证、无二次验证,保存记入操作审计,请求体不是严格的 `{"enabled": }` 时返回 400 且不保存。请求路径只读内存中的布尔值(保存即替换,60 秒后台刷新以同步多实例),不查询数据库,每次转发开始时读取一次。启动时读取失败或存储值不是严格的 `{"enabled": }` 则停止启动;运行中刷新失败保留当前值。 -开启时,OpenAI 平台的 API Key、OAuth 和 Setup Token 账号(含自动透传)在原生 Responses(含 compact)与 Chat Completions 转 Responses 的 HTTP 请求上生效:上游以结构化错误码 `invalid_encrypted_content` 或 `thinking_signature_invalid` 拒绝、且尚无语义输出时,去掉推理项的 `encrypted_content`,在同一账号重发一次:上游用 `input[N]` 指明条目时只去掉该项,否则去掉请求里全部推理项的密文。上游未用 `input[N]` 指明条目、而请求里还有其他密文载体(压缩项、多智能体 `agent_message` 的正文段)时,只有上游错误消息按条目 id 点名的是带密文的推理项才恢复,否则不恢复;这条点名规则只在推理项 id 随请求发出的路径上成立(API Key 账号与自动透传),OAuth / Setup Token 的非透传请求发出前会去掉推理项 id,遇到其他密文载体仍不恢复。其他载体的密文任何情况下都不改动。被去掉的密文按来源在 Redis 记忆 24 小时,期间同源请求发送前即去掉;记忆按每批 64 条读写,读与写共用每个请求 100 毫秒的 I/O 预算。本账号修不好时换号:在 `POST /v1/responses`(含 compact 与自动透传)上,上游报告请求里的密文无法解密或校验(结构化错误码、同类错误消息,或 `could not be decrypted` 的措辞),而本账号无法修复(不能恢复,或去掉推理密文后的重发、发送前已去掉密文的请求仍得到这类拒绝),且客户端尚未收到本次尝试的任何输出时,请求原样交给分组里的下一个账号——密文只能由签发它的上游组织解开,别的账号可能直接接受。每个请求最多因此换号 3 次(同时计入正常换号预算),下一个账号各自按上述规则恢复一次;被换下的账号不计健康惩罚,也不做同账号重试。换号上限或可选账号用尽时,客户端收到第一个报告密文问题的账号的那个错误;途中某个账号给出自己的终态(例如读得懂密文但请求本身无效)或网关侧容量限制时,返回该结果。没有账号给出回答时,被这次换号挪到后续账号上的粘性绑定回到第一个报告密文问题的账号,调度器保留在别处的绑定不动;有账号接受或已开始输出时,会话留在该账号。普通的请求校验错误、带 `previous_response_id` 或 `conversation` 的请求、上游状态为 401/402/403/407/429 时不换号;先试过透传账号再换到非透传账号时,沿用既有规则整块去掉加密推理项。经原生 Responses 路径转发的其他平台账号同样适用;Chat Completions 转 Responses 不换号。Ops 续接诊断用 `recovery.account_mismatch` 标出因密文读不了而交给下一个账号的尝试。恢复重发因账号自身的原因失败时也换号(同样只在 `POST /v1/responses`,含 compact 与自动透传;Chat Completions 转 Responses 不换号):重发已发出、失败属于首次发送时就会换号的那一类(额度用尽、限流、鉴权、5xx、流内的上游失败),且客户端尚未收到本次尝试的任何输出时,请求按普通账号失败交给下一个账号——下一个账号收到原样请求,本账号不再重试,换号预算同普通换号,账号健康按首次发送时的分类记录(调度失败上报、同账号重试用尽后的冷却):请求预算被拒(402/403 `balance_insufficient`)仍只算本次请求,结构化的 `model_not_supported` 仍只算该账号与模型,`budget_exceeded` 429 仍算账号,其余池模式可重试的状态仍不计入 API Key 健康熔断。Ops 续接诊断用 `recovery.account_failure` 标出这样交给下一个账号的尝试。重发的 HTTP 错误应答不执行转发层在首次发送时的账号状态写入(标记限流、置为错误、停用、预算用尽标记),也不开启 OAuth 429 的同账号重试窗口,这些由之后的首次发送处理;首次发送因同时停用账号而把池模式可重试状态计入账号的情形,这里无从得知,按不计入处理;处理器侧的冷却照常执行,其中 OAuth 账号遇到硬额度 429 时会写入额度状态。重发遇到请求校验错误、拒绝类结果、传输错误、响应头超时或未归类为可换号的错误时,仍在本账号结束。Messages 桥接与 WebSocket 不恢复。关闭时不重发也不预先去除,签名拒绝仍作为请求级终态返回,不切换账号。 +开启时,OpenAI 平台的 API Key、OAuth 和 Setup Token 账号(含自动透传)在原生 Responses(含 compact)与 Chat Completions 转 Responses 的 HTTP 请求上生效:上游以结构化错误码 `invalid_encrypted_content` 或 `thinking_signature_invalid` 拒绝、且尚无语义输出时,去掉推理项的 `encrypted_content`,在同一账号重发一次:上游用 `input[N]` 指明条目时只去掉该项,否则去掉请求里全部推理项的密文。上游未用 `input[N]` 指明条目、而请求里还有其他密文载体(压缩项、多智能体 `agent_message` 的正文段)时,只有上游错误消息按条目 id 点名的是带密文的推理项才恢复,否则不恢复;这条点名规则只在推理项 id 随请求发出的路径上成立(API Key 账号与自动透传),OAuth / Setup Token 的非透传请求发出前会去掉推理项 id,遇到其他密文载体仍不恢复。其他载体的密文任何情况下都不改动。被去掉的密文按来源在 Redis 记忆 24 小时,期间同源请求发送前即去掉;记忆按每批 64 条读写,读与写共用每个请求 100 毫秒的 I/O 预算。本账号修不好时换号:在 `POST /v1/responses`(含 compact 与自动透传)上,上游报告请求里的密文无法解密或校验(结构化错误码、同类错误消息,或 `could not be decrypted` 的措辞),而本账号无法修复(不能恢复,或去掉推理密文后的重发、发送前已去掉密文的请求仍得到这类拒绝),且客户端尚未收到本次尝试的任何输出时,请求原样交给分组里的下一个账号——密文只能由签发它的上游组织解开,别的账号可能直接接受。每个请求最多因此换号 3 次(同时计入正常换号预算),下一个账号各自按上述规则恢复一次;被换下的账号不计健康惩罚,也不做同账号重试。换号上限或可选账号用尽时,客户端收到第一个报告密文问题的账号的那个错误;途中某个账号给出自己的终态(例如读得懂密文但请求本身无效)或网关侧容量限制时,返回该结果。没有账号给出回答时,被这次换号挪到后续账号上的粘性绑定回到第一个报告密文问题的账号,调度器保留在别处的绑定不动;有账号接受或已开始输出时,会话留在该账号。普通的请求校验错误、带 `previous_response_id` 或 `conversation` 的请求、上游状态为 401/402/403/407/429 时不换号;先试过透传账号再换到非透传账号时,沿用既有规则整块去掉加密推理项。经原生 Responses 路径转发的其他平台账号同样适用;Chat Completions 转 Responses 不换号。Ops 续接诊断用 `recovery.account_mismatch` 标出因密文读不了而交给下一个账号的尝试。恢复重发因账号自身的原因失败时也换号(同样只在 `POST /v1/responses`,含 compact 与自动透传;Chat Completions 转 Responses 不换号):重发已发出、失败属于该路径首次发送时就会换号的那一类,且客户端尚未收到本次尝试的任何输出时,请求按普通账号失败交给下一个账号——下一个账号收到原样请求,本账号不再重试,换号预算同普通换号。哪些失败算这一类以各路径首次发送的换号规则为准:非透传包括额度用尽、限流、鉴权失败、5xx 与流内的上游失败;自动透传的规则更窄,OAuth / Setup Token 透传的普通 5xx、不带账号状态信息的 401/403 等不在其内,仍在本账号结束。Ops 续接诊断用 `recovery.account_failure` 标出这样交给下一个账号的尝试。账号健康:失败的分类与首次发送相同(请求预算被拒 402/403 `balance_insufficient` 只算本次请求,结构化的 `model_not_supported` 只算该账号与模型,`budget_exceeded` 429 算账号,其余池模式可重试的状态不计入 API Key 健康熔断),调度失败照常上报,流内失败照常写入账号状态;重发的 HTTP 错误应答则不执行转发层在首次发送时的账号状态写入(标记限流、置为错误、停用、预算用尽标记、API Key 的瞬时失败计数),也不开启 OAuth 429 的同账号重试窗口,留给之后的首次发送处理,其中首次发送因同时停用账号而把池模式可重试状态计入账号的情形按不计入处理。处理器侧同账号重试用尽后的冷却照常调用,它只对 OAuth / Setup Token 账号起作用(硬额度 429 会写入额度状态);OpenAI API Key 账号在 HTTP 错误应答下不被冷却,下一个请求仍可能先发给它,由那次首次发送写入状态。重发遇到请求校验错误、拒绝类结果、传输错误、响应头超时或未归类为可换号的错误时,仍在本账号结束;重发再次被报告密文无法解密时,按上文的密文规则换号。Messages 桥接与 WebSocket 不恢复。关闭时不重发也不预先去除,签名拒绝仍作为请求级终态返回,不切换账号。 迁移 `267_purge_account_reasoning_options.sql` 一次性删除各账号 `extra` 中原有的两个账号级推理选项键(键名见该文件);此后没有代码读写它们,已存的账号级取值不带入全局开关。Chat Completions 转 Responses 的请求只由客户端消息转换而来,网关不保存响应中的推理项。 diff --git a/backend/internal/handler/openai_continuation_account_mismatch_test.go b/backend/internal/handler/openai_continuation_account_mismatch_test.go index 09fdaf301c7a..fec991336bce 100644 --- a/backend/internal/handler/openai_continuation_account_mismatch_test.go +++ b/backend/internal/handler/openai_continuation_account_mismatch_test.go @@ -367,8 +367,9 @@ func TestOpenAIGatewayHandler_ContinuationStrippedRetryAccountFailureIsNotAMisma require.Equal(t, int64(2), continuationMismatchBoundAccount(t, cache), "no mismatch, so no binding is moved back") } -// The account that failed on its stripped retry gets the cooldown of an account -// whose same-account attempts are used up: the next request is not sent to it. +// An OAuth account that failed on its stripped retry gets the cooldown of an +// account whose same-account attempts are used up: after a 401 the next request +// is not sent to it. (For an OpenAI API-key account that cooldown does nothing.) func TestOpenAIGatewayHandler_ContinuationStrippedRetryAccountFailureCoolsTheAccount(t *testing.T) { gin.SetMode(gin.TestMode) upstream := &continuationMismatchUpstream{ diff --git a/backend/internal/handler/openai_gateway_handler.go b/backend/internal/handler/openai_gateway_handler.go index 63123da85669..5751e568bf5a 100644 --- a/backend/internal/handler/openai_gateway_handler.go +++ b/backend/internal/handler/openai_gateway_handler.go @@ -1052,9 +1052,11 @@ func (h *OpenAIGatewayHandler) Responses(c *gin.Context) { // A mismatch says nothing against the account: no // same-account retry and no cooldown, the next account is tried. case failoverErr.IsOpenAIRecoveryRetrySpent(): - // The account failed on its stripped retry. It is not asked - // again, and gets the cooldown of an account whose - // same-account attempts are used up. + // The account failed on its stripped retry and is not asked + // again for this request. The retry-exhausted cooldown runs as + // for used-up same-account attempts; it does nothing for an + // OpenAI API-key account, whose penalties the forwarding + // layer writes on a first send. h.gatewayService.CooldownOpenAIRetryExhausted(c.Request.Context(), account, account.GetMappedModel(routingModel), failoverErr) default: retryAction = retryState.HandleHTTP( @@ -4081,12 +4083,13 @@ func (h *OpenAIGatewayHandler) handleOpenAINoAccountError(c *gin.Context, classi h.handleStreamingAwareErrorWithCode(c, classification.Status, classification.ErrType, code, classification.Message, streamStarted, false) } -// A recovery terminal ends the request: it is any attempt that spent its -// stripped retry, or had output committed, and was not handed on. Preserve its -// classification without allowing refusal, account, or transport retries to -// re-enter the loop. An account mismatch and an account failure of the -// stripped retry, both possible only before anything reached the client, are -// returned as failovers instead and never reach this function. +// A recovery terminal ends the request: it is an attempt that spent its +// stripped retry, or one whose ciphertext rejection arrived after output was +// committed, and that was not handed on. Preserve its classification without +// allowing refusal, account, or transport retries to re-enter the loop. An +// account mismatch and an account failure of the stripped retry, both +// possible only before anything reached the client, are returned as failovers +// instead and never reach this function. func (h *OpenAIGatewayHandler) handleReasoningRecoveryTerminal(c *gin.Context, err error, selection *service.AccountSelectionResult, account *service.Account, model string, streamStarted bool, reporter openAIFailoverSelectionReporter) bool { var terminal *service.OpenAIReasoningRecoveryTerminalError if !errors.As(err, &terminal) { diff --git a/backend/internal/service/openai_gateway_forward.go b/backend/internal/service/openai_gateway_forward.go index bc3d268c9583..2c368234d31b 100644 --- a/backend/internal/service/openai_gateway_forward.go +++ b/backend/internal/service/openai_gateway_forward.go @@ -1088,7 +1088,8 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco if reasoningRecovery.RecoveryAttempt() { failure := reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, respBody) // Only an answer that a first send would move to another account - // is the account's failure; any other ends the request here. + // is the account's failure. Any other ends the request here, + // unless StopError hands it on as a ciphertext mismatch. retryMsg := sanitizeUpstreamErrorMessage(strings.TrimSpace(extractUpstreamErrorMessage(respBody))) if !s.shouldFailoverOpenAIUpstreamResponse(account, resp.StatusCode, retryMsg, respBody) { failure.NextAccountAction = NextAccountStop diff --git a/backend/internal/service/openai_gateway_passthrough.go b/backend/internal/service/openai_gateway_passthrough.go index c29dca044fe2..7e12f364e6ad 100644 --- a/backend/internal/service/openai_gateway_passthrough.go +++ b/backend/internal/service/openai_gateway_passthrough.go @@ -439,7 +439,8 @@ retryUpstream: if reasoningRecovery.RecoveryAttempt() { failure := reasoningRecovery.FailureForResponse(resp.StatusCode, resp.Header, probeBody) // Only an answer that a first send would move to another account is - // the account's failure; any other ends the request here. + // the account's failure. Any other ends the request here, unless + // StopError hands it on as a ciphertext mismatch. if !shouldFailoverOpenAIPassthroughResponse(account, resp.StatusCode, probeBody) { failure.NextAccountAction = NextAccountStop } else if failure.ShouldRetryNextAccount() { diff --git a/backend/internal/service/openai_reasoning_recovery_test.go b/backend/internal/service/openai_reasoning_recovery_test.go index cc94b7cf1abb..a01c3eb5a9c0 100644 --- a/backend/internal/service/openai_reasoning_recovery_test.go +++ b/backend/internal/service/openai_reasoning_recovery_test.go @@ -566,7 +566,8 @@ func TestOpenAIReasoningRecoveryRetryAccountFailureKeepsTheFirstSendClassificati } // An answer to the stripped retry that a first send would not move to another -// account is not the account's failure: the request ends on this account. +// account, and that is no ciphertext rejection either, is not the account's +// failure: the request ends on this account. func TestOpenAIReasoningRecoveryRetryAnswerThatIsNoFailoverEndsTheRequest(t *testing.T) { for _, passthrough := range []bool{false, true} { t.Run(map[bool]string{true: "passthrough", false: "native"}[passthrough], func(t *testing.T) {