Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions .downstream/codex-quality-diagnostics.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

此入口用于管理员在真实 `/v1/responses` 链路上运行有界对照。路由材料、完整响应、上游模型声明和回答质量是四种不同证据;采集成功不代表回答质量恢复。

本文描述协议与账号身份修复合入后的组合行为:诊断与普通业务共用最终出站身份、workspace、连接租约检查与模型声明守卫;诊断授权、预算、原始观测和路由材料仍独立于普通调度状态。

## 接口与范围

所有管理操作位于 `/api/v1/admin/accounts/:id/codex-quality-runs`,要求真实管理员身份。运行绑定该管理员已有的业务 Key、其分组、目标 Pro OAuth 账号主体、代理、`gpt-6-astra`、显式 `high` 和题面 SHA-256。
Expand All @@ -10,7 +12,7 @@
| --- | --- | --- |
| `POST` 基础路径 | `run_id`、`api_key_id`、`prompt_sha256`、`max_sends`、`ttl_seconds` | 创建或恢复同一运行,换发短时 grant;最多 60 次发送、最长 7200 秒 |
| `GET /:run_id` | 无正文 | 读取预算、原始上游观测和私有路由资格 |
| `POST /:run_id/renew-route` | `operation_id` | 显式采集与业务出口复验,均计入同一预算 |
| `POST /:run_id/renew-route` | `operation_id` | 用 `gpt-6-astra/high` 轻量探针显式采集与业务出口复验,均计入同一预算 |
| `POST /:run_id/close` | 空对象 | 关闭 grant,释放该运行自身连接,擦除私有 Cookie 值,保留账本 |

`run_id`、`operation_id` 与 trial 均使用规范化 UUID。同一 run 恢复时不能改变绑定范围或增加预算,换发 grant 不恢复已花费次数;closed 运行不能重开。
Expand All @@ -24,9 +26,14 @@ grant 与 trial header 在 handler 开始时移入私有执行上下文,不转
- 只在私有选号判断中跳过账号手动停调度开关;账号、Key、分组和代理配置不修改。当前 Key 状态、到期、额度、渠道模型限制、账号健康、并发及路由资格仍检查。
- acquire、verify、business 共用持久 CAS 预算,在本地准备与检查结束、上游 I/O 开始前预留。传输结果未知保留为 unknown,不能重放该 trial;预留次数是保守的发送上限,不能宣称每次预留均已得到上游响应。
- 诊断材料使用独立 bundle、clock 与连接,不能覆盖普通资格,不登记自动续获。GET 不抢占或恢复普通 half-open probe。
- 私有资格和普通资格共用本进程租约检查:连接必须仍存在、未被已知关闭,主体、scope 与有效期须一致。检查不发送探针、不重拨或延长原租约;持久 bundle 本身不能证明进程重启后的连接仍可用。
- 普通健康、调度恢复、proxy circuit 和自动恢复通知与诊断隔离;实际用量账单及额度快照按真实请求记录。
- 记录来源是网关改写前的原始响应。`created_model`、`terminal_model`、HTTP/事件模型头分别保留;模型头冲突保存在 `header_models`。缺失 token 数据为 null,不替换为零。
- `completed` 仅代表成功终态观测,不能替代模型核验或数学判分。当前仍保留修前探针形态,探针未显式给出的 effort 如实记录为空。
- 读取次序固定为网络 → 原始诊断观察 → 模型声明守卫 → 业务响应改写。`created_model`、`terminal_model`、HTTP/事件模型头分别保留,模型头名称不区分大小写;模型头冲突保存在 `header_models`。被守卫拦下的已读原始帧仍记入同一次预算,未读取的正文模型保持未知。缺失 token 数据为 null,不替换为零。
- 诊断私有发送同样使用模型守卫;首输出前的模型错配不写入下游,已经输出的请求不会自动重放。资格要求终态模型与所有已提供的原始模型声明一致,早期声明或模型头冲突不能被最后一个正确标签覆盖。失败只撤销本诊断运行对应连接的资格,不修改普通账号健康状态。
- `acquire`、`verify` 的模型发送显式使用 `high`,仍是轻量 `ping`/`pong` 协议探针,总读取预算保持 128 KiB。`business` 的 SSE 单事件上限复用 `Gateway.MaxLineSize`,JSON 复用 `Gateway.UpstreamResponseReadMaxBytes`;不会套用探针上限,也不使用无界响应缓冲。普通非诊断探针未指定 effort 时继续省略该字段。
- `completed` 只代表原始上游成功终态观测;同一条记录仍可能因早期模型、模型头冲突或客户端取消而不能作为通过样本。它不能替代模型核验或数学判分。

普通 Codex 的 Responses-Lite 请求保留 `input` 中已有基础提示和工具,不再额外填入顶层默认 `instructions`。OAuth STATE 只在同一账号主体、会话和显式逻辑 turn 内保留首个已提交值;Messages 只有完整响应成功写入下游后才保存。缺少 turn 证据不复用旧值,API Key 会话续接契约保持。

## 运维执行器

Expand All @@ -52,4 +59,4 @@ python scripts/invoke-codex-quality-acceptance.py status

诊断基础设施先独立发布并采集修前基线。协议与账号身份修复随后发布,在共同宿主基线、同一账号和同一预算下完成修后序列。不能先应用修复再把结果命名为修前基线。

后续协议整合必须保持读取次序:网络 → 原始诊断观察 → 模型声明检查 → 业务响应改写。诊断业务流的读取限界需与普通业务一致,轻量采集探针保持自身预算。
组合版本只用于修后序列。原始诊断观察始终位于模型守卫内层,避免守卫丢弃错误帧后账本反而看不到错误声明;上面的业务与探针读取限界同样适用于诊断私有传输。
7 changes: 4 additions & 3 deletions .github/workflows/downstream-verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,15 @@ jobs:
- name: Agent protocol contracts
run: bash .github/scripts/test_agent_protocol.sh
- name: First-party plugin contracts and signed process package
# Includes codex-runtime routing outcomes and legacy-state migration.
run: bash .github/scripts/test_first_party_plugins.sh
- name: Reasoning replay and recovery contracts
run: bash .github/scripts/test_reasoning_replay.sh
- name: Model context capacity contracts
run: bash .github/scripts/test_model_context_capacity.sh
- name: Service tests
working-directory: backend
run: go test -p=1 -tags unit ./internal/service -run 'Test.*(Refusal|CyberPolicy|Continuation|InvalidEncryptedContent|PreviousResponseNotFound|CapacityShed|RequestScopedTransient|ServerOverloaded|BusinessSystemPrompt|RemoteSkill|Cindy|CompatibilityModes|AlphaSearch|CodexQuality)|Test(UpdateService|NormalizeAccountTaxonomyName|FilterConsoleAccounts|AccountConsolePlan|AccountFacetMatcher)' -count=1
run: go test -p=1 -tags unit ./internal/service -run 'Test.*(Refusal|CyberPolicy|Continuation|InvalidEncryptedContent|PreviousResponseNotFound|CapacityShed|RequestScopedTransient|ServerOverloaded|BusinessSystemPrompt|RemoteSkill|Cindy|CompatibilityModes|AlphaSearch|CodexQuality|SelectedWorkspace)|Test(UpdateService|NormalizeAccountTaxonomyName|FilterConsoleAccounts|AccountConsolePlan|AccountFacetMatcher)' -count=1
- name: Upstream automation classifier tests
run: |
python3 .github/scripts/test_upstream_risk.py
Expand All @@ -77,10 +78,10 @@ jobs:
- name: Plugin policy cancellation race tests
working-directory: backend
run: go test -race -tags unit ./internal/service -run '^(TestPluginPolicyCancellationSurvivesUpstreamDetachWithoutCancelingBilling|TestPluginPolicyContextsStopOnConfigurationChangeAndDisable|TestSavingIdenticalPluginConfigurationDoesNotCancelHostWork|TestPluginOwnedHostJobHoldsUpdateLeaseUntilCleanup|TestAccountJobRetryPreservesOwnerAndAdmitsCurrentGeneration|TestLegacyImportJobUsesCurrentPluginLifetimeWithoutRewritingHistory|TestPluginJobCompletionDoesNotCancelButPolicyStopPersistsCancellation)$' -count=1
- name: Cindy manual balance, split, image, and turn-state tests
- name: Account identity, routing leases, balance, image, and turn-state tests
working-directory: backend
run: |
go test -p=1 -tags unit ./internal/repository ./internal/service ./internal/handler ./internal/handler/admin -run 'Test.*(AccountCredentialIdentity|AccountJobCindyMutation|ProviderIdentityMismatch|CindyBalanceProbe|CindyGroupSplit|CindyCatalog|CindyCodexModels|ImageStudioEligibleKeys|CindyCompatibility|LegacyCindyRuntimeCompatibility|CodexImageGenerationBridgeOverridePrecedence|TurnStateCommits|SchedulerCache|SchedulerRefresh|KeepsCindyRoutingIdentity|StrictCindyGroup|MaterializesCindyIdentity|DisabledOrdinaryMember|SkipsCindyAggregate|GatewayModelCapabilities|GatewayModels|WriteOpenAIModelsList|AccountHandlerGetAvailableModels|ForwardResponsesOllamaClampPreservesContinuationPolicy|AccountAvailableModels|FetchOpenAIAccountModels|SetAnthropicAPIKeyAuthHeader|OllamaCloud.*(MaxTokens|MaxOutputTokens|Bearer)|Clamps.*OllamaBase)' -count=1
go test -p=1 -tags unit ./internal/repository ./internal/service ./internal/handler ./internal/handler/admin -run 'Test.*(AccountCredentialIdentity|AccountJobCindyMutation|ProviderIdentityMismatch|CindyBalanceProbe|CindyGroupSplit|CindyCatalog|CindyCodexModels|ImageStudioEligibleKeys|CindyCompatibility|LegacyCindyRuntimeCompatibility|CodexImageGenerationBridgeOverridePrecedence|TurnStateCommits|LegacyCodexRoutingImport|CodexConnectionLease|SchedulerCache|SchedulerRefresh|KeepsCindyRoutingIdentity|StrictCindyGroup|MaterializesCindyIdentity|DisabledOrdinaryMember|SkipsCindyAggregate|GatewayModelCapabilities|GatewayModels|WriteOpenAIModelsList|AccountHandlerGetAvailableModels|ForwardResponsesOllamaClampPreservesContinuationPolicy|AccountAvailableModels|FetchOpenAIAccountModels|SetAnthropicAPIKeyAuthHeader|OllamaCloud.*(MaxTokens|MaxOutputTokens|Bearer)|Clamps.*OllamaBase)' -count=1
go test -p=1 -tags unit ./migrations -run 'Test.*(Migration222|Migration223|Migration224|Migration230|Migration231|Migration232|Migration235|Migration236|GroupModelAllowlistRepairMigration|ReleaseAcceptance)' -count=1
- name: Request body recovery tests
working-directory: backend
Expand Down
23 changes: 12 additions & 11 deletions backend/internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -1028,8 +1028,8 @@ type GatewayConfig struct {
// OpenAICompactModel: /responses/compact 上游使用的模型。
// compact 端点支持模型滞后于普通 /responses 时,可用该配置降级规避上游错误。
OpenAICompactModel string `mapstructure:"openai_compact_model"`
// OpenAICodexTicket: ChatGPT OAuth 账号按 (账号, 模型) 捕获 292 长度
// x-codex-turn-state,并在住宅 IP 业务请求中注入该头。默认关闭。
// OpenAICodexTicket: Codex 路由插件的旧配置兼容入口。路由材料须按
// 所选账号、模型与业务出口验证,不使用 STATE 长度判定资格。默认关闭。
OpenAICodexTicket OpenAICodexTicketConfig `mapstructure:"openai_codex_ticket"`
// OpenAIWS: OpenAI Responses WebSocket 配置(默认开启,可按需回滚到 HTTP)
OpenAIWS GatewayOpenAIWSConfig `mapstructure:"openai_ws"`
Expand Down Expand Up @@ -1248,14 +1248,13 @@ func (c *UserMessageQueueConfig) GetEffectiveMode() string {
return ""
}

// OpenAICodexTicketConfig 控制 ChatGPT OAuth 的 x-codex-turn-state 门票。
// 打票走 harvest_proxy_url(SOCKS),业务出站仍用账号住宅 proxy_id,只替换该请求头。
// 门票默认有效 3600 秒,临近过期前 refresh_before_seconds 重新打票。
// OpenAICodexTicketConfig retains the legacy bootstrap configuration. Runtime
// route validation and expiry are owned by the Codex routing plugin.
type OpenAICodexTicketConfig struct {
Enabled bool `mapstructure:"enabled"`
TargetLength int `mapstructure:"target_length"`
TTLSeconds int `mapstructure:"ttl_seconds"`
RefreshBeforeSeconds int `mapstructure:"refresh_before_seconds"`
TargetLength int `mapstructure:"target_length"` // Deprecated: accepted for old files, ignored by routing.
TTLSeconds int `mapstructure:"ttl_seconds"` // Deprecated: not a routing-material lifetime.
RefreshBeforeSeconds int `mapstructure:"refresh_before_seconds"` // Deprecated: ignored by routing.
HarvestProxyURL string `mapstructure:"harvest_proxy_url"`
HarvestProbeIntervalSeconds int `mapstructure:"harvest_probe_interval_seconds"`
HarvestAttemptTimeoutSeconds int `mapstructure:"harvest_attempt_timeout_seconds"`
Expand Down Expand Up @@ -2436,9 +2435,6 @@ func setDefaults() {
viper.SetDefault("gateway.openai_passthrough_allow_timeout_headers", false)
viper.SetDefault("gateway.openai_compact_model", "gpt-5.5")
viper.SetDefault("gateway.openai_codex_ticket.enabled", false)
viper.SetDefault("gateway.openai_codex_ticket.target_length", 292)
viper.SetDefault("gateway.openai_codex_ticket.ttl_seconds", 3600)
viper.SetDefault("gateway.openai_codex_ticket.refresh_before_seconds", 60)
viper.SetDefault("gateway.openai_codex_ticket.harvest_proxy_url", "")
viper.SetDefault("gateway.openai_codex_ticket.harvest_probe_interval_seconds", 6)
viper.SetDefault("gateway.openai_codex_ticket.harvest_attempt_timeout_seconds", 25)
Expand Down Expand Up @@ -2656,6 +2652,11 @@ func setEnvReachableDefaults() {
viper.SetDefault("gateway.user_message_queue.mode", "")
viper.SetDefault("update.proxy_url", "")

// Retired Codex bootstrap fields are accepted for compatibility and ignored by routing.
viper.SetDefault("gateway.openai_codex_ticket.target_length", 0)
viper.SetDefault("gateway.openai_codex_ticket.ttl_seconds", 0)
viper.SetDefault("gateway.openai_codex_ticket.refresh_before_seconds", 0)

// sticky_escape_enabled is the one exception to the zero-value rule: its
// effective default is true, applied post-unmarshal via a viper.IsSet guard.
// Registering false would make IsSet always report true and permanently
Expand Down
2 changes: 1 addition & 1 deletion backend/internal/repository/account_repo.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ var schedulerNeutralExtraKeyPrefixes = []string{
"codex_5h_",
"codex_7d_",
"codex_reset_credit_",
// 292 门票是纯运行态凭据:它不在 filterSchedulerExtra 的投影白名单里,
// 历史 STATE 材料是运行态凭据:它不在 filterSchedulerExtra 的投影白名单里,
// 因此 bucket 重建事件永远搬不动门票状态,续期时开事务+发 outbox 是白干。
// 归为观测型后仍会同步单账号快照(见 UpdateExtra),不丢任何新鲜度。
"codex_turn_ticket:",
Expand Down
15 changes: 0 additions & 15 deletions backend/internal/repository/account_repo_codex_lifecycle.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,18 +34,3 @@ func scanTicketState(row interface{ Scan(...any) error }) (*service.CodexTicketL
}
return s, nil
}

func ticketFromAccount(a *service.Account, model string) *service.CodexTicketRecord {
if a == nil {
return nil
}
raw, err := json.Marshal(a.Extra["codex_turn_ticket:"+model])
if err != nil {
return nil
}
var t service.CodexTicketRecord
if json.Unmarshal(raw, &t) != nil || t.State == "" {
return nil
}
return &t
}
42 changes: 40 additions & 2 deletions backend/internal/repository/codex_connection_lease.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,31 @@ type codexConnectionLease struct {
expiresAt time.Time
client *http.Client
transport *http.Transport
alive *atomic.Bool
}

type codexLeasedConn struct {
net.Conn
alive *atomic.Bool
}

func (c *codexLeasedConn) Close() error {
c.alive.Store(false)
return c.Conn.Close()
}

func (s *httpUpstreamService) CheckCodexConnectionLease(accountID int64, scope, leaseID string, expiresAt time.Time) error {
s.mu.RLock()
lease := s.codexConnectionLeases[leaseID]
s.mu.RUnlock()
now := time.Now()
if lease == nil || lease.alive == nil || !lease.alive.Load() || !now.Before(lease.expiresAt) || !now.Before(expiresAt) {
return service.ErrCodexConnectionLeaseExpired
}
if accountID <= 0 || lease.accountID != accountID || scope == "" || lease.scope != scope || expiresAt.After(lease.expiresAt) {
return service.ErrCodexConnectionLeaseScope
}
return nil
}

func (s *httpUpstreamService) DoWithCodexConnectionLease(req *http.Request, proxyURL string, accountID int64, scope string, leaseID string, expiresAt time.Time, profile *tlsfingerprint.Profile) (*http.Response, string, error) {
Expand Down Expand Up @@ -57,6 +82,13 @@ func (s *httpUpstreamService) DoWithCodexConnectionLease(req *http.Request, prox
if lease.accountID != accountID || lease.scope != scope || lease.origin != origin || lease.transportHash != transportHash {
return nil, "", service.ErrCodexConnectionLeaseScope
}
checkDeadline := expiresAt
if checkDeadline.IsZero() {
checkDeadline = lease.expiresAt
}
if err := s.CheckCodexConnectionLease(accountID, scope, leaseID, checkDeadline); err != nil {
return nil, "", err
}
} else {
if expiresAt.IsZero() || expiresAt.After(now.Add(120*time.Second)) {
expiresAt = now.Add(120 * time.Second)
Expand All @@ -80,12 +112,18 @@ func (s *httpUpstreamService) DoWithCodexConnectionLease(req *http.Request, prox
// Exactly one physical dial is authorized. A disconnected/expired lease
// must fail before any request can reach a different residential exit.
var dialed atomic.Bool
alive := &atomic.Bool{}
wrap := func(dial func(context.Context, string, string) (net.Conn, error)) func(context.Context, string, string) (net.Conn, error) {
return func(ctx context.Context, network, address string) (net.Conn, error) {
if !time.Now().Before(expiresAt) || !dialed.CompareAndSwap(false, true) {
return nil, service.ErrCodexConnectionLeaseExpired
}
return dial(ctx, network, address)
connection, err := dial(ctx, network, address)
if err != nil {
return nil, err
}
alive.Store(true)
return &codexLeasedConn{Conn: connection, alive: alive}, nil
}
}
if transport.DialContext != nil {
Expand All @@ -101,7 +139,7 @@ func (s *httpUpstreamService) DoWithCodexConnectionLease(req *http.Request, prox
}
leaseID = hex.EncodeToString(id[:])
lease = &codexConnectionLease{accountID: accountID, scope: scope, origin: origin, transportHash: transportHash,
expiresAt: expiresAt, transport: transport,
expiresAt: expiresAt, transport: transport, alive: alive,
client: &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}}
s.mu.Lock()
if s.codexConnectionLeases == nil {
Expand Down
46 changes: 46 additions & 0 deletions backend/internal/repository/codex_connection_lease_check_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package repository

import (
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"

"github.com/Wei-Shaw/sub2api/internal/service"
"github.com/stretchr/testify/require"
)

func TestCodexConnectionLeaseLocalCheckDoesNotDial(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
calls.Add(1)
_, _ = io.WriteString(w, "complete")
}))
t.Cleanup(server.Close)
upstream, ok := NewHTTPUpstream(nil).(*httpUpstreamService)
require.True(t, ok)
request, err := http.NewRequestWithContext(t.Context(), http.MethodGet, server.URL, nil)
require.NoError(t, err)
deadline := time.Now().Add(time.Minute)
response, id, err := upstream.DoWithCodexConnectionLease(request, "", 71, "fixture-scope", "", deadline, nil)
require.NoError(t, err)
_, err = io.Copy(io.Discard, response.Body)
require.NoError(t, err)
require.NoError(t, response.Body.Close())
require.NoError(t, upstream.CheckCodexConnectionLease(71, "fixture-scope", id, deadline))
require.ErrorIs(t, upstream.CheckCodexConnectionLease(72, "fixture-scope", id, deadline), service.ErrCodexConnectionLeaseScope)
require.ErrorIs(t, upstream.CheckCodexConnectionLease(71, "different-scope", id, deadline), service.ErrCodexConnectionLeaseScope)
require.ErrorIs(t, upstream.CheckCodexConnectionLease(71, "fixture-scope", id, deadline.Add(time.Second)), service.ErrCodexConnectionLeaseScope)
restarted, ok := NewHTTPUpstream(nil).(*httpUpstreamService)
require.True(t, ok)
require.ErrorIs(t, restarted.CheckCodexConnectionLease(71, "fixture-scope", id, deadline), service.ErrCodexConnectionLeaseExpired)
upstream.mu.RLock()
lease := upstream.codexConnectionLeases[id]
upstream.mu.RUnlock()
lease.transport.CloseIdleConnections()
require.ErrorIs(t, upstream.CheckCodexConnectionLease(71, "fixture-scope", id, deadline), service.ErrCodexConnectionLeaseExpired)
require.EqualValues(t, 1, calls.Load())
upstream.removeCodexConnectionLease(id, lease)
}
Loading
Loading