diff --git a/.downstream/gateway-borrow-notices.md b/.downstream/gateway-borrow-notices.md index 0147f7b12bd0..efc5f3e8854a 100644 --- a/.downstream/gateway-borrow-notices.md +++ b/.downstream/gateway-borrow-notices.md @@ -11,6 +11,18 @@ commit `fd1b5ee4eeb20961fbb783fa6f136a1704271e90`; retained host adaptations are listed in [the scoped review](upstream-review-ranxi-v2.10.3.md). The repository [LICENSE](../LICENSE) retains the LGPL-3.0 terms. +Optional HTTP node selection additionally adapts `astra_rotation.go` and +`astra_rotation_cache.go` from v2.10.5, commit +`d977e7a5e24e5536db6ce8b8478743e0c4b11f77`. The bounded node cursor, one-hour +default target/node/gateway cooling, one-minute source cooling, duplicate-window +rule and five-minute repeated-host search pause come from that implementation. +The host adapter uses explicitly selected registered proxies instead of a managed +Mihomo listener. Sol and Astra require separate target proofs and cooldowns. +Interrupted target probes retain the host's v16 no-negative-cache behavior; +source failures still share source/node cooling across models. Business STATE +replacement and incomplete SSE invalidate future selection without replaying +the generation or changing the upstream response. + HTML fence selection also references `web/src/lib/render.ts` from [iwyxdxl/gpttesticu](https://github.com/iwyxdxl/gpttesticu), commit `1b3e237c15dcdf89c4aa66d3521f8d6485d6da4a`. Its MIT notice follows. diff --git a/.downstream/native-domains.md b/.downstream/native-domains.md index c4b296081f18..4228f5bf524c 100644 --- a/.downstream/native-domains.md +++ b/.downstream/native-domains.md @@ -89,13 +89,17 @@ countdowns expire routes without renewing them. `GET/PUT /admin/codex-gateway-borrow/config` owns one JSON setting, `codex_gateway_borrow_config`: `enabled`, `source_account_ids`, -`target_account_ids` and `models`. A missing row is off with empty account +`target_account_ids` and `models`. Optional `rotate_nodes`, `node_proxy_ids`, +`max_node_attempts` and `node_cooldown_seconds` enable bounded HTTP node selection +only for those configured targets. Proxy ID `0` explicitly adds the current +server's direct egress as a candidate; it is never an implicit fallback. +A missing row is off with empty account lists and the two models `gpt-6-astra` and `gpt-6.1-sol`. Only existing OpenAI OAuth-like accounts can be selected; the source and target lists are disjoint. The extension never rewrites account model mappings, proxies, WS switches, groups, quota or scheduling state. Normal credential refresh remains available. -The protocol reference is ranxi v2.10.5 (`d977e7a5`), non-rotating branch. +The protocol reference is ranxi v2.10.5 (`d977e7a5`). With node selection off, Candidates are stored per source with an active-source preference. Their maximum lease is 230 seconds, capped by earlier upstream expiry; seeing a value again while its lease is live does not extend it. Source acquisition uses the normal @@ -105,6 +109,64 @@ transport configuration and no business service tier inserted into their body. The actual selected model is preserved: Astra follows the original; Sol is a separate downstream model extension with independent qualification. +With node selection enabled, the source, target probe and business request use +the same immutable registered proxy definition and fresh-connect transport. +Account bindings and shared proxy records are not changed. Inactive or expired +nodes are skipped; a changed proxy definition invalidates prepared dispatch. +Each preparation has a 180-second bound and tries at most the configured number +of distinct eligible nodes (default 3, maximum 10), with 45 seconds per source +acquisition. Explicit target rejection cools both the node and the named gateway +for that account/model (default 3600 seconds). Source failures cool the donor/node +for one minute across models. Duplicate observations cannot extend a cooling +window. Discovering a known cooling gateway pauses that model's source search +for five minutes; an exhausted search has a 15-second pause. A target transport +interruption or missing terminal does not poison the next independent request: +it still requires a fresh full probe pair. No business generation is replayed. +Cooling decisions survive setting revisions but are process-local, as upstream. +Optional `wait_for_qualified_route` defaults to false and requires node selection. +Optional `immediate_reborrow_on_failure` also defaults to false and requires both +waiting and node selection. Only pending HTTP business requests use its zero +cooldown: failed qualifications trigger fresh source acquisition and two complete +strict probes, bypassing preparation/source/node/gateway failure caches. Each +180-second bounded search starts its next round immediately, without a total +recovery deadline. Reacquiring the same cookie cannot extend an active lease; +an expired proof requires new acquisition and complete qualification as upstream. +Administrator diagnostics retain ordinary +cooling and remain bounded. Business dispatch still requires a current successful +proof and is never replayed. Missing eligible nodes or invalid configuration stop +recovery rather than spinning without network work. Acquisition and validation +phases remain separate from a recovered proof and a completed business response. +Saving immediate mode does not start background preparation; acquisition only +runs while a pending business request needs it or an administrator explicitly +invokes the existing bounded preparation/verification interface. +For full HTTP Responses business preparation it retains the original request while +eligible routes recover; administrator probes and Pelican jobs remain bounded. +Each search remains bounded to 180 seconds and the configured distinct-node limit. +No total recovery deadline is added while the caller is connected. Without +immediate reborrow the next cycle waits for node/source/gateway cooling; failures without a future eligibility time +wait at least 15 seconds. Cancellation of the last waiter cancels shared work; +configuration, target identity or account invalidation stops recovery. +Recovery first reacquires and requalifies an expired lease on the previously +qualified node; it changes nodes when that route fails or is unavailable. A +candidate that rediscovers a cooling gateway is skipped while other eligible +nodes remain in the bounded round. The gateway/node cooldown remains intact; +the upstream five-minute blanket search pause remains for non-waiting mode. +Normal token refresh +may update authentication, never business STATE, input, model, tier or history. +Recovery emits no downstream bytes before actual business response headers, so +the real STATE header remains deliverable. Existing proxy/client timeouts still +apply. Recovery status, rounds and initiated source/probe counts are distinct +from dispatched/completed business counters; shared preparations are labelled. +Source Cookie leases retain the 230-second cap and same-value no-extension rule +across selected nodes. Every model has its own target proof and current route. +The status API reports actual proxy IDs, gateway metadata, expiry and cooling; +selection is not a claim of long-term availability or model quality. Full +Responses business STATE replacement and incomplete SSE invalidate future +selection while preserving the response. Compact/lite JSON keeps its distinct +terminal/STATE protocol. Node selection does not enable WS and fails closed if +a selected target attempts borrowed WS; ordinary and unselected paths retain +their existing behavior. + Source acquisition has a 90-second bound. The two target probes have 45 seconds each and require completed HTTP 200 streams. The first must return STATE; the second may omit it or return the same value. Probes clone the business headers, diff --git a/.github/workflows/downstream-verify.yml b/.github/workflows/downstream-verify.yml index 7514d3190bc6..2b04fd054199 100644 --- a/.github/workflows/downstream-verify.yml +++ b/.github/workflows/downstream-verify.yml @@ -50,7 +50,7 @@ jobs: go test -p=1 -tags integration ./internal/repository -run '^TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle$' -count=1 - name: Codex preparation coordination race tests working-directory: backend - run: go test -p=1 -race -tags unit ./internal/service -run '^TestCodexBorrow(Flight|DifferentTargets|CancelledParent|DiagnoseHTTPTerminalAndDispatchEvidence)|^TestCodexGatewayBorrowRunnerCancellationStopsQueuedAndInflightAndRetainsRawOutput$' -count=1 + run: go test -p=1 -race -tags unit ./internal/service -run '^TestCodexBorrow(Immediate|Recovery|Node|Flight|DifferentTargets|CancelledParent|DiagnoseHTTPTerminalAndDispatchEvidence)|^TestCodexGatewayBorrowRunnerCancellationStopsQueuedAndInflightAndRetainsRawOutput$' -count=1 - name: Account test keepalive lifecycle race tests working-directory: backend run: go test -p=1 -race -tags unit ./internal/service -run '^TestAccountTestKeepalive' -count=1 diff --git a/backend/cmd/server/wire_gen.go b/backend/cmd/server/wire_gen.go index 98ba69699e45..92c3fe347f6a 100644 --- a/backend/cmd/server/wire_gen.go +++ b/backend/cmd/server/wire_gen.go @@ -300,7 +300,7 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { auditLogHandler := admin.NewAuditLogHandler(auditLogService, totpService) systemPromptHandler := admin.NewSystemPromptHandler(systemPromptService) codexGatewayBorrowProbeUpstream := repository.NewCodexGatewayBorrowProbeUpstream(httpUpstream) - codexGatewayBorrowService, err := service.ProvideCodexGatewayBorrowService(settingRepository, accountRepository, openAIGatewayService, codexGatewayBorrowProbeUpstream, tlsFingerprintProfileService) + codexGatewayBorrowService, err := service.ProvideCodexGatewayBorrowService(settingRepository, accountRepository, openAIGatewayService, codexGatewayBorrowProbeUpstream, tlsFingerprintProfileService, proxyRepository) if err != nil { return nil, err } diff --git a/backend/internal/handler/ops_codex_borrow_test.go b/backend/internal/handler/ops_codex_borrow_test.go index 49b9c96355d0..f7f457233954 100644 --- a/backend/internal/handler/ops_codex_borrow_test.go +++ b/backend/internal/handler/ops_codex_borrow_test.go @@ -27,3 +27,41 @@ func TestOpsCodexBorrowPreparationIsNotAnUpstream503(t *testing.T) { require.Equal(t, "target_state_changed", entry.UpstreamErrors[0].Reason) require.Contains(t, *entry.UpstreamErrorDetail, "full diagnostic detail") } + +func TestOpsCodexBorrowPreparationUsesSelectedNode(t *testing.T) { + for _, tc := range []struct { + name string + stage string + verification *service.CodexGatewayBorrowVerification + proxyID int64 + proxyName string + }{ + {"legacy binding", "target_validation", nil, 47, "saved binding"}, + {"exhausted pool retains last probe", "node_selection", &service.CodexGatewayBorrowVerification{ProxyID: 43, ProxyName: "selected node", NodeIdentity: "selected-identity"}, 43, "selected node"}, + {"explicit direct", "target_validation", &service.CodexGatewayBorrowVerification{ProxyName: "Server direct", NodeIdentity: "direct-identity"}, 0, "direct/no_proxy"}, + {"no selected node", "node_selection", nil, 0, "unknown"}, + } { + t.Run(tc.name, func(t *testing.T) { + c, _ := gin.CreateTestContext(httptest.NewRecorder()) + c.Request = httptest.NewRequest("POST", "/v1/responses", nil) + binding := int64(47) + account := &service.Account{ID: 2, Platform: "openai", ProxyID: &binding, Proxy: &service.Proxy{ID: 47, Name: "saved binding"}} + failure := &service.CodexGatewayBorrowFailure{Cause: service.ErrCodexGatewayBorrowUnavailable, Stage: tc.stage, Reason: "node_candidates_exhausted", Verification: tc.verification} + service.RecordCodexGatewayBorrowPreparationFailure(c, account, failure) + entry := &service.OpsInsertErrorLogInput{} + applyOpsUpstreamFieldsFromContext(c, entry) + require.Len(t, entry.UpstreamErrors, 1) + event := entry.UpstreamErrors[0] + if tc.proxyID == 0 { + require.Nil(t, event.ProxyID) + } else { + require.NotNil(t, event.ProxyID) + require.Equal(t, tc.proxyID, *event.ProxyID) + } + require.Equal(t, tc.proxyName, event.ProxyName) + require.NotNil(t, entry.UpstreamStatusCode) + require.Zero(t, *entry.UpstreamStatusCode, "local qualification failure is not a business upstream 503") + require.Equal(t, int64(47), *account.ProxyID, "diagnostics cannot rewrite account bindings") + }) + } +} diff --git a/backend/internal/service/codex_gateway_borrow.go b/backend/internal/service/codex_gateway_borrow.go index fac15dee56b5..302224a24716 100644 --- a/backend/internal/service/codex_gateway_borrow.go +++ b/backend/internal/service/codex_gateway_borrow.go @@ -4,7 +4,8 @@ package service // from ranxi2001/sub2api at 5ca3cca21eeaf4ca8a694a7f2f8f0ecd9575c549: // backend/internal/repository/{codex_gateway_pin,astra_target_validation}.go // and backend/internal/service/openai_codex_state_probe.go (LGPL-3.0). -// This native implementation owns no exit routing, account health or quota. +// Optional node selection changes only scoped outgoing request routes; account +// bindings, account health and quota remain owned by the host. import ( "bytes" @@ -115,10 +116,16 @@ func codexBorrowFailureStage(err error) (string, string) { type CodexGatewayBorrowProbeUpstream interface{ HTTPUpstream } type CodexGatewayBorrowConfig struct { - Enabled bool `json:"enabled"` - SourceAccountIDs []int64 `json:"source_account_ids"` - TargetAccountIDs []int64 `json:"target_account_ids"` - Models []string `json:"models"` + ImmediateReborrowOnFailure bool `json:"immediate_reborrow_on_failure,omitempty"` + WaitForQualifiedRoute bool `json:"wait_for_qualified_route,omitempty"` + RotateNodes bool `json:"rotate_nodes,omitempty"` + NodeProxyIDs []int64 `json:"node_proxy_ids,omitempty"` + MaxNodeAttempts int `json:"max_node_attempts,omitempty"` + NodeCooldownSeconds int `json:"node_cooldown_seconds,omitempty"` + Enabled bool `json:"enabled"` + SourceAccountIDs []int64 `json:"source_account_ids"` + TargetAccountIDs []int64 `json:"target_account_ids"` + Models []string `json:"models"` } func DefaultCodexGatewayBorrowConfig() CodexGatewayBorrowConfig { @@ -143,6 +150,9 @@ func DecodeCodexGatewayBorrowConfig(raw []byte) (CodexGatewayBorrowConfig, error } func normalizeCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) (CodexGatewayBorrowConfig, error) { + if err := validateCodexBorrowNodeConfig(cfg); err != nil { + return cfg, err + } if cfg.SourceAccountIDs == nil { cfg.SourceAccountIDs = []int64{} } @@ -175,6 +185,7 @@ func normalizeCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) (CodexGatew } func cloneCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) CodexGatewayBorrowConfig { + cfg.NodeProxyIDs = append([]int64(nil), cfg.NodeProxyIDs...) cfg.SourceAccountIDs = append([]int64{}, cfg.SourceAccountIDs...) cfg.TargetAccountIDs = append([]int64{}, cfg.TargetAccountIDs...) cfg.Models = append([]string{}, cfg.Models...) @@ -182,6 +193,9 @@ func cloneCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) CodexGatewayBor } type CodexGatewayBorrowCandidateStatus struct { + NodeIdentity string `json:"node_identity,omitempty"` + ProxyID int64 `json:"proxy_id,omitempty"` + Gateway string `json:"gateway,omitempty"` SourceRequestEncoding string `json:"source_request_encoding,omitempty"` SourceAccountID int64 `json:"source_account_id"` ExpiresAt time.Time `json:"expires_at"` @@ -190,6 +204,8 @@ type CodexGatewayBorrowCandidateStatus struct { } type CodexGatewayBorrowSourceStatus struct { + NodeIdentity string `json:"node_identity,omitempty"` + ProxyID int64 `json:"proxy_id,omitempty"` AccountID int64 `json:"account_id"` State string `json:"state"` Reason string `json:"reason"` @@ -200,6 +216,10 @@ type CodexGatewayBorrowSourceStatus struct { } type CodexGatewayBorrowVerification struct { + ProxyID int64 `json:"proxy_id,omitempty"` + ProxyName string `json:"proxy_name,omitempty"` + NodeIdentity string `json:"node_identity,omitempty"` + Gateway string `json:"gateway,omitempty"` RetryOnNextRequest bool `json:"retry_on_next_request,omitempty"` FailedShot string `json:"failed_shot,omitempty"` FailureKind string `json:"failure_kind,omitempty"` @@ -240,6 +260,7 @@ type CodexGatewayBorrowTargetStatus struct { } type CodexGatewayBorrowStatus struct { + Nodes []CodexGatewayBorrowNodeStatus `json:"nodes,omitempty"` Acquisition CodexGatewayBorrowAcquisition `json:"acquisition"` ObservedSince time.Time `json:"observed_since"` RecentUsage []CodexGatewayBorrowUsage `json:"recent_usage"` @@ -258,6 +279,7 @@ type CodexGatewayBorrowStatus struct { // Application contains only opaque identities. The selected cookie itself is // available solely in the cloned outbound request's Cookie header. type CodexGatewayBorrowApplication struct { + node *codexBorrowNodeBinding Verification CodexGatewayBorrowVerification Applied bool SourceAccountID int64 @@ -267,6 +289,7 @@ type CodexGatewayBorrowApplication struct { } type codexGatewayBorrowCandidate struct { + node *codexBorrowNode sourceRequestEncoding string cookie http.Cookie expires time.Time @@ -291,6 +314,11 @@ type codexGatewayBorrowTargetCheck struct { } type CodexGatewayBorrowService struct { + proxies ProxyRepository + node *codexBorrowNode + nodeLease func(string, time.Time) time.Time + rotation codexBorrowRotation + rotationCandidates map[codexGatewayBorrowTargetKey]*codexGatewayBorrowCandidate validationSequence uint64 sourceSequence uint64 qualifications map[string]codexGatewayBorrowTargetCheck @@ -337,8 +365,9 @@ func NewCodexGatewayBorrowService(settings SettingRepository, accounts AccountRe return s } -func ProvideCodexGatewayBorrowService(settings SettingRepository, accounts AccountRepository, gateway *OpenAIGatewayService, upstream CodexGatewayBorrowProbeUpstream, tlsProfiles *TLSFingerprintProfileService) (*CodexGatewayBorrowService, error) { +func ProvideCodexGatewayBorrowService(settings SettingRepository, accounts AccountRepository, gateway *OpenAIGatewayService, upstream CodexGatewayBorrowProbeUpstream, tlsProfiles *TLSFingerprintProfileService, proxies ProxyRepository) (*CodexGatewayBorrowService, error) { s := NewCodexGatewayBorrowService(settings, accounts, gateway, upstream, tlsProfiles) + s.proxies = proxies ctx, cancel := context.WithTimeout(context.Background(), codexGatewayBorrowStoreTimeout) defer cancel() if _, err := s.GetConfig(ctx); err != nil { @@ -391,6 +420,11 @@ func (s *CodexGatewayBorrowService) SaveConfig(ctx context.Context, cfg CodexGat if err != nil { return cfg, err } + if cfg.RotateNodes { + if _, err := s.borrowNodes(ctx, cfg.NodeProxyIDs, true); err != nil { + return cfg, err + } + } // Validation only loads local rows and resolves existing parent credentials. // It never discovers models, touches quota or changes any account fields. for _, id := range append(append([]int64{}, cfg.SourceAccountIDs...), cfg.TargetAccountIDs...) { @@ -419,7 +453,7 @@ func (s *CodexGatewayBorrowService) SaveConfig(ctx context.Context, cfg CodexGat func (s *CodexGatewayBorrowService) publishConfig(cfg CodexGatewayBorrowConfig, saved bool) { s.mu.Lock() - changed := !s.loaded || !slices.Equal(cfg.SourceAccountIDs, s.config.SourceAccountIDs) || !slices.Equal(cfg.TargetAccountIDs, s.config.TargetAccountIDs) || !slices.Equal(cfg.Models, s.config.Models) || cfg.Enabled != s.config.Enabled + changed := !s.loaded || !slices.Equal(cfg.SourceAccountIDs, s.config.SourceAccountIDs) || !slices.Equal(cfg.TargetAccountIDs, s.config.TargetAccountIDs) || !slices.Equal(cfg.Models, s.config.Models) || cfg.Enabled != s.config.Enabled || !borrowNodeConfigEqual(cfg, s.config) if !changed && !saved { s.mu.Unlock() return @@ -432,7 +466,7 @@ func (s *CodexGatewayBorrowService) publishConfig(cfg CodexGatewayBorrowConfig, s.candidates = nil s.acquisition = CodexGatewayBorrowAcquisition{} s.setup = CodexGatewayBorrowSetup{} - if saved && cfg.Enabled { + if saved && cfg.Enabled && !cfg.ImmediateReborrowOnFailure { s.setup = CodexGatewayBorrowSetup{State: "queued", Total: len(cfg.TargetAccountIDs) * len(cfg.Models), StartedAt: time.Now()} } s.usage = nil @@ -440,12 +474,13 @@ func (s *CodexGatewayBorrowService) publishConfig(cfg CodexGatewayBorrowConfig, s.sources = map[int64]CodexGatewayBorrowSourceStatus{} s.targets = map[codexGatewayBorrowTargetKey]codexGatewayBorrowTargetCheck{} s.qualifications = map[string]codexGatewayBorrowTargetCheck{} + s.rotationCandidates = nil s.prepareFailedUntil, s.prepareFailure, s.preparing, s.prepareRuns = time.Time{}, "", false, 0 ctx := s.revisionCtx s.mu.Unlock() // Exactly one finite preparation per save. Loading settings or restarting // never calls a model. Individual source/target deadlines remain 90 seconds. - if saved && cfg.Enabled { + if saved && cfg.Enabled && !cfg.ImmediateReborrowOnFailure { go func() { prepareCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) defer cancel() @@ -459,8 +494,9 @@ func (s *CodexGatewayBorrowService) Stop() { return } s.mu.Lock() - defer s.mu.Unlock() s.cancel() + s.mu.Unlock() + s.rotation.stop() } func codexGatewayBorrowAccountSupported(a *Account) bool { @@ -510,13 +546,24 @@ func (s *CodexGatewayBorrowService) Status() CodexGatewayBorrowStatus { return CodexGatewayBorrowStatus{Config: DefaultCodexGatewayBorrowConfig(), Sources: []CodexGatewayBorrowSourceStatus{}, Targets: []CodexGatewayBorrowTargetStatus{}, ModelEfforts: codexGatewayBorrowModelEfforts()} } s.mu.Lock() + snapshotConfig, snapshotRevision := cloneCodexGatewayBorrowConfig(s.config), s.revision + s.mu.Unlock() + nodes := s.rotation.snapshot(snapshotConfig, snapshotRevision) + s.mu.Lock() defer s.mu.Unlock() + if s.revision != snapshotRevision { + nodes = nil + } now := time.Now() s.currentCandidateLocked() - out := CodexGatewayBorrowStatus{Enabled: s.config.Enabled, Revision: s.revision, GeneratedAt: now, Preparing: s.preparing || s.prepareRuns > 0 || s.setup.State == "queued", + out := CodexGatewayBorrowStatus{Nodes: nodes, Enabled: s.config.Enabled, Revision: s.revision, GeneratedAt: now, Preparing: s.preparing || s.prepareRuns > 0 || s.setup.State == "queued", Acquisition: s.acquisition, ObservedSince: s.observedSince, RecentUsage: []CodexGatewayBorrowUsage{}, Setup: s.setup, Config: cloneCodexGatewayBorrowConfig(s.config), Sources: []CodexGatewayBorrowSourceStatus{}, Targets: []CodexGatewayBorrowTargetStatus{}, ModelEfforts: codexGatewayBorrowModelEfforts()} if c := s.candidate; c != nil { out.Candidate = &CodexGatewayBorrowCandidateStatus{SourceAccountID: c.sourceID, SourceRequestEncoding: c.sourceRequestEncoding, ExpiresAt: c.expires, RemainingSeconds: borrowRemaining(c.expires, now), CookieFingerprint: borrowHash(c.cookie.Value)} + if c.node != nil { + out.Candidate.ProxyID, out.Candidate.Gateway = c.node.id, codexBorrowGateway(c.cookie.Value) + out.Candidate.NodeIdentity = c.node.identity + } } for _, usage := range s.usage { out.RecentUsage = append(out.RecentUsage, usage) @@ -543,12 +590,16 @@ func (s *CodexGatewayBorrowService) Status() CodexGatewayBorrowStatus { for _, model := range s.config.Models { row := CodexGatewayBorrowTargetStatus{CodexGatewayBorrowVerification: CodexGatewayBorrowVerification{AccountID: id, Model: model, Reason: "not_verified"}, State: "waiting"} if c, ok := s.targets[codexGatewayBorrowTargetKey{id, model}]; ok { + candidate := s.candidate + if s.config.RotateNodes { + candidate = s.rotationCandidates[codexGatewayBorrowTargetKey{id, model}] + } row.CodexGatewayBorrowVerification = c.result row.Acquisition = c.acquisition - row.CacheValid = c.policyRevision == currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision && c.result.Success && s.candidate != nil && c.cookieKey == borrowHash(s.candidate.cookie.Value) && now.Before(c.expires) && now.Before(s.candidate.expires) + row.CacheValid = c.policyRevision == currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision && c.result.Success && candidate != nil && c.cookieKey == borrowHash(candidate.cookie.Value) && now.Before(c.expires) && now.Before(candidate.expires) expires := c.expires - if s.candidate != nil && c.cookieKey == borrowHash(s.candidate.cookie.Value) && s.candidate.expires.Before(expires) { - expires = s.candidate.expires + if candidate != nil && c.cookieKey == borrowHash(candidate.cookie.Value) && candidate.expires.Before(expires) { + expires = candidate.expires } if !expires.IsZero() { row.ExpiresAt = &expires @@ -564,7 +615,7 @@ func (s *CodexGatewayBorrowService) Status() CodexGatewayBorrowStatus { } else if c.result.Success { row.State, row.Reason = "expired", "route_expired" } - if c.result.Success && s.candidate != nil && c.cookieKey != borrowHash(s.candidate.cookie.Value) && now.Before(c.expires) { + if c.result.Success && candidate != nil && c.cookieKey != borrowHash(candidate.cookie.Value) && now.Before(c.expires) { row.State, row.Reason = "waiting", "target_route_changed" } if c.policyRevision != currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision && c.result.Success { @@ -575,6 +626,23 @@ func (s *CodexGatewayBorrowService) Status() CodexGatewayBorrowStatus { row.RetryAfter = &retry } } + if s.config.RotateNodes && row.NodeIdentity != "" { + // The child probe's short cache is not the node's quiet window. + // Report the selected node snapshot, including after lease expiry. + row.RetryAfter, row.Acquisition.RetryAfter = nil, nil + for _, node := range nodes { + if node.AccountID == id && node.Model == model && node.NodeIdentity == row.NodeIdentity && node.RetryAfter != nil && now.Before(*node.RetryAfter) { + if row.CacheValid { + // A late failure on another node may cool this gateway + // after the stored probe passed. Match dispatch eligibility. + row.CacheValid, row.State, row.Reason = false, "rejected", "node_cooling" + } + retry := *node.RetryAfter + row.RetryAfter, row.Acquisition.RetryAfter = &retry, &retry + break + } + } + } if !s.config.Enabled { row.State, row.Reason, row.CacheValid = "disabled", "disabled", false } @@ -653,9 +721,11 @@ func (s *CodexGatewayBorrowService) Prepare(ctx context.Context) (CodexGatewayBo } s.setSetup(rev, progress) } - if err := s.prepareSource(runCtx, rev, revisionCtx); err != nil { - finish(err) - return nil, err + if !cfg.RotateNodes { + if err := s.prepareSource(runCtx, rev, revisionCtx); err != nil { + finish(err) + return nil, err + } } var lastErr error for _, id := range cfg.TargetAccountIDs { @@ -699,16 +769,17 @@ func (s *CodexGatewayBorrowService) prepareSource(ctx context.Context, rev uint6 return ErrCodexGatewayBorrowUnavailable } trigger, _ := ctx.Value(borrowAcquisitionTriggerKey{}).(string) + fresh, _ := ctx.Value(borrowFreshSourceKey{}).(bool) if trigger == "" { trigger = "save_or_manual" } - _, err := s.sourceFlight.do(ctx, revisionCtx, fmt.Sprint(rev), codexGatewayBorrowTimeout, func(ctx context.Context) (any, error) { + _, err := s.sourceFlight.do(ctx, revisionCtx, fmt.Sprintf("%d:%t", rev, fresh), codexGatewayBorrowTimeout, func(ctx context.Context) (any, error) { s.mu.Lock() if s.revision != rev || !s.config.Enabled { s.mu.Unlock() return nil, ErrCodexGatewayBorrowChanged } - if c := s.currentCandidateLocked(); c != nil && time.Until(c.expires) > codexGatewayBorrowTimeout { + if c := s.currentCandidateLocked(); !fresh && c != nil && time.Until(c.expires) > codexGatewayBorrowTimeout { s.mu.Unlock() return nil, nil } @@ -716,7 +787,7 @@ func (s *CodexGatewayBorrowService) prepareSource(ctx context.Context, rev uint6 s.mu.Unlock() return nil, ErrCodexGatewayBorrowUnavailable } - if time.Now().Before(s.prepareFailedUntil) { + if !fresh && time.Now().Before(s.prepareFailedUntil) { detail, retry := s.prepareFailure, s.prepareFailedUntil s.mu.Unlock() return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "source_preparation", Reason: "source_prepare_failed", Detail: detail, RetryAfter: &retry} @@ -762,6 +833,10 @@ func (s *CodexGatewayBorrowService) prepareSource(ctx context.Context, rev uint6 if acquireErr == nil && candidate != nil && now.Before(candidate.expires) { s.storeCandidateLocked(candidate) row.State, row.Reason, row.Error, row.ExpiresAt = "ready", "source_qualified", "", &candidate.expires + if candidate.node != nil { + row.ProxyID = candidate.node.id + row.NodeIdentity = candidate.node.identity + } s.sources[id] = row s.acquisition.Phase = "candidate_acquired" s.prepareFailedUntil, s.prepareFailure = time.Time{}, "" @@ -777,7 +852,7 @@ func (s *CodexGatewayBorrowService) prepareSource(ctx context.Context, rev uint6 s.sources[id] = row // Original non-rotation preparation may still have a usable primary or // backup after a failed refresh. Preserve it without extending its lease. - if current := s.currentCandidateLocked(); current != nil && time.Now().Before(current.expires) { + if current := s.currentCandidateLocked(); !fresh && current != nil && time.Now().Before(current.expires) { s.acquisition.Phase, s.acquisition.Error = "candidate_retained", lastErr.Error() s.prepareFailedUntil, s.prepareFailure = time.Time{}, "" s.mu.Unlock() @@ -854,11 +929,15 @@ func (s *CodexGatewayBorrowService) apply(req *http.Request, account *Account, m return req, nil, nil } rev, revisionCtx := s.revision, s.revisionCtx + rotation := s.config.RotateNodes candidate := s.currentCandidateLocked() s.mu.Unlock() if !codexGatewayBorrowAccountSupported(account) { return req, nil, ErrCodexGatewayBorrowUnavailable } + if rotation { + return s.applyRotated(req, account, model, proxy, profile, cachedOnly, force, rev, revisionCtx) + } if candidate == nil || !time.Now().Before(candidate.expires) { if cachedOnly { return req, nil, ErrCodexGatewayBorrowUnavailable diff --git a/backend/internal/service/codex_gateway_borrow_candidates.go b/backend/internal/service/codex_gateway_borrow_candidates.go index 303b93ab6aa9..1161bfa9dc04 100644 --- a/backend/internal/service/codex_gateway_borrow_candidates.go +++ b/backend/internal/service/codex_gateway_borrow_candidates.go @@ -44,6 +44,11 @@ func (s *CodexGatewayBorrowService) storeCandidateLocked(candidate *codexGateway } func (s *CodexGatewayBorrowService) liveCookieLocked(key string) bool { + for _, c := range s.rotationCandidates { + if c != nil && borrowHash(c.cookie.Value) == key && time.Now().Before(c.expires) { + return true + } + } if c := s.candidate; c != nil && borrowHash(c.cookie.Value) == key && time.Now().Before(c.expires) { return true } diff --git a/backend/internal/service/codex_gateway_borrow_core_test.go b/backend/internal/service/codex_gateway_borrow_core_test.go index 57ba9555f114..cdd5e021bb47 100644 --- a/backend/internal/service/codex_gateway_borrow_core_test.go +++ b/backend/internal/service/codex_gateway_borrow_core_test.go @@ -120,7 +120,7 @@ func TestCodexGatewayBorrowConfigDefaultsAndLoadNeverProbe(t *testing.T) { return nil, errors.New("must not call upstream") }) store := &borrowCoreSettings{} - s, err := ProvideCodexGatewayBorrowService(store, nil, nil, probe, nil) + s, err := ProvideCodexGatewayBorrowService(store, nil, nil, probe, nil, nil) require.NoError(t, err) t.Cleanup(s.Stop) require.Equal(t, DefaultCodexGatewayBorrowConfig(), s.ConfigSnapshot()) diff --git a/backend/internal/service/codex_gateway_borrow_http.go b/backend/internal/service/codex_gateway_borrow_http.go index baee156f5ab1..33da924b84f7 100644 --- a/backend/internal/service/codex_gateway_borrow_http.go +++ b/backend/internal/service/codex_gateway_borrow_http.go @@ -67,9 +67,22 @@ func RecordCodexGatewayBorrowPreparationFailure(c *gin.Context, account *Account event.ProxyID, event.ProxyName = opsUpstreamProxyID(account), opsUpstreamProxyName(account) event.Platform, event.AccountID, event.AccountName = account.Platform, account.ID, account.Name } + if stage == "node_selection" { + // A saved account binding does not identify a node-selection attempt. + // Keep the route unknown unless the failed probe provides its snapshot. + event.ProxyID, event.ProxyName = nil, opsProxyNameUnknown + } var failure *CodexGatewayBorrowFailure if errors.As(err, &failure) && failure.Verification != nil { verification := *failure.Verification + if verification.NodeIdentity != "" { + event.ProxyID, event.ProxyName = nil, verification.ProxyName + if verification.ProxyID > 0 { + event.ProxyID = &verification.ProxyID + } else { + event.ProxyName = opsProxyNameDirect + } + } event.BorrowQualification = &CodexGatewayBorrowFailureEvidence{CacheHit: failure.CacheHit, RetryAfter: failure.RetryAfter, Verification: &verification} } appendOpsUpstreamError(c, event) @@ -111,15 +124,25 @@ func (s *OpenAIGatewayService) prepareCodexGatewayBorrowHTTP(ctx context.Context } started := time.Now() tracker := s.gatewayBorrow.beginAttempt(ctx, account.ID, model, "http") + ctx = context.WithValue(ctx, codexBorrowUsageContextKey{}, tracker) req = req.WithContext(context.WithValue(req.Context(), codexBorrowUsageContextKey{}, tracker)) prepared := codexGatewayBorrowHTTPPreparation{service: s.gatewayBorrow, accountID: account.ID, model: model, proxy: proxy} prepared.service.mu.Lock() prepared.revision = prepared.service.revision + revisionCtx := prepared.service.revisionCtx + wait := prepared.service.config.WaitForQualifiedRoute && req.URL.Path == "/backend-api/codex/responses" && borrowDiagnosticFromContext(ctx) == nil && !IsPelicanGeneration(ctx) prepared.service.mu.Unlock() if tier, ok := req.Context().Value(codexBorrowServiceTierContextKey{}).(string); ok { ctx = withCodexBorrowServiceTier(ctx, tier) } - borrowed, application, err := prepared.service.Apply(req.WithContext(ctx), account, model, proxy, nil, false) + var borrowed *http.Request + var application *CodexGatewayBorrowApplication + var err error + if wait { + borrowed, application, err = prepared.service.recoverQualifiedHTTP(ctx, req, account, model, proxy, prepared.revision, revisionCtx) + } else { + borrowed, application, err = prepared.service.Apply(req.WithContext(ctx), account, model, proxy, nil, false) + } if err != nil { tracker.blocked(err) return nil, 0, err @@ -130,7 +153,7 @@ func (s *OpenAIGatewayService) prepareCodexGatewayBorrowHTTP(ctx context.Context return nil, 0, failure } prepared.application = application - req = borrowed.WithContext(context.WithValue(req.Context(), codexGatewayBorrowHTTPPreparationContextKey{}, prepared)) + req = borrowed.WithContext(context.WithValue(borrowed.Context(), codexGatewayBorrowHTTPPreparationContextKey{}, prepared)) return req, time.Since(started), nil } @@ -190,6 +213,9 @@ func (s *OpenAIGatewayService) applyCodexGatewayBorrowHTTP(req *http.Request, ac // The preparation already canonicalized this Cookie header. Validation // must keep the bound plaintext request identity when it changes no bytes. if slices.Equal(req.Header.Values("Cookie"), borrowed.Header.Values("Cookie")) { + if application.node != nil && HTTPUpstreamProfileFromContext(req.Context()) != HTTPUpstreamProfileOpenAIHarvest { + return req.WithContext(WithHTTPUpstreamProfile(req.Context(), HTTPUpstreamProfileOpenAIHarvest)), nil + } return req, nil } return borrowed, nil @@ -200,7 +226,12 @@ type codexGatewayBorrowRecoveryContext struct { values context.Context } -func (c codexGatewayBorrowRecoveryContext) Value(key any) any { return c.values.Value(key) } +func (c codexGatewayBorrowRecoveryContext) Value(key any) any { + if value := c.values.Value(key); value != nil { + return value + } + return c.Context.Value(key) +} // A prepared recovery request already carries the caller's cancellation and // deadline. Preserve that exact context unless an old header guard was replaced. diff --git a/backend/internal/service/codex_gateway_borrow_immediate_test.go b/backend/internal/service/codex_gateway_borrow_immediate_test.go new file mode 100644 index 000000000000..77865b2e9a2b --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_immediate_test.go @@ -0,0 +1,290 @@ +package service + +import ( + "context" + "errors" + "io" + "net/http" + "sync/atomic" + "testing" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" + "github.com/stretchr/testify/require" +) + +func enableImmediateBorrow(t *testing.T, s *CodexGatewayBorrowService) { + t.Helper() + cfg := s.ConfigSnapshot() + cfg.WaitForQualifiedRoute, cfg.ImmediateReborrowOnFailure = true, true + cfg.NodeProxyIDs, cfg.MaxNodeAttempts = []int64{10}, 1 + s.publishConfig(cfg, false) +} + +func TestCodexBorrowImmediateConfigPreservesLegacyDefaults(t *testing.T) { + cfg, err := DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":false,"node_cooldown_seconds":0}`)) + require.NoError(t, err) + require.False(t, cfg.ImmediateReborrowOnFailure) + _, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"immediate_reborrow_on_failure":true}`)) + require.ErrorContains(t, err, "requires recovery waiting and node selection") + cfg, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"source_account_ids":[1],"target_account_ids":[2],"rotate_nodes":true,"node_proxy_ids":[10],"wait_for_qualified_route":true,"immediate_reborrow_on_failure":true,"node_cooldown_seconds":0}`)) + require.NoError(t, err) + require.True(t, cfg.ImmediateReborrowOnFailure) + require.Zero(t, cfg.NodeCooldownSeconds) +} + +func TestCodexBorrowImmediateSaveDoesNotProbeWithoutWaitingRequests(t *testing.T) { + s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) + enableImmediateBorrow(t, s) + s.settings = &borrowCoreSettings{} + var requests atomic.Int32 + base := s.upstream + s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + requests.Add(1) + return base.DoWithTLS(w, proxy, id, n, profile) + }) + _, err := s.SaveConfig(t.Context(), s.ConfigSnapshot()) + require.NoError(t, err) + require.Empty(t, s.Status().Setup.State) + require.Never(t, func() bool { return requests.Load() != 0 }, 50*time.Millisecond, time.Millisecond) +} + +func TestCodexBorrowImmediateReacquiresAfterCachedRejectionAndEachFailedRound(t *testing.T) { + var pairs atomic.Int32 + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return pairs.Add(1) >= 3 }) + enableImmediateBorrow(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + req.Header.Set("X-Codex-Turn-State", "business-state") + _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) + require.Error(t, err) + require.Len(t, *calls, 3) + identity := borrowHash("http://node-10.invalid:8080") + child := s.rotation.children[identity] + child.mu.Lock() + originalExpiry := child.candidate.expires + child.prepareFailedUntil = time.Now().Add(time.Hour) + child.prepareFailure = "old source failure" + child.mu.Unlock() + s.rotation.sourceCache.rejectSource(1, identity, time.Now()) + s.rotation.cache("gpt-6-astra").pauseSourceSearch(2, time.Now()) + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + prepared, elapsed, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + require.Less(t, elapsed, time.Second, "must not wait 15 seconds, source cooling, or a target gateway deadline") + require.Len(t, *calls, 9, "old pair plus two fresh source acquisitions and complete target pairs") + require.Equal(t, originalExpiry, child.candidate.expires, "reacquiring the same cookie must not extend its lease") + require.Equal(t, "business-state", prepared.Header.Get("X-Codex-Turn-State")) + row := s.Status().RecentUsage[0] + require.EqualValues(t, 2, row.Recovery.Rounds) + require.EqualValues(t, 2, row.Recovery.SourceRequests) + require.EqualValues(t, 4, row.Recovery.ProbeRequests) + require.Equal(t, "recovered", row.Recovery.Phase) + require.False(t, row.Dispatched) + // The failed evidence is not erased or converted into success for Sol. + require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, identity, "", time.Now()))) + require.Zero(t, s.rotation.cache("gpt-6.1-sol").blockedUntil(2, identity, "", time.Now())) + var sent atomic.Int32 + s.gateway.httpUpstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { + sent.Add(1) + require.Equal(t, int64(2), id) + require.Equal(t, "http://node-10.invalid:8080", proxy) + require.Equal(t, "business-state", w.Header.Get("X-Codex-Turn-State")) + body, err := io.ReadAll(w.Body) + require.NoError(t, err) + require.Equal(t, "business body remains unread", string(body)) + return borrowCoreResponse("gpt-6-astra", "done", "actual-response-state"), nil + }) + resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved") + require.NoError(t, err) + require.Equal(t, "actual-response-state", resp.Header.Get("X-Codex-Turn-State")) + _, err = io.ReadAll(resp.Body) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + require.EqualValues(t, 1, sent.Load()) +} + +func TestCodexBorrowImmediateFailedSourceCannotRetainRejectedCandidate(t *testing.T) { + var pairs atomic.Int32 + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return pairs.Add(1) > 1 }) + enableImmediateBorrow(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) + require.Error(t, err) + child := s.rotation.children[borrowHash("http://node-10.invalid:8080")] + base := child.upstream + var sources atomic.Int32 + child.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + if id == 1 && sources.Add(1) == 1 { + return nil, errors.New("synthetic source transport failure") + } + return base.DoWithTLS(w, proxy, id, n, profile) + }) + ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) + defer cancel() + _, _, err = s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + require.EqualValues(t, 2, sources.Load()) + require.Len(t, *calls, 6, "a failed recapture must not start probes using the retained old candidate") +} + +func TestCodexBorrowImmediateNoEligibleNodesStopsWithoutSpinning(t *testing.T) { + s, calls, proxies := nodeFixture(t, 1, func(int64, string) bool { return false }) + enableImmediateBorrow(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + delete(proxies.rows, 10) + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + require.ErrorContains(t, err, "node_candidates_unavailable") + require.Empty(t, *calls) + require.Equal(t, "stopped", s.Status().RecentUsage[0].Recovery.Phase) +} + +func TestCodexBorrowImmediateExpiredProofRequiresFreshAcquisitionAndProbes(t *testing.T) { + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) + enableImmediateBorrow(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) + require.NoError(t, err) + child := s.rotation.children[borrowHash("http://node-10.invalid:8080")] + expired := time.Now().Add(-time.Second) + child.mu.Lock() + child.candidate.expires = expired + value := child.candidate.cookie.Value + for key, check := range child.qualifications { + check.expires = expired + child.qualifications[key] = check + } + child.mu.Unlock() + s.rotation.mu.Lock() + s.rotation.leases[borrowHash(value)] = expired + s.rotation.mu.Unlock() + _, _, err = s.Apply(req, a, "gpt-6-astra", "saved", nil, true) + require.Error(t, err) + require.Len(t, *calls, 3, "an expired proof cannot dispatch or probe underneath business sending") + _, _, err = s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + require.Len(t, *calls, 6, "lease expiry requires a new source response and two complete target probes, even if the routing cookie value is the same") + require.True(t, time.Now().Before(child.candidate.expires)) + require.LessOrEqual(t, time.Until(child.candidate.expires), 230*time.Second) +} + +func TestCodexBorrowImmediateCancelledProbeReleasesPreparation(t *testing.T) { + s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) + enableImmediateBorrow(t, s) + base := s.upstream + started := make(chan struct{}, 1) + s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + if id == 2 { + started <- struct{}{} + <-w.Context().Done() + return nil, w.Context().Err() + } + return base.DoWithTLS(w, proxy, id, n, profile) + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + <-started + cancel() + require.ErrorIs(t, <-done, context.Canceled) + require.Eventually(t, func() bool { + s.rotation.flights.mu.Lock() + defer s.rotation.flights.mu.Unlock() + return len(s.rotation.flights.active) == 0 + }, time.Second, time.Millisecond) + require.Equal(t, "cancelled", s.Status().RecentUsage[0].Recovery.Phase) +} + +func TestCodexBorrowImmediateSharedPreparationSurvivesOneCancellation(t *testing.T) { + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) + enableImmediateBorrow(t, s) + base := s.upstream + entered, release := make(chan struct{}, 1), make(chan struct{}) + s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + if id == 1 { + entered <- struct{}{} + select { + case <-release: + case <-req.Context().Done(): + return nil, req.Context().Err() + } + } + return base.DoWithTLS(req, proxy, id, n, profile) + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + first, second := make(chan error, 1), make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + first <- err + }() + <-entered + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + second <- err + }() + require.Eventually(t, func() bool { + s.rotation.flights.mu.Lock() + defer s.rotation.flights.mu.Unlock() + for _, f := range s.rotation.flights.active { + if f.waiters == 2 { + return true + } + } + return false + }, time.Second, time.Millisecond) + cancel() + require.ErrorIs(t, <-first, context.Canceled) + close(release) + require.NoError(t, <-second) + require.Len(t, *calls, 3) +} + +type immediateBorrowAccountGuard struct { + AccountRepository + inactive atomic.Bool +} + +func (r *immediateBorrowAccountGuard) GetByID(ctx context.Context, id int64) (*Account, error) { + a, err := r.AccountRepository.GetByID(ctx, id) + if err == nil && id == 2 && r.inactive.Load() { + copy := *a + copy.Status = "disabled" + return ©, nil + } + return a, err +} + +func TestCodexBorrowImmediateDisabledAccountCancelsAnOngoingProbe(t *testing.T) { + s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) + enableImmediateBorrow(t, s) + guard := &immediateBorrowAccountGuard{AccountRepository: s.accounts} + s.accounts, s.gateway.accountRepo = guard, guard + base := s.upstream + entered := make(chan struct{}, 1) + s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + if id == 2 { + entered <- struct{}{} + <-w.Context().Done() + return nil, w.Context().Err() + } + return base.DoWithTLS(w, proxy, id, n, profile) + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithTimeout(t.Context(), 8*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + <-entered + guard.inactive.Store(true) + require.ErrorContains(t, <-done, "account_inactive") + require.False(t, s.Status().RecentUsage[0].Dispatched) +} diff --git a/backend/internal/service/codex_gateway_borrow_probe.go b/backend/internal/service/codex_gateway_borrow_probe.go index 633def15e889..24f2a852a0ed 100644 --- a/backend/internal/service/codex_gateway_borrow_probe.go +++ b/backend/internal/service/codex_gateway_borrow_probe.go @@ -120,6 +120,11 @@ func (s *CodexGatewayBorrowService) acquireSource(ctx context.Context, id int64) if err != nil { return nil, err.Error(), err } + if s.node != nil { + proxy = s.node.proxy + req = req.WithContext(WithHTTPUpstreamProfile(req.Context(), HTTPUpstreamProfileOpenAIHarvest)) + req.Close = true + } shot, err := s.sendObservation(req, a, proxy, nil, codexGatewayBorrowSourceModel, codexGatewayBorrowProbeMaxBody) if err != nil { return nil, shot.errorText, err @@ -136,7 +141,14 @@ func (s *CodexGatewayBorrowService) acquireSource(ctx context.Context, id int64) return nil, "routing cookie missing, invalid, deleted or expired", errors.New("source_routing_cookie_unavailable") } candidate.sourceID = id + candidate.node = s.node candidate.sourceRequestEncoding = shot.requestEncoding + if s.node != nil && s.node.expires != nil && s.node.expires.Before(candidate.expires) { + candidate.expires = *s.node.expires + } + if s.nodeLease != nil { + candidate.expires = s.nodeLease(candidate.cookie.Value, candidate.expires) + } return candidate, "", nil } @@ -228,6 +240,10 @@ func (s *CodexGatewayBorrowService) probeTarget(ctx context.Context, template *h ctx = withCodexBorrowPlainProbe(ctx) expires := candidate.expires result := CodexGatewayBorrowVerification{AccountID: account.ID, Model: model, CheckedAt: time.Now(), ExpiresAt: &expires, Reason: "target_probe_failed", RequestShape: codexBorrowProbeRanxiLiteral} + if s.node != nil { + result.ProxyID, result.NodeIdentity, result.Gateway = s.node.id, s.node.identity, codexBorrowGateway(candidate.cookie.Value) + result.ProxyName = s.node.name + } if shape, _ := ctx.Value(codexBorrowProbeShapeContextKey{}).(string); shape != "" { result.RequestShape = shape } else { @@ -477,6 +493,7 @@ func (s *CodexGatewayBorrowService) sendObservation(req *http.Request, account * shot.requestEncoding = "identity" } resp, phase, attempts, err := s.sendBorrowObservationHTTP(wire, proxy, account, profile) + codexBorrowUsageFromContext(req.Context()).recoveryObservation(req.Context().Value(codexBorrowSourceObservationKey{}) == true, attempts) shot.transportPhase, shot.transportAttempts = phase, attempts if err != nil { if resp != nil && resp.Body != nil { diff --git a/backend/internal/service/codex_gateway_borrow_recovery.go b/backend/internal/service/codex_gateway_borrow_recovery.go new file mode 100644 index 000000000000..5d812a63ec66 --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_recovery.go @@ -0,0 +1,302 @@ +package service + +import ( + "context" + "errors" + "log" + "net/http" + "time" +) + +// Only request-owned recovery can bypass preparation cooldowns. Administrators +// keep ordinary rules; dispatch still requires a matching successful proof. +type borrowImmediateReborrowKey struct{} +type borrowFreshSourceKey struct{} + +func borrowImmediateReborrow(ctx context.Context) bool { + value, _ := ctx.Value(borrowImmediateReborrowKey{}).(bool) + return value +} + +// Request-owned recovery never writes downstream headers or starts a business +// generation. Every shared preparation remains bounded; only waiting is open-ended. +type CodexGatewayBorrowRecovery struct { + Phase string `json:"phase"` + StartedAt time.Time `json:"started_at"` + NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"` + Rounds uint64 `json:"rounds"` + SourceRequests uint64 `json:"source_requests"` + ProbeRequests uint64 `json:"probe_requests"` + SharedPreparation bool `json:"shared_preparation,omitempty"` + LastStage string `json:"last_stage,omitempty"` + LastReason string `json:"last_reason,omitempty"` + LastError string `json:"last_error,omitempty"` +} + +func (t *codexBorrowUsageTracker) recovering(phase string, next *time.Time, err error) { + if t == nil { + return + } + t.mu.Lock() + defer t.mu.Unlock() + if t.finished || t.row.Dispatched { + return + } + if t.row.Recovery == nil && phase != "searching" { + return + } + r := CodexGatewayBorrowRecovery{StartedAt: time.Now()} + if t.row.Recovery != nil { + r = *t.row.Recovery + } + r.Phase, r.NextAttemptAt = phase, next + if phase == "searching" { + r.Rounds++ + } + if err != nil { + r.LastStage, r.LastReason = codexBorrowFailureStage(err) + r.LastError = err.Error() + } + t.row.Recovery = &r + t.publish(0, 0, 0) + log.Printf("[codex_borrow_recovery] request_id=%s account_id=%d model=%s phase=%s round=%d reason=%s next_attempt_at=%v source_requests=%d probe_requests=%d", t.row.GatewayRequestID, t.key.account, t.key.model, phase, r.Rounds, r.LastReason, next, r.SourceRequests, r.ProbeRequests) +} + +func (t *codexBorrowUsageTracker) recoveryObservation(source bool, attempts int) { + if t == nil || attempts == 0 { + return + } + t.mu.Lock() + defer t.mu.Unlock() + if t.row.Recovery == nil { + return + } + r := *t.row.Recovery + if source { + r.SourceRequests += uint64(attempts) + } else { + r.ProbeRequests += uint64(attempts) + } + t.row.Recovery = &r + t.publish(0, 0, 0) + log.Printf("[codex_borrow_recovery_observation] request_id=%s account_id=%d model=%s source=%t attempts=%d", t.row.GatewayRequestID, t.key.account, t.key.model, source, attempts) +} + +func (t *codexBorrowUsageTracker) sharedRecoveryPreparation() { + if t == nil { + return + } + t.mu.Lock() + defer t.mu.Unlock() + if t.finished || t.row.Recovery == nil { + return + } + r := *t.row.Recovery + r.SharedPreparation = true + t.row.Recovery = &r + t.publish(0, 0, 0) +} + +func borrowRecoveryRetryable(err error) bool { + if errors.Is(err, ErrCodexGatewayBorrowChanged) || errors.Is(err, context.Canceled) { + return false + } + var failure *CodexGatewayBorrowFailure + if errors.As(err, &failure) { + if v := failure.Verification; v != nil && (v.Reason == "account_unavailable" || v.MintStatus == 401 || v.ContinueStatus == 401) { + return false + } + switch failure.Reason { + case "account_unavailable", "account_inactive", "model_not_allowed", "model_mapping_mismatch", "configuration_changed", "node_proxy_repository_unavailable", "node_candidates_unavailable": + return false + } + return true + } + return errors.Is(err, ErrCodexGatewayBorrowUnavailable) || errors.Is(err, context.DeadlineExceeded) +} + +// Compute the next actual eligible node, rather than spinning every 15 seconds +// through an hour-long rejection. Source/node and target/gateway quiet windows +// are combined; a global source-search pause is a lower bound for all nodes. +func (s *CodexGatewayBorrowService) nextBorrowRecoveryAttempt(ctx context.Context, accountID int64, model string, err error) time.Time { + now := time.Now() + if borrowImmediateReborrow(ctx) { + return now + } + fallback := now.Add(codexGatewayBorrowFailureWait) + cfg := s.ConfigSnapshot() + nodes, loadErr := s.borrowNodes(ctx, cfg.NodeProxyIDs) + if loadErr != nil || len(nodes) == 0 { + return fallback + } + cache := s.rotation.cache(model) + s.rotation.mu.Lock() + global := s.rotation.retry[codexGatewayBorrowTargetKey{accountID, model}] + s.rotation.mu.Unlock() + if until := cache.sourceSearchPausedUntil(accountID, now); !cfg.WaitForQualifiedRoute && until.After(global) { + global = until + } + var earliest time.Time + for _, node := range nodes { + until := cache.blockedUntil(accountID, node.identity, "", now) + if sourceUntil := s.rotation.sourceCache.sourcesBlockedUntil(cfg.SourceAccountIDs, node.identity, now); sourceUntil.After(until) { + until = sourceUntil + } + if global.After(until) { + until = global + } + if !now.Before(until) { + return fallback + } + if earliest.IsZero() || until.Before(earliest) { + earliest = until + } + } + if earliest.IsZero() { + return fallback + } + return earliest +} + +func (s *CodexGatewayBorrowService) recoveryAccount(ctx context.Context, id int64, model string) (*Account, *http.Request, error) { + a, template, _, err := s.accountTemplate(ctx, id, model) + if err != nil { + return nil, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "recovery", Reason: "account_unavailable"} + } + return a, template, nil +} + +func (s *CodexGatewayBorrowService) waitBorrowRecovery(ctx context.Context, next time.Time, accountID int64) error { + timer := time.NewTimer(time.Until(next)) + defer timer.Stop() + ticker := time.NewTicker(5 * time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-timer.C: + return nil + case <-ticker.C: + a, err := s.accounts.GetByID(ctx, accountID) + if err != nil || !codexGatewayBorrowAccountSupported(a) || a.Status != StatusActive { + return &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"} + } + } + } +} + +func (s *CodexGatewayBorrowService) recoverQualifiedHTTP(ctx context.Context, req *http.Request, account *Account, model, proxy string, revision uint64, revisionCtx context.Context) (*http.Request, *CodexGatewayBorrowApplication, error) { + caller := ctx + ctx = codexGatewayBorrowRecoveryContext{Context: ctx, values: req.Context()} + ctx, cancel := context.WithCancelCause(ctx) + if s.ConfigSnapshot().ImmediateReborrowOnFailure { + ctx = context.WithValue(ctx, borrowImmediateReborrowKey{}, true) + guardCtx := ctx + go func() { + ticker := time.NewTicker(5 * time.Second) + defer ticker.Stop() + for { + select { + case <-guardCtx.Done(): + return + case <-ticker.C: + checkCtx, stop := context.WithTimeout(guardCtx, codexGatewayBorrowStoreTimeout) + current, err := s.accounts.GetByID(checkCtx, account.ID) + stop() + if err != nil || !codexGatewayBorrowAccountSupported(current) || current.Status != StatusActive || (account.Schedulable && !current.Schedulable) { + cancel(&CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"}) + return + } + } + } + }() + } + policy := currentCodexFingerprintPolicyForAccount(account).revision + ctx, cancelPolicy := borrowPolicyContext(ctx, account.ID, policy) + defer cancelPolicy() + changed := func() bool { + return revisionCtx.Err() != nil || currentCodexFingerprintPolicyForAccount(account).revision != policy + } + stopRevision := context.AfterFunc(revisionCtx, func() { cancel(ErrCodexGatewayBorrowChanged) }) + defer stopRevision() + defer cancel(nil) + tracker := codexBorrowUsageFromContext(ctx) + originalAccount := req.Header.Get("ChatGPT-Account-ID") + wire := req.Clone(ctx) // Shares Body/GetBody; the pending generation is never read or replayed. + for { + if err := ctx.Err(); err != nil { + if changed() { + return nil, nil, ErrCodexGatewayBorrowChanged + } + return nil, nil, context.Cause(ctx) + } + s.mu.Lock() + current := s.revision == revision && s.config.Enabled && s.config.WaitForQualifiedRoute && s.config.RotateNodes + s.mu.Unlock() + if !current { + return nil, nil, ErrCodexGatewayBorrowChanged + } + checkCtx, cancelCheck := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) + fresh, auth, err := s.recoveryAccount(checkCtx, account.ID, model) + cancelCheck() + if err != nil { + return nil, nil, err + } + if auth.Header.Get("ChatGPT-Account-ID") != originalAccount { + return nil, nil, ErrCodexGatewayBorrowChanged + } + wire.Header.Set("Authorization", auth.Header.Get("Authorization")) + tracker.recovering("searching", nil, nil) + roundCtx, cancelRound := context.WithTimeout(ctx, 180*time.Second) + prepared, app, err := s.Apply(wire.WithContext(roundCtx), fresh, model, proxy, nil, false) + cancelRound() + if err == nil && app != nil && app.Applied { + checkCtx, cancelCheck := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) + latest, latestAuth, checkErr := s.recoveryAccount(checkCtx, account.ID, model) + cancelCheck() + if checkErr != nil { + return nil, nil, checkErr + } + if account.Schedulable && !latest.Schedulable { + return nil, nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"} + } + if latestAuth.Header.Get("ChatGPT-Account-ID") != originalAccount || latestAuth.Header.Get("X-OpenAI-Fedramp") != wire.Header.Get("X-OpenAI-Fedramp") { + return nil, nil, ErrCodexGatewayBorrowChanged + } + if latestAuth.Header.Get("Authorization") != wire.Header.Get("Authorization") { + continue + } + tracker.recovering("recovered", nil, nil) + return prepared.WithContext(codexGatewayBorrowRecoveryContext{Context: caller, values: prepared.Context()}), app, nil + } + if err == nil { + err = ErrCodexGatewayBorrowChanged + } + if ctx.Err() != nil { + continue + } + if errors.Is(err, ErrCodexGatewayBorrowChanged) { + // Revision cancellation stops above. Lease expiry or a changed node + // within a live configuration requires a new qualified route. + err = &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "route_changed"} + } + if !borrowRecoveryRetryable(err) { + return nil, nil, err + } + next := s.nextBorrowRecoveryAttempt(ctx, account.ID, model, err) + if borrowImmediateReborrow(ctx) { + // The next round starts immediately, but still performs new network + // acquisition and full probes. Never spin on an empty candidate pool. + tracker.recovering("reborrowing", nil, err) + continue + } + tracker.recovering("cooling", &next, err) + if err := s.waitBorrowRecovery(ctx, next, account.ID); err != nil { + if changed() { + return nil, nil, ErrCodexGatewayBorrowChanged + } + return nil, nil, err + } + } +} diff --git a/backend/internal/service/codex_gateway_borrow_recovery_test.go b/backend/internal/service/codex_gateway_borrow_recovery_test.go new file mode 100644 index 000000000000..688282fa9770 --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_recovery_test.go @@ -0,0 +1,401 @@ +package service + +import ( + "context" + "errors" + "io" + "net/http" + "sync/atomic" + "testing" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" + "github.com/stretchr/testify/require" +) + +func enableBorrowRecovery(t *testing.T, s *CodexGatewayBorrowService) { + t.Helper() + cfg := s.ConfigSnapshot() + cfg.WaitForQualifiedRoute, cfg.NodeProxyIDs = true, []int64{10} + s.publishConfig(cfg, false) + s.rotation.mu.Lock() + err := s.rotation.initializeLocked(s.revision) + s.rotation.mu.Unlock() + require.NoError(t, err) +} + +func seedBorrowRecoveryCooling(s *CodexGatewayBorrowService, model string, until time.Time) { + s.rotation.cache(model).reject(2, borrowHash("http://node-10.invalid:8080"), "", until, time.Now()) + s.rotation.mu.Lock() + s.rotation.retry[codexGatewayBorrowTargetKey{2, model}] = until + s.rotation.mu.Unlock() +} + +func TestCodexBorrowRecoveryRetainsRequestAndDispatchesOnceAfterCooling(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + req.Header.Set("X-Codex-Turn-State", "original-business-state") + req.Header.Set("Session-Id", "original-session") + req.Header.Set("X-Codex-Turn-Metadata", `{"thread_id":"original-thread"}`) + body := &struct{ io.ReadCloser }{req.Body} + req.Body = body + until := time.Now().Add(100 * time.Millisecond) + seedBorrowRecoveryCooling(s, "gpt-6-astra", until) + prepared, elapsed, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved-exit") + require.NoError(t, err) + require.GreaterOrEqual(t, elapsed, 80*time.Millisecond) + require.Same(t, body, prepared.Body) + require.Equal(t, "original-business-state", prepared.Header.Get("X-Codex-Turn-State")) + require.Equal(t, req.Header.Get("X-Codex-Turn-Metadata"), prepared.Header.Get("X-Codex-Turn-Metadata")) + require.Equal(t, "original-session", prepared.Header.Get("Session-Id")) + require.Empty(t, req.Header.Get("Cookie")) + require.Len(t, *calls, 3) + row := s.Status().RecentUsage[0] + require.False(t, row.Dispatched) + require.Equal(t, "preparing", row.Outcome) + require.Equal(t, "recovered", row.Recovery.Phase) + require.EqualValues(t, 2, row.Recovery.Rounds) + require.EqualValues(t, 1, row.Recovery.SourceRequests) + require.EqualValues(t, 2, row.Recovery.ProbeRequests) + var dispatched atomic.Int32 + s.gateway.httpUpstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { + dispatched.Add(1) + require.Equal(t, "http://node-10.invalid:8080", proxy) + require.Equal(t, int64(2), id) + require.Equal(t, "original-business-state", w.Header.Get("X-Codex-Turn-State")) + data, err := io.ReadAll(w.Body) + require.NoError(t, err) + require.Equal(t, "business body remains unread", string(data)) + return borrowCoreResponse("gpt-6-astra", "done", "returned-business-state"), nil + }) + resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved-exit") + require.NoError(t, err) + require.Equal(t, "returned-business-state", resp.Header.Get("X-Codex-Turn-State")) + _, err = io.ReadAll(resp.Body) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + require.EqualValues(t, 1, dispatched.Load()) + row = s.Status().RecentUsage[0] + require.True(t, row.Dispatched) + require.Equal(t, "completed", row.Outcome) +} + +func TestCodexBorrowRecoveryCancelAndConfigChangeStopWithoutProbes(t *testing.T) { + for _, change := range []bool{false, true} { + t.Run(map[bool]string{false: "cancel", true: "configuration"}[change], func(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + require.Eventually(t, func() bool { + rows := s.Status().RecentUsage + return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" + }, time.Second, time.Millisecond) + if change { + cfg := s.ConfigSnapshot() + cfg.WaitForQualifiedRoute = false + s.publishConfig(cfg, false) + } else { + cancel() + } + select { + case err := <-done: + if change { + require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) + } else { + require.ErrorIs(t, err, context.Canceled) + require.Equal(t, "cancelled", s.Status().RecentUsage[0].Recovery.Phase) + } + case <-time.After(time.Second): + t.Fatal("recovery leaked after cancellation") + } + require.Empty(t, *calls) + }) + } +} + +func TestCodexBorrowRecoveryUsesLatestEligibleNodeDeadline(t *testing.T) { + s, _, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + now := time.Now() + until := now.Add(time.Hour) + seedBorrowRecoveryCooling(s, "gpt-6-astra", until) + next := s.nextBorrowRecoveryAttempt(t.Context(), 2, "gpt-6-astra", ErrCodexGatewayBorrowUnavailable) + require.Equal(t, until, next) + other := s.nextBorrowRecoveryAttempt(t.Context(), 2, "gpt-6.1-sol", ErrCodexGatewayBorrowUnavailable) + require.WithinDuration(t, now.Add(15*time.Second), other, time.Second) +} + +func TestCodexBorrowRecoveryDisabledAndDiagnosticsRemainBounded(t *testing.T) { + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + require.Error(t, err) + require.Len(t, *calls, 3) + require.Nil(t, s.Status().RecentUsage[0].Recovery) + _, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"source_account_ids":[1],"target_account_ids":[2],"models":["gpt-6-astra"],"wait_for_qualified_route":true}`)) + require.ErrorContains(t, err, "requires node selection") + require.False(t, borrowRecoveryRetryable(context.Canceled)) + require.False(t, borrowRecoveryRetryable(ErrCodexGatewayBorrowChanged)) + require.False(t, borrowRecoveryRetryable(errors.New("not a borrowing failure"))) + require.True(t, borrowRecoveryRetryable(&CodexGatewayBorrowFailure{Cause: io.ErrUnexpectedEOF, Reason: "target_probe_failed"})) + require.False(t, borrowRecoveryRetryable(&CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Verification: &CodexGatewayBorrowVerification{MintStatus: 401}})) +} + +func TestCodexBorrowRecoverySharedPreparationSurvivesOneCallerCancelling(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + base := s.upstream + entered := make(chan struct{}, 1) + release := make(chan struct{}) + s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + if id == 1 { + entered <- struct{}{} + select { + case <-release: + case <-req.Context().Done(): + return nil, req.Context().Err() + } + } + return base.DoWithTLS(req, proxy, id, n, profile) + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + first, second := make(chan error, 1), make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + first <- err + }() + select { + case <-entered: + case <-time.After(time.Second): + t.Fatal("source did not start") + } + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + second <- err + }() + require.Eventually(t, func() bool { + s.rotation.flights.mu.Lock() + defer s.rotation.flights.mu.Unlock() + for _, f := range s.rotation.flights.active { + if f.waiters == 2 { + return true + } + } + return false + }, time.Second, time.Millisecond) + cancel() + require.ErrorIs(t, <-first, context.Canceled) + close(release) + require.NoError(t, <-second) + require.Len(t, *calls, 3, "the remaining caller reuses the same bounded preparation") +} + +func TestCodexBorrowRecoveryCoolingDoesNotBlockOtherModel(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + require.Eventually(t, func() bool { + rows := s.Status().RecentUsage + return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" + }, time.Second, time.Millisecond) + _, sol := nodeRequest(t, s, "gpt-6.1-sol") + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), sol, a, "gpt-6.1-sol", "saved") + require.NoError(t, err) + cancel() + require.ErrorIs(t, <-done, context.Canceled) + require.Len(t, *calls, 3) + require.True(t, s.Status().Targets[1].CacheValid) +} + +func TestCodexBorrowRecoveryLastCallerCancellationStopsTheProbe(t *testing.T) { + s, _, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + entered, closed := make(chan struct{}), make(chan struct{}) + s.upstream = borrowCoreProbe(func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { + close(entered) + <-req.Context().Done() + close(closed) + return nil, req.Context().Err() + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + select { + case <-entered: + case <-time.After(time.Second): + t.Fatal("probe did not start") + } + cancel() + require.ErrorIs(t, <-done, context.Canceled) + select { + case <-closed: + case <-time.After(time.Second): + t.Fatal("probe outlived its last waiter") + } + require.Eventually(t, func() bool { + s.rotation.flights.mu.Lock() + defer s.rotation.flights.mu.Unlock() + return len(s.rotation.flights.active) == 0 + }, time.Second, time.Millisecond) +} + +type borrowRecoveryAccountSwitch struct { + AccountRepository + inactive atomic.Bool +} + +func (r *borrowRecoveryAccountSwitch) GetByID(ctx context.Context, id int64) (*Account, error) { + a, err := r.AccountRepository.GetByID(ctx, id) + if err != nil || a == nil { + return a, err + } + copy := *a + if id == 2 && r.inactive.Load() { + copy.Status = "disabled" + } + return ©, nil +} + +func TestCodexBorrowRecoveryStopsWhenAccountIsDisabledDuringCooling(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + repo := &borrowRecoveryAccountSwitch{AccountRepository: s.accounts} + s.accounts = repo + s.gateway.accountRepo = repo + a, req := nodeRequest(t, s, "gpt-6-astra") + seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + require.Eventually(t, func() bool { + rows := s.Status().RecentUsage + return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" + }, time.Second, time.Millisecond) + repo.inactive.Store(true) + select { + case err := <-done: + require.ErrorContains(t, err, "account_inactive") + case <-time.After(8 * time.Second): + t.Fatal("disabled account kept waiting") + } + require.Empty(t, *calls) +} + +func TestCodexBorrowRecoveryCoolingGatewayDoesNotHideUntestedNodes(t *testing.T) { + s, calls, proxies := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 12 }) + cfg := s.ConfigSnapshot() + cfg.WaitForQualifiedRoute = true + cfg.NodeProxyIDs = []int64{10, 11, 12} + s.publishConfig(cfg, false) + base := s.upstream + alias := proxies.rows[11] + s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, p *tlsfingerprint.Profile) (*http.Response, error) { + resp, err := base.DoWithTLS(req, proxy, id, n, p) + if err == nil && id == 1 && proxy == alias.URL() { + resp.Header.Set("Set-Cookie", "__oailb="+borrowNodeCookie("chat.gateway.unified-10.api.openai.com")+"; Path=/; Secure; Max-Age=230") + } + return resp, err + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithTimeout(t.Context(), time.Second) + defer cancel() + prepared, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + proof, ok := prepared.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) + require.True(t, ok) + require.Equal(t, int64(12), proof.application.Verification.ProxyID) + require.Len(t, *calls, 7, "two strict pairs and three source acquisitions; no target call to the cooling alias") + require.EqualValues(t, 1, s.Status().RecentUsage[0].Recovery.Rounds) + require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()))) +} + +func TestCodexBorrowRecoveryIdentityEditCancelsCoolingWait(t *testing.T) { + s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) + enableBorrowRecovery(t, s) + a, req := nodeRequest(t, s, "gpt-6-astra") + seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + done := make(chan error, 1) + go func() { + _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") + done <- err + }() + require.Eventually(t, func() bool { + rows := s.Status().RecentUsage + return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" + }, time.Second, time.Millisecond) + notifyCodexFingerprintAccountChanged(2) + select { + case err := <-done: + require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) + case <-time.After(time.Second): + t.Fatal("old identity kept recovering") + } + require.Empty(t, *calls) +} + +func TestCodexBorrowRecoveryRenewsTheQualifiedNodeBeforeRotating(t *testing.T) { + s, calls, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) + cfg := s.ConfigSnapshot() + cfg.WaitForQualifiedRoute = true + cfg.NodeProxyIDs = []int64{10, 11} + s.publishConfig(cfg, false) + base := s.upstream + s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, p *tlsfingerprint.Profile) (*http.Response, error) { + resp, err := base.DoWithTLS(req, proxy, id, n, p) + if err == nil && id == 1 { + cookies := resp.Cookies() + require.Len(t, cookies, 1) + cookies[0].MaxAge = 1 + resp.Header.Set("Set-Cookie", cookies[0].String()) + } + return resp, err + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + first, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + proof, ok := first.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) + require.True(t, ok) + timer := time.NewTimer(time.Until(proof.application.ExpiresAt.Add(20 * time.Millisecond))) + defer timer.Stop() + <-timer.C + second, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") + require.NoError(t, err) + next, ok := second.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) + require.True(t, ok) + require.Equal(t, int64(10), next.application.Verification.ProxyID) + require.Len(t, *calls, 6, "renewal reacquires a lease and repeats both target probes") + for _, call := range *calls { + require.Contains(t, call, "10:") + } + require.True(t, next.application.ExpiresAt.After(proof.application.ExpiresAt)) +} diff --git a/backend/internal/service/codex_gateway_borrow_rotation.go b/backend/internal/service/codex_gateway_borrow_rotation.go new file mode 100644 index 000000000000..793ccac5cbda --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_rotation.go @@ -0,0 +1,667 @@ +package service + +// Node selection and cooling follow ranxi2001/sub2api d977e7a5: +// repository/astra_rotation{,_cache}.go (LGPL-3.0). Registered host proxies +// replace Mihomo's private listeners. Proxy/account rows are never modified. + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "net/http" + "regexp" + "slices" + "sort" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" +) + +type codexBorrowNode struct { + id int64 + identity, name, proxy string + expires *time.Time +} + +type codexBorrowNodeBinding struct { + owner *CodexGatewayBorrowService + child *CodexGatewayBorrowService + revision uint64 + accountID int64 + model, gateway string + node codexBorrowNode +} + +type codexBorrowNodeContextKey struct{} + +type CodexGatewayBorrowNodeStatus struct { + AccountID int64 `json:"account_id"` + Model string `json:"model"` + ProxyID int64 `json:"proxy_id"` + ProxyName string `json:"proxy_name"` + NodeIdentity string `json:"node_identity"` + Gateway string `json:"gateway,omitempty"` + State string `json:"state"` + RetryAfter *time.Time `json:"retry_after,omitempty"` + ExpiresAt *time.Time `json:"expires_at,omitempty"` + Reason string `json:"reason,omitempty"` +} + +type codexBorrowRotation struct { + mu sync.Mutex + revision uint64 + children map[string]*CodexGatewayBorrowService + active map[codexGatewayBorrowTargetKey]string + cursor map[codexGatewayBorrowTargetKey]int + retry map[codexGatewayBorrowTargetKey]time.Time + caches map[string]*codexBorrowRotationCache + sourceCache codexBorrowRotationCache + leases map[string]time.Time + flights codexBorrowFlights +} + +func validateCodexBorrowNodeConfig(cfg CodexGatewayBorrowConfig) error { + if cfg.ImmediateReborrowOnFailure && (!cfg.WaitForQualifiedRoute || !cfg.RotateNodes) { + return errors.New("immediate reborrow requires recovery waiting and node selection") + } + if cfg.Enabled && cfg.WaitForQualifiedRoute && !cfg.RotateNodes { + return errors.New("waiting for a qualified route requires node selection") + } + if cfg.MaxNodeAttempts < 0 || cfg.MaxNodeAttempts > 10 { + return errors.New("node attempts must be 0–10 (0 uses 3)") + } + if cfg.NodeCooldownSeconds != 0 && (cfg.NodeCooldownSeconds < 60 || cfg.NodeCooldownSeconds > 86400) { + return errors.New("node cooldown must be 60–86400 seconds (0 uses 3600)") + } + if len(cfg.NodeProxyIDs) > 64 || (cfg.RotateNodes && len(cfg.NodeProxyIDs) == 0) { + return errors.New("node selection requires 1–64 explicit proxy IDs") + } + seen := map[int64]bool{} + for _, id := range cfg.NodeProxyIDs { + if id < 0 || seen[id] { + return errors.New("node proxy IDs must be unique; 0 explicitly selects server direct") + } + seen[id] = true + } + return nil +} + +func borrowNodeConfigEqual(a, b CodexGatewayBorrowConfig) bool { + return a.ImmediateReborrowOnFailure == b.ImmediateReborrowOnFailure && a.WaitForQualifiedRoute == b.WaitForQualifiedRoute && a.RotateNodes == b.RotateNodes && a.MaxNodeAttempts == b.MaxNodeAttempts && a.NodeCooldownSeconds == b.NodeCooldownSeconds && slices.Equal(a.NodeProxyIDs, b.NodeProxyIDs) +} + +func borrowNodeFromProxy(p *Proxy) (codexBorrowNode, error) { + if p == nil || p.Status != StatusActive || (p.ExpiresAt != nil && !time.Now().Before(*p.ExpiresAt)) || (p.Protocol != "http" && p.Protocol != "https" && p.Protocol != "socks5" && p.Protocol != "socks5h") { + return codexBorrowNode{}, errors.New("node proxy is unavailable") + } + url := p.URL() + return codexBorrowNode{id: p.ID, identity: borrowHash(url), name: p.Name, proxy: url, expires: p.ExpiresAt}, nil +} + +func (s *CodexGatewayBorrowService) borrowNodes(ctx context.Context, ids []int64, strict ...bool) ([]codexBorrowNode, error) { + if s.proxies == nil { + return nil, errors.New("node proxy repository unavailable") + } + databaseIDs := []int64{} + for _, id := range ids { + if id > 0 { + databaseIDs = append(databaseIDs, id) + } + } + rows, err := s.proxies.ListByIDs(ctx, databaseIDs) + if err != nil { + return nil, err + } + byID := map[int64]*Proxy{} + for i := range rows { + byID[rows[i].ID] = &rows[i] + } + seen := map[string]bool{} + nodes := []codexBorrowNode{} + for _, id := range ids { + if id == 0 { + nodes = append(nodes, codexBorrowNode{id: 0, identity: borrowHash("codex-borrow-server-direct"), name: "Server direct"}) + continue + } + node, err := borrowNodeFromProxy(byID[id]) + if err != nil { + if len(strict) > 0 && strict[0] { + return nil, fmt.Errorf("proxy %d: %w", id, err) + } + continue + } + if !seen[node.identity] { + nodes = append(nodes, node) + seen[node.identity] = true + } + } + return nodes, nil +} + +func (r *codexBorrowRotation) cache(model string) *codexBorrowRotationCache { + r.mu.Lock() + defer r.mu.Unlock() + if r.caches == nil { + r.caches = map[string]*codexBorrowRotationCache{} + } + if r.caches[model] == nil { + r.caches[model] = &codexBorrowRotationCache{} + } + return r.caches[model] +} + +func (s *CodexGatewayBorrowService) nodeChild(rev uint64, cfg CodexGatewayBorrowConfig, node codexBorrowNode) (*CodexGatewayBorrowService, error) { + r := &s.rotation + r.mu.Lock() + defer r.mu.Unlock() + if err := r.initializeLocked(rev); err != nil { + return nil, err + } + if child := r.children[node.identity]; child != nil { + return child, nil + } + // Reuse the tested cookie/probe implementation with a separate cache for + // each immutable node. Its gateway dependency still belongs to the host. + child := NewCodexGatewayBorrowService(nil, s.accounts, nil, s.upstream, s.tlsProfiles) + child.gateway = s.gateway + child.node = &node + child.nodeLease = r.cookieExpiry + cfg.RotateNodes = false + cfg.WaitForQualifiedRoute = false + cfg.ImmediateReborrowOnFailure = false + child.publishConfig(cfg, false) + r.children[node.identity] = child + return child, nil +} + +func (r *codexBorrowRotation) initializeLocked(rev uint64) error { + if r.revision > rev { + return ErrCodexGatewayBorrowChanged + } + if r.revision != rev || r.children == nil { + for _, child := range r.children { + child.Stop() + } + r.children = map[string]*CodexGatewayBorrowService{} + r.active = map[codexGatewayBorrowTargetKey]string{} + r.cursor = map[codexGatewayBorrowTargetKey]int{} + r.retry = map[codexGatewayBorrowTargetKey]time.Time{} + r.revision = rev + } + return nil +} + +func (r *codexBorrowRotation) stop() { + r.mu.Lock() + defer r.mu.Unlock() + for _, child := range r.children { + child.Stop() + } +} + +func (r *codexBorrowRotation) cookieExpiry(value string, proposed time.Time) time.Time { + r.mu.Lock() + defer r.mu.Unlock() + now := time.Now() + key := borrowHash(value) + if r.leases == nil { + r.leases = map[string]time.Time{} + } + if old, ok := r.leases[key]; ok && now.Before(old) && old.Before(proposed) { + return old + } + if len(r.leases) >= 4096 { + for k, v := range r.leases { + if !now.Before(v) { + delete(r.leases, k) + } + } + if len(r.leases) >= 4096 { + for k := range r.leases { + delete(r.leases, k) + break + } + } + } + r.leases[key] = proposed + return proposed +} + +var codexBorrowGatewayPattern = regexp.MustCompile(`^chat\.gateway\.unified-[0-9]+\.api\.openai\.com$`) + +// Descriptive routing metadata only, never JWT verification or model identity. +func codexBorrowGateway(value string) string { + parts := strings.Split(value, ".") + if len(parts) != 3 { + return "" + } + raw, err := base64.RawURLEncoding.DecodeString(parts[1]) + if err != nil { + return "" + } + var claims struct { + Host string `json:"host"` + } + if json.Unmarshal(raw, &claims) != nil || !codexBorrowGatewayPattern.MatchString(claims.Host) { + return "" + } + return claims.Host +} + +func (s *CodexGatewayBorrowService) nodeCurrent(ctx context.Context, b *codexBorrowNodeBinding) error { + if b == nil || b.owner != s || s.proxies == nil { + return ErrCodexGatewayBorrowChanged + } + s.mu.Lock() + current := s.config.Enabled && s.config.RotateNodes && s.revision == b.revision && s.revisionCtx.Err() == nil && slices.Contains(s.config.NodeProxyIDs, b.node.id) + s.mu.Unlock() + if !current { + return ErrCodexGatewayBorrowChanged + } + if b.node.id != 0 { + p, err := s.proxies.GetByID(ctx, b.node.id) + if err != nil { + return ErrCodexGatewayBorrowChanged + } + n, err := borrowNodeFromProxy(p) + if err != nil || n.identity != b.node.identity { + return ErrCodexGatewayBorrowChanged + } + } else if b.node.proxy != "" || b.node.identity != borrowHash("codex-borrow-server-direct") { + return ErrCodexGatewayBorrowChanged + } + if !borrowImmediateReborrow(ctx) && time.Now().Before(s.rotation.cache(b.model).blockedUntil(b.accountID, b.node.identity, b.gateway, time.Now())) { + return &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_cooling"} + } + return nil +} + +func borrowNodeBinding(req *http.Request) *codexBorrowNodeBinding { + if req == nil { + return nil + } + if p, ok := req.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation); ok && p.application != nil && p.application.node != nil { + return p.application.node + } + b, _ := req.Context().Value(codexBorrowNodeContextKey{}).(*codexBorrowNodeBinding) + return b +} + +func (s *CodexGatewayBorrowService) applyNode(req *http.Request, account *Account, model string, profile *tlsfingerprint.Profile, cachedOnly, force bool, b *codexBorrowNodeBinding) (*http.Request, *CodexGatewayBorrowApplication, error) { + if b.accountID != account.ID || b.model != model { + return req, nil, ErrCodexGatewayBorrowChanged + } + if err := s.nodeCurrent(req.Context(), b); err != nil { + return req, nil, err + } + wire, app, err := b.child.apply(req, account, model, b.node.proxy, profile, cachedOnly, force) + if app != nil { + app.node = b + app.Verification.ProxyID = b.node.id + app.Verification.ProxyName = b.node.name + app.Verification.NodeIdentity = b.node.identity + app.Verification.Gateway = b.gateway + wire = wire.WithContext(context.WithValue(wire.Context(), codexBorrowNodeContextKey{}, b)) + wire = wire.WithContext(WithHTTPUpstreamProfile(wire.Context(), HTTPUpstreamProfileOpenAIHarvest)) + } + return wire, app, err +} + +func (s *CodexGatewayBorrowService) applyRotated(req *http.Request, account *Account, model, _ string, profile *tlsfingerprint.Profile, cachedOnly, force bool, rev uint64, revisionCtx context.Context) (*http.Request, *CodexGatewayBorrowApplication, error) { + if bound := borrowNodeBinding(req); bound != nil && cachedOnly { + return s.applyNode(req, account, model, profile, true, force, bound) + } + cfg := s.ConfigSnapshot() + immediate := cfg.ImmediateReborrowOnFailure && borrowImmediateReborrow(req.Context()) && !cachedOnly + key := codexGatewayBorrowTargetKey{account.ID, model} + r := &s.rotation + cache := r.cache(model) + nodes, err := s.borrowNodes(req.Context(), cfg.NodeProxyIDs) + if err != nil { + if immediate { + return req, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "node_selection", Reason: "node_candidates_unavailable"} + } + return req, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "node_selection", Reason: "node_unavailable"} + } + if len(nodes) == 0 { + return req, nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_candidates_unavailable"} + } + // Cached-only dispatch may use the selected node, never search or probe. + if cachedOnly { + r.mu.Lock() + identity := r.active[key] + child := r.children[identity] + r.mu.Unlock() + for _, node := range nodes { + if node.identity == identity && child != nil { + child.mu.Lock() + c := child.currentCandidateLocked() + host := "" + if c != nil { + host = codexBorrowGateway(c.cookie.Value) + } + child.mu.Unlock() + return s.applyNode(req, account, model, profile, true, force, &codexBorrowNodeBinding{s, child, rev, account.ID, model, host, node}) + } + } + return req, nil, ErrCodexGatewayBorrowUnavailable + } + r.mu.Lock() + initializeErr := r.initializeLocked(rev) + r.mu.Unlock() + if initializeErr != nil { + return req, nil, initializeErr + } + flightKey := fmt.Sprintf("%d:%d:%s:%s", rev, account.ID, model, borrowRequestFingerprint(req, model, "", profile, "")) + var preparationOwner atomic.Bool + value, err := r.flights.do(req.Context(), revisionCtx, flightKey, 180*time.Second, func(ctx context.Context) (any, error) { + preparationOwner.Store(true) + limit := cfg.MaxNodeAttempts + if limit == 0 { + limit = 3 + } + seen := map[string]bool{} + attempts := 0 + transientTargetFailure := false + var lastErr error + for scanned := 0; scanned < len(nodes)+1 && attempts < limit; scanned++ { + if err := ctx.Err(); err != nil { + return nil, err + } + s.mu.Lock() + current := s.revision == rev && s.config.Enabled + s.mu.Unlock() + if !current { + return nil, ErrCodexGatewayBorrowChanged + } + r.mu.Lock() + identity := r.active[key] + retry := r.retry[key] + child := r.children[identity] + r.mu.Unlock() + if pausedUntil := cache.sourceSearchPausedUntil(account.ID, time.Now()); !cfg.WaitForQualifiedRoute && pausedUntil.After(retry) { + retry = pausedUntil + } + if !immediate && time.Now().Before(retry) { + return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_search_cooling", RetryAfter: &retry} + } + var node codexBorrowNode + if scanned == 0 && child != nil { + child.mu.Lock() + candidate := child.currentCandidateLocked() + live := candidate != nil && time.Now().Before(candidate.expires) + child.mu.Unlock() + if live || cfg.WaitForQualifiedRoute { + for _, n := range nodes { + if n.identity == identity { + node = n + break + } + } + } + } + if node.identity == "" { + r.mu.Lock() + cursor := r.cursor[key] + node = nodes[cursor%len(nodes)] + if r.cursor == nil { + r.cursor = map[codexGatewayBorrowTargetKey]int{} + } + r.cursor[key] = cursor + 1 + r.mu.Unlock() + } + if seen[node.identity] || (!immediate && time.Now().Before(cache.blockedUntil(account.ID, node.identity, "", time.Now()))) { + continue + } + allSourcesCooling := true + for _, id := range cfg.SourceAccountIDs { + if !r.sourceCache.sourceCooling(id, node.identity, time.Now()) { + allSourcesCooling = false + } + } + if !immediate && allSourcesCooling { + continue + } + seen[node.identity] = true + attempts++ + child, err := s.nodeChild(rev, cfg, node) + if err != nil { + return nil, err + } + child.mu.Lock() + candidate := child.currentCandidateLocked() + childRev, childCtx := child.revision, child.revisionCtx + previous := child.targets[key] + child.mu.Unlock() + freshSource := immediate && (candidate == nil || !time.Now().Before(candidate.expires) || identity != node.identity || (!previous.result.Success && previous.result.Reason != "")) + if freshSource || candidate == nil || !time.Now().Before(candidate.expires) { + codexBorrowUsageFromContext(ctx).recovering("reborrowing", nil, nil) + sourceCtx, sourceCancel := context.WithTimeout(ctx, 45*time.Second) + if freshSource { + sourceCtx = context.WithValue(sourceCtx, borrowFreshSourceKey{}, true) + sourceCtx = context.WithValue(sourceCtx, borrowAcquisitionTriggerKey{}, "immediate_reborrow") + } + err = child.prepareSource(sourceCtx, childRev, childCtx) + sourceCancel() + if err != nil { + lastErr = err + if ctx.Err() != nil { + return nil, ctx.Err() + } + for _, id := range cfg.SourceAccountIDs { + r.sourceCache.rejectSource(id, node.identity, time.Now()) + } + continue + } + child.mu.Lock() + candidate = child.currentCandidateLocked() + child.mu.Unlock() + } + if candidate == nil || !time.Now().Before(candidate.expires) { + lastErr = ErrCodexGatewayBorrowUnavailable + continue + } + host := codexBorrowGateway(candidate.cookie.Value) + cache.observe(node.identity, host, time.Now()) + if !immediate && time.Now().Before(cache.blockedUntil(account.ID, node.identity, host, time.Now())) { + if cfg.WaitForQualifiedRoute { + // This node rediscovered a rejected gateway. Other distinct + // nodes may still qualify within this round's existing budget. + // observe() ensures this alias is skipped on later rounds. + lastErr = &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "gateway_cooling"} + continue + } + cache.pauseSourceSearch(account.ID, time.Now()) + retry := cache.sourceSearchPausedUntil(account.ID, time.Now()) + return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "gateway_cooling", RetryAfter: &retry} + } + binding := &codexBorrowNodeBinding{s, child, rev, account.ID, model, host, node} + codexBorrowUsageFromContext(ctx).recovering("validating", nil, nil) + _, app, err := s.applyNode(req.WithContext(ctx), account, model, profile, false, force || freshSource, binding) + s.publishNodeObservation(rev, key, binding, candidate) + if err == nil && app != nil { + r.mu.Lock() + if r.revision != rev { + r.mu.Unlock() + return nil, ErrCodexGatewayBorrowChanged + } + r.active[key] = node.identity + delete(r.retry, key) + r.mu.Unlock() + return binding, nil + } + lastErr = err + if ctx.Err() != nil { + return nil, ctx.Err() + } + if immediate && !borrowRecoveryRetryable(err) { + return nil, err + } + var failure *CodexGatewayBorrowFailure + if errors.As(err, &failure) && failure.Verification != nil { + seconds := cfg.NodeCooldownSeconds + if seconds == 0 { + seconds = 3600 + } + if failure.Verification.RetryOnNextRequest { + // Preserve v16: an interrupted target probe has no negative + // cache. The next independent request runs a full new pair. + transientTargetFailure = true + } else { + cache.reject(account.ID, node.identity, host, time.Now().Add(time.Duration(seconds)*time.Second), time.Now()) + } + } else if errors.Is(err, ErrCodexGatewayBorrowChanged) { + return nil, err + } + r.mu.Lock() + if r.active[key] == node.identity { + delete(r.active, key) + } + r.mu.Unlock() + } + retry := time.Now().Add(15 * time.Second) + if transientTargetFailure || immediate { + retry = time.Time{} + } + r.mu.Lock() + if r.revision != rev { + r.mu.Unlock() + return nil, ErrCodexGatewayBorrowChanged + } + if r.retry == nil { + r.retry = map[codexGatewayBorrowTargetKey]time.Time{} + } + r.retry[key] = retry + r.mu.Unlock() + failure := &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_candidates_exhausted", RetryAfter: &retry} + if retry.IsZero() { + failure.RetryAfter = nil + } + if lastErr != nil { + failure.Detail = lastErr.Error() + var previous *CodexGatewayBorrowFailure + if errors.As(lastErr, &previous) { + failure.Verification = previous.Verification + } + } + return nil, failure + }) + if !preparationOwner.Load() { + codexBorrowUsageFromContext(req.Context()).sharedRecoveryPreparation() + } + if err != nil { + return req, nil, err + } + binding, ok := value.(*codexBorrowNodeBinding) + if !ok { + return req, nil, ErrCodexGatewayBorrowUnavailable + } + return s.applyNode(req, account, model, profile, true, false, binding) +} + +func (s *CodexGatewayBorrowService) publishNodeObservation(rev uint64, key codexGatewayBorrowTargetKey, b *codexBorrowNodeBinding, candidate *codexGatewayBorrowCandidate) { + b.child.mu.Lock() + check := b.child.targets[key] + sources := make(map[int64]CodexGatewayBorrowSourceStatus, len(b.child.sources)) + for id, row := range b.child.sources { + sources[id] = row + } + b.child.mu.Unlock() + check.result.ProxyID = b.node.id + check.result.ProxyName = b.node.name + check.result.NodeIdentity = b.node.identity + check.result.Gateway = b.gateway + s.mu.Lock() + defer s.mu.Unlock() + if s.revision != rev { + return + } + s.targets[key] = check + if s.rotationCandidates == nil { + s.rotationCandidates = map[codexGatewayBorrowTargetKey]*codexGatewayBorrowCandidate{} + } + s.rotationCandidates[key] = candidate + s.candidate = candidate + for id, row := range sources { + s.sources[id] = row + } +} + +func (r *codexBorrowRotation) snapshot(cfg CodexGatewayBorrowConfig, revision uint64) []CodexGatewayBorrowNodeStatus { + if !cfg.RotateNodes { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + now := time.Now() + out := []CodexGatewayBorrowNodeStatus{} + for _, id := range cfg.TargetAccountIDs { + for _, model := range cfg.Models { + for identity, child := range r.children { + if child.node == nil { + continue + } + node := child.node + if !slices.Contains(cfg.NodeProxyIDs, node.id) { + continue + } + row := CodexGatewayBorrowNodeStatus{AccountID: id, Model: model, ProxyID: node.id, ProxyName: node.name, NodeIdentity: identity, State: "candidate"} + child.mu.Lock() + candidate := child.currentCandidateLocked() + if candidate != nil { + expires := candidate.expires + row.ExpiresAt = &expires + } + row.Reason = child.targets[codexGatewayBorrowTargetKey{id, model}].result.Reason + child.mu.Unlock() + if cache := r.caches[model]; cache != nil { + cache.mu.Lock() + row.Gateway = cache.hosts[identity].host + cache.mu.Unlock() + until := cache.blockedUntil(id, identity, row.Gateway, now) + if !cfg.ImmediateReborrowOnFailure && now.Before(until) { + row.State = "cooling" + row.RetryAfter = &until + } + } + if row.State != "cooling" && !cfg.ImmediateReborrowOnFailure { + until := r.sourceCache.sourcesBlockedUntil(cfg.SourceAccountIDs, identity, now) + if now.Before(until) { + row.State = "cooling" + row.Reason = "source_probe_failed" + row.RetryAfter = &until + } + } + if row.State != "cooling" && r.revision == revision && r.active[codexGatewayBorrowTargetKey{id, model}] == identity { + row.State = "selected" + if row.ExpiresAt == nil || !now.Before(*row.ExpiresAt) { + row.State = "expired" + } + } + if row.State == "candidate" && row.ExpiresAt != nil && !now.Before(*row.ExpiresAt) { + row.State = "expired" + } + if cfg.ImmediateReborrowOnFailure && row.State == "candidate" && row.Reason != "" && row.Reason != "target_probe_passed" && row.Reason != "validating" { + row.State = "rejected" + } + out = append(out, row) + } + } + } + sort.Slice(out, func(i, j int) bool { + a, b := out[i], out[j] + if a.AccountID != b.AccountID { + return a.AccountID < b.AccountID + } + if a.Model != b.Model { + return a.Model < b.Model + } + return a.ProxyID < b.ProxyID + }) + return out +} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_cache.go b/backend/internal/service/codex_gateway_borrow_rotation_cache.go new file mode 100644 index 000000000000..df211a3025ae --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_rotation_cache.go @@ -0,0 +1,165 @@ +// Adapted from ranxi2001/sub2api d977e7a5, astra_rotation_cache.go (LGPL-3.0). +// Each model owns an independent instance; the upstream cache rules are unchanged. +package service + +import ( + "sync" + "time" +) + +// Cache decisions survive settings revisions, not process restarts. They never +// contain Cookies or credentials, and are not evidence of an upstream cache TTL. +type codexBorrowRotationCache struct { + mu sync.Mutex + blocked map[codexBorrowRotationCacheKey]time.Time + hosts map[string]codexBorrowNodeHost +} +type codexBorrowRotationCacheKey struct { + target int64 + kind, value string +} +type codexBorrowNodeHost struct { + host string + expires time.Time +} + +func (c *codexBorrowRotationCache) prune(now time.Time) { + for k, v := range c.blocked { + if !now.Before(v) { + delete(c.blocked, k) + } + } + for k, v := range c.hosts { + if !now.Before(v.expires) { + delete(c.hosts, k) + } + } +} +func (c *codexBorrowRotationCache) observe(node, host string, now time.Time) { + if node == "" || host == "" { + return + } + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + if c.hosts == nil { + c.hosts = map[string]codexBorrowNodeHost{} + } + if len(c.hosts) >= 4096 { + for k := range c.hosts { + delete(c.hosts, k) + break + } + } + c.hosts[node] = codexBorrowNodeHost{host: host, expires: now.Add(24 * time.Hour)} +} +func (c *codexBorrowRotationCache) reject(target int64, node, host string, until, now time.Time) { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + if c.blocked == nil { + c.blocked = map[codexBorrowRotationCacheKey]time.Time{} + } + for kind, value := range map[string]string{"node": node, "host": host} { + if value == "" { + continue + } + k := codexBorrowRotationCacheKey{target: target, kind: kind, value: value} + // A late duplicate observation must not slide an existing quiet window. + if _, exists := c.blocked[k]; exists { + continue + } + if len(c.blocked) >= 16384 { + for old := range c.blocked { + delete(c.blocked, old) + break + } + } + c.blocked[k] = until + } +} +func (c *codexBorrowRotationCache) blockedUntil(target int64, node, host string, now time.Time) time.Time { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + if host == "" { + host = c.hosts[node].host + } + a := c.blocked[codexBorrowRotationCacheKey{target: target, kind: "node", value: node}] + b := c.blocked[codexBorrowRotationCacheKey{target: target, kind: "host", value: host}] + if b.After(a) { + return b + } + return a +} + +// Source transport/qualification failures are scoped to the donor and node, +// independently from the longer target-host quiet window. +const codexBorrowSourceFailureCooldown = time.Minute + +func (c *codexBorrowRotationCache) sourceCooling(source int64, node string, now time.Time) bool { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + return now.Before(c.blocked[codexBorrowRotationCacheKey{target: source, kind: "source", value: node}]) +} + +func (c *codexBorrowRotationCache) sourcesBlockedUntil(ids []int64, node string, now time.Time) time.Time { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + var earliest time.Time + for _, id := range ids { + until := c.blocked[codexBorrowRotationCacheKey{target: id, kind: "source", value: node}] + if !now.Before(until) { + return time.Time{} + } + if earliest.IsZero() || until.Before(earliest) { + earliest = until + } + } + return earliest +} +func (c *codexBorrowRotationCache) rejectSource(source int64, node string, now time.Time) { + if node == "" { + return + } + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + if c.blocked == nil { + c.blocked = map[codexBorrowRotationCacheKey]time.Time{} + } + key := codexBorrowRotationCacheKey{target: source, kind: "source", value: node} + if _, ok := c.blocked[key]; ok { + return + } + if len(c.blocked) >= 16384 { + for old := range c.blocked { + delete(c.blocked, old) + break + } + } + c.blocked[key] = now.Add(codexBorrowSourceFailureCooldown) +} + +// A new exit which rediscovers a cooling host must not trigger another full +// source sweep on every request. Keep the pause across settings revisions. +func (c *codexBorrowRotationCache) pauseSourceSearch(target int64, now time.Time) { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + if c.blocked == nil { + c.blocked = map[codexBorrowRotationCacheKey]time.Time{} + } + key := codexBorrowRotationCacheKey{target: target, kind: "source_search"} + if _, exists := c.blocked[key]; !exists { + c.blocked[key] = now.Add(5 * time.Minute) + } +} +func (c *codexBorrowRotationCache) sourceSearchPausedUntil(target int64, now time.Time) time.Time { + c.mu.Lock() + defer c.mu.Unlock() + c.prune(now) + return c.blocked[codexBorrowRotationCacheKey{target: target, kind: "source_search"}] +} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_http.go b/backend/internal/service/codex_gateway_borrow_rotation_http.go new file mode 100644 index 000000000000..63dc057c249b --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_rotation_http.go @@ -0,0 +1,192 @@ +package service + +import ( + "context" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" + + "github.com/tidwall/gjson" +) + +// Business failures only affect future selection. A dispatched generation is +// never replayed by node selection, including an incomplete or ambiguous stream. +func (b *codexBorrowNodeBinding) rejectBusiness(reason string, explicit bool) { + if b == nil { + return + } + s := b.owner + s.mu.Lock() + current := s.revision == b.revision + cfg := s.config + s.mu.Unlock() + if !current { + return + } + seconds := 60 + if explicit { + seconds = cfg.NodeCooldownSeconds + if seconds == 0 { + seconds = 3600 + } + } + host := "" + if explicit { + host = b.gateway + } + now := time.Now() + until := now.Add(time.Duration(seconds) * time.Second) + cache := s.rotation.cache(b.model) + cache.reject(b.accountID, b.node.identity, host, until, now) + until = cache.blockedUntil(b.accountID, b.node.identity, b.gateway, now) + key := codexGatewayBorrowTargetKey{b.accountID, b.model} + b.child.mu.Lock() + check := b.child.targets[key] + check.result.Success = false + check.result.Reason = reason + check.result.RetryOnNextRequest = !explicit + check.retryAfter = until + check.acquisition.Phase = "failed" + check.acquisition.Reason = reason + check.acquisition.RetryAfter = &until + b.child.targets[key] = check + prefix := fmt.Sprintf("%d:%s:", b.accountID, b.model) + for k := range b.child.qualifications { + if strings.HasPrefix(k, prefix) { + delete(b.child.qualifications, k) + } + } + b.child.mu.Unlock() + s.rotation.mu.Lock() + if s.rotation.active[key] == b.node.identity { + delete(s.rotation.active, key) + } + s.rotation.mu.Unlock() + s.mu.Lock() + defer s.mu.Unlock() + if s.revision == b.revision && s.targets[key].result.NodeIdentity == b.node.identity { + check.result.ProxyID = b.node.id + check.result.ProxyName = b.node.name + check.result.NodeIdentity = b.node.identity + check.result.Gateway = b.gateway + s.targets[key] = check + delete(s.rotationCandidates, key) + } +} + +func trackCodexBorrowNodeHTTP(req *http.Request, resp *http.Response, err error) (*http.Response, error) { + b := borrowNodeBinding(req) + if b == nil { + return resp, err + } + if req.Context().Err() != nil { + return resp, err + } + if err != nil || resp == nil { + b.rejectBusiness("business_transport_failed", false) + return resp, err + } + old, next := req.Header.Get("X-Codex-Turn-State"), resp.Header.Get("X-Codex-Turn-State") + explicitRejected := false + if req.URL.Path == "/backend-api/codex/responses" && old != "" && next != "" && old != next { + b.rejectBusiness("business_state_changed", true) + explicitRejected = true + } + if sent, e := req.Cookie("__oailb"); e == nil { + for _, got := range resp.Cookies() { + if got.Name == "__oailb" && (got.Value != sent.Value || got.MaxAge < 0 || (!got.Expires.IsZero() && !time.Now().Before(got.Expires))) { + b.rejectBusiness("business_route_changed", true) + explicitRejected = true + break + } + } + } + if resp.StatusCode != http.StatusOK || resp.Body == nil { + if !explicitRejected { + b.rejectBusiness("business_http_failed", false) + } + return resp, err + } + // Compact/lite JSON has its own terminal and STATE protocol. Never turn + // the absence of SSE framing on that path into a stream failure. + if req.URL.Path == "/backend-api/codex/responses" && (resp.Header.Get("Content-Type") == "" || strings.Contains(resp.Header.Get("Content-Type"), "text/event-stream")) { + resp.Body = &codexBorrowNodeBody{ReadCloser: resp.Body, binding: b, ctx: req.Context(), rejected: explicitRejected} + } + return resp, err +} + +type codexBorrowNodeBody struct { + io.ReadCloser + binding *codexBorrowNodeBinding + ctx context.Context + mu sync.Mutex + line []byte + payload []byte + completed, failed bool + rejected bool + once sync.Once +} + +func (b *codexBorrowNodeBody) observe() { + line := strings.TrimSpace(string(b.line)) + b.line = nil + if line != "" { + if strings.HasPrefix(line, "data:") && len(b.payload) < 1<<20 { + b.payload = append(b.payload, strings.TrimSpace(strings.TrimPrefix(line, "data:"))...) + b.payload = append(b.payload, '\n') + } + return + } + payload := strings.TrimSpace(string(b.payload)) + b.payload = nil + if payload == "" || payload == "[DONE]" { + return + } + if !gjson.Valid(payload) { + b.failed = true + return + } + kind := gjson.Get(payload, "type").String() + switch kind { + case "response.completed", "response.done": + b.completed = gjson.Get(payload, "response.status").String() == "completed" && !gjson.Get(payload, "response.error").IsObject() + case "response.failed", "response.incomplete", "response.cancelled", "response.canceled", "error": + b.failed = true + } +} + +func (b *codexBorrowNodeBody) finish() { + b.once.Do(func() { + if !b.rejected && b.ctx.Err() == nil && (!b.completed || b.failed) { + b.binding.rejectBusiness("business_stream_incomplete", false) + } + }) +} + +func (b *codexBorrowNodeBody) Read(p []byte) (int, error) { + n, err := b.ReadCloser.Read(p) + b.mu.Lock() + defer b.mu.Unlock() + for _, ch := range p[:n] { + if ch == '\n' { + b.observe() + } else if len(b.line) < 1<<20 { + b.line = append(b.line, ch) + } + } + if err != nil { + b.finish() + } + return n, err +} + +func (b *codexBorrowNodeBody) Close() error { + err := b.ReadCloser.Close() + b.mu.Lock() + defer b.mu.Unlock() + b.finish() + return err +} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_test.go b/backend/internal/service/codex_gateway_borrow_rotation_test.go new file mode 100644 index 000000000000..454da47b8c95 --- /dev/null +++ b/backend/internal/service/codex_gateway_borrow_rotation_test.go @@ -0,0 +1,423 @@ +package service + +import ( + "context" + "encoding/base64" + "fmt" + "io" + "net/http" + "strings" + "sync" + "testing" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" + "github.com/stretchr/testify/require" + "github.com/tidwall/gjson" +) + +type borrowNodeProxies struct { + ProxyRepository + rows map[int64]Proxy +} + +func (p *borrowNodeProxies) GetByID(_ context.Context, id int64) (*Proxy, error) { + v, ok := p.rows[id] + if !ok { + return nil, ErrProxyNotFound + } + return &v, nil +} +func (p *borrowNodeProxies) ListByIDs(_ context.Context, ids []int64) ([]Proxy, error) { + out := []Proxy{} + for _, id := range ids { + if p, ok := p.rows[id]; ok { + out = append(out, p) + } + } + return out, nil +} + +func borrowNodeCookie(host string) string { + return "header." + base64.RawURLEncoding.EncodeToString([]byte(`{"host":"`+host+`"}`)) + ".signature" +} + +func nodeFixture(t *testing.T, limit int, decide func(int64, string) bool) (*CodexGatewayBorrowService, *[]string, *borrowNodeProxies) { + t.Helper() + var calls []string + var mu sync.Mutex + proxies := &borrowNodeProxies{rows: map[int64]Proxy{}} + for i := int64(10); i <= 13; i++ { + proxies.rows[i] = Proxy{ID: i, Name: fmt.Sprintf("node-%d", i), Host: fmt.Sprintf("node-%d.invalid", i), Port: 8080, Protocol: "http", Status: StatusActive} + } + s := newBorrowCoreTest(t, func(req *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { + mu.Lock() + defer mu.Unlock() + var pid int64 + for k, p := range proxies.rows { + if p.URL() == proxy { + pid = k + } + } + require.NotZero(t, pid, "must select a configured node, never the account proxy") + require.Equal(t, HTTPUpstreamProfileOpenAIHarvest, HTTPUpstreamProfileFromContext(req.Context())) + body := borrowCoreBody(t, req) + model := gjson.GetBytes(body, "model").String() + calls = append(calls, fmt.Sprintf("%d:%d:%s", pid, id, model)) + host := fmt.Sprintf("chat.gateway.unified-%d.api.openai.com", pid) + if id == 1 { + return borrowCoreResponse("gpt-6-astra", "OK", "source-state", "__oailb="+borrowNodeCookie(host)+"; Path=/; Secure; Max-Age=230"), nil + } + require.Equal(t, int64(2), id) + require.Equal(t, "Bearer synthetic-token", req.Header.Get("Authorization")) + c, err := req.Cookie("__oailb") + require.NoError(t, err) + require.Equal(t, borrowNodeCookie(host), c.Value) + state := "first-state" + if req.Header.Get("X-Codex-Turn-State") != "" { + state = "" + if !decide(pid, model) { + state = "different-state" + } + } + return borrowCoreResponse(model, "OK", state), nil + }, borrowCoreAccount(1), borrowCoreAccount(2), borrowCoreAccount(3)) + s.proxies = proxies + s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6-astra", "gpt-6.1-sol"}, RotateNodes: true, NodeProxyIDs: []int64{10, 11, 12, 13}, MaxNodeAttempts: limit}, false) + return s, &calls, proxies +} + +func nodeRequest(t *testing.T, s *CodexGatewayBorrowService, model string) (*Account, *http.Request) { + t.Helper() + a, r, _, err := s.accountTemplate(t.Context(), 2, model) + require.NoError(t, err) + r.Body = io.NopCloser(strings.NewReader("business body remains unread")) + return a, r +} + +func TestCodexBorrowNodeSelectionPinsBusinessAndCoolsFailure(t *testing.T) { + s, calls, _ := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 11 }) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, app, err := s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(11), app.Verification.ProxyID) + require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "11:1:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6-astra"}, *calls) + data, err := io.ReadAll(wire.Body) + require.NoError(t, err) + require.Equal(t, "business body remains unread", string(data)) + require.Empty(t, req.Header.Get("Cookie")) + accounts, ok := s.accounts.(*borrowCoreAccounts) + require.True(t, ok) + require.Nil(t, accounts.rows[1].ProxyID) + require.Nil(t, accounts.rows[2].ProxyID) + _, app, err = s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(11), app.Verification.ProxyID) + require.Len(t, *calls, 6) + rows := s.Status().Targets + require.True(t, rows[0].CacheValid) + cache := s.rotation.cache("gpt-6-astra") + require.True(t, time.Now().Before(cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()))) + _, other, err := s.Apply(req, borrowCoreAccount(3), "gpt-6-astra", "ordinary-original", nil, false) + require.NoError(t, err) + require.Nil(t, other) + require.Len(t, *calls, 6) +} + +func TestCodexBorrowNodeModelsHaveIndependentVerdicts(t *testing.T) { + s, calls, _ := nodeFixture(t, 1, func(_ int64, model string) bool { return model == "gpt-6.1-sol" }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) + require.Error(t, err) + a, req = nodeRequest(t, s, "gpt-6.1-sol") + _, app, err := s.Apply(req, a, "gpt-6.1-sol", "account-original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(10), app.Verification.ProxyID) + require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6.1-sol", "10:2:gpt-6.1-sol"}, *calls) +} + +func TestCodexBorrowNodeBoundedSearchAndRevisionPreservesCooldown(t *testing.T) { + s, calls, _ := nodeFixture(t, 2, func(id int64, _ string) bool { return id == 12 }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.ErrorContains(t, err, "node_candidates_exhausted") + require.Len(t, *calls, 6) + _, _, err = s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.ErrorContains(t, err, "node_search_cooling") + require.Len(t, *calls, 6) + s.publishConfig(s.ConfigSnapshot(), false) // Same config load cannot reset cooling. + cache := s.rotation.cache("gpt-6-astra") + before := cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()) + cache.reject(2, borrowHash("http://node-10.invalid:8080"), "", time.Now().Add(2*time.Hour), time.Now()) + require.Equal(t, before, cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now())) +} + +func TestCodexBorrowNodeStatusReportsActualCoolingDeadline(t *testing.T) { + s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.Error(t, err) + status := s.Status() + row := status.Targets[0] + require.False(t, row.CacheValid) + require.NotNil(t, row.RetryAfter) + require.True(t, row.RetryAfter.After(time.Now().Add(59*time.Minute)), "must not display the child's 15-second cache as node recovery") + require.Equal(t, row.RetryAfter, row.Acquisition.RetryAfter) + require.Equal(t, status.Nodes[0].RetryAfter, row.RetryAfter) + cache := s.rotation.cache("gpt-6-astra") + cache.pauseSourceSearch(a.ID, time.Now()) + until := cache.sourceSearchPausedUntil(a.ID, time.Now()) + _, _, err = s.Apply(req, a, "gpt-6-astra", "original", nil, false) + var failure *CodexGatewayBorrowFailure + require.ErrorAs(t, err, &failure) + require.Equal(t, "node_search_cooling", failure.Reason) + require.Equal(t, &until, failure.RetryAfter, "must not expose a zero deadline for a five-minute pause") + require.Len(t, *calls, 3, "reading cooling state must not issue new probes") +} + +func TestCodexBorrowNodeStatusRejectsAProofWhoseGatewayEnteredCooling(t *testing.T) { + s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + require.True(t, s.Status().Targets[0].CacheValid) + // An older in-flight request on another node can reject the same gateway. + // The stored probe is still a historical success, but cannot authorize send. + until := time.Now().Add(time.Hour) + s.rotation.cache("gpt-6-astra").reject(a.ID, "previous-node", app.Verification.Gateway, until, time.Now()) + _, _, err = s.Apply(wire, a, "gpt-6-astra", "original", nil, true) + require.ErrorContains(t, err, "node_cooling") + row := s.Status().Targets[0] + require.False(t, row.CacheValid, "status must agree with dispatch even while a historical proof is unexpired") + require.Equal(t, "rejected", row.State) + require.Equal(t, "node_cooling", row.Reason) + require.Equal(t, &until, row.RetryAfter) +} + +func TestCodexBorrowNodeDispatchRejectsEditedProxyAndNeverReplaysStream(t *testing.T) { + s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + p := proxies.rows[10] + p.Host = "edited.invalid" + proxies.rows[10] = p + _, _, err = s.Apply(wire, a, "gpt-6-astra", "original", nil, true) + require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) + require.Len(t, *calls, 3) + p.Host = "node-10.invalid" + proxies.rows[10] = p + resp, err := trackCodexBorrowNodeHTTP(wire, &http.Response{StatusCode: 200, Header: http.Header{}, Body: io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n"))}, nil) + require.NoError(t, err) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + require.Len(t, *calls, 3, "incomplete generation must never be replayed") + require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, app.node.node.identity, "", time.Now()))) + require.False(t, s.Status().Targets[0].CacheValid) +} + +func TestCodexBorrowNodeActualSenderUsesSelectedProxyAndBusinessState(t *testing.T) { + s, calls, _ := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 11 }) + s.gateway.httpUpstream = s.upstream + a, req := nodeRequest(t, s, "gpt-6-astra") + req.Header.Set("X-Codex-Turn-State", "business-state") + req.Body = io.NopCloser(strings.NewReader(`{"model":"gpt-6-astra","stream":true,"store":false,"input":[],"reasoning":{"effort":"xhigh"},"service_tier":"default"}`)) + prepared, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved-account-proxy") + require.NoError(t, err) + resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved-account-proxy", "gpt-6-astra") + require.NoError(t, err) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + require.Equal(t, "business-state", req.Header.Get("X-Codex-Turn-State")) + require.Len(t, *calls, 7) + require.Equal(t, "11:2:gpt-6-astra", (*calls)[6]) + require.True(t, s.Status().Targets[0].CacheValid) + usage := s.Status().RecentUsage + require.Len(t, usage, 1) + require.Equal(t, int64(11), usage[0].ProxyID) + require.True(t, usage[0].Applied) +} + +func TestCodexBorrowNodeBusinessReplacementRemainsVisibleAndCoolsGateway(t *testing.T) { + s, calls, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + wire.Header.Set("X-Codex-Turn-State", "old-business-state") + resp, err := trackCodexBorrowNodeHTTP(wire, borrowCoreResponse("gpt-6-astra", "OK", "new-business-state"), nil) + require.NoError(t, err) + require.Equal(t, "new-business-state", resp.Header.Get("X-Codex-Turn-State")) + _, err = io.Copy(io.Discard, resp.Body) + require.NoError(t, err) + require.NoError(t, resp.Body.Close()) + require.Len(t, *calls, 3) + require.Greater(t, time.Until(s.rotation.cache("gpt-6-astra").blockedUntil(2, app.node.node.identity, app.node.gateway, time.Now())), 59*time.Minute) + require.False(t, s.Status().Targets[0].CacheValid) + require.Equal(t, "business_state_changed", s.Status().Targets[0].Reason) +} + +func TestCodexBorrowNodeStatusUsesEachModelsRoute(t *testing.T) { + s, _, _ := nodeFixture(t, 3, func(id int64, model string) bool { return model == "gpt-6-astra" || id == 11 }) + for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { + a, req := nodeRequest(t, s, model) + _, _, err := s.Apply(req, a, model, "original", nil, false) + require.NoError(t, err) + } + status := s.Status() + require.True(t, status.Targets[0].CacheValid) + require.True(t, status.Targets[1].CacheValid) + require.Equal(t, int64(10), status.Targets[0].ProxyID) + require.Equal(t, int64(11), status.Targets[1].ProxyID) + a, req := nodeRequest(t, s, "gpt-6-astra") + ctx, cancel := context.WithCancel(req.Context()) + cancel() + _, _, err := s.Apply(req.WithContext(ctx), a, "gpt-6-astra", "original", nil, false) + require.ErrorIs(t, err, context.Canceled) + require.True(t, s.Status().Targets[0].CacheValid, "a cancelled caller must not cool a valid node") +} + +func TestCodexBorrowNodeCoolingHostSkippedAcrossSettingsRevision(t *testing.T) { + s, calls, proxies := nodeFixture(t, 1, func(id int64, _ string) bool { return id == 12 }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.Error(t, err) + require.Len(t, *calls, 3) + cache := s.rotation.cache("gpt-6-astra") + alias := proxies.rows[11] + cache.observe(borrowHash(alias.URL()), "chat.gateway.unified-10.api.openai.com", time.Now()) + cfg := s.ConfigSnapshot() + cfg.MaxNodeAttempts = 3 + s.publishConfig(cfg, false) + _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(12), app.Verification.ProxyID) + require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "12:1:gpt-6-astra", "12:2:gpt-6-astra", "12:2:gpt-6-astra"}, *calls) +} + +func TestCodexBorrowNodeTransientFailureDoesNotCoolGatewayForAnHour(t *testing.T) { + s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) + base := s.upstream + badProxy := proxies.rows[10] + s.upstream = borrowCoreProbe(func(req *http.Request, p string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + resp, err := base.DoWithTLS(req, p, id, n, profile) + if id == 2 && p == badProxy.URL() && req.Header.Get("X-Codex-Turn-State") != "" { + resp.Body = io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n")) + } + return resp, err + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(11), app.Verification.ProxyID) + require.Len(t, *calls, 6) + cache := s.rotation.cache("gpt-6-astra") + bad := proxies.rows[10] + require.True(t, cache.blockedUntil(2, borrowHash(bad.URL()), "", time.Now()).IsZero()) + require.True(t, cache.blockedUntil(2, "another-node", "chat.gateway.unified-10.api.openai.com", time.Now()).IsZero()) +} + +func TestCodexBorrowNodeSingleTransientFailureAllowsNextIndependentPair(t *testing.T) { + s, calls, proxies := nodeFixture(t, 1, func(int64, string) bool { return true }) + cfg := s.ConfigSnapshot() + cfg.NodeProxyIDs = []int64{10} + s.publishConfig(cfg, false) + base := s.upstream + interrupted := false + badProxy := proxies.rows[10] + s.upstream = borrowCoreProbe(func(req *http.Request, p string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { + resp, err := base.DoWithTLS(req, p, id, n, profile) + if !interrupted && id == 2 && p == badProxy.URL() && req.Header.Get("X-Codex-Turn-State") != "" { + interrupted = true + resp.Body = io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n")) + } + return resp, err + }) + a, req := nodeRequest(t, s, "gpt-6-astra") + _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.Error(t, err) + require.Len(t, *calls, 3) + _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + require.True(t, app.Verification.Success) + require.Len(t, *calls, 5, "the next request needs both fresh probe shots, with no source or business replay") +} + +func TestCodexBorrowNodeSameCookieAcrossNodesCannotExtendActiveLease(t *testing.T) { + r := &codexBorrowRotation{} + first := time.Now().Add(20 * time.Second) + require.Equal(t, first, r.cookieExpiry("same-routing-value", first)) + require.Equal(t, first, r.cookieExpiry("same-routing-value", time.Now().Add(230*time.Second))) + other := time.Now().Add(230 * time.Second) + require.Equal(t, other, r.cookieExpiry("different-routing-value", other)) +} + +func TestCodexBorrowNodeSourceCoolingIsSharedButTargetVerdictsAreNot(t *testing.T) { + s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) + p := proxies.rows[10] + s.rotation.sourceCache.rejectSource(1, borrowHash(p.URL()), time.Now()) + for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { + a, req := nodeRequest(t, s, model) + _, app, err := s.Apply(req, a, model, "original", nil, false) + require.NoError(t, err) + require.Equal(t, int64(11), app.Verification.ProxyID) + } + require.Equal(t, []string{"11:1:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6.1-sol", "11:2:gpt-6.1-sol"}, *calls) +} + +func TestCodexBorrowNodeCompactJSONDoesNotBecomeAnSSEFailure(t *testing.T) { + s, _, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) + require.NoError(t, err) + wire.URL.Path = "/backend-api/codex/responses/lite" + wire.Header.Set("X-Codex-Turn-State", "full-response-state") + response := &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"application/json"}, "X-Codex-Turn-State": {"compact-state"}}, Body: io.NopCloser(strings.NewReader(`{"type":"compaction","output":[]}`))} + response, err = trackCodexBorrowNodeHTTP(wire, response, nil) + require.NoError(t, err) + _, err = io.Copy(io.Discard, response.Body) + require.NoError(t, err) + require.NoError(t, response.Body.Close()) + require.Equal(t, "compact-state", response.Header.Get("X-Codex-Turn-State")) + require.True(t, s.Status().Targets[0].CacheValid) +} + +func TestCodexBorrowNodeDirectRequiresExplicitSelection(t *testing.T) { + var calls int + s := newBorrowCoreTest(t, func(req *http.Request, p string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { + require.Empty(t, p, "explicit direct node must not use the saved account proxy") + calls++ + if id == 1 { + return borrowCoreResponse("gpt-6-astra", "OK", "source", "__oailb="+borrowNodeCookie("chat.gateway.unified-99.api.openai.com")+"; Path=/; Secure; Max-Age=230"), nil + } + state := "first" + if req.Header.Get("X-Codex-Turn-State") != "" { + state = "" + } + return borrowCoreResponse("gpt-6-astra", "OK", state), nil + }, borrowCoreAccount(1), borrowCoreAccount(2)) + s.proxies = &borrowNodeProxies{rows: map[int64]Proxy{}} + cfg := s.ConfigSnapshot() + cfg.TargetAccountIDs = []int64{2} + cfg.RotateNodes = true + cfg.NodeProxyIDs = []int64{0} + _, err := normalizeCodexGatewayBorrowConfig(cfg) + require.NoError(t, err) + s.publishConfig(cfg, false) + a, req := nodeRequest(t, s, "gpt-6-astra") + wire, app, err := s.Apply(req, a, "gpt-6-astra", "saved-proxy", nil, false) + require.NoError(t, err) + require.NotEmpty(t, app.Verification.NodeIdentity) + require.Zero(t, app.Verification.ProxyID) + require.Len(t, s.Status().Nodes, 2) + require.Equal(t, 3, calls) + _, _, err = s.Apply(wire, a, "gpt-6-astra", "saved-proxy", nil, true) + require.NoError(t, err) + require.Equal(t, 3, calls) + cfg.NodeProxyIDs = []int64{99} + s.publishConfig(cfg, false) + _, _, err = s.Apply(wire, a, "gpt-6-astra", "saved-proxy", nil, true) + require.Error(t, err) + require.Equal(t, 3, calls, "removing direct cannot silently fall back to it") +} diff --git a/backend/internal/service/codex_gateway_borrow_usage.go b/backend/internal/service/codex_gateway_borrow_usage.go index 01c74fe544d0..54420b63ea70 100644 --- a/backend/internal/service/codex_gateway_borrow_usage.go +++ b/backend/internal/service/codex_gateway_borrow_usage.go @@ -2,6 +2,7 @@ package service import ( "context" + "errors" "io" "net/http" "slices" @@ -14,25 +15,29 @@ import ( ) type CodexGatewayBorrowUsage struct { - Dispatched bool `json:"dispatched"` - FailureStage string `json:"failure_stage,omitempty"` - ClientRequestID string `json:"client_request_id,omitempty"` - GatewayRequestID string `json:"gateway_request_id,omitempty"` - AttemptCount uint64 `json:"attempt_count"` - BlockedCount uint64 `json:"blocked_count"` - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Transport string `json:"transport"` - Origin string `json:"origin"` - Applied bool `json:"applied"` - Reason string `json:"reason"` - StartedAt time.Time `json:"started_at"` - FinishedAt *time.Time `json:"finished_at,omitempty"` - Outcome string `json:"outcome"` - RequestID string `json:"request_id,omitempty"` - ReportedModel string `json:"reported_model,omitempty"` - Count uint64 `json:"count"` - AppliedCount uint64 `json:"applied_count"` + Recovery *CodexGatewayBorrowRecovery `json:"recovery,omitempty"` + ProxyID int64 `json:"proxy_id,omitempty"` + NodeIdentity string `json:"node_identity,omitempty"` + Gateway string `json:"gateway,omitempty"` + Dispatched bool `json:"dispatched"` + FailureStage string `json:"failure_stage,omitempty"` + ClientRequestID string `json:"client_request_id,omitempty"` + GatewayRequestID string `json:"gateway_request_id,omitempty"` + AttemptCount uint64 `json:"attempt_count"` + BlockedCount uint64 `json:"blocked_count"` + AccountID int64 `json:"account_id"` + Model string `json:"model"` + Transport string `json:"transport"` + Origin string `json:"origin"` + Applied bool `json:"applied"` + Reason string `json:"reason"` + StartedAt time.Time `json:"started_at"` + FinishedAt *time.Time `json:"finished_at,omitempty"` + Outcome string `json:"outcome"` + RequestID string `json:"request_id,omitempty"` + ReportedModel string `json:"reported_model,omitempty"` + Count uint64 `json:"count"` + AppliedCount uint64 `json:"applied_count"` } type codexBorrowUsageKey struct { @@ -146,6 +151,14 @@ func (t *codexBorrowUsageTracker) blocked(err error) { } t.row.FailureStage, t.row.Reason = codexBorrowFailureStage(err) t.row.Outcome, t.terminal = "blocked", true + if t.row.Recovery != nil { + copy := *t.row.Recovery + copy.Phase, copy.NextAttemptAt = "stopped", nil + if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { + copy.Phase, t.row.Outcome = "cancelled", "cancelled" + } + t.row.Recovery = © + } // Mark finished while holding the tracker lock so repeated error owners // cannot count the same preparation rejection twice. t.finished = true diff --git a/backend/internal/service/codex_gateway_borrow_ws_anchor.go b/backend/internal/service/codex_gateway_borrow_ws_anchor.go index 3b56b9dc106e..3d619b8ff1b7 100644 --- a/backend/internal/service/codex_gateway_borrow_ws_anchor.go +++ b/backend/internal/service/codex_gateway_borrow_ws_anchor.go @@ -226,6 +226,11 @@ func (s *OpenAIGatewayService) prepareCodexGatewayBorrowWSTurn(ctx context.Conte if s == nil || s.gatewayBorrow == nil || account == nil { return nil, nil } + borrowConfig := s.gatewayBorrow.ConfigSnapshot() + diagnosticMode := borrowDiagnosticFromContext(ctx) + if borrowConfig.Enabled && borrowConfig.RotateNodes && slices.Contains(borrowConfig.TargetAccountIDs, account.ID) && slices.Contains(borrowConfig.Models, model) && (diagnosticMode == nil || diagnosticMode.borrow) { + return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_selection_http_only"} + } req, err := codexGatewayBorrowWSRequest(ctx, wsURL, headers) if err != nil { return nil, err diff --git a/backend/internal/service/openai_codex_request_compression.go b/backend/internal/service/openai_codex_request_compression.go index f2ea64b334bd..a07f9e30287c 100644 --- a/backend/internal/service/openai_codex_request_compression.go +++ b/backend/internal/service/openai_codex_request_compression.go @@ -164,6 +164,16 @@ func (s *OpenAIGatewayService) doOpenAICodexUpstream(req *http.Request, account return nil, err } tracker := s.gatewayBorrow.beginUsage(wire.Context(), account.ID, model, "http", applied) + if node := borrowNodeBinding(wire); node != nil { + proxyURL = node.node.proxy + if tracker != nil { + tracker.mu.Lock() + tracker.row.ProxyID, tracker.row.NodeIdentity, tracker.row.Gateway = node.node.id, node.node.identity, node.gateway + tracker.publish(0, 0, 0) + tracker.mu.Unlock() + } + } resp, sendErr := s.httpUpstream.Do(wire, proxyURL, account.ID, account.Concurrency) + resp, sendErr = trackCodexBorrowNodeHTTP(wire, resp, sendErr) return s.gatewayBorrow.trackHTTPResponse(tracker, resp, sendErr) } diff --git a/frontend/src/api/admin/codexGatewayBorrow.ts b/frontend/src/api/admin/codexGatewayBorrow.ts index 956ec36946f5..026f54022a27 100644 --- a/frontend/src/api/admin/codexGatewayBorrow.ts +++ b/frontend/src/api/admin/codexGatewayBorrow.ts @@ -5,6 +5,12 @@ export const BORROW_MODELS = ['gpt-6-astra', 'gpt-6.1-sol'] as const export const PELICAN_BORROW_PROMPT = '创建一个HTML，内容是SVG绘制一个鹈鹕骑自行车的2D动画，你不需要任何测试' export interface CodexGatewayBorrowConfig { + immediate_reborrow_on_failure?: boolean + wait_for_qualified_route?: boolean + rotate_nodes?: boolean + node_proxy_ids?: number[] + max_node_attempts?: number + node_cooldown_seconds?: number enabled: boolean source_account_ids: number[] target_account_ids: number[] @@ -12,6 +18,8 @@ export interface CodexGatewayBorrowConfig { } export interface BorrowSourceStatus { + proxy_id?: number + node_identity?: string account_id: number state: string reason: string @@ -27,6 +35,8 @@ export interface BorrowAcquisition { } export interface BorrowTargetStatus extends BorrowSourceStatus { + node_identity?: string + gateway?: string retry_on_next_request?: boolean failed_shot?: string failure_kind?: string @@ -49,6 +59,8 @@ export interface BorrowTargetStatus extends BorrowSourceStatus { } export interface CodexGatewayBorrowUsage { + recovery?: { phase: 'searching' | 'reborrowing' | 'validating' | 'cooling' | 'recovered' | 'cancelled' | 'stopped'; started_at: string; next_attempt_at?: string; rounds: number; source_requests: number; probe_requests: number; shared_preparation?: boolean; last_stage?: string; last_reason?: string; last_error?: string } + proxy_id?: number; node_identity?: string; gateway?: string account_id: number; model: string; transport: string; origin: string applied: boolean; reason: string; started_at: string; finished_at?: string outcome: string; request_id?: string; reported_model?: string; count: number; applied_count: number @@ -70,6 +82,7 @@ export interface CodexBorrowDiagnosticEvent { export interface CodexBorrowDiagnosticRequest { account_id: number; model: string; transport: 'http' | 'ws'; request_limit?: number; scenario?: 'codex_session'; mode?: 'ordinary' | 'borrowed'; service_tier?: string } export interface CodexGatewayBorrowStatus { + nodes?: Array<{ account_id: number; model: string; proxy_id: number; proxy_name: string; node_identity: string; gateway?: string; state: string; retry_after?: string; expires_at?: string; reason?: string }> acquisition?: BorrowAcquisition setup?: { state: string; phase: string; account_id: number; model: string; completed: number; total: number; failed: number; started_at: string; finished_at?: string; error?: string } recent_usage?: CodexGatewayBorrowUsage[] @@ -80,6 +93,9 @@ export interface CodexGatewayBorrowStatus { preparing: boolean config: CodexGatewayBorrowConfig candidate?: { + proxy_id?: number + node_identity?: string + gateway?: string source_account_id: number expires_at: string remaining_seconds: number @@ -91,6 +107,10 @@ export interface CodexGatewayBorrowStatus { } export interface BorrowVerification { + proxy_id?: number + proxy_name?: string + node_identity?: string + gateway?: string retry_on_next_request?: boolean failed_shot?: string failure_kind?: string diff --git a/frontend/src/components/admin/codex/CodexBorrowActivity.vue b/frontend/src/components/admin/codex/CodexBorrowActivity.vue index e67a238adbf0..82741fe173e1 100644 --- a/frontend/src/components/admin/codex/CodexBorrowActivity.vue +++ b/frontend/src/components/admin/codex/CodexBorrowActivity.vue @@ -4,6 +4,15 @@

{{ text('最近实际请求', 'Recent actual requests') }}

{{ text('打开鹈鹕测试', 'Open Pelican tests') }} +
+

{{ accountName(row.account_id) }} · {{ row.model }} · {{ recoveryPhase(row.recovery!.phase) }}

+

{{ text('恢复开始', 'Recovery started') }}: {{ time(row.recovery!.started_at) }} · {{ text('轮数', 'Rounds') }} {{ row.recovery!.rounds }}

+

{{ text('下一检查时间', 'Next eligibility check') }}: {{ time(row.recovery!.next_attempt_at) }}

+

{{ text('本请求发起的上游调用：来源采集', 'Upstream calls initiated for this request: source') }} {{ row.recovery!.source_requests }} · {{ text('目标探针', 'Target probes') }} {{ row.recovery!.probe_requests }} · {{ row.dispatched ? text('业务已发送，结果见下方记录', 'Business dispatched; see its outcome below') : text('业务尚未发送，恢复等待不代表成功', 'Business not dispatched; waiting is not success') }}

+

{{ row.recovery!.last_stage }} / {{ row.recovery!.last_reason }}

+

{{ text('复用了其他并发请求的准备结果，共享探针不重复计数。', 'Reused preparation from another concurrent request; shared probes are not counted twice.') }}

+
{{ text('最近一次恢复失败详情', 'Latest recovery failure details') }}
{{ row.recovery!.last_error }}
+

{{ text('这里记录业务从借用准备到上游完成的结果。准备失败表示业务尚未发送；诊断成功不代表业务成功。', 'Business results cover preparation through upstream completion. A preparation failure means the request was not sent; a diagnostic success does not prove business success.') }}

{{ text('还没有实际发送记录。可运行下方对照，或在正常使用后刷新状态。', 'No dispatch has been observed. Run a comparison below, or refresh after normal use.') }}

@@ -66,6 +75,8 @@ const usageGroups = computed(() => ['business', 'diagnostic'].map(origin => ({ origin, title: origin === 'business' ? text('最近业务请求', 'Recent business requests') : text('诊断请求', 'Diagnostic requests'), rows: (props.status.recent_usage || []).filter(row => row.origin === origin) })).filter(group => group.rows.length)) +const recoveryRows = computed(() => (props.status.recent_usage || []).filter(row => row.recovery)) +function recoveryPhase(value: string) { return ({ searching: text('正在寻找合格路由', 'Searching for a qualified route'), reborrowing: text('正在重新取票', 'Reacquiring a route'), validating: text('正在严格验证', 'Strict qualification in progress'), cooling: text('等待冷却', 'Waiting for cooling'), recovered: text('严格验证已恢复', 'Strict qualification recovered'), cancelled: text('已取消恢复', 'Recovery cancelled'), stopped: text('恢复已停止', 'Recovery stopped') } as Record)[value] || value } watch(pairs, value => { if (!value.some(pair => pair.key === selection.value)) selection.value = value[0]?.key || '' }, { immediate: true }) let controller: AbortController | undefined let disposed = false diff --git a/frontend/src/components/admin/codex/__tests__/CodexBorrowActivity.spec.ts b/frontend/src/components/admin/codex/__tests__/CodexBorrowActivity.spec.ts index acfcb5098b37..8ca7f78d2a68 100644 --- a/frontend/src/components/admin/codex/__tests__/CodexBorrowActivity.spec.ts +++ b/frontend/src/components/admin/codex/__tests__/CodexBorrowActivity.spec.ts @@ -19,6 +19,21 @@ function open(locale = 'zh') { beforeEach(() => { diagnose.mockReset() }) describe('Codex actual-request diagnostics', () => { + it('shows recovery waiting and cancellation without counting it as business success or invoking a diagnostic', async () => { + const wrapper = open('en') + const row = { account_id: 2, model: 'gpt-6-astra', transport: 'http', origin: 'business', applied: false, dispatched: false, reason: 'preparing', started_at: new Date().toISOString(), outcome: 'preparing', count: 0, applied_count: 0, recovery: { phase: 'cooling' as const, started_at: new Date().toISOString(), next_attempt_at: new Date(Date.now()+3600000).toISOString(), rounds: 2, source_requests: 1, probe_requests: 2, last_reason: 'target_state_changed' } } + await wrapper.setProps({ status: { ...status, recent_usage: [row] } }) + expect(wrapper.get('[data-test="borrow-recovery"]').text()).toContain('Waiting for cooling') + expect(wrapper.get('[data-test="borrow-recovery"]').text()).toContain('waiting is not success') + await wrapper.setProps({ status: { ...status, recent_usage: [{ ...row, recovery: { ...row.recovery, phase: 'reborrowing', next_attempt_at: undefined } }] } }) + expect(wrapper.get('[data-test="borrow-recovery"]').text()).toContain('Reacquiring a route') + await wrapper.setProps({ status: { ...status, recent_usage: [{ ...row, recovery: { ...row.recovery, phase: 'validating', next_attempt_at: undefined } }] } }) + expect(wrapper.get('[data-test="borrow-recovery"]').text()).toContain('Strict qualification in progress') + await wrapper.setProps({ status: { ...status, recent_usage: [{ ...row, outcome: 'cancelled', recovery: { ...row.recovery, phase: 'cancelled', next_attempt_at: undefined } }] } }) + expect(wrapper.get('[data-test="borrow-recovery"]').text()).toContain('Recovery cancelled') + expect(diagnose).not.toHaveBeenCalled() + wrapper.unmount() + }) it('keeps blocked business attempts visible ahead of successful diagnostics', () => { const wrapper = open('en') const diagnostic = { account_id: 2, model: 'gpt-6-astra', transport: 'http', origin: 'diagnostic', applied: true, dispatched: true, reason: 'borrow_applied', started_at: new Date().toISOString(), outcome: 'completed', count: 2, applied_count: 1 } diff --git a/frontend/src/views/admin/CodexGatewayBorrowStatusView.vue b/frontend/src/views/admin/CodexGatewayBorrowStatusView.vue index c75c0ffd24fe..3bd400731b42 100644 --- a/frontend/src/views/admin/CodexGatewayBorrowStatusView.vue +++ b/frontend/src/views/admin/CodexGatewayBorrowStatusView.vue @@ -43,7 +43,7 @@
{{ t('admin.codexGatewayBorrow.currentSource') }}

{{ accountName(status.candidate.source_account_id) }} #{{ status.candidate.source_account_id }}

-

{{ t('admin.codexGatewayBorrow.proxy') }}: {{ proxyLabel(accountById(status.candidate.source_account_id)) }}

+

{{ proxyTitle }}: {{ actualProxyLabel(status.candidate.proxy_id, status.candidate.source_account_id, status.candidate.node_identity) }}

{{ t('admin.codexGatewayBorrow.remainingSeconds', { seconds: candidateRemaining }) }}

{{ t('admin.codexGatewayBorrow.candidateExpired') }}

@@ -98,7 +98,7 @@
{{ t('admin.codexGatewayBorrow.rawReason') }}
{{ source.reason || '—' }}
{{ t('admin.codexGatewayBorrow.checkedAt') }}
{{ formatTime(source.checked_at) }}
{{ t('admin.codexGatewayBorrow.expiresAt') }}
{{ formatTime(source.expires_at) }}
-
{{ t('admin.codexGatewayBorrow.proxy') }}
{{ proxyLabel(accountById(source.account_id)) }}
+
{{ proxyTitle }}
{{ actualProxyLabel(source.proxy_id, source.account_id, source.node_identity) }}
{{ source.error }}
@@ -114,13 +114,14 @@

{{ t('admin.codexGatewayBorrow.accountModel') }}

{{ accountName(target.account_id) }} #{{ target.account_id }}

{{ target.model }}

-

{{ t('admin.codexGatewayBorrow.proxy') }}: {{ proxyLabel(accountById(target.account_id)) }}

+

{{ proxyTitle }}: {{ actualProxyLabel(target.proxy_id, target.account_id, target.node_identity) }}

+

{{ target.gateway }}

{{ t('admin.codexGatewayBorrow.validation') }}

- {{ t(`admin.codexGatewayBorrow.lineStates.${targetState(target)}`) }} + {{ targetLineLabel(target) }}

{{ reasonLabel(target) }}

-

{{ text('探针中断；下一次请求重新进行完整验证，通过后才发送业务。', 'The probe was interrupted. The next request runs a complete validation before business dispatch.') }}

+

{{ recoveryPending(target) ? text('验证中断；当前请求等待下一轮完整验证，通过后才发送业务。', 'Validation was interrupted. This request waits for another complete validation before business dispatch.') : text('探针中断；下一次请求重新进行完整验证，通过后才发送业务。', 'The probe was interrupted. The next request runs a complete validation before business dispatch.') }}

{{ t('admin.codexGatewayBorrow.checkedAt') }}
{{ formatTime(target.checked_at) }}
@@ -163,6 +164,16 @@

{{ t('admin.codexGatewayBorrow.noConfiguredTargets') }}

+
+

{{ text('节点选择与冷却', 'Node selection and cooling') }}

+

{{ text('按目标账号和模型分别记录。当前选中仅表示本次验证有效，不代表长期稳定或任务质量通过。', 'Recorded separately for each target and model. Selection indicates a current validation, not long-term stability or task-quality acceptance.') }}

+
+

{{ accountName(node.account_id) }} #{{ node.account_id }}

{{ node.model }}

+

{{ node.proxy_id === 0 ? text('原服务器直连', 'Server direct') : `${node.proxy_name} #${node.proxy_id}` }}

{{ node.gateway || '—' }}

+

{{ nodeStateLabel(node) }}

{{ text('冷却截止', 'Cooling until') }}: {{ formatTime(node.retry_after) }} · {{ remainingSeconds(node.retry_after) }}s

{{ borrowReasonLabel(node.reason, t) }}

+
+

{{ text('尚未尝试候选节点。', 'Candidate nodes have not been attempted.') }}

+
@@ -187,6 +198,22 @@ const lifecycle = new AbortController() const { loadAccounts, accountById, accountName, accountStateLabel, proxyLabel, wsLabel, modelMappingLabel } = useCodexBorrowInventory(lifecycle.signal) const { now, receiveServerTime, cacheUsable, remainingSeconds } = useCodexBorrowClock() const status = ref(null) +const proxyTitle = computed(() => status.value?.config.rotate_nodes ? text('实际出口', 'Actual exit') : t('admin.codexGatewayBorrow.proxy')) +function actualProxyLabel(id?: number, accountID?: number, identity?: string) { + const selected = identity ? status.value?.nodes?.find(row => row.node_identity === identity) : undefined + if (selected?.proxy_id === 0) return text('原服务器直连', 'Server direct') + if (id) { const node = status.value?.nodes?.find(row => row.proxy_id === id); return `${node?.proxy_name || ''} #${id}`.trim() } + return status.value?.config.rotate_nodes ? text('尚未选中节点', 'No node selected') : proxyLabel(accountID ? accountById(accountID) : undefined) +} +function nodeStateLabel(node: NonNullable[number]) { + if (statusStale.value) return text('上次记录，等待更新', 'Last observed; awaiting an update') + if (status.value?.config.immediate_reborrow_on_failure && node.state === 'cooling') return text('失败待重借（0 秒冷却）', 'Rejected; ready for immediate reborrow') + if (status.value?.config.immediate_reborrow_on_failure && node.state === 'rejected') return text('验证失败，可立即重借', 'Rejected; immediate reborrow available') + if (node.retry_after && remainingSeconds(node.retry_after)) return text('冷却中', 'Cooling') + if (node.state === 'selected') return node.expires_at && !remainingSeconds(node.expires_at) ? text('已过期', 'Expired') : text('当前已选', 'Selected') + if (node.state === 'expired') return text('已过期', 'Expired') + return text('待验证', 'Awaiting validation') +} const loading = ref(true) const refreshing = ref(false) const statusError = ref('') @@ -219,6 +246,10 @@ const canAct = computed(() => !!status.value?.enabled && !statusStale.value && ! const shouldPoll = computed(() => pageVisible.value && (!!status.value?.enabled || !!status.value?.preparing || !!actionBusy.value)) const preparationLabel = computed(() => { if (statusStale.value) return text('状态待刷新', 'Status needs refreshing') + const waiting = (status.value?.recent_usage || []).filter(row => !row.dispatched && !row.finished_at && ['searching', 'reborrowing', 'validating', 'cooling'].includes(row.recovery?.phase || '')) + if (waiting.some(row => row.recovery?.phase === 'reborrowing')) return text('正在重新取票，原请求仍在等待', 'Reacquiring a route; the original request is still waiting') + if (waiting.some(row => row.recovery?.phase === 'validating')) return text('正在严格验证，业务尚未发送', 'Strict qualification in progress; business not dispatched') + if (waiting.length) return waiting.some(row => row.recovery?.phase === 'searching') ? text('正在恢复合格路由，业务尚未发送', 'Recovering a qualified route; business not dispatched') : text('等待冷却，原请求仍在等待', 'Cooling; the original request is still waiting') if (status.value?.preparing || actionBusy.value === 'prepare') return t('admin.codexGatewayBorrow.preparationPending') if (actionBusy.value || serverValidating.value) return t('admin.codexGatewayBorrow.validating') if (!status.value?.enabled || !targets.value.length) return text('未配置或未启用', 'Not configured or disabled') @@ -239,6 +270,14 @@ const acquisitionLabel = computed(() => { }) function pairKey(id: number, model: string) { return `${id}:${model}` } +function recoveryPending(target: BorrowTargetStatus) { return status.value?.recent_usage?.some(row => row.account_id === target.account_id && row.model === target.model && !row.dispatched && !row.finished_at && ['searching', 'reborrowing', 'validating', 'cooling'].includes(row.recovery?.phase || '')) } +function targetLineLabel(target: BorrowTargetStatus) { + const row = status.value?.recent_usage?.find(row => row.account_id === target.account_id && row.model === target.model && !row.dispatched && !row.finished_at) + if (!statusStale.value && row?.recovery?.phase === 'reborrowing') return text('重新取票', 'Reacquiring a route') + if (!statusStale.value && row?.recovery?.phase === 'validating') return text('严格验证中', 'Strict qualification in progress') + if (!statusStale.value && row?.recovery?.phase === 'searching') return text('寻找合格路由', 'Searching for a qualified route') + return t(`admin.codexGatewayBorrow.lineStates.${targetState(target)}`) +} function formatTime(value?: string) { if (!value || value.startsWith('0001-01-01')) return '—' const date = new Date(value) @@ -247,6 +286,7 @@ function formatTime(value?: string) { function targetState(target: BorrowTargetStatus) { if (!status.value?.enabled) return 'waiting' if (statusStale.value) return 'stale' + if (recoveryPending(target)) return 'validating' const state = borrowTargetState(target, now.value) if (state !== 'ready' && actionBusy.value === pairKey(target.account_id, target.model)) return 'validating' if (state === 'waiting' && (status.value.preparing || actionBusy.value === 'prepare')) return 'validating' @@ -257,6 +297,7 @@ function reasonLabel(target: BorrowTargetStatus) { const state = targetState(target) if (state === 'stale') return text('上次读取的结果，等待更新', 'Last observed result; awaiting an update') if (state === 'expired') return borrowReasonLabel('route_expired', t) + if (target.reason === 'node_cooling') return text('该出口或上游网关正在冷却，先前验证暂不可用于发送。', 'This exit or upstream gateway is cooling; the earlier validation cannot currently authorize dispatch.') if (state === 'validating' && (actionBusy.value || target.state === 'waiting')) return borrowReasonLabel('validating', t) return borrowReasonLabel(target.reason, t) } diff --git a/frontend/src/views/admin/CodexGatewayBorrowView.vue b/frontend/src/views/admin/CodexGatewayBorrowView.vue index 1f47b651befd..4c8f300e2e97 100644 --- a/frontend/src/views/admin/CodexGatewayBorrowView.vue +++ b/frontend/src/views/admin/CodexGatewayBorrowView.vue @@ -72,6 +72,30 @@ {{ model === 'gpt-6-astra' ? 'Astra' : '6.1 Sol' }} {{ model }} +
+ {{ text('候选节点与失败冷却', 'Candidate nodes and failure cooling') }} + + +

{{ t('admin.codexGatewayBorrow.localInventoryHint') }}

@@ -100,7 +124,8 @@ import Toggle from '@/components/common/Toggle.vue' import { BORROW_MODELS, codexGatewayBorrowAPI, type CodexGatewayBorrowConfig, type CodexGatewayBorrowStatus } from '@/api/admin/codexGatewayBorrow' import { useCodexBorrowInventory } from '@/composables/useCodexBorrowUI' import { useAppStore } from '@/stores/app' -import type { AccountListItem } from '@/types' +import type { AccountListItem, Proxy } from '@/types' +import { list as listProxies } from '@/api/admin/proxies' import { extractApiErrorMessage } from '@/utils/apiError' const { t, locale } = useI18n() @@ -120,6 +145,9 @@ const savedNotice = ref(false) const loadError = ref('') const statusError = ref('') const saveError = ref('') +const proxies = ref([]) +const proxyError = ref('') +const proxiesLoaded = ref(false) const DRAFT_KEY = 'codex-gateway-borrow-draft' type AccountRole = 'source_account_ids' | 'target_account_ids' @@ -131,13 +159,40 @@ const accountSearch = ref>({ source_account_ids: '', const dirty = computed(() => !!draft.value && !!saved.value && !sameConfig(draft.value, saved.value)) const configError = computed(() => { if (!draft.value) return '' + if (draft.value.immediate_reborrow_on_failure && (!draft.value.wait_for_qualified_route || !draft.value.rotate_nodes)) return text('立即重借需要启用等待恢复和节点选择。', 'Immediate reborrow requires recovery waiting and node selection.') + if (draft.value.enabled && draft.value.wait_for_qualified_route && !draft.value.rotate_nodes) return text('等待恢复需要启用节点选择。', 'Recovery waiting requires node selection.') if (draft.value.source_account_ids.some(id => draft.value!.target_account_ids.includes(id))) return t('admin.codexGatewayBorrow.disjointRequired') if (draft.value.enabled && (!draft.value.source_account_ids.length || !draft.value.target_account_ids.length || !draft.value.models.length)) return t('admin.codexGatewayBorrow.selectionRequired') + if (draft.value.rotate_nodes && (!draft.value.node_proxy_ids?.length || !Number.isInteger(draft.value.max_node_attempts) || draft.value.max_node_attempts! < 1 || draft.value.max_node_attempts! > 10 || !Number.isInteger(draft.value.node_cooldown_seconds) || draft.value.node_cooldown_seconds! < 60 || draft.value.node_cooldown_seconds! > 86400)) return text('请选择候选节点，尝试次数须为 1–10，冷却时间须为 60–86400 秒。', 'Select candidate nodes, 1–10 attempts and a cooldown of 60–86400 seconds.') return '' }) function copyConfig(value: CodexGatewayBorrowConfig): CodexGatewayBorrowConfig { - return { enabled: value.enabled, source_account_ids: [...value.source_account_ids], target_account_ids: [...value.target_account_ids], models: [...value.models] } + const copy: CodexGatewayBorrowConfig = { enabled: value.enabled, source_account_ids: [...value.source_account_ids], target_account_ids: [...value.target_account_ids], models: [...value.models] } + if (value.wait_for_qualified_route) copy.wait_for_qualified_route = true + if (value.immediate_reborrow_on_failure) copy.immediate_reborrow_on_failure = true + if (value.rotate_nodes || value.node_proxy_ids?.length || value.max_node_attempts || value.node_cooldown_seconds) Object.assign(copy, { rotate_nodes: !!value.rotate_nodes, node_proxy_ids: [...(value.node_proxy_ids || [])], max_node_attempts: value.max_node_attempts || 3, node_cooldown_seconds: value.node_cooldown_seconds || 3600 }) + return copy +} +async function loadNodes() { + if (proxies.value.length || lifecycle.signal.aborted) return + proxyError.value = '' + try { + const found: Proxy[] = [] + for (let page = 1; !lifecycle.signal.aborted; page++) { + const result = await listProxies(page, 100, { status: 'active' }, { signal: lifecycle.signal }) + found.push(...result.items) + if (!result.items.length || page * 100 >= result.total) break + } + if (!lifecycle.signal.aborted) { proxies.value = found; proxiesLoaded.value = true } + } catch (error) { if (!lifecycle.signal.aborted) proxyError.value = extractApiErrorMessage(error, text('候选节点读取失败', 'Could not load candidate nodes')) } +} +function initializeNodes() { + if (!draft.value?.rotate_nodes) return + draft.value.node_proxy_ids ||= [] + draft.value.max_node_attempts ||= 3 + draft.value.node_cooldown_seconds ||= 3600 + void loadNodes() } function sameConfig(left: CodexGatewayBorrowConfig, right: CodexGatewayBorrowConfig) { return JSON.stringify(copyConfig(left)) === JSON.stringify(copyConfig(right)) } function isConfig(value: unknown): value is CodexGatewayBorrowConfig { @@ -146,6 +201,11 @@ function isConfig(value: unknown): value is CodexGatewayBorrowConfig { return typeof config.enabled === 'boolean' && [config.source_account_ids, config.target_account_ids].every(ids => Array.isArray(ids) && ids.every(id => Number.isSafeInteger(id) && id > 0)) && Array.isArray(config.models) && config.models.every(model => typeof model === 'string' && (BORROW_MODELS as readonly string[]).includes(model)) + && (config.rotate_nodes === undefined || typeof config.rotate_nodes === 'boolean') + && (config.wait_for_qualified_route === undefined || typeof config.wait_for_qualified_route === 'boolean') + && (config.immediate_reborrow_on_failure === undefined || typeof config.immediate_reborrow_on_failure === 'boolean') + && (config.node_proxy_ids === undefined || (Array.isArray(config.node_proxy_ids) && config.node_proxy_ids.every(id => Number.isSafeInteger(id) && id >= 0))) + && [config.max_node_attempts, config.node_cooldown_seconds].every(value => value === undefined || (Number.isInteger(value) && value >= 0)) } function clearStoredDraft() { try { sessionStorage.removeItem(DRAFT_KEY) } catch { /* The form remains usable when storage is unavailable. */ } } function restoredDraft(base: CodexGatewayBorrowConfig): CodexGatewayBorrowConfig { @@ -170,6 +230,7 @@ function receiveConfig(value: CodexGatewayBorrowConfig, restore = false) { const nextDraft = restore ? restoredDraft(base) : copyConfig(base) saved.value = base draft.value = nextDraft + initializeNodes() if (!restore) clearStoredDraft() } function resetDraft() { diff --git a/frontend/src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts b/frontend/src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts index 3a5092f4214c..7d8c7ea19822 100644 --- a/frontend/src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts +++ b/frontend/src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts @@ -78,6 +78,58 @@ afterEach(() => { }) describe('CodexGatewayBorrowStatusView request and expiry boundaries', () => { + it('shows request-owned recovery above the last validation summary without issuing new probes', async () => { + const state = makeStatus() + state.recent_usage = [{ account_id: 2, model: 'gpt-6-astra', transport: 'http', origin: 'business', applied: false, dispatched: false, reason: 'preparing', started_at: new Date().toISOString(), outcome: 'preparing', count: 0, applied_count: 0, recovery: { phase: 'cooling', started_at: new Date().toISOString(), rounds: 1, source_requests: 0, probe_requests: 0 } }] + mocks.getStatus.mockResolvedValue(state) + const wrapper = await mountView() + expect(wrapper.get('[data-test="borrow-preparation-status"]').text()).toContain('原请求仍在等待') + expect(mocks.prepare).not.toHaveBeenCalled() + expect(mocks.verify).not.toHaveBeenCalled() + }) + it('identifies explicit server direct without presenting the saved proxy as the actual exit', async () => { + const state = makeStatus() + state.config.rotate_nodes = true + state.config.node_proxy_ids = [0] + state.targets[0].node_identity = 'direct-node' + state.nodes = [{ account_id: 2, model: 'gpt-6-astra', proxy_id: 0, proxy_name: 'Server direct', node_identity: 'direct-node', state: 'selected', expires_at: state.targets[0].expires_at }] + mocks.getStatus.mockResolvedValue(state) + const wrapper = await mountView() + expect(wrapper.get('[data-test="borrow-status-2-gpt-6-astra"] [data-test="borrow-actual-node"]').text()).toContain('原服务器直连') + expect(wrapper.get('[data-test="borrow-node-status"]').text()).not.toContain('#0') + expectNoUnrelatedCalls() + }) + it('shows reacquisition over a retained rejection without claiming validation or business success', async () => { + const state = makeStatus() + state.config.rotate_nodes = state.config.wait_for_qualified_route = state.config.immediate_reborrow_on_failure = true + state.targets[0] = { ...state.targets[0], state: 'rejected', cache_valid: false, reason: 'target_state_changed', new_ticket: true } + state.recent_usage = [{ account_id: 2, model: 'gpt-6-astra', transport: 'http', origin: 'business', applied: false, dispatched: false, reason: 'preparing', started_at: new Date().toISOString(), outcome: 'preparing', count: 0, applied_count: 0, recovery: { phase: 'reborrowing', started_at: new Date().toISOString(), rounds: 3, source_requests: 3, probe_requests: 4, last_reason: 'target_state_changed' } }] + mocks.getStatus.mockResolvedValue(state) + const wrapper = await mountView() + expect(wrapper.get('[data-test="borrow-preparation-status"]').text()).toContain('正在重新取票') + expect(wrapper.get('[data-test="borrow-status-2-gpt-6-astra"] [data-test="borrow-line-state"]').text()).toBe('重新取票') + expect(mocks.prepare).not.toHaveBeenCalled() + expect(mocks.verify).not.toHaveBeenCalled() + }) + it('shows the selected node and model-specific cooling instead of the saved account proxy', async () => { + const state = makeStatus() + state.config.rotate_nodes = true + state.config.node_proxy_ids = [43, 34] + state.targets[0].proxy_id = 43 + state.targets[0].gateway = 'chat.gateway.unified-198.api.openai.com' + state.nodes = [ + { account_id: 2, model: 'gpt-6-astra', proxy_id: 43, proxy_name: 'Selected candidate', node_identity: 'selected', gateway: state.targets[0].gateway, state: 'selected', expires_at: state.targets[0].expires_at }, + { account_id: 2, model: 'gpt-6.1-sol', proxy_id: 34, proxy_name: 'Cooling candidate', node_identity: 'cooling', state: 'cooling', retry_after: new Date(Date.now() + 3_600_000).toISOString(), reason: 'target_state_changed' } + ] + mocks.getStatus.mockResolvedValue(state) + const wrapper = await mountView() + expect(wrapper.get('[data-test="borrow-status-2-gpt-6-astra"] [data-test="borrow-actual-node"]').text()).toContain('Selected candidate #43') + expect(wrapper.get('[data-test="borrow-node-status"]').text()).toContain('冷却中') + expect(wrapper.get('[data-test="borrow-node-status"]').text()).toContain('3600s') + expect(wrapper.get('[data-test="borrow-node-status"]').text()).toContain('不代表长期稳定') + expect(mocks.prepare).not.toHaveBeenCalled() + expect(mocks.verify).not.toHaveBeenCalled() + }) it('replaces an idle green result with interrupted validation before the route expires', async () => { const initial = makeStatus() const interrupted = makeStatus() diff --git a/frontend/src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts b/frontend/src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts index c525ff0e884b..fcbaf0fd0f48 100644 --- a/frontend/src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts +++ b/frontend/src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts @@ -6,7 +6,7 @@ import type { CodexGatewayBorrowConfig, CodexGatewayBorrowStatus } from '@/api/a const mocks = vi.hoisted(() => ({ getConfig: vi.fn(), saveConfig: vi.fn(), getStatus: vi.fn(), prepare: vi.fn(), verify: vi.fn(), listTests: vi.fn(), getTest: vi.fn(), streamTests: vi.fn(), - listAccounts: vi.fn(), getAvailableModels: vi.fn(), testAccount: vi.fn(), showSuccess: vi.fn() + listAccounts: vi.fn(), listProxies: vi.fn(), getAvailableModels: vi.fn(), testAccount: vi.fn(), showSuccess: vi.fn() })) vi.mock('@/api/admin/codexGatewayBorrow', async importOriginal => ({ ...(await importOriginal()), @@ -17,6 +17,7 @@ vi.mock('@/api/admin/accounts', () => { return { accountsAPI, default: accountsAPI } }) vi.mock('@/stores/app', () => ({ useAppStore: () => ({ showSuccess: mocks.showSuccess }) })) +vi.mock('@/api/admin/proxies', () => ({ list: mocks.listProxies, default: { list: mocks.listProxies } })) vi.mock('vue-i18n', async importOriginal => ({ ...(await importOriginal()), useI18n: () => ({ locale: { value: 'zh' }, t: (key: string, params?: Record) => params ? `${key}:${JSON.stringify(params)}` : key, te: () => true }) @@ -65,10 +66,36 @@ beforeEach(() => { mocks.listAccounts.mockResolvedValue({ items: localAccounts, total: 4, page: 1, page_size: 100, pages: 1 }) mocks.getStatus.mockResolvedValue(makeStatus()) mocks.saveConfig.mockImplementation(async config => structuredClone(config)) + mocks.listProxies.mockResolvedValue({ items: [{ id: 43, name: 'Candidate A' }, { id: 34, name: 'Candidate B' }], total: 2 }) }) afterEach(() => { wrappers.splice(0).forEach(wrapper => wrapper.unmount()); sessionStorage.clear() }) describe('Codex borrow settings', () => { + it('saves zero-cooldown reborrow explicitly without changing the legacy cooldown value', async () => { + const config = { ...enabledConfig, rotate_nodes: true, wait_for_qualified_route: true, node_proxy_ids: [43, 34], max_node_attempts: 5, node_cooldown_seconds: 3600 } + const wrapper = await mountView(config) + await wrapper.get('[data-test="borrow-immediate-reborrow"]').setValue(true) + expect(wrapper.get('[data-test="borrow-node-cooldown"]').attributes('disabled')).toBeDefined() + await wrapper.get('[data-test="borrow-save"]').trigger('click') + await flushPromises() + expect(mocks.saveConfig).toHaveBeenCalledWith({ ...config, immediate_reborrow_on_failure: true }, expect.any(AbortSignal)) + expectNoModelRequests() + await wrapper.get('[data-test="borrow-wait-qualified"]').setValue(false) + expect(wrapper.text()).toContain('立即重借需要启用等待恢复和节点选择') + expect(wrapper.get('[data-test="borrow-save"]').attributes('disabled')).toBeDefined() + }) + it('preserves ordered node selection and independent bounds through a normal save', async () => { + const config = { ...enabledConfig, rotate_nodes: true, wait_for_qualified_route: true, node_proxy_ids: [43, 34], max_node_attempts: 5, node_cooldown_seconds: 3600 } + const wrapper = await mountView(config) + expect(wrapper.find('[data-test="borrow-unsaved"]').exists()).toBe(false) + expectNoModelRequests() + await wrapper.get('[data-test="borrow-node-attempts"]').setValue(4) + await wrapper.get('[data-test="borrow-save"]').trigger('click') + await flushPromises() + expect(mocks.saveConfig).toHaveBeenCalledWith({ ...config, max_node_attempts: 4 }, expect.any(AbortSignal)) + expect(wrapper.text()).toContain('#43 → #34') + expectNoModelRequests() + }) it('loads local settings and inventory only, with complete account details outside selection labels', async () => { const wrapper = await mountView(disabledConfig) expect(mocks.getConfig).toHaveBeenCalledOnce()