diff --git a/.downstream/gateway-borrow-notices.md b/.downstream/gateway-borrow-notices.md deleted file mode 100644 index efc5f3e8854a..000000000000 --- a/.downstream/gateway-borrow-notices.md +++ /dev/null @@ -1,50 +0,0 @@ -# Gateway borrowing implementation sources - -The native Cookie lease, two-shot STATE qualification and WebSocket anchor -behavior were adapted from [ranxi2001/sub2api](https://github.com/ranxi2001/sub2api), -commit `5ca3cca21eeaf4ca8a694a7f2f8f0ecd9575c549`, under LGPL-3.0. The source -files are `codex_gateway_pin.go`, `astra_routing_upstream.go`, -`astra_target_validation.go`, `openai_codex_state_probe.go`, -`openai_codex_ws_anchor.go`, `pelicanHtml.ts` and the pelican preview components. -The non-rotating source/target protocol was subsequently realigned with v2.10.3, -commit `fd1b5ee4eeb20961fbb783fa6f136a1704271e90`; retained host adaptations -are listed in [the scoped review](upstream-review-ranxi-v2.10.3.md). -The repository [LICENSE](../LICENSE) retains the LGPL-3.0 terms. - -Optional HTTP node selection additionally adapts `astra_rotation.go` and -`astra_rotation_cache.go` from v2.10.5, commit -`d977e7a5e24e5536db6ce8b8478743e0c4b11f77`. The bounded node cursor, one-hour -default target/node/gateway cooling, one-minute source cooling, duplicate-window -rule and five-minute repeated-host search pause come from that implementation. -The host adapter uses explicitly selected registered proxies instead of a managed -Mihomo listener. Sol and Astra require separate target proofs and cooldowns. -Interrupted target probes retain the host's v16 no-negative-cache behavior; -source failures still share source/node cooling across models. Business STATE -replacement and incomplete SSE invalidate future selection without replaying -the generation or changing the upstream response. - -HTML fence selection also references `web/src/lib/render.ts` from -[iwyxdxl/gpttesticu](https://github.com/iwyxdxl/gpttesticu), commit -`1b3e237c15dcdf89c4aa66d3521f8d6485d6da4a`. Its MIT notice follows. - -MIT License - -Copyright (c) 2026 iwyxdxl - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. diff --git a/.downstream/native-domains.md b/.downstream/native-domains.md index 4228f5bf524c..94a181cc81da 100644 --- a/.downstream/native-domains.md +++ b/.downstream/native-domains.md @@ -19,12 +19,10 @@ nothing. While no Codex runtime row is stored, ## Codex fingerprint simulation -The Extensions sidebar owns one Codex entry. `/admin/codex` combines borrowing -configuration and status; `/admin/codex/identity` shows client identity; and -`/admin/codex/advanced` owns manual UA/version, identifier modes and compression. -The previous runtime, fingerprint, borrowing and status URLs redirect to these -pages. Configured borrowing opens with its editor folded; a new setup opens the -three-step editor. Full saved/effective identity fields remain available in +The Extensions sidebar owns one Codex entry. `/admin/codex` redirects to +`/admin/codex/identity`; `/admin/codex/advanced` owns manual UA/version, +identifier modes and compression. Previous runtime and fingerprint URLs retain +their redirects. Full saved/effective identity fields remain available in keyboard-accessible disclosures. General settings and account create/edit/bulk forms do not submit fingerprint configuration. `GET/PUT /admin/settings/codex-fingerprint` uses `enabled`, `user_agent`, @@ -75,252 +73,6 @@ Account connection tests exclude and reject `ultra` across every capability source before dispatch; stale UI selections reset to default. Codex client orchestration catalogs retain their existing capabilities. -## Codex gateway borrowing - -The unified Codex page separates configuration, route qualification, actual -request usage and upstream completion. Pelican stays at `/admin/pelican-tests`; -its legacy comparison URL redirects there. Page reads and navigation only read -local data. Unsaved configuration stays in browser-session storage. Saving an -enabled configuration starts exactly one finite preparation, shown as queued -before the worker starts. While enabled and visible, status polls every five -seconds, including idle periods so later business failures become visible. Hidden pages stop polling and -refresh on return. Failed or stale reads cannot display current readiness; -countdowns expire routes without renewing them. - -`GET/PUT /admin/codex-gateway-borrow/config` owns one JSON setting, -`codex_gateway_borrow_config`: `enabled`, `source_account_ids`, -`target_account_ids` and `models`. Optional `rotate_nodes`, `node_proxy_ids`, -`max_node_attempts` and `node_cooldown_seconds` enable bounded HTTP node selection -only for those configured targets. Proxy ID `0` explicitly adds the current -server's direct egress as a candidate; it is never an implicit fallback. -A missing row is off with empty account -lists and the two models `gpt-6-astra` and `gpt-6.1-sol`. Only existing OpenAI -OAuth-like accounts can be selected; the source and target lists are disjoint. -The extension never rewrites account model mappings, proxies, WS switches, -groups, quota or scheduling state. Normal credential refresh remains available. - -The protocol reference is ranxi v2.10.5 (`d977e7a5`). With node selection off, -Candidates are stored per source with an active-source preference. Their maximum -lease is 230 seconds, capped by earlier upstream expiry; seeing a value again -while its lease is live does not extend it. Source acquisition uses the normal -OAuth account-test payload and OpenAI transport. Target probes use the fixed -plaintext JSON template and the host's OpenAIHarvest policy, with no copied -transport configuration and no business service tier inserted into their body. -The actual selected model is preserved: Astra follows the original; Sol is a -separate downstream model extension with independent qualification. - -With node selection enabled, the source, target probe and business request use -the same immutable registered proxy definition and fresh-connect transport. -Account bindings and shared proxy records are not changed. Inactive or expired -nodes are skipped; a changed proxy definition invalidates prepared dispatch. -Each preparation has a 180-second bound and tries at most the configured number -of distinct eligible nodes (default 3, maximum 10), with 45 seconds per source -acquisition. Explicit target rejection cools both the node and the named gateway -for that account/model (default 3600 seconds). Source failures cool the donor/node -for one minute across models. Duplicate observations cannot extend a cooling -window. Discovering a known cooling gateway pauses that model's source search -for five minutes; an exhausted search has a 15-second pause. A target transport -interruption or missing terminal does not poison the next independent request: -it still requires a fresh full probe pair. No business generation is replayed. -Cooling decisions survive setting revisions but are process-local, as upstream. -Optional `wait_for_qualified_route` defaults to false and requires node selection. -Optional `immediate_reborrow_on_failure` also defaults to false and requires both -waiting and node selection. Only pending HTTP business requests use its zero -cooldown: failed qualifications trigger fresh source acquisition and two complete -strict probes, bypassing preparation/source/node/gateway failure caches. Each -180-second bounded search starts its next round immediately, without a total -recovery deadline. Reacquiring the same cookie cannot extend an active lease; -an expired proof requires new acquisition and complete qualification as upstream. -Administrator diagnostics retain ordinary -cooling and remain bounded. Business dispatch still requires a current successful -proof and is never replayed. Missing eligible nodes or invalid configuration stop -recovery rather than spinning without network work. Acquisition and validation -phases remain separate from a recovered proof and a completed business response. -Saving immediate mode does not start background preparation; acquisition only -runs while a pending business request needs it or an administrator explicitly -invokes the existing bounded preparation/verification interface. -For full HTTP Responses business preparation it retains the original request while -eligible routes recover; administrator probes and Pelican jobs remain bounded. -Each search remains bounded to 180 seconds and the configured distinct-node limit. -No total recovery deadline is added while the caller is connected. Without -immediate reborrow the next cycle waits for node/source/gateway cooling; failures without a future eligibility time -wait at least 15 seconds. Cancellation of the last waiter cancels shared work; -configuration, target identity or account invalidation stops recovery. -Recovery first reacquires and requalifies an expired lease on the previously -qualified node; it changes nodes when that route fails or is unavailable. A -candidate that rediscovers a cooling gateway is skipped while other eligible -nodes remain in the bounded round. The gateway/node cooldown remains intact; -the upstream five-minute blanket search pause remains for non-waiting mode. -Normal token refresh -may update authentication, never business STATE, input, model, tier or history. -Recovery emits no downstream bytes before actual business response headers, so -the real STATE header remains deliverable. Existing proxy/client timeouts still -apply. Recovery status, rounds and initiated source/probe counts are distinct -from dispatched/completed business counters; shared preparations are labelled. -Source Cookie leases retain the 230-second cap and same-value no-extension rule -across selected nodes. Every model has its own target proof and current route. -The status API reports actual proxy IDs, gateway metadata, expiry and cooling; -selection is not a claim of long-term availability or model quality. Full -Responses business STATE replacement and incomplete SSE invalidate future -selection while preserving the response. Compact/lite JSON keeps its distinct -terminal/STATE protocol. Node selection does not enable WS and fails closed if -a selected target attempts borrowed WS; ordinary and unselected paths retain -their existing behavior. - -Source acquisition has a 90-second bound. The two target probes have 45 seconds -each and require completed HTTP 200 streams. The first must return STATE; the -second may omit it or return the same value. Probes clone the business headers, -clear business STATE and old body headers, preserve native session/thread/window -headers, and generate a new legacy `session_id` per shot. The second shot carries -only the first probe's STATE and target `__cflb` with the same borrowed `__oailb`. -Nothing is written back to the business session. The business request keeps its -own model, tier, cookies, STATE and history; borrowing replaces only `__oailb`. -Native HTTP business STATE echo follows the original provenance rule: unknown or expired -process-local provenance preserves the caller's STATE; a live foreign-account -record strips it. This does not relax two-shot target qualification or the -existing WS/WS-to-HTTP bridge provenance requirements. - -Observations report their connect/TLS/write/header phase and attempt count. -They may retry once only with transport trace proving a connection failure before -HTTP request headers were written. Unobserved errors, responses, written headers, -cancellation or an exhausted diagnostic budget forbid retry. Ordinary business -generations retain their existing retry policy. - -The original cache identity includes route cookie, proxy, Authorization, -ChatGPT-Account-ID, UA, Originator, Version, business STATE and TLS profile. -The host adds actual model and fingerprint policy revision. Trace IDs, native -session headers and business tier are not additional cache rules. STATE changes -therefore require fresh validation. A completed rejection or protocol failure -cools that qualification for 15 seconds. A transport interruption instead retains failed evidence with -`retry_on_next_request`: the next independent request starts a fresh complete -two-shot validation through the existing shared flight, without a negative-cache -rejection. It never retries an uncertain observation within the failed request, -admits an incomplete probe or replays a business generation. Neither failure -discards a source candidate or extends its lease. After a rejection cooldown -the next request may validate again; missing or expired candidates -are acquired on demand. There is no renewal timer, exit rotation, ordinary fallback -or business replay. Business WS keeps its existing continuation anchors. - -Status/config/history reads do not call models. Saving an enabled config starts -one finite preparation; business misses prepare on demand. There is no renewal -timer. `/prepare` and `/verify` retain their interfaces. Same-revision, same-wire -qualifications share work; cancelling one caller does not cancel other waiters, -and the final departing waiter cancels abandoned work. Distinct targets no longer -fail because another target is validating. Outbound observations share the existing -ten execution slots and serialize by account. The two target shots retain one -account lease from mint through continuation, so a different model or client -window cannot interleave another probe. Source preparation finishes before that -lease is acquired; nested shots reuse it without taking a second permit. -Preparation has a ten-minute bound, continues after a target failure, and reports -partial readiness after rechecking -expiry. Policy, credential/model-map and exit changes invalidate displayed evidence; -actual dispatch still verifies the finalized request fingerprint. - -The finalized business service tier stays in request context for diagnostics. -Probe and business inputs are reported separately. `acquisition` reports the -source trigger, stage, start/end time, failure and cooldown; each target also -reports its latest validation interval. A source candidate is not target proof. - -Status adds `setup`, `observed_since` and `recent_usage`. The last actual dispatch -and in-process counters are retained per configured target/model/protocol/purpose; -configuration publication resets this index. HTTP/SSE, pooled WS and native WS -relay observations record application independently from upstream completion. -HTTP application is checked against the immutable preparation proof and final -cookie, including after restoring the caller context. A parsed successful terminal -and visible text remain authoritative when read-ahead cleanup later reports -cancellation; that detail remains available as `read_error`. Captures are snapshotted -under a lock, and the usage index preserves the first terminal event. -Diagnostic and business requests are distinguished. This index contains no request -payloads or credential/cookie values; the existing administrator error records remain. -Attempts now start before preparation: `attempt_count` and `blocked_count` are -separate from existing `count` (dispatches) and `applied_count`. `dispatched`, -`failure_stage`, client and gateway request IDs distinguish a local rejection -from an inference failure. Original target reasons survive failure cooldowns; -Ops classifies local borrowing rejection as gateway/routing with upstream status -zero, retaining the compatible client 503 code. The event also retains probe -shot/transport evidence, next-request recovery policy and whether a rejection -came from the qualification cache. Business evidence is displayed -ahead of diagnostics. Candidate expiry never implies a failed check is usable. - -`POST /admin/codex-gateway-borrow/diagnose` streams an explicit fixed-account -comparison with `account_id`, `model`, `transport` (`http` or `ws`), and optional -`request_limit` (1–8, default 8). Its ordinary and borrowed requests use the normal -sender with observation isolation and a fixed short prompt. WS uses separate -short-lived diagnostic anchors and performs one continuation on each socket; -account/global WS settings must already permit WS. Source/target probes and all -compatibility sends count toward the same hard request limit. Events include -`phase`, `request`, `result`, `done`, and `error`; results contain original output, -completion, actual borrow application and reported model. Cancellation releases -owned work and sockets. The UI retains the current run and can download JSON; -there is no new table or migration. These observations do not prove intelligence -or physical model identity. Normal token refresh remains permitted. -Optional `scenario: codex_session` checks HTTP native client metadata, a fixed -side-effect-free echo tool call, returned STATE and complete-history continuation. -It defaults to the borrowed path to fit preparation and two business turns within -the same eight-request bound. Optional `mode` selects `ordinary` or `borrowed`; -omitting it for the original short scenario retains both-path comparison. -Optional `service_tier` exercises the effective routing tier. Results include the -specific verification (shape, actual model/tier, both completions, STATE lengths -and route replacement), failure stage and tool-round-trip result. HTTP diagnostic -history and STATE live only within that invocation, never in client sessions. -Complete `response.output_item.done` events rebuild tool calls and reasoning -items when the final successful envelope has empty output; a successful terminal -is still required. This matches native streaming continuation instead of assuming -all output is repeated inside `response.completed`. - -The HTTP-only `scenario: probe_contract` compares a fixed candidate under three -request bodies: the prior sorted-map body, the current pinned template with the -same effective tier, and the pinned template without a tier. With no effective -tier, the redundant final pair is skipped. The older serializer runs first to -avoid always confounding its position with later upstream failures. -Native identity, routing hint, model, -proxy and cookie stay fixed; each shot still gets its own legacy `session_id`. -The original probe selects Astra; the result explicitly marks Sol as a downstream -model extension. `probe_only: true` results carry per-shot completion and STATE -facts under `verification`, and their common template/cookie and plaintext-body -fingerprints under `probe_context`. Top-level business dispatch, application and -completion stay false. These comparisons never publish business qualifications -or replace real usage observations. Source acquisition and all shots share the -existing eight-request cap; the snapshot's expiry cancels remaining comparisons -without silently selecting a different candidate. Page reads do not invoke it. -Optional `comparison: encoding` keeps the same current body, tier, identity, -cookie and exit, and compares plaintext probes with the configured business -codec. It changes no global setting or business qualification; actual per-shot -encoding distinguishes an enabled zstd comparison from two plaintext samples. -The same probe-only scenario can select a configured source for one plaintext, -no-tier Astra pair against its own current live candidate. It cannot generate, -test another source model, acquire a replacement or substitute a different -candidate owner. `probe_context.subject_role` distinguishes source self-checks -from target checks. Self-check results do not qualify business traffic or mutate -the source's health/status. - -Administrator `scenario: codex_replay` accepts explicit HTTP `mode: ordinary` or -`borrowed` and `replay: {body, headers}` for controlled task/tool comparisons. -The JSON body is limited to 1 MiB, must select the requested model, stream with -`store: false`, use complete history without `previous_response_id`, and carry a -`codex-acceptance-` session in both prompt-cache and native session/thread headers. -Only native client identity/metadata/STATE headers are accepted; auth, cookies, -URL overrides and duplicate fields are rejected. The ordinary source control -must use a saved source and configured model; borrowing still requires a saved -target. Each call retains the eight-physical-attempt cap and a five-minute -generation budget. Results return actual business `turn_state` for the next -isolated replay turn; probe STATE is never returned or copied. The controller -executes synthetic tools locally and preserves full-history items, rather than -inventing tool results or executing model code in the production service. - -Ranxi reference: v2.10.5 (`d977e7a5e24e5536db6ce8b8478743e0c4b11f77`). Its -fingerprint, gateway-cookie, target-probe and automatic-setup core files are -unchanged from v2.10.2. The scoped update and retained host adaptations are -listed in [the review](upstream-review-ranxi-v2.10.5.md). OAuth search-history compatibility -is supplied by the official v0.2.15 integration. Automatic group membership, -Mihomo exit rotation and removal of harvesting attempt/concurrency bounds are -not adopted; existing account mappings, network routes and protocol switches stay -under their existing owners. - -The historical `/admin/codex-gateway-borrow/tests` endpoints retain their -cache-qualified, 90-second request contract. They share execution admission with -the independent tests described below. - ## Independent Pelican tests `/admin/pelican-tests` is a task/result workbench. The new-task composer loads @@ -349,12 +101,11 @@ complete administrator evidence. The page supports 2–4 same/cross-task compari filters, enlargement, downloads, parameter copying and failed-item retries into a new reviewed task. History, browsing, reconnecting and comparison are read-only. The fixed Pelican prompt is preserved. The actual sending path, credentials, -identity, model mapping, proxy, TLS, protocol and configured borrowing remain -owned by each account. No API Key, group policy, model score or fallback account +identity, model mapping, proxy, TLS and protocol remain owned by each account. No API Key, group policy, model score or fallback account is manufactured for a test. All batches share ten model execution slots, with one request per account. -Nested borrowing probes participate in the same coordination and normal business +Ordinary selected-account sends participate in normal business account capacity. Generation starts after preparation and admission; its default budget is 600 seconds, configurable from 60 to 1800 seconds. Queue, preparation and generation durations are recorded separately. Observed business account slots @@ -372,13 +123,28 @@ binary upstream envelopes use lossless Base64. Truncated or incomplete output is recorded without another generation or automatic repair. Result previews use short random capability URLs under -`/pelican-tests/preview/` (the historical borrowing path remains an alias), +`/pelican-tests/preview/` only, expire within 30 minutes and the task's remaining lifetime, and use an opaque script-enabled sandbox and their own response CSP. Inline animation and external scripts/styles/fonts/images are allowed. Preview documents receive no administrator token or UI bridge. Adaptation sources and retained notices are in -[gateway borrowing notices](gateway-borrow-notices.md). +[third-party notices](../THIRD_PARTY_NOTICES.md). + +## Retired gateway borrowing + +Gateway borrowing has no service, forwarding hooks, administrative interfaces, +page or compatibility routes. Migration 278 deletes only +`settings.codex_gateway_borrow_config` and changes the task execution-mode default +to `account`. Accounts, model mappings, proxies, groups and identity settings +are not changed. Historical migrations retain their filenames and checksums. + +The independent Pelican repository retains the historical +`codex_gateway_borrow_test_tasks` and `codex_gateway_borrow_test_results` table +names to preserve records. Historical `execution_mode` and `borrow_applied` +remain readable facts. New tasks always use `account` and never apply borrowing; +non-account historical tasks cannot be executed or replayed. Account-mode request +hashes retain their original field order and wire representation. ## Startup and stored data diff --git a/.downstream/ranxi-borrow-reference.json b/.downstream/ranxi-borrow-reference.json deleted file mode 100644 index a466d1d410ed..000000000000 --- a/.downstream/ranxi-borrow-reference.json +++ /dev/null @@ -1,82 +0,0 @@ -{ - "repository": "ranxi2001/sub2api", - "tag": "v2.10.5", - "commit": "d977e7a5e24e5536db6ce8b8478743e0c4b11f77", - "adaptations": [ - "Test-only identifier prefix; original function bodies.", - "Host HTTPUpstream, TLS profile, context markers, turn-state extractor and error display helpers.", - "Route container retains only cookie/expires; unused Mihomo/proxy fields are omitted.", - "Source payload calls the original Pelican builder with Astra, OAuth, fixed prompt and medium effort." - ], - "files": { - "backend/internal/service/openai_codex_turn_state.go": { - "sha256": "d5c7720ad556e68bd4505c6dcec009d8489141e5bd4b6c7984d45a38430c7440", - "declarations": { - "guardOpenAICodexTurnStateEcho": "5ce7aabcb3057409d9b746c333a14b70823e0847299f8836c6a9a5d342b938c2" - } - }, - "backend/internal/service/openai_codex_state_probe.go": { - "sha256": "ee7de916510cf6d48c94a542c731aa52fd7c4b0e8dc410b0d217229df05e99ba", - "declarations": { - "runOpenAICodexStateProbe": "eccfd327d172e198d947dcead28d77a5921e60a90c2e00e5660f56ded41a86c6", - "ProbeOpenAICodexStateRoute": "13e1cf1f0d8492d65aced50bec502350506da3c0986f56091bafda9162d15ec1", - "fail": "de12462962b42aab47fef6c33329d80648f6089388f1159d69ad5629964a1897", - "shotUsable": "9e74562f9555a5cfa40fd8c6237191ea0c2507577af5ebdf0a69f3c3507a04b4", - "openAICodexStateVerdictReason": "936f726961005cc672e1e30ab293c92f4f00f29b87c4bb103f1b6202c00f62f1", - "fireOpenAICodexStateShotRequest": "9b3b1c7cfd29e9274ed3e28034e2abd1e39ab6c7d93eec13eca2d31eb19b6256", - "openAICodexStateRouteCookies": "f1c37c5f2411b18b2e1ad72b8cf68fa5e6e25c31bbb3cbcd4f77ba633e7ae68b", - "openAICodexStateDetail": "b35df3e1ee98dfb8ad423a8f37826e5ee4ec971b3329f338d3b62617f9bd70c4", - "openAICodexStateStreamEvents": "734c354320e4a9fe9debb331cafb747040de4e7b9c53c113357eaf9a86036107", - "openAICodexStateStreamModel": "6e6e64daea561c6ef38f52472f105b4670caab7de3a2bfe039751d3126df4e44", - "openAICodexStateStreamErrorPayload": "76abf87c7a62cd22398fa7779e15bcf140a2850b26a5d507c92a07f43dea1496" - } - }, - "backend/internal/service/openai_codex_probe_response.go": { - "sha256": "cff3af2e6c01bbc7308ec8a4f9ea4596ebfd96689e38e03ef09761c9b2eb6a88", - "declarations": { - "validateCodexProbeResponse": "43b9a73bf19ca3fd04c42f85150459dfd5d3823b8f963463cdc06d1726c4daea" - } - }, - "backend/internal/service/openai_codex_ws_anchor.go": { - "sha256": "b5a3100719948d0f6c5f6a4052f7bbb70d668eaa3ce4e4480d2edd3706adabb5", - "declarations": { - "replaceCodexWSAnchorCookie": "ee9b8934f40b8a0d33b9ced3e716bd21d2cdb81bae4e345c69782231d0269f3f" - } - }, - "backend/internal/service/openai_codex_ticket.go": { - "sha256": "186ea9b390324b58613948a1371944a0eb6b6457c3aa52ff6f24fddb35d59831", - "declarations": { - "jsonString": "d595eca92f7086de35294d4fc27edf07df49d70b057642fe77c2510e909ee311", - "normalizeOpenAICodexTicketModel": "8b4bf5d56064064308040b1a1eda56b220e76f2cf078a59c6042b48a3b940b70" - } - }, - "backend/internal/repository/codex_gateway_pin.go": { - "sha256": "e9f6eb813a440282c9d39ab5d4e03a96f8862438bcfd143fa61c1ac23513bdfb", - "declarations": { - "codexGatewayCookiePathMatches": "91628f271432d6fb6426783b670e1ed08f7c72811565963d2f13255e840958bc", - "codexGatewayRouteFromResponse": "31a7c69a8a39c482f01be40b3b6909b07140400f5893cd990890346087e60d85" - } - }, - "backend/internal/service/account_test_service.go": { - "sha256": "6c90400653ac6ab22987cb717db235530a353af4dc0e00ed35fd77f876283fcd", - "declarations": { - "createOpenAITestPayload": "35451845d4ae6c6b373ec9afe050fd2606f73f7fecef795662eb3cba2e94fe86" - } - }, - "backend/internal/service/pelican_test_service.go": { - "sha256": "a601a52aae9ccef507d90b7faa1f1cc029e9e907af735e949a236883f87bf4ee", - "declarations": { - "createPelicanOpenAIPayload": "56e2c4d395beefb8f564d331ffe2afc98ed4018c1ca1ab33ab9917f07cdf0a55", - "promptOrDefault": "b807f11ad9bc4f9e6d90d6df9bb2b1a60024731553893041ad9b87f6866acdcd", - "normalizePelicanReasoningEffort": "50d726f8594b9f5ad0dcf95921b46b73ab4a8e7078873d3281352d46b5ce6934" - } - } - }, - "generated_sha256": "9ad61fb4b2a7d3230f374568ba6145635a8434579763492b292ee0ce345c1f48", - "reviewed_host_policies": { - "backend/internal/repository/astra_target_validation.go": "420b9a2f631ae382ee5455aa922b71bdde413bfbe3ea25b048d8fc0d55f9f1d5", - "backend/internal/repository/astra_routing_upstream.go": "3969048bfe953dbded5a686378eb88be52b685cfb208f9f0a97a10c52f337b13", - "backend/internal/service/astra_automatic_setup.go": "c60fc40730d667ae5b8f55d6084c9961073a107fe7ba40e68d577f722389874f", - "backend/internal/service/openai_codex_fingerprint.go": "c974262d3b0fff3c8d1eb8286078234be2905b2402245f4a46b4e9bced63eeb9" - } -} diff --git a/.github/scripts/generate_ranxi_borrow_reference.py b/.github/scripts/generate_ranxi_borrow_reference.py deleted file mode 100644 index 9e405ab67f14..000000000000 --- a/.github/scripts/generate_ranxi_borrow_reference.py +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/env python3 -"""Extract pinned upstream declarations; no hand-translated reference algorithm. - -Run from the repository root. --check validates the committed generated file and -manifest against Git objects, without network access or rewriting files. -""" -import argparse -import hashlib -import json -from pathlib import Path -import re -import subprocess - -REVISION = 'd977e7a5e24e5536db6ce8b8478743e0c4b11f77' -ROOT = Path(__file__).resolve().parents[2] -OUTPUT = ROOT / 'backend/internal/service/codex_gateway_borrow_ranxi_generated_test.go' -MANIFEST = ROOT / '.downstream/ranxi-borrow-reference.json' -FILES = { - 'backend/internal/service/openai_codex_turn_state.go': ['guardOpenAICodexTurnStateEcho'], - 'backend/internal/service/openai_codex_state_probe.go': [ - 'runOpenAICodexStateProbe', 'ProbeOpenAICodexStateRoute', 'fail', - 'shotUsable', 'openAICodexStateVerdictReason', - 'fireOpenAICodexStateShotRequest', 'openAICodexStateRouteCookies', - 'openAICodexStateDetail', 'openAICodexStateStreamEvents', - 'openAICodexStateStreamModel', 'openAICodexStateStreamErrorPayload'], - 'backend/internal/service/openai_codex_probe_response.go': ['validateCodexProbeResponse'], - 'backend/internal/service/openai_codex_ws_anchor.go': ['replaceCodexWSAnchorCookie'], - 'backend/internal/service/openai_codex_ticket.go': ['jsonString', 'normalizeOpenAICodexTicketModel'], - 'backend/internal/repository/codex_gateway_pin.go': ['codexGatewayCookiePathMatches', 'codexGatewayRouteFromResponse'], - 'backend/internal/service/account_test_service.go': ['createOpenAITestPayload'], - 'backend/internal/service/pelican_test_service.go': ['createPelicanOpenAIPayload', 'promptOrDefault', 'normalizePelicanReasoningEffort'], -} - -def read(path): - return subprocess.check_output(['git', 'show', f'{REVISION}:{path}'], cwd=ROOT) - -def extract(source, name): - pattern = r'(?m)^func (?:\([^\n]*\) )?' + re.escape(name) + r'\([^\n]*' - start = re.search(pattern, source) - if not start: - raise ValueError('missing declaration: ' + name) - end = re.search(r'(?m)^}', source[start.start():]) - if not end: - raise ValueError('unterminated declaration: ' + name) - return source[start.start():start.start() + end.end()] + '\n' - -def generate(): - manifest = {'repository': 'ranxi2001/sub2api', 'tag': 'v2.10.5', 'commit': REVISION, - 'adaptations': ['Test-only identifier prefix; original function bodies.', - 'Host HTTPUpstream, TLS profile, context markers, turn-state extractor and error display helpers.', - 'Route container retains only cookie/expires; unused Mihomo/proxy fields are omitted.', - 'Source payload calls the original Pelican builder with Astra, OAuth, fixed prompt and medium effort.'], 'files': {}} - parts = [] - names = set() - for path, functions in FILES.items(): - raw = read(path) - source = raw.decode('utf-8') - fragments = [extract(source, name) for name in functions] - names.update(functions) - if path.endswith('openai_codex_state_probe.go'): - type_names = ['OpenAICodexStateVerdict', 'OpenAICodexStateProbeResult', 'openAICodexStateShot', 'openAICodexStateProbeContextKey'] - for name in type_names: - # Multi-line structs, unlike the string alias and empty key. - start = source.index('type ' + name + ' ') - line_end = source.index('\n', start) - if source[start:line_end].endswith('{'): - end = re.search(r'(?m)^}', source[line_end:]) - fragment = source[start:line_end+end.end()]+'\n' - else: - fragment = source[start:line_end]+'\n' - fragments.append(fragment) - names.add(name) - for block in re.findall(r'(?m)^const \([\s\S]*?^\)', source): - fragments.append(block+'\n') - names.update(re.findall(r'(?m)^\t([A-Za-z]\w*)', block)) - for name, fragment in zip(functions, fragments): - manifest['files'].setdefault(path, {'sha256': hashlib.sha256(raw).hexdigest(), 'declarations': {}})['declarations'][name] = hashlib.sha256(fragment.encode()).hexdigest() - parts.extend(fragments) - names.add('codexGatewayRoute') - combined = '\n'.join(parts) - combined = re.sub(r'\b('+'|'.join(sorted(names, key=len, reverse=True))+r')\b', lambda m: 'ranxiRef'+m[0][0].upper()+m[0][1:], combined) - header = '''// Code generated by .github/scripts/generate_ranxi_borrow_reference.py; DO NOT EDIT. -// ranxi2001/sub2api v2.10.5, LGPL-3.0, commit '''+REVISION+'''. -package service -import ( - "bufio" - "bytes" - "context" - "encoding/json" - "errors" - "fmt" - "io" - "net/http" - "strings" - "time" - "github.com/google/uuid" - "github.com/gin-gonic/gin" - "github.com/Wei-Shaw/sub2api/internal/pkg/openai" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" -) -type ranxiRefCodexGatewayRoute struct { cookie http.Cookie; expires time.Time } -''' - data = subprocess.run(['gofmt'], input=(header+combined).encode(), capture_output=True, check=True).stdout.replace(b'\r\n', b'\n') - manifest['generated_sha256'] = hashlib.sha256(data).hexdigest() - # Adjacent host-dependent policies are reviewed by source hash, not replaced - # with a second hand-written implementation or claimed as executed functions. - manifest['reviewed_host_policies'] = {p: hashlib.sha256(read(p)).hexdigest() for p in [ - 'backend/internal/repository/astra_target_validation.go', - 'backend/internal/repository/astra_routing_upstream.go', - 'backend/internal/service/astra_automatic_setup.go', - 'backend/internal/service/openai_codex_fingerprint.go']} - return data, (json.dumps(manifest, indent=2)+'\n').encode() - -if __name__ == '__main__': - parser = argparse.ArgumentParser(); parser.add_argument('--check', action='store_true') - args = parser.parse_args() - generated, manifest = generate() - if args.check: - assert OUTPUT.read_bytes().replace(b'\r\n', b'\n') == generated - assert MANIFEST.read_bytes().replace(b'\r\n', b'\n') == manifest - else: - OUTPUT.write_bytes(generated); MANIFEST.write_bytes(manifest) - print('Pinned ranxi reference '+('verified' if args.check else 'generated')) diff --git a/.github/workflows/downstream-verify.yml b/.github/workflows/downstream-verify.yml index 2b04fd054199..c372917a1e53 100644 --- a/.github/workflows/downstream-verify.yml +++ b/.github/workflows/downstream-verify.yml @@ -42,15 +42,15 @@ jobs: - name: Claude stream failure and signature recovery contracts working-directory: backend run: go test -p=1 -tags unit ./internal/service ./internal/handler -run '^TestClaude(ClassifyStreamError|ThinkingSignatureError|Prelude_|Signature|StreamError|StreamFailureStatus|OpsErrorLogger)|^TestGatewayService_Forward_(PreOutputSSE|PostOutputSSE)' -count=1 - - name: Native gateway borrowing contracts + - name: Independent Pelican contracts working-directory: backend run: | python3 ../.github/scripts/prepare_test_images.py - go test -p=1 -tags unit ./internal/service ./internal/repository ./internal/handler ./internal/handler/admin -run '^Test(CodexGatewayBorrow|Pelican|AccountObservation|ObservedAccountLease)' -count=1 - go test -p=1 -tags integration ./internal/repository -run '^TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle$' -count=1 - - name: Codex preparation coordination race tests + go test -p=1 -tags unit ./internal/service ./internal/repository ./internal/handler ./internal/handler/admin ./internal/server/routes -run '^Test(Pelican|AccountObservation|ObservedAccountLease)' -count=1 + go test -p=1 -tags integration ./internal/repository -run '^TestPelicanTestRepositoryIntegrationLifecycle$' -count=1 + - name: Pelican cancellation race test working-directory: backend - run: go test -p=1 -race -tags unit ./internal/service -run '^TestCodexBorrow(Immediate|Recovery|Node|Flight|DifferentTargets|CancelledParent|DiagnoseHTTPTerminalAndDispatchEvidence)|^TestCodexGatewayBorrowRunnerCancellationStopsQueuedAndInflightAndRetainsRawOutput$' -count=1 + run: go test -p=1 -race -tags unit ./internal/service -run '^TestPelicanRunnerCancellationStopsQueuedAndInflightAndRetainsRawOutput$' -count=1 - name: Account test keepalive lifecycle race tests working-directory: backend run: go test -p=1 -race -tags unit ./internal/service -run '^TestAccountTestKeepalive' -count=1 @@ -148,15 +148,10 @@ jobs: src/components/payment/__tests__/StripePaymentInline.contrast.spec.ts \ src/views/user/__tests__/StripePaymentView.spec.ts \ src/views/admin/ops/components/__tests__/OpsErrorDetailModal.spec.ts \ - src/api/admin/__tests__/codexGatewayBorrow.spec.ts \ - src/components/admin/codex/__tests__/BorrowPelicanPreview.spec.ts \ - src/views/admin/__tests__/CodexGatewayBorrowView.spec.ts \ - src/views/admin/__tests__/CodexGatewayBorrowStatusView.spec.ts \ - src/components/admin/codex/__tests__/CodexBorrowActivity.spec.ts \ + src/components/admin/codex/__tests__/PelicanPreview.spec.ts \ src/views/admin/__tests__/CodexPelicanComparisonView.spec.ts \ src/api/admin/__tests__/pelicanTests.spec.ts \ - src/components/admin/__tests__/CodexBorrowNav.spec.ts \ - src/router/__tests__/codex-borrow-routes.spec.ts \ + src/router/__tests__/codex-routes.spec.ts \ src/components/account/__tests__/EditAccountModal.spec.ts \ src/utils/__tests__/openaiProtocolModeEdits.spec.ts \ src/components/account/__tests__/ModelWhitelistSelector.spec.ts \ diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 6ea704ba47cf..28c89f2df8b7 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -41,3 +41,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +## Pelican previews and HTML extraction + +The Pelican HTML/SVG extraction and isolated preview components retain adaptations +from ranxi2001/sub2api, commit `5ca3cca21eeaf4ca8a694a7f2f8f0ecd9575c549`, +under the repository's LGPL-3.0 license. The retained files are +`backend/internal/service/pelican_preview.go` and +`frontend/src/components/admin/codex/PelicanPreview.vue`. + +HTML fence selection also references `web/src/lib/render.ts` from +iwyxdxl/gpttesticu, commit `1b3e237c15dcdf89c4aa66d3521f8d6485d6da4a`, +under the following MIT license. + +MIT License + +Copyright (c) 2026 iwyxdxl + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/backend/cmd/server/wire.go b/backend/cmd/server/wire.go index f8fbd4264a82..fc320ef3c926 100644 --- a/backend/cmd/server/wire.go +++ b/backend/cmd/server/wire.go @@ -131,9 +131,8 @@ func provideCleanup( antigravityOAuth *service.AntigravityOAuthService, grokOAuth *service.GrokOAuthService, openAIGateway *service.OpenAIGatewayService, - codexGatewayBorrow *service.CodexGatewayBorrowService, - codexGatewayBorrowTests *service.CodexGatewayBorrowTestRunner, - codexGatewayBorrowPreview *service.CodexGatewayBorrowPreviewService, + pelicanTests *service.PelicanTestRunner, + pelicanPreview *service.PelicanPreviewService, scheduledTestRunner *service.ScheduledTestRunnerService, upstreamModelCatalogRefresh *service.UpstreamModelCatalogRefreshService, backupSvc *service.BackupService, @@ -161,15 +160,12 @@ func provideCleanup( // 应用层清理步骤可并行执行，基础设施资源（Redis/Ent）最后按顺序关闭。 parallelSteps := []cleanupStep{ - {"CodexGatewayBorrow", func() error { - if codexGatewayBorrow != nil { - codexGatewayBorrow.Stop() + {"PelicanTests", func() error { + if pelicanTests != nil { + pelicanTests.Stop() } - if codexGatewayBorrowTests != nil { - codexGatewayBorrowTests.Stop() - } - if codexGatewayBorrowPreview != nil { - codexGatewayBorrowPreview.Stop() + if pelicanPreview != nil { + pelicanPreview.Stop() } return nil }}, diff --git a/backend/cmd/server/wire_gen.go b/backend/cmd/server/wire_gen.go index ea875c897464..9eca5e2863c5 100644 --- a/backend/cmd/server/wire_gen.go +++ b/backend/cmd/server/wire_gen.go @@ -299,22 +299,16 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { auditLogService := service.ProvideAuditLogService(auditLogRepository, settingService) auditLogHandler := admin.NewAuditLogHandler(auditLogService, totpService) systemPromptHandler := admin.NewSystemPromptHandler(systemPromptService) - codexGatewayBorrowProbeUpstream := repository.NewCodexGatewayBorrowProbeUpstream(httpUpstream) - codexGatewayBorrowService, err := service.ProvideCodexGatewayBorrowService(settingRepository, accountRepository, openAIGatewayService, codexGatewayBorrowProbeUpstream, tlsFingerprintProfileService, proxyRepository) - if err != nil { - return nil, err - } - codexGatewayBorrowTestRepository := repository.NewCodexGatewayBorrowTestRepository(db) - codexGatewayBorrowTestRunner := service.ProvidePelicanTestRunner(codexGatewayBorrowTestRepository, codexGatewayBorrowService, accountRepository, accountTestService, gatewayService, geminiMessagesCompatService, concurrencyService) - codexGatewayBorrowPreviewService := service.NewCodexGatewayBorrowPreviewService(codexGatewayBorrowTestRepository) - codexGatewayBorrowHandler := admin.NewCodexGatewayBorrowHandler(codexGatewayBorrowService, codexGatewayBorrowTestRunner, codexGatewayBorrowPreviewService) pelicanTestCatalog := service.NewPelicanTestCatalog(accountRepository) - pelicanTestHandler := admin.NewPelicanTestHandler(pelicanTestCatalog, codexGatewayBorrowTestRunner, codexGatewayBorrowPreviewService) + pelicanTestRepository := repository.NewPelicanTestRepository(db) + pelicanTestRunner := service.ProvidePelicanTestRunner(pelicanTestRepository, accountRepository, accountTestService, gatewayService, geminiMessagesCompatService, concurrencyService) + pelicanPreviewService := service.NewPelicanPreviewService(pelicanTestRepository) + pelicanTestHandler := admin.NewPelicanTestHandler(pelicanTestCatalog, pelicanTestRunner, pelicanPreviewService) upstreamBillingProbeService := service.ProvideUpstreamBillingProbeService(accountRepository, accountTestService, settingService, leaderLockCache, db) openCodeGoUsageService := service.ProvideOpenCodeGoUsageService(accountRepository, httpUpstream, settingService, leaderLockCache, db) idempotencyCoordinator := service.ProvideIdempotencyCoordinator(idempotencyRepository, configConfig) claudeResetCreditService := service.ProvideClaudeResetCreditService(accountRepository, claudeTokenProvider, proxyRepository, settingService, idempotencyCoordinator, leaderLockCache) - adminHandlers := handler.ProvideAdminHandlers(dashboardHandler, adminUserHandler, groupHandler, accountHandler, accountJobHandler, adminAnnouncementHandler, dataManagementHandler, backupHandler, oAuthHandler, openAIOAuthHandler, geminiOAuthHandler, antigravityOAuthHandler, grokOAuthHandler, cnProviderHandler, proxyHandler, adminRedeemHandler, promoHandler, settingHandler, opsHandler, systemHandler, adminSubscriptionHandler, adminUsageHandler, userAttributeHandler, errorPassthroughHandler, tlsFingerprintProfileHandler, pluginHandler, adminAPIKeyHandler, scheduledTestHandler, channelHandler, channelMonitorHandler, channelMonitorRequestTemplateHandler, contentModerationHandler, promptAdminHandler, paymentHandler, affiliateHandler, complianceHandler, auditLogHandler, systemPromptHandler, codexGatewayBorrowHandler, pelicanTestHandler, upstreamBillingProbeService, ollamaCloudUsageService, openCodeGoUsageService, claudeResetCreditService) + adminHandlers := handler.ProvideAdminHandlers(dashboardHandler, adminUserHandler, groupHandler, accountHandler, accountJobHandler, adminAnnouncementHandler, dataManagementHandler, backupHandler, oAuthHandler, openAIOAuthHandler, geminiOAuthHandler, antigravityOAuthHandler, grokOAuthHandler, cnProviderHandler, proxyHandler, adminRedeemHandler, promoHandler, settingHandler, opsHandler, systemHandler, adminSubscriptionHandler, adminUsageHandler, userAttributeHandler, errorPassthroughHandler, tlsFingerprintProfileHandler, pluginHandler, adminAPIKeyHandler, scheduledTestHandler, channelHandler, channelMonitorHandler, channelMonitorRequestTemplateHandler, contentModerationHandler, promptAdminHandler, paymentHandler, affiliateHandler, complianceHandler, auditLogHandler, systemPromptHandler, pelicanTestHandler, upstreamBillingProbeService, ollamaCloudUsageService, openCodeGoUsageService, claudeResetCreditService) usageRecordWorkerPool := service.NewUsageRecordWorkerPool(configConfig) userMsgQueueCache := repository.NewUserMsgQueueCache(redisClient) userMessageQueueService := service.ProvideUserMessageQueueService(userMsgQueueCache, rpmCache, configConfig) @@ -382,7 +376,7 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { channelMonitorV2Aggregator := service.ProvideChannelMonitorV2Aggregator(channelMonitorV2Repository, db, settingService) userPlatformQuotaUsageFlusher := service.ProvideUserPlatformQuotaUsageFlusher(configConfig, billingCache, serviceUserPlatformQuotaRepository, timingWheelService) openAIQuotaAutoResetService := service.ProvideOpenAIQuotaAutoResetService(accountRepository, openAIQuotaService, rateLimitService, idempotencyCoordinator, auditLogService, settingService, leaderLockCache) - v := provideCleanup(client, redisClient, gatewayCache, opsMetricsCollector, opsAggregationService, opsAlertEvaluatorService, opsCleanupService, opsScheduledReportService, opsSystemLogSink, opsService, opsIngressRejectAggregator, apiKeyService, authCacheInvalidationWorker, schedulerSnapshotService, tokenRefreshService, accountExpiryService, cnProviderBalanceCheckService, openAICodexVersionSyncService, codexClientIdentityBackfillService, claudeCodeVersionSyncService, proxyExpiryService, subscriptionExpiryService, usageCleanupService, idempotencyCleanupService, geminiCachedContentService, batchImageCleanupService, batchImageWorkerRuntime, pricingService, emailQueueService, billingCacheService, usageRecordWorkerPool, subscriptionService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, grokOAuthService, openAIGatewayService, codexGatewayBorrowService, codexGatewayBorrowTestRunner, codexGatewayBorrowPreviewService, scheduledTestRunnerService, upstreamModelCatalogRefreshService, backupService, paymentOrderExpiryService, channelMonitorRunner, channelMonitorV2Aggregator, userPlatformQuotaUsageFlusher, upstreamBillingProbeService, ollamaCloudUsageService, openCodeGoUsageService, auditLogService, openAIQuotaAutoResetService, promptService, accountJobRuntime, pluginManager) + v := provideCleanup(client, redisClient, gatewayCache, opsMetricsCollector, opsAggregationService, opsAlertEvaluatorService, opsCleanupService, opsScheduledReportService, opsSystemLogSink, opsService, opsIngressRejectAggregator, apiKeyService, authCacheInvalidationWorker, schedulerSnapshotService, tokenRefreshService, accountExpiryService, cnProviderBalanceCheckService, openAICodexVersionSyncService, codexClientIdentityBackfillService, claudeCodeVersionSyncService, proxyExpiryService, subscriptionExpiryService, usageCleanupService, idempotencyCleanupService, geminiCachedContentService, batchImageCleanupService, batchImageWorkerRuntime, pricingService, emailQueueService, billingCacheService, usageRecordWorkerPool, subscriptionService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, grokOAuthService, openAIGatewayService, pelicanTestRunner, pelicanPreviewService, scheduledTestRunnerService, upstreamModelCatalogRefreshService, backupService, paymentOrderExpiryService, channelMonitorRunner, channelMonitorV2Aggregator, userPlatformQuotaUsageFlusher, upstreamBillingProbeService, ollamaCloudUsageService, openCodeGoUsageService, auditLogService, openAIQuotaAutoResetService, promptService, accountJobRuntime, pluginManager) application := &Application{ Server: httpServer, PromptAudit: promptService, @@ -468,9 +462,8 @@ func provideCleanup( antigravityOAuth *service.AntigravityOAuthService, grokOAuth *service.GrokOAuthService, openAIGateway *service.OpenAIGatewayService, - codexGatewayBorrow *service.CodexGatewayBorrowService, - codexGatewayBorrowTests *service.CodexGatewayBorrowTestRunner, - codexGatewayBorrowPreview *service.CodexGatewayBorrowPreviewService, + pelicanTests *service.PelicanTestRunner, + pelicanPreview *service.PelicanPreviewService, scheduledTestRunner *service.ScheduledTestRunnerService, upstreamModelCatalogRefresh *service.UpstreamModelCatalogRefreshService, backupSvc *service.BackupService, @@ -497,15 +490,12 @@ func provideCleanup( } parallelSteps := []cleanupStep{ - {"CodexGatewayBorrow", func() error { - if codexGatewayBorrow != nil { - codexGatewayBorrow.Stop() - } - if codexGatewayBorrowTests != nil { - codexGatewayBorrowTests.Stop() + {"PelicanTests", func() error { + if pelicanTests != nil { + pelicanTests.Stop() } - if codexGatewayBorrowPreview != nil { - codexGatewayBorrowPreview.Stop() + if pelicanPreview != nil { + pelicanPreview.Stop() } return nil }}, diff --git a/backend/cmd/server/wire_gen_test.go b/backend/cmd/server/wire_gen_test.go index 6f140fa27bc7..c33991cc3f36 100644 --- a/backend/cmd/server/wire_gen_test.go +++ b/backend/cmd/server/wire_gen_test.go @@ -98,9 +98,8 @@ func minimalDependencyCleanup(gatewayCache service.GatewayCache, autoReset *serv antigravityOAuthSvc, nil, // grokOAuth nil, // openAIGateway - nil, // codexGatewayBorrow - nil, // codexGatewayBorrowTests - nil, // codexGatewayBorrowPreview + nil, // pelicanTests + nil, // pelicanPreview nil, // scheduledTestRunner nil, // upstreamModelCatalogRefresh nil, // backupSvc diff --git a/backend/internal/handler/admin/codex_gateway_borrow_handler.go b/backend/internal/handler/admin/codex_gateway_borrow_handler.go deleted file mode 100644 index 5bdde206532b..000000000000 --- a/backend/internal/handler/admin/codex_gateway_borrow_handler.go +++ /dev/null @@ -1,307 +0,0 @@ -package admin - -import ( - "context" - "encoding/json" - "errors" - "fmt" - "net/http" - "strconv" - "strings" - "sync" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/response" - "github.com/Wei-Shaw/sub2api/internal/server/middleware" - "github.com/Wei-Shaw/sub2api/internal/service" - "github.com/gin-gonic/gin" - "github.com/google/uuid" -) - -type CodexGatewayBorrowHandler struct { - core *service.CodexGatewayBorrowService - runner *service.CodexGatewayBorrowTestRunner - preview *service.CodexGatewayBorrowPreviewService -} - -func NewCodexGatewayBorrowHandler(core *service.CodexGatewayBorrowService, runner *service.CodexGatewayBorrowTestRunner, preview *service.CodexGatewayBorrowPreviewService) *CodexGatewayBorrowHandler { - return &CodexGatewayBorrowHandler{core: core, runner: runner, preview: preview} -} - -func codexGatewayBorrowPrivateHeaders(c *gin.Context) { - c.Header("Cache-Control", "private, no-store") - c.Header("Referrer-Policy", "no-referrer") - c.Header("X-Content-Type-Options", "nosniff") -} - -func (h *CodexGatewayBorrowHandler) GetConfig(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - config, err := h.core.GetConfig(c.Request.Context()) - if err != nil { - response.Error(c, http.StatusInternalServerError, err.Error()) - return - } - response.Success(c, config) -} - -func (h *CodexGatewayBorrowHandler) SaveConfig(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request service.CodexGatewayBorrowConfig - raw, err := c.GetRawData() - if err == nil { - request, err = service.DecodeCodexGatewayBorrowConfig(raw) - } - if err != nil { - response.BadRequest(c, err.Error()) - return - } - config, err := h.core.SaveConfig(c.Request.Context(), request) - if err != nil { - response.BadRequest(c, err.Error()) - return - } - middleware.SetAuditExtra(c, map[string]any{"result": "codex_gateway_borrow_config_saved", "enabled": config.Enabled}) - response.Success(c, config) -} - -func (h *CodexGatewayBorrowHandler) Status(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - response.Success(c, h.core.CurrentStatus(c.Request.Context())) -} - -func (h *CodexGatewayBorrowHandler) Prepare(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - status, err := h.core.Prepare(c.Request.Context()) - if err != nil { - response.BadRequest(c, err.Error()) - return - } - response.Success(c, status) -} - -func (h *CodexGatewayBorrowHandler) Verify(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request struct { - AccountID int64 `json:"account_id"` - Model string `json:"model"` - } - if err := c.ShouldBindJSON(&request); err != nil { - response.BadRequest(c, err.Error()) - return - } - if request.AccountID <= 0 || strings.TrimSpace(request.Model) == "" { - response.BadRequest(c, "account_id and model are required") - return - } - result, err := h.core.Verify(c.Request.Context(), request.AccountID, strings.TrimSpace(request.Model)) - if err != nil { - response.BadRequest(c, err.Error()) - return - } - response.Success(c, result) -} - -func codexGatewayBorrowTestError(c *gin.Context, err error) { - status := http.StatusInternalServerError - switch { - case errors.Is(err, service.ErrCodexGatewayBorrowTestReplayConflict): - status = http.StatusConflict - case errors.Is(err, service.ErrCodexGatewayBorrowTestInvalidRequest): - status = http.StatusBadRequest - case errors.Is(err, service.ErrCodexGatewayBorrowTestExpired): - status = http.StatusGone - case errors.Is(err, service.ErrCodexGatewayBorrowTestNotFound): - status = http.StatusNotFound - } - response.Error(c, status, err.Error()) -} - -func (h *CodexGatewayBorrowHandler) StartTests(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request service.CodexGatewayBorrowTestRequest - if err := c.ShouldBindJSON(&request); err != nil { - response.BadRequest(c, err.Error()) - return - } - if _, err := uuid.Parse(strings.TrimSpace(request.ClientTaskID)); err != nil { - response.BadRequest(c, "client_task_id must be a UUID") - return - } - if len(request.Targets) == 0 { - response.BadRequest(c, "targets is required") - return - } - var createdBy int64 - if subject, ok := middleware.GetAuthSubjectFromContext(c); ok { - createdBy = subject.UserID - } - task, replayed, err := h.runner.Create(c.Request.Context(), createdBy, request) - if err != nil { - codexGatewayBorrowTestError(c, err) - return - } - middleware.SetAuditExtra(c, map[string]any{"result": "codex_gateway_borrow_manual_test", "task_id": task.ID, "replayed": replayed, "targets": task.Total}) - ctx, cancel := context.WithCancel(c.Request.Context()) - defer cancel() - c.Header("Content-Type", "text/event-stream") - c.Header("Connection", "keep-alive") - c.Header("X-Accel-Buffering", "no") - c.Status(http.StatusOK) - var writeMu sync.Mutex - emit := func(event service.CodexGatewayBorrowTestEvent) { - if ctx.Err() != nil { - return - } - if event.Task != nil { - event.Task = h.preview.DecorateTask(event.Task) - } - if event.Result != nil { - event.Result = h.preview.DecorateResult(event.Result) - } - payload, marshalErr := json.Marshal(event) - if marshalErr != nil { - cancel() - return - } - writeMu.Lock() - defer writeMu.Unlock() - if ctx.Err() != nil { - return - } - if _, writeErr := fmt.Fprintf(c.Writer, "data: %s\n\n", payload); writeErr != nil { - cancel() - return - } - c.Writer.Flush() - } - emit(service.CodexGatewayBorrowTestEvent{Type: "task_start", Task: task}) - if replayed { - emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task}) - return - } - keepAliveDone, keepAliveStopped := make(chan struct{}), make(chan struct{}) - go func() { - defer close(keepAliveStopped) - ticker := time.NewTicker(15 * time.Second) - defer ticker.Stop() - for { - select { - case <-keepAliveDone: - return - case <-ctx.Done(): - return - case <-ticker.C: - writeMu.Lock() - if _, writeErr := fmt.Fprint(c.Writer, ": keep-alive\n\n"); writeErr != nil { - cancel() - } else { - c.Writer.Flush() - } - writeMu.Unlock() - } - } - }() - runErr := h.runner.Run(ctx, task, emit) - close(keepAliveDone) - <-keepAliveStopped - if runErr != nil && ctx.Err() == nil { - // An unexpected persistence failure remains fully visible on the live - // stream even if the database cannot save that final failure observation. - task.Status, task.Error = "incomplete", runErr.Error() - emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task}) - } -} - -func (h *CodexGatewayBorrowHandler) ListTests(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - page, size := response.ParsePagination(c) - if rawSize := c.Query("size"); rawSize != "" { - if parsed, err := strconv.Atoi(rawSize); err == nil && parsed > 0 { - size = parsed - } - } - list, err := h.runner.List(c.Request.Context(), page, size) - if err != nil { - codexGatewayBorrowTestError(c, err) - return - } - response.Success(c, list) -} - -func (h *CodexGatewayBorrowHandler) GetTest(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - id := strings.TrimSpace(c.Param("id")) - if id == "last" { - list, err := h.runner.List(c.Request.Context(), 1, 1) - if err != nil { - codexGatewayBorrowTestError(c, err) - return - } - if len(list.Items) == 0 { - response.NotFound(c, "no unexpired pelican test") - return - } - id = list.Items[0].ID - } - parsed, err := uuid.Parse(id) - if err != nil { - response.BadRequest(c, "test ID must be a UUID or last") - return - } - task, err := h.runner.Get(c.Request.Context(), parsed.String()) - if err != nil { - codexGatewayBorrowTestError(c, err) - return - } - response.Success(c, h.preview.DecorateTask(task)) -} - -func (h *CodexGatewayBorrowHandler) ServePreview(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - c.Header("Content-Security-Policy", service.CodexGatewayBorrowPreviewCSP) - c.Header("X-Frame-Options", "SAMEORIGIN") - c.Header("Cross-Origin-Resource-Policy", "cross-origin") - html, err := h.preview.Resolve(c.Request.Context(), strings.TrimSpace(c.Param("cap"))) - if err != nil { - c.Status(http.StatusGone) - return - } - c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html)) -} - -// Diagnose streams explicit administrator observations; opening a page never invokes it. -func (h *CodexGatewayBorrowHandler) Diagnose(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request service.CodexBorrowDiagnosticRequest - if err := c.ShouldBindJSON(&request); err != nil { - response.BadRequest(c, err.Error()) - return - } - middleware.SetAuditExtra(c, map[string]any{"result": "codex_gateway_borrow_diagnostic", "account_id": request.AccountID, "model": request.Model, "transport": request.Transport, "scenario": request.Scenario, "comparison": request.Comparison, "mode": request.Mode, "service_tier": request.ServiceTier}) - c.Header("Content-Type", "text/event-stream") - c.Header("X-Accel-Buffering", "no") - c.Status(http.StatusOK) - var emission sync.Mutex - var observedRequests int32 - emit := func(event service.CodexBorrowDiagnosticEvent) { - emission.Lock() - defer emission.Unlock() - if event.Requests > observedRequests { - observedRequests = event.Requests - } - if event.Type == "error" { - event.Requests = observedRequests - } - if c.Request.Context().Err() != nil { - return - } - raw, err := json.Marshal(event) - if err == nil { - _, _ = fmt.Fprintf(c.Writer, "data: %s\n\n", raw) - c.Writer.Flush() - } - } - if err := h.core.Diagnose(c.Request.Context(), request, emit); err != nil { - emit(service.CodexBorrowDiagnosticEvent{Type: "error", Error: err.Error(), Limit: service.CodexBorrowDiagnosticMaxRequests}) - } -} diff --git a/backend/internal/handler/admin/codex_gateway_borrow_handler_test.go b/backend/internal/handler/admin/codex_gateway_borrow_handler_test.go deleted file mode 100644 index 8bd65bacbc36..000000000000 --- a/backend/internal/handler/admin/codex_gateway_borrow_handler_test.go +++ /dev/null @@ -1,172 +0,0 @@ -//go:build unit - -package admin - -import ( - "context" - "encoding/json" - "net/http" - "net/http/httptest" - "strings" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/config" - "github.com/Wei-Shaw/sub2api/internal/server/middleware" - "github.com/Wei-Shaw/sub2api/internal/service" - "github.com/gin-gonic/gin" - "github.com/google/uuid" - "github.com/stretchr/testify/require" -) - -func TestCodexGatewayBorrowDiagnosePreflightNeverDispatches(t *testing.T) { - gin.SetMode(gin.TestMode) - for _, test := range []struct { - name, body string - status int - }{ - {"disabled", `{"account_id":1,"model":"gpt-6-astra","transport":"http"}`, http.StatusOK}, - {"unsupported transport", `{"account_id":1,"model":"gpt-6-astra","transport":"invalid"}`, http.StatusOK}, - {"malformed", `{`, http.StatusBadRequest}, - } { - t.Run(test.name, func(t *testing.T) { - router := gin.New() - handler := NewCodexGatewayBorrowHandler(nil, nil, nil) - router.POST("/diagnose", handler.Diagnose) - response := httptest.NewRecorder() - request := httptest.NewRequest(http.MethodPost, "/diagnose", strings.NewReader(test.body)) - request.Header.Set("Content-Type", "application/json") - router.ServeHTTP(response, request) - require.Equal(t, test.status, response.Code) - require.Equal(t, "private, no-store", response.Header().Get("Cache-Control")) - if test.status == http.StatusOK { - require.Contains(t, response.Header().Get("Content-Type"), "text/event-stream") - require.Contains(t, response.Body.String(), `"type":"error"`) - require.Contains(t, response.Body.String(), `"requests":0`) - require.NotContains(t, response.Body.String(), `"type":"request"`) - } - }) - } -} - -type codexGatewayBorrowHandlerTestRepo struct { - task *service.CodexGatewayBorrowTestTask - result *service.CodexGatewayBorrowTestResult - creates int - writes int - reads int -} - -func (r *codexGatewayBorrowHandlerTestRepo) Create(context.Context, *service.CodexGatewayBorrowTestTask) (*service.CodexGatewayBorrowTestTask, bool, error) { - r.creates++ - return r.task, true, nil -} -func (r *codexGatewayBorrowHandlerTestRepo) Get(context.Context, string) (*service.CodexGatewayBorrowTestTask, error) { - r.reads++ - return r.task, nil -} -func (r *codexGatewayBorrowHandlerTestRepo) GetResult(context.Context, string) (*service.CodexGatewayBorrowTestResult, error) { - r.reads++ - return r.result, nil -} -func (r *codexGatewayBorrowHandlerTestRepo) List(context.Context, int, int) (*service.CodexGatewayBorrowTestList, error) { - return &service.CodexGatewayBorrowTestList{Items: []*service.CodexGatewayBorrowTestTask{r.task}}, nil -} -func (r *codexGatewayBorrowHandlerTestRepo) StartTask(context.Context, string, time.Time) error { - r.writes++ - return nil -} -func (r *codexGatewayBorrowHandlerTestRepo) SaveResult(context.Context, *service.CodexGatewayBorrowTestResult) error { - r.writes++ - return nil -} -func (r *codexGatewayBorrowHandlerTestRepo) FinishTask(context.Context, string, string, string, time.Time) error { - r.writes++ - return nil -} -func (r *codexGatewayBorrowHandlerTestRepo) MarkInterrupted(context.Context, time.Time) error { - return nil -} -func (r *codexGatewayBorrowHandlerTestRepo) CleanupExpired(context.Context, time.Time) error { - return nil -} - -func newCodexGatewayBorrowHandlerFixture(t *testing.T) (*CodexGatewayBorrowHandler, *codexGatewayBorrowHandlerTestRepo) { - t.Helper() - now := time.Now().UTC() - client, err := uuid.NewV7() - require.NoError(t, err) - result := &service.CodexGatewayBorrowTestResult{ID: uuid.NewString(), Status: "complete", AccountID: 42, ModelID: "gpt-6-astra", Effort: "high", RawAnswer: "", RawHTML: "", HTML: "", Error: "original upstream error token=admin-visible\x00tail", ExpiresAt: now.Add(time.Hour)} - task := &service.CodexGatewayBorrowTestTask{ID: uuid.NewString(), ClientTaskID: client.String(), Status: "complete", Total: 1, Completed: 1, CreatedAt: now, ExpiresAt: result.ExpiresAt, Results: []*service.CodexGatewayBorrowTestResult{result}} - result.TaskID = task.ID - repo := &codexGatewayBorrowHandlerTestRepo{task: task, result: result} - runner := service.NewCodexGatewayBorrowTestRunner(repo, nil, nil) - preview := service.NewCodexGatewayBorrowPreviewService(repo) - t.Cleanup(runner.Stop) - t.Cleanup(preview.Stop) - return NewCodexGatewayBorrowHandler(nil, runner, preview), repo -} - -func TestCodexGatewayBorrowHandlerPreviewUsesOnlyCapabilityAndOverridesCSP(t *testing.T) { - gin.SetMode(gin.TestMode) - h, repo := newCodexGatewayBorrowHandlerFixture(t) - decorated := h.preview.DecorateResult(repo.result) - r := gin.New() - r.Use(middleware.SecurityHeaders(config.CSPConfig{Enabled: true}, nil)) - r.GET("/api/v1/codex-gateway-borrow/preview/:cap/index.html", h.ServePreview) - r.GET("/api/v1/pelican-tests/preview/:cap/index.html", h.ServePreview) - r.GET("/", func(c *gin.Context) { c.String(http.StatusOK, "main app") }) - w := httptest.NewRecorder() - request := httptest.NewRequest(http.MethodGet, decorated.PreviewURL, nil) - request.Header.Set("Authorization", "Bearer SHOULD-NOT-APPEAR") - request.Header.Set("Cookie", "admin_jwt=SHOULD-NOT-APPEAR") - r.ServeHTTP(w, request) - require.Equal(t, http.StatusOK, w.Code) - require.Equal(t, repo.result.HTML, w.Body.String()) - require.Equal(t, service.CodexGatewayBorrowPreviewCSP, w.Header().Get("Content-Security-Policy")) - require.Equal(t, "private, no-store", w.Header().Get("Cache-Control")) - require.Equal(t, "no-referrer", w.Header().Get("Referrer-Policy")) - require.Equal(t, "nosniff", w.Header().Get("X-Content-Type-Options")) - require.Equal(t, "SAMEORIGIN", w.Header().Get("X-Frame-Options")) - require.NotContains(t, w.Body.String(), "SHOULD-NOT-APPEAR") - require.NotContains(t, w.Body.String(), "bridge_token") - main := httptest.NewRecorder() - r.ServeHTTP(main, httptest.NewRequest(http.MethodGet, "/", nil)) - require.Contains(t, main.Header().Get("Content-Security-Policy"), "'nonce-") - require.NotEqual(t, service.CodexGatewayBorrowPreviewCSP, main.Header().Get("Content-Security-Policy")) - reads := repo.reads - unknown := httptest.NewRecorder() - r.ServeHTTP(unknown, httptest.NewRequest(http.MethodGet, "/api/v1/codex-gateway-borrow/preview/"+strings.Repeat("x", 43)+"/index.html", nil)) - require.Equal(t, http.StatusGone, unknown.Code) - require.Equal(t, reads, repo.reads) -} - -func TestCodexGatewayBorrowHandlerReadAndSSEReplayPreserveFullErrorsWithoutDispatch(t *testing.T) { - gin.SetMode(gin.TestMode) - h, repo := newCodexGatewayBorrowHandlerFixture(t) - r := gin.New() - r.GET("/tests/:id", h.GetTest) - r.POST("/tests", h.StartTests) - w := httptest.NewRecorder() - r.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/tests/"+repo.task.ID, nil)) - require.Equal(t, http.StatusOK, w.Code) - var envelope struct { - Data service.CodexGatewayBorrowTestTask `json:"data"` - } - require.NoError(t, json.Unmarshal(w.Body.Bytes(), &envelope)) - require.Equal(t, repo.result.Error, envelope.Data.Results[0].Error) - require.NotEmpty(t, envelope.Data.Results[0].PreviewURL) - requestBody := `{"client_task_id":"` + repo.task.ClientTaskID + `","targets":[{"account_id":42,"model_id":"gpt-6-astra"}]}` - stream := httptest.NewRecorder() - request := httptest.NewRequest(http.MethodPost, "/tests", strings.NewReader(requestBody)) - request.Header.Set("Content-Type", "application/json") - r.ServeHTTP(stream, request) - require.Equal(t, http.StatusOK, stream.Code) - require.Equal(t, "text/event-stream", stream.Header().Get("Content-Type")) - require.Contains(t, stream.Body.String(), `"type":"task_start"`) - require.Contains(t, stream.Body.String(), `"type":"task_complete"`) - require.NotContains(t, stream.Body.String(), `"type":"result_started"`) - require.Contains(t, stream.Body.String(), "original upstream error token=admin-visible") - require.Equal(t, 1, repo.creates) - require.Zero(t, repo.writes, "SSE replay must never start or regenerate an existing task") -} diff --git a/backend/internal/handler/admin/pelican_background_handler.go b/backend/internal/handler/admin/pelican_background_handler.go index ef8a75c20983..7f4a18f0a614 100644 --- a/backend/internal/handler/admin/pelican_background_handler.go +++ b/backend/internal/handler/admin/pelican_background_handler.go @@ -16,8 +16,8 @@ import ( ) func (h *PelicanTestHandler) StartTask(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request service.CodexGatewayBorrowTestRequest + pelicanPrivateHeaders(c) + var request service.PelicanTestRequest if err := c.ShouldBindJSON(&request); err != nil { response.BadRequest(c, err.Error()) return @@ -28,7 +28,7 @@ func (h *PelicanTestHandler) StartTask(c *gin.Context) { } task, err := h.runner.StartBackground(c.Request.Context(), createdBy, request, h.catalog) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } middleware.SetAuditExtra(c, map[string]any{"result": "pelican_background_test", "task_id": task.ID, "replayed": task.Replayed, "targets": task.Total}) @@ -36,7 +36,7 @@ func (h *PelicanTestHandler) StartTask(c *gin.Context) { } func pelicanTaskID(c *gin.Context) (string, bool) { - codexGatewayBorrowPrivateHeaders(c) + pelicanPrivateHeaders(c) id, err := uuid.Parse(strings.TrimSpace(c.Param("id"))) if err != nil { response.BadRequest(c, "task ID must be a UUID") @@ -52,7 +52,7 @@ func (h *PelicanTestHandler) GetTask(c *gin.Context) { } task, err := h.runner.TaskSnapshot(c.Request.Context(), id) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } response.Success(c, task) @@ -67,7 +67,7 @@ func (h *PelicanTestHandler) GetTaskResults(c *gin.Context) { accountID, _ := strconv.ParseInt(c.Query("account_id"), 10, 64) result, err := h.runner.TaskResults(c.Request.Context(), id, service.PelicanResultFilter{Page: page, Size: size, AccountID: accountID, Model: c.Query("model"), Status: c.Query("status")}) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } response.Success(c, result) @@ -85,7 +85,7 @@ func (h *PelicanTestHandler) GetTaskResult(c *gin.Context) { } result, err := h.runner.TaskResult(c.Request.Context(), id, resultID.String()) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } response.Success(c, h.preview.DecorateResult(result)) @@ -98,7 +98,7 @@ func (h *PelicanTestHandler) CancelTask(c *gin.Context) { } task, err := h.runner.CancelBackground(c.Request.Context(), id) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } middleware.SetAuditExtra(c, map[string]any{"result": "pelican_cancel_requested", "task_id": task.ID}) @@ -114,7 +114,7 @@ func (h *PelicanTestHandler) ObserveTask(c *gin.Context) { } task, err := h.runner.TaskSnapshot(c.Request.Context(), id) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } c.Header("Content-Type", "text/event-stream") diff --git a/backend/internal/handler/admin/pelican_background_handler_test.go b/backend/internal/handler/admin/pelican_background_handler_test.go index 4fd7311e0d0f..f27d5a5351f0 100644 --- a/backend/internal/handler/admin/pelican_background_handler_test.go +++ b/backend/internal/handler/admin/pelican_background_handler_test.go @@ -14,22 +14,22 @@ import ( ) type pelicanSummaryHandlerRepo struct { - *codexGatewayBorrowHandlerTestRepo + *pelicanHandlerTestRepo } func (r *pelicanSummaryHandlerRepo) GetTaskSnapshot(context.Context, string) (*service.PelicanTaskSnapshot, error) { copy := *r.task copy.Results = nil - return &service.PelicanTaskSnapshot{CodexGatewayBorrowTestTask: ©, Counts: map[string]int{"complete": 1}}, nil + return &service.PelicanTaskSnapshot{PelicanTestTask: ©, Counts: map[string]int{"complete": 1}}, nil } func (r *pelicanSummaryHandlerRepo) ListTaskResults(context.Context, string, service.PelicanResultFilter) (*service.PelicanResultPage, error) { return &service.PelicanResultPage{Items: []*service.PelicanResultSummary{{ID: r.result.ID, TaskID: r.task.ID, Status: "complete", HasPreview: true}}, Total: 1, Page: 1, PageSize: 24}, nil } func TestPelicanBackgroundReadEndpointsNeverGenerateOrSendRawInProgress(t *testing.T) { gin.SetMode(gin.TestMode) - legacy, base := newCodexGatewayBorrowHandlerFixture(t) + legacy, base := newPelicanHandlerFixture(t) repo := &pelicanSummaryHandlerRepo{base} - runner := service.NewCodexGatewayBorrowTestRunner(repo, nil, nil) + runner := service.NewPelicanTestRunner(repo, nil, nil) t.Cleanup(runner.Stop) h := NewPelicanTestHandler(nil, runner, legacy.preview) router := gin.New() diff --git a/backend/internal/handler/admin/pelican_fixture_test.go b/backend/internal/handler/admin/pelican_fixture_test.go new file mode 100644 index 000000000000..aff81f43299e --- /dev/null +++ b/backend/internal/handler/admin/pelican_fixture_test.go @@ -0,0 +1,110 @@ +//go:build unit + +package admin + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/Wei-Shaw/sub2api/internal/config" + "github.com/Wei-Shaw/sub2api/internal/server/middleware" + "github.com/Wei-Shaw/sub2api/internal/service" + "github.com/gin-gonic/gin" + "github.com/google/uuid" + "github.com/stretchr/testify/require" +) + +type pelicanHandlerTestRepo struct { + task *service.PelicanTestTask + result *service.PelicanTestResult + creates int + writes int + reads int +} + +func (r *pelicanHandlerTestRepo) Create(context.Context, *service.PelicanTestTask) (*service.PelicanTestTask, bool, error) { + r.creates++ + return r.task, true, nil +} +func (r *pelicanHandlerTestRepo) Get(context.Context, string) (*service.PelicanTestTask, error) { + r.reads++ + return r.task, nil +} +func (r *pelicanHandlerTestRepo) GetResult(context.Context, string) (*service.PelicanTestResult, error) { + r.reads++ + return r.result, nil +} +func (r *pelicanHandlerTestRepo) List(context.Context, int, int) (*service.PelicanTestList, error) { + return &service.PelicanTestList{Items: []*service.PelicanTestTask{r.task}}, nil +} +func (r *pelicanHandlerTestRepo) StartTask(context.Context, string, time.Time) error { + r.writes++ + return nil +} +func (r *pelicanHandlerTestRepo) SaveResult(context.Context, *service.PelicanTestResult) error { + r.writes++ + return nil +} +func (r *pelicanHandlerTestRepo) FinishTask(context.Context, string, string, string, time.Time) error { + r.writes++ + return nil +} +func (r *pelicanHandlerTestRepo) MarkInterrupted(context.Context, time.Time) error { + return nil +} +func (r *pelicanHandlerTestRepo) CleanupExpired(context.Context, time.Time) error { + return nil +} + +func newPelicanHandlerFixture(t *testing.T) (*PelicanTestHandler, *pelicanHandlerTestRepo) { + t.Helper() + now := time.Now().UTC() + client, err := uuid.NewV7() + require.NoError(t, err) + result := &service.PelicanTestResult{ID: uuid.NewString(), Status: "complete", AccountID: 42, ModelID: "gpt-6-astra", Effort: "high", RawAnswer: "", RawHTML: "", HTML: "", Error: "original upstream error token=admin-visible\x00tail", ExpiresAt: now.Add(time.Hour)} + task := &service.PelicanTestTask{ID: uuid.NewString(), ClientTaskID: client.String(), Status: "complete", Total: 1, Completed: 1, CreatedAt: now, ExpiresAt: result.ExpiresAt, Results: []*service.PelicanTestResult{result}} + result.TaskID = task.ID + repo := &pelicanHandlerTestRepo{task: task, result: result} + runner := service.NewPelicanTestRunner(repo, nil, nil) + preview := service.NewPelicanPreviewService(repo) + t.Cleanup(runner.Stop) + t.Cleanup(preview.Stop) + return NewPelicanTestHandler(nil, runner, preview), repo +} + +func TestPelicanHandlerPreviewUsesOnlyCapabilityAndOverridesCSP(t *testing.T) { + gin.SetMode(gin.TestMode) + h, repo := newPelicanHandlerFixture(t) + decorated := h.preview.DecorateResult(repo.result) + r := gin.New() + r.Use(middleware.SecurityHeaders(config.CSPConfig{Enabled: true}, nil)) + r.GET("/api/v1/pelican-tests/preview/:cap/index.html", h.ServePreview) + r.GET("/", func(c *gin.Context) { c.String(http.StatusOK, "main app") }) + w := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodGet, decorated.PreviewURL, nil) + request.Header.Set("Authorization", "Bearer SHOULD-NOT-APPEAR") + request.Header.Set("Cookie", "admin_jwt=SHOULD-NOT-APPEAR") + r.ServeHTTP(w, request) + require.Equal(t, http.StatusOK, w.Code) + require.Equal(t, repo.result.HTML, w.Body.String()) + require.Equal(t, service.PelicanPreviewCSP, w.Header().Get("Content-Security-Policy")) + require.Equal(t, "private, no-store", w.Header().Get("Cache-Control")) + require.Equal(t, "no-referrer", w.Header().Get("Referrer-Policy")) + require.Equal(t, "nosniff", w.Header().Get("X-Content-Type-Options")) + require.Equal(t, "SAMEORIGIN", w.Header().Get("X-Frame-Options")) + require.NotContains(t, w.Body.String(), "SHOULD-NOT-APPEAR") + require.NotContains(t, w.Body.String(), "bridge_token") + main := httptest.NewRecorder() + r.ServeHTTP(main, httptest.NewRequest(http.MethodGet, "/", nil)) + require.Contains(t, main.Header().Get("Content-Security-Policy"), "'nonce-") + require.NotEqual(t, service.PelicanPreviewCSP, main.Header().Get("Content-Security-Policy")) + reads := repo.reads + unknown := httptest.NewRecorder() + r.ServeHTTP(unknown, httptest.NewRequest(http.MethodGet, "/api/v1/pelican-tests/preview/"+strings.Repeat("x", 43)+"/index.html", nil)) + require.Equal(t, http.StatusGone, unknown.Code) + require.Equal(t, reads, repo.reads) +} diff --git a/backend/internal/handler/admin/pelican_test_handler.go b/backend/internal/handler/admin/pelican_test_handler.go index 71b5611d3daf..5e41fe8c3920 100644 --- a/backend/internal/handler/admin/pelican_test_handler.go +++ b/backend/internal/handler/admin/pelican_test_handler.go @@ -18,16 +18,15 @@ import ( "github.com/google/uuid" ) -// PelicanTestHandler owns the independent administrator entry point. Borrowing -// configuration and preparation remain in CodexGatewayBorrowHandler; this entry -// selects only an account, a model, an effort, and a generation budget. +// PelicanTestHandler selects only an account, a model, an effort, +// and a generation budget for ordinary account execution. type PelicanTestHandler struct { catalog *service.PelicanTestCatalog - runner *service.CodexGatewayBorrowTestRunner - preview *service.CodexGatewayBorrowPreviewService + runner *service.PelicanTestRunner + preview *service.PelicanPreviewService } -func NewPelicanTestHandler(catalog *service.PelicanTestCatalog, runner *service.CodexGatewayBorrowTestRunner, preview *service.CodexGatewayBorrowPreviewService) *PelicanTestHandler { +func NewPelicanTestHandler(catalog *service.PelicanTestCatalog, runner *service.PelicanTestRunner, preview *service.PelicanPreviewService) *PelicanTestHandler { return &PelicanTestHandler{catalog: catalog, runner: runner, preview: preview} } @@ -36,7 +35,7 @@ func (h *PelicanTestHandler) GetOptions(c *gin.Context) { } func (h *PelicanTestHandler) GetAccountOptions(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) + pelicanPrivateHeaders(c) var request struct { AccountIDs []int64 `json:"account_ids"` } @@ -48,7 +47,7 @@ func (h *PelicanTestHandler) GetAccountOptions(c *gin.Context) { } func (h *PelicanTestHandler) options(c *gin.Context, accountIDs []int64) { - codexGatewayBorrowPrivateHeaders(c) + pelicanPrivateHeaders(c) options, err := h.catalog.Options(c.Request.Context(), accountIDs) if err != nil { if errors.Is(err, service.ErrPelicanTestOptionsInvalidRequest) { @@ -62,8 +61,8 @@ func (h *PelicanTestHandler) options(c *gin.Context, accountIDs []int64) { } func (h *PelicanTestHandler) StartTests(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - var request service.CodexGatewayBorrowTestRequest + pelicanPrivateHeaders(c) + var request service.PelicanTestRequest if err := c.ShouldBindJSON(&request); err != nil { response.BadRequest(c, err.Error()) return @@ -76,16 +75,13 @@ func (h *PelicanTestHandler) StartTests(c *gin.Context) { response.BadRequest(c, "targets is required") return } - // Only this server-side entry point can select account execution. A JSON - // field cannot change the legacy borrowing handler's existing contract. - request.Standalone = true var createdBy int64 if subject, ok := middleware.GetAuthSubjectFromContext(c); ok { createdBy = subject.UserID } task, replayed, err := h.runner.Create(c.Request.Context(), createdBy, request) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } middleware.SetAuditExtra(c, map[string]any{"result": "pelican_manual_test", "task_id": task.ID, "replayed": replayed, "targets": task.Total}) @@ -96,7 +92,7 @@ func (h *PelicanTestHandler) StartTests(c *gin.Context) { c.Header("X-Accel-Buffering", "no") c.Status(http.StatusOK) var writeMu sync.Mutex - emit := func(event service.CodexGatewayBorrowTestEvent) { + emit := func(event service.PelicanTestEvent) { if ctx.Err() != nil { return } @@ -122,9 +118,9 @@ func (h *PelicanTestHandler) StartTests(c *gin.Context) { } c.Writer.Flush() } - emit(service.CodexGatewayBorrowTestEvent{Type: "task_start", Task: task}) + emit(service.PelicanTestEvent{Type: "task_start", Task: task}) if replayed { - emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task}) + emit(service.PelicanTestEvent{Type: "task_complete", Task: task}) return } keepAliveDone, keepAliveStopped := make(chan struct{}), make(chan struct{}) @@ -156,12 +152,12 @@ func (h *PelicanTestHandler) StartTests(c *gin.Context) { <-keepAliveStopped if runErr != nil && ctx.Err() == nil { task.Status, task.Error = "incomplete", runErr.Error() - emit(service.CodexGatewayBorrowTestEvent{Type: "task_complete", Task: task}) + emit(service.PelicanTestEvent{Type: "task_complete", Task: task}) } } func (h *PelicanTestHandler) ListTests(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) + pelicanPrivateHeaders(c) page, size := response.ParsePagination(c) if rawSize := c.Query("size"); rawSize != "" { if parsed, err := strconv.Atoi(rawSize); err == nil && parsed > 0 { @@ -170,19 +166,19 @@ func (h *PelicanTestHandler) ListTests(c *gin.Context) { } list, err := h.runner.List(c.Request.Context(), page, size) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } response.Success(c, list) } func (h *PelicanTestHandler) GetTest(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) + pelicanPrivateHeaders(c) id := strings.TrimSpace(c.Param("id")) if id == "last" { list, err := h.runner.List(c.Request.Context(), 1, 1) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } if len(list.Items) == 0 { @@ -198,15 +194,15 @@ func (h *PelicanTestHandler) GetTest(c *gin.Context) { } task, err := h.runner.Get(c.Request.Context(), parsed.String()) if err != nil { - codexGatewayBorrowTestError(c, err) + pelicanTestError(c, err) return } response.Success(c, h.preview.DecorateTask(task)) } func (h *PelicanTestHandler) ServePreview(c *gin.Context) { - codexGatewayBorrowPrivateHeaders(c) - c.Header("Content-Security-Policy", service.CodexGatewayBorrowPreviewCSP) + pelicanPrivateHeaders(c) + c.Header("Content-Security-Policy", service.PelicanPreviewCSP) c.Header("X-Frame-Options", "SAMEORIGIN") c.Header("Cross-Origin-Resource-Policy", "cross-origin") html, err := h.preview.Resolve(c.Request.Context(), strings.TrimSpace(c.Param("cap"))) @@ -216,3 +212,24 @@ func (h *PelicanTestHandler) ServePreview(c *gin.Context) { } c.Data(http.StatusOK, "text/html; charset=utf-8", []byte(html)) } + +func pelicanPrivateHeaders(c *gin.Context) { + c.Header("Cache-Control", "private, no-store") + c.Header("Referrer-Policy", "no-referrer") + c.Header("X-Content-Type-Options", "nosniff") +} + +func pelicanTestError(c *gin.Context, err error) { + status := http.StatusInternalServerError + switch { + case errors.Is(err, service.ErrPelicanTestReplayConflict): + status = http.StatusConflict + case errors.Is(err, service.ErrPelicanTestInvalidRequest): + status = http.StatusBadRequest + case errors.Is(err, service.ErrPelicanTestExpired): + status = http.StatusGone + case errors.Is(err, service.ErrPelicanTestNotFound): + status = http.StatusNotFound + } + response.Error(c, status, err.Error()) +} diff --git a/backend/internal/handler/admin/pelican_test_handler_test.go b/backend/internal/handler/admin/pelican_test_handler_test.go index b3c5f7abfc5f..91db4c3c4fef 100644 --- a/backend/internal/handler/admin/pelican_test_handler_test.go +++ b/backend/internal/handler/admin/pelican_test_handler_test.go @@ -22,13 +22,13 @@ type pelicanTestHandlerLocalAccountReader struct { } type pelicanTestHandlerReplayCaptureRepo struct { - *codexGatewayBorrowHandlerTestRepo - incoming *service.CodexGatewayBorrowTestTask + *pelicanHandlerTestRepo + incoming *service.PelicanTestTask } -func (r *pelicanTestHandlerReplayCaptureRepo) Create(ctx context.Context, task *service.CodexGatewayBorrowTestTask) (*service.CodexGatewayBorrowTestTask, bool, error) { +func (r *pelicanTestHandlerReplayCaptureRepo) Create(ctx context.Context, task *service.PelicanTestTask) (*service.PelicanTestTask, bool, error) { r.incoming = task - return r.codexGatewayBorrowHandlerTestRepo.Create(ctx, task) + return r.pelicanHandlerTestRepo.Create(ctx, task) } func (r *pelicanTestHandlerLocalAccountReader) GetByIDs(_ context.Context, _ []int64) ([]*service.Account, error) { @@ -78,7 +78,7 @@ func TestPelicanTestHandlerOptionsDefaultsAndSelectedLocalModels(t *testing.T) { func TestPelicanTestHandlerSSEReplayAndHistoryNeverGenerateAndPreserveRawContent(t *testing.T) { gin.SetMode(gin.TestMode) - legacy, repo := newCodexGatewayBorrowHandlerFixture(t) + legacy, repo := newPelicanHandlerFixture(t) h := NewPelicanTestHandler(service.NewPelicanTestCatalog(nil), legacy.runner, legacy.preview) router := gin.New() router.GET("/tests/:id", h.GetTest) @@ -89,7 +89,7 @@ func TestPelicanTestHandlerSSEReplayAndHistoryNeverGenerateAndPreserveRawContent router.ServeHTTP(history, httptest.NewRequest(http.MethodGet, "/tests/last", nil)) require.Equal(t, http.StatusOK, history.Code) var envelope struct { - Data service.CodexGatewayBorrowTestTask `json:"data"` + Data service.PelicanTestTask `json:"data"` } require.NoError(t, json.Unmarshal(history.Body.Bytes(), &envelope)) require.Equal(t, repo.result.Error, envelope.Data.Results[0].Error) @@ -99,7 +99,7 @@ func TestPelicanTestHandlerSSEReplayAndHistoryNeverGenerateAndPreserveRawContent router.ServeHTTP(preview, httptest.NewRequest(http.MethodGet, envelope.Data.Results[0].PreviewURL, nil)) require.Equal(t, http.StatusOK, preview.Code) require.Equal(t, repo.result.HTML, preview.Body.String()) - require.Equal(t, service.CodexGatewayBorrowPreviewCSP, preview.Header().Get("Content-Security-Policy")) + require.Equal(t, service.PelicanPreviewCSP, preview.Header().Get("Content-Security-Policy")) stream := httptest.NewRecorder() body := fmt.Sprintf(`{"client_task_id":%q,"generation_timeout_seconds":600,"targets":[{"account_id":42,"model_id":"gpt-6-astra"}]}`, repo.task.ClientTaskID) request := httptest.NewRequest(http.MethodPost, "/tests", strings.NewReader(body)) @@ -117,10 +117,10 @@ func TestPelicanTestHandlerSSEReplayAndHistoryNeverGenerateAndPreserveRawContent func TestPelicanTestHandlerForcesAccountModeAndFixedPromptBeforeReplay(t *testing.T) { gin.SetMode(gin.TestMode) - _, existing := newCodexGatewayBorrowHandlerFixture(t) - repo := &pelicanTestHandlerReplayCaptureRepo{codexGatewayBorrowHandlerTestRepo: existing} - runner := service.NewCodexGatewayBorrowTestRunner(repo, nil, nil) - preview := service.NewCodexGatewayBorrowPreviewService(repo) + _, existing := newPelicanHandlerFixture(t) + repo := &pelicanTestHandlerReplayCaptureRepo{pelicanHandlerTestRepo: existing} + runner := service.NewPelicanTestRunner(repo, nil, nil) + preview := service.NewPelicanPreviewService(repo) t.Cleanup(runner.Stop) t.Cleanup(preview.Stop) h := NewPelicanTestHandler(service.NewPelicanTestCatalog(nil), runner, preview) @@ -134,7 +134,7 @@ func TestPelicanTestHandlerForcesAccountModeAndFixedPromptBeforeReplay(t *testin require.Equal(t, http.StatusOK, w.Code) require.NotNil(t, repo.incoming) require.Equal(t, service.PelicanExecutionModeAccount, repo.incoming.ExecutionMode) - require.Equal(t, service.CodexGatewayBorrowPelicanPrompt, repo.incoming.Prompt) + require.Equal(t, service.PelicanPrompt, repo.incoming.Prompt) require.Equal(t, 600, repo.incoming.GenerationTimeoutSeconds) require.Equal(t, int64(42), repo.incoming.Results[0].AccountID) require.Empty(t, repo.incoming.Results[0].Effort, "the generator applies only the selected model's supported default") @@ -143,7 +143,7 @@ func TestPelicanTestHandlerForcesAccountModeAndFixedPromptBeforeReplay(t *testin func TestPelicanTestHandlerRejectsGenerationBudgetBeforeCreate(t *testing.T) { gin.SetMode(gin.TestMode) - legacy, repo := newCodexGatewayBorrowHandlerFixture(t) + legacy, repo := newPelicanHandlerFixture(t) h := NewPelicanTestHandler(service.NewPelicanTestCatalog(nil), legacy.runner, legacy.preview) router := gin.New() router.POST("/tests", h.StartTests) diff --git a/backend/internal/handler/codex_gateway_borrow_failover_test.go b/backend/internal/handler/codex_gateway_borrow_failover_test.go deleted file mode 100644 index 3138eb604d79..000000000000 --- a/backend/internal/handler/codex_gateway_borrow_failover_test.go +++ /dev/null @@ -1,27 +0,0 @@ -package handler - -import ( - "context" - "errors" - "net/http" - "testing" - - "github.com/Wei-Shaw/sub2api/internal/service" - "github.com/stretchr/testify/require" -) - -func TestCodexGatewayBorrowFailoverSkipsSameAccountRetry(t *testing.T) { - account := &service.Account{ID: 2, Platform: service.PlatformOpenAI, Type: service.AccountTypeOAuth} - failure := service.NewCodexGatewayBorrowRequestFailure(errors.New("source proxy connection refused")) - require.Zero(t, openAISameAccountRetryLimit(account, failure, true)) - require.True(t, failure.ShouldRetryNextAccount()) - require.False(t, failure.ShouldReportAccountScheduleFailure()) - state := newOpenAIFailoverRetryState() - action := state.HandleHTTP(context.Background(), &service.OpenAIGatewayService{}, account, "gpt-6-astra", failure, true, 0, "borrow-test") - require.Equal(t, openAIFailoverRetrySwitchAccount, action) - require.Zero(t, state.sameAccountRetryCount[account.ID]) - // The new reason is narrow: ordinary transport and upstream failures retain - // their existing bounded retry on OAuth accounts. - require.Equal(t, 1, openAISameAccountRetryLimit(account, &service.UpstreamFailoverError{StatusCode: http.StatusBadGateway, Reason: service.OpenAITransientTransportFailureReason}, true)) - require.Equal(t, 1, openAISameAccountRetryLimit(account, &service.UpstreamFailoverError{StatusCode: http.StatusServiceUnavailable}, true)) -} diff --git a/backend/internal/handler/codex_gateway_borrow_ws_failure_test.go b/backend/internal/handler/codex_gateway_borrow_ws_failure_test.go deleted file mode 100644 index 96d95fd9c631..000000000000 --- a/backend/internal/handler/codex_gateway_borrow_ws_failure_test.go +++ /dev/null @@ -1,23 +0,0 @@ -//go:build unit - -package handler - -import ( - "errors" - "testing" - - "github.com/Wei-Shaw/sub2api/internal/service" - coderws "github.com/coder/websocket" - "github.com/stretchr/testify/require" -) - -func TestCodexGatewayBorrowWSPreparationFailureDoesNotReportTargetHealth(t *testing.T) { - local := &service.CodexGatewayBorrowFailure{Cause: errors.New("dial tcp synthetic-source.invalid:443: connect: connection refused")} - require.False(t, shouldReportOpenAIWSProxyAccountFailure(local)) - classified := service.NewCodexGatewayBorrowRequestFailure(local) - require.True(t, classified.ShouldRetryNextAccount()) - require.False(t, shouldReportOpenAIWSProxyAccountFailure(classified)) - closed := service.NewOpenAIWSClientCloseError(coderws.StatusTryAgainLater, "local preparation unavailable", classified) - require.False(t, shouldReportOpenAIWSProxyAccountFailure(closed)) - require.True(t, shouldReportOpenAIWSProxyAccountFailure(errors.New("ordinary business websocket connection refused"))) -} diff --git a/backend/internal/handler/handler.go b/backend/internal/handler/handler.go index 71d59a1c44c4..0e3a3a17c078 100644 --- a/backend/internal/handler/handler.go +++ b/backend/internal/handler/handler.go @@ -45,7 +45,6 @@ type AdminHandlers struct { Compliance *admin.ComplianceHandler AuditLog *admin.AuditLogHandler SystemPrompt *admin.SystemPromptHandler - CodexGatewayBorrow *admin.CodexGatewayBorrowHandler PelicanTest *admin.PelicanTestHandler } diff --git a/backend/internal/handler/openai_failover_retry.go b/backend/internal/handler/openai_failover_retry.go index cdca81067a29..438f0c9daa5d 100644 --- a/backend/internal/handler/openai_failover_retry.go +++ b/backend/internal/handler/openai_failover_retry.go @@ -120,11 +120,7 @@ func openAISameAccountRetryLimit(account *service.Account, failoverErr *service. if account == nil || failoverErr == nil || !failoverErr.ShouldRetryNextAccount() { return 0 } - if failoverErr.Reason == service.CodexGatewayBorrowPreparationFailureReason { - // A local preparation failure uses normal switch-account handling, but - // replaying it on the same account is not a network/inference retry. - return 0 - } + if account.Type == service.AccountTypeAPIKey { if !account.IsPoolMode() || failoverErr.StatusCode <= 0 || !account.IsPoolModeRetryableStatus(failoverErr.StatusCode) || diff --git a/backend/internal/handler/openai_gateway_handler.go b/backend/internal/handler/openai_gateway_handler.go index 8be6e043a5f9..ecbb8eac3593 100644 --- a/backend/internal/handler/openai_gateway_handler.go +++ b/backend/internal/handler/openai_gateway_handler.go @@ -114,9 +114,6 @@ func openAIWSPostOutputCyberClose(err error) (*service.OpenAIWSClientCloseError, } func shouldReportOpenAIWSProxyAccountFailure(err error) bool { - if service.IsCodexGatewayBorrowRequestFailure(err) { - return false - } if _, postOutputCyber := openAIWSPostOutputCyberClose(err); postOutputCyber { return false } @@ -916,7 +913,6 @@ func (h *OpenAIGatewayHandler) Responses(c *gin.Context) { }) } if err != nil { - if result != nil && result.ClientDisconnect { reqLog.Info("openai.client_disconnected", zap.Int64("account_id", account.ID), @@ -940,7 +936,6 @@ func (h *OpenAIGatewayHandler) Responses(c *gin.Context) { zap.Error(err), ) } else { - var failoverErr *service.UpstreamFailoverError if errors.As(err, &failoverErr) { if failoverClientGone(c) { diff --git a/backend/internal/handler/ops_codex_borrow_test.go b/backend/internal/handler/ops_codex_borrow_test.go deleted file mode 100644 index f7f457233954..000000000000 --- a/backend/internal/handler/ops_codex_borrow_test.go +++ /dev/null @@ -1,67 +0,0 @@ -package handler - -import ( - "net/http/httptest" - "testing" - - "github.com/Wei-Shaw/sub2api/internal/service" - "github.com/gin-gonic/gin" - "github.com/stretchr/testify/require" -) - -func TestOpsCodexBorrowPreparationIsNotAnUpstream503(t *testing.T) { - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - c.Request = httptest.NewRequest("POST", "/v1/responses", nil) - err := &service.CodexGatewayBorrowFailure{Cause: service.ErrCodexGatewayBorrowUnavailable, Stage: "target_validation", Reason: "target_state_changed", Detail: "full diagnostic detail"} - service.RecordCodexGatewayBorrowPreparationFailure(c, &service.Account{ID: 2, Platform: "openai"}, err) - phase, limited, owner, source := classifyOpsErrorLog(c, "upstream_error", "no qualified Codex gateway borrow route is available", "CODEX_GATEWAY_BORROW_UNAVAILABLE", 503) - require.Equal(t, "routing", phase) - require.False(t, limited, "a platform failure still counts as a failed client request") - require.Equal(t, "platform", owner) - require.Equal(t, "gateway", source) - entry := &service.OpsInsertErrorLogInput{} - applyOpsUpstreamFieldsFromContext(c, entry) - require.NotNil(t, entry.UpstreamStatusCode) - require.Zero(t, *entry.UpstreamStatusCode) - require.Len(t, entry.UpstreamErrors, 1) - require.Equal(t, "target_state_changed", entry.UpstreamErrors[0].Reason) - require.Contains(t, *entry.UpstreamErrorDetail, "full diagnostic detail") -} - -func TestOpsCodexBorrowPreparationUsesSelectedNode(t *testing.T) { - for _, tc := range []struct { - name string - stage string - verification *service.CodexGatewayBorrowVerification - proxyID int64 - proxyName string - }{ - {"legacy binding", "target_validation", nil, 47, "saved binding"}, - {"exhausted pool retains last probe", "node_selection", &service.CodexGatewayBorrowVerification{ProxyID: 43, ProxyName: "selected node", NodeIdentity: "selected-identity"}, 43, "selected node"}, - {"explicit direct", "target_validation", &service.CodexGatewayBorrowVerification{ProxyName: "Server direct", NodeIdentity: "direct-identity"}, 0, "direct/no_proxy"}, - {"no selected node", "node_selection", nil, 0, "unknown"}, - } { - t.Run(tc.name, func(t *testing.T) { - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - c.Request = httptest.NewRequest("POST", "/v1/responses", nil) - binding := int64(47) - account := &service.Account{ID: 2, Platform: "openai", ProxyID: &binding, Proxy: &service.Proxy{ID: 47, Name: "saved binding"}} - failure := &service.CodexGatewayBorrowFailure{Cause: service.ErrCodexGatewayBorrowUnavailable, Stage: tc.stage, Reason: "node_candidates_exhausted", Verification: tc.verification} - service.RecordCodexGatewayBorrowPreparationFailure(c, account, failure) - entry := &service.OpsInsertErrorLogInput{} - applyOpsUpstreamFieldsFromContext(c, entry) - require.Len(t, entry.UpstreamErrors, 1) - event := entry.UpstreamErrors[0] - if tc.proxyID == 0 { - require.Nil(t, event.ProxyID) - } else { - require.NotNil(t, event.ProxyID) - require.Equal(t, tc.proxyID, *event.ProxyID) - } - require.Equal(t, tc.proxyName, event.ProxyName) - require.NotNil(t, entry.UpstreamStatusCode) - require.Zero(t, *entry.UpstreamStatusCode, "local qualification failure is not a business upstream 503") - require.Equal(t, int64(47), *account.ProxyID, "diagnostics cannot rewrite account bindings") - }) - } -} diff --git a/backend/internal/handler/ops_error_logger.go b/backend/internal/handler/ops_error_logger.go index f8b6116a4f3c..5893548b189b 100644 --- a/backend/internal/handler/ops_error_logger.go +++ b/backend/internal/handler/ops_error_logger.go @@ -1804,7 +1804,7 @@ func applyOpsUpstreamErrorEvents(entry *service.OpsInsertErrorLogInput, events [ entry.UpstreamStatusCode = nil entry.UpstreamErrorMessage = nil entry.UpstreamErrorDetail = nil - if last.Stage == string(service.GatewayFailureStageAccountAuth) || last.Kind == string(service.CodexGatewayBorrowPreparationFailureReason) { + if last.Stage == string(service.GatewayFailureStageAccountAuth) { code := 0 entry.UpstreamStatusCode = &code } else if last.UpstreamStatusCode > 0 { diff --git a/backend/internal/handler/wire.go b/backend/internal/handler/wire.go index d21fcd75055c..31df87c5cc6b 100644 --- a/backend/internal/handler/wire.go +++ b/backend/internal/handler/wire.go @@ -49,7 +49,6 @@ func ProvideAdminHandlers( complianceHandler *admin.ComplianceHandler, auditLogHandler *admin.AuditLogHandler, systemPromptHandler *admin.SystemPromptHandler, - codexGatewayBorrowHandler *admin.CodexGatewayBorrowHandler, pelicanTestHandler *admin.PelicanTestHandler, upstreamBillingProbe *service.UpstreamBillingProbeService, ollamaCloudUsage *service.OllamaCloudUsageService, @@ -99,7 +98,6 @@ func ProvideAdminHandlers( Compliance: complianceHandler, AuditLog: auditLogHandler, SystemPrompt: systemPromptHandler, - CodexGatewayBorrow: codexGatewayBorrowHandler, PelicanTest: pelicanTestHandler, } } @@ -302,7 +300,6 @@ var ProviderSet = wire.NewSet( admin.NewComplianceHandler, admin.NewAuditLogHandler, admin.NewSystemPromptHandler, - admin.NewCodexGatewayBorrowHandler, admin.NewPelicanTestHandler, ProvideAccountJobRuntime, diff --git a/backend/internal/repository/codex_gateway_borrow_upstream.go b/backend/internal/repository/codex_gateway_borrow_upstream.go deleted file mode 100644 index b8fd2c19d13b..000000000000 --- a/backend/internal/repository/codex_gateway_borrow_upstream.go +++ /dev/null @@ -1,10 +0,0 @@ -package repository - -import "github.com/Wei-Shaw/sub2api/internal/service" - -// Borrow probes use the host transport and its provider profiles, exactly as -// ranxi fd1b5ee4's non-rotating path. Observation isolation belongs to the -// request context; copying transport configuration here would drift again. -func NewCodexGatewayBorrowProbeUpstream(upstream service.HTTPUpstream) service.CodexGatewayBorrowProbeUpstream { - return upstream -} diff --git a/backend/internal/repository/codex_gateway_borrow_upstream_test.go b/backend/internal/repository/codex_gateway_borrow_upstream_test.go deleted file mode 100644 index a41aab63c8bd..000000000000 --- a/backend/internal/repository/codex_gateway_borrow_upstream_test.go +++ /dev/null @@ -1,29 +0,0 @@ -package repository - -import ( - "github.com/Wei-Shaw/sub2api/internal/config" - "github.com/Wei-Shaw/sub2api/internal/service" - "github.com/stretchr/testify/require" - "testing" -) - -func TestCodexBorrowPinnedTransportPolicy(t *testing.T) { - cfg := &config.Config{Gateway: config.GatewayConfig{OpenAIHTTP2: config.GatewayOpenAIHTTP2Config{Enabled: true}}} - upstream := NewHTTPUpstream(cfg) - require.Same(t, upstream, NewCodexGatewayBorrowProbeUpstream(upstream)) - manager, ok := upstream.(*httpUpstreamService) - require.True(t, ok) - require.Equal(t, upstreamProtocolModeOpenAIH2, manager.resolveProtocolMode(service.HTTPUpstreamProfileCodexBorrowSource, "", nil)) - mode := manager.resolveProtocolMode(service.HTTPUpstreamProfileCodexBorrowTarget, "", nil) - require.Equal(t, upstreamProtocolModeOpenAIH1NoReuse, mode) - transport, err := buildUpstreamTransport(defaultPoolSettings(cfg), nil, mode) - require.NoError(t, err) - defer transport.CloseIdleConnections() - require.False(t, transport.ForceAttemptHTTP2) - require.True(t, transport.DisableKeepAlives) - require.Zero(t, transport.MaxIdleConns) - require.Zero(t, transport.MaxIdleConnsPerHost) - require.Empty(t, transport.TLSNextProto) - cfg.Gateway.OpenAIHTTP2.Enabled = false - require.Equal(t, upstreamProtocolModeOpenAIH1, manager.resolveProtocolMode(service.HTTPUpstreamProfileCodexBorrowSource, "", nil)) -} diff --git a/backend/internal/repository/http_upstream.go b/backend/internal/repository/http_upstream.go index 60331b36a53a..f10ab2906d61 100644 --- a/backend/internal/repository/http_upstream.go +++ b/backend/internal/repository/http_upstream.go @@ -39,7 +39,6 @@ import ( // 默认配置常量 // 这些值在配置文件未指定时作为回退默认值使用 const ( - upstreamProtocolModeOpenAIH1NoReuse = "openai_h1_no_reuse" // directProxyKey: 无代理时的缓存键标识 directProxyKey = "direct" // defaultMaxIdleConns: 默认最大空闲连接总数 @@ -1077,9 +1076,6 @@ func (s *httpUpstreamService) resolveOpenAIHTTP2Settings() openAIHTTP2Settings { } func (s *httpUpstreamService) resolveProtocolMode(profile service.HTTPUpstreamProfile, proxyKey string, parsedProxy *url.URL) string { - if profile == service.HTTPUpstreamProfileOpenAIHarvest { - return upstreamProtocolModeOpenAIH1NoReuse - } if profile == service.HTTPUpstreamProfileLongStream { return upstreamProtocolModeLongStreamH2 } @@ -1414,13 +1410,6 @@ func buildUpstreamTransport(settings poolSettings, proxyURL *url.URL, protocolMo transport.DisableCompression = true } switch protocolMode { - case upstreamProtocolModeOpenAIH1NoReuse: - // Fixed ranxi v2.10.3 harvest policy; each probe opens a fresh connection. - transport.ForceAttemptHTTP2 = false - transport.DisableKeepAlives = true - transport.MaxIdleConns = 0 - transport.MaxIdleConnsPerHost = 0 - transport.TLSNextProto = make(map[string]func(string, *tls.Conn) http.RoundTripper) case upstreamProtocolModeLongStreamH2, upstreamProtocolModeOpenAIH2: transport.ForceAttemptHTTP2 = true // 显式配置 http2 并启用 PING 健康探测，剔除代理/NAT 静默掐断的死连接， diff --git a/backend/internal/repository/pelican_background_repo.go b/backend/internal/repository/pelican_background_repo.go index 130d3773a23f..31715697e3ff 100644 --- a/backend/internal/repository/pelican_background_repo.go +++ b/backend/internal/repository/pelican_background_repo.go @@ -10,18 +10,18 @@ import ( "github.com/Wei-Shaw/sub2api/internal/service" ) -var _ service.PelicanTaskReader = (*codexGatewayBorrowTestRepository)(nil) +var _ service.PelicanTaskReader = (*pelicanTestRepository)(nil) -func (r *codexGatewayBorrowTestRepository) GetTaskSnapshot(ctx context.Context, id string) (*service.PelicanTaskSnapshot, error) { - task, err := scanCodexGatewayBorrowTestTask(r.db.QueryRowContext(ctx, `SELECT `+codexGatewayBorrowTestTaskColumns+` +func (r *pelicanTestRepository) GetTaskSnapshot(ctx context.Context, id string) (*service.PelicanTaskSnapshot, error) { + task, err := scanPelicanTestTask(r.db.QueryRowContext(ctx, `SELECT `+pelicanTestTaskColumns+` FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND (id=$1 OR client_task_id=$1) AND expires_at>CURRENT_TIMESTAMP`, id)) if errors.Is(err, sql.ErrNoRows) { - return nil, service.ErrCodexGatewayBorrowTestNotFound + return nil, service.ErrPelicanTestNotFound } if err != nil { return nil, err } - snapshot := &service.PelicanTaskSnapshot{CodexGatewayBorrowTestTask: task, Counts: make(map[string]int)} + snapshot := &service.PelicanTaskSnapshot{PelicanTestTask: task, Counts: make(map[string]int)} rows, err := r.db.QueryContext(ctx, `SELECT status, COUNT(*), COUNT(*) FILTER (WHERE status='complete' AND octet_length(html)=0) FROM codex_gateway_borrow_test_results WHERE task_id=$1 GROUP BY status`, task.ID) if err != nil { @@ -46,14 +46,14 @@ func (r *codexGatewayBorrowTestRepository) GetTaskSnapshot(ctx context.Context, return snapshot, rows.Err() } -// Keep this projection independent from codexGatewayBorrowTestResultColumns: +// Keep this projection independent from pelicanTestResultColumns: // no raw_answer, raw_response, raw_html, html or complete error enters a page. const pelicanResultSummaryColumns = `r.id,r.task_id,r.ordinal,r.account_id,r.account_name,r.platform,r.model_id,r.effort,r.status, (r.status='complete' AND octet_length(r.html)>0), (r.status='incomplete' AND t.error=convert_to('process stopped before the test finished; this task will not be replayed','UTF8')), r.started_at,r.finished_at,r.generation_started_at,r.duration_ms,r.queue_duration_ms,r.preparation_duration_ms,r.generation_duration_ms,t.expires_at` -func (r *codexGatewayBorrowTestRepository) ListTaskResults(ctx context.Context, id string, filter service.PelicanResultFilter) (*service.PelicanResultPage, error) { +func (r *pelicanTestRepository) ListTaskResults(ctx context.Context, id string, filter service.PelicanResultFilter) (*service.PelicanResultPage, error) { if filter.Page < 1 { filter.Page = 1 } @@ -82,7 +82,7 @@ func (r *codexGatewayBorrowTestRepository) ListTaskResults(ctx context.Context, case "pending", "running", "complete", "failed", "incomplete", "cancelled", "skipped": add("r.status=$%d", filter.Status) default: - return nil, service.ErrCodexGatewayBorrowTestInvalidRequest + return nil, service.ErrPelicanTestInvalidRequest } from := ` FROM codex_gateway_borrow_test_results r JOIN codex_gateway_borrow_test_tasks t ON t.id=r.task_id WHERE ` + strings.Join(where, " AND ") page := &service.PelicanResultPage{Items: []*service.PelicanResultSummary{}, Page: filter.Page, PageSize: filter.Size} diff --git a/backend/internal/repository/pelican_background_repo_integration_test.go b/backend/internal/repository/pelican_background_repo_integration_test.go index 31607a5d1a7b..8ab0f58a10cf 100644 --- a/backend/internal/repository/pelican_background_repo_integration_test.go +++ b/backend/internal/repository/pelican_background_repo_integration_test.go @@ -15,11 +15,11 @@ import ( func TestPelicanBackgroundSummaryPaginationRetainsFullDetails(t *testing.T) { ctx := context.Background() - db := codexGatewayBorrowPrivateIntegrationDB(t, ctx) - repo := NewCodexGatewayBorrowTestRepository(db) + db := pelicanPrivateIntegrationDB(t, ctx) + repo := NewPelicanTestRepository(db) reader, ok := repo.(service.PelicanTaskReader) require.True(t, ok) - task := codexGatewayBorrowIntegrationTask(t, 42) + task := pelicanIntegrationTask(t, 42) first := task.Results[0] for i := 2; i <= 3; i++ { copy := *first @@ -69,5 +69,5 @@ func TestPelicanBackgroundSummaryPaginationRetainsFullDetails(t *testing.T) { _, err = db.ExecContext(ctx, "UPDATE codex_gateway_borrow_test_tasks SET created_at=$2::timestamptz - INTERVAL '24 hours',expires_at=$2::timestamptz WHERE id=$1", task.ID, time.Now().Add(-time.Second)) require.NoError(t, err) _, err = reader.GetTaskSnapshot(ctx, task.ID) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestNotFound) + require.ErrorIs(t, err, service.ErrPelicanTestNotFound) } diff --git a/backend/internal/repository/codex_gateway_borrow_test_repo.go b/backend/internal/repository/pelican_test_repo.go similarity index 70% rename from backend/internal/repository/codex_gateway_borrow_test_repo.go rename to backend/internal/repository/pelican_test_repo.go index b154fcbcd6a3..2db307131e08 100644 --- a/backend/internal/repository/codex_gateway_borrow_test_repo.go +++ b/backend/internal/repository/pelican_test_repo.go @@ -9,24 +9,24 @@ import ( "github.com/Wei-Shaw/sub2api/internal/service" ) -type codexGatewayBorrowTestRepository struct{ db *sql.DB } +type pelicanTestRepository struct{ db *sql.DB } -func NewCodexGatewayBorrowTestRepository(db *sql.DB) service.CodexGatewayBorrowTestRepository { - return &codexGatewayBorrowTestRepository{db: db} +func NewPelicanTestRepository(db *sql.DB) service.PelicanTestRepository { + return &pelicanTestRepository{db: db} } -const codexGatewayBorrowTestTaskColumns = `id, client_task_id, created_by, request_hash, status, prompt, +const pelicanTestTaskColumns = `id, client_task_id, created_by, request_hash, status, prompt, created_at, expires_at, started_at, finished_at, total, completed, error, generation_timeout_seconds, execution_mode` -const codexGatewayBorrowTestResultColumns = `r.id, r.task_id, r.ordinal, r.account_id, r.account_name, r.model_id, +const pelicanTestResultColumns = `r.id, r.task_id, r.ordinal, r.account_id, r.account_name, r.model_id, r.upstream_model, r.effort, r.status, r.raw_answer, r.raw_response, r.raw_html, r.html, r.error, r.started_at, r.finished_at, r.duration_ms, t.expires_at, r.platform, r.actual_endpoint, r.actual_protocol, r.actual_transport, r.borrow_applied, r.queue_duration_ms, r.preparation_duration_ms, r.generation_duration_ms, r.generation_started_at` -type codexGatewayBorrowTestScanner interface{ Scan(...any) error } +type pelicanTestScanner interface{ Scan(...any) error } -func scanCodexGatewayBorrowTestTask(row codexGatewayBorrowTestScanner) (*service.CodexGatewayBorrowTestTask, error) { - task := &service.CodexGatewayBorrowTestTask{} +func scanPelicanTestTask(row pelicanTestScanner) (*service.PelicanTestTask, error) { + task := &service.PelicanTestTask{} var started, finished sql.NullTime var rawError []byte if err := row.Scan(&task.ID, &task.ClientTaskID, &task.CreatedBy, &task.RequestHash, &task.Status, &task.Prompt, @@ -43,8 +43,8 @@ func scanCodexGatewayBorrowTestTask(row codexGatewayBorrowTestScanner) (*service return task, nil } -func scanCodexGatewayBorrowTestResult(row codexGatewayBorrowTestScanner) (*service.CodexGatewayBorrowTestResult, error) { - result := &service.CodexGatewayBorrowTestResult{} +func scanPelicanTestResult(row pelicanTestScanner) (*service.PelicanTestResult, error) { + result := &service.PelicanTestResult{} var answer, rawResponse, rawHTML, html, rawError []byte var started, finished, generationStarted sql.NullTime if err := row.Scan(&result.ID, &result.TaskID, &result.Ordinal, &result.AccountID, &result.AccountName, &result.ModelID, @@ -73,28 +73,28 @@ func scanCodexGatewayBorrowTestResult(row codexGatewayBorrowTestScanner) (*servi // The conflict read intentionally includes expired UUIDs. Reposting a UUID can // never re-run a task while its row exists, even before expired-row cleanup. -func (r *codexGatewayBorrowTestRepository) Create(ctx context.Context, task *service.CodexGatewayBorrowTestTask) (*service.CodexGatewayBorrowTestTask, bool, error) { +func (r *pelicanTestRepository) Create(ctx context.Context, task *service.PelicanTestTask) (*service.PelicanTestTask, bool, error) { tx, err := r.db.BeginTx(ctx, nil) if err != nil { return nil, false, err } defer func() { _ = tx.Rollback() }() - stored, err := scanCodexGatewayBorrowTestTask(tx.QueryRowContext(ctx, `INSERT INTO codex_gateway_borrow_test_tasks + stored, err := scanPelicanTestTask(tx.QueryRowContext(ctx, `INSERT INTO codex_gateway_borrow_test_tasks (id,client_task_id,created_by,request_hash,status,prompt,total,created_at,expires_at,generation_timeout_seconds,execution_mode) VALUES ($1,$2,$3,$4,'pending',$5,$6,$7,$8,$9,$10) - ON CONFLICT (client_task_id) DO NOTHING RETURNING `+codexGatewayBorrowTestTaskColumns, + ON CONFLICT (client_task_id) DO NOTHING RETURNING `+pelicanTestTaskColumns, task.ID, task.ClientTaskID, task.CreatedBy, task.RequestHash, task.Prompt, task.Total, task.CreatedAt, task.ExpiresAt, task.GenerationTimeoutSeconds, task.ExecutionMode)) if errors.Is(err, sql.ErrNoRows) { - stored, err = scanCodexGatewayBorrowTestTask(tx.QueryRowContext(ctx, `SELECT `+codexGatewayBorrowTestTaskColumns+` + stored, err = scanPelicanTestTask(tx.QueryRowContext(ctx, `SELECT `+pelicanTestTaskColumns+` FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND client_task_id=$1 FOR UPDATE`, task.ClientTaskID)) if err != nil { return nil, false, err } if !stored.ExpiresAt.After(time.Now().UTC()) { - return nil, true, service.ErrCodexGatewayBorrowTestExpired + return nil, true, service.ErrPelicanTestExpired } if stored.RequestHash != task.RequestHash { - return nil, true, service.ErrCodexGatewayBorrowTestReplayConflict + return nil, true, service.ErrPelicanTestReplayConflict } if err = tx.Commit(); err != nil { return nil, true, err @@ -119,25 +119,25 @@ func (r *codexGatewayBorrowTestRepository) Create(ctx context.Context, task *ser return stored, false, nil } -func (r *codexGatewayBorrowTestRepository) Get(ctx context.Context, id string) (*service.CodexGatewayBorrowTestTask, error) { - task, err := scanCodexGatewayBorrowTestTask(r.db.QueryRowContext(ctx, `SELECT `+codexGatewayBorrowTestTaskColumns+` +func (r *pelicanTestRepository) Get(ctx context.Context, id string) (*service.PelicanTestTask, error) { + task, err := scanPelicanTestTask(r.db.QueryRowContext(ctx, `SELECT `+pelicanTestTaskColumns+` FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND (id=$1 OR client_task_id=$1) AND expires_at>CURRENT_TIMESTAMP`, id)) if errors.Is(err, sql.ErrNoRows) { - return nil, service.ErrCodexGatewayBorrowTestNotFound + return nil, service.ErrPelicanTestNotFound } if err != nil { return nil, err } - rows, err := r.db.QueryContext(ctx, `SELECT `+codexGatewayBorrowTestResultColumns+` + rows, err := r.db.QueryContext(ctx, `SELECT `+pelicanTestResultColumns+` FROM codex_gateway_borrow_test_results r JOIN codex_gateway_borrow_test_tasks t ON t.id=r.task_id WHERE t.scope='manual_pelican' AND t.id=$1 AND t.expires_at>CURRENT_TIMESTAMP ORDER BY r.ordinal`, task.ID) if err != nil { return nil, err } defer func() { _ = rows.Close() }() - task.Results = make([]*service.CodexGatewayBorrowTestResult, 0, task.Total) + task.Results = make([]*service.PelicanTestResult, 0, task.Total) for rows.Next() { - result, scanErr := scanCodexGatewayBorrowTestResult(rows) + result, scanErr := scanPelicanTestResult(rows) if scanErr != nil { return nil, scanErr } @@ -147,25 +147,25 @@ func (r *codexGatewayBorrowTestRepository) Get(ctx context.Context, id string) ( return nil, err } if !task.ExpiresAt.After(time.Now().UTC()) { - return nil, service.ErrCodexGatewayBorrowTestNotFound + return nil, service.ErrPelicanTestNotFound } return task, nil } -func (r *codexGatewayBorrowTestRepository) GetResult(ctx context.Context, id string) (*service.CodexGatewayBorrowTestResult, error) { - result, err := scanCodexGatewayBorrowTestResult(r.db.QueryRowContext(ctx, `SELECT `+codexGatewayBorrowTestResultColumns+` +func (r *pelicanTestRepository) GetResult(ctx context.Context, id string) (*service.PelicanTestResult, error) { + result, err := scanPelicanTestResult(r.db.QueryRowContext(ctx, `SELECT `+pelicanTestResultColumns+` FROM codex_gateway_borrow_test_results r JOIN codex_gateway_borrow_test_tasks t ON t.id=r.task_id WHERE t.scope='manual_pelican' AND r.id=$1 AND t.expires_at>CURRENT_TIMESTAMP`, id)) if errors.Is(err, sql.ErrNoRows) { - return nil, service.ErrCodexGatewayBorrowTestNotFound + return nil, service.ErrPelicanTestNotFound } if err == nil && !result.ExpiresAt.After(time.Now().UTC()) { - return nil, service.ErrCodexGatewayBorrowTestNotFound + return nil, service.ErrPelicanTestNotFound } return result, err } -func (r *codexGatewayBorrowTestRepository) List(ctx context.Context, page, size int) (*service.CodexGatewayBorrowTestList, error) { +func (r *pelicanTestRepository) List(ctx context.Context, page, size int) (*service.PelicanTestList, error) { if page < 1 { page = 1 } @@ -175,19 +175,19 @@ func (r *codexGatewayBorrowTestRepository) List(ctx context.Context, page, size if size > 100 { size = 100 } - list := &service.CodexGatewayBorrowTestList{Items: make([]*service.CodexGatewayBorrowTestTask, 0, size), Page: page, Size: size, PageSize: size} + list := &service.PelicanTestList{Items: make([]*service.PelicanTestTask, 0, size), Page: page, Size: size, PageSize: size} if err := r.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND expires_at>CURRENT_TIMESTAMP`).Scan(&list.Total); err != nil { return nil, err } - rows, err := r.db.QueryContext(ctx, `SELECT `+codexGatewayBorrowTestTaskColumns+` FROM codex_gateway_borrow_test_tasks + rows, err := r.db.QueryContext(ctx, `SELECT `+pelicanTestTaskColumns+` FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND expires_at>CURRENT_TIMESTAMP ORDER BY created_at DESC,id DESC LIMIT $1 OFFSET $2`, size, (page-1)*size) if err != nil { return nil, err } defer func() { _ = rows.Close() }() for rows.Next() { - task, scanErr := scanCodexGatewayBorrowTestTask(rows) + task, scanErr := scanPelicanTestTask(rows) if scanErr != nil { return nil, scanErr } @@ -198,7 +198,7 @@ func (r *codexGatewayBorrowTestRepository) List(ctx context.Context, page, size return list, rows.Err() } -func codexGatewayBorrowRequireAffected(result sql.Result, err error) error { +func pelicanRequireAffected(result sql.Result, err error) error { if err != nil { return err } @@ -207,24 +207,24 @@ func codexGatewayBorrowRequireAffected(result sql.Result, err error) error { return err } if count == 0 { - return service.ErrCodexGatewayBorrowTestNotFound + return service.ErrPelicanTestNotFound } return nil } -func (r *codexGatewayBorrowTestRepository) StartTask(ctx context.Context, id string, started time.Time) error { - return codexGatewayBorrowRequireAffected(r.db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks +func (r *pelicanTestRepository) StartTask(ctx context.Context, id string, started time.Time) error { + return pelicanRequireAffected(r.db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks SET status='running',started_at=$2,updated_at=CURRENT_TIMESTAMP WHERE scope='manual_pelican' AND id=$1 AND status='pending' AND expires_at>CURRENT_TIMESTAMP`, id, started)) } -func (r *codexGatewayBorrowTestRepository) SaveResult(ctx context.Context, result *service.CodexGatewayBorrowTestResult) error { +func (r *pelicanTestRepository) SaveResult(ctx context.Context, result *service.PelicanTestResult) error { tx, err := r.db.BeginTx(ctx, nil) if err != nil { return err } defer func() { _ = tx.Rollback() }() - if err = codexGatewayBorrowRequireAffected(tx.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_results r + if err = pelicanRequireAffected(tx.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_results r SET account_name=$3,upstream_model=$4,effort=$5,status=$6,raw_answer=$7,raw_response=$8,raw_html=$9,html=$10,error=$11, started_at=$12,finished_at=$13,duration_ms=$14,platform=$15,actual_endpoint=$16,actual_protocol=$17, actual_transport=$18,borrow_applied=$19,queue_duration_ms=$20,preparation_duration_ms=$21, @@ -246,13 +246,13 @@ func (r *codexGatewayBorrowTestRepository) SaveResult(ctx context.Context, resul return tx.Commit() } -func (r *codexGatewayBorrowTestRepository) FinishTask(ctx context.Context, id, status, errorText string, finished time.Time) error { - return codexGatewayBorrowRequireAffected(r.db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks +func (r *pelicanTestRepository) FinishTask(ctx context.Context, id, status, errorText string, finished time.Time) error { + return pelicanRequireAffected(r.db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks SET status=$2,error=$3,finished_at=$4,updated_at=CURRENT_TIMESTAMP WHERE scope='manual_pelican' AND id=$1 AND expires_at>CURRENT_TIMESTAMP`, id, status, []byte(errorText), finished)) } -func (r *codexGatewayBorrowTestRepository) MarkInterrupted(ctx context.Context, now time.Time) error { +func (r *pelicanTestRepository) MarkInterrupted(ctx context.Context, now time.Time) error { tx, err := r.db.BeginTx(ctx, nil) if err != nil { return err @@ -273,7 +273,7 @@ func (r *codexGatewayBorrowTestRepository) MarkInterrupted(ctx context.Context, return tx.Commit() } -func (r *codexGatewayBorrowTestRepository) CleanupExpired(ctx context.Context, now time.Time) error { +func (r *pelicanTestRepository) CleanupExpired(ctx context.Context, now time.Time) error { _, err := r.db.ExecContext(ctx, `DELETE FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND expires_at<=$1`, now) return err diff --git a/backend/internal/repository/codex_gateway_borrow_test_repo_integration_test.go b/backend/internal/repository/pelican_test_repo_integration_test.go similarity index 82% rename from backend/internal/repository/codex_gateway_borrow_test_repo_integration_test.go rename to backend/internal/repository/pelican_test_repo_integration_test.go index e277e70c1353..bf5aba36b9c2 100644 --- a/backend/internal/repository/codex_gateway_borrow_test_repo_integration_test.go +++ b/backend/internal/repository/pelican_test_repo_integration_test.go @@ -21,7 +21,7 @@ import ( // The real repository opens its own transactions, so a private schema keeps // its expiry cleanup separate from every other integration fixture and ledger. // Copy the harness connection configuration without displaying its credentials. -func codexGatewayBorrowPrivateIntegrationDB(t *testing.T, ctx context.Context) *sql.DB { +func pelicanPrivateIntegrationDB(t *testing.T, ctx context.Context) *sql.DB { t.Helper() conn, err := integrationDB.Conn(ctx) require.NoError(t, err) @@ -38,7 +38,7 @@ func codexGatewayBorrowPrivateIntegrationDB(t *testing.T, ctx context.Context) * require.NoError(t, err) require.NoError(t, closeErr) - schema := "codex_borrow_it_" + strings.ReplaceAll(uuid.NewString(), "-", "") + schema := "pelican_it_" + strings.ReplaceAll(uuid.NewString(), "-", "") quotedSchema := pgx.Identifier{schema}.Sanitize() _, err = integrationDB.ExecContext(ctx, "CREATE SCHEMA "+quotedSchema) require.NoError(t, err) @@ -73,7 +73,7 @@ func codexGatewayBorrowPrivateIntegrationDB(t *testing.T, ctx context.Context) * legacyAnswer := []byte("legacy original answer\x00tail") _, err = privateDB.ExecContext(ctx, `INSERT INTO codex_gateway_borrow_test_tasks (id,client_task_id,created_by,request_hash,status,prompt,total,created_at,expires_at) - VALUES ($1,$2,7,'legacy-hash','complete',$3,1,$4,$5)`, legacyTask, legacyClient.String(), service.CodexGatewayBorrowPelicanPrompt, created, created.Add(service.CodexGatewayBorrowTestTTL)) + VALUES ($1,$2,7,'legacy-hash','complete',$3,1,$4,$5)`, legacyTask, legacyClient.String(), service.PelicanPrompt, created, created.Add(service.PelicanTestTTL)) require.NoError(t, err) _, err = privateDB.ExecContext(ctx, `INSERT INTO codex_gateway_borrow_test_results (id,task_id,ordinal,account_id,model_id,effort,status,raw_answer,error) @@ -83,6 +83,25 @@ func codexGatewayBorrowPrivateIntegrationDB(t *testing.T, ctx context.Context) * require.NoError(t, err) _, err = privateDB.ExecContext(ctx, string(ddl)) require.NoError(t, err, "apply the actual additive execution migration") + _, err = privateDB.ExecContext(ctx, `CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT NOT NULL); + INSERT INTO settings VALUES ('codex_gateway_borrow_config','{"enabled":true}'),('unrelated_setting','preserve');`) + require.NoError(t, err) + retirement, err := migrations.FS.ReadFile("278_retire_codex_gateway_borrow_config.sql") + require.NoError(t, err) + for i := 0; i < 2; i++ { + _, err = privateDB.ExecContext(ctx, string(retirement)) + require.NoError(t, err) + } + var remaining int + require.NoError(t, privateDB.QueryRowContext(ctx, `SELECT count(*) FROM settings WHERE key='codex_gateway_borrow_config'`).Scan(&remaining)) + require.Zero(t, remaining) + var kept string + require.NoError(t, privateDB.QueryRowContext(ctx, `SELECT value FROM settings WHERE key='unrelated_setting'`).Scan(&kept)) + require.Equal(t, "preserve", kept) + var modeDefault string + require.NoError(t, privateDB.QueryRowContext(ctx, `SELECT column_default FROM information_schema.columns + WHERE table_schema=current_schema() AND table_name='codex_gateway_borrow_test_tasks' AND column_name='execution_mode'`).Scan(&modeDefault)) + require.Equal(t, "'account'::text", modeDefault) var answer, originalError []byte var executionMode string var timeout int @@ -94,26 +113,26 @@ func codexGatewayBorrowPrivateIntegrationDB(t *testing.T, ctx context.Context) * require.NoError(t, err) require.Equal(t, legacyAnswer, answer) require.Equal(t, legacyAnswer, originalError) - require.Equal(t, service.PelicanExecutionModeLegacyCache, executionMode) + require.Equal(t, "legacy_cache", executionMode) require.Equal(t, 90, timeout, "historical observations retain their actual legacy budget") - require.Equal(t, int64(service.CodexGatewayBorrowTestTTL/time.Second), retention) + require.Equal(t, int64(service.PelicanTestTTL/time.Second), retention) _, err = privateDB.ExecContext(ctx, `DELETE FROM codex_gateway_borrow_test_tasks WHERE id=$1`, legacyTask) require.NoError(t, err) return privateDB } -func codexGatewayBorrowIntegrationTask(t *testing.T, accountID int64) *service.CodexGatewayBorrowTestTask { +func pelicanIntegrationTask(t *testing.T, accountID int64) *service.PelicanTestTask { t.Helper() clientID, err := uuid.NewV7() require.NoError(t, err) created := time.Now().UTC().Truncate(time.Microsecond) - task := &service.CodexGatewayBorrowTestTask{ + task := &service.PelicanTestTask{ ID: uuid.NewString(), ClientTaskID: clientID.String(), CreatedBy: 7, - RequestHash: strings.Repeat("a", 64), Status: "pending", Prompt: service.CodexGatewayBorrowPelicanPrompt, - CreatedAt: created, ExpiresAt: created.Add(service.CodexGatewayBorrowTestTTL), Total: 1, + RequestHash: strings.Repeat("a", 64), Status: "pending", Prompt: service.PelicanPrompt, + CreatedAt: created, ExpiresAt: created.Add(service.PelicanTestTTL), Total: 1, GenerationTimeoutSeconds: service.PelicanDefaultGenerationTimeoutSeconds, ExecutionMode: service.PelicanExecutionModeAccount, } - task.Results = []*service.CodexGatewayBorrowTestResult{{ + task.Results = []*service.PelicanTestResult{{ ID: uuid.NewString(), TaskID: task.ID, Ordinal: 1, AccountID: accountID, AccountName: "private integration fixture", Platform: "openai", ModelID: "gpt-6-astra", Effort: "high", Status: "pending", ExpiresAt: task.ExpiresAt, @@ -121,12 +140,12 @@ func codexGatewayBorrowIntegrationTask(t *testing.T, accountID int64) *service.C return task } -func TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle(t *testing.T) { +func TestPelicanTestRepositoryIntegrationLifecycle(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() - db := codexGatewayBorrowPrivateIntegrationDB(t, ctx) - repo := NewCodexGatewayBorrowTestRepository(db) - task := codexGatewayBorrowIntegrationTask(t, 987654321) + db := pelicanPrivateIntegrationDB(t, ctx) + repo := NewPelicanTestRepository(db) + task := pelicanIntegrationTask(t, 987654321) // No account fixture is needed: these are retained observations, not an // account relationship or a model request. The genuine schema enforces this. @@ -138,7 +157,7 @@ func TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle(t *testing.T) { require.Len(t, stored.Results, 1) require.Equal(t, service.PelicanDefaultGenerationTimeoutSeconds, stored.GenerationTimeoutSeconds) require.Equal(t, service.PelicanExecutionModeAccount, stored.ExecutionMode) - require.Equal(t, service.CodexGatewayBorrowTestTTL, stored.ExpiresAt.Sub(stored.CreatedAt)) + require.Equal(t, service.PelicanTestTTL, stored.ExpiresAt.Sub(stored.CreatedAt)) started := time.Now().UTC().Truncate(time.Microsecond) require.NoError(t, repo.StartTask(ctx, task.ID, started)) @@ -191,7 +210,7 @@ func TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle(t *testing.T) { // A different server-side ID cannot replace an existing client UUID. The // replay returns the old complete task and its exact BYTEA-backed results. - replayInput := codexGatewayBorrowIntegrationTask(t, result.AccountID) + replayInput := pelicanIntegrationTask(t, result.AccountID) replayInput.ClientTaskID = task.ClientTaskID old, replayed, err := repo.Create(ctx, replayInput) require.NoError(t, err) @@ -202,12 +221,12 @@ func TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle(t *testing.T) { replayInput.RequestHash = strings.Repeat("b", 64) _, replayed, err = repo.Create(ctx, replayInput) require.True(t, replayed) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestReplayConflict) + require.ErrorIs(t, err, service.ErrPelicanTestReplayConflict) var clientRows int require.NoError(t, db.QueryRowContext(ctx, `SELECT COUNT(*) FROM codex_gateway_borrow_test_tasks WHERE client_task_id=$1`, task.ClientTaskID).Scan(&clientRows)) require.Equal(t, 1, clientRows) - keeper := codexGatewayBorrowIntegrationTask(t, 987654322) + keeper := pelicanIntegrationTask(t, 987654322) _, replayed, err = repo.Create(ctx, keeper) require.NoError(t, err) require.False(t, replayed) @@ -222,15 +241,15 @@ func TestCodexGatewayBorrowTestRepositoryIntegrationLifecycle(t *testing.T) { require.NoError(t, err) expiredCreated := time.Now().UTC().Add(-25 * time.Hour).Truncate(time.Microsecond) - _, err = db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks SET created_at=$2,expires_at=$3 WHERE id=$1`, task.ID, expiredCreated, expiredCreated.Add(service.CodexGatewayBorrowTestTTL)) + _, err = db.ExecContext(ctx, `UPDATE codex_gateway_borrow_test_tasks SET created_at=$2,expires_at=$3 WHERE id=$1`, task.ID, expiredCreated, expiredCreated.Add(service.PelicanTestTTL)) require.NoError(t, err) _, err = repo.Get(ctx, task.ID) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestNotFound) + require.ErrorIs(t, err, service.ErrPelicanTestNotFound) _, err = repo.GetResult(ctx, result.ID) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestNotFound) + require.ErrorIs(t, err, service.ErrPelicanTestNotFound) _, replayed, err = repo.Create(ctx, replayInput) require.True(t, replayed) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestExpired) + require.ErrorIs(t, err, service.ErrPelicanTestExpired) list, err := repo.List(ctx, 1, 10) require.NoError(t, err) require.Equal(t, int64(1), list.Total) diff --git a/backend/internal/repository/codex_gateway_borrow_test_repo_test.go b/backend/internal/repository/pelican_test_repo_test.go similarity index 78% rename from backend/internal/repository/codex_gateway_borrow_test_repo_test.go rename to backend/internal/repository/pelican_test_repo_test.go index f06f0d55a0cd..7b240728ed27 100644 --- a/backend/internal/repository/codex_gateway_borrow_test_repo_test.go +++ b/backend/internal/repository/pelican_test_repo_test.go @@ -15,41 +15,41 @@ import ( "github.com/stretchr/testify/require" ) -func newCodexGatewayBorrowRepoTest(t *testing.T) (*codexGatewayBorrowTestRepository, sqlmock.Sqlmock) { +func newPelicanRepoTest(t *testing.T) (*pelicanTestRepository, sqlmock.Sqlmock) { t.Helper() db, mock, err := sqlmock.New() require.NoError(t, err) t.Cleanup(func() { _ = db.Close() }) - return &codexGatewayBorrowTestRepository{db: db}, mock + return &pelicanTestRepository{db: db}, mock } -func codexGatewayBorrowTaskTestRows(task *service.CodexGatewayBorrowTestTask) *sqlmock.Rows { +func pelicanTaskTestRows(task *service.PelicanTestTask) *sqlmock.Rows { return sqlmock.NewRows([]string{"id", "client_task_id", "created_by", "request_hash", "status", "prompt", "created_at", "expires_at", "started_at", "finished_at", "total", "completed", "error", "generation_timeout_seconds", "execution_mode"}).AddRow( task.ID, task.ClientTaskID, task.CreatedBy, task.RequestHash, task.Status, task.Prompt, task.CreatedAt, task.ExpiresAt, nil, nil, task.Total, task.Completed, []byte(task.Error), task.GenerationTimeoutSeconds, task.ExecutionMode) } -func codexGatewayBorrowResultTestRows(result *service.CodexGatewayBorrowTestResult) *sqlmock.Rows { +func pelicanResultTestRows(result *service.PelicanTestResult) *sqlmock.Rows { return sqlmock.NewRows([]string{"id", "task_id", "ordinal", "account_id", "account_name", "model_id", "upstream_model", "effort", "status", "raw_answer", "raw_response", "raw_html", "html", "error", "started_at", "finished_at", "duration_ms", "expires_at", "platform", "actual_endpoint", "actual_protocol", "actual_transport", "borrow_applied", "queue_duration_ms", "preparation_duration_ms", "generation_duration_ms", "generation_started_at"}).AddRow( result.ID, result.TaskID, result.Ordinal, result.AccountID, result.AccountName, result.ModelID, result.UpstreamModel, result.Effort, result.Status, []byte(result.RawAnswer), []byte(result.RawResponse), []byte(result.RawHTML), []byte(result.HTML), []byte(result.Error), result.StartedAt, result.FinishedAt, result.DurationMS, result.ExpiresAt, result.Platform, result.ActualEndpoint, result.ActualProtocol, result.ActualTransport, result.BorrowApplied, result.QueueDurationMS, result.PreparationDurationMS, result.GenerationDurationMS, result.GenerationStartedAt) } -func codexGatewayBorrowRepoFixture(t *testing.T) *service.CodexGatewayBorrowTestTask { +func pelicanRepoFixture(t *testing.T) *service.PelicanTestTask { t.Helper() now := time.Now().UTC() clientID, err := uuid.NewV7() require.NoError(t, err) - task := &service.CodexGatewayBorrowTestTask{ID: uuid.NewString(), ClientTaskID: clientID.String(), CreatedBy: 7, RequestHash: "hash", Status: "pending", Prompt: service.CodexGatewayBorrowPelicanPrompt, CreatedAt: now, ExpiresAt: now.Add(service.CodexGatewayBorrowTestTTL), Total: 1, + task := &service.PelicanTestTask{ID: uuid.NewString(), ClientTaskID: clientID.String(), CreatedBy: 7, RequestHash: "hash", Status: "pending", Prompt: service.PelicanPrompt, CreatedAt: now, ExpiresAt: now.Add(service.PelicanTestTTL), Total: 1, GenerationTimeoutSeconds: service.PelicanDefaultGenerationTimeoutSeconds, ExecutionMode: service.PelicanExecutionModeAccount} - task.Results = []*service.CodexGatewayBorrowTestResult{{ID: uuid.NewString(), TaskID: task.ID, Ordinal: 1, AccountID: 42, AccountName: "paused target", Platform: "openai", ModelID: "gpt-6-astra", Effort: "high", Status: "pending", ExpiresAt: task.ExpiresAt}} + task.Results = []*service.PelicanTestResult{{ID: uuid.NewString(), TaskID: task.ID, Ordinal: 1, AccountID: 42, AccountName: "paused target", Platform: "openai", ModelID: "gpt-6-astra", Effort: "high", Status: "pending", ExpiresAt: task.ExpiresAt}} return task } -func TestCodexGatewayBorrowTestRepositoryCreateAtomicAndReplayReadsWithoutWrites(t *testing.T) { - repo, mock := newCodexGatewayBorrowRepoTest(t) - task := codexGatewayBorrowRepoFixture(t) +func TestPelicanTestRepositoryCreateAtomicAndReplayReadsWithoutWrites(t *testing.T) { + repo, mock := newPelicanRepoTest(t) + task := pelicanRepoFixture(t) mock.ExpectBegin() - mock.ExpectQuery("INSERT INTO codex_gateway_borrow_test_tasks").WithArgs(task.ID, task.ClientTaskID, task.CreatedBy, task.RequestHash, task.Prompt, task.Total, task.CreatedAt, task.ExpiresAt, task.GenerationTimeoutSeconds, task.ExecutionMode).WillReturnRows(codexGatewayBorrowTaskTestRows(task)) + mock.ExpectQuery("INSERT INTO codex_gateway_borrow_test_tasks").WithArgs(task.ID, task.ClientTaskID, task.CreatedBy, task.RequestHash, task.Prompt, task.Total, task.CreatedAt, task.ExpiresAt, task.GenerationTimeoutSeconds, task.ExecutionMode).WillReturnRows(pelicanTaskTestRows(task)) result := task.Results[0] mock.ExpectExec("INSERT INTO codex_gateway_borrow_test_results").WithArgs(result.ID, task.ID, 1, result.AccountID, result.AccountName, result.ModelID, result.Effort, result.Platform).WillReturnResult(sqlmock.NewResult(0, 1)) mock.ExpectCommit() @@ -65,10 +65,10 @@ func TestCodexGatewayBorrowTestRepositoryCreateAtomicAndReplayReadsWithoutWrites result.Status, result.RawAnswer, result.RawResponse, result.Error = "complete", "answer\x00tail", "full raw stream\x00end", "untruncated upstream diagnostic" mock.ExpectBegin() mock.ExpectQuery("INSERT INTO codex_gateway_borrow_test_tasks").WillReturnRows(sqlmock.NewRows([]string{"id"})) - mock.ExpectQuery("FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND client_task_id=\\$1 FOR UPDATE").WithArgs(task.ClientTaskID).WillReturnRows(codexGatewayBorrowTaskTestRows(task)) + mock.ExpectQuery("FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND client_task_id=\\$1 FOR UPDATE").WithArgs(task.ClientTaskID).WillReturnRows(pelicanTaskTestRows(task)) mock.ExpectCommit() - mock.ExpectQuery("FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican'.*expires_at>CURRENT_TIMESTAMP").WithArgs(task.ID).WillReturnRows(codexGatewayBorrowTaskTestRows(task)) - mock.ExpectQuery("FROM codex_gateway_borrow_test_results r JOIN codex_gateway_borrow_test_tasks t.*t.expires_at>CURRENT_TIMESTAMP").WithArgs(task.ID).WillReturnRows(codexGatewayBorrowResultTestRows(result)) + mock.ExpectQuery("FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican'.*expires_at>CURRENT_TIMESTAMP").WithArgs(task.ID).WillReturnRows(pelicanTaskTestRows(task)) + mock.ExpectQuery("FROM codex_gateway_borrow_test_results r JOIN codex_gateway_borrow_test_tasks t.*t.expires_at>CURRENT_TIMESTAMP").WithArgs(task.ID).WillReturnRows(pelicanResultTestRows(result)) stored, replayed, err = repo.Create(context.Background(), task) require.NoError(t, err) require.True(t, replayed) @@ -79,43 +79,43 @@ func TestCodexGatewayBorrowTestRepositoryCreateAtomicAndReplayReadsWithoutWrites require.NoError(t, mock.ExpectationsWereMet()) } -func TestCodexGatewayBorrowTestRepositoryRejectsConflictingAndExpiredReplay(t *testing.T) { +func TestPelicanTestRepositoryRejectsConflictingAndExpiredReplay(t *testing.T) { for _, expired := range []bool{false, true} { t.Run(map[bool]string{false: "conflict", true: "expired"}[expired], func(t *testing.T) { - repo, mock := newCodexGatewayBorrowRepoTest(t) - task := codexGatewayBorrowRepoFixture(t) + repo, mock := newPelicanRepoTest(t) + task := pelicanRepoFixture(t) original := *task if expired { original.CreatedAt = time.Now().Add(-25 * time.Hour) - original.ExpiresAt = original.CreatedAt.Add(service.CodexGatewayBorrowTestTTL) + original.ExpiresAt = original.CreatedAt.Add(service.PelicanTestTTL) } else { original.RequestHash = "different" } mock.ExpectBegin() mock.ExpectQuery("INSERT INTO codex_gateway_borrow_test_tasks").WillReturnRows(sqlmock.NewRows([]string{"id"})) - mock.ExpectQuery("client_task_id=\\$1 FOR UPDATE").WillReturnRows(codexGatewayBorrowTaskTestRows(&original)) + mock.ExpectQuery("client_task_id=\\$1 FOR UPDATE").WillReturnRows(pelicanTaskTestRows(&original)) mock.ExpectRollback() _, replayed, err := repo.Create(context.Background(), task) require.True(t, replayed) if expired { - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestExpired) + require.ErrorIs(t, err, service.ErrPelicanTestExpired) } else { - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestReplayConflict) + require.ErrorIs(t, err, service.ErrPelicanTestReplayConflict) } require.NoError(t, mock.ExpectationsWereMet()) }) } } -func TestCodexGatewayBorrowTestRepositoryExpiredReadsAndScopedCleanup(t *testing.T) { - repo, mock := newCodexGatewayBorrowRepoTest(t) +func TestPelicanTestRepositoryExpiredReadsAndScopedCleanup(t *testing.T) { + repo, mock := newPelicanRepoTest(t) id := uuid.NewString() mock.ExpectQuery("FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican'.*expires_at>CURRENT_TIMESTAMP").WithArgs(id).WillReturnRows(sqlmock.NewRows([]string{"id"})) _, err := repo.Get(context.Background(), id) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestNotFound) + require.ErrorIs(t, err, service.ErrPelicanTestNotFound) mock.ExpectQuery("WHERE t.scope='manual_pelican' AND r.id=\\$1 AND t.expires_at>CURRENT_TIMESTAMP").WithArgs(id).WillReturnRows(sqlmock.NewRows([]string{"id"})) _, err = repo.GetResult(context.Background(), id) - require.ErrorIs(t, err, service.ErrCodexGatewayBorrowTestNotFound) + require.ErrorIs(t, err, service.ErrPelicanTestNotFound) now := time.Now().UTC() mock.ExpectExec(regexp.QuoteMeta("DELETE FROM codex_gateway_borrow_test_tasks WHERE scope='manual_pelican' AND expires_at<=$1")).WithArgs(now).WillReturnResult(sqlmock.NewResult(0, 2)) require.NoError(t, repo.CleanupExpired(context.Background(), now)) @@ -133,9 +133,9 @@ func TestCodexGatewayBorrowTestRepositoryExpiredReadsAndScopedCleanup(t *testing require.NotContains(t, string(data), "DROP") } -func TestCodexGatewayBorrowTestRepositorySavesFullRawValuesAndMarksRestartIncomplete(t *testing.T) { - repo, mock := newCodexGatewayBorrowRepoTest(t) - task := codexGatewayBorrowRepoFixture(t) +func TestPelicanTestRepositorySavesFullRawValuesAndMarksRestartIncomplete(t *testing.T) { + repo, mock := newPelicanRepoTest(t) + task := pelicanRepoFixture(t) result := task.Results[0] result.Status, result.UpstreamModel, result.RawAnswer, result.RawResponse, result.RawHTML, result.HTML, result.Error = "incomplete", "reported", "answer\x00tail", "SSE\x00raw", "source\x00raw", "html\x00raw", "API returned 403: entire original body\x00tail" started := time.Now().UTC() @@ -147,7 +147,7 @@ func TestCodexGatewayBorrowTestRepositorySavesFullRawValuesAndMarksRestartIncomp mock.ExpectExec("UPDATE codex_gateway_borrow_test_tasks t.*completed=.*status NOT IN").WithArgs(task.ID).WillReturnResult(sqlmock.NewResult(0, 1)) mock.ExpectCommit() require.NoError(t, repo.SaveResult(context.Background(), result)) - mock.ExpectQuery("WHERE t.scope='manual_pelican' AND r.id=\\$1 AND t.expires_at>CURRENT_TIMESTAMP").WithArgs(result.ID).WillReturnRows(codexGatewayBorrowResultTestRows(result)) + mock.ExpectQuery("WHERE t.scope='manual_pelican' AND r.id=\\$1 AND t.expires_at>CURRENT_TIMESTAMP").WithArgs(result.ID).WillReturnRows(pelicanResultTestRows(result)) stored, err := repo.GetResult(context.Background(), result.ID) require.NoError(t, err) result.PreviewUnavailable = "test incomplete" diff --git a/backend/internal/repository/wire.go b/backend/internal/repository/wire.go index 9bef5ef19d10..aef47d918fb8 100644 --- a/backend/internal/repository/wire.go +++ b/backend/internal/repository/wire.go @@ -75,7 +75,7 @@ var ProviderSet = wire.NewSet( NewAdminAccountRepository, NewScheduledTestPlanRepository, // 定时测试计划仓储 NewScheduledTestResultRepository, // 定时测试结果仓储 - NewCodexGatewayBorrowTestRepository, + NewPelicanTestRepository, NewProxyRepository, NewRedeemCodeRepository, NewPromoCodeRepository, @@ -166,7 +166,6 @@ var ProviderSet = wire.NewSet( NewClaudeUsageFetcher, NewClaudeOAuthClient, NewHTTPUpstream, - NewCodexGatewayBorrowProbeUpstream, NewOpenAIOAuthClient, NewOpenAIReferralClient, NewGrokOAuthClient, diff --git a/backend/internal/server/routes/admin.go b/backend/internal/server/routes/admin.go index 1565fa988196..4c71b8b397a6 100644 --- a/backend/internal/server/routes/admin.go +++ b/backend/internal/server/routes/admin.go @@ -22,7 +22,6 @@ func RegisterAdminRoutes( ) { // 插件 UI 使用短时能力 URL，仅提供经过安装校验的静态资源。 v1.GET("/plugin-ui/:token/*path", h.Admin.Plugin.ServeUIAsset) - v1.GET("/codex-gateway-borrow/preview/:cap/index.html", h.Admin.CodexGatewayBorrow.ServePreview) v1.GET("/pelican-tests/preview/:cap/index.html", h.Admin.PelicanTest.ServePreview) admin := v1.Group("/admin") @@ -132,7 +131,6 @@ func RegisterAdminRoutes( registerSystemPromptRoutes(admin, h) // 推理恢复全局开关 - registerCodexGatewayBorrowRoutes(admin, h) registerPelicanTestRoutes(admin, h) // 邀请返利（专属用户管理） @@ -143,19 +141,6 @@ func RegisterAdminRoutes( } } -func registerCodexGatewayBorrowRoutes(admin *gin.RouterGroup, h *handler.Handlers) { - borrow := admin.Group("/codex-gateway-borrow") - borrow.GET("/config", h.Admin.CodexGatewayBorrow.GetConfig) - borrow.PUT("/config", h.Admin.CodexGatewayBorrow.SaveConfig) - borrow.GET("/status", h.Admin.CodexGatewayBorrow.Status) - borrow.POST("/prepare", h.Admin.CodexGatewayBorrow.Prepare) - borrow.POST("/verify", h.Admin.CodexGatewayBorrow.Verify) - borrow.POST("/diagnose", h.Admin.CodexGatewayBorrow.Diagnose) - borrow.POST("/tests", h.Admin.CodexGatewayBorrow.StartTests) - borrow.GET("/tests", h.Admin.CodexGatewayBorrow.ListTests) - borrow.GET("/tests/:id", h.Admin.CodexGatewayBorrow.GetTest) -} - func registerAccountJobRoutes(admin *gin.RouterGroup, h *handler.Handlers) { jobs := admin.Group("/account-jobs") { diff --git a/backend/internal/server/routes/pelican_retirement_test.go b/backend/internal/server/routes/pelican_retirement_test.go new file mode 100644 index 000000000000..c9312bb49b56 --- /dev/null +++ b/backend/internal/server/routes/pelican_retirement_test.go @@ -0,0 +1,41 @@ +//go:build unit + +package routes + +import ( + "net/http" + "net/http/httptest" + "testing" + + "github.com/Wei-Shaw/sub2api/internal/handler" + "github.com/Wei-Shaw/sub2api/internal/server/middleware" + "github.com/gin-gonic/gin" + "github.com/stretchr/testify/require" +) + +func TestPelicanRoutesRetireBorrowing(t *testing.T) { + gin.SetMode(gin.TestMode) + router := gin.New() + noop := func(c *gin.Context) { c.Next() } + RegisterAdminRoutes(router.Group("/api/v1"), &handler.Handlers{Admin: &handler.AdminHandlers{}}, + middleware.AdminAuthMiddleware(noop), middleware.AuditLogMiddleware(noop), middleware.StepUpAuthMiddleware(noop), nil, + middleware.NewPanelRateLimiter(nil, nil)) + registered := map[string]bool{} + for _, route := range router.Routes() { + registered[route.Method+" "+route.Path] = true + } + require.True(t, registered["POST /api/v1/admin/pelican-tests/tasks"]) + require.True(t, registered["GET /api/v1/admin/pelican-tests/tasks/:id"]) + require.True(t, registered["GET /api/v1/pelican-tests/preview/:cap/index.html"]) + for _, endpoint := range []struct{ method, path string }{ + {"GET", "/api/v1/admin/codex-gateway-borrow/config"}, {"PUT", "/api/v1/admin/codex-gateway-borrow/config"}, + {"GET", "/api/v1/admin/codex-gateway-borrow/status"}, {"POST", "/api/v1/admin/codex-gateway-borrow/prepare"}, + {"POST", "/api/v1/admin/codex-gateway-borrow/verify"}, {"POST", "/api/v1/admin/codex-gateway-borrow/diagnose"}, + {"POST", "/api/v1/admin/codex-gateway-borrow/tests"}, {"GET", "/api/v1/admin/codex-gateway-borrow/tests"}, + {"GET", "/api/v1/admin/codex-gateway-borrow/tests/one"}, {"GET", "/api/v1/codex-gateway-borrow/preview/one/index.html"}, + } { + response := httptest.NewRecorder() + router.ServeHTTP(response, httptest.NewRequest(endpoint.method, endpoint.path, nil)) + require.Equal(t, http.StatusNotFound, response.Code, endpoint.method+" "+endpoint.path) + } +} diff --git a/backend/internal/service/account_test_service.go b/backend/internal/service/account_test_service.go index 6ce29c2da17d..3f01791e696c 100644 --- a/backend/internal/service/account_test_service.go +++ b/backend/internal/service/account_test_service.go @@ -167,7 +167,6 @@ type AccountTestService struct { pelicanGatewayService *GatewayService pelicanGeminiService *GeminiMessagesCompatService pelicanConcurrencyService *ConcurrencyService - pelicanBorrowService *CodexGatewayBorrowService modelMetadataRegistryMu sync.Mutex modelMetadataRegistry map[string]modelsDevProvider modelMetadataRegistryAt time.Time @@ -198,18 +197,7 @@ func (s *AccountTestService) SetPluginManager(pluginManager *PluginManager) { func (s *AccountTestService) SetOpenAIGatewayService(gateway *OpenAIGatewayService) { if s != nil { s.openaiGatewayService = gateway - if gateway != nil && s.pelicanBorrowService != nil { - gateway.SetCodexGatewayBorrowService(s.pelicanBorrowService) - } - } -} -func (s *AccountTestService) SetCodexGatewayBorrowService(borrow *CodexGatewayBorrowService) { - if s != nil { - s.pelicanBorrowService = borrow - if s.openaiGatewayService != nil { - s.openaiGatewayService.SetCodexGatewayBorrowService(borrow) - } } } diff --git a/backend/internal/service/codex_gateway_borrow.go b/backend/internal/service/codex_gateway_borrow.go deleted file mode 100644 index 302224a24716..000000000000 --- a/backend/internal/service/codex_gateway_borrow.go +++ /dev/null @@ -1,1158 +0,0 @@ -package service - -// Cookie scope, the 230-second lease and two-shot STATE comparison are adapted -// from ranxi2001/sub2api at 5ca3cca21eeaf4ca8a694a7f2f8f0ecd9575c549: -// backend/internal/repository/{codex_gateway_pin,astra_target_validation}.go -// and backend/internal/service/openai_codex_state_probe.go (LGPL-3.0). -// Optional node selection changes only scoped outgoing request routes; account -// bindings, account health and quota remain owned by the host. - -import ( - "bytes" - "context" - "crypto/sha256" - "encoding/hex" - "encoding/json" - "errors" - "fmt" - "io" - "net/http" - "slices" - "sort" - "strings" - "sync" - "time" - - infraerrors "github.com/Wei-Shaw/sub2api/internal/pkg/errors" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" -) - -const SettingKeyCodexGatewayBorrowConfig = "codex_gateway_borrow_config" - -const ( - codexGatewayBorrowTTL = 230 * time.Second - codexGatewayBorrowTimeout = 90 * time.Second - codexGatewayBorrowShotTimeout = 45 * time.Second - codexGatewayBorrowFailureWait = 15 * time.Second - codexGatewayBorrowStoreTimeout = 5 * time.Second - codexGatewayBorrowSourceModel = "gpt-6-astra" -) - -var ( - ErrCodexGatewayBorrowUnavailable = infraerrors.ServiceUnavailable("CODEX_GATEWAY_BORROW_UNAVAILABLE", "no qualified Codex gateway borrow route is available") - ErrCodexGatewayBorrowBusy = infraerrors.Conflict("CODEX_GATEWAY_BORROW_BUSY", "target validation is already in progress") - ErrCodexGatewayBorrowChanged = infraerrors.Conflict("CODEX_GATEWAY_BORROW_CHANGED", "gateway borrow configuration or route changed") -) - -// A local route qualification failure is distinct from an actual business -// request failing upstream. Callers must surface it before ordinary account -// failover, stream-timeout or health hooks. Unwrap preserves the specific cause. -type CodexGatewayBorrowFailure struct { - CacheHit bool - Revision uint64 - CookieFingerprint string - Cause error - Stage string - Reason string - Detail string - RetryAfter *time.Time - Verification *CodexGatewayBorrowVerification -} - -func (e *CodexGatewayBorrowFailure) Error() string { - if e == nil || e.Cause == nil { - return "Codex gateway borrow qualification failed" - } - detail := e.Cause.Error() - if e.Detail != "" { - detail += ": " + e.Detail - } - if e.Reason != "" { - detail = e.Stage + "/" + e.Reason + ": " + detail - } - return "Codex gateway borrow qualification failed: " + detail -} -func (e *CodexGatewayBorrowFailure) Unwrap() error { - if e == nil { - return nil - } - return e.Cause -} -func IsCodexGatewayBorrowFailure(err error) bool { - var failure *CodexGatewayBorrowFailure - return errors.As(err, &failure) -} - -func borrowVerificationFailure(check codexGatewayBorrowTargetCheck, revision uint64) *CodexGatewayBorrowFailure { - result := check.result - failure := &CodexGatewayBorrowFailure{Revision: revision, Cause: ErrCodexGatewayBorrowUnavailable, Stage: "target_validation", Reason: result.Reason, Detail: result.Error, Verification: &result, CookieFingerprint: check.cookieKey} - if !check.retryAfter.IsZero() { - retry := check.retryAfter - failure.RetryAfter = &retry - } - return failure -} - -func codexBorrowFailureStage(err error) (string, string) { - var failure *CodexGatewayBorrowFailure - if errors.As(err, &failure) && failure.Reason != "" { - return failure.Stage, failure.Reason - } - if errors.Is(err, ErrCodexGatewayBorrowChanged) { - return "qualification", "configuration_changed" - } - if errors.Is(err, context.Canceled) { - return "preparation", "cancelled" - } - if errors.Is(err, context.DeadlineExceeded) { - return "preparation", "deadline_exceeded" - } - return "preparation", "borrow_unavailable" -} - -// CodexGatewayBorrowProbeUpstream is deliberately a distinct injection type. -// It delegates to the host transport; profiles select the pinned upstream -// policy without maintaining a separate transport configuration. -type CodexGatewayBorrowProbeUpstream interface{ HTTPUpstream } - -type CodexGatewayBorrowConfig struct { - ImmediateReborrowOnFailure bool `json:"immediate_reborrow_on_failure,omitempty"` - WaitForQualifiedRoute bool `json:"wait_for_qualified_route,omitempty"` - RotateNodes bool `json:"rotate_nodes,omitempty"` - NodeProxyIDs []int64 `json:"node_proxy_ids,omitempty"` - MaxNodeAttempts int `json:"max_node_attempts,omitempty"` - NodeCooldownSeconds int `json:"node_cooldown_seconds,omitempty"` - Enabled bool `json:"enabled"` - SourceAccountIDs []int64 `json:"source_account_ids"` - TargetAccountIDs []int64 `json:"target_account_ids"` - Models []string `json:"models"` -} - -func DefaultCodexGatewayBorrowConfig() CodexGatewayBorrowConfig { - return CodexGatewayBorrowConfig{SourceAccountIDs: []int64{}, TargetAccountIDs: []int64{}, Models: []string{"gpt-6-astra", "gpt-6.1-sol"}} -} - -func DecodeCodexGatewayBorrowConfig(raw []byte) (CodexGatewayBorrowConfig, error) { - cfg := DefaultCodexGatewayBorrowConfig() - if trimmed := bytes.TrimSpace(raw); len(trimmed) == 0 || trimmed[0] != '{' { - return cfg, errors.New("configuration must be a JSON object") - } - d := json.NewDecoder(bytes.NewReader(raw)) - d.DisallowUnknownFields() - if err := d.Decode(&cfg); err != nil { - return cfg, err - } - var extra any - if err := d.Decode(&extra); err != io.EOF { - return cfg, errors.New("configuration must contain one JSON object") - } - return normalizeCodexGatewayBorrowConfig(cfg) -} - -func normalizeCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) (CodexGatewayBorrowConfig, error) { - if err := validateCodexBorrowNodeConfig(cfg); err != nil { - return cfg, err - } - if cfg.SourceAccountIDs == nil { - cfg.SourceAccountIDs = []int64{} - } - if cfg.TargetAccountIDs == nil { - cfg.TargetAccountIDs = []int64{} - } - if cfg.Models == nil { - cfg.Models = DefaultCodexGatewayBorrowConfig().Models - } - seen := map[int64]bool{} - for _, ids := range [][]int64{cfg.SourceAccountIDs, cfg.TargetAccountIDs} { - for _, id := range ids { - if id <= 0 || seen[id] { - return cfg, errors.New("source and target account IDs must be positive, unique and disjoint") - } - seen[id] = true - } - } - models := map[string]bool{} - for _, model := range cfg.Models { - if (model != "gpt-6-astra" && model != "gpt-6.1-sol") || models[model] { - return cfg, errors.New("models must be a unique selection of gpt-6-astra and gpt-6.1-sol") - } - models[model] = true - } - if cfg.Enabled && (len(cfg.SourceAccountIDs) == 0 || len(cfg.TargetAccountIDs) == 0 || len(cfg.Models) == 0) { - return cfg, errors.New("enabled gateway borrowing requires sources, targets and models") - } - return cloneCodexGatewayBorrowConfig(cfg), nil -} - -func cloneCodexGatewayBorrowConfig(cfg CodexGatewayBorrowConfig) CodexGatewayBorrowConfig { - cfg.NodeProxyIDs = append([]int64(nil), cfg.NodeProxyIDs...) - cfg.SourceAccountIDs = append([]int64{}, cfg.SourceAccountIDs...) - cfg.TargetAccountIDs = append([]int64{}, cfg.TargetAccountIDs...) - cfg.Models = append([]string{}, cfg.Models...) - return cfg -} - -type CodexGatewayBorrowCandidateStatus struct { - NodeIdentity string `json:"node_identity,omitempty"` - ProxyID int64 `json:"proxy_id,omitempty"` - Gateway string `json:"gateway,omitempty"` - SourceRequestEncoding string `json:"source_request_encoding,omitempty"` - SourceAccountID int64 `json:"source_account_id"` - ExpiresAt time.Time `json:"expires_at"` - RemainingSeconds int64 `json:"remaining_seconds"` - CookieFingerprint string `json:"cookie_fingerprint"` -} - -type CodexGatewayBorrowSourceStatus struct { - NodeIdentity string `json:"node_identity,omitempty"` - ProxyID int64 `json:"proxy_id,omitempty"` - AccountID int64 `json:"account_id"` - State string `json:"state"` - Reason string `json:"reason"` - CheckedAt *time.Time `json:"checked_at,omitempty"` - ExpiresAt *time.Time `json:"expires_at,omitempty"` - RemainingSeconds int64 `json:"remaining_seconds"` - Error string `json:"error,omitempty"` -} - -type CodexGatewayBorrowVerification struct { - ProxyID int64 `json:"proxy_id,omitempty"` - ProxyName string `json:"proxy_name,omitempty"` - NodeIdentity string `json:"node_identity,omitempty"` - Gateway string `json:"gateway,omitempty"` - RetryOnNextRequest bool `json:"retry_on_next_request,omitempty"` - FailedShot string `json:"failed_shot,omitempty"` - FailureKind string `json:"failure_kind,omitempty"` - MintTransportPhase string `json:"mint_transport_phase,omitempty"` - ContinueTransportPhase string `json:"continue_transport_phase,omitempty"` - MintTransportAttempts int `json:"mint_transport_attempts,omitempty"` - ContinueTransportAttempts int `json:"continue_transport_attempts,omitempty"` - MintRequestEncoding string `json:"mint_request_encoding,omitempty"` - ContinueRequestEncoding string `json:"continue_request_encoding,omitempty"` - RequestShape string `json:"request_shape,omitempty"` - ServiceTier string `json:"service_tier,omitempty"` - MintCompleted bool `json:"mint_completed"` - ContinueCompleted bool `json:"continue_completed"` - MintStateLength int `json:"mint_state_length"` - ContinueStateLength int `json:"continue_state_length"` - RouteChanged bool `json:"route_changed"` - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Success bool `json:"success"` - Reason string `json:"reason"` - Error string `json:"error,omitempty"` - MintStatus int `json:"mint_status"` - ContinueStatus int `json:"continue_status"` - Minted bool `json:"minted"` - NewTicket bool `json:"new_ticket"` - ReportedModel string `json:"reported_model,omitempty"` - CheckedAt time.Time `json:"checked_at"` - ExpiresAt *time.Time `json:"expires_at,omitempty"` -} - -type CodexGatewayBorrowTargetStatus struct { - Acquisition CodexGatewayBorrowAcquisition `json:"acquisition"` - CodexGatewayBorrowVerification - State string `json:"state"` - CacheValid bool `json:"cache_valid"` - RemainingSeconds int64 `json:"remaining_seconds"` - RetryAfter *time.Time `json:"retry_after,omitempty"` -} - -type CodexGatewayBorrowStatus struct { - Nodes []CodexGatewayBorrowNodeStatus `json:"nodes,omitempty"` - Acquisition CodexGatewayBorrowAcquisition `json:"acquisition"` - ObservedSince time.Time `json:"observed_since"` - RecentUsage []CodexGatewayBorrowUsage `json:"recent_usage"` - Setup CodexGatewayBorrowSetup `json:"setup"` - Enabled bool `json:"enabled"` - Revision uint64 `json:"revision"` - GeneratedAt time.Time `json:"generated_at"` - Preparing bool `json:"preparing"` - Config CodexGatewayBorrowConfig `json:"config"` - Candidate *CodexGatewayBorrowCandidateStatus `json:"candidate,omitempty"` - Sources []CodexGatewayBorrowSourceStatus `json:"sources"` - Targets []CodexGatewayBorrowTargetStatus `json:"targets"` - ModelEfforts map[string][]string `json:"model_efforts"` -} - -// Application contains only opaque identities. The selected cookie itself is -// available solely in the cloned outbound request's Cookie header. -type CodexGatewayBorrowApplication struct { - node *codexBorrowNodeBinding - Verification CodexGatewayBorrowVerification - Applied bool - SourceAccountID int64 - ExpiresAt time.Time - Fingerprint string - CookieFingerprint string -} - -type codexGatewayBorrowCandidate struct { - node *codexBorrowNode - sourceRequestEncoding string - cookie http.Cookie - expires time.Time - sourceID int64 -} - -type codexGatewayBorrowTargetKey struct { - accountID int64 - model string -} -type codexGatewayBorrowTargetCheck struct { - acquisition CodexGatewayBorrowAcquisition - validationID uint64 - policyRevision uint64 - identityRevision string - key string - cookieKey string - expires time.Time - retryAfter time.Time - validating bool - result CodexGatewayBorrowVerification -} - -type CodexGatewayBorrowService struct { - proxies ProxyRepository - node *codexBorrowNode - nodeLease func(string, time.Time) time.Time - rotation codexBorrowRotation - rotationCandidates map[codexGatewayBorrowTargetKey]*codexGatewayBorrowCandidate - validationSequence uint64 - sourceSequence uint64 - qualifications map[string]codexGatewayBorrowTargetCheck - settings SettingRepository - accounts AccountRepository - gateway *OpenAIGatewayService - upstream CodexGatewayBorrowProbeUpstream - // A profile supplied by a business caller is reused verbatim. Current Codex - // HTTP and WS business paths use standard TLS, so manual preparation uses nil. - tlsProfiles *TLSFingerprintProfileService - storeMu sync.Mutex - mu sync.Mutex - config CodexGatewayBorrowConfig - loaded bool - revision uint64 - revisionCtx context.Context - cancel context.CancelFunc - candidates map[int64]*codexGatewayBorrowCandidate - candidate *codexGatewayBorrowCandidate - sources map[int64]CodexGatewayBorrowSourceStatus - targets map[codexGatewayBorrowTargetKey]codexGatewayBorrowTargetCheck - acquisition CodexGatewayBorrowAcquisition - preparing bool - prepareRuns int - prepareFailedUntil time.Time - prepareFailure string - sourceFlight codexBorrowFlights - targetFlight codexBorrowFlights - setupFlight codexBorrowFlights - setup CodexGatewayBorrowSetup - usage map[codexBorrowUsageKey]CodexGatewayBorrowUsage - observedSince time.Time - wsAnchors codexGatewayBorrowWSAnchorStore -} - -func NewCodexGatewayBorrowService(settings SettingRepository, accounts AccountRepository, gateway *OpenAIGatewayService, upstream CodexGatewayBorrowProbeUpstream, tlsProfiles *TLSFingerprintProfileService) *CodexGatewayBorrowService { - ctx, cancel := context.WithCancel(context.Background()) - s := &CodexGatewayBorrowService{settings: settings, accounts: accounts, gateway: gateway, upstream: upstream, tlsProfiles: tlsProfiles, - config: DefaultCodexGatewayBorrowConfig(), revisionCtx: ctx, cancel: cancel, - sources: map[int64]CodexGatewayBorrowSourceStatus{}, targets: map[codexGatewayBorrowTargetKey]codexGatewayBorrowTargetCheck{}} - if gateway != nil { - gateway.SetCodexGatewayBorrowService(s) - } - return s -} - -func ProvideCodexGatewayBorrowService(settings SettingRepository, accounts AccountRepository, gateway *OpenAIGatewayService, upstream CodexGatewayBorrowProbeUpstream, tlsProfiles *TLSFingerprintProfileService, proxies ProxyRepository) (*CodexGatewayBorrowService, error) { - s := NewCodexGatewayBorrowService(settings, accounts, gateway, upstream, tlsProfiles) - s.proxies = proxies - ctx, cancel := context.WithTimeout(context.Background(), codexGatewayBorrowStoreTimeout) - defer cancel() - if _, err := s.GetConfig(ctx); err != nil { - return nil, err - } - return s, nil -} - -func (s *OpenAIGatewayService) SetCodexGatewayBorrowService(borrow *CodexGatewayBorrowService) { - s.gatewayBorrow = borrow -} - -func (s *CodexGatewayBorrowService) ConfigSnapshot() CodexGatewayBorrowConfig { - if s == nil { - return DefaultCodexGatewayBorrowConfig() - } - s.mu.Lock() - defer s.mu.Unlock() - return cloneCodexGatewayBorrowConfig(s.config) -} - -func (s *CodexGatewayBorrowService) GetConfig(ctx context.Context) (CodexGatewayBorrowConfig, error) { - if s == nil || s.settings == nil { - return DefaultCodexGatewayBorrowConfig(), ErrCodexGatewayBorrowUnavailable - } - s.storeMu.Lock() - defer s.storeMu.Unlock() - dbCtx, cancel := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) - defer cancel() - raw, err := s.settings.GetValue(dbCtx, SettingKeyCodexGatewayBorrowConfig) - cfg := DefaultCodexGatewayBorrowConfig() - if !errors.Is(err, ErrSettingNotFound) { - if err != nil { - return cfg, err - } - cfg, err = DecodeCodexGatewayBorrowConfig([]byte(raw)) - if err != nil { - return cfg, fmt.Errorf("stored gateway borrow configuration: %w", err) - } - } - s.publishConfig(cfg, false) - return cfg, nil -} - -func (s *CodexGatewayBorrowService) SaveConfig(ctx context.Context, cfg CodexGatewayBorrowConfig) (CodexGatewayBorrowConfig, error) { - if s == nil || s.settings == nil || s.accounts == nil { - return cfg, ErrCodexGatewayBorrowUnavailable - } - cfg, err := normalizeCodexGatewayBorrowConfig(cfg) - if err != nil { - return cfg, err - } - if cfg.RotateNodes { - if _, err := s.borrowNodes(ctx, cfg.NodeProxyIDs, true); err != nil { - return cfg, err - } - } - // Validation only loads local rows and resolves existing parent credentials. - // It never discovers models, touches quota or changes any account fields. - for _, id := range append(append([]int64{}, cfg.SourceAccountIDs...), cfg.TargetAccountIDs...) { - a, err := s.accounts.GetByID(ctx, id) - if err != nil || !codexGatewayBorrowAccountSupported(a) { - return cfg, fmt.Errorf("account %d must be an existing OpenAI OAuth-like account", id) - } - if _, err := resolveCredentialAccount(ctx, s.accounts, a); err != nil { - return cfg, fmt.Errorf("account %d has no valid credential parent", id) - } - } - raw, err := json.Marshal(cfg) - if err != nil { - return cfg, err - } - s.storeMu.Lock() - defer s.storeMu.Unlock() - dbCtx, cancel := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) - defer cancel() - if err := s.settings.Set(dbCtx, SettingKeyCodexGatewayBorrowConfig, string(raw)); err != nil { - return cfg, err - } - s.publishConfig(cfg, true) - return cfg, nil -} - -func (s *CodexGatewayBorrowService) publishConfig(cfg CodexGatewayBorrowConfig, saved bool) { - s.mu.Lock() - changed := !s.loaded || !slices.Equal(cfg.SourceAccountIDs, s.config.SourceAccountIDs) || !slices.Equal(cfg.TargetAccountIDs, s.config.TargetAccountIDs) || !slices.Equal(cfg.Models, s.config.Models) || cfg.Enabled != s.config.Enabled || !borrowNodeConfigEqual(cfg, s.config) - if !changed && !saved { - s.mu.Unlock() - return - } - s.cancel() - s.revisionCtx, s.cancel = context.WithCancel(context.Background()) - s.revision++ - s.config, s.loaded = cloneCodexGatewayBorrowConfig(cfg), true - s.candidate = nil - s.candidates = nil - s.acquisition = CodexGatewayBorrowAcquisition{} - s.setup = CodexGatewayBorrowSetup{} - if saved && cfg.Enabled && !cfg.ImmediateReborrowOnFailure { - s.setup = CodexGatewayBorrowSetup{State: "queued", Total: len(cfg.TargetAccountIDs) * len(cfg.Models), StartedAt: time.Now()} - } - s.usage = nil - s.observedSince = time.Now() - s.sources = map[int64]CodexGatewayBorrowSourceStatus{} - s.targets = map[codexGatewayBorrowTargetKey]codexGatewayBorrowTargetCheck{} - s.qualifications = map[string]codexGatewayBorrowTargetCheck{} - s.rotationCandidates = nil - s.prepareFailedUntil, s.prepareFailure, s.preparing, s.prepareRuns = time.Time{}, "", false, 0 - ctx := s.revisionCtx - s.mu.Unlock() - // Exactly one finite preparation per save. Loading settings or restarting - // never calls a model. Individual source/target deadlines remain 90 seconds. - if saved && cfg.Enabled && !cfg.ImmediateReborrowOnFailure { - go func() { - prepareCtx, cancel := context.WithTimeout(ctx, 10*time.Minute) - defer cancel() - _, _ = s.Prepare(prepareCtx) - }() - } -} - -func (s *CodexGatewayBorrowService) Stop() { - if s == nil { - return - } - s.mu.Lock() - s.cancel() - s.mu.Unlock() - s.rotation.stop() -} - -func codexGatewayBorrowAccountSupported(a *Account) bool { - return a != nil && a.IsOpenAIOAuthLike() && !a.IsOpenAIAgentIdentity() && !a.IsSyntheticUITest() -} - -func CodexGatewayBorrowRequestEligible(req *http.Request) bool { - return req != nil && req.URL != nil && req.Method == http.MethodPost && req.URL.Scheme == "https" && req.URL.Host == "chatgpt.com" && req.URL.User == nil && - (req.Host == "" || req.Host == "chatgpt.com") && (req.URL.Path == "/backend-api/codex/responses" || req.URL.Path == "/backend-api/codex/responses/lite") -} - -func (s *CodexGatewayBorrowService) RequestEligible(req *http.Request) bool { - return CodexGatewayBorrowRequestEligible(req) -} -func (s *CodexGatewayBorrowService) ModelEligible(model string) bool { - cfg := s.ConfigSnapshot() - return cfg.Enabled && slices.Contains(cfg.Models, model) -} - -type codexGatewayBorrowModelContextKey struct{} -type codexGatewayBorrowObservationContextKey struct{} - -// The token provider may refresh credentials normally, but must not attach an -// observation failure to account health, scheduling or quota. -func WithCodexGatewayBorrowObservation(ctx context.Context) context.Context { - return context.WithValue(WithAccountObservation(ctx), codexGatewayBorrowObservationContextKey{}, true) -} -func IsCodexGatewayBorrowObservation(ctx context.Context) bool { - return ctx != nil && ctx.Value(codexGatewayBorrowObservationContextKey{}) == true -} - -// Builders stage the model they actually resolved. This gate does not inspect, -// consume, copy or limit the business body (which can be the full 100 MiB). -func WithCodexGatewayBorrowModel(ctx context.Context, model string) context.Context { - return context.WithValue(ctx, codexGatewayBorrowModelContextKey{}, model) -} -func CodexGatewayBorrowModelFromContext(ctx context.Context) string { - if ctx == nil { - return "" - } - model, _ := ctx.Value(codexGatewayBorrowModelContextKey{}).(string) - return model -} - -func (s *CodexGatewayBorrowService) Status() CodexGatewayBorrowStatus { - if s == nil { - return CodexGatewayBorrowStatus{Config: DefaultCodexGatewayBorrowConfig(), Sources: []CodexGatewayBorrowSourceStatus{}, Targets: []CodexGatewayBorrowTargetStatus{}, ModelEfforts: codexGatewayBorrowModelEfforts()} - } - s.mu.Lock() - snapshotConfig, snapshotRevision := cloneCodexGatewayBorrowConfig(s.config), s.revision - s.mu.Unlock() - nodes := s.rotation.snapshot(snapshotConfig, snapshotRevision) - s.mu.Lock() - defer s.mu.Unlock() - if s.revision != snapshotRevision { - nodes = nil - } - now := time.Now() - s.currentCandidateLocked() - out := CodexGatewayBorrowStatus{Nodes: nodes, Enabled: s.config.Enabled, Revision: s.revision, GeneratedAt: now, Preparing: s.preparing || s.prepareRuns > 0 || s.setup.State == "queued", - Acquisition: s.acquisition, ObservedSince: s.observedSince, RecentUsage: []CodexGatewayBorrowUsage{}, Setup: s.setup, Config: cloneCodexGatewayBorrowConfig(s.config), Sources: []CodexGatewayBorrowSourceStatus{}, Targets: []CodexGatewayBorrowTargetStatus{}, ModelEfforts: codexGatewayBorrowModelEfforts()} - if c := s.candidate; c != nil { - out.Candidate = &CodexGatewayBorrowCandidateStatus{SourceAccountID: c.sourceID, SourceRequestEncoding: c.sourceRequestEncoding, ExpiresAt: c.expires, RemainingSeconds: borrowRemaining(c.expires, now), CookieFingerprint: borrowHash(c.cookie.Value)} - if c.node != nil { - out.Candidate.ProxyID, out.Candidate.Gateway = c.node.id, codexBorrowGateway(c.cookie.Value) - out.Candidate.NodeIdentity = c.node.identity - } - } - for _, usage := range s.usage { - out.RecentUsage = append(out.RecentUsage, usage) - } - sort.Slice(out.RecentUsage, func(i, j int) bool { return out.RecentUsage[i].StartedAt.After(out.RecentUsage[j].StartedAt) }) - for _, id := range s.config.SourceAccountIDs { - row, ok := s.sources[id] - if !ok { - row = CodexGatewayBorrowSourceStatus{AccountID: id, State: "waiting", Reason: "not_prepared"} - } - if !s.config.Enabled { - row.State, row.Reason = "disabled", "disabled" - } - if row.ExpiresAt != nil { - row.RemainingSeconds = borrowRemaining(*row.ExpiresAt, now) - if row.State == "ready" && row.RemainingSeconds == 0 { - row.State, row.Reason = "expired", "route_expired" - } - } - - out.Sources = append(out.Sources, row) - } - for _, id := range s.config.TargetAccountIDs { - for _, model := range s.config.Models { - row := CodexGatewayBorrowTargetStatus{CodexGatewayBorrowVerification: CodexGatewayBorrowVerification{AccountID: id, Model: model, Reason: "not_verified"}, State: "waiting"} - if c, ok := s.targets[codexGatewayBorrowTargetKey{id, model}]; ok { - candidate := s.candidate - if s.config.RotateNodes { - candidate = s.rotationCandidates[codexGatewayBorrowTargetKey{id, model}] - } - row.CodexGatewayBorrowVerification = c.result - row.Acquisition = c.acquisition - row.CacheValid = c.policyRevision == currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision && c.result.Success && candidate != nil && c.cookieKey == borrowHash(candidate.cookie.Value) && now.Before(c.expires) && now.Before(candidate.expires) - expires := c.expires - if candidate != nil && c.cookieKey == borrowHash(candidate.cookie.Value) && candidate.expires.Before(expires) { - expires = candidate.expires - } - if !expires.IsZero() { - row.ExpiresAt = &expires - } - row.RemainingSeconds = borrowRemaining(expires, now) - row.State = "rejected" - if row.CacheValid { - row.State = "ready" - } else if c.validating { - row.State = "validating" - } else if c.result.RetryOnNextRequest { - row.State = "retry_pending" - } else if c.result.Success { - row.State, row.Reason = "expired", "route_expired" - } - if c.result.Success && candidate != nil && c.cookieKey != borrowHash(candidate.cookie.Value) && now.Before(c.expires) { - row.State, row.Reason = "waiting", "target_route_changed" - } - if c.policyRevision != currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision && c.result.Success { - row.State, row.Reason = "waiting", "identity_changed" - } - if now.Before(c.retryAfter) && !c.result.Success { - retry := c.retryAfter - row.RetryAfter = &retry - } - } - if s.config.RotateNodes && row.NodeIdentity != "" { - // The child probe's short cache is not the node's quiet window. - // Report the selected node snapshot, including after lease expiry. - row.RetryAfter, row.Acquisition.RetryAfter = nil, nil - for _, node := range nodes { - if node.AccountID == id && node.Model == model && node.NodeIdentity == row.NodeIdentity && node.RetryAfter != nil && now.Before(*node.RetryAfter) { - if row.CacheValid { - // A late failure on another node may cool this gateway - // after the stored probe passed. Match dispatch eligibility. - row.CacheValid, row.State, row.Reason = false, "rejected", "node_cooling" - } - retry := *node.RetryAfter - row.RetryAfter, row.Acquisition.RetryAfter = &retry, &retry - break - } - } - } - if !s.config.Enabled { - row.State, row.Reason, row.CacheValid = "disabled", "disabled", false - } - - out.Targets = append(out.Targets, row) - } - } - return out -} - -func borrowRemaining(expires, now time.Time) int64 { - remaining := int64(expires.Sub(now).Seconds()) - if remaining < 0 { - return 0 - } - return remaining -} -func borrowHash(value string) string { - h := sha256.Sum256([]byte(value)) - return hex.EncodeToString(h[:]) -} - -func borrowRevisionContext(parent, revision context.Context, timeout time.Duration) (context.Context, context.CancelFunc) { - ctx, cancel := context.WithTimeout(parent, timeout) - stop := context.AfterFunc(revision, cancel) - return ctx, func() { stop(); cancel() } -} - -func (s *CodexGatewayBorrowService) Prepare(ctx context.Context) (CodexGatewayBorrowStatus, error) { - if s == nil { - return s.Status(), ErrCodexGatewayBorrowUnavailable - } - s.mu.Lock() - cfg, rev, revisionCtx := cloneCodexGatewayBorrowConfig(s.config), s.revision, s.revisionCtx - s.mu.Unlock() - if !cfg.Enabled { - return s.Status(), ErrCodexGatewayBorrowUnavailable - } - _, err := s.setupFlight.do(ctx, revisionCtx, fmt.Sprint(rev), 10*time.Minute, func(runCtx context.Context) (any, error) { - progress := CodexGatewayBorrowSetup{State: "running", Phase: "source", Total: len(cfg.TargetAccountIDs) * len(cfg.Models), StartedAt: time.Now()} - s.setSetup(rev, progress) - s.mu.Lock() - if s.revision != rev { - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - s.prepareRuns++ - s.mu.Unlock() - defer func() { - s.mu.Lock() - if s.revision == rev { - s.prepareRuns-- - } - s.mu.Unlock() - }() - finish := func(err error) { - now := time.Now() - progress.FinishedAt = &now - progress.Phase = "complete" - if err != nil { - progress.Error = err.Error() - progress.State = "failed" - } else { - ready := 0 - for _, row := range s.Status().Targets { - if row.CacheValid { - ready++ - } - } - progress.State = "failed" - if ready == progress.Total { - progress.State = "ready" - } else if ready > 0 { - progress.State = "partial" - } - } - s.setSetup(rev, progress) - } - if !cfg.RotateNodes { - if err := s.prepareSource(runCtx, rev, revisionCtx); err != nil { - finish(err) - return nil, err - } - } - var lastErr error - for _, id := range cfg.TargetAccountIDs { - for _, model := range cfg.Models { - if err := runCtx.Err(); err != nil { - finish(err) - return nil, err - } - progress.Phase, progress.AccountID, progress.Model = "target", id, model - s.setSetup(rev, progress) - a, req, proxy, itemErr := s.accountTemplate(runCtx, id, model) - if itemErr == nil { - _, _, itemErr = s.Apply(req, a, model, proxy, nil, false) - } else { - s.mu.Lock() - if s.revision == rev { - s.targets[codexGatewayBorrowTargetKey{id, model}] = codexGatewayBorrowTargetCheck{result: CodexGatewayBorrowVerification{AccountID: id, Model: model, CheckedAt: time.Now(), Reason: "account_unavailable", Error: itemErr.Error()}} - } - s.mu.Unlock() - } - progress.Completed++ - if itemErr != nil { - progress.Failed++ - lastErr = itemErr - } - s.setSetup(rev, progress) - } - } - if progress.Failed == progress.Total { - finish(lastErr) - return nil, lastErr - } - finish(nil) - return nil, nil - }) - return s.Status(), err -} - -func (s *CodexGatewayBorrowService) prepareSource(ctx context.Context, rev uint64, revisionCtx context.Context) error { - if s == nil { - return ErrCodexGatewayBorrowUnavailable - } - trigger, _ := ctx.Value(borrowAcquisitionTriggerKey{}).(string) - fresh, _ := ctx.Value(borrowFreshSourceKey{}).(bool) - if trigger == "" { - trigger = "save_or_manual" - } - _, err := s.sourceFlight.do(ctx, revisionCtx, fmt.Sprintf("%d:%t", rev, fresh), codexGatewayBorrowTimeout, func(ctx context.Context) (any, error) { - s.mu.Lock() - if s.revision != rev || !s.config.Enabled { - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - if c := s.currentCandidateLocked(); !fresh && c != nil && time.Until(c.expires) > codexGatewayBorrowTimeout { - s.mu.Unlock() - return nil, nil - } - if s.accounts == nil || s.gateway == nil { - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowUnavailable - } - if !fresh && time.Now().Before(s.prepareFailedUntil) { - detail, retry := s.prepareFailure, s.prepareFailedUntil - s.mu.Unlock() - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "source_preparation", Reason: "source_prepare_failed", Detail: detail, RetryAfter: &retry} - } - ids := append([]int64{}, s.config.SourceAccountIDs...) - if s.sources == nil { - s.sources = make(map[int64]CodexGatewayBorrowSourceStatus) - } - now := time.Now() - s.acquisition = CodexGatewayBorrowAcquisition{Trigger: trigger, Phase: "acquiring", StartedAt: &now} - s.preparing = true - s.sourceSequence++ - sequence := s.sourceSequence - s.mu.Unlock() - probeCtx, cancel := borrowRevisionContext(ctx, revisionCtx, codexGatewayBorrowTimeout) - defer func() { - cancel() - s.mu.Lock() - if s.revision == rev && s.sourceSequence == sequence { - s.preparing = false - finished := time.Now() - s.acquisition.FinishedAt = &finished - if s.acquisition.Phase == "acquiring" { - s.acquisition.Phase, s.acquisition.Reason = "cancelled", "cancelled" - } - } - s.mu.Unlock() - }() - var lastErr error = ErrCodexGatewayBorrowUnavailable - for _, id := range ids { - if probeCtx.Err() != nil { - lastErr = probeCtx.Err() - break - } - candidate, detail, acquireErr := s.acquireSource(probeCtx, id) - now := time.Now() - s.mu.Lock() - if s.revision != rev || revisionCtx.Err() != nil || s.sourceSequence != sequence || errors.Is(probeCtx.Err(), context.Canceled) { - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - row := CodexGatewayBorrowSourceStatus{AccountID: id, State: "rejected", Reason: "source_probe_failed", CheckedAt: &now, Error: detail} - if acquireErr == nil && candidate != nil && now.Before(candidate.expires) { - s.storeCandidateLocked(candidate) - row.State, row.Reason, row.Error, row.ExpiresAt = "ready", "source_qualified", "", &candidate.expires - if candidate.node != nil { - row.ProxyID = candidate.node.id - row.NodeIdentity = candidate.node.identity - } - s.sources[id] = row - s.acquisition.Phase = "candidate_acquired" - s.prepareFailedUntil, s.prepareFailure = time.Time{}, "" - s.mu.Unlock() - return nil, nil - } - if acquireErr != nil { - lastErr = acquireErr - if detail != "" { - lastErr = errors.Join(acquireErr, errors.New(detail)) - } - } - s.sources[id] = row - // Original non-rotation preparation may still have a usable primary or - // backup after a failed refresh. Preserve it without extending its lease. - if current := s.currentCandidateLocked(); !fresh && current != nil && time.Now().Before(current.expires) { - s.acquisition.Phase, s.acquisition.Error = "candidate_retained", lastErr.Error() - s.prepareFailedUntil, s.prepareFailure = time.Time{}, "" - s.mu.Unlock() - return nil, nil - } - s.mu.Unlock() - } - s.mu.Lock() - if s.revision == rev && s.sourceSequence == sequence { - s.acquisition.Phase, s.acquisition.Reason, s.acquisition.Error = "failed", "source_prepare_failed", lastErr.Error() - if !errors.Is(probeCtx.Err(), context.Canceled) && !errors.Is(lastErr, context.Canceled) && !errors.Is(lastErr, ErrCodexBorrowDiagnosticBudget) { - s.prepareFailedUntil = time.Now().Add(codexGatewayBorrowFailureWait) - s.prepareFailure = lastErr.Error() - retry := s.prepareFailedUntil - s.acquisition.RetryAfter = &retry - } - } - s.mu.Unlock() - return nil, lastErr - }) - return err -} - -func borrowRequestFingerprint(req *http.Request, model, proxy string, profile *tlsfingerprint.Profile, cookie string) string { - tlsJSON, _ := json.Marshal(profile) - // Original non-rotation qualification identity, plus the actual model for - // the downstream Sol extension. Request tracing and business tiers do not - // add probe rules. Business STATE intentionally invalidates qualification. - values := []string{cookie, proxy, req.Header.Get("Authorization"), req.Header.Get("ChatGPT-Account-ID"), req.Header.Get("User-Agent"), req.Header.Get("Originator"), req.Header.Get("Version"), req.Header.Get("X-Codex-Turn-State"), string(tlsJSON), model} - identity, _ := json.Marshal(values) - return borrowHash(string(identity)) -} - -func borrowTargetFingerprint(req *http.Request, account *Account, model, proxy string, profile *tlsfingerprint.Profile, cookie string) string { - return borrowHash(borrowRequestFingerprint(req, model, proxy, profile, cookie) + fmt.Sprint(currentCodexFingerprintPolicyForAccount(account).revision)) -} - -// Apply changes only __oailb on a request clone. The target's auth, STATE, -// __cflb, other cookies, body and proxy stay exactly as supplied by its caller. -// cachedOnly is used by manual generation: it never acquires or validates. -func (s *CodexGatewayBorrowService) Apply(req *http.Request, account *Account, model, proxy string, profile *tlsfingerprint.Profile, cachedOnly bool) (*http.Request, *CodexGatewayBorrowApplication, error) { - wire, application, err := s.apply(req, account, model, proxy, profile, cachedOnly, false) - if err != nil && !IsCodexGatewayBorrowFailure(err) { - err = &CodexGatewayBorrowFailure{Cause: err} - } - var diagnostic *codexBorrowDiagnostic - if req != nil { - diagnostic = borrowDiagnosticFromContext(req.Context()) - } - if d := diagnostic; d != nil { - if application != nil { - result := application.Verification - d.verification.Store(&result) - } else { - var failure *CodexGatewayBorrowFailure - if errors.As(err, &failure) { - d.verification.Store(failure.Verification) - } - } - } - return wire, application, err -} - -func (s *CodexGatewayBorrowService) apply(req *http.Request, account *Account, model, proxy string, profile *tlsfingerprint.Profile, cachedOnly, force bool) (*http.Request, *CodexGatewayBorrowApplication, error) { - if s == nil || !CodexGatewayBorrowRequestEligible(req) || account == nil { - return req, nil, nil - } - if d := borrowDiagnosticFromContext(req.Context()); d != nil && !d.borrow { - return req, nil, nil - } - s.mu.Lock() - if !s.config.Enabled || !slices.Contains(s.config.Models, model) || !slices.Contains(s.config.TargetAccountIDs, account.ID) { - s.mu.Unlock() - return req, nil, nil - } - rev, revisionCtx := s.revision, s.revisionCtx - rotation := s.config.RotateNodes - candidate := s.currentCandidateLocked() - s.mu.Unlock() - if !codexGatewayBorrowAccountSupported(account) { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - if rotation { - return s.applyRotated(req, account, model, proxy, profile, cachedOnly, force, rev, revisionCtx) - } - if candidate == nil || !time.Now().Before(candidate.expires) { - if cachedOnly { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - if err := s.prepareSource(withBorrowAcquisitionTrigger(req.Context(), candidate), rev, revisionCtx); err != nil { - if IsCodexGatewayBorrowFailure(err) { - return req, nil, err - } - return req, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "source_preparation", Reason: "source_prepare_failed"} - } - s.mu.Lock() - candidate = s.currentCandidateLocked() - s.mu.Unlock() - } - if candidate == nil || !time.Now().Before(candidate.expires) || !borrowCookiePathMatches(req.URL.Path, candidate.cookie.Path) { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - policyRevision := currentCodexFingerprintPolicyForAccount(account).revision - key := borrowTargetFingerprint(req, account, model, proxy, profile, candidate.cookie.Value) - targetKey := codexGatewayBorrowTargetKey{account.ID, model} - // Cached proofs do not wait for preparation on another target or identity. - if !force { - if check, ok, err := s.cachedTarget(rev, targetKey, key, candidate); ok || err != nil { - if err != nil { - return req, nil, err - } - return borrowApplyCookie(req, candidate, check), borrowApplication(candidate, key, check.expires, check.result), nil - } - } - if cachedOnly { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - value, err := s.targetFlight.do(req.Context(), revisionCtx, fmt.Sprintf("%d:%d:%s:%s", rev, account.ID, model, key), codexGatewayBorrowTimeout, func(operation context.Context) (any, error) { - if !force { - if check, ok, err := s.cachedTarget(rev, targetKey, key, candidate); ok || err != nil { - if err != nil { - return nil, err - } - return check, nil - } - } - s.mu.Lock() - if operation.Err() != nil || !s.candidateCurrentLocked(rev, candidate) { - if s.revision == rev { - delete(s.targets, targetKey) - } - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - started := time.Now() - trigger := "qualification_missing" - if force { - trigger = "manual_validation" - } - acquisition := CodexGatewayBorrowAcquisition{Trigger: trigger, Phase: "validating", StartedAt: &started} - s.validationSequence++ - validationID := s.validationSequence - s.targets[targetKey] = codexGatewayBorrowTargetCheck{acquisition: acquisition, validationID: validationID, policyRevision: policyRevision, key: key, cookieKey: borrowHash(candidate.cookie.Value), validating: true, - result: CodexGatewayBorrowVerification{AccountID: account.ID, Model: model, Reason: "validating", CheckedAt: time.Now()}} - s.mu.Unlock() - probeCtx, cancel := borrowRevisionContext(operation, revisionCtx, codexGatewayBorrowTimeout) - probeCtx, policyCancel := borrowPolicyContext(probeCtx, account.ID, policyRevision) - defer policyCancel() - defer cancel() - credential, credentialErr := resolveCredentialAccount(probeCtx, s.accounts, account) - identityRevision := borrowAccountIdentityRevision(account, credential) - var result CodexGatewayBorrowVerification - if credentialErr != nil { - result = CodexGatewayBorrowVerification{AccountID: account.ID, Model: model, CheckedAt: time.Now(), Reason: "account_unavailable", Error: credentialErr.Error()} - } else { - result = s.probeTarget(probeCtx, req, account, model, proxy, profile, candidate) - } - s.mu.Lock() - if errors.Is(operation.Err(), context.Canceled) || currentCodexFingerprintPolicyForAccount(account).revision != policyRevision || !s.candidateCurrentLocked(rev, candidate) { - if s.revision == rev && s.targets[targetKey].validationID == validationID { - delete(s.targets, targetKey) - } - s.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - finished := time.Now() - acquisition.FinishedAt, acquisition.Reason, acquisition.Error = &finished, result.Reason, result.Error - acquisition.Phase = "ready" - if !result.Success { - acquisition.Phase = "failed" - } - check := codexGatewayBorrowTargetCheck{acquisition: acquisition, validationID: validationID, identityRevision: identityRevision, policyRevision: policyRevision, key: key, cookieKey: borrowHash(candidate.cookie.Value), expires: candidate.expires, result: result} - if !result.Success && !result.RetryOnNextRequest { - check.retryAfter = time.Now().Add(codexGatewayBorrowFailureWait) - check.acquisition.RetryAfter = &check.retryAfter - } - if s.targets[targetKey].validationID == validationID { - s.targets[targetKey] = check - } - if s.qualifications == nil { - s.qualifications = make(map[string]codexGatewayBorrowTargetCheck) - } - qualificationKey := fmt.Sprintf("%d:%s:%s", account.ID, model, key) - // Bound request-identity variants; expired proofs are discarded. - if len(s.qualifications) >= 1024 { - oldestKey := "" - var oldest time.Time - for k, v := range s.qualifications { - if !time.Now().Before(v.expires) { - delete(s.qualifications, k) - } else if oldestKey == "" || v.result.CheckedAt.Before(oldest) { - oldestKey, oldest = k, v.result.CheckedAt - } - } - if len(s.qualifications) >= 1024 { - delete(s.qualifications, oldestKey) - } - } - s.qualifications[qualificationKey] = check - s.mu.Unlock() - if !result.Success { - return nil, borrowVerificationFailure(check, rev) - } - return check, nil - }) - if err != nil { - return req, nil, err - } - check, ok := value.(codexGatewayBorrowTargetCheck) - if !ok { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - return borrowApplyCookie(req, candidate, check), borrowApplication(candidate, key, check.expires, check.result), nil -} - -func (s *CodexGatewayBorrowService) cachedTarget(rev uint64, key codexGatewayBorrowTargetKey, fingerprint string, candidate *codexGatewayBorrowCandidate) (codexGatewayBorrowTargetCheck, bool, error) { - s.mu.Lock() - defer s.mu.Unlock() - if !s.candidateCurrentLocked(rev, candidate) { - return codexGatewayBorrowTargetCheck{}, false, ErrCodexGatewayBorrowChanged - } - old, ok := s.qualifications[fmt.Sprintf("%d:%s:%s", key.accountID, key.model, fingerprint)] - if !ok { - old, ok = s.targets[key] - } - if !ok || old.key != fingerprint || old.validating { - return old, false, nil - } - now := time.Now() - if !now.Before(old.expires) { - return old, false, nil - } - if old.result.Success && now.Before(old.expires) { - if candidate.expires.Before(old.expires) { - old.expires = candidate.expires - } - return old, true, nil - } - if !old.result.Success && !old.result.RetryOnNextRequest && now.Before(old.retryAfter) { - failure := borrowVerificationFailure(old, rev) - failure.CacheHit = true - return old, false, failure - } - return old, false, nil -} - -func borrowApplication(candidate *codexGatewayBorrowCandidate, fingerprint string, expires time.Time, verification ...CodexGatewayBorrowVerification) *CodexGatewayBorrowApplication { - app := &CodexGatewayBorrowApplication{Applied: true, SourceAccountID: candidate.sourceID, ExpiresAt: expires, Fingerprint: fingerprint, CookieFingerprint: borrowHash(candidate.cookie.Value)} - if len(verification) > 0 { - app.Verification = verification[0] - } - return app -} - -func borrowApplyCookie(req *http.Request, candidate *codexGatewayBorrowCandidate, check codexGatewayBorrowTargetCheck) *http.Request { - clone := req.Clone(context.WithValue(req.Context(), codexBorrowAppliedContextKey{}, true)) - replaceCodexGatewayBorrowCookie(clone.Header, "__oailb", candidate.cookie.Value) - return clone -} - -func replaceCodexGatewayBorrowCookie(headers http.Header, name, value string) { - var kept []string - for _, header := range headers.Values("Cookie") { - for _, part := range strings.Split(header, ";") { - part = strings.TrimSpace(part) - cookieName, _, ok := strings.Cut(part, "=") - if part != "" && (!ok || strings.TrimSpace(cookieName) != name) { - kept = append(kept, part) - } - } - } - if value != "" { - kept = append(kept, (&http.Cookie{Name: name, Value: value}).String()) - } - if len(kept) == 0 { - headers.Del("Cookie") - } else { - headers.Set("Cookie", strings.Join(kept, "; ")) - } -} - -func borrowCookiePathMatches(path, scope string) bool { - return path == scope || (strings.HasPrefix(path, scope) && (strings.HasSuffix(scope, "/") || strings.HasPrefix(strings.TrimPrefix(path, scope), "/"))) -} - -func (s *CodexGatewayBorrowService) Verify(ctx context.Context, accountID int64, model string) (CodexGatewayBorrowVerification, error) { - result := CodexGatewayBorrowVerification{AccountID: accountID, Model: model, CheckedAt: time.Now(), Reason: "not_configured"} - cfg := s.ConfigSnapshot() - if !cfg.Enabled || !slices.Contains(cfg.TargetAccountIDs, accountID) || !slices.Contains(cfg.Models, model) { - return result, ErrCodexGatewayBorrowUnavailable - } - a, req, proxy, err := s.accountTemplate(ctx, accountID, model) - if err != nil { - return result, err - } - _, _, err = s.apply(req, a, model, proxy, nil, false, true) - s.mu.Lock() - if check, ok := s.targets[codexGatewayBorrowTargetKey{accountID, model}]; ok { - result = check.result - } - s.mu.Unlock() - return result, err -} diff --git a/backend/internal/service/codex_gateway_borrow_candidate_retry_test.go b/backend/internal/service/codex_gateway_borrow_candidate_retry_test.go deleted file mode 100644 index 997cf2bbc226..000000000000 --- a/backend/internal/service/codex_gateway_borrow_candidate_retry_test.go +++ /dev/null @@ -1,90 +0,0 @@ -package service - -import ( - "context" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" - "net/http" - "testing" - "time" -) - -func TestCodexBorrowNonRotationRejectsWithoutReplacingSource(t *testing.T) { - var sources, targets int - account, other := borrowCoreAccount(2), borrowCoreAccount(3) - reject := true - s := newBorrowCoreTest(t, func(req *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 { - sources++ - return borrowCoreResponse("gpt-6-astra", "OK", "source", "__oailb=fresh; Secure; Path=/; Max-Age=230"), nil - } - targets++ - state := "first" - if id == 2 && reject && req.Header.Get("X-Codex-Turn-State") != "" { - state = "changed" - } - return borrowCoreResponse("gpt-6.1-sol", "OK", state), nil - }, borrowCoreAccount(1), account, other) - borrowCoreCandidate(s, time.Now().Add(120*time.Second)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - original := s.candidate - for range 2 { - _, applied, err := s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.Nil(t, applied) - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Equal(t, "target_state_changed", failure.Reason) - require.NotNil(t, failure.RetryAfter) - } - require.Zero(t, sources) - require.Equal(t, 2, targets, "cooldown retains verdict without another probe") - require.Same(t, original, s.candidate, "target failure never retires source") - _, otherReq, _, err := s.accountTemplate(context.Background(), 3, "gpt-6.1-sol") - require.NoError(t, err) - _, app, err := s.Apply(otherReq, other, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.True(t, app.Applied) - reject = false - for key, check := range s.qualifications { - check.retryAfter = time.Now().Add(-time.Second) - s.qualifications[key] = check - } - _, app, err = s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.True(t, app.Applied, "next request resumes after cooldown") - require.Equal(t, 6, targets) - require.Zero(t, sources) - original.expires = time.Now().Add(-time.Second) - _, app, err = s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.True(t, app.Applied) - require.Equal(t, 1, sources) - require.Equal(t, 8, targets) - status := s.Status() - require.Equal(t, "expired_route", status.Acquisition.Trigger) - require.Equal(t, "candidate_acquired", status.Acquisition.Phase) - require.NotNil(t, status.Acquisition.FinishedAt) - require.False(t, status.Preparing) -} -func TestCodexBorrowSourcePoolPreservesOtherCandidates(t *testing.T) { - s := newBorrowCoreTest(t, nil) - borrowCoreCandidate(s, time.Now().Add(time.Minute)) - first := s.candidate - s.storeCandidateLocked(first) - second := *first - second.sourceID, second.cookie.Value = 4, "second" - s.config.SourceAccountIDs = []int64{1, 4} - s.storeCandidateLocked(&second) - require.True(t, s.candidateCurrentLocked(s.revision, first)) - repeated := *first - repeated.expires = time.Now().Add(230 * time.Second) - s.storeCandidateLocked(&repeated) - require.Equal(t, first.expires, repeated.expires) - repeated.expires = time.Now().Add(-time.Second) - require.Same(t, &second, s.currentCandidateLocked()) - oldRev := s.revision - s.publishConfig(CodexGatewayBorrowConfig{}, false) - require.False(t, s.candidateCurrentLocked(oldRev, &second)) - require.Empty(t, s.candidates) -} diff --git a/backend/internal/service/codex_gateway_borrow_candidates.go b/backend/internal/service/codex_gateway_borrow_candidates.go deleted file mode 100644 index 1161bfa9dc04..000000000000 --- a/backend/internal/service/codex_gateway_borrow_candidates.go +++ /dev/null @@ -1,92 +0,0 @@ -package service - -import ( - "context" - "time" -) - -// Candidates belong to sources, never to a target verdict. The active source -// is preferred until expiry; a target rejection only cools that qualification. -func (s *CodexGatewayBorrowService) currentCandidateLocked() *codexGatewayBorrowCandidate { - if s.candidate != nil && time.Now().Before(s.candidate.expires) { - return s.candidate - } - for _, id := range s.config.SourceAccountIDs { - if c := s.candidates[id]; c != nil && time.Now().Before(c.expires) { - s.candidate = c - return c - } - } - return s.candidate // Retain the expired candidate for status and trigger evidence. -} - -func (s *CodexGatewayBorrowService) storeCandidateLocked(candidate *codexGatewayBorrowCandidate) { - if s.candidates == nil { - s.candidates = make(map[int64]*codexGatewayBorrowCandidate) - } - now := time.Now() - clamp := func(old *codexGatewayBorrowCandidate) { - if old != nil && now.Before(old.expires) && old.cookie.Value == candidate.cookie.Value && old.expires.Before(candidate.expires) { - candidate.expires = old.expires - } - } - clamp(s.candidate) - for _, old := range s.candidates { - clamp(old) - } - s.candidates[candidate.sourceID] = candidate - // A backup does not displace a live primary from a different source. - if s.candidate == nil || !now.Before(s.candidate.expires) || s.candidate.sourceID == candidate.sourceID { - s.candidate = candidate - } - // Other source proofs remain usable. Cookie, model and identity are already - // in their keys, so no target can accidentally inherit another qualification. -} - -func (s *CodexGatewayBorrowService) liveCookieLocked(key string) bool { - for _, c := range s.rotationCandidates { - if c != nil && borrowHash(c.cookie.Value) == key && time.Now().Before(c.expires) { - return true - } - } - if c := s.candidate; c != nil && borrowHash(c.cookie.Value) == key && time.Now().Before(c.expires) { - return true - } - for _, c := range s.candidates { - if borrowHash(c.cookie.Value) == key && time.Now().Before(c.expires) { - return true - } - } - return false -} - -func (s *CodexGatewayBorrowService) candidateCurrentLocked(rev uint64, candidate *codexGatewayBorrowCandidate) bool { - if candidate == nil || s.revision != rev || !s.config.Enabled || s.revisionCtx.Err() != nil || !time.Now().Before(candidate.expires) { - return false - } - current := s.candidates[candidate.sourceID] - if current == nil { - current = s.candidate - } - return current != nil && current.sourceID == candidate.sourceID && current.cookie.Value == candidate.cookie.Value && current.cookie.Path == candidate.cookie.Path && current.expires.Equal(candidate.expires) -} - -type CodexGatewayBorrowAcquisition struct { - Trigger string `json:"trigger,omitempty"` - Phase string `json:"phase,omitempty"` - StartedAt *time.Time `json:"started_at,omitempty"` - FinishedAt *time.Time `json:"finished_at,omitempty"` - Reason string `json:"reason,omitempty"` - Error string `json:"error,omitempty"` - RetryAfter *time.Time `json:"retry_after,omitempty"` -} - -type borrowAcquisitionTriggerKey struct{} - -func withBorrowAcquisitionTrigger(ctx context.Context, candidate *codexGatewayBorrowCandidate) context.Context { - trigger := "missing_route" - if candidate != nil { - trigger = "expired_route" - } - return context.WithValue(ctx, borrowAcquisitionTriggerKey{}, trigger) -} diff --git a/backend/internal/service/codex_gateway_borrow_cookie_contract_test.go b/backend/internal/service/codex_gateway_borrow_cookie_contract_test.go deleted file mode 100644 index 2e2122b04b0a..000000000000 --- a/backend/internal/service/codex_gateway_borrow_cookie_contract_test.go +++ /dev/null @@ -1,174 +0,0 @@ -package service - -import ( - "context" - "errors" - "io" - "net/http" - "strings" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -type borrowHeaderRejectedBody struct{ reads, closes int } - -func (b *borrowHeaderRejectedBody) Read([]byte) (int, error) { - b.reads++ - return 0, io.ErrUnexpectedEOF -} -func (b *borrowHeaderRejectedBody) Close() error { b.closes++; return nil } - -// Fixed contract: ranxi fd1b5ee4 ProbeOpenAICodexStateRoute checks every -// replacement __oailb at response headers, before consuming a potentially -// stalled/error body. Cookie scope does not hide a target route replacement. -func TestCodexBorrowPinnedRouteRejectsAtHeaders(t *testing.T) { - for _, cookie := range []string{ - "__oailb=changed; Path=/unrelated; Domain=other.invalid", - "__oailb=synthetic-borrowed-cookie; Max-Age=230; Expires=Thu, 01 Jan 1970 00:00:00 GMT", - "__oailb=synthetic-borrowed-cookie; Max-Age=0", - } { - body := &borrowHeaderRejectedBody{} - calls := 0 - account := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - return &http.Response{StatusCode: 200, Header: http.Header{"Set-Cookie": []string{cookie}}, Body: body}, nil - }, account) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), account.ID, "gpt-6-astra") - require.NoError(t, err) - _, applied, err := s.Apply(req, account, "gpt-6-astra", "", nil, false) - require.Nil(t, applied) - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Equal(t, "target_route_changed", failure.Reason) - require.True(t, failure.Verification.RouteChanged) - require.False(t, failure.Verification.MintCompleted) - require.Zero(t, body.reads) - require.Equal(t, 1, body.closes) - require.Equal(t, 1, calls) - } -} - -func TestCodexBorrowPinnedContinuationUsesFirstUsableCflb(t *testing.T) { - account := borrowCoreAccount(2) - calls := 0 - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - if calls == 1 { - return borrowCoreResponse("gpt-6-astra", "OK", "state", - "__cflb=deleted; Max-Age=0", "__cflb=first; Path=/other; Expires=Thu, 01 Jan 1970 00:00:00 GMT", "__cflb=later"), nil - } - require.Equal(t, "__cflb=first; __oailb=synthetic-borrowed-cookie", req.Header.Get("Cookie")) - return borrowCoreResponse("gpt-6-astra", "OK", "state"), nil - }, account) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), account.ID, "gpt-6-astra") - require.NoError(t, err) - _, applied, err := s.Apply(req, account, "gpt-6-astra", "", nil, false) - require.NoError(t, err) - require.True(t, applied.Applied) - require.Equal(t, 2, calls) -} - -func TestCodexBorrowSourceRequiresSSEAndExplicitCookieLifetime(t *testing.T) { - for _, test := range []struct{ contentType, cookie string }{ - {"application/json", "__oailb=value; Secure; Path=/; Max-Age=230"}, - {"text/event-stream", "__oailb=value; Secure; Path=/"}, - } { - source := borrowCoreAccount(1) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - response := borrowCoreResponse("gpt-6-astra", "OK", "state", test.cookie) - response.Header.Set("Content-Type", test.contentType) - return response, nil - }, source) - candidate, _, err := s.acquireSource(context.Background(), 1) - require.Error(t, err) - require.Nil(t, candidate) - } -} - -func TestCodexBorrowSourceRefreshFailureKeepsLivePrimary(t *testing.T) { - calls := 0 - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - return nil, errors.New("synthetic refresh failure") - }, borrowCoreAccount(1)) - borrowCoreCandidate(s, time.Now().Add(50*time.Second)) - primary := s.candidate - backup := *primary - backup.sourceID = 4 - backup.cookie.Value = "backup" - s.storeCandidateLocked(&backup) - require.Same(t, primary, s.currentCandidateLocked(), "a backup does not replace a live primary") - require.NoError(t, s.prepareSource(context.Background(), s.revision, s.revisionCtx)) - require.Same(t, primary, s.currentCandidateLocked()) - require.Equal(t, 1, calls) - require.True(t, s.prepareFailedUntil.IsZero()) - require.Equal(t, "candidate_retained", s.Status().Acquisition.Phase) -} - -func TestCodexBorrowPinnedTargetRequiresOriginalCompletedStream(t *testing.T) { - for _, kind := range []string{"valid", "unterminated", "late_error", "done_alias"} { - t.Run(kind, func(t *testing.T) { - account := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - body := borrowCoreSSE("gpt-6.1-sol", "OK") - switch kind { - case "unterminated": - body = strings.TrimSuffix(body, "\n") - case "late_error": - body += "data: {\"type\":\"error\",\"error\":{\"message\":\"retained failure\"}}\n\n" - case "done_alias": - body = strings.ReplaceAll(body, "response.completed", "response.done") - } - return &http.Response{StatusCode: 200, Header: http.Header{"X-Codex-Turn-State": []string{"state"}}, Body: io.NopCloser(strings.NewReader(body))}, nil - }, account) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), account.ID, "gpt-6.1-sol") - require.NoError(t, err) - _, applied, err := s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - if kind == "valid" { - require.NoError(t, err) - require.True(t, applied.Applied) - } else { - require.Error(t, err) - require.Nil(t, applied) - } - }) - } -} - -type borrowProbeWaitForEOF struct { - io.Reader - ctx context.Context -} - -func (b *borrowProbeWaitForEOF) Read(data []byte) (int, error) { - n, err := b.Reader.Read(data) - if err == io.EOF { - <-b.ctx.Done() - return n, b.ctx.Err() - } - return n, err -} -func (b *borrowProbeWaitForEOF) Close() error { return nil } -func TestCodexBorrowPinnedTargetDoesNotCertifyBeforeEOF(t *testing.T) { - account := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - return &http.Response{StatusCode: 200, Header: http.Header{"X-Codex-Turn-State": []string{"state"}}, Body: &borrowProbeWaitForEOF{Reader: strings.NewReader(borrowCoreSSE("gpt-6-astra", "OK")), ctx: req.Context()}}, nil - }, account) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), account.ID, "gpt-6-astra") - require.NoError(t, err) - ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) - defer cancel() - result := s.probeTarget(ctx, req, account, "gpt-6-astra", "", nil, s.candidate) - require.False(t, result.Success) - require.False(t, result.MintCompleted) - require.Zero(t, result.ContinueStatus) - require.ErrorIs(t, ctx.Err(), context.DeadlineExceeded) -} diff --git a/backend/internal/service/codex_gateway_borrow_coordination.go b/backend/internal/service/codex_gateway_borrow_coordination.go deleted file mode 100644 index 625b412933f6..000000000000 --- a/backend/internal/service/codex_gateway_borrow_coordination.go +++ /dev/null @@ -1,94 +0,0 @@ -package service - -import ( - "context" - "sync" - "time" -) - -// A preparation belongs to its configuration revision, not to the first caller. -// Cancelling a waiter only cancels the work when no other waiter needs it. -type codexBorrowFlights struct { - mu sync.Mutex - active map[string]*codexBorrowFlight -} - -type codexBorrowFlight struct { - done chan struct{} - cancel context.CancelFunc - waiters int - value any - err error -} - -func (g *codexBorrowFlights) do(ctx, revision context.Context, key string, timeout time.Duration, run func(context.Context) (any, error)) (any, error) { - if err := ctx.Err(); err != nil { - return nil, err - } - g.mu.Lock() - if g.active == nil { - g.active = make(map[string]*codexBorrowFlight) - } - f := g.active[key] - if f == nil { - operation, cancel := context.WithTimeout(context.WithoutCancel(ctx), timeout) - stop := context.AfterFunc(revision, cancel) - f = &codexBorrowFlight{done: make(chan struct{}), cancel: cancel} - g.active[key] = f - go func() { - defer stop() - defer cancel() - f.value, f.err = run(operation) - g.mu.Lock() - if g.active[key] == f { - delete(g.active, key) - } - close(f.done) - g.mu.Unlock() - }() - } - f.waiters++ - g.mu.Unlock() - defer func() { - g.mu.Lock() - f.waiters-- - if f.waiters == 0 { - f.cancel() - if g.active[key] == f { - delete(g.active, key) - } - } - g.mu.Unlock() - }() - select { - case <-ctx.Done(): - return nil, ctx.Err() - case <-f.done: - // A concurrently completed flight cannot outlive this waiter's deadline. - if err := ctx.Err(); err != nil { - return nil, err - } - return f.value, f.err - } -} - -type CodexGatewayBorrowSetup struct { - State string `json:"state"` - Phase string `json:"phase"` - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Completed int `json:"completed"` - Total int `json:"total"` - Failed int `json:"failed"` - StartedAt time.Time `json:"started_at"` - FinishedAt *time.Time `json:"finished_at,omitempty"` - Error string `json:"error,omitempty"` -} - -func (s *CodexGatewayBorrowService) setSetup(revision uint64, setup CodexGatewayBorrowSetup) { - s.mu.Lock() - defer s.mu.Unlock() - if s.revision == revision { - s.setup = setup - } -} diff --git a/backend/internal/service/codex_gateway_borrow_coordination_test.go b/backend/internal/service/codex_gateway_borrow_coordination_test.go deleted file mode 100644 index f8cd58665831..000000000000 --- a/backend/internal/service/codex_gateway_borrow_coordination_test.go +++ /dev/null @@ -1,212 +0,0 @@ -package service - -import ( - "context" - "errors" - "io" - "net/http" - "strings" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -func TestCodexBorrowFlightCancellationIsPerWaiter(t *testing.T) { - var flights codexBorrowFlights - var calls atomic.Int32 - started, finish := make(chan struct{}), make(chan struct{}) - run := func(ctx context.Context) (any, error) { - calls.Add(1) - close(started) - select { - case <-finish: - return "ready", nil - case <-ctx.Done(): - return nil, ctx.Err() - } - } - first, cancel := context.WithCancel(context.Background()) - firstResult, secondResult := make(chan error, 1), make(chan error, 1) - go func() { - _, err := flights.do(first, context.Background(), "same", time.Second, run) - firstResult <- err - }() - <-started - go func() { - value, err := flights.do(context.Background(), context.Background(), "same", time.Second, run) - if err == nil && value != "ready" { - err = errors.New("missing shared value") - } - secondResult <- err - }() - require.Eventually(t, func() bool { flights.mu.Lock(); defer flights.mu.Unlock(); return flights.active["same"].waiters == 2 }, time.Second, time.Millisecond) - cancel() - require.ErrorIs(t, <-firstResult, context.Canceled) - close(finish) - require.NoError(t, <-secondResult) - require.EqualValues(t, 1, calls.Load()) -} - -func TestCodexBorrowFlightRevisionCancelsPreparation(t *testing.T) { - var flights codexBorrowFlights - revision, cancel := context.WithCancel(context.Background()) - started := make(chan struct{}) - result := make(chan error, 1) - go func() { - _, err := flights.do(context.Background(), revision, "old", time.Second, func(ctx context.Context) (any, error) { close(started); <-ctx.Done(); return nil, ctx.Err() }) - result <- err - }() - <-started - cancel() - require.ErrorIs(t, <-result, context.Canceled) -} - -func TestCodexBorrowPreparePreservesPartialSuccess(t *testing.T) { - s := newBorrowCoreTest(t, func(r *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id == 2 { - return nil, errors.New("target two rejected") - } - return borrowCoreResponse("gpt-6-astra", "OK", "stable-target-state", ""), nil - }, borrowCoreAccount(1), borrowCoreAccount(2), borrowCoreAccount(3)) - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2, 3}, Models: []string{"gpt-6-astra"}}, false) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - status, err := s.Prepare(context.Background()) - require.NoError(t, err) - require.Equal(t, "partial", status.Setup.State) - require.Equal(t, 2, status.Setup.Completed) - require.Equal(t, 1, status.Setup.Failed) - require.False(t, status.Targets[0].CacheValid) - require.True(t, status.Targets[1].CacheValid) - repo, ok := s.accounts.(*borrowCoreAccounts) - require.True(t, ok) - account := repo.rows[3] - account.Credentials["model_mapping"] = map[string]any{"gpt-6-astra": "gpt-6-luna"} - require.Equal(t, "model_mapping_mismatch", s.CurrentStatus(context.Background()).Targets[1].Reason) -} - -func TestCodexBorrowDifferentTargetsDoNotReturnBusy(t *testing.T) { - started, finish := make(chan struct{}), make(chan struct{}) - var blocked atomic.Bool - s := newBorrowCoreTest(t, func(r *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id == 2 && blocked.CompareAndSwap(false, true) { - close(started) - select { - case <-finish: - case <-r.Context().Done(): - return nil, r.Context().Err() - } - } - return borrowCoreResponse("gpt-6-astra", "OK", "stable", ""), nil - }, borrowCoreAccount(1), borrowCoreAccount(2), borrowCoreAccount(3)) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - first := make(chan error, 1) - go func() { _, err := s.Verify(context.Background(), 2, "gpt-6-astra"); first <- err }() - <-started - ctx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - _, err := s.Verify(ctx, 3, "gpt-6-astra") - close(finish) - require.NoError(t, err) - require.NoError(t, <-first) -} - -func TestCodexBorrowUsagePreservesStreamAndNewestAttempt(t *testing.T) { - s := newBorrowCoreTest(t, nil) - raw := borrowCoreSSE("gpt-6-astra", "OK") - tracker := s.beginUsage(context.Background(), 2, "gpt-6-astra", "http", true) - response, _ := s.trackHTTPResponse(tracker, &http.Response{StatusCode: 200, Header: http.Header{}, Body: io.NopCloser(strings.NewReader(raw))}, nil) - read, err := io.ReadAll(response.Body) - require.NoError(t, err) - require.NoError(t, response.Body.Close()) - require.Equal(t, raw, string(read)) - status := s.Status() - require.Len(t, status.RecentUsage, 1) - require.Equal(t, "completed", status.RecentUsage[0].Outcome) - old := s.beginUsage(context.Background(), 2, "gpt-6-astra", "http", false) - newer := s.beginUsage(context.Background(), 2, "gpt-6-astra", "http", true) - old.finish(errors.New("old failure")) - newer.observe([]byte(`{"type":"response.completed","response":{"id":"latest","model":"gpt-6-astra","status":"completed"}}`)) - newer.finish(nil) - require.Equal(t, "latest", s.Status().RecentUsage[0].RequestID) - require.EqualValues(t, 3, s.Status().RecentUsage[0].Count) -} - -func TestCodexBorrowDiagnosticBudgetIncludesPreparation(t *testing.T) { - var requests atomic.Int32 - ctx := context.WithValue(context.Background(), codexBorrowDiagnosticContextKey{}, &codexBorrowDiagnostic{requests: &requests, limit: 2}) - require.NoError(t, consumeBorrowDiagnosticRequest(ctx)) - require.NoError(t, consumeBorrowDiagnosticRequest(WithCodexGatewayBorrowObservation(ctx))) - require.ErrorIs(t, consumeBorrowDiagnosticRequest(ctx), ErrCodexBorrowDiagnosticBudget) - require.EqualValues(t, 2, requests.Load()) -} - -func TestCodexBorrowDistinctRequestProofsRemainReusable(t *testing.T) { - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - return borrowCoreResponse("gpt-6-astra", "OK", "stable", ""), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, first, _, err := s.accountTemplate(context.Background(), 2, "gpt-6-astra") - require.NoError(t, err) - second := first.Clone(first.Context()) - second.Header.Set("X-Codex-Turn-State", "a-different-client-state") - _, _, err = s.Apply(first, a, "gpt-6-astra", "", nil, false) - require.NoError(t, err) - _, _, err = s.Apply(second, a, "gpt-6-astra", "", nil, false) - require.NoError(t, err) - _, applied, err := s.Apply(first, a, "gpt-6-astra", "", nil, true) - require.NoError(t, err) - require.True(t, applied.Applied, "another request's validation must not replace this proof") -} - -func TestCodexBorrowUsageKeepsFirstTerminal(t *testing.T) { - completed := []byte(`{"type":"response.completed","response":{"status":"completed"}}`) - failed := []byte(`{"type":"error","error":{"code":"synthetic_failure"}}`) - for _, firstError := range []bool{false, true} { - s := newBorrowCoreTest(t, nil) - tracker := s.beginUsage(context.Background(), 2, "gpt-6-astra", "http", true) - if firstError { - tracker.observe(failed) - tracker.observe(completed) - } else { - tracker.observe(completed) - tracker.observe(failed) - } - tracker.finish(context.Canceled) - want := "completed" - if firstError { - want = "upstream_error" - } - require.Equal(t, want, s.Status().RecentUsage[0].Outcome) - } -} - -func TestCodexBorrowHTTPAppliedChecksFinalCookie(t *testing.T) { - req, err := http.NewRequest(http.MethodPost, chatgptCodexURL, nil) - require.NoError(t, err) - proof := codexGatewayBorrowHTTPPreparation{application: &CodexGatewayBorrowApplication{Applied: true, CookieFingerprint: borrowHash("qualified-cookie")}} - req = req.WithContext(context.WithValue(req.Context(), codexGatewayBorrowHTTPPreparationContextKey{}, proof)) - require.False(t, codexBorrowHTTPApplied(req)) - req.Header.Set("Cookie", "__oailb=changed-cookie") - require.False(t, codexBorrowHTTPApplied(req)) - req.Header.Set("Cookie", "__oailb=qualified-cookie") - require.True(t, codexBorrowHTTPApplied(req)) -} - -func TestCodexBorrowCancelledParentCannotAdmitWaitingChild(t *testing.T) { - parent, cancel := context.WithCancel(context.Background()) - coordinator := newPelicanExecutionCoordinator(1) - state := newPelicanExecutionState(parent, coordinator, time.Minute, time.Now) - defer state.finish() - // Represent the cancellation propagation gap: the parent is cancelled, - // but this child's cancellation signal is not visible to its waiter yet. - ctx := context.WithValue(context.WithoutCancel(state.parent), pelicanExecutionContextKey{}, state) - cancel() - _, release, err := AcquirePelicanExecution(ctx, 2) - require.ErrorIs(t, err, context.Canceled) - require.Nil(t, release) - require.Zero(t, coordinator.active) -} diff --git a/backend/internal/service/codex_gateway_borrow_core_test.go b/backend/internal/service/codex_gateway_borrow_core_test.go deleted file mode 100644 index cdd5e021bb47..000000000000 --- a/backend/internal/service/codex_gateway_borrow_core_test.go +++ /dev/null @@ -1,492 +0,0 @@ -package service - -import ( - "context" - "encoding/json" - "errors" - "io" - "net/http" - "strings" - "sync" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/klauspost/compress/zstd" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -// Embedding nil ports makes any health, quota, recovery or account write panic. -// Every request in these tests terminates at the synthetic HTTPUpstream below. -type borrowCoreAccounts struct { - AccountRepository - rows map[int64]*Account -} - -func (r *borrowCoreAccounts) GetByID(_ context.Context, id int64) (*Account, error) { - return r.rows[id], nil -} - -type borrowCoreSettings struct { - SettingRepository - raw string - writes int -} - -func (s *borrowCoreSettings) GetValue(context.Context, string) (string, error) { - if s.raw == "" { - return "", ErrSettingNotFound - } - return s.raw, nil -} -func (s *borrowCoreSettings) Set(_ context.Context, _, value string) error { - s.raw, s.writes = value, s.writes+1 - return nil -} - -type borrowCoreProbe func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) - -func (f borrowCoreProbe) Do(r *http.Request, proxy string, id int64, concurrency int) (*http.Response, error) { - return f(r, proxy, id, concurrency, nil) -} -func (f borrowCoreProbe) DoWithTLS(r *http.Request, proxy string, id int64, concurrency int, p *tlsfingerprint.Profile) (*http.Response, error) { - return f(r, proxy, id, concurrency, p) -} - -func borrowCoreAccount(id int64) *Account { - return &Account{ID: id, Platform: PlatformOpenAI, Type: AccountTypeOAuth, Status: StatusActive, Concurrency: 4, - Credentials: map[string]any{"access_token": "synthetic-token", "chatgpt_account_id": "synthetic-chat-id"}, Extra: map[string]any{}} -} - -func newBorrowCoreTest(t *testing.T, upstream borrowCoreProbe, rows ...*Account) *CodexGatewayBorrowService { - t.Helper() - repo := &borrowCoreAccounts{rows: map[int64]*Account{}} - for _, row := range rows { - repo.rows[row.ID] = row - } - gateway := &OpenAIGatewayService{accountRepo: repo} - s := NewCodexGatewayBorrowService(nil, repo, gateway, upstream, nil) - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2, 3}, Models: []string{"gpt-6-astra", "gpt-6.1-sol"}}, false) - t.Cleanup(s.Stop) - return s -} - -func borrowCoreCandidate(s *CodexGatewayBorrowService, expires time.Time) { - s.mu.Lock() - defer s.mu.Unlock() - s.candidate = &codexGatewayBorrowCandidate{cookie: http.Cookie{Name: "__oailb", Value: "synthetic-borrowed-cookie", Secure: true, Path: "/"}, expires: expires, sourceID: 1} -} - -func borrowCoreSSE(model, text string) string { - output := []map[string]any{} - if text != "" { - output = append(output, map[string]any{"type": "message", "content": []map[string]any{{"type": "output_text", "text": text}}}) - } - raw, _ := json.Marshal(map[string]any{"type": "response.completed", "response": map[string]any{"status": "completed", "model": model, "output": output}}) - return "data: " + string(raw) + "\n\n" -} - -func borrowCoreResponse(model, text, state string, cookies ...string) *http.Response { - h := make(http.Header) - if state != "" { - h.Set("X-Codex-Turn-State", state) - } - for _, cookie := range cookies { - h.Add("Set-Cookie", cookie) - } - return &http.Response{StatusCode: http.StatusOK, Header: h, Body: io.NopCloser(strings.NewReader(borrowCoreSSE(model, text)))} -} - -func borrowCoreBody(t *testing.T, req *http.Request) []byte { - t.Helper() - data, err := io.ReadAll(req.Body) - require.NoError(t, err) - if req.Header.Get("Content-Encoding") == "zstd" { - decoder, err := zstd.NewReader(nil) - require.NoError(t, err) - defer decoder.Close() - data, err = decoder.DecodeAll(data, nil) - require.NoError(t, err) - } - return data -} - -func TestCodexGatewayBorrowConfigDefaultsAndLoadNeverProbe(t *testing.T) { - var calls atomic.Int32 - probe := borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - calls.Add(1) - return nil, errors.New("must not call upstream") - }) - store := &borrowCoreSettings{} - s, err := ProvideCodexGatewayBorrowService(store, nil, nil, probe, nil, nil) - require.NoError(t, err) - t.Cleanup(s.Stop) - require.Equal(t, DefaultCodexGatewayBorrowConfig(), s.ConfigSnapshot()) - require.Zero(t, store.writes) - require.Zero(t, calls.Load()) - store.raw = `{"enabled":true,"source_account_ids":[1],"target_account_ids":[2],"models":["gpt-6.1-sol"]}` - cfg, err := s.GetConfig(context.Background()) - require.NoError(t, err) - require.True(t, cfg.Enabled) - require.Zero(t, calls.Load()) - for _, raw := range []string{`null`, `{"enabled":true}`, `{"source_account_ids":[1],"target_account_ids":[1]}`, `{"models":["gpt-6-luna"]}`, `{"models":["gpt-6-astra","gpt-6-astra"]}`, `{"unknown":true}`, `{} {}`} { - _, err := DecodeCodexGatewayBorrowConfig([]byte(raw)) - require.Error(t, err, raw) - } - _, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":false}`)) - require.NoError(t, err) -} - -func TestCodexGatewayBorrowCookieScopeAndExpiry(t *testing.T) { - now := time.Now() - valid := http.Cookie{Name: "__oailb", Value: "route", Secure: true, Path: "/backend-api/codex", MaxAge: 230} - for _, tc := range []struct { - name string - change func(*http.Cookie) - ttl time.Duration - valid bool - }{ - {"session without expiry", func(c *http.Cookie) { c.MaxAge = 0 }, 0, false}, - {"actual max age", func(c *http.Cookie) { c.MaxAge = 30 }, 30 * time.Second, true}, - {"local cap", func(c *http.Cookie) { c.MaxAge = 1000000 }, 230 * time.Second, true}, - {"actual expires", func(c *http.Cookie) { c.MaxAge = 0; c.Expires = now.Add(20 * time.Second) }, 20 * time.Second, true}, - {"default path", func(c *http.Cookie) { c.Path = "" }, 230 * time.Second, true}, - {"foreign domain", func(c *http.Cookie) { c.Domain = "example.invalid" }, 0, false}, - {"wrong path", func(c *http.Cookie) { c.Path = "/backend-api/codex/responses-other" }, 0, false}, - {"not secure", func(c *http.Cookie) { c.Secure = false }, 0, false}, - {"deleted", func(c *http.Cookie) { c.MaxAge = -1 }, 0, false}, - {"expired", func(c *http.Cookie) { c.MaxAge = 0; c.Expires = now.Add(-time.Second) }, 0, false}, - } { - t.Run(tc.name, func(t *testing.T) { - cookie := valid - tc.change(&cookie) - candidate := borrowCandidateFromCookies([]*http.Cookie{&cookie}, "/backend-api/codex/responses", now) - if !tc.valid { - require.Nil(t, candidate) - return - } - require.NotNil(t, candidate) - require.Equal(t, now.Add(tc.ttl), candidate.expires) - }) - } - deleted := valid - deleted.MaxAge = -1 - require.Nil(t, borrowCandidateFromCookies([]*http.Cookie{&valid, &deleted}, "/backend-api/codex/responses", now)) -} - -func TestCodexGatewayBorrowSourceUsesOwnExitAndNeverExtendsSameCookie(t *testing.T) { - proxyID := int64(11) - source := borrowCoreAccount(1) - source.ProxyID, source.Proxy = &proxyID, &Proxy{Protocol: "http", Host: "source.invalid", Port: 8080} - var calls int - s := newBorrowCoreTest(t, func(req *http.Request, proxy string, id int64, _ int, profile *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - require.Equal(t, int64(1), id) - require.Equal(t, source.Proxy.URL(), proxy) - require.Nil(t, profile) - require.Equal(t, HTTPUpstreamProfileCodexBorrowSource, HTTPUpstreamProfileFromContext(req.Context())) - require.True(t, IsCodexGatewayBorrowObservation(req.Context())) - require.True(t, HTTPUpstreamRedirectsDisabled(req.Context())) - deadline, ok := req.Context().Deadline() - require.True(t, ok) - require.InDelta(t, 90, time.Until(deadline).Seconds(), 2) - body := borrowCoreBody(t, req) - require.Equal(t, "gpt-6-astra", gjson.GetBytes(body, "model").String()) - require.Equal(t, "medium", gjson.GetBytes(body, "reasoning.effort").String()) - require.Equal(t, "Reply with OK only.", gjson.GetBytes(body, "input.0.content.0.text").String()) - require.Empty(t, req.Header.Get("Cookie")) - require.Empty(t, req.Header.Get("X-Codex-Turn-State")) - return borrowCoreResponse("gpt-6-astra", "OK", "source-state", "__oailb=synthetic-borrowed-cookie; Secure; Path=/; Max-Age=230", "__cflb=source-cookie; Secure; Path=/"), nil - }, source) - expires := time.Now().Add(50 * time.Second) - borrowCoreCandidate(s, expires) - require.NoError(t, s.prepareSource(context.Background(), s.revision, s.revisionCtx)) - require.Equal(t, 1, calls) - require.Equal(t, expires, s.candidate.expires) - borrowCoreCandidate(s, time.Now().Add(150*time.Second)) - require.NoError(t, s.prepareSource(context.Background(), s.revision, s.revisionCtx)) - require.Equal(t, 1, calls, "fresh candidate skips a new source invocation") -} - -func TestCodexGatewayBorrowTargetCacheModelAndFingerprintIsolation(t *testing.T) { - var calls int - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, proxy string, id int64, _ int, profile *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - require.Equal(t, int64(2), id) - require.Equal(t, "target-exit", proxy) - require.Nil(t, profile) - require.Equal(t, HTTPUpstreamProfileCodexBorrowTarget, HTTPUpstreamProfileFromContext(req.Context())) - require.True(t, IsCodexGatewayBorrowObservation(req.Context())) - require.True(t, req.Close) - deadline, ok := req.Context().Deadline() - require.True(t, ok) - require.InDelta(t, 45, time.Until(deadline).Seconds(), 2) - body := borrowCoreBody(t, req) - model := gjson.GetBytes(body, "model").String() - require.Equal(t, "Reply with OK.", gjson.GetBytes(body, "input.0.content.0.text").String()) - require.False(t, gjson.GetBytes(body, "reasoning.effort").Exists()) - borrowed, err := req.Cookie("__oailb") - require.NoError(t, err) - require.Equal(t, "synthetic-borrowed-cookie", borrowed.Value) - if req.Header.Get("X-Codex-Turn-State") == "" { - require.NotContains(t, req.Header.Get("Cookie"), "source-cookie") - return borrowCoreResponse(model, "", "minted-state", "__cflb=target-cookie; Secure; Path=/"), nil - } - require.Equal(t, "minted-state", req.Header.Get("X-Codex-Turn-State")) - require.Contains(t, req.Header.Get("Cookie"), "__cflb=target-cookie") - // Completed with no visible text is valid for the STATE criterion. - return borrowCoreResponse(model, "", ""), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, template, _, err := s.accountTemplate(context.Background(), 2, "gpt-6-astra") - require.NoError(t, err) - template.Header.Set("Cookie", "__oailb=old; __cflb=business-cookie; other=keep") - template.Header.Set("X-Codex-Turn-State", "business-state") - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - wire, app, err := s.Apply(template, a, model, "target-exit", nil, false) - require.NoError(t, err) - require.True(t, app.Applied) - require.Equal(t, "business-state", wire.Header.Get("X-Codex-Turn-State")) - require.Contains(t, wire.Header.Get("Cookie"), "__cflb=business-cookie") - require.Contains(t, wire.Header.Get("Cookie"), "other=keep") - require.NotContains(t, wire.Header.Get("Cookie"), "minted-state") - require.Contains(t, template.Header.Get("Cookie"), "__oailb=old", "input request is not modified") - } - require.Equal(t, 4, calls) - _, app, err := s.Apply(template, a, "gpt-6.1-sol", "target-exit", nil, false) - require.NoError(t, err) - require.True(t, app.Applied, "hot cache is read before global target lock") - continuation := template.Clone(template.Context()) - continuation.Header.Set("X-Codex-Turn-State", "next-business-state") - continuation.Header.Set("X-Client-Request-ID", "next-trace") - _, _, err = s.Apply(continuation, a, "gpt-6.1-sol", "target-exit", nil, true) - require.Error(t, err, "original cache identity includes business STATE") - require.Equal(t, "business-state", template.Header.Get("X-Codex-Turn-State")) - traceOnly := template.Clone(template.Context()) - traceOnly.Header.Set("X-Client-Request-ID", "another-trace") - _, reused, err := s.Apply(traceOnly, a, "gpt-6.1-sol", "target-exit", nil, true) - require.NoError(t, err) - require.Equal(t, app.ExpiresAt, reused.ExpiresAt) - for _, header := range []string{"Authorization", "ChatGPT-Account-ID", "User-Agent", "Originator", "Version"} { - req := template.Clone(template.Context()) - req.Header.Set(header, req.Header.Get(header)+"-changed") - _, _, err := s.Apply(req, a, "gpt-6.1-sol", "target-exit", nil, true) - require.Error(t, err, header) - } - _, _, err = s.Apply(template, a, "gpt-6.1-sol", "other-exit", nil, true) - require.Error(t, err) - _, _, err = s.Apply(template, a, "gpt-6.1-sol", "target-exit", &tlsfingerprint.Profile{Name: "changed"}, true) - require.Error(t, err) - require.Equal(t, 4, calls, "cached-only mismatches do not validate") -} - -func TestCodexGatewayBorrowTargetChangedStateRouteAndFailureCooldown(t *testing.T) { - for _, mode := range []string{"state", "cookie", "missing", "incomplete", "upstream_failed"} { - t.Run(mode, func(t *testing.T) { - var calls int - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - borrowCoreBody(t, req) - if mode == "missing" { - return borrowCoreResponse("gpt-6.1-sol", "", ""), nil - } - if mode == "cookie" { - return borrowCoreResponse("gpt-6.1-sol", "", "first", "__oailb=changed; Secure; Path=/"), nil - } - if mode == "incomplete" { - return &http.Response{StatusCode: 200, Header: http.Header{"X-Codex-Turn-State": []string{"first"}}, Body: io.NopCloser(strings.NewReader("data: [DONE]\n\n"))}, nil - } - if mode == "upstream_failed" { - return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader("data: {\"type\":\"response.failed\"}\n\n"))}, nil - } - state := "first" - if calls == 2 { - state = "changed" - } - return borrowCoreResponse("gpt-6.1-sol", "", state), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - _, _, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.Error(t, err) - initial := calls - _, _, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.Error(t, err) - status := s.Status() - require.False(t, status.Targets[1].CacheValid) - if mode == "incomplete" { - require.Equal(t, initial+1, calls, "each independent request revalidates the incomplete stream") - require.Nil(t, status.Targets[1].RetryAfter) - require.True(t, status.Targets[1].RetryOnNextRequest) - } else { - require.Equal(t, initial, calls, "rejection stays in 15-second cooldown") - require.NotNil(t, status.Targets[1].RetryAfter) - } - }) - } -} - -func TestCodexGatewayBorrowSourceSingleFlightCooldownAndRevisionCancellation(t *testing.T) { - started, release := make(chan struct{}), make(chan struct{}) - var calls atomic.Int32 - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if calls.Add(1) == 1 { - close(started) - } - <-release - return &http.Response{StatusCode: http.StatusForbidden, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(`{"error":{"message":"full synthetic source failure"}}`))}, nil - }, borrowCoreAccount(1)) - var wg sync.WaitGroup - for i := 0; i < 4; i++ { - wg.Add(1) - go func() { defer wg.Done(); _ = s.prepareSource(context.Background(), s.revision, s.revisionCtx) }() - } - <-started - close(release) - wg.Wait() - require.Equal(t, int32(1), calls.Load()) - err := s.prepareSource(context.Background(), s.revision, s.revisionCtx) - require.ErrorContains(t, err, "full synthetic source failure") - require.Equal(t, int32(1), calls.Load()) - started, release = make(chan struct{}), make(chan struct{}) - calls.Store(0) - s2 := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - close(started) - <-release - return borrowCoreResponse("gpt-6-astra", "OK", "", "__oailb=late; Secure; Path=/"), nil - }, borrowCoreAccount(1)) - done := make(chan error, 1) - rev, revCtx := s2.revision, s2.revisionCtx - go func() { done <- s2.prepareSource(context.Background(), rev, revCtx) }() - <-started - s2.publishConfig(DefaultCodexGatewayBorrowConfig(), false) - close(release) - require.Error(t, <-done) - require.Nil(t, s2.Status().Candidate) - require.Empty(t, s2.Status().Sources) -} - -type borrowCoreUntouchedBody struct{ reads int } - -func (b *borrowCoreUntouchedBody) Read([]byte) (int, error) { - b.reads++ - return 0, errors.New("business body must not be inspected") -} -func (*borrowCoreUntouchedBody) Close() error { return nil } - -func TestCodexGatewayBorrowGatePreserves100MiBBodyAndOtherRoutes(t *testing.T) { - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - t.Fatal("must not probe") - return nil, nil - }, a) - body := &borrowCoreUntouchedBody{} - req, err := http.NewRequest(http.MethodPost, chatgptCodexURL, body) - require.NoError(t, err) - req.ContentLength = 100 << 20 - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - key := borrowTargetFingerprint(req, a, "gpt-6.1-sol", "", nil, s.candidate.cookie.Value) - s.targets[codexGatewayBorrowTargetKey{2, "gpt-6.1-sol"}] = codexGatewayBorrowTargetCheck{policyRevision: currentCodexFingerprintPolicyForAccount(a).revision, key: key, cookieKey: borrowHash(s.candidate.cookie.Value), expires: s.candidate.expires, result: CodexGatewayBorrowVerification{Success: true}} - wire, app, err := s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.True(t, app.Applied) - require.Same(t, body, wire.Body) - require.Equal(t, int64(100<<20), wire.ContentLength) - require.Zero(t, body.reads) - for _, url := range []string{"https://chatgpt.com/backend-api/codex/responses/compact", "https://chatgpt.com/backend-api/codex/models", "https://api.openai.com/v1/responses", "https://chatgpt.com:443/backend-api/codex/responses", "https://other.invalid/backend-api/codex/responses"} { - other, _ := http.NewRequest(http.MethodPost, url, body) - result, app, err := s.Apply(other, a, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.Same(t, other, result) - require.Nil(t, app) - } - result, app, err := s.Apply(req, a, "gpt-6-luna", "", nil, false) - require.NoError(t, err) - require.Same(t, req, result) - require.Nil(t, app) - require.Zero(t, body.reads) -} - -func TestCodexGatewayBorrowPelicanCachedOnlyPausedTargetAndFullOutput(t *testing.T) { - var calls int - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - body := borrowCoreBody(t, req) - if gjson.GetBytes(body, "input.0.content.0.text").String() == CodexGatewayBorrowPelicanPrompt { - require.Equal(t, "high", gjson.GetBytes(body, "reasoning.effort").String()) - return borrowCoreResponse("gpt-6.1-sol", "鹈鹕骑自行车", ""), nil - } - state := "first" - if calls == 2 { - state = "" - } - return borrowCoreResponse("gpt-6.1-sol", "", state), nil - }, a) - result, err := s.GeneratePelican(context.Background(), 2, "gpt-6.1-sol", "high") - require.NoError(t, err) - require.Equal(t, "skipped", result.Status) - require.Zero(t, calls) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - _, _, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.Equal(t, 2, calls) - a.Status, a.Schedulable = "paused", false - result, err = s.GeneratePelican(context.Background(), 2, "gpt-6.1-sol", "") - require.NoError(t, err) - require.Equal(t, "complete", result.Status) - require.Equal(t, "鹈鹕骑自行车", result.RawAnswer) - require.Contains(t, result.RawResponse, "response.completed") - require.Equal(t, 3, calls) - require.Equal(t, "paused", a.Status) - require.False(t, a.Schedulable) - result, err = s.GeneratePelican(context.Background(), 2, "gpt-6-astra", "high") - require.NoError(t, err) - require.Equal(t, "skipped", result.Status) - require.Equal(t, 3, calls) -} - -func TestCodexGatewayBorrowSSERequiresTerminalAndPreservesErrors(t *testing.T) { - for _, stream := range []string{"data: [DONE]\n\n", `data: {"type":"response.output_text.delta","delta":"partial"}` + "\n\n", `data: {"type":"response.failed","response":{"error":{"code":"synthetic_error","message":"full upstream message"}}}` + "\n\n", "data: {bad json}\n\n"} { - got := parseCodexGatewayBorrowSSE([]byte(stream), "gpt-6-astra", codexGatewayBorrowProbeMaxBody) - require.False(t, got.complete) - if strings.Contains(stream, "full upstream message") { - require.Contains(t, got.errorText, "full upstream message") - } - } - createdOnly := `data: {"type":"response.created","response":{"model":"gpt-6-astra"}}` + "\n\n" + `data: {"type":"response.completed","response":{"status":"completed","output":[]}}` + "\n\n" - got := parseCodexGatewayBorrowSSE([]byte(createdOnly), "gpt-6-astra", codexGatewayBorrowProbeMaxBody) - require.True(t, got.complete) - require.Empty(t, got.model, "source requires a final model, not earlier advertised identity") - got = parseCodexGatewayBorrowSSE([]byte(borrowCoreSSE("gpt-6.1-sol", "answer")), "gpt-6-astra", codexGatewayBorrowProbeMaxBody) - require.True(t, got.complete) - require.True(t, got.modelMismatch) - require.Equal(t, "answer", got.answer) -} - -func TestCodexGatewayBorrowQualificationFailuresRemainTyped(t *testing.T) { - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - return nil, errors.New("synthetic source connection refused") - }, borrowCoreAccount(1), a) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6-astra") - require.NoError(t, err) - _, _, err = s.Apply(req, a, "gpt-6-astra", "", nil, false) - require.True(t, IsCodexGatewayBorrowFailure(err)) - require.ErrorContains(t, err, "synthetic source connection refused") - _, _, err = s.Apply(req, a, "gpt-6-astra", "", nil, true) - require.True(t, IsCodexGatewayBorrowFailure(err)) - require.ErrorIs(t, err, ErrCodexGatewayBorrowUnavailable) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, _, err = s.Apply(req, a, "gpt-6-astra", "", nil, false) - require.True(t, IsCodexGatewayBorrowFailure(err)) - require.ErrorIs(t, err, ErrCodexGatewayBorrowUnavailable) - require.False(t, IsCodexGatewayBorrowFailure(errors.New("ordinary business failure"))) -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose.go b/backend/internal/service/codex_gateway_borrow_diagnose.go deleted file mode 100644 index baa0fa271c20..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose.go +++ /dev/null @@ -1,393 +0,0 @@ -package service - -import ( - "context" - "encoding/json" - "errors" - "net/http" - "net/http/httptest" - "slices" - "strings" - "sync/atomic" - "time" - - "github.com/gin-gonic/gin" - "github.com/google/uuid" -) - -const CodexBorrowDiagnosticMaxRequests = 8 - -var ErrCodexBorrowDiagnosticBudget = errors.New("diagnostic request budget exhausted (including preparation)") - -type codexBorrowDiagnosticContextKey struct{} -type codexBorrowAppliedContextKey struct{} - -type codexBorrowDiagnostic struct { - replay *CodexBorrowReplay - scenario string - serviceTier string - history []any - turnState string - verification atomic.Pointer[CodexGatewayBorrowVerification] - limit int32 - onRequest func(int32) - requests *atomic.Int32 - borrow bool - transport string - session string - anchors codexGatewayBorrowWSAnchorStore -} - -func borrowDiagnosticFromContext(ctx context.Context) *codexBorrowDiagnostic { - if ctx == nil { - return nil - } - d, _ := ctx.Value(codexBorrowDiagnosticContextKey{}).(*codexBorrowDiagnostic) - return d -} - -func consumeBorrowDiagnosticRequest(ctx context.Context) error { - d := borrowDiagnosticFromContext(ctx) - if d == nil { - return nil - } - if err := ctx.Err(); err != nil { - return err - } - for { - count := d.requests.Load() - limit := d.limit - if limit == 0 { - limit = CodexBorrowDiagnosticMaxRequests - } - if count >= limit { - return ErrCodexBorrowDiagnosticBudget - } - if d.requests.CompareAndSwap(count, count+1) { - if d.onRequest != nil { - d.onRequest(count + 1) - } - return nil - } - } -} - -type CodexBorrowDiagnosticRequest struct { - Replay *CodexBorrowReplay `json:"replay,omitempty"` - Comparison string `json:"comparison,omitempty"` - Scenario string `json:"scenario,omitempty"` - Mode string `json:"mode,omitempty"` - ServiceTier string `json:"service_tier,omitempty"` - RequestLimit int `json:"request_limit,omitempty"` - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Transport string `json:"transport"` -} - -type CodexBorrowDiagnosticResult struct { - TurnState string `json:"turn_state,omitempty"` - ProbeOnly bool `json:"probe_only,omitempty"` - ProbeContext *CodexBorrowProbeContext `json:"probe_context,omitempty"` - Scenario string `json:"scenario,omitempty"` - FailureStage string `json:"failure_stage,omitempty"` - FailureReason string `json:"failure_reason,omitempty"` - Verification *CodexGatewayBorrowVerification `json:"verification,omitempty"` - ToolRoundTrip bool `json:"tool_round_trip,omitempty"` - StateLength int `json:"state_length,omitempty"` - ReadError string `json:"read_error,omitempty"` - Dispatched bool `json:"dispatched"` - Mode string `json:"mode"` - Turn int `json:"turn"` - Applied bool `json:"applied"` - Completed bool `json:"completed"` - ResponseID string `json:"response_id,omitempty"` - ReportedModel string `json:"reported_model,omitempty"` - Answer string `json:"answer"` - RawResponse string `json:"raw_response"` - Error string `json:"error,omitempty"` - DurationMS int64 `json:"duration_ms"` -} - -type CodexBorrowDiagnosticEvent struct { - Type string `json:"type"` - Mode string `json:"mode,omitempty"` - Requests int32 `json:"requests"` - Limit int `json:"limit"` - Result *CodexBorrowDiagnosticResult `json:"result,omitempty"` - Error string `json:"error,omitempty"` -} - -// Diagnose runs fixed short questions through the normal account sender. The -// internal observation purpose isolates health/billing; it is never user input. -func (s *CodexGatewayBorrowService) Diagnose(ctx context.Context, request CodexBorrowDiagnosticRequest, emit func(CodexBorrowDiagnosticEvent)) error { - if request.Scenario != "" && request.Scenario != "codex_session" && request.Scenario != "probe_contract" && request.Scenario != "codex_replay" { - return errors.New("scenario must be empty, codex_session, probe_contract or codex_replay") - } - if err := validateCodexBorrowReplay(request); err != nil { - return err - } - if request.Mode != "" && request.Mode != "ordinary" && request.Mode != "borrowed" { - return errors.New("mode must be empty, ordinary or borrowed") - } - if request.Scenario == "codex_session" && request.Transport != "http" { - return errors.New("codex_session uses HTTP; existing WS diagnostics already cover continuation") - } - if request.Scenario == "probe_contract" && (request.Transport != "http" || request.Mode == "ordinary") { - return errors.New("probe_contract requires borrowed HTTP observations") - } - if request.Comparison != "" && (request.Scenario != "probe_contract" || (request.Comparison != "body" && request.Comparison != "encoding")) { - return errors.New("comparison must be body or encoding in probe_contract") - } - if request.ServiceTier != "" && request.ServiceTier != "default" && request.ServiceTier != OpenAIFastTierPriority && request.ServiceTier != OpenAIFastTierFlex && request.ServiceTier != OpenAIFastTierUltrafast { - return errors.New("unsupported service_tier") - } - if request.AccountID <= 0 || (request.Transport != "http" && request.Transport != "ws") { - return errors.New("account_id and transport (http or ws) are required") - } - limit := request.RequestLimit - if limit == 0 { - limit = CodexBorrowDiagnosticMaxRequests - } - if limit < 1 || limit > CodexBorrowDiagnosticMaxRequests { - return errors.New("request_limit must be between 1 and 8") - } - cfg := s.ConfigSnapshot() - sourceReplay := request.Scenario == "codex_replay" && request.Mode == "ordinary" && slices.Contains(cfg.SourceAccountIDs, request.AccountID) && slices.Contains(cfg.Models, request.Model) - sourceProbe := request.Scenario == "probe_contract" && request.Model == codexGatewayBorrowSourceModel && slices.Contains(cfg.SourceAccountIDs, request.AccountID) - if sourceProbe && (request.ServiceTier != "" || request.Comparison == "encoding") { - return errors.New("source self-check uses the current plaintext Astra template without a service tier") - } - if !cfg.Enabled || (!sourceProbe && !sourceReplay && (!slices.Contains(cfg.TargetAccountIDs, request.AccountID) || !slices.Contains(cfg.Models, request.Model))) { - return errors.New("select a saved, enabled target and model") - } - account, err := s.accounts.GetByID(ctx, request.AccountID) - if err != nil || !codexGatewayBorrowAccountSupported(account) { - return errors.New("selected account is unavailable") - } - if account.Status != StatusActive || !account.IsModelSupported(request.Model) || normalizeOpenAIModelForUpstream(account, account.GetMappedModel(request.Model)) != request.Model { - return errors.New("selected account is inactive or maps the requested model to another model") - } - if request.Transport == "ws" && s.gateway.getOpenAIWSProtocolResolver().Resolve(account).Transport != OpenAIUpstreamTransportResponsesWebsocketV2 { - return errors.New("WebSocket is not enabled for this account; settings were not changed") - } - s.mu.Lock() - revisionCtx := s.revisionCtx - revision := s.revision - s.mu.Unlock() - ctx, cancel := borrowRevisionContext(WithAccountObservation(ctx), revisionCtx, 10*time.Minute) - defer cancel() - if request.Scenario == "probe_contract" { - return s.diagnoseProbeContract(ctx, revision, revisionCtx, request, sourceProbe, limit, emit) - } - var requests atomic.Int32 - modes := []bool{false, true} - if request.Mode == "borrowed" || (request.Scenario == "codex_session" && request.Mode == "") { - modes = []bool{true} - } else if request.Mode == "ordinary" { - modes = []bool{false} - } - for _, borrowed := range modes { - if err := ctx.Err(); err != nil { - return err - } - mode := "ordinary" - if borrowed { - mode = "borrowed" - } - d := &codexBorrowDiagnostic{limit: int32(limit), onRequest: func(count int32) { - emit(CodexBorrowDiagnosticEvent{Type: "request", Mode: mode, Requests: count, Limit: limit}) - }, requests: &requests, borrow: borrowed, transport: request.Transport, session: uuid.NewString(), scenario: request.Scenario, serviceTier: request.ServiceTier, replay: request.Replay} - operation := context.WithValue(ctx, codexBorrowDiagnosticContextKey{}, d) - emit(CodexBorrowDiagnosticEvent{Type: "phase", Mode: mode, Requests: requests.Load(), Limit: limit}) - previous := "" - turns := 1 - if request.Transport == "ws" || request.Scenario == "codex_session" { - turns = 2 - } - for turn := 1; turn <= turns; turn++ { - result := s.diagnoseTurn(operation, account, request.Model, previous) - result.Mode, result.Turn = mode, turn - emit(CodexBorrowDiagnosticEvent{Type: "result", Mode: mode, Result: &result, Requests: requests.Load(), Limit: limit}) - if !result.Completed || result.Error != "" { - break - } - previous = result.ResponseID - if request.Transport == "ws" && previous == "" { - break - } - } - d.anchors.mu.Lock() - for key, entry := range d.anchors.entries { - if entry.connID != "" { - s.gateway.getOpenAIWSConnPool().evictConn(key.account, entry.connID) - } - } - d.anchors.mu.Unlock() - if requests.Load() >= int32(limit) { - break - } - } - emit(CodexBorrowDiagnosticEvent{Type: "done", Requests: requests.Load(), Limit: limit}) - return ctx.Err() -} - -func (s *CodexGatewayBorrowService) diagnoseTurn(ctx context.Context, account *Account, model, previous string) CodexBorrowDiagnosticResult { - start := time.Now() - result := CodexBorrowDiagnosticResult{} - budget := 90 * time.Second - if diagnostic := borrowDiagnosticFromContext(ctx); diagnostic != nil && diagnostic.replay != nil { - budget = 5 * time.Minute - } - state := newPelicanExecutionState(ctx, sharedPelicanExecution, budget, time.Now) - ctx = context.WithValue(state.parent, pelicanExecutionContextKey{}, state) - defer state.finish() - BeginPelicanPreparation(ctx) - capture := &pelicanGenerationCapture{account: account, concurrency: s.gateway.concurrencyService, openaiGateway: s.gateway, fallbackModel: model} - ctx = context.WithValue(ctx, pelicanGenerationContextKey{}, capture) - w := httptest.NewRecorder() - c, _ := gin.CreateTestContext(w) - capture.ginContext = c - c.Request = httptest.NewRequest(http.MethodPost, "/internal/admin/codex-diagnostic", nil).WithContext(ctx) - d := borrowDiagnosticFromContext(ctx) - d.verification.Store(nil) - c.Request.Header.Set("session_id", d.session) - if d.transport == "ws" { - SetOpenAIClientTransport(c, OpenAIClientTransportWS) - } else { - SetOpenAIClientTransport(c, OpenAIClientTransportHTTP) - } - payload := map[string]any{"model": model, "stream": true, "store": false, "input": []map[string]any{{"role": "user", "content": []map[string]string{{"type": "input_text", "text": "Reply with exactly OK."}}}}} - if previous != "" { - payload["previous_response_id"] = previous - } - if levels, _ := AccountTestReasoningOptions(account, model); slices.Contains(levels, "low") { - payload["reasoning"] = map[string]string{"effort": "low"} - } - if d.serviceTier != "" { - payload["service_tier"] = d.serviceTier - } - if d.scenario == "codex_session" { - d.prepareHTTPSessionTurn(c.Request.Header, payload) - } - body, _ := json.Marshal(payload) - if d.replay != nil { - body = append([]byte(nil), d.replay.Body...) - c.Request.Header.Del("session_id") - d.replay.applyHeaders(c.Request.Header) - } - forward, sendErr := s.gateway.Forward(ctx, c, account, body) - result.Scenario, result.Verification = d.scenario, d.verification.Load() - if sendErr != nil { - result.FailureStage, result.FailureReason = codexBorrowFailureStage(sendErr) - if result.Verification != nil && !result.Verification.Success { - result.FailureStage, result.FailureReason = "target_validation", result.Verification.Reason - } else if !IsCodexGatewayBorrowRequestFailure(sendErr) { - result.FailureStage, result.FailureReason = "upstream", "request_failed" - } - } - result.DurationMS = time.Since(start).Milliseconds() - if forward != nil { - result.ResponseID = firstNonEmpty(forward.ResponseID, forward.RequestID) - result.ReportedModel = forward.UpstreamResponseModel - } - last := snapshotPelicanAttempt(capture) - if last == nil { - result.RawResponse = w.Body.String() - if sendErr != nil { - result.Error = sendErr.Error() - } else { - result.Error = "no upstream request was dispatched" - } - return result - } - result.Dispatched = last.dispatched - if last.err != nil { - result.ReadError = last.err.Error() - } - result.RawResponse = last.raw.String() - if d.scenario == "codex_replay" && last.dispatched { - state.mu.Lock() - result.Applied = state.invocation.BorrowApplied - state.mu.Unlock() - result.StateLength = len(extractOpenAICodexTurnState(w.Header())) - response, terminalErr := codexBorrowSessionTerminal(result.RawResponse) - result.Completed = sendErr == nil && terminalErr == nil && last.status >= 200 && last.status < 300 - result.ResponseID, result.ReportedModel = response.Get("id").String(), response.Get("model").String() - if sendErr != nil { - result.Error = sendErr.Error() - } else if terminalErr != nil { - result.Error = terminalErr.Error() - } - // The replay controller needs the actual business state, never probe state. - result.TurnState = extractOpenAICodexTurnState(w.Header()) - return result - } - if d.scenario == "codex_session" && last.dispatched { - state.mu.Lock() - result.Applied = state.invocation.BorrowApplied - state.mu.Unlock() - result.StateLength = len(extractOpenAICodexTurnState(w.Header())) - d.turnState = extractOpenAICodexTurnState(w.Header()) - d.completeHTTPSessionTurn(&result, sendErr, last.status) - return result - } - if last.wsFrames && last.raw.Len() == 0 && sendErr != nil { - var eventErr *openAIWSUpstreamEventError - if errors.As(sendErr, &eventErr) && len(eventErr.payload) > 0 { - result.RawResponse = string(eventErr.payload) - result.Error = string(eventErr.payload) - } else { - result.Error = sendErr.Error() - } - return result - } - - var parseErr error - var limited bool - if last.wsFrames { - result.Answer, limited, result.ReportedModel, parseErr = parsePelicanWSFrames(result.RawResponse) - } else { - result.Answer, limited, result.ReportedModel, parseErr = parsePelicanTextResponse(last.protocol, result.RawResponse) - } - state.mu.Lock() - result.Applied = state.invocation.BorrowApplied - state.mu.Unlock() - // A valid completed terminal with text is authoritative. Read-ahead cleanup - // (often context cancellation after that terminal) cannot turn it into failure. - result.Completed = sendErr == nil && parseErr == nil && !limited && last.status >= 200 && last.status < 300 && strings.TrimSpace(result.Answer) != "" - if sendErr != nil { - result.Error = sendErr.Error() - } else if parseErr != nil { - result.Error = parseErr.Error() - } else if !result.Completed { - result.Error = "upstream did not complete with visible text" - } - return result -} - -func (d *codexBorrowDiagnostic) prepareWS(ctx context.Context, s *CodexGatewayBorrowService, req *http.Request, account *Account, model, previous, proxy string) (*codexGatewayBorrowWSTurn, error) { - key := codexGatewayBorrowWSAnchorKey{account: account.ID, model: model, scope: "diagnostic:" + d.session} - turn, err := d.anchors.begin(key, previous, time.Now()) - if err != nil { - return nil, err - } - turn.diagnostic, turn.diagnosticApplied = true, d.borrow - turn.headers = cloneHeader(req.Header) - if previous == "" && d.borrow { - borrowed, application, err := s.Apply(req, account, model, proxy, nil, false) - if err != nil { - turn.finish(nil, nil, err) - return nil, err - } - if application == nil || !application.Applied { - turn.finish(nil, nil, ErrCodexGatewayBorrowUnavailable) - return nil, ErrCodexGatewayBorrowUnavailable - } - turn.headers = cloneHeader(borrowed.Header) - } - return turn, nil -} - -func (turn *codexGatewayBorrowWSTurn) borrowed() bool { - return turn != nil && (!turn.diagnostic || turn.diagnosticApplied) -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose_contract.go b/backend/internal/service/codex_gateway_borrow_diagnose_contract.go deleted file mode 100644 index 7a0ccfffbb3a..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose_contract.go +++ /dev/null @@ -1,165 +0,0 @@ -package service - -import ( - "context" - "encoding/json" - "errors" - "sync/atomic" - "time" - - "github.com/google/uuid" - "github.com/tidwall/gjson" -) - -type codexBorrowProbeShapeContextKey struct{} - -const ( - codexBorrowProbeRanxiLiteral = "ranxi_literal_without_tier" - codexBorrowProbeRanxiTier = "ranxi_literal_with_final_tier" - codexBorrowProbeCurrent = "current_final_tier" - codexBorrowProbePlain = "current_plaintext" - codexBorrowProbeConfigured = "current_configured_encoding" - codexBorrowProbeLegacySorted = "legacy_sorted_final_tier" -) - -// These are observations of one fixed candidate, never qualifications usable -// by business requests. The original pinned route probe selects Astra; rendering -// a selected Sol model here is explicitly the downstream model extension. -type CodexBorrowProbeContext struct { - SourceAccountID int64 `json:"source_account_id"` - SubjectRole string `json:"subject_role"` - SourceRequestEncoding string `json:"source_request_encoding,omitempty"` - TargetProxyID *int64 `json:"target_proxy_id"` - CookieFingerprint string `json:"cookie_fingerprint"` - CandidateExpiresAt time.Time `json:"candidate_expires_at"` - TemplateFingerprint string `json:"template_fingerprint"` - RequestedTier string `json:"requested_service_tier"` - FinalTier string `json:"final_service_tier"` - RoutingHint string `json:"routing_hint"` - PlaintextBodyFingerprint string `json:"plaintext_body_fingerprint"` - OriginalAstraModel bool `json:"original_astra_model"` -} - -// Literal request field order from ranxi2001/sub2api d3e43f2, LGPL-3.0, -// openai_codex_state_probe.go:320-347. Tier is the only semantic addition in -// the second variant. Model always stays the explicitly selected real model. -func borrowRanxiLiteralPayload(model, tier string) []byte { - modelJSON, _ := json.Marshal(model) - body := `{"model":` + string(modelJSON) + `,"instructions":"Reply with OK.","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Reply with OK."}]}],"stream":true,"store":false,"parallel_tool_calls":true,"include":["reasoning.encrypted_content"]` - if tier != "" { - tierJSON, _ := json.Marshal(tier) - body += `,"service_tier":` + string(tierJSON) - } - return []byte(body + `}`) -} - -func (s *CodexGatewayBorrowService) diagnoseProbeContract(ctx context.Context, revision uint64, revisionCtx context.Context, request CodexBorrowDiagnosticRequest, sourceProbe bool, limit int, emit func(CodexBorrowDiagnosticEvent)) error { - var requests atomic.Int32 - d := &codexBorrowDiagnostic{limit: int32(limit), requests: &requests, borrow: true, session: uuid.NewString(), serviceTier: request.ServiceTier, - onRequest: func(count int32) { - emit(CodexBorrowDiagnosticEvent{Type: "request", Mode: "borrowed", Requests: count, Limit: limit}) - }} - ctx = context.WithValue(WithCodexGatewayBorrowObservation(ctx), codexBorrowDiagnosticContextKey{}, d) - emit(CodexBorrowDiagnosticEvent{Type: "phase", Mode: "borrowed", Limit: limit}) - account, template, proxy, err := s.accountTemplate(ctx, request.AccountID, request.Model) - if err != nil { - return err - } - policyBody, err := s.gateway.applyOpenAIFastPolicyToBody(ctx, account, request.Model, borrowObservationPayload(request.Model, "Reply with OK.", "", request.ServiceTier)) - if err != nil { - return err // A blocked local tier must not consume a source or probe request. - } - finalTier := gjson.GetBytes(policyBody, "service_tier").String() - if sourceProbe && finalTier != "" { - return errors.New("local tier policy changes the source self-check template; no model request was sent") - } - s.mu.Lock() - candidate := s.currentCandidateLocked() - current := s.candidateCurrentLocked(revision, candidate) - s.mu.Unlock() - if sourceProbe && (!current || candidate.sourceID != account.ID) { - return errors.New("source self-check requires its current live candidate; no different source is prepared or substituted") - } - if !current { - if err := s.prepareSource(ctx, revision, revisionCtx); err != nil { - return err - } - s.mu.Lock() - candidate = s.currentCandidateLocked() - current = s.candidateCurrentLocked(revision, candidate) - s.mu.Unlock() - } - if !current { - return ErrCodexGatewayBorrowChanged - } - role := "target" - if sourceProbe { - role = "source" - } - // The snapshot cannot be silently replaced halfway through the comparison. - ctx, cancel := context.WithDeadline(ctx, candidate.expires) - defer cancel() - d.prepareHTTPSessionTurn(template.Header, make(map[string]any)) - credential, err := resolveCredentialAccount(ctx, s.accounts, account) - if err != nil { - return err - } - enforceCodexIdentityHeadersForAccount(template.Header, credential, s.gateway.codexIdentityOverrideUA(account)) - setOpenAICodexRoutingHint(template.Header, account, request.Model, finalTier) - template = template.WithContext(withCodexBorrowServiceTier(ctx, finalTier)) - identity := borrowRequestFingerprint(template, request.Model, proxy, nil, candidate.cookie.Value) - // Reverse the earlier comparison: the legacy serializer runs first, so a - // later transient failure cannot always be confused with its field order. - shapes := []string{codexBorrowProbeLegacySorted, codexBorrowProbeCurrent, codexBorrowProbeRanxiLiteral} - if finalTier == "" { - shapes = []string{codexBorrowProbeLegacySorted, codexBorrowProbeCurrent} - } - if request.Comparison == "encoding" { - shapes = []string{codexBorrowProbePlain, codexBorrowProbeConfigured} - } - if sourceProbe { - shapes = []string{codexBorrowProbeCurrent} - } - for _, shape := range shapes { - if err := ctx.Err(); err != nil { - return err - } - if requests.Load() >= int32(limit) { - return ErrCodexBorrowDiagnosticBudget - } - tier := finalTier - if shape == codexBorrowProbeRanxiLiteral { - tier = "" - } - operation := withCodexBorrowServiceTier(context.WithValue(ctx, codexBorrowProbeShapeContextKey{}, shape), tier) - if request.Comparison == "encoding" { - operation = context.WithValue(operation, codexBorrowProbeConfiguredEncodingKey{}, shape == codexBorrowProbeConfigured) - } - variant := template.WithContext(operation) - started := time.Now() - verification := s.probeTarget(operation, variant, account, request.Model, proxy, nil, candidate) - body := borrowRanxiLiteralPayload(request.Model, tier) - if shape == codexBorrowProbeLegacySorted { - body = borrowObservationPayload(request.Model, "Reply with OK.", "", tier) - } - result := CodexBorrowDiagnosticResult{Scenario: "probe_contract", Mode: "borrowed", ProbeOnly: true, - Verification: &verification, DurationMS: time.Since(started).Milliseconds(), ReportedModel: verification.ReportedModel, - ProbeContext: &CodexBorrowProbeContext{SourceAccountID: candidate.sourceID, TargetProxyID: account.ProxyID, - SubjectRole: role, - SourceRequestEncoding: candidate.sourceRequestEncoding, - CookieFingerprint: borrowHash(candidate.cookie.Value), CandidateExpiresAt: candidate.expires, - TemplateFingerprint: identity, RequestedTier: request.ServiceTier, FinalTier: finalTier, RoutingHint: template.Header.Get(openAICodexRoutingHintHeader), - PlaintextBodyFingerprint: borrowHash(string(body)), OriginalAstraModel: request.Model == codexGatewayBorrowSourceModel}} - if !verification.Success { - result.FailureStage, result.FailureReason, result.Error = "target_validation", verification.Reason, verification.Error - } - // No business send occurred. Completed/applied/dispatched remain false; - // the verification contains each probe's actual terminal and STATE facts. - emit(CodexBorrowDiagnosticEvent{Type: "result", Mode: "borrowed", Result: &result, Requests: requests.Load(), Limit: limit}) - if errors.Is(ctx.Err(), context.Canceled) { - return ctx.Err() - } - } - emit(CodexBorrowDiagnosticEvent{Type: "done", Requests: requests.Load(), Limit: limit}) - return ctx.Err() -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose_contract_test.go b/backend/internal/service/codex_gateway_borrow_diagnose_contract_test.go deleted file mode 100644 index 5c4c5c8c47e8..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose_contract_test.go +++ /dev/null @@ -1,148 +0,0 @@ -package service - -import ( - "bytes" - "context" - "net/http" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexBorrowProbeContractKeepsOneCandidateAndDoesNotQualifyBusiness(t *testing.T) { - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - t.Run(model, func(t *testing.T) { - account := borrowCoreAccount(2) - var bodies [][]byte - var headers []http.Header - s := newBorrowCoreTest(t, func(req *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - assert.EqualValues(t, 2, id) - bodies = append(bodies, borrowCoreBody(t, req)) - headers = append(headers, req.Header.Clone()) - state := "minted-state" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "changed-state" // A failed verdict must not end the comparison. - } - return borrowCoreResponse(model, "OK", state, "__cflb=target-route; Secure; Path=/"), nil - }, account) - borrowCoreCandidate(s, time.Now().Add(time.Minute)) - var results []*CodexBorrowDiagnosticResult - var used int32 - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 2, Model: model, Transport: "http", ServiceTier: "priority"}, func(e CodexBorrowDiagnosticEvent) { - used = e.Requests - if e.Result != nil { - results = append(results, e.Result) - } - }) - require.NoError(t, err) - require.Len(t, results, 3) - require.EqualValues(t, 6, used) - require.Len(t, bodies, 6) - for i, result := range results { - require.True(t, result.ProbeOnly) - require.False(t, result.Dispatched) - require.False(t, result.Applied) - require.False(t, result.Completed) - require.Equal(t, "target_state_changed", result.Verification.Reason) - require.True(t, result.Verification.MintCompleted && result.Verification.ContinueCompleted) - require.Equal(t, model == "gpt-6-astra", result.ProbeContext.OriginalAstraModel) - require.Equal(t, results[0].ProbeContext.TemplateFingerprint, result.ProbeContext.TemplateFingerprint) - require.Equal(t, results[0].ProbeContext.CookieFingerprint, result.ProbeContext.CookieFingerprint) - require.Equal(t, model, gjson.GetBytes(bodies[i*2], "model").String()) - require.Equal(t, headers[0].Get("session-id"), headers[i*2].Get("session-id")) - require.NotEqual(t, headers[i*2].Get("session_id"), headers[i*2+1].Get("session_id")) - require.Equal(t, "minted-state", headers[i*2+1].Get("X-Codex-Turn-State")) - require.Contains(t, headers[i*2].Get("Cookie"), "__oailb=synthetic-borrowed-cookie") - require.Contains(t, headers[i*2+1].Get("Cookie"), "__cflb=target-route") - } - require.True(t, bytes.HasPrefix(bodies[2], []byte(`{"model":`))) - require.Equal(t, "priority", gjson.GetBytes(bodies[2], "service_tier").String()) - require.JSONEq(t, string(bodies[0]), string(bodies[2])) - require.NotEqual(t, string(bodies[0]), string(bodies[2]), "the first comparison changes field order only") - require.True(t, bytes.HasPrefix(bodies[4], []byte(`{"model":`))) - require.False(t, gjson.GetBytes(bodies[4], "service_tier").Exists()) - require.Empty(t, s.targets) - require.Empty(t, s.qualifications) - require.Empty(t, s.Status().RecentUsage, "probe-only results cannot replace real request evidence") - }) - } -} - -func TestCodexBorrowProbeContractRequestBudgetAndCancellation(t *testing.T) { - for _, cancelled := range []bool{false, true} { - account := borrowCoreAccount(2) - var calls int - ctx, cancel := context.WithCancel(context.Background()) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - if cancelled { - cancel() - } - return borrowCoreResponse("gpt-6-astra", "OK", "state"), nil - }, account) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - err := s.Diagnose(ctx, CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 2, Model: "gpt-6-astra", Transport: "http", RequestLimit: 1}, func(CodexBorrowDiagnosticEvent) {}) - cancel() - require.Error(t, err) - require.Equal(t, 1, calls) - require.Empty(t, s.targets) - } -} - -func TestCodexBorrowProbeContractStopsAtFrozenCandidateExpiry(t *testing.T) { - var sources, targets int - var s *CodexGatewayBorrowService - s = newBorrowCoreTest(t, func(req *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 { - sources++ - return borrowCoreResponse("gpt-6-astra", "OK", "source-state", "__oailb=short-candidate; Secure; Path=/; Max-Age=1"), nil - } - targets++ - deadline, ok := req.Context().Deadline() - assert.True(t, ok) - assert.True(t, deadline.Equal(s.candidate.expires), "a comparison cannot outlive its fixed cookie") - <-req.Context().Done() - return nil, req.Context().Err() - }, borrowCoreAccount(1), borrowCoreAccount(2)) - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 2, Model: "gpt-6-astra", Transport: "http"}, func(CodexBorrowDiagnosticEvent) {}) - require.ErrorIs(t, err, context.DeadlineExceeded) - require.Equal(t, 1, sources) - require.Equal(t, 1, targets) - require.Empty(t, s.qualifications) -} - -func TestCodexBorrowProbeContractRespectsTierPolicyBeforeModelCalls(t *testing.T) { - for _, action := range []string{BetaPolicyActionBlock, BetaPolicyActionFilter} { - t.Run(action, func(t *testing.T) { - var calls int - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - assert.False(t, gjson.GetBytes(borrowCoreBody(t, req), "service_tier").Exists()) - return borrowCoreResponse("gpt-6-astra", "OK", "state"), nil - }, borrowCoreAccount(2)) - policy := openAIFastFilterPriorityPolicy() - policy.Rules[0].Action = action - s.gateway.settingService = newOpenAIGatewayServiceWithSettings(t, policy).settingService - if action == BetaPolicyActionFilter { - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - } - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 2, Model: "gpt-6-astra", Transport: "http", ServiceTier: "priority"}, func(e CodexBorrowDiagnosticEvent) { - if e.Result != nil { - assert.Empty(t, e.Result.ProbeContext.FinalTier) - } - }) - if action == BetaPolicyActionBlock { - var blocked *OpenAIFastBlockedError - require.ErrorAs(t, err, &blocked) - require.Zero(t, calls) - } else { - require.NoError(t, err) - require.Equal(t, 4, calls, "a filtered tier needs no redundant middle comparison") - } - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose_session.go b/backend/internal/service/codex_gateway_borrow_diagnose_session.go deleted file mode 100644 index 775b218629c6..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose_session.go +++ /dev/null @@ -1,145 +0,0 @@ -package service - -import ( - "encoding/json" - "errors" - "net/http" - "sort" - "strings" - - "github.com/google/uuid" - "github.com/tidwall/gjson" - "github.com/tidwall/sjson" -) - -const codexBorrowEchoTool = "codex_borrow_echo" -const codexBorrowEchoValue = "borrow-continuation-check" - -// A fixed, side-effect-free tool round trip exercises native client metadata, -// STATE echo and full-history HTTP continuation without reading a user's chat. -// Each diagnostic owns its history; nothing enters business continuation stores. -func (d *codexBorrowDiagnostic) prepareHTTPSessionTurn(headers http.Header, payload map[string]any) { - headers.Set("User-Agent", CodexCanonicalUserAgent()) - headers.Set("originator", codexTUIOriginator) - headers.Set("session-id", d.session) - headers.Set("thread-id", d.session) - headers.Set("x-client-request-id", uuid.NewString()) - headers.Set("x-codex-window-id", d.session+":0") - headers.Set("x-codex-installation-id", d.session) - metadata, _ := json.Marshal(map[string]string{"installation_id": d.session, "session_id": d.session, "thread_id": d.session, "turn_id": uuid.NewString(), "window_id": d.session + ":0"}) - headers.Set("x-codex-turn-metadata", string(metadata)) - if d.turnState != "" { - headers.Set("x-codex-turn-state", d.turnState) - } - payload["client_metadata"] = map[string]string{"session_id": d.session, "thread_id": d.session, "x-codex-installation-id": d.session, "x-codex-turn-metadata": string(metadata)} - payload["prompt_cache_key"] = d.session - payload["include"] = []string{"reasoning.encrypted_content"} - payload["parallel_tool_calls"] = false - payload["tools"] = []any{map[string]any{"type": "function", "name": codexBorrowEchoTool, "description": "Return value unchanged.", "strict": true, "parameters": map[string]any{"type": "object", "properties": map[string]any{"value": map[string]string{"type": "string"}}, "required": []string{"value"}, "additionalProperties": false}}} - delete(payload, "previous_response_id") // HTTP sends complete history, as the native client does. - if len(d.history) == 0 { - d.history = []any{map[string]any{"role": "user", "content": []any{map[string]string{"type": "input_text", "text": "Call codex_borrow_echo with value borrow-continuation-check. After its result, reply with that value only."}}}} - payload["tool_choice"] = map[string]string{"type": "function", "name": codexBorrowEchoTool} - } else { - payload["tool_choice"] = "none" - } - payload["input"] = d.history -} - -func codexBorrowSessionTerminal(raw string) (gjson.Result, error) { - // Native Codex streams may omit output from the final envelope. Preserve - // completed output items (including encrypted reasoning) by their stream - // index; deltas alone never prove a complete item or a completed response. - items := make(map[int]json.RawMessage) - for _, line := range strings.Split(raw, "\n") { - if !strings.HasPrefix(line, "data:") { - continue - } - event := gjson.Parse(strings.TrimSpace(strings.TrimPrefix(line, "data:"))) - switch event.Get("type").String() { - case "response.output_item.done": - index := int(event.Get("output_index").Int()) - item := event.Get("item") - if index < 0 || index >= 128 || !item.IsObject() { - return gjson.Result{}, errors.New("invalid diagnostic output item") - } - items[index] = json.RawMessage(item.Raw) - case "response.completed", "response.done": - response := event.Get("response") - if response.Get("status").String() == "completed" && !response.Get("error").IsObject() { - if len(response.Get("output").Array()) == 0 && len(items) > 0 { - indexes := make([]int, 0, len(items)) - for index := range items { - indexes = append(indexes, index) - } - sort.Ints(indexes) - output := make([]json.RawMessage, 0, len(items)) - for _, index := range indexes { - output = append(output, items[index]) - } - encoded, err := json.Marshal(output) - if err != nil { - return gjson.Result{}, err - } - materialized, err := sjson.SetRaw(response.Raw, "output", string(encoded)) - if err != nil { - return gjson.Result{}, err - } - response = gjson.Parse(materialized) - } - return response, nil - } - return response, errors.New("upstream terminal was not completed") - case "error", "response.failed", "response.incomplete", "response.cancelled", "response.canceled": - return gjson.Result{}, errors.New(event.Raw) - } - } - return gjson.Result{}, errors.New("upstream SSE ended without a completed response") -} - -func (d *codexBorrowDiagnostic) completeHTTPSessionTurn(result *CodexBorrowDiagnosticResult, sendErr error, status int) { - response, err := codexBorrowSessionTerminal(result.RawResponse) - result.Completed = err == nil && status >= 200 && status < 300 - if sendErr != nil { - result.Error = sendErr.Error() - return - } - if err != nil { - result.Error = err.Error() - return - } - if status < 200 || status >= 300 { - result.Error = "upstream HTTP request did not succeed" - return - } - result.ResponseID, result.ReportedModel = response.Get("id").String(), response.Get("model").String() - if len(d.history) == 1 { - var callID, arguments string - for _, item := range response.Get("output").Array() { - d.history = append(d.history, json.RawMessage(item.Raw)) - if item.Get("type").String() == "function_call" && item.Get("name").String() == codexBorrowEchoTool { - callID, arguments = item.Get("call_id").String(), item.Get("arguments").String() - } - } - if callID == "" || gjson.Get(arguments, "value").String() != codexBorrowEchoValue { - result.Error = "upstream did not return the requested fixed echo tool call" - return - } - d.history = append(d.history, map[string]string{"type": "function_call_output", "call_id": callID, "output": codexBorrowEchoValue}) - result.Completed, result.Answer = true, "codex_borrow_echo(\""+codexBorrowEchoValue+"\")" - return - } - var answer strings.Builder - for _, item := range response.Get("output").Array() { - for _, content := range item.Get("content").Array() { - if content.Get("type").String() == "output_text" { - _, _ = answer.WriteString(content.Get("text").String()) - } - } - } - result.Answer = strings.TrimSpace(answer.String()) - result.ToolRoundTrip = result.Answer == codexBorrowEchoValue - if !result.ToolRoundTrip { - result.Error = "continuation did not return the fixed tool result" - } -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose_session_test.go b/backend/internal/service/codex_gateway_borrow_diagnose_session_test.go deleted file mode 100644 index 821257e16e4b..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose_session_test.go +++ /dev/null @@ -1,48 +0,0 @@ -package service - -import ( - "encoding/json" - "testing" - - "github.com/stretchr/testify/require" -) - -func TestCodexBorrowSessionRebuildsCompletedStreamItems(t *testing.T) { - // Captures the production shape without retaining any real response data: - // complete items are emitted in SSE, while response.completed has output:[]. - first := "data: " + `{"type":"response.output_item.done","output_index":1,"item":{"type":"function_call","name":"codex_borrow_echo","call_id":"synthetic-call","arguments":"{\"value\":\"borrow-continuation-check\"}"}}` + "\n\n" + - "data: " + `{"type":"response.output_item.done","output_index":0,"item":{"type":"reasoning","encrypted_content":"synthetic-encrypted-item"}}` + "\n\n" + - "data: " + `{"type":"response.completed","response":{"id":"synthetic-1","model":"gpt-6.1-sol","status":"completed","output":[]}}` + "\n\n" - diagnostic := &codexBorrowDiagnostic{history: []any{map[string]string{"role": "user"}}} - result := CodexBorrowDiagnosticResult{RawResponse: first} - diagnostic.completeHTTPSessionTurn(&result, nil, 200) - require.True(t, result.Completed, result.Error) - require.Len(t, diagnostic.history, 4, "preserve reasoning, tool call and its linked output") - reasoning, ok := diagnostic.history[1].(json.RawMessage) - require.True(t, ok) - require.Contains(t, string(reasoning), "synthetic-encrypted-item") - second := "data: " + `{"type":"response.output_item.done","output_index":0,"item":{"type":"message","role":"assistant","content":[{"type":"output_text","text":"borrow-continuation-check"}]}}` + "\n\n" + - "data: " + `{"type":"response.completed","response":{"id":"synthetic-2","model":"gpt-6.1-sol","status":"completed","output":[]}}` + "\n\n" - result = CodexBorrowDiagnosticResult{RawResponse: second} - diagnostic.completeHTTPSessionTurn(&result, nil, 200) - require.True(t, result.Completed, result.Error) - require.True(t, result.ToolRoundTrip) - require.Equal(t, codexBorrowEchoValue, result.Answer) -} - -func TestCodexBorrowSessionItemsStillRequireSuccessfulTerminal(t *testing.T) { - item := "data: " + `{"type":"response.output_item.done","output_index":0,"item":{"type":"message","content":[{"type":"output_text","text":"borrow-continuation-check"}]}}` + "\n\n" - for _, ending := range []string{"", "data: " + `{"type":"response.failed","response":{"status":"failed"}}` + "\n\n"} { - _, err := codexBorrowSessionTerminal(item + ending) - require.Error(t, err) - } -} - -func TestCodexBorrowSessionSeparatesCompletionFromTaskResult(t *testing.T) { - diagnostic := &codexBorrowDiagnostic{history: []any{"user", "tool-output"}} - result := CodexBorrowDiagnosticResult{RawResponse: "data: " + `{"type":"response.completed","response":{"status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"different answer"}]}]}}` + "\n\n"} - diagnostic.completeHTTPSessionTurn(&result, nil, 200) - require.True(t, result.Completed, "upstream completion is independent of the fixed-answer check") - require.False(t, result.ToolRoundTrip) - require.NotEmpty(t, result.Error) -} diff --git a/backend/internal/service/codex_gateway_borrow_diagnose_test.go b/backend/internal/service/codex_gateway_borrow_diagnose_test.go deleted file mode 100644 index 70265d7bb2e2..000000000000 --- a/backend/internal/service/codex_gateway_borrow_diagnose_test.go +++ /dev/null @@ -1,250 +0,0 @@ -//go:build unit - -package service - -import ( - "context" - "encoding/json" - "io" - "net/http" - "strings" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -// A real HTTP stream can deliver its final bytes together with cancellation -// while the gateway shuts down read-ahead after committing response.completed. -type borrowDiagnosticCancelledTail struct{ *strings.Reader } - -func (r *borrowDiagnosticCancelledTail) Read(p []byte) (int, error) { - n, err := r.Reader.Read(p) - if r.Reader.Len() == 0 { - return n, context.Canceled - } - return n, err -} -func (*borrowDiagnosticCancelledTail) Close() error { return nil } - -func TestCodexBorrowDiagnoseHTTPTerminalAndDispatchEvidence(t *testing.T) { - for _, complete := range []bool{true, false} { - t.Run(map[bool]string{true: "completed then read cancellation", false: "missing terminal"}[complete], func(t *testing.T) { - account := borrowCoreAccount(2) - generator, repo := newPelicanGeneratorForTest(account, &pelicanGeneratorUpstream{}) - gateway := generator.openaiGatewayService - raw := borrowCoreSSE("gpt-6-astra", "OK") - if !complete { - raw = "data: {\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\n" - } - gateway.httpUpstream = borrowCoreProbe(func(req *http.Request, _ string, id int64, n int, _ *tlsfingerprint.Profile) (*http.Response, error) { - _, finish, err := PreparePelicanHTTPRequest(req, id, n, "http") - if err != nil { - return nil, err - } - return finish(&http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"text/event-stream"}}, Body: &borrowDiagnosticCancelledTail{strings.NewReader(raw)}}, nil) - }) - s := NewCodexGatewayBorrowService(nil, repo, gateway, borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - return borrowCoreResponse("gpt-6-astra", "OK", "stable", ""), nil - }), nil) - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6-astra"}}, false) - defer s.Stop() - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - var results []CodexBorrowDiagnosticResult - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{AccountID: 2, Model: "gpt-6-astra", Transport: "http"}, func(e CodexBorrowDiagnosticEvent) { - if e.Result != nil { - results = append(results, *e.Result) - } - }) - require.NoError(t, err) - require.Len(t, results, 2) - for _, r := range results { - require.Equal(t, complete, r.Completed, r.Error) - require.True(t, r.Dispatched) - } - require.False(t, results[0].Applied) - require.True(t, results[1].Applied) - usage := s.Status().RecentUsage - require.Len(t, usage, 1) - require.True(t, usage[0].Applied) - require.EqualValues(t, 1, usage[0].AppliedCount) - if complete { - require.Equal(t, "completed", usage[0].Outcome) - } else { - require.NotEqual(t, "completed", usage[0].Outcome) - } - }) - } -} - -var _ io.ReadCloser = (*borrowDiagnosticCancelledTail)(nil) - -func TestCodexBorrowDiagnoseHTTPUsesRealSenderAndIsolatesModes(t *testing.T) { - account := borrowCoreAccount(2) - upstream := &pelicanGeneratorUpstream{body: pelicanResponsesSSE("gpt-6-astra", "completed", "")} - generator, repo := newPelicanGeneratorForTest(account, upstream) - var probes int - s := NewCodexGatewayBorrowService(nil, repo, generator.openaiGatewayService, borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - probes++ - return borrowCoreResponse("gpt-6-astra", "OK", "stable", ""), nil - }), nil) - cfg := CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6-astra"}} - s.publishConfig(cfg, false) - defer s.Stop() - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - var events []CodexBorrowDiagnosticEvent - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{AccountID: 2, Model: "gpt-6-astra", Transport: "http"}, func(event CodexBorrowDiagnosticEvent) { events = append(events, event) }) - require.NoError(t, err) - var results []CodexBorrowDiagnosticResult - for _, event := range events { - if event.Result != nil { - results = append(results, *event.Result) - } - } - require.Len(t, results, 2) - require.True(t, results[0].Completed, results[0].Error) - require.False(t, results[0].Applied) - require.True(t, results[1].Completed, results[1].Error) - require.True(t, results[1].Applied) - require.Len(t, upstream.requests, 2) - require.NotContains(t, upstream.requests[0].Header.Get("Cookie"), "__oailb") - require.Contains(t, upstream.requests[1].Header.Get("Cookie"), "__oailb=synthetic-borrowed-cookie") - require.Equal(t, cfg, s.ConfigSnapshot()) - require.EqualValues(t, len(upstream.requests)+probes, events[len(events)-1].Requests) - require.LessOrEqual(t, events[len(events)-1].Requests, int32(8)) - require.Equal(t, "diagnostic", s.Status().RecentUsage[0].Origin) -} - -func testCodexBorrowDiagnoseWS(t *testing.T, prewarm bool) { - gateway, account, borrow, probes := borrowWSFixture(t, 2) - gateway.cfg.Gateway.MaxLineSize = defaultMaxLineSize - gateway.cfg.Gateway.OpenAIWS.PrewarmGenerateEnabled = prewarm - repo := &borrowCoreAccounts{rows: map[int64]*Account{account.ID: account}} - gateway.accountRepo = repo - borrow.accounts = repo - borrow.upstream = borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - return borrowCoreResponse("gpt-6-astra", "OK", "stable", ""), nil - }) - completed := func(id string) []byte { - raw, _ := json.Marshal(map[string]any{"type": "response.completed", "response": map[string]any{"id": id, "model": "gpt-6-astra", "status": "completed", "output": []any{map[string]any{"type": "message", "role": "assistant", "content": []any{map[string]any{"type": "output_text", "text": "OK"}}}}, "usage": map[string]any{"input_tokens": 2, "output_tokens": 1}}}) - return raw - } - ordinary := &openAIWSCaptureConn{events: [][]byte{completed("ordinary-1"), completed("ordinary-2")}} - borrowed := &openAIWSCaptureConn{events: [][]byte{completed("borrowed-1"), completed("borrowed-2")}} - if prewarm { - ordinary.events = append([][]byte{completed("ordinary-prewarm")}, ordinary.events...) - borrowed.events = append([][]byte{completed("borrowed-prewarm")}, borrowed.events...) - } - dialer := &borrowWSDialer{conns: []openAIWSClientConn{ordinary, borrowed}} - gateway.openaiWSPool.setClientDialerForTest(dialer) - var requests int32 - var results []CodexBorrowDiagnosticResult - err := borrow.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{AccountID: account.ID, Model: "gpt-6-astra", Transport: "ws"}, func(event CodexBorrowDiagnosticEvent) { - requests = event.Requests - if event.Result != nil { - results = append(results, *event.Result) - } - }) - require.NoError(t, err) - require.Len(t, results, 4) - for _, result := range results { - require.True(t, result.Completed, result.Error) - } - require.False(t, results[0].Applied) - require.True(t, results[2].Applied) - require.Len(t, dialer.headers, 2) - expected := 2 - if prewarm { - expected = 3 - require.EqualValues(t, 8, requests) - } - require.Len(t, ordinary.writes, expected) - require.Len(t, borrowed.writes, expected) - require.Equal(t, "ordinary-1", ordinary.writes[len(ordinary.writes)-1]["previous_response_id"]) - require.Equal(t, "borrowed-1", borrowed.writes[len(borrowed.writes)-1]["previous_response_id"]) - require.Empty(t, probes.requests) - require.Empty(t, borrow.wsAnchors.entries, "diagnostics cannot create client anchors") -} - -func TestCodexBorrowDiagnoseWSContinuesOwnSockets(t *testing.T) { testCodexBorrowDiagnoseWS(t, false) } -func TestCodexBorrowDiagnoseWSCountsConfiguredPrewarm(t *testing.T) { - testCodexBorrowDiagnoseWS(t, true) -} - -func TestCodexBorrowDiagnoseHTTPToolContinuation(t *testing.T) { - for _, tier := range []string{"priority", "default", "flex", "ultrafast", ""} { - t.Run("tier_"+tier, func(t *testing.T) { testCodexBorrowDiagnoseHTTPToolContinuation(t, tier) }) - } -} - -func testCodexBorrowDiagnoseHTTPToolContinuation(t *testing.T, tier string) { - account := borrowCoreAccount(2) - generator, repo := newPelicanGeneratorForTest(account, &pelicanGeneratorUpstream{}) - gateway := generator.openaiGatewayService - var business, probes int - gateway.httpUpstream = borrowCoreProbe(func(req *http.Request, _ string, id int64, n int, _ *tlsfingerprint.Profile) (*http.Response, error) { - business++ - body := borrowCoreBody(t, req) - assert.Equal(t, "gpt-6.1-sol", gjson.GetBytes(body, "model").String()) - assert.Equal(t, tier, gjson.GetBytes(body, "service_tier").String()) - assert.NotEmpty(t, req.Header.Get("session-id")) - assert.Contains(t, req.Header.Get("Cookie"), "__oailb=synthetic-borrowed-cookie") - var output []any - if business == 1 { - assert.Empty(t, req.Header.Get("X-Codex-Turn-State")) - output = []any{map[string]any{"type": "function_call", "name": codexBorrowEchoTool, "call_id": "echo-call", "arguments": `{"value":"borrow-continuation-check"}`}} - } else { - assert.Equal(t, "business-turn-state", req.Header.Get("X-Codex-Turn-State")) - assert.Equal(t, "none", gjson.GetBytes(body, "tool_choice").String()) - assert.False(t, gjson.GetBytes(body, "previous_response_id").Exists()) - assert.Equal(t, "function_call_output", gjson.GetBytes(body, "input.2.type").String()) - assert.NotEmpty(t, gjson.GetBytes(body, "input.1.call_id").String()) - assert.Equal(t, gjson.GetBytes(body, "input.1.call_id").String(), gjson.GetBytes(body, "input.2.call_id").String()) - output = []any{map[string]any{"type": "message", "role": "assistant", "content": []any{map[string]string{"type": "output_text", "text": codexBorrowEchoValue}}}} - } - raw, _ := json.Marshal(map[string]any{"type": "response.completed", "response": map[string]any{"id": "response-test", "model": "gpt-6.1-sol", "status": "completed", "output": output, "usage": map[string]int{"input_tokens": 5, "output_tokens": 3}}}) - _, finish, err := PreparePelicanHTTPRequest(req, id, n, "http") - if err != nil { - return nil, err - } - headers := http.Header{"Content-Type": {"text/event-stream"}} - headers.Set("X-Codex-Turn-State", "business-turn-state") - return finish(&http.Response{StatusCode: 200, Header: headers, Body: io.NopCloser(strings.NewReader("data: " + string(raw) + "\n\n"))}, nil) - }) - s := NewCodexGatewayBorrowService(nil, repo, gateway, borrowCoreProbe(func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - probes++ - assert.False(t, gjson.GetBytes(borrowCoreBody(t, req), "service_tier").Exists()) - return borrowCoreResponse("gpt-6.1-sol", "OK", "probe-state"), nil - }), nil) - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6.1-sol"}}, false) - defer s.Stop() - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - var results []CodexBorrowDiagnosticResult - var count int32 - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{AccountID: 2, Model: "gpt-6.1-sol", Transport: "http", Scenario: "codex_session", ServiceTier: tier}, func(e CodexBorrowDiagnosticEvent) { - if e.Result != nil { - results = append(results, *e.Result) - } - count = e.Requests - }) - require.NoError(t, err) - require.Len(t, results, 2) - for _, r := range results { - require.True(t, r.Completed, r.Error) - require.True(t, r.Applied) - require.True(t, r.Dispatched) - require.NotNil(t, r.Verification) - } - require.True(t, results[1].ToolRoundTrip) - require.Equal(t, 2, business) - require.Equal(t, 4, probes, "original qualification identity includes the returned business STATE") - require.EqualValues(t, business+probes, count) - require.LessOrEqual(t, count, int32(8)) - usage := s.Status().RecentUsage[0] - require.EqualValues(t, 2, usage.AttemptCount) - require.EqualValues(t, 2, usage.Count) - require.Zero(t, usage.BlockedCount) -} diff --git a/backend/internal/service/codex_gateway_borrow_http.go b/backend/internal/service/codex_gateway_borrow_http.go deleted file mode 100644 index 33da924b84f7..000000000000 --- a/backend/internal/service/codex_gateway_borrow_http.go +++ /dev/null @@ -1,265 +0,0 @@ -package service - -import ( - "context" - "errors" - "net/http" - "slices" - "time" - - "github.com/gin-gonic/gin" -) - -const CodexGatewayBorrowPreparationFailureReason GatewayFailureReason = "codex_gateway_borrow_preparation_failed" - -type CodexGatewayBorrowFailureEvidence struct { - CacheHit bool `json:"cache_hit"` - RetryAfter *time.Time `json:"retry_after,omitempty"` - Verification *CodexGatewayBorrowVerification `json:"verification,omitempty"` -} - -// The existing failover owner may select another account. A local preparation -// failure is not an inference attempt or evidence against the selected account. -func NewCodexGatewayBorrowRequestFailure(_ error) *UpstreamFailoverError { - return &UpstreamFailoverError{ - StatusCode: http.StatusServiceUnavailable, - ResponseBody: []byte(`{"error":{"type":"upstream_error","code":"CODEX_GATEWAY_BORROW_UNAVAILABLE","message":"no qualified Codex gateway borrow route is available"}}`), - Scope: GatewayFailureScopeRequest, - Reason: CodexGatewayBorrowPreparationFailureReason, - NextAccountAction: NextAccountRetry, - RequestScopedTransient: true, - SuppressAccountHealthPenalty: true, - SafeToFailoverAfterWrite: true, - ClientStatusCode: http.StatusServiceUnavailable, - ClientErrorCode: "CODEX_GATEWAY_BORROW_UNAVAILABLE", - ClientErrorType: "upstream_error", - ClientMessage: "no qualified Codex gateway borrow route is available", - } -} - -func IsCodexGatewayBorrowRequestFailure(err error) bool { - if IsCodexGatewayBorrowFailure(err) { - return true - } - var failure *UpstreamFailoverError - return errors.As(err, &failure) && failure.Reason == CodexGatewayBorrowPreparationFailureReason -} - -// Record before converting to failover metadata: the administrator keeps the -// complete local/source cause, while exhausted client responses use their own -// existing error contract. No account or scheduler state is touched here. -func RecordCodexGatewayBorrowPreparationFailure(c *gin.Context, account *Account, err error) { - if err == nil { - return - } - detail := err.Error() - stage, reason := codexBorrowFailureStage(err) - setOpsUpstreamError(c, 0, detail, detail) - event := OpsUpstreamErrorEvent{ - Kind: string(CodexGatewayBorrowPreparationFailureReason), - Scope: string(GatewayFailureScopeRequest), - Stage: stage, - Reason: reason, - Message: detail, - Detail: detail, - } - if account != nil { - event.ProxyID, event.ProxyName = opsUpstreamProxyID(account), opsUpstreamProxyName(account) - event.Platform, event.AccountID, event.AccountName = account.Platform, account.ID, account.Name - } - if stage == "node_selection" { - // A saved account binding does not identify a node-selection attempt. - // Keep the route unknown unless the failed probe provides its snapshot. - event.ProxyID, event.ProxyName = nil, opsProxyNameUnknown - } - var failure *CodexGatewayBorrowFailure - if errors.As(err, &failure) && failure.Verification != nil { - verification := *failure.Verification - if verification.NodeIdentity != "" { - event.ProxyID, event.ProxyName = nil, verification.ProxyName - if verification.ProxyID > 0 { - event.ProxyID = &verification.ProxyID - } else { - event.ProxyName = opsProxyNameDirect - } - } - event.BorrowQualification = &CodexGatewayBorrowFailureEvidence{CacheHit: failure.CacheHit, RetryAfter: failure.RetryAfter, Verification: &verification} - } - appendOpsUpstreamError(c, event) -} - -type codexGatewayBorrowHTTPPreparationContextKey struct{} -type codexGatewayBorrowHTTPFirstOutputContextKey struct{} - -type codexGatewayBorrowHTTPPreparation struct { - service *CodexGatewayBorrowService - revision uint64 - accountID int64 - model string - proxy string - application *CodexGatewayBorrowApplication -} - -func (s *OpenAIGatewayService) codexGatewayBorrowHTTPConfigured(account *Account, model string) bool { - if s == nil || s.gatewayBorrow == nil || !codexGatewayBorrowAccountSupported(account) { - return false - } - borrow := s.gatewayBorrow - borrow.mu.Lock() - defer borrow.mu.Unlock() - return borrow.config.Enabled && slices.Contains(borrow.config.TargetAccountIDs, account.ID) && slices.Contains(borrow.config.Models, model) -} - -// Qualify the final request using the original client's cancellation/deadline, -// before the business first-output guard. WithContext shares Body/GetBody: this -// phase neither reads nor copies the business body, including a 100 MiB body. -func (s *OpenAIGatewayService) prepareCodexGatewayBorrowHTTP(ctx context.Context, req *http.Request, account *Account, model, proxy string) (*http.Request, time.Duration, error) { - // Ordinary requests retain their exact identity: continuation diagnostics - // bind an immutable body snapshot to this pointer after GetBody is disabled. - if d := borrowDiagnosticFromContext(ctx); d != nil && !d.borrow { - return req, 0, nil - } - if !CodexGatewayBorrowRequestEligible(req) || !s.codexGatewayBorrowHTTPConfigured(account, model) { - return req, 0, nil - } - started := time.Now() - tracker := s.gatewayBorrow.beginAttempt(ctx, account.ID, model, "http") - ctx = context.WithValue(ctx, codexBorrowUsageContextKey{}, tracker) - req = req.WithContext(context.WithValue(req.Context(), codexBorrowUsageContextKey{}, tracker)) - prepared := codexGatewayBorrowHTTPPreparation{service: s.gatewayBorrow, accountID: account.ID, model: model, proxy: proxy} - prepared.service.mu.Lock() - prepared.revision = prepared.service.revision - revisionCtx := prepared.service.revisionCtx - wait := prepared.service.config.WaitForQualifiedRoute && req.URL.Path == "/backend-api/codex/responses" && borrowDiagnosticFromContext(ctx) == nil && !IsPelicanGeneration(ctx) - prepared.service.mu.Unlock() - if tier, ok := req.Context().Value(codexBorrowServiceTierContextKey{}).(string); ok { - ctx = withCodexBorrowServiceTier(ctx, tier) - } - var borrowed *http.Request - var application *CodexGatewayBorrowApplication - var err error - if wait { - borrowed, application, err = prepared.service.recoverQualifiedHTTP(ctx, req, account, model, proxy, prepared.revision, revisionCtx) - } else { - borrowed, application, err = prepared.service.Apply(req.WithContext(ctx), account, model, proxy, nil, false) - } - if err != nil { - tracker.blocked(err) - return nil, 0, err - } - if application == nil || !application.Applied { - failure := &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowChanged} - tracker.blocked(failure) - return nil, 0, failure - } - prepared.application = application - req = borrowed.WithContext(context.WithValue(borrowed.Context(), codexGatewayBorrowHTTPPreparationContextKey{}, prepared)) - return req, time.Since(started), nil -} - -// Forward already performed its one preparation before starting the business -// clock. At dispatch only matching cached evidence may be used. Expiry, a save -// or an identity change must not start another probe underneath that clock. -func (s *OpenAIGatewayService) applyCodexGatewayBorrowHTTP(req *http.Request, account *Account, model, proxy string) (wire *http.Request, failure error) { - defer func() { - if failure != nil { - codexBorrowUsageFromContext(req.Context()).blocked(failure) - } - }() - prepared, ok := req.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) - if !ok { - if s.gatewayBorrow == nil { - return req, nil - } - if CodexGatewayBorrowRequestEligible(req) && s.codexGatewayBorrowHTTPConfigured(account, model) && codexBorrowUsageFromContext(req.Context()) == nil { - if d := borrowDiagnosticFromContext(req.Context()); d == nil || d.borrow { - tracker := s.gatewayBorrow.beginAttempt(req.Context(), account.ID, model, "http") - req = req.WithContext(context.WithValue(req.Context(), codexBorrowUsageContextKey{}, tracker)) - } - } - borrowed, _, err := s.gatewayBorrow.Apply(req, account, model, proxy, nil, false) - return borrowed, err - } - changed := func() (*http.Request, error) { - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowChanged} - } - if prepared.application == nil || !prepared.application.Applied { - if s.codexGatewayBorrowHTTPConfigured(account, model) && CodexGatewayBorrowRequestEligible(req) { - return changed() - } - return req, nil - } - if err := req.Context().Err(); err != nil { - return nil, err - } - if s.gatewayBorrow != prepared.service || account.ID != prepared.accountID || model != prepared.model || proxy != prepared.proxy { - return changed() - } - prepared.service.mu.Lock() - current := prepared.service.revision == prepared.revision && prepared.service.revisionCtx.Err() == nil - prepared.service.mu.Unlock() - if !current || !time.Now().Before(prepared.application.ExpiresAt) { - return changed() - } - borrowed, application, err := prepared.service.Apply(req, account, model, proxy, nil, true) - if err != nil { - return nil, err - } - if application == nil || !application.Applied || application.Fingerprint != prepared.application.Fingerprint || - application.CookieFingerprint != prepared.application.CookieFingerprint || application.SourceAccountID != prepared.application.SourceAccountID || - !application.ExpiresAt.Equal(prepared.application.ExpiresAt) { - return changed() - } - // The preparation already canonicalized this Cookie header. Validation - // must keep the bound plaintext request identity when it changes no bytes. - if slices.Equal(req.Header.Values("Cookie"), borrowed.Header.Values("Cookie")) { - if application.node != nil && HTTPUpstreamProfileFromContext(req.Context()) != HTTPUpstreamProfileOpenAIHarvest { - return req.WithContext(WithHTTPUpstreamProfile(req.Context(), HTTPUpstreamProfileOpenAIHarvest)), nil - } - return req, nil - } - return borrowed, nil -} - -type codexGatewayBorrowRecoveryContext struct { - context.Context - values context.Context -} - -func (c codexGatewayBorrowRecoveryContext) Value(key any) any { - if value := c.values.Value(key); value != nil { - return value - } - return c.Context.Value(key) -} - -// A prepared recovery request already carries the caller's cancellation and -// deadline. Preserve that exact context unless an old header guard was replaced. -// In that race retain request values, but bind lifecycle to the original caller -// without adding cancellation callbacks or letting the old guard cancel it. -func codexGatewayBorrowHTTPPreparedContext(caller, prepared context.Context, replacedGuard, recovery bool) context.Context { - if !replacedGuard { - return prepared - } - values := context.WithoutCancel(prepared) - if recovery { - return codexGatewayBorrowRecoveryContext{Context: caller, values: values} - } - return values -} - -func withCodexGatewayBorrowHTTPFirstOutputStart(ctx context.Context, start time.Time) context.Context { - return context.WithValue(ctx, codexGatewayBorrowHTTPFirstOutputContextKey{}, start) -} - -func codexGatewayBorrowHTTPFirstOutputStart(ctx context.Context, fallback time.Time) time.Time { - if IsPelicanGeneration(ctx) { - return PelicanFirstOutputStart(ctx, fallback) - } - if ctx != nil { - if start, ok := ctx.Value(codexGatewayBorrowHTTPFirstOutputContextKey{}).(time.Time); ok { - return start - } - } - return fallback -} diff --git a/backend/internal/service/codex_gateway_borrow_http_observation.go b/backend/internal/service/codex_gateway_borrow_http_observation.go deleted file mode 100644 index dd81075f45fd..000000000000 --- a/backend/internal/service/codex_gateway_borrow_http_observation.go +++ /dev/null @@ -1,21 +0,0 @@ -package service - -import "net/http" - -// Prepared forwarding restores the caller's context after qualification. Its -// immutable application proof survives that restoration; the transient Apply -// marker does not. Compare the proof against the cookie at the actual boundary. -func codexBorrowHTTPApplied(req *http.Request) bool { - if req == nil { - return false - } - cookie, err := req.Cookie("__oailb") - if err != nil || cookie.Value == "" { - return false - } - if prepared, ok := req.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation); ok { - application := prepared.application - return application != nil && application.Applied && application.CookieFingerprint == borrowHash(cookie.Value) - } - return req.Context().Value(codexBorrowAppliedContextKey{}) == true -} diff --git a/backend/internal/service/codex_gateway_borrow_http_test.go b/backend/internal/service/codex_gateway_borrow_http_test.go deleted file mode 100644 index b0127c75332a..000000000000 --- a/backend/internal/service/codex_gateway_borrow_http_test.go +++ /dev/null @@ -1,258 +0,0 @@ -package service - -import ( - "bytes" - "context" - "errors" - "io" - "net/http" - "net/http/httptest" - "strings" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/config" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/gin-gonic/gin" - "github.com/stretchr/testify/require" -) - -type borrowHTTPMutationRepo struct { - *borrowCoreAccounts - writes atomic.Int32 -} - -func (r *borrowHTTPMutationRepo) SetError(context.Context, int64, string) error { - r.writes.Add(1) - return nil -} -func (r *borrowHTTPMutationRepo) SetRateLimited(context.Context, int64, time.Time) error { - r.writes.Add(1) - return nil -} -func (r *borrowHTTPMutationRepo) SetTempUnschedulable(context.Context, int64, time.Time, string) error { - r.writes.Add(1) - return nil -} -func (r *borrowHTTPMutationRepo) UpdateExtra(context.Context, int64, map[string]any) error { - r.writes.Add(1) - return nil -} - -func TestCodexGatewayBorrowHTTPPreparationExcludedFromFirstOutputBudgets(t *testing.T) { - gin.SetMode(gin.TestMode) - var probeCalls, businessCalls atomic.Int32 - probe := borrowCoreProbe(func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if probeCalls.Add(1) == 1 { - // Longer than the entire business first-output budget. This local - // observation is bound by the original client, not that business guard. - timer := time.NewTimer(1200 * time.Millisecond) - defer timer.Stop() - select { - case <-req.Context().Done(): - return nil, req.Context().Err() - case <-timer.C: - } - return borrowCoreResponse("gpt-6-astra", "", "synthetic-state"), nil - } - return borrowCoreResponse("gpt-6-astra", "", ""), nil - }) - account := borrowCoreAccount(2) - account.Schedulable = true - borrow := newBorrowCoreTest(t, probe, account) - borrowCoreCandidate(borrow, time.Now().Add(time.Minute)) - repo := &borrowHTTPMutationRepo{borrowCoreAccounts: &borrowCoreAccounts{rows: map[int64]*Account{2: account}}} - borrow.accounts, borrow.gateway.accountRepo = repo, repo - svc := borrow.gateway - svc.cfg = &config.Config{Gateway: config.GatewayConfig{OpenAIFirstOutputTimeoutSeconds: 1, MaxLineSize: defaultMaxLineSize}} - svc.rateLimitService = &RateLimitService{accountRepo: repo} - svc.httpUpstream = borrowCoreProbe(func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - businessCalls.Add(1) - // Exercise both response-header and semantic-output waiting, after - // preparation already exceeded the original one-second deadline. - timer := time.NewTimer(100 * time.Millisecond) - defer timer.Stop() - select { - case <-req.Context().Done(): - return nil, req.Context().Err() - case <-timer.C: - } - reader, writer := io.Pipe() - go func() { - defer func() { _ = writer.Close() }() - select { - case <-req.Context().Done(): - return - case <-time.After(100 * time.Millisecond): - } - _, _ = io.WriteString(writer, "data: {\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\n"+borrowCoreSSE("gpt-6-astra", "OK")) - }() - return &http.Response{StatusCode: http.StatusOK, Header: http.Header{"Content-Type": []string{"text/event-stream"}}, Body: reader}, nil - }) - body := []byte(`{"model":"gpt-6-astra","stream":true,"reasoning":{"effort":"low"},"input":"hello"}`) - recorder := httptest.NewRecorder() - c, _ := gin.CreateTestContext(recorder) - clientCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - c.Request = httptest.NewRequest(http.MethodPost, "/v1/responses", bytes.NewReader(body)).WithContext(clientCtx) - started := time.Now() - result, err := svc.Forward(clientCtx, c, account, body) - require.NoError(t, err) - require.NotNil(t, result) - require.Equal(t, int32(2), probeCalls.Load()) - require.Equal(t, int32(1), businessCalls.Load()) - require.Contains(t, recorder.Body.String(), "OK") - require.GreaterOrEqual(t, result.Duration, 1200*time.Millisecond) - require.NotNil(t, result.FirstTokenMs) - require.GreaterOrEqual(t, *result.FirstTokenMs, 1200, "TTFT must retain preparation wall time") - require.GreaterOrEqual(t, time.Since(started), 1200*time.Millisecond) - require.Zero(t, repo.writes.Load()) - require.False(t, svc.isOpenAIAccountRuntimeBlocked(account)) - require.False(t, svc.isOpenAIAccountModelRuntimeBlocked(account, "gpt-6-astra")) -} - -func TestCodexGatewayBorrowHTTPFailureUsesAccountNeutralFailover(t *testing.T) { - for _, cause := range []error{errors.New("source proxy connection refused: complete diagnostic"), ErrCodexGatewayBorrowBusy, ErrCodexGatewayBorrowUnavailable, ErrCodexGatewayBorrowChanged} { - t.Run(cause.Error(), func(t *testing.T) { - account := borrowCoreAccount(2) - repo := &borrowHTTPMutationRepo{borrowCoreAccounts: &borrowCoreAccounts{}} - svc := &OpenAIGatewayService{accountRepo: repo, rateLimitService: &RateLimitService{accountRepo: repo}} - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - c.Request = httptest.NewRequest(http.MethodPost, "/v1/responses", nil) - err := svc.handleOpenAIUpstreamTransportError(context.Background(), c, account, &CodexGatewayBorrowFailure{Cause: cause}, false) - var failure *UpstreamFailoverError - require.ErrorAs(t, err, &failure) - require.Equal(t, GatewayFailureScopeRequest, failure.Scope) - require.Equal(t, CodexGatewayBorrowPreparationFailureReason, failure.Reason) - require.True(t, failure.SuppressAccountHealthPenalty) - require.True(t, failure.ShouldRetryNextAccount()) - require.False(t, failure.ShouldReportAccountScheduleFailure()) - // This is the same exhaustion owner used by the handler after its - // narrow no-same-account-retry policy selects another account. - svc.CooldownOpenAIRetryExhausted(context.Background(), account, "gpt-6-astra", failure) - require.Zero(t, repo.writes.Load()) - require.False(t, svc.isOpenAIAccountRuntimeBlocked(account)) - require.False(t, svc.isOpenAIAccountModelRuntimeBlocked(account, "gpt-6-astra")) - value, ok := c.Get(OpsUpstreamErrorsKey) - require.True(t, ok) - events, ok := value.([]*OpsUpstreamErrorEvent) - require.True(t, ok) - require.Len(t, events, 1) - require.Contains(t, events[0].Detail, cause.Error()) - }) - } - ordinary := (&OpenAIGatewayService{}).handleOpenAIUpstreamTransportError(context.Background(), nil, borrowCoreAccount(2), errors.New("connection refused"), false) - var failure *UpstreamFailoverError - require.ErrorAs(t, ordinary, &failure) - require.Equal(t, OpenAIPersistentTransportFailureReason, failure.Reason) - require.False(t, failure.SuppressAccountHealthPenalty) -} - -func TestCodexGatewayBorrowHTTPPreparedDispatchNeverReprepares(t *testing.T) { - for _, change := range []string{"expired", "configuration", "identity"} { - t.Run(change, func(t *testing.T) { - var probeCalls, businessCalls atomic.Int32 - probe := borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - if probeCalls.Add(1) == 1 { - return borrowCoreResponse("gpt-6-astra", "", "synthetic-state"), nil - } - return borrowCoreResponse("gpt-6-astra", "", ""), nil - }) - account := borrowCoreAccount(2) - borrow := newBorrowCoreTest(t, probe, account) - borrowCoreCandidate(borrow, time.Now().Add(time.Minute)) - req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, chatgptCodexURL, nil) - require.NoError(t, err) - req.Header.Set("Authorization", "Bearer synthetic-target") - prepared, _, err := borrow.gateway.prepareCodexGatewayBorrowHTTP(context.Background(), req, account, "gpt-6-astra", "") - require.NoError(t, err) - switch change { - case "expired": - borrow.mu.Lock() - borrow.candidate.expires = time.Now().Add(-time.Second) - borrow.mu.Unlock() - case "configuration": - cfg := borrow.ConfigSnapshot() - cfg.Enabled = false - borrow.publishConfig(cfg, false) - case "identity": - prepared.Header.Set("Authorization", "Bearer changed-target") - } - borrow.gateway.httpUpstream = borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - businessCalls.Add(1) - return nil, errors.New("must not dispatch on changed evidence") - }) - _, err = borrow.gateway.doOpenAICodexUpstream(prepared, account, "", "gpt-6-astra") - require.True(t, IsCodexGatewayBorrowFailure(err), "changed evidence must fail locally") - require.Equal(t, int32(2), probeCalls.Load(), "the guarded dispatch may only check cached evidence") - require.Zero(t, businessCalls.Load()) - }) - } -} - -type borrowHTTPUnreadBody struct{ reads atomic.Int32 } - -func (r *borrowHTTPUnreadBody) Read([]byte) (int, error) { - r.reads.Add(1) - return 0, errors.New("business body must not be inspected by the borrow gate") -} -func (*borrowHTTPUnreadBody) Close() error { return nil } - -func TestCodexGatewayBorrowHTTPOrdinaryGatingDoesNotInspectBody(t *testing.T) { - for _, ordinary := range []string{"other_model", "other_host", "other_method", "other_path", "feature_off", "other_account", "nil_service"} { - t.Run(ordinary, func(t *testing.T) { - var probes, business atomic.Int32 - probe := borrowCoreProbe(func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - probes.Add(1) - return nil, errors.New("ordinary request must not probe") - }) - account := borrowCoreAccount(2) - borrow := newBorrowCoreTest(t, probe, account) - model := "gpt-6-astra" - req, err := http.NewRequest(http.MethodPost, chatgptCodexURL, nil) - require.NoError(t, err) - switch ordinary { - case "other_model": - model = "gpt-5.5" - case "other_host": - req.URL.Host = "api.openai.com" - case "other_method": - req.Method = http.MethodGet - case "other_path": - req.URL.Path += "/compact" - case "feature_off": - cfg := borrow.ConfigSnapshot() - cfg.Enabled = false - borrow.publishConfig(cfg, false) - case "other_account": - account = borrowCoreAccount(9) - case "nil_service": - borrow.gateway.gatewayBorrow = nil - } - body := &borrowHTTPUnreadBody{} - req.Body, req.ContentLength = body, 100<<20 - req.GetBody = func() (io.ReadCloser, error) { t.Fatal("borrow gate must not reread GetBody"); return nil, nil } - prepared, elapsed, err := borrow.gateway.prepareCodexGatewayBorrowHTTP(context.Background(), req, account, model, "") - require.NoError(t, err) - require.Zero(t, elapsed) - require.Same(t, req, prepared, "ordinary requests must retain their diagnostic identity") - require.Nil(t, prepared.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{})) - require.Same(t, body, prepared.Body) - borrow.gateway.httpUpstream = borrowCoreProbe(func(got *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - business.Add(1) - require.Same(t, body, got.Body) - return &http.Response{StatusCode: http.StatusOK, Header: http.Header{}, Body: io.NopCloser(strings.NewReader("ordinary"))}, nil - }) - oldCompression := codexRequestZstd.Load() - SetCodexRequestZstdEnabled(false) - defer SetCodexRequestZstdEnabled(oldCompression) - resp, err := borrow.gateway.doOpenAICodexUpstream(prepared, account, "", model) - require.NoError(t, err) - _ = resp.Body.Close() - require.Zero(t, probes.Load()) - require.Zero(t, body.reads.Load()) - require.Equal(t, int32(1), business.Load()) - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_immediate_test.go b/backend/internal/service/codex_gateway_borrow_immediate_test.go deleted file mode 100644 index 77865b2e9a2b..000000000000 --- a/backend/internal/service/codex_gateway_borrow_immediate_test.go +++ /dev/null @@ -1,290 +0,0 @@ -package service - -import ( - "context" - "errors" - "io" - "net/http" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -func enableImmediateBorrow(t *testing.T, s *CodexGatewayBorrowService) { - t.Helper() - cfg := s.ConfigSnapshot() - cfg.WaitForQualifiedRoute, cfg.ImmediateReborrowOnFailure = true, true - cfg.NodeProxyIDs, cfg.MaxNodeAttempts = []int64{10}, 1 - s.publishConfig(cfg, false) -} - -func TestCodexBorrowImmediateConfigPreservesLegacyDefaults(t *testing.T) { - cfg, err := DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":false,"node_cooldown_seconds":0}`)) - require.NoError(t, err) - require.False(t, cfg.ImmediateReborrowOnFailure) - _, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"immediate_reborrow_on_failure":true}`)) - require.ErrorContains(t, err, "requires recovery waiting and node selection") - cfg, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"source_account_ids":[1],"target_account_ids":[2],"rotate_nodes":true,"node_proxy_ids":[10],"wait_for_qualified_route":true,"immediate_reborrow_on_failure":true,"node_cooldown_seconds":0}`)) - require.NoError(t, err) - require.True(t, cfg.ImmediateReborrowOnFailure) - require.Zero(t, cfg.NodeCooldownSeconds) -} - -func TestCodexBorrowImmediateSaveDoesNotProbeWithoutWaitingRequests(t *testing.T) { - s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) - enableImmediateBorrow(t, s) - s.settings = &borrowCoreSettings{} - var requests atomic.Int32 - base := s.upstream - s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - requests.Add(1) - return base.DoWithTLS(w, proxy, id, n, profile) - }) - _, err := s.SaveConfig(t.Context(), s.ConfigSnapshot()) - require.NoError(t, err) - require.Empty(t, s.Status().Setup.State) - require.Never(t, func() bool { return requests.Load() != 0 }, 50*time.Millisecond, time.Millisecond) -} - -func TestCodexBorrowImmediateReacquiresAfterCachedRejectionAndEachFailedRound(t *testing.T) { - var pairs atomic.Int32 - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return pairs.Add(1) >= 3 }) - enableImmediateBorrow(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - req.Header.Set("X-Codex-Turn-State", "business-state") - _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) - require.Error(t, err) - require.Len(t, *calls, 3) - identity := borrowHash("http://node-10.invalid:8080") - child := s.rotation.children[identity] - child.mu.Lock() - originalExpiry := child.candidate.expires - child.prepareFailedUntil = time.Now().Add(time.Hour) - child.prepareFailure = "old source failure" - child.mu.Unlock() - s.rotation.sourceCache.rejectSource(1, identity, time.Now()) - s.rotation.cache("gpt-6-astra").pauseSourceSearch(2, time.Now()) - ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) - defer cancel() - prepared, elapsed, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - require.Less(t, elapsed, time.Second, "must not wait 15 seconds, source cooling, or a target gateway deadline") - require.Len(t, *calls, 9, "old pair plus two fresh source acquisitions and complete target pairs") - require.Equal(t, originalExpiry, child.candidate.expires, "reacquiring the same cookie must not extend its lease") - require.Equal(t, "business-state", prepared.Header.Get("X-Codex-Turn-State")) - row := s.Status().RecentUsage[0] - require.EqualValues(t, 2, row.Recovery.Rounds) - require.EqualValues(t, 2, row.Recovery.SourceRequests) - require.EqualValues(t, 4, row.Recovery.ProbeRequests) - require.Equal(t, "recovered", row.Recovery.Phase) - require.False(t, row.Dispatched) - // The failed evidence is not erased or converted into success for Sol. - require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, identity, "", time.Now()))) - require.Zero(t, s.rotation.cache("gpt-6.1-sol").blockedUntil(2, identity, "", time.Now())) - var sent atomic.Int32 - s.gateway.httpUpstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - sent.Add(1) - require.Equal(t, int64(2), id) - require.Equal(t, "http://node-10.invalid:8080", proxy) - require.Equal(t, "business-state", w.Header.Get("X-Codex-Turn-State")) - body, err := io.ReadAll(w.Body) - require.NoError(t, err) - require.Equal(t, "business body remains unread", string(body)) - return borrowCoreResponse("gpt-6-astra", "done", "actual-response-state"), nil - }) - resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved") - require.NoError(t, err) - require.Equal(t, "actual-response-state", resp.Header.Get("X-Codex-Turn-State")) - _, err = io.ReadAll(resp.Body) - require.NoError(t, err) - require.NoError(t, resp.Body.Close()) - require.EqualValues(t, 1, sent.Load()) -} - -func TestCodexBorrowImmediateFailedSourceCannotRetainRejectedCandidate(t *testing.T) { - var pairs atomic.Int32 - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return pairs.Add(1) > 1 }) - enableImmediateBorrow(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) - require.Error(t, err) - child := s.rotation.children[borrowHash("http://node-10.invalid:8080")] - base := child.upstream - var sources atomic.Int32 - child.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 && sources.Add(1) == 1 { - return nil, errors.New("synthetic source transport failure") - } - return base.DoWithTLS(w, proxy, id, n, profile) - }) - ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second) - defer cancel() - _, _, err = s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - require.EqualValues(t, 2, sources.Load()) - require.Len(t, *calls, 6, "a failed recapture must not start probes using the retained old candidate") -} - -func TestCodexBorrowImmediateNoEligibleNodesStopsWithoutSpinning(t *testing.T) { - s, calls, proxies := nodeFixture(t, 1, func(int64, string) bool { return false }) - enableImmediateBorrow(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - delete(proxies.rows, 10) - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - require.ErrorContains(t, err, "node_candidates_unavailable") - require.Empty(t, *calls) - require.Equal(t, "stopped", s.Status().RecentUsage[0].Recovery.Phase) -} - -func TestCodexBorrowImmediateExpiredProofRequiresFreshAcquisitionAndProbes(t *testing.T) { - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) - enableImmediateBorrow(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "saved", nil, false) - require.NoError(t, err) - child := s.rotation.children[borrowHash("http://node-10.invalid:8080")] - expired := time.Now().Add(-time.Second) - child.mu.Lock() - child.candidate.expires = expired - value := child.candidate.cookie.Value - for key, check := range child.qualifications { - check.expires = expired - child.qualifications[key] = check - } - child.mu.Unlock() - s.rotation.mu.Lock() - s.rotation.leases[borrowHash(value)] = expired - s.rotation.mu.Unlock() - _, _, err = s.Apply(req, a, "gpt-6-astra", "saved", nil, true) - require.Error(t, err) - require.Len(t, *calls, 3, "an expired proof cannot dispatch or probe underneath business sending") - _, _, err = s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - require.Len(t, *calls, 6, "lease expiry requires a new source response and two complete target probes, even if the routing cookie value is the same") - require.True(t, time.Now().Before(child.candidate.expires)) - require.LessOrEqual(t, time.Until(child.candidate.expires), 230*time.Second) -} - -func TestCodexBorrowImmediateCancelledProbeReleasesPreparation(t *testing.T) { - s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) - enableImmediateBorrow(t, s) - base := s.upstream - started := make(chan struct{}, 1) - s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - if id == 2 { - started <- struct{}{} - <-w.Context().Done() - return nil, w.Context().Err() - } - return base.DoWithTLS(w, proxy, id, n, profile) - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithCancel(t.Context()) - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - <-started - cancel() - require.ErrorIs(t, <-done, context.Canceled) - require.Eventually(t, func() bool { - s.rotation.flights.mu.Lock() - defer s.rotation.flights.mu.Unlock() - return len(s.rotation.flights.active) == 0 - }, time.Second, time.Millisecond) - require.Equal(t, "cancelled", s.Status().RecentUsage[0].Recovery.Phase) -} - -func TestCodexBorrowImmediateSharedPreparationSurvivesOneCancellation(t *testing.T) { - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) - enableImmediateBorrow(t, s) - base := s.upstream - entered, release := make(chan struct{}, 1), make(chan struct{}) - s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 { - entered <- struct{}{} - select { - case <-release: - case <-req.Context().Done(): - return nil, req.Context().Err() - } - } - return base.DoWithTLS(req, proxy, id, n, profile) - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - first, second := make(chan error, 1), make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - first <- err - }() - <-entered - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - second <- err - }() - require.Eventually(t, func() bool { - s.rotation.flights.mu.Lock() - defer s.rotation.flights.mu.Unlock() - for _, f := range s.rotation.flights.active { - if f.waiters == 2 { - return true - } - } - return false - }, time.Second, time.Millisecond) - cancel() - require.ErrorIs(t, <-first, context.Canceled) - close(release) - require.NoError(t, <-second) - require.Len(t, *calls, 3) -} - -type immediateBorrowAccountGuard struct { - AccountRepository - inactive atomic.Bool -} - -func (r *immediateBorrowAccountGuard) GetByID(ctx context.Context, id int64) (*Account, error) { - a, err := r.AccountRepository.GetByID(ctx, id) - if err == nil && id == 2 && r.inactive.Load() { - copy := *a - copy.Status = "disabled" - return ©, nil - } - return a, err -} - -func TestCodexBorrowImmediateDisabledAccountCancelsAnOngoingProbe(t *testing.T) { - s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) - enableImmediateBorrow(t, s) - guard := &immediateBorrowAccountGuard{AccountRepository: s.accounts} - s.accounts, s.gateway.accountRepo = guard, guard - base := s.upstream - entered := make(chan struct{}, 1) - s.upstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - if id == 2 { - entered <- struct{}{} - <-w.Context().Done() - return nil, w.Context().Err() - } - return base.DoWithTLS(w, proxy, id, n, profile) - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithTimeout(t.Context(), 8*time.Second) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - <-entered - guard.inactive.Store(true) - require.ErrorContains(t, <-done, "account_inactive") - require.False(t, s.Status().RecentUsage[0].Dispatched) -} diff --git a/backend/internal/service/codex_gateway_borrow_pelican.go b/backend/internal/service/codex_gateway_borrow_pelican.go deleted file mode 100644 index 70d3c71652fb..000000000000 --- a/backend/internal/service/codex_gateway_borrow_pelican.go +++ /dev/null @@ -1,124 +0,0 @@ -package service - -import ( - "context" - "errors" - "net/http" - "slices" - "strings" - "time" -) - -type CodexGatewayBorrowPelicanResult struct { - RawAnswer string `json:"raw_answer"` - RawResponse string `json:"raw_response"` - ModelID string `json:"model_id"` - Effort string `json:"effort"` - Status string `json:"status"` - Error string `json:"error"` -} - -func codexGatewayBorrowModelEfforts() map[string][]string { - result := map[string][]string{} - for _, model := range DefaultCodexGatewayBorrowConfig().Models { - levels := configuredCodexGPTReasoningLevels(model) - efforts := make([]string, 0, len(levels)) - for _, level := range levels { - // The local manifest's Ultra entry describes Codex orchestration. - // The inference observation sends only native API effort values. - if level.Effort != "ultra" { - efforts = append(efforts, level.Effort) - } - } - result[model] = efforts - } - return result -} - -// GeneratePelican observes exactly one request on an already-qualified route. -// Missing, expired or mismatched evidence produces "skipped" without acquiring -// a source cookie or validating a target. The runner owns its bounded parallel -// work, persistence and preview; this method changes no route or account state. -func (s *CodexGatewayBorrowService) GeneratePelican(ctx context.Context, accountID int64, model, effort string) (*CodexGatewayBorrowPelicanResult, error) { - result := &CodexGatewayBorrowPelicanResult{ModelID: model, Effort: effort, Status: "skipped"} - if effort == "" { - effort, result.Effort = "high", "high" - } - if !slices.Contains(codexGatewayBorrowModelEfforts()[model], effort) { - result.Error = "reasoning effort is unsupported for this model" - return result, errors.New(result.Error) - } - cfg := s.ConfigSnapshot() - if !cfg.Enabled || !slices.Contains(cfg.TargetAccountIDs, accountID) || !slices.Contains(cfg.Models, model) { - result.Error = "target model is not configured for gateway borrowing" - return result, nil - } - s.mu.Lock() - rev, revisionCtx := s.revision, s.revisionCtx - s.mu.Unlock() - s.mu.Lock() - candidate := s.currentCandidateLocked() - cached := candidate != nil && time.Now().Before(candidate.expires) - s.mu.Unlock() - if !cached { - result.Error = "qualified route cache is missing or expired" - return result, nil - } - requestCtx, cancel := borrowRevisionContext(ctx, revisionCtx, codexGatewayBorrowTimeout) - defer cancel() - a, template, proxy, err := s.accountTemplate(requestCtx, accountID, model, true) - if err != nil { - result.Error = err.Error() - return result, err - } - if supported, _ := AccountTestReasoningOptions(a, model); !slices.Contains(supported, effort) { - result.Error = "selected reasoning effort is not supported by the account's local model metadata" - return result, errors.New(result.Error) - } - // Use the same finalized target identity as manual verification. Effort - // and prompt do not enter the route key; model/auth/exit/business STATE do. - req, err := borrowObservationRequest(requestCtx, template.Header, model, CodexGatewayBorrowPelicanPrompt, effort, "", nil, HTTPUpstreamProfileCodexBorrowTarget) - if err != nil { - result.Error = err.Error() - return result, err - } - borrowed, applied, err := s.Apply(req, a, model, proxy, nil, true) - if err != nil || applied == nil || !applied.Applied { - result.Error = "matching qualified route cache is missing, expired or changed" - return result, nil - } - s.mu.Lock() - valid := s.revision == rev && revisionCtx.Err() == nil && s.liveCookieLocked(applied.CookieFingerprint) - s.mu.Unlock() - if !valid { - result.Error = "gateway borrow configuration or route changed before dispatch" - return result, nil - } - // The final cached check happened immediately before this single dispatch. - // It does not write qualification results, even when the response fails. - shot, sendErr := s.sendObservation(borrowed, a, proxy, nil, model, codexGatewayBorrowPelicanMaxBody) - result.RawAnswer, result.RawResponse = shot.answer, shot.raw - if shot.model != "" { - result.ModelID = shot.model - } - if sendErr != nil { - result.Status, result.Error = "failed", borrowObservationError(shot, sendErr) - if shot.incomplete { - result.Status = "incomplete" - } - return result, sendErr - } - if shot.status != http.StatusOK || shot.errorText != "" { - result.Status, result.Error = "failed", borrowObservationError(shot, nil) - if shot.incomplete { - result.Status = "incomplete" - } - return result, nil - } - if !shot.complete || strings.TrimSpace(shot.answer) == "" { - result.Status, result.Error = "incomplete", borrowObservationError(shot, nil) - return result, nil - } - result.Status = "complete" - return result, nil -} diff --git a/backend/internal/service/codex_gateway_borrow_policy.go b/backend/internal/service/codex_gateway_borrow_policy.go deleted file mode 100644 index 495a8ed6c60e..000000000000 --- a/backend/internal/service/codex_gateway_borrow_policy.go +++ /dev/null @@ -1,51 +0,0 @@ -package service - -import ( - "context" - "sync" -) - -// Only active observations register here. Identity edits cancel their work, -// without changing or replaying a business request that has already been sent. -var borrowPolicyObservers = struct { - sync.Mutex - next uint64 - accounts map[int64]map[uint64]context.CancelFunc -}{accounts: make(map[int64]map[uint64]context.CancelFunc)} - -func borrowPolicyContext(parent context.Context, accountID int64, revision uint64) (context.Context, context.CancelFunc) { - ctx, cancel := context.WithCancel(parent) - borrowPolicyObservers.Lock() - borrowPolicyObservers.next++ - id := borrowPolicyObservers.next - if borrowPolicyObservers.accounts[accountID] == nil { - borrowPolicyObservers.accounts[accountID] = make(map[uint64]context.CancelFunc) - } - borrowPolicyObservers.accounts[accountID][id] = cancel - if currentCodexFingerprintPolicyForAccount(&Account{ID: accountID}).revision != revision { - cancel() - } - borrowPolicyObservers.Unlock() - return ctx, func() { - cancel() - borrowPolicyObservers.Lock() - delete(borrowPolicyObservers.accounts[accountID], id) - if len(borrowPolicyObservers.accounts[accountID]) == 0 { - delete(borrowPolicyObservers.accounts, accountID) - } - borrowPolicyObservers.Unlock() - } -} - -func cancelBorrowPolicyObservers(accountID int64) { - borrowPolicyObservers.Lock() - defer borrowPolicyObservers.Unlock() - for id, callbacks := range borrowPolicyObservers.accounts { - if accountID != 0 && id != accountID { - continue - } - for _, cancel := range callbacks { - cancel() - } - } -} diff --git a/backend/internal/service/codex_gateway_borrow_policy_test.go b/backend/internal/service/codex_gateway_borrow_policy_test.go deleted file mode 100644 index 9605fda2d93b..000000000000 --- a/backend/internal/service/codex_gateway_borrow_policy_test.go +++ /dev/null @@ -1,49 +0,0 @@ -package service - -import ( - "context" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" - "net/http" - "testing" - "time" -) - -func TestCodexBorrowPolicyEditCancelsOldPreparation(t *testing.T) { - for _, id := range []int64{1, 2} { - t.Run(map[int64]string{1: "source", 2: "target"}[id], func(t *testing.T) { - entered := make(chan struct{}) - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - close(entered) - <-req.Context().Done() - return nil, req.Context().Err() - }, borrowCoreAccount(1), a) - if id == 2 { - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - } - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6-astra") - require.NoError(t, err) - done := make(chan error, 1) - go func() { _, _, err := s.Apply(req, a, "gpt-6-astra", "", nil, false); done <- err }() - select { - case <-entered: - case <-time.After(time.Second): - t.Fatal("probe did not start") - } - notifyCodexFingerprintAccountChanged(id) - select { - case err := <-done: - require.Error(t, err) - case <-time.After(time.Second): - t.Fatal("old policy continued probing") - } - status := s.Status() - require.False(t, status.Preparing) - for _, target := range status.Targets { - require.False(t, target.CacheValid) - require.NotEqual(t, "validating", target.State) - } - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_probe.go b/backend/internal/service/codex_gateway_borrow_probe.go deleted file mode 100644 index 24f2a852a0ed..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe.go +++ /dev/null @@ -1,760 +0,0 @@ -package service - -// Adapted from the scoped-cookie and STATE probe protocol in ranxi2001/sub2api, -// commit fd1b5ee4eeb20961fbb783fa6f136a1704271e90 (LGPL-3.0), specifically -// backend/internal/service/openai_codex_state_probe.go. These are observation -// requests: no normal response handlers, quota writes, recovery or health hooks. - -import ( - "bufio" - "bytes" - "context" - "encoding/json" - "errors" - "fmt" - "io" - "net" - "net/http" - "strings" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/google/uuid" - "github.com/tidwall/gjson" -) - -const codexGatewayBorrowProbeMaxBody = 1 << 20 -const codexGatewayBorrowPelicanMaxBody = 32 << 20 - -type codexBorrowProbePinnedRouteKey struct{} -type codexBorrowSourceObservationKey struct{} - -type codexGatewayBorrowObservation struct { - transportPhase string - transportAttempts int - routeChanged bool - requestEncoding string - status int - state string - model string - answer string - raw string - errorText string - complete bool - incomplete bool - interrupted bool - modelMismatch bool - cookies []*http.Cookie - cookiesObservedAt time.Time -} - -// accountTemplate reuses the native credential and identity builders without -// entering business forwarding or AccountTest's mutating response handlers. -// Its model is exact; account mappings that select another model cannot lend -// one model's qualification to the requested model. -func (s *CodexGatewayBorrowService) accountTemplate(ctx context.Context, id int64, model string, allowInactive ...bool) (*Account, *http.Request, string, error) { - ctx = WithCodexGatewayBorrowObservation(ctx) - if s == nil || s.accounts == nil || s.gateway == nil { - return nil, nil, "", ErrCodexGatewayBorrowUnavailable - } - a, err := s.accounts.GetByID(ctx, id) - if err != nil || !codexGatewayBorrowAccountSupported(a) { - return nil, nil, "", fmt.Errorf("account %d is unavailable or not an eligible OAuth-like account", id) - } - if a.Status != StatusActive && (len(allowInactive) == 0 || !allowInactive[0]) { - return nil, nil, "", fmt.Errorf("account %d is inactive", id) - } - if !a.IsModelSupported(model) { - return nil, nil, "", fmt.Errorf("account %d does not allow model %s", id, model) - } - actual := normalizeOpenAIModelForUpstream(a, a.GetMappedModel(model)) - if actual != model { - return nil, nil, "", fmt.Errorf("account %d maps %s to a different model %s", id, model, actual) - } - credential, err := resolveCredentialAccount(ctx, s.accounts, a) - if err != nil { - return nil, nil, "", err - } - // GetAccessToken may perform the existing normal token refresh. It is the - // only ordinary account write allowed in this observation path. - token, _, err := s.gateway.GetAccessToken(ctx, a) - if err != nil { - return nil, nil, "", err - } - if strings.TrimSpace(token) == "" { - return nil, nil, "", errors.New("account access token is empty") - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, chatgptCodexURL, nil) - if err != nil { - return nil, nil, "", err - } - req.Host = "chatgpt.com" - req.Header.Set("Authorization", "Bearer "+token) - setOpenAIChatGPTAccountHeaders(req.Header, credential) - applyOpenAICodexProbeHeaders(req.Header) - // Manual preparation and cached-only generation rebuild this template. - // Give that synthetic window a stable identity, without changing any real - // client's window or the per-shot legacy session_id. A random window here - // would make a freshly validated proof unusable by its own next caller. - req.Header.Set("X-Codex-Window-ID", uuid.NewSHA1(uuid.NameSpaceOID, []byte(fmt.Sprintf("codex-borrow-template:%d:%s", id, model))).String()) - enforceCodexIdentityHeadersForAccount(req.Header, credential, s.gateway.codexIdentityOverrideUA(a)) - setOpenAICodexRoutingHint(req.Header, a, model, "") - req = req.WithContext(WithCodexGatewayBorrowModel(req.Context(), actual)) - proxy := "" - // Same resolver used by the native HTTP and WS forwarding paths. A shadow - // takes its parent's credential, but retains its own selected proxy. - if a.ProxyID != nil && a.Proxy != nil { - proxy = a.Proxy.URL() - } - return a, req, proxy, nil -} - -func (s *CodexGatewayBorrowService) acquireSource(ctx context.Context, id int64) (*codexGatewayBorrowCandidate, string, error) { - ctx, policyCancel := borrowPolicyContext(ctx, id, currentCodexFingerprintPolicyForAccount(&Account{ID: id}).revision) - defer policyCancel() - a, template, proxy, err := s.accountTemplate(ctx, id, codexGatewayBorrowSourceModel) - if err != nil { - return nil, err.Error(), err - } - req, err := borrowSourceObservationRequest(ctx, template.Header) - if err != nil { - return nil, err.Error(), err - } - if s.node != nil { - proxy = s.node.proxy - req = req.WithContext(WithHTTPUpstreamProfile(req.Context(), HTTPUpstreamProfileOpenAIHarvest)) - req.Close = true - } - shot, err := s.sendObservation(req, a, proxy, nil, codexGatewayBorrowSourceModel, codexGatewayBorrowProbeMaxBody) - if err != nil { - return nil, shot.errorText, err - } - if shot.status != http.StatusOK || !shot.complete || strings.TrimSpace(shot.answer) == "" || shot.model != codexGatewayBorrowSourceModel || shot.modelMismatch { - detail := shot.errorText - if detail == "" { - detail = fmt.Sprintf("source response is unqualified: HTTP %d, completed=%t, reported model=%q, visible text=%t", shot.status, shot.complete, shot.model, strings.TrimSpace(shot.answer) != "") - } - return nil, detail, errors.New("source_response_not_qualified") - } - candidate := borrowCandidateFromCookies(shot.cookies, "/backend-api/codex/responses", shot.cookiesObservedAt) - if candidate == nil { - return nil, "routing cookie missing, invalid, deleted or expired", errors.New("source_routing_cookie_unavailable") - } - candidate.sourceID = id - candidate.node = s.node - candidate.sourceRequestEncoding = shot.requestEncoding - if s.node != nil && s.node.expires != nil && s.node.expires.Before(candidate.expires) { - candidate.expires = *s.node.expires - } - if s.nodeLease != nil { - candidate.expires = s.nodeLease(candidate.cookie.Value, candidate.expires) - } - return candidate, "", nil -} - -// The pinned ranxi source acquisition uses the ordinary OAuth account-test -// payload, not the target's two-shot STATE-probe protocol. Preserve the shared -// account identity and observation isolation, but do not manufacture probe -// sessions, a window or a routing hint, or force this normal source connection -// closed. The source keeps its own configured OpenAI transport and proxy. -func borrowSourceObservationRequest(ctx context.Context, headers http.Header) (*http.Request, error) { - ctx = WithCodexGatewayBorrowObservation(WithHTTPUpstreamRedirectsDisabled(WithHTTPUpstreamProfile(ctx, HTTPUpstreamProfileCodexBorrowSource))) - ctx = context.WithValue(ctx, codexBorrowProbeConfiguredEncodingKey{}, false) - ctx = context.WithValue(ctx, codexBorrowSourceObservationKey{}, true) - payload := createOpenAITestPayload(codexGatewayBorrowSourceModel, true, "Reply with OK only.") - payload["reasoning"] = map[string]string{"effort": "medium"} - body, err := json.Marshal(payload) - if err != nil { - return nil, err - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, chatgptCodexURL, bytes.NewReader(body)) - if err != nil { - return nil, err - } - req.Host, req.Header = "chatgpt.com", headers.Clone() - if req.Header == nil { - req.Header = make(http.Header) - } - for _, name := range []string{"Cookie", "Content-Length", "Content-Encoding", "Content-MD5", "X-Codex-Turn-State", "X-Codex-Turn-Metadata", "X-OpenAI-Internal-Codex-Responses-Lite", "conversation_id", "session_id", "session-id", "thread-id", "x-client-request-id", "x-codex-window-id", openAICodexRoutingHintHeader} { - deleteOpenAIHeaderEqualFold(req.Header, name) - } - req.Header.Set("Accept", "text/event-stream") - req.Header.Set("Content-Type", "application/json") - req.Header.Set("OpenAI-Beta", "responses=experimental") - return req, nil -} - -func borrowCookieScoped(cookie *http.Cookie, path string) bool { - if cookie == nil || (cookie.Domain != "" && strings.TrimPrefix(cookie.Domain, ".") != "chatgpt.com") { - return false - } - scope := cookie.Path - if scope == "" { - scope = path[:strings.LastIndex(path, "/")] - } - return strings.HasPrefix(scope, "/") && borrowCookiePathMatches(path, scope) -} - -func borrowCookieDeleted(cookie *http.Cookie, now time.Time) bool { - return cookie.Value == "" || cookie.MaxAge < 0 || (cookie.MaxAge == 0 && !cookie.Expires.IsZero() && !now.Before(cookie.Expires)) -} - -func borrowCandidateFromCookies(cookies []*http.Cookie, path string, now time.Time) *codexGatewayBorrowCandidate { - var candidate *codexGatewayBorrowCandidate - for _, cookie := range cookies { - if cookie.Name != "__oailb" || !cookie.Secure || !borrowCookieScoped(cookie, path) { - continue - } - // Deletion applies even after an otherwise usable Set-Cookie line. - if borrowCookieDeleted(cookie, now) { - return nil - } - expires := now.Add(codexGatewayBorrowTTL) - if cookie.MaxAge > 0 { - if cookie.MaxAge < int(codexGatewayBorrowTTL/time.Second) { - expires = now.Add(time.Duration(cookie.MaxAge) * time.Second) - } - } else { - // Pinned upstream requires an explicit lifetime; a session cookie - // is not a transferable bounded route lease. - if cookie.Expires.IsZero() { - continue - } - if cookie.Expires.Before(expires) { - expires = cookie.Expires - } - } - if !now.Before(expires) { - continue - } - copy := *cookie - if copy.Path == "" { - copy.Path = path[:strings.LastIndex(path, "/")] - } - candidate = &codexGatewayBorrowCandidate{cookie: copy, expires: expires} - } - return candidate -} - -func (s *CodexGatewayBorrowService) probeTarget(ctx context.Context, template *http.Request, account *Account, model, proxy string, profile *tlsfingerprint.Profile, candidate *codexGatewayBorrowCandidate) CodexGatewayBorrowVerification { - ctx = withCodexBorrowPlainProbe(ctx) - expires := candidate.expires - result := CodexGatewayBorrowVerification{AccountID: account.ID, Model: model, CheckedAt: time.Now(), ExpiresAt: &expires, Reason: "target_probe_failed", RequestShape: codexBorrowProbeRanxiLiteral} - if s.node != nil { - result.ProxyID, result.NodeIdentity, result.Gateway = s.node.id, s.node.identity, codexBorrowGateway(candidate.cookie.Value) - result.ProxyName = s.node.name - } - if shape, _ := ctx.Value(codexBorrowProbeShapeContextKey{}).(string); shape != "" { - result.RequestShape = shape - } else { - // The pinned verdict was measured with this fixed byte template, not a - // generic map serializer. Only the selected model is parameterized. - ctx = context.WithValue(ctx, codexBorrowProbeShapeContextKey{}, codexBorrowProbeRanxiLiteral) - } - // Business tier stays on the real request; the original probe has none. - shape, _ := ctx.Value(codexBorrowProbeShapeContextKey{}).(string) - if shape != codexBorrowProbeRanxiLiteral { - result.ServiceTier = codexBorrowRequestServiceTier(template) - } - ctx = withCodexBorrowServiceTier(ctx, result.ServiceTier) - // A two-shot STATE verdict is one observation. Releasing account capacity - // after mint would let another model/window mint a ticket before continuation. - // Keep the shared account lease across both shots; source preparation already - // finished, so this cannot hold a target permit while waiting for a source. - ctx, release, err := acquireCodexBorrowObservation(ctx, account.ID) - if err != nil { - result.Error = err.Error() - return result - } - defer release() - cookies := []*http.Cookie{{Name: "__oailb", Value: candidate.cookie.Value}} - fire := func(state string, cookies []*http.Cookie) (codexGatewayBorrowObservation, error) { - shotCtx, cancel := context.WithTimeout(ctx, codexGatewayBorrowShotTimeout) - defer cancel() - shotCtx = context.WithValue(shotCtx, codexBorrowProbePinnedRouteKey{}, candidate.cookie.Value) - return s.fireObservation(shotCtx, template.Header, account, model, "Reply with OK.", "", proxy, profile, state, cookies, HTTPUpstreamProfileCodexBorrowTarget, codexGatewayBorrowProbeMaxBody) - } - mint, err := fire("", cookies) - result.MintTransportPhase, result.MintTransportAttempts = mint.transportPhase, mint.transportAttempts - result.MintStatus, result.ReportedModel = mint.status, mint.model - result.MintRequestEncoding = mint.requestEncoding - result.MintCompleted, result.MintStateLength = mint.complete, len(mint.state) - if mint.routeChanged { - result.RouteChanged = true - result.Reason, result.Error = "target_route_changed", "target response changed or deleted the fixed __oailb route" - return result - } - if err != nil || !borrowShotUsable(mint) { - borrowTargetObservationFailure(ctx, &result, "mint", mint, err) - return result - } - if mint.state == "" { - result.Reason, result.Error = "target_state_missing", "first response did not return x-codex-turn-state" - return result - } - result.Minted = true - // Only the target's newly returned __cflb travels into the second shot. - // __oailb remains the exact borrowed candidate, never a target replacement. - for _, cookie := range mint.cookies { - if cookie.Name == "__cflb" && cookie.Value != "" && cookie.MaxAge >= 0 { - // The original replacement helper retains the target's __cflb first, - // then appends the pinned __oailb. No business cookie header is changed. - cookies = append([]*http.Cookie{{Name: cookie.Name, Value: cookie.Value}}, cookies...) - break - } - } - continued, err := fire(mint.state, cookies) - result.ContinueTransportPhase, result.ContinueTransportAttempts = continued.transportPhase, continued.transportAttempts - result.ContinueStatus = continued.status - result.ContinueRequestEncoding = continued.requestEncoding - result.ContinueCompleted, result.ContinueStateLength = continued.complete, len(continued.state) - if continued.model != "" { - result.ReportedModel = continued.model - } - if continued.routeChanged { - result.RouteChanged = true - result.Reason, result.Error = "target_route_changed", "target response changed or deleted the fixed __oailb route" - return result - } - if err != nil || !borrowShotUsable(continued) { - borrowTargetObservationFailure(ctx, &result, "continue", continued, err) - return result - } - result.NewTicket = continued.state != "" && continued.state != mint.state - if result.NewTicket { - result.Reason = "target_state_changed" - return result - } - result.Success, result.Reason = true, "target_probe_passed" - return result -} - -func borrowShotUsable(shot codexGatewayBorrowObservation) bool { - return shot.status == http.StatusOK && shot.complete && shot.errorText == "" -} - -// A transport interruption is inconclusive, not a reusable rejection. Keep its -// evidence, but let the next independent request run a fresh complete pair. -// Never retry this observation or replay an uncertain business generation here. -func borrowTargetObservationFailure(ctx context.Context, result *CodexGatewayBorrowVerification, phase string, shot codexGatewayBorrowObservation, err error) { - result.Error = borrowObservationError(shot, err) - result.FailedShot, result.FailureKind = phase, "response" - if ctx.Err() != nil || errors.Is(err, context.Canceled) { - result.FailureKind = "cancelled" - return - } - if shot.interrupted { - result.FailureKind = "stream_incomplete" - result.RetryOnNextRequest = true - } - var transport *codexBorrowTransportError - var network net.Error - if err != nil && (errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) || errors.As(err, &transport) || errors.As(err, &network)) { - result.FailureKind = "transport" - result.RetryOnNextRequest = true - } -} - -func borrowObservationError(shot codexGatewayBorrowObservation, err error) string { - if shot.errorText != "" { - return shot.errorText - } - if err != nil { - return err.Error() - } - if shot.status != http.StatusOK { - return fmt.Sprintf("API returned %d: %s", shot.status, shot.raw) - } - if !shot.complete { - return "upstream SSE ended without a completed response" - } - return "completed response contained no visible output text" -} - -func borrowTargetRouteChanged(cookies []*http.Cookie, candidate *codexGatewayBorrowCandidate, now time.Time) bool { - for _, cookie := range cookies { - if cookie.Name == "__oailb" && (cookie.Value != candidate.cookie.Value || cookie.MaxAge < 0 || (!cookie.Expires.IsZero() && !now.Before(cookie.Expires))) { - return true - } - } - return false -} - -func borrowObservationPayload(model, prompt, effort string, serviceTier ...string) []byte { - body := map[string]any{"model": model, "instructions": prompt, - "input": []map[string]any{{"type": "message", "role": "user", "content": []map[string]any{{"type": "input_text", "text": prompt}}}}, - "stream": true, "store": false, "parallel_tool_calls": true, "include": []string{"reasoning.encrypted_content"}} - if effort != "" { - body["reasoning"] = map[string]string{"effort": effort} - } - if len(serviceTier) > 0 && serviceTier[0] != "" { - body["service_tier"] = serviceTier[0] - } - raw, _ := json.Marshal(body) - return raw -} - -// The native sender owns this hint and derives it from the finalized body. -// Reading its tier avoids consuming/rebuilding the potentially large business -// body and keeps the short probe's body consistent with its routing headers. -func borrowProbeServiceTier(headers http.Header) string { - for _, part := range strings.Split(headers.Get(openAICodexRoutingHintHeader), ";") { - if key, value, ok := strings.Cut(part, "="); ok && strings.TrimSpace(key) == "tier" { - switch value = strings.TrimSpace(value); value { - case OpenAIFastTierPriority, OpenAIFastTierFlex, OpenAIFastTierUltrafast: - return value - } - } - } - return "" -} - -func borrowObservationRequest(ctx context.Context, headers http.Header, model, prompt, effort, state string, cookies []*http.Cookie, purpose HTTPUpstreamProfile) (*http.Request, error) { - ctx = WithCodexGatewayBorrowObservation(WithHTTPUpstreamRedirectsDisabled(WithHTTPUpstreamProfile(ctx, purpose))) - tier := codexBorrowRequestServiceTier((&http.Request{Header: headers}).WithContext(ctx)) - body := borrowObservationPayload(model, prompt, effort, tier) - if shape, _ := ctx.Value(codexBorrowProbeShapeContextKey{}).(string); shape != "" && shape != codexBorrowProbeLegacySorted { - body = borrowRanxiLiteralPayload(model, tier) - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, chatgptCodexURL, bytes.NewReader(body)) - if err != nil { - return nil, err - } - req.Host, req.Close, req.Header = "chatgpt.com", true, headers.Clone() - if req.Header == nil { - req.Header = make(http.Header) - } - for _, name := range []string{"Cookie", "Content-Length", "Content-Encoding", "Content-MD5", "X-Codex-Turn-State", "X-Codex-Turn-Metadata", "X-OpenAI-Internal-Codex-Responses-Lite", "conversation_id"} { - req.Header.Del(name) - } - req.Header.Set("Accept", "text/event-stream") - req.Header.Set("Content-Type", "application/json") - req.Header.Set("OpenAI-Beta", "responses=experimental") - // Pinned ranxi d3e43f2: each shot gets a fresh legacy session_id, while - // native session/thread/window headers in the template remain unchanged. - req.Header.Set("session_id", uuid.NewString()) - if state != "" { - req.Header.Set("X-Codex-Turn-State", state) - } - for _, cookie := range cookies { - req.AddCookie(cookie) - } - return req, nil -} - -func (s *CodexGatewayBorrowService) fireObservation(ctx context.Context, headers http.Header, account *Account, model, prompt, effort, proxy string, profile *tlsfingerprint.Profile, state string, cookies []*http.Cookie, purpose HTTPUpstreamProfile, maxBody int64) (codexGatewayBorrowObservation, error) { - var observation codexGatewayBorrowObservation - if s.upstream == nil { - return observation, ErrCodexGatewayBorrowUnavailable - } - req, err := borrowObservationRequest(ctx, headers, model, prompt, effort, state, cookies, purpose) - if err != nil { - return observation, err - } - return s.sendObservation(req, account, proxy, profile, model, maxBody) -} - -func (s *CodexGatewayBorrowService) sendObservation(req *http.Request, account *Account, proxy string, profile *tlsfingerprint.Profile, expectedModel string, maxBody int64) (codexGatewayBorrowObservation, error) { - var shot codexGatewayBorrowObservation - if err := consumeBorrowDiagnosticRequest(req.Context()); err != nil { - return shot, err - } - leasedCtx, release, acquireErr := acquireCodexBorrowObservation(req.Context(), account.ID) - if acquireErr != nil { - shot.errorText = acquireErr.Error() - return shot, acquireErr - } - defer release() - req = req.WithContext(leasedCtx) - if pelicanExecutionFromContext(leasedCtx) != nil { - var capacity *ConcurrencyService - if capture := pelicanCaptureFromContext(leasedCtx); capture != nil { - capacity = capture.concurrency - } else if s.gateway != nil { - capacity = s.gateway.concurrencyService - } - releaseCapacity, capacityErr := acquirePelicanBusinessAccountSlot(leasedCtx, capacity, account.ID, account.Concurrency) - if capacityErr != nil { - shot.errorText = capacityErr.Error() - return shot, capacityErr - } - defer releaseCapacity() - } - wire := req - if configured, probe := req.Context().Value(codexBorrowProbeConfiguredEncodingKey{}).(bool); !probe || configured { - var err error - wire, err = prepareOpenAICodexWireRequest(req, account) - if err != nil { - return shot, err - } - } - shot.requestEncoding = strings.TrimSpace(wire.Header.Get("Content-Encoding")) - if shot.requestEncoding == "" { - shot.requestEncoding = "identity" - } - resp, phase, attempts, err := s.sendBorrowObservationHTTP(wire, proxy, account, profile) - codexBorrowUsageFromContext(req.Context()).recoveryObservation(req.Context().Value(codexBorrowSourceObservationKey{}) == true, attempts) - shot.transportPhase, shot.transportAttempts = phase, attempts - if err != nil { - if resp != nil && resp.Body != nil { - _ = resp.Body.Close() - } - shot.errorText = err.Error() - return shot, err - } - if resp == nil { - return shot, errors.New("nil upstream observation response") - } - shot.status, shot.state, shot.cookies, shot.cookiesObservedAt = resp.StatusCode, strings.TrimSpace(extractOpenAICodexTurnState(resp.Header)), resp.Cookies(), time.Now() - // Match the original target wrapper: a changed route ends the probe at - // headers, even when the body would stall, fail or report a completed stream. - if pinned, _ := req.Context().Value(codexBorrowProbePinnedRouteKey{}).(string); pinned != "" && borrowTargetRouteChanged(shot.cookies, &codexGatewayBorrowCandidate{cookie: http.Cookie{Value: pinned}}, shot.cookiesObservedAt) { - shot.routeChanged, shot.errorText = true, "target_route_changed" - if resp.Body != nil { - _ = resp.Body.Close() - } - return shot, errors.New("target_route_changed") - } - if resp.Body == nil { - shot.errorText = "upstream observation response has no body" - return shot, nil - } - defer func() { _ = resp.Body.Close() }() - var data []byte - var readErr error - pinnedProbe, _ := req.Context().Value(codexBorrowProbePinnedRouteKey{}).(string) - sourceProbe := req.Context().Value(codexBorrowSourceObservationKey{}) == true - if shot.status == http.StatusOK && pinnedProbe == "" { - data, readErr = readCodexGatewayBorrowSSE(resp.Body, maxBody, sourceProbe) - } else { - data, readErr = io.ReadAll(io.LimitReader(resp.Body, maxBody+1)) - } - shot.raw = string(data) - if sourceProbe && shot.status == http.StatusOK { - contentType := strings.ToLower(resp.Header.Get("Content-Type")) - if contentType != "" && !strings.HasPrefix(contentType, "text/event-stream") { - shot.errorText = "source response content type is not an SSE stream: " + shot.raw - return shot, nil - } - } - if shot.status == http.StatusOK { - stream := parseCodexGatewayBorrowSSE(data, expectedModel, maxBody) - shot.model, shot.answer, shot.complete, shot.modelMismatch, shot.errorText, shot.incomplete = stream.model, stream.answer, stream.complete, stream.modelMismatch, stream.errorText, stream.incomplete - if pinnedProbe != "" { - // Original target probes read the bounded body to EOF and require a valid - // completion with a terminated SSE event. Do not substitute the ordinary - // business sender's early-terminal optimization for this criterion. - validationErr := validateCodexGatewayBorrowPinnedProbeResponse(data) - shot.complete = validationErr == nil - shot.interrupted = errors.Is(validationErr, errCodexBorrowProbeNoCompletion) || errors.Is(validationErr, errCodexBorrowProbeUnterminated) - if validationErr == nil { - shot.errorText = "" - } else if shot.errorText == "" { - shot.errorText = validationErr.Error() - } - } - if sourceProbe && shot.complete && !bytes.HasSuffix(data, []byte("\n\n")) && !bytes.HasSuffix(data, []byte("\r\n\r\n")) { - shot.complete, shot.errorText = false, "unterminated source probe event" - } - } - if readErr != nil { - shot.complete, shot.incomplete = false, true - if shot.errorText != "" { - shot.errorText += "\n" - } - shot.errorText += readErr.Error() - return shot, readErr - } - if int64(len(data)) > maxBody { - shot.complete, shot.incomplete, shot.errorText = false, true, fmt.Sprintf("observation response exceeded %d bytes", maxBody) - return shot, errors.New(shot.errorText) - } - if shot.status != http.StatusOK { - shot.errorText = fmt.Sprintf("API returned %d: %s", shot.status, shot.raw) - return shot, nil - } - return shot, nil -} - -// Both public-request preparation and manual diagnostics use the same account -// and global permits. A sequential target pair lends its active lease to each -// shot; a standalone source observation acquires and releases its own permit. -func acquireCodexBorrowObservation(ctx context.Context, accountID int64) (context.Context, func(), error) { - if pelicanExecutionFromContext(ctx) != nil { - return AcquirePelicanExecution(ctx, accountID) - } - if err := ctx.Err(); err != nil { - return ctx, nil, err - } - if lease, _ := ctx.Value(pelicanExecutionLeaseContextKey{}).(*pelicanExecutionLease); lease != nil && lease.active.Load() { - if lease.accountID != accountID { - return ctx, nil, errors.New("borrow observation cannot acquire another account while holding a probe lease") - } - return ctx, func() {}, nil - } - release, err := sharedPelicanExecution.acquire(ctx, accountID) - if err != nil { - return ctx, nil, err - } - lease := &pelicanExecutionLease{accountID: accountID} - lease.active.Store(true) - return context.WithValue(ctx, pelicanExecutionLeaseContextKey{}, lease), func() { - if lease.active.Swap(false) { - release() - } - }, nil -} - -// A Responses terminal event is authoritative even when the HTTP connection -// remains open. Collect the original bytes, without waiting for EOF or reading -// beyond that terminal payload; sendObservation closes the body immediately. -// The existing pure parser still decides completion, text, models and errors. -func readCodexGatewayBorrowSSE(body io.Reader, maxBody int64, requireDelimiter ...bool) ([]byte, error) { - strict := len(requireDelimiter) > 0 && requireDelimiter[0] - reader := bufio.NewReader(io.LimitReader(body, maxBody+1)) - var raw bytes.Buffer - var lines []string - terminal := func() bool { - payload := strings.TrimSpace(strings.Join(lines, "\n")) - if !gjson.Valid(payload) { - return false - } - switch gjson.Get(payload, "type").String() { - case "response.completed", "response.done", "response.failed", "response.incomplete", "response.cancelled", "response.canceled", "error": - return true - default: - return false - } - } - for { - line, err := reader.ReadString('\n') - _, _ = raw.WriteString(line) - if int64(raw.Len()) > maxBody { - return raw.Bytes(), nil // The caller reports the unchanged body bound. - } - text := strings.TrimSuffix(strings.TrimSuffix(line, "\n"), "\r") - if text == "" { - if strict && terminal() { - return raw.Bytes(), nil - } - lines = nil - } else if strings.HasPrefix(text, "data:") { - lines = append(lines, strings.TrimPrefix(strings.TrimPrefix(text, "data:"), " ")) - if !strict && terminal() { - // Preserve an already-received blank event delimiter byte for - // byte. Never block waiting for a delimiter or subsequent EOF. - if buffered := reader.Buffered(); buffered > 0 { - remaining, _ := reader.Peek(buffered) - length := 0 - if remaining[0] == '\n' { - length = 1 - } else if len(remaining) >= 2 && remaining[0] == '\r' && remaining[1] == '\n' { - length = 2 - } - _, _ = raw.Write(remaining[:length]) - _, _ = reader.Discard(length) - } - return raw.Bytes(), nil - } - } - if err != nil { - if errors.Is(err, io.EOF) { - return raw.Bytes(), nil - } - return raw.Bytes(), err - } - } -} - -// Parsing is local and records full error payloads. A bare [DONE], malformed -// event, premature EOF, cancellation or failed/incomplete response cannot be -// mistaken for a completed observation. Source and manual-generation callers -// separately require a visible answer; the target STATE criterion does not. -func parseCodexGatewayBorrowSSE(data []byte, expectedModel string, maxEvent int64) codexGatewayBorrowObservation { - var out codexGatewayBorrowObservation - scanner := bufio.NewScanner(bytes.NewReader(data)) - scanner.Buffer(make([]byte, 4096), int(maxEvent)+1) - var lines []string - var deltas strings.Builder - var terminal bool - flush := func() { - if len(lines) == 0 { - return - } - payload := strings.TrimSpace(strings.Join(lines, "\n")) - lines = nil - if payload == "" || payload == "[DONE]" { - return - } - if !gjson.Valid(payload) { - out.errorText = "malformed upstream SSE data: " + payload - return - } - event := gjson.Parse(payload) - model := event.Get("response.model").String() - if model != "" { - out.model = model - if model != expectedModel { - out.modelMismatch = true - } - } - switch event.Get("type").String() { - case "response.output_text.delta": - _, _ = deltas.WriteString(event.Get("delta").String()) - case "error", "response.failed", "response.incomplete", "response.cancelled", "response.canceled": - out.errorText = payload - out.incomplete = event.Get("type").String() == "response.incomplete" - terminal = true - case "response.completed", "response.done": - if terminal { - out.errorText = "multiple upstream terminal events: " + payload - return - } - terminal = true - // The source requires the terminal response's explicit model, not - // a model advertised only by an earlier response.created event. - out.model = model - if event.Get("response.status").String() != "completed" || (event.Get("response.error").Exists() && event.Get("response.error").Type != gjson.Null) { - out.errorText = payload - return - } - var final strings.Builder - for _, item := range event.Get("response.output").Array() { - for _, content := range item.Get("content").Array() { - if content.Get("type").String() == "output_text" { - _, _ = final.WriteString(content.Get("text").String()) - } - } - } - out.answer = final.String() - if out.answer == "" { - out.answer = deltas.String() - } - out.complete = true - } - } - for scanner.Scan() { - line := strings.TrimSuffix(scanner.Text(), "\r") - if line == "" { - flush() - continue - } - if strings.HasPrefix(line, "data:") { - lines = append(lines, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " ")) - } - } - flush() - if err := scanner.Err(); err != nil { - out.errorText = err.Error() - } - if out.answer == "" { - out.answer = deltas.String() - } - if out.errorText != "" { - out.complete = false - } - if !terminal && out.errorText == "" { - out.incomplete = true - } - return out -} diff --git a/backend/internal/service/codex_gateway_borrow_probe_serial_test.go b/backend/internal/service/codex_gateway_borrow_probe_serial_test.go deleted file mode 100644 index e565fe980cb2..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe_serial_test.go +++ /dev/null @@ -1,116 +0,0 @@ -package service - -import ( - "context" - "net/http" - "sync" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -func TestCodexBorrowProbePairCannotInterleaveOnSameAccount(t *testing.T) { - for _, manual := range []bool{false, true} { - name := "business" - if manual { - name = "diagnostic" - } - t.Run(name, func(t *testing.T) { - ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) - defer cancel() - if manual { - state := newPelicanExecutionState(ctx, sharedPelicanExecution, time.Minute, time.Now) - defer state.finish() - ctx = context.WithValue(state.parent, pelicanExecutionContextKey{}, state) - } - account := borrowCoreAccount(92001) - started, release := make(chan struct{}), make(chan struct{}) - var mu sync.Mutex - var order []string - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - id := req.Header.Get("X-Codex-Window-ID") - phase := "mint" - if req.Header.Get("X-Codex-Turn-State") != "" { - phase = "continue" - } - mu.Lock() - order = append(order, id+":"+phase) - mu.Unlock() - if id == "first" && phase == "mint" { - close(started) - select { - case <-release: - case <-req.Context().Done(): - return nil, req.Context().Err() - } - } - return borrowCoreResponse("gpt-6-astra", "OK", id), nil - }, account) - borrowCoreCandidate(s, time.Now().Add(time.Minute)) - run := func(id string, done chan<- CodexGatewayBorrowVerification) { - template := &http.Request{Header: http.Header{"X-Codex-Window-Id": []string{id}}} - done <- s.probeTarget(ctx, template, account, "gpt-6-astra", "", nil, s.candidate) - } - first, second := make(chan CodexGatewayBorrowVerification, 1), make(chan CodexGatewayBorrowVerification, 1) - go run("first", first) - select { - case <-started: - case <-ctx.Done(): - t.Fatal("first probe did not start") - } - go run("second", second) - queued := false - deadline := time.Now().Add(time.Second) - for !queued && time.Now().Before(deadline) { - sharedPelicanExecution.mu.Lock() - for _, waiter := range sharedPelicanExecution.waiting { - queued = queued || waiter.accountID == account.ID - } - sharedPelicanExecution.mu.Unlock() - if !queued { - time.Sleep(time.Millisecond) - } - } - close(release) - require.True(t, queued, "second probe must wait on the same account") - require.True(t, (<-first).Success) - require.True(t, (<-second).Success) - mu.Lock() - defer mu.Unlock() - require.Equal(t, []string{"first:mint", "first:continue", "second:mint", "second:continue"}, order) - }) - } -} - -func TestCodexBorrowProbeLeaseCancellationDoesNotBlockOtherAccounts(t *testing.T) { - ctx, cancel := context.WithTimeout(context.Background(), time.Second) - defer cancel() - leased, release, err := acquireCodexBorrowObservation(ctx, 92002) - require.NoError(t, err) - defer release() - _, nestedRelease, err := acquireCodexBorrowObservation(leased, 92002) - require.NoError(t, err) - nestedRelease() // A nested shot cannot release the whole pair's lease. - _, releaseOther, err := acquireCodexBorrowObservation(ctx, 92003) - require.NoError(t, err, "another account can use remaining global capacity") - releaseOther() - _, _, err = acquireCodexBorrowObservation(leased, 92003) - require.Error(t, err, "a pair cannot wait for a source while retaining its target lease") - queued, cancelQueued := context.WithCancel(ctx) - done := make(chan error, 1) - go func() { - _, releaseQueued, queuedErr := acquireCodexBorrowObservation(queued, 92002) - if releaseQueued != nil { - releaseQueued() - } - done <- queuedErr - }() - cancelQueued() - require.ErrorIs(t, <-done, context.Canceled) - release() - _, releaseAgain, err := acquireCodexBorrowObservation(ctx, 92002) - require.NoError(t, err, "cancellation and nested releases must not leak account capacity") - releaseAgain() -} diff --git a/backend/internal/service/codex_gateway_borrow_probe_stream_test.go b/backend/internal/service/codex_gateway_borrow_probe_stream_test.go deleted file mode 100644 index 7d69b17fae67..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe_stream_test.go +++ /dev/null @@ -1,164 +0,0 @@ -//go:build unit - -package service - -import ( - "context" - "errors" - "io" - "net/http" - "strings" - "sync" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -type borrowCoreStreamBody struct { - *io.PipeReader - closed chan struct{} - once sync.Once -} - -func (b *borrowCoreStreamBody) Close() error { - b.once.Do(func() { close(b.closed) }) - return b.PipeReader.Close() -} - -func TestCodexGatewayBorrowObservationStopsAtTerminal(t *testing.T) { - for _, action := range []string{"source", "target", "manual", "failed", "done", "no event delimiter", "disconnected", "bounded"} { - t.Run(action, func(t *testing.T) { - var bodies []*borrowCoreStreamBody - var sent []string - calls := 0 - source, target := borrowCoreAccount(1), borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, proxy string, accountID int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - require.Equal(t, "Bearer synthetic-token", req.Header.Get("Authorization")) - require.Equal(t, "synthetic-chat-id", req.Header.Get("ChatGPT-Account-ID")) - require.Empty(t, proxy) - borrowCoreBody(t, req) - model, answer := "gpt-6-astra", "OK" - if accountID == 2 { - model = "gpt-6.1-sol" - } - if action == "manual" { - answer = "鹈鹕骑自行车" - } - stream := ": exact raw comment\r\n" + strings.ReplaceAll(borrowCoreSSE(model, answer), "\n", "\r\n") - if action == "failed" { - stream = "data: {\"type\":\"response.failed\",\"response\":{\"error\":{\"message\":\"full terminal error\"}}}\r\n\r\n" - } - if action == "done" { - stream = strings.ReplaceAll(stream, "response.completed", "response.done") - } - if action == "no event delimiter" { - stream = strings.TrimSuffix(stream, "\r\n") - } - if action == "disconnected" { - stream = "data: {\"type\":\"response.output_text.delta\",\"delta\":\"partial answer\"}\r\n\r\n" - } - if action == "bounded" { - stream = strings.Repeat("x", 65) - } - reader, writer := io.Pipe() - body := &borrowCoreStreamBody{PipeReader: reader, closed: make(chan struct{})} - bodies, sent = append(bodies, body), append(sent, stream) - go func() { - _, _ = io.WriteString(writer, stream) - if action == "target" { - _ = writer.Close() - return - } - if action == "disconnected" { - _ = writer.CloseWithError(io.ErrUnexpectedEOF) - return - } - select { - case <-req.Context().Done(): - _ = writer.CloseWithError(req.Context().Err()) - case <-body.closed: - _ = writer.Close() - } - }() - headers := make(http.Header) - if accountID == 1 { - headers.Set("Set-Cookie", "__oailb=synthetic-borrowed-cookie; Secure; Path=/; Max-Age=230") - } - if action == "target" && calls == 1 { - headers.Set("X-Codex-Turn-State", "minted-state") - } - return &http.Response{StatusCode: 200, Header: headers, Body: body}, nil - }, source, target) - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) - defer cancel() - started := time.Now() - switch action { - case "source": - candidate, _, err := s.acquireSource(ctx, 1) - require.NoError(t, err) - require.NotNil(t, candidate) - require.Equal(t, 1, calls) - case "target": - _, template, proxy, err := s.accountTemplate(ctx, 2, "gpt-6.1-sol") - require.NoError(t, err) - candidate := &codexGatewayBorrowCandidate{cookie: http.Cookie{Name: "__oailb", Value: "synthetic-borrowed-cookie", Secure: true, Path: "/"}, sourceID: 1, expires: time.Now().Add(codexGatewayBorrowTTL)} - result := s.probeTarget(ctx, template, target, "gpt-6.1-sol", proxy, nil, candidate) - require.True(t, result.Success, result.Error) - require.Equal(t, 2, calls) - case "manual": - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, template, _, err := s.accountTemplate(ctx, 2, "gpt-6.1-sol") - require.NoError(t, err) - key := borrowTargetFingerprint(template, target, "gpt-6.1-sol", "", nil, s.candidate.cookie.Value) - s.targets[codexGatewayBorrowTargetKey{2, "gpt-6.1-sol"}] = codexGatewayBorrowTargetCheck{policyRevision: currentCodexFingerprintPolicyForAccount(target).revision, key: key, cookieKey: borrowHash(s.candidate.cookie.Value), expires: s.candidate.expires, result: CodexGatewayBorrowVerification{Success: true}} - result, err := s.GeneratePelican(ctx, 2, "gpt-6.1-sol", "high") - require.NoError(t, err) - require.Equal(t, "complete", result.Status) - require.Equal(t, "鹈鹕骑自行车", result.RawAnswer) - require.Equal(t, sent[0], result.RawResponse) - require.Equal(t, 1, calls) - default: - _, template, _, err := s.accountTemplate(ctx, 1, "gpt-6-astra") - require.NoError(t, err) - limit := int64(codexGatewayBorrowProbeMaxBody) - if action == "bounded" { - limit = 64 - } - shot, err := s.fireObservation(ctx, template.Header, source, "gpt-6-astra", "Reply with OK only.", "medium", "", nil, "", nil, HTTPUpstreamProfileCodexBorrowSource, limit) - require.Equal(t, 1, calls) - if action == "disconnected" { - require.ErrorIs(t, err, io.ErrUnexpectedEOF) - require.False(t, shot.complete) - require.True(t, shot.incomplete) - require.Equal(t, "partial answer", shot.answer) - } else if action == "bounded" { - require.ErrorContains(t, err, "exceeded 64 bytes") - require.False(t, shot.complete) - require.True(t, shot.incomplete) - } else { - require.NoError(t, err) - if action == "failed" { - require.False(t, shot.complete) - require.Contains(t, shot.errorText, "full terminal error") - } else { - require.True(t, shot.complete) - require.Equal(t, "OK", shot.answer) - } - } - require.Equal(t, sent[0], shot.raw) - } - require.Less(t, time.Since(started), 500*time.Millisecond, "authoritative terminal must not wait for HTTP EOF") - for _, body := range bodies { - select { - case <-body.closed: - default: - t.Fatal("response body was not closed before returning") - } - } - require.False(t, errors.Is(ctx.Err(), context.DeadlineExceeded), "the open stream must not force a timeout") - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_probe_terminal.go b/backend/internal/service/codex_gateway_borrow_probe_terminal.go deleted file mode 100644 index 52f46752650e..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe_terminal.go +++ /dev/null @@ -1,100 +0,0 @@ -package service - -// Target-probe terminal contract from ranxi2001/sub2api v2.10.3, -// fd1b5ee4eeb20961fbb783fa6f136a1704271e90, LGPL-3.0. -// The original route probe calls this without an expected-model argument: -// reported labels are display evidence, not the STATE qualification criterion. - -import ( - "bufio" - "bytes" - "encoding/json" - "errors" - "strings" -) - -var ( - errCodexBorrowProbeNoCompletion = errors.New("probe did not complete successfully") - errCodexBorrowProbeUnterminated = errors.New("unterminated probe event") -) - -// A clean EOF or [DONE] is not evidence of a successful Responses generation. -func validateCodexGatewayBorrowPinnedProbeResponse(body []byte) error { - completed := false - inspect := func(data []byte) error { - if bytes.Equal(bytes.TrimSpace(data), []byte("[DONE]")) { - return nil - } - var event struct { - Model string `json:"model"` - Type string `json:"type"` - Status string `json:"status"` - Error json.RawMessage `json:"error"` - Response *struct { - Model string `json:"model"` - Status string `json:"status"` - } `json:"response"` - } - if json.Unmarshal(data, &event) != nil { - return errors.New("invalid probe event") - } - - if event.Type == "error" || event.Type == "response.failed" || event.Type == "response.incomplete" || event.Status == "failed" || event.Status == "incomplete" || (len(event.Error) > 0 && string(event.Error) != "null") { - return errors.New("probe response failed") - } - if event.Response != nil && (event.Response.Status == "failed" || event.Response.Status == "incomplete") { - return errors.New("probe response failed") - } - if event.Type == "response.completed" { - if event.Response == nil || event.Response.Status != "completed" { - return errors.New("invalid probe completion") - } - completed = true - } - if event.Type == "" && event.Status == "completed" { - completed = true - } - return nil - } - trimmed := bytes.TrimSpace(body) - if len(trimmed) > 0 && trimmed[0] == '{' { - if err := inspect(trimmed); err != nil { - return err - } - } else { - scanner := bufio.NewScanner(bytes.NewReader(body)) - scanner.Buffer(make([]byte, 4096), 1<<20) - var data []string - flush := func() error { - if len(data) == 0 { - return nil - } - err := inspect([]byte(strings.Join(data, "\n"))) - data = nil - return err - } - for scanner.Scan() { - line := strings.TrimSuffix(scanner.Text(), "\r") - if line == "" { - if err := flush(); err != nil { - return err - } - continue - } - if strings.HasPrefix(line, "data:") { - data = append(data, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " ")) - } - } - if scanner.Err() != nil { - return errors.New("invalid probe stream") - } - // A trailing unterminated event may be truncated, even if its JSON parses. - if len(data) > 0 { - return errCodexBorrowProbeUnterminated - } - } - if !completed { - return errCodexBorrowProbeNoCompletion - } - return nil -} diff --git a/backend/internal/service/codex_gateway_borrow_probe_wire.go b/backend/internal/service/codex_gateway_borrow_probe_wire.go deleted file mode 100644 index 6d80eb768872..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe_wire.go +++ /dev/null @@ -1,15 +0,0 @@ -package service - -import "context" - -// The pinned source and two-shot STATE protocols send plaintext JSON. They are -// independent of the user's business-request compression setting. Only an -// explicit administrator encoding comparison may opt into the configured codec. -type codexBorrowProbeConfiguredEncodingKey struct{} - -func withCodexBorrowPlainProbe(ctx context.Context) context.Context { - if _, set := ctx.Value(codexBorrowProbeConfiguredEncodingKey{}).(bool); set { - return ctx - } - return context.WithValue(ctx, codexBorrowProbeConfiguredEncodingKey{}, false) -} diff --git a/backend/internal/service/codex_gateway_borrow_probe_wire_test.go b/backend/internal/service/codex_gateway_borrow_probe_wire_test.go deleted file mode 100644 index cd3d3d578dbd..000000000000 --- a/backend/internal/service/codex_gateway_borrow_probe_wire_test.go +++ /dev/null @@ -1,79 +0,0 @@ -package service - -import ( - "context" - "net/http" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestCodexBorrowProbesStayPlainWhileGenerationKeepsCompression(t *testing.T) { - oldCompression := codexRequestZstd.Load() - SetCodexRequestZstdEnabled(true) - defer SetCodexRequestZstdEnabled(oldCompression) - var sources, targets int - s := newBorrowCoreTest(t, func(req *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 { - sources++ - assert.Empty(t, req.Header.Get("Content-Encoding"), "the pinned source protocol sends plain JSON") - return borrowCoreResponse("gpt-6-astra", "OK", "source", "__oailb=source-cookie; Secure; Path=/; Max-Age=230"), nil - } - targets++ - if targets <= 2 { - assert.Empty(t, req.Header.Get("Content-Encoding"), "a body helper must not silently recompress the two-shot probe") - } else { - assert.Equal(t, "zstd", req.Header.Get("Content-Encoding"), "ordinary generation retains the enabled runtime policy") - } - return borrowCoreResponse("gpt-6.1-sol", "OK", "stable-state"), nil - }, borrowCoreAccount(1), borrowCoreAccount(2)) - _, err := s.Verify(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - result, err := s.GeneratePelican(context.Background(), 2, "gpt-6.1-sol", "low") - require.NoError(t, err) - require.Equal(t, "complete", result.Status) - require.Equal(t, 1, sources) - require.Equal(t, 3, targets) - require.True(t, codexRequestZstd.Load(), "diagnostic protocol isolation cannot toggle the global setting") - require.Equal(t, "identity", s.Status().Candidate.SourceRequestEncoding) -} - -func TestCodexBorrowEncodingComparisonChangesOnlyWireEncoding(t *testing.T) { - oldCompression := codexRequestZstd.Load() - SetCodexRequestZstdEnabled(true) - defer SetCodexRequestZstdEnabled(oldCompression) - var encodings []string - var bodies []string - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - encodings = append(encodings, req.Header.Get("Content-Encoding")) - bodies = append(bodies, string(borrowCoreBody(t, req))) - return borrowCoreResponse("gpt-6.1-sol", "OK", "stable"), nil - }, borrowCoreAccount(2)) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - var results []*CodexBorrowDiagnosticResult - var requests int32 - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", Comparison: "encoding", AccountID: 2, Model: "gpt-6.1-sol", Transport: "http", ServiceTier: "priority"}, func(e CodexBorrowDiagnosticEvent) { - requests = e.Requests - if e.Result != nil { - results = append(results, e.Result) - } - }) - require.NoError(t, err) - require.EqualValues(t, 4, requests) - require.Equal(t, []string{"", "", "zstd", "zstd"}, encodings) - for _, body := range bodies { - require.Equal(t, bodies[0], body) - } - require.Len(t, results, 2) - require.Equal(t, results[0].ProbeContext.CookieFingerprint, results[1].ProbeContext.CookieFingerprint) - require.Equal(t, results[0].ProbeContext.PlaintextBodyFingerprint, results[1].ProbeContext.PlaintextBodyFingerprint) - require.Equal(t, "identity", results[0].Verification.MintRequestEncoding) - require.Equal(t, "identity", results[0].Verification.ContinueRequestEncoding) - require.Equal(t, "zstd", results[1].Verification.MintRequestEncoding) - require.Equal(t, "zstd", results[1].Verification.ContinueRequestEncoding) - require.True(t, codexRequestZstd.Load()) - require.Empty(t, s.qualifications) -} diff --git a/backend/internal/service/codex_gateway_borrow_ranxi_generated_test.go b/backend/internal/service/codex_gateway_borrow_ranxi_generated_test.go deleted file mode 100644 index c2b1fd191b15..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ranxi_generated_test.go +++ /dev/null @@ -1,644 +0,0 @@ -// Code generated by .github/scripts/generate_ranxi_borrow_reference.py; DO NOT EDIT. -// ranxi2001/sub2api v2.10.5, LGPL-3.0, commit d977e7a5e24e5536db6ce8b8478743e0c4b11f77. -package service - -import ( - "bufio" - "bytes" - "context" - "encoding/json" - "errors" - "fmt" - "github.com/Wei-Shaw/sub2api/internal/pkg/openai" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/gin-gonic/gin" - "github.com/google/uuid" - "io" - "net/http" - "strings" - "time" -) - -type ranxiRefCodexGatewayRoute struct { - cookie http.Cookie - expires time.Time -} - -func (s *OpenAIGatewayService) ranxiRefGuardOpenAICodexTurnStateEcho(c *gin.Context, account *Account, h http.Header) { - if s == nil || h == nil || account == nil { - return - } - if strings.TrimSpace(h.Get(openAICodexTurnStateHeader)) == "" { - return - } - seed := openAICodexTurnStateSeed(c) - if seed == "" { - return - } - raw, ok := s.openaiCodexTurnStateOrigins.Load(seed) - if !ok { - return - } - origin, ok := raw.(openAICodexTurnStateOrigin) - if !ok { - s.openaiCodexTurnStateOrigins.Delete(seed) - return - } - if !origin.expiresAt.IsZero() && time.Now().After(origin.expiresAt) { - s.openaiCodexTurnStateOrigins.Delete(seed) - return - } - if origin.accountID != account.ID { - h.Del(openAICodexTurnStateHeader) - } -} - -func ranxiRefRunOpenAICodexStateProbe(ctx context.Context, result *ranxiRefOpenAICodexStateProbeResult, seed string, fire func(string, string) (ranxiRefOpenAICodexStateShot, error)) { - mint, err := fire("", seed) - result.MintStatus = mint.status - result.TicketLength = len(mint.state) - if !result.ranxiRefShotUsable(ctx, "打票", mint, err) { - return - } - if mint.state == "" { - result.ranxiRefFail(ranxiRefOpenAICodexStateFailureNoTicket, "打票成功但未返回门票头（x-codex-turn-state），无法进行续接判据", "") - return - } - result.Minted = true - cookies := strings.Join(mint.cookies, "; ") - if seed != "" { - // A borrowed route remains pinned when mint only returns __cflb. - h := http.Header{"Cookie": []string{cookies}} - request := &http.Request{Header: http.Header{"Cookie": []string{seed}}} - if pinned, err := request.Cookie("__oailb"); err == nil { - ranxiRefReplaceCodexWSAnchorCookie(h, pinned.Value) - } - cookies = h.Get("Cookie") - } - cont, err := fire(mint.state, cookies) - result.ContinueStatus = cont.status - result.ContinueTicketLength = len(cont.state) - if !result.ranxiRefShotUsable(ctx, "门票续接", cont, err) { - return - } - result.NewTicket = cont.state != "" && cont.state != mint.state - result.ReportedModel = cont.model - if result.ReportedModel == "" { - result.ReportedModel = mint.model - } - if result.NewTicket { - result.Verdict = ranxiRefOpenAICodexStateDegraded - } else { - result.Verdict = ranxiRefOpenAICodexStateHealthy - } -} - -func ranxiRefProbeOpenAICodexStateRoute(ctx context.Context, upstream HTTPUpstream, template *http.Request, proxy string, accountID int64, concurrency int, profile *tlsfingerprint.Profile) *ranxiRefOpenAICodexStateProbeResult { - result := &ranxiRefOpenAICodexStateProbeResult{AccountID: accountID, Model: ranxiRefOpenAICodexStateProbeDefaultModel, Verdict: ranxiRefOpenAICodexStateInconclusive, StartedAt: time.Now()} - defer func() { - result.FinishedAt = time.Now() - result.LatencyMs = result.FinishedAt.Sub(result.StartedAt).Milliseconds() - if result.Reason == "" { - result.Reason = ranxiRefOpenAICodexStateVerdictReason(result.Verdict) - } - }() - if upstream == nil || template == nil { - result.ranxiRefFail(ranxiRefOpenAICodexStateFailureUnsupported, "缺少网关探针请求", "") - return result - } - pinned, err := template.Cookie("__oailb") - if err != nil || pinned.Value == "" { - result.ranxiRefFail("route_changed", "缺少借用路由", "") - return result - } - routeChanged := false - ranxiRefRunOpenAICodexStateProbe(ctx, result, pinned.String(), func(state, cookie string) (ranxiRefOpenAICodexStateShot, error) { - return ranxiRefFireOpenAICodexStateShotRequest(ctx, template.Header, result.Model, state, cookie, func(req *http.Request) (*http.Response, error) { - req = req.WithContext(WithHTTPUpstreamRedirectsDisabled(req.Context())) - resp, err := upstream.DoWithTLS(req, proxy, accountID, concurrency, profile) - if resp != nil { - for _, got := range resp.Cookies() { - if got.Name == "__oailb" && (got.Value != pinned.Value || got.MaxAge < 0 || (!got.Expires.IsZero() && !time.Now().Before(got.Expires))) { - routeChanged = true - } - } - } - if routeChanged { - if resp != nil && resp.Body != nil { - _ = resp.Body.Close() - } - return nil, errors.New("target_route_changed") - } - return resp, err - }) - }) - if routeChanged { - result.ranxiRefFail("route_changed", "探针期间网关路由已改变", "") - } - return result -} - -func (r *ranxiRefOpenAICodexStateProbeResult) ranxiRefFail(kind, reason, detail string) { - r.Verdict = ranxiRefOpenAICodexStateInconclusive - r.Failure = kind - r.Reason = reason - r.Detail = detail -} - -func (r *ranxiRefOpenAICodexStateProbeResult) ranxiRefShotUsable(ctx context.Context, step string, shot ranxiRefOpenAICodexStateShot, err error) bool { - if err != nil { - if ctx.Err() != nil { - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureCancelled, step+"请求被取消或超时", "") - return false - } - if errors.Is(err, context.DeadlineExceeded) { - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureNetworkError, fmt.Sprintf("%s请求超时（%d 秒内没有完成）", step, int(ranxiRefOpenAICodexStateProbeShotTimeout/time.Second)), "") - return false - } - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureNetworkError, step+"请求失败（网络或代理异常）", SanitizeUpstreamErrorMessage(truncateString(err.Error(), ranxiRefOpenAICodexStateProbeMaxDetailByte))) - return false - } - switch { - case shot.status == http.StatusOK: - if shot.streamErr != nil { - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureStreamError, step+"返回 200 但回复流没有正常结束（上游过载或额度用尽等），本次无法判断", shot.detail) - return false - } - return true - case shot.status == http.StatusUnauthorized || shot.status == http.StatusForbidden: - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureAccountError, fmt.Sprintf("%s被上游拒绝（HTTP %d），账号令牌可能失效或被封", step, shot.status), shot.detail) - case shot.status == http.StatusTooManyRequests: - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureRateLimited, step+"被上游限流（HTTP 429），请稍后再试", shot.detail) - case shot.status == http.StatusBadRequest && strings.Contains(strings.ToLower(shot.detail), "not supported"): - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureModelUnsupported, "该账号的套餐不支持这个模型（HTTP 400），请换一个模型再测", shot.detail) - default: - r.ranxiRefFail(ranxiRefOpenAICodexStateFailureUpstreamError, fmt.Sprintf("%s返回异常状态（HTTP %d）", step, shot.status), shot.detail) - } - return false -} - -func ranxiRefOpenAICodexStateVerdictReason(verdict ranxiRefOpenAICodexStateVerdict) string { - switch verdict { - case ranxiRefOpenAICodexStateDegraded: - return "带原门票续接时上游回了新的门票（x-codex-turn-state），符合降智特征：账号很可能已被降到 gpt-5.6-luna 档，建议尽快换号。" - case ranxiRefOpenAICodexStateHealthy: - return "带原门票续接时上游没有回新门票，门票延续正常：账号仍处于满血（gpt-6-astra）状态。" - default: - return "未取得有效的门票续接探测结果，无法判断账号是否降智。" - } -} - -func ranxiRefFireOpenAICodexStateShotRequest(ctx context.Context, headers http.Header, model, turnState, cookie string, send func(*http.Request) (*http.Response, error)) (ranxiRefOpenAICodexStateShot, error) { - var out ranxiRefOpenAICodexStateShot - ctx = context.WithValue(ctx, ranxiRefOpenAICodexStateProbeContextKey{}, true) - shotCtx, cancel := context.WithTimeout(ctx, ranxiRefOpenAICodexStateProbeShotTimeout) - defer cancel() - body := []byte(`{"model":` + ranxiRefJsonString(model) + `,"instructions":"Reply with OK.","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Reply with OK."}]}],"stream":true,"store":false,"parallel_tool_calls":true,"include":["reasoning.encrypted_content"]}`) - req, err := http.NewRequestWithContext(shotCtx, http.MethodPost, chatgptCodexURL, bytes.NewReader(body)) - if err != nil { - return out, err - } - req = req.WithContext(WithHTTPUpstreamProfile(req.Context(), HTTPUpstreamProfileOpenAIHarvest)) - req.Close = true - req.Host = "chatgpt.com" - req.Header = headers.Clone() - req.Header.Set("Accept", "text/event-stream") - req.Header.Set("Content-Type", "application/json") - req.Header.Set("OpenAI-Beta", "responses=experimental") - req.Header.Set("session_id", uuid.NewString()) - // Never carry a business turn, request size, or compressed/lite body into a probe. - for _, name := range []string{responsesLiteHeaderKey, openAICodexTurnStateHeader, "Cookie", "Content-Length", "Content-Encoding", "conversation_id", "x-codex-turn-metadata"} { - req.Header.Del(name) - } - if turnState = strings.TrimSpace(turnState); turnState != "" { - req.Header.Set(openAICodexTurnStateHeader, turnState) - } - if cookie = strings.TrimSpace(cookie); cookie != "" { - req.Header.Set("Cookie", cookie) - } - - resp, err := send(req) - if err != nil { - if resp != nil && resp.Body != nil { - _ = resp.Body.Close() - } - return out, err - } - if resp == nil { - return out, errors.New("nil upstream response") - } - out.status = resp.StatusCode - out.state = strings.TrimSpace(extractOpenAICodexTurnState(resp.Header)) - out.cookies = ranxiRefOpenAICodexStateRouteCookies(resp) - if resp.Body == nil { - out.streamErr = errors.New("probe response body missing") - return out, nil - } - defer func() { _ = resp.Body.Close() }() - data, err := io.ReadAll(io.LimitReader(resp.Body, ranxiRefOpenAICodexStateProbeMaxBody+1)) - if err != nil { - if shotCtx.Err() != nil { - return out, shotCtx.Err() - } - out.streamErr = errors.New("probe response incomplete") - return out, nil - } - if len(data) > ranxiRefOpenAICodexStateProbeMaxBody { - out.streamErr = errors.New("probe response too large") - return out, nil - } - if out.status != http.StatusOK { - out.detail = ranxiRefOpenAICodexStateDetail(data) - return out, nil - } - out.model = ranxiRefOpenAICodexStateStreamModel(data) - if out.streamErr = ranxiRefValidateCodexProbeResponse(data); out.streamErr != nil { - out.detail = ranxiRefOpenAICodexStateDetail(ranxiRefOpenAICodexStateStreamErrorPayload(data)) - } - return out, nil -} - -func ranxiRefOpenAICodexStateRouteCookies(resp *http.Response) []string { - var out []string - seen := map[string]bool{} - for _, cookie := range resp.Cookies() { - if cookie.Name != "__cflb" && cookie.Name != "__oailb" { - continue - } - if seen[cookie.Name] || cookie.Value == "" || cookie.MaxAge < 0 { - continue - } - seen[cookie.Name] = true - out = append(out, cookie.Name+"="+cookie.Value) - } - return out -} - -func ranxiRefOpenAICodexStateDetail(body []byte) string { - msg := strings.TrimSpace(extractUpstreamErrorMessage(body)) - if msg == "" { - msg = strings.TrimSpace(string(body)) - } - return SanitizeUpstreamErrorMessage(truncateString(msg, ranxiRefOpenAICodexStateProbeMaxDetailByte)) -} - -func ranxiRefOpenAICodexStateStreamEvents(data []byte) [][]byte { - var events [][]byte - scanner := bufio.NewScanner(bytes.NewReader(data)) - scanner.Buffer(make([]byte, 4096), ranxiRefOpenAICodexStateProbeMaxBody) - var lines []string - flush := func() { - if len(lines) == 0 { - return - } - payload := strings.TrimSpace(strings.Join(lines, "\n")) - lines = nil - if payload != "" && payload != "[DONE]" { - events = append(events, []byte(payload)) - } - } - for scanner.Scan() { - line := strings.TrimSuffix(scanner.Text(), "\r") - if line == "" { - flush() - continue - } - if strings.HasPrefix(line, "data:") { - lines = append(lines, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " ")) - } - } - flush() - return events -} - -func ranxiRefOpenAICodexStateStreamModel(data []byte) string { - completed := "" - for _, payload := range ranxiRefOpenAICodexStateStreamEvents(data) { - var event struct { - Type string `json:"type"` - Response struct { - Model string `json:"model"` - } `json:"response"` - } - if json.Unmarshal(payload, &event) != nil { - continue - } - model := strings.TrimSpace(event.Response.Model) - if model == "" { - continue - } - switch event.Type { - case "response.created": - return model - case "response.completed": - completed = model - } - } - return completed -} - -func ranxiRefOpenAICodexStateStreamErrorPayload(data []byte) []byte { - for _, payload := range ranxiRefOpenAICodexStateStreamEvents(data) { - var event struct { - Type string `json:"type"` - } - if json.Unmarshal(payload, &event) != nil { - continue - } - switch event.Type { - case "error", "response.failed", "response.incomplete": - var wrapped struct { - Response struct { - Error json.RawMessage `json:"error"` - } `json:"response"` - } - if json.Unmarshal(payload, &wrapped) == nil && len(wrapped.Response.Error) > 0 && string(wrapped.Response.Error) != "null" { - return []byte(`{"error":` + string(wrapped.Response.Error) + `}`) - } - return payload - } - } - return nil -} - -type ranxiRefOpenAICodexStateVerdict string - -type ranxiRefOpenAICodexStateProbeResult struct { - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Verdict ranxiRefOpenAICodexStateVerdict `json:"verdict"` - Reason string `json:"reason"` - Failure string `json:"failure,omitempty"` - // Detail 是脱敏截断后的上游报错原文，便于看出「模型不支持」之类的具体原因。 - Detail string `json:"detail,omitempty"` - - MintStatus int `json:"mint_status"` - ContinueStatus int `json:"continue_status"` - Minted bool `json:"minted"` - NewTicket bool `json:"new_ticket"` - // TicketLength 是第一发打到的门票长度；判据只在 780 长度的完整请求上验证过。 - TicketLength int `json:"ticket_length"` - ContinueTicketLength int `json:"continue_ticket_length"` - ReportedModel string `json:"reported_model,omitempty"` - - LatencyMs int64 `json:"latency_ms"` - StartedAt time.Time `json:"started_at"` - FinishedAt time.Time `json:"finished_at"` -} - -type ranxiRefOpenAICodexStateShot struct { - status int - state string - cookies []string - model string - // streamErr 非空表示 200 但回复流没有完整跑完（上游过载、额度用尽等流内失败）。 - streamErr error - detail string -} - -type ranxiRefOpenAICodexStateProbeContextKey struct{} - -const ( - // ranxiRefOpenAICodexStateHealthy：带原门票续接时上游没回新门票，账号仍是满血（astra）。 - ranxiRefOpenAICodexStateHealthy ranxiRefOpenAICodexStateVerdict = "healthy" - // ranxiRefOpenAICodexStateDegraded：带原门票续接时上游回了新门票，账号大概率已被降到 luna 档。 - ranxiRefOpenAICodexStateDegraded ranxiRefOpenAICodexStateVerdict = "degraded" - // ranxiRefOpenAICodexStateInconclusive：两发没有都干净跑通，本次无法判断。 - ranxiRefOpenAICodexStateInconclusive ranxiRefOpenAICodexStateVerdict = "inconclusive" -) - -const ( - ranxiRefOpenAICodexStateFailureUnsupported = "unsupported" - ranxiRefOpenAICodexStateFailureAccountError = "account_error" - ranxiRefOpenAICodexStateFailureRateLimited = "rate_limited" - ranxiRefOpenAICodexStateFailureModelUnsupported = "model_unsupported" - ranxiRefOpenAICodexStateFailureUpstreamError = "upstream_error" - ranxiRefOpenAICodexStateFailureNetworkError = "network_error" - ranxiRefOpenAICodexStateFailureStreamError = "stream_error" - ranxiRefOpenAICodexStateFailureNoTicket = "no_ticket" - ranxiRefOpenAICodexStateFailureCancelled = "cancelled" -) - -const ( - ranxiRefOpenAICodexStateProbeDefaultModel = "gpt-6-astra" - - ranxiRefOpenAICodexStateProbeShotTimeout = 45 * time.Second - ranxiRefOpenAICodexStateProbeMaxBody = 1 << 20 - ranxiRefOpenAICodexStateProbeMaxDetailByte = 300 -) - -func ranxiRefValidateCodexProbeResponse(body []byte, expectedModel ...string) error { - completed := false - inspect := func(data []byte) error { - if bytes.Equal(bytes.TrimSpace(data), []byte("[DONE]")) { - return nil - } - var event struct { - Model string `json:"model"` - Type string `json:"type"` - Status string `json:"status"` - Error json.RawMessage `json:"error"` - Response *struct { - Model string `json:"model"` - Status string `json:"status"` - } `json:"response"` - } - if json.Unmarshal(data, &event) != nil { - return errors.New("invalid probe event") - } - if len(expectedModel) > 0 { - models := []string{event.Model} - if event.Response != nil { - models = append(models, event.Response.Model) - } - for _, served := range models { - if served != "" && ranxiRefNormalizeOpenAICodexTicketModel(served) != ranxiRefNormalizeOpenAICodexTicketModel(expectedModel[0]) { - return errors.New("probe response model mismatch") - } - } - } - if event.Type == "error" || event.Type == "response.failed" || event.Type == "response.incomplete" || event.Status == "failed" || event.Status == "incomplete" || (len(event.Error) > 0 && string(event.Error) != "null") { - return errors.New("probe response failed") - } - if event.Response != nil && (event.Response.Status == "failed" || event.Response.Status == "incomplete") { - return errors.New("probe response failed") - } - if event.Type == "response.completed" { - if event.Response == nil || event.Response.Status != "completed" { - return errors.New("invalid probe completion") - } - completed = true - } - if event.Type == "" && event.Status == "completed" { - completed = true - } - return nil - } - trimmed := bytes.TrimSpace(body) - if len(trimmed) > 0 && trimmed[0] == '{' { - if err := inspect(trimmed); err != nil { - return err - } - } else { - scanner := bufio.NewScanner(bytes.NewReader(body)) - scanner.Buffer(make([]byte, 4096), 1<<20) - var data []string - flush := func() error { - if len(data) == 0 { - return nil - } - err := inspect([]byte(strings.Join(data, "\n"))) - data = nil - return err - } - for scanner.Scan() { - line := strings.TrimSuffix(scanner.Text(), "\r") - if line == "" { - if err := flush(); err != nil { - return err - } - continue - } - if strings.HasPrefix(line, "data:") { - data = append(data, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " ")) - } - } - if scanner.Err() != nil { - return errors.New("invalid probe stream") - } - // A trailing unterminated event may be truncated, even if its JSON parses. - if len(data) > 0 { - return errors.New("unterminated probe event") - } - } - if !completed { - return errors.New("probe did not complete successfully") - } - return nil -} - -func ranxiRefReplaceCodexWSAnchorCookie(h http.Header, value string) { - var keep []string - for _, raw := range h.Values("Cookie") { - for _, part := range strings.Split(raw, ";") { - part = strings.TrimSpace(part) - name, _, _ := strings.Cut(part, "=") - if part != "" && strings.TrimSpace(name) != "__oailb" { - keep = append(keep, part) - } - } - } - keep = append(keep, (&http.Cookie{Name: "__oailb", Value: value}).String()) - h.Set("Cookie", strings.Join(keep, "; ")) -} - -func ranxiRefJsonString(v string) string { - b, err := json.Marshal(v) - if err != nil { - return `""` - } - return string(b) -} - -func ranxiRefNormalizeOpenAICodexTicketModel(model string) string { - return strings.TrimSpace(model) -} - -func ranxiRefCodexGatewayCookiePathMatches(path, scope string) bool { - return path == scope || (strings.HasPrefix(path, scope) && (strings.HasSuffix(scope, "/") || strings.HasPrefix(strings.TrimPrefix(path, scope), "/"))) -} - -func ranxiRefCodexGatewayRouteFromResponse(resp *http.Response, path string, now time.Time, configuredTTL ...int) *ranxiRefCodexGatewayRoute { - ttl := 230 - if len(configuredTTL) > 0 && configuredTTL[0] >= 30 && configuredTTL[0] <= 240 { - ttl = configuredTTL[0] - } - var route *ranxiRefCodexGatewayRoute - for _, cookie := range resp.Cookies() { - if cookie.Name != "__oailb" || !cookie.Secure || (cookie.Domain != "" && strings.TrimPrefix(cookie.Domain, ".") != "chatgpt.com") { - continue - } - if cookie.Path == "" { - cookie.Path = path[:strings.LastIndex(path, "/")] - } - if !strings.HasPrefix(cookie.Path, "/") || !ranxiRefCodexGatewayCookiePathMatches(path, cookie.Path) { - continue - } - // A deletion sentinel is applied even if the successful response has no new route. - if cookie.MaxAge < 0 || cookie.Value == "" || (cookie.MaxAge == 0 && !cookie.Expires.IsZero() && !now.Before(cookie.Expires)) { - return &ranxiRefCodexGatewayRoute{} - } - expires := now.Add(time.Duration(ttl) * time.Second) - if cookie.MaxAge > 0 && cookie.MaxAge < ttl { - expires = now.Add(time.Duration(cookie.MaxAge) * time.Second) - } else if cookie.MaxAge == 0 { - if cookie.Expires.IsZero() { - continue - } - if cookie.Expires.Before(expires) { - expires = cookie.Expires - } - } - route = &ranxiRefCodexGatewayRoute{cookie: *cookie, expires: expires} - } - return route -} - -func ranxiRefCreateOpenAITestPayload(modelID string, isOAuth bool) map[string]any { - payload := map[string]any{ - "model": modelID, - "input": []map[string]any{ - { - "role": "user", - "content": []map[string]any{ - { - "type": "input_text", - "text": "hi", - }, - }, - }, - }, - "stream": true, - } - - // OAuth accounts using ChatGPT internal API require store: false - if isOAuth { - payload["store"] = false - } - - // All accounts require instructions for Responses API - payload["instructions"] = openai.DefaultInstructions - - return payload -} - -func ranxiRefCreatePelicanOpenAIPayload(modelID string, isOAuth bool, prompt, reasoningEffort string) map[string]any { - payload := ranxiRefCreateOpenAITestPayload(modelID, isOAuth) - input, ok := payload["input"].([]map[string]any) - if ok && len(input) > 0 { - content, ok := input[0]["content"].([]map[string]any) - if ok && len(content) > 0 { - content[0]["text"] = ranxiRefPromptOrDefault(prompt) - } - } - if effort := ranxiRefNormalizePelicanReasoningEffort(reasoningEffort); effort != "" { - payload["reasoning"] = map[string]any{"effort": effort} - } - return payload -} - -func ranxiRefPromptOrDefault(prompt string) string { - if value := strings.TrimSpace(prompt); value != "" { - return value - } - return "hi" -} - -func ranxiRefNormalizePelicanReasoningEffort(value string) string { - switch strings.ToLower(strings.TrimSpace(value)) { - case "none", "minimal", "low", "medium", "high", "xhigh", "max", "ultra": - return strings.ToLower(strings.TrimSpace(value)) - default: - return "" - } -} diff --git a/backend/internal/service/codex_gateway_borrow_ranxi_reference_test.go b/backend/internal/service/codex_gateway_borrow_ranxi_reference_test.go deleted file mode 100644 index 18846e6b24c4..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ranxi_reference_test.go +++ /dev/null @@ -1,141 +0,0 @@ -package service - -import ( - "context" - "encoding/json" - "io" - "net/http" - "strings" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/google/uuid" - "github.com/stretchr/testify/require" -) - -// Both implementations consume identical responses. The reference functions -// are extracted from the pinned Git commit, not restated as expected results. -func TestCodexGatewayBorrowRanxiReference(t *testing.T) { - t.Run("business STATE provenance", func(t *testing.T) { - for _, tc := range []struct { - name string - owner int64 - ttl time.Duration - }{ - {"unknown", 0, 0}, {"same account", 2, time.Minute}, {"foreign account", 1, time.Minute}, {"expired foreign record", 1, -time.Minute}, - } { - t.Run(tc.name, func(t *testing.T) { - gateway := &OpenAIGatewayService{} - c, _ := newTurnStateTestContext(t, 7, "native-session") - if tc.owner != 0 { - gateway.openaiCodexTurnStateOrigins.Store(openAICodexTurnStateSeed(c), openAICodexTurnStateOrigin{accountID: tc.owner, expiresAt: time.Now().Add(tc.ttl)}) - } - h := http.Header{"X-Codex-Turn-State": {"business-state"}} - original := h.Clone() - gateway.ranxiRefGuardOpenAICodexTurnStateEcho(c, &Account{ID: 2}, original) - gateway.guardOpenAICodexTurnStateEcho(c, &Account{ID: 2}, h) - require.Equal(t, original, h) - }) - } - }) - t.Run("source payload", func(t *testing.T) { - req, err := borrowSourceObservationRequest(context.Background(), http.Header{}) - require.NoError(t, err) - original := ranxiRefCreatePelicanOpenAIPayload("gpt-6-astra", true, "Reply with OK only.", "medium") - want, err := json.Marshal(original) - require.NoError(t, err) - require.Equal(t, string(want), string(borrowCoreBody(t, req))) - }) - for _, tc := range []struct { - name, first, second, firstBody, secondBody, route string - status int - }{ - {name: "same STATE", first: "mint", second: "mint"}, - {name: "omitted second STATE", first: "mint"}, - {name: "new second STATE", first: "mint", second: "replacement"}, - {name: "missing first STATE"}, - {name: "route replaced", first: "mint", route: "__oailb=replaced; Path=/; Secure"}, - {name: "route deleted", first: "mint", route: "__oailb=; Max-Age=0; Path=/; Secure"}, - {name: "stream failed", first: "mint", secondBody: "data: {\"type\":\"response.failed\",\"response\":{\"status\":\"failed\"}}\n\n"}, - {name: "unterminated completion", first: "mint", secondBody: strings.TrimSuffix(borrowCoreSSE("gpt-6-astra", "OK"), "\n\n")}, - {name: "HTTP failure", first: "mint", status: 503}, - } { - t.Run(tc.name, func(t *testing.T) { - type sent struct { - Body string - Header http.Header - Close bool - Profile HTTPUpstreamProfile - } - runs := make([][]sent, 2) - sender := func(run int) borrowCoreProbe { - return func(req *http.Request, proxy string, id int64, n int, p *tlsfingerprint.Profile) (*http.Response, error) { - require.Equal(t, "synthetic-exit", proxy) - require.Equal(t, int64(2), id) - header := req.Header.Clone() - _, err := uuid.Parse(header.Get("session_id")) - require.NoError(t, err) - header.Del("session_id") // Each original shot deliberately generates a UUID. - runs[run] = append(runs[run], sent{string(borrowCoreBody(t, req)), header, req.Close, HTTPUpstreamProfileFromContext(req.Context())}) - state, body := tc.first, tc.firstBody - if len(runs[run]) == 2 { - state, body = tc.second, tc.secondBody - } - resp := borrowCoreResponse("gpt-6-astra", "OK", state, "__cflb=target-affinity; Path=/; Secure") - if body != "" { - resp.Body = io.NopCloser(strings.NewReader(body)) - } - if tc.route != "" { - resp.Header.Add("Set-Cookie", tc.route) - } - if tc.status != 0 { - resp.StatusCode = tc.status - } - return resp, nil - } - } - headers := http.Header{} - for k, v := range map[string]string{"Authorization": "Bearer target-only", "ChatGPT-Account-ID": "target-only", "User-Agent": "current-client", "Originator": "Codex Desktop", "Version": "0.162.0-alpha.17.2", "session-id": "native-session", "thread-id": "native-thread", "x-codex-window-id": "native-window", "X-Codex-Turn-State": "business-state", "X-Codex-Turn-Metadata": "business-metadata", "Cookie": "__oailb=borrowed; business=untouched", "Content-Encoding": "zstd"} { - headers.Set(k, v) - } - originalHeaders := headers.Clone() - req, _ := http.NewRequest(http.MethodPost, chatgptCodexURL, nil) - req.Header = headers - reference := ranxiRefProbeOpenAICodexStateRoute(context.Background(), sender(0), req, "synthetic-exit", 2, 4, nil) - service := newBorrowCoreTest(t, sender(1), borrowCoreAccount(1), borrowCoreAccount(2)) - candidate := &codexGatewayBorrowCandidate{sourceID: 1, cookie: http.Cookie{Name: "__oailb", Value: "borrowed"}, expires: time.Now().Add(time.Minute)} - local := service.probeTarget(context.Background(), req, borrowCoreAccount(2), "gpt-6-astra", "synthetic-exit", nil, candidate) - require.Equal(t, reference.Verdict == ranxiRefOpenAICodexStateHealthy, local.Success) - require.Equal(t, reference.NewTicket, local.NewTicket) - require.Equal(t, runs[0], runs[1], "both implementations must send the same two-shot protocol") - require.Equal(t, originalHeaders, req.Header, "probes must not overwrite business history or cookies") - }) - } - t.Run("cookie scope and lifetime", func(t *testing.T) { - now := time.Date(2026, 10, 11, 0, 0, 0, 0, time.UTC) - for _, cookies := range [][]string{ - {"__oailb=route; Secure; Path=/; Max-Age=600"}, - {"__oailb=route; Secure; Path=/; Max-Age=30"}, - {"__oailb=session; Secure; Path=/"}, - {"__oailb=route; Secure; Path=/other; Max-Age=60"}, - {"__oailb=route; Secure; Domain=other.example; Max-Age=60"}, - {"__oailb=route; Path=/; Max-Age=60"}, - {"__oailb=route; Secure; Path=/; Max-Age=60", "__oailb=; Secure; Path=/; Max-Age=0"}, - } { - h := http.Header{} - for _, c := range cookies { - h.Add("Set-Cookie", c) - } - response := &http.Response{Header: h} - ref := ranxiRefCodexGatewayRouteFromResponse(response, "/backend-api/codex/responses", now) - local := borrowCandidateFromCookies(response.Cookies(), "/backend-api/codex/responses", now) - usable := ref != nil && ref.cookie.Value != "" && now.Before(ref.expires) - require.Equal(t, usable, local != nil, cookies) - if usable { - require.Equal(t, ref.cookie, local.cookie) - require.Equal(t, ref.expires, local.expires) - } - } - }) -} diff --git a/backend/internal/service/codex_gateway_borrow_recovery.go b/backend/internal/service/codex_gateway_borrow_recovery.go deleted file mode 100644 index 5d812a63ec66..000000000000 --- a/backend/internal/service/codex_gateway_borrow_recovery.go +++ /dev/null @@ -1,302 +0,0 @@ -package service - -import ( - "context" - "errors" - "log" - "net/http" - "time" -) - -// Only request-owned recovery can bypass preparation cooldowns. Administrators -// keep ordinary rules; dispatch still requires a matching successful proof. -type borrowImmediateReborrowKey struct{} -type borrowFreshSourceKey struct{} - -func borrowImmediateReborrow(ctx context.Context) bool { - value, _ := ctx.Value(borrowImmediateReborrowKey{}).(bool) - return value -} - -// Request-owned recovery never writes downstream headers or starts a business -// generation. Every shared preparation remains bounded; only waiting is open-ended. -type CodexGatewayBorrowRecovery struct { - Phase string `json:"phase"` - StartedAt time.Time `json:"started_at"` - NextAttemptAt *time.Time `json:"next_attempt_at,omitempty"` - Rounds uint64 `json:"rounds"` - SourceRequests uint64 `json:"source_requests"` - ProbeRequests uint64 `json:"probe_requests"` - SharedPreparation bool `json:"shared_preparation,omitempty"` - LastStage string `json:"last_stage,omitempty"` - LastReason string `json:"last_reason,omitempty"` - LastError string `json:"last_error,omitempty"` -} - -func (t *codexBorrowUsageTracker) recovering(phase string, next *time.Time, err error) { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - if t.finished || t.row.Dispatched { - return - } - if t.row.Recovery == nil && phase != "searching" { - return - } - r := CodexGatewayBorrowRecovery{StartedAt: time.Now()} - if t.row.Recovery != nil { - r = *t.row.Recovery - } - r.Phase, r.NextAttemptAt = phase, next - if phase == "searching" { - r.Rounds++ - } - if err != nil { - r.LastStage, r.LastReason = codexBorrowFailureStage(err) - r.LastError = err.Error() - } - t.row.Recovery = &r - t.publish(0, 0, 0) - log.Printf("[codex_borrow_recovery] request_id=%s account_id=%d model=%s phase=%s round=%d reason=%s next_attempt_at=%v source_requests=%d probe_requests=%d", t.row.GatewayRequestID, t.key.account, t.key.model, phase, r.Rounds, r.LastReason, next, r.SourceRequests, r.ProbeRequests) -} - -func (t *codexBorrowUsageTracker) recoveryObservation(source bool, attempts int) { - if t == nil || attempts == 0 { - return - } - t.mu.Lock() - defer t.mu.Unlock() - if t.row.Recovery == nil { - return - } - r := *t.row.Recovery - if source { - r.SourceRequests += uint64(attempts) - } else { - r.ProbeRequests += uint64(attempts) - } - t.row.Recovery = &r - t.publish(0, 0, 0) - log.Printf("[codex_borrow_recovery_observation] request_id=%s account_id=%d model=%s source=%t attempts=%d", t.row.GatewayRequestID, t.key.account, t.key.model, source, attempts) -} - -func (t *codexBorrowUsageTracker) sharedRecoveryPreparation() { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - if t.finished || t.row.Recovery == nil { - return - } - r := *t.row.Recovery - r.SharedPreparation = true - t.row.Recovery = &r - t.publish(0, 0, 0) -} - -func borrowRecoveryRetryable(err error) bool { - if errors.Is(err, ErrCodexGatewayBorrowChanged) || errors.Is(err, context.Canceled) { - return false - } - var failure *CodexGatewayBorrowFailure - if errors.As(err, &failure) { - if v := failure.Verification; v != nil && (v.Reason == "account_unavailable" || v.MintStatus == 401 || v.ContinueStatus == 401) { - return false - } - switch failure.Reason { - case "account_unavailable", "account_inactive", "model_not_allowed", "model_mapping_mismatch", "configuration_changed", "node_proxy_repository_unavailable", "node_candidates_unavailable": - return false - } - return true - } - return errors.Is(err, ErrCodexGatewayBorrowUnavailable) || errors.Is(err, context.DeadlineExceeded) -} - -// Compute the next actual eligible node, rather than spinning every 15 seconds -// through an hour-long rejection. Source/node and target/gateway quiet windows -// are combined; a global source-search pause is a lower bound for all nodes. -func (s *CodexGatewayBorrowService) nextBorrowRecoveryAttempt(ctx context.Context, accountID int64, model string, err error) time.Time { - now := time.Now() - if borrowImmediateReborrow(ctx) { - return now - } - fallback := now.Add(codexGatewayBorrowFailureWait) - cfg := s.ConfigSnapshot() - nodes, loadErr := s.borrowNodes(ctx, cfg.NodeProxyIDs) - if loadErr != nil || len(nodes) == 0 { - return fallback - } - cache := s.rotation.cache(model) - s.rotation.mu.Lock() - global := s.rotation.retry[codexGatewayBorrowTargetKey{accountID, model}] - s.rotation.mu.Unlock() - if until := cache.sourceSearchPausedUntil(accountID, now); !cfg.WaitForQualifiedRoute && until.After(global) { - global = until - } - var earliest time.Time - for _, node := range nodes { - until := cache.blockedUntil(accountID, node.identity, "", now) - if sourceUntil := s.rotation.sourceCache.sourcesBlockedUntil(cfg.SourceAccountIDs, node.identity, now); sourceUntil.After(until) { - until = sourceUntil - } - if global.After(until) { - until = global - } - if !now.Before(until) { - return fallback - } - if earliest.IsZero() || until.Before(earliest) { - earliest = until - } - } - if earliest.IsZero() { - return fallback - } - return earliest -} - -func (s *CodexGatewayBorrowService) recoveryAccount(ctx context.Context, id int64, model string) (*Account, *http.Request, error) { - a, template, _, err := s.accountTemplate(ctx, id, model) - if err != nil { - return nil, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "recovery", Reason: "account_unavailable"} - } - return a, template, nil -} - -func (s *CodexGatewayBorrowService) waitBorrowRecovery(ctx context.Context, next time.Time, accountID int64) error { - timer := time.NewTimer(time.Until(next)) - defer timer.Stop() - ticker := time.NewTicker(5 * time.Second) - defer ticker.Stop() - for { - select { - case <-ctx.Done(): - return ctx.Err() - case <-timer.C: - return nil - case <-ticker.C: - a, err := s.accounts.GetByID(ctx, accountID) - if err != nil || !codexGatewayBorrowAccountSupported(a) || a.Status != StatusActive { - return &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"} - } - } - } -} - -func (s *CodexGatewayBorrowService) recoverQualifiedHTTP(ctx context.Context, req *http.Request, account *Account, model, proxy string, revision uint64, revisionCtx context.Context) (*http.Request, *CodexGatewayBorrowApplication, error) { - caller := ctx - ctx = codexGatewayBorrowRecoveryContext{Context: ctx, values: req.Context()} - ctx, cancel := context.WithCancelCause(ctx) - if s.ConfigSnapshot().ImmediateReborrowOnFailure { - ctx = context.WithValue(ctx, borrowImmediateReborrowKey{}, true) - guardCtx := ctx - go func() { - ticker := time.NewTicker(5 * time.Second) - defer ticker.Stop() - for { - select { - case <-guardCtx.Done(): - return - case <-ticker.C: - checkCtx, stop := context.WithTimeout(guardCtx, codexGatewayBorrowStoreTimeout) - current, err := s.accounts.GetByID(checkCtx, account.ID) - stop() - if err != nil || !codexGatewayBorrowAccountSupported(current) || current.Status != StatusActive || (account.Schedulable && !current.Schedulable) { - cancel(&CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"}) - return - } - } - } - }() - } - policy := currentCodexFingerprintPolicyForAccount(account).revision - ctx, cancelPolicy := borrowPolicyContext(ctx, account.ID, policy) - defer cancelPolicy() - changed := func() bool { - return revisionCtx.Err() != nil || currentCodexFingerprintPolicyForAccount(account).revision != policy - } - stopRevision := context.AfterFunc(revisionCtx, func() { cancel(ErrCodexGatewayBorrowChanged) }) - defer stopRevision() - defer cancel(nil) - tracker := codexBorrowUsageFromContext(ctx) - originalAccount := req.Header.Get("ChatGPT-Account-ID") - wire := req.Clone(ctx) // Shares Body/GetBody; the pending generation is never read or replayed. - for { - if err := ctx.Err(); err != nil { - if changed() { - return nil, nil, ErrCodexGatewayBorrowChanged - } - return nil, nil, context.Cause(ctx) - } - s.mu.Lock() - current := s.revision == revision && s.config.Enabled && s.config.WaitForQualifiedRoute && s.config.RotateNodes - s.mu.Unlock() - if !current { - return nil, nil, ErrCodexGatewayBorrowChanged - } - checkCtx, cancelCheck := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) - fresh, auth, err := s.recoveryAccount(checkCtx, account.ID, model) - cancelCheck() - if err != nil { - return nil, nil, err - } - if auth.Header.Get("ChatGPT-Account-ID") != originalAccount { - return nil, nil, ErrCodexGatewayBorrowChanged - } - wire.Header.Set("Authorization", auth.Header.Get("Authorization")) - tracker.recovering("searching", nil, nil) - roundCtx, cancelRound := context.WithTimeout(ctx, 180*time.Second) - prepared, app, err := s.Apply(wire.WithContext(roundCtx), fresh, model, proxy, nil, false) - cancelRound() - if err == nil && app != nil && app.Applied { - checkCtx, cancelCheck := context.WithTimeout(ctx, codexGatewayBorrowStoreTimeout) - latest, latestAuth, checkErr := s.recoveryAccount(checkCtx, account.ID, model) - cancelCheck() - if checkErr != nil { - return nil, nil, checkErr - } - if account.Schedulable && !latest.Schedulable { - return nil, nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "account_inactive"} - } - if latestAuth.Header.Get("ChatGPT-Account-ID") != originalAccount || latestAuth.Header.Get("X-OpenAI-Fedramp") != wire.Header.Get("X-OpenAI-Fedramp") { - return nil, nil, ErrCodexGatewayBorrowChanged - } - if latestAuth.Header.Get("Authorization") != wire.Header.Get("Authorization") { - continue - } - tracker.recovering("recovered", nil, nil) - return prepared.WithContext(codexGatewayBorrowRecoveryContext{Context: caller, values: prepared.Context()}), app, nil - } - if err == nil { - err = ErrCodexGatewayBorrowChanged - } - if ctx.Err() != nil { - continue - } - if errors.Is(err, ErrCodexGatewayBorrowChanged) { - // Revision cancellation stops above. Lease expiry or a changed node - // within a live configuration requires a new qualified route. - err = &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "recovery", Reason: "route_changed"} - } - if !borrowRecoveryRetryable(err) { - return nil, nil, err - } - next := s.nextBorrowRecoveryAttempt(ctx, account.ID, model, err) - if borrowImmediateReborrow(ctx) { - // The next round starts immediately, but still performs new network - // acquisition and full probes. Never spin on an empty candidate pool. - tracker.recovering("reborrowing", nil, err) - continue - } - tracker.recovering("cooling", &next, err) - if err := s.waitBorrowRecovery(ctx, next, account.ID); err != nil { - if changed() { - return nil, nil, ErrCodexGatewayBorrowChanged - } - return nil, nil, err - } - } -} diff --git a/backend/internal/service/codex_gateway_borrow_recovery_test.go b/backend/internal/service/codex_gateway_borrow_recovery_test.go deleted file mode 100644 index 688282fa9770..000000000000 --- a/backend/internal/service/codex_gateway_borrow_recovery_test.go +++ /dev/null @@ -1,401 +0,0 @@ -package service - -import ( - "context" - "errors" - "io" - "net/http" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -func enableBorrowRecovery(t *testing.T, s *CodexGatewayBorrowService) { - t.Helper() - cfg := s.ConfigSnapshot() - cfg.WaitForQualifiedRoute, cfg.NodeProxyIDs = true, []int64{10} - s.publishConfig(cfg, false) - s.rotation.mu.Lock() - err := s.rotation.initializeLocked(s.revision) - s.rotation.mu.Unlock() - require.NoError(t, err) -} - -func seedBorrowRecoveryCooling(s *CodexGatewayBorrowService, model string, until time.Time) { - s.rotation.cache(model).reject(2, borrowHash("http://node-10.invalid:8080"), "", until, time.Now()) - s.rotation.mu.Lock() - s.rotation.retry[codexGatewayBorrowTargetKey{2, model}] = until - s.rotation.mu.Unlock() -} - -func TestCodexBorrowRecoveryRetainsRequestAndDispatchesOnceAfterCooling(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - req.Header.Set("X-Codex-Turn-State", "original-business-state") - req.Header.Set("Session-Id", "original-session") - req.Header.Set("X-Codex-Turn-Metadata", `{"thread_id":"original-thread"}`) - body := &struct{ io.ReadCloser }{req.Body} - req.Body = body - until := time.Now().Add(100 * time.Millisecond) - seedBorrowRecoveryCooling(s, "gpt-6-astra", until) - prepared, elapsed, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved-exit") - require.NoError(t, err) - require.GreaterOrEqual(t, elapsed, 80*time.Millisecond) - require.Same(t, body, prepared.Body) - require.Equal(t, "original-business-state", prepared.Header.Get("X-Codex-Turn-State")) - require.Equal(t, req.Header.Get("X-Codex-Turn-Metadata"), prepared.Header.Get("X-Codex-Turn-Metadata")) - require.Equal(t, "original-session", prepared.Header.Get("Session-Id")) - require.Empty(t, req.Header.Get("Cookie")) - require.Len(t, *calls, 3) - row := s.Status().RecentUsage[0] - require.False(t, row.Dispatched) - require.Equal(t, "preparing", row.Outcome) - require.Equal(t, "recovered", row.Recovery.Phase) - require.EqualValues(t, 2, row.Recovery.Rounds) - require.EqualValues(t, 1, row.Recovery.SourceRequests) - require.EqualValues(t, 2, row.Recovery.ProbeRequests) - var dispatched atomic.Int32 - s.gateway.httpUpstream = borrowCoreProbe(func(w *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - dispatched.Add(1) - require.Equal(t, "http://node-10.invalid:8080", proxy) - require.Equal(t, int64(2), id) - require.Equal(t, "original-business-state", w.Header.Get("X-Codex-Turn-State")) - data, err := io.ReadAll(w.Body) - require.NoError(t, err) - require.Equal(t, "business body remains unread", string(data)) - return borrowCoreResponse("gpt-6-astra", "done", "returned-business-state"), nil - }) - resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved-exit") - require.NoError(t, err) - require.Equal(t, "returned-business-state", resp.Header.Get("X-Codex-Turn-State")) - _, err = io.ReadAll(resp.Body) - require.NoError(t, err) - require.NoError(t, resp.Body.Close()) - require.EqualValues(t, 1, dispatched.Load()) - row = s.Status().RecentUsage[0] - require.True(t, row.Dispatched) - require.Equal(t, "completed", row.Outcome) -} - -func TestCodexBorrowRecoveryCancelAndConfigChangeStopWithoutProbes(t *testing.T) { - for _, change := range []bool{false, true} { - t.Run(map[bool]string{false: "cancel", true: "configuration"}[change], func(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - require.Eventually(t, func() bool { - rows := s.Status().RecentUsage - return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" - }, time.Second, time.Millisecond) - if change { - cfg := s.ConfigSnapshot() - cfg.WaitForQualifiedRoute = false - s.publishConfig(cfg, false) - } else { - cancel() - } - select { - case err := <-done: - if change { - require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) - } else { - require.ErrorIs(t, err, context.Canceled) - require.Equal(t, "cancelled", s.Status().RecentUsage[0].Recovery.Phase) - } - case <-time.After(time.Second): - t.Fatal("recovery leaked after cancellation") - } - require.Empty(t, *calls) - }) - } -} - -func TestCodexBorrowRecoveryUsesLatestEligibleNodeDeadline(t *testing.T) { - s, _, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - now := time.Now() - until := now.Add(time.Hour) - seedBorrowRecoveryCooling(s, "gpt-6-astra", until) - next := s.nextBorrowRecoveryAttempt(t.Context(), 2, "gpt-6-astra", ErrCodexGatewayBorrowUnavailable) - require.Equal(t, until, next) - other := s.nextBorrowRecoveryAttempt(t.Context(), 2, "gpt-6.1-sol", ErrCodexGatewayBorrowUnavailable) - require.WithinDuration(t, now.Add(15*time.Second), other, time.Second) -} - -func TestCodexBorrowRecoveryDisabledAndDiagnosticsRemainBounded(t *testing.T) { - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - require.Error(t, err) - require.Len(t, *calls, 3) - require.Nil(t, s.Status().RecentUsage[0].Recovery) - _, err = DecodeCodexGatewayBorrowConfig([]byte(`{"enabled":true,"source_account_ids":[1],"target_account_ids":[2],"models":["gpt-6-astra"],"wait_for_qualified_route":true}`)) - require.ErrorContains(t, err, "requires node selection") - require.False(t, borrowRecoveryRetryable(context.Canceled)) - require.False(t, borrowRecoveryRetryable(ErrCodexGatewayBorrowChanged)) - require.False(t, borrowRecoveryRetryable(errors.New("not a borrowing failure"))) - require.True(t, borrowRecoveryRetryable(&CodexGatewayBorrowFailure{Cause: io.ErrUnexpectedEOF, Reason: "target_probe_failed"})) - require.False(t, borrowRecoveryRetryable(&CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Verification: &CodexGatewayBorrowVerification{MintStatus: 401}})) -} - -func TestCodexBorrowRecoverySharedPreparationSurvivesOneCallerCancelling(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - base := s.upstream - entered := make(chan struct{}, 1) - release := make(chan struct{}) - s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - if id == 1 { - entered <- struct{}{} - select { - case <-release: - case <-req.Context().Done(): - return nil, req.Context().Err() - } - } - return base.DoWithTLS(req, proxy, id, n, profile) - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - first, second := make(chan error, 1), make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - first <- err - }() - select { - case <-entered: - case <-time.After(time.Second): - t.Fatal("source did not start") - } - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - second <- err - }() - require.Eventually(t, func() bool { - s.rotation.flights.mu.Lock() - defer s.rotation.flights.mu.Unlock() - for _, f := range s.rotation.flights.active { - if f.waiters == 2 { - return true - } - } - return false - }, time.Second, time.Millisecond) - cancel() - require.ErrorIs(t, <-first, context.Canceled) - close(release) - require.NoError(t, <-second) - require.Len(t, *calls, 3, "the remaining caller reuses the same bounded preparation") -} - -func TestCodexBorrowRecoveryCoolingDoesNotBlockOtherModel(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - require.Eventually(t, func() bool { - rows := s.Status().RecentUsage - return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" - }, time.Second, time.Millisecond) - _, sol := nodeRequest(t, s, "gpt-6.1-sol") - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), sol, a, "gpt-6.1-sol", "saved") - require.NoError(t, err) - cancel() - require.ErrorIs(t, <-done, context.Canceled) - require.Len(t, *calls, 3) - require.True(t, s.Status().Targets[1].CacheValid) -} - -func TestCodexBorrowRecoveryLastCallerCancellationStopsTheProbe(t *testing.T) { - s, _, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - entered, closed := make(chan struct{}), make(chan struct{}) - s.upstream = borrowCoreProbe(func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - close(entered) - <-req.Context().Done() - close(closed) - return nil, req.Context().Err() - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - select { - case <-entered: - case <-time.After(time.Second): - t.Fatal("probe did not start") - } - cancel() - require.ErrorIs(t, <-done, context.Canceled) - select { - case <-closed: - case <-time.After(time.Second): - t.Fatal("probe outlived its last waiter") - } - require.Eventually(t, func() bool { - s.rotation.flights.mu.Lock() - defer s.rotation.flights.mu.Unlock() - return len(s.rotation.flights.active) == 0 - }, time.Second, time.Millisecond) -} - -type borrowRecoveryAccountSwitch struct { - AccountRepository - inactive atomic.Bool -} - -func (r *borrowRecoveryAccountSwitch) GetByID(ctx context.Context, id int64) (*Account, error) { - a, err := r.AccountRepository.GetByID(ctx, id) - if err != nil || a == nil { - return a, err - } - copy := *a - if id == 2 && r.inactive.Load() { - copy.Status = "disabled" - } - return ©, nil -} - -func TestCodexBorrowRecoveryStopsWhenAccountIsDisabledDuringCooling(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - repo := &borrowRecoveryAccountSwitch{AccountRepository: s.accounts} - s.accounts = repo - s.gateway.accountRepo = repo - a, req := nodeRequest(t, s, "gpt-6-astra") - seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - require.Eventually(t, func() bool { - rows := s.Status().RecentUsage - return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" - }, time.Second, time.Millisecond) - repo.inactive.Store(true) - select { - case err := <-done: - require.ErrorContains(t, err, "account_inactive") - case <-time.After(8 * time.Second): - t.Fatal("disabled account kept waiting") - } - require.Empty(t, *calls) -} - -func TestCodexBorrowRecoveryCoolingGatewayDoesNotHideUntestedNodes(t *testing.T) { - s, calls, proxies := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 12 }) - cfg := s.ConfigSnapshot() - cfg.WaitForQualifiedRoute = true - cfg.NodeProxyIDs = []int64{10, 11, 12} - s.publishConfig(cfg, false) - base := s.upstream - alias := proxies.rows[11] - s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, p *tlsfingerprint.Profile) (*http.Response, error) { - resp, err := base.DoWithTLS(req, proxy, id, n, p) - if err == nil && id == 1 && proxy == alias.URL() { - resp.Header.Set("Set-Cookie", "__oailb="+borrowNodeCookie("chat.gateway.unified-10.api.openai.com")+"; Path=/; Secure; Max-Age=230") - } - return resp, err - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithTimeout(t.Context(), time.Second) - defer cancel() - prepared, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - proof, ok := prepared.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) - require.True(t, ok) - require.Equal(t, int64(12), proof.application.Verification.ProxyID) - require.Len(t, *calls, 7, "two strict pairs and three source acquisitions; no target call to the cooling alias") - require.EqualValues(t, 1, s.Status().RecentUsage[0].Recovery.Rounds) - require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()))) -} - -func TestCodexBorrowRecoveryIdentityEditCancelsCoolingWait(t *testing.T) { - s, calls, _ := nodeFixture(t, 5, func(int64, string) bool { return true }) - enableBorrowRecovery(t, s) - a, req := nodeRequest(t, s, "gpt-6-astra") - seedBorrowRecoveryCooling(s, "gpt-6-astra", time.Now().Add(time.Hour)) - ctx, cancel := context.WithCancel(t.Context()) - defer cancel() - done := make(chan error, 1) - go func() { - _, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6-astra", "saved") - done <- err - }() - require.Eventually(t, func() bool { - rows := s.Status().RecentUsage - return len(rows) > 0 && rows[0].Recovery != nil && rows[0].Recovery.Phase == "cooling" - }, time.Second, time.Millisecond) - notifyCodexFingerprintAccountChanged(2) - select { - case err := <-done: - require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) - case <-time.After(time.Second): - t.Fatal("old identity kept recovering") - } - require.Empty(t, *calls) -} - -func TestCodexBorrowRecoveryRenewsTheQualifiedNodeBeforeRotating(t *testing.T) { - s, calls, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) - cfg := s.ConfigSnapshot() - cfg.WaitForQualifiedRoute = true - cfg.NodeProxyIDs = []int64{10, 11} - s.publishConfig(cfg, false) - base := s.upstream - s.upstream = borrowCoreProbe(func(req *http.Request, proxy string, id int64, n int, p *tlsfingerprint.Profile) (*http.Response, error) { - resp, err := base.DoWithTLS(req, proxy, id, n, p) - if err == nil && id == 1 { - cookies := resp.Cookies() - require.Len(t, cookies, 1) - cookies[0].MaxAge = 1 - resp.Header.Set("Set-Cookie", cookies[0].String()) - } - return resp, err - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - first, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - proof, ok := first.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) - require.True(t, ok) - timer := time.NewTimer(time.Until(proof.application.ExpiresAt.Add(20 * time.Millisecond))) - defer timer.Stop() - <-timer.C - second, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved") - require.NoError(t, err) - next, ok := second.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) - require.True(t, ok) - require.Equal(t, int64(10), next.application.Verification.ProxyID) - require.Len(t, *calls, 6, "renewal reacquires a lease and repeats both target probes") - for _, call := range *calls { - require.Contains(t, call, "10:") - } - require.True(t, next.application.ExpiresAt.After(proof.application.ExpiresAt)) -} diff --git a/backend/internal/service/codex_gateway_borrow_replay.go b/backend/internal/service/codex_gateway_borrow_replay.go deleted file mode 100644 index f31d405ea571..000000000000 --- a/backend/internal/service/codex_gateway_borrow_replay.go +++ /dev/null @@ -1,94 +0,0 @@ -package service - -import ( - "bytes" - "encoding/json" - "errors" - "net/http" - "strings" - - "github.com/tidwall/gjson" -) - -// Administrator-only replay shares the real sender, account allowlist and -// observation isolation. The caller executes local fixture tools and supplies -// complete history. It cannot inject credentials, cookies, destinations, or -// anchors belonging to a real user conversation. -type CodexBorrowReplay struct { - Body json.RawMessage `json:"body"` - Headers map[string]string `json:"headers"` -} - -var codexBorrowReplayHeaders = map[string]bool{ - "user-agent": true, "originator": true, "version": true, "session-id": true, "thread-id": true, - "x-client-request-id": true, "x-codex-window-id": true, "x-codex-installation-id": true, - "x-codex-turn-metadata": true, "x-codex-turn-state": true, -} - -func validateCodexBorrowReplay(request CodexBorrowDiagnosticRequest) error { - if request.Scenario != "codex_replay" { - if request.Replay != nil { - return errors.New("replay requires codex_replay scenario") - } - return nil - } - if request.Transport != "http" || (request.Mode != "ordinary" && request.Mode != "borrowed") || request.Replay == nil { - return errors.New("codex_replay requires HTTP, explicit mode and replay") - } - raw := request.Replay.Body - if len(raw) == 0 || len(raw) > 1<<20 || !gjson.ValidBytes(raw) || !gjson.ParseBytes(raw).IsObject() { - return errors.New("replay body must be a JSON object up to 1 MiB") - } - body := gjson.ParseBytes(raw) - if body.Get("model").String() != request.Model || body.Get("stream").Type != gjson.True || body.Get("store").Type != gjson.False || body.Get("previous_response_id").Exists() { - return errors.New("replay requires the selected model, streaming, store=false and complete history") - } - session := body.Get("prompt_cache_key").String() - if !strings.HasPrefix(session, "codex-acceptance-") || len(session) > 128 { - return errors.New("replay requires an isolated codex-acceptance session") - } - if request.ServiceTier != "" && body.Get("service_tier").String() != request.ServiceTier { - return errors.New("replay service tier must match its body") - } - seen := map[string]bool{} - for name, value := range request.Replay.Headers { - lower := strings.ToLower(name) - if !codexBorrowReplayHeaders[lower] || seen[lower] || strings.ContainsAny(value, "\r\n") || len(value) > 16384 { - return errors.New("unsupported or ambiguous replay header") - } - seen[lower] = true - if (lower == "session-id" || lower == "thread-id") && value != session { - return errors.New("replay session headers must match its isolated session") - } - } - if !seen["session-id"] || !seen["thread-id"] { - return errors.New("replay requires isolated session and thread headers") - } - // Reject trailing values and duplicate top-level keys, so validation and the - // gateway's JSON reader cannot select different models or session identifiers. - decoder := json.NewDecoder(bytes.NewReader(raw)) - _, _ = decoder.Token() - keys := map[string]bool{} - for decoder.More() { - token, err := decoder.Token() - if err != nil { - return err - } - key, ok := token.(string) - if !ok || keys[key] { - return errors.New("duplicate replay body key") - } - keys[key] = true - var value json.RawMessage - if err = decoder.Decode(&value); err != nil { - return err - } - } - return nil -} - -func (r *CodexBorrowReplay) applyHeaders(headers http.Header) { - for name, value := range r.Headers { - headers.Set(name, value) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_replay_test.go b/backend/internal/service/codex_gateway_borrow_replay_test.go deleted file mode 100644 index 9aef71e3b3c1..000000000000 --- a/backend/internal/service/codex_gateway_borrow_replay_test.go +++ /dev/null @@ -1,98 +0,0 @@ -//go:build unit - -package service - -import ( - "context" - "encoding/json" - "io" - "net/http" - "strings" - "testing" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func borrowReplayFixture() CodexBorrowDiagnosticRequest { - return CodexBorrowDiagnosticRequest{Scenario: "codex_replay", Mode: "ordinary", Transport: "http", AccountID: 2, Model: "gpt-6-astra", ServiceTier: "default", Replay: &CodexBorrowReplay{ - Body: json.RawMessage(`{"model":"gpt-6-astra","stream":true,"store":false,"service_tier":"default","reasoning":{"effort":"xhigh"},"prompt_cache_key":"codex-acceptance-synthetic","input":[{"role":"user","content":[{"type":"input_text","text":"synthetic task"}]}],"tools":[{"type":"function","name":"read_file","parameters":{"type":"object","properties":{}}}]}`), - Headers: map[string]string{"session-id": "codex-acceptance-synthetic", "thread-id": "codex-acceptance-synthetic", "User-Agent": "Codex Desktop/0.162.0-alpha.17.2", "x-codex-turn-state": "previous-business-state"}, - }} -} - -func TestCodexGatewayBorrowReplayBoundaries(t *testing.T) { - require.NoError(t, validateCodexBorrowReplay(borrowReplayFixture())) - for _, tc := range []struct { - name string - edit func(*CodexBorrowDiagnosticRequest) - }{ - {"credentials", func(r *CodexBorrowDiagnosticRequest) { r.Replay.Headers["Authorization"] = "Bearer other" }}, - {"cookies", func(r *CodexBorrowDiagnosticRequest) { r.Replay.Headers["Cookie"] = "__oailb=other" }}, - {"ambiguous header", func(r *CodexBorrowDiagnosticRequest) { r.Replay.Headers["Session-Id"] = "another" }}, - {"real session", func(r *CodexBorrowDiagnosticRequest) { - r.Replay.Body = json.RawMessage(strings.ReplaceAll(string(r.Replay.Body), "codex-acceptance-synthetic", "real-user-session")) - }}, - {"wrong model", func(r *CodexBorrowDiagnosticRequest) { r.Model = "gpt-6.1-sol" }}, - {"different history anchor", func(r *CodexBorrowDiagnosticRequest) { - r.Replay.Body = append([]byte(`{"previous_response_id":"real-user-response",`), r.Replay.Body[1:]...) - }}, - {"duplicate body key", func(r *CodexBorrowDiagnosticRequest) { - r.Replay.Body = append([]byte(`{"model":"gpt-6-astra",`), r.Replay.Body[1:]...) - }}, - {"implicit mode", func(r *CodexBorrowDiagnosticRequest) { r.Mode = "" }}, - {"wrong scenario", func(r *CodexBorrowDiagnosticRequest) { r.Scenario = "short" }}, - {"oversize", func(r *CodexBorrowDiagnosticRequest) { - r.Replay.Body = json.RawMessage(strings.Repeat(" ", 1<<20) + string(r.Replay.Body)) - }}, - } { - t.Run(tc.name, func(t *testing.T) { - request := borrowReplayFixture() - tc.edit(&request) - require.Error(t, validateCodexBorrowReplay(request)) - }) - } -} - -func TestCodexGatewayBorrowReplayPreservesTasksAndReturnsBusinessState(t *testing.T) { - for _, id := range []int64{1, 2} { - account := borrowCoreAccount(id) - generator, repo := newPelicanGeneratorForTest(account, &pelicanGeneratorUpstream{}) - gateway := generator.openaiGatewayService - var calls int - gateway.httpUpstream = borrowCoreProbe(func(req *http.Request, _ string, accountID int64, n int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - require.Equal(t, id, accountID) - body := borrowCoreBody(t, req) - require.Equal(t, "xhigh", gjson.GetBytes(body, "reasoning.effort").String()) - require.Equal(t, "default", gjson.GetBytes(body, "service_tier").String()) - require.Equal(t, "synthetic task", gjson.GetBytes(body, "input.0.content.0.text").String()) - require.Equal(t, "read_file", gjson.GetBytes(body, "tools.0.name").String()) - require.Equal(t, "previous-business-state", req.Header.Get("x-codex-turn-state")) - _, finish, err := PreparePelicanHTTPRequest(req, accountID, n, "http") - require.NoError(t, err) - raw := `data: {"type":"response.completed","response":{"id":"synthetic-response","model":"gpt-6-astra","status":"completed","output":[{"type":"function_call","name":"read_file","call_id":"test-call","arguments":"{}"}]}}` + "\n\n" - return finish(&http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"text/event-stream"}, "X-Codex-Turn-State": {"actual-business-state"}}, Body: io.NopCloser(strings.NewReader(raw))}, nil) - }) - s := NewCodexGatewayBorrowService(nil, repo, gateway, nil, nil) - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6-astra"}}, false) - request := borrowReplayFixture() - request.AccountID = id - var result *CodexBorrowDiagnosticResult - err := s.Diagnose(context.Background(), request, func(e CodexBorrowDiagnosticEvent) { - if e.Result != nil { - result = e.Result - } - }) - s.Stop() - require.NoError(t, err) - require.Equal(t, 1, calls) - require.NotNil(t, result) - require.True(t, result.Dispatched) - require.True(t, result.Completed, result.Error) - require.False(t, result.Applied) - require.Equal(t, "actual-business-state", result.TurnState) - require.Contains(t, result.RawResponse, "function_call") - } -} diff --git a/backend/internal/service/codex_gateway_borrow_request_context.go b/backend/internal/service/codex_gateway_borrow_request_context.go deleted file mode 100644 index 00caccdba2eb..000000000000 --- a/backend/internal/service/codex_gateway_borrow_request_context.go +++ /dev/null @@ -1,22 +0,0 @@ -package service - -import ( - "context" - "net/http" -) - -type codexBorrowServiceTierContextKey struct{} - -// The finalized body owns service_tier. The routing hint intentionally omits -// default/auto/scale, so it cannot reconstruct every actual request condition. -// Keep the exact effective tier locally; never add an internal transport header. -func withCodexBorrowServiceTier(ctx context.Context, tier string) context.Context { - return context.WithValue(ctx, codexBorrowServiceTierContextKey{}, normalizedOpenAIServiceTierValue(tier)) -} - -func codexBorrowRequestServiceTier(req *http.Request) string { - if tier, ok := req.Context().Value(codexBorrowServiceTierContextKey{}).(string); ok { - return tier - } - return borrowProbeServiceTier(req.Header) // Compatibility for existing manual templates. -} diff --git a/backend/internal/service/codex_gateway_borrow_request_context_test.go b/backend/internal/service/codex_gateway_borrow_request_context_test.go deleted file mode 100644 index e17f8284784f..000000000000 --- a/backend/internal/service/codex_gateway_borrow_request_context_test.go +++ /dev/null @@ -1,37 +0,0 @@ -package service - -import ( - "context" - "net/http" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexBorrowPinnedProbeKeepsBusinessTierOutOfQualification(t *testing.T) { - account := borrowCoreAccount(2) - var received []string - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - tier := gjson.GetBytes(borrowCoreBody(t, req), "service_tier").String() - received = append(received, tier) - assert.Equal(t, "model=gpt-6.1-sol", req.Header.Get(openAICodexRoutingHintHeader)) - return borrowCoreResponse("gpt-6.1-sol", "OK", "stable"), nil - }, account) - borrowCoreCandidate(s, time.Now().Add(time.Minute)) - _, req, _, err := s.accountTemplate(context.Background(), account.ID, "gpt-6.1-sol") - require.NoError(t, err) - for _, tier := range []string{"default", "auto", "scale", ""} { - wire := req.WithContext(withCodexBorrowServiceTier(req.Context(), tier)) - _, applied, err := s.Apply(wire, account, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.Empty(t, applied.Verification.ServiceTier) - } - require.Equal(t, []string{"", ""}, received) - _, _, err = s.Apply(req.WithContext(withCodexBorrowServiceTier(req.Context(), "default")), account, "gpt-6.1-sol", "", nil, true) - require.NoError(t, err, "the first tier's independent successful proof remains reusable") - require.Len(t, received, 2) -} diff --git a/backend/internal/service/codex_gateway_borrow_request_repair_test.go b/backend/internal/service/codex_gateway_borrow_request_repair_test.go deleted file mode 100644 index 38b40e3e2e58..000000000000 --- a/backend/internal/service/codex_gateway_borrow_request_repair_test.go +++ /dev/null @@ -1,164 +0,0 @@ -package service - -import ( - "context" - "net/http" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/ctxkey" - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexBorrowNilRequestDoesNotCreateDiagnosticEvidence(t *testing.T) { - s := newBorrowCoreTest(t, nil) - wire, application, err := s.Apply(nil, borrowCoreAccount(2), "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.Nil(t, wire) - require.Nil(t, application) -} - -// The pinned ranxi probe changes only the legacy session_id per shot. Native -// session/thread headers belong to the template and must survive both shots. -// A synthetic upstream rotates STATE when those headers change, reproducing -// the false rejection introduced by manufacturing a new native session per shot. -func TestCodexBorrowProbePreservesPinnedSessionContract(t *testing.T) { - oldCompression := codexRequestZstd.Load() - SetCodexRequestZstdEnabled(true) - defer SetCodexRequestZstdEnabled(oldCompression) - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - t.Run(model, func(t *testing.T) { - var shots []*http.Request - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - assert.Equal(t, int64(2), id) - assert.Equal(t, "target-exit", proxy) - body := borrowCoreBody(t, req) - assert.Equal(t, model, gjson.GetBytes(body, "model").String()) - assert.False(t, gjson.GetBytes(body, "service_tier").Exists()) - assert.Equal(t, `{"model":"`+model+`","instructions":"Reply with OK.","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Reply with OK."}]}],"stream":true,"store":false,"parallel_tool_calls":true,"include":["reasoning.encrypted_content"]}`, string(body), "use the fixed upstream byte template with only the actual model parameterized") - assert.Equal(t, "model="+model+";tier=priority", req.Header.Get(openAICodexRoutingHintHeader)) - assert.Empty(t, req.Header.Get("Content-Encoding")) - assert.Empty(t, req.Header.Get("X-Codex-Turn-Metadata")) - assert.NotEmpty(t, req.Header.Get("session_id")) - shots = append(shots, req.Clone(context.Background())) - state := "stable-probe-state" - if len(shots) == 2 { - assert.NotEqual(t, shots[0].Header.Get("session_id"), req.Header.Get("session_id")) - assert.Equal(t, "stable-probe-state", req.Header.Get("X-Codex-Turn-State")) - assert.Contains(t, req.Header.Get("Cookie"), "__cflb=target-route") - assert.Equal(t, "__cflb=target-route; __oailb=synthetic-borrowed-cookie", req.Header.Get("Cookie")) - if shots[0].Header.Get("session-id") != req.Header.Get("session-id") || shots[0].Header.Get("thread-id") != req.Header.Get("thread-id") { - state = "different-session-state" - } - } else { - assert.Empty(t, req.Header.Get("X-Codex-Turn-State")) - } - return borrowCoreResponse(model, "OK", state, "__cflb=target-route; Secure; Path=/"), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, model) - require.NoError(t, err) - for k, v := range map[string]string{"session-id": "client-session", "thread-id": "client-thread", "x-client-request-id": "client-request", "x-codex-window-id": "client-window", "X-Codex-Turn-State": "business-state", "X-Codex-Turn-Metadata": `{"session_id":"client-session"}`, "Content-Encoding": "zstd"} { - req.Header.Set(k, v) - } - setOpenAICodexRoutingHint(req.Header, a, model, "priority") - original := req.Header.Clone() - wire, applied, err := s.Apply(req, a, model, "target-exit", nil, false) - require.NoError(t, err) - require.True(t, applied.Applied) - require.Equal(t, original, req.Header) - require.Equal(t, "business-state", wire.Header.Get("X-Codex-Turn-State")) - for _, shot := range shots { - for _, key := range []string{"session-id", "thread-id", "x-client-request-id", "x-codex-window-id"} { - require.Equal(t, original.Get(key), shot.Header.Get(key), key) - } - } - }) - } -} - -func TestCodexBorrowPreparationFailureRetainsReasonAndCounts(t *testing.T) { - var calls int - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - state := "first-state" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "replacement-state" - } - return borrowCoreResponse("gpt-6.1-sol", "OK", state), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - for _, id := range []string{"first-client", "retry-client"} { - ctx := context.WithValue(context.Background(), ctxkey.ClientRequestID, id) - ctx = context.WithValue(ctx, ctxkey.RequestID, "gateway-"+id) - req.Header.Set("x-client-request-id", id) - _, _, err = s.gateway.prepareCodexGatewayBorrowHTTP(ctx, req, a, "gpt-6.1-sol", "") - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Equal(t, "target_validation", failure.Stage) - require.Equal(t, "target_state_changed", failure.Reason) - require.NotNil(t, failure.RetryAfter) - require.NotNil(t, failure.Verification) - require.True(t, failure.Verification.MintCompleted) - require.True(t, failure.Verification.ContinueCompleted) - require.Equal(t, len("replacement-state"), failure.Verification.ContinueStateLength) - } - require.Equal(t, 2, calls, "a different tracing ID does not bypass the failure cooldown") - usage := s.Status().RecentUsage - require.Len(t, usage, 1) - require.EqualValues(t, 2, usage[0].AttemptCount) - require.EqualValues(t, 2, usage[0].BlockedCount) - require.Zero(t, usage[0].Count) - require.Zero(t, usage[0].AppliedCount) - require.False(t, usage[0].Dispatched) - require.Equal(t, "retry-client", usage[0].ClientRequestID) - require.Equal(t, "gateway-retry-client", usage[0].GatewayRequestID) - require.Equal(t, "target_state_changed", usage[0].Reason) -} - -func TestCodexBorrowProofUsesOriginalCacheFields(t *testing.T) { - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(*http.Request, string, int64, int, *tlsfingerprint.Profile) (*http.Response, error) { - return borrowCoreResponse("gpt-6.1-sol", "OK", "stable"), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - _, _, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - for _, header := range []string{"session-id", "thread-id", "x-codex-window-id", "x-codex-installation-id", "x-codex-beta-features", openAICodexRoutingHintHeader} { - changed := req.Clone(req.Context()) - changed.Header.Set(header, "changed") - _, _, err = s.Apply(changed, a, "gpt-6.1-sol", "", nil, true) - require.NoError(t, err, header) - } - req.Header.Set("x-client-request-id", "another-trace") - _, application, err := s.Apply(req, a, "gpt-6.1-sol", "", nil, true) - require.NoError(t, err) - require.True(t, application.Applied) -} - -func TestCodexBorrowAttemptCountersSurviveOutOfOrderDispatch(t *testing.T) { - s := newBorrowCoreTest(t, nil) - first := s.beginAttempt(context.Background(), 2, "gpt-6.1-sol", "http") - newer := s.beginAttempt(context.Background(), 2, "gpt-6.1-sol", "http") - first.dispatched(true) - newer.blocked(&CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "target_validation", Reason: "target_state_changed"}) - newer.blocked(ErrCodexGatewayBorrowUnavailable) - first.observe([]byte(`{"type":"response.completed","response":{"status":"completed"}}`)) - first.finish(nil) - row := s.Status().RecentUsage[0] - require.Equal(t, "blocked", row.Outcome) - require.EqualValues(t, 2, row.AttemptCount) - require.EqualValues(t, 1, row.BlockedCount) - require.EqualValues(t, 1, row.Count) - require.EqualValues(t, 1, row.AppliedCount) - require.False(t, row.Dispatched) -} diff --git a/backend/internal/service/codex_gateway_borrow_rotation.go b/backend/internal/service/codex_gateway_borrow_rotation.go deleted file mode 100644 index 793ccac5cbda..000000000000 --- a/backend/internal/service/codex_gateway_borrow_rotation.go +++ /dev/null @@ -1,667 +0,0 @@ -package service - -// Node selection and cooling follow ranxi2001/sub2api d977e7a5: -// repository/astra_rotation{,_cache}.go (LGPL-3.0). Registered host proxies -// replace Mihomo's private listeners. Proxy/account rows are never modified. - -import ( - "context" - "encoding/base64" - "encoding/json" - "errors" - "fmt" - "net/http" - "regexp" - "slices" - "sort" - "strings" - "sync" - "sync/atomic" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" -) - -type codexBorrowNode struct { - id int64 - identity, name, proxy string - expires *time.Time -} - -type codexBorrowNodeBinding struct { - owner *CodexGatewayBorrowService - child *CodexGatewayBorrowService - revision uint64 - accountID int64 - model, gateway string - node codexBorrowNode -} - -type codexBorrowNodeContextKey struct{} - -type CodexGatewayBorrowNodeStatus struct { - AccountID int64 `json:"account_id"` - Model string `json:"model"` - ProxyID int64 `json:"proxy_id"` - ProxyName string `json:"proxy_name"` - NodeIdentity string `json:"node_identity"` - Gateway string `json:"gateway,omitempty"` - State string `json:"state"` - RetryAfter *time.Time `json:"retry_after,omitempty"` - ExpiresAt *time.Time `json:"expires_at,omitempty"` - Reason string `json:"reason,omitempty"` -} - -type codexBorrowRotation struct { - mu sync.Mutex - revision uint64 - children map[string]*CodexGatewayBorrowService - active map[codexGatewayBorrowTargetKey]string - cursor map[codexGatewayBorrowTargetKey]int - retry map[codexGatewayBorrowTargetKey]time.Time - caches map[string]*codexBorrowRotationCache - sourceCache codexBorrowRotationCache - leases map[string]time.Time - flights codexBorrowFlights -} - -func validateCodexBorrowNodeConfig(cfg CodexGatewayBorrowConfig) error { - if cfg.ImmediateReborrowOnFailure && (!cfg.WaitForQualifiedRoute || !cfg.RotateNodes) { - return errors.New("immediate reborrow requires recovery waiting and node selection") - } - if cfg.Enabled && cfg.WaitForQualifiedRoute && !cfg.RotateNodes { - return errors.New("waiting for a qualified route requires node selection") - } - if cfg.MaxNodeAttempts < 0 || cfg.MaxNodeAttempts > 10 { - return errors.New("node attempts must be 0–10 (0 uses 3)") - } - if cfg.NodeCooldownSeconds != 0 && (cfg.NodeCooldownSeconds < 60 || cfg.NodeCooldownSeconds > 86400) { - return errors.New("node cooldown must be 60–86400 seconds (0 uses 3600)") - } - if len(cfg.NodeProxyIDs) > 64 || (cfg.RotateNodes && len(cfg.NodeProxyIDs) == 0) { - return errors.New("node selection requires 1–64 explicit proxy IDs") - } - seen := map[int64]bool{} - for _, id := range cfg.NodeProxyIDs { - if id < 0 || seen[id] { - return errors.New("node proxy IDs must be unique; 0 explicitly selects server direct") - } - seen[id] = true - } - return nil -} - -func borrowNodeConfigEqual(a, b CodexGatewayBorrowConfig) bool { - return a.ImmediateReborrowOnFailure == b.ImmediateReborrowOnFailure && a.WaitForQualifiedRoute == b.WaitForQualifiedRoute && a.RotateNodes == b.RotateNodes && a.MaxNodeAttempts == b.MaxNodeAttempts && a.NodeCooldownSeconds == b.NodeCooldownSeconds && slices.Equal(a.NodeProxyIDs, b.NodeProxyIDs) -} - -func borrowNodeFromProxy(p *Proxy) (codexBorrowNode, error) { - if p == nil || p.Status != StatusActive || (p.ExpiresAt != nil && !time.Now().Before(*p.ExpiresAt)) || (p.Protocol != "http" && p.Protocol != "https" && p.Protocol != "socks5" && p.Protocol != "socks5h") { - return codexBorrowNode{}, errors.New("node proxy is unavailable") - } - url := p.URL() - return codexBorrowNode{id: p.ID, identity: borrowHash(url), name: p.Name, proxy: url, expires: p.ExpiresAt}, nil -} - -func (s *CodexGatewayBorrowService) borrowNodes(ctx context.Context, ids []int64, strict ...bool) ([]codexBorrowNode, error) { - if s.proxies == nil { - return nil, errors.New("node proxy repository unavailable") - } - databaseIDs := []int64{} - for _, id := range ids { - if id > 0 { - databaseIDs = append(databaseIDs, id) - } - } - rows, err := s.proxies.ListByIDs(ctx, databaseIDs) - if err != nil { - return nil, err - } - byID := map[int64]*Proxy{} - for i := range rows { - byID[rows[i].ID] = &rows[i] - } - seen := map[string]bool{} - nodes := []codexBorrowNode{} - for _, id := range ids { - if id == 0 { - nodes = append(nodes, codexBorrowNode{id: 0, identity: borrowHash("codex-borrow-server-direct"), name: "Server direct"}) - continue - } - node, err := borrowNodeFromProxy(byID[id]) - if err != nil { - if len(strict) > 0 && strict[0] { - return nil, fmt.Errorf("proxy %d: %w", id, err) - } - continue - } - if !seen[node.identity] { - nodes = append(nodes, node) - seen[node.identity] = true - } - } - return nodes, nil -} - -func (r *codexBorrowRotation) cache(model string) *codexBorrowRotationCache { - r.mu.Lock() - defer r.mu.Unlock() - if r.caches == nil { - r.caches = map[string]*codexBorrowRotationCache{} - } - if r.caches[model] == nil { - r.caches[model] = &codexBorrowRotationCache{} - } - return r.caches[model] -} - -func (s *CodexGatewayBorrowService) nodeChild(rev uint64, cfg CodexGatewayBorrowConfig, node codexBorrowNode) (*CodexGatewayBorrowService, error) { - r := &s.rotation - r.mu.Lock() - defer r.mu.Unlock() - if err := r.initializeLocked(rev); err != nil { - return nil, err - } - if child := r.children[node.identity]; child != nil { - return child, nil - } - // Reuse the tested cookie/probe implementation with a separate cache for - // each immutable node. Its gateway dependency still belongs to the host. - child := NewCodexGatewayBorrowService(nil, s.accounts, nil, s.upstream, s.tlsProfiles) - child.gateway = s.gateway - child.node = &node - child.nodeLease = r.cookieExpiry - cfg.RotateNodes = false - cfg.WaitForQualifiedRoute = false - cfg.ImmediateReborrowOnFailure = false - child.publishConfig(cfg, false) - r.children[node.identity] = child - return child, nil -} - -func (r *codexBorrowRotation) initializeLocked(rev uint64) error { - if r.revision > rev { - return ErrCodexGatewayBorrowChanged - } - if r.revision != rev || r.children == nil { - for _, child := range r.children { - child.Stop() - } - r.children = map[string]*CodexGatewayBorrowService{} - r.active = map[codexGatewayBorrowTargetKey]string{} - r.cursor = map[codexGatewayBorrowTargetKey]int{} - r.retry = map[codexGatewayBorrowTargetKey]time.Time{} - r.revision = rev - } - return nil -} - -func (r *codexBorrowRotation) stop() { - r.mu.Lock() - defer r.mu.Unlock() - for _, child := range r.children { - child.Stop() - } -} - -func (r *codexBorrowRotation) cookieExpiry(value string, proposed time.Time) time.Time { - r.mu.Lock() - defer r.mu.Unlock() - now := time.Now() - key := borrowHash(value) - if r.leases == nil { - r.leases = map[string]time.Time{} - } - if old, ok := r.leases[key]; ok && now.Before(old) && old.Before(proposed) { - return old - } - if len(r.leases) >= 4096 { - for k, v := range r.leases { - if !now.Before(v) { - delete(r.leases, k) - } - } - if len(r.leases) >= 4096 { - for k := range r.leases { - delete(r.leases, k) - break - } - } - } - r.leases[key] = proposed - return proposed -} - -var codexBorrowGatewayPattern = regexp.MustCompile(`^chat\.gateway\.unified-[0-9]+\.api\.openai\.com$`) - -// Descriptive routing metadata only, never JWT verification or model identity. -func codexBorrowGateway(value string) string { - parts := strings.Split(value, ".") - if len(parts) != 3 { - return "" - } - raw, err := base64.RawURLEncoding.DecodeString(parts[1]) - if err != nil { - return "" - } - var claims struct { - Host string `json:"host"` - } - if json.Unmarshal(raw, &claims) != nil || !codexBorrowGatewayPattern.MatchString(claims.Host) { - return "" - } - return claims.Host -} - -func (s *CodexGatewayBorrowService) nodeCurrent(ctx context.Context, b *codexBorrowNodeBinding) error { - if b == nil || b.owner != s || s.proxies == nil { - return ErrCodexGatewayBorrowChanged - } - s.mu.Lock() - current := s.config.Enabled && s.config.RotateNodes && s.revision == b.revision && s.revisionCtx.Err() == nil && slices.Contains(s.config.NodeProxyIDs, b.node.id) - s.mu.Unlock() - if !current { - return ErrCodexGatewayBorrowChanged - } - if b.node.id != 0 { - p, err := s.proxies.GetByID(ctx, b.node.id) - if err != nil { - return ErrCodexGatewayBorrowChanged - } - n, err := borrowNodeFromProxy(p) - if err != nil || n.identity != b.node.identity { - return ErrCodexGatewayBorrowChanged - } - } else if b.node.proxy != "" || b.node.identity != borrowHash("codex-borrow-server-direct") { - return ErrCodexGatewayBorrowChanged - } - if !borrowImmediateReborrow(ctx) && time.Now().Before(s.rotation.cache(b.model).blockedUntil(b.accountID, b.node.identity, b.gateway, time.Now())) { - return &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_cooling"} - } - return nil -} - -func borrowNodeBinding(req *http.Request) *codexBorrowNodeBinding { - if req == nil { - return nil - } - if p, ok := req.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation); ok && p.application != nil && p.application.node != nil { - return p.application.node - } - b, _ := req.Context().Value(codexBorrowNodeContextKey{}).(*codexBorrowNodeBinding) - return b -} - -func (s *CodexGatewayBorrowService) applyNode(req *http.Request, account *Account, model string, profile *tlsfingerprint.Profile, cachedOnly, force bool, b *codexBorrowNodeBinding) (*http.Request, *CodexGatewayBorrowApplication, error) { - if b.accountID != account.ID || b.model != model { - return req, nil, ErrCodexGatewayBorrowChanged - } - if err := s.nodeCurrent(req.Context(), b); err != nil { - return req, nil, err - } - wire, app, err := b.child.apply(req, account, model, b.node.proxy, profile, cachedOnly, force) - if app != nil { - app.node = b - app.Verification.ProxyID = b.node.id - app.Verification.ProxyName = b.node.name - app.Verification.NodeIdentity = b.node.identity - app.Verification.Gateway = b.gateway - wire = wire.WithContext(context.WithValue(wire.Context(), codexBorrowNodeContextKey{}, b)) - wire = wire.WithContext(WithHTTPUpstreamProfile(wire.Context(), HTTPUpstreamProfileOpenAIHarvest)) - } - return wire, app, err -} - -func (s *CodexGatewayBorrowService) applyRotated(req *http.Request, account *Account, model, _ string, profile *tlsfingerprint.Profile, cachedOnly, force bool, rev uint64, revisionCtx context.Context) (*http.Request, *CodexGatewayBorrowApplication, error) { - if bound := borrowNodeBinding(req); bound != nil && cachedOnly { - return s.applyNode(req, account, model, profile, true, force, bound) - } - cfg := s.ConfigSnapshot() - immediate := cfg.ImmediateReborrowOnFailure && borrowImmediateReborrow(req.Context()) && !cachedOnly - key := codexGatewayBorrowTargetKey{account.ID, model} - r := &s.rotation - cache := r.cache(model) - nodes, err := s.borrowNodes(req.Context(), cfg.NodeProxyIDs) - if err != nil { - if immediate { - return req, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "node_selection", Reason: "node_candidates_unavailable"} - } - return req, nil, &CodexGatewayBorrowFailure{Cause: err, Stage: "node_selection", Reason: "node_unavailable"} - } - if len(nodes) == 0 { - return req, nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_candidates_unavailable"} - } - // Cached-only dispatch may use the selected node, never search or probe. - if cachedOnly { - r.mu.Lock() - identity := r.active[key] - child := r.children[identity] - r.mu.Unlock() - for _, node := range nodes { - if node.identity == identity && child != nil { - child.mu.Lock() - c := child.currentCandidateLocked() - host := "" - if c != nil { - host = codexBorrowGateway(c.cookie.Value) - } - child.mu.Unlock() - return s.applyNode(req, account, model, profile, true, force, &codexBorrowNodeBinding{s, child, rev, account.ID, model, host, node}) - } - } - return req, nil, ErrCodexGatewayBorrowUnavailable - } - r.mu.Lock() - initializeErr := r.initializeLocked(rev) - r.mu.Unlock() - if initializeErr != nil { - return req, nil, initializeErr - } - flightKey := fmt.Sprintf("%d:%d:%s:%s", rev, account.ID, model, borrowRequestFingerprint(req, model, "", profile, "")) - var preparationOwner atomic.Bool - value, err := r.flights.do(req.Context(), revisionCtx, flightKey, 180*time.Second, func(ctx context.Context) (any, error) { - preparationOwner.Store(true) - limit := cfg.MaxNodeAttempts - if limit == 0 { - limit = 3 - } - seen := map[string]bool{} - attempts := 0 - transientTargetFailure := false - var lastErr error - for scanned := 0; scanned < len(nodes)+1 && attempts < limit; scanned++ { - if err := ctx.Err(); err != nil { - return nil, err - } - s.mu.Lock() - current := s.revision == rev && s.config.Enabled - s.mu.Unlock() - if !current { - return nil, ErrCodexGatewayBorrowChanged - } - r.mu.Lock() - identity := r.active[key] - retry := r.retry[key] - child := r.children[identity] - r.mu.Unlock() - if pausedUntil := cache.sourceSearchPausedUntil(account.ID, time.Now()); !cfg.WaitForQualifiedRoute && pausedUntil.After(retry) { - retry = pausedUntil - } - if !immediate && time.Now().Before(retry) { - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_search_cooling", RetryAfter: &retry} - } - var node codexBorrowNode - if scanned == 0 && child != nil { - child.mu.Lock() - candidate := child.currentCandidateLocked() - live := candidate != nil && time.Now().Before(candidate.expires) - child.mu.Unlock() - if live || cfg.WaitForQualifiedRoute { - for _, n := range nodes { - if n.identity == identity { - node = n - break - } - } - } - } - if node.identity == "" { - r.mu.Lock() - cursor := r.cursor[key] - node = nodes[cursor%len(nodes)] - if r.cursor == nil { - r.cursor = map[codexGatewayBorrowTargetKey]int{} - } - r.cursor[key] = cursor + 1 - r.mu.Unlock() - } - if seen[node.identity] || (!immediate && time.Now().Before(cache.blockedUntil(account.ID, node.identity, "", time.Now()))) { - continue - } - allSourcesCooling := true - for _, id := range cfg.SourceAccountIDs { - if !r.sourceCache.sourceCooling(id, node.identity, time.Now()) { - allSourcesCooling = false - } - } - if !immediate && allSourcesCooling { - continue - } - seen[node.identity] = true - attempts++ - child, err := s.nodeChild(rev, cfg, node) - if err != nil { - return nil, err - } - child.mu.Lock() - candidate := child.currentCandidateLocked() - childRev, childCtx := child.revision, child.revisionCtx - previous := child.targets[key] - child.mu.Unlock() - freshSource := immediate && (candidate == nil || !time.Now().Before(candidate.expires) || identity != node.identity || (!previous.result.Success && previous.result.Reason != "")) - if freshSource || candidate == nil || !time.Now().Before(candidate.expires) { - codexBorrowUsageFromContext(ctx).recovering("reborrowing", nil, nil) - sourceCtx, sourceCancel := context.WithTimeout(ctx, 45*time.Second) - if freshSource { - sourceCtx = context.WithValue(sourceCtx, borrowFreshSourceKey{}, true) - sourceCtx = context.WithValue(sourceCtx, borrowAcquisitionTriggerKey{}, "immediate_reborrow") - } - err = child.prepareSource(sourceCtx, childRev, childCtx) - sourceCancel() - if err != nil { - lastErr = err - if ctx.Err() != nil { - return nil, ctx.Err() - } - for _, id := range cfg.SourceAccountIDs { - r.sourceCache.rejectSource(id, node.identity, time.Now()) - } - continue - } - child.mu.Lock() - candidate = child.currentCandidateLocked() - child.mu.Unlock() - } - if candidate == nil || !time.Now().Before(candidate.expires) { - lastErr = ErrCodexGatewayBorrowUnavailable - continue - } - host := codexBorrowGateway(candidate.cookie.Value) - cache.observe(node.identity, host, time.Now()) - if !immediate && time.Now().Before(cache.blockedUntil(account.ID, node.identity, host, time.Now())) { - if cfg.WaitForQualifiedRoute { - // This node rediscovered a rejected gateway. Other distinct - // nodes may still qualify within this round's existing budget. - // observe() ensures this alias is skipped on later rounds. - lastErr = &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "gateway_cooling"} - continue - } - cache.pauseSourceSearch(account.ID, time.Now()) - retry := cache.sourceSearchPausedUntil(account.ID, time.Now()) - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "gateway_cooling", RetryAfter: &retry} - } - binding := &codexBorrowNodeBinding{s, child, rev, account.ID, model, host, node} - codexBorrowUsageFromContext(ctx).recovering("validating", nil, nil) - _, app, err := s.applyNode(req.WithContext(ctx), account, model, profile, false, force || freshSource, binding) - s.publishNodeObservation(rev, key, binding, candidate) - if err == nil && app != nil { - r.mu.Lock() - if r.revision != rev { - r.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - r.active[key] = node.identity - delete(r.retry, key) - r.mu.Unlock() - return binding, nil - } - lastErr = err - if ctx.Err() != nil { - return nil, ctx.Err() - } - if immediate && !borrowRecoveryRetryable(err) { - return nil, err - } - var failure *CodexGatewayBorrowFailure - if errors.As(err, &failure) && failure.Verification != nil { - seconds := cfg.NodeCooldownSeconds - if seconds == 0 { - seconds = 3600 - } - if failure.Verification.RetryOnNextRequest { - // Preserve v16: an interrupted target probe has no negative - // cache. The next independent request runs a full new pair. - transientTargetFailure = true - } else { - cache.reject(account.ID, node.identity, host, time.Now().Add(time.Duration(seconds)*time.Second), time.Now()) - } - } else if errors.Is(err, ErrCodexGatewayBorrowChanged) { - return nil, err - } - r.mu.Lock() - if r.active[key] == node.identity { - delete(r.active, key) - } - r.mu.Unlock() - } - retry := time.Now().Add(15 * time.Second) - if transientTargetFailure || immediate { - retry = time.Time{} - } - r.mu.Lock() - if r.revision != rev { - r.mu.Unlock() - return nil, ErrCodexGatewayBorrowChanged - } - if r.retry == nil { - r.retry = map[codexGatewayBorrowTargetKey]time.Time{} - } - r.retry[key] = retry - r.mu.Unlock() - failure := &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_candidates_exhausted", RetryAfter: &retry} - if retry.IsZero() { - failure.RetryAfter = nil - } - if lastErr != nil { - failure.Detail = lastErr.Error() - var previous *CodexGatewayBorrowFailure - if errors.As(lastErr, &previous) { - failure.Verification = previous.Verification - } - } - return nil, failure - }) - if !preparationOwner.Load() { - codexBorrowUsageFromContext(req.Context()).sharedRecoveryPreparation() - } - if err != nil { - return req, nil, err - } - binding, ok := value.(*codexBorrowNodeBinding) - if !ok { - return req, nil, ErrCodexGatewayBorrowUnavailable - } - return s.applyNode(req, account, model, profile, true, false, binding) -} - -func (s *CodexGatewayBorrowService) publishNodeObservation(rev uint64, key codexGatewayBorrowTargetKey, b *codexBorrowNodeBinding, candidate *codexGatewayBorrowCandidate) { - b.child.mu.Lock() - check := b.child.targets[key] - sources := make(map[int64]CodexGatewayBorrowSourceStatus, len(b.child.sources)) - for id, row := range b.child.sources { - sources[id] = row - } - b.child.mu.Unlock() - check.result.ProxyID = b.node.id - check.result.ProxyName = b.node.name - check.result.NodeIdentity = b.node.identity - check.result.Gateway = b.gateway - s.mu.Lock() - defer s.mu.Unlock() - if s.revision != rev { - return - } - s.targets[key] = check - if s.rotationCandidates == nil { - s.rotationCandidates = map[codexGatewayBorrowTargetKey]*codexGatewayBorrowCandidate{} - } - s.rotationCandidates[key] = candidate - s.candidate = candidate - for id, row := range sources { - s.sources[id] = row - } -} - -func (r *codexBorrowRotation) snapshot(cfg CodexGatewayBorrowConfig, revision uint64) []CodexGatewayBorrowNodeStatus { - if !cfg.RotateNodes { - return nil - } - r.mu.Lock() - defer r.mu.Unlock() - now := time.Now() - out := []CodexGatewayBorrowNodeStatus{} - for _, id := range cfg.TargetAccountIDs { - for _, model := range cfg.Models { - for identity, child := range r.children { - if child.node == nil { - continue - } - node := child.node - if !slices.Contains(cfg.NodeProxyIDs, node.id) { - continue - } - row := CodexGatewayBorrowNodeStatus{AccountID: id, Model: model, ProxyID: node.id, ProxyName: node.name, NodeIdentity: identity, State: "candidate"} - child.mu.Lock() - candidate := child.currentCandidateLocked() - if candidate != nil { - expires := candidate.expires - row.ExpiresAt = &expires - } - row.Reason = child.targets[codexGatewayBorrowTargetKey{id, model}].result.Reason - child.mu.Unlock() - if cache := r.caches[model]; cache != nil { - cache.mu.Lock() - row.Gateway = cache.hosts[identity].host - cache.mu.Unlock() - until := cache.blockedUntil(id, identity, row.Gateway, now) - if !cfg.ImmediateReborrowOnFailure && now.Before(until) { - row.State = "cooling" - row.RetryAfter = &until - } - } - if row.State != "cooling" && !cfg.ImmediateReborrowOnFailure { - until := r.sourceCache.sourcesBlockedUntil(cfg.SourceAccountIDs, identity, now) - if now.Before(until) { - row.State = "cooling" - row.Reason = "source_probe_failed" - row.RetryAfter = &until - } - } - if row.State != "cooling" && r.revision == revision && r.active[codexGatewayBorrowTargetKey{id, model}] == identity { - row.State = "selected" - if row.ExpiresAt == nil || !now.Before(*row.ExpiresAt) { - row.State = "expired" - } - } - if row.State == "candidate" && row.ExpiresAt != nil && !now.Before(*row.ExpiresAt) { - row.State = "expired" - } - if cfg.ImmediateReborrowOnFailure && row.State == "candidate" && row.Reason != "" && row.Reason != "target_probe_passed" && row.Reason != "validating" { - row.State = "rejected" - } - out = append(out, row) - } - } - } - sort.Slice(out, func(i, j int) bool { - a, b := out[i], out[j] - if a.AccountID != b.AccountID { - return a.AccountID < b.AccountID - } - if a.Model != b.Model { - return a.Model < b.Model - } - return a.ProxyID < b.ProxyID - }) - return out -} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_cache.go b/backend/internal/service/codex_gateway_borrow_rotation_cache.go deleted file mode 100644 index df211a3025ae..000000000000 --- a/backend/internal/service/codex_gateway_borrow_rotation_cache.go +++ /dev/null @@ -1,165 +0,0 @@ -// Adapted from ranxi2001/sub2api d977e7a5, astra_rotation_cache.go (LGPL-3.0). -// Each model owns an independent instance; the upstream cache rules are unchanged. -package service - -import ( - "sync" - "time" -) - -// Cache decisions survive settings revisions, not process restarts. They never -// contain Cookies or credentials, and are not evidence of an upstream cache TTL. -type codexBorrowRotationCache struct { - mu sync.Mutex - blocked map[codexBorrowRotationCacheKey]time.Time - hosts map[string]codexBorrowNodeHost -} -type codexBorrowRotationCacheKey struct { - target int64 - kind, value string -} -type codexBorrowNodeHost struct { - host string - expires time.Time -} - -func (c *codexBorrowRotationCache) prune(now time.Time) { - for k, v := range c.blocked { - if !now.Before(v) { - delete(c.blocked, k) - } - } - for k, v := range c.hosts { - if !now.Before(v.expires) { - delete(c.hosts, k) - } - } -} -func (c *codexBorrowRotationCache) observe(node, host string, now time.Time) { - if node == "" || host == "" { - return - } - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - if c.hosts == nil { - c.hosts = map[string]codexBorrowNodeHost{} - } - if len(c.hosts) >= 4096 { - for k := range c.hosts { - delete(c.hosts, k) - break - } - } - c.hosts[node] = codexBorrowNodeHost{host: host, expires: now.Add(24 * time.Hour)} -} -func (c *codexBorrowRotationCache) reject(target int64, node, host string, until, now time.Time) { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - if c.blocked == nil { - c.blocked = map[codexBorrowRotationCacheKey]time.Time{} - } - for kind, value := range map[string]string{"node": node, "host": host} { - if value == "" { - continue - } - k := codexBorrowRotationCacheKey{target: target, kind: kind, value: value} - // A late duplicate observation must not slide an existing quiet window. - if _, exists := c.blocked[k]; exists { - continue - } - if len(c.blocked) >= 16384 { - for old := range c.blocked { - delete(c.blocked, old) - break - } - } - c.blocked[k] = until - } -} -func (c *codexBorrowRotationCache) blockedUntil(target int64, node, host string, now time.Time) time.Time { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - if host == "" { - host = c.hosts[node].host - } - a := c.blocked[codexBorrowRotationCacheKey{target: target, kind: "node", value: node}] - b := c.blocked[codexBorrowRotationCacheKey{target: target, kind: "host", value: host}] - if b.After(a) { - return b - } - return a -} - -// Source transport/qualification failures are scoped to the donor and node, -// independently from the longer target-host quiet window. -const codexBorrowSourceFailureCooldown = time.Minute - -func (c *codexBorrowRotationCache) sourceCooling(source int64, node string, now time.Time) bool { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - return now.Before(c.blocked[codexBorrowRotationCacheKey{target: source, kind: "source", value: node}]) -} - -func (c *codexBorrowRotationCache) sourcesBlockedUntil(ids []int64, node string, now time.Time) time.Time { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - var earliest time.Time - for _, id := range ids { - until := c.blocked[codexBorrowRotationCacheKey{target: id, kind: "source", value: node}] - if !now.Before(until) { - return time.Time{} - } - if earliest.IsZero() || until.Before(earliest) { - earliest = until - } - } - return earliest -} -func (c *codexBorrowRotationCache) rejectSource(source int64, node string, now time.Time) { - if node == "" { - return - } - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - if c.blocked == nil { - c.blocked = map[codexBorrowRotationCacheKey]time.Time{} - } - key := codexBorrowRotationCacheKey{target: source, kind: "source", value: node} - if _, ok := c.blocked[key]; ok { - return - } - if len(c.blocked) >= 16384 { - for old := range c.blocked { - delete(c.blocked, old) - break - } - } - c.blocked[key] = now.Add(codexBorrowSourceFailureCooldown) -} - -// A new exit which rediscovers a cooling host must not trigger another full -// source sweep on every request. Keep the pause across settings revisions. -func (c *codexBorrowRotationCache) pauseSourceSearch(target int64, now time.Time) { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - if c.blocked == nil { - c.blocked = map[codexBorrowRotationCacheKey]time.Time{} - } - key := codexBorrowRotationCacheKey{target: target, kind: "source_search"} - if _, exists := c.blocked[key]; !exists { - c.blocked[key] = now.Add(5 * time.Minute) - } -} -func (c *codexBorrowRotationCache) sourceSearchPausedUntil(target int64, now time.Time) time.Time { - c.mu.Lock() - defer c.mu.Unlock() - c.prune(now) - return c.blocked[codexBorrowRotationCacheKey{target: target, kind: "source_search"}] -} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_http.go b/backend/internal/service/codex_gateway_borrow_rotation_http.go deleted file mode 100644 index 63dc057c249b..000000000000 --- a/backend/internal/service/codex_gateway_borrow_rotation_http.go +++ /dev/null @@ -1,192 +0,0 @@ -package service - -import ( - "context" - "fmt" - "io" - "net/http" - "strings" - "sync" - "time" - - "github.com/tidwall/gjson" -) - -// Business failures only affect future selection. A dispatched generation is -// never replayed by node selection, including an incomplete or ambiguous stream. -func (b *codexBorrowNodeBinding) rejectBusiness(reason string, explicit bool) { - if b == nil { - return - } - s := b.owner - s.mu.Lock() - current := s.revision == b.revision - cfg := s.config - s.mu.Unlock() - if !current { - return - } - seconds := 60 - if explicit { - seconds = cfg.NodeCooldownSeconds - if seconds == 0 { - seconds = 3600 - } - } - host := "" - if explicit { - host = b.gateway - } - now := time.Now() - until := now.Add(time.Duration(seconds) * time.Second) - cache := s.rotation.cache(b.model) - cache.reject(b.accountID, b.node.identity, host, until, now) - until = cache.blockedUntil(b.accountID, b.node.identity, b.gateway, now) - key := codexGatewayBorrowTargetKey{b.accountID, b.model} - b.child.mu.Lock() - check := b.child.targets[key] - check.result.Success = false - check.result.Reason = reason - check.result.RetryOnNextRequest = !explicit - check.retryAfter = until - check.acquisition.Phase = "failed" - check.acquisition.Reason = reason - check.acquisition.RetryAfter = &until - b.child.targets[key] = check - prefix := fmt.Sprintf("%d:%s:", b.accountID, b.model) - for k := range b.child.qualifications { - if strings.HasPrefix(k, prefix) { - delete(b.child.qualifications, k) - } - } - b.child.mu.Unlock() - s.rotation.mu.Lock() - if s.rotation.active[key] == b.node.identity { - delete(s.rotation.active, key) - } - s.rotation.mu.Unlock() - s.mu.Lock() - defer s.mu.Unlock() - if s.revision == b.revision && s.targets[key].result.NodeIdentity == b.node.identity { - check.result.ProxyID = b.node.id - check.result.ProxyName = b.node.name - check.result.NodeIdentity = b.node.identity - check.result.Gateway = b.gateway - s.targets[key] = check - delete(s.rotationCandidates, key) - } -} - -func trackCodexBorrowNodeHTTP(req *http.Request, resp *http.Response, err error) (*http.Response, error) { - b := borrowNodeBinding(req) - if b == nil { - return resp, err - } - if req.Context().Err() != nil { - return resp, err - } - if err != nil || resp == nil { - b.rejectBusiness("business_transport_failed", false) - return resp, err - } - old, next := req.Header.Get("X-Codex-Turn-State"), resp.Header.Get("X-Codex-Turn-State") - explicitRejected := false - if req.URL.Path == "/backend-api/codex/responses" && old != "" && next != "" && old != next { - b.rejectBusiness("business_state_changed", true) - explicitRejected = true - } - if sent, e := req.Cookie("__oailb"); e == nil { - for _, got := range resp.Cookies() { - if got.Name == "__oailb" && (got.Value != sent.Value || got.MaxAge < 0 || (!got.Expires.IsZero() && !time.Now().Before(got.Expires))) { - b.rejectBusiness("business_route_changed", true) - explicitRejected = true - break - } - } - } - if resp.StatusCode != http.StatusOK || resp.Body == nil { - if !explicitRejected { - b.rejectBusiness("business_http_failed", false) - } - return resp, err - } - // Compact/lite JSON has its own terminal and STATE protocol. Never turn - // the absence of SSE framing on that path into a stream failure. - if req.URL.Path == "/backend-api/codex/responses" && (resp.Header.Get("Content-Type") == "" || strings.Contains(resp.Header.Get("Content-Type"), "text/event-stream")) { - resp.Body = &codexBorrowNodeBody{ReadCloser: resp.Body, binding: b, ctx: req.Context(), rejected: explicitRejected} - } - return resp, err -} - -type codexBorrowNodeBody struct { - io.ReadCloser - binding *codexBorrowNodeBinding - ctx context.Context - mu sync.Mutex - line []byte - payload []byte - completed, failed bool - rejected bool - once sync.Once -} - -func (b *codexBorrowNodeBody) observe() { - line := strings.TrimSpace(string(b.line)) - b.line = nil - if line != "" { - if strings.HasPrefix(line, "data:") && len(b.payload) < 1<<20 { - b.payload = append(b.payload, strings.TrimSpace(strings.TrimPrefix(line, "data:"))...) - b.payload = append(b.payload, '\n') - } - return - } - payload := strings.TrimSpace(string(b.payload)) - b.payload = nil - if payload == "" || payload == "[DONE]" { - return - } - if !gjson.Valid(payload) { - b.failed = true - return - } - kind := gjson.Get(payload, "type").String() - switch kind { - case "response.completed", "response.done": - b.completed = gjson.Get(payload, "response.status").String() == "completed" && !gjson.Get(payload, "response.error").IsObject() - case "response.failed", "response.incomplete", "response.cancelled", "response.canceled", "error": - b.failed = true - } -} - -func (b *codexBorrowNodeBody) finish() { - b.once.Do(func() { - if !b.rejected && b.ctx.Err() == nil && (!b.completed || b.failed) { - b.binding.rejectBusiness("business_stream_incomplete", false) - } - }) -} - -func (b *codexBorrowNodeBody) Read(p []byte) (int, error) { - n, err := b.ReadCloser.Read(p) - b.mu.Lock() - defer b.mu.Unlock() - for _, ch := range p[:n] { - if ch == '\n' { - b.observe() - } else if len(b.line) < 1<<20 { - b.line = append(b.line, ch) - } - } - if err != nil { - b.finish() - } - return n, err -} - -func (b *codexBorrowNodeBody) Close() error { - err := b.ReadCloser.Close() - b.mu.Lock() - defer b.mu.Unlock() - b.finish() - return err -} diff --git a/backend/internal/service/codex_gateway_borrow_rotation_test.go b/backend/internal/service/codex_gateway_borrow_rotation_test.go deleted file mode 100644 index 454da47b8c95..000000000000 --- a/backend/internal/service/codex_gateway_borrow_rotation_test.go +++ /dev/null @@ -1,423 +0,0 @@ -package service - -import ( - "context" - "encoding/base64" - "fmt" - "io" - "net/http" - "strings" - "sync" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -type borrowNodeProxies struct { - ProxyRepository - rows map[int64]Proxy -} - -func (p *borrowNodeProxies) GetByID(_ context.Context, id int64) (*Proxy, error) { - v, ok := p.rows[id] - if !ok { - return nil, ErrProxyNotFound - } - return &v, nil -} -func (p *borrowNodeProxies) ListByIDs(_ context.Context, ids []int64) ([]Proxy, error) { - out := []Proxy{} - for _, id := range ids { - if p, ok := p.rows[id]; ok { - out = append(out, p) - } - } - return out, nil -} - -func borrowNodeCookie(host string) string { - return "header." + base64.RawURLEncoding.EncodeToString([]byte(`{"host":"`+host+`"}`)) + ".signature" -} - -func nodeFixture(t *testing.T, limit int, decide func(int64, string) bool) (*CodexGatewayBorrowService, *[]string, *borrowNodeProxies) { - t.Helper() - var calls []string - var mu sync.Mutex - proxies := &borrowNodeProxies{rows: map[int64]Proxy{}} - for i := int64(10); i <= 13; i++ { - proxies.rows[i] = Proxy{ID: i, Name: fmt.Sprintf("node-%d", i), Host: fmt.Sprintf("node-%d.invalid", i), Port: 8080, Protocol: "http", Status: StatusActive} - } - s := newBorrowCoreTest(t, func(req *http.Request, proxy string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - mu.Lock() - defer mu.Unlock() - var pid int64 - for k, p := range proxies.rows { - if p.URL() == proxy { - pid = k - } - } - require.NotZero(t, pid, "must select a configured node, never the account proxy") - require.Equal(t, HTTPUpstreamProfileOpenAIHarvest, HTTPUpstreamProfileFromContext(req.Context())) - body := borrowCoreBody(t, req) - model := gjson.GetBytes(body, "model").String() - calls = append(calls, fmt.Sprintf("%d:%d:%s", pid, id, model)) - host := fmt.Sprintf("chat.gateway.unified-%d.api.openai.com", pid) - if id == 1 { - return borrowCoreResponse("gpt-6-astra", "OK", "source-state", "__oailb="+borrowNodeCookie(host)+"; Path=/; Secure; Max-Age=230"), nil - } - require.Equal(t, int64(2), id) - require.Equal(t, "Bearer synthetic-token", req.Header.Get("Authorization")) - c, err := req.Cookie("__oailb") - require.NoError(t, err) - require.Equal(t, borrowNodeCookie(host), c.Value) - state := "first-state" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "" - if !decide(pid, model) { - state = "different-state" - } - } - return borrowCoreResponse(model, "OK", state), nil - }, borrowCoreAccount(1), borrowCoreAccount(2), borrowCoreAccount(3)) - s.proxies = proxies - s.publishConfig(CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{1}, TargetAccountIDs: []int64{2}, Models: []string{"gpt-6-astra", "gpt-6.1-sol"}, RotateNodes: true, NodeProxyIDs: []int64{10, 11, 12, 13}, MaxNodeAttempts: limit}, false) - return s, &calls, proxies -} - -func nodeRequest(t *testing.T, s *CodexGatewayBorrowService, model string) (*Account, *http.Request) { - t.Helper() - a, r, _, err := s.accountTemplate(t.Context(), 2, model) - require.NoError(t, err) - r.Body = io.NopCloser(strings.NewReader("business body remains unread")) - return a, r -} - -func TestCodexBorrowNodeSelectionPinsBusinessAndCoolsFailure(t *testing.T) { - s, calls, _ := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 11 }) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, app, err := s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(11), app.Verification.ProxyID) - require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "11:1:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6-astra"}, *calls) - data, err := io.ReadAll(wire.Body) - require.NoError(t, err) - require.Equal(t, "business body remains unread", string(data)) - require.Empty(t, req.Header.Get("Cookie")) - accounts, ok := s.accounts.(*borrowCoreAccounts) - require.True(t, ok) - require.Nil(t, accounts.rows[1].ProxyID) - require.Nil(t, accounts.rows[2].ProxyID) - _, app, err = s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(11), app.Verification.ProxyID) - require.Len(t, *calls, 6) - rows := s.Status().Targets - require.True(t, rows[0].CacheValid) - cache := s.rotation.cache("gpt-6-astra") - require.True(t, time.Now().Before(cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()))) - _, other, err := s.Apply(req, borrowCoreAccount(3), "gpt-6-astra", "ordinary-original", nil, false) - require.NoError(t, err) - require.Nil(t, other) - require.Len(t, *calls, 6) -} - -func TestCodexBorrowNodeModelsHaveIndependentVerdicts(t *testing.T) { - s, calls, _ := nodeFixture(t, 1, func(_ int64, model string) bool { return model == "gpt-6.1-sol" }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "account-original", nil, false) - require.Error(t, err) - a, req = nodeRequest(t, s, "gpt-6.1-sol") - _, app, err := s.Apply(req, a, "gpt-6.1-sol", "account-original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(10), app.Verification.ProxyID) - require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6.1-sol", "10:2:gpt-6.1-sol"}, *calls) -} - -func TestCodexBorrowNodeBoundedSearchAndRevisionPreservesCooldown(t *testing.T) { - s, calls, _ := nodeFixture(t, 2, func(id int64, _ string) bool { return id == 12 }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.ErrorContains(t, err, "node_candidates_exhausted") - require.Len(t, *calls, 6) - _, _, err = s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.ErrorContains(t, err, "node_search_cooling") - require.Len(t, *calls, 6) - s.publishConfig(s.ConfigSnapshot(), false) // Same config load cannot reset cooling. - cache := s.rotation.cache("gpt-6-astra") - before := cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now()) - cache.reject(2, borrowHash("http://node-10.invalid:8080"), "", time.Now().Add(2*time.Hour), time.Now()) - require.Equal(t, before, cache.blockedUntil(2, borrowHash("http://node-10.invalid:8080"), "", time.Now())) -} - -func TestCodexBorrowNodeStatusReportsActualCoolingDeadline(t *testing.T) { - s, calls, _ := nodeFixture(t, 1, func(int64, string) bool { return false }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.Error(t, err) - status := s.Status() - row := status.Targets[0] - require.False(t, row.CacheValid) - require.NotNil(t, row.RetryAfter) - require.True(t, row.RetryAfter.After(time.Now().Add(59*time.Minute)), "must not display the child's 15-second cache as node recovery") - require.Equal(t, row.RetryAfter, row.Acquisition.RetryAfter) - require.Equal(t, status.Nodes[0].RetryAfter, row.RetryAfter) - cache := s.rotation.cache("gpt-6-astra") - cache.pauseSourceSearch(a.ID, time.Now()) - until := cache.sourceSearchPausedUntil(a.ID, time.Now()) - _, _, err = s.Apply(req, a, "gpt-6-astra", "original", nil, false) - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Equal(t, "node_search_cooling", failure.Reason) - require.Equal(t, &until, failure.RetryAfter, "must not expose a zero deadline for a five-minute pause") - require.Len(t, *calls, 3, "reading cooling state must not issue new probes") -} - -func TestCodexBorrowNodeStatusRejectsAProofWhoseGatewayEnteredCooling(t *testing.T) { - s, _, _ := nodeFixture(t, 1, func(int64, string) bool { return true }) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - require.True(t, s.Status().Targets[0].CacheValid) - // An older in-flight request on another node can reject the same gateway. - // The stored probe is still a historical success, but cannot authorize send. - until := time.Now().Add(time.Hour) - s.rotation.cache("gpt-6-astra").reject(a.ID, "previous-node", app.Verification.Gateway, until, time.Now()) - _, _, err = s.Apply(wire, a, "gpt-6-astra", "original", nil, true) - require.ErrorContains(t, err, "node_cooling") - row := s.Status().Targets[0] - require.False(t, row.CacheValid, "status must agree with dispatch even while a historical proof is unexpired") - require.Equal(t, "rejected", row.State) - require.Equal(t, "node_cooling", row.Reason) - require.Equal(t, &until, row.RetryAfter) -} - -func TestCodexBorrowNodeDispatchRejectsEditedProxyAndNeverReplaysStream(t *testing.T) { - s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - p := proxies.rows[10] - p.Host = "edited.invalid" - proxies.rows[10] = p - _, _, err = s.Apply(wire, a, "gpt-6-astra", "original", nil, true) - require.ErrorIs(t, err, ErrCodexGatewayBorrowChanged) - require.Len(t, *calls, 3) - p.Host = "node-10.invalid" - proxies.rows[10] = p - resp, err := trackCodexBorrowNodeHTTP(wire, &http.Response{StatusCode: 200, Header: http.Header{}, Body: io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n"))}, nil) - require.NoError(t, err) - _, err = io.Copy(io.Discard, resp.Body) - require.NoError(t, err) - require.NoError(t, resp.Body.Close()) - require.Len(t, *calls, 3, "incomplete generation must never be replayed") - require.True(t, time.Now().Before(s.rotation.cache("gpt-6-astra").blockedUntil(2, app.node.node.identity, "", time.Now()))) - require.False(t, s.Status().Targets[0].CacheValid) -} - -func TestCodexBorrowNodeActualSenderUsesSelectedProxyAndBusinessState(t *testing.T) { - s, calls, _ := nodeFixture(t, 3, func(id int64, _ string) bool { return id == 11 }) - s.gateway.httpUpstream = s.upstream - a, req := nodeRequest(t, s, "gpt-6-astra") - req.Header.Set("X-Codex-Turn-State", "business-state") - req.Body = io.NopCloser(strings.NewReader(`{"model":"gpt-6-astra","stream":true,"store":false,"input":[],"reasoning":{"effort":"xhigh"},"service_tier":"default"}`)) - prepared, _, err := s.gateway.prepareCodexGatewayBorrowHTTP(t.Context(), req, a, "gpt-6-astra", "saved-account-proxy") - require.NoError(t, err) - resp, err := s.gateway.doOpenAICodexUpstream(prepared, a, "saved-account-proxy", "gpt-6-astra") - require.NoError(t, err) - _, err = io.Copy(io.Discard, resp.Body) - require.NoError(t, err) - require.NoError(t, resp.Body.Close()) - require.Equal(t, "business-state", req.Header.Get("X-Codex-Turn-State")) - require.Len(t, *calls, 7) - require.Equal(t, "11:2:gpt-6-astra", (*calls)[6]) - require.True(t, s.Status().Targets[0].CacheValid) - usage := s.Status().RecentUsage - require.Len(t, usage, 1) - require.Equal(t, int64(11), usage[0].ProxyID) - require.True(t, usage[0].Applied) -} - -func TestCodexBorrowNodeBusinessReplacementRemainsVisibleAndCoolsGateway(t *testing.T) { - s, calls, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - wire.Header.Set("X-Codex-Turn-State", "old-business-state") - resp, err := trackCodexBorrowNodeHTTP(wire, borrowCoreResponse("gpt-6-astra", "OK", "new-business-state"), nil) - require.NoError(t, err) - require.Equal(t, "new-business-state", resp.Header.Get("X-Codex-Turn-State")) - _, err = io.Copy(io.Discard, resp.Body) - require.NoError(t, err) - require.NoError(t, resp.Body.Close()) - require.Len(t, *calls, 3) - require.Greater(t, time.Until(s.rotation.cache("gpt-6-astra").blockedUntil(2, app.node.node.identity, app.node.gateway, time.Now())), 59*time.Minute) - require.False(t, s.Status().Targets[0].CacheValid) - require.Equal(t, "business_state_changed", s.Status().Targets[0].Reason) -} - -func TestCodexBorrowNodeStatusUsesEachModelsRoute(t *testing.T) { - s, _, _ := nodeFixture(t, 3, func(id int64, model string) bool { return model == "gpt-6-astra" || id == 11 }) - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - a, req := nodeRequest(t, s, model) - _, _, err := s.Apply(req, a, model, "original", nil, false) - require.NoError(t, err) - } - status := s.Status() - require.True(t, status.Targets[0].CacheValid) - require.True(t, status.Targets[1].CacheValid) - require.Equal(t, int64(10), status.Targets[0].ProxyID) - require.Equal(t, int64(11), status.Targets[1].ProxyID) - a, req := nodeRequest(t, s, "gpt-6-astra") - ctx, cancel := context.WithCancel(req.Context()) - cancel() - _, _, err := s.Apply(req.WithContext(ctx), a, "gpt-6-astra", "original", nil, false) - require.ErrorIs(t, err, context.Canceled) - require.True(t, s.Status().Targets[0].CacheValid, "a cancelled caller must not cool a valid node") -} - -func TestCodexBorrowNodeCoolingHostSkippedAcrossSettingsRevision(t *testing.T) { - s, calls, proxies := nodeFixture(t, 1, func(id int64, _ string) bool { return id == 12 }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.Error(t, err) - require.Len(t, *calls, 3) - cache := s.rotation.cache("gpt-6-astra") - alias := proxies.rows[11] - cache.observe(borrowHash(alias.URL()), "chat.gateway.unified-10.api.openai.com", time.Now()) - cfg := s.ConfigSnapshot() - cfg.MaxNodeAttempts = 3 - s.publishConfig(cfg, false) - _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(12), app.Verification.ProxyID) - require.Equal(t, []string{"10:1:gpt-6-astra", "10:2:gpt-6-astra", "10:2:gpt-6-astra", "12:1:gpt-6-astra", "12:2:gpt-6-astra", "12:2:gpt-6-astra"}, *calls) -} - -func TestCodexBorrowNodeTransientFailureDoesNotCoolGatewayForAnHour(t *testing.T) { - s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) - base := s.upstream - badProxy := proxies.rows[10] - s.upstream = borrowCoreProbe(func(req *http.Request, p string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - resp, err := base.DoWithTLS(req, p, id, n, profile) - if id == 2 && p == badProxy.URL() && req.Header.Get("X-Codex-Turn-State") != "" { - resp.Body = io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n")) - } - return resp, err - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(11), app.Verification.ProxyID) - require.Len(t, *calls, 6) - cache := s.rotation.cache("gpt-6-astra") - bad := proxies.rows[10] - require.True(t, cache.blockedUntil(2, borrowHash(bad.URL()), "", time.Now()).IsZero()) - require.True(t, cache.blockedUntil(2, "another-node", "chat.gateway.unified-10.api.openai.com", time.Now()).IsZero()) -} - -func TestCodexBorrowNodeSingleTransientFailureAllowsNextIndependentPair(t *testing.T) { - s, calls, proxies := nodeFixture(t, 1, func(int64, string) bool { return true }) - cfg := s.ConfigSnapshot() - cfg.NodeProxyIDs = []int64{10} - s.publishConfig(cfg, false) - base := s.upstream - interrupted := false - badProxy := proxies.rows[10] - s.upstream = borrowCoreProbe(func(req *http.Request, p string, id int64, n int, profile *tlsfingerprint.Profile) (*http.Response, error) { - resp, err := base.DoWithTLS(req, p, id, n, profile) - if !interrupted && id == 2 && p == badProxy.URL() && req.Header.Get("X-Codex-Turn-State") != "" { - interrupted = true - resp.Body = io.NopCloser(strings.NewReader("data: {\"type\":\"response.created\"}\n\n")) - } - return resp, err - }) - a, req := nodeRequest(t, s, "gpt-6-astra") - _, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.Error(t, err) - require.Len(t, *calls, 3) - _, app, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - require.True(t, app.Verification.Success) - require.Len(t, *calls, 5, "the next request needs both fresh probe shots, with no source or business replay") -} - -func TestCodexBorrowNodeSameCookieAcrossNodesCannotExtendActiveLease(t *testing.T) { - r := &codexBorrowRotation{} - first := time.Now().Add(20 * time.Second) - require.Equal(t, first, r.cookieExpiry("same-routing-value", first)) - require.Equal(t, first, r.cookieExpiry("same-routing-value", time.Now().Add(230*time.Second))) - other := time.Now().Add(230 * time.Second) - require.Equal(t, other, r.cookieExpiry("different-routing-value", other)) -} - -func TestCodexBorrowNodeSourceCoolingIsSharedButTargetVerdictsAreNot(t *testing.T) { - s, calls, proxies := nodeFixture(t, 3, func(int64, string) bool { return true }) - p := proxies.rows[10] - s.rotation.sourceCache.rejectSource(1, borrowHash(p.URL()), time.Now()) - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - a, req := nodeRequest(t, s, model) - _, app, err := s.Apply(req, a, model, "original", nil, false) - require.NoError(t, err) - require.Equal(t, int64(11), app.Verification.ProxyID) - } - require.Equal(t, []string{"11:1:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6-astra", "11:2:gpt-6.1-sol", "11:2:gpt-6.1-sol"}, *calls) -} - -func TestCodexBorrowNodeCompactJSONDoesNotBecomeAnSSEFailure(t *testing.T) { - s, _, _ := nodeFixture(t, 3, func(int64, string) bool { return true }) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, _, err := s.Apply(req, a, "gpt-6-astra", "original", nil, false) - require.NoError(t, err) - wire.URL.Path = "/backend-api/codex/responses/lite" - wire.Header.Set("X-Codex-Turn-State", "full-response-state") - response := &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"application/json"}, "X-Codex-Turn-State": {"compact-state"}}, Body: io.NopCloser(strings.NewReader(`{"type":"compaction","output":[]}`))} - response, err = trackCodexBorrowNodeHTTP(wire, response, nil) - require.NoError(t, err) - _, err = io.Copy(io.Discard, response.Body) - require.NoError(t, err) - require.NoError(t, response.Body.Close()) - require.Equal(t, "compact-state", response.Header.Get("X-Codex-Turn-State")) - require.True(t, s.Status().Targets[0].CacheValid) -} - -func TestCodexBorrowNodeDirectRequiresExplicitSelection(t *testing.T) { - var calls int - s := newBorrowCoreTest(t, func(req *http.Request, p string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - require.Empty(t, p, "explicit direct node must not use the saved account proxy") - calls++ - if id == 1 { - return borrowCoreResponse("gpt-6-astra", "OK", "source", "__oailb="+borrowNodeCookie("chat.gateway.unified-99.api.openai.com")+"; Path=/; Secure; Max-Age=230"), nil - } - state := "first" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "" - } - return borrowCoreResponse("gpt-6-astra", "OK", state), nil - }, borrowCoreAccount(1), borrowCoreAccount(2)) - s.proxies = &borrowNodeProxies{rows: map[int64]Proxy{}} - cfg := s.ConfigSnapshot() - cfg.TargetAccountIDs = []int64{2} - cfg.RotateNodes = true - cfg.NodeProxyIDs = []int64{0} - _, err := normalizeCodexGatewayBorrowConfig(cfg) - require.NoError(t, err) - s.publishConfig(cfg, false) - a, req := nodeRequest(t, s, "gpt-6-astra") - wire, app, err := s.Apply(req, a, "gpt-6-astra", "saved-proxy", nil, false) - require.NoError(t, err) - require.NotEmpty(t, app.Verification.NodeIdentity) - require.Zero(t, app.Verification.ProxyID) - require.Len(t, s.Status().Nodes, 2) - require.Equal(t, 3, calls) - _, _, err = s.Apply(wire, a, "gpt-6-astra", "saved-proxy", nil, true) - require.NoError(t, err) - require.Equal(t, 3, calls) - cfg.NodeProxyIDs = []int64{99} - s.publishConfig(cfg, false) - _, _, err = s.Apply(wire, a, "gpt-6-astra", "saved-proxy", nil, true) - require.Error(t, err) - require.Equal(t, 3, calls, "removing direct cannot silently fall back to it") -} diff --git a/backend/internal/service/codex_gateway_borrow_source_contract_test.go b/backend/internal/service/codex_gateway_borrow_source_contract_test.go deleted file mode 100644 index 2dffdb990340..000000000000 --- a/backend/internal/service/codex_gateway_borrow_source_contract_test.go +++ /dev/null @@ -1,37 +0,0 @@ -package service - -import ( - "context" - "net/http" - "testing" - - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexBorrowSourceUsesNormalAccountTestContract(t *testing.T) { - headers := http.Header{"Authorization": {"Bearer synthetic-source"}, "User-Agent": {"synthetic-persisted-identity"}, "Originator": {"codex-tui"}, "Version": {"0.162.1"}} - for _, name := range []string{"session-id", "thread-id", "session_id", "x-codex-window-id", "x-codex-turn-state", "x-codex-turn-metadata", "x-codex-routing-hint", "Cookie", "Content-Encoding"} { - headers.Set(name, "target-only-value") - } - original := headers.Clone() - req, err := borrowSourceObservationRequest(context.Background(), headers) - require.NoError(t, err) - require.False(t, req.Close, "source acquisition keeps normal connection policy") - require.Equal(t, original, headers) - require.Equal(t, "Bearer synthetic-source", req.Header.Get("Authorization")) - require.Equal(t, "synthetic-persisted-identity", req.Header.Get("User-Agent")) - for _, name := range []string{"session-id", "thread-id", "session_id", "x-codex-window-id", "x-codex-turn-state", "x-codex-turn-metadata", "x-codex-routing-hint", "Cookie", "Content-Encoding"} { - require.Empty(t, req.Header.Get(name), name) - } - body := borrowCoreBody(t, req) - want := createOpenAITestPayload("gpt-6-astra", true, "Reply with OK only.") - require.Equal(t, want["instructions"], gjson.GetBytes(body, "instructions").String()) - require.Equal(t, "Reply with OK only.", gjson.GetBytes(body, "input.0.content.0.text").String()) - require.Equal(t, "medium", gjson.GetBytes(body, "reasoning.effort").String()) - require.False(t, gjson.GetBytes(body, "parallel_tool_calls").Exists()) - require.False(t, gjson.GetBytes(body, "include").Exists()) - require.Equal(t, HTTPUpstreamProfileCodexBorrowSource, HTTPUpstreamProfileFromContext(req.Context())) - require.True(t, IsCodexGatewayBorrowObservation(req.Context())) - require.True(t, HTTPUpstreamRedirectsDisabled(req.Context())) -} diff --git a/backend/internal/service/codex_gateway_borrow_source_self_test.go b/backend/internal/service/codex_gateway_borrow_source_self_test.go deleted file mode 100644 index 263dec23df6d..000000000000 --- a/backend/internal/service/codex_gateway_borrow_source_self_test.go +++ /dev/null @@ -1,51 +0,0 @@ -package service - -import ( - "context" - "net/http" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexBorrowSourceSelfCheckUsesOnlyCurrentOwner(t *testing.T) { - var calls int - source := borrowCoreAccount(1) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - assert.EqualValues(t, 1, id) - assert.Equal(t, codexGatewayBorrowSourceModel, gjson.GetBytes(borrowCoreBody(t, req), "model").String()) - assert.Empty(t, req.Header.Get("Content-Encoding")) - return borrowCoreResponse(codexGatewayBorrowSourceModel, "OK", "source-state"), nil - }, source) - borrowCoreCandidate(s, time.Now().Add(time.Minute)) - var results []*CodexBorrowDiagnosticResult - err := s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 1, Model: codexGatewayBorrowSourceModel, Transport: "http"}, func(e CodexBorrowDiagnosticEvent) { - if e.Result != nil { - results = append(results, e.Result) - } - }) - require.NoError(t, err) - require.Equal(t, 2, calls) - for _, r := range results { - require.Equal(t, "source", r.ProbeContext.SubjectRole) - require.True(t, r.ProbeOnly) - require.False(t, r.Applied || r.Dispatched || r.Completed) - } - require.Empty(t, s.qualifications) - require.Empty(t, s.Status().RecentUsage) - for _, request := range []CodexBorrowDiagnosticRequest{ - {AccountID: 1, Model: codexGatewayBorrowSourceModel, Transport: "http", Mode: "borrowed"}, - {Scenario: "probe_contract", AccountID: 1, Model: "gpt-6.1-sol", Transport: "http"}, - } { - require.Error(t, s.Diagnose(context.Background(), request, func(CodexBorrowDiagnosticEvent) {})) - } - require.Equal(t, 2, calls, "source selection cannot enter generation or another model") - s.candidate = nil - require.Error(t, s.Diagnose(context.Background(), CodexBorrowDiagnosticRequest{Scenario: "probe_contract", AccountID: 1, Model: codexGatewayBorrowSourceModel, Transport: "http"}, func(CodexBorrowDiagnosticEvent) {})) - require.Equal(t, 2, calls, "a self-check does not silently prepare a replacement candidate") -} diff --git a/backend/internal/service/codex_gateway_borrow_status.go b/backend/internal/service/codex_gateway_borrow_status.go deleted file mode 100644 index 4fd85d000911..000000000000 --- a/backend/internal/service/codex_gateway_borrow_status.go +++ /dev/null @@ -1,66 +0,0 @@ -package service - -import ( - "context" - "encoding/json" - "fmt" -) - -// Read local records without refreshing credentials or probing an upstream. -// The send path still checks the finalized wire identity immediately at dispatch. -func (s *CodexGatewayBorrowService) CurrentStatus(ctx context.Context) CodexGatewayBorrowStatus { - status := s.Status() - if s == nil || s.accounts == nil { - return status - } - accounts := make(map[int64]*Account) - for i := range status.Targets { - row := &status.Targets[i] - account, loaded := accounts[row.AccountID] - if !loaded { - account, _ = s.accounts.GetByID(ctx, row.AccountID) - accounts[row.AccountID] = account - } - reason := "" - if !codexGatewayBorrowAccountSupported(account) { - reason = "account_unavailable" - } else if account.Status != StatusActive { - reason = "account_inactive" - } else if !account.IsModelSupported(row.Model) { - reason = "model_not_allowed" - } else if normalizeOpenAIModelForUpstream(account, account.GetMappedModel(row.Model)) != row.Model { - reason = "model_mapping_mismatch" - } else if row.CacheValid { - credential, err := resolveCredentialAccount(ctx, s.accounts, account) - if err != nil || credential == nil { - reason = "account_unavailable" - } else { - s.mu.Lock() - check := s.targets[codexGatewayBorrowTargetKey{row.AccountID, row.Model}] - s.mu.Unlock() - if check.identityRevision != "" && check.identityRevision != borrowAccountIdentityRevision(account, credential) { - reason = "identity_changed" - } - } - } - if reason != "" { - row.CacheValid = false - row.State = "waiting" - row.Reason = reason - row.Error = fmt.Sprintf("account %d: %s", row.AccountID, reason) - } - } - return status -} - -func borrowAccountIdentityRevision(account, credential *Account) string { - if account == nil || credential == nil { - return "" - } - proxy := "" - if account.Proxy != nil { - proxy = account.Proxy.URL() - } - value, _ := json.Marshal([]any{credential.Credentials, credential.Extra["codex_client_identity"], credential.Extra["codex_fingerprint_seed"], account.Extra["codex_fingerprint_mode"], account.Credentials["model_mapping"], proxy, account.ProxyID}) - return borrowHash(string(value)) -} diff --git a/backend/internal/service/codex_gateway_borrow_token_test.go b/backend/internal/service/codex_gateway_borrow_token_test.go deleted file mode 100644 index 301e178d9640..000000000000 --- a/backend/internal/service/codex_gateway_borrow_token_test.go +++ /dev/null @@ -1,65 +0,0 @@ -//go:build unit - -package service - -import ( - "context" - "testing" - "time" - - "github.com/stretchr/testify/require" -) - -type borrowTokenObservationRepository struct { - AccountRepository - errorWrites int -} - -func (r *borrowTokenObservationRepository) SetError(context.Context, int64, string) error { - r.errorWrites++ - return nil -} - -type borrowTokenObservationBlocker struct{ calls int } - -func (b *borrowTokenObservationBlocker) BlockAccountScheduling(*Account, time.Time, string) { - b.calls++ -} - -func (b *borrowTokenObservationBlocker) ClearAccountSchedulingBlock(int64) { - b.calls++ -} - -func TestCodexGatewayBorrowTokenObservationKeepsBusinessProtection(t *testing.T) { - for _, observation := range []bool{false, true} { - name := "ordinary_business" - if observation { - name = "borrow_observation" - } - t.Run(name, func(t *testing.T) { - repo := &borrowTokenObservationRepository{} - blocker := &borrowTokenObservationBlocker{} - provider := NewOpenAITokenProvider(repo, nil, nil) - provider.SetAccountRuntimeBlocker(blocker) - account := &Account{ID: 73, Platform: PlatformOpenAI, Type: AccountTypeOAuth, Status: StatusActive, - Credentials: map[string]any{ - "access_token": "synthetic-expired-token", - "expires_at": time.Now().Add(-time.Hour).Format(time.RFC3339), - }} - ctx := context.Background() - if observation { - ctx = WithCodexGatewayBorrowObservation(ctx) - } - token, err := provider.GetAccessToken(ctx, account) - require.Empty(t, token) - require.ErrorContains(t, err, "refresh_token is missing") - writes := 1 - if observation { - writes = 0 - } - require.Equal(t, writes, repo.errorWrites) - require.Equal(t, writes, blocker.calls) - require.Equal(t, StatusActive, account.Status) - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_transient_test.go b/backend/internal/service/codex_gateway_borrow_transient_test.go deleted file mode 100644 index 9b0900035d10..000000000000 --- a/backend/internal/service/codex_gateway_borrow_transient_test.go +++ /dev/null @@ -1,195 +0,0 @@ -package service - -import ( - "context" - "fmt" - "io" - "net/http" - "net/http/httptest" - "net/url" - "strings" - "sync" - "sync/atomic" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/gin-gonic/gin" - "github.com/stretchr/testify/require" -) - -// Exercise real net/http framing: a response advertises more bytes than it -// delivers. Neither a partial stream nor a completed event followed by a -// transport error may qualify the target. Only a new, complete pair can do so. -func TestCodexGatewayBorrowTransientEOFFreshPairOnNextRequest(t *testing.T) { - for _, tc := range []struct { - name string - shot int32 - completedBeforeEOF bool - cleanEOF bool - }{{"mint", 1, false, false}, {"continue", 2, false, false}, {"completed_then_eof", 2, true, false}, {"clean_eof_without_completion", 1, false, true}, {"unterminated_completion", 2, true, true}} { - t.Run(tc.name, func(t *testing.T) { - var calls atomic.Int32 - var statesMu sync.Mutex - var states []string - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - n := calls.Add(1) - statesMu.Lock() - states = append(states, r.Header.Get("X-Codex-Turn-State")) - statesMu.Unlock() - w.Header().Set("X-Codex-Turn-State", "probe-state") - w.Header().Set("Content-Type", "text/event-stream") - body := borrowCoreSSE("gpt-6.1-sol", "OK") - if n == tc.shot { - if !tc.completedBeforeEOF { - body = "data: {\"type\":\"response.created\"}\n\n" - } - if tc.cleanEOF && tc.completedBeforeEOF { - body = strings.TrimSuffix(body, "\n\n") - } - if !tc.cleanEOF { - w.Header().Set("Content-Length", fmt.Sprint(len(body)+100)) - } - } - _, _ = io.WriteString(w, body) - })) - defer server.Close() - targetURL, err := url.Parse(server.URL) - require.NoError(t, err) - account := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(r *http.Request, _ string, id int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - if id != 2 { - return nil, fmt.Errorf("unexpected source acquisition") - } - wire := r.Clone(r.Context()) - wire.URL = targetURL - return server.Client().Do(wire) - }, account) - expires := time.Now().Add(codexGatewayBorrowTTL) - borrowCoreCandidate(s, expires) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - req.Header.Set("X-Codex-Turn-State", "business-state") - _, app, err := s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.Nil(t, app) - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Equal(t, "target_probe_failed", failure.Reason) - if !tc.cleanEOF { - require.Contains(t, failure.Detail, "unexpected EOF") - } - require.False(t, failure.CacheHit) - require.Nil(t, failure.RetryAfter) - require.True(t, failure.Verification.RetryOnNextRequest) - expectedKind := "transport" - if tc.cleanEOF { - expectedKind = "stream_incomplete" - } - require.Equal(t, expectedKind, failure.Verification.FailureKind) - expectedShot := "mint" - if tc.shot == 2 { - expectedShot = "continue" - } - require.Equal(t, expectedShot, failure.Verification.FailedShot) - require.Equal(t, tc.shot, calls.Load(), "the failed request never replays an uncertain probe") - status := s.Status().Targets[1] - require.Equal(t, "retry_pending", status.State) - require.False(t, status.CacheValid) - require.False(t, status.Success) - require.Nil(t, status.RetryAfter) - - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - RecordCodexGatewayBorrowPreparationFailure(c, account, failure) - events, exists := c.Get(OpsUpstreamErrorsKey) - require.True(t, exists) - records, ok := events.([]*OpsUpstreamErrorEvent) - require.True(t, ok) - require.Len(t, records, 1) - evidence := records[0].BorrowQualification - require.NotNil(t, evidence) - require.False(t, evidence.CacheHit) - require.Equal(t, expectedShot, evidence.Verification.FailedShot) - - // No sleep, manual Verify, STATE deletion or cached-proof override. - wire, app, err := s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.True(t, app.Applied) - require.True(t, app.Verification.MintCompleted) - require.True(t, app.Verification.ContinueCompleted) - require.False(t, app.Verification.RetryOnNextRequest) - require.Equal(t, expires, app.ExpiresAt, "recovery does not extend the lease") - require.Equal(t, tc.shot+2, calls.Load(), "both new shots are required") - statesMu.Lock() - lastStates := append([]string(nil), states[len(states)-2:]...) - statesMu.Unlock() - require.Equal(t, []string{"", "probe-state"}, lastStates) - require.Equal(t, "business-state", wire.Header.Get("X-Codex-Turn-State")) - require.Equal(t, "business-state", req.Header.Get("X-Codex-Turn-State")) - require.True(t, s.Status().Targets[1].CacheValid) - _, _, err = s.Apply(req, account, "gpt-6.1-sol", "", nil, false) - require.NoError(t, err) - require.Equal(t, tc.shot+2, calls.Load(), "a fully qualified proof remains reusable") - }) - } -} - -func TestCodexGatewayBorrowTransientRecoveryStillRejectsChangedState(t *testing.T) { - var calls int - a := borrowCoreAccount(2) - s := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - if calls == 1 { - return nil, io.ErrUnexpectedEOF - } - state := "first" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "changed" - } - return borrowCoreResponse("gpt-6.1-sol", "OK", state), nil - }, a) - borrowCoreCandidate(s, time.Now().Add(codexGatewayBorrowTTL)) - _, req, _, err := s.accountTemplate(context.Background(), 2, "gpt-6.1-sol") - require.NoError(t, err) - _, app, err := s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.Error(t, err) - require.Nil(t, app) - _, app, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - var failure *CodexGatewayBorrowFailure - require.ErrorAs(t, err, &failure) - require.Nil(t, app) - require.Equal(t, "target_state_changed", failure.Reason) - require.False(t, failure.Verification.RetryOnNextRequest) - require.NotNil(t, failure.RetryAfter) - require.False(t, failure.CacheHit) - _, app, err = s.Apply(req, a, "gpt-6.1-sol", "", nil, false) - require.ErrorAs(t, err, &failure) - require.Nil(t, app) - require.True(t, failure.CacheHit) - require.Equal(t, 3, calls, "a real rejection still cools down without new probes") - require.Equal(t, "rejected", s.Status().Targets[1].State) -} - -func TestCodexGatewayBorrowTransientClassificationNeverGrantsSuccess(t *testing.T) { - for _, tc := range []struct { - name string - err error - cancelled bool - retry bool - }{ - {"eof", io.ErrUnexpectedEOF, false, true}, - {"caller_cancelled", io.ErrUnexpectedEOF, true, false}, - {"protocol_incomplete", nil, false, false}, - } { - t.Run(tc.name, func(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - if tc.cancelled { - cancel() - } - var result CodexGatewayBorrowVerification - borrowTargetObservationFailure(ctx, &result, "mint", codexGatewayBorrowObservation{status: 200, raw: strings.Repeat(" ", 3)}, tc.err) - require.False(t, result.Success) - require.Equal(t, tc.retry, result.RetryOnNextRequest) - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_transport.go b/backend/internal/service/codex_gateway_borrow_transport.go deleted file mode 100644 index 1588ab08da90..000000000000 --- a/backend/internal/service/codex_gateway_borrow_transport.go +++ /dev/null @@ -1,91 +0,0 @@ -package service - -import ( - "crypto/tls" - "fmt" - "net/http" - "net/http/httptrace" - "sync/atomic" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" -) - -// Trace evidence, rather than error text, decides whether another connection is -// safe. An uninstrumented transport or any written HTTP header forbids retry. -// This wrapper is used only by isolated source/target observations. -type codexBorrowConnectionTrace struct { - wrote atomic.Bool - failedBeforeHTTP atomic.Bool - phase atomic.Int32 -} - -func (t *codexBorrowConnectionTrace) trace() *httptrace.ClientTrace { - return &httptrace.ClientTrace{ - ConnectStart: func(_, _ string) { t.phase.Store(1); t.failedBeforeHTTP.Store(false) }, - ConnectDone: func(_, _ string, err error) { t.failedBeforeHTTP.Store(err != nil) }, - TLSHandshakeStart: func() { t.phase.Store(2); t.failedBeforeHTTP.Store(false) }, - TLSHandshakeDone: func(_ tls.ConnectionState, err error) { t.failedBeforeHTTP.Store(err != nil) }, - GotConn: func(httptrace.GotConnInfo) { t.phase.Store(3); t.failedBeforeHTTP.Store(false) }, - WroteHeaders: func() { t.wrote.Store(true); t.phase.Store(4) }, - WroteRequest: func(httptrace.WroteRequestInfo) { t.wrote.Store(true); t.phase.Store(4) }, - GotFirstResponseByte: func() { t.wrote.Store(true); t.phase.Store(5) }, - } -} - -func (t *codexBorrowConnectionTrace) name() string { - switch t.phase.Load() { - case 1: - return "connect" - case 2: - return "tls_handshake" - case 3: - return "connected" - case 4: - return "request_write" - case 5: - return "response_headers" - default: - return "unobserved" - } -} - -type codexBorrowTransportError struct { - phase string - attempts int - cause error -} - -func (e *codexBorrowTransportError) Error() string { - return fmt.Sprintf("borrow transport %s after %d attempt(s): %s", e.phase, e.attempts, e.cause) -} -func (e *codexBorrowTransportError) Unwrap() error { return e.cause } - -func (s *CodexGatewayBorrowService) sendBorrowObservationHTTP(req *http.Request, proxy string, account *Account, profile *tlsfingerprint.Profile) (*http.Response, string, int, error) { - current := req - for attempt := 1; ; attempt++ { - trace := &codexBorrowConnectionTrace{} - current = current.WithContext(httptrace.WithClientTrace(req.Context(), trace.trace())) - resp, err := s.upstream.DoWithTLS(current, proxy, account.ID, account.Concurrency, profile) - if err == nil { - return resp, trace.name(), attempt, nil - } - if resp != nil && resp.Body != nil { - _ = resp.Body.Close() - } - retry := attempt == 1 && resp == nil && trace.failedBeforeHTTP.Load() && !trace.wrote.Load() && req.GetBody != nil && req.Context().Err() == nil - if !retry { - return nil, trace.name(), attempt, &codexBorrowTransportError{trace.name(), attempt, err} - } - body, bodyErr := req.GetBody() - if bodyErr != nil { - return nil, trace.name(), attempt, &codexBorrowTransportError{trace.name(), attempt, err} - } - // Preparation and diagnostic accounting includes every physical attempt. - if budgetErr := consumeBorrowDiagnosticRequest(req.Context()); budgetErr != nil { - _ = body.Close() - return nil, trace.name(), attempt, budgetErr - } - current = req.Clone(req.Context()) - current.Body = body - } -} diff --git a/backend/internal/service/codex_gateway_borrow_transport_test.go b/backend/internal/service/codex_gateway_borrow_transport_test.go deleted file mode 100644 index 5db42a0ed4da..000000000000 --- a/backend/internal/service/codex_gateway_borrow_transport_test.go +++ /dev/null @@ -1,75 +0,0 @@ -package service - -import ( - "context" - "crypto/tls" - "errors" - "net/http" - "net/http/httptrace" - "strings" - "sync/atomic" - "testing" - - "github.com/Wei-Shaw/sub2api/internal/pkg/tlsfingerprint" - "github.com/stretchr/testify/require" -) - -func TestCodexGatewayBorrowTransportRetryRequiresPreHTTPProof(t *testing.T) { - for _, kind := range []string{"tls", "connect", "unobserved", "headers_written", "response_received", "budget_exhausted", "cancelled"} { - t.Run(kind, func(t *testing.T) { - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - var count atomic.Int32 - count.Store(1) - limit := int32(2) - if kind == "budget_exhausted" { - limit = 1 - } - ctx = context.WithValue(ctx, codexBorrowDiagnosticContextKey{}, &codexBorrowDiagnostic{requests: &count, limit: limit}) - calls := 0 - service := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - calls++ - require.Equal(t, "identical payload", string(borrowCoreBody(t, req))) - if calls == 2 { - return borrowCoreResponse("gpt-6-astra", "OK", ""), nil - } - trace := httptrace.ContextClientTrace(req.Context()) - switch kind { - case "unobserved": - case "connect": - trace.ConnectStart("tcp", "synthetic") - trace.ConnectDone("tcp", "synthetic", errors.New("connect failed")) - default: - trace.TLSHandshakeStart() - trace.TLSHandshakeDone(tls.ConnectionState{}, errors.New("handshake failed")) - if kind == "headers_written" { - trace.WroteHeaders() - } - if kind == "cancelled" { - cancel() - } - } - if kind == "response_received" { - return borrowCoreResponse("gpt-6-astra", "OK", ""), errors.New("late error") - } - return nil, errors.New("synthetic transport failure") - }) - req, _ := http.NewRequestWithContext(ctx, http.MethodPost, chatgptCodexURL, strings.NewReader("identical payload")) - resp, phase, attempts, err := service.sendBorrowObservationHTTP(req, "", borrowCoreAccount(2), nil) - if kind == "tls" || kind == "connect" { - require.NoError(t, err) - require.Equal(t, 2, calls) - require.Equal(t, 2, attempts) - require.Equal(t, int32(2), count.Load()) - _ = resp.Body.Close() - } else { - require.Error(t, err) - require.Equal(t, 1, calls) - require.Equal(t, 1, attempts) - } - if kind == "headers_written" { - require.Equal(t, "request_write", phase) - } - }) - } -} diff --git a/backend/internal/service/codex_gateway_borrow_usage.go b/backend/internal/service/codex_gateway_borrow_usage.go deleted file mode 100644 index 54420b63ea70..000000000000 --- a/backend/internal/service/codex_gateway_borrow_usage.go +++ /dev/null @@ -1,314 +0,0 @@ -package service - -import ( - "context" - "errors" - "io" - "net/http" - "slices" - "strings" - "sync" - "time" - - "github.com/Wei-Shaw/sub2api/internal/pkg/ctxkey" - "github.com/tidwall/gjson" -) - -type CodexGatewayBorrowUsage struct { - Recovery *CodexGatewayBorrowRecovery `json:"recovery,omitempty"` - ProxyID int64 `json:"proxy_id,omitempty"` - NodeIdentity string `json:"node_identity,omitempty"` - Gateway string `json:"gateway,omitempty"` - Dispatched bool `json:"dispatched"` - FailureStage string `json:"failure_stage,omitempty"` - ClientRequestID string `json:"client_request_id,omitempty"` - GatewayRequestID string `json:"gateway_request_id,omitempty"` - AttemptCount uint64 `json:"attempt_count"` - BlockedCount uint64 `json:"blocked_count"` - AccountID int64 `json:"account_id"` - Model string `json:"model"` - Transport string `json:"transport"` - Origin string `json:"origin"` - Applied bool `json:"applied"` - Reason string `json:"reason"` - StartedAt time.Time `json:"started_at"` - FinishedAt *time.Time `json:"finished_at,omitempty"` - Outcome string `json:"outcome"` - RequestID string `json:"request_id,omitempty"` - ReportedModel string `json:"reported_model,omitempty"` - Count uint64 `json:"count"` - AppliedCount uint64 `json:"applied_count"` -} - -type codexBorrowUsageKey struct { - account int64 - model, transport, origin string -} - -// Only configured pairs are retained, one last observation per protocol/purpose. -// Payloads, credentials and routing cookies are never copied into this index. -func (s *CodexGatewayBorrowService) beginUsage(ctx context.Context, accountID int64, model, transport string, applied bool) *codexBorrowUsageTracker { - t := codexBorrowUsageFromContext(ctx) - if t == nil || t.service != s || t.key.account != accountID || t.key.model != model || t.key.transport != transport || !t.pendingDispatch() { - t = s.beginAttempt(ctx, accountID, model, transport) - } - t.dispatched(applied) - return t -} - -func (t *codexBorrowUsageTracker) pendingDispatch() bool { - t.mu.Lock() - defer t.mu.Unlock() - return !t.finished && !t.row.Dispatched -} - -type codexBorrowUsageContextKey struct{} - -func codexBorrowUsageFromContext(ctx context.Context) *codexBorrowUsageTracker { - if ctx == nil { - return nil - } - t, _ := ctx.Value(codexBorrowUsageContextKey{}).(*codexBorrowUsageTracker) - return t -} - -func (s *CodexGatewayBorrowService) beginAttempt(ctx context.Context, accountID int64, model, transport string) *codexBorrowUsageTracker { - if s == nil { - return nil - } - s.mu.Lock() - defer s.mu.Unlock() - if !slices.Contains(s.config.TargetAccountIDs, accountID) || !slices.Contains(s.config.Models, model) { - return nil - } - origin := "business" - if IsPelicanGeneration(ctx) || borrowDiagnosticFromContext(ctx) != nil { - origin = "diagnostic" - } - key := codexBorrowUsageKey{accountID, model, transport, origin} - previous := s.usage[key] - row := CodexGatewayBorrowUsage{AccountID: accountID, Model: model, Transport: transport, Origin: origin, StartedAt: time.Now(), Outcome: "preparing", Reason: "preparing", Count: previous.Count, AppliedCount: previous.AppliedCount, AttemptCount: previous.AttemptCount + 1, BlockedCount: previous.BlockedCount} - row.ClientRequestID, _ = ctx.Value(ctxkey.ClientRequestID).(string) - row.GatewayRequestID, _ = ctx.Value(ctxkey.RequestID).(string) - if !s.config.Enabled { - row.Reason = "disabled" - } - if s.usage == nil { - s.usage = make(map[codexBorrowUsageKey]CodexGatewayBorrowUsage) - } - s.usage[key] = row - return &codexBorrowUsageTracker{service: s, key: key, row: row, revision: s.revision} -} - -// Counter changes apply even when a newer attempt owns the displayed row. A -// late finish may not overwrite that newer request or roll its counters back. -func (t *codexBorrowUsageTracker) publish(sent, applied, blocked uint64) { - t.service.mu.Lock() - defer t.service.mu.Unlock() - current, ok := t.service.usage[t.key] - if !ok || t.service.revision != t.revision { - return - } - current.Count += sent - current.AppliedCount += applied - current.BlockedCount += blocked - if current.AttemptCount == t.row.AttemptCount { - t.row.Count, t.row.AppliedCount, t.row.BlockedCount = current.Count, current.AppliedCount, current.BlockedCount - current = t.row - } - t.service.usage[t.key] = current -} - -func (t *codexBorrowUsageTracker) dispatched(applied bool) { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - if t.finished || t.row.Dispatched { - return - } - t.row.Dispatched, t.row.Applied, t.row.Outcome = true, applied, "sent" - if t.row.Reason != "disabled" { - t.row.Reason = "not_applied" - } - var appliedCount uint64 - if applied { - t.row.Reason = "borrow_applied" - appliedCount = 1 - } - t.publish(1, appliedCount, 0) -} - -func (t *codexBorrowUsageTracker) blocked(err error) { - if t == nil { - return - } - t.mu.Lock() - if t.finished || t.row.Dispatched { - t.mu.Unlock() - return - } - t.row.FailureStage, t.row.Reason = codexBorrowFailureStage(err) - t.row.Outcome, t.terminal = "blocked", true - if t.row.Recovery != nil { - copy := *t.row.Recovery - copy.Phase, copy.NextAttemptAt = "stopped", nil - if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { - copy.Phase, t.row.Outcome = "cancelled", "cancelled" - } - t.row.Recovery = © - } - // Mark finished while holding the tracker lock so repeated error owners - // cannot count the same preparation rejection twice. - t.finished = true - now := time.Now() - t.row.FinishedAt = &now - t.publish(0, 0, 1) - t.mu.Unlock() -} - -type codexBorrowUsageTracker struct { - mu sync.Mutex - terminal bool - finished bool - revision uint64 - service *CodexGatewayBorrowService - key codexBorrowUsageKey - row CodexGatewayBorrowUsage - once sync.Once -} - -func (t *codexBorrowUsageTracker) observe(message []byte) { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - if t.finished || t.terminal { - return - } - typeName := gjson.GetBytes(message, "type").String() - if id := gjson.GetBytes(message, "response.id").String(); id != "" { - t.row.RequestID = id - } - if model := gjson.GetBytes(message, "response.model").String(); model != "" { - t.row.ReportedModel = model - } - switch typeName { - case "response.completed", "response.done": - t.terminal = true - if gjson.GetBytes(message, "response.status").String() == "completed" && !gjson.GetBytes(message, "response.error").IsObject() { - t.row.Outcome = "completed" - } else if gjson.GetBytes(message, "response.status").String() != "" || gjson.GetBytes(message, "response.error").IsObject() { - t.row.Outcome = "upstream_error" - } else { - t.row.Outcome = "unobserved" - } - case "error", "response.failed", "response.incomplete", "response.cancelled", "response.canceled": - t.terminal = true - t.row.Outcome = "upstream_error" - } -} - -func (t *codexBorrowUsageTracker) finish(err error) { - if t == nil { - return - } - t.once.Do(func() { - t.mu.Lock() - defer t.mu.Unlock() - if t.finished { - return - } - if !t.terminal { - if err != nil { - t.row.Outcome = "transport_error" - } else { - t.row.Outcome = "incomplete" - } - } - now := time.Now() - t.row.FinishedAt = &now - t.finished = true - t.publish(0, 0, 0) - }) -} - -type codexBorrowUsageBody struct { - mu sync.Mutex - io.ReadCloser - tracker *codexBorrowUsageTracker - line []byte - err error -} - -func (b *codexBorrowUsageBody) Read(p []byte) (int, error) { - n, err := b.ReadCloser.Read(p) - b.mu.Lock() - defer b.mu.Unlock() - for _, ch := range p[:n] { - if ch == '\n' { - if strings.HasPrefix(string(b.line), "data:") { - b.tracker.observe(b.line[5:]) - } - b.line = b.line[:0] - } else if len(b.line) < 1<<20 { - b.line = append(b.line, ch) - } - } - if err != nil { - if len(b.line) > 5 && strings.HasPrefix(string(b.line), "data:") { - b.tracker.observe(b.line[5:]) - b.line = nil - } - if err != io.EOF { - b.err = err - } - b.tracker.finish(b.err) - } - return n, err -} - -func (b *codexBorrowUsageBody) Close() error { - err := b.ReadCloser.Close() - b.mu.Lock() - defer b.mu.Unlock() - if b.err == nil { - b.err = err - } - b.tracker.finish(b.err) - return err -} - -func (s *CodexGatewayBorrowService) trackHTTPResponse(t *codexBorrowUsageTracker, resp *http.Response, err error) (*http.Response, error) { - if t == nil { - return resp, err - } - if resp == nil { - t.finish(err) - return resp, err - } - t.mu.Lock() - t.row.RequestID = resp.Header.Get("X-Request-ID") - if resp.StatusCode >= 400 { - t.row.Outcome = "upstream_error" - t.terminal = true - } - t.mu.Unlock() - if err != nil || resp.Body == nil { - t.finish(err) - } else { - resp.Body = &codexBorrowUsageBody{ReadCloser: resp.Body, tracker: t} - } - return resp, err -} - -func (t *codexBorrowUsageTracker) terminalObserved() bool { - if t == nil { - return false - } - t.mu.Lock() - defer t.mu.Unlock() - return t.terminal -} diff --git a/backend/internal/service/codex_gateway_borrow_usage_ws.go b/backend/internal/service/codex_gateway_borrow_usage_ws.go deleted file mode 100644 index 2d62505da473..000000000000 --- a/backend/internal/service/codex_gateway_borrow_usage_ws.go +++ /dev/null @@ -1,62 +0,0 @@ -package service - -import ( - "context" - "sync" - - openaiwsv2 "github.com/Wei-Shaw/sub2api/internal/service/openai_ws_v2" - coderws "github.com/coder/websocket" - "github.com/tidwall/gjson" -) - -// Wrap the final native WS relay, after policy rewriting. Handshakes and -// blocked client frames never enter this observer. -type codexBorrowUsageFrameConn struct { - openaiwsv2.FrameConn - service *CodexGatewayBorrowService - accountID int64 - applied bool - mu sync.Mutex - current *codexBorrowUsageTracker - prepared *codexBorrowUsageTracker -} - -func (c *codexBorrowUsageFrameConn) WriteFrame(ctx context.Context, kind coderws.MessageType, payload []byte) error { - c.mu.Lock() - if gjson.GetBytes(payload, "type").String() == "response.create" { - c.current.finish(nil) - if c.prepared != nil { - ctx = context.WithValue(ctx, codexBorrowUsageContextKey{}, c.prepared) - c.prepared = nil - } - c.current = c.service.beginUsage(ctx, c.accountID, gjson.GetBytes(payload, "model").String(), "ws", c.applied) - } - tracker := c.current - c.mu.Unlock() - err := c.FrameConn.WriteFrame(ctx, kind, payload) - if err != nil { - c.mu.Lock() - tracker.finish(err) - c.mu.Unlock() - } - return err -} - -func (c *codexBorrowUsageFrameConn) ReadFrame(ctx context.Context) (coderws.MessageType, []byte, error) { - kind, payload, err := c.FrameConn.ReadFrame(ctx) - c.mu.Lock() - c.current.observe(payload) - if err != nil { - c.current.finish(err) - } else if c.current.terminalObserved() { - c.current.finish(nil) - } - c.mu.Unlock() - return kind, payload, err -} - -func (c *codexBorrowUsageFrameConn) finish() { - c.mu.Lock() - defer c.mu.Unlock() - c.current.finish(nil) -} diff --git a/backend/internal/service/codex_gateway_borrow_ws_anchor.go b/backend/internal/service/codex_gateway_borrow_ws_anchor.go deleted file mode 100644 index 3d619b8ff1b7..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ws_anchor.go +++ /dev/null @@ -1,418 +0,0 @@ -package service - -import ( - "context" - "crypto/sha256" - "errors" - "fmt" - "net/http" - "net/url" - "slices" - "strings" - "sync" - "time" - - "github.com/gin-gonic/gin" - "github.com/tidwall/gjson" -) - -const ( - codexGatewayBorrowWSAnchorTTL = time.Hour - codexGatewayBorrowWSAnchorLimit = 1024 -) - -type codexGatewayBorrowWSAnchorKey struct { - account, apiKey, group int64 - model, scope string - identity [32]byte -} - -type codexGatewayBorrowWSAnchorEntry struct { - responseID, connID string - expires time.Time -} - -type codexGatewayBorrowWSAnchorStore struct { - mu sync.Mutex - entries map[codexGatewayBorrowWSAnchorKey]codexGatewayBorrowWSAnchorEntry - busy map[codexGatewayBorrowWSAnchorKey]bool -} - -type codexGatewayBorrowWSTurn struct { - usage *codexBorrowUsageTracker - store *codexGatewayBorrowWSAnchorStore - key codexGatewayBorrowWSAnchorKey - previousID, connID, scope, model string - expires time.Time - headers http.Header - qualified bool - finished sync.Once - pelicanOneShot bool - diagnostic bool - diagnosticApplied bool -} - -// Passthrough owns a direct socket for the entire client session. Its anchor is -// local to that socket and can never be resumed by another direct dial. -type codexGatewayBorrowWSDirectSession struct { - turn *codexGatewayBorrowWSTurn - mu sync.Mutex - responseID string - valid bool -} - -func (session *codexGatewayBorrowWSDirectSession) check(c *gin.Context, account *Account, headers http.Header, rawClientBody []byte, model, proxyURL string) error { - if session == nil || session.turn == nil { - return nil - } - key, err := codexGatewayBorrowWSKey(c, account, headers, rawClientBody, model, proxyURL) - if err != nil || key != session.turn.key || !time.Now().Before(session.turn.expires) { - return NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - previous := strings.TrimSpace(gjson.GetBytes(rawClientBody, "previous_response_id").String()) - session.mu.Lock() - defer session.mu.Unlock() - if previous != "" && (!session.valid || previous != session.responseID) { - return NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - return nil -} - -func (session *codexGatewayBorrowWSDirectSession) completed(result *OpenAIForwardResult) { - if session == nil || session.turn == nil { - return - } - session.mu.Lock() - defer session.mu.Unlock() - session.valid = result != nil && isOpenAIWSSuccessTerminalEvent(result.UpstreamTerminalEvent) && - result.UpstreamResponseModel == session.turn.model && !result.UpstreamResponseModelConflict && result.RequestID != "" - if session.valid { - session.responseID = result.RequestID - } else { - session.responseID = "" - } -} - -func (session *codexGatewayBorrowWSDirectSession) close(pool *openAIWSConnPool) { - if session == nil || session.turn == nil { - return - } - turn := session.turn - turn.finished.Do(func() { - session.mu.Lock() - responseID, valid := session.responseID, session.valid && time.Now().Before(turn.expires) - session.mu.Unlock() - store := turn.store - store.mu.Lock() - delete(store.busy, turn.key) - previous := store.entries[turn.key] - if valid { - // Empty connID records a closed direct socket. The same bounded - // anchor budget rejects reconnects without retaining that socket. - store.entries[turn.key] = codexGatewayBorrowWSAnchorEntry{responseID: responseID, expires: turn.expires} - } - store.mu.Unlock() - if valid && previous.connID != "" && pool != nil { - pool.evictConn(turn.key.account, previous.connID) - } - }) -} - -// The HTTP template is only for eligibility and qualification. The real dial -// retains the existing WebSocket URL, account credentials and account proxy. -func codexGatewayBorrowWSRequest(ctx context.Context, wsURL string, headers http.Header) (*http.Request, error) { - u, err := url.Parse(wsURL) - if err != nil { - return nil, err - } - switch u.Scheme { - case "wss": - u.Scheme = "https" - case "ws": - u.Scheme = "http" - } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, u.String(), nil) - if err != nil { - return nil, err - } - req.Header = cloneHeader(headers) - return req, nil -} - -func (s *OpenAIGatewayService) codexGatewayBorrowWSSelected(account *Account, model string, req *http.Request) bool { - if s == nil || s.gatewayBorrow == nil || account == nil || !account.IsOpenAIOAuthLike() || - !s.gatewayBorrow.ModelEligible(model) || !s.gatewayBorrow.RequestEligible(req) { - return false - } - settings := s.gatewayBorrow.ConfigSnapshot() - return settings.Enabled && slices.Contains(settings.TargetAccountIDs, account.ID) -} - -func codexGatewayBorrowWSKey(c *gin.Context, account *Account, headers http.Header, rawClientBody []byte, model, proxyURL string) (codexGatewayBorrowWSAnchorKey, error) { - apiKeyID := getAPIKeyIDFromContext(c) - scope, _ := resolveOpenAIWSExecutionScope(c, rawClientBody, apiKeyID) - if apiKeyID <= 0 || scope == "" { - return codexGatewayBorrowWSAnchorKey{}, errors.New("codex gateway borrow websocket requires an authenticated API key and explicit session or thread identity") - } - // Turn state and cookie generations deliberately do not enter the key: a - // response continuation remains on its original authenticated connection. - identity := sha256.Sum256([]byte(headers.Get("Authorization") + "\x00" + - headers.Get("Chatgpt-Account-Id") + "\x00" + account.GetCredential("access_token") + "\x00" + proxyURL)) - return codexGatewayBorrowWSAnchorKey{account: account.ID, apiKey: apiKeyID, group: getOpenAIGroupIDFromContext(c), - model: strings.TrimSpace(model), scope: scope, identity: identity}, nil -} - -func (store *codexGatewayBorrowWSAnchorStore) begin(key codexGatewayBorrowWSAnchorKey, previousID string, now time.Time) (*codexGatewayBorrowWSTurn, error) { - store.mu.Lock() - defer store.mu.Unlock() - if store.entries == nil { - store.entries = make(map[codexGatewayBorrowWSAnchorKey]codexGatewayBorrowWSAnchorEntry) - store.busy = make(map[codexGatewayBorrowWSAnchorKey]bool) - } - for k, entry := range store.entries { - if !now.Before(entry.expires) && !store.busy[k] { - // Expiring an anchor never clears an account pool or closes a socket. - delete(store.entries, k) - } - } - if store.busy[key] { - return nil, errors.New("codex gateway borrow websocket session already has an active request") - } - entry, found := store.entries[key] - previousID = strings.TrimSpace(previousID) - if previousID != "" && (!found || entry.responseID != previousID || entry.connID == "" || !now.Before(entry.expires)) { - return nil, NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - if !found { - occupied := len(store.entries) - for busyKey := range store.busy { - if _, exists := store.entries[busyKey]; !exists { - occupied++ - } - } - if occupied >= codexGatewayBorrowWSAnchorLimit { - return nil, errors.New("codex gateway borrow websocket anchor capacity reached") - } - } - turn := &codexGatewayBorrowWSTurn{store: store, key: key, previousID: previousID, model: key.model, - scope: fmt.Sprintf("%d:%d:%d:%s:%s:%x", key.account, key.apiKey, key.group, key.model, key.scope, key.identity)} - if previousID != "" { - turn.connID = entry.connID - turn.expires = entry.expires - } else { - turn.expires = now.Add(codexGatewayBorrowWSAnchorTTL) - } - store.busy[key] = true - return turn, nil -} - -func (store *codexGatewayBorrowWSAnchorStore) knowsContinuation(key codexGatewayBorrowWSAnchorKey, previousID string) (exact, knownResponse bool) { - store.mu.Lock() - defer store.mu.Unlock() - _, exact = store.entries[key] - if previousID != "" { - for storedKey, entry := range store.entries { - if storedKey.account == key.account && entry.responseID == previousID { - knownResponse = true - break - } - } - } - return exact, knownResponse -} - -func (s *OpenAIGatewayService) prepareCodexGatewayBorrowWSTurn(ctx context.Context, c *gin.Context, account *Account, - wsURL string, headers http.Header, rawClientBody []byte, model, previousID, proxyURL string, executionScope ...string) (preparedTurn *codexGatewayBorrowWSTurn, preparationErr error) { - if s == nil || s.gatewayBorrow == nil || account == nil { - return nil, nil - } - borrowConfig := s.gatewayBorrow.ConfigSnapshot() - diagnosticMode := borrowDiagnosticFromContext(ctx) - if borrowConfig.Enabled && borrowConfig.RotateNodes && slices.Contains(borrowConfig.TargetAccountIDs, account.ID) && slices.Contains(borrowConfig.Models, model) && (diagnosticMode == nil || diagnosticMode.borrow) { - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowUnavailable, Stage: "node_selection", Reason: "node_selection_http_only"} - } - req, err := codexGatewayBorrowWSRequest(ctx, wsURL, headers) - if err != nil { - return nil, err - } - selected := s.codexGatewayBorrowWSSelected(account, model, req) - diagnostic := borrowDiagnosticFromContext(ctx) - if selected && (diagnostic == nil || diagnostic.borrow) { - tracker := s.gatewayBorrow.beginAttempt(ctx, account.ID, model, "ws") - ctx = context.WithValue(ctx, codexBorrowUsageContextKey{}, tracker) - req = req.WithContext(ctx) - defer func() { - if preparationErr != nil { - tracker.blocked(preparationErr) - } else if preparedTurn != nil { - preparedTurn.usage = tracker - } - }() - } - if diagnostic != nil { - return diagnostic.prepareWS(ctx, s.gatewayBorrow, req, account, model, previousID, proxyURL) - } - if IsPelicanGeneration(ctx) { - if !selected { - return nil, nil - } - if previousID != "" { - return nil, errors.New("pelican generation starts a new conversation") - } - // A trusted administrator invocation has no customer's API key or - // continuation identity. It borrows through the normal preparation path - // but never creates a persistent client continuation anchor. - borrowed, application, applyErr := s.gatewayBorrow.Apply(req, account, model, proxyURL, nil, false) - if applyErr != nil { - return nil, applyErr - } - if borrowed == nil || application == nil || !application.Applied { - return nil, &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowChanged} - } - return &codexGatewayBorrowWSTurn{key: codexGatewayBorrowWSAnchorKey{account: account.ID, model: model}, - model: model, headers: cloneHeader(borrowed.Header), expires: application.ExpiresAt, pelicanOneShot: true}, nil - } - key, err := codexGatewayBorrowWSKey(c, account, headers, rawClientBody, model, proxyURL) - if err != nil { - if !selected { - _, knownResponse := s.gatewayBorrow.wsAnchors.knowsContinuation(codexGatewayBorrowWSAnchorKey{account: account.ID}, previousID) - if knownResponse { - return nil, NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - return nil, nil - } - return nil, err - } - if len(executionScope) > 0 && strings.TrimSpace(executionScope[0]) != "" { - key.scope = strings.TrimSpace(executionScope[0]) - } - // Existing conversations survive a setting edit or disable. Only new roots - // require the current configuration and a current shared cookie. - knownAnchor := false - if previousID != "" { - var knownResponse bool - knownAnchor, knownResponse = s.gatewayBorrow.wsAnchors.knowsContinuation(key, previousID) - if !knownAnchor && knownResponse { - // A model, credential or authenticated client change cannot convert - // an existing borrowed response into an ordinary reconnect. - return nil, NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - } - if !selected && !knownAnchor { - return nil, nil - } - turn, err := s.gatewayBorrow.wsAnchors.begin(key, previousID, time.Now()) - if err != nil { - return nil, err - } - if turn.previousID != "" { - // No cookie reacquisition and no source or target probe on continuation. - turn.headers = cloneHeader(headers) - return turn, nil - } - borrowedReq, application, err := s.gatewayBorrow.Apply(req, account, model, proxyURL, nil, false) - if err != nil { - turn.finish(s.getOpenAIWSConnPool(), nil, err) - return nil, err - } - if application == nil || !application.Applied || borrowedReq == nil { - failure := &CodexGatewayBorrowFailure{Cause: ErrCodexGatewayBorrowChanged} - turn.finish(s.getOpenAIWSConnPool(), nil, failure) - return nil, failure - } - turn.headers = borrowedReq.Header - return turn, nil -} - -func codexGatewayBorrowWSPreparationError(c *gin.Context, account *Account, err error) error { - if !IsCodexGatewayBorrowFailure(err) { - return err - } - RecordCodexGatewayBorrowPreparationFailure(c, account, err) - return NewCodexGatewayBorrowRequestFailure(err) -} - -func (turn *codexGatewayBorrowWSTurn) applyToAcquire(req *openAIWSAcquireRequest) { - if turn == nil || req == nil { - return - } - req.AnchorScope = turn.scope - req.Headers = cloneHeader(turn.headers) - if turn.previousID == "" { - req.PreferredConnID = "" - req.ForcePreferredConn = false - req.ForceNewConn = true - } else { - req.PreferredConnID = turn.connID - req.ForcePreferredConn = true - req.ForceNewConn = false - } -} - -func (turn *codexGatewayBorrowWSTurn) bindLease(lease *openAIWSConnLease) error { - if turn == nil { - return nil - } - if lease == nil || lease.conn == nil || (turn.previousID != "" && turn.connID != lease.ConnID()) { - return NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - turn.connID = lease.ConnID() - if turn.pelicanOneShot { - return nil - } - lease.conn.anchorUntilNano.Store(turn.expires.UnixNano()) - return nil -} - -func (turn *codexGatewayBorrowWSTurn) observe(message []byte) { - if turn == nil || gjson.GetBytes(message, "type").String() != "response.completed" { - return - } - response := gjson.GetBytes(message, "response") - turn.qualified = strings.TrimSpace(response.Get("model").String()) == turn.model && - response.Get("status").String() == "completed" && !response.Get("error").IsObject() -} - -func (turn *codexGatewayBorrowWSTurn) finish(pool *openAIWSConnPool, result *OpenAIForwardResult, resultErr error) { - if turn == nil { - return - } - if resultErr != nil { - turn.usage.blocked(resultErr) - } - if turn.pelicanOneShot { - turn.finished.Do(func() { - if pool != nil && turn.connID != "" { - pool.evictConn(turn.key.account, turn.connID) - } - }) - return - } - if turn.store == nil { - return - } - turn.finished.Do(func() { - store := turn.store - store.mu.Lock() - delete(store.busy, turn.key) - previous := store.entries[turn.key] - responseID := "" - if result != nil { - responseID = firstNonEmpty(result.ResponseID, result.RequestID) - } - success := resultErr == nil && result != nil && !result.ClientDisconnect && turn.qualified && responseID != "" && - result.UpstreamResponseModel == turn.model && !result.UpstreamResponseModelConflict && time.Now().Before(turn.expires) - if success { - store.entries[turn.key] = codexGatewayBorrowWSAnchorEntry{responseID: responseID, connID: turn.connID, expires: turn.expires} - } else if turn.previousID != "" { - delete(store.entries, turn.key) - } - store.mu.Unlock() - if success && turn.previousID == "" && previous.connID != "" && previous.connID != turn.connID && pool != nil { - pool.evictConn(turn.key.account, previous.connID) - } - if !success && turn.connID != "" && pool != nil { - pool.evictConn(turn.key.account, turn.connID) - } - }) -} diff --git a/backend/internal/service/codex_gateway_borrow_ws_passthrough_test.go b/backend/internal/service/codex_gateway_borrow_ws_passthrough_test.go deleted file mode 100644 index 1855954e4d84..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ws_passthrough_test.go +++ /dev/null @@ -1,102 +0,0 @@ -package service - -import ( - "context" - "net/http" - "net/http/httptest" - "strings" - "testing" - "time" - - coderws "github.com/coder/websocket" - "github.com/stretchr/testify/require" - "github.com/tidwall/gjson" -) - -func TestCodexGatewayBorrowWS_PassthroughFinalModelAndContinuationUseOneDirectDial(t *testing.T) { - svc, account, borrow, probe := borrowWSFixture(t, 1) - upstream := newStagedPassthroughConn() - dialer := &borrowWSDialer{conns: []openAIWSClientConn{upstream}} - svc.openaiWSPassthroughDialer = dialer - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6.1-sol") - turns := make(chan struct{}, 3) - hooks := &OpenAIWSIngressHooks{ - MapRequestModel: func(int, string) (string, error) { return "gpt-6.1-sol", nil }, - AfterTurn: func(_ int, _ *OpenAIForwardResult, _ error) { turns <- struct{}{} }, - } - serverErr := make(chan error, 1) - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { - client, err := coderws.Accept(w, req, nil) - if err != nil { - serverErr <- err - return - } - defer func() { _ = client.CloseNow() }() - _, first, err := client.Read(context.Background()) - if err != nil { - serverErr <- err - return - } - c := borrowWSContext() - c.Request = req - serverErr <- svc.proxyResponsesWebSocketV2Passthrough(context.Background(), c, client, account, "target-token", first, - hooks, svc.getOpenAIWSProtocolResolver().Resolve(account)) - })) - defer server.Close() - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - client, _, err := coderws.Dial(ctx, "ws"+strings.TrimPrefix(server.URL, "http"), &coderws.DialOptions{HTTPHeader: borrowWSContext().Request.Header}) - require.NoError(t, err) - defer func() { _ = client.CloseNow() }() - require.NoError(t, client.Write(ctx, coderws.MessageText, []byte(`{"type":"response.create","model":"public-sol-alias","input":"hello","store":false}`))) - firstWire := requirePassthroughUpstreamWrite(t, upstream, time.Second) - require.Equal(t, "gpt-6.1-sol", gjson.GetBytes(firstWire, "model").String(), "borrowing is chosen by the final normalized model") - upstream.Send(string(borrowWSCompleted("gpt-6.1-sol", "resp_direct_1"))) - _, _, err = client.Read(ctx) - require.NoError(t, err) - select { - case <-turns: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - borrow.mu.Lock() - borrow.config.Enabled = false - borrow.candidate.expires = time.Now().Add(-time.Second) - borrow.targets = nil - borrow.mu.Unlock() - require.NoError(t, client.Write(ctx, coderws.MessageText, []byte(`{"type":"response.create","model":"public-sol-alias","previous_response_id":"resp_direct_1","input":"next","store":false}`))) - secondWire := requirePassthroughUpstreamWrite(t, upstream, time.Second) - require.Equal(t, "resp_direct_1", gjson.GetBytes(secondWire, "previous_response_id").String()) - upstream.Send(string(borrowWSCompleted("gpt-6.1-sol", "resp_direct_2"))) - _, _, err = client.Read(ctx) - require.NoError(t, err) - select { - case <-turns: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - _ = client.Close(coderws.StatusNormalClosure, "done") - select { - case <-serverErr: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - require.Len(t, dialer.headers, 1) - require.Equal(t, "Bearer target-token", dialer.headers[0].Get("Authorization")) - require.Equal(t, "__oailb=shared-native-route", dialer.headers[0].Get("Cookie")) - require.Equal(t, account.Proxy.URL(), dialer.proxies[0]) - require.Empty(t, probe.requests) - borrow.wsAnchors.mu.Lock() - require.Len(t, borrow.wsAnchors.entries, 1) - for _, entry := range borrow.wsAnchors.entries { - require.Equal(t, "resp_direct_2", entry.responseID) - require.Empty(t, entry.connID) - } - borrow.wsAnchors.mu.Unlock() - reconnectBody := []byte(`{"type":"response.create","model":"public-sol-alias","previous_response_id":"resp_direct_2","input":"reconnect"}`) - err = svc.proxyResponsesWebSocketV2Passthrough(context.Background(), borrowWSContext(), client, account, "target-token", reconnectBody, - hooks, svc.getOpenAIWSProtocolResolver().Resolve(account)) - require.Error(t, err) - require.Len(t, dialer.headers, 1, "a closed direct anchor must not be redialed even after feature disable") - require.Empty(t, probe.requests) -} diff --git a/backend/internal/service/codex_gateway_borrow_ws_phase_test.go b/backend/internal/service/codex_gateway_borrow_ws_phase_test.go deleted file mode 100644 index a6a6aa80324d..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ws_phase_test.go +++ /dev/null @@ -1,108 +0,0 @@ -package service - -import ( - "context" - "errors" - "net/http" - "net/http/httptest" - "strings" - "testing" - "time" - - coderws "github.com/coder/websocket" - "github.com/stretchr/testify/require" -) - -func TestCodexGatewayBorrowWS_PreparationFailureUsesNeutralExistingFailoverBeforeDial(t *testing.T) { - for _, mode := range []string{"http_ws_v2", "native_ctx_pool", "native_passthrough"} { - t.Run(mode, func(t *testing.T) { - svc, account, borrow, probe := borrowWSFixture(t, 1) - dialer := &borrowWSDialer{} - svc.openaiWSPool.setClientDialerForTest(dialer) - svc.openaiWSPassthroughDialer = dialer - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6.1-sol") - for key, check := range borrow.targets { - check.result.Success = false - check.result.Reason = "target_state_changed" - check.retryAfter = time.Now().Add(codexGatewayBorrowFailureWait) - borrow.targets[key] = check - } - var resultErr error - if mode == "http_ws_v2" { - _, resultErr = runBorrowWSV2(t, svc, account, "gpt-6.1-sol", "") - } else { - serverResult := make(chan error, 1) - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { - client, err := coderws.Accept(w, req, nil) - if err != nil { - serverResult <- err - return - } - defer func() { _ = client.CloseNow() }() - _, first, err := client.Read(context.Background()) - if err != nil { - serverResult <- err - return - } - c := borrowWSContext() - c.Request = req - if mode == "native_ctx_pool" { - err = svc.ProxyResponsesWebSocketFromClient(context.Background(), c, client, account, "target-token", first, nil) - } else { - err = svc.proxyResponsesWebSocketV2Passthrough(context.Background(), c, client, account, "target-token", first, nil, - svc.getOpenAIWSProtocolResolver().Resolve(account)) - } - serverResult <- err - })) - defer server.Close() - ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) - defer cancel() - client, _, err := coderws.Dial(ctx, "ws"+strings.TrimPrefix(server.URL, "http"), &coderws.DialOptions{HTTPHeader: borrowWSContext().Request.Header}) - require.NoError(t, err) - defer func() { _ = client.CloseNow() }() - require.NoError(t, client.Write(ctx, coderws.MessageText, []byte(`{"type":"response.create","model":"gpt-6.1-sol","input":"hello","store":false}`))) - select { - case resultErr = <-serverResult: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - } - var failure *UpstreamFailoverError - require.ErrorAs(t, resultErr, &failure) - require.Equal(t, CodexGatewayBorrowPreparationFailureReason, failure.Reason) - require.Equal(t, GatewayFailureScopeRequest, failure.Scope) - require.True(t, failure.SuppressAccountHealthPenalty) - require.True(t, failure.RequestScopedTransient) - require.False(t, failure.ShouldReportAccountScheduleFailure()) - require.True(t, failure.ShouldRetryNextAccount(), "the existing failover owner may switch accounts") - require.False(t, failure.RetryableOnSameAccount) - _, reconnect := classifyOpenAIWSReconnectReason(resultErr) - require.False(t, reconnect, "local qualification must not enter the WS reconnect loop") - require.NotErrorIs(t, resultErr, errOpenAIWSPassthroughRetrySameAccount) - require.Empty(t, dialer.headers, "no business dial occurs for a failed local preparation") - require.Empty(t, probe.requests, "the cached failed qualification is not reprobed") - require.Empty(t, borrow.wsAnchors.busy) - usage := borrow.Status().RecentUsage - require.Len(t, usage, 1) - require.EqualValues(t, 1, usage[0].AttemptCount) - require.EqualValues(t, 1, usage[0].BlockedCount) - require.Zero(t, usage[0].Count) - require.Equal(t, "target_state_changed", usage[0].Reason) - }) - } -} - -func TestCodexGatewayBorrowWS_PreparationDiagnosticsKeepCompleteCause(t *testing.T) { - c := borrowWSContext() - cause := errors.New("dial tcp synthetic-source.invalid:443: connect: connection refused; complete local observation") - err := codexGatewayBorrowWSPreparationError(c, &Account{ID: 96, Platform: PlatformOpenAI}, &CodexGatewayBorrowFailure{Cause: cause}) - require.True(t, IsCodexGatewayBorrowRequestFailure(err)) - rawEvents, exists := c.Get(OpsUpstreamErrorsKey) - require.True(t, exists) - events, ok := rawEvents.([]*OpsUpstreamErrorEvent) - require.True(t, ok) - require.Len(t, events, 1) - require.Contains(t, events[0].Detail, cause.Error()) - ordinary := errors.New("ordinary websocket business dial failure") - require.Same(t, ordinary, codexGatewayBorrowWSPreparationError(c, nil, ordinary)) -} diff --git a/backend/internal/service/codex_gateway_borrow_ws_test.go b/backend/internal/service/codex_gateway_borrow_ws_test.go deleted file mode 100644 index 82e3f9e16908..000000000000 --- a/backend/internal/service/codex_gateway_borrow_ws_test.go +++ /dev/null @@ -1,325 +0,0 @@ -package service - -import ( - "context" - "errors" - "fmt" - "net/http" - "net/http/httptest" - "strings" - "sync" - "testing" - "time" - - "github.com/Wei-Shaw/sub2api/internal/config" - coderws "github.com/coder/websocket" - "github.com/gin-gonic/gin" - "github.com/stretchr/testify/require" -) - -type borrowWSDialer struct { - mu sync.Mutex - conns []openAIWSClientConn - headers []http.Header - proxies []string -} - -func (d *borrowWSDialer) Dial(_ context.Context, _ string, headers http.Header, proxyURL string) (openAIWSClientConn, int, http.Header, error) { - d.mu.Lock() - defer d.mu.Unlock() - d.headers = append(d.headers, cloneHeader(headers)) - d.proxies = append(d.proxies, proxyURL) - if len(d.conns) == 0 { - return nil, 0, nil, errors.New("unexpected additional websocket dial") - } - conn := d.conns[0] - d.conns = d.conns[1:] - return conn, http.StatusSwitchingProtocols, http.Header{}, nil -} - -func borrowWSContext() *gin.Context { - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - c.Request = httptest.NewRequest(http.MethodPost, "/v1/responses", nil) - c.Request.Header.Set("User-Agent", "borrow-ws-test") - c.Request.Header.Set("Session-Id", "borrow-client-session") - groupID := int64(91) - c.Set("api_key", &APIKey{ID: 92, GroupID: &groupID}) - return c -} - -func borrowWSFixture(t *testing.T, maxConns int) (*OpenAIGatewayService, *Account, *CodexGatewayBorrowService, *httpUpstreamRecorder) { - t.Helper() - gin.SetMode(gin.TestMode) - cfg := newOpenAIWSV2TestConfig() - cfg.Gateway.ForceCodexCLI = true - cfg.Gateway.OpenAIWS.MaxConnsPerAccount = maxConns - cfg.Gateway.OpenAIWS.MinIdlePerAccount = 0 - cfg.Gateway.OpenAIWS.MaxIdlePerAccount = maxConns - cfg.Gateway.OpenAIWS.ReadTimeoutSeconds = 3 - cfg.Gateway.OpenAIWS.WriteTimeoutSeconds = 3 - cfg.Security.URLAllowlist.Enabled = false - pool := newOpenAIWSConnPool(cfg) - t.Cleanup(pool.Close) - svc := &OpenAIGatewayService{cfg: cfg, cache: &stubGatewayCache{}, openaiWSPool: pool, - openaiWSResolver: NewOpenAIWSProtocolResolver(cfg), toolCorrector: NewCodexToolCorrector()} - proxyID := int64(93) - account := &Account{ID: 94, Platform: PlatformOpenAI, Type: AccountTypeOAuth, Status: StatusActive, - Schedulable: true, Concurrency: maxConns, ProxyID: &proxyID, - Proxy: &Proxy{ID: proxyID, Protocol: "http", Host: "target-proxy.invalid", Port: 8080}, - Credentials: map[string]any{"access_token": "target-token", "chatgpt_account_id": "target-chatgpt"}, - Extra: map[string]any{"openai_oauth_responses_websockets_v2_enabled": true}} - probe := &httpUpstreamRecorder{err: errors.New("unexpected borrow observation")} - borrow := &CodexGatewayBorrowService{gateway: svc, upstream: probe, loaded: true, revision: 1, revisionCtx: context.Background(), - config: CodexGatewayBorrowConfig{Enabled: true, SourceAccountIDs: []int64{95}, TargetAccountIDs: []int64{account.ID}, Models: []string{"gpt-6-astra", "gpt-6.1-sol"}}, - candidate: &codexGatewayBorrowCandidate{cookie: http.Cookie{Name: "__oailb", Value: "shared-native-route", Path: "/backend-api/codex"}, - expires: time.Now().Add(codexGatewayBorrowTTL), sourceID: 95}, - targets: make(map[codexGatewayBorrowTargetKey]codexGatewayBorrowTargetCheck)} - svc.gatewayBorrow = borrow - return svc, account, borrow, probe -} - -func seedBorrowWSTarget(t *testing.T, svc *OpenAIGatewayService, borrow *CodexGatewayBorrowService, account *Account, c *gin.Context, model string) { - t.Helper() - headers, _, err := svc.buildOpenAIWSHeaders(context.Background(), c, account, "target-token", - svc.getOpenAIWSProtocolResolver().Resolve(account), true, "", "", "", model, "") - require.NoError(t, err) - wsURL, err := svc.buildOpenAIResponsesWSURL(account) - require.NoError(t, err) - req, err := codexGatewayBorrowWSRequest(context.Background(), wsURL, headers) - require.NoError(t, err) - candidate := borrow.candidate - key := borrowTargetFingerprint(req, account, model, account.Proxy.URL(), nil, candidate.cookie.Value) - borrow.targets[codexGatewayBorrowTargetKey{account.ID, model}] = codexGatewayBorrowTargetCheck{policyRevision: currentCodexFingerprintPolicyForAccount(account).revision, key: key, - cookieKey: borrowHash(candidate.cookie.Value), expires: candidate.expires, result: CodexGatewayBorrowVerification{Success: true}} -} - -func borrowWSCompleted(model, id string) []byte { - // Tool-only completions are authoritative native Responses completions. - return []byte(fmt.Sprintf(`{"type":"response.completed","response":{"id":%q,"model":%q,"status":"completed","output":[{"type":"function_call","id":"fc_test","call_id":"call_test","name":"tool","arguments":"{}"}],"usage":{"input_tokens":1,"output_tokens":1}}}`, id, model)) -} - -func runBorrowWSV2(t *testing.T, svc *OpenAIGatewayService, account *Account, model, previousID string) (*OpenAIForwardResult, error) { - t.Helper() - body := map[string]any{"model": model, "input": "hello", "stream": false, "store": false} - if previousID != "" { - body["previous_response_id"] = previousID - } - recoveryTried := false - return svc.forwardOpenAIWSV2(context.Background(), borrowWSContext(), account, body, "", "", "target-token", - svc.getOpenAIWSProtocolResolver().Resolve(account), true, false, model, model, time.Now(), 1, "", &recoveryTried) -} - -func TestCodexGatewayBorrowWS_TwoModelsContinueOriginalConnectionsAfterCookieExpiryAndDisable(t *testing.T) { - svc, account, borrow, probe := borrowWSFixture(t, 2) - astraConn := &openAIWSCaptureConn{events: [][]byte{borrowWSCompleted("gpt-6-astra", "resp_astra_1"), borrowWSCompleted("gpt-6-astra", "resp_astra_2")}} - solConn := &openAIWSCaptureConn{events: [][]byte{borrowWSCompleted("gpt-6.1-sol", "resp_sol_1"), borrowWSCompleted("gpt-6.1-sol", "resp_sol_2")}} - dialer := &borrowWSDialer{conns: []openAIWSClientConn{astraConn, solConn}} - svc.openaiWSPool.setClientDialerForTest(dialer) - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6-astra") - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6.1-sol") - first, err := runBorrowWSV2(t, svc, account, "gpt-6-astra", "") - require.NoError(t, err) - require.Equal(t, "resp_astra_1", first.ResponseID) - second, err := runBorrowWSV2(t, svc, account, "gpt-6.1-sol", "") - require.NoError(t, err) - require.Equal(t, "resp_sol_1", second.ResponseID) - require.Len(t, dialer.headers, 2) - for i, headers := range dialer.headers { - require.Equal(t, "Bearer target-token", headers.Get("Authorization")) - require.Equal(t, "__oailb=shared-native-route", headers.Get("Cookie")) - require.Equal(t, account.Proxy.URL(), dialer.proxies[i]) - } - borrow.wsAnchors.mu.Lock() - expires := make(map[string]time.Time) - connections := make(map[string]string) - for key, entry := range borrow.wsAnchors.entries { - expires[key.model] = entry.expires - connections[key.model] = entry.connID - } - borrow.wsAnchors.mu.Unlock() - require.Len(t, expires, 2) - require.NotEqual(t, connections["gpt-6-astra"], connections["gpt-6.1-sol"]) - borrow.mu.Lock() - borrow.candidate.expires = time.Now().Add(-time.Second) - borrow.config.Enabled = false - borrow.targets = nil - borrow.mu.Unlock() - _, err = runBorrowWSV2(t, svc, account, "gpt-6.1-sol", "resp_astra_1") - require.Error(t, err, "disabling borrowing must not let a different model take over an existing response") - continued, err := runBorrowWSV2(t, svc, account, "gpt-6-astra", "resp_astra_1") - require.NoError(t, err) - require.Equal(t, "resp_astra_2", continued.ResponseID) - continued, err = runBorrowWSV2(t, svc, account, "gpt-6.1-sol", "resp_sol_1") - require.NoError(t, err) - require.Equal(t, "resp_sol_2", continued.ResponseID) - require.Len(t, dialer.headers, 2, "continuations must not redial") - require.Empty(t, probe.requests, "cached roots and every continuation must avoid source/target observations") - require.False(t, astraConn.closed) - require.False(t, solConn.closed) - borrow.wsAnchors.mu.Lock() - defer borrow.wsAnchors.mu.Unlock() - for key, entry := range borrow.wsAnchors.entries { - require.Equal(t, expires[key.model], entry.expires, "anchor deadlines never slide") - } -} - -func TestCodexGatewayBorrowWS_NewRootAtCapacityPreservesIdleAnchor(t *testing.T) { - svc, account, borrow, _ := borrowWSFixture(t, 1) - conn := &openAIWSCaptureConn{events: [][]byte{borrowWSCompleted("gpt-6-astra", "resp_kept")}} - dialer := &borrowWSDialer{conns: []openAIWSClientConn{conn}} - svc.openaiWSPool.setClientDialerForTest(dialer) - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6-astra") - _, err := runBorrowWSV2(t, svc, account, "gpt-6-astra", "") - require.NoError(t, err) - _, err = runBorrowWSV2(t, svc, account, "gpt-6-astra", "") - require.ErrorIs(t, err, errOpenAIWSConnQueueFull) - require.Len(t, dialer.headers, 1) - require.False(t, conn.closed, "ForceNewConn must not evict a valid idle anchor") - borrow.wsAnchors.mu.Lock() - defer borrow.wsAnchors.mu.Unlock() - require.Len(t, borrow.wsAnchors.entries, 1) - for _, entry := range borrow.wsAnchors.entries { - require.Equal(t, "resp_kept", entry.responseID) - } -} - -func TestCodexGatewayBorrowWS_PoolCleanupAndIncompatibleEvictionRespectAnchorUntil(t *testing.T) { - cfg := &config.Config{} - cfg.Gateway.OpenAIWS.MaxIdlePerAccount = 0 - pool := newOpenAIWSConnPool(cfg) - t.Cleanup(pool.Close) - ap := pool.getOrCreateAccountPool(201) - conn := newOpenAIWSConn("borrow_anchor", 201, &openAIWSFakeConn{}, nil) - conn.createdAtNano.Store(time.Now().Add(-2 * time.Hour).UnixNano()) - conn.lastUsedNano.Store(time.Now().Add(-20 * time.Minute).UnixNano()) - conn.anchorUntilNano.Store(time.Now().Add(time.Hour).UnixNano()) - conn.handshakeCompatibility.anchorScope = "borrow-model-session" - ap.conns[conn.id] = conn - require.Empty(t, pool.cleanupAccountLocked(ap, time.Now(), 1)) - require.Nil(t, pool.pickOldestIdleConnLocked(ap)) - require.Nil(t, pool.pickOldestIdleConnWithoutHandshakeCompatibilityLocked(ap, openAIWSHandshakeCompatibilityKey{})) - require.False(t, conn.isClosed()) - conn.anchorUntilNano.Store(time.Now().Add(-time.Second).UnixNano()) - require.Same(t, conn, pool.pickOldestIdleConnLocked(ap), "expired anchors return to ordinary pool retirement") -} - -func TestCodexGatewayBorrowWS_ModelIdentityTTLAndCapacityAreStrict(t *testing.T) { - store := &codexGatewayBorrowWSAnchorStore{} - now := time.Now() - key := codexGatewayBorrowWSAnchorKey{account: 1, apiKey: 2, group: 3, scope: "client", model: "gpt-6-astra"} - turn, err := store.begin(key, "", now) - require.NoError(t, err) - require.Equal(t, now.Add(time.Hour), turn.expires) - turn.connID, turn.qualified = "original", true - turn.finish(nil, &OpenAIForwardResult{ResponseID: "resp_one", UpstreamResponseModel: key.model}, nil) - otherModel := key - otherModel.model = "gpt-6.1-sol" - _, err = store.begin(otherModel, "resp_one", now.Add(time.Second)) - require.Error(t, err) - _, err = store.begin(key, "resp_one", now.Add(time.Hour)) - require.Error(t, err) - require.Empty(t, store.entries) - for i := 0; i < codexGatewayBorrowWSAnchorLimit; i++ { - entryKey := key - entryKey.scope = fmt.Sprint(i) - store.entries[entryKey] = codexGatewayBorrowWSAnchorEntry{responseID: "r", connID: "c", expires: now.Add(time.Hour)} - } - key.scope = "over-limit" - _, err = store.begin(key, "", now) - require.EqualError(t, err, "codex gateway borrow websocket anchor capacity reached") - require.Len(t, store.entries, 1024) -} - -func TestCodexGatewayBorrowWS_DirectClosedEvidenceRejectsReconnectAfterDisable(t *testing.T) { - svc, account, borrow, probe := borrowWSFixture(t, 1) - c := borrowWSContext() - seedBorrowWSTarget(t, svc, borrow, account, c, "gpt-6.1-sol") - headers, _, err := svc.buildOpenAIWSHeaders(context.Background(), c, account, "target-token", - svc.getOpenAIWSProtocolResolver().Resolve(account), true, "", "", "", "gpt-6.1-sol", "") - require.NoError(t, err) - wsURL, err := svc.buildOpenAIResponsesWSURL(account) - require.NoError(t, err) - turn, err := svc.prepareCodexGatewayBorrowWSTurn(context.Background(), c, account, wsURL, headers, nil, - "gpt-6.1-sol", "", account.Proxy.URL()) - require.NoError(t, err) - session := &codexGatewayBorrowWSDirectSession{turn: turn} - session.completed(&OpenAIForwardResult{RequestID: "resp_direct", UpstreamResponseModel: "gpt-6.1-sol", UpstreamTerminalEvent: "response.completed"}) - borrow.config.Enabled = false - borrow.candidate.expires = time.Now().Add(-time.Second) - require.NoError(t, session.check(c, account, headers, []byte(`{"previous_response_id":"resp_direct"}`), "gpt-6.1-sol", account.Proxy.URL())) - require.Error(t, session.check(c, account, headers, []byte(`{"previous_response_id":"resp_direct"}`), "gpt-6-astra", account.Proxy.URL())) - session.close(svc.openaiWSPool) - _, err = svc.prepareCodexGatewayBorrowWSTurn(context.Background(), c, account, wsURL, headers, nil, - "gpt-6.1-sol", "resp_direct", account.Proxy.URL()) - require.Error(t, err) - require.Empty(t, probe.requests) - require.Len(t, borrow.wsAnchors.entries, 1) - for _, entry := range borrow.wsAnchors.entries { - require.Empty(t, entry.connID) - require.Equal(t, turn.expires, entry.expires) - } -} - -func TestCodexGatewayBorrowWS_NativeIngressDoesNotPrepareAgainOnLaterFrame(t *testing.T) { - svc, account, borrow, probe := borrowWSFixture(t, 1) - conn := &openAIWSCaptureConn{events: [][]byte{borrowWSCompleted("gpt-6.1-sol", "resp_ingress_1"), borrowWSCompleted("gpt-6.1-sol", "resp_ingress_2")}} - dialer := &borrowWSDialer{conns: []openAIWSClientConn{conn}} - svc.openaiWSPool.setClientDialerForTest(dialer) - seedBorrowWSTarget(t, svc, borrow, account, borrowWSContext(), "gpt-6.1-sol") - turns := make(chan struct{}, 2) - serverErr := make(chan error, 1) - server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { - client, err := coderws.Accept(w, req, nil) - if err != nil { - serverErr <- err - return - } - defer func() { _ = client.CloseNow() }() - _, first, err := client.Read(context.Background()) - if err != nil { - serverErr <- err - return - } - c := borrowWSContext() - c.Request = req - serverErr <- svc.ProxyResponsesWebSocketFromClient(context.Background(), c, client, account, "target-token", first, - &OpenAIWSIngressHooks{AfterTurn: func(_ int, _ *OpenAIForwardResult, _ error) { turns <- struct{}{} }}) - })) - defer server.Close() - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - defer cancel() - client, _, err := coderws.Dial(ctx, "ws"+strings.TrimPrefix(server.URL, "http"), &coderws.DialOptions{HTTPHeader: borrowWSContext().Request.Header}) - require.NoError(t, err) - defer func() { _ = client.CloseNow() }() - require.NoError(t, client.Write(ctx, coderws.MessageText, []byte(`{"type":"response.create","model":"gpt-6.1-sol","input":"hello","store":false}`))) - _, _, err = client.Read(ctx) - require.NoError(t, err) - select { - case <-turns: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - borrow.mu.Lock() - borrow.config.Enabled = false - borrow.candidate.expires = time.Now().Add(-time.Second) - borrow.mu.Unlock() - require.NoError(t, client.Write(ctx, coderws.MessageText, []byte(`{"type":"response.create","model":"gpt-6.1-sol","previous_response_id":"resp_ingress_1","input":"next","store":false}`))) - _, _, err = client.Read(ctx) - require.NoError(t, err) - select { - case <-turns: - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - _ = client.Close(coderws.StatusNormalClosure, "") - select { - case err := <-serverErr: - require.NoError(t, err) - case <-ctx.Done(): - t.Fatal(ctx.Err()) - } - require.Len(t, dialer.headers, 1) - require.Empty(t, probe.requests) - require.Len(t, conn.writes, 2) - require.Equal(t, "resp_ingress_1", conn.writes[1]["previous_response_id"]) -} diff --git a/backend/internal/service/http_upstream_profile.go b/backend/internal/service/http_upstream_profile.go index b729ee3ffc15..92020abb357c 100644 --- a/backend/internal/service/http_upstream_profile.go +++ b/backend/internal/service/http_upstream_profile.go @@ -11,11 +11,6 @@ const ( HTTPUpstreamProfileOpenAI HTTPUpstreamProfile = "openai" HTTPUpstreamProfileGrok HTTPUpstreamProfile = "grok" HTTPUpstreamProfileLongStream HTTPUpstreamProfile = "long_stream" - // The pinned upstream uses ordinary OpenAI for source acquisition and - // OpenAIHarvest for the two target probes. Keep aliases for local callers. - HTTPUpstreamProfileOpenAIHarvest HTTPUpstreamProfile = "openai_harvest" - HTTPUpstreamProfileCodexBorrowSource = HTTPUpstreamProfileOpenAI - HTTPUpstreamProfileCodexBorrowTarget = HTTPUpstreamProfileOpenAIHarvest ) type httpUpstreamProfileContextKey struct{} @@ -43,8 +38,7 @@ func HTTPUpstreamProfileFromContext(ctx context.Context) HTTPUpstreamProfile { return HTTPUpstreamProfileDefault } switch profile { - case HTTPUpstreamProfileOpenAI, HTTPUpstreamProfileGrok, HTTPUpstreamProfileLongStream, - HTTPUpstreamProfileOpenAIHarvest: + case HTTPUpstreamProfileOpenAI, HTTPUpstreamProfileGrok, HTTPUpstreamProfileLongStream: return profile default: return HTTPUpstreamProfileDefault diff --git a/backend/internal/service/openai_alpha_search.go b/backend/internal/service/openai_alpha_search.go index 34f38a622d20..81208b5a7b9e 100644 --- a/backend/internal/service/openai_alpha_search.go +++ b/backend/internal/service/openai_alpha_search.go @@ -73,7 +73,7 @@ func (s *OpenAIGatewayService) ForwardAlphaSearch(ctx context.Context, c *gin.Co } upstreamStart := time.Now() - resp, err := s.doOpenAICodexUpstream(req, account, proxyURL, upstreamModel) + resp, err := s.doOpenAICodexUpstream(req, account, proxyURL) SetOpsLatencyMs(c, OpsUpstreamLatencyMsKey, time.Since(upstreamStart).Milliseconds()) if err != nil { return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, true) @@ -163,7 +163,7 @@ func (s *OpenAIGatewayService) forwardAlphaSearchViaResponsesWebSearch( SetActualOpenAIUpstreamEndpoint(c, "/v1/responses") upstreamStart := time.Now() - resp, err := s.doOpenAICodexUpstream(req, account, proxyURL, upstreamModel) + resp, err := s.doOpenAICodexUpstream(req, account, proxyURL) SetOpsLatencyMs(c, OpsUpstreamLatencyMsKey, time.Since(upstreamStart).Milliseconds()) if err != nil { return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, true) diff --git a/backend/internal/service/openai_codex_request_compression.go b/backend/internal/service/openai_codex_request_compression.go index a07f9e30287c..452a346e6e83 100644 --- a/backend/internal/service/openai_codex_request_compression.go +++ b/backend/internal/service/openai_codex_request_compression.go @@ -143,37 +143,10 @@ func prepareOpenAICodexWireRequest(req *http.Request, account *Account) (*http.R // doOpenAICodexUpstream 是 OpenAI 网关所有 Responses 端点 POST 的统一发送入口：先按上述 // 规则生成线上请求（满足条件时压缩），再交给 httpUpstream。读取失败作为传输错误返回。 -func (s *OpenAIGatewayService) doOpenAICodexUpstream(req *http.Request, account *Account, proxyURL string, resolvedModels ...string) (*http.Response, error) { - model := CodexGatewayBorrowModelFromContext(req.Context()) - if len(resolvedModels) > 0 { - model = resolvedModels[0] - } - borrowed, err := s.applyCodexGatewayBorrowHTTP(req, account, model, proxyURL) - if err != nil { - return nil, err - } - req = borrowed +func (s *OpenAIGatewayService) doOpenAICodexUpstream(req *http.Request, account *Account, proxyURL string) (*http.Response, error) { wire, err := prepareOpenAICodexWireRequest(req, account) if err != nil { - codexBorrowUsageFromContext(req.Context()).blocked(err) return nil, err } - applied := codexBorrowHTTPApplied(wire) - if err := consumeBorrowDiagnosticRequest(wire.Context()); err != nil { - codexBorrowUsageFromContext(req.Context()).blocked(err) - return nil, err - } - tracker := s.gatewayBorrow.beginUsage(wire.Context(), account.ID, model, "http", applied) - if node := borrowNodeBinding(wire); node != nil { - proxyURL = node.node.proxy - if tracker != nil { - tracker.mu.Lock() - tracker.row.ProxyID, tracker.row.NodeIdentity, tracker.row.Gateway = node.node.id, node.node.identity, node.gateway - tracker.publish(0, 0, 0) - tracker.mu.Unlock() - } - } - resp, sendErr := s.httpUpstream.Do(wire, proxyURL, account.ID, account.Concurrency) - resp, sendErr = trackCodexBorrowNodeHTTP(wire, resp, sendErr) - return s.gatewayBorrow.trackHTTPResponse(tracker, resp, sendErr) + return s.httpUpstream.Do(wire, proxyURL, account.ID, account.Concurrency) } diff --git a/backend/internal/service/openai_gateway_chat_completions.go b/backend/internal/service/openai_gateway_chat_completions.go index 9f840f561168..ce5f8e9076cd 100644 --- a/backend/internal/service/openai_gateway_chat_completions.go +++ b/backend/internal/service/openai_gateway_chat_completions.go @@ -387,7 +387,7 @@ func (s *OpenAIGatewayService) forwardAsChatCompletions( upstreamReq.Header.Set("session_id", generateSessionUUID(sessionKey)) } } - resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, upstreamModel) + resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) if err != nil { cancelUpstream() @@ -950,7 +950,6 @@ func (s *OpenAIGatewayService) handleChatStreamingResponse( } processDataLine := func(payload string) bool { - if firstChunk { firstChunk = false ms := int(time.Since(startTime).Milliseconds()) @@ -1228,7 +1227,6 @@ func (s *OpenAIGatewayService) handleChatStreamingResponse( } } missingTerminalErr := func() (*OpenAIForwardResult, error) { - return resultWithUsage(), fmt.Errorf("stream usage incomplete: missing terminal event") } processFrame := func(frame openAICompatSSEFrame) bool { diff --git a/backend/internal/service/openai_gateway_forward.go b/backend/internal/service/openai_gateway_forward.go index d59017f2fdd9..2d2b9c813b92 100644 --- a/backend/internal/service/openai_gateway_forward.go +++ b/backend/internal/service/openai_gateway_forward.go @@ -1023,7 +1023,6 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco agentTaskRecoveryTried := false rejectedFieldRetryState := newOpenAIResponsesRejectedFieldRetryState(body) var wireBody []byte - var borrowPreparationElapsed time.Duration for { // Read the final attempt payload. A compatibility retry may have changed // the request, so neither the original alias nor prior attempt is usage @@ -1042,9 +1041,9 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco upstreamCtx, releaseUpstreamCtx := detachUpstreamContext(ctx) var headerGuard *openAIFirstOutputHeaderGuard var pelicanHeaderPolicy *pelicanFirstOutputPolicy - if firstOutputTimeout > 0 && !IsPelicanGeneration(ctx) && !s.codexGatewayBorrowHTTPConfigured(account, upstreamModel) { + if firstOutputTimeout > 0 && !IsPelicanGeneration(ctx) { upstreamCtx, headerGuard = newOpenAIFirstOutputHeaderGuard( - upstreamCtx, releaseUpstreamCtx, startTime.Add(borrowPreparationElapsed).Add(firstOutputTimeout), + upstreamCtx, releaseUpstreamCtx, startTime.Add(firstOutputTimeout), ) } upstreamReq, err := s.buildUpstreamRequest(upstreamCtx, c, account, body, token, reqStream, promptCacheKey, isCodexCLI) @@ -1080,75 +1079,26 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco } return nil, integrityErr } - // Qualification observes the fully finalized request, after recovery and - // integrity checks, using the original client's cancellation/deadline. - // It must finish before the business header and semantic-output budgets. - prepared, preparationElapsed, preparationErr := s.prepareCodexGatewayBorrowHTTP(ctx, upstreamReq, account, upstreamModel, proxyURL) - if preparationErr != nil { - if headerGuard != nil { - headerGuard.close() - } - return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, preparationErr, false) - } - upstreamReq = prepared - if preparationElapsed > 0 { - borrowPreparationElapsed += preparationElapsed - // A save may have enabled borrowing after the earlier configuration - // check. Drop that old guard without treating its cancellation as an - // upstream timeout, and retain the finalized request's context values. - replacedGuard := headerGuard != nil - if headerGuard != nil { - headerGuard.close() - headerGuard = nil - } - upstreamReq = upstreamReq.WithContext(codexGatewayBorrowHTTPPreparedContext(ctx, upstreamReq.Context(), replacedGuard, httpInvalidEncryptedContentRetryTried)) - } - firstOutputStartedAt := startTime.Add(borrowPreparationElapsed) if firstOutputTimeout > 0 && headerGuard == nil { if IsPelicanGeneration(ctx) { pelicanHeaderPolicy = &pelicanFirstOutputPolicy{timeout: firstOutputTimeout} upstreamReq = upstreamReq.WithContext(context.WithValue(upstreamReq.Context(), pelicanFirstOutputPolicyContextKey{}, pelicanHeaderPolicy)) } else { - upstreamCtx, headerGuard = newOpenAIFirstOutputHeaderGuard(upstreamReq.Context(), releaseUpstreamCtx, firstOutputStartedAt.Add(firstOutputTimeout)) + upstreamCtx, headerGuard = newOpenAIFirstOutputHeaderGuard(upstreamReq.Context(), releaseUpstreamCtx, startTime.Add(firstOutputTimeout)) upstreamReq = upstreamReq.WithContext(upstreamCtx) } } - if _, borrowed := upstreamReq.Context().Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation); borrowed { - finalized, finalizationErr := s.applyCodexGatewayBorrowHTTP(upstreamReq, account, upstreamModel, proxyURL) - if finalizationErr != nil { - if headerGuard != nil { - headerGuard.close() - } - return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, finalizationErr, false) - } - upstreamReq = finalized - } - // Bind only after preparation, context/guard changes and the final - // cached Cookie canonicalization. Dispatch preserves this plaintext - // pointer; wire compression still uses its separate outbound copy. - // Send request freezeOpenAIContinuationWire(c, upstreamReq, wireBody) upstreamStart := time.Now() - resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, upstreamModel) + resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) if pelicanHeaderPolicy != nil { headerGuard = pelicanHeaderPolicy.guard - if !pelicanHeaderPolicy.startedAt.IsZero() { - firstOutputStartedAt = pelicanHeaderPolicy.startedAt - } } SetOpsLatencyMs(c, OpsUpstreamLatencyMsKey, time.Since(upstreamStart).Milliseconds()) - if IsCodexGatewayBorrowRequestFailure(err) { - if resp != nil && resp.Body != nil { - _ = resp.Body.Close() - } - if headerGuard != nil { - headerGuard.close() - } - return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, false) - } + headerTimedOut := false if pelicanHeaderPolicy != nil { headerTimedOut = pelicanHeaderPolicy.stopHeaderWait() @@ -1330,13 +1280,8 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco var imageOutputSizes []string if reqStream { setOpenAIRefusalEarlyStreamEligibility(c, account, body) - streamCtx := ctx - if borrowPreparationElapsed > 0 || IsPelicanGeneration(ctx) { - streamCtx = withCodexGatewayBorrowHTTPFirstOutputStart(ctx, firstOutputStartedAt) - } - streamResult, err := s.handleStreamingResponseWithReasoning(streamCtx, resp, c, account, startTime, originalModel, upstreamModel, reasoningEffortValue) + streamResult, err := s.handleStreamingResponseWithReasoning(ctx, resp, c, account, startTime, originalModel, upstreamModel, reasoningEffortValue) if err != nil { - if signal, ok := asOpenAICompactFallbackSignal(err); ok { if retryBody, fallbackModel, retry := s.prepareOpenAICompactFallbackRetry( c, account, originalModel, body, http.StatusBadRequest, signal.message, signal.payload, compactModelFallbackRetried, @@ -1386,7 +1331,6 @@ func (s *OpenAIGatewayService) Forward(ctx context.Context, c *gin.Context, acco } else { nonStreamResult, err := s.handleNonStreamingResponse(ctx, resp, c, account, originalModel, upstreamModel) if err != nil { - if signal, ok := asOpenAICompactFallbackSignal(err); ok { if retryBody, fallbackModel, retry := s.prepareOpenAICompactFallbackRetry( c, account, originalModel, body, http.StatusBadRequest, signal.message, signal.payload, compactModelFallbackRetried, @@ -1631,7 +1575,6 @@ func (s *OpenAIGatewayService) buildUpstreamRequest(ctx context.Context, c *gin. // x-codex-beta-features：按真实 Codex 的会话级行为补注（在账号级覆写之后， // 保证不被覆盖丢失）。 applyOpenAICodexBetaFeatures(c, account, req.Header) - req = req.WithContext(withCodexBorrowServiceTier(req.Context(), gjson.GetBytes(body, "service_tier").String())) if err := s.finalizeCodexOutboundHeaders(req.Context(), c, account, req.Header, gjson.GetBytes(body, "model").String(), gjson.GetBytes(body, "service_tier").String()); err != nil { return nil, err } diff --git a/backend/internal/service/openai_gateway_messages.go b/backend/internal/service/openai_gateway_messages.go index 84ee86294f3f..0d68bed92e58 100644 --- a/backend/internal/service/openai_gateway_messages.go +++ b/backend/internal/service/openai_gateway_messages.go @@ -386,7 +386,6 @@ func (s *OpenAIGatewayService) ForwardAsAnthropic( // 清除身份头。真正发送前恢复完整 Codex 身份，避免 ChatGPT Codex 上游因缺失 // originator/OpenAI-Beta 返回 404（issue #3901）。 ensureCodexIdentityHeaders(upstreamReq.Header) - upstreamReq = upstreamReq.WithContext(withCodexBorrowServiceTier(upstreamReq.Context(), gjson.GetBytes(responsesBody, "service_tier").String())) if err := s.finalizeCodexOutboundHeaders(upstreamReq.Context(), c, account, upstreamReq.Header, upstreamModel, gjson.GetBytes(responsesBody, "service_tier").String()); err != nil { return nil, err } @@ -424,7 +423,7 @@ func (s *OpenAIGatewayService) ForwardAsAnthropic( return nil, fmt.Errorf("build grok retry request: %w", err) } } - resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, upstreamModel) + resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) if err != nil { return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, false) } @@ -909,7 +908,6 @@ func (s *OpenAIGatewayService) readOpenAICompatBufferedTerminal( return response, usage, acc, []byte(payload), nil } if failedResponse, ok := openAICompatBareErrorResponse(&event, []byte(payload)); ok { - return failedResponse, usage, acc, []byte(payload), nil } diff --git a/backend/internal/service/openai_gateway_passthrough.go b/backend/internal/service/openai_gateway_passthrough.go index 01a3441a0d15..6d0bd8da6283 100644 --- a/backend/internal/service/openai_gateway_passthrough.go +++ b/backend/internal/service/openai_gateway_passthrough.go @@ -416,11 +416,10 @@ retryUpstream: freezeOpenAIContinuationWire(c, upstreamReq, wireBody) upstreamStart := time.Now() - resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, actualModel) + resp, err = s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) SetOpsLatencyMs(c, OpsUpstreamLatencyMsKey, time.Since(upstreamStart).Milliseconds()) if err != nil { - // Transport-level failure (proxy/DNS/TCP/TLS — no HTTP response). Convert to // a failover so the handler switches to a healthy account. return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, true) @@ -511,7 +510,6 @@ retryUpstream: setOpenAIRefusalEarlyStreamEligibility(c, account, body) result, err := s.handleStreamingResponsePassthrough(ctx, resp, c, account, startTime, reqModel, upstreamPassthroughModel) if err != nil { - if retryBody, fallbackModel, retry := s.applyOpenAIPassthroughCompactFallbackFromSignal( c, account, reqModel, body, err, compactModelFallbackRetried, resp, ); retry { @@ -538,7 +536,6 @@ retryUpstream: } else { result, err := s.handleNonStreamingResponsePassthrough(ctx, resp, c, account, reqModel, upstreamPassthroughModel) if err != nil { - if retryBody, fallbackModel, retry := s.applyOpenAIPassthroughCompactFallbackFromSignal( c, account, reqModel, body, err, compactModelFallbackRetried, resp, ); retry { @@ -792,7 +789,6 @@ func (s *OpenAIGatewayService) buildUpstreamRequestOpenAIPassthrough( // x-codex-beta-features：按真实 Codex 的会话级行为补注（在账号级覆写之后， // 保证不被覆盖丢失）。 applyOpenAICodexBetaFeatures(c, account, req.Header) - req = req.WithContext(withCodexBorrowServiceTier(req.Context(), gjson.GetBytes(body, "service_tier").String())) if err := s.finalizeCodexOutboundHeaders(req.Context(), c, account, req.Header, gjson.GetBytes(body, "model").String(), gjson.GetBytes(body, "service_tier").String()); err != nil { return nil, err } @@ -2217,7 +2213,6 @@ func (s *OpenAIGatewayService) handleStreamingResponsePassthrough( upstreamEventType := effectiveOpenAISSEEventType(rawDataBytes, pendingSSEEventType) s.parseSSEUsageBytesWithType(rawDataBytes, upstreamEventType, usage) if upstreamEventType == "response.failed" || upstreamEventType == "error" || (upstreamEventType == "response.done" && gjson.GetBytes(rawDataBytes, "response.status").String() == "failed") { - if failoverErr, ok := s.openAIBudgetExceededHTTPResponseTerminalFailover(ctx, c, account, resp.StatusCode, resp.Header, rawDataBytes); ok { // Classification must happen on the untouched event, before error // rules, namespace restoration or the client stream see it. The @@ -2267,7 +2262,6 @@ func (s *OpenAIGatewayService) handleStreamingResponsePassthrough( } eventType := effectiveOpenAISSEEventType(dataBytes, pendingSSEEventType) if eventType == "response.completed" || eventType == "response.done" { - // A response terminal supersedes a non-authoritative bare error. sawBareError = false bareErrorPayload = nil diff --git a/backend/internal/service/openai_gateway_response_handling.go b/backend/internal/service/openai_gateway_response_handling.go index c3f42fd8e4db..d60d8b1a8f59 100644 --- a/backend/internal/service/openai_gateway_response_handling.go +++ b/backend/internal/service/openai_gateway_response_handling.go @@ -328,9 +328,9 @@ func (s *OpenAIGatewayService) handleStreamingResponseWithReasoning(ctx context. var firstOutputTimer *time.Timer var firstOutputCh <-chan time.Time if firstOutputTimeout > 0 { - // Borrow preparation is outside the business first-output budget; TTFT - // and the total request duration continue to use the original startTime. - remaining := time.Until(codexGatewayBorrowHTTPFirstOutputStart(ctx, startTime).Add(firstOutputTimeout)) + // Pelican queueing is outside its first-output budget; ordinary + // requests keep the original startTime. + remaining := time.Until(PelicanFirstOutputStart(ctx, startTime).Add(firstOutputTimeout)) if remaining <= 0 { remaining = time.Nanosecond } @@ -495,7 +495,6 @@ func (s *OpenAIGatewayService) handleStreamingResponseWithReasoning(ctx context. lastDownstreamWriteAt = time.Now() } finalizeStream := func() (*openaiStreamingResult, error) { - if stageFirstOutput && eventInProgress { // EOF dispatches the final SSE event even without a trailing blank line. completeGuardedEvent(true) @@ -540,7 +539,6 @@ func (s *OpenAIGatewayService) handleStreamingResponseWithReasoning(ctx context. return resultWithUsage(), fmt.Errorf("stream usage incomplete: missing terminal event") } if sawFailedEvent { - return resultWithUsage(), fmt.Errorf("upstream response failed: %s", failedMessage) } if terminalResponseErr != nil { diff --git a/backend/internal/service/openai_gateway_service.go b/backend/internal/service/openai_gateway_service.go index e7648e07880d..5e217705ff3c 100644 --- a/backend/internal/service/openai_gateway_service.go +++ b/backend/internal/service/openai_gateway_service.go @@ -475,7 +475,6 @@ type OpenAIGatewayService struct { settingService *SettingService userPlatformQuotaRepo UserPlatformQuotaRepository systemPrompts *SystemPromptService - gatewayBorrow *CodexGatewayBorrowService usageCache *UsageCache usageCommitObserver UsageCommitObserver liveAttestation liveattestation.Provider @@ -1102,7 +1101,7 @@ func (s *OpenAIGatewayService) detectCodexClientRestriction(c *gin.Context, acco // A trusted administrator operation has no downstream client to admit. // Its purpose is installed internally; upstream identity/authentication and // every account/global sending policy still use the ordinary gateway path. - if c != nil && c.Request != nil && IsPelicanGeneration(c.Request.Context()) && !IsCodexGatewayBorrowObservation(c.Request.Context()) { + if c != nil && c.Request != nil && IsPelicanGeneration(c.Request.Context()) { return CodexClientRestrictionDetectionResult{Reason: CodexClientRestrictionReasonDisabled} } // 安全默认：即便缺 settingService（仅测试/误配可达）也保持指纹门为默认种子， diff --git a/backend/internal/service/openai_images_responses.go b/backend/internal/service/openai_images_responses.go index 960388ae4ae1..a37c3dd733d2 100644 --- a/backend/internal/service/openai_images_responses.go +++ b/backend/internal/service/openai_images_responses.go @@ -1902,7 +1902,7 @@ func (s *OpenAIGatewayService) forwardOpenAIImagesOAuth( upstreamStart := time.Now() // 非 direct 分支是发往 /backend-api/codex/responses 的 OAuth POST，与其他 Responses // 端点同一发送入口（含 zstd 压缩边界）；direct-images URL 不满足门控，保持明文。 - resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, upstreamModel) + resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) SetOpsLatencyMs(c, OpsUpstreamLatencyMsKey, time.Since(upstreamStart).Milliseconds()) if err != nil { return nil, s.handleOpenAIImagesUpstreamTransportError(upstreamCtx, c, account, err, upstreamReq.URL.String()) diff --git a/backend/internal/service/openai_upstream_transport_error.go b/backend/internal/service/openai_upstream_transport_error.go index b9e4761ba9ef..ee7570f8cf45 100644 --- a/backend/internal/service/openai_upstream_transport_error.go +++ b/backend/internal/service/openai_upstream_transport_error.go @@ -118,10 +118,7 @@ func (s *OpenAIGatewayService) handleOpenAIUpstreamTransportError(ctx context.Co if isClientCanceledTransportError(ctx, err) { return err } - if IsCodexGatewayBorrowRequestFailure(err) { - RecordCodexGatewayBorrowPreparationFailure(c, account, err) - return NewCodexGatewayBorrowRequestFailure(err) - } + safeErr := sanitizeUpstreamErrorMessage(err.Error()) setOpsUpstreamError(c, 0, safeErr, "") appendOpsUpstreamError(c, OpsUpstreamErrorEvent{ diff --git a/backend/internal/service/openai_ws_forwarder_ingress.go b/backend/internal/service/openai_ws_forwarder_ingress.go index c51efb3cb2cf..8b60f0a80960 100644 --- a/backend/internal/service/openai_ws_forwarder_ingress.go +++ b/backend/internal/service/openai_ws_forwarder_ingress.go @@ -953,13 +953,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( agentTaskRecoveryTried := false initialTransportRetryUsed := false - var borrowTurn *codexGatewayBorrowWSTurn - var lastBorrowTurn *codexGatewayBorrowWSTurn - defer func() { - if borrowTurn != nil { - borrowTurn.finish(pool, nil, errors.New("websocket borrow turn did not complete")) - } - }() var acquireTurnLease func(int, string, bool, bool) (*openAIWSConnLease, error) acquireTurnLease = func(turn int, preferred string, forcePreferredConn bool, forceNewConn bool) (*openAIWSConnLease, error) { req := cloneOpenAIWSAcquireRequest(baseAcquireReq) @@ -967,7 +960,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( req.ForcePreferredConn = forcePreferredConn // dedicated 模式及重试均新建连接，避免复用陈旧上下文。 req.ForceNewConn = dedicatedMode || forceNewConn || (turn == 1 && initialTransportRetryUsed) - borrowTurn.applyToAcquire(&req) acquireCtx, acquireCancel := context.WithTimeout(ctx, acquireTimeout) lease, acquireErr := pool.Acquire(acquireCtx, req) acquireCancel() @@ -1060,11 +1052,7 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( return nil, acquireErr } connID := strings.TrimSpace(lease.ConnID()) - if bindErr := borrowTurn.bindLease(lease); bindErr != nil { - lease.MarkBroken() - lease.Release() - return nil, bindErr - } + logOpenAIWSModeInfo( "ingress_ws_upstream_connected account_id=%d turn=%d conn_id=%s conn_reused=%v conn_idle_ms=%d conn_age_ms=%d upstream_pings=%d conn_pick_ms=%d queue_wait_ms=%d preferred_conn_id=%s", account.ID, @@ -1220,7 +1208,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( eventType, eventResponseID, _ := parseOpenAIWSEventEnvelope(upstreamMessage) responseModelObserver.ObserveOpenAI(upstreamMessage, eventType) - borrowTurn.observe(rawUpstreamMessage) if responseID == "" && eventResponseID != "" { responseID = eventResponseID } @@ -1276,7 +1263,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( fallbackReason == string(openAIContinuationStateErrorInvalidEncryptedContent) recoverablePrevNotFound := fallbackReason == openAIWSIngressStagePreviousResponseNotFound && turnPreviousResponseID != "" && - borrowTurn == nil && s.openAIWSIngressPreviousResponseRecoveryEnabled() && !downstreamOutputStarted() if recoverablePrevNotFound { @@ -1617,9 +1603,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( preferredConnID = "" } recoverIngressPrevResponseNotFound := func(relayErr error, turn int, connID string) bool { - if borrowTurn != nil { - return false - } if !isOpenAIWSIngressPreviousResponseNotFound(relayErr) { return false } @@ -1646,9 +1629,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( return true } retryIngressTurn := func(relayErr error, turn int, connID string) bool { - if borrowTurn != nil && borrowTurn.previousID != "" { - return false - } _, transportFailure := openAIWSIngressTurnTransportCause(relayErr) if transportFailure && initialTransportRetryUsed { return false @@ -1804,7 +1784,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( return fmt.Errorf("prepare websocket request: %w", prepareErr) } wireFields := gjson.GetManyBytes(wirePayload, "prompt_cache_key", "model", "service_tier") - ctx = withCodexBorrowServiceTier(ctx, wireFields[2].String()) finalHeaders, _, headerErr := s.buildOpenAIWSHeaders(ctx, c, account, token, wsDecision, isCodexCLI, turnState, strings.TrimSpace(c.GetHeader(openAIWSTurnMetadataHeader)), wireFields[0].String(), wireFields[1].String(), wireFields[2].String()) if headerErr != nil { @@ -1820,29 +1799,7 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( } baseAcquireReq.FingerprintPolicy = codexFingerprintPolicyForContext(c, account) baseAcquireReq.Headers = finalHeaders - if borrowTurn == nil { - borrowPreviousID := currentPreviousResponseID - // Later frames without an explicit previous ID still belong to the - // live native session. They need no fresh cookie or qualification. - if borrowPreviousID == "" && lastBorrowTurn != nil && sessionLease != nil && - !contextWindowBoundary.Changed && lastBorrowTurn.model == wireFields[1].String() { - borrowPreviousID = lastTurnResponseID - } - var borrowPrepareErr error - borrowTurn, borrowPrepareErr = s.prepareCodexGatewayBorrowWSTurn(ctx, c, account, wsURL, finalHeaders, - currentClientPayload, wireFields[1].String(), borrowPreviousID, baseAcquireReq.ProxyURL) - if borrowPrepareErr != nil { - return codexGatewayBorrowWSPreparationError(c, account, borrowPrepareErr) - } - if borrowTurn != nil { - if sessionLease != nil && (borrowTurn.previousID == "" || borrowTurn.connID != sessionConnID) { - resetSessionLease(false) - } - preferredConnID = borrowTurn.connID - } else if sessionLease != nil && sessionLease.conn.handshakeCompatibility.anchorScope != "" { - resetSessionLease(false) - } - } + forcePreferredConn := isStrictAffinityTurn(currentPayload) && strings.TrimSpace(preferredConnID) != "" if sessionLease == nil { acquiredLease, acquireErr := acquireTurnLease(turn, preferredConnID, forcePreferredConn, turnRetry > 0) @@ -1914,8 +1871,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( continue } if retryIngressTurn(relayErr, turn, connID) { - borrowTurn.finish(pool, nil, relayErr) - borrowTurn = nil continue } finalErr := relayErr @@ -1947,9 +1902,6 @@ func (s *OpenAIGatewayService) ProxyResponsesWebSocketFromClient( turnPrevRecoveryTried = false lastTurnFinishedAt = time.Now() lastTurnClean = true - borrowTurn.finish(pool, result, nil) - lastBorrowTurn = borrowTurn - borrowTurn = nil if hooks != nil && hooks.AfterTurn != nil { hooks.AfterTurn(turn, result, nil) } diff --git a/backend/internal/service/openai_ws_forwarder_support.go b/backend/internal/service/openai_ws_forwarder_support.go index 9e0d6ee99347..8790a0550aa6 100644 --- a/backend/internal/service/openai_ws_forwarder_support.go +++ b/backend/internal/service/openai_ws_forwarder_support.go @@ -78,9 +78,6 @@ func (s *OpenAIGatewayService) performOpenAIWSGeneratePrewarm( prewarmPayload["generate"] = false prewarmPayloadJSON := payloadAsJSONBytes(prewarmPayload) - if err := consumeBorrowDiagnosticRequest(ctx); err != nil { - return err - } if err := lease.WriteJSONWithContextTimeout(ctx, prewarmPayload, s.openAIWSWriteTimeout()); err != nil { lease.MarkBroken() logOpenAIWSModeInfo( diff --git a/backend/internal/service/openai_ws_forwarder_v2.go b/backend/internal/service/openai_ws_forwarder_v2.go index 84eb847b45ee..ecb862660aba 100644 --- a/backend/internal/service/openai_ws_forwarder_v2.go +++ b/backend/internal/service/openai_ws_forwarder_v2.go @@ -8,7 +8,6 @@ import ( "fmt" "net/http" "net/url" - "slices" "strings" "time" @@ -37,7 +36,7 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( attempt int, lastFailureReason string, agentTaskRecoveryTried *bool, -) (borrowResult *OpenAIForwardResult, borrowErr error) { +) (*OpenAIForwardResult, error) { if s == nil || account == nil { return nil, wrapOpenAIWSFallback("invalid_state", errors.New("service or account is nil")) } @@ -175,7 +174,6 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( storeDisabledConnMode := s.openAIWSStoreDisabledConnMode() forceNewConnByPolicy := shouldForceNewConnOnStoreDisabled(storeDisabledConnMode, lastFailureReason) forceNewConn := forceNewConnByPolicy && storeDisabled && previousResponseID == "" && sessionHash != "" && preferredConnID == "" - ctx = withCodexBorrowServiceTier(ctx, openAIWSPayloadString(payload, "service_tier")) wsHeaders, sessionResolution, buildHdrErr := s.buildOpenAIWSHeaders( ctx, c, @@ -197,27 +195,6 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( if account.ProxyID != nil && account.Proxy != nil { proxyURL = account.Proxy.URL() } - borrowTurn, borrowPrepareErr := s.prepareCodexGatewayBorrowWSTurn(ctx, c, account, wsURL, wsHeaders, - payloadAsJSONBytes(reqBody), openAIWSPayloadString(payload, "model"), previousResponseID, proxyURL, executionScope) - if borrowPrepareErr != nil { - return nil, codexGatewayBorrowWSPreparationError(c, account, borrowPrepareErr) - } - defer func() { - if borrowTurn != nil && borrowTurn.previousID != "" && borrowErr != nil { - var fallback *openAIWSFallbackError - if errors.As(borrowErr, &fallback) { - // A borrowed continuation can never fall back or redial after its - // original upstream socket has become unavailable. - borrowErr = NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - } - borrowTurn.finish(s.getOpenAIWSConnPool(), borrowResult, borrowErr) - }() - if borrowTurn != nil { - wsHeaders = cloneHeader(borrowTurn.headers) - preferredConnID = borrowTurn.connID - forceNewConn = borrowTurn.previousID == "" - } turnState = strings.TrimSpace(wsHeaders.Get(openAIWSTurnStateHeader)) logOpenAIWSModeDebug( "acquire_start account_id=%d account_type=%s transport=%s preferred_conn_id=%s has_previous_response_id=%v session_hash=%s has_turn_state=%v turn_state_len=%d has_turn_metadata=%v turn_metadata_len=%d store_disabled=%v store_disabled_conn_mode=%s retry_last_reason=%s force_new_conn=%v header_user_agent=%s header_openai_beta=%s header_originator=%s header_accept_language=%s header_session_id=%s header_conversation_id=%s session_id_source=%s conversation_id_source=%s has_prompt_cache_key=%v has_chatgpt_account_id=%v has_authorization=%v has_session_id=%v has_conversation_id=%v proxy_enabled=%v", @@ -266,7 +243,6 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( ForceNewConn: forceNewConn, ProxyURL: proxyURL, } - borrowTurn.applyToAcquire(&acquireReq) lease, err := s.getOpenAIWSConnPool().Acquire(acquireCtx, acquireReq) if err != nil { var agentDialErr *openAIWSDialError @@ -310,15 +286,13 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( // 释放租约，避免等待中的业务请求取得它后又被借用清理关闭。 cleanExit := false defer func() { - if !cleanExit || (borrowTurn != nil && borrowTurn.pelicanOneShot) { + if !cleanExit { lease.MarkBroken() } lease.Release() }() connID := strings.TrimSpace(lease.ConnID()) - if borrowBindErr := borrowTurn.bindLease(lease); borrowBindErr != nil { - return nil, borrowBindErr - } + logOpenAIWSModeDebug( "connected account_id=%d account_type=%s transport=%s conn_id=%s conn_reused=%v conn_idle_ms=%d conn_age_ms=%d upstream_pings=%d conn_pick_ms=%d queue_wait_ms=%d has_previous_response_id=%v", account.ID, @@ -393,35 +367,11 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( var observePelicanFrame func([]byte) error var releasePelican func() - pelicanPreparationCtx := ctx - ctx, observePelicanFrame, releasePelican, err = preparePelicanWSSend(ctx, account, wsURL, payload, borrowTurn.borrowed()) + ctx, observePelicanFrame, releasePelican, err = preparePelicanWSSend(ctx, account, wsURL, payload) if err != nil { return nil, err } defer releasePelican() - if IsPelicanGeneration(ctx) && borrowTurn != nil && !borrowTurn.diagnostic && !time.Now().Before(borrowTurn.expires) { - // No prompt has been written. Release the expired route and all execution - // resources, then perform the ordinary preparation on this same account. - releasePelican() - lease.MarkBroken() - lease.Release() - borrowTurn.finish(s.getOpenAIWSConnPool(), nil, ErrCodexGatewayBorrowChanged) - return s.forwardOpenAIWSV2(pelicanPreparationCtx, c, account, reqBody, clientPromptCacheKey, executionScope, token, decision, - isCodexCLI, reqStream, originalModel, mappedModel, startTime, attempt, lastFailureReason, agentTaskRecoveryTried) - } - if IsPelicanGeneration(ctx) && borrowTurn != nil && !borrowTurn.diagnostic { - prepared, prepareErr := codexGatewayBorrowWSRequest(ctx, wsURL, wsHeaders) - if prepareErr != nil { - return nil, codexGatewayBorrowWSPreparationError(c, account, prepareErr) - } - borrowed, application, applyErr := s.gatewayBorrow.Apply(prepared, account, openAIWSPayloadString(payload, "model"), proxyURL, nil, true) - if applyErr != nil { - return nil, codexGatewayBorrowWSPreparationError(c, account, applyErr) - } - if application == nil || !application.Applied || !application.ExpiresAt.Equal(borrowTurn.expires) || !slices.Equal(prepared.Header.Values("Cookie"), borrowed.Header.Values("Cookie")) { - return nil, codexGatewayBorrowWSPreparationError(c, account, ErrCodexGatewayBorrowChanged) - } - } if err := s.performOpenAIWSGeneratePrewarm( ctx, lease, @@ -442,9 +392,6 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( } } - if err := consumeBorrowDiagnosticRequest(ctx); err != nil { - return nil, err - } if capture := pelicanCaptureFromContext(ctx); capture != nil { capture.mu.Lock() if len(capture.attempts) > 0 { @@ -455,12 +402,6 @@ func (s *OpenAIGatewayService) forwardOpenAIWSV2( } capture.mu.Unlock() } - usageContext := ctx - if borrowTurn != nil && borrowTurn.usage != nil { - usageContext = context.WithValue(ctx, codexBorrowUsageContextKey{}, borrowTurn.usage) - } - usageTracker := s.gatewayBorrow.beginUsage(usageContext, account.ID, mappedModel, "ws", borrowTurn.borrowed()) - defer func() { usageTracker.finish(borrowErr) }() if err := lease.WriteJSONWithContextTimeout(ctx, payload, s.openAIWSWriteTimeout()); err != nil { lease.MarkBroken() logOpenAIWSModeInfo( @@ -686,7 +627,6 @@ readLoop: } message, readErr = lease.ReadMessageWithContextTimeout(upstreamReadCtx, currentReadTimeout) if readErr == nil { - usageTracker.observe(message) if observeErr := observePelicanFrame(message); observeErr != nil { lease.MarkBroken() return nil, observeErr @@ -794,7 +734,6 @@ readLoop: continue } responseModelObserver.ObserveOpenAI(message, eventType) - borrowTurn.observe(message) eventCount++ if firstEventType == "" { firstEventType = eventType diff --git a/backend/internal/service/openai_ws_http_bridge.go b/backend/internal/service/openai_ws_http_bridge.go index b13aa10d5b66..e77c0a3e1820 100644 --- a/backend/internal/service/openai_ws_http_bridge.go +++ b/backend/internal/service/openai_ws_http_bridge.go @@ -600,7 +600,7 @@ func (s *OpenAIGatewayService) proxyOpenAIWSHTTPBridgeTurn( } turnStart := time.Now() - resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL, strings.TrimSpace(gjson.GetBytes(body, "model").String())) + resp, err := s.doOpenAICodexUpstream(upstreamReq, account, proxyURL) if err != nil { if turn == 1 { return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, true) @@ -649,7 +649,7 @@ func (s *OpenAIGatewayService) proxyOpenAIWSHTTPBridgeTurn( return nil, err } applyBridgeOwnedTurnState(retryReq) - resp, err = s.doOpenAICodexUpstream(retryReq, account, proxyURL, strings.TrimSpace(gjson.GetBytes(body, "model").String())) + resp, err = s.doOpenAICodexUpstream(retryReq, account, proxyURL) if err != nil { return nil, s.handleOpenAIUpstreamTransportError(ctx, c, account, err, true) } diff --git a/backend/internal/service/openai_ws_v2_passthrough_adapter.go b/backend/internal/service/openai_ws_v2_passthrough_adapter.go index 0f17c3398cd3..a5b3688007de 100644 --- a/backend/internal/service/openai_ws_v2_passthrough_adapter.go +++ b/backend/internal/service/openai_ws_v2_passthrough_adapter.go @@ -888,7 +888,6 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( turnState = strings.TrimSpace(c.GetHeader(openAIWSTurnStateHeader)) turnMetadata = strings.TrimSpace(c.GetHeader(openAIWSTurnMetadataHeader)) } - ctx = withCodexBorrowServiceTier(ctx, gjson.GetBytes(firstClientMessage, "service_tier").String()) headers, _, buildHdrErr := s.buildOpenAIWSHeaders( ctx, c, @@ -909,34 +908,6 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( if account.ProxyID != nil && account.Proxy != nil { proxyURL = account.Proxy.URL() } - borrowRequest, borrowRequestErr := codexGatewayBorrowWSRequest(ctx, wsURL, headers) - if borrowRequestErr != nil { - return borrowRequestErr - } - knownBorrowAnchor := false - if requestPreviousResponseID != "" && s.gatewayBorrow != nil { - if key, keyErr := codexGatewayBorrowWSKey(c, account, headers, originalFirstClientMessage, initialUpstreamModel, proxyURL); keyErr == nil { - exact, knownResponse := s.gatewayBorrow.wsAnchors.knowsContinuation(key, requestPreviousResponseID) - knownBorrowAnchor = exact || knownResponse - } - } - if requestPreviousResponseID != "" && (knownBorrowAnchor || s.codexGatewayBorrowWSSelected(account, initialUpstreamModel, borrowRequest)) { - // A direct passthrough socket closes with its client. A reconnect must - // not acquire a new cookie and replay a continuation on a new socket. - return NewOpenAIContinuationStateUnavailableError(http.StatusBadRequest, nil, nil) - } - borrowTurn, borrowPrepareErr := s.prepareCodexGatewayBorrowWSTurn(ctx, c, account, wsURL, headers, - originalFirstClientMessage, initialUpstreamModel, requestPreviousResponseID, proxyURL) - if borrowPrepareErr != nil { - return codexGatewayBorrowWSPreparationError(c, account, borrowPrepareErr) - } - var borrowSession *codexGatewayBorrowWSDirectSession - if borrowTurn != nil { - headers = cloneHeader(borrowTurn.headers) - borrowSession = &codexGatewayBorrowWSDirectSession{turn: borrowTurn} - defer borrowSession.close(s.getOpenAIWSConnPool()) - } - dialer := s.getOpenAIWSPassthroughDialer() if dialer == nil { return errors.New("openai ws passthrough dialer is nil") @@ -1011,13 +982,8 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( if !ok { return errors.New("openai ws passthrough upstream connection does not support frame relay") } - usageConn := &codexBorrowUsageFrameConn{FrameConn: upstreamFrameConn, service: s.gatewayBorrow, accountID: account.ID, applied: borrowTurn.borrowed()} - if borrowTurn != nil { - usageConn.prepared = borrowTurn.usage - } - defer usageConn.finish() relayUpstreamFrameConn := &openAIWSPassthroughFirstOutputFrameConn{ - inner: usageConn, + inner: upstreamFrameConn, activeReadTimeout: s.openAIWSPassthroughIdleTimeout(), deadlineChanged: make(chan struct{}, 1), resolveDeadline: func(payload []byte) openAIWSPassthroughFirstOutputDeadline { @@ -1074,7 +1040,6 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( return payload, nil, nil } eventType := strings.TrimSpace(gjson.GetBytes(payload, "type").String()) - rawBorrowClientFrame := payload isResponseCreate := eventType == "response.create" // Request integrity snapshot of the client frame before any rewrite in // this filter (response.create only; other frames carry no compared field). @@ -1246,9 +1211,6 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( // 覆盖（Store(nil)），因为 OpenAI 上游对该帧实际不传 // service_tier 时按 default 处理，billing 应如实反映。 if policyErr == nil && blocked == nil && isResponseCreate { - if borrowErr := borrowSession.check(c, account, headers, rawBorrowClientFrame, model, proxyURL); borrowErr != nil { - return payload, nil, borrowErr - } usageMeta.updateFromResponseCreate(out, model, requestModelForThisFrame) responseCreateAtCopy := responseCreateAt acceptedTurnStartedAt.Store(&responseCreateAtCopy) @@ -1386,7 +1348,6 @@ func (s *OpenAIGatewayService) proxyResponsesWebSocketV2PassthroughAttempt( Duration: turn.Duration, FirstTokenMs: turn.FirstTokenMs, } - borrowSession.completed(turnResult) logOpenAIWSV2Passthrough( "relay_turn_completed account_id=%d turn=%d request_id=%s terminal_event=%s turn_requested_model=%s turn_upstream_model=%s duration_ms=%d first_token_ms=%d input_tokens=%d output_tokens=%d cache_read_tokens=%d", account.ID, diff --git a/backend/internal/service/ops_upstream_context.go b/backend/internal/service/ops_upstream_context.go index 8ab3dfc4f614..cf67c6cc0d00 100644 --- a/backend/internal/service/ops_upstream_context.go +++ b/backend/internal/service/ops_upstream_context.go @@ -498,8 +498,8 @@ type OpsUpstreamErrorEvent struct { // Signature recovery evidence is bounded and contains fingerprints only; // bookkeeping must not replace the corresponding upstream failure. SignatureRecovery *ClaudeSignatureRecoveryDiagnostic `json:"signature_recovery,omitempty"` - // Probe evidence belongs to preparation, not to the business upstream status. - BorrowQualification *CodexGatewayBorrowFailureEvidence `json:"borrow_qualification,omitempty"` + // Preserve the raw historical preparation evidence; new attempts never set it. + BorrowQualification json.RawMessage `json:"borrow_qualification,omitempty"` // SkipMonitoring is request-local rule state. It is intentionally excluded // from persisted attempt JSON. The logger consults it only when this event is diff --git a/backend/internal/service/pelican_account_generator.go b/backend/internal/service/pelican_account_generator.go index 0ac2318bb065..3e7f10dd9383 100644 --- a/backend/internal/service/pelican_account_generator.go +++ b/backend/internal/service/pelican_account_generator.go @@ -80,7 +80,6 @@ func PelicanFirstOutputStart(ctx context.Context, fallback time.Time) time.Time type pelicanGenerationCapture struct { account *Account concurrency *ConcurrencyService - openaiGateway *OpenAIGatewayService ginContext *gin.Context fallbackModel string fallbackEffort string @@ -116,8 +115,8 @@ func pelicanCaptureFromContext(ctx context.Context) *pelicanGenerationCapture { } // PreparePelicanHTTPRequest is the shared outbound boundary. Ordinary requests -// retain their exact pointer and context. Borrow probes coordinate separately; -// their old observation marker must never start the work-generation budget. +// retain their exact pointer and context. Only selected-account Pelican sends +// acquire the execution lease and start the generation budget. // The returned completion function owns the execution/account leases until the // response body closes, including error responses and same-account retries. func PreparePelicanHTTPRequest(req *http.Request, accountID int64, accountConcurrency int, transport string) (*http.Request, func(*http.Response, error) (*http.Response, error), error) { @@ -126,50 +125,20 @@ func PreparePelicanHTTPRequest(req *http.Request, accountID int64, accountConcur return req, finish, nil } capture := pelicanCaptureFromContext(req.Context()) - if capture == nil || capture.account.ID != accountID || IsCodexGatewayBorrowObservation(req.Context()) { + if capture == nil || capture.account.ID != accountID { return req, finish, nil } - var ctx context.Context - var release func() - for { - requestCtx := req.Context() - var acquireErr error - var releaseExecution func() - ctx, releaseExecution, acquireErr = AcquirePelicanExecution(requestCtx, accountID) - if acquireErr != nil { - return nil, finish, acquireErr - } - releaseAccount, acquireErr := acquirePelicanBusinessAccountSlot(ctx, capture.concurrency, accountID, accountConcurrency) - if acquireErr != nil { - releaseExecution() - return nil, finish, acquireErr - } - var releaseOnce sync.Once - release = func() { releaseOnce.Do(func() { releaseAccount(); releaseExecution() }) } - // Admission can outlast a borrowed cookie. Recheck the prepared cache - // binding after all waits, without a nested probe under this lease. - prepared, hasPreparation := requestCtx.Value(codexGatewayBorrowHTTPPreparationContextKey{}).(codexGatewayBorrowHTTPPreparation) - if hasPreparation && capture.openaiGateway != nil { - checked, prepareErr := capture.openaiGateway.applyCodexGatewayBorrowHTTP(req.WithContext(ctx), capture.account, prepared.model, prepared.proxy) - if prepareErr != nil { - release() - if prepared.application != nil && !time.Now().Before(prepared.application.ExpiresAt) && - capture.openaiGateway.codexGatewayBorrowHTTPConfigured(capture.account, prepared.model) && errors.Is(prepareErr, ErrCodexGatewayBorrowChanged) { - // Refresh an expired route through the normal preparation path - // after releasing every account/global execution resource. - refreshed, _, refreshErr := capture.openaiGateway.prepareCodexGatewayBorrowHTTP(requestCtx, req.WithContext(requestCtx), capture.account, prepared.model, prepared.proxy) - if refreshErr != nil { - return nil, finish, refreshErr - } - req = refreshed - continue - } - return nil, finish, prepareErr - } - req = checked - } - break + ctx, releaseExecution, acquireErr := AcquirePelicanExecution(req.Context(), accountID) + if acquireErr != nil { + return nil, finish, acquireErr } + releaseAccount, acquireErr := acquirePelicanBusinessAccountSlot(ctx, capture.concurrency, accountID, accountConcurrency) + if acquireErr != nil { + releaseExecution() + return nil, finish, acquireErr + } + var releaseOnce sync.Once + release := func() { releaseOnce.Do(func() { releaseAccount(); releaseExecution() }) } req = req.WithContext(ctx) fallbackModel := capture.fallbackModel if capture.ginContext != nil { @@ -190,9 +159,8 @@ func PreparePelicanHTTPRequest(req *http.Request, accountID int64, accountConcur firstOutputPolicy, _ := ctx.Value(pelicanFirstOutputPolicyContextKey{}).(*pelicanFirstOutputPolicy) ctx = firstOutputPolicy.arm(ctx) req = req.WithContext(ctx) - borrowApplied := codexBorrowHTTPApplied(req) invocation := PelicanInvocation{Platform: capture.account.Platform, Model: model, Effort: effort, - Endpoint: req.URL.Scheme + "://" + req.URL.Host + req.URL.EscapedPath(), Protocol: protocol, Transport: transport, BorrowApplied: borrowApplied} + Endpoint: req.URL.Scheme + "://" + req.URL.Host + req.URL.EscapedPath(), Protocol: protocol, Transport: transport} RecordPelicanInvocation(ctx, invocation) attempt := &pelicanHTTPAttempt{protocol: protocol, model: model, effort: effort, dispatched: true} capture.mu.Lock() @@ -215,13 +183,13 @@ func PreparePelicanHTTPRequest(req *http.Request, accountID int64, accountConcur // Gateway error readers can have a smaller diagnostic cap. Capture the // actual error body first so the administrator keeps its complete text. if resp.StatusCode >= http.StatusBadRequest { - raw, readErr := io.ReadAll(io.LimitReader(resp.Body, codexGatewayBorrowPelicanMaxBody+1)) + raw, readErr := io.ReadAll(io.LimitReader(resp.Body, pelicanMaxBody+1)) _ = resp.Body.Close() _, _ = attempt.raw.Write(raw) attempt.err = readErr release() resp.Body = io.NopCloser(bytes.NewReader(raw)) - if int64(len(raw)) > codexGatewayBorrowPelicanMaxBody { + if int64(len(raw)) > pelicanMaxBody { attempt.err = errors.New("upstream response exceeds the pelican record size limit") } return resp, sendErr @@ -259,7 +227,7 @@ func acquirePelicanBusinessAccountSlot(ctx context.Context, concurrency *Concurr // The WS sender calls this after normal route/socket preparation. Generation // starts separately, immediately before response.create, so a pool wait or // generate:false prewarm does not spend the work-generation budget. -func preparePelicanWSSend(ctx context.Context, account *Account, endpoint string, payload map[string]any, borrowed bool) (context.Context, func([]byte) error, func(), error) { +func preparePelicanWSSend(ctx context.Context, account *Account, endpoint string, payload map[string]any) (context.Context, func([]byte) error, func(), error) { observe := func([]byte) error { return nil } capture := pelicanCaptureFromContext(ctx) if capture == nil || account == nil || capture.account.ID != account.ID { @@ -279,7 +247,7 @@ func preparePelicanWSSend(ctx context.Context, account *Account, endpoint string model := openAIWSPayloadString(payload, "model") effort := gjson.GetBytes(payloadAsJSONBytes(payload), "reasoning.effort").String() RecordPelicanInvocation(ctx, PelicanInvocation{Platform: account.Platform, Model: model, Effort: effort, - Endpoint: endpoint, Protocol: "responses", Transport: "websocket", BorrowApplied: borrowed}) + Endpoint: endpoint, Protocol: "responses", Transport: "websocket"}) attempt := &pelicanHTTPAttempt{protocol: "responses", status: http.StatusOK, model: model, effort: effort, wsFrames: true} capture.mu.Lock() capture.attempts = append(capture.attempts, attempt) @@ -287,7 +255,7 @@ func preparePelicanWSSend(ctx context.Context, account *Account, endpoint string observe = func(message []byte) error { attempt.mu.Lock() defer attempt.mu.Unlock() - if attempt.raw.Len()+len(message)+1 > int(codexGatewayBorrowPelicanMaxBody) { + if attempt.raw.Len()+len(message)+1 > int(pelicanMaxBody) { attempt.err = errors.New("upstream response exceeds the pelican record size limit") return attempt.err } @@ -310,7 +278,7 @@ func (b *pelicanCapturingBody) Read(data []byte) (int, error) { b.attempt.mu.Lock() defer b.attempt.mu.Unlock() if n > 0 { - if b.attempt.raw.Len()+n > int(codexGatewayBorrowPelicanMaxBody) { + if b.attempt.raw.Len()+n > int(pelicanMaxBody) { b.attempt.err = errors.New("upstream response exceeds the pelican record size limit") return 0, b.attempt.err } @@ -350,12 +318,12 @@ func pelicanFinalHTTPRequest(req *http.Request, fallbackModel, fallbackEffort st } } { - raw, readErr := io.ReadAll(io.LimitReader(body, codexGatewayBorrowPelicanMaxBody+1)) + raw, readErr := io.ReadAll(io.LimitReader(body, pelicanMaxBody+1)) _ = body.Close() if readErr != nil { return model, effort, protocol, readErr } - if int64(len(raw)) > codexGatewayBorrowPelicanMaxBody { + if int64(len(raw)) > pelicanMaxBody { return model, effort, protocol, errors.New("pelican request body exceeds its record size limit") } if req.GetBody == nil { @@ -365,7 +333,7 @@ func pelicanFinalHTTPRequest(req *http.Request, fallbackModel, fallbackEffort st req.Body = io.NopCloser(bytes.NewReader(raw)) } if readErr == nil && strings.EqualFold(req.Header.Get("Content-Encoding"), "zstd") { - if decoder, decodeErr := zstd.NewReader(nil, zstd.WithDecoderMaxMemory(codexGatewayBorrowPelicanMaxBody)); decodeErr == nil { + if decoder, decodeErr := zstd.NewReader(nil, zstd.WithDecoderMaxMemory(pelicanMaxBody)); decodeErr == nil { raw, readErr = decoder.DecodeAll(raw, nil) decoder.Close() } @@ -402,8 +370,8 @@ func pelicanFinalHTTPRequest(req *http.Request, fallbackModel, fallbackEffort st // GeneratePelican sends the fixed题面 using the selected account's ordinary // business sender. Only that sender owns credential refresh and compatibility // retries. An UpstreamFailoverError is recorded here and never enters selection. -func (s *AccountTestService) GeneratePelican(ctx context.Context, accountID int64, model, effort string) (*CodexGatewayBorrowPelicanResult, error) { - result := &CodexGatewayBorrowPelicanResult{ModelID: model, Effort: effort, Status: "failed"} +func (s *AccountTestService) GeneratePelican(ctx context.Context, accountID int64, model, effort string) (*PelicanResult, error) { + result := &PelicanResult{ModelID: model, Effort: effort, Status: "failed"} ctx = WithAccountObservation(ctx) if pelicanExecutionFromContext(ctx) == nil { state := newPelicanExecutionState(ctx, sharedPelicanExecution, time.Duration(PelicanDefaultGenerationTimeoutSeconds)*time.Second, time.Now) @@ -438,7 +406,7 @@ func (s *AccountTestService) GeneratePelican(ctx context.Context, accountID int6 } result.Effort = effort capture := &pelicanGenerationCapture{account: account, concurrency: s.pelicanConcurrencyService, - openaiGateway: s.openaiGatewayService, fallbackModel: option.UpstreamModel, fallbackEffort: effort} + fallbackModel: option.UpstreamModel, fallbackEffort: effort} ctx = context.WithValue(ctx, pelicanGenerationContextKey{}, capture) w := httptest.NewRecorder() c, _ := gin.CreateTestContext(w) @@ -447,14 +415,7 @@ func (s *AccountTestService) GeneratePelican(ctx context.Context, accountID int6 // This represents the server's internal generation operation. No client // UA, protocol handshake, API-key subject or group policy is synthesized. sendErr := s.sendPelicanWithAccount(ctx, c, account, model, option.UpstreamModel, effort) - // The gateway keeps client-facing failover text generic, but records the - // complete local borrow cause on this invocation's Gin context. Preserve it - // in administrator task history even when no generation request was sent. - if sendErr != nil && IsCodexGatewayBorrowRequestFailure(sendErr) { - if detail := c.GetString(OpsUpstreamErrorDetailKey); detail != "" && !strings.Contains(sendErr.Error(), detail) { - sendErr = fmt.Errorf("%w\n%s", sendErr, detail) - } - } + last := snapshotPelicanAttempt(capture) if last == nil { result.RawResponse = w.Body.String() @@ -534,7 +495,7 @@ func pelicanOutputBudget(account *Account, upstreamModel string) int64 { func pelicanMessagesBody(model, upstreamModel, effort string, budget int64) []byte { payload := map[string]any{"model": model, "stream": true, "max_tokens": budget, - "messages": []map[string]any{{"role": "user", "content": CodexGatewayBorrowPelicanPrompt}}} + "messages": []map[string]any{{"role": "user", "content": PelicanPrompt}}} if effort != "" { payload["output_config"] = map[string]any{"effort": effort} if claude.EffortUsesAdaptiveThinking(upstreamModel) { @@ -549,7 +510,7 @@ func (s *AccountTestService) sendPelicanWithAccount(ctx context.Context, c *gin. budget := pelicanOutputBudget(account, upstreamModel) switch { case account.IsGemini(), account.Platform == PlatformAntigravity && account.Type != AccountTypeUpstream && isAntigravityGeminiModel(upstreamModel): - payload := map[string]any{"contents": []map[string]any{{"role": "user", "parts": []map[string]any{{"text": CodexGatewayBorrowPelicanPrompt}}}}, + payload := map[string]any{"contents": []map[string]any{{"role": "user", "parts": []map[string]any{{"text": PelicanPrompt}}}}, "generationConfig": map[string]any{"maxOutputTokens": budget}} body, _ := json.Marshal(payload) if account.IsGemini() { @@ -586,7 +547,7 @@ func (s *AccountTestService) sendPelicanWithAccount(ctx context.Context, c *gin. return errors.New("OpenAI-compatible account sender is unavailable") } payload := map[string]any{"model": model, "stream": true, "max_output_tokens": budget, - "input": []map[string]any{{"role": "user", "content": []map[string]any{{"type": "input_text", "text": CodexGatewayBorrowPelicanPrompt}}}}} + "input": []map[string]any{{"role": "user", "content": []map[string]any{{"type": "input_text", "text": PelicanPrompt}}}}} if effort != "" { payload["reasoning"] = map[string]any{"effort": effort} } diff --git a/backend/internal/service/pelican_account_generator_test.go b/backend/internal/service/pelican_account_generator_test.go index 6c76c5257ce4..752f3cb02a30 100644 --- a/backend/internal/service/pelican_account_generator_test.go +++ b/backend/internal/service/pelican_account_generator_test.go @@ -171,7 +171,7 @@ func TestPelicanAccountGeneratorSharedOpenAISettings(t *testing.T) { require.Equal(t, []string{"saved-identity"}, upstream.requests[0].Header[resolveWireCasing("x-account-identity")]) require.Equal(t, target, gjson.GetBytes(upstream.bodies[0], "model").String()) require.False(t, gjson.GetBytes(upstream.bodies[0], "reasoning.effort").Exists(), "omitted effort preserves the model default") - require.Contains(t, string(upstream.bodies[0]), CodexGatewayBorrowPelicanPrompt) + require.Contains(t, string(upstream.bodies[0]), PelicanPrompt) require.Contains(t, string(upstream.bodies[0]), "saved site instructions") require.Greater(t, gjson.GetBytes(upstream.bodies[0], "max_output_tokens").Int(), int64(1024)) require.True(t, IsAccountObservation(upstream.requests[0].Context())) @@ -179,32 +179,6 @@ func TestPelicanAccountGeneratorSharedOpenAISettings(t *testing.T) { } } -func TestPelicanAccountGeneratorKeepsBorrowPreparationCause(t *testing.T) { - account := borrowCoreAccount(2) - upstream := &pelicanGeneratorUpstream{} - svc, _ := newPelicanGeneratorForTest(account, upstream) - shots := 0 - borrow := newBorrowCoreTest(t, func(req *http.Request, _ string, _ int64, _ int, _ *tlsfingerprint.Profile) (*http.Response, error) { - shots++ - state := "first" - if req.Header.Get("X-Codex-Turn-State") != "" { - state = "changed" - } - return borrowCoreResponse("gpt-6-astra", "OK", state), nil - }, account) - borrowCoreCandidate(borrow, time.Now().Add(codexGatewayBorrowTTL)) - svc.openaiGatewayService.SetCodexGatewayBorrowService(borrow) - result, err := svc.GeneratePelican(context.Background(), account.ID, "gpt-6-astra", "low") - require.Error(t, err) - require.True(t, IsCodexGatewayBorrowRequestFailure(err)) - require.Contains(t, result.Error, "target_validation/target_state_changed") - require.Contains(t, result.Error, "CODEX_GATEWAY_BORROW_UNAVAILABLE") - require.Equal(t, "failed", result.Status) - require.Empty(t, result.RawAnswer) - require.Empty(t, upstream.requests, "a rejected qualification never reaches generation") - require.Equal(t, 2, shots) -} - func TestPelicanAccountGeneratorHTTPFailureKeepsSelectedAccountAndRawBody(t *testing.T) { raw := `{"error":{"type":"rate_limit_error","message":"` + strings.Repeat("complete original detail ", 2000) + `"}}` account := &Account{ID: 28, Platform: PlatformOpenAI, Type: AccountTypeAPIKey, Credentials: map[string]any{"api_key": "local-key", "base_url": "http://fake.example"}} @@ -280,7 +254,7 @@ func TestPelicanAccountGeneratorNativeTextDispatch(t *testing.T) { require.Equal(t, "work", result.RawAnswer) require.Len(t, upstream.requests, 1) require.Contains(t, upstream.requests[0].URL.Path, scenario.path) - require.Contains(t, string(upstream.bodies[0]), CodexGatewayBorrowPelicanPrompt) + require.Contains(t, string(upstream.bodies[0]), PelicanPrompt) }) } } @@ -299,7 +273,7 @@ func TestPelicanParsersKeepJSONDataURLsAndPrettyWSFrames(t *testing.T) { require.Equal(t, "whole", text) } -func TestPelicanHTTPRequestOrdinaryAndPreparationPurposeAreNoops(t *testing.T) { +func TestPelicanHTTPRequestOrdinaryRequestsAreNoops(t *testing.T) { req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, "http://fake.example/responses", strings.NewReader(`{}`)) require.NoError(t, err) prepared, finish, err := PreparePelicanHTTPRequest(req, 17, 1, "http") @@ -309,18 +283,7 @@ func TestPelicanHTTPRequestOrdinaryAndPreparationPurposeAreNoops(t *testing.T) { actual, err := finish(resp, nil) require.NoError(t, err) require.Same(t, resp, actual) - capture := &pelicanGenerationCapture{account: &Account{ID: 17}} - state := newPelicanExecutionState(context.Background(), newPelicanExecutionCoordinator(1), time.Minute, time.Now) - defer state.finish() - ctx := context.WithValue(state.parent, pelicanExecutionContextKey{}, state) - ctx = context.WithValue(ctx, pelicanGenerationContextKey{}, capture) - ctx = WithCodexGatewayBorrowObservation(ctx) - req = req.WithContext(ctx) - prepared, _, err = PreparePelicanHTTPRequest(req, 17, 1, "http") - require.NoError(t, err) - require.Same(t, req, prepared) - require.True(t, state.generationAt.IsZero(), "borrow preparation must keep its own clock") - require.Empty(t, capture.attempts) + } func TestPelicanFirstOutputPolicyStartsAfterQueueAndPreservesNilReplay(t *testing.T) { diff --git a/backend/internal/service/pelican_background.go b/backend/internal/service/pelican_background.go index 8b2f47755a51..afe0163324c7 100644 --- a/backend/internal/service/pelican_background.go +++ b/backend/internal/service/pelican_background.go @@ -34,7 +34,7 @@ type PelicanResultSummary struct { } type PelicanTaskSnapshot struct { - *CodexGatewayBorrowTestTask + *PelicanTestTask Counts map[string]int `json:"counts"` } @@ -65,7 +65,7 @@ type pelicanBackgroundTask struct { const pelicanInterruptedMessage = "process stopped before the test finished; this task will not be replayed" -func pelicanResultSummary(result *CodexGatewayBorrowTestResult, phase string) *PelicanResultSummary { +func pelicanResultSummary(result *PelicanTestResult, phase string) *PelicanResultSummary { return &PelicanResultSummary{ID: result.ID, TaskID: result.TaskID, Ordinal: result.Ordinal, AccountID: result.AccountID, AccountName: result.AccountName, Platform: result.Platform, ModelID: result.ModelID, Effort: result.Effort, Status: result.Status, Phase: phase, @@ -75,16 +75,15 @@ func pelicanResultSummary(result *CodexGatewayBorrowTestResult, phase string) *P PreparationDurationMS: result.PreparationDurationMS, GenerationDurationMS: result.GenerationDurationMS, ExpiresAt: result.ExpiresAt} } -func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, createdBy int64, request CodexGatewayBorrowTestRequest, catalog *PelicanTestCatalog) (*PelicanTaskSnapshot, error) { - request.Standalone = true - normalized, err := normalizeCodexGatewayBorrowTestRequest(request, r.now().UTC()) +func (r *PelicanTestRunner) StartBackground(ctx context.Context, createdBy int64, request PelicanTestRequest, catalog *PelicanTestCatalog) (*PelicanTaskSnapshot, error) { + normalized, err := normalizePelicanTestRequest(request, r.now().UTC()) if err != nil { return nil, err } // Legacy normalization deduplicates; the new workbench must reject conflicting // efforts instead of silently dropping a later row. if len(normalized.Targets) != len(request.Targets) { - return nil, fmt.Errorf("%w: one effort per account/model; edit the existing combination", ErrCodexGatewayBorrowTestInvalidRequest) + return nil, fmt.Errorf("%w: one effort per account/model; edit the existing combination", ErrPelicanTestInvalidRequest) } if err := ctx.Err(); err != nil { return nil, err @@ -104,12 +103,12 @@ func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, crea // Replays compare their immutable normalized parameters before consulting // today's account capabilities, and read summaries rather than every raw work. if existing, readErr := r.TaskSnapshot(ctx, normalized.ClientTaskID); readErr == nil { - if existing.RequestHash != codexGatewayBorrowTestRequestHash(normalized) { - return nil, ErrCodexGatewayBorrowTestReplayConflict + if existing.RequestHash != pelicanTestRequestHash(normalized) { + return nil, ErrPelicanTestReplayConflict } existing.Replayed = true return existing, nil - } else if !errors.Is(readErr, ErrCodexGatewayBorrowTestNotFound) { + } else if !errors.Is(readErr, ErrPelicanTestNotFound) { return nil, readErr } if catalog == nil { @@ -131,7 +130,7 @@ func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, crea } copyTask := *task copyTask.Results = nil - snapshot := &PelicanTaskSnapshot{CodexGatewayBorrowTestTask: ©Task, Counts: map[string]int{"pending": task.Total}} + snapshot := &PelicanTaskSnapshot{PelicanTestTask: ©Task, Counts: map[string]int{"pending": task.Total}} // No HTTP request values, Gin writer or observer cancellation enters the run. runCtx, cancel := context.WithCancel(r.stopCtx) control := &pelicanBackgroundTask{cancel: cancel, phases: make(map[string]*PelicanResultSummary)} @@ -145,7 +144,7 @@ func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, crea go func() { defer r.backgroundRunning.Done() defer cancel() - runErr := r.Run(runCtx, task, func(event CodexGatewayBorrowTestEvent) { + runErr := r.Run(runCtx, task, func(event PelicanTestEvent) { if event.Result == nil { return } @@ -156,7 +155,7 @@ func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, crea }) if runErr != nil { slog.Error("pelican_background_task_failed", "task_id", task.ID, "error", runErr) - if !errors.Is(runErr, ErrCodexGatewayBorrowTestNotFound) { + if !errors.Is(runErr, ErrPelicanTestNotFound) { writeCtx, cancelWrite := context.WithTimeout(context.Background(), 10*time.Second) _ = r.repo.FinishTask(writeCtx, task.ID, "incomplete", runErr.Error(), r.now().UTC()) cancelWrite() @@ -169,7 +168,7 @@ func (r *CodexGatewayBorrowTestRunner) StartBackground(ctx context.Context, crea return snapshot, nil } -func (r *CodexGatewayBorrowTestRunner) TaskSnapshot(ctx context.Context, id string) (*PelicanTaskSnapshot, error) { +func (r *PelicanTestRunner) TaskSnapshot(ctx context.Context, id string) (*PelicanTaskSnapshot, error) { if err := r.initialize(ctx); err != nil { return nil, err } @@ -180,7 +179,7 @@ func (r *CodexGatewayBorrowTestRunner) TaskSnapshot(ctx context.Context, id stri return reader.GetTaskSnapshot(ctx, id) } -func (r *CodexGatewayBorrowTestRunner) TaskResults(ctx context.Context, id string, filter PelicanResultFilter) (*PelicanResultPage, error) { +func (r *PelicanTestRunner) TaskResults(ctx context.Context, id string, filter PelicanResultFilter) (*PelicanResultPage, error) { if _, err := r.TaskSnapshot(ctx, id); err != nil { return nil, err } @@ -205,7 +204,7 @@ func (r *CodexGatewayBorrowTestRunner) TaskResults(ctx context.Context, id strin return page, nil } -func (r *CodexGatewayBorrowTestRunner) CancelBackground(ctx context.Context, id string) (*PelicanTaskSnapshot, error) { +func (r *PelicanTestRunner) CancelBackground(ctx context.Context, id string) (*PelicanTaskSnapshot, error) { r.submissionMu.Lock() defer r.submissionMu.Unlock() snapshot, err := r.TaskSnapshot(ctx, id) @@ -226,7 +225,7 @@ func (r *CodexGatewayBorrowTestRunner) CancelBackground(ctx context.Context, id return r.TaskSnapshot(ctx, id) } -func (r *CodexGatewayBorrowTestRunner) TaskResult(ctx context.Context, taskID, resultID string) (*CodexGatewayBorrowTestResult, error) { +func (r *PelicanTestRunner) TaskResult(ctx context.Context, taskID, resultID string) (*PelicanTestResult, error) { if err := r.initialize(ctx); err != nil { return nil, err } @@ -235,20 +234,20 @@ func (r *CodexGatewayBorrowTestRunner) TaskResult(ctx context.Context, taskID, r return nil, err } if result.TaskID != taskID { - return nil, ErrCodexGatewayBorrowTestNotFound + return nil, ErrPelicanTestNotFound } return result, nil } // Validate exactly the capabilities used again by the native generator. Reads // are local; paused/error/limited accounts stay explicitly selectable. -func (s *PelicanTestCatalog) ValidateTargets(ctx context.Context, targets []CodexGatewayBorrowTestTarget) error { +func (s *PelicanTestCatalog) ValidateTargets(ctx context.Context, targets []PelicanTestTarget) error { ids := make([]int64, 0, len(targets)) for _, target := range targets { ids = append(ids, target.AccountID) } if len(ids) == 0 || len(ids) > pelicanTestCatalogMaxAccounts { - return ErrCodexGatewayBorrowTestInvalidRequest + return ErrPelicanTestInvalidRequest } accounts, err := s.accounts.GetByIDs(ctx, ids) if err != nil { @@ -263,10 +262,10 @@ func (s *PelicanTestCatalog) ValidateTargets(ctx context.Context, targets []Code for _, target := range targets { option := PelicanTestModelOptions(byID[target.AccountID], target.ModelID) if !option.TextSupported { - return fmt.Errorf("%w: account %d / %s: %s", ErrCodexGatewayBorrowTestInvalidRequest, target.AccountID, target.ModelID, option.CapabilityReason) + return fmt.Errorf("%w: account %d / %s: %s", ErrPelicanTestInvalidRequest, target.AccountID, target.ModelID, option.CapabilityReason) } if target.Effort != "" && !slices.Contains(option.ReasoningEfforts, strings.ToLower(strings.TrimSpace(target.Effort))) { - return fmt.Errorf("%w: account %d / %s does not support effort %q", ErrCodexGatewayBorrowTestInvalidRequest, target.AccountID, target.ModelID, target.Effort) + return fmt.Errorf("%w: account %d / %s does not support effort %q", ErrPelicanTestInvalidRequest, target.AccountID, target.ModelID, target.Effort) } } return nil diff --git a/backend/internal/service/pelican_background_test.go b/backend/internal/service/pelican_background_test.go index 5f23325f48d6..80249d920e12 100644 --- a/backend/internal/service/pelican_background_test.go +++ b/backend/internal/service/pelican_background_test.go @@ -15,28 +15,28 @@ import ( ) type pelicanBackgroundRepo struct { - CodexGatewayBorrowTestRepository + PelicanTestRepository mu sync.Mutex - tasks map[string]*CodexGatewayBorrowTestTask + tasks map[string]*PelicanTestTask fullReads int } -func clonePelicanTestTask(task *CodexGatewayBorrowTestTask) *CodexGatewayBorrowTestTask { +func clonePelicanTestTask(task *PelicanTestTask) *PelicanTestTask { copy := *task - copy.Results = make([]*CodexGatewayBorrowTestResult, 0, len(task.Results)) + copy.Results = make([]*PelicanTestResult, 0, len(task.Results)) for _, row := range task.Results { item := *row copy.Results = append(copy.Results, &item) } return © } -func (r *pelicanBackgroundRepo) Create(_ context.Context, task *CodexGatewayBorrowTestTask) (*CodexGatewayBorrowTestTask, bool, error) { +func (r *pelicanBackgroundRepo) Create(_ context.Context, task *PelicanTestTask) (*PelicanTestTask, bool, error) { r.mu.Lock() defer r.mu.Unlock() for _, old := range r.tasks { if old.ClientTaskID == task.ClientTaskID { if old.RequestHash != task.RequestHash { - return nil, true, ErrCodexGatewayBorrowTestReplayConflict + return nil, true, ErrPelicanTestReplayConflict } return clonePelicanTestTask(old), true, nil } @@ -44,16 +44,16 @@ func (r *pelicanBackgroundRepo) Create(_ context.Context, task *CodexGatewayBorr r.tasks[task.ID] = clonePelicanTestTask(task) return clonePelicanTestTask(task), false, nil } -func (r *pelicanBackgroundRepo) Get(_ context.Context, id string) (*CodexGatewayBorrowTestTask, error) { +func (r *pelicanBackgroundRepo) Get(_ context.Context, id string) (*PelicanTestTask, error) { r.mu.Lock() defer r.mu.Unlock() r.fullReads++ if task := r.tasks[id]; task != nil { return clonePelicanTestTask(task), nil } - return nil, ErrCodexGatewayBorrowTestNotFound + return nil, ErrPelicanTestNotFound } -func (r *pelicanBackgroundRepo) GetResult(_ context.Context, id string) (*CodexGatewayBorrowTestResult, error) { +func (r *pelicanBackgroundRepo) GetResult(_ context.Context, id string) (*PelicanTestResult, error) { r.mu.Lock() defer r.mu.Unlock() for _, task := range r.tasks { @@ -64,7 +64,7 @@ func (r *pelicanBackgroundRepo) GetResult(_ context.Context, id string) (*CodexG } } } - return nil, ErrCodexGatewayBorrowTestNotFound + return nil, ErrPelicanTestNotFound } func (r *pelicanBackgroundRepo) StartTask(_ context.Context, id string, now time.Time) error { r.mu.Lock() @@ -73,7 +73,7 @@ func (r *pelicanBackgroundRepo) StartTask(_ context.Context, id string, now time r.tasks[id].StartedAt = &now return nil } -func (r *pelicanBackgroundRepo) SaveResult(_ context.Context, item *CodexGatewayBorrowTestResult) error { +func (r *pelicanBackgroundRepo) SaveResult(_ context.Context, item *PelicanTestResult) error { r.mu.Lock() defer r.mu.Unlock() task := r.tasks[item.TaskID] @@ -128,10 +128,10 @@ func (r *pelicanBackgroundRepo) GetTaskSnapshot(_ context.Context, id string) (* counts["no_preview"]++ } } - return &PelicanTaskSnapshot{CodexGatewayBorrowTestTask: ©, Counts: counts}, nil + return &PelicanTaskSnapshot{PelicanTestTask: ©, Counts: counts}, nil } } - return nil, ErrCodexGatewayBorrowTestNotFound + return nil, ErrPelicanTestNotFound } func (r *pelicanBackgroundRepo) ListTaskResults(_ context.Context, id string, filter PelicanResultFilter) (*PelicanResultPage, error) { r.mu.Lock() @@ -144,20 +144,20 @@ func (r *pelicanBackgroundRepo) ListTaskResults(_ context.Context, id string, fi return page, nil } -func newPelicanBackgroundFixture(t *testing.T, generate func(context.Context, int64, string, string) (*CodexGatewayBorrowPelicanResult, error)) (*CodexGatewayBorrowTestRunner, *pelicanBackgroundRepo, *PelicanTestCatalog) { +func newPelicanBackgroundFixture(t *testing.T, generate func(context.Context, int64, string, string) (*PelicanResult, error)) (*PelicanTestRunner, *pelicanBackgroundRepo, *PelicanTestCatalog) { t.Helper() - repo := &pelicanBackgroundRepo{tasks: map[string]*CodexGatewayBorrowTestTask{}} - runner := newCodexGatewayBorrowTestRunner(repo, nil, nil) - runner.SetStandaloneGenerator(codexGatewayBorrowFakeGenerator{generate: generate}) + repo := &pelicanBackgroundRepo{tasks: map[string]*PelicanTestTask{}} + runner := newPelicanTestRunner(repo, nil, nil) + runner.generator = pelicanFakeGenerator{generate: generate} t.Cleanup(runner.Stop) - catalog := &PelicanTestCatalog{now: time.Now, accounts: &pelicanTestCatalogReaderStub{accounts: []*Account{borrowCoreAccount(1)}}} + catalog := &PelicanTestCatalog{now: time.Now, accounts: &pelicanTestCatalogReaderStub{accounts: []*Account{pelicanFixtureAccount(1)}}} return runner, repo, catalog } func TestPelicanBackgroundSurvivesCallerAndReplaysOnlySnapshot(t *testing.T) { entered, release := make(chan struct{}), make(chan struct{}) var calls atomic.Int32 - runner, repo, catalog := newPelicanBackgroundFixture(t, func(ctx context.Context, _ int64, _ string, effort string) (*CodexGatewayBorrowPelicanResult, error) { + runner, repo, catalog := newPelicanBackgroundFixture(t, func(ctx context.Context, _ int64, _ string, effort string) (*PelicanResult, error) { require.Empty(t, effort) calls.Add(1) close(entered) @@ -166,10 +166,10 @@ func TestPelicanBackgroundSurvivesCallerAndReplaysOnlySnapshot(t *testing.T) { case <-ctx.Done(): return nil, ctx.Err() } - return &CodexGatewayBorrowPelicanResult{Status: "complete", RawAnswer: "saved work", RawResponse: "complete raw response"}, nil + return &PelicanResult{Status: "complete", RawAnswer: "saved work", RawResponse: "complete raw response"}, nil }) caller, cancel := context.WithCancel(context.Background()) - request := CodexGatewayBorrowTestRequest{ClientTaskID: codexGatewayBorrowV7(t), Targets: []CodexGatewayBorrowTestTarget{{AccountID: 1, ModelID: "gpt-6-astra"}}} + request := PelicanTestRequest{ClientTaskID: pelicanV7(t), Targets: []PelicanTestTarget{{AccountID: 1, ModelID: "gpt-6-astra"}}} task, err := runner.StartBackground(caller, 7, request, catalog) require.NoError(t, err) <-entered @@ -181,7 +181,7 @@ func TestPelicanBackgroundSurvivesCallerAndReplaysOnlySnapshot(t *testing.T) { require.EqualValues(t, 1, calls.Load()) request.GenerationTimeoutSeconds = 900 _, err = runner.StartBackground(context.Background(), 7, request, catalog) - require.ErrorIs(t, err, ErrCodexGatewayBorrowTestReplayConflict) + require.ErrorIs(t, err, ErrPelicanTestReplayConflict) page, err := runner.TaskResults(context.Background(), task.ID, PelicanResultFilter{}) require.NoError(t, err) require.Len(t, page.Items, 1) @@ -207,12 +207,12 @@ func TestPelicanBackgroundExplicitCancelAndShutdownInterrupt(t *testing.T) { for _, shutdown := range []bool{false, true} { t.Run(map[bool]string{false: "explicit_cancel", true: "shutdown"}[shutdown], func(t *testing.T) { entered := make(chan struct{}) - runner, _, catalog := newPelicanBackgroundFixture(t, func(ctx context.Context, _ int64, _ string, _ string) (*CodexGatewayBorrowPelicanResult, error) { + runner, _, catalog := newPelicanBackgroundFixture(t, func(ctx context.Context, _ int64, _ string, _ string) (*PelicanResult, error) { close(entered) <-ctx.Done() - return &CodexGatewayBorrowPelicanResult{RawAnswer: "partial work", RawResponse: "partial response"}, ctx.Err() + return &PelicanResult{RawAnswer: "partial work", RawResponse: "partial response"}, ctx.Err() }) - task, err := runner.StartBackground(context.Background(), 7, CodexGatewayBorrowTestRequest{ClientTaskID: codexGatewayBorrowV7(t), Targets: []CodexGatewayBorrowTestTarget{{AccountID: 1, ModelID: "gpt-6-astra"}}}, catalog) + task, err := runner.StartBackground(context.Background(), 7, PelicanTestRequest{ClientTaskID: pelicanV7(t), Targets: []PelicanTestTarget{{AccountID: 1, ModelID: "gpt-6-astra"}}}, catalog) require.NoError(t, err) <-entered expected := "cancelled" @@ -240,16 +240,16 @@ func TestPelicanBackgroundExplicitCancelAndShutdownInterrupt(t *testing.T) { } func TestPelicanBackgroundRejectsDuplicateAndUnsupportedBeforeCreating(t *testing.T) { - runner, repo, catalog := newPelicanBackgroundFixture(t, func(context.Context, int64, string, string) (*CodexGatewayBorrowPelicanResult, error) { + runner, repo, catalog := newPelicanBackgroundFixture(t, func(context.Context, int64, string, string) (*PelicanResult, error) { t.Error("must not generate") return nil, errors.New("unexpected") }) - for _, targets := range [][]CodexGatewayBorrowTestTarget{ + for _, targets := range [][]PelicanTestTarget{ {{AccountID: 1, ModelID: "gpt-6-astra", Effort: "low"}, {AccountID: 1, ModelID: "gpt-6-astra", Effort: "high"}}, {{AccountID: 1, ModelID: "codex-auto-review"}}, } { - _, err := runner.StartBackground(context.Background(), 7, CodexGatewayBorrowTestRequest{ClientTaskID: codexGatewayBorrowV7(t), Targets: targets}, catalog) - require.ErrorIs(t, err, ErrCodexGatewayBorrowTestInvalidRequest) + _, err := runner.StartBackground(context.Background(), 7, PelicanTestRequest{ClientTaskID: pelicanV7(t), Targets: targets}, catalog) + require.ErrorIs(t, err, ErrPelicanTestInvalidRequest) } require.Empty(t, repo.tasks) } diff --git a/backend/internal/service/pelican_borrow_ws_test.go b/backend/internal/service/pelican_borrow_ws_test.go deleted file mode 100644 index f061685e531f..000000000000 --- a/backend/internal/service/pelican_borrow_ws_test.go +++ /dev/null @@ -1,138 +0,0 @@ -//go:build unit - -package service - -import ( - "context" - "encoding/json" - "net/http/httptest" - "strings" - "testing" - "time" - - "github.com/gin-gonic/gin" - "github.com/stretchr/testify/require" -) - -// Capture the shared pool's actual lease state when the socket closes. A -// one-shot test socket must be evicted while still leased, before it can be -// offered to a waiting business request. -type pelicanBorrowClosingConn struct { - *openAIWSCaptureConn - pool *openAIWSConnPool - accountID int64 - leasedConn *openAIWSConn - closedAfterRelease bool -} - -func (c *pelicanBorrowClosingConn) WriteJSON(ctx context.Context, value any) error { - ap := c.pool.getOrCreateAccountPool(c.accountID) - ap.mu.Lock() - for _, conn := range ap.conns { - if conn.ws == c { - c.leasedConn = conn - break - } - } - ap.mu.Unlock() - return c.openAIWSCaptureConn.WriteJSON(ctx, value) -} - -func (c *pelicanBorrowClosingConn) Close() error { - if c.leasedConn != nil { - c.closedAfterRelease = len(c.leasedConn.leaseCh) > 0 - } - return c.openAIWSCaptureConn.Close() -} - -func TestPelicanBorrowWSUsesAccountWithoutCustomerAnchor(t *testing.T) { - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - t.Run(model, func(t *testing.T) { - svc, account, borrow, upstream := borrowWSFixture(t, 1) - c, _ := gin.CreateTestContext(httptest.NewRecorder()) - capture := &pelicanGenerationCapture{account: account, openaiGateway: svc} - ctx := context.WithValue(WithAccountObservation(context.Background()), pelicanGenerationContextKey{}, capture) - c.Request = httptest.NewRequest("POST", "/internal/admin/pelican-tests", nil).WithContext(ctx) - seedBorrowWSTarget(t, svc, borrow, account, c, model) - headers, _, err := svc.buildOpenAIWSHeaders(ctx, c, account, "target-token", - svc.getOpenAIWSProtocolResolver().Resolve(account), true, "", "", "", model, "") - require.NoError(t, err) - wsURL, err := svc.buildOpenAIResponsesWSURL(account) - require.NoError(t, err) - turn, err := svc.prepareCodexGatewayBorrowWSTurn(ctx, c, account, wsURL, headers, - []byte(`{"model":"`+model+`","input":"one question"}`), model, "", account.Proxy.URL()) - require.NoError(t, err) - require.NotNil(t, turn) - require.True(t, turn.pelicanOneShot) - require.Nil(t, turn.store) - require.Equal(t, account.ID, turn.key.account) - require.Contains(t, turn.headers.Get("Cookie"), "__oailb=shared-native-route") - require.Zero(t, getAPIKeyIDFromContext(c)) - require.Empty(t, upstream.requests, "hot qualification must not prepare another route") - _, err = svc.prepareCodexGatewayBorrowWSTurn(ctx, c, account, wsURL, headers, - []byte(`{"model":"`+model+`"}`), model, "previous-response", account.Proxy.URL()) - require.ErrorContains(t, err, "new conversation") - }) - } -} - -func TestPelicanBorrowWSSharedForwardGeneratesOnceAndReleasesItsSocket(t *testing.T) { - for _, model := range []string{"gpt-6-astra", "gpt-6.1-sol"} { - t.Run(model, func(t *testing.T) { - gateway, account, borrow, probes := borrowWSFixture(t, 1) - gateway.cfg.Gateway.MaxLineSize = defaultMaxLineSize - account.Status, account.Schedulable = StatusError, false - repo := &pelicanGeneratorAccountRepo{accounts: map[int64]*Account{account.ID: account}} - gateway.accountRepo = repo - sender := &AccountTestService{accountRepo: repo, cfg: gateway.cfg, openaiGatewayService: gateway} - seed, _ := gin.CreateTestContext(httptest.NewRecorder()) - seed.Request = httptest.NewRequest("POST", "/internal/admin/pelican-tests", nil) - seedBorrowWSTarget(t, gateway, borrow, account, seed, model) - answer := "" - delta, err := json.Marshal(map[string]any{"type": "response.output_text.delta", "delta": answer}) - require.NoError(t, err) - completed, err := json.Marshal(map[string]any{"type": "response.completed", "response": map[string]any{ - "id": "resp_pelican", "model": model, "status": "completed", - "output": []any{map[string]any{"type": "message", "role": "assistant", "content": []any{map[string]any{"type": "output_text", "text": answer}}}}, - "usage": map[string]any{"input_tokens": 4, "output_tokens": 8}, - }}) - require.NoError(t, err) - conn := &pelicanBorrowClosingConn{openAIWSCaptureConn: &openAIWSCaptureConn{events: [][]byte{delta, completed}}, - pool: gateway.openaiWSPool, accountID: account.ID} - dialer := &borrowWSDialer{conns: []openAIWSClientConn{conn}} - gateway.openaiWSPool.setClientDialerForTest(dialer) - state := newPelicanExecutionState(context.Background(), newPelicanExecutionCoordinator(10), 10*time.Minute, time.Now) - defer state.finish() - ctx := context.WithValue(state.parent, pelicanExecutionContextKey{}, state) - result, err := sender.GeneratePelican(ctx, account.ID, model, "high") - require.NoError(t, err) - require.Equal(t, "complete", result.Status, result.Error) - require.Equal(t, answer, result.RawAnswer) - require.Equal(t, string(delta)+"\n"+string(completed)+"\n", result.RawResponse) - require.Equal(t, []int64{account.ID}, repo.reads) - require.Len(t, conn.writes, 1, "the real shared Forward sends the fixed prompt only once") - wire, err := json.Marshal(conn.writes[0]) - require.NoError(t, err) - require.Contains(t, string(wire), CodexGatewayBorrowPelicanPrompt) - require.Equal(t, model, conn.writes[0]["model"]) - require.Len(t, dialer.headers, 1) - require.Equal(t, "Bearer target-token", dialer.headers[0].Get("Authorization")) - require.Equal(t, "target-chatgpt", dialer.headers[0].Get("Chatgpt-Account-Id")) - require.Contains(t, dialer.headers[0].Get("Cookie"), "__oailb=shared-native-route") - require.Equal(t, []string{account.Proxy.URL()}, dialer.proxies) - require.Empty(t, probes.requests, "qualified routes must not issue extra probes") - snapshot := state.finish() - require.NotNil(t, snapshot.generationStartedAt) - require.Equal(t, "websocket", snapshot.invocation.Transport) - require.Equal(t, "responses", snapshot.invocation.Protocol) - require.Equal(t, model, snapshot.invocation.Model) - require.Equal(t, "high", snapshot.invocation.Effort) - require.True(t, snapshot.invocation.BorrowApplied) - require.True(t, strings.HasPrefix(snapshot.invocation.Endpoint, "wss://")) - require.True(t, conn.closed, "a one-shot admin test must release its borrowed socket") - require.NotNil(t, conn.leasedConn) - require.False(t, conn.closedAfterRelease, "evict before exposing the socket to a waiting business request") - require.Empty(t, borrow.wsAnchors.entries, "admin tests must not create customer continuation anchors") - }) - } -} diff --git a/backend/internal/service/pelican_execution.go b/backend/internal/service/pelican_execution.go index 0ba03fb30211..a25267ebc47b 100644 --- a/backend/internal/service/pelican_execution.go +++ b/backend/internal/service/pelican_execution.go @@ -14,14 +14,11 @@ const ( PelicanMinGenerationTimeoutSeconds = 60 PelicanMaxGenerationTimeoutSeconds = 1800 PelicanExecutionModeAccount = "account" - PelicanExecutionModeLegacyCache = "legacy_cache" ) // Every manual Pelican runner in this process shares the same outbound capacity. -// Preparation and generation acquire it while sending model requests. A target -// STATE probe retains one lease across its two sequential shots, after source -// preparation. No preparation retains a target permit while waiting for a source -// account or the remaining global permits. +// Selected-account requests acquire capacity after preparation and retain it +// until their response is consumed or closed. var sharedPelicanExecution = newPelicanExecutionCoordinator(PelicanExecutionConcurrency) type pelicanExecutionContextKey struct{} @@ -113,13 +110,12 @@ type pelicanExecutionLease struct { // PelicanInvocation records the resolved request at the actual sending boundary. // It deliberately contains no group/client simulation settings or credentials. type PelicanInvocation struct { - Platform string - Model string - Effort string - Endpoint string - Protocol string - Transport string - BorrowApplied bool + Platform string + Model string + Effort string + Endpoint string + Protocol string + Transport string } type pelicanExecutionSnapshot struct { @@ -176,8 +172,8 @@ func CancelPelicanExecution(ctx context.Context, cause error) { } } -// AcquirePelicanExecution coordinates an actual model request, including a -// nested borrow source/target probe, with all other manual Pelican requests. +// AcquirePelicanExecution coordinates an actual model request with all other +// manual Pelican requests. // Call it after route preparation and retain its lease until the response body // has been consumed/closed. Nested wrappers for the same request/account reuse // the lease; a leased context must not be shared by parallel model requests. @@ -256,8 +252,8 @@ func BeginPelicanQueueWait(ctx context.Context) func() { } } -// BeginPelicanPreparation marks account/credential/route preparation. Borrow -// preparation keeps its own existing budgets; none of this starts generation. +// BeginPelicanPreparation marks account and credential preparation without +// starting the generation budget. func BeginPelicanPreparation(ctx context.Context) { state := pelicanExecutionFromContext(ctx) if state == nil { @@ -371,7 +367,7 @@ func (s *pelicanExecutionState) finish() pelicanExecutionSnapshot { return snapshot } -func applyPelicanExecutionSnapshot(result *CodexGatewayBorrowTestResult, snapshot pelicanExecutionSnapshot) { +func applyPelicanExecutionSnapshot(result *PelicanTestResult, snapshot pelicanExecutionSnapshot) { if snapshot.invocation.Platform != "" { result.Platform = snapshot.invocation.Platform } @@ -382,7 +378,7 @@ func applyPelicanExecutionSnapshot(result *CodexGatewayBorrowTestResult, snapsho result.ActualEndpoint = snapshot.invocation.Endpoint result.ActualProtocol = snapshot.invocation.Protocol result.ActualTransport = snapshot.invocation.Transport - result.BorrowApplied = snapshot.invocation.BorrowApplied + result.BorrowApplied = false result.GenerationStartedAt = snapshot.generationStartedAt result.QueueDurationMS = snapshot.queueDurationMS result.PreparationDurationMS = snapshot.preparationDurationMS diff --git a/backend/internal/service/pelican_fixture_test.go b/backend/internal/service/pelican_fixture_test.go new file mode 100644 index 000000000000..d838e7941dbc --- /dev/null +++ b/backend/internal/service/pelican_fixture_test.go @@ -0,0 +1,8 @@ +//go:build unit + +package service + +func pelicanFixtureAccount(id int64) *Account { + return &Account{ID: id, Platform: PlatformOpenAI, Type: AccountTypeOAuth, Status: StatusActive, Concurrency: 4, + Credentials: map[string]any{"access_token": "synthetic-token", "chatgpt_account_id": "synthetic-chat-id"}, Extra: map[string]any{}} +} diff --git a/backend/internal/service/codex_gateway_borrow_preview.go b/backend/internal/service/pelican_preview.go similarity index 67% rename from backend/internal/service/codex_gateway_borrow_preview.go rename to backend/internal/service/pelican_preview.go index 1f36381024b9..fd15e1737236 100644 --- a/backend/internal/service/codex_gateway_borrow_preview.go +++ b/backend/internal/service/pelican_preview.go @@ -12,18 +12,18 @@ import ( ) const ( - CodexGatewayBorrowPreviewTTL = 30 * time.Minute - codexGatewayBorrowPreviewMaxCapabilities = 4096 + PelicanPreviewTTL = 30 * time.Minute + pelicanPreviewMaxCapabilities = 4096 // The HTTP header applies before any model-authored markup is parsed. The // iframe has the same sandbox without allow-same-origin, so its origin is opaque. - CodexGatewayBorrowPreviewCSP = "default-src 'none'; sandbox allow-scripts; script-src 'unsafe-inline' http: https: data: blob:; style-src 'unsafe-inline' http: https: data: blob:; img-src http: https: data: blob:; font-src http: https: data: blob:; media-src data: blob:; connect-src 'none'; frame-src 'none'; child-src 'none'; worker-src 'none'; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'self'; navigate-to 'none'" + PelicanPreviewCSP = "default-src 'none'; sandbox allow-scripts; script-src 'unsafe-inline' http: https: data: blob:; style-src 'unsafe-inline' http: https: data: blob:; img-src http: https: data: blob:; font-src http: https: data: blob:; media-src data: blob:; connect-src 'none'; frame-src 'none'; child-src 'none'; worker-src 'none'; object-src 'none'; base-uri 'none'; form-action 'none'; frame-ancestors 'self'; navigate-to 'none'" ) var ( - codexGatewayBorrowHTMLMarker = regexp.MustCompile(`(?i)<(?:!doctype\s+html|html|svg)[\s>]`) - codexGatewayBorrowHTMLTag = regexp.MustCompile(`(?i)]`) - codexGatewayBorrowSVGTag = regexp.MustCompile(`(?i)]`) - ErrCodexGatewayBorrowPreviewUnavailable = errors.New("preview capability unavailable or expired") + pelicanHTMLMarker = regexp.MustCompile(`(?i)<(?:!doctype\s+html|html|svg)[\s>]`) + pelicanHTMLTag = regexp.MustCompile(`(?i)]`) + pelicanSVGTag = regexp.MustCompile(`(?i)]`) + ErrPelicanPreviewUnavailable = errors.New("preview capability unavailable or expired") ) // Extraction follows ranxi2001/sub2api@5ca3cca's fenced/bare HTML/SVG @@ -32,7 +32,7 @@ var ( // deliberately not used: raw source and JavaScript remain intact, with the // boundary enforced by the response header and opaque sandbox instead. // rawHTML is the selected source before any SVG wrapper is added. -func ExtractCodexGatewayBorrowHTML(answer string) (rawHTML, html string) { +func ExtractPelicanHTML(answer string) (rawHTML, html string) { var selected string var outside strings.Builder for cursor := 0; cursor < len(answer); { @@ -49,8 +49,8 @@ func ExtractCodexGatewayBorrowHTML(answer string) (rawHTML, html string) { if closing >= 0 { bodyEnd = bodyStart + closing } - body, allowed := codexGatewayBorrowHTMLFenceContent(answer[bodyStart:bodyEnd]) - if allowed && codexGatewayBorrowHTMLMarker.MatchString(body) { + body, allowed := pelicanHTMLFenceContent(answer[bodyStart:bodyEnd]) + if allowed && pelicanHTMLMarker.MatchString(body) { selected = strings.TrimSpace(body) } if closing < 0 { @@ -61,7 +61,7 @@ func ExtractCodexGatewayBorrowHTML(answer string) (rawHTML, html string) { if selected == "" { selected = strings.TrimSpace(outside.String()) } - start := codexGatewayBorrowHTMLMarker.FindStringIndex(selected) + start := pelicanHTMLMarker.FindStringIndex(selected) if start == nil { return "", "" } @@ -71,13 +71,13 @@ func ExtractCodexGatewayBorrowHTML(answer string) (rawHTML, html string) { selected = selected[:end+len("")] } rawHTML = selected - if !codexGatewayBorrowHTMLTag.MatchString(selected) && codexGatewayBorrowSVGTag.MatchString(selected) { + if !pelicanHTMLTag.MatchString(selected) && pelicanSVGTag.MatchString(selected) { return rawHTML, `Pelican test` + selected + `` } return rawHTML, selected } -func codexGatewayBorrowHTMLFenceContent(body string) (string, bool) { +func pelicanHTMLFenceContent(body string) (string, bool) { if newline := strings.IndexByte(body, '\n'); newline >= 0 { language := strings.ToLower(strings.TrimSpace(body[:newline])) if language == "" || language == "html" || language == "htm" || language == "xml" || language == "svg" { @@ -94,15 +94,15 @@ func codexGatewayBorrowHTMLFenceContent(body string) (string, bool) { return trimmed, strings.HasPrefix(trimmed, "<") } -type codexGatewayBorrowPreviewCapability struct { +type pelicanPreviewCapability struct { ResultID string ExpiresAt time.Time } -type CodexGatewayBorrowPreviewService struct { - repo CodexGatewayBorrowTestRepository +type PelicanPreviewService struct { + repo PelicanTestRepository mu sync.Mutex - capabilities map[string]codexGatewayBorrowPreviewCapability + capabilities map[string]pelicanPreviewCapability stopCh chan struct{} doneCh chan struct{} stopOnce sync.Once @@ -110,14 +110,14 @@ type CodexGatewayBorrowPreviewService struct { randomToken func() (string, error) } -func NewCodexGatewayBorrowPreviewService(repo CodexGatewayBorrowTestRepository) *CodexGatewayBorrowPreviewService { - p := &CodexGatewayBorrowPreviewService{repo: repo, capabilities: make(map[string]codexGatewayBorrowPreviewCapability), - stopCh: make(chan struct{}), doneCh: make(chan struct{}), now: time.Now, randomToken: randomCodexGatewayBorrowPreviewToken} +func NewPelicanPreviewService(repo PelicanTestRepository) *PelicanPreviewService { + p := &PelicanPreviewService{repo: repo, capabilities: make(map[string]pelicanPreviewCapability), + stopCh: make(chan struct{}), doneCh: make(chan struct{}), now: time.Now, randomToken: randomPelicanPreviewToken} go p.cleanupLoop() return p } -func randomCodexGatewayBorrowPreviewToken() (string, error) { +func randomPelicanPreviewToken() (string, error) { var bytes [32]byte if _, err := rand.Read(bytes[:]); err != nil { return "", err @@ -125,11 +125,11 @@ func randomCodexGatewayBorrowPreviewToken() (string, error) { return base64.RawURLEncoding.EncodeToString(bytes[:]), nil } -func (p *CodexGatewayBorrowPreviewService) Stop() { +func (p *PelicanPreviewService) Stop() { p.stopOnce.Do(func() { close(p.stopCh); <-p.doneCh; p.mu.Lock(); clear(p.capabilities); p.mu.Unlock() }) } -func (p *CodexGatewayBorrowPreviewService) cleanupLoop() { +func (p *PelicanPreviewService) cleanupLoop() { defer close(p.doneCh) ticker := time.NewTicker(time.Minute) defer ticker.Stop() @@ -145,7 +145,7 @@ func (p *CodexGatewayBorrowPreviewService) cleanupLoop() { } } -func (p *CodexGatewayBorrowPreviewService) pruneLocked(now time.Time) { +func (p *PelicanPreviewService) pruneLocked(now time.Time) { for token, capability := range p.capabilities { if !capability.ExpiresAt.After(now) { delete(p.capabilities, token) @@ -155,7 +155,7 @@ func (p *CodexGatewayBorrowPreviewService) pruneLocked(now time.Time) { // DecorateResult only adds an independently minted read capability. It makes no // upstream request and never turns incomplete HTML into a successful test. -func (p *CodexGatewayBorrowPreviewService) DecorateResult(result *CodexGatewayBorrowTestResult) *CodexGatewayBorrowTestResult { +func (p *PelicanPreviewService) DecorateResult(result *PelicanTestResult) *PelicanTestResult { decorated := *result decorated.PreviewURL, decorated.PreviewExpiresAt = "", nil if decorated.Status != "complete" { @@ -171,7 +171,7 @@ func (p *CodexGatewayBorrowPreviewService) DecorateResult(result *CodexGatewayBo decorated.PreviewUnavailable = "test expired" return &decorated } - expires := now.Add(CodexGatewayBorrowPreviewTTL) + expires := now.Add(PelicanPreviewTTL) if decorated.ExpiresAt.Before(expires) { expires = decorated.ExpiresAt } @@ -184,7 +184,7 @@ func (p *CodexGatewayBorrowPreviewService) DecorateResult(result *CodexGatewayBo p.pruneLocked(now) // Bound the in-memory capability store even if many administrators keep // reloading results. The oldest capability expires first and is evicted. - if len(p.capabilities) >= codexGatewayBorrowPreviewMaxCapabilities { + if len(p.capabilities) >= pelicanPreviewMaxCapabilities { var oldestToken string var oldest time.Time for candidate, capability := range p.capabilities { @@ -194,17 +194,17 @@ func (p *CodexGatewayBorrowPreviewService) DecorateResult(result *CodexGatewayBo } delete(p.capabilities, oldestToken) } - p.capabilities[token] = codexGatewayBorrowPreviewCapability{ResultID: decorated.ID, ExpiresAt: expires} + p.capabilities[token] = pelicanPreviewCapability{ResultID: decorated.ID, ExpiresAt: expires} p.mu.Unlock() decorated.PreviewURL = "/api/v1/pelican-tests/preview/" + token + "/index.html" decorated.PreviewExpiresAt, decorated.PreviewUnavailable = &expires, "" return &decorated } -func (p *CodexGatewayBorrowPreviewService) DecorateTask(task *CodexGatewayBorrowTestTask) *CodexGatewayBorrowTestTask { +func (p *PelicanPreviewService) DecorateTask(task *PelicanTestTask) *PelicanTestTask { decorated := *task if task.Results != nil { - decorated.Results = make([]*CodexGatewayBorrowTestResult, 0, len(task.Results)) + decorated.Results = make([]*PelicanTestResult, 0, len(task.Results)) for _, result := range task.Results { decorated.Results = append(decorated.Results, p.DecorateResult(result)) } @@ -214,9 +214,9 @@ func (p *CodexGatewayBorrowPreviewService) DecorateTask(task *CodexGatewayBorrow // Resolve checks the capability and re-reads exactly its bound result. Expired // task data is inaccessible even when the periodic SQL cleanup has not run. -func (p *CodexGatewayBorrowPreviewService) Resolve(ctx context.Context, token string) (string, error) { +func (p *PelicanPreviewService) Resolve(ctx context.Context, token string) (string, error) { if len(token) != 43 { - return "", ErrCodexGatewayBorrowPreviewUnavailable + return "", ErrPelicanPreviewUnavailable } p.mu.Lock() now := p.now().UTC() @@ -224,7 +224,7 @@ func (p *CodexGatewayBorrowPreviewService) Resolve(ctx context.Context, token st if !found || !capability.ExpiresAt.After(now) { delete(p.capabilities, token) p.mu.Unlock() - return "", ErrCodexGatewayBorrowPreviewUnavailable + return "", ErrPelicanPreviewUnavailable } p.mu.Unlock() result, err := p.repo.GetResult(ctx, capability.ResultID) @@ -233,7 +233,7 @@ func (p *CodexGatewayBorrowPreviewService) Resolve(ctx context.Context, token st p.mu.Lock() delete(p.capabilities, token) p.mu.Unlock() - return "", ErrCodexGatewayBorrowPreviewUnavailable + return "", ErrPelicanPreviewUnavailable } return result.HTML, nil } diff --git a/backend/internal/service/codex_gateway_borrow_preview_test.go b/backend/internal/service/pelican_preview_test.go similarity index 57% rename from backend/internal/service/codex_gateway_borrow_preview_test.go rename to backend/internal/service/pelican_preview_test.go index 1f2eed0f0980..9a2a407108d3 100644 --- a/backend/internal/service/codex_gateway_borrow_preview_test.go +++ b/backend/internal/service/pelican_preview_test.go @@ -13,42 +13,42 @@ import ( "github.com/stretchr/testify/require" ) -func TestCodexGatewayBorrowHTMLExtractionPreservesSelectedSourceAndScripts(t *testing.T) { +func TestPelicanHTMLExtractionPreservesSelectedSourceAndScripts(t *testing.T) { const final = `` - raw, html := ExtractCodexGatewayBorrowHTML("reasoning\n```json\n{}\n```\n```html\nold\n```\n```html\n" + final + "\n```\nfinished") + raw, html := ExtractPelicanHTML("reasoning\n```json\n{}\n```\n```html\nold\n```\n```html\n" + final + "\n```\nfinished") require.Equal(t, final, raw) require.Equal(t, final, html) require.NotContains(t, html, "Content-Security-Policy") const svg = `` - raw, html = ExtractCodexGatewayBorrowHTML("```svg\n" + svg + "\n```") + raw, html = ExtractPelicanHTML("```svg\n" + svg + "\n```") require.Equal(t, svg, raw) require.Contains(t, html, svg) require.Contains(t, html, ``) require.NotContains(t, html, "DOMPurify") - raw, html = ExtractCodexGatewayBorrowHTML("Explanation\npelican\nmore prose") + raw, html = ExtractPelicanHTML("Explanation\npelican\nmore prose") require.Equal(t, `pelican`, raw) require.Equal(t, raw, html) - raw, html = ExtractCodexGatewayBorrowHTML("There is a pelican riding a bicycle.") + raw, html = ExtractPelicanHTML("There is a pelican riding a bicycle.") require.Empty(t, raw) require.Empty(t, html) - raw, html = ExtractCodexGatewayBorrowHTML("```json\n{\"example\":\"quoted markup\"}\n```") + raw, html = ExtractPelicanHTML("```json\n{\"example\":\"quoted markup\"}\n```") require.Empty(t, raw, "markup quoted in a non-HTML code fence is not a preview") require.Empty(t, html) } -func TestCodexGatewayBorrowHTMLExtractionAcceptsUnclosedValidFenceWithoutRepair(t *testing.T) { +func TestPelicanHTMLExtractionAcceptsUnclosedValidFenceWithoutRepair(t *testing.T) { const partial = `
    -
  1. - - - {{ t(item.label) }} - -
    2. -
- - - - diff --git a/frontend/src/components/admin/__tests__/CodexBorrowNav.spec.ts b/frontend/src/components/admin/__tests__/CodexBorrowNav.spec.ts deleted file mode 100644 index c3976e362951..000000000000 --- a/frontend/src/components/admin/__tests__/CodexBorrowNav.spec.ts +++ /dev/null @@ -1,47 +0,0 @@ -import { flushPromises, mount } from '@vue/test-utils' -import { createMemoryHistory, createRouter } from 'vue-router' -import { describe, expect, it, vi } from 'vitest' -import CodexBorrowNav from '../CodexBorrowNav.vue' - -vi.mock('vue-i18n', () => ({ - useI18n: () => ({ t: (key: string) => key }), -})) - -const paths = [ - '/admin/codex-gateway-borrow', - '/admin/codex-gateway-borrow/status', -] - -async function renderNav(path: string) { - const router = createRouter({ - history: createMemoryHistory(), - routes: paths.map(routePath => ({ path: routePath, component: { template: '
' } })), - }) - await router.push(path) - await router.isReady() - const wrapper = mount(CodexBorrowNav, { global: { plugins: [router] } }) - return { router, wrapper } -} - -describe('Codex borrowing page navigation', () => { - it.each(paths)('renders real links and marks only the current page on %s', async path => { - const { wrapper } = await renderNav(path) - - expect(wrapper.findAll('a').map(link => link.attributes('href'))).toEqual(paths) - expect(wrapper.findAll('a[aria-current="page"]')).toHaveLength(1) - expect(wrapper.get('a[aria-current="page"]').attributes('href')).toBe(path) - wrapper.unmount() - }) - - it('navigates between pages and supports browser Back', async () => { - const { router, wrapper } = await renderNav(paths[0]) - - await wrapper.get(`a[href="${paths[1]}"]`).trigger('click') - await flushPromises() - expect(router.currentRoute.value.path).toBe(paths[1]) - router.back() - await vi.waitFor(() => expect(router.currentRoute.value.path).toBe(paths[0])) - expect(wrapper.get('a[aria-current="page"]').attributes('href')).toBe(paths[0]) - wrapper.unmount() - }) -}) diff --git a/frontend/src/components/admin/codex/CodexBorrowActivity.vue b/frontend/src/components/admin/codex/CodexBorrowActivity.vue deleted file mode 100644 index 82741fe173e1..000000000000 --- a/frontend/src/components/admin/codex/CodexBorrowActivity.vue +++ /dev/null @@ -1,102 +0,0 @@ - - - diff --git a/frontend/src/components/admin/codex/CodexLayout.vue b/frontend/src/components/admin/codex/CodexLayout.vue index 50a57b4118fd..fc7bdffa8c31 100644 --- a/frontend/src/components/admin/codex/CodexLayout.vue +++ b/frontend/src/components/admin/codex/CodexLayout.vue @@ -3,7 +3,7 @@

Codex

-

{{ text('配置线路借用、管理客户端身份，并核实请求实际使用了什么。', 'Configure route borrowing, manage client identity, and inspect what requests actually used.') }}

+

{{ text('管理客户端身份与 Codex 请求设置。', 'Manage client identity and Codex request settings.') }}