diff --git a/.downstream/codex-quality-diagnostics.md b/.downstream/codex-quality-diagnostics.md index 8869339a441c..fe3969780886 100644 --- a/.downstream/codex-quality-diagnostics.md +++ b/.downstream/codex-quality-diagnostics.md @@ -44,7 +44,7 @@ grant 与 trial header 在 handler 开始时移入私有执行上下文，不转 账本存于 `sub2api_plugin_state`：plugin key `codexrip.codex-runtime`，namespace `codex-routing-private`，key `quality-run.`。新记录显式保存 `model`、`reasoning_effort` 和凭据主体 `owner_identity`；账本不保存 grant、Key、题面、Cookie 或 STATE 值。 -路由资格诊断时期写入的旧记录（无 `model`）按 `gpt-6-astra`/`high` 显示且只读：GET 和 close 仍可用，同一 `run_id` 不能再创建、恢复或发送（409，需换用新 run id）。旧记录的 `scope`、`qualification`、`route_generation`、`route_runtime_generation` 及旧 attempt 字段在重写（如 close）时原样保留；视图只对旧记录只读显示 `route_generation`、`route_expires_at` 和 `connection_fingerprint`，旧 attempt 可能带 `acquire`/`verify` 阶段。关闭旧记录不再清理其遗留的私有 Cookie 状态。 +路由资格诊断时期写入的旧记录（无 `model`）按 `gpt-6-astra`/`high` 显示且只读：GET 和 close 仍可用，同一 `run_id` 不能再创建、恢复或发送（409，需换用新 run id）。旧记录的 `scope`、`qualification`、`route_generation`、`route_runtime_generation` 及旧 attempt 字段在重写（如 close）时原样保留；视图只对旧记录只读显示 `route_generation`、`route_expires_at` 和 `connection_fingerprint`，旧 attempt 可能带 `acquire`/`verify` 阶段。旧记录引用的路由 bundle 与 Cookie 状态（`bundle.*`、`clock.*`、`seen.*`）已由迁移 263 删除，账本本身原样保留。 ## 运维执行器 diff --git a/.downstream/native-domains.md b/.downstream/native-domains.md index 260323485567..5435074a47ed 100644 --- a/.downstream/native-domains.md +++ b/.downstream/native-domains.md @@ -24,11 +24,12 @@ and not stored again; any other key, or a `request_zstd` that is not a boolean The route acquisition endpoints (`/admin/accounts/codex-tickets/*`, `/admin/accounts/:id/codex-tickets/*`, `/admin/accounts/:id/codex-routing/validate` -and `/admin/settings/openai-codex-ticket/*`) are removed. Jobs of the retired -kinds `codex_ticket_harvest`, `codex_ticket_stop` and `extension_operation` -still list and read with `retry_eligible=false` and -`retry_unavailable_reason=kind_unsupported`; a retry returns HTTP 409, and their -items still pending fail with `kind_unsupported`. Ordinary account bulk editing +and `/admin/settings/openai-codex-ticket/*`) are removed. Migration 263 deletes +the job rows of the retired kinds `codex_ticket_harvest`, `codex_ticket_stop` +and `extension_operation`; their display names are gone. A job whose kind this +version has no executor for still lists and reads with `retry_eligible=false` +and `retry_unavailable_reason=kind_unsupported`; a retry returns HTTP 409, and +its items still pending fail with `kind_unsupported`. Ordinary account bulk editing remains HTTP 202; this change does not replace background tasks with synchronous editing. Existing API-key reveal and account-field protections remain. @@ -94,15 +95,23 @@ the first start advances `runtime_generation` once and records the new configuration hash and version; the encrypted source keeps its old bytes until the next save. -Data written by the retired route acquisition stays in place and is no longer -read: the `openai_codex_ticket_*` settings, the `tickets`, `routing-demand` and -`proxy-trust` state namespaces, the route keys in `codex-routing-private` -(`bundle.*`, `clock.*`, `seen.*`, `spent.*`, `validation.*`; `quality-run.*` and -`wire.*` stay in use), `sub2api_plugin_leases` rows, the account `extra` keys -`codex_turn_ticket:*`, `codex_ticket_runtime:*`, `codex_harvest_proxy_url` and -`plugin_account_projections`, and job rows of the retired kinds. Migrations 244 -and 245 and their triggers are unchanged. A later release removes this data with -a forward migration. +Migration `263_purge_retired_codex_ticket_data.sql` deletes the data written by +the retired route acquisition: the triggers, trigger functions and tables of +migrations 244 and 245; the settings `openai_codex_ticket_enabled` and +`openai_codex_ticket_harvest_proxy_url`; the `codexrip.codex-runtime` state +namespaces `tickets`, `routing-demand` and `proxy-trust` and its +`routing-account` and `tickets` leases; the route and Cookie material in +`codex-routing-private` (`bundle.*`, `clock.*`, `seen.*`, `validation-result.*`); +the account `extra` keys `codex_turn_ticket:*`, `codex_ticket_runtime:*` and +`codex_harvest_proxy_url` and the `codexrip.codex-runtime` entry of +`plugin_account_projections` (the map goes once it is empty); and job rows of +the retired kinds. `codex-routing-private` keeps the ledgers `quality-run.*`, +`validation.*` and `spent.*`, and the `wire.*` observations without the raw +Cookie values (`cookies`) they stored. Accounts without these keys are not +rewritten. Account create, update and bulk edit drop these `extra` keys and that +projection entry, so importing an account export taken before this release does +not bring them back. The encrypted Codex runtime source keeps its bytes until +the next save, as described above. Migration `254_plugin_job_action_digest.sql` preserves full 64-character historical task action digests. History and retries continue to use saved targets; unsupported @@ -125,6 +134,12 @@ runtime generation or delete its native anchor**. A plain image change does not perform this reverse conversion. Such a legacy-host rollback has not been run as part of the native migration and is not an automatic failure path. +Migration 263 has no down path; the data it deletes (listed above) comes back +only from a database dump taken before it. An image that still has the route +acquisition starts without its tickets and routes. Images from +`v0.2.8-codexrip.5` (source `9f7b08dfc`) on never read the 244/245 tables; older +images still query them, so returning to one of those needs that dump. + ## Verification Normal required PR checks validate the native domain packages, account scope, diff --git a/DOWNSTREAM.md b/DOWNSTREAM.md index cf35f8909886..9ba1233cfb36 100644 --- a/DOWNSTREAM.md +++ b/DOWNSTREAM.md @@ -32,11 +32,12 @@ request route gates, and their admin endpoints, settings and job kinds. OpenAI OAuth accounts are scheduled without route qualification and send through their ordinary proxy and transport. An OAuth Codex `/responses` reply over HTTP that declares a different model than the request is still rejected. The Codex -runtime setting keeps only request compression. Old jobs of the retired kinds -stay readable but cannot be retried; data the feature wrote stays in place, -unread, until a later release purges it. Quality diagnostics bind an explicit -model and reasoning effort and send through the account's ordinary path; see -[quality diagnostics](.downstream/codex-quality-diagnostics.md). +runtime setting keeps only request compression. Migration 263 deletes the data +the feature wrote, old jobs of the retired kinds included; quality ledgers and +the observed outbound requests stay, without stored Cookie values (see +[native domains](.downstream/native-domains.md)). Quality diagnostics bind an +explicit model and reasoning effort and send through the account's ordinary +path; see [quality diagnostics](.downstream/codex-quality-diagnostics.md). ## Official behavior and downstream contracts diff --git a/backend/internal/repository/account_plugin_projection.go b/backend/internal/repository/account_plugin_projection.go deleted file mode 100644 index e1f73f14aa8f..000000000000 --- a/backend/internal/repository/account_plugin_projection.go +++ /dev/null @@ -1,27 +0,0 @@ -package repository - -import ( - "github.com/Wei-Shaw/sub2api/internal/service" - "maps" -) - -func withoutPluginAccountProjection(extra map[string]any) map[string]any { - if _, exists := extra[service.NativeCodexAccountProjectionKey]; !exists { - return extra - } - copy := maps.Clone(extra) - delete(copy, service.NativeCodexAccountProjectionKey) - return copy -} - -func preservePluginAccountProjection(incoming, current map[string]any) map[string]any { - extra := withoutPluginAccountProjection(incoming) - if value, exists := current[service.NativeCodexAccountProjectionKey]; exists { - extra = maps.Clone(extra) - if extra == nil { - extra = make(map[string]any) - } - extra[service.NativeCodexAccountProjectionKey] = value - } - return extra -} diff --git a/backend/internal/repository/account_repo.go b/backend/internal/repository/account_repo.go index 632f392e9cc5..a89f678304b9 100644 --- a/backend/internal/repository/account_repo.go +++ b/backend/internal/repository/account_repo.go @@ -108,6 +108,48 @@ func stripCodexFingerprintSeedFromExtraUpdate(extra map[string]any) map[string]a return stripped } +// isRetiredCodexTicketExtraKey matches the account extra keys of the retired +// Codex route acquisition that migration 263 purged. +func isRetiredCodexTicketExtraKey(key string) bool { + return strings.HasPrefix(key, "codex_turn_ticket:") || + strings.HasPrefix(key, "codex_ticket_runtime:") || + key == "codex_harvest_proxy_url" +} + +// withoutRetiredCodexTicketExtra keeps account writes from storing the data +// migration 263 purged, so importing an account export taken before it cannot +// bring that data back. Like 263 it also drops the codexrip.codex-runtime entry +// of plugin_account_projections, and the map only once no other entry is left. +// extra is returned as is when it holds none of this. +func withoutRetiredCodexTicketExtra(extra map[string]any) map[string]any { + projections, _ := extra["plugin_account_projections"].(map[string]any) + _, projected := projections["codexrip.codex-runtime"] + stripped := make(map[string]any, len(extra)) + for key, value := range extra { + if !isRetiredCodexTicketExtraKey(key) { + stripped[key] = value + } + } + if !projected { + if len(stripped) == len(extra) { + return extra + } + return stripped + } + rest := make(map[string]any, len(projections)) + for key, value := range projections { + if key != "codexrip.codex-runtime" { + rest[key] = value + } + } + if len(rest) == 0 { + delete(stripped, "plugin_account_projections") + } else { + stripped["plugin_account_projections"] = rest + } + return stripped +} + // NewAccountRepository 创建账户仓储实例。 // 这是对外暴露的构造函数，返回接口类型以便于依赖注入。 func NewAccountRepository(client *dbent.Client, sqlDB *sql.DB, schedulerCache service.SchedulerCache) service.AccountRepository { @@ -156,7 +198,7 @@ func createAccountRecord(ctx context.Context, client *dbent.Client, account *ser SetPlatform(account.Platform). SetType(account.Type). SetCredentials(normalizeJSONMap(account.Credentials)). - SetExtra(normalizeJSONMap(withoutPluginAccountProjection(account.Extra))). + SetExtra(normalizeJSONMap(withoutRetiredCodexTicketExtra(account.Extra))). SetConcurrency(account.Concurrency). SetPriority(account.Priority). SetStatus(account.Status). @@ -740,7 +782,7 @@ func lockAndMergeAccountProbeExtra( currentExtra = nil } } - extra = preservePluginAccountProjection(extra, currentExtra) + extra = withoutRetiredCodexTicketExtra(extra) // Preserve the system prompt binding read under the row lock. A stale // general account editor cannot overwrite the dedicated binding endpoint. delete(extra, service.AccountExtraSystemPromptKey) @@ -2878,7 +2920,7 @@ func (r *accountRepository) AutoPauseExpiredAccounts(ctx context.Context, now ti } func (r *accountRepository) UpdateExtra(ctx context.Context, id int64, updates map[string]any) error { - updates = withoutPluginAccountProjection(updates) + updates = withoutRetiredCodexTicketExtra(updates) updates = stripCodexFingerprintSeedFromExtraUpdate(updates) if len(updates) == 0 { return nil @@ -3234,7 +3276,7 @@ func (r *accountRepository) BulkUpdate(ctx context.Context, ids []int64, updates return 0, nil } updates.Extra = stripCodexFingerprintSeedFromExtraUpdate(updates.Extra) - updates.Extra = withoutPluginAccountProjection(updates.Extra) + updates.Extra = withoutRetiredCodexTicketExtra(updates.Extra) setClauses := make([]string, 0, 8) args := make([]any, 0, 8) diff --git a/backend/internal/repository/account_repo_model_context_test.go b/backend/internal/repository/account_repo_model_context_test.go index c11a517c9a45..8087940bc94f 100644 --- a/backend/internal/repository/account_repo_model_context_test.go +++ b/backend/internal/repository/account_repo_model_context_test.go @@ -183,8 +183,8 @@ func TestLockAndMergeAccountProbeExtraUsesLockedModelContextValues(t *testing.T) } // This path serves every account update: a non-object stored extra must not -// fail the edit, only skip carrying locked values over. A dormant scheduling -// projection supplied by the editor is still never written. +// fail the edit, only skip carrying locked values over. Data of the retired +// Codex route acquisition supplied by the editor or an import is never written. func TestLockAndMergeAccountExtraDegradesOnUnparsableExtra(t *testing.T) { db, mock, err := sqlmock.New() require.NoError(t, err) @@ -195,14 +195,28 @@ func TestLockAndMergeAccountExtraDegradesOnUnparsableExtra(t *testing.T) { WithArgs(int64(41), service.PlatformOpenAI, service.AccountTypeOAuth, `{"access_token":"test"}`, nil). WillReturnRows(sqlmock.NewRows([]string{"identity_unchanged", "ollama_group_unchanged", "ollama_proxy_unchanged", "enabled", "rate_sync_enabled", "snapshot", "ollama_session", "ollama_auto", "ollama_snapshot", "model_context_overrides", "upstream_model_metadata", "current_extra", "opencode_group_unchanged", "opencode_auto", "opencode_snapshot"}). AddRow(true, false, true, nil, nil, nil, nil, nil, nil, nil, nil, []byte(`[1,2,3]`), false, nil, nil)) - account := &service.Account{ID: 41, Platform: service.PlatformOpenAI, Type: service.AccountTypeOAuth, Credentials: map[string]any{"access_token": "test"}, Extra: map[string]any{service.NativeCodexAccountProjectionKey: map[string]any{"forged": true}, "new_admin_setting": true}} + account := &service.Account{ID: 41, Platform: service.PlatformOpenAI, Type: service.AccountTypeOAuth, Credentials: map[string]any{"access_token": "test"}, Extra: map[string]any{"plugin_account_projections": map[string]any{"codexrip.codex-runtime": map[string]any{"forged": true}}, "codex_turn_ticket:gpt-6-astra": map[string]any{"state": "forged"}, "codex_harvest_proxy_url": "http://proxy.example.test:8080", "new_admin_setting": true}} extra, err := lockAndMergeAccountProbeExtra(context.Background(), client, account, nil, nil) require.NoError(t, err, "unparsable extra must not fail the account update") require.Equal(t, true, extra["new_admin_setting"]) - require.NotContains(t, extra, service.NativeCodexAccountProjectionKey) + for _, key := range []string{"plugin_account_projections", "codex_turn_ticket:gpt-6-astra", "codex_harvest_proxy_url"} { + require.NotContains(t, extra, key) + } require.NoError(t, mock.ExpectationsWereMet()) } +// Like migration 263, only the codexrip.codex-runtime projection goes; another +// plugin's entry and a key that merely shares a name stay. +func TestWithoutRetiredCodexTicketExtraKeepsOtherEntries(t *testing.T) { + kept := map[string]any{"codex_turn_ticket_summary": "not the retired prefix", "plugin_account_projections": map[string]any{"acme.other-plugin": true}} + require.Equal(t, kept, withoutRetiredCodexTicketExtra(kept)) + require.Equal(t, kept, withoutRetiredCodexTicketExtra(map[string]any{ + "codex_turn_ticket_summary": "not the retired prefix", + "codex_ticket_runtime:gpt-6-astra": map[string]any{"phase": "ready"}, + "plugin_account_projections": map[string]any{"codexrip.codex-runtime": map[string]any{"identity": "owner"}, "acme.other-plugin": true}, + })) +} + func TestShouldEnqueueSchedulerOutboxForExtraUpdatesModelContextKeys(t *testing.T) { for _, key := range []string{ service.ModelContextOverridesExtraKey, diff --git a/backend/internal/repository/codex_ticket_purge_migration_integration_test.go b/backend/internal/repository/codex_ticket_purge_migration_integration_test.go new file mode 100644 index 000000000000..5add297bd3e2 --- /dev/null +++ b/backend/internal/repository/codex_ticket_purge_migration_integration_test.go @@ -0,0 +1,246 @@ +//go:build integration + +package repository + +import ( + "context" + "database/sql" + "strings" + "testing" + + dbmigrations "github.com/Wei-Shaw/sub2api/migrations" + "github.com/stretchr/testify/require" +) + +const codexTicketPurgeMigration = "263_purge_retired_codex_ticket_data.sql" + +// TestMain applied every migration, 263 included, to an empty database. The +// test restores the state the previous release left behind (the objects of +// 244/245 and representative dormant rows next to rows that must survive), +// applies 263, checks what was purged and what stayed, and applies it again. +func TestMigration263PurgesRetiredCodexTicketData(t *testing.T) { + tx := testTx(t) + ctx := context.Background() + exec := func(query string, args ...any) { + t.Helper() + _, err := tx.ExecContext(ctx, query, args...) + require.NoError(t, err) + } + text := func(query string, args ...any) string { + t.Helper() + var value string + require.NoError(t, tx.QueryRowContext(ctx, query, args...).Scan(&value)) + return value + } + count := func(query string, args ...any) int { + t.Helper() + var value int + require.NoError(t, tx.QueryRowContext(ctx, query, args...).Scan(&value)) + return value + } + migration := func(name string) string { + t.Helper() + content, err := dbmigrations.FS.ReadFile(name) + require.NoError(t, err) + return string(content) + } + + requireCodexTicketObjects(t, ctx, tx, false) + exec(migration("244_codex_ticket_lifecycle.sql")) + exec(migration("245_codex_ticket_account_status_independent.sql")) + requireCodexTicketObjects(t, ctx, tx, true) + + var userID int64 + require.NoError(t, tx.QueryRowContext(ctx, ` +INSERT INTO users (email, password_hash, role, status, balance, concurrency) +VALUES ('migration-263@example.com', 'hash', 'admin', 'active', 0, 1) +RETURNING id`).Scan(&userID)) + insertAccount := func(name, platform, extra string) int64 { + t.Helper() + var id int64 + require.NoError(t, tx.QueryRowContext(ctx, ` +INSERT INTO accounts (name, platform, type, extra) VALUES ($1, $2, 'oauth', $3::jsonb) RETURNING id`, + name, platform, extra).Scan(&id)) + return id + } + ticketID := insertAccount("migration-263-ticket", "openai", `{ + "openai_long_context_billing_enabled": true, + "codex_fingerprint_seed": "11111111-1111-4111-8111-111111111111", + "codex_turn_ticket:gpt-6-astra": {"state": "gAAAAA-dormant-ticket", "length": 292}, + "codex_ticket_runtime:gpt-6-astra": {"phase": "ready"}, + "codex_harvest_proxy_url": "http://user:fixture-proxy-secret@proxy.example.test:8080", + "codex_turn_ticket_summary": "kept: not the retired prefix", + "plugin_account_projections": {"codexrip.codex-runtime": {"identity": "owner", "scheduling": {"gpt-6-astra": {"effect": "deny"}}}} + }`) + sharedID := insertAccount("migration-263-shared-projection", "openai", `{ + "openai_long_context_billing_enabled": false, + "plugin_account_projections": {"codexrip.codex-runtime": {"identity": "owner"}, "acme.other-plugin": {"identity": "kept"}} + }`) + cleanID := insertAccount("migration-263-clean", "openai", `{ + "openai_long_context_billing_enabled": true, + "codex_fingerprint_seed": "22222222-2222-4222-8222-222222222222", + "plugin_account_projections": {"acme.other-plugin": {"identity": "kept"}} + }`) + arrayID := insertAccount("migration-263-array-extra", "anthropic", `[1, 2, 3]`) + accountIDs := []int64{ticketID, sharedID, cleanID, arrayID} + + exec(`INSERT INTO settings (key, value) VALUES + ('openai_codex_ticket_enabled', 'true'), + ('openai_codex_ticket_harvest_proxy_url', 'http://user:fixture-proxy-secret@proxy.example.test:8080'), + ('codex_native_runtime_source', 'encrypted-source-kept'), + ('codex_native_runtime_config', '{"version":1,"config_version":3,"config_sha256":"kept"}'), + ('openai_codex_client_version', '0.156.0') + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value`) + exec(`INSERT INTO openai_codex_ticket_runtime (account_id, model, identity, phase) VALUES ($1, 'gpt-6-astra', 'owner', 'ready')`, ticketID) + exec(`INSERT INTO openai_codex_ticket_proxy_trust (proxy_key, certificates) VALUES ('migration-263', '["fixture-public-certificate"]')`) + + const codex, private = "codexrip.codex-runtime", "codex-routing-private" + insertState := func(plugin, namespace, key, value string) { + t.Helper() + exec(`INSERT INTO sub2api_plugin_state (plugin_key, namespace, state_key, value) VALUES ($1, $2, $3, $4::jsonb)`, + plugin, namespace, key, value) + } + purgedState := [][2]string{ + {"tickets", "m263.ticket"}, + {"routing-demand", "m263.ticket"}, + {"proxy-trust", "m263.proxy"}, + {private, "bundle.m263.live"}, + {private, "bundle.quality.m263.candidate"}, + {private, "clock.m263"}, + {private, "clock.quality.m263"}, + {private, "seen.m263"}, + {private, "validation-result.m263"}, + } + for _, row := range purgedState { + insertState(codex, row[0], row[1], `{"cookies":[{"name":"__cflb","value":"fixture-cookie-value"}]}`) + } + keptState := map[string]string{ + "quality-run.11111111-1111-4111-8111-111111111111": `{"run_id":"11111111-1111-4111-8111-111111111111","qualification":{"bundle":{"key":"bundle.quality.m263.live"}}}`, + "validation.22222222-2222-4222-8222-222222222222": `{"account_id":41,"used":2,"stages":{"gpt-6-astra:http:acquire":true}}`, + "spent.m263": `{"spent":true}`, + "wire.m263-observed": `{"user_agent":"codex_cli_rs/0.156.0","state_present":false}`, + "other.m263-unclassified": `{"kept":true}`, + } + for key, value := range keptState { + insertState(codex, private, key, value) + } + insertState(codex, private, "wire.m263-cookies", `{"user_agent":"codex_cli_rs/0.156.0","state":"opaque-state","cookie_versions":{"__cflb":"0123456789abcdef"},"cookies":[{"name":"__cflb","value":"fixture-cookie-value","routing":true}]}`) + insertState("acme.other-plugin", "tickets", "m263.ticket", `{"kept":true}`) + insertState("acme.other-plugin", private, "bundle.m263.live", `{"kept":true}`) + + insertLease := func(plugin, namespace, key string) { + t.Helper() + exec(`INSERT INTO sub2api_plugin_leases (plugin_key, namespace, lease_key, owner, expires_at) VALUES ($1, $2, $3, 'owner', NOW())`, + plugin, namespace, key) + } + insertLease(codex, "routing-account", "m263") + insertLease(codex, "tickets", "m263.ticket") + insertLease("acme.other-plugin", "routing-account", "m263") + + insertJob := func(kind string) int64 { + t.Helper() + var id int64 + require.NoError(t, tx.QueryRowContext(ctx, ` +INSERT INTO admin_account_jobs (created_by, kind, idempotency_key, request_hash, status, target_count, processed_count, failed_count) +VALUES ($1, $2, $3, $4, 'failed', 1, 1, 1) RETURNING id`, + userID, kind, "migration-263-"+kind, strings.Repeat("a", 64)).Scan(&id)) + exec(`INSERT INTO admin_account_job_items (job_id, ordinal, target_account_id, status, metadata) VALUES ($1, 1, $2, 'failed', '{"model_id":"gpt-6-astra"}')`, + id, ticketID) + return id + } + retiredJobs := []int64{insertJob("codex_ticket_harvest"), insertJob("codex_ticket_stop"), insertJob("extension_operation")} + keptJob := insertJob("account_bulk_update") + + ctid := func(table, column string, key any) string { + t.Helper() + return text(`SELECT ctid::text FROM `+table+` WHERE `+column+` = $1`, key) + } + stateCTID := func(key string) string { + t.Helper() + return text(`SELECT ctid::text || '/' || revision FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = $3`, codex, private, key) + } + untouchedAccounts := map[int64]string{cleanID: ctid("accounts", "id", cleanID), arrayID: ctid("accounts", "id", arrayID)} + untouchedState := map[string]string{} + for key := range keptState { + untouchedState[key] = stateCTID(key) + } + untouchedSettings := map[string]string{} + for _, key := range []string{"codex_native_runtime_source", "codex_native_runtime_config", "openai_codex_client_version"} { + untouchedSettings[key] = ctid("settings", "key", key) + } + outboxBefore := count(`SELECT count(*) FROM scheduler_outbox WHERE account_id = ANY($1)`, accountIDs) + wireRevision := count(`SELECT revision FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = 'wire.m263-cookies'`, codex, private) + + exec(migration(codexTicketPurgeMigration)) + + requireCodexTicketObjects(t, ctx, tx, false) + require.Zero(t, count(`SELECT count(*) FROM settings WHERE key IN ('openai_codex_ticket_enabled', 'openai_codex_ticket_harvest_proxy_url')`)) + require.Equal(t, "encrypted-source-kept", text(`SELECT value FROM settings WHERE key = 'codex_native_runtime_source'`)) + for key, before := range untouchedSettings { + require.Equal(t, before, ctid("settings", "key", key), key) + } + + require.JSONEq(t, `{ + "openai_long_context_billing_enabled": true, + "codex_fingerprint_seed": "11111111-1111-4111-8111-111111111111", + "codex_turn_ticket_summary": "kept: not the retired prefix" + }`, text(`SELECT extra::text FROM accounts WHERE id = $1`, ticketID)) + require.JSONEq(t, `{ + "openai_long_context_billing_enabled": false, + "plugin_account_projections": {"acme.other-plugin": {"identity": "kept"}} + }`, text(`SELECT extra::text FROM accounts WHERE id = $1`, sharedID)) + for id, before := range untouchedAccounts { + require.Equal(t, before, ctid("accounts", "id", id), "account %d holds no retired key and is not rewritten", id) + } + require.Equal(t, outboxBefore, count(`SELECT count(*) FROM scheduler_outbox WHERE account_id = ANY($1)`, accountIDs)) + + for _, row := range purgedState { + require.Zero(t, count(`SELECT count(*) FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = $3`, codex, row[0], row[1]), row) + } + require.Zero(t, count(`SELECT count(*) FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace IN ('tickets', 'routing-demand', 'proxy-trust')`, codex)) + for key, before := range untouchedState { + require.Equal(t, before, stateCTID(key), key) + } + require.JSONEq(t, `{"user_agent":"codex_cli_rs/0.156.0","state":"opaque-state","cookie_versions":{"__cflb":"0123456789abcdef"}}`, + text(`SELECT value::text FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = 'wire.m263-cookies'`, codex, private)) + require.Equal(t, wireRevision+1, count(`SELECT revision FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = 'wire.m263-cookies'`, codex, private)) + require.Equal(t, 2, count(`SELECT count(*) FROM sub2api_plugin_state WHERE plugin_key = 'acme.other-plugin' AND state_key IN ('m263.ticket', 'bundle.m263.live')`)) + + require.Zero(t, count(`SELECT count(*) FROM sub2api_plugin_leases WHERE plugin_key = $1 AND namespace IN ('routing-account', 'tickets')`, codex)) + require.Equal(t, 1, count(`SELECT count(*) FROM sub2api_plugin_leases WHERE plugin_key = 'acme.other-plugin' AND lease_key = 'm263'`)) + + require.Zero(t, count(`SELECT count(*) FROM admin_account_jobs WHERE id = ANY($1)`, retiredJobs)) + require.Zero(t, count(`SELECT count(*) FROM admin_account_job_items WHERE job_id = ANY($1)`, retiredJobs)) + require.Equal(t, 1, count(`SELECT count(*) FROM admin_account_jobs WHERE id = $1`, keptJob)) + require.Equal(t, 1, count(`SELECT count(*) FROM admin_account_job_items WHERE job_id = $1`, keptJob)) + + // The triggers went before their tables, so account status/credential + // writes and setting writes no longer reach the dropped tables. + exec(`UPDATE accounts SET status = 'disabled', credentials = credentials || '{"chatgpt_account_id":"changed"}'::jsonb WHERE id = $1`, ticketID) + exec(`UPDATE settings SET value = '0.157.0' WHERE key = 'openai_codex_client_version'`) + + ticketCTID := ctid("accounts", "id", ticketID) + exec(migration(codexTicketPurgeMigration)) + requireCodexTicketObjects(t, ctx, tx, false) + require.Equal(t, ticketCTID, ctid("accounts", "id", ticketID), "a second run rewrites no account") + require.Equal(t, wireRevision+1, count(`SELECT revision FROM sub2api_plugin_state WHERE plugin_key = $1 AND namespace = $2 AND state_key = 'wire.m263-cookies'`, codex, private)) + require.Equal(t, 1, count(`SELECT count(*) FROM admin_account_jobs WHERE id = $1`, keptJob)) +} + +func requireCodexTicketObjects(t *testing.T, ctx context.Context, tx *sql.Tx, present bool) { + t.Helper() + var relations, triggers, functions int + require.NoError(t, tx.QueryRowContext(ctx, ` +SELECT + (SELECT count(*) FROM unnest(ARRAY['openai_codex_ticket_runtime', 'openai_codex_ticket_due_idx', + 'openai_codex_ticket_proxy_trust', 'openai_codex_ticket_proxy_generation']) AS name + WHERE to_regclass(name) IS NOT NULL), + (SELECT count(*) FROM pg_trigger WHERE tgname IN ('codex_ticket_account_claim_guard', 'codex_ticket_proxy_trust_guard')), + (SELECT count(*) FROM pg_proc WHERE proname IN ('invalidate_codex_ticket_account_claims', 'invalidate_codex_ticket_proxy_trust')) +`).Scan(&relations, &triggers, &functions)) + if present { + require.Equal(t, []int{4, 2, 2}, []int{relations, triggers, functions}) + return + } + require.Equal(t, []int{0, 0, 0}, []int{relations, triggers, functions}) +} diff --git a/backend/internal/repository/scheduler_cache_reasoning_policy_test.go b/backend/internal/repository/scheduler_cache_reasoning_policy_test.go index 8bfe3ea02478..8a0c92b2a3a2 100644 --- a/backend/internal/repository/scheduler_cache_reasoning_policy_test.go +++ b/backend/internal/repository/scheduler_cache_reasoning_policy_test.go @@ -10,17 +10,6 @@ import ( "github.com/stretchr/testify/require" ) -// The retired route-qualification projection stays dormant in account extra -// and never enters a scheduler snapshot. -func TestSchedulerCacheDropsRetiredCodexProjection(t *testing.T) { - projection := map[string]any{"identity": "owner", "scheduling": map[string]any{"gpt-6-astra": map[string]any{"effect": "deny"}}} - source := map[string]any{service.NativeCodexAccountProjectionKey: map[string]any{"codexrip.codex-runtime": projection}, "openai_quota_status": "ok"} - filtered := filterSchedulerExtra(source) - require.NotContains(t, filtered, service.NativeCodexAccountProjectionKey) - require.Equal(t, "ok", filtered["openai_quota_status"]) - require.Contains(t, source, service.NativeCodexAccountProjectionKey, "the persistent account extra is not rewritten") -} - func TestSchedulerCacheReasoningPolicyProjectionPreservesMissingAndBooleanSemantics(t *testing.T) { keys := []string{service.OpenAIChatReasoningReplayEnabledExtraKey, service.OpenAIReasoningSignatureRecoveryEnabledExtraKey} for _, source := range []map[string]any{nil, {}, {"unrelated": true}, {"quota_limit": nil}} { diff --git a/backend/internal/service/account_job.go b/backend/internal/service/account_job.go index 925f02aae73a..51e8f2c2b5b1 100644 --- a/backend/internal/service/account_job.go +++ b/backend/internal/service/account_job.go @@ -300,8 +300,8 @@ func (s *AccountJobService) decorateRetryEligibility(ctx context.Context, job *A return } if !validAccountJobKind(job.Kind) { - // A retired kind (route acquisition, renewal stop, legacy extension - // operation) still lists, but it has no executor to retry with. + // A kind this version has no executor for (a row written by another + // version) still lists, but it cannot be retried. job.RetryUnavailableReason = "kind_unsupported" return } diff --git a/backend/internal/service/account_job_test.go b/backend/internal/service/account_job_test.go index e6e78b72d717..3de4bcde2c61 100644 --- a/backend/internal/service/account_job_test.go +++ b/backend/internal/service/account_job_test.go @@ -262,10 +262,10 @@ type accountJobTestExecutor struct { cleanupCalls int } -// Retired kinds (route acquisition, renewal stop, legacy extension operation) -// keep their rows readable, but nothing can submit or retry them. +// A row of a kind this version has no executor for (written by another +// version) stays readable, but nothing can submit or retry that kind. func TestAccountJobRetiredKindsStayReadableButCannotRun(t *testing.T) { - for _, kind := range []string{"codex_ticket_harvest", "codex_ticket_stop", "extension_operation"} { + for _, kind := range []string{"kind_from_another_version"} { t.Run(kind, func(t *testing.T) { repo := newAccountJobTestRepo() jobs := NewAccountJobService(repo, accountJobTestCipher{}) diff --git a/backend/internal/service/account_test_prompt_test.go b/backend/internal/service/account_test_prompt_test.go index b7ebb4b581b3..1905bb881c4d 100644 --- a/backend/internal/service/account_test_prompt_test.go +++ b/backend/internal/service/account_test_prompt_test.go @@ -110,7 +110,7 @@ func TestAccountTestPromptOAuthMappedModelReportsFinalWire(t *testing.T) { a.Credentials["model_mapping"] = map[string]any{"alias": "gpt-5.6-sol"} a.Credentials["header_override_enabled"] = true a.Credentials["header_overrides"] = map[string]any{openAICodexTurnStateHeader: "header-override-state"} - a.Extra = map[string]any{"codex_turn_ticket:gpt-5.6-sol": map[string]any{"state": "dormant-state"}, NativeCodexAccountProjectionKey: map[string]any{NativeCodexPluginKey: map[string]any{"scheduling": map[string]any{"gpt-5.6-sol": map[string]any{"effect": "deny"}}}}} + a.Extra = map[string]any{"codex_turn_ticket:gpt-5.6-sol": map[string]any{"state": "dormant-state"}, "plugin_account_projections": map[string]any{NativeCodexPluginKey: map[string]any{"scheduling": map[string]any{"gpt-5.6-sol": map[string]any{"effect": "deny"}}}}} upstream := &queuedHTTPUpstream{responses: []*http.Response{newJSONResponse(200, "data: {\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\ndata: {\"type\":\"response.completed\",\"response\":{\"status\":\"completed\",\"model\":\"gpt-5.6-sol\"}}\n\n")}} gateway := &OpenAIGatewayService{cfg: cfg, httpUpstream: upstream, accountRepo: &codexAccountRepositoryFixture{account: a}, nativeCodexRuntime: nativeCodexTestRuntime(t)} svc := &AccountTestService{cfg: cfg, httpUpstream: upstream, openAIGatewayService: gateway} diff --git a/backend/internal/service/codex_native_contract.go b/backend/internal/service/codex_native_contract.go index 1cf01745adec..af8c6527bd64 100644 --- a/backend/internal/service/codex_native_contract.go +++ b/backend/internal/service/codex_native_contract.go @@ -13,11 +13,6 @@ const NativeCodexConfigSettingKey = "codex_native_runtime_config" const NativeCodexSourceSettingKey = "codex_native_runtime_source" const NativeCodexRetirementSettingKey = "deplugin_retired_plugins" -// NativeCodexAccountProjectionKey names the account extra key that held the -// retired route-qualification scheduling projection. Nothing reads it; account -// writes only carry an existing value over unchanged until the data is purged. -const NativeCodexAccountProjectionKey = "plugin_account_projections" - var ErrNativeCodexRuntimeChanged = errors.New("native Codex runtime metadata changed") var ErrNativeCodexRuntimeUnavailable = errors.New("native Codex runtime unavailable") diff --git a/backend/internal/service/openai_account_scheduler_test.go b/backend/internal/service/openai_account_scheduler_test.go index 5eaaee67aad6..b07610094f63 100644 --- a/backend/internal/service/openai_account_scheduler_test.go +++ b/backend/internal/service/openai_account_scheduler_test.go @@ -4091,7 +4091,7 @@ func TestSelectAccountWithSchedulerServesOAuthAccountsWithoutRouteQualification( ID: 42, Platform: PlatformOpenAI, Type: AccountTypeOAuth, Status: StatusActive, Schedulable: true, Concurrency: 1, GroupIDs: []int64{groupID}, Credentials: map[string]any{"access_token": "scheduler-token", "chatgpt_account_id": "scheduler-principal", "model_mapping": map[string]any{"gpt-6-astra": "gpt-6-astra"}}, - Extra: map[string]any{NativeCodexAccountProjectionKey: map[string]any{NativeCodexPluginKey: map[string]any{ + Extra: map[string]any{"plugin_account_projections": map[string]any{NativeCodexPluginKey: map[string]any{ "identity": "dormant", "scheduling": map[string]any{"gpt-6-astra": map[string]any{"model": "gpt-6-astra", "effect": "deny"}}, }}}, } diff --git a/backend/migrations/263_purge_retired_codex_ticket_data.sql b/backend/migrations/263_purge_retired_codex_ticket_data.sql new file mode 100644 index 000000000000..849668b56532 --- /dev/null +++ b/backend/migrations/263_purge_retired_codex_ticket_data.sql @@ -0,0 +1,151 @@ +-- Purge the data of the retired Codex route acquisition ("ticket") feature. +-- The previous release removed every reader and writer of it, so nothing in +-- the application reads what this migration deletes. Evidence names the writer +-- in the last source that had one (git d81cd8be4, paths below +-- backend/internal), or the plugin-era source where noted. +-- +-- Deleted: +-- * The objects of 244/245 (no later migration altered them): the triggers +-- codex_ticket_account_claim_guard (accounts) and +-- codex_ticket_proxy_trust_guard (settings) and their functions +-- invalidate_codex_ticket_account_claims() and +-- invalidate_codex_ticket_proxy_trust() go first, because the account +-- trigger writes openai_codex_ticket_runtime and a function cannot be dropped +-- while a trigger uses it. Then the tables openai_codex_ticket_runtime (with +-- openai_codex_ticket_due_idx), openai_codex_ticket_proxy_trust and +-- openai_codex_ticket_proxy_generation. +-- * settings openai_codex_ticket_enabled and +-- openai_codex_ticket_harvest_proxy_url, the only keys the feature stored +-- (service/domain_constants.go:723,725; openai_codex_ticket_clear_proxy and +-- openai_codex_ticket_harvest_proxy_configured were a request flag and a +-- response field, never stored). +-- * accounts.extra keys codex_turn_ticket:* (service/openai_codex_ticket.go:15), +-- codex_ticket_runtime:* (service/codex_ticket_lifecycle.go:14) and +-- codex_harvest_proxy_url (service/openai_codex_ticket.go:167-171), and the +-- codexrip.codex-runtime entry of plugin_account_projections, which only the +-- Codex runtime wrote (repository/codex_native_state.go:56-66,78-113); the +-- projection map itself goes only when no other entry is left. Rows that hold +-- none of these keys are not written. +-- * sub2api_plugin_state rows of plugin codexrip.codex-runtime in the +-- namespaces tickets (codexruntime/tickets/module.go:494,509), +-- routing-demand (codexruntime/tickets/routing.go:208-227) and proxy-trust +-- (pinned harvest-proxy certificates, codexruntime/tickets/proxy.go:364,423), +-- and the route and Cookie material of codex-routing-private: +-- bundle.* route bundles holding raw Cookie values +-- (service/codex_routing_cookies.go:45-55; keys from +-- service/codex_routing_redaction.go:21 and +-- service/codex_quality_runtime.go:536) +-- clock.* Cookie clocks holding raw Cookie values +-- (service/codex_routing_cookies.go:38-43; keys from +-- service/codex_routing_host.go:154 and +-- service/codex_quality_runtime.go:543) +-- seen.* first-seen index per Cookie value +-- (service/codex_routing_cookies.go:173-216) +-- validation-result.* stored verified route probe with its bundle and +-- connection lease (service/codex_routing_validation.go:168,190-195) +-- * sub2api_plugin_leases rows of codexrip.codex-runtime in routing-account +-- (codexruntime/tickets/routing.go:45, service/codex_quality_route.go:75, +-- service/codex_routing_validation.go:136) and tickets (plugin-era per-ticket +-- leases, git 769361555 plugins/codex-runtime/internal/tickets/module.go:391). +-- * admin_account_jobs of the kinds codex_ticket_harvest, codex_ticket_stop and +-- extension_operation (service/account_job.go:19,23,24); their items follow +-- through ON DELETE CASCADE (232). +-- +-- Kept in codex-routing-private: +-- * quality-run.* quality diagnostic ledger, read by this release; it stores no +-- Cookie or STATE value (service/codex_quality_state.go:104-128). +-- * validation.* route validation budget ledger +-- (service/codex_routing_validation.go:41-45,54-99). +-- * spent.* send-once ledger written before probe IO, value +-- {"spent":true} (service/codex_routing_host.go:136-148). +-- * wire.* last outbound request of an account, read by this release, +-- without its retired "cookies" member: the name/value pairs of the Cookie +-- header as sent (service/codex_routing_fingerprint.go:74-96,120,204). Its +-- other members, digests included, stay. +-- Audit logs, every other setting (codex_native_runtime_source and +-- codex_native_runtime_config included), other plugins' state, account +-- credentials and any certificate other than the harvest-proxy pins above are +-- not touched. +-- +-- accounts has no updated_at trigger. Writing extra fires the long-context +-- billing triggers of 175, which keep the stored flag and therefore queue no +-- scheduler_outbox event; the scheduler rebuild at startup reloads cached +-- account payloads. Every statement is a no-op once its objects or rows are +-- gone, so the migration can run again. +SET LOCAL lock_timeout = '10s'; +SET LOCAL statement_timeout = '120s'; + +DROP TRIGGER IF EXISTS codex_ticket_account_claim_guard ON accounts; +DROP TRIGGER IF EXISTS codex_ticket_proxy_trust_guard ON settings; +DROP FUNCTION IF EXISTS invalidate_codex_ticket_account_claims(); +DROP FUNCTION IF EXISTS invalidate_codex_ticket_proxy_trust(); +DROP TABLE IF EXISTS openai_codex_ticket_runtime; +DROP TABLE IF EXISTS openai_codex_ticket_proxy_trust; +DROP TABLE IF EXISTS openai_codex_ticket_proxy_generation; + +DELETE FROM settings +WHERE key IN ('openai_codex_ticket_enabled', 'openai_codex_ticket_harvest_proxy_url'); + +WITH retired AS ( + SELECT a.id, + a.extra - ARRAY( + SELECT k + FROM jsonb_object_keys(a.extra) AS k + WHERE starts_with(k, 'codex_turn_ticket:') + OR starts_with(k, 'codex_ticket_runtime:') + OR k = 'codex_harvest_proxy_url' + ) AS extra, + COALESCE( + jsonb_typeof(a.extra -> 'plugin_account_projections') = 'object' + AND (a.extra -> 'plugin_account_projections') ? 'codexrip.codex-runtime', + FALSE + ) AS projected + FROM accounts AS a + WHERE jsonb_typeof(a.extra) = 'object' +) +UPDATE accounts AS a +SET extra = CASE + WHEN NOT r.projected THEN r.extra + WHEN (r.extra -> 'plugin_account_projections') - 'codexrip.codex-runtime' = '{}'::jsonb + THEN r.extra - 'plugin_account_projections' + ELSE jsonb_set( + r.extra, + '{plugin_account_projections}', + (r.extra -> 'plugin_account_projections') - 'codexrip.codex-runtime' + ) + END +FROM retired AS r +WHERE a.id = r.id + AND (r.projected OR r.extra <> a.extra); + +DELETE FROM sub2api_plugin_state +WHERE plugin_key = 'codexrip.codex-runtime' + AND ( + namespace IN ('tickets', 'routing-demand', 'proxy-trust') + OR ( + namespace = 'codex-routing-private' + AND ( + starts_with(state_key, 'bundle.') + OR starts_with(state_key, 'clock.') + OR starts_with(state_key, 'seen.') + OR starts_with(state_key, 'validation-result.') + ) + ) + ); + +UPDATE sub2api_plugin_state +SET value = value - 'cookies', + revision = revision + 1, + updated_at = NOW() +WHERE plugin_key = 'codexrip.codex-runtime' + AND namespace = 'codex-routing-private' + AND starts_with(state_key, 'wire.') + AND jsonb_typeof(value) = 'object' + AND value ? 'cookies'; + +DELETE FROM sub2api_plugin_leases +WHERE plugin_key = 'codexrip.codex-runtime' + AND namespace IN ('routing-account', 'tickets'); + +DELETE FROM admin_account_jobs +WHERE kind IN ('codex_ticket_harvest', 'codex_ticket_stop', 'extension_operation'); diff --git a/frontend/src/components/admin/account-jobs/AccountOperationProgress.vue b/frontend/src/components/admin/account-jobs/AccountOperationProgress.vue index e0f7bbf88954..3695b49ceb1b 100644 --- a/frontend/src/components/admin/account-jobs/AccountOperationProgress.vue +++ b/frontend/src/components/admin/account-jobs/AccountOperationProgress.vue @@ -45,19 +45,11 @@

{{ itemLabel(item) }}

-

{{ item.metadata.model_id }}

-

{{ item.metadata.label }}

{{ statusLabel(item.status) }}

{{ item.metadata.message }}

[{{ item.error_code }}]{{ item.error_message }}

-
- -