Skip to content

feat(accounts): candy-question quality check in the account test dialog - #231

Merged
HTExplicit merged 1 commit into
mainfrom
feat/account-test-quality-check
Oct 3, 2026
Merged

HTExplicit merged 1 commit into
mainfrom
feat/account-test-quality-check

Conversation

@HTExplicit

Copy link
Copy Markdown
Owner

Problem

With the ticket subsystem gone (#229), admins need a quick, manual way to see whether an OpenAI OAuth account currently answers like a degraded model. Our own records show the candy question separates the two reliably: correct answers are 21, while degraded answers are typically 29.

Change

Account test dialog, OpenAI OAuth accounts only (owner-approved preview, 2026-10-03):

  • New test mode. 「测试模式」 gets a third option, 「质量判分(糖果题)」. Selecting it switches reasoning effort to high when the model offers it; it can be changed, and leaving the mode restores the previous choice.
  • Fixed question. It replaces the custom prompt and is shown read-only, collapsed by default. It is sent unchanged over the ordinary account-test path (SHA-256 05cc85bb…f5a7a7, asserted by a unit test).
  • Verdict card. Shown after the run:
    • 初判: 未见异常 (final number 21), 疑似降智 (another number), or 无法判定 (no number, an incomplete stream or a failed test);
    • the final number and the reasoning tokens, with a hint when the count is exactly 516 (every such answer in our records was wrong);
    • the model the upstream declared, compared with the requested upstream model;
    • a note that the verdict is a hint only.
  • Nothing is stored and scheduling is untouched. A successful run is still a successful connection test, same as before.

The final-number parser prefers explicit answer markers (\boxed{}, 答案/至少需要…/at least …). It skips markers inside hypothetical clauses (「若完全随机摸取,则答案会是 29 个」), ignores counts of shapes, flavours and dates, and resolves arithmetic chains. Over the 134 recorded answers in our evidence it never reports 21 for a wrong answer.

Backend: the account-test stream now reports reasoning_tokens on test_complete for OpenAI Responses tests. The field is optional and omitted otherwise, so other platforms are unchanged.

Dialog-wide: a test stream that ends without a terminal event now shows 「连接在测试完成前结束」 instead of staying on "connecting" (owner decision).

Verification

  • accountQualityCheck.spec.ts: 52 cases (hash, Chinese and English answers, verdict states);
  • AccountTestModal.qualityCheck.spec.ts, plus the existing AccountTestModal specs;
  • check:i18n, vue-tsc --noEmit, eslint on the changed files;
  • go build ./..., and TestAccountConnectionStreamReportsOpenAIReasoningTokens / TestAccountConnectionStreamContract (-tags unit);
  • a local mock-data preview of every state, with no console errors.

Not run: real model calls.

🤖 Generated with Claude Code

OpenAI OAuth accounts get a third test mode, 质量判分(糖果题). It sends the
fixed candy question (SHA-256 05cc85bb…f5a7a7) over the ordinary account
test path with high reasoning effort by default, shows the question
read-only and collapsed, and ends with a verdict card: a first verdict from
the final answer number (21 = no anomaly, another number = suspected
degradation, otherwise undetermined), reasoning tokens with a hint when
they are exactly 516, and the upstream-declared model. Nothing is stored
and scheduling is unaffected.

The test stream now reports reasoning tokens for OpenAI Responses tests,
and a test stream that ends without a terminal event shows an error
instead of staying on "connecting".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HTExplicit
HTExplicit force-pushed the feat/account-test-quality-check branch from 2e4a807 to 25f5579 Compare October 3, 2026 02:20
@HTExplicit
HTExplicit merged commit 06e865f into main Oct 3, 2026
18 checks passed
@HTExplicit
HTExplicit deleted the feat/account-test-quality-check branch October 3, 2026 03:24
HTExplicit added a commit that referenced this pull request Oct 3, 2026
Removes the pieces the Codex ticket removal (#229-#231) left behind and restores the upstream or pre-ticket form in each place.

Deleted: the response model-declaration guard (upstream behaviour: reply delivered, usage log records upstream_model_mismatch); the codex-quality-runs diagnostics; the Codex outbound fingerprint panel with its per-request wire observation and profile selection; plugin-owner job metadata decoding; the retired-key, retired-extra and retired-job tolerance.

Replaced: the native Codex runtime host by one plain settings row {request_zstd} and direct calls into codexruntime/profile and codexruntime/recovery.

Restored: x-codex-turn-state handling to the fork's contract from before the routing commits; Responses-Lite default instructions to upstream.

Migration 264 carries the compression value over from the recorded config hash, drops sub2api_plugin_state and sub2api_plugin_leases, removes the retired codex-runtime installation's stored configuration and package, and writes the fingerprint mode explicitly for OAuth accounts that had none.

A differing upstream model is now shown in the single account test and makes the quality verdict suspect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant