test(evals): compare production Standard scan time budgets

[mldangelo-oai]

Summary

Compare production Standard scans with and without an advisory time allowance. Stacked on the runtime support in #1674; this PR contains only evaluation code, fixtures, documentation, and test/build wiring.

Changes

• Run both arms through CodexSecurity.run, including real prompt construction, canonical artifact validation, completion, and delegated-worker usage accounting. Only the treatment sets maxTimeMinutes.
• Alternate arm order and use opaque per-arm workspaces outside evaluator artifacts. Check API-key and production budget prerequisites before either arm starts; this developer eval does not read or manage saved logins. Archive output and remove state even when client cleanup fails; retain scan and cleanup diagnostics.
• Capture actual prompts, evaluator/shared-helper/source/full-plugin/built-SDK hashes, package versions, execution failures, monotonic elapsed time, and worker-inclusive token usage even when completion fails. Exclude generated reports from evaluator hashes.
• Grade a synthetic application with four vulnerability families and matching safe controls. Automatic credit requires a root_control or sink citation; retain other-role findings and control overlaps for manual adjudication rather than automatically crediting or labeling them false positives.
• Report completed-pair quality/time/token deltas alongside failures and missing usage. Reuse production types, canonical relative source paths, and existing fixture helpers instead of generalizing the unrelated secret-discovery runner.
• Cover production completion and accounting with deterministic tests, wired into the existing source-eval CI path.

Testing

• build:plugin, build:evals, SDK types, Prettier/Markdown checks, and git diff --check pass.
• All 134 deterministic Standard/secret eval tests pass against the rebuilt SDK. All 24 pure Standard tests also pass without an SDK tree or production adapter.
• Regressions reject secondary supporting_context evidence for automatic recall, preserve known primary-role credit and descriptive safe-control advisories, and verify evaluator/shared-helper changes alter provenance while generated reports do not.
• Production integration covers unlabeled dispatch working directories/environment, canonical completion, post-model and client-close failures, state cleanup, readable archives, actual prompts, and aggregate parent/worker usage. A small prerequisite test covers API keys, empty authentication state, and the SDK's deadline validation before model work. All 134 tests also pass under a group-writable default umask.
• Three fresh native reviews plus an independent verifier report no findings. No new live-model comparison was run for this revision.

Risk and rollout

The evaluator is opt-in and requires an existing OPENAI_API_KEY or CODEX_API_KEY, with no effect on production CLI/SDK/plugin login or runtime defaults. Live model runs consume usage and have no enforced time or spending cap. Generated artifacts stay ignored and are not included in this PR.

The small synthetic corpus is a smoke comparison, not evidence of real-world effectiveness. Source matching is heuristic; adjudicate findings and compare recall, false positives, completion, tokens, and elapsed time together. The arms are not spend-matched, and missing usage must remain unknown rather than zero.

Public disclosure review

• No customer, partner, prospect, or user identities, data, or identifying details are included.
• No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
• I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-17T03:27:24.797255Z	0b8be06	New commits
🔒 Security Review	✅ Completed 2026-10-17T03:27:07.355285Z	0b8be06	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 453299b0aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e60b241fe3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 972d96c5cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a843299654

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e102079737

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6dee75e349

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".