Skip to content

Test the manual SkillEval CLI handoff lifecycle - #291

Merged
Dong1017 merged 1 commit into
mainfrom
test/skill-eval-local-cli-handoff
Oct 31, 2026
Merged

Dong1017 merged 1 commit into
mainfrom
test/skill-eval-local-cli-handoff

Conversation

@Dong1017

Copy link
Copy Markdown
Collaborator

Scope

Refs #253. This test-only slice covers the existing in-process CLI entrypoint; it does not add another runner or change runtime behavior.

  • Exercise prepare, both arms' stage/capture, and report through the real argument parser and dispatch.
  • Retain exact synthetic source, separate author-supplied logs and receipts; reject stale heads, missing flags/submission and repeated capture.
  • Delete live workspaces/submissions before reporting, then verify replay uses retained bytes without changing original result/receipt files.
  • Trap automatic execution and explicitly test a fake verifier plus SandboxUnavailable. Unknown usage and author-reported, unverified comparability remain visible.

Verification and boundaries

Relevant six-suite aggregate: 184 passed, 2 skipped. The 3 new cases also passed independent review/test. Whole-tree Ruff and diff checks passed. Published tree is byte-identical to the reviewed local tree; normal PR CI remains required.

These are synthetic in-process CLI software controls, not a subprocess-isolation qualification, actual local-agent run, 19-case production-verifier result or measured skill benefit. No model/provider/hardware calls. The real author execution → author PR evidence → independent disposition acceptance in #253 remains open; historical #90 cloud results are not rebound to its newer head. No runtime, controller, task fixture, workflow or paid configuration changes.

Authorship

Actual implementation and independent review: Codex / gpt-6-astra / xhigh.

{"schema_version":1,"head_sha":"72baef53605b31f54698627b824507f18e5e2431","human_only":false,"material_ai_authors":[{"tool":"Codex (SkillEval CLI handoff regression)","model_id":"gpt-6-astra","effort":"xhigh","role":"author"}]}

Refs #253. In-process CLI entrypoint software controls only, with an
explicit synthetic verifier seam. No agent/model/hardware run or runtime change.

Implementation: Codex / gpt-6-astra / xhigh.
Signed-off-by: Codex <codex@openai.com>
@Dong1017
Dong1017 marked this pull request as ready for review October 31, 2026 13:59
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T14:00:51.128691Z 72baef5 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped software LGTM for 72baef53605b31f54698627b824507f18e5e2431 against ae0d3221: no blocking finding in this one-file, test-only change. The published test bytes match the independently reviewed local tree. The tests exercise real in-process CLI parsing/dispatch, retained-source replay and refusal paths without changing runtime behavior.

Evidence and acceptance limits
  • Independent offline checks: the 3 new cases passed; the existing local-evaluation suite passed 24 tests. Automatic execution is trapped; the verifier is explicitly synthetic or unavailable.
  • Author's relevant aggregate: 184 passed, 2 pre-existing isolation skips. These skips are not qualification passes.
  • Fresh readback: Ready run 41208043444 completed successfully with all five jobs passing; latest declarations passed. The superseded cancelled run is not counted as a pass. Current-head native review reports no findings; no review threads are open.
  • This supports the CLI regression scope only: not subprocess confinement, an actual author/agent run, the 19-check production verifier, or skill effectiveness. The real author execution → retained PR evidence → independent disposition acceptance in #253 remains open; useful PLANNED guidance is separate work.

This is a COMMENT review, not formal approval or authorization to execute an experiment.

@Dong1017
Dong1017 merged commit 33de003 into main Oct 31, 2026
10 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant