[napetrov]

Summary

repo run and repo reach had one built-in harness, Claude Code. This adds the Codex CLI and the Copilot CLI, including each CLI's own review mode, so the same task can be compared across guidance arms × harness × model.

 dokimi/repos/
 ├── profiles/
 │   ├── claude-code.yaml
+│   ├── codex.yaml            # codex exec --json … -- {prompt}
+│   ├── codex-review.yaml     # codex exec review --commit HEAD (native prompt)
+│   ├── copilot.yaml          # copilot -p {prompt} --output-format json
+│   └── copilot-review.yaml   # copilot -p "/review {prompt}"
 ├── events.py                 # trace: + codex-json, copilot-json
+├── cli_traces.py             # the two JSON converters -> events.jsonl
+├── answers.py                # answer_format: codex-review -> findings block
 ├── runner.py                 # answer_format applied; bin_companions mounted
 └── reach.py                  # refuses native_prompt; now writes config_files

harness_profile.v1 is widened in place with three optional keys:

native_prompt: true          # the harness's own prompt; invoke has no {prompt}
answer_format: codex-review  # "- [P1] title — path:12-14" -> ```json [{file, line, severity, rule, message}]
bin_companions: [codex-code-mode-host]   # mounted beside /eval/harness/codex

repo run cell (codex-review)
  agent container: /eval/harness/codex + codex-code-mode-host
    codex exec review --commit HEAD --json      # task prompt not passed
  convert("codex-json") -> events.jsonl
  convert_answer("codex-review") -> answer.txt  # text kept, ```json block appended
  grade                                         # review grader unchanged

Things that only show up when running the real CLIs:

• codex review --commit refuses a prompt.
• Codex 0.160 runs every tool call through codex-code-mode-host beside its binary, so a single-file mount leaves it with no tools.
• An empty Copilot --available-tools list leaves every tool on, so none names a tool that doesn't exist.
• npm installs both CLIs as Node launchers; --bin has to name the native binary.

Not here: hosted PR reviewers (Copilot code review, @codex review, CodeRabbit). They are not CLIs and need a separate PR-replay adapter.

Evidence

• Before: repo run --harness codex fails with no harness profile 'codex' (built-in: claude-code). A profile with a Codex provider in config_files probed via repo reach hit wss://api.openai.com with a 401, because reach never wrote the file.
  After: live runs on oneDAL review_seeded_fc (fork dev/agent-evals c4e2667f5, image agent-benchmark/onedal:a28e8a6c33a7, gpt-6-luna through a LiteLLM proxy, guidance:raw, 1 rep):

  harness	recall	events	notes
  codex	0.50	12 command, 9 read
  codex-review	0.375	7 command, 8 read	same recall as an out-of-tree harness on 10-04
  copilot-review	0.625	5 command, 14 read, 13 tool	11 turns

  The first codex-review run, without bin_companions, answered "the repository command tool failed to start" and got recall 0.0.

• Live repo reach on oneDAL main 7f6d9f86 (3 scenarios, gpt-6-luna):

                         cwd=.  cwd=cpp/daal  read=cpp/daal
  codex   AGENTS.md        L       L             L
          cpp/AGENTS.md    .       L             .
          cpp/daal/AGENTS  .       L             .
  copilot AGENTS.md        L       L             .   <- model refused to list tokens (prose answer)
          cpp/AGENTS.md    .       L             .
          cpp/daal/AGENTS  .       L             .
  

  Both CLIs load AGENTS.md from the git root down to cwd, and reading a file in a subtree does not pull in that subtree's AGENTS.md. In read= Codex skipped the read, because the reach ask says "use the Read tool" and Codex has no tool by that name. Rewording the ask is left for a follow-up: it would change Claude's recorded ask too.

• Tests: tests/test_repo_cli_harnesses.py (29 tests) covers:

  • argv for each profile;
  • BYOK env pass-through;
  • refused native profiles;
  • shell_reads, including pipes, redirects and sed -i;
  • both converters, on shapes taken from real traces;
  • codex_review_findings;
  • a repo run cell that grades a native review answer;
  • bin_companions mounted, or a stop before any agent runs.

  test_repo_reach.py adds a test that a profile's config_files reach the probe.

• make ci-local: every row passes except test. There, every PR-touched test passes, but 1–3 unrelated tests fail at random, a different one each run: test_process Popen ResourceWarning, test_container_agent "AF_UNIX path too long". This happens under 224 xdist workers on a shared box. pytest -n 32 gives 5208 passed, 2 skipped. Those two files alone: 149 passed.

Merge Danger

Door: two-way

New profiles and optional schema keys only. Profiles that already exist are left as they were. schema_compat passes against the base.

Blast Radius: repo reach

Every reach run now writes the profile's config_files. For claude-code that adds the no-background hook settings.json to each scenario's config dir. Reach scenarios run with tools none/read, so the Bash hook never fires. Only repo run / repo reach read the new keys.

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.81707% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.53%. Comparing base (9a46235) to head (5d4eff6).

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/cli_traces.py	93.52%	17 Missing and 13 partials ⚠️
dokimi/dokimi/repos/harness_profile.py	92.85%	1 Missing and 1 partial ⚠️
dokimi/dokimi/repos/transcript.py	96.66%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #116      +/-   ##
==========================================
+ Coverage   93.50%   93.53%   +0.02%     
==========================================
  Files         273      276       +3     
  Lines       35569    36153     +584     
  Branches     6040     6174     +134     
==========================================
+ Hits        33260    33814     +554     
- Misses       1492     1509      +17     
- Partials      817      830      +13     

Files with missing lines	Coverage Δ
dokimi/dokimi/repos/answers.py	100.00% <100.00%> (ø)
dokimi/dokimi/repos/events.py	91.47% <100.00%> (-1.31%)	⬇️
dokimi/dokimi/repos/reach.py	96.61% <100.00%> (+0.06%)	⬆️
dokimi/dokimi/repos/runner.py	95.45% <100.00%> (+0.18%)	⬆️
dokimi/dokimi/repos/harness_profile.py	96.42% <92.85%> (+0.45%)	⬆️
dokimi/dokimi/repos/transcript.py	96.66% <96.66%> (ø)
dokimi/dokimi/repos/cli_traces.py	93.52% <93.52%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🟡 Changes recommended

Trace conversion can distort evaluation results, and reach configuration produces invalid JSON on Windows.

4 open issues

• Windows paths produce invalid JSON in Claude profile commands · New
• Shell quote splitting drops reads from multi-range sed commands · New
• Directory changes are ignored when resolving subsequent file reads · New
• Interrupted commands are omitted from converted traces · New

What changed in this PR

Adds Codex and Copilot CLI harnesses so repository evaluations can compare guidance across agents, including native review modes.

Changes:

• Adds four harness profiles and optional profile settings.
• Converts CLI traces and native review answers into evaluation artifacts.
• Applies profile configuration during reach probes, with tests and documentation.

File	Description
dokimi/​tests/​test_repo_reach.py	Tests reach configuration generation.
dokimi/​tests/​test_repo_cli_harnesses.py	Tests profiles, converters, and runner integration.
dokimi/​dokimi/​schemas/​harness_profile.v1.json	Adds optional harness capabilities.
dokimi/​dokimi/​repos/​runner.py	Mounts companion binaries and converts answers.
dokimi/​dokimi/​repos/​reach.py	Writes configuration and rejects native prompts.
dokimi/​dokimi/​repos/​profiles/​copilot.yaml	Adds Copilot execution profile.
dokimi/​dokimi/​repos/​profiles/​copilot-review.yaml	Adds Copilot review profile.
dokimi/​dokimi/​repos/​profiles/​codex.yaml	Adds Codex execution profile.
dokimi/​dokimi/​repos/​profiles/​codex-review.yaml	Adds Codex native review profile.
dokimi/​dokimi/​repos/​harness_profile.py	Parses and validates new settings.
dokimi/​dokimi/​repos/​events.py	Dispatches new trace formats.
dokimi/​dokimi/​repos/​cli_traces.py	Normalizes Codex and Copilot events.
dokimi/​dokimi/​repos/​answers.py	Converts native review findings.
dokimi/​docs/​repo-evaluation-contract.md	Documents profiles and evaluation behavior.
dokimi/​docs/​architecture.md	Describes new modules and harnesses.
dokimi/​changelog.d/​20261019_050000_nikolay.a.petrov_codex_copilot_profiles.md	Records harness additions and reach fix.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Trace correlation and path attribution defects affect evaluation accuracy, and typechecking and Windows test issues remain.

5 open issues

• Correlate command events by item ID · New
• Remove duplicate event annotation · New
• Resolve reads relative to the execution directory · New
• Compose and normalize relative command paths · New
• Normalize expected paths in Windows test · New

4 resolved since last review

• Windows paths produce invalid JSON in Claude profile commands
• Interrupted commands are omitted from converted traces
• Directory changes are ignored when resolving subsequent file reads
• Shell quote splitting drops reads from multi-range sed commands

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The test-package conversion leaves sibling imports unresolved, breaking test collection and execution.

6 open issues

• Fix remaining sibling imports after converting tests to a package · New
• Remove duplicate event annotation
• Correlate command events by item ID
• Normalize expected paths in Windows test
• Compose and normalize relative command paths
• Resolve reads relative to the execution directory

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Overlapping identical Codex commands produce incorrect trace events used by graders.

7 open issues

• Track overlapping identical commands by item ID · New
• Fix remaining sibling imports after converting tests to a package
• Remove duplicate event annotation
• Correlate command events by item ID
• Normalize expected paths in Windows test
• Compose and normalize relative command paths
• Resolve reads relative to the execution directory

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Trace accuracy and the reported incomplete CI pass need human verification before approval.

5 open issues

• Track overlapping identical commands by item ID
• Correlate command events by item ID
• Normalize expected paths in Windows test
• Compose and normalize relative command paths
• Resolve reads relative to the execution directory

2 resolved since last review

• Fix remaining sibling imports after converting tests to a package
• Remove duplicate event annotation

_{Review effort: Balanced}

[antonwolfy]

Review at 27911d6. The profiles, schema widening and mounting logic hold up, but the trace converters have several wrong-result cases, and the Codex findings parser is fragile. Four things are worth fixing before merge; the inline comments have the inputs and outputs.

What I ran, on a Linux review host at 27911d6:

• make ci-local: 19 passed, 0 failed, 4 skipped (docker/semgrep/jobs that do not run on a PR); changed-line coverage gate passes.
• Reproductions for the code findings, with no model or network calls.
• A mutation run over the new tests: 36 of 83 mutants of cli_traces.py pass the suite; two of the severity-mapping mutants in answers.py pass too.

I agree with Copilot that Codex commands are matched by text, not item.id; it still holds (two overlapping make build items give rc [None, 0, 1]).

Checked, no issues

• Schema: the harness_profile.v1.json change only widens the trace enum and adds optional keys, so no version bump is needed; schema_compat passes against the base; bin_companions names reject .. and /.
• Docs: flags, keys, section numbers and the profile table match the code; the changelog fragment is present; no tests/__init__.py or uv.lock remains after the reverts.
• Mounts and reach: a missing companion fails with a clear error, a symlinked --bin finds the companion beside the real file, and repo reach refuses native_prompt profiles before --dry-run.

[napetrov]

This review has no body text, and each of its inline threads has its own answer in the thread, so nothing in the review itself needs a change.

[napetrov]

This review has no body text, and each of its inline threads has its own answer in the thread, so nothing in the review itself needs a change.

[napetrov]

Both inline comments of this review (comment blocks in the profiles) are fixed in 902d3d6 and answered in their threads.

[napetrov]

The five inline comments of this review are fixed and answered in their threads: quoted redirects 2eb0369, computed cd 12bc4bb, failed Copilot reads 2aabefe, companion digests 7283f17, config file names cf5fe01.

[napetrov]

All five findings in this summary are fixed and answered in their threads: reach.py:373 in cf5fe01, harness_profile.py:163 in 7283f17, cli_traces.py:142 in 12bc4bb, cli_traces.py:618 in 2aabefe, cli_traces.py:133 in 2eb0369.

[napetrov]

(a) Waiting on a decision before I change this: an agent run can have an earlier message with a [P1] finding and a final message that says "No issues found." Should the earlier finding still count? Options: (1) the final verdict replaces earlier drafts, so only the final message counts; (2) findings from every message count, as now. Recommendation: (1), since the final message is the verdict and a draft it retracts should not block. Reply 1 or 2. (b) Test added in 8ee9e27: an em-dash-only drifted header after a finding withholds the block. Pinned by test_convert_answer_withholds_the_block_for_a_drifted_header_with_no_path; a mutant without the em-dash alternative fails three tests. (c) Fixed in 9557e72 by Nikolay Petrov: a body bullet such as - [P2] see b.py:4 stays body text. (d) Fixed in 9557e72 by Nikolay Petrov: Codex cached input is split out of input tokens, pinned by test_codex_cached_input_is_split_out_of_input_tokens_as_cache_reads.

[napetrov]

This review has no body text, and its inline thread has its own answer in the thread, so nothing in the review itself needs a change.

[exai-bot]

The answer_format text says codex-review appends a findings block, but convert_answer (repos/answers.py) leaves the answer unchanged in three cases: the run failed, the answer is empty, or the text has more [Pn] markers than parsed findings (format drift). It also sets severity to null for a priority outside P0–P3. A reader can't tell from the doc that a missing block means "no review" or "drifted format", not a clean review. A clean review gets []. Add one sentence covering these cases.