[vchamarthi]

Summary

run, compare, arms, tasks, explore and repo read user-supplied files (spec, snapshot, arms, task, batch, manifest, --matrix-cells, --harness, --static) and write --out-json/--out-md without handling OS, decode or nesting errors, so a bad file or path ends in a traceback.

Changes

• Each affected read or write reports the path and reason in a one-line message and exits.
• run and compare exit 2 on a bad spec, snapshot or document selection (DocSelectionError, CompareInputError) and 1 on an unreadable document or an unwritable output. 1 is also the --strict verdict code; the README says so. Errors raised inside metrics or the regression check keep their traceback.
• repo report and repo compare check every field they read in result.json and name the file when one is unusable. A field of the wrong shape that the checks miss is reported per cell, naming its files; errors in the statistics code are not caught.
• explore rejects a non-string repo_root (including 0 and []) and names the task file on an invalid YAML timestamp.

Risk

Mostly error paths. Three changes affect recorded or read data:

• repo run records a non-integer agent.num_turns as null and notes it in agent.trace_errors.
• A grader score that is NaN, infinite or beyond 1e150 is a grader failure, both when repo run grades and when repo report/repo compare read a batch that already stores one.
• repo report and repo compare leave a process counter or grade.json metric that is beyond 1e150 (or an integer too large for a float) out of the means, as they already did for NaN and infinity.

arms run and arms matrix run catch ValueError around the whole arm build, as on main; this PR adds OSError and RecursionError to that catch.

Testing

make ci-local on Linux, Python 3.13: every row passes except pip-audit (filelock advisories, failing on main too). test_unbalanced_quote_in_hook_completes_quickly (a 1 s timing bound, code this PR does not touch) fails intermittently in the full parallel run (1.03 to 1.04 s on a loaded host) and passed alone 3 times on this branch and on main.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A): README run exit codes
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

• _pin_batch still allows OSError from path.read_text(...) (e.g., permission denied, transient… · New
• commands/run.py imports and relies on a private function (_load_spec) from dokimi.runner.run.… · New
• The emitted CLI error prefix here is error: (lowercase), while other updated commands/tests in… · New
• The emitted CLI error prefix here is error: (lowercase), while other updated commands/tests in… · New

What changed in this PR

Improves CLI robustness by converting common file read/write/decode failures across multiple commands into concise, user-facing error messages with deterministic exit codes (notably exit code 2 for run/compare input issues).

Changes:

• Add structured handling for unreadable/malformed input files (YAML/JSON/UTF-8) across run, compare, repo, arms, explore, and tasks.
• Add handling for unwritable output paths for JSON/Markdown report generation.
• Add/extend tests covering the new error paths and ensure tracebacks are not emitted.

File	Description
dokimi/​tests/​test_tasks_command.py	Adds coverage for unwritable tasks run outputs and artifact preservation behavior.
dokimi/​tests/​test_run_compare_errors.py	New tests asserting run/compare exit 2 with path+reason on bad inputs.
dokimi/​tests/​test_repo_run.py	Adds coverage for unreadable batch pin handling with a user-facing message.
dokimi/​tests/​test_repo_report_unreadable_cell.py	New tests for repo report/compare behavior on corrupt cell files.
dokimi/​tests/​test_repo_l0_cases.py	Adds coverage for non-UTF-8 inputs in repo scan paths.
dokimi/​tests/​test_repo_harness_profile_unreadable.py	New tests ensuring --harness load errors name the file and exit cleanly.
dokimi/​tests/​test_record_readers.py	Adds coverage for unreadable arms inputs exiting cleanly.
dokimi/​tests/​test_explore_task_files.py	New tests for malformed/unreadable explore task YAML inputs.
dokimi/​tests/​test_explore_run_outputs.py	New tests for unwritable explore run outputs.
dokimi/​tests/​test_arms_output_errors.py	New tests for arms plugin-delta unwritable output handling.
dokimi/​tests/​test_arms_matrix_cells_input.py	New tests for unreadable/invalid --matrix-cells handling.
dokimi/​dokimi/​runner/​spec.py	Treats decode failures as readable errors when loading spec YAML.
dokimi/​dokimi/​runner/​compare.py	Unifies snapshot read/parse/decode failures into ValueError with path context.
dokimi/​dokimi/​repos/​report.py	Adds CellFileError and centralized JSON object reading/validation for batch files.
dokimi/​dokimi/​repos/​harness_profile.py	Converts unreadable/non-UTF-8 harness profiles into ProfileError with path.
dokimi/​dokimi/​explore/​tasks.py	Centralizes task YAML reading with path-aware errors; strengthens task schema checks.
dokimi/​dokimi/​eval/​cells.py	Adds path-aware errors for unreadable/invalid matrix-cells JSON files.
dokimi/​dokimi/​commands/​tasks.py	Converts output write failures into one-line SystemExit messages.
dokimi/​dokimi/​commands/​run.py	Adds early spec validation + exit 2 on bad spec/snapshot; improves compare spec parsing errors.
dokimi/​dokimi/​commands/​repo_run.py	Adds detection of unreadable batch pin JSON and emits actionable error.
dokimi/​dokimi/​commands/​repo.py	Treats decode/JSON parse issues in repo scan inputs as clean errors.
dokimi/​dokimi/​commands/​explore.py	Handles explore task parse/read failures and output write failures without tracebacks.
dokimi/​dokimi/​commands/​arms.py	Adds shared helpers to write reports/save artifacts with clean errors; broadens read error handling.
dokimi/​changelog.d/​20261025_120000_vchamarthi_cli-input-run-arms-repo.md	Changelog entry describing the improved error handling and exit codes.

_{Review effort: Lite}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 99.20844% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.48%. Comparing base (0a63f4e) to head (9e8067f).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
dokimi/dokimi/repos/report.py	98.19%	1 Missing and 1 partial ⚠️
dokimi/dokimi/runner/compare.py	97.14%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #238      +/-   ##
==========================================
+ Coverage   93.26%   93.48%   +0.21%     
==========================================
  Files         272      272              
  Lines       35117    35363     +246     
  Branches     5964     6001      +37     
==========================================
+ Hits        32752    33058     +306     
+ Misses       1547     1491      -56     
+ Partials      818      814       -4     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/arms.py	95.48% <100.00%> (+1.75%)	⬆️
dokimi/dokimi/commands/explore.py	81.25% <100.00%> (+25.20%)	⬆️
dokimi/dokimi/commands/repo.py	96.48% <100.00%> (+0.08%)	⬆️
dokimi/dokimi/commands/repo_run.py	93.96% <100.00%> (+0.16%)	⬆️
dokimi/dokimi/commands/run.py	91.52% <100.00%> (+22.19%)	⬆️
dokimi/dokimi/commands/subjects.py	86.63% <100.00%> (ø)
dokimi/dokimi/commands/tasks.py	92.70% <100.00%> (+0.23%)	⬆️
dokimi/dokimi/eval/cells.py	82.50% <100.00%> (+0.76%)	⬆️
dokimi/dokimi/explore/tasks.py	95.04% <100.00%> (+0.75%)	⬆️
dokimi/dokimi/repos/compare.py	99.48% <100.00%> (ø)
... and 10 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review at 4eaef5b. The error handling that is there is right and the new tests fail against the base modules. But the title says these commands exit with a message on bad input or output, and several paths of the same kind still end in a traceback. Five of those are worth fixing before merge, or the title and changelog should be narrowed.

Copilot's point on _pin_batch does not apply: its only caller sits inside try ... except _errors(), and _errors() includes OSError, so a permission error on batch.json already exits 1 with a message.

What I ran, on a Linux review host at 4eaef5b:

• make ci-local: 29 passed, 0 failed, 7 skipped (docker, network and org-level checks).
• Reproductions for the findings, in process through cli.main(), with no model calls.
• The new tests run against the base modules: the tests of the new behaviour fail there.

Checked, no issues

• _pin_batch: a decode or JSON error in batch.json gives the message naming the batch and file.
• Explore tasks: the six tasks under data/explore_tasks and the task-format fixture still load with the new validation.
• Repo rules: changelog fragment format, SPDX headers on the new test files, no generated files. The last commit's subject says "growth baseline", but it only adds the fragment.

[antonwolfy]

I pushed a commit per nit, up to 51a65ee, which fix the nit from my review:

• 51a65ee: dokimi/dokimi/commands/run.py:16: imports the new public load_benchmark_spec (an alias of _load_spec in runner/run.py) instead of the private name.

make ci-local at 51a65ee: 29 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the five risks from my review.

[vchamarthi]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 51a65ee93b59, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 12 of 95 turns, 0.9 of 48 minutes, 383k in / 5k out tokens.

The PR wraps the input reads and output writes of run, compare, arms, tasks, explore and repo so a bad file ends in a one-line message and a non-zero exit. The input handling is careful and well tested. The output-write handling is incomplete in the places that matter most. run, compare and arms run/matrix-run still write their outputs unguarded, although the title and description say output errors are covered.

3 findings are on the changed lines, as comments below.

For a human reviewer

• Decide whether run/compare output-write errors and arms run/matrix-run writes belong in this PR or in a follow-up; the description currently says they are covered.
• Check that exit 2 for a bad spec or snapshot in run/compare, versus exit 1 elsewhere, is the intended CLI contract, since argparse errors also exit 2.

[exai-bot]

Note

AI-generated review of 345757bee1b7, requested by @vchamarthi, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 11 of 94 turns, 0.8 of 48 minutes, 442k in / 4k out tokens.

Re-review of 345757bee1b7: 2 new findings.

2 findings are on the changed lines, as comments below.

[vchamarthi]

@napetrov, on your review:

1. Title and body refreshed; the body now lists the recorded-data changes (num_turns, the grader-failure reading of a stored bad score) under Risk.
2. Changelog line reworded: it is a tasks run --tasks name that matches no task, which exits with a message.

• repo scan --manifest-file catches RecursionError (commands/repo.py:321); the 100000-deep case is in test_a_deeply_nested_manifest_file_exits_1_naming_it.
• run ValueError: see the run.py 34 thread.
• num_turns: a nulled value is now noted in agent.trace_errors (result num_turns '3' is not an integer; recorded as null).
• parse_constant: the bound test parses the output with a parse_constant that raises, so a NaN or Infinity in the output fails it.

@xaleryb, on batches already stored with a NaN, infinite or >1e150 score: Normalized on read: load_cells turns such a stored grade.score into a grader failure (status: grader_failure, success: false, score: null, with a reason) before the field checks, the same record repo run now writes. test_a_stored_unmeasurable_score_is_read_as_a_grader_failure covers NaN, Infinity, -Infinity, 1e200 and a 400-digit integer through repo report and repo compare.