fix(cli): run, compare, arms, tasks, explore and repo exit with a message on bad input or output

dokimi/README.md

@@ -202,7 +202,7 @@ The default static benchmark is configured in `benchmarks/spec.v1.yaml` and curr
 | `readability` | Estimates text readability with a Flesch-Kincaid-style score. |
 | `example_pass_rate` | Executes supported fenced code examples in isolation. |
 
-Weights are normalized across enabled metrics. The benchmark also supports soft gates, hard gates, critical bands, and regression thresholds. Hard gates and critical bands are enforced only when `--strict` is passed. `--strict` also exits 1 when no document was measured.
+Weights are normalized across enabled metrics. The benchmark also supports soft gates, hard gates, critical bands, and regression thresholds. Hard gates and critical bands are enforced only when `--strict` is passed. With `--strict`, a spec without a `golden_manifest` also exits 1 when no document was measured. A spec with one, as `spec.v1.yaml` has, exits 2 with or without `--strict` when fewer documents than its `min_docs` match, none included. An empty or absolute `include` pattern exits 2 as well. Exit 1 is not only the `--strict` verdict: a document that cannot be read, or an `--out-json` or `--out-md` path that cannot be written, also exits 1, with an `Error:` or `❌` line on stderr saying which.
 
 ## LLM and treatment evaluation workflow
 

dokimi/changelog.d/20261025_120000_vchamarthi_cli-input-run-arms-repo.md

@@ -0,0 +1,6 @@
+### Fixed
+
+- **`run`, `compare`, `arms`, `tasks`, `explore` and `repo`** exit with a message instead of a traceback on the unreadable, non-UTF-8 or corrupt spec, snapshot, result, plan, task and question files their tests cover, and on an unwritable output. This covers a deeply nested `--manifest-file`, `--static` or `--harness` file, an unreadable or non-UTF-8 `.agent-eval/repo-eval.yaml`, a `tasks run --tasks` name that matches no task, and an `explore` task file with an invalid YAML timestamp, a bad `lines` range or a `repo_root` that is not a string.
+- `run` and `compare` exit 2 on a bad spec or snapshot (a non-numeric summary value, an empty or absolute `golden_manifest` include pattern, or fewer documents than its `min_docs`) and exit 1 when an output cannot be written, as `arms` does. An error inside a metric is no longer reported as bad input. `tasks` error messages use the `Error:` prefix.
+- `repo report` and `repo compare` exit 1 on a result file without a usable `grade`, `agent`, `task`, `arm` or `model`, naming the file. A grader score that is NaN, infinite or beyond 1e150 is recorded as a grader failure, and a batch that already stores one is read the same way. A `process` counter or `grade.json` metric that is not finite or beyond 1e150 is left out of the means, as a non-number is, instead of ending in an overflow traceback.
+- `repo run` records `agent.num_turns` as null when the harness reports anything but an integer, and notes it in `agent.trace_errors`; a finished run with such a value still counts as started.

dokimi/dokimi/commands/arms.py

@@ -88,7 +88,7 @@ def cmd_arms_run(args: argparse.Namespace) -> None:
         output = _build_arms_output(args, specs, questions)
     except EmptyQuestionError:
         raise  # a bad input record, not an arm setup error: cli.main reports it (exit 2)
-    except (ValueError, FileNotFoundError) as exc:
+    except (ValueError, OSError, RecursionError) as exc:
         print(f"Error building arms/plugins: {exc}", file=sys.stderr)
         sys.exit(1)
 
@@ -108,7 +108,6 @@ def cmd_arms_run(args: argparse.Namespace) -> None:
 
 def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
     """Run every cell in a matrix.cells descriptor and write a rollup."""
-    from dokimi.artifacts import save_artifact
     from dokimi.eval.cells import load_matrix_cells
     from dokimi.eval.matrix_rollup import build_matrix_rollup, strip_internal_payloads
     from dokimi.report.matrix_rollup_report import render_matrix_rollup_report
@@ -124,14 +123,17 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
             cell_args = argparse.Namespace(**vars(args))
             _apply_matrix_cell(cell_args, cell)
             _check_sampling_support(cell_args)
-    except (ValueError, FileNotFoundError) as exc:
+    except (ValueError, OSError, RecursionError) as exc:
         print(f"Error loading matrix run inputs: {exc}", file=sys.stderr)
         sys.exit(1)
 
     out_dir = Path(args.out_dir) if args.out_dir else Path(f"results/arms/{args.product}-matrix")
-    out_dir.mkdir(parents=True, exist_ok=True)
     cells_dir = out_dir / "cells"
-    cells_dir.mkdir(parents=True, exist_ok=True)
+    try:
+        cells_dir.mkdir(parents=True, exist_ok=True)
+    except OSError as exc:
+        print(f"Error: cannot write {cells_dir}: {exc}", file=sys.stderr)
+        sys.exit(1)
     cell_artifacts = []
     for cell in cells:
         cell_args = argparse.Namespace(**vars(args))
@@ -143,7 +145,7 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
             output = _build_arms_output(cell_args, specs, questions)
         except EmptyQuestionError:
             raise  # a bad input record, not a cell error: cli.main reports it (exit 2)
-        except (ValueError, FileNotFoundError) as exc:
+        except (ValueError, OSError, RecursionError) as exc:
             print(f"Error running matrix cell {cell.name!r}: {exc}", file=sys.stderr)
             sys.exit(1)
         _save_arms_output(output, out_json, out_md)
@@ -160,24 +162,22 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
         if not payload:
             continue
         delta_json = Path(row["artifact"])
-        save_artifact("plugin_delta", payload, delta_json)
-        delta_md = delta_json.with_suffix(".md")
-        delta_md.write_text(render_plugin_delta_report(payload), encoding="utf-8")
+        _save_artifact_or_exit("plugin_delta", payload, delta_json)
+        _write_report(delta_json.with_suffix(".md"), render_plugin_delta_report(payload))
         print(f"OK Saved plugin delta:    {delta_json}")
 
     persisted = strip_internal_payloads(rollup)
     out_json = Path(args.out_json) if args.out_json else out_dir / "matrix-rollup.json"
     out_md = Path(args.out_md) if args.out_md else out_json.with_suffix(".md")
-    save_artifact("matrix_rollup", persisted, out_json)
-    out_md.parent.mkdir(parents=True, exist_ok=True)
-    out_md.write_text(render_matrix_rollup_report(persisted), encoding="utf-8")
+    _save_artifact_or_exit("matrix_rollup", persisted, out_json)
+    _write_report(out_md, render_matrix_rollup_report(persisted))
     print(f"OK Saved matrix rollup:   {out_json}")
     print(f"OK Saved matrix report:   {out_md}")
 
 
 def cmd_arms_plugin_delta(args: argparse.Namespace) -> None:
     """Compare paired no-plugin/plugin arms artifacts."""
-    from dokimi.artifacts import load_artifact, save_artifact
+    from dokimi.artifacts import load_artifact
     from dokimi.eval.plugin_delta import PluginDeltaError, compare_plugin_runs
     from dokimi.records import records
     from dokimi.report.plugin_delta_report import render_plugin_delta_report
@@ -189,17 +189,16 @@ def cmd_arms_plugin_delta(args: argparse.Namespace) -> None:
             for section in ("answers", "evaluations"):
                 records(artifact.get(section) or [], source=f"{path} {section}", kind=section)
         output = compare_plugin_runs(baseline, plugin)
-    except (PluginDeltaError, ValueError, FileNotFoundError) as exc:
+    except (PluginDeltaError, ValueError, OSError, RecursionError) as exc:
         print(f"Error computing plugin delta: {exc}", file=sys.stderr)
         sys.exit(1)
 
     out_json = Path(args.out_json) if args.out_json else Path("results/arms/plugin_delta.json")
-    save_artifact("plugin_delta", output, out_json)
+    _save_artifact_or_exit("plugin_delta", output, out_json)
     print(f"OK Saved plugin delta: {out_json}")
 
     out_md = Path(args.out_md) if args.out_md else out_json.with_suffix(".md")
-    out_md.parent.mkdir(parents=True, exist_ok=True)
-    out_md.write_text(render_plugin_delta_report(output), encoding="utf-8")
+    _write_report(out_md, render_plugin_delta_report(output))
     print(f"OK Saved plugin delta report: {out_md}")
 
     print(
@@ -485,7 +484,7 @@ def _refuse_external_trigger_set(path: str) -> None:
 
     try:
         data = json.loads(Path(path).read_text(encoding="utf-8"))
-    except (OSError, ValueError):
+    except (OSError, ValueError, RecursionError):
         return  # the questions loader reports it
     fmt = detect_trigger_format(data)
     if fmt and fmt != FORMAT_QUESTIONS:
@@ -726,14 +725,38 @@ def _build_arms_output(args: argparse.Namespace, specs: list[str], questions: li
     )
 
 
+def _write_report(path: Path, text: str) -> None:
+    """Write a Markdown report, or exit 1 naming ``path`` when it cannot be written."""
+    try:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(text, encoding="utf-8")
+    except OSError as exc:
+        print(f"Error: cannot write {path}: {exc}", file=sys.stderr)
+        sys.exit(1)
+
+
+def _save_artifact_or_exit(kind: str, payload: dict, path: Path) -> None:
+    """``save_artifact``, or exit 1 naming ``path`` when it cannot be written."""
+    from dokimi.artifacts import save_artifact
+
+    try:
+        save_artifact(kind, payload, path)
+    except OSError as exc:
+        print(f"Error: cannot write {path}: {exc}", file=sys.stderr)
+        sys.exit(1)
+
+
 def _save_arms_output(output: dict, out_json: Path, out_md: Path) -> None:
     from dokimi.eval.arm_runner import ArmRunner
     from dokimi.report.arms_report import render_arms_report
 
-    ArmRunner.save(output, out_json)
+    try:
+        ArmRunner.save(output, out_json)
+    except OSError as exc:
+        print(f"Error: cannot write {out_json}: {exc}", file=sys.stderr)
+        sys.exit(1)
     print(f"OK Saved arms comparison: {out_json}")
-    out_md.parent.mkdir(parents=True, exist_ok=True)
-    out_md.write_text(render_arms_report(output), encoding="utf-8")
+    _write_report(out_md, render_arms_report(output))
     print(f"OK Saved arms report:     {out_md}")
 
 

dokimi/dokimi/commands/explore.py

@@ -6,6 +6,7 @@
 from __future__ import annotations
 
 import argparse
+import sys
 from pathlib import Path
 
 _TASKS_ROOT_HELP = (
@@ -26,7 +27,7 @@ def cmd_explore_list(args: argparse.Namespace) -> None:
     default_repo_root = Path(args.repo_root) if getattr(args, "repo_root", None) else None
     try:
         tasks = discover_explore_tasks(root, default_repo_root=default_repo_root)
-    except FileNotFoundError as exc:
+    except (FileNotFoundError, ValueError) as exc:  # a task file that cannot be read or parsed
         raise SystemExit(str(exc)) from exc
     for task in tasks:
         n_refs = len(task.references)
@@ -90,7 +91,7 @@ def cmd_explore_run(args: argparse.Namespace) -> None:
             tasks = [
                 load_explore_task(t, root=root, default_repo_root=default_repo_root) for t in ids
             ]
-    except FileNotFoundError as exc:
+    except (FileNotFoundError, ValueError) as exc:  # a task file that cannot be read or parsed
         raise SystemExit(str(exc)) from exc
     if not tasks:
         raise SystemExit("No tasks selected. Pass --tasks <id[,id]> or --all.")
@@ -111,14 +112,18 @@ def cmd_explore_run(args: argparse.Namespace) -> None:
     output = runner.run()
 
     out_json = Path(args.out_json) if args.out_json else output_root / "explore_runs.json"
-    save_artifact("explore_runs", output, out_json)
+    try:
+        save_artifact("explore_runs", output, out_json)
+    except OSError as exc:
+        print(f"Error: cannot write {out_json}: {exc}", file=sys.stderr)
+        raise SystemExit(1) from exc
     print(f"Saved explore run artifact: {out_json}")
 
     if args.out_md:
         from dokimi.report.exploration_report import render_explore_report
+        from dokimi.utils import write_text_or_exit
 
-        Path(args.out_md).parent.mkdir(parents=True, exist_ok=True)
-        Path(args.out_md).write_text(render_explore_report(output), encoding="utf-8")
+        write_text_or_exit(Path(args.out_md), render_explore_report(output))
         print(f"Saved exploration report: {args.out_md}")
 
     for arm, stats in output["summary"]["per_arm"].items():

dokimi/dokimi/commands/repo.py

@@ -11,6 +11,7 @@
 import re
 import sys
 from pathlib import Path
+from typing import Any
 
 
 def _normalize_whitespace(text: str, max_len: int | None = 100) -> str:
@@ -73,6 +74,14 @@ def cmd_repo_validate(args: argparse.Namespace) -> None:
         sys.exit(1)
 
 
+def _read_static(path: str) -> Any:
+    try:
+        return json.loads(Path(path).read_text(encoding="utf-8"))
+    except (OSError, ValueError, RecursionError) as exc:  # ValueError: bad JSON or UTF-8
+        print(f"✗ --static {path}: {exc}", file=sys.stderr)
+        sys.exit(1)
+
+
 def cmd_repo_scan(args: argparse.Namespace) -> None:
     """L0: check every mechanically checkable claim in the guidance against the same tree."""
     from dokimi.artifacts import save_artifact, stamp
@@ -82,10 +91,10 @@ def cmd_repo_scan(args: argparse.Namespace) -> None:
     try:
         tree = open_source(RepoSource.parse(args.source), _cache_dir(args))
         manifest = _load_manifest(tree, args, guidance_default=True)
-        static = json.loads(Path(args.static).read_text(encoding="utf-8")) if args.static else None
-    except (SourceError, ManifestError, OSError, json.JSONDecodeError) as exc:
+    except (SourceError, ManifestError, OSError, ValueError, RecursionError) as exc:
         print(f"✗ {exc}", file=sys.stderr)
         sys.exit(1)
+    static = _read_static(args.static) if args.static else None
     payload = scan_repo(tree, manifest, static)
     if args.out:
         try:
@@ -314,7 +323,7 @@ def _load_manifest(tree, args: argparse.Namespace, *, guidance_default: bool = F
         return load_manifest(tree, path)
     try:
         raw = yaml.safe_load(Path(args.manifest_file).read_text(encoding="utf-8"))
-    except (OSError, yaml.YAMLError) as exc:
+    except (OSError, ValueError, RecursionError, yaml.YAMLError) as exc:  # ValueError: not UTF-8
         raise ManifestError(args.manifest_file, [str(exc)]) from exc
     return parse_manifest(raw, path)
 

dokimi/dokimi/commands/repo_run.py

@@ -142,8 +142,13 @@ def _uses_builtin_graders(content: Path, manifest: RepoManifest) -> bool:
 
 def _pin_batch(batch_dir: Path, pin: dict[str, str]) -> None:
     path = batch_dir / "_eval" / "batch.json"
-    if path.is_file():
-        old = json.loads(path.read_text(encoding="utf-8"))
+    if path.exists():
+        try:
+            old = json.loads(path.read_text(encoding="utf-8"))
+        except (OSError, ValueError, RecursionError) as exc:  # unreadable, bad JSON or UTF-8, deep nesting
+            old = exc
+        if not isinstance(old, dict):
+            raise ValueError(f"batch {batch_dir.name} has an unreadable pin ({path}); use another --batch")
         changed = [k for k in pin if old.get(k) != pin[k]]
         if changed:
             was = ", ".join(f"{k} {old.get(k)}" for k in changed)
@@ -379,19 +384,24 @@ def _harness_bin(name: str) -> Path:
 
 def cmd_repo_report(args: argparse.Namespace) -> None:
     """Per (task, arm, model): gated success, grader failures, missing cells, cost, turns."""
-    from dokimi.repos.report import format_report, summarize
+    from dokimi.repos.report import CellFileError, format_report, summarize
 
     batch_dir = Path(args.batch_dir)
     if not batch_dir.is_dir():
         print(f"✗ not a directory: {batch_dir}", file=sys.stderr)
         sys.exit(1)
-    summary = summarize(batch_dir)
+    try:
+        summary = summarize(batch_dir)
+    except CellFileError as exc:
+        print(f"✗ {exc}", file=sys.stderr)
+        sys.exit(1)
     print(json.dumps(summary, indent=2) if args.json else format_report(summary))
 
 
 def cmd_repo_compare(args: argparse.Namespace) -> None:
     """Per (treatment, model, measure): per-task deltas against the baseline, 95% CI, sign-flip p."""
     from dokimi.repos.compare import CompareError, compare, format_compare
+    from dokimi.repos.report import CellFileError
 
     dirs = [Path(d) for d in args.batch_dirs]
     for d in dirs:
@@ -402,7 +412,7 @@ def cmd_repo_compare(args: argparse.Namespace) -> None:
     try:
         summary = compare(dirs, args.baseline, arms, args.model, tuple(args.metric or ()),
                           pass_k=args.pass_k, path_stability=args.path_stability)
-    except CompareError as exc:
+    except (CompareError, CellFileError) as exc:
         print(f"✗ {exc}", file=sys.stderr)
         sys.exit(1)
     print(json.dumps(summary, indent=2) if args.json else format_compare(summary))