fix(cli): input and usage refusals exit 2, not 1 (#94 A2)

dokimi/changelog.d/20261028_120000_napetrov_fix-cli-exit-codes-audit.md

@@ -0,0 +1,3 @@
+### Changed
+
+- **Input and usage refusals exit 2, not 1** — the exit-code table (new in `docs/quickstart.md`, section Exit codes) gives 2 to a missing, unreadable, malformed or unusable input, a flag combination a command refuses, an output file that cannot be written, and a benchmark run or preflight with no provider credential. Exit 1 is left for a verdict or a gate that ran and failed, and for a runtime or model failure. The messages are unchanged. Scripts that branch on `$? -eq 1` for these cases must also accept 2. Affected: `personas` (approve, discover), `questions`, `eval` (score, ragas, grounding, validity, panel-score), `orchestrate`, `report` (eval, generate, model-compare), `baseline`, `novelty`, `catalog` (and `catalog probe`), `answers`, `arms`, `explore`, `tasks`, `subjects`, `repo` (`validate`, `scan`, `inspect`, `budget`, `fetch`, `reach`, `report`, `compare`), `library`, `tools`, `dashboard`, `run`, and `benchmark` (run, preflight, batch). Only the validation step of a command exits 2; a failure while it runs keeps 1. Specifically: `benchmark preflight` exits 2 when a provider credential, the output dir, the question set or the doc source is invalid, even when a failed check is also reported, and 1 when a check alone fails (an unreachable doc source); `benchmark batch` with nothing to run exits 2; `repo validate`, `scan`, `inspect`, `budget`, `fetch` and `reach` exit 2 for a malformed source string, an unpinned remote, a pinned local path that is not the repository root, a manifest path that leaves the tree, a manifest or profile error, or a bad scenario selection, and 1 for a clone, fetch or pin resolution that fails (including a pinned local source that is not a git repository, a pin absent locally, and a remote pin missing from origin) and for a harness run; `arms run` and `arms matrix-run` exit 2 for a bad spec or setup input, and 1 for a run failure or a failed sampling probe; a matrix probes every cell before its first answer, so a cell refused at the probe makes no answer call (probe calls are still model calls and can fail, exit 1); `catalog trigger-set` exits 2 for a duplicate question id, a question without text or an `@field` rule that labels none of its source's questions (refused before any `--rewrite` call), and 1 for a `--strict` refusal (a lint gate, as on main) and when a `--rewrite` returns nothing; a missing RAGAS extra stays 1. `repo run` and `repo tasks check` are not converted: their input refusals still exit 1. `tools preference` still exits 2 when every call failed or `--max-cost-usd` skipped calls. A `--static` file that cannot be read or is not JSON exits 2 in `repo scan`; a schema-invalid document exits 1.

dokimi/docs/benchmarking-and-comparison.md

@@ -796,8 +796,8 @@ stderr as not compared, and `--strict` exits 1 on it as well, since it cannot
 vouch for that role. A malformed record in either file (no `question_id`) exits 2,
 naming the record on stderr, so it never reads as a failed `--strict` check.
 `baseline compare` also exits 2 on an eval file, saved baseline or `manifest.json`
-that is not valid JSON, not a JSON object, or cannot be read. `baseline save` exits 1
-on a bad eval file or `manifest.json`, and `baseline list` exits 1 on a bad
+that is not valid JSON, not a JSON object, or cannot be read. `baseline save` exits 2
+on a bad eval file or `manifest.json`, and `baseline list` exits 2 on a bad
 `manifest.json`. A `manifest.json` whose `baselines` is not a list, or whose entries
 lack a string `name` and `path` or have a `summary` that is not an object, is rejected
 the same way.

dokimi/docs/external-content.md

@@ -186,7 +186,7 @@ WARNING questions root C:\Users\Public\content-repo\questions has no golden set
 ```
 
 `benchmark run` logs a warning and continues with generated questions only.
-`benchmark preflight` treats a missing golden set as an error and exits 1.
+`benchmark preflight` refuses a missing golden set and exits 2.
 If you are comparing arms, that is usually not what you want; check the coverage matrix first.
 
 `benchmark preflight` resolves the whole plan without calling a model, and
@@ -213,7 +213,7 @@ Resolved benchmark plan
   golden questions: C:\Users\Public\content-repo\questions\demo_golden.json (2 items)
 ```
 
-(It exited 1 here because no API keys were set: `✗ missing OPENAI_API_KEY for
+(It exited 2 here because no API keys were set: `✗ missing OPENAI_API_KEY for
 provider openai`. Key checks are part of the pre-flight.)
 
 For a direct N-arm comparison, hand `arms run` the question file itself:

dokimi/docs/quickstart.md

@@ -241,6 +241,22 @@ so results from before are not comparable with older baselines; re-baseline —
 Costs scale roughly linearly with question count. Token counts are recorded
 per answer in `answers/<product>.json`.
 
+## Exit codes
+
+`dokimi` commands use three codes, so a script can tell a bad invocation from a
+failed check.
+
+| Code | Meaning | Examples |
+|------|---------|----------|
+| 0 | The command ran and, if it checks something, the check passed. | |
+| 1 | A verdict or a gate that ran and failed, or a runtime failure (a model call, a run, a clone or fetch). | `baseline compare --strict` over a changed model, `benchmark preflight` with a failed check alone, `repo reach` with a harness binary that cannot run, `catalog trigger-set --strict` over a lint finding |
+| 2 | A bad invocation: a missing, unreadable or malformed input file, a flag combination the command refuses, an output file that cannot be written, a missing provider credential, a source string that is malformed, unpinned or not a repository root. | `baseline save` on a bad eval file, `benchmark preflight` with no golden set, `repo scan --manifest ../x.yaml` |
+
+A command that validates its input and then runs exits 2 for the first and 1
+for the second. Not every command follows this yet: `repo run` and
+`repo tasks check` still exit 1 for their input refusals, and `tools preference`
+exits 2 when every call failed or `--max-cost-usd` skipped calls.
+
 ## Troubleshooting
 
 | Symptom | Fix |

dokimi/docs/repo-evaluation-contract.md

@@ -1161,8 +1161,9 @@ scanned once, as the file it names.
 The report lists `warnings` for guidance the manifest names but the tree lacks:
 a glob with no match, a missing `also_scan` file, no guidance at all, and a
 guidance file that cannot be read (a link that dangles or leaves the tree, a
-binary file, a file over 2 MB), named with the reason. An invalid `--static`
-document is not merged, and the command exits 1. `--fail-on-fail` also exits 1
+binary file, a file over 2 MB), named with the reason. A `--static`
+file that cannot be read or is not JSON exits 2; a document that fails the
+`repo_static.v1` schema is not merged, and the command exits 1. `--fail-on-fail` also exits 1
 when any claim fails, an `also_scan` file is missing, a guidance file cannot be
 read, a Bazel build input (`BUILD`, `.bzl`, `.bazelrc`, ...) cannot be read, or no guidance file matched
 (`incomplete`). A glob that matches nothing

dokimi/dokimi/commands/_output.py

@@ -9,12 +9,14 @@
 import sys
 from pathlib import Path
 
+from dokimi.exit_codes import EXIT_USAGE
+
 
 @contextlib.contextmanager
 def exit_on_unwritable(path: Path):
-    """Exit 1 naming ``path`` when an output write inside the block fails."""
+    """Exit 2 naming ``path`` when an output write inside the block fails (exit-code table in docs/quickstart.md)."""
     try:
         yield
     except OSError as exc:
         print(f"Error: cannot write {path}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)

dokimi/dokimi/commands/answers.py

@@ -19,6 +19,7 @@
     DEFAULT_MAX_TOKENS_PER_QUESTION,
     DEFAULT_RETRIEVAL_TOP_K,
 )
+from dokimi.exit_codes import EXIT_USAGE
 
 
 def _load_runtime_config() -> dict:
@@ -33,18 +34,18 @@ def _load_runtime_config() -> dict:
         loaded = yaml.safe_load(config_path.read_text(encoding="utf-8"))
     except yaml.YAMLError as exc:
         print(f"Error: invalid YAML in {config_path}: {exc}", file=sys.stderr)
-        raise SystemExit(1) from exc
+        raise SystemExit(EXIT_USAGE) from exc
     except (OSError, ValueError) as exc:  # ValueError covers bad UTF-8
         print(f"Error: cannot read {config_path}: {exc}", file=sys.stderr)
-        raise SystemExit(1) from exc
+        raise SystemExit(EXIT_USAGE) from exc
     if loaded is not None and not isinstance(loaded, dict):
         print(f"Error: expected a mapping in {config_path}", file=sys.stderr)
-        raise SystemExit(1)
+        raise SystemExit(EXIT_USAGE)
     config = loaded or {}
     for key in ("products", "retrieval"):
         if config.get(key) is not None and not isinstance(config[key], dict):
             print(f"Error: '{key}' in {config_path} must be a mapping", file=sys.stderr)
-            raise SystemExit(1)
+            raise SystemExit(EXIT_USAGE)
     return config
 
 
@@ -55,10 +56,10 @@ def _load_questions(path: str) -> list:
         return load_questions(path).questions
     except ValueError as exc:  # ArtifactValidationError: names the record and the fix
         print(f"Error: {exc}", file=sys.stderr)
-        raise SystemExit(1) from exc
+        raise SystemExit(EXIT_USAGE) from exc
     except OSError as exc:
         print(f"Error: cannot read {path}: {exc.strerror or exc}", file=sys.stderr)
-        raise SystemExit(1) from exc
+        raise SystemExit(EXIT_USAGE) from exc
 
 
 def _output_path(args: argparse.Namespace) -> Path:
@@ -104,7 +105,7 @@ def cmd_answers_generate(args: argparse.Namespace) -> None:
             )
         except ValueError as exc:
             print(f"Error: {exc}", file=sys.stderr)
-            raise SystemExit(1) from exc
+            raise SystemExit(EXIT_USAGE) from exc
 
     print(f"Generating answers for {args.product} (library_id={library_id})")
     print(f"Using model: {args.provider}/{args.model}")

dokimi/dokimi/commands/arms.py

@@ -11,6 +11,7 @@
 from pathlib import Path
 from typing import TYPE_CHECKING
 
+from dokimi.exit_codes import EXIT_FAILURE, EXIT_USAGE
 from dokimi.utils import question_id
 from dokimi.eval.judge import JUDGE_PROVIDERS
 from dokimi.defaults import (
@@ -53,21 +54,32 @@ def judge_panel_roles(value: str) -> str:
 
 def cmd_arms_run(args: argparse.Namespace) -> None:
     """Run an N-arm treatment comparison and optionally judge it."""
-    from dokimi.eval.arms_run import LibraryIdError, run_arms
+    from dokimi.eval.arms_run import LibraryIdError, ProbeError, prepare_arms, run_prepared
 
+    # Setup (inputs, arms, plugins, sampling, judge) exits 2; a failed sampling probe is a run failure (1).
     try:
         specs = _parse_arm_specs(args.arms)
         questions = _load_questions(args.questions)
         config = _with_named_cell(run_config_from_args(args), args.matrix_cells, args.matrix_cell)
-        output = run_arms(config, specs, questions)
+        prepared = prepare_arms(config, specs, questions)
     except EmptyQuestionError:
         raise  # a bad input record, not an arm setup error: cli.main reports it (exit 2)
+    except ProbeError as exc:
+        print(f"Error running arms: {exc}", file=sys.stderr)
+        sys.exit(EXIT_FAILURE)
     except LibraryIdError as exc:
         print(f"Error: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
     except (ValueError, OSError, RecursionError) as exc:
         print(f"Error building arms/plugins: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
+    try:
+        output = run_prepared(prepared, config, questions)
+    except EmptyQuestionError:
+        raise  # a bad input record, not a run error: cli.main reports it (exit 2)
+    except (ValueError, OSError, RecursionError) as exc:
+        print(f"Error running arms: {exc}", file=sys.stderr)
+        sys.exit(EXIT_FAILURE)
 
     out_json = Path(args.out_json) if args.out_json else Path(f"results/arms/{args.product}.json")
     out_md = Path(args.out_md) if args.out_md else Path(f"results/arms/{args.product}.md")
@@ -95,6 +107,29 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None:
     )
 
 
+def _load_matrix_inputs(config: ArmsRunConfig, arms: str, questions: str, matrix_cells: str):
+    """The arms, questions and cells of a matrix run, checked before any cell is set up. A refused
+    input exits 2; a failed sampling probe, a real call, exits 1."""
+    from dokimi.eval import arms_run
+    from dokimi.eval.cells import load_matrix_cells
+
+    try:
+        specs = _parse_arm_specs(arms)
+        loaded_questions = _load_questions(questions)
+        cells = load_matrix_cells(matrix_cells)
+        _validate_unique_cell_filenames(cells)
+        arms_run.check_matrix(config, cells)
+    except EmptyQuestionError:
+        raise  # a bad input record, not a cell error: cli.main reports it (exit 2)
+    except arms_run.ProbeError as exc:
+        print(f"Error loading matrix run inputs: {exc}", file=sys.stderr)
+        sys.exit(EXIT_FAILURE)
+    except (ValueError, OSError, RecursionError) as exc:
+        print(f"Error loading matrix run inputs: {exc}", file=sys.stderr)
+        sys.exit(EXIT_USAGE)
+    return specs, loaded_questions, cells
+
+
 def run_matrix(config: ArmsRunConfig, *, arms: str, questions: str, matrix_cells: str,
                out_dir: Path, out_json: Path | None = None, out_md: Path | None = None,
                plugin_deltas: bool = True) -> None:
@@ -105,32 +140,24 @@ def run_matrix(config: ArmsRunConfig, *, arms: str, questions: str, matrix_cells
     ``.md`` suffix.
     """
     from dokimi.eval import arms_run
-    from dokimi.eval.cells import load_matrix_cells
     from dokimi.eval.matrix_rollup import build_matrix_rollup, strip_internal_payloads
     from dokimi.report.matrix_rollup_report import render_matrix_rollup_report
     from dokimi.report.plugin_delta_report import render_plugin_delta_report
 
-    try:
-        specs = _parse_arm_specs(arms)
-        loaded_questions = _load_questions(questions)
-        cells = load_matrix_cells(matrix_cells)
-        _validate_unique_cell_filenames(cells)
-        arms_run.check_matrix(config, cells)
-    except (ValueError, OSError, RecursionError) as exc:
-        print(f"Error loading matrix run inputs: {exc}", file=sys.stderr)
-        sys.exit(1)
+    specs, loaded_questions, cells = _load_matrix_inputs(config, arms, questions, matrix_cells)
 
+    # The cells dir is made only when a cell finishes: a cell refused in setup writes nothing.
     cells_dir = out_dir / "cells"
-    try:
-        cells_dir.mkdir(parents=True, exist_ok=True)
-    except OSError as exc:
-        print(f"Error: cannot write {cells_dir}: {exc}", file=sys.stderr)
-        sys.exit(1)
     cell_artifacts = []
 
     def save_cell(cell, output: dict) -> None:
         safe_name = _safe_name(cell.name)
         cell_json = cells_dir / f"{safe_name}.json"
+        try:
+            cells_dir.mkdir(parents=True, exist_ok=True)
+        except OSError as exc:
+            print(f"Error: cannot write {cells_dir}: {exc}", file=sys.stderr)
+            sys.exit(EXIT_USAGE)
         _save_arms_output(output, cell_json, cells_dir / f"{safe_name}.md")
         cell_artifacts.append((cell.name, cell_json, output))
 
@@ -140,10 +167,11 @@ def save_cell(cell, output: dict) -> None:
         raise  # a bad input record, not a cell error: cli.main reports it (exit 2)
     except arms_run.LibraryIdError as exc:
         print(f"Error: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
     except arms_run.MatrixCellError as exc:
-        print(f"Error running matrix cell {exc.cell!r}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        verb = "building" if exc.usage else "running"
+        print(f"Error {verb} matrix cell {exc.cell!r}: {exc}", file=sys.stderr)
+        sys.exit(EXIT_USAGE if exc.usage else EXIT_FAILURE)
 
     rollup = build_matrix_rollup(
         library_name=config.product,
@@ -185,7 +213,7 @@ def cmd_arms_plugin_delta(args: argparse.Namespace) -> None:
         output = compare_plugin_runs(baseline, plugin)
     except (PluginDeltaError, ValueError, OSError, RecursionError) as exc:
         print(f"Error computing plugin delta: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
 
     out_json = Path(args.out_json) if args.out_json else Path("results/arms/plugin_delta.json")
     _save_artifact_or_exit("plugin_delta", output, out_json)
@@ -488,7 +516,7 @@ def _write_report(path: Path, text: str) -> None:
         path.write_text(text, encoding="utf-8")
     except OSError as exc:
         print(f"Error: cannot write {path}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
 
 
 def _save_artifact_or_exit(kind: str, payload: dict, path: Path) -> None:
@@ -499,7 +527,7 @@ def _save_artifact_or_exit(kind: str, payload: dict, path: Path) -> None:
         save_artifact(kind, payload, path)
     except OSError as exc:
         print(f"Error: cannot write {path}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
 
 
 def _save_arms_output(output: dict, out_json: Path, out_md: Path) -> None:
@@ -510,7 +538,7 @@ def _save_arms_output(output: dict, out_json: Path, out_md: Path) -> None:
         ArmRunner.save(output, out_json)
     except OSError as exc:
         print(f"Error: cannot write {out_json}: {exc}", file=sys.stderr)
-        sys.exit(1)
+        sys.exit(EXIT_USAGE)
     print(f"OK Saved arms comparison: {out_json}")
     _write_report(out_md, render_arms_report(output))
     print(f"OK Saved arms report:     {out_md}")

dokimi/dokimi/commands/baseline.py

@@ -12,6 +12,7 @@
 from typing import Any, Dict, List, Optional, Tuple
 
 from dokimi.utils import normalize_model_ref
+from dokimi.exit_codes import EXIT_USAGE
 
 BASELINES_DIR = Path("baselines/eval")
 
@@ -37,7 +38,7 @@ def _read_json_object(path: Path, code: int) -> Dict[str, Any]:
     return data
 
 
-def _baseline_manifest(code: int = 1) -> Dict[str, Any]:
+def _baseline_manifest(code: int = EXIT_USAGE) -> Dict[str, Any]:
     """Load or create the baseline manifest; a corrupt one exits ``code``."""
     manifest_path = BASELINES_DIR / "manifest.json"
     if not manifest_path.exists():
@@ -140,7 +141,7 @@ def cmd_baseline_save(args: argparse.Namespace) -> None:
 
     eval_path = Path(args.from_eval)
     manifest = _baseline_manifest()  # first: a corrupt manifest fails fast and leaves no orphan file
-    eval_data = _read_json_object(eval_path, 1)
+    eval_data = _read_json_object(eval_path, EXIT_USAGE)
     product = args.product or eval_data.get("product", "unknown")
     timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%f")
     name = args.name or f"{product}-{timestamp}"
@@ -151,13 +152,13 @@ def cmd_baseline_save(args: argparse.Namespace) -> None:
         summary = _compute_summary(eval_data)
     except RecordError as exc:
         print(f"❌ {eval_path}: {exc}", file=sys.stderr)
-        raise SystemExit(1) from exc
+        raise SystemExit(EXIT_USAGE) from exc
 
     BASELINES_DIR.mkdir(parents=True, exist_ok=True)
     dest = BASELINES_DIR / f"{name}.json"
     if dest.exists():
         print(f"❌ Baseline already exists: {dest} (choose a different --name)", file=sys.stderr)
-        raise SystemExit(1)
+        raise SystemExit(EXIT_USAGE)
     dest.write_text(json.dumps(eval_data, indent=2), encoding="utf-8")
 
     manifest["baselines"].append({
@@ -205,7 +206,7 @@ def _pick_entry(args: argparse.Namespace, baselines: list, eval_data: dict) -> d
         entry = next((b for b in baselines if b["name"] == args.baseline), None)
         if not entry:
             print(f"❌ Baseline '{args.baseline}' not found.")
-            raise SystemExit(1)
+            raise SystemExit(EXIT_USAGE)
     else:
         product = args.product or eval_data.get("product")
         candidates = [b for b in baselines if b.get("product") == product] if product else baselines
@@ -260,7 +261,7 @@ def cmd_baseline_compare(args: argparse.Namespace) -> None:
     if not baselines:
         print("No baselines found. Save one first:")
         print("  dokimi baseline save --from-eval eval/oneTBB.json --product oneTBB")
-        raise SystemExit(1)
+        raise SystemExit(EXIT_USAGE)
 
     entry = _pick_entry(args, baselines, eval_data)
     baseline_data = _read_json_object(Path(entry["path"]), 2)