diff --git a/dokimi/changelog.d/20261028_120000_napetrov_fix-cli-exit-codes-audit.md b/dokimi/changelog.d/20261028_120000_napetrov_fix-cli-exit-codes-audit.md new file mode 100644 index 000000000..6152a6485 --- /dev/null +++ b/dokimi/changelog.d/20261028_120000_napetrov_fix-cli-exit-codes-audit.md @@ -0,0 +1,3 @@ +### Changed + +- **Input and usage refusals exit 2, not 1** — the exit-code table (new in `docs/quickstart.md`, section Exit codes) gives 2 to a missing, unreadable, malformed or unusable input, a flag combination a command refuses, an output file that cannot be written, and a benchmark run or preflight with no provider credential. Exit 1 is left for a verdict or a gate that ran and failed, and for a runtime or model failure. The messages are unchanged. Scripts that branch on `$? -eq 1` for these cases must also accept 2. Affected: `personas` (approve, discover), `questions`, `eval` (score, ragas, grounding, validity, panel-score), `orchestrate`, `report` (eval, generate, model-compare), `baseline`, `novelty`, `catalog` (and `catalog probe`), `answers`, `arms`, `explore`, `tasks`, `subjects`, `repo` (`validate`, `scan`, `inspect`, `budget`, `fetch`, `reach`, `report`, `compare`), `library`, `tools`, `dashboard`, `run`, and `benchmark` (run, preflight, batch). Only the validation step of a command exits 2; a failure while it runs keeps 1. Specifically: `benchmark preflight` exits 2 when a provider credential, the output dir, the question set or the doc source is invalid, even when a failed check is also reported, and 1 when a check alone fails (an unreachable doc source); `benchmark batch` with nothing to run exits 2; `repo validate`, `scan`, `inspect`, `budget`, `fetch` and `reach` exit 2 for a malformed source string, an unpinned remote, a pinned local path that is not the repository root, a manifest path that leaves the tree, a manifest or profile error, or a bad scenario selection, and 1 for a clone, fetch or pin resolution that fails (including a pinned local source that is not a git repository, a pin absent locally, and a remote pin missing from origin) and for a harness run; `arms run` and `arms matrix-run` exit 2 for a bad spec or setup input, and 1 for a run failure or a failed sampling probe; a matrix probes every cell before its first answer, so a cell refused at the probe makes no answer call (probe calls are still model calls and can fail, exit 1); `catalog trigger-set` exits 2 for a duplicate question id, a question without text or an `@field` rule that labels none of its source's questions (refused before any `--rewrite` call), and 1 for a `--strict` refusal (a lint gate, as on main) and when a `--rewrite` returns nothing; a missing RAGAS extra stays 1. `repo run` and `repo tasks check` are not converted: their input refusals still exit 1. `tools preference` still exits 2 when every call failed or `--max-cost-usd` skipped calls. A `--static` file that cannot be read or is not JSON exits 2 in `repo scan`; a schema-invalid document exits 1. diff --git a/dokimi/docs/benchmarking-and-comparison.md b/dokimi/docs/benchmarking-and-comparison.md index 4d771b83e..8ee19935a 100644 --- a/dokimi/docs/benchmarking-and-comparison.md +++ b/dokimi/docs/benchmarking-and-comparison.md @@ -796,8 +796,8 @@ stderr as not compared, and `--strict` exits 1 on it as well, since it cannot vouch for that role. A malformed record in either file (no `question_id`) exits 2, naming the record on stderr, so it never reads as a failed `--strict` check. `baseline compare` also exits 2 on an eval file, saved baseline or `manifest.json` -that is not valid JSON, not a JSON object, or cannot be read. `baseline save` exits 1 -on a bad eval file or `manifest.json`, and `baseline list` exits 1 on a bad +that is not valid JSON, not a JSON object, or cannot be read. `baseline save` exits 2 +on a bad eval file or `manifest.json`, and `baseline list` exits 2 on a bad `manifest.json`. A `manifest.json` whose `baselines` is not a list, or whose entries lack a string `name` and `path` or have a `summary` that is not an object, is rejected the same way. diff --git a/dokimi/docs/external-content.md b/dokimi/docs/external-content.md index ba7929a61..370a67e36 100644 --- a/dokimi/docs/external-content.md +++ b/dokimi/docs/external-content.md @@ -186,7 +186,7 @@ WARNING questions root C:\Users\Public\content-repo\questions has no golden set ``` `benchmark run` logs a warning and continues with generated questions only. -`benchmark preflight` treats a missing golden set as an error and exits 1. +`benchmark preflight` refuses a missing golden set and exits 2. If you are comparing arms, that is usually not what you want; check the coverage matrix first. `benchmark preflight` resolves the whole plan without calling a model, and @@ -213,7 +213,7 @@ Resolved benchmark plan golden questions: C:\Users\Public\content-repo\questions\demo_golden.json (2 items) ``` -(It exited 1 here because no API keys were set: `✗ missing OPENAI_API_KEY for +(It exited 2 here because no API keys were set: `✗ missing OPENAI_API_KEY for provider openai`. Key checks are part of the pre-flight.) For a direct N-arm comparison, hand `arms run` the question file itself: diff --git a/dokimi/docs/quickstart.md b/dokimi/docs/quickstart.md index f95bc0317..7fc46655b 100644 --- a/dokimi/docs/quickstart.md +++ b/dokimi/docs/quickstart.md @@ -241,6 +241,22 @@ so results from before are not comparable with older baselines; re-baseline — Costs scale roughly linearly with question count. Token counts are recorded per answer in `answers/.json`. +## Exit codes + +`dokimi` commands use three codes, so a script can tell a bad invocation from a +failed check. + +| Code | Meaning | Examples | +|------|---------|----------| +| 0 | The command ran and, if it checks something, the check passed. | | +| 1 | A verdict or a gate that ran and failed, or a runtime failure (a model call, a run, a clone or fetch). | `baseline compare --strict` over a changed model, `benchmark preflight` with a failed check alone, `repo reach` with a harness binary that cannot run, `catalog trigger-set --strict` over a lint finding | +| 2 | A bad invocation: a missing, unreadable or malformed input file, a flag combination the command refuses, an output file that cannot be written, a missing provider credential, a source string that is malformed, unpinned or not a repository root. | `baseline save` on a bad eval file, `benchmark preflight` with no golden set, `repo scan --manifest ../x.yaml` | + +A command that validates its input and then runs exits 2 for the first and 1 +for the second. Not every command follows this yet: `repo run` and +`repo tasks check` still exit 1 for their input refusals, and `tools preference` +exits 2 when every call failed or `--max-cost-usd` skipped calls. + ## Troubleshooting | Symptom | Fix | diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index a61f27da3..dae1183cc 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -1162,8 +1162,9 @@ scanned once, as the file it names. The report lists `warnings` for guidance the manifest names but the tree lacks: a glob with no match, a missing `also_scan` file, no guidance at all, and a guidance file that cannot be read (a link that dangles or leaves the tree, a -binary file, a file over 2 MB), named with the reason. An invalid `--static` -document is not merged, and the command exits 1. `--fail-on-fail` also exits 1 +binary file, a file over 2 MB), named with the reason. A `--static` +file that cannot be read or is not JSON exits 2; a document that fails the +`repo_static.v1` schema is not merged, and the command exits 1. `--fail-on-fail` also exits 1 when any claim fails, an `also_scan` file is missing, a guidance file cannot be read, a Bazel build input (`BUILD`, `.bzl`, `.bazelrc`, ...) cannot be read, or no guidance file matched (`incomplete`). A glob that matches nothing diff --git a/dokimi/dokimi/commands/_output.py b/dokimi/dokimi/commands/_output.py index c2c6eed40..1d72ba552 100644 --- a/dokimi/dokimi/commands/_output.py +++ b/dokimi/dokimi/commands/_output.py @@ -9,12 +9,14 @@ import sys from pathlib import Path +from dokimi.exit_codes import EXIT_USAGE + @contextlib.contextmanager def exit_on_unwritable(path: Path): - """Exit 1 naming ``path`` when an output write inside the block fails.""" + """Exit 2 naming ``path`` when an output write inside the block fails (exit-code table in docs/quickstart.md).""" try: yield except OSError as exc: print(f"Error: cannot write {path}: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) diff --git a/dokimi/dokimi/commands/answers.py b/dokimi/dokimi/commands/answers.py index 4b08abe6a..8a998ee5f 100644 --- a/dokimi/dokimi/commands/answers.py +++ b/dokimi/dokimi/commands/answers.py @@ -19,6 +19,7 @@ DEFAULT_MAX_TOKENS_PER_QUESTION, DEFAULT_RETRIEVAL_TOP_K, ) +from dokimi.exit_codes import EXIT_USAGE def _load_runtime_config() -> dict: @@ -33,18 +34,18 @@ def _load_runtime_config() -> dict: loaded = yaml.safe_load(config_path.read_text(encoding="utf-8")) except yaml.YAMLError as exc: print(f"Error: invalid YAML in {config_path}: {exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc except (OSError, ValueError) as exc: # ValueError covers bad UTF-8 print(f"Error: cannot read {config_path}: {exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc if loaded is not None and not isinstance(loaded, dict): print(f"Error: expected a mapping in {config_path}", file=sys.stderr) - raise SystemExit(1) + raise SystemExit(EXIT_USAGE) config = loaded or {} for key in ("products", "retrieval"): if config.get(key) is not None and not isinstance(config[key], dict): print(f"Error: '{key}' in {config_path} must be a mapping", file=sys.stderr) - raise SystemExit(1) + raise SystemExit(EXIT_USAGE) return config @@ -55,10 +56,10 @@ def _load_questions(path: str) -> list: return load_questions(path).questions except ValueError as exc: # ArtifactValidationError: names the record and the fix print(f"Error: {exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc except OSError as exc: print(f"Error: cannot read {path}: {exc.strerror or exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc def _output_path(args: argparse.Namespace) -> Path: @@ -104,7 +105,7 @@ def cmd_answers_generate(args: argparse.Namespace) -> None: ) except ValueError as exc: print(f"Error: {exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc print(f"Generating answers for {args.product} (library_id={library_id})") print(f"Using model: {args.provider}/{args.model}") diff --git a/dokimi/dokimi/commands/arms.py b/dokimi/dokimi/commands/arms.py index b61ea4c99..0fbca01cb 100644 --- a/dokimi/dokimi/commands/arms.py +++ b/dokimi/dokimi/commands/arms.py @@ -7,10 +7,12 @@ import argparse from collections import Counter +import os import sys from pathlib import Path from typing import TYPE_CHECKING +from dokimi.exit_codes import EXIT_FAILURE, EXIT_USAGE from dokimi.utils import question_id from dokimi.eval.judge import JUDGE_PROVIDERS from dokimi.defaults import ( @@ -53,21 +55,32 @@ def judge_panel_roles(value: str) -> str: def cmd_arms_run(args: argparse.Namespace) -> None: """Run an N-arm treatment comparison and optionally judge it.""" - from dokimi.eval.arms_run import LibraryIdError, run_arms + from dokimi.eval.arms_run import LibraryIdError, ProbeError, prepare_arms, run_prepared + # Setup (inputs, arms, plugins, sampling, judge) exits 2; a failed sampling probe is a run failure (1). try: specs = _parse_arm_specs(args.arms) questions = _load_questions(args.questions) config = _with_named_cell(run_config_from_args(args), args.matrix_cells, args.matrix_cell) - output = run_arms(config, specs, questions) + prepared = prepare_arms(config, specs, questions) except EmptyQuestionError: raise # a bad input record, not an arm setup error: cli.main reports it (exit 2) + except ProbeError as exc: + print(f"Error running arms: {exc}", file=sys.stderr) + sys.exit(EXIT_FAILURE) except LibraryIdError as exc: print(f"Error: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) except (ValueError, OSError, RecursionError) as exc: print(f"Error building arms/plugins: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) + try: + output = run_prepared(prepared, config, questions) + except EmptyQuestionError: + raise # a bad input record, not a run error: cli.main reports it (exit 2) + except (ValueError, OSError, RecursionError) as exc: + print(f"Error running arms: {exc}", file=sys.stderr) + sys.exit(EXIT_FAILURE) out_json = Path(args.out_json) if args.out_json else Path(f"results/arms/{args.product}.json") out_md = Path(args.out_md) if args.out_md else Path(f"results/arms/{args.product}.md") @@ -95,6 +108,39 @@ def cmd_arms_matrix_run(args: argparse.Namespace) -> None: ) +def _load_matrix_inputs(config: ArmsRunConfig, arms: str, questions: str, matrix_cells: str): + """The arms, questions and cells of a matrix run, checked before any cell is set up. A refused + input exits 2; a failed sampling probe, a real call, exits 1.""" + from dokimi.eval import arms_run + from dokimi.eval.cells import load_matrix_cells + + try: + specs = _parse_arm_specs(arms) + loaded_questions = _load_questions(questions) + cells = load_matrix_cells(matrix_cells) + _validate_unique_cell_filenames(cells) + arms_run.check_matrix(config, cells) + except EmptyQuestionError: + raise # a bad input record, not a cell error: cli.main reports it (exit 2) + except arms_run.ProbeError as exc: + print(f"Error loading matrix run inputs: {exc}", file=sys.stderr) + sys.exit(EXIT_FAILURE) + except (ValueError, OSError, RecursionError) as exc: + print(f"Error loading matrix run inputs: {exc}", file=sys.stderr) + sys.exit(EXIT_USAGE) + return specs, loaded_questions, cells + + +def _check_cells_dir(cells_dir: Path) -> None: + """Exit 2 when *cells_dir* cannot be made: the nearest existing ancestor must be a directory + the process can write to. Creates nothing.""" + # A dangling symlink exists as a link but not as a target: it is the nearest entry, and it is refused. + existing = next(p for p in (cells_dir, *cells_dir.parents) if p.exists() or p.is_symlink()) + if not existing.is_dir() or not os.access(existing, os.W_OK | os.X_OK): + print(f"Error: cannot write {cells_dir}: {existing} is not a writable directory", file=sys.stderr) + sys.exit(EXIT_USAGE) + + def run_matrix(config: ArmsRunConfig, *, arms: str, questions: str, matrix_cells: str, out_dir: Path, out_json: Path | None = None, out_md: Path | None = None, plugin_deltas: bool = True) -> None: @@ -105,32 +151,26 @@ def run_matrix(config: ArmsRunConfig, *, arms: str, questions: str, matrix_cells ``.md`` suffix. """ from dokimi.eval import arms_run - from dokimi.eval.cells import load_matrix_cells from dokimi.eval.matrix_rollup import build_matrix_rollup, strip_internal_payloads from dokimi.report.matrix_rollup_report import render_matrix_rollup_report from dokimi.report.plugin_delta_report import render_plugin_delta_report - try: - specs = _parse_arm_specs(arms) - loaded_questions = _load_questions(questions) - cells = load_matrix_cells(matrix_cells) - _validate_unique_cell_filenames(cells) - arms_run.check_matrix(config, cells) - except (ValueError, OSError, RecursionError) as exc: - print(f"Error loading matrix run inputs: {exc}", file=sys.stderr) - sys.exit(1) + specs, loaded_questions, cells = _load_matrix_inputs(config, arms, questions, matrix_cells) + # The cells dir is made only when a cell finishes, so a cell refused in setup writes nothing. + # An uncreatable one is refused here, before any answer is paid for and then lost on exit. cells_dir = out_dir / "cells" - try: - cells_dir.mkdir(parents=True, exist_ok=True) - except OSError as exc: - print(f"Error: cannot write {cells_dir}: {exc}", file=sys.stderr) - sys.exit(1) + _check_cells_dir(cells_dir) cell_artifacts = [] def save_cell(cell, output: dict) -> None: safe_name = _safe_name(cell.name) cell_json = cells_dir / f"{safe_name}.json" + try: + cells_dir.mkdir(parents=True, exist_ok=True) + except OSError as exc: + print(f"Error: cannot write {cells_dir}: {exc}", file=sys.stderr) + sys.exit(EXIT_USAGE) _save_arms_output(output, cell_json, cells_dir / f"{safe_name}.md") cell_artifacts.append((cell.name, cell_json, output)) @@ -140,10 +180,11 @@ def save_cell(cell, output: dict) -> None: raise # a bad input record, not a cell error: cli.main reports it (exit 2) except arms_run.LibraryIdError as exc: print(f"Error: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) except arms_run.MatrixCellError as exc: - print(f"Error running matrix cell {exc.cell!r}: {exc}", file=sys.stderr) - sys.exit(1) + verb = "building" if exc.usage else "running" + print(f"Error {verb} matrix cell {exc.cell!r}: {exc}", file=sys.stderr) + sys.exit(EXIT_USAGE if exc.usage else EXIT_FAILURE) rollup = build_matrix_rollup( library_name=config.product, @@ -185,7 +226,7 @@ def cmd_arms_plugin_delta(args: argparse.Namespace) -> None: output = compare_plugin_runs(baseline, plugin) except (PluginDeltaError, ValueError, OSError, RecursionError) as exc: print(f"Error computing plugin delta: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) out_json = Path(args.out_json) if args.out_json else Path("results/arms/plugin_delta.json") _save_artifact_or_exit("plugin_delta", output, out_json) @@ -488,7 +529,7 @@ def _write_report(path: Path, text: str) -> None: path.write_text(text, encoding="utf-8") except OSError as exc: print(f"Error: cannot write {path}: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) def _save_artifact_or_exit(kind: str, payload: dict, path: Path) -> None: @@ -499,7 +540,7 @@ def _save_artifact_or_exit(kind: str, payload: dict, path: Path) -> None: save_artifact(kind, payload, path) except OSError as exc: print(f"Error: cannot write {path}: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) def _save_arms_output(output: dict, out_json: Path, out_md: Path) -> None: @@ -510,7 +551,7 @@ def _save_arms_output(output: dict, out_json: Path, out_md: Path) -> None: ArmRunner.save(output, out_json) except OSError as exc: print(f"Error: cannot write {out_json}: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) print(f"OK Saved arms comparison: {out_json}") _write_report(out_md, render_arms_report(output)) print(f"OK Saved arms report: {out_md}") diff --git a/dokimi/dokimi/commands/baseline.py b/dokimi/dokimi/commands/baseline.py index 20769c53d..c36fb7ca0 100644 --- a/dokimi/dokimi/commands/baseline.py +++ b/dokimi/dokimi/commands/baseline.py @@ -12,6 +12,7 @@ from typing import Any, Dict, List, Optional, Tuple from dokimi.utils import normalize_model_ref +from dokimi.exit_codes import EXIT_USAGE BASELINES_DIR = Path("baselines/eval") @@ -37,7 +38,7 @@ def _read_json_object(path: Path, code: int) -> Dict[str, Any]: return data -def _baseline_manifest(code: int = 1) -> Dict[str, Any]: +def _baseline_manifest(code: int = EXIT_USAGE) -> Dict[str, Any]: """Load or create the baseline manifest; a corrupt one exits ``code``.""" manifest_path = BASELINES_DIR / "manifest.json" if not manifest_path.exists(): @@ -140,7 +141,7 @@ def cmd_baseline_save(args: argparse.Namespace) -> None: eval_path = Path(args.from_eval) manifest = _baseline_manifest() # first: a corrupt manifest fails fast and leaves no orphan file - eval_data = _read_json_object(eval_path, 1) + eval_data = _read_json_object(eval_path, EXIT_USAGE) product = args.product or eval_data.get("product", "unknown") timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%f") name = args.name or f"{product}-{timestamp}" @@ -151,13 +152,13 @@ def cmd_baseline_save(args: argparse.Namespace) -> None: summary = _compute_summary(eval_data) except RecordError as exc: print(f"❌ {eval_path}: {exc}", file=sys.stderr) - raise SystemExit(1) from exc + raise SystemExit(EXIT_USAGE) from exc BASELINES_DIR.mkdir(parents=True, exist_ok=True) dest = BASELINES_DIR / f"{name}.json" if dest.exists(): print(f"❌ Baseline already exists: {dest} (choose a different --name)", file=sys.stderr) - raise SystemExit(1) + raise SystemExit(EXIT_USAGE) dest.write_text(json.dumps(eval_data, indent=2), encoding="utf-8") manifest["baselines"].append({ @@ -205,7 +206,7 @@ def _pick_entry(args: argparse.Namespace, baselines: list, eval_data: dict) -> d entry = next((b for b in baselines if b["name"] == args.baseline), None) if not entry: print(f"❌ Baseline '{args.baseline}' not found.") - raise SystemExit(1) + raise SystemExit(EXIT_USAGE) else: product = args.product or eval_data.get("product") candidates = [b for b in baselines if b.get("product") == product] if product else baselines @@ -260,7 +261,7 @@ def cmd_baseline_compare(args: argparse.Namespace) -> None: if not baselines: print("No baselines found. Save one first:") print(" dokimi baseline save --from-eval eval/oneTBB.json --product oneTBB") - raise SystemExit(1) + raise SystemExit(EXIT_USAGE) entry = _pick_entry(args, baselines, eval_data) baseline_data = _read_json_object(Path(entry["path"]), 2) diff --git a/dokimi/dokimi/commands/benchmark.py b/dokimi/dokimi/commands/benchmark.py index 2c50c724e..57abb83ae 100644 --- a/dokimi/dokimi/commands/benchmark.py +++ b/dokimi/dokimi/commands/benchmark.py @@ -9,7 +9,7 @@ import os import sys from pathlib import Path -from typing import Any, List, cast +from typing import Any, Callable, List, cast from dokimi.commands.evaluate import _warn_judge_independence from dokimi.eval.judge import JUDGE_PROVIDERS @@ -25,6 +25,7 @@ JUDGE_MODEL_HELP, ) from dokimi.commands.library import _load_registry +from dokimi.exit_codes import EXIT_USAGE def _run_single_library(entry, output_dir: str, model: str, provider: str, judge_model: str, judge_provider: str = DEFAULT_JUDGE_PROVIDER, doc_source_override=None, max_tokens_per_question: int = DEFAULT_MAX_TOKENS_PER_QUESTION, force_regen: bool = False, concurrency: int = DEFAULT_CONCURRENCY, questions_from=None, questions_root=None) -> dict: @@ -123,71 +124,93 @@ def _check_questions_from(questions_from: str, key: str) -> tuple[Path, list, st return qf, [], f"questions source invalid: {qf} ({exc})" -def cmd_benchmark_preflight(args: argparse.Namespace) -> None: - """Validate the resolved benchmark plan without running LLM calls.""" +def _check_golden_set(questions_root: str, key: str, refuse: Callable[[str], None]) -> None: + """Preflight the golden set under a questions root; each problem is an input refusal.""" from dokimi.orchestrator.pipeline import load_questions_payload + + root = Path(questions_root) + golden = root / f"{key.lower()}_golden.json" + if not root.is_dir(): + refuse(f"questions root not found: {root}") + return + if not golden.is_file(): + refuse(f"questions root has no golden set: {golden}") + return + try: + golden_questions = load_questions_payload(golden) + print(f" golden questions: {golden} ({len(golden_questions)} items)") + if not golden_questions: + refuse(f"golden set is empty: {golden}") + return + bad = next((i for i, q in enumerate(cast(List[Any], golden_questions)) if not isinstance(q, dict)), None) + if bad is not None: + refuse(f"golden set item {bad} is not a dict: {golden}") + except Exception as exc: + refuse(f"golden set invalid: {golden} ({exc})") + + +def _check_doc_source(plan: dict, entry: Any, refuse: Callable[[str], None], errors: list[str], + warnings: list[str]) -> None: + """The doc source must build. A --doc-source value the factory rejects is a refusal (exit 2); + a source that cannot be built is a failed check (exit 1).""" from dokimi.mcp.factory import create_doc_source_client + try: + create_doc_source_client(plan["doc_source"]) + if plan["doc_source"] == "context7" and getattr(entry, "context7_id", None): + print(f" context7_id: {entry.context7_id}") + elif plan["doc_source"] == "context7": + warnings.append("context7 id not pinned; client will resolve by target name") + except ValueError as exc: + refuse(f"doc source invalid: {plan['doc_source']} ({exc})") + except Exception as exc: # the source cannot be reached or built: a failed check + errors.append(f"doc source check failed: {plan['doc_source']} ({exc})") + + +def cmd_benchmark_preflight(args: argparse.Namespace) -> None: + """Validate the resolved benchmark plan without running LLM calls.""" registry = _load_registry(args) try: entry = registry.get(args.library) except KeyError as exc: print(f"Error: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) plan = _resolved_benchmark_plan(entry, args) _print_plan(plan) errors: list[str] = [] warnings: list[str] = [] + # Refusals are missing or invalid inputs and credentials (exit 2, as `benchmark run` refuses + # them). The other errors are failed checks (exit 1). + refusals: list[str] = [] + + def refuse(message: str) -> None: + refusals.append(message) + errors.append(message) for provider in {args.provider, args.judge_provider}: key = _credential_env_var(provider) if not os.environ.get(key): - errors.append(f"missing {key} for provider {provider}") + refuse(f"missing {key} for provider {provider}") try: Path(plan["output_dir"]).mkdir(parents=True, exist_ok=True) except OSError as exc: - errors.append(f"output dir not writable: {plan['output_dir']} ({exc})") + refuse(f"output dir not writable: {plan['output_dir']} ({exc})") - try: - client = create_doc_source_client(plan["doc_source"]) - if plan["doc_source"] == "context7" and getattr(entry, "context7_id", None): - print(f" context7_id: {entry.context7_id}") - elif plan["doc_source"] == "context7": - warnings.append("context7 id not pinned; client will resolve by target name") - _ = client - except Exception as exc: # pragma: no cover - defensive for plugin clients - errors.append(f"doc source invalid: {plan['doc_source']} ({exc})") + _check_doc_source(plan, entry, refuse, errors, warnings) if plan["questions_from"]: qf, questions, problem = _check_questions_from(plan["questions_from"], entry.key) if problem: - errors.append(problem) + refuse(problem) else: print(f" questions: {qf} ({len(questions)} items)") # Check golden set only if no external questions source (--questions-from takes precedence) if plan["questions_root"] and not plan["questions_from"]: - root = Path(plan["questions_root"]) - golden = root / f"{entry.key.lower()}_golden.json" - if not root.is_dir(): - errors.append(f"questions root not found: {root}") - elif not golden.is_file(): - errors.append(f"questions root has no golden set: {golden}") - else: - try: - golden_questions = load_questions_payload(golden) - print(f" golden questions: {golden} ({len(golden_questions)} items)") - if not golden_questions: - errors.append(f"golden set is empty: {golden}") - else: - bad = next((i for i, q in enumerate(cast(List[Any], golden_questions)) if not isinstance(q, dict)), None) - if bad is not None: - errors.append(f"golden set item {bad} is not a dict: {golden}") - except Exception as exc: - errors.append(f"golden set invalid: {golden} ({exc})") + _check_golden_set(plan["questions_root"], entry.key, refuse) same = _warn_judge_independence( answer_provider=args.provider, @@ -207,7 +230,9 @@ def cmd_benchmark_preflight(args: argparse.Namespace) -> None: print("Errors:", file=sys.stderr) for item in errors: print(f" ✗ {item}", file=sys.stderr) - sys.exit(1) + # A missing credential or input is a usage refusal (2), even with a failed check beside it; + # a failed check alone is a failed gate (1). + sys.exit(EXIT_USAGE if refusals else 1) print("✅ Preflight OK") @@ -219,7 +244,7 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None: entry = registry.get(args.library) except KeyError as exc: print(f"Error: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) output_dir = args.output_dir or f"results/{entry.key}{DEFAULT_RESULTS_SUFFIX}" n_runs = getattr(args, "multi_run", 1) @@ -231,7 +256,7 @@ def cmd_benchmark_run(args: argparse.Namespace) -> None: _, _, problem = _check_questions_from(questions_from, entry.key) if problem: print(f"Error: {problem}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) if n_runs == 1: _run_single_library( @@ -303,7 +328,7 @@ def cmd_benchmark_batch(args: argparse.Namespace) -> None: if not entries: print("No libraries to run.", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) print(f"Batch run: {len(entries)} libraries → {args.output_dir}") results = [] diff --git a/dokimi/dokimi/commands/catalog.py b/dokimi/dokimi/commands/catalog.py index f81c5a7bd..9e941cf46 100644 --- a/dokimi/dokimi/commands/catalog.py +++ b/dokimi/dokimi/commands/catalog.py @@ -16,6 +16,7 @@ import json import sys from pathlib import Path +from dokimi.exit_codes import usage_exit def _skill_row(skill, review) -> dict: @@ -62,7 +63,7 @@ def _scan_skills(args: argparse.Namespace, root: Path, report: dict, as_json: bo try: skills = read_skill_manifest(root) except ValueError as exc: - raise SystemExit(f"skills.yaml: {exc}") from exc + usage_exit(f"skills.yaml: {exc}") reviews = review_skills(skills) counts = {"ok": 0, "warn": 0, "error": 0} failed = False @@ -102,7 +103,7 @@ def _scan_tasks(args: argparse.Namespace, root: Path, report: dict, as_json: boo try: tasks = read_task_manifest(root, tasks_subdir=args.tasks_subdir) except ValueError as exc: - raise SystemExit(str(exc)) from exc + usage_exit(str(exc)) if not as_json: print(f"# tasks ({len(tasks)} declared under {root})") failed = False @@ -121,7 +122,7 @@ def _scan_tasks(args: argparse.Namespace, root: Path, report: dict, as_json: boo def cmd_catalog_scan(args: argparse.Namespace) -> None: root = Path(args.root) if not root.is_dir(): - raise SystemExit(f"--root is not a directory: {root}") + usage_exit(f"--root is not a directory: {root}") as_json = args.format == "json" report: dict = {"root": str(root), "skills": [], "tasks": []} @@ -137,7 +138,7 @@ def cmd_catalog_scan(args: argparse.Namespace) -> None: def _write_trigger_set(args: argparse.Namespace, queries, catalog_root: Path) -> None: - """Write the query set to ``--out`` in ``--format``, or exit 1 naming ``--out`` when it cannot be written.""" + """Write the query set to ``--out`` in ``--format``, or exit 2 naming ``--out`` when it cannot be written.""" import json from dokimi.artifacts import ArtifactValidationError @@ -154,9 +155,9 @@ def _write_trigger_set(args: argparse.Namespace, queries, catalog_root: Path) -> else ts.to_skill_creator(queries, args.target)) Path(args.out).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") except ArtifactValidationError as exc: # the path was writable; the payload failed the schema - raise SystemExit(f"Error: questions set rejected: {exc}") from exc + usage_exit(f"Error: questions set rejected: {exc}") except (OSError, ValueError) as exc: # unwritable --out: a directory, a file as parent, no permission, empty path (pathlib ValueError) - raise SystemExit(f"Error: cannot write --out {args.out}: {exc}") from exc + usage_exit(f"Error: cannot write --out {args.out}: {exc}") def _discover_catalog_skills(catalog_root: Path) -> list: @@ -165,7 +166,7 @@ def _discover_catalog_skills(catalog_root: Path) -> list: try: return discover_skills(catalog_root) except (OSError, ValueError, RecursionError) as exc: # unreadable, non-UTF-8, bad frontmatter or nesting too deep - raise SystemExit(f"--catalog {catalog_root}: {exc}") from exc + usage_exit(f"--catalog {catalog_root}: {exc}") def cmd_catalog_trigger_set(args: argparse.Namespace) -> None: @@ -175,42 +176,40 @@ def cmd_catalog_trigger_set(args: argparse.Namespace) -> None: catalog_root = Path(args.catalog) if not catalog_root.is_dir(): - raise SystemExit(f"--catalog is not a directory: {catalog_root}") + usage_exit(f"--catalog is not a directory: {catalog_root}") catalog = _discover_catalog_skills(catalog_root) # Checked before any --rewrite call, so a bad invocation costs no LLM calls. if args.format == "skill-creator": if not args.target: - raise SystemExit("--format skill-creator needs --target ") + usage_exit("--format skill-creator needs --target ") if args.target not in {s.name for s in catalog}: - raise SystemExit(f"--target {args.target!r} is not a skill in --catalog " - f"(have: {', '.join(sorted(s.name for s in catalog))})") + usage_exit(f"--target {args.target!r} is not a skill in --catalog " + f"(have: {', '.join(sorted(s.name for s in catalog))})") sources = _question_sources(args.questions or []) sources += _trigger_set_sources(args.trigger_set or [], catalog) if not sources: - raise SystemExit("give --questions PATH=LABEL or --trigger-set PATH[=TARGET]") - specs = list(args.questions or []) + list(args.trigger_set or []) + usage_exit("give --questions PATH=LABEL or --trigger-set PATH[=TARGET]") + # A --trigger-set source is labelled by @expected_skill whether or not its file has labels, + # so only --questions sources (named here) get the no-labels refusal. + specs = list(args.questions or []) + [None] * len(args.trigger_set or []) rewrite = None if args.rewrite: if not args.model: - raise SystemExit("--rewrite needs --model") + usage_exit("--rewrite needs --model") from dokimi.llm import llm_call def rewrite(text: str) -> str: return llm_call(ts.REWRITE_PROMPT + text, model=args.model, provider=args.provider) try: - queries = ts.build_trigger_set(sources, catalog, rewrite=rewrite) + queries = ts.build_trigger_set(sources, catalog, rewrite=rewrite, specs=specs) + except ts.TriggerSetInputError as exc: + usage_exit(str(exc)) except ValueError as exc: - raise SystemExit(str(exc)) from exc - # An @field rule that resolves for no question in its source is a typo in the - # path, not a set of negatives; refuse it rather than write "nothing loads". - start = 0 - for (qs, rule), spec in zip(sources, specs, strict=True): - chunk = queries[start:start + len(qs)] - start += len(qs) - if rule.startswith("@") and chunk and not any(q.labelled for q in chunk): - raise SystemExit(f"{spec}: {rule} resolves for none of {len(chunk)} questions") + # A --rewrite that returned nothing: a runtime failure, not a bad invocation. + print(str(exc), file=sys.stderr) + sys.exit(1) summary = ts.lint_summary(queries) if summary["label_unresolved"]: print(f"warning: {summary['label_unresolved']} questions have an unresolved @field label; " @@ -219,9 +218,9 @@ def rewrite(text: str) -> str: args.strict and (summary["names_skill"] or summary["exam_framing"] or summary["label_unresolved"]) ): # Refuse before writing, so a pipeline that only checks for the file - # cannot pick up a set with a mistyped label. + # cannot pick up a set with a mistyped label. A lint gate verdict: exit 1, as on main. print(json.dumps(summary, indent=2)) - raise SystemExit(1) + sys.exit(1) _write_trigger_set(args, queries, catalog_root) print(json.dumps(summary, indent=2)) @@ -241,7 +240,7 @@ def cmd_catalog_formats(args: argparse.Namespace) -> None: root = Path(args.root) if not root.is_dir(): - raise SystemExit(f"--root is not a directory: {root}") + usage_exit(f"--root is not a directory: {root}") packages = formats.detect_skill_packages(root) summary = formats.summarize(packages) if args.json: @@ -266,11 +265,11 @@ def _question_sources(specs: list) -> list: for spec in specs: path, _, rule = spec.rpartition("=") # labels never contain '=', paths may if not path or not rule: - raise SystemExit(f"--questions needs PATH=LABEL (a skill name, 'none' or '@field'): {spec}") + usage_exit(f"--questions needs PATH=LABEL (a skill name, 'none' or '@field'): {spec}") data = read_json_input(path, flag="--questions") questions = data.get("questions", data) if isinstance(data, dict) else data if not isinstance(questions, list) or not all(isinstance(q, dict) for q in questions): - raise SystemExit(f"{path}: expected a list of question objects") + usage_exit(f"{path}: expected a list of question objects") sources.append((questions, rule)) return sources @@ -295,16 +294,16 @@ def _trigger_set_sources(specs: list, catalog) -> list: try: fmt, queries = ts.read_trigger_set(data, target=target) except ValueError as exc: - raise SystemExit(f"{path}: {exc}") from exc + usage_exit(f"{path}: {exc}") if fmt == ts.FORMAT_SKILL_CREATOR: if target not in names: - raise SystemExit(f"{spec}: skill-creator target {target!r} is not a skill in " - f"--catalog (have: {', '.join(sorted(names))})") + usage_exit(f"{spec}: skill-creator target {target!r} is not a skill in " + f"--catalog (have: {', '.join(sorted(names))})") if len(names) > 1: print(f"note: {path}: skill-creator negatives are read as 'no skill loads'; in a " f"catalog of {len(names)} a rival load counts as a false trigger", file=sys.stderr) elif target: - raise SystemExit(f"{spec}: =TARGET applies to skill-creator sets only, this is {fmt}") + usage_exit(f"{spec}: =TARGET applies to skill-creator sets only, this is {fmt}") print(f"read {path} as {fmt} ({len(queries)} queries)", file=sys.stderr) rows = ts.to_questions(queries)["questions"] for row, q in zip(rows, queries, strict=True): diff --git a/dokimi/dokimi/commands/catalog_probe.py b/dokimi/dokimi/commands/catalog_probe.py index abecfcdda..4cfb97a28 100644 --- a/dokimi/dokimi/commands/catalog_probe.py +++ b/dokimi/dokimi/commands/catalog_probe.py @@ -13,6 +13,7 @@ import threading from concurrent.futures import ThreadPoolExecutor from pathlib import Path +from dokimi.exit_codes import usage_exit def cmd_catalog_trigger_probe(args: argparse.Namespace) -> None: @@ -109,7 +110,7 @@ def _labelled_queries(args: argparse.Namespace, names: set) -> list: data = json.loads(Path(args.trigger_set).read_text(encoding="utf-8")) _fmt, queries = ts.read_trigger_set(data, target=args.target) except (OSError, ValueError) as exc: - raise SystemExit(f"--trigger-set: {exc}") from exc + usage_exit(f"--trigger-set: {exc}") queries = [q for q in queries if q.labelled][: args.limit or None] if not queries: # else nothing runs, and an empty summary reads as a probe that ran raise InputError(f"--trigger-set has no labelled queries: {args.trigger_set}" diff --git a/dokimi/dokimi/commands/dashboard.py b/dokimi/dokimi/commands/dashboard.py index f0087e70a..af64a5fec 100644 --- a/dokimi/dokimi/commands/dashboard.py +++ b/dokimi/dokimi/commands/dashboard.py @@ -8,6 +8,7 @@ import argparse import sys from pathlib import Path +from dokimi.exit_codes import EXIT_USAGE def cmd_dashboard_generate(args: argparse.Namespace) -> None: @@ -24,7 +25,7 @@ def cmd_dashboard_generate(args: argparse.Namespace) -> None: if not data.products: print("⚠ No evaluation results found. Run benchmark first.", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) print(f"Found {len(data.products)} product(s): {', '.join(p.product for p in data.products)}") @@ -40,7 +41,7 @@ def cmd_dashboard_generate(args: argparse.Namespace) -> None: print(f"✅ JSON data: {json_path}") except OSError as exc: print(f"Error: cannot write the dashboard to {output_dir}: {exc}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) # Print quick summary to stdout print(f"\n{'─'*50}") diff --git a/dokimi/dokimi/commands/evaluate.py b/dokimi/dokimi/commands/evaluate.py index 9d3602f31..ba725e85b 100644 --- a/dokimi/dokimi/commands/evaluate.py +++ b/dokimi/dokimi/commands/evaluate.py @@ -21,6 +21,7 @@ from dokimi.eval.judge import JUDGE_PROVIDERS from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY, NoUsablePairs from dokimi.utils import normalize_model_ref, write_text_or_exit +from dokimi.exit_codes import EXIT_USAGE def _warn_judge_independence( @@ -53,7 +54,7 @@ def _load_answers(path: str): sys.exit(2) -def _write_json_or_exit(path: Path, payload: dict, code: int = 1) -> None: +def _write_json_or_exit(path: Path, payload: dict, code: int = EXIT_USAGE) -> None: """Write ``payload`` as JSON to ``path``, or exit ``code`` naming ``path`` when it cannot be written.""" write_text_or_exit(path, json.dumps(payload, indent=2), code) @@ -110,11 +111,11 @@ def cmd_eval_panel_score(args: argparse.Namespace) -> None: f"Error: --judge-configs entry '{entry}' must be 'role:model:provider'.", file=sys.stderr, ) - sys.exit(1) + sys.exit(EXIT_USAGE) role, model, provider = parts if role not in JUDGE_ROLES: print(f"Error: unknown role '{role}'. Valid: {list(JUDGE_ROLES)}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) judges.append(JudgeConfig(role=role.strip(), model=model.strip(), provider=provider.strip(), temperature=args.judge_temperature)) roles = [j.role for j in judges] @@ -123,7 +124,7 @@ def cmd_eval_panel_score(args: argparse.Namespace) -> None: unknown = [r for r in roles if r not in JUDGE_ROLES] if unknown: print(f"Error: unknown roles: {unknown}. Valid: {list(JUDGE_ROLES)}", file=sys.stderr) - sys.exit(1) + sys.exit(EXIT_USAGE) judges = [JudgeConfig(role=r, model=args.model, provider=args.provider, temperature=args.judge_temperature) for r in roles] diff --git a/dokimi/dokimi/commands/explore.py b/dokimi/dokimi/commands/explore.py index c958300d2..b64656295 100644 --- a/dokimi/dokimi/commands/explore.py +++ b/dokimi/dokimi/commands/explore.py @@ -8,6 +8,7 @@ import argparse import sys from pathlib import Path +from dokimi.exit_codes import EXIT_USAGE, usage_exit _TASKS_ROOT_HELP = ( "Directory of exploration task YAML files (default: data/explore_tasks of a " @@ -28,7 +29,7 @@ def cmd_explore_list(args: argparse.Namespace) -> None: try: tasks = discover_explore_tasks(root, default_repo_root=default_repo_root) except (FileNotFoundError, ValueError) as exc: # a task file that cannot be read or parsed - raise SystemExit(str(exc)) from exc + usage_exit(str(exc)) for task in tasks: n_refs = len(task.references) print(f"{task.id}\t{task.product}\t{n_refs} ref(s)\t{task.query[:60]}") @@ -56,7 +57,7 @@ def _build_arms(specs: list[str], output_root: Path): elif spec.startswith("command:"): template = spec[len("command:"):] if not template: - raise SystemExit("'command:' arm requires a template") + usage_exit("'command:' arm requires a template") def _factory(task, _template=template, _spec=spec): return CommandExplorer( @@ -67,9 +68,7 @@ def _factory(task, _template=template, _spec=spec): arms[spec] = _factory else: - raise SystemExit( - f"Unknown explore arm: '{spec}'. Valid: oracle, empty, command: