fix: fail incomplete task runs and add opt-in baseline regression gates

.github/workflows/agent-quality.yml

@@ -25,27 +25,8 @@ defaults:
 
 # Runners: see the header of ci.yml.
 jobs:
-  # THE executable-track proof for this repo. It is now the only job that
-  # builds and verifies a Docker task, because the task definitions no longer
-  # live here: they are supplied from an external content checkout via
-  # --tasks-root (see dokimi/docs/contributing-terminal-bench-task.md
-  # for the format). That makes this the right shape of test anyway -- it
-  # exercises the framework against content it did not author.
-  #
-  # Real end-to-end proof that this repo's own harnesses can pick up and run a
-  # skill/task from an external catalog. Clones a real, public
-  # uxlfoundation/skills checkout, scans it with `catalog scan`, then runs one
-  # of its real tasks through our own `docker-oracle` harness (build + solve +
-  # verify: the build and verifier steps `docker-claude`/`docker-claude-skill`
-  # share) and `docker-probe` (the in-container agent launch those arms use,
-  # without the LLM). It asserts the actual
-  # pass rate in "Verify pass rate" below -- `tasks run`'s own exit code does
-  # not reflect pass/fail (it is a reporting command, used by callers that want
-  # partial results), so without that explicit check this job would stay green
-  # even if the external task failed verification. Pinned to a fixed commit
-  # (UXL_SKILLS_COMMIT below) and retried 3x on clone so this job's result
-  # reflects our own code, not an upstream repo edit or a transient network
-  # blip.
+  # Runs a real task from a pinned external catalog through docker-oracle and docker-probe.
+  # `tasks run` exits 0 for a failed verification, so "Verify pass rate" below asserts the pass rate.
   external-catalog-smoke:
     name: External catalog smoke test (uxlfoundation/skills)
     runs-on: ${{ github.repository_owner == 'intel' && github.event_name == 'pull_request' && 'dokimi-ec2' || 'ubuntu-latest' }}

dokimi/changelog.d/20261021_180000_napetrov_outcome-completeness-2.md

@@ -1,3 +1,3 @@
 ### Fixed
 
-- **Nothing measured is not a pass (second batch)** — `tasks run` exits 1 as `incomplete` when every cell errored (the artifact and report are still written; partly errored runs keep exit 0 and their counts). The novelty gate's status is `NOT MEASURED`, not `PASS`, for a cell with no check measured. `run` refuses with one line and exit 1 when a selected document does not load, instead of a KeyError or OSError traceback. `baseline compare` says "Nothing to compare (0 scored)" when no question is scored on both sides, and its "No notable changes" line counts the questions compared; the snapshot `compare` report says "Nothing to compare (0 scored)" instead of "No regressions" when either snapshot has no document.
+- **Nothing measured is not a pass (second batch)** — `tasks run` exits 1 as `incomplete` when every cell errored (the artifact and report are still written). The novelty gate's status is `NOT MEASURED`, not `PASS`, for a cell with no check measured. `run` refuses with one line and exit 1 when a selected document does not load, instead of a KeyError or OSError traceback. `baseline compare` says "Nothing to compare (0 scored)" when no question is scored on both sides, and its "No notable changes" line counts the questions compared; the snapshot `compare` report says "Nothing to compare (0 scored)" instead of "No regressions" when either snapshot has no document.

dokimi/changelog.d/20261026_evaluation_completion.md

@@ -0,0 +1,19 @@
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+### Changed
+
+- Task runs preserve artifacts and exit nonzero for any errored, unscored or missing planned cell, including partly errored runs that previously exited 0; supported dry runs remain plans and verifier failures remain scored outcomes.
+
+### Fixed
+
+- `baseline compare` now treats a blank, non-string or case-variant "Unknown" provider or model, and a panel member without a `provider/` prefix, as not recorded, for every run: the "not compared" warning appears and `--strict` exits 1 where it used to pass.
+- A command harness with no reward file records its exit code as reward 1.0 or 0.0, so `tasks run` counts it as scored; exit 126, 127, signal exits and a reward file with no reward in it are no verdict, even with a `reward: x` on the output: the cell is errored and stays out of `pass_rate` and the paired test. `avg_reward` of command harnesses now includes these exit-code cells, so it does not compare with older `task_runs` artifacts.
+- `tasks run` exits 2 before running when a task or harness is listed twice, including one task under two spellings and two tasks with the same directory name.
+- Undecided judge-validity checks visibly report INCONCLUSIVE, with the existing default and strict exit policies.
+
+### Added
+
+- Opt-in baseline context-score and paired-lift regression thresholds, requiring complete matched scoring and known matching model identities. A regression and an identity mismatch exit 1; partial or insufficient evidence exits 2. `--min-matched` without a threshold exits 2. Existing `--strict` remains an identity check.

dokimi/docs/benchmarking-and-comparison.md

@@ -486,8 +486,12 @@ The report gives, per
 degradation, the pairs scored, the gate accuracy with its 95% CI and PASS or
 FAIL; `--sequential` adds a `mode` line and marks a degradation still undecided
 at the stop `continue`, with the reason under INCONCLUSIVE (cap reached, or the
-answer pool exhausted). Exit 1 on FAIL, and on an inconclusive pass with
-`--strict`. A malformed input, such as a record that would be judged with no
+answer pool exhausted). An undecided result prints `RESULT: INCONCLUSIVE`
+when it is above the gate and `RESULT: FAIL (INCONCLUSIVE, see above)` when it
+is below, never PASS. Exit policy is unchanged: a below-threshold point estimate exits 1;
+an above-threshold inconclusive result warns and exits 0 by default, or 1 with
+`--strict`. A settled PASS exits 0. JSON retains `passed` (the point-estimate
+check) and `conclusive`; promotion requires both to be true. A malformed input, such as a record that would be judged with no
 question text, exits 2, so it never reads as a failed gate.
 
 `--mode pairwise` shows the judge both answers, in both orders, and asks which
@@ -811,6 +815,28 @@ on a bad eval file or `manifest.json`, and `baseline list` exits 2 on a bad
 lack a string `name` and `path` or have a `summary` that is not an object, is rejected
 the same way.
 
+Regression promotion is opt-in and separate from identity-only `--strict`:
+
+```bash
+python cli.py baseline compare --baseline onetbb-gpt55-docs \
+  --eval results/onetbb_candidate/eval/onetbb.json \
+  --max-context-drop 1 --max-lift-drop 0.5 --min-matched 20
+```
+
+`--max-context-drop` limits the mean saved-minus-current context score.
+`--max-lift-drop` limits the mean saved-minus-current paired lift, where lift
+is context score minus the no-context score for the same question. Both use
+unrounded per-question aggregates, not stored rounded deltas or manifest means.
+Thresholds are finite, non-negative score points; equality passes. Either flag
+enables the gate; when both are supplied both must pass. `--min-matched` defaults
+to 1 and needs a threshold flag (alone it exits 2). The gate exits 1 on a
+regression or an unknown or mismatched answer/judge identity. Insufficient
+evidence exits 2, since the input can be fixed: every question in both artifacts
+must match uniquely and have finite scores in both conditions, and at least
+`--min-matched` must match. Unmatched questions, duplicates, errors and partial
+scoring cannot promote a candidate. Invalid thresholds and malformed records
+exit 2. Issue #94 moves these thresholds to `gate`.
+
 Use baselines for release-to-release tracking. Use `--questions-from` for
 model-to-model fairness.
 

dokimi/docs/coding-harnesses.md

@@ -579,10 +579,12 @@ a verdict. The rules mirror intel/skills' `tools/check_harbor_job.py`:
 | no trial file (with or without the job's), more than one trial, or not a JSON object | errored |
 
 An errored cell is out of the pass-rate denominator and the paired test, as
-for `docker-*` and BenchFlow. When every cell of a `tasks run` errored, it
-still writes the artifact and the report, then exits 1 as `incomplete`
-(`0 scored, N errored of N cells`); a run with some cells scored exits 0 and
-prints the errored count per harness. `metrics.n_trials` counts the trial files and
+for `docker-*` and BenchFlow. When any planned cell of a `tasks run` errors, lacks a finite score, or is missing,
+the command writes the artifact and report, then exits 1 as `incomplete`.
+It reports scored, errored, unscored and missing counts and names missing cells.
+A completed verifier failure (reward 0) is scored evidence and does not itself
+change the exit code. A supported `--dry-run` plan exits 0 without claiming
+execution; unsupported cells still fail. `metrics.n_trials` counts the trial files and
 `metrics.result_json` names the one read. A `harbor` that `dokimi` kills at
 the task budget is a failed cell (reward 0, returncode 124), as above.
 
@@ -604,8 +606,11 @@ refuses the task instead of running it with the internet the task forbade.
 Both now land as errored cells.
 
 A `command:` harness (including a `--command-template` that wraps `harbor`)
-keeps its own rule: a `reward.txt` / `reward.json` under the run dir, else a
-`reward: x` on its output, else exit code 0 is a pass.
+keeps its own rule: a `reward.txt` / `reward.json` under the run dir, else (no
+reward file) a `reward: x` on its output, else exit code 0 is a pass. Exit 126 or 127, a
+signal exit, or a reward file with no reward in it is no verdict: the cell is
+errored (`metrics.solver_error`). Only a reward file with a reward in it outranks such an
+exit; a `reward: x` on the output does not.
 
 ### Network policy
 

dokimi/docs/reference/cli-interface.json

@@ -1578,6 +1578,54 @@
               "required": false,
               "type": null
             },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--max-context-drop",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "dokimi.commands.baseline_regression.threshold#83cba01472d6"
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--max-lift-drop",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "dokimi.commands.baseline_regression.threshold#83cba01472d6"
+            },
+            {
+              "action": "_StoreAction",
+              "choices": null,
+              "const": null,
+              "default": null,
+              "group_key": null,
+              "group_required": null,
+              "hidden": false,
+              "is_flag": false,
+              "key": "--min-matched",
+              "kind": "option",
+              "nargs": null,
+              "position": null,
+              "required": false,
+              "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"
+            },
             {
               "action": "_StoreAction",
               "choices": null,

dokimi/docs/user-journeys/journeys.md

@@ -717,19 +717,21 @@ cost.
 `python cli.py baseline compare --strict`), as do static gates
 (`python cli.py repo scan --fail-on-fail`). `baseline compare --strict` adds a
 non-zero exit only when a model cannot be verified or differs from the
-baseline; a regression in the scores is printed and exits 0. Failing on a regression beyond
-a threshold is **proposed** (AC-J12-001). Blocking policy is **partial**
-(AC-J12-002): `repo tasks check --allow-unsupported` decides host-unsupported
-tasks; budget-stopped and inconclusive judge results have no policy.
+baseline. Explicit `--max-context-drop` and `--max-lift-drop` thresholds block
+score regressions, requiring complete matched scores and known matching models
+(AC-J12-001). `eval judge-validity --strict` blocks inconclusive checks; without
+it an above-threshold inconclusive check warns and exits 0. Blocking policy remains
+partial for budget-stopped checks (AC-J12-002). Native `tasks run` rejects any
+missing, unscored or errored planned cell and preserves the artifacts (AC-J12-004).
 
 ### Use cases and acceptance
 
 #### UC-J12-001 Gate a PR on a budgeted check
 
-- **AC-J12-001** A strict baseline comparison exits non-zero on a regression beyond the threshold.
-- **AC-J12-002** An inconclusive or budget-stopped check is reported as such, and whether it blocks is the configured policy's decision. *Partial: one flag, for host-unsupported tasks; budget-stopped and inconclusive judge results have none.*
+- **AC-J12-001** An explicitly gated baseline comparison exits non-zero on a regression beyond its context-score or paired-lift threshold; `--strict` alone checks model identity.
+- **AC-J12-002** An inconclusive or budget-stopped check is reported as such, and whether it blocks is the configured policy's decision. *Partial: explicit flags cover host-unsupported tasks and inconclusive judge checks; general budget-stop policy remains proposed.*
 - **AC-J12-003** The reusable CI check runs only the tasks a change touches, adds a smoke set when shared harness files change, and fails on an unknown task name instead of checking nothing.
-- **AC-J12-004** A gate on a run whose result lacks planned cells reports it incomplete and names the missing cells; it never passes on the cells that finished. *Proposed.*
+- **AC-J12-004** A gate on a run whose result lacks planned cells reports it incomplete and names the missing cells; it never passes on the cells that finished. *Partial: enforced for native task runs.*
 
 ---
 

dokimi/dokimi/commands/baseline.py

@@ -13,6 +13,7 @@
 from typing import Any, Dict, List, Optional, Tuple
 
 from dokimi.utils import model_identity, same_model
+from dokimi.commands import baseline_regression
 from dokimi.exit_codes import EXIT_USAGE
 
 BASELINES_DIR = Path("baselines/eval")
@@ -99,7 +100,8 @@ def _recorded_models(eval_data: Dict[str, Any]) -> Dict[str, Optional[Tuple[str,
         meta = {}
 
     def pair(provider: Any, model: Any) -> Optional[Tuple[str, str]]:
-        if not provider or not model or "unknown" in (provider, model):
+        if any(not isinstance(v, str) or not v.strip() or v.strip().lower() == "unknown"
+               for v in (provider, model)):
             return None
         if model_identity(str(model))[0] is None:  # a route with no id after it ("anthropic/") names no model
             return None
@@ -109,7 +111,8 @@ def pair(provider: Any, model: Any) -> Optional[Tuple[str, str]]:
     if str(judge_model or "").startswith("panel("):
         panel = eval_data.get("panel_config")
         models = panel.get("models") if isinstance(panel, dict) else None
-        if not (isinstance(models, list) and all(isinstance(m, str) for m in models)):
+        if not (isinstance(models, list) and all(isinstance(m, str) and "/" in m
+                                                and pair(*m.split("/", 1)) is not None for m in models)):
             models = []
         judge = ("panel", ",".join(sorted(models))) if models else None
     else:
@@ -276,7 +279,8 @@ def _pick_entry(args: argparse.Namespace, baselines: list, eval_data: dict) -> d
 
 def _report_model_checks(args: argparse.Namespace, entry: dict, baseline_data: dict, eval_data: dict) -> None:
     mismatches, unverified = _model_mismatches(baseline_data, eval_data)
-    strict = getattr(args, "strict", False)
+    strict = getattr(args, "strict", False) or baseline_regression.enabled(args)
+    policy = "--strict" if getattr(args, "strict", False) else "regression gate"
     base_rec, cur_rec = _recorded_models(baseline_data), _recorded_models(eval_data)
     for role in unverified:
         b, c = base_rec[role], cur_rec[role]
@@ -287,13 +291,13 @@ def _report_model_checks(args: argparse.Namespace, entry: dict, baseline_data: d
         sides = [name for name, rec in (("baseline", base_rec), ("current eval", cur_rec)) if rec[role] is None]
         print(f"⚠ {role} model not recorded in {' and '.join(sides)}; not compared", file=sys.stderr)
     if unverified and strict:
-        print(f"❌ --strict cannot verify the {' and '.join(unverified)} model against "
+        print(f"❌ {policy} cannot verify the {' and '.join(unverified)} model against "
               f"baseline '{entry['name']}'", file=sys.stderr)
         raise SystemExit(1)
     if mismatches:
         detail = "".join(f"\n  {line}" for line in mismatches)
         if strict:
-            print(f"❌ Baseline '{entry['name']}' was scored with different models (--strict):{detail}",
+            print(f"❌ Baseline '{entry['name']}' was scored with different models ({policy}):{detail}",
                   file=sys.stderr)
             raise SystemExit(1)
         print(f"⚠ Baseline model warning: '{entry['name']}' was scored with different models;"
@@ -316,6 +320,7 @@ def _print_metric_table(cur_summary: dict, base_summary: dict) -> None:
 
 def cmd_baseline_compare(args: argparse.Namespace) -> None:
     """Compare an eval result against a saved baseline."""
+    baseline_regression.require_gate_for_min_matched(args)
     eval_path = Path(args.eval)
     eval_data = _read_json_object(eval_path, 2)  # 1 is the --strict verdict
     manifest = _baseline_manifest(2)
@@ -333,8 +338,9 @@ def cmd_baseline_compare(args: argparse.Namespace) -> None:
 
     # Read both record lists before the model checks: a malformed file must exit 2, not reach the --strict verdict.
     try:
-        baseline_evals = {e.key: e for e in records(baseline_data.get("evaluations", []),
-                                                    source=entry["path"], kind="evaluations")}
+        baseline_records = records(baseline_data.get("evaluations", []),
+                                   source=entry["path"], kind="evaluations")
+        baseline_evals = {e.key: e for e in baseline_records}
         current_evals = records(eval_data.get("evaluations", []), source=str(eval_path), kind="evaluations")
     except RecordError as exc:
         print(f"❌ {exc}", file=sys.stderr)
@@ -377,6 +383,8 @@ def cmd_baseline_compare(args: argparse.Namespace) -> None:
     else:
         print(f"\n✅ No notable changes vs baseline ({n_compared} of {len(current_evals)} questions compared).")
 
+    baseline_regression.check(args, baseline_records, current_evals)
+
 
 def register(sub, positive_int) -> None:
     """Add the `baseline` subcommand group."""
@@ -403,4 +411,12 @@ def register(sub, positive_int) -> None:
     bl_cmp.add_argument("--strict", action="store_true",
                         help="Exit 1 instead of warning when the baseline's answer or judge model "
                              "differs from the current eval's")
+    bl_cmp.add_argument("--max-context-drop", type=baseline_regression.threshold, default=None,
+                        help="Gate on the mean matched context-score drop in score points (equality passes)")
+    bl_cmp.add_argument("--max-lift-drop", type=baseline_regression.threshold, default=None,
+                        help="Gate on the mean matched (context minus baseline) lift drop in score points")
+    bl_cmp.add_argument("--min-matched", type=positive_int, default=None,
+                        help="Minimum matched scored questions under a regression gate (default: 1); "
+                             "needs --max-context-drop or --max-lift-drop; complete coverage and "
+                             "matching known models are always required")
     bl_cmp.set_defaults(func=cmd_baseline_compare)