fix(eval): infra errors and empty replies are counted as errors, never scored

dokimi/changelog.d/20261028_120200_napetrov_feat-run-hygiene.md

@@ -0,0 +1,18 @@
+### Added
+
+- **Errors are counted apart.** An answer whose call failed is tallied as `errors` in `eval reference` summaries and per arm in arms summaries. The arms summary shows an `Errors` column, and the console summary shows the count per arm.
+
+### Changed
+
+- **Earlier figures are not comparable.** Pass rates, deltas and selection rates from runs before this change do not match new ones for the same inputs. Re-run `arms judge` or `eval reference` to compare.
+- **Errors leave the scores out.** An answer whose call failed leaves `n`, the pass rate and the mean out. Before, it was lumped with answers that no check could score.
+- **Empty replies are errors, never scores.** An empty, whitespace-only or content-less model reply is an error on its cell in arms runs and in `eval reference`: it is never judged or graded (a negated check no longer scores `""` as 1.0), and it leaves `n`, the pass rate and the mean out.
+- **Judge failures with an empty message are counted.** A judge exception whose message is empty (for example `TimeoutError()`) is recorded under its class name, so it is counted as an error rather than dropped.
+- **`delta_vs_baseline` is paired.** It is the mean of arm minus baseline over the questions both arms scored. Before, it was the difference of two unpaired means, so a question one arm errored on moved the delta.
+- **A judge failure is an error, not a pass.** In `eval reference`, a judge check whose call raised, or whose reply had no YES or NO, makes the answer an error: no score and no pass, even when its other checks passed. Before, the other checks decided the pass. The block carries `error`, as an arm cell does.
+- **An empty reply stays an error when an output shaper adds text.** Emptiness is read on the model's reply before the shapers run, so a footer on an empty reply is not an answer.
+
+### Fixed
+
+- **Empty replies and content-less replies are not cached.** Neither is stored in or replayed from the response cache, so a later run asks the model again.
+- **`delta_vs_baseline` is always present.** It is `null` when no question has both arms scored, instead of missing.

dokimi/docs/answer-formats.md

@@ -76,14 +76,22 @@ value is blank or missing.
 Every check accepts `mode` (`any`/`all` over its `values`), `negate` and
 `weight`. A question's score is the weighted mean of the checks that were
 counted. It passes when it reaches `threshold`; without a threshold, it passes
-when every counted check passes. A failed answer call leaves its checks
-unscored rather than failed. `--no-judge` leaves the `judge` checks unscored
+when every counted check passes. A failed answer call is an error, not a failed
+check: the answer gets no score and no pass. `--no-judge` leaves the `judge` checks unscored
 and makes no model calls; an arms artifact written that way has `judge_model`
 and `judge_provider` set to `reference`. The summary's `judge_unscored` counts the
 counted answers whose pass leaves out a judge check that had no verdict. The
 summary's `check_unscored` does the same for a deterministic check that did not
 run to a result (a `regex` past its limit): the pass is decided on the checks
-that ran, and the count shows how many passes left one out.
+that ran, and the count shows how many passes left one out. An answer whose
+call failed is counted as `errors`, not `unscored`, and leaves the pass rate and
+the mean out just as an unscored answer does. A judge call that raises, or whose
+reply has no YES or NO after its retries, is an error too: the answer gets no score
+and no pass, even when its other checks passed. The block's `error` names it (for
+example `judge: TimeoutError`), as an arm cell's `error` does. An arms summary has the same
+count per arm as `errors`. An empty or content-less reply is an error, never graded:
+it leaves the pass rate and the mean out, and is never cached (on store or replay),
+so a rerun asks the model again.
 
 `eval reference` writes eval.v1 for an `answers generate` file and an arms
 artifact for an `arms run` file. Each block carries `scorer: reference`, and
@@ -119,7 +127,7 @@ because the native run had no dokimi doc context. The tool's own verdicts
 become an eval.v1 file with `scorer: native:<tool>`. Running `eval reference`
 on that answers file re-grades the same completions with dokimi's checks,
 which shows where the two graders disagree. An Inspect sample that errored
-keeps its error on the answer, so `eval reference` leaves it unscored, as
-Inspect does. A promptfoo output must come from
+keeps its error on the answer, so `eval reference` counts it as an error, not a
+score, as Inspect does. A promptfoo output must come from
 a single provider. Supported log formats are Inspect `json` logs (inspect-ai
 0.3.283) and promptfoo `-o *.json` (0.125.0).

dokimi/docs/evaluating-treatments.md

@@ -718,10 +718,10 @@ Two gates are deliberately statistical rather than absolute:
 - **coverage, not zero errors.** A judge call fails occasionally (~1.25% observed against
   the Intel proxy), and a failed call leaves `aggregate: null`. The gate requires ≥95% of
   questions scored in *both* arms.
-- **paired deltas, not the artifact's own number.** `delta_vs_baseline` averages each arm
-  over whatever it scored, so one failed call subtracts means taken over different
-  questions (39 vs 40 was observed, worth ~0.4 points). The script recomputes over
-  questions both arms scored. It pools questions; a mean of per-cell means differs slightly
+- **paired deltas.** The artifact's `delta_vs_baseline` is paired over the questions both
+  arms scored, so one failed call does not subtract means taken over different
+  questions (39 vs 40 was observed, worth ~0.4 points). The script recomputes it over
+  those questions. It pools questions; a mean of per-cell means differs slightly
   when the sets have different sizes — say which one a report uses.
 
 ## Scope and caveats

dokimi/dokimi/caching.py

@@ -278,15 +278,32 @@ def draw_extra(extra: Dict[str, Any], draw: int) -> Dict[str, Any]:
     return {**extra, "draw": draw} if draw else extra
 
 
-def keeps(cache_if: Optional[Callable[[str], bool]], text: str) -> bool:
+def is_empty_reply(text: Optional[str]) -> bool:
+    """Whether a reply carries no answer: no content (``None``) or only whitespace.
+
+    The one test every path shares (response cache, single-turn and session arms,
+    ``eval reference``, the judge). An empty reply is an error on the cell: it is
+    never scored, never counted as a 0, and never stored or replayed.
+    """
+    return text is None or not text.strip()
+
+
+def keeps(cache_if: Optional[Callable[[str], bool]], text: Optional[str]) -> bool:
     """Whether a completion may be stored, or a stored one replayed.
 
     *cache_if* is the caller's own acceptance test. The judge passes "this reply
     parses": without it an unparseable reply is stored like any other, and once
     all three of the judge's attempts are stored that way, every later run
     replays the failure without calling the model. Checking on replay too means
     entries stored before the check existed stop poisoning the run.
+
+    An empty reply (see :func:`is_empty_reply`) is never kept, on store or on
+    replay: a stored empty answer would be replayed as a failed answer on every
+    later run, and a content-less one (stored as null) as "no answer". ``text`` is
+    ``None`` for a content-less reply.
     """
+    if text is None or is_empty_reply(text):  # the None test narrows the type for cache_if
+        return False
     return cache_if is None or cache_if(text)
 
 

dokimi/dokimi/commands/arms.py

@@ -80,7 +80,8 @@ def cmd_arms_run(args: argparse.Namespace) -> None:
             delta = stats.get("delta_vs_baseline")
             avg_s = "n/a" if avg is None else f"{avg:.1f}"
             delta_s = "" if (delta is None or arm == args.baseline_arm) else f" (delta {delta:+.1f})"
-            print(f"  {arm:<24} {avg_s}{delta_s}")
+            err_s = f" ({stats['errors']} errors)" if stats.get("errors") else ""
+            print(f"  {arm:<24} {avg_s}{delta_s}{err_s}")
 
 
 def cmd_arms_matrix_run(args: argparse.Namespace) -> None:

dokimi/dokimi/eval/arm_runner.py

@@ -18,14 +18,15 @@
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Sequence
 
+from dokimi.caching import is_empty_reply
 from dokimi.utils import question_id  # noqa: F401  (re-exported: callers import it from here)
 from dokimi.artifacts import save_artifact
 from dokimi.llm import llm_call_with_usage
 from dokimi.defaults import DEFAULT_ANSWER_MODEL, DEFAULT_ANSWER_PROVIDER
 from dokimi.eval import stats as arm_stats
 from dokimi.eval.interactions import interaction_significance
 from dokimi.eval.arm_summary import (
-    cost_summary, summarize_agentic_usage, summarize_skill_selection,
+    cell_values, cost_summary, empty_reply_cell, paired_delta, summarize_agentic_usage, summarize_skill_selection, unjudged_cell,
 )
 from dokimi.eval.cells import Cell
 from dokimi.eval.question_text import question_text_of, require_question_texts
@@ -393,11 +394,13 @@ def _answer_one_arm(
             return {"error": f"generation_failed: {exc}", "answer": None,
                     "metadata": cfg_metadata}
 
-        raw_chars = len(answer_text or "")
+        rec = to_record(usage, self.model, self.provider)
+        # Read on the reply the model gave, before the shapers: a footer added to nothing is not an answer.
+        if is_empty_reply(answer_text):  # an error cell, as a session's empty turn; the usage still counts
+            return empty_reply_cell(rec, cfg_metadata, t0)
         final_text = self._apply_shapers(shapers, answer_text)
+        raw_chars = len(answer_text or "")
         final_chars = len(final_text or "")
-
-        rec = to_record(usage, self.model, self.provider)
         return {
             "answer": final_text,
             "raw_answer": answer_text if shapers else None,
@@ -566,8 +569,13 @@ def _answer_agentic_arm(
                     "tool_usage": tool_usage, "metadata": cfg_metadata}
 
         raw_answer = result["answer"]
-        final_answer = self._apply_shapers(shapers, raw_answer)
         rec = to_record(result.get("usage") or result["token_usage"], self.model, self.provider)
+        if is_empty_reply(raw_answer):  # read before the shapers: a footer added to nothing is not an answer
+            return empty_reply_cell(
+                rec, cfg_metadata, t0, agentic=True, transcript=result["transcript"],
+                tool_call_count=result["tool_call_count"], tool_usage=result.get("tool_usage", {}),
+                iterations=result["iterations"], stopped_reason=result["stopped_reason"])
+        final_answer = self._apply_shapers(shapers, raw_answer)
         metrics = rec.as_metrics_dict(
             answer_chars=len(raw_answer or ""), final_chars=len(final_answer or "")
         )
@@ -614,8 +622,8 @@ def _score(rec: Record) -> Dict[str, Any]:
             gt = rec.ground_truth
             for arm_name in arm_names(rec):
                 answer = rec.answer(arm_name)
-                if answer is None or not answer.text:
-                    scores[arm_name] = None
+                if answer is None or is_empty_reply(answer.text):
+                    scores[arm_name] = unjudged_cell(answer)  # an errored or empty answer is never judged
                     continue
                 context, source = self.judge_context.resolve_with_source(
                     rec.question_id, arm_name, dict(answer.raw)
@@ -630,7 +638,7 @@ def _score(rec: Record) -> Dict[str, Any]:
                     )
                 except Exception as exc:
                     logger.exception("Judging arm %s failed", arm_name)
-                    score = {"error": str(exc)}
+                    score = {"error": str(exc) or type(exc).__name__}
                 if isinstance(score, dict):
                     score = {**score, "judge_context_source": source}
                     digest = (self.judge_context.digest_of(rec.question_id, arm_name)
@@ -752,25 +760,18 @@ def build_output(
     def _summarize(
         self, evaluations: List[Dict[str, Any]], baseline_arm: str
     ) -> Dict[str, Any]:
-        per_arm: Dict[str, List[float]] = {name: [] for name in self.arm_names}
-        for ev in as_records(evaluations, source="evaluations", kind="evaluations"):
-            for arm_name in ev.scores:
-                if (v := ev.score(arm_name)) is not None:
-                    per_arm.setdefault(arm_name, []).append(float(v))
+        per_arm, errors = cell_values(evaluations, self.arm_names)
 
         summary: Dict[str, Any] = {"per_arm": {}}
-        base_avg = None
         for arm_name, vals in per_arm.items():
             avg = round(sum(vals) / len(vals), 1) if vals else None
-            summary["per_arm"][arm_name] = {"avg_aggregate": avg, "n": len(vals)}
-            if arm_name == baseline_arm:
-                base_avg = avg
-        if base_avg is not None:
-            for arm_name, stats in summary["per_arm"].items():
-                avg = stats["avg_aggregate"]
-                stats["delta_vs_baseline"] = (
-                    round(avg - base_avg, 1) if avg is not None and arm_name != baseline_arm else None
-                )
+            summary["per_arm"][arm_name] = {"avg_aggregate": avg, "n": len(vals), "errors": errors.get(arm_name, 0)}
+        # Paired over the questions both arms scored; the unpaired means above are still reported.
+        # Always present: null for the baseline, and for an arm with no question both arms scored.
+        for arm_name, stats in summary["per_arm"].items():
+            stats["delta_vs_baseline"] = (
+                paired_delta(evaluations, baseline_arm, arm_name) if arm_name != baseline_arm else None
+            )
 
         # Paired statistics. Every arm is tested against the *same* baseline
         # sample question-by-question, so the arms are mutually comparable and

dokimi/dokimi/eval/arm_summary.py

@@ -13,10 +13,12 @@
 
 from __future__ import annotations
 
-from typing import Any, Dict, Iterable, Mapping
+import time
+from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple
 
-from dokimi.records import RecordLike
+from dokimi.records import Answer, RecordLike
 from dokimi.records import records as as_records
+from dokimi.eval.stats import paired_vectors
 
 #: Stamp on an agentic answer's ``tool_usage["counts"]``: its tool counts skip
 #: calls that failed (``error`` in the transcript). Unstamped answers counted
@@ -154,7 +156,10 @@ def summarize_skill_selection(records: Iterable[RecordLike]) -> Dict[str, Any]:
             })
             if failed:
                 s["failed"] += 1
-                continue
+                # An empty final reply fails the answer, not the choice: the skills were
+                # loaded before it came back empty, so the choice is still scored.
+                if arm.get("error") != "empty_reply":
+                    continue
             first = skill.get("selected")
             # Skills loaded together in the first turn are one choice. It is a hit
             # only if every one of them is acceptable: crediting "the right one is
@@ -264,3 +269,49 @@ def cost_summary(records: Iterable[RecordLike]) -> Dict[str, Any]:
                                 if prompt else None),
         }
     return out
+
+
+def unjudged_cell(answer: Optional[Answer]) -> Optional[Dict[str, str]]:
+    """The cell of an answer that is not judged: an error for an errored or empty one, ``None`` for none."""
+    if answer is None:
+        return None
+    return {"error": answer.error or "empty_reply"}
+
+
+def empty_reply_cell(rec: Any, metadata: Any, started: float, **extra: Any) -> Dict[str, Any]:
+    """The answer cell of a call that returned no text: an error, never an answer.
+
+    The call was paid for, so its usage (*rec*) is still recorded. *started* is the ``time.time()``
+    at the start of the arm; *extra* carries what the agent loop adds to a cell.
+    """
+    return {"error": "empty_reply", "answer": None, "token_usage": rec.as_token_usage_dict(),
+            "metrics": rec.as_metrics_dict(), "metadata": metadata,
+            "elapsed_sec": round(time.time() - started, 2), **extra}
+
+
+def paired_delta(evaluations: Iterable[Any], baseline_arm: str, arm_name: str) -> Optional[float]:
+    """The mean of ``arm - baseline`` over the questions that both arms scored; ``None`` if none.
+
+    A question one arm errored on (or left unscored) is not in the delta at all: the
+    difference of two unpaired means would compare different questions.
+    """
+    arm_vals, base_vals = paired_vectors(evaluations, arm_name, baseline_arm)
+    diffs = [a - b for a, b in zip(arm_vals, base_vals, strict=True)]
+    return round(sum(diffs) / len(diffs), 1) if diffs else None
+
+
+def cell_values(evaluations: Iterable[Any], arm_names: Sequence[str]) -> Tuple[Dict[str, List[float]], Dict[str, int]]:
+    """Per arm, the aggregates that count, and how many cells errored instead.
+
+    An errored cell (the answer call failed, or the judge raised) has no aggregate, so it
+    is left out of the mean and its ``n``; the errors are counted apart, not as scores.
+    """
+    values: Dict[str, List[float]] = {name: [] for name in arm_names}
+    errors: Dict[str, int] = {name: 0 for name in arm_names}
+    for ev in as_records(evaluations, source="evaluations", kind="evaluations"):
+        for arm_name, cell in ev.scores.items():
+            if (v := ev.score(arm_name)) is not None:
+                values.setdefault(arm_name, []).append(float(v))
+            elif cell is not None and cell.error:
+                errors[arm_name] = errors.get(arm_name, 0) + 1
+    return values, errors

dokimi/dokimi/eval/checklist.py

@@ -105,16 +105,30 @@ def consistent(self) -> Optional[bool]:
         return self.covered != self.absent
 
 
-def check_answer(ask: Ask, question: str, answer: str, points: Sequence[str]) -> List[PointVerdict]:
-    """Both polarities for every point."""
+class JudgeFailed(RuntimeError):
+    """A judge call raised, or its reply had no YES or NO after every draw (``strict`` checks only)."""
+
+
+def check_answer(ask: Ask, question: str, answer: str, points: Sequence[str], *,
+                 strict: bool = False) -> List[PointVerdict]:
+    """Both polarities for every point.
+
+    By default a failed call or an unparseable reply leaves its point unscored. With ``strict``
+    it raises :class:`JudgeFailed` instead, for a caller that must not grade an answer on the
+    points that did parse.
+    """
     def one(point: str, q: str) -> Optional[bool]:
         try:
             # Retried under new draws, and an unparseable reply is never cached.
             verdict, _ = ask_parsed(ask, _TEMPLATE.format(question=question, answer=answer, point=point, ask=q),
                                     parse_yes_no)
         except Exception as exc:  # one failed call is an unparsed point, not a lost run
+            if strict:
+                raise JudgeFailed(str(exc) or type(exc).__name__) from exc
             logger.warning("Checklist call failed, point left unscored: %s", exc)
             return None
+        if verdict is None and strict:
+            raise JudgeFailed("no YES or NO in the reply")
         return verdict
 
     return [