diff --git a/dokimi/changelog.d/20261029_120000_antonwolfy_replay-hit-pairs.md b/dokimi/changelog.d/20261029_120000_antonwolfy_replay-hit-pairs.md new file mode 100644 index 000000000..685e7e1e7 --- /dev/null +++ b/dokimi/changelog.d/20261029_120000_antonwolfy_replay-hit-pairs.md @@ -0,0 +1,4 @@ +### Added + +- **Replay hits you can check** — a `dokimi:review_replay` grade now records `details.matches`: per credited thread, the thread (id, anchor, gold comment) and the finding that hit it (its index, location, severity, message) with their distance in lines, so a hit can be read from `grade.json` alone. The metrics are unchanged. `dokimi repo report --hits` lists those pairs by task, arm, model and rep, as the cell's `result.json` names them even when a match carries a key of the same name (`--json` for records), and counts the cells graded before `matches` existed. +- **Parse errors in `repo report`** — a row notes how many of its cells had an agent answer whose findings JSON did not parse (`grade.json` `parse_error`): scored 0 on a score-only row, counted as failed on a gate row, and not counted for a cell already counted as a grader failure. `--json` gives `parse_errors` per row, so a lower recall from answers that did not parse can be told from one from worse reviews. diff --git a/dokimi/docs/reference/cli-interface.json b/dokimi/docs/reference/cli-interface.json index 3778ecf3d..ad1f9d568 100644 --- a/dokimi/docs/reference/cli-interface.json +++ b/dokimi/docs/reference/cli-interface.json @@ -10792,6 +10792,22 @@ "required": false, "type": null }, + { + "action": "_StoreTrueAction", + "choices": null, + "const": true, + "default": false, + "group_key": null, + "group_required": null, + "hidden": false, + "is_flag": true, + "key": "--hits", + "kind": "option", + "nargs": 0, + "position": null, + "required": false, + "type": null + }, { "action": "_StoreTrueAction", "choices": null, diff --git a/dokimi/docs/repo-evaluation-contract.md b/dokimi/docs/repo-evaluation-contract.md index 5a05bba8c..38bcfaaea 100644 --- a/dokimi/docs/repo-evaluation-contract.md +++ b/dokimi/docs/repo-evaluation-contract.md @@ -616,7 +616,7 @@ failed the task, and on a task with `score` that is a failed run with score | `grader` | data next to `task.json` (key that renames it) | `pass` | `metrics` | `details` | |---|---|---|---|---| | `dokimi:review_seeds` | `manifest.json` (`manifest`); oracle answer `oracle.json` (`oracle_answer`), a list of findings | `null`; for a manifest with no seeds (a clean-diff control), `true` when no finding is `blocker` or `major` | `n_findings`, `n_severe`, `n_decoys`, `unmatched`, `recall` (`null` with no seeds) | `seeds_hit`, `decoys_flagged`, `recall_by_class` (`"hit/total"`) | -| `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `recall_by_kind`, `decoys_flagged` (always `[]`) | +| `dokimi:review_replay` | `gold.json` (`gold`); `task.json` `window`, default 5 | `null` (score-only) | `n_findings`, `n_severe`, `n_threads`, `recall` (= `recall_all`), `recall_addressed`, `precision_proxy` | `threads_hit`, `matches`, `recall_by_kind`, `decoys_flagged` (always `[]`) | `manifest.json` is `{"seeds": [{id, class, file, lines, kw}], "decoys": [{id, file, lines, kw}]}`; `file` `"*"` matches any file, and `kw` is a @@ -637,6 +637,12 @@ review thread of a merged PR, anchored on a line of the reviewed revision. - A finding on a thread's file within `window` lines of its anchor can hit it. Matching is one-to-one and nearest first: one finding credits at most one thread, and one thread at most one finding. +- `matches` holds one object per credited pair, in `threads_hit` order: the + thread's `thread` (id), `thread_file`, `thread_line` and `comment`, the + finding's `finding` (its index among the answer's objects), `file` + (repository-relative), `line` (as read for matching), `severity` and + `message`, and their `distance` in lines; `comment`, `severity` and + `message` are cut to 200 characters. `repo report --hits` lists them (§9). - `recall_addressed` counts only threads with `addressed: true` (the code changed in response), `recall_by_kind` groups by `kind`, and `precision_proxy` is the share of findings within the window of any @@ -1330,7 +1336,19 @@ python cli.py repo compare repo-runs/replay --baseline guidance:none --metric re When cells carry `result.json` `process` (§4.1), a line under the row gives the mean of each count over the cells that have it (`guidance_read` counted as paths; `counter:` for `process_metrics`); a `null` or absent - `process` is left out, not counted as zero. + `process` is left out, not counted as zero. A row whose cells include + agent answers with no findings JSON that parsed (`grade.json` + `parse_error`, §4.5) notes how many, `(scored 0)` on a score-only row and + `(counted as failed)` on a gate row; a cell whose `result.json` says + `grader_failure` is counted under `grader_fail` instead. `--json` gives + `parse_errors` on every row. `repo report --hits` prints, instead of the table, each credited pair of + the `dokimi:review_replay` cells (`details.matches`, §4.5) by task, arm, + model and rep: the thread and its comment, then the finding that hit it. + `--hits --json` gives `{batch, replay_cells, without_matches, hits}`, one + record in `hits` per pair (`task`, `arm`, `model`, `rep`, `cell` and the + match's fields; the first five come from the cell's `result.json`, and a + match key of the same name does not replace them). A cell graded before `matches` was recorded is counted + (`N cell(s) graded without hit details`), not listed. - `repo compare` reduces cells to one value per (task, arm, model) first: repetitions share a task's difficulty and are not paired across arms, so the task is the unit. Per treatment arm, model and measure (gate share, score, diff --git a/dokimi/dokimi/commands/repo_run.py b/dokimi/dokimi/commands/repo_run.py index 585a7a544..e4992e0d9 100644 --- a/dokimi/dokimi/commands/repo_run.py +++ b/dokimi/dokimi/commands/repo_run.py @@ -383,14 +383,18 @@ def _harness_bin(name: str) -> Path: def cmd_repo_report(args: argparse.Namespace) -> None: - """Per (task, arm, model): gated success, grader failures, missing cells, cost, turns.""" - from dokimi.repos.report import CellFileError, format_report, summarize + """Per (task, arm, model): gated success, grader failures, missing cells, cost, turns; or the replay hits.""" + from dokimi.repos.report import CellFileError, format_hits, format_report, load_hits, summarize batch_dir = Path(args.batch_dir) if not batch_dir.is_dir(): print(f"✗ not a directory: {batch_dir}", file=sys.stderr) sys.exit(1) try: + if args.hits: + found = load_hits(batch_dir) + print(json.dumps(found, indent=2) if args.json else format_hits(found)) + return summary = summarize(batch_dir) except CellFileError as exc: print(f"✗ {exc}", file=sys.stderr) @@ -545,6 +549,9 @@ def register(repo_sub, positive_int, source_help: str, cache_help: str) -> None: report_p = repo_sub.add_parser("report", help="Per (task, arm, model) table over a batch directory") report_p.add_argument("batch_dir", help="/ of `repo run`") report_p.add_argument("--json", action="store_true", help="Print the summary as JSON") + report_p.add_argument("--hits", action="store_true", + help="Instead of the table: each (thread, finding) pair that credited a " + "dokimi:review_replay thread, by task, arm, model and rep (grade.json details.matches)") report_p.set_defaults(func=cmd_repo_report) compare_p = repo_sub.add_parser( diff --git a/dokimi/dokimi/repos/graders/review.py b/dokimi/dokimi/repos/graders/review.py index 721f2cfc8..7eeeac56f 100644 --- a/dokimi/dokimi/repos/graders/review.py +++ b/dokimi/dokimi/repos/graders/review.py @@ -41,7 +41,12 @@ to one and nearest first, so one finding credits at most one thread and one thread at most one finding. Location is the only criterion (no judge, deterministic), so ``pass`` is null: humans miss problems and raise -non-defects, and neither recall nor precision is a verdict. +non-defects, and neither recall nor precision is a verdict. ``matches`` lists +each credited pair in ``threads_hit`` order, so a grade can be checked without +the task: the thread (``thread``, ``thread_file``, ``thread_line``, +``comment``), the finding (``finding``, its index among the answer's objects; +``file`` repository-relative and ``line`` as matched; ``severity``, +``message``) and their ``distance``; strings are cut to 200 characters. ``precision_proxy`` is the share of findings within the window of any thread, a lower bound on precision. ``recall`` equals ``recall_all`` and ``decoys_flagged`` is ``[]``, so a seeds-style report applies unchanged. @@ -58,7 +63,9 @@ import json import re -VERSION = "1" # bump when a result for the same inputs changes; grade.json records it +# Bump when a result for the same inputs changes (a metric or ``pass``); grade.json records it. A field added +# to ``details`` that leaves the result as it was (``matches``) does not: a reader tells it by its presence. +VERSION = "1" SEVERE = ("blocker", "major") @@ -216,18 +223,33 @@ def _ratio(a, b): def _replay_pairs(gold, findings, window): - """``(pairs, near)``: every (distance, finding index, thread id) within ``window``, and how many - findings are near any thread.""" + """``(pairs, near)``: every (distance, finding index, thread id, thread) within ``window``, and how + many findings are near any thread.""" pairs, near = [], 0 for i, f in enumerate(findings): path, _, line = finding_fields(f) - close = [(abs(line - g["line"]), i, g["id"]) for g in gold + close = [(abs(line - g["line"]), i, g["id"], g) for g in gold if same_file(g, path) and abs(line - g["line"]) <= window] pairs += close near += bool(close) return pairs, near +CUT = 200 # characters of a comment or message a match keeps + + +def _cut(value): + return "" if value is None else str(value)[:CUT] + + +def _match(g, i, f, distance): + """One credited (thread, finding) pair of ``details.matches``: enough to check it without the task.""" + path, _, line = finding_fields(f) + return {"thread": g["id"], "thread_file": g["file"], "thread_line": g["line"], "comment": _cut(g.get("comment")), + "finding": i, "file": path, "line": line, "severity": _cut(f.get("severity")), + "message": _cut(f.get("message")), "distance": distance} + + def grade_replay(task, task_dir, answer): """``review_replay``: score-only, ``recall`` over the human threads.""" gold = _read_json(task_dir, task.get("gold", "gold.json")) @@ -237,16 +259,17 @@ def grade_replay(task, task_dir, answer): return {"pass": False, "parse_error": True, "recall": zero, "recall_all": zero, "recall_addressed": _ratio(0, sum(bool(g.get("addressed")) for g in gold))} pairs, near = _replay_pairs(gold, findings, task.get("window", 5)) - hit, used = set(), set() + hit, used, credited = set(), set(), {} # nearest first: one finding per thread, one thread per finding; ids in _id_key order, as oneDAL's min() - for _, i, gid in sorted(pairs, key=lambda p: (p[0], p[1], _id_key(p[2]))): + for dist, i, gid, g in sorted(pairs, key=lambda p: (p[0], p[1], _id_key(p[2]))): if i not in used and gid not in hit: used.add(i) hit.add(gid) + credited[gid] = _match(g, i, findings[i], dist) addressed = [g for g in gold if g.get("addressed")] recall_all = _ratio(len(hit), len(gold)) return {"pass": None, "n_findings": len(findings), "n_severe": n_severe(findings), - "threads_hit": _ids(hit), "n_threads": len(gold), + "threads_hit": _ids(hit), "matches": [credited[gid] for gid in _ids(hit)], "n_threads": len(gold), "recall_all": recall_all, "recall": recall_all, "recall_addressed": _ratio(sum(g["id"] in hit for g in addressed), len(addressed)), "recall_by_kind": _per_group(gold, "kind", hit), diff --git a/dokimi/dokimi/repos/grading.py b/dokimi/dokimi/repos/grading.py index c4d0174e6..0c0cbf942 100644 --- a/dokimi/dokimi/repos/grading.py +++ b/dokimi/dokimi/repos/grading.py @@ -226,18 +226,25 @@ def replaced_files(run_dir: Path) -> list[str]: return out -def read_grade(run_dir: Path) -> tuple[dict[str, Any] | None, list[str]]: - path = run_dir / "grade.json" +def read_grade_json(path: Path) -> tuple[Any, list[str]]: + """The JSON value of a cell's ``grade.json``, read as the host must: a link, FIFO or other non-regular file + is not read, and a file nested too deep to parse is no grade. ``(None, problems)`` when it is not one.""" try: if not stat.S_ISREG(os.lstat(path).st_mode): return None, ["grade.json is not a regular file"] # O_NOFOLLOW: a link swapped in after the lstat is not followed either with os.fdopen(os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)), "rb") as f: - doc = json.loads(f.read().decode("utf-8")) + return json.loads(f.read().decode("utf-8")), [] except FileNotFoundError: return None, ["no grade.json written"] - except (OSError, ValueError) as exc: + except (OSError, ValueError, RecursionError) as exc: return None, [f"grade.json is not JSON: {exc}"] + + +def read_grade(run_dir: Path) -> tuple[dict[str, Any] | None, list[str]]: + doc, problems = read_grade_json(run_dir / "grade.json") + if problems: + return None, problems problems = check_grade(doc) return (None, [f"grade.json is not {GRADE_SCHEMA}: {p}" for p in problems]) if problems else (doc, []) diff --git a/dokimi/dokimi/repos/report.py b/dokimi/dokimi/repos/report.py index 3244f035b..76a4385ec 100644 --- a/dokimi/dokimi/repos/report.py +++ b/dokimi/dokimi/repos/report.py @@ -31,6 +31,17 @@ A row's ``process`` holds the mean ``result.json`` ``process`` counts of its cells (:mod:`.process_counts`), printed on a line of its own under the row. + +A row's ``parse_errors`` counts its cells whose ``grade.json`` has +``metrics.parse_error`` true: the grader ran, and the agent's answer held no +findings JSON that parsed, so the run counts as failed (contract §4.5). A +cell whose ``result.json`` says ``grader_failure`` is counted there, not +here. The table notes it only when it is not 0: ``scored 0`` on a score row, +``counted as failed`` on a gate row. + +``--hits`` (:func:`load_hits`) lists instead each credited (thread, finding) +pair of the ``dokimi:review_replay`` cells, from ``grade.json`` +``details.matches``. """ from __future__ import annotations @@ -45,7 +56,7 @@ import math -from .grading import SCORE_BOUND +from .grading import SCORE_BOUND, read_grade_json from .process_counts import process_line, process_means from .runner import UNSUPPORTED_FILE, slug @@ -251,6 +262,8 @@ def summarize(batch_dir: Path) -> dict[str, Any]: if expecting: row["skill_expected"] = len(expecting) row["skill_selected_correct"] = sum(bool(u["skill_selected_correct"]) for u in expecting) + row["parse_errors"] = sum(r["grade"]["status"] != "grader_failure" # counted in grader_failures + and _metrics(batch_dir / r["cell"]).get("parse_error") is True for r in recs) rows.append(row) return {"batch": str(batch_dir), "cells": len(cells), "missing": missing, "rows": rows, "unsupported": load_unsupported(batch_dir), @@ -278,14 +291,7 @@ def format_report(summary: dict[str, Any]) -> str: turns = "-" if r["turns"] is None else f"{r['turns']:.1f}" lines.append(f"{r['task']:<{width}} {r['arm']:<{arm_w}} {r['model']:<{model_w}} {r['n']:>3} " f"{result:<22} {r['grader_failures']:>11} {r['missing']:>7} {cost:>8} {turns:>6}") - noted = ("agent_timeouts", "agent_errors", "over_budget", "cost_unknown") - notes = [f"{r[k]} {k.replace('_', ' ')}" for k in noted if r[k]] - notes += [f"{r[k]}/{r[d]} {k.replace('_', ' ')}" - for k, d in (("skill_invoked", "skill_measured"), ("skill_selected_correct", "skill_expected")) - if k in r] - if "uptake" in r: - notes.append(f"arm's additions used in {r['uptake']['used']}/{r['uptake']['cells']} cell(s)" - + unmeasured_note(r["uptake"])) + notes = _row_notes(r) if notes: lines.append(" " * (width + 2) + " " + ", ".join(notes)) lines += process_line(r.get("process"), width) @@ -301,6 +307,91 @@ def format_report(summary: dict[str, Any]) -> str: return "\n".join(lines) +def _row_notes(r: dict[str, Any]) -> list[str]: + noted = ("agent_timeouts", "agent_errors", "over_budget", "cost_unknown") + notes = [f"{r[k]} {k.replace('_', ' ')}" for k in noted if r[k]] + if r.get("parse_errors"): # the grader ran; the agent's findings JSON is what failed + counted = "scored 0" if r["kind"] == "score" else "counted as failed" + notes.append(f"{r['parse_errors']} agent answer(s) whose findings JSON did not parse ({counted})") + notes += [f"{r[k]}/{r[d]} {k.replace('_', ' ')}" + for k, d in (("skill_invoked", "skill_measured"), ("skill_selected_correct", "skill_expected")) + if k in r] + if "uptake" in r: + notes.append(f"arm's additions used in {r['uptake']['used']}/{r['uptake']['cells']} cell(s)" + + unmeasured_note(r["uptake"])) + return notes + + +def _grade_doc(cell_dir: Path) -> dict[str, Any]: + """The cell's ``grade.json``, read as the grader's host reads it; empty when it is missing, not a regular + file, too deep to parse or not a JSON object (a grader failure says why).""" + doc, _ = read_grade_json(cell_dir / "grade.json") + return doc if isinstance(doc, dict) else {} + + +def _metrics(cell_dir: Path) -> dict[str, Any]: + found = _grade_doc(cell_dir).get("metrics") + return found if isinstance(found, dict) else {} + + +REPLAY = "dokimi:review_replay" + + +def load_hits(batch_dir: Path) -> dict[str, Any]: + """``repo report --hits``: each credited pair of the batch's ``dokimi:review_replay`` cells, from their + ``grade.json`` ``details.matches``, by task, arm, model and rep. ``replay_cells`` counts the cells whose + ``grade.json`` names that grader; ``without_matches`` those graded before ``matches`` was recorded.""" + cells, _ = load_cells(batch_dir) + bad = next((r for r in cells if not isinstance(r.get("rep"), int)), None) # load_cells checks the rest of the key + if bad is not None: + raise CellFileError(f"{batch_dir / bad['cell'] / 'result.json'} has no usable `rep`") + hits: list[dict[str, Any]] = [] + replay = without = 0 + for rec in sorted(cells, key=lambda r: (r["task"], r["arm"], r["model"], r["rep"])): + grade = _grade_doc(batch_dir / rec["cell"]) + if grade.get("grader") != REPLAY: + continue + replay += 1 + if rec["grade"]["status"] == "grader_failure": # the table scores this cell 0: no hit is credited to it + continue + details = grade.get("details") + details = details if isinstance(details, dict) else {} + matches = details.get("matches") + if not isinstance(matches, list): # a parse error or a grader error credits no thread: nothing is missing + without += "threads_hit" in details + continue + hits += [{**m, "task": rec["task"], "arm": rec["arm"], "model": rec["model"], "rep": rec["rep"], + "cell": rec["cell"]} for m in matches if isinstance(m, dict)] # result.json's key wins over a match's + return {"batch": str(batch_dir), "replay_cells": replay, "without_matches": without, "hits": hits} + + +def format_hits(found: dict[str, Any], width: int = 100) -> str: + """Two lines per credited pair: the thread and its comment, then the finding that hit it.""" + if not found["replay_cells"]: + return f"{found['batch']}: no {REPLAY} cell" + lines = [f"{found['batch']}: {len(found['hits'])} credited hit(s) in {found['replay_cells']} {REPLAY} cell(s)"] + task = cell = None + for h in found["hits"]: + if h["task"] != task: + task = h["task"] + lines.append(str(task)) + if h["cell"] != cell: + cell = h["cell"] + lines.append(f" {h['arm']} {h['model']} r{h['rep']}") + lines.append(f" {h.get('thread')} {h.get('thread_file')}:{h.get('thread_line')} " + + _one_line(h.get("comment"), width)) + lines.append(f" finding {h.get('finding')} {h.get('file')}:{h.get('line')} {h.get('severity')}, " + f"{h.get('distance')} line(s) away: " + _one_line(h.get("message"), width)) + if found["without_matches"]: + lines.append(f"{found['without_matches']} cell(s) graded without hit details") + return "\n".join(lines) + + +def _one_line(text: Any, width: int) -> str: + flat = " ".join(str(text or "").split()) + return flat if len(flat) <= width else flat[:width - 3] + "..." + + def _unsupported_lines(cells: list[dict[str, Any]]) -> list[str]: by_task: dict[tuple[str, str], int] = defaultdict(int) for u in cells: diff --git a/dokimi/tests/data/cli_flags_allowlist.txt b/dokimi/tests/data/cli_flags_allowlist.txt index 6e9ed384d..be3e93b1e 100644 --- a/dokimi/tests/data/cli_flags_allowlist.txt +++ b/dokimi/tests/data/cli_flags_allowlist.txt @@ -76,6 +76,7 @@ --grounded-from-topics --groups --harnesses +--hits # repo report replay hits (#297) adds it: a view switch has no registry concept yet --image --include-disabled # novelty routing (#172) adds it: no registry concept yet; a F1b migration names it --include-resources diff --git a/dokimi/tests/test_repo_builtin_graders.py b/dokimi/tests/test_repo_builtin_graders.py index aaa0f3b41..e877d88c8 100644 --- a/dokimi/tests/test_repo_builtin_graders.py +++ b/dokimi/tests/test_repo_builtin_graders.py @@ -204,6 +204,34 @@ def test_replay_is_one_to_one_nearest_first_within_the_window(tmp_path): "recall_addressed": 0.0} +def test_replay_matches_record_each_credited_pair_nearest_first(tmp_path): + d = _task_dir(tmp_path, "replay_mul") + # object 1 (line 4) is near both threads: t_doc at 0 takes it, so t_op goes to object 2 at 1, not object 0 at 4 + findings = ["not an object", _f(9, message="far"), _f(4, severity="major", message="near both"), + _f("6", file="/eval/run/workspace/src/calc.py", message="m" * 300)] + g = review.grade_replay({}, d, fence(findings)) + assert g["threads_hit"] == ["t_doc", "t_op"] and g["recall"] == 1.0 and g["precision_proxy"] == 1.0 + assert g["matches"] == [ + {"thread": "t_doc", "thread_file": "src/calc.py", "thread_line": 4, "comment": "docstring?", + "finding": 1, "file": "src/calc.py", "line": 4, "severity": "major", "message": "near both", "distance": 0}, + {"thread": "t_op", "thread_file": "src/calc.py", "thread_line": 5, + "comment": "mul adds instead of multiplying\nsecond line", "finding": 2, "file": "src/calc.py", "line": 6, + "severity": "minor", "message": "m" * 200, "distance": 1}] + # the entry script keeps them under details, so grade.json alone is enough to check a hit + doc = json.loads((_script(tmp_path, "grade", "review_replay", "replay_mul", fence(findings)) / "grade.json") + .read_text()) + assert doc["details"]["matches"] == g["matches"] and "matches" not in doc["metrics"] and check_grade(doc) == [] + assert review.grade_replay({}, d, fence([_f(30)]))["matches"] == [] + # a thread with no comment and a finding with no severity or message keep "", not "None" + write(tmp_path, "bare/gold.json", json.dumps([{"id": "t", "file": "src/calc.py", "line": 5, "kind": "docs"}])) + assert review.grade_replay({}, tmp_path / "bare", fence([{"file": "src/calc.py", "line": 5}]))["matches"] == [ + {"thread": "t", "thread_file": "src/calc.py", "thread_line": 5, "comment": "", + "finding": 0, "file": "src/calc.py", "line": 5, "severity": "", "message": "", "distance": 0}] + # t_op is credited first (distance 0), but matches follow threads_hit + later = review.grade_replay({}, d, fence([_f(5), _f(3)]))["matches"] + assert [(m["thread"], m["finding"], m["distance"]) for m in later] == [("t_doc", 1, 1), ("t_op", 0, 0)] + + @pytest.mark.outcome("cjm-builtin-review-graders") def test_matching_rules_as_the_contract_states_them(tmp_path): """§4.5: same file only, kw case-insensitive, replay within ``window`` (default 5) inclusive.""" diff --git a/dokimi/tests/test_repo_report_hits.py b/dokimi/tests/test_repo_report_hits.py new file mode 100644 index 000000000..50acc93ef --- /dev/null +++ b/dokimi/tests/test_repo_report_hits.py @@ -0,0 +1,234 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""`repo report --hits` (each credited replay pair) and the parse-error count of `repo report`, over hand-written +batches graded by the real ``dokimi:review_replay`` grader.""" + +from __future__ import annotations + +import contextlib +import json +import os +import threading +from pathlib import Path + +import pytest + +from dokimi.cli import build_parser +from dokimi.repos.graders import main, review +from dokimi.repos.report import format_hits, format_report, load_hits, summarize +from tests.test_repo_builtin_graders import GOLD, _f, fence +from tests.test_repo_compare import _cell + +REPLAY = "dokimi:review_replay" + + +def _graded(tmp: Path, answer: str, *, old: bool = False, grader: str = REPLAY) -> dict: + """``grade.json`` as the entry script writes it for ``answer`` against ``GOLD``; ``old``: as graded before + ``details.matches`` existed.""" + (tmp / "gold").mkdir(exist_ok=True) + (tmp / "gold" / "gold.json").write_text(json.dumps(GOLD)) + metrics, details = main.split_result(review.grade_replay({}, tmp / "gold", answer)) + if old: + details.pop("matches") + return {"schema": "repo_grade.v1", "grader": grader, "pass": None, "metrics": metrics, + "details": {**details, "grader_version": review.VERSION}, "errors": []} + + +def _put(batch: Path, task: str, arm: str, rep: int, grade: dict) -> None: + _cell(batch, task, arm, rep, score=grade["metrics"].get("recall") or 0.0) + (batch / f"{task}__{arm.replace(':', '-')}__m-1__r{rep}" / "grade.json").write_text(json.dumps(grade)) + + +def _batch(tmp: Path) -> Path: + batch = tmp / "b1" + todo = _f(6, message="The TODO is repeated five\ntimes and has no issue URL") + _put(batch, "pr3098-t2", "guidance:none", 1, _graded(tmp, fence([todo]))) + _put(batch, "pr3098-t2", "guidance:none", 2, _graded(tmp, fence([_f(4, message="x" * 150), _f(5, message="mul adds")]))) + _put(batch, "pr3098-t2", "guidance:raw", 1, _graded(tmp, fence([_f(5)]), old=True)) + # the parse error the dpnp run hit: a "line" JSON cannot read, so the whole fence is unreadable + unparsed = '```json\n[{"file": "dpnp/dpnp_iface_indexing.py", "line": 1:1, "message": "m"}]\n```' + _put(batch, "pr3014-t1", "guidance:none", 2, _graded(tmp, unparsed)) + _put(batch, "seeds", "guidance:none", 1, _graded(tmp, fence([_f(5)]), grader="dokimi:review_seeds")) + return batch + + +def test_hits_list_each_credited_pair_by_task_arm_and_rep(tmp_path): + batch = _batch(tmp_path) + # by directory name r10 sorts before r2; the hits follow rep order + _put(batch, "pr3098-t2", "guidance:none", 10, _graded(tmp_path, fence([_f(5)]))) + found = load_hits(batch) + assert (found["replay_cells"], found["without_matches"]) == (5, 1) # the parse error has no hit to miss + assert [(h["task"], h["arm"], h["rep"], h["thread"], h["finding"], h["distance"]) for h in found["hits"]] == [ + ("pr3098-t2", "guidance:none", 1, "t_op", 0, 1), + ("pr3098-t2", "guidance:none", 2, "t_doc", 0, 0), ("pr3098-t2", "guidance:none", 2, "t_op", 1, 0), + ("pr3098-t2", "guidance:none", 10, "t_op", 0, 0)] + assert found["hits"][0] == { + "task": "pr3098-t2", "arm": "guidance:none", "model": "m-1", "rep": 1, "cell": "pr3098-t2__guidance-none__m-1__r1", + "thread": "t_op", "thread_file": "src/calc.py", "thread_line": 5, + "comment": "mul adds instead of multiplying\nsecond line", "finding": 0, "file": "src/calc.py", "line": 6, + "severity": "minor", "message": "The TODO is repeated five\ntimes and has no issue URL", "distance": 1} + assert format_hits(found).splitlines() == [ + f"{batch}: 4 credited hit(s) in 5 {REPLAY} cell(s)", + "pr3098-t2", + " guidance:none m-1 r1", + " t_op src/calc.py:5 mul adds instead of multiplying second line", + " finding 0 src/calc.py:6 minor, 1 line(s) away: The TODO is repeated five times and has no issue URL", + " guidance:none m-1 r2", + " t_doc src/calc.py:4 docstring?", + " finding 0 src/calc.py:4 minor, 0 line(s) away: " + "x" * 97 + "...", + " t_op src/calc.py:5 mul adds instead of multiplying second line", + " finding 1 src/calc.py:5 minor, 0 line(s) away: mul adds", + " guidance:none m-1 r10", + " t_op src/calc.py:5 mul adds instead of multiplying second line", + " finding 0 src/calc.py:5 minor, 0 line(s) away: ", + "1 cell(s) graded without hit details"] + + +def _cli(capsys, *argv: str) -> str: + args = build_parser().parse_args(["repo", "report", *argv]) + args.func(args) + return capsys.readouterr().out + + +def test_report_hits_flag_prints_the_pairs_and_the_default_report_stays_the_table(tmp_path, capsys): + batch = _batch(tmp_path) + text = _cli(capsys, str(batch), "--hits") + assert text.startswith(f"{batch}: 3 credited hit(s)") and "finding 0 src/calc.py:6" in text + records = json.loads(_cli(capsys, str(batch), "--hits", "--json")) + assert [(h["cell"], h["thread"]) for h in records["hits"]][:2] == [ + ("pr3098-t2__guidance-none__m-1__r1", "t_op"), ("pr3098-t2__guidance-none__m-1__r2", "t_doc")] + assert records["without_matches"] == 1 + table = _cli(capsys, str(batch)) + assert table.startswith(f"{batch}: 5 finished cells") and "credited" not in table and "finding 0" not in table + + +def test_a_match_key_named_like_the_cell_key_does_not_move_the_hit(tmp_path, capsys): + batch = tmp_path / "b1" + grade = _graded(tmp_path, fence([_f(5)])) + grade["details"]["matches"][0].update(task="other-task", arm="guidance:raw", model="m-9", rep=99, cell="elsewhere") + _put(batch, "a", "guidance:none", 1, grade) + records = json.loads(_cli(capsys, str(batch), "--hits", "--json")) + assert [(h["task"], h["arm"], h["model"], h["rep"], h["cell"], h["thread"]) for h in records["hits"]] == [ + ("a", "guidance:none", "m-1", 1, "a__guidance-none__m-1__r1", "t_op")] + assert _cli(capsys, str(batch), "--hits").splitlines()[1:3] == ["a", " guidance:none m-1 r1"] + + +def test_hits_on_a_batch_with_no_replay_cell_say_so(tmp_path): + _cell(tmp_path, "a", "guidance:none", 1) + found = load_hits(tmp_path) + assert found["hits"] == [] and found["replay_cells"] == 0 + assert format_hits(found) == f"{tmp_path}: no {REPLAY} cell" + + +def test_report_counts_answers_that_did_not_parse_per_row(tmp_path): + batch = _batch(tmp_path) + rows = {(r["task"], r["arm"]): r for r in summarize(batch)["rows"]} + unparsed = rows[("pr3014-t1", "guidance:none")] + assert unparsed["parse_errors"] == 1 and unparsed["grader_failures"] == 0 # the agent failed, not the grader + assert [r["parse_errors"] for k, r in rows.items() if k[0] != "pr3014-t1"] == [0, 0, 0] + lines = format_report(summarize(batch)).splitlines() + at = next(i for i, line in enumerate(lines) if line.startswith("pr3014-t1")) + assert lines[at + 1].strip() == "1 agent answer(s) whose findings JSON did not parse (scored 0)" # in the mean + assert sum("did not parse" in line for line in lines) == 1 + + +def test_the_parse_error_note_of_a_gate_row_says_counted_as_failed(tmp_path): + _cell(tmp_path, "gate", "guidance:none", 1, success=False, metrics={"parse_error": True}) + _cell(tmp_path, "gate", "guidance:none", 2) + lines = format_report(summarize(tmp_path)).splitlines() + at = next(i for i, line in enumerate(lines) if line.startswith("gate")) + assert "1/2 pass" in lines[at] + assert lines[at + 1].strip() == "1 agent answer(s) whose findings JSON did not parse (counted as failed)" + + +def test_hits_and_report_skip_unreadable_grades_and_list_a_match_without_text(tmp_path): + batch = tmp_path / "b1" + bad = {1: "{not json", 2: "[]", + 3: json.dumps({"grader": REPLAY, "metrics": {"parse_error": "yes"}, "details": []}), + 4: json.dumps({"grader": REPLAY, "metrics": [], "details": {"threads_hit": ["t"], + "matches": [None, "t", {"thread": "t", "finding": 0, "distance": 0}]}})} + for rep, text in bad.items(): + _cell(batch, "a", "guidance:none", rep) + (batch / f"a__guidance-none__m-1__r{rep}" / "grade.json").write_text(text) + found = load_hits(batch) + assert (found["replay_cells"], found["without_matches"]) == (2, 0) # 1 and 2 name no grader + assert [(h["rep"], h["thread"]) for h in found["hits"]] == [(4, "t")] + assert format_hits(found).splitlines()[-2:] == [ + " t None:None ", " finding 0 None:None None, 0 line(s) away: "] + assert [r["parse_errors"] for r in summarize(batch)["rows"]] == [0] # a non-boolean parse_error is not one + + +def test_hits_refuse_a_cell_with_no_rep_as_the_table_refuses_a_malformed_cell(tmp_path, capsys): + batch = _batch(tmp_path) + path = batch / "pr3098-t2__guidance-none__m-1__r1" / "result.json" + rec = json.loads(path.read_text()) + del rec["rep"] # the table never reads it; --hits sorts by it + path.write_text(json.dumps(rec)) + with pytest.raises(SystemExit) as stopped: + _cli(capsys, str(batch), "--hits") + assert stopped.value.code == 1 + assert capsys.readouterr().err.strip() == f"✗ {path} has no usable `rep`" + + +def test_a_grade_nested_too_deep_or_linked_out_of_the_batch_is_no_grade(tmp_path): + batch = tmp_path / "b1" + outside = tmp_path / "outside.json" # a link out of the batch is not followed + outside.write_text(json.dumps({"grader": REPLAY, "metrics": {"parse_error": True}, + "details": {"matches": [{"thread": "outside", "finding": 0}]}})) + _put(batch, "a", "guidance:none", 1, _graded(tmp_path, fence([_f(5)]))) + _cell(batch, "a", "guidance:none", 2, score=0.0) + _cell(batch, "a", "guidance:none", 3, score=0.0) + (batch / "a__guidance-none__m-1__r2" / "grade.json").write_text("[" * 200000) # RecursionError to parse + link = batch / "a__guidance-none__m-1__r3" / "grade.json" + link.unlink() + link.symlink_to(outside) + assert summarize(batch)["rows"][0]["parse_errors"] == 0 + found = load_hits(batch) + assert (found["replay_cells"], [h["rep"] for h in found["hits"]]) == (1, [1]) + + +@pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="needs os.mkfifo") +def test_a_grade_that_is_a_fifo_is_no_grade_and_does_not_block_the_report(tmp_path): + batch = tmp_path / "b1" + _put(batch, "a", "guidance:none", 1, _graded(tmp_path, fence([_f(5)]))) + _cell(batch, "a", "guidance:none", 2, score=0.0) + fifo = batch / "a__guidance-none__m-1__r2" / "grade.json" + fifo.unlink() + os.mkfifo(fifo) + out: dict = {} + reader = threading.Thread(target=lambda: out.update(summary=summarize(batch), hits=load_hits(batch)), + daemon=True) # a read of a FIFO waits for a writer: the thread is kept apart + reader.start() + reader.join(10) + blocked = reader.is_alive() + for _ in range(30): # open the write end so each blocked read ends (the table's, then the one of --hits) + if not reader.is_alive(): + break + with contextlib.suppress(OSError): # ENXIO: no read of the FIFO is waiting right now + os.close(os.open(fifo, os.O_WRONLY | os.O_NONBLOCK)) + reader.join(1) + assert not blocked, "a grade.json that is a FIFO blocked the report" + assert out["summary"]["rows"][0]["parse_errors"] == 0 + assert (out["hits"]["replay_cells"], [h["rep"] for h in out["hits"]["hits"]]) == (1, [1]) + + +def test_hits_credit_no_match_of_a_cell_the_table_scores_as_a_grader_failure(tmp_path): + batch = tmp_path / "b1" # written, then the run failed: the table counts the cell as a grader failure + _cell(batch, "a", "guidance:none", 1, failure=True) + (batch / "a__guidance-none__m-1__r1" / "grade.json").write_text(json.dumps(_graded(tmp_path, fence([_f(5)])))) + _put(batch, "a", "guidance:none", 2, _graded(tmp_path, fence([_f(5)]))) + assert [r["grader_failures"] for r in summarize(batch)["rows"]] == [1] + found = load_hits(batch) + assert (found["replay_cells"], [h["rep"] for h in found["hits"]]) == (2, [2]) + + +def test_parse_errors_leave_out_a_cell_the_table_counts_as_a_grader_failure(tmp_path): + batch = tmp_path / "b1" + unparsed = _graded(tmp_path, "no findings JSON here") + assert unparsed["metrics"]["parse_error"] is True + _cell(batch, "a", "guidance:none", 1, failure=True, score=0.0) # the grade was written, then the run failed + (batch / "a__guidance-none__m-1__r1" / "grade.json").write_text(json.dumps(unparsed)) + _put(batch, "a", "guidance:none", 2, unparsed) + row = summarize(batch)["rows"][0] + assert (row["grader_failures"], row["parse_errors"]) == (1, 1)