[antonwolfy]

Summary

Follow-up to the review of #217 (catalog trigger-probe), which left two Nits:

• a trigger set with no labelled query runs nothing and exits 0
  (review);
• the not_listed guard is skipped without a word when no run reports any listed skill
  (review).

On main, with a two-skill catalog (sum-fast, io-read) and a stub claude that prints a
stream-json transcript:

Input	main	This PR
--trigger-set whose questions have no expected_skill	writes probe.json with "answers": [], prints "summary": {} and wrote probe.json, exit 0	Error: --trigger-set has no labelled queries: ts.json, exit 2, no probe.json, no run
same, with --target sum-fast	as above, exit 0	as above, the message ends (read with --target 'sum-fast')
{"queries": []} (empty skill-reach set)	as above, exit 0	as above, exit 2
init event "skills": [] in every run	not_listed: [], recall 0.5 scored, exit 0, nothing on stderr	the same JSON and exit 0, plus warning: Claude Code listed no skills in any run; not_listed was not checked on stderr
init event lists both skills	not_listed: [], exit 0	unchanged, no warning

--target does not filter the queries of any format: it labels the rows of a skill-creator set,
where every row is labelled. So the "--target matches none" case the review names cannot occur
at this head. The error still names --target when it was given, for a set whose labels are
missing anyway.

Changes

• dokimi/dokimi/commands/catalog_probe.py, _labelled_queries: when no labelled query is left
  after the filter, raise InputError naming --trigger-set, its path and --target when given.
  cli.main prints it as Error: ... and exits 2, as it does the command's other input errors.
  This happens before <out>.partial or --out is written and before any claude run.
• Same file, cmd_catalog_trigger_probe: when at least one run happened and no run listed any
  skill, print one warning line to stderr. not_listed stays [] in the output and the exit code
  is unchanged. Nothing is printed when there were no runs.
• dokimi/docs/evaluating-treatments.md, the catalog trigger-probe paragraph: one sentence for
  each behaviour.
• dokimi/tests/test_trigger_probe.py: test_cli_exits_2_on_a_set_with_no_labelled_query_before_any_run
  (three cases: unlabelled questions, the same with --target, an empty skill-reach set). It goes
  through cli.main and checks exit 2, the message, that no probe.json or .partial file was
  written and that probe_trigger_set was never called.
  test_cli_warns_when_no_run_listed_any_skill checks that the warning appears for "skills": []
  with not_listed == [], and that no warning appears for a run that lists skills.
• Changelog fragment (Fixed).

Risk

A trigger set with no labelled query was accepted and produced an empty output with exit 0. It
now exits 2 and writes nothing. A script that relied on that empty probe.json will now see the
error. The new stderr line appears only when runs happened and none listed any skill. The output
JSON (no schema change) and stdout are unchanged in every case.

Testing

On a Linux review host, at 9ba04bf (on main 3a1f287):

• make ci-local: 28 passed, 0 failed, 7 skipped (agent-quality.yml:external-catalog-smoke,
  which needs docker and network; ci.yml:test-latest, docs.yml:deploy and
  pip-audit.yml:report, whose job conditions are false on a PR; cli-interface.yml "Package the
  result", whose expression ci-local cannot evaluate; conda-forge.yml:conda-forge-deps, with no
  micromamba; semgrep secrets, which runs on the PR). 9302 tests passed, 4 skipped; changed-line
  coverage 100% (4 of 4 lines). An earlier run on a busy host failed one wall-clock test,
  tests/test_reference_checks.py::test_the_regex_limit_is_shared_by_every_pattern_of_one_check
  (2.02 s against a 1.5 s bound). That module is not touched here, and the test passed alone and
  in the rerun above.
• tests/test_trigger_probe.py, tests/test_docs_code_refs.py, tests/test_growth_ratchet.py,
  tests/test_audit_test_quality.py, tests/test_import_architecture.py: 311 passed. ruff check
  passes on the changed files, and so does mypy on catalog_probe.py.
• On main's code the new test file fails 4 of its 24 tests (the three empty-set cases do not
  raise SystemExit, and the warning is missing). At this head all 24 pass.
• Mutants, each run against the committed tree:
  • dropping the empty-set check (if not queries: → if False:) fails all three
    test_cli_exits_2_on_a_set_with_no_labelled_query_before_any_run cases;
  • dropping the warning (if runs and not listed: → if False:) fails
    test_cli_warns_when_no_run_listed_any_skill.
• The table above was run through cli.main with a stub claude script, at this head and on
  main.
• Not tested: the "runs happened" half of the warning condition. With the new empty-set check, the
  command always has at least one run, so the no-run case cannot be reached from the CLI.

Revert

git revert of the one commit. It leaves no stored records and no schema versions behind.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 9ba04bf58615, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 3 of 80 turns, 0.2 of 40 minutes, 46k in / 857 out tokens.

The change makes catalog trigger-probe exit 2 with an InputError when no labelled query is left, before any run or output file. It also prints a stderr warning when runs happened but none listed a skill. I found no defects. cli.main already turns InputError into Error: ... and exit 2, and the check sits before the thread pool and the first write. The tests cover both behaviours and go through cli.main.

No issues found.

For a human reviewer

• Exit 2 instead of an empty probe.json with exit 0 is a behaviour change for any script that relied on the old output.

[vchamarthi]

No blocking findings. Both behaviours work as the description says. Ready to approve once the remaining GitHub checks on this head (the py3.11 and py3.14 test legs, mypy and CodeQL) finish green.

Checked on 9ba04bf on a Linux host:

• make ci-local: 27 passed, 7 skipped, 1 failed. The failure is tests/test_process.py::test_sighup_kills_the_live_groups_too, a TimeoutExpired during the full-suite run. This PR does not touch that file, and the test passed 3 of 3 alone on this head and 3 of 3 on main, so it looks like a timing flake under load. Three reruns do not prove that.
• The 24 tests in tests/test_trigger_probe.py pass.
• Mutants, run against those tests: replacing the empty-set check with if False: fails all three test_cli_exits_2_on_a_set_with_no_labelled_query_before_any_run cases; replacing if runs and not listed: with if False: fails test_cli_warns_when_no_run_listed_any_skill.
• The description's table through cli.py catalog trigger-probe with a stub claude, head against main: a set with no expected_skill and a {"queries": []} set both exit 2 with Error: --trigger-set has no labelled queries: <path> and write no --out file (main: exit 0 and a file). An init event with "skills": [] gives not_listed: [], exit 0 and the new warning (main: no warning). An init event listing both skills gives no warning on either tree.

The empty-set check in _labelled_queries runs before <out>.partial is written and before any claude run, and cli.main turns the InputError into Error: ... with exit 2 (cli.py:116).

Two notes, neither blocking:

• catalog_probe.py:68: the runs half of runs and not listed cannot be false once a labelled query exists, as the description says. Removing it leaves all 24 tests passing, so the guard could go.
• catalog_probe.py:113: an unreadable or malformed --trigger-set still exits 1 (SystemExit(f"--trigger-set: {exc}")), while the new empty-set case exits 2. That predates this PR; it fits the shared InputError follow-up if there is one.

Coverage: read the whole diff and ran everything above. A probe against a real Claude Code run was not done: the changed code acts on the parsed transcript, which the stub produces in the same stream-json shape the existing tests use.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.81%. Comparing base (3a1f287) to head (9ba04bf).
⚠️ Report is 3 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #267      +/-   ##
==========================================
- Coverage   92.83%   92.81%   -0.02%     
==========================================
  Files         262      262              
  Lines       33016    33020       +4     
  Branches     5510     5512       +2     
==========================================
  Hits        30649    30649              
- Misses       1571     1574       +3     
- Partials      796      797       +1     

Files with missing lines	Coverage Δ
dokimi/dokimi/commands/catalog_probe.py	89.24% <100.00%> (+0.48%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.