[antonwolfy]

Summary

Follow-up to the review of #172, which found that the novelty routing --all-vs-all tests leave some outputs unpinned (review). The gold != name filter on sibling_queries that the review also named is already pinned on main (test_an_attractor_with_queries_of_its_own_is_rated_over_the_other_skills_queries_only). The other three are pinned here.

On main, each change below still passes all 298 tests of tests/test_novelty_routing_all_vs_all.py, test_novelty_cli.py, test_novelty_routing.py, test_novelty_routing_field.py and test_novelty_routing_cost.py:

Change on main	Where	Tests on main
seed=args.seed -> seed=0	_cmd_routing_all_vs_all, commands/novelty.py:372	298 passed
section["attractors"][:5] -> [:3] (or [:6])	_print_attractors, commands/novelty.py:456	298 passed
section = result[arm] -> result["lexical"], or -> result["model"] when there is a model arm: one arm's rows under both labels	_print_attractors, commands/novelty.py:455	298 passed
attractor sort by name reversed, or name key dropped	_attractor_table, novelty/all_vs_all.py:246	298 passed
from sort with the name key dropped	_attractor_table, novelty/all_vs_all.py:243	298 passed

Reversing the from name order at :243 is already caught on main, by test_the_model_arm_attractor_is_the_skill_the_selector_always_picks (two golds tied 2–2). Dropping the name key there is not.

Changes

• tests/test_novelty_routing_all_vs_all.py, test_the_seed_on_the_command_line_is_the_seed_the_saved_model_arm_records: a CLI run with --model m --seed 5 and the selector mocked; the saved result.model records model == "m" and seed == 5. The seed is in the payload only in the model arm; the lexical arm takes no seed.
• test_console_prints_the_top_five_attractors_of_each_arm_by_count: six skills each take some of gold's queries, six down to one (21 queries). The lexical arm gives each query to the skill whose card holds its word. The mocked selector gives it to the mirror skill's card instead (alpha's word to foxtrot's card, foxtrot's to alpha's), so the two arms rank the skills in opposite orders. The console prints exactly these ten attractor lines, five per arm, in count order, and each arm leaves out its sixth skill:
  lexical: foxtrot picked for 6 of 21 sibling queries (from gold) ... lexical: bravo picked for 2 of 21 sibling queries (from gold), then model: alpha picked for 6 of 21 sibling queries (from gold) ... model: echo picked for 2 of 21 sibling queries (from gold). _word_client takes an optional swap of word to word for this.
• test_attractors_tied_on_count_and_golds_tied_in_from_are_in_name_order: zeta and alpha are each picked for 2 sibling queries, with zeta first in the field. zeta takes one query each from mu and beta, with mu first in the queries file, and zeta also has a query of its own. The test asserts the exact rows: alpha (5 sibling queries, rate 0.4, from mu) before zeta (4, rate 0.5, from beta, mu).
• Tests only, nothing under dokimi/dokimi/, so no changelog fragment.

Risk

None: no code changes, only three new tests and their selector helper.

Testing

On a Linux review host, at 7737770 (on main 0a63f4e):

• make ci-local: 25 passed, 3 failed, 7 skipped (the external-catalog smoke, which needs Docker; test-latest, docs.yml:deploy and pip-audit.yml:report, whose job conditions are false on a PR; the cli-surface "Package the result" step; conda-forge-deps, with micromamba not installed; semgrep secrets).
  10181 tests passed, 4 skipped; changed-line coverage: 0 package lines changed.
  Install gitleaks and Scan commits failed because the review host's proxy answers the gitleaks release download with 504. The same gitleaks 8.30.1, downloaded through another proxy and checked against ci.yml's GITLEAKS_SHA256, ran the step's PR scan (--log-opts="--diff-merges=remerge origin/main..HEAD"): 2 commits, no leaks.
  The other FAIL is pip-audit.yml:pip-audit: 10 advisories against filelock==3.32.7 in requirements.lock.txt and requirements-ci.lock.txt, which this PR does not touch. The workflow's paths: filter runs it only on a PR that edits a lock, a requirements file or pyproject.toml, so it does not run on this PR; the fix is a lock PR.
• ruff check is clean on the touched file.
• Mutants, each in a copy of 7737770, against tests/test_novelty_routing_all_vs_all.py (69 tests):
  • seed=0 at commands/novelty.py:372: the seed test fails, alone.
  • [:3] and [:6] at commands/novelty.py:456: the top-five test fails, alone.
  • section = result["lexical"], and section = result["model"] if "model" in result else result[arm], at commands/novelty.py:455: the top-five test fails, alone. At 8bc08b7, where both arms printed the same five lines, both passed all 69 tests.
  • :246 reversed ((wrongly_picked, skill), reverse=True) and name key dropped: the tie-break test fails, alone.
  • :243 reversed: the tie-break test fails, as does the existing model-arm test. :243 name key dropped: the tie-break test fails, alone.
  • Each of these mutants passes all 298 novelty tests on main (table above).

Revert

git revert of the squash-merged commit. It leaves nothing behind.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 8bc08b711909, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 6 of 80 turns, 0.7 of 40 minutes, 128k in / 2k out tokens.

This PR adds three tests to test_novelty_routing_all_vs_all.py. They pin the CLI --seed reaching the saved model arm, the top-five console cap per arm, and the name tie-break on attractor rows and on from. I found no defects. I could not run pytest because the container has no network. I traced each test by hand against _attractor_table, _print_attractors and _cmd_routing_all_vs_all, and each expected value matches the code, so the tests should pass and should fail under the mutants the description lists.

No issues found.

Not verified

I did not run the new tests or the mutants, because pytest could not be installed offline. I relied on tracing the code by hand and on the author's reported mutant results.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.26%. Comparing base (0a63f4e) to head (8bc08b7).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #287   +/-   ##
=======================================
  Coverage   93.26%   93.26%           
=======================================
  Files         272      272           
  Lines       35117    35117           
  Branches     5964     5964           
=======================================
  Hits        32752    32752           
  Misses       1547     1547           
  Partials      818      818           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vchamarthi]

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Review details

• Commit reviewed: 7737770 against merge base 0a63f4e
• Coverage: 1 files in 1 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 80k tokens (55k from cache), 30s
• Tool calls: 7 (5 code_search, 2 file_read)
• dokimi/tests/test_novelty_routing_all_vs_all.py: 1 file