[napetrov]

Summary

This PR adds a mutation-testing pilot for the measurement core: the paired statistics, the gates and the response-cache key. In this code a surviving mutant means a wrong number would ship and no test would catch it. The pilot uses mutmut 3.8.0, with an exact pin, hash-locked in its own requirements-mutation.lock.txt that only mutation.yml installs (and pip-audit.yml audits). A weekly or on-demand workflow writes the score and the survivors to the job summary. It never fails a build.

 .github/workflows/
+└── mutation.yml          # weekly + workflow_dispatch, contents: read, non-blocking
 dokimi/
 ├── pyproject.toml        # + [tool.mutmut]: only_mutate = the 7 pilot modules,
 │                         #   test selection = their 14 test files
 ├── requirements-mutation.txt       # mutmut==3.8.0
├── requirements-mutation.lock.txt  # hash-locked closure, constrained to requirements.lock.txt
 ├── CONTRIBUTING.md       # + "Mutation pilot" paragraph
 └── tests/                # + kills for the survivors that were real gaps

Mutant counts below are from the first local run; later review rounds added tests that kill further survivors.

Module	Killed before	Killed after
eval.stats	77.2%	87.7%
eval.binary_stats	92.0%	95.8%
gate.regression	65.6%	100.0%
gate.score_gate	88.0%	96.0%
gate.critical_bands	72.4%	95.5%
gate.trust_gate	64.1%	97.5%
caching	75.5%	87.1%
all 2087 mutants	75.3%	92.0%

The new tests pin behaviour that no existing test checked:

• Regression gate: every default threshold at its exact value. Spec thresholds replace the defaults.
• Critical bands: each condition reads its own summary key, and "below" is strict.
• Trust gate:
  • a hand-computed value, threshold and severity for every check
  • each threshold is inclusive
  • the rendered block, whole
• Paired test: the Wilcoxon veto on a significant t-test, and MIN_PAIRS_FOR_TEST.
• Bootstrap CI: the two-sided alpha and the n_resamples=1 edge.
• Arm significance: confidence passes through, and holm and bh give different results.
• McNemar (binary_stats): pseudo_replicated.
• Cache key: it separates role, sampling parameters and image blocks, and a pinned hash keeps stored entries findable.

What is still left is mostly equivalent mutants:

• scipy lowercases "ZSPLIT".
• zip(strict=) after a length check.
• caps that never bind, such as min(2.0, p) with p ≤ 1.
• <= vs < that differs only at p == 0.05 exactly.
• message text and display-rounding digits.

gate/novelty_gate.py is out of scope. mutmut 3 copies the whole 440-line evaluate_novelty once per mutant, and generation alone grew past 26 MB. A diff-only mode for PRs is a follow-up.

Evidence

• Before: on main, the mutant dokimi.gate.regression.x__classify_regression__mutmut_7 survives:

  -    if drop >= critical_threshold:
  +    if drop > critical_threshold:

  tests/test_regression.py from main against that mutant gives 6 passed. A drop of exactly the 0.08 default is reported as WARN instead of CRITICAL, and nothing notices.

  After: the same mutant against this branch's tests:

  FAILED test_default_thresholds_apply_at_their_exact_values[score--0.08-CRITICAL]
  FAILED test_default_thresholds_apply_at_their_exact_values[coverage--0.12-CRITICAL]
  

• Whole pilot: the local mutmut run --max-children 16 takes about 1 minute after the first mutant generation. Before: 1566 killed, 514 survived. After: 1913 killed, 167 survived.

• Workflow summary: the step-summary script was run on the local results and produced the table above.

• Checks:

  • pytest -n 16 --outcomes: 6986 passed, 2 skipped
  • ruff check .: clean
  • mypy: no issues in 222 files
  • zizmor --offline --min-severity medium .github/workflows: no findings
  • scripts/audit_test_quality.py: no new flags
  • tests/test_dependency_locks.py: passes
  • scripts/check_docs_review_triggers.py: no page to review

Merge Danger

Door: two-way. The workflow is new and never runs on PRs or pushes. Removing it, together with the [tool.mutmut] table, undoes the change.

Blast Radius: tests

• No package code changes.
• mutmut and its dependencies (libcst, textual, setproctitle and a few more) live only in requirements-mutation.lock.txt; the tools lock other jobs install is unchanged.
• The step summary flags a run cut short (mutants not checked, or a module with nothing scored).
• test_the_key_of_a_known_request_does_not_change pins the response-cache key format. A future change to the key has to update that hash on purpose, because a new key format orphans every stored entry.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

🤖 Generated with Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.51%. Comparing base (f0f95a2) to head (7a0dc61).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #178      +/-   ##
==========================================
+ Coverage   93.48%   93.51%   +0.02%     
==========================================
  Files         272      272              
  Lines       35363    35363              
  Branches     6001     6001              
==========================================
+ Hits        33060    33068       +8     
+ Misses       1489     1485       -4     
+ Partials      814      810       -4     

see 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

The tests are real and they kill what the PR says they kill. I'd fix the gaps below before merge, because the PR's point is that a surviving mutant is a wrong number that ships, and several mutants of the same kind survive the new tests.

What I ran, on a Linux review host at 432f037:

• make ci-local: 27 passed, 1 failed, 7 skipped; 7,175 tests passed. The one failure, test_repo_environment.py::test_unbalanced_quote_in_hook_completes_quickly (1.7 s against a 1 s limit), passes on its own (513 passed in that file). Python 3.13, so the 3.11 leg is not covered here.
• Hand-made mutants of the seven pilot modules (about 310), run against the new test files and against the 14-file selection in [tool.mutmut]. The drop >= critical_threshold -> > mutant from the description is killed. The survivors are in the inline comments.
• The step-summary script from mutation.yml, run on made-up mutmut results lines (function and class-method names, no tests, timeout, suspicious): the module grouping and the table are right. zizmor --offline on the workflow: no findings.
• I did not run mutmut run itself, so the 75.3% -> 92.0% figures are not checked.

Findings that cannot be anchored inline:

1. The mutation-only tool now lands in the shared tools lock. requirements-ci.lock.txt gains 352 lines for mutmut: libcst, textual, linkify-it-py, mdit-py-plugins, pyyaml-ft, setproctitle and others. ci.yml, release.yml, pip-audit.yml and the eval image install that whole lock, so each now installs and audits packages only the weekly workflow uses. An advisory on textual or libcst would fail the daily pip-audit. Suggestion: a separate requirements-mutation.lock.txt that only mutation.yml installs. If the shared lock is intended, say so in the PR description.

Checked, no issues

• Gate modules: all 22 mutants of critical_bands.py and all 11 of score_gate.py I tried are killed (per-condition keys, strict <, defaults, guards).
• Cache key: the pinned-hash test hashes an explicit JSON literal, not a stored digest, so it is stable across Python versions and platforms. Dropping the role, model, extra or sort_keys, or changing ensure_ascii, fails it.
• Config: all 14 files in the test selection exist, the only_mutate entries end in .py as mutmut 3 requires, the lock entry for mutmut==3.8.0 is hashed, and tests/test_dependency_locks.py passes.
• Tests: all four gate test files pass in other orders and with different pytest-randomly seeds; they use no I/O.

[antonwolfy]

I pushed a commit per nit, up to d57ed0e, which fix the nit from my review:

• d57ed0e: dokimi/CONTRIBUTING.md:147: now names the score, regression, critical-band and trust gates and says novelty_gate.py is not mutated.

make ci-local at d57ed0e: 28 passed, 0 failed, 7 skipped.

Not approving yet. Still open: the seven risks from my review.

[napetrov]

Review-body item 1 (mutmut in the shared tools lock): fixed in c5c6715. requirements-ci.txt and requirements-ci.lock.txt are back to their pre-PR content. mutmut==3.8.0 now sits in requirements-mutation.txt, locked into requirements-mutation.lock.txt (uv pip compile with the same options as the tools lock: --constraint=requirements.lock.txt, --universal, --python-version=3.11, --exclude-newer=P7D, --generate-hashes). Only mutation.yml installs it (requirements.lock.txt plus the new lock). tests/test_dependency_locks.py now covers the new lock (hashes, header, cooldown, stacked pins agree with the main lock, mutmut not in the tools lock); Renovate's pip-compile pattern and .licenserc.yaml include it. pip-audit does not audit the new lock; that keeps its closure out of the daily audit, as you suggested. Say so if you want it audited.

[napetrov]

Checked the new lock: regenerating requirements-mutation.lock.txt with the command in its header (uv 0.11.9, same options as the tools lock: --constraint=requirements.lock.txt, --universal, --python-version=3.11, --exclude-newer=P7D, --generate-hashes) gives the committed file byte for byte; the header, hashes and stacked pins pass tests/test_dependency_locks.py, and mutation.yml installs only requirements.lock.txt and this lock. One gap found and fixed in c6f1bfe: moving mutmut out of the tools lock had dropped its closure from pip-audit, so pip-audit.yml now audits requirements-mutation.lock.txt too (test_pip_audit_audits_every_lock fails without it), and CONTRIBUTING no longer says it is unaudited.

[antonwolfy]

Review at 91d9ae6. The six earlier threads are fixed: the default thresholds, the score flags, the both-arms filter, the coverage floor, MIN_PAIRS_FOR_TEST and the step timeout all hold. The new tests still leave mutants alive in the same modules. The six inline comments and the first numbered item below are worth fixing before merge.

What I ran, on a Linux review host at 91d9ae6:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,315 tests; changed-line coverage 100%. The failure is pip-audit: gitpython 3.2.0 (CVE-2026-114145, fix 3.2.1), pinned in requirements.lock.txt on the PR's base as well, so it is not caused by this PR.
• About 350 hand-made mutants of the seven pilot modules, run against the new test files and the 14-file selection from [tool.mutmut]. The survivors I re-ran are in the comments and below. I did not run mutmut run itself, so the 75.3% -> 92.0% figures are not checked.
• zizmor --offline --min-severity medium on the workflows: no findings. tests/test_dependency_locks.py: 28 passed. The summary script from mutation.yml on made-up mutmut results lines: the table is right.

Findings that cannot be anchored inline:

1. Length guards and the no-SciPy seed are pinned in one direction only. Each of these passes the 14-file selection (881 passed):
   • paired_test: len(treatment) != len(control) -> len(treatment) > len(control) (stats.py:479). A shorter treatment vector raises from zip(strict=True) instead of returning None.
   • mean_ci: bootstrap_ci(vals, confidence=confidence, seed=seed) -> without seed=seed (stats.py:154). Without SciPy the seed is ignored.
     Suggestion: assert stats.paired_test([1, 2, 3], [1, 2, 3, 4]) is None and the shorter-treatment mirror; with HAS_SCIPY monkeypatched to False, assert mean_ci(vals, seed=7) differs from the default seed on a sample whose bounds depend on it.
2. Nit: dokimi/SECURITY.md (outside this diff) lists the locks pip-audit runs against and does not name requirements-mutation.lock.txt. Suggestion: add it to that sentence.

Checked, no issues

• Fixes from my review: the four default-lowering mutants, has_critical/has_warnings dropping the score and == "WARN" -> != "OK", the dropped context half of the both-arms filter, min_retrieval_coverage 0.80 -> 0.75, MIN_PAIRS_FOR_TEST 6 -> 5, and the alpha, lo_idx rounding and seed + 1 mutants are all killed now.
• Locks: requirements-ci.lock.txt is back to the base content, mutmut sits only in requirements-mutation.lock.txt, installed by mutation.yml alone, and pip-audit.yml audits it.
• Workflow: Run mutmut has timeout-minutes: 75 under the job's 90, with continue-on-error and an if: always() summary step. Actions are SHA-pinned, permissions: contents: read.
• Determinism: the four gate test files and test_stats.py, test_binary_stats.py, test_caching.py pass under three different pytest-randomly seeds.
• Cache key: the pinned hash is of a literal payload with sorted keys, so it is platform-stable.

[antonwolfy]

I pushed a commit per nit, up to b5785d6, which fix the nit from my review:

• b5785d6: dokimi/SECURITY.md:47: the list of locks pip-audit runs against now names requirements-mutation.lock.txt.

make ci-local at b5785d6: 28 passed, 1 failed (pip-audit, same as at the reviewed head), 7 skipped.

Not approving yet. Still open: the seven risks from my review.

[xaleryb]

• Ran: the full suite, mutmut 3.8.0 as mutation.yml installs it, the summary script verbatim, and mutations of the pip-audit step and of the bootstrap defaults.
• Summary on a partial run: a step-timeout SIGINT leaves 1228 mutants not checked, yet eval.stats prints 94.7% with no incomplete marker. A failing baseline prints 0|0|n/a rows instead of "No results". Expected: a banner when any mutant is not checked or killed+survived is 0.
• Test selection: is None -> is not None in first_replayed passes the 14 files but fails test_judge_response_cache.py, and arm_significance_31 is killed only by test_arm_runner.py. Expected: survivors are mutants no test kills. Either drop the selection or widen it.
• The PR body still places mutmut in requirements-ci.lock.txt and quotes pre-merge totals.
• Pushed:
  • 60a99b6: the pip-audit lock test passed with locks dropped from the audited list; it now parses the step.
  • 55b32d8: pins bootstrap_ci's default seed and n_resamples, which grounding uses.
• Left: the summary banner, the selection, the PR body, and a rebase for pip-audit.

[antonwolfy]

Review at 55b32d8. The only change since b5785d6 is two commits by @xaleryb: the pip-audit lock test now parses the step, and a new test pins bootstrap_ci's defaults. Both are right. Five of my six threads still stand, and one item from my earlier review body is still open.

What I ran, on a Linux review host at 55b32d8:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,316 tests passed. The failure is pip-audit: gitpython 3.2.0, pinned in requirements.lock.txt on the PR's base as well, so this PR does not cause it.
• The mutants from my threads and from my earlier body, re-run against test_stats.py, test_binary_stats.py, test_regression.py, test_trust_gate.py, test_critical_bands.py, test_score_gate.py and test_caching.py (267 passed on each): 10 of 10 survive, listed below and in the threads.
• Ten mutants of the cache key (role, sampling parameters, image blocks, sort_keys, ensure_ascii, default=str, block joining): all killed. The two bootstrap_ci default mutants (seed 0 -> 1, n_resamples 10,000 -> 5,000): both killed by the new test.
• The five test files above under two pytest-randomly seeds and with sockets disabled: 221 passed each time. I did not run mutmut itself.

Findings that cannot be anchored inline:

1. Length guard and the no-SciPy seed are still pinned in one direction only. This is the first item of my earlier review body, not yet addressed. Both survive the seven test files:
   • paired_test: len(treatment) != len(control) -> len(treatment) > len(control) (stats.py:479). A control longer than the treatment is no longer rejected as degenerate.
   • mean_ci: bootstrap_ci(vals, confidence=confidence, seed=seed) -> without seed=seed (stats.py:154). The seed is ignored on the bootstrap fallback.
     Suggestion: assert stats.paired_test([1, 2, 3], [1, 2, 3, 4]) is None and the shorter-treatment mirror; with HAS_SCIPY monkeypatched to False, assert mean_ci(vals, seed=7) differs from the default seed on a sample whose bounds depend on it.

I agree with @xaleryb that the weekly summary does not mark a partial run. New evidence: made-up mutmut results lines with 1 killed, 1 survived and 3 not checked print | dokimi.eval.stats | 1 | 1 | 3 | 50.0% |, and a module with nothing checked prints n/a; nothing says the run was cut short.

Still open from my earlier review:

• dokimi/tests/test_stats.py:598: the hi_idx/lo_idx mutants (int(...) without round, n_resamples for n_resamples - 1) all still pass; the new defaults test does not reach them.
• dokimi/tests/test_binary_stats.py:211: removing seed=seed from mcnemar_test's bootstrap_ci call still passes.
• dokimi/tests/test_regression.py:197: has_warnings without metric_regressions, and has_critical with only the first metric, both still pass.
• dokimi/tests/test_trust_gate.py:318: if agreement_scores: -> if len(agreement_scores) > 1: still passes.
• dokimi/tests/test_trust_gate.py:404: if verdict.warnings: -> if verdict.failures: still passes.

Checked, no issues

• pip-audit test: test_pip_audit_audits_every_lock now reads the pip-audit step's run text. A lock name that survives only in a YAML comment no longer satisfies it: removing it from the sed command fails the test.
• Cache key: the new tests kill every key mutant I tried; the pinned hash is of a literal payload.
• Determinism: the pinned bootstrap_ci interval uses only random.Random, so it does not depend on SciPy or NumPy.
• Locks and workflow: mutmut sits only in requirements-mutation.lock.txt, installed by mutation.yml alone and audited by pip-audit.yml; Run mutmut has a 75-minute step timeout.

[napetrov]

Review-body items, by sha (fork/ci/mutation-pilot at d27b2e3):

1. Length guard and no-SciPy seed: 705e015. test_paired_test_rejects_a_length_mismatch_in_both_directions (the len(treatment) > len(control) mutant fails it) and test_mean_ci_forwards_the_seed_to_the_bootstrap_fallback (monkeypatches HAS_SCIPY and bootstrap_ci and asserts seed=7 arrives; the dropped-seed mutant fails it). I checked that a seed-differs literal does not exist for mean_ci at 10,000 resamples, hence the forwarding check.
2. The five inline mutants: replied on each thread, same sha.
3. Summary banner on a partial run (xaleryb): d27b2e3. The step summary prints an 'Incomplete run' note when any mutant is not checked or a module has nothing scored. Run against made-up results lines: banner shown for not-checked and for a 0/0 module, absent for a clean run.
4. PR body: edited. mutmut is described in requirements-mutation.lock.txt, not the tools lock; the counts are labelled as first-run numbers.
5. Merge: intel/main merged in 60eb92b (one conflict in CONTRIBUTING.md, both sides kept).
6. Not done, needs a decision: the test selection in [tool.mutmut] (drop it or widen it to the tests that kill first_replayed and arm_significance mutants). Left to the author.

[antonwolfy]

LGTM. The five tests added since my last review kill the mutants from my threads, and the new "Incomplete run" banner reads the right counts. Two nits inline.

What I ran, on a Linux review host at d27b2e3:

• make ci-local: 28 passed, 1 failed, 7 skipped; 7,856 tests passed. The failure is pip-audit on filelock 3.32.7 and setuptools 84.0.0, pinned in requirements.lock.txt and requirements-ci.lock.txt, which this PR does not touch.
• The mutants from my five threads, the length-guard and no-SciPy-seed mutants from my earlier body, and about 20 further hand-made mutants of the seven pilot modules (thresholds, boundaries, defaults, cache-key and usage handling), run against the seven pilot test files: all killed except the one in the caching.py nit. I did not run mutmut run itself.
• The summary script from mutation.yml on made-up mutmut results lines (clean run, not checked, a module with nothing scored, empty file): the banner appears only for the incomplete runs.

@xaleryb's point on the [tool.mutmut] test selection waits for the author. At this head caching.draw_extra ({**extra, "draw": draw} for draw 0) and caching.first_replayed (range(attempts - 1)) mutants pass the seven pilot test files and fail only test_judge_response_cache.py, so they will show as survivors although a test kills them.

Checked, no issues

• Threads: hi_idx/lo_idx (no round, n_resamples for n_resamples - 1), seed=seed in mcnemar_test, has_warnings without metric_regressions, has_critical with metric_regressions[:1], len(agreement_scores) > 1 and verdict.failures for verdict.warnings all fail the new tests.
• Length guard and seed: len(treatment) > len(control) and mean_ci without seed=seed are killed.
• Config: all 7 only_mutate paths and 14 selected test files exist; tests/test_dependency_locks.py passes.

[antonwolfy]

Nit: _env_flag has a surviving mutant that no test kills. raw.strip().lower() not in _FALSEY -> raw.lower() not in _FALSEY (caching.py:94) passes the seven pilot test files (297 passed). test_prompt_cache_env_switch (line 123, outside this diff) tries only 0, false, off, "", 1 and yes. With the mutant, DOKIMI_..._CACHE="off\n" or " off " leaves the cache on, and OFF or FALSE would too without .lower().

Suggestion: add (" off ", False), ("OFF", False) and ("False\n", False) to the parametrization of test_prompt_cache_env_switch.

[napetrov]

Applied in 9f5e03e (Anton Volkov). test_prompt_cache_env_switch now also covers " off ", "OFF" and "False\n". Pushed in b74f999; tests/test_caching.py passes on that head.

[antonwolfy]

Nit: the banner says "0 mutants were not checked (a step timeout or a crash cut the run short)" when only a module has nothing scored. A module whose mutants are all no tests or timeout has unchecked == 0 and still gets that text.

Suggestion: build the text from the parts that apply, for example print the "not checked" clause only if unchecked, and the "nothing was scored in ..." clause only if unscored.

[napetrov]

Applied in 3ea86b2 (Anton Volkov). The banner names only the clauses that apply and is printed only when a mutant is not checked or a module has nothing scored. Pushed in b74f999. The step script is not under a pytest test; checked by running it on made-up mutmut results lines: clean run (no banner), partial run (not-checked clause), unscored module (nothing-scored clause).

[napetrov]

Fixed in 3ea86b2: the banner text is built only from the parts that apply. The "not checked" clause prints only if unchecked, and the "nothing was scored in ..." clause only if unscored. A module where every mutant is no tests or timeout now gets only the second clause.

[napetrov]

Item 1 (mutmut in the shared tools lock) is fixed in c5c6715 by napetrov: requirements-ci.txt and requirements-ci.lock.txt have no diff against the merge base on head, and mutmut sits in requirements-mutation.lock.txt. The inline risks are their own threads, all resolved.

[napetrov]

Status review. The seven risks it lists are fixed on head: the lock split in c5c6715, the tail-index, seed, flag and render mutants in 705e015, the banner in d27b2e3. Later reviews on this PR confirm them.

[napetrov]

Item 1 (length guards, no-SciPy seed) is fixed in 705e015: on head len(treatment) > len(control) and a fallback without seed=seed both fail tests/test_stats.py. Item 2 (SECURITY.md) is fixed in b5785d6 by Anton Volkov: line 49 names requirements-mutation.lock.txt.

[napetrov]

Fixed in b5785d6 by Anton Volkov: dokimi/SECURITY.md names requirements-mutation.lock.txt among the locks pip-audit runs against.

[napetrov]

Item 1 is fixed in 705e015 (both mutants fail tests/test_stats.py on head). The partial-run summary is fixed in d27b2e3: mutation.yml prints "Incomplete run" for not-checked mutants and for a module with nothing scored. The five open threads of that review are resolved.

[napetrov]

The two inline nits are the threads on mutation.yml:89 and test_caching.py:402, answered there. The test-selection point is a decision for the author and is asked on the comment from @xaleryb (IC_kwDOVBg7g88AAAABgAyVFQ).

[napetrov]

The one question of this review is the thread on mutation.yml:98 (was the workflow dispatched, does the summary render). It is answered there, with the local run on 9b13d65.

[napetrov]

No new claim. The three failing checks listed in the CI table are green now: gh pr checks 178 on 9b13d65 shows every check passing, including pip-audit (locks), Tests (py3.11)/(py3.14) and the static job.

[napetrov]

This review is the wrapper of the six inline findings of the OpenCodeReview run. Five are fixed (one commit per thread, shas on each thread: a9c43fd, edd5ab2, 31e9e2c, 6968fb5, 5060fea); the _round_p one is pushed back with probe output on its thread.

[napetrov]

Installing the Codecov GitHub app is an organisation setting, not a change to this PR. The report itself says all modified and coverable lines are covered.

[napetrov]

Summary banner on a partial run: d27b2e3. PR body: edited (mutmut is in requirements-mutation.lock.txt; the counts are labelled first-run numbers). pip-audit: the base moved, and intel/main is merged in (60eb92b and again now). The test-selection point is repeated in your later comment and asked there (IC_kwDOVBg7g88AAAABgAyVFQ).

[napetrov]

Waiting on a decision before I change this: which test selection should [tool.mutmut] use? The banner, the PR body and pip-audit are fixed. Context: I reproduced it on head. The range(attempts - 1) mutant of caching survives the 14-file selection (980 passed) and fails once tests/test_judge_response_cache.py and tests/test_arm_runner.py are added (1 failed, 996 passed). Options: (A) add those two files; (B) add every test file that imports a pilot module, which per your run also kills draw_extra and takes about 40% longer; (C) drop the selection and run the full suite per mutant, the slowest. My recommendation: A, because it kills the five named survivors for the smallest run-time cost, and the step has a 75-minute timeout inside a 90-minute job. I did not measure the run time. The thread stays open until you choose.

[napetrov]

All six findings of this summary are inline threads. Five are fixed, one commit per thread (a9c43fd, edd5ab2, 31e9e2c, 6968fb5, 5060fea), each with the mutant that survived the old test and now fails. The _round_p finding is pushed back on its thread with probe output.

[vchamarthi]

The name says the model decides, but only the positive case (bedrock + Claude) is asserted. The system-breakpoint test has a meta.llama3 negative, this one doesn't. Mutation: build_messages checks provider in {"bedrock", ...} without looking at the model. It would still set the breakpoint on a Llama model on Bedrock, and this test passes. Add a bedrock meta.llama3-70b-instruct-v1:0 case asserting cached is False and that the content is the plain string LONG + "q".

[vchamarthi]

The None, NaN and empty cases are not covered. {"score": None} makes float(None) raise TypeError, and float('nan') < threshold is False, so a NaN metric passes the gate silently. The new tests only cover a missing key. Add tests that pin the intended behaviour for None and NaN (raise, or treat as a violation). Also cover a spec with no critical_bands key, and an empty fail_on list in the parametrized set.

[vchamarthi]

Nit: for the ci lock this repeats the existing test_tools_lock_is_constrained_to_main_lock, so it adds no new failure mode. Delete the old test, or parametrize it and keep only one.

[vchamarthi]

Nit: this hardcodes the mutmut version, so every Renovate bump of the lock fails the test, which defeats the lock tracking. If mutmut is missing from the lock, the ["mutmut"] lookup raises a bare KeyError. Instead, assert that mutmut is in the mutation lock and not in the ci lock. Then compare the exact pin in requirements-mutation.txt with the locked version, as test_tools_lock_has_every_requested_pin does for the ci lock. The substring check would also accept mutmut==3.8.01.

[vchamarthi]

This assertion cannot fail. The mean of two draws from [0, 1] is always 0.0, 0.5 or 1.0. The lo == hi check is also trivial with one resample, since lo_idx and hi_idx both clamp to 0. Mutating the confidence handling or swapping lo/hi still passes. Pin the exact value for seed 0 (for example res["lo"] == res["hi"] == <literal>) so a seed or indexing mutation is caught.

[vchamarthi]

This test does not show that the tolerance scales with the mean. Noise of 1e-10 gives sd of about 1e-10, which is below 1e-9 even with an absolute tolerance. Mutation: change max(1.0, abs(mean)) to 1.0 and this test still passes. Only the next test catches that mutation. Use noise that is degenerate only under the scaled tolerance, as the second test does.

[vchamarthi]

This compares the value to its own 4-place rounding, so it passes for any rounding of 4 or fewer digits. Mutation: change _round_p to round(p, 3), or round to a coarser value, and the test still passes. Only dropping the rounding entirely would fail it. The 0 < p < 1 check adds almost nothing. Pin the literal, for example assert res["p_wilcoxon"] == <expected 4-place value>.

[vchamarthi]

The name (and the docstring of the test above: "serializes measured as false (the report JSON reads it)") promise a serialization check, but both tests only read attributes of the in-memory check. The serialized key is written in report/generator.py ("measured": c.measured), and no test in the repo asserts report["trust"]["checks"][...]["measured"]. Mutation: delete "measured": c.measured from that dict, or hard-code it to True. Both tests still pass. Assert on the generated report JSON, or rename the tests to what they check.

[exai-bot]

🔍 OpenCodeReview found 4 issue(s) in this PR.

By severity: 1 medium, 3 low

Severity	Category	Location	Finding
medium	maintainability	pip-audit.yml:53	This edit lands in a 10-line # block (lines 51-60).
low	style	pyproject.toml:279	This comment block is 10 lines.
low	style	mutation.yml:6	The 8-line header comment, and the 4–5 line comments above Install deps and Run mutmut, are longer than the repo's two-line comment l...
low	maintainability	mutation.yml:51-52	continue-on-error: true here plus || true on mutmut results mean a broken run (mutmut crash, bad config, or a missing test file) le...

Review details

• Commit reviewed: 7a0dc61 against merge base f0f95a2
• Coverage: 6 files in 2 groups
• Engine: OpenCodeReview v1.13.3, us.anthropic.claude-sonnet-5-5
• Usage: 362k tokens (245k from cache), 54s
• Tool calls: 21 (8 code_search, 8 file_find, 3 file_read, 2 code_comment)
• Mutation testing setup (workflow, config, ignores): 3 files
• CI/repo tooling config (license, renovate, pip-audit): 3 files

[exai-bot]

The 8-line header comment, and the 4–5 line comments above Install deps and Run mutmut, are longer than the repo's two-line comment limit. If the check covers YAML, this fails CI. Trim each to the constraint and link the PR or an issue for the backstory.

[exai-bot]

continue-on-error: true here plus || true on mutmut results mean a broken run (mutmut crash, bad config, or a missing test file) leaves the job green. Only the summary text shows it. That is acceptable for a pilot, but nobody reads a summary on a green weekly job. Consider failing the job, or emitting a ::warning:: annotation, when the results file is empty or has not-checked mutants.

[exai-bot]

This edit lands in a 10-line # block (lines 51-60). scripts/check_comment_blocks.py fails any block of 3+ lines once a PR edits a line in it, so this will likely fail CI's static job. The edited line is also over-long. Shorten the block to at most two lines, for example by moving the per-lock list out of the comment.

Suggestion:

Suggested change

	# requirements-mutation.lock.txt for the weekly mutation run), so --no-deps --disable-pip audits them as written
	# The locks are the full closure (docs, tools, mutation), so --no-deps --disable-pip audits them as written.
	# Each --ignore-vuln is documented in SECURITY.md and osv-scanner.toml; change all three together.

[exai-bot]

This comment block is 10 lines. The repo rule (scripts/check_comment_blocks.py, AGENTS guide) caps comments at two lines and fails a PR that adds a block of three or more # lines. The scripts/check_comment_blocks.py file isn't in the tree I could search, so I can't confirm it covers TOML. Shorten it to the constraint (what source_paths / only_mutate mean, why novelty_gate.py is excluded) and move the rationale to the PR description.