[vlad-perevezentsev]

Summary

arms run, arms matrix-run and subjects run all end in one arms run, each by its own path. matrix-run copies its Namespace for each cell and _apply_matrix_cell mutates the copy. subjects run builds a Namespace by hand from FORWARDED_RUN_CONFIG_FIELDS. _build_arms_output reads most fields with getattr(args, ..., default), so a field one path drops looks exactly like a user who chose the default. This PR changes only tests. It pins what reaches the runner from each entry point, so that a later move to a typed run config (#4 Wave 5) can show an empty test diff.

tests/test_arms_run_config_parity.py (new, 11 tests):

• Input: the argv goes through dokimi.cli.parse_args, the function main calls (it also fills in the --judge-model default from --judge-provider), and then to the command's handler. No hand-built Namespace, and no assert on forwarded_run_config(): a refactor changes both.
• Boundary: ArmRunner.__init__, run, judge and build_output, recorded on the class and then run for real. Arguments the caller left out are recorded with the runner's own default (inspect.signature(...).bind + apply_defaults). top_k/rerank_threshold are read from the doc treatment create_treatments built. The judge is read from the Judge/JudgePanel object passed to runner.judge. Because the methods are patched on the class, the tests also pass when ArmRunner, Judge and create_treatments are imported at module top (checked).
• LLM calls: provider_calls (adapter seam, response cache off, a bypass to litellm fails the test), with a judge-shaped reply for every request.
• Cases: each case asserts the whole row (model, provider, harness, matrix cell, plugin set, 9 sampling/budget fields, doc top-k/threshold, library name/id, concurrency, the judge or panel, baseline arm), not single calls.

entry point	every field it takes set away from its default	nothing set	other
arms run	all 20 FORWARDED_RUN_CONFIG_FIELDS except the session pair, plus --plugins, --baseline-arm	with and without --judge	panel (--judge-model left to --judge-provider); --session-turns/--turn-reminder; named --matrix-cell beats --model/--provider/--plugins
arms matrix-run	same flags; two cells, each with its own name, model, provider and plugins	per cell	panel per cell
subjects run	the 13 flags it has; a profile as the baseline, so the baseline_arm override is visible	defaults	—

What the tests pin about today's behaviour

These are facts, not fixes; nothing in the package changes.

• subjects run has no --temperature, --seed, --judge-temperature, --judge-panel, --stats-correction, --session-turns or --turn-reminder. Its runs always get None/None/None/single judge/holm/no session.
• arms matrix-run has no --session-turns/--turn-reminder. The flags it does share with arms run reach every cell, but --model/--provider/--plugins are replaced by the cell's values.
• In subjects run, the forwarded model, provider and harness never reach the runner. The matrix cell sets all three, from the descriptor's [[matrix.cells]] or, without them, from the one cell built from these flags.

Evidence

The branch is upstream/main + one new test file, so the tests describe main as it is: 11 passed.

Mutation sweep: 57 single edits to commands/arms.py, commands/subjects.py and cli.py, each run against this file and then reverted:

mutations	result
each ArmRunner(...) keyword dropped (13); run library_id/concurrency; judge baseline_arm/concurrency; build_output baseline_arm; Judge temperature / model; JudgeConfig temperature / provider; create_treatments top_k / rerank_threshold; --context7-id ignored; --judge ignored; --judge-panel ignored; _apply_matrix_cell name / model / provider / plugins dropped; the stats_correction getattr default holm→bh; the subjects run baseline_arm override dropped; its default cell model replaced; the --judge-model provider default in parse_args turned off	all fail the tests
FORWARDED_RUN_CONFIG_FIELDS minus context7_id, judge, judge_model, judge_provider, top_k, rerank_threshold, max_iterations, max_tokens, grounded_from_topics, concurrency	all fail the tests
FORWARDED_RUN_CONFIG_FIELDS minus model, provider, harness	pass: the cell sets these (see above)
FORWARDED_RUN_CONFIG_FIELDS minus the 7 fields subjects run has no flag for	pass: never forwarded, the flags do not exist
grounded_from_topics getattr default False→True	passes: every caller sets the field, so the default is never read
matrix-run sharing one Namespace across cells instead of copying it	passes: each cell overwrites all five fields it sets

Each of the survivors is a change that does not alter behaviour today, not a gap in the test.

• Full pytest: 9256 passed, 57 skipped. tests/test_audit_test_quality.py passes (no flagged shapes). Each test passes alone and under xdist.
• Growth ratchet passes; the new file is 338 lines.

Merge Danger

Door: two-way. One new test file; nothing in the package changes.

Blast Radius: none at runtime. A future change to how these three commands configure an arms run fails these tests until its effect is intended and the tables are updated.

Overlap: #238 and #255 touch commands/arms.py/commands/subjects.py. This PR does not, and its file name does not clash with #238's tests (test_arms_output_errors.py, test_arms_matrix_cells_input.py). If #255 adds a provider choice, nothing here changes.

Not in this PR:

• The typed run config itself (FORWARDED_RUN_CONFIG_FIELDS / forwarded_run_config / the getattr defaults → one config object). That is the next PR, and these tests must pass on it unchanged.
• The flags missing from subjects run and matrix-run (listed above): adding them changes behaviour, so it is a separate change.

Checklist

• Tests added and pytest passes (9256 passed, 57 skipped)
• ruff, mypy, and schema checks pass (make ci-local)
• Docs: none needed; no changelog fragment (only dokimi/tests/ changes)
• Self-reviewed per "Before opening a PR" in AGENTS.md: mutation sweep above
• No ad-hoc run artifacts committed
• New files carry the SPDX license header

Refs #4

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.89%. Comparing base (3a1f287) to head (9452b2f).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #264      +/-   ##
==========================================
+ Coverage   92.83%   92.89%   +0.06%     
==========================================
  Files         262      262              
  Lines       33016    33016              
  Branches     5510     5510              
==========================================
+ Hits        30649    30670      +21     
+ Misses       1571     1553      -18     
+ Partials      796      793       -3     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.