[antonwolfy]

Summary

Follow-up to the review of #250, where the dashboard and the report model-compare run summary were left averaging each arm over its own scored questions while report generate, report eval and the pipeline average both over the complete pairs (review). The author and the reviewers agreed in that thread to keep it for a follow-up, because it changes reported numbers. The model-compare By Difficulty table has the same bug: its context mean covers every context-scored question in a bucket, beside a delta over the pairs.

On main, an eval file with three questions, q1 80/60, q2 40/40 and q3 100/– (no baseline score), gives:

Consumer	Context	Baseline	Delta
report generate	60.0	50.0	+10.0
dashboard (ProductSnapshot, and each by_source row)	73.3	50.0	+10.0
report model-compare Overall Summary	73.3	50.0	+10.0
report model-compare By Difficulty row	73.3	(no column)	+10.0

The dashboard and model-compare context mean covers q3 and the baseline mean and the delta do not, so the delta printed beside them is not their difference. With this PR every row prints 60.0 for the context arm and +10.0 for the delta, and 50.0 where it has a baseline column.

Changes

• dashboard/aggregator.py: the snapshot (avg_with_docs, avg_without_docs, avg_delta) and each _compute_by_source group take their numbers from one helper, _arm_means, which averages both arms over the questions scored in both. The delta is the difference of the pair means, as eval.stats.paired_summary computes it. It used to be the mean of the stored per-question delta; for a file the judge writes, the two agree. QuestionResult.delta still carries the stored value.
• report/model_compare.py _run_summary: with_avg and without_avg are over the same rows as delta_avg. n, n_paired, unpaired, improved and degraded are unchanged.
• report/model_compare.py _difficulty_cells: when a bucket has a context score, the row's context mean and delta come from _run_summary on the bucket's questions, so a difficulty row follows the same rule as the Overall Summary. A bucket with no context score still shows its baseline mean, marked (baseline), as before.
• Both consumers handle a run with no pair the same way. A run that scored one arm only keeps that arm's mean, because the dashboard's doc_score (and its status band) and model-compare's baseline-only ranking read it. When both arms are scored but never on the same question, neither mean is given (None, rendered —). In the difficulty table, a bucket the run scored in the context arm only keeps its context mean with — for the delta, and a bucket with both arms scored but no pair shows — for both.
• eval/stats.py module docstring: it no longer names the dashboard and model-compare as exceptions.
• The unreleased fix(report): report eval, generate and pipeline use one paired arm summary (#229 P1-3) #250 fragment (changelog.d/20261026_130000_napetrov_report-eval-generate-agree.md): removed the one sentence saying the dashboard and the model-compare run summary "still average each arm over its own scored questions", which is no longer true.
• Docs: docs/quickstart.md (dashboard step) and docs/benchmarking-and-comparison.md (model-compare metrics, now with the difficulty rows) say the means are over the questions scored in both arms.
• tests/fixtures/model_compare_sections.json: re-recorded (DOKIMI_UPDATE_GOLDEN=1). Only the Overall Summary context/baseline cells, the absolute-quality ranking cells and the By Difficulty context cells change. In every changed summary row the context mean minus the baseline mean now equals the printed delta, for example 68.8 / 56.2 / +17.1 → 70.0 / 52.9 / +17.1, and 45.0 / 45.8 / −10.0 → 45.0 / 55.0 / −10.0. In the By Difficulty table one cell changes, the first run's easy context mean, in eight sections: 63.3 → 65.0 (seven sections) and 68.8 → 70.0 (pinned_arm).
• scripts/growth_baseline.json: _load_snapshot dropped from complexity 13 to 10, so its entry is removed; report/model_compare.py dropped from 1209 to 1207 lines.
• tests/test_records_guard.py: report.model_compare's count of condition-key reads lowered from 17 to 16, since _difficulty_cells reads one fewer.
• Changelog fragment (Fixed).

Risk

The numbers change for any run with a question scored in one arm only:

• dashboard: the context and baseline means per library and per source (DASHBOARD.md and dashboard.json), and with them the good/fair/poor status, which follows the context mean;
• report model-compare: the context and baseline columns of the Overall Summary and the absolute-quality ranking, and the context column of the By Difficulty table.

Runs where every question is scored in both arms print the same numbers as before. A dashboard delta can also change where a stored delta disagrees with the two stored scores. A run, source group or difficulty bucket with both arms scored but no question scored in both now shows — where it showed means; on the dashboard such a run reads as no-data. No input that was accepted before is refused now.

eval/judge.py (avg_with / avg_without / avg_delta) and commands/baseline.py _compute_summary were checked and need no change: every record they average has a delta, which the judge sets only when both arms were judged and baseline.py only when both scores are present, so their means are already over the pairs.

Testing

On a Linux review host, at 347bb9c (on main 0a63f4e):

• make ci-local: 27 passed, 1 failed, 7 skipped (skipped: external-catalog-smoke, which needs docker; test-latest, docs deploy and the pip-audit report, whose job conditions are false on a PR; cli-surface "Package the result"; conda-forge-deps, since micromamba is not installed; semgrep secrets).
  10185 tests passed, 4 skipped; changed-line coverage 100% (22 lines).
  The one failure is pip-audit: it reports ten advisories against filelock 3.32.7 in the CI lock file (fix versions 4.0.9 / 4.2.1). This PR does not touch dependencies or the lock file, so main fails the same way. It is left for a dependency update.
• The repro above: on main, the dashboard and model-compare print 73.3 / 50.0 / 10.0 against report generate's 60.0 / 50.0 / 10.0; at this head all three print 60.0 / 50.0 / 10.0. The By Difficulty row for the same questions reads | easy | 3 | 73.3 | +10.0 | on main and | easy | 3 | 60.0 | +10.0 | here.
• New tests, on two pairs (80/60, 40/40), one question scored in the context arm only and one in the baseline arm only:
  • test_dashboard.py::test_aggregator_averages_both_arms_over_the_complete_pairs: the snapshot is 60.0 / 50.0 / 10.0, equal to paired_summary on the same records;
  • test_dashboard.py::test_by_source_averages_over_the_pairs_and_a_group_without_one_has_no_means: the persona row is 60.0 / 50.0 / 10.0, and a chunk group with one context-only and one baseline-only question gives None for all three;
  • test_dashboard.py::test_baseline_only_run_keeps_its_baseline_mean: a run with no context arm still scores 50.0 (fair);
  • test_model_compare.py::test_run_summary_averages_both_arms_over_the_pairs: the Overall Summary row reads 4 | 2 (2 unscored) | 60.0 | 50.0 | **+10.0** | 1 | 0;
  • test_model_compare.py::test_run_summary_gives_no_means_for_two_arms_without_a_pair: that row reads — for both means.
• New difficulty-table tests:
  • test_model_compare.py::test_difficulty_row_averages_the_context_arm_over_the_pairs: two pairs (80/60, 40/40) and one context-only question give | easy | 3 | 60.0 | +10.0 |;
  • test_model_compare.py::test_difficulty_row_without_a_pair_shows_no_context_mean: a hard bucket with one context-only and one baseline-only question reads | hard | 2 | — | — |, and a medium bucket scored in the context arm only keeps | medium | 1 | 70.0 | — |.
• On main's code, the six tests of the new behaviour fail. The two difficulty tests also fail on the first commit of this PR, before the difficulty fix. The baseline-only test passes on main: it guards behaviour that should not change.
• Mutants, each run on a copy of the commit, each fails its test:
  • dashboard: the pair branch dropped (the snapshot and by-source tests fail); the no-pair None branch dropped (by-source); the single-arm fallback dropped (baseline-only); the delta sign flipped (snapshot);
  • model-compare: arm means over all scores again (the pairs test); the no-pair None branch dropped (the no-pair test); the single-arm fallback dropped (4 baseline-only goldens in test_model_compare_sections.py);
  • model-compare difficulty rows: the context mean over every context score again (both difficulty tests); the no-pair None branch of _run_summary dropped (the no-pair difficulty test, hard shows 80.0); its single-arm fallback dropped (the no-pair difficulty test, medium shows —).
• ruff check on the changed files and mypy on the whole package are clean.
• Merged together with the other open follow-up branches (test/novelty-all-vs-all-pins, ci/filelock-advisories) on main: 10189 tests passed, 4 skipped; mypy, ruff check and the growth check pass.

Revert

A plain revert restores the per-arm means. Nothing persisted changes: the dashboard and model-compare outputs are regenerated from the eval files.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 8c8fd399977f, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 11 of 80 turns, 0.5 of 40 minutes, 186k in / 2k out tokens.

The change makes the dashboard snapshot, each by_source row and the model-compare run summary average both arms over the questions scored in both, as report generate and paired_summary do. I found no defects. In _arm_means and _run_summary, the no-pair cases behave as the description says. A one-arm run keeps its mean. Two arms that never share a question give None, so the dashboard status reads no-data. The new tests cover the pair, one-arm and no-pair cases for both consumers.

No issues found.

For a human reviewer

• Product call: a run with both arms scored but no shared question now reads no-data on the dashboard instead of showing two means.

Not verified

I did not run the tests or regenerate the golden fixture model_compare_sections.json. I did not check its re-recorded numbers.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.27%. Comparing base (0a63f4e) to head (347bb9c).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #286      +/-   ##
==========================================
+ Coverage   93.26%   93.27%   +0.01%     
==========================================
  Files         272      272              
  Lines       35117    35119       +2     
  Branches     5964     5964              
==========================================
+ Hits        32752    32759       +7     
+ Misses       1547     1546       -1     
+ Partials      818      814       -4     

Files with missing lines	Coverage Δ
dokimi/dokimi/dashboard/aggregator.py	93.61% <100.00%> (+3.54%)	⬆️
dokimi/dokimi/eval/stats.py	96.20% <ø> (ø)
dokimi/dokimi/report/model_compare.py	93.06% <100.00%> (+0.02%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

@exai-bot /rereview

[exai-bot]

@antonwolfy I could not read that command: unknown command /rereview. See @exai-bot /help.

[antonwolfy]

@exai-bot /review

[exai-bot]

Note

AI-generated review of 347bb9c444f6, requested by @antonwolfy, using the default review skill. It can be wrong: verify a finding before acting on it. React with a thumbs up or down on a finding to tell the bot whether it helped.
Model us.anthropic.claude-sonnet-5-5, effort default, 11 of 80 turns, 0.8 of 40 minutes, 391k in / 4k out tokens.

Re-review of 347bb9c444f6: nothing new.

[vchamarthi]

✅ OpenCodeReview: Review complete: 0 finding(s) across 10 selected item(s).

[vchamarthi]

Approving. The change does what the description says: the dashboard snapshot, each by_source row, the model-compare Overall Summary and the By Difficulty rows now average both arms over the questions scored in both, so the printed delta is the difference of the printed means.

Checked:

• The description's example (q1 80/60, q2 40/40, q3 100/-) through _run_summary and _difficulty_cells: 73.3 / 50.0 / +10.0 on main, 60.0 / 50.0 / +10.0 on this head.
• The six new tests fail on main and pass on this head.
• A stored delta beside a missing score cannot reach _run_summary: extract_scores sets the delta only when both scores are present, so pairing on delta matches the dashboard's pairing on scores.
• One-arm runs, a group with both arms scored and no pair, and empty groups on both trees.
• make ci-local on Linux at 347bb9c: 26 passed, 2 failed, 7 skipped. pip-audit fails on the filelock advisories, as on main. tests/test_process.py::test_sighup_kills_the_live_groups_too timed out under the parallel run and passes alone 3/3 on this head and 3/3 on main, so it is a load-sensitive flake outside this diff.

Not looked at: the 3.11, 3.12 and 3.14 legs locally (the PR's CI ran them).