Skip to content

fix(evaluation): restore term-based CAR scoring - #630

Merged
DhavalRepo18 merged 2 commits into
aafeedback_changesfrom
fix/restore-car-term-scorer
Oct 13, 2026
Merged

DhavalRepo18 merged 2 commits into
aafeedback_changesfrom
fix/restore-car-term-scorer

Conversation

@ShuxinLin

@ShuxinLin ShuxinLin commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Reverts the CAR scorer change from #627 (f8805be). A CAR answer again passes only when it is a single-key object whose key matches the gold mode and whose value contains every required term extracted from the gold value. The score is the F1 over the mode key plus those terms.

Fix Details

  • src/evaluation/scorers/static_json.py: restored byte-for-byte to its pre-f8805be state.
  • src/evaluation/tests/test_static_json_scorer.py: the pre-f8805be CAR tests describe the groundtruth_eval.json metadata scoring added in 79d6356 (car_score = 0.6·mode + 0.4·coverage, optional terms, pass on any one required term). 8f1e538 took that out of the scorer two days later but left the tests, so 6 of them fail against this code. Replaced them with tests of what the code actually does: a missing term fails, every term is required, a wrong key still earns partial F1 from terms, the key is case-insensitive, a non-object answer scores 0, and the StaticJsonScorer wrapper reports pass/score.
  • docs/static-json-evaluation.md: CAR section rewritten to match the restored behaviour. The old text still described the 79d6356 metadata scoring. No scenario in the aa_v1 suite ships a groundtruth_eval.json, so CAR terms come from groundtruth.txt either way.

Impact on Benchmarking

  • No change to baselines: This fix only improves stability/performance.
  • Baseline change: This fix corrects a scoring error. (Please provide "Before vs. After" results).

Every aa_v1 job graded before 9584e28 used this scorer. Re-scoring their saved CAR answers with it reproduces the recorded f1 exactly (0 mismatches across 223 trials in 5 jobs). Restoring it puts later runs back on the same scale.

CAR, 15 scenarios × 3 attempts, saved answers re-scored against the current gold:

Run pass, mode-key only (#627) pass, restored mean f1, mode-key only mean f1, restored
aws-claude-opus-5 high 48.9% 8.9% 0.489 0.362
azure-gpt-5.6-sol xhigh 44.4% 0.0% 0.444 0.224
gcp-gemini-3.6-flash high 51.1% 8.9% 0.511 0.311
MiniMax-M3 42.2% 6.7% 0.422 0.274
deepseek-v4.1-flash max 46.7% 2.2% 0.467 0.294
grok-4.7 53.3% 0.0% 0.533 0.262
glm-5.3 46.7% 0.0% 0.467 0.298

Only the gpt-5.6-sol xhigh job was graded with the #627 scorer, so its stored CAR rewards need re-verifying after this merges.

Related Issues

Verification Steps

  1. uv run pytest src/evaluation -k "not integration": 111 passed.
  2. uv run pytest src/ -k "not integration": 753 passed. 2 failures in src/observability/tests/test_file_exporter.py also fail on the base branch (missing google.protobuf locally).
  3. Re-scored all saved aa_v1 CAR answers with the restored scorer and compared with the recorded verifier reports, as described above.

Checklist

  • I have added tests that prove my fix is effective.
  • My code follows the project's Ruff formatting and linting rules. (The two remaining ruff findings in static_json.py, SIM118 and ISC004, are in the restored code and also present on the base branch.)
  • I have signed off my commits (DCO).

Reverts the scorer change from f8805be. A CAR answer again passes only
when it is a single-key object whose key matches the gold mode and whose
value contains every required term extracted from the gold value; the
score is the F1 over the mode key plus those terms.

This is the scorer every aa_v1 job before 9584e28 was graded with.
Re-scoring their saved CAR answers with it reproduces the recorded f1
exactly, so restoring it puts the 9584e28+ runs back on the same scale.

The pre-f8805be CAR tests and docs describe the groundtruth_eval.json
metadata scoring added in 79d6356 (car_score = 0.6 mode + 0.4 coverage,
optional terms, pass on any one required term). 8f1e538 took that out of
the scorer two days later but left the tests and docs, so six tests fail
against this code. They are replaced with tests of the behaviour the
code has, and the docs are corrected to match.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin
ShuxinLin force-pushed the fix/restore-car-term-scorer branch from 906c411 to f091eb4 Compare October 13, 2026 03:34
The suite's CAR groundtruth_eval.json files were removed on 2026-10-07,
and static_json never read them.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@DhavalRepo18
DhavalRepo18 merged commit 684827c into aafeedback_changes Oct 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants