Repository navigation
fix(evaluation): restore term-based CAR scoring - #630
Merged
Merged
Conversation
Reverts the scorer change from f8805be. A CAR answer again passes only when it is a single-key object whose key matches the gold mode and whose value contains every required term extracted from the gold value; the score is the F1 over the mode key plus those terms. This is the scorer every aa_v1 job before 9584e28 was graded with. Re-scoring their saved CAR answers with it reproduces the recorded f1 exactly, so restoring it puts the 9584e28+ runs back on the same scale. The pre-f8805be CAR tests and docs describe the groundtruth_eval.json metadata scoring added in 79d6356 (car_score = 0.6 mode + 0.4 coverage, optional terms, pass on any one required term). 8f1e538 took that out of the scorer two days later but left the tests and docs, so six tests fail against this code. They are replaced with tests of the behaviour the code has, and the docs are corrected to match. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
ShuxinLin
force-pushed
the
fix/restore-car-term-scorer
branch
from
October 13, 2026 03:34
906c411 to
f091eb4
Compare
The suite's CAR groundtruth_eval.json files were removed on 2026-10-07, and static_json never read them. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
8 of 11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reverts the CAR scorer change from #627 (f8805be). A CAR answer again passes only when it is a single-key object whose key matches the gold mode and whose value contains every required term extracted from the gold value. The score is the F1 over the mode key plus those terms.
Fix Details
src/evaluation/scorers/static_json.py: restored byte-for-byte to its pre-f8805be state.src/evaluation/tests/test_static_json_scorer.py: the pre-f8805be CAR tests describe thegroundtruth_eval.jsonmetadata scoring added in 79d6356 (car_score = 0.6·mode + 0.4·coverage, optional terms, pass on any one required term). 8f1e538 took that out of the scorer two days later but left the tests, so 6 of them fail against this code. Replaced them with tests of what the code actually does: a missing term fails, every term is required, a wrong key still earns partial F1 from terms, the key is case-insensitive, a non-object answer scores 0, and theStaticJsonScorerwrapper reports pass/score.docs/static-json-evaluation.md: CAR section rewritten to match the restored behaviour. The old text still described the 79d6356 metadata scoring. No scenario in the aa_v1 suite ships agroundtruth_eval.json, so CAR terms come fromgroundtruth.txteither way.Impact on Benchmarking
Every aa_v1 job graded before 9584e28 used this scorer. Re-scoring their saved CAR answers with it reproduces the recorded f1 exactly (0 mismatches across 223 trials in 5 jobs). Restoring it puts later runs back on the same scale.
CAR, 15 scenarios × 3 attempts, saved answers re-scored against the current gold:
Only the gpt-5.6-sol xhigh job was graded with the #627 scorer, so its stored CAR rewards need re-verifying after this merges.
Related Issues
Verification Steps
uv run pytest src/evaluation -k "not integration": 111 passed.uv run pytest src/ -k "not integration": 753 passed. 2 failures insrc/observability/tests/test_file_exporter.pyalso fail on the base branch (missinggoogle.protobuflocally).Checklist
static_json.py, SIM118 and ISC004, are in the restored code and also present on the base branch.)