Repository navigation
Conversation
A CAR scenario's groundtruth_eval.json now supplies its required and
optional concepts. Each concept is {"concept", "any_of": [...]} or a
plain string, and is covered when any surface form appears as whole
words in the answer. Without the file, concepts are extracted from the
gold value as before.
Ported from the CAR part of d1a668c (#617, closed unmerged):
- concepts are read only when the mode key is right, so a wrong-mode
answer scores 0 instead of earning credit for echoed terms;
- pass needs the mode plus every required concept;
- the file's "mode" is checked against the gold key, never used in its
place, and reported as mode_spec_matches_gold;
- optional concepts and car_score are reported, never scored.
Unlike d1a668c, f1 stays the F1 over the mode key plus required
concepts, (1 + k) / (1 + n), the same counting as every other
static_json scenario. StaticJsonScorer now passes
scenario.evaluation_metadata through; the loader and the Harbor task
generator already carry groundtruth_eval.json.
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Collaborator
|
@ShuxinLin Please put the PR against aa_feedback and then reach out to them for the evaluation part only. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CAR scenarios score against the curated concepts in their
groundtruth_eval.jsoninstead of terms scraped from the gold sentence. This is a port of the CAR part of d1a668c (#617, closed unmerged). The numeric and time relaxations in that commit are not included.{"concept", "any_of": [...]}or a plain string. A concept is covered when any of its surface forms appears as whole words in the answer, so paraphrases are accepted. Without the file, concepts are extracted from the gold value as before.f1is the F1 over the mode key plus the required concepts,(1 + k) / (1 + n), the same counting as every otherstatic_jsonscenario. d1a668c replacedf1withcar_score. That madef1 >= 0.5equivalent to "picked the right mode", so this PR keepscar_scoreas a reported field only.modefield: checked against the gold key, never used instead of it. A mismatch is reported asmode_spec_matches_gold(with an average in the report).StaticJsonScorernow passesscenario.evaluation_metadatato the scorer. The loader and the Harbor task generator already carriedgroundtruth_eval.json.Type of Change
Impact
This takes effect only for scenarios whose folder holds a
groundtruth_eval.json. The private suite's CAR files (scenarios 151–200) were restored unchanged from before their removal on 2026-10-07. Harbor tasks need regenerating sotests/scenarios/carries them.CAR results on the aa_v1 runs (15 scenarios × 3 attempts), with saved answers re-scored offline:
Related Issues
Testing & Validation
uv run pytest src/evaluation -k "not integration"passes (118).uv run pytest src/ -k "not integration": 760 passed. The 2 failures insrc/observability/tests/test_file_exporter.pyalso fail on the base branch (missinggoogle.protobuflocally).groundtruth_eval.jsonthroughload_scenariosand scored all 312 saved answers withStaticJsonScorer. Pass and score match d1a668c's verdict andexact_f1on every answer.Checklist
static_json.pyand UP017 inreport.py, are on lines this PR does not touch.)