Skip to content

feat(evaluation): score CAR concepts from groundtruth_eval.json - #631

Open
ShuxinLin wants to merge 1 commit into
fix/restore-car-term-scorerfrom
feat/car-groundtruth-eval
Open

ShuxinLin wants to merge 1 commit into
fix/restore-car-term-scorerfrom
feat/car-groundtruth-eval

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Stacked on #630. Merge #630 first, then retarget this PR to aafeedback_changes.

Description

CAR scenarios score against the curated concepts in their groundtruth_eval.json instead of terms scraped from the gold sentence. This is a port of the CAR part of d1a668c (#617, closed unmerged). The numeric and time relaxations in that commit are not included.

  • Concepts: each entry is {"concept", "any_of": [...]} or a plain string. A concept is covered when any of its surface forms appears as whole words in the answer, so paraphrases are accepted. Without the file, concepts are extracted from the gold value as before.
  • Wrong mode: concepts are read only when the mode key is right, so a wrong-mode answer scores 0. Before this change, it earned partial credit for echoing terms.
  • Pass: right mode + every required concept.
  • Score: f1 is the F1 over the mode key plus the required concepts, (1 + k) / (1 + n), the same counting as every other static_json scenario. d1a668c replaced f1 with car_score. That made f1 >= 0.5 equivalent to "picked the right mode", so this PR keeps car_score as a reported field only.
  • mode field: checked against the gold key, never used instead of it. A mismatch is reported as mode_spec_matches_gold (with an average in the report).
  • Optional concepts: reported only, never scored.
  • Wiring: StaticJsonScorer now passes scenario.evaluation_metadata to the scorer. The loader and the Harbor task generator already carried groundtruth_eval.json.

Type of Change

  • New Benchmark Scenario (Industry/Asset type)
  • Evaluation Metric / Scorer
  • Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
  • Infrastructure / Tooling Improvement

Impact

This takes effect only for scenarios whose folder holds a groundtruth_eval.json. The private suite's CAR files (scenarios 151–200) were restored unchanged from before their removal on 2026-10-07. Harbor tasks need regenerating so tests/scenarios/ carries them.

CAR results on the aa_v1 runs (15 scenarios × 3 attempts), with saved answers re-scored offline:

Run right mode strict pass: #630 → this PR f1 ≥ 0.5: #630 → this PR mean f1: #630 → this PR
aws-claude-opus-5 high 48.9% 8.9% → 26.7% 42.2% → 48.9% 0.362 → 0.426
azure-gpt-5.6-sol xhigh 44.4% 0.0% → 4.4% 17.8% → 37.8% 0.224 → 0.301
gcp-gemini-3.6-flash high 51.1% 8.9% → 20.0% 35.6% → 48.9% 0.311 → 0.412
MiniMax-M3 42.2% 6.7% → 8.9% 17.8% → 35.6% 0.274 → 0.306
deepseek-v4.1-flash max 46.7% 2.2% → 20.0% 24.4% → 44.4% 0.294 → 0.377
grok-4.7 53.3% 0.0% → 20.0% 26.7% → 51.1% 0.262 → 0.414
glm-5.3 46.7% 0.0% → 17.8% 28.9% → 44.4% 0.298 → 0.364

Related Issues

Testing & Validation

  • Unit Tests: uv run pytest src/evaluation -k "not integration" passes (118). uv run pytest src/ -k "not integration": 760 passed. The 2 failures in src/observability/tests/test_file_exporter.py also fail on the base branch (missing google.protobuf locally).
  • Scenario Validation: loaded the 15 aa_v1 CAR scenario folders with their groundtruth_eval.json through load_scenarios and scored all 312 saved answers with StaticJsonScorer. Pass and score match d1a668c's verdict and exact_f1 on every answer.
  • Data Integrity: no scenario content. The tests and docs use the synthetic "main unit" example.

Checklist

  • My code follows the project's Ruff formatting and linting rules. (The remaining findings, SIM118 and ISC004 in static_json.py and UP017 in report.py, are on lines this PR does not touch.)
  • I have performed a self-review of my code.
  • I have updated the documentation (README or /docs) accordingly.
  • I have signed off my commits (DCO).

A CAR scenario's groundtruth_eval.json now supplies its required and
optional concepts. Each concept is {"concept", "any_of": [...]} or a
plain string, and is covered when any surface form appears as whole
words in the answer. Without the file, concepts are extracted from the
gold value as before.

Ported from the CAR part of d1a668c (#617, closed unmerged):
- concepts are read only when the mode key is right, so a wrong-mode
  answer scores 0 instead of earning credit for echoed terms;
- pass needs the mode plus every required concept;
- the file's "mode" is checked against the gold key, never used in its
  place, and reported as mode_spec_matches_gold;
- optional concepts and car_score are reported, never scored.

Unlike d1a668c, f1 stays the F1 over the mode key plus required
concepts, (1 + k) / (1 + n), the same counting as every other
static_json scenario. StaticJsonScorer now passes
scenario.evaluation_metadata through; the loader and the Harbor task
generator already carry groundtruth_eval.json.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@DhavalRepo18

Copy link
Copy Markdown
Collaborator

@ShuxinLin Please put the PR against aa_feedback and then reach out to them for the evaluation part only.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants