Repository navigation
fix(mcq): load JSON prediction tables for evaluation - #1766
rudycelekli wants to merge 2 commits into
Conversation
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
|
@rudycelekli Thank you for your contribution. |
load_prediction_file is for inference results only. Evaluation-written intermediate tables are reloaded with load_table_file, which does not require a prediction column and only restores JSON record lists to a DataFrame. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks, agreed. Pushed 97bc271: |
With PRED_FORMAT=json, native prediction exports contain record lists. MCQ evaluators read these through ordinary load(), then call DataFrame methods; scored prediction tables saved as JSON fail again when reloaded. Use the existing prediction-table reader for the tabular reads in text/image MCQ evaluation and their image overrides. Judge-cache dictionaries and ordinary JSON loading retain their existing semantics.
This complements #1172: its format/filename changes leave these MCQ reader calls unchanged. Merged #1729 wires the table-aware reader into inference resume; this change covers evaluation consumers. One grouped change covers31 source-audited table read sites.
Validation: native local TSV dataset constructors, actual prediction exports, original exact matcher/scorer producing native resume caches, and full cached evaluation/reporting. Six JSON input/reload cases fail against the base while eight controls pass. All14 focused cases pass after the fix; existing JSON-resume/file/NumPy neighbors bring the total to67 passes. XLSX, TSV, physical Pillow image prompts, generic text/image MCQ, and a local CVBench subtype are covered. Ordinary JSON dict/list return types remain unchanged. All nine pinned pre-commit hooks pass on materialized source/metadata.
Tests isolate unrelated optional model imports and execute exact shipping definitions. No model/provider calls, GPU benchmarks, downloaded datasets, benchmark accuracy, hosted CI, or validation of every specialized scoring branch is claimed. AI assistance: prepared with Codex and verified using native local fixtures.