Skip to content

fix(mcq): load JSON prediction tables for evaluation - #1766

Open
rudycelekli wants to merge 2 commits into
open-compass:mainfrom
rudycelekli:fix/mcq-json-prediction-evaluation-20261013
Open

rudycelekli wants to merge 2 commits into
open-compass:mainfrom
rudycelekli:fix/mcq-json-prediction-evaluation-20261013

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

With PRED_FORMAT=json, native prediction exports contain record lists. MCQ evaluators read these through ordinary load(), then call DataFrame methods; scored prediction tables saved as JSON fail again when reloaded. Use the existing prediction-table reader for the tabular reads in text/image MCQ evaluation and their image overrides. Judge-cache dictionaries and ordinary JSON loading retain their existing semantics.

This complements #1172: its format/filename changes leave these MCQ reader calls unchanged. Merged #1729 wires the table-aware reader into inference resume; this change covers evaluation consumers. One grouped change covers31 source-audited table read sites.

Validation: native local TSV dataset constructors, actual prediction exports, original exact matcher/scorer producing native resume caches, and full cached evaluation/reporting. Six JSON input/reload cases fail against the base while eight controls pass. All14 focused cases pass after the fix; existing JSON-resume/file/NumPy neighbors bring the total to67 passes. XLSX, TSV, physical Pillow image prompts, generic text/image MCQ, and a local CVBench subtype are covered. Ordinary JSON dict/list return types remain unchanged. All nine pinned pre-commit hooks pass on materialized source/metadata.

Tests isolate unrelated optional model imports and execute exact shipping definitions. No model/provider calls, GPU benchmarks, downloaded datasets, benchmark accuracy, hosted CI, or validation of every specialized scoring branch is claimed. AI assistance: prepared with Codex and verified using native local fixtures.

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
@ssiq

ssiq commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

@rudycelekli Thank you for your contribution. load_prediction_file should only be used to load inference result files. It should not be used for files generated during evaluation, as those files are not guaranteed to contain a DataFrame or a prediction column.

load_prediction_file is for inference results only. Evaluation-written
intermediate tables are reloaded with load_table_file, which does not require a
prediction column and only restores JSON record lists to a DataFrame.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@rudycelekli

Copy link
Copy Markdown
Contributor Author

Thanks, agreed. Pushed 97bc271: load_prediction_file is now used only for reading the inference output (eval_file). The tables that evaluation itself writes and reloads (the _result / _vanilla_*_result / eval-record files) go through a new load_table_file in vlmeval/smp/file.py, which does not require a prediction column and only turns the record list that JSON serialization produces back into a DataFrame; any other loaded value is returned unchanged. The test now reloads the scored table with it. Local run: tests/test_mcq_prediction_tables.py and tests/test_json_resume.py pass (18 passed); full CI not yet re-checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants