[mottopanikeiku]

The aimo_progress_prize_1 task passes math_normalizer as both normalize_gold and normalize_pred. The gold is the plain integer answer (str(line["answer"])), and math_normalizer only keeps the content of the last \boxed{}. When there is no box it returns "", so every gold becomes "". A non-empty answer without a box then matches the gold and scores 1, and a boxed answer scores 0 even when it is correct.

This PR keeps math_normalizer on the prediction only, so the boxed answer is compared with the integer gold. The Inspect path (lighteval eval) uses math_scorer and is not affected.

Reproducer:

from lighteval.models.model_output import ModelResponse
from lighteval.tasks.tasks.aimo import aimo_prompt, task

line = {"problem": "...", "answer": 52}  # problem 229ee8 of lighteval/aimo_progress_prize_1
doc = aimo_prompt(line, task_name="aimo_progress_prize_1")
[metric] = task.metrics
for text in ["So the answer is $\\boxed{52}$.", "So the answer is $\\boxed{40}$.", "So the answer is 40."]:
    score = metric.compute_sample(doc=doc, model_response=ModelResponse(text=[text]))
    print(repr(text), list(score.values())[0])

prediction (gold 52)	main (7fa19dc)	this PR
So the answer is $\boxed{52}$.	0	1
So the answer is $\boxed{40}$.	0	0
So the answer is 40.	1	0

I also ran lighteval accelerate "model_name=<model>,device=cpu" "aimo_progress_prize_1|0" --save-details on all 10 problems and re-scored the saved answers with this branch:

model	em on main	em with this PR	answers that are correct
HuggingFaceTB/SmolLM2-135M-Instruct	1.00	0.00	0/10 (no answer is boxed)
Qwen/Qwen2.5-0.5B-Instruct	0.10	0.00	0/10 (the 0.10 is the one answer cut off at 2048 tokens before boxing)

Writing each reference answer as The answer is $\boxed{<answer>}$. scores 0/10 on main and 10/10 with this change.

The metric key becomes em:normalize_pred=<function math_normalizer ...>.

I added tests/unit/tasks/test_aimo.py. Two of its three cases fail on main, and all three pass with this change.

I found this by reading and testing the code.

I made this PR and directed its development, with AI assistance from Claude and GPT-6 Astra.