[mottopanikeiku]

The built-in triviaqa task lowercases the gold aliases and strips their punctuation in triviaqa_prompt. Its metric is plain Metrics.exact_match, which only strips whitespace from the prediction. So a correct answer scores 0 whenever it has a capital letter or punctuation. For example, John Nash does not match the alias john nash.

The prediction normalizer was lost when the tasks were moved to individual files in #1016. Before that, the task used Metrics.exact_match(sample_params={"strip_strings": True, "normalize_pred": harness_triviaqa_normalizer}). This PR restores that line. lm-eval-harness also scores TriviaQA with ignore_case and ignore_punctuation.

Reproducer:

from lighteval.models.model_output import ModelResponse
from lighteval.tasks.tasks.triviaqa import triviaqa, triviaqa_prompt

line = {
    "question": "Name the pioneering Game Theory US mathematician featured in the film A Beautiful Mind?",
    "answer": {"aliases": ["John Nash (disambiguation)", "Johnny Nash (disambiguation)", "John nash", "John Nash", "Nash, John"]},
}
doc = triviaqa_prompt(line, task_name="triviaqa")
[metric] = triviaqa.metrics
print(metric.compute_sample(doc=doc, model_response=ModelResponse(text=[" John Nash"])))

	main (7fa19dc)	this PR
John Nash (validation question bb_5713)	0	1
1599-1660 (validation question wh_4325)	0	1
lighteval accelerate "model_name=HuggingFaceTB/SmolLM2-135M,device=cpu" "triviaqa|0" --max-samples 1000	em 0.003	em 0.039

The model outputs were the same in both runs. Scoring those 1000 outputs with the lm-eval-harness rule gives 0.042. The remaining 3 differences come from _remove_prefixes in the prompt function, which this PR does not change.

Like other tasks that pass a normalizer function in sample_params, the metric key is now em:strip_strings=True&normalize_pred=<function harness_triviaqa_normalizer ...>, as it was before #1016.

I added tests/unit/tasks/test_triviaqa.py. Two of its three cases fail on main, and all three pass with this change.

I found this by reading and testing the code.

I made this PR and directed its development, with AI assistance from Claude and GPT-6 Astra.