[CY-HYUN]

Adds KMMLU (HAERAE-HUB/KMMLU), a Korean multiple-choice benchmark built from original Korean exams: 35,030 questions in 45 subjects (arXiv 2402.11548). This follows the proposal in #1424.

What it adds

One file, src/lighteval/tasks/tasks/kmmlu.py: 45 tasks named kmmlu:<subject> (for example kmmlu:accounting), one per HF config, test split, loglikelihood_acc.

The prompt copies the default kmmlu task of lm-evaluation-harness (v0.4.13), so scores from the two tools can be compared: the question, the four options as A. to D., then 정답： (full-width colon), choices " A" to " D", gold answer - 1. The dataset revision is pinned (d61b3f1). In #1424 I asked whether you'd rather have the multilingual template style (get_mcq_prompt_function, like cmmlu). Happy to switch if you prefer it.

Update (2026-10-28): this branch now also has 45 kmmlu_direct:<subject> tasks. They are generative, five-shot, and scored by exact match. @GeonRyoung and team A of the 2026 Open-Source AI Contributor Challenge wrote them. I merged them from CY-HYUN#1. Their PR compares all 45 subjects (35,030 questions) with lm-eval 0.4.13. Prompts, generated text and scores differ in 0 cases. That is on one model, Qwen2.5-0.5B-Instruct.

How I checked it

I ran this task as a --custom-tasks file with lighteval 0.13.0, ran lm-eval 0.4.13 on the same model, and compared each question. Setup: Qwen/Qwen2.5-0.5B-Instruct (revision 7ae5576), float32, CPU, batch size 1, 0-shot, chat template off.

Subject	n	lighteval acc	lm-eval acc	prompts differ	predictions differ	max abs log-prob diff
Health	100	0.3300	0.3300	0	0	5.15e-05
Education	100	0.3600	0.3600	0	0	5.53e-05
Korean-History	100	0.2300	0.2300	0	0	4.01e-05
Accounting	100	0.2500	0.2500	0	0	5.05e-05
Math	300	0.2733	0.2733	0	0	4.24e-05

All 700 questions got the same prompt, tokens and predicted choice. Code and a reproduce.py that prints this table: https://github.com/CY-HYUN/kmmlu-lighteval

Update (2026-10-28): @GeonRyoung, @ojm6135 and @hyeongrae-kim ran the same kmmlu comparison on all 45 subjects on an RTX 4070 Ti SUPER (float32, batch size 1, same model and dataset revisions). All 35,030 questions got the same prompt, tokens and predicted choice. Micro accuracy is 0.3245 and macro 0.3157 in both tools. Log-prob gaps reach 3.48e-03 on that GPU (5.53e-05 on CPU), and no prediction flipped. They ran the task as a custom-task file with lighteval 0.13.0; its 45 kmmlu tasks are defined the same way as in this PR's file. Per-subject table: https://github.com/CY-HYUN/kmmlu-lighteval#all-45-subjects-on-a-gpu

On main (7fa19dc) I checked that the config fields and Metrics.loglikelihood_acc the file uses are unchanged, and ran ruff check and ruff format with the repo's config (ruff 0.11.10). I haven't run it from a source install of main.

Limits

• One small model. kmmlu was compared on 5 subjects on CPU and on all 45 on one GPU (update above).
• Log-likelihood (kmmlu) and generative five-shot (kmmlu_direct) only; kmmlu_hard and chain-of-thought variants are not included.
• lighteval tokenizes prompt and choice separately and lm-eval tokenizes them together, so they agree only when no token crosses the boundary. That held for the Qwen2.5 tokenizer on all 35,030 questions.

I used an AI coding assistant (Claude Code) to write and check this.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[CY-HYUN]

@GeonRyoung @ojm6135 @hyeongrae-kim Thank you for running the full 45-subject kmmlu comparison on your GPU. All 35,030 questions matched lm-eval 0.4.13: same prompt, tokens and predicted choice. I've updated the description and the Limits above.