[mohsenhariri]

Feature design: #1440.

Bayes@N adds posterior uncertainty to repeated generative evaluation. A task selects Metrics.bayes_at_n(sample_params={"n": 16}) to score 16 candidates per question and report bayes@n (posterior mean) and bayes@n_sigma (posterior standard deviation). Metrics.bayes_at_n_math uses the existing math answer extraction.

Each question has an independent categorical Dirichlet posterior with one pseudo-count per category. The corpus result is the mean score over the fixed question set. Binary exact match is the default; custom categorical scorers provide category weights. Optional prior outcomes are attached to the current document through a callable. The native NumPy implementation follows Algorithm 1 of Don't Pass@k and the MIT-licensed Scorio reference, without adding a dependency.

The collectors reject missing generations, ragged outcome rows, invalid category ids and inconsistent weights or prior availability. Reward scaling avoids overflow in posterior variances. Grouped metric names accept CLI sampling parameters. The metric documentation includes task configuration, categorical scoring and prior examples.

Task and suite summaries retain Lighteval's equal-task mean. For the Bayes SD, they use sqrt(sum(sigma_t**2))/number_of_tasks, assuming independent, non-overlapping question sets. Per-task means and SDs remain available when that assumption does not hold. The metric adds neither a bootstrap stderr nor a credible interval.

Validation uses the checkout's Python 3.13.11 .venv with .[dev] and CPU PyTorch:

• python -m pytest -q tests/unit/metrics/test_bayes_at_n.py tests/unit/metrics/test_normalizations.py: 48 passed. Cases include exact rational binary results, an independent categorical Fraction oracle, prior alignment through serialization/reordering, registered math scoring, Registry CLI parameters and real task/logger aggregation, pipeline metric execution and JSON/Parquet export.
• 200 categorical/prior/large-scale cases match the pinned Scorio reference to rtol=1e-13.
• make quality: passed, with existing suppression-comment warnings in test_evaluation_tracker.py.
• HF_DATASETS_CACHE=/tmp/lighteval-datasets python -m pytest -q tests/unit/logging/test_evaluation_tracker.py: 8 passed, 1 skipped and 133 passed subtests.
• Repository-pinned pre-commit hooks passed on all changed files.
• A wider nearby suite produced 55 passed, 1 skipped and 133 passed subtests; 10 failed because the host's default Hugging Face/model cache paths are read-only. The identical 10 tests fail on unchanged base 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7. Full GPU/model/download suites were not run.

[mohsenhariri]

Please review the metric API linked in #1440 and enable the pending fork workflows for this PR when appropriate. Tests, Quality, Build PR Documentation and Slow end to end tests currently need approval.

The refreshed head 75b58fd exposes Metrics.bayes_at_n(sample_params={"n": 16}) and carries posterior mean and SD through task/suite logging, with categorical scoring and per-document priors. The local validation recorded in the PR passed 48 metric/normalization tests and make quality; full GPU/model suites were not run.