[ningyixi]

The bootstrap path reports standard errors that are too small and silently drops requested resamples. bootstrap_stderr passes the bootstrap metric values to mean_stderr, dividing their sample standard deviation by sqrt(B). Their standard deviation already estimates the metric's standard error, so at the default 1,000 iterations this understates uncertainty by about 31.6 times, and quadrupling iterations halves the reported error. Separately, requests for 1,001 or 1,500 resamples currently evaluate only 1,000.

Return _stddev(res), matching the bootstrap definition and lm-eval's sample_stddev(res). Round the chunk count up and limit the final chunk to the remaining draws. Full chunks retain their existing seeds and sizes. The analytic mean_stderr path for mean aggregations is unchanged.

The tests call the public stderr-function factory and use real multiprocessing workers. A shared recording metric verifies the exact draw count and compares the result with statistics.stdev of the actual replicates. An independent Bernoulli-accuracy check compares 1,000- and 4,000-iteration estimates with the known sqrt(p*(1-p)/n) sampling error. A separate control protects the analytic mean path.

Validation (Python 3.12.14, CPU Torch 2.14.1):

• On original upstream, the 1,001- and 1,500-draw regressions fail because only 1,000 evaluations run.
• Before correcting the formula, the updated stderr tests give 6 failed, 1 passed. For 500 balanced binary samples, the old result is about 0.00069 (1,000 iterations) and 0.00034 (4,000), versus analytic sampling error 0.02236.
• After both fixes, 7 stderr tests pass.
• Expanded metric/base-metric/harness-reference/local-logging validation: 552 passed, 4 skipped, 5 deselected. All five excluded failures were reproduced with unchanged upstream stderr.py: two BLEU fixtures need NLTK tokenizer data unavailable through the configured proxy; extractiveness and two dynamic multilingual tests need the optional multilingual extra. No other failures in the expanded run.
• make quality (341 files), pre-commit run --all-files, and git diff --check: passed. The real harness reference fixture was fetched through Git LFS.

The full repository/GPU suite was not run. No model API calls, Hub uploads, or dependency changes. Bootstrap-derived _stderr values intentionally increase to their proper scale; metric scores and the analytic mean-error formula remain unchanged.

Prepared and verified with OpenAI Codex assistance. No human approval is claimed.

[HarshShroff]

I ran this at f875198 (macOS arm64, Python 3.12). tests/unit/metrics/test_stderr.py gives 4 passed, and with main's stderr.py swapped back in the 1001 and 1500 cases fail, so the remainder fix does what it says.

While checking the numbers I think the bootstrap path has a bigger, pre-existing problem in the line right after the loop: bootstrap_stderr returns mean_stderr(res), which is stdev(res) / sqrt(len(res)). res holds the bootstrap replicates of the metric, so their standard deviation is already the bootstrap standard error. Dividing by sqrt(B) gives the standard error of the mean of the replicates, which goes to zero as you add iterations. lm-eval's bootstrap_stderr, which this file says it was taken from, returns sample_stddev(res) with no division.

Quick check with get_stderr_function (on this branch), plus a plain reference bootstrap (stdev of B replicates):

metric	B	lighteval	plain bootstrap
accuracy as a non-"mean" aggregation, n=500, p=0.69 (analytic SE 0.0207)	1000	0.00067	0.0212
same	4000	0.00034	0.0209
CorpusLevelF1Score(average=None).compute_corpus, n=300	1000	0.00076	0.0239
same	4000	0.00038	0.0238

So any corpus-level metric that goes through the bootstrap (anything whose aggregation name doesn't contain "mean") reports a stderr about 31.6x too small at the default 1000, and it halves every time bootstrap_iters quadruples. That also means this PR changes 1500-iteration results downward (0.00067 to about 0.00055) rather than making them more accurate.

Changing that return to _stddev(res) gives 0.0212 / 0.0212 and 0.0239 / 0.0240 for the rows above. tests/unit/metrics is otherwise unchanged with it (the extractiveness.json case in test_automated_metrics_pytest.py fails with or without it). The new test here would need its expected value changed to statistics.stdev(draws), since it currently asserts the divided-by-sqrt(n) formula; I didn't run the slow reference-score tests, so I don't know whether any stored _stderr values come from this path. Happy for that to be a separate PR, but it seemed worth raising before the test locks in the current formula.