Error Bars for Benchmark Scores

The problem

A model scored 62% on 200 questions, another 60%. Is the first one really better? A score measured on a finite set of questions is an estimate: a different set of 200 questions would give a slightly different score. The bootstrap measures that uncertainty without formulas: pretend the questions you have are the whole population, resample them with replacement many times (draw 200 questions out of 200: some twice, some not at all), compute the score of each resample, and look at how much it varies.

Confidence interval of one score

bootstrap_ci(correct, rng, n_boot=2000, level=0.95): correct is a 0/1 array, one entry per question.

1. Draw n_boot resamples of the question indices, all at once: idx = rng.integers(0, n, size=(n_boot, n)).
2. Compute the accuracy of each resample.
3. Return (accuracy, low, high): the accuracy on the original data, and the (1 - level) / 2 and (1 + level) / 2 quantiles of the resampled accuracies (np.quantile): for level = 0.95, the 2.5th and 97.5th percentiles.

Example. Model A is right on 124 of 200 questions: accuracy $0.62$, interval about $[0.555, 0.685]$. Model B, 120 of 200: $0.60$, about $[0.535, 0.665]$. The intervals overlap almost entirely.

Comparing two models on the same questions

paired_bootstrap(correct_a, correct_b, rng, n_boot=2000, level=0.95) does the same for the difference of accuracies, A minus B, using the same resampled indices for both models (idx is drawn once). The questions are shared, so a question both models get right or both get wrong adds nothing to the difference: only the questions where they disagree matter, and pairing removes most of the noise.

significant(low, high) is True when the interval excludes 0: the data shows a difference.

Example. The same models. They disagree on only 8 questions: A is right and B wrong on 6, the reverse on 2. The difference is $0.62 - 0.60 = +0.02$, and the paired interval is about $[-0.005, +0.05]$. It contains 0: significant is False. A two-point gap on 200 questions is within the noise.

What to implement

• bootstrap_ci(correct, rng, n_boot=2000, level=0.95): (accuracy, low, high).
• paired_bootstrap(correct_a, correct_b, rng, n_boot=2000, level=0.95): (difference, low, high).
• significant(low, high): whether the interval excludes 0.

Requirements

• Use the given random draws exactly, so the tests can reproduce the intervals.
• No Python loops over resamples.