Expected Calibration Error

The problem

When a model says it is 80% sure, is it right 80% of the time? A model whose confidence matches its accuracy is calibrated. Modern networks are often overconfident: they say 95% and are right 75% of the time. The ECE (expected calibration error) measures the mismatch with one number.

For $n$ predictions with confidences $c_i \in [0, 1]$ (the probability of the predicted class) and correctness $y_i \in \{0, 1\}$, split $[0, 1]$ into $M$ equal-width bins and compare, in each bin, the accuracy (the fraction right) with the mean confidence:

where $B_m$ is the set of predictions in bin $m$, $|B_m|$ its size, and each gap is weighted by how many predictions the bin holds.

The bins

Bin $m$ (counting from 0) holds the confidences in $(m/M, (m+1)/M]$: the left end is excluded and the right end included, and confidence exactly 0 goes in bin 0. So with $M = 10$: $c = 0.1$ is in bin 0, $c = 0.6$ in bin 5, $c = 1.0$ in bin 9. The bin of a confidence is $\lceil cM \rceil - 1$, clipped to $[0, M-1]$ (so that 0 does not give $-1$).

Example. Six predictions with $M = 5$ bins of width $0.2$: confidences $0.95, 0.90, 0.85, 0.60, 0.55, 0.30$ and correctness $1, 0, 1, 1, 0, 0$.

• 2: $(0.4, 0.6]$ — holds: $0.60, 0.55$; count: 2; accuracy: $0.5$; mean confidence: $0.575$; gap: $0.075$
• 1: $(0.2, 0.4]$ — holds: $0.30$; count: 1; accuracy: $0$; mean confidence: $0.3$; gap: $0.3$
• 4: $(0.8, 1]$ — holds: $0.95, 0.90, 0.85$; count: 3; accuracy: $0.667$; mean confidence: $0.9$; gap: $0.233$

(Bins 0 and 3 are empty: count 0, accuracy and confidence 0, no contribution.) ECE $= \tfrac{2}{6} \cdot 0.075 + \tfrac{1}{6} \cdot 0.3 + \tfrac{3}{6} \cdot 0.233 = 0.192$.

What to implement

• bin_index(conf, n_bins): the bin of each confidence.
• reliability(conf, correct, n_bins): three arrays of length $M$: the count, the accuracy and the mean confidence of each bin (accuracy and confidence are 0 for empty bins).
• ece(conf, correct, n_bins=15): the expected calibration error. Empty bins contribute nothing.

Requirements

• No loops over predictions.

Hint

np.ceil(conf * n_bins) - 1, as an integer and clipped to $[0, M-1]$, gives the bin; np.bincount(bins, weights=values, minlength=n_bins) sums values per bin, and without weights it counts.