Temperature Scaling

The problem

A model that is overconfident says 98% when it is right only 75% of the time. Temperature scaling (Guo et al., 2017) is the simplest fix: divide every logit by one number $T > 0$ before the softmax. With $T > 1$ the probabilities get softer. It does not change which logit is largest, so the predictions, and the accuracy, stay exactly the same: only the confidences change. $T$ is chosen on a held-out validation set, to minimize the NLL (negative log-likelihood, section 2).

Example. Eight examples; the logits are $(4, 0)$ or $(0, 4)$ and every prediction has confidence $0.982$. The right classes make the model right on 6 of 8: accuracy $0.75$.

$T$	confidence	NLL
1	$0.982$	$1.02$
2	$0.88$	$0.63$
5	$0.69$	$0.57$
$3.64$ (the best)	$0.75$	$0.56$

At the best temperature the confidence is exactly the accuracy.

Finding $T$: golden-section search

The NLL as a function of $T$ has one minimum here. Golden-section search finds the minimum of such a function by shrinking an interval $[a, b]$ that contains it:

1. evaluate two interior points $x_1 = b - \varphi(b - a)$ and $x_2 = a + \varphi(b - a)$, with $\varphi = (\sqrt 5 - 1)/2 \approx 0.618$;
2. if $f(x_1) < f(x_2)$ the minimum is to the left of $x_2$: set $b = x_2$; otherwise set $a = x_1$;
3. repeat; after iters iterations return $(a + b)/2$.

Each iteration shrinks the interval to $0.618$ of its length. After 60 iterations it is $10^{-13}$ of the original.

What to implement

• nll(logits, labels, T): the mean negative log-likelihood of softmax(logits / T) against integer labels, computed stably.
• fit_temperature(logits, labels, lo=0.05, hi=10.0, iters=60): golden-section search on $[lo, hi]$.
• confidences(logits, T): the maximum softmax probability of each row at temperature $T$.

Requirements

• Stable log-softmax (subtract the row maximum).