Mean Squared Error

The problem

When a model predicts real numbers (a price, a reward, the brightness of a pixel), the standard loss is the MSE (mean squared error): for every prediction, take the error, square it, and average all the squares. It is the negative log-likelihood of Gaussian noise (see the theory page), and its gradient is the simplest one in this section.

The loss

For predictions and targets of the same shape, with $n$ numbers in total:

where:

• $\hat y$ ("y hat") is the prediction pred, and $\hat y_i$ is its number $i$;
• $y$ is the target, and $y_i$ is its number $i$;
• $\hat y_i - y_i$ is the error on number $i$, and $i$ runs over all $n$ numbers, whatever the shape.

Squaring does two things: every error becomes positive, so errors in opposite directions cannot cancel out; and large errors count much more than small ones (an error of 2 costs 4, an error of 10 costs 100).

Example. pred = [1.5, 2, 2], target = [1, 2, 3].

1. Errors: $1.5 - 1 = 0.5$, $2 - 2 = 0$, $2 - 3 = -1$.
2. Squares: $0.25$, $0$, $1$.
3. Average: $(0.25 + 0 + 1) / 3 \approx 0.417$.

The gradient

The gradient says, for every prediction, how the loss changes when that prediction moves. Only one term of the sum contains $\hat y_i$, and the derivative of $(\hat y_i - y_i)^2$ is $2(\hat y_i - y_i)$. With the $1/n$ in front:

where the symbols are the same as above.

Example. Same numbers: errors $(0.5, 0, -1)$, $n = 3$, so the gradient is $\tfrac{2}{3}(0.5, 0, -1) \approx (0.333, 0, -0.667)$.

Training moves against the gradient: the first prediction goes down (it was too high), the second stays (it was right), the third goes up (it was too low).

What to implement

• mse(pred, target): the loss, a single number (a Python float or a 0-d array).
• mse_backward(pred, target): the gradient, an array shaped like pred.

Requirements

• Any shape: the mean is over all numbers, not over one axis.
• If the shapes differ, raise a ValueError. Otherwise broadcasting hides a classic bug: pred of shape $(N, 1)$ minus target of shape $(N,)$ is an $N \times N$ table, and the loss is wrong without any error. For example, [[1], [2]] - [1, 2] is [[0, -1], [1, 0]], whose MSE is $0.5$, although the predictions are perfect.
• No Python loops.

Hint

pred.size is the total number of elements $n$, whatever the shape.