Backpropagation by Hand

A network learns by adjusting its parameters: the numbers inside it, such as the entries of its weight matrices. Training repeats one step millions of times: measure how wrong the network is (the loss $L$, a single number), then nudge every parameter a little in the direction that makes $L$ smaller. To know that direction, we need a derivative for every parameter. This page explains what those derivatives are and how backpropagation computes all of them at once.

Derivatives and gradients

The derivative of the loss with respect to one parameter $\theta$ (theta) answers one question: if I change $\theta$ by a tiny amount, how much does $L$ change, and in which direction? It is written $\partial L / \partial \theta$.

Example. Take $L = \theta^2$ at $\theta = 3$, so $L = 9$. Move $\theta$ to $3.001$: now $L = 9.006001$. The loss grew by about $0.006$ for a step of $0.001$, a ratio of about $6$. That ratio is the derivative: $\partial L / \partial \theta = 2\theta = 6$.

A positive derivative means "increasing $\theta$ increases the loss", so training decreases $\theta$. With a step size of $0.1$: $\theta \leftarrow 3 - 0.1 \cdot 6 = 2.4$, and the loss drops from $9$ to $5.76$.

The gradient is the list of these derivatives, one per parameter. A small LLM (large language model) has millions of parameters, so we need millions of derivatives, at every training step, cheaply. Frameworks such as PyTorch compute them automatically, and every one of them uses the same idea, which you will code in this section.

The chain rule

A network is a long chain of simple operations: add, multiply, matrix product, tanh, and so on. Each operation takes the result of earlier ones. Drawn as a diagram, with one box per operation and arrows from inputs to outputs, this is the computational graph. The loss $L$ is the last box.

Suppose $L$ depends on $x$ only through an intermediate value $y = f(x)$. The chain rule says:

where:

• $\partial L / \partial y$ is the upstream gradient: how the loss reacts to $y$. It is computed first, coming back from the end of the graph;
• $\partial y / \partial x$ is the local gradient: how $y$ reacts to $x$. It depends only on the operation $f$ itself.

In words: how $x$ affects the loss = how $x$ affects $y$, times how $y$ affects the loss.

Example. Take $y = 3x$ and $L = y^2$, at $x = 1$ (so $y = 3$ and $L = 9$).

1. Upstream: $L = y^2$, so $\partial L / \partial y = 2y = 6$.
2. Local: $y = 3x$, so $\partial y / \partial x = 3$.
3. Chain rule: $\partial L / \partial x = 6 \cdot 3 = 18$.

Check directly: $L = (3x)^2 = 9x^2$, whose derivative $18x$ is $18$ at $x = 1$.

So each operation needs just one rule: "given the upstream gradient, produce the gradient for my inputs". It never needs to know what the rest of the network looks like. A few rules cover most of a network. In the table, $g$ is the upstream gradient $\partial L / \partial y$:

How to read the rows:

• addition passes the gradient unchanged to both inputs;
• multiplication sends each input the gradient times the other input;
• tanh multiplies the gradient by $1 - y^2$, using its own output $y$;
• $\max(0, x)$, called ReLU (rectified linear unit), lets the gradient through only where the input was positive.

Backpropagation, step by step

Backpropagation applies these rules from the end of the graph to the start: first the forward pass computes every value, then the backward pass computes every gradient, in the opposite order.

Example. $c = a \cdot b + a$ with $a = 2$ and $b = -3$.

Forward. $d = a \cdot b = -6$, then $c = d + a = -4$.

Backward, starting from $\partial c / \partial c = 1$:

1. $c = d + a$ is an addition: it passes $1$ to $d$ and $1$ to $a$. So far $\partial c / \partial d = 1$ and $a$ has received $1$.
2. $d = a \cdot b$ is a multiplication with upstream gradient $1$: $a$ receives $1 \cdot b = -3$, and $b$ receives $1 \cdot a = 2$.
3. $a$ was used twice, so its two contributions add up: $1 + (-3) = -2$.

Result: $\partial c / \partial a = -2$ and $\partial c / \partial b = 2$. Check directly: $c = ab + a$, so $\partial c / \partial a = b + 1 = -2$ and $\partial c / \partial b = a = 2$.

The backward pass visits the operations in reverse topological order: every operation comes after all the operations that use its result. In the example, $c$ comes before $d$, because $c$ uses $d$. This guarantees that when the pass reaches a value, every path from that value to $L$ has already sent its contribution, so its gradient is complete.

The total cost is a small constant times the cost of the forward pass, whatever the number of parameters. This is reverse-mode automatic differentiation. Linnainmaa's 1970 master's thesis is usually cited as its first published description (the idea was found independently several times), and Rumelhart, Hinton and Williams popularized it for training neural networks in 1986.

Two rules that prevent most bugs

Gradients add up. When a value is used in more than one place, each use sends back its own contribution, and these must be summed: in code grad += ..., never grad = .... In the example above, $a$ received $1$ and $-3$. Another case: $y = a \cdot a$ sends back $g\,a$ twice, once per factor, for a total of $2ga$, the derivative of $a^2$.

Broadcasting is the same rule in disguise. A bias $b$ added to every row of a batch is used once per row, so its gradient is the sum of the gradients of all the rows.

Shapes match. The gradient with respect to an array always has the shape of that array. For a matrix product $Y = XW$, with upstream gradient $G = \partial L / \partial Y$, this alone almost forces the answer:

where $X^\top$ is $X$ transposed: rows become columns. These are the only two products whose shapes work.

Example. $X$ has shape $(2, 3)$ and $W$ has shape $(3, 4)$, so $Y$ and $G$ have shape $(2, 4)$.

• $X^\top G$ is $(3, 2) \times (2, 4) = (3, 4)$: the shape of $W$.
• $G W^\top$ is $(2, 4) \times (4, 3) = (2, 3)$: the shape of $X$.

Any other combination, such as $G X$, does not even have shapes that can be multiplied.

Gradient checking

A wrong backward pass can still make the loss go down a little, so you cannot trust it just because training "works". Test it against the definition of the derivative instead: nudge a parameter a tiny amount and measure how the loss moves. The central difference nudges it both ways:

where:

• $\varepsilon$ (epsilon) is the small nudge, such as $10^{-5}$;
• $L(\theta + \varepsilon)$ and $L(\theta - \varepsilon)$ are the loss with $\theta$ nudged up and down;
• dividing by $2\varepsilon$, the distance between the two points, gives the slope.

Its error shrinks like $\varepsilon^2$, while the one-sided version, $(L(\theta+\varepsilon) - L(\theta))/\varepsilon$, only shrinks like $\varepsilon$.

Example. $L(\theta) = \theta^3$ at $\theta = 1$, where the true derivative is $3$. With $\varepsilon = 0.01$:

• central: $(1.01^3 - 0.99^3) / 0.02 = 3.0001$;
• one-sided: $(1.01^3 - 1^3) / 0.01 = 3.0301$, a hundred times further off.

To compare your gradient $a$ with the numerical one $b$, use a relative error, because gradients can be tiny or huge, and a difference of $0.001$ means nothing until you know the size of the numbers:

where $a_i$ and $b_i$ are entry $i$ of the two gradients. In words: the largest difference, divided by the size of the numbers being compared.

How to read it: around $10^{-7}$ or below is a pass; $10^{-2}$ means a bug. Use float64 and $\varepsilon$ between about $10^{-6}$ and $10^{-4}$. A check costs two forward passes per parameter, so run it on small inputs only, never during real training.

References

• D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, Nature 323, 533–536, 1986.
• S. Linnainmaa, Taylor expansion of the accumulated rounding error, BIT 16(2), 146–160, 1976. The journal paper based on his 1970 thesis.
• A. Griewank, Who invented the reverse mode of differentiation?, Documenta Mathematica, Extra Volume: Optimization Stories, 389–400, 2012. A short history.
• A. G. Baydin, B. A. Pearlmutter, A. A. Radul, J. M. Siskind, Automatic differentiation in machine learning: a survey, JMLR 18(153), 1–43, 2018.
• A. Griewank, A. Walther, Evaluating Derivatives, 2nd ed., SIAM, 2008. The reference book on automatic differentiation.
• I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016. Chapter 6: Deep Feedforward Networks, section 6.5 on back-propagation.
• V. Nair, G. E. Hinton, Rectified linear units improve restricted Boltzmann machines, ICML, 2010.
• Stanford CS231n notes: Backpropagation, intuitions and gradient checks.
• A. Karpathy, micrograd: a scalar autograd engine in about 100 lines, the model for the last exercise of this section.