ReLU & Tanh Backward

The problem

A network alternates two kinds of layers: matrix products, and activation functions applied to every number separately. The activations are what make the network more than one big matrix product: two matrix products in a row are just another matrix product (multiplying by 2 and then by 3 is the same as multiplying by 6), so without something in between, a deep network could learn nothing a single layer could not.

You will write the forward pass (compute the output) and the backward pass (compute the gradient) of the two classic activations, ReLU (rectified linear unit) and tanh (hyperbolic tangent).

Both work number by number, so their backward pass is number by number too. By the chain rule, the gradient with respect to the input is:

where:

• $x$ is the input and $y$ the output of the activation;
• $\partial L / \partial y$ is grad_out, the upstream gradient: how the loss reacts to the output. It is given to you;
• $\partial y / \partial x$ is the local derivative of the activation at that point.

So each backward pass is "grad_out times the local derivative", entry by entry.

ReLU

ReLU keeps positive numbers and replaces negative ones with 0:

Its local derivative is simple. Where $x > 0$ the output is $x$ itself, so the slope is 1. Where $x < 0$ the output is always 0, so the slope is 0. At exactly $x = 0$ there is no single slope; by convention, as in PyTorch, use 0.

Example. x = [-2.0, 0.0, 3.0] and grad_out = [5.0, 5.0, 5.0].

• Forward: [0.0, 0.0, 3.0].
• Local derivative: [0, 0, 1] (only the last input is positive).
• Backward: grad_out times it, [0.0, 0.0, 5.0]. The gradient passes only where the input was positive.

Tanh

Tanh squashes any number into the range between $-1$ and $1$: large positive inputs give almost 1, large negative ones almost $-1$, and 0 gives 0.

Its local derivative has a convenient form, written with the output $y = \tanh(x)$:

So the backward pass does not need $x$ at all, only the output $y$ that the forward pass already computed. Saving what the backward pass will need is what frameworks call the cache.

Example. x = [-1.0, 0.0, 0.5, 3.0] and grad_out = [2.0, 2.0, 2.0, 2.0].

• Forward: $y \approx$ [-0.762, 0.0, 0.462, 0.995].
• Local derivative $1 - y^2 \approx$ [0.420, 1.0, 0.786, 0.010].
• Backward: grad_out times it, about [0.840, 2.0, 1.573, 0.020].

At $x = 3$ the output is almost flat at 1, so almost no gradient gets through: tanh "saturates".

What to implement

• relu_forward(x): ReLU of every entry.
• relu_backward(x, grad_out): the gradient with respect to x, given the input x and the upstream gradient grad_out.
• tanh_forward(x): tanh of every entry (np.tanh).
• tanh_backward(y, grad_out): the gradient with respect to the input. Note that this one receives the output y = tanh(x), not the input.

Requirements

• Every function works on arrays of any shape and returns the shape of its input.
• ReLU's local derivative at exactly 0 is 0.
• No Python loops.

Hint

The local derivative of ReLU is a boolean mask: x > 0 is an array of True and False, and multiplying it by numbers treats them as 1 and 0.