Inverted Dropout

The problem

Dropout randomly switches off parts of a network during training, so that it cannot rely on any single number and has to spread what it learns. Each activation is set to 0 with probability $p$. The survivors are multiplied by $1/(1-p)$, so that the average value of every activation stays the same ("inverted" dropout). When the model is used for real (evaluation), the layer does nothing.

Forward

where:

• $p$ is the probability of dropping an entry;
• $m$ is the mask: for each entry, 1 (keep) with probability $1 - p$, and 0 (drop) otherwise;
• $\odot$ is elementwise multiplication, and dividing by $1 - p$ makes up for the dropped entries.

To build the mask, draw one random number between 0 and 1 for every entry, u = rng.random(x.shape), and keep the entries where u >= p: that happens with probability $1 - p$.

Example. $p = 0.5$, x = [1, 1, 1, 1], and the random numbers come out as u = [0.7, 0.2, 0.9, 0.6].

1. Mask: u >= 0.5 is [True, False, True, True].
2. Keep the entries: [1, 0, 1, 1].
3. Divide by $1 - 0.5 = 0.5$: y = [2, 0, 2, 2].

Each entry is kept half of the time with value 2, and dropped the other half: on average it is still 1.

Backward

An entry that was dropped did not influence the output, so its gradient is 0. An entry that was kept was multiplied by $1/(1-p)$, so its gradient is multiplied by the same factor. In short, the backward pass applies the same mask to the upstream gradient:

Example. Same mask, upstream gradient [3, 3, 3, 3]: the result is [6, 0, 6, 6].

What to implement

• dropout_forward(x, p, training, rng), returning (y, mask):
  • training (training=True): draw u = rng.random(x.shape) once, keep the entries where u >= p. mask is that boolean array.
  • evaluation (training=False): return x unchanged and mask = None. Do not draw any random numbers.
• dropout_backward(grad_y, mask, p): the gradient with respect to x. When mask is None (evaluation), the gradient passes through unchanged.

Requirements

• p = 0 keeps everything. p = 1 drops everything: return zeros, without dividing by zero.
• Use the rng you are given (a np.random.Generator), exactly as described, so the tests can reproduce your mask.