The Diffusion Forward Process

The problem

A diffusion model learns to make images by learning to undo noise. First we need the forward process: adding noise to an image, a little at a time, in $T$ steps. You will write the schedule of how much noise is added at each step, and the function that jumps straight to any step. This is the DDPM (Denoising Diffusion Probabilistic Models) schedule.

The schedule

At step $t$ a small amount of noise of variance $\beta_t$ is added. Two derived quantities:

\alpha_t = 1 - \beta_t, \qquad \bar\alpha_t = \prod_{s=1}^{t} \alpha_s

where $\alpha_t$ is the share of the signal that survives step $t$ , and $\bar\alpha_t$ ("alpha bar") is the product of the survivals up to step $t$ : the share of the original image still in the sample. DDPM's $\beta$ grows linearly from $10^{-4}$ to $0.02$ .

Example. $T = 4$ , $\beta = (0.1, 0.2, 0.3, 0.4)$ .

$\alpha = 1 - \beta = (0.9, 0.8, 0.7, 0.6)$ .

$\bar\alpha$ : running product: $0.9$ , then $0.9 \cdot 0.8 = 0.72$ , then $0.72 \cdot 0.7 = 0.504$ , then $0.504 \cdot 0.6 = 0.3024$ .

Jumping to step $t$

You do not need to add noise $t$ times: the sum of many small Gaussian noises is one Gaussian noise, so

x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon

where $x_0$ is the clean data, $\epsilon$ is standard Gaussian noise (given in the tests, shaped like $x_0$ ), and the two coefficients say how much of the signal and how much of the noise are in $x_t$ .

Example. Same schedule, $x_0 = 2$ , noise $\epsilon = 1$ , step $t = 2$ ( $\bar\alpha_2 = 0.72$ ): $x_2 = \sqrt{0.72} \cdot 2 + \sqrt{0.28} \cdot 1 = 1.697 + 0.529 = 2.226$ . In the code, $\bar\alpha_2$ is abar[1]: the array is indexed from 0.

The training loss

The network predicts the noise that was added; the loss is the mean squared error between its prediction and the true noise.

Example. The true noise is $(1, 0, -1)$ and the prediction $(0.5, 0, -1)$ : squared errors $(0.25, 0, 0)$ , mean $0.083$ .

What to implement

linear_betas(T, beta_1=1e-4, beta_T=0.02): $T$ values increasing linearly from beta_1 to beta_T. Element t - 1 is $\beta_t$ .
alpha_bars(betas): $\bar\alpha_t$ for $t = 1, \dots, T$ , same indexing.
q_sample(x0, t, noise, abar): $x_t$ for an integer step $t \in [1, T]$ , using abar[t - 1].
simple_loss(eps_pred, noise): the mean squared error between the predicted and the true noise, over all elements.

Requirements

Steps are numbered from 1 as in the paper; arrays are indexed from 0. Mind the shift.
No loops: np.cumprod computes the running product.