Classifier-Free Guidance

The problem

To generate "a photo of a cat", the diffusion network gets a condition $c$ (here a class id, or a text embedding). A weak condition produces images that only loosely follow it. Classifier-free guidance (Ho and Salimans, 2022) strengthens it with a trick: train one network that works both with the condition and without it (a null condition $\varnothing$), then, when sampling, push the prediction away from the unconditional one and towards the conditional one.

Training: dropping the condition

During training, in a fraction p_uncond of the examples the true condition is replaced by a special "null" id, so the same network learns both jobs. Draw one random number per example, u = rng.random(conds.shape), and replace the id where u < p_uncond.

Example. conds = [3, 1, 4, 1, 5], p_uncond = 0.5, null_id = 0, and the random numbers come out [0.64, 0.27, 0.04, 0.02, 0.81]. The ones below 0.5 are at positions 1, 2 and 3, so the result is [3, 0, 0, 0, 5].

Sampling: guided prediction

At each step, ask the network twice, with the condition and with the null id, and extrapolate:

where $\tilde\epsilon$ is the guided noise prediction, $\epsilon(x_t, c)$ the conditional one, $\epsilon(x_t, \varnothing)$ the unconditional one, and $w$ the guidance weight. The same formula can be written $\epsilon_u + (1 + w)(\epsilon_c - \epsilon_u)$: start from the unconditional prediction and move along "what the condition adds", $\epsilon_c - \epsilon_u$, by a factor $1 + w$.

Example. For one number, $\epsilon_c = 0.8$ and $\epsilon_u = 0.5$.

$w$	guided prediction
0	$0.8$ (the conditional prediction, unchanged)
1	$1.1$
2	$1.4$
5	$2.3$

A larger $w$ follows the condition more closely and gives less varied images.

What to implement

• drop_conditions(conds, p_uncond, null_id, rng): training-time dropout. conds is an integer array of condition ids, one per example. Draw u = rng.random(conds.shape) once and replace the ids where u < p_uncond with null_id. Return a new array.
• guided_eps(eps_model, x_t, t, cond, null_id, w): call eps_model(x_t, t, cond) and eps_model(x_t, t, null_id) and combine them.

Requirements

• w = 0 must give the plain conditional prediction, and the unconditional model must be queried with null_id.
• drop_conditions must not modify its input.