Classifier-Free Diffusion Guidance, annotated

3.1 Classifier guidance · original

Everyday picture

At every step, show the half-finished sample to a classifier trained on noisy images, ask how to change it to look more like the label, and nudge it that way as well as along the diffusion model's own direction. The strength of the nudge is w.

Tiny example

In the two-class toy at z = 0.1, a perfect noisy-data classifier says “+” with probability 0.7087, and the slope of its log-probability is +2.59: move right to become more “+”. The class-conditional noise guess for “+” is (0.1 − 1.6)/0.6 = −2.5. With w = 1, the guided guess is −2.5 − 1 × 0.6 × 2.59 = −4.05: a guess of more negative noise, so a step that moves z further right, toward “+”.

In words: “the guided noise guess is the labelled guess minus w times the classifier's push toward the label, rescaled; it is the noise guess you would get for a distribution whose log-probability adds w times the classifier's log-probability.”

With the numbers: −2.5 − 1 × 0.6 × 2.5897 = −4.0538. The classifier's slope comes from Bayes' rule on the two bell curves at centres ±1.6 with variance 0.36: log-odds 8.889 × z, so a slope of 8.889 × (1 − 0.7087) = 2.5897.

In Python:

import math
z, sigma, var = 0.1, 0.6, 0.36
# p(+ | z) by Bayes' rule: a logistic curve in z, with slope k = 2 × 1.6 × 2 / (2 × 0.36)
k = 4 * 1.6 / (2 * var)
p_plus = 1 / (1 + math.exp(-k * z))
round(p_plus, 4), round(k, 3)  # → (0.7087, 8.889)
# ∇ log p(+ | z) = k (1 − p)
grad = k * (1 - p_plus)
eps_c = (z - 1.6) / sigma
round(grad, 4), round(eps_c, 4)  # → (2.5897, -2.5)
# ε̃ = ε_c − w σ ∇ log p(c | z), with w = 1
round(eps_c - 1 * sigma * grad, 4)  # → -4.0538

Using ε̃ in place of the labelled guess samples, approximately, from a sharpened distribution:

In words: “take the class's own distribution and multiply in the classifier's confidence w times: places where the classifier is unsure lose weight, places where it is sure keep theirs.”

With the numbers: compare the ambiguous z = 0.1 with the class centre 1.6, where the classifier is certain. Unguided, z = 0.1 is 0.0439 times as likely as 1.6. With w = 1 that falls to 0.0311 (× 0.7087), and with w = 3 to 0.0156 (× 0.7087³). Ambiguous samples are squeezed out.

In Python:

import math
def density(z, centre, var=0.36):
    return math.exp(-(z - centre) ** 2 / (2 * var)) / math.sqrt(2 * math.pi * var)
def p_plus(z):
    return density(z, 1.6) / (density(z, 1.6) + density(z, -1.6))
# p(z | c) p(c | z)^w at z = 0.1, relative to the class centre z = 1.6
ratio = density(0.1, 1.6) / density(1.6, 1.6)
[round(ratio * (p_plus(0.1) / p_plus(1.6)) ** w, 4) for w in (0, 1, 3)]  # → [0.0439, 0.0311, 0.0156]

Try it: the paper's Figure 2 shows three classes, each a round bell curve in the plane. Drag w and watch each class's guided distribution.

Reading it: each colour is one class (its data a round bell curve of spread 0.7 around a centre 1.0 from the middle, illustrative numbers), and the brightness is the density of the equal mixture of the three guided class distributions p(x | c) p(c | x)^w, each renormalised, computed exactly on a grid. At w = 0 it is the plain data distribution: three overlapping blobs that blur into one another in the middle. Drag w up and each class retreats from the others: its mass moves outward, away from where the classifier could confuse it, and bunches into a smaller, lopsided region, no longer a round bell curve. The readout tracks one class. That is the paper's Figure 2, redrawn: higher confidence and lower variety, the Inception-score-up, diversity-down trade in miniature.

Guiding an unconditional model

The paper also notes that guiding an unconditional model with weight w + 1 gives the same thing as guiding a conditional model with weight w, since, by Bayes' rule, p(z | c) is proportional to p(z) p(c | z):

In words: “start from the unlabelled noise guess and push toward the label once more than before: the first push turns the unlabelled guess into the labelled one, the rest is guidance.”

With the numbers: the toy's unlabelled guess at z = 0.1 is −0.9462 (computed in §3.2 below). −0.9462 − 2 × 0.6 × 2.5897 = −4.0538: the same guided guess as before.

In Python:

eps_u, sigma, grad, w = -0.946191, 0.6, 2.589682, 1
# ε_u − (w + 1) σ ∇ log p(c | z)
round(eps_u - (w + 1) * sigma * grad, 4)  # → -4.0538

In theory the two are the same; in practice the earlier work got its best results guiding an already conditional model, so the paper stays with that.

Why it matters today

Classifier guidance showed that a diffusion model can be sharpened after training, with one knob. Its costs (a second, noise-trained classifier, and steps along that classifier's gradient) are what the next section removes.

3.2 Classifier-free guidance · original

“Eq. 6 has no classifier gradient present, so taking a step in the ε̃_θ direction cannot be interpreted as a gradient-based adversarial attack on an image classifier.”Ho and Salimans (2022), §3.2

Everyday picture

Instead of hiring a critic, teach the artist to draw both ways: sometimes you tell them the subject, sometimes you don't. Then, while drawing, the artist compares “what I'd draw for a dog” with “what I'd draw for anything” and exaggerates the difference. The difference between those two is itself a critic's opinion, recovered without a critic.

Tiny example

At some step the unlabelled guess is 0.2 and the labelled guess is 0.5 (the diffusion lesson's numbers). The label moves the guess by 0.3. With w = 2, go 2 × 0.3 = 0.6 past the labelled guess: 1.1.

Reading it: on the left, training is DDPM's loop with one extra box, the second from the top: with probability p_uncond the label is replaced by the null label ∅, so the same network learns both the labelled and the unlabelled guess. Everything else (pick a level, noise the picture, score the noise guess, take a gradient step) is unchanged. On the right, sampling starts from pure noise, and at every level calls the network twice, once with the label and once with ∅. The two guesses meet in the mixing box, (1 + w)ε_c − w ε_u, which is the whole of classifier-free guidance. The mixed guess predicts a clean picture x̃, and an ordinary sampler step (the paper's own, or DDIM, as its Algorithm 2 notes) moves to the next level. After the last level, x̃ is the output.

The formula

In words: “take the labelled guess, and move w times further in the direction from the unlabelled guess to the labelled one.” Rearranged, it is ε_u + (1 + w)(ε_c − ε_u), which is the diffusion lesson's formula with its weight equal to 1 + w.

With the numbers: (1 + 2) × 0.5 − 2 × 0.2 = 1.5 − 0.4 = 1.1. At w = 0 it is 0.5, the plain labelled guess. The lesson writes the same mix as 0.2 + 3 × (0.5 − 0.2) = 1.1, with its weight 3.

In Python:

eps_c, eps_u = 0.5, 0.2
# the paper: ε̃ = (1 + w) ε_c − w ε_u
[round((1 + w) * eps_c - w * eps_u, 2) for w in (0, 1, 2)]  # → [0.5, 0.8, 1.1]
# the lesson: ε_u + w_lesson (ε_c − ε_u), with w_lesson = 1 + w
[round(eps_u + (1 + w) * (eps_c - eps_u), 2) for w in (0, 1, 2)]  # → [0.5, 0.8, 1.1]

The implicit classifier

Where does the formula come from? Bayes' rule gives a classifier hidden inside any pair of conditional and unconditional models, pⁱ(c | z) ∝ p(z | c)/p(z). Its slope can be read off the two exact noise guesses:

In words: “the difference between the labelled and the unlabelled noise guess, flipped and rescaled, is exactly the slope of the classifier that Bayes' rule hides in the two models.”

With the numbers: in the two-class toy at z = 0.1, the exact labelled guess is −2.5 and the exact unlabelled guess (averaging the two classes' guesses by their 71% / 29% odds) is −0.9462. Then −(−2.5 + 0.9462)/0.6 = 2.5897: the classifier slope of §3.1, found with no classifier. Plug it into classifier guidance with w = 1 and you get (1 + 1) × (−2.5) − 1 × (−0.9462) = −4.0538, equation 6 exactly.

In Python:

import math
z, sigma, var = 0.1, 0.6, 0.36
centres = [1.6, -1.6]
# the two classes' odds at z, by Bayes' rule
dens = [math.exp(-(z - m) ** 2 / (2 * var)) for m in centres]
odds = [d / sum(dens) for d in dens]
# exact noise guesses: labelled "+", and unlabelled (the odds-weighted average)
eps_c = (z - centres[0]) / sigma
eps_u = sum(p * (z - m) / sigma for p, m in zip(odds, centres))
round(eps_c, 4), round(eps_u, 4)  # → (-2.5, -0.9462)
# the implicit classifier's slope, and equation 6 with w = 1
round(-(eps_c - eps_u) / sigma, 4), round(2 * eps_c - eps_u, 4)  # → (2.5897, -4.0538)

The paper stresses a subtlety. With exact scores the two methods agree, as above. But a trained network's outputs are not guaranteed to be the slope of any function at all (they need not form a conservative vector field), so the learned difference ε_θ(z, c) − ε_θ(z) is in general not the slope of any classifier. Classifier-free guidance is inspired by an implicit classifier, not equal to one, and so it cannot be an adversarial attack on one.

It was also not obvious that it would work: a classifier built from a generative model by Bayes' rule is often worse than one trained directly, and can behave badly when the model is imperfect. The experiments are the evidence.

Try it: the same three classes as Figure 2 above, but now actually sampled: 300 samples of the top class, using the exact labelled and unlabelled noise guesses, equation 6, and 50 DDIM steps (see the DDIM companion). Drag w and compare the dots with the shaded target.

Reading it: the shading is the top class's guided target p(x | c) p(c | x)^w from Figure 2, and the dots are what guided sampling actually produces from the same 300 starting noises. At w = 0 the dots roughly match the class's plain bell curve (300 samples and 50 big steps leave small differences in the readout). As w grows the dots move up and away from the other two classes and tighten, just as the shading does, but much further: at w = 4 the samples' centre is about 2.5 from the middle, against 1.45 for the shaded target. The reason is that guidance is applied at every noise level, including the very noisy ones where the classes overlap heavily and the push is strong. So guided sampling only approximates the sharpened distribution of §3.1, and overshoots it toward exaggeration; the paper, for its part, notes that strongly guided ImageNet samples show saturated colours.

Why it matters today

This is the whole method, and it is a few lines of code: drop the label now and then while training; run the network twice and mix while sampling. It needs no classifier, so the “label” can be anything the network can read, such as a sentence.

In code: network_inputs adds the one-hot label with its extra “no label” slot, train_noise_predictor hides labels at random (Algorithm 1), guided_noise is equation 6 in the lesson's form, and ddim_sample applies it at every jump (Algorithm 2 with a DDIM step).

4.1 Varying the classifier-free guidance strength · original

“…here we clearly see that increasing classifier-free guidance strength has the expected effect of decreasing sample variety and increasing individual sample fidelity.”Ho and Salimans (2022), §4.1

Everyday picture

Turn the dial and the gallery changes character: at w = 0 a varied crowd of dogs, some odd; at w = 3 or 4, every dog is an unmistakable textbook dog, in saturated colours, and they start to look alike. The paper's image figures show exactly this for the malamute class.

Tiny example

A little guidance helps both scores (FID 1.8 → 1.55, the best in the table). More guidance keeps raising the Inception score, to 260, while FID climbs to 26: every sample more convincing, the collection less like the real, varied data.

Reading it: each line is one trained model, and each point on it is one guidance weight w: the x-axis is Inception score, the y-axis is FID. Guidance grows from left to right along every line, from w = 0 at the far left to w = 4 at the far right, because Inception score rises steadily with w. Look first at the bottom-left corner: the lines dip slightly (FID improves) for the first small step of guidance, then turn up. After that it is a straight trade: every gain in Inception score is paid for in FID. The three lines are three values of p_uncond (§4.2). This is the paper's Figure 4, redrawn from the numbers in its Table 1.

The paper's best 128 × 128 numbers beat the classifier-guided model it compares with at w = 0.3 (FID 2.43 against 2.97), and at w = 4 beat BigGAN-deep at its best-Inception-score setting on both scores (FID 21.53 against 25, Inception score 421.03 against 253, all from its Table 2).

One sentence in §4.1 reads the wrong way round: it describes “FID monotonically decreasing and IS monotonically increasing with w”, but the paper's own tables and figures show FID increasing with w beyond the first small step, as the chart above does. The trade-off the sentence goes on to describe is the one the numbers show.

Why it matters today

The shape of this curve is why products expose w as a knob and pick a default well above 0: people usually prefer each image to follow the prompt convincingly over the collection matching the data's variety.

In code: guidance_sweep measures the lesson's version of this trade: the share of samples on the requested blob and their spread, at several weights.

4.2 Varying the unconditional training probability · original

Everyday picture

How often should the artist practise without being told the subject? Enough to know what “anything” looks like, but not so often that the labelled drawing suffers.

Tiny example

Out of 400,000 training steps with p_uncond = 0.1, about 40,000 steps' worth of examples are unlabelled. At 0.5, half of all training goes to the unlabelled task.

In Python:

steps = 400_000
# how many steps' worth of examples train the unlabelled guess
[int(steps * p) for p in (0.1, 0.2, 0.5)]  # → [40000, 80000, 200000]

In the chart above, the p_uncond = 0.5 line sits above the other two all the way along (worse FID at the same Inception score), while 0.1 and 0.2 are almost on top of each other. So a small share of the network's effort, 10 to 20%, is enough for the unlabelled guess that guidance needs. The paper notes the parallel with classifier guidance, where small classifiers were found to suffice.

Why it matters today

Dropping the condition 10 to 20% of the time, replacing it with an empty or null prompt, became the default recipe for training text-conditioned diffusion models.

In code: train_noise_predictor takes p_uncond as an argument, 0.2 by default.

4.3 Varying the number of sampling steps · original

Everyday picture

More steps give a more careful drawing, but here every step costs two network calls instead of one. A fair race counts calls, not steps.

Tiny example

256 guided steps cost 2 × 256 = 512 network passes. The classifier-guided model the paper compares against used about 256 steps of the same architecture (plus its classifier). So, as the paper says, the fair comparison in network passes is its T = 128 setting, whose best FID (3.02, at w = 0.4) is worse than the classifier-guided 2.97.

In Python:

calls_per_step = 2
# network passes for the paper's three step counts
[T * calls_per_step for T in (128, 256, 1024)]  # → [256, 512, 2048]

Reading it: as in the chart of §4.1, each line traces w from 0 (left) to 4 (right), with Inception score across and FID up; here the lines are the 128 × 128 model sampled with T = 128, 256 and 1,024 steps. The T = 128 line sits a little above the other two for small w, where FID is lowest, while 256 and 1,024 steps almost coincide. So T = 256 is the good balance the paper names. This is the paper's Figure 5, redrawn from its Table 2. (The printed Figure 5 labels its third line T = 512, but its caption, the text and Table 2 all say 1,024; this redraw uses the table.)

Why it matters today

The doubled cost is real and still paid; a common implementation runs the conditional and unconditional passes together as one batch of two. Cutting the step count, with samplers like DDIM, matters twice as much when every step costs double.