Denoising Diffusion Probabilistic Models, annotated

3.1 Forward process and L_T · original

Everyday picture

If the static dial is bolted in place, how much static is on screen at the end is not something training can change. The paper fixes every β_t to a constant, so the forward process has no learnable parameters.

Tiny example

The toy chain's L_T = 1.7165 depends only on x₀ and the fixed betas. No weight of the network appears in it, so its gradient is zero and training can drop it. (The paper notes the betas could be learned by reparameterization, as in the VAE paper, but chooses not to.)

Why it matters today

A fixed, parameter-free forward process is what separates diffusion models from VAEs: the “encoder” never has to be trained, and its end point is always the same known noise to sample from.

3.2 Reverse process and L_1:T−1: guess the noise · original

“The complete sampling procedure, Algorithm 2, resembles Langevin dynamics with ε_θ as a learned gradient of the data density.”Ho, Jain and Abbeel (2020), §3.2

Everyday picture

First choice: do not learn the width of each reverse step. The paper fixes Σ_θ = σ_t²I and tries σ_t² = β_t and σ_t² = β̃_t, with similar results (the first is optimal if the data were pure noise, the second if the data were one single point). That leaves the centre. The obvious design asks the network for the centre directly. The paper's design asks it for something simpler: the noise that was added. A photo restorer who practises by guessing where known dust went is learning exactly this.

Tiny example

In the toy chain, x₂ = 1.3913 is a one-jump mix of x₀ = 2.0 and some total noise ε: 1.3913 = √0.72 × 2.0 + √0.28 × ε, so ε = −0.5779. Plug that ε into the paper's formula for the centre and you get 1.7997: exactly the true backward centre μ̃₂ from §2. A network that guesses the noise perfectly gets the reverse step exactly right.

The middle terms are squared distances between centres

In words: “when both bell curves have fixed widths, the only part of each middle term training can change is the squared distance between the true backward centre and the model's centre, divided by twice the model's variance.”

With the numbers: toy chain, t = 2, σ₂² = β₂ = 0.2 and an illustrative model centre of 1.90: (1.7997 − 1.90)² / 0.4 = 0.0252. The full KL is 0.2185, and the gap, C = 0.1934, depends only on β̃₂ and σ₂², never on the network.

In Python:

import math
x0 = 2.0
x2 = math.sqrt(0.8) * (math.sqrt(0.9) * x0 + math.sqrt(0.1) * 0.5) + math.sqrt(0.2) * -1.0
# the true backward step from the previous block
mu_tilde = math.sqrt(0.9) * 0.2 / 0.28 * x0 + math.sqrt(0.8) * 0.1 / 0.28 * x2
beta_tilde = 0.1 / 0.28 * 0.2
mu_theta, sigma2 = 1.90, 0.2
# the part that depends on θ: ‖μ̃ − μ_θ‖² / (2σ²)
theta_part = (mu_tilde - mu_theta) ** 2 / (2 * sigma2)
round(theta_part, 4)  # → 0.0252
# the full KL between N(μ̃, β̃) and N(μ_θ, σ²)
kl = 0.5 * (beta_tilde / sigma2 + (mu_tilde - mu_theta) ** 2 / sigma2 - 1 - math.log(beta_tilde / sigma2))
round(kl, 4)  # → 0.2185
# C: what is left, the same whatever μ_θ is
round(kl - theta_part, 4)  # → 0.1934

Predict the noise instead of the centre

Write the noisy picture with the shortcut, x_t = √ᾱ_t x₀ + √(1 − ᾱ_t) ε, and substitute into μ̃_t: the target centre becomes (x_t − β_t/√(1 − ᾱ_t) ε) / √α_t. The network already sees x_t, so the only unknown is ε. The paper's parameterization lets the network supply it:

In words: “to step back, subtract this step's share of the guessed noise, undo this step's shrink, then add a small fresh wobble.”

With the numbers: toy chain with the perfect guess ε = −0.5779: (1.3913 − 0.2/√0.28 × (−0.5779)) / √0.8 = (1.3913 + 0.2184) / 0.8944 = 1.7997 = μ̃₂. The diffusion lesson's example step, x_t = 1.0, β_t = 0.19, ᾱ_t = 0.75 and a guess of 0.5, gives (1.0 − 0.38 × 0.5)/0.9 = 0.9, and with σ_t = √0.19 and z = 0.5 the next point is 1.1179.

In Python:

import math
x0 = 2.0
xt = math.sqrt(0.8) * (math.sqrt(0.9) * x0 + math.sqrt(0.1) * 0.5) + math.sqrt(0.2) * -1.0
beta_t, alpha_bar_t = 0.2, 0.72
alpha_t = 1 - beta_t
# the total noise that turned x_0 into x_t in one jump
eps = (xt - math.sqrt(alpha_bar_t) * x0) / math.sqrt(1 - alpha_bar_t)
round(eps, 4)  # → -0.5779
# μ_θ = (x_t − β_t / √(1 − ᾱ_t) · ε_θ) / √α_t, with a perfect guess ε_θ = ε
mu = (xt - beta_t / math.sqrt(1 - alpha_bar_t) * eps) / math.sqrt(alpha_t)
round(mu, 4)  # → 1.7997
# one sampling step from the diffusion lesson's example, σ_t = √β_t
x_t, b, ab, eps_hat, z = 1.0, 0.19, 0.75, 0.5, 0.5
round((x_t - b / math.sqrt(1 - ab) * eps_hat) / math.sqrt(1 - b) + math.sqrt(b) * z, 4)  # → 1.1179

Put this centre back into equation 8 and the middle terms become a weighted squared error on the noise:

In words: “noise a real picture to step t in one jump, ask the network what noise was added, and score the squared miss, weighted by a number that depends only on the step.”

With the numbers: with σ_t² = β_t the weight simplifies to β_t / (2α_t(1 − ᾱ_t)). Toy chain, t = 2: 0.2 / (2 × 0.8 × 0.28) = 0.4464. Paper's schedule: 0.5 at t = 1, falling to 0.0102 at t = 1,000, a ratio of 49.

In Python:

beta_t, alpha_bar_t = 0.2, 0.72
sigma2 = beta_t
# β² / (2σ² α (1 − ᾱ)), with σ² = β
round(beta_t ** 2 / (2 * sigma2 * (1 - beta_t) * (1 - alpha_bar_t)), 4)  # → 0.4464
# the paper's schedule, first and last step
T = 1000
betas = [1e-4 + (0.02 - 1e-4) * i / (T - 1) for i in range(T)]
alpha_bars, ab = [], 1.0
for b in betas:
    ab *= 1 - b
    alpha_bars.append(ab)
w = [b / (2 * (1 - b) * (1 - a)) for b, a in zip(betas, alpha_bars)]
round(w[0], 3), round(w[-1], 4), round(w[0] / w[-1])  # → (0.5, 0.0102, 49)

This is the paper's central observation. The loss “resembles denoising score matching over multiple noise scales”: guessing the noise is guessing the score, the direction towards more data, flipped and rescaled. And the sampling step, “move a little along a learned direction, then add a little noise”, is the shape of Langevin dynamics. Training this bound trains that sampler.

Reading it: on the left, training takes three random choices at the top (a real picture, a step, a noise draw), mixes them with the shortcut into one noisy picture, and asks the network for the noise. Only the noise draw also takes the long wire down the right-hand side, straight to the loss, where the guess is graded against it. The whole column repeats until the loss stops improving; there is no chain to simulate. On the right, sampling starts from pure noise and loops T times: guess the noise, compute the centre μ_θ, add a fresh wobble σ_tz (except on the last step), step t down by one. Each lap is one full run of the network, so a sample costs T = 1,000 network calls.

Why it matters today

“Predict the noise” is still the most common parameterization, and the two algorithms are the ones a first diffusion implementation writes. Later samplers such as DDIM reuse the same trained ε_θ with far fewer steps; see the diffusion lesson.

In code: train_noise_predictor is Algorithm 1 on a small DenoiserMLP; ddpm_step is one step of Algorithm 2 with σ_t = √β_t, and ddpm_sample runs the loop; noise_to_score turns the noise guess into the score.

3.3 Data scaling, reverse process decoder, and L₀ · original

Everyday picture

Real pixels are whole numbers from 0 to 255, not smooth values. So the very last step does not ask “how likely is exactly this value?” (a question with no good answer for a continuous bell curve) but “how much of the bell curve falls in this pixel's bin?”, the way a histogram counts everything inside a bar.

Tiny example

Pixel values are scaled linearly from {0, …, 255} to [−1, 1]. A pixel of 128 becomes 128/127.5 − 1 = 0.0039, and its bin runs half a step either side, from 0.0000 to 0.0078. If the model's last step is a bell curve centred at 0.01 (illustrative) with spread σ₁ = √β₁ = 0.01 (the paper's schedule), the chance of landing in that bin is 0.256.

In words: “the probability of the final picture is, for every pixel independently, the area under that pixel's bell curve over the pixel's bin; the outermost bins stretch to infinity so no probability is lost off the ends.”

With the numbers: Φ((0.0078 − 0.01)/0.01) − Φ((0.0000 − 0.01)/0.01) = Φ(−0.22) − Φ(−1.0) = 0.256 for the pixel of 128. Multiply such areas over all D = 32 × 32 × 3 = 3,072 numbers of a CIFAR10 image to get p_θ(x₀ | x₁).

In Python:

from statistics import NormalDist
# scale a pixel from {0, ..., 255} to [−1, 1]
x = 128 / 127.5 - 1
round(x, 4)  # → 0.0039
# the bin: δ−(x) = x − 1/255, δ+(x) = x + 1/255
lo, hi = x - 1 / 255, x + 1 / 255
round(lo, 4), round(hi, 4)  # → (-0.0, 0.0078)
# area of N(μ, σ_1²) over the bin, with an illustrative μ = 0.01 and σ_1 = √0.0001 = 0.01
step = NormalDist(0.01, 0.01)
round(step.cdf(hi) - step.cdf(lo), 3)  # → 0.256
# D for a CIFAR10 image
32 * 32 * 3  # → 3072

Why it matters today

This makes the bound an honest codelength in bits per dimension, comparable with other likelihood models. At the end of sampling, the paper shows μ_θ(x₁, 1) without adding the last bit of noise.

3.4 The simplified training objective · original

Everyday picture

The exact bound grades every step's homework with a different weight, and it weights the easiest homework (removing a whisper of noise from a nearly clean picture) the most. The paper's simplification: grade every step equally. Students then spend their effort on the hard exercises, the heavily noised pictures, where the big decisions about the image are made.

Tiny example

True noise (0.5, −1.0), network guess (0.4, −0.8): the miss is (0.1, −0.2) and the squared miss 0.01 + 0.04 = 0.05. Under the exact bound this miss would be multiplied by the step's weight (0.5 at step 1, 0.0102 at step 1,000). Under L_simple it counts as 0.05 at every step.

In words: “pick a random step, a random real picture and random noise; noise the picture to that step in one jump; score the squared distance between the real noise and the network's guess; average over everything.”

With the numbers: ‖(0.5, −1.0) − (0.4, −0.8)‖² = 0.05. A network that always guesses zero scores the average of ε₁² + ε₂², which is 2 for 2-number noise: the score to beat.

In Python:

import random
eps, guess = [0.5, -1.0], [0.4, -0.8]
# ‖ε − ε_θ‖²
round(sum((e - g) ** 2 for e, g in zip(eps, guess)), 2)  # → 0.05
# always guessing 0: the average of ‖ε‖² over many draws of 2-number noise
random.seed(0)
draws = [random.gauss(0, 1) ** 2 + random.gauss(0, 1) ** 2 for _ in range(100_000)]
round(sum(draws) / len(draws), 1)  # → 2.0

Reading it: the x-axis is the step t on a logarithmic scale, so the first few steps get room; the y-axis is the weight the exact bound puts on the noise-guessing error at that step (with σ_t² = β_t, the paper's schedule). It starts at 0.5 at t = 1, halves almost at once, and settles near 0.01 for most of the chain. L_simple replaces this whole curve with a flat line: every step counts the same. Relative to the exact bound, that shifts attention from the first handful of steps (almost clean pictures) to the hundreds of noisy ones, which is the down-weighting of small t the paper describes. Computed live from the schedule.

Why it matters today

L_simple gave the paper's best samples (FID 3.17 against 13.51 for the exact bound; §4.2), and plain, unweighted noise regression became the default. Later work returned to the question of how to weight the steps, but the unweighted loss is still the baseline every paper compares against.

In code: train_noise_predictor trains on exactly this loss, averaged over a batch of random steps, pictures and noises.

4.1 Sample quality · original

Everyday picture

How do you grade an artist who paints pictures nobody asked for? FID compares the whole gallery of generated pictures with a gallery of real ones, as clouds of features from a pretrained image network: the closer the clouds, the lower the score. Lower is better; 0 would be indistinguishable.

Tiny example

Two galleries whose feature clouds have the same shape but whose centres sit 3 units apart score an FID of 3² = 9; move the centres to 1 unit apart and it drops to 1. (FID also compares the clouds' spreads; with equal spreads that part is zero.)

Reading it: each bar is one model's FID on unconditional CIFAR10 (BigGAN is class-conditional, so it had extra help), drawn to one scale from 0 to 70; shorter is better. The striped bars are the paper's two diffusion models. Trained on the exact bound, the model sits among earlier methods at 13.51. Trained on L_simple, it reaches 3.17, just ahead of StyleGAN2 + ADA's 3.26. Its FID against the test set rather than the training set is 5.24. The NLL column shows the other side: the Sparse Transformer's 2.80 bits per dimension beats both diffusion models' 3.70 and 3.75.

Why it matters today

This table is why diffusion models took off: a likelihood-based model, trained by regression, matching the best GANs on the standard image benchmark. On 256 × 256 LSUN, the paper reports FID 7.89 on churches and 4.90 on bedrooms (larger model).

4.2 Reverse process parameterization and training objective ablation · original

Everyday picture

Two knobs, four settings, one winner. Knob one: does the network predict the centre μ̃ or the noise ε? Knob two: train on the exact bound, or on unweighted squared error? And a side question: should the network also learn the width of each step?

What the rows say

• Predicting the centre works only on the exact bound; trained with unweighted squared error on the centre it was unstable.
• Learning the widths made training unstable and samples worse, for both parameterizations.
• On the exact bound, predicting the noise is about as good as predicting the centre (13.51 against 13.22). With L_simple, it is far better (3.17).

Why it matters today

The winning combination is a pair: ε-prediction and the unweighted loss. The paper also mentions trying to predict x₀ directly, which gave worse samples early on. Later work revisited learned widths with a different parameterization, but the ε-prediction, fixed-width recipe of this table is where most implementations start.

4.3 Progressive coding · original

“More than half of the lossless codelength describes imperceptible distortions.”Ho, Jain and Abbeel (2020), §4.3

Everyday picture

Picture a slow internet connection loading a photo: first a blurry thumbnail, then sharper and sharper. The bound can be read as exactly such a transmission: send x_T, then x_T−1, and so on, each costing the bits of one KL term. At any moment the receiver can guess the final picture from what has arrived.

Tiny example

Midway, the receiver holds the toy's x_t = 1.9 at a level with ᾱ_t = 0.64, and the network guesses the noise in it as 0.5. Undoing the shortcut gives the receiver's best guess of the clean value: (1.9 − 0.6 × 0.5) / 0.8 = 2.0.

In words: “the best guess of the clean picture is the noisy one with the guessed noise taken out, scaled back up by the surviving signal share.”

With the numbers: (1.9 − √0.36 × 0.5) / √0.64 = (1.9 − 0.3) / 0.8 = 2.0.

In Python:

import math
x_t, alpha_bar_t, eps_hat = 1.9, 0.64, 0.5
# x̂_0 = (x_t − √(1 − ᾱ_t) ε_θ) / √ᾱ_t
round((x_t - math.sqrt(1 - alpha_bar_t) * eps_hat) / math.sqrt(alpha_bar_t), 4)  # → 2.0

Reading it: both charts share an x-axis, the reverse-process time T − t + 1 as the paper's Table 4 lists it, from step 100 to step 1,000 of generation. The top chart is distortion, the root-mean-squared error of the receiver's guess x̂₀ on a 0 to 255 pixel scale: it falls steadily from 67.6 to 12.0 by step 900 and to 0.95 at the end. The bottom chart is rate, the bits per dimension received so far: almost nothing until step 800 (0.054), 0.12 at step 900, and then a jump to 1.78 in the last hundred steps. So most of the bits buy the last drop in error, from 12 grey levels to under 1: details the eye barely sees. This is Figure 5 of the paper, redrawn from its Table 4.

The paper's best CIFAR10 model has a rate of 1.78 bits per dimension and a distortion of 1.97 bits per dimension, a root-mean-squared error of 0.95 on the 0 to 255 scale. Its train and test codelengths differ by at most 0.03 bits per dimension, so it is not overfitting.

Progressive generation

Running the sampler and showing x̂₀ along the way, large features (layout, colour) appear first and fine details last. Freezing x_t and sampling several endings shows the same thing from the other side: when t is large only the coarse features are shared; when t is small, all but the finest details are.

A connection to autoregressive models

Tiny example: take a 3-pixel image and a strange “diffusion” that blanks one pixel per step: step 1 blanks pixel 1, step 2 pixel 2, step 3 pixel 3, ending at a blank image. Running it backwards means predicting pixel 3, then pixel 2 given pixel 3, then pixel 1 given the other two: that is an autoregressive model. The paper shows the bound becomes exactly an autoregressive loss under that choice, and so reads Gaussian diffusion as autoregression along a generalised “bit ordering” that no reordering of pixels could express. T also need not equal the number of pixels: T = 1,000 is less than the 3,072 numbers of a CIFAR10 image.

Why it matters today

The coarse-to-fine behaviour is why step budgets matter: cut the early steps and the layout suffers, cut the late ones and only fine texture does. It also explains why diffusion models make better pictures than their likelihoods suggest: the likelihood pays for invisible details. The paper notes Algorithms 3 and 4 are a proof of concept, since the coding procedure they assume is not tractable for high-dimensional data.

4.4 Interpolation · original

Everyday picture

Blend two faces by mixing their pixels and you get a ghostly double exposure. Instead, noise both faces part of the way, blend the noisy versions, and let the reverse process clean up the blend: it removes the ghosting and paints a single, plausible face between the two.

Tiny example

Two pictures noised to the same step (with the same noise, so only the pictures differ) have values 1.9 and −0.5 at one pixel. A quarter of the way from the first to the second: 0.75 × 1.9 + 0.25 × (−0.5) = 1.3. The reverse process then denoises that blended value.

In words: “take a share 1 − λ of the first noised picture and a share λ of the second, then run the reverse process from the blend.”

With the numbers: λ = 0.25: 0.75 × 1.9 + 0.25 × (−0.5) = 1.3. The printed formula in the paper writes the clean pictures x₀ and x′₀ on the right; the surrounding text (“the linearly interpolated latent”, with the noise fixed so that x_t and x′_t stay the same) makes clear the noised versions are meant, as written here.

In Python:

x_t, x_t_prime, lam = 1.9, -0.5, 0.25
# x̄_t = (1 − λ) x_t + λ x′_t
round((1 - lam) * x_t + lam * x_t_prime, 4)  # → 1.3

What the paper found

On CelebA-HQ faces at t = 500, interpolations vary pose, skin tone, hairstyle, expression and background smoothly, though not eyewear. More noising steps before blending give coarser, more varied interpolations; after 1,000 steps the source pictures are forgotten and the results are new samples.

Why it matters today

“Noise part of the way, then denoise” became a standard editing tool: image-to-image translation and sketch-to-picture tools use exactly this move, choosing how far to noise as a strength knob.