Wasserstein GAN, annotated

How to read this page

Nothing here assumes you already know the jargon. This paper is heavier on mathematics than most, so the page leans on one tiny example throughout: two piles of two grains. The real pile has half its sand at position 0 and half at 3. The fake pile has half at 1 and half at 4. Every distance in the paper is computed on those four numbers.

• Any dotted word explains itself when you hover it, tab to it, or tap it.
• Every symbol inside an equation does the same, and every equation is followed by a table of its symbols, a sentence reading it aloud, the worked numbers, and the same numbers in plain Python.
• The pictures are live. Example 1 in §2 is the heart of the paper: slide the fake line and watch one distance move while the others refuse to.

Each idea climbs the same ladder: an everyday picture, a tiny example, a diagram, the math, and why it matters today. It builds on the GAN companion, and the GAN lesson computes the Wasserstein and Jensen-Shannon distances from scratch.

Abstract · original

“In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches.”Arjovsky, Chintala and Bottou (2017), Abstract

Everyday picture

A GAN's detective answers one question: real or fake? Once real and fake notes look nothing alike, it answers “fake” with total confidence, and a confident “fake” says nothing about how far off the forger is. This paper swaps the detective for a critic that gives a score, and trains it so that the gap between its average scores on real and fake samples measures how much work it would take to turn one pile of samples into the other.

What the paper claims

• Of the usual ways to compare two distributions, only the earth mover's distance changes smoothly as the generator's weights change, even when real and generated data sit on thin, non-overlapping surfaces.
• A GAN can minimise it: train a critic whose slope is capped (a Lipschitz constraint), enforced crudely by clipping its weights.
• In practice the critic's loss falls as the samples improve, training is far less sensitive to the architecture, and the authors saw no mode collapse.

Why it matters today

The clipping did not last, but the idea did: a critic whose slope is kept under control, a loss that measures distance rather than distinguishability, and a whole line of work (gradient penalties, spectral normalization) on how to keep that slope in check.

1 Introduction · original

“In other words, the added noise term is clearly incorrect for the problem, but is needed to make the maximum likelihood approach work.”Arjovsky, Chintala and Bottou (2017), §1

Everyday picture

The classic way to fit a model is maximum likelihood: tune it until the real data looks as probable as possible. That only works if the model gives every real example some probability. Real images, though, are thought to lie on a thin sheet inside the enormous space of all pixel grids, a low-dimensional manifold. A model whose samples lie on its own thin sheet will almost never touch the data's sheet, so it gives the real images probability zero. The usual patch is to blur the model with noise until it covers everything, which makes samples blurry.

Tiny example

Two real points, 0.5 and 1.5. The model is a bell curve of width 1 centred at θ. Its average log-likelihood is −1.544 at θ = 0, −1.044 at θ = 1 and −1.544 at θ = 2, so maximum likelihood picks θ = 1, the middle. Now the blur: the paper cites a model whose best noise was a spread of about 0.1 per pixel on pixels scaled 0 to 1. Noise with that spread lands within ±0.196 of the true value 95% of the time (1.96 spreads), turning a mid-grey pixel of 0.5 into anything from about 0.3 to 0.7, which is why, the paper notes, authors showed their samples without it.

In words: “choose the model's parameters so that the average log-probability it gives the real examples is as large as possible.” The paper adds that, with enough data, this is the same as minimising the KL divergence from the real distribution to the model.

With the numbers: at θ = 1 each point is 0.5 away from the centre, where the bell's height is 0.352, and log 0.352 = −1.044. At θ = 0 or 2 one point is 0.5 away and the other 1.5 away, averaging −1.544.

In Python:

import math
def P(x, theta):
    # a bell curve of width 1 centred at theta
    return math.exp(-0.5 * (x - theta) ** 2) / math.sqrt(2 * math.pi)
x = [0.5, 1.5]
# (1/m) Σ log P_θ(x^(i)) for three choices of θ
[round(sum(math.log(P(x_i, theta)) for x_i in x) / len(x), 3) for theta in (0, 1, 2)]  # → [-1.544, -1.044, -1.544]
# the blur the paper criticises: 95% of noise with spread 0.1 lands within 1.96 spreads
round(1.96 * 0.1, 3)  # → 0.196

The alternative, which GANs and VAEs share, is to skip the density altogether: draw noise z and push it through a network g_θ, whose outputs follow some distribution ℙ_θ. That can live on a thin sheet, and sampling is just a forward pass. What remains is to choose how to measure the distance ρ(ℙ_θ, ℙ_r) between the model's samples and the real ones.

Reading it: follow the chain from the top. The weights θ define a generator; the generator's outputs pile up into a distribution ℙ_θ; a distance ρ compares that pile with the real pile ℙ_r (the box on the right); and that number is the loss. Gradient descent needs the loss to change smoothly when θ changes a little. The paper's point is that this depends entirely on the amber box: with some choices of ρ, a small change of θ can make the loss jump, and then there is no gradient to follow.

Why it matters today

This framing, “which distance between distributions am I really minimising, and does it give a gradient?”, is a question worth asking of any generative training objective. The paper names the property it wants a weaker distance: one under which more sequences of distributions count as converging, so that more generators give a continuous loss.

2 Different distances · original

Everyday picture

Two piles of sand, and four ways to say how different they are. Three of them only look at how much the piles overlap. The fourth asks how far you would have to carry the sand. The section shows that only the fourth notices when a pile moves closer.

Why it matters

A GAN is only as good as the distance its training secretly minimises. The GAN companion's Theorem 1 showed that the original GAN minimises the Jensen-Shannon divergence; this section shows why that is a poor choice when the piles do not overlap.

TV, KL and JS · original

Everyday picture

The total variation distance finds the region where the two piles disagree most and reports how much. The KL divergence measures the surprise of expecting one pile when the truth is the other. The Jensen-Shannon divergence compares each pile with their half-and-half mix.

Tiny example

On our two piles (real: ½ at 0 and ½ at 3; fake: ½ at 1 and ½ at 4) they share no position at all. Take A = {0, 3}: the real pile puts all its sand there and the fake pile none, so TV = 1, the most it can be. The real pile has sand at 0, where the fake pile has none, so KL is infinite. JS comes out as log 2 = 0.693, also its largest possible value. None of the three would change if the fake pile sat at 100 and 103 instead.

In words: “over every set of outcomes you could pick, find the biggest gap between the probability the real distribution gives that set and the probability the fake one gives it.”

With the numbers: the set {0, 3} gets 1 from the real pile and 0 from the fake one; no set does better, so δ = 1.

In Python:

from itertools import combinations
P_r = {0: 0.5, 3: 0.5}
P_g = {1: 0.5, 4: 0.5}
positions = sorted(set(P_r) | set(P_g))
# every set A of positions: Σ, here all 16 subsets of {0, 1, 3, 4}
sets = [A for k in range(len(positions) + 1) for A in combinations(positions, k)]
def gap(A):
    return abs(sum(P_r.get(a, 0) for a in A) - sum(P_g.get(a, 0) for a in A))
# δ = sup over A of |P_r(A) − P_g(A)|
max(gap(A) for A in sets)  # → 1.0
max(sets, key=gap)  # → (0, 3)

In words: “wherever the real distribution has mass, weigh the log of how many times more likely the real density makes that point than the fake density does, and add it all up.”

With the numbers: at x = 0 the real density is ½ and the fake density is 0, so the ratio ½ / 0 is infinite and so is KL. The paper notes KL is lopsided (swapping the two piles changes it) and infinite whenever the fake pile misses a spot the real one covers.

In Python:

import math
P_r = {0: 0.5, 3: 0.5}
P_g = {1: 0.5, 4: 0.5}
def KL(P, Q):
    # ∫ log(P(x) / Q(x)) P(x): infinite as soon as Q misses a spot P covers
    if any(Q.get(x, 0) == 0 for x in P if P[x] > 0):
        return math.inf
    return sum(P[x] * math.log(P[x] / Q[x]) for x in P if P[x] > 0)
KL(P_r, P_g)  # → inf
# the same piles compared with themselves
KL(P_r, P_r)  # → 0.0

In words: “pour the two piles together half and half, and average how far each pile is from that mix.” Because the mix covers every spot either pile covers, JS is never infinite. (The line in §2 of the paper prints this without the two halves; its Example 1 value, log 2, and its proofs in Appendix C use the halved form shown here.)

With the numbers: the mix has ¼ at each of 0, 1, 3 and 4. Each pile has ½ where the mix has ¼, so each KL is ½ log 2 + ½ log 2 = log 2, and JS = ½ log 2 + ½ log 2 = 0.693.

In Python:

import math
P_r = {0: 0.5, 3: 0.5}
P_g = {1: 0.5, 4: 0.5}
# P_m: the half-and-half mix
P_m = {x: (P_r.get(x, 0) + P_g.get(x, 0)) / 2 for x in sorted(set(P_r) | set(P_g))}
P_m  # → {0: 0.25, 1: 0.25, 3: 0.25, 4: 0.25}
def KL(P, Q):
    return sum(P[x] * math.log(P[x] / Q[x]) for x in P if P[x] > 0)
JS = KL(P_r, P_m) / 2 + KL(P_g, P_m) / 2
round(JS, 3)  # → 0.693
round(math.log(2), 3)  # → 0.693

Why it matters today

All three are blind to distance: they only see overlap. The js_divergence function in the lesson shows the same flatness on histograms, stuck at log 2 whenever the two share no bin.

The earth mover distance (equation 1) · original

Everyday picture

You have to reshape one pile of sand into the other with a wheelbarrow. Every load costs its weight times the distance you push it. A transport plan says how much sand goes from each spot to each spot. The Wasserstein-1 or earth mover's distance is the cost of the cheapest plan.

Tiny example

Two plans move our real pile onto the fake pile. In order: 0 → 1 and 3 → 4, each half the sand moved 1 step, total 0.5 + 0.5 = 1. Crossed: 0 → 4 and 3 → 1, total 0.5 × 4 + 0.5 × 2 = 3. The cheapest is 1, so W = 1. Slide the fake pile to 100 and 103, and W grows to 100, while TV, KL and JS do not budge.

Reading it: the number line runs from 0 to 4. The two filled squares above it are the real pile, half the sand at 0 and half at 3; the two open circles below it are where the fake pile's sand sits, at 1 and 4. Each arrow is one load of the plan, labelled with its cost, half the sand times the distance travelled. Switch between the plans: both end with exactly the fake pile, but crossing the loads costs three times as much. The earth mover's distance takes the cheapest plan, and in one dimension that is always the one that keeps the grains in order.

In words: “among all ways of moving the real pile onto the fake pile, find the one whose average distance moved, weighted by the amount of sand, is smallest; that smallest cost is the distance.”

With the numbers: in order: ½ × |0 − 1| + ½ × |3 − 4| = 1. Crossed: ½ × |0 − 4| + ½ × |3 − 1| = 3. The infimum over these plans is 1.

In Python:

# each plan γ: (from x, to y, how much sand)
in_order = [(0, 1, 0.5), (3, 4, 0.5)]
crossed = [(0, 4, 0.5), (3, 1, 0.5)]
def cost(plan):
    # E over (x, y) ~ γ of |x − y|
    return sum(mass * abs(x - y) for x, y, mass in plan)
cost(in_order), cost(crossed)  # → (1.0, 3.0)
# W = inf over plans
min(cost(in_order), cost(crossed))  # → 1.0
# the fake pile slid to 100 and 103: W grows with the distance
cost([(0, 100, 0.5), (3, 103, 0.5)])  # → 100.0

Why it matters today

The earth mover's distance is the one yardstick in this section that shrinks as the fake pile approaches. In one dimension it is easy to compute: sort both samples and pair them up, which is exactly wasserstein_1d in the lesson. In high dimensions the infimum over plans is hopeless, which is why §3 needs a trick.

Example 1: learning parallel lines · original

Everyday picture

The real data is a thin vertical line at x = 0, sand spread evenly up it. The generator draws the same line, shifted sideways to x = θ. The right answer is obviously θ = 0, and the right loss should get smaller as θ gets closer to it.

Tiny example

At θ = 0.5 the lines are half a unit apart. Every grain has to move 0.5 sideways, so W = 0.5. The lines do not overlap anywhere, so JS = log 2, TV = 1 and both KLs are infinite. At θ = 0.01 the lines are almost on top of each other, W = 0.01, and the other three have not changed at all.

Reading it: the picture at the top shows the square from x = −1 to 1: the dotted line is the real data at x = 0 and the solid line the generator at x = θ. Drag θ and watch the readout: W follows the gap exactly, while JS, TV and KL stay pinned at their largest values until the lines coincide. The chart below is the paper's Figure 1 redrawn. The solid V is W = |θ|: it slopes down towards 0 from both sides, so its gradient always points home. The dashed line is JS: flat at log 2 = 0.693 everywhere except the single point θ = 0, where it drops to 0. A flat line has zero slope, so gradient descent on JS goes nowhere.

In words: “the earth mover's distance between the two lines is just how far apart they are; the Jensen-Shannon divergence is log 2 unless they sit exactly on top of each other.” The paper adds that both KLs are +∞ and TV is 1 for every θ ≠ 0.

With the numbers: θ = 0.5 gives W = 0.5 and JS = 0.693; θ = 0.01 gives W = 0.01 and still JS = 0.693; only θ = 0 gives JS = 0.

In Python:

import math
def W(theta):
    # every grain moves |θ| sideways
    return abs(theta)
def JS(theta):
    # the lines overlap only when θ = 0
    return 0.0 if theta == 0 else math.log(2)
[W(t) for t in (0.5, 0.01, 0)]  # → [0.5, 0.01, 0]
[round(JS(t), 3) for t in (0.5, 0.01, 0)]  # → [0.693, 0.693, 0.0]

“The EM plot is continuous and provides a usable gradient everywhere. The JS plot is not continuous and does not provide a usable gradient.”Arjovsky, Chintala and Bottou (2017), Figure 1 caption

Why it matters today

This two-line toy is the whole argument of the paper in one picture. The paper notes it is not a special case: two thin surfaces in a big space, placed at random, almost never overlap in more than a set of size zero, which is exactly the situation of a young generator and real images. The GAN lesson redraws the same comparison for a pile sliding along a line.

Theorems 1 and 2: continuity, and which distance is weakest · original

Everyday picture

A thermostat that jumps straight from 15 to 25 degrees cannot be tuned; one that turns smoothly can. Theorem 1 says the earth mover's distance turns smoothly: if the generator changes smoothly with its weights, W changes smoothly too, and for networks it has a slope almost everywhere (the Corollary). JS and the KLs do not, and Example 1 is the counterexample. Theorem 2 ranks the distances by how strict they are about calling two piles close: KL is strictest, JS and TV next (they agree with each other), and W is the most lenient, agreeing with plain convergence of the samples.

Tiny example

Put all the fake sand at the single point 1/n, and all the real sand at 0. As n grows, the fake point creeps towards the real one: W = 1, 0.1, 0.01 for n = 1, 10, 100. TV stays at 1 and JS at log 2 for every n, because the points never quite coincide. Under W the sequence converges; under TV and JS it never does. That is what “weaker” means here.

In Python:

import math
ns = [1, 10, 100]
# W between a point at 0 and a point at 1/n: the gap
[1 / n for n in ns]  # → [1.0, 0.1, 0.01]
# TV between two points that don't coincide: always 1
[1 for n in ns]  # → [1, 1, 1]
# JS: always log 2
[round(math.log(2), 3) for n in ns]  # → [0.693, 0.693, 0.693]

Reading it: each box is a way for a sequence of distributions ℙ_n to approach ℙ, and each arrow reads “if this happens, so does the one below”. Going down, the conditions get easier to meet. The creeping point from the tiny example lives only in the green box: it converges under W but never makes it into the yellow or amber boxes. A loss built on a lower box is continuous for more generators, which is exactly what training needs.

Why it matters today

The paper concludes that KL, JS and TV are “not sensible cost functions” for distributions on thin surfaces, and that W is. Every later Wasserstein-style GAN rests on this argument.

3 Wasserstein GAN · original

Everyday picture

Finding the cheapest way to move a pile of a million images onto another is hopeless. But there is a back door. Instead of planning the moves, hire a critic who assigns every position a price, with one rule: prices may not change faster than distance (move one step, the price changes by at most 1). The critic tries to make real sand as expensive and fake sand as cheap as possible. The biggest gap it can achieve equals the cost of the cheapest move.

Why it matters

That back door turns a planning problem into a training problem: the critic is a neural network, and the gap is something you can estimate from minibatches and backpropagate through, exactly like a GAN.

The duality (equation 2) · original

Everyday picture

A function is K-Lipschitz if its value never changes more than K times as fast as its input: a speed limit on its slope. The Kantorovich-Rubinstein duality says the earth mover's distance equals the largest gap any 1-Lipschitz price function can create between the average price of real and fake sand.

Tiny example

Try the price f(x) = −x, which has slope exactly 1. Real sand averages −(0 + 3)/2 = −1.5 and fake sand −(1 + 4)/2 = −2.5, a gap of 1, which is W. A lazy critic, f(x) = 0, gets a gap of 0. A cheating critic, f(x) = −2x, gets 2, but it breaks the speed limit (its slope is 2), and with slope K the gap is K times W.

In words: “the earth mover's distance is the biggest gap between the average score of real samples and the average score of generated samples that any function with slope at most 1 can produce.”

With the numbers: f(x) = −x gives −1.5 − (−2.5) = 1 = W. The speed limit is what makes the answer finite: f(x) = −2x gives 2, and f(x) = −100x would give 100.

In Python:

real = [0, 3]
fake = [1, 4]
def gap(f):
    # E over real of f(x) − E over fake of f(x)
    return sum(f(x) for x in real) / len(real) - sum(f(x) for x in fake) / len(fake)
# a 1-Lipschitz critic that tilts the right way
gap(lambda x: -x)  # → 1.0
# a lazy critic
gap(lambda x: 0)  # → 0.0
# slope 2 breaks the speed limit and doubles the gap: K · W with K = 2
gap(lambda x: -2 * x)  # → 2.0

In words: “f is K-Lipschitz when, for any two inputs, the outputs are never further apart than K times the inputs are.” (The paper uses the notation ‖f‖_L ≤ K without writing out this definition; it is the standard one.)

With the numbers: for f(x) = −2x, the inputs 1 and 4 are 3 apart and the outputs −2 and −8 are 6 apart, so K must be at least 6 / 3 = 2. For f(x) = −x the ratio is 1.

In Python:

points = [0, 1, 3, 4]
def worst_ratio(f):
    # the largest |f(a) − f(b)| / |a − b| over every pair: the smallest K that works
    return max(abs(f(a) - f(b)) / abs(a - b) for a in points for b in points if a != b)
worst_ratio(lambda x: -x)  # → 1.0
worst_ratio(lambda x: -2 * x)  # → 2.0

Why it matters today

Every Wasserstein-style GAN is built on this line: the critic is the f, and all the engineering is about keeping its slope under control. The lesson's section on the Wasserstein distance puts it the same way: the estimate is only valid if the critic's slope is at most 1 everywhere.

The critic (equation 3) and weight clipping · original

“Weight clipping is a clearly terrible way to enforce a Lipschitz constraint.”Arjovsky, Chintala and Bottou (2017), §3

Everyday picture

You cannot search over every 1-Lipschitz function, so search over a network f_w instead, and keep its slope bounded somehow. The paper's crude but simple way: after every update, force every weight back into a small box, [−0.01, 0.01]. A network with small weights cannot be steep, so its slope is bounded by some K. The gap it finds is then K × W: the right distance, times a constant that does not matter for training.

Tiny example

Take the smallest possible critic, f_w(x) = w · x, with weight clipping at c = 0.01. Its gap on our piles is w × (1.5 − 2.5) = −w. The best clipped choice is w = −0.01, giving a gap of 0.01: exactly W = 1 scaled by K = 0.01.

In words: “over all allowed critic weights, find the critic that makes real samples score as high as possible, on average, compared with generated ones.”

With the numbers: with f_w(x) = w · x and w clipped to [−0.01, 0.01], the value is −w, largest at w = −0.01, where it is 0.01 = 0.01 × W.

In Python:

real = [0, 3]
fake = [1, 4]
c = 0.01
def objective(w):
    # E over real of f_w(x) − E over fakes of f_w(g_θ(z)), with f_w(x) = w · x
    return sum(w * x for x in real) / len(real) - sum(w * x for x in fake) / len(fake)
# try every allowed weight on a fine grid of the box [−c, c]
grid = [c * k / 100 for k in range(-100, 101)]
best_w = max(grid, key=objective)
best_w  # → -0.01
round(objective(best_w), 4)  # → 0.01

The paper is blunt about the cost. Clip too loosely and the weights take a long time to reach their limits, so the critic trains slowly. Clip too tightly and gradients vanish in deep networks or without batch normalization. The authors tried alternatives, such as projecting the weights onto a sphere, found little difference, and explicitly invited better methods.

Why it matters today

The invitation was taken up almost immediately: Wasserstein GANs with a gradient penalty (WGAN-GP) replaced clipping by fining the critic when its slope strays from 1, and spectral normalization caps each layer's stretch. The lesson builds the second with largest_stretch and spectrally_normalize.

Theorem 3: the generator's gradient · original

Everyday picture

Once the critic has set its prices, the forger does not need to know anything else: it moves each of its samples in the direction that makes the critic's price go up. Theorem 3 says that this direction, averaged over samples, is exactly the gradient of the earth mover's distance, provided the critic is the best one.

Tiny example

Let the generator shift its pile: g_θ(z) = z + θ, where z is 1 or 4. With the best critic, f(x) = −x, the price of a generated sample is −(z + θ), whose slope in θ is −1. So the gradient of W is −(−1) = +1: moving the fake pile right (θ up) makes W bigger by 1 per unit, and moving it left makes it smaller. Check: at θ = −0.5 the fake pile sits at 0.5 and 3.5, and W = 0.5, down from 1.

In words: “to find how the distance changes with the generator's weights, average, over noise draws, how the best critic's score on each generated sample changes with those weights, and flip the sign.”

With the numbers: ∇_θ f(g_θ(z)) = ∇_θ[−(z + θ)] = −1 for both z, the average is −1, and the minus sign makes ∇_θW = +1.

In Python:

z_values = [1, 4]
real = [0, 3]
def f(x):
    # the best 1-Lipschitz critic for these piles
    return -x
def g(z, theta):
    return z + theta
h = 1e-6
# ∇_θ f(g_θ(z)) for each z, by nudging θ
slopes = [(f(g(z, h)) - f(g(z, -h))) / (2 * h) for z in z_values]
[round(s, 6) for s in slopes]  # → [-1.0, -1.0]
# ∇_θ W = −E_z[∇_θ f(g_θ(z))]
round(-sum(slopes) / len(slopes), 6)  # → 1.0
# check directly: W with the fake pile moved to θ = −0.5
sum(abs(a - b) for a, b in zip(real, [g(z, -0.5) for z in z_values])) / 2  # → 0.5

Why it matters today

This is what licenses the practical recipe: estimate the critic, then backpropagate its score into the generator, just as a GAN backpropagates the discriminator's verdict. The difference is that the signal is the slope of a distance, not the slope of a classifier's confidence.

Algorithm 1 · original

Everyday picture

The same turn-taking as the original GAN, with three changes: the critic gets several turns (five) for each of the forger's, its weights are clipped after every turn, and both players use RMSProp with a tiny step. The paper's defaults are α = 0.00005, c = 0.01, a batch of m = 64 and n_critic = 5.

Tiny example

With the one-weight critic f_w(x) = w · x on our piles, the critic's gradient is (average real) − (average fake) = 1.5 − 2.5 = −1. Starting from w = −0.00999, one step of size α = 0.00005 in that direction gives −0.01004, and clipping pulls it back to −0.01. Then the forger's gradient, for the shift g_θ(z) = z + θ, is −w = 0.01, so its step moves θ down: the fake pile slides left, towards the real one.

Reading it: the outer box is one generator iteration and the inner box is the critic's turn, repeated five times. In each critic turn a batch of real samples and a batch of fakes feed one RMSProp step that widens the gap in average score (blue), and then the clip (amber) forces every critic weight back into [−0.01, 0.01]. Only then does the generator draw fresh noise and take one step that raises the critic's average score on its samples (green). The arrow on the right starts the next iteration. Compared with the GAN's Algorithm 1, there is no log and no sigmoid anywhere: the critic's raw score is the loss.

The math

Line 7, the critic's gradient, and line 9, the clip:

In words: “find which way each critic weight should move to widen the gap between its average score on the real batch and on the fake batch; after the step, snap every weight back into the box from −c to c.”

With the numbers: for f_w(x) = w · x, g_w = 1.5 − 2.5 = −1. RMSProp scales the step to about α per weight here, so w = −0.00999 becomes −0.01004 and the clip returns it to −0.01.

In Python:

real, fake = [0, 3], [1, 4]
alpha, c = 0.00005, 0.01
# g_w for f_w(x) = w · x: mean of real x minus mean of fake x
g_w = sum(real) / len(real) - sum(fake) / len(fake)
g_w  # → -1.0
# RMSProp divides by the root of a running mean of g_w²; with a steady gradient that is |g_w|
w = -0.00999
w = w + alpha * g_w / abs(g_w)
round(w, 5)  # → -0.01004
# clip(w, −c, c)
w = max(-c, min(c, w))
w  # → -0.01

Line 12, the generator's gradient:

In words: “the generator's gradient is minus the slope, with respect to its weights, of the critic's average score on its samples”; the step θ ← θ − α · RMSProp(θ, g_θ) then raises that score.

With the numbers: f_w(g_θ(z)) = w(z + θ) has slope w = −0.01 in θ, so g_θ = +0.01, and the step moves θ down by about α = 0.00005: the fake pile slides left, towards the real pile.

In Python:

w, alpha = -0.01, 0.00005
z_values = [1, 4]
h = 1e-6
def mean_score(theta):
    # (1/m) Σ f_w(g_θ(z^(i))) with f_w(x) = w · x and g_θ(z) = z + θ
    return sum(w * (z + theta) for z in z_values) / len(z_values)
g_theta = -(mean_score(h) - mean_score(-h)) / (2 * h)
round(g_theta, 6)  # → 0.01
# θ ← θ − α · RMSProp step, which is about α times the sign of g_θ
theta = 0.0 - alpha * g_theta / abs(g_theta)
theta  # → -5e-05

Why it matters today

Several critic steps per generator step, and a critic whose raw score is the loss, are the two habits every Wasserstein-style GAN inherits from this algorithm. The optimizer choice was a response to trouble with Adam that §4 describes.

Figure 2: train the critic to the end · original

Everyday picture

In an ordinary GAN, a detective trained too well becomes useless: it rejects every fake with certainty, and certainty is flat. A critic trained too well cannot become useless, because its speed limit forces it to keep sloping. So a WGAN wants the critic trained as well as possible before each generator step, the opposite of the GAN's balancing act.

Tiny example

Two bell curves of width 1, the real one centred at −2 and the fake at +2. The best GAN detective is D*(x) = 1 / (1 + e^4x). At the fake pile's centre, x = 2, its verdict is 0.0003 and its slope is about −0.001: almost nothing for the forger to follow. The best critic here is the straight line f(x) = −x, with slope −1 everywhere, including at the fakes.

Reading it: the x-axis is position. The first two curves are the real and fake piles, bell curves drawn to a peak of 1. The third is the best GAN discriminator D*(x), the share of what lands at x that is real. The fourth is the best critic, the straight line f(x) = −x, drawn rescaled so that it fits the same 0 to 1 axis; its true slope is −1 everywhere. The readout gives both slopes at the fake pile's centre, the place the generator needs directions. Drag the piles apart: D* turns into a cliff between the piles, flat at 0 over the fakes, and its slope there collapses towards zero; the critic's slope does not change. Drag them close together and D* becomes gentle again. These are two 1-D bell curves computed in your browser, not the paper's experiment.

In Python:

import math
# real pile centred at −2, fake at +2: log p_r(x) − log p_g(x) = −4x, so D*(x) = σ(−4x)
def D_star(x):
    return 1 / (1 + math.exp(4 * x))
round(D_star(2), 4)  # → 0.0003
# D*'s slope at the fakes' centre, by nudging x
h = 1e-6
round((D_star(2 + h) - D_star(2 - h)) / (2 * h), 4)  # → -0.0013
# the critic f(x) = −x has slope −1 there, as everywhere
def f(x):
    return -x
round((f(2 + h) - f(2 - h)) / (2 * h), 4)  # → -1.0

“Perhaps more importantly, the fact that we can train the critic till optimality makes it impossible to collapse modes when we do.”Arjovsky, Chintala and Bottou (2017), §3

Why it matters today

The paper's argument about mode collapse is that, against a fixed discriminator, the forger's best move is to put all its samples on the few points the discriminator likes most. A critic trained to the end does not hand the forger such a point to collapse onto. The GAN lesson's mode-collapse experiment on the ring of eight clouds shows the collapse this is meant to prevent.

4 Empirical results · original

Everyday picture

The paper claims two practical benefits: a loss curve you can actually read, and training that survives architectures that usually break a GAN. The testbed is LSUN bedrooms, generated at 64 × 64 pixels, against a DCGAN baseline trained the standard way with the −log D trick. Every experiment uses Algorithm 1's default settings.

Why it matters

Before this paper a GAN's losses told you little about whether training was working; you had to look at samples. A loss that tracks quality changes how people debug and tune these models.

4.2 A loss that means something · original

Everyday picture

A GAN detective's loss is like a referee's whistle count: it goes up and down with the game, not with how good the players are. A WGAN critic's gap is like a distance on a map: it shrinks as the forger gets closer.

Tiny example

The paper also tracks the GAN's own number for comparison. Its detective maximises L, the value of the GAN game, which is at most 2 JS − 2 log 2; the paper plots ½ L + log 2, an estimate of JS from below. A detective that can only guess (D = ½ everywhere) gives L = −1.386 and an estimate of 0. A detective that is always right gives L = 0 and an estimate of log 2 = 0.693, the ceiling, whatever the samples look like. The paper reports the GAN curves often sit near that ceiling.

In words: “the value of the GAN game for a given detective is never more than twice the Jensen-Shannon divergence minus 2 log 2, so half of it plus log 2 is an estimate of JS from below.”

With the numbers: D = ½: L = 2 log ½ = −1.386 and ½ L + log 2 = 0. D always right: L = 0 and ½ L + log 2 = 0.693.

In Python:

import math
# a guessing detective: log(1/2) on real, log(1 − 1/2) on fake
L = math.log(0.5) + math.log(1 - 0.5)
round(L, 3)  # → -1.386
round(L / 2 + math.log(2), 3)  # → 0.0
# a detective that is always right: log 1 + log 1
L = 0.0
round(L / 2 + math.log(2), 3)  # → 0.693

What the figures show

• Figure 3 (WGAN): for an MLP generator and a DCGAN generator, the critic's estimate of the earth mover's distance falls steadily as the samples improve. A run with learning rates set so high that training failed shows a flat loss and unchanging samples.
• Figure 4 (standard GAN): the JS estimate stays flat or rises even while a DCGAN's samples improve, and swings up and down regardless of quality for an all-MLP GAN.

The authors are careful: the critic's scale depends on its architecture, so the number cannot yet compare models with different critics, and a finite critic only approximates W. But they report using it to validate experiments “repeatedly and without failure”.

A negative result about momentum

Training became unstable at times with Adam (β₁ > 0) on the critic, or with high learning rates. When the loss blew up, the cosine between Adam's step and the gradient usually turned negative: the momentum was pushing against the current gradient, because the critic's loss keeps changing underneath it. Hence RMSProp. For example, with made-up two-weight numbers, a step of (1, 1) against a gradient of (−1, 0.2) has cosine −0.55: the step points more against the gradient than along it.

In Python:

import math
# illustrative numbers, not from the paper
step, grad = (1, 1), (-1, 0.2)
dot = sum(s * g for s, g in zip(step, grad))
round(dot / (math.hypot(*step) * math.hypot(*grad)), 2)  # → -0.55

Why it matters today

A training curve that tracks sample quality is the practical reason many people reached for this paper. And the observation that momentum fights a moving target is why GAN code commonly lowers Adam's β₁, as the lesson's train_gan does with β₁ = 0.5.

4.3 Improved stability · original

“In no experiment did we see evidence of mode collapse for the WGAN algorithm.”Arjovsky, Chintala and Bottou (2017), §4.3

Everyday picture

A standard GAN is like a seesaw that only balances with the right two children on it: change either player's build and it tips over. With a critic trained to the end, there is no balance to keep, so the generator can be almost anything that produces images.

Tiny example

The paper keeps a DCGAN critic (or discriminator) fixed and swaps in three generators. We summarise Figures 5 to 7 in words:

In Python:

# the three generators in §4.3, and whether each method still produced samples
results = {
    "DCGAN": ("WGAN works", "GAN works"),
    "no batch norm, constant filters": ("WGAN works", "GAN fails"),
    "4-layer ReLU MLP": ("WGAN works", "GAN collapses"),
}
# how many of the three each method survives
sum(w == "WGAN works" for w, g in results.values()), sum(g == "GAN works" for w, g in results.values())  # → (3, 1)

Why it matters today

Robustness to architecture is what made Wasserstein-style training attractive. The lesson shows the flip side on a toy: with the ordinary GAN loss, changing only the detective's learning rate decides whether the forger finds all eight clouds.

5 Related work · original

Everyday picture

Equation 2 is one member of a family. Pick any club of allowed critics; the distance is the largest gap in average score any club member can create between the piles. The club decides everything. Slope-limited critics give the earth mover's distance. Critics that are merely bounded, however steep, can build a cliff between the piles and give the total variation distance, with all its flatness.

Tiny example

On our piles, a bounded critic that scores +1 on the real spots and −1 on the fake ones scores a gap of 2 whether the fake pile is at 1 and 4 or at 100 and 103: it does not care how far away the fakes are. The slope-limited critic f(x) = −x scores 1 and 100 in those two cases.

In words: “for a chosen class of functions 𝓕, the distance is the biggest gap between average scores on real and generated samples that any function in the class can produce.” This is an integral probability metric.

With the numbers: bounded class, the ±1 critic: 1 − (−1) = 2 for the near fakes and again 2 for the far ones. Slope-limited class, f(x) = −x: 1 and 100.

In Python:

real = [0, 3]
near, far = [1, 4], [100, 103]
def gap(f, fake):
    return sum(f(x) for x in real) / len(real) - sum(f(x) for x in fake) / len(fake)
def bounded(x):
    # +1 on the real spots, −1 anywhere else: bounded, but as steep as it likes
    return 1 if x in real else -1
gap(bounded, near), gap(bounded, far)  # → (2.0, 2.0)
gap(lambda x: -x, near), gap(lambda x: -x, far)  # → (1.0, 100.0)

The relatives the paper compares

• Energy-based GANs end up minimising total variation (proved in the paper's Appendix D), so they inherit the flatness of JS.
• Maximum mean discrepancy needs no critic network, but its cost grows with the square of the batch size, and generators trained on it directly (generative moment matching networks) saw limited success. The paper's estimate for 64 × 64 images: batches of at least 4,096, and a cost per iteration of 4,096², over five orders of magnitude more than a GAN iteration with a batch of 64.

In Python:

# MMD's cost grows with the square of the batch; a GAN's with the batch
mmd_cost, gan_cost = 4096 ** 2, 64
mmd_cost // gan_cost  # → 262144
# how many orders of magnitude that is
len(str(mmd_cost // gan_cost)) - 1  # → 5

Why it matters today

The integral-probability-metric view is a useful way to read any critic-based loss: say which critics are allowed, and you have said which distance is minimised and how it behaves when the piles do not overlap.

6 Conclusion and appendices · original

Everyday picture

Change the yardstick and the game changes with it: a critic that measures distance cannot be won so completely that it stops teaching.

What the paper concludes

WGAN improves the stability of learning, avoids mode collapse in the authors' experiments, gives learning curves useful for debugging and hyperparameter search, and rests on a sound optimisation problem. The appendices supply the mathematics: why W is a weak distance and TV a strong one (A), the regularity assumption on the generator (B), the proofs of Theorems 1 to 3 and the Corollary (C), the result that energy-based GANs minimise total variation (D), the standard GAN's generator cost during training (E), and full sheets of samples (F).

Why it matters today

The strong-versus-weak framing explains, in one idea, why the original GAN's gradients vanish when real and generated data do not overlap, and what a fix has to preserve: a loss that still shrinks as the fake pile moves closer.

What changed since 2017

The GAN lesson's fixes section runs these descendants side by side: a gradient penalty that turns the two-number Dirac GAN's orbit into an inward spiral, spectral normalization by power iteration, and the Wasserstein distance itself. For where it all started, see the GAN companion.

Glossary

Every term with hover guidance on this page, in one place.