Flow Straight and Fast: Rectified Flow, annotated

2.1 Overview · original

“By fitting the drift v with X₁ − X₀, the rectified flow causalizes the paths of linear interpolation X_t, yielding an ODE flow that can be simulated without seeing the future.”Liu, Gong and Liu (2022), §2.1

Everyday picture

Draw a straight line from each starting point to its partner. A point moving along its line needs to know where it is going: that is “seeing the future”, and a generator can't. So train a network to look only at where a point is and when, and to guess the line's direction. Where several lines pass through the same spot, the network can only give their average. Following those averaged arrows is an ordinary differential equation that anyone can run forwards from noise, with no knowledge of partners.

Tiny example

Noise X₀ = −1 paired with data X₁ = 2. A quarter of the way along, at t = 0.25, the line passes −0.25, and it moves at 2 − (−1) = 3 the whole way. The network is shown the point −0.25 and the time 0.25 and asked for 3; a guess of 2.5 costs (3 − 2.5)² = 0.25. These are the same numbers as the diffusion lesson's flow matching step, because it is the same loss.

In words: “find the velocity field that, averaged over every pair and every moment, best matches the straight line's direction at the point where the line is at that moment.”

With the numbers: X_t = 0.25 × 2 + 0.75 × (−1) = −0.25; target 2 − (−1) = 3; a guess of 2.5 scores 0.25.

In Python:

x0, x1, t = -1.0, 2.0, 0.25
# X_t = t X_1 + (1 − t) X_0: where the line is at time t
x_t = t * x1 + (1 - t) * x0
x_t  # → -0.25
# X_1 − X_0: the line's direction, the same at every t
target = x1 - x0
target  # → 3.0
# a guess of 2.5, graded by squared error
(target - 2.5) ** 2  # → 0.25

Reading it: the top box is one training step, and it contains no simulation: a random pair and a random time give a point on the pair's line and the line's direction, the network sees only the point and the time, and the squared error is the loss. The bottom box starts once the network is trained: follow its arrows from fresh noise to get a sample, which pairs each starting point Z₀ with an end point Z₁. Those new pairs can go round again (the long wire back to the top, “reflow”), which makes the next flow straighter, or be distilled into a network that jumps from Z₀ to Z₁ in one call.

Figure 2, redrawn: crossing lines, rewired paths

Reading it: hollow circles are starting points (π₀, two blobs on the left), filled dots are where they end (π₁, two blobs on the right); 40 of the 250 journeys are drawn, solid if they start in the upper blob and dashed if in the lower. Random pairs: half the lines cross over to the far blob, making a big X in the middle. 1-rectified flow: the learned paths never cross; at the crossing they are “rewired”, each staying on its own side and bending to the nearer target blob. Its new pairs: join each start to where its flow ended, with straight lines: almost no crossings left. 2-rectified flow: the flow trained on those pairs is nearly straight. Now drag N down to 1. The 1-rectified flow's single step sends each point to the average end of the few training lines that start near it; with random pairs those ends are half up and half down, so the samples land in the empty gap between the two right blobs, scattered up and down and crossing each other. The 2-rectified flow's single step lands close to where its 60-step paths end. The readout gives each picture's average squared jump length (its transport cost) and its straightness (0 means perfectly straight).

Why it matters today

The loss is the same one the flow matching paper arrived at independently, with its tiny σ_min set to 0. The lesson trains it in train_velocity_predictor, with flow_point as X_t and flow_target as X₁ − X₀, and its figure of crossing training lines and bending learned paths is the picture above.

2.2 Main results and properties · original

“It is not recommended to apply too many reflow steps as it may accumulate estimation error on v^X.”Liu, Gong and Liu (2022), §2.2

The best possible arrow: an average

Everyday picture: at a busy crossroads, people heading many ways pass the same spot at the same moment. A single signpost there can only point the average way. Tiny example: in the one-dimensional toy, many lines pass through 2 at t = 0.5: from 0 to 4 (speed 4), from 1 to 3 (speed 2), from −1 to 5 (speed 6), and so on. Weighted by how likely each pairing is, their average speed is exactly 2.4.

In words: “the best velocity at a place and time is the average direction of all the lines that pass through that place at that time” (the paper's equation 2, written with the average's condition pulled out front).

With the numbers: for two independent bell curves the average is a straight-line function of x: v^X(x, t) = 3 + (c/q)(x − 3t), where q = (1 − t)² + 0.25t² is the spread of X_t squared and c = 0.25t − (1 − t) is how X₁ − X₀ moves with X_t. At t = 0.5, x = 2: q = 0.3125, c = −0.375, so v = 3 − 1.2 × 0.5 = 2.4.

In Python:

t, x = 0.5, 2.0
# three of the many lines through x = 2 at t = 0.5, and their speeds
for x0, x1 in [(0, 4), (1, 3), (-1, 5)]:
    print((1 - t) * x0 + t * x1, x1 - x0)  # → 2.0 4 2.0 2 2.0 6
# the exact average for noise N(0, 1) paired at random with data N(3, 0.5²):
# q is the variance of X_t, c its covariance with X_1 − X_0
q = (1 - t) ** 2 + 0.25 * t * t
c = 0.25 * t - (1 - t)
q, c  # → (0.3125, -0.375)
3 + c / q * (x - 3 * t)  # → 2.4

Marginals are preserved, couplings get cheaper

Following the average arrows keeps the crowd at every moment exactly as the straight lines had it (Theorem 3.3, §3.1): same spread of positions at t = 0.25, 0.5, 0.9, everything. So the flow still carries π₀ onto π₁. And it gets there more cheaply (Theorem 3.5). The paper's picture proof, for the cost “distance travelled”, redrawn:

Reading it: each panel has two starting points, A and B, on its left and two destinations, C and D, on its right; the solid line is A's journey and the dashed line B's. Left: the random pairing sends A to D and B to C; the lines cross at P, each 4.47 long, 8.94 together. Middle: a flow cannot let paths cross, so at P each path turns: A ends at C and B at D. The rewired paths are the same four half-lines rearranged, so the total is still 8.94: that is the “(**)” step of the paper's proof. Right: the new pairs, A with C and B with D, joined straight. A straight line is never longer than a bent one, so these are 4 each, 8 together: the “(*)” step, the triangle inequality.

In words: “the new pairs are no farther apart, on average, than the lengths of the flow's paths, which are the old straight lines cut and re-joined, so no longer than the old pairs.”

With the numbers: old pairs √(4² + 2²) = 4.472 each; rewired paths 2 × √(2² + 1²) = 4.472 each; new pairs 4 each. Average: 4 ≤ 4.472 = 4.472.

In Python:

import math
A, B, C, D, P = (0, 2), (0, 0), (4, 2), (4, 0), (2, 1)
# the random pairs A→D and B→C, as straight lines
old = (math.dist(A, D) + math.dist(B, C)) / 2
# the rewired paths A→P→C and B→P→D: the same pieces
rewired = (math.dist(A, P) + math.dist(P, C) + math.dist(B, P) + math.dist(P, D)) / 2
# the new pairs A→C and B→D, straight
new = (math.dist(A, C) + math.dist(B, D)) / 2
round(new, 3), round(rewired, 3), round(old, 3)  # → (4.0, 4.472, 4.472)

Straightness, and why reflow gives it

Everyday picture: a path is straight when the direction you are moving at every moment is the same as the direction from start to finish. Tiny example: the path z_t = t² goes from 0 to 1, so the overall direction is 1, but its speed at time t is 2t: slow, then fast. Its straightness score is the average of (1 − 2t)² over the journey: 1/3. The straight path z_t = t scores 0.

In words: “straightness is the average squared gap between the velocity at each moment and the overall start-to-finish direction; zero means every path is a straight line travelled at constant speed.”

With the numbers: for z_t = t², the integral of (1 − 2t)² from 0 to 1 is 1/3. For the one-dimensional toy's 1-rectified flow, whose paths bend (§3.4), it is 0.2146.

In Python:

n = 100_000
# the path z_t = t²: overall direction 1, velocity 2t at time t
S = sum((1 - 2 * ((i + 0.5) / n)) ** 2 for i in range(n)) / n
round(S, 4)  # → 0.3333

Theorem 3.7 promises that reflow drives straightness toward zero, at a guaranteed pace:

In words: “after K rounds of reflow, at least one of the flows has straightness no worse than the original pairs' average squared distance divided by K.” The total straightness you can ever remove is bounded by the starting cost, because each round's straightening is paid for by a drop in cost (§3.3).

With the numbers: the toy starts at 10.25, so the bound is 10.25 after one round, 5.125 after two, 1.025 after ten. It is a guarantee, and loose: the toy's first flow already has straightness 0.2146, and its second is exactly straight (§3.4).

In Python:

# E‖X_1 − X_0‖² for noise N(0, 1) paired at random with data N(3, 0.5²)
cost = 3 ** 2 + 1 + 0.5 ** 2
cost  # → 10.25
[cost / K for K in (1, 2, 10)]  # → [10.25, 5.125, 1.025]

Reading it: the x-axis counts rectifications: 0 is the random pairing, 1 the 1-rectified flow, and so on. The y-axis is relative: the solid line is the straightness of each flow as a share of the first flow's, and the dashed line the transport cost of each coupling as a share of the random pairing's. Straightness collapses after one reflow (to a few percent of where it started) and creeps down after that; the cost drops once, when the crossing lines are removed, and then hardly moves. That matches the paper's advice that one or two reflows are enough, and that too many can pile up estimation error.

Burgers' equation. A flow is exactly straight when every point keeps its velocity for the whole trip. That makes the velocity field special: it must satisfy the inviscid Burgers' equation.

In words: “if you ride along with the flow, the velocity you feel never changes: the change at a fixed spot is exactly cancelled by the change from moving to a new spot.”

With the numbers: the field v(z, t) = z/(1 + t) carries each point along a straight line z_t = z₀(1 + t). At z = 2, t = 0.5: ∂_tv = −0.8889 and (∂_zv)v = +0.8889; they cancel.

In Python:

def v(z, t):
    return z / (1 + t)
z, t, d = 2.0, 0.5, 1e-6
# ∂_t v and ∂_z v by small finite differences
dv_dt = (v(z, t + d) - v(z, t - d)) / (2 * d)
dv_dz = (v(z + d, t) - v(z - d, t)) / (2 * d)
round(dv_dt, 4), round(dv_dz * v(z, t), 4)  # → (-0.8889, 0.8889)
abs(dv_dt + dv_dz * v(z, t)) < 1e-6  # → True

Distillation, and learning the arrows without a network

After k rounds, a network can be trained to jump from Z₀ to Z₁ in one call: distillation. Taking the jump as z₀ + v(z₀, 0) turns its loss into the rectified flow loss at t = 0 alone. The paper stresses the difference: distillation copies a coupling faithfully, while rectification changes the coupling to a cheaper, straighter one, so distillation belongs at the very end.

For low-dimensional toys the paper replaces the network with a smoothing formula that averages the lines near a point, the Nadaraya-Watson estimator (equation 5). Every picture on this page that learns a flow uses it.

In words: “to guess the arrow at z, look at every training line's position at time t; for each, the direction that would carry z to that line's end point in the time left; average those directions, weighting lines that pass close to z far more.” The kernel κ_h(x, z) = exp(−‖x − z‖²/2h²) sets what “close” means.

With the numbers: three training pairs (0, 4), (1, 2), (−2, 3) sit at 2, 1.5 and 0.5 at t = 0.5. At z = 1.8 with h = 0.5, their weights are 0.923, 0.835 and 0.034, and the directions to their end points are 4.4, 0.4 and 2.4. The weighted average is 2.498: mostly the two nearby lines, split nearly evenly.

In Python:

import math
pairs = [(0, 4), (1, 2), (-2, 3)]
t, z, h = 0.5, 1.8, 0.5
# κ_h(X_t, z): how close each line passes to z at time t
kappa = [math.exp(-(((1 - t) * x0 + t * x1) - z) ** 2 / (2 * h * h)) for x0, x1 in pairs]
[round(k, 3) for k in kappa]  # → [0.923, 0.835, 0.034]
# (X_1 − z)/(1 − t): the direction that reaches each end point in the time left
aims = [(x1 - z) / (1 - t) for x0, x1 in pairs]
[round(a, 1) for a in aims]  # → [4.4, 0.4, 2.4]
# ω_h = κ_h / Σ κ_h, then the weighted average
round(sum(k * a for k, a in zip(kappa, aims)) / sum(kappa), 3)  # → 2.498

Why it matters today

This section is the reason “one or two steps” is possible at all. Every sampler takes straight steps; straight paths are the only ones such steps follow without error. Reflow comes with a guarantee that learned paths get straighter, and distilling a straightened flow is a route to one-step generators with no adversarial training.

2.3 A nonlinear extension · original

“Such generalized rectified flows can still transport π₀ to π₁ (Theorem 3.3), but no longer guarantee to decrease convex transport costs, or have the straightening effect.”Liu, Gong and Liu (2022), §2.3

Everyday picture

Nothing forces the roads between partners to be straight. Any smooth curve from X₀ to X₁ works, and averaging its direction still gives a flow that moves the crowd correctly. What you lose is the two guarantees above, cheaper couplings and straightening, which relied on straight lines being the shortest roads.

Tiny example

Take the road X_t = t²X₁ + (1 − t²)X₀. It is the same straight line as before, but travelled slow-then-fast. With X₀ = −1 and X₁ = 2, at t = 0.5 it is at −0.25 (where the steady road was at t = 0.25) and moving at speed 3.

In words: “the same loss with any road: match the road's velocity at the road's position, weighting each moment by w_t.” With α_t = t, β_t = 1 − t and w_t = 1 it is equation 1 again.

With the numbers: α = 0.25, β = 0.75, α̇ = 1, β̇ = −1 at t = 0.5: X_t = 0.5 − 0.75 = −0.25, Ẋ_t = 2 + 1 = 3.

In Python:

t, x0, x1 = 0.5, -1.0, 2.0
# α_t = t², β_t = 1 − t², and their rates of change
alpha, beta = t * t, 1 - t * t
alpha_dot, beta_dot = 2 * t, -2 * t
alpha * x1 + beta * x0, alpha_dot * x1 + beta_dot * x0  # → (-0.25, 3.0)

Why it matters today

This one generalisation is enough to fit diffusion's samplers inside the framework, as the next subsection shows, and to compare roads on equal terms.

2.3.1 Probability flow ODEs and DDIM · original

“The linear rectified flow yields nearly straight trajectories with one step of reflow. But the trajectories of VP ODE and sub-VP ODE are curved and can not be straightened by reflowing.”Liu, Gong and Liu (2022), Figure 4 caption

Everyday picture

The score SDE paper's probability flow ODEs, and DDIM, turn out to be rectified flows along curved, unevenly paced roads. The shape of the road was never chosen for sampling: it was inherited from a noising process designed years earlier for stochastic models.

Tiny example

Halfway along the VP road (t = 0.5), the moving point is only 28% data and 96% noise (for VP the squares of the two shares add to 1, not the shares). On the straight road it is 50% of each.

In words: “the data share rises slowly and then steeply toward 1, following the exponential fade of the score SDE's VP noising read backwards; the noise share is set so the two shares' squares add to 1 (VP), or so it shrinks faster (sub-VP).” The defaults a = 19.9 and b = 0.1 are the score SDE paper's β_max − β_min and β_min.

With the numbers: at t = 0.5: α = e^−1.26875 = 0.2812; VP β = 0.9597; sub-VP β = 0.9209. The straight road has α = β = 0.5.

In Python:

import math
a, b, t = 19.9, 0.1, 0.5
# α_t = exp(−¼ a (1 − t)² − ½ b (1 − t))
alpha = math.exp(-0.25 * a * (1 - t) ** 2 - 0.5 * b * (1 - t))
round(alpha, 4)  # → 0.2812
# β for VP (squares add to 1) and for sub-VP
round(math.sqrt(1 - alpha ** 2), 4), round(1 - alpha ** 2, 4)  # → (0.9597, 0.9209)

Reading it: the x-axis is time from noise (0) to data (1). With data share α_t selected, the straight line is rectified flow, rising steadily from 0 to 1; the dashed curve is the shared α of VP and sub-VP, which stays below 0.2 until about t = 0.4 and then shoots up. With noise share β_t selected, rectified flow falls steadily, while VP and sub-VP hold near 1 for the first half. Both pictures say the same thing: the diffusion roads do almost nothing early and everything late, the uneven pace the paper objects to. Based on Liu, Gong and Liu (2022), Figure 6, redrawn from equations 7 and 8.

Figures 4 and 5, redrawn: the same noise, four roads, N steps

Reading it: the three outlined squares are the target clusters; dots are 300 samples, all starting from the same noise, after N Euler steps; faint lines are 40 of their journeys. Rectified flow, N = 1: everything lands on the centre, the mean of the data, exactly as the paper says a single step of a 1-rectified flow must. At N = 2 the samples have already split toward the three clusters, and by 3 to 5 steps they sit on them. VP ODE: at N = 1 or 2 the samples have barely left the noise, because the road hardly moves during the early steps; it needs about 10 steps to match what rectified flow does in 3 to 5. Sub-VP is a little better than VP. VP ODE with α = t follows VP's curved roads at a steadier pace in α, as the paper proposes; on this toy that alone does not help (at 5 steps it lands further off than VP), because the roads still bend sharply near the end. The readout gives the average distance to the nearest cluster centre; about 0.37 is the floor set by the clusters' own spread.

The VE ODE

The variance exploding ODE uses α_t = 1 and a noise share that shrinks from σ_max to almost 0, so its starting crowd is a very wide bell curve. Its roads are straight (the direction ξ never changes) but travelled at a very uneven pace; the paper leaves it out of its toys because it cannot start from the standard bell curve.

Why it matters today

The paper's conclusion is a design rule that stuck: the ODE can be learned directly, any road can be chosen, the starting distribution is free, and the canonical choice is the straight line at constant speed. Its score SDE ancestor derived its ODE from an SDE; this paper, and flow matching, choose the road first.

3.1 The marginal preserving property · original

“Because Z_t is driven by the same velocity field v^X, its marginal law Law(Z_t) solves the very same equation with the same initial condition (Z₀ = X₀).”Liu, Gong and Liu (2022), §3.1

Everyday picture

Film a crowd crossing a square from above and blur the individuals: you see a smooth density moving. That density only changes as fast as people flow in and out of each spot, and the flow at each spot is the density times the average velocity there (the continuity equation). The straight-line walkers and the sleepwalkers have the same average velocity everywhere, by construction, and start as the same crowd. So the film looks the same, frame by frame, even though individuals take different routes.

Tiny example

At t = 0.5, the straight lines' crowd X_0.5 = 0.5X₀ + 0.5X₁ has mean 1.5 and variance 0.25 + 0.0625 = 0.3125. Push 10,000 noise samples through the rectified flow to t = 0.5 with small Euler steps: their mean is 1.5 and their variance 0.31.

In words: “at every moment, the flow's crowd is distributed exactly as the straight lines' crowd”; at t = 1 that means the flow delivers exactly the data distribution (Theorem 3.3).

With the numbers: mean 1.5 and variance 0.3125 for the lines; 1.5 and 0.31 for 10,000 simulated flow paths (the small gap is sampling noise and step error).

In Python:

import random
random.seed(0)
# q(t) = Var X_t and c(t) = Cov(X_1 − X_0, X_t) for the toy
def q(t):
    return (1 - t) ** 2 + 0.25 * t * t
def c(t):
    return 0.25 * t - (1 - t)
zs = [random.gauss(0, 1) for _ in range(10_000)]
steps = 200
h = 0.5 / steps
# follow v^X(z, t) = 3 + (c/q)(z − 3t) from t = 0 to 0.5
for i in range(steps):
    t = i * h
    zs = [z + h * (3 + c(t) / q(t) * (z - 3 * t)) for z in zs]
mean = sum(zs) / len(zs)
var = sum((z - mean) ** 2 for z in zs) / len(zs)
round(mean, 2), round(var, 2), q(0.5)  # → (1.5, 0.31, 0.3125)

Why it matters today

This is the property that makes any of it a generator: the flow is guaranteed to deliver π₁ if the averaged velocity is learned well. It holds for every road, straight or not, which is why diffusion's ODEs work too.

3.2 Reducing convex transport costs · original

“The proof is based on elementary applications of Jensen's inequality.”Liu, Gong and Liu (2022), §3.2

Everyday picture

A convex cost punishes long jumps at least in proportion: squared distance, plain distance, distance to the fourth power. The proof uses one fact twice, Jensen's inequality: averaging first and then paying the cost is never dearer than paying the cost first and then averaging. The flow averages directions, so its pairs can only be cheaper.

Tiny example

In the toy, random pairs cost 10.25 in squared distance; the rectified pairs (z₀, 3 + 0.5z₀) cost 3² + 0.25 = 9.25. In plain distance, 3.0025 against 3.0000: smaller again, though barely, because both jumps are mostly the same 3 to the right.

In words: “for any convex cost at all, the rectified pairs cost no more than the pairs you started with” (Theorem 3.5). No particular cost is being minimised: all of them go down together.

With the numbers: X₁ − X₀ is a bell curve with mean 3 and variance 1.25; Z₁ − Z₀ = 3 − 0.5z₀ has mean 3 and variance 0.25. Squared: 10.25 ≥ 9.25. Absolute: 3.0025 ≥ 3.0000.

In Python:

import math
# E[c] for a bell curve of mean mu and spread s, for c(x) = x² and c(x) = |x|
def e_square(mu, s):
    return mu ** 2 + s ** 2
def e_abs(mu, s):
    tail = 0.5 * (1 + math.erf(-mu / s / math.sqrt(2)))
    return s * math.sqrt(2 / math.pi) * math.exp(-mu ** 2 / (2 * s ** 2)) + mu * (1 - 2 * tail)
# X_1 − X_0 ~ N(3, 1.25) for random pairs; Z_1 − Z_0 = 3 − 0.5 z_0 ~ N(3, 0.25) after rectifying
old, new = (3, math.sqrt(1.25)), (3, 0.5)
round(e_square(*new), 4), round(e_square(*old), 4)  # → (9.25, 10.25)
round(e_abs(*new), 4), round(e_abs(*old), 4)  # → (3.0, 3.0025)

Why it matters today

Cheaper couplings mean shorter journeys, and deterministic ones mean each noise sample has one definite image: a usable latent space for editing, which the paper uses in §5.2.

3.3 The straightening effect · original

“Then, we prove that recursive rectification straightens the coupling and its related flow with a O(1/k) rate, where k is the number of rectification steps.”Liu, Gong and Liu (2022), §3.3

Everyday picture

A coupling is straight when rectifying it changes nothing: its straight lines never cross, so the average arrow at every spot is the one line through it, and the flow simply follows the lines. Each round of reflow removes crossings, and every bit of crossing it removes is paid for from a fixed budget, the starting cost. A budget that only goes down cannot keep paying forever, so the crossings, and the bending, must fade.

Tiny example

In the toy, one rectification lowers the squared cost from 10.25 to 9.25, a drop of exactly 1. That drop splits into two parts: how much the new flow bends, S = 1 − π/4 = 0.2146, and how much the old straight lines crossed, V = π/4 = 0.7854.

In words: “V measures how much the straight lines disagree with their own average where they meet, which is zero exactly when no two lines cross; and one round of rectification lowers the squared cost by the new flow's bending plus the old lines' crossing.” Summing this over rounds gives Theorem 3.7's bound.

With the numbers: V integrates 1.25 − c²/q; S integrates (0.5 + c/√q)². They come to 0.7854 and 0.2146, and add to 1.0 = 10.25 − 9.25.

In Python:

import math
def q(t):
    return (1 - t) ** 2 + 0.25 * t * t
def c(t):
    return 0.25 * t - (1 - t)
n = 100_000
ts = [(i + 0.5) / n for i in range(n)]
# V: variance of X_1 − X_0 (1.25) minus the part the average arrow explains, c²/q
V = sum(1.25 - c(t) ** 2 / q(t) for t in ts) / n
# S: the flow's velocity c/√q · z_0 + 3 against its overall direction 3 − 0.5 z_0
S = sum((0.5 + c(t) / math.sqrt(q(t))) ** 2 for t in ts) / n
round(V, 4), round(S, 4), round(V + S, 4)  # → (0.7854, 0.2146, 1.0)
round(math.pi / 4, 4)  # → 0.7854

A note on the source: equation 13 in the paper prints the cost drop without the squares, as E‖X₁ − X₀‖ − E‖Z₁ − Z₀‖; the sentence before it (take c(x) = ‖x‖²) and the equation after it both have them, and only the squared version is true. The toy confirms it: the plain-distance drop is 0.0025, nowhere near 1.

Reading it: time runs left to right; height is position. Nine noise samples, evenly spread from −2 to 2, start on the left; the data crowd is centred at 3 with spread 0.5. Random pairs: each start is joined by a straight line to a random data point; the lines criss-cross. 1-rectified flow: the exact flow from the formula above. The paths never cross; they first squeeze together (the crowd narrows from spread 1 toward 0.45 around t = 0.8) and then fan out again to spread 0.5, so they bend. The order of the starts is kept: the lowest noise goes to the lowest data. 2-rectified flow: the same starts and the same ends, joined straight, because in one dimension the ordered pairing's lines never cross: one reflow makes the flow exactly straight.

Why it matters today

The straightening theorem is why reflow works in practice: the paper's CIFAR-10 flows go from recognisable-but-blurry images at one step to clear ones after a single reflow (§5.2).

3.4 Straight vs. optimal couplings · original

“Hence, it is “easier” to find a straight coupling than a c-optimal couplings.”Liu, Gong and Liu (2022), §3.4

Everyday picture

Straight and optimal are different goals. Optimal means the cheapest pairing for one chosen cost. Straight only means no two lines cross. Every optimal pairing is straight (if two lines crossed, swapping partners would be cheaper), but not every straight pairing is optimal, except on a line.

Tiny example

On a line (one dimension), a pairing is straight exactly when it keeps the order: the lowest start goes to the lowest destination, and so on. That ordered pairing is the cheapest for every convex cost at once. The toy's rectified pairing, z₀ → 3 + 0.5z₀, keeps the order, so it is both straight and optimal: 9.25 is the least squared cost any pairing of these two crowds can reach. In two or more dimensions, rectification finds a straight pairing, which need not be the cheapest; the paper points to follow-up work that restricts the velocity to the gradient of a function to reach the squared-cost optimum.

Why it matters today

For fast sampling, any straight pairing is as good as any other: all of them can be followed exactly in one step. The paper argues that chasing exact optimal transport is therefore unnecessary for generation, and also shows that an earlier conjecture, that DDIM's pairing is optimal for the squared cost, was already known to be false.

3.5 Denoising diffusion models and probability flow ODEs · original

“We prove that the probability flow ODEs (PF-ODEs) of [73] can be viewed as nonlinear rectified flows in (6) with X_t = α_tX₁ + β_tξ.”Liu, Gong and Liu (2022), §3.5

Everyday picture

A diffusion model's ODE is trained with a target written in the language of noise processes: a shrink rate η_t and a noise strength σ_t. Proposition 3.11 translates that target into road language, and finds it is exactly the velocity of the road X_t = α_tX₁ + β_tξ: the diffusion ODE was a rectified flow all along.

Tiny example

On the VP road at t = 0.5, with data X₁ = 1 and noise ξ = 0.5: the road's own velocity is α̇X₁ + β̇ξ = 1.4129 − 0.2070 = 1.2059. The diffusion ODE's training target, built from η_0.5 = −5.025 and σ_0.5² = 10.05, is also 1.2059. (That σ² is the score SDE paper's β(0.5), read in reversed time.)

In words: “the probability flow's training target, a shrink term plus a noise term, equals the road's velocity once the shrink rate and noise strength are written in terms of the road's two shares.”

With the numbers: α = 0.2812, α̇ = 1.4129, β = 0.9597, β̇ = −0.4140; η = −5.025, σ² = 10.05; X_t = 0.2812 + 0.4798 = 0.7610; Ỹ = 5.025 × 0.7610 − (10.05/1.9193) × 0.5 = 1.2059, the same as α̇ × 1 + β̇ × 0.5.

In Python:

import math
a, b, t = 19.9, 0.1, 0.5
u = 1 - t
# the VP road: α, β and their rates of change
alpha = math.exp(-0.25 * a * u * u - 0.5 * b * u)
alpha_dot = alpha * (0.5 * a * u + 0.5 * b)
beta = math.sqrt(1 - alpha ** 2)
beta_dot = -alpha * alpha_dot / beta
# η and σ² written in terms of the road
eta = -alpha_dot / alpha
sigma2 = 2 * beta ** 2 * (alpha_dot / alpha - beta_dot / beta)
round(eta, 3), round(sigma2, 2)  # → (-5.025, 10.05)
x1, xi = 1.0, 0.5
x_t = alpha * x1 + beta * xi
# the diffusion ODE's target and the road's velocity
y_tilde = -eta * x_t - sigma2 / (2 * beta) * xi
round(y_tilde, 4), round(alpha_dot * x1 + beta_dot * xi, 4)  # → (1.2059, 1.2059)

Two notes on the source. The proof's second line writes the shrink rate with a dot (η̇_t) where the rest of the section, and the computation above, use η_t itself; and the nonlinear rectified flow's definition in §2.3 prints E[Ẋ_t | X_t = t] where the condition must be X_t = z. Both read as typos; neither changes a result.

Why it matters today

Seen this way, the difference between a diffusion ODE and a rectified flow is only the road: curved and unevenly paced, or straight and steady. That is the whole case for training on straight roads, and it is why the DDIM companion's sampler and the lesson's euler_step are the same kind of step on different roads.

5.1 Toy examples · original

Everyday picture

To see the theory without a neural network's quirks, the toys use the kernel estimator of equation 5, averaged over the 100 nearest training lines with bandwidth 1 (the paper finds results insensitive to both). A small network also works (two hidden layers of 64), fitting the averaged arrows a little less crisply; the paper finds that stronger L2 regularization, which makes the network smoother, helps straighten the flow further.

How straightness and cost were measured

Straightness is equation 3 estimated from simulated paths, as in Figure 2 above. For cost, the paper compares each coupling with the best possible pairing of the same end points, found by solving the discrete optimal transport problem, and notes a trap: in high dimensions that relative cost is near zero even for a random network, which is how an earlier study came to believe DDIM was optimal.

Why it matters today

Low-dimensional pictures like these are still the fastest way to see what a sampler does to paths; the redrawn figures on this page are built the same way.

5.2 Unconditioned image generation · original

“In particular, the distilled 2-rectified flow achieves an FID of 4.85, beating the best known one-step generative model with U-net architecture, 8.91 (TDPM, Table 1 (b)).”Liu, Gong and Liu (2022), §5.2

Everyday picture

On CIFAR-10 the paper uses the score SDE paper's DDPM++ U-Net and code, so the comparison with the VP and sub-VP ODEs is like for like: same network, different road. Quality is measured by FID and Inception score, and diversity by recall.

Read the one-step rows first: without reflow, one step is useless (FID 378, the blurred average image of Figure 5's first panel); after one reflow it is 12.21 without any distillation, and 4.85 with it. With a full solver, the 1-rectified flow is the best ODE on the page (FID 2.58 with 127 calls) and close to the 2,000-step SDE. The paper also reports the trade-off honestly: reflow helps a lot below about 80 steps but slightly hurts with many steps, because each round adds some estimation error.

Seeing straightness in pictures

Figure 10 of the paper extrapolates from the middle of a path to its end with one straight jump. On a straight path that guess never changes as you move along it.

In words: “from where the path is now, jump straight to the end, at the current velocity, for all the time that is left.”

With the numbers: in the one-dimensional toy, the 1-rectified flow from z₀ = 1 is at 2.0590 at t = 0.5 with velocity 2.3292, so the guess is 3.2236, though the path really ends at 3.5. On the 2-rectified flow the path is the straight line from 1 to 3.5, and the guess is 3.5 at every t.

In Python:

import math
def q(t):
    return (1 - t) ** 2 + 0.25 * t * t
def c(t):
    return 0.25 * t - (1 - t)
t, z0 = 0.5, 1.0
# the 1-rectified flow: z_t = 3t + z_0 √q(t), velocity 3 + (c/q)(z_t − 3t)
z_t = 3 * t + z0 * math.sqrt(q(t))
v = 3 + c(t) / q(t) * (z_t - 3 * t)
round(z_t, 4), round(v, 4)  # → (2.059, 2.3292)
round(z_t + (1 - t) * v, 4), 3 + 0.5 * z0  # → (3.2236, 3.5)
# the 2-rectified flow is the straight line from z_0 to 3.5
z2 = (1 - t) * z0 + t * 3.5
z2 + (1 - t) * (3.5 - z0)  # → 3.5

On cat faces, the 2-rectified flow's extrapolated image hardly changes along the path, and even the 1-rectified flow's is recognisable by t ≈ 0.1, where the sub-VP ODE's needs t ≈ 0.6. The paper also shows 256 × 256 images (bedrooms, churches, faces, cats) from the 1-rectified flow, and simple edits: run the flow backwards from a stitched-together image to its noise, nudge the noise toward more probable values, and run forwards again.

Why it matters today

This table is the evidence behind “reflow, then distill” as a route to few-step generation, and the backwards-then-forwards edit shows the practical value of a deterministic, low-cost coupling.

5.3 Image-to-image translation · original

Everyday picture

Set π₀ to photos of cats and π₁ to photos of wild animals, with no pairs, and the same algorithm learns to turn one into the other. There is no need for the cycle-consistency penalty of CycleGAN, because an ODE run backwards returns exactly where it started. But the goal here is not an exact match to π₁: it is to change the style while keeping the subject, so the loss only counts errors that change the style.

Tiny example

A two-number “image” whose first number is its style and second its content. A pair moves from (0, 0) to (2, 1); the network guesses the velocity (1.5, 3.0). The plain loss counts both errors: 0.5² + 2² = 4.25. With the style feature h(x) = first number, whose slope is (1, 0), only the style error counts: 0.5² = 0.25.

In words: “the rectified flow loss, but measured only in the directions that change the style features, weighted by how much they change them.” The paper takes h from a classifier trained to tell the two domains apart.

With the numbers: the error is (0.5, −2.0); plain loss 4.25; style-weighted loss (1 × 0.5 + 0 × (−2.0))² = 0.25.

In Python:

x0, x1, v = (0.0, 0.0), (2.0, 1.0), (1.5, 3.0)
# ∇h: the style feature is the first number, so its slope is (1, 0)
grad_h = (1.0, 0.0)
err = [x1[i] - x0[i] - v[i] for i in range(2)]
err  # → [0.5, -2.0]
# the plain loss, and the loss seen through ∇h
sum(e * e for e in err), sum(g * e for g, e in zip(grad_h, err)) ** 2  # → (4.25, 0.25)

Why it matters today

Generation and translation become the same algorithm with a different π₀, which is the paper's unifying claim made concrete. The results (cats to wild animals, faces to painted portraits and back) are shown as images, one step being enough after one reflow.

5.4 Domain adaptation · original

Everyday picture

A classifier trained on photos stumbles on sketches. Instead of retraining it, move the sketches' features into the photos' feature distribution with a rectified flow, and classify the moved features. Here π₀ and π₁ are the test and training features from a pretrained model's last hidden layer, and the flow is run with 100 Euler steps.

The paper's own wording is careful: “better or on par with” the best baseline. On DomainNet the difference is inside the error bars.

Why it matters today

It shows that the recipe is not tied to images or to noise: any two piles of vectors can be transported, with no pairs and no adversary.