Interactive demo · Foundations of Deep Learning · Chapter 20

The forward process and its closed form

A diffusion model destroys data by adding a little Gaussian noise at every step, zt = √(1 − βt) zt−1 + √βt εt (20.3). Because each step is linear and Gaussian, t steps collapse into one: zt = √αt x + √(1 − αt) ε with αt = ∏(1 − βτ) (20.6)–(20.8). Blue points are run through the chain one step at a time, orange points jump straight to step t; the two clouds have the same distribution. The lower panel asks when the reverse step can be modelled by a Gaussian: it computes q(zt−1 | zt) exactly for a one-dimensional mixture (Figures 20.3 and 20.4).

Schedule

Number of steps T

β1 and βT (log scale)

Step t

Show

Run

Reverse step (lower panel): β and zt

Try these

300 data points from the spiral (each arm a chain of small Gaussians). The circles are the 2-standard-deviation circles of q(zt | x) = N(√αtx, (1 − αt)I) for three data points. Lower panel: q(zt−1) is a fixed three-component mixture (red), the reverse density (blue) is proportional to q(zt | zt−1) q(zt−1), and the dashed curve is the Gaussian of Exercise 20.7, mean zt + β(zt/2 + s), variance β + β²(1 + h), with s and h the first two derivatives of ln q(zt−1) at zt.

In the lecture notes