Interactive demo · Bayesian and Probabilistic Machine Learning · Advanced topics

Denoising diffusion: linear against cosine schedules, and a reverse sampler

The target is a three-component Gaussian mixture (means (−1.5, 0), (1.5, 0), (0, 1.5), standard deviation 0.25, weights 0.4, 0.4, 0.2). The forward process gives xt = √ᾱt x0 + √(1 − ᾱt) ε, where ᾱt is the product of 1 − βs up to step t. Because the noised marginal of a Gaussian mixture is again a Gaussian mixture, its score is exact, and the reverse process is the ancestral DDPM sampler driven by that score. The two schedules differ only in how fast ᾱt falls, and that decides how close the final step is to pure noise.

Schedule

Steps T

Forward time t

Reverse sampler

Try these

Linear: βt rises evenly from 10−4 to 0.02 over the T steps. Cosine: ᾱt = cos2((t/T + 0.008)/1.008 · π/2), divided by its value at t = 0, and βt = 1 − ᾱt/ᾱt−1. Early jump: βt = 0.2 for t < 10, then the linear schedule. For the linear schedule with T = 200, ᾱT ends near 0.13, not at 0: the final noisy sample still carries signal, so starting the reverse sampler from white noise N(0, I) is an approximation there. The cosine schedule ends at ᾱT = 0 exactly. The reverse step is xt−1 = (xt − βt/√(1 − ᾱt) ε̂)/√αt + σt z with ε̂ = −√(1 − ᾱt) ∇ log pt(xt) and σt2 = βt; the noise is added at every step, as written. The seeds are fixed, so a class sees the same pictures.

In the lecture notes