Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 25

Reverse diffusion samplers with an exact score

Every diffusion sampler integrates the reverse process from noise at t = 1 to data at t = 0 using the score ∇ log qt. Here the data distribution is a Gaussian mixture, so every noised marginal qt is a mixture too and its score is exact: the network is perfect, and what you see is only the error of the sampler. Change the sampler and the number of steps, and compare the log-density of the samples with that of exact draws.

Data distribution

Sampler

Number of steps N

Samples

Try these

VP SDE with β(t) rising linearly from 0.1 to 20, integrated from t = 1 to t = 0.001 on an evenly spaced grid. Ancestral sampling uses the posterior variance β̃; DDIM uses η = 0. Langevin runs N steps of size 0.002 at t = 0.001 from N(0, I) starts; annealed Langevin spreads the N steps over 10 noise levels (step 0.002 · σt2/σmin2, at most 0.05). The 95% highest-density region (HDR) is {x: log q0(x) ≥ its 5% quantile under q0}; an exact sampler puts 95% of its draws there. Fixed seeds: the same settings always give the same picture.

In the lecture notes