Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 25

Discrete diffusion: the transition kernels of D3PM

A discrete diffusion model corrupts a categorical variable with a Markov chain, q(xt | xt−1) = Cat(xt−1Qt). The t-step marginal is a row of the product Q̄t = Q1···Qt, and Bayes' rule gives the posterior q(xt−1 | xt, x0) that a reverse model is trained to match. Choose the kernel and watch the data distribution (the horizontal position of a quantised Swiss roll, 20 levels) flow to the stationary distribution.

Kernel

Number of steps T

Time t

Posterior: clean state x0

Posterior: noisy state xt

Try these

Schedules (Section 25.7.4): the uniform kernel uses the cosine schedule of Nichol and Dhariwal; the absorbing kernel uses βt = 1/(T − t + 1); the discretised Gaussian kernel has βt rising linearly from 0.0002 to 0.05. The absorbing chain has a 21st state, MASK. Distances are total variation; the prior loss is the average of KL(q(xT | x0) || stationary) over the data, in nats.

In the lecture notes