Interactive demo · Foundations of Deep Learning · Chapter 20

Reverse sampling with a trained denoiser

A network g(z, w, t) with three hidden layers of 64 units was trained in Python with Algorithm 20.1 to predict the noise in zt = √αtx + √(1 − αt)ε for points x of a three-arm spiral; its weights run here, forward only. Generation starts from white noise and removes noise step by step. Algorithm 20.2 takes a step towards the predicted clean point and adds fresh noise; the DDIM update with η = 0 adds none, so the map from zT to x is deterministic and survives much larger steps. Switch to the exact predictor E[ε | zt], which the network approximates, to separate the network's error from the sampler's.

Noise predictor

Sampler

Stochasticity η (0 = deterministic)

Number of steps S

Points

Run

Loading the network ...

Try these

Steps run at S times spread evenly over 1..200. Algorithm 20.2 on a sub-sequence uses β = 1 − αt/αt′ between consecutive times. DDIM: x̂ = (z − √(1 − αt)g)/√αt, then z′ = √αt′x̂ + √(1 − αt′ − s²)g + sε, with s = η√((1 − αt′)/(1 − αt))√(1 − αt/αt′); η = 1 is the stochastic sampler with the variance (20.17). Quality is the mean of ln p(x) under the true spiral density: real data score real data score −1.51minus;1.53.

In the lecture notes