Interactive demo · Foundations of Deep Learning · Chapter 19

The ELBO gap: a Gaussian q against a bimodal posterior

The model has one latent variable, z ~ N(0, 1), and a decoder that traces a circle: p(x|z) = N(x | (sin z, cos z), σ²I). Click the left panel to choose a data point x*. Its exact posterior p(z|x*) is computed on a grid; a Gaussian q(z) = N(m, s²) is then fitted by stochastic gradient ascent on the ELBO using the reparameterisation trick z = m + sε. Whatever q does, ln p(x*) = ELBO + KL(q‖p(z|x*)) (19.6): when the posterior has two modes, a Gaussian must choose one, and the gap cannot close.

Decoder noise σ

Starting mean of q

Samples per step L

Step size

Gradient steps

Run

Try these

Updates (per step, with f(z) = ln p(x*|z) + ln p(z) and εl ~ N(0, 1)): m ← m + η meanl f′(zl), ln s ← ln s + η(meanl f′(zl) sεl + 1), the "+1" being the gradient of the entropy of q. ln p(x*) and the ELBO plotted are exact (quadrature). The best Gaussian is found by a grid search over (m, ln s).

In the lecture notes