Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 10

The ELBO, the IWAE bound and the EUBO

The model is z ~ N(0, 1), x | z ~ N(z2, σx2): for x > 0 the posterior has two symmetric modes at z ≈ ±√x. Pick a Gaussian q(z) = N(m, s2) and compare three numbers with the exact log evidence log p(x): the ELBO (below it by KL(q ∥ p)), the IWAE bound with S importance samples (between the two, rising with S), and the EUBO (above it by KL(p ∥ q)). A narrow q on one mode has a good ELBO but a huge EUBO, and its IWAE bound stalls about log 2 below log p(x) because its samples never reach the other mode.

Observation x and noise σx

Variational mean m and standard deviation s

Importance samples S

Fit q

Try these

The ELBO, the EUBO and log p(x) are computed by quadrature on a grid of 6001 points on [−6, 6]. The IWAE bound E[log (1/S) Σ ws] is estimated by averaging 500 independent draws of the S samples (fixed seed); the readout gives its Monte Carlo standard error. The curve on the right uses 200 draws per point. "ELBO fit" maximises the ELBO numerically from the right-hand mode; "moment match" sets m and s to the posterior mean and standard deviation, which minimises the EUBO.

In the lecture notes