Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 5

Diagonal Gaussian approximations of a correlated Gaussian

The target p is a zero-mean Gaussian with standard deviations σ1, σ2 = 1 and correlation ρ. The approximation q has a diagonal covariance diag(v1, v2). Minimising the forward KL D(p‖q) gives the marginal variances, vi = Σii (teal, too wide). Minimising the reverse KL D(q‖p) gives vi = 1/Λii = Σii(1 − ρ2) (gold, too narrow). The right panel is the plane of (log10 v1, log10 v2) coloured by the chosen KL; your own q (pink) is set by two sliders.

Target p

Your q (pink)

Colour the right panel by

Try these

KL divergences are in nats. The colour scale is log(1 + KL), so the minimum is the darkest point; it sits exactly at the teal marker for the forward KL and at the gold marker for the reverse KL.

In the lecture notes