Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 10

Gaussian approximations: Laplace, KL(q‖p) and KL(p‖q)

Three ways to replace a distribution p by a simpler q. The Laplace approximation fits a Gaussian to the curvature at the mode. Variational inference minimises KL(q‖p), which punishes q for putting mass where p has little: q is compact and seeks a mode. Moment matching minimises KL(p‖q), which punishes q for missing mass of p: q is broad and covers everything (this is the step used by expectation propagation). See it on a skewed density (A), a correlated Gaussian with a factorised q (B) and a density with two modes (C).

Example

Scale

p(z) ∝ exp(−z²/2) σ(az + b): slope a

Shift b

Correlation ρ of p

Factorised q minimises

Coordinate ascent (10.13), (10.15)

Separation of the two modes

Gaussian q minimises

Start of the optimisation (or click the plot)

Scoreboard

Try these

Expectations under q use a 61-point (61 × 61 in two dimensions) Gaussian grid rule, the same rule as the Python code; KL(q‖p) is minimised by quasi-Newton steps (BFGS). In example A, moment matching is exactly what one EP update does when p has a single non-Gaussian factor.

In the lecture notes

Chapter 10 notes: Section 10.1 (the decomposition ln p(X) = L(q) + KL(q‖p) and the Gaussian variational fit of Figure 10.1), Section 10.1.2 (factorised approximations, (10.12) to (10.17), Figures 10.2 and 10.3, and the alpha family of divergences) and Section 10.7 (expectation propagation: minimising KL(p‖q) by moment matching, Figure 10.14).
The Laplace approximation itself is Section 4.4 of the book (Figure 4.14 shows the same skewed density).
Python code: code/ch10_variational_inference.py (functions laplace_1d, gauss_kl_qp_1d, moments_1d, mean_field_gaussian, factorised_kl_pq, gauss_kl_qp_2d, moment_match_mixture, figures fig10_1 to fig10_3).