Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 7

Approximating a posterior: grid, Laplace, variational, MCMC

A coin shows N1 heads and N0 tails; with a uniform prior the exact posterior is Beta(N1+1, N0+1), so every approximation can be scored against the truth. The blue curve is exact. Pick a method (or all of them) and watch how it fails: the MAP point has no spread, a coarse grid is blocky, the Laplace approximation in θ is symmetric and leaks outside [0, 1], the same approximation in the logit is better, the variational Gaussian is mode seeking, and MCMC converges only if the proposal is sensible. The table gives the divergence KL(p ∥ q) of each approximation from the posterior and its estimate of the log evidence log p(D).

Method shown

Observed heads N1 and tails N0

Grid points K

MCMC: step size in the logit and samples

Try these

The prior is uniform on θ (Beta(1,1)). The variational family is a Gaussian on the logit α = log(θ/(1−θ)), fitted by gradient ascent on the ELBO with a fixed quadrature (a deterministic stand-in for the stochastic ADVI of the notes). KL is computed numerically on a fine grid; for the Laplace approximation in θ the Gaussian is truncated to [0, 1] and renormalised. MCMC is random-walk Metropolis on the logit from α = 0, with the first 20% of the samples discarded, and the seed is fixed.

In the lecture notes