Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 10

Mean-field variational Bayes for the mean and precision of a Gaussian

The exact posterior of the mean μ and precision τ of a Gaussian is a Normal-Gamma distribution (green contours): μ and τ are dependent, since a larger τ makes the mean better determined. Variational Bayes replaces it by a product q(μ)q(τ) (Bishop 2006, Section 10.1.3) and improves one factor at a time with (10.26)–(10.30); each update raises the lower bound L(q), which approaches ln p(D) from below. Press the update buttons in turn, as in Figure 10.4, then compare the marginals.

Data size (N)

Update

Scoreboard

Try these

Prior (10.22), (10.23): μ | τ ~ N(0, 1/τ) and τ ~ Gam(1, 1), i.e. μ₀ = 0, λ₀ = 1, a₀ = b₀ = 1. The first factorised guess is q(μ) = N(0.55, 0.17²), q(τ) = Gam(25, 25/1.3), placed as in Figure 10.4(a). Contours are drawn at 10, 30, 50, 70 and 90% of each density's maximum. (10.29) uses a_N = a₀ + (N + 1)/2, as corrected in the errata.

In the lecture notes

Chapter 10 notes: Section 10.1.3 (the example: the updates (10.25) to (10.30), Figure 10.4 and the fixed point (10.33)) and Section 10.1.1 (the general mean-field solution (10.9) that produces these updates).
Python code: code/ch10_variational_inference.py (functions vb_update_mu, vb_update_tau, vb_gaussian_bound, normal_gamma_posterior, fig10_4); the data, prior and first guess are the same as in the reproduced Figure 10.4.