Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 10

Expectation propagation for a logistic model

Twelve observations yn = ±1 with inputs xn have likelihood factors fn(θ) = σ(ynxnθ) and the prior is θ ~ N(0, 10). Expectation propagation replaces each factor by a Gaussian "site". To refresh site n it divides the site out of q (the cavity), multiplies the exact factor in (the tilted distribution), matches the mean and variance of the result (the projection), and keeps the ratio of projection to cavity as the new site. Left: the exact posterior, the current Gaussian and the variational (reverse KL) Gaussian. Right: the cavity, factor, tilted distribution and projection of the chosen site, as in Figure 10.15.

Method

Sweeps over the 12 sites

Damping δ (1 = none)

Power α (1 = plain EP)

Site shown on the right

Try these

All integrals use a grid of 3201 points on [−8, 8]. EP visits the sites in order 1 to 12 in every sweep; ADF makes one pass and keeps no sites to revise. Damping mixes the old and new site natural parameters, (1 − δ) old + δ new; power EP divides out α of the site, projects the cavity times fnα, and raises the ratio to 1/α. The variational Gaussian maximises the ELBO (computed once, by quadrature).

In the lecture notes