Interactive demo · Probabilistic Machine Learning · Chapter 6

Forward versus reverse KL: mode covering and mode seeking

The target p is a mixture of two Gaussians with equal spread; the approximation q is one Gaussian. Minimising the forward KL divergence D(p ‖ q) punishes q for putting little mass where p has some, so q spreads over both modes (it matches the mean and variance of p). Minimising the reverse KL divergence D(q ‖ p) punishes q for putting mass where p has little, so q locks onto one mode; there are two such local minima. The right panel shows each divergence as a function of the mean of q, with the standard deviation chosen best for each mean.

The Gaussian q is

Separation of the modes m (means ±m)

Weight of the left component

Spread of each component s

Manual q: mean and sd

Try these

Both divergences are computed by summing over a grid of 1201 points on [−12, 12] (nats). The forward optimum is the moment-matched Gaussian; each reverse optimum is found by Nelder–Mead started at one of the two modes.

In the lecture notes