Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 2

Alpha divergence: mode seeking or mass covering

The target p (pink) is a mixture of two Gaussians. For each α the page finds the Gaussian q (blue) that minimises Minka's alpha divergence Dα(p‖q) = (1 − ∫ pαq1−α) / (α(1 − α)). The limit α → 1 is KL(p‖q), which matches the mean and variance of p and covers both modes; α → 0 is KL(q‖p), which settles on one mode; α = 1/2 is a symmetric (Hellinger) compromise. Move α and the separation of the modes.

Divergence parameter α

Gap between the modes

Weight of the left mode

Try these

p = w N(μ1, 0.8²) + (1 − w) N(μ2, 1²) with mean 0.2; the defaults reproduce Figure 2.20. The optimum is found by Nelder–Mead from four starting points on a fine grid, so for α ≤ 0 the result can be a local minimum.

In the lecture notes