For any distribution q(Z) over the latent labels, ln p(X|θ) = ℒ(q, θ) + KL(q‖p) (15.52): the evidence lower bound ℒ plus the gap KL between q and the posterior. EM is coordinate ascent on ℒ: the E step chooses q (the gap closes), the M step chooses θ (the bound rises, and the log likelihood rises at least as much). Step through EM on a one-dimensional Gaussian mixture and watch the three numbers; the lower right panel shows ln p and the bounds along the straight line from the old to the new parameters, the computed version of the book's Figure 15.16.
The run starts from θ0 and q0(znk = 1) = 1/K. Exact E step: q = posterior (15.12). Partial: q ← (1 − λ)q + λ × posterior. Exact M step: (15.16), (15.18), (15.21). ECM: one block per M step (means, then variances, then weights). Gradient: one step of size η/N on ℒ in the coordinates (μk, ln σk2, ln πk). The prior is an inverse-gamma IG(2, 0.1) on each variance (MAP-EM, (15.59)); with it, every number shown includes ln p(θ) (up to a constant). Variances are kept above 10−10.
code/ch15_discrete_latent_variables.py (em1d_data, em1d_run, em1d_slice; Figures 15.13 to 15.16 are preset 1 of this page). With the same settings the values of ln p and ℒ shown here equal those of em1d_run.