Each binarised 28 × 28 digit is a vector of 784 bits. A single multivariate Bernoulli distribution treats the bits as independent and can only learn the average image; a mixture of K of them (15.37) has a hidden label for each image, and EM learns, without seeing any digit labels, K prototype images μk and their weights πk. The E step (15.46) computes each component's responsibility for each image; the M step (15.49)–(15.50) sets μk to the responsibility-weighted average image. Below the prototypes, new images are generated by ancestral sampling: choose a component with probabilities π, then switch each pixel on with probability μki.
Data: the first 200 training images of each chosen digit (MNIST), binarised at 0.5. πk starts at 1/K; μki = 0.25 + 0.5u with u uniform (seeded). "Book" then divides each μk by its sum, as printed under Figure 15.12; "identical" gives all components the same μ. Logs of μ are taken after clipping to [10−10, 1 − 10−10].
code/ch15_discrete_latent_variables.py (bern_init_js, bern_em; Figure 15.12 is preset 1 of this page). With the same digits, K, start and seed, the log likelihoods shown here equal those of bern_em_js.