Interactive demo · Foundations of Deep Learning · Chapter 15

A mixture of Bernoulli distributions on handwritten digits

Each binarised 28 × 28 digit is a vector of 784 bits. A single multivariate Bernoulli distribution treats the bits as independent and can only learn the average image; a mixture of K of them (15.37) has a hidden label for each image, and EM learns, without seeing any digit labels, K prototype images μk and their weights πk. The E step (15.46) computes each component's responsibility for each image; the M step (15.49)–(15.50) sets μk to the responsibility-weighted average image. Below the prototypes, new images are generated by ancestral sampling: choose a component with probabilities π, then switch each pixel on with probability μki.

Digits used

Components K

Starting values of μki

Run EM

Try these

Data: the first 200 training images of each chosen digit (MNIST), binarised at 0.5. πk starts at 1/K; μki = 0.25 + 0.5u with u uniform (seeded). "Book" then divides each μk by its sum, as printed under Figure 15.12; "identical" gives all components the same μ. Logs of μ are taken after clipping to [10−10, 1 − 10−10].

In the lecture notes