Interactive demo · Foundations of Deep Learning · Chapter 16

Eigen-digits: compressing handwritten threes with PCA

The 6000 training threes are 784-dimensional vectors of grey levels (0 to 255). Their mean and the eigenvectors of their covariance matrix are images too: the eigen-digits. Keeping the first M of them, each digit is stored as M numbers ai = uiT(x − x̄) and rebuilt as x̄ + Σi≤M aiui (16.20). Move M and watch the reconstruction sharpen, the residual shrink, and the error follow the sum of the discarded eigenvalues, which is its average over the training set (16.18).

Digit (13 images)

Number of components M

Play

Loading the eigenvectors…

Try these

Data: the first 6000 MNIST training threes; PCA as in Figures 16.3 to 16.5 of the book. The eigenvectors (first 250 of 784) are stored as 16-bit integers, which changes reconstructions by less than 0.01 grey levels. Images 2 to 9 are test-set threes, 10 to 13 are a 7, a 2, an 8 and a 5. The probabilistic PCA model uses the maximum likelihood solution (16.46)–(16.47) with the same M.

In the lecture notes