The 6000 training threes are 784-dimensional vectors of grey levels (0 to 255). Their mean and the eigenvectors of their covariance matrix are images too: the eigen-digits. Keeping the first M of them, each digit is stored as M numbers ai = uiT(x − x̄) and rebuilt as x̄ + Σi≤M aiui (16.20). Move M and watch the reconstruction sharpen, the residual shrink, and the error follow the sum of the discarded eigenvalues, which is its average over the training set (16.18).
Data: the first 6000 MNIST training threes; PCA as in Figures 16.3 to 16.5 of the book. The eigenvectors (first 250 of 784) are stored as 16-bit integers, which changes reconstructions by less than 0.01 grey levels. Images 2 to 9 are test-set threes, 10 to 13 are a 7, a 2, an 8 and a 5. The probabilistic PCA model uses the maximum likelihood solution (16.46)–(16.47) with the same M.
code/ch16_continuous_latent_variables.py (pca, reconstruct, export_eigendigits_json, Figures 16.3 to 16.5); the numbers on this page equal the Python ones.