PCA is fitted to 600 images of handwritten digits (8 × 8 pixels), and 400 further images are held out. Reconstruct a digit from its first \(L\) principal components and watch the error. The error on the training images falls with \(L\) by construction; the error on the held-out images falls too, because PCA is not a probabilistic model that can overfit, so there is no U shape. The U shape appears only when we measure the error against the clean image after adding pixel noise: the late components reproduce noise, not digit. The scree plot and the profile likelihood of (20.31) give two ways of picking \(L\) without the clean image.
The PCA basis is always fitted to the (noisy) training images with the divisor \(N\). RMSE is the square root of the mean squared pixel error. The noise is a fixed table of standard normal numbers multiplied by \(\sigma\), so the curves are reproducible. Data: the 1797 UCI digits shipped with scikit-learn (1000 used).
code/ch20_dimensionality_reduction.py (pca_eig, recon_error_curve, profile_loglik).