The network y = W2W1x with M hidden units is trained by gradient descent on the sum-of-squares error (19.1) for 500 centred 14 × 14 MNIST digits, here in your browser. Its error falls to exactly the PCA error and the span of its weight vectors becomes the principal subspace (the principal angles go to zero), yet the weight vectors themselves stay mixtures of the eigenvectors: any invertible C gives the same network through W2C and C−1W1. A small weight decay removes most of that freedom, and the left singular vectors of W2 then are the eigenvectors.
Data: the digits are centred (the optimal biases do exactly this), so there are no biases. Error E/N = (1/2N)Σn‖W2W1xn − xn‖² (+ decay/2 times the squared weights); the PCA optimum is half the sum of the discarded eigenvalues. Updates: full-batch gradient descent, W ← W − η∇E/N. Weights start as (scale) × N(0, 1).
code/ch19_autoencoders.py (linear_ae_train); the digits are in data/ch19-mnist14.json and, with the same seed, the errors here equal linear_ae_train to rounding error.