Interactive demo · Foundations of Deep Learning · Chapter 19

A linear autoencoder against PCA

The network y = W2W1x with M hidden units is trained by gradient descent on the sum-of-squares error (19.1) for 500 centred 14 × 14 MNIST digits, here in your browser. Its error falls to exactly the PCA error and the span of its weight vectors becomes the principal subspace (the principal angles go to zero), yet the weight vectors themselves stay mixtures of the eigenvectors: any invertible C gives the same network through W2C and C−1W1. A small weight decay removes most of that freedom, and the left singular vectors of W2 then are the eigenvectors.

Hidden units M

Learning rate η

Weight decay

Initial weight scale

Gradient steps

Run

Try these

Data: the digits are centred (the optimal biases do exactly this), so there are no biases. Error E/N = (1/2N)Σn‖W2W1xn − xn‖² (+ decay/2 times the squared weights); the PCA optimum is half the sum of the discarded eigenvalues. Updates: full-batch gradient descent, W ← W − η∇E/N. Weights start as (scale) × N(0, 1).

In the lecture notes