Interactive demo · Probabilistic Machine Learning · Chapter 7

Standardising and whitening data: PCA and ZCA

Standardising gives every column mean 0 and variance 1 but leaves the correlation alone. Whitening goes further: a matrix \(\mathbf W\) with \(\mathbf W\boldsymbol\Sigma\mathbf W^{\mathsf T}=\mathbf I\) turns the centred data into data with identity covariance. With \(\boldsymbol\Sigma=\mathbf E\mathbf D\mathbf E^{\mathsf T}\), PCA whitening \(\mathbf W_{\rm pca}=\mathbf D^{-1/2}\mathbf E^{\mathsf T}\) rotates onto the principal axes and rescales; every \(\mathbf R\mathbf W_{\rm pca}\) with \(\mathbf R\) orthogonal also whitens. Turn the rotation and watch the mean squared displacement \(\frac1N\sum_n\|\mathbf W\mathbf x_n-\mathbf x_n\|^2\) (bottom panel): its minimum is ZCA whitening, \(\mathbf W_{\rm zca}=\boldsymbol\Sigma^{-1/2}\), the whitened data closest to the original.

Data

Correlation of the simulated data ρ

Transformation

Rotation of the whitened data φ (degrees; W = R(φ) Wpca)

Try these

Covariances use the divisor \(N\). The eigenvector signs are fixed so that \(\mathbf E\) is a rotation with a positive first entry; then ZCA is the rotation \(\mathbf R(\varphi^*)=\mathbf E\). Height and weight: the 88 men of the Davis (1990) study (R package carData), a stand-in for the book's data set.

In the lecture notes