With \(D=2\) and \(L=1\) the model is \(\mathbf x=z\mathbf w+\boldsymbol\epsilon\). The E step holds the rod \(\mathbf w\) fixed and slides each point's attachment \(z_n\) along it; the M step holds the attachments fixed and rotates and stretches the rod. With zero noise this is a power iteration on the span of \(\mathbf S\mathbf w\), so the angle to the first principal axis shrinks by the factor \(\lambda_2/\lambda_1\) per sweep. With noise \(\sigma^2\) learned too (probabilistic PCA) the iteration converges to the closed-form maximum likelihood estimate, \(\sigma^2=\lambda_2\) and \(\|\mathbf w\|^2=\lambda_1-\lambda_2\).
The rod starts with length 1.2. In the zero-noise model the objective is the mean squared reconstruction error; in probabilistic PCA it is the log likelihood per data point, with \(\sigma^2\) started at \(\mathrm{tr}(\mathbf S)/D\). Height and weight: the Davis (1990) data (R package carData), standardised, a stand-in for the book's data set.
code/ch20_dimensionality_reduction.py (em_pca, ppca_em_1d, ppca_mle).