Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 12

Bayesian PCA, automatic relevance determination and factor analysis

Ten-dimensional data with standard deviation 1 along three directions and 0.5 along the other seven are fitted by probabilistic PCA with M = 9 latent dimensions, using EM. Each square of the Hinton diagram is one element of W (white positive, black negative, area proportional to size). Maximum likelihood spreads the three real directions over all nine columns. Bayesian PCA puts a prior N(0, α_i⁻¹I) on each column and re-estimates α_i = D/‖w_i‖² after every EM step: columns the data do not need are shrunk, their α_i grows, and they switch off (Bishop 2006, Section 12.2.3). The last view compares factor analysis with PPCA (Section 12.2.4).

Method

Data points (N)

Data set

EM

Rescale variable x₁ by

Scoreboard

Try these

EM uses the sample covariance S directly: the E step (12.54), (12.55) and the M step (12.56), (12.57) only need SW, so each step costs O(D²M). Bayesian PCA replaces (12.56) by (12.63). Columns with α_i above 10¹² are treated as switched off (α is capped there).

In the lecture notes

Chapter 12 notes: Section 12.2.3 (Bayesian PCA: the ARD prior (12.60), the re-estimation (12.62), the modified M step (12.63) and Figure 12.14), Section 12.2.2 (EM for PPCA, (12.54) to (12.57)) and Section 12.2.4 (factor analysis, (12.64) to (12.70), and its behaviour under rescaling).
Maximum likelihood PPCA in closed form, (12.45) and (12.46): Section 12.2.1; a two-dimensional picture is in the PCA demo.
Python code: code/ch12_continuous_latent.py (functions ppca_em, bayesian_pca, factor_analysis_em, bpca_data, fig12_14, fig12_15); the 300 points and the starting W of the first data set are those of our Figure 12.14.