Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 12

Kernel PCA

Kernel PCA does ordinary PCA in a feature space φ(x) that is never formed: everything is computed from the kernel k(x, x′) = exp(−‖x − x′‖²/c) through the centred Gram matrix K̃ (Bishop 2006, Section 12.3). Each panel colours the plane by the projection of φ(x) onto one principal direction in feature space, with contour lines where that projection is constant (dark: low, bright: high). Change the data, the kernel width c and the centring, and compare with linear PCA, whose components are only straight lines.

Data

Kernel width c (log scale)

Show

Component

Centre the Gram matrix (12.85)

Method

Scoreboard

Try these

The eigenvalues printed are those of the centred Gram matrix K̃, which are Nλ_i by (12.80), as in the book's Figure 12.17. Projections of new points use the kernel centred with the training means; without that, (12.87) differs only by a constant, so the contour lines are the same.

In the lecture notes

Chapter 12 notes: Section 12.3 (kernel PCA: the eigenproblem (12.79), (12.80), the normalisation (12.81), projections (12.82), centring in feature space (12.83) to (12.85), the Gaussian kernel (12.86) and Figure 12.17).
Linear PCA for comparison: Section 12.1; the same picture for ordinary PCA is in the PCA demo.
Python code: code/ch12_continuous_latent.py (functions gaussian_kernel, centre_gram, kernel_pca, kernel_pca_project, kpca_toy_data, fig12_17); checks_ch12.py verifies (12.85) against explicit centring in an explicit feature space.