Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 6

Gaussian process classification: the Laplace approximation

A Gaussian process on a latent function a(x), squashed through a logistic sigmoid, becomes a classifier. The posterior over the latent values at the training points is not Gaussian, so the Laplace approximation finds its single mode by Newton-Raphson (an IRLS iteration), fits a Gaussian there, and gives a predictive probability and an approximate evidence for the hyperparameters (Figure 6.12). Compare a short and a long length scale, learn the hyperparameters, and step through the Newton iteration.

Try these

Data: a synthetic two-class set of the kind in Appendix A of Bishop (2006), Figure A.7: one Gaussian for the blue class and a mixture of two Gaussians for the red class, equal priors, generated in the browser with a seeded generator (the book’s own points are not available). Kernel θ₀ exp(−θ₁‖x−x′‖²/2) + νδ, ν = 10⁻⁴. The gradient of the evidence uses (I + CW)⁻¹ in (6.94); the printed (I + WC)⁻¹ does not match its own derivation (see the notes).

In the lecture notes

Chapter 6 notes: Section 6.4.5 (Gaussian processes for classification: the latent function and the sigmoid), Section 6.4.6 (Laplace approximation: Newton-Raphson for the mode, the predictive distribution and the evidence).
Python code: code/ch06_kernel_methods.py (functions laplace_mode, laplace_predict, laplace_evidence, gpc_fit; Figures 6.11 and 6.12) and code/checks_ch06.py (mode gradient zero, Newton monotone, evidence gradient by finite differences).