Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 6

Gaussian process regression: kernels, hyperparameters, ARD

A Gaussian process is a prior over functions defined by a kernel: any finite set of function values is jointly Gaussian. Draw functions from the prior (Figure 6.4), condition on noisy data to get a predictive mean and an uncertainty band that grows away from the data (Figure 6.8), see how the length scale and the noise level change the fit (Figure 6.5), let the marginal likelihood choose the hyperparameters, and watch automatic relevance determination switch off an irrelevant input (Figures 6.9 and 6.10).

Try these

Data: sin(2πx) plus noise of sd 0.3 at equally spaced inputs on [0, 1] (the first N of ten, as in Figure 6.8 where the right-most three are left out), generated with a seeded generator. Inputs on [0, 1] are half the width of the book’s [−1, 1], so a given θ₁ here corresponds to θ₁/4 there. ARD data: x₁ ~ N(0, 0.35²), x₂ = x₁ + noise, x₃ independent, N = 100. Optimiser: Newton (one input scale and noise) or L-BFGS (ARD).

In the lecture notes

Chapter 6 notes: Section 6.4 (Gaussian processes: prior over functions, noise-free and noisy data), Section 6.4.2 (GP regression, predictive mean and variance), Section 6.4.3 (learning the hyperparameters by the marginal likelihood), Section 6.4.4 (automatic relevance determination).
Python code: code/ch06_kernel_methods.py (functions gp_predict, gp_log_marginal, gp_fit, classes GaussKernel and ExpKernel; Figures 6.4 to 6.10) and code/checks_ch06.py (predictive against direct conditioning, gradient of the marginal likelihood by finite differences).