Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 18

Sparse GPs: inducing points and the variance they lose

Ninety noisy points from a smooth function with a gap in the middle are fitted by the exact GP (dashed) and by an approximation that summarises the data through M inducing inputs (triangles). Choose the approximation, the number and the placement of the inducing points. The approximations share a mean but differ in the uncertainty they keep: SOR collapses it far from the inducing points, DTC and the variational bound restore it, and the variational objective never exceeds the exact log marginal likelihood.

Approximation

Inducing points M

Placement

Try these

The kernel is the SE kernel with length scale 0.8 and variance 1, and the noise variance is 0.04, all held fixed so that only the approximation changes. The objective is the log marginal likelihood of the approximate model (SOR, DTC, FITC) or the collapsed variational bound (VFE, which subtracts the trace term). Greedy placement adds, one at a time, the data point that raises the variational bound most, and is limited to 15 points. The latent standard deviation excludes the observation noise.

In the lecture notes