Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 7

The relevance vector machine: sparsity from the evidence

The RVM is a linear model in kernel basis functions centred on the data, y(x) = ∑_n w_n k(x, x_n) + w₀, with a separate Gaussian prior precision α_i on every weight. Maximising the marginal likelihood over the α_i drives most of them to infinity, which removes those basis functions: the survivors are the relevance vectors. Press One update to apply one re-estimation step α_i = γ_i/m_i² (and the noise update), or Run, and watch the evidence rise and the basis functions drop out. In regression, click a data point to see its own evidence curve λ(α_i) (Figure 7.11).

Problem

Number of points N

Noise standard deviation

overlay the ν-SVM regression (ν = 0.7)

Kernel width γ in exp(−γ‖x−x′‖²)

Hyperparameter updates

Try these

Regression data: t = sin(2πx) + noise at evenly spaced x. Classification data: two overlapping classes, each a mixture of two Gaussians (a stand-in for the synthetic set of Bishop 2006, Appendix A), 50 points per class; the SVM beside it is the C-SVM with C chosen so that about half of the points are support vectors. Green rings: relevance vectors; orange rings: support vectors.

In the lecture notes

Chapter 7 notes: Section 7.2 (relevance vector machines), Section 7.2.1 (regression, the re-estimation equations), Section 7.2.2 (the analysis of sparsity, sparsity and quality), Section 7.2.3 (classification by the Laplace approximation).
Python code: code/ch07_sparse_kernel_machines.py (functions rvm_regression, rvm_fast_regression, rvm_classification; Figures 7.9 to 7.12) and the checks code/checks_ch07.py. The support vector machine is in the previous demo.