Interactive demo · Probabilistic Machine Learning · Chapter 13

Radial basis function networks

The hidden layer of an RBF network is φk(x) = exp(−(x − μk)2/(2σ2)) for K fixed centres μk (13.101)–(13.102), and the output is linear in these features (13.104). With the centres fixed, the weights solve a (ridge) least-squares problem for regression or a convex logistic regression for classification. In regression the bandwidth σ decides how local each bump is: tiny σ gives a spiky fit, huge σ makes all features nearly constant. In classification four Gaussian bumps at the corners solve the XOR problem that no linear model can.

Problem

Bandwidth log10 σ

Number of centres K

Prototypes

Ridge penalty log10 λ

Try these

Regression data: 50 simulated points on [0, 20] (a stand-in for the book's data), K equally spaced centres on [0, 20], an unpenalised intercept, weights by ridge regression. Classification data: 100 jittered XOR points; the corner prototypes are (0,0), (0,1), (1,0), (1,1); the weights come from Newton's method for the penalised logistic loss (the intercept is not penalised). The effective number of parameters is the trace of the hat matrix.

In the lecture notes