The hidden layer of an RBF network is φk(x) = exp(−(x − μk)2/(2σ2)) for K fixed centres μk (13.101)–(13.102), and the output is linear in these features (13.104). With the centres fixed, the weights solve a (ridge) least-squares problem for regression or a convex logistic regression for classification. In regression the bandwidth σ decides how local each bump is: tiny σ gives a spiky fit, huge σ makes all features nearly constant. In classification four Gaussian bumps at the corners solve the XOR problem that no linear model can.
Regression data: 50 simulated points on [0, 20] (a stand-in for the book's data), K equally spaced centres on [0, 20], an unpenalised intercept, weights by ridge regression. Classification data: 100 jittered XOR points; the corner prototypes are (0,0), (0,1), (1,0), (1,1); the weights come from Newton's method for the penalised logistic loss (the intercept is not penalised). The effective number of parameters is the trace of the hat matrix.
code/ch13_neural_networks_tabular.py (rbf_features, rbf_ridge, logreg_newton).