A network with M logistic hidden units, f(x) = β0 + Σm βmσ(α0m + αmTx), is fitted to 100 training points by minimising the residual sum of squares plus λ times the sum of all squared weights (weight decay), with the quasi-Newton (BFGS) method of the nnet package, from starting weights drawn uniformly on [−0.7, 0.7]. The test error, over 10,000 test points, is shown relative to the Bayes error σ2: a value of 1 is perfect, and the constant fit scores 1 + SNR = 5. Change M, λ and the start, then let the page refit all ten starts for every M (the boxplots of Figures 11.6 and 11.7) or every λ (Figure 11.8).
Both models have inputs with independent standard normal components and noise variance equal to a quarter of the variance of the true function (signal-to-noise ratio 4). The radial function Πφ(Xm) of ten inputs is rescaled to unit variance, so that the decay values mean the same thing for both models. Weight decay acts on all weights, biases included. The training and test sets and the starting weights are drawn from a seeded generator, so every number on this page is reproducible.
code/ch11_neural_networks.py (Net, vmmin, sim_data, sim_experiment).