The model y(x, w) = Σj wj φj(x) is a network with a single layer of weights: each basis function φj is an input node and the output adds them up. Its weights are fitted to noisy samples of sin(2πx) by minimising the sum-of-squares error, either exactly (maximum likelihood, w = Φ†t) or with the penalty (λ/2)‖w‖2. The upper right panel shows each term wjφj(x) of the sum; the lower right panel shows the singular values of the design matrix Φ and how much regularisation shrinks each direction.
Data: x uniform on (0, 1), t = sin(2πx) plus Gaussian noise, drawn with the seeded generator (seed shown in the readout); the test error uses 500 fresh points. Gaussian and sigmoidal centres are equally spaced on [0, 1]; the polynomial basis is xj. The leftmost position of the λ slider is λ = 0 (maximum likelihood). The SVD solver treats singular values below 10−12 times the largest as zero, which gives the minimum-norm least-squares solution.
code/ch04_single_layer_regression.py (sine_data, design_matrix, fit_ml, fit_ridge, fit_normal_equations).