Interactive demo · Foundations of Deep Learning · Chapter 4

Linear regression with basis functions: least squares, the pseudo-inverse and regularisation

The model y(x, w) = Σj wj φj(x) is a network with a single layer of weights: each basis function φj is an input node and the output adds them up. Its weights are fitted to noisy samples of sin(2πx) by minimising the sum-of-squares error, either exactly (maximum likelihood, w = Φ†t) or with the penalty (λ/2)‖w‖2. The upper right panel shows each term wjφj(x) of the sum; the lower right panel shows the singular values of the design matrix Φ and how much regularisation shrinks each direction.

Basis functions

Number of parameters M (bias included)

Width s (Gaussian and sigmoid)

Regularisation ln λ

Training points N

Noise sd

Solver

Try these

Data: x uniform on (0, 1), t = sin(2πx) plus Gaussian noise, drawn with the seeded generator (seed shown in the readout); the test error uses 500 fresh points. Gaussian and sigmoidal centres are equally spaced on [0, 1]; the polynomial basis is xj. The leftmost position of the λ slider is λ = 0 (maximum likelihood). The SVD solver treats singular values below 10−12 times the largest as zero, which gives the minimum-norm least-squares solution.

In the lecture notes