Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 1

Polynomial curve fitting: over-fitting, regularisation and Bayesian prediction

The data are t = sin(2πx) + noise with standard deviation 0.3 at N equally spaced inputs (blue circles; the green curve is the truth). A polynomial of order M (red) minimises the sum of squares, optionally plus the penalty (λ/2)‖w‖². With M = 9 and N = 10 the curve passes through every point and swings wildly in between: over-fitting. More data, or a penalty, or a Bayesian treatment that averages over all weight vectors, tames it.

Show

Method

Number of data points N

Polynomial order M

Regularisation ln λ

Prior precision α

Noise precision β

Try these

The training inputs are x_n = (n−1)/(N−1) and the noise comes from a seeded generator, so the first data set is the one of the figures in the lecture notes (the same numbers as code/ch01_introduction.py). The test set has 100 points from the same process. “No penalty” is the leftmost λ value (λ = 0). The fit is computed by Householder least squares on the design matrix stacked with √λI, because the normal equations are numerically useless for M = 9 (condition number about 10¹⁴). The Bayesian band is the mean ± one standard deviation of the predictive distribution (1.69)–(1.72).

In the lecture notes

Chapter 1 notes: Section 1.1 (polynomial curve fitting, over-fitting, regularisation), Section 1.2.5 (curve fitting re-visited: maximum likelihood and MAP), Section 1.2.6 (Bayesian curve fitting), Section 1.3 (model selection and cross-validation).
Python code: code/ch01_introduction.py (functions js_noisy_sine, fit_poly, rms_error, map_poly, bayes_predictive; Figures 1.2 to 1.8 and 1.17) and the checks in code/checks_ch01.py.