Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 3

The bias-variance decomposition and regularisation

Draw L = 100 data sets of N points from sin(2πx) plus noise, fit the same flexible model (24 Gaussian basis functions and a bias) to each by regularised least squares, and look at the fits. A large λ makes every fit similar but systematically wrong (high bias); a small λ makes the fits track the noise (high variance). The expected squared error splits exactly as (bias)² + variance + noise.

View

Regularisation ln λ

Points per data set N

Noise standard deviation σ

Try these

Data set l uses the seeded generator with seed 1000 + l, the same numbers as bias_variance_experiment in the Python code. The bias and variance integrals are averages over 200 equally spaced x on [0, 1] (uniform p(x)); the test error shown is (bias)² + variance + σ², its expectation over test noise. Basis functions: Gaussians of width 0.1 with centres equally spaced on [0, 1]; the penalty (λ/2)w^Tw includes the bias weight, as in (3.27).

In the lecture notes

Chapter 3 notes: Section 3.2 (the bias-variance decomposition, Figures 3.5 and 3.6), with the regularised least squares of Section 3.1.4.
Python code: code/ch03_linear_regression.py (functions ridge_weights, bias_variance, bias_variance_experiment; Figures 3.5 and 3.6) and the exact identity check in code/checks_ch03.py.