Interactive demo · Probabilistic Machine Learning · Chapter 4

Ridge regression and cross-validation

Twenty-one noisy points are fitted by a polynomial of chosen degree (default 14) with the penalty λ‖w‖2 on all coefficients except the intercept (4.99). With λ near zero the fit chases the noise: tiny training error, large test error. Increasing λ trades variance for bias until the model underfits. The right panel plots the training error, the error on 500 fresh points and the K-fold cross-validation estimate (4.106) with its standard error; the one-standard-error rule picks the largest λ whose cross-validated error is within one standard error of the minimum.

Regularisation log10 λ

Polynomial degree

Folds K

Try these

Data: 21 equally spaced inputs on [0, 20], truth 4 sin(x/3) + 0.25(x − 10), Gaussian noise with standard deviation 2. Features are the Legendre polynomials P1, ..., Pdegree of (x − 10)/10; the intercept is not penalised. Folds are contiguous blocks of the 21 points (sizes as in numpy's array_split); K = 21 is leave-one-out. The standard error is the standard deviation of the 21 out-of-fold squared errors divided by √21.

In the lecture notes