Fifty training points are drawn from Y = f(X) + ε with Var(ε) = 1 and a known f (white curve). A smoothing spline is fitted with a chosen number of degrees of freedom, from 2 (the least-squares line) to 30 (a very wiggly curve). The training MSE keeps falling as the flexibility grows; the test MSE, measured on new data, falls and then rises again, and can never go below the irreducible error Var(ε) = 1 (dashed). Where the bottom of the U lies depends on how far f is from a straight line.
The inputs are fixed (one in each of 50 equal bins of [0, 100]); only the noise changes between training samples. The curves are exact expectations over the noise: the training MSE is (1/n)(‖(I − S)f‖² + σ² tr[(I − S)²]) and the test MSE is σ² + bias² + variance at a new input uniform on [0, 100] (500 equally spaced points). The squares are the training and test MSE of the fit on the current sample.
code/ch02_statistical_learning.py (SmoothingSpline, exact_curves, sample_mse, Figures 2.9 to 2.12).