Twenty points y = sin(x) + ε, σ = 0.3, are fitted by a natural cubic spline with d basis functions (knots at d quantiles of the x's). Up to d = 20 this is least squares; at d = 20 = n the spline passes through every point; beyond it there are infinitely many interpolants, and the one with the smallest ∑βj2 is taken. The test error rises to a peak at the interpolation threshold and then falls again: this is the double descent of Figure 10.20. The bottom panel shows why: the size of the coefficients peaks at the threshold too.
The basis is the subspace of cubic B-splines whose second derivative vanishes at the two boundary knots, with orthonormal coefficients; the minimum-norm fit is β = BT(BBT)+y and ridge is BT(BBT + λI)−1y. The test error is the expected squared error at a new point, σ2 plus the average of (sin x − f̂(x))2 over [−5, 5], so it carries no test-set noise. The grey dashed curve is the median test error of minimum-norm fits over 100 other training samples.
code/ch10_deep_learning.py (ns_basis_full, min_norm_fit, dd_curve).