Interactive demo · Applied Statistical Learning in R and Python · Chapter 2

The bias-variance decomposition

Draw many training sets from the same model Y = f(X) + ε and fit a smoothing spline with the same flexibility to each. The fits (thin teal curves) scatter around their average (gold). The squared bias is how far the average fit is from f (white); the variance is how much the fits spread around their average. Equation (2.7): expected test MSE = variance + squared bias + Var(ε), at every input x0 and therefore on average over the inputs. Stiff fits have small variance and large bias; wiggly fits the reverse.

True function

Flexibility (degrees of freedom)

Input x0

Try these

The 50 inputs are fixed and only the noise changes, so a smoothing spline is a linear smoother, f̂(x0) = l(x0)Ty, and its bias and variance are exact: bias = l(x0)Tf − f(x0), variance = σ²‖l(x0)‖². The "simulated" column estimates the same averages from the training sets drawn so far (bias² from the average fit, variance from the spread of the fits); it approaches the exact column as you draw more.

In the lecture notes