How well does a polynomial in horsepower predict the fuel consumption (mpg) of a car it has not seen? Each method holds some cars out, fits on the rest and scores the held-out cars. Draw new random splits and watch the estimates move: a single 50/50 validation split gives curves that jump by several units and disagree about the best degree; 10-fold cross-validation barely moves; leave-one-out cross-validation has no randomness at all and costs a single fit per degree, thanks to formula (5.2).
Every fit is ordinary least squares on orthogonal polynomials built from the training cars (the same column space as 1, x, …, xd). The validation estimate trains on 196 cars and scores the other 196; CV(k) is the average of the k fold MSEs (5.3); LOOCV uses the shortcut (5.2) with the leverages of the full fit.
code/ch05_resampling_methods.py (Figures 5.2 and 5.4; the presets below use the same splits and folds).