Interactive demo · Applied Statistical Learning in R and Python · Chapter 5

Validation set, LOOCV and k-fold CV on the Auto data

How well does a polynomial in horsepower predict the fuel consumption (mpg) of a car it has not seen? Each method holds some cars out, fits on the rest and scores the held-out cars. Draw new random splits and watch the estimates move: a single 50/50 validation split gives curves that jump by several units and disagree about the best degree; 10-fold cross-validation barely moves; leave-one-out cross-validation has no randomness at all and costs a single fit per degree, thanks to formula (5.2).

Method

Number of folds k

Degree shown

Try these

Every fit is ordinary least squares on orthogonal polynomials built from the training cars (the same column space as 1, x, …, xd). The validation estimate trains on 196 cars and scores the other 196; CV(k) is the average of the k fold MSEs (5.3); LOOCV uses the shortcut (5.2) with the leverages of the full fit.

In the lecture notes