A hyperparameter (the order M, or the weight-decay coefficient λ) cannot be chosen by minimising the training error: that always prefers the most flexible model. S-fold cross-validation trains on S − 1 groups, scores on the held-out group, and averages over the S choices (Figure 1.11). The second view shows why the training error misleads: averaged over many data sets it is smaller than the error on fresh targets by exactly 2σ2p/N, where p is the number of fitted coefficients.
code/ch01_deep_learning_revolution.py (functions folds, cv_rms, cv_curve, loo_shortcut, optimism_mc, optimism_theory).