Interactive demo · Foundations of Deep Learning · Chapter 1

Model selection: cross-validation and the optimism of the training error

A hyperparameter (the order M, or the weight-decay coefficient λ) cannot be chosen by minimising the training error: that always prefers the most flexible model. S-fold cross-validation trains on S − 1 groups, scores on the held-out group, and averages over the S choices (Figure 1.11). The second view shows why the training error misleads: averaged over many data sets it is smaller than the error on fresh targets by exactly 2σ2p/N, where p is the number of fitted coefficients.

View

Training data

Number of folds S

Hyperparameter

Value

Data set size N

Simulated data sets

Scoreboard

Try these

In the lecture notes