Train a network by gradient descent and its training error keeps falling, while the error on held-out data falls and then rises: the network's effective complexity grows with the number of steps. Stopping at the minimum of the validation error is early stopping. On a quadratic error the link to weight decay is exact: after τ steps of size ρ from the origin, each eigen-direction of the Hessian keeps the fraction 1 − (1 − ρηj)τ of the minimiser, while weight decay keeps ηj/(ηj + λ); with λ = 1/(ρτ) the two agree within a factor of 2 in every direction.
Network: y = ∑j vj tanh(wjx + bj) + c, full-batch gradient descent on ½∑(y − t)2 + (λ/2)(|w|2 + |b|2 + |v|2), 10 training and 20 validation points of sin(2πx) plus noise of standard deviation 0.3, errors as root mean squares, recorded 50 times per run.
code/ch09_regularization.py (sine_data, net_init, train_net, gd_filter, ridge_filter, Figures 9.7, 9.8 and 9.S2); the learning curves here equal the Python ones.