Interactive demo · Foundations of Deep Learning · Chapter 9

Early stopping versus weight decay

Train a network by gradient descent and its training error keeps falling, while the error on held-out data falls and then rises: the network's effective complexity grows with the number of steps. Stopping at the minimum of the validation error is early stopping. On a quadratic error the link to weight decay is exact: after τ steps of size ρ from the origin, each eigen-direction of the Hessian keeps the fraction 1 − (1 − ρηj)τ of the minimiser, while weight decay keeps ηj/(ηj + λ); with λ = 1/(ρτ) the two agree within a factor of 2 in every direction.

Mode

Hidden units M

Learning rate ρ

Gradient steps

Weight decay λ (leftmost: none)

Data set

Show the network after step τ

Steps τ (log scale)

Learning rate ρ (eigenvalues 1 and 1/κ)

Condition number κ

Try these

Network: y = ∑j vj tanh(wjx + bj) + c, full-batch gradient descent on ½∑(y − t)2 + (λ/2)(|w|2 + |b|2 + |v|2), 10 training and 20 validation points of sin(2πx) plus noise of standard deviation 0.3, errors as root mean squares, recorded 50 times per run.

In the lecture notes