Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 5

Backpropagation: a two-layer network, over-fitting, weight decay and early stopping

A network with M tanh hidden units and one linear output, y(x) = b + Σ_j w_j tanh(a_j(x)), is trained by full-batch gradient descent (Adam) on the sum-of-squares error, with the gradient computed by backpropagation. The dashed curves are the outputs z_j = tanh(a_j) of the individual hidden units (rescaled to fit the plot); the red curve is the network function. The lower right panel compares every backpropagated derivative with a central finite difference.

Target function

Hidden units M

Training points N

Noise sd

Weight decay λ

Step size

validation set (same size)

Try these

Data: x uniform on (0, 1) for sin(2πx) and on (−1, 1) for the other targets, plus Gaussian noise with the chosen standard deviation. Training minimises E(w)/N + (λ/2)‖w‖² with Adam, full batch, from weights drawn N(0, 1) (first layer) and N(0, 1/M) (second layer); the error curves show the RMS error √(2E/N). All random numbers are seeded, so a class sees the same pictures.

In the lecture notes

Chapter 5 notes: Section 5.1 (network functions, hidden units), Section 5.2 (error functions and training), Section 5.3 (backpropagation and its gradient check) and Section 5.5 (weight decay, early stopping).
Python code: code/ch05_neural_networks.py (class Net, adam_train, Figures 5.3, 5.9, 5.10, 5.12, 5.13) and the checks in code/checks_ch05.py. The data, initial weights and Adam steps of this page are the same as data_sine, init_weights and adam_train with JSRandom(seed), number for number.