Interactive demo · Foundations of Deep Learning · Chapter 8

Backpropagation step by step, with a gradient check

A two-layer network with two inputs, M hidden units and linear, logistic or softmax outputs computes aj = Σi wji(1)xi, zj = h(aj) and ak = Σj wkj(2)zj (8.18)–(8.20). Step through Algorithm 8.1: the forward pass, the output errors δk = yk − tk, the backward recursion δj = h′(aj) Σk wkjδk (8.13) and the derivatives δjzi (8.10). The lower panel checks one derivative against finite differences (8.24) and central differences (8.25) for every step size ε.

Stage of Algorithm 8.1

Hidden units M

Weights (seed of the random draw)

Hidden activation h

Output and error

Skip-layer weights (Exercise 8.4)

Put hidden unit 1 at its kink

Weight to check

Step size log10 ε

Try these

Input x = (0.6, −0.4). Targets: t = (0.5, −0.3) for linear outputs with the sum-of-squares error (8.17); t = 1 for one logistic output with the cross-entropy; t = (0, 1, 0) for three softmax outputs with the cross-entropy. Weights are normal with standard deviation 0.9, drawn in the order W(1), W(2), U (skip); index 0 is the bias.

In the lecture notes