Interactive demo · Probabilistic Machine Learning · Chapter 13

Vanishing and exploding gradients

A chain of L layers zl = φ(Wlzl−1) with 64 units each, weights N(0, c2/64), is fed a batch of 16 standard normal inputs; the objective is half the mean squared norm of the output. Backpropagation (13.69) multiplies the gradient by one Jacobian per layer, so its size changes geometrically with depth: it vanishes if the Jacobians shrink, explodes if they grow. The panels show the gradient with respect to each layer's activations (relative to the output), the size of the activations themselves, and the gradient of each weight matrix. Choose the activation, the weight scale c (He initialisation is c = √2 for ReLU, Glorot and LeCun give c = 1 here) and whether the layers are residual blocks (13.87).

Depth L

Activation

Weight scale c (std = c / √64)

Residual blocks

Branch weight β

Random draw

Try these

Layers have no bias. The residual form is zl = zl−1 + β φ(Wlzl−1); the gradient then passes through the identity as well as through the layer, the mechanism of (13.88)–(13.92). Curves leaving the plot are clipped at 10±40. SELU uses the constants of Klambauer et al.'s self-normalising network with c = 1.

In the lecture notes