Interactive demo · Foundations of Deep Learning · Chapter 7

Initialisation and activation statistics through depth

A batch of 100 random inputs passes through L layers of M units, a(l) = W(l)z(l−1) (7.19), with weights drawn from N(0, ε2), ε = g/√M. He initialisation (7.23) is g = √2. A random error signal is then backpropagated. The top panels show, layer by layer, the mean square of the pre-activations and of the backpropagated errors δ = ∂E/∂a on a log scale; the bottom panel shows the distribution of the pre-activations (or activations) in one layer.

Activation function

Weight scale g (ε = g/√M)

Depth L and width M

Normalisation before the activation

Histogram: layer and quantity

Try these

Inputs and the output error signal are standard normal; biases are zero. With ReLU and no normalisation the theory line is E[(a(l))2] = g2(g2/2)l−1; for linear units it is g2l. Batch normalisation (7.52)–(7.54) uses the 100 examples of the batch, layer normalisation (7.58)–(7.60) the M units of each example, δ = 10−5, γ = 1, β = 0. All random numbers are seeded.

In the lecture notes