A batch of 100 random inputs passes through L layers of M units, a(l) = W(l)z(l−1) (7.19), with weights drawn from N(0, ε2), ε = g/√M. He initialisation (7.23) is g = √2. A random error signal is then backpropagated. The top panels show, layer by layer, the mean square of the pre-activations and of the backpropagated errors δ = ∂E/∂a on a log scale; the bottom panel shows the distribution of the pre-activations (or activations) in one layer.
Inputs and the output error signal are standard normal; biases are zero. With ReLU and no normalisation the theory line is E[(a(l))2] = g2(g2/2)l−1; for linear units it is g2l. Batch normalisation (7.52)–(7.54) uses the 100 examples of the batch, layer normalisation (7.58)–(7.60) the M units of each example, δ = 10−5, γ = 1, β = 0. All random numbers are seeded.
code/ch07_gradient_descent.py (depth_stats, supplementary Figure 7.S1). With js=True it uses the same random numbers as this page and gives the same per-layer values.