Interactive demo · Probabilistic Machine Learning · Chapter 14

Batch, layer, instance and group normalisation

A convolutional layer produces a tensor with an example index n, a channel index c and a 4×4 map of pixels. Every normalisation layer replaces each activation by (z − μ)/√(σ2 + ε), as in Equations 14.17 to 14.21; the layers differ only in the set Si of activations over which μ and σ2 are computed. Each square on the left is one (example, channel) block of 16 pixels. Click a block: the gold squares are the blocks pooled with it. The data have different means and scales per channel and a random offset and gain per example, so the layers do visibly different things. On the right: how noisy a batch mean is, as a function of the batch size.

Normalisation

Batch size B

Show

Batch norm statistics

Try these

The tensor has 32 examples, 6 channels and 4×4 pixels (the left panel shows the first B examples). ε = 10−5; the scale and shift are the identity (γ = 1, β = 0). "Frozen" batch norm uses the mean and variance of all 32 examples in place of the current batch, as at test time. The right panel uses disjoint consecutive batches of the 32 examples, channel 1.

In the lecture notes