Interactive demo · Bayesian and Probabilistic Machine Learning · Advanced topics

Normalizing flows: an affine coupling flow, layer by layer

Each layer keeps one coordinate xa and maps the other one by xb′ = xb exp(s(xa)) + t(xa), where s = tanh(MLPs) and t = MLPt are small networks (1 to 8 to 1 with tanh). The roles of the two coordinates alternate from layer to layer, so both are transformed. The base is a standard Gaussian or a Student-t with 3 degrees of freedom. The left panel is the exact density after the chosen number of layers, by the change of variables formula (Bishop, Section 1.2.1): the inverse map gives the base point, and the log determinant of the layers corrects the density. The right panel pushes 400 base samples through the same layers.

Base distribution

Layers K

Scale of s

Try these

Weights: the networks are drawn from a seeded generator (seed 11), with hidden weights and biases standard normal (the biases times 0.5) and the output weights scaled by 0.2, so the layers start close to the identity. The scale slider multiplies s (s = scale · tanh(MLPs)), so a larger scale stretches the density more. The integral is a midpoint sum of the exact density on a 160 by 160 grid over [−3, 3]2. It is close to 1 only when most of the mass stays in the box: mass outside the box is not counted, so a stretched flow can give a smaller integral. The Student-t base is the multivariate t with identity scale and 3 degrees of freedom.

In the lecture notes