Interactive demo · Bayesian and Probabilistic Machine Learning · Advanced topics

Normalizing flows: the book's 2-d flow, layer by layer

The flow pushes a standard Gaussian base z through six affine maps (each one a matrix and a bias) and five leaky ReLU stages with learned slopes. It is the 2-d model of Murphy's book code (pyprobml, flow_2d_mlp), trained for the book's 5000 Adam steps on the target y = 0.8 x2 + 0.2 ε2, x = 0.6 ε1. The left panel is the exact density after the chosen layer, by the change of variables formula, with base samples pushed to that layer. The right panel is the target density, or the base density. The horizontal axis is the first coordinate (y) and the vertical axis the second (x), as in the book's code.

Weights

Layer shown k

Right panel

Try these

Layer k means the state after the k-th affine map and, for k = 1 to 5, the leaky ReLU that follows it (so layer 6 is the output of the last affine map). The density after layer k is exact: the inverse of the first k stages is applied to the point, and each inverse stage adds its log Jacobian, −log|det M| for an affine map and −log(slope) for a negative input to a leaky stage. The integral is a midpoint sum on a 160 by 160 grid over [−3, 3]2. The weights are the book's trained weights (after 5000 steps of Adam, lr 1e-3, batch 100) and their initial values, stored in code/data/book_flow_2d.json and copied into this page. The random initial weights come from PyTorch, not JAX, so the untrained weights are not the book's random draw. The target density is p(y, x) = N(x; 0, 0.36) N(y; 0.8 x2, 0.04).

In the lecture notes