Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 23

Autoregressive flows on a cross-shaped density

One affine autoregressive layer sets x1 = μ1 + eα1u1 and x2 = μ2(x1) + eα2(x1)u2, so every conditional p(x2 | x1) is Gaussian, however flexible the networks μ2 and α2 are. The cross below has two-humped conditionals, so one layer fails; stacking layers and reversing the order of the variables between them fixes it. The flows were trained in Python by maximum likelihood; your browser runs them: the density by the parallel inverse pass, new samples by the sequential forward pass.

Number of layers

Slice x1 for the conditional

Test points shown in u-space

Overlay

Try these

The cross is an equal mixture of two zero-mean Gaussians with standard deviations 1.5 along the diagonals and 0.25 across them. Each layer's conditioner is a network with two hidden layers of 32 tanh units; the order of the variables is reversed after every layer. Log-likelihoods are averages over the same 2 000 test draws, in nats; the true density scores −2.350 on them (its entropy is 2.346 nats). The red curve on the right is the exact conditional of the cross, a mixture of N(±0.946 x1, 0.3492).

In the lecture notes