Interactive demo · Foundations of Deep Learning · Chapter 16

A nonlinear latent variable model in two dimensions

Draw z ~ N(0, 1), map it to the plane by a curve g(z), add noise: p(x|z) = N(x | g(z), σ2I) (16.80). The marginal p(x) (16.81) is a blurred copy of the curve. Click anywhere in the left panel to choose a point x: the top-right panel shows which z could have produced it (the posterior), a Gaussian q(z) you control, and the identity ln p(x) = ELBO + KL (16.57). Then replace the circle by a small neural network g(z, w) and train it by maximising the likelihood estimate (16.83), with a quadrature grid or with random prior samples.

Mapping g

Data (200 points)

Noise σ

Likelihood estimate for training

Train the network (Adam, rate 0.02)

Gaussian q(z) = N(m, s²)

Try these

Network: g(z, w) = V tanh(az + b) + c with 16 hidden units. Training maximises (1/N)Σn ln Σk πk N(xn | g(zk, w), σ2I): "grid" uses K equally spaced zk on [−4, 4] with πk ∝ N(zk|0, 1); "Monte Carlo" draws K new zk ~ N(0, 1) at every step with πk = 1/K, which is (16.83). The ELBO, the posterior and ln p(x) use 801 grid points on [−5, 5].

In the lecture notes