Four small networks were trained on 20 000 MNIST digits with a two-dimensional code: a deterministic autoencoder and VAEs with KL weight β = 0.25, 1 and 4 (one hidden layer of 256 units each, 30 epochs). Each dot is the encoder mean of one test digit, coloured by its class; the circles are the 1 and 2 standard-deviation contours of the prior N(0, I). Click or drag in the plane to decode a point. The autoencoder spreads its codes over an arbitrary region, with empty space (holes) between classes where prior draws decode to nonsense; raising β packs the codes into the prior and fills the holes, at the cost of blurrier digits and a lower rate.
The decoded image is a bilinear blend of the true decoder output on a 21 × 21 grid over the plotted square, stored with 36 grey levels (a few percent blurrier than the network itself); points outside the square are decoded at the nearest edge. Interpolation decodes nine equally spaced points on the straight line between A and B. The numbers are computed on 2 000 held-out digits: reconstruction error in nats per image (one latent sample), rate = E KL(q(z|x) ∥ p(z)) split exactly as rate = Iq(x; z) + KL(q(z) ∥ p(z)), and the hole fraction, the share of 4 000 prior draws farther than 0.4 from every one of 1 000 stored codes. Trained on the CPU at small scale, so the digits are blurrier than a large GPU model would give.
code/ch21_variational_autoencoders.py (Figures 21.x1 and 21.x3).