Seven VAEs, identical except for the weight β on the KL term, were trained on binarised MNIST with M = 10 latent dimensions: each minimises distortion D = −Eq[ln p(x|z)] plus β times the rate R = KL(q(z|x)‖p(z)), averaged over the data. Their decoders run here in your browser. Small β: sharp reconstructions, a code far from the prior, and samples from the prior that look unlike digits. Large β: the rate falls, whole latent dimensions collapse to the prior (they carry no information: moving along them changes nothing), and reconstructions blur. Click a bar in the KL chart to traverse that dimension.
Encoder 784–256–20 (tanh; means and log-variances), decoder 10–128–784 (tanh, then a sigmoid giving Bernoulli means), Adam for 12 epochs, one reparameterised sample per digit. R and D are in nats per digit on the 10 000 test digits; the ELBO is −(D + R) whatever β was used in training. A dimension is active if its encoder mean varies across the test set (variance > 0.01). Images show decoder means; traversal sets zj to −3, …, 3 with the other coordinates at the digit's encoder mean.
code/ch19_autoencoders.py (train_vae, vae_eval, export_vae); the decoders (float16) and numbers are in data/ch19-vae-summary.json and data/ch19-vae-b<β>.json, and the images here equal vae_decode to rounding error.