Take zero-mean Gaussian data with covariance eigenvalues λ1 ≥ λ2 ≥ …, a linear decoder with noise variance σ2 and the loss −E log p(x|z) + β KL. The optimum is available in closed form. A direction with λ ≤ βσ2 is collapsed: the posterior equals the prior, the rate is 0 and the whole variance λ is left as error. A direction above the water level βσ2 is used: it costs (1/2) log(λ/βσ2) nats and leaves exactly βσ2 of error. Raise β or σ2 and the water rises over the spectrum, switching directions off from the weakest up; at β = 1 this is probabilistic PCA.
Everything is evaluated from the closed form: with ui = max(1, λi/(βσ2)) the rate is (1/2) log ui, the error λi/ui = min(λi, βσ2), the posterior variance 1/ui, and the loss is error/(2σ2) + β · rate. The MNIST spectrum is that of the pixel covariance of 20 000 training digits (784 directions; the first 40 are drawn, the totals use all of them). Nats throughout.
code/ch21_variational_autoencoders.py (Figure 21.x4, linear_vae_theory and fit_linear_vae).