A VQ-VAE replaces each encoder output ze by the nearest of K codebook vectors. Instead of a gradient step, the codebook can follow the encoder outputs by the exponential moving averages Ni ← γNi + (1 − γ)ni, mi ← γmi + (1 − γ)Σjzi,j, ei = mi/Ni (Murphy 2023, (21.81) to (21.83)). Here the “encoder” is the identity, so the codebook is learned on the 800 data points themselves: an online k-means. Watch the decay γ trade speed for stability, and see what happens to codes that are never the nearest one.
The data are 800 points from five Gaussian blobs. “Corner start” places six codes near the origin and two far outside the data; “random start” draws eight data points. A minibatch of 800 uses all the data every step (deterministic); smaller batches are drawn without replacement with a fixed seed. Every code starts at N = 1, m = e. The dashed line is the quantisation error that Lloyd’s k-means reaches from the same start. Mean squared error per point, in data units.
code/ch21_variational_autoencoders.py (Figure 21.x5, vq_ema).