Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 21

The VQ-VAE codebook and its moving-average update

A VQ-VAE replaces each encoder output ze by the nearest of K codebook vectors. Instead of a gradient step, the codebook can follow the encoder outputs by the exponential moving averages Ni ← γNi + (1 − γ)ni, mi ← γmi + (1 − γ)Σjzi,j, ei = mi/Ni (Murphy 2023, (21.81) to (21.83)). Here the “encoder” is the identity, so the codebook is learned on the 800 data points themselves: an online k-means. Watch the decay γ trade speed for stability, and see what happens to codes that are never the nearest one.

Initial codebook

Decay γ

Minibatch

Run

Try these

The data are 800 points from five Gaussian blobs. “Corner start” places six codes near the origin and two far outside the data; “random start” draws eight data points. A minibatch of 800 uses all the data every step (deterministic); smaller batches are drawn without replacement with a fixed seed. Every code starts at N = 1, m = e. The dashed line is the quantisation error that Lloyd’s k-means reaches from the same start. Mean squared error per point, in data units.

In the lecture notes