Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 24

Contrastive divergence on a tiny RBM, with no sampling noise

A binary restricted Boltzmann machine with 6 visible and 3 hidden units has only 64 visible states, so every expectation in the learning rule (24.21) can be computed exactly. The data are three noisy prototypes (111000, 000111, 101010). The “unclamped” term is taken from the model itself (exact maximum likelihood), from k block Gibbs steps started at the data (CD-k, expected update) or from a persistent chain (PCD). Pick one, train 6000 updates, and compare the final KL divergence DKL(pD ‖ pθ).

Unclamped term

Gibbs steps k

Learning rate

Try these

Weights start from the same small random matrix every time. CD-k uses the exact distribution of the visible state after k Gibbs sweeps from the data distribution; PCD advances one persistent distribution by k sweeps per update, so neither has Monte Carlo noise. The gap to exact maximum likelihood is the bias of the truncated chain.

In the lecture notes