A binary restricted Boltzmann machine with 6 visible and 3 hidden units has only 64 visible states, so every expectation in the learning rule (24.21) can be computed exactly. The data are three noisy prototypes (111000, 000111, 101010). The “unclamped” term is taken from the model itself (exact maximum likelihood), from k block Gibbs steps started at the data (CD-k, expected update) or from a persistent chain (PCD). Pick one, train 6000 updates, and compare the final KL divergence DKL(pD ‖ pθ).
Weights start from the same small random matrix every time. CD-k uses the exact distribution of the visible state after k Gibbs sweeps from the data distribution; PCD advances one persistent distribution by k sweeps per update, so neither has Monte Carlo noise. The gap to exact maximum likelihood is the bias of the truncated chain.
code/ch24_energy_based_models.py (Figure 24.x2, class RBMEnum, rbm_train).