Interactive demo · Statistical Learning: Theory and Methods · Chapter 17

Fitting a Boltzmann machine: exact gradients against contrastive divergence

The log-likelihood of a Boltzmann machine has gradient data expectation minus model expectation of XjXk (17.34), (17.37). The model expectation needs the partition function, a sum over every state. With 11 binary units (the flow-cytometry proteins, coded 1 above their median) there are only 2048 states, so this page computes it exactly and also runs contrastive divergence CD-k: the model expectation is replaced by the average after k Gibbs sweeps started at the data. Both runs take the same steps from the same start; the curves show the exact log-likelihood of each.

Model

Hidden units (RBM) H

Gibbs sweeps in CD-k

Step size η

Iterations and seed

Try these

Full-batch gradient ascent on the average log-likelihood, θ ← θ + η(Ê(T) − E(T)), from Θ = 0 (fully visible) or from weights 0.1 × N(0, 1) and zero biases (RBM, the same start for both runs). Fully visible: node terms sit on the diagonal of Θ, and a CD sweep updates the 11 nodes in turn from their logistic conditionals (17.30). RBM: hidden units given visible, visible given hidden, and the last hidden layer enters through its probabilities. All 7466 cells are used at every step; the random numbers come from the seeded generator of the course.

In the lecture notes