Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 28

Collapsed Gibbs sampling for LDA

Sixteen short documents use five words. Topic 1 is uniform on {money, loan, bank}, topic 2 on {river, stream, bank}. The sampler sees only the words. After integrating out the topic proportions and the topic-word probabilities it resamples the topic of one token at a time from p(m = k | rest) ∝ (Nvk + β)/(Nk + Vβ) · (Nik + α). Each row of the left picture is a document, each dot a token (teal: topic 1, pink: topic 2); the word "bank" is assigned by its neighbours. The right panels show the estimated topic-word probabilities against the truth, and the collapsed log joint.

Document-topic concentration α

Topic-word concentration β

Try these

The 16 documents, the random initial assignment and the sampler's random numbers are fixed, so a preset always gives the same picture. Topic labels are only identified up to a permutation, so the estimated topics are matched to the true ones before the error is computed. α and β are the symmetric Dirichlet parameters of the document proportions and of the topics.

In the lecture notes