Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 5

The information bottleneck on word-class counts

The data are document counts of the twelve words that say most about which newsgroup family a message comes from (comp, rec, sci, talk): this gives a joint distribution p(x, y) of word X and family Y. The bottleneck chooses a stochastic clustering p(z|x) to minimise βI(Z;X) − I(Z;Y), by iterating p(z|x) ∝ m(z) exp(−D(p(y|x)‖p(y|z))/β). Large β punishes memory of x and gives one cluster; as β falls the clusters split, one transition at a time, and the point (I(Z;X), I(Z;Y)) climbs the information-plane curve.

Trade-off

Try these

Information is in nats. The sweep starts at β = 10 and falls geometrically to about 0.003 in 44 steps, each step starting from the previous solution; the slider picks one step. The dashed line is the ceiling I(X;Y), reached only when no information about Y is lost. The cluster slots that nobody uses stay empty.

In the lecture notes