Interactive demo · Probabilistic Machine Learning · Chapter 6

Entropy, cross entropy, KL divergence and perplexity

Two distributions p and q on five states are set by the sliders (the numbers are weights, normalised to sum to one). The entropy H(p) is the average number of bits needed to code a draw from p with a code designed for p; the cross entropy H(p, q) is the average when the code is designed for q instead; the KL divergence is the extra cost, D(p ‖ q) = H(p, q) − H(p) ≥ 0. The perplexity 2H is the effective number of equally likely states.

Weights of p (teal)

Weights of q (gold)

Try these

Logarithms are base 2, so the units are bits; 0 log 0 = 0. A state with weight 0 has probability 0. The value ∞ appears when q gives probability 0 to a state that p can produce: such a code cannot encode that state at all.

In the lecture notes