Interactive demo · Foundations of Deep Learning · Chapter 2

Entropy, KL divergence and mutual information

Entropy H[p] = −Σ p ln p measures how spread out a distribution is; the Kullback–Leibler divergence KL(p‖q) is the extra code length paid for using q when the data come from p; it is never negative, is zero only when q = p, and is not symmetric. Minimising KL(p‖q) over q is maximum likelihood and covers all of p's mass; minimising KL(q‖p) (as variational methods do) locks onto one mode. Mutual information is the KL divergence between a joint distribution and the product of its marginals.

View

Width of the distribution (bins)

Number of states M

p = N(μp, σp2)

q = N(μq, σq2)

Separation of the two modes (±d)

Weight of the left mode

Reverse-KL search starts at μ =

p(x = 1)

p(y = 1 | x = 0)

p(y = 1 | x = 1)

Scoreboard

Try these

In the lecture notes