Entropy H[p] = −Σ p ln p measures how spread out a distribution is; the Kullback–Leibler divergence KL(p‖q) is the extra code length paid for using q when the data come from p; it is never negative, is zero only when q = p, and is not symmetric. Minimising KL(p‖q) over q is maximum likelihood and covers all of p's mass; minimising KL(q‖p) (as variational methods do) locks onto one mode. Mutual information is the KL divergence between a joint distribution and the product of its marginals.
code/ch02_probabilities.py (functions entropy, binned_gauss, kl_gauss, fit_forward_kl, fit_reverse_kl, discrete_mi).