A convex function g lies above a line through the point (μ, g(μ)), with μ = E(X). Average both sides over the distribution of X: the line averages to g(μ), the curve to E g(X), so E g(X) ≥ g(E X). The red segment is the gap. Each dot is a value x of X placed on the curve, with area proportional to its probability; the red dot, their centre of mass, always sits above the curve. The last data choice turns the same picture into a proof that the Kullback-Leibler divergence is never negative.
Nerve data: the 799 waiting times (seconds) between nerve pulses of the book's file nerve.dat, each with probability 1/799. For 1/x, −log x and x log x, two-point values are kept at 0.05 or more. Kullback-Leibler: p is the histogram of the nerve data on 29 bins of width 0.05, q the probabilities of the same bins under an exponential distribution with mean m (the last bin is open to the right); the variable is Y = q(X)/p(X) with X drawn from p, and g = −log.
code/ch04_inequalities.py (jensen_summary, kl_hist_exponential, nerve; Figure 4.S5).