Interactive demo · Probability and Statistical Inference · Chapter 4

Jensen's inequality, seen through a supporting line

A convex function g lies above a line through the point (μ, g(μ)), with μ = E(X). Average both sides over the distribution of X: the line averages to g(μ), the curve to E g(X), so E g(X) ≥ g(E X). The red segment is the gap. Each dot is a value x of X placed on the curve, with area proportional to its probability; the red dot, their centre of mass, always sits above the curve. The last data choice turns the same picture into a proof that the Kullback-Leibler divergence is never negative.

Distribution of X

Convex function g

Value a (probability w)

Value b (probability 1 − w)

Weight w

Mean of the exponential model m

Try these

Nerve data: the 799 waiting times (seconds) between nerve pulses of the book's file nerve.dat, each with probability 1/799. For 1/x, −log x and x log x, two-point values are kept at 0.05 or more. Kullback-Leibler: p is the histogram of the nerve data on 29 bins of width 0.05, q the probabilities of the same bins under an exponential distribution with mean m (the last bin is open to the right); the variable is Y = q(X)/p(X) with X drawn from p, and g = −log.

In the lecture notes