Interactive demo · Foundations of Deep Learning · Chapter 3

The exponential family: maximum likelihood is moment matching

Write a distribution as p(x|η) = h(x) g(η) exp{η u(x)} and let ψ(η) = −ln g(η). Then ψ′(η) is the mean of u(x) and ψ″(η) its variance, so the log likelihood is concave and its maximiser solves one equation: model mean of u = sample mean of u. The data enter only through the sufficient statistic Σu(xn). Newton's method on this equation uses the variance as its slope.

Family

Try these

Data come from a seeded generator, the same numbers as the Python code (expfam_demo_data, expfam_ml_newton).

In the lecture notes