Interactive demo · Probability and Statistical Inference · Chapter 9

Computing the MLE: Newton–Raphson against EM

Both methods turn the hard problem "maximise ℓ(θ)" into a sequence of easy ones. Newton–Raphson (blue) replaces ℓ by its second-order Taylor parabola at the current point, whose curvature is the observed information, and jumps to that parabola's top. EM (gold) fills in the missing data by their conditional expectation and maximises the result; in one dimension this is a parabola that lies below ℓ, touches it at the current point, and has the curvature of the complete data. So EM can never decrease the likelihood, but its steps are too short by the fraction of information that is missing, and it converges linearly at that rate; Newton converges quadratically once it is close, and may wander off, stop at a saddle point or break down when it is not. Two problems: right-censored Normal data (we see min(X, c) and whether X > c), and the two-Normal mixture of Example 9.50 with equal weights and known spread.

Problem

Sample size n

Censoring point c (data N(1, 1))

Separation Δ (data ½N(0, 1) + ½N(Δ, 1))

Starting value

Show iterations up to j

Try these

Censored data: n draws from N(1, 1); values above c are replaced by "above c". EM step: μ ← [Σobserved y + m(μ + λ(c − μ))]/n, with m censored values and λ(t) = φ(t)/(1 − Φ(t)). Mixture: weights ½, standard deviation 1 (0.4 for the Old Faithful eruption times, in minutes); EM step: responsibility-weighted means. On the contour map, click to set the starting point. The MLE used as the target is EM's limit from the lower and upper quartiles of the data, polished by Newton steps; for the mixture the two labellings count as the same answer. A Newton run stops when the Hessian is singular or the iterate leaves the picture by a factor of a million.

In the lecture notes