Interactive demo · Statistical Learning: Theory and Methods · Chapter 12

Mixture discriminant analysis: several prototypes per class

LDA gives each class one Gaussian prototype; MDA gives it R subclasses, P(X | G = k) = ∑r πkr φ(X; μkr, Σ), all with the same covariance Σ. EM fits it: the E-step shares each observation among the subclasses of its own class in proportion to πkrφ(xi; μkr, Σ); the M-step is a weighted LDA with the subclasses as classes. The start is a k-means partition within each class. Change R, the random start and the number of EM iterations; the boundary is where the two class posteriors are equal.

Subclasses per class R

Random start (k-means seed)

EM iterations (200 = run to convergence)

Try these

Blue and orange discs are the subclass centres of the two classes, with the one-standard-deviation ellipse of Σ around each. The white curve is the MDA boundary, the purple dashed curve the Bayes boundary. The test error uses the 10,000 test points of Chapter 2; EM stops when the log-likelihood changes by less than 10−10 of its value.

In the lecture notes