Interactive demo · Probabilistic Machine Learning · Chapter 9

LDA, QDA and naive Bayes: three kinds of decision boundary

A generative classifier fits a Gaussian to each class and applies Bayes' rule (9.3). What matters is the covariance model. A separate covariance for each class gives quadratic boundaries (QDA, 9.4), one shared covariance gives straight boundaries (LDA, 9.7), and diagonal covariances give the naive Bayes classifier, whose Gaussians are aligned with the axes. The slider shrinks the covariance towards its diagonal as in (9.22), and the prior tilt shows that class priors only slide the boundary.

Data set

Covariance model

Shrinkage to the diagonal λ in (9.22)

Prior tilt log-odds shift towards the red class

Display

Try these

Each class is fitted by the maximum likelihood estimates (9.19) to (9.21) on the training points (circles); the test error uses 600 fresh points from the same distribution (not drawn). Data sets: simulated classes with unequal covariances in the spirit of Murphy's Figure 9.1; simulated classes sharing one covariance; and the petal length and width of 90 iris flowers (60 test). Contours are the 1 and 2 standard-deviation ellipses of the fitted Gaussians.

In the lecture notes