Interactive demo · Applied Statistical Learning in R and Python · Chapter 4

LDA, QDA, logistic regression and naive Bayes on two Gaussian classes

Each class is a bivariate normal with unit variances; you choose the two correlations. When they are equal the Bayes boundary is a straight line and LDA, which assumes a common covariance, estimates it with few parameters. When they differ the Bayes boundary is a conic, which only QDA can follow, but QDA pays for its extra parameters with variance when the sample is small. Naive Bayes ignores the correlations altogether, and logistic regression fits a straight line without any distributional assumption. The test error of every method is measured on 10,000 fresh points; the Bayes error is the best any method can achieve.

Correlation in class 1 (orange)

Correlation in class 2 (blue)

Training points per class n

Separation of the means s

Boundaries

Shade the regions of

Try these

Class means are −s(1, 0.4) and s(1, 0.4). LDA uses the pooled covariance (divisor n1 + n2 − 2), QDA one covariance per class (divisor nk − 1), naive Bayes one normal density per class and predictor; logistic regression is fitted by Newton–Raphson. The Bayes rule uses the true means and covariances.

In the lecture notes