Interactive demo · Statistical Learning: Theory and Methods · Chapter 4

Masking: regression of an indicator matrix versus LDA

Code each class by an indicator Yk (1 in class k, 0 otherwise), regress every indicator on the inputs by least squares and classify to the largest fitted value (4.3)–(4.4). When three or more classes have centroids on a line, the fitted function of an inner class is too flat ever to be the largest: the class is masked, although a straight line separates it perfectly. Linear discriminant analysis, with the same kind of linear boundaries, does not suffer from this. Adding polynomial terms of higher degree to the regression unmasks the classes; the bottom panel shows the fitted indicators along the line through the centroids, as in the book's Figure 4.3.

Number of classes K

Centroids

Degree of the regression terms

Distance between neighbouring centroids (sd = 1)

Points per class

Try these

Each class is a spherical Gaussian with standard deviation 1, drawn from a seeded generator. The regression uses every monomial x1ax2b with a + b ≤ degree (inputs divided by 6 first), solved by the normal equations; LDA uses the class means, the pooled covariance (divisor N − K) and equal priors. Errors are training error rates; the Bayes error for centroids on a line is 2(K − 1)/K · Φ(−d/2).

In the lecture notes