Interactive demo · Statistical Learning: Theory and Methods · Chapter 12

Discriminant analysis by regression: optimal scoring

Regress the 528 × 11 indicator matrix Y of the vowel classes on a basis H of the ten inputs, giving fits Ŷ = SY. The eigenvectors θℓ of YTŶ/N, normalised by the class proportions, are the optimal scores: the first is constant, the others turn each class label into a number that regression on H predicts as well as possible (eigenvalue αℓ2 = 1 − residual). The fitted scores ηℓ(x) are discriminant functions; classify to the nearest class centroid with weights 1/[rℓ2(1 − rℓ2)]. With the linear basis this is LDA; with a richer basis it is flexible discriminant analysis, and a ridge penalty makes it penalised discriminant analysis.

Regression basis

Degrees of freedom per input (natural splines)

Ridge penalty λ (0 = none)

Dimension L (discriminant functions used)

Try these

Columns of H are divided by their training standard deviations; the constant is never penalised. Spline knots sit at equally spaced quantiles of each training input. Coordinates are scaled by √wℓ, so distances in the plot are the classification distances (12.54).

In the lecture notes