Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 4

Least squares, Fisher's discriminant and the perceptron

A linear classifier cuts the input space with the hyperplane y(x) = w^Tx + w₀ = 0. Least squares fits the 1-of-K targets by linear regression, which is convenient but treats the targets as Gaussian: a few far-away points tilt the boundary, and with three classes one class can be masked altogether. Fisher's discriminant chooses the direction w that makes the projected classes far apart relative to their spread, J(w) = (w^TS_Bw) / (w^TS_Ww). The perceptron adds t_nφ_n to w for each misclassified point and stops after finitely many updates on separable data.

Method

add 10 outliers (Fig 4.4, right) also show logistic regression (green)

Direction w, angle in degrees

Data

Fit

Scoreboard

Try these

In the lecture notes

Section 4.1.3 (least squares for classification, the pseudo-inverse solution (4.16), why it is not robust, Figures 4.4 and 4.5).
Section 4.1.4 (Fisher's linear discriminant, (4.25) to (4.30), Figure 4.6) and Section 4.1.7 (the perceptron algorithm, (4.54) and (4.55), Figure 4.7).
Python code: code/ch04_linear_classification.py (functions least_squares_classifier, fisher_two_class, fisher_criterion, perceptron, irls_logistic, multiclass_irls; Figures 4.4 to 4.7) and the checks in code/checks_ch04.py.