Interactive demo · Applied Statistical Learning in R and Python · Chapter 9

Hinge loss against logistic loss: who pulls on the boundary

Written as loss plus penalty (9.25), the support vector classifier minimises ∑i max[0, 1 − yif(xi)] + λ∑jβj². Replace the hinge by the logistic loss log(1 + e−y f) and the same penalty gives ridge-penalised logistic regression. The two losses have the same shape (Figure 9.12), so the two boundaries are usually close; the difference is in who is allowed to pull. In the fitting equations observation i pulls with weight equal to the slope of its loss: for the hinge that is exactly 0 once yif(xi) > 1, so points beyond the margin have no say; for the logistic loss it is 1 − Pr(correct class), small but never 0. Push the well-classified points away and see which boundary moves.

Penalty λ (log scale)

Push points with y f(x) > 1.5 away by

Try these

Data: the 50 points of Section 9.6.1 of the Python lab (default_rng(1), class +1 shifted by 1). The hinge problem is solved as LIBSVM's cost form with C′ = 1/(2λ) by a small SMO solver; the logistic problem by Newton's method. Neither penalty touches the intercept. The pushed points are those with y f(x) > 1.5 under the support vector classifier on the original data; each moves along the normal of that hyperplane, away from it.

In the lecture notes