Interactive demo · Probabilistic Machine Learning · Chapter 10

Robust logistic regression: mixture likelihood and tempered losses

Fifty setosa (class 0) and fifty versicolor (class 1) flowers are classified from the sepal length. Up to six versicolor flowers with abnormally low sepal length (they look like setosa) can be added as label noise; the standard logistic fit is dragged towards them and its decision boundary moves. Two remedies from Section 10.4 are available: a likelihood that says a label is replaced by a fair coin with probability π (10.99), and the bi-tempered loss, whose bounded loss (t1 < 1) and heavy-tailed link (t2 > 1) limit the influence of a mislabelled point (10.102, 10.105). The right panel shows the loss that a class-1 point pays as a function of its logit a: the standard loss grows without bound as the logit becomes very negative, the robust ones flatten.

Outliers added k

Model

Flip probability π

Loss temperature t1

Link temperature t2

Try these

Data: the 100 iris flowers of species setosa and versicolor (sepal length, cm); the six outliers have sepal length 3.5 to 4.2 and label 1. The fit minimises the mean loss plus 0.005 w2 for the slope w, in the model p(y = 1) = σ(b + w (x − 5.47)); the mixture model maximises the likelihood (10.99) for the chosen π rather than averaging over π. The mixture and bi-tempered objectives are not convex, so the page starts the search from five points and keeps the best. With t1 = t2 = 1 the bi-tempered loss is the ordinary logistic loss. Large t2 together with small t1 can trap the search in a poor local minimum on this one-dimensional example; try it.

In the lecture notes