Interactive demo · Probabilistic Machine Learning · Chapter 12

Link functions for binary data

A binary response can be modelled with P(y = 1 | x) = ℓ−1(w0 + w1x) for different links ℓ. The logit is the canonical one; the probit uses the Gaussian CDF (12.25); the complementary log-log (12.26) is asymmetric: it approaches 1 much faster than it leaves 0. The data here come from a complementary log-log truth. The probit and logit curves are almost the same up to a rescaling of w, so they cannot be told apart; the complementary log-log fits better, but only with enough data and only in the tails. All fits are maximum likelihood, by Fisher scoring (IRLS), which is Newton's method for the canonical logit link and a weighted least squares iteration with expected weights for the others.

Sample size N (first N points)

Links shown

Vertical scale

Try these

Data: 600 inputs uniform on [−3.2, 2.4]; truth P(y = 1 | x) = 1 − exp(−exp(−0.4 + 0.95x)) (dashed). Deviance is 2 × the negative log likelihood here (the saturated model has likelihood 1 for binary data); AIC adds 4 for the two parameters. The log scales show the probability of a 1 (lower tail) or of a 0 (upper tail), where the links differ most.

In the lecture notes