Interactive demo · Probabilistic Machine Learning · Chapter 12

Predicting insurance claims: ridge against Poisson regression

Simulated motor policies have an exposure (years insured), three binned risk factors and a claim count; the target is the claim frequency, claims per year of exposure. Three predictors are fitted on the training policies, all weighted by exposure: a constant, a ridge regression (linear mean, Gaussian loss) and a Poisson regression (log link, maximum likelihood by IRLS). They are scored on held-out policies by the mean squared error, the mean absolute error and the Poisson deviance (12.37), and by a calibration curve: sort the test policies by predicted frequency, split into ten groups of equal exposure, and compare the mean prediction with the mean observed frequency. The linear model can predict negative frequencies and miscalibrates the low-risk end; the Poisson model cannot.

Training policies N

Ridge penalty log10 α (ridge model)

Risk factors used

Model in the left plot

Try these

Data: 24,000 simulated policies, 12,000 available for training (the slider uses the first N of them) and 12,000 held out; about 94% have no claim. The risk factors are binned driver age (8 levels), vehicle age (6) and bonus-malus (6), one-hot encoded. The Poisson model carries a tiny ridge penalty (10−4) only to keep the Newton steps well defined when a factor level has no claims. Floor of 0.001 on predictions inside the deviance, since the linear model can go negative.

In the lecture notes