Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 3

Which degree? The evidence against the information criteria

A polynomial of degree d is fitted to noisy data with a Gaussian prior on the weights. The log marginal likelihood averages the fit over the prior, so a model that is too flexible wastes prior mass and is penalised automatically (the Bayesian Occam's razor, Section 3.8.1). The criteria BIC and AIC penalise the maximum-likelihood fit instead; exact leave-one-out and WAIC estimate predictive accuracy. Change the noise, the prior precision or the sample and see which ones agree.

Data

Sample size N

Noise in the data (sd)

Noise assumed by the model σ

Prior precision of the weights α (log scale)

Degree shown on the left

Try these

Model: y = ∑k≤d wk xk + ε, ε ~ N(0, σ2), w ~ N(0, α−1 I). Every score is in log-probability units, higher is better: log p(D|d), log p(D|ŵ) − (d+1)/2 · log N (BIC), log p(D|ŵ) − (d+1) (AIC), the exact leave-one-out sum (N refits) and WAIC = lppd − pWAIC.

Scores by degree (best in each column in green)

In the lecture notes