A polynomial of degree d is fitted to noisy data with a Gaussian prior on the weights. The log marginal likelihood averages the fit over the prior, so a model that is too flexible wastes prior mass and is penalised automatically (the Bayesian Occam's razor, Section 3.8.1). The criteria BIC and AIC penalise the maximum-likelihood fit instead; exact leave-one-out and WAIC estimate predictive accuracy. Change the noise, the prior precision or the sample and see which ones agree.
Model: y = ∑k≤d wk xk + ε, ε ~ N(0, σ2), w ~ N(0, α−1 I). Every score is in log-probability units, higher is better: log p(D|d), log p(D|ŵ) − (d+1)/2 · log N (BIC), log p(D|ŵ) − (d+1) (AIC), the exact leave-one-out sum (N refits) and WAIC = lppd − pWAIC.
code/ch03_statistics.py (beta_logml, bic_logml_coin); all figures in the code index.