Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 3

The evidence function: model order, hyperparameters and the effective number of parameters

The marginal likelihood p(t | α, β) = ∫ p(t | w, β) p(w | α) dw has a closed form (3.86), so it can be used to compare models and to set α and β without a validation set. Compare polynomial orders, scan α, run the re-estimation equations (3.92) and (3.95), and see how many of the weights the data determine: γ = Σ_i λ_i / (α + λ_i).

View

Data points N

Noise standard deviation σ (data; fixed β = 1/σ²)

Prior precision ln α

Try these

Order view: x equally spaced on [0, 1], t = sin(2πx) + noise, polynomial of order M = 0..8, α = 5×10⁻³ (as in the text), β = 1/σ². The other views use nine Gaussian basis functions of width 0.1 plus a bias (ten weights) and x uniform on (0, 1). Iteration starts from α = β = 1 and updates both hyperparameters.

In the lecture notes

Chapter 3 notes: Section 3.4 (Bayesian model comparison, the evidence (3.86)), 3.5 (the evidence approximation), 3.5.1 (maximising the evidence), 3.5.2 (the effective number of parameters, Figures 3.15 to 3.17).
Python code: code/ch03_linear_regression.py (functions log_evidence, gamma_eff, evidence_update, evidence_iterate; Figures 3.14 to 3.17) and the checks against numerical integration and direct maximisation in code/checks_ch03.py.