Thirty observations, ten features, three of them relevant (true weights 2.5, −1.8, 1.2; noise standard deviation 1). Automatic relevancy determination (ARD) gives every weight its own prior precision αj and chooses all of them, and the noise precision, by maximising the marginal likelihood. Irrelevant features end with αj very large and a weight of almost exactly zero. Left: the estimated weights of ARD, ridge and the lasso. Right: how each αj evolves with the iteration.
ARD starts from αj = 1 and noise precision 1/(0.1 var y). MacKay: αj ← γj/μj2 with γj = 1 − αjΣjj; EM: αj ← 1/(μj2 + Σjj). Precisions are capped at 1010. Ridge minimises ½‖y − Xw‖2 + ½λ‖w‖2; the lasso minimises ½‖y − Xw‖2 + λ‖w‖1 by coordinate descent.
code/ch15_generalized_linear_models.py (function ard_fit).