Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 15

Automatic relevancy determination against ridge and the lasso

Thirty observations, ten features, three of them relevant (true weights 2.5, −1.8, 1.2; noise standard deviation 1). Automatic relevancy determination (ARD) gives every weight its own prior precision αj and chooses all of them, and the noise precision, by maximising the marginal likelihood. Irrelevant features end with αj very large and a weight of almost exactly zero. Left: the estimated weights of ARD, ridge and the lasso. Right: how each αj evolves with the iteration.

ARD update

ARD iterations

Ridge penalty λ (log scale)

Lasso penalty λ

Try these

ARD starts from αj = 1 and noise precision 1/(0.1 var y). MacKay: αj ← γj/μj2 with γj = 1 − αjΣjj; EM: αj ← 1/(μj2 + Σjj). Precisions are capped at 1010. Ridge minimises ½‖y − Xw‖2 + ½λ‖w‖2; the lasso minimises ½‖y − Xw‖2 + λ‖w‖1 by coordinate descent.

In the lecture notes