Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 28

Scale mixtures and their shrinkage rules

A prior of the form x = εz with z Gaussian and a random scale ε is a Gaussian scale mixture. Observe y = x + noise and ask what the posterior mean E[x|y] is. Every scale mixture shrinks y towards 0, but by very different amounts: a Laplace prior subtracts a constant, a Student or horseshoe prior leaves large observations almost alone, a spike and slab prior kills small ones. The left panel shows log p(x), the right panel the rule E[x|y] (dotted: no shrinkage).

Highlighted prior

Student degrees of freedom ν

Laplace rate λ

Spike-and-slab slab probability π

Noise standard deviation s

Observation y

Try these

The Gaussian prior is N(0, 1) and the slab of the spike and slab is N(0, 1). Posterior means are computed by summing over a fine grid of x; the spike and slab rule is the closed form w(y) y/(1 + s2) with w the posterior slab probability.

In the lecture notes