Interactive demo · Probabilistic Machine Learning · Chapter 5

Risk functions of estimators: Bayes, minimax and Stein's paradox

Data are N draws from N(θ*, 1) and the loss is squared error, so the risk of an estimator is its mean squared error as a function of the unknown θ*. No estimator is best for every θ*: the constant θ0 wins at θ* = θ0 and is terrible elsewhere, the posterior means win near θ0, and the sample mean has the smallest worst case. The right panel shows that in dimension d ≥ 3 the sample mean is even dominated everywhere by the James-Stein estimator.

Sample size N

Prior strength κ (posterior means)

Prior guess θ0

Range of θ* for the maximum risk [−R, R]

Prior on θ* for the Bayes risk N(0, τ²)

Dimension d (right panel)

Try these

Risks of the mean, the constant and the posterior means are exact formulas; the median's risk is the exact variance of the median of N standard normals (a table computed by numerical integration of the order-statistic density). The dots are Monte Carlo estimates from a seeded generator, a check that risk really is an average over data sets. In the right panel the James-Stein risk is d − (d−2)2 E[1/||X||2] with a Poisson mixture for the non-central chi-square.

In the lecture notes