Interactive demo · Probability and Statistical Inference · Chapter 6

Mean squared error = bias² + variance: the price and the reward of shrinking

An estimator that is pulled towards a fixed value (shrunk) is biased, but its variance is smaller. Theorem 6.9 says that the mean squared error E(θ̂ − θ)² is exactly the squared bias plus the variance, so a little bias can buy a large reduction in error. Choose a family of estimators and its tuning value: panel (a) shows the three exact curves against the tuning value, panel (b) the sampling distribution of the estimator from simulated data sets, and panel (c) the mean squared error at every value of the true parameter, which shows why no estimator is best everywhere.

Model and estimator

Sample size n

True parameter

Tuning value

Simulated data sets R

Try these

Normal model: Xi ~ N(θ, 1), estimator c X̄n. Bernoulli model: estimator (S + a)/(n + 2a), which adds a imaginary successes and a imaginary failures. Uniform model: Xi ~ Uniform(0, θ), estimator c maxi Xi. Exact values use the closed forms in the notes; the simulation uses the seeded generator (seed in the readout), and its variance divides by R.

In the lecture notes