Interactive demo · Nonparametric Statistical Inference · Chapter 4

The bias–variance trade-off at a point

To estimate a density f at 0, count the number X of observations in the window (−h/2, h/2) and set f̂n(0) = X/(nh). Since X is Binomial(n, ph), the bias, variance and mean squared error are known exactly. A wide window averages f over a region where it changes (bias); a narrow one catches few points (variance). For a smooth f the mse is about Ah4 + B/(nh), so the best width shrinks like n−1/5 and the best mse like n−4/5. A density with a cusp at 0 breaks this.

Density f

Sample size n

Window width h

Try these

Densities: the standard Normal; the bimodal mixture ½N(−1.5, 1) + ½N(1.5, 1), which has a trough at 0 (f″(0) > 0, so the bias is positive); and the Laplace density ½e−|x|, which has a cusp at 0, so the Taylor expansion behind Ah4 fails: its bias is about −h/8 and the approximation becomes h2/64 + 1/(2nh). The width runs from 0.001 to 5 on a log scale. The optimal h minimises the exact mse (golden-section search on log h).

In the lecture notes