Interactive demo · Probabilistic Machine Learning · Chapter 16

Kernel density estimation and kernel regression

The Parzen window estimator places one scaled copy of a density kernel on every data point and averages them (16.33): the bandwidth h sets the width of the copies and so the complexity of the estimate. The same idea gives regression: the Nadaraya-Watson estimator (16.39) is a weighted average of the observed responses with weights given by the kernel, and a locally weighted straight line (16.47) removes its bias at the edges of the data. Switch between the two modes, change the kernel and the bandwidth, and watch the estimate move from spiky to oversmoothed. The rule-of-thumb bandwidth h = σ(4/(3N))1/5 of Section 16.3.3 (with σ from the median absolute deviation) is marked.

What to estimate

Data

Sample size N

Kernel

Bandwidth h

Estimator

Kernel copies

Spread (16.45)

Try these

Six-point data: the standard example of the Parzen estimator, as in Figure 16.9. Sample data: a simulated bimodal sample (40 percent N(−1, 0.42), 60 percent N(1.2, 0.72)), dashed curve = its true density. Regression data: 200 simulated points with the true mean 0.9 exp(−1.5x2) + 0.1 sin(2.5x) and noise whose standard deviation grows towards the edges; the page uses the first N. The kernels are the four of Table 16.1; the bandwidth is the scale factor in Kh(u) = K(u/h)/h.

In the lecture notes