Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 2

Density estimation: histograms, kernels and nearest neighbours

Fifty points drawn from a two-peaked density (green curve) are all we see. A histogram (bin width Δ), a kernel estimate (bandwidth h) and a K-nearest-neighbour estimate each need one smoothing parameter: too small gives a noisy model, too large washes out the peaks. The right-hand plot shows the average log likelihood of 2000 held-out points, which picks out the best value. In the last mode the same ideas classify two Gaussian classes.

Method

Data

Try these

Data are seeded and nested: raising N adds points to the same sample. The K-nearest-neighbour density is not a normalised density (its tails decay like 1/|x|); the readout gives its integral over a wide interval.

In the lecture notes

Chapter 2 notes: Section 2.5 (nonparametric methods), Section 2.5.1 (kernel density estimators), Section 2.5.2 (nearest-neighbour methods and the Cover-Hart bound) and Section 2.5.3 (the K-nearest-neighbour classifier).
Python code: code/ch02_probability_distributions.py (functions histogram_density, kde_gaussian, knn_density, knn_classify; Figures 2.24 to 2.28) and the checks in code/checks_ch02.py. The demo's estimates equal the Python ones on the same data.