Interactive demo · Probabilistic Machine Learning · Chapter 15

Kernel regression as attention

Nadaraya–Watson regression predicts at a query \(x\) the weighted average \(f(x)=\sum_i\alpha_i(x)\,y_i\) of the training labels, with weights \(\alpha_i=\operatorname{softmax}_i\big(-\tfrac12\beta^2(x-x_i)^2\big)\) (Equations 15.36 to 15.41). That is attention with the test points as queries, the training inputs as keys and the labels as values. The left panel is the attention matrix (rows: test points, columns: sorted training inputs): its diagonal band narrows as \(\beta\) grows. A small \(\beta\) averages almost everything (smooth, biased); a large \(\beta\) copies the nearest label (jagged, overfit). For the default sample the best bandwidth is not the book's \(\beta=1\) but about four times larger. The bottom panel shows the leave-one-out squared error, a way to choose \(\beta\) from the data alone. Move the probe point to see the weights one query uses.

Training points n

Noise standard deviation

Inverse bandwidth β = 1/σ

Probe point x

Try these

Data: \(x_i\sim U(0,5)\) (the first \(n\) of a fixed pool of 100, sorted), \(y_i=2\sin x_i+x_i^{0.8}+\sigma\varepsilon_i\) with fixed standard normal \(\varepsilon_i\); a simulated stand-in for the data of Figure 15.17. The error against the true curve is measured on 400 grid points of \([0,5]\). The leave-one-out score drops the diagonal weight: \(\hat f_{-i}(x_i)\) is the softmax average over the other \(n-1\) points; its minimum over a grid of 60 values of \(\beta\) between \(10^{-0.7}\) and \(10^{1.7}\) is marked.

In the lecture notes