Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 6

Kernel functions: construction, Nadaraya-Watson and the dual form

A kernel k(x, x′) is an inner product of feature vectors, so it never needs the features themselves. Build one from basis functions (Figure 6.1), combine kernels by the rules of Section 6.2 and watch the Gram matrix stay positive semidefinite (or fail to), average neighbouring targets with Nadaraya-Watson weights that sum to one (Figure 6.3), solve ridge regression in its dual form, and make a kernel from a generative model.

Try these

Data: the sinusoid sin(2πx) plus Gaussian noise, as in Appendix A of Bishop (2006), generated in the browser with a seeded generator. The kernels in the build view use a Gaussian of sd 0.3, (xx′+1)², xx′, exp(−3|x−x′|) and tanh(3xx′−1).

In the lecture notes

Chapter 6 notes: Section 6.1 (dual representations, ridge regression), Section 6.2 (constructing kernels: from basis functions, the closure rules, generative models), Section 6.3 (radial basis functions and the Nadaraya-Watson model).
Python code: code/ch06_kernel_methods.py (functions basis_kernel, k_poly, k_gauss, k_sigmoid, ridge_dual, nadaraya_watson; Figures 6.1 to 6.3) and code/checks_ch06.py (dual equals primal, positive semidefinite Gram matrices, closure rules, weights sum to one).