Interactive demo · Bayesian and Probabilistic Machine Learning · Chapter 7

Support vector machines: maximum margin, soft margin and the ε-tube

A support vector machine is a kernel model y(x) = ∑_n a_n t_n k(x, x_n) + b whose coefficients are found from a convex dual problem, and most a_n come out exactly zero. The points with a_n > 0 are the support vectors: they alone define the boundary. The solver here is a small sequential minimal optimisation (SMO), the same algorithm as in the Python code. Click on the plot to add or remove points and watch the solution change. In regression, errors smaller than ε are free, so points inside the tube are not support vectors.

Problem

Data

Kernel

Kernel width γ in exp(−γ‖x−x′‖²)

Slack penalty C

hard margin (C = ∞)

Tube half-width ε

Click on the plot to

Try these

Classification: red is t = +1 and blue t = −1; the solid line is y(x) = 0 and the dashed lines the margins y(x) = ±1. Gold rings: support vectors on the margin (0 < a < C); violet rings: bounded support vectors (a = C, inside the margin or misclassified). Data are simulated; the overlapping classes are a stand-in for the synthetic set of Bishop (2006), Appendix A.

In the lecture notes

Chapter 7 notes: Section 7.1 (maximum margin classifiers and the dual), Section 7.1.1 (overlapping classes, the soft margin and C), Section 7.1.2 (relation to logistic regression, the hinge error), Section 7.1.4 (regression and the ε-insensitive error).
Python code: code/ch07_sparse_kernel_machines.py (functions smo_solve, svc_fit, svr_fit; Figures 7.1, 7.2, 7.4, 7.5, 7.6, 7.8) and the checks code/checks_ch07.py. The relevance vector machine, a sparser Bayesian relative, is in the next demo.