Mathematical foundations, demonstrations and code
A course that follows Christopher Bishop's Pattern Recognition and Machine Learning (Springer, 2006) chapter by chapter. It covers the whole book, from the first chapters on regression, classification, neural networks and kernel methods to graphical models and belief propagation, mixture models and the EM algorithm, approximate (variational) inference and expectation propagation, Monte Carlo sampling and Markov chain Monte Carlo, continuous latent variables from PCA to kernel PCA, hidden Markov models and Kalman filters, and ways of combining models from boosting to mixtures of experts.
Lecture notes that fill the gaps. The notes follow the book section by section and use its equation numbers. Wherever the book says that a result "can be shown", is "easily verified" or is left to an exercise, the notes write out the whole argument, hidden until you open it.
Every numerical figure, reproduced. The Python code rebuilds the chapter's numerical experiments from scratch with numpy, scipy and matplotlib. The notes show these reproductions, and each module comes with numerical checks of the derivations.
Demonstrations tied to results. Thirty-nine browser demonstrations let you run the algorithms step by step. The notes open each one already set to the case being discussed, and each demonstration links back to the notes.
Prerequisites. Linear algebra (eigenvectors, determinants, positive definite matrices), multivariable calculus, and elementary probability. The course is self-contained and builds on itself: it starts at Chapter 1, and each chapter lists the earlier chapters and sections it uses, as links. Readers who already know the material can start at any chapter. The identities used most often are proved once in the mathematical toolkit.
The book. Bishop's book is available as a free PDF from Microsoft Research. Keep it open next to the notes: section, equation, figure and exercise numbers match.
Polynomial curve fitting and over-fitting, probability theory, the curse of dimensionality, decision theory and information theory.
Binomial, beta, multinomial, Dirichlet and Gaussian distributions, conjugate Bayesian inference, the exponential family and nonparametric density estimation.
Least squares and ridge regression, the bias-variance decomposition, Bayesian linear regression, the equivalent kernel and the evidence approximation.
Discriminant functions, Fisher's discriminant, the perceptron, generative and discriminative models, logistic regression with IRLS, the Laplace approximation and Bayesian logistic regression.
Network functions, training, backpropagation and the Hessian, regularisation, mixture density networks and Bayesian neural networks.
Dual representations, constructing kernels, radial basis function networks and Gaussian processes for regression and classification.
Maximum margin classifiers, support vector machines for classification and regression, and the relevance vector machine.
Bayesian networks and Markov random fields, conditional independence and d-separation, factor graphs, the sum-product and max-sum algorithms, and loopy belief propagation.
K-means, Gaussian mixtures by maximum likelihood, the EM algorithm in general and why it never decreases the likelihood, Bernoulli mixtures, EM for Bayesian linear regression.
Variational inference and mean field, the variational mixture of Gaussians, variational linear and logistic regression, local variational bounds, and expectation propagation.
Transformation and rejection sampling, importance sampling and SIR, Monte Carlo EM, Markov chains, Metropolis–Hastings, Gibbs sampling, slice sampling, Hamiltonian (hybrid) Monte Carlo, and estimating partition functions.
PCA by maximum variance and minimum error, probabilistic PCA and its maximum-likelihood solution, EM for PCA, Bayesian PCA, factor analysis, kernel PCA, and nonlinear latent variable models.
Markov models, hidden Markov models with forward-backward, scaling, Viterbi and Baum-Welch, linear dynamical systems with the Kalman filter and smoother, and particle filters.
Bayesian model averaging versus combination, committees and bagging, AdaBoost as stagewise minimisation of the exponential error, decision trees, and conditional mixtures up to mixtures of experts.
Jensen's inequality, KL divergence, Lagrange multipliers, matrix derivatives, the Woodbury identity, Gaussian identities, exponential families, and the Gamma, Dirichlet, Wishart and Student t distributions, each with a proof.
One module per chapter, one function per figure, readable implementations of every algorithm, and numerical checks of the derivations. Runs with numpy, scipy and matplotlib.