Mathematical foundations, demonstrations and code
A course that follows Christopher Bishop and Hugh Bishop's Deep Learning: Foundations and Concepts (Springer, 2024) chapter by chapter: probability and single-layer networks, deep networks, gradient descent and backpropagation, regularisation, convolutional networks, transformers and graph neural networks, sampling and latent variables, and the generative models behind modern AI: GANs, normalizing flows, autoencoders and diffusion models.
Lecture notes that fill the gaps. The notes follow the book section by section, with its section, equation and figure numbers. Wherever the book says that a result can be shown, is easy to see or is left as an exercise, the notes write out the whole argument in a "Filling the gap" box: 212 derivations in all, from the backpropagation recursion and initialisation variances to the equivariance of convolutions and message passing, the scaling of attention, the GAN optimum, the change of variables for flows and the diffusion evidence lower bound. All 352 exercises have worked solutions. Where we found a slip in the book or its solutions manual, the notes say so and show the evidence.
The book's figures, reproduced. The Python code rebuilds 286 figures with numpy and PyTorch on an ordinary CPU, one function per figure. Where the book shows the output of a very large model, the code shows the same effect at small scale, and the caption says so.
Demonstrations you can run. 65 browser demonstrations let you train small networks live, step through backpropagation and attention, sample from flows and diffusion models, and watch a GAN on a ring of Gaussians. The notes open each one at the case being discussed, and each links back to the notes.
Prerequisites. Linear algebra, multivariable calculus and elementary probability. The mathematical appendices collect the linear algebra, the calculus of variations and Lagrange multipliers the course uses, each with a proof.
The book. A free-to-read digital version of Bishop and Bishop's book is available from its website. Keep it open next to the notes: section, equation, figure and exercise numbers match.
Companion course. Several chapters overlap the Bayesian and Probabilistic Machine Learning course, which follows Bishop's 2006 book. Each notes page lists the overlapping sections and links them, so a result proved in full there is not repeated here.
Polynomial curve fitting, error functions, over-fitting and regularisation, model selection, and a short history of neural networks from the perceptron to deep learning.
7 figures reproduced · 7 gap-filling derivations
The rules of probability and Bayes' theorem, densities and the change of variables, the Gaussian and its maximum likelihood estimates, and information theory: entropy, KL divergence and mutual information.
14 figures reproduced · 20 gap-filling derivations · 41 exercises with solutions
Discrete distributions, the multivariate Gaussian (conditionals, marginals, linear-Gaussian models), mixtures, periodic variables, the exponential family and nonparametric density estimation.
16 figures reproduced · 12 gap-filling derivations · 38 exercises with solutions
Linear regression with basis functions, least squares as projection, sequential learning, regularised least squares, decision theory for regression and the bias-variance decomposition.
6 figures reproduced · 7 gap-filling derivations · 12 exercises with solutions
Discriminant functions, decision theory and ROC curves, generative classifiers with Gaussian classes, logistic and softmax regression, probit regression and canonical links.
14 figures reproduced · 11 gap-filling derivations · 24 exercises with solutions
Limitations of fixed basis functions, multilayer networks and activation functions, universal approximation, depth and linear regions, error functions and mixture density networks.
18 figures reproduced · 13 gap-filling derivations · 21 exercises with solutions
Error surfaces and the Hessian, gradient descent and its convergence, stochastic and mini-batch gradients, initialisation, momentum, Adam, and batch and layer normalisation.
14 figures reproduced · 10 gap-filling derivations · 14 exercises with solutions
Backpropagation for general networks, numerical differentiation, the Jacobian and the Hessian, and automatic differentiation in forward and reverse mode.
7 figures reproduced · 10 gap-filling derivations · 18 exercises with solutions
Inductive bias, invariance and data augmentation, weight decay and L1 sparsity, early stopping, parameter sharing, residual connections, model averaging and dropout.
17 figures reproduced · 11 gap-filling derivations · 18 exercises with solutions
Convolution and its equivariance, padding, stride and pooling, multi-channel layers, the backward pass, visualising trained networks, adversarial examples and detection.
19 figures reproduced · 11 gap-filling derivations · 13 exercises with solutions
Directed graphical models, conditional independence and d-separation, explaining away, naive Bayes, linear-Gaussian models, and Markov and autoregressive models of sequences.
13 figures reproduced · 9 gap-filling derivations · 19 exercises with solutions
Attention, self-attention and multi-head attention, positional encodings, the transformer layer, tokenisation and language models, sampling strategies and vision transformers.
15 figures reproduced · 10 gap-filling derivations · 16 exercises with solutions
Graphs and permutation symmetry, message passing, graph convolutional networks, node, edge and graph tasks, over-smoothing, graph attention and geometric deep learning.
11 figures reproduced · 11 gap-filling derivations · 10 exercises with solutions
Monte Carlo estimation, rejection and importance sampling, Markov chain Monte Carlo, Langevin dynamics, energy-based models and gradient estimators for expectations.
19 figures reproduced · 9 gap-filling derivations · 18 exercises with solutions
K-means and vector quantisation, Gaussian mixtures and the EM algorithm, Bernoulli mixtures, and the evidence lower bound that later chapters build on.
16 figures reproduced · 7 gap-filling derivations · 24 exercises with solutions
Principal component analysis, probabilistic PCA, factor analysis, independent component analysis, Kalman filters and nonlinear latent variable models.
19 figures reproduced · 13 gap-filling derivations · 26 exercises with solutions
The adversarial game, the optimal discriminator and the Jensen-Shannon divergence, training difficulties and mode collapse, Wasserstein GANs, conditional GANs and CycleGAN.
13 figures reproduced · 10 gap-filling derivations · 3 exercises with solutions
The change of variables for invertible networks, coupling flows, masked autoregressive and inverse autoregressive flows, and continuous flows with neural ODEs.
8 figures reproduced · 8 gap-filling derivations · 11 exercises with solutions
Deterministic, sparse, denoising and masked autoencoders, the link to PCA and to the score, and variational autoencoders with the reparameterisation trick.
14 figures reproduced · 8 gap-filling derivations · 6 exercises with solutions
The forward noising process, the reverse process and its evidence lower bound, noise prediction and score matching, stochastic differential equations and guidance.
16 figures reproduced · 8 gap-filling derivations · 20 exercises with solutions
Linear algebra (determinants, inverses, eigenvectors, matrix derivatives), the calculus of variations (functional derivatives and the Euler-Lagrange equation) and Lagrange multipliers with the KKT conditions.
10 figures reproduced · 7 gap-filling derivations
One module per chapter and one function per figure, with readable numpy and PyTorch implementations that run on a CPU in seconds to minutes.
The same format for Bishop's Pattern Recognition and Machine Learning (2006), with a module on deep generative models. See also Statistical Methods for Modern Biology.