Mathematical foundations, demonstrations and code
A course that follows Larry Wasserman's All of Nonparametric Statistics (Springer, 2006) chapter by chapter: the empirical cdf and statistical functionals, the bootstrap and the jackknife, smoothing, nonparametric regression and density estimation, normal means and minimax theory, orthogonal series, wavelets and adaptive methods, and topics from measurement error to the Dirichlet process.
Lecture notes that prove what the book states. The book covers a great deal of modern theory in a short space and quotes many results from the research literature. The notes follow it section by section, with its numbering, and supply the argument in a "Filling the gap" box: 165 derivations in all, from the DKW and Vapnik-Chervonenkis inequalities to bootstrap accuracy, the tube formula for confidence bands, optimal bandwidths, Pinsker's theorem, Stein's unbiased risk estimate, oracle inequalities and wavelet thresholding. All 124 exercises have worked solutions. Where we found a slip in the book, the notes say so and show the evidence.
The book's figures, reproduced. The Python code rebuilds 119 figures from the book's own data sets (the cosmic microwave background, LIDAR, galaxy redshifts, motorcycle and glass data, among others) and from the simulations the book uses, one function per figure.
Demonstrations you can run. 31 browser demonstrations let you change a bandwidth, a number of basis functions, a threshold, a noise level or a sample size and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.
Prerequisites. A first graduate course in mathematical statistics, such as the companion course Probability and Statistical Inference, which follows the same author's All of Statistics and is the natural first course before this one.
The book. The book's data sets are on its website. Keep the book open next to the notes: section, equation, figure and exercise numbers match.
Companion courses. Each notes page lists the overlapping sections of Probability and Statistical Inference, Statistical Learning: Theory and Methods, Bayesian and Probabilistic Machine Learning and the other courses on this site, and links them.
What nonparametric inference means, the modes of convergence, the delta method, pointwise, uniform and finite-sample confidence sets, and the inequalities of Hoeffding, Mill, Bernstein and Berry-Esseen.
5 figures reproduced · 15 gap-filling derivations · 5 exercises with solutions
The empirical cdf, Glivenko-Cantelli and the DKW band, plug-in estimators of statistical functionals, influence functions and the nonparametric delta method, and the Vapnik-Chervonenkis inequality.
5 figures reproduced · 11 gap-filling derivations · 15 exercises with solutions
The jackknife estimates of bias and variance, the bootstrap and the parametric bootstrap, the normal, pivotal, percentile and studentized intervals, and Edgeworth arguments for their accuracy.
6 figures reproduced · 15 gap-filling derivations · 13 exercises with solutions
The bias-variance trade-off, kernels, the choice of loss, pointwise and simultaneous confidence sets, cross-validation, and the curse of dimensionality.
8 figures reproduced · 8 gap-filling derivations · 12 exercises with solutions
Linear smoothers, choosing the smoothing parameter, local polynomial regression, smoothing and regression splines, variance estimation, confidence bands by the tube formula, local likelihood, and additive, tree and projection pursuit models.
31 figures reproduced · 29 gap-filling derivations · 24 exercises with solutions
Histograms and kernel density estimators: risk, optimal bandwidths and rates, cross-validation, confidence bands, local polynomials, multivariate problems, and density estimation by regression.
14 figures reproduced · 16 gap-filling derivations · 10 exercises with solutions
The normal means model, Sobolev spaces, Stein's unbiased risk estimate, Pinsker's minimax theorem, James-Stein shrinkage, adaptive estimation, confidence balls and the oracle inequalities behind sparsity.
9 figures reproduced · 18 gap-filling derivations · 15 exercises with solutions
Orthogonal series for regression and density estimation: the cosine basis, risk estimation and modulators, confidence balls and bands, irregular designs, and tensor products.
12 figures reproduced · 13 gap-filling derivations · 14 exercises with solutions
Haar and Daubechies wavelets, multiresolution analysis, wavelet regression, hard, soft and SURE thresholding, Besov spaces, confidence sets, and other adaptive methods such as intersecting confidence intervals.
17 figures reproduced · 22 gap-filling derivations · 9 exercises with solutions
Measurement error and SIMEX, deconvolution and inverse problems, the Dirichlet process, semiparametric inference, correlated errors, classification, sieves, shape-restricted inference, testing and computation.
12 figures reproduced · 18 gap-filling derivations · 7 exercises with solutions
One module per chapter and one function per figure, with readable implementations of the jackknife and bootstrap, local polynomial and spline smoothers, kernel density estimators, SURE, wavelet transforms and thresholding.
The same author's introductory treatment of the bootstrap, curve estimation and orthogonal functions. See also Statistical Learning: Theory and Methods for splines and kernel smoothing.