Course · Lecture notes, demonstrations and code

Nonparametric Statistical Inference

Mathematical foundations, demonstrations and code

A course that follows Larry Wasserman's All of Nonparametric Statistics (Springer, 2006) chapter by chapter: the empirical cdf and statistical functionals, the bootstrap and the jackknife, smoothing, nonparametric regression and density estimation, normal means and minimax theory, orthogonal series, wavelets and adaptive methods, and topics from measurement error to the Dirichlet process.

TrungTin Nguyen · Teaching page · Homepage

What each chapter offers

Lecture notes that prove what the book states. The book covers a great deal of modern theory in a short space and quotes many results from the research literature. The notes follow it section by section, with its numbering, and supply the argument in a "Filling the gap" box: 165 derivations in all, from the DKW and Vapnik-Chervonenkis inequalities to bootstrap accuracy, the tube formula for confidence bands, optimal bandwidths, Pinsker's theorem, Stein's unbiased risk estimate, oracle inequalities and wavelet thresholding. All 124 exercises have worked solutions. Where we found a slip in the book, the notes say so and show the evidence.

The book's figures, reproduced. The Python code rebuilds 119 figures from the book's own data sets (the cosmic microwave background, LIDAR, galaxy redshifts, motorcycle and glass data, among others) and from the simulations the book uses, one function per figure.

Demonstrations you can run. 31 browser demonstrations let you change a bandwidth, a number of basis functions, a threshold, a noise level or a sample size and see the effect at once. The notes open each one at the case being discussed, and each links back to the notes.

Before you start

Prerequisites. A first graduate course in mathematical statistics, such as the companion course Probability and Statistical Inference, which follows the same author's All of Statistics and is the natural first course before this one.

The book. The book's data sets are on its website. Keep the book open next to the notes: section, equation, figure and exercise numbers match.

Companion courses. Each notes page lists the overlapping sections of Probability and Statistical Inference, Statistical Learning: Theory and Methods, Bayesian and Probabilistic Machine Learning and the other courses on this site, and links them.

Chapters

Chapter 1

Introduction

What nonparametric inference means, the modes of convergence, the delta method, pointwise, uniform and finite-sample confidence sets, and the inequalities of Hoeffding, Mill, Bernstein and Berry-Esseen.

5 figures reproduced · 15 gap-filling derivations · 5 exercises with solutions

Chapter 2

Estimating the cdf and Statistical Functionals

The empirical cdf, Glivenko-Cantelli and the DKW band, plug-in estimators of statistical functionals, influence functions and the nonparametric delta method, and the Vapnik-Chervonenkis inequality.

5 figures reproduced · 11 gap-filling derivations · 15 exercises with solutions

Chapter 3

The Bootstrap and the Jackknife

The jackknife estimates of bias and variance, the bootstrap and the parametric bootstrap, the normal, pivotal, percentile and studentized intervals, and Edgeworth arguments for their accuracy.

6 figures reproduced · 15 gap-filling derivations · 13 exercises with solutions

Chapter 4

Smoothing: General Concepts

The bias-variance trade-off, kernels, the choice of loss, pointwise and simultaneous confidence sets, cross-validation, and the curse of dimensionality.

8 figures reproduced · 8 gap-filling derivations · 12 exercises with solutions

Chapter 5

Nonparametric Regression

Linear smoothers, choosing the smoothing parameter, local polynomial regression, smoothing and regression splines, variance estimation, confidence bands by the tube formula, local likelihood, and additive, tree and projection pursuit models.

31 figures reproduced · 29 gap-filling derivations · 24 exercises with solutions

Chapter 6

Density Estimation

Histograms and kernel density estimators: risk, optimal bandwidths and rates, cross-validation, confidence bands, local polynomials, multivariate problems, and density estimation by regression.

14 figures reproduced · 16 gap-filling derivations · 10 exercises with solutions

Chapter 7

Normal Means and Minimax Theory

The normal means model, Sobolev spaces, Stein's unbiased risk estimate, Pinsker's minimax theorem, James-Stein shrinkage, adaptive estimation, confidence balls and the oracle inequalities behind sparsity.

9 figures reproduced · 18 gap-filling derivations · 15 exercises with solutions

Chapter 8

Nonparametric Inference Using Orthogonal Functions

Orthogonal series for regression and density estimation: the cosine basis, risk estimation and modulators, confidence balls and bands, irregular designs, and tensor products.

12 figures reproduced · 13 gap-filling derivations · 14 exercises with solutions

Chapter 9

Wavelets and Other Adaptive Methods

Haar and Daubechies wavelets, multiresolution analysis, wavelet regression, hard, soft and SURE thresholding, Besov spaces, confidence sets, and other adaptive methods such as intersecting confidence intervals.

17 figures reproduced · 22 gap-filling derivations · 9 exercises with solutions

Chapter 10

Other Topics

Measurement error and SIMEX, deconvolution and inverse problems, the Dirichlet process, semiparametric inference, correlated errors, classification, sieves, shape-restricted inference, testing and computation.

12 figures reproduced · 18 gap-filling derivations · 7 exercises with solutions

Shared resources

Python

Code that reproduces the figures

One module per chapter and one function per figure, with readable implementations of the jackknife and bootstrap, local polynomial and spline smoothers, kernel density estimators, SURE, wavelet transforms and thresholding.

Companion courses

Probability and Statistical Inference

The same author's introductory treatment of the bootstrap, curve estimation and orthogonal functions. See also Statistical Learning: Theory and Methods for splines and kernel smoothing.

Using the material in class

  1. Before the lecture, students read a section of the book and the matching section of the notes, with the derivations closed.
  2. In the lecture, state a result, ask the class to predict what it implies, then open the demonstration at the linked preset and test the prediction.
  3. In the tutorial, open the derivations one at a time and work through the exercises at the end of each chapter.
  4. For assignments, students change one setting in the Python code (a bandwidth, a number of basis functions, a threshold, a noise level) and explain what changes in the figure.