Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 20

Judging a generative model: likelihood, FID, precision and recall

Grey dots are held-out data, red dots are samples drawn from the fitted model, and the shading is the model density. Choose a model family and its complexity and read three kinds of score. The held-out negative log-likelihood punishes a model that puts little mass where new data fall, even if its samples look perfect (a kernel estimator with a tiny bandwidth memorises the training set). The Fréchet distance compares only the mean and covariance of the two clouds (here in the raw plane, standing in for the features of a pretrained network), so a single Gaussian can score almost perfectly while its samples fill the gap between the moons. Precision (are samples near real data?) and recall (is real data near samples?) separate quality from coverage.

Data

Model family

Mixture components K

Kernel bandwidth h

Shading

Try these

Data: 400 training and 400 held-out points. The Gaussian mixture is fitted by EM from a fixed start (full covariances, a small ridge 10−4); the kernel estimator is the Parzen estimator with an isotropic Gaussian kernel. Every score is computed against the 400 held-out points with 400 model samples; Fréchet distance = |Δμ|2 + tr(Σ1 + Σ2 − 2(Σ1Σ2)1/2); precision and recall use the k = 3 nearest-neighbour balls.

In the lecture notes