Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 19

Label shift and blackbox shift estimation

Three classes with one-dimensional Gaussian class-conditionals p(x | y=k) = N(μk, 1), μ = (−d, 0, d). A classifier is trained on source data whose class prior is p = (0.45, 0.45, 0.10); it is the Bayes rule under p, so its decision boundaries (grey dashed) are known exactly. At test time only the class prior has moved to q (the label-shift assumption), and the target labels are hidden. The classifier's predicted-class fractions μ = Cq reveal q through the confusion matrix C measured on the source: q = C−1μ. Estimate μ from n unlabelled target points, invert, and correct the classifier by (19.23): its boundaries move (solid) towards the new rare classes.

Class separation d

Target prior of class 1 q₁

Target prior of class 3 q₃

Unlabelled target points n

Try these

The confusion matrix, μ = Cq, the asymptotic standard deviations and the accuracies are exact (integrals of Gaussians over intervals); the "repeated" values come from 300 independent draws of n target points (the confusion matrix is taken as known, so the spread is the pure sampling error of the predicted-class fractions). Class 2 is the middle class; the middle class is never predicted if the boundaries cross.

In the lecture notes