Interactive demo · Probabilistic Machine Learning · Chapter 5

Thresholds, costs, ROC and precision-recall curves

A classifier outputs a probability p = p(y=1 | x). Which threshold to use is a decision problem: the Bayes-optimal rule predicts 1 when p ≥ 1/(1+c), where c is the cost of a false negative divided by the cost of a false positive. The ROC curve and the precision-recall curve show every threshold at once; one point on each curve is the threshold you have chosen. The population is either a binormal model (scores N(0,1) for negatives and N(d,1) for positives, so the posterior is exactly known) or the held-out patients of a real classifier.

Population

Separation d (binormal)

Prevalence of positives (binormal)

Cost ratio c = cost(FN) / cost(FP)

Threshold on p(y=1|x)

Reject option (Chow's rule, error cost 1)

Try these

Everything is computed exactly on a fine grid of scores (binormal) or from the 190 test patients (real data); the posterior of the binormal model is p = sigma(logit(prevalence) + d x - d2/2). The curves are traced by sweeping the threshold over all values of p.

In the lecture notes