A pool of 300 unlabeled points, two of them labeled to start. A Bayesian logistic regression (Laplace posterior on RBF features) is refitted after every query. Three rules choose the next point: random; maximum entropy, the point whose predicted label is most uncertain; and BALD, the point whose label would tell us most about the parameters, the predictive entropy minus the average entropy given the parameters (19.51). Left: the predictive probability, the pool (dot size grows with the chosen rule's score) and the next query (gold ring). Right: test accuracy of the three rules from the same start. In the second data set a patch of points has labels that are pure coin flips, which look uncertain to every model but carry no information about the boundary.
Data: two moons (noise 0.25), 300 pool points and 1000 test points, simulated. The second pool has 200 moons points (noise 0.12) plus 100 points in a Gaussian patch around (1, 0.25) with random labels; its test set is 1000 clean moons points. Model: logistic regression on a fixed 7 by 5 grid of Gaussian features, prior N(0, I), Laplace posterior; predictive probability sigmoid(m / sqrt(1 + πs2/8)) where m and s2 are the mean and variance of the logit. BALD uses a closed-form approximation of the expected entropy (within 0.03 nats of Monte Carlo). “Random” follows a fixed random order of the pool for each starting pair.
code/ch19_learning_with_fewer_labels.py (laplace_logreg, bald_closed_form, active_run).