Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 32

The InfoNCE bound, negatives and temperature

An anchor x and a positive y are two Gaussians with correlation ρ; the K−1 negatives are independent draws of y. With the best possible critic (the log density ratio) the InfoNCE loss L gives the estimate log K − L of the mutual information I(X;Y) = −½ log(1 − ρ2): it is below the truth, below log K, and needs K well above eI to be close. On the right, the softmax over the K candidates that the gradient of the loss uses: a low temperature puts nearly all the push on the few hardest negatives.

Pair and batch

Try these

200 anchors with their positives and up to 127 negatives are a fixed table of standard normal draws, so the estimates carry Monte Carlo error of about 0.05 nats. The estimate uses the critic log p(y|x)/p(y) at temperature 1; the temperature slider divides the critic by τ and affects the right panel and the loss reported below it.

In the lecture notes