An anchor x and a positive y are two Gaussians with correlation ρ; the K−1 negatives are independent draws of y. With the best possible critic (the log density ratio) the InfoNCE loss L gives the estimate log K − L of the mutual information I(X;Y) = −½ log(1 − ρ2): it is below the truth, below log K, and needs K well above eI to be close. On the right, the softmax over the K candidates that the gradient of the loss uses: a low temperature puts nearly all the push on the few hardest negatives.
200 anchors with their positives and up to 127 negatives are a fixed table of standard normal draws, so the estimates carry Monte Carlo error of about 0.05 nats. The estimate uses the critic log p(y|x)/p(y) at temperature 1; the temperature slider divides the critic by τ and affects the right panel and the loss reported below it.
code/ch32_representation_learning.py (infonce_bound, gauss_critic); data: data/ch32-infonce.json.