Take one direction of the data with variance s and two augmented views that add independent noise of variance σ2. The encoder is a number w, the predictor a number p, and the loss is E(p w x1 − target)2. The constant solution w = 0 has zero loss, so the question is whether gradient flow goes there. With a stop-gradient and a predictor the quantity w2 − p2 never changes and p settles at b/a, so the encoder collapses only if p02 ≥ w02 + (b/a)2. Without the predictor, or without the stop-gradient, it always collapses.
Gradient flow integrated by the classical fourth-order Runge–Kutta method, step 0.02, up to time 30. a = s + σ² is the variance of a view and b = s the covariance of the two views. η matters only for the moving-average variants.
code/ch32_representation_learning.py (byol_rhs, byol_run). The model is a deliberately minimal cousin of the linear analysis the book cites, not a reproduction of it.