Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 32

Why BYOL and SimSiam need not collapse

Take one direction of the data with variance s and two augmented views that add independent noise of variance σ2. The encoder is a number w, the predictor a number p, and the loss is E(p w x1 − target)2. The constant solution w = 0 has zero loss, so the question is whether gradient flow goes there. With a stop-gradient and a predictor the quantity w2 − p2 never changes and p settles at b/a, so the encoder collapses only if p02 ≥ w02 + (b/a)2. Without the predictor, or without the stop-gradient, it always collapses.

Variant

Start and data

Try these

Gradient flow integrated by the classical fourth-order Runge–Kutta method, step 0.02, up to time 30. a = s + σ² is the variance of a view and b = s the covariance of the two views. η matters only for the moving-average variants.

In the lecture notes