Interactive demo · Foundations of Deep Learning · Chapter 7

Optimisers on a two-dimensional error surface

Each optimiser of Chapter 7 is run on an error function E(w) of two weights, so the whole path through weight space can be seen. The left panel shows the contours of E with the path (click anywhere on it to choose a new starting point); the right panel shows E − Emin against the step on a log scale. Gradient noise of standard deviation σ stands in for the error of a mini-batch gradient.

Error surface

Valley: condition number κ and angle θ

Optimiser

Learning rate η

Momentum μ

Decay rates β₁, β₂

β₁

Gradient noise σ and schedule

Steps

Try these

Updates: gradient descent (7.16); momentum (7.31) Δw = −η∇E + μΔw; Nesterov (7.34) evaluates the gradient at w + μΔw; AdaGrad (7.39)–(7.40); RMSProp (7.41)–(7.42) with β = β₂; Adam (7.43)–(7.47) with bias correction; δ = 10−8. Schedules: linear (7.36) to η/20 over 200 steps, power law (7.37) η(1 + τ/20)−1, exponential (7.38) η·0.5τ/50. Noise is seeded, so a class sees the same paths.

In the lecture notes