Each optimiser of Chapter 7 is run on an error function E(w) of two weights, so the whole path through weight space can be seen. The left panel shows the contours of E with the path (click anywhere on it to choose a new starting point); the right panel shows E − Emin against the step on a log scale. Gradient noise of standard deviation σ stands in for the error of a mini-batch gradient.
Updates: gradient descent (7.16); momentum (7.31) Δw = −η∇E + μΔw; Nesterov (7.34) evaluates the gradient at w + μΔw; AdaGrad (7.39)–(7.40); RMSProp (7.41)–(7.42) with β = β₂; Adam (7.43)–(7.47) with bias correction; δ = 10−8. Schedules: linear (7.36) to η/20 over 200 steps, power law (7.37) η(1 + τ/20)−1, exponential (7.38) η·0.5τ/50. Noise is seeded, so a class sees the same paths.
code/ch07_gradient_descent.py (SURFACES, opt_update, optimise_2d, Figures 7.1, 7.3, 7.6). The paths drawn here are the paths of optimise_2d, number for number.