Near a minimum the error is a quadratic, E = ½(λminα12 + λmaxα22) in the eigen-coordinates αi = uiT(w − w*) of the Hessian (7.11). Gradient descent multiplies αi by 1 − ηλi at every step (7.27). Here λmax = 1 and λmin = 1/κ, so the condition number κ sets how elongated the bowl is. Watch which direction limits the step size and which one limits the speed.
Heavy ball is momentum (7.31): αt+1 = (1 + μ − ηλ)αt − μαt−1. Nesterov (7.34): αt+1 = (1 + μ)(1 − ηλ)αt − μ(1 − ηλ)αt−1. Their contraction per step is the larger modulus of the roots of the matching quadratic; the dashed grey line in the lower panel is gradient descent with its best fixed step, for comparison. Start: α = (1, 1).
code/ch07_gradient_descent.py (quad_eigen_path, spectral_radius, optimal_gd, optimal_heavy_ball, supplementary Figure 7.S2); the paths here equal quad_eigen_path, number for number.