Interactive demo · Foundations of Deep Learning · Chapter 8

The Hessian, the outer-product approximation and Hessian-vector products

For the sum-of-squares error the Hessian splits as H = Σn ∇yn∇ynT + Σn (yn − tn)∇∇yn (8.39). Dropping the second sum gives the outer-product (Levenberg–Marquardt, or Gauss–Newton) approximation HOP (8.40), which needs only first derivatives and is never indefinite. Train a 1-M-1 tanh network on t = sin(πx) + noise and watch when the dropped term matters. The exact H is assembled from W Hessian-vector products, each one forward and one backward pass of the R-operator.

Hidden units M (W = 3M + 1)

Data points N

Target noise σ

Adam steps (η = 0.01)

Try these

Data: xn evenly spaced on [−1, 1], noise from a fixed seed. Weights per hidden unit (bias, input weight), then the output bias and weights; heat maps use this order and a cube-root colour scale (red positive, blue negative). Eigenvalues below 10−12 in size are not resolved.

In the lecture notes