The natural gradient \(\bm F^{-1}\bm g\) measures steps by the Kullback–Leibler divergence between the distributions they induce, not by the Euclidean length of the parameter change. Two problems show what that buys. In the coupled Gaussian of Equation (6.91) the Fisher matrix has condition number about 6900, so steepest descent crawls along the valley floor while natural gradient runs straight to the optimum. In the iris Gaussian the loss is the mean negative log-likelihood of a Gaussian fitted to the 50 versicolor sepal lengths; the same fit is parameterised by \((\mu,\sigma)\) or by \((\mu,\log\sigma)\). Steepest descent follows a different path in each parameterisation; natural gradient follows (almost) the same path in both.
Coupled Gaussian: \(p(\bm x;\bm\theta)=\mathcal N(\bm x\,|\,(3\theta_1+\theta_2/3,\ \theta_1/3),\bm I)\), data from \(p(\bm x;\bm 0)\), so \(L(\bm\theta)=\tfrac12\bm\theta^\mathsf T\bm F\bm\theta\) with \(\bm F=\bm J^\mathsf T\bm J\) (Equation 6.94). Iris Gaussian: \(L=\log\sigma+\tfrac12\log 2\pi+\frac{s^2+(\bar x-\mu)^2}{2\sigma^2}\) with the exact Fisher matrix \(\mathrm{diag}(1/\sigma^2,\,2/\sigma^2)\) in \((\mu,\sigma)\) and \(\mathrm{diag}(1/\sigma^2,\,2)\) in \((\mu,\log\sigma)\). Both panels plot the \((\mu,\sigma)\) plane. The loss gap is the loss minus its minimum.
code/ch06_optimization.py (ngd_example, fisher_gauss_musigma).