A straight line y = w0 + w1x is fitted one data point at a time by the update w ← w + η(tn − wTφn)φn, which is stochastic gradient descent on the sum-of-squares error. Because there are only two weights, the whole path can be drawn over the elliptical contours of the error; the cross marks the least-squares solution wML. Click in the weight-space panel to choose a new starting point.
Data: N = 20 points, x uniform on (0, 1), t = −0.3 + 0.8x plus Gaussian noise, from the seeded generator. One epoch is N LMS updates, or one step of batch gradient descent on the mean error, w ← w + (η/N)ΦT(t − Φw), which has the same expected step. The decreasing schedule is ητ = η/(1 + τ/N), with τ the update count. The lower panel plots log10 of ED(w) − ED(wML).
code/ch04_single_layer_regression.py (line_data, lms, batch_gd).