Thirty points lie near the line y = −1 + 4x, but eight of them (the red circles) have been pushed far below it. Least squares is dragged towards the outliers because squared error punishes a large residual quadratically. The Laplace likelihood (least absolute deviations), the Huber loss and the Student-t likelihood all grow more slowly in the tails and ignore most of the contamination. Slide the outliers up or down, change the Huber threshold δ or the Student degrees of freedom ν, or delete the outliers altogether.
Fits: least squares (closed form); Laplace: the exact minimiser of the sum of absolute residuals (a line through two data points, found by checking all pairs); Huber: iteratively reweighted least squares; Student-t: EM started from the Laplace fit, with the scale estimated. The truth is the dashed white line. The data are fixed (seed 4); the outliers have been displaced downwards by 3.5 to 6.
code/ch11_linear_regression.py (laplace_lp, huber_irls, student_em, Figure 11.19).