In the Credit data a card holder's credit limit and credit rating move almost in lockstep (correlation 0.997), while limit and age are nearly unrelated. Regress balance on limit together with age, and the residual sum of squares has a sharp bowl-shaped minimum: both coefficients are well determined. Swap age for rating, and the bowl becomes a long narrow valley: many pairs (βLimit, βRating) fit almost equally well, the standard error of βLimit grows about twelvefold and its t-statistic collapses. The variance inflation factor measures this growth. The simulated mode shows the same effect on repeated samples: as the correlation ρ of two predictors rises, the cloud of estimates stretches along a line.
RSS contours are drawn in millions, at 21.25, 21.5 and 21.8, with the intercept re-fitted for every pair of slopes, over a window of four standard errors either side of the least-squares estimates (as in Figure 3.15). VIFj = 1/(1 − R²Xj|X−j), the factor by which collinearity multiplies the variance of β̂j; with two predictors it is 1/(1 − r²). Simulated mode: n = 100, X1, X2 standard normal with correlation ρ, Y = 1 + X1 + X2 + N(0, 1), new predictors and noise in every data set; the same random numbers are reused at ρ = 0 to give the reference spread.
code/ch03_linear_regression.py (vif, profile_rss, ols; see the code index).