Interactive demo · Statistical Learning: Theory and Methods · Chapter 3

Least angle regression step by step, and the lasso path

Least angle regression (Algorithm 3.2) starts from zero, finds the input most correlated with the residual and moves its coefficient until a second input is equally correlated; then it moves both along the direction that keeps their correlations tied, and so on. Step through it: on the left the absolute correlations fall together and new inputs join the tie; on the right the coefficients move in straight lines. When a coefficient reaches zero, LAR carries it through to the other side, while the lasso (Algorithm 3.2a) drops the variable, which is the only difference between the two paths. Incremental forward stagewise (Algorithm 3.4), taking many tiny steps, traces almost the same path.

Data

Algorithm

Step size ε (forward stagewise)

Step

Try these

Every step length is computed exactly (Exercise 3.25): the inputs not yet active are checked for the value of the step at which their |inner product with the residual| catches up with the common value of the active set, and, for the lasso, the active coefficients for the step at which they reach zero. Correlations are shown as |xjTr|/N with inputs of variance 1. The faint dashed path on the right is the other algorithm, for comparison.

In the lecture notes