Interactive demo · Statistical Learning: Theory and Methods · Chapter 15

Gini against permutation importance on the spam data

A random forest offers two importance measures. The Gini importance of an input adds up, over all splits on it in all trees, how much the split lowered the node's Gini index (weighted by the node size). The permutation importance shuffles the input among each tree's out-of-bag cases and records how much the tree's out-of-bag accuracy drops. Both are shown here, each in percent of its own largest value, for forests of 500 trees on the spam data (4601 e-mails, 57 inputs, the book's training/test split). Change m, add a copy of an input or add pure noise, and compare with what happens when the forest is refitted without an input.

Forest

Sort by

Inputs shown K

Try these

Gini importance: mean over the trees of the total decrease of node size × Gini index at the splits on the input. Permutation importance: mean over the trees of the drop in out-of-bag accuracy (percentage points), with ±2 standard errors across trees; an input that a tree never uses contributes exactly zero for that tree. The refit panel uses forests of 200 trees with m = 7 grown without each of the 12 inputs with the largest permutation importance; differences of a few tenths of a percentage point are within the noise of the error estimates.

In the lecture notes