Interactive demo · Statistical Learning: Theory and Methods · Chapter 10

Opening the black box: importance and partial dependence on the spam data

A gradient-boosted model of the log-odds of spam is a sum of hundreds of small trees, impossible to read directly. The relative importance of an input (left) adds up the squared-error improvements of all the splits on it. The partial dependence of the log-odds on an input XS is the model averaged over the other inputs as they occur in the 3065 training e-mails, f̄S(xS) = (1/N)∑i f(xS, xiC): it is the average of the thin individual curves, one per e-mail. It is not the conditional expectation E(f(X) | XS), which also carries the effect of whatever else comes with XS. Choose a second input to see a two-variable partial dependence, and switch to the stump model to see what "additive" means.

Model (binomial deviance, ν = 0.1)

Input

Second input (two-variable partial dependence)

Try these

Partial dependence is computed exactly from the trees: each leaf contributes its value times the fraction of training e-mails whose other inputs satisfy the leaf's conditions on them (Exercise 10.11), and the curves are centred to average zero over the training e-mails. The data-free traversal replaces those fractions by the training counts of the nodes, which is exact only in special cases. Red ticks are the deciles of the input. Click a bar to choose an input.

In the lecture notes