At every node the learner tries every feature and every threshold between consecutive values, keeps the split that lowers the weighted impurity most, and recurses. Pick the data, the impurity measure and the depth. The right-hand plots show the gain of every candidate split at the root; the shaded plane shows the resulting partition. Click a point (or press the button) to leave one observation out and watch the tree change.
Node cost of a node with class weights s and total W: Gini W(1 − Σc (sc/W)2), entropy −Σc sc log(sc/W), misclassification W − maxc sc. The gain of a split is the cost of the node minus the costs of its two children, shown as a fraction of the root cost.
code/ch18_trees_forests_boosting.py (CART, iris_tree_probe, Figure 18.3).