The black box is p(x) = σ(2 sin(1.4 x1) + 1.5 x2 − 1.2 x1x2), a classifier with an interaction. To explain it at the dot, fit a plane by weighted least squares with Gaussian weights of width h around the dot (the infinite-sample limit of a local surrogate such as LIME). A narrow kernel gives a faithful but tiny explanation that agrees with the gradient; a wide kernel explains more of the model and is no longer faithful anywhere. Click the picture to move the dot.
Fidelity is the weighted R² of the plane against the probability; class agreement is the weighted share of the neighbourhood where the plane and the black box predict the same class (threshold 1/2). Stability is the cosine between the coefficient vectors at the point and at the point shifted along x1. The weights are placed on a 41 × 41 grid over ±4h, so there is no sampling noise.
code/ch33_interpretability.py (local_surrogate, local_report).