A network with three convolutional layers (16, 24 and 32 channels), max-pooling, a global pooling layer and 11 outputs (the ten digits and “background”) was trained on MNIST in Python and runs here in your browser. See its learned 5×5 filters, the feature maps they produce, how the maps move when you shift the digit (equivariance) while the class probabilities hardly change (invariance), and where the network “looks” for each class (Grad-CAM, (10.9)–(10.10)). You can also draw your own digit.
Feature maps are after the ReLU, each scaled by its own maximum. Grad-CAM weights each last-layer channel by the average gradient of the chosen class's pre-softmax output (10.9) and sums the maps (10.10): the book sums the pre-activations, Selvaraju et al. the post-ReLU maps; only the positive part is overlaid. Each cell of the 7 × 7 map stands for a 24 × 24 receptive field. The digit pixels are values in [0, 1].
code/ch10_convolutional_networks.py (SmallCNN, np_forward, gradcam; the weights are in data/ch10-cnn.json). The logits and Grad-CAM maps computed here equal Python's to about 10−13.