Five networks of increasing structure classify 16×16 handwritten digits (Figure 11.10): Net-1 has no hidden layer; Net-2 one fully connected layer of 12 units; Net-3 two locally connected layers, each unit seeing only a 3×3 (then 5×5) patch of the layer below; Net-4 adds weight sharing in the first layer (two 8×8 feature maps, each computed with one 3×3 kernel); Net-5 shares weights in both hidden layers. Each network was trained by back-propagation on 320 digits that were shifted horizontally at random, as in the book's example, and is evaluated here, in your browser, on 300 shifted test digits. Click a hidden unit to see its receptive field on the digit.
Hidden units are tanh, outputs logistic; inputs and hidden maps are padded with the background value (−1) and 0 respectively. A link is a connection (biases included); a weight is a free parameter, so a shared kernel counts once for all the links it serves. Grey levels of the hidden cells: dark = −1, light = +1. The test curves are averages over five training runs on all 2007 shifted test digits; the trained weights shown here are those of the first run.
code/ch11_neural_networks.py (zip_layers, zip_counts, zip_torch_net, zip_forward_np).