Interactive demo · Statistical Learning: Theory and Methods · Chapter 11

Local connectivity and weight sharing: the five ZIP code networks

Five networks of increasing structure classify 16×16 handwritten digits (Figure 11.10): Net-1 has no hidden layer; Net-2 one fully connected layer of 12 units; Net-3 two locally connected layers, each unit seeing only a 3×3 (then 5×5) patch of the layer below; Net-4 adds weight sharing in the first layer (two 8×8 feature maps, each computed with one 3×3 kernel); Net-5 shares weights in both hidden layers. Each network was trained by back-propagation on 320 digits that were shifted horizontally at random, as in the book's example, and is evaluated here, in your browser, on 300 shifted test digits. Click a hidden unit to see its receptive field on the digit.

Network

Test digit (1 to 300)

Try these

Hidden units are tanh, outputs logistic; inputs and hidden maps are padded with the background value (−1) and 0 respectively. A link is a connection (biases included); a weight is a free parameter, so a shared kernel counts once for all the links it serves. Grey levels of the hidden cells: dark = −1, light = +1. The test curves are averages over five training runs on all 2007 shifted test digits; the trained weights shown here are those of the first run.

In the lecture notes