A network with hidden layers zl = φ(Wlzl−1 + bl) ends in a logit, turned into a probability by the sigmoid (13.13)–(13.16). The left panel shows P(y = 1 | x) over the plane with the 0.5 contour; the right panels show the training loss and what every hidden unit computes, as in Figure 13.3. Train with full-batch Adam or plain gradient descent, change the depth, the width, the activation function and the weight decay, and see when the model underfits (no hidden layer, linear activations), when it is stuck on a plateau, and when it overfits.
Each point is one training example (200 per data set). The loss is the mean cross entropy (10.8); the penalty is (λ/2)Σw2 over the weights, not the biases (Section 13.5.2). Weights start from N(0, 2/nin) for ReLU and N(0, 2/(nin+nout)) otherwise (Section 13.4.5.1), biases at 0. With no hidden layer the model is logistic regression. Hidden-unit tiles are scaled to their own range.
code/ch13_neural_networks_tabular.py (MLP, playground_run).