Interactive demo · Applied Statistical Learning in R and Python · Chapter 10

A single-hidden-layer network on the Hitters data

The network f(X) = β0 + ∑k βk max(0, wk0 + ∑j wkjXj) (10.1) predicts a baseball player's 1987 salary from 20 standardised inputs. It is trained here, in your browser, by RMSprop on the squared error (10.23), with the gradients of (10.29)–(10.30), on the 176 training players of the R lab, and judged on the 87 test players. Compare the network's test error, epoch by epoch, with least squares and the lasso: Section 10.6's question of when deep learning is worth it.

Hidden units K

Dropout rate φ

Learning rate ρ

Epochs

Minibatch

Seed

Try these

Initial weights are uniform on ±1/√(fan-in), as in torch. RMSprop divides each gradient by the running root-mean-square of its past values (decay 0.99). With "full" every step uses all 176 players and, without dropout, a run is deterministic; minibatches of 32 make 6 steps per epoch, drawn at random as in stochastic gradient descent. Dropout zeroes each hidden unit with probability φ during training and scales the survivors by 1/(1 − φ). The inputs are standardised over all 263 players, as in the lab.

In the lecture notes