Interactive demo · Foundations of Deep Learning · Chapter 9

Dropout and ensembles

Averaging the predictions of M networks cancels the parts of their errors that differ. If the errors had zero mean and were uncorrelated, the committee's error would be the members' average error divided by M (9.50); in practice they are correlated, and with average error correlation c̄ the committee error is EAV(c̄ + (1 − c̄)/M), exactly, when averaged over all choices of M members from a pool. Dropout trains one network whose hidden units are deleted at random, an implicit average over 2M thinned networks; at test time the average is approximated by scaling the outgoing weights by the keep probability ρ, or by Monte Carlo over a few masks.

Mode

Members trained on

Number of members M

Keep probability ρ (1: no dropout)

Monte Carlo masks K

Hidden units

Adam steps

Data set

Try these

Each network is y = ∑j vj tanh(wjx + bj) + c, trained by full-batch Adam (step 0.01) on ½∑(y − t)2 for 20 noisy points of sin(2πx). Errors are measured against h(x) = sin(2πx) on 201 points of [0, 1], as in (9.44)–(9.47). Dropout draws a fresh mask for every data point at every step and acts on the hidden units only.

In the lecture notes