Interactive demo · Foundations of Deep Learning · Chapter 1

Three phases of neural networks: perceptron, backpropagation, depth

Section 1.3 of the book tells the history in three phases. Single layer: the perceptron learns a line by correcting mistakes, and stops after at most (R/γ)2 corrections when a line exists, but no line separates XOR. Two layers: with a differentiable activation, the error can be differentiated and backpropagation trains hidden units that solve XOR. Many layers: the backpropagated signal shrinks layer by layer through sigmoid units; residual connections keep it alive.

Phase

Data

Perceptron learning

Hidden units H

Learning rate

Gradient descent

Depth L

Activation

Weight scale (gain)

Scoreboard

Try these

In the lecture notes