Section 1.3 of the book tells the history in three phases. Single layer: the perceptron learns a line by correcting mistakes, and stops after at most (R/γ)2 corrections when a line exists, but no line separates XOR. Two layers: with a differentiable activation, the error can be differentiated and backpropagation trains hidden units that solve XOR. Many layers: the backpropagated signal shrinks layer by layer through sigmoid units; residual connections keep it alive.
code/ch01_deep_learning_revolution.py (functions perceptron_data, perceptron, max_margin, mlp_init, mlp_grad, mlp_train, deep_chain_signal).