Interactive demo · Foundations of Deep Learning · Chapter 6

Two-layer networks: universal approximation and the activation function

The network y(x) = b + Σj vj h(wjx + cj) has M hidden units with activation h and a linear output (equations (6.7)–(6.12) of the book). It is trained in your browser by full-batch Adam on the sum-of-squares error, with the gradient computed by backpropagation. The dashed curves are the contributions vj h(·) of the individual hidden units (each rescaled to fit the panel), whose sum (plus b) is the red network function: this is how a handful of bent or kinked basis functions approximates a curve.

Target function

Activation h(a)

Hidden units M

Initial first-layer scale

Step size

Try these

Data: 50 equally spaced points on [−1, 1], no noise. Training minimises E/N = (1/2N) Σ (yn − tn)2 with Adam (full batch) from first-layer weights and biases drawn N(0, s2) and output weights N(0, 1/M), all from a seeded generator, so a class sees the same pictures. Leaky ReLU uses α = 0.1 and swish uses β = 1.

In the lecture notes