Interactive demo · Probabilistic Machine Learning · Chapter 19

Few-shot regression: a learned starting point

Every task is a sinusoid y = A sin(x − b); you see only K noise-free points of it (black triangles) and may take a few gradient steps on them. Three networks of the same size (1-40-40-1, ReLU) start from different weights: a random initialisation, an initialisation pre-trained by ordinary regression on thousands of sinusoids, and a MAML initialisation, trained so that one gradient step on the support points already gives a good fit on the rest of the same task. All three adapt here with plain gradient descent in your browser. The right-hand plot gives the test error over the whole interval against the number of steps.

Amplitude A

Phase b

Points seen K

Gradient steps

Step size

Try these

The three initialisations were trained offline (6000 meta-iterations of second-order MAML with 16 tasks of 10 support and 10 query points and one inner step of size 0.01; the pre-trained network used the same data, optimiser and number of iterations on the pooled regression loss) with amplitudes in [0.1, 5], phases in [0, π] and inputs in [−5, 5]; their weights are loaded here. The support inputs are a fixed random draw (the first K of ten). The test error is the mean squared error on 201 equally spaced points of [−5, 5]. Outside the training range of amplitudes and phases all three get worse; try A = 0.1.

In the lecture notes