Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 8

Online logistic regression by filtering

Treat the weights w = (w0, w1, w2) of p(y = 1 | x) = σ(w0 + w1x1 + w2x2) as a hidden state and see the 60 examples one at a time. ADF keeps a diagonal Gaussian and after each example matches the exact mean and variance of the one-dimensional logit (by quadrature); the EEKF keeps a full Gaussian and linearises the mean σ(xTw). Both are single passes costing O(D) or O(D2) per example. The red line is the offline Laplace approximation using all the data at once.

Method

Order of the data

Prior and drift

Try these

Prior N(0, v0 I) for every method; the drift adds q to the variances before each example (a random walk on the weights). Predictive maps use E[σ(η)] ≈ σ(m / √(1 + πv/8)). Simulated data (60 points), the same as in the notes.

In the lecture notes