Local variational bounds and Bayesian logistic regression

A local variational method replaces one awkward factor by a simpler bound with an adjustable parameter. A convex function lies above all its tangent lines, and the intercept of the tangent with slope η defines the conjugate g(η) (view Convex duality). For the logistic sigmoid this gives an exponential upper bound and the Jaakkola–Jordan lower bound σ(x) ≥ σ(ξ) exp{(x − ξ)/2 − λ(ξ)(x² − ξ²)}, the exponential of a quadratic that touches at x = ±ξ (view Bounds on σ). With one such bound per data point the logistic likelihood becomes Gaussian in w, so the posterior over the weights is Gaussian, and EM over the ξ_n raises a lower bound on the evidence at every step (view Logistic regression).