A single layer of weights computes activations ak = wkTφ(x) and passes them through an activation function: none for least squares, the logistic sigmoid for logistic regression, the normal distribution function for probit regression, the softmax for K classes. For the logistic and softmax models with the cross-entropy error the gradient is the same simple sum Σn(yn − tn)φn as for linear regression. This page trains the weights by batch gradient descent, starting from w = 0, with forward and backward passes written out in JavaScript.
code/ch05_single_layer_classification.py (functions logistic_gd, softmax_gd, least_squares_classifier, logistic_irls, probit_fit).