Interactive demo · Applied Statistical Learning in R and Python · Chapter 1

Up and down days on the stock market

The Smarket data record the percentage change of the S&P 500 index on each of 1250 trading days from 2001 to 2005, together with the changes on the five previous days (Lag1 to Lag5) and the trading volume. The output is qualitative: did the market go Down or Up today? Left: compare a past return on the Up and Down days; the boxes are almost identical, and a two-sample t statistic says how far apart the means really are. Right: fit quadratic discriminant analysis (QDA) on the days of 2001 to 2004, predict the probability of a Down day for each day of 2005, and call a day Down when that probability exceeds the threshold. Compare the accuracy with the rule that always says Up.

Left panel: variable

Right panel: QDA inputs

Threshold for calling a day Down

Try these

Data: the Smarket data of the book (1250 days; 602 Down, 648 Up). Boxplots follow R's conventions (Tukey's hinges, whiskers to the most extreme points within 1.5 box lengths); the diamond is the mean. The t statistic is Welch's, mean on Down days minus mean on Up days. QDA: each class has its own mean vector and covariance matrix (divisor nk − 1) estimated from the 998 days of 2001 to 2004, and the prior is the class proportion; the 252 days of 2005 are the test set. The standard error of an accuracy p on 252 days is √(p(1 − p)/252).

In the lecture notes