Interactive demo · Probabilistic Machine Learning: Advanced Topics · Chapter 33

Designing a user study of explanations

An explanation method is better if people do better with it, so the evidence is a user study. Suppose the score of a participant has standard deviation 1 and the explanation raises it by d standard deviations. In a between-subject design two groups of n people see one condition each; in a within-subject (matched) design n people see both, and a person's two scores have correlation ρ, which the pairing removes. Power is that of the exact two-sided t-test. With m explanation types compared with a baseline, the chance of at least one false discovery grows unless the level is corrected.

The effect and the design

Comparisons

Try these

Power is computed exactly from the noncentral t distribution: between, noncentrality d√(n/2) with 2n − 2 degrees of freedom; within, d√(n/(2(1 − ρ))) with n − 1. The level used for each comparison is α or, with the Bonferroni correction, α/m. The simulation draws m independent comparisons of which half have a real effect d, and reports the share of false discoveries among the true nulls (any one counts) and the power on the real effects under no correction, Bonferroni and Holm; it changes with the random seed of this page, not with the controls above.

In the lecture notes