Lecture notes · First-year statistics

Estimation and Hypothesis Testing

How to learn about an unknown population quantity from data, how sure we can be, and how to weigh evidence for a claim. One running example ties the notes together: the body temperature of healthy adults, and the question of whether 37 °C is really "normal".

Main references

  1. D. P. Kroese and J. C. C. Chan, Statistical Modeling and Computation, Springer (2014): Chapter 5, Statistical Inference, and Chapter 6, Likelihood.
  2. P. Lafaye de Micheaux, R. Drouilhet and B. Liquet, The R Software: Fundamentals of Programming and Statistical Analysis, Springer (2013): Chapter 13, Confidence Intervals and Hypothesis Testing, for the R code.
  3. J. L. Devore and K. N. Berk, Modern Mathematical Statistics with Applications, 2nd edition, Springer (2012): Chapter 7, Point Estimation; Chapter 8, Statistical Intervals Based on a Single Sample; Chapter 9, Tests of Hypotheses Based on a Single Sample; and Chapter 10, Inferences Based on Two Samples.
  4. B. L. Welch (1947), The generalization of "Student's" problem when several different population variances are involved, Biometrika, 34(1–2), 28–35: the origin of the unequal-variance (Welch) \(t\)-test in Section 8.

Scope: estimation and hypothesis testing for a first-year course. The derivations in Section 2 use first-year calculus (logarithms, derivatives and the first- and second-derivative tests for extrema). The notes, examples and exercises are my own.

How these notes work

They follow my teaching philosophy: intuition before formalism, and ideas shown where they live in current research. Each part opens with an Explore first box: make a prediction, run a demonstration, and notice what happens. Only then come the definitions and formulas that explain what you saw, followed by worked examples and R code. From research boxes show where the same idea reappears in my own work, and the exercises ask you to reason rather than recall.

Part I. Estimation

Explore first: predict, then run

  1. Open Estimators on trial, choose Spread with \(n=5\). Before pressing +1000, predict: if we estimate a variance by averaging squared deviations from \(\bar x\) (dividing by \(n\)), will the average guess be too high, too low, or right? Then run it.
  2. Open Confidence intervals. If a hundred teams each build a 95% interval, how many intervals do you expect to miss the truth? Run 100 studies. Then predict what happens at \(n=5\) with the normal multiplier, and check.

Keep your predictions: Sections 2 and 3 explain both results.

1. Models, parameters and the running example

Statistics starts from a model: a family of probability distributions that could have produced the data, indexed by unknown parameters. The data are fixed numbers once collected, but we treat them as the outcome of random variables so that we can reason about what would happen if the study were repeated.

Running example: body temperature

Oral temperatures of \(n=130\) healthy adults have average \(\bar x = 36.81\,^\circ\mathrm{C}\) and standard deviation \(s = 0.41\,^\circ\mathrm{C}\) (Shoemaker, 1996). A natural model is

\[ X_1,\dots,X_n \overset{\text{iid}}{\sim} \mathcal N(\mu,\sigma^2), \qquad \mu,\ \sigma^2 \text{ unknown}. \]

Questions: what is our best guess for \(\mu\)? How precise is it? Is the textbook value \(\mu = 37\,^\circ\)C consistent with the data?

A second model we will use is for counts of successes: if \(X\) is the number of successes in \(n\) independent trials with the same success probability \(p\), then \(X \sim \mathrm{Bin}(n,p)\), and the parameter of interest is \(p\).

2. Estimates and estimators

Definition

An estimate \(\hat\theta\) of a parameter \(\theta\) is a number computed from the observed data \(x_1,\dots,x_n\). The corresponding estimator is the same formula applied to the random variables \(X_1,\dots,X_n\): it is a random variable, describing the estimates we would get from repeated studies. Its distribution is the sampling distribution.

For example, the estimate \(\bar x = 36.81\) is one outcome of the estimator \(\bar X = \frac1n\sum_{i=1}^n X_i\). We judge a recipe for estimating, not a single number: the properties below belong to the estimator.

Method of moments. Match population moments to their sample averages and solve for the parameters. For \(\mathcal N(\mu,\sigma^2)\), \(E(X)=\mu\) and \(E(X^2)=\sigma^2+\mu^2\), so

\[ \hat\mu = \bar x, \qquad \hat\sigma^2 = \frac1n\sum_{i=1}^n x_i^2 - \bar x^2 = \frac1n\sum_{i=1}^n (x_i-\bar x)^2 . \]

Estimators from calculus. Many estimators are the minimiser or maximiser of a criterion, found with derivatives (using derivatives and the first- and second-derivative tests from first-year calculus).

Unbiasedness and standard error

An estimator \(\hat\Theta\) of \(\theta\) is unbiased if \(E(\hat\Theta)=\theta\); its bias is \(E(\hat\Theta)-\theta\). Its standard error (SE) is its standard deviation, \(\sqrt{\mathrm{Var}(\hat\Theta)}\). For iid data with mean \(\mu\) and variance \(\sigma^2\) (normality not needed),

\[ E(\bar X) = \mu, \qquad \mathrm{Var}(\bar X) = \frac{\sigma^2}{n}, \qquad \mathrm{SE}(\bar X) = \frac{\sigma}{\sqrt n} \approx \frac{s}{\sqrt n}. \]

So \(\bar X\) is unbiased, and its spread shrinks like \(1/\sqrt n\): four times the data halves the SE. For body temperature, \(\mathrm{SE} \approx 0.41/\sqrt{130} = 0.036\,^\circ\)C.

Why divide by \(n-1\)? The method-of-moments variance \(\hat\sigma^2\) divides by \(n\) and is biased: \(E(\hat\sigma^2) = \frac{n-1}{n}\sigma^2\), because the deviations are measured from \(\bar x\), which is fitted to the same data and so sits closer to them than \(\mu\) does. The sample variance

\[ s^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i-\bar x)^2 = \frac{n}{n-1}\,\hat\sigma^2 \]

is unbiased. The difference matters for small \(n\) (at \(n=5\) the divisor-\(n\) version is 20% too small on average) and vanishes for large \(n\).

Comparing unbiased estimators. For normal data both the sample mean and the sample median are centred on \(\mu\), but the mean has smaller spread (for large \(n\) the median's variance is about \(\pi/2 \approx 1.57\) times larger). Among unbiased estimators we prefer the one with the smaller SE; more generally we compare the mean squared error \(\mathrm{MSE} = \mathrm{bias}^2 + \mathrm{Var}\).

From research to the classroom

The balance \(\mathrm{MSE}=\mathrm{bias}^2+\mathrm{Var}\) drives model choice in research too. In penalised model selection for mixtures of experts, adding experts makes the model more flexible (less bias) but harder to estimate (more variance); a penalty chosen from the data balances the two. See the slope-heuristic model selection demo, built from my paper in the Electronic Journal of Statistics (2022).

▶ Demonstration: Estimators on trial: bias, spread and the n − 1 Repeat a study thousands of times and watch the sampling distributions of two competing estimators.
▶ Demonstration: Maximum likelihood: climbing the likelihood Draw yes/no samples one at a time and watch the top of the log-likelihood sit at \(\hat p = x/n\), with its curvature giving the standard error.

3. Confidence intervals

An estimate without a measure of precision is of little use. A confidence interval reports a range of plausible values together with a guarantee about the method.

Definition

A random interval \((T_1,T_2)\), computed from the data and not depending on \(\theta\), is a \(100(1-\alpha)\%\) confidence interval for \(\theta\) if \(P(T_1<\theta<T_2) \ge 1-\alpha\) for every value of \(\theta\). The observed interval \((t_1,t_2)\) is its numerical version.

Interpretation

"95%" describes the procedure: used over and over, it captures the true \(\theta\) about 95 times in 100. A single computed interval, such as \((36.74, 36.88)\), either contains \(\mu\) or it does not, and we never find out which. It is not correct to say "\(\mu\) lies in this interval with probability 0.95", because \(\mu\) is fixed.

The pivot recipe. Find a function of the data and the parameter whose distribution is known and free of unknowns (a pivot), trap it between two quantiles, and rearrange the inequalities to isolate the parameter. Every interval below comes from this recipe.

(a) Mean, large sample (approximate). By the central limit theorem, \(\frac{\bar X-\mu}{S/\sqrt n}\) is approximately \(\mathcal N(0,1)\) for large \(n\), whatever the population shape (unless it is extremely skewed). With \(z_{\alpha/2}\) the \(1-\alpha/2\) quantile of \(\mathcal N(0,1)\) (\(z_{0.025}=1.96\)):

\[ \bar x \pm z_{\alpha/2}\,\frac{s}{\sqrt n}. \]

The half-width \(z_{\alpha/2}\,s/\sqrt n\) is the margin of error. To achieve margin \(E\), plan \(n \approx (z_{\alpha/2}\,\sigma/E)^2\) using a pilot estimate of \(\sigma\).

(b) Mean, normal data (exact). If the data are normal, \(\frac{\bar X-\mu}{S/\sqrt n} \sim t_{n-1}\), Student's \(t\) distribution with \(n-1\) degrees of freedom. It is bell-shaped with heavier tails than the normal, and approaches \(\mathcal N(0,1)\) as \(n\) grows. With \(t_{\alpha/2,n-1}\) its \(1-\alpha/2\) quantile:

\[ \bar x \pm t_{\alpha/2,\,n-1}\,\frac{s}{\sqrt n}. \]

Body temperature: 95% CI for \(\mu\)

\(t_{0.025,129} = 1.979\), so \(36.81 \pm 1.979 \times 0.036 = (36.74,\ 36.88)\,^\circ\)C. The large-sample \(z\) version gives \((36.74, 36.88)\) too: with \(n=130\) the two agree. The value 37 lies outside the interval, a first hint about Section 5.

Small samples are where \(t\) matters: \(t_{0.025,4}=2.776\) and \(t_{0.025,9}=2.262\), compared with 1.96. Using 1.96 with \(n=5\) gives intervals that are too narrow, covering the truth only about 88% of the time instead of 95%.

▶ Demonstration: Confidence intervals: does the method catch the truth? A hundred studies, a hundred intervals: see coverage, the effect of \(n\), and the repair by Student's \(t\).
▶ Demonstration: Averages on trial: why ± 1.96 works Average skewed waiting times: as \(n\) grows the standardised averages follow the bell curve, so about 95% land within 1.96 standard errors (the central limit theorem).
▶ Demonstration: Why Student's t? With small samples, estimating \(\sigma\) by \(s\) gives the statistic heavier tails than the bell curve, so the multiplier must be larger than 1.96.

From research to the classroom

"Does the method catch the truth at its stated rate?" is exactly how uncertainty is checked in research. In my work on robust Bayesian parameter balancing for kinetic models in systems biology, each kinetic parameter is measured by many independent sources. Leave-one-source-out cross-validation holds out one whole source, refits on the rest, and checks how often the 95% predictive intervals contain the held-out measurements: the same coverage check, on real data. See the Bayesian parameter balancing demo.

(c) Variance, normal data. \(\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}\), the chi-squared distribution with \(n-1\) degrees of freedom (non-negative and right-skewed). With \(q_1, q_2\) its \(\alpha/2\) and \(1-\alpha/2\) quantiles:

\[ \left( \frac{(n-1)s^2}{q_2},\ \frac{(n-1)s^2}{q_1} \right) \text{ for } \sigma^2, \quad \text{and take square roots for } \sigma. \]

Body temperature: \(q_1=99.45\), \(q_2=162.33\) for 129 df, giving \((0.134, 0.218)\) for \(\sigma^2\) and \((0.37, 0.47)\,^\circ\)C for \(\sigma\). This interval relies heavily on normality.

(d) Difference of two means (Welch). For independent samples of sizes \(n_1, n_2\) from \(\mathcal N(\mu_X,\sigma_X^2)\) and \(\mathcal N(\mu_Y,\sigma_Y^2)\), with variances not assumed equal:

\[ \bar x - \bar y \pm t_{\alpha/2,\,\nu}\sqrt{\frac{s_X^2}{n_1}+\frac{s_Y^2}{n_2}}, \qquad \nu = \frac{\left(\frac{s_X^2}{n_1}+\frac{s_Y^2}{n_2}\right)^2}{\frac{(s_X^2/n_1)^2}{n_1-1}+\frac{(s_Y^2/n_2)^2}{n_2-1}} \]

(the Welch–Satterthwaite effective degrees of freedom). By the central limit theorem it also works approximately for non-normal data when both samples are reasonably large. When working by hand, a conservative alternative is \(\nu=\min(n_1,n_2)-1\), which gives a slightly wider interval.

(d′) Difference of two means, equal variances (pooled, exact). If we may assume \(\sigma_X=\sigma_Y\), pool the two sample variances, \(s_p^2=\frac{(n_1-1)s_X^2+(n_2-1)s_Y^2}{n_1+n_2-2}\), and use

\[ \bar x - \bar y \pm t_{\alpha/2,\,n_1+n_2-2}\; s_p\sqrt{\frac1{n_1}+\frac1{n_2}}, \]

which has exact coverage for normal data with a common variance.

Women versus men

In the same data, 65 women average \(36.89\,^\circ\)C (\(s=0.41\)) and 65 men \(36.73\,^\circ\)C (\(s=0.39\)). The difference is \(0.16\), SE \(0.070\), \(\nu \approx 127.7\) (127.5 from the unrounded data), and the 95% CI is \((0.02,\ 0.30)\,^\circ\)C: it excludes 0, so women's average temperature appears slightly higher. The pooled interval (\(s_p=0.40\), 128 df) is the same to two decimals, as expected when the spreads are similar.

(e) One proportion (approximate). With \(\hat p = x/n\) and the normal approximation to the binomial,

\[ \hat p \pm z_{\alpha/2}\sqrt{\frac{\hat p(1-\hat p)}{n}}, \]

reasonable when there are at least 5 successes and 5 failures. Example: 64 of 80 seeds germinate, \(\hat p = 0.80\), SE \(0.045\), 95% CI \((0.71,\ 0.89)\).

(f) Difference of two proportions (approximate).

\[ \hat p_X-\hat p_Y \pm z_{\alpha/2}\sqrt{\frac{\hat p_X(1-\hat p_X)}{n_1}+\frac{\hat p_Y(1-\hat p_Y)}{n_2}}. \]

Example: a reminder email is followed by a response from 45 of 150 people, against 30 of 150 without it. The difference is \(0.10\), SE \(0.050\), 95% CI \((0.003,\ 0.197)\).

# R: confidence intervals
t.test(x, conf.level = 0.95)$conf.int            # t interval for a mean (needs the raw data x)
xbar <- 36.81; s <- 0.41; n <- 130               # from summary statistics:
xbar + c(-1, 1) * qt(0.975, n - 1) * s / sqrt(n)  # 36.739 36.881
(n - 1) * s^2 / qchisq(c(0.975, 0.025), n - 1)    # CI for sigma^2: 0.134 0.218
t.test(x, y)$conf.int                              # Welch CI for muX - muY
prop.test(c(45, 30), c(150, 150), correct = FALSE)$conf.int

Part II. Hypothesis testing

Explore first: predict, then run

  1. Open A hypothesis test on trial with \(d=0\) (the null hypothesis is true), \(n=30\), \(\alpha=0.05\). Predict: out of 1000 studies, how many will wrongly reject \(H_0\)? What will the histogram of P-values look like? Run it.
  2. Now set \(d=0.5\) and \(n=5\). Predict the percentage of studies that detect this real effect. Then try \(n=50\).

Sections 4 to 6 explain what you saw: P-values, the Type I error rate \(\alpha\), and power.

4. The logic of a hypothesis test

A test weighs the data against a claim. The null hypothesis \(H_0\) is the status quo or "no effect" statement (e.g. \(\mu=37\)); the alternative \(H_1\) is what we would like to demonstrate (e.g. \(\mu \ne 37\)). The roles are not symmetric, as in a court of law: \(H_0\) stands unless the evidence against it is convincing.

Test statistic and P-value

A test statistic \(T\) is a function of the data whose distribution under \(H_0\) is known. With observed value \(t\), the P-value is the probability, computed assuming \(H_0\) is true, of a value of \(T\) at least as extreme as \(t\) in the direction(s) of \(H_1\):

\[ \text{left-sided: } P_{H_0}(T\le t), \qquad \text{right-sided: } P_{H_0}(T\ge t), \qquad \text{two-sided: } 2\,P_{H_0}(T\ge |t|) \ \text{(symmetric } T). \]

A small P-value means the data would be surprising if \(H_0\) were true. Rough scale: below 0.10 weak, below 0.05 moderate, below 0.01 strong evidence against \(H_0\). Decision rule at significance level \(\alpha\) (often 0.05): reject \(H_0\) if \(p<\alpha\). A P-value is not the probability that \(H_0\) is true, and "not rejecting" \(H_0\) is not proof that it is true.

A checklist for every test

  1. State a model for the data.
  2. State \(H_0\) and \(H_1\) (one- or two-sided).
  3. Choose a test statistic.
  4. Find its distribution under \(H_0\).
  5. Compute its observed value.
  6. Compute the P-value.
  7. Decide, and state the conclusion in the context of the problem.

5. One-sample \(t\)-test

Model \(X_1,\dots,X_n \overset{\text{iid}}{\sim} \mathcal N(\mu,\sigma^2)\); test \(H_0:\mu=\mu_0\). Measuring the distance from \(\mu_0\) in units of the standard error gives a statistic whose null distribution does not depend on the unknown \(\sigma\):

\[ T = \frac{\bar X-\mu_0}{S/\sqrt n} \;\sim\; t_{n-1} \quad \text{under } H_0. \]

Is 37 °C the normal body temperature?

\(H_0:\mu=37\) versus \(H_1:\mu\ne 37\). Observed \(t = \frac{36.81-37}{0.41/\sqrt{130}} = -5.28\) on 129 df, so the two-sided P-value is \(2P(T_{129}\ge 5.28) \approx 5\times10^{-7}\). (These use the rounded summaries; the full data, \(\bar x = 36.805\) and \(s = 0.407\), give \(t=-5.46\) and \(p\approx 2\times10^{-7}\). The conclusion is the same.) There is very strong evidence that the mean oral temperature of healthy adults is not 37 °C; the data point to about 36.8 °C.

Tests and intervals agree. A two-sided test at level \(\alpha\) rejects \(H_0:\mu=\mu_0\) exactly when \(\mu_0\) lies outside the \(100(1-\alpha)\%\) \(t\) interval: 37 is outside \((36.74, 36.88)\).

t.test(x, mu = 37)                          # two-sided (default)
t.test(x, mu = 37, alternative = "less")    # one-sided
2 * pt(-abs((36.81 - 37) / (0.41 / sqrt(130))), df = 129)   # from summary statistics
▶ Demonstration: Tests and intervals: two views of one object Drag the hypothesised mean across the 95% interval: the test rejects exactly when the value lies outside it, as 37 °C does.

6. Type I and Type II errors, and power

Decision\(H_0\) true\(H_1\) true
Do not reject \(H_0\)correctType II error (probability \(\beta\))
Reject \(H_0\)Type I error (probability \(\le\alpha\))correct (probability = power \(=1-\beta\))

The rule "reject if \(p<\alpha\)" caps the Type I error rate at \(\alpha\). Within that cap we want high power, the probability of rejecting \(H_0\) when a particular alternative is true. For the one-sample \(t\)-test, power depends on the standardised effect \(d=(\mu-\mu_0)/\sigma\), on \(n\), on \(\alpha\) and on whether the test is one- or two-sided.

How much data do we need?

One-sided test at \(\alpha=0.05\) when the truth is half a standard deviation above \(\mu_0\) (\(d=0.5\)): power is 0.24 with \(n=5\), 0.43 with \(n=10\), 0.85 with \(n=30\) and 0.97 with \(n=50\). About \(n=27\) observations give 80% power. A small study can easily miss a real effect.

Errors for a fixed rejection rule

A supplier claims that 70% of its seeds germinate. We suspect fewer, plant \(n=20\) seeds, and decide in advance to reject \(H_0:p=0.7\) in favour of \(H_1:p<0.7\) if \(X\le 10\) germinate, where \(X\sim\mathrm{Bin}(20,p)\).

  • Type I error: \(\alpha=P(X\le10 \mid p=0.7)=0.048\).
  • Type II error if in fact \(p=0.5\): \(\beta=P(X\ge11\mid p=0.5)=0.41\), so power is 0.59.
  • If \(p=0.4\): \(\beta=P(X\ge11\mid p=0.4)=0.13\), power 0.87. The further the truth is from \(H_0\), the higher the power.

In R: pbinom(10, 20, 0.7) and 1 - pbinom(10, 20, 0.5).

power.t.test(n = 5, delta = 0.5, sd = 1, sig.level = 0.05,
             type = "one.sample", alternative = "one.sided")   # power = 0.239
power.t.test(delta = 0.5, sd = 1, power = 0.8, sig.level = 0.05,
             type = "one.sample", alternative = "one.sided")   # n = 26.1, so 27
▶ Demonstration: A hypothesis test on trial: P-values, errors and power Run thousands of \(t\)-tests: false alarms at rate \(\alpha\) when \(H_0\) is true, and power when it is not.
▶ Demonstration: Power and sample size: two bell curves Slide the effect size and the sample size: the areas α, β and power change, and the demo gives the \(n\) needed for 80% or 90% power.

From research to the classroom

Choosing how many components a mixture model needs is a decision with two ways to go wrong, much like Type I and Type II errors: too few components miss real structure, too many fit noise. Criteria such as AIC and BIC settle this by penalising extra components. In my recent paper with Hien Duy Nguyen (Annals of the Institute of Statistical Mathematics, 2026) we adjust the BIC penalty by a logarithmic factor so that it provably selects the right number as \(n\) grows, even for heavy-tailed families the classical theory does not cover. In the demo, watch over- and under-selection rates change with \(n\), just as error rates and power did above. See the BIC order-selection demo.

7. One-sample test for a proportion

Model \(X \sim \mathrm{Bin}(n,p)\); test \(H_0:p=p_0\).

Taste test

14 of 18 tasters prefer a new recipe. \(H_0:p=0.5\) versus \(H_1:p>0.5\). Exact: \(P(X\ge14)=0.015\) for \(X\sim\mathrm{Bin}(18,0.5)\). Approximation: \(z=2.36\), \(p=0.009\). With only 18 people the approximation understates the P-value; prefer the exact test for small \(n\).

binom.test(14, 18, p = 0.5, alternative = "greater")$p.value                 # 0.0154
prop.test(14, 18, p = 0.5, alternative = "greater", correct = FALSE)$p.value # 0.0092
▶ Demonstration: Counting successes: exact and approximate methods Why the exact test for 18 tasters has size 0.048 rather than 0.05, and how often the ± 1.96 interval for a proportion really covers \(p\).

8. Two samples: Welch, pooled and paired \(t\)-tests

Welch test (independent samples, variances not assumed equal): \(H_0:\mu_X=\mu_Y\),

\[ T=\frac{\bar X-\bar Y}{\sqrt{S_X^2/n_1+S_Y^2/n_2}} \approx t_\nu \quad \text{under } H_0, \ \nu \text{ as in the Welch CI.} \]

Pooled test (assumes \(\sigma_X=\sigma_Y\)): with \(S_p^2=\frac{(n_1-1)S_X^2+(n_2-1)S_Y^2}{n_1+n_2-2}\), \(T=\frac{\bar X-\bar Y}{S_p\sqrt{1/n_1+1/n_2}}\sim t_{n_1+n_2-2}\) exactly under \(H_0\). Welch is the safer default; the two agree closely when the sample sizes or spreads are similar.

Women versus men, as a test

\(H_0:\mu_W=\mu_M\) versus \(H_1:\mu_W\ne\mu_M\): Welch \(t=2.28\) on 127.7 df, \(p=0.024\); pooled \(t=2.28\) on 128 df, \(p=0.024\) (with equal group sizes the two statistics coincide). Moderate evidence of a difference, consistent with the CI \((0.02, 0.30)\) excluding 0.

Paired data. When each subject is measured twice (before/after, two methods on the same items), the two samples are dependent. Analyse the differences \(D_i = X_i - Y_i\) with a one-sample \(t\)-test of \(H_0: \mu_D=0\), with \(n-1\) df.

Sleep before and after a routine change

Eight people's nightly sleep (hours) before: 7.1, 6.4, 6.8, 5.9, 7.4, 6.2, 6.9, 6.5; after: 7.6, 6.9, 6.7, 6.6, 7.9, 6.8, 7.5, 6.6. Differences: mean \(0.425\) h, \(s_D=0.276\), \(t=4.35\) on 7 df, two-sided \(p=0.0034\), 95% CI \((0.19,\ 0.66)\) h. Treating the samples as independent would ignore the large person-to-person variation and lose most of this evidence.

t.test(x, y)                      # Welch (default)
t.test(x ~ group, data = d)       # same, with a two-level grouping variable
t.test(x, y, var.equal = TRUE)    # pooled
t.test(after, before, paired = TRUE)
▶ Demonstration: Two samples: pooled or Welch, paired or not? When group sizes and spreads both differ, the pooled test rejects a true \(H_0\) about 21% of the time while Welch stays near 5%; and for paired sleep data pairing raises the power from about 28% to 96%.

9. Two-sample test for proportions

Model \(X\sim\mathrm{Bin}(n_1,p_X)\), \(Y\sim\mathrm{Bin}(n_2,p_Y)\), independent; \(H_0:p_X=p_Y\). Under \(H_0\) both groups share one \(p\), estimated by the pooled proportion \(\hat p=\frac{X+Y}{n_1+n_2}\):

\[ Z=\frac{\hat P_X-\hat P_Y}{\sqrt{\hat P(1-\hat P)\left(\frac1{n_1}+\frac1{n_2}\right)}} \approx \mathcal N(0,1) \quad \text{under } H_0. \]

Reminder emails

45/150 versus 30/150, \(H_1:p_X>p_Y\). Pooled \(\hat p=75/300=0.25\), \(z=2.00\), one-sided \(p=0.023\): moderate evidence that reminders help. (The test uses the pooled SE because it assumes \(H_0\); the confidence interval in Section 3(f) uses the unpooled SE.)

prop.test(c(45, 30), c(150, 150), alternative = "greater", correct = FALSE)   # p = 0.0228

Summary table

SituationEstimate and intervalTest statistic (null distribution)R
One mean, normal data\(\bar x \pm t_{\alpha/2,n-1}\, s/\sqrt n\)\(\frac{\bar X-\mu_0}{S/\sqrt n}\) (\(t_{n-1}\))t.test(x, mu=)
One mean, large \(n\)\(\bar x \pm z_{\alpha/2}\, s/\sqrt n\)same (\(\approx\mathcal N(0,1)\))t.test
One variance, normal data\(\big((n-1)s^2/q_2,\ (n-1)s^2/q_1\big)\)\((n-1)S^2/\sigma_0^2\) (\(\chi^2_{n-1}\))qchisq
Two means, independent\(\bar x-\bar y \pm t_{\alpha/2,\nu}\sqrt{s_X^2/n_1+s_Y^2/n_2}\)Welch \(T\) (\(\approx t_\nu\))t.test(x, y)
Two means, paired\(\bar d \pm t_{\alpha/2,n-1}\, s_D/\sqrt n\)\(\frac{\bar D}{S_D/\sqrt n}\) (\(t_{n-1}\))t.test(x, y, paired=TRUE)
One proportion\(\hat p \pm z_{\alpha/2}\sqrt{\hat p(1-\hat p)/n}\)\(X\) (\(\mathrm{Bin}(n,p_0)\)), or \(Z\) (\(\approx\mathcal N(0,1)\))binom.test, prop.test
Two proportions\(\hat p_X-\hat p_Y \pm z_{\alpha/2}\sqrt{\cdots}\) (unpooled)pooled \(Z\) (\(\approx\mathcal N(0,1)\))prop.test

Exercises

  1. Seven steps, filled in. A manufacturer claims its batteries last 10 hours on average. A consumer group tests \(n=25\) batteries and finds \(\bar x=9.6\) h and \(s=0.9\) h. Is there evidence that the mean life is less than claimed? Work through: (a) model, (b) \(H_0\) and \(H_1\), (c) test statistic, (d) its null distribution, (e) observed value, (f) P-value (which side?), (g) decision, (h) conclusion in context.
    Answer

    (a) \(X_1,\dots,X_{25}\overset{\text{iid}}{\sim}\mathcal N(\mu,\sigma^2)\). (b) \(H_0:\mu=10\), \(H_1:\mu<10\). (c) \(T=\frac{\bar X-10}{S/\sqrt{25}}\). (d) \(t_{24}\). (e) \(t=\frac{9.6-10}{0.9/5}=-2.22\). (f) Left-sided: \(P(T_{24}\le-2.22)=0.018\). (g) Reject \(H_0\) at \(\alpha=0.05\). (h) Moderate evidence that the mean battery life is below 10 h; a 95% CI for \(\mu\) is \((9.23,\ 9.97)\) h. In R: pt(-2.22, 24).

  2. Show that \(\hat\sigma^2 = \frac1n\sum (x_i-\bar x)^2\) equals \(\frac1n\sum x_i^2-\bar x^2\), and hence that \(s^2=\frac{n}{n-1}\hat\sigma^2\).
    Answer

    Expand \((x_i-\bar x)^2 = x_i^2-2x_i\bar x+\bar x^2\) and use \(\sum x_i = n\bar x\): \(\sum (x_i-\bar x)^2 = \sum x_i^2 - n\bar x^2\). Divide by \(n\) or by \(n-1\).

  3. A pilot study gives \(s\approx 0.41\,^\circ\)C. How many people are needed for a 95% interval for \(\mu\) with margin of error 0.05 °C?
    Answer

    \(n\approx(1.96\times0.41/0.05)^2 = 258.3\), so about 259 people.

  4. In the taste test, what is the smallest number of the 18 tasters who must prefer the new recipe for the exact one-sided test to reject \(H_0:p=0.5\) at \(\alpha=0.05\)?
    Answer

    \(P(X\ge13)=0.048\) and \(P(X\ge12)=0.119\) for \(X\sim\mathrm{Bin}(18,0.5)\), so at least 13. In R: 1 - pbinom(12, 18, 0.5).

  5. A study reports "not significant, \(p=0.30\)" with \(n=8\). Explain why this is weak evidence that there is no effect.
    Answer

    With \(n=8\) the power to detect even a moderate effect is low (for \(d=0.5\), one-sided \(\alpha=0.05\), power is under 0.4), so a real effect would often produce a non-significant result. Failing to reject \(H_0\) is not evidence that \(H_0\) is true; report a confidence interval to show which effect sizes remain plausible.

Demonstrations

Ten interactive experiments that run in your browser, each tied to a section of these notes. The advanced notes link each one to the theorem it illustrates.

DemonstrationWhat you can seeSections
Estimators on trialBias, spread and MSE of competing estimators; why we divide by \(n-1\)2
Maximum likelihoodThe likelihood of each sample, its top (the estimate) and its curvature (the standard error)2
Averages on trialAverages gather at the mean and their histogram becomes the bell curve2, 3
Confidence intervalsA hundred intervals at a time: how many catch the truth3
Why Student's t?Why small samples need the t multiplier, and when normality matters3, 5
Tests and intervalsA test rejects exactly the values outside the confidence interval3, 5
A hypothesis test on trialFalse alarms at rate α, flat P-values, and power by simulation4–6
Power and sample sizeThe areas α, β and power, and the sample size for a target power6
Counting successesExact and approximate tests and intervals for a proportion3, 7
Two samplesPooled versus Welch, and the power gained by pairing8

References

  1. Kroese, D. P. and Chan, J. C. C. (2014). Statistical Modeling and Computation. Springer, New York. doi:10.1007/978-1-4614-8775-3. Chapters 5–6. Main reference.
  2. Lafaye de Micheaux, P., Drouilhet, R. and Liquet, B. (2013). The R Software: Fundamentals of Programming and Statistical Analysis. Statistics and Computing. Springer, New York. doi:10.1007/978-1-4614-9020-3. Chapter 13. Main reference.
  3. Devore, J. L. and Berk, K. N. (2012). Modern Mathematical Statistics with Applications, 2nd edition. Springer Texts in Statistics. Springer, New York. doi:10.1007/978-1-4614-0391-3. Chapters 7–10. Main reference.
  4. Shoemaker, A. L. (1996). What's normal? Temperature, gender, and heart rate. Journal of Statistics Education, 4(2).
  5. Mackowiak, P. A., Wasserman, S. S. and Levine, M. M. (1992). A critical appraisal of 98.6 °F, the upper limit of the normal body temperature, and other legacies of Carl Reinhold August Wunderlich. JAMA, 268(12), 1578–1580.
  6. Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. doi:10.1093/biomet/34.1-2.28. Main reference.

The sleep, seed, taste-test, reminder-email and battery examples are illustrative numbers constructed for these notes. All numerical results were computed in R.