The same topics as the beginner notes, now built up from the axioms of probability. Every formula is stated as a result and proved, so that you can see why each confidence interval and test works, and exactly which assumptions it needs.
Prerequisites. Sets and functions, sums and integrals, derivatives, and for one proof (Theorem 0.9) orthogonal matrices. Everything about probability is developed from the axioms in Part 0.
Proofs are hidden. Read the statement first, decide whether you believe it (try a small case, or run a demonstration), then open the proof. The button below opens or closes all of them at once. After most results a ▶ Demo box opens an interactive demonstration already set to show that result; the full list is in Demonstrations.
Honest foundations. A few deep results are used without proof and are labelled (F1)–(F5) in Section 0.1: they are proved in a second-year probability course. Everything else is proved here from them and from the axioms. Results marked Going further are optional.
Scope: estimation and hypothesis testing for a first-year course. The text, proofs, examples and exercises are my own.
A probability \(P\) assigns to each event \(A\) of a sample space \(\Omega\) a number with (A1) \(P(A)\ge 0\); (A2) \(P(\Omega)=1\); (A3) if \(A_1,A_2,\dots\) are pairwise disjoint, \(P\big(\bigcup_i A_i\big)\)
(a) \(P(A^c)=1-P(A)\). (b) If \(A\subseteq B\) then \(P(A)\le P(B)\). (c) \(P(A\cup B)\)
Finite additivity first. Taking every \(A_i=\varnothing\) in (A3) gives \(P(\varnothing)=\sum_i P(\varnothing)\), which is impossible for a finite number unless \(P(\varnothing)=0\). So (A3) also holds for finitely many disjoint events: pad the list with empty sets.
(a) \(A\) and \(A^c\) are disjoint with union \(\Omega\), so by (A3) and (A2), \(P(A)+P(A^c)=1\). (b) \(B=A\cup(B\cap A^c)\) is a disjoint union, so \(P(B)=P(A)+P(B\cap A^c)\ge P(A)\) by (A1). (c) \(A\cup B=A\cup(B\cap A^c)\) and \(B=(B\cap A)\cup(B\cap A^c)\), both disjoint; subtract the two applications of (A3). The inequality uses \(P(A\cap B)\ge0\). ∎
A random variable \(X\) is a numerical function of the outcome. Its distribution is described by the cumulative distribution function \(F(x)=P(X\le x)\), and by a probability mass function \(p(x)\) (discrete case) or a density \(f(x)\) with \(P(a\lt X\le b)=\int_a^b f\) (continuous case). The expectation is \(E(X)=\sum_x x\,p(x)\) or \(\int x f(x)\,dx\), and \(E\,g(X)=\sum g(x)p(x)\) or \(\int g(x)f(x)\,dx\). Random variables are independent if events defined by them are independent. When \(X,Y\) have a joint density, they are independent exactly when it factorises, \(f_{X,Y}(x,y)=f_X(x)f_Y(y)\), and likewise for \(n\) variables.
For constants \(a,b\) and random variables \(X,Y\) with finite variances:
\[ E(aX+bY)=aE(X)+bE(Y) \]\[ \mathrm{Var}(X)=E(X^2)-(EX)^2 \]\[ \mathrm{Var}(aX+b)=a^2\mathrm{Var}(X) \] \[ \mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\,\mathrm{Cov}(X,Y) \]\[ \mathrm{Cov}(X,Y)=E\big[(X-EX)(Y-EY)\big]. \]If \(X,Y\) are independent, \(E(XY)=E(X)E(Y)\), hence \(\mathrm{Cov}(X,Y)=0\) and variances add.
Linearity (continuous case; the discrete case replaces integrals by sums): \(E(aX+bY)\)
Variance formula: with \(\mu=EX\), \(\mathrm{Var}(X)\)
Sum: expand \(\big[(X-EX)+(Y-EY)\big]^2\) and take expectations term by term.
Independence: \(E(XY)\)
If \(X_1,\dots,X_n\) are independent with common mean \(\mu\) and variance \(\sigma^2\), then \(\bar X=\frac1n\sum_{i=1}^n X_i\) has \(E(\bar X)=\mu\) and \(\mathrm{Var}(\bar X)=\sigma^2/n\).
By linearity \(E(\bar X)=\frac1n\sum E(X_i)=\mu\). By independence the covariances vanish, so \(\mathrm{Var}(\sum X_i)\)
If \(Y\ge 0\) and \(a\gt0\), then \(P(Y\ge a)\le E(Y)/a\). Consequently, for any \(X\) with mean \(\mu\) and variance \(\sigma^2\), and any \(\varepsilon\gt0\), \(P(|X-\mu|\ge \varepsilon)\)
Let \(I\) be the indicator of \(\{Y\ge a\}\). Then \(aI\le Y\) always (if \(I=1\), \(Y\ge a\); if \(I=0\), \(Y\ge 0\)). Taking expectations, \(aP(Y\ge a)\le E(Y)\). For Chebyshev apply Markov to \(Y=(X-\mu)^2\) and \(a=\varepsilon^2\), noting \(\{|X-\mu|\)
For iid \(X_i\) with mean \(\mu\) and finite variance, \(P(|\bar X_n-\mu|\ge\varepsilon)\)
By Chebyshev and Corollary 0.3, \(P(|\bar X_n-\mu|\ge\varepsilon)\)
\(Z\sim\mathcal N(0,1)\) has density \(\varphi(z)\)
(a) \(M_X(t)=e^{\mu t+\sigma^2t^2/2}\) for \(X\sim\mathcal N(\mu,\sigma^2)\). (b) If \(X,Y\) are independent then \(M_{X+Y}=M_XM_Y\). (c) If \(X_i\sim\mathcal N(\mu_i,\sigma_i^2)\) are independent, then \(\sum a_iX_i\sim\mathcal N\big(\sum a_i\mu_i,\ \sum a_i^2\sigma_i^2\big)\). In particular \(\bar X\sim\mathcal N(\mu,\sigma^2/n)\) for an iid normal sample.
(a) Complete the square: \(tz-\tfrac{z^2}{2}\)
(b) \(e^{t(X+Y)}=e^{tX}e^{tY}\) and independence gives \(E(e^{tX}e^{tY})=E(e^{tX})E(e^{tY})\) (Theorem 0.2 applied to functions of independent variables).
(c) By (a) and (b), \(M_{\sum a_iX_i}(t)\)
If \(X_1,X_2,\dots\) are iid with mean \(\mu\) and variance \(\sigma^2\in(0,\infty)\), then for every \(z\), \(P\!\left(\dfrac{\bar X_n-\mu}{\sigma/\sqrt n}\le z\right)\)
Let \(Y_i=(X_i-\mu)/\sigma\), so \(E Y_i=0\), \(E Y_i^2=1\), and \(Z_n\)
using \((1+c/n+o(1/n))^n\to e^c\). This is the MGF of \(\mathcal N(0,1)\), and the continuity theorem (F3) converts MGF convergence into convergence of the CDFs. The general proof (without an MGF) uses characteristic functions in the same way. ∎
If \(Z_1,\dots,Z_k\) are iid \(\mathcal N(0,1)\), then \(V=\sum_{j=1}^kZ_j^2\) has the chi-squared distribution with \(k\) degrees of freedom, \(V\sim\chi^2_k\). If \(Z\sim\mathcal N(0,1)\) and \(V\sim\chi^2_k\) are independent, then \(T=Z/\sqrt{V/k}\) has Student's t distribution with \(k\) degrees of freedom, \(T\sim t_k\).
(a) \(M_V(t)=(1-2t)^{-k/2}\) for \(t\lt\tfrac12\); \(E(V)=k\), \(\mathrm{Var}(V)=2k\). (b) If \(V_1\sim\chi^2_{a}\) and \(V_2\sim\chi^2_{b}\) are independent, \(V_1+V_2\sim\chi^2_{a+b}\). (c) \(t_k\) is symmetric about 0, and \(t_k\to\mathcal N(0,1)\) in distribution as \(k\to\infty\).
(a) For one term, \(E(e^{tZ^2})\)
(b) \(M_{V_1+V_2}\)
(c) \(Z\) and \(-Z\) have the same distribution and both are independent of \(V\), so the pairs \((Z,V)\) and \((-Z,V)\) have the same joint distribution; hence \(T\) and \(-T\) have the same distribution. As \(k\to\infty\), \(V/k\) is an average of \(k\) iid \(Z_j^2\) with mean 1, so \(V/k\to1\) in probability by the law of large numbers, and \(T=Z\cdot(V/k)^{-1/2}\to Z\) by Slutsky (F4). ∎
This single result is behind every exact interval and test for normal data in these notes.
Let \(X_1,\dots,X_n\) be iid \(\mathcal N(\mu,\sigma^2)\), \(n\ge2\), and \(S^2\)
Standardise: \(Z_i=(X_i-\mu)/\sigma\) are iid \(\mathcal N(0,1)\), with joint density \((2\pi)^{-n/2}\exp(-\tfrac12\|\mathbf z\|^2)\). Then \(\bar X=\mu+\sigma\bar Z\) and \((n-1)S^2/\sigma^2\)
An orthogonal rotation. The unit vector \(\mathbf u\)
Reading off the pieces. \(Y_1\)
(i) \(\bar Z\)
a standard normal divided by the square root of an independent \(\chi^2_{n-1}\) over its degrees of freedom: \(t_{n-1}\) by definition. ∎
The proof shows it geometrically: the \(n\) deviations \(Z_i-\bar Z\) are forced to sum to zero, so they live in an \((n-1)\)-dimensional subspace, and their squared length is a sum of only \(n-1\) independent squared normals. One dimension was used up by estimating the mean.
A statistical model is a family of distributions \(\{P_\theta:\theta\in\Theta\}\) for the data \(\mathbf X=(X_1,\dots,X_n)\). A statistic is any function of \(\mathbf X\) that does not involve \(\theta\). An estimator of \(\theta\) is a statistic \(\hat\Theta=g(\mathbf X)\) used to guess \(\theta\); its value \(\hat\theta=g(\mathbf x)\) on observed data is the estimate. The distribution of \(\hat\Theta\) under \(P_\theta\) is its sampling distribution.
\(\hat\Theta\) is unbiased if \(E_\theta(\hat\Theta)=\theta\) for every \(\theta\); its bias is \(E_\theta(\hat\Theta)-\theta\); its standard error is \(\sqrt{\mathrm{Var}_\theta(\hat\Theta)}\); its mean squared error is \(\mathrm{MSE}\)
Let \(m=E(\hat\Theta)\). Then \(\hat\Theta-\theta\)
The middle term is zero because \(E(\hat\Theta-m)=0\). ∎
If \(\mathrm{MSE}(\hat\Theta_n)\to0\) for every \(\theta\), then \(\hat\Theta_n\) is consistent. In particular an unbiased estimator whose variance tends to 0 is consistent.
Markov's inequality applied to \((\hat\Theta_n-\theta)^2\): \(P(|\hat\Theta_n-\theta|\ge\varepsilon)\)
For iid data with mean \(\mu\) and variance \(\sigma^2\): (a) \(\bar X\) is unbiased for \(\mu\) with standard error \(\sigma/\sqrt n\), and consistent. (b) \(S^2\) is unbiased for \(\sigma^2\), whereas \(\hat\sigma^2\)
(a) Corollary 0.3 and Theorem 1.2.
(b) First the algebraic identity \(\sum(X_i-\bar X)^2\)
Dividing by \(n-1\) gives an unbiased estimator; dividing by \(n\) gives mean \(\frac{n-1}{n}\sigma^2\).
(c) Write \(X=\sum_{i=1}^n B_i\) with \(B_i\) iid Bernoulli(\(p\)): \(E B_i=p\), \(E B_i^2=p\), \(\mathrm{Var}(B_i)=p-p^2\). Then \(\hat P=\bar B\) and Corollary 0.3 applies.
(d) By Theorem 0.9(ii), \(S^2=\frac{\sigma^2}{n-1}V\) with \(V\sim\chi^2_{n-1}\), and \(\mathrm{Var}(V)=2(n-1)\) by Theorem 0.8, so \(\mathrm{Var}(S^2)\)
For normal data, compare \(S^2\) with \(\hat\sigma^2=\frac{n-1}{n}S^2\). Using Theorems 1.1 and 1.3, \(\mathrm{MSE}(S^2)\)
The \(k\)-th population moment \(E_\theta(X^k)\) is a function of \(\theta\); the \(k\)-th sample moment \(\frac1n\sum X_i^k\) estimates it consistently by the law of large numbers (Theorem 0.5, when \(E X^{2k}\lt\infty\)). The method of moments sets as many sample moments equal to population moments as there are parameters, and solves.
For \(\mathcal N(\mu,\sigma^2)\): \(E X=\mu\), \(E X^2=\sigma^2+\mu^2\). Solving \(\bar x=\mu\), \(\frac1n\sum x_i^2=\sigma^2+\mu^2\) gives \(\hat\mu=\bar x\) and \(\hat\sigma^2\)
For observed data \(\mathbf x\), the likelihood is \(L(\theta)\)
If \(X\sim\mathrm{Bin}(n,p)\) and \(x\) is observed, the MLE is \(\hat p=x/n\).
\(L(p)=\binom nx p^x(1-p)^{n-x}\). If \(0\lt x\lt n\), \(\ell(p)=c+x\ln p+(n-x)\ln(1-p)\) on \((0,1)\), with \(\ell'(p)=\frac xp-\frac{n-x}{1-p}\). Setting \(\ell'=0\) gives \(x(1-p)=(n-x)p\), i.e. \(p=x/n\). Since \(\ell''(p)\)
For an iid \(\mathcal N(\mu,\sigma^2)\) sample with \(\sum(x_i-\bar x)^2\gt0\), the MLE is \(\hat\mu=\bar x\), \(\hat\sigma^2\)
\(\ell(\mu,\sigma^2)\)
So the MLE of \(\sigma^2\) is the biased \(\hat\sigma^2\). Maximum likelihood is a general recipe, not a guarantee of unbiasedness; its strength is large-sample behaviour (consistency and asymptotic efficiency, proved in later courses).
Let \(X_1,\dots,X_n\) be iid with density \(f_\theta\), and suppose differentiation in \(\theta\) may be taken inside the integrals below (F5) and the support does not depend on \(\theta\). Define the Fisher information \(I(\theta)\)
Let \(U\)
Unbiasedness says \(\int g(\mathbf x)\prod f_\theta(x_i)\,d\mathbf x\)
For \(\mathcal N(\mu,\sigma^2)\) with \(\sigma\) known, \(\frac{\partial}{\partial\mu}\ln f\)
Let \(T_1\)
Repeat a study independently \(m\) times with an exact \(100(1-\alpha)\%\) interval, and let \(C_m\) be the number of intervals that contain \(\theta\). Then \(C_m\sim\mathrm{Bin}(m,1-\alpha)\), and \(C_m/m\to1-\alpha\) in probability.
Let \(I_j\) indicate that interval \(j\) covers \(\theta\). The studies are independent, so the \(I_j\) are iid Bernoulli with success probability \(P_\theta(T_1\lt\theta\lt T_2)\)
For example, 100 independent 95% intervals cover the truth on average 95 times with standard deviation \(\sqrt{100\times0.95\times0.05}\)
Suppose \(Q=q(\mathbf X,\theta)\) has a distribution that does not depend on \(\theta\) (a pivot), with \(P(a\lt Q\lt b)=1-\alpha\). If, for every data set, \(\{\theta: a\)
The events \(\{a\lt q(\mathbf X,\theta)\lt b\}\) and \(\{T_1\lt\theta\lt T_2\}\) are the same event by construction, so they have the same probability, \(1-\alpha\), whatever the value of \(\theta\). ∎
For an iid \(\mathcal N(\mu,\sigma^2)\) sample, with \(t_{\alpha/2,n-1}\) the upper \(\alpha/2\) quantile of \(t_{n-1}\):
\[ \text{(a) } \sigma \text{ known: } \bar X\pm z_{\alpha/2}\frac{\sigma}{\sqrt n} \]\[ \text{(b) } \sigma \text{ unknown: } \bar X\pm t_{\alpha/2,n-1}\frac{S}{\sqrt n} \]are exact \(100(1-\alpha)\%\) confidence intervals for \(\mu\).
(b) By Theorem 0.9(iv), \(Q\)
Apply Theorem 3.3. (a) is identical with the pivot \(\frac{\bar X-\mu}{\sigma/\sqrt n}\sim\mathcal N(0,1)\) from Theorem 0.6. ∎
With \(n=130\), \(\bar x=36.81\), \(s=0.41\): \(t_{0.025,129}=1.979\), giving \(36.81\)
For iid data with finite variance \(\sigma^2\gt0\) (any shape), \(\bar X\pm z_{\alpha/2}S/\sqrt n\) is an approximate \(100(1-\alpha)\%\) confidence interval for \(\mu\).
Write \(\frac{\bar X-\mu}{S/\sqrt n}\)
Sample size. The half-width ("margin of error") is \(E=z_{\alpha/2}\sigma/\sqrt n\). Solving for \(n\) gives \(n=(z_{\alpha/2}\sigma/E)^2\), rounded up; for \(\sigma\approx0.41\) and \(E=0.05\), \(n=258.3\to259\).
Let \(q_1,q_2\) be the lower and upper \(\alpha/2\) quantiles of \(\chi^2_{n-1}\). For normal data, \(\Big(\dfrac{(n-1)S^2}{q_2},\ \dfrac{(n-1)S^2}{q_1}\Big)\) is an exact \(100(1-\alpha)\%\) confidence interval for \(\sigma^2\); square roots of the endpoints give one for \(\sigma\).
\(Q\)
Body temperature: \(q_1=99.45\), \(q_2=162.33\) on 129 df give \((0.134,\,0.218)\) for \(\sigma^2\) and \((0.37,\,0.47)\) °C for \(\sigma\). Unlike Theorem 3.5, there is no CLT rescue here: the result depends on normality even for large \(n\), because \(\mathrm{Var}(S^2)\) depends on the fourth moment of the data.
Let \(X_1,\dots,X_{n_1}\) iid \(\mathcal N(\mu_X,\sigma^2)\) and \(Y_1,\dots,Y_{n_2}\) iid \(\mathcal N(\mu_Y,\sigma^2)\) be independent samples with a common \(\sigma^2\), and \(S_p^2\)
so \(\bar X-\bar Y\)
Numerator. \(\bar X-\bar Y\) is a linear combination of independent normals, so by Theorem 0.6 it is \(\mathcal N\big(\mu_X-\mu_Y,\ \sigma^2(1/n_1+1/n_2)\big)\); standardising gives \(Z\sim\mathcal N(0,1)\).
Denominator. By Theorem 0.9(ii) applied to each sample, \((n_1-1)S_X^2/\sigma^2\sim\chi^2_{n_1-1}\) and \((n_2-1)S_Y^2/\sigma^2\sim\chi^2_{n_2-1}\), independent because the samples are. By Theorem 0.8(b), \(V\)
Independence. \(\bar X\) is independent of \(S_X^2\) (Theorem 0.9iii) and of the whole \(Y\)-sample; likewise for \(\bar Y\). So \(Z\) (a function of \(\bar X,\bar Y\)) is independent of \(V\) (a function of \(S_X^2,S_Y^2\)).
The ratio equals \(Z/\sqrt{V/(n_1+n_2-2)}\), since the unknown \(\sigma\) cancels: \(t_{n_1+n_2-2}\) by definition. The interval follows as in Theorem 3.4. ∎
Without assuming \(\sigma_X=\sigma_Y\), use \(\bar X-\bar Y\)
No exact pivot is known here (this is the Behrens–Fisher problem), so we approximate. Let \(W=S_X^2/n_1+S_Y^2/n_2\), the estimated variance of \(\bar X-\bar Y\). If \(W\) behaved like \(E(W)\cdot\chi^2_\nu/\nu\) for some \(\nu\), the studentised difference would be exactly \(t_\nu\) by the argument of Theorem 3.7. Choose \(\nu\) so that this model has the right mean and variance. The model has \(\mathrm{Var}(W)\)
By Theorem 1.3, \(E(W)=\sigma_X^2/n_1+\sigma_Y^2/n_2\), and by Theorem 1.3(d) and independence, \(\mathrm{Var}(W)\)
Bounds. Put \(a=S_X^2/n_1\), \(b=S_Y^2/n_2\), \(k_1=n_1-1\), \(k_2=n_2-1\), so \(\nu=(a+b)^2/(a^2/k_1+b^2/k_2)\). Upper: by Cauchy–Schwarz, \((a+b)^2\)
The bounds explain two rules of thumb: using \(\min(n_1,n_2)-1\) degrees of freedom by hand is conservative (wider interval), and Welch can never claim more degrees of freedom than the pooled method. Women versus men in the temperature data: difference \(0.16\), SE \(0.070\), \(\nu\approx127.7\) from these rounded summaries (127.5 from the full data), interval \((0.02,\ 0.30)\) °C.
If \(X\sim\mathrm{Bin}(n,p)\) with \(0\lt p\lt1\), then \(\hat P\)
\(\hat P\) is the mean of \(n\) iid Bernoulli(\(p\)) variables (Theorem 1.3c), so by the CLT \(\frac{\hat P-p}{\sqrt{p(1-p)/n}}\)
The approximation is reasonable when there are at least about 5 successes and 5 failures in each sample; for smaller counts, exact binomial methods are preferred. Examples: 64 of 80 seeds gives \((0.71,\ 0.89)\); 45/150 versus 30/150 gives \((0.003,\ 0.197)\) for the difference.
A hypothesis test of \(H_0:\theta\in\Theta_0\) against \(H_1:\theta\in\Theta_1\) is a rule that rejects \(H_0\) when the data fall in a rejection region \(R\). Its power function is \(\pi(\theta)\)
If the test rejects for large values of a statistic \(T\), the P-value of observed data is \(p(\mathbf x)\)
Let \(H_0:\theta=\theta_0\) be simple. (a) If \(T\) has a continuous CDF under \(H_0\), then \(p(\mathbf X)\sim\mathrm{Uniform}(0,1)\) under \(H_0\), so the rule "reject if \(p\le\alpha\)" has size exactly \(\alpha\). (b) In general (for example a discrete \(T\)), \(P_{\theta_0}(p(\mathbf X)\le\alpha)\)
Let \(G(t)=P_{\theta_0}(T\ge t)\), a non-increasing function, so that \(p(\mathbf X)=G(T)\).
(b) Fix \(\alpha\), and let \(c=\inf\{t: G(t)\le\alpha\}\). Because \(G\) is non-increasing, \(\{G(T)\le\alpha\}\) is either \(\{T\ge c\}\) or \(\{T\gt c\}\). In the first case \(G(c)\le\alpha\) (the infimum is attained), so \(P(T\ge c)=G(c)\le\alpha\). In the second case, the events \(\{T\ge t\}\) increase to \(\{T\gt c\}\) as \(t\downarrow c\), so \(P(T\gt c)\)
(a) If the CDF of \(T\) is continuous, \(G\) is continuous, so by the intermediate value theorem there is a \(c\) with \(G(c)=\alpha\), and then \(P(T\ge c)=\alpha\) exactly and \(\{G(T)\le\alpha\}=\{T\ge c\}\) up to an event of probability zero. Hence \(P(p\le\alpha)=\alpha\) for every \(\alpha\): the CDF of \(p(\mathbf X)\) is that of a Uniform(0,1) variable. ∎
This is why, in the hypothesis-testing demonstration, the histogram of P-values is flat when \(H_0\) is true (and piles up near 0 when it is false). It also shows what a P-value is not: it is a random variable computed assuming \(H_0\), not the probability that \(H_0\) is true.
(a) If \(C(\mathbf X)\) is a \(100(1-\alpha)\%\) confidence set for \(\theta\), then for each \(\theta_0\) the test "reject \(H_0:\theta=\theta_0\) iff \(\theta_0\notin C(\mathbf X)\)" has level \(\alpha\). (b) Conversely, if for each \(\theta_0\) a level-\(\alpha\) test has acceptance region \(A(\theta_0)\), then \(C(\mathbf x)\)
(a) \(P_{\theta_0}(\text{reject})\)
For the \(t\) procedures: \(|\bar x-\mu_0|\)
For iid normal data and \(T=\frac{\bar X-\mu_0}{S/\sqrt n}\), the tests "reject \(H_0:\mu=\mu_0\) if \(|T|\gt t_{\alpha/2,n-1}\)" (two-sided), "if \(T\gt t_{\alpha,n-1}\)" (\(H_1:\mu\gt\mu_0\)) and "if \(T\lt-t_{\alpha,n-1}\)" (\(H_1:\mu\lt\mu_0\)) all have size exactly \(\alpha\), and each is equivalent to "P-value \(\le\alpha\)" (up to an event of probability 0).
Under \(H_0\), \(T\sim t_{n-1}\) (Theorem 0.9iv with \(\mu=\mu_0\)). By the definition of the quantiles and symmetry, \(P(|T|\gt t_{\alpha/2,n-1})\)
Body temperature: \(t\)
For the one-sided \(z\)-test that rejects \(H_0:\mu=\mu_0\) in favour of \(H_1:\mu\gt\mu_0\) when \(Z\)
and, when \(d\gt0\), power at least \(1-\beta\) requires
\[ n\ \ge\ \Big(\frac{z_\alpha+z_\beta}{d}\Big)^2 . \]Under the true mean \(\mu\), \(Z\)
Consequences, all visible in the formula: power equals \(\alpha\) at \(d=0\), increases with \(d\), increases with \(n\) when \(d\gt0\), and decreases as \(\alpha\) is made smaller. For \(d=0.5\), \(\alpha=0.05\), 80% power: \(n\)power.t.test gives 26.1). The price of not knowing \(\sigma\) is two observations.
To test a simple \(H_0:f=f_0\) against a simple \(H_1:f=f_1\), consider the likelihood-ratio test that rejects when \(f_1(\mathbf x)\gt k f_0(\mathbf x)\), with \(k\ge0\) chosen so that its size is \(\alpha\). Every other test of level \(\alpha\) has power no greater than this one.
Describe a test by \(\phi(\mathbf x)\in\{0,1\}\) (1 = reject). Let \(\phi^*\) be the likelihood-ratio test and \(\phi\) any test with \(E_0\phi\le\alpha=E_0\phi^*\). For every \(\mathbf x\),
\[ \big(\phi^*(\mathbf x)-\phi(\mathbf x)\big)\big(f_1(\mathbf x)-kf_0(\mathbf x)\big)\ \ge\ 0, \]because when \(f_1\gt kf_0\), \(\phi^*=1\ge\phi\), and when \(f_1\le kf_0\), \(\phi^*=0\le\phi\). Integrate over \(\mathbf x\):
\[ E_1\phi^*-E_1\phi\ \ge\ k\,\big(E_0\phi^*-E_0\phi\big)\ \ge\ 0 . \]So the power of \(\phi^*\) is at least the power of \(\phi\). ∎
For iid \(\mathcal N(\mu,\sigma^2)\) with \(\sigma\) known, test \(\mu=\mu_0\) against a single \(\mu_1\gt\mu_0\). The likelihood ratio is
\[ \frac{f_1(\mathbf x)}{f_0(\mathbf x)}=\exp\!\Big(\frac{\mu_1-\mu_0}{\sigma^2}\sum x_i-\frac{n(\mu_1^2-\mu_0^2)}{2\sigma^2}\Big), \]an increasing function of \(\bar x\). So "ratio \(\gt k\)" is "\(\bar x\gt c\)", and size \(\alpha\) forces \(c=\mu_0+z_\alpha\sigma/\sqrt n\): the z-test. The rejection region does not depend on which \(\mu_1\gt\mu_0\) we picked, so the z-test is simultaneously most powerful against every \(\mu_1\gt\mu_0\): it is uniformly most powerful.
Let \(X\sim\mathrm{Bin}(n,p)\), \(H_0:p=p_0\), \(H_1:p\gt p_0\). (a) The exact test with P-value \(P_{p_0}(X\ge x)\) has level \(\alpha\) (usually size strictly below \(\alpha\)). (b) The approximate test based on \(Z\)
(a) Theorem 4.1(b) with \(T=X\). The size is \(P_{p_0}(X\ge c)\) for the cut-off \(c\) found there, and because \(X\) takes only the values \(0,\dots,n\), this usually cannot equal \(\alpha\) exactly. (b) Under \(H_0\), \(Z\to\mathcal N(0,1)\) by the CLT (Theorem 1.3c gives the mean and variance of \(\hat P\)). No Slutsky step is needed: under \(H_0\) the standard error is known, which is why the test uses \(p_0\) rather than \(\hat P\). ∎
14 of 18 tasters prefer a new recipe; \(H_0:p=0.5\). Exact: \(P(X\ge14)=0.015\). Approximate: \(z=2.36\), \(p=0.009\). At level 0.05 the exact rejection region is \(X\ge13\), whose size is \(P(X\ge13)=0.048\), because \(P(X\ge12)=0.119\) is too large: no region of the form \(X\ge c\) has size exactly 0.05. Similarly, the seed rule "reject \(p=0.7\) if \(X\le10\) out of 20" has size \(0.048\), and power \(0.59\) at \(p=0.5\) and \(0.87\) at \(p=0.4\).
By the duality theorem (4.2), each interval in Section 3.4 gives a test of \(H_0:\mu_X=\mu_Y\): the pooled statistic \(\frac{\bar X-\bar Y}{S_p\sqrt{1/n_1+1/n_2}}\) is exactly \(t_{n_1+n_2-2}\) under \(H_0\) when the variances are equal (Theorem 3.7), and the Welch statistic is approximately \(t_\nu\) without that assumption (Theorem 3.8).
If \(n_1=n_2=n\), the pooled and Welch statistics are identical; they differ only in their degrees of freedom.
With \(n_1=n_2=n\), \(S_p^2=\frac{S_X^2+S_Y^2}{2}\), so \(S_p^2\big(\frac1n+\frac1n\big)\)
Women versus men (65 each): both statistics equal \(t=2.28\), on 127.7 and 128 df, \(p=0.024\). When the sample sizes are very unequal and the variances differ, the pooled test's size can be far from \(\alpha\), which is why Welch is the safer default (Welch, 1947).
Let \((X_i,Y_i)\) be measured on the same subject, \(D_i=X_i-Y_i\), with \(\mathrm{Var}(X)=\sigma_X^2\), \(\mathrm{Var}(Y)=\sigma_Y^2\) and correlation \(\rho\). Then \(\mathrm{Var}(D)\)
Theorem 0.2 with \(Y\) replaced by \(-Y\): \(\mathrm{Var}(X-Y)\)
The sample standard deviations are 0.493 h (before) and 0.512 h (after), with correlation \(r=0.85\) between a person's two nights. The variance of the differences is \(0.076\), against \(0.493^2+0.512^2\approx0.505\) if the samples were independent: pairing removes about 85% of the noise. Paired test: mean difference \(0.425\), \(t=4.35\) on 7 df, \(p=0.0034\). Ignoring the pairing (Welch): \(t=1.69\), \(p=0.11\).
For independent \(X\sim\mathrm{Bin}(n_1,p_X)\), \(Y\sim\mathrm{Bin}(n_2,p_Y)\) and \(H_0:p_X=p_Y\), let \(\hat P=\frac{X+Y}{n_1+n_2}\). Under \(H_0\), as \(n_1,n_2\to\infty\),
\[ Z=\frac{\hat P_X-\hat P_Y}{\sqrt{\hat P(1-\hat P)\big(\frac1{n_1}+\frac1{n_2}\big)}}\ \longrightarrow\ \mathcal N(0,1). \]Under \(H_0\) both samples share one \(p\). By Theorem 1.3(c) and independence, \(\hat P_X-\hat P_Y\) has mean 0 and variance \(p(1-p)\big(\frac1{n_1}+\frac1{n_2}\big)\), and the CLT (applied to each sample) makes its standardised version approximately \(\mathcal N(0,1)\). Under \(H_0\), \(X+Y\sim\mathrm{Bin}(n_1+n_2,p)\), so \(\hat P\to p\) in probability, and Slutsky lets us replace \(p\) by \(\hat P\). Using all \(n_1+n_2\) observations to estimate the common \(p\) is what the null hypothesis permits; the confidence interval (Theorem 3.9) cannot assume \(p_X=p_Y\), so it uses the separate estimates. ∎
Reminder emails: 45/150 versus 30/150, \(\hat p=0.25\), \(z=2.00\), one-sided \(p=0.023\).
\(S\) is biased. Show that \(E(S)\le\sigma\), with equality only if \(S\) is constant. (Hint: \(\mathrm{Var}(S)\ge0\).)
\(0\)
Two estimators for a uniform. Let \(X_i\) be iid Uniform\((0,\theta)\). Show that the method-of-moments estimator is \(2\bar X\) and the MLE is \(M=\max_i X_i\). Show \(\mathrm{MSE}(2\bar X)\)
\(E X=\theta/2\), so \(\bar x=\theta/2\) gives \(2\bar X\). \(L(\theta)=\theta^{-n}\) if \(\theta\ge\max x_i\) and 0 otherwise; it is decreasing on \([\max x_i,\infty)\), so \(\hat\theta=M\) (calculus fails here: the maximum is at a boundary). \(2\bar X\) is unbiased with variance \(4\cdot\frac{\theta^2/12}{n}\)
▶ Check it in a demo: the uniform maximum in the likelihood demonstration
An exponential rate. For iid Exponential(\(\lambda\)) data (density \(\lambda e^{-\lambda x}\), mean \(1/\lambda\)), find the MLE of \(\lambda\). Given that \(\sum X_i\) has the Gamma(\(n,\lambda\)) distribution, for which \(E\big(1/\sum X_i\big)\)
\(\ell(\lambda)\)
▶ Check it in a demo: the exponential rate in the likelihood demonstration
Two-sided sample size. Adapt Theorem 5.2 to the two-sided z-test (ignore the negligible probability of rejecting in the wrong direction). How many observations give 80% power at \(d=0.5\), \(\alpha=0.05\)?
Rejection on the correct side has probability \(\Phi(\sqrt n|d|-z_{\alpha/2})\), so \(n\)
▶ Check it in a demo: the two-sided test in the power demonstration
Seven steps, filled in. A manufacturer claims a mean battery life of 10 h; \(n=25\), \(\bar x=9.6\), \(s=0.9\). Test against \(\mu\lt10\), stating which theorem justifies each step.
Model: iid \(\mathcal N(\mu,\sigma^2)\). Hypotheses: \(H_0:\mu=10\), \(H_1:\mu\lt10\). Statistic: \(T=\frac{\bar X-10}{S/5}\), which is \(t_{24}\) under \(H_0\) by Theorem 0.9(iv). Observed \(t=-0.4/0.18=-2.22\); left-sided P-value \(P(T_{24}\le-2.22)=0.018\). By Theorems 4.1 and 5.1, rejecting when \(p\le0.05\) has size 0.05, so reject \(H_0\): moderate evidence that the mean life is below 10 h. The 95% interval \((9.23,\,9.97)\) excludes 10, consistent with Theorem 4.2 (whose two-sided test also rejects).
▶ Check it in a demo: the battery data in the test–interval demonstration
Uniform P-values, by simulation. In R, simulate 10,000 samples of size 10 from \(\mathcal N(0,1)\), compute the two-sided P-value of t.test(x, mu = 0) for each, and plot a histogram. Which theorem predicts the shape? What changes if you use mu = 0.5?
p <- replicate(1e4, t.test(rnorm(10), mu = 0)$p.value); hist(p) gives a flat histogram (Theorem 4.1a), with about 5% of values below 0.05. With mu = 0.5 the null is false (the true mean is 0, so \(d=-0.5\)), and the P-values pile up near 0; the fraction below 0.05 estimates the power (about 0.29 for this \(n\) and \(d\)).
▶ Check it in a demo: flat P-values in the hypothesis-testing demonstration
Each demonstration runs in your browser and is tied to the results it illustrates; the ▶ Demo boxes above open them already set to the relevant case. Simulations use a fixed seed, so a class sees the same pictures.
| Demonstration | What you can see | Results |
|---|---|---|
| Averages on trial: the law of large numbers and the central limit theorem | Averages gather at the mean; standardised, they become the bell curve for any population shape | Cor. 0.3, Thm 0.4, 0.5, 0.7, 3.5 |
| Why Student's t? Inside the normal-sample theorem | \(\bar X\) and \(S\) independent, the chi-squared and t pivots, and how each fails without normality | Thm 0.8, 0.9, 3.4, 3.5, 3.6 |
| Estimators on trial: bias, spread and the n − 1 | Sampling distributions of competing estimators: bias, standard error and MSE | Thm 1.1, 1.3, Example, Thm 2.4 |
| Maximum likelihood: climbing the likelihood | The log-likelihood of each sample, its top and curvature, and the MLE's spread against the Cramér–Rao bound | Thm 2.2, 2.3, 2.4, Ex. 2, 3 |
| Confidence intervals: does the method catch the truth? | A hundred intervals at a time: coverage, width, and the failure of 1.96 for small n | Def. 3.1, Thm 3.2, 3.3, 3.4 |
| Two samples: pooled or Welch, paired or not? | When the pooled test breaks, why Welch holds, and how much pairing gains | Thm 3.7, 3.8, Prop. 8.1, Thm 8.2 |
| Counting successes: exact and approximate methods for proportions | Exact sizes of binomial tests and the exact coverage of the Wald interval | Thm 3.9, 4.1(b), 7.1 |
| A hypothesis test on trial: P-values, errors and power | Thousands of t-tests: false alarms at rate \(\alpha\), flat P-values, and simulated power | Thm 4.1, 5.1, 5.2, Ex. 6 |
| Tests and intervals: two views of one object | The P-value as a function of \(\mu_0\): the values not rejected form the interval | Thm 4.2, 3.4, 5.1, Ex. 5 |
| Power and sample size: two bell curves | The areas \(\alpha\), \(\beta\) and power, the power curve, and the sample size for a target power | Thm 5.2, 6.1, Ex. 4 |
The sleep, seed, taste-test, reminder-email and battery examples are illustrative numbers constructed for these notes. All numerical results were computed in R.