Hypothesis testing
& inference

A concrete question

Statistical inference asks whether the data are consistent with a claim about the process that produced them. Suppose one student claims the mean economics grade in a population is at most 65%, while another claims it exceeds 65%. We cannot observe the whole population; we draw a sample and must decide, from that sample alone, which claim the evidence supports. Writing the unknown population mean as \(\mu\), the two claims become a null and an alternative hypothesis:

$$H_0:\ \mu\le65 \qquad\text{versus}\qquad H_A:\ \mu>65.$$

This is a one-sided test: the alternative points in a single direction. The entire machinery of testing is a disciplined way of deciding between these two statements while controlling the chance of getting it wrong.

The sampling distribution of the estimator

Take a sample of \(N=25\) grades, modelled as draws \(x\sim N(\mu,\sigma^2)\) with, say, known variance \(\sigma^2=100\). The natural estimator of \(\mu\) is the sample mean \(\hat\mu\). Its sampling distribution — the distribution of the estimator across hypothetical repeated samples — is

$$\hat\mu\sim N\!\left(\mu,\ \frac{\sigma^2}{N}\right), \qquad \frac{\sigma^2}{N}=\frac{100}{25}=4.$$

So \(\hat\mu\) is centred on the truth \(\mu\) with standard error \(\sigma/\sqrt{N}=10/5=2\). Everything that follows is read off this distribution. This is exactly why sampling distributions matter: they turn a single number computed from data into a probability statement we can test.

p-values

Fix the null at its boundary, \(\mu=65\). A p-value is the probability, computed under that null, of observing a sample mean at least as extreme as the one we actually saw. Using the \(N(65,4)\) distribution,

$$\Pr\{\hat\mu>67\mid\mu=65\}=0.159, \qquad \Pr\{\hat\mu>70\mid\mu=65\}=0.0062.$$

A sample mean of 67 is unremarkable if the null holds (it would happen about 16% of the time), so it is weak evidence against \(H_0\). A sample mean of 70 is very unlikely under the null (about 0.6%), so it is strong evidence against it. Small p-values are evidence against the null; the smaller the p-value, the more surprising the data would be if \(H_0\) were true.

Significance level, size and the critical value

To make a firm decision we fix in advance how much risk of a false rejection we will tolerate. The significance level (or size) is the probability of rejecting the null when it is in fact true — a Type I error. The conventional choice is 5% (with 10% and 1% also common). We then find the critical value \(c\) such that

$$\Pr\{\hat\mu>c\mid\mu=65\}=0.05.$$

With the \(N(65,4)\) sampling distribution this gives \(c=68.29\). The decision rule is simply: accept \(H_0\) if \(\hat\mu\le68.29\), and reject \(H_0\) in favour of \(H_A\) if \(\hat\mu>68.29\). By construction, if the null is true we will wrongly reject it only 5% of the time.

The standardised (z) test

Comparing \(\hat\mu\) with a case-specific number like 68.29 is awkward. The standard move is to standardise. Subtracting the null mean and dividing by the standard error, the rule \(\hat\mu\le68.29\) is equivalent to

$$\hat\mu-65\le3.29 \quad\Longleftrightarrow\quad \frac{\hat\mu-65}{2}\le1.645.$$

The quantity on the left is the z-statistic,

$$z=\frac{\hat\mu-\mu_0}{\sigma/\sqrt{N}},$$

which under the null has a standard normal distribution. We now compare \(z\) with a fixed critical value from the normal tables: for a one-sided 5% test that value is 1.645, and we reject when \(z>1.645\). Standardising means the same handful of critical values (1.645, 1.96, 2.33) works for every test, whatever the units of the data.

The same logic across the course

Every test you meet in ECON30401 is this template with a different estimator plugged in. Testing whether a sample autocorrelation is zero compares \(\sqrt{T}\,\hat\rho_T(k)\) with 1.96; testing a single regression or AR coefficient uses a \(t\)-statistic against a \(t\) (or, in large samples, normal) critical value; testing several coefficients jointly uses an \(F\)- or \(\chi^2\)-statistic. The moving parts are always the same: a null and alternative, an estimator with a known sampling distribution, a significance level that fixes the Type I error rate, and a critical value (or p-value) that turns the estimate into a decision. A two-sided alternative \(H_A:\mu\neq\mu_0\) simply splits the rejection probability across both tails (critical value 1.96 at 5%), and inverting the accept region gives the associated confidence interval. It is also worth separating statistical from economic significance: with enough data a tiny, economically trivial effect can be statistically significant, and vice versa.

Check your understanding

With \(\hat\mu\sim N(\mu,4)\), a sample gives \(\hat\mu=69\). Test \(H_0:\mu\le65\) against \(H_A:\mu>65\) at 5%.

The standard error is \(\sigma/\sqrt N=2\). The z-statistic under \(\mu_0=65\) is \(z=(69-65)/2=2.0\). The one-sided 5% critical value is 1.645. Since \(2.0>1.645\), we reject \(H_0\): the evidence supports a mean above 65. (Equivalently, \(\hat\mu=69>68.29=c\).)

Interpret a p-value of 0.0062 for the same test.

It is the probability, if the null \(\mu=65\) were true, of drawing a sample mean at least as large as the one observed. At 0.62% this is very unlikely under the null, so the data are strong evidence against \(H_0\); we would reject at the 5% and even the 1% level. Note the p-value is a statement about the data under the null, not the probability that the null is true.

What is a Type I error, and how does the significance level relate to it?

A Type I error is rejecting the null when it is actually true. The significance level (size) is exactly the probability of that error, which we fix in advance — typically 5%. Choosing the critical value so that \(\Pr\{\text{reject}\mid H_0\ \text{true}\}=0.05\) is precisely controlling the Type I error rate at 5%. (Failing to reject a false null is the complementary Type II error.)

Related notes