Hypothesis testing: errors, power and which test to use

A hypothesis test is a decision rule with two ways to be wrong, and the whole design consists of choosing how much of each you will tolerate. In research the type that matters is usually the one nobody controls: failing to detect a real effect because the sample was never large enough.

Decision	Null is true	Null is false
Reject	Type I error, probability $\alpha$	Correct — power, $1 - \beta$
Fail to reject	Correct	Type II error, probability $\beta$

Table 4.1 · The two errors. You choose

\alpha

directly.

\beta

follows from the effect size, the variance and the sample size — which is why power is something you design for rather than observe.

Equation 4.2

Sample size from power

The two-sample size needed to detect a difference $\delta$ with power $1-\beta$ at level $\alpha$ .

n \approx \frac{2\left(z_{\alpha/2} + z_{\beta}\right)^2\sigma^2}{\delta^2}

$z_{\alpha/2} + z_\beta$: For $\alpha = 5\%$ and $80\%$ power this is $1.96 + 0.84 = 2.80$ .
$\delta$: The minimum effect you care about detecting, chosen before you look at the data.

Example 4.3

You want to detect a $2\%$ improvement in a conversion rate from a base of $20\%$ , at $5\%$ significance and $80\%$ power. How many per arm?

Show the worked solutionHide the worked solution

Worked solution

Formula $n = \frac{2(z_{\alpha/2} + z_\beta)^2 p(1-p)}{\delta^2}$
Substitute $p = 0.20, \ \delta = 0.02, \ (1.96 + 0.84)^2 = 7.84$
Solve $p(1-p) = 0.16$
$n = \frac{2 \times 7.84 \times 0.16}{0.0004}$
Answer $\approx 6{,}300 \text{ per arm}$

Sanity check. Halving the effect you want to detect quadruples the sample, since $\delta$ enters squared. That single fact explains why most business experiments are underpowered: the effects are small and nobody budgets for the square.

Figure 4.4 · What a $5\%$ test actually rejects. Both shaded tails together hold five per cent of the distribution, which is what

\alpha = 5\%

buys you: reject outside

\pm 1.96

and you will be wrong one time in twenty when the null is true. Power is a separate picture — it asks how much of a *different* distribution lands in the same two tails.

Question	Test	Assumes
Mean against a value, $\sigma$ known	$z$ -test	Normal or large $n$
Mean against a value, $\sigma$ estimated	$t$ -test	Normal or large $n$
Two means	Two-sample $t$	Independent groups; Welch if variances differ
Before and after on the same units	Paired $t$	Differences are what you test
Two proportions	Two-proportion $z$	Enough successes in each arm
Variances	$F$ -test	Very sensitive to non-normality
Several means	ANOVA	Equal variances; follow up carefully
Categorical association	Chi-square	Expected counts above about five
No distributional assumption	Permutation test	Exchangeability under the null

Table 4.5 · Which test. The paired row is the one candidates miss. When the same units are measured twice, pairing removes the between-unit variance and can cut the required sample by an order of magnitude.

Hypothesis testing: errors, power and which test to use

After this lesson you should be able to

Sample size from power

The rest of this lesson is in Premium