p-values, p-hacking and the multiple-testing problem

STAT · Chapter 513 min readAsked at Two Sigma, Citadel, DE Shaw, AQR

Assumes The law of large numbers and the central limit theorem.

After this lesson you should be able to

  • State precisely what a p-value is and list three things it is not.
  • Compute the family-wise error rate for a set of independent tests.
  • Choose between Bonferroni and false-discovery-rate control, and say why.

Signal research is thousands of hypothesis tests wearing a trench coat. Understanding what a p-value measures, and what happens to it when you run a test ten thousand times, is the difference between a research process that discovers things and one that manufactures them.

Definition 5.1

What a p-value is

p-value, p=Pr⁡( T≥tobs∣H0 )p = \Pr(\,T \ge t_{\text{obs}} \mid H_0\,) — The probability of seeing data at least as extreme as what you observed, *given that the null hypothesis is true*. It is a statement about data under an assumption, not about the assumption.

Proposition 5.2

Three things it is not

A p-value is not the probability the null is true. It is not the probability your result is a fluke. And one minus the p-value is not the probability the effect is real. All three require a prior, which the p-value does not have.

Holds when

  • Pr⁡(data∣H0)\Pr(\text{data} \mid H_0) is not Pr⁡(H0∣data)\Pr(H_0 \mid \text{data}) — this is the same inversion error as the disease-testing brainteaser.
  • A p-value of 0.05 is routinely consistent with a posterior probability of the null well above 20%.

Equation 5.3

The family-wise error rate

The probability of at least one false positive across mm independent tests at level α\alpha.

FWER=1−(1−α)m\mathrm{FWER} = 1 - (1 - \alpha)^m
mm
The number of hypotheses tested — including the ones you tried and discarded.
α\alpha
The per-test significance level.

Example 5.4

You backtest 100 signals, none of which has any edge, and keep any that is significant at the 5%5\% level. How many do you expect to keep, and what is the chance you keep at least one?

Show the worked solutionHide the worked solution

Worked solution

  1. Formula
    E[false positives]=mα,FWER=1−(1−α)m\mathbb{E}[\text{false positives}] = m\alpha, \qquad \mathrm{FWER} = 1 - (1-\alpha)^m
  2. Substitute
    m=100, α=0.05m = 100, \ \alpha = 0.05
  3. Solve
    E=100×0.05=5\mathbb{E} = 100 \times 0.05 = 5
  4. FWER=1−0.95100≈1−0.0059\mathrm{FWER} = 1 - 0.95^{100} \approx 1 - 0.0059
  5. Answer
    five signals, and near certainty of at least one\text{five signals, and near certainty of at least one}

Sanity check. Every one of those five will have a plausible story attached, because you will construct one. That is the whole problem.

  • Clears the 5%5\% bar anyway — 5 of 100
  • Correctly discarded — 95 of 100
Figure 5.5 · A hundred signals, none of them real. Not one of these hundred has any edge. Five survive the cut, and each of the five will arrive with a plausible story attached, because you will be the one constructing it.

Two different fears. Bonferroni controls the chance of making even one false discovery; Benjamini–Hochberg controls the expected share of your discoveries that are false. A researcher running thousands of backtests can live with one live signal in twenty being junk, but cannot live with a procedure so strict it never rejects anything. Which fear you are managing decides which correction is right.

The rest of this lesson is in Premium

You have read the opening. 12 more sections follow, including 4 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.