Statistics and inference

Family-wise error rate

Says how many worthless signals will look significant once you have screened enough of them.

FWER=1−(1−α)m,E[false positives]=mα\mathrm{FWER} = 1 - (1-\alpha)^m, \qquad \mathbb{E}[\text{false positives}] = m\alpha

Where

mm
Every hypothesis tested, including the ones you abandoned.

Assumptions

  • Independent tests. Correlated candidates give a smaller effective mm, but rarely by much.

Sanity check. 100 tests at 5% yield five hits and near-certainty of at least one. Each will have a story attached.

Where this is taught