Resampling: the bootstrap, permutation tests and where they break

STAT · Chapter 612 min readAsked at Two Sigma, QuantCo, AQR, Citadel

Assumes Hypothesis testing: errors, power and which test to use.

After this lesson you should be able to

  • Describe the bootstrap and what it estimates.
  • Use a permutation test where no distributional assumption is available.
  • Say why the ordinary bootstrap fails on a time series, and what replaces it.

Resampling replaces a distributional assumption with computation: instead of deriving the sampling distribution of a statistic, you generate it by re-drawing from the data you have. It is the right tool when the statistic is awkward — a median, a Sharpe ratio, a maximum drawdown — and the wrong one the moment the observations are not exchangeable.

Definition 6.1

The bootstrap

Bootstrap, resample n points with replacement, recompute, repeat\text{resample } n \text{ points with replacement, recompute, repeat} — Treat the sample as if it were the population, draw new samples of the same size from it with replacement, and compute your statistic on each. The spread of those values estimates the sampling distribution — which gives you a standard error or a confidence interval for any statistic at all, including ones with no closed form.

Drawn at least once63.2Never drawn36.8
Figure 6.2 · What one bootstrap resample contains. Resampling nn points with replacement leaves each one out with probability (1−1/n)n(1-1/n)^n, which converges to 1/e1/e. So every resample is built from about 63%63\% of the data and the remaining third is what makes the resamples differ from each other.

Why re-using the same data is not circular. The obvious objection is that you cannot manufacture information by shuffling what you already have, and you cannot. What the bootstrap estimates is not the parameter — that still comes from the sample — but the *variability* of the estimate, and for that the empirical distribution is a legitimate stand-in for the true one. The plug-in step is the whole trick: the relationship between population and sample is approximated by the relationship between sample and resample.

MethodQuestion it answersNote
BootstrapHow variable is my estimate?Works for almost any statistic
JackknifeSame, by leaving one out at a timeCheaper, and fails on non-smooth statistics like the median
Permutation testCould this difference be chance?Exact under exchangeability; no distribution assumed
Block bootstrapVariability with dependent dataResample blocks, not points
Stationary bootstrapSame, with random block lengthsAvoids artefacts at fixed block boundaries
Table 6.3 · What each method is for. The permutation test is the one to reach for when you are asked "is this difference real?" and cannot justify any distributional assumption — it makes only the assumption already implied by the null.

Derivation 6.4

A permutation test

Under the null the labels carry no information, so any relabelling is equally likely.

  1. tobs=xˉA−xˉBt_{\text{obs}} = \bar{x}_A - \bar{x}_B

    Compute the statistic on the real labelling.

  2. shuffle the labels, recompute t(k), many times\text{shuffle the labels, recompute } t^{(k)}, \text{ many times}

    This generates the null distribution directly.

  3. p=#{∣t(k)∣≥∣tobs∣}+1K+1p = \frac{\#\{|t^{(k)}| \ge |t_{\text{obs}}|\} + 1}{K + 1}

    The +1+1s include the observed value and keep the test valid.

an exact p-value, assuming only exchangeability\text{an exact } p\text{-value, assuming only exchangeability}

The rest of this lesson is in Premium

You have read the opening. 11 more sections follow, including 5 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.