The overfitting problem: deflated Sharpe and what discipline looks like

This is the central problem of the field. Financial data are short and noisy, computers are fast, and a sufficiently determined search will find something in any sample. Everything else in signal research is a defence against that fact.

Equation 8.1

What luck produces

The expected best Sharpe ratio among $N$ worthless strategies over $T$ years. Any reported Sharpe must clear this before it is evidence of anything.

\mathbb{E}\left[\max_{i \le N} \mathrm{SR}_i\right] \approx \sqrt{\frac{2\ln N}{T}}

$N$: Every variant tried — parameter settings, universes, horizons, abandoned attempts.
$\sqrt{2\ln N}$: Grows slowly, so the fix is more data far more than fewer trials.

Trials	2 years	5 years	10 years	20 years
10	1.52	0.96	0.68	0.48
100	2.15	1.36	0.96	0.68
1,000	2.63	1.66	1.18	0.83
10,000	3.03	1.92	1.36	0.96

Table 8.2 · The luck threshold. Read the row for how many things you tried and the column for how much data you have. A Sharpe of 1.5 on two years is below what luck gives after a hundred attempts, and comfortably above the bar after ten attempts on twenty years.

Proposition 8.3

Minimum backtest length

Invert the formula: to claim a Sharpe of $S$ after $N$ trials, you need roughly $T \ge 2\ln N / S^2$ years. A Sharpe of $1$ after a thousand trials needs about fourteen years — which is more history than most signals have, and is the honest reason so many published results do not replicate.

Holds when

The requirement falls with the square of the Sharpe, so a genuinely strong signal needs far less data.
It rises only logarithmically in the trial count, so cutting trials from 1,000 to 100 helps surprisingly little.
More independent data is the only strong lever, which is why breadth across assets matters so much.

Figure 8.4 · The best of nothing. With a thousand strategies that have no edge at all, the best one shows a Sharpe near

3.7

by luck alone. This is why the number of configurations tried is part of the result: without it, a Sharpe is not evidence of anything.

Why fewer trials helps less than you think. The threshold grows as $\sqrt{\ln N}$ , so going from ten thousand trials to a hundred lowers the bar by only about thirty per cent — which is a genuinely uncomfortable result, because reducing the search is the intervention everyone reaches for first. What it means is that discipline about the *number* of tests is necessary and nowhere near sufficient. The effective levers are longer or wider data, which enters as $\sqrt{T}$ , and a stronger prior from an economic mechanism, which changes the base rate rather than the threshold.

The overfitting problem: deflated Sharpe and what discipline looks like

After this lesson you should be able to

What luck produces

Minimum backtest length

The rest of this lesson is in Premium