The research pipeline: from hypothesis to live capital

Signal research is a pipeline with a brutal attrition rate: most ideas die, and the discipline is in killing them cheaply and in the right order. Working the stages in sequence — hypothesis, data, feature, evaluation, risk, capacity, live — means the expensive checks only ever run on ideas that have survived the cheap ones.

Stage	Question	What kills it here
Hypothesis	Why should this predict anything?	No economic mechanism
Data	Is it available point-in-time?	Look-ahead, survivorship, revisions
Feature	How do I express it numerically?	The construction destroys the signal
Evaluation	Does it forecast, out of sample?	IC indistinguishable from zero
Risk	Is it a known factor in disguise?	Alpha vanishes after neutralisation
Capacity	How much can it carry?	Costs exceed the edge at size
Live	Does it work with real money?	Live performance well below backtest

Table 1.1 · The stages. The order is a cost ordering. Checking the mechanism takes a minute and checking capacity takes weeks, so anything that would fail the first check should never reach the last.

Proposition 1.2

The hypothesis comes first

Write down what you expect to find and why, before touching the data. It is not a formality: it fixes the number of hypotheses being tested at one, which is the only thing that makes a p-value or a Sharpe ratio interpretable. An idea that arrives from a data sweep has an unknown trial count attached, and no amount of subsequent validation recovers it.

Holds when

A mechanism can be behavioural, structural or a risk premium — but it must exist and be stateable.
"The data say so" is not a hypothesis; it is a description of a search.
Pre-registering the universe, the horizon and the evaluation metric is part of the same discipline.

Why the mechanism does the work. There are far more plausible-looking patterns in financial data than there are real ones, so statistics alone cannot separate them — the multiple-testing arithmetic makes that clear. What the mechanism buys is a *prior*: an idea with a reason to work should be believed on much weaker evidence than one without. It also tells you when to abandon it, because a signal with a stated cause has a stated condition under which the cause stops operating. A signal with no mechanism has no such condition, so you learn nothing from its decay.

Research hygiene

Version the code and the data together, so a result can be reproduced exactly.
Log every variant tried, including the abandoned ones, and keep the count.
Build the evaluation before the model, so you cannot tune the evaluation to the result.
Keep a final holdout untouched until the decision is made.
Write the conclusion you expect before you run it; note when you were wrong.

Example 1.3

Attrition arithmetic

A team tests 200 ideas a year, and one in twenty is genuinely real. At $5\%$ significance and $60\%$ power, how many findings are false?

Show the worked solutionHide the worked solution

Worked solution

Formula $\text{true positives} = \pi N \beta, \qquad \text{false positives} = (1-\pi)N\alpha$
Substitute $N = 200, \ \pi = 0.05, \ \alpha = 0.05, \ \text{power} = 0.6$
Solve $\text{true} = 200 \times 0.05 \times 0.6 = 6$
$\text{false} = 200 \times 0.95 \times 0.05 = 9.5$
Answer $\text{about } 61\% \text{ of findings are false}$

Sanity check. Even with a respectable hit rate and standard thresholds, the majority of significant results are wrong. Raising the prior — by only testing ideas with a mechanism — is far more effective than tightening the threshold, because it changes $\pi$ rather than $\alpha$ .

The research pipeline: from hypothesis to live capital

After this lesson you should be able to

The hypothesis comes first

Research hygiene

Attrition arithmetic

The rest of this lesson is in Premium