Market data in pandas, and where look-ahead hides

Most backtests that look too good are not wrong about the model; they are wrong about the timestamps. Handling market data well is mostly a discipline about what was knowable when, and the pandas idioms exist to encode that discipline rather than to be clever.

Definition 4.1

Look-ahead bias

Look-ahead bias — Using information in a decision that was not available at the time of the decision. It rarely arrives as an obvious mistake; it arrives as a shift in the wrong direction, a close-to-close signal traded at the same close, a restated fundamental, or an index membership list downloaded today and applied to 2015.

Figure 4.2 · How much of a backtest the warm-up eats. A two-hundred-day average needs two hundred days before it says anything. On a single year of data that is four fifths of the sample — and a backtest that quietly starts at the first non-null row is measuring a different period from the one it claims.

Bias	What it is	The fix
Look-ahead	Using data not yet published at decision time	Lag by the publication delay, not by one bar
Survivorship	A universe containing only firms that still exist	Point-in-time universe including delistings
Restatement	Fundamentals as revised, not as first reported	Point-in-time fundamentals with an as-of date
Selection	A universe or period chosen after seeing results	Fix the universe before the test, and count the variants tried

Table 4.3 · The four that matter. Survivorship is the one candidates name; look-ahead is the one that actually appears in their code.

import pandas as pd

# WRONG: the signal at time t is multiplied by the return realised at time t.
pnl = signal * returns

# RIGHT: act on the signal at t, earn the return from t to t+1.
pnl = signal.shift(1) * returns

# RIGHT, and honest about execution: a signal computed on the close
# of day t is traded at the open of t+1 at the earliest.
position = signal.shift(1)
pnl = position * returns - position.diff().abs() * cost_per_turn

Listing 4.4 · Aligning a signal with the return it predicts. The first line is the single most common bug in a first backtest, and it is worth knowing that it usually produces a spectacular Sharpe ratio rather than a subtle one. Time O(n) · Space O(n).

Proposition 4.5

Why `merge_asof` exists

Market data arrives on irregular timestamps, and an ordinary merge on time joins only exact matches — which for tick data means almost nothing joins. merge_asof joins each row to the most recent row on the other side at or before its timestamp, which is exactly the "what did I know at this instant" semantics a backtest needs.

Holds when

Both frames must be sorted by the join key, or the result is silently wrong.
The default direction is backward. Using direction="forward" in a backtest is look-ahead written explicitly.
tolerance caps how stale a match may be, which is how you avoid quoting off an hour-old print.

Market data in pandas, and where look-ahead hides

After this lesson you should be able to

Look-ahead bias

Why `merge_asof` exists

The rest of this lesson is in Premium