Forecast evaluation: walk-forward design and backtest overfitting

Evaluating a forecast on financial data is harder than producing one. The design has to respect time, the comparison has to account for the fact that both forecasts saw the same data, and the final number has to be discounted for how many candidates were tried to get it.

Figure 7.1 · One walk-forward window, in days. The two small blocks are the whole point. A label that takes five days to resolve reaches five days into the future, so the five days either side of the boundary have to be thrown away — and a backtest that keeps them is reporting a number it did not earn.

Proposition 7.2

Walk-forward design

Fit on a window, test on the block that follows, roll forward, repeat. Every parameter — including the hyperparameters and the feature selection — must be chosen using only data from before the test block, or the evaluation is contaminated.

Holds when

Expanding window if the data-generating process is stable; rolling if you expect regime change.
Report the per-period results, not just the average — a strategy that worked only in one regime should be visible.
Anything fitted outside the loop, including a scaling factor or a universe filter, leaks.

Leak	How it hides	Fix
Hyperparameters tuned on all data	Looks like one model, is really hundreds	Nest the tuning inside the loop
Full-sample normalisation	No code reads a future row	Expanding or cross-sectional statistics
Universe chosen today	Survivorship	Point-in-time membership
Overlapping labels	Train and test share information	Purge and embargo
Reused test set	Each look is a trial	Touch the final holdout once

Table 7.3 · Where evaluations leak. The first and the last are the same failure at different scales: every decision informed by the test data is a use of it, whether or not any code reads a future row.

Equation 7.4

Diebold–Mariano

Test whether two forecasts differ in accuracy by testing whether their loss differential has mean zero.

DM = \frac{\bar{d}}{\sqrt{\widehat{\mathrm{Var}}(\bar{d})}}, \qquad d_t = L(e_{1t}) - L(e_{2t})

$L$: The loss function — squared error, absolute error, or something economic.
$\widehat{\mathrm{Var}}$: Must be HAC, since the loss differential is autocorrelated.

Why you cannot just compare two error rates. Both forecasts were evaluated on the same periods, so their errors are heavily correlated — when the market gapped, both were wrong. Comparing their mean squared errors independently ignores that and produces a standard error far too large, so real differences look insignificant. Diebold–Mariano tests the *difference* series directly, where the common component has already cancelled, which is why it can detect improvements that a naive comparison misses entirely.

Forecast evaluation: walk-forward design and backtest overfitting

After this lesson you should be able to

Walk-forward design

Diebold–Mariano

The rest of this lesson is in Premium