AppliedMultiple choiceFree solution

Why is ordinary shuffled k-fold cross-validation wrong on a financial time series?

  1. AIt is too slow to run on large datasets in practice
  2. BIt trains on rows after the test rows, and labels overlap
  3. CFinancial data is not identically distributed across folds
  4. DFive folds is too few for the amount of data available
Show the answer and worked solution

Answer: B – It trains on rows after the test rows, and labels overlap

Two leaks at once. Shuffling puts rows from after the test period into the training set, so the model learns from the future; and when the label is a forward return over several days, training rows adjacent to the test set share part of their label window with it, so information crosses the boundary even without shuffling. The result is an inflated score that will not survive live. The fixes are a time-ordered split, purging the training rows whose label window overlaps the validation set, and an embargo after it.

Worked solution

  1. Formula
    leak=training rows after the test, plus overlapping labels\text{leak} = \text{training rows after the test, plus overlapping labels}
  2. Substitute
    shuffled k-fold on ordered data\text{shuffled } k\text{-fold on ordered data}
  3. Solve
    shuffling trains on the future\text{shuffling trains on the future}
  4. multi-day labels overlap across the split\text{multi-day labels overlap across the split}
  5. Answer
    Both leaks; the score is invalid\text{Both leaks; the score is invalid}

Sanity check. The fixes are a time-ordered split, purging the overlapping label windows, and an embargo after the validation set.

  • A. A performance objection, not a correctness one.
  • B. Correct – both leaks. The fixes are a time-ordered split, purging and an embargo.
  • C. True, and a real problem, but the leak is what makes the score invalid.
  • D. More folds would not repair a leak.

Takeaway: Shuffled k-fold trains on the future.

Answer it in practice – your answer is marked and recorded.

Reported in interviews at

Citadel, Two Sigma, Point72 and Cubist

More machine learning questions

Why is ordinary shuffled k-fold cross-validation wrong on a financial time series? · QuantMax