Machine learning
Why is ordinary shuffled k-fold cross-validation wrong on a financial time series?
- AIt is too slow to run on large datasets in practice
- BIt trains on rows after the test rows, and labels overlap
- CFinancial data is not identically distributed across folds
- DFive folds is too few for the amount of data available
Try it
Pick an answer
Show the answer and worked solution
Answer: B – It trains on rows after the test rows, and labels overlap
Two leaks at once. Shuffling puts rows from after the test period into the training set, so the model learns from the future; and when the label is a forward return over several days, training rows adjacent to the test set share part of their label window with it, so information crosses the boundary even without shuffling. The result is an inflated score that will not survive live. The fixes are a time-ordered split, purging the training rows whose label window overlaps the validation set, and an embargo after it.
Worked solution
- Formula
- Substitute
- Solve
- Answer
Sanity check. The fixes are a time-ordered split, purging the overlapping label windows, and an embargo after the validation set.
- A. A performance objection, not a correctness one.
- B. Correct – both leaks. The fixes are a time-ordered split, purging and an embargo.
- C. True, and a real problem, but the leak is what makes the score invalid.
- D. More folds would not repair a leak.
Takeaway: Shuffled k-fold trains on the future.
Report an issue with this question
Learn the method
Of the kind reported at
More machine learning questions
- What does embargoing do that purging does not?Advanced · Free solution
- A classifier predicts probability 0.8 for an example that is in fact positive.Foundation · Premium
- Cross-validating a return forecaster, part 1 of 3Foundation · Premium
- A classifier assigns scores at random. What is its expected AUC?Applied · Premium
- A classifier flags 500 cases, of which 40 are true positives, and misses 60 positives.Applied · Premium
- An anomaly detector has sensitivity 90% and specificity 95%.Applied · Premium