Overfit

Machine learning on 24 indicators: no better than shuffled data

· 2 min read

The idea

Let a model combine all 24 indicators. Three models, retrained each year from 2008 with purged data and a 20-session embargo: an elastic-net logistic model at 21 and 63 sessions, and extra-trees at depth 3 for 63 sessions. Each had a control trained on shuffled labels.

What happened

Model Sharpe Shuffle Sharpe Beat its shuffle control by 0.05? Exposure-matched SPY
Elastic-net, 21 sessions 0.30 0.34 No (worse than shuffle, by 0.044) 0.41
Elastic-net, 63 sessions 0.32 0.31 No (ahead by 0.016, short of 0.05) 0.41
Extra-trees, 63 sessions 0.24 0.24 No (ahead by 0.004, short of 0.05) 0.41

Drawdowns were smaller than SPY's, which wasn't enough.

The full results

Each model predicts the years 2008 to 2016, retrained every year on data up to that point with purging and a 20-session embargo; all numbers are default costs, with idle cash credited at the 3-month T-bill rate. A model had to beat its own shuffled-label control by 0.05 and beat the exposure-matched market comparison by 0.05 to pass. None did either.

Predicting 2008–2016SharpeYearly returnWorst dropTime investedShuffled-label SharpeModel minus shuffle
Elastic-net, 21 sessions0.2994.11%38.5%64%0.342−0.044
Elastic-net, 63 sessions0.3234.38%37.5%63%0.306+0.016
Extra-trees, 63 sessions0.2453.08%28.4%49%0.240+0.004
SPY, fully invested Reference0.4057.47%55.2%100%——
Paired bars of each walk-forward model's Sharpe and its shuffled-label control's Sharpe, with exposure-matched SPY marked at 0.41.
Each model sits within a few hundredths of its own shuffled-label copy, and all three sit below the exposure-matched market comparison. All three had smaller worst drops than SPY's, which was not enough.

This table shows three decimals, so the margins in the last column can be checked against the numbers beside them.

Why it failed

If a model can't beat itself trained on noise by a margin we set in advance, it hasn't found a pattern. All three Sharpes (0.24 to 0.32) fell short of an exposure-matched market comparison at 0.41. None beat its shuffled-data copy by the 0.05 margin we had set: the 21-session elastic-net was worse than its shuffle (0.30 against 0.34); the other two were only barely ahead (0.32 against 0.31, and 0.24 against 0.24, ahead by only 0.004). With 23 ETFs and daily bars there may simply not be enough independent data for these models to learn from.

Receipts: the plan was written down and committed first. 3 official trials plus shuffled controls.