Overfit
Machine learning on 24 indicators: no better than shuffled data
The idea
Let a model combine all 24 indicators. Three models, retrained each year from 2008 with purged data and a 20-session embargo: an elastic-net logistic model at 21 and 63 sessions, and extra-trees at depth 3 for 63 sessions. Each had a control trained on shuffled labels.
What happened
| Model | Sharpe | Shuffle Sharpe | Beat its shuffle control by 0.05? | Exposure-matched SPY |
|---|---|---|---|---|
| Elastic-net, 21 sessions | 0.30 | 0.34 | No (worse than shuffle, by 0.044) | 0.41 |
| Elastic-net, 63 sessions | 0.32 | 0.31 | No (ahead by 0.016, short of 0.05) | 0.41 |
| Extra-trees, 63 sessions | 0.24 | 0.24 | No (ahead by 0.004, short of 0.05) | 0.41 |
Drawdowns were smaller than SPY's, which wasn't enough.
The full results
Each model predicts the years 2008 to 2016, retrained every year on data up to that point with purging and a 20-session embargo; all numbers are default costs, with idle cash credited at the 3-month T-bill rate. A model had to beat its own shuffled-label control by 0.05 and beat the exposure-matched market comparison by 0.05 to pass. None did either.
| Predicting 2008–2016 | Sharpe | Yearly return | Worst drop | Time invested | Shuffled-label Sharpe | Model minus shuffle |
|---|---|---|---|---|---|---|
| Elastic-net, 21 sessions | 0.299 | 4.11% | 38.5% | 64% | 0.342 | −0.044 |
| Elastic-net, 63 sessions | 0.323 | 4.38% | 37.5% | 63% | 0.306 | +0.016 |
| Extra-trees, 63 sessions | 0.245 | 3.08% | 28.4% | 49% | 0.240 | +0.004 |
| SPY, fully invested Reference | 0.405 | 7.47% | 55.2% | 100% | — | — |
This table shows three decimals, so the margins in the last column can be checked against the numbers beside them.
Why it failed
If a model can't beat itself trained on noise by a margin we set in advance, it hasn't found a pattern. All three Sharpes (0.24 to 0.32) fell short of an exposure-matched market comparison at 0.41. None beat its shuffled-data copy by the 0.05 margin we had set: the 21-session elastic-net was worse than its shuffle (0.30 against 0.34); the other two were only barely ahead (0.32 against 0.31, and 0.24 against 0.24, ahead by only 0.004). With 23 ETFs and daily bars there may simply not be enough independent data for these models to learn from.
Receipts: the plan was written down and committed first. 3 official trials plus shuffled controls.