Overfit
1,192 indicator combinations: the winner was no better than luck
The idea
Combine up to three indicators into a ranking score, hold the top 5 or 8 ETFs, rebalance every 5 or 21 sessions. Search everything.
What we did
- 12 indicators, all 1–3 indicator subsets, 2 rebalance speeds, 2 portfolio sizes: 1,192 configurations.
- Choose on training years only (2006–2016). Validation (2017–2021) was reported, not used to choose.
- A null: run 200 complete searches on time-shifted indicators and see what the best of each looks like.
- Deflated Sharpe and probability of backtest overfitting on the whole set.
What happened
| Result | |
|---|---|
| Best on training years | Sharpe 0.744 (a three-indicator combo, rebalanced every 5 sessions) |
| Typical config on training | median Sharpe 0.50; 90th percentile 0.61 |
| Best of a scrambled-data search | median 0.83; 95th percentile 1.03 |
| How often scrambled data beat our winner | empirical p = 0.73 |
| Deflated Sharpe (1,192 tries) | 0.910 (bar 0.95) |
| Probability of backtest overfitting | 0.78 (bar 0.20) |
Rebuilt in a realistic engine (next-open fills, settled cash, costs), the finalist got Sharpe 0.70 on training and 0.77 on 2017–2021, about 9.2% a year against SPY's 18.28% (about 18.3%) in that window. It traded about 41–45 times its capital a year.
The full results
This search used a fast approximate simulator (no next-open fills, flat cost), so its Sharpes differ a little from the realistic-engine rebuild described above. The report kept only summary numbers for the 1,192 configurations, not each one's result, so the chart shows those saved summary points rather than a full histogram. We did not re-run the search just to redraw it.
The ten best on the training years, with their 2017–2021 Sharpe alongside. Validation was reported, not used to choose:
| Configuration | Rebalance | Holdings | Training Sharpe | 2017–2021 Sharpe |
|---|---|---|---|---|
| #1: 3-month return + Bollinger z-score (20-day) + distance below 12-month high | every 5 days | top 5 | 0.744 | 0.809 |
| #2: 3-month return + 12-month return per unit of risk + price vs 200-day average | every 21 days | top 5 | 0.732 | 0.735 |
| #3: 3-month return | every 21 days | top 5 | 0.725 | 0.548 |
| #4: 6-month return + Bollinger z-score (20-day) + distance below 12-month high | every 5 days | top 5 | 0.709 | 0.765 |
| #5: 3-month return + 50-day vs 200-day average + distance below 12-month high | every 21 days | top 5 | 0.703 | 0.913 |
| #6: 3-month return + 12-1 month momentum + price vs 200-day average | every 5 days | top 5 | 0.703 | 0.767 |
| #7: 3-month return + distance below 12-month high | every 5 days | top 5 | 0.688 | 0.775 |
| #8: 3-month return + 6-month return + 12-month return per unit of risk | every 21 days | top 8 | 0.683 | 0.746 |
| #9: 3-month return | every 5 days | top 8 | 0.682 | 1.030 |
| #10: 50-day vs 200-day average + Bollinger z-score (20-day) + distance below 12-month high | every 5 days | top 5 | 0.682 | 0.616 |
| Median of all 1,192 configurations | 0.501 | 0.760 |
The ten best on the search years averaged 0.77 on 2017–2021, against 0.76 for the median of all 1,192.
Why it failed
Picking the training winner added nothing later: the top ten on the search years averaged a Sharpe of 0.77 on the later years, against a 0.76 median for all 1,192 configurations. Searching more than a thousand combinations on 23 funds will find something that looks good. That is all this search did. A scrambled-data search has the same budget of tries, so its lucky winner can look even stronger than the real one — here the scrambled best had a median Sharpe of 0.83, above our real winner's 0.744, about 73% of the time. The deflated Sharpe (a probability that asks how believable the best result still is after 1,192 tries, not a marked-down Sharpe ratio) was 0.910, under our 0.95 bar, and the probability of overfitting was 0.78, far above our 0.20 bar.
Receipts: 1,192 distinct configurations were recorded in our trial registry.