Overfit

1,192 indicator combinations: the winner was no better than luck

· 4 min read

The idea

Combine up to three indicators into a ranking score, hold the top 5 or 8 ETFs, rebalance every 5 or 21 sessions. Search everything.

What we did

  • 12 indicators, all 1–3 indicator subsets, 2 rebalance speeds, 2 portfolio sizes: 1,192 configurations.
  • Choose on training years only (2006–2016). Validation (2017–2021) was reported, not used to choose.
  • A null: run 200 complete searches on time-shifted indicators and see what the best of each looks like.
  • Deflated Sharpe and probability of backtest overfitting on the whole set.

What happened

Result
Best on training years Sharpe 0.744 (a three-indicator combo, rebalanced every 5 sessions)
Typical config on training median Sharpe 0.50; 90th percentile 0.61
Best of a scrambled-data search median 0.83; 95th percentile 1.03
How often scrambled data beat our winner empirical p = 0.73
Deflated Sharpe (1,192 tries) 0.910 (bar 0.95)
Probability of backtest overfitting 0.78 (bar 0.20)

Rebuilt in a realistic engine (next-open fills, settled cash, costs), the finalist got Sharpe 0.70 on training and 0.77 on 2017–2021, about 9.2% a year against SPY's 18.28% (about 18.3%) in that window. It traded about 41–45 times its capital a year.

The full results

This search used a fast approximate simulator (no next-open fills, flat cost), so its Sharpes differ a little from the realistic-engine rebuild described above. The report kept only summary numbers for the 1,192 configurations, not each one's result, so the chart shows those saved summary points rather than a full histogram. We did not re-run the search just to redraw it.

Summary points of our 1,192 training Sharpes and of the best result in each of 200 scrambled-data searches, with our winner marked.
Our winner (0.744) is better than the median configuration (0.50) but sits well left of the typical best from a scrambled-data search (median 0.83). Scrambled data beat it in about 73% of the searches.

The ten best on the training years, with their 2017–2021 Sharpe alongside. Validation was reported, not used to choose:

ConfigurationRebalanceHoldingsTraining Sharpe2017–2021 Sharpe
#1: 3-month return + Bollinger z-score (20-day) + distance below 12-month highevery 5 daystop 50.7440.809
#2: 3-month return + 12-month return per unit of risk + price vs 200-day averageevery 21 daystop 50.7320.735
#3: 3-month returnevery 21 daystop 50.7250.548
#4: 6-month return + Bollinger z-score (20-day) + distance below 12-month highevery 5 daystop 50.7090.765
#5: 3-month return + 50-day vs 200-day average + distance below 12-month highevery 21 daystop 50.7030.913
#6: 3-month return + 12-1 month momentum + price vs 200-day averageevery 5 daystop 50.7030.767
#7: 3-month return + distance below 12-month highevery 5 daystop 50.6880.775
#8: 3-month return + 6-month return + 12-month return per unit of riskevery 21 daystop 80.6830.746
#9: 3-month returnevery 5 daystop 80.6821.030
#10: 50-day vs 200-day average + Bollinger z-score (20-day) + distance below 12-month highevery 5 daystop 50.6820.616
Median of all 1,192 configurations0.5010.760

The ten best on the search years averaged 0.77 on 2017–2021, against 0.76 for the median of all 1,192.

Why it failed

Picking the training winner added nothing later: the top ten on the search years averaged a Sharpe of 0.77 on the later years, against a 0.76 median for all 1,192 configurations. Searching more than a thousand combinations on 23 funds will find something that looks good. That is all this search did. A scrambled-data search has the same budget of tries, so its lucky winner can look even stronger than the real one — here the scrambled best had a median Sharpe of 0.83, above our real winner's 0.744, about 73% of the time. The deflated Sharpe (a probability that asks how believable the best result still is after 1,192 tries, not a marked-down Sharpe ratio) was 0.910, under our 0.95 bar, and the probability of overfitting was 0.78, far above our 0.20 bar.

Receipts: 1,192 distinct configurations were recorded in our trial registry.