Overfit

Wait for the stars to align: 24 setups, 0 passed

· 5 min read

The idea

Instead of always being invested, wait for a "confluence" of two or three conditions, hold a fixed 1, 3 or 6 months, and sit in T-bills otherwise.

What we did

  • 8 setups defined from indicators we already had (for example: a deep pullback in an ETF still in a long-term uptrend, entered once momentum turns back up; a volatility spike that calms; a breakout after a quiet stretch). 3 horizons (21, 63 and 126 sessions). 24 tests.
  • Rule written first: at least 30 events on 15 unique dates; beat random entry dates in the same ETF; a bootstrap confidence interval entirely above zero; false-discovery rate q ≤ 0.10 across all 24.

What happened

0 of 24 passed. The closest:

Setup Horizon Extra return vs random dates Events FDR q
Deep pullback in an uptrend, momentum turning up 6 months +5.9% 55 (43 dates) 0.16
Calming volatility 1 month +1.3% 484 0.16
Same pullback setup 3 months +3.5% 58 0.16

Setups that fired often were indistinguishable from random entries in the same ETF. One setup never had 30 events at any horizon.

We also report eight 63-day portfolio books for context. The best Sharpe (0.57, against SPY's 0.41) belonged to a setup that was 83% invested with about no event edge, which is just market exposure.

The full results

All 24 event tests, training years 2005–2016. "Extra return" is the average executable return of the setup's events (buy the next open, sell the hold period later) minus the average for random entry dates in the same ETF. "Interval floor" is the bottom of the one-sided 95% bootstrap interval for that extra return; it needed to be above zero. The false-discovery q-value had to be 0.10 or lower.

Dot plot of the false-discovery q-values of all 24 confluence tests against the 0.10 bar.
No test met the bar: the best q-value was 0.16, against our 0.10.
SetupHoldEvents (dates)Extra return vs random datesInterval floorFDR qSurvived
Deep pullback, 200-day still rising, momentum turning up126 sessions55 (43)+5.9%1.6%0.16No
VIX spike, then calming21 sessions484 (154)+1.3%0.3%0.16No
Deep pullback, 200-day still rising, momentum turning up63 sessions58 (46)+3.5%0.9%0.16No
Quiet-stretch squeeze plus 52-week breakout126 sessions69 (55)+2.6%0.2%0.22No
Deep pullback, 200-day still rising, momentum turning up21 sessions81 (67)+1.4%−0.2%0.35No
Quiet-stretch squeeze plus 52-week breakout63 sessions73 (59)+0.9%−1.1%0.61No
Quiet-stretch squeeze plus 52-week breakout21 sessions84 (69)+0.2%−0.3%0.61No
Relative strength vs SPY just turned positive63 sessions381 (341)+0.2%−0.6%0.61No
All 24 tests, closest first
SetupHoldEvents (dates)Extra return vs random datesInterval floorFDR qSurvived
Deep pullback, 200-day still rising, momentum turning up126 sessions55 (43)+5.9%1.6%0.16No
VIX spike, then calming21 sessions484 (154)+1.3%0.3%0.16No
Deep pullback, 200-day still rising, momentum turning up63 sessions58 (46)+3.5%0.9%0.16No
Quiet-stretch squeeze plus 52-week breakout126 sessions69 (55)+2.6%0.2%0.22No
Deep pullback, 200-day still rising, momentum turning up21 sessions81 (67)+1.4%−0.2%0.35No
Quiet-stretch squeeze plus 52-week breakout63 sessions73 (59)+0.9%−1.1%0.61No
Quiet-stretch squeeze plus 52-week breakout21 sessions84 (69)+0.2%−0.3%0.61No
Relative strength vs SPY just turned positive63 sessions381 (341)+0.2%−0.6%0.61No
VIX spike, then calming63 sessions251 (108)+0.4%−1.6%0.61No
Oversold pullback in an uptrend21 sessions1,413 (683)+0.1%−0.4%0.61No
SPY 200-day regime just switched on21 sessions294 (27)+0.3%−1.0%0.61No
Oversold pullback in an uptrend63 sessions623 (393)+0.1%−0.9%0.61No
Deep 1-year drawdown with an RSI-2 hook21 sessions191 (101)+0.1%−3.8%0.61No
SPY 200-day regime just switched on63 sessions227 (25)+0.1%−2.2%0.61No
Volatility spike, then calming, in an uptrend126 sessions23 (17)+0.2%−4.3%0.61No
Deep 1-year drawdown with an RSI-2 hook126 sessions63 (42)+0.5%−11.0%0.61No
Oversold pullback in an uptrend126 sessions348 (240)0.0%−1.5%0.61No
Relative strength vs SPY just turned positive126 sessions266 (238)+0.0%−1.6%0.61No
Deep 1-year drawdown with an RSI-2 hook63 sessions97 (66)−0.4%−6.0%0.61No
SPY 200-day regime just switched on126 sessions172 (22)−0.7%−6.8%0.61No
VIX spike, then calming126 sessions164 (68)−0.4%−3.9%0.61No
Relative strength vs SPY just turned positive21 sessions617 (536)0.0%−0.5%0.61No
Volatility spike, then calming, in an uptrend21 sessions28 (19)−0.3%−3.0%0.61No
Volatility spike, then calming, in an uptrend63 sessions24 (18)−1.2%−3.5%0.85No

For context only, the eight 63-day portfolio books. These were reported for reference and carried no pass or fail:

63-day book, training yearsSharpeYearly returnWorst dropTime invested
Oversold pullback in an uptrend0.506.33%25.5%81%
Volatility spike, then calming, in an uptrend0.202.09%9.3%15%
VIX spike, then calming0.253.58%42.7%41%
Quiet-stretch squeeze plus 52-week breakout0.323.29%11.4%34%
Deep pullback, 200-day still rising, momentum turning up0.404.15%12.6%22%
Relative strength vs SPY just turned positive Highest Sharpe0.577.31%23.1%83%
SPY 200-day regime just switched on0.434.24%23.6%34%
Deep 1-year drawdown with an RSI-2 hook0.203.06%50.2%29%
SPY, fully invested Reference0.417.47%55.2%100%

The highest Sharpe belonged to a setup that was in the market about 83% of the time with no event edge, which is mostly market exposure.

Why it failed

With 24 tries on 23 ETFs, an extra 5.9% on 55 events does not stand out from luck. The closest q-value (a false-discovery score) was 0.16, and our bar was 0.10. We did not loosen that bar after seeing the result.

Receipts: the setups were written down and committed before any returns were computed. 24 event trials plus 8 reporting-only books.