Method
How we test
What is overfitting?
Try enough ideas and one will look brilliant by accident. Overfitting is when a strategy is shaped (by you, or by a search) to fit the past so well that it is mostly fitting noise, so it falls apart on data it hasn't seen. For a plain-language walkthrough, see Investopedia’s explainer on overfitting (external).
The cure isn't cleverness. It is bookkeeping: write the rule down first, count every try, and judge the result on years you didn't peek at.
What we do
1. Pre-register
The rule, the windows and the pass criteria are written into a file and saved in version control before the run. The runner refuses to start on a file that has unsaved changes. If we change the rule after seeing results, that is a different idea and the page says so.
2. Split the years
- Training: 2005–2016. Searching and tuning are allowed here. Some studies begin in 2006 when a long lookback, such as a 200-day average, would otherwise be incomplete. Each write-up names the years it actually used.
- Validation: 2017–2021, a strongly trending market.
- Second window: 2022–2023, when stocks and bonds fell together.
- Reserved: from 2 January 2024 on. Never used for any published result. (Honest footnote: a few early demo runs, before we enforced this, did see some of it.)
3. Count every try
Each backtest, parameter combination and variant goes into a trial registry. A result with no trial count is just a pretty line.
4. Judge against what chance would do
- Deflated Sharpe (Bailey and López de Prado, Journal of Portfolio Management, 2014 (PDF, external)): given how many things we tried, how believable is the best Sharpe ratio (return per unit of bumpiness)? Our bar is 0.95.
- Probability of backtest overfitting (Bailey, Borwein, López de Prado and Zhu, 2017 (PDF, external)): how often does the winner from the search years land in the bottom half later? Our general bar is 0.20 (lower is better); for one family we set a looser 0.50 in advance and it still failed.
- False-discovery control (Benjamini and Hochberg, Journal of the Royal Statistical Society, Series B, 1995) when running many tests at once. We often use a q-value bar of 0.10.
- Luck comparisons: shift the indicator in time, pick random entry dates, or train on shuffled answers. If scrambled data does as well, the "edge" isn't one.
5. Compare fairly
A strategy that is invested 50% of the time looks safer than SPY for free. So we also compare against an exposure-matched benchmark: SPY held at the same invested level (rest in T-bills) and at the same bumpiness, plus an equal-weight basket of the same funds. That is how the drawdown brake can pass the test it was written for and still show no edge over sitting out.
6. Trade it realistically
Decide at the close, fill at the next open. Spreads, impact and fees included. Cash settles the next day, like a retirement account, with no leverage. Idle cash earns the 3-month T-bill rate.
A quick Sharpe scale
On a simple daily strategy, a yearly Sharpe below 0.3 is weak — often not useful once you allow for costs and doubt, but not the same as earning nothing over cash. 0.3 to 0.7 is ordinary (the stock market over long periods sits around here); 0.7 to 1.0 is good; above 1.0 for years is rare and worth checking hard; and above 2.0 on daily ETF data is almost always a bug or overfitting.
In our 2017–2021 tests, plain SPY scored 0.92, which is the bar a lot of ideas had to beat. The full scale, and more of our examples, are in the glossary.
How things die
| Type | Meaning |
|---|---|
| Overfit | The winner doesn't survive an honest count of tries, or is no better than a shuffled luck test. |
| Not tradable | The effect may exist but can't be captured with our realistic execution. |
| Edge decayed | It worked in one window and failed a later one we fixed in advance. |
| Inconsistent | The sign or result flipped between the windows we fixed in advance, so we don't treat it as a real effect. |
| Works but not worth it | It does something real (often smaller drops) but fails its bar or the protection costs more return than we said we'd accept. |
| Passed its test, no edge | It cleared the pre-registered test it was written for, but did not beat simply holding the market on the same days. That is not an edge, and it has not been traded with real money. |
| Untestable | Not enough data to test it honestly. We say so instead of inventing a p-value. |
Further reading
- Glossary — the terms on this page, in plain words
- Understanding overfitting (external) — Investopedia
- Overfitting (external) — Wikipedia
- The Deflated Sharpe Ratio (PDF, external) — Bailey and López de Prado, Journal of Portfolio Management, 2014. This is the paper behind the deflated-Sharpe gate we use.
- The Probability of Backtest Overfitting (PDF, external) — Bailey, Borwein, López de Prado and Zhu, Journal of Computational Finance, 2017. This is the paper behind the overfitting-probability gate we use.
What this can't tell you
- We used 23 liquid ETFs and some other free-data baskets. Results may not carry to single stocks.
- The data is free-tier and not survivorship-free.
- Everything is paper. Simulated costs are not a brokerage statement.
- A failed test says "we couldn't find it with this data and rule", not "can never work".
- A Sharpe ratio has its own failure modes, including looking great after a big search.