What is overfitting?

Try enough ideas and one will look brilliant by accident. Overfitting is when a strategy is shaped (by you, or by a search) to fit the past so well that it is mostly fitting noise, so it falls apart on data it hasn't seen. For a plain-language walkthrough, see Investopedia’s explainer on overfitting (external).

The cure isn't cleverness. It is bookkeeping: write the rule down first, count every try, and judge the result on years you didn't peek at.

What we do

1. Pre-register

The rule, the windows and the pass criteria are written into a file and saved in version control before the run. The runner refuses to start on a file that has unsaved changes. If we change the rule after seeing results, that is a different idea and the page says so.

2. Split the years

  • Training: 2005–2016. Searching and tuning are allowed here. Some studies begin in 2006 when a long lookback, such as a 200-day average, would otherwise be incomplete. Each write-up names the years it actually used.
  • Validation: 2017–2021, a strongly trending market.
  • Second window: 2022–2023, when stocks and bonds fell together.
  • Reserved: from 2 January 2024 on. Never used for any published result. (Honest footnote: a few early demo runs, before we enforced this, did see some of it.)

3. Count every try

Each backtest, parameter combination and variant goes into a trial registry. A result with no trial count is just a pretty line.

4. Judge against what chance would do

5. Compare fairly

A strategy that is invested 50% of the time looks safer than SPY for free. So we also compare against an exposure-matched benchmark: SPY held at the same invested level (rest in T-bills) and at the same bumpiness, plus an equal-weight basket of the same funds. That is how the drawdown brake can pass the test it was written for and still show no edge over sitting out.

6. Trade it realistically

Decide at the close, fill at the next open. Spreads, impact and fees included. Cash settles the next day, like a retirement account, with no leverage. Idle cash earns the 3-month T-bill rate.

A quick Sharpe scale

On a simple daily strategy, a yearly Sharpe below 0.3 is weak — often not useful once you allow for costs and doubt, but not the same as earning nothing over cash. 0.3 to 0.7 is ordinary (the stock market over long periods sits around here); 0.7 to 1.0 is good; above 1.0 for years is rare and worth checking hard; and above 2.0 on daily ETF data is almost always a bug or overfitting.

In our 2017–2021 tests, plain SPY scored 0.92, which is the bar a lot of ideas had to beat. The full scale, and more of our examples, are in the glossary.

How things die

Type Meaning
Overfit The winner doesn't survive an honest count of tries, or is no better than a shuffled luck test.
Not tradable The effect may exist but can't be captured with our realistic execution.
Edge decayed It worked in one window and failed a later one we fixed in advance.
Inconsistent The sign or result flipped between the windows we fixed in advance, so we don't treat it as a real effect.
Works but not worth it It does something real (often smaller drops) but fails its bar or the protection costs more return than we said we'd accept.
Passed its test, no edge It cleared the pre-registered test it was written for, but did not beat simply holding the market on the same days. That is not an edge, and it has not been traded with real money.
Untestable Not enough data to test it honestly. We say so instead of inventing a p-value.

Further reading

What this can't tell you

  • We used 23 liquid ETFs and some other free-data baskets. Results may not carry to single stocks.
  • The data is free-tier and not survivorship-free.
  • Everything is paper. Simulated costs are not a brokerage statement.
  • A failed test says "we couldn't find it with this data and rule", not "can never work".
  • A Sharpe ratio has its own failure modes, including looking great after a big search.