Works but not worth it

A take-or-skip judge on top of our setups: it took nearly everything

· 3 min read

The idea

Meta-labeling: don't ask a model to predict prices, ask whether a given signal is worth acting on. We applied this to three signals: the deep-pullback setup, the calming-volatility setup, and 200-day overlay turn-on events. Conditions were rendered as simple labels (up, flat, down) for a fixed-choice decision gate (JJM).

What happened

The gate can't yet accept a chronological holdout (it holds out a random 20%), so reported numbers use a plain logistic-regression stand-in trained on the same labels. It took 100% of the pullback and overlay events and 79% of the calming-volatility events. For the one setup where it skipped anything, taken events were not significantly better than skipped (p = 0.27). On the training years, that setup's stand-in book had a Sharpe of 0.89, essentially the same as taking every signal (also 0.89). The other two books were identical to take-all, as expected when nothing is skipped. We did not open a later window.

The full results

Training years only; no later window was opened. As the page says, the reported numbers use a plain logistic-regression stand-in for the gate, trained on the same labels, because the gate itself cannot yet take a chronological holdout. A numeric baseline on raw scores was specified and not run, so there is nothing to show for it here.

Event results, predicting 2008–2016. "Average return" is the average executable return per event over the hold shown.

Trigger (events, 2008–2016)EventsTakenTake rateAverage return, takenAverage return, skippedp (taken better than skipped)
Deep pullback, 126-session hold4949100%10.25%none skippedn/a
Calming volatility, 21-session hold43334179%2.30%2.01%0.27
200-day overlay turning on, 63-session hold158158100%2.60%none skippedn/a

The books built from those decisions (idle cash credited at the 3-month T-bill rate). Where the stand-in skipped nothing, its book is the same as take-every-signal:

Training yearsSharpeYearly returnWorst dropTime invested
Deep pullback, 126-session hold · take every signal0.444.87%15.8%32%
Deep pullback, 126-session hold · gate stand-in Identical0.444.87%15.8%32%
Calming volatility, 21-session hold · take every signal0.898.16%8.6%24%
Calming volatility, 21-session hold · gate stand-in0.897.66%9.1%23%
200-day overlay turning on, 63-session hold · take every signal0.384.68%26.3%54%
200-day overlay turning on, 63-session hold · gate stand-in Identical0.384.68%26.3%54%

For the one setup where the stand-in skipped anything, the take-every-signal and gated books have essentially the same Sharpe (0.89 against 0.89), and the event test (p = 0.27) did not find the taken events better than the skipped ones.

Why it failed

A filter that says yes to nearly everything doesn't filter. It took 100% of the pullback and 200-day-overlay events, and 79% of the calming-volatility events. For the one setup where it skipped anything, the taken events were not clearly better than the skipped ones (a p-value of 0.27). A numeric baseline on raw scores was specified and not run. We did not reopen it.

Receipts: the plan was written down and committed first.