A backtest that looks too good usually is. Two causes explain most of it: the strategy saw the future (lookahead bias), or you tried so many versions that the best one won by luck (overfitting). This guide shows how to check for both. The examples use OptAlgo, but the ideas apply to any backtester.
Signs your backtest is lying
- A small change to one parameter makes the return jump or collapse.
- It wins on one coin or one year and loses everywhere else.
- A handful of trades make most of the profit.
- You have lost count of how many variants you tried on the same data.
- A short-timeframe strategy that trades several times a day looks great before fees and flat after them.
Lookahead bias: the strategy saw the future
Lookahead means a decision on bar i uses data from after bar i closed. It hides in innocent-looking code: shifting a series backwards, np.roll(close, -1), centred windows, or a statistic computed over the whole series. The backtest then trades on information nobody had at the time.
OptAlgo makes every run prove, before its numbers count, that it decides the same with the future removed. The engine runs the strategy again on data cut at 6 evenly spaced and 6 random bars of the last 3,000, and re-decides up to 24 bars where it entered, exited or set a target, on data that ends at that bar. It also runs the same parameters twice to prove the code is deterministic. Any difference is lookahead: the run is graded D and never deploys. Validation runs the same check on synthetic bars for free, before you spend a unit.
Know its limits too. It samples bars rather than re-running every one, plots are not checked, and it is not an overfitting check: parameters tuned on the same window can look great without any lookahead.
Overfitting: the best of many tries
Run 35 variants of one idea on one window and the best of them looks good partly by luck, even if each one alone grades B. The grade judges one run's mechanics, not the search around it. That is why OptAlgo counts the variants you ran on overlapping data in the last 30 days and computes a deflated Sharpe ratio (Bailey & López de Prado): the probability that your Sharpe beats what the best of that many worthless variants would reach by luck.
- ok: one run, or a deflated Sharpe probability of at least 0.95.
- caution: below 0.95.
- likely overfit: below 0.5.
- no edge: the run lost money. Change the idea, not its parameters.
Check 1: a walk-forward instead of hand tuning
Do not sweep parameters with many single runs. A walk-forward splits history into windows: optimise on a training span, test on the next span the optimiser never saw, step forward and repeat. The default plan is 180 days of training and 60 days of testing. The number that matters is the chained out-of-sample return across the test folds, and the verdict is robust, inconclusive or likely overfit. Inconclusive is not a pass.
To validate parameters you already chose, without searching, run a walk-forward with a single configuration. Every fold tests the same parameters on data after its training span, and it counts as one trial:
{"protocol": "walk_forward", "plan": {"plan": "rolling", "train_days": 365, "test_days": 90, "step_days": 90},
"space": {"lb1": {"choices": [42]}, "lb4": {"choices": [336]}}, "sampler": "grid", "n_trials": 1, "objective": "sharpe"}Check 2: a holdout you never touch
A deflated Sharpe discounts luck; a holdout removes it. Group one idea's runs into a study (a tag such as study:rsi-4h) and declare its holdout, for example the last 6 months, before you tune. While the holdout is unused, OptAlgo refuses the study's runs over it. At the end you run one final test on it:
- holds: it makes money and keeps at least half of the in-sample Sharpe.
- degraded: it makes money with less than half of the in-sample Sharpe.
- failed: it lost money on data it never saw.
Once used, the holdout is spent. To test a changed strategy again, declare a new holdout on later data.
Check 3: does it survive small changes?
A robustness check reruns a finished backtest under four pressures: its trades resampled 1,000 times, fees and slippage doubled, each numeric parameter moved by 10%, and the start of the window pushed 5, 10 and 15% later. The verdict is robust, fragile or broken. A strategy that only works at one exact setting, or only from one start date, is telling you something.
Check 4: net of fees, and on more than one coin
Judge every result after fees and funding. Fewer, larger moves usually beat more trades on a short timeframe. Then test the same rules on a basket of coins in one run: OptAlgo splits the capital equally and the grade is the worst symbol's, not the best.
A checklist before you trust a number
- The honesty grade is A to C and you have read its findings.
- The search verdict is ok, or a walk-forward of the same code came back robust.
- The holdout final test holds.
- The profit does not come from a few trades, and it survives doubled costs.
- It runs on paper first, and the paper trades match a backtest of the same period.
Each of these is built into OptAlgo's crypto backtesting, and the Free plan's 50 units a month are enough to try it on your own idea.