Trust an AI-written backtest only after it passes seven checks: real data with fees and funding, no lookahead, an honest count of tries, out-of-sample evidence, enough trades with a drawdown you can live with, costs that survive frequent trading, and a paper run. ChatGPT or Claude can produce a good-looking equity curve in a minute, and most of the ways it can be wrong do not show on the chart. Each check below gives you a question to ask and what a bad answer looks like.
The examples are real runs made on OptAlgo on 2026-11-22 with Binance data. Trading is risky, and no backtest is proof, including the good-looking one below.
Why an AI-written backtest needs extra checks
An assistant writes strategy code quickly, and the code usually runs. That is the problem. A backtest has two parts: the strategy and the simulator that fills its orders. When the assistant writes both in one script, nothing checks the simulator. Orders fill at perfect prices, fees are left out, and the result looks better than anything you could have traded.
The second problem is the conversation itself. Each time you say “make it better”, the assistant changes a number and runs again. After twenty rounds you are looking at the best of twenty tries, and some of that result is luck.
The checks below work with any tool. Where OptAlgo does a check for you, the text says how.
Check 1: Real data, fees and funding
Ask: which exchange is the data from, what bar size were fills simulated on, which fees were charged, and was funding paid on perpetual contracts?
A bad answer: “I used sample prices”, “fees are not included yet”, “orders fill at the close of the signal bar”, or silence about funding. On a perpetual contract, funding is a real payment between longs and shorts, and a position held for weeks pays or receives it many times.
On OptAlgo this question is answered by the honesty grade, from A to D. A means every fill settled on real data with fees, funding and exchange rules. D means not realistic: fix it and rerun. Read the grade before any return figure.
Example A. An EMA cross trend strategy (long when EMA 20 crosses above EMA 80, 3 ATR trailing stop) on the BTCUSDT perpetual, 4h bars, 2022-01-01 to 2026-01-01. Honesty grade A. Return +70.8% after fees and funding, CAGR 14.3%, max drawdown -15.0%, Sharpe 0.85, 55 trades, win rate 40%, profit factor 1.79. This is one backtest, in-sample. Keep it in mind: it passes check 1 and fails check 4.
Check 2: Lookahead
Ask: when the strategy decides at bar i, does it use only data up to bar i?
A bad answer is usually in the code, not in the chat. Look for arrays shifted the wrong way, a daily high or low used before the day has ended, or an indicator computed over the whole history at once. The usual symptom is an equity curve that is too smooth and a win rate that is too high.
Example D. A strategy compared each bar with the next bar's close:
nxt = np.concatenate([c[1:], c[-1:]]) # next bar's close
OptAlgo's free validation refused it before any run, with this finding: “the strategy looks ahead: enter at bar 1607 changes when the bars after 1607 are removed (bar i may only use data up to bar i)”. The fix came with it: “use indicators.shift(x, 1) for the previous bar”.
The test behind that message is simple and you can ask any tool to do it: cut the data at a bar, run again, and see whether the decision at that bar changed. If it did, the strategy was reading the future. More cases are in the lookahead bias examples.
Check 3: The number of tries
Ask: how many versions of this strategy did we run on this data before this one?
A bad answer: “this is the best of the ones I tried”, with no count. Or a clean final script and no record of the fifteen that came before it.
Every run is a try. If you test enough worthless variants, one of them will show a good Sharpe ratio by chance. A deflated Sharpe ratio corrects for this: it asks whether your Sharpe beats what the best of that many worthless variants would reach by luck.
OptAlgo counts the variants you ran on overlapping data in the last 30 days and attaches a search verdict to every result: ok, caution, likely overfit, or no edge when the run lost money. If you use another tool, keep the count yourself and be honest about it.
Check 4: Out-of-sample or walk-forward
Ask: were the parameters chosen on one period and judged on a period the optimiser never saw?
A bad answer: “the backtest covers four years, so it is robust”. Length is not the same as out-of-sample. If the numbers were picked while looking at all four years, all four years are in-sample.
Example B. The same EMA cross idea from check 1, as one walk-forward research run: 30 configurations per fold (fast 10 to 40, slow 60 to 160, trail 2, 3 or 4 ATR), 608-day train, 180-day test, 6 rolling folds, 2021-01-01 to 2026-01-01. The verdict was likely overfit.
- Out-of-sample return +23.4%, out-of-sample annualised Sharpe 0.49.
- 4 of 6 test folds positive, 30 out-of-sample trades.
- It failed the deflated Sharpe check, with warnings for a short track record and for train-to-test degradation.
- The chosen parameters changed in every fold, for example fast 30 and slow 120, then 25 and 160, then 40 and 140.
That last line is the clearest sign. If a real edge sat at one setting, the optimiser would keep finding the same neighbourhood. Here it found a different winner each time, which is what fitting noise looks like.
So the grade A run with +70.8% was honest about fills and still not proof of an edge. The grade answers “could these trades have happened?”. Walk-forward answers “would I have picked these rules in advance?”. You need both. The overfitting guide covers how to read a research run and how to keep a holdout.
Check 5: Trade count and drawdown
Ask: how many trades is this based on, what was the max drawdown, and how long did the worst losing stretch last?
A bad answer: a large return from a dozen trades, or a result where two trades carry all the profit. A few trades cannot separate skill from luck. In Example B, 30 out-of-sample trades was enough to trigger the short track record warning.
Read drawdown as something you would have had to sit through. Example A lost 15.0% from a peak at its worst, with a win rate of 40%, which means most trades lost and losing streaks were normal. Ask yourself whether you would have kept the bot running at that point. If not, the return figure is not yours.
Check 6: Cost sensitivity for frequent trading
Ask: how many trades per day does it make, and what happens to the result if costs are a little higher?
A bad answer: a strategy on 1-minute or 5-minute bars with hundreds of trades and an average profit per trade that is smaller than one round trip of fees. A strategy pays the fee on every entry and every exit, so the more it trades, the more the result depends on the cost assumption.
A quick way to see it: divide net profit by the number of trades. If the average trade earns only a small multiple of its own cost, a slightly wider spread or a worse fill can erase the edge. Example A made 55 trades in four years, so costs were a small part of its result. A scalping strategy is the opposite case, and it deserves the hardest look here.
Check 7: A paper run before money
Ask: has this exact code run forward, on live prices, without real money?
A bad answer: “the backtest is enough”. A backtest only knows the past. Paper trading runs the same rules on prices that did not exist when you wrote them, which is the one test you cannot overfit.
On OptAlgo you deploy a backtest graded A to C and the code runs at every bar close, on paper, in My Bots. The deploy gate refuses a losing backtest, and refuses one judged likely overfit unless an out-of-sample research run of the same code passed. Compare with backtest then matches the bot's real paper trades against a backtest of the same period, trade by trade. Chat connectors are paper-only, and going live is your decision in the app, on Plus or Pro.
The checklist as rules for your assistant
You can paste these into the chat at the start. They turn the seven checks into habits.
- State the data source, fees and funding, and the honesty grade, before any return.
- Prove each decision uses only past bars.
- Tell me how many variants we have run on this data.
- Do not tune by single runs. Use one walk-forward and report the out-of-sample result.
- Report trade count, max drawdown and win rate with every result.
- Report the average profit per trade next to the cost per trade.
- Call a single backtest in-sample, never proof. Paper comes before money.
All seven are built into OptAlgo backtesting, and the Free plan is enough to try them: 50 backtest units a month, no card. Create a free account, or read how the AI strategy builder works first.
Frequently asked questions
Can I trust a backtest written by ChatGPT or Claude?
Not on its own. The code may run and still fill orders at impossible prices, skip fees or read future bars. Trust it only after it ran on real exchange data with fees and funding, passed a lookahead check, held up out of sample and ran on paper.
What is the most common mistake in an AI-written backtest?
Two are very common. The first is lookahead: the strategy uses data from a bar that had not closed yet. The second is tuning by repetition: asking for one more variation many times on the same data and keeping the best one.
My backtest has a good return and an honest grade. Is that enough?
No. An honest grade says the trades could have happened with real costs. It does not say the rules were chosen in advance. In our example, a grade A run with +70.8% was judged likely overfit by a walk-forward of the same idea.
How many trades does a backtest need?
There is no fixed number, but a few dozen trades is thin. In our walk-forward example, 30 out-of-sample trades triggered a short track record warning. Fewer trades means a wider range of results that luck alone can explain.
Does a passed checklist mean the strategy will make money?
No. The checks remove known ways a backtest lies. They do not predict the future. Markets change, and a strategy that passed every check can still lose. Start on paper and risk only what you can afford to lose.