The backtest improved. Did the strategy learn anything real?

A better score can mean you asked more questions

You change a trading rule and its historical result improves. You change it again and the chart becomes smoother. After enough attempts, the best version looks persuasive. The difficulty is that the history has now helped choose the rule. Asking it to provide an independent verdict afterward gives the same evidence two jobs.

Hard to Fool, a preprint submitted on 4 November by Henry Zhang, describes an open source research loop in which a large language model proposes trading rules and separate evaluation code tests them. The AI chooses from a fixed vocabulary expressed as structured JSON data. It does not write arbitrary executable code for the evaluator to run. The design gives the model room to suggest ideas while keeping the acceptance rules outside its control.

One reported candidate passed development and validation with Sharpe ratios near 3, then fell to 0.26 on the final holdout. A Sharpe ratio compares excess returns with their variability; the sharp decline weakened the apparent result. A holdout is a reserved period of data kept out of the search for a strategy. This is the author's reported experiment, not evidence of guaranteed future returns or a result IWTLP has reproduced.

Source: Hard to Fool preprint, submitted 4 November 2026

The clock belongs inside the data model

For an original teaching exercise, use invented observations of a fictional market. Each row records an event time, the time its information became available and a later outcome. The event time alone is insufficient. A report describing Monday's activity may not have been released until Tuesday. A program simulating a Monday decision cannot legitimately use its Tuesday publication.

Write the rule in terms of information available at the decision time. Then create a test row whose event occurred earlier but whose availability time is later. If the simulated decision changes when that unavailable row is added, the experiment has revealed a timing error. No financial sophistication is needed to understand the failure; it is a problem of which records the code was allowed to see.

Chronological evaluation also requires care with preprocessing. A normalization computed from the entire dataset can carry later information into earlier decisions. Preserve the period used to fit each transformation. A split in the final table does not repair a feature that already incorporated information from the future.

A test stops being untouched when it influences the next idea

Imagine trying several rules on a development period and selecting one for a later evaluation period. That later period can help you reject the idea. If you repeatedly inspect its results and redesign the rule around them, it has become another development resource. Calling it a test set does not preserve its independence.

A final holdout is useful when the procedure limits how its result can influence subsequent decisions. The preprint records a one-use opening before revealing that final result. It also keeps failed proposals and evaluation outcomes. These controls matter in an automated AI loop: repeated searching can produce an impressive historical score even when no durable pattern has been found.

Keep failed proposals as well as successful ones. A directory containing only the winning run cannot show how many alternatives were searched. In our exercise, a simple ledger can retain the candidate description, the input revision, the evaluation period and the result. That record makes a later claim easier to assess, even when every candidate fails.

Source: Authors' staged-evaluation and provenance design

Costs change the question a backtest answers

A second toy example makes the accounting visible. Suppose an invented rule earns ten arbitrary units before costs across ten completed transactions. If each transaction costs one unit, the net result is zero. Reporting the gross result alone answers a different question from reporting the amount remaining after the stated cost model.

Now let the costs vary and preserve that assumption with the output. A sensitivity comparison can show how dependent the conclusion is on a convenient setting. It cannot establish what execution would cost in every real market. The same restraint belongs around uncertainty estimates and sample size: state what was measured and under which assumptions.

For the fictional exercise, finish with one candidate, one untouched evaluation period and a stated cost model. Record the result even if it is disappointing. The useful question is whether the procedure could reject your favourite idea. If every poor result simply starts another round of tuning on the same data, the final chart has stopped being an independent test.