A screenshot is not a model evaluation
An AI label can make a trading claim sound technically sophisticated before the underlying evidence has been shown. A screenshot of winning trades may be genuine yet still leave out losing accounts, open positions, costs, and the period selected. A fluent explanation of a market move may have been written after the outcome. None of those objects establishes predictive ability on future observations. Evaluating the claim begins with defining the result, the sample, the information available at the time, and the account conditions under which the result was obtained.
The CFTC's January 2024 advisory about AI trading bots warns about schemes using AI claims and unrealistic or guaranteed returns. It is a dated consumer warning, not a fresh benchmark of every 2026 product. We use it as context for asking better evidence questions. This article has not tested a commercial bot, verified a vendor, or run an automated trading strategy. Its examples are original arithmetic exercises showing why five appealing numbers can remain insufficiently defined.
First number: the win rate
Imagine ten fictional trades. Nine earn $1 each and one loses $20. The win rate is 90%, but total gross profit is nine dollars minus twenty dollars, or a loss of $11. A high proportion of winners therefore does not imply a positive total result. The size of wins and losses matters, as do costs and position sizing. A post displaying only the percentage of winning trades omits information needed to understand the economic outcome, even when the percentage itself has been counted correctly.
Now change the example to five wins of $5 and five losses of $3. The win rate is 50%, but gross profit is $10. This does not identify a superior strategy; both examples are invented and too small for a performance conclusion. Their purpose is to separate a frequency measure from a money measure. Before comparing advertised win rates, ask what counts as a trade, how partial closes are handled, and whether open losses are excluded. Different counting rules can make two apparently comparable percentages describe different populations.
Second number: the sample size
A result based on ten selected observations contains less information than the same descriptive result across a clearly documented larger sample, but sample size alone does not solve selection problems. Ten thousand duplicated or cherry-picked observations do not become representative because the count is large. The evaluation needs the collection rule, the date range, the instruments included, and the handling of missing or failed trades. Without those details, a large number can provide an impression of rigor while leaving the most important questions about how the evidence was chosen unanswered.
Suppose a vendor examines 100 fictional settings and advertises only the one with the highest historical return. The published record may accurately describe that setting on those observations while overstating what has been learned about future performance. The selection process belongs in the disclosure. A reader should ask how many alternatives were tried and whether the reported test set was consulted during development. This article does not assign a probability to the resulting overstatement; doing so would require a defined statistical setup rather than a generic warning about small samples.
Third number: the backtest return
A backtest applies rules to historical data under assumptions about what was known and how trades could execute. If the rule uses information that became available later, the historical result can contain leakage. An obvious example is using a day's closing price to choose a trade supposedly entered earlier that same day. Less obvious cases involve revised economic series, future-adjusted labels, or preprocessing fitted using the entire dataset. A credible method describes the chronology precisely enough for another person to identify whether the simulated decision had access to future information.
Separating training and evaluation periods can help, but the separation must be respected. If an author repeatedly changes the model after looking at the evaluation period, that period has become part of the development process. Calling it out of sample does not restore independence. The question is what decisions were made using which observations. A useful disclosure names the cutoff, the version of the data, the rule-selection process, and any later retesting. This article explains those questions without presenting an invented experiment as a successful or unsuccessful AI benchmark.
Fourth number: the net result
A model can predict a direction correctly often enough to look impressive while producing an unfavorable net result after costs. Return calculations need spreads, commissions, financing, conversion, and other applicable charges under a stated execution model. A historical midpoint is not necessarily an executable buy or sell price. A calculation that ignores this distinction can create a simulated edge from transactions that could not have occurred at the assumed levels. The relevant evidence is not simply a clean chart but a reproducible account of how simulated orders were priced.
Consider a fictional system earning $40 gross across twenty completed trades. If each completed trade costs $3 in total, charges are $60 and the net result is a $20 loss. These are invented charges, not a quotation from any broker. The example shows why a small gross advantage can disappear without any change in the prediction count. A vendor's phrase after fees should identify which fees and whether the calculation also includes execution differences. Otherwise, net can mean different things on different product pages.
Fifth number: the maximum drawdown
An attractive ending balance can coexist with a difficult path. A fictional account might rise from $1,000 to $1,200, fall to $720, and finish at $1,100. The final return relative to the starting point is positive, but the decline from the $1,200 peak to $720 is 40%. That path may be hidden by a summary emphasizing only the endpoint. The drawdown measure also depends on observation frequency and whether open positions are included. A daily curve can miss an intraday trough, and a closed-balance curve can omit unrealized losses.
An AI description does not remove these measurement choices. A reader should ask whether the drawdown was calculated from account equity, what timestamps were used, and whether deposits or withdrawals were adjusted. Ask whether the reported figure comes from a backtest, a demonstration account, or independently verified live records. Those labels are not interchangeable. This article has not verified any of them for a commercial product. The point is to request evidence that matches the claim rather than let a technological label substitute for the missing account definitions.
Language fluency and trading evidence are different
A system that summarizes a central-bank statement can be useful for organizing reading without having a tested ability to forecast prices. A system that generates code can help implement an idea while still introducing errors in timing, units, or assumptions. Evaluate the particular task instead of treating AI as one uniform capability. A polished explanation should be checked against its cited source. A numerical claim should be recalculated. A claimed trading result should be evaluated under the same disclosure standards regardless of whether the vendor calls the underlying method neural, statistical, or rules based.
This distinction also helps define a bounded learning exercise. Ask a tool to list the variables in a hypothetical cost calculation, then verify the calculation independently. Ask it to compare two public definitions, then inspect the originals. Do not infer a performance record from its ability to complete those tasks. The article does not provide a prompt for automatic orders or a claim that a general-purpose system can choose appropriate exposure. The useful outcome is better documentation and checking, not a shortcut around the evidence needed for financial claims.
Ask for a reproducible evidence package
A meaningful evaluation package identifies the strategy version, data source, chronological split, instruments, transaction assumptions, costs, and outcome definitions. It reports unfavorable results as well as favorable ones and distinguishes development experiments from a final evaluation. If independent verification is claimed, the scope and verifier should be identifiable. A badge or testimonial without a method does not supply those details. The absence of a package does not prove every claim false, but it limits what a careful reader can responsibly conclude from the information presented.
For a personal review, write down the exact claim before examining the chart. Is it about prediction accuracy, net account return, reduced drawdown, or convenience in analyzing documents? Then list the evidence that would be required to support that particular claim. This prevents the discussion from shifting to a different impressive feature whenever a missing metric is questioned. It also allows an honest vendor to describe a narrow, useful capability without being forced into an unsupported promise that the product can do everything suggested by the AI label.
What the numbers can and cannot establish
The arithmetic examples establish that win rate, gross return, net return, and drawdown are distinct measures. They do not establish the profitability or safety of any bot. The cited advisory provides consumer-protection context, while our discussion of evaluation design is a practical framework for identifying missing information. No proprietary dataset, live account, or vendor model was tested. That limitation should remain visible when the article is shared, because removing it would change an educational explanation into an apparent review of products we have not examined.
Before accepting a dramatic screenshot, ask what was counted, what was omitted, when the rule was chosen, how transactions were priced, and whether the full path is available. If the answers are missing, leave the performance claim unresolved. A system may still be useful for a clearly bounded task, but usefulness in one task is not evidence of guaranteed investment returns. Keeping those conclusions separate is a stronger response to AI marketing than either automatic enthusiasm or a blanket claim that every automated method must fail.
A useful editorial check is to remove the letters AI from the claim and read it again. If the remaining statement still promises a guaranteed return without a defined method or verifiable record, the missing evidence has not been supplied by the label. Conversely, a modest claim about organizing documents may be useful without implying any market prediction. This exercise keeps the evaluation tied to observable tasks. It also avoids confusing a criticism of an unsupported financial promise with a claim that every computational tool is useless for research, documentation, or calculation.
