Visa published its first agentic commerce benchmark on November 5, and three accounts of it say the same numbers in three different orders. Read together, they point at one figure that the headline result hides: more than half of the failed runs ended in a purchase.
What Visa published
The Visa Agentic Commerce Benchmark put 22 AI models from 12 labs through 15 shopping scenarios, such as buying a TV, a laptop or a personal care appliance. The shopper in each scenario leaves details unsaid, so the model has to find the budget and the other requirements by talking to the customer. If nothing on offer fits, the correct outcome is no purchase. A run passes only when the model uncovers the requirements the task defines and makes the right buy or no-buy call.
The results, as Visa states them: an overall pass rate of 57.2%, with the best models passing 82.2% of attempts and the weakest 28.9%. The five models that found the most hidden requirements averaged 70.2%, against 44.4% for the five that found the fewest. The five most expensive models per run passed 67.1% of runs and the five cheapest 50.2%. Each model had three attempts at every scenario, and all three passed in 34.2% of cases. Of the failed runs, 54% still ended with the AI making a purchase.
Advertisement
Visa’s release gives the reason it ran the test. “AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. It also points to the company’s Trust Index from September 2026, which found that 72% of US consumers had used an AI assistant and 23% trusted AI to handle payment transactions on their behalf.
Three accounts, three emphases
All three accounts below carry the 57.2%, 82.2%, 28.9%, 34.2% and 54% figures without alteration. None of them disputes a number. They differ in which number they lead with and what they say it means.
Payments Industry Intelligence framed the result as a reliability gap and gave a section the heading “Consistency Remains the Bigger Problem”. Its argument is that a model can complete a task once and fail it on repeat, so the 34.2% three-for-three figure carries more weight than the 57.2% average. Its closing view is that the industry will need to solve “consistency, explainability and control” before autonomous purchasing scales.
The Paypers led with the pass rate and then moved to consumer sentiment. It repeated the Trust Index figures and added a point Visa’s release does not contain: a SEON survey in which half of surveyed US adults said they were uncomfortable with AI agents using their card details. For The Paypers, the benchmark is one more data point on a trust gap that already existed.
The Next Web led with the failure rate in its headline and then did something the other two did not. It noted that Visa’s statement did not name the models, and it supplied lab-level scores from a Fortune report that put agents from two labs at the top and a Claude agent third. It also carried a second Birwadker quote from that Fortune interview: “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.”
So the accounts split on three points. Payments Industry Intelligence treats repeatability as the central defect. The Paypers treats the shopper’s willingness as the central constraint. The Next Web is the one account that goes past Visa’s statement to say where named models placed, and it gets that from a second outlet’s reporting. Visa’s release gives no per-model results, which means a reader of the release cannot tell whether the 57.2% average is carried by a few strong models or spread evenly.
The figure none of them leads with
Every account reports that 54% of failed runs ended in a purchase. Only one of them gives it more than a sentence, and none builds its main argument on it. It is the number with a direct payments consequence.
A failed run in this benchmark has two forms. In one, the model recommended a purchase that did not meet the customer’s identified requirements. In the other, it bought before it had found all the requirements. In both, money moves. The other 46% of failed runs, by implication of Visa’s own arithmetic, ended without a purchase: the model either declined or never completed the buy. Those failures cost the shopper time and cost the merchant a sale. They do not obviously leave a payment to dispute.
Using Visa’s figures, 42.8% of runs failed and 54% of those ended in a purchase, which puts roughly 23% of all runs in the category where a wrong purchase went through. That is our calculation from the published percentages. Visa and the three outlets do not state it. The inputs are rounded, so the exact share will differ slightly.
The benchmark also stops short of the step where a card is charged. Visa says it has not yet tested checkout, payment or order placement and plans to extend testing there. The purchase decision in these runs is therefore a recommendation to buy, scored against the scenario’s answer key. A model that “buys” the wrong television in the test has not yet charged anyone. The 54% measures how often the decision layer reaches for the buy button when it should not.
What it means for the finance leader
In our reading, the three accounts together describe a control problem that sits upstream of the payment. The tests measure whether a model can stop itself. Visa’s design rewards a correct no-buy and counts an unjustified buy as a failure, and the results say models fail that test more often than they fail by declining.
That has practical consequences for three groups.
Issuers and card programs. If a share of agent-initiated purchases rests on an incomplete read of what the customer wanted, the disputes that follow will not look like fraud. The cardholder authorised an agent, and the agent acted on a partial brief. Programs that write agentic dispute rules now will have a category for this. We covered the open question of who bears that loss in an earlier piece on agent shopping and refund rules.
Merchants. The benchmark’s cost finding, that the priciest models passed more often but the cheapest still passed half the time, matters to a merchant deciding which agents to accept traffic from. Pass rates of 28.9% to 82.2% across models show that agent quality varies widely, so an acceptance policy that treats every agent alike is treating very different buyers alike.
CFOs approving agent spend. Spend limits cap the size of an error and do nothing about its frequency. A treasury team piloting agents for procurement should ask a vendor for repeat-run results on its own scenarios, since the benchmark shows that one passing run says little about the next. This follows from Visa’s three-attempts design. The recommendation is ours and Visa makes no such claim.
These threads connect to earlier coverage here. Our look at the Trust Index findings reported the gap between using an AI assistant and trusting it with payment. The benchmark supplies a mechanism for that gap. And our reading of the coverage of Stripe’s agent purchase growth covered the volume side of the same market, where the question of how many of those purchases were right never came up.
What to watch
Three things would change this reading. Visa’s extension of the test to checkout, payment and order placement will show whether the buy-anyway rate holds when a payment credential is in play. Per-model results, which Visa’s statement does not contain, would show whether the failures cluster. And a second benchmark from another network or lab would show whether 57.2% is a property of the models or of Visa’s scenarios.
Until then, the safest reading of the three accounts is the narrow one. They agree on the data, they differ on the diagnosis, and the 54% buy-anyway figure, the one with the most direct payments consequence, is reported by all three and built on by none.
Source: Visa newsroom
