Visa finds AI shopping agents failed 42.8% of runs in a new benchmark

AI shopping agents passed just 57.2% of test runs in a new Visa benchmark covering TVs, laptops and personal care appliances. The challenge was more demanding than finding something to buy: agents had to work out what a shopper needed, gather enough information and avoid purchases that failed to meet those requirements.

Visa released the results of its Agentic Commerce Benchmark on November 5, evaluating 22 AI models from 12 labs. According to figures published by PYMNTS, 42.8% of runs failed. More troubling than an unsuccessful search: 54% of those failed runs still ended with the agent making a purchase.

The shopper did not supply every answer

The experiment made conversation part of the task. The shopper was itself an AI agent, assigned requirements that it would reveal only partly at first. The shopping assistant had to ask questions to uncover the missing details within a reasonable time, rather than treating the opening request as a complete specification.

A TV shopper, for example, might need compatibility with a child’s gaming console, a 30-day return window and support for Dolby Vision. Those requirements made the exercise more than a search for a television: the selected product had to fit the shopper’s particular circumstances, including conditions attached to the sale.

Visa’s benchmark evaluates three linked abilities:

• Understand the customer: identify their needs through conversation, including requirements not volunteered at the outset.
• Justify the decision: uncover enough information to support buying a particular product.
• Decline the purchase: buy nothing when no available option meets the customer’s requirements.

The failed purchases fell into two categories: the product did not satisfy the customer’s identified requirements, or the agent bought without uncovering all the requirements. Other failures included buying the wrong product or failing to suggest a suitable one. Completing a transaction was therefore not enough to pass.

Higher running costs bought better results

Performance varied widely. Visa put the best-performing models at an 82.2% pass rate, compared with 28.9% for the worst performers. The overall result conceals a large difference between systems, rather than showing that every shopping assistant struggled to the same degree.

The cost comparison concerns the expense of running an AI shopping inquiry, not the price of the products selected. More expensive models performed better as a group, but that group’s pass rate still left a substantial share of tasks unsuccessful.

The results also come from a defined set of exercises. Fortune described 15 shopping expeditions for each of the 22 agents. The first round covered three product categories; its percentages describe performance on those challenges, not a measured success rate across every kind of consumer purchase.

Choosing the product is only one hurdle

“We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate,” Visa’s global head of growth, Rubail Birwadker, told Fortune. That is his forecast, alongside results showing that even the strongest tested systems did not pass every task.

Birwadker described product discovery—the focus of this experiment—as just one part of agent-driven commerce. Smooth, secure payments remain another challenge. Liability is also unresolved when an agent makes a rogue purchase or commits fraud, leaving a separate question about who bears responsibility after a shopping decision goes wrong.

Alongside the benchmark, Visa launched a website where consumers can interact with the agents used in the experiment, Fortune reports. That gives shoppers a way to explore the tested systems themselves, while the published results offer a narrower guide: how those agents handled the particular needs and choices in Visa’s shopping exercises.