Visa benchmark finds AI shopping models fail 42.8% of tests

In a press release, Visa introduced an AI shopping benchmark that recorded a 57.2% overall pass rate across 22 models from 12 labs. The models completed 15 scenarios involving products such as televisions, laptops and personal care appliances.

Each model had to question a customer, identify hidden requirements and select a suitable product. It also had to decline to buy when no product met those requirements. The best performing models passed 82.2% of attempts, while the lowest scoring models passed 28.9%.

Models passed all three attempts at the same scenario only 34.2% of the time. Among failed attempts, 54% still ended with a purchase that did not meet identified requirements or was made before all requirements were uncovered. Visa plans to add more products and scenarios, then extend testing to checkout, payment and order placement.