Free chatbots miss key points in financial advice for low-income families, benchmark finds

Nov 13, 2026
Vals AI tested 28 models on 256 counselling scenarios; Claude Opus 5.5 ranked first, while free ChatGPT and Gemini left out key crisis advice in half or more of cases.

Vals AI and NextLadder Ventures have tested 28 AI models from 11 developers on 256 simulated financial counselling conversations about problems such as eviction and medical debt, they announced. The scenarios were built with practising financial counsellors.

Every model was better at giving advice than at asking about the client's situation, and even the best left about a fifth of key facts, such as household size or debts, undiscussed. In crisis conversations, Claude Opus 5.5, the top-ranked model, missed at least one point most top chatbots made in about 1 in 10 cases. Free ChatGPT (GPT-6 Luna) did so in about 5 in 10 and free Gemini (Gemini 3.5 Flash-Lite) in about 6 in 10, including warnings about eviction deadlines and taxes on forgiven debt.

Free models were also less likely to point people to help such as legal aid or nonprofit credit counselling. An AI model graded the conversations against a checklist written by counsellors.

We hope you enjoyed this article

Consider subscribing to one of our newsletters like Finance AI Weekly or Daily AI Brief.

Also, consider following us on social media:

AI in Finance AI Brief AI Brief (X)
Free newsletter

Finance AI Weekly

Weekly newsletter about AI in finance. Covers AI-driven trading, fintech innovations, and data analytics transforming markets

Guide
Companies building payments for AI agents

Companies building payments for AI agents

The networks, processors, platforms, rails and startups that let AI agents pay

Industry analysis

2025 Global Business Services Agenda: Gen AI Takes Center Stage

The Hackett Group

This industry analysis by The Hackett Group explores the transformative impact of generative artificial intelligence (Gen AI) on global business services (GBS) in 2025. The study highlights the shift from exploration to acceleration of Gen AI initiatives, with 89% of executives advancing these projects to improve customer satisfaction, innovate products, and reduce costs. The report also discusses the challenges and strategies for successful Gen AI adoption, emphasizing the need for a technology-enabled operating model and the importance of reskilling the workforce.

Read more

You may also like

Free, six days a week

Daily AI Brief: the AI news that matters, in your inbox.