AI models give sound legal basics but miss local help in legal aid test of 23 models

Vals AI and NextLadder Ventures released the Access to Justice Benchmark on 10 November, evaluating 23 AI models across 252 legal scenarios written by legal aid experts in six states. Models met 78% to 87% of the criteria for general legal explanations, but only 30% to 50% of the criteria for identifying resources such as local legal aid programs and court help centers.

Qwen 3.8 Max led with 76.4%, though the top three models were within the margin of error of each other; the top two OpenAI models scored about 66%. Across all models, housing and landlord questions produced the highest average score at 75.4%, while public benefits questions ranked lowest at 57.3%.

Free-tier chatbot models averaged 69.7%, compared with 70.4% for premium models. In one New York City tenant scenario, only four of the 23 models referred the user to the city's Right to Counsel program. NextLadder Ventures, which backs AI tools for low-income Americans, says it hopes both independent builders and frontier labs will take up the gaps.