Pearl Finds AI Models Match Expert Judgment Only 70% of the Time
Leading AI models from OpenAI, Anthropic, Google DeepMind, Microsoft, and other developers align with expert judgment only about 70% of the time, according to a press release from Pearl Enterprise. The company evaluated 25 models using more than 500 professional questions across five domains: business, health, law, pets, and technology.
Pearl’s Expert Alignment Leaderboard found that OpenAI’s GPT 5.5 led with 72.7% expert alignment, followed closely by GPT 5 at 72.5%, GPT 5.1 at 72.0%, and Anthropic’s Claude Opus 4.7 at 71.9%. No model exceeded 73% overall alignment, indicating that current systems may be converging below expert-level performance.
Performance varied sharply by domain. Top scores reached 80.9% in business, but some widely used models dropped to around 20% in areas such as law and health. Pearl also noted that increasing reasoning depth improved results by only up to 2.6 percentage points, and in some cases reduced response quality.
Each model received identical prompts and was scored on correctness, completeness, prioritization, and professional judgment. Pearl stated that the dataset used for evaluation was not previously available to model developers. The full leaderboard is available on Pearl’s website.
We hope you enjoyed this article
Consider subscribing to one of our newsletters like Legal AI Weekly, AI Policy Brief or Daily AI Brief.
Also, consider following us on social media:
More from Legal AI
Nov 18 OpenAI opens Astra for Law to all ChatGPT Enterprise customers Nov 18 Litera says AI tripled engineer output as it names Greg Ingino chief technology and AI officer Nov 18 Wimbledon names Harvey its first official legal AI partner from 2027 Nov 18 SK Group rolls out Harvey's legal AI across more than 20 affiliates Nov 18 London's Perry AI raises $2.2 million for AI legal agents for private capital fundsMore from AI Safety
Nov 19 US Catholic bishops form AI task force and push for guardrails and worker protections Nov 18 Nvidia scandal complicates US-China AI talks ahead of Trump-Xi meeting, Wired reports Nov 18 Beijing AI safety group Concordia AI briefed OpenAI staff on China's approach to AI risk Nov 18 OpenAI details how it contains frontier models, including a pause after a DNS exploit Nov 18 Coalition including Hugging Face and xAI co-founders says Anthropic "is not a safe AI company"AI Policy Brief
Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.
Industry analysis
2025 Global Business Services Agenda: Gen AI Takes Center Stage
This industry analysis by The Hackett Group explores the transformative impact of generative artificial intelligence (Gen AI) on global business services (GBS) in 2025. The study highlights the shift from exploration to acceleration of Gen AI initiatives, with 89% of executives advancing these projects to improve customer satisfaction, innovate products, and reduce costs. The report also discusses the challenges and strategies for successful Gen AI adoption, emphasizing the need for a technology-enabled operating model and the importance of reskilling the workforce.
Read moreYou may also like
Epoch AI finds no model meets its employee standard, but Fable 5.2 matches contractors on data center research
Artificial Analysis Index v5: open-weight models fall 12 points behind
AI models give sound legal basics but miss local help in legal aid test of 23 models
Scale AI and Singapore's IMDA benchmark finds AI models often miss facts they need for legal answers
SemiAnalysis GDPx benchmark: best frontier model scores 12% across 11 high-value domains
Daily AI Brief: the AI news that matters, in your inbox.