Scale AI and Singapore's IMDA benchmark finds AI models often miss facts they need for legal answers

Scale AI and Singapore's Infocomm Media Development Authority have released SG-LegalClarify, a benchmark that tests how well AI models reason about Singapore law and whether they notice when a question leaves out facts they need, Scale said on November 12. It is the first research result of the two organisations' collaboration, announced in July.

Legal professionals wrote 1,100 questions across 10 areas of law, including criminal, employment, property and family law. Of these, 760 deliberately omit information needed for a correct answer, and models can ask a simulated user for the missing facts. Scale tested 19 models from developers including OpenAI, Anthropic, Google DeepMind and DeepSeek, with and without web search.

Without web search, the best model met 69.8% of the mandatory grading criteria and the average was 53.8%. Web search raised the average to 63.0% and the top score to 77.8%. In about 39% of scenarios with missing facts, models asked no clarifying question at all, and even when a web-enabled model asked about every missing fact, it reached the correct legal conclusion only 55.6% of the time. Scale plans to release a public subset of the benchmark for further research.