Axiomatic AI publishes its own test results for Lemma paper-checking AI

Axiomatic AI has published the first results from its own tests of Lemma, an AI co-scientist that checks the mathematics, proofs, claims and references in scientific papers. The company says Lemma shows the validation steps and reasoning behind each finding, and says so when it cannot confirm something.

On the public SPOT benchmark, Axiomatic used the 26 mathematics and computer science papers whose known errors are in equations or proofs, 29 errors in all. It says Lemma found 26 of them, or 89.7%, the same figure Google reports for its Paper Assistant Tool, while the general-purpose models Axiomatic ran found between 48% and 79%. On an unpublished set of 50 recent physics papers that their authors later corrected or withdrew, Axiomatic says Lemma found 84% of the errors and Gemini 3.1 Pro 52%.

Axiomatic says Lemma raised fewer than a third as many issues per page on 20 journal papers with no known errors, did not flag errors that authors had already fixed, and gave the right verdict on 103 of 104 test references. The company notes the tests cover only mathematics, computer science and physics, that models may have seen the public papers in training, and that it could not run Google's tool itself. Lemma is in open beta.