OpenAI publishes its pretraining perplexity and proposes industry limits on AI development

Nov 22, 2026
GPT-4 sat near 0.91 log-perplexity and a ~0.76 model produced maths breakthroughs; OpenAI wants shared limits until monitoring catches up.

OpenAI has published internal measurements of its pretraining progress and proposed that frontier developers agree on "common limits" they would not cross until they can better understand and monitor AI behaviour, in a post arguing for perplexity as a shared industry metric. Perplexity measures how well a model predicts held-out text. On OpenAI's main evaluation dataset, GPT-3.5 sat at a log-perplexity of about 0.99 and GPT-4 at about 0.91; below 0.8, the post says, transformative capabilities emerged in fields like cybersecurity and mathematics, and once OpenAI upgraded to a model at about 0.76 it "started producing groundbreaking results" on open maths problems.

The post says further transformative capabilities should be expected in the 0.68 to 0.7 range, along with further degradation of current alignment auditing methods, and that an avalanche of pretraining advances could come as soon as 2027. It proposes an industry standard setting a first limit on development, measured by a metric tracked across labs, and what must be achieved, such as a successor to chain-of-thought monitors, before going beyond it. Publishing the numbers gives competitors a clearer picture of OpenAI's position, the post says, but that is outweighed by the urgency of shared safety measures. CEO Sam Altman called it "the most important thing OpenAI has published in awhile" in a post on X, adding that he hopes the field can agree to common limits, which he said "would be very wise and help ensure safety and control." In a second post, he said labs agreed under the White House Accords to establish standards and best practices to improve safety, and that a discussion about perplexity can further these goals. The post also notes limits of the metric: labs use different datasets, and chain-of-thought reasoning raises capability without lowering perplexity.

We hope you enjoyed this article

Consider subscribing to one of our newsletters like AI Policy Brief or Daily AI Brief.

Also, consider following us on social media:

Free newsletter

AI Policy Brief

Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.

Whitepaper

Tensordyne Napier: What If One Rack Could Do the Work of Nine?

Tensordyne

This Tensordyne whitepaper presents Napier, an inference-focused AI processor and rack-scale system based on the company’s TDN Math logarithmic number system. It examines infrastructure requirements for large mixture-of-experts and agentic models, compares major inference architecture approaches, and details the TDN AIP processor, TDN72 pod, TDN Link fabric, and Napier Ultra configuration. The paper reports simulation-based performance, cost, and accuracy-validation results, including Tensordyne’s projected comparison of one Napier rack with a nine-rack Nvidia Rubin plus Groq deployment; the chip is reported as taped out and in fabrication.

Read more
Free, six days a week

Daily AI Brief: the AI news that matters, in your inbox.