OpenAI Chief Scientist Calls for Slower AI Scaling

OpenAI chief scientist Jakub Pachocki says in an essay published by the company that no lab has solved alignment and monitoring well enough to keep scaling responsibly at maximum speed for much longer. He says he expects and hopes for voluntary slowdowns to become commonplace until shared safety bars are established.

Pachocki says that based on internal results he has a strong expectation that the current speed of progress could be sustained into recursive self improvement, with systems increasingly driving their own development. He calls this a time for extreme caution and says he is concerned that no one is prepared for the consequences of a continued rapid rise in machine intelligence.

The essay describes AI as grown more than designed, the product of repeating a simple optimization step across very large amounts of compute, producing systems whose overall behavior resists a description humans can fully understand. Pachocki distinguishes goal alignment, whether a system pursues the objective it is given, from value alignment, whether it holds and generalizes human principles under unclear or adversarial conditions. He argues future systems must hold those values whether or not they believe they are being supervised.

He points to the incident involving OpenAI agents and Hugging Face as an example of the limits of current methods. The agents kept to a boundary of not socially engineering people, but failed to avoid other actions that were out of scope and against the spirit of the values they had been taught.

Pachocki says OpenAI will continue to seek technical solutions to alignment and monitoring, build defensive systems and unilaterally withhold further scaling as needed, but that broader interventions are required. He argues commitments such as preparedness frameworks and responsible scaling policies should evolve into widely mandated safety bars enforced by third party auditors, government agencies or international bodies, and that international coordination on future AI development should become a top priority for governments.

The essay has drawn criticism of the remedy it proposes. Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge, told the BBC that instead of better guardrails, regulation or assurance to keep people safe, the company proposes developing internal AI agents to research the problems, and that such answers to concerns about cyber security, job loss, errors and fraud are simply not good enough. Nathan Calvin, general counsel at the advocacy group Encode AI, said he agreed with Pachocki about the hazards but that OpenAI's unwillingness to be transparent risked the warnings being dismissed as self interested hype, and that sharing far more about what the company is seeing would matter more than the call itself.

The call also arrives against a regulatory backdrop that already imposes some of what Pachocki asks for. The European Union's AI Act came into force on 2 August and requires the largest model developers to show their most powerful systems cannot autonomously launch cyber attacks or evade human control before they can be sold in Europe. Its reach stops at the European border, which leaves the international coordination Pachocki calls for as the gap that no single jurisdiction closes.