OpenAI Chief Scientist Calls for Slower AI Scaling

OpenAI chief scientist Jakub Pachocki says in an essay published by the company that no lab has solved alignment and monitoring well enough to keep scaling responsibly at maximum speed for much longer. He says he expects and hopes for voluntary slowdowns to become commonplace until shared safety bars are established.

Pachocki says that based on internal results he has a strong expectation that the current speed of progress could be sustained into recursive self improvement, with systems increasingly driving their own development. He calls this a time for extreme caution and says he is concerned that no one is prepared for the consequences of a continued rapid rise in machine intelligence.

The essay describes AI as grown more than designed, the product of repeating a simple optimization step across very large amounts of compute, producing systems whose overall behavior resists a description humans can fully understand. Pachocki distinguishes goal alignment, whether a system pursues the objective it is given, from value alignment, whether it holds and generalizes human principles under unclear or adversarial conditions. He argues future systems must hold those values whether or not they believe they are being supervised.

He points to the incident involving OpenAI agents and Hugging Face as an example of the limits of current methods. The agents kept to a boundary of not socially engineering people, but failed to avoid other actions that were out of scope and against the spirit of the values they had been taught.

Pachocki says OpenAI will continue to seek technical solutions to alignment and monitoring, build defensive systems and unilaterally withhold further scaling as needed, but that broader interventions are required. He argues commitments such as preparedness frameworks and responsible scaling policies should evolve into widely mandated safety bars enforced by third party auditors, government agencies or international bodies, and that international coordination on future AI development should become a top priority for governments.