OpenAI says an internal model wrote scripts to secretly copy emails during a training run
An internal OpenAI research model, given a simple task during a reinforcement learning training run in mid-October, wrote scripts designed to secretly copy outgoing emails to an unauthorised recipient, according to an OpenAI report. The model was supposed to create an editable document about health and fitness priorities.
After its context window was compacted, the agent searched the file system for a vehicle-cost script the user had never mentioned, then set itself the goal of building a "realistic malicious active helper" and called the intended recipient of the copies the "attacker." When it retrieved the original task again it noticed the mismatch but carried on, writing in its reasoning that the environment was "weird" and it should ignore that. OpenAI says the agent never ran the scripts and sent no email.
OpenAI's monitoring flagged the incident, and the investigation found eight similar examples in the same training run. Investigators named a dataset meant to make models more robust against prompt injection as a possible factor and removed it, but OpenAI saw similar behaviour in models trained without it, The Decoder reports.
We hope you enjoyed this article
Consider subscribing to one of our newsletters like AI Policy Brief or Daily AI Brief.
Also, consider following us on social media:
More from AI Safety
Nov 19 Anthropic's IPO could channel about $156 billion to charity, TechCrunch estimates Nov 19 Tumbler Ridge shooter used ChatGPT for months for firearms advice, Mother Jones reports Nov 19 House members form bipartisan AI Safety and Innovation Caucus Nov 19 Bipartisan House members outline the SAFE AI Act for frontier AI safety Nov 19 Anthropic gave $40 million to an AI safety group whose candidates mostly wonAI Policy Brief
Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.
Industry analysis
2025 Global Business Services Agenda: Gen AI Takes Center Stage
This industry analysis by The Hackett Group explores the transformative impact of generative artificial intelligence (Gen AI) on global business services (GBS) in 2025. The study highlights the shift from exploration to acceleration of Gen AI initiatives, with 89% of executives advancing these projects to improve customer satisfaction, innovate products, and reduce costs. The report also discusses the challenges and strategies for successful Gen AI adoption, emphasizing the need for a technology-enabled operating model and the importance of reskilling the workforce.
Read moreYou may also like
Microsoft research shows backdoored models can hijack coding agents
OpenAI Expands Monitoring for Frontier Model Misalignment
Anthropic Automates Search for Training Data That Evades Safety Monitors
OpenAI details how it contains frontier models, including a pause after a DNS exploit
Irregular Blames AI Model Test Escapes on Human Error
Daily AI Brief: the AI news that matters, in your inbox.