Anthropic Study Finds Just 250 Documents Can Backdoor Large Language Models

Anthropic, in collaboration with the UK Government's AI Security Institute and the Alan Turing Institute, has found that large language models can be backdoored with a surprisingly small amount of poisoned data, according to a research paper published on Anthropic’s website.

The study shows that adding just 250 malicious documents—roughly 0.00016% of total training data—can trigger backdoor behaviors in models ranging from 600 million to 13 billion parameters. The attack used a trigger phrase, “,” which caused the affected models to output gibberish text. Researchers observed that model size and total training data volume had no effect on the attack’s success.

The team trained 72 models across different configurations to confirm that poisoning success depends on the absolute number of poisoned samples rather than the proportion of the dataset. Even models trained on twenty times more clean data were equally vulnerable once they encountered the same number of malicious documents.

Anthropic’s researchers said the findings challenge common assumptions about data poisoning, suggesting that attackers may not need large-scale data access to compromise models. The team shared the results to encourage further research into scalable defenses against such vulnerabilities.