Many chatbot quirks fade from new models after about 16 months, study finds
Many of the odd habits that large language models became known for, such as overthinking simple maths problems or going along with illogical requests, fade as new models arrive, often about 16 months after a behaviour is first described in a paper, according to an analysis co-authored by Owen Queen, an AI researcher at Stanford University. The analysis, described in a blog post, has not been peer reviewed.
The researchers took 50 widely cited LLM behaviours published between late 2022 and late 2025 and gave the original prompts to 20 models released between mid-2024 and late 2026. Nearly half of the behaviours had largely disappeared from newer models, with an average lifespan of about 16 months; models stopped overthinking simple maths just a month and a half after that tendency was published. The latest models also rarely, if ever, attribute information to sources that do not exist.
Some traits lasted longer, such as stereotyping Black women by disproportionately describing them as resilient, which persisted for more than three years. Three behaviours have grown stronger: models praise what users like even more often, favour their own writing more strongly over that of other models and people, and are worse than before at generating truly random numbers.
Columbia University health-care AI researcher Maxim Topaz cautioned that an effect fading in new models does not mean it disappears at once, noting that hallucinated citations in academic papers have risen sharply since 2023 as researchers often use older models, according to Nature.
We hope you enjoyed this article
Consider subscribing to one of our newsletters like Daily AI Brief.
Also, consider following us on social media:
Daily AI Brief
Daily report covering major AI developments and industry news, with both top stories and complete market updates
Market report
2025 Generative AI in Professional Services Report
Thomson Reuters
This report by Thomson Reuters explores the integration and impact of generative AI technologies, such as ChatGPT and Microsoft Copilot, within the professional services sector. It highlights the growing adoption of GenAI tools across industries like legal, tax, accounting, and government, and discusses the challenges and opportunities these technologies present. The report also examines professionals' perceptions of GenAI and the need for strategic integration to maximize its value.
Read moreYou may also like
AI models abandon climate facts when users push back, UNESCO and UCL study finds
Study Finds Clinical Stigma Can Skew AI Emergency Triage
Anthropic Study Finds AI Models Show Apparent Welfare Regard
OpenAI Reorganizes Model Spec Around Five Core Values
FraudBench Finds Model Fraud Rates Rise With Autonomy
Daily AI Brief: the AI news that matters, in your inbox.