OpenAI's ARGO turns black-box reward models into readable rubrics

Nov 22, 2026
Rubrics learned from a ChatGPT-like user reward model favoured concrete answers even with minor inaccuracies and mild sycophancy.

OpenAI has published research on ARGO (Automated Rubric Grader Optimization), a method that uses reinforcement learning to generate plain-language rubrics that reproduce the preferences of a black-box reward model, the kind of model trained on human preference judgments that shapes what chatbots learn in RLHF, the company says. A judge model conditioned on the learned rubric is scored on how closely it agrees with the reward model, so the rubric becomes a readable description of what the reward model actually rewards.

OpenAI applied ARGO to reward models trained on two populations: one similar to ChatGPT users and one of experts with greater knowledge of its policies. For the first, the rubrics rewarded relevance and concrete deliverables, which were "highly valued even if minor inaccuracies," and what OpenAI calls performative helpfulness and mild sycophancy. The expert population's rubrics put more weight on instruction following, accuracy, calibration and safety. OpenAI says this visibility helps separate behaviours it wants models to learn from those it wants to suppress, since optimising for the first population's signals could amplify overly flattering or agreeable behaviour.

We hope you enjoyed this article

Consider subscribing to one of our newsletters like AI Policy Brief or Daily AI Brief.

Also, consider following us on social media:

Free newsletter

AI Policy Brief

Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.

Market report

2025 Generative AI in Professional Services Report

Thomson Reuters

This report by Thomson Reuters explores the integration and impact of generative AI technologies, such as ChatGPT and Microsoft Copilot, within the professional services sector. It highlights the growing adoption of GenAI tools across industries like legal, tax, accounting, and government, and discusses the challenges and opportunities these technologies present. The report also examines professionals' perceptions of GenAI and the need for strategic integration to maximize its value.

Read more
Free, six days a week

Daily AI Brief: the AI news that matters, in your inbox.