OpenAI's ARGO turns black-box reward models into readable rubrics
OpenAI has published research on ARGO (Automated Rubric Grader Optimization), a method that uses reinforcement learning to generate plain-language rubrics that reproduce the preferences of a black-box reward model, the kind of model trained on human preference judgments that shapes what chatbots learn in RLHF, the company says. A judge model conditioned on the learned rubric is scored on how closely it agrees with the reward model, so the rubric becomes a readable description of what the reward model actually rewards.
OpenAI applied ARGO to reward models trained on two populations: one similar to ChatGPT users and one of experts with greater knowledge of its policies. For the first, the rubrics rewarded relevance and concrete deliverables, which were "highly valued even if minor inaccuracies," and what OpenAI calls performative helpfulness and mild sycophancy. The expert population's rubrics put more weight on instruction following, accuracy, calibration and safety. OpenAI says this visibility helps separate behaviours it wants models to learn from those it wants to suppress, since optimising for the first population's signals could amplify overly flattering or agreeable behaviour.
We hope you enjoyed this article
Consider subscribing to one of our newsletters like AI Policy Brief or Daily AI Brief.
Also, consider following us on social media:
More from AI Safety
Nov 21 Microsoft AI chief Suleyman 'really, really nervous' as models help improve their successors Nov 21 Hundreds march in London for stricter AI regulation Nov 21 Dan Hendrycks calls for indefinite pause on frontier AI training at DealBook Nov 21 Ex-Anthropic researcher Jacob Coxon calls private control of AI 'insane' on IPO day Nov 21 Human auditors catch only 17% of planted sabotage in AI safety research, Redwood study findsAI Policy Brief
Weekly report on AI regulations, safety standards, government policies, and compliance requirements worldwide.
Market report
2025 Generative AI in Professional Services Report
Thomson Reuters
This report by Thomson Reuters explores the integration and impact of generative AI technologies, such as ChatGPT and Microsoft Copilot, within the professional services sector. It highlights the growing adoption of GenAI tools across industries like legal, tax, accounting, and government, and discusses the challenges and opportunities these technologies present. The report also examines professionals' perceptions of GenAI and the need for strategic integration to maximize its value.
Read moreYou may also like
OpenAI Finds GPT-6 Astra Underperforms on Safety Research
OpenAI Details Activation Classifiers for Model Safety
OpenAI Introduces BioBasicsBench for Wet Lab Analysis
Anthropic safety guide shows Claude Fable 5.1 scored 94% in app but 60% via API on self-harm tests
OpenAI opens Astra for Law to all ChatGPT Enterprise customers
Daily AI Brief: the AI news that matters, in your inbox.