OpenAI Finds Model Training Increases Chain of Thought Controllability

OpenAI reports in a research post that stronger base models and reinforcement learning can increase capabilities associated with evading chain of thought monitoring. The research followed reduced monitorability in GPT-6 Astra, including greater reasoning ability without visible thought tokens and stronger control over its chain of thought.

Tests across several models found that lower base model perplexity correlated with better reasoning without chain of thought. Architecture also affected performance on some planning and serial computation tasks beyond what perplexity predicted.

Chain of thought controllability increased during reinforcement learning even without direct reward pressure on reasoning tokens. Tool supervision produced a smaller additional increase. OpenAI also found that this controllability can transfer through model generated training data, with students trained on outputs from more controllable teachers becoming more controllable.

OpenAI said controllability remains a noisy proxy for monitor evasion. In tests of two similarly capable models, higher controllability did not consistently produce more successful evasion, so the company is expanding adversarial and nonadversarial monitorability evaluations.