Goodfire trains AI models to avoid misleading shortcuts

Goodfire AI introduced mechanistic shaping, a training framework that uses a model's internal representations to control which concepts it relies on and expresses, according to a company research post. The framework adds penalties to the training objective for relying on or expressing selected concepts.

Goodfire tested the method on a vision model trained to identify collapsed lungs in chest X-ray images. The training data linked collapsed lungs with visible chest drains, which could cause the model to use the drain rather than features of the lung. Researchers used a sparse autoencoder and a language model agent to find internal features associated with drains, then penalized the model for relying on them.

The researchers report that this reliance shaping method outperformed standard training and several earlier methods. It also exceeded the accuracy of a model trained on balanced data in the experiment.

Goodfire separately used mechanistic shaping to limit sycophancy and reward hacking in language models while retaining capability gains from training. The researchers found that targeted concepts remained detectable after shaping across their experiments, although monitoring sometimes required a newly trained probe.