OpenAI Releases Circuit-Sparsity Model and Toolkit on Hugging Face

The OpenAI research team has released its Circuit-Sparsity toolkit and model, announced on Hugging Face. The release includes a 0.4 billion parameter GPT-2–style sparse Transformer model and accompanying open-source tools for analyzing interpretable circuits.

Circuit-Sparsity enforces extreme weight sparsity during training, where roughly one in a thousand weights remain nonzero. The model also maintains moderate activation sparsity, with about a quarter of neuron activations active at any time. This approach allows researchers to identify small, explicit computational circuits that perform specific tasks, such as quote closing, bracket counting, and variable type tracking.

The toolkit connects sparse and dense models through “activation bridges,” encoder–decoder pairs that map activations between the two representations. These bridges allow researchers to transfer interpretable features from sparse models into standard dense models for controlled experimentation.

The model, named *csp_yolo2* in the accompanying paper, and the full toolkit are available under the Apache 2.0 license. The model can be downloaded from Hugging Face, and the related codebase is hosted on GitHub, providing model checkpoints, lightweight inference scripts, and circuit visualization utilities.