Google Unveils VaultGemma: A Differentially Private Language Model

Google has introduced VaultGemma, a 1 billion-parameter language model trained from scratch with differential privacy, announced on their website. This model is designed to integrate privacy at its core by adding calibrated noise to prevent data memorization, a significant step in AI development.

VaultGemma is the largest open model of its kind, and its weights are available on platforms like Hugging Face and Kaggle. The model's development was guided by new research on scaling laws for differentially private language models, conducted in collaboration with Google DeepMind. These laws help in understanding the trade-offs between compute, privacy, and utility, providing a framework for optimizing training configurations.

The model's training involved advanced techniques such as Poisson sampling to ensure strong privacy protections while maintaining high utility. VaultGemma's performance is comparable to non-private models from five years ago, demonstrating the progress in differentially private training methods. The model also comes with a formal privacy guarantee, ensuring that it does not memorize specific training data sequences.