Hugging Face Introduces SmolLM3: A Multilingual, Long-Context AI Model

Hugging Face has announced the release of SmolLM3, a new language model featuring 3 billion parameters. This model is designed to provide multilingual support across six languages, including English, French, Spanish, German, Italian, and Portuguese. SmolLM3 is positioned as a competitive alternative to larger models, offering efficient performance with long-context reasoning capabilities.

The model is built on a transformer decoder architecture and incorporates several innovations to enhance its efficiency and performance. Key features include Grouped Query Attention (GQA) and NoPE, which improve long-context performance without compromising short-context capabilities. SmolLM3's training involved a three-stage strategy using a diverse data mixture, ensuring robust performance across various domains.

SmolLM3 supports dual-mode reasoning, allowing users to toggle between reasoning and non-reasoning modes. This flexibility is facilitated through a chat template that enables seamless switching, enhancing user interaction with the model. The model's performance has been validated across multiple benchmarks, demonstrating its capability in knowledge, reasoning, math, and coding tasks.

The release of SmolLM3 includes a comprehensive engineering blueprint, detailing the architecture, data mixtures, and training methodologies used. This transparency aims to assist researchers and developers in understanding and building upon the model's capabilities.