Red Hat Launches llm-d for Scalable AI Inference

Red Hat has launched llm-d, a new open-source initiative designed to address the growing need for scalable generative AI inference, announced in a press release. This project is supported by founding contributors such as CoreWeave, Google Cloud, IBM Research, and NVIDIA, and aims to make AI inference as ubiquitous as Linux.

The llm-d project leverages a native Kubernetes architecture, vLLM-based distributed inference, and AI-aware network routing to optimize compute resources and deliver AI inference at a massive scale. This approach is intended to meet the demanding service-level objectives of production environments without compromising performance.

Key innovations of llm-d include vLLM, which supports a wide range of accelerators, and Prefill and Decode Disaggregation, which separates AI input context and token generation into discrete operations. Additionally, the project features KV Cache Offloading to reduce memory burdens and AI-Aware Network Routing for efficient request scheduling.

The initiative has garnered support from industry leaders such as AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, as well as academic institutions like the University of California, Berkeley, and the University of Chicago. This collaboration underscores the industry's commitment to advancing large-scale AI inference capabilities.