Perplexity Open Sources Lily Inference Engine for Apple Silicon

Perplexity AI open sources Lily, a local inference engine built specifically for Apple silicon and the Qwen3.6-35B-A3B model, as detailed in a company blog post. Lily uses a Rust runtime and custom Metal kernels, with neither PyTorch nor MLX in its execution path, and exposes an OpenAI compatible chat completions API. A standalone demo is on GitHub.

In tests on a single MacBook Pro with an M5 Max, a 40 core GPU and 128 GB of unified memory, Lily averaged 1.23 times the prompt processing throughput and 1.35 times the response generation throughput of MLX-LM, Apple's general purpose library. The sweep covered ten prompt lengths and ten context lengths, from 256 to 128K tokens. At a 4K token prompt and 4K token context, Lily reached 5,749.9 prompt tokens per second and 186.6 generated tokens per second, against 4,737.5 and 140.9 for MLX-LM.

The model is what creates the opportunity. Qwen3.6-35B-A3B holds 35 billion parameters but activates only about 3 billion per token, routing each one to eight of 256 expert subnetworks plus a shared expert, and it mixes 10 full attention layers with 30 layers that compress earlier context into a fixed size recurrent state. That design cuts computation but produces irregular work, which a general purpose engine supporting many architectures cannot specialise for.

Perplexity's optimisations follow from that. Four bit quantisation reduces the checkpoint from roughly 70 GB of bfloat16 weights to 19.4 GB, and the packed weights are unpacked inside the matrix multiplication rather than written back to memory first. Expert routing metadata stays on the GPU, and the runtime switches to a different attention layout at contexts of 32K tokens or longer.

The engine currently targets one model on one family of chips. Perplexity says future work will widen coverage across models, chips and serving workloads, and turn the mechanisms validated on this configuration into more general runtime policy. Lily powers the hybrid compute feature in the company's Mac app, which splits a task between cloud models handling research and reasoning and a local model working with private files.