Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That Beats MLX-LM in Its Test

Perplexity has open-sourced Lily, the local inference engine behind Hybrid Compute in Perplexity Computer. In the company’s benchmark, Lily ran Qwen3.6-35B-A3B faster than MLX-LM on an M5 Max MacBook Pro—not by offering a general Mac AI framework, but by committing to one model and Apple silicon.

Lily is a single-process Rust runtime that loads the model and drives generation, while hand-written Metal kernels execute its computations. It offers a streaming API compatible with OpenAI’s chat-completions format, without PyTorch or MLX in the execution path. A standalone greedy-generation server is available in Perplexity’s pplx-garden repository.

A runtime shaped around one model

The target, Qwen3.6-35B-A3B, is a sparse mixture-of-experts model with 35 billion parameters that activates roughly 3 billion per token. Each token is routed to eight of 256 experts plus a shared expert; the model also combines 10 full-attention layers with 30 Gated DeltaNet layers. Those design choices create uneven expert workloads, a growing attention cache and a recurrent state for the runtime to manage.

The optimizations follow the workload

Prompt processing, called prefill, works through many input tokens at once. Lily reconstructs 4-bit weights inside grouped matrix multiplication rather than expanding a full weight array in unified memory. It also keeps expert routing on the GPU, avoiding CPU synchronization during mixture-of-experts work. Perplexity says those two changes raised end-to-end prefill throughput by 77.4% and 89%, respectively, at a 512-token prompt.

Generation has a different bottleneck

• Decode generates one token at a time, so Lily keeps the selected token on the GPU for the next step and schedules independent kernels concurrently.
• Its grouped-query-attention packing and alternate attention layouts improved end-to-end decode throughput by up to 40.2% at a 128K-token context in Perplexity’s testing.

A useful release with narrow evidence

Across prompt and context lengths from 256 to 128K tokens, Perplexity measured a 1.23-times prefill advantage and a 1.35-times decode advantage over MLX-LM. A teacher-forced check across 192 positions found Lily’s perplexity was 0.04% higher than the comparison and its top-ranked token matched at 96.35% of positions; that checks output consistency, not broad model behavior.

The 4-bit checkpoint is 19.4 GB, with 32 GB or more of unified memory identified as a realistic Mac floor. Lily is therefore most relevant to developers running this exact Qwen configuration locally. Perplexity also found speculative decoding cut batch-one decode performance by 18%, a reminder that an optimization suited to one inference path can be a poor fit for another.