Superpower Daily
Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That B...Uber and Wayve Put 15 Supervised Robotaxis on London’s StreetsLlamaIndex and Kaggle Launch a Document Benchmark for Cost and...Lyte Raises $165M at $1.6B Valuation for Robot PerceptionAbliteration Removes Refusal Mechanisms for Offensive Cyber WorkNew York City Public Schools Will Block Student AI Use Through...Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That B...Uber and Wayve Put 15 Supervised Robotaxis on London’s StreetsLlamaIndex and Kaggle Launch a Document Benchmark for Cost and...Lyte Raises $165M at $1.6B Valuation for Robot PerceptionAbliteration Removes Refusal Mechanisms for Offensive Cyber WorkNew York City Public Schools Will Block Student AI Use Through...
Infrastructurepublished

Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That Beats MLX-LM in Its Test

The Rust-and-Metal runtime gives developers a public, deployable version of Perplexity’s local inference stack. Its reported gains come from one Qwen model, one hardware family and a batch-one comparison on a high-memory M5 Max.

By 3 min read
Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That Beats MLX-LM in Its Test
Perplexity Open-Sources Lily, a Qwen-Specific Mac Engine That Beats MLX-LM in Its Test
Sign in to save and react

Listen to this story

The audio brief

About 1:44
0:001:44
Read transcript
Perplexity has open-sourced Lily, a Rust-and-Metal inference engine that it says runs one Qwen model faster than MLX-LM on Apple silicon. The important qualification is that Lily is not a general Mac AI framework. It is tuned for Qwen3.6-35B-A3B, a sparse mixture-of-experts model, and for batch-one generation on a high-memory M5 Max. On a 40-core machine with 128 gigabytes of unified memory, Perplexity measured 4,156 prefill tokens per second, versus 3,388 for MLX-LM. During decoding, Lily reached 170 tokens per second, compared with 126.4. Across prompts and contexts from 256 to 128,000 tokens, the reported averages were 1.23 times higher for prefill and 1.35 times higher for decode. The gains come from specializing the runtime. Lily reconstructs four-bit weights inside grouped matrix multiplication, keeps expert routing on the GPU, and schedules parts of one-token generation concurrently. It runs as a single Rust process with handwritten Metal kernels, without PyTorch or MLX in the execution path, and includes an OpenAI-compatible streaming API. For developers, the practical target is narrow: the four-bit checkpoint is 19.4 gigabytes, and Perplexity identifies 32 gigabytes of unified memory as a realistic floor. Quality checks showed close output consistency, but speculative decoding actually reduced batch-one speed by 18 percent. The constraint to watch is whether Lily’s advantage survives beyond this one Qwen model, Mac hardware family, and inference path.

Story brief

3 key points

Perplexity has released Lily, a Rust-and-Metal runtime whose performance comes from specializing in Qwen3.6-35B-A3B rather than supporting Mac inference broadly. In batch-one tests on a 40-core, 128GB M5 Max, Lily reached 4,156 prefill and 170 decode tokens per second, ahead of MLX-LM. The practical audience is developers running this specific 4-bit model locally: its checkpoint is 19.4GB, 32GB unified memory is the...

  1. 01

    Across 256–128K-token tests, Lily averaged 1.23× higher prefill and 1.35× higher decode throughput than MLX-LM.

  2. 02

    GPU-resident expert routing and fused 4-bit weight reconstruction targeted MoE prefill bottlenecks; reported gains reached 89% and 77.4%.

  3. 03

    Quality checks found 96.35% top-token agreement and 0.04% higher perplexity, measuring consistency rather than broad model behavior.

Perplexity has open-sourced Lily, the local inference engine behind Hybrid Compute in Perplexity Computer. In the company’s benchmark, Lily ran Qwen3.6-35B-A3B faster than MLX-LM on an M5 Max MacBook Pro—not by offering a general Mac AI framework, but by committing to one model and Apple silicon.

Lily is a single-process Rust runtime that loads the model and drives generation, while hand-written Metal kernels execute its computations. It offers a streaming API compatible with OpenAI’s chat-completions format, without PyTorch or MLX in the execution path. A standalone greedy-generation server is available in Perplexity’s pplx-garden repository.

A runtime shaped around one model

The target, Qwen3.6-35B-A3B, is a sparse mixture-of-experts model with 35 billion parameters that activates roughly 3 billion per token. Each token is routed to eight of 256 experts plus a shared expert; the model also combines 10 full-attention layers with 30 Gated DeltaNet layers. Those design choices create uneven expert workloads, a growing attention cache and a recurrent state for the runtime to manage.

The optimizations follow the workload

Prompt processing, called prefill, works through many input tokens at once. Lily reconstructs 4-bit weights inside grouped matrix multiplication rather than expanding a full weight array in unified memory. It also keeps expert routing on the GPU, avoiding CPU synchronization during mixture-of-experts work. Perplexity says those two changes raised end-to-end prefill throughput by 77.4% and 89%, respectively, at a 512-token prompt.

Generation has a different bottleneck

  • Decode generates one token at a time, so Lily keeps the selected token on the GPU for the next step and schedules independent kernels concurrently.
  • Its grouped-query-attention packing and alternate attention layouts improved end-to-end decode throughput by up to 40.2% at a 128K-token context in Perplexity’s testing.

A useful release with narrow evidence

Across prompt and context lengths from 256 to 128K tokens, Perplexity measured a 1.23-times prefill advantage and a 1.35-times decode advantage over MLX-LM. A teacher-forced check across 192 positions found Lily’s perplexity was 0.04% higher than the comparison and its top-ranked token matched at 96.35% of positions; that checks output consistency, not broad model behavior.

The 4-bit checkpoint is 19.4 GB, with 32 GB or more of unified memory identified as a realistic Mac floor. Lily is therefore most relevant to developers running this exact Qwen configuration locally. Perplexity also found speculative decoding cut batch-one decode performance by 18%, a reminder that an optimization suited to one inference path can be a poor fit for another.

Sources

  1. marktechpost.comPerplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon

Related coverage

Continue reading