Perplexity has open sourced Lily, a Rust and Metal inference engine built for Qwen3.6-35B-A3B on Apple silicon. The project powers the local side of Hybrid Compute in Perplexity Computer, a feature that splits tasks between cloud and local AI models.
The announcements dated September 1, 2026 and September 2, 2026 put a working local model engine in developers’ hands, rather than keeping it inside the Mac app. A standalone Lily demo is available in the pplx-garden repository, where it offers greedy text generation through a minimal OpenAI-compatible HTTP API.
That makes Lily easier to test as a separate engine. The project is focused, too: it targets one model, one hardware family, and a clear goal of running a large model within the memory limits of Apple silicon Macs.
A 35B Model Built for Local Memory
Qwen3.6-35B-A3B has 35B parameters, but only 3B parameters activate for each token. Its design includes 256 experts, with eight experts selected for a token, along with 10 full-attention layers, 16 query heads, and two KV heads.
Lily uses groupwise affine 4-bit quantization to shrink about 70 GB of bfloat16 weights into a 19.4 GB checkpoint. Perplexity recommends an Apple silicon Mac with 32 GB or more of unified memory for Lily, giving the model room to run alongside the rest of the system.
The shipping Hybrid Compute product lists macOS 15+, a 24 GB minimum, and 32 GB for best results. Those figures give the project two practical entry points: the product has a stated minimum, while the Lily engine has a stronger memory recommendation for its standalone use.
Perplexity’s Jon Staff described the Mac app’s handling of cloud-bound information this way: “This is integrated directly into the Mac app. Any time you try to upload files or send information, we’re going to automatically check for sensitive content and make sure that you want to share that data to the cloud.”
Where the Engine Gains Speed
Lily reconstructs weights inside the grouped GEMM, one tile at a time. During prefill, the engine keeps weights packed and keeps routing on the GPU, choices that target the cost of processing a prompt before generation begins.
Moving from 16-row tiles to 32-row tiles with four simdgroups added 13.2% at 2K tokens. A register-resident Gated DeltaNet scan added another 5.6%, while expert GEMMs took roughly 90% of prefill time. Together, these changes produced a 77.4% end-to-end prefill increase, with 89% happening inside a single GPU command buffer.
Decode presented a different problem. Instead of focusing on prompt processing, Lily worked to minimize the bytes moved for each token. One recorded step launched 795 kernels across 555 sequential stages, so the path between one generated token and the next mattered as much as the individual operations.
The selected token goes straight into the next step’s GPU-resident input slot. Coalesced cache reads lifted key bandwidth from 33.8 GB/s to 47.9 GB/s, while value bandwidth rose from 42.0 GB/s to 61.8 GB/s.
GQA packing improved decode by 23.8% at 32K. A fixed-block attention layout at 32K and above added 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K. These gains focus on long context sizes, where cache movement and attention layout place more pressure on the decode path.
Performance and Accuracy
On a 40-core, 128 GB M5 Max, Lily averaged 4,156 prefill tokens/sec and 170.0 decode tokens/sec. The comparison figures were 3,388 prefill tokens/sec and 126.4 decode tokens/sec, giving Lily a 1.23x prefill speed and a 1.35x decode speed.
Those results show why the project centers on Apple silicon optimization instead of treating the hardware as a generic target. The engine’s gains come from decisions across memory packing, GPU routing, tile size, cache reads, attention layout, and the handoff between decode steps.
Speed is only useful if the output stays close to the model’s expected behavior. In a teacher-forced check across 192 positions, Lily’s perplexity was 0.04% higher, and it produced the same top-ranked token 96.35% of the time.
Hybrid Compute also gives the technical work a product role. Staff said, “The short answer is that a fully frontier output is going to almost always be better in terms of raw artifact creation. It’s more expensive, it’s more capable.” The split between cloud and local models is therefore part of how Perplexity Computer handles different levels of work.
Staff also said, “The system generates an output, you can write follow-up instructions as usual. You can also use an iPhone to queue up tasks.” Lily’s release connects that broader workflow to an open-source local engine, a public demo, and a small HTTP interface that developers can inspect and use.
Based on
