Mozilla.ai Benchmarks Local AI Servers, Finds Setup Drives Bigger Gains

The fastest local AI setup may not come from switching server brands. Mozilla.ai’s new benchmark found that four popular tools were generally close on prompt processing when they used the same model files and environment, while build and configuration choices produced the much larger gains.

The study compared llama.cpp, llamafile, LM Studio and Ollama on an Apple M4 Max, an NVIDIA L40S Linux system and a Steam Deck. It used matching GGUF model weights across Qwen models at roughly 0.8 billion, 9 billion and, where memory allowed, 27 billion parameters.

That control matters. Mozilla.ai says all four servers use a llama.cpp-based core for GGUF workloads, which helps explain why their prompt-processing results usually landed within a few percent once the weights and test environment were held constant. The study is a practical snapshot rather than a final ranking, and each runtime kept its own batching defaults.

The benchmark separates two jobs often collapsed into one speed claim. Prompt processing is the work of reading an input before an answer begins; token generation is the rate at which the answer appears. The first can dominate long-document or large-codebase work, while the second is more visible in chat and code completion.

On the L40S, enabling CUDA graphs in llamafile improved generation throughput by 16.8% on the 0.8B model. The gain fell to 6.5% for 9B and 4.3% for 27B.

The Steam Deck produced the widest spread, but Mozilla.ai traced most of it to the Vulkan shader compiler rather than a different inference engine. Rebuilding the same llamafile/llama.cpp code with a newer toolchain raised prompt processing by 25.9% on the 0.8B model and 63.3% on the 9B model. Decode speed for the 9B model rose 18.2%.

Speculative-decoding settings also resisted a universal recommendation. For the tested 27B configuration, draft length 2 worked best on Metal, while draft length 4 beat it on CUDA by up to 16%. On the M4 Max, a four-token draft was slower than no speculation in the reported comparison.

Some apparent generation gaps may also be less about GPU compute than the software around it. The researchers measured an approximately fixed host-side cost per generated token: 0.4 milliseconds on the Mac, 1.8 on the L40S and 5.4 on the Deck. A near-constant delay becomes a larger percentage penalty when the underlying model is faster.

Mozilla.ai also tested the Apple-focused MLX model artifacts that Ollama and LM Studio offer. Each improved 27B prompt processing by roughly 10% over the shared GGUF setup, but the comparison did not establish one engine’s superiority: the artifacts differed, and Ollama enabled speculative decoding while LM Studio did not.

Compatibility can matter before throughput. On an approximately decade-old Intel MacBook Pro, llama.cpp and llamafile ran on the installed macOS, while the current Ollama distribution did not support that system version and LM Studio’s current release supported Apple Silicon rather than Intel Macs. That check was not a comparable performance test.