Liquid AI’s Pipette Tests 1,000+ On-Device AI Setups—and Limits Cross-Device Rankings

Liquid AI has released Pipette, an open-source benchmarking platform built around a consequential premise: on-device AI behavior belongs to the full deployment configuration, not the model alone. The launch dataset covers more than 1,000 configurations across more than 30 models, giving developers a public way to inspect model, quantization, runtime, device and context length together.

The tool targets a mismatch in model selection. Model cards can report quality under server-class, full-precision conditions, while a phone deployment may use a quantized model and a particular software stack. Pipette measures throughput, latency and memory for each configuration, with quality tracked separately.

From a model score to a deployment test

Artificial Analysis partnered with Liquid AI to independently review and verify Pipette’s methodology. Initial verified results cover a MacBook Pro with M5 Max, an iPhone 17 Pro and a Galaxy S26 Ultra. The dataset supports llama.cpp builds across macOS, iOS, Windows and Android, using context lengths from 256 to 8,192 tokens.

What ships with Pipette

• Apache 2.0-licensed infrastructure: pipette-mgmt, pipette-clients and pipette-scores.
• A public results dataset and hosted dashboard for inspecting benchmark configurations.
• Benchmark clients for macOS, Windows, iOS and Android, plus native iOS and Android apps.

One phone, two context curves

Pipette’s published examples show why model size alone is a weak shortcut. On the same Galaxy S26 Ultra, using Q4_K_M quantization, two 350M Granite models retained sharply different shares of their decode speed as the input context grew from 256 to 4,096 tokens.

Decode-throughput retention at 4,096 tokens

0178.4%

Granite-4.0-H-350M

At Q4_K_M on a Galaxy S26 Ultra, Granite-4.0-H-350M retained 78.4% of its 256-token decode throughput at a 4,096-token input context.

0233.8%

Granite-4.0-350M

Under the same conditions, Granite-4.0-350M retained 33.8% of its 256-token decode throughput at a 4,096-token input context.

Reproducibility has a scope limit

Performance runs use fixed token shapes, greedy decoding, a discarded warm-up, five measured repetitions, and thermal and system-load readiness checks. The scoring path is separated from generation: pipette-scores uses deterministic, model-blind scoring without generation provenance.

The initial release omits consistent NPU comparisons and Android GPU results. It also advises against comparing results across devices because iOS and Android runs use different execution environments; the current data is intended for comparisons among configurations on the same device.

Pipette presents IFBench, GPQA Diamond and MATH-500 quality evaluations beside on-device performance. Those quality scores come from llama.cpp runs on NVIDIA H100 80GB reference systems, then are matched to runs sharing the same model and quantization. They are not quality measurements made on the phone or laptop.