端側 AI 與評測
Pipette Open-Sources On-Device Model Benchmarking for Accuracy, Latency, and Memory—but Rankings Still Cannot Be Compared Directly Across Devices
Liquid AI and Artificial Analysis have released Pipette, which measures on-device inference across more than 1,000 combinations of models, quantizations, runtimes, devices, and context configurations. Speed and memory are measured on physical devices, while quality scores are generated separately on H100 GPUs, so the two testing paths must be distinguished when interpreting its combined charts.
Liquid AI and Artificial Analysis launched Pipette on August 24, aiming to fill a gap commonly left by model cards: how a quantized model actually performs on a specific phone, operating system, and inference runtime. The initial dataset covers more than 1,000 “model × quantization × runtime × device × context” configurations, over 30 models, and context lengths ranging from 256 to 8,192 tokens. The client, orchestration, and evaluation components are all available under the Apache 2.0 license.
Pipette measures more than output tokens per second. It fixes input and output lengths, disables prompt caching and early stopping, discards one warm-up run, and then repeats each measurement five times. It also checks device temperature and system load before execution. Submission records preserve the model artifact, quantization format, runtime version, flags, hardware, and operating system, preventing different deployment artifacts bearing the same model name from being conflated. The Rust client currently supports llama.cpp, MLX, OpenVINO, vLLM, and SGLang, while native iOS and Android apps are also available.
Initial iPhone 17 Pro results show that 33 of 41 quantized builds ran successfully. In the standard test with a 1,024-token input and 256-token output, LFM2.5-2.6B required 8.0 seconds and 2.3 GB of memory. Nanbeige4.2-3B, which received the same quality score, required 21.4 seconds and 4.0 GB, demonstrating that similar quality does not imply similar deployment costs.
The biggest interpretive pitfall is that quality and performance are not measured during the same on-device run. Quality evaluations for BFCL, IFBench, AA-Omniscience, GPQA Diamond, and MATH-500 use the same quantized builds but are conducted on an H100 reference system; devices are used only for speed and memory measurements. In addition, iOS uses Metal, while the current Android path primarily uses the CPU, and memory-accounting methods and thermal conditions also differ. At this stage, comparisons between configurations on the same device are the most reliable. The leaderboard should not be used to claim that a particular phone or runtime is superior across the board.