端側推論
smolperfbenchmark Launches On-Device LLM Energy-Efficiency Tests, but Its Live Leaderboard Currently Shows Zero Results
The new open-source benchmark runs small models on Jetson devices, Macs, Raspberry Pis, and Android devices, measuring decoding speed, tokens per joule, latency, and temperature instead of focusing solely on data-center GPU throughput. Its creator says roughly 1,000 Jetson configurations have been tested, but the public leaderboard currently loads no data, leaving users to verify results through repository reports and raw artifacts.
After its September 12 release, smolperfbenchmark sparked significant discussion on LocalLLaMA by addressing an on-device dimension overlooked by mainstream model leaderboards: whether a model fits into 4GB, 8GB, or 16GB of memory, and how many tokens it can actually generate under constrained cooling and power budgets. The creator says the initial dataset includes 13 model families and roughly 1,000 configurations on the Jetson Orin Nano Super 8GB, covering throughput, TTFT, ITL, end-to-end latency, power consumption, and temperature.
Rather than reporting only a single tok/s figure, the benchmarking pipeline fixes prompt and generation lengths, runs each combination—typically about 20 times—at concurrency 1 using aiperf 0.11.0, and then calculates p50 latency and throughput. Jetson tests are locked to 7W, 15W, 25W, or MAXN mode, with power and temperature records aligned through `tegrastats`; scripts can run the same GGUF separately with llama.cpp and Ollama. The Mac workflow compares llama.cpp Metal with MLX-LM and samples CPU, GPU, and ANE power consumption using `powermetrics`.
A key aspect of the design is that it puts output tok/J on the homepage. For always-on local agents, offline RAG, or battery-powered devices, the fastest model is not necessarily the one with the lowest energy and thermal costs. However, the project also explicitly acknowledges that tok/J calculations are not yet consistent across hardware: Jetson and Mac use energy consumed during the decoding phase, while Raspberry Pi calculates the metric from average power over the entire benchmark window. The devices therefore cannot be ranked against one another directly.
The biggest issue at present is the mismatch between the project's published claims and its actual release state. The creator's post and aggregation page still cite 13 families and roughly 1,000 configurations, but at the time of verification the live leaderboard showed `0 families / 0 configs / 0 live`. The complete reports actually listed in the GitHub README are also largely limited to eight small Jetson models ranging from 135M to 1.2B parameters, plus the Bonsai series; Mac, Raspberry Pi, Android, and multi-node MoE results are mostly still being built or awaiting publication.
At this stage, it is therefore better suited as reproducible benchmarking scaffolding than as a mature hardware-buying guide. Engineers should inspect the scripts, pinned versions, and raw artifacts in each device directory directly. The next key milestones will be restoring the leaderboard data, adding quality metrics, and establishing whether third parties can reproduce results for the same model across different backends, context lengths, and thermal conditions.