ComputeArena publishes local inference benchmarking workflow with signed reports documenting models and environments

Base Compute introduced ComputeArena on September 22, turning its internal local inference benchmarking workflow into an open-source command-line tool and a public leaderboard. It records the model, quantization format, chip, and runtime environment alongside performance figures, making the test conditions behind tokens-per-second results traceable. [Technical announcement](https://huggingface.co/blog/basecompute/local-inference-benchmarks-computearena)

The tool currently integrates standalone benchmarking programs for BaseRT and llama.cpp and supports macOS on Apple Silicon and Linux. Adapters for MLX and vLLM are not yet available. The default prefill sweep ranges from 128 to 16,384 tokens, while the decoding test generates 128 tokens, with three measurements recorded. Custom workloads can be saved as local reports; leaderboard submissions require completing the default sweep. [Public implementation](https://github.com/basecompute/computearena-cli)

Model identification distinguishes among the upstream model, the actual converted artifact, and provenance evidence. The download workflow pins file versions and verifies hashes; files with unknown or ambiguous provenance remain unresolved. BaseRT and GGUF weights both labeled as four-bit therefore remain distinct variants, and their results cannot be merged based on the model name alone. [Model provenance rules](https://github.com/basecompute/computearena-cli/blob/main/docs/model-identity.md)

Reports retain token counts and elapsed times for each measurement, along with environmental information such as temperature and memory pressure. Model files and executables are hashed both before and after testing, and the report is not signed if their contents change. Results remain local until users review the data to be made public and submit them. The leaderboard currently displays rankings for specified prefill and decoding workloads. [Report design](https://huggingface.co/blog/basecompute/local-inference-benchmarks-computearena), [Public leaderboard](https://computearena.ai/)

Cooldown modes also differ in implementation: BaseRT currently waits once before the entire benchmark round, while llama.cpp waits before each separate workload process. This suggests that even when a mode with the same name is selected, thermal conditions may differ at the start of each test segment, requiring engineers to review the measurement logs. [Execution conditions](https://github.com/basecompute/computearena-cli)

The comparison still has limitations. The two runtimes differ in their warm-up behavior and workload order. A signature proves only that a report has not been modified since signing; it cannot establish that the machine reported its performance truthfully. Model provenance checks also do not guarantee that publishers' labels are correct, nor can they establish that different quantization formats deliver equivalent output quality. [Trust boundaries](https://github.com/basecompute/computearena-cli/blob/main/docs/model-identity.md), [Author's explanation](https://huggingface.co/blog/basecompute/local-inference-benchmarks-computearena)

For deployment teams, this workflow offers a starting point for rerunning tests and investigating performance differences. In practice, teams should still use matching workloads and context conditions, inspect individual timings and environmental logs, and separately assess quality on Chinese-language tasks. Developments to watch include whether the community can expand hardware coverage and whether test conditions can be aligned more closely across runtimes.