端側推論
BaseRT Directly Drives M5 Neural Accelerators via Metal 4, Achieving Up to 6.4× Faster Prompt Processing Than llama.cpp
BaseRT has added hand-written dense and MoE GEMM and FlashAttention kernels, offloading matrix operations during the prompt-processing stage to the neural accelerators inside the M5 GPU. The team reports substantial leads over llama.cpp and MLX on the M5 Pro, although the figures currently come solely from the developers’ own single-machine tests.
Base Compute has released a new version of BaseRT alongside a technical report demonstrating how to bypass general-purpose frameworks and access the neural accelerators built into each Apple M5 GPU core directly through the Metal 4 tensor API. These units are well suited to large matrix multiplications, so the new release adds hand-written dense and mixture-of-experts (MoE) GEMM kernels, quantized matrix kernels, and FlashAttention kernels for prefill. Memory-bandwidth-bound token-by-token decoding continues to use the existing specialized kernels rather than being forced onto the matrix units.
The team tested 15 model configurations on an M5 Pro, covering Qwen 3, Qwen 3.5/3.6, Llama 3.2, and Gemma 4, with model sizes ranging from under 1 billion to 35 billion parameters. According to the report, BaseRT’s prompt-processing throughput is up to 6.4× that of llama.cpp and 3.9× that of MLX; decoding is up to 1.75× and 1.33× faster, respectively. The gap is particularly pronounced for MoE models because prefill requires extensive matrix computation that can be handled by the neural accelerators, whereas decoding remains more constrained by unified memory bandwidth.
The technical significance goes beyond introducing another local LLM runtime: it shows that the optimization frontier on Apple Silicon has expanded from GPU shaders to dedicated matrix hardware. For developers handling long prompts, RAG documents, or batches of concurrent requests, faster prefill may be more valuable than generation speed alone. A native Metal path can also reduce dependence on Python and large frameworks.
However, all comparisons were provided by the BaseRT team and focused on a single M5 Pro configuration. Quantization formats, prompt lengths, power consumption, memory usage, and output consistency could all alter the rankings. The next step is to see whether third parties can reproduce the results using identical model files, precision settings, and thermal and power conditions, and to determine whether the hand-written kernels can keep pace with changes in new model architectures and the Metal API.