GitHub Repo
Ollama Community Test: Quantized Decision Models Are Slower Than BF16 at Prompt Processing on M5 Pro
An October 7 Ollama community test found higher latency for long-prompt requests with three decision models using MLX quantized formats on M5 Pro than with BF16. The report includes measurement methods and reproduction instructions, but maintainers have yet to confirm whether dequantization is a bottleneck.
On October 7, the Ollama community posted a performance report worth noting for local-agent developers: on an M5 Pro with 48 GB of memory and Ollama 0.40.0, MLX-quantized versions of three decision models were slower than their BF16 versions when processing longer prompts. The test came from a real-time Snake game, where each model decision had to finish before the next game update cycle, so the metric was request latency. Test report
The reporter used /v1/systemone and tested eight distinct requests at each of four input lengths: about 120, 500, 1,100, and 1,900 tokens. No other models were loaded during testing. At about 1,900 tokens, tev1 had a median latency of 685 ms in BF16 and 759 ms in NVFP4; nimble measured 1,234 ms and 1,433 ms, respectively; and clef-flash measured 1,280 ms in BF16 and 1,423 ms in MXFP8. The gap was smaller for short inputs and widened as prompts grew longer. Measurement data
The result is a reminder for engineers to validate memory footprint and response speed separately. MLX’s quantized matrix multiplication interface uses group-scale information in addition to compressing weights, with computation controlled by bit width and quantization mode. The documentation describes the data representation; it does not guarantee that using fewer bits will reduce latency for every workload. From an engineering perspective, if an application spends most of its time processing the full prompt, teams should directly compare measured time for that stage. MLX technical documentation
The reporter attributed the gap to prefill and speculated that dequantization costs might affect computation. No kernel profiling has confirmed this explanation, and there is no maintainer response or fix linked yet. This single-machine test cannot be generalized to all Apple chips, models, or text-generation scenarios. Teams preparing to deploy real-time decision agents should rerun the public instructions with their own input lengths, record memory use and latency distributions, and watch for upstream cross-hardware reproductions and quantized-kernel analysis. Reproduction steps and current status