推論系統
llama.cpp Reworks Adreno Matrix Multiplication Routing, Speeding Up Prompt Prefill by 9% to 25% Across Two Chip Generations
llama.cpp b10687 avoids inefficient matrix multiplication paths selected by the OpenCL compiler on Snapdragon X2 and older A7X GPUs. Maintainers measured prompt-prefill speedups of roughly 20% to 25% for gpt-oss-20b and 9% for Gemma 3n, while per-token decoding saw no improvement.
On August 29, llama.cpp merged an OpenCL routing fix contributed by Qualcomm. Rather than adding new kernels, the change makes two generations of Adreno GPUs select existing, faster implementations by default. The Adreno X2-90 in Snapdragon X2 Elite previously used `kernel_mul_mm_f16_f32_l4_lm` to process F16 weights and F32 activations. On the standard gpt-oss-20b model, this kernel ran attention projections at only about one-quarter the speed of the optimized Q4_0 GEMM on the same machine and accounted for 40.8% of prefill GPU time. The new version therefore enables the xmem F16×F32 path by default on X2E, with maintainers reporting prefill gains of roughly 20% to 25% in separate summaries.
Another change addresses an OpenCL compiler issue on Adreno A7X GPUs. The older register allocator caused tiled F32 GEMM to use 488 bytes of private memory per work item, compared with 304 bytes on the next generation, resulting in more K-loop spills. Performance was only about one-tenth that of the F16 and Q4_K kernels on the same chip. For F32×F32 operations with batch widths greater than eight, the new routing uses a row-wise kernel instead, making Gemma-3n-E4B prefill about 9% faster on the Adreno 740. Small batches retain the original path, and the change can be disabled by setting `GGML_OPENCL_A7X_F32_LM_BYPASS=0`.
The fix shows that on-device model performance depends on more than quantization bit width and advertised compute throughput. Layers that remain in F16 or F32, the driver compiler version, and GEMM dispatch can all become major bottlenecks. However, both changes apply only to larger matrix batches, so they do not improve per-token decoding. Q8 attention already uses the DP4A path and will not benefit either. Engineering teams should benchmark prefill and decode separately using their own GGUF files, drivers, and prompt lengths rather than extrapolating one model’s percentage gains to every Snapdragon device.