GitHub Repo
Transformers integrates ggml quantization kernels for direct inference with compressed GGUF weights on Apple Silicon
Hugging Face has announced GGUF inference integration on the main branch, bringing compressed weights into existing Python and PyTorch workflows. Initial support focuses on Apple Silicon and Qwen3.5 architectures, though the benchmarks use different measurement methods.
On September 22, Hugging Face announced GGUF quantized inference integration in Transformers, allowing developers to keep weights compressed on Apple Silicon while continuing to use Python model and generation interfaces. The announcement covers the integration approach and benchmark results; the official instructions require installation from the main branch and do not yet describe it as a stable release feature. [Official announcement](https://huggingface.co/blog/transformers-llama-cpp-quants)
The key difference is how computation proceeds after loading. Compatible ggml kernels perform matrix operations directly on packed quantization blocks, eliminating the need to first expand the entire set of weights. Developers simply specify `gguf_file` when loading and can then use their existing generation workflow. If no compatible kernel is available, the system falls back to full dequantization, increasing memory requirements. Successfully loading a file therefore does not guarantee that the memory-saving path is being used. The documentation also notes that loading compressed weights automatically selects a single-precision compute dtype because it is faster on MPS for this path; explicitly specifying another dtype triggers a warning. Weight storage precision and compute dtype therefore need to be checked separately. [GGUF documentation](https://huggingface.co/docs/transformers/main/en/quantization/gguf)
The generation loop also reduces waiting: redundant masks are removed early for supported inputs without padding, while stopping-condition checks use asynchronous copies and defer reading the results by one step, allowing the CPU to continue queuing GPU work. Any extra output beyond the stopping point is trimmed. These changes had already been merged before the announcement and should not be interpreted as having all launched that day. [Stopping-check fix](https://github.com/huggingface/transformers/pull/47975), [mask fix](https://github.com/huggingface/transformers/pull/48814)
The performance comparisons still differ in methodology. Hugging Face tested three models on an M2 Max with 32 GB of unified memory, but the llama.cpp figures average three decoding measurements, while the Transformers figures use the best result after three warmup runs and include prefill for a short prompt. These results can serve as a starting point for local experiments, but they do not establish that the two runtimes deliver identical performance. [Benchmark methodology](https://huggingface.co/blog/transformers-llama-cpp-quants)
The compressed inference path is currently limited to MPS, with support focused on Qwen3.5 dense and mixture-of-experts architectures. Padding and batching still need improvement. For researchers, the value lies in bringing quantized checkpoints into familiar evaluation, model-hook, and custom decoding workflows. When adopting the integration, pin the commit and kernel versions, verify that no dequantization fallback occurs, and measure quality on Chinese-language tasks, time to first token, and peak memory usage separately. Also record the initial kernel download and warmup times to avoid mixing startup costs into steady-state speed measurements. Results from single-user conversations with short prompts remain insufficient to represent long-context workloads or multi-user serving. [Scope of support](https://huggingface.co/docs/transformers/main/en/quantization/gguf)