Transformers Adds GGUF Compressed-Weight Inference, Initially Focused on Apple Silicon

On September 22, Hugging Face announced a GGUF compressed-weight inference path for Transformers, allowing developers to use ggml’s Metal kernels through Python model and generation interfaces. Initial support focuses on single-conversation workloads on Apple Silicon and Qwen3.5 dense and mixture-of-experts architectures; the announcement also lists compatible Qwen3.8 weights. Installation from the main branch is currently required, and developers should not assume that standard stable releases already provide full support. [Official announcement](https://huggingface.co/blog/transformers-llama-cpp-quants)

The key difference is how weights are used during execution. The previous import path first expands quantized weights, while the new path lets matrix operations read compressed blocks directly, reducing the memory overhead of fully expanding them. Specify `gguf_file` when loading a model, then continue using existing generation and evaluation workflows. If no compatible quantization kernel is found, the loader falls back to dequantization, so deployment teams must check actual memory usage. [GGUF documentation](https://huggingface.co/docs/transformers/main/en/quantization/gguf)

The generation loop also reduces the time the CPU and GPU spend waiting for each other. The implementation asynchronously copies the device-side stop flag for reading at the next step, allowing the host to continue queuing computation. This can result in one extra step, after which the excess tokens, output records, and cache entries are trimmed. This requires a cache that supports rollback and cannot be assumed to work with every generation mode. [Implementation and limitations](https://github.com/huggingface/transformers/pull/47975)

Hugging Face tested three sets of weights on an M2 Max with 32 GB of unified memory and reported throughput close to llama.cpp. However, the llama.cpp benchmark measures only the generation of 128 tokens and averages three runs, while the Transformers benchmark includes prefill for a 12-token prompt and takes the best of three post-warm-up runs. These different measurement methods are insufficient to establish equivalent performance. [Benchmark methodology](https://huggingface.co/blog/transformers-llama-cpp-quants)

For researchers, this path makes it easier to inspect intermediate outputs, modify decoding rules, and evaluate quantization error within the same PyTorch model. Engineers should keep in mind that compressed-weight inference is currently limited to MPS, and other architectures may continue to use dequantization. Batches that require padding also cannot use every optimization. Next steps to watch include integration into a stable release, batching support, and quality, latency, and peak memory usage on long-form Chinese text and coding tasks. [Scope of support](https://huggingface.co/docs/transformers/main/en/quantization/gguf)

For Chinese-language applications, quality comparisons between original and quantized weights should use identical chat templates, output limits, and sampling settings. For long documents, time to first token and subsequent generation speed should be recorded separately. Testing should also cover cold starts and sustained serving, avoiding estimates of user experience based solely on short-prompt benchmarks after warm-up. These deployment checks remain to be completed; they are not Chinese-language benchmark results provided in the announcement.