推論與量化
Diffusers Natively Integrates Nunchaku Lite, Using W4A4 to Reduce Diffusion Model VRAM Usage and Latency
Hugging Face Diffusers can now directly load Nunchaku Lite checkpoints to run diffusion Transformers with 4-bit weights and activations. Official single-machine tests showed up to roughly half the peak VRAM usage and a 1.8× speedup, but hardware coverage and image-quality validation remain limited.
On July 23, Hugging Face announced native Diffusers integration for Nunchaku Lite, enabling developers to load SVDQuant-quantized diffusion models using the standard `from_pretrained()` method. This eliminates the need to maintain the separate Nunchaku inference engine or compile CUDA extensions locally. Core operators are downloaded through the Hugging Face `kernels` package, while model repositories record the quantization method, precision, group size, low-rank rank, and target modules in `config.json`. As a result, quantized checkpoints remain compatible with Diffusers workflows involving schedulers, LoRA, CPU offload, and `torch.compile`.
Most weight-only quantization methods restore weights to a higher precision during computation, primarily reducing VRAM usage without necessarily lowering latency. Nunchaku Lite applies SVDQ W4A4 to attention and MLP projections: it first migrates hard-to-quantize activation outliers into the weights, then uses a small 16-bit low-rank branch for correction, while the remaining matrix uses 4-bit weights and activations. More sensitive modulation layers, or those limited by memory bandwidth, use AWQ W4A16 instead. Before loading the weights, Diffusers replaces the specified `nn.Linear` layers with the corresponding runtime modules.
In official tests of ERNIE-Image-Turbo at 1024×1024 on an RTX PRO 6000 Blackwell, the BF16 pipeline took approximately 3.00 seconds and used 31.1 GB of peak VRAM. The NVFP4 version reduced this to 2.27 seconds and 20.6 GB, or approximately 1.68 seconds with `torch.compile`. Quantizing the text encoder further reduced VRAM usage to around 16 GB. However, these remain team-run tests using a single model, GPU, and prompt configuration. NVFP4 supports only Blackwell; older Turing, Ampere, and Ada GPUs must use INT4, while Volta and Hopper are currently unsupported. In addition, the general-purpose Lite path lacks the model-specific QKV and MLP fusion available in the original Nunchaku, so deployments targeting maximum throughput may still require architecture-specific adapters.