Nunchaku Lite Lands Natively in Diffusers, Reducing Diffusion Transformer VRAM and Latency With W4A4 Quantization

Hugging Face Diffusers has added a native loading path for Nunchaku Lite, allowing users to run SVDQuant checkpoints through the standard `from_pretrained()` interface. Traditional weight-only quantization dequantizes low-precision weights before computation, which saves VRAM but does not necessarily improve latency. SVDQuant instead moves activation outliers into the weights, preserving hard-to-quantize signals with a small 16-bit low-rank branch while allowing the main linear layers to run in W4A4. For precision-sensitive layers such as normalization and modulation layers, the integration uses W4A16 AWQ.

Before loading the weights, the new path replaces compatible `nn.Linear` modules in a standard Diffusers model with SVDQ or AWQ execution layers. CUDA kernels are downloaded through the Hugging Face `kernels` package, with no local compilation required. In official tests using ERNIE-Image-Turbo at 1024×1024 on an RTX PRO 6000 Blackwell, peak VRAM usage fell from 31.1 GB to as little as 16.0 GB. The full pipeline took 2.27 seconds with NVFP4 alone and 1.68 seconds with `torch.compile`, compared with a BF16 baseline of 3.00 seconds.

Generality also brings trade-offs: Nunchaku Lite does not automatically perform architecture-specific rewrites such as QKV fusion. As a result, the basic path is about 30% faster but may still be slower than the specialized fused kernels in the original Nunchaku. NVFP4 is supported only on Blackwell; older NVIDIA GPUs must use INT4, while Volta and Hopper are currently excluded from support for its 4-bit kernels. Teams should remeasure image quality, LoRA compatibility, compilation time, and batch throughput on their target models rather than extrapolating directly from results for a single image model.