模型量化與推論
ReRound Uses a Diffusion Prior to Rechoose Quantization Rounding, Boosting Average Accuracy of 3-/4-Bit Small Models by Up to 1.6 Points
ReRound uses no calibration corpus. Instead, it trains a conditional diffusion model on the model’s own weights to reconsider whether weights near the midpoint of a quantization interval should be rounded up or down. It preserves the existing low-bit format at deployment with no additional inference overhead, but each model requires several hours of model-specific reconstruction upfront.
Low-bit quantization typically leaves weights to round-to-nearest (RTN). However, for weights close to the midpoint between two quantized values, the scalar error is nearly identical whether rounding up or down, yet the accumulation of these small choices can still alter an entire linear layer. [ReRound](https://arxiv.org/abs/2608.11045), released on August 11, specifically revisits these ambiguous rounding decisions without reading calibration text, activations, or downstream labels.
The method first extracts 64×64 blocks from the weight matrices of the model being quantized and trains a conditional U-Net diffusion model to reconstruct continuous weights from low-bit observations. The reconstructed values serve only as a prior for the rounding direction and do not become the deployed weights; RTN is retained for values farther from an interval midpoint. Different tolerance settings produce multiple candidate sets of integer weights. The system then compares the leading singular values of each dequantized matrix with those of the original matrix and selects the candidate that better preserves the dominant transformation structure.
Across eight models ranging from 1B to 2.7B parameters—including Gemma, Qwen3, OLMo, SmolLM2, Llama, Pythia, and Phi—ReRound achieved higher four-task average scores than RTN at both 3-bit and 4-bit precision under the same scale, zero-point, group size of 128, and quantized-layer scope. The gains ranged from 0.1 to 1.6 percentage points. In 4-bit comparisons on six models, it also outperformed AdaRound, which used 128 C4 calibration samples, by as much as 1.9 points.
The cost is concentrated in the offline stage. According to the paper, diffusion training requires two GPUs and approximately 1.76 to 3.43 hours, while reconstruction on a single GPU takes another 3.65 to 9.55 hours. Once that is complete, the actual quantization takes about 42 to 124 seconds, and the inference format is identical to RTN. The [codebase](https://github.com/louisYen/ReRound) already provides configurations for the RTX 3090 and RTX 4090. Engineering teams should still note that the evaluation covers only four zero-shot tasks and a single seed, and spectral similarity does not guarantee optimal performance on a specific downstream task. Because the diffusion model must be retrained for every checkpoint, the method may not be cost-effective for large numbers of model versions.