模型推論與壓縮
ICBQ Revisits Quantized Block Boundaries, Cutting Qwen3-8B Ternary-Model Perplexity From 5752.9 to 29.9
Rather than replacing the quantizer, ICBQ jointly recalibrates the boundaries between adjacent Transformer blocks a second time, reducing the accumulation of early errors with depth. The quality improvement comes at the cost of a 21% average increase in quantization time, and the authors have not yet released an implementation.
Existing cross-block post-training quantization methods typically use two Transformer blocks as a window, calibrating them sequentially from the front of the model to the back. Once the window moves on, errors introduced earlier are no longer addressed. Interleaved Cross-Block Quantization (ICBQ) changes neither weight encoding nor the loss function, but the schedule: it divides the network into multiple chunks, allowing each pair of blocks at a chunk boundary to be optimized first at the end of the preceding chunk and then revisited at the beginning of the next.
The design can be applied to ternary DBF and can also use GPTQ as its internal quantizer. In ternary experiments calibrated on C4, Qwen3-8B recorded a perplexity of 5752.9 under conventional sequential cross-block quantization; with ICBQ using a chunk size of 4, it fell to 29.90. Mistral-7B dropped from 31.1 to 16.8, while Qwen3-14B fell from 44.82 to 24.67. ICBQ achieved lower perplexity across all seven major models in the ternary comparison, while average zero-shot accuracy was unchanged or higher on six models. Llama-2-13B was the exception, with the sequential version ahead by just 0.0002.
The gains are not free. Across 11 models, end-to-end quantization took an average of 1.21 times as long as the sequential method. For Qwen3-14B, the time increased from about 15 hours to 19 hours and 39 minutes. Storing calibration activations also increased memory usage—by roughly 4.3GB for 7B/8B-class configurations and about 5.2GB for 13B/14B-class configurations. This is a quality-versus-cost trade-off for offline checkpoint generation and should not be interpreted as an improvement in inference throughput.
From an engineering perspective, the results suggest that low-bit quantization failures may stem from the optimization order, not only from the quantization codebook. Existing GPTQ or cross-block pipelines could experiment with revisiting chunk boundaries, but should measure calibration time, peak memory usage, downstream-task performance, and actual kernel speed as well. The paper has not yet released code or quantized checkpoints; until the results are independently reproduced, the dramatic Qwen improvements may also depend heavily on the authors' DBF implementation and calibration configuration.