模型壓縮研究
Wringer Compresses Low-Rank Distillation Results Back into Low-Bit Weights, but So Far It Only Saves Storage Space
Wringer first repairs a 2.655 bpw model using temporary LoRA adapters, then re-solves the corrections into integer codes and scales so no adapter is retained at inference time. The released model preserves about 94.65% of the BF16 score across three benchmarks, but the full model is actually 4.69 bpw and lacks a native low-bit kernel.
Independent researcher Weiciao Wu has released Wringer, a “fill then wring” low-bit compression pipeline, along with the corresponding Agents‑A1‑4B checkpoint. The target is a 32-layer Qwen 3.5 hybrid model comprising 25 linear-attention layers and seven global-attention layers. The experiments were conducted on a single 96GB RTX PRO 6000, and the code, 69 preregistered experiments, evaluation outputs, and compressed container have been published on GitHub and Hugging Face.
The first step jointly solves for block scales using GPTQ with a full second-order matrix, a signed even grid, and Tikhonov regularization. Next, the “fill” stage freezes the integer codes and attaches a rank-128 adapter to every quantized linear layer, using the original BF16 model for self-distillation over roughly 100 million tokens. The author says this stage takes about 7.5 hours. Finally, “wring” uses `Wq + BA/r` as the new target and re-solves the integer codes and scales before discarding the adapters. This differs from conventional compensation methods that retain LoRA: the additional capacity exists only during training.
One pass preserves an average of 93.7% of the original scores across IFEval, HumanEval, and GSM8K; the released two-pass model reports 94.65%. However, the 2.655 bits per weight figure counts only the 3.565 billion linear weights in the Transformer body. Once the tied embedding, normalization layers, and small SSM parameters that remain in BF16 are included, the entire language model comes to 4.69 bpw. The author has not yet rerun the evaluations with a quantized embedding. Although the compressed file is only about 1.102 GiB, the current inference path still decompresses it to BF16 before passing it to vLLM. The file size therefore cannot be interpreted directly as an improvement in VRAM use, latency, or throughput.
The results also cover only one model and three benchmarks. The primary baselines are GGUF curves constructed by the author, with no comparison yet against AQLM, QuIP#, or other low-bit methods using the same data and compute budget. The calibration prompts also overlap heavily with ten HumanEval problems. Excluding those problems does not improve the headline result, but the sample remains too small. The next meaningful milestones will be native packed-code GEMM, embedding quantization, and independent reproduction across multiple models and random seeds.