本機推論與模型量化
GSQ-RCO Allocates Bits by Tensor Sensitivity, Compressing Qwen3.8-27B into an 8.4GB Standard GGUF
ISTA DASLab has released three mixed-precision quantized versions of Qwen3.8-27B. GSQ learns the quantized values, while RCO selects the precision for each tensor under a fixed capacity budget. The smallest version is just 8.4GB and can be loaded directly by llama.cpp, Ollama, and LM Studio, although the near-lossless claim is still based on the team’s own evaluation.
ISTA’s DASLab has released GSQ-RCO GGUF versions of Qwen3.8-27B at 2.50, 2.75, and 3.00 bpw, with file sizes of 8.4GB, 9.3GB, and 10.1GB, respectively. A separate 0.9GB BF16 vision projector is also available. Unlike uniform quantization, which applies the same format to all weights, these models select different quantization types for individual tensors according to their sensitivity to task loss. The result is still exported as standard GGUF and requires no modifications to llama.cpp, Ollama, or LM Studio.
The process combines two existing research contributions. GSQ uses a Gumbel-Softmax relaxation to turn discrete quantization grids into a differentiable problem, jointly learning each coordinate’s grid assignment and the scaling value for each group. RCO reformulates the total-capacity limit as a constraint on a Riemannian manifold, then uses task loss directly to determine which quantization format should be assigned to each tensor. This allows the system to reserve more bits for sensitive weights within a strict file-size budget instead of relying on manually configured per-layer precision.
In the team’s tests, the 9.3GB version matched the BF16 baseline with a score of 100 on AIME25, while the 10.1GB version came within roughly one point of the baseline on GPQA-Diamond and LiveCodeBench v6. The smallest version’s average across five zero-shot evaluations was even slightly higher than BF16. However, occasional gains over the baseline may result from evaluation variance and should not be interpreted as evidence that compression generally improves capability. The current results also do not cover long-running agents, tool-calling reliability, vision tasks, or real-world throughput across different hardware. Engineering teams should next run regression tests against their own prompt distributions and verify whether low-bit kernels deliver genuine latency and power-efficiency gains rather than merely reducing disk and VRAM usage.