本地推論／模型量化
Qwen3.8-27B GGUF Switches to Per-Tensor Configurations, So the Same Quantization Names No Longer Represent the Old Bit Allocations
Bartowski has re-uploaded the Qwen3.8-27B GGUF files, assigning quantization formats according to each tensor’s sensitivity and publishing reproducible configuration files and KLD measurements. Some old and new files share the same names, but the meaning of `_L` and the actual bit allocations have changed, so deployers cannot compare filenames alone.
Hugging Face community quantization maintainer Bartowski has rebuilt the Qwen3.8-27B GGUF files using a new per-tensor layout. Traditional llama.cpp rules apply fixed quantization formats based on tensor names and model types. The new process instead starts with sensitivity priors derived from cross-model measurements, then allocates additional bits to weights that are more sensitive to quantization error. Each model must still pass a “canary” check: if the new configuration does not outperform the baseline on the Kullback–Leibler divergence versus file-size curve, the process falls back to the original heuristic rules.
This release includes more than the finished artifacts. Each applicable file comes with the tensor list actually passed to `llama-quantize --tensor-type-file`, along with JSON recording the generator, llama.cpp version, and commit, allowing others to at least reproduce the bit allocation. The Qwen3.8-27B model card reports KLD ratios of 0.96, 0.94, 0.79, and 0.78 for Q6_K, Q4_K_M, Q3_K_M, and IQ2_XXS, respectively, when compared with the standard configuration curve at the same file size. A ratio below 1 indicates better bit efficiency under this measurement methodology, not a proportional improvement in model task scores.
Naming compatibility is the easiest pitfall to miss. The new S, M, and L designations indicate that roughly 90%, 70%, and 50% of the model body, respectively, retain the base quantization type, while the remaining capacity is allocated according to sensitivity. As a result, `Q4_K_L` and `Q6_K_L` no longer simply preserve the embedding and output weights as Q8_0. The existing `Q2_K_L`, `Q3_K_XL`, and `Q5_K_L` files were not rebuilt but remain in the repository, so any automated download script should pin the revision, SHA256, and exact filename together.
The new quantizations still retain the vision projector and MTP layers, and speculative decoding can be enabled in recent llama.cpp builds with `--spec-type draft-mtp`. However, the public evidence primarily consists of the maintainer’s perplexity, KLD, and top-p agreement measurements on fixed text samples; it does not yet cover task-level evaluations for Chinese, code, vision, or tool calling. Engineering teams should treat these files as more transparent quantization candidates, not as drop-in replacements proven to outperform the older versions across the board.