Whisper Drops Six Encoder Layers, but Average Error Rate Across Four Languages Remains Higher After Distillation

The study “Six Layers Less,” released on September 23, reduces the encoder in Whisper large-v3-turbo from 32 layers to 26, then uses knowledge distillation to restore recognition performance. This approach preserves the network’s existing computational structure, offering a compression path compatible with existing inference frameworks. However, average word error rate across the four languages tested remains higher than that of the original model. [Paper](https://arxiv.org/abs/2609.27980)

The authors first remove encoder layers one at a time, rank them by the resulting change in word error rate, and then remove the six layers with smaller effects. The public repository provides workflows for layer selection, pruning sweeps, distillation, and a control using randomly selected layers. The distillation example uses English speech and runs for 2,000 steps, training the smaller encoder to match the original model’s hidden representations using mean squared error. Thus, “unlabeled” describes the recovery training; the preceding layer-selection stage still relies on recognition evaluations with reference transcripts. [Reproduction workflow](https://github.com/rasgaard/whisper-encoder-layer-prune)

In the paper, average word error rate across Danish, English, German, and French rises from 18.2% for the original model to 21.9% after pruning alone, then falls to 20.1% after distillation—still an increase of 1.9 percentage points over the original. The smaller model therefore comes with a measurable accuracy cost: the recovery should not be interpreted as fully preserving its capabilities. Chinese was not included in this evaluation. [Results and methods](https://arxiv.org/html/2609.27980v1)

The performance figures require particular attention to the baseline. The paper tests a 60-second English audio clip on an M4 Pro. Its table reports a 1.75× end-to-end speedup for the pruned model at batch size eight, but the baseline is the full model at batch size one. In the same table, the full model at batch size eight already achieves a 1.50× speedup. Calculated from the reported values, pruning therefore provides roughly a 1.17× speedup at the same batch size of eight; at batch size one, the speedup is 1.08×. The 1.75× figure cannot be attributed entirely to removing six layers. [Benchmark table and notes](https://arxiv.org/html/2609.27980v1)

For Chinese-language speech services, the public workflow provides a starting point for replication, but it does not yet answer questions about recognition quality across dialects, proper nouns, noise, and long recordings. Another limitation is that both layer selection and the reported results use the stated FLEURS test set. Deployment evaluations should reserve a separate, independent dataset to avoid letting the selection process skew assessments of generalization. The next step should be to repeat the benchmarks with the same batch size, decoding settings, and hardware, while also accounting for manual correction costs, to determine whether the computational savings are worthwhile.

Anyone repeating layer selection for their own speech domain must also account for the annotation and evaluation costs required by that stage; counting only distillation steps underestimates the full workflow. These results offer a testable set of trade-offs that still needs support from a broader range of languages and recording conditions.