ASR Benchmark Fitting Quantified with Three Probe Types; Some Models Reproduce Erroneous Reference Transcripts 18% to 30% of the Time

A new study covering 11 open-source automatic speech recognition models reframes the vague problem of “benchmark contamination” as an actionable behavioral test. The researchers designed three types of probes. First, they identified VoxPopuli segments in which the audio and reference transcript contradicted each other. Next, they silenced numbers in LibriSpeech or VoxPopuli recordings to test whether models would still restore the original answers. Finally, they used homophonic spelling variants such as “Mr.” and “Mister” to examine whether models could use dataset-specific acoustic cues to switch to the spelling preferred by each test set.

The results show that some models with lower WER on public leaderboards were actually more likely to copy erroneous reference transcripts. Within the inspected sample, the research team labeled about 40% of VoxPopuli test segments as potentially containing reference errors, affecting roughly 3% of reference words. Models exhibiting benchmark-conditioned behavior selected the erroneous reference transcript 18% to 30% of the time. When target numbers were completely silenced, several models still achieved exact restoration rates approaching 40% on public datasets. Six of the 11 models also performed significantly above the 0.5 random-switching baseline in the honorific spelling test.

Mechanistic experiments further indicate that the trigger signals are not confined to the text decoder. When the same sentences were retained but rendered using newly recorded parliamentary voices or generic synthetic voices, the tendency to copy the answers generally declined. Appending eight seconds of ordinary conversation to the original segment also caused multiple models to return to outputs that more faithfully reflected the audio. The authors were additionally able to alter this behavior through low-rank linear steering, suggesting that models may infer “which benchmark this is” from narrow cues involving timbre, recording environment, or data source.

Hugging Face has added reference-transcript divergence and orthographic-switching metrics to the Open ASR Leaderboard and released the evaluation code and unnormalized outputs. When selecting models, engineering teams should evaluate private test sets built from newly recorded audio, temporal splits, or speaker-disjoint partitions alongside WER. An important limitation is that the study covers only two major English-language datasets and 11 models. Answer-like behavior could arise from training data, fine-tuning, model selection, or shortcuts based on the data distribution; the scores alone cannot establish that developers directly included the test sets in training.