模型評測
HIVE Finds Speech Transcription Harms LLM Reasoning More Than Keyboard Typos—and a Larger Thinking Budget Cannot Make Up the Difference
A new study tests instruction-tuned models using speech transcription and QWERTY keyboard perturbations, finding that structural rewrites introduced by speech consistently reduce answer performance. Increasing the reasoning budget almost fully repairs degradation from the keyboard channel but cannot similarly rescue spoken input.
Voice agents commonly send automatic speech recognition output directly to language models, yet system evaluations are generally conducted using clean text prompts. A study released on August 4 introduces the Human Input-Variation Engine (HIVE), which separately simulates QWERTY keyboard errors, disfluencies found in ordinary speech transcription, and the compression and restructuring of sentences by AI dictation tools to measure how input channels affect the performance of instruction-tuned models.
The authors report that speech-transcription perturbations reduce accuracy across all instruction-tuned models tested. The main issue is not the addition of filler words such as “um,” but the restructuring of the original sentence during transcription. Keyboard typos have a smaller effect: models can tolerate substantial surface noise until critical tokens are corrupted. The study identifies a shared predictor of failure across both channels: how many tokens from the original question remain intact after perturbation. Inserting extra tokens generally carries a limited cost, whereas deleting or corrupting original tokens significantly harms results.
The gap between channels appears mainly in tasks that require models to construct or derive answers themselves; when the answer is already present among multiple-choice options, the difference is not significant. The authors also test higher thinking budgets, finding that they almost completely recover the losses caused by keyboard perturbations but do not repair spoken-language transcription. Compressed speech input may even perform worse with additional reasoning. Lightweight adaptation training likewise fails to eliminate the problem, meaning it cannot simply be assumed that adding a small amount of noisy training data will solve it.
For voice-agent engineers, ASR word error rate is insufficient to represent downstream risk. At the same error rate, deleting a negation, condition, or entity can have a far greater impact than adding redundant words. More practical safeguards include retaining both the raw transcript and its normalized version, flagging low-confidence segments, and asking users to confirm critical parameters before executing tools. This remains a single preprint, and its abstract does not provide cross-lingual results. Chinese word segmentation, homophones, and omitted subjects may produce different failure modes, warranting further validation using the public code and complete dataset.