評測與資料集
Open ASR Leaderboard Adds Hindi and Indian English, Uses Orthographic Lattice References to Correct WER Bias
Hugging Face and Voice Arena have expanded the open speech recognition leaderboard with the Monsoon dataset. Four public and private splits cover 4,888 speakers. For Hindi, reference lattices that accept multiple spellings replace a single canonical transcription, preventing conventional WER from counting correct recognitions as errors.
The Open ASR Leaderboard has added Monsoon en-IN and hi-IN, bringing Hindi and Indian English into a major open speech recognition evaluation for the first time. The data comes from dual-channel, unscripted conversations that were segmented into short, single-speaker clips. Four non-overlapping public and private splits encompass 4,888 speakers, 17 hours of audio, and hundreds of regions and phone models. The [release announcement](https://huggingface.co/blog/open-asr-leaderboard-global-south) says each sample includes 12 attributes—including age, gender, education, occupation, home region, and recording device—allowing researchers to break down error rates by demographic group instead of examining only a single average.
The most technically significant change is how Hindi is scored. Everyday Hindi contains many English loanwords, code-switching, and variations in whether compound words are joined or separated, so the same sentence may have multiple valid spellings in Devanagari. With only one reference transcription, conventional Word Error Rate (WER) rewards models for imitating the annotator’s orthographic choices. Monsoon hi-IN therefore stores a lattice of acceptable variants for each text segment and uses Orthographically-Informed WER to find a valid alignment. The team has also released the `voi-oiwer` implementation. When the lattices are flattened into a single reference, error rates rise for every model, and some models swap positions in the rankings.
Grouped results also show how aggregate scores can conceal deployment risks. Across eight models, overall WER on the public Indian English set differs by only 0.18 points, but Whisper Large V3 Turbo has a regional gap of 0.46 points, while Voxtral Mini 3B reaches 1.68 points. Indian English results are now included in the default average score. The maintainers run evaluations on the private splits, reducing opportunities to tune models specifically for public samples. The leaderboard code is [open source under Apache 2.0](https://github.com/huggingface/open_asr_leaderboard), and the public data is available for independent reruns.
One limitation is that the dataset intentionally prioritizes speaker diversity, with each person typically contributing only a few seconds of audio. It is therefore unsuitable for drawing conclusions about long-form transcription, streaming latency, or speaker continuity. The Hindi samples are also concentrated in the Hindi Belt, where real-world usage is highest. As more models are submitted, key questions will be whether lattice-based scoring remains stable, whether the private sets can resist benchmark fitting, and whether similar designs can be extended to more languages with multiple orthographic representations.