Speech transcription benchmarks often score a model by comparing its words with a published reference transcript. Hume researchers documented how that score can reward an error when the reference transcript itself is wrong. Their paper was submitted to arXiv on August 20, followed by a Hugging Face write-up on August 21.
The team evaluated 11 open speech-recognition models using recordings from VoxPopuli and LibriSpeech. In one European Parliament clip, the speaker audibly said a brief thank-you that the reference omitted. Six models repeated that omission. When researchers recreated the sentence in newer voices, most of those systems included the missing words.
The researchers also silenced numbers inside recordings. Some strong benchmark performers still supplied the missing values, with recovery rates around 30% to 40% in selected LibriSpeech examples. Another test found models switching between equivalent spellings, such as an abbreviated title and its written-out form, to match the convention of the dataset.
These experiments show benchmark-specific behavior in the evaluated models and recordings. They do not prove that every transcription model memorized its training audio. Hume's team added a benchmark-fitting analysis to the Open ASR Leaderboard so model comparisons can examine more than word error rate.