Toward Zero Oracle Word Error Rate on the Switchboard Benchmark

  • The “Switchboard benchmark” is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best amongThe “Switchboard benchmark” is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best among a list of alternatives. A phrase alternatives representation is compared to utterance-level N-best lists and word-level data structures; using dense lattices and adding out-of-vocabulary words, this achieves an oracle WER of 0.18%.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Arlo Faria, Adam Janin, Sidhi Adkoli, Korbinian RiedhammerORCiD
DOI:https://doi.org/10.21437/Interspeech.2022-10959
Parent Title (English):Interspeech 2022
Publisher:ISCA
Place of publication:ISCA
Document Type:conference proceeding (article)
Language:English
Reviewed:Begutachtet/Reviewed
Release Date:2024/08/02
Tag:ASR evaluation; N-best lists; Switchboard benchmark; oracle word error rate; phrase alternatives
Pagenumber:5
First Page:3973
Last Page:3977
Konferenzangabe:Interspeech 2022, 18-22 September 2022, Incheon, Korea
institutes:Fakultät Informatik
Zentrum für Künstliche Intelligenz (KIZ)
Research Themes:Digitalisierung & Künstliche Intelligenz
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.