Toward Zero Oracle Word Error Rate on the Switchboard Benchmark
- The “Switchboard benchmark” is a very well-known test set
in automatic speech recognition (ASR) research, establishing
record-setting performance for systems that claim human-level
transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring
methodology. In this more detailed and reproducible scheme,
even commercial ASR systems can score below 5% WER and
the established record for a research system is lowered to 2.3%.
An alternative metric of transcript precision is proposed, which
does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial
ASR systems are still below this threshold, a research system
is shown to clearly surpass the accuracy of commercial human
speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best
amongThe “Switchboard benchmark” is a very well-known test set
in automatic speech recognition (ASR) research, establishing
record-setting performance for systems that claim human-level
transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring
methodology. In this more detailed and reproducible scheme,
even commercial ASR systems can score below 5% WER and
the established record for a research system is lowered to 2.3%.
An alternative metric of transcript precision is proposed, which
does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial
ASR systems are still below this threshold, a research system
is shown to clearly surpass the accuracy of commercial human
speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best
among a list of alternatives. A phrase alternatives representation
is compared to utterance-level N-best lists and word-level data
structures; using dense lattices and adding out-of-vocabulary
words, this achieves an oracle WER of 0.18%.…


| Author: | Arlo Faria, Adam Janin, Sidhi Adkoli, Korbinian RiedhammerORCiD |
|---|---|
| DOI: | https://doi.org/10.21437/Interspeech.2022-10959 |
| Parent Title (English): | Interspeech 2022 |
| Publisher: | ISCA |
| Place of publication: | ISCA |
| Document Type: | conference proceeding (article) |
| Language: | English |
| Reviewed: | Begutachtet/Reviewed |
| Release Date: | 2024/08/02 |
| Tag: | ASR evaluation; N-best lists; Switchboard benchmark; oracle word error rate; phrase alternatives |
| Pagenumber: | 5 |
| First Page: | 3973 |
| Last Page: | 3977 |
| Konferenzangabe: | Interspeech 2022, 18-22 September 2022, Incheon, Korea |
| institutes: | Fakultät Informatik |
| Zentrum für Künstliche Intelligenz (KIZ) | |
| Research Themes: | Digitalisierung & Künstliche Intelligenz |
