TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias ED - Pesak, Krisztina T1 - Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models N2 - This paper explores the improvement of post-training quantization (PTQ) after knowledge distillation in the Whisper speech foundation model family. We address the challenge of outliers in weights and activation tensors, known to impede quantization quality in transformer-based language and vision models. Extending this observation to Whisper, we demonstrate that these outliers are also present when transformer-based models are trained to perform automatic speech recognition, necessitating mitigation strategies for PTQ. We show that outliers can be reduced by a recently proposed gating mechanism in the attention blocks of the student model, enabling effective 8-bit quantization, and lower word error rates compared to student models without the gating mechanism in place. KW - post-training quantization, Whisper, gated attention, outliers Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2105 SP - 4623 EP - 4627 ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Wagner, Dominik A1 - Noeth, Elmar A1 - Riedhammer, Korbinian T1 - Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0 T2 - Interspeech 2022 N2 - Stuttering is a varied speech disorder that harms an individual’s communication ability. Persons who stutter (PWS) often use speech therapy to cope with their condition. Improving speech recognition systems for people with such non-typical speech or tracking the effectiveness of speech therapy would require systems that can detect dysfluencies while at the same time being able to detect speech techniques acquired in therapy. This paper shows that fine-tuning wav2vec 2.0 [1] for the classification of stuttering on a sizeable English corpus containing stuttered speech, in conjunction with multi-task learning, boosts the effectiveness of the general-purpose wav2vec 2.0 features for detecting stuttering in speech; both within and across languages. We evaluate our method on FluencyBank , [2] and the German therapy-centric Kassel State of Fluency (KSoF) [3] dataset by training Support Vector Machine classifiers using features extracted from the finetuned models for six different stuttering-related event types: blocks, prolongations, sound repetitions, word repetitions, interjections, and – specific to therapy – speech modifications. Using embeddings from the fine-tuned models leads to relative classification performance gains up to 27% w.r.t. F1-score. KW - stuttering KW - pathological speech KW - computational paralinguistics Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10908 SP - 2868 EP - 2872 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Wagner, Dominik A1 - Churchill, Alexander A1 - Sigtia, Siddharth A1 - Marchi, Erik T1 - SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline. KW - multi-task KW - multimodal KW - virtual assistant KW - large language KW - model KW - low-rank adaptation Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10890139 SN - 2379-190X SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Lee, Seanie A1 - Seong, Haebin A1 - Bok Lee, Dong A1 - Kang, Minki A1 - Chen, Xiaoyin A1 - Wagner, Dominik A1 - Bengio, Joshua A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models N2 - Safety guard models that detect malicious queries aimed at large language models(LLMs) are essential for ensuring the secure and responsible deployment of LLMs in real-world applications. However, deploying existing safety guard models with billions of parameters alongside LLMs on mobile devices is impractical due to substantial memory requirements and latency. To reduce this cost, we distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. Due to the limited diversity of harmful instructions in the existing labeled dataset, naively distilled models tend to underperform compared to larger models. To bridge the gap between small and large models, we propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Given a prompt such as, “Make a single harmful instruction prompt that would elicit offensive content”, we add an affirmative prefix (e.g., “I have an idea for a prompt:”) to the LLM’s response. This encourages the LLM to continue generating the rest of the response, leading to sampling harmful instructions. Another LLM generates a response to the harmful instruction, and the teacher model labels the instruction-response pair. We empirically show that our HarmAug outperforms other relevant baselines. Moreover, a 435-millionparameter safety guard model trained with HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost. Our code, safety guard model, and synthetic dataset are publicly available. Y1 - 2025 U6 - https://doi.org/10.48550/arXiv.2410.01524 SP - 1 EP - 20 PB - ICLR ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Self-supervised Training with BEST-RQ for Speech Recognition T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multicodebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on testother using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training. KW - self-supervised learning KW - speech recognition KW - conformer KW - BEST-RQ Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889362 SP - 1 EP - 5 PB - IEEE ER - TY - JOUR A1 - Lee, Seanie A1 - Bok Lee, Dong A1 - Wagner, Dominik A1 - Kang, Minki A1 - Seong, Haebin A1 - Bocklet, Tobias A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models, JF - Findings of the Association for Computational Linguistics: ACL 2025 N2 - Deploying large language models (LLMs) in real-world applications requires robust safety guard models to detect and block harmful user prompts. While large safety guard models achieve strong performance, their computational cost is substantial. To mitigate this, smaller distilled models are used, but they often underperform on “hard” examples where the larger model provides accurate predictions. We observe that many inputs can be reliablyhandled by the smaller model, while only a small fraction require the larger model’s capacity. Motivated by this, we propose SafeRoute, a binary router that distinguishes hard examples from easy ones. Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy compared to solely using the larger safety guard model. Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance, outperforming relevant baselines. Y1 - 2025 UR - https://aclanthology.org/2025.findings-acl.105.pdf PB - Association for Computational Linguistics CY - Vienna, Austria ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Engert, Natalie A1 - Lee, Seanie A1 - Nöth, Elmar A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ∼23% and ∼22%, respectively. Further improvements (∼5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ∼7% relative WER improvement over personalized fine-tuning alone. KW - speech recognition KW - dysarthric speech KW - personalization KW - low-rank adaptation KW - synthetic data Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2155 SN - 2958-1796 SP - 3294 EP - 3298 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Pathology-Aware Speech Encoding and Data Augmentation for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - Automatic speech recognition (ASR) for pathologic speech remains a major challenge due to high variability in articulation, phonation, and prosody distortions. In this work, we propose a pathology-aware speech encoder based on BEST-RQ pre-training, which incorporates 46k hours of speech, including pathologic and atypical speech. We continue pre-training for domain adaptation and experiment with etiology-specific codebooks. We achieve a 13.2% relative word error rate (WER) improvement using the pathology-aware speech encoder with etiology-specific continued pre-training. Additionally, we examine the impact of incorporating synthetic and out-of-domain (OOD) data to further enhance ASR performance. Synthetic data reduces WER by up to 8.7%, while OOD data improves WER by 12.2%. Finally, we introduce a semantic similaritybased data augmentation technique to optimize data selection, achieving a WER improvement of up to 9.7% while minimizing the need for additional training data. KW - pathologic speech recognition KW - speech augmentation KW - self-supervised learning Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2724 SN - 2958-1796 SP - 3289 EP - 3293 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias T1 - Vocoder-Free Non-parallel Conversion of Whispered Speech With Masked Cycle-Consistent Generative Adversarial Networks N2 - Cycle-consistent generative adversarial networks have been widely used in non-parallel voice conversion (VC). Their ability to learn mappings between source and target features without relying on parallel training data eliminates the need for temporal alignments. However, most methods decouple the conversion of acoustic features from synthesizing the audio signal by using separate models for conversion and waveform synthesis. This work unifies conversion and synthesis into a single model, thereby eliminating the need for a separate vocoder. By leveraging cycle-consistent training and a self-supervised auxiliary training task, our model is able to efficiently generate converted high-quality raw audio waveforms. Subjective listening tests showed that our unified approach achieved improvements of up to 6.7% relative to the baseline in whispered VC. Mean opinion score predictions also yielded stable results in conventional VC (between 0.5% and 2.4% relative improvement). KW - voice conversion KW - generative adversarial networks KW - cycle-consistency KW - masking KW - whispered speech Y1 - 2025 SN - 978-3-032-02548-7 U6 - https://doi.org/10.1007/978-3-032-02548-7_20 PB - Springer CY - Cham ER - TY - JOUR A1 - Ranzenberger, Thomas A1 - Baumann, Ilja A1 - Bayerl, Sebastian A1 - Wagner, Dominik A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Evaluation of recognition errors of hybrid and transformer-based ASR systems in German video lectures JF - Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2025 - Book N2 - We analyze different errors in speech recognition systems, focusing on consecutive insertions and deletions, known as hallucinations and elisions in transformer-based end-to-end automatic speech recognition (ASR) systems. We compare errors from a TDNN-HMM, and whisper-based models on English and German spontaneous speech. Based on a human annotated subset of German lecture videos, we investigate whether these blocks of deletions affect the semantics of the utterance. Whisper performs best and preserves the meaning in 90% of the annotated error segments even containing consecutive deletions on this subset. We analyze the word error rate and do further analysis of errors using natural language processing to detect lemmatization errors, compound word errors, and out-of-vocabulary words. We discuss possible reasons and mitigations. Y1 - 2025 UR - https://www.essv.de/pdf/2025_101_108.pdf PB - ESSV 2025 CY - Halle, Deutschland ER -