Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition

  • In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ∼23% and ∼22%, respectively. Further improvements (∼5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ∼7% relative WER improvement over personalized fine-tuning alone.

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Dominik Wagner, Ilja Baumann, Natalie Engert, Seanie Lee, Elmar Nöth, Korbinian Riedhammer, Tobias BockletORCiD
DOI:https://doi.org/10.21437/Interspeech.2025-2155
ISSN:2958-1796
ArXiv Id:http://arxiv.org/abs/2505.12991v1
Parent Title (English):Interspeech 2025
Publisher:ISCA
Place of publication:ISCA
Document Type:conference proceeding (article)
Language:English
Date of first Publication:2025/05/19
Reviewed:Begutachtet/Reviewed
Release Date:2025/11/04
Tag:dysarthric speech; low-rank adaptation; personalization; speech recognition; synthetic data
Pagenumber:5
First Page:3294
Last Page:3298
institutes:Fakultät Informatik
Zentrum für Künstliche Intelligenz (KIZ)
Research Themes:Digitalisierung & Künstliche Intelligenz
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.