TY - JOUR A1 - Agchar, Ismael A1 - Baumann, Ilja A1 - Braun, Franziska A1 - Perez-Toro, Paula Andrea A1 - Riedhammer, Korbinian A1 - Trump, Sebastian A1 - Ullrich, Martin T1 - A Survey of Music Generation in the Context of Interaction N2 - In recent years, machine learning, and in particular generative adversarial neural networks (GANs) and attention-based neural networks (transformers), have been successfully used to compose and generate music, both melodies and polyphonic pieces. Current research focuses foremost on style replication (e.g., generating a Bach-style chorale) or style transfer (e.g., classical to jazz) based on large amounts of recorded or transcribed music, which in turn also allows for fairly straight-forward “performance” evaluation. However, most of these models are not suitable for human-machine co-creation through live interaction, neither is clear, how such models and resulting creations would be evaluated. This article presents a thorough review of music representation, feature analysis, heuristic algorithms, statistical and parametric modelling, and human and automatic evaluation measures, along with a discussion of which approaches and models seem most suitable for live interaction. Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2402.15294 ER - TY - JOUR A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias ED - Baumann, Ilja T1 - Generative adversarial networks for whispered to voiced speech conversion: a comparative study JF - International Journal of Speech Technology N2 - Generative Adversarial Networks (GANs) have demonstrated promising results as end-to-end models for whispered to voiced speech conversion. Leveraging non-autoregressive systems like GANs capable of performing conditional waveform generation eliminates the need for separate models to estimate voiced speech features, and leads to faster inference compared to autoregressive methods. This study aims to identify the optimal GAN architecture for the whispered to voiced speech conversion task by comparing six state-of-the-art models. Furthermore, we present a method for evaluating the preservation of speaker identity and local accent, using embeddings obtained from speaker- and language identification systems. Our experimental results show that building the speech conversion system based on the HiFi-GAN architecture yields the best objective evaluation scores, outperforming the baseline by ∼9% relative using frequency-weighted Signal-to-Noise Ratio and Log Likelihood Ratio, as well as by ∼29% relative using Root Mean Squared Error. In subjective tests, HiFi-GAN yielded a mean opinion score of 2.9, significantly outperforming the baseline with a score of 1.4. Furthermore, HiFi-GAN enhanced ASR performance and preserved speaker identity and accent, with correct language detection rates of up to ∼98%. KW - Speech conversion · Generative adversarial networks · Whispered speech · Voiced speech Y1 - 2024 U6 - https://doi.org/10.1007/s10772-024-10161-1 VL - 27 ER - TY - JOUR A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Detecting Vocal Fatigue with Neural Embeddings JF - Journal of Voice N2 - Vocal fatigue refers to the feeling of tiredness and weakness of voice due to extended utilization. This paper investigates the effectiveness of neural embeddings for the detection of vocal fatigue. We compare x-vectors, ECAPA-TDNN, and wav2vec 2.0 embeddings on a corpus of academic spoken English. Low-dimensional mappings of the data reveal that neural embeddings capture information about the change in vocal characteristics of a speaker during prolonged voice usage. We show that vocal fatigue can be reliably predicted using all three types of neural embeddings after 40 minutes of continuous speaking when temporal smoothing and normalization are applied to the extracted embeddings. We employ support vector machines for classification and achieve accuracy scores of 81% using x-vectors, 85% using ECAPA-TDNN embeddings, and 82% using wav2vec 2.0 embeddings as input features. We obtain an accuracy score of 76%, when the trained system is applied to a different speaker and recording environment without any adaptation. KW - Vocal fatigue KW - Neural embeddings KW - Visualization KW - Detection Y1 - 2023 U6 - https://doi.org/10.1016/j.jvoice.2023.01.012 SN - 0892-1997 PB - Elsevier BV ER - TY - JOUR A1 - Escobar-Grisales, Daniel A1 - Ríos-Urrego, Cristian-David A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Garcia, Adolfo A1 - Orozco-Arroyave, Juan rafael T1 - It’s Time to Take Action: Acoustic Modeling of Motor Verbs to Detect Parkinson’s Disease N2 - Pre-trained models generate speech representations that are used in different tasks, including the automatic detection of Parkinson’s disease (PD). Although these models can yield high accuracy, their interpretation is still challenging. This paper used a pre-trained Wav2vec 2.0 model to represent speech frames of 25ms length and perform a frame-by-frame discrimination between PD patients and healthy control (HC) subjects. This fine granularity prediction enabled us to identify specific linguistic segments with high discrimination capability. Speech representations of all produced verbs were compared w.r.t. nouns and the first ones yielded higher accuracies. To gaina deeper understanding of this pattern, representations of motor and non-motor verbs were compared and the first ones yielded better results, with accuracies of around 83% in an independent test set. These findings support well-established neurocognitive models about action-related language highlighted as key drivers of PD. Index Terms: computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease KW - computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2205 ER - TY - JOUR A1 - Ranzenberger, Thomas A1 - Baumann, Ilja A1 - Bayerl, Sebastian A1 - Wagner, Dominik A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Evaluation of recognition errors of hybrid and transformer-based ASR systems in German video lectures JF - Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2025 - Book N2 - We analyze different errors in speech recognition systems, focusing on consecutive insertions and deletions, known as hallucinations and elisions in transformer-based end-to-end automatic speech recognition (ASR) systems. We compare errors from a TDNN-HMM, and whisper-based models on English and German spontaneous speech. Based on a human annotated subset of German lecture videos, we investigate whether these blocks of deletions affect the semantics of the utterance. Whisper performs best and preserves the meaning in 90% of the annotated error segments even containing consecutive deletions on this subset. We analyze the word error rate and do further analysis of errors using natural language processing to detect lemmatization errors, compound word errors, and out-of-vocabulary words. We discuss possible reasons and mitigations. Y1 - 2025 UR - https://www.essv.de/pdf/2025_101_108.pdf PB - ESSV 2025 CY - Halle, Deutschland ER -