TY - JOUR A1 - Freier, Carolin A1 - Bocklet, Tobias A1 - Helten, Anne-Kathrin A1 - Hoffmann, Franziska A1 - Hunger, Marianne A1 - Kovács, László A1 - Richter, Florian A1 - Riedhammer, Korbinian A1 - Schmohl, Tobias A1 - Simon, Claudia T1 - Wie kann videogestütztes Lernen die Erwartungen Studierender und Dozierender erfüllen? JF - Soziale Passagen N2 - ZusammenfassungIm BMBF-Verbundprojekt HAnS entwickeln und implementieren neun Hochschulen sowie drei hochschulübergreifende Einrichtungen ein intelligentes Hochschul-Assistenz-System als Open-Source-Lösung. Videobasierte Lehrmaterialien werden verschriftlicht und durch eine Indexierung Stichwortsuchen ermöglicht; geplant ist, über einen KI-Tutor automatisiert Übungsaufgaben zu generieren. Studierende sollen so in ihrem Selbststudium digital unterstützt werden. Die technische Entwicklung wird interdisziplinär – auch sozialwissenschaftlich und pädagogisch – begleitet und in einem iterativen Vorgehen evidenzbasiert entsprechend Design-Based-Research angepasst. Wissen und Wertesystem der Anwender*innen, Didaktik, Ethik, Akzeptanz und Datenschutz werden dabei im Entwicklungsprozess einbezogen. KW - Hochschulforschung · Intelligente Assistenzsysteme · Evaluation · Video-Lernen · Adressat*innen-zentrierte Technikentwicklung Y1 - 2023 U6 - https://doi.org/10.1007/s12592-023-00478-0 SN - 1867-0180 VL - 15 IS - 2 SP - 631 EP - 635 PB - Springer VS ER - TY - JOUR A1 - Schuster, Maria A1 - Arias-Vergara, Tomas A1 - Müller-Hörner, Rainer A1 - Winterholler, Cordula A1 - Bocklet, Tobias T1 - Verstehen mich mit der Maske noch alle? T1 - Speech quality changes due to face masks BT - Coronavirus-Pandemie JF - MMW - Fortschritte der Medizin N2 - In Arztpraxen nimmt der Infektionsschutz mittels Mund-Nasen-Maske momentan eine zentrale Rolle ein. Wenig nachgedacht wird dabei über die Folgen für die Kommunikation. Gerade schwerhörigen Patienten bereiten gedämpfte Stimmen und abgedeckte Lippen Probleme. KW - Face mask, speech quality, computer-assisted speech assessment, hearing loss Y1 - 2020 U6 - https://doi.org/10.1007/s15006-020-0749-4 VL - 162 IS - 14 SP - 42 EP - 44 ER - TY - JOUR A1 - Bocklet, Tobias A1 - Nöth, Elmar A1 - Riedhammer, Korbinian T1 - Künstliche Intelligenz für die Analyse pathologischer Sprache JF - Sprache - Stimme - Gehör N2 - Sprache kann eine Vielzahl von diagnostisch relevanten Informationen enthalten. In diesem Übersichtsartikel wird aufgezeigt, wie Methoden der Künstlichen Intelligenz, insbesondere Maschinelles Lernen und Sprachverarbeitung, angewendet auf Sprachsignale eingesetzt werden können: zur Bewertung von Verständlichkeit, zur Automatisierung von standardisierten Tests und zur Bestimmung medizinischer Skalen und Diagnosen. Eine abschließende kritischen Betrachtung von akustischen Merkmalen über eine Vielzahl von Pathologien gibt Grund zur Annahme, dass diese Marker tatsächlich diagnostisch relevante Informationen enthalten. N2 - Speech can contain a variety of diagnostically relevant cues. In this article, it is shown how artificial intelligence, in particular machine learning and speech processing, can be applied to speech signals: to assess intelligibility, to automate standardized tests, and to determine medical scales and diagnoses. We conclude with critical review of acoustic features across a variety of pathologies that give reason to believe that these markers do indeed contain diagnostically relevant information. T2 - Artificial Intelligence for the Analysis of Pathologic Speech KW - Künstliche Intelligenz - Machine Learning - pathologische Sprache - Spracherkennung - Sprachverarbeitung Y1 - 2023 U6 - https://doi.org/10.1055/a-2089-5778 VL - 47 IS - 3 ER - TY - CHAP A1 - Braun, Franziska A1 - Bayerl, Sebastian P. A1 - Pérez-Toro, Paula A. A1 - Hönig, Florian A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Classifying Dementia in the Presence of Depression BT - A Cross-Corpus Study N2 - Automated dementia screening enables early detection and intervention, reducing costs to healthcare systems and increasing quality of life for those affected. Depression has shared symptoms with dementia, adding complexity to diagnoses. The research focus so far has been on binary classification of dementia (DEM) and healthy controls (HC) using speech from picture description tests from a single dataset. In this work, we apply established baseline systems to discriminate cognitive impairment in speech from the semantic Verbal Fluency Test and the Boston Naming Test using text, audio and emotion embeddings in a 3-class classification problem (HC vs. MCI vs. DEM). We perform cross-corpus and mixed-corpus experiments on two independently recorded German datasets to investigate generalization to larger populations and different recording conditions. In a detailed error analysis, we look at depression as a secondary diagnosis to understand what our classifiers actually learn. Y1 - 2023 U6 - https://doi.org/10.48550/arXiv.2308.08306 ER - TY - CHAP A1 - Riedhammer, Korbinian A1 - Baumann, Ilja A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias A1 - Braun, Franziska A1 - Wagner’, Dominik T1 - Medical Speech Processing for Diagnosis and Monitoring BT - Clinical Use Cases N2 - In recent years, speech processing for medical applications got significant traction. While pioneering work in the 1990ies focused on processing sustained vowels or isolated utterances, work in the 2000s already showed, that speech recognition systems, prosodic analysis and natural language processing be used to assess a large variety of speech pathologies.Here, we give an overview of how to classify selected speech pathologies including stuttering, language development, speech intelligibility after surgery, dementia and Alzheimers, depression and state-of-mind. While each of those poses a rather well-defined problem in a lab setting, we discuss the issues when integrating such methods in a clinical workflow such as diagnosis or monitoring. Starting from the question if such detectors can be used for general screening or rather as a specialist's tool, we explore the legal and privacy-related implications: patient-doctor conversations, working with children or demented seniors, bias towards examiner or patient, on-device vs. cloud processing.We conclude with a set of open questions that should be addressed to help bringing all this research from the lab to routine clinical use. Y1 - 2023 SP - 1417 EP - 1420 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Braun, Franziska A1 - Bayerl, Sebastian P. A1 - Nöth, Elmar A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Multi-class Detection of Pathological Speech with Latent Features BT - How does it perform on unseen data? N2 - The detection of pathologies from speech features is usually defined as a binary classification task with one class representing a specific pathology and the other class representing healthy speech. In this work, we train neural networks, large margin classifiers, and tree boosting machines to distinguish between four pathologies: Parkinson's disease, laryngeal cancer, cleft lip and palate, and oral squamous cell carcinoma. We show that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be effectively used to classify these types of pathological voices. We evaluate the robustness of our classifiers by adding room impulse responses to the test data and by applying them to unseen speech corpora. Our approach achieves unweighted average F1-Scores between 74.1% and 97.0%, depending on the model and the noise conditions used. The systems generalize and perform well on unseen data of healthy speakers sampled from a variety of different sources. Y1 - 2023 U6 - https://doi.org/10.21437/Interspeech.2023-464 SN - 2958-1796 SP - 2318 EP - 2322 ER - TY - CHAP A1 - Chen, Wenda A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Length-and Noise-aware Training Techniques for Short-utterance Speaker Recognition N2 - Speaker recognition performance has been greatly improved with the emergence of deep learning. Deep neural networks show the capacity to effectively deal with impacts of noise and reverberation, making them attractive to far-field speaker recognition systems. The x-vector framework is a popular choice for generating speaker embeddings in recent literature due to its robust training mechanism and excellent performance in various test sets. In this paper, we start with early work on including invariant representation learning (IRL) to the loss function and modify the approach with centroid alignment (CA) and length variability cost (LVC) techniques to further improve robustness in noisy, far-field applications. This work mainly focuses on improvements for short-duration test utterances (1-8s). We also present improved results on long-duration tasks. In addition, this work discusses a novel self-attention mechanism. On the VOiCES far-field corpus, the combination of the proposed techniques achieves relative improvements of 7.0% for extremely short and 8.2% for full-duration test utterances on equal error rate (EER) over our baseline system. KW - speaker recognition, invariant representation learning, centroid alignment, x-vector, far-field Y1 - 2020 U6 - https://doi.org/10.48550/arXiv.2008.12218 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bayerl, Sebastian P. A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Speaker Adaptation for End-To-End Speech Recognition Systems in Noisy Environments N2 - We analyze the impact of speaker adaptation in end-to-end automatic speech recognition models based on transformers and wav2vec 2.0 under different noise conditions. By including speaker embeddings obtained from x-vector and ECAPA-TDNN systems, as well as i-vectors, we achieve relative word error rate improvements of up to 16.3% on LibriSpeech and up to 14.5% on Switchboard. We show that the proven method of concatenating speaker vectors to the acoustic features and supplying them as auxiliary model inputs remains a viable option to increase the robustness of end-to-end architectures. The effect on transformer models is stronger, when more noise is added to the input speech. The most substantial benefits for systems based on wav2vec 2.0 are achieved under moderate or no noise conditions. Both x-vectors and ECAPA-TDNN embeddings outperform i-vectors as speaker representations. The optimal embedding size depends on the dataset and also varies with the noise condition. KW - speaker adaptation, automatic speech recognition, end-to-end systems, transformer, wav2vec 2.0 Y1 - 2023 U6 - https://doi.org/10.48550/arXiv.2211.08774 ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias A1 - Draxler, Christoph T1 - Implementing Easy-to-Use Recipes for the Switchboard Benchmark N2 - We report on our contribution of templates for tokenization, language modeling, and automatic speech recognition (ASR) on the Switchboard benchmark to the open-source general-purpose toolkit SpeechBrain. Three recipes for the training of end-to-end ASR systems were implemented. We describe their model architectures, as well as the necessary data preparation steps. The word error rates achievable with our models are comparable to or better than those of other popular toolkits. Pre-trained ASR models were made available on HuggingFace. They can be easily integrated into research projects or used directly for quick inference via a hosted inference API. Y1 - 2023 SP - 150 EP - 157 PB - TUDpress, Dresden ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Detection of Vowel Errors in Children’s Speech Using Synthetic Phonetic Transcripts T2 - 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Taipei, Taiwan, 2023 N2 - The analysis of phonological processes is crucial in evaluating speech development disorders in children, but encounters challenges due to limited children audio data. This work focuses on automatic vowel error detection using a two-stage pipeline. The first stage uses a fine-tuned cross-lingual phone recognizer (wav2vec 2.0) to extract phone sequences from audio. The second stage employs a language model (BERT) for classification from a phone sequence, entirely trained on synthetic transcripts, to counteract the very broad range of potential mistakes. We evaluate the system on nonword audio recordings recited by preschool children from a speech development test. The results show that the classifier trained on synthetic data performs well, but its efficacy relies on the quality of the phone recognizer. The best classifier achieves an 94.7% F1 score when evaluated against phonetic ground truths, whereas the F1 score is 76.2% when using automatically recognized phone sequences. KW - Conferences ; Pipelines ; Phonetics ; Audio recording ; Speech processing ; Synthetic data ; Automatic speech recognition ; children’s speech ; vowel errors ; nonwords Y1 - 2023 U6 - https://doi.org/10.1109/ASRU57964.2023.10389704 SP - 1 EP - 8 ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias T1 - Nonwords Pronunciation Classification in Language Development Tests for Preschool Children T2 - Interspeech 2022 : 18-22 September 2022, Incheon, Korea N2 - This work aims to automatically evaluate whether the language development of children is age-appropriate. Validated speech and language tests are used for this purpose to test the auditory memory. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures: Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR acoustic model). Each of the approaches provides input for VGG-like 5-layer CNN classifiers. We also examine the adaptation per non-word. The evaluation of the proposed systems was performed using recordings from different kindergartens of spoken non- words. ECAPA-TDNN and low-level FFT features do not explicitly model phonetic information; wav2vec2.0 is trained on grapheme labels, our ASR acoustic model features contain (sub-)phonetic information. We found that the more granular the phonetic modeling is, the higher are the achieved recognition rates. The best system trained on ASR acoustic model features with VTLN achieved an accuracy of 89.4% and an area under the ROC (Receiver Operating Characteristic) curve (AUC) of 0.923. This corresponds to an improvement in accuracy of 20.2% and AUC of 0.309 relative compared to the FFT-baseline KW - children’s speech, speech assessment, transfer learning Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10777 SP - 3643 EP - 3647 ER - TY - CHAP A1 - Georges, Munir A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Compact Speaker Embedding BT - lrx-vector N2 - Deep neural networks (DNN) have recently been widely used in speaker recognition systems, achieving state-of-the-art performance on various benchmarks. The x-vector architecture is especially popular in this research community, due to its excellent performance and manageable computational complexity. In this paper, we present the lrx-vector system, which is the low-rank factorized version of the x-vector embedding network. The primary objective of this topology is to further reduce the memory requirement of the speaker recognition system. We discuss the deployment of knowledge distillation for training the lrx-vector system and compare against low-rank factorization with SVD. On the VOiCES 2019 far-field corpus we were able to reduce the weights by 28% compared to the full-rank x-vector system while keeping the recognition rate constant (1.83% EER). KW - speaker recognition, x-vector, low power Y1 - 2020 U6 - https://doi.org/10.48550/arXiv.2008.05011 ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian A1 - Sojka, Petr A1 - Kopeček, Ivan A1 - Pala, Karel A1 - Horák, Aleš T1 - The Influence of Dataset Partitioning on Dysfluency Detection Systems N2 - This paper empirically investigates the influence of different data splits and splitting strategies on the performance of dysfluency detection systems. For this, we perform experiments using wav2vec 2.0 models with a classification head as well as support vector machines (SVM) in conjunction with the features extracted from the wav2vec 2.0 model to detect dysfluencies. We train and evaluate the systems with different non-speaker-exclusive and speaker-exclusive splits of the Stuttering Events in Podcasts (SEP-28k) dataset to shed some light on the variability of results w.r.t. to the partition method used. Furthermore, we show that the SEP-28k dataset is dominated by only a few speakers, making it difficult to evaluate. To remedy this problem, we created SEP-28k-Extended (SEP-28k-E), containing semi-automatically generated speaker and gender information for the SEP-28k corpus, and suggest different data splits, each useful for evaluating other aspects of methods for dysfluency detection. KW - stuttering · dysfluencies · pathological speech · SEP-28k Y1 - 2022 U6 - https://doi.org/10.48550/arXiv.2206.03400 PB - Springer International Publishing ER - TY - CHAP A1 - Lopatka, Kuba A1 - Bocklet, Tobias T1 - State Sequence Pooling Training of Acoustic Models for Keyword Spotting T2 - Proceedings Interspeech 2020 N2 - We propose a new training method to improve HMM-based keyword spotting. The loss function is based on a score computed with the keyword/filler model from the entire input sequence. It is equivalent to max/attention pooling but is based on prior acoustic knowledge. We also employ a multi-task learning setup by predicting both LVCSR and keyword posteriors. We compare our model to a baseline trained on frame-wise cross entropy, with and without per-class weighting. We employ a low-footprint TDNN for acoustic modeling. The proposed training yields significant and consistent improvement over the baseline in adverse noise conditions. The FRR on cafeteria noise is reduced from 13.07% to 5.28% at 9 dB SNR and from 37.44% to 6.78% at 5 dB SNR. We obtain these results with only 600 unique training keyword samples. The training method is independent of the frontend and acoustic model topology. KW - keyword spotting, machine learning, speech recognition Y1 - 2020 U6 - https://doi.org/10.21437/Interspeech.2020-2722 SN - 2958-1796 SP - 4338 EP - 4342 ER - TY - CHAP A1 - Simic, Christopher A1 - Bocklet, Tobias T1 - Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models T2 - ICASSP 2024 - 2024 IEEE International Y1 - 2024 U6 - https://doi.org/10.1109/ICASSP48485.2024.10448047 SP - 12787 EP - 12791 ER - TY - CHAP A1 - Seeberger, Philipp A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Information Type Classification with Contrastive Task-Specialized Sentence Encoders N2 - User-generated information content has become an important information source in crisis situations. However, classification models suffer from noise and event-related biases which still poses a challenging task and requires sophisticated task-adaptation. To address these challenges, we propose the use of contrastive task-specialized sentence encoders for downstream classification. We apply the task-specialization on the CrisisLex, HumAID, and TrecIS information type classification tasks and show performance gains w.r.t. F1-score. Furthermore, we analyse the cross-corpus and cross-lingual capabilities for two German event relevancy classification datasets. Y1 - 2023 U6 - https://doi.org/10.48550/arXiv.2312.11020 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Schuster, Maria A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Towards Interpretability of Automatic Phoneme Analysis in Cleft Lip and Palate Speech T2 - IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) N2 - Cleft Lip and Palate ranks among the most common congenital abnormalities and significantly influences speech articulation, resulting in varying phonemic impacts. In a clinical context, a detailed diagnosis is carried out by time-consuming perceptual evaluations. We use perceptual ratings of different articulatory modifications on phoneme-level as ground-truth and propose a system based on wav2vec 2.0, trained to the downstream task of classifying phonemic criteria as a multi-class and multi-label problem. The system is trained for detection on utterance level, without the usage of phoneme labels. To gain a clearer understanding of which areas of the speech signal have the greatest impact on classification, we assess the extent to which our system aligns with expert ratings at the phoneme level. Additionally, we examine which specific phonemes play a decisive role in determining the final classification of the labeled criteria. The results show that salient phonemes marked by experts contribute remarkably greater to the classification of the correct class using feature relevance explanation methods. To the best of our knowledge, this is the first study incorporating various utterance-level articulatory modifications classification and phoneme-level interpretation, offering a more comprehensive understanding for potential clinical applications. KW - pathologic speech, cleft lip and palate, children’s speech, automatic assessment Y1 - 2024 U6 - https://doi.org/10.1109/ICASSP48485.2024.10447632 SP - 12602 EP - 12606 ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI Workshops 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education KW - Learning Experience Platform KW - Open Source Software KW - Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 PB - Gesellschaft für Informatik e.V. ER - TY - CHAP A1 - Braun, Franziska A1 - Bayerl, Sebastian A1 - Hönig, Florian A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Infusing Acoustic Pause Context into Text-Based Dementia Assessment N2 - Speech pauses, alongside content and structure, offer a valuable and non-invasive biomarker for detecting dementia. This work investigates the use of pause-enriched transcripts in transformer-based language models to differentiate the cognitive states of subjects with no cognitive impairment, mild cognitive impairment, and Alzheimer’s dementia based on their speech from a clinical assessment. We address three binary classification tasks: Onset, monitoring, and dementia exclusion. The performance is evaluated through experiments on a German Verbal Fluency Test and a Picture Description Test, comparing the model’s effectiveness across different speech production contexts. Starting from a textual baseline, we investigate the effect of incorporation of pause information and acoustic context. We show the test should be chosen depending on the task, and similarly, lexical pause information and acoustic cross-attention contribute differently. KW - speech biomarkers KW - dementia assessment KW - neuropsychological tests KW - pathological speech Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2496 SN - 2958-1796 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Ranzenberger, Thomas A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalizing Large Sequence-to-Sequence Speech Foundation Models With Speaker Representations N2 - We present a method to personalize large transformer-based encoderdecoder speech foundation models without the need for changes in the underlying model structure or training from scratch. This is achieved by projecting speaker-specific information into the latent space of the transformer decoder via a small neural network and learning to process the speaker information along with domainspecific information via parameter-efficient finetuning. We use this method to improve the automatic speech recognition results of spoken academic German and English. Our approach yields average relative word error rate (WER) improvements of approximately 29% on German academic speech and 25% on English academic speech. It also translates well to conversational speech, achieving relative WER improvements of up to 36%, and demonstrates modest gains of up to 5% on read speech. Moreover, we observe that incorporating utterances from the recent past as personalization context yields the most significant overall improvements and that changes in voice characteristics resulting from prolonged speaking have a minimal effect on the personalization quality of academic lectures. Y1 - 2024 U6 - https://doi.org/10.1109/SLT61566.2024.10832252 ER - TY - JOUR A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias ED - Baumann, Ilja T1 - Generative adversarial networks for whispered to voiced speech conversion: a comparative study JF - International Journal of Speech Technology N2 - Generative Adversarial Networks (GANs) have demonstrated promising results as end-to-end models for whispered to voiced speech conversion. Leveraging non-autoregressive systems like GANs capable of performing conditional waveform generation eliminates the need for separate models to estimate voiced speech features, and leads to faster inference compared to autoregressive methods. This study aims to identify the optimal GAN architecture for the whispered to voiced speech conversion task by comparing six state-of-the-art models. Furthermore, we present a method for evaluating the preservation of speaker identity and local accent, using embeddings obtained from speaker- and language identification systems. Our experimental results show that building the speech conversion system based on the HiFi-GAN architecture yields the best objective evaluation scores, outperforming the baseline by ∼9% relative using frequency-weighted Signal-to-Noise Ratio and Log Likelihood Ratio, as well as by ∼29% relative using Root Mean Squared Error. In subjective tests, HiFi-GAN yielded a mean opinion score of 2.9, significantly outperforming the baseline with a score of 1.4. Furthermore, HiFi-GAN enhanced ASR performance and preserved speaker identity and accent, with correct language detection rates of up to ∼98%. KW - Speech conversion · Generative adversarial networks · Whispered speech · Voiced speech Y1 - 2024 U6 - https://doi.org/10.1007/s10772-024-10161-1 VL - 27 ER - TY - CHAP A1 - Wagner, Dominik A1 - Lee, Seanie A1 - Baumann, Ilja A1 - Seeberger, Philipp A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Speculative Sampling for {GPU} Hardware Accelerators T2 - Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing N2 - In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.emnlp-main.370 PB - Association for Computational Linguistics CY - Miami, Florida, USA ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Glockner, Kevin A1 - Herygers, Aaricia A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Zakaria, Khabbab T1 - EXTENDING HANS: LARGE LANGUAGE MODELS FOR QUESTION ANSWERING, SUMMARIZATION, AND TOPIC SEGMENTATION IN AN ML-BASED LEARNING EXPERIENCE PLATFORM T2 - Elektronische Sprachsignalverarbeitung 2024, Tagungsband der 35. Konferenz, Regensburg, 6.-8. März 2024 N2 - Abstract: The use of chatbots based on large language models (LLMs) and their impact on society are influencing our learning experience platform Hochschul Assistenz-System (HAnS). HAnS uses machine learning (ML) methods to support students and lecturers in the online learning and teaching processes [1]. This paper introduces LLM-based features available in HAnS which are using the transcript of our improved Automatic Speech Recognition (ASR) pipeline with an average transcription duration of 45 seconds and an average word error rate (WER) of 6.66% on over 8 hours of audio data of 7 lecture videos. A LLM-based chatbot could be used to answer questions on the lecture content as the ASR transcript is provided as context. The summarization and topic segmentation uses the LLM to improve our learning experience platform. We generate multiple choice questions using the LLM and the ASR transcript as context during playback in a period of 3 minutes and display them in the HAnS frontend Y1 - 2024 SN - 978-3-95908-325-6 PB - TUPress CY - Dresden ER - TY - JOUR A1 - Pérez-Toro, Paula Andrea A1 - Vásquez-Correa, Juan Camilo A1 - Bocklet, Tobias A1 - Nöth, Elmar A1 - Orozco-Arroyave, Juan Rafael T1 - User State Modeling Based on the Arousal-Valence Plane: Applications in Customer Satisfaction and Health-Care JF - IEEE Transactions on Affective Computing N2 - The acoustic analysis helps to discriminate emotions according to non-verbal information, while linguistics aims to capture verbal information from written sources. Acoustic and linguistic analyses can be addressed for different applications, where information related to emotions, mood, or affect are involved. The Arousal-Valence plane is commonly used to model emotional states in a multidimensional space. This study proposes a methodology focused on modeling the user’s state based on the Arousal-Valence plane in different scenarios. Acoustic and linguistic information are used as input to feed different deep learning architectures mainly based on convolutional and recurrent neural networks, which are trained to model the Arousal-Valence plane. The proposed approach is used for the evaluation of customer satisfaction in call-centers and for health-care applications in the assessment of depression in Parkinson’s disease and the discrimination of Alzheimer’s disease. F-scores of up to 0.89 are obtained for customer satisfaction, of up to 0.82 for depression in Parkinson’s patients, and of up to 0.80 for Alzheimer’s patients. The proposed approach confirms that there is information embedded in the Arousal-Valence plane that can be used for different purposes. KW - Arousal-valence plane KW - acoustic KW - linguistic KW - customer satisfaction KW - alzheimer’s disease KW - depression Y1 - 2021 U6 - https://doi.org/10.1109/taffc.2021.3112543 SN - 1949-3045 VL - 14 IS - 2 SP - 1533 EP - 1546 PB - Institute of Electrical and Electronics Engineers (IEEE) ER - TY - CHAP A1 - Perez-Toro, P. A. A1 - Vasquez-Correa, J. C. A1 - Arias-Vergara, T. A1 - Klumpp, P. A1 - Sierra-Castrillon, M. A1 - Roldan-Lopez, M. E. A1 - Aguillon, D. A1 - Hincapie-Henao, L. A1 - Tobon-Quintero, C. A. A1 - Bocklet, Tobias A1 - Schuster, M. A1 - Orozco-Arroyave, J. R. A1 - Nöth, E. T1 - Acoustic and Linguistic Analyses to Assess Early-Onset and Genetic Alzheimer’s Disease T2 - ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - The PSEN1-E280A or Paisa mutation is responsible for most of Early-Onset Alzheimer’s (EOA) disease cases in Colombia. It affects a large kindred of over 5000 members that present the same phenotype. The most common symptoms are related to language disorders, where speech fluency is also affected due to the difficulty to access semantic information intentionally. This study proposes the use of acoustic and linguistic methods to extract features from speech recordings and their transcriptions to discriminate people with conditions related to the Paisa mutation. We consider state-of-the-art word-embedding methods like Word2Vec and Bidirectional Encoder Representations from Transformer to process the transcripts. The speech signals are modeled by using traditional acoustic features and speaker embeddings. To the best of our knowledge, this is the first study focused on evaluating genetic Alzheimer’s and EOA using acoustics and linguistics. KW - PSEN1–E280A KW - Alzheimer’s Disease KW - Acoustic Analysis KW - Linguistic Analysis Y1 - 2021 SN - 978-1-7281-7605-5 U6 - https://doi.org/10.1109/ICASSP39728.2021.9414009 SP - 8338 EP - 8342 PB - IEEE ER - TY - CHAP A1 - Scheuerer, Ralph A1 - Haderlein, Tino A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Applying X-Vectors on Pathological Speech After Larynx Removal T2 - 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) N2 - Speaker embeddings extracted from time delayed neural networks (TDNNs) contributed to major recent advancements in speaker recognition and verification. We use an X-Vector system trained on augmented VoxCeleb1 and VoxCeleb2 data to obtain embeddings for pathological speech after total or partial larynx removal. We show that our model is able to effectively distinguish and visualize patient groups when generating embeddings. We further compare various regression models on the task of automatically predicting different perceptual ratings by speech therapists (intelligibility, vocal effort, and overall quality) based on the extracted speaker embeddings. For both patient groups we show Pearson correlations in the range of +0.8; we find that Random Forest and Support Vector Regression produce scores that best resemble the experts' assessments. KW - laryngectomy KW - intelligibility KW - pathological speech KW - x-vectors Y1 - 2021 SN - 978-1-6654-3739-4 U6 - https://doi.org/10.1109/asru51503.2021.9688278 VL - 2021 SP - 1079 EP - 1086 PB - IEEE ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas H. A1 - Bayerl, Sebastian P. A1 - Auerbach, Thomas A1 - Bocklet, Tobias T1 - An Acoustical Machine Learning Approach to Determine Abrasive Belt Wear of Wide Belt Sanders T2 - 2022 IEEE Sensors N2 - This paper describes a machine learning approach to determine the abrasive belt wear of wide belt sanders used in industrial processes based on acoustic data, regardless of the sanding process-related parameters, Feed speed, Grit Size, and Type of material. Our approach utilizes Decision Tree, Random Forest, k-nearest Neighbors, and Neural network Classifiers to detect the belt wear from Spectrograms, Mel Spectrograms, MFCC, IMFCC, and LFCC, yielding an accuracy of up to 86.1% on five levels of belt wear. A 96% accuracy could be achieved with different Decision Tree Classifiers specialized in different sanding parameter configurations. The classifiers could also determine with an accuracy of 97% if the machine is currently sanding or is idle and with an accuracy of 98.4% and 98.8% detect the sanding parameters Feed speed and Grit Size. We can show that low-dimensional mappings of high-dimensional features can be used to visualize belt wear and sanding parameters meaningfully. KW - Acoustic sensors KW - Abrasive belt wear KW - Tool wear KW - Machine learning KW - Industrial process KW - Wide belt sanding machines Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/SENSORS52175.2022.9967324 VL - 2022 PB - IEEE ER - TY - CHAP A1 - Schmitt, Thomas H. A1 - Bundscherer, Maximilian A1 - Drechsel, Ralf A1 - Bocklet, Tobias T1 - Machine learning based optimization of a ceramic bushing manufacturing process T2 - 2022 IEEE Sensors N2 - Machine learning (ML) has shown great promise in a variety of domains in recent years. ML models are known to require large amounts of labeled training data, keeping small to medium-sized business from utilizing them. This paper presents ML based approach to optimize a ceramic bushing manufac-turing process, by predicting the employed press-fit process as a function of press punch position. Accurate predictions would ensure optimal process configuration, guaranteeing quality and reducing waste. Models are trained in a supervised manner to predict the press-fit process and the ceramic defect probabilities as functions of press punch position. We were able to predict the press-fit process with a mean correlation of 0.996 and assess whether the process would damage the ceramic with a mean precision of 96.7%. Our results exemplify how ML can be used to predict and optimize highly specialised processes even with small datasets. KW - Manufacturing KW - machine learning KW - optimization KW - ceramic bushing Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/sensors52175.2022.9967124 PB - IEEE ER - TY - CHAP A1 - Klumpp, P. A1 - Bocklet, Tobias A1 - Arias-Vergara, T. A1 - Vásquez-Correa, J. C. A1 - Pérez-Toro, P.A. A1 - Bayerl, Sebastian P. A1 - Orozco-Arroyave, J. R. A1 - Nöth, Elmar T1 - The Phonetic Footprint of Covid-19? T2 - Interspeech 2021 N2 - Against the background of the ongoing pandemic, this year’s Computational Paralinguistics Challenge featured a classification problem to detect Covid-19 from speech recordings. The presented approach is based on a phonetic analysis of speech samples, thus it enabled us not only to discriminate between Covid and non-Covid samples, but also to better understand how the condition influenced an individual’s speech signal. Our deep acoustic model was trained with datasets collected exclusively from healthy speakers. It served as a tool for segmentation and feature extraction on the samples from the challenge dataset. Distinct patterns were found in the embeddings of phonetic classes that have their place of articulation deep inside the vocal tract. We observed profound differences in classification results for development and test splits, similar to the baseline method. We concluded that, based on our phonetic findings, it was safe to assume that our classifier was able to reliably detect a pathological condition located in the respiratory tract. However, we found no evidence to claim that the system was able to discriminate between Covid-19 and other respiratory diseases. KW - COVID-19 Y1 - 2021 U6 - https://doi.org/10.21437/Interspeech.2021-1488 SN - 2958-1796 SP - 441 EP - 445 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias T1 - Nonwords Pronunciation Classification in Language Development Tests for Preschool Children T2 - Interspeech 2022 N2 - This work aims to automatically evaluate whether the language development of children is age-appropriate. Validated speech and language tests are used for this purpose to test the auditory memory. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures: Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR acoustic model). Each of the approaches provides input for VGG-like 5-layer CNN classifiers. We also examine the adaptation per nonword. The evaluation of the proposed systems was performed using recordings from different kindergartens of spoken nonwords. ECAPA-TDNN and low-level FFT features do not explicitly model phonetic information; wav2vec2.0 is trained on grapheme labels, our ASR acoustic model features contain (sub-)phonetic information. We found that the more granular the phonetic modeling is, the higher are the achieved recognition rates. The best system trained on ASR acoustic model features with VTLN achieved an accuracy of 89.4% and an area under the ROC (Receiver Operating Characteristic) curve (AUC) of 0.923. This corresponds to an improvement in accuracy of 20.2% and AUC of 0.309 relative compared to the FFT-baseline. Y1 - 2022 U6 - https://doi.org/10.21437/interspeech.2022-10777 SN - 2958-1796 VL - 2022 SP - 3643 EP - 3647 PB - ISCA ER - TY - CHAP A1 - Chen, Wenda A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Length- and Noise-Aware Training Techniques for Short-Utterance Speaker Recognition T2 - Interspeech 2020 N2 - Speaker recognition performance has been greatly improved with the emergence of deep learning. Deep neural networks show the capacity to effectively deal with impacts of noise and reverberation, making them attractive to far-field speaker recognition systems. The x-vector framework is a popular choice for generating speaker embeddings in recent literature due to its robust training mechanism and excellent performance in various test sets. In this paper, we start with early work on including invariant representation learning (IRL) to the loss function and modify the approach with centroid alignment (CA) and length variability cost (LVC) techniques to further improve robustness in noisy, far-field applications. This work mainly focuses on improvements for short-duration test utterances (1-8s). We also present improved results on long-duration tasks. In addition, this work discusses a novel self-attention mechanism. On the VOiCES far-field corpus, the combination of the proposed techniques achieves relative improvements of 7.0% for extremely short and 8.2% for full-duration test utterances on equal error rate (EER) over our baseline system. KW - speaker recognition KW - invariant representation learning KW - centroid alignment KW - x-vector KW - far-field Y1 - 2020 U6 - https://doi.org/10.21437/interspeech.2020-2872 SN - 2958-1796 SP - 3835 EP - 3839 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Detecting Vocal Fatigue with Neural Embeddings JF - Journal of Voice N2 - Vocal fatigue refers to the feeling of tiredness and weakness of voice due to extended utilization. This paper investigates the effectiveness of neural embeddings for the detection of vocal fatigue. We compare x-vectors, ECAPA-TDNN, and wav2vec 2.0 embeddings on a corpus of academic spoken English. Low-dimensional mappings of the data reveal that neural embeddings capture information about the change in vocal characteristics of a speaker during prolonged voice usage. We show that vocal fatigue can be reliably predicted using all three types of neural embeddings after 40 minutes of continuous speaking when temporal smoothing and normalization are applied to the extracted embeddings. We employ support vector machines for classification and achieve accuracy scores of 81% using x-vectors, 85% using ECAPA-TDNN embeddings, and 82% using wav2vec 2.0 embeddings as input features. We obtain an accuracy score of 76%, when the trained system is applied to a different speaker and recording environment without any adaptation. KW - Vocal fatigue KW - Neural embeddings KW - Visualization KW - Detection Y1 - 2023 U6 - https://doi.org/10.1016/j.jvoice.2023.01.012 SN - 0892-1997 PB - Elsevier BV ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - The Influence of Dataset Partitioning on Dysfluency Detection Systems T2 - Text, Speech, and Dialogue N2 - This paper empirically investigates the influence of different data splits and splitting strategies on the performance of dysfluency detection systems. For this, we perform experiments using wav2vec 2.0 models with a classification head as well as support vector machines (SVM) in conjunction with the features extracted from the wav2vec 2.0 model to detect dysfluencies. We train and evaluate the systems with different non-speaker-exclusive and speaker-exclusive splits of the Stuttering Events in Podcasts (SEP-28k) dataset to shed some light on the variability of results w.r.t. to the partition method used. Furthermore, we show that the SEP-28k dataset is dominated by only a few speakers, making it difficult to evaluate. To remedy this problem, we created SEP-28k-Extended (SEP-28k-E), containing semi-automatically generated speaker and gender information for the SEP-28k corpus, and suggest different data splits, each useful for evaluating other aspects of methods for dysfluency detection. KW - Computer Science KW - Informatics Y1 - 2022 SN - 9783031162695 U6 - https://doi.org/10.1007/978-3-031-16270-1_35 SN - 0302-9743 SP - 423 EP - 436 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Maruri, Hector A. Cordourier A1 - Bocklet, Tobias T1 - Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech T2 - 2022 IEEE Spoken Language Technology Workshop (SLT) N2 - This work adapts two recent architectures of generative models and evaluates their effectiveness for the conversion of whispered speech to normal speech. We incorporate the normal target speech into the training criterion of vector-quantized variational autoencoders (VQ-VAEs) and Mel-GANs, thereby conditioning the systems to recover voiced speech from whispered inputs. Objective and subjective quality measures indicate that both VQ-VAEs and MelGANs can be modified to perform the conversion task. We find that the proposed approaches significantly improve the Mel cepstral distortion (MCD) metric by at least 25% relative to a Disco-GAN baseline. Subjective listening tests suggest that the MelGAN-based system significantly improves naturalness, intelligibility, and voicing compared to the whispered input speech. A novel evaluation measure based on differences between latent speech representations also indicates that our MelGAN-based approach yields improvements relative to the baseline. KW - whispered speech KW - speech conversion KW - VAE KW - GAN KW - generative models Y1 - 2023 SN - 979-8-3503-9690-4 U6 - https://doi.org/10.1109/SLT54892.2023.10022796 SP - 943 EP - 948 PB - IEEE ER - TY - CHAP A1 - Vasquez-Correa, J. C. A1 - Bocklet, Tobias A1 - Orozco-Arroyave, J. R. A1 - Nöth, E. T1 - Comparison of User Models Based on GMM-UBM and I-Vectors for Speech, Handwriting, and Gait Assessment of Parkinson’s Disease Patients T2 - ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Parkinson's disease is a neurodegenerative disorder characterized by the presence of different motor impairments. Information from speech, handwriting, and gait signals have been considered to evaluate the neurological state of the patients. On the other hand, user models based on Gaussian mixture models - universal background models (GMMUBM) and i-vectors are considered the state-of-the-art in biometric applications like speaker verification because they are able to model specific speaker traits. This study introduces the use of GMM-UBM and i-vectors to evaluate the neurological state of Parkinson's patients using information from speech, handwriting, and gait. The results show the importance of different feature sets from each type of signal in the assessment of the neurological state of the patients. KW - Parkinson’s disease KW - GMM-UBM KW - ivectors KW - gait analysis KW - handwriting analysis KW - speech analysis Y1 - 2020 SN - 978-1-5090-6631-5 U6 - https://doi.org/10.1109/icassp40776.2020.9054348 SP - 6544 EP - 6548 PB - IEEE ER - TY - JOUR A1 - Schuster, Maria A1 - Arias-Vergara, Tomás A1 - Müller-Hörner, Rainer A1 - Winterholler, Cordula A1 - Bocklet, Tobias T1 - "Verstehen mich mit der Maske eigentlich noch alle?" JF - Geriatrie-Report Y1 - 2020 U6 - https://doi.org/10.1007/s42090-020-0627-3 SN - 1862-5363 VL - 15 IS - 3 SP - 8 EP - 9 PB - Springer Science and Business Media LLC ER - TY - CHAP A1 - Georges, Munir A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Compact Speaker Embedding: lrx-Vector BT - Proceedings of the Annual Conference of the International Speech Communication Association T2 - Interspeech 2020 N2 - Deep neural networks (DNN) have recently been widely used in speaker recognition systems, achieving state-of-the-art performance on various benchmarks. The x-vector architecture is especially popular in this research community, due to its excellent performance and manageable computational complexity. In this paper, we present the lrx-vector system, which is the low-rank factorized version of the x-vector embedding network. The primary objective of this topology is to further reduce the memory requirement of the speaker recognition system. We discuss the deployment of knowledge distillation for training the lrx-vector system and compare against low-rank factorization with SVD. On the VOiCES 2019 far-field corpus we were able to reduce the weights by 28% compared to the full-rank x-vector system while keeping the recognition rate constant (1.83% EER). KW - speaker recognition KW - x-vector KW - low power Y1 - 2020 U6 - https://doi.org/10.21437/interspeech.2020-2106 SN - 2958-1796 SP - 3236 EP - 3240 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas A1 - Bocklet, Tobias T1 - Machine Learning in Industrial Quality Control of Glass Bottle Prints N2 - In industrial manufacturing of glass bottles, quality control of bottle prints is necessary as numerous factors can negatively affect the printing process. Even minor defects in the bottle prints must be detected despite reflections in the glass or manufacturing-related deviations. In cooperation with our medium-sized industrial partner, two ML-based approaches for quality control of these bottle prints were developed and evaluated, which can also be used in this challenging scenario. Our first approach utilized different filters to supress reflections (e.g. Sobel or Canny) and image quality metrics for image comparison (e.g. MSE or SSIM) as features for different supervised classification models (e.g. SVM or k-Neighbors), which resulted in an accuracy of 84%. The images were aligned based on the ORB algorithm, which allowed us to estimate the rotations of the prints, which may serve as an indicator for anomalies in the manufacturing process. In our second approach, we fine-tuned different pre-trained CNN models (e.g. ResNet or VGG) for binary classification, which resulted in an accuracy of 87%. Utilizing Grad-Cam on our fine-tuned ResNet-34, we were able to localize and visualize frequently defective bottle print regions. This method allowed us to provide insights that could be used to optimize the actual manufacturing process. This paper also describes our general approach and the challenges we encountered in practice with data collection during ongoing production, unsupervised preselection, and labeling. KW - Machine Learning, Quality Control, Industrial Manufacturing, Glass Bottle Printsuring Optimization, Glass Printing Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2409.20132 ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Large Language Models for Dysfluency Detection in Stuttered Speech N2 - Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task. KW - dysfluency detection, stuttering, large language models, wav2vec 2.0, Whisper, pathological speech Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2406.11025 ER - TY - RPRT A1 - Schmitt, Thomas A1 - Bundscherer, Maximilian A1 - Bocklet, Tobias T1 - Semmeldetector: Application of Machine Learning in Commercial Bakeries N2 - The Semmeldetector, is a machine learning application that utilizes object detection models to detect, classify and count baked goods in images. Our application allows commercial bakers to track unsold baked goods, which allows them to optimize production and increase resource efficiency. We compiled a dataset comprising 1151 images that distinguishes between 18 different types of baked goods to train our detection models. To facilitate model training, we used a Copy-Paste augmentation pipeline to expand our dataset. We trained the state-of-the-art object detection model YOLOv8 on our detection task. We tested the impact of different training data, model scale, and online image augmentation pipelines on model performance. Our overall best performing model, achieved an AP@0.5 of 89.1% on our test set. Based on our results, we conclude that machine learning can be a valuable tool even for unforeseen industries like bakeries, even with very limited datasets. KW - machine learning, object detection, YOLOv8, image composition, baked goods, food inspection, industrial automation Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2406.04050 ER - TY - JOUR A1 - Kammerbauer, Roland A1 - Schmitt, Thomas A1 - Bocklet, Tobias T1 - Segmenting Wood Rot using Computer Vision Models N2 - In the woodworking industry, a huge amount of effort has to be invested into the initial quality assessment of the raw material. In this study we present an AI model to detect, quantify and localize defects on wooden logs. This model aims to both automate the quality control process and provide a more consistent and reliable quality assessment. For this purpose a dataset of 1424 sample images of wood logs is created. A total of 5 annotators possessing different levels of expertise is involved in dataset creation. An inter-annotator agreement analysis is conducted to analyze the impact of expertise on the annotation task and to highlight subjective differences in annotator judgement. We explore, train and fine-tune the state-of-the-art InternImage and ONE-PEACE architectures for semantic segmentation. The best model created achieves an average IoU of 0.71, and shows detection and quantification capabilities close to the human annotators. KW - machine learning, image segmentation, semantic segmentation, InternImage, ONEPEACE, lumbering, industrial quality control, industrial automation Y1 - 2024 ER - TY - CHAP A1 - Baumann, Ilja A1 - Unger, Nicole A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Automatic Evaluation of a Sentence Memory Test for Preschool Children N2 - Assessment of memory capabilities in preschool-aged children is crucial for early detection of potential speech development impairments or delays. We present an approach for the automatic evaluation of a standardized sentence memory test specifically for preschool children. Our methodology leverages automatic transcription of recited sentences and evaluation based on natural language processing techniques. We demonstrate the effectiveness of our approach on a dataset comprised of recited sentences from preschool-aged children, incorporating ratings of semantic and syntactic correctness. The best performing systems achieve an F1 score of 91.7% for semantic correctness and 86.1% for syntactic correctness using automatic transcripts. Our results showcase the potential of automated evaluation systems in providing reliable and efficient assessments of memory capabilities in early childhood, facilitating timely interventions and support for children with language development needs. KW - speech development, children’s speech, automatic assessment Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2125 SP - 5158 EP - 5162 ER - TY - JOUR A1 - Schmitt, Thomas A1 - Bundscherer, Maximilian A1 - Bocklet, Tobias T1 - Training a Computer Vision Model for Commercial Bakeries with Primarily Synthetic Images N2 - In the food industry, reprocessing returned product is a vital step to increase resource efficiency. [SBB23] presented an AI application that automates the tracking of returned bread buns. We extend their work by creating an expanded dataset comprising 2432 images and a wider range of baked goods. To increase model robustness, we use generative models pix2pix and CycleGAN to create synthetic images. We train state-of-the-art object detection model YOLOv9 and YOLOv8 on our detection task. Our overall best-performing model achieved an average precision AP@0.5 of 90.3% on our test set. KW - machine learning, object detection, YOLOv9, image composition, baked goods, food industry, industrial automation Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2409.20122 ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Schuster, Maria A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Towards Self-Attention Understanding for Automatic Articulatory Processes Analysis in Cleft Lip and Palate Speech N2 - Cleft lip and palate (CLP) speech presents unique challenges for automatic phoneme analysis due to its distinct acoustic characteristics and articulatory anomalies. We perform phoneme analysis in CLP speech using a pre-trained wav2vec 2.0 model with a multi-head self-attention classification module to capture long-range dependencies within the speech signal, thereby enabling better contextual understanding of phoneme sequences. We demonstrate the effectiveness of our approach in the classification of various articulatory processes in CLP speech. Furthermore, we investigate the interpretability of self-attention to gain insights into the model’s understanding of CLP speech characteristics. Our findings highlight the potential of the selfattention mechanisms for improving automatic phoneme analysis in CLP speech, paving the way for enhanced diagnostics, adding interpretability for therapists and affected patients. KW - pathologic speech, cleft lip and palate, children’s speech, automatic assessment Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2134 SP - 2430 EP - 2434 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias ED - Pesak, Krisztina T1 - Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models N2 - This paper explores the improvement of post-training quantization (PTQ) after knowledge distillation in the Whisper speech foundation model family. We address the challenge of outliers in weights and activation tensors, known to impede quantization quality in transformer-based language and vision models. Extending this observation to Whisper, we demonstrate that these outliers are also present when transformer-based models are trained to perform automatic speech recognition, necessitating mitigation strategies for PTQ. We show that outliers can be reduced by a recently proposed gating mechanism in the attention blocks of the student model, enabling effective 8-bit quantization, and lower word error rates compared to student models without the gating mechanism in place. KW - post-training quantization, Whisper, gated attention, outliers Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2105 SP - 4623 EP - 4627 ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education; Learning Experience Platform; Open Source Software; Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 SN - 2944-7682 PB - Gesellschaft für Informatik e.V. CY - Bonn ER - TY - JOUR A1 - Escobar-Grisales, Daniel A1 - Ríos-Urrego, Cristian-David A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Garcia, Adolfo A1 - Orozco-Arroyave, Juan rafael T1 - It’s Time to Take Action: Acoustic Modeling of Motor Verbs to Detect Parkinson’s Disease N2 - Pre-trained models generate speech representations that are used in different tasks, including the automatic detection of Parkinson’s disease (PD). Although these models can yield high accuracy, their interpretation is still challenging. This paper used a pre-trained Wav2vec 2.0 model to represent speech frames of 25ms length and perform a frame-by-frame discrimination between PD patients and healthy control (HC) subjects. This fine granularity prediction enabled us to identify specific linguistic segments with high discrimination capability. Speech representations of all produced verbs were compared w.r.t. nouns and the first ones yielded higher accuracies. To gaina deeper understanding of this pattern, representations of motor and non-motor verbs were compared and the first ones yielded better results, with accuracies of around 83% in an independent test set. These findings support well-established neurocognitive models about action-related language highlighted as key drivers of PD. Index Terms: computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease KW - computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2205 ER - TY - CHAP A1 - Braun, Franziska A1 - Witzl, Christopher A1 - Erzigkeit, Andreas A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Pitfalls and Limits in Automatic Dementia Assessment T2 - Interspeech 2025 N2 - Current work on speech-based dementia assessment focuses on either feature extraction to predict assessment scales, or on the automation of existing test procedures. Most research uses public data unquestioningly and rarely performs a detailed error analysis, focusing primarily on numerical performance. We perform an in-depth analysis of an automated standardized dementia assessment, the Syndrom-Kurz-Test. We find that while there is a high overall correlation with human annotators, due to certain artifacts, we observe high correlations for the severely impaired individuals, which is less true for the healthy or mildly impaired ones. Speech production decreases with cognitive decline, leading to overoptimistic correlations when test scoring relies on word naming. Depending on the test design, fallback handling introduces further biases that favor certain groups. These pitfalls remain independent of group distributions in datasets and require differentiated analysis of target groups. KW - dementia screening KW - neuropsychological tests KW - pathological speech Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2280 SN - 2958-1796 SP - 5663 EP - 5667 PB - ISCA ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas A1 - Baumann, Ilja A1 - Bocklet, Tobias T1 - Digital Operating Mode Classification of Real-World Amateur Radio Transmissions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - This study presents an ML approach for classifying digital radio operating modes evaluated on real-world transmissions. We generated 98 different parameterized radio signals from 17 digital operating modes, transmitted each of them on the 70 cm (UHF) amateur radio band, and recorded our transmissions with two different architectures of SDR receivers. Three lightweight ML models were trained exclusively on spectrograms of limited non-transmitted signals with random characters as payloads. This training involved an online data augmentation pipeline to simulate various radio channel impairments. Our best model, EfficientNetB0, achieved an accuracy of 93.80% across the 17 operating modes and 85.47% across all 98 parameterized radio signals, evaluated on our real-world transmissions with Wikipedia articles as payloads. Furthermore, we analyzed the impact of varying signal durations & the number of FFT bins on classification, assessed the effectiveness of our simulated channel impairments, and tested our models across multiple simulated SNRs. KW - Automatic Modulation Classification KW - Amateur KW - Radio KW - Spectrum Monitoring KW - Cognitive Radio KW - Machine Learning Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889837 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Self-supervised Training with BEST-RQ for Speech Recognition T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multicodebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on testother using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training. KW - self-supervised learning KW - speech recognition KW - conformer KW - BEST-RQ Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889362 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Freisinger, Steffen A1 - Seeberger, Philipp A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation T2 - Interspeech 2025 N2 - Segmenting speech transcripts into thematic sections benefits both downstream processing and users who depend on written text for accessibility. We introduce a novel approach to hierarchical topic segmentation in transcripts, generating multi-level tables of contents that capture both topic and subtopic boundaries. We compare zero-shot prompting and LoRA fine-tuning on large language models, while also exploring the integration of high-level speech pause features. Evaluations on English meeting recordings and multilingual lecture transcripts (Portuguese, German) show significant improvements over established topic segmentation baselines. Additionally, we adapt a common evaluation measure for multi-level segmentation, taking into account all hierarchical levels within one metric. KW - topic segmentation KW - spoken content segmentation KW - table of contents generation KW - hierarchical segmentation Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2792 SN - 2958-1796 SP - 276 EP - 280 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Lee, Seanie A1 - Bok Lee, Dong A1 - Wagner, Dominik A1 - Kang, Minki A1 - Seong, Haebin A1 - Bocklet, Tobias A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models, JF - Findings of the Association for Computational Linguistics: ACL 2025 N2 - Deploying large language models (LLMs) in real-world applications requires robust safety guard models to detect and block harmful user prompts. While large safety guard models achieve strong performance, their computational cost is substantial. To mitigate this, smaller distilled models are used, but they often underperform on “hard” examples where the larger model provides accurate predictions. We observe that many inputs can be reliablyhandled by the smaller model, while only a small fraction require the larger model’s capacity. Motivated by this, we propose SafeRoute, a binary router that distinguishes hard examples from easy ones. Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy compared to solely using the larger safety guard model. Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance, outperforming relevant baselines. Y1 - 2025 UR - https://aclanthology.org/2025.findings-acl.105.pdf PB - Association for Computational Linguistics CY - Vienna, Austria ER -