TY - RPRT A1 - Albrecht, Jens A1 - Lehmann, Robert A1 - Zauter, Sigrid T1 - Digitale Assistenz in der psychosozialen Beratung BT - Projekt-Abschlussbericht ; 9.2019-2.2021 N2 - In vielen Unternehmen werden heutzutage Chatbots eingesetzt, um Standard-Anfragen von Kunden effizient zu beantworten. Auch im psychosozialen Beratungsumfeld können natürlichsprachliche digitale Assistenzen eine sinnvolle Ergänzung zu menschlichen Beratern sein, denn sie sind jederzeit verfügbare, anonyme Ansprechpartner. Die Hürden für den Einsatz in der psychosozialen Beratung sind allerdings hoch, da die Entwicklung und der Betrieb von Chatbots zunächst Kosten verursacht und systematische Studien zum Mehrwehrt derartiger Systeme in diesem Anwendungsfeld bisher nicht vorliegen. Ziel des Projekts DIA (Digitale Assistenz) war die prototypische Entwicklung eines lernfähigen Chatbots für die psychosoziale Beratung. Dabei wurde insbesondere die Frage untersucht, wie aufwändig es ist, einen KI-basierten Chatbot ausschließlich mit Open-Source-Technologie zu entwickeln. Das von der Staedtler-Stiftung geförderte Projekt lief von September 2019 bis Februar 2021. Die Arbeiten wurden in einer interdisziplinären Kooperation vom Institut für E-Beratung mit der Fakultät Informatik der TH Nürnberg durchgeführt. Als Anwendungsdomäne diente die Bildungsberatung für den Zertifikatskurs Online-Beratung, die bisher hauptsächlich über E-Mail-Kommunikation erfolgt. T3 - E-Beratung - 2021,1 KW - Onlineberatung, Onlinekommunikation, digitale Kommunikation, Chatbot KW - Psychosoziale Beratung KW - computerunterstützte Kommunikation KW - ChatBot Y1 - 2021 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:92-opus4-7792 VL - 2021 IS - 1 ER - TY - RPRT A1 - Lehmann, Robert A1 - Albrecht, Jens A1 - Zauter, Sigrid T1 - CaSoTex, Computerunterstützte Analyse Sozialwissenschaftlicher Texte T2 - E-Beratung N2 - Ziel des Forschungsprojektes war die Erforschung der Einsatzmöglichkeiten computerlinguistischer Methoden und maschineller Lernverfahren zur automatisierten Analyse sozialwissenschaftlicher Texte. Die Datengrundlage bildeten digital vorliegende Beiträge aus moderierten, sozialen Beratungsforen. Anhand konkreter sozial-wissenschaftlicher Fragestellungen zu Wirkungszusammenhängen in der Onlineberatung sollte untersucht werden, inwiefern Computer-Linguistik und maschinelle Lernverfahren qualitative Analysen unterstützen bzw. ergänzen können, wo die Grenzen der Verfahren liegen und wie bei deren Einsatz vorzugehen ist. Im Projekt wurden erste konkrete Erkenntnisse generiert, der Forschungsverbund vergrößerte sich, Praxis-Forschungsprojekte wurden beantragt und es entstanden weitere Überlegungen zu größeren Forschungsprojekten. Die aktuellen Ergebnisse sind vielversprechend. Überwachte Lernverfahren können mit ausreichend vielen und gut vorbereiteten Trainingsdaten interpretierbare und verwertbare Erkenntnisse liefern. T3 - E-Beratung - 2020,1 KW - Künstliche Intelligenz KW - Computerunterstützte Kommunikation KW - Onlineberatung KW - Onlinekommunikation KW - digitale Kommunikation Y1 - 2020 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:92-opus4-6197 VL - 2020 IS - 1 CY - Nürnberg ER - TY - CHAP A1 - Braun, Franziska A1 - Erzigkeit, Andreas A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Riedhammer, Korbinian A1 - Bayerl, Sebastian P. T1 - Going Beyond the Cookie Theft Picture Test: Detecting Cognitive Impairments Using Acoustic Features T2 - Text, Speech, and Dialogue Y1 - 2022 SN - 9783031162695 SN - 978-3-031-16270-1 U6 - https://doi.org/10.1007/978-3-031-16270-1_36 SN - 0302-9743 SP - 437 EP - 448 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Frassetto, Tommaso A1 - Jauernig, Patrick A1 - Riedhammer, Korbinian A1 - Sadeghi, Ahmad-Reza A1 - Schneider, Thomas A1 - Stapf, Emmanuel A1 - Weinert, Christian T1 - Offline Model Guard: Secure and Private ML on Mobile Devices T2 - 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE) N2 - Performing machine learning tasks in mobile applications yields a challenging conflict of interest: highly sensitive client information (e.g., speech data) should remain private while also the intellectual property of service providers (e.g., model parameters) must be protected. Cryptographic techniques offer secure solutions for this, but have an unacceptable overhead and moreover require frequent network interaction.In this work, we design a practically efficient hardware-based solution. Specifically, we build OFFLINE MODEL GUARD (OMG) to enable privacy-preserving machine learning on the predominant mobile computing platform ARM—even in offline scenarios. By leveraging a trusted execution environment for strict hardware-enforced isolation from other system components, OMG guarantees privacy of client data, secrecy of provided models, and integrity of processing algorithms. Our prototype implementation on an ARM HiKey 960 development board performs privacy-preserving keyword recognition using TensorFlow Lite for Microcontrollers in real time. Y1 - 2020 U6 - https://doi.org/10.23919/DATE48585.2020.9116560 PB - IEEE ER - TY - CHAP A1 - Rudolph, Eric A1 - Seer, Hanna A1 - Mothes, Carina A1 - Albrecht, Jens T1 - Automated feedback generation in an intelligent tutoring system for counselor education T2 - Annals of Computer Science and Information Systems N2 - This paper investigates the implementation of AI- driven feedback in an intelligent tutoring system (ITS) developed for training of counselors. By using LLMs, the study explores the automatic generation of feedback for communication-intensive tasks such as online counseling. The evaluation compares dif- ferent feedback methods, including the sandwich, WWW and STATE methods, and assesses their emotional and objective impact. The results show that AI-generated feedback fulfills objective criteria better than emotional ones. Fine-tuning an open source LLM can improve both the emotional and objective quality of feedback. Furthermore, the study examines the accep- tance of AI feedback among aspiring counselors, highlighting the influence of familiarity with AI on acceptance levels. Ethical con- siderations, including bias and hallucination, are addressed, with recommendations for risk mitigation through multi-feedback options and expert supervision. This research contributes to the understanding of the role of AI in improving digital counseling practices and highlights the need for continuous evaluation and ethical considerations. KW - Intelligent Tutoring System KW - Automated Feedback Generation KW - Artificial Intelligence KW - Counselling Y1 - 2024 U6 - https://doi.org/10.15439/2024F1649 SN - 2300-5963 VL - 39 SP - 501 EP - 512 PB - IEEE ER - TY - CHAP A1 - Albrecht, Jens A1 - Rudolph, Eric A1 - Poltermann, Aleksandra A1 - Lehmann, Robert T1 - The Virtual Client: Leveraging Generative AI for Innovative Online-Counselor Education T2 - ICERI Proceedings N2 - In an increasingly intricate world, individuals turn to online platforms for guidance from trained counselors regarding various psychological and social challenges. The training of competent and effective counselors is challenging, requiring specific strategies, methods and realistic role-playing scenarios. VirCo (Virtual Client for Online Counseling) is an innovative educational platform that employs an open-source large language model (LLM) to simulate clients facing psycho-social problems. VirCo enables learners to engage with a variety of client personas through realistic conversations, thereby enhancing their counseling abilities in a self-directed manner. Furthermore, it features tools for course development, assignments, and multiple feedback systems, fostering a collaborative learning environment. This innovative use of AI technology offers scalable and practical training solutions, contributing to a skilled and proficient workforce in online counseling. KW - Education, Role-Play, Counseling, Generative AI Y1 - 2024 U6 - https://doi.org/10.21125/iceri.2024.0454 SN - 2340-1095 SP - 1522 EP - 1531 PB - IATED ER - TY - CHAP A1 - Lehmann, Robert A1 - Albrecht, Jens A1 - Zauter, Sigrid T1 - Die Computerunterstützte Analyse Sozialwissenschaftlicher Texte – Ergebnisse des Forschungsprojekts „Casotex“ T2 - Gegenwart und Zukunft sozialer Dienstleistungsarbeit N2 - Ein Datensatz aus mehreren deutschsprachigen Onlineberatungsforen wurde mit verschiedenen maschinellen Lernverfahren untersucht. Dabei können unüberwachte Lernverfahren nur in Bezug auf die linguistische Gestaltung der Beiträge relevante Ergebnisse zeigen. Überwachte maschinelle Lernverfahren zeigen ein sehr großes Potenzial zur Klassifikation von Textstellen analog zur qualitativen Inhaltsanalyse. Damit eignen sich maschinelle Lernverfahren prinzipiell für die qualitative Analyse großer Textmengen, wie sie z. B. in der Onlineberatung vorliegen. Y1 - 2021 SN - 9783658325558 U6 - https://doi.org/10.1007/978-3-658-32556-5_12 SN - 2569-2127 SP - 167 EP - 180 PB - Springer Fachmedien Wiesbaden CY - Wiesbaden ER - TY - RPRT A1 - Albrecht, Jens A1 - Lehmann, Robert A1 - Poltermann, Aleksandra T1 - GeCCo 1.0 - Erstellung eines öffentlichen Datensatzes für die KI-basierte Inhaltsanalyse in der Online-Beratung T2 - Schriftenreihe der Technischen Hochschule Nürnberg Georg Simon Ohm N2 - Ein Einsatzfeld Künstlicher Intelligenz im Bereich der Sozialen Arbeit ist die automatische Inhaltsanalyse von Konversationen aus der psychosozialen Onlineberatung. Mögliche Anwendungen sind die statistische Untersuchung von Wirkzusammenhängen sowie die Entwicklung KI-basierter Dialog-und Unterstützungssysteme. Die Grundlage für das Training entsprechender Modelle sind aussagekräftige und realistische Trainingsdaten. Im Rahmen des Projektes GeCCo (German e-Counseling Conversation Dataset) wurde ein erster deutschsprachiger Datensatz für die Analyse psychosozialer Beratungskonversationen erstellt und veröffentlicht. Wesentlicher Bestandteil ist ein neu entwickeltes Kategoriensystem, das eine feingranulare inhaltliche Klassifikation in 40 Berater-und 28 Klienten-Kategorien ermöglicht. Basierend auf dem Datensatz konnten erfolgreich verschiedene Modelle mithilfe maschineller Lernverfahren trainiert und für die wissenschaftliche Nutzung bereitgestellt werden. KW - Online-Beratung KW - KI KW - Datensatz KW - automatische Inhaltsanalyse Y1 - 2024 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:92-opus4-13894 SP - 3 EP - 15 ER - TY - CHAP A1 - Riedhammer, Korbinian A1 - Baumann, Ilja A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias A1 - Braun, Franziska A1 - Wagner’, Dominik T1 - Medical Speech Processing for Diagnosis and Monitoring BT - Clinical Use Cases N2 - In recent years, speech processing for medical applications got significant traction. While pioneering work in the 1990ies focused on processing sustained vowels or isolated utterances, work in the 2000s already showed, that speech recognition systems, prosodic analysis and natural language processing be used to assess a large variety of speech pathologies.Here, we give an overview of how to classify selected speech pathologies including stuttering, language development, speech intelligibility after surgery, dementia and Alzheimers, depression and state-of-mind. While each of those poses a rather well-defined problem in a lab setting, we discuss the issues when integrating such methods in a clinical workflow such as diagnosis or monitoring. Starting from the question if such detectors can be used for general screening or rather as a specialist's tool, we explore the legal and privacy-related implications: patient-doctor conversations, working with children or demented seniors, bias towards examiner or patient, on-device vs. cloud processing.We conclude with a set of open questions that should be addressed to help bringing all this research from the lab to routine clinical use. Y1 - 2023 SP - 1417 EP - 1420 ER - TY - CHAP A1 - Freier, Carolin A1 - Schmohl, Tobias A1 - Schelling, Katrin A1 - Go, Stefanie A1 - Hunger, Marianne A1 - Hofmann, Franziska A1 - Helten, Anne-Katrin A1 - Richter, Florian T1 - Combining NLP, speech recognition, and indexing: an AI-based learning assistant for higher education N2 - This paper presents the ongoing development of HAnS (Hochschul-Assistenz-System), an Intelligent Tutoring System (ITS) designed to support self-directed digital learning in higher education. Initiated by twelve collaborating German universities and research institutes, HAnS is developed 2021–2025 with the goal of utilizing artificial intelligence (AI) and Big Data in academic settings to enhance technology-based learning. The system employs AI for speech recognition and the indexing of existing learning resources, enabling users to search and compile these materials based on various parameters. Here, we provide an overview of the project, showcasing how iterative design and development processes contribute to innovative educational research in the evolving field of AI-based ITS in higher education. Notwithstanding the potential of HAnS, we also deliberate upon the challenges associated with ensuring a suitable dataset for training the AI, refining complex algorithms for personalization, and maintaining data privacy. KW - artificial intelligence in education (AIED), teaching/learning materials, intelligent tutoring system (ITS), adaptive learning, digital education. Y1 - 2023 UR - https://conference.pixel-online.net/files/foe/ed0013/FP/3516-CDEV5940-FP-FOE13.pdf ER - TY - JOUR A1 - Axenie, Cristian A1 - Halilov, Ertan A1 - Main, Julian A1 - Weiss, David T1 - Edge neuro-statistical learning for event-based visual motion detection and tracking in roadside safety systems JF - Neuromorphic Computing and Engineering N2 - The Vision Zero Program's purpose is to reduce traffic-related fatalities and serious injuries while promoting equitable, safe, and healthy mobility for all. Ultimately, the challenge is to detect pedestrians during the day and especially at night to implement safety measures. The current study introduces an award-winning low-power solution employing neuromorphic visual sensing and hybrid neuro-statistical processing developed by the Technische Hochschule Nürnberg team for the TinyML Vision Zero San Jose Competition. The solution proposes a novel neuromorphic edge fusion of spiking neural networks and event-based expectation maximization for the detection and tracking of pedestrians and bicyclists. We provide a deployment-ready evaluation of the detection performance along with robustness, energy footprint, and weatherization while emphasizing the advantages of the neuro-statistical edge solution and its city-level scaling capabilities. Y1 - 2025 U6 - https://doi.org/10.1088/2634-4386/adcbcb SN - 2634-4386 PB - IOP Publishing ER - TY - JOUR A1 - Axenie, Cristian T1 - Antifragile control systems in neuronal processing: a sensorimotor perspective JF - Biological Cybernetics N2 - The stability–robustness–resilience–adaptiveness continuum in neuronal processing follows a hierarchical structure that explains interactions and information processing among the different time scales. Interestingly, using “canonical” neuronal computational circuits, such as Homeostatic Activity Regulation, Winner-Take-All, and Hebbian Temporal Correlation Learning, one can extend the behavior spectrum towards antifragility. Cast already in both probability theory and dynamical systems, antifragility can explain and define the interesting interplay among neural circuits, found, for instance, in sensorimotor control in the face of uncertainty and volatility. This perspective proposes a new framework to analyze and describe closed-loop neuronal processing using principles of antifragility, targeting sensorimotor control. Our objective is two-fold. First, we introduce antifragile control as a conceptual framework to quantify closed-loop neuronal network behaviors that gain from uncertainty and volatility. Second, we introduce neuronal network design principles, opening the path to neuromorphic implementations and transfer to technical systems. Y1 - 2025 U6 - https://doi.org/10.1007/s00422-025-01003-7 SN - 1432-0770 VL - 119 IS - 2-3 PB - Springer Nature ER - TY - CHAP A1 - Braun, Franziska A1 - Förstel, Markus A1 - Oppermann, Bastian A1 - Erzigkeit, Andreas A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Riedhammer, Korbinian T1 - Automated Evaluation of Standardized Dementia Screening Tests T2 - Interspeech 2022 N2 - For dementia screening and monitoring, standardized tests play a key role in clinical routine since they aim at minimizing subjectivity by measuring performance on a variety of cognitive tasks. In this paper, we report a study consisting of a semistandardized history taking followed by two standardized neuropsychological tests, namely the SKT and the CERAD-NB. The tests include basic tasks such as naming objects, learning word lists, but also widely used tools such as the MMSE. Most of the tasks are performed verbally and should thus be suitable for automated scoring based on transcripts. For the first batch of 30 patients, we analyze the correlation between expert manual evaluations and automatic evaluations based on manual and automatic transcriptions. For both SKT and CERAD-NB, we observe high to perfect correlations using manual transcripts; for certain tasks with lower correlation, the automatic scoring is stricter than the human reference since it is limited to the audio. Using automatic transcriptions, correlations drop as expected and are related to recognition accuracy; however, we still observe high correlations of up to 0.98 (SKT) and 0.85 (CERADNB). We show that using word alternatives helps to mitigate recognition errors and subsequently improves correlation with expert scores. KW - dementia screening KW - neuropsychological tests Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10436 SP - 2478 EP - 2482 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Agchar, Ismael A1 - Baumann, Ilja A1 - Braun, Franziska A1 - Perez-Toro, Paula Andrea A1 - Riedhammer, Korbinian A1 - Trump, Sebastian A1 - Ullrich, Martin T1 - A Survey of Music Generation in the Context of Interaction N2 - In recent years, machine learning, and in particular generative adversarial neural networks (GANs) and attention-based neural networks (transformers), have been successfully used to compose and generate music, both melodies and polyphonic pieces. Current research focuses foremost on style replication (e.g., generating a Bach-style chorale) or style transfer (e.g., classical to jazz) based on large amounts of recorded or transcribed music, which in turn also allows for fairly straight-forward “performance” evaluation. However, most of these models are not suitable for human-machine co-creation through live interaction, neither is clear, how such models and resulting creations would be evaluated. This article presents a thorough review of music representation, feature analysis, heuristic algorithms, statistical and parametric modelling, and human and automatic evaluation measures, along with a discussion of which approaches and models seem most suitable for live interaction. Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2402.15294 ER - TY - CHAP A1 - Albrecht, Jens A1 - Fiedler, Marc A1 - Kiefer, Tim T1 - A Rule-Based Approach to Local Anonymization for Exclusivity Handling in Statistical Databases T2 - In: Domingo-Ferrer, J., Pejić-Bach, M. (eds) Privacy in Statistical Databases. PSD 2016. Lecture Notes in Computer Science, vol 9867. N2 - Statistical databases in general and data warehouses in particular are used to analyze large amounts of business data in pre-defined as well as ad-hoc reports. Operators of statistical databases must ensure that individual sensitive data, e.g., personal data, medical data, or business-critical data, are not revealed to unprivileged users while making use of these data in aggregates. Business rules must be defined and enforced to prevent disclosure. The unsupervised nature of ad-hoc reports, defined by the user and unknown to the database operator upfront, adds to the complexity of keeping data secure. Storing sensitive data in statistical databases demands automated methods to prevent direct or indirect disclosure of such sensitive data. This document describes a rule-based approach to local recoding of sensitive data. It introduces the notion of exclusivity to describe quasiidentifiers with local rules based on the multidimensional data model. It further defines options to treat exclusive entities that may disclose sensitive data. The local anonymization minimizes information loss and works well even with very large data sets. Y1 - 2016 U6 - https://doi.org/10.1007/978-3-319-45381-1_7 PB - Springer CY - Cham ER - TY - CHAP A1 - Gheorghe, Lisca A1 - Matthias, Kindler A1 - Tanja, Lerchl A1 - Axenie, Cristian A1 - Thomas, Grauschopf A1 - Veit, Senner T1 - The potential of reinforcement learning for lumbar load prediction in multi body models of the spine N2 - Multi body models (MBS) of the spine are an integral part of clinical and biomechanical research. Their noninvasive and adaptive character makes them a promising tool to address a large variety of questions regarding spinal loading, its causes, and consequences for the healthy and pathological spine. In sports science and athletic training, the predictive simulations of human movement can explain how changes in training, technique, or equipment affect performance and the body's biomechanical responses, helping athletes and coaches to make informed decisions. For solving predictive simulations optimization algorithms like collocation method are the most performant ones. The Reinforcement Learning (RL) algorithms propose a transition from optimization to learning. They formulate the problem of predictive simulation as learning to generate new data that describes human movement. This formulation enables them to leverage the generative power of the Artificial Neural Networks (ANNs), capable of learning models from high-dimensional and nonlinear movement data, and subsequently using these models to generate new ones within new boundary conditions. Most studies including MBS of the spine use a combination of inverse kinematics and optimization for muscle force and lumbar load estimation. However, these approaches either use generic assumptions for loading tasks of low complexity or require kinematic data from experimental studies. In this study, we address the following question: What is the potential of RL algorithms for predicting the spine’s joint torques during an extension? Y1 - 2024 U6 - https://doi.org/10.17028/rd.lboro.27045154.v1 ER - TY - JOUR A1 - Freisinger, Steffen A1 - Schneider, Fabian A1 - Herygers, Aaricia A1 - Georges, Munir A1 - Riedhammer, Korbinian ED - Pesak, Krisztina T1 - Unsupervised Multilingual Topic Segmentation of Video Lectures: What can Hierarchical Labels tell us about the Performance? N2 - The current shift from in-person to online education, e.g., through lecture videos, requires novel techniques for quickly searching for and navigating through media content. At this point, an automatic segmentation of the videos into thematically coherent units can be beneficial. Like in a book, the topics in an educational video are often structured hierarchically. There are larger topics, which in turn are divided into different subtopics. We thus propose a metric that considers the hierarchical levels in the reference segmentation when evaluating segmentation algorithms. In addition, we propose a multilingual, unsupervised topic segmentation approach and evaluate it on three datasets with English, Portuguese and German lecture videos. We achieve WindowDiff scores of up to 0.373 and show the usefulness of our hierarchical metric. Y1 - 2023 U6 - https://doi.org/10.21437/slate.2023-27 ER - TY - CHAP A1 - Grandeit, Philipp A1 - Haberkern, Carolyn A1 - Lang, Maximiliane A1 - Albrecht, Jens A1 - Lehmann, Robert T1 - Using BERT for Qualitative Content Analysis in Psychosocial Online Counseling T2 - Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science N2 - Qualitative content analysis is a systematic method commonly used in the social sciences to analyze textual data from interviews or online discussions. However, this method usually requires high expertise and manual effort because human coders need to read, interpret, and manually annotate text passages. This is especially true if the system of categories used for annotation is complex and semantically rich. Therefore, qualitative content analysis could benefit greatly from automated coding. In this work, we investigate the usage of machine learning-based text classification models for automatic coding in the area of psycho-social online counseling. We developed a system of over 50 categories to analyze counseling conversations, labeled over 10.000 text passages manually, and evaluated the performance of different machine learning-based classifiers against human coders. Y1 - 2020 U6 - https://doi.org/10.18653/v1/2020.nlpcss-1.2 SP - 11 EP - 23 PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Wagner, Dominik A1 - Churchill, Alexander A1 - Sigtia, Siddharth A1 - Georgiou, Panayiotis A1 - Mirsamadi, Matt A1 - Mishra, Aarshee A1 - Marchi, Erik T1 - A Multimodal Approach to Device-Directed Speech Detection with Large Language Models N2 - Interactions with virtual assistants typically start with a predefined trigger phrase followed by the user command. To make interactions with the assistant more intuitive, we explore whether it is feasible to drop the requirement that users must begin each command with a trigger phrase. We explore this task in three ways: First, we train classifiers using only acoustic information obtained from the audio waveform. Second, we take the decoder outputs of an automatic speech recognition (ASR) system, such as 1-best hypotheses, as input features to a large language model (LLM). Finally, we explore a multimodal system that combines acoustic and lexical features, as well as ASR decoder signals in an LLM. Using multimodal information yields relative equal-error-rate improvements over text-only and audio-only models of up to 39% and 61%. Increasing the size of the LLM and training with low-rank adaption leads to further relative EER reductions of up to 18% on our dataset. KW - device-directed speech detection, large language model, multimodal, conditional generation Y1 - 2024 U6 - https://doi.org/10.1109/ICASSP48485.2024.10446224 SP - 10451 EP - 10455 PB - IEEE ER - TY - CHAP A1 - Kiesler, Natalie A1 - Scholz, Ingo A1 - Albrecht, Jens A1 - Stappert, Friedhelm A1 - Wienkop, Uwe T1 - Novice Learners of Programming and Generative AI - Prior Knowledge Matters T2 - Proceedings of the 24th Koli Calling International Conference on Computing Education Research N2 - With the broad availability of Generative AI (GenAI), introductory programming education is starting to change. At Nuremberg Tech, we observed the doubling of failure rates to approximately 50% in the first semester course “Procedural Programming” across students of all study programs. Due to these exam results in winter 2023/24, we conducted a pilot study to gather students’ use of GenAI tools, their exam results, and prior programming education and experience. The results imply significant differences of students’ use of GenAI tools depending on their prior programming education. We will therefore extend the investigation in winter term 2024/25. Y1 - 2024 U6 - https://doi.org/10.1145/3699538.3699580 SP - 1 EP - 2 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Faria, Arlo A1 - Janin, Adam A1 - Adkoli, Sidhi A1 - Riedhammer, Korbinian T1 - Toward Zero Oracle Word Error Rate on the Switchboard Benchmark T2 - Interspeech 2022 N2 - The “Switchboard benchmark” is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best among a list of alternatives. A phrase alternatives representation is compared to utterance-level N-best lists and word-level data structures; using dense lattices and adding out-of-vocabulary words, this achieves an oracle WER of 0.18%. KW - ASR evaluation KW - Switchboard benchmark KW - oracle word error rate KW - N-best lists KW - phrase alternatives Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10959 SP - 3973 EP - 3977 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI Workshops 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education KW - Learning Experience Platform KW - Open Source Software KW - Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 PB - Gesellschaft für Informatik e.V. ER - TY - CHAP A1 - Braun, Franziska A1 - Bayerl, Sebastian A1 - Hönig, Florian A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Infusing Acoustic Pause Context into Text-Based Dementia Assessment N2 - Speech pauses, alongside content and structure, offer a valuable and non-invasive biomarker for detecting dementia. This work investigates the use of pause-enriched transcripts in transformer-based language models to differentiate the cognitive states of subjects with no cognitive impairment, mild cognitive impairment, and Alzheimer’s dementia based on their speech from a clinical assessment. We address three binary classification tasks: Onset, monitoring, and dementia exclusion. The performance is evaluated through experiments on a German Verbal Fluency Test and a Picture Description Test, comparing the model’s effectiveness across different speech production contexts. Starting from a textual baseline, we investigate the effect of incorporation of pause information and acoustic context. We show the test should be chosen depending on the task, and similarly, lexical pause information and acoustic cross-attention contribute differently. KW - speech biomarkers KW - dementia assessment KW - neuropsychological tests KW - pathological speech Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2496 SN - 2958-1796 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Ranzenberger, Thomas A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalizing Large Sequence-to-Sequence Speech Foundation Models With Speaker Representations N2 - We present a method to personalize large transformer-based encoderdecoder speech foundation models without the need for changes in the underlying model structure or training from scratch. This is achieved by projecting speaker-specific information into the latent space of the transformer decoder via a small neural network and learning to process the speaker information along with domainspecific information via parameter-efficient finetuning. We use this method to improve the automatic speech recognition results of spoken academic German and English. Our approach yields average relative word error rate (WER) improvements of approximately 29% on German academic speech and 25% on English academic speech. It also translates well to conversational speech, achieving relative WER improvements of up to 36%, and demonstrates modest gains of up to 5% on read speech. Moreover, we observe that incorporating utterances from the recent past as personalization context yields the most significant overall improvements and that changes in voice characteristics resulting from prolonged speaking have a minimal effect on the personalization quality of academic lectures. Y1 - 2024 U6 - https://doi.org/10.1109/SLT61566.2024.10832252 ER - TY - JOUR A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias ED - Baumann, Ilja T1 - Generative adversarial networks for whispered to voiced speech conversion: a comparative study JF - International Journal of Speech Technology N2 - Generative Adversarial Networks (GANs) have demonstrated promising results as end-to-end models for whispered to voiced speech conversion. Leveraging non-autoregressive systems like GANs capable of performing conditional waveform generation eliminates the need for separate models to estimate voiced speech features, and leads to faster inference compared to autoregressive methods. This study aims to identify the optimal GAN architecture for the whispered to voiced speech conversion task by comparing six state-of-the-art models. Furthermore, we present a method for evaluating the preservation of speaker identity and local accent, using embeddings obtained from speaker- and language identification systems. Our experimental results show that building the speech conversion system based on the HiFi-GAN architecture yields the best objective evaluation scores, outperforming the baseline by ∼9% relative using frequency-weighted Signal-to-Noise Ratio and Log Likelihood Ratio, as well as by ∼29% relative using Root Mean Squared Error. In subjective tests, HiFi-GAN yielded a mean opinion score of 2.9, significantly outperforming the baseline with a score of 1.4. Furthermore, HiFi-GAN enhanced ASR performance and preserved speaker identity and accent, with correct language detection rates of up to ∼98%. KW - Speech conversion · Generative adversarial networks · Whispered speech · Voiced speech Y1 - 2024 U6 - https://doi.org/10.1007/s10772-024-10161-1 VL - 27 ER - TY - CHAP A1 - Seeberger, Philipp A1 - Wagner, Dominik A1 - Riedhammer, Korbinian T1 - Multimodal Multimedia Event Argument Extraction with Unified Template Filling T2 - Findings of the Association for Computational Linguistics: EMNLP 2024 N2 - With the advancement of multimedia technologies, news documents and user-generated content are often represented as multiple modalities, making Multimedia Event Extraction (MEE) an increasingly important challenge. However, recent MEE methods employ weak alignment strategies and data augmentation with simple classification models, which ignore the capabilities of natural language-formulated event templates for the challenging Event Argument Extraction (EAE) task. In this work, we focus on EAE and address this issue by introducing a unified template filling model that connects the textual and visual modalities via textual prompts. This approach enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics. Experiments on the M2E2 benchmark demonstrate the effectiveness of our approach. Our system surpasses the current SOTA on textual EAE by +7% F1, and performs generally better than the second-best systems for multimedia EAE. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.findings-emnlp.381 PB - Association for Computational Linguistics CY - Miami, Florida, USA ER - TY - CHAP A1 - Wagner, Dominik A1 - Lee, Seanie A1 - Baumann, Ilja A1 - Seeberger, Philipp A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Speculative Sampling for {GPU} Hardware Accelerators T2 - Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing N2 - In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.emnlp-main.370 PB - Association for Computational Linguistics CY - Miami, Florida, USA ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Glockner, Kevin A1 - Herygers, Aaricia A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Zakaria, Khabbab T1 - EXTENDING HANS: LARGE LANGUAGE MODELS FOR QUESTION ANSWERING, SUMMARIZATION, AND TOPIC SEGMENTATION IN AN ML-BASED LEARNING EXPERIENCE PLATFORM T2 - Elektronische Sprachsignalverarbeitung 2024, Tagungsband der 35. Konferenz, Regensburg, 6.-8. März 2024 N2 - Abstract: The use of chatbots based on large language models (LLMs) and their impact on society are influencing our learning experience platform Hochschul Assistenz-System (HAnS). HAnS uses machine learning (ML) methods to support students and lecturers in the online learning and teaching processes [1]. This paper introduces LLM-based features available in HAnS which are using the transcript of our improved Automatic Speech Recognition (ASR) pipeline with an average transcription duration of 45 seconds and an average word error rate (WER) of 6.66% on over 8 hours of audio data of 7 lecture videos. A LLM-based chatbot could be used to answer questions on the lecture content as the ASR transcript is provided as context. The summarization and topic segmentation uses the LLM to improve our learning experience platform. We generate multiple choice questions using the LLM and the ASR transcript as context during playback in a period of 3 minutes and display them in the HAnS frontend Y1 - 2024 SN - 978-3-95908-325-6 PB - TUPress CY - Dresden ER - TY - JOUR A1 - Pérez-Toro, Paula Andrea A1 - Vásquez-Correa, Juan Camilo A1 - Bocklet, Tobias A1 - Nöth, Elmar A1 - Orozco-Arroyave, Juan Rafael T1 - User State Modeling Based on the Arousal-Valence Plane: Applications in Customer Satisfaction and Health-Care JF - IEEE Transactions on Affective Computing N2 - The acoustic analysis helps to discriminate emotions according to non-verbal information, while linguistics aims to capture verbal information from written sources. Acoustic and linguistic analyses can be addressed for different applications, where information related to emotions, mood, or affect are involved. The Arousal-Valence plane is commonly used to model emotional states in a multidimensional space. This study proposes a methodology focused on modeling the user’s state based on the Arousal-Valence plane in different scenarios. Acoustic and linguistic information are used as input to feed different deep learning architectures mainly based on convolutional and recurrent neural networks, which are trained to model the Arousal-Valence plane. The proposed approach is used for the evaluation of customer satisfaction in call-centers and for health-care applications in the assessment of depression in Parkinson’s disease and the discrimination of Alzheimer’s disease. F-scores of up to 0.89 are obtained for customer satisfaction, of up to 0.82 for depression in Parkinson’s patients, and of up to 0.80 for Alzheimer’s patients. The proposed approach confirms that there is information embedded in the Arousal-Valence plane that can be used for different purposes. KW - Arousal-valence plane KW - acoustic KW - linguistic KW - customer satisfaction KW - alzheimer’s disease KW - depression Y1 - 2021 U6 - https://doi.org/10.1109/taffc.2021.3112543 SN - 1949-3045 VL - 14 IS - 2 SP - 1533 EP - 1546 PB - Institute of Electrical and Electronics Engineers (IEEE) ER - TY - CHAP A1 - Perez-Toro, P. A. A1 - Vasquez-Correa, J. C. A1 - Arias-Vergara, T. A1 - Klumpp, P. A1 - Sierra-Castrillon, M. A1 - Roldan-Lopez, M. E. A1 - Aguillon, D. A1 - Hincapie-Henao, L. A1 - Tobon-Quintero, C. A. A1 - Bocklet, Tobias A1 - Schuster, M. A1 - Orozco-Arroyave, J. R. A1 - Nöth, E. T1 - Acoustic and Linguistic Analyses to Assess Early-Onset and Genetic Alzheimer’s Disease T2 - ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - The PSEN1-E280A or Paisa mutation is responsible for most of Early-Onset Alzheimer’s (EOA) disease cases in Colombia. It affects a large kindred of over 5000 members that present the same phenotype. The most common symptoms are related to language disorders, where speech fluency is also affected due to the difficulty to access semantic information intentionally. This study proposes the use of acoustic and linguistic methods to extract features from speech recordings and their transcriptions to discriminate people with conditions related to the Paisa mutation. We consider state-of-the-art word-embedding methods like Word2Vec and Bidirectional Encoder Representations from Transformer to process the transcripts. The speech signals are modeled by using traditional acoustic features and speaker embeddings. To the best of our knowledge, this is the first study focused on evaluating genetic Alzheimer’s and EOA using acoustics and linguistics. KW - PSEN1–E280A KW - Alzheimer’s Disease KW - Acoustic Analysis KW - Linguistic Analysis Y1 - 2021 SN - 978-1-7281-7605-5 U6 - https://doi.org/10.1109/ICASSP39728.2021.9414009 SP - 8338 EP - 8342 PB - IEEE ER - TY - CHAP A1 - Scheuerer, Ralph A1 - Haderlein, Tino A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Applying X-Vectors on Pathological Speech After Larynx Removal T2 - 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) N2 - Speaker embeddings extracted from time delayed neural networks (TDNNs) contributed to major recent advancements in speaker recognition and verification. We use an X-Vector system trained on augmented VoxCeleb1 and VoxCeleb2 data to obtain embeddings for pathological speech after total or partial larynx removal. We show that our model is able to effectively distinguish and visualize patient groups when generating embeddings. We further compare various regression models on the task of automatically predicting different perceptual ratings by speech therapists (intelligibility, vocal effort, and overall quality) based on the extracted speaker embeddings. For both patient groups we show Pearson correlations in the range of +0.8; we find that Random Forest and Support Vector Regression produce scores that best resemble the experts' assessments. KW - laryngectomy KW - intelligibility KW - pathological speech KW - x-vectors Y1 - 2021 SN - 978-1-6654-3739-4 U6 - https://doi.org/10.1109/asru51503.2021.9688278 VL - 2021 SP - 1079 EP - 1086 PB - IEEE ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas H. A1 - Bayerl, Sebastian P. A1 - Auerbach, Thomas A1 - Bocklet, Tobias T1 - An Acoustical Machine Learning Approach to Determine Abrasive Belt Wear of Wide Belt Sanders T2 - 2022 IEEE Sensors N2 - This paper describes a machine learning approach to determine the abrasive belt wear of wide belt sanders used in industrial processes based on acoustic data, regardless of the sanding process-related parameters, Feed speed, Grit Size, and Type of material. Our approach utilizes Decision Tree, Random Forest, k-nearest Neighbors, and Neural network Classifiers to detect the belt wear from Spectrograms, Mel Spectrograms, MFCC, IMFCC, and LFCC, yielding an accuracy of up to 86.1% on five levels of belt wear. A 96% accuracy could be achieved with different Decision Tree Classifiers specialized in different sanding parameter configurations. The classifiers could also determine with an accuracy of 97% if the machine is currently sanding or is idle and with an accuracy of 98.4% and 98.8% detect the sanding parameters Feed speed and Grit Size. We can show that low-dimensional mappings of high-dimensional features can be used to visualize belt wear and sanding parameters meaningfully. KW - Acoustic sensors KW - Abrasive belt wear KW - Tool wear KW - Machine learning KW - Industrial process KW - Wide belt sanding machines Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/SENSORS52175.2022.9967324 VL - 2022 PB - IEEE ER - TY - CHAP A1 - Schmitt, Thomas H. A1 - Bundscherer, Maximilian A1 - Drechsel, Ralf A1 - Bocklet, Tobias T1 - Machine learning based optimization of a ceramic bushing manufacturing process T2 - 2022 IEEE Sensors N2 - Machine learning (ML) has shown great promise in a variety of domains in recent years. ML models are known to require large amounts of labeled training data, keeping small to medium-sized business from utilizing them. This paper presents ML based approach to optimize a ceramic bushing manufac-turing process, by predicting the employed press-fit process as a function of press punch position. Accurate predictions would ensure optimal process configuration, guaranteeing quality and reducing waste. Models are trained in a supervised manner to predict the press-fit process and the ceramic defect probabilities as functions of press punch position. We were able to predict the press-fit process with a mean correlation of 0.996 and assess whether the process would damage the ceramic with a mean precision of 96.7%. Our results exemplify how ML can be used to predict and optimize highly specialised processes even with small datasets. KW - Manufacturing KW - machine learning KW - optimization KW - ceramic bushing Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/sensors52175.2022.9967124 PB - IEEE ER - TY - CHAP A1 - Klumpp, P. A1 - Bocklet, Tobias A1 - Arias-Vergara, T. A1 - Vásquez-Correa, J. C. A1 - Pérez-Toro, P.A. A1 - Bayerl, Sebastian P. A1 - Orozco-Arroyave, J. R. A1 - Nöth, Elmar T1 - The Phonetic Footprint of Covid-19? T2 - Interspeech 2021 N2 - Against the background of the ongoing pandemic, this year’s Computational Paralinguistics Challenge featured a classification problem to detect Covid-19 from speech recordings. The presented approach is based on a phonetic analysis of speech samples, thus it enabled us not only to discriminate between Covid and non-Covid samples, but also to better understand how the condition influenced an individual’s speech signal. Our deep acoustic model was trained with datasets collected exclusively from healthy speakers. It served as a tool for segmentation and feature extraction on the samples from the challenge dataset. Distinct patterns were found in the embeddings of phonetic classes that have their place of articulation deep inside the vocal tract. We observed profound differences in classification results for development and test splits, similar to the baseline method. We concluded that, based on our phonetic findings, it was safe to assume that our classifier was able to reliably detect a pathological condition located in the respiratory tract. However, we found no evidence to claim that the system was able to discriminate between Covid-19 and other respiratory diseases. KW - COVID-19 Y1 - 2021 U6 - https://doi.org/10.21437/Interspeech.2021-1488 SN - 2958-1796 SP - 441 EP - 445 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias T1 - Nonwords Pronunciation Classification in Language Development Tests for Preschool Children T2 - Interspeech 2022 N2 - This work aims to automatically evaluate whether the language development of children is age-appropriate. Validated speech and language tests are used for this purpose to test the auditory memory. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures: Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR acoustic model). Each of the approaches provides input for VGG-like 5-layer CNN classifiers. We also examine the adaptation per nonword. The evaluation of the proposed systems was performed using recordings from different kindergartens of spoken nonwords. ECAPA-TDNN and low-level FFT features do not explicitly model phonetic information; wav2vec2.0 is trained on grapheme labels, our ASR acoustic model features contain (sub-)phonetic information. We found that the more granular the phonetic modeling is, the higher are the achieved recognition rates. The best system trained on ASR acoustic model features with VTLN achieved an accuracy of 89.4% and an area under the ROC (Receiver Operating Characteristic) curve (AUC) of 0.923. This corresponds to an improvement in accuracy of 20.2% and AUC of 0.309 relative compared to the FFT-baseline. Y1 - 2022 U6 - https://doi.org/10.21437/interspeech.2022-10777 SN - 2958-1796 VL - 2022 SP - 3643 EP - 3647 PB - ISCA ER - TY - JOUR A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Detecting Vocal Fatigue with Neural Embeddings JF - Journal of Voice N2 - Vocal fatigue refers to the feeling of tiredness and weakness of voice due to extended utilization. This paper investigates the effectiveness of neural embeddings for the detection of vocal fatigue. We compare x-vectors, ECAPA-TDNN, and wav2vec 2.0 embeddings on a corpus of academic spoken English. Low-dimensional mappings of the data reveal that neural embeddings capture information about the change in vocal characteristics of a speaker during prolonged voice usage. We show that vocal fatigue can be reliably predicted using all three types of neural embeddings after 40 minutes of continuous speaking when temporal smoothing and normalization are applied to the extracted embeddings. We employ support vector machines for classification and achieve accuracy scores of 81% using x-vectors, 85% using ECAPA-TDNN embeddings, and 82% using wav2vec 2.0 embeddings as input features. We obtain an accuracy score of 76%, when the trained system is applied to a different speaker and recording environment without any adaptation. KW - Vocal fatigue KW - Neural embeddings KW - Visualization KW - Detection Y1 - 2023 U6 - https://doi.org/10.1016/j.jvoice.2023.01.012 SN - 0892-1997 PB - Elsevier BV ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - The Influence of Dataset Partitioning on Dysfluency Detection Systems T2 - Text, Speech, and Dialogue N2 - This paper empirically investigates the influence of different data splits and splitting strategies on the performance of dysfluency detection systems. For this, we perform experiments using wav2vec 2.0 models with a classification head as well as support vector machines (SVM) in conjunction with the features extracted from the wav2vec 2.0 model to detect dysfluencies. We train and evaluate the systems with different non-speaker-exclusive and speaker-exclusive splits of the Stuttering Events in Podcasts (SEP-28k) dataset to shed some light on the variability of results w.r.t. to the partition method used. Furthermore, we show that the SEP-28k dataset is dominated by only a few speakers, making it difficult to evaluate. To remedy this problem, we created SEP-28k-Extended (SEP-28k-E), containing semi-automatically generated speaker and gender information for the SEP-28k corpus, and suggest different data splits, each useful for evaluating other aspects of methods for dysfluency detection. KW - Computer Science KW - Informatics Y1 - 2022 SN - 9783031162695 U6 - https://doi.org/10.1007/978-3-031-16270-1_35 SN - 0302-9743 SP - 423 EP - 436 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Maruri, Hector A. Cordourier A1 - Bocklet, Tobias T1 - Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech T2 - 2022 IEEE Spoken Language Technology Workshop (SLT) N2 - This work adapts two recent architectures of generative models and evaluates their effectiveness for the conversion of whispered speech to normal speech. We incorporate the normal target speech into the training criterion of vector-quantized variational autoencoders (VQ-VAEs) and Mel-GANs, thereby conditioning the systems to recover voiced speech from whispered inputs. Objective and subjective quality measures indicate that both VQ-VAEs and MelGANs can be modified to perform the conversion task. We find that the proposed approaches significantly improve the Mel cepstral distortion (MCD) metric by at least 25% relative to a Disco-GAN baseline. Subjective listening tests suggest that the MelGAN-based system significantly improves naturalness, intelligibility, and voicing compared to the whispered input speech. A novel evaluation measure based on differences between latent speech representations also indicates that our MelGAN-based approach yields improvements relative to the baseline. KW - whispered speech KW - speech conversion KW - VAE KW - GAN KW - generative models Y1 - 2023 SN - 979-8-3503-9690-4 U6 - https://doi.org/10.1109/SLT54892.2023.10022796 SP - 943 EP - 948 PB - IEEE ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas A1 - Bocklet, Tobias T1 - Machine Learning in Industrial Quality Control of Glass Bottle Prints N2 - In industrial manufacturing of glass bottles, quality control of bottle prints is necessary as numerous factors can negatively affect the printing process. Even minor defects in the bottle prints must be detected despite reflections in the glass or manufacturing-related deviations. In cooperation with our medium-sized industrial partner, two ML-based approaches for quality control of these bottle prints were developed and evaluated, which can also be used in this challenging scenario. Our first approach utilized different filters to supress reflections (e.g. Sobel or Canny) and image quality metrics for image comparison (e.g. MSE or SSIM) as features for different supervised classification models (e.g. SVM or k-Neighbors), which resulted in an accuracy of 84%. The images were aligned based on the ORB algorithm, which allowed us to estimate the rotations of the prints, which may serve as an indicator for anomalies in the manufacturing process. In our second approach, we fine-tuned different pre-trained CNN models (e.g. ResNet or VGG) for binary classification, which resulted in an accuracy of 87%. Utilizing Grad-Cam on our fine-tuned ResNet-34, we were able to localize and visualize frequently defective bottle print regions. This method allowed us to provide insights that could be used to optimize the actual manufacturing process. This paper also describes our general approach and the challenges we encountered in practice with data collection during ongoing production, unsupervised preselection, and labeling. KW - Machine Learning, Quality Control, Industrial Manufacturing, Glass Bottle Printsuring Optimization, Glass Printing Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2409.20132 ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Large Language Models for Dysfluency Detection in Stuttered Speech N2 - Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task. KW - dysfluency detection, stuttering, large language models, wav2vec 2.0, Whisper, pathological speech Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2406.11025 ER - TY - RPRT A1 - Schmitt, Thomas A1 - Bundscherer, Maximilian A1 - Bocklet, Tobias T1 - Semmeldetector: Application of Machine Learning in Commercial Bakeries N2 - The Semmeldetector, is a machine learning application that utilizes object detection models to detect, classify and count baked goods in images. Our application allows commercial bakers to track unsold baked goods, which allows them to optimize production and increase resource efficiency. We compiled a dataset comprising 1151 images that distinguishes between 18 different types of baked goods to train our detection models. To facilitate model training, we used a Copy-Paste augmentation pipeline to expand our dataset. We trained the state-of-the-art object detection model YOLOv8 on our detection task. We tested the impact of different training data, model scale, and online image augmentation pipelines on model performance. Our overall best performing model, achieved an AP@0.5 of 89.1% on our test set. Based on our results, we conclude that machine learning can be a valuable tool even for unforeseen industries like bakeries, even with very limited datasets. KW - machine learning, object detection, YOLOv8, image composition, baked goods, food inspection, industrial automation Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2406.04050 ER - TY - JOUR A1 - Kammerbauer, Roland A1 - Schmitt, Thomas A1 - Bocklet, Tobias T1 - Segmenting Wood Rot using Computer Vision Models N2 - In the woodworking industry, a huge amount of effort has to be invested into the initial quality assessment of the raw material. In this study we present an AI model to detect, quantify and localize defects on wooden logs. This model aims to both automate the quality control process and provide a more consistent and reliable quality assessment. For this purpose a dataset of 1424 sample images of wood logs is created. A total of 5 annotators possessing different levels of expertise is involved in dataset creation. An inter-annotator agreement analysis is conducted to analyze the impact of expertise on the annotation task and to highlight subjective differences in annotator judgement. We explore, train and fine-tune the state-of-the-art InternImage and ONE-PEACE architectures for semantic segmentation. The best model created achieves an average IoU of 0.71, and shows detection and quantification capabilities close to the human annotators. KW - machine learning, image segmentation, semantic segmentation, InternImage, ONEPEACE, lumbering, industrial quality control, industrial automation Y1 - 2024 ER - TY - CHAP A1 - Baumann, Ilja A1 - Unger, Nicole A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Automatic Evaluation of a Sentence Memory Test for Preschool Children N2 - Assessment of memory capabilities in preschool-aged children is crucial for early detection of potential speech development impairments or delays. We present an approach for the automatic evaluation of a standardized sentence memory test specifically for preschool children. Our methodology leverages automatic transcription of recited sentences and evaluation based on natural language processing techniques. We demonstrate the effectiveness of our approach on a dataset comprised of recited sentences from preschool-aged children, incorporating ratings of semantic and syntactic correctness. The best performing systems achieve an F1 score of 91.7% for semantic correctness and 86.1% for syntactic correctness using automatic transcripts. Our results showcase the potential of automated evaluation systems in providing reliable and efficient assessments of memory capabilities in early childhood, facilitating timely interventions and support for children with language development needs. KW - speech development, children’s speech, automatic assessment Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2125 SP - 5158 EP - 5162 ER - TY - JOUR A1 - Schmitt, Thomas A1 - Bundscherer, Maximilian A1 - Bocklet, Tobias T1 - Training a Computer Vision Model for Commercial Bakeries with Primarily Synthetic Images N2 - In the food industry, reprocessing returned product is a vital step to increase resource efficiency. [SBB23] presented an AI application that automates the tracking of returned bread buns. We extend their work by creating an expanded dataset comprising 2432 images and a wider range of baked goods. To increase model robustness, we use generative models pix2pix and CycleGAN to create synthetic images. We train state-of-the-art object detection model YOLOv9 and YOLOv8 on our detection task. Our overall best-performing model achieved an average precision AP@0.5 of 90.3% on our test set. KW - machine learning, object detection, YOLOv9, image composition, baked goods, food industry, industrial automation Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2409.20122 ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Schuster, Maria A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Towards Self-Attention Understanding for Automatic Articulatory Processes Analysis in Cleft Lip and Palate Speech N2 - Cleft lip and palate (CLP) speech presents unique challenges for automatic phoneme analysis due to its distinct acoustic characteristics and articulatory anomalies. We perform phoneme analysis in CLP speech using a pre-trained wav2vec 2.0 model with a multi-head self-attention classification module to capture long-range dependencies within the speech signal, thereby enabling better contextual understanding of phoneme sequences. We demonstrate the effectiveness of our approach in the classification of various articulatory processes in CLP speech. Furthermore, we investigate the interpretability of self-attention to gain insights into the model’s understanding of CLP speech characteristics. Our findings highlight the potential of the selfattention mechanisms for improving automatic phoneme analysis in CLP speech, paving the way for enhanced diagnostics, adding interpretability for therapists and affected patients. KW - pathologic speech, cleft lip and palate, children’s speech, automatic assessment Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2134 SP - 2430 EP - 2434 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias ED - Pesak, Krisztina T1 - Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models N2 - This paper explores the improvement of post-training quantization (PTQ) after knowledge distillation in the Whisper speech foundation model family. We address the challenge of outliers in weights and activation tensors, known to impede quantization quality in transformer-based language and vision models. Extending this observation to Whisper, we demonstrate that these outliers are also present when transformer-based models are trained to perform automatic speech recognition, necessitating mitigation strategies for PTQ. We show that outliers can be reduced by a recently proposed gating mechanism in the attention blocks of the student model, enabling effective 8-bit quantization, and lower word error rates compared to student models without the gating mechanism in place. KW - post-training quantization, Whisper, gated attention, outliers Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2105 SP - 4623 EP - 4627 ER - TY - JOUR A1 - Freier, Carolin A1 - Bocklet, Tobias A1 - Helten, Anne-Kathrin A1 - Hoffmann, Franziska A1 - Hunger, Marianne A1 - Kovács, László A1 - Richter, Florian A1 - Riedhammer, Korbinian A1 - Schmohl, Tobias A1 - Simon, Claudia T1 - Wie kann videogestütztes Lernen die Erwartungen Studierender und Dozierender erfüllen? JF - Soziale Passagen N2 - ZusammenfassungIm BMBF-Verbundprojekt HAnS entwickeln und implementieren neun Hochschulen sowie drei hochschulübergreifende Einrichtungen ein intelligentes Hochschul-Assistenz-System als Open-Source-Lösung. Videobasierte Lehrmaterialien werden verschriftlicht und durch eine Indexierung Stichwortsuchen ermöglicht; geplant ist, über einen KI-Tutor automatisiert Übungsaufgaben zu generieren. Studierende sollen so in ihrem Selbststudium digital unterstützt werden. Die technische Entwicklung wird interdisziplinär – auch sozialwissenschaftlich und pädagogisch – begleitet und in einem iterativen Vorgehen evidenzbasiert entsprechend Design-Based-Research angepasst. Wissen und Wertesystem der Anwender*innen, Didaktik, Ethik, Akzeptanz und Datenschutz werden dabei im Entwicklungsprozess einbezogen. KW - Hochschulforschung · Intelligente Assistenzsysteme · Evaluation · Video-Lernen · Adressat*innen-zentrierte Technikentwicklung Y1 - 2023 U6 - https://doi.org/10.1007/s12592-023-00478-0 SN - 1867-0180 VL - 15 IS - 2 SP - 631 EP - 635 PB - Springer VS ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education; Learning Experience Platform; Open Source Software; Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 SN - 2944-7682 PB - Gesellschaft für Informatik e.V. CY - Bonn ER - TY - JOUR A1 - Escobar-Grisales, Daniel A1 - Ríos-Urrego, Cristian-David A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Garcia, Adolfo A1 - Orozco-Arroyave, Juan rafael T1 - It’s Time to Take Action: Acoustic Modeling of Motor Verbs to Detect Parkinson’s Disease N2 - Pre-trained models generate speech representations that are used in different tasks, including the automatic detection of Parkinson’s disease (PD). Although these models can yield high accuracy, their interpretation is still challenging. This paper used a pre-trained Wav2vec 2.0 model to represent speech frames of 25ms length and perform a frame-by-frame discrimination between PD patients and healthy control (HC) subjects. This fine granularity prediction enabled us to identify specific linguistic segments with high discrimination capability. Speech representations of all produced verbs were compared w.r.t. nouns and the first ones yielded higher accuracies. To gaina deeper understanding of this pattern, representations of motor and non-motor verbs were compared and the first ones yielded better results, with accuracies of around 83% in an independent test set. These findings support well-established neurocognitive models about action-related language highlighted as key drivers of PD. Index Terms: computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease KW - computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2205 ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Wagner, Dominik A1 - Noeth, Elmar A1 - Riedhammer, Korbinian T1 - Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0 T2 - Interspeech 2022 N2 - Stuttering is a varied speech disorder that harms an individual’s communication ability. Persons who stutter (PWS) often use speech therapy to cope with their condition. Improving speech recognition systems for people with such non-typical speech or tracking the effectiveness of speech therapy would require systems that can detect dysfluencies while at the same time being able to detect speech techniques acquired in therapy. This paper shows that fine-tuning wav2vec 2.0 [1] for the classification of stuttering on a sizeable English corpus containing stuttered speech, in conjunction with multi-task learning, boosts the effectiveness of the general-purpose wav2vec 2.0 features for detecting stuttering in speech; both within and across languages. We evaluate our method on FluencyBank , [2] and the German therapy-centric Kassel State of Fluency (KSoF) [3] dataset by training Support Vector Machine classifiers using features extracted from the finetuned models for six different stuttering-related event types: blocks, prolongations, sound repetitions, word repetitions, interjections, and – specific to therapy – speech modifications. Using embeddings from the fine-tuned models leads to relative classification performance gains up to 27% w.r.t. F1-score. KW - stuttering KW - pathological speech KW - computational paralinguistics Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10908 SP - 2868 EP - 2872 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Roccabruna, Gabriel A1 - Chowdhury, Shammur Absar A1 - Ciulli, Tommaso A1 - Danieli, Morena A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - What can Speech and Language Tell us About the Working Alliance in Psychotherapy T2 - Interspeech 2022 N2 - We are interested in the problem of conversational analysis and its application to the health domain. Cognitive Behavioral Therapy is a structured approach in psychotherapy, allowing the therapist to help the patient to identify and modify the malicious thoughts, behavior, or actions. This cooperative effort can be evaluated using the Working Alliance Inventory Observer-rated Shortened – a 12 items inventory covering task, goal, and relationship – which has a relevant influence on therapeutic outcomes. In this work, we investigate the relation between this alliance inventory and the spoken conversations (sessions) between the patient and the psychotherapist. We have delivered eight weeks of e-therapy, collected their audio and video call sessions, and manually transcribed them. The spoken conversations have been annotated and evaluated with WAI ratings by professional therapists. We have investigated speech and language features and their association with WAI items. The feature types include turn dynamics, lexical entrainment, and conversational descriptors extracted from the speech and language signals. Our findings provide strong evidence that a subset of these features are strong indicators of working alliance. To the best of our knowledge, this is the first and a novel study to exploit speech and language for characterising working alliance. KW - conversational analysis KW - working alliance KW - psychotherapy Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-347 SP - 2443 EP - 2447 PB - ISCA CY - ISCA ER - TY - BOOK A1 - Axenie, Cristian A1 - Bauer, Roman A1 - López Corona, Oliver A1 - West, Jeffrey T1 - Applied Antifragility in Natural Systems BT - From Principles to Applications N2 - As coined in the book of Nassim Taleb, antifragility is a property of a system to gain from uncertainty, randomness, and volatility, opposite to what fragility would incur. An antifragile system’s response to external perturbations is beyond robust, such that small stressors can strengthen the future response of the system by adding a strong anticipation component. Such principles are already well suited for describing behaviors in natural systems but also in approaching therapy designs and eco-system modelling and eco-system analysis. The purpose of this book is to build a foundational knowledge base by applying antifragile system design, analysis, and development in natural systems, including biomedicine, neuroscience, and ecology as main fields. We are interested in formalizing principles and an apparatus that turns the basic concept of antifragility into a tool for designing and building closed-loop systems that behave beyond robust in the face of uncertainty when characterizing and intervening in biomedical and ecological (eco)systems. The book introduces the framework of applied antifragility and possible paths to build systems that gain from uncertainty. We draw from the body of literature on natural systems (e.g. cancer therapy, antibiotics, neuroscience, and agricultural pest management) in an attempt to unify the scales of antifragility in one framework. The work of the Applied Antifragility Group in oncology, neuroscience, and ecology led by the authors provides a good overview on the current research status. T2 - with a foreword from Nassim Taleb Y1 - 2025 SN - 9783031903908 U6 - https://doi.org/10.1007/978-3-031-90391-5 SN - 2191-5768 PB - Springer Nature Switzerland CY - Cham ER - TY - BOOK A1 - Axenie, Cristian A1 - Akbarzadeh, Meisam A1 - Makridis, Michail A. A1 - Saveriano, Matteo A1 - Stancu, Alexandru T1 - Applied Antifragility in Technical Systems BT - From Principles to Applications N2 - The book purpose is to build a foundational knowledge base by applying antifragile system design, analysis, and development in technical systems, with a focus on traffic engineering, robotics, and control engineering. The authors are interested in formalizing principles and an apparatus that turns the basic concept of antifragility into a tool for designing and building closed-loop technical systems that behave beyond robust in the face of uncertainty. As coined in the book of Nassim Taleb, antifragility is a property of a system to gain from uncertainty, randomness, and volatility, opposite to what fragility would incur. An antifragile system’s response to external perturbations is beyond robust, such that small stressors can strengthen the future response of the system by adding a strong anticipation component. The work of the Applied Antifragility Group in traffic control and robotics, led by the authors, provides a good overview on the current research status. T2 - with a foreword from Nassim Taleb Y1 - 2025 SN - 9783031904240 U6 - https://doi.org/10.1007/978-3-031-90425-7 SN - 2191-5768 PB - Springer Nature Switzerland CY - Cham ER - TY - CHAP A1 - Wagner, Dominik A1 - Churchill, Alexander A1 - Sigtia, Siddharth A1 - Marchi, Erik T1 - SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline. KW - multi-task KW - multimodal KW - virtual assistant KW - large language KW - model KW - low-rank adaptation Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10890139 SN - 2379-190X SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Lee, Seanie A1 - Seong, Haebin A1 - Bok Lee, Dong A1 - Kang, Minki A1 - Chen, Xiaoyin A1 - Wagner, Dominik A1 - Bengio, Joshua A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models N2 - Safety guard models that detect malicious queries aimed at large language models(LLMs) are essential for ensuring the secure and responsible deployment of LLMs in real-world applications. However, deploying existing safety guard models with billions of parameters alongside LLMs on mobile devices is impractical due to substantial memory requirements and latency. To reduce this cost, we distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. Due to the limited diversity of harmful instructions in the existing labeled dataset, naively distilled models tend to underperform compared to larger models. To bridge the gap between small and large models, we propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Given a prompt such as, “Make a single harmful instruction prompt that would elicit offensive content”, we add an affirmative prefix (e.g., “I have an idea for a prompt:”) to the LLM’s response. This encourages the LLM to continue generating the rest of the response, leading to sampling harmful instructions. Another LLM generates a response to the harmful instruction, and the teacher model labels the instruction-response pair. We empirically show that our HarmAug outperforms other relevant baselines. Moreover, a 435-millionparameter safety guard model trained with HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost. Our code, safety guard model, and synthetic dataset are publicly available. Y1 - 2025 U6 - https://doi.org/10.48550/arXiv.2410.01524 SP - 1 EP - 20 PB - ICLR ER - TY - CHAP A1 - Braun, Franziska A1 - Witzl, Christopher A1 - Erzigkeit, Andreas A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Pitfalls and Limits in Automatic Dementia Assessment T2 - Interspeech 2025 N2 - Current work on speech-based dementia assessment focuses on either feature extraction to predict assessment scales, or on the automation of existing test procedures. Most research uses public data unquestioningly and rarely performs a detailed error analysis, focusing primarily on numerical performance. We perform an in-depth analysis of an automated standardized dementia assessment, the Syndrom-Kurz-Test. We find that while there is a high overall correlation with human annotators, due to certain artifacts, we observe high correlations for the severely impaired individuals, which is less true for the healthy or mildly impaired ones. Speech production decreases with cognitive decline, leading to overoptimistic correlations when test scoring relies on word naming. Depending on the test design, fallback handling introduces further biases that favor certain groups. These pitfalls remain independent of group distributions in datasets and require differentiated analysis of target groups. KW - dementia screening KW - neuropsychological tests KW - pathological speech Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2280 SN - 2958-1796 SP - 5663 EP - 5667 PB - ISCA ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas A1 - Baumann, Ilja A1 - Bocklet, Tobias T1 - Digital Operating Mode Classification of Real-World Amateur Radio Transmissions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - This study presents an ML approach for classifying digital radio operating modes evaluated on real-world transmissions. We generated 98 different parameterized radio signals from 17 digital operating modes, transmitted each of them on the 70 cm (UHF) amateur radio band, and recorded our transmissions with two different architectures of SDR receivers. Three lightweight ML models were trained exclusively on spectrograms of limited non-transmitted signals with random characters as payloads. This training involved an online data augmentation pipeline to simulate various radio channel impairments. Our best model, EfficientNetB0, achieved an accuracy of 93.80% across the 17 operating modes and 85.47% across all 98 parameterized radio signals, evaluated on our real-world transmissions with Wikipedia articles as payloads. Furthermore, we analyzed the impact of varying signal durations & the number of FFT bins on classification, assessed the effectiveness of our simulated channel impairments, and tested our models across multiple simulated SNRs. KW - Automatic Modulation Classification KW - Amateur KW - Radio KW - Spectrum Monitoring KW - Cognitive Radio KW - Machine Learning Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889837 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Self-supervised Training with BEST-RQ for Speech Recognition T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multicodebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on testother using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training. KW - self-supervised learning KW - speech recognition KW - conformer KW - BEST-RQ Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889362 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Freisinger, Steffen A1 - Seeberger, Philipp A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation T2 - Interspeech 2025 N2 - Segmenting speech transcripts into thematic sections benefits both downstream processing and users who depend on written text for accessibility. We introduce a novel approach to hierarchical topic segmentation in transcripts, generating multi-level tables of contents that capture both topic and subtopic boundaries. We compare zero-shot prompting and LoRA fine-tuning on large language models, while also exploring the integration of high-level speech pause features. Evaluations on English meeting recordings and multilingual lecture transcripts (Portuguese, German) show significant improvements over established topic segmentation baselines. Additionally, we adapt a common evaluation measure for multi-level segmentation, taking into account all hierarchical levels within one metric. KW - topic segmentation KW - spoken content segmentation KW - table of contents generation KW - hierarchical segmentation Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2792 SN - 2958-1796 SP - 276 EP - 280 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Lee, Seanie A1 - Bok Lee, Dong A1 - Wagner, Dominik A1 - Kang, Minki A1 - Seong, Haebin A1 - Bocklet, Tobias A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models, JF - Findings of the Association for Computational Linguistics: ACL 2025 N2 - Deploying large language models (LLMs) in real-world applications requires robust safety guard models to detect and block harmful user prompts. While large safety guard models achieve strong performance, their computational cost is substantial. To mitigate this, smaller distilled models are used, but they often underperform on “hard” examples where the larger model provides accurate predictions. We observe that many inputs can be reliablyhandled by the smaller model, while only a small fraction require the larger model’s capacity. Motivated by this, we propose SafeRoute, a binary router that distinguishes hard examples from easy ones. Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy compared to solely using the larger safety guard model. Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance, outperforming relevant baselines. Y1 - 2025 UR - https://aclanthology.org/2025.findings-acl.105.pdf PB - Association for Computational Linguistics CY - Vienna, Austria ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Engert, Natalie A1 - Lee, Seanie A1 - Nöth, Elmar A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ∼23% and ∼22%, respectively. Further improvements (∼5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ∼7% relative WER improvement over personalized fine-tuning alone. KW - speech recognition KW - dysarthric speech KW - personalization KW - low-rank adaptation KW - synthetic data Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2155 SN - 2958-1796 SP - 3294 EP - 3298 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Pathology-Aware Speech Encoding and Data Augmentation for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - Automatic speech recognition (ASR) for pathologic speech remains a major challenge due to high variability in articulation, phonation, and prosody distortions. In this work, we propose a pathology-aware speech encoder based on BEST-RQ pre-training, which incorporates 46k hours of speech, including pathologic and atypical speech. We continue pre-training for domain adaptation and experiment with etiology-specific codebooks. We achieve a 13.2% relative word error rate (WER) improvement using the pathology-aware speech encoder with etiology-specific continued pre-training. Additionally, we examine the impact of incorporating synthetic and out-of-domain (OOD) data to further enhance ASR performance. Synthetic data reduces WER by up to 8.7%, while OOD data improves WER by 12.2%. Finally, we introduce a semantic similaritybased data augmentation technique to optimize data selection, achieving a WER improvement of up to 9.7% while minimizing the need for additional training data. KW - pathologic speech recognition KW - speech augmentation KW - self-supervised learning Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2724 SN - 2958-1796 SP - 3289 EP - 3293 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias T1 - Vocoder-Free Non-parallel Conversion of Whispered Speech With Masked Cycle-Consistent Generative Adversarial Networks N2 - Cycle-consistent generative adversarial networks have been widely used in non-parallel voice conversion (VC). Their ability to learn mappings between source and target features without relying on parallel training data eliminates the need for temporal alignments. However, most methods decouple the conversion of acoustic features from synthesizing the audio signal by using separate models for conversion and waveform synthesis. This work unifies conversion and synthesis into a single model, thereby eliminating the need for a separate vocoder. By leveraging cycle-consistent training and a self-supervised auxiliary training task, our model is able to efficiently generate converted high-quality raw audio waveforms. Subjective listening tests showed that our unified approach achieved improvements of up to 6.7% relative to the baseline in whispered VC. Mean opinion score predictions also yielded stable results in conventional VC (between 0.5% and 2.4% relative improvement). KW - voice conversion KW - generative adversarial networks KW - cycle-consistency KW - masking KW - whispered speech Y1 - 2025 SN - 978-3-032-02548-7 U6 - https://doi.org/10.1007/978-3-032-02548-7_20 PB - Springer CY - Cham ER - TY - JOUR A1 - Ranzenberger, Thomas A1 - Baumann, Ilja A1 - Bayerl, Sebastian A1 - Wagner, Dominik A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Evaluation of recognition errors of hybrid and transformer-based ASR systems in German video lectures JF - Studientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2025 - Book N2 - We analyze different errors in speech recognition systems, focusing on consecutive insertions and deletions, known as hallucinations and elisions in transformer-based end-to-end automatic speech recognition (ASR) systems. We compare errors from a TDNN-HMM, and whisper-based models on English and German spontaneous speech. Based on a human annotated subset of German lecture videos, we investigate whether these blocks of deletions affect the semantics of the utterance. Whisper performs best and preserves the meaning in 90% of the annotated error segments even containing consecutive deletions on this subset. We analyze the word error rate and do further analysis of errors using natural language processing to detect lemmatization errors, compound word errors, and out-of-vocabulary words. We discuss possible reasons and mitigations. Y1 - 2025 UR - https://www.essv.de/pdf/2025_101_108.pdf PB - ESSV 2025 CY - Halle, Deutschland ER - TY - CHAP A1 - Rudolph, Eric A1 - Engert, Natalie A1 - Albrecht, Jens T1 - An AI-Based Virtual Client for Educational Role-Playing in the Training of Online Counselors T2 - Proceedings of the 16th International Conference on Computer Supported Education - Volume 2 KW - Artificial Intelligence KW - Social Counseling Y1 - 2024 SN - 978-989-758-697-2 U6 - https://doi.org/10.5220/0012690700003693 SN - 2184-5026 SP - 108 EP - 117 PB - SCITEPRESS - Science and Technology Publications ER - TY - JOUR A1 - Lippert, Carolyn A1 - Rudolph, Eric A1 - Poltermann, Aleksandra A1 - Engert, Natalie A1 - Lehmann, Robert A1 - Albrecht, Jens T1 - Generative KI in der beraterischen Ausbildung BT - Der Einsatz eines*r virtuellen Klient*in als Übungstool für angehende Onlineberater*innen JF - e-beratungsjournal.net – Zeitschrift für Online-Beratung und computervermittelte Kommunikation N2 - Der Beitrag beleuchtet das Projekt "Virtuelle*r Klient*in (ViKl)" am Institut für E-Beratung der Technischen Hochschule Nürnberg Georg Simon Ohm, das sich mit der Nutzung eines KI-basierten Chatbots für die Simulation von Klient*innen in der Lehrveranstaltung "Querschnitt Onlineberatung" im Bachelor-Studiengang Soziale Arbeit befasst. Ziel ist es, praxisnahe Erfahrungen zu vermitteln und die Fähigkeiten der Studierenden im digitalen Beratungsgespräch zu verbessern. Durch den Einsatz verschiedener maschineller Lernarchitekturen werden passende Antworten generiert, die zum Gesprächsverlauf und zur psychosozialen Problemstellung passen. Die Nutzung einer umfangreichen Datenbasis aus dem Zertifikatskurs "Onlineberatung" sowie ein zielgruppenadäquates didaktisches Konzept tragen zur Qualität des*der ViKl bei. Die Diskussion umfasst auch zukünftige Entwicklungen in der Onlineberatung, die didaktische Gestaltung von KI-basierten Lehr-Lern-Formaten, sowie Herausforderungen und Vorteile dieser Ansätze. KW - Generative KI KW - Hochschule KW - Lehre KW - Computerunterstützte Kommunikation KW - Psychosoziale Beratung KW - Onlinekommunikation KW - digitale Kommunikation KW - Generative KI KW - Onlineberatung KW - Chatberatung Y1 - 2024 U6 - https://doi.org/10.48341/tcgc-st69 SN - 1816-7632 VL - 20 IS - 1 ER - TY - CHAP A1 - Steigerwald, Philipp A1 - Albrecht, Jens T1 - Enhancing Psychosocial Counselling with AI: A Multifaceted Support System for Professionals T2 - Frontiers of Artificial Intelligence, Ethics and Multidisciplinary Applications N2 - The expanding field of psychosocial online email counselling has created a demand for advanced tools to support counsellors in managing diverse client interactions. This paper introduces CAIA (Counsellor Artificial Intelligence Assistant), a comprehensive AI-driven system currently under active development, designed to condense and present information to counsellors in a structured manner. CAIA integrates several AI-based features, including Automatic Subject Generation, Case Summarisation, Timeline Management, Social Network Graph creation, and Case Classification. The features are designed to assist the counsellors, allowing them to engage more deeply with each case while ensuring that the client's needs remain paramount. Each feature presented in the paper addresses a specific problem in psychosocial online counselling, proposes a solution, discusses a technical implementation approach, and highlights the challenges encountered. Additionally, the paper discusses ethical considerations associated with an AI integration. Y1 - 2025 SN - 9789819679447 U6 - https://doi.org/10.1007/978-981-96-7945-4_35 SN - 2731-8125 SP - 597 EP - 611 PB - Springer Nature CY - Singapore ER - TY - CHAP A1 - Steigerwald, Philipp A1 - Albrecht, Jens T1 - Comparing Large Language Models for Automated Subject Line Generation in e-Mental Health: A Performance Study T2 - Proceedings of the 11th International Conference on Information and Communication Technologies for Ageing Well and e-Health N2 - Large Language Models (LLMs) have the potential to enhance e-mental health and psychosocial e-mail counselling by automating tasks such as generating concise and relevant subject lines for client communications. However, concerns regarding accuracy, reliability, data privacy and resource efficiency persist. This study investigates the performance of several LLMs in generating subject lines for e-mail threads, yielding a total of 253 generated subjects. Each subject line was assessed by six raters, including five counselling professionals and one AI system, using a three-category quality scale (Good, Fair, Poor). The results show that LLMs can generally produce concise subject lines considered helpful by experts. While GPT-4o and GPT-3.5 Turbo outperformed other models, their use is restricted in mental health settings due to data protection concerns, making the evaluation of open-source models crucial. Among open-source models, SauerkrautLM LLama 3 70b (4-bit) and SauerkrautLM Mixtral 8x7b (both 8-bit and 4-bit versions) delivered promising results with potential for further development. In contrast, models with lower parameter counts produced predominantly poor outputs. Y1 - 2025 U6 - https://doi.org/10.5220/0013294100003938 VL - 2025 SP - 70 EP - 77 PB - SCITEPRESS - Science and Technology Publications ER - TY - CHAP A1 - Steigerwald, Philipp A1 - Bienlein, Nico A1 - Burghardt, Jennifer A1 - Stieler, Mara A1 - Lehmann, Robert A1 - Albrecht, Jens T1 - CAIA in Practice: Field Evaluation of an AI-Assisted Support System for Text-Based Online Counselling T2 - 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI) N2 - Rising global demand for mental health support creates significant service delivery challenges, with asynchronous email counselling serving as a crucial low-threshold channel for accessing care. This paper presents CAIA, a co-designed AIbased tool suite that demonstrates responsible AI integration into counselling practice through seven LLM-driven functions enhanced by retrieval-augmented generation. A field evaluation involved 34 professional counsellors conducting authentic sessions with trained student counsellees (36 threads, 321 messages, 1,257 AI outputs). User behaviour analysis confirms substantial adoption, revealing that professional autonomy and information accuracy are decisive for sustained acceptance, with counsellors particularly valuing interpretive functionalities that provide new perspectives and stimulate professional reflection. KW - AI-Assisted Counselling KW - Email Counselling KW - Large Language Models KW - Retrieval-Augmented Generation KW - User KW - Adoption Study KW - Human-AI Collaboration Y1 - 2025 U6 - https://doi.org/10.1109/ICTAI66417.2025.00214 SP - 1476 EP - 1483 PB - IEEE ER - TY - CHAP A1 - Burghardt, Jennifer A1 - Stieler, Mara A1 - Steigerwald, Philipp A1 - Albrecht, Jens A1 - Lehmann, Robert T1 - KI-Assistenz in der psychosozialen Onlineberatung – Erkenntnisse zur Akzeptanz bei Fachkräften KW - Soziale Arbeit Y1 - 2026 UR - https://www.eventclass.it/dgpm2026/scientific/online-program/session?s=Symposium+56#e479 ER - TY - CHAP A1 - Burghardt, Jennifer A1 - Stieler, Mara A1 - Steigerwald, Philipp A1 - Rudolph, Eric A1 - Albrecht, Jens A1 - Lehmann, Robert T1 - KI-gestützte Beratung – Assistenzsysteme für Ratsuchende und Fachkräfte in der digitalen Beratung T2 - Deutscher Suchtkongress (DSK) Y1 - 2025 U6 - https://doi.org/10.18416/DSK.2025.2259 VL - 2 IS - 1 ER -