TY - CHAP A1 - Baumann, Timo A1 - Eller, Korbinian A1 - Gagarina, Natalia ED - Lal, Yash Kumar ED - Clark, Elizabeth ED - Iyyer, Mohit ED - Chaturvedi, Snigdha ED - Brei, Anneliese ED - Brahman, Faeze ED - Chandu, Khyathi Raghavi T1 - BERT-based Annotation of Oral Texts Elicited via Multilingual Assessment Instrument for Narratives T2 - Proceedings of the the 6th Workshop on Narrative Understanding (WNU), November 12th to November 16th 2024, Miami, Florida, USA N2 - We investigate how NLP can help annotate the structure and complexity of oral narrative texts elicited via the Multilingual Assessment Instrument for Narratives (MAIN). MAIN is a theory-based tool designed to evaluate the narrative abilities of children who are learning one or more languages from birth or early in their development. It provides a standardized way to measure how well children can comprehend and produce stories across different languages and referential norms for children between 3 and 12 years old. MAIN has been adapted to over ninety languages and is used in over 65 countries. The MAIN analysis focuses on story structure and story complexity which are typically evaluated manually based on scoring sheets. We here investigate the automation of this process using BERT-based classification which already yields promising results. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.wnu-1.16 SP - 99 EP - 104 PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Mühlhausen, Sara A1 - Gomez, Sarah A1 - Lauer, Norina A1 - Baumann, Timo ED - Grawunder, Sven T1 - Cross lingual transfer learning does not improve aphasic speech recognition T2 - Elektronische Sprachsignalverarbeitung 2025: Tagungsband der 36. Konferenz Halle/Saale, 05.–07. MÄRZ 2025 N2 - In addressing the particular linguistic challenges posed by patients suffering from aphasia, a language disorder, this paper proposes a fine-tuning approach to enhance the speech recognition capabilities of existing models. The available aphasic research data in German is highly limited. To address this constraint, we propose a cross-lingual transfer approach to utilize English data to improve performance in German. This advancement aims to support the development of a therapy platform tailored for patients with aphasia. For the base speech recognition model, we choose to use OpenAI’s Whisper model, and for fine-tuning, we make use of TalkBank’s AphasiaBank. The experimental findings demonstrate that the transcription of aphasic audio with Whisper is less successful than non-aphasic audio. However, fine-tuning the transcription in the respective language resulted in an enhancement of its quality. In contrast, fine-tuning the transcription in another language and expecting a transfer of the learned aphasic speech properties led to a deterioration in its quality. KW - Automatische Spracherkennung KW - Sprachdialogsystem KW - Aphasie Y1 - 2025 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-80518 UR - https://www.essv.de/pdf/2025_77_84.pdf SN - 978-3-95908-803-9 SN - 0940-6832 PB - TUDpress CY - Dresden ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard ED - Schneider, Birgit ED - Löffler, Beate ED - Mager, Tino ED - Hein, Carola T1 - Free Verse Prosodies: Identifying and Classifying Spoken Poetry Using Literary and Computational Perspectives (Rhythmicalizer) T2 - Mixing Methods: Practical Insights from the Humanities in the Digital Age N2 - At least 80% of modern and postmodern poems exhibit neither rhyme nor metrical schemes such as iamb or trochee. However, does this mean that they are free of any rhythmical features?TheUS American research onfree verse prosody claimsthe opposite: Modern poets like Whitman, the Imagists, the Beat poets and contemporary Slam poets have developed a postmetrical idea of prosody, using rhythmical features of everyday language, prose, and musical styles like Jazz or Hip Hop. It has spawned a large and complex variety intheir poetic prosodies which,however,appearto bemuchharderto quantify and regularize than traditional patterns. In our project, we examinethe largest portal for spoken poetry Lyrikline and analysed and classified such rhythmical patterns by using pattern recognition and classification techniques. We integrate a human-in-the-loop approach in which we interleave manual annotation with computational modelling and data-based analysis. Our results are integrated into the website of Lyrikline. Our follow-up project makes our research results available to a wider audience, in particular to high school-level teaching. Y1 - 2023 U6 - https://doi.org/10.1515/9783839469132-018 SP - 167 EP - 186 PB - Bielefeld University Press CY - Bielefeld ER - TY - CHAP A1 - Baumann, Timo A1 - Buß, Okko A1 - Atterer, Michaela A1 - Schlangen, David T1 - Evaluating the Potential Utility of ASR N-Best Lists for Incremental Spoken Dialogue Systems T2 - Proceedings of Interspeech 2009 : speech and intelligence ; 6 - 10 September, 2009, Brighton, UK N2 - The potential of using ASR n-best lists for dialogue systems has often been recognised (if less often realised): it is often the case that even when the top-ranked hypothesis is erroneous, a bet- ter one can be found at a lower rank. In this paper, we describe metrics for evaluating whether the same potential carries over to incremental dialogue systems, where ASR output is consumed and reacted upon while speech is still ongoing. We show that even small N can provide an advantage for semantic process- ing, at a cost of a computational overhead. KW - dialogue systems KW - speech recognition KW - naturallanguage understanding KW - incrementality Y1 - 2009 U6 - https://doi.org/10.21437/Interspeech.2009-318 SP - 1031 EP - 1034 PB - ISCA CY - Brighton, UK ER - TY - CHAP A1 - Baumann, Timo A1 - Atterer, Michaela A1 - Schlangen, David T1 - Assessing and Improving the Performance of Speech Recognition for Incremental Systems T2 - NAACL '09: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, May 31 - June 5, 2009, Boulder, Colorado, USA N2 - In incremental spoken dialogue systems, par- tial hypotheses about what was said are re- quired even while the utterance is still ongo- ing. We define measures for evaluating the quality of incremental ASR components with respect to the relative correctness of the par- tial hypotheses compared to hypotheses that can optimize over the complete input, the tim- ingof hypothesisformationrelative to the por- tion ofthe inputthey areabout, andhypothesis stability, defined as the number of times they are revised. We show that simple incremen- tal post-processing can improve stability dra- matically, at the cost of timeliness (from 90% of edits of hypotheses being spurious down to 10% at a lag of 320ms). The measures are not independent,and we show how system de- signers can find a desired operating point for their ASR. To our knowledge, we are the first to suggest and examine a variety of measures for assessing incremental ASR and improve performance on this basis. Y1 - 2009 UR - https://dl.acm.org/doi/10.5555/1620754.1620810 SN - 978-1-932432-41-1 SP - 380 EP - 388 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Timo A1 - Buß, Okko A1 - Schlangen, David T1 - InproTK in Action: Open-Source Software for Building German-Speaking Incremental Spoken Dialogue Systems T2 - Electronic speech signal processing 2010 : proceedings of the 21st conference, Berlin, 8 - 10 September 2010 N2 - We present INPROTK, a toolkit for building incremental spoken dia-logue systems. Incremental spoken dialogue systems (systems that may react whilethe user’s utterance is ongoing) are a fairly recent research topic and allow for ex-citing new features. Even though toolkits exist that help in building conventionaldialogue systems, INPROTK offers both a tested architecture for building incre-mental SDSs as well as many of the building blocks necessary when building suchsystems. With INPROTK a researcher can avoid many of the technical difficulties,which hopefully further fosters research in this area. Y1 - 2010 UR - https://www.researchgate.net/publication/233540470_InproTK_in_Action_Open-Source_Software_for_Building_German-Speaking_Incremental_Spoken_Dialogue_Systems SP - 204 EP - 211 PB - TUDpress CY - Berlin, Germany ER - TY - CHAP A1 - Atterer, Michaela A1 - Baumann, Timo A1 - Schlangen, David T1 - No Sooner Said Than Done: Testing the Incrementality of Semantic Interpretations of Spontaneous Speech T2 - Proceedings of Interspeech 2009 : 6 - 10 September 2009, Brighton, U.K. N2 - Ideally, a spoken dialogue system should react without much delay to a user’s utterance. Such a system would already select an object, for instance, before the user has finished her utterance about moving this particular object to a particular place. A prerequisite for such a prompt reaction is that semantic representations are built up on the fly and passed on to other modules. Few approaches to incremental semantics construction exist, and, to our knowledge, none of those has been systematically tested on a spontaneous speech corpus. In this paper, we develop measures to test empirically on transcribed spontaneous speech to what extent we can create semantic interpretation on the fly with an incremental semantic chunker that builds a frame semantics. KW - incrementality KW - spoken dialogue systems KW - spontaneous speech KW - evaluation Y1 - 2009 U6 - https://doi.org/10.21437/Interspeech.2009-539 SP - 1855 EP - 1858 PB - International Speech Communication Association CY - Brighton, UK ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Predicting the Micro-Timing of User Input for an Incremental Spoken Dialogue System that Completes a User’s Ongoing Turn T2 - SIGDIAL 2011 Conference, 12th annual meeting of the Special Interest Group on Discourse and Dialogue, Co-located with ACL HLT 2011, Portland, Oregon, 17 - 18 June 2011 N2 - We present the novel task of predicting tem-poral features of continuations of user input,while that input is still ongoing. We show that the remaining duration of an ongoing word, aswell as the duration of the next can be predicted reasonably well, and we put this information touse in a system that synchronously completesa user’s speech. While we focus on collaborative completions, the techniques presented here may also be useful for the alignment of back-channels and immediate turn-taking in anincremental SDS, or to synchronously monitorthe user’s speech fluency for other reasons. Y1 - 2011 UR - https://www.researchgate.net/publication/220794517_Predicting_the_Micro-Timing_of_User_Input_for_an_Incremental_Spoken_Dialogue_System_that_Completes_a_User%27s_Ongoing_Turn SN - 978-1-61839-242-8 SP - 120 EP - 129 PB - Curran CY - Red Hook, NY ER - TY - CHAP A1 - Buschmeier, Hendrik A1 - Baumann, Timo A1 - Dorsch, Benjamin A1 - Kopp, Stefan A1 - Schlangen, David T1 - Combining Incremental Language Generation and Incremental Speech Synthesis for Adaptive Information Presentation T2 - SIGDIAL '12: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Seoul, South Korea, July 5 - 6, 2012 N2 - Participants in a conversation are normally receptive to their surroundings and their interlocutors, even while they are speaking and can, if necessary, adapt their ongoing utterance. Typical dialogue systems are not receptive and cannot adapt while uttering. We present combin-able components for incremental natural lan-guage generation and incremental speech syn-thesis and demonstrate the flexibility they can achieve with an example system that adapts to a listener's acoustic understanding problems by pausing, repeating and possibly rephrasing problematic parts of an utterance. In an evaluation, this system was rated as significantly more natural than two systems representing the current state of the art that either ignore the interrupting event or just pause; it also has a lower response time. Y1 - 2012 U6 - https://doi.org/10.5555/2392800.2392852 SP - 295 EP - 303 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Heintze, Silvan A1 - Baumann, Timo A1 - Schlangen, David T1 - Comparing Local and Sequential Models for Statistical Incremental Natural Language Understanding T2 - SIGDIAL '10: Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Tokyo, Japan,September 24 - 25, 2010 N2 - Incremental natural language understanding is the task of assigning semantic representations to successively larger prefixes of utterances. We compare two types of statistical models for this task: a) local models, which predict a single class for an input; and b), sequential models, which align a sequence of classes to a sequence of input tokens. We show that, with some modifications, the first type of model can be improved and made to approximate the output of the second, even though the latter is more informative. We show on two different data sets that both types of model achieve comparable performance (significantly better than a baseline), with the first type requiring simpler training data. Results for the first type of model have been reported in the literature; we show that for our kind of data our more sophisticated variant of the model performs better. Y1 - 2010 SN - 978-1-932432-85-5 U6 - https://doi.org/10.5555/1944506.1944508 PB - Association for Computational Linguistics CY - Tokyo, Japan ER -