TY - CHAP A1 - Baumann, Timo A1 - Siegert, Ingo ED - Alt, Florian ED - Schneegass, Stefan ED - Hornecker, Eva T1 - Prosodic addressee-detection : ensuring privacy in always-on spoken dialog systems T2 - Tagungsband Mensch und Computer (MuC'20): 06.09.2020 - 09.09.2020, Magdeburg N2 - We analyze the addressee detection task for complexity-identical dialog for both human conversation and device-directed speech. Our recurrent neural model performs at least as good as humans, who have problems with this task, even native speakers, who profit from the relevant linguistic skills. We perform ablation experiments on the features used by our model and show that fundamental frequency variation is the single most relevant feature class. Therefore, we conclude that future systems can detect whether they are addressed based only on speech prosody which does not (or only to a very limited extent) reveal the content of conversations not intended for the system. KW - addressee detection KW - complexity-identical human-computer interaction KW - computational paralinguistics KW - fundamental frequency variation KW - recurrent neural network Y1 - 2020 SN - 9781450375405 U6 - https://doi.org/10.1145/3404983.3410021 SP - 195 EP - 198 PB - Association for Computing Machinery CY - New York, NY, United States ER - TY - JOUR A1 - Krüger, Norbert A1 - Fischer, Kerstin A1 - Manoonpong, Poramate A1 - Palinko, Oskar A1 - Bodenhagen, Leon A1 - Baumann, Timo A1 - Kjærum, Jens A1 - Rano, Ignacio A1 - Naik, Lakshadeep A1 - Juel, William Kristian A1 - Haarslev, Frederik A1 - Ignasov, Jevgeni A1 - Marchetti, Emanuela A1 - Langedijk, Rosalyn Melissa A1 - Kollakidou, Avgi A1 - Jeppesen, Kasper Camillus A1 - Heidtmann, Conny A1 - Dalgaard, Lars T1 - The SMOOTH-Robot: A Modular, Interactive Service Robot JF - Frontiers in Robotics and AI N2 - The SMOOTH-robot is a mobile robot that-due to its modularity-combines a relatively low price with the possibility to be used for a large variety of tasks in a wide range of domains. In this article, we demonstrate the potential of the SMOOTH-robot through three use cases, two of which were performed in elderly care homes. The robot is designed so that it can either make itself ready or be quickly changed by staff to perform different tasks. We carefully considered important design parameters such as the appearance, intended and unintended interactions with users, and the technical complexity, in order to achieve high acceptability and a sufficient degree of utilization of the robot. Three demonstrated use cases indicate that such a robot could contribute to an improved work environment, having the potential to free resources of care staff which could be allocated to actual care-giving tasks. Moreover, the SMOOTH-robot can be used in many other domains, as we will also exemplify in this article. KW - human–robot interaction KW - mobile robots KW - proactive control KW - service robots KW - socially aware navigation Y1 - 2021 U6 - https://doi.org/10.3389/frobt.2021.645639 VL - 8 PB - Frontiers ER - TY - JOUR A1 - Fischer, Kerstin A1 - Bodenhagen, Leon A1 - Krüger, Norbert A1 - Andersen, Morten Østergaard A1 - Baumann, Timo T1 - Trinvise og delte strategier for kontrol af velfærdsrobotter JF - Medicoteknik Y1 - 2015 UR - http://ipaper.ipapercms.dk/TechMedia/Medicoteknik/2015/1/?Page=11 VL - 2 IS - 1 SP - 11 EP - 13 PB - TechMedia ER - TY - CHAP A1 - Kousidis, Spyros A1 - Kennington, Casey A1 - Baumann, Timo A1 - Buschmeier, Hendrik A1 - Kopp, Stefan A1 - Schlangen, David T1 - Situationally Aware In-Car Information Presentation Using Incremental Speech Generation: Safer, and More Effective T2 - Proceedings of the EACL 2014 Workshop on Dialogue in Motion, Gothenburg, Sweden N2 - Holding non-co-located conversationswhile driving is dangerous (Horrey and- Wickens, 2006; Strayer et al., 2006), much more so than conversations with physically present, “situated” interlocutors (Drews et al., 2004). In-car dialogue systems typically resemble non-co-located conversations more, and share their negative impact (Strayer et al., 2013). We implemented and tested a simple strategy for making in-car dialogue systems aware of the driving situation, by giving them the capability to interrupt themselves when a dangerous situation is detected,and resume when over. We show that this improves both driving performance and recall of system-presented information, compared to a non-adaptive strategy. KW - in-car information presentation KW - incremental speech generation KW - situation awareness Y1 - 2014 U6 - https://doi.org/10.3115/v1/W14-0212 SP - 68 EP - 72 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Nayak, Shravan A1 - Schuler, Christian A1 - Saha, Debjoy A1 - Baumann, Timo ED - Tumuluri, Raj ED - Sebe, Nicu ED - Pingali, Gopal T1 - A Deep Dive Into Neural Synchrony Evaluation for Audio-visual Translation T2 - ICMI '22, Proceedings of the 2022 International Conference on Multimodal Interaction: November 7 - 11 2022, Bengaluru India N2 - We present a comprehensive analysis of the neural audio-visual synchrony evaluation tool SyncNet. We assess the agreement of SyncNet scores vis-a-vis human perception and whether we can use these as a reliable metric for evaluating audio-visual lip-synchrony in generation tasks with no ground truth reference audio-video pair. We further look into the underlying elements in audio and video which vitally affect synchrony using interpretable explanations from SyncNet predictions and analyse its susceptibility by introducing adversarial noise. SyncNet has been used in numerous papers on visually-grounded text-to-speech for scenarios such as dubbing. We focus on this scenario which features many local asynchronies (something that SyncNet isn’t made for). KW - audio-visual synchrony KW - speech-lip synchrony KW - dubbing Y1 - 2022 SN - 978-1-4503-9390-4 U6 - https://doi.org/10.1145/3536221.3556621 SP - 642 EP - 647 PB - Association for Computing Machinery CY - New York ER - TY - CHAP A1 - Melzer, Matthias A1 - Dünnweber, Jan A1 - Baumann, Timo T1 - Towards Smart Home Data Interpretation Using Analogies to Natural Language Processing T2 - 2022 IEEE International Conference on Smart Internet of Things: IEEE SmartIoT 2022, proceedings, 19-21 August 2022, Suzhou, China, hybrid conference (onsite and virtual) N2 - Recent advances in the development of smart homes have led to the availability of a wide variety of devices providing a high level of convenience via gesture and speech control or fully automated operation. Many smart home appliances also address the aspects of safety and electricity savings by automatically powering themselves off after not being used for a while. However, many devices remain in a typical household that are not themselves "smart", or are not primarily electric (such as heating systems). We address the savings aspect by identifying processes involving the use of multiple devices in the electrical flow data, as captured by a smart meter in a modern household, rather than focusing on a single appliance. Therefore, we introduce a novel approach to usage pattern analysis based on the idea that a pattern of device usages as a result of a resident's 'routine' (such as making breakfast) can be interpreted similarly to a natural language 'sentence'; Natural Language Processing (NLP) algorithms can then be used for interpreting the residents' behavior. We introduce the notion of bag-of-devices (BoD), derived from the bag-of-words model used in document classification. In an experiment, we show how we use this model to infer predictions about the inhabitants from device usage, such as the resident leaving for the day or just to fetch the newspaper. KW - IoT KW - AI KW - NLP KW - Smart City Y1 - 2022 SN - 978-1-6654-7952-3 U6 - https://doi.org/10.1109/SmartIoT55134.2022.00020 SP - 65 EP - 71 PB - IEEE ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Automatic Detection of Enjambment in German Readout Poetry T2 - Proceedings of Speech Prosody, 2018, Poznán N2 - One of the most important patterns in ancient as well as modern poetry is the enjambment, the continuation of a sentence beyond the end of a line, couplet, or stanza. The paper reports first activities towards the development of a digital tool to analyze the accentuation of poetic enjambments in readout poetry. The aim in this contribution is to recognize two forms of enjambment (emphasized and unemphasized) in poems using audio and text data. We use data from lyrikline which is a major online portal for spoken poetry whereas poems are read aloud by the original authors. We identified by hermeneutical means based on literary analysis a total of 69 poems being characteristic for the use of enjambments in modern and postmodern German poetry and train classifiers to differentiate the emphasized/unemphasized ategorization. A remarkable result of our automated analyses (and to our knowledge the first data-driven analysis of this kind) is the identification of a cultural difference in the accentuation of enjambments: statistically speaking, poets from the former GDR tend to emphasize the enjambment, whereas poets from the FRG do not. We use features derived from speech-to-text alignment and statistical parsing information such as pause lengths, number of lines with verbs, and number of lines with punctuation. The best classification results, calculated by the F-measure, for the both types of enjambment (emphasized/unemphasized) is 0.69. KW - modern and postmodern poetry KW - free verse prosody KW - emphasized and unemphasized enjambment Y1 - 2018 U6 - https://doi.org/10.21437/SpeechProsody.2018-67 SP - 329 EP - 333 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Analysing the Focus of a Hierarchical Attention Network: the Importance of Enjambments When Classifying Post-modern Poetry T2 - Proceedings of Interspeech, 2018, Hyderabad N2 - After overcoming the traditional metrics, modern and postmodern poetry developed a large variety of ‘free verse prosodies’ that falls along a spectrum from a more fluent to a more disfluent and choppy style. We present a method, grounded in philological analysis and theories on cognitive (dis)fluency, to analyze this ‘free verse spectrum’ into six classes of poetic styles as well as to differentiate three types of poems with enjambments. We use a model for automatic prosodic analysis of spoken free verse poetry which uses deep hierarchical attention networks to integrate the source text and audio and predict the assigned class. We then analyze and fine-tune the model with a particular focus on enjambments and in two ways: we drill down on classification performance by analyzing whether the model focuses on similar traits of poems as humans would, specifically, whether it internally builds a notion of enjambment. We find that our model is similarly good as humans in finding enjambments; however, when we employ the model for classifying enjambment-dominated poem types, it does not pay particular attention to those lines. Adding enjambment labels to the training only marginally improves performance, indicating that all other lines are similarly informative for the model. KW - digital humanities KW - free verse poetry KW - prosodic analysis KW - enjambment detection KW - hierarchical attention network Y1 - 2018 U6 - https://doi.org/10.21437/Interspeech.2018-2533 SP - 2162 EP - 2166 ER - TY - CHAP A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo ED - Burghardt, M. ED - Müller-Birn, C. T1 - Analysis and Classification of Prosodic Styles in Post-modern Spoken Poetry T2 - INF-DH 2018 Workshopband N2 - We present our research on computer-supported analysis of prosodic styles in post-modern poetry. Our project is unique in making use of both the written as well as the spoken form of the poem as read by the original author. In particular, we use speech and natural language processing technology to align speech and text and to perform textual analyses. We then explore, based on literary theory, the quantitative value of various types of features in differentiating various prosodic classes of post-modern poetry using machine-learning techniques. We contrast this feature-driven approach with a theoretically less informed neural networks-based approach and explore the relative strengths of both models, as well as how to integrate higher-level knowledge into the NN. In this paper, we give an overview of our project, our approach, and particularly focus on the challenges encountered and lessons learned in our interdisciplinary endeavour. The classification results of the rhythmical patterns (six classes) using NN-based approaches are better than by feature-based approaches. KW - modern and postmodern poetry KW - free verse prosody KW - rhythmical patters Y1 - 2018 U6 - https://doi.org/10.18420/infdh2018-09 PB - Gesellschaft für Informatik e.V ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Analysis of Rhythmic Phrasing: Feature Engineering vs. Representation Learning for Classifying Readout Poetry T2 - Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL-2018), August 25, 2018, Santa Fe, New Mexico, USA N2 - We show how to classify the phrasing of readout poems with the help of machine learning algorithms that use manually engineered features or automatically learn representations. We investigate modern and postmodern poems from the webpage lyrikline, and focus on two exemplary rhythmical patterns in order to detect the rhythmic phrasing: The Parlando and the Variable Foot. These rhythmical patterns have been compared by using two important theoretical works: The Generative Theory of Tonal Music and the Rhythmic Phrasing in English Verse. Using both, we focus on a combination of four different features: The grouping structure, the metrical structure, the time-span-variation, and the prolongation in order to detect the rhythmic phrasing in the two rhythmical types. We use manually engineered features based on text-speech alignment and parsing for classification. We also train a neural network to learn its own representation based on text, speech and audio during pauses. The neural network outperforms manual feature engineering, reaching an f-measure of 0.85. Y1 - 2018 UR - https://aclanthology.org/W18-4505 SP - 44 EP - 49 PB - Association for Computational Linguistics CY - Santa Fe, New Mexico ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Style Detection for Free Verse Poetry from Text and Speech T2 - Proceedings of the 27th International Conference on Computational Linguistics (COLING), 20-26.08.2018, Santa Fe, New Mexico, USA N2 - Modern and post-modern free verse poems feature a large and complex variety in their poetic prosodies that falls along a continuum from a more fluent to a more disfluent and choppy style. As the poets of modernism overcame rhyme and meter, they oriented themselves in these two opposing directions, creating a free verse spectrum that calls for new analyses of prosodic forms. We present a method, grounded in philological analysis and current research on cognitive (dis)fluency, for automatically analyzing this spectrum. We define and relate six classes of poetic styles (ranging from parlando to lettristic decomposition) by their gradual differentiation. Based on this discussion, we present a model for automatic prosodic classification of spoken free verse poetry that uses deep hierarchical attention networks to integrate the source text and audio and predict the assigned class. We evaluate our model on a large corpus of German author-read post-modern poetry and find that classes can reliably be differentiated, reaching a weighted f-measure of 0.73, when combining textual and phonetic evidence. In our further analyses, we validate the model’s decision-making process, the philologically hypothesized continuum of fluency and investigate the relative importance of various features. Y1 - 2018 UR - https://aclanthology.org/C18-1164.pdf SP - 1929 EP - 1940 ER -