TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Analysing the Focus of a Hierarchical Attention Network: the Importance of Enjambments When Classifying Post-modern Poetry T2 - Proceedings of Interspeech, 2018, Hyderabad N2 - After overcoming the traditional metrics, modern and postmodern poetry developed a large variety of ‘free verse prosodies’ that falls along a spectrum from a more fluent to a more disfluent and choppy style. We present a method, grounded in philological analysis and theories on cognitive (dis)fluency, to analyze this ‘free verse spectrum’ into six classes of poetic styles as well as to differentiate three types of poems with enjambments. We use a model for automatic prosodic analysis of spoken free verse poetry which uses deep hierarchical attention networks to integrate the source text and audio and predict the assigned class. We then analyze and fine-tune the model with a particular focus on enjambments and in two ways: we drill down on classification performance by analyzing whether the model focuses on similar traits of poems as humans would, specifically, whether it internally builds a notion of enjambment. We find that our model is similarly good as humans in finding enjambments; however, when we employ the model for classifying enjambment-dominated poem types, it does not pay particular attention to those lines. Adding enjambment labels to the training only marginally improves performance, indicating that all other lines are similarly informative for the model. KW - digital humanities KW - free verse poetry KW - prosodic analysis KW - enjambment detection KW - hierarchical attention network Y1 - 2018 U6 - https://doi.org/10.21437/Interspeech.2018-2533 SP - 2162 EP - 2166 ER - TY - CHAP A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo ED - Burghardt, M. ED - Müller-Birn, C. T1 - Analysis and Classification of Prosodic Styles in Post-modern Spoken Poetry T2 - INF-DH 2018 Workshopband N2 - We present our research on computer-supported analysis of prosodic styles in post-modern poetry. Our project is unique in making use of both the written as well as the spoken form of the poem as read by the original author. In particular, we use speech and natural language processing technology to align speech and text and to perform textual analyses. We then explore, based on literary theory, the quantitative value of various types of features in differentiating various prosodic classes of post-modern poetry using machine-learning techniques. We contrast this feature-driven approach with a theoretically less informed neural networks-based approach and explore the relative strengths of both models, as well as how to integrate higher-level knowledge into the NN. In this paper, we give an overview of our project, our approach, and particularly focus on the challenges encountered and lessons learned in our interdisciplinary endeavour. The classification results of the rhythmical patterns (six classes) using NN-based approaches are better than by feature-based approaches. KW - modern and postmodern poetry KW - free verse prosody KW - rhythmical patters Y1 - 2018 U6 - https://doi.org/10.18420/infdh2018-09 PB - Gesellschaft für Informatik e.V ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Analysis of Rhythmic Phrasing: Feature Engineering vs. Representation Learning for Classifying Readout Poetry T2 - Proceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL-2018), August 25, 2018, Santa Fe, New Mexico, USA N2 - We show how to classify the phrasing of readout poems with the help of machine learning algorithms that use manually engineered features or automatically learn representations. We investigate modern and postmodern poems from the webpage lyrikline, and focus on two exemplary rhythmical patterns in order to detect the rhythmic phrasing: The Parlando and the Variable Foot. These rhythmical patterns have been compared by using two important theoretical works: The Generative Theory of Tonal Music and the Rhythmic Phrasing in English Verse. Using both, we focus on a combination of four different features: The grouping structure, the metrical structure, the time-span-variation, and the prolongation in order to detect the rhythmic phrasing in the two rhythmical types. We use manually engineered features based on text-speech alignment and parsing for classification. We also train a neural network to learn its own representation based on text, speech and audio during pauses. The neural network outperforms manual feature engineering, reaching an f-measure of 0.85. Y1 - 2018 UR - https://aclanthology.org/W18-4505 SP - 44 EP - 49 PB - Association for Computational Linguistics CY - Santa Fe, New Mexico ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard T1 - Style Detection for Free Verse Poetry from Text and Speech T2 - Proceedings of the 27th International Conference on Computational Linguistics (COLING), 20-26.08.2018, Santa Fe, New Mexico, USA N2 - Modern and post-modern free verse poems feature a large and complex variety in their poetic prosodies that falls along a continuum from a more fluent to a more disfluent and choppy style. As the poets of modernism overcame rhyme and meter, they oriented themselves in these two opposing directions, creating a free verse spectrum that calls for new analyses of prosodic forms. We present a method, grounded in philological analysis and current research on cognitive (dis)fluency, for automatically analyzing this spectrum. We define and relate six classes of poetic styles (ranging from parlando to lettristic decomposition) by their gradual differentiation. Based on this discussion, we present a model for automatic prosodic classification of spoken free verse poetry that uses deep hierarchical attention networks to integrate the source text and audio and predict the assigned class. We evaluate our model on a large corpus of German author-read post-modern poetry and find that classes can reliably be differentiated, reaching a weighted f-measure of 0.73, when combining textual and phonetic evidence. In our further analyses, we validate the model’s decision-making process, the philologically hypothesized continuum of fluency and investigate the relative importance of various features. Y1 - 2018 UR - https://aclanthology.org/C18-1164.pdf SP - 1929 EP - 1940 ER - TY - CHAP A1 - Köhn, Arne A1 - Baumann, Timo A1 - Dörfler, Oskar T1 - An Empirical Analysis of the Correlation of Syntax and Prosody T2 - Proceedings of Interspeech 2018, 02.-06.09.2018, Hyderabad N2 - The relation of syntax and prosody (the syntax-prosody interface) has been an active area of research, mostly in linguistics and typically studied under controlled conditions. More recently, prosody has also been successfully used in the data-based training of syntax parsers. However, there is a gap between the controlled and detailed study of the individual effects between syntax and prosody and the large-scale application of prosody in syntactic parsing with only a shallow analysis of the respective influences. In this paper, we close the gap by investigating the significance of correlations of prosodic realization with specific syntactic functions using linear mixed effects models in a very large corpus of read-out German encyclopedic texts. Using this corpus, we are able to analyze prosodic structuring performed by a diverse set of speakers while they try to optimize factual content delivery. After normalization by speaker, we obtain significant effects, e.g. confirming that the subject function, as compared to the object function, has a positive effect on pitch and duration of a word, but a negative effect on loudness. KW - syntax KW - prosody KW - corpus-based analysis KW - linear mixed effects models Y1 - 2018 U6 - https://doi.org/10.21437/Interspeech.2018-2530 SP - 2157 EP - 2161 ER - TY - CHAP A1 - Koller, Alexander A1 - Baumann, Timo A1 - Köhn, Arne T1 - DialogOS: Simple and extensible dialog modeling T2 - Proceedings of Interspeech: Hyderabad, India 2-6 September 2018 N2 - We present the open-source extensible dialog manager DialogOS. DialogOS features simple finite-state based dialog management (which can be expanded to more complex DM strategies via a full-fledged scripting language) in combination with integrated speech recognition and synthesis in multiple languages. DialogOS runs on all major platforms, provides a simple-to-use graphical interface and can easily be extended via well-defined plugin and client interfaces, or can be integrated server-side into larger existing software infrastructures. We hope that DialogOS will help foster research and teaching given that it lowers the bar of entry into building and testing spoken dialog systems and provides paths to extend one’s system as development progresses. Y1 - 2018 UR - https://edoc.sub.uni-hamburg.de/informatik/volltexte/2018/246/pdf/koller_dialogOS.pdf CY - Hyderabad, India ER - TY - CHAP A1 - Baumann, Timo T1 - Learning to Determine Who is the Better Speaker T2 - Proceedings of the 9th International Conference on Speech Prosody, 13-16 June 2018, Poznań, Poland N2 - Speech can be more or less likable in various ways and comparing speakers by likability has important applications such as speaker selection or matching. Determining the likability of a speaker is a difficult task which can be simplified by breaking it down into pairwise preference decisions. Using a corpus of 5440 pairwise preference ratings collected previously through crowd-sourcing, we train classifiers to determine which of two speakers is “better”. We find that modeling the speech feature sequences using LSTMs outperforms conventional methods that pre-aggregate feature averages by a large margin, indicating that the prosodic structure should be taken into account when determining speech quality. Our classifier reaches an accuracy of 97 % for coarse-grained decisions, where differences between speech quality in both stimuli is relatively large. Y1 - 2018 U6 - https://doi.org/10.21437/SpeechProsody.2018-165 SP - 819 EP - 822 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo ED - Moore, Roger K. ED - Thill, Serge ED - Marxer, Ricard T1 - Highly Responsive Vocal Interaction through Incremental Processing T2 - Vocal Interactivity in-and-between Humans, Animals and Robots (VIHAR), October 30-4, 2016 N2 - This seminar was held in late 2016 and brought together, for the first time, researchers studying vocal interaction in a variety of different domains covering communications between all possible combinations of humans, animals, and robots. While each of these sub-domains has extensive histories of research progress, there is much potential for cross-fertilisation that currently remains underexplored. This seminar aimed at bridging this gap. In this report, we present the nascent research field of VIHAR and the major outputs from our seminar in the form of prioritised open research questions, abstracts from stimulus talks given by prominent researchers in their respective fields, and open problem statements by all participants. Y1 - 2017 U6 - https://doi.org/10.4230/DagRep.6.10.154 VL - 6 IS - 10 SP - 154 EP - 194 PB - Schloss Dagstuhl ‐ Leibniz-Zentrum für Informatik CY - Dagstuhl, Germany ER - TY - CHAP A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo ED - Eibl, Maximilian ED - Gaedke, Martin T1 - Rhythmicalizer: Data Analysis for the Identification of Rhythmic Patterns in Readout Poetry T2 - INFORMATIK 2017, Tagung vom 25.- 29. September 2017 in Chemnitz T2 - GI-Edition Lecture notes in informatics. Proceedings ; volume P-275 N2 - The most important development in modern and postmodern poetry is the replacement of traditional meter by new rhythmical patterns. Ever since Walt Whitman's Leaves of Grass (1855), modern (nineteenth-to twenty-first-century) poets have been searching for novel forms of prosody, accent, rhythm, and intonation. Along with the rejection of older metrical units such as the iamb or trochee, a structure of lyrical language was developed that renounced traditional forms like rhyme and meter. This development is subsumed under the term free verse prosody. Our project will test this theory by applying machine learning or deep learning techniques to a corpus of modern and postmodern poems as read aloud by the original authors. To this end, we examine “lyrikline”, the most famous online portal for spoken poetry. First, about 17 different patterns being characteristic for the lyrikline-poems have been identified by the philological scholar of this project. This identification was based on a certain philological method including three different steps: a) grammetrical ranking; b) rhythmic phrasing; and c) mapping rubato and prosodic phrasing. In this paper we will show how to combine this philological and a digital analysis by using the prosody detection available in speech processing technology. In order to analyse the data, we want to use different tools for the following tasks: PoS-tagging, alignment, intonation, phrases and pauses, and tempo. We also analyzed the lyrikline-data by identifying the occurrence of the mentioned patterns. This analysis is a first step towards an automatic classification based on machine learning or deep learning techniques. Y1 - 2017 SN - 978-3-88579-669-5 U6 - https://doi.org/10.18420/in2017_218 SP - 2189 EP - 2200 PB - Gesellschaft für Informatik CY - Bonn ER - TY - CHAP A1 - Baumann, Timo A1 - Meyer-Sickendiek, Burkhard T1 - Large-scale Analysis of Spoken Free-verse Poetry T2 - Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities, Osaka, Japan, December 11-17 2016 N2 - Most modern and post-modern poems have developed a post-metrical idea of lyrical prosody that employs rhythmical features of everyday language and prose instead of a strict adherence to rhyme and metrical schemes. This development is subsumed under the term free verse prosody. We present our methodology for the large-scale analysis of modern and post-modern poetry in both their written form and as spoken aloud by the author. We employ language processing tools to align text and speech, to generate a null-model of how the poem would be spoken by a naïve reader, and to extract contrastive prosodic features used by the poet. On these, we intend to build our model of free verse prosody, which will help to understand, differentiate and relate the different styles of free verse poetry. We plan to use our processing scheme on large amounts of data to iteratively build models of styles, to validate and guide manual style annotation, to identify further rhythmical categories, and ultimately to broaden our understanding of free verse poetry. In this paper, we report on a proof-of-concept of our methodology using smaller amounts of poems and a limited set of features. We find that our methodology helps to extract differentiating features in the authors’ speech that can be explained by philological insight. Thus, our automatic method helps to guide the literary analysis and this in turn helps to improve our computational models. Y1 - 2016 UR - https://aclanthology.org/W16-4017 SP - 125 EP - 130 PB - The COLING 2016 Organizing Committee ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Tonality in Language: The Generative Theory of Tonal Music as a Framework for Prosodic Analysis of Poetry T2 - 6th International Symposium on Tonal Aspects of Languages (TAL 2018), 18-20 June 2018, Berlin, Germany N2 - This contribution focuses on structural similarities between tonality and cadences in music on the one hand, and rhythmical patterns in poetic languages respectively poetry on the other hand. We investigate two exemplary rhythmical patterns in modern and postmodern poetry to detect these tonality-like features in poetic language: The Parlando and the Variable Foot. German poems readout from the original poets are collected from the webpage of our partner lyrikline. We compared these rhythmical features with tonality rules, explained in two important theoretical volumes: The Generative Theory of Tonal Music and the Rhythmic Phrasing in English Verse. Using both volumes, we focused on a certain combination of four different features: The grouping structure, the metrical structure, the time-span-variation and the prolongation, in order to detect the two important rhythmical patterns which use tonality-like features in poetic language (Parlando and Variable Foot). Different features including pause and parser information are used in this classification process. The best classification result, calculated by the f-measure, for Parlando and Variable Foot is 0.69. KW - tonality in poetic language KW - prosodic patterns in postwar poems KW - automatic detection of prosodic features Y1 - 2018 U6 - https://doi.org/10.21437/TAL.2018-36 SP - 178 EP - 182 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo A1 - Kennington, Casey A1 - Hough, Julian A1 - Schlangen, David ED - Jokinen, Kristiina ED - Wilcock, Graham T1 - Recognising Conversational Speech: What an Incremental ASR Should Do for a Dialogue System and How to Get There T2 - Dialogues with social robots: Enablements analyses and evaluation T2 - Lecture Notes in Electrical Engineering: Bd. 427 N2 - Automatic speech recognition (ASR) is not only becoming increasingly accurate, but also increasingly adapted for producing timely, incremental output. However, overall accuracy and timeliness alone are insufficient when it comes to interactive dialogue systems which require stability in the output and responsivity to the utterance as it is unfolding. Furthermore, for a dialogue system to deal with phenomena such as disfluencies, to achieve deep understanding of user utterances these should be preserved or marked up for use by downstream components, such as language understanding, rather than be filtered out. Similarly, word timing can be informative for analyzing deictic expressions in a situated environment and should be available for analysis. Here we investigate the overall accuracy and incremental performance of three widely used systems and discuss their suitability for the aforementioned perspectives. From the differing performance along these measures we provide a picture of the requirements for incremental ASR in dialogue systems and describe freely available tools for using and evaluating incremental ASR. KW - Incremental ASR KW - Conversational speech KW - System requirements KW - Evaluation Y1 - 2017 SN - 978-981-10-2584-6 U6 - https://doi.org/10.1007/978-981-10-2585-3_35 SP - 421 EP - 432 PB - Springer CY - Singapore ER - TY - CHAP A1 - Köhn, Arne A1 - Baumann, Timo T1 - Predictive Incremental Parsing Helps Language Modeling T2 - Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan N2 - Predictive incremental parsing produces syntactic representations of sentences as they are produced, e.g. by typing or speaking. In order to generate connected parses for such unfinished sentences, upcoming word types can be hypothesized and structurally integrated with already realized words. For example, the presence of a determiner as the last word of a sentence prefix may indicate that a noun will appear somewhere in the completion of that sentence, and the determiner can be attached to the predicted noun. We combine the forward-looking parser predictions with backward-looking N-gram histories and analyze in a set of experiments the impact on language models, i.e. stronger discriminative power but also higher data sparsity. Conditioning N-gram models, MaxEnt models or RNN-LMs on parser predictions yields perplexity reductions of about 6%. Our method (a) retains online decoding capabilities and (b) incurs relatively little computational overhead which sets it apart from previous approaches that use syntax for language modeling. Our method is particularly attractive for modular systems that make use of a syntax parser anyway, e.g. as part of an understanding pipeline where predictive parsing improves language modeling at no additional cost. Y1 - 2016 UR - https://aclanthology.org/C16-1026 SP - 268 EP - 277 PB - The COLING 2016 Organizing Committee ER - TY - CHAP A1 - Baumann, Timo T1 - Large-Scale Speaker Ranking from Crowdsourced Pairwise Listener Ratings T2 - Interspeech 2017 : 20-24 August 2017 N2 - Speech quality and likability is a multi-faceted phenomenon consisting of a combination of perceptory features that cannot easily be computed nor weighed automatically. Yet, it is often easy to decide which of two voices one likes better, even though it would be hard to describe why, or to name the underlying basic perceptory features. Although likability is inherently subjective and individual preferences differ frequently, generalizations are useful and there is often a broad intersubjective consensus about whether one speaker is more likable than another. However, breaking down likability rankings into pairwise comparisons leads to a quadratic explosion of rating pairs. We present a methodology and software to efficiently create a likability ranking for many speakers from crowdsourced pairwise likability ratings. We collected pairwise likability ratings for many (>220) speakers from many raters (>160) and turn these ratings into one likability ranking. We investigate the resulting speaker ranking stability under different conditions: limiting the number of ratings and the dependence on rater and speaker characteristics. We also analyze the ranking wrt. acoustic correlates to find out what factors influence likability. We publish our ranking and the underlying ratings in order to facilitate further research. KW - ranking KW - speech quality KW - likability KW - found data KW - ratings KW - crowd-sourcing Y1 - 2017 U6 - https://doi.org/10.21437/Interspeech.2017-1697 SP - 2262 EP - 2266 PB - ISCA ER - TY - CHAP A1 - Rohde, Marcel A1 - Baumann, Timo T1 - Navigating the Spoken Wikipedia T2 - Proceedings of the 7th Workshop on Speech and Language Processing for Assistive Technologies (SLPAT 2016) N2 - The Spoken Wikipedia project unites volunteer readers of encyclopedic entries. Their recordings make encyclopedic knowledge accessible to persons who are unable to read (out of alexia, visual impairment, or because their sight is currently occupied, e. g. while driving). However, on Wikipedia, recordings are available as raw audio files that can only be consumed linearly, without the possibility for targeted navigation or search. We present a reading application which uses an alignment between the recording, text and article structure and which allows to navigate spoken articles, through a graphical or voice-based user interface (or a combination thereof). We present the results of a usability study in which we compare the two interaction modalities. We find that both types of interaction enable users to navigate articles and to find specific information much more quickly compared to a sequential presentation of the full article. In particular when the VUI is not restricted by speech recognition and understanding issues, this interface is on par with the graphical interface and thus a real option for browsing the Wikipedia without the need for vision or reading. KW - Mensch-Maschine-Interaktion KW - Sprechschnittstelle KW - Sprachdialogverarbeitung Y1 - 2016 U6 - https://doi.org/10.21437/SLPAT.2016-2 SP - 9 EP - 13 ER - TY - CHAP A1 - Köhn, Arne A1 - Stegen, Florian A1 - Baumann, Timo T1 - Mining the Spoken Wikipedia for Speech Data and Beyond T2 - Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16) N2 - We present a corpus of time-aligned spoken data of Wikipedia articles as well as the pipeline that allows to generate such corpora for many languages. There are initiatives to create and sustain spoken Wikipedia versions in many languages and hence the data is freely available, grows over time, and can be used for automatic corpus creation. Our pipeline automatically downloads and aligns this data. The resulting German corpus currently totals 293h of audio, of which we align 71h in full sentences and another 86h of sentences with some missing words. The English corpus consists of 287h, for which we align 27h in full sentence and 157h with some missing words. Results are publically available. KW - Spoken Wikipedia KW - Long Audio Alignment KW - Speech Resource Y1 - 2016 UR - https://aclanthology.org/L16-1735 SP - 4644 EP - 4647 PB - European Language Resources Association (ELRA) ER - TY - CHAP A1 - Kirsch, Kolja A1 - Steinicke, Frank A1 - Baumann, Timo T1 - Entwicklung und Evaluierung einer Notenwender-App für Klaviernoten T2 - Mensch und Computer 2016 - Tagungsband, 4. - 7. September 2016, Aachen N2 - Das ständige Umblättern von Noten ist für Musiker ein wiederkehrendes Problem. Dieses wird häufig durch einen Assistenten des Musikers, dem sogenannten Notenwender, gelöst. Diese Unterstützung haben allerdings viele Musiker nur selten während des Übens. In diesem Artikel stellen wir eine Anwendung für mobile Geräte vor, die auf verschiedene Arten das Umblättern von Klavierpartituren unterstützt. In einer Studie mit professionellen Musikern und Klavierschülern wurden diese Arten gegeneinander abgewogen. Die Ergebnisse zeigen auf, dass computer-unterstütztes Blättern Vorteile gegenüber herkömmlichem Blättern hat. Y1 - 2016 U6 - https://doi.org/10.18420/muc2016-mci-0159 PB - Gesellschaft für Informatik e.V. CY - Aachen ER - TY - CHAP A1 - Twiefel, Johannes A1 - Baumann, Timo A1 - Heinrich, Stefan A1 - Wermter, Stefan T1 - Improving Domain-independent Cloud-based Speech Recognition with Domain-dependent Phonetic Post-processing T2 - AAAI 2014 – Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI-14), Québec, CA, July 27–31, 2014 N2 - Automatic speech recognition (ASR) technology has been developed to such a level that off-the-shelf distributed speech recognition services are available (free of cost), which allow researchers to integrate speech into their applications with little development effort or expert knowledge leading to better results compared with previously used open-source tools. Often, however, such services do not accept language models or grammars but process free speech from any domain. While results are very good given the enormous size of the search space, results frequently contain out-of-domain words or constructs that cannot be understood by subsequent domain-dependent natural language understanding (NLU) components. We present a versatile post-processing technique based on phonetic distance that integrates domain knowledge with open-domain ASR results, leading to improved ASR performance. Notably, our technique is able to make use of domain restrictions using various degrees of domain knowledge, ranging from pure vocabulary restrictions via grammars or N-Grams to restrictions of the acceptable utterances. We present results for a variety of corpora (mainly from human-robot interaction) where our combined approach significantly outperforms Google ASR as well as a plain open-source ASR solution. Y1 - 2014 UR - https://www2.informatik.uni-hamburg.de/wtm/publications/2014/TBHW14/Twiefel_AAAI2014_CR.pdf U6 - https://doi.org/10.1609/aaai.v28i1.8929 SP - 1529 EP - 1535 PB - Association for the Advancement of Artificial Intelligence ER - TY - CHAP A1 - Kennington, Casey A1 - Kousidis, Spyros A1 - Baumann, Timo A1 - Buschmeier, Hendrik A1 - Kopp, Stefan A1 - Schlangen, David ED - Miller, Erika ED - Wu, Yuqing T1 - Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation T2 - Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI '14, Seattle WA, USA, September 17 - 19, 2014 N2 - It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers information via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes. Y1 - 2014 UR - 978-1-4503-3212-5 U6 - https://doi.org/10.1145/2667317.2667332 SP - 1 EP - 7 PB - ACM CY - New York, NY ER - TY - CHAP A1 - Baumann, Timo T1 - Partial Representations Improve the Prosody of Incremental Speech Synthesis T2 - Interspeech 2014, Singapore, 14-18 September 2014 N2 - When humans speak, they do not plan their full utterance inall detail before beginning to speak, nor do they speak piece-by-piece and ignoring their full message – instead humans usepartial representations in which they fill in the missing partsas the utterance unfolds. Incremental speech synthesizers, incontrast, have not yet made use of partial representations and theinformation contained there-in.We analyze the quality of prosodic parameter assignments(pitch and duration) generated from partial utterance specifi-cations (substituting defaults for missing features) in order todetermine the requirements that symbolic incremental prosodymodelling should meet. We find that broader, higher-level infor-mation helps to improve prosody even if lower-level informationabout the near future is yet unavailable. Furthermore, we findthat symbolic phrase-level or utterance-level information is mosthelpful towards the end of the phrase or utterance, respectively,that is, when this information is becoming available even in theincremental case. Thus, the negative impact of incremental pro-cessing can be minimized by using partial representations thatare filled in incrementally. KW - incremental processing KW - prosody KW - speech synthesis KW - spoken dialogue systems KW - simultaneous interpreting Y1 - 2014 U6 - https://doi.org/10.21437/Interspeech.2014-199 SP - 2932 EP - 2936 ER - TY - CHAP A1 - Kousidis, Spyros A1 - Kennington, Casey A1 - Baumann, Timo A1 - Buschmeier, Hendrik A1 - Kopp, Stefan A1 - Schlangen, David ED - Ali Salah, Albert T1 - A Multimodal In-Car Dialogue System That Tracks The Driver's Attention T2 - Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14, November 12 - 16 2014, Istanbul N2 - When a passenger speaks to a driver, he or she is co-located with the driver, is generally aware of the situation, and can stop speaking to allow the driver to focus on the driving task. In-car dialogue systems ignore these important aspects, making them more distracting than even cell-phone conversations. We developed and tested a "situationally-aware" dialogue system that can interrupt its speech when a situation which requires more attention from the driver is detected, and can resume when driving conditions return to normal. Furthermore, our system allows driver-controlled resumption of interrupted speech via verbal or visual cues (head nods). Over two experiments, we found that the situationally-aware spoken dialogue system improves driving performance and attention to the speech content, while driver-controlled speech resumption does not hinder performance in either of these two tasks Y1 - 2014 SN - 9781450328852 U6 - https://doi.org/10.1145/2663204.2663244 SP - 26 EP - 33 PB - ACM CY - New York, NY ER - TY - CHAP A1 - Baumann, Timo A1 - Lindner, Felix ED - Tapus, Adriana ED - André, Elisabeth ED - Martin, Jean-Claude T1 - Incremental Speech Production for Polite and Natural Personal-Space Intrusion T2 - Social robotics : 7th International Conference, ICSR 2015, Paris, France, October 26-30, 2015, Proceedings T2 - Lecture notes in Computer science ; 9388 N2 - We propose to use a model of personal space to initiate communication while passing a human thereby acknowledging that humans are not just a special kind of obstacle to be avoided but potential interaction partners. As a simple form of interaction, our system communicates an apology while closely passing a human. To this end, we present a software architecture that integrates a social-spaces knowledge base and a component for incremental speech production. Incrementality ensures that the robot’s utterance can be adapted to fit the developing situation in a natural way. Observer ratings show that personal-space intrusion is perceived as both natural and polite if the robot has the capability to utter and adapt an apology in an incremental way whereas it is perceived as unfriendly if the robot intrudes personal space without saying anything. Moreover, the robot is perceived as less natural if it does not adapt. Y1 - 2015 SN - 978-3-319-25553-8 U6 - https://doi.org/10.1007/978-3-319-25554-5_8 SP - 72 EP - 82 PB - Springer CY - Cham ER - TY - CHAP A1 - Baumann, Timo A1 - Kennington, Casey A1 - Hough, Julian A1 - Schlangen, David T1 - Recognising Conversational Speech: What an Incremental ASR Should Do for a Dialogue System and How to Get There T2 - Proceedings of the International Workshop Series on Spoken Dialogue Systems Technology (IWSDS) 2016 N2 - Automatic speech recognition (ASR) is not only becoming increasingly accurate, but also increasingly adapted for producing timely, incremental output. However, overall accuracy and timeliness alone are insufficient when it comes to interactive dialogue systems which require stability in the output and responsivity to the utterance as it is unfolding. Furthermore, for a dialogue system to deal with phenomena such as disfluencies, to achieve deep understanding of user utterances these should be preserved or marked up for use by downstream components, such as language understanding, rather than be filtered out. Similarly, word timing can be informative for analyzing deictic expressions in a situated environment and should be available for analysis. Here we investigate the overall accuracy and incremental performance of three widely used systems and discuss their suitability for the aforementioned perspectives. From the differing performance along these measures we provide a picture of the requirements for incremental ASR in dialogue systems and describe freely available tools for using and evaluating incremental ASR. Y1 - 2016 ER - TY - CHAP A1 - Baumann, Timo A1 - Lindner, Felix T1 - Towards a Social Robot that Incrementally Justifies Personal-Space Intrusion T2 - Proceedings of the Workshop Towards a Framework for Joint Action 2015 N2 - Robots should appropriately give reasons for their actions when these actions affect a human’s action or goal space. Communicating reasons may help the human understand the robot’s intents and may initiate joint action, i. e., accepting the robot’s goals and cooperating on the robot’s actions. However, to be efficient, the communication of reasons should be limited to the necessary rather than to completeness, conforming to the Gricean Maxim of Quantity. Furthermore, what is necessary only becomes apparent as the situation evolves and hence, for seamless interaction, ongoing utterances must be adapted as they happen. We present a system that flexibly gives reasons in a reduced setting in which the robot needs to intrude a human’s personal space in order to reach its goal. Y1 - 2015 UR - https://fja.sciencesconf.org/conference/fja/TB_FL_joint_action_abstract.pdf ER - TY - CHAP A1 - Pouget, Maël A1 - Hueber, Thomas A1 - Bailly, Gérard A1 - Baumann, Timo T1 - HMM training strategy for incremental speech synthesis T2 - Interspeech 2015, Dresden, Germany, 6-10 September 2015 N2 - Incremental speech synthesis aims at delivering the synthetic voice while the sentence is still being typed. One of the main challenges is the online estimation of the target prosody from a partial knowledge of the sentence's syntactic structure. In the context of HMM-based speech synthesis, this typically results in missing segmental and suprasegmental features, which describe the linguistic context of each phoneme. This study describes a voice training procedure which integrates explicitly a potential uncertainty on some contextual features. The proposed technique is compared to a baseline approach (previously published), which consists in substituting a missing contextual feature by a default value calculated on the training set. Both techniques were implemented in a HMM-based Text-To-Speech system for French, and compared using objective and perceptual measurements. Experimental results show that the proposed strategy outperforms the baseline technique for this language. KW - HMM-based speech synthesis KW - incremental KW - TTS KW - HTS KW - prosody Y1 - 2015 U6 - https://doi.org/10.21437/Interspeech.2015-304 SP - 1201 EP - 1205 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo T1 - Decision Tree Usage for Incremental Parametric Speech Synthesis T2 - Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2014, 4-9 May 2014, Florence, Italy N2 - Human speakers plan and deliver their utterances incrementally, piece-by-piece, and it is obvious that their choice regarding phonetic details (and the details' peculiarities) is rarely determined by globally optimal solutions. In contrast, parametric speech synthesizers use a full-utterance context when optimizing vocoding parameters and when determing HMM states. Apart from being cognitively implausible, this impedes incremental use-cases, where the future context is often at least partially unavailable. This paper investigates the `locality' of features in parametric speech synthesis voices and takes some missing steps towards better HMM state selection and prosody modelling for incremental speech synthesis. KW - Speech Synthesis KW - Incremental Processing KW - HMM Synthesis KW - Interactivity KW - Spoken Dialogue Systems Y1 - 2014 U6 - https://doi.org/10.1109/ICASSP.2014.6854316 SP - 3847 EP - 3851 PB - IEEE ER - TY - CHAP A1 - Baumann, Timo A1 - Bangalore, Srinivas A1 - Hirschberg, Julia ED - Federico, Marcello ED - Stüker, Sebastian ED - Yvon, François T1 - Towards Simultaneous Interpreting: The Timing of Incremental Machine Translation and Speech Synthesis T2 - Proceedings of the 11th International Workshop on Spoken Language Translation. Papers. Lake Tahoe, December 4th and 5th, 2014 N2 - In simultaneous interpreting, human experts incrementally construct and extend partial hypotheses about the source speaker’s message, and start to verbalize a corresponding message in the target language, based on a partial translation – which may have to be corrected occasionally. They commence the target utterance in the hope that they will be able to finish understanding the source speaker’s message and determine its translation in time for the unfolding delivery. Of course, both incremental understanding and translation by humans can be garden-pathed, although experts are able to optimize their delivery so as to balance the goals of minimal latency, translation quality and high speech fluency with few corrections. We investigate the temporal properties of both translation input and output to evaluate the tradeoff between low latency and translation quality. In addition, we estimate the improvements that can be gained with a tempo-elastic speech synthesizer. KW - incremental processing KW - speech synthesis KW - timing KW - simultaneous interpreting Y1 - 2014 UR - https://aclanthology.org/2014.iwslt-papers.2.pdf SP - 163 EP - 168 ER - TY - JOUR A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo T1 - Towards the Creation of a Poetry Translation Mapping System JF - Archives of Data Science, Series A N2 - The translation of poetry is a complex, multifaceted challenge: the translated text should communicate the same meaning, similar metaphoric expressions, and also match the style and prosody of the original poem. Research on machine poetry translation is existing since 2010, but for four reasons it is still rather insufficient: 1. The few approaches existing completely lack any knowledge about current developments in both lyric theory and translation theory. 2. They are based on very small datasets. 3. They mostly ignored the neural learning approach that superseded the long-standing dominance of phrase-based approaches within machine translation. 4. They have no concept concerning the pragmatic function of their research and the resulting tools. Our paper describes how to improve the existing research and technology for poetry translations in exactly these four points. With regards to 1) we will describe the “Poetics of Translation”. With regards to 2) we will introduce the Worlds largest corpus for poetry translations from lyrikline. With regards to 3) we will describe first steps towards a neural machine translation of poetry. With regards to 4) we will describe first steps towards the development of a poetry translation mapping system. Y1 - 2018 U6 - https://doi.org/10.5445/KSP/1000087327/21 SN - 2363-9881 VL - 5 IS - 1 SP - 1 EP - 15 PB - Towards the Creation of a Poetry Translation Mapping System ER - TY - PAT A1 - Ward, Nigel A1 - Baumann, Timo A1 - Karkhedkar, Shreyas A1 - Novick, David T1 - Dynamic control of voice codec data rate N2 - A method, system, and computer-usable non-transitory storage device for dynamic voice codec adaptation are disclosed. The voice codec adapts in real time to devote more bits to audio quality when it is most needed, and fewer bits to less important parts of utterances are disclosed. Dialog knowledge is utilized for compression opportunities to adjust the bitrate moment-by-moment, based on the inferred value of each frame. Frame importance and appropriate transmission fidelity is predicted based on prosodic features and models of dialog dynamics. This technique provides the same communications quality with less spectrum needs, fewer antennas, and less battery drain. Y1 - 2015 UR - https://assignment.uspto.gov/patent/index.html#/patent/search/resultAssignor?assignorName=BAUMANN,%20TIMO ER - TY - CHAP A1 - Johannßen, Dirk A1 - Biemann, Chris A1 - Remus, Steffen A1 - Baumann, Timo A1 - Scheffer, David T1 - GermEval 2020 Task 1 on the Classification and Regression of Cognitive and Motivational style from Text T2 - Proceedings of the 5th SwissText & 16th KONVENS Joint Conference 2020 N2 - This paper describes the tasks, databases, baseline systems, and summarizes submissions and results for the GermEval 2020 Shared Task 1 on the Classification and Regression of Cognitive and Motivational Style from Text. This shared task is divided into two subtasks, a regression task, and a classification task. Subtask 1 asks participants to reproduce a ranking of students based on average aptitude indicators such as different high school grades and different IQ scores. The second subtask aims to classify so-called implicit motives, which are projective testing procedures that can reveal unconscious desires. Besides five implicit motives, the target labels of Subtask 2 also contain one of six levels that describe the type of self-regulation when acting out a motive, which makes this task a multiclass-classification with 30 target labels. 3 participants submitted multiple systems. Subtask 1 was solved (best r = .3701) mainly with non-neural systems and statistical language representations, submissions for Subtask 2 utilized neural approaches and word embeddings (best macro F1 = 70.40). Not only were the tasks solvable, analyses by the participants even showed connections to the implicit psychometrics theory and behavioral observations made by psychologists. This paper describes the tasks, databases, baseline systems, and summarizes submissions and results for the GermEval 2020 Shared Task 1 on the Classification and Regression of Cognitive and Motivational Style from Text. This shared task is divided into two subtasks, a regression task, and a classification task. Subtask 1 asks participants to reproduce a ranking of students based on average aptitude indicators such as different high school grades and different IQ scores. The second subtask aims to classify so-called implicit motives, which are projective testing procedures that can reveal unconscious desires. Besides five implicit motives, the target labels of Subtask 2 also contain one of six levels that describe the type of self-regulation when acting out a motive, which makes this task a multiclass-classification with 30 target labels. 3 participants submitted multiple systems. Subtask 1 was solved (best r =.3701) mainly with non-neural systems and statistical language representations, submissions for Subtask 2 utilized neural approaches and word embeddings (best macro F1 = 70.40). Not only were the tasks solvable, analyses by the participants even showed connections to the implicit psychometrics theory and behavioral observations made by psychologists. Y1 - 2020 UR - https://www.inf.uni-hamburg.de/en/inst/ab/lt/publications/2020-johannssen-germeval20-1-companion-paper.pdf SP - 1 EP - 10 CY - Zurich, Switzerland ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Free Verse and Beyond: How to Classify Post-modern Spoken Poetry T2 - Proceedings of Speech Prosody: Tokyo, Japan, 25-28 May 2020 N2 - This paper presents the classification of rhythmical patterns detected in post-modern spoken poetry by means of machine learning algorithms that use manually engineered features or automatically learnt representations. We used the world's largest corpus of spoken poetry from our partner lyrikline. We identified nine rhythmical patterns within a spectrum raging from a more fluent to a more disfluent poetic style. The text data analyzed by a statistical parser. Prosodic features of rhythmical patterns are identified by using the parser information. For the classification of rhythmical patterns, we used a neural networks-based approach which use text, audio, and pause information between poetic lines as features. Different combinations of features as well as the integration of feature engineering in the neural networks-based approach are tested. We compared the performance of both approaches (feature-based and neural network-based) using combinations of different features. The results show – by using the weighted average of f-measure for the evaluation – that the neural networks-based approach performed much better in classification of rhythmical patterns. The important improvement of the classification results lies in the use of the audio information. The integration of feature engineering in the neural networks-based approach yielded a very small result improvement. Y1 - 2020 U6 - https://doi.org/10.21437/SpeechProsody.2020-141 SP - 690 EP - 694 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo A1 - Meyer-Sickendiek, Burkhard ED - Krauwer, Steven ED - Fišer, Darja T1 - Deep Learning meets Post-modern Poetry T2 - TwinTalks 2020: Understanding and Facilitating Collaboration in Digital Humanities 2020, proceedings of the Twin Talks 2 and 3 workshops at DHN 2020 and DH 2020, Ottawa Canada and Riga Latvia, July 23 and October 20, 2020 N2 - We summarize our project Rhythmicalizer in which we analyze a corpus of post-modern poetry in a combination of qualitative hermeneutical and computational methods, as we have run the project over the course of the past three years (and preparing it for some time before that). Interdisciplinary work is always challenging and we here focus on some of the highlights of our collaboration. KW - Literary Studies KW - Machine Learning KW - Meta-Research Y1 - 2020 UR - http://ceur-ws.org/Vol-2717/paper03.pdf SN - 1613-0073 SP - 30 EP - 36 PB - RWTH Aachen ER - TY - CHAP A1 - Baumann, Timo T1 - How a Listener Influences the Speaker T2 - Proceedings of Speech Prosody: Tokyo, Japan, 25-28 May 2020 N2 - Listeners typically provide feedback while listening to a speaker in conversation and thereby engage in the co-construction of the interaction. We analyze the influence of the listener on the speaker by investigating how her verbal feedback signals help in modeling the speaker's language. We find that feedback from the listener may help in modeling the speaker's language, whether through the listener's feedback as transcribed, or the acoustic signal directly. We find the largest positive effects for end of sentence as well as for pauses mid-utterance, but also effects that indicate we successfully model elaborations of ongoing utterances that may result from the presence or absence of listener feedback. KW - CNN KW - conversation KW - feedback KW - language model KW - RNNLM Y1 - 2020 U6 - https://doi.org/10.21437/SpeechProsody.2020-198 SP - 970 EP - 974 PB - ISCA ER - TY - JOUR A1 - Baumann, Timo A1 - Köhn, Arne A1 - Hennig, Felix T1 - The Spoken Wikipedia Corpus collection: Harvesting, alignment and an application to hyperlistening JF - Language Resources and Evaluation N2 - Spoken corpora are important for speech research, but are expensive to create and do not necessarily reflect (read or spontaneous) speech ‘in the wild’. We report on our conversion of the preexisting and freely available Spoken Wikipedia into a speech resource. The Spoken Wikipedia project unites volunteer readers of Wikipedia articles. There are initiatives to create and sustain Spoken Wikipedia versions in many languages and hence the available data grows over time. Thousands of spoken articles are available to users who prefer a spoken over the written version. We turn these semi-structured collections into structured and time-aligned corpora, keeping the exact correspondence with the original hypertext as well as all available metadata. Thus, we make the Spoken Wikipedia accessible for sustainable research. We present our open-source software pipeline that downloads, extracts, normalizes and text–speech aligns the Spoken Wikipedia. Additional language versions can be exploited by adapting configuration files or extending the software if necessary for language peculiarities. We also present and analyze the resulting corpora for German, English, and Dutch, which presently total 1005 h and grow at an estimated 87 h per year. The corpora, together with our software, are available via http://islrn.org/resources/684-927-624-257-3/. As a prototype usage of the time-aligned corpus, we describe an experiment about the preferred modalities for interacting with information-rich read-out hypertext. We find alignments to help improve user experience and factual information access by enabling targeted interaction. KW - Annotation KW - Eyes-free speech access KW - Found data KW - Robust text–speech alignment KW - Speech corpus KW - Spoken hypertext KW - Wikipedia Y1 - 2019 U6 - https://doi.org/10.1007/s10579-017-9410-y VL - 53 IS - 2 SP - 303 EP - 329 PB - Springer Nature ER - TY - CHAP A1 - Saboo, Ashutosh A1 - Baumann, Timo ED - Bojar, Ondřej ED - Chatterjee, Rajen ED - Federmann, Christian ED - Fishel, Mark ED - Graham, Yvette ED - Haddow, Barry ED - Huck, Matthias ED - Yepes, Antonio Jimeno ED - Koehn, Philipp ED - Martins, André ED - Monz, Christof ED - Negri, Matteo ED - Névéol, Aurélie ED - Neves, Mariana ED - Post, Matt ED - Turchi, Marco ED - Verspoor, Karin T1 - Integration of Dubbing Constraints into Machine Translation T2 - Proceedings of the Fourth Conference on Machine Translation, Volume 1: Research Papers N2 - Translation systems aim to perform a meaning-preserving conversion of linguistic material (typically text but also speech) from a source to a target language (and, to a lesser degree, the corresponding socio-cultural contexts). Dubbing, i.e., the lip-synchronous translation and revoicing of speech adds to this constraints about the close matching of phonetic and resulting visemic synchrony characteristics of source and target material. There is an inherent conflict between a translation’s meaning preservation and ‘dubbability’ and the resulting trade-off can be controlled by weighing the synchrony constraints. We introduce our work, which to the best of our knowledge is the first of its kind, on integrating synchrony constraints into the machine translation paradigm. We present first results for the integration of synchrony constraints into encoder decoder-based neural machine translation and show that considerably more ‘dubbable’ translations can be achieved with only a small impact on BLEU score, and dubbability improves more steeply than BLEU degrades. Y1 - 2019 U6 - https://doi.org/10.18653/v1/W19-5210 SP - 94 EP - 101 PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Identification of Concrete Poetry within a Modern-Poetry Corpus using Neural Networks T2 - Proceedings of the Quantitative Approaches to Versification Conference N2 - This work aims to discern the poetics of concrete poetry by using a corpus-based classification focusing on the two most important techniques used within concrete poetry: semantic decomposition and syntactic permutation. We demonstrate how to identify concrete poetry in modern and postmodern free verse. A class contrasting to concrete poetry is defined on the basis of poems with complete and correct sentences. We used the data from lyrikline, which contain both the written as well as the spoken form of poems as read by the original author. We explored two approaches for the identification of concrete poetry. The first is based on the definition of concrete poetry in literary theory by the extraction of various types of features derived from a parser, such as verb, noun, comma, sentence ending, conjunction, and asemantic material. The second is a neural network-based approach, which is theoretically less informed by human insight, as it does not have access to features established by scholars. This approach used the following inputs: textual information and the spoken recitation of poetic lines as well as information about pauses between lines. The results based on the neural network are more accurate than the feature-based approach. The best results, calculated by the weighted F-measure, for the classification of concrete poetry vis-à-vis the contrasting class is 0.96 Y1 - 2019 UR - https://versologie.cz/conference2019/proceedings/hussein-meyer-sickendiek-baumann.pdf SP - 95 EP - 104 CY - Prague, Czech Republic ER - TY - CHAP A1 - Baumann, Timo ED - Weiss, Benjamin ED - Trouvain, Jürgen ED - Barkat-Defradas, Mélissa ED - Ohala, John J. T1 - Ranking and Comparing Speakers Based on Crowdsourced Pairwise Listener Ratings T2 - Voice attractiveness: Studies on Sexy, Likable, and Charismatic Speakers N2 - Speech quality and likability is a multi-faceted phenomenon consisting of a combination of perceptory features that cannot easily be computed nor weighed automatically. Yet, it is often easy to decide which of two voices one likes better, even though it would be hard to describe why, or to name the underlying basic perceptory features. Although likability is inherently subjective and individual preferences differ, generalizations are useful and there is often a broad intersubjective consensus about whether one speaker is more likeable than another. We present a methodology to efficiently create a likability ranking for many speakers from crowdsourced pairwise likability ratings which focuses manual rating effort on pairs of similar quality using an active sampling technique. Using this methodology, we collected pairwise likability ratings for many speakers (>220) from many raters (>160). We analyze listener preferences by correlating the resulting ranking with various acoustic and prosodic features. We also present a neural network that is able to model the complexity of listener preferences and the underlying temporal evolution of features. The recurrent neural network achieves remarkably high performance in estimating the pairwise decisions and an ablation study points toward the criticality of modeling temporal aspects in speech quality assessment. KW - Crowdsourcing KW - Found data KW - Likability ratings KW - Ranking KW - Sequence modelling KW - Speech quality Y1 - 2021 SN - 978-981-15-6626-4 SN - 978-981-15-6627-1 U6 - https://doi.org/10.1007/978-981-15-6627-1_14 SP - 263 EP - 279 PB - Springer CY - Singapore ER - TY - CHAP A1 - Tsai, Vivian A1 - Baumann, Timo A1 - Pecune, Florian A1 - Cassell, Justine ED - D'Haro, Luis Fernando ED - Banchs, Rafael E. ED - Li, Haizhou T1 - Faster Responses Are Better Responses: Introducing Incrementality into Sociable Virtual Personal Assistants T2 - 9th International Workshop on Spoken Dialogue System Technology N2 - Speech-based interactive systems, such as virtual personal assistants, inevitably use complex architectures, with a multitude of modules working in series (or less often in parallel) to perform a task (e.g., giving personalized movie recommendations via dialog). Add modules for evoking and sustaining sociability with the user and the accumulation of processing latencies through the modules results in considerable turn-taking delays. We introduce incremental speech processing into the generation pipeline of the system to overcome this challenge with only minimal changes to the system architecture, through partial underspecification that is resolved as necessary. A user study with a sociable movie recommendation agent objectively diminishes turn-taking delays; furthermore, users not only rate the incremental system as more responsive, but also rate its recommendation performance as higher. Y1 - 2019 SN - 978-981-13-9442-3 SN - 978-981-13-9445-4 SN - 978-981-13-9443-0 U6 - https://doi.org/10.1007/978-981-13-9443-0_10 SN - 1876-1119 SN - 1876-1100 VL - 579 SP - 111 EP - 118 PB - Springer Singapore CY - Singapore ER - TY - CHAP A1 - Nayak, Shravan A1 - Baumann, Timo A1 - Bhattacharya, Supratik A1 - Karakanta, Alina A1 - Negri, Matteo A1 - Turchi, Marco ED - Truong, Khiet T1 - See me Speaking? Differentiating on Whether Words are Spoken On Screen or Off to Optimize Machine Dubbing T2 - ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction, 25.10.2020 - 29.10.2020; Virtual Event Netherlands N2 - Dubbing is the art of finding a translation from a source into a target language that can be lip-synchronously revoiced, i. e., that makes the target language speech appear as if it was spoken by the very actors all along. Lip synchrony is essential for the full-fledged reception of foreign audiovisual media, such as movies and series, as violated constraints of synchrony between video (lips) and audio (speech) lead to cognitive dissonance and reduce the perceptual quality. Of course, synchrony constraints only apply to the translation when the speaker's lips are visible on screen. Therefore, deciding whether to apply synchrony constraints requires an automatic method for detecting whether an actor's lips are visible on screen for a given stretch of speech or not. In this paper, we attempt, for the first time, to classify on- from off-screen speech based on a corpus of real-world television material that has been annotated word-by-word for the visibility of talking lips on screen. We present classification experiments in which we classify Y1 - 2020 SN - 9781450380027 U6 - https://doi.org/10.1145/3395035.3425640 SP - 130 EP - 134 PB - Association for Computing Machinery CY - New York,NY,United States ER - TY - CHAP A1 - Geislinger, Robert A1 - Milde, Benjamin A1 - Baumann, Timo A1 - Biemann, Chris T1 - Live Subtitling for BigBlueButton with Open-Source Software T2 - INTERSPEECH 2021: (Show & Tell Contribution), 30 August – 3 September, 2021, Brno, Czechia N2 - We present an open source plugin for live subtitling in the popular open source video conferencing software BigBlueBut-ton. Our plugin decodes each speaker’s audio stream separately and in parallel, thereby obliviating the need for speaker di-arization and seamlessly handling overlapped talk. Any Kaldi-compatible nnet3 model can be used with our plugin and we demonstrate it using freely available TDNN-HMM-based ASR models for English and German. Our subtitles can be used as they are (e.g., in loud environments) or can form the basis for further NLP processes. Our tool can also simplify the collection of remotely recorded multi-party dialogue corpora. KW - automatic speech recognition KW - automatic subtitles KW - computer-supported collaborative work KW - meeting transcription KW - multi-party dialogue KW - videoconferencing KW - VoIP Y1 - 2021 UR - https://www.isca-speech.org/archive/pdfs/interspeech_2021/geislinger21_interspeech.pdf SP - 3319 EP - 3320 CY - Brno, Czechia ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Elbeshausen, Jasper ED - Draude, Claude ED - Lange, Martin ED - Sick, Bernhard T1 - A Tool for Human-in-the-Loop Analysis and Exploration of (not only) Prosodic Classifications for Post-modern Poetry T2 - Informatik 2019 : 50 Jahre Gesellschaft für Informatik Workshop-Beiträge Fachtagung vom 23.-26. September 2019 in Kassel N2 - Data-based analyses are becoming more and more common in the Digital Humanities and tools are needed that focus human efforts on the most interesting and important aspects of exploration, analysis and annotation by using active machine learning techniques. We present our ongoing work on a tool that supports classification tasks for spoken documents (in our case: read-out post-modern poetry) using a neural networks-based classification backend and a web-based exploration and classification environment. KW - classification KW - data mining KW - free verse poetry KW - human-in-the-loop KW - rhythmical patterns Y1 - 2019 UR - https://dl.gi.de/bitstream/handle/20.500.12116/25047/paper03_08.pdf?sequence=1&isAllowed=y SN - 978-3-88579-689-3 SP - 151 EP - 156 PB - Gesellschaft für Informatik CY - Bonn ER - TY - CHAP A1 - Karakanta, Alina A1 - Bhattacharya, Supratik A1 - Nayak, Shravan A1 - Baumann, Timo A1 - Negri, Matteo A1 - Turchi, Marco ED - Scott, Donia ED - Bel, Nuria ED - Zong, Chengqing T1 - The Two Shades of Dubbing in Neural Machine Translation T2 - Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020): Barcelona, Spain (Online) N2 - Dubbing has two shades; synchronisation constraints are applied only when the actor’s mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depending on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive features between on- and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease translation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation. Y1 - 2020 U6 - https://doi.org/10.18653/v1/2020.coling-main.382 SP - 4327 EP - 4333 PB - International Committee on Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Fischer, Kerstin A1 - Naik, Lakshadeep A1 - Langedijk, Rosalyn Melissa A1 - Baumann, Timo A1 - Jelínek, Matouš A1 - Palinko, Oskar ED - Bethel, Cindy ED - Paiva, Ana T1 - Initiating Human-Robot Interactions Using Incremental Speech Adaptation T2 - HRI '21 Companion, Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction: March 8 - 11, 2021, Boulder CO, USA N2 - In this paper, we present a study in which a robot initiates interactions with people passing by in an in-the-wild scenario. The robot adapts the loudness of its voice dynamically to the distance of the respective person approached, thus indicating who it is talking to. It furthermore tracks people based on information on body orientation and eye gaze and adapts the text produced based on people's distance autonomously. Our study shows that the adaptation of the loudness of its voice is perceived as personalization by the participants and that the likelihood that they stop by and interact with the robot increases when the robot incrementally adjusts its behavior. KW - gaze KW - incrementalit KW - initiating interaction KW - personalization KW - voice Y1 - 2021 SN - 9781450382908 U6 - https://doi.org/10.1145/3434074.3447205 SP - 421 EP - 425 PB - Association for Computing Machinery CY - New York, NY ER - TY - CHAP A1 - Saha, Debjoy A1 - Nayak, Shravan A1 - Baumann, Timo T1 - Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcasts T2 - Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), Marseille, 20-25 June 2022 N2 - We introduce the Merkel Podcast Corpus, an audio-visual-text corpus in German collected from 16 years of (almost) weekly Internet podcasts of former German chancellor Angela Merkel. To the best of our knowledge, this is the first single speaker corpus in the German language consisting of audio, visual and text modalities of comparable size and temporal extent. We describe the methods used with which we have collected and edited the data which involves downloading the videos, transcripts and other metadata, forced alignment, performing active speaker recognition and face detection to finally curate the single speaker dataset consisting of utterances spoken by Angela Merkel. The proposed pipeline is general and can be used to curate other datasets of similar nature, such as talk show contents. Through various statistical analyses and applications of the dataset in talking face generation and TTS, we show the utility of the dataset. We argue that it is a valuable contribution to the research community, in particular, due to its realistic and challenging material at the boundary between prepared and spontaneous speech. Accepted at LREC 2022 KW - corpus KW - speaker diarization KW - multi-modal KW - forced alignment KW - single-speaker KW - cross-modal learning KW - German Y1 - 2022 UR - http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.270.pdf N1 - Preprint unter: https://arxiv.org/abs/2205.12194 SP - 2536 EP - 2540 ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David ED - Fingscheidt, T. T1 - The INPROTK 2012 release BT - A toolkit for incremental spoken dialogue processing T2 - Sprachkommunikation 2012 : Beiträge zur 10. ITG-Fachtagung vom 26. bis 28. September 2012 in Braunschweig N2 - We describe the 2012 release of INPROTK1, our “Incremental Processing Toolkit“ which combines a powerful and extensible architecture for incremental processing with components for incremental speech recognition and, new to this release, incremental speech synthesis. These components work domainindependently; we also provide example implementations of higher-level components such as natural language understanding and dialogue management that are somewhat more tied to a particular domain. The toolkit is accompanied by evaluation tools for analysing timing behaviour, and we highlight some timing results on conversational speech input in this paper. We offer our toolkit to foster research in this new and exciting area, which promises to help increase the naturalness of behaviours that can be modelled in such systems. KW - Speech synthesis Y1 - 2012 UR - https://ieeexplore.ieee.org/document/6309600 SN - 978-3-8007-3455-9 SP - 147 EP - 150 PB - VDE-Verl CY - Berlin ; Offenbach ER - TY - GEN A1 - Baumann, Timo A1 - Geiser, Dorothee A1 - Menzel, Wolfgang A1 - Mohr, Mario A1 - Neef, Svenja A1 - Nykamp, Sören A1 - Rokita, Nils T1 - Concurrent Sub-turn Interaction Specification and Dialogue Management with an Application to Interactive Storytelling T2 - GSCL Workshop Gesprochene Sprache und Sprachverarbeitung (GSS), Darmstadt, Germany, 2013 N2 - Conventional dialogue management centers around an interaction style that is best described as a ping-pong game, with full turns being the units at which speech is delivered (and expected) by the system, which greatly simplifies the interaction management, delivery and understanding components of the system. While the resulting mode of interaction works well for task-based systems, it is insufficient for more conversational interaction styles, where content is delivered and grounded in units finer than full turns (Poesio and Traum 1997) and where turns are delivered concurrently by both interlocutors and hence overlap more frequently. One domain with particularly frequent overlapping contributions is interactive storytelling: a storyteller that is responsive to listeners will integrate their feedback immediately while still speaking a current contribution, and listener’s remarks or propositions regarding the story will typically be uttered immediately when the related content is delivered. We present our work on a dialogue manager that leverages recent advances in incremental speech delivery and reception (Baumann 2013) to provide for an interactive and concurrent storytelling experience. Our system uses an interaction graph, which uses a word-by-word granularity and allows to specify for individual stretches of speech where (and with what content) users may interrupt/comment, how this is interpreted, and how it is integrated into the storytelling process, thus providing for in-utterance alternatives to be spoken even without audibly interrupting the system’s ongoing utterance. The system’s interaction graph is specified in an XML language and may be hand-crafted by a story designer but can also be automatically generated. In our current system, speech recognition results are interpreted only when the user utterance is finished; however, we plan to integrate incremental speech recognition and understanding capabilities, and to immediately react to the start of user contributions. Y1 - 2013 CY - Darmstadt, Germany ER - TY - CHAP A1 - Schenke, Diana Marie A1 - Baumann, Timo ED - Möller, Sebastian ED - Knoeferle, Pia ED - Schulte, Britta ED - Feldhus, Nils T1 - Controlled Diversity: Length-optimized Natural Language Generation T2 - Proceedings of the ISCA/ITG Workshop on Diversity in Large Speech and Language Models, February 20, 2025 in Berlin, Germany N2 - LLMs are not generally able to adjust the length of their outputs based on strict length requirements, a capability that would improve their usefulness in applications that require adherence to diverse user and system requirements. We present an approach to train LLMs to acquire this capability by augmenting existing data and applying existing fine-tuning techniques, which we compare based on the trained models’ adherence to the length requirement and overall response quality relative to the baseline model. Our results demonstrate that these techniques can be successfully applied to train LLMs to adhere to length requirements, with the trained models generating texts which better align to the length requirements. Our results indicate that our method may change the response quality when using training data that was not generated by the baseline model. This allows simultaneous alignment to another training objective in certain scenarios, but is undesirable otherwise. Training on a dataset containing the model’s own responses eliminates this issue. Y1 - 2025 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-80534 ER - TY - INPR A1 - Amanpour, Ali A1 - Baumann, Timo A1 - Hofmann, Ulrich G. T1 - Optimizing Neural Data Analysis: Determining Minimum Recording Length for Unambigous Signal Processing N2 - Advanced silicon electrode arrays facilitate the recording of thousands of neurons, generating extensive neural data that imposes a significant burden on researchers and processing algorithms. Thus, real-time analysis pipelines are gaining increasing value, while at the same time having to deal with non-stationary and noisy data. We intend to apply Machine Learning (ML) algorithms to a dense set of recordings from rat brains in order to prepare a functional atlas, correlating neuronal signals with anatomical position. While doing so, we needed to decide on a rational way which recording snippet length would best represent the original, longer source sequence and thus suffices to be further processed for anatomical correlation. We implemented an algorithm to evaluate the spectral information of systematically length varied records based on similarity to the original record. For our dataset a recording duration of 3 seconds satisfied moderate requirements across all channels, thus allowing us to reduce computational load for ongoing ML classification of microprobe sourced electrophysiologic signals. Y1 - 2025 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-80606 ER - TY - CHAP A1 - Morales, Juan A. A1 - Baumann, Timo A1 - Pérez, José L. A1 - Peinado, Antonio M. A1 - Gómez, Ángel M. T1 - Implementaciòn de un reconocedor distribuido de voz en tiempo real sobre IP T2 - Actas de las IV Jornadas en Tecnologías del Habla N2 - En este trabajo se presenta una implementación de unsistema de reconocimiento distribuido del habla en tiem-po real para su aplicación en un entorno de Internet. Desa-rrollado como una aplicación cliente-servidor, el clien-te hace uso del front-end estándar definido por la ETSI.Incluye un detector de voz para sólo enviar informacióncuando el locutor habla y la transmisión de la informaciónse hace de acuerdo al RFC 3557. El servidor realiza el re-conocimiento utilizando programación dinámica, incluyea su vez un detector de voz y utiliza la técnica de mitiga-ción de pérdida de paquetes propuesta en el estándar de laETSI. La modularidad del diseño permite la utilización decualquier otro reconocedor sin que por ello se vean afec-tados los clientes. Las pruebas reales de funcionamientopara la aplicación particular desarrollada han demostradouna alta fiabilidad en varias condiciones de transmisión. Y1 - 2006 UR - https://www.researchgate.net/publication/319702803_IMPLEMENTACION_DE_UN_RECONOCEDOR_DISTRIBUIDO_DE_VOZ_EN_TIEMPO_REAL_SOBRE_IP ER - TY - JOUR A1 - Nigel G. Ward, A1 - Vega, Alejandro A1 - Baumann, Timo T1 - Prosodic and Temporal Features for Language Modeling for Dialog JF - Speech Communication N2 - If we can model the cognitive and communicative processes underlying speech, we should be able to better predict what a speaker will do. With this idea as inspiration, we examine a number of prosodic and timing features as potential sources of information on what words the speaker is likely to say next. In spontaneous dialog we find that word probabilities do vary with such features. Using perplexity as the metric, the most informative of these included recent speaking rate, volume, and pitch, and time until end of utterance. Using simple combinations of such features to augment trigram language models gave up to a 8.4% perplexity benefit on the Switchboard corpus, and up to a 1.0% relative reduction in word error rate (0.3% absolute) on the Verbmobil II corpus. KW - Dialog dynamics KW - Dialog state KW - Prosody KW - Interlocutor behavior KW - Word probabilities KW - Prediction KW - Perplexity KW - Speech recognition KW - Switchboard corpus KW - Verbmobil corpus Y1 - 2012 U6 - https://doi.org/10.1016/j.specom.2011.07.009 VL - 54 IS - 2 SP - 161 EP - 174 PB - ELSEVIER ER - TY - CHAP A1 - Lohmann, Kris A1 - Eichhorn, Ole A1 - Baumann, Timo T1 - Generating Situated Assisting Utterances to Facilitate Tactile-Map Understanding: A Prototype System T2 - Proceedings of the Third Workshop on Speech and Language Processing for Assistive Technologies (SLPAT), Montreal, Canada, June 7–8, 2012 N2 - Tactile maps are important substitutes for visual maps for blind and visually impaired people and the efficiency of tactile-map reading can largely be improved by giving assisting utterances that make use of spatial language. In this paper, we elaborate earlier ideas for a system that generates such utterances and present a prototype implementation based on a semantic conceptualization of the movements that the map user performs. A worked example shows the plausibility of the solution and the output that the prototype generates given input derived from experimental data. Y1 - 2012 UR - https://aclanthology.org/W12-2908.pdf SP - 56 EP - 65 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Timo T1 - Feedback in Adaptive Interactive Storytelling T2 - 13th annual conference of the International Speech Communication Association 2012 (INTERSPEECH 2012): Portland, Oregon, USA, 9 - 13 September 2012, Interdisciplinary Workshop on Feedback Behaviors in Dialog N2 - Telling stories is different from reading out text: a speakerreponds to the listener’s feedback and incorporates thisinto the ongoing talk. However, current computer sys-tems are unable to do this and instead just non-attentivelyread out a text, disregarding all feedback (or the absencethereof). I propose and discuss an idea for a small researchproject and a plan for how an attentive listening storytellercan be built. KW - Incrementality KW - Feedback KW - Storytelling KW - Adaptation KW - Prosody Y1 - 2012 UR - https://www.researchgate.net/publication/236879509_Feedback_in_Adaptive_Interactive_Storytelling SN - 978-1-62276-759-5 CY - Stevenson, USA ER - TY - CHAP A1 - Baumann, Timo T1 - Integrating Prosodic Modelling with Incremental Speech Recognition T2 - Proceedings of DiaHolmia, the 13th International Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL 2009), 24-26 JUNE, 2009, Sttockhom, Sweden N2 - We describe ongoing and proposed work concerning incremental prosody extraction and classification for a spoken dialogue system. The system described will be tightly integrated with the SDS's speech recogntion which also works incrementally. The proposed architecture should allow for more control over the user interaction experience, for example allowing more precise and timely end-of-utterance vs. hesitation distinction, and auditive or visual back-channel generation. Y1 - 2009 UR - https://www.isca-speech.org/archive_v0/diaholmia_2009/dho9_143.html SP - 143 EP - 144 PB - Royal Institute of Technology ER - TY - THES A1 - Baumann, Timo T1 - Automatische Erkennung von Akzentuierungen und Phrasierungen in Sprachsynthesekorpora Y1 - 2007 UR - https://nats-www.informatik.uni-hamburg.de/pub/User/TimoBaumann/Publications/da.pdf N1 - Univ. Hamburg, Diplomarbeit CY - Hamburg ER - TY - CHAP A1 - Penzkofer, Vinzent A1 - Baumann, Timo T1 - Evaluating and Fine-Tuning Retrieval-Augmented Language Models to Generate Text With Accurate Citations T2 - Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024), September 10-13, 2024, Vienna, Austria N2 - Retrieval Augmented Generation (RAG) is becoming an essential tool for easily accessing large amounts of textual information. However, it is often challenging to determine whether the information in a given response originates from the retrieved context, the training, or is a result of hallucination. Our contribution in this area is twofold. Firstly, we demonstrate how existing datasets for information retrieval evaluation can be used to assess the ability of Large Language Models (LLMs) to correctly identify relevantsources. Our findings indicate that there are notable discrepancies in the performance of different current LLMs in this task. Secondly, we utilise the datasets and metrics for citation evaluation to enhance the citation quality of small open-weight LLMs through fine-tuning. We achieve significant performance gains in this task, matching the results of much larger models. Y1 - 2024 UR - https://aclanthology.org/2024.konvens-main.6.pdf SP - 57 EP - 64 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Timo A1 - Eller, Korbinian A1 - Gagarina, Natalia ED - Lal, Yash Kumar ED - Clark, Elizabeth ED - Iyyer, Mohit ED - Chaturvedi, Snigdha ED - Brei, Anneliese ED - Brahman, Faeze ED - Chandu, Khyathi Raghavi T1 - BERT-based Annotation of Oral Texts Elicited via Multilingual Assessment Instrument for Narratives T2 - Proceedings of the the 6th Workshop on Narrative Understanding (WNU), November 12th to November 16th 2024, Miami, Florida, USA N2 - We investigate how NLP can help annotate the structure and complexity of oral narrative texts elicited via the Multilingual Assessment Instrument for Narratives (MAIN). MAIN is a theory-based tool designed to evaluate the narrative abilities of children who are learning one or more languages from birth or early in their development. It provides a standardized way to measure how well children can comprehend and produce stories across different languages and referential norms for children between 3 and 12 years old. MAIN has been adapted to over ninety languages and is used in over 65 countries. The MAIN analysis focuses on story structure and story complexity which are typically evaluated manually based on scoring sheets. We here investigate the automation of this process using BERT-based classification which already yields promising results. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.wnu-1.16 SP - 99 EP - 104 PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Mühlhausen, Sara A1 - Gomez, Sarah A1 - Lauer, Norina A1 - Baumann, Timo ED - Grawunder, Sven T1 - Cross lingual transfer learning does not improve aphasic speech recognition T2 - Elektronische Sprachsignalverarbeitung 2025: Tagungsband der 36. Konferenz Halle/Saale, 05.–07. MÄRZ 2025 N2 - In addressing the particular linguistic challenges posed by patients suffering from aphasia, a language disorder, this paper proposes a fine-tuning approach to enhance the speech recognition capabilities of existing models. The available aphasic research data in German is highly limited. To address this constraint, we propose a cross-lingual transfer approach to utilize English data to improve performance in German. This advancement aims to support the development of a therapy platform tailored for patients with aphasia. For the base speech recognition model, we choose to use OpenAI’s Whisper model, and for fine-tuning, we make use of TalkBank’s AphasiaBank. The experimental findings demonstrate that the transcription of aphasic audio with Whisper is less successful than non-aphasic audio. However, fine-tuning the transcription in the respective language resulted in an enhancement of its quality. In contrast, fine-tuning the transcription in another language and expecting a transfer of the learned aphasic speech properties led to a deterioration in its quality. KW - Automatische Spracherkennung KW - Sprachdialogsystem KW - Aphasie Y1 - 2025 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-80518 UR - https://www.essv.de/pdf/2025_77_84.pdf SN - 978-3-95908-803-9 SN - 0940-6832 PB - TUDpress CY - Dresden ER - TY - CHAP A1 - Baumann, Timo A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard ED - Schneider, Birgit ED - Löffler, Beate ED - Mager, Tino ED - Hein, Carola T1 - Free Verse Prosodies: Identifying and Classifying Spoken Poetry Using Literary and Computational Perspectives (Rhythmicalizer) T2 - Mixing Methods: Practical Insights from the Humanities in the Digital Age N2 - At least 80% of modern and postmodern poems exhibit neither rhyme nor metrical schemes such as iamb or trochee. However, does this mean that they are free of any rhythmical features?TheUS American research onfree verse prosody claimsthe opposite: Modern poets like Whitman, the Imagists, the Beat poets and contemporary Slam poets have developed a postmetrical idea of prosody, using rhythmical features of everyday language, prose, and musical styles like Jazz or Hip Hop. It has spawned a large and complex variety intheir poetic prosodies which,however,appearto bemuchharderto quantify and regularize than traditional patterns. In our project, we examinethe largest portal for spoken poetry Lyrikline and analysed and classified such rhythmical patterns by using pattern recognition and classification techniques. We integrate a human-in-the-loop approach in which we interleave manual annotation with computational modelling and data-based analysis. Our results are integrated into the website of Lyrikline. Our follow-up project makes our research results available to a wider audience, in particular to high school-level teaching. Y1 - 2023 U6 - https://doi.org/10.1515/9783839469132-018 SP - 167 EP - 186 PB - Bielefeld University Press CY - Bielefeld ER - TY - CHAP A1 - Baumann, Timo A1 - Buß, Okko A1 - Atterer, Michaela A1 - Schlangen, David T1 - Evaluating the Potential Utility of ASR N-Best Lists for Incremental Spoken Dialogue Systems T2 - Proceedings of Interspeech 2009 : speech and intelligence ; 6 - 10 September, 2009, Brighton, UK N2 - The potential of using ASR n-best lists for dialogue systems has often been recognised (if less often realised): it is often the case that even when the top-ranked hypothesis is erroneous, a bet- ter one can be found at a lower rank. In this paper, we describe metrics for evaluating whether the same potential carries over to incremental dialogue systems, where ASR output is consumed and reacted upon while speech is still ongoing. We show that even small N can provide an advantage for semantic process- ing, at a cost of a computational overhead. KW - dialogue systems KW - speech recognition KW - naturallanguage understanding KW - incrementality Y1 - 2009 U6 - https://doi.org/10.21437/Interspeech.2009-318 SP - 1031 EP - 1034 PB - ISCA CY - Brighton, UK ER - TY - CHAP A1 - Baumann, Timo A1 - Atterer, Michaela A1 - Schlangen, David T1 - Assessing and Improving the Performance of Speech Recognition for Incremental Systems T2 - NAACL '09: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, May 31 - June 5, 2009, Boulder, Colorado, USA N2 - In incremental spoken dialogue systems, par- tial hypotheses about what was said are re- quired even while the utterance is still ongo- ing. We define measures for evaluating the quality of incremental ASR components with respect to the relative correctness of the par- tial hypotheses compared to hypotheses that can optimize over the complete input, the tim- ingof hypothesisformationrelative to the por- tion ofthe inputthey areabout, andhypothesis stability, defined as the number of times they are revised. We show that simple incremen- tal post-processing can improve stability dra- matically, at the cost of timeliness (from 90% of edits of hypotheses being spurious down to 10% at a lag of 320ms). The measures are not independent,and we show how system de- signers can find a desired operating point for their ASR. To our knowledge, we are the first to suggest and examine a variety of measures for assessing incremental ASR and improve performance on this basis. Y1 - 2009 UR - https://dl.acm.org/doi/10.5555/1620754.1620810 SN - 978-1-932432-41-1 SP - 380 EP - 388 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Timo A1 - Buß, Okko A1 - Schlangen, David T1 - InproTK in Action: Open-Source Software for Building German-Speaking Incremental Spoken Dialogue Systems T2 - Electronic speech signal processing 2010 : proceedings of the 21st conference, Berlin, 8 - 10 September 2010 N2 - We present INPROTK, a toolkit for building incremental spoken dia-logue systems. Incremental spoken dialogue systems (systems that may react whilethe user’s utterance is ongoing) are a fairly recent research topic and allow for ex-citing new features. Even though toolkits exist that help in building conventionaldialogue systems, INPROTK offers both a tested architecture for building incre-mental SDSs as well as many of the building blocks necessary when building suchsystems. With INPROTK a researcher can avoid many of the technical difficulties,which hopefully further fosters research in this area. Y1 - 2010 UR - https://www.researchgate.net/publication/233540470_InproTK_in_Action_Open-Source_Software_for_Building_German-Speaking_Incremental_Spoken_Dialogue_Systems SP - 204 EP - 211 PB - TUDpress CY - Berlin, Germany ER - TY - CHAP A1 - Atterer, Michaela A1 - Baumann, Timo A1 - Schlangen, David T1 - No Sooner Said Than Done: Testing the Incrementality of Semantic Interpretations of Spontaneous Speech T2 - Proceedings of Interspeech 2009 : 6 - 10 September 2009, Brighton, U.K. N2 - Ideally, a spoken dialogue system should react without much delay to a user’s utterance. Such a system would already select an object, for instance, before the user has finished her utterance about moving this particular object to a particular place. A prerequisite for such a prompt reaction is that semantic representations are built up on the fly and passed on to other modules. Few approaches to incremental semantics construction exist, and, to our knowledge, none of those has been systematically tested on a spontaneous speech corpus. In this paper, we develop measures to test empirically on transcribed spontaneous speech to what extent we can create semantic interpretation on the fly with an incremental semantic chunker that builds a frame semantics. KW - incrementality KW - spoken dialogue systems KW - spontaneous speech KW - evaluation Y1 - 2009 U6 - https://doi.org/10.21437/Interspeech.2009-539 SP - 1855 EP - 1858 PB - International Speech Communication Association CY - Brighton, UK ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Predicting the Micro-Timing of User Input for an Incremental Spoken Dialogue System that Completes a User’s Ongoing Turn T2 - SIGDIAL 2011 Conference, 12th annual meeting of the Special Interest Group on Discourse and Dialogue, Co-located with ACL HLT 2011, Portland, Oregon, 17 - 18 June 2011 N2 - We present the novel task of predicting tem-poral features of continuations of user input,while that input is still ongoing. We show that the remaining duration of an ongoing word, aswell as the duration of the next can be predicted reasonably well, and we put this information touse in a system that synchronously completesa user’s speech. While we focus on collaborative completions, the techniques presented here may also be useful for the alignment of back-channels and immediate turn-taking in anincremental SDS, or to synchronously monitorthe user’s speech fluency for other reasons. Y1 - 2011 UR - https://www.researchgate.net/publication/220794517_Predicting_the_Micro-Timing_of_User_Input_for_an_Incremental_Spoken_Dialogue_System_that_Completes_a_User%27s_Ongoing_Turn SN - 978-1-61839-242-8 SP - 120 EP - 129 PB - Curran CY - Red Hook, NY ER - TY - CHAP A1 - Buschmeier, Hendrik A1 - Baumann, Timo A1 - Dorsch, Benjamin A1 - Kopp, Stefan A1 - Schlangen, David T1 - Combining Incremental Language Generation and Incremental Speech Synthesis for Adaptive Information Presentation T2 - SIGDIAL '12: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Seoul, South Korea, July 5 - 6, 2012 N2 - Participants in a conversation are normally receptive to their surroundings and their interlocutors, even while they are speaking and can, if necessary, adapt their ongoing utterance. Typical dialogue systems are not receptive and cannot adapt while uttering. We present combin-able components for incremental natural lan-guage generation and incremental speech syn-thesis and demonstrate the flexibility they can achieve with an example system that adapts to a listener's acoustic understanding problems by pausing, repeating and possibly rephrasing problematic parts of an utterance. In an evaluation, this system was rated as significantly more natural than two systems representing the current state of the art that either ignore the interrupting event or just pause; it also has a lower response time. Y1 - 2012 U6 - https://doi.org/10.5555/2392800.2392852 SP - 295 EP - 303 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Heintze, Silvan A1 - Baumann, Timo A1 - Schlangen, David T1 - Comparing Local and Sequential Models for Statistical Incremental Natural Language Understanding T2 - SIGDIAL '10: Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Tokyo, Japan,September 24 - 25, 2010 N2 - Incremental natural language understanding is the task of assigning semantic representations to successively larger prefixes of utterances. We compare two types of statistical models for this task: a) local models, which predict a single class for an input; and b), sequential models, which align a sequence of classes to a sequence of input tokens. We show that, with some modifications, the first type of model can be improved and made to approximate the output of the second, even though the latter is more informative. We show on two different data sets that both types of model achieve comparable performance (significantly better than a baseline), with the first type requiring simpler training data. Results for the first type of model have been reported in the literature; we show that for our kind of data our more sophisticated variant of the model performs better. Y1 - 2010 SN - 978-1-932432-85-5 U6 - https://doi.org/10.5555/1944506.1944508 PB - Association for Computational Linguistics CY - Tokyo, Japan ER - TY - JOUR A1 - Baumann, Timo A1 - Buß, Okko A1 - Schlangen, David T1 - Evaluation and Optimisation of Incremental Processors JF - Dialogue & Discourse N2 - Incremental spoken dialogue systems, which process user input as it unfolds, pose additionalengineering challenges compared to more standard non-incremental systems: Their processingcomponents must be able to accept partial, and possibly subsequently revised input, and mustproduce output that is at the same time as accurate as possible and delivered with as little delay aspossible. In this article, we define metrics that measure how well a given processor meets thesechallenges, and we identify types of gold standards for evaluation. We exemplify these metrics inthe evaluation of several incremental processors that we have developed. We also present genericmeans to optimise some of the measures, if certain trade-offs are accepted. We believe that thiswork will help enable principled comparison of components for incremental dialogue systems andportability of results. KW - evaluation KW - incrementality, KW - spoken dialogue systems KW - incremental processing KW - timing Y1 - 2011 U6 - https://doi.org/10.5087/dad.2011.106 VL - 2 IS - 1 SP - 113 EP - 141 PB - OJS ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Interactional Adequacy as a Factor in the Perception of Synthesized Speech T2 - Proceedings of Speech Synthesis Workshop (SSW8) N2 - Speaking as part of a conversation is different from reading out aloud. Speech synthesis systems, however, are typically developed using assumptions (at least implicitly) that are more true of the latter than the former situation. We address one particular aspect, which is the assumption that a fully formulated sentence is available for synthesis. We have built a system that does not make this assumption but rather can synthesize speech given incrementally extended input. In an evaluation experiment, we found that in a dynamic domain where what is talked about changes quickly, subjects rated the output of this system as more ‘naturally pronounced’ than that of a baseline system that employed standard synthesis, despite the synthesis quality objectively being degraded. Our results highlight the importance of considering a synthesizer’s ability to support interactive use-cases when determining the adequacy of synthesized speech. KW - speech synthesis KW - incremental processing KW - interactive behaviour KW - evaluation KW - adequacy Y1 - 2013 UR - https://www.isca-speech.org/archive_v0/ssw8/papers/ssw8_223.pdf CY - Barcelona, Spain ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Open-ended, Extensible System Utterances Are Preferred, Even If They Require Filled Pauses T2 - 14th annual meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL 2013) : Metz, France, 22 - 24 August 2013 N2 - In many environments (e. g. sports commentary), situations incrementally unfold over time and often the future appearance of a relevant event can be predicted, but not in all its details or precise timing. We have built a simulation framework that uses our incremental speech synthesis component to assemble in a timely manner complex commentary utterances. In our evaluation, the resulting output is preferred over that from a baseline system that uses a simpler commenting strategy. Even in cases where the incremental system overcommits temporally and requires a filled pause to wait for the upcoming event, the system is preferred over the baseline. Y1 - 2013 UR - https://aclanthology.org/W13-4042.pdf SP - 280 EP - 283 PB - Curran CY - Red Hook, NY ER - TY - CHAP A1 - Atterer, Michaela A1 - Baumann, Timo A1 - Schlangen, David T1 - Towards Incremental End-of-Utterance Detection in Dialogue Systems T2 - COLING '08: Proceedings of the 22nd International Conference on Computational Linguistics, Manchester United Kingdom August 18 - 22, 2008 N2 - We define the task of incremental or 0-lag utterance segmentation, that is, the task of segmenting an ongoing speech recognition stream into utterance units, and present first results. We use a combination of hidden event language model, features from an incremental parser, and acoustic / prosodic features to train classifiers on real-world conversational data (from the Switchboard corpus). The best classifiers reach an F-score of around 56%, improving over baseline and related work. Y1 - 2008 UR - https://www.researchgate.net/publication/221101778_Towards_Incremental_End-of-Utterance_Detection_in_Dialogue_Systems SN - 978-1-905593-44-6 SP - 11 EP - 14 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Evaluating Prosodic Processing for Incremental Speech Synthesis T2 - Proceedings of Interspeech 2012 N2 - Incremental speech synthesis (iSS) accepts input and produces output in consecutive chunks that only together result in a full utterance. Systems that use iSS thus have the ability to adapt their utterances while they are ongoing. Having available less than the full utterance to plan the acoustic realisation has downsides, however, as global optimisation is not possible anymore. In this paper we present a strategy for incrementalizing the symbolic pre-processing component of speech synthesis and assess the influence of a reduction in "lookahead", i. e. in knowledge about the rest of the utterance, on prosodic quality. We found that high quality incremental output can be achieved even with a lookahead of slightly less than one phrase, allowing for timely system reaction. KW - speech synthesis KW - spoken dialogue systems KW - incrementality KW - prosody Y1 - 2012 U6 - https://doi.org/10.21437/Interspeech.2012-152 SP - 438 EP - 441 PB - International Speech Communication Association ER - TY - CHAP A1 - Baumann, Timo A1 - Schlangen, David T1 - Inpro_iSS: A Component for Just-In-Time Incremental Speech Synthesis T2 - The 13th Conference of the European Chapter of the Association for Computational Linguistics - proceedings of the System Demonstrations, April 23 - 27 2012, Avignon France N2 - We present a component for incremental speech synthesis (iSS) and a set of applications that demonstrate its capabilities. This component can be used to increase the responsivity and naturalness of spoken interactive systems. While iSS can show its full strength in systems that generate output incrementally, we also discuss how even otherwise unchanged systems may profit from its capabilities. Y1 - 2012 PB - Association for Computational Linguistics (ACL) ER - TY - CHAP A1 - Peldszus, Andreas A1 - Buß, Okko A1 - Baumann, Timo A1 - Schlangen, David T1 - Joint Satisfaction of Syntactic and Pragmatic Constraints Improves Incremental Spoken Language Understanding T2 - EACL 2012 Joint Workshop of LINGVIS & UNCLH, Visualization of Linguistic Patterns and Uncovering Language History from Multilingual Resources, Proceedings of the Workshop, April 23 - 24 2012, Avignon France N2 - We present a model of semantic processing of spoken language that (a) is robust against ill-formed input, such as can be expected from automatic speech recognisers, (b) respects both syntactic and pragmatic constraints in the computation of most likely interpretations, (c) uses a principled, expressive semantic representation formalism (RMRS) with a well-defined model theory, and (d) works continuously (producing meaning representations on a word-by-word basis, rather than only for full utterances) and incrementally (computing only the additional contribution by the new word, rather than re-computing for the whole utterance-so-far). We show that the joint satisfaction of syntactic and pragmatic constraints improves the performance of the NLU component (around 10 % absolute, over a syntax-only baseline). Y1 - 2012 UR - https://www.researchgate.net/publication/233540473_Joint_Satisfaction_of_Syntactic_and_Pragmatic_Constraints_Improves_Incremental_Spoken_Language_Understanding SN - 978-1-937284-19-0 SP - 514 EP - 523 PB - The Association for Computational Linguistics ER - TY - CHAP A1 - Buß, Okko A1 - Baumann, Timo A1 - Schlangen, David T1 - Collaborating on Utterances with a Spoken Dialogue System Using an ISU-based Approach to Incremental Dialogue Management T2 - SIGDIAL 10, Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Tokyo Japan September 24 - 25, 2010 N2 - When dialogue systems, through theuse of incremental processing, arenot bounded anymore by strict, non-overlapping turn-taking, a whole range ofadditional interactional devices becomesavailable. We explore the use of one suchdevice, trial intonation. We elaborateour approach to dialogue managementin incremental systems, based on theInformation-State-Update approach, anddiscuss an implementation in a micro-domain that lends itself to the use ofimmediate feedback, trial intonations andexpansions. In an overhearer evaluation,the incremental system was judged as sig-nificantly more human-like and reactivethan a non-incremental version. Y1 - 2010 UR - https://www.researchgate.net/publication/220794407_Collaborating_on_Utterances_with_a_Spoken_Dialogue_System_Using_an_ISU-based_Approach_to_Incremental_Dialogue_Management SN - 978-1-932432-85-5 PB - Association for Computational Linguistics CY - Tokyo, Japan ER - TY - CHAP A1 - Schlangen, David A1 - Baumann, Timo A1 - Atterer, Michaela T1 - Incremental Reference Resolution: The Task, Metrics for Evaluation, and a Bayesian Filtering Model that is Sensitive to Disfluencies T2 - Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, September 11 - 12, 2009, London United Kingdom N2 - In this paper we do two things: a) we discuss in general terms the task of incre mental reference resolution (IRR), in particular resolution of exophoric reference, and specify metrics for measuring the performance of dialogue system components tackling this task, and b) we present a simple Bayesian filtering model of IRR that performs reasonably well just using words directly (no structure information and no hand-coded semantics): it picks the right referent out of 12 for around 50 % of real world dialogue utterances in our test corpus. It is also able to learn to interpret not only words but also hesitations, just as humans have shown to do in similar situations, namely as markers of references tohard-to-describe entities. Y1 - 2009 UR - https://clp.ling.uni-potsdam.de/publications/Schlangen-2009-2.pdf SP - 30 EP - 37 PB - Association for Computational Linguistics CY - London, UK ER - TY - CHAP A1 - Schlangen, David A1 - Baumann, Timo A1 - Buschmeier, Hendrik A1 - Buß, Okko A1 - Kopp, Stefan A1 - Skantze, Gabriel A1 - Yaghoubzadeh, Ramin T1 - Middleware for Incremental Processing in Conversational Agents T2 - Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SigDial 2010), September 24 - 25, 2010, Tokyo Japan N2 - We describe work done at three sites on designing conversational agents capable of incremental processing. We focus on the ‘middleware’ layer in these systems, which takes care of passing around and maintaining incremental information between the modules of such agents. All implementations are based on the abstract model of incremental dialogue processing proposed by Schlangen and Skantze (2009), and the paper shows what different instantiations of the model can look like given specific requirements and application areas. Y1 - 2010 UR - https://www.researchgate.net/publication/47735874_Middleware_for_Incremental_Processing_in_Conversational_Agents SP - 51 EP - 54 PB - Association for Computational Linguistics CY - Tokyo, Japan ER - TY - CHAP A1 - van Welbergen, Herwin A1 - Baumann, Timo A1 - Schlangen, David A1 - Kopp, Stefan T1 - Incremental, Adaptive and Interruptive Speech Realization for Fluent Conversation with ECAs T2 - Intelligent Virtual Agents N2 - This project aimed to establish the feasibility of creating a procedural system for generating expressive facial animation based on an affective agent. A procedural system supporting a limited number of emotional expression changes was created alongside keyframed animations of these same emotional expression changes, and audience response to these two approaches was tested empirically. Results seem to partially support the procedural animations generated being comparable with keyframed, in terms of perceptual validity. KW - artificial intelligence KW - cognitive modeling KW - gestures KW - nonverbal communication KW - social interaction Y1 - 2013 UR - https://link.springer.com/content/pdf/bbm:978-3-642-40415-3/1?pdf=chapter%20toc SN - 978-3-642-40414-6 SP - 468 EP - 469 PB - Springer ER - TY - CHAP A1 - von der Malsburg, Titus A1 - Baumann, Timo A1 - Schlangen, David T1 - TELIDA: A Package for Manipulation and Visualisation of Timed Linguistic Data T2 - Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, September 11 - 12, 2009, London United Kingdom N2 - We present a toolkit for manipulating andvisualising time-aligned linguistic datasuch as dialogue transcripts or languageprocessing data. The package comple-ments existing editing tools by allowingfor conversion between their formats, in-formation extraction from the raw files,and by adding sophisticated, and easily ex-tended methods for visualising the dynam-ics of dialogue processing. To illustratethe versatility of the package, we describeits use in three different projects at our site. Y1 - 2009 U6 - https://doi.org/10.5555/1708376.1708419 SP - 302 EP - 305 PB - Association for Computational Linguistics CY - London, UK ER - TY - CHAP A1 - Baumann, Timo A1 - Pohl, Martin A1 - Selgrad, Kai ED - Schäfle, Claudia ED - Dölling, Hanna T1 - Einführung und erste Evaluation des Selbstlernkonzepts CodeClinic an der OTH Regensburg T2 - Tagungsband MINT Symposium 2025. Zukunft MINT Lehre: Was bleibt? Was kommt? Was wirkt? N2 - Mit der CodeClinic wollen wir das Programmieren-Lernen – eine der großen Einstiegshürden in Informatik- und informatiknahen Studiengängen – in einen Prozess überführen, der das gemeinschaftliche Lernen in der Hochschullehre stärker als bisher in den Vordergrund rückt. Davon profitieren Studierende in ihrer fachlichen und persönlichen Entwicklung, und das Lernumfeld spiegelt die tatsächliche Realität der beruflichen Praxis besser wider. Wir unterstützen dieses Konzept durch einen für selbstgesteuertes und gemeinschaftliches Lernen ausgelegten SCALE-UP-Raum, den wir an der OTH Regensburg eingerichtet haben. In unserem Beitrag erläutern wir Konzept und Raum und liefern eine erste Evaluation zum Erfolg unserer Maßnahmen. KW - Hochschuldidaktik KW - Programmierung KW - Studieneingangsphase KW - SCALE-UP KW - Lernen durch Lehren (LdL) KW - BayernMINT Y1 - 2025 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-86138 SN - 1612-4537 PB - BayZiel – Bayerisches Zentrum für Innovative Lehre CY - München ER -