TY - JOUR A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo T1 - Towards the Creation of a Poetry Translation Mapping System JF - Archives of Data Science, Series A N2 - The translation of poetry is a complex, multifaceted challenge: the translated text should communicate the same meaning, similar metaphoric expressions, and also match the style and prosody of the original poem. Research on machine poetry translation is existing since 2010, but for four reasons it is still rather insufficient: 1. The few approaches existing completely lack any knowledge about current developments in both lyric theory and translation theory. 2. They are based on very small datasets. 3. They mostly ignored the neural learning approach that superseded the long-standing dominance of phrase-based approaches within machine translation. 4. They have no concept concerning the pragmatic function of their research and the resulting tools. Our paper describes how to improve the existing research and technology for poetry translations in exactly these four points. With regards to 1) we will describe the “Poetics of Translation”. With regards to 2) we will introduce the Worlds largest corpus for poetry translations from lyrikline. With regards to 3) we will describe first steps towards a neural machine translation of poetry. With regards to 4) we will describe first steps towards the development of a poetry translation mapping system. Y1 - 2018 U6 - https://doi.org/10.5445/KSP/1000087327/21 SN - 2363-9881 VL - 5 IS - 1 SP - 1 EP - 15 PB - Towards the Creation of a Poetry Translation Mapping System ER - TY - PAT A1 - Ward, Nigel A1 - Baumann, Timo A1 - Karkhedkar, Shreyas A1 - Novick, David T1 - Dynamic control of voice codec data rate N2 - A method, system, and computer-usable non-transitory storage device for dynamic voice codec adaptation are disclosed. The voice codec adapts in real time to devote more bits to audio quality when it is most needed, and fewer bits to less important parts of utterances are disclosed. Dialog knowledge is utilized for compression opportunities to adjust the bitrate moment-by-moment, based on the inferred value of each frame. Frame importance and appropriate transmission fidelity is predicted based on prosodic features and models of dialog dynamics. This technique provides the same communications quality with less spectrum needs, fewer antennas, and less battery drain. Y1 - 2015 UR - https://assignment.uspto.gov/patent/index.html#/patent/search/resultAssignor?assignorName=BAUMANN,%20TIMO ER - TY - CHAP A1 - Johannßen, Dirk A1 - Biemann, Chris A1 - Remus, Steffen A1 - Baumann, Timo A1 - Scheffer, David T1 - GermEval 2020 Task 1 on the Classification and Regression of Cognitive and Motivational style from Text T2 - Proceedings of the 5th SwissText & 16th KONVENS Joint Conference 2020 N2 - This paper describes the tasks, databases, baseline systems, and summarizes submissions and results for the GermEval 2020 Shared Task 1 on the Classification and Regression of Cognitive and Motivational Style from Text. This shared task is divided into two subtasks, a regression task, and a classification task. Subtask 1 asks participants to reproduce a ranking of students based on average aptitude indicators such as different high school grades and different IQ scores. The second subtask aims to classify so-called implicit motives, which are projective testing procedures that can reveal unconscious desires. Besides five implicit motives, the target labels of Subtask 2 also contain one of six levels that describe the type of self-regulation when acting out a motive, which makes this task a multiclass-classification with 30 target labels. 3 participants submitted multiple systems. Subtask 1 was solved (best r = .3701) mainly with non-neural systems and statistical language representations, submissions for Subtask 2 utilized neural approaches and word embeddings (best macro F1 = 70.40). Not only were the tasks solvable, analyses by the participants even showed connections to the implicit psychometrics theory and behavioral observations made by psychologists. This paper describes the tasks, databases, baseline systems, and summarizes submissions and results for the GermEval 2020 Shared Task 1 on the Classification and Regression of Cognitive and Motivational Style from Text. This shared task is divided into two subtasks, a regression task, and a classification task. Subtask 1 asks participants to reproduce a ranking of students based on average aptitude indicators such as different high school grades and different IQ scores. The second subtask aims to classify so-called implicit motives, which are projective testing procedures that can reveal unconscious desires. Besides five implicit motives, the target labels of Subtask 2 also contain one of six levels that describe the type of self-regulation when acting out a motive, which makes this task a multiclass-classification with 30 target labels. 3 participants submitted multiple systems. Subtask 1 was solved (best r =.3701) mainly with non-neural systems and statistical language representations, submissions for Subtask 2 utilized neural approaches and word embeddings (best macro F1 = 70.40). Not only were the tasks solvable, analyses by the participants even showed connections to the implicit psychometrics theory and behavioral observations made by psychologists. Y1 - 2020 UR - https://www.inf.uni-hamburg.de/en/inst/ab/lt/publications/2020-johannssen-germeval20-1-companion-paper.pdf SP - 1 EP - 10 CY - Zurich, Switzerland ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Free Verse and Beyond: How to Classify Post-modern Spoken Poetry T2 - Proceedings of Speech Prosody: Tokyo, Japan, 25-28 May 2020 N2 - This paper presents the classification of rhythmical patterns detected in post-modern spoken poetry by means of machine learning algorithms that use manually engineered features or automatically learnt representations. We used the world's largest corpus of spoken poetry from our partner lyrikline. We identified nine rhythmical patterns within a spectrum raging from a more fluent to a more disfluent poetic style. The text data analyzed by a statistical parser. Prosodic features of rhythmical patterns are identified by using the parser information. For the classification of rhythmical patterns, we used a neural networks-based approach which use text, audio, and pause information between poetic lines as features. Different combinations of features as well as the integration of feature engineering in the neural networks-based approach are tested. We compared the performance of both approaches (feature-based and neural network-based) using combinations of different features. The results show – by using the weighted average of f-measure for the evaluation – that the neural networks-based approach performed much better in classification of rhythmical patterns. The important improvement of the classification results lies in the use of the audio information. The integration of feature engineering in the neural networks-based approach yielded a very small result improvement. Y1 - 2020 U6 - https://doi.org/10.21437/SpeechProsody.2020-141 SP - 690 EP - 694 PB - ISCA ER - TY - CHAP A1 - Baumann, Timo A1 - Meyer-Sickendiek, Burkhard ED - Krauwer, Steven ED - Fišer, Darja T1 - Deep Learning meets Post-modern Poetry T2 - TwinTalks 2020: Understanding and Facilitating Collaboration in Digital Humanities 2020, proceedings of the Twin Talks 2 and 3 workshops at DHN 2020 and DH 2020, Ottawa Canada and Riga Latvia, July 23 and October 20, 2020 N2 - We summarize our project Rhythmicalizer in which we analyze a corpus of post-modern poetry in a combination of qualitative hermeneutical and computational methods, as we have run the project over the course of the past three years (and preparing it for some time before that). Interdisciplinary work is always challenging and we here focus on some of the highlights of our collaboration. KW - Literary Studies KW - Machine Learning KW - Meta-Research Y1 - 2020 UR - http://ceur-ws.org/Vol-2717/paper03.pdf SN - 1613-0073 SP - 30 EP - 36 PB - RWTH Aachen ER - TY - CHAP A1 - Baumann, Timo T1 - How a Listener Influences the Speaker T2 - Proceedings of Speech Prosody: Tokyo, Japan, 25-28 May 2020 N2 - Listeners typically provide feedback while listening to a speaker in conversation and thereby engage in the co-construction of the interaction. We analyze the influence of the listener on the speaker by investigating how her verbal feedback signals help in modeling the speaker's language. We find that feedback from the listener may help in modeling the speaker's language, whether through the listener's feedback as transcribed, or the acoustic signal directly. We find the largest positive effects for end of sentence as well as for pauses mid-utterance, but also effects that indicate we successfully model elaborations of ongoing utterances that may result from the presence or absence of listener feedback. KW - CNN KW - conversation KW - feedback KW - language model KW - RNNLM Y1 - 2020 U6 - https://doi.org/10.21437/SpeechProsody.2020-198 SP - 970 EP - 974 PB - ISCA ER - TY - JOUR A1 - Baumann, Timo A1 - Köhn, Arne A1 - Hennig, Felix T1 - The Spoken Wikipedia Corpus collection: Harvesting, alignment and an application to hyperlistening JF - Language Resources and Evaluation N2 - Spoken corpora are important for speech research, but are expensive to create and do not necessarily reflect (read or spontaneous) speech ‘in the wild’. We report on our conversion of the preexisting and freely available Spoken Wikipedia into a speech resource. The Spoken Wikipedia project unites volunteer readers of Wikipedia articles. There are initiatives to create and sustain Spoken Wikipedia versions in many languages and hence the available data grows over time. Thousands of spoken articles are available to users who prefer a spoken over the written version. We turn these semi-structured collections into structured and time-aligned corpora, keeping the exact correspondence with the original hypertext as well as all available metadata. Thus, we make the Spoken Wikipedia accessible for sustainable research. We present our open-source software pipeline that downloads, extracts, normalizes and text–speech aligns the Spoken Wikipedia. Additional language versions can be exploited by adapting configuration files or extending the software if necessary for language peculiarities. We also present and analyze the resulting corpora for German, English, and Dutch, which presently total 1005 h and grow at an estimated 87 h per year. The corpora, together with our software, are available via http://islrn.org/resources/684-927-624-257-3/. As a prototype usage of the time-aligned corpus, we describe an experiment about the preferred modalities for interacting with information-rich read-out hypertext. We find alignments to help improve user experience and factual information access by enabling targeted interaction. KW - Annotation KW - Eyes-free speech access KW - Found data KW - Robust text–speech alignment KW - Speech corpus KW - Spoken hypertext KW - Wikipedia Y1 - 2019 U6 - https://doi.org/10.1007/s10579-017-9410-y VL - 53 IS - 2 SP - 303 EP - 329 PB - Springer Nature ER - TY - CHAP A1 - Saboo, Ashutosh A1 - Baumann, Timo ED - Bojar, Ondřej ED - Chatterjee, Rajen ED - Federmann, Christian ED - Fishel, Mark ED - Graham, Yvette ED - Haddow, Barry ED - Huck, Matthias ED - Yepes, Antonio Jimeno ED - Koehn, Philipp ED - Martins, André ED - Monz, Christof ED - Negri, Matteo ED - Névéol, Aurélie ED - Neves, Mariana ED - Post, Matt ED - Turchi, Marco ED - Verspoor, Karin T1 - Integration of Dubbing Constraints into Machine Translation T2 - Proceedings of the Fourth Conference on Machine Translation, Volume 1: Research Papers N2 - Translation systems aim to perform a meaning-preserving conversion of linguistic material (typically text but also speech) from a source to a target language (and, to a lesser degree, the corresponding socio-cultural contexts). Dubbing, i.e., the lip-synchronous translation and revoicing of speech adds to this constraints about the close matching of phonetic and resulting visemic synchrony characteristics of source and target material. There is an inherent conflict between a translation’s meaning preservation and ‘dubbability’ and the resulting trade-off can be controlled by weighing the synchrony constraints. We introduce our work, which to the best of our knowledge is the first of its kind, on integrating synchrony constraints into the machine translation paradigm. We present first results for the integration of synchrony constraints into encoder decoder-based neural machine translation and show that considerably more ‘dubbable’ translations can be achieved with only a small impact on BLEU score, and dubbability improves more steeply than BLEU degrades. Y1 - 2019 U6 - https://doi.org/10.18653/v1/W19-5210 SP - 94 EP - 101 PB - Association for Computational Linguistics CY - Stroudsburg, PA, USA ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo T1 - Identification of Concrete Poetry within a Modern-Poetry Corpus using Neural Networks T2 - Proceedings of the Quantitative Approaches to Versification Conference N2 - This work aims to discern the poetics of concrete poetry by using a corpus-based classification focusing on the two most important techniques used within concrete poetry: semantic decomposition and syntactic permutation. We demonstrate how to identify concrete poetry in modern and postmodern free verse. A class contrasting to concrete poetry is defined on the basis of poems with complete and correct sentences. We used the data from lyrikline, which contain both the written as well as the spoken form of poems as read by the original author. We explored two approaches for the identification of concrete poetry. The first is based on the definition of concrete poetry in literary theory by the extraction of various types of features derived from a parser, such as verb, noun, comma, sentence ending, conjunction, and asemantic material. The second is a neural network-based approach, which is theoretically less informed by human insight, as it does not have access to features established by scholars. This approach used the following inputs: textual information and the spoken recitation of poetic lines as well as information about pauses between lines. The results based on the neural network are more accurate than the feature-based approach. The best results, calculated by the weighted F-measure, for the classification of concrete poetry vis-à-vis the contrasting class is 0.96 Y1 - 2019 UR - https://versologie.cz/conference2019/proceedings/hussein-meyer-sickendiek-baumann.pdf SP - 95 EP - 104 CY - Prague, Czech Republic ER - TY - CHAP A1 - Baumann, Timo ED - Weiss, Benjamin ED - Trouvain, Jürgen ED - Barkat-Defradas, Mélissa ED - Ohala, John J. T1 - Ranking and Comparing Speakers Based on Crowdsourced Pairwise Listener Ratings T2 - Voice attractiveness: Studies on Sexy, Likable, and Charismatic Speakers N2 - Speech quality and likability is a multi-faceted phenomenon consisting of a combination of perceptory features that cannot easily be computed nor weighed automatically. Yet, it is often easy to decide which of two voices one likes better, even though it would be hard to describe why, or to name the underlying basic perceptory features. Although likability is inherently subjective and individual preferences differ, generalizations are useful and there is often a broad intersubjective consensus about whether one speaker is more likeable than another. We present a methodology to efficiently create a likability ranking for many speakers from crowdsourced pairwise likability ratings which focuses manual rating effort on pairs of similar quality using an active sampling technique. Using this methodology, we collected pairwise likability ratings for many speakers (>220) from many raters (>160). We analyze listener preferences by correlating the resulting ranking with various acoustic and prosodic features. We also present a neural network that is able to model the complexity of listener preferences and the underlying temporal evolution of features. The recurrent neural network achieves remarkably high performance in estimating the pairwise decisions and an ablation study points toward the criticality of modeling temporal aspects in speech quality assessment. KW - Crowdsourcing KW - Found data KW - Likability ratings KW - Ranking KW - Sequence modelling KW - Speech quality Y1 - 2021 SN - 978-981-15-6626-4 SN - 978-981-15-6627-1 U6 - https://doi.org/10.1007/978-981-15-6627-1_14 SP - 263 EP - 279 PB - Springer CY - Singapore ER - TY - CHAP A1 - Tsai, Vivian A1 - Baumann, Timo A1 - Pecune, Florian A1 - Cassell, Justine ED - D'Haro, Luis Fernando ED - Banchs, Rafael E. ED - Li, Haizhou T1 - Faster Responses Are Better Responses: Introducing Incrementality into Sociable Virtual Personal Assistants T2 - 9th International Workshop on Spoken Dialogue System Technology N2 - Speech-based interactive systems, such as virtual personal assistants, inevitably use complex architectures, with a multitude of modules working in series (or less often in parallel) to perform a task (e.g., giving personalized movie recommendations via dialog). Add modules for evoking and sustaining sociability with the user and the accumulation of processing latencies through the modules results in considerable turn-taking delays. We introduce incremental speech processing into the generation pipeline of the system to overcome this challenge with only minimal changes to the system architecture, through partial underspecification that is resolved as necessary. A user study with a sociable movie recommendation agent objectively diminishes turn-taking delays; furthermore, users not only rate the incremental system as more responsive, but also rate its recommendation performance as higher. Y1 - 2019 SN - 978-981-13-9442-3 SN - 978-981-13-9445-4 SN - 978-981-13-9443-0 U6 - https://doi.org/10.1007/978-981-13-9443-0_10 SN - 1876-1119 SN - 1876-1100 VL - 579 SP - 111 EP - 118 PB - Springer Singapore CY - Singapore ER - TY - CHAP A1 - Nayak, Shravan A1 - Baumann, Timo A1 - Bhattacharya, Supratik A1 - Karakanta, Alina A1 - Negri, Matteo A1 - Turchi, Marco ED - Truong, Khiet T1 - See me Speaking? Differentiating on Whether Words are Spoken On Screen or Off to Optimize Machine Dubbing T2 - ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction, 25.10.2020 - 29.10.2020; Virtual Event Netherlands N2 - Dubbing is the art of finding a translation from a source into a target language that can be lip-synchronously revoiced, i. e., that makes the target language speech appear as if it was spoken by the very actors all along. Lip synchrony is essential for the full-fledged reception of foreign audiovisual media, such as movies and series, as violated constraints of synchrony between video (lips) and audio (speech) lead to cognitive dissonance and reduce the perceptual quality. Of course, synchrony constraints only apply to the translation when the speaker's lips are visible on screen. Therefore, deciding whether to apply synchrony constraints requires an automatic method for detecting whether an actor's lips are visible on screen for a given stretch of speech or not. In this paper, we attempt, for the first time, to classify on- from off-screen speech based on a corpus of real-world television material that has been annotated word-by-word for the visibility of talking lips on screen. We present classification experiments in which we classify Y1 - 2020 SN - 9781450380027 U6 - https://doi.org/10.1145/3395035.3425640 SP - 130 EP - 134 PB - Association for Computing Machinery CY - New York,NY,United States ER -