TY - CHAP A1 - Milde, Benjamin A1 - Geislinger, Robert A1 - Lindt, Irina A1 - Baumann, Timo T1 - Open Source Automatic Lecture Subtitling T2 - Proceedings of Elektronische Sprachsignalverarbeitung (ESSV): Tagungsband der 32. Konferenz, Berlin, 3.-5. März 2021 N2 - We present a fully automatic solution for German video subtitling, with a focus on lecture videos. We rely entirely on open source models and scripts for German ASR, automatic punctuation reconstruction and subtitle segmentation. All training scripts, 1000h of German speech training data, pre-trained models and the final subtitling program are publicly available. It can readily be integrated into lecture video platforms such as Lecture2Go. The automatically generated subtitles can also serve as a basis to make the video material more accessible (e.g. via search, keyword clouds, and the like) or for further manual revision, potentially helping in significantly speeding up manual work. A particular challenge that we observe in lectures are technical terms that are frequent in a particular lecture, but infrequent in a typical language model and that might be out of vocabulary for a general purpose ASR. We approach this challenge by extracting texts from accompanying lecture slides to adapt the language model of our TDNN-HMM based ASR system. We demonstrate the usability of the full system and its generated subtitles and evaluate on a dataset of manually transcribed lectures with an average of 26.3% WER. KW - automatische Spracherkennung KW - freie Software KW - Untertitelung Y1 - 2021 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-2523 UR - https://www.essv.de/pdf/2021_128_135.pdf?id=1109 SP - 128 EP - 135 PB - Förderverein Elektronische Sprachsignalverabeitung e.V. CY - Berlin, Germany ER - TY - CHAP A1 - Hussein, Hussein A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo ED - Birkholz, Peter ED - Stone, Simon T1 - How to identify elliptical poems within a digital corpus of auditory poetry T2 - Elektronische Sprachsignalverarbeitung 2019 (ESSV), Tagungsband der 30. Konferenz, Dresden, 3.-8. März 2019 N2 - Ellipses denote the omission of one or more grammatically necessary phrases. In this paper, we will demonstrate how to identify such ellipses as a rhythmical pattern in modern and postmodern free verse poetry by using data from lyrikline which contain the corresponding audio recording of each poem as spoken by the original author. We present a feature engineering approach based on literary analysis as well as a neural networks based approach for the identification of ellipses within the lines of a poem. A contrast class to the ellipsis is defined from poems consisting of complete and correct sentences. The feature-based approach used features derived from a parser such as verb, comma, and sentence ending punctuation. The classifier of neural networks is trained on the line level to integrate the textual information, the spoken recitation, and the pause information between lines, and to integrate information across the lines within the poem. A statistic analysis of poet's gender showed that 65% of all elliptical poems were written by female poets. The best results, calculated by the weighted F-measure, for the classification of ellipsis with the contrast class is 0.94 with the neural networks based approach. The best results for classification of elliptical lines is 0.62 with the feature-based approach. Y1 - 2019 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-2579 UR - http://www.essv.de/paper.php?id=88 SN - 978-3-959081-57-3 SP - 247 EP - 254 PB - TUDpress CY - Dresden, Germany ER - TY - CHAP A1 - Baumann, Timo A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein T1 - How to Identify Speech When Translating Unpunctuated Poetry T2 - Proceedings of Elektronische Sprachsignalverarbeitung: Tagungsband der 31. Konferenz, Magdeburg, 4.-6. März 2020 N2 - A large proportion of (post)-modern poetry contains no or hardly any punctuation. In our contribution, we will investigate how well punctuation information can be recovered for postmodern poetry based on the information contained in the text and speech of free verse poems. We use the world's largest corpus of spoken (post-)modern poetry from our partner lyrikline which contains the corresponding audio recording of each poem as spoken by the original author and features translations for many of the poems. We identify lines that contain a phrase break in the middle of the poetic line, which may already be helpful for philological analysis on one hand, and identify the position of the break in the line on the other hand. We select those poetic lines that contain one or more punctuation characters that typically indicate a phrase break in poetry (.,;:!?/) somewhere in the middle (rather than only at the end of the line) as our target class. We train a neural network (bidirectional recurrent neural network (RNN) based on gated recurrent units (GRU) with attention) that combines audio and textual features to identify the punctuation with the goal of applying it to reconstruct them within a corpus of unpunctuated poems. Our results clearly indicate that speech is helpful for recovering the constituency structure of post-modern poetry that is partially obfuscated by missing punctuation. Y1 - 2020 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-2587 UR - http://www.essv.de/paper.php?id=452 SP - 165 EP - 172 PB - Förderverein Elektronische Sprachsignalverabeitung e.V. CY - Magdeburg, Germany ER - TY - CHAP A1 - Baumann, Timo A1 - Saboo, Ashutosh ED - Ehrmann, Maud T1 - Evaluating Heuristics for Audio-Visual Translation T2 - Proceedings of the Conference of Computational Humanities Research, (CHR2021): November 17-19, 2021, Amsterdam, The Netherlands N2 - Dubbing, i.e., the lip-synchronous translation and revoicing of audio-visual media into a target language from a different source language, is essential for the full-fledged reception of foreign audio-visual media, be it movies, instructional videos or short social media clips. In this paper, we objectify influences on the ‘dubbability’ of translations, i.e., how well a translation would be synchronously revoiceable to the lips on screen. We explore the value of traditional heuristics used in evaluating the qualitative aspects, in particular matching bilabial consonants and the jaw opening while producing vowels, and control for quantity, i.e., that translations are similar to the source in length. We perform an ablation study using an adversarial neural classifier which is trained to differentiate “true” dubbing translations from machine translations. While we are able to confirm the value of matching lip closure in dubbing, we find that the opening angle of the jaw as determined by the realized vowel may be less relevant than frequently considered in audio-visual translation. KW - ablation study KW - audiovisual translation KW - dubbing KW - lip synchrony KW - machine translation Y1 - 2021 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-2592 UR - http://ceur-ws.org/Vol-2989/short_paper46.pdf SP - 171 EP - 180 PB - CEUR Workshops CY - Amsterdam, the Netherlands ER - TY - CHAP A1 - Baumann, Timo T1 - Coordinating Speech Delivery to Gesture Progress for Deictic Expressions with Incremental Speech Synthesis T2 - Proceedings of the Timing in Human-Robot Interaction Workshop held in conjunction with the 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2014), Bielefeld, 3. March 2014 N2 - This paper presents ongoing work in incremental speech synthesis that enables a system to adapt speech delivery to unforeseen changes in the timing of motor events (e. g. a robot actuator working faster or slower than anticipated) in order to improve the coordination of speech and gestures for deictic expressions. KW - Incremental processing KW - speech synthesis KW - online adaptation KW - Inkrementelle Verarbeitung, KW - Sprachsynthese KW - Zeigegesten KW - Echtzeitadaptierung KW - Sprachverarbeitung Y1 - 2014 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-1991 UR - https://edoc.sub.uni-hamburg.de/informatik/volltexte/2014/199/pdf/timo_baumann_coordinating_speech_delivery.pdf ER - TY - GEN A1 - Meyer-Sickendiek, Burkhard A1 - Baumann, Timo A1 - Hussein, Hussein T1 - Requirements on the Punctuation Reconstruction for the Translation of Post-modern Poetry T2 - DHd 2020 ; Spielräume ; Digital Humanities zwischen Modellierung und Interpretation ; Konferenzabstracts ; Universität Paderborn 02. bis 06. März 2020 Y1 - 2020 SN - 978-3-945437-07-0 U6 - https://doi.org/10.5281/zenodo.4621722 PB - Universität Paderborn CY - Paderborn ER - TY - GEN A1 - Baumann, Timo ED - Devillers, Laurence ED - Kawahara, Tatsuya ED - Moore, Roger K. ED - Scheutz, Matthias T1 - Ubiquity of Computing and SLIVAR BT - Towards Effective and Ethical Interaction T2 - Spoken Language Interaction with Virtual Agents and Robots (SLIVAR): Towards Effective and Ethical Interaction KW - human-robot interaction KW - spoken language processing KW - virtual agents Y1 - 2020 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0030-drops-124009 VL - 10 PB - Schloss Dagstuhl ‐ Leibniz-Zentrum für Informatik CY - Dagstuhl, Germany ER - TY - GEN A1 - Meyer-Sickendiek, Burkhard A1 - Hussein, Hussein A1 - Baumann, Timo T1 - From Fluency To Disfluency: Ranking Prosodic Features Of Poetry By Using Neural Networks T2 - Proceedings of the International Conference on Digital Humanities (DH 2019), Utrecht, the Netherlands 9-12 July, 2019 Y1 - 2019 U6 - https://doi.org/10.34894/EGZNMI ER - TY - CHAP A1 - Windbühler, Anna A1 - Okkesim, Sükrü A1 - Christ, Olaf A1 - Mottaghi, Soheil A1 - Rastogi, Shavika A1 - Schmuker, Michael A1 - Baumann, Timo A1 - Hofmann, Ulrich G. T1 - Machine Learning Approaches to Classify Anatomical Regions in Rodent Brain from High Density Recordings T2 - 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2022): 11-15 July 2022, Glasgow, Scotland, United Kingdom N2 - Identifying different functional regions during a brain surgery is a challenging task usually performed by highly specialized neurophysiologists. Progress in this field may be used to improve in situ brain navigation and will serve as an important building block to minimize the number of animals in preclinical brain research required by properly positioning implants intraoperatively. The study at hand aims to correlate recorded extracellular signals with the volume of origin by deep learning methods. Our work establishes connections between the position in the brain and recorded high-density neural signals. This was achieved by evaluating the performance of BLSTM, BGRU, QRNN and CNN neural network architectures on multisite electrophysiological data sets. All networks were able to successfully distinguish cortical and thalamic brain regions according to their respective neural signals. The BGRU provides the best results with an accuracy of 88.6 % and demonstrates that this classification task might be solved in higher detail while minimizing complex preprocessing steps. Y1 - 2022 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-35296 SP - 3530 EP - 3533 PB - IEEE ER - TY - GEN A1 - Tsai, Vivian A1 - Baumann, Timo A1 - Pecune, Florian A1 - Cassell, Justine T1 - Faster Responses are Better Responses: Introducing Incrementality into Sociable Virtual Personal Assistants Y1 - 2018 UR - https://colips.org/conferences/iwsds2018/wp/wp-content/uploads/2018/05/IWSDS-2018_paper_42_presentation.pdf ER - TY - GEN A1 - Paetzel, Maike A1 - Schlesinger, Philipp A1 - Pricop, Mircea A1 - Comaneci, Radu A1 - Baumann, Timo A1 - Menzel, Wolfgang T1 - Inkrementelle und zustandsbasierte Verarbeitung in einem hybriden Sprachdialogsystem BT - Postersession der Sektion Computerlinguistik Y1 - 2013 UR - https://www.researchgate.net/publication/236879516_Inkrementelle_und_zustandsbasierte_Verarbeitung_in_einem_hybriden_Sprachdialogsystem UR - https://www.timobaumann.de/work/uploads/Main/pentomino2013dgfsposter.pdf ER - TY - CHAP A1 - Baumann, Timo A1 - Paetzel, Maike A1 - Schlesinger, Philipp A1 - Menzel, Wolfgang T1 - Using Affordances to Shape the Interaction in a Hybrid Spoken Dialogue System T2 - Elektronische Sprachsignalverarbeitung 2013 : Tagungsband der 24. Konferenz, Bielefeld, 26. - 28.3.2013 ; [ESSV 2013] N2 - Affordances manifest possibilities of interaction with a spoken dialogsystem. For example, the act of asking a question affords to the recipient thepossibility of answering. In the system we present, the observable act of maneuveringaffords the possibility of controlling a motion. Our system thus uses the affordanceprinciple to shape the interaction: to trigger the usage of instructions that are easyto understand and process, the system gives immediate visual feedback to afforduser commands that can then be reacted upon. This tightening of the interactionloop requires an incremental processing paradigm to allow fast reactions and to beable to alter ongoing system actions. Our system is a hybrid of incremental andnon-incremental processing components, combining conventional, state graph-basedprocessing, which has the advantage of widely available toolkits and well-understooddialog management, with incremental dialog processing which allows for the tightfeedback loop that provides for quick reactions. We tested our approach in a smalluser study and found that users used simpler and setting-independent commandsmore often and were more efficient when faced with the affordance-based version ofour system. Y1 - 2013 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:gbv:18-228-7-1836 UR - https://edoc.sub.uni-hamburg.de//informatik/volltexte/2013/183/ SN - 978-3-94431-03-4 SP - 12 EP - 19 PB - TUDpress ER - TY - BOOK A1 - Baumann, Timo T1 - Incremental Spoken Dialogue Processing: Architecture and Lower-level Components Y1 - 2012 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:hbz:361-25819101 SN - 9783848200221 N1 - Bielefeld, Univ., Diss., 2013 PB - Bielefeld University ER -