TY - CHAP A1 - Windbühler, Anna A1 - Okkesim, Sükrü A1 - Christ, Olaf A1 - Mottaghi, Soheil A1 - Rastogi, Shavika A1 - Schmuker, Michael A1 - Baumann, Timo A1 - Hofmann, Ulrich G. T1 - Machine Learning Approaches to Classify Anatomical Regions in Rodent Brain from High Density Recordings T2 - 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2022): 11-15 July 2022, Glasgow, Scotland, United Kingdom N2 - Identifying different functional regions during a brain surgery is a challenging task usually performed by highly specialized neurophysiologists. Progress in this field may be used to improve in situ brain navigation and will serve as an important building block to minimize the number of animals in preclinical brain research required by properly positioning implants intraoperatively. The study at hand aims to correlate recorded extracellular signals with the volume of origin by deep learning methods. Our work establishes connections between the position in the brain and recorded high-density neural signals. This was achieved by evaluating the performance of BLSTM, BGRU, QRNN and CNN neural network architectures on multisite electrophysiological data sets. All networks were able to successfully distinguish cortical and thalamic brain regions according to their respective neural signals. The BGRU provides the best results with an accuracy of 88.6 % and demonstrates that this classification task might be solved in higher detail while minimizing complex preprocessing steps. Y1 - 2022 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-35296 SP - 3530 EP - 3533 PB - IEEE ER - TY - CHAP A1 - Nayak, Shravan A1 - Schuler, Christian A1 - Saha, Debjoy A1 - Baumann, Timo ED - Tumuluri, Raj ED - Sebe, Nicu ED - Pingali, Gopal T1 - A Deep Dive Into Neural Synchrony Evaluation for Audio-visual Translation T2 - ICMI '22, Proceedings of the 2022 International Conference on Multimodal Interaction: November 7 - 11 2022, Bengaluru India N2 - We present a comprehensive analysis of the neural audio-visual synchrony evaluation tool SyncNet. We assess the agreement of SyncNet scores vis-a-vis human perception and whether we can use these as a reliable metric for evaluating audio-visual lip-synchrony in generation tasks with no ground truth reference audio-video pair. We further look into the underlying elements in audio and video which vitally affect synchrony using interpretable explanations from SyncNet predictions and analyse its susceptibility by introducing adversarial noise. SyncNet has been used in numerous papers on visually-grounded text-to-speech for scenarios such as dubbing. We focus on this scenario which features many local asynchronies (something that SyncNet isn’t made for). KW - audio-visual synchrony KW - speech-lip synchrony KW - dubbing Y1 - 2022 SN - 978-1-4503-9390-4 U6 - https://doi.org/10.1145/3536221.3556621 SP - 642 EP - 647 PB - Association for Computing Machinery CY - New York ER - TY - CHAP A1 - Melzer, Matthias A1 - Dünnweber, Jan A1 - Baumann, Timo T1 - Towards Smart Home Data Interpretation Using Analogies to Natural Language Processing T2 - 2022 IEEE International Conference on Smart Internet of Things: IEEE SmartIoT 2022, proceedings, 19-21 August 2022, Suzhou, China, hybrid conference (onsite and virtual) N2 - Recent advances in the development of smart homes have led to the availability of a wide variety of devices providing a high level of convenience via gesture and speech control or fully automated operation. Many smart home appliances also address the aspects of safety and electricity savings by automatically powering themselves off after not being used for a while. However, many devices remain in a typical household that are not themselves "smart", or are not primarily electric (such as heating systems). We address the savings aspect by identifying processes involving the use of multiple devices in the electrical flow data, as captured by a smart meter in a modern household, rather than focusing on a single appliance. Therefore, we introduce a novel approach to usage pattern analysis based on the idea that a pattern of device usages as a result of a resident's 'routine' (such as making breakfast) can be interpreted similarly to a natural language 'sentence'; Natural Language Processing (NLP) algorithms can then be used for interpreting the residents' behavior. We introduce the notion of bag-of-devices (BoD), derived from the bag-of-words model used in document classification. In an experiment, we show how we use this model to infer predictions about the inhabitants from device usage, such as the resident leaving for the day or just to fetch the newspaper. KW - IoT KW - AI KW - NLP KW - Smart City Y1 - 2022 SN - 978-1-6654-7952-3 U6 - https://doi.org/10.1109/SmartIoT55134.2022.00020 SP - 65 EP - 71 PB - IEEE ER - TY - CHAP A1 - Saha, Debjoy A1 - Nayak, Shravan A1 - Baumann, Timo T1 - Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcasts T2 - Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), Marseille, 20-25 June 2022 N2 - We introduce the Merkel Podcast Corpus, an audio-visual-text corpus in German collected from 16 years of (almost) weekly Internet podcasts of former German chancellor Angela Merkel. To the best of our knowledge, this is the first single speaker corpus in the German language consisting of audio, visual and text modalities of comparable size and temporal extent. We describe the methods used with which we have collected and edited the data which involves downloading the videos, transcripts and other metadata, forced alignment, performing active speaker recognition and face detection to finally curate the single speaker dataset consisting of utterances spoken by Angela Merkel. The proposed pipeline is general and can be used to curate other datasets of similar nature, such as talk show contents. Through various statistical analyses and applications of the dataset in talking face generation and TTS, we show the utility of the dataset. We argue that it is a valuable contribution to the research community, in particular, due to its realistic and challenging material at the boundary between prepared and spontaneous speech. Accepted at LREC 2022 KW - corpus KW - speaker diarization KW - multi-modal KW - forced alignment KW - single-speaker KW - cross-modal learning KW - German Y1 - 2022 UR - http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.270.pdf N1 - Preprint unter: https://arxiv.org/abs/2205.12194 SP - 2536 EP - 2540 ER -