Detecting Emotion Carriers by Combining Acoustic and Lexical Representations

  • Personal narratives (PN) - spoken or written - are recollections of facts, people, events, and thoughts from one's own experience. Emotion recognition and sentiment analysis tasks are usually defined at the utterance or document level. However, in this work, we focus on Emotion Carriers (EC) defined as the segments (speech or text) that best explain the emotional state of the narrator ("loss of father", "made me choose"). Once extracted, such EC can provide a richer representation of the user state to improve natural language understanding and dialogue modeling. In previous work, it has been shown that EC can be identified using lexical features. However, spoken narratives should provide a richer description of the context and the users' emotional state. In this paper, we leverage word-based acoustic and textual embeddings as well as early and late fusion techniques for the detection of ECs in spoken narratives. For the acoustic word-level representations, we use Residual Neural Networks (ResNet) pretrained on separate speech emotionPersonal narratives (PN) - spoken or written - are recollections of facts, people, events, and thoughts from one's own experience. Emotion recognition and sentiment analysis tasks are usually defined at the utterance or document level. However, in this work, we focus on Emotion Carriers (EC) defined as the segments (speech or text) that best explain the emotional state of the narrator ("loss of father", "made me choose"). Once extracted, such EC can provide a richer representation of the user state to improve natural language understanding and dialogue modeling. In previous work, it has been shown that EC can be identified using lexical features. However, spoken narratives should provide a richer description of the context and the users' emotional state. In this paper, we leverage word-based acoustic and textual embeddings as well as early and late fusion techniques for the detection of ECs in spoken narratives. For the acoustic word-level representations, we use Residual Neural Networks (ResNet) pretrained on separate speech emotion corpora and fine-tuned to detect EC. Experiments with different fusion and system combination strategies show that late fusion leads to significant improvements for this task.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Sebastian P. BayerlORCiD, Aniruddha Tammewar, Korbinian Riedhammer, Giuseppe Riccardi
DOI:https://doi.org/10.48550/arXiv.2112.06603
Document Type:conference proceeding (article)
Language:English
Date of first Publication:2021/12/13
Reviewed:Begutachtet/Reviewed
Release Date:2024/07/03
Tag:emotion carrier, speech emotion recognition, natural language understanding
Pagenumber:8
Konferenzangabe:IEEE Workshop on Speech Recognition and Understanding (ASRU)
institutes:Fakultät Informatik
Licence (German):Keine Lizenz - Deutsches Urheberrecht gilt
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.