TY - CHAP A1 - Tammewar, Aniruddha A1 - Braun, Franziska A1 - Roccabruna, Gabriel A1 - Bayerl, Sebastian P. A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - Annotation of Valence for Spoken Personal Narratives N2 - Personal Narrative (PN) is the recollection of individuals’ life experiences, events, and thoughts along with the associated emotions in the form of a story. Compared to other genres such as social media texts or microblogs, where people write about ex-perienced events or products, the spoken PNs are complex to analyze and understand. They are usually long and unstructured, involving multiple and related events, characters as well as thoughts and emotions associated with events, objects, and persons. In spoken PNs, emotions are conveyed by changing the speech signal characteristics as well as the lexical content of the narrative. In this work, we annotate a corpus of spoken personal narratives, with the emotion valence using discrete values. The PNs are segmented into speech segments, and the annotators annotate them in the discourse context, with values on a 5 point bipolar scale ranging from -2 to +2 (0 for neutral). In this way, we capture the unfolding of the PNs events and changes in the emotional state of the narrator. We perform an in-depth analysis of the inter-annotator agreement, the relation between the label distribution w.r.t. the stimulus (positive/negative) used for the elicitation of the narrative, and compare the segment-level annotations to a baseline continuous annotation. We find that the neutral score plays an important role in the agreement. We observe that it is easy to differentiate the positive from the negative valence while the confusion with the neutral label is high. KW - Personal Narratives, Emotion Annotation, Segment Level Annotation Y1 - 2022 ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Roccabruna, Gabriel A1 - Chowdhury, ShammurA A1 - Ciulli, Tommaso A1 - Danieli, Morena A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - What can Speech and Language Tell us About the Working Alliance in Psychotherapy N2 - We are interested in the problem of conversational analysis and its application to the health domain. Cognitive Behavioral Therapy is a structured approach in psychotherapy, allowing the therapist to help the patient to identify and modify the malicious thoughts, behavior, or actions. This cooperative effort can be evaluated using the Working Alliance Inventory Observer-rated Shortened - a 12 items inventory covering task, goal, and relationship - which has a relevant influence on therapeutic outcomes. In this work, we investigate the relation between this alliance inventory and the spoken conversations (sessions) between the patient and the psychotherapist. We have delivered eight weeks of e-therapy, collected their audio and video call sessions, and manually transcribed them. The spoken conversations have been annotated and evaluated with WAI ratings by professional therapists. We have investigated speech and language features and their association with WAI items. The feature types include turn dynamics, lexical entrainment, and conversational descriptors extracted from the speech and language signals. Our findings provide strong evidence that a subset of these features are strong indicators of working alliance. To the best of our knowledge, this is the first and a novel study to exploit speech and language for characterising working alliance. KW - conversational analysis, working alliance, psy- chotherapy Y1 - 2022 U6 - https://doi.org/10.48550/arXiv.2206.08835 ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Tammewar, Aniruddha A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - Detecting Emotion Carriers by Combining Acoustic and Lexical Representations N2 - Personal narratives (PN) - spoken or written - are recollections of facts, people, events, and thoughts from one's own experience. Emotion recognition and sentiment analysis tasks are usually defined at the utterance or document level. However, in this work, we focus on Emotion Carriers (EC) defined as the segments (speech or text) that best explain the emotional state of the narrator ("loss of father", "made me choose"). Once extracted, such EC can provide a richer representation of the user state to improve natural language understanding and dialogue modeling. In previous work, it has been shown that EC can be identified using lexical features. However, spoken narratives should provide a richer description of the context and the users' emotional state. In this paper, we leverage word-based acoustic and textual embeddings as well as early and late fusion techniques for the detection of ECs in spoken narratives. For the acoustic word-level representations, we use Residual Neural Networks (ResNet) pretrained on separate speech emotion corpora and fine-tuned to detect EC. Experiments with different fusion and system combination strategies show that late fusion leads to significant improvements for this task. KW - emotion carrier, speech emotion recognition, natural language understanding Y1 - 2021 U6 - https://doi.org/10.48550/arXiv.2112.06603 ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Roccabruna, Gabriel A1 - Chowdhury, Shammur Absar A1 - Ciulli, Tommaso A1 - Danieli, Morena A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - What can Speech and Language Tell us About the Working Alliance in Psychotherapy T2 - Interspeech 2022 N2 - We are interested in the problem of conversational analysis and its application to the health domain. Cognitive Behavioral Therapy is a structured approach in psychotherapy, allowing the therapist to help the patient to identify and modify the malicious thoughts, behavior, or actions. This cooperative effort can be evaluated using the Working Alliance Inventory Observer-rated Shortened – a 12 items inventory covering task, goal, and relationship – which has a relevant influence on therapeutic outcomes. In this work, we investigate the relation between this alliance inventory and the spoken conversations (sessions) between the patient and the psychotherapist. We have delivered eight weeks of e-therapy, collected their audio and video call sessions, and manually transcribed them. The spoken conversations have been annotated and evaluated with WAI ratings by professional therapists. We have investigated speech and language features and their association with WAI items. The feature types include turn dynamics, lexical entrainment, and conversational descriptors extracted from the speech and language signals. Our findings provide strong evidence that a subset of these features are strong indicators of working alliance. To the best of our knowledge, this is the first and a novel study to exploit speech and language for characterising working alliance. KW - conversational analysis KW - working alliance KW - psychotherapy Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-347 SP - 2443 EP - 2447 PB - ISCA CY - ISCA ER -