Comparing Large Language Models for Automated Subject Line Generation in e-Mental Health: A Performance Study

  • Large Language Models (LLMs) have the potential to enhance e-mental health and psychosocial e-mail counselling by automating tasks such as generating concise and relevant subject lines for client communications. However, concerns regarding accuracy, reliability, data privacy and resource efficiency persist. This study investigates the performance of several LLMs in generating subject lines for e-mail threads, yielding a total of 253 generated subjects. Each subject line was assessed by six raters, including five counselling professionals and one AI system, using a three-category quality scale (Good, Fair, Poor). The results show that LLMs can generally produce concise subject lines considered helpful by experts. While GPT-4o and GPT-3.5 Turbo outperformed other models, their use is restricted in mental health settings due to data protection concerns, making the evaluation of open-source models crucial. Among open-source models, SauerkrautLM LLama 3 70b (4-bit) and SauerkrautLM Mixtral 8x7b (both 8-bit and 4-bit versions) deliveredLarge Language Models (LLMs) have the potential to enhance e-mental health and psychosocial e-mail counselling by automating tasks such as generating concise and relevant subject lines for client communications. However, concerns regarding accuracy, reliability, data privacy and resource efficiency persist. This study investigates the performance of several LLMs in generating subject lines for e-mail threads, yielding a total of 253 generated subjects. Each subject line was assessed by six raters, including five counselling professionals and one AI system, using a three-category quality scale (Good, Fair, Poor). The results show that LLMs can generally produce concise subject lines considered helpful by experts. While GPT-4o and GPT-3.5 Turbo outperformed other models, their use is restricted in mental health settings due to data protection concerns, making the evaluation of open-source models crucial. Among open-source models, SauerkrautLM LLama 3 70b (4-bit) and SauerkrautLM Mixtral 8x7b (both 8-bit and 4-bit versions) delivered promising results with potential for further development. In contrast, models with lower parameter counts produced predominantly poor outputs.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Philipp SteigerwaldORCiD, Jens AlbrechtORCiD
DOI:https://doi.org/10.5220/0013294100003938
Parent Title (English):Proceedings of the 11th International Conference on Information and Communication Technologies for Ageing Well and e-Health
Publisher:SCITEPRESS - Science and Technology Publications
Document Type:conference proceeding (article)
Language:English
Reviewed:Begutachtet/Reviewed
Release Date:2025/11/26
Volume:2025
Pagenumber:8
First Page:70
Last Page:77
institutes:Institut für e-Beratung
Fakultät Informatik
Fakultät Sozialwissenschaften
Zentrum für Künstliche Intelligenz (KIZ)
Research Themes:Digitalisierung & Künstliche Intelligenz
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.