Overview of the CLEF-2024 CheckThat! Lab Task 2 on Subjectivity in News Articles

  • We present an overview of Task 2 of the seventh edition of the CheckThat! lab at the 2024 iteration of the Conference and Labs of the Evaluation Forum (CLEF). The task focuses on subjectivity detection in news articles and was o ered in five languages: Arabic, Bulgarian, English, German, and Italian, as well as in a multilingual setting. The datasets for each language were carefully curated and annotated, comprising over 10,000 sentences from news articles. The task challenged participants to develop systems capable of distinguishing between subjective statements (refecting personal opinions or biases) and objective ones (presenting factual information) at the sentence level. A total of 15 teams participated in the task, submitting 36 valid runs across all language tracks. The participants used a variety of approaches, with transformer-based models being the most popular choice. Strategies included fine-tuning monolingual and multilingual models, and leveraging English models with automatic translation for the non-English datasets.We present an overview of Task 2 of the seventh edition of the CheckThat! lab at the 2024 iteration of the Conference and Labs of the Evaluation Forum (CLEF). The task focuses on subjectivity detection in news articles and was o ered in five languages: Arabic, Bulgarian, English, German, and Italian, as well as in a multilingual setting. The datasets for each language were carefully curated and annotated, comprising over 10,000 sentences from news articles. The task challenged participants to develop systems capable of distinguishing between subjective statements (refecting personal opinions or biases) and objective ones (presenting factual information) at the sentence level. A total of 15 teams participated in the task, submitting 36 valid runs across all language tracks. The participants used a variety of approaches, with transformer-based models being the most popular choice. Strategies included fine-tuning monolingual and multilingual models, and leveraging English models with automatic translation for the non-English datasets. Some teams also explored ensembles, feature engineering, and innovative techniques such as few-shot learning and in-context learning with large language models. The evaluation was based on macro-averaged F1 score. The results varied across languages, with the best performance achieved for Italian and German, followed by English. The Arabic track proved particularly challenging, with no team surpassing an F1 score of 0.50. This task contributes to the broader goal of enhancing the reliability of automated content analysis in the context of misinformation detection and fact-checking. The paper provides detailed insights into the datasets, participant approaches, and results, o ering a benchmark for the current state of subjectivity detection across multiple languages.zeige mehrzeige weniger

Metadaten exportieren

Weitere Dienste

Suche bei Google Scholar
Metadaten
Verfasserangaben:Julia Maria StrußORCiDGND, Federico Ruggeri, Alberto Barrón-Cedeño, Firoj Alam, Dimitar Dimitrov, Andrea Galassi, Georgi Pachov, Ivan Koychev, Preslav Nakov, Melanie Siegel, Michael Wiegand, Maram Hasanain, Reem Suwaileh, Wajdi Zaghouani
URN:urn:nbn:de:0074-3740-3
URL:https://ceur-ws.org/Vol-3740/paper-25.pdf
ISSN:1613-0073
Titel des übergeordneten Werkes (Englisch):CLEF 2024 Working Notes : Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024)
Untertitel (Englisch):Notebook for the CheckThat! Lab at CLEF 2024
Verlagsort:Frankreich
Herausgeber*in:Guglielmo Faggioli, Nicola Ferro, Petra Galuščáková, Alba García Seco de Herrera
Dokumentart:Konferenzveröffentlichung
Sprache:Englisch
Jahr der Erstveröffentlichung:2024
Veröffentlichende Institution:Fachhochschule Potsdam
Urhebende Körperschaft:University of Grenoble Alpes
Datum der Freischaltung:26.03.2025
GND-Schlagwort:Fehlinformation; Zeitungsartikel
Erste Seite:287
Letzte Seite:298
Fachbereiche und Zentrale Einrichtungen:FB5 Informationswissenschaften
FB5 Informationswissenschaften / Publikationen des FB Informationswissenschaften
DDC-Klassifikation:000 Informatik, Informationswissenschaft, allgemeine Werke / 000 Informatik, Wissen, Systeme
Open Access:Gold Open Access
Lizenz (Deutsch):Creative Commons - CC BY - Namensnennung 4.0 International
Einverstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.