Multi-Query Focused Disaster Summarization via Instruction-Based Prompting

  • Automatic summarization of mass-emergency events plays a critical role in disaster management. The second edition of CrisisFACTS aims to advance disaster summarization based on multi-stream fact-finding with a focus on web sources such as Twitter, Reddit, Facebook, and Webnews. Here, participants are asked to develop systems that can extract key facts from several disaster-related events, which ultimately serve as a summary. This paper describes our method to tackle this challenging task. We follow previous work and propose to use a combination of retrieval, reranking, and an embarrassingly simple instruction-following summarization. The two-stage retrieval pipeline relies on BM25 and MonoT5, while the summarizer module is based on the open-source Large Language Model (LLM) LLaMA-13b. For summarization, we explore a Question Answering (QA)-motivated prompting approach and find the evidence useful for extracting query-relevant facts. The automatic metrics and human evaluation show strong results but also highlight the gap betweenAutomatic summarization of mass-emergency events plays a critical role in disaster management. The second edition of CrisisFACTS aims to advance disaster summarization based on multi-stream fact-finding with a focus on web sources such as Twitter, Reddit, Facebook, and Webnews. Here, participants are asked to develop systems that can extract key facts from several disaster-related events, which ultimately serve as a summary. This paper describes our method to tackle this challenging task. We follow previous work and propose to use a combination of retrieval, reranking, and an embarrassingly simple instruction-following summarization. The two-stage retrieval pipeline relies on BM25 and MonoT5, while the summarizer module is based on the open-source Large Language Model (LLM) LLaMA-13b. For summarization, we explore a Question Answering (QA)-motivated prompting approach and find the evidence useful for extracting query-relevant facts. The automatic metrics and human evaluation show strong results but also highlight the gap between open-source and proprietary systems.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Philipp Seeberger, Korbinian Riedhammer
DOI:https://doi.org/10.48550/arXiv.2402.09008
ArXiv Id:http://arxiv.org/abs/arXiv:2402.09008
Publisher:National Institute of Standards and Technology (NIST)
Document Type:conference proceeding (article)
Language:English
Date of first Publication:2024/02/14
Reviewed:Begutachtet/Reviewed
Release Date:2024/07/08
Pagenumber:7
Konferenzangabe:Proceedings of the 32nd Text REtrieval Conference (TREC 2023)
institutes:Fakultät Informatik
Licence (German):Keine Lizenz - Deutsches Urheberrecht gilt
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.