Better Call Claude: Can LLMs Detect Changes of Writing Style?

  • This article explores the zero-shot performance of state-ofthe-art large language models (LLMs) on one of the most challenging tasks in authorship analysis: sentence-level style change detection. Benchmarking four LLMs on the official PAN 2024 and 2025 “MultiAuthor Writing Style Analysis” datasets, we present several observations. First, state-of-the-art generative models are sensitive to variations in writing style—even at the granular level of individual sentences. Second, their accuracy establishes a challenging baseline for the task, outperforming suggested baselines of the PAN competition. Finally, we explore the influence of semantics on model predictions and present evidence suggesting that the latest generation of LLMs may be more sensitive to content-independent and purely stylistic signals than previously reported.

Download full text files

  • This article explores the zero-shot performance of state-of-the-art large language models (LLMs) on one of the most challenging tasks in authorship analysis: sentence-level style change detection. Benchmarking four LLMs on the official PAN 2024 and 2025 “Multi-Author Writing Style Analysis” datasets, we present several observations. First, state-of-the-art generative models are sensitive to variations in writing style—even at the granular level of individual sentences. Second, their accuracy establishes a challenging baseline for the task, outperforming suggested baselines of the PAN competition. Finally, we explore the influence of semantics on model predictions and present evidence suggesting that the latest generation of LLMs may be more sensitive to content-independent and purely stylistic signals than previously reported.

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Johannes Römisch, Svetlana Gorovaia, Mariia Halchynska, Gleb Schmidt, Ivan P. Yamshchikov
URN:urn:nbn:de:bvb:863-opus-62822
DOI:https://doi.org/10.1007/978-3-032-04354-2_4
ISBN:9783032043535
ISSN:0302-9743
Parent Title (English):Lecture Notes in Computer Science
Publisher:Springer Nature Switzerland
Place of publication:Cham
Document Type:Conference Proceeding
Language:English
Year of publication:2025
Release Date:2025/09/23
Tag:LLM; Style Detection
Pages/Size:15
First Page:42
Last Page:56
Institutes and faculty:Institute / Center for Artificial Intelligence (CAIRO)
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.