TY - JOUR A1 - Hackl, Veronika A1 - Müller, Alexandra Elena A1 - Granitzer, Michael A1 - Sailer, Maximilian T1 - Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings JF - Frontiers in Education N2 - This study reports the Intraclass Correlation Coefficients of feedback ratings produced by OpenAI's GPT-4, a large language model (LLM), across various iterations, time frames, and stylistic variations. The model was used to rate responses to tasks related to macroeconomics in higher education (HE), based on their content and style. Statistical analysis was performed to determine the absolute agreement and consistency of ratings in all iterations, and the correlation between the ratings in terms of content and style. The findings revealed high interrater reliability, with ICC scores ranging from 0.94 to 0.99 for different time periods, indicating that GPT-4 is capable of producing consistent ratings. The prompt used in this study is also presented and explained. KW - artificial intelligence KW - GPT-4 KW - large language model KW - prompt engineering KW - feedback KW - higher education Y1 - 2023 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:739-opus4-18348 SN - 2504-284X VL - 2023 IS - 8 PB - Frontiers CY - Lausanne ER -