TY - CHAP A1 - Staufer, Susanne A1 - Hauser, Florian A1 - Ezer, Timur A1 - Grabinger, Lisa A1 - Nadimpalli, Vamsi Krishna A1 - Röhrl, Simon A1 - Bugert, Flemming A1 - Bittner, Dominik A1 - Mottok, Jürgen T1 - EVALUATION OF THE LEARNING MANAGEMENT SYSTEM PYTHIA T2 - EDULEARN Proceedings: 16th International Conference on Education and New Learning Technologies, Palma, Spain. 1-3 July, 2024 N2 - Learning management systems gain importance due to the Corona pandemic. To personalize such a learning management system, a Moodle adaption named Pythia was created. Pythia is implemented as a Moodle plugin solution. The possibility of generating learning paths with various algorithms and the change of the graphical user interface is significant. Learning paths are generated with two different algorithms. Nestor algorithm generates learning paths with a Bayesian network, while Tyche algorithm uses a Markov model. The graphical user interface is adapted to the generated learning paths where learning element symbols are customized and the individual learning paths are depicted. This paper presents the results of a qualitative survey among German students with the aim of evaluating the first version of the learning management system Pythia. 25 students took the subject “Software Engineering for Safe and Secure Systems” in the winter term of 2023/24. The 68 asked questions focus on usage behavior, graphical user interface, usability, preferences, volume, and quality of learning elements, and learning paths. Our Moodle course for the lecture contains seven sections, each section has minimum one subsection on which the learning path is calculated. Last three sections are designed by presenting minimum one learning element of each learning element category except the collaboration tool. In a previous work, ten learning element categories were designed and evaluated comprising manuscript, brief overview, learning goal, quiz, exercise, summary, three types of additional material, and collaboration tool. The first sections are left as they are initially designed by the lecturer (normal section). The survey considers the difference between normal sections and diverse sections. Furthermore, the generated learning paths are evaluated. 13 students undertook Nestor learning paths, whereas 12 undertook Tyche learning paths. The results suggest that for the majority of students, the learning element symbols are well-chosen. Moreover, all students wish the elements relevant to the exam to be labelled. 23 students are overwhelmed by the number of learning elements in the last sections of the Moodle course. In the future, some suggestions will be implemented such as the labelling of exam relevant learning elements. To overcome the overload of learning elements, a strategy could be developed like hiding learning elements not preferred by the learning path algorithm. KW - Learning management system (LMS) KW - questionnaire (study) KW - learning elements KW - learning paths KW - usability KW - graphical user interface (GUI) KW - Moodle KW - higher education area (HEA) Y1 - 2024 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-73474 SN - 2340-1117 SP - 9089 EP - 9098 PB - IATED ER - TY - CHAP A1 - Grabinger, Lisa A1 - Mottok, Jürgen T1 - On selecting hypothesis tests for group differences T2 - ICERI2024 Proceedings: 17th Annual International Conference of Education, Research and Innovation, 11th-13th November 2024, Seville, Spain N2 - Deciding on the right method of analyzing empirical research data can be difficult – especially when it comes to choosing specific methods of inferential statistics, where there is often not one right way, but a variety of valid options (e.g., using an ANOVA or its non-parametric alternative for data that is not perfectly normally distributed). Novice researchers not only lack the experience to know when a particular hypothesis test is appropriate, but struggle to find suitable literature to familiarize themselves (i.e., literature that is not too superficial, yet comprehensible). With the present article we provide a remedy following the didactic method of scaffolding: We present a systematization of the most elementary inferential statistical methods, namely hypothesis tests for group differences. We start by explaining basic terms (e.g., independent variable) and then give step-by-step instructions for choosing a proper hypothesis test based on data properties, implementing it from scratch, and reporting or interpreting its results. With these practical cookbook-like guidelines, this article serves as a concise starting point for young researchers entering the field of empirical research, as a valuable resource for their instructors, and as a basis for automating statistical procedures in a software system. KW - empirical research KW - data analysis KW - inferential statistics KW - guidelines Y1 - 2024 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-78200 SP - 702 EP - 712 PB - IATED ER - TY - JOUR A1 - Grabinger, Lisa A1 - Hauser, Florian A1 - Mottok, Jürgen T1 - On the perception of graph layouts JF - Journal of Software: Evolution and Process N2 - In the field of software engineering, graph-based models are used for a variety of applications. Usually, the layout of those graphs is determined at the discretion of the user. This article empirically investigates whether different layouts affect the comprehensibility or popularity of a graph and whether one can predict the perception of certain aspects in the graph using basic graphical laws from psychology (i.e., Gestalt principles). Data on three distinct layouts of one causal graph is collected from 29 subjects using eye tracking and a print questionnaire. The evaluation of the collected data suggests that the layout of a graph does matter and that the Gestalt principles are a valuable tool for assessing partial aspects of a layout. KW - causal graphs KW - eye tracking KW - gestalt principles KW - graph layouts KW - modeling languages Y1 - 2023 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-62973 N1 - Corresponding author: Lisa Grabinger PB - Wiley ER - TY - CHAP A1 - Grabinger, Lisa A1 - Hauser, Florian A1 - Mottok, Jürgen T1 - Accessing the Presentation of Causal Graphs and an Application of Gestalt Principles with Eye Tracking T2 - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2022), 1st Workshop on Advances in Human-Centric Experiments in Software Engineering (HUMAN 2022): 15-18 March 2022, Honolulu, HI, USA N2 - The discipline of causal inference uses so-called causal graphs to model cause and effect relations of random variables. As those graphs only encode a relation structure there is no hard rule concerning their alignment. The present paper presents a study with the aim of working out the optimal alignment of causal graphs with respect to comprehensibility and interestingness. In addition, the study examines whether the central gestalt principles of psychology apply for causal graphs. Data from 29 participants is acquired by triangulating eye tracking with a questionnaire. The results of the study suggest that causal graphs should be aligned downwards. Moreover, the gestalt principles proximity, similarity and closure are shown to hold true for causal graphs. KW - causal graphs KW - gestalt principles KW - eye tracking Y1 - 2022 U6 - https://doi.org/10.1109/SANER53432.2022.00153 SP - 1278 EP - 1285 PB - IEEE ER - TY - CHAP A1 - Grabinger, Lisa A1 - Hauser, Florian A1 - Mottok, Jürgen T1 - Evaluating Graph-based Modeling Languages T2 - Proceedings of the 5th European Conference on Software Engineering Education, Seeon/Germany, June 19 - 21, 2023 N2 - As humans, we tend to use models to describe reality. Modeling languages provide the formal frameworks for creating such models. Usually, the graphical design of individual model elements is based on subjective decisions; their suitability is determined at most by the prevalence of the modeling language. With other words: there is no objective way to compare different designs of model elements. The present paper addresses this issue: it introduces a systematic approach for evaluating the elements of graph-based modeling languages comprising 14 criteria – derived from standards, usability analyses, or the design theories ‘Physics of Notations’ and ‘Cognitive Dimensions of Notations’. The criteria come with measurement procedures and evaluation schemes based on reasoning, eye tracking, and questioning. The developed approach is demonstrated with a specific use case: three distinct sets of node elements for causal graphs are evaluated in an eye tracking study with 41 subjects. Y1 - 2023 SN - 978-1-4503-9956-2 U6 - https://doi.org/10.1145/3593663.3593664 SP - 120 EP - 129 PB - ACM ER - TY - CHAP A1 - Hauser, Florian A1 - Grabinger, Lisa A1 - Mottok, Jürgen T1 - Something Short Gets Even Shorter: Adapting the LIST-K for the Use in an Online Learning Management System T2 - Proceedings of the 5th European Conference on Software Engineering Education, Seeon/Germany, June 19 - 21, 2023 N2 - This study examines how Klingsieck’s LIST-K questionnaire [22] can be shortened and adapted to the requirements of an online learning management system. In a study with 213 participants, the questionnaire is subjected to an exploitative factor analysis. In a next step, the results are evaluated in terms of their reliability. This process creates a modified factor structure for the LIST-K, comprising a total of eight factors. The reliability of the modified questionnaire is at an α of .770. The shortened version of the LIST-K questionnaire is currently being used on an experimental basis in different courses. Y1 - 2023 SN - 978-1-4503-9956-2 U6 - https://doi.org/10.1145/3593663.3593684 SP - 65 EP - 72 PB - ACM ER - TY - CHAP A1 - Hauser, Florian A1 - Grabinger, Lisa A1 - Mottok, Jürgen A1 - Gruber, Hans T1 - Visual Expertise in Code Reviews: Using Holistic Models of Image Perception to Analyze and Interpret Eye Movements T2 - ETRA '23: 2023 Symposium on Eye Tracking Research and Applications, Tubingen Germany 30 May 2023- 2 June 2023 N2 - This study uses holistic models of image perception to analyze and interpret eye movements during a code review. 23 participants (15 novices and 8 experts) take part in the experiment. The subjects’ task is to review six short code examples in C programming language and identify possible errors. During the experiment, their eye movements are recorded by an SMI 250 REDmobile. Additional data is collected through questionnaires and retrospective interviews. The results implicate that holistic models of image perception provide a suitable theoretical background for the analysis and interpretation of eye movements during code reviews. The assumptions of these models are particularly evident for expert programmers. Their approach can be divided into different phases with characteristic eye movement patterns. It is best described as switching between scans of the code example (global viewing) and the detailed examination of errors (focal viewing). Y1 - 2023 U6 - https://doi.org/10.1145/3588015.3589189 SP - 1 EP - 7 PB - ACM ER - TY - CHAP A1 - Röhrl, Simon A1 - Staufer, Susanne A1 - Nadimpalli, Vamsi Krishna A1 - Bugert, Flemming A1 - Bugert, Flemming A1 - Hauser, Florian A1 - Grabinger, Lisa A1 - Bittner, Dominik A1 - Ezer, Timur A1 - Mottok, Jürgen T1 - PYTHIA - AI SUGGESTED INDIVIDUAL LEARNING PATHS FOR EVERY STUDENT T2 - INTED2024 Proceedings: 18th International Technology, Education and Development Conference, Valencia, Spain. 4-6 March, 2024 N2 - During the COVID-19 pandemic, the importance of digital course rooms, where teachers provide their learning materials, increased dramatically. While these platforms are crucial for providing teaching materials, they often fall short in addressing individual student needs. A system within an academic setting, capable of creating and presenting individual learning paths for each student, can solve these issues. These paths are composed of various learning elements - defined in our previous work as units of educational content with which a learner works. Currently, there is no suitable system that enables the integration of learning path generating algorithms into a digital course room. Therefore we present an application that enables this integration into the Moodle Learning Management System (LMS). More precisely, this paper presents a Moodle plugin together with its framework. It describes the mechanism for effectively collecting data from Moodle, which AI algorithms then use to generate personalized learning paths. Subsequently these paths are visualized with the help of the Moodle plugin. We started with a set of requirements and use cases for the interface connecting Moodle to the AI system, which were established with a group of experts. Based on the requirements, various relevant technologies were assessed, and the best ones were chosen for implementation. Following that, the paper develops a strategy for software structuring as well as an architecture, focusing on performance, modularity, and ease of deployment for widespread use. Furthermore, the architecture ensures a simple method for integrating the algorithms. Afterwards, the framework's concrete implementation is described. A technique for enriching learning elements with metadata is presented, and additionally a concept for presenting these learning elements within a hierarchy. Moreover, it is shown how questionnaire responses and learning analytics are utilized for data collection. We cover in detail techniques for extracting and storing data from the Moodle database, as well as methods for customizing Moodle course rooms and a standard API for incorporating AI algorithms. Finally, the paper discusses the application of the proposed framework in an actual course and how student feedback is collected, which could enhance the framework. It concludes with an assessment of the outcomes obtained and prospects for the framework's future advancements. KW - Personalized Learning Paths KW - Learning Management System KW - Software Architecture KW - Moodle KW - Artificial Intelligence Y1 - 2024 SN - 978-84-09-59215-9 U6 - https://doi.org/10.21125/inted.2024.0783 SN - 2340-1079 SP - 2871 EP - 2880 ER - TY - CHAP A1 - Staufer, Susanne A1 - Hauser, Florian A1 - Grabinger, Lisa A1 - Bittner, Dominik A1 - Nadimpalli, Vamsi Krishna A1 - Bugert, Flemming A1 - Ezer, Timur A1 - Röhrl, Simon A1 - Mottok, Jürgen T1 - Learning elements in LMS - a survey among students T2 - INTED2024 Proceedings: 18th International Technology, Education and Development Conference, Valencia, Spain. 4-6 March, 2024 N2 - In higher education, improving learning and learning success are goals of general improvement. Lecturers teaches content and students acquire that content in an efficient way. To structure content, learning element categories are evaluated from the student's point of view in higher education area. The aim is to validate given definitions of ten learning element categories within a Learning Management System (LMS). This paper evaluates a categorization of learning elements for organizing learning content in online education within LMSs. Therefore, ten categories of learning elements and corresponding definitions were defined in a previous work as base for this paper. The learning elements to examine are manuscript, exercise, quiz, brief overview, learning goal, summary, collaboration tool, auditory additional material, textual additional material, and visual additional material. To validate the definitions and to get improvements to each learning element a survey is processed. Beside the demographic data questions, the survey consists of two questions to the acceptance of the definitions and asks for improvements. 148 students between the ages 19 and 35 participate in the survey in summer term 2023. The education level of the participants ranges from undergraduates to Ph.D. students. The results of this paper are that more than 80% accept the given definitions. Some definitions of the learning elements are changed, but the changes are restricted to additions of maximal four words. This categorization of learning elements could lead to improvements in learning by giving the content more structure. With the structure students get the possibility to learn with preferred learning elements which could lead to more success in learning and to a decreasing dropout rate in universities. In the future, the learning elements allow to classify content within LMSs with the goal of generating individual learning paths. Furthermore, our project will integrate these learning elements, use them to generate learning paths, and could set a new standard in the way of personalized learning. KW - Learning elements KW - learning management system KW - learning objects KW - questionnaire KW - higher education area KW - content organization Y1 - 2024 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-71135 SN - 978-84-09-59215-9 SN - 2340-1079 SP - 4224 EP - 4231 PB - IATED ER - TY - CHAP A1 - Grabinger, Lisa A1 - Ezer, Timur A1 - Hauser, Florian A1 - Mottok, Jürgen T1 - The impact of eyenalyzer T2 - ICERI2024 Proceedings: 17th Annual International Conference of Education, Research and Innovation, 11th-13th November 2024, Seville, Spain N2 - Empirical research poses numerous challenges for beginners. This is especially true for data analysis – a task that usually requires knowledge from two distinct areas: statistics and programming. To support prospective researchers with that task, we developed a web-based tool called eyenalyzer. It supports common activities in the data analysis phase of empirical studies in a way that is suitable for novices in both, statistics and programming. The present article describes a controlled experiment investigating the impact of this tool with a total of 20 participants. All of them are given a set of common data analysis tasks. Half of the participants complete the tasks using eyenalyzer, the other half can use anything except for eyenalyzer. For each task and participant, we record the time in minutes, the task score, and the perceived difficulty. The results confirm that our tool is a valuable support for novice researchers: With eyenalyzer, the participants are significantly faster, achieve higher scores, and perceive the tasks to be less difficult. KW - tool evaluation KW - data analysis KW - eye tracking KW - empirical research Y1 - 2024 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:898-opus4-78215 SN - 978-84-09-63010-3 SN - 2340-1095 SP - 695 EP - 701 PB - IATED ER -