TY - CHAP A1 - Lohr, Dominic A1 - Kiesler, Natalie A1 - Keuning, Hieke A1 - Jeuring, Johan T1 - "Let Them Try to Figure It Out First" - Reasons Why Experts (Do Not) Provide Feedback to Novice Programmers T2 - Proceedings of the 2024 on Innovation and Technology in Computer Science Education Vol. 1 N2 - A recent ITiCSE working group investigated when and how experts give feedback and hints at steps novice programmers take when solving programming problems. Based on the feedback literature and an analysis of expert feedback on steps, the working group designed guidelines for when and how to give feedback. The feed-back provided by educators using these guidelines on a number of sequences of student steps varied a lot. In this paper, we try to answer the question of why educators give feedback at particular steps to novice learners of programming. We prepared six authentic sequences of student steps when solving an introductory programming task. The preprocessed sequences were used in a survey to gather information about when and why an expert would give feedback. Respondents annotated each step from one sequence with if and why they would give feedback at that step. Our survey received 47 responses. We qualitatively analyzed the responses, resulting in a coding scheme consisting of 19 different reasons for why experts intervene (or not) when novice learners work on introductory programming tasks. We found a considerable variety of reasons experts give for when and how to help students with feedback and hints. Also, sometimes one expert uses a reason at a step to explain why they do intervene, and another expert uses the same reason at the step to not intervene. The categories of experts’ feedback indicators will pave the way for several future studies and applications, including learning systems trying to resemble expert feedback strategies. KW - Learning programming, expert feedback, feedback rationale, feed- back guidelines, novice programmers Y1 - 2024 U6 - https://doi.org/10.1145/3649217.3653530 SP - 38 EP - 44 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Brockmann, Patricia A1 - Schuhbauer, Heidi A1 - Bienert, Margot T1 - Based E-Learning to Increase Motivation of Stem-Students : Gamification of A Startup Enterpriser T2 - Proceedings of the International Teaching and Education Conference N2 - First semester students often face major challenges in adjusting to university studies. Inexperienced students may not appreciate the practical relevance of abstract, theoretical concepts taught using traditional, instructor-centered lectures. Furthermore, the rise of generative artificial intelligence (GenAI) can mislead some to believe they no longer need to exert effort to learn academic skills. To encourage students to engage in learning material and thus improve retention rates, two complementary teaching methods were integrated into a large, introductory course for first semester STEM students: Problem-Based Learning (PBL) and gamification. Problem-Based Learning was implemented to make the subject matter more meaningful by simulating a real-world experience: How to start up a small business. This entrepreneurship task was simulated in an online game, to increase student motivation. Quantitative questionnaires of student motivation and experiences with e-learning were evaluated. Based on input obtained in expert interviews, a fictitious startup enterprise was designed to serve as the used case. An online e-learning game was developed, which leveraged gamification elements to try to increase motivation. Each phase of the startup process was represented as one level of the e-learning game. The e-learning game was tested by a group of first semester students. Their opinions were collected using an anonymous online survey. Aggregated results of the survey are discussed and plans for further research are presented. Y1 - 2025 ER - TY - JOUR A1 - Schuhbauer, Heidi A1 - Schötteler, Sebastian A1 - Niu, Johannes A1 - Schiffer, Bernhard A1 - Wolfharth, David A1 - Groß, Tim A1 - Kreidl, Thomas A1 - Model, Fabian T1 - Analysis of Politicians’ Tweets to Explore Political Communication with Social Network Analysis JF - International Journal On Advances in Networks and Services N2 - This paper illustrates the practical application of cluster analysis, social network analysis, sentiment analysis, and topic analysis in a case study on Twitter data. These techniques provide insights into the public communication patterns between German Members of Parliament (MPs) on Twitter around the time of the 2021 federal election. The question of this work was to determine whether a potential shift in communication towards the inaugurated “Ampel” coalition, made up of the parties SPD, Greens, and FDP, can be derived from Twitter interactions. Twitter data were collected and separated into two time slots: before and after the election. In distinct scenarios, mention, retweet, and reply interactions are first considered together and then separately. In these scenarios,the Girvan-Newman Algorithm detects clusters of MPs dependent on the interactions observed. Then, the average inbreeding homophily and other network metrics of the pre- and post-election area are compared. An additional scenario focuses on intra- and inter-party sentiments conveyed within tweet texts. In a fourth scenario, MPs are grouped according to their party affiliation, the average inbreeding homophily values of parties, and potential coalitions. A topic analysis handled relevant discussion topics between the successful coalition parties. Changes in communication behavior at these two different time slots are visible. The communication clusters of those MPs differ mostly before and after the election. The average sentiment of the parties towards each other changed positively, although no significant tendency could be derived regarding later coalition formations. A determination of the density of the network for the topics also indicates that the importance of the individual topics before and after the formation of the government has a consistent relevance. KW - Cluster Analysis; Microblog; Network Metrics; Sentiment Analysis; Social Network Analysis; Topic Analysis Y1 - 2023 SN - 1942-2644 VL - 16 IS - 1/2 ER - TY - CHAP A1 - Kiesler, Natalie A1 - Scholz, Ingo A1 - Albrecht, Jens A1 - Stappert, Friedhelm A1 - Wienkop, Uwe T1 - Novice Learners of Programming and Generative AI - Prior Knowledge Matters T2 - Proceedings of the 24th Koli Calling International Conference on Computing Education Research N2 - With the broad availability of Generative AI (GenAI), introductory programming education is starting to change. At Nuremberg Tech, we observed the doubling of failure rates to approximately 50% in the first semester course “Procedural Programming” across students of all study programs. Due to these exam results in winter 2023/24, we conducted a pilot study to gather students’ use of GenAI tools, their exam results, and prior programming education and experience. The results imply significant differences of students’ use of GenAI tools depending on their prior programming education. We will therefore extend the investigation in winter term 2024/25. Y1 - 2024 U6 - https://doi.org/10.1145/3699538.3699580 SP - 1 EP - 2 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Keuning, Hieke A1 - Luxton-Reilly, Andrew A1 - Ott, Claudia A1 - Petersen, Andrew A1 - Kiesler, Natalie T1 - Goodbye Hello World - Research Questions for a Future CS1 Curriculum T2 - Proceedings of the 24th Koli Calling International Conference on Computing Education Research N2 - Generative AI (GenAI) is currently capable of generating correct code for introductory level programming problems, and its performance is improving. We believe that this capability can be leveraged to improve student motivation, broaden students’ understanding of software development, and engage them in more authentic learning. We defined a set of assumptions about GenAI’s future capabilities (e.g., the ability to generate small pieces of code and to compose these pieces of code via user prompts) and engaged in a backcasting exercise to identify what else is needed to develop a CS1 course that places GenAI in a central role. Undertaking this thought experiment immediately revealed that aspects of the software development process usually reserved for later in the curriculum, such as requirements elicitation and design, could be introduced earlier in the process. With GenAI tools bearing the load of generating correct code snippets, students could focus on higher-level software design and construction skills and practice them in an authentic environment. Our thought experiment identified a set of questions that need to be addressed for such a course to actually exist, including questions about student preparation, and the ability of students to decompose problems effectively and to resolve problems that arise when integrating pieces of code. We also identified questions related to the design of a GenAI centered course, such as the impact on student motivation of using GenAI instead of engaging directly with code, the extent to which social learning theories apply to interactions with GenAI, and how existing pedagogies can integrate GenAI tools. Y1 - 2024 U6 - https://doi.org/10.1145/3699538.3699591 SP - 1 EP - 2 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Alshaigy, Bedour A1 - Grande, Virginia A1 - Kiesler, Natalie A1 - Settle, Amber T1 - How Do You Solve A Problem Like Recruitment? On The Hiring and Retention of Computing Academics T2 - Proceedings of the 2024 on ACM Virtual Global Computing Education Conference Vol. 1 N2 - This paper critically examines persistent inequities in existing computing faculty hiring and retention practices, which gravely impact computing educators from marginalized groups. Throughout these processes, applicants fight against multiple systemic barriers, including but not limited to, biased job ads and discriminatory interview practices. The increasing use of generative AI tools to aid in tasks connected to the hiring process, such as writing recommendation letters, exacerbates these biases. The inequities persist despite global initiatives and legal mandates and serve as a direct contradiction to widespread institutional commitments to diversity and inclusion. By building on literature and the lived experiences of the SIGCSE community represented in a recent Technical Symposium session, we raise concerns about the different stages of this process, highlighting the importance of clear expectations and adequate support. The paper concludes with a call to align hiring practices with inclusive institutional values, requiring the academic community to reflect on and revise hiring policies for a more equitable future. It is of paramount importance to address the role of these practices in the erosion of marginalized communities from the computing education community, a marginalization that occurs in many different contexts and negatively impacts everyone involved. KW - CS academics, recruitment, retention Y1 - 2024 U6 - https://doi.org/10.1145/3649165.3703622 SP - 263 EP - 266 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Kiesler, Natalie A1 - Schiffner, Daniel T1 - Conferences are Exclusive by Nature T2 - Proceedings of the 2024 on RESPECT Annual Conference N2 - This paper presents a perspective on equity by examining the inherent exclusivity of academic conferences within the context of computing education research. Our perspective delves into unequal opportunities faced by (young) researchers, which are rooted in the challenges of limited mobility, finances, and, for example, family obligations preventing equitable participation. By identifying and interrogating these barriers, the paper advocates for a reevaluation of conference structures to foster inclusivity and address the systemic issues that disproportionately impact certain demographics. Therefore, this paper contributes to the ongoing discourse on equity within academic spaces, urging a transformative approach to conference organization. KW - equity, diversity, conferences, resources, young researchers, mobility, finances, inclusion, sustainability Y1 - 2024 U6 - https://doi.org/10.1145/3653666.3656077 SP - 288 EP - 292 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Kiesler, Natalie A1 - Lohr, Dominic A1 - Keuning, Hieke T1 - Exploring the Potential of Large Language Models to Generate Formative Programming Feedback T2 - 2023 IEEE Frontiers in Education Conference (FIE) N2 - Ever since the emergence of large language models (LLMs) and related applications, such as ChatGPT, its performance and error analysis for programming tasks have been subject to research. In this work-in-progress paper, we explore the potential of such LLMs for computing educators and learners, as we analyze the feedback it generates to a given input containing program code. In particular, we aim at (1) exploring how an LLM like ChatGPT responds to students seeking help with their introductory programming tasks, and (2) identifying feedback types in its responses. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT along with questions required to elicit feedback and correct solutions. The results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors, which means that students can potentially benefit. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices. Y1 - 2023 U6 - https://doi.org/10.1109/FIE58773.2023.10343457 PB - IEEE ER - TY - CHAP A1 - Scarcella, Diego A1 - Schneider, Jan A1 - Kiesler, Natalie A1 - Schiffner, Daniel T1 - Mathkinetics: Solving Arithmetics While Running out of Breath T2 - Proceedings of the 16th International Conference on Computer Supported Education N2 - To benefit from most of the current digital educational technologies, learners are required to sit down and look closely at a computer monitor or smart device screen for hours, which can have side effects on learners’ health and lifestyle. As an attempt to address this, we developed MathKinetics, an application designed to support the practice of cognitive skills such as arithmetic while engaging in physical activity by integrating the principles of Multimodal Learning, Life Kinetik, and Gamification. MathKinetics is a variant of an endless running game where users control an avatar through their body posture and dodge obstacles. At the same time, they pick up arithmetic problems whose answers need to be verbalized. In this paper, we present an exploratory evaluation of MathKinetics and its user experience. We conducted user tests with 20 participants. Results from our tests indicate that MathKinetics is a fun way to practice arithmetic skills and train executive cognitive functions such as task switching. KW - Game-Based Learning, Multimodal Learning, Natural Interaction Y1 - 2024 SN - 978-989-758-697-2 U6 - https://doi.org/10.5220/0012536900003693 SN - 2184-5026 SP - 250 EP - 256 PB - SCITEPRESS - Science and Technology Publications ER - TY - CHAP A1 - Kiesler, Natalie A1 - Röpke, René A1 - Schiffner, Daniel A1 - Schulz, Sandra A1 - Strickroth, Sven A1 - Ehlenz, Matthias A1 - Heinemann, Birte A1 - Wilhelm-Weidner, Arno ED - Schulz, Sandra ED - Kiesler, Natalie T1 - Towards Open Science at the DELFI Conference T2 - 22. Fachtagung Bildungstechnologien (DELFI) N2 - Despite the increasing awareness of Open Science within the educational technology community, conferences, such as DELFI, do not yet foster the publication of research data including software. To address this, we conducted a survey eliciting the community’s needs, perspectives, and publication preferences. The analysis of 24 valid responses reveals a variety of research data formats used, and several uncertainties, e. g., regarding data ownership. Associated barriers comprise legal concerns and lacking resources to publish data. Nonetheless, researchers seem open for new publication formats. Moreover, we analyzed author’s intentions to publish data related to their DELFI submissions in 2023 (n=66). Many researchers assume not to have data to share (n=28), or no intention to publish data in the future (n=16). Overall, the results imply a lack of awareness and recognition of data publications, so that further efforts and incentives are required to move toward Open Science practices in the DELFI community. KW - Open Science , Open Data , Data publication , FAIR , DELFI , Educational Technology Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_22 SP - 251 EP - 265 ER - TY - CHAP A1 - Scholl, Andreas A1 - Schiffner, Daniel A1 - Kiesler, Natalie ED - Schulz, Sandra ED - Kiesler, Natalie T1 - Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT T2 - 22. Fachtagung Bildungstechnologien (DELFI) N2 - Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats’ progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students’ interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education. KW - ChatGPT-3.5, large language models, students, interaction pattern, application, chat protocols, introductory programming, higher education Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_05 SN - 2944-7682 SP - 63 EP - 79 PB - Gesellschaft für Informatik e.V. CY - Bonn ER - TY - JOUR A1 - Wolf, Philipp A1 - Götzelmann, Timo T1 - VEPdgets: Towards Richer Interaction Elements Based on Visually Evoked Potentials JF - Sensors N2 - For brain–computer interfaces, a variety of technologies and applications already exist. However, current approaches use visual evoked potentials (VEP) only as action triggers or in combination with other input technologies. This paper shows that the losing visually evoked potentials after looking away from a stimulus is a reliable temporal parameter. The associated latency can be used to control time-varying variables using the VEP. In this context, we introduced VEP interaction elements (VEP widgets) for a value input of numbers, which can be applied in various ways and is purely based on VEP technology. We carried out a user study in a desktop as well as in a virtual reality setting. The results for both settings showed that the temporal control approach using latency correction could be applied to the input of values using the proposed VEP widgets. Even though value input is not very accurate under untrained conditions, users could input numerical values. Our concept of applying latency correction to VEP widgets is not limited to the input of numbers. KW - brain–computer interface; BCI; VEP; interaction; HCI; interaction elements; control elements; controls; widgets; VEP-widget Y1 - 2023 U6 - https://doi.org/10.3390/s23229127 SN - 1424-8220 VL - 23 IS - 22 PB - MDPI AG ER - TY - CHAP A1 - MacNeil, Stephen A1 - Leinonen, Juho A1 - Denny, Paul A1 - Kiesler, Natalie A1 - Hellas, Arto A1 - Prather, James A1 - Becker, Brett A. A1 - Wermelinger, Michel A1 - Reid, Karen T1 - Discussing the Changing Landscape of Generative AI in Computing Education T2 - Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2 N2 - In a previous Birds of a Feather discussion, we delved into the nascent applications of generative AI, contemplating its potential and speculating on future trajectories. Since then, the landscape has continued to evolve revealing the capabilities and limitations of these models. Despite this progress, the computing education research community still faces uncertainty around pivotal aspects such as (1) academic integrity and assessments, (2) curricular adaptations, (3) pedagogical strategies, and (4) the competencies students require to instill responsible use of these tools. The goal of this Birds of a Feather discussion is to unravel these pressing and persistent issues with computing educators and researchers, fostering a collaborative exploration of strategies to navigate the educational implications of advancing generative AI technologies. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed leaders to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education. KW - academic integrity; assessment; computing education; curriculum; large language models; pedagogy Y1 - 2024 SN - 979-8-4007-0424-6 U6 - https://doi.org/10.1145/3626253.3635369 PB - ACM CY - New York, NY, USA ER - TY - RPRT A1 - Trump, Sebastian A1 - Agchar, Ismael A1 - Baumann, Ilja A1 - Braun, Franziska A1 - Riedhammer, Korbinian A1 - Siemandel, Lea A1 - Ullrich, Martin T1 - Spirio Sessions BT - Experiments in Human-Machine Improvisation with a Digital Player Piano N2 - This paper presents an ongoing interdisciplinary research project that deals with free improvisation and human-machine interaction, involving a digital player piano and other musical instruments. Various technical concepts are developed by student participants in the project and continuously evaluated in artistic performances. Our goal is to explore methods for co-creative collaborations with artificial intelligences embodied in the player piano, enabling it to act as an equal improvisation partner for human musicians. KW - Human-Machine Improvisation, Co-creativity, Player Piano Y1 - 2021 ER - TY - CHAP A1 - Götzelmann, Timo T1 - <> 3D Printable Hand Exoskeleton for the Haptic Exploration of Virtual 3D Scenes T2 - PETRA '17: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments N2 - Virtual reality is currently experiencing a comeback. A considerable market has developed for VR computer games and educational applications. Some solutions integrate tracked devices which allow users to freely move within a certain space. Virtual 3D model can be visually explored, implemented collision detected allows users to get a feedback for instance by sound or vibration. For research projects there are several approaches which offer to get the actual feedback for the fingers of a hand, when the users virtually touches the surface of a 3D model. However, in the consumer market currently no product is sold which offers this direct feedback for the whole hand. In this paper we introduce a low-cost hand exoskeleton which is usable in conjunction with commodity hardware. It covers each of the five fingers of the user's hand, its design is open-source, low-cost, can be customized and 3D printed by individuals. It aims at improving the haptic perception of users, bases of a popular physical computing platform and is designed to be assembled even by electronically unexperienced users. We show the integration of our lean interface of the wireless exoskeleton into exemplary VR environment and describe a calibration process which is flexible for customizations. Y1 - 2017 SN - 978-1-4503-5227-7 U6 - https://doi.org/10.1145/3056540.3064950 VL - 2017 SP - 63 EP - 66 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Götzelmann, Timo A1 - Winkler, Klaus T1 - SmartTactMaps: A Smartphone-Based Approach to Support Blind Persons in Exploring Tactile Maps T2 - Proc. 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments N2 - Despite increasing digitalization of our society many blind persons still have very limited access to predominantly pictorial information such as maps. In this paper we introduce a novel approach to improve the accessibility of maps for blind users by utilizing the abilities of standard smartphones. A major issue of tactile maps is the limited discriminability of the humans' tactile sense. Textual annotation of maps is crucial, but adds much complexity to tactile maps. Additionally, only few Braille labels can be accommodated to maintain legibility. In our approach we link smartphones with adapted tactile maps which transforms the physical maps into interactive surfaces using both the tactile and the auditory modality. We integrate machine readable metadata into these maps which can be recognized by the smartphones' camera to immediately obtain detailed map descriptions from a free global database. During tactile exploration of the map, blind users can request auditory explanations by interacting with the mobile application. An experimental application and a user study demonstrate the feasibility of our approach. Y1 - 2015 SN - 978-1-4503-3452-5 U6 - https://doi.org/10.1145/2769493.2769497 VL - 2015 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Götzelmann, Timo A1 - Branz, Lisa A1 - Heidenreich, Claudia A1 - Otto, Markus T1 - A Personal Computer-based Approach for 3D-Printing Accessible to Blind People T2 - PETRA '17: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments N2 - Tactile materials play a major role in making information available to blind people and support their understanding for spatial matters. Due to the complex manual manufacturing process there is still a lack of suitable models for the visually impaired. Millions of 3D models are currently available on the internet and can be searched by dedicated retrieval sites. Most of them can be printed by 3D printers; however, this often isn't a trivial task even for sighted users. Blind peoples' self-dependence could be drastically increased if they were able to autonomously print 3D models at home. This paper analyses the individual tasks to actually print 3D models and adapts them to steps accessible for blind people. We introduce a workflow for the combined use of 3D printing software and consumer hardware. We verified our approach by a formal user study with visually impaired people which showed its feasibility. Y1 - 2017 SN - 978-1-4503-5227-7 U6 - https://doi.org/10.1145/3056540.3064954 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Kreimeier, Julian A1 - Götzelmann, Timo T1 - First Steps Towards Walk-In-Place Locomotion and Haptic Feedback in Virtual Reality for Visually Impaired T2 - CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems N2 - This paper presents the first results on a user study in which people with visual impairments (PVI) explored a virtual environment (VE) by walking in a virtual reality (VR) treadmill. As recently suggested, we have now acquired first results from our feasibility study investigating this walk-in-place interaction. This represents a new, more intuitive way of for example virtually exploring unknown spaces in advance. Our prototype consists of off-the-shelf VR components (i.e., treadmill, headphones, glasses, and controller) providing a simplified white cane simulation and was tested by six visually impaired subjects. Our results indicate that this interaction is yet difficult, but promising and an important step to make VR more and better usable for PVIs. As an impact on the CHI community, we would like to make this research field known to a wider audience by sharing our intermediate results and suggestions for improvements, on some of which we are already working on. Y1 - 2019 SN - 978-1-4503-5971-9 U6 - https://doi.org/10.1145/3290607.3312944 VL - 2019 SP - 1 EP - 6 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Götzelmann, Timo T1 - LucentMaps: 3D Printed Audiovisual Tactile Maps for Blind and Visually Impaired People T2 - ASSETS '16: Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility N2 - Tactile maps support blind and visually impaired people in orientation and to familiarize with unfamiliar environments. Interactive approaches complement these maps with auditory feedback. However, commonly these approaches focus on blind people. We present an approach which incorporates visually impaired people by visually augmenting relevant parts of tactile maps. These audiovisual tactile maps can be used in conjunction with common tablet computers and smartphones. By integrating conductive elements into 3D printed tactile maps, they can be recognized by a single touch on the mobile device's display, which eases the handling for blind and visually impaired people. To allow multiple elevation levels in our transparent tactile maps, we conducted a study to reconcile technical and physiological requirements of off-the-shelf 3D printers, capacitive touch inputs and the human tactile sense. We propose an interaction concept for 3D printed audiovisual tactile maps, verify its feasibility and test it with a user study. Our discussion includes economic considerations crucial for a broad dissemination of tactile maps for both blind and visually impaired people. Y1 - 2016 SN - 978-1-4503-4124-0 U6 - https://doi.org/10.1145/2982142.2982163 VL - 2016 SP - 81 EP - 90 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Götzelmann, Timo T1 - Interactive Tactile Maps for Blind People using Smartphones' Integrated Cameras T2 - Proc. 9th ACM International Conference on Interactive Tabletops and Surfaces (ITS'14) N2 - Tactile maps may support blind persons in orientation and understanding geographical relations, but their availability is still very limited. However, recent technologies such as 3D printers allow to autonomously print individual tactile maps which can be linked with interactive applications. Besides geographical depictions, textual annotation of maps is crucial. However, this often adds much complexity to tactile maps. To limit tactile complexity, interactive approaches may help to complement maps by the auditive modality. The presented approach integrates barcodes into tactile maps to allow their detection by standard smartphones' cameras. Automatically, more detailed map data is obtained to auditively support the exploration of the tactile map. Our experimental implementation shows the principal feasibility and provides the basis of ongoing comprehensive user studies. Y1 - 2014 SN - 978-1-4503-2587-5 U6 - https://doi.org/10.1145/2669485.2669550 VL - 2014 SP - 381 EP - 385 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Fischer, Alexander A1 - Tavakoli Kolagari, Ramin T1 - Harmonization of Security Standards for Embedded Systems in a Comprehensive Metamodel: The Security Abstraction Model T2 - 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C) N2 - The development of reliable systems is inextricably linked to the consideration of security aspects during the development process. Especially in the context of autonomous vehicles, security is crucial to ensure user trust in this technology. This work presents an overview of the harmonization of the Security Abstration Model (SAM). The results demonstrate that our approach is compliant with relevant security standards for embedded systems, indicating compliance and robustness. The utilization of SAM improves conceptual understanding and promotes clarity in security engineering. Y1 - 2024 SN - 979-8-3503-6565-8 U6 - https://doi.org/10.1109/QRS-C63300.2024 SP - 1314 EP - 1315 PB - IEEE ER - TY - CHAP A1 - Wolf, Philipp A1 - Götzelmann, Timo T1 - A Systematic Review of Interaction Approaches based on Visually Evoked Potentials N2 - BCIs are already present in the mass market. Visual evoked potentials (VEP) are based on the recognition of recurrent visual stimuli by specific BCI. A variety of approaches use this technology to implement a whole range of different application scenarios. This paper provides an overview of approaches that use VEP either as a stand-alone technology or in combination with other technologies. First, terms for different VEP technologies and their possible procedures are introduced. The papers are classified according to the technology used, the quality of the classification, the number of simultaneous stimuli, and different application areas. One focus of the paper is value input through VEP, especially for approaches that do not work in combination with other devices. It is shown that the value input has been addressed only very little so far. Y1 - 2023 U6 - https://doi.org/10.1145/3594806.3594862 SP - 396 EP - 401 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Auernhammer, Katja A1 - Tavakoli Kolagari, Ramin T1 - Workshop for Automotive Software Systems Engineering Education T2 - 2020 IEEE 32nd Conference on Software Engineering Education and Training (CSEE&T) N2 - In view of the advent of autonomous driving, the automotive industry is longing for adequately trained employees in a wide range of fields; these include not so much the classic automotive engineering and embedded systems topics, but rather software development, robotics, machine learning, statistics and design. As a particularly specific, industry-oriented discipline, automotive software engineering is hardly represented in academic teaching, except for a few AUTOSAR courses. But as a beacon project of digitization, autonomous driving can arouse broad interest among students who are interested in how diverse disciplines can be profitably combined to realize an impressive project. This interest is an excellent chance for both universities and industry to engage in constructive dialogue, to carry out exciting multidisciplinary projects at universities and to meet the needs of industry with graduates who have a relevant and versatile education. This workshop provides an opportunity for both sides to meet, share requirements and experiences and develop a best practice course according to industry requirements and academic feasibility. KW - Automotive engineering , Software engineering , Industries , Education , Conferences , Machine learning , Automobiles Y1 - 2020 U6 - https://doi.org/10.1109/CSEET49119.2020.9206169 PB - IEEE ER - TY - CHAP A1 - Götzelmann, Timo T1 - CapMaps: Capacitive Sensing 3D Printed Audio-Tactile Maps T2 - Proc. 15th International Conference on Computers Helping People with Special Needs N2 - Tactile maps can be useful tools for blind people for navigation and orientation tasks. Apart from static maps, there are techniques to augment tactile maps with audio content. They can be used to interact with the map content, to offer extra information and to reduce the tactile complexity of a map. Studies show that audio-tactile maps can be more efficient and satisfying for the user than pure tactile maps without audio feedback. A major challenge of audio-tactile maps is the linkage of tactile elements with audio content and interactivity. This paper introduces a novel approach to link 3D printed tactile maps with mobile devices, such as smartphones and tablets, in a flexible way to enable interactivity and audio-support. By integrating conductive filaments into the printed maps it seamlessly integrates into the 3D printing process. This allows to automatically recognize the tactile map by a single press at its corner. Additionally, the arrangement of the tactile map on the mobile device is flexible and detected automatically which eases the use of these maps. The practicability of this approach is shown by a dedicated feasibility study. Y1 - 2015 SN - 978-3-319-41266-5 U6 - https://doi.org/10.1007/978-3-319-41267-2_20 VL - 2015 SP - 146 EP - 152 PB - Springer CY - Cham ER - TY - CHAP A1 - Feitl, Selina A1 - Kreimeier, Julian A1 - Götzelmann, Timo T1 - Accessible Electrostatic Surface Haptics: Towards an Interactive Audiotactile Map Interface for People With Visual Impairments T2 - Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments N2 - Tactile models, such as floor plans of a familiar or unfamiliar environment, can be helpful for people with visual impairments to grasp and interpret spatial information. Such plans are usually fabricated physically in a time-consuming process and are not interactive. This paper suggests presenting tactile floor plans using surface haptic feedback on an electrostatic display to overcome these limitations. Besides audiohaptically exploring tactile floor plans, our prototype also allows for voice interaction and demonstrates the control of smart home devices in this context. The evaluation was conducted in two stages with eight participants with visual impairments: First, it was investigated how individual rooms can be identified and assigned with electrostatic tactile feedback over a common dot matrix display. Second, the generation of a mental map when exploring an interactive detailed floor plan with several rooms was evaluated. Our results show that electrostatic haptic feedback enables people with visual impairments to recognize and understand graphic elements like rooms and a floor plan. However, this entails significantly more time for exploration and a higher cognitive workload when compared to a comparable but more expensive dot matrix display, though spatial and semantic smart home contextual information can be added through a voice interface, such as ”What is the room’s name I am touching?” or ”Turn on the power socket in this room.” Our preliminary but innovative approach reveals the initial potential of electrostatic feedback for accessibility and aims to help situate the opportunities and challenges in this context for wider dissemination. Y1 - 2022 U6 - https://doi.org/10.1145/3529190.3534781 SP - 522 EP - 531 PB - ACM CY - New York, NY, USA ER - TY - CHAP A1 - Groß, Rainer A1 - Freudenthaler, Kurt A1 - Ulrich, Thomas T1 - PROJECT-BASED LEARNING: KEY SUCCESS FACTORS AND BENEFITS FOR A TEACHING-LEARNING SETTING IN THE CONTEXT OF UNIVERSITY-INDUSTRY COOPERATION BASED ON TEN YEARS OF EXPERIENCE T2 - ICERI Proceedings N2 - Industrial companies increasingly realize that they cannot rely exclusively on their internal research and development activities for successful innovations. Targeted cooperation with universities enables them to obtain innovative and creative solutions to challenging industrial problems. Cooperations also deliver access to a pool of talents and skills, while overcoming their own operational blindness and also saving R&D costs in the long term. However, these cooperations are not a one-way street. They are also welcome at universities, as they complement the mission of practice-oriented teaching and thus contribute to applied research and the transfer of knowledge to industry. For the students, cooperative learning provides contact to industry, which can be used on the one hand for topics of final theses and on the other hand as an introduction to potential employers. For ten years now, so-called "idea competitions" have been conducted within a cooperation between Nuremberg Tech's Faculty of Computer Science and a locally-based, multinational conglomerate corporation. The idea competitions enable a didactic approach of project-based learning within a master-level course. From the beginning, the question was raised as to how the teaching-learning setting of the idea competition could be adapted to the needs and benefits of the partners involved. This setting was subject to a continuous improvement process. In addition, key success factors for the effective conduct of the course were derived. In this article, a phase-based model of the underlying course concept is presented and explained in detail. During the preparation phase of the course, the lecturers and the cooperation partner focus on finding a current, innovative project topic that is realizable for students within a master-level course. The following phase, the initialization of the project, then involves the students. In addition to the coordination of administrative subjects, the focus is above all on conveying the project assignments and the required basic and contextual knowledge. Furthermore, the student groups who compete with one another are composed. In the next phase, project work begins, with the generation of ideas by the groups. The lecturers limit themselves to a coaching role. Before transitioning to the following course phase, an additional contact with the cooperation partner takes place, with a discussion of the group exposés. During this step, the promising ideas worth pursuing are identified. These are then further developed in detail and solutions are worked out. This phase ends with the preparation of the project report, in combination with the final presentation of each group. In the final phase of the course, the first step is to critically evaluate the presentations of the groups by the lecturers. In these dry runs, the focus is on optimizing the logical sequence and the management suitability of the presentations. The highlight is then the closing event, at which the results achieved are presented and discussed in front of a jury consisting of representatives of the cooperation partner and the lecturers. The award ceremony and handing out of the certificates conclude the idea competition. In this paper, the essential experiences encountered as well as the key success factors derived and the emerging benefits gained are discussed. Finally, the application of these experiences to the current idea competition in the summer semester of 2019 is presented. KW - Project-based learning, university-industry cooperation, computer science education, idea competition, teaching-learning setting Y1 - 2019 U6 - https://doi.org/10.21125/iceri.2019.0662 SN - 2340-1095 VL - 1 SP - 2492 EP - 2499 PB - IATED ER - TY - CHAP A1 - Faria, Arlo A1 - Janin, Adam A1 - Adkoli, Sidhi A1 - Riedhammer, Korbinian T1 - Toward Zero Oracle Word Error Rate on the Switchboard Benchmark T2 - Interspeech 2022 N2 - The “Switchboard benchmark” is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best among a list of alternatives. A phrase alternatives representation is compared to utterance-level N-best lists and word-level data structures; using dense lattices and adding out-of-vocabulary words, this achieves an oracle WER of 0.18%. KW - ASR evaluation KW - Switchboard benchmark KW - oracle word error rate KW - N-best lists KW - phrase alternatives Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10959 SP - 3973 EP - 3977 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Tammewar, Aniruddha A1 - Braun, Franziska A1 - Roccabruna, Gabriel A1 - Bayerl, Sebastian P. A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - Annotation of Valence for Spoken Personal Narratives N2 - Personal Narrative (PN) is the recollection of individuals’ life experiences, events, and thoughts along with the associated emotions in the form of a story. Compared to other genres such as social media texts or microblogs, where people write about ex-perienced events or products, the spoken PNs are complex to analyze and understand. They are usually long and unstructured, involving multiple and related events, characters as well as thoughts and emotions associated with events, objects, and persons. In spoken PNs, emotions are conveyed by changing the speech signal characteristics as well as the lexical content of the narrative. In this work, we annotate a corpus of spoken personal narratives, with the emotion valence using discrete values. The PNs are segmented into speech segments, and the annotators annotate them in the discourse context, with values on a 5 point bipolar scale ranging from -2 to +2 (0 for neutral). In this way, we capture the unfolding of the PNs events and changes in the emotional state of the narrator. We perform an in-depth analysis of the inter-annotator agreement, the relation between the label distribution w.r.t. the stimulus (positive/negative) used for the elicitation of the narrative, and compare the segment-level annotations to a baseline continuous annotation. We find that the neutral score plays an important role in the agreement. We observe that it is easy to differentiate the positive from the negative valence while the confusion with the neutral label is high. KW - Personal Narratives, Emotion Annotation, Segment Level Annotation Y1 - 2022 ER - TY - CHAP A1 - Seeberger, Philipp A1 - Riedhammer, Korbinian T1 - Combining Deep Neural Reranking and Unsupervised Extraction for Multi-Query Focused Summarization N2 - The CrisisFACTS Track aims to tackle challenges such as multi-stream fact-finding in the domain of event tracking; participants' systems extract important facts from several disaster-related events while incorporating the temporal order. We propose a combination of retrieval, reranking, and the well-known Integer Linear Programming (ILP) and Maximal Marginal Relevance (MMR) frameworks. In the former two modules, we explore various methods including an entity-based baseline, pre-trained and fine-tuned Question Answering systems, and ColBERT. We then use the latter module as an extractive summarization component by taking diversity and novelty criteria into account. The automatic scoring runs show strong results across the evaluation setups but also reveal shortcomings and challenges. Y1 - 2023 U6 - https://doi.org/10.48550/arXiv.2302.01148 PB - National Institute of Standards and Technology (NIST) ER - TY - CHAP A1 - Seeberger, Philipp A1 - Riedhammer, Korbinian T1 - Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning N2 - Social media has become an important information source for crisis management and provides quick access to ongoing developments and critical information. However, classification models suffer from event-related biases and highly imbalanced label distributions which still poses a challenging task. To address these challenges, we propose a combination of entity-masked language modeling and hierarchical multi-label classification as a multi-task learning problem. We evaluate our method on tweets from the TREC-IS dataset and show an absolute performance gain w.r.t. F1-score of up to 10% for actionable information types. Moreover, we found that entity-masking reduces the effect of overfitting to in-domain events and enables improvements in cross-event generalization. Y1 - 2022 U6 - https://doi.org/10.48550/arXiv.2211.11468 SP - 70 EP - 78 PB - Association for Computational Linguistics ER - TY - CHAP A1 - Zoppelt, Markus A1 - Tavakoli Kolagari, Ramin T1 - SAM: A Security Abstraction Model for Automotive Software Systems T2 - Lecture Notes in Computer Science Y1 - 2019 SN - 9783030168735 U6 - https://doi.org/10.1007/978-3-030-16874-2_5 SN - 0302-9743 SP - 59 EP - 74 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Wägemann, Tobias A1 - Tavakoli Kolagari, Ramin A1 - Schmid, Klaus T1 - ADOOPLA - Combining Product-Line- and Product-Level Criteria in Multi-objective Optimization of Product Line Architectures T2 - Lecture Notes in Computer Science Y1 - 2019 SN - 9783030299828 U6 - https://doi.org/10.1007/978-3-030-29983-5_9 SN - 0302-9743 SP - 126 EP - 142 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Wagemann, Tobias A1 - Tavakoli Kolagari, Ramin A1 - Schmid, Klaus T1 - Exploring Automotive Stakeholder Requirements for Architecture Optimization Support T2 - 2019 IEEE International Conference on Software Architecture Companion (ICSA-C) Y1 - 2019 U6 - https://doi.org/10.1109/ICSA-C.2019.00015 SP - 37 EP - 44 PB - IEEE ER - TY - CHAP A1 - Wägemann, Tobias A1 - Langer, Tobias A1 - Mottok, Jürgen A1 - Osinski, Lukas A1 - Stappert, Friedhelm A1 - Tavakoli Kolagari, Ramin T1 - Models for Dependable Heterogenous Multi- and Many-Core System Software Design Revisited T2 - 29th International Conference on Architecture of Computing Systems Y1 - 2016 ER - TY - CHAP A1 - Auernhammer, Katja A1 - Tavakoli Kolagari, Ramin A1 - Zoppelt, Markus T1 - Attacks on machine learning: Lurking danger for accountability T2 - Proceedings of the 2019 AAAI Workshop on Artificial Intelligence Safety, SafeAI 2019 Y1 - 2019 VL - 2301 PB - CEUR-WS ER - TY - CHAP A1 - Cuenot, Philippe A1 - Frey, Patrick A1 - Johansson, Rolf A1 - Lönn, Henrik A1 - Papadopoulos, Yiannis A1 - Reiser, Mark-Oliver A1 - Sandberg, Anders A1 - Servat, David A1 - Tavakoli Kolagari, Ramin A1 - Törngren, Martin A1 - Weber, Matthias T1 - 11 The EAST-ADL Architecture Description Language for Automotive Embedded Software T2 - Lecture Notes in Computer Science N2 - Current trends in automotive embedded systems focus on how to manage the increasing software content, with a strong emphasis on standardization of the embedded software structure. The management of engineering information remains a critical challenge in order to support development and other stages of the life-cycle. System modelling based on an Architecture Description Language (ADL) is a way to keep these assets within one information structure. This paper presents the EAST-ADL2 modelling language, developed in the ITEA EAST-EEA project and further enhanced in the ATESST project (www.atesst.org). EAST-ADL2 supports comprehensive model-based development of embedded systems and provides dedicated constructs to facilitate variability and product line management, requirements engineering, representation of functional as well as software/hardware solutions, and timing and safety analysis. Y1 - 2010 SN - 9783642162763 U6 - https://doi.org/10.1007/978-3-642-16277-0_11 SN - 0302-9743 SP - 297 EP - 307 PB - Springer Berlin Heidelberg CY - Berlin, Heidelberg ER - TY - CHAP A1 - Marutschke, Daniel Moritz A1 - Brockmann, Patricia T1 - A Conceptual Framework for Multidisciplinary Data Science Education Based on IoT Projects T2 - International Workshop on Artificial Intelligence and Education (WAIE) N2 - The necessity for professionals to make predictions using digital data has been recognized for many decades. With its roots in statistical analysis and later implemented as data analytics, Data Science emerged as a widely accepted discipline. The data science life cycle begins by setting a goal and ends with reporting/reflecting on the gained results. It incorporates known analytical techniques, including modern artificial intelligence and machine learning techniques. While often considered an independent discipline, both engineering and liberal arts colleges have recognized the need for an understanding of data science in the general workforce. The literature still suggests a disconnect between the study fields and how the Internet of Things could be leveraged in Project-Based Learning (PBL) style data science education. In this paper, the authors propose a conceptual framework to teach multidisciplinary data science at university level to engineering and liberal arts students using data from real-world Internet of Things (IoT) projects. KW - Data Science KW - Internet of Things KW - Educational Technology KW - Higher Education Y1 - 2024 U6 - https://doi.org/10.1109/WAIE63876.2024.00065 VL - 2024 SP - 331 EP - 335 PB - IEEE Xplore ER - TY - CHAP A1 - Domanski, Jan A1 - Reif, Jonas A1 - Schaab, Lea A1 - Schötteler, Sebastian A1 - Schuhbauer, Heidi ED - Sever Mališ, ED - Načinović Braje, ED - Raguž Krištić, T1 - JUDGED BY ITS COVER: UNDERSTANDING THE FORMATION OF BOOK TESTIMONIAL NETWORKS T2 - Proceedings of FEB Zagreb 16th International Odyssey Conference on Economics and Business N2 - Testimonials, short promotional statements often written by notable authors or critics, are a crucial marketing tool in the publishing industry, contributing to a book’s market success. This study investigates the formation of ties in testimonial networks, focusing on both network-endogenous factors (reciprocity, transitivity and preferential attachment) and exogenous factors tied to social homophily, such as shared publisher, genre and gender. A network of 575 authors was constructed from testimonials in fiction books published between 2020 and 2024 and analyzed using Exponential Random Graph Models (ERGM). The results reveal strong effects of reciprocity, transitivity, and homophily, particularly with respect to publisher and genre. Reciprocity appears to play a key role in testimonial exchanges, suggesting that authors who provide testimonials are significantly more likely to receive testimonials in return. Authors are likely to form connections through shared relationships. However, the network does not exhibit significant inequality in tie distribution, indicating that preferential attachment is not a dominant mechanism. The results strongly confirm that shared attributes, especially publisher and genre, significantly influence testimonial formation, with publisher-based homophily being the most impactful. This reflects the role of publishers in orchestrating testimonial exchanges, as they may actively encourage authors within their networks to provide testimonials for one another. These findings offer insights into the structural dynamics of testimonial networks and provide practical implications for emerging authors seeking to establish connections in the competitive publishing market. The strong evidence for reciprocity underscores the im- portance of actively engaging in the testimonial exchange process. The lack of evidence for preferential attachment is a promising finding for emerging authors, as it suggests they are not disadvantaged by having fewer initial connections. KW - Social Network Analysis, Testimonials, Book Market, ERGM Y1 - 2025 PB - University of Zagreb, Faculty of Economics & Business CY - Zagreb ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI Workshops 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education KW - Learning Experience Platform KW - Open Source Software KW - Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 PB - Gesellschaft für Informatik e.V. ER - TY - CHAP A1 - Braun, Franziska A1 - Bayerl, Sebastian A1 - Hönig, Florian A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Infusing Acoustic Pause Context into Text-Based Dementia Assessment N2 - Speech pauses, alongside content and structure, offer a valuable and non-invasive biomarker for detecting dementia. This work investigates the use of pause-enriched transcripts in transformer-based language models to differentiate the cognitive states of subjects with no cognitive impairment, mild cognitive impairment, and Alzheimer’s dementia based on their speech from a clinical assessment. We address three binary classification tasks: Onset, monitoring, and dementia exclusion. The performance is evaluated through experiments on a German Verbal Fluency Test and a Picture Description Test, comparing the model’s effectiveness across different speech production contexts. Starting from a textual baseline, we investigate the effect of incorporation of pause information and acoustic context. We show the test should be chosen depending on the task, and similarly, lexical pause information and acoustic cross-attention contribute differently. KW - speech biomarkers KW - dementia assessment KW - neuropsychological tests KW - pathological speech Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2496 SN - 2958-1796 ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Ranzenberger, Thomas A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalizing Large Sequence-to-Sequence Speech Foundation Models With Speaker Representations N2 - We present a method to personalize large transformer-based encoderdecoder speech foundation models without the need for changes in the underlying model structure or training from scratch. This is achieved by projecting speaker-specific information into the latent space of the transformer decoder via a small neural network and learning to process the speaker information along with domainspecific information via parameter-efficient finetuning. We use this method to improve the automatic speech recognition results of spoken academic German and English. Our approach yields average relative word error rate (WER) improvements of approximately 29% on German academic speech and 25% on English academic speech. It also translates well to conversational speech, achieving relative WER improvements of up to 36%, and demonstrates modest gains of up to 5% on read speech. Moreover, we observe that incorporating utterances from the recent past as personalization context yields the most significant overall improvements and that changes in voice characteristics resulting from prolonged speaking have a minimal effect on the personalization quality of academic lectures. Y1 - 2024 U6 - https://doi.org/10.1109/SLT61566.2024.10832252 ER - TY - JOUR A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias ED - Baumann, Ilja T1 - Generative adversarial networks for whispered to voiced speech conversion: a comparative study JF - International Journal of Speech Technology N2 - Generative Adversarial Networks (GANs) have demonstrated promising results as end-to-end models for whispered to voiced speech conversion. Leveraging non-autoregressive systems like GANs capable of performing conditional waveform generation eliminates the need for separate models to estimate voiced speech features, and leads to faster inference compared to autoregressive methods. This study aims to identify the optimal GAN architecture for the whispered to voiced speech conversion task by comparing six state-of-the-art models. Furthermore, we present a method for evaluating the preservation of speaker identity and local accent, using embeddings obtained from speaker- and language identification systems. Our experimental results show that building the speech conversion system based on the HiFi-GAN architecture yields the best objective evaluation scores, outperforming the baseline by ∼9% relative using frequency-weighted Signal-to-Noise Ratio and Log Likelihood Ratio, as well as by ∼29% relative using Root Mean Squared Error. In subjective tests, HiFi-GAN yielded a mean opinion score of 2.9, significantly outperforming the baseline with a score of 1.4. Furthermore, HiFi-GAN enhanced ASR performance and preserved speaker identity and accent, with correct language detection rates of up to ∼98%. KW - Speech conversion · Generative adversarial networks · Whispered speech · Voiced speech Y1 - 2024 U6 - https://doi.org/10.1007/s10772-024-10161-1 VL - 27 ER - TY - CHAP A1 - Seeberger, Philipp A1 - Wagner, Dominik A1 - Riedhammer, Korbinian T1 - Multimodal Multimedia Event Argument Extraction with Unified Template Filling T2 - Findings of the Association for Computational Linguistics: EMNLP 2024 N2 - With the advancement of multimedia technologies, news documents and user-generated content are often represented as multiple modalities, making Multimedia Event Extraction (MEE) an increasingly important challenge. However, recent MEE methods employ weak alignment strategies and data augmentation with simple classification models, which ignore the capabilities of natural language-formulated event templates for the challenging Event Argument Extraction (EAE) task. In this work, we focus on EAE and address this issue by introducing a unified template filling model that connects the textual and visual modalities via textual prompts. This approach enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics. Experiments on the M2E2 benchmark demonstrate the effectiveness of our approach. Our system surpasses the current SOTA on textual EAE by +7% F1, and performs generally better than the second-best systems for multimedia EAE. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.findings-emnlp.381 PB - Association for Computational Linguistics CY - Miami, Florida, USA ER - TY - CHAP A1 - Wagner, Dominik A1 - Lee, Seanie A1 - Baumann, Ilja A1 - Seeberger, Philipp A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Speculative Sampling for {GPU} Hardware Accelerators T2 - Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing N2 - In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods. Y1 - 2024 U6 - https://doi.org/10.18653/v1/2024.emnlp-main.370 PB - Association for Computational Linguistics CY - Miami, Florida, USA ER - TY - CHAP A1 - Bergler, Matthias A1 - Tolvanen, Juha-Pekka A1 - Tavakoli Kolagari, Ramin T1 - Integrating Security and Safety with Systems Engineering: a Model-Based Approach N2 - Development of reliable systems requires that safety and security concerns are acknowledged during system development. Adding them afterwards is risky as many concerns are missed if not elicited together with the system requirements. Unfortunately, languages for systems engineering, like SysML, typically ignore security and safety forcing development teams to split the work into different formats, languages and tools without easy collaboration, with limited traceability, separate versioning and restricted use of automation that tools can provide. We present a model-based approach targeting automotive that integrates safety and security aspects with other system development practices. This is achieved via a comprehensive domain-specific modeling language that is extendable by language users. We demonstrate this approach with practical examples on how security and safety concerns are recognized along with traditional system design and analysis phases. KW - model-based development KW - security KW - safety KW - domainspecific language KW - system engineering KW - software engineering Y1 - 2022 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:92-opus4-14863 ER - TY - CHAP A1 - Brost, Julian A1 - Egger, Christoph A1 - Lai, Russell A1 - Schmid, Fritz A1 - Schröder, Dominique A1 - Zoppelt, Markus T1 - Threshold Password-Hardened Encryption Services N2 - Password-hardened encryption (PHE) was introduced by Lai et al. at USENIX 2018 and immediately productized by VirgilSecurity. PHE is a password-based key derivation protocol that involves an oblivious external crypto service for key derivation. The security of PHE protects against offline brute-force attacks, even when the attacker is given the entire database. Furthermore, the crypto service neither learns the derived key nor the password. PHE supports key-rotation meaning that both the server and crypto service can update their keys without involving the user. While PHE significantly strengthens data security, it introduces a single point of failure because key-derivation always requires access to the crypto service. In this work, we address this issue and simultaneously increase security by introducing threshold password-hardened encryption. Our formalization of this primitive revealed shortcomings of the original PHE definition that we also address in this work. Following the spirit of prior works, we give a simple and efficient construction using lightweight tools only. We also implement our construction and evaluate its efficiency. Our experiments confirm the practical efficiency of our scheme and show that it is more efficient than common memory-hard functions, such as scrypt. From a practical perspective this means that threshold PHE can be used as an alternative to scrypt for password protection and key-derivation, offering better security in terms of offline brute force attacks. CCS CONCEPTS • Security and privacy → Cryptography. Y1 - 2020 U6 - https://doi.org/10.1145/3372297.3417266 SP - 409 EP - 424 PB - ACM ER - TY - CHAP A1 - Bergler, Matthias A1 - Tavakoli Kolagari, Ramin A1 - Lundqvist, Kristina T1 - Case study on the use of the SafeML approach in training autonomous driving vehicles N2 - The development quality for the control software for autonomous vehicles is rapidly progressing, so that the control units in the field generally perform very reliably. Nevertheless, fatal misjudgments occasionally occur putting people at risk: such as the recent accident in which a Tesla vehicle in Autopilot mode rammed a police vehicle. Since the object recognition software which is a part of the control software is based on machine learning (ML) algorithms at its core, one can distinguish a training phase from a deployment phase of the software. In this paper we investigate to what extent the deployment phase has an impact on the robustness and reliability of the software; because just as traditional, software based on ML degrades with time. A widely known effect is the so-called concept drift: in this case, one finds that the deployment conditions in the field have changed and the software, based on the outdated training data, no longer responds adequately to the current field situation. In a previous research paper, we developed the SafeML approach with colleagues from the University of Hull, where datasets are compared for their statistical distance measures. In doing so, we detected that for simple, benchmark data, the statistical distance correlates with the classification accuracy in the field. The contribution of this paper is to analyze the applicability of the SafeML approach to complex, multidimensional data used in autonomous driving. In our analysis, we found that the SafeML approach can be used for this data as well. In practice, this would mean that a vehicle could constantly check itself and detect concept drift situation early. KW - Automotive KW - Safety KW - SafeML KW - Machine Learning KW - Autonomous Driving Y1 - 2022 ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Glockner, Kevin A1 - Herygers, Aaricia A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Zakaria, Khabbab T1 - EXTENDING HANS: LARGE LANGUAGE MODELS FOR QUESTION ANSWERING, SUMMARIZATION, AND TOPIC SEGMENTATION IN AN ML-BASED LEARNING EXPERIENCE PLATFORM T2 - Elektronische Sprachsignalverarbeitung 2024, Tagungsband der 35. Konferenz, Regensburg, 6.-8. März 2024 N2 - Abstract: The use of chatbots based on large language models (LLMs) and their impact on society are influencing our learning experience platform Hochschul Assistenz-System (HAnS). HAnS uses machine learning (ML) methods to support students and lecturers in the online learning and teaching processes [1]. This paper introduces LLM-based features available in HAnS which are using the transcript of our improved Automatic Speech Recognition (ASR) pipeline with an average transcription duration of 45 seconds and an average word error rate (WER) of 6.66% on over 8 hours of audio data of 7 lecture videos. A LLM-based chatbot could be used to answer questions on the lecture content as the ASR transcript is provided as context. The summarization and topic segmentation uses the LLM to improve our learning experience platform. We generate multiple choice questions using the LLM and the ASR transcript as context during playback in a period of 3 minutes and display them in the HAnS frontend Y1 - 2024 SN - 978-3-95908-325-6 PB - TUPress CY - Dresden ER - TY - CHAP A1 - Zoppelt, Markus A1 - Tavakoli Kolagari, Ramin T1 - What Today's Serious Cyber Attacks on Cars Tell Us BT - Consequences for Automotive Security and Dependability T2 - Model-Based Safety and Assessment : 6th International Symposium, IMBSA 2019, Thessaloniki, Greece, October 16–18, 2019, Proceedings N2 - Highly connected with the environment via various interfaces, cars have been the focus of malicious cyber attacks for years. These attacks are becoming an increasing burden for a society with growing vehicle autonomization: they are the sword of Damocles of future mobility. Therefore, research is particularly active in the area of vehicle IT security, and in part also in the area of dependability, in order to develop effective countermeasures and to maintain a minimum of one step ahead of hackers. This paper examines the known state-of-the-art security and dependability measures based on a detailed and systematic analysis of published cyber attacks on automotive software systems. The sobering result of the analysis of the cyber attacks with the model-based technique SAM (Security Abstraction Model) and a categorization of the examined attacks in relation to the known security and dependability measures is that most countermeasures against cyber attacks are hardly effective. They either are not applicable to the underlying problem or take effect too late; the intruder has already gained access to a substantial part of the vehicle when the countermeasures apply. The paper is thus contributing to an understanding of the gaps that exist today in the area of vehicle security and dependability and concludes concrete research challenges. KW - Automotive Security KW - Automotive System Architecture KW - Dependability KW - Model-Driven Engineering Methodologies Y1 - 2019 SN - 978-3-030-32872-6 U6 - https://doi.org/10.1007/978-3-030-32872-6_18 SP - 219 EP - 285 PB - Springer CY - Cham ER - TY - CHAP A1 - Zoppelt, Markus A1 - Tavakoli Kolagari, Ramin T1 - UnCle SAM: Modeling Cloud Attacks with the Automotive Security Abstraction Model T2 - CLOUD COMPUTING 2019 : The Tenth International Conference on Cloud Computing, GRIDs, and Virtualization N2 - Driverless (autonomous) vehicles will have greater attack potential than any other individual mobility vehicles ever before. Most intelligent vehicles require communication interfaces to the environment, direct connections (e.g., Vehicle-to-X (V2X)) to an Original Equipment Manufacturer (OEM) backend service or a cloud. By connecting to the Internet, which is not only necessary for the infotainment systems, cars could increasingly turn into targets for malware or botnet attacks. Remote control via the Internet by a remote attacker is also conceivable, as has already been impressively demonstrated. This paper examines security modeling for cloud-based remote attacks on autonomous vehicles using a Security Abstraction Model (SAM) for automotive software systems). SAM adds to the early phases of (automotive) software architecture development by explicitly documenting attacks and handling them with security techniques. SAM also provides the basis for comprehensive security analysis techniques, such as the already available Common Vulnerability Scoring System (CVSS) or any other attack assessment system. KW - Automotive Security KW - Automotive Software Engineering KW - Security Modeling KW - Cloud Attacks KW - OTA Updates Y1 - 2019 SN - 978-1-61208-703-0 ER - TY - CHAP A1 - Wägemann, Tobias A1 - Werner, Albert T1 - Generating Multi-objective Programs from Variant-rich EAST-ADL Product Line Architectures T2 - INFORMATIK 2015 N2 - The design of robust architectures for software-intensive systems in the automotive domain is a complex task and a considerable engineering challenge. Even conventional systems are usually subject to a multitude of conflicting design goals, such as unit cost and weight minimization, dependability augmentation and timing control. One feasible approach to resolve these conflicting levers on a system's architecture is to perform an optimization analysis on a concretely defined design space, which in general is vast. For the purpose of the approach presented here, this design space is represented by an automated identification of variation points relevant for design space exploration. The identified variability information is then transformed into a convenient mathematical representation for product-line-aware architecture optimization. KW - Model-based Analysis KW - Multi-objective Programming KW - Architecture Optimization KW - Pareto Optimality KW - Product Line Engineering Y1 - 2015 SN - 978-3-88579-640-4 SN - 1617-5468 SP - 1673 EP - 1685 PB - Gesellschaft für Informatik e.V.. CY - Bonn ER - TY - JOUR A1 - Pérez-Toro, Paula Andrea A1 - Vásquez-Correa, Juan Camilo A1 - Bocklet, Tobias A1 - Nöth, Elmar A1 - Orozco-Arroyave, Juan Rafael T1 - User State Modeling Based on the Arousal-Valence Plane: Applications in Customer Satisfaction and Health-Care JF - IEEE Transactions on Affective Computing N2 - The acoustic analysis helps to discriminate emotions according to non-verbal information, while linguistics aims to capture verbal information from written sources. Acoustic and linguistic analyses can be addressed for different applications, where information related to emotions, mood, or affect are involved. The Arousal-Valence plane is commonly used to model emotional states in a multidimensional space. This study proposes a methodology focused on modeling the user’s state based on the Arousal-Valence plane in different scenarios. Acoustic and linguistic information are used as input to feed different deep learning architectures mainly based on convolutional and recurrent neural networks, which are trained to model the Arousal-Valence plane. The proposed approach is used for the evaluation of customer satisfaction in call-centers and for health-care applications in the assessment of depression in Parkinson’s disease and the discrimination of Alzheimer’s disease. F-scores of up to 0.89 are obtained for customer satisfaction, of up to 0.82 for depression in Parkinson’s patients, and of up to 0.80 for Alzheimer’s patients. The proposed approach confirms that there is information embedded in the Arousal-Valence plane that can be used for different purposes. KW - Arousal-valence plane KW - acoustic KW - linguistic KW - customer satisfaction KW - alzheimer’s disease KW - depression Y1 - 2021 U6 - https://doi.org/10.1109/taffc.2021.3112543 SN - 1949-3045 VL - 14 IS - 2 SP - 1533 EP - 1546 PB - Institute of Electrical and Electronics Engineers (IEEE) ER - TY - CHAP A1 - Perez-Toro, P. A. A1 - Vasquez-Correa, J. C. A1 - Arias-Vergara, T. A1 - Klumpp, P. A1 - Sierra-Castrillon, M. A1 - Roldan-Lopez, M. E. A1 - Aguillon, D. A1 - Hincapie-Henao, L. A1 - Tobon-Quintero, C. A. A1 - Bocklet, Tobias A1 - Schuster, M. A1 - Orozco-Arroyave, J. R. A1 - Nöth, E. T1 - Acoustic and Linguistic Analyses to Assess Early-Onset and Genetic Alzheimer’s Disease T2 - ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - The PSEN1-E280A or Paisa mutation is responsible for most of Early-Onset Alzheimer’s (EOA) disease cases in Colombia. It affects a large kindred of over 5000 members that present the same phenotype. The most common symptoms are related to language disorders, where speech fluency is also affected due to the difficulty to access semantic information intentionally. This study proposes the use of acoustic and linguistic methods to extract features from speech recordings and their transcriptions to discriminate people with conditions related to the Paisa mutation. We consider state-of-the-art word-embedding methods like Word2Vec and Bidirectional Encoder Representations from Transformer to process the transcripts. The speech signals are modeled by using traditional acoustic features and speaker embeddings. To the best of our knowledge, this is the first study focused on evaluating genetic Alzheimer’s and EOA using acoustics and linguistics. KW - PSEN1–E280A KW - Alzheimer’s Disease KW - Acoustic Analysis KW - Linguistic Analysis Y1 - 2021 SN - 978-1-7281-7605-5 U6 - https://doi.org/10.1109/ICASSP39728.2021.9414009 SP - 8338 EP - 8342 PB - IEEE ER - TY - CHAP A1 - Scheuerer, Ralph A1 - Haderlein, Tino A1 - Nöth, Elmar A1 - Bocklet, Tobias T1 - Applying X-Vectors on Pathological Speech After Larynx Removal T2 - 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) N2 - Speaker embeddings extracted from time delayed neural networks (TDNNs) contributed to major recent advancements in speaker recognition and verification. We use an X-Vector system trained on augmented VoxCeleb1 and VoxCeleb2 data to obtain embeddings for pathological speech after total or partial larynx removal. We show that our model is able to effectively distinguish and visualize patient groups when generating embeddings. We further compare various regression models on the task of automatically predicting different perceptual ratings by speech therapists (intelligibility, vocal effort, and overall quality) based on the extracted speaker embeddings. For both patient groups we show Pearson correlations in the range of +0.8; we find that Random Forest and Support Vector Regression produce scores that best resemble the experts' assessments. KW - laryngectomy KW - intelligibility KW - pathological speech KW - x-vectors Y1 - 2021 SN - 978-1-6654-3739-4 U6 - https://doi.org/10.1109/asru51503.2021.9688278 VL - 2021 SP - 1079 EP - 1086 PB - IEEE ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas H. A1 - Bayerl, Sebastian P. A1 - Auerbach, Thomas A1 - Bocklet, Tobias T1 - An Acoustical Machine Learning Approach to Determine Abrasive Belt Wear of Wide Belt Sanders T2 - 2022 IEEE Sensors N2 - This paper describes a machine learning approach to determine the abrasive belt wear of wide belt sanders used in industrial processes based on acoustic data, regardless of the sanding process-related parameters, Feed speed, Grit Size, and Type of material. Our approach utilizes Decision Tree, Random Forest, k-nearest Neighbors, and Neural network Classifiers to detect the belt wear from Spectrograms, Mel Spectrograms, MFCC, IMFCC, and LFCC, yielding an accuracy of up to 86.1% on five levels of belt wear. A 96% accuracy could be achieved with different Decision Tree Classifiers specialized in different sanding parameter configurations. The classifiers could also determine with an accuracy of 97% if the machine is currently sanding or is idle and with an accuracy of 98.4% and 98.8% detect the sanding parameters Feed speed and Grit Size. We can show that low-dimensional mappings of high-dimensional features can be used to visualize belt wear and sanding parameters meaningfully. KW - Acoustic sensors KW - Abrasive belt wear KW - Tool wear KW - Machine learning KW - Industrial process KW - Wide belt sanding machines Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/SENSORS52175.2022.9967324 VL - 2022 PB - IEEE ER - TY - CHAP A1 - Schmitt, Thomas H. A1 - Bundscherer, Maximilian A1 - Drechsel, Ralf A1 - Bocklet, Tobias T1 - Machine learning based optimization of a ceramic bushing manufacturing process T2 - 2022 IEEE Sensors N2 - Machine learning (ML) has shown great promise in a variety of domains in recent years. ML models are known to require large amounts of labeled training data, keeping small to medium-sized business from utilizing them. This paper presents ML based approach to optimize a ceramic bushing manufac-turing process, by predicting the employed press-fit process as a function of press punch position. Accurate predictions would ensure optimal process configuration, guaranteeing quality and reducing waste. Models are trained in a supervised manner to predict the press-fit process and the ceramic defect probabilities as functions of press punch position. We were able to predict the press-fit process with a mean correlation of 0.996 and assess whether the process would damage the ceramic with a mean precision of 96.7%. Our results exemplify how ML can be used to predict and optimize highly specialised processes even with small datasets. KW - Manufacturing KW - machine learning KW - optimization KW - ceramic bushing Y1 - 2022 SN - 978-1-6654-8464-0 U6 - https://doi.org/10.1109/sensors52175.2022.9967124 PB - IEEE ER - TY - CHAP A1 - Klumpp, P. A1 - Bocklet, Tobias A1 - Arias-Vergara, T. A1 - Vásquez-Correa, J. C. A1 - Pérez-Toro, P.A. A1 - Bayerl, Sebastian P. A1 - Orozco-Arroyave, J. R. A1 - Nöth, Elmar T1 - The Phonetic Footprint of Covid-19? T2 - Interspeech 2021 N2 - Against the background of the ongoing pandemic, this year’s Computational Paralinguistics Challenge featured a classification problem to detect Covid-19 from speech recordings. The presented approach is based on a phonetic analysis of speech samples, thus it enabled us not only to discriminate between Covid and non-Covid samples, but also to better understand how the condition influenced an individual’s speech signal. Our deep acoustic model was trained with datasets collected exclusively from healthy speakers. It served as a tool for segmentation and feature extraction on the samples from the challenge dataset. Distinct patterns were found in the embeddings of phonetic classes that have their place of articulation deep inside the vocal tract. We observed profound differences in classification results for development and test splits, similar to the baseline method. We concluded that, based on our phonetic findings, it was safe to assume that our classifier was able to reliably detect a pathological condition located in the respiratory tract. However, we found no evidence to claim that the system was able to discriminate between Covid-19 and other respiratory diseases. KW - COVID-19 Y1 - 2021 U6 - https://doi.org/10.21437/Interspeech.2021-1488 SN - 2958-1796 SP - 441 EP - 445 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Bocklet, Tobias T1 - Nonwords Pronunciation Classification in Language Development Tests for Preschool Children T2 - Interspeech 2022 N2 - This work aims to automatically evaluate whether the language development of children is age-appropriate. Validated speech and language tests are used for this purpose to test the auditory memory. In this work, the task is to determine whether spoken nonwords have been uttered correctly. We compare different approaches that are motivated to model specific language structures: Low-level features (FFT), speaker embeddings (ECAPA-TDNN), grapheme-motivated embeddings (wav2vec 2.0), and phonetic embeddings in form of senones (ASR acoustic model). Each of the approaches provides input for VGG-like 5-layer CNN classifiers. We also examine the adaptation per nonword. The evaluation of the proposed systems was performed using recordings from different kindergartens of spoken nonwords. ECAPA-TDNN and low-level FFT features do not explicitly model phonetic information; wav2vec2.0 is trained on grapheme labels, our ASR acoustic model features contain (sub-)phonetic information. We found that the more granular the phonetic modeling is, the higher are the achieved recognition rates. The best system trained on ASR acoustic model features with VTLN achieved an accuracy of 89.4% and an area under the ROC (Receiver Operating Characteristic) curve (AUC) of 0.923. This corresponds to an improvement in accuracy of 20.2% and AUC of 0.309 relative compared to the FFT-baseline. Y1 - 2022 U6 - https://doi.org/10.21437/interspeech.2022-10777 SN - 2958-1796 VL - 2022 SP - 3643 EP - 3647 PB - ISCA ER - TY - CHAP A1 - Chen, Wenda A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Length- and Noise-Aware Training Techniques for Short-Utterance Speaker Recognition T2 - Interspeech 2020 N2 - Speaker recognition performance has been greatly improved with the emergence of deep learning. Deep neural networks show the capacity to effectively deal with impacts of noise and reverberation, making them attractive to far-field speaker recognition systems. The x-vector framework is a popular choice for generating speaker embeddings in recent literature due to its robust training mechanism and excellent performance in various test sets. In this paper, we start with early work on including invariant representation learning (IRL) to the loss function and modify the approach with centroid alignment (CA) and length variability cost (LVC) techniques to further improve robustness in noisy, far-field applications. This work mainly focuses on improvements for short-duration test utterances (1-8s). We also present improved results on long-duration tasks. In addition, this work discusses a novel self-attention mechanism. On the VOiCES far-field corpus, the combination of the proposed techniques achieves relative improvements of 7.0% for extremely short and 8.2% for full-duration test utterances on equal error rate (EER) over our baseline system. KW - speaker recognition KW - invariant representation learning KW - centroid alignment KW - x-vector KW - far-field Y1 - 2020 U6 - https://doi.org/10.21437/interspeech.2020-2872 SN - 2958-1796 SP - 3835 EP - 3839 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Detecting Vocal Fatigue with Neural Embeddings JF - Journal of Voice N2 - Vocal fatigue refers to the feeling of tiredness and weakness of voice due to extended utilization. This paper investigates the effectiveness of neural embeddings for the detection of vocal fatigue. We compare x-vectors, ECAPA-TDNN, and wav2vec 2.0 embeddings on a corpus of academic spoken English. Low-dimensional mappings of the data reveal that neural embeddings capture information about the change in vocal characteristics of a speaker during prolonged voice usage. We show that vocal fatigue can be reliably predicted using all three types of neural embeddings after 40 minutes of continuous speaking when temporal smoothing and normalization are applied to the extracted embeddings. We employ support vector machines for classification and achieve accuracy scores of 81% using x-vectors, 85% using ECAPA-TDNN embeddings, and 82% using wav2vec 2.0 embeddings as input features. We obtain an accuracy score of 76%, when the trained system is applied to a different speaker and recording environment without any adaptation. KW - Vocal fatigue KW - Neural embeddings KW - Visualization KW - Detection Y1 - 2023 U6 - https://doi.org/10.1016/j.jvoice.2023.01.012 SN - 0892-1997 PB - Elsevier BV ER - TY - CHAP A1 - Bayerl, Sebastian P. A1 - Wagner, Dominik A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - The Influence of Dataset Partitioning on Dysfluency Detection Systems T2 - Text, Speech, and Dialogue N2 - This paper empirically investigates the influence of different data splits and splitting strategies on the performance of dysfluency detection systems. For this, we perform experiments using wav2vec 2.0 models with a classification head as well as support vector machines (SVM) in conjunction with the features extracted from the wav2vec 2.0 model to detect dysfluencies. We train and evaluate the systems with different non-speaker-exclusive and speaker-exclusive splits of the Stuttering Events in Podcasts (SEP-28k) dataset to shed some light on the variability of results w.r.t. to the partition method used. Furthermore, we show that the SEP-28k dataset is dominated by only a few speakers, making it difficult to evaluate. To remedy this problem, we created SEP-28k-Extended (SEP-28k-E), containing semi-automatically generated speaker and gender information for the SEP-28k corpus, and suggest different data splits, each useful for evaluating other aspects of methods for dysfluency detection. KW - Computer Science KW - Informatics Y1 - 2022 SN - 9783031162695 U6 - https://doi.org/10.1007/978-3-031-16270-1_35 SN - 0302-9743 SP - 423 EP - 436 PB - Springer International Publishing CY - Cham ER - TY - CHAP A1 - Wagner, Dominik A1 - Bayerl, Sebastian P. A1 - Maruri, Hector A. Cordourier A1 - Bocklet, Tobias T1 - Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech T2 - 2022 IEEE Spoken Language Technology Workshop (SLT) N2 - This work adapts two recent architectures of generative models and evaluates their effectiveness for the conversion of whispered speech to normal speech. We incorporate the normal target speech into the training criterion of vector-quantized variational autoencoders (VQ-VAEs) and Mel-GANs, thereby conditioning the systems to recover voiced speech from whispered inputs. Objective and subjective quality measures indicate that both VQ-VAEs and MelGANs can be modified to perform the conversion task. We find that the proposed approaches significantly improve the Mel cepstral distortion (MCD) metric by at least 25% relative to a Disco-GAN baseline. Subjective listening tests suggest that the MelGAN-based system significantly improves naturalness, intelligibility, and voicing compared to the whispered input speech. A novel evaluation measure based on differences between latent speech representations also indicates that our MelGAN-based approach yields improvements relative to the baseline. KW - whispered speech KW - speech conversion KW - VAE KW - GAN KW - generative models Y1 - 2023 SN - 979-8-3503-9690-4 U6 - https://doi.org/10.1109/SLT54892.2023.10022796 SP - 943 EP - 948 PB - IEEE ER - TY - CHAP A1 - Vasquez-Correa, J. C. A1 - Bocklet, Tobias A1 - Orozco-Arroyave, J. R. A1 - Nöth, E. T1 - Comparison of User Models Based on GMM-UBM and I-Vectors for Speech, Handwriting, and Gait Assessment of Parkinson’s Disease Patients T2 - ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Parkinson's disease is a neurodegenerative disorder characterized by the presence of different motor impairments. Information from speech, handwriting, and gait signals have been considered to evaluate the neurological state of the patients. On the other hand, user models based on Gaussian mixture models - universal background models (GMMUBM) and i-vectors are considered the state-of-the-art in biometric applications like speaker verification because they are able to model specific speaker traits. This study introduces the use of GMM-UBM and i-vectors to evaluate the neurological state of Parkinson's patients using information from speech, handwriting, and gait. The results show the importance of different feature sets from each type of signal in the assessment of the neurological state of the patients. KW - Parkinson’s disease KW - GMM-UBM KW - ivectors KW - gait analysis KW - handwriting analysis KW - speech analysis Y1 - 2020 SN - 978-1-5090-6631-5 U6 - https://doi.org/10.1109/icassp40776.2020.9054348 SP - 6544 EP - 6548 PB - IEEE ER - TY - CHAP A1 - Bergler, Matthias A1 - Tolvanen, Juha-Pekka A1 - Zoppelt, Markus A1 - Tavakoli Kolagari, Ramin T1 - Social Engineering Exploits in Automotive Software Security BT - Modeling Human-targeted Attacks with SAM T2 - Proceedings of the 31st European Safety and Reliability Conference (ESREL 2021) N2 - Security cannot be implemented into a system retrospectively without considerable effort, so security must be taken into consideration already at the beginning of the system development. The engineering of automotive software is by no means an exception to this rule. For addressing automotive security, the AUTOSAR and EAST-ADL standards for domain-specific system and component modeling provide the central foundation as a start. The EASTADL extension SAM enables fully integrated security modeling for traditional feature-targeted attacks. Due to the COVID-19 pandemic, the number of cyber-attacks has increased tremendously and of these, about 98 percent are based on social engineering attacks. These social engineering attacks exploit vulnerabilities in human behaviors, rather than vulnerabilities in a system, to inflict damage. And these social engineering attacks also play a relevant but nonetheless regularly neglected role for automotive software. The contribution of this paper is a novel modeling concept for social engineering attacks and their criticality assessment integrated into a general automotive software security modeling approach. This makes it possible to relate social engineering exploits with feature-related attacks. To elevate the practical usage, we implemented an integration of this concept into the established, domain-specific modeling tool MetaEdit+. The tool support enables collaboration between stakeholders, calculates vulnerability scores, and enables the specification of security objectives and measures to eliminate vulnerabilities. KW - automotive systems, social engineering attacks, design, model-based development, modeling, security Y1 - 2021 U6 - https://doi.org/10.3850/978-981-18-2016-8_720-cd SP - 2502 EP - 2509 PB - Research Publishing Services CY - Singapore ER - TY - CHAP A1 - Georges, Munir A1 - Huang, Jonathan A1 - Bocklet, Tobias T1 - Compact Speaker Embedding: lrx-Vector BT - Proceedings of the Annual Conference of the International Speech Communication Association T2 - Interspeech 2020 N2 - Deep neural networks (DNN) have recently been widely used in speaker recognition systems, achieving state-of-the-art performance on various benchmarks. The x-vector architecture is especially popular in this research community, due to its excellent performance and manageable computational complexity. In this paper, we present the lrx-vector system, which is the low-rank factorized version of the x-vector embedding network. The primary objective of this topology is to further reduce the memory requirement of the speaker recognition system. We discuss the deployment of knowledge distillation for training the lrx-vector system and compare against low-rank factorization with SVD. On the VOiCES 2019 far-field corpus we were able to reduce the weights by 28% compared to the full-rank x-vector system while keeping the recognition rate constant (1.83% EER). KW - speaker recognition KW - x-vector KW - low power Y1 - 2020 U6 - https://doi.org/10.21437/interspeech.2020-2106 SN - 2958-1796 SP - 3236 EP - 3240 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Axenie, Cristian A1 - López-Corona, Oliver A1 - Makridis, Michail A. A1 - Akbarzadeh, Meisam A1 - Saveriano, Matteo A1 - Stancu, Alexandru A1 - West, Jeffrey T1 - Antifragility in complex dynamical systems JF - npj Complexity N2 - Antifragility characterizes the benefit of a dynamical system derived from the variability in environmental perturbations. Antifragility carries a precise definition that quantifies a system’s output response to input variability. Systems may respond poorly to perturbations (fragile) or benefit from perturbations (antifragile). In this manuscript, we review a range of applications of antifragility theory in technical systems (e.g., traffic control, robotics) and natural systems (e.g., cancer therapy, antibiotics). While there is a broad overlap in methods used to quantify and apply antifragility across disciplines, there is a need for precisely defining the scales at which antifragility operates. Thus, we provide a brief general introduction to the properties of antifragility in applied systems and review relevant literature for both natural and technical systems’ antifragility. We frame this review within three scales common to technical systems: intrinsic (input–output nonlinearity), inherited (extrinsic environmental signals), and induced (feedback control), with associated counterparts in biological systems: ecological (homogeneous systems), evolutionary (heterogeneous systems), and interventional (control). We use the common noun in designing systems that exhibit antifragile behavior across scales and guide the reader along the spectrum of fragility–adaptiveness–resilience–robustness–antifragility, the principles behind it, and its practical implications. Y1 - 2024 U6 - https://doi.org/10.1038/s44260-024-00014-y SN - 2731-8753 VL - 1 IS - 1 PB - Springer Science and Business Media LLC ER - TY - CHAP A1 - Schuhbauer, Heidi A1 - Brockmann, Patricia A1 - Schötteler, Sebastian T1 - Agile learning: Use of Eduscrum in higher education T2 - Education and New Developments 2024 – Volume 2 N2 - Eduscrum is a methodology that applies agile project management principles to the field of education. The Scrum methodology is an established approach in software development, where iterative product development is carried out by a small development team in work cycles. Originally, Scrum was an agile framework that originated in software development but has since been adapted to various industries, including education. The idea behind Eduscrum is to apply agile project management principles to education, aiming to be more flexible in addressing the needs of learners, fostering collaboration, and enabling continuous feedback. Eduscrum organizes the learning process into sprints, which are time-boxed periods typically lasting one to four weeks. During each sprint, students and teachers work collaboratively to achieve specific learning objectives. Scrum defines roles which are sometimes difficult to adapt to the educational context. The role of the product owner is often taken on by the teacher, who defines the learning goals and prioritizes the content. The Scrum master moderates the learning process, removes obstacles, and ensures that the Scrum framework is followed effectively. Students take on the role of the development team. Regular meetings, often referred to as "stand-ups" or "Scrum meetings,"are held to discuss progress and challenges. These meetings provide a platform for students and teachers to collaborate, share insights, and identify any impediments. The Scrum process includes artifacts, like a backlog to prioritize tasks and charts which track the progress of the sprints. One of the key principles of Eduscrum is flexibility. It allows for adjustments based on student needs, enabling a dynamic and responsive learning environment. Eduscrum promotes regular feedback loops between students and teachers, fostering a culture of continuous improvement. Collaboration is a central theme in Eduscrum. Students work together on projects, share ideas, and contribute to the overall learning experience. The paper describes how the methodology is used in a master’s level class for computer science. Content of the course subject is social network analysis. The students work together in small groups. They perform together a self-chosen project and write a research paper together. Every process step is accompanied by the professor and an assistant, in accordance with Eduscrum. Each project group gets constant feedback. The students are guided step by step through the whole process. In the course evaluation, the students state their high learning outcomes, but they find that the projects are a lot of work. KW - Agile learning, Eduscrum, higher education, science education, social network analysis Y1 - 2024 SN - 78-989-35728-0-1 U6 - https://doi.org/10.36315/2024v2end034 SN - 2184-1489 PB - inScience Press ER - TY - CHAP A1 - Ranzenberger, Thomas A1 - Freier, Carolin A1 - Reinold, Luca A1 - Riedhammer, Korbinian A1 - Schneider, Fabian A1 - Simic, Christopher A1 - Simon, Claudia A1 - Freisinger, Steffen A1 - Georges, Munir A1 - Bocklet, Tobias T1 - A Multidisciplinary Approach to AI-based self-motivated Learning and Teaching with Large Language Models T2 - Proceedings of DELFI 2024 N2 - We present a learning experience platform that uses machine learning methods to support students and lecturers in self-motivated online learning and teaching processes. The platform is being developed as an agile open-source collaborative project supported by multiple universities and partners. The development is guided didactically, reviewed, and scientifically evaluated in several cycles. Transparency, data protection and the copyright compliant use of the system is a central part of the project. The system further employs large language models (LLMs). Due to privacy concerns, we utilize locally hosted LLM instances and explicitly do not rely on available cloud products. Students and lecturers can interact with an LLM-based chatbot in the current prototype. The AI-generated outputs contain cross-references to the current educational video’s context, indicating if sections are based on the lectures context or world knowledge. We present the prototype and results of our qualitative evaluation from the perspective of lecturers and students. KW - Artificial Intelligence in Education; Learning Experience Platform; Open Source Software; Large Language Models Y1 - 2024 U6 - https://doi.org/10.18420/delfi2024_11 SN - 2944-7682 PB - Gesellschaft für Informatik e.V. CY - Bonn ER - TY - JOUR A1 - Brockmann, Patricia A1 - Schuhbauer, Heidi T1 - Generative AI-chatbots in higher education: Challenges and opportunities in student motivation and authentic assessments JF - Education and New Developments N2 - The rapid pace of development in generative artificial intelligence (AI) has resulted in the introduction of extremely advanced chatbots, such as ChatGPT, Google Bard and Copilot. These chatbots are capable of simulating human behavior quite successfully, because they have been trained on huge data sets with large language models (LLM). New avenues for application of generative AI-chatbots have arisen not only in industry, science and medicine, but also in education. The consideration of social and ethical consequences of this new technology has not kept pace with the speed of its development. This paper explores challenges and opportunities which have arisen in higher education with the emergence of generative AI-chatbots. For university students, the temptation to use chatbots to complete their assignments may seem like a good way to save time and effort. The disadvantage is that they may not learn anything by simply copying answers from a chatbot. Inexperienced students may trust the validity of answers which are incorrect, especially if they do not have prerequisite knowledge to evaluate the output of a chatbot. For educators, however, it can be extremely difficult to determine whether students have completed their assignments themselves, or whether they have submitted the output from a chatbot. Instructors have the responsibility to teach university students to avoid plagiarism by judiciously citing the sources used. Furthermore, competencies taught at the university level should go beyond mere reproduction of facts. The development of analytical capabilities and critical thinking often require hard work to learn from mistakes, so-called “productive failure”. Creative teaching methods, such as gamification, may help to motivate students to engage with learning materials. A number of questions which have arisen from these challenges will be addressed in this paper. (1) Can educators determine whether student submissions are original or were written by a chatbot?; (2) Should students be allowed to use generative AI-chatbots to work on assessments? If so, how?; (3) Can creative teaching methods, such as gamification, motivate students to engage in learning?; (4) What kinds of assessments can help to evaluate whether students have achieved learning goals? Challenges encountered in exploring these questions when teaching undergraduate university courses will be presented. Opportunities to increase student engagement and design of authentic assessments will be discussed. KW - Artificial intelligence, chatbots, motivation, gamification, assessment. Y1 - 2024 SN - 978-989-35106-9-8 U6 - https://doi.org/10.36315/2024v2end024 SN - 2184-044X ER - TY - JOUR A1 - Altenbernd, Peter A1 - Gustafsson, Jan A1 - Lisper, Björn A1 - Stappert, Friedhelm T1 - Early execution time-estimation through automatically generated timing models JF - Real-Time Systems N2 - Traditional timing analysis, such as worst-case execution time analysis, is normally applied only in the late stages of embedded system software development, when the hardware is available and the code is compiled and linked. However, preliminary timing estimates are often needed in early stages of system development as an essential prerequisite for the configuration of the hardware setup and dimensioning of the system. During this phase the hardware is often not available, and the code might not be ready to link. This article describes an approach to predict the execution time of software through an early, source-level timing analysis. A timing model for source code is automatically derived from a given combination of hardware architecture and compiler. The model is identified from measured execution times for a set of synthetic training programs, compiled for the hardware platform in question. It can be used to estimate the execution time for code running on the platform: the estimation is then done directly from the source code, without compiling and running it. Our experiments show that, using this model, we can predict the execution times of the final, compiled code surprisingly well. For instance, we achieve an average deviation of 8% for a set of benchmark programs for the ARM7 architecture. KW - Early timing estimates · Timing model identification · WCET analysis Y1 - 2016 U6 - https://doi.org/10.1007/s11241-016-9250-7 SN - 0922-6443 VL - 52 IS - 6 SP - 731 EP - 760 PB - Springer Science and Business Media LLC ER - TY - JOUR A1 - Escobar-Grisales, Daniel A1 - Ríos-Urrego, Cristian-David A1 - Baumann, Ilja A1 - Riedhammer, Korbinian A1 - Nöth, Elmar A1 - Bocklet, Tobias A1 - Garcia, Adolfo A1 - Orozco-Arroyave, Juan rafael T1 - It’s Time to Take Action: Acoustic Modeling of Motor Verbs to Detect Parkinson’s Disease N2 - Pre-trained models generate speech representations that are used in different tasks, including the automatic detection of Parkinson’s disease (PD). Although these models can yield high accuracy, their interpretation is still challenging. This paper used a pre-trained Wav2vec 2.0 model to represent speech frames of 25ms length and perform a frame-by-frame discrimination between PD patients and healthy control (HC) subjects. This fine granularity prediction enabled us to identify specific linguistic segments with high discrimination capability. Speech representations of all produced verbs were compared w.r.t. nouns and the first ones yielded higher accuracies. To gaina deeper understanding of this pattern, representations of motor and non-motor verbs were compared and the first ones yielded better results, with accuracies of around 83% in an independent test set. These findings support well-established neurocognitive models about action-related language highlighted as key drivers of PD. Index Terms: computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease KW - computational paralinguistics, interpretability of pre-trained models, action verbs, Parkinson’s disease Y1 - 2024 U6 - https://doi.org/10.21437/Interspeech.2024-2205 ER - TY - CHAP A1 - Brockmann, Patricia T1 - Problem-Based E-Learning to Increase Motivationn of STEM-Students BT - Gamification of a Startup Enterprise T2 - Proceedings of the International Teaching and Education Conference N2 - First semester students often face major challenges in adjusting to university studies. Inexperienced students may not appreciate the practical relevance of abstract, theoretical concepts taught using traditional, instructor-centered lectures. Furthermore, the rise of generative artificial intelligence (GenAI) can mislead some to believe they no longer need to exert effort to learn academic skills. To encourage students to engage in learning material and thus improve retention rates, two complementary teaching methods were integrated into a large, introductory course for first semester STEM students: Problem-Based Learning (PBL) and gamification. Problem-Based Learning was implemented to make the subject matter more meaningful by simulating a real world experience: How to start up a small business. This entrepreneurship task was simulated in an online game, to increase student motivation. Quantitative questionnaires of student motivation and experiences with e-learning were evaluated. Based on input obtained in expert interviews, a fictitious startup enterprise was designed to serve as the used case. An online e-learning game was developed, which leveraged gamification elements to try to increase motivation. Each phase of the startup process was represented as one level of the e-learning game. The e-learning game was tested by a group of first semester students. Their opinions were collected using an anonymous online survey. Aggregated results of the survey are discussed and plans for further research are presented. KW - E-Learning, Gamification, STEM, Problem-Baseed Learning, Used-Case Learning Y1 - 2024 U6 - https://doi.org/10.33422/teduconf.v1i1.746 SN - 3030-1521 VL - 1 IS - 1 PB - Diamond Scientific Publishing ER - TY - CHAP A1 - Marutschke, Daniel Moritz A1 - Brockmann, Patricia T1 - Multicultural Observations in a Three-University Global Distributed Software Development Course BT - A UML Case Study T2 - Proceedings of the 2024 The 6th World Symposium on Software Engineering (WSSE) N2 - In our increasingly interconnected world, software engineering students need increased exposure to not only state of the art technology, but to its usage and development in multicultural environments. This paper reports on findings from a global software engineering course taught at three universities in Japan, Germany, and Indonesia. Questionnaire and interview data gathered ex-ante (before) and ex-post (after) the semester about cultural dimensions and UML usage are analyzed qualitatively and quantitatively. The perceived importance of cultural dimensions was compared to previous two-university classes. Experience reports from instructors are described. Unique dynamics in students’ perception of cultural dimensions could be observed. UML challenges self-reported are consistent with previous research findings. An ex-ante to ex-post comparison supports the implementation of real-world projects taught in PBL-style. These findings together are presented as best practices and lessons learned. KW - Global Software Engineering KW - Education KW - Distributed Teams KW - Cultural Dimenstions KW - UML Y1 - 2024 SN - 979-8-4007-1708-6 U6 - https://doi.org/10.1145/3698062.3698077 VL - 2024 SP - 115 EP - 120 PB - ACM Digital Library ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Wagner, Dominik A1 - Noeth, Elmar A1 - Riedhammer, Korbinian T1 - Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0 T2 - Interspeech 2022 N2 - Stuttering is a varied speech disorder that harms an individual’s communication ability. Persons who stutter (PWS) often use speech therapy to cope with their condition. Improving speech recognition systems for people with such non-typical speech or tracking the effectiveness of speech therapy would require systems that can detect dysfluencies while at the same time being able to detect speech techniques acquired in therapy. This paper shows that fine-tuning wav2vec 2.0 [1] for the classification of stuttering on a sizeable English corpus containing stuttered speech, in conjunction with multi-task learning, boosts the effectiveness of the general-purpose wav2vec 2.0 features for detecting stuttering in speech; both within and across languages. We evaluate our method on FluencyBank , [2] and the German therapy-centric Kassel State of Fluency (KSoF) [3] dataset by training Support Vector Machine classifiers using features extracted from the finetuned models for six different stuttering-related event types: blocks, prolongations, sound repetitions, word repetitions, interjections, and – specific to therapy – speech modifications. Using embeddings from the fine-tuned models leads to relative classification performance gains up to 27% w.r.t. F1-score. KW - stuttering KW - pathological speech KW - computational paralinguistics Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-10908 SP - 2868 EP - 2872 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Seibt, Simon A1 - Chang, Thomas A1 - Latoschik, Marc Erich A1 - Kuth, Bastian A1 - von Rymon Lipinski, Bartosz T1 - Multidimensional image morphing-fast image-based rendering of open 3D and VR environments T2 - Virtual Reality & Intelligent Hardware N2 - In recent years, the demand for interactive photorealistic three-dimensional (3D) environments has increased in various fields, including architecture, engineering, and entertainment. However, achieving a balance between the quality and efficiency of high-performance 3D applications and virtual reality (VR) remains challenging. Methods This study addresses this issue by revisiting and extending view interpolation for image-based rendering (IBR), which enables the exploration of spacious open environments in 3D and VR. Therefore, we introduce multimorphing, a novel rendering method based on the spatial data structure of 2D image patches, called the image graph. Using this approach, novel views can be rendered with up to six degrees of freedom using only a sparse set of views. The rendering process does not require 3D reconstruction of the geometry or per-pixel depth information, and all relevant data for the output are extracted from the local morphing cells of the image graph. The detection of parallax image regions during preprocessing reduces rendering artifacts by extrapolating image patches from adjacent cells in real-time. In addition, a GPU-based solution was presented to resolve exposure inconsistencies within a dataset, enabling seamless transitions of brightness when moving between areas with varying light intensities. Results Experiments on multiple real-world and synthetic scenes demonstrate that the presented method achieves high "VR-compatible" frame rates, even on mid-range and legacy hardware, respectively. While achieving adequate visual quality even for sparse datasets, it outperforms other IBR and current neural rendering approaches. Conclusions Using the correspondence-based decomposition of input images into morphing cells of 2D image patches, multidimensional image morphing provides high-performance novel view generation, supporting open 3D and VR environments. Nevertheless, the handling of morphing artifacts in the parallax image regions remains a topic for future research. KW - Computer graphics; 3D real-time rendering; Computer vision; Image morphing; Virtual reality Y1 - 2024 U6 - https://doi.org/10.1016/j.vrih.2023.06.007 VL - 7 IS - 2 ER - TY - CHAP A1 - Chang, Thomas A1 - Seibt, Simon A1 - von Rymon Lipinski, Bartosz T1 - Hierarchical Histogram Threshold Segmentation – Auto-terminating High-detail Oversegmentation T2 - 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) N2 - Superpixels play a crucial role in image processing by partitioning an image into clusters of pixels with similar visual attributes. This facilitates subsequent image processing tasks, offering computational advantages over the manipulation of individual pixels. While numerous oversegmentation techniques have emerged in recent years, many rely on predefined initialization and termination criteria. In this paper, a novel top-down superpixel segmentation algorithm called Hierarchical Histogram Threshold Segmentation (HHTS) is introduced. It eliminates the need for initialization and implements auto-termination, outperforming state-of-the-art methods w.r.t. boundary recall. This is achieved by iteratively partitioning individual pixel segments into foreground and background and applying intensity thresholding across multiple color channels. The underlying iterative process constructs a superpixel hierarchy that adapts to local detail distributions until color information exhaustion. Experimental results demonstrate the superiority of the proposed approach in terms of boundary adherence, while maintaining competitive runtime performance on the BSDS500 and NYUV2 datasets. Furthermore, an application of HHTS in refining machine learningbased semantic segmentation masks produced by the Segment Anything Foundation Model (SAM) is presented. Y1 - 2024 U6 - https://doi.org/10.1109/CVPR52733.2024.00308 SP - 3195 EP - 3204 ER - TY - GEN A1 - Seibt, Simon A1 - Chang, Thomas A1 - von Rymon Lipinski, Bartosz A1 - Latoschik, Marc Erich T1 - Dense 3D Gaussian Splatting Initialization for Sparse Image Data N2 - This paper presents advancements in novel-view synthesis with 3D Gaussian Splatting (3DGS) using a dense and accurate SfM point cloud initialization approach. We address the challenge of achieving photorealistic renderings from sparse image data, where basic 3DGS training may result in suboptimal convergence, thus leading to visual artifacts. The proposed method enhances precision and density of initially reconstructed point clouds by refining 3D positions and extrapolating additional points, even for difficult image regions, e.g. with repeating patterns and suboptimal visual coverage. Our contributions focus on improving “Dense Feature Matching for Structure-from-Motion” (DFM4SfM) based on a homographic decomposition of the image space to support 3DGS training: First, a grid-based feature detection method is introduced for DFM4SfM to ensure a welldistributed 3D Gaussian initialization uniformly over all depth planes. Second, the SfM feature matching is complemented by a geometric plausibility check, priming the homography estimation and thereby improving the initial placement of 3D Gaussians. Experimental results on the NeRF-LLFF dataset demonstrate that this approach achieves superior qualitative and quantitative results, even for fewer views, and the potential for a significantly accelerated 3DGS training with faster convergence. Y1 - 2024 U6 - https://doi.org/10.2312/egp.20241038 ER - TY - CHAP A1 - von Rymon Lipinski, Bartosz A1 - Seibt, Simon A1 - Roth, Johannes A1 - Abé, Dominik T1 - Level Graph – Incremental Procedural Generation of Indoor Levels using Minimum Spanning Trees N2 - Procedural generation of virtual worlds is an important aspect of game development since decades, typically for increasing replayability or for speeding up the level creation process. However, the utilization of this potential has always been a great challenge due to the difficult controllability of the underlying algorithms or limitations to specific level geometries, like 2D regular structures. In this paper, we present a novel approach for semi-automatic generation of a wide variety of 2D/3D corridor and room systems. The underlying processing pipeline is based on a separation between a user-guided generation of a graph-based abstract level structure and a fully-automatic construction of the corresponding geometry using a pre-modeled component library. The core algorithm is built on the computation of an extended minimal spanning tree, which can be controlled by a set of intuitive vertex and edge parameters. First experimental results have shown that our incremental generation pipeline allows the efficient creation of complex indoor levels, minimizing limitations on level and game designers’ creativity. Y1 - 2019 U6 - https://doi.org/10.1109/CIG.2019.8847956 ER - TY - CHAP A1 - Seibt, Simon A1 - von Rymon Lipinski, Bartosz A1 - Chang, Thomas A1 - Latoschik, Marc Erich T1 - DFM4SFM - Dense Feature Matching for Structure from Motion T2 - 2023 IEEE International Conference on Image Processing Challenges and Workshops (ICIPCW) N2 - Structure from motion (SfM) is a fundamental task in computer vision and allows recovering the 3D structure of a stationary scene from an image set. Finding robust and accurate feature matches plays a crucial role in the early stages of SfM. So in this work, we propose a novel method for computing image correspondences based on dense feature matching (DFM) using homographic decomposition: The underlying pipeline provides refinement of existing matches through iterative rematching, detection of occlusions and extrapolation of additional matches in critical image areas between image pairs. Our main contributions are improvements of DFM specifically for SfM, resulting in global refinement and global extrapolation of image correspondences between related views. Furthermore, we propose an iterative version of the Delaunay-triangulation-based outlier detection algorithm for robust processing of repeated image patterns. Through experiments, we demonstrate that the proposed method significantlv improves the reconstruction accuracy. Y1 - 2023 U6 - https://doi.org/10.1109/ICIPC59416.2023.10328368 SP - 3678 EP - 3682 ER - TY - CHAP A1 - Seibt, Simon A1 - Arold, Michael A1 - von Rymon Lipinski, Bartosz A1 - Wienkopf, Uwe A1 - Latoschik, Marc Erich T1 - Parallax-Aware Image Stitching Based on Homographic Decomposition N2 - Image stitching plays a crucial role for various computer vision applications, like panoramic photography, video production, medical imaging and satellite imagery. It makes it possible to align two images captured at different views onto a single image with a wider field of view. However, for 3D scenes with high depth complexity and images captured from two different positions, the resulting image pair may exhibit significant parallaxes. Stitching images with multiple or large apparent motion shifts remains a challenging task, and existing methods often fail in such cases. In this paper, a novel image stitching pipeline is introduced, addressing the aforementioned challenge: First, iterative dense feature matching is performed, which results in a multi-homography decomposition. Then, this output is used to compute a per-pixel multidimensional weight map of the estimated homographies for image alignment via weighted warping. Additionally, the homographic image space decomposition is exploited using combinatorial analysis to identify parallaxes, resulting in a parallax-aware overlapping region: Parallax-free overlapping areas only require weighted warping and blending. For parallax areas, these operations are omitted to avoid ghosting artifacts. Instead, histogram- and mask-based color mapping is performed to ensure visual color consistency. The presented experiments demonstrate that the proposed method provides superior results regarding precision and handling of parallaxes. Y1 - 2024 SN - 978-3-031-54605-1 U6 - https://doi.org/10.1007/978-3-031-54605-1_13 PB - Springer CY - Cham ER - TY - JOUR A1 - Latoschik, Marc Erich A1 - VON RYMON LIPINSKI, BARTOSZ A1 - Seibt, Simon T1 - Dense Feature Matching Based on Homographic Decomposition JF - IEEE Access N2 - Finding robust and accurate feature matches is a fundamental problem in computer vision. However, incorrect correspondences and suboptimal matching accuracies lead to significant challenges for many real-world applications. In conventional feature matching, corresponding features in an image pair are greedily searched using their descriptor distance. The resulting matching set is then typically used as input for geometric model fitting methods to find an appropriate fundamental matrix and filter out incorrect matches. Unfortunately, this basic approach cannot solve all practical problems, such as fundamental matrix degeneration, matching ambiguities caused by repeated patterns and rejection of initially mismatched features without further reconsideration. In this paper we introduce a novel matching pipeline, which addresses all of the aforementioned challenges at once: First, we perform iterative rematching to give mismatched feature points a further chance for being considered in later processing steps. Thereby, we are searching for inliers that exhibit the same homographic transformation per iteration. The resulting homographic decomposition is used for refining matches, occlusion detection (e.g. due to parallaxes) and extrapolation of additional features in critical image areas. Furthermore, Delaunay triangulation of the matching set is utilized to minimize the repeated pattern problem and to implement focused matching. Doing so, enables us to further increase matching quality by concentrating on local image areas, defined by the triangular mesh. We present and discuss experimental results with multiple real-world matching datasets. Our contributions, besides improving matching recall and precision for image processing applications in general, also relate to use cases in image-based computer graphics. KW - Delaunay triangulation, Extrapolation, Feature Matching, Homography matrix, Repeated pattern matching. Y1 - 2022 U6 - https://doi.org/10.1109/ACCESS.2022.3152539 VL - 10 ER - TY - CHAP A1 - Bayerl, Sebastian Peter A1 - Roccabruna, Gabriel A1 - Chowdhury, Shammur Absar A1 - Ciulli, Tommaso A1 - Danieli, Morena A1 - Riedhammer, Korbinian A1 - Riccardi, Giuseppe T1 - What can Speech and Language Tell us About the Working Alliance in Psychotherapy T2 - Interspeech 2022 N2 - We are interested in the problem of conversational analysis and its application to the health domain. Cognitive Behavioral Therapy is a structured approach in psychotherapy, allowing the therapist to help the patient to identify and modify the malicious thoughts, behavior, or actions. This cooperative effort can be evaluated using the Working Alliance Inventory Observer-rated Shortened – a 12 items inventory covering task, goal, and relationship – which has a relevant influence on therapeutic outcomes. In this work, we investigate the relation between this alliance inventory and the spoken conversations (sessions) between the patient and the psychotherapist. We have delivered eight weeks of e-therapy, collected their audio and video call sessions, and manually transcribed them. The spoken conversations have been annotated and evaluated with WAI ratings by professional therapists. We have investigated speech and language features and their association with WAI items. The feature types include turn dynamics, lexical entrainment, and conversational descriptors extracted from the speech and language signals. Our findings provide strong evidence that a subset of these features are strong indicators of working alliance. To the best of our knowledge, this is the first and a novel study to exploit speech and language for characterising working alliance. KW - conversational analysis KW - working alliance KW - psychotherapy Y1 - 2022 U6 - https://doi.org/10.21437/Interspeech.2022-347 SP - 2443 EP - 2447 PB - ISCA CY - ISCA ER - TY - JOUR A1 - Axenie, Cristian A1 - Saveriano, Matteo T1 - Antifragile Control Systems: The Case of Mobile Robot Trajectory Tracking Under Uncertainty and Volatility JF - IEEE Access N2 - Mobile robots are ubiquitous. Such vehicles benefit from well-designed and calibrated control algorithms ensuring their task execution under precise uncertainty bounds. Yet, in tasks involving humans in the loop, such as elderly or mobility impaired, the problem takes a new dimension. In such cases, the system needs not only to compensate for uncertainty and volatility in its operation but at the same time to anticipate and offer responses that go beyond robust. Such robots operate in cluttered, complex environments, akin to human residences, and need to face during their operation sensor and, even, actuator faults, and still operate. This is where our thesis comes into the foreground. We propose a new control design framework based on the principles of antifragility. Such a design is meant to offer a high uncertainty anticipation given previous exposure to failures and faults, and exploit this anticipation capacity to provide performance beyond robust. In the current instantiation of antifragile control applied to mobile robot trajectory tracking, we provide controller design steps, the analysis of performance under parametrizable uncertainty and faults, as well as an extended comparative evaluation against state-of-the-art controllers. We believe in the potential antifragile control has in achieving closed-loop performance in the face of uncertainty and volatility by using its exposures to uncertainty to increase its capacity to anticipate and compensate for such events. KW - Antifragile Control; Mobile Robotics; Trajectory Tracking; Uncertainty Y1 - 2023 U6 - https://doi.org/10.1109/ACCESS.2023.3339988 SN - 2169-3536 VL - 11 SP - 138188 EP - 138200 PB - Institute of Electrical and Electronics Engineers (IEEE) ER - TY - JOUR A1 - Cogno, Nicolò A1 - Axenie, Cristian A1 - Bauer, Roman A1 - Vavourakis, Vasileios T1 - Agent-based modeling in cancer biomedicine: applications and tools for calibration and validation JF - Cancer Biology & Therapy N2 - Computational models are not just appealing because they can simulate and predict the development of biological phenomena across multiple spatial and temporal scales, but also because they can integrate information from well-established in vitro and in vivo models and test new hypotheses in cancer biomedicine. Agent-based models and simulations are especially interesting candidates among computational modeling procedures in cancer research due to the capability to, for instance, recapitulate the dynamics of neoplasia and tumor – host interactions. Yet, the absence of methods to validate the consistency of the results across scales can hinder adoption by turning fine-tuned models into black boxes. This review compiles relevant literature that explores strategies to leverage high-fidelity simulations of multi-scale, or multi-level, cancer models with a focus on verification approached as simulation calibration. KW - Agent-based modeling; multi-scale; multi-level; calibration; validation; optimization; biomechanics; biophysics; cancer simulation; precision oncolog Y1 - 2024 U6 - https://doi.org/10.1080/15384047.2024.2344600 SN - 1538-4047 VL - 25 IS - 1 PB - Informa UK Limited ER - TY - BOOK A1 - Axenie, Cristian A1 - Bauer, Roman A1 - López Corona, Oliver A1 - West, Jeffrey T1 - Applied Antifragility in Natural Systems BT - From Principles to Applications N2 - As coined in the book of Nassim Taleb, antifragility is a property of a system to gain from uncertainty, randomness, and volatility, opposite to what fragility would incur. An antifragile system’s response to external perturbations is beyond robust, such that small stressors can strengthen the future response of the system by adding a strong anticipation component. Such principles are already well suited for describing behaviors in natural systems but also in approaching therapy designs and eco-system modelling and eco-system analysis. The purpose of this book is to build a foundational knowledge base by applying antifragile system design, analysis, and development in natural systems, including biomedicine, neuroscience, and ecology as main fields. We are interested in formalizing principles and an apparatus that turns the basic concept of antifragility into a tool for designing and building closed-loop systems that behave beyond robust in the face of uncertainty when characterizing and intervening in biomedical and ecological (eco)systems. The book introduces the framework of applied antifragility and possible paths to build systems that gain from uncertainty. We draw from the body of literature on natural systems (e.g. cancer therapy, antibiotics, neuroscience, and agricultural pest management) in an attempt to unify the scales of antifragility in one framework. The work of the Applied Antifragility Group in oncology, neuroscience, and ecology led by the authors provides a good overview on the current research status. T2 - with a foreword from Nassim Taleb Y1 - 2025 SN - 9783031903908 U6 - https://doi.org/10.1007/978-3-031-90391-5 SN - 2191-5768 PB - Springer Nature Switzerland CY - Cham ER - TY - BOOK A1 - Axenie, Cristian A1 - Akbarzadeh, Meisam A1 - Makridis, Michail A. A1 - Saveriano, Matteo A1 - Stancu, Alexandru T1 - Applied Antifragility in Technical Systems BT - From Principles to Applications N2 - The book purpose is to build a foundational knowledge base by applying antifragile system design, analysis, and development in technical systems, with a focus on traffic engineering, robotics, and control engineering. The authors are interested in formalizing principles and an apparatus that turns the basic concept of antifragility into a tool for designing and building closed-loop technical systems that behave beyond robust in the face of uncertainty. As coined in the book of Nassim Taleb, antifragility is a property of a system to gain from uncertainty, randomness, and volatility, opposite to what fragility would incur. An antifragile system’s response to external perturbations is beyond robust, such that small stressors can strengthen the future response of the system by adding a strong anticipation component. The work of the Applied Antifragility Group in traffic control and robotics, led by the authors, provides a good overview on the current research status. T2 - with a foreword from Nassim Taleb Y1 - 2025 SN - 9783031904240 U6 - https://doi.org/10.1007/978-3-031-90425-7 SN - 2191-5768 PB - Springer Nature Switzerland CY - Cham ER - TY - CHAP A1 - Sun, Linghang A1 - Zhang, Yifan A1 - Axenie, Cristian A1 - Grossi, Margherita A1 - Kouvelas, Anastasios A1 - Makridis, Michail T1 - The fragile nature of road transportation systems : conference paper N2 - Major cities worldwide experience problems with the performance of their road transportation systems, and the continuous increase in traffic demand presents a substantial challenge to the optimal operation of urban road networks and the efficiency of traffic control strategies. Although robust and resilient transportation systems have been extensively researched over the past decades, their performance under an ever-growing traffic demand can still be questionable. The operation of transportation systems is widely believed to display fragile property, i.e., the loss in performance increases exponentially with the linearly increasing magnitude of disruptions, which undermines their continuous operation. Nowadays, the risk engineering community is embracing the novel concept of antifragility, which enables systems to learn from historical disruptions and exhibit improved performance as disruption levels reach unprecedented magnitudes. In this study, we demonstrate the fragile nature of road transportation systems when faced with demand or supply disruptions. First, we conducted a rigorous mathematical analysis to establish the fragile nature of the systems theoretically. Subsequently, by taking into account real-world stochasticity, we implemented a numerical simulation with realistic network data to bridge the gap between the theoretical proof and the real-world operations, to reflect the potential impact of uncertainty on the fragile property of the systems. This work aims to demonstrate the fragility of road transportation systems and help researchers better comprehend the necessity to explicitly consider antifragile design for future traffic control strategies, coping with the constantly growing traffic demand and subsequent traffic accidents. KW - (anti-)fragility, road transportation systems, macroscopic fundamental diagram, model stochasticity Y1 - 2024 U6 - https://doi.org/10.3929/ethz-b-000681460 ER - TY - CHAP A1 - Wagner, Dominik A1 - Churchill, Alexander A1 - Sigtia, Siddharth A1 - Marchi, Erik T1 - SELMA: A Speech-Enabled Language Model for Virtual Assistant Interactions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - In this work, we present and evaluate SELMA, a Speech-Enabled Language Model for virtual Assistant interactions that integrates audio and text as inputs to a Large Language Model (LLM). SELMA is designed to handle three primary and two auxiliary tasks related to interactions with virtual assistants simultaneously within a single end-to-end model. We employ low-rank adaptation modules for parameter-efficient training of both the audio encoder and the LLM. Additionally, we implement a feature pooling strategy enabling the system to recognize global patterns and improve accuracy on tasks less reliant on individual sequence elements. Experimental results on Voice Trigger (VT) detection, Device-Directed Speech Detection (DDSD), and Automatic Speech Recognition (ASR), demonstrate that our approach both simplifies the typical input processing pipeline of virtual assistants significantly and also improves performance compared to dedicated models for each individual task. SELMA yields relative Equal-Error Rate improvements of 64% on the VT detection task, and 22% on DDSD, while also achieving word error rates close to the baseline. KW - multi-task KW - multimodal KW - virtual assistant KW - large language KW - model KW - low-rank adaptation Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10890139 SN - 2379-190X SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Lee, Seanie A1 - Seong, Haebin A1 - Bok Lee, Dong A1 - Kang, Minki A1 - Chen, Xiaoyin A1 - Wagner, Dominik A1 - Bengio, Joshua A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models N2 - Safety guard models that detect malicious queries aimed at large language models(LLMs) are essential for ensuring the secure and responsible deployment of LLMs in real-world applications. However, deploying existing safety guard models with billions of parameters alongside LLMs on mobile devices is impractical due to substantial memory requirements and latency. To reduce this cost, we distill a large teacher safety guard model into a smaller one using a labeled dataset of instruction-response pairs with binary harmfulness labels. Due to the limited diversity of harmful instructions in the existing labeled dataset, naively distilled models tend to underperform compared to larger models. To bridge the gap between small and large models, we propose HarmAug, a simple yet effective data augmentation method that involves jailbreaking an LLM and prompting it to generate harmful instructions. Given a prompt such as, “Make a single harmful instruction prompt that would elicit offensive content”, we add an affirmative prefix (e.g., “I have an idea for a prompt:”) to the LLM’s response. This encourages the LLM to continue generating the rest of the response, leading to sampling harmful instructions. Another LLM generates a response to the harmful instruction, and the teacher model labels the instruction-response pair. We empirically show that our HarmAug outperforms other relevant baselines. Moreover, a 435-millionparameter safety guard model trained with HarmAug achieves an F1 score comparable to larger models with over 7 billion parameters, and even outperforms them in AUPRC, while operating at less than 25% of their computational cost. Our code, safety guard model, and synthetic dataset are publicly available. Y1 - 2025 U6 - https://doi.org/10.48550/arXiv.2410.01524 SP - 1 EP - 20 PB - ICLR ER - TY - CHAP A1 - Fischer, Alexander A1 - Tolvanen, Juha-Pekka A1 - Tavakoli Kolagari, Ramin T1 - Embedded Systems Security Co-design: Modeling Support for Managers and Developers T2 - Lecture Notes in Business Information Processing N2 - The proliferation of connected and autonomous vehicle technologies has significantly increased cybersecurity risks. Modern vehicles, as complex and networked computer systems, require comprehensive protection against malicious external attacks, much like conventional computers. Addressing these challenges requires robust tools that align established automotive model-based development approaches with the ISO/SAE 21434 standard for automotive cybersecurity, which became mandatory following its publication in 2021. Building on prior research, this paper introduces key innovations in the conceptual framework and the tool support that integrate seamlessly into existing automotive development methodologies. These advancements are rooted in extensions to the Security Abstraction Model (SAM) informed by the ISO/SAE 21434 standard. Notably, SAM now incorporates advanced methods for score calculation, including an attack potential-based approach for assessing attack feasibility and the computation of risk scores using risk matrices. Usability improvements are also a contribution, achieved through the introduction of BPMN-style (Business Process Model and Notation) diagrams tailored for the accessible visualization of otherwise complex security models. These diagrams make multifaceted attack trees easier to interpret, enabling managers and other non-technical stakeholders to intuitively understand security vulnerabilities and make informed decisions. Additionally, the tool supports updated metrics for impact and risk analysis, demonstrated through practical applications involving automotive subsystems such as braking. These examples illustrate improved traceability between SAM and functional design, ensuring that cybersecurity requirements are effectively integrated into the broader development lifecycle. Y1 - 2025 SN - 978-3-031-84913-8 U6 - https://doi.org/10.1007/978-3-031-84913-8_8 SN - 1865-1348 SP - 206 EP - 232 PB - Springer Nature Switzerland CY - Cham ER - TY - CHAP A1 - Fischer, Alexander A1 - Burk, Louis A1 - Tavakoli Kolagari, Ramin A1 - Wienkop, Uwe T1 - Machine-Readable by Design: Language Specifications as the Key to Integrating LLMs into Industrial Tools T2 - Annals of Computer Science and Information Systems N2 - We propose a meta-language-based approach enabling Large Language Models (LLMs) to reliably generate structured, machine-readable artifacts referred to as Meta-Language-defined Structures (MLDS) adapted to domain requirements, without adhering strictly to standard formats like JSON or XML. By embedding explicit schema instructions within prompts, we evaluated the method across diverse use cases, including automated Virtual Reality environment generation and automotive security modeling. Our experiments demonstrate that the meta-language approach significantly improves LLM-generated structure compliance, with an 88 % validation rate across 132 test scenarios. Compared to traditional methods using LangChain and Pydantic, our MLDS method reduces setup complexity by approximately 80 %, despite a marginally higher error rate. Furthermore, the MLDS artifacts produced were easily editable, enabling rapid iterative refinement. This flexibility greatly alleviates the “blank page syndrome” by providing structured initial artifacts suitable for immediate use or further human enhancement, making our approach highly practical for rapid prototyping and integration into complex industrial workflows. Y1 - 2025 SN - 978-83-973291-6-4 U6 - https://doi.org/10.15439/2025F5613 SN - 2300-5963 VL - 43 SP - 531 EP - 542 PB - IEEE ER - TY - CHAP A1 - Fischer, Alexander A1 - Tavakoli Kolagari, Ramin T1 - Towards Robust Autonomous Vehicle Software: Extending SAM for a Holistic Security Analysis T2 - 2024 8th International Conference on System Reliability and Safety (ICSRS) N2 - The Security Abstraction Model (SAM) is a meta-model that enables comprehensive security modeling and does not only consider technological aspects, but also takes into consideration the influence of social engineering. This plays an important role by exploiting vulnerabilities in human behavior and thus represents a dimension that must not be neglected in possible attacks on vehicle systems. This work is dedicated to the advancement of the Security Abstraction Model (SAM) with the aim of enabling improved modelling and analysis of vulnerabilities in the context of software and system security; the relevant standards and best practices of the international security literature for embedded systems are harmonized in one metamodel. By extending SAM, this research enables a more nuanced understanding and representation of vulnerabilities, attacks and scenarios, improving the effectiveness of security analysis and remediation strategies. The integration of different perspectives, including technological, social and behavioral factors, underlines the comprehensive nature of the proposed model. This interdisciplinary approach recognizes the complexity of security threats in autonomous vehicle systems and enables stakeholders to develop more effective countermeasures. This work presents an overview of the extension of the Security Abstration Model (SAM) and evaluates this security technology through expert interviews. The results demonstrate that our approach is compliant not only in the automotive domain, but also with standards for embedded systems from other domains. It thus provides a comprehensive and universal approach for the development of secure systems. Y1 - 2024 SN - 979-8-3503-5450-8 U6 - https://doi.org/10.1109/ICSRS63046.2024.10927601 SP - 578 EP - 586 PB - IEEE ER - TY - CHAP A1 - Braun, Franziska A1 - Witzl, Christopher A1 - Erzigkeit, Andreas A1 - Lehfeld, Hartmut A1 - Hillemacher, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Pitfalls and Limits in Automatic Dementia Assessment T2 - Interspeech 2025 N2 - Current work on speech-based dementia assessment focuses on either feature extraction to predict assessment scales, or on the automation of existing test procedures. Most research uses public data unquestioningly and rarely performs a detailed error analysis, focusing primarily on numerical performance. We perform an in-depth analysis of an automated standardized dementia assessment, the Syndrom-Kurz-Test. We find that while there is a high overall correlation with human annotators, due to certain artifacts, we observe high correlations for the severely impaired individuals, which is less true for the healthy or mildly impaired ones. Speech production decreases with cognitive decline, leading to overoptimistic correlations when test scoring relies on word naming. Depending on the test design, fallback handling introduces further biases that favor certain groups. These pitfalls remain independent of group distributions in datasets and require differentiated analysis of target groups. KW - dementia screening KW - neuropsychological tests KW - pathological speech Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2280 SN - 2958-1796 SP - 5663 EP - 5667 PB - ISCA ER - TY - CHAP A1 - Bundscherer, Maximilian A1 - Schmitt, Thomas A1 - Baumann, Ilja A1 - Bocklet, Tobias T1 - Digital Operating Mode Classification of Real-World Amateur Radio Transmissions T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - This study presents an ML approach for classifying digital radio operating modes evaluated on real-world transmissions. We generated 98 different parameterized radio signals from 17 digital operating modes, transmitted each of them on the 70 cm (UHF) amateur radio band, and recorded our transmissions with two different architectures of SDR receivers. Three lightweight ML models were trained exclusively on spectrograms of limited non-transmitted signals with random characters as payloads. This training involved an online data augmentation pipeline to simulate various radio channel impairments. Our best model, EfficientNetB0, achieved an accuracy of 93.80% across the 17 operating modes and 85.47% across all 98 parameterized radio signals, evaluated on our real-world transmissions with Wikipedia articles as payloads. Furthermore, we analyzed the impact of varying signal durations & the number of FFT bins on classification, assessed the effectiveness of our simulated channel impairments, and tested our models across multiple simulated SNRs. KW - Automatic Modulation Classification KW - Amateur KW - Radio KW - Spectrum Monitoring KW - Cognitive Radio KW - Machine Learning Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889837 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Optimized Self-supervised Training with BEST-RQ for Speech Recognition T2 - ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) N2 - Self-supervised learning has been successfully used for various speech related tasks, including automatic speech recognition. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ) has achieved state-of-the-art results in speech recognition. In this work, we further optimize the BEST-RQ approach using Kullback-Leibler divergence as an additional regularizing loss and multicodebook extension per cluster derived from low-level feature clustering. Preliminary experiments on train-100 split of LibriSpeech result in a relative improvement of 11.2% on test-clean by using multiple codebooks, utilizing a combination of cross-entropy and Kullback-Leibler divergence further reduces the word error rate by 4.5%. The proposed optimizations on full LibriSpeech pre-training and fine-tuning result in relative word error rate improvements of up to 23.8% on test-clean and 30.6% on testother using 6 codebooks. Furthermore, the proposed setup leads to faster convergence in pre-training and fine-tuning and additionally stabilizes the pre-training. KW - self-supervised learning KW - speech recognition KW - conformer KW - BEST-RQ Y1 - 2025 U6 - https://doi.org/10.1109/ICASSP49660.2025.10889362 SP - 1 EP - 5 PB - IEEE ER - TY - CHAP A1 - Freisinger, Steffen A1 - Seeberger, Philipp A1 - Ranzenberger, Thomas A1 - Bocklet, Tobias A1 - Riedhammer, Korbinian T1 - Towards Multi-Level Transcript Segmentation: LoRA Fine-Tuning for Table-of-Contents Generation T2 - Interspeech 2025 N2 - Segmenting speech transcripts into thematic sections benefits both downstream processing and users who depend on written text for accessibility. We introduce a novel approach to hierarchical topic segmentation in transcripts, generating multi-level tables of contents that capture both topic and subtopic boundaries. We compare zero-shot prompting and LoRA fine-tuning on large language models, while also exploring the integration of high-level speech pause features. Evaluations on English meeting recordings and multilingual lecture transcripts (Portuguese, German) show significant improvements over established topic segmentation baselines. Additionally, we adapt a common evaluation measure for multi-level segmentation, taking into account all hierarchical levels within one metric. KW - topic segmentation KW - spoken content segmentation KW - table of contents generation KW - hierarchical segmentation Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2792 SN - 2958-1796 SP - 276 EP - 280 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Sun, Linghang A1 - Makridis, Michail A1 - Genser, Alexander A1 - Axenie, Cristian A1 - Grossi, Margherita A1 - Kouvelas, Anastasios T1 - Antifragile perimeter control BT - Anticipating and gaining from disruptions with reinforcement learning N2 - The optimal operation of transportation networks is often susceptible to unexpected disruptions, such as traffic incidents and social events. Many established control strategies rely on mathematical models that often struggle to cope with real-world uncertainties, leading to a significant decline in their effectiveness when faced with substantial disruptions. While previous research works have dedicated efforts to enhancing the robustness or resilience of transportation systems against disruptions, in this paper, we use the concept of antifragility to better design a traffic control strategy for urban road networks. Antifragility represents a system's ability to not only withstand stressors, shocks, and volatility but also thrive and enhance performance in the presence of such disruptions. Hence, modern transport systems call for solutions that are antifragile. In this work, we propose a model-free deep Reinforcement Learning (RL) algorithm to regulate perimeter control in a two-region urban traffic network to exploit and strengthen the learning capability of RL under disruptions and achieve antifragility. By incorporating antifragility terms based on the change rate and curvature of the traffic state into the RL framework, the proposed algorithm further gains knowledge of the traffic state, which helps in anticipating imminent disruptions. An additional term is also integrated into the RL algorithm as redundancy to enhance the performance under disruption scenarios. When compared to a state-of-the-art model predictive control approach and a state-of-the-art RL algorithm, our proposed method demonstrates two antifragility-related properties: (a) gradual performance improvement under disruptions of similar magnitude; and (b) increasingly superior performance under growing disruptions. Y1 - 2024 U6 - https://doi.org/10.48550/arXiv.2402.12665 ER - TY - JOUR A1 - Lee, Seanie A1 - Bok Lee, Dong A1 - Wagner, Dominik A1 - Kang, Minki A1 - Seong, Haebin A1 - Bocklet, Tobias A1 - Lee, Juho A1 - Hwang, Sung Ju T1 - SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models, JF - Findings of the Association for Computational Linguistics: ACL 2025 N2 - Deploying large language models (LLMs) in real-world applications requires robust safety guard models to detect and block harmful user prompts. While large safety guard models achieve strong performance, their computational cost is substantial. To mitigate this, smaller distilled models are used, but they often underperform on “hard” examples where the larger model provides accurate predictions. We observe that many inputs can be reliablyhandled by the smaller model, while only a small fraction require the larger model’s capacity. Motivated by this, we propose SafeRoute, a binary router that distinguishes hard examples from easy ones. Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy compared to solely using the larger safety guard model. Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance, outperforming relevant baselines. Y1 - 2025 UR - https://aclanthology.org/2025.findings-acl.105.pdf PB - Association for Computational Linguistics CY - Vienna, Austria ER - TY - CHAP A1 - Wagner, Dominik A1 - Baumann, Ilja A1 - Engert, Natalie A1 - Lee, Seanie A1 - Nöth, Elmar A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - In this work, we present our submission to the Speech Accessibility Project challenge for dysarthric speech recognition. We integrate parameter-efficient fine-tuning with latent audio representations to improve an encoder-decoder ASR system. Synthetic training data is generated by fine-tuning Parler-TTS to mimic dysarthric speech, using LLM-generated prompts for corpus-consistent target transcripts. Personalization with x-vectors consistently reduces word error rates (WERs) over non-personalized fine-tuning. AdaLoRA adapters outperform full fine-tuning and standard low-rank adaptation, achieving relative WER reductions of ∼23% and ∼22%, respectively. Further improvements (∼5% WER reduction) come from incorporating wav2vec 2.0-based audio representations. Training with synthetic dysarthric speech yields up to ∼7% relative WER improvement over personalized fine-tuning alone. KW - speech recognition KW - dysarthric speech KW - personalization KW - low-rank adaptation KW - synthetic data Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2155 SN - 2958-1796 SP - 3294 EP - 3298 PB - ISCA CY - ISCA ER - TY - CHAP A1 - Baumann, Ilja A1 - Wagner, Dominik A1 - Riedhammer, Korbinian A1 - Bocklet, Tobias T1 - Pathology-Aware Speech Encoding and Data Augmentation for Dysarthric Speech Recognition T2 - Interspeech 2025 N2 - Automatic speech recognition (ASR) for pathologic speech remains a major challenge due to high variability in articulation, phonation, and prosody distortions. In this work, we propose a pathology-aware speech encoder based on BEST-RQ pre-training, which incorporates 46k hours of speech, including pathologic and atypical speech. We continue pre-training for domain adaptation and experiment with etiology-specific codebooks. We achieve a 13.2% relative word error rate (WER) improvement using the pathology-aware speech encoder with etiology-specific continued pre-training. Additionally, we examine the impact of incorporating synthetic and out-of-domain (OOD) data to further enhance ASR performance. Synthetic data reduces WER by up to 8.7%, while OOD data improves WER by 12.2%. Finally, we introduce a semantic similaritybased data augmentation technique to optimize data selection, achieving a WER improvement of up to 9.7% while minimizing the need for additional training data. KW - pathologic speech recognition KW - speech augmentation KW - self-supervised learning Y1 - 2025 U6 - https://doi.org/10.21437/Interspeech.2025-2724 SN - 2958-1796 SP - 3289 EP - 3293 PB - ISCA CY - ISCA ER -