Fakultät Informatik und Mathematik
Refine
Year of publication
Document Type
- conference proceeding (article) (437)
- Article (396)
- conference proceeding (presentation, abstract) (92)
- Part of a Book (88)
- conference proceeding (volume) (23)
- Preprint (22)
- Book (21)
- Report (16)
- Working Paper (11)
- conference talk (8)
Is part of the Bibliography
- no (1137)
Keywords
- Bildgebendes Verfahren (22)
- Deep Learning (20)
- Produktionsplanung (18)
- Betriebliches Informationssystem (16)
- Künstliche Intelligenz (16)
- Artificial Intelligence (14)
- Diagnose (14)
- Gebärmutterhalskrebs (14)
- Informationstechnik (14)
- Maschinelles Lernen (14)
Institute
- Fakultät Informatik und Mathematik (1137)
- Labor für Technikfolgenabschätzung und Angewandte Ethik (LaTe) (184)
- Regensburg Medical Image Computing (ReMIC) (164)
- Institut für Sozialforschung und Technikfolgenabschätzung (IST) (145)
- Fakultät Angewandte Sozial- und Gesundheitswissenschaften (97)
- Labor Empirische Sozialforschung (93)
- Regensburg Strategic IT Management (ReSITM) (89)
- Labor für Digitalisierung (LFD) (84)
- Labor eHealth (eH) (66)
- Regensburg Center of Health Sciences and Technology - RCHST (57)
Begutachtungsstatus
- peer-reviewed (495)
- begutachtet (5)
Künstliche Intelligenz (KI) stellt eine Schlüsseltechnologie des gesellschaftlichen Wandels im 21. Jahrhundert dar. Mittlerweile werden zahlreiche technologische Anwendungen genutzt, die auf maschinellem Lernen und den damit verbundenen Möglichkeiten der Datensamm¬lung, -nutzung und -verwertung aufbauen. Indem KI große Datenmengen beherrschbar und verborgene Muster und Zusammenhänge sichtbar macht, wird vieles schneller, einfacher und effizienter – sei es im Alltag, in der Arbeit oder in Organisationen. Offen bleibt jedoch nach wie vor die Frage, welche tiefgreifenden und teilweise latenten Folgen für den Menschen als soziales Wesen und das gesellschaftliche Zusammenleben mit dem Einsatz und der Entwick¬lung von KI verbunden sind. Wie wandelt sich das Verhältnis von Mensch und Technik durch KI und wie ist dieser Wandel zu bewerten? Welche Chancen, aber auch Risiken eröffnen sich durch den Einsatz und die Entwicklung von KI für Mensch und Gesellschaft? Welchen Grenzen unterliegt der Wandel und welche Gestaltungsmöglichkeiten bieten sich? Und nicht zuletzt: Was und wer bestimmt die Entwicklungspfade, die KI nimmt – mit welchen Folgen und für wen?
Background
This study evaluated the effect of an artificial intelligence (AI)-based clinical decision support system on the performance and diagnostic confidence of endoscopists in their assessment of Barrett’s esophagus (BE).
Methods
96 standardized endoscopy videos were assessed by 22 endoscopists with varying degrees of BE experience from 12 centers. Assessment was randomized into two video sets: group A (review first without AI and second with AI) and group B (review first with AI and second without AI). Endoscopists were required to evaluate each video for the presence of Barrett’s esophagus-related neoplasia (BERN) and then decide on a spot for a targeted biopsy. After the second assessment, they were allowed to change their clinical decision and confidence level.
Results
AI had a stand-alone sensitivity, specificity, and accuracy of 92.2%, 68.9%, and 81.3%, respectively. Without AI, BE experts had an overall sensitivity, specificity, and accuracy of 83.3%, 58.1%, and 71.5%, respectively. With AI, BE nonexperts showed a significant improvement in sensitivity and specificity when videos were assessed a second time with AI (sensitivity 69.8% [95%CI 65.2%–74.2%] to 78.0% [95%CI 74.0%–82.0%]; specificity 67.3% [95%CI 62.5%–72.2%] to 72.7% [95%CI 68.2%–77.3%]). In addition, the diagnostic confidence of BE nonexperts improved significantly with AI.
Conclusion
BE nonexperts benefitted significantly from additional AI. BE experts and nonexperts remained significantly below the stand-alone performance of AI, suggesting that there may be other factors influencing endoscopists’ decisions to follow or discard AI advice.
Aims
Recent evidence suggests the possibility of intraprocedural phase recognition in surgical operations as well as endoscopic interventions such as peroral endoscopic myotomy and endoscopic submucosal dissection (ESD) by AI-algorithms. The intricate measurement of intraprocedural phase distribution may deepen the understanding of the procedure. Furthermore, real-time quality assessment as well as automation of reporting may become possible. Therefore, we aimed to develop an AI-algorithm for intraprocedural phase recognition during ESD.
Methods
A training dataset of 364385 single images from 9 full-length ESD videos was compiled. Each frame was classified into one procedural phase. Phases included scope manipulation, marking, injection, application of electrical current and bleeding. Allocation of each frame was only possible to one category. This training dataset was used to train a Video Swin transformer to recognize the phases. Temporal information was included via logarithmic frame sampling. Validation was performed using two separate ESD videos with 29801 single frames.
Results
The validation yielded sensitivities of 97.81%, 97.83%, 95.53%, 85.01% and 87.55% for scope manipulation, marking, injection, electric application and bleeding, respectively. Specificities of 77.78%, 90.91%, 95.91%, 93.65% and 84.76% were measured for the same parameters.
Conclusions
The developed algorithm was able to classify full-length ESD videos on a frame-by-frame basis into the predefined classes with high sensitivities and specificities. Future research will aim at the development of quality metrics based on single-operator phase distribution.
Aims
While AI has been successfully implemented in detecting and characterizing colonic polyps, its role in therapeutic endoscopy remains to be elucidated. Especially third space endoscopy procedures like ESD and peroral endoscopic myotomy (POEM) pose a technical challenge and the risk of operator-dependent complications like intraprocedural bleeding and perforation. Therefore, we aimed at developing an AI-algorithm for intraprocedural real time vessel detection during ESD and POEM.
Methods
A training dataset consisting of 5470 annotated still images from 59 full-length videos (47 ESD, 12 POEM) and 179681 unlabeled images was used to train a DeepLabV3+neural network with the ECMT semi-supervised learning method. Evaluation for vessel detection rate (VDR) and time (VDT) of 19 endoscopists with and without AI-support was performed using a testing dataset of 101 standardized video clips with 200 predefined blood vessels. Endoscopists were stratified into trainees and experts in third space endoscopy.
Results
The AI algorithm had a mean VDR of 93.5% and a median VDT of 0.32 seconds. AI support was associated with a statistically significant increase in VDR from 54.9% to 73.0% and from 59.0% to 74.1% for trainees and experts, respectively. VDT significantly decreased from 7.21 sec to 5.09 sec for trainees and from 6.10 sec to 5.38 sec for experts in the AI-support group. False positive (FP) readings occurred in 4.5% of frames. FP structures were detected significantly shorter than true positives (0.71 sec vs. 5.99 sec).
Conclusions
AI improved VDR and VDT of trainees and experts in third space endoscopy and may reduce performance variability during training. Further research is needed to evaluate the clinical impact of this new technology.
Aims
Artificial Intelligence (AI) systems in gastrointestinal endoscopy are narrow because they are trained to solve only one specific task. Unlike Narrow-AI, general AI systems may be able to solve multiple and unrelated tasks. We aimed to understand whether an AI system trained to detect, characterize, and segment early Barrett’s neoplasia (Barrett’s AI) is only capable of detecting this pathology or can also detect and segment other diseases like early squamous cell cancer (SCC).
Methods
120 white light (WL) and narrow-band endoscopic images (NBI) from 60 patients (1 WL and 1 NBI image per patient) were extracted from the endoscopic database of the University Hospital Augsburg. Images were annotated by three expert endoscopists with extensive experience in the diagnosis and endoscopic resection of early esophageal neoplasias. An AI system based on DeepLabV3+architecture dedicated to early Barrett’s neoplasia was tested on these images. The AI system was neither trained with SCC images nor had it seen the test images prior to evaluation. The overlap between the three expert annotations („expert-agreement“) was the ground truth for evaluating AI performance.
Results
Barrett’s AI detected early SCC with a mean intersection over reference (IoR) of 92% when at least 1 pixel of the AI prediction overlapped with the expert-agreement. When the threshold was increased to 5%, 10%, and 20% overlap with the expert-agreement, the IoR was 88%, 85% and 82%, respectively. The mean Intersection Over Union (IoU) – a metric according to segmentation quality between the AI prediction and the expert-agreement – was 0.45. The mean expert IoU as a measure of agreement between the three experts was 0.60.
Conclusions
In the context of this pilot study, the predictions of SCC by a Barrett’s dedicated AI showed some overlap to the expert-agreement. Therefore, features learned from Barrett’s cancer-related training might be helpful also for SCC prediction. Our results allow different possible explanations. On the one hand, some Barrett’s cancer features generalize toward the related task of assessing early SCC. On the other hand, the Barrett’s AI is less specific to Barrett’s cancer than a general predictor of pathological tissue. However, we expect to enhance the detection quality significantly by extending the training to SCC-specific data. The insight of this study opens the way towards a transfer learning approach for more efficient training of AI to solve tasks in other domains.
Aims
Endoscopic retrograde cholangiopancreaticography (ERCP) is the gold standard in the diagnosis as well as treatment of diseases of the pancreatobiliary tract. However, it is technically complex and has a relatively high complication rate. In particular, cannulation of the papillary ostium remains challenging. The aim of this study is to examine whether a deep-learning algorithm can be used to detect the major duodenal papilla and in particular the papillary ostium reliably and could therefore be a valuable tool for inexperienced endoscopists, particularly in training situation.
Methods
We analyzed a total of 654 retrospectively collected images of 85 patients. Both the major duodenal papilla and the ostium were then segmented. Afterwards, a neural network was trained using a deep-learning algorithm. A 5-fold cross-validation was performed. Subsequently, we ran the algorithm on 5 prospectively collected videos of ERCPs.
Results
5-fold cross-validation on the 654 labeled data resulted in an F1 value of 0.8007, a sensitivity of 0.8409 and a specificity of 0.9757 for the class papilla, and an F1 value of 0.5724, a sensitivity of 0.5456 and a specificity of 0.9966 for the class ostium. Regardless of the class, the average F1 value (class papilla and class ostium) was 0.6866, the sensitivity 0.6933 and the specificity 0.9861. In 100% of cases the AI-detected localization of the papillary ostium in the prospectively collected videos corresponded to the localization of the cannulation performed by the endoscopist.
Conclusions
In the present study, the neural network was able to identify the major duodenal papilla with a high sensitivity and high specificity. In detecting the papillary ostium, the sensitivity was notably lower. However, when used on videos, the AI was able to identify the location of the subsequent cannulation with 100% accuracy. In the future, the neural network will be trained with more data. Thus, a suitable tool for ERCP could be established, especially in the training situation.
Case study research is one of the most widely used research methods in Information Systems (IS). In recent years, an increasing number of publications have used case studies with few sources of evidence, such as single interviews per case. While there is much methodological guidance on rigorously conducting multiple case studies, it remains unclear how researchers can achieve an acceptable level of rigour for this emerging type of multiple case study with few sources of evidence, i.e., multiple mini case studies. In this context, we synthesise methodological guidance for multiple case study research from a cross-disciplinary perspective to develop an analytical framework. Furthermore, we calibrate this analytical framework to multiple mini case studies by reviewing previous IS publications that use multiple mini case studies to provide guidelines to conduct multiple mini case studies rigorously. We also offer a conceptual definition of multiple mini case studies, distinguish them from other research approaches, and position multiple mini case studies as a pragmatic and rigorous approach to research emerging and innovative phenomena in IS.
It remains difficult to segregate pelagic habitats since structuring processes are dynamic on a wide range of scales and clear boundaries in the open ocean are non-existent. However, to improve our knowledge about existing ecological niches and the processes shaping the enormous diversity of marine plankton, we need a better understanding of the driving forces behind plankton patchiness. Here we describe a new machine-learning method to detect and quantify pelagic habitats based on hydrographic measurements. An Autoencoder learns two-dimensional, meaningful representations of higher-dimensional micro-habitats, which are characterized by a variety of biotic and abiotic measurements from a high-speed ROTV. Subsequently, we apply a density-based clustering algorithm to group similar micro-habitats into associated pelagic macro-habitats in the German Bight of the North Sea. Three distinct macro-habitats, a “surface mixed layer,” a “bottom layer,” and an exceptionally “productive layer” are consistently identified, each with its distinct plankton community. We provide evidence that the model detects relevant features like the doming of the thermocline within an Offshore Wind Farm or the presence of a tidal mixing front.
Aims
Human-computer interactions (HCI) may have a relevant impact on the performance of Artificial Intelligence (AI). Studies show that although endoscopists assessing Barrett’s esophagus (BE) with AI improve their performance significantly, they do not achieve the level of the stand-alone performance of AI. One aspect of HCI is the impact of AI on the degree of certainty and confidence displayed by the endoscopist. Indirectly, diagnostic confidence when using AI may be linked to trust and acceptance of AI. In a BE video study, we aimed to understand the impact of AI on the diagnostic confidence of endoscopists and the possible correlation with diagnostic performance.
Methods
22 endoscopists from 12 centers with varying levels of BE experience reviewed ninety-six standardized endoscopy videos. Endoscopists were categorized into experts and non-experts and randomly assigned to assess the videos with and without AI. Participants were randomized in two arms: Arm A assessed videos first without AI and then with AI, while Arm B assessed videos in the opposite order. Evaluators were tasked with identifying BE-related neoplasia and rating their confidence with and without AI on a scale from 0 to 9.
Results
The utilization of AI in Arm A (without AI first, with AI second) significantly elevated confidence levels for experts and non-experts (7.1 to 8.0 and 6.1 to 6.6, respectively). Only non-experts benefitted from AI with a significant increase in accuracy (68.6% to 75.5%). Interestingly, while the confidence levels of experts without AI were higher than those of non-experts with AI, there was no significant difference in accuracy between these two groups (71.3% vs. 75.5%). In Arm B (with AI first, without AI second), experts and non-experts experienced a significant reduction in confidence (7.6 to 7.1 and 6.4 to 6.2, respectively), while maintaining consistent accuracy levels (71.8% to 71.8% and 67.5% to 67.1%, respectively).
Conclusions
AI significantly enhanced confidence levels for both expert and non-expert endoscopists. Endoscopists felt significantly more uncertain in their assessments without AI. Furthermore, experts with or without AI consistently displayed higher confidence levels than non-experts with AI, irrespective of comparable outcomes. These findings underscore the possible role of AI in improving diagnostic confidence during endoscopic assessment.
Effect of AI on performance of endoscopists to detect Barrett neoplasia: A Randomized Tandem Trial
()
Background and study aims
To evaluate the effect of an AI-based clinical decision support system (AI) on the performance and diagnostic confidence of endoscopists during the assessment of Barrett's esophagus (BE).
Patients and Methods
Ninety-six standardized endoscopy videos were assessed by 22 endoscopists from 12 different centers with varying degrees of BE experience.
The assessment was randomized into two video sets: Group A (review first without AI and second with AI) and group B (review first with AI and second without AI). Endoscopists were required to evaluate each video for the presence of Barrett's esophagus-related neoplasia (BERN) and then decide on a spot for a targeted biopsy. After the second assessment, they were allowed to change their clinical decision and confidence level.
Results
AI had a standalone sensitivity, specificity, and accuracy of 92.2%, 68.9%, and 81.6%, respectively. Without AI, BE experts had an overall sensitivity, specificity, and accuracy of 83.3%, 58.1 and 71.5%, respectively. With AI, BE nonexperts showed a significant improvement in sensitivity and specificity when videos were assessed a second time with AI (sensitivity 69.7% (95% CI, 65.2% - 74.2%) to 78.0% (95% CI, 74.0% - 82.0%); specificity 67.3% (95% CI, 62.5% - 72.2%) to 72.7% (95 CI, 68.2% - 77.3%). In addition, the diagnostic confidence of BE nonexperts improved significantly with AI.
Conclusion
BE nonexperts benefitted significantly from the additional AI. BE experts and nonexperts remained below the standalone performance of AI, suggesting that there may be other factors influencing endoscopists to follow or discard AI advice.
Control and Automation of services of the urban infrastructure offered to citizens and tourists are elementary parts of a smart city. But both rely on a stable supply of data from sensors spread across the whole city, e. g., the fill level sensors of waste bins needed for a waste management tool which we developed in a collaboration with the Regensburg city council for the on-demand collection of waste bins. Europe has a lot of historic cities like Regensburg with narrow streets and huge building walls, some made from granite and fieldstones, which often represents an insurmountable obstacle to wireless data transmission. The reduction of the road traffic volume poses an additional challenge for city planners. By means of networked planning and simulation software, the situation, state and efficiency of citywide logistic services can be monitored and optimized. In the course of such optimizations, we propose the combination of digital and logistic services. As an example, we show that monitoring state information, such as the waste bin fill levels, can be accomplished using the same vehicles and the same planning software, that is used for luggage transportation. Moreover, we describe how we adapted a solver for a variant of the TSP, namely the prize-collecting traveling salesman, to optimize the route planning dynamically.
Small and medium-sized enterprises (SMEs) increasingly need to manage nformation technology (IT) effectively in order to remain competitive. However, compared to larger organizations, SMEs often face challenges in terms of resources and employer attractiveness, and regularly do not have the need to employ a Chief Information Officer (CIO) on a full-time basis. To address this issue, a growing number of global experts have begun to provide CIO services on a part-time basis for multiple clients simultaneously. This approach allows SMEs to tap into the expertise of experienced IT leaders at a fraction of the cost and without committing to long-term arrangements. While these professionals, known as “Fractional CIOs”, have proven their value in the field, there has been a lack of academic research on this emerging trend. Therefore, we carried out a comprehensive research project between 2020 and 2023, involving 62 Fractional CIOs from 10 countries. The research produced a definition, different types of engagements, and success factors for Fractional CIOs and their engagements. This paper summarizes these findings for a wider audience of academics and practitioners.
Der zweite ITG-Workshop „Sprachassistenten – Anwendungen, Implikationen, Entwicklungen“ fand am 5. März 2024 in Regensburg statt. Er bot eine organisatorische und inhaltliche Fortführung des ersten Workshops vor vier Jahren in Magdeburg 2020. Auch in diesem Jahr war er wieder der Konferenz Elektronische Sprachsignalverarbeitung angegliedert. Auf dem Workshop wurden vielfältige und interdisziplinäre Beiträge in eingeladenen Vorträgen und als eingereichte Poster präsentiert. Durch die gute Mischung von Beitragenden sowohl aus der Hochschullandschaft als auch aus der Industrie wurden die verschiedensten Aspekte anwendungsnah diskutiert.
One of the tasks PAULE[1, 2] solves is finding suitable control parameter (cp-)trajectories for a given target acoustic. These cp-trajectories can be used to synthesize speech with the articulatory speech synthesizer of the VocalTractLab (VTL) [3]. If the target acoustic contains substantial microphone noise or other background noises, occasionally PAULE optimizes not for the speech in the target, but for this background noises. By adding a speech/non-speech classifier to the feedback and planning-loop in PAULE this resynthesis of background noises should be mitigated. Unfortunately, the improvements were minor, which might be due to uninformative gradients of the classifier. The importance of informative gradients and the use classifiers to adapt PAULE to different tasks are explained and discussed.
There is high confidence for the hypothesis that in speech perception the cycles of a θ-oscillation segment the auditory signal into syllables [8]. Yet the functionality of the oscillator generating the θ-oscillation is unknown. We follow the finding that, within an auditory scene, speech is perceived as a stream given by temporal coherence [12]. We work with the hypotheses that the θ-oscillator is driven by temporal features providing this coherence. We propose a new temporal feature called O-distance, which detects the onset of a syllable - the starting point to of a θ-cycle–triggered by the temporal distance from to to the instance of the maximal rise of the loudness curve of the vowel. To extract to from the auditory signal, we use the statistical properties of this distance based on the C-center hypothesis [25], which predicts a close temporal relation of the onset consonants to the onset of a vowel. The statistics are derived from reference O-distance extracted from an articulatory database, where the minima and maxima of the loudness are related to maxima and minima of the lower incisor and tongue tip. To judge the quality of the O-distance extracted from the auditory signal, we regard the temporal deviation of the O-distance to the reference O-distance. Currently we achieve a mean deviation of 34ms.
This paper addresses the challenges and advancements in speech recognition for singing, a domain distinctly different from standard speech recognition. Singing encompasses unique challenges, including extensive pitch variations, diverse vocal styles, and background music interference. We explore key areas such as phoneme recognition, language identification in songs, keyword spotting, and full lyrics transcription. I will describe some of my own experiences when performing research on these tasks just as they were starting to gain traction, but will also show how recent developments in deep learning and large-scale datasets have propelled progress in this field. My goal is to illuminate the complexities of applying speech recognition to singing, evaluate current capabilities, and outline future research directions.
Real-time computational speed and a high degree of precision are requirements for computer-assisted interventions. Applying a segmentation network to a medical video processing task can introduce significant inter-frame prediction noise. Existing approaches can reduce inconsistencies by including temporal information but often impose requirements on the architecture or dataset. This paper proposes a method to include temporal information in any segmentation model and, thus, a technique to improve video segmentation performance without alterations during training or additional labeling. With Motion-Corrected Moving Average, we refine the exponential moving average between the current and previous predictions. Using optical flow to estimate the movement between consecutive frames, we can shift the prior term in the moving-average calculation to align with the geometry of the current frame. The optical flow calculation does not require the output of the model and can therefore be performed in parallel, leading to no significant runtime penalty for our approach. We evaluate our approach on two publicly available segmentation datasets and two proprietary endoscopic datasets and show improvements over a baseline approach.
Limitations in computer-assisted diagnosis include lack of labeled data and inability to model the relation between what experts see and what computers learn. Even though artificial intelligence and machine learning have demonstrated remarkable performances in medical image computing, their accountability and transparency level must be improved to transfer this success into clinical practice. The reliability of machine learning decisions must be explained and interpreted, especially for supporting the medical diagnosis. While deep learning techniques are broad so that unseen information might help learn patterns of interest, human insights to describe objects of interest help in decision-making. This paper proposes a novel approach, DeepCraftFuse, to address the challenge of combining information provided by deep networks with visual-based features to significantly enhance the correct identification of cancerous tissues in patients affected with Barrett’s esophagus (BE). We demonstrate that DeepCraftFuse outperforms state-of-the-art techniques on private and public datasets, reaching results of around 95% when distinguishing patients affected by BE that is either positive or negative to esophageal cancer.
Heavy smoke development represents an important challenge for operating physicians during laparoscopic procedures and can potentially affect the success of an intervention due to reduced visibility and orientation. Reliable and accurate recognition of smoke is therefore a prerequisite for the use of downstream systems such as automated smoke evacuation systems. Current approaches distinguish between non-smoked and smoked frames but often ignore the temporal context inherent in endoscopic video data. In this work, we therefore present a method that utilizes the pixel-wise displacement from randomly sampled images to the preceding frames determined using the optical flow algorithm by providing the transformed magnitude of the displacement as an additional input to the network. Further, we incorporate the temporal context at evaluation time by applying an exponential moving average on the estimated class probabilities of the model output to obtain more stable and robust results over time. We evaluate our method on two convolutional-based and one state-of-the-art transformer architecture and show improvements in the classification results over a baseline approach, regardless of the network used.
Training data for Neural Networks is often scarce in the medical domain, which often results in models that struggle to generalize and consequently showpoor performance on unseen datasets. Generally, adding augmentation methods to the training pipeline considerably enhances a model’s performance. Using the dataset of the Foot Ulcer Segmentation Challenge, we analyze two additional augmentation methods in the domain of chronic foot wounds - local warping of wound edges along with projection and blurring of shapes inside wounds. Our experiments show that improvements in the Dice similarity coefficient and Normalized Surface Distance metrics depend on a sensible selection of those augmentation methods.
Seit mehr als 25 Jahren ist der Workshop "Bildverarbeitung für die Medizin" als erfolgreiche Veranstaltung etabliert. Ziel ist auch 2024 wieder die Darstellung aktueller Forschungsergebnisse und die Vertiefung der Gespräche zwischen Wissenschaftlern, Industrie und Anwendern. Die Beiträge dieses Bandes - viele davon in englischer Sprache - umfassen alle Bereiche der medizinischen Bildverarbeitung, insbesondere die Bildgebung und -akquisition, Segmentierung und Analyse, Visualisierung und Animation, computerunterstützte Diagnose sowie bildgestützte Therapieplanung und Therapie. Hierbei kommen Methoden des maschinelles Lernens, der biomechanischen Modellierung sowie der Validierung und Qualitätssicherung zum Einsatz.
The use of chatbots based on large language models (LLMs) and their impact on society are influencing our learning experience platform Hochschul-Assistenz-System (HAnS). HAnS uses machine learning (ML) methods to support students and lecturers in the online learning and teaching processes [1]. This paper introduces LLM-based features available in HAnS which are using the transcript of our improved Automatic Speech Recognition (ASR) pipeline with an average transcription duration of 45 seconds and an average word error rate (WER) of 6.66% on over 8 hours of audio data of 7 lecture videos. A LLM-based chatbot could be used to answer questions on the lecture content as the ASR transcript is provided as context. The summarization and topic segmentation uses the LLM to improve our learning experience platform. We generate multiple choice questions using the LLM and the ASR transcript as context during playback in a period of 3 minutes and display them in the HAnS frontend.
In this study, we address the complex dynamics of emotional speech and
comprehensively examine the integration of rhythmic and vocal features to recognize emotional patterns. Our exploration is conducted using two German emotional corpora: VMEmo and EmoDB. Employing a combination of supervised methods (here linear discriminant analysis, LDA) and unsupervised techniques (here k-means clustering), we aim to uncover nuanced patterns within the emotional speech in these corpora. The application of LDA highlights salient patterns across different feature sets and focuses on the classification of speakers and prosodic characteristics. In addition, k-means clustering uncovers latent structures that reveal subtle mapping between emotions and speech behavior. Our results suggest that it is possible to cluster data based on prosodic behaviors that are influenced by emotional changes. Although precise mapping to the actual clusters derived from emotional labels could not be fully achieved, the results nonetheless reveal a moderate level of success in this investigation.
This study investigates the effects of speech segmentation methods on speaker recognition models, particularly with regard to the use of rhythmic feature sets. Using three automatic methods and one manual method on the German database of Kiel corpus, segmentation was performed based on the identification of vowel onsets. Subsequently rhythmic variability indices derived from these intervals were calculated and used for principal component analysis and support vector machine model in order to investigate the variation among speakers. The results underline the influence of signal segmentation methods on speaker recognition models.
PROM surveys, used to measure the effect of rehabilitation treatments, are typically filled out on paper, and often suffer from low response rates. Replacing it with a multimodal survey system, supporting touch and speech interaction, could lead to lower hurdles and therefore more data quantity. To do this, it requires task-specific training samples for the Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) to classify spoken answers into one of the standardized PROM answer options.
Due to the lack of training data for medical PROM surveys, we created augmented text samples with each answer option description, combined with different templates. To improve training capabilities, introduce a proper test set, and evaluate the ASR, we also collected 1,797 real voice samples within an empirical study. Further, we incorporate the contextual knowledge of the current question into our NLU architecture by implementing one classifier for every question scale.
Our results reveal that training with empirical data leads to better results than augmented data from templates and original answer option descriptions. Because of participant mislabeling of 33% due to the ambiguity of the task, we receive overall low NLU performances with up to 51.1% accuracy, and rank-1-accuracy up to 79.3%. We also find that our implementation of many scale-specific NLU classifiers significantly outperforms one NLU classifier for all labels, that incorporates the same contextual knowledge after the prediction, by 8 percent points.
Speech Recognition Errors in ASR Engines and Their Impact on Linguistic Analysis in Psychotherapies
(2024)
Modern intervention planning in psychotherapies may benefit from predicting process relevant psychotherapy constructs by automated speech analysis. One essential step is the extraction of relevant linguistic speech markers by ASR engines, which because of highly sensible data, work offline. We analyze transcription errors from NeMo, Whisper, and Wav2Vec2.0, focusing on their impact on linguistic markers usually requiring high quality transcripts. By utilizing part-of-speech tagging, we examine error occurrences among different word types. The Linguistic Inquiry and Word Count (LIWC) software aids in extracting markers. We highlight challenges in transcribing spontaneous speech, prevalent in therapy, and compare results with the Mozilla CommonVoice dataset, which features read speech.
This paper investigated whether predictability-based adjustments in production have listener-oriented consequences in perception. By manipulating the acoustic features of a target syllable in different predictability contexts in German, we tested 40 listeners’ perceptual preference for the manipulation. Four source words underwent acoustic modifications on the target syllable. Our results revealed a general preference for the original (unmodified) version over the modified one. However, listeners generally favored the unmodified version more when the source word had a higher predictable context compared to a less predictable one. The results showed that predictability-based adjustments have perceptual consequences and that listeners have predictability-based expectations in perception.
Recent neural text-to-speech (TTS) models are able to synthesize highly
natural speech signals using deep learning techniques. In practical applications, it can be desirable to have explicit control over the prosody (speech rate, fundamental frequency, and energy) of the synthesized speech. Such controllability can be achieved by adding prosody prediction modules, whose main purpose is to estimate plausible prosody features for each phoneme in the text input. This explicit modeling also allows for changing prosody features at inference time, consequently enabling the adjustment of the prosody in the synthesized audio. In this paper, we evaluate to which extent deliberate manipulation of such prosody features is reflected in the resulting speech audio. We focus particularly on changing the pitch (i.e., fundamental frequency) while applying different normalization strategies.
Generative models for audio are commonly used for music composition, sound effects generation for video game development, audio restoration, voice cloning, etc. The ease of generating indistinguishable fake audio with deep learning poses a major threat to personal privacy, online security, and political discourse. Evaluating the quality and realism of these synthetic utterances is crucial for mitigating the potential for misinformation and harm. To assess this threat, this paper conducts a systematic review, using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), on how these deepfake models are currently evaluated. The analysis of 86 papers shows that the majority of the evaluation is conducted on a machine level and highlights a research gap regarding the human perception of deepfakes. This paper explores various methods and perceptual measures employed in assessing audio deepfakes and evaluating their strengths, limitations, and future directions.
In this work we assess whether there is information in pauses in-between utterances of the same or different speakers that are predictive of the following speaker’s utterance. We present models that connect a person’s visual features before they speak to their upcoming utterance. In our experiments we find that outof-the-box pre-trained models can already reach a better-than-chance performance in correlating video embeddings to utterance embeddings. In contrast, models that attempt to predict the first word after the pause do not outperform a unigram model, indicating that our models do not read lips (based e.g. on co-articulation effects) but rather capture more fundamental aspects of the upcoming utterance.
Wine making is usually considered a domain being far off the processing of speech and language. But in a particular aspect, the domains of speech processing and wine making are related, namely, in the description of wine aromas. These descriptors are used for creating wine expertise as well as more general (advertisement-like) textual representations. In the current paper, we use Natural Language Processing techniques, especially Named Entity Recognition, to identify Aspects and Opinions, reflecting wine characteristics. These are combined with analyses of respective relations (triplet extraction) building Aspect-Opinion-Pairs to establish indicative aroma descriptors, also trying to approach the complex interplay amongst these individual statements. In our experiments, we rely on the Falstaff corpus comprising a huge set of wine descriptions. This results in an average F1 score of around 0.85 for Aspect-Opinion classification. For triplet generation multiple strategies were compared, resulting in an average F1 score of 0.67 in this challenging task. For both tasks we rely only on a handful of manually annotated samples, applying pseudo-labeling methods from seed data to achieve automatic labeling.
Despite the potential of AI, only a small percentage of small and medium-sized enterprises (SMEs) are adopting it due to data issues, expertise gaps, and implementation barriers. Zero-shot learning offers a promising approach for SMEs by minimizing these obstacles. This paper explores the use of zero-shot learning in a real-world NLP classification task on online comments (comparable with intent classification tasks) from the e-learning platform Sofatutor. While finetuning has achieved high accuracy (82.3–86.5%), zero-shot models have shown lower performance (39.3–61.4%) due to different label selection, grouping of different scenarios in one class and the type of classification task. Even if the current accuracy is not sufficient for practical application, pre-filtering the data using zeroshot learning might be a promising option for SMEs.
Wines are complex beverages whose taste can be described either numerically or textually, with the former involving the rating of the intensities of different aroma characteristics often with the help of a wine tasting wheel, and the latter with the help of crisp terms often in a poetic fashion. These are often done with the help of wine sommeliers who with one sniff can describe the wine. Usually, each sommelier has a unique style when it comes to textually describing a wine, research has shown that such differences have no negative impact in correctly classifying wines on the basis of their color, grape variety, region etc. Given the recent advancements in the field of Natural Language Processing, especially with the emergence of Large Language Models, we aim to check the capability of Llama 2 in its ability to generate texts pertaining to a specific color of a wine, given a list of aroma intensities as input prompts. In our experiments, we relied on data from Meininger and Falstaff, and on a combination of domain adaptation and pseudo-labeling techniques to create the corpus to train the Llama 2 model on. Also, we relied on a voting scheme of three differently trained classifiers to evaluate the wine-color specific text generation capabilities of Llama 2. Additionally, we employed the services of domain experts to evaluate the quality of a sample set of texts that was generated by Llama 2.
Our paper introduces a new technology for posture research and training:
the INteractive POsture COrrector, IN-POCO. The device warns its users about unfavorable postures when speaking (e.g., sitting in video conferences) and is thus suitable as an aid for rhetoric trainers. In addition, IN-POCO can also collect time-aligned posture and speech signals for researching prosody-posture relationships in the speech sciences. We outline the motivation for the development of IN-POCO and describe the key technical specifications and operational characteristics. The paper concludes
with a pilot experiment in which we provide initial evidence that, for a communicative (public) speaking task, posture does indeed affect speech prosody in gender-specific ways – in line with claims of rhetoric trainers and guidebooks, and such that an unfavorable (e.g,, humped) posture can be assumed to reduce the speaker's vocal charisma.
Das Continuous-Response-Measurement-Verfahren bildet durch die kontinuierliche Bewertungsmöglichkeit eine wichtige Ergänzung zu den gängigen Methoden im Repertoire der Wirkungsforschung. Um diesen Mehrwert voll ausschöpfen zu können, wird als Verfahrensoptimierung die Entwicklung einer Softwarelösung vorgestellt. Die Überprüfung des optimierten CRM-Verfahrens erfolgt mittels eines Anwendungsfalls aus der sprechwissenschaftlichen Telekommunikationsforschung im Rahmen eines User-Acceptance-Tests. Dabei wird die Funktionalität und Bedienerfreundlichkeit der entwickelten CRM-Softwarelösung unter Beachtung der für die Sprechwirkungsforschung relevanten Kriterien in Form einer A-BStudie getestet.
Das Gesamtergebnis des User-Acceptance-Tests fällt für die Software Evalue positiv aus. Mit Hilfe der Verfahrensoptimierung des CRM-Verfahrens ist eine variabel einsetzbare und damit vielfältig nutzbare CRM-Softwarelösung entstanden.
The growing prevalence of voice assistants has sparked privacy concerns with respect to content privacy and potential human-based attacks such as eavesdropping which make users feel uncomfortable utilizing them in public. To address these challenges, understanding human privacy perceptions in acoustic environments becomes paramount. This understanding can empower voice assistants to accurately quantify privacy perceptions, adapt conversational patterns, and ultimately enhance human-machine interaction. This study draws inspiration from human-tohuman interactions and previous research on acoustic privacy, to quantify privacy perceptions in environments characterized by babble noise. The primary objective is a comprehensive evaluation of both objective and subjective measures to quantitatively capture privacy perceptions in acoustic environments.
Speaker recognition systems often use mel-scaled cepstral coefficients (MFCCs) as main features. In contrast to MFCCs, Godoy et al. (2015) proposed a different type of short-term spectral analysis that provides features related to the lower vocal tract (LVT). They are calculated as the ratio of the acoustic shorttime spectra during the closed and open phases of the glottal oscillation cycles based on a pitch-synchronous analysis. These features were suggested to be particularly speaker-specific and might therefore be suitable to substitute or complement MFCCs in speaker recognition systems. The present study investigated the benefit of these features in an i-vector-based speaker recognition system. Using the LVT features alone, the system achieved a speaker recognition rate of 92.3% with 63 enrolled speakers. When the LVT features were fused with conventional MFCC features, the recognition rate was about equal to the recognition rate using MFCC features alone (> 98%).
Octra Backend ist eine portable web-basierte Infrastruktur für Transkriptionsprojekte, die lokal im Feld oder geschützten Bereichen, im begrenzten Intranet oder weltweit erreichbar im Internet eingesetzt werden kann. Entwicklungsziele waren die Gewährleistung möglichst hoher Sicherheitsanforderungen, eine gute Skalierbarkeit sowie eine einfache Installation auch ohne Administratorrechte. Octra Backend ist in Node.js implementiert und für MacOS, Windows und Linux verfügbar.
Wir repräsentieren eine Bedeutung als Liste von Mustersignalen, und unser Ziel ist es, ein weiteres ankommendes Signal damit zu vergleichen. Die Quantenlogik motiviert die Verwendung von Orthogonalprojektoren, um die gesuchte Ähnlichkeit als Projektionswahrscheinlichkeit darzustellen. Die Ergebnisse des quantenlogischen Verfahrens hängen davon ab, in welcher Weise die Signale vorverarbeitet werden. In diesem Aufsatz untersuchen und diskutieren wir vier verschiedene Möglichkeiten der Vorverarbeitung.
NoiSLU: a Noisy speech corpus for Spoken Language Understanding in the Public Transport Domain
(2024)
The use of local public transport requires the barrier-free purchase of a ticket. Travellers who are not proficient in the local language benefit from a multilingual human(ticket)machine voice interaction. This paper presents a nearly parallel audio dataset with 13218 annotated user queries from 20 speakers for English, German and Dutch. The domain-specific speech corpus can be understood as an evaluation dataset for future research in Spoken Language Understanding (SLU) and thus, it enables researches to improve the quality of human-machine interaction applications. Furthermore, we compare the SLU performance of different compositions of Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) models in baseline experiments on different test datasets.
Um bei Minimalistischen Grammatiken (MG) Übergenerierung zu vermeiden, kann man Einträge mit leeren Exponenten (ε-Einträge) verwenden. Ein Eintrag besteht aus einem Exponenten, der die Äußerung oder Schrift eines Wortes repräsentiert, einer Merkmalsliste, welche die Syntax kodiert und einem λ-Ausdruck, der die Semantik repräsentiert. Leere Einträge führen allerdings zu einer schlechteren Verwendbarkeit der Grammatik für das Parsen. Die vorliegende Arbeit wird ein Umformungsalgorithmus für MGs vorstellen, sodass die Anzahl der ε-Einträge verringert werden kann, um sie wieder für Parser verwendbar zu machen. Hierzu werden die ε-Einträge mit den anderen Einträgen vorverarbeitet und dadurch neue Einträge geschaffen. Die nun überflüssigen ε-Einträge können dann problemlos entfernt werden. Der Algorithmus wurde anhand von über 260 Zahlwortgrammatiken getestet.
Die vorliegende Studie untersucht, inwiefern sich die in der Phonetik verbreiteten Skripte zur automatisierten Feststellung von verschiedenen Aspekten des Sprechflusses von de Jong et al. zur Beurteilung des Sprachstands bei Kindern eignen und in welcher Art und Weise die Methodik angepasst werden könnte. Dazu wurden Sprachdaten von Vorschulkindern mit Deutsch als Muttersprache bzw. als Zweitsprache mithilfe eines Serious Game zur Sprachstandserhebung elizitiert. Die Audiodaten wurden bezüglich Artikulationsrate, Pausen und Füllpartikeln sowohl automatisiert durch die Skripte als auch manuell annotiert. Die Ergebnisse zeigen, dass sich die Skripte zur Ermittlung der Artikulationsrate mit einer relativ hohen Übereinstimmung mit der manuellen Ermittlung zur Verwendung in Sprachstandserhebungsverfahren eignen. Auch die automatische Erkennung von Sprechpausen weist einen hohen Precision-Wert auf und könnte als Instrument in Sprachstandserhebungen verwendet werden. Eine solche Verwendung würde mit
einer Erweiterung um die hier vorgestellte manuelle Methode zum Annotieren von disfluenten und nicht disfluenten Pausen profitieren. Bei den Füllpartikeln stellte sich die automatisierte Klassifizierung hingegen als weniger geeignet heraus. Hier wurde keine hohe Übereinstimmung mit der menschlichen Annotation festgestellt. Um in der Praxis Verwendung finden zu können, muss diese Methode noch erweitert werden, zum Beispiel durch Miteinbeziehen der Pausenerkennung.
Synchrony of Glottal Area Waveform Parameters During the Production of Obstruents in Vowel Context
(2024)
Obstruents are phonemes which require partial or total obstruction of airflow through the vocal tract. Their articulation also requires adjustments of the laryngeal settings, e. g., an abduction gesture to stop vocal fold vibration for voiceless obstruents. This study investigated the laryngeal settings during the production of voiced and voiceless obstruents in vowel context to analyze the degree of synchrony of the involved glottal gestures. High-speed laryngoscopy images were used to determine the glottal area waveform, from which the time functions of the parameters open quotient (OQ), fundamental frequency (f0), and AC and DC amplitude (ACA and DCA) were calculated and analyzed. Significant correlations were found between all pairs of parameters, with strong correlations between some of them, e.g. Open Quotient and AC Amplitude. Correlations were also either consistently positive or negative for specific pairs of parameters across all investigated phonemes. These results could point to consistent patterns in laryngeal gestures that could enhance articulatory speech synthesis.
Concatenative text-to-speech (TTS) systems remain a widely used cheaper alternative to neural TTS systems. Yet concatenation of prerecorded units entails some drawbacks, such as spectral distortion, the perceptual consequences of which remain unclear. In an attempt to bridge this gap, our study focused on the effect of spectral distortion in vowel formants on perceived speech quality in
naturally-read manipulated German words as well as non-words. More specifically, we explored the distortion effect on a varying number of affected formants, at different magnitude and directionality in two corner vowels /a:/ and /i:/. The results indicate that single formant manipulations have a less pronounced effect on the listeners’ perception compared to multiple formant perturbations. The threshold at which the distortion became generally audible was estimated to lie between 0.4 and 1.0 bandwidth. The directionality of the distortion was not found to be significant.
The continuous advancement of digitization extends beyond educational institutions, giving rise to numerous innovations, particularly in the realm of study information [1]. One avenue for incorporating digital methodologies involves leveraging conversational agents (CAs) [2], serving as interactive interfaces bridging the gap between humans and computers. In the broader context, conversational agents are gaining prominence, offering several benefits to their users. The overarching goal is to comprehensively assist users through these intelligent systems. Consequently, exploring existing university chatbots becomes imperative to discern the areas where they excel. This research aims to scrutinize diverse chatbot systems, delving into their use cases and the challenges they encounter, employing a systematic review. Here it turns out that chatbots support universities the most in the fields of administration, e-learning and mental health. Furthermore, the study will investigate practical experiences on the potential applications and implementation of these systems in university settings, incorporating insights from an online survey and interviews, both made with experts. Here it comes to conclusion that preparation in relation to a chatbot implementation is the key factor to success. Otherwise, a failed system is nearly impossible to be saved, once users lost trust in the system. Therefore, carefully made preparations in the technical and organisational field are necessary to provide a helpful assistant.
Die Therapie-App aphaDIGITAL wird im Rahmen eines Forschungsprojekts entwickelt, um Menschen mit Aphasie im häuslichen Umfeld zu
unterstützen. Das Projekt kombiniert bewährte Therapiemethoden mit digitalen Technologien, einschließlich künstlicher Intelligenz und einem interaktiven Avatar namens Eva. Dieser Artikel betrachtet die Analyse und Entwicklung der Interaktionsmechanismen, die für eine digitale Assistenz von Aphasietherapie den größten Einfluss haben. Es wurden dazu reale Therapiesitzungen zwischen sprachgeschädigten Menschen und ihren Therapeuten konversationsanalytisch nach spezifischen Merkmalen untersucht. Darauf aufbauend wurde durch manuelle Animation ein prototypisches Mundbild modelliert und ein eigenes Artikulationssystem konzipiert, um ein entsprechend authentisches Modell der deutschen Artikulations- und Koartikulationsmuster abzubilden.
This paper describes the usability evaluation of the parts of the CHATU chatbot. The evaluation was conducted with 21 participants. A focus of this paper is the description of the carefully designed evaluation procedure, which aims to avoid textual priming of the participants. The general evaluation procedure can be applied to other speech- or text-based conversational systems, and additional material is provided. The evaluation results show that the usability and user experience of CHATU are positively rated. However, the naturalness and novelty of the interaction are not optimal, and the potential influence of users’ experience with LLMs on the evaluation is discussed.
This paper describes a field study conducted with a museum chatbot at the Städel Museum Frankfurt. The chatbot uses the BERT language model for natural language processing and can be operated via touchscreen as well as via speech input. Prior to the study, hypotheses regarding the user experience of the system were formulated and a system-specific questionnaire was designed, which was used to inquire (among other things) about the perceived quality of the speech output and the frequency of audio guide use in museums. During the interaction with the chatbot, log data was collected and stored in the back-end system. The results show a significant correlation between perceived speech quality and user experience. An exploratory data analysis revealed that participants who used only speech input rated the system as significantly more stimulating than participants who used only touch input. Touch input turned out to be the most efficient input modality in terms of answer correctness and was rated highest regarding pragmatic quality. Interestingly touch input was preferred by younger participants. We discuss our findings and conclude that speech interaction should be seriously considered to create engaging conversational user experiences in museums.
Utility of Smartphone-based Three-dimensional Surface Imaging for Digital Facial Anthropometry
(2024)
Background
The utilization of three-dimensional (3D) surface imaging for facial anthropometry is a significant asset for patients undergoing maxillofacial surgery. Notably, there have been recent advancements in smartphone technology that enable 3D surface imaging.
In this study, anthropometric assessments of the face were performed using a smartphone and a sophisticated 3D surface imaging system.
Methods
30 healthy volunteers (15 females and 15 males) were included in the study. An iPhone 14 Pro (Apple Inc., USA) using the application 3D Scanner App (Laan Consulting Corp., USA) and the Vectra M5 (Canfield Scientific, USA) were employed to create 3D surface models. For each participant, 19 anthropometric measurements were conducted on the 3D surface models. Subsequently, the anthropometric measurements generated by the two approaches were compared. The statistical techniques employed included the paired t-test, paired Wilcoxon signed-rank test, Bland–Altman analysis, and calculation of the intraclass correlation coefficient (ICC).
Results
All measurements showed excellent agreement between smartphone-based and Vectra M5-based measurements (ICC between 0.85 and 0.97). Statistical analysis revealed no statistically significant differences in the central tendencies for 17 of the 19 linear measurements. Despite the excellent agreement found, Bland–Altman analysis revealed that the 95% limits of agreement between the two methods exceeded ±3 mm for the majority of measurements.
Conclusion
Digital facial anthropometry using smartphones can serve as a valuable supplementary tool for surgeons, enhancing their communication with patients. However, the proposed data suggest that digital facial anthropometry using smartphones may not yet be suitable for certain diagnostic purposes that require high accuracy.