Refine
Document Type
Is part of the Bibliography
- no (52)
Keywords
- Artificial Intelligence (6)
- Deep Learning (5)
- Endoscopy (5)
- Künstliche Intelligenz (3)
- Bildverarbeitung (2)
- Cholecystectomy (2)
- Medical Image Computing (2)
- Adenocarcinoma (1)
- Barrett’s cancer (1)
- Barrett’s esophagus detection (1)
Institute
Begutachtungsstatus
- peer-reviewed (42)
Ziele:
Das Ziel der Studie war es, den Einfluss von KI auf die diagnostische Sicherheit (Konfidenzniveau) von Endoskopikern anhand von BÖ-Videos zu untersuchen und mögliche Korrelationen mit der Untersuchungsqualität zu erforschen.
Methodik:
22 Endoskopiker aus zwölf Zentren mit unterschiedlicher Barrett-Erfahrung untersuchten 96 standardisierte Endoskopievideos. Die Untersucher wurden in Experten und Nicht-Experten eingeteilt und nach dem Zufallsprinzip für die Bewertung der Videos mit oder ohne KI eingeteilt. Die Teilnehmer wurden in zwei Gruppen aufgeteilt: Arm A bewertete zunächst Videos ohne KI und dann mit KI, während Arm B die umgekehrte Reihenfolge einhielt. Die Untersucher hatten die Aufgabe, BÖ-assoziierte Neoplasien zu erkennen und ihr Konfidenzniveau sowohl mit als auch ohne KI auf einer Skala von 0 bis 9 anzugeben.
Ergebnis:
In Arm A erhöhte der Einsatz von KI das Konfidenzniveau bei beiden signifikant (p<0.001). Bemerkenswert ist, dass jedoch nur Nicht-Experten durch die KI eine signifikante Verbesserung der Sensitivität und Spezifität (p<0.001 bzw. p<0.05) erfuhren. Während Experten ohne KI im Vergleich zu Nicht-Experten mit KI ein höheres Konfidenzniveau aufwiesen, gab es keinen signifikanten Unterschied in der Genauigkeit. In Arm B zeigten beide Gruppen eine signifikante Abnahme des Konfidenzniveaus (p<0.001) bei gleichbleibender Genauigkeit. Darüber hinaus wurden in 9% der Entscheidungen trotz korrekter KI eine falsche Wahl getroffen.
Schlussfolgerung:
Der Einsatz künstlicher Intelligenz steigerte das Konfidenzniveau sowohl bei Experten als auch bei Nicht-Experten signifikant – ein Effekt, der im Studienmodell reversibel war. Darüber hinaus wiesen Experten mit oder ohne KI durchweg höhere Konfidenzniveaus auf als Nicht-Experten mit KI, trotz vergleichbarer Ergebnisse. Zudem konnte beobachtet werden, dass die Untersucher in 9% der Fälle die KI zuungunsten des Patienten ignorierten.
Diese vorläufige monozentrische Studie zeigt, dass ein aus einem Barrett-Ösophagus-KI-System feinjustiertes Deep-Learning-Modell Magenkarzinome in westlichen multimodalen endoskopischen Bilddaten zuverlässig detektieren und präzise segmentieren kann. Die hohe Segmentierungsgenauigkeit und Detektionssensitivität über verschiedene Bildmodalitäten hinweg unterstreichen die Machbarkeit eines pathologiegestützten KI-Ansatzes auch in einer westlichen Niedriginzidenzpopulation. Aufgrund der ausschließlichen Verwendung von Bildern mit sichtbaren Tumoren lassen sich keine Aussagen zur Spezifität treffen; eine Übertragbarkeit auf Screening- oder Mischkollektive ist daher limitiert. Weitere Studien mit a) größerem Datensatz inklusive Videodaten, b) externer Validierung an einer multizentrischen westlichen Kohorte, sowie c) Anwendung und Prüfung an nicht-neoplastischen Vergleichsbildern oder anderen Pathologien sind erforderlich.
Nach unserem Kenntnisstand zählt dieses System zu den ersten in einer westlichen Population entwickelten endoskopischen KI-Ansätzen zur Magenkarzinomdetektion, und zu wenigen, die vollständige ESD-präparatbasierte Referenzdaten für Training und Validierung nutzen.
Aims
Human-computer interactions (HCI) may have a relevant impact on the performance of Artificial Intelligence (AI). Studies show that although endoscopists assessing Barrett’s esophagus (BE) with AI improve their performance significantly, they do not achieve the level of the stand-alone performance of AI. One aspect of HCI is the impact of AI on the degree of certainty and confidence displayed by the endoscopist. Indirectly, diagnostic confidence when using AI may be linked to trust and acceptance of AI. In a BE video study, we aimed to understand the impact of AI on the diagnostic confidence of endoscopists and the possible correlation with diagnostic performance.
Methods
22 endoscopists from 12 centers with varying levels of BE experience reviewed ninety-six standardized endoscopy videos. Endoscopists were categorized into experts and non-experts and randomly assigned to assess the videos with and without AI. Participants were randomized in two arms: Arm A assessed videos first without AI and then with AI, while Arm B assessed videos in the opposite order. Evaluators were tasked with identifying BE-related neoplasia and rating their confidence with and without AI on a scale from 0 to 9.
Results
The utilization of AI in Arm A (without AI first, with AI second) significantly elevated confidence levels for experts and non-experts (7.1 to 8.0 and 6.1 to 6.6, respectively). Only non-experts benefitted from AI with a significant increase in accuracy (68.6% to 75.5%). Interestingly, while the confidence levels of experts without AI were higher than those of non-experts with AI, there was no significant difference in accuracy between these two groups (71.3% vs. 75.5%). In Arm B (with AI first, without AI second), experts and non-experts experienced a significant reduction in confidence (7.6 to 7.1 and 6.4 to 6.2, respectively), while maintaining consistent accuracy levels (71.8% to 71.8% and 67.5% to 67.1%, respectively).
Conclusions
AI significantly enhanced confidence levels for both expert and non-expert endoscopists. Endoscopists felt significantly more uncertain in their assessments without AI. Furthermore, experts with or without AI consistently displayed higher confidence levels than non-experts with AI, irrespective of comparable outcomes. These findings underscore the possible role of AI in improving diagnostic confidence during endoscopic assessment.
Aims
Systematic visualization of anatomical landmarks is critical to prevent "blind spots" during esophagogastroduodenoscopy (EGD), which are a primary cause of missed upper gastrointestinal cancers. Standardized documentation serves as a secondary but essential tool for quality assurance. We aimed to develop a deep learning approach for the automated classification of 28 anatomical categories, including a 22-segment mapping of the stomach based on the Japanese Systemic Stomach Screening (SSS) protocol. The primary goal was to provide a technical foundation for real-time monitoring of mucosal coverage to ensure examination completeness. Primary outcomes were macro F1-score and overall accuracy for this 28-class classification task.
Methods
We used a multi-center dataset of 9.012 endoscopic images from 1.553 patients, merging the public GastroHUN database (5.202 images; 385 patients) [1] and an internal dataset (3.810 images; 1.168 patients) from a European tertiary referral centre (Augsburg, Germany). All images were obtained during routine EGD using Olympus endoscopes and anonymized prior to analysis. Each image was labelled at image level into one of 28 anatomical classes: oropharynx, esophagus, esophagogastric junction, duodenal bulb, second part of the duodenum, papilla and 22 gastric segments based on the SSS classification using a consensus-based multi-annotator process (n=4). A trained and supervised specialist annotated additional data from Augsburg. To handle this complexity, we employed a Vision Transformer (ViT-L/14) foundation model pre-trained via DINO-v2. Instead of full-parameter fine-tuning, which risks overfitting on medical datasets, we implemented Low-Rank Adaptation (LoRA) [2]. By targeting only the Q- and V-projection layers in the final six transformer blocks, we reduced the trainable parameters to 312.000 (0.36% of the total model). Evaluation was performed using a strict patient-wise 5-fold cross-validation.
Results
The model achieved a mean macro F1-score of 0.88±0.01 and an overall accuracy of 0.88±0.01. Beyond robustly identifying major regions like the esophagus and duodenum, the model showed high precision in the more challenging differentiation of fine-grained SSS segments. The use of LoRA enabled stable training and strong generalization across the combined international and local datasets ([Abb. 1]).
Conclusions
Integrating transformer-based foundation models with LoRA-based fine-tuning allows for highly accurate UGIT anatomical mapping. An F1-score of 0.88 across 28 classes suggests that this approach is robust enough for automated documentation and real-time quality monitoring. Future work will focus on expanding the multi-center data volume and implementing expert consensus-labeling across all subsets to fürther refine model reliability, alongside developing a real-time visualization tool that provides an intuitive overview of the explored segments. To our knowledge, this is among the first studies to leverage a public endoscopy dataset together with institutional data for upper GI anatomical site classification.
Die zuverlässige endoskopische Einschätzung der Infiltrationstiefe bei Neoplasien des Barrett-Ösophagus (BO) ist entscheidend für die Therapieentscheidung zwischen organerhaltender endoskopischer Resektion und chirurgischer Resektion. Ziel dieser Studie ist die Entwicklung eines KIVision- Modells, das endoskopisch kurativ resezierbare Läsionen (LGD; HGD; T1a m1 –T1b sm1, L0, V0 und G1-2; EREAC) von nicht kurativ endoskopisch resezierbaren Läsionen (≥ T1b sm2 und/oder L1, V1, G3; NEREAC) anhand endoskopischer Bildgebung unterscheidet. In einer Pilotstudie mit 116 Patient:innen wurde bei der Unterscheidung von T1a vs. T1b mit 230 Bildern ein F1-Score, eine Sensitivität und Spezifität von 74%, 77% und 64% erreicht. [1]
Material und Methodik Es handelt sich um eine ambidirektionale multizentrische Studie mit insgesamt n=578 Patient:innen und n=651 Läsionen im Zeitraum Januar 2014 bis Juni 2025. Eingeschlossen wurden Patient:innen mit gesicherter BO-assoziierter Neoplasie anhand des histologischen Goldstandards durch a) endoskopische Submukosadissektion (ESD) oder b) chirurgische Resektion. Insgesamt wurden 526 Läsionen im Arm EREAC und 125 Läsionen der Gruppe im Arm NEREAC akquiriert. Es werden Bild- und Videodaten aus Olympus und Fuji-Endoskopen verwendet. Aus der Videodokumentation wurden Frames extrahiert und anhand eines vordefinierten Schemas annotiert (u.a. Lasionssichtbarkeit, virtuelle (NBI, TXI) und farbbasierte Chromoendoskopie, Zoommodus). Es erfolgte eine mehrstufige Annotation durch einen medizinischen Doktoranden (J.B.), mit 1:1 Supervision durch einen endoskopisch tätigen Facharzt im Bereich Gastroenterologie (D.R.), sowie durch einen Barrett-Experten (A.E). Aufgrund einer Klassenimbalance zugunsten der EREAC-Gruppe erfolgt zunachst eine Zwischenanalyse aus 185 Patient:innen mit 215 Läsionen mit a 500 Frames. Mittels ConvNeXt Tiny Architektur wurde eine 5-fache Cross-Validierung mit Split auf Patientenebene durchgeführt. Primare Leistungsmetriken sind F1-Score, Sensitivität und Spezifität.
Ergebnisse Im aktuellen Test erreichte das KI-Modell für die Klassifikation (EREAC vs. NEREAC) einen mittleren F1-Score von 78.3% (resektabel: 79.8%, nicht 2 resektabel: 76.8%). Die mittlere Sensitivität und Spezifität lag bei je 78.3%. Die klassenspezifische Sensitivität und Spezifität zur Erkennung von NEREAC lag bei 72.0% und 84.6%.
Zusammenfassung Ein KI-Assistenzmodell kann die endoskopische Einschatzung der Resektabilität von Barrett-Neoplasien unterstutzen, indem es die Infiltrationstiefe aus endoskopischer Bildgebung pradiziert. Klinisch besonders relevant ist die hohe Spezifität in der Erkennung nicht-resektabler Läsionen.
Nächste Schritte sind das weitere Training mit höherer Datenmenge, die adressierte Aufarbeitung der Klassenimbalance, externe Validierung sowie die Echtzeit-Implementation.
Note: A script for extracting the individual frames from the video files while preserving the challenge-compliant directory structure and frame-to-mask naming conventions is available on GitHub and can be accessed here: https://github.com/remic-othr/PhaKIR_Dataset.
The dataset is described in the following publications:
Rueckert, Tobias et al.: Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge. arXiv preprint, https://arxiv.org/abs/2507.16559. 2025.
Rueckert, Tobias et al.: Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR). arXiv preprint, https://arxiv.org/abs/2511.06549. 2025.
The proposed dataset was used as the training dataset in the PhaKIR challenge (https://phakir.re-mic.de/) as part of EndoVis-2024 at MICCAI 2024 and consists of eight real-world videos of human cholecystectomies ranging from 23 to 60 minutes in duration. The procedures were performed by experienced physicians, and the videos were recorded in three hospitals. In addition to existing datasets, our annotations provide pixel-wise instance segmentation masks of surgical instruments for a total of 19 categories, coordinates of relevant instrument keypoints (instrument tip(s), shaft-tip transition, shaft), both at an interval of one frame per second, and specifications regarding the intervention phases for a total of eight different phase categories for each individual frame in one dataset and thus comprehensively cover instrument localization and the context of the operation. Furthermore, the provision of the complete video sequences offers the opportunity to include the temporal information regarding the respective tasks and thus further optimize the resulting methods and outcomes.
Aims
Artificial Intelligence (AI) systems in gastrointestinal endoscopy are narrow because they are trained to solve only one specific task. Unlike Narrow-AI, general AI systems may be able to solve multiple and unrelated tasks. We aimed to understand whether an AI system trained to detect, characterize, and segment early Barrett’s neoplasia (Barrett’s AI) is only capable of detecting this pathology or can also detect and segment other diseases like early squamous cell cancer (SCC).
Methods
120 white light (WL) and narrow-band endoscopic images (NBI) from 60 patients (1 WL and 1 NBI image per patient) were extracted from the endoscopic database of the University Hospital Augsburg. Images were annotated by three expert endoscopists with extensive experience in the diagnosis and endoscopic resection of early esophageal neoplasias. An AI system based on DeepLabV3+architecture dedicated to early Barrett’s neoplasia was tested on these images. The AI system was neither trained with SCC images nor had it seen the test images prior to evaluation. The overlap between the three expert annotations („expert-agreement“) was the ground truth for evaluating AI performance.
Results
Barrett’s AI detected early SCC with a mean intersection over reference (IoR) of 92% when at least 1 pixel of the AI prediction overlapped with the expert-agreement. When the threshold was increased to 5%, 10%, and 20% overlap with the expert-agreement, the IoR was 88%, 85% and 82%, respectively. The mean Intersection Over Union (IoU) – a metric according to segmentation quality between the AI prediction and the expert-agreement – was 0.45. The mean expert IoU as a measure of agreement between the three experts was 0.60.
Conclusions
In the context of this pilot study, the predictions of SCC by a Barrett’s dedicated AI showed some overlap to the expert-agreement. Therefore, features learned from Barrett’s cancer-related training might be helpful also for SCC prediction. Our results allow different possible explanations. On the one hand, some Barrett’s cancer features generalize toward the related task of assessing early SCC. On the other hand, the Barrett’s AI is less specific to Barrett’s cancer than a general predictor of pathological tissue. However, we expect to enhance the detection quality significantly by extending the training to SCC-specific data. The insight of this study opens the way towards a transfer learning approach for more efficient training of AI to solve tasks in other domains.
Aims
AI has proven great potential in assisting endoscopists in diagnostics, however its role in therapeutic endoscopy remains unclear. Endoscopic submucosal dissection (ESD) is a technically demanding intervention with a slow learning curve and relevant risks like bleeding and perforation. Therefore, we aimed to develop an algorithm for the real-time detection and delineation of relevant structures during third-space endoscopy.
Methods
5470 still images from 59 full length videos (47 ESD, 12 POEM) were annotated. 179681 additional unlabeled images were added to the training dataset. Consequently, a DeepLabv3+ neural network architecture was trained with the ECMT semi-supervised algorithm (under review elsewhere). Evaluation of vessel detection was performed on a dataset of 101 standardized video clips from 15 separate third-space endoscopy videos with 200 predefined blood vessels.
Results
Internal validation yielded an overall mean Dice score of 85% (68% for blood vessels, 86% for submucosal layer, 88% for muscle layer). On the video test data, the overall vessel detection rate (VDR) was 94% (96% for ESD, 74% for POEM). The median overall vessel detection time (VDT) was 0.32 sec (0.3 sec for ESD, 0.62 sec for POEM).
Conclusions
Evaluation of the developed algorithm on a video test dataset showed high VDR and quick VDT, especially for ESD. Further research will focus on a possible clinical benefit of the AI application for VDR and VDT during third-space endoscopy.
This data article describes the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, a collection of eight complete laparoscopic cholecystectomy videos acquired during routine minimally invasive procedures at three German medical centers. The recordings were captured with different monocular endoscopic camera systems at 25 frames per second and a resolution of 1920 × 1080 pixels, with durations ranging from 28 to 58 minutes. Segments showing regions outside the abdominal cavity were removed for anonymization, and the corresponding cut indices are provided.
The dataset offers unified annotations for three interrelated tasks on the same complete surgical sequences: surgical phase recognition, annotated for every frame at 25 fps (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames), each annotated at one frame per second (every 25th frame). Surgical phases follow the seven-phase Cholec80 scheme, extended by an undefined label for transitional frames. The instrument annotations distinguish 19 instrument classes as well as individual instances of the same class, with keypoint annotations comprising two to four class-dependent points labeled with COCO-style visibility states. Segmentation and keypoint labels were created manually using the Computer Vision Annotation Tool (CVAT), whereas phase labels were derived from documented phase-transition timestamps, and all annotations underwent a multi-stage review by a medically trained team. The data are provided in open formats (MP4, CSV, JSON, PNG) together with a frame-extraction script under the CC BY-NC-SA license and are available through controlled access upon request via the Zenodo platform.
Because the dataset combines procedural context, instrument pose, and pixel-accurate instance segmentations within full-length recordings from multiple institutions, it can be reused for single-task or multi-task model development, for temporally aware approaches that exploit motion continuity, and for cross-institutional generalization protocols such as leave-one-hospital-out evaluation. The dataset served as the training resource for the PhaKIR Challenge at the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024.
We address domain generalization (DG) in mitotic-cell (MC) detection by combining a β-variational autoencoder (VAE) for domain transformations with feature-space alignment together with an object detector. The β-VAE synthesizes domain-transformed images, and the detector is trained to map originals and their transformed counterparts to equal representations. On the MIDOG++ dataset, this approach improves out-of-domain detection F1 scores by 7 and 3 percentage points compared to the color-variation augmentation and stain-normalization baselines. Results further suggest that morphology shifts hinder generalization more than stain shifts.
