Refine
Year of publication
Document Type
- Article (60)
- conference proceeding (article) (52)
- conference proceeding (presentation, abstract) (33)
- Preprint (4)
- Book (1)
- Part of a Book (1)
Is part of the Bibliography
- no (151)
Keywords
- Bildgebendes Verfahren (18)
- Deep Learning (16)
- Diagnose (13)
- Artificial Intelligence (12)
- Künstliche Intelligenz (12)
- Gehirn (11)
- Maschinelles Lernen (11)
- Kernspintomografie (9)
- Dreidimensionale Bildverarbeitung (8)
- Registrierung <Bildverarbeitung> (8)
Institute
- Fakultät Informatik und Mathematik (151) (remove)
Begutachtungsstatus
- peer-reviewed (77)
- begutachtet (1)
Einleitung Die Endoskopische Retrograde Cholangiopankreatikographie (ERCP) ist der Goldstandard in der Diagnostik und Therapie von Erkrankungen des pankreatobiliären Trakts. Jedoch ist sie technisch sehr anspruchsvoll und weist eine vergleichsweise hohe Komplikationsrate auf.
Ziele
In der vorliegenden Machbarkeitsstudie soll geprüft werden, ob mithilfe eines Deep-learning-Algorithmus die Papille und das Ostium zuverlässig detektiert werden können und somit für Endoskopiker mit geringer Erfahrung ein geeignetes Hilfsmittel, insbesondere für die Ausbildungssituation, darstellen könnten.
Methodik
Wir betrachteten insgesamt 606 Bilddatensätze von 65 Patienten. In diesen wurde sowohl die Papilla duodeni major als auch das Ostium segmentiert. Anschließend wurde eine neuronales Netz mittels eines Deep-learning-Algorithmus trainiert. Außerdem erfolgte eine 5-fache Kreuzvaldierung.
Ergebnisse
Bei einer 5-fachen Kreuzvaldierung auf den 606 gelabelten Daten konnte für die Klasse Papille eine F1-Wert von 0,7908, eine Sensitivität von 0,7943 und eine Spezifität von 0,9785 erreicht werden, für die Klasse Ostium eine F1-Wert von 0,5538, eine Sensitivität von 0,5094 und eine Spezifität von 0,9970 (vgl. [Tab. 1]). Unabhängig von der Klasse zeigte sich gemittelt (Klasse Papille und Klasse Ostium) ein F1-Wert von 0,6673, eine Sensitivität von 0,6519 und eine Spezifität von 0,9877 (vgl. [Tab. 2]).
Schlussfolgerung
In vorliegende Machbarkeitsstudie konnte das neuronale Netz die Papilla duodeni major mit einer hohen Sensitivität und sehr hohen Spezifität identifizieren. Bei der Detektion des Ostiums war die Sensitivität deutlich geringer. Zukünftig soll das das neuronale Netz mit mehr Daten trainiert werden. Außerdem ist geplant, den Algorithmus auch auf Videos anzuwenden. Somit könnte langfristig ein geeignetes Hilfsmittel für die ERCP etabliert werden.
Aims
Endoscopic retrograde cholangiopancreaticography (ERCP) is the gold standard in the diagnosis as well as treatment of diseases of the pancreatobiliary tract. However, it is technically complex and has a relatively high complication rate. In particular, cannulation of the papillary ostium remains challenging. The aim of this study is to examine whether a deep-learning algorithm can be used to detect the major duodenal papilla and in particular the papillary ostium reliably and could therefore be a valuable tool for inexperienced endoscopists, particularly in training situation.
Methods
We analyzed a total of 654 retrospectively collected images of 85 patients. Both the major duodenal papilla and the ostium were then segmented. Afterwards, a neural network was trained using a deep-learning algorithm. A 5-fold cross-validation was performed. Subsequently, we ran the algorithm on 5 prospectively collected videos of ERCPs.
Results
5-fold cross-validation on the 654 labeled data resulted in an F1 value of 0.8007, a sensitivity of 0.8409 and a specificity of 0.9757 for the class papilla, and an F1 value of 0.5724, a sensitivity of 0.5456 and a specificity of 0.9966 for the class ostium. Regardless of the class, the average F1 value (class papilla and class ostium) was 0.6866, the sensitivity 0.6933 and the specificity 0.9861. In 100% of cases the AI-detected localization of the papillary ostium in the prospectively collected videos corresponded to the localization of the cannulation performed by the endoscopist.
Conclusions
In the present study, the neural network was able to identify the major duodenal papilla with a high sensitivity and high specificity. In detecting the papillary ostium, the sensitivity was notably lower. However, when used on videos, the AI was able to identify the location of the subsequent cannulation with 100% accuracy. In the future, the neural network will be trained with more data. Thus, a suitable tool for ERCP could be established, especially in the training situation.
GraphMIC is a cross-platform image processing application utilizing the libraries ITK and OpenCV. The abstract structure of image processing pipelines is visually represented by user interface components based on modern QtQuick technology and allows users to focus on arrangement and parameterization of operations rather than implementing the equivalent functionality natively in C++. The application's central goal is to improve and simplify the typical workflow by providing various high level features and functions like multi threading, image sequence processing and advanced error handling. A built-in python interpreter allows the creation of custom nodes, where user defined algorithms can be integrated to extend basic functionality. An embedded 2d/3d visual-izer gives feedback of the resulting image of an operation or the whole pipeline. User inputs like seed points, contours or regions are forwarded to the processing pipeline as parameters to offer semi-automatic image computing. We report the main concept of the application and introduce several features and their implementation. Finally, the current state of development as well as future perspectives of GraphMIC are discussed
Backround
Scaphoidectomy and midcarpal fusion can be performed using traditional fixation methods like K-wires, staples, screws or different dorsal (non)locking arthrodesis systems. The aim of this study is to test the Aptus four corner locking plate and to compare the clinical findings to the data revealed by CT scans and semi-automated segmentation.
Methods:
This is a retrospective review of eleven patients suffering from scapholunate advanced collapse (SLAC) or scaphoid non-union advanced collapse (SNAC) wrist, who received a four corner fusion between August 2011 and July 2014. The clinical evaluation consisted of measuring the range of motion (ROM), strength and pain on a visual analogue scale (VAS). Additionally, the Disabilities of the Arm, Shoulder and Hand (QuickDASH) and the Mayo Wrist Score were assessed. A computerized tomography (CT) of the wrist was obtained six weeks postoperatively. After semi-automated segmentation of the CT scans, the models were post processed and surveyed.
Results
During the six-month follow-up mean range of motion (ROM) of the operated wrist was 60°, consisting of 30° extension and 30° flexion. While pain levels decreased significantly, 54% of grip strength and 89% of pinch strength were preserved compared to the contralateral healthy wrist. Union could be detected in all CT scans of the wrist. While X-ray pictures obtained postoperatively revealed no pathology, two user related technical complications were found through the 3D analysis, which correlated to the clinical outcome.
Conclusion
Due to semi-automated segmentation and 3D analysis it has been proved that the plate design can keep up to the manufacturers’ promises. Over all, this case series confirmed that the plate can compete with the coexisting techniques concerning clinical outcome, union and complication rate.
The Impact of Semi-Automated Segmentation and 3D Analysis on Testing New Osteosynthesis Material
(2017)
A new protocol for testing osteosynthesis material postoperatively combining semi-automated segmentation and 3D analysis of surface meshes is proposed. By various steps of transformation and measuring, objective data can be collected. In this study the specifications of a locking plate used for mediocarpal arthrodesis of the wrist were examined. The results show, that union of the lunate, triquetrum, hamate and capitate was achieved and that the plate is comparable to coexisting arthrodesis systems. Additionally, it was shown, that the complications detected correlate to the clinical outcome. In synopsis, this protocol is considered beneficial and should be taken into account in further studies.
In this paper, we address the problem of retrospective color shading correction. An extension of the established gray-level shading correction algorithm based on signal envelope (SE) estimation to color images is developed using principal color components. Compared to the probably most general shading correction algorithm based on entropy minimization, SE estimation does not need any computationally expensive optimization and thus can be implemented more effciently. We tested our new shading correction scheme on artificial as well as real endoscopic images and observed promising results. Additionally, an indepth analysis of the stop criterion used in the SE estimation algorithm is provided leading to the conclusion that a fixed, user-defined threshold is generally not feasible. Thus, we present new ideas how to develop a non-parametric version of the SE estimation algorithm using entropy.
We present the Regensburg Breast Shape Model (RBSM)—a 3D statistical shape model of the female breast built from 110 breast scans acquired in a standing position, and the first publicly available. Together with the model, a fully automated, pairwise surface registration pipeline used to establish dense correspondence among 3D breast scans is introduced. Our method is computationally efficient and requires only four landmarks to guide the registration process. A major challenge when modeling female breasts from surface-only 3D breast scans is the non-separability of breast and thorax. In order to weaken the strong coupling between breast and surrounding areas, we propose to minimize the variance outside the breast region as much as possible. To achieve this goal, a novel concept called breast probability masks (BPMs) is introduced. A BPM assigns probabilities to each point of a 3D breast scan, telling how likely it is that a particular point belongs to the breast area. During registration, we use BPMs to align the template to the target as accurately as possible inside the breast region and only roughly outside. This simple yet effective strategy significantly reduces the unwanted variance outside the breast region, leading to better statistical shape models in which breast shapes are quite well decoupled from the thorax. The RBSM is thus able to produce a variety of different breast shapes as independently as possible from the shape of the thorax. Our systematic experimental evaluation reveals a generalization ability of 0.17 mm and a specificity of 2.8 mm. To underline the expressiveness of the proposed model, we finally demonstrate in two showcase applications how the RBSM can be used for surgical outcome simulation and the prediction of a missing breast from the remaining one. Our model is available at https://www.rbsm.re-mic.de/.
We present the Regensburg Breast Shape Model (RBSM) – a 3D statistical shape model of the female breast built from 110 breast scans, and the first ever publicly available. Together with the model, a fully automated, pairwise surface registration pipeline used to establish correspondence among 3D breast scans is introduced. Our method is computationally efficient and requires only four landmarks to guide the registration process. In order to weaken the strong coupling between breast and thorax, we propose to minimize the variance outside the breast region as much as possible. To achieve this goal, a novel concept called breast probability masks (BPMs) is introduced. A BPM assigns probabilities to each point of a 3D breast scan, telling how likely it is that a particular point belongs to the breast area. During registration, we use BPMs to align the template to the target as accurately as possible inside the breast region and only roughly outside. This simple yet effective strategy significantly reduces the unwanted variance outside the breast region, leading to better statistical shape models in which breast shapes are quite well decoupled from the thorax. The RBSM is thus able to produce a variety of different breast shapes as independently as possible from the shape of the thorax. Our systematic experimental evaluation reveals a generalization ability of 0.17 mm and a specificity of 2.8 mm for the RBSM. Ultimately, our model is seen as a first step towards combining physically motivated deformable models of the breast and statistical approaches in order to enable more realistic surgical outcome simulation.
Whole-body PET/CT imaging
(2008)
Aim
Combined whole-body (WB) PET/CT imaging provides better overall co-registration compared to separate CT and PET. However, in clinical routine local PET-CT mis-registration cannot be avoided. Thus, the reconstructed PET tracer distribution may be biased when using the misaligned CT transmission data for CT-based attenuation correction (CT-AC). We investigate the feasibility of retrospective co-registration techniques to align CT and PET images prior to CT-AC, thus improving potentially the quality of combined PET/CT imaging in clinical routine.
Methods
First, using a commercial software registration package CT images were aligned to the uncorrected PET data by rigid and non-rigid registration methods. Co-registration accuracy of both alignment approaches was assessed by reviewing the PET tracer uptake patterns (visual, linked cursor display) following attenuation correction based on the original and co-registered CT. Second, we investigated non-rigid registration based on a prototype ITK implementation of the B-spline algorithm on a similar targeted MR-CT registration task, there showing promising results.
Results
Manual rigid, landmark-based co-registration introduced unacceptable misalignment, in particular in peripheral areas of the whole-body images. Manual, non-rigid landmark-based co-registration prior to CT-AC was successful with minor loco-regional distortions. Nevertheless, neither rigid nor non-rigid automatic co-registration based on the Mutual Information image to image metric succeeded in co-registering the CT and noAC-PET images. In contrast to widely available commercial software registration our implementation of an alternative automated, non-rigid B-spline co-registration technique yielded promising results in this setting with MR-CT data.
Conclusion
In clinical PET/CT imaging, retrospective registration of CT and uncorrected PET images may improve the quality of the AC-PET images. As of today no validated and clinically viable commercial registration software is in routine use. This has triggered our efforts in pursuing new approaches to a validated, non-rigid co-registration algorithm applicable to whole-body PET/CT imaging of which first results are presented here. This approach appears suitable for applications in retrospective WB-PET/CT alignment.
Ziel
Kombinierte PET/CT-Bildgebung ermöglicht verbesserte Koregistrierung von PET- und CT-Daten gegenüber separat akquirierten Bildern. Trotzdem entstehen in der klinischen Anwendung lokale Fehlregistrierungen, die zu Fehlern in der rekonstruierten PET- Tracerverteilung führen können, falls die unregistrierten CT-Daten zur Schwächungskorrektur (AC) der Emissionsdaten verwendet werden. Wir untersuchen daher die Anwendung von Bildregistrierungsalgorithmen vor der CT-basierten AC zur Verbesserung der PET-Aufnahmen.
Methoden
Mittels einer kommerziellen Registrierungssoftware wurden die CT-Daten eines PET/CT- Tomographen durch landmarken- und intensitätsbasierte rigide (starre) und nicht-rigide Registrierungsverfahren räumlich an die unkorrigierten PET-Emissionsdaten angepasst und zur AC verwendet. Zur Bewertung wurden die Tracerverteilungen in den PET-Bildern (vor AC, CT-AC, CT-AC nach Koregistrierung) visuell und mit Hilfe korrelierter Fadenkreuze verglichen. Zusätzlich untersuchten wir die ITK-Implementierung der bekannten B-spline basierten, nicht-rigiden Registrierungsansätze im Hinblick auf ihre Verwendbarkeit für die multimodale PET/CT-Ganzkörperregistrierung.
Ergebnisse
Mittels landmarkenbasierter, nicht-rigider Registrierung konnte die Tracerverteilung in den PET-Daten lokal verbessert werden. Landmarkenbasierte rigide Registrierung führte zu starker Fehlregistrierung in entfernten Körperregionen. Automatische rigide und nicht-rigide Registrierung unter Verwendung der Mutual-Information-Ähnlichkeitsmetrik versagte auf allen verwendeten Datensätzen. Die automatische Registrierung mit B-spline-Funktionen zeigte vielversprechende Resultate in der Anwendung auf einem ähnlich gelagerten CT–MR-Registrierungsproblem.
Fazit
Retrospektive, nicht-rigide Registrierung unkorrigierter PET- und CT-Aufnahmen aus kombinierten Aufnahmensystemen vor der AC kann die Qualität von PET-Aufnahmen im klinischen Einsatz verbessern. Trotzdem steht bis heute im klinischen Alltag keine validierte, automatische Registrierungssoftware zur Verfügung. Wir verfolgen dazu Ansätze für validierte, nicht-rigide Bildregistrierung für den klinischen Einsatz und präsentieren erste Ergebnisse.
Vascular malformations (VMs) are a rare condition. They can be categorized into high-flow and low-flow VMs, which is a challenging task for radiologists. In this work, a very heterogeneous set of MRI images with only rough annotations are used for classification with a convolutional neural network. The main focus is to describe the challenging data set and strategies to deal with such data in terms of preprocessing, annotation usage and choice of the network architecture. We achieved a classification result of 89.47 % F1-score with a 3D ResNet 18.
Structural MRI brain analysis and segmentation is a crucial part in the daily routine in neurosurgery for intervention planning. Exemplarily, the free software FSL-FAST (FMRIB’s Segmentation Library – FMRIB’s Automated Segmentation Tool) in version 4 is used for segmentation of brain tissue types. To speed up the segmentation procedure by parallel execution, we transferred FSL-FAST to a General Purpose Graphics Processing Unit (GPGPU) using Open Computing Language (OpenCL) [1]. The necessary steps for parallelization resulted in substantially different and less useful results. Therefore, the underlying methods were revised and adapted yielding computational overhead. Nevertheless, we achieved a speed-up factor of 3.59 from CPU to GPGPU execution, as well providing similar useful or even better results.
Die Lehre der medizinischen Bildverarbeitung vermittelt Kenntnisse mit einem breiten Methodenspektrum. Neben den Grundlagen der Verfahren soll ein Gefühl für eine geeignete Ausführungsreihenfolge und ihrer Wirkung auf medizinische Bilddaten entwickelt werden. Die Komplexität der Methoden erfordert vertiefte Programmierkenntnisse, sodass bereits einfache Operationen mit großem Programmieraufwand verbunden sind. Die Software GraphMIC stellt Bildverarbeitungsoperationen in Form interaktiver Knoten zur Verfügung und erlaubt das Arrangieren, Parametrisieren und Ausführen komplexer Verarbeitungssequenzen in einem Graphen. Durch den Fokus auf das Design einer Pipeline, weg von sprach- und frameworkspezifischen Implementierungsdetails, lassen sich grundlegende Prinzipien der Bildverarbeitung anschaulich erlernen. In diesem Beitrag stellen wir die visuelle Programmierung mit GraphMIC der nativen Implementierung äquivalenter Funktionen gegenüber. Die in C++ entwickelte Applikation basiert auf Qt, ITK, OpenCV, VTK und MITK.
Limitations in computer-assisted diagnosis include lack of labeled data and inability to model the relation between what experts see and what computers learn. Even though artificial intelligence and machine learning have demonstrated remarkable performances in medical image computing, their accountability and transparency level must be improved to transfer this success into clinical practice. The reliability of machine learning decisions must be explained and interpreted, especially for supporting the medical diagnosis. While deep learning techniques are broad so that unseen information might help learn patterns of interest, human insights to describe objects of interest help in decision-making. This paper proposes a novel approach, DeepCraftFuse, to address the challenge of combining information provided by deep networks with visual-based features to significantly enhance the correct identification of cancerous tissues in patients affected with Barrett’s esophagus (BE). We demonstrate that DeepCraftFuse outperforms state-of-the-art techniques on private and public datasets, reaching results of around 95% when distinguishing patients affected by BE that is either positive or negative to esophageal cancer.
Reduktion von Rissartefakten durch nicht-lineare Registrierung in histologischen Schnittbildern
(2009)
In dieser Arbeit wird ein Verfahren vorgestellt, das Rissartefakte, die in histologischen Rattenhirnschnitten vorkommen können, durch nicht-lineare Registrierung reduziert. Um die Optimierung in der Rissregion zu leiten, wird der Curvature Registrierungsansatz um eine Metrik basierend auf der Segmentierung der Bilder erweitert. Dabei erzielten Registrierungen mit der ausschließlichen Segmentierung des Risses bessere Ergebnisse als Registrierungen mit einer Segmentierung des gesamten Hirnschnitts. Insgesamt zeigt sich eine deutliche Verbesserung in der Rissregion, wobei der verbleibende reduzierte Riss auf die Glattheitsbedingungen des Regularisierers zurückzuführen ist.
Aims
Recent evidence suggests the possibility of intraprocedural phase recognition in surgical operations as well as endoscopic interventions such as peroral endoscopic myotomy and endoscopic submucosal dissection (ESD) by AI-algorithms. The intricate measurement of intraprocedural phase distribution may deepen the understanding of the procedure. Furthermore, real-time quality assessment as well as automation of reporting may become possible. Therefore, we aimed to develop an AI-algorithm for intraprocedural phase recognition during ESD.
Methods
A training dataset of 364385 single images from 9 full-length ESD videos was compiled. Each frame was classified into one procedural phase. Phases included scope manipulation, marking, injection, application of electrical current and bleeding. Allocation of each frame was only possible to one category. This training dataset was used to train a Video Swin transformer to recognize the phases. Temporal information was included via logarithmic frame sampling. Validation was performed using two separate ESD videos with 29801 single frames.
Results
The validation yielded sensitivities of 97.81%, 97.83%, 95.53%, 85.01% and 87.55% for scope manipulation, marking, injection, electric application and bleeding, respectively. Specificities of 77.78%, 90.91%, 95.91%, 93.65% and 84.76% were measured for the same parameters.
Conclusions
The developed algorithm was able to classify full-length ESD videos on a frame-by-frame basis into the predefined classes with high sensitivities and specificities. Future research will aim at the development of quality metrics based on single-operator phase distribution.
Background and aims
Celiac disease with its endoscopic manifestation of villous atrophy is underdiagnosed worldwide. The application of artificial intelligence (AI) for the macroscopic detection of villous atrophy at routine esophagogastroduodenoscopy may improve diagnostic performance.
Methods
A dataset of 858 endoscopic images of 182 patients with villous atrophy and 846 images from 323 patients with normal duodenal mucosa was collected and used to train a ResNet 18 deep learning model to detect villous atrophy. An external data set was used to test the algorithm, in addition to six fellows and four board certified gastroenterologists. Fellows could consult the AI algorithm’s result during the test. From their consultation distribution, a stratification of test images into “easy” and “difficult” was performed and used for classified performance measurement.
Results
External validation of the AI algorithm yielded values of 90 %, 76 %, and 84 % for sensitivity, specificity, and accuracy, respectively. Fellows scored values of 63 %, 72 % and 67 %, while the corresponding values in experts were 72 %, 69 % and 71 %, respectively. AI consultation significantly improved all trainee performance statistics. While fellows and experts showed significantly lower performance for “difficult” images, the performance of the AI algorithm was stable.
Conclusion
In this study, an AI algorithm outperformed endoscopy fellows and experts in the detection of villous atrophy on endoscopic still images. AI decision support significantly improved the performance of non-expert endoscopists. The stable performance on “difficult” images suggests a further positive add-on effect in challenging cases.
Aims
Celiac disease (CD) is a complex condition caused by an autoimmune reaction to ingested gluten. Due to its polymorphic manifestation and subtle endoscopic presentation, the diagnosis is difficult and thus the disorder is underreported. We aimed to use deep learning to identify celiac disease on endoscopic images of the small bowel.
Methods
Patients with small intestinal histology compatible with CD (MARSH classification I-III) were extracted retrospectively from the database of Augsburg University hospital. They were compared to patients with no clinical signs of CD and histologically normal small intestinal mucosa. In a first step MARSH III and normal small intestinal mucosa were differentiated with the help of a deep learning algorithm. For this, the endoscopic white light images were divided into five equal-sized subsets. We avoided splitting the images of one patient into several subsets. A ResNet-50 model was trained with the images from four subsets and then validated with the remaining subset. This process was repeated for each subset, such that each subset was validated once. Sensitivity, specificity, and harmonic mean (F1) of the algorithm were determined.
Results
The algorithm showed values of 0.83, 0.88, and 0.84 for sensitivity, specificity, and F1, respectively. Further data showing a comparison between the detection rate of the AI model and that of experienced endoscopists will be available at the time of the upcoming conference.
Conclusions
We present the first clinical report on the use of a deep learning algorithm for the detection of celiac disease using endoscopic images. Further evaluation on an external data set, as well as in the detection of CD in real-time, will follow. However, this work at least suggests that AI can assist endoscopists in the endoscopic diagnosis of CD, and ultimately may be able to do a true optical biopsy in live-time.
Aims
While AI has been successfully implemented in detecting and characterizing colonic polyps, its role in therapeutic endoscopy remains to be elucidated. Especially third space endoscopy procedures like ESD and peroral endoscopic myotomy (POEM) pose a technical challenge and the risk of operator-dependent complications like intraprocedural bleeding and perforation. Therefore, we aimed at developing an AI-algorithm for intraprocedural real time vessel detection during ESD and POEM.
Methods
A training dataset consisting of 5470 annotated still images from 59 full-length videos (47 ESD, 12 POEM) and 179681 unlabeled images was used to train a DeepLabV3+neural network with the ECMT semi-supervised learning method. Evaluation for vessel detection rate (VDR) and time (VDT) of 19 endoscopists with and without AI-support was performed using a testing dataset of 101 standardized video clips with 200 predefined blood vessels. Endoscopists were stratified into trainees and experts in third space endoscopy.
Results
The AI algorithm had a mean VDR of 93.5% and a median VDT of 0.32 seconds. AI support was associated with a statistically significant increase in VDR from 54.9% to 73.0% and from 59.0% to 74.1% for trainees and experts, respectively. VDT significantly decreased from 7.21 sec to 5.09 sec for trainees and from 6.10 sec to 5.38 sec for experts in the AI-support group. False positive (FP) readings occurred in 4.5% of frames. FP structures were detected significantly shorter than true positives (0.71 sec vs. 5.99 sec).
Conclusions
AI improved VDR and VDT of trainees and experts in third space endoscopy and may reduce performance variability during training. Further research is needed to evaluate the clinical impact of this new technology.
Aims
AI has proven great potential in assisting endoscopists in diagnostics, however its role in therapeutic endoscopy remains unclear. Endoscopic submucosal dissection (ESD) is a technically demanding intervention with a slow learning curve and relevant risks like bleeding and perforation. Therefore, we aimed to develop an algorithm for the real-time detection and delineation of relevant structures during third-space endoscopy.
Methods
5470 still images from 59 full length videos (47 ESD, 12 POEM) were annotated. 179681 additional unlabeled images were added to the training dataset. Consequently, a DeepLabv3+ neural network architecture was trained with the ECMT semi-supervised algorithm (under review elsewhere). Evaluation of vessel detection was performed on a dataset of 101 standardized video clips from 15 separate third-space endoscopy videos with 200 predefined blood vessels.
Results
Internal validation yielded an overall mean Dice score of 85% (68% for blood vessels, 86% for submucosal layer, 88% for muscle layer). On the video test data, the overall vessel detection rate (VDR) was 94% (96% for ESD, 74% for POEM). The median overall vessel detection time (VDT) was 0.32 sec (0.3 sec for ESD, 0.62 sec for POEM).
Conclusions
Evaluation of the developed algorithm on a video test dataset showed high VDR and quick VDT, especially for ESD. Further research will focus on a possible clinical benefit of the AI application for VDR and VDT during third-space endoscopy.
Einleitung
Third space Endoskopieprozeduren wie die endoskopische Submukosadissektion (ESD) und die perorale endoskopische Myotomie (POEM) sind technisch anspruchsvoll und gehen mit untersucherabhängigen Komplikationen wie Blutungen und Perforationen einher. Grund hierfür ist die unabsichtliche Durchschneidung von submukosalen Blutgefäßen ohne präemptive Koagulation.
Ziele
Die Forschungsfrage, ob ein KI-Algorithmus die intraprozedurale Gefäßerkennung bei ESD und POEM unterstützen und damit Komplikationen wie Blutungen verhindern könnte, erscheint in Anbetracht des erfolgreichen Einsatzes von KI bei der Erkennung von Kolonpolypen interessant.
Methoden
Auf 5470 Einzelbildern von 59 third space Endoscopievideos wurden submukosale Blutgefäße annotiert. Zusammen mit weiteren 179.681 nicht-annotierten Bildern wurde ein DeepLabv3+neuronales Netzwerk mit dem ECMT-Verfahren für semi-supervised learning trainiert, um Blutgefäße in Echtzeit erkennen zu können. Für die Evaluation wurde ein Videotest mit 101 Videoclips aus 15 vom Trainingsdatensatz separaten Prozeduren mit 200 vordefinierten Gefäßen erstellt. Die Gefäßdetektionsrate, -zeit und -dauer, definiert als der Prozentsatz an Einzelbildern eines Videos bezogen auf den Goldstandard, auf denen ein definiertes Gefäß erkannt wurde, wurden erhoben. Acht erfahrene Endoskopiker wurden mithilfe dieses Videotests im Hinblick auf Gefäßdetektion getestet, wobei eine Hälfte der Videos nativ, die andere Hälfte nach Markierung durch den KI-Algorithmus angesehen wurde.
Ergebnisse
Der mittlere Dice Score des Algorithmus für Blutgefäße war 68%. Die mittlere Gefäßdetektionsrate im Videotest lag bei 94% (96% für ESD; 74% für POEM). Die mediane Gefäßdetektionszeit des Algorithmus lag bei 0,32 Sekunden (0,3 Sekunden für ESD; 0,62 Sekunden für POEM). Die mittlere Gefäßdetektionsdauer lag bei 59,1% (60,6% für ESD; 44,8% für POEM) des Goldstandards. Alle Endoskopiker hatten mit KI-Unterstützung eine höhere Gefäßdetektionsrate als ohne KI. Die mittlere Gefäßdetektionsrate ohne KI lag bei 56,4%, mit KI bei 71,2% (p<0.001).
Schlussfolgerung
KI-Unterstützung war mit einer statistisch signifikant höheren Gefäßdetektionsrate vergesellschaftet. Die mediane Gefäßdetektionszeit von deutlich unter einer Sekunde sowie eine Gefäßdetektionsdauer von größer 50% des Goldstandards wurden für den klinischen Einsatz als ausreichend erachtet. In prospektiven Anwendungsstudien sollte der KI-Algorithmus auf klinische Relevanz getestet werden.
Clinical setting
Third space procedures such as endoscopic submucosal dissection (ESD) and peroral endoscopic myotomy (POEM) are complex minimally invasive techniques with an elevated risk for operator-dependent adverse events such as bleeding and perforation. This risk arises from accidental dissection into the muscle layer or through submucosal blood vessels as the submucosal cutting plane within the expanding resection site is not always apparent. Deep learning algorithms have shown considerable potential for the detection and characterization of gastrointestinal lesions. So-called AI – clinical decision support solutions (AI-CDSS) are commercially available for polyp detection during colonoscopy. Until now, these computer programs have concentrated on diagnostics whereas an AI-CDSS for interventional endoscopy has not yet been introduced. We aimed to develop an AI-CDSS („Smart ESD“) for real-time intra-procedural detection and delineation of blood vessels, tissue structures and endoscopic instruments during third-space endoscopic procedures.
Characteristics of Smart ESD
An AI-CDSS was invented that delineates blood vessels, tissue structures and endoscopic instruments during third-space endoscopy in real-time. The output can be displayed by an overlay over the endoscopic image with different modes of visualization, such as a color-coded semitransparent area overlay, or border tracing (demonstration video). Hereby the optimal layer for dissection can be visualized, which is close above or directly at the muscle layer, depending on the applied technique (ESD or POEM). Furthermore, relevant blood vessels (thickness> 1mm) are delineated. Spatial proximity between the electrosurgical knife and a blood vessel triggers a warning signal. By this guidance system, inadvertent dissection through blood vessels could be averted.
Technical specifications
A DeepLabv3+ neural network architecture with KSAC and a 101-layer ResNeSt backbone was used for the development of Smart ESD. It was trained and validated with 2565 annotated still images from 27 full length third-space endoscopic videos. The annotation classes were blood vessel, submucosal layer, muscle layer, electrosurgical knife and endoscopic instrument shaft. A test on a separate data set yielded an intersection over union (IoU) of 68%, a Dice Score of 80% and a pixel accuracy of 87%, demonstrating a high overlap between expert and AI segmentation. Further experiments on standardized video clips showed a mean vessel detection rate (VDR) of 85% with values of 92%, 70% and 95% for POEM, rectal ESD and esophageal ESD respectively. False positive measurements occurred 0.75 times per minute. 7 out of 9 vessels which caused intraprocedural bleeding were caught by the algorithm, as well as both vessels which required hemostasis via hemostatic forceps.
Future perspectives
Smart ESD performed well for vessel and tissue detection and delineation on still images, as well as on video clips. During a live demonstration in the endoscopy suite, clinical applicability of the innovation was examined. The lag time for processing of the live endoscopic image was too short to be visually detectable for the interventionist. Even though the algorithm could not be applied during actual dissection by the interventionist, Smart ESD appeared readily deployable during visual assessment by ESD experts. Therefore, we plan to conduct a clinical trial in order to obtain CE-certification of the algorithm. This new technology may improve procedural safety and speed, as well as training of modern minimally invasive endoscopic resection techniques.
ARTIFICIAL INTELLIGENCE (AI) – ASSISTED VESSEL AND TISSUE RECOGNITION IN THIRD-SPACE ENDOSCOPY
(2022)
Aims
Third-space endoscopy procedures such as endoscopic submucosal dissection (ESD) and peroral endoscopic myotomy (POEM) are complex interventions with elevated risk of operator-dependent adverse events, such as intra-procedural bleeding and perforation. We aimed to design an artificial intelligence clinical decision support solution (AI-CDSS, “Smart ESD”) for the detection and delineation of vessels, tissue structures, and instruments during third-space endoscopy procedures.
Methods
Twelve full-length third-space endoscopy videos were extracted from the Augsburg University Hospital database. 1686 frames were annotated for the following categories: Submucosal layer, blood vessels, electrosurgical knife and endoscopic instrument. A DeepLabv3+neural network with a 101-layer ResNet backbone was trained and validated internally. Finally, the ability of the AI system to detect visible vessels during ESD and POEM was determined on 24 separate video clips of 7 to 46 seconds duration and showing 33 predefined vessels. These video clips were also assessed by an expert in third-space endoscopy.
Results
Smart ESD showed a vessel detection rate (VDR) of 93.94%, while an average of 1.87 false positive signals were recorded per minute. VDR of the expert endoscopist was 90.1% with no false positive findings. On the internal validation data set using still images, the AI system demonstrated an Intersection over Union (IoU), mean Dice score and pixel accuracy of 63.47%, 76.18% and 86.61%, respectively.
Conclusions
This is the first AI-CDSS aiming to mitigate operator-dependent limitations during third-space endoscopy. Further clinical trials are underway to better understand the role of AI in such procedures.
Einleitung
Third-Space Interventionen wie die endoskopische Submukosadissektion (ESD) und die perorale endoskopische Myotomie (POEM) sind technisch anspruchsvoll und mit einem erhöhten Risiko für intraprozedurale Komplikationen wie Blutung oder Perforation assoziiert. Moderne Computerprogramme zur Unterstützung bei diagnostischen Entscheidungen werden unter Einsatz von künstlicher Intelligenz (KI) in der Endoskopie bereits erfolgreich eingesetzt. Ziel der vorliegenden Arbeit war es, relevante anatomische Strukturen mithilfe eines Deep-Learning Algorithmus zu detektieren und segmentieren, um die Sicherheit und Anwendbarkeit von ESD und POEM zu erhöhen.
Methoden
Zwölf Videoaufnahmen in voller Länge von Third-Space Endoskopien wurden aus der Datenbank des Universitätsklinikums Augsburg extrahiert. 1686 Einzelbilder wurden für die Kategorien Submukosa, Blutgefäß, Dissektionsmesser und endoskopisches Instrument annotiert und segmentiert. Mit diesem Datensatz wurde ein DeepLabv3+neuronales Netzwerk auf der Basis eines ResNet mit 101 Schichten trainiert und intern anhand der Parameter Intersection over Union (IoU), Dice Score und Pixel Accuracy validiert. Die Fähigkeit des Algorithmus zur Gefäßdetektion wurde anhand von 24 Videoclips mit einer Spieldauer von 7 bis 46 Sekunden mit 33 vordefinierten Gefäßen evaluiert. Anhand dieses Tests wurde auch die Gefäßdetektionsrate eines Experten in der Third-Space Endoskopie ermittelt.
Ergebnisse
Der Algorithmus zeigte eine Gefäßdetektionsrate von 93,94% mit einer mittleren Rate an falsch positiven Signalen von 1,87 pro Minute. Die Gefäßdetektionsrate des Experten lag bei 90,1% ohne falsch positive Ergebnisse. In der internen Validierung an Einzelbildern wurde eine IoU von 63,47%, ein mittlerer Dice Score von 76,18% und eine Pixel Accuracy von 86,61% ermittelt.
Zusammenfassung
Dies ist der erste KI-Algorithmus, der für den Einsatz in der therapeutischen Endoskopie entwickelt wurde. Präliminäre Ergebnisse deuten auf eine mit Experten vergleichbare Detektion von Gefäßen während der Untersuchung hin. Weitere Untersuchungen sind nötig, um die Leistung des Algorithmus im Vergleich zum Experten genauer zu eruieren sowie einen möglichen klinischen Nutzen zu ermitteln.
Aims
VA is an endoscopic finding of celiac disease (CD), which can easily be missed if pretest probability is low. In this study, we aimed to develop an artificial intelligence (AI) algorithm for the detection of villous atrophy on endoscopic images.
Methods
858 images from 182 patients with VA and 846 images from 323 patients with normal duodenal mucosa were used for training and internal validation of an AI algorithm (ResNet18). A separate dataset was used for external validation, as well as determination of detection performance of experts, trainees and trainees with AI support. According to the AI consultation distribution, images were stratified into “easy” and “difficult”.
Results
Internal validation showed 82%, 85% and 84% for sensitivity, specificity and accuracy. External validation showed 90%, 76% and 84%. The algorithm was significantly more sensitive and accurate than trainees, trainees with AI support and experts in endoscopy. AI support in trainees was associated with significantly improved performance. While all endoscopists showed significantly lower detection for “difficult” images, AI performance remained stable.
Conclusions
The algorithm outperformed trainees and experts in sensitivity and accuracy for VA detection. The significant improvement with AI support suggests a potential clinical benefit. Stable performance of the algorithm in “easy” and “difficult” test images may indicate an advantage in macroscopically challenging cases.
In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images and videos. In particular, the determination of the position and type of instruments is of great interest. Current work involves both spatial and temporal information, with the idea that predicting the movement of surgical tools over time may improve the quality of the final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify and characterize datasets used for method development and evaluation and quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images and videos. The paper focuses on methods that work purely visually, without markers of any kind attached to the instruments, considering both single-frame semantic and instance segmentation approaches, as well as those that incorporate temporal information. The publications analyzed were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were “instrument segmentation”, “instrument tracking”, “surgical tool segmentation”, and “surgical tool tracking”, resulting in a total of 741 articles published between 01/2015 and 07/2023, of which 123 were included using systematic selection criteria. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing the available potential for future developments.
Aims
Eosinophilic esophagitis (EoE) is easily missed during endoscopy, either because physicians are not familiar with its endoscopic features or the morphologic changes are too subtle. In this preliminary paper, we present the first attempt to detect EoE in endoscopic white light (WL) images using a deep learning network (EoE-AI).
Methods
401 WL images of eosinophilic esophagitis and 871 WL images of normal esophageal mucosa were evaluated. All images were assessed for the Endoscopic Reference score (EREFS) (edema, rings, exudates, furrows, strictures). Images with strictures were excluded. EoE was defined as the presence of at least 15 eosinophils per high power field on biopsy. A convolutional neural network based on the ResNet architecture with several five-fold cross-validation runs was used. Adding auxiliary EREFS-classification branches to the neural network allowed the inclusion of the scores as optimization criteria during training. EoE-AI was evaluated for sensitivity, specificity, and F1-score. In addition, two human endoscopists evaluated the images.
Results
EoE-AI showed a mean sensitivity, specificity, and F1 of 0.759, 0.976, and 0.834 respectively, averaged over the five distinct cross-validation runs. With the EREFS-augmented architecture, a mean sensitivity, specificity, and F1-score of 0.848, 0.945, and 0.861 could be demonstrated respectively. In comparison, the two human endoscopists had an average sensitivity, specificity, and F1-score of 0.718, 0.958, and 0.793.
Conclusions
To the best of our knowledge, this is the first application of deep learning to endoscopic images of EoE which were also assessed after augmentation with the EREFS-score. The next step is the evaluation of EoE-AI using an external dataset. We then plan to assess the EoE-AI tool on endoscopic videos, and also in real-time. This preliminary work is encouraging regarding the ability for AI to enhance physician detection of EoE, and potentially to do a true “optical biopsy” but more work is needed.
The endoscopic features associated with eosinophilic esophagitis (EoE) may be missed during routine endoscopy. We aimed to develop and evaluate an Artificial Intelligence (AI) algorithm for detecting and quantifying the endoscopic features of EoE in white light images, supplemented by the EoE Endoscopic Reference Score (EREFS). An AI algorithm (AI-EoE) was constructed and trained to differentiate between EoE and normal esophagus using endoscopic white light images extracted from the database of the University Hospital Augsburg. In addition to binary classification, a second algorithm was trained with specific auxiliary branches for each EREFS feature (AI-EoE-EREFS). The AI algorithms were evaluated on an external data set from the University of North Carolina, Chapel Hill (UNC), and compared with the performance of human endoscopists with varying levels of experience. The overall sensitivity, specificity, and accuracy of AI-EoE were 0.93 for all measures, while the AUC was 0.986. With additional auxiliary branches for the EREFS categories, the AI algorithm (AI-EoEEREFS) performance improved to 0.96, 0.94, 0.95, and 0.992 for sensitivity, specificity, accuracy, and AUC, respectively. AI-EoE and AI-EoE-EREFS performed significantly better than endoscopy beginners and senior fellows on the same set of images. An AI algorithm can be trained to detect and quantify endoscopic features of EoE with excellent performance scores. The addition of the EREFS criteria improved the performance of the AI algorithm, which performed significantly better than endoscopists with a lower or medium experience level.
Digital nerve lesions result in a loss of tactile sensation reflected by an anesthetic area (AA) at the radial or ulnar aspect of the respective digit. Yet, available tools to monitor the recovery of tactile sense have been criticized for their lack of validity. However, the precise quantification of AA dynamics by three-dimensional (3-D) imaging could serve as an accurate surrogate to monitor recovery following digital nerve repair.
For validation, AAs were marked on digits of healthy volunteers to simulate the AA of an impaired cutaneous innervation. Three dimensional models were composed from raw images that had been acquired with a 3-D camera (Vectra H2) to precisely quantify relative AA for each digit (3-D models, n= 80). Operator properties varied regarding individual experience in 3-D imaging and image processing. Additionally, the concept was applied in a clinical case study.
Images taken by experienced photographers were rated better quality (p< 0.001) and needed less processing time (p= 0.020). Quantification of the relative AA was neither altered significantly by experience levels of the photographer (p= 0.425) nor the image assembler (p= 0.749).
The proposed concept allows precise and reliable surface quantification of digits and can be performed consistently without relevant distortion by lack of examiner experience. Routine 3-D imaging of the AA has the great potential to provide visual evidence of various returning states of sensation and to convert sensory nerve recovery into a metric variable with high responsiveness to temporal progress.
In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images. Especially the determination of the position and type of the instruments is of great interest here. Current work involves both spatial and temporal information with the idea, that the prediction of movement of surgical tools over time may improve the quality of final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify datasets used for method development and evaluation, as well as quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images. The paper focuses on methods that work purely visually without attached markers of any kind on the instruments, taking into account both single-frame segmentation approaches as well as those involving temporal information. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing available potential for future developments. The publications considered were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were "instrument segmentation", "instrument tracking", "surgical tool segmentation", and "surgical tool tracking" and result in 408 articles published between 2015 and 2022 from which 109 were included using systematic selection criteria.
Heavy smoke development represents an important challenge for operating physicians during laparoscopic procedures and can potentially affect the success of an intervention due to reduced visibility and orientation. Reliable and accurate recognition of smoke is therefore a prerequisite for the use of downstream systems such as automated smoke evacuation systems. Current approaches distinguish between non-smoked and smoked frames but often ignore the temporal context inherent in endoscopic video data. In this work, we therefore present a method that utilizes the pixel-wise displacement from randomly sampled images to the preceding frames determined using the optical flow algorithm by providing the transformed magnitude of the displacement as an additional input to the network. Further, we incorporate the temporal context at evaluation time by applying an exponential moving average on the estimated class probabilities of the model output to obtain more stable and robust results over time. We evaluate our method on two convolutional-based and one state-of-the-art transformer architecture and show improvements in the classification results over a baseline approach, regardless of the network used.
Aims
Human-computer interactions (HCI) may have a relevant impact on the performance of Artificial Intelligence (AI). Studies show that although endoscopists assessing Barrett’s esophagus (BE) with AI improve their performance significantly, they do not achieve the level of the stand-alone performance of AI. One aspect of HCI is the impact of AI on the degree of certainty and confidence displayed by the endoscopist. Indirectly, diagnostic confidence when using AI may be linked to trust and acceptance of AI. In a BE video study, we aimed to understand the impact of AI on the diagnostic confidence of endoscopists and the possible correlation with diagnostic performance.
Methods
22 endoscopists from 12 centers with varying levels of BE experience reviewed ninety-six standardized endoscopy videos. Endoscopists were categorized into experts and non-experts and randomly assigned to assess the videos with and without AI. Participants were randomized in two arms: Arm A assessed videos first without AI and then with AI, while Arm B assessed videos in the opposite order. Evaluators were tasked with identifying BE-related neoplasia and rating their confidence with and without AI on a scale from 0 to 9.
Results
The utilization of AI in Arm A (without AI first, with AI second) significantly elevated confidence levels for experts and non-experts (7.1 to 8.0 and 6.1 to 6.6, respectively). Only non-experts benefitted from AI with a significant increase in accuracy (68.6% to 75.5%). Interestingly, while the confidence levels of experts without AI were higher than those of non-experts with AI, there was no significant difference in accuracy between these two groups (71.3% vs. 75.5%). In Arm B (with AI first, without AI second), experts and non-experts experienced a significant reduction in confidence (7.6 to 7.1 and 6.4 to 6.2, respectively), while maintaining consistent accuracy levels (71.8% to 71.8% and 67.5% to 67.1%, respectively).
Conclusions
AI significantly enhanced confidence levels for both expert and non-expert endoscopists. Endoscopists felt significantly more uncertain in their assessments without AI. Furthermore, experts with or without AI consistently displayed higher confidence levels than non-experts with AI, irrespective of comparable outcomes. These findings underscore the possible role of AI in improving diagnostic confidence during endoscopic assessment.
Celiac disease is an autoimmune disorder caused by gluten that results in an inflammatory response of the small intestine.We investigated whether celiac disease can be detected using endoscopic images through a deep learning approach. The results show that additional clinical parameters can improve the classification accuracy. In this work, we distinguished between healthy tissue and Marsh III, according to the Marsh score system. We first trained a baseline network to classify endoscopic images of the small bowel into these two classes and then augmented the approach with a multimodality component that took the antibody status into account.
Presenting images from different modalities seems to be a trivial task considering the challenges to obtain registered images as a pre-requisite for image fusion. In combined tomographs like PET/CT, image registration is intrinsic. However, informative image fusion mandates careful preparation owing to the large amount of information that is presented to the observer. In complex imaging situations it is required to provide tools that are easy to handle and still powerful enough to help the observer discriminating important details from background patterns. We investigated several options for color tables applied to brain and non-brain images obtained with PET, MRI and CT.
Volumetric representations of autoradiographic and histological images gain ever more interest as a base to interpret data obtained with /spl mu/-imaging devices like microPET. Beyond supporting spatial orientation within rat brains especially autoradiographic images may serve as a base to quantitatively evaluate the complex uptake patterns of microPET studies with receptor ligands or tumor tracers. They may also serve for the development of rat brain atlases or data models, which can be explored during further image analysis or simulation studies. In all cases a consistent spatial representation of the rat brain, i.e. its anatomy and the corresponding quantitative uptake pattern, is required. This includes both, a restacking of the individual two-dimensional images and the exact registration of the respective volumes. We propose strategies how these volumes can be created in a consistent way and trying to limit the requirements on the circumstances during data acquisition, i.e. being independent from other sources like video imaging of the block face prior to cutting or high resolution micro-X-ray CT or micro MRI.
The number of patients with Barret’s esophagus (BE) has increased in the last decades. Considering the dangerousness of the disease and its evolution to adenocarcinoma, an early diagnosis of BE may provide a high probability of cancer remission. However, limitations regarding traditional methods of detection and management of BE demand alternative solutions. As such, computer-aided tools have been recently used to assist in this problem, but the challenge still persists. To manage the problem, we introduce the infinity Restricted Boltzmann Machines (iRBMs) to the task of automatic identification of Barrett’s esophagus from endoscopic images of the lower esophagus. Moreover, since iRBM requires a proper selection of its meta-parameters, we also present a discriminative iRBM fine-tuning using six meta-heuristic optimization techniques. We showed that iRBMs are suitable for the context since it provides competitive results, as well as the meta-heuristic techniques showed to be appropriate for such task.
In neuroscience, small-animal studies frequently involve dealing with series of images from multiple modalities such as histology and autoradiography. The consistent and bias-free restacking of multi-modality image series is obligatory as a starting point for subsequent non-rigid registration procedures and for quantitative comparisons with positron emission tomography (PET) and other in vivo data. Up to now, consistency between 2D slices without cross validation using an inherent 3D modality is frequently presumed to be close to the true morphology due to the smooth appearance of the contours of anatomical structures. However, in multi-modality stacks consistency is difficult to assess. In this work, consistency is defined in terms of smoothness of neighboring slices within a single modality and between different modalities. Registration bias denotes the distortion of the registered stack in comparison to the true 3D morphology and shape. Based on these metrics, different restacking strategies of multi-modality rat brain slices are experimentally evaluated. Experiments based on MRI-simulated and real dual-tracer autoradiograms reveal a clear bias of the restacked volume despite quantitatively high consistency and qualitatively smooth brain structures. However, different registration strategies yield different inter-consistency metrics. If no genuine 3D modality is available, the use of the so-called SOP (slice-order preferred) or MOSOP (modality-and-slice-order preferred) strategy is recommended.
Evaluierung von Registrierungsstrategien zur multimodalen 3D-Rekonstruktion von Rattenhirnschnitten
(2006)
In dieser Arbeit werden drei Strategien zur 3D Stapelung von multimodalen Schnittbildern vorgestellt. Die Strategien werden experimentell anhand von Dualtracer-Autoradiographien evaluiert. Dazu werden neue Maße zur Beschreibung der Konsistenz innerhalb einer Modalität und der Konsistenz der Modalitäten untereinander entwickelt, die auf bekannten Registrierungsmetriken basieren. Gerade bezüglich der Konsistenz der Modalitäten untereinander zeigen zwei Strategien die besten Resultate: (1) abwechselnde multimodale Registrierung (2) monomodale Rekonstruktion einer Modalität und multimodale 2D Registrierung der zweiten Modalität.
Introduction – Diagnosis of abnormal cilia function is based on ultrastructural analysis of axoneme defects, especialy the features of inner and outer dynein arms which are the motors of ciliar motility. Sub-optimal biopsy material, methodical, and intrinsic electron microscopy factors pose difficulty in ciliary defects evaluation. We present a computer-assisted approach based on state-of-the-art image analysis and object recognition methods yielding a time-saving and efficient diagnosis of cilia dysfunction. Method – The presented approach is based on a pipeline of basal image processing methods like smoothing, thresholding and ellipse fitting. However, integration of application specific knowledge results in robust segmentations even in cases of image artifacts. The method is build hierarchically starting with the detection of cilia within the image, followed by the detection of nine doublets within each analyzable cilium, and ending with the detection of dynein arms of each doublet. The process is concluded by a rough classification of the dynein arms as basis for a computer-assisted diagnosis. Additionally, the interaction possibilities are designed in a way, that the results are still reproducible given the completion report. Results – A qualitative evaluation showed reasonable detection results for cilia, doublets and dynein arms. However, since a ground truth is missing, the variation of the computer-assisted diagnosis should be within the subjective bias of human diagnosticians. The results of a first quantitative evaluation with five human experts and six images with 12 analyzable cilia showed, that with default parameterization 91.6% of the cilia and 98% of the doublets were found. The computer-assisted approach rated 66% of those inner and outer dynein arms correct, where all human experts agree. However, especially the quality of the dynein arm classification may be improved in future work.
Quantitative Farbmessungen sollen die Diagnostik laryngealer Erkrankungen unterstützen. Dabei wird der Farbeindruck nicht nur durch die Reflexionseigenschaften des Gewebes sondern auch durch die Farbe der verwendeten Lichtquelle beeinflußt. Der hier vorgestellte Farbkonstanz-Algorithmus basiert auf dem dichromatischen Reflexionsmodell und liefert eine pixelweise Trennung des Farbbildes in seine beiden Faxbanteile. Die Körperfarbe entspricht dabei der gewebespezifischen Reflexion, die Oberfächenfarbe der Strahlung der Lichtquelle.
Nutzung eines Farbkonstanz-Algorithmus zur Entfernung von Glanzlichtern in laryngoskopischen Bildern
(1998)
1 Einführung
Funktionelle und organische Störungen im Larynx beeinträchtigen die Ausdrucksfähigkeit des Menschen. Zur Diagnostik und Verlaufkontrolle werden die Stimmlippen im Larynx mit Hilfe der Video-Laryngoskopie aufgenommen. Zur optimalen Farbmessung wird dazu an das Lupenendoskop eine 3-Chip-CCD-Kamera angeschlossen, die eine unabhängige Aufnahme der drei Farbkanäle erlaubt. Die bisherige subjektive Befundung ist von der Erfahrung des Untersuchers abhängig und läßt nur eine grobe Klassifikation der Krankheitsbilder zu. Zur Objektivierung werden daher quantitative Parameter für Farbe, Textur und Schwingung entwickelt. Neben dem Einfluß der wechselnden Lichtquellenfarbe auf den Farbeindruck ist die Sekretauflage auf den Stimmlippen ein Problem bei der Farb-und Texturanalyse. Sie kann zu ausgedehnten Glanzlichtern führen und so weite Bereiche der Stimmlippen für die Farb-und Texturanalyse unbrauchbar machen. Dieser Beitrag stellt einen Farbkonstanz-Algorithmus vor, der unabhängig von der Lichtquelle quantitative Farbwerte des Gewebes liefert und die Glanzlichtdetektion und -elimination ermöglicht.
2 Methodik
Ziel des Farbkonstanz-Algorithmus ist die Trennung von Lichtquellen-und Gewebefarbe. Unter Verwendung des dichromatischen Reflexionsmodells [1] kann die Oberflächenreflexion mit der Farbe der Lichtquelle und die Körperreflexion mit der Gewebefarbe identifiziert werden. Der Farbeindruck entsteht aus der Linearkombination beider Farbkomponenten. Ihre Gewichtung ist von der Aufnahmegeometrie abhängig, insbesondere vom Winkel zwischen Oberflächennormalen und dem Positionsvektor der Lichtquelle. In einem zweistufigen Verfahren wird zunächst die Lichtquellenfarbe geschätzt, dann die Gewebefarbe ermittelt. Hieraus können beide Farbanteile durch die Berechnung der Gewichtsfaktoren pixelweise getrennt werden.
Zur diagnostischen Unterstützung bei der Befundung laryngealer Erkrankungen soll eine Farb- und Formanalyse der Stimmlippen durchgeführt werden. In diesem Beitrag wird ein Verfahren zur Trennung der spiegelnden und diffusen Reflexionsanteile in Farbbildern des Larynx vorgestellt. Die Farbe der diffusen Komponente entspricht dabei der beleuchtungsunabhängigen Objektfarbe, während deren Wichtungsfaktoren als Eingabe für Shape-from-Shading-Verfahren zur Oberflächenrekonstruktion dienen.
Time-Dependent Joint Probability Speed Function for Level-Set Segmentation of Rat-Brain Slices
(2008)
The segmentation of rat brain slices suffers from illumination inhomogeneities and staining effects. State-of-the-art level-set methods model slice and background with intensity mixture densities defining the speed function as difference between the respective probabilites. Nevertheless, the overlap of these distributions causes an inaccurate stopping at the slice border. In this work, we propose the characterisation of the border area with intensity pairs for inside and outside estimating joint intensity probabilities. Method - In contrast to global object and background models, we focus on the object border characterised by a joint mixture density. This specifies the probability of the occurance of an inside and an outside value in direct adjacency. These values are not known beforehand, because inside and outside depend on the level-set evolution and change during time. Therefore, the speed function is computed time-dependently at the position of the current zero level-set. Along this zero level-set curve, the inside and outside values are derived as mean along the curvature normal directing inside and outside the object. Advantage of the joint probability distribution is to resolve the distribution overlaps, because these are assumed to be not located at the same border position. Results - The novel time-dependent joint probability based speed function is compared expermimentally with single probability based speed functions. Two rat brains with about 40 slices are segmented and the results analysed using manual segmentations and the Tanimoto overlap measure. Improved results are recognised for both data sets.
Laryngoscopic images of the vocal tract are used for diagnostic purposes. Quantitative mea-surements like changes of the glottis size or the surface of the vocal cords during an image sequence can be helpful to describe the healing process or to compare the findings of diffe-rent patients. Typically the endoscopic images are circulary symmetric distorted (barrel di-stortion). Therefore measurements of geometric dimensions depend on the object´s position in the image. In this paper an algorithm is presented which allows the computation of the translational invariant "real" object size by correcting the image distortion without using additional calibration of the optical environment.
Zur Analyse von Lippenbewegungsabläufen wird ein aktives Konturmodell eingesetzt. Probleme bereitet die hohe Sprechgeschwindigkeit, die in star ken Objektverschiebungen result iert und bislang nicht durch eine alleinige Konturanpassung kompensiert werden kann. In diesem Beitrag werden die klassischen aktiven Konturmodelle um eine Vorjustierung der Grobkonturen erweitert, die eine energiebasierte Konturanpassung erst möglich macht. Die Schätzung der Verschiebung zur Vorjustierung basiert auf dem Gradientenbild und einem prädikatenlogisch formulierten Regelwerk, das Annahmen und Nebenbedingungen als Wissensbasis enthält. Mit Hilfe dieser Erweiterungen ist eine automatisierte Konturverfolgung der Lippen möglich.
Statistische Eigenschaften natürlicher Grauwerttexturen werden mit Co-Occurrence Matrizen, basierend auf der Grauwertstatistik zweiter Ordnung, modelliert. Die Matrix gibt dann die apriori Wahrscheinlichkeiten aller Grauwertpaare an. Da in der medizinischen Bildverarbeitung verstärkt Multispektralbilder ausgewertet werden, wird das bekannte Konzept hier auf beliebige Vektorbilder erweitert. Dadurch kann bei der Texturklassifikation die zur Verfügung stehende Information vollständig genutzt werden. Insbesondere zur Detektion von Farbtexturen ist dieser Ansatz geeignet, da Wertepaare unterschiedlicher Spektralebenen ausgewertet werden können. Ebenso kann die Methode auch bei der Multiskalendekomposition von Intensitätsbildern zur Verbesserung der Texturerkennung beitragen. Die in den Matrizen entstehenden Muster lassen dann über die Extraktion geeigneter Texturdeskriptoren Rückschlüsse auf die Texturen des Bildes zu.
Bestimmung der Lichtquellenfarbe bei der Endoskopie mikrotexturierter Oberflächen des Kehlkopfes
(1999)
Zur Unterstützung der Diagnose von Stimmlippenerkrankungen werden innerhalb des Forschungsprojektes Quantitative Digitale Laryngoskopie objektive Parameter zur Beschreibung der Bewegung, der Farbe sowie der Form der Stimmlippen entwickelt und klinisch evaluiert. Während die Bewegungsanalyse Aufschluß über funktionelle Stimmstörungen gibt, beschreiben Parameter der Farb- und Formanalyse morphologische Veränderungen des Stimmlippengewebes. In diesem Beitrag werden die Methoden und bisherigen Ergebnisse zur Bewegungs- und Farbanalyse vorgestellt.
Die Bewegungsanalyse wurde mit einem erweiterten Konturmodell (Snakes) durchgeführt. Aufgrund des modifizierten Konturmodells konnten die Konturen der Stimmlippen automatisch über die gesmate Bildsequenz zuverlässig detektiert werden. Die Vermssung der Konturen liefert neue quantitative Parameter zur Befundung von laryngoskopischen Stimmlippenaufnahmen.
Um die Farbeigenschaften der Stimmlippen zu bestimmen, wurde ausgehend vom RGB-Bild die Objektfarbe unabhängig von der Farbe der Lichtquelle durch Verwendung von Clusterverfahren und der Viertelkreisanalyse berechnet. Mit dieser Farbanalyse konnte die Farbe der Lichtquelle ermittelt und das beleuchtungsunabhängige Farbbild berechnet werden. Die Quanitifizierung der Rötung der Stimmlippen ist z.B. ein entscheidendes Kriterium zur Diagnostik der akuten Laryngitis.
Color Texture Analysis of Moving Vocal Cords Using Approaches from Statistics and Signal Theory
(2000)
Textural features are applied for detection of morphological pathologies of vocal cords. Cooccurrence matrices as statistical features are presented as well as filter bank analysis by Gabor filters. Both methods are extended to handle color images. Their robustness against camera movement and vibration of vocal cords is evaluated. Classification results due to three in vivo sequences are in between 94.4 % and 98.9%. The classification errors decrease if color features are used instead of grayscale features for both statistical and Fourier features
A method for automated segmentation of vocal cords in stroboscopic video sequences is presented.
In contrast to earlier approaches, the inner and outer contours of the vocal cords are independently delineated. Automatic segmentation of the low contrasted images is carried out by connecting the shape constraint of a point distribution model to a multi-channel regionbased balloon model. This enables us to robustly compute a vibration profile that is used as a new diagnostic tool to visualize several vibration parameters in only one graphic. The vibration profiles are studied in two cases: one physiological vibration and one functional pathology.
Objective:
Complex hue/saturation images as a new approach for color texture classification using Gabor filters are introduced and compared with common techniques.
Method:
The interpretation of hue and saturationas polar coordinates allows direct use of the HSV-colorspace for Fourier transform. This technique is applied for Gabor feature extraction of color textures. In contrast to other color features based on the RGB-colorspace [1] the combination of color bands is done previous to the filtering.
Results:
The performance of the new HS-featuresis compared with that of RGB based as well as grayscale Gabor features by evaluating the classifi-cation of 30 natural textures. The new HS-featuresshow same results like the best RGB features but allow a more compact representation. On the averagethe color features improve the results of grayscale features.
Conclusion:
The consideration of the color information enhances the classification of color texture. The choice of colorspace cannot be adjudged finally, but the introduced features suggest the use of the HSV-colorspace with less features than RGB.
Fusion of histology and MRI is frequently demanded in biomedical research to study in vitro tissue properties in an in vivo reference space. Distortions and artifacts caused by cutting and staining of histological slices as well as differences in spatial resolution make even the rigid fusion a difficult task. State-of- the-art methods start with a mono-modal restacking yielding a histological pseudo-3D volume. The 3D information of the MRI reference is considered subsequently. However, consistency of the histology volume and consistency due to the corresponding MRI seem to be diametral goals. Therefore, we propose a novel fusion framework optimizing histology/histology and histology/MRI consistency at the same time finding a balance between both goals. Method - Direct slice-to-slice correspondence even in irregularly-spaced cutting sequences is achieved by registration-based interpolation of the MRI. Introducing a weighted multi-image mutual information metric (WI), adjacent histology and corresponding MRI are taken into account at the same time. Therefore, the reconstruction of the histological volume as well as the fusion with the MRI is done in a single step. Results - Based on two data sets with more than 110 single registrations in all, the results are evaluated quantitatively based on Tanimoto overlap measures and qualitatively showing the fused volumes. In comparison to other multi-image metrics, the reconstruction based on WI is significantly improved. We evaluated different parameter settings with emphasis on the weighting term steering the balance between intra- and inter-modality consistency.
Zur Lösung komplexer Segmentierungsprobleme wird eine hierarchische und farbbasierte Wasserscheidentransformation vorgestellt. Geringe Modifikationen bezüglich Startpunktwahl und Flutungsprozess resultieren in signifikanten Verbesserungen der Segmentierung. Das Verfahren wurde zur Lippendetektion in Farbbildsequenzen eingesetzt, die zur quantitativen Beschreibung von Sprechbewegungsabläufen automatisch ausgewertet werden. Die Experimente mit 245 Bildern aus 6 Sequenzen zeigten eine Fehlerrate von 13%.
3D rat brain tumors
(2005)
Polarised light imaging (PLI) utilises the birefringence of the myelin sheaths in order to visualise the orientation of nerve fibres in microtome sections of adult human post-mortem brains at ultra-high spatial resolution. The preparation of post-mortem brains for PLI involves fixation, freezing and cutting into 100-μm-thick sections. Hence, geometrical distortions of histological sections are inevitable and have to be removed for 3D reconstruction and subsequent fibre tracking. We here present a processing pipeline for 3D reconstruction of these sections using PLI derived multimodal images of post-mortem brains. Blockface images of the brains were obtained during cutting; they serve as reference data for alignment and elimination of distortion artefacts. In addition to the spatial image transformation, fibre orientation vectors were reoriented using the transformation fields, which consider both affine and subsequent non-linear registration. The application of this registration and reorientation approach results in a smooth fibre vector field, which reflects brain morphology. PLI combined with 3D reconstruction and fibre tracking is a powerful tool for human brain mapping. It can also serve as an independent method for evaluating in vivo fibre tractography.
Serial 2D section images with high resolution, resulting from innovative imaging methods become even more valuable, if they are fused with in vivo volumes. Achieving this goal, the 3D context of the sections would be restored, the deformations would be corrected and the artefacts would be eliminated. However, the registration in this field faces big challenges and is not solved in general. On the other hand, several approaches have been introduced dealing at least with some of these difficulties. Here, a brief overview of the topic is given and some of the solutions are presented. It does not constitute the claim to be a complete review, but could be a starting point for those who are interested in this field.
Integrative Co-occurrence matrices are introduced as novel features for color texture classification. The extended Co-occurrence notation allows the comparison between integrative and parallel color texture concepts. The information profit of the new matrices is shown quantitatively using the Kolmogorov distance and by extensive classification experiments on two datasets. Applying them to the RGB and the LUV color space the combined color and intensity textures are studied and the existence of intensity independent pure color patterns is demonstrated. The results are compared with two baselines: gray-scale texture analysis and color histogram analysis. The novel features improve the classification results up to 20% and 32% for the first and second baseline, respectively.
The field of AI is characterized by robust promises, astonishing successes, and remarkable breakthroughs. AI will play a major role in all domains of clinical medicine, but the role of AI in relation to the physician is not yet completely determined. The term artificial intelligence or AI is broad, and several different terms are used in this context that must be organized and demystified. This chapter will review the key concepts and methods of AI, and will introduce some of the different roles for AI in relation to the physician.
GinJinn: An object-detection pipeline for automated feature extraction from herbarium specimens
(2020)
PREMISE:
The generation of morphological data in evolutionary, taxonomic, and ecological studies of plants using herbarium material has traditionally been a labor-intensive task. Recent progress in machine learning using deep artificial neural networks (deep learning) for image classification and object detection has facilitated the establishment of a pipeline for the automatic recognition and extraction of relevant structures in images of herbarium specimens.
METHODS AND RESULTS:
We implemented an extendable pipeline based on state-of-the-art deep-learning object-detection methods to collect leaf images from herbarium specimens of two species of the genus Leucanthemum. Using 183 specimens as the training data set, our pipeline extracted one or more intact leaves in 95% of the 61 test images.
CONCLUSIONS:
We establish GinJinn as a deep-learning object-detection tool for the automatic recognition and extraction of individual leaves or other structures from herbarium specimens. Our pipeline offers greater flexibility and a lower entrance barrier than previous image-processing approaches based on hand-crafted features.
Metals are involved in many processes of life. They are needed for enzymatic reactions, are involved in healthy processes but also yield diseases if the metal homeostasis is disordered. Therefore, the interest to assess the spatial distribution of metals is rising in biomedical science. Imaging metal (and non-metal) isotopes by laser ablation mass spectrometry with inductively coupled plasma (LA-ICP-MS) requires a special software solution to process raw data obtained by scanning a sample line-by-line. As no software ready to use was available we developed an interactive software tool for Image Generation and Analysis (IMAGENA). Unless optimised for LA-ICP-MS, IMAGENA can handle other raw data as well. The general purpose was to reconstruct images from a continuous list of raw data points, to visualise these images, and to convert them into a commonly readable image file format that can be further analysed by standard image analysis software. The generation of the image starts with loading a text file that holds a data column of every measured isotope. Specifying general spatial domain settings like the data offset and the image dimensions is done by the user getting a direct feedback by means of a preview image. IMAGENA provides tools for calibration and to correct for a signal drift in the y-direction. Images are visualised in greyscale as well a pseudo-colours with possibilities for contrast enhancement. Image analysis is performed in terms of smoothed line plots in row and column direction.
The success of artificial intelligence in medicine is based on the need for large amounts of high quality training data. Sharing of medical image data, however, is often restricted by laws such as doctor-patient confidentiality. Although there are publicly available medical datasets, their quality and quantity are often low. Moreover, datasets are often imbalanced and only represent a fraction of the images generated in hospitals or clinics and can thus usually only be used as training data for specific problems. The introduction of generative adversarial networks (GANs) provides a mean to generate artificial images by training two convolutional networks. This paper proposes a method which uses GANs trained on medical images in order to generate a large number of artificial images that could be used to train other artificial intelligence algorithms. This work is a first step towards alleviating data privacy concerns and being able to publicly share data that still contains a substantial amount of the information in the original private data. The method has been evaluated on several public datasets and quantitative and qualitative tests showing promising results.
Local gray level dependencies of natural images can be modelled by means of co-occurrence matrices containing joint probabilities of gray-level pairs. Texture, however, is a resolution-dependent phenomenon and hence, classification depends on the chosen scale. Since there is no optimal scale for all textures we employ a multiscale approach that acquires textural features at several scales. Thus linear and nonlinear scale-spaces are analyzed by multiscale co-occurrence matrices that describe the statistical behavior of a texture in scale-space. Classification is then performed on the basis of texture features taken from the individual scale with the highest discriminatory power. By considering cross-scale occurrences of gray level pairs, the impact of filters on the feature is described and used for classification of natural textures. This novel method was found to improve classification rates of the common co-occurrence matrix approach on standard textures significantly.
Real-time computational speed and a high degree of precision are requirements for computer-assisted interventions. Applying a segmentation network to a medical video processing task can introduce significant inter-frame prediction noise. Existing approaches can reduce inconsistencies by including temporal information but often impose requirements on the architecture or dataset. This paper proposes a method to include temporal information in any segmentation model and, thus, a technique to improve video segmentation performance without alterations during training or additional labeling. With Motion-Corrected Moving Average, we refine the exponential moving average between the current and previous predictions. Using optical flow to estimate the movement between consecutive frames, we can shift the prior term in the moving-average calculation to align with the geometry of the current frame. The optical flow calculation does not require the output of the model and can therefore be performed in parallel, leading to no significant runtime penalty for our approach. We evaluate our approach on two publicly available segmentation datasets and two proprietary endoscopic datasets and show improvements over a baseline approach.
We investigate contrastive learning in a multi-task learning setting classifying and segmenting early Barrett’s cancer. How can contrastive learning be applied in a domain with few classes and low inter-class and inter-sample variance, potentially enabling image retrieval or image attribution? We introduce a data sampling strategy that mines per-lesion data for positive samples and keeps a queue of the recent projections as negative samples. We propose a masking strategy for the NT-Xent loss that keeps the negative set pure and removes samples from the same lesion. We show cohesion and uniqueness improvements of the proposed method in feature space. The introduction of the auxiliary objective does not affect the performance but adds the ability to indicate similarity between lesions. Therefore, the approach could enable downstream auto-documentation tasks on homogeneous medical image data.
Semantic segmentation is an essential task in medical imaging research. Many powerful deep-learning-based approaches can be employed for this problem, but they are dependent on the availability of an expansive labeled dataset. In this work, we augment such supervised segmentation models to be suitable for learning from unlabeled data. Our semi-supervised approach, termed Error-Correcting Mean-Teacher, uses an exponential moving average model like the original Mean Teacher but introduces our new paradigm of error correction. The original segmentation network is augmented to handle this secondary correction task. Both tasks build upon the core feature extraction layers of the model. For the correction task, features detected in the input image are fused with features detected in the predicted segmentation and further processed with task-specific decoder layers. The combination of image and segmentation features allows the model to correct present mistakes in the given input pair. The correction task is trained jointly on the labeled data. On unlabeled data, the exponential moving average of the original network corrects the student’s prediction. The combined outputs of the students’ prediction with the teachers’ correction form the basis for the semi-supervised update. We evaluate our method with the 2017 and 2018 Robotic Scene Segmentation data, the ISIC 2017 and the BraTS 2020 Challenges, a proprietary Endoscopic Submucosal Dissection dataset, Cityscapes, and Pascal VOC 2012. Additionally, we analyze the impact of the individual components and examine the behavior when the amount of labeled data varies, with experiments performed on two distinct segmentation architectures. Our method shows improvements in terms of the mean Intersection over Union over the supervised baseline and competing methods. Code is available at https://github.com/CloneRob/ECMT.
We propose an automatic approach for early detection of adenocarcinoma in the esophagus. High-definition endoscopic images (50 cancer, 50 Barrett) are partitioned into a dataset containing approximately equal amounts of patches showing cancerous and non-cancerous regions. A deep convolutional neural network is adapted to the data using a transfer learning approach. The final classification of an image is determined by at least one patch, for which the probability being a cancer patch exceeds a given threshold. The model was evaluated with leave one patient out cross-validation. With sensitivity and specificity of 0.94 and 0.88, respectively, our findings improve recently published results on the same image data base considerably. Furthermore, the visualization of the class probabilities of each individual patch indicates, that our approach might be extensible to the segmentation domain.
Pixel-level classification is an essential part of computer vision. For learning from labeled data, many powerful deep learning models have been developed recently. In this work, we augment such supervised segmentation models by allowing them to learn from unlabeled data. Our semi-supervised approach, termed Error-Correcting Supervision, leverages a collaborative strategy. Apart from the supervised training on the labeled data, the segmentation network is judged by an additional network.
Pixel-level classification is an essential part of computer vision. For learning from labeled data, many powerful deep learning models have been developed recently. In this work, we augment such supervised segmentation models by allowing them to learn from unlabeled data. Our semi-supervised approach, termed Error-Correcting Supervision, leverages a collaborative strategy. Apart from the supervised training on the labeled data, the segmentation network is judged by an additional network. The secondary correction network learns on the labeled data to optimally spot correct predictions, as well as to amend incorrect ones. As auxiliary regularization term, the corrector directly influences the supervised training of the segmentation network. On unlabeled data, the output of the correction network is essential to create a proxy for the unknown truth. The corrector’s output is combined with the segmentation network’s prediction to form the new target. We propose a loss function that incorporates both the pseudo-labels as well as the predictive certainty of the correction network. Our approach can easily be added to supervised segmentation models. We show consistent improvements over a supervised baseline on experiments on both the Pascal VOC 2012 and the Cityscapes datasets with varying amounts of labeled data.
Aims
Barrett´s esophagus related neoplasia (BERN) is difficult to detect and characterize during endoscopy, even for expert endoscopists. We aimed to assess the add-on effect of an Artificial Intelligence (AI) algorithm (Barrett-Ampel) as a decision support system (DSS) for non-expert endoscopists in the evaluation of Barrett’s esophagus (BE) and BERN.
Methods
Twelve videos with multimodal imaging white light (WL), narrow-band imaging (NBI), texture and color enhanced imaging (TXI) of histologically confirmed BE and BERN were assessed by expert and non-expert endoscopists. For each video, endoscopists were asked to identify the area of BERN and decide on the biopsy spot. Videos were assessed by the AI algorithm and regions of BERN were highlighted in real-time by a transparent overlay. Finally, endoscopists were shown the AI videos and asked to either confirm or change their initial decision based on the AI support.
Results
Barrett-Ampel correctly identified all areas of BERN, irrespective of the imaging modality (WL, NBI, TXI), but misinterpreted two inflammatory lesions (Accuracy=75%). Expert endoscopists had a similar performance (Accuracy=70,8%), while non-experts had an accuracy of 58.3%. When AI was implemented as a DSS, non-expert endoscopists improved their diagnostic accuracy to 75%.
Conclusions
AI may have the potential to support non-expert endoscopists in the assessment of videos of BE and BERN. Limitations of this study include the low number of videos used. Randomized clinical trials in a real-life setting should be performed to confirm these results.
Einleitung
Die sichere Detektion und Charakterisierung von Barrett-Ösophagus assoziierten Neoplasien (BERN) stellt selbst für erfahrene Endoskopiker eine Herausforderung dar.
Ziel
Ziel dieser Studie ist es, den Add-on Effekt eines künstlichen Intelligenz (KI) Systems (Barrett-Ampel) als Entscheidungsunterstüzungssystem für Endoskopiker ohne Expertise bei der Untersuchung von BERN zu evaluieren.
Material und Methodik
Zwölf Videos in „Weißlicht“ (WL), „narrow-band imaging“ (NBI) und „texture and color enhanced imaging“ (TXI) von histologisch bestätigten Barrett-Metaplasien oder BERN wurden von Experten und Untersuchern ohne Barrett-Expertise evaluiert. Die Probanden wurden dazu aufgefordert in den Videos auftauchende BERN zu identifizieren und gegebenenfalls die optimale Biopsiestelle zu markieren. Unser KI-System wurde demselben Test unterzogen, wobei dieses BERN in Echtzeit segmentierte und farblich von umliegendem Epithel differenzierte. Anschließend wurden den Probanden die Videos mit zusätzlicher KI-Unterstützung gezeigt. Basierend auf dieser neuen Information, wurden die Probanden zu einer Reevaluation ihrer initialen Beurteilung aufgefordert.
Ergebnisse
Die „Barrett-Ampel“ identifizierte unabhängig von den verwendeten Darstellungsmodi (WL, NBI, TXI) alle BERN. Zwei entzündlich veränderte Läsionen wurden fehlinterpretiert (Genauigkeit=75%). Während Experten vergleichbare Ergebnisse erzielten (Genauigkeit=70,8%), hatten Endoskopiker ohne Expertise bei der Beurteilung von Barrett-Metaplasien eine Genauigkeit von lediglich 58,3%. Wurden die nicht-Experten allerdings von unserem KI-System unterstützt, erreichten diese eine Genauigkeit von 75%.
Zusammenfassung
Unser KI-System hat das Potential als Entscheidungsunterstützungssystem bei der Differenzierung zwischen Barrett-Metaplasie und BERN zu fungieren und so Endoskopiker ohne entsprechende Expertise zu assistieren. Eine Limitation dieser Studie ist die niedrige Anzahl an eingeschlossenen Videos. Um die Ergebnisse dieser Studie zu bestätigen, müssen randomisierte kontrollierte klinische Studien durchgeführt werden.
Aims
Evaluation of the add-on effect an artificial intelligence (AI) based clinical decision support system has on the performance of endoscopists with different degrees of expertise in the field of Barrett's esophagus (BE) and Barrett's esophagus-related neoplasia (BERN).
Methods
The support system is based on a multi-task deep learning model trained to solve a segmentation and several classification tasks. The training approach represents an extension of the ECMT semi-supervised learning algorithm. The complete system evaluates a decision tree between estimated motion, classification, segmentation, and temporal constraints, to decide when and how the prediction is highlighted to the observer. In our current study, ninety-six video cases of patients with BE and BERN were prospectively collected and assessed by Barrett's specialists and non-specialists. All video cases were evaluated twice – with and without AI assistance. The order of appearance, either with or without AI support, was assigned randomly. Participants were asked to detect and characterize regions of dysplasia or early neoplasia within the video sequences.
Results
Standalone sensitivity, specificity, and accuracy of the AI system were 92.16%, 68.89%, and 81.25%, respectively. Mean sensitivity, specificity, and accuracy of expert endoscopists without AI support were 83,33%, 58,20%, and 71,48 %, respectively. Gastroenterologists without Barrett's expertise but with AI support had a comparable performance with a mean sensitivity, specificity, and accuracy of 76,63%, 65,35%, and 71,36%, respectively.
Conclusions
Non-Barrett's experts with AI support had a similar performance as experts in a video-based study.
Einleitung
Die Differenzierung zwischen nicht dysplastischem Barrett-Ösophagus (NDBE) und mit Barrett-Ösophagus assoziierten Neoplasien (BERN) während der endoskopischen Inspektion erfordert viel Expertise. Die frühe Diagnosestellung ist wichtig für die weitere Prognose des Barrett-Karzinoms. In Deutschland werden Patient:innen mit einem Barrett-Ösophagus (BE) in der Regel im niedergelassenen Sektor überwacht.
Ziele
Ziel ist es, den Einfluss von einem auf Künstlicher Intelligenz (KI) basierenden klinischen Entscheidungsunterstützungssystems (CDSS) auf die Performance von niedergelassenen Gastroenterolog:innen (NG) bei der Evaluation von Barrett-Ösophagus (BE) zu untersuchen.
Methodik
Es erfolgte die prospektive Sammlung von 96 unveränderten hochauflösenden Videos mit Fällen von Patient:innen mit histologisch bestätigtem NDBE und BERN. Alle eingeschlossenen Fälle enthielten mindestens zwei der folgenden Darstellungsmethoden: HD-Weißlichtendoskopie, Narrow Band Imaging oder Texture and Color Enhancement Imaging. Sechs NG von sechs unterschiedlichen Praxen wurden als Proband:innen eingeschlossen. Es erfolgte eine permutierte Block-Randomisierung der Videofälle in entweder Gruppe A oder Gruppe B. Gruppe A implizierte eine Evaluation des Falls durch Proband:innen zunächst ohne KI und anschließend mit KI als CDSS. In Gruppe B erfolgte die Evaluation in umgekehrter Reihenfolge. Anschließend erfolgte eine zufällige Wiedergabe der so entstandenen Subgruppen im Rahmen des Tests.
Ergebnis
In diesem Test konnte ein von uns entwickeltes KI-System (Barrett-Ampel) eine Sensitivität von 92,2%, eine Spezifität von 68,9% und eine Accuracy von 81,3% erreichen. Mit der Hilfe von KI verbesserte sich die Sensitivität der NG von 64,1% auf 71,2% (p<0,001) und die Accuracy von 66,3% auf 70,8% (p=0,006) signifikant. Eine signifikante Verbesserung dieser Parameter zeigte sich ebenfalls, wenn die Proband:innen die Fälle zunächst ohne KI evaluierten (Gruppe A). Wurde der Fall jedoch als Erstes mit der Hilfe von KI evaluiert (Gruppe B), blieb die Performance nahezu konstant.
Schlussfolgerung
Es konnte ein performantes KI-System zur Evaluation von BE entwickelt werden. NG verbessern sich bei der Evaluation von BE durch den Einsatz von KI.
Barrett-Ampel
(2022)
Hintergrund
Adenokarzinome des Ösophagus sind bis heute mit einer infausten Prognose vergesellschaftet (1). Obwohl Endoskopiker mit Barrett-Ösophagus als Präkanzerose konfrontiert werden, ist vor allem für nicht-Experten die Differenzierung zwischen Barrett-Ösophagus ohne Dysplasie und assoziierten Neoplasien mitunter schwierig. Existierende Biopsieprotokolle (z.B. Seattle Protokoll) sind oftmals unzuverlässig (2). Eine frühzeitige Diagnose des Adenokarzinoms ist allerdings von fundamentaler Bedeutung für die Prognose des Patienten.
Forschungsansatz
Auf der Grundlage dieser Problematik, entwickelten wir in Kooperation mit dem Forschungslabor „Regensburg Medical Image Computing (ReMIC)“ der OTH Regensburg ein auf künstlicher Intelligenz (KI) basiertes Entscheidungsunterstützungssystem (CDSS). Das auf einer DeepLabv3+ neuronalen Netzwerkarchitektur basierende CDSS differenziert mittels Mustererkennung Barrett- Ösophagus ohne Dysplasie von Barrett-Ösophagus mit Dysplasie bzw. Neoplasie („Klassifizierung“). Hierbei werden gemittelte Ausgabewahrscheinlichkeiten mit einem vom Benutzer definierten Schwellenwert verglichen. Für Vorhersagen, die den Schwellenwert überschreiten, berechnen wir die Kontur der Region und die Fläche. Sobald die vorhergesagte Läsion eine bestimmte Größe in der Eingabe überschreitet, heben wir sie und ihren Umriss hervor. So ermöglicht eine farbkodierte Visualisierung eine Abgrenzung zwischen Dysplasie bzw. Neoplasie und normalem Barrett-Epithel („Segmentierung“).
In einer Studie an Bildern in „Weißlicht“ (WL) und „Narrow Band Imaging“ (NBI) demonstrierten wir eine Sensitivität von mehr als 90% und eine Spezifität von mehr als 80% (3). In einem nächsten Schritt, differenzierte unser KI-Algorithmus Barrett- Metaplasien von assoziierten Neoplasien anhand von zufällig abgegriffenen Bildern in Echtzeit mit einer Accuracy von 89.9% (4). Darauf folgend, entwickelten wir unser System dahingehend weiter, dass unser Algorithmus nun auch dazu in der Lage ist, Untersuchungsvideos in WL, NBI und „Texture and Color Enhancement Imaging“ (TXI) in Echtzeit zu analysieren (5).
Aktuell führen wir eine Studie in einem randomisiert-kontrollierten Ansatz an unveränderten Untersuchungsvideos in WL, NBI und TXI durch.
Ausblick
Um Patienten mit aus Barrett-Metaplasien resultierenden Neoplasien frühestmöglich an „High-Volume“-Zentren überweisen zu können, soll unser KI-Algorithmus zukünftig vor allem Endoskopiker ohne extensive Erfahrung bei der Beurteilung von Barrett- Ösophagus in der Krebsfrüherkennung unterstützen.
Einleitung
Übermäßige Bewegung im Bild kann die Performance von auf künstlicher Intelligenz (KI) basierenden klinischen Entscheidungsunterstützungssystemen (CDSS) reduzieren. Optical Flow (OF) ist eine Methode zur Lokalisierung und Quantifizierung von Bewegungen zwischen aufeinanderfolgenden Bildern.
Ziel
Ziel ist es, die Mensch-Computer-Interaktion (HCI) zu verbessern und Endoskopiker die unser KI-System „Barrett-Ampel“ zur Unterstützung bei der Beurteilung von Barrett-Ösophagus (BE) verwenden, ein Echtzeit-Feedback zur aktuellen Datenqualität anzubieten.
Methodik
Dazu wurden unveränderte Videos in „Weißlicht“ (WL), „Narrow Band Imaging“ (NBI) und „Texture and Color Enhancement Imaging“ (TXI) von acht endoskopischen Untersuchungen von histologisch gesichertem BE und mit Barrett-Ösophagus assoziierten Neoplasien (BERN) durch unseren KI-Algorithmus analysiert. Der zur Bewertung der Bildqualität verwendete OF beinhaltete die mittlere Magnitude und die Entropie des Histogramms der Winkel. Frames wurden automatisch extrahiert, wenn die vordefinierten Schwellenwerte von 3,0 für die mittlere Magnitude und 9,0 für die Entropie des Histogramms der Winkel überschritten wurden. Experten sahen sich zunächst die Videos ohne KI-Unterstützung an und bewerteten, ob Störfaktoren die Sicherheit mit der eine Diagnose im vorliegenden Fall gestellt werden kann negativ beeinflussen. Anschließend überprüften sie die extrahierten Frames.
Ergebnis
Gleichmäßige Bewegung in eine Richtung, wie etwa beim Vorschieben des Endoskops, spiegelte sich, bei insignifikant veränderter Entropie, in einer Erhöhung der Magnitude wider. Chaotische Bewegung, zum Beispiel während dem Spülen, war mit erhöhter Entropie assoziiert. Insgesamt war eine unruhige endoskopische Darstellung, Flüssigkeit sowie übermäßige Ösophagusmotilität mit erhöhtem OF assoziiert und korrelierte mit der Meinung der Experten über die Qualität der Videos. Der OF und die subjektive Wahrnehmung der Experten über die Verwertbarkeit der vorliegenden Bildsequenzen korrelierten direkt proportional. Wenn die vordefinierten Schwellenwerte des OF überschritten wurden, war die damit verbundene Bildqualität in 94% der Fälle für eine definitive Interpretation auch für Experten unzureichend.
Schlussfolgerung
OF hat das Potenzial Endoskopiker ein Echtzeit-Feedback über die Qualität des Dateninputs zu bieten und so nicht nur die HCI zu verbessern, sondern auch die optimale Performance von KI-Algorithmen zu ermöglichen.
Effect of AI on performance of endoscopists to detect Barrett neoplasia: A Randomized Tandem Trial
()
Background and study aims
To evaluate the effect of an AI-based clinical decision support system (AI) on the performance and diagnostic confidence of endoscopists during the assessment of Barrett's esophagus (BE).
Patients and Methods
Ninety-six standardized endoscopy videos were assessed by 22 endoscopists from 12 different centers with varying degrees of BE experience.
The assessment was randomized into two video sets: Group A (review first without AI and second with AI) and group B (review first with AI and second without AI). Endoscopists were required to evaluate each video for the presence of Barrett's esophagus-related neoplasia (BERN) and then decide on a spot for a targeted biopsy. After the second assessment, they were allowed to change their clinical decision and confidence level.
Results
AI had a standalone sensitivity, specificity, and accuracy of 92.2%, 68.9%, and 81.6%, respectively. Without AI, BE experts had an overall sensitivity, specificity, and accuracy of 83.3%, 58.1 and 71.5%, respectively. With AI, BE nonexperts showed a significant improvement in sensitivity and specificity when videos were assessed a second time with AI (sensitivity 69.7% (95% CI, 65.2% - 74.2%) to 78.0% (95% CI, 74.0% - 82.0%); specificity 67.3% (95% CI, 62.5% - 72.2%) to 72.7% (95 CI, 68.2% - 77.3%). In addition, the diagnostic confidence of BE nonexperts improved significantly with AI.
Conclusion
BE nonexperts benefitted significantly from the additional AI. BE experts and nonexperts remained below the standalone performance of AI, suggesting that there may be other factors influencing endoscopists to follow or discard AI advice.
Background
This study evaluated the effect of an artificial intelligence (AI)-based clinical decision support system on the performance and diagnostic confidence of endoscopists in their assessment of Barrett’s esophagus (BE).
Methods
96 standardized endoscopy videos were assessed by 22 endoscopists with varying degrees of BE experience from 12 centers. Assessment was randomized into two video sets: group A (review first without AI and second with AI) and group B (review first with AI and second without AI). Endoscopists were required to evaluate each video for the presence of Barrett’s esophagus-related neoplasia (BERN) and then decide on a spot for a targeted biopsy. After the second assessment, they were allowed to change their clinical decision and confidence level.
Results
AI had a stand-alone sensitivity, specificity, and accuracy of 92.2%, 68.9%, and 81.3%, respectively. Without AI, BE experts had an overall sensitivity, specificity, and accuracy of 83.3%, 58.1%, and 71.5%, respectively. With AI, BE nonexperts showed a significant improvement in sensitivity and specificity when videos were assessed a second time with AI (sensitivity 69.8% [95%CI 65.2%–74.2%] to 78.0% [95%CI 74.0%–82.0%]; specificity 67.3% [95%CI 62.5%–72.2%] to 72.7% [95%CI 68.2%–77.3%]). In addition, the diagnostic confidence of BE nonexperts improved significantly with AI.
Conclusion
BE nonexperts benefitted significantly from additional AI. BE experts and nonexperts remained significantly below the stand-alone performance of AI, suggesting that there may be other factors influencing endoscopists’ decisions to follow or discard AI advice.
Laser ablation inductively coupled plasma mass spectrometry (LA-ICP-MS) has been established as a powerful technique for the determination of metal and nonmetal distributions within biological systems with high sensitivity. An imaging LA-ICP-MS technique for Fe, Cu, Zn, and Mn was developed to produce large series of quantitative element maps in native brain sections of mice subchronically intoxicated with 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridin (MPTP) as a model of Parkinson’s disease. Images were calibrated using matrix-matched laboratory standards. A software solution allowing a precise delineation of anatomical structures was implemented. Coronal brain sections were analyzed crossing the striatum and the substantia nigra, respectively. Animals sacrificed 2 h, 7 d, or 28 d after the last MPTP injection and controls were investigated.
We observed significant decreases of Cu concentrations in the periventricular zone and the fascia dentata at 2 h and 7d and a recovery or overcompensation at 28 d, most pronounced in the rostral periventricular zone (+40%). In the cortex Cu decreased slightly to −10%. Fe increased in the interpeduncular nucleus (+40%) but not in the substantia nigra. This pattern is in line with a differential regulation of periventricular and parenchymal Cu, and with the histochemical localization of Fe, and congruent to regions of preferential MPTP binding described in the rodent brain.
The LA-ICP-MS technique yielded valid and statistically robust results in the present study on 39 slices from 19 animals. Our findings underline the value of routine micro-local analytical techniques in the life sciences and affirm a role of Cu availability in Parkinson’s disease.
Object
The consistency of parametric registration in multi-temporal magnetic resonance (MR) imaging studies was evaluated.
Materials and methods
Serial MRI scans of adult patients with a brain tumor (glioma) were aligned by parametric registration. The performance of low-order spatial alignment (6/9/12 degrees of freedom) of different 3D serial MR-weighted images is evaluated. A registration protocol for the alignment of all images to one reference coordinate system at baseline is presented. Registration results were evaluated for both, multimodal intra-timepoint and mono-modal multi-temporal registration. The latter case might present a challenge to automatic intensity-based registration algorithms due to ill-defined correspondences. The performance of our algorithm was assessed by testing the inverse registration consistency. Four different similarity measures were evaluated to assess consistency.
Results
Careful visual inspection suggests that images are well aligned, but their consistency may be imperfect. Sub-voxel inconsistency within the brain was found for allsimilarity measures used for parametric multi-temporal registration. T1-weighted images were most reliable for establishing spatial correspondence between different timepoints.
Conclusions
The parametric registration algorithm is feasible for use in this application. The sub-voxel resolution mean displacement error of registration transformations demonstrates that the algorithm converges to an almost identical solution for forward and reverse registration.
Background: Currently, it is common practice to use three-dimensional (3D) printers not only for rapid prototyping in the industry, but also in the medical area to create medical applications for training inexperienced surgeons. In a clinical training simulator for minimally invasive bone drilling to fix hand fractures with Kirschner-wires (K-wires), a 3D-printed hand phantom must not only be geometrically but also haptically correct. Due to a limited view during an operation, surgeons need to perfectly localize underlying risk structures only by feeling of specific bony protrusions of the human hand.
Methods: The goal of this experiment is to imitate human soft tissue with its haptic and elasticity for a realistic hand phantom fabrication, using only a dual-material 3D printer and support-material-filled metamaterial between skin and bone. We present our workflow to generate lattice structures between hard bone and soft skin with iterative cube edge (CE) or cube face (CF) unit cells. Cuboid and finger shaped sample prints with and without inner hard bone in different lattice thickness are constructed and 3D printed.
Results: The most elastic available rubber-like material is too firm to imitate soft tissue. By reducing the amount of rubber in the inner volume through support material (SUP), objects become significantly softer. Without metamaterial, after disintegration, the SUP can be shifted through the volume and thus the body loses its original shape. Although the CE design increases the elasticity, it cannot restore the fabric form. In contrast to CE, the CF design increases not only the elasticity but also guarantees a local limitation of the SUP. Therefore, the body retains its shape and internal bones remain in its intended place. Various unit cell sizes, lattice thickening and skin thickness regulate the rubber material and SUP ratio. Test prints with higher SUP and lower rubber material percentage appear softer and vice versa. This was confirmed by an expert surgeon evaluation. Subjects adjudged pure rubber-like material as too firm and samples only filled with SUP or lattice structure in CE design as not suitable for imitating tissue. 3D-printed finger samples in CF design were rated as realistic compared to the haptic of human tissue with a good palpable bone structure.
Conclusions: We developed a new dual-material 3D print technique to imitate soft tissue of the human hand with its haptic properties. Blowy SUP is trapped within a lattice structure to soften rubber-like 3D print material, which makes it possible to reproduce a realistic replica of human hand soft tissue.
Currently, it is common practice to use three-dimensional (3D) printers not only for rapid prototyping in the industry, but also in the medical area to create medical applications for training inexperienced surgeons. In a clinical training simulator for minimally invasive bone drilling to fix hand fractures with Kirschner-wires (K-wires), a 3D printed hand phantom must not only be geometrically but also haptically correct. Due to a limited view during an operation, surgeons need to perfectly localize underlying risk structures only by feeling of specific bony protrusions of the human hand.
Background: For surgical fixation of bone fractures of the human hand, so-called Kirschner-wires (K-wires) are drilled through bone fragments. Due to the minimally invasive drilling procedures without a view of risk structures like vessels and nerves, a thorough training of young surgeons is necessary. For the development of a virtual reality (VR) based training system, a three-dimensional (3D) printed phantom hand is required. To ensure an intuitive operation, this phantom hand has to be realistic in both, its position relative to the driller as well as in its haptic features. The softest 3D printing material available on the market, however, is too hard to imitate human soft tissue. Therefore, a support-material (SUP) filled metamaterial is used to soften the raw material. Realistic haptic features are important to palpate protrusions of the bone to determine the drilling starting point and angle. An optical real-time tracking is used to transfer position and rotation to the training system.
Methods: A metamaterial already developed in previous work is further improved by use of a new unit cell. Thus, the amount of SUP within the volume can be increased and the tissue is softened further. In addition, the human anatomy is transferred to the entire hand model. A subcutaneous fat layer and penetration of air through pores into the volume simulate shiftability of skin layers. For optical tracking, a rotationally symmetrical marker attached to the phantom hand with corresponding reference marker is developed. In order to ensure trouble-free position transmission, various types of marker point applications are tested.
Results: Several cuboid and forearm sample prints lead to a final 30 centimeter long hand model. The whole haptic phantom could be printed faultless within about 17 hours. The metamaterial consisting of the new unit cell results in an increased SUP share of 4.32%. Validated by an expert surgeon study, this allows in combination with a displacement of the uppermost skin layer a good palpability of the bones. Tracking of the hand marker in dodecahedron design works trouble-free in conjunction with a reference marker attached to the worktop of the training system.
Conclusions: In this work, an optically tracked and haptically correct phantom hand was developed using dual-material 3D printing, which can be easily integrated into a surgical training system.
One common method to fix fractures of the human hand after an accident is an osteosynthesis with Kirschner wires (K-wires) to stabilize the bone fragments. The insertion of K-wires is a delicate minimally invasive surgery, because surgeons operate almost without a sight. Since realistic training methods are time consuming, costly and insufficient, a virtual-reality (VR) based training system for the placement of K-wires was developed. As part of this, the current work deals with the real-time bone drilling simulation using a haptic force-feedback device.
To simulate the drilling, we introduce a virtual fixture based force-feedback drilling approach. By decomposition of the drilling task into individual phases, each phase can be handled individually to perfectly control the drilling procedure. We report about the related finite state machine (FSM), describe the haptic feedback of each state and explain, how to avoid jerking of the haptic force-feedback during state transition.
The usage of the virtual fixture approach results in a good haptic performance and a stable drilling behavior. This was confirmed by 26 expert surgeons, who evaluated the virtual drilling on the simulator and rated it as very realistic. To make the system even more convincing, we determined real drilling feed rates through experimental pig bone drilling and transferred them to our system. Due to a constant simulation thread we can guarantee a precise drilling motion.
Virtual fixtures based force-feedback calculation is able to simulate force-feedback assisted bone drilling with high quality and, thus, will have a great potential in developing medical applications.
In order to fix a fracture using minimally invasive surgery approaches, surgeons are drilling complex and tiny bones with a 2 dimensional X-ray as single imaging modality in the operating room. Our novel haptic force-feedback and visual assisted training system will potentially help hand surgeons to learn the drilling procedure in a realistic visual environment. Within the simulation, the collision detection as well as the interaction between virtual drill, bone voxels and surfaces are important. In this work, the chai3d collision detection and force calculation algorithms are combined with a physics engine to simulate the bone drilling process. The chosen Bullet-Physics-Engine provides a stable simulation of rigid bodies, if the collision model of the drill and the tool holder is generated as a compound shape. Three haptic points are added to the K-wire tip for removing single voxels from the bone. For the drilling process three modes are proposed to emulate the different phases of drilling in restricting the movement of a haptic device.
The German workshop on medical image computing (BVM) has been held in different locations in Germany for more than 20 years. In terms of content, BVM focused on the computer-aided analysis of medical image data with a wide range of applications, e.g. in the area of imaging, diagnostics, operation planning, computer-aided intervention and visualization.
During this time, there have been remarkable methodological developments and upheavals, on which the BVM community has worked intensively. The area of machine learning should be emphasized, which has led to significant improvements, especially for tasks of classification and segmentation, but increasingly also in image formation and registration. As a result, work in connection with deep learning now dominates the BVM. These developments have also contributed to the establishment of medical image processing at the interface between computer science and medicine as one of the key technologies for the digitization of the health system.
In addition to the presentation of current research results, a central aspect of the BVM is primarily the promotion of young scientists from the diverse BVM community, covering not only Germany but also Austria, Switzerland, The Netherland and other European neighbors. The conference serves primarily doctoral students and postdocs, but also students with excellent bachelor and master theses as a platform to present their work, to enter into professional discourse with the community, and to establish networks with specialist colleagues. Despite the many conferences and congresses that are also relevant for medical image processing, the BVM has therefore lost none of its importance and attractiveness and has retained its permanent place in the annual conference rhythm.
Building on this foundation, there are some innovations and changes this year. The BVM 2021 was organized for the first time at the Ostbayerische Technische Hochschule Regensburg (OTH Regensburg, a technical university of applied sciences). After Aachen, Berlin, Erlangen, Freiburg, Hamburg, Heidelberg, Leipzig, Lübeck, and Munich, Regensburg is not just a new venue. OTH Regensburg is the first representative of the universities of applied sciences (HAW) to organize the conference, which differs to universities, university hospitals, or research centers like Fraunhofer or Helmholtz. This also considers the further development of the research landscape in Germany, where HAWs increasingly contribute to applied research in addition to their focus on teaching. This development is also reflected in the contributions submitted to the BVM in recent years.
At BVM 2021, which was held in a virtual format for the first time due to the Corona pandemic, an attractive and high-quality program was offered. Fortunately, the number of submissions increased significantly. Out of 97 submissions, 26 presentations, 51 posters and 5 software demonstrations were accepted via an anonymized reviewing process with three reviews each. The three best works have been awarded BVM prizes, selected by a separate committee.
Based on these high-quality submissions, we are able to present another special issue in the International Journal of Computer Assisted Radiology and Surgery (IJCARS). Out of the 97 submissions, the ones with the highest scores have been invited to submit an extended version of their paper to be presented in IJCARS. As a result, we are now able to present this special issue with seven excellent articles. Many submissions focus on machine learning in a medical context.
The estimation of illuminant color is mandatory for many applications in the field of color image quantification. However, it is an unresolved problem if no additional heuristics or restrictive assumptions apply. Assuming uniformly colored and roundly shaped objects, Lee has presented a theory and a method for computing the scene-illuminant chromaticity from specular highlights [H. C. Lee, J. Opt. Soc. Am. A 3, 1694 (1986)]. However, Lee’s method, called image path search, is less robust to noise and is limited in the handling of microtextured surfaces. We introduce a novel approach to estimate the color of a single illuminant for noisy and microtextured images, which frequently occur in real-world scenes. Using dichromatic regions of different colored surfaces, our approach, named color line search, reverses Lee’s strategy of image path search. Reliable color lines are determined directly in the domain of the color diagrams by three steps. First, regions of interest are automatically detected around specular highlights, and local color diagrams are computed. Second, color lines are determined according to the dichromatic reflection model by Hough transform of the color diagrams. Third, a consistency check is applied by a corresponding path search in the image domain. Our method is evaluated on 40 natural images of fruit and vegetables. In comparison with those of Lee’s method, accuracy and stability are substantially improved. In addition, the color line search approach can easily be extended to scenes of objects with macrotextured surfaces.
Einleitung:
Blutdruck gilt als sogenannter Vitalparameter als einer der grundlegenden Indikatoren für den Gesundheitszustand einer Person. Sowohl zu niedriger als auch zu hoher Blutdruck kann lebensbedrohend sein, letzerer ist darüber hinaus ein Risikofaktor insbesondere für Herz-Kreislauferkrankungen, die trotz wichtiger Fortschritte in der Behandlung immer noch die häufigste Todesursache in Deutschland darstellen. Die Hämodynamik, also die raumzeitliche Dynamik des Blutflusses, und der Gefäßzustand sind eng verbunden mit dem Blutdruck und ebenfalls von hoher klinischer Relevanz, u.a. zur Identifikation von Durchblutungsstörungen und ungünstigen Druckverteilungen der Gefäßwand. Innovationen in der Messtechnik als auch in der Datenanalyse bieten heute neue Möglichkeiten der Erfassung und Bewertung von Blutdruck, Hämodynamik und Gefäßzustand [1], [2], [3], [4].
Methodik:
In einer gemeinsamen Workshopserie der AG Medizinische Bild- und Signalverarbeitung der GMDS und des Fachausschusses Biosignale der DGBMT werden wir neue Ansätze und Lösungen für Mess- und Analyseverfahren zu Blutdruck und -fluss sowie zum Gefäßzustand vorstellen und diskutieren. Dabei stehen im ersten Workshop auf der GMDS Jahrestagung Bildbasierte Verfahren im Zentrum, während der zweite Workshop auf der DGBMT Jahrestagung den Fokus auf Biosignalbasierten Verfahren legt. Es werden aktuelle Forschungsergebnisse vorgestellt und diskutiert. Es sind jeweils mehrere Vorträge geplant mit ausreichend Zeit zur Diskussion. Folgende Vorträge sind geplant (Arbeitstitel):
Sebastian Zaunseder: Videobasierte Erfassung des Blutdrucks
Anja Hennemuth: A Visualization Toolkit for the Analysis of Aortic Anatomy and Pressure Distribution
Lars Krenkel: Numerische Analyse der Rupturwahrscheinlichkeit zerebraler Aneurysmata
Susanne Schnell: Messung des Blutflusses und hämodynamischer Parameter mit 4D flow MRI: Möglichkeiten und Herausforderungen
Ergebnisse:
Ziel des Workshops ist die Identifikation von innovativen Ansätzen und neuen Methoden zur qualitativen und quantitativen Bestimmung von hämodynamischen Parametern sowie deren kritische Bewertung durch die Community für die Eignung in der klinischen Entscheidungsunterstützung.
Diskussion:
Der Workshop leistet inhaltlich einen Beitrag zu zentralen Aspekten für die Herz-Kreislauf-Medizin. Er bringt dabei Expertise aus verschiedenen Bereichen zusammen und schlägt die Brücke zwischen Kardiologie, Medizininformatik und Medizintechnik.
Schlussfolgerung:
Innovative Technologien aus Medizintechnik und Informatik ermöglichen zunehmend einfache und raumzeitlich aufgelöste Erfassung und Bewertung wichtiger Informationen zur Unterstützung von Diagnose und Therapieverfolgung.
[1] Zaunseder S, Trumpp A, Wedekind D, Malberg H. Cardiovascular assessment by imaging photoplethysmography - a review. Biomed Tech (Berl). 2018 Oct 25;63(5):617–34.
[2] Huellebrand M, Messroghli D, Tautz L, Kuehne T, Hennemuth A. An extensible software platform for interdisciplinary cardiovascular imaging research. Comput Methods Programs Biomed. 2020 Feb;184:105277.
[3] Schmitter S, Adriany G, Waks M, Moeller S, Aristova M, Vali A, et al. Bilateral Multiband 4D Flow MRI of the Carotid Arteries at 7T. Magn Reson Med. 2020 Oct;84(4):1947–60.
[4] Birkenmaier C, and Krenkel, L. Flow in Artificial Lungs. In: New Results in Numerical and Experimental Fluid Mechanics XIII. Contributions to the 22nd STAB/DGLR Symposium. Springer; 2021.