Completion of Missing Parts in Medical Images Using Generative Adversarial Networks and Transformers
- Image inpainting (image completion) is a technique to remove undesirable elements or fill missing (repair damaged) sections in an image [1]. In medical images, inpainting can be applied for various applications. Issues like image artifacts (e.g., metal artifacts in CT and MRI) can introduce alterations to medical images. In simpler terms, these artifacts may introduce unwanted elements or anomalies into the medical images. These unwanted elements can be replaced with appropriate intensity in the images. In the context of defect reconstruction, which involves repairing skull defects, medical images are taken. To design an implant for a defective skull, image inpainting can be applied to an image containing the defective skull. This process generates an image with skull tissue, which can then be utilized for implant design. Additionally, in situations involving a restricted field of view, such as when only accessing the lower part of the full-body MRI, predicting the missing information (e.g., the upper part of the full-body MRI) can be beneficial for subsequent tasks, such as extracting bone structures for 3D shape modeling studies. Another application of inpainting is observed during actual stereo-fluoroscopic X-ray measurements. In these measurements, the calibration grid, which is beneficial for correcting distortions, introduces unwanted black dots into the image content. These black dots need to be removed from the image and replaced with the appropriate intensity. This thesis extends an existing architecture named Mask-aware Transformer (MAT) to fill in missing information in medical scans. The proposed method utilizes Generative Adversarial Networks and a Mask-aware Swin Transformer. Various adjustments, including changes in data type, model architecture, and adaptation for multinode training, have been implemented to address missing information in three different datasets. In one of our datasets, where black dots disrupt the image content, we applied our method to fill in the missing information. The results indicate that our approach outperforms other model-driven methods. In the second dataset, we employed the German National Cohort full-body MRI dataset to train a network for predicting the upper part of the full-body MRI. The outcomes indicate that the generated 2D slices exhibit realism and perform well in both pixel-wise and perceptual metrics. However, when these 2D slices are stacked into a 3D volume, inconsistencies between different slices become apparent. Furthermore, when we compare our 2D-aware image inpainting method to 3D inpainting algorithms used in the Brain Tumor Segmentation 2023 challenge, the results confirm the findings of the earlier experiment. The 2D slices produced are realistic, but there is inconsistency between them. In summary, theMAT demonstrates its ability to capture global dependencies by utilizing an adapted Swin Transformer for X-ray images, inherently 2D images. The results for 2D slices from a 3D MRI volume are also reasonable because the network is trained specifically on 2D slices. However, due to the lack of awareness regarding this architecture in the third dimension, inconsistencies between slices are observed.
| Author: | Javad Kasravi |
|---|---|
| Document Type: | Master's Thesis |
| Tag: | Generative Adversarial Networks; Image Inpainting; Mask-Tware Transformer; Swin Transformer |
| Granting Institution: | Freie Universität Berlin |
| Advisor: | Stefan Zachow, Cagdas Aslan |
| Date of final exam: | 2024/02/21 |
| Year of first publication: | 2024 |
| Page Number: | 70 |

