TY - CHAP A1 - Knollmeyer, Simon A1 - Caymazer, Oğuz A1 - Koval, Leonid A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Mathias, Selvine George A1 - Großmann, Daniel ED - Gruenwald, Le ED - Masciari, Elio ED - Bernardino, Jorge T1 - Benchmarking of Retrieval Augmented Generation: A Comprehensive Systematic Literature Review on Evaluation Dimensions, Evaluation Metrics and Datasets T2 - Proceedings of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2024) - Volume 3 N2 - Despite the rapid advancements in the field of Large Language Models (LLM), traditional benchmarks have proven to be inadequate for assessing the performance of Retrieval Augmented Generation (RAG) systems. Therefore, this paper presents a comprehensive systematic literature review of evaluation dimensions, metrics, and datasets for RAG systems. This review identifies key evaluation dimensions such as context relevance, faithfulness, answer relevance, correctness, and citation quality. For each evaluation dimension, several metrics and evaluators are proposed on how to assess them. This paper synthesizes the findings from 12 relevant papers and presents a concept matrix that categorizes each evaluation approach. The results provide a foundation for the development of robust evaluation frameworks and suitable datasets that are essential for the effective implementation and deployment of RAG systems in real-world applications. UR - https://doi.org/10.5220/0013065700003838 Y1 - 2024 UR - https://doi.org/10.5220/0013065700003838 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-58342 SN - 978-989-758-716-0 SP - 137 EP - 148 PB - SciTePress CY - Setúbal ER - TY - JOUR A1 - Koval, Leonid A1 - Knollmeyer, Simon A1 - Mathias, Selvine George A1 - Asif, Saara A1 - Akmal, Muhammad Uzair A1 - Großmann, Daniel A1 - Bregulla, Markus T1 - Unlocking the Potential of Information Modeling for Root Cause Analysis in a Production Environment: A Comprehensive State-of-the-Art Review Using the Kitchenham Methodology JF - IEEE Access N2 - Data from production environments is now available in unprecedented volumes, making the problem-solving of incidents through root cause analysis straightforward. However, the root cause analysis process remains time-consuming. This study employs the Kitchenham standard systematic literature review methodology to explore how information models and deep learning can streamline this process. By conducting a comprehensive search across four major databases, we evaluate the current technological advancements and their application in root cause analysis. The aim of this study is to assesses the impact of information models for root cause analysis in a production environment. Our findings reveal that integrating knowledge graphs, association rule mining, and deep learning algorithms significantly improves the speed and depth of root cause analysis compared to traditional methods. Specifically, the use of neural networks in recent literature shows substantial advancements in analyzing complex datasets, facilitating large-scale data integration, and enabling automated learning capabilities. Comparing our findings with other recent studies highlights the advantages of using information modeling and deep learning technologies in root cause analysis. This comparison underscores the superior accuracy and efficiency of these advanced methodologies over traditional manual interpretation methods. The effective implementation of these technologies requires a robust foundation of clean, standardized data, giving rise to the concept of “Production IT.” Furthermore, it is crucial for this data to be openly available to facilitate academic research, thereby enabling the development of new methods for more efficient and effective root cause analysis. UR - https://doi.org/10.1109/ACCESS.2024.3406020 Y1 - 2024 UR - https://doi.org/10.1109/ACCESS.2024.3406020 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-49240 SN - 2169-3536 VL - 12 SP - 80266 EP - 80282 PB - IEEE CY - New York ER - TY - CHAP A1 - Koval, Leonid A1 - Pfaller, Daniel A1 - Bilal, Mühenad A1 - Bregulla, Markus A1 - Cupek, Rafal T1 - An analysis of convolutional neural network models for classifying machine tools T2 - Advances in Computational Collective Intelligence UR - https://doi.org/10.1007/978-3-030-88113-9_37 KW - image classification KW - domain relevance KW - F1-score KW - confusion-matrix KW - machine tools Y1 - 2021 UR - https://doi.org/10.1007/978-3-030-88113-9_37 SN - 978-3-030-88112-2 SN - 978-3-030-88113-9 SN - 1865-0929 SN - 1865-0937 SP - 461 EP - 473 PB - Springer CY - Cham ER - TY - JOUR A1 - Bilal, Mühenad A1 - Podishetti, Ranadheer A1 - Koval, Leonid A1 - Gaafar, Mahmoud A. A1 - Großmann, Daniel A1 - Bregulla, Markus T1 - The Effect of Annotation Quality on Wear Semantic Segmentation by CNN JF - Sensors N2 - In this work, we investigate the impact of annotation quality and domain expertise on the performance of Convolutional Neural Networks (CNNs) for semantic segmentation of wear on titanium nitride (TiN) and titanium carbonitride (TiCN) coated end mills. Using an innovative measurement system and customized CNN architecture, we found that domain expertise significantly affects model performance. Annotator 1 achieved maximum mIoU scores of 0.8153 for abnormal wear and 0.7120 for normal wear on TiN datasets, whereas Annotator 3 with the lowest expertise achieved significantly lower scores. Sensitivity to annotation inconsistencies and model hyperparameters were examined, revealing that models for TiCN datasets showed a higher coefficient of variation (CV) of 16.32% compared to 8.6% for TiN due to the subtle wear characteristics, highlighting the need for optimized annotation policies and high-quality images to improve wear segmentation. UR - https://doi.org/10.3390/s24154777 Y1 - 2024 UR - https://doi.org/10.3390/s24154777 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-49551 SN - 1424-8220 VL - 24 IS - 15 PB - MDPI CY - Basel ER - TY - CHAP A1 - Bilal, Mühenad A1 - Kancharana, Sunil A1 - Mayer, Christian A1 - Pfaller, Daniel A1 - Koval, Leonid A1 - Bregulla, Markus A1 - Cupek, Rafal A1 - Ziębiński, Adam ED - Farinella, Giovanni Maria ED - Radeva, Petia ED - Bouatouch, Kadi T1 - High Resolution Mask R-CNN-based Damage Detection on Titanium Nitride Coated Milling Tools for Condition Monitoring by using a New Illumination Technique T2 - Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications N2 - The implementation of intelligent software in the manufacturing industry is a technology of growing importance and has highlighted the need for improvement in automatization, production, inspection, and quality assurance. An automated inspection system based on deep learning methods can help to enhance inspection and provide a consistent overview of the production line. Camera-based imaging systems are among the most widely used tools, replacing manual industrial quality control tasks. Moreover, an automatized damage detection system on milling tools can be employed in quality control during the coating process and to simplify measuring tool life. Deep Convolutional Neural Networks (DCNNs) are state-of-the-art methods used to extract visual features and classify objects. Hence, there is great interest in applying DCNN in damage detection and classification. However, training a DCNN model on Titanium-Nitride coated (TiN) milling tools is extremely challenging. Due to the coating, the optical properties such as reflection and light scattering on the milling tool surface make image capturing for computer vision tasks quite challenging. In addition to the reflection and scattering, the helical-shaped surface of the cutting tools creates shadows, preventing the neural network from efficient training and damage detection. Here, in the context of applying an automatized deep learning-based method to detect damages on coated milling tools for quality control, the light has been shed on a novel illumination technique that allows capturing high-quality images which makes efficient damage detection for condition monitoring and quality control reliable. The method is outlined along with results obtained in training a ResNet 50 and ResNet 101 model reaching an overall accuracy of 83% from a dataset containing bounding box annotated damages. For instance and semantic segmentation, the state-of-the-art framework Mask R-CNN is employed. UR - https://doi.org/10.5220/0010781800003124 KW - Predictive Maintenance KW - Machine Learning KW - Damage Detection KW - Illumination Source KW - Mask R-CNN Y1 - 2022 UR - https://doi.org/10.5220/0010781800003124 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-27498 SN - 978-989-758-555-5 SN - 2184-4321 VL - vol. 5: VISAPP SP - 305 EP - 314 PB - SciTePress CY - Setúbal ER - TY - THES A1 - Koval, Leonid T1 - Methodology for Evaluating and Optimizing Machine Learning Applications in Industrial Production N2 - This dissertation examines the challenges of integrating machine learning into manufacturing environments and introduces a structured optimization methodology, termedUnderstanding and Transformation, Classification and Estimation, Optimization, Results and Evaluation (UT-CORE), to align technical solutions with strategic production objectives. The initial chapters establish the historical context and current state of data-driven production, emphasizing the complexities introduced by imbalanced datasets, constrained budgets, and evolving AI maturity within the industry. Cost emerges as a central determinant for success, prompting in-depth analyses of model-centric and data-centric pipeline design approaches. Building on these foundations, UT-CORE is presented as a four-phase process that employs a morphological box to isolate critical pipeline components and to quantify them through cost, time, availability, and complexity. By transforming high-level goals into systematic ranking and selection mechanisms using multi-criteria decision-making tools, UTCORE pinpoints the most impactful aspects of an ML pipeline, such as labeling, deployment, or model tuning, and visualizes their break-even points for more transparent managerial oversight. This process supports incremental improvements and comprehensive overhauls and can be adapted to industrial use cases. An extensive real-world application in a small to medium-sized enterprise validates the method’s utility. The findings underscore the importance of optimizing meta-characteristics selectively rather than attempting to address an entire pipeline simultaneously. Concluding discussions highlight UT-CORE’s adaptability to emerging technologies such as automated machine learning, and robotic process automation, as well as the potential for integrated information modeling to enhance future iterations of the method. Ultimately, this dissertation contributes a robust, modular framework to facilitate cost-effective, data-driven quality assurance across diverse production contexts. Y1 - 2025 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-63465 PB - Technische Hochschule Ingolstadt CY - Ingolstadt ER - TY - JOUR A1 - Bilal, Mühenad A1 - Podishetti, Ranadheer A1 - Koval, Leonid A1 - Gaafar, Mahmoud A. A1 - Großmann, Daniel A1 - Bregulla, Markus T1 - Automatized End Mill Wear Inspection Using a Novel Illumination Unit and Convolutional Neural Network JF - IEEE Access N2 - Ensuring cutting tools are in optimal condition is essential for achieving peak machining performance, given their direct impact on both workpiece quality and process efficiency. However, accurately assessing wear on end mills, especially those with complex geometries, pose a significant challenge due to their reflective surfaces and varied wear patterns. Presented here is a novel method that addresses this challenge by employing a customized illumination unit in conjunction with a convolutional neural network (CNN) for end mill wear analysis. This innovative approach involves utilizing the specially designed illumination unit to capture high-quality images, enabling precise examination of material wear on helically shaped end mills. Notably, this method is tailored to illuminate reflective surfaces and represents a pioneering application in the realm of wear testing.We validate the viability of this approach by employing CNN-based models to segment wear on complex-shaped end mills coated with titanium carbonitride (TiCN) and titanium nitride (TiN). We achieved remarkable mean Intersection over Union (mIoU) results in wear detection on a test dataset: 0.99 for tool segmentation, 0.78 for abnormal wear, and 0.71 for normal wear segmentation. UR - https://doi.org/10.1109/ACCESS.2024.3454692 Y1 - 2024 UR - https://doi.org/10.1109/ACCESS.2024.3454692 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-58160 SN - 2169-3536 VL - 12 SP - 124282 EP - 124297 PB - IEEE CY - New York ER - TY - CHAP A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Koval, Leonid A1 - Mathias, Selvine George A1 - Knollmeyer, Simon A1 - Großmann, Daniel ED - Koprinkova-Hristova, Petia ED - Kasabov, Nikola T1 - Layered Data-Centric AI to Streamline Data Quality Practices for Enhanced Automation T2 - Artificial Intelligence: Methodology, Systems, and Applications: 19th International Conference, AIMSA 2024, Varna, Bulgaria, September 18–20, 2024, Proceedings UR - https://doi.org/10.1007/978-3-031-81542-3_11 Y1 - 2025 UR - https://doi.org/10.1007/978-3-031-81542-3_11 SN - 978-3-031-81542-3 SP - 128 EP - 142 PB - Springer CY - Cham ER - TY - CHAP A1 - Mathias, Selvine George A1 - Asif, Saara A1 - Akmal, Muhammad Uzair A1 - Knollmeyer, Simon A1 - Koval, Leonid A1 - Großmann, Daniel ED - Rocha, Ana Paula ED - Steels, Luc ED - van den Herik, Jaap T1 - Industrial Image Grouping Through Pre-Trained CNN Encoder-Based Feature Extraction and Sub-Clustering T2 - Proceedings of the 17th International Conference on Agents and Artificial Intelligence (ICAART 2025) - Volume 2 N2 - A common challenge faced by many industries today is the classification of unlabeled image data from production processes into meaningful groups or patterns for better documentation and analysis. This paper presents a sequential approach for leveraging industrial image data to identify patterns in products or processes for plant floor operators. The dataset used is sourced from steel production, and the model architecture integrates feature reduction through convolutional neural networks (CNNs) like VGG, EfficientNet, and ResNet, followed by clustering algorithms to assign appropriate labels to the observed data. The model’s selection criteria combine clustering metrics, including entropy minimization and silhouette score maximization. Once primary clusters are identified, sub-clustering is performed using near-labels, which are pre-assigned to images with initial distinctions. A novel metric, C-Score, is introduced to assess cluster convergence and grouping accuracy. Experimental re sults demonstrate that this method can address challenges in detecting variations across images, improving pattern recognition and classification. UR - https://doi.org/10.5220/0013189000003890 Y1 - 2025 UR - https://doi.org/10.5220/0013189000003890 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-58004 SN - 978-989-758-737-5 SP - 496 EP - 506 PB - SciTePress CY - Setúbal ER - TY - JOUR A1 - Mathias, Selvine George A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Koval, Leonid A1 - Knollmeyer, Simon A1 - Großmann, Daniel T1 - Pattern Identifications in Transformed Acoustic Signals Using Classification Models JF - Procedia CIRP N2 - Pattern identifications in signals is necessary to discern variations from approved normal values in different scenarios. With machine learning algorithms, it is possible to use hybrid methods of pattern identifications such as feature extractions followed by classifications and/or clustering. This paper presents a pattern identification approach of acoustic signals using their transformations as inputs to classification algorithms. The analysis is carried out on two transformed versions of acoustic emission (AE) hits such as log transformations of peak hits and binary sequencing based on threshold crossing. A comparative analysis using custom data loss metrics is made to determine which inputs provide the best information in predictive methods for identifying commonly occurring patterns while acknowledging significant data loss. The methodology is conducted on transformed versions of a public dataset and the results show that patterns can be discerned to above 90% accuracy with the transformed datasets. The experimental results yield that actual source signals need not be utilized depending on which transformations suit the practical application. UR - https://doi.org/10.1016/j.procir.2024.10.061 Y1 - 2024 UR - https://doi.org/10.1016/j.procir.2024.10.061 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-56233 SN - 2212-8271 VL - 2024 IS - 130 SP - 93 EP - 99 PB - Elsevier CY - Amsterdam ER - TY - CHAP A1 - Knollmeyer, Simon A1 - Akmal, Muhammad Uzair A1 - Koval, Leonid A1 - Asif, Saara A1 - Mathias, Selvine George A1 - Großmann, Daniel ED - Facchinetti, Tullio ED - Cenedese, Angelo ED - Lo Bello, Lucia ED - Vitturi, Stefano ED - Sauter, Thilo ED - Tramarin, Federico T1 - Document Knowledge Graph to Enhance Question Answering with Retrieval Augmented Generation T2 - 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA) UR - https://doi.org/10.1109/ETFA61755.2024.10711054 Y1 - 2024 UR - https://doi.org/10.1109/ETFA61755.2024.10711054 SN - 979-8-3503-6123-0 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Asif, Saara A1 - Akmal, Muhammad Uzair A1 - Koval, Leonid A1 - Knollmeyer, Simon A1 - Mathias, Selvine George A1 - Großmann, Daniel ED - Facchinetti, Tullio ED - Cenedese, Angelo ED - Lo Bello, Lucia ED - Vitturi, Stefano ED - Sauter, Thilo ED - Tramarin, Federico T1 - Supervised Anomaly Detection for Production Line Images using Data Augmentation and Convolutional Neural Network T2 - 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA) UR - https://doi.org/10.1109/ETFA61755.2024.10710718 Y1 - 2024 UR - https://doi.org/10.1109/ETFA61755.2024.10710718 SN - 979-8-3503-6123-0 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Koval, Leonid A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Mathias, Selvine George A1 - Knollmeyer, Simon A1 - Großmann, Daniel T1 - Optimizing AI-Driven Production in Industry 4.0: A Morphological Box and Taxonomy Approach T2 - 2025 International Conference on Computer Technology Applications (ICCTA) UR - https://doi.org/10.1109/ICCTA65425.2025.11166106 Y1 - 2025 UR - https://doi.org/10.1109/ICCTA65425.2025.11166106 SN - 979-8-3315-1265-1 SP - 272 EP - 278 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Koval, Leonid A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Mathias, Selvine George A1 - Knollmeyer, Simon A1 - Großmann, Daniel ED - Facchinetti, Tullio ED - Cenedese, Angelo ED - Lo Bello, Lucia ED - Vitturi, Stefano ED - Sauter, Thilo ED - Tramarin, Federico T1 - Addressing the complexity of AI Integration in Manufacturing: A Morphological Analysis T2 - 2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA) UR - https://doi.org/10.1109/ETFA61755.2024.10711011 Y1 - 2024 UR - https://doi.org/10.1109/ETFA61755.2024.10711011 SN - 979-8-3503-6123-0 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Asif, Saara A1 - Akmal, Muhammad Uzair A1 - Koval, Leonid A1 - Mathias, Selvine George A1 - Knollmeyer, Simon A1 - Großmann, Daniel ED - Sgurev, Vassil ED - Jotsov, Vladimir ED - Piuri, Vincenzo ED - Doukovska, Luybka ED - Yoshinov, Radoslav T1 - A Conceptual Framework for Addressing Class Imbalance in Image Data: Challenges and Strategies T2 - 2024 IEEE 12th International Conference on Intelligent Systems (IS): Proceedings UR - https://doi.org/10.1109/IS61756.2024.10705251 Y1 - 2024 UR - https://doi.org/10.1109/IS61756.2024.10705251 SN - 979-8-3503-5098-2 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Koval, Leonid A1 - Wächter, Sonja A1 - Erdogan, Hüseyin A1 - Großmann, Daniel T1 - Ontology-Driven Modeling and Integration of Production Processes in Advanced Driver-Assistance Systems within the Gaia-X Ecosystem T2 - 2025 11th International Conference on Computer Technology Applications (ICCTA 2025) UR - https://doi.org/10.1109/ICCTA65425.2025.11166225 Y1 - 2025 UR - https://doi.org/10.1109/ICCTA65425.2025.11166225 SN - 979-8-3315-1265-1 SP - 211 EP - 216 PB - IEEE CY - Piscataway ER - TY - CHAP A1 - Mathias, Selvine George A1 - Akmal, Muhammad Uzair A1 - Asif, Saara A1 - Knollmeyer, Simon A1 - Koval, Leonid A1 - Grossmann, Daniel ED - Maglogiannis, Ilias ED - Iliadis, Lazaros ED - Andreou, Andreas ED - Papaleonidas, Antonios T1 - Handling Anonymized Non-numerical Features in Data Using Transformations for Regression Models T2 - Artificial Intelligence Applications and Innovations: 21st IFIP WG 12.5 International Conference, AIAI 2025, Limassol, Cyprus, June 26–29, 2025, Proceedings, Part III UR - https://doi.org/10.1007/978-3-031-96231-8_8 Y1 - 2025 UR - https://doi.org/10.1007/978-3-031-96231-8_8 SN - 978-3-031-96231-8 SP - 99 EP - 111 PB - Springer CY - Cham ER - TY - CHAP A1 - Akmal, Muhammad Uzair A1 - Mathias, Selvine George A1 - Asif, Saara A1 - Koval, Leonid A1 - Knollmeyer, Simon A1 - Großmann, Daniel T1 - Humanizing AI: A Human-Centered Architecture to Developing Trustworthy Intelligent Systems T2 - CS & IT Conference Proceedings Y1 - 2025 UR - https://www.csitcp.com/abstract/15/158csit03 SN - 978-1-923107-57-1 SP - 27 EP - 44 PB - AIRCC CY - [s. l.] ER -