TY - JOUR A1 - Vivekanandan, Deepak A1 - Wirth, Samuel A1 - Karlbauer, Patrick A1 - Klarmann, Noah T1 - A Reinforcement Learning Approach for Scheduling Problems with Improved Generalization through Order Swapping JF - Machine Learning and Knowledge Extraction N2 - The scheduling of production resources (such as associating jobs to machines) plays a vital role for the manufacturing industry not only for saving energy, but also for increasing the overall efficiency. Among the different job scheduling problems, the Job Shop Scheduling Problem (JSSP) is addressed in this work. JSSP falls into the category of NP-hard Combinatorial Optimization Problem (COP), in which solving the problem through exhaustive search becomes unfeasible. Simple heuristics such as First-In, First-Out, Largest Processing Time First and metaheuristics such as taboo search are often adopted to solve the problem by truncating the search space. The viability of the methods becomes inefficient for large problem sizes as it is either far from the optimum or time consuming. In recent years, the research towards using Deep Reinforcement Learning (DRL) to solve COPs has gained interest and has shown promising results in terms of solution quality and computational efficiency. In this work, we provide an novel approach to solve the JSSP examining the objectives generalization and solution effectiveness using DRL. In particular, we employ the Proximal Policy Optimization (PPO) algorithm that adopts the policy-gradient paradigm that is found to perform well in the constrained dispatching of jobs. We incorporated a new method called Order Swapping Mechanism (OSM) in the environment to achieve better generalized learning of the problem. The performance of the presented approach is analyzed in depth by using a set of available benchmark instances and comparing our results with the work of other groups. KW - Job Shop Scheduling KW - Production Scheduling KW - Reinforcement Learning KW - Markov Decision Process KW - Generalization KW - Industry 4.0 Y1 - 2023 U6 - https://doi.org/10.3390/make5020025 VL - 5 IS - 2 SP - 418 EP - 430 ER - TY - CHAP A1 - Josifovski, Josip A1 - Malmir, Mohammadhossein A1 - Klarmann, Noah A1 - Žagar, Bare Luka A1 - Navarro-Guerrero, Nicolás A1 - Knoll, Alois T1 - Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks T2 - 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan N2 - Randomization is currently a widely used approach in Sim2Real transfer for data-driven learning algorithms in robotics. Still, most Sim2Real studies report results for a specific randomization technique and often on a highly customized robotic system, making it difficult to evaluate different randomization approaches systematically. To address this problem, we define an easy-to-reproduce experimental setup for a robotic reach-and-balance manipulator task, which can serve as a benchmark for comparison. We compare four randomization strategies with three randomized parameters both in simulation and on a real robot. Our results show that more randomization helps in Sim2Real transfer, yet it can also harm the ability of the algorithm to find a good policy in simulation. Fully randomized simulations and fine-tuning show differentiated results and translate better to the real robot than the other approaches tested KW - Sim2Real KW - Robotics KW - Reinforcment Learning Y1 - 2022 U6 - https://doi.org/10.1109/IROS47612.2022.9981951 SP - 10193 EP - 10200 ER - TY - JOUR A1 - Mehta, Devang A1 - Klarmann, Noah T1 - Autoencoder-Based Visual Anomaly Localization for Manufacturing Quality Control JF - Machine Learning and Knowledge Extraction N2 - Manufacturing industries require the efficient and voluminous production of high-quality finished goods. In the context of Industry 4.0, visual anomaly detection poses an optimistic solution for automatically controlled product quality with high precision. In general, automation based on computer vision is a promising solution to prevent bottlenecks at the product quality checkpoint. We considered recent advancements in machine learning to improve visual defect localization, but challenges persist in obtaining a balanced feature set and database of the wide variety of defects occurring in the production line. Hence, this paper proposes a defect localizing autoencoder with unsupervised class selection by clustering with k-means the features extracted from a pretrained VGG16 network. Moreover, the selected classes of defects are augmented with natural wild textures to simulate artificial defects. The study demonstrates the effectiveness of the defect localizing autoencoder with unsupervised class selection for improving defect detection in manufacturing industries. The proposed methodology shows promising results with precise and accurate localization of quality defects on melamine-faced boards for the furniture industry. Incorporating artificial defects into the training data shows significant potential for practical implementation in real-world quality control scenarios. KW - anomaly detection KW - artifical defect simulation KW - autoencoder KW - computer vision KW - defect detection KW - unsupervised clustering KW - visual product quality control Y1 - 2024 U6 - https://doi.org/10.3390/make6010001 VL - 6 IS - 1 SP - 1 EP - 17 ER -