TY - CHAP A1 - Tresp, Volker A1 - Udluft, Steffen A1 - Hein, Daniel A1 - Hauptmann, Werner A1 - Leib, Martin A1 - Mutschler, Christopher A1 - Scherer, Daniel D. A1 - Mauerer, Wolfgang T1 - Workshop Summary: Quantum Machine Learning T2 - 2023 IEEE International Conference on Quantum Computing and Engineering, Bellevue, WA, United States, September 17-22, 2023 Y1 - 2023 U6 - https://doi.org/10.1109/QCE57702.2023.10174 PB - IEEE ER - TY - JOUR A1 - Franz, Maja A1 - Wolf, Lucas A1 - Periyasamy, Maniraman A1 - Ufrecht, Christian A1 - Scherer, Daniel D. A1 - Plinge, Axel A1 - Mutschler, Christopher A1 - Mauerer, Wolfgang T1 - Uncovering Instabilities in Variational-Quantum Deep Q-Networks JF - Journal of the Franklin Institute N2 - Deep Reinforcement Learning (RL) has considerably advanced over the past decade. At the same time, state-of-the-art RL algorithms require a large computational budget in terms of training time to converge. Recent work has started to approach this problem through the lens of quantum computing, which promises theoretical speed-ups for several traditionally hard tasks. In this work, we examine a class of hybrid quantumclassical RL algorithms that we collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show that VQ-DQN approaches are subject to instabilities that cause the learned policy to diverge, study the extent to which this afflicts reproduciblity of established results based on classical simulation, and perform systematic experiments to identify potential explanations for the observed instabilities. Additionally, and in contrast to most existing work on quantum reinforcement learning, we execute RL algorithms on an actual quantum processing unit (an IBM Quantum Device) and investigate differences in behaviour between simulated and physical quantum systems that suffer from implementation deficiencies. Our experiments show that, contrary to opposite claims in the literature, it cannot be conclusively decided if known quantum approaches, even if simulated without physical imperfections, can provide an advantage as compared to classical approaches. Finally, we provide a robust, universal and well-tested implementation of VQ-DQN as a reproducible testbed for future experiments. KW - Artificial Intelligence KW - Computer Science KW - Quantum Physics Y1 - 2022 U6 - https://doi.org/10.1016/j.jfranklin.2022.08.021 SN - 0016-0032 N1 - Corresponding author: Maja Franz PB - Elsevier ET - In Press, Corrected Proof ER - TY - INPR A1 - Periyasamy, Maniraman A1 - Plinge, Axel A1 - Mutschler, Christopher A1 - Scherer, Daniel D. A1 - Mauerer, Wolfgang T1 - Guided-SPSA: Simultaneous Perturbation Stochastic Approximation assisted by the Parameter Shift Rule N2 - The study of variational quantum algorithms (VQCs) has received significant attention from the quantum computing community in recent years. These hybrid algorithms, utilizing both classical and quantum components, are well-suited for noisy intermediate-scale quantum devices. Though estimating exact gradients using the parameter-shift rule to optimize the VQCs is realizable in NISQ devices, they do not scale well for larger problem sizes. The computational complexity, in terms of the number of circuit evaluations required for gradient estimation by the parameter-shift rule, scales linearly with the number of parameters in VQCs. On the other hand, techniques that approximate the gradients of the VQCs, such as the simultaneous perturbation stochastic approximation (SPSA), do not scale with the number of parameters but struggle with instability and often attain suboptimal solutions. In this work, we introduce a novel gradient estimation approach called Guided-SPSA, which meaningfully combines the parameter-shift rule and SPSA-based gradient approximation. The Guided-SPSA results in a 15% to 25% reduction in the number of circuit evaluations required during training for a similar or better optimality of the solution found compared to the parameter-shift rule. The Guided-SPSA outperforms standard SPSA in all scenarios and outperforms the parameter-shift rule in scenarios such as suboptimal initialization of the parameters. We demonstrate numerically the performance of Guided-SPSA on different paradigms of quantum machine learning, such as regression, classification, and reinforcement learning. Y1 - 2024 ER -