Überblick Statistik: Seitenaufrufe (grau) und PDF-Downloads (blau)

ELRA: Exponential learning rate adaption gradient descent optimization method

  • We present a novel, fast (exponential rate adaption), ab initio (hyper-parameter-free) gradient based optimizer algorithm. The main idea of the method is to adapt the learning rate α by situational awareness, mainly striving for orthogonal neighboring gradients. The method has a high success and fast convergence rate and does not rely on hand-tuned parameters giving it greater universality. It can be applied to problems of any dimensions n and scales only linearly (of order O(n)) with the dimension of the problem. It optimizes convex and non-convex continuous landscapes providing some kind of gradient. In contrast to the Ada-family (AdaGrad, AdaMax, AdaDelta, Adam, etc.) the method is rotation invariant: optimization path and performance are independent of coordinate choices. The impressive performance is demonstrated by extensive experiments on the MNIST benchmark data-set against state-of-the-art optimizers. We name this new class of optimizers after its core idea Exponential Learning Rate Adaption - ELRA. We present it in two variants c2min and p2min with slightly different control. The authors strongly believe that ELRA will open a completely new research direction for gradient descent optimize.

Download full text files

Export metadata

Additional Services

Search Google Scholar
Metadaten
URN:urn:nbn:de:kobv:526-opus4-20839
Publisher DOI:https://doi.org/10.48550/arXiv.2309.06274
Author:Alexander KleinsorgeORCiD, Stefan KupperORCiD, Alexander FauckORCiDGND, Felix Rothe
ISSN:2331-8422
Document Type:Preprint
Language:English
Year of Publication:2023
Publisher:arXiv.org
Faculties an central facilities:Fachbereich Ingenieur- und Naturwissenschaften
Publishing Institution:Technische Hochschule Wildau
Source:Kleinsorge, A., Kupper, S., Fauck, A., & Rothe, F. (2023). ELRA: Exponential learning rate adaption gradient descent optimization method. ArXiv.org. doi:10.48550/arXiv.2309.06274
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 006 Spezielle Computerverfahren
5 Naturwissenschaften und Mathematik / 51 Mathematik / 519 Wahrscheinlichkeiten, angewandte Mathematik
Licence (German):Creative Commons - CC BY-NC-SA - Namensnennung - Nicht kommerziell - Weitergabe unter gleichen Bedingungen 4.0 International
Release Date:2025/09/29
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.