A Novel Exponential Continuous Learning Rate Adaption Gradient Descent Optimization Method
- We present two novel, fast gradient based optimizer algorithms with dynamic learning rate. The main idea is to adapt the learning rate α by situational awareness, mainly striving for orthogonal neighboring gradients. The method has a high success and fast convergence rate and relies much less on hand-tuned hyper-parameters, providing greater universality. It scales linearly (of order O(n)) with dimension and is rotation invariant, thereby overcoming known limitations. The method is presented in two variants C2Min and P2Min, with slightly different control. Their impressive performance is demonstrated by experiments on several benchmark data-sets (ranging from MNIST to Tiny ImageNet) against the state-of-the-art optimizers Adam and Lion.
| URN: | urn:nbn:de:kobv:526-opus4-20785 |
|---|---|
| Publisher DOI: | https://doi.org/10.52825/th-wildau-ensp.v2i.2939 |
| Author: | Alexander KleinsorgeORCiD, Alexander FauckORCiDGND, Stefan KupperORCiD |
| Parent Title (German): | Wildauer Konferenz für Künstliche Intelligenz 2025 (WiKKI25) |
| Series (Serial Number): | TH Wildau Engineering and Natural Sciences Proceedings (2) |
| Document Type: | Conference Proceeding |
| Language: | English |
| Year of Publication: | 2025 |
| Publisher: | TIB Open Publishing |
| Place of publication: | Hannover |
| Faculties an central facilities: | Fachbereich Ingenieur- und Naturwissenschaften |
| Editor: | Jörg Reiff-StephanORCiDGND, Anja Beuster |
| Publishing Institution: | Technische Hochschule Wildau |
| Tag: | neural network; optimizer; training |
| Source: | Kleinsorge, A., Fauck, A., & Kupper, S. (2025). A Novel Exponential Continuous Learning Rate Adaption Gradient Descent Optimization Method. TH Wildau Engineering and Natural Sciences Proceedings , 2. https://doi.org/10.52825/th-wildau-ensp.v2i.2939 |
| Dewey Decimal Classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 006 Spezielle Computerverfahren |
| 5 Naturwissenschaften und Mathematik / 51 Mathematik / 519 Wahrscheinlichkeiten, angewandte Mathematik | |
| Funding: | Publikationsfonds für Open-Access-Monografien des Landes Brandenburg |
| Licence (German): | Creative Commons - CC BY - Namensnennung 4.0 International |
| Release Date: | 2025/09/26 |



