Fractional-order stochastic gradient descent method with momentum and energy for deep neural networks.

Autor: Zhou X; School of Information Science and Engineering, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China; School of Nuclear Science and Technology, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China., You Z; School of Information Science and Engineering, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China., Sun W; School of Information Science and Engineering, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China., Zhao D; School of Information Science and Engineering, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China., Yan S; School of Information Science and Engineering, Lanzhou University, 222 Tianshui South Road, Chengguan District, Lanzhou, Gansu Province, Lanzhou, 730000, Gansu, China. Electronic address: yanshi@lzu.edu.cn.
Jazyk: angličtina
Zdroj: Neural networks : the official journal of the International Neural Network Society [Neural Netw] 2025 Jan; Vol. 181, pp. 106810. Date of Electronic Publication: 2024 Oct 19.
DOI: 10.1016/j.neunet.2024.106810
Abstrakt: In this paper, a novel fractional-order stochastic gradient descent with momentum and energy (FOSGDME) approach is proposed. Specifically, to address the challenge of converging to a real extreme point encountered by the existing fractional gradient algorithms, a novel fractional-order stochastic gradient descent (FOSGD) method is presented by modifying the definition of the Caputo fractional-order derivative. A FOSGD with moment (FOSGDM) is established by incorporating momentum information to accelerate the convergence speed and accuracy further. In addition, to improve the robustness and accuracy, a FOSGD with moment and energy is established by further introducing energy formation. The extensive experimental results on the image classification CIFAR-10 dataset obtained with ResNet and DenseNet demonstrate that the proposed FOSGD, FOSGDM and FOSGDME algorithms are superior to the integer order optimization algorithms, and achieve state-of-the-art performance.
Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
(Copyright © 2024. Published by Elsevier Ltd.)
Databáze: MEDLINE
Externí odkaz: