A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition
In this paper, a deep learning optimization algorithm is proposed, which is based on the Grünwald–Letnikov (G-L) fractional order definition. An optimizer fractional calculus gradient descent based on the G-L fractional order definition (FCGD_G-L) is designed. Using the short-memory effect of the G-...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-01-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/11/2/316 |
| _version_ | 1797439016415526912 |
|---|---|
| author | Xiaojun Zhou Chunna Zhao Yaqun Huang |
| author_facet | Xiaojun Zhou Chunna Zhao Yaqun Huang |
| author_sort | Xiaojun Zhou |
| collection | DOAJ |
| description | In this paper, a deep learning optimization algorithm is proposed, which is based on the Grünwald–Letnikov (G-L) fractional order definition. An optimizer fractional calculus gradient descent based on the G-L fractional order definition (FCGD_G-L) is designed. Using the short-memory effect of the G-L fractional order definition, the derivation only needs 10 time steps. At the same time, via the transforming formula of the G-L fractional order definition, the Gamma function is eliminated. Thereby, it can achieve the unification of the fractional order and integer order in FCGD_G-L. To prevent the parameters falling into local optimum, a small disturbance is added in the unfolding process. According to the stochastic gradient descent (SGD) and Adam, two optimizers’ fractional calculus stochastic gradient descent based on the G-L definition (FCSGD_G-L), and the fractional calculus Adam based on the G-L definition (FCAdam_G-L), are obtained. These optimizers are validated on two time series prediction tasks. With the analysis of train loss, related experiments show that FCGD_G-L has the faster convergence speed and better convergence accuracy than the conventional integer order optimizer. Because of the fractional order property, the optimizer exhibits stronger robustness and generalization ability. Through the test sets, using the saved optimal model to evaluate, FCGD_G-L also shows a better evaluation effect than the conventional integer order optimizer. |
| first_indexed | 2024-03-09T11:46:45Z |
| format | Article |
| id | doaj.art-af87f13a2b8646d4b757165bb362dd92 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-09T11:46:45Z |
| publishDate | 2023-01-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-af87f13a2b8646d4b757165bb362dd922023-11-30T23:20:30ZengMDPI AGMathematics2227-73902023-01-0111231610.3390/math11020316A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order DefinitionXiaojun Zhou0Chunna Zhao1Yaqun Huang2School of Information Science and Engineering, Yunnan University, Kunming 650500, ChinaSchool of Information Science and Engineering, Yunnan University, Kunming 650500, ChinaSchool of Information Science and Engineering, Yunnan University, Kunming 650500, ChinaIn this paper, a deep learning optimization algorithm is proposed, which is based on the Grünwald–Letnikov (G-L) fractional order definition. An optimizer fractional calculus gradient descent based on the G-L fractional order definition (FCGD_G-L) is designed. Using the short-memory effect of the G-L fractional order definition, the derivation only needs 10 time steps. At the same time, via the transforming formula of the G-L fractional order definition, the Gamma function is eliminated. Thereby, it can achieve the unification of the fractional order and integer order in FCGD_G-L. To prevent the parameters falling into local optimum, a small disturbance is added in the unfolding process. According to the stochastic gradient descent (SGD) and Adam, two optimizers’ fractional calculus stochastic gradient descent based on the G-L definition (FCSGD_G-L), and the fractional calculus Adam based on the G-L definition (FCAdam_G-L), are obtained. These optimizers are validated on two time series prediction tasks. With the analysis of train loss, related experiments show that FCGD_G-L has the faster convergence speed and better convergence accuracy than the conventional integer order optimizer. Because of the fractional order property, the optimizer exhibits stronger robustness and generalization ability. Through the test sets, using the saved optimal model to evaluate, FCGD_G-L also shows a better evaluation effect than the conventional integer order optimizer.https://www.mdpi.com/2227-7390/11/2/316deep learning optimizerstochastic gradient descentfractional orderAdamtime series prediction |
| spellingShingle | Xiaojun Zhou Chunna Zhao Yaqun Huang A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition Mathematics deep learning optimizer stochastic gradient descent fractional order Adam time series prediction |
| title | A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition |
| title_full | A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition |
| title_fullStr | A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition |
| title_full_unstemmed | A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition |
| title_short | A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition |
| title_sort | deep learning optimizer based on grunwald letnikov fractional order definition |
| topic | deep learning optimizer stochastic gradient descent fractional order Adam time series prediction |
| url | https://www.mdpi.com/2227-7390/11/2/316 |
| work_keys_str_mv | AT xiaojunzhou adeeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition AT chunnazhao adeeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition AT yaqunhuang adeeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition AT xiaojunzhou deeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition AT chunnazhao deeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition AT yaqunhuang deeplearningoptimizerbasedongrunwaldletnikovfractionalorderdefinition |