On the Convergence Proof of AMSGrad and a New Version
The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic, and they have also proposed a variant of Adam called AMSGrad as a fix. In this pa...
Main Authors: | Phuong Thi Tran, Le Trieu Phong |
---|---|
Format: | Article |
Language: | English |
Published: |
IEEE
2019-01-01
|
Series: | IEEE Access |
Subjects: | |
Online Access: | https://ieeexplore.ieee.org/document/8713445/ |
Similar Items
-
Unified Algorithm Framework for Nonconvex Stochastic Optimization in Deep Neural Networks
by: Yini Zhu, et al.
Published: (2021-01-01) -
The buffered optimization methods for online transfer function identification employed on DEAP actuator
by: Jakub Bernat, et al.
Published: (2023-09-01) -
GAPCNN with HyPar: Global Average Pooling convolutional neural network with novel NNLU activation function and HYBRID parallelism
by: Gousia Habib, et al.
Published: (2022-11-01) -
A ResNet‐based approach for accurate radiographic diagnosis of knee osteoarthritis
by: Yu Wang, et al.
Published: (2022-09-01) -
Communication-Efficient Distributed SGD with Error-Feedback, Revisited
by: Tran Thi Phuong, et al.
Published: (2021-04-01)