AG-SGD: Angle-Based Stochastic Gradient Descent
In the field of neural network, stochastic gradient descent is often employed as an effective method of accelerating the result's convergence. Generating the new gradient from the past gradient is a common method adopted by many existing optimization algorithms. Since the past gradient is not c...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9343305/ |
| _version_ | 1819276839492780032 |
|---|---|
| author | Chongya Song Alexander Pons Kang Yen |
| author_facet | Chongya Song Alexander Pons Kang Yen |
| author_sort | Chongya Song |
| collection | DOAJ |
| description | In the field of neural network, stochastic gradient descent is often employed as an effective method of accelerating the result's convergence. Generating the new gradient from the past gradient is a common method adopted by many existing optimization algorithms. Since the past gradient is not computed based on the most updated stochastic gradient descent state, it can introduce a deviation to the new gradient computation, negatively impacting its rate of convergence. To resolve this problem, we propose an algorithm that quantifies this deviation based on the angle between the past and the current gradients, which is then applied to calibrate these two gradients, generating a more accurate new gradient. To demonstrate the broad applicability of the algorithm, the proposed method is implemented into a neural network and a logistic regression classifier which are evaluated on the datasets MNIST and NSL-KDD, respectively. An in-depth analysis is performed to compare our algorithm with nine optimization algorithms in two experiments, demonstrating the advantages in the cost and the error rate reductions from adopting the proposed method. |
| first_indexed | 2024-12-23T23:46:36Z |
| format | Article |
| id | doaj.art-e4904363afe749dbbd704516f78d6aaf |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-23T23:46:36Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-e4904363afe749dbbd704516f78d6aaf2022-12-21T17:25:29ZengIEEEIEEE Access2169-35362021-01-019230072302410.1109/ACCESS.2021.30559939343305AG-SGD: Angle-Based Stochastic Gradient DescentChongya Song0https://orcid.org/0000-0003-4208-5738Alexander Pons1Kang Yen2Department of Electrical and Computer Engineering, Florida International University, Miami, FL, USADepartment of Electrical and Computer Engineering, Florida International University, Miami, FL, USADepartment of Electrical and Computer Engineering, Florida International University, Miami, FL, USAIn the field of neural network, stochastic gradient descent is often employed as an effective method of accelerating the result's convergence. Generating the new gradient from the past gradient is a common method adopted by many existing optimization algorithms. Since the past gradient is not computed based on the most updated stochastic gradient descent state, it can introduce a deviation to the new gradient computation, negatively impacting its rate of convergence. To resolve this problem, we propose an algorithm that quantifies this deviation based on the angle between the past and the current gradients, which is then applied to calibrate these two gradients, generating a more accurate new gradient. To demonstrate the broad applicability of the algorithm, the proposed method is implemented into a neural network and a logistic regression classifier which are evaluated on the datasets MNIST and NSL-KDD, respectively. An in-depth analysis is performed to compare our algorithm with nine optimization algorithms in two experiments, demonstrating the advantages in the cost and the error rate reductions from adopting the proposed method.https://ieeexplore.ieee.org/document/9343305/Stochastic gradient descentoptimization algorithmgradient deviationneural network |
| spellingShingle | Chongya Song Alexander Pons Kang Yen AG-SGD: Angle-Based Stochastic Gradient Descent IEEE Access Stochastic gradient descent optimization algorithm gradient deviation neural network |
| title | AG-SGD: Angle-Based Stochastic Gradient Descent |
| title_full | AG-SGD: Angle-Based Stochastic Gradient Descent |
| title_fullStr | AG-SGD: Angle-Based Stochastic Gradient Descent |
| title_full_unstemmed | AG-SGD: Angle-Based Stochastic Gradient Descent |
| title_short | AG-SGD: Angle-Based Stochastic Gradient Descent |
| title_sort | ag sgd angle based stochastic gradient descent |
| topic | Stochastic gradient descent optimization algorithm gradient deviation neural network |
| url | https://ieeexplore.ieee.org/document/9343305/ |
| work_keys_str_mv | AT chongyasong agsgdanglebasedstochasticgradientdescent AT alexanderpons agsgdanglebasedstochasticgradientdescent AT kangyen agsgdanglebasedstochasticgradientdescent |