Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks
In this paper, we propose a new iterative thresholding algorithm based optimizer (Itao) for deep neural networks. It is a first-order gradient-based algorithm with Tikhonov regularization for stochastic objective functions. It is fast and straightforward to implement. It acts on the parameters and t...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9950492/ |
| _version_ | 1811211660108496896 |
|---|---|
| author | Mohamed Merrouchi Khalid Atifi Mustapha Skittou Taoufiq Gadi |
| author_facet | Mohamed Merrouchi Khalid Atifi Mustapha Skittou Taoufiq Gadi |
| author_sort | Mohamed Merrouchi |
| collection | DOAJ |
| description | In this paper, we propose a new iterative thresholding algorithm based optimizer (Itao) for deep neural networks. It is a first-order gradient-based algorithm with Tikhonov regularization for stochastic objective functions. It is fast and straightforward to implement. It acts on the parameters and their gradients, with respect to the objective function, in only one step in the backpropagation system when training a neural network. This reduces the learning time and makes it well suited for neural networks with large parameters and/or large datasets. We have experimented this algorithm on several types of loss functions such as mean squared error, mean absolute error and categorical crossentropy. Different types of models such as regression and classification are studied. The robustness of this optimizer against the noisy labels is also verified. Many of the empirical results of conducted experiments in this study show that our optimizer works well in practice. It can outperform other state-of-the-art optimizers in terms of accuracy or at least give the same results in addition to the reduction of learning time. |
| first_indexed | 2024-04-12T05:16:23Z |
| format | Article |
| id | doaj.art-cdd95b48f2b64281af39929d0f4ec7b8 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T05:16:23Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-cdd95b48f2b64281af39929d0f4ec7b82022-12-22T03:46:37ZengIEEEIEEE Access2169-35362022-01-011012088012089010.1109/ACCESS.2022.32223119950492Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural NetworksMohamed Merrouchi0https://orcid.org/0000-0001-8652-2798Khalid Atifi1https://orcid.org/0000-0001-6425-0015Mustapha Skittou2https://orcid.org/0000-0002-6337-8620Taoufiq Gadi3Faculty of Science and Technology, Hassan First University of Settat, Settat, MoroccoFaculty of Science and Technology, Cadi Ayyad University of Marrakech, Marrakesh, MoroccoFaculty of Science and Technology, Hassan First University of Settat, Settat, MoroccoFaculty of Science and Technology, Hassan First University of Settat, Settat, MoroccoIn this paper, we propose a new iterative thresholding algorithm based optimizer (Itao) for deep neural networks. It is a first-order gradient-based algorithm with Tikhonov regularization for stochastic objective functions. It is fast and straightforward to implement. It acts on the parameters and their gradients, with respect to the objective function, in only one step in the backpropagation system when training a neural network. This reduces the learning time and makes it well suited for neural networks with large parameters and/or large datasets. We have experimented this algorithm on several types of loss functions such as mean squared error, mean absolute error and categorical crossentropy. Different types of models such as regression and classification are studied. The robustness of this optimizer against the noisy labels is also verified. Many of the empirical results of conducted experiments in this study show that our optimizer works well in practice. It can outperform other state-of-the-art optimizers in terms of accuracy or at least give the same results in addition to the reduction of learning time.https://ieeexplore.ieee.org/document/9950492/Iterative thresholding algorithm based optimizer (Itao)deep learningoptimizer algorithmneural networks |
| spellingShingle | Mohamed Merrouchi Khalid Atifi Mustapha Skittou Taoufiq Gadi Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks IEEE Access Iterative thresholding algorithm based optimizer (Itao) deep learning optimizer algorithm neural networks |
| title | Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks |
| title_full | Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks |
| title_fullStr | Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks |
| title_full_unstemmed | Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks |
| title_short | Itao: A New Iterative Thresholding Algorithm Based Optimizer for Deep Neural Networks |
| title_sort | itao a new iterative thresholding algorithm based optimizer for deep neural networks |
| topic | Iterative thresholding algorithm based optimizer (Itao) deep learning optimizer algorithm neural networks |
| url | https://ieeexplore.ieee.org/document/9950492/ |
| work_keys_str_mv | AT mohamedmerrouchi itaoanewiterativethresholdingalgorithmbasedoptimizerfordeepneuralnetworks AT khalidatifi itaoanewiterativethresholdingalgorithmbasedoptimizerfordeepneuralnetworks AT mustaphaskittou itaoanewiterativethresholdingalgorithmbasedoptimizerfordeepneuralnetworks AT taoufiqgadi itaoanewiterativethresholdingalgorithmbasedoptimizerfordeepneuralnetworks |