A hybrid model of heuristic algorithm and gradient descent to optimize neural networks
Training a neural network can be a challenging task, particularly when working with complex models and large amounts of training data, as it consumes significant time and resources. This research proposes a hybrid model that combines population-based heuristic algorithms with traditional gradient-ba...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Polish Academy of Sciences
2023-10-01
|
| Series: | Bulletin of the Polish Academy of Sciences: Technical Sciences |
| Subjects: | |
| Online Access: | https://journals.pan.pl/Content/129003/PDF/BPASTS_2023_71_6_3773.pdf |
| _version_ | 1797360979205423104 |
|---|---|
| author | Amer Mirkhan Numan Çelebi |
| author_facet | Amer Mirkhan Numan Çelebi |
| author_sort | Amer Mirkhan |
| collection | DOAJ |
| description | Training a neural network can be a challenging task, particularly when working with complex models and large amounts of training data, as it consumes significant time and resources. This research proposes a hybrid model that combines population-based heuristic algorithms with traditional gradient-based techniques to enhance the training process. The proposed approach involves using a dynamic population-based heuristic algorithm to identify good initial values for the neural network weight vector. This is done as an alternative to the traditional technique of starting with random weights. After several cycles of distributing search agents across the search domain, the training process continues using a gradient-based technique that starts with the best initial weight vector identified by the heuristic algorithm. Experimental analysis confirms that exploring the search domain during the training process decreases the number of cycles needed for gradient descent to train a neural network. Furthermore, a dynamic population strategy is applied during the heuristic search, with objects added and removed dynamically based on their progress. This approach yields better results compared to traditional heuristic algorithms that use the same population members throughout the search process. |
| first_indexed | 2024-03-08T15:47:21Z |
| format | Article |
| id | doaj.art-ffd3535b8c3d4126b361f54e0fda451c |
| institution | Directory Open Access Journal |
| issn | 2300-1917 |
| language | English |
| last_indexed | 2024-03-08T15:47:21Z |
| publishDate | 2023-10-01 |
| publisher | Polish Academy of Sciences |
| record_format | Article |
| series | Bulletin of the Polish Academy of Sciences: Technical Sciences |
| spelling | doaj.art-ffd3535b8c3d4126b361f54e0fda451c2024-01-09T10:13:46ZengPolish Academy of SciencesBulletin of the Polish Academy of Sciences: Technical Sciences2300-19172023-10-01716https://doi.org/10.24425/bpasts.2023.147924A hybrid model of heuristic algorithm and gradient descent to optimize neural networksAmer Mirkhan0Numan Çelebi1https://orcid.org/0000-0001-7489-9053Sakarya University, Computer Engineering DepartmentSakarya University, Information Systems Engineering DepartmentTraining a neural network can be a challenging task, particularly when working with complex models and large amounts of training data, as it consumes significant time and resources. This research proposes a hybrid model that combines population-based heuristic algorithms with traditional gradient-based techniques to enhance the training process. The proposed approach involves using a dynamic population-based heuristic algorithm to identify good initial values for the neural network weight vector. This is done as an alternative to the traditional technique of starting with random weights. After several cycles of distributing search agents across the search domain, the training process continues using a gradient-based technique that starts with the best initial weight vector identified by the heuristic algorithm. Experimental analysis confirms that exploring the search domain during the training process decreases the number of cycles needed for gradient descent to train a neural network. Furthermore, a dynamic population strategy is applied during the heuristic search, with objects added and removed dynamically based on their progress. This approach yields better results compared to traditional heuristic algorithms that use the same population members throughout the search process.https://journals.pan.pl/Content/129003/PDF/BPASTS_2023_71_6_3773.pdfoptimizationheuristic algorithmsneural networksdynamic population |
| spellingShingle | Amer Mirkhan Numan Çelebi A hybrid model of heuristic algorithm and gradient descent to optimize neural networks Bulletin of the Polish Academy of Sciences: Technical Sciences optimization heuristic algorithms neural networks dynamic population |
| title | A hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| title_full | A hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| title_fullStr | A hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| title_full_unstemmed | A hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| title_short | A hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| title_sort | hybrid model of heuristic algorithm and gradient descent to optimize neural networks |
| topic | optimization heuristic algorithms neural networks dynamic population |
| url | https://journals.pan.pl/Content/129003/PDF/BPASTS_2023_71_6_3773.pdf |
| work_keys_str_mv | AT amermirkhan ahybridmodelofheuristicalgorithmandgradientdescenttooptimizeneuralnetworks AT numancelebi ahybridmodelofheuristicalgorithmandgradientdescenttooptimizeneuralnetworks AT amermirkhan hybridmodelofheuristicalgorithmandgradientdescenttooptimizeneuralnetworks AT numancelebi hybridmodelofheuristicalgorithmandgradientdescenttooptimizeneuralnetworks |