Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning
Incremental learning is a methodology that continuously uses the sequential input data to extend the existing network’s knowledge. The layer sharing algorithm is one of the representative methods which leverages general knowledge by sharing some initial layers of the existing network. To determine t...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-03-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/5/2171 |
| _version_ | 1797416520315305984 |
|---|---|
| author | Bomi Kim Taehyeon Kim Yoonsik Choe |
| author_facet | Bomi Kim Taehyeon Kim Yoonsik Choe |
| author_sort | Bomi Kim |
| collection | DOAJ |
| description | Incremental learning is a methodology that continuously uses the sequential input data to extend the existing network’s knowledge. The layer sharing algorithm is one of the representative methods which leverages general knowledge by sharing some initial layers of the existing network. To determine the performance of the incremental network, it is critical to estimate how much the initial convolutional layers in the existing network can be shared as the fixed feature extractors. However, the existing algorithm selects the sharing configuration through improper optimization strategy but a brute force manner such as searching for all possible sharing layers case. This is a non-convex and non-differential problem. Accordingly, this can not be solved using powerful optimization techniques such as the gradient descent algorithm or other convex optimization problem, and it leads to high computational complexity. To solve this problem, we firstly define this as a discrete combinatorial optimization problem, and propose a novel efficient incremental learning algorithm-based Bayesian optimization, which guarantees the global convergence in a non-convex and non-differential optimization. Additionally, our proposed algorithm can adaptively find the optimal number of sharing layers via adjusting the threshold accuracy parameter in the proposed loss function. With the proposed method, the global optimal sharing layer can be found in only six or eight iterations without searching for all possible layer cases. Hence, the proposed method can find the global optimal sharing layers by utilizing Bayesian optimization, which achieves both high combined accuracy and low computational complexity. |
| first_indexed | 2024-03-09T06:05:24Z |
| format | Article |
| id | doaj.art-6c24c7a9be324cf0b78a3d47fa729395 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-09T06:05:24Z |
| publishDate | 2021-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-6c24c7a9be324cf0b78a3d47fa7293952023-12-03T12:04:41ZengMDPI AGApplied Sciences2076-34172021-03-01115217110.3390/app11052171Bayesian Optimization Based Efficient Layer Sharing for Incremental LearningBomi Kim0Taehyeon Kim1Yoonsik Choe2Department of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, KoreaDepartment of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, KoreaDepartment of Electrical and Electronic Engineering, Yonsei University, Seoul 03722, KoreaIncremental learning is a methodology that continuously uses the sequential input data to extend the existing network’s knowledge. The layer sharing algorithm is one of the representative methods which leverages general knowledge by sharing some initial layers of the existing network. To determine the performance of the incremental network, it is critical to estimate how much the initial convolutional layers in the existing network can be shared as the fixed feature extractors. However, the existing algorithm selects the sharing configuration through improper optimization strategy but a brute force manner such as searching for all possible sharing layers case. This is a non-convex and non-differential problem. Accordingly, this can not be solved using powerful optimization techniques such as the gradient descent algorithm or other convex optimization problem, and it leads to high computational complexity. To solve this problem, we firstly define this as a discrete combinatorial optimization problem, and propose a novel efficient incremental learning algorithm-based Bayesian optimization, which guarantees the global convergence in a non-convex and non-differential optimization. Additionally, our proposed algorithm can adaptively find the optimal number of sharing layers via adjusting the threshold accuracy parameter in the proposed loss function. With the proposed method, the global optimal sharing layer can be found in only six or eight iterations without searching for all possible layer cases. Hence, the proposed method can find the global optimal sharing layers by utilizing Bayesian optimization, which achieves both high combined accuracy and low computational complexity.https://www.mdpi.com/2076-3417/11/5/2171Bayesian optimizationincremental learninglayer sharing algorithm |
| spellingShingle | Bomi Kim Taehyeon Kim Yoonsik Choe Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning Applied Sciences Bayesian optimization incremental learning layer sharing algorithm |
| title | Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning |
| title_full | Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning |
| title_fullStr | Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning |
| title_full_unstemmed | Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning |
| title_short | Bayesian Optimization Based Efficient Layer Sharing for Incremental Learning |
| title_sort | bayesian optimization based efficient layer sharing for incremental learning |
| topic | Bayesian optimization incremental learning layer sharing algorithm |
| url | https://www.mdpi.com/2076-3417/11/5/2171 |
| work_keys_str_mv | AT bomikim bayesianoptimizationbasedefficientlayersharingforincrementallearning AT taehyeonkim bayesianoptimizationbasedefficientlayersharingforincrementallearning AT yoonsikchoe bayesianoptimizationbasedefficientlayersharingforincrementallearning |