Feature fusion-based collaborative learning for knowledge distillation
Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performanc...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Hindawi - SAGE Publishing
2021-11-01
|
| Series: | International Journal of Distributed Sensor Networks |
| Online Access: | https://doi.org/10.1177/15501477211057037 |
| _version_ | 1797766822732234752 |
|---|---|
| author | Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou |
| author_facet | Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou |
| author_sort | Yiting Li |
| collection | DOAJ |
| description | Deep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher’s knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods. |
| first_indexed | 2024-03-12T20:29:46Z |
| format | Article |
| id | doaj.art-e64c67ea766f46eea337ddceeef31f02 |
| institution | Directory Open Access Journal |
| issn | 1550-1477 |
| language | English |
| last_indexed | 2024-03-12T20:29:46Z |
| publishDate | 2021-11-01 |
| publisher | Hindawi - SAGE Publishing |
| record_format | Article |
| series | International Journal of Distributed Sensor Networks |
| spelling | doaj.art-e64c67ea766f46eea337ddceeef31f022023-08-02T00:11:28ZengHindawi - SAGE PublishingInternational Journal of Distributed Sensor Networks1550-14772021-11-011710.1177/15501477211057037Feature fusion-based collaborative learning for knowledge distillationYiting Li0Liyuan Sun1Jianping Gou2Lan Du3Weihua Ou4School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, ChinaSchool of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, ChinaSchool of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, ChinaDepartment of Data Science and Artificial Intelligence, Faculty of Information Technology, Monash University, Clayton, VIC, AustraliaSchool of Big Data and Computer Science, Guizhou Normal University, Guiyang, ChinaDeep neural networks have achieved a great success in a variety of applications, such as self-driving cars and intelligent robotics. Meanwhile, knowledge distillation has received increasing attention as an effective model compression technique for training very efficient deep models. The performance of the student network obtained through knowledge distillation heavily depends on whether the transfer of the teacher’s knowledge can effectively guide the student training. However, most existing knowledge distillation schemes require a large teacher network pre-trained on large-scale data sets, which can increase the difficulty of knowledge distillation in different applications. In this article, we propose a feature fusion-based collaborative learning for knowledge distillation. Specifically, during knowledge distillation, it enables networks to learn from each other using the feature/response-based knowledge in different network layers. We concatenate the features learned by the teacher and the student networks to obtain a more representative feature map for knowledge transfer. In addition, we also introduce a network regularization method to further improve the model performance by providing a positive knowledge during training. Experiments and ablation studies on two widely used data sets demonstrate that the proposed method, feature fusion-based collaborative learning, significantly outperforms recent state-of-the-art knowledge distillation methods.https://doi.org/10.1177/15501477211057037 |
| spellingShingle | Yiting Li Liyuan Sun Jianping Gou Lan Du Weihua Ou Feature fusion-based collaborative learning for knowledge distillation International Journal of Distributed Sensor Networks |
| title | Feature fusion-based collaborative learning for knowledge distillation |
| title_full | Feature fusion-based collaborative learning for knowledge distillation |
| title_fullStr | Feature fusion-based collaborative learning for knowledge distillation |
| title_full_unstemmed | Feature fusion-based collaborative learning for knowledge distillation |
| title_short | Feature fusion-based collaborative learning for knowledge distillation |
| title_sort | feature fusion based collaborative learning for knowledge distillation |
| url | https://doi.org/10.1177/15501477211057037 |
| work_keys_str_mv | AT yitingli featurefusionbasedcollaborativelearningforknowledgedistillation AT liyuansun featurefusionbasedcollaborativelearningforknowledgedistillation AT jianpinggou featurefusionbasedcollaborativelearningforknowledgedistillation AT landu featurefusionbasedcollaborativelearningforknowledgedistillation AT weihuaou featurefusionbasedcollaborativelearningforknowledgedistillation |