Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance
Knowledge distillation (KD) is a powerful technique that enables a well-trained large model to assist a small model. However, KD is constrained in a teacher-student manner. Thus, this method may not be appropriate in general situations, where the learning abilities of two models are uncertain or not...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2018-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8409945/ |
| _version_ | 1818853834707959808 |
|---|---|
| author | Jinzhuo Wang Wenmin Wang Wen Gao |
| author_facet | Jinzhuo Wang Wenmin Wang Wen Gao |
| author_sort | Jinzhuo Wang |
| collection | DOAJ |
| description | Knowledge distillation (KD) is a powerful technique that enables a well-trained large model to assist a small model. However, KD is constrained in a teacher-student manner. Thus, this method may not be appropriate in general situations, where the learning abilities of two models are uncertain or not significantly different. In this paper, we propose a collaborative learning (CL) method, which is a flexible strategy to achieve bidirectional model assistance for two models using a mutual knowledge base (MKB). The MKB is used to collect mutual information and provide assistance, and it is updated along with the learning process of the two models and separately deployed when converged. We show that CL can be applied to any two deep neural networks and is easily extended to multiple networks. Compared with the teacher-student framework, CL can achieve bidirectional assistance and does not impose specific requirements on the involved models, such as pretraining and different abilities. The experimental results demonstrate that CL can efficiently improve the learning ability and convergence speed of the two models, with superior performance to a series of relevant methods, such as ensemble learning and a series of KD-based methods. More importantly, we show that the state-of-the-art models, such as DenseNet, can be greatly improved using CL along with other popular models. |
| first_indexed | 2024-12-19T07:43:07Z |
| format | Article |
| id | doaj.art-ce65c4bcb7c4471d88566211a4b7b212 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-19T07:43:07Z |
| publishDate | 2018-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-ce65c4bcb7c4471d88566211a4b7b2122022-12-21T20:30:25ZengIEEEIEEE Access2169-35362018-01-016394903950010.1109/ACCESS.2018.28549188409945Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model AssistanceJinzhuo Wang0https://orcid.org/0000-0002-9464-4426Wenmin Wang1Wen Gao2School of Electronic and Computer Engineering, Peking University, Shenzhen, ChinaSchool of Electronic and Computer Engineering, Peking University, Shenzhen, ChinaSchool of Electronic Engineering and Computer Science, Peking University, Beijing, ChinaKnowledge distillation (KD) is a powerful technique that enables a well-trained large model to assist a small model. However, KD is constrained in a teacher-student manner. Thus, this method may not be appropriate in general situations, where the learning abilities of two models are uncertain or not significantly different. In this paper, we propose a collaborative learning (CL) method, which is a flexible strategy to achieve bidirectional model assistance for two models using a mutual knowledge base (MKB). The MKB is used to collect mutual information and provide assistance, and it is updated along with the learning process of the two models and separately deployed when converged. We show that CL can be applied to any two deep neural networks and is easily extended to multiple networks. Compared with the teacher-student framework, CL can achieve bidirectional assistance and does not impose specific requirements on the involved models, such as pretraining and different abilities. The experimental results demonstrate that CL can efficiently improve the learning ability and convergence speed of the two models, with superior performance to a series of relevant methods, such as ensemble learning and a series of KD-based methods. More importantly, we show that the state-of-the-art models, such as DenseNet, can be greatly improved using CL along with other popular models.https://ieeexplore.ieee.org/document/8409945/Bidirectional model assistancecollaborative learningdeep neural networksmutual knowledge base |
| spellingShingle | Jinzhuo Wang Wenmin Wang Wen Gao Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance IEEE Access Bidirectional model assistance collaborative learning deep neural networks mutual knowledge base |
| title | Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance |
| title_full | Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance |
| title_fullStr | Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance |
| title_full_unstemmed | Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance |
| title_short | Beyond Knowledge Distillation: Collaborative Learning for Bidirectional Model Assistance |
| title_sort | beyond knowledge distillation collaborative learning for bidirectional model assistance |
| topic | Bidirectional model assistance collaborative learning deep neural networks mutual knowledge base |
| url | https://ieeexplore.ieee.org/document/8409945/ |
| work_keys_str_mv | AT jinzhuowang beyondknowledgedistillationcollaborativelearningforbidirectionalmodelassistance AT wenminwang beyondknowledgedistillationcollaborativelearningforbidirectionalmodelassistance AT wengao beyondknowledgedistillationcollaborativelearningforbidirectionalmodelassistance |