Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we d...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-11-01
|
| Series: | Algorithms |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-4893/14/11/334 |
| _version_ | 1827677684209025024 |
|---|---|
| author | Nicola Landro Ignazio Gallo Riccardo La Grassa |
| author_facet | Nicola Landro Ignazio Gallo Riccardo La Grassa |
| author_sort | Nicola Landro |
| collection | DOAJ |
| description | Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings. |
| first_indexed | 2024-03-10T05:47:23Z |
| format | Article |
| id | doaj.art-b5508f2d11a04b6db13a9ff530052572 |
| institution | Directory Open Access Journal |
| issn | 1999-4893 |
| language | English |
| last_indexed | 2024-03-10T05:47:23Z |
| publishDate | 2021-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Algorithms |
| spelling | doaj.art-b5508f2d11a04b6db13a9ff5300525722023-11-22T22:05:03ZengMDPI AGAlgorithms1999-48932021-11-01141133410.3390/a14110334Is One Teacher Model Enough to Transfer Knowledge to a Student Model?Nicola Landro0Ignazio Gallo1Riccardo La Grassa2Department of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyDepartment of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyDepartment of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyNowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.https://www.mdpi.com/1999-4893/14/11/334deep learningtransfer learningloss functions |
| spellingShingle | Nicola Landro Ignazio Gallo Riccardo La Grassa Is One Teacher Model Enough to Transfer Knowledge to a Student Model? Algorithms deep learning transfer learning loss functions |
| title | Is One Teacher Model Enough to Transfer Knowledge to a Student Model? |
| title_full | Is One Teacher Model Enough to Transfer Knowledge to a Student Model? |
| title_fullStr | Is One Teacher Model Enough to Transfer Knowledge to a Student Model? |
| title_full_unstemmed | Is One Teacher Model Enough to Transfer Knowledge to a Student Model? |
| title_short | Is One Teacher Model Enough to Transfer Knowledge to a Student Model? |
| title_sort | is one teacher model enough to transfer knowledge to a student model |
| topic | deep learning transfer learning loss functions |
| url | https://www.mdpi.com/1999-4893/14/11/334 |
| work_keys_str_mv | AT nicolalandro isoneteachermodelenoughtotransferknowledgetoastudentmodel AT ignaziogallo isoneteachermodelenoughtotransferknowledgetoastudentmodel AT riccardolagrassa isoneteachermodelenoughtotransferknowledgetoastudentmodel |