Is One Teacher Model Enough to Transfer Knowledge to a Student Model?

Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we d...

Full description

Bibliographic Details
Main Authors: Nicola Landro, Ignazio Gallo, Riccardo La Grassa
Format: Article
Language:English
Published: MDPI AG 2021-11-01
Series:Algorithms
Subjects:
Online Access:https://www.mdpi.com/1999-4893/14/11/334
_version_ 1827677684209025024
author Nicola Landro
Ignazio Gallo
Riccardo La Grassa
author_facet Nicola Landro
Ignazio Gallo
Riccardo La Grassa
author_sort Nicola Landro
collection DOAJ
description Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.
first_indexed 2024-03-10T05:47:23Z
format Article
id doaj.art-b5508f2d11a04b6db13a9ff530052572
institution Directory Open Access Journal
issn 1999-4893
language English
last_indexed 2024-03-10T05:47:23Z
publishDate 2021-11-01
publisher MDPI AG
record_format Article
series Algorithms
spelling doaj.art-b5508f2d11a04b6db13a9ff5300525722023-11-22T22:05:03ZengMDPI AGAlgorithms1999-48932021-11-01141133410.3390/a14110334Is One Teacher Model Enough to Transfer Knowledge to a Student Model?Nicola Landro0Ignazio Gallo1Riccardo La Grassa2Department of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyDepartment of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyDepartment of Theoretical and Applied Sciences—DISTA, University of Insubria, Via J.H. Dunant, 3, 21100 Varese, ItalyNowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.https://www.mdpi.com/1999-4893/14/11/334deep learningtransfer learningloss functions
spellingShingle Nicola Landro
Ignazio Gallo
Riccardo La Grassa
Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
Algorithms
deep learning
transfer learning
loss functions
title Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_full Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_fullStr Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_full_unstemmed Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_short Is One Teacher Model Enough to Transfer Knowledge to a Student Model?
title_sort is one teacher model enough to transfer knowledge to a student model
topic deep learning
transfer learning
loss functions
url https://www.mdpi.com/1999-4893/14/11/334
work_keys_str_mv AT nicolalandro isoneteachermodelenoughtotransferknowledgetoastudentmodel
AT ignaziogallo isoneteachermodelenoughtotransferknowledgetoastudentmodel
AT riccardolagrassa isoneteachermodelenoughtotransferknowledgetoastudentmodel