Data-Efficient Sensor Upgrade Path Using Knowledge Distillation
Deep neural networks have achieved state-of-the-art performance in image classification. Due to this success, deep learning is now also being applied to other data modalities such as multispectral images, lidar and radar data. However, successfully training a deep neural network requires a large red...
| Main Authors: | , , , , , , , |
|---|---|
| 格式: | 文件 |
| 语言: | English |
| 出版: |
MDPI AG
2021-09-01
|
| 丛编: | Sensors |
| 主题: | |
| 在线阅读: | https://www.mdpi.com/1424-8220/21/19/6523 |
| _version_ | 1827680562647662592 |
|---|---|
| author | Pieter Van Molle Cedric De Boom Tim Verbelen Bert Vankeirsbilck Jonas De Vylder Bart Diricx Pieter Simoens Bart Dhoedt |
| author_facet | Pieter Van Molle Cedric De Boom Tim Verbelen Bert Vankeirsbilck Jonas De Vylder Bart Diricx Pieter Simoens Bart Dhoedt |
| author_sort | Pieter Van Molle |
| collection | DOAJ |
| description | Deep neural networks have achieved state-of-the-art performance in image classification. Due to this success, deep learning is now also being applied to other data modalities such as multispectral images, lidar and radar data. However, successfully training a deep neural network requires a large reddataset. Therefore, transitioning to a new sensor modality (e.g., from regular camera images to multispectral camera images) might result in a drop in performance, due to the limited availability of data in the new modality. This might hinder the adoption rate and time to market for new sensor technologies. In this paper, we present an approach to leverage the knowledge of a teacher network, that was trained using the original data modality, to improve the performance of a student network on a new data modality: a technique known in literature as knowledge distillation. By applying knowledge distillation to the problem of sensor transition, we can greatly speed up this process. We validate this approach using a multimodal version of the MNIST dataset. Especially when little data is available in the new modality (i.e., 10 images), training with additional teacher supervision results in increased performance, with the student network scoring a test set accuracy of 0.77, compared to an accuracy of 0.37 for the baseline. We also explore two extensions to the default method of knowledge distillation, which we evaluate on a multimodal version of the CIFAR-10 dataset: an annealing scheme for the hyperparameter <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>α</mi></semantics></math></inline-formula> and selective knowledge distillation. Of these two, the first yields the best results. Choosing the optimal annealing scheme results in an increase in test set accuracy of 6%. Finally, we apply our method to the real-world use case of skin lesion classification. |
| first_indexed | 2024-03-10T06:51:28Z |
| format | Article |
| id | doaj.art-bf71d20ae35e4f95a2d1fcc487956e78 |
| institution | Directory Open Access Journal |
| issn | 1424-8220 |
| language | English |
| last_indexed | 2024-03-10T06:51:28Z |
| publishDate | 2021-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Sensors |
| spelling | doaj.art-bf71d20ae35e4f95a2d1fcc487956e782023-11-22T16:47:24ZengMDPI AGSensors1424-82202021-09-012119652310.3390/s21196523Data-Efficient Sensor Upgrade Path Using Knowledge DistillationPieter Van Molle0Cedric De Boom1Tim Verbelen2Bert Vankeirsbilck3Jonas De Vylder4Bart Diricx5Pieter Simoens6Bart Dhoedt7IDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumIDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumIDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumIDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumBarco Healthcare, Barco N.V., 8500 Kortrijk, BelgiumBarco Healthcare, Barco N.V., 8500 Kortrijk, BelgiumIDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumIDLab, Department of Information and Technology, Ghent University, 9052 Gent, BelgiumDeep neural networks have achieved state-of-the-art performance in image classification. Due to this success, deep learning is now also being applied to other data modalities such as multispectral images, lidar and radar data. However, successfully training a deep neural network requires a large reddataset. Therefore, transitioning to a new sensor modality (e.g., from regular camera images to multispectral camera images) might result in a drop in performance, due to the limited availability of data in the new modality. This might hinder the adoption rate and time to market for new sensor technologies. In this paper, we present an approach to leverage the knowledge of a teacher network, that was trained using the original data modality, to improve the performance of a student network on a new data modality: a technique known in literature as knowledge distillation. By applying knowledge distillation to the problem of sensor transition, we can greatly speed up this process. We validate this approach using a multimodal version of the MNIST dataset. Especially when little data is available in the new modality (i.e., 10 images), training with additional teacher supervision results in increased performance, with the student network scoring a test set accuracy of 0.77, compared to an accuracy of 0.37 for the baseline. We also explore two extensions to the default method of knowledge distillation, which we evaluate on a multimodal version of the CIFAR-10 dataset: an annealing scheme for the hyperparameter <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>α</mi></semantics></math></inline-formula> and selective knowledge distillation. Of these two, the first yields the best results. Choosing the optimal annealing scheme results in an increase in test set accuracy of 6%. Finally, we apply our method to the real-world use case of skin lesion classification.https://www.mdpi.com/1424-8220/21/19/6523deep learningknowledge distillationcross-modal distillationsensor upgradeskin lesion classificationmultispectral imaging |
| spellingShingle | Pieter Van Molle Cedric De Boom Tim Verbelen Bert Vankeirsbilck Jonas De Vylder Bart Diricx Pieter Simoens Bart Dhoedt Data-Efficient Sensor Upgrade Path Using Knowledge Distillation Sensors deep learning knowledge distillation cross-modal distillation sensor upgrade skin lesion classification multispectral imaging |
| title | Data-Efficient Sensor Upgrade Path Using Knowledge Distillation |
| title_full | Data-Efficient Sensor Upgrade Path Using Knowledge Distillation |
| title_fullStr | Data-Efficient Sensor Upgrade Path Using Knowledge Distillation |
| title_full_unstemmed | Data-Efficient Sensor Upgrade Path Using Knowledge Distillation |
| title_short | Data-Efficient Sensor Upgrade Path Using Knowledge Distillation |
| title_sort | data efficient sensor upgrade path using knowledge distillation |
| topic | deep learning knowledge distillation cross-modal distillation sensor upgrade skin lesion classification multispectral imaging |
| url | https://www.mdpi.com/1424-8220/21/19/6523 |
| work_keys_str_mv | AT pietervanmolle dataefficientsensorupgradepathusingknowledgedistillation AT cedricdeboom dataefficientsensorupgradepathusingknowledgedistillation AT timverbelen dataefficientsensorupgradepathusingknowledgedistillation AT bertvankeirsbilck dataefficientsensorupgradepathusingknowledgedistillation AT jonasdevylder dataefficientsensorupgradepathusingknowledgedistillation AT bartdiricx dataefficientsensorupgradepathusingknowledgedistillation AT pietersimoens dataefficientsensorupgradepathusingknowledgedistillation AT bartdhoedt dataefficientsensorupgradepathusingknowledgedistillation |