Compressing medical deep neural network models for edge devices using knowledge distillation
Recently, deep neural networks (DNNs) have been used successfully in many fields, particularly, in medical diagnosis. However, deep learning (DL) models are expensive in terms of memory and computing resources, which hinders their implementation in limited-resources devices or for delay-sensitive sy...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2023-07-01
|
| Series: | Journal of King Saud University: Computer and Information Sciences |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S1319157823001702 |
| _version_ | 1827871836191326208 |
|---|---|
| author | F. MohiEldeen Alabbasy A.S. Abohamama Mohammed F. Alrahmawy |
| author_facet | F. MohiEldeen Alabbasy A.S. Abohamama Mohammed F. Alrahmawy |
| author_sort | F. MohiEldeen Alabbasy |
| collection | DOAJ |
| description | Recently, deep neural networks (DNNs) have been used successfully in many fields, particularly, in medical diagnosis. However, deep learning (DL) models are expensive in terms of memory and computing resources, which hinders their implementation in limited-resources devices or for delay-sensitive systems. Therefore, these deep models need to be accelerated and compressed to smaller sizes to be deployed on edge devices without noticeably affecting their performance. In this paper, recent accelerating and compression approaches of DNN are analyzed and compared regarding their performance, applications, benefits, and limitations with a more focus on the knowledge distillation approach as a successful emergent approach in this field. In addition, a framework is proposed to develop knowledge distilled DNN models that can be deployed on fog/edge devices for automatic disease diagnosis. To evaluate the proposed framework, two compressed medical diagnosis systems are proposed based on knowledge distillation deep neural models for both COVID-19 and Malaria. The experimental results show that these knowledge distilled models have been compressed by 18.4% and 15% of the original model and their responses accelerated by 6.14x and 5.86%, respectively, while there were no significant drop in their performance (dropped by 0.9% and 1.2%, respectively). Furthermore, the distilled models are compared with other pruned and quantized models. The obtained results revealed the superiority of the distilled models in terms of compression rates and response time. |
| first_indexed | 2024-03-12T16:15:40Z |
| format | Article |
| id | doaj.art-92f82407e206426c8d5b0f4bcb9baa8b |
| institution | Directory Open Access Journal |
| issn | 1319-1578 |
| language | English |
| last_indexed | 2024-03-12T16:15:40Z |
| publishDate | 2023-07-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Journal of King Saud University: Computer and Information Sciences |
| spelling | doaj.art-92f82407e206426c8d5b0f4bcb9baa8b2023-08-09T04:32:05ZengElsevierJournal of King Saud University: Computer and Information Sciences1319-15782023-07-01357101616Compressing medical deep neural network models for edge devices using knowledge distillationF. MohiEldeen Alabbasy0A.S. Abohamama1Mohammed F. Alrahmawy2Department of Computer Science, Faculty of Computers and Information, Mansoura University, Egypt; Corresponding author.Department of Computer Science, Faculty of Computers and Information, Mansoura University, Egypt; Department of Computer Science, Arab East Colleges, Saudi ArabiaDepartment of Computer Science, Faculty of Computers and Information, Mansoura University, EgyptRecently, deep neural networks (DNNs) have been used successfully in many fields, particularly, in medical diagnosis. However, deep learning (DL) models are expensive in terms of memory and computing resources, which hinders their implementation in limited-resources devices or for delay-sensitive systems. Therefore, these deep models need to be accelerated and compressed to smaller sizes to be deployed on edge devices without noticeably affecting their performance. In this paper, recent accelerating and compression approaches of DNN are analyzed and compared regarding their performance, applications, benefits, and limitations with a more focus on the knowledge distillation approach as a successful emergent approach in this field. In addition, a framework is proposed to develop knowledge distilled DNN models that can be deployed on fog/edge devices for automatic disease diagnosis. To evaluate the proposed framework, two compressed medical diagnosis systems are proposed based on knowledge distillation deep neural models for both COVID-19 and Malaria. The experimental results show that these knowledge distilled models have been compressed by 18.4% and 15% of the original model and their responses accelerated by 6.14x and 5.86%, respectively, while there were no significant drop in their performance (dropped by 0.9% and 1.2%, respectively). Furthermore, the distilled models are compared with other pruned and quantized models. The obtained results revealed the superiority of the distilled models in terms of compression rates and response time.http://www.sciencedirect.com/science/article/pii/S1319157823001702Knowledge distillationDeep modelsEdge devicesDeep models' compressing techniques |
| spellingShingle | F. MohiEldeen Alabbasy A.S. Abohamama Mohammed F. Alrahmawy Compressing medical deep neural network models for edge devices using knowledge distillation Journal of King Saud University: Computer and Information Sciences Knowledge distillation Deep models Edge devices Deep models' compressing techniques |
| title | Compressing medical deep neural network models for edge devices using knowledge distillation |
| title_full | Compressing medical deep neural network models for edge devices using knowledge distillation |
| title_fullStr | Compressing medical deep neural network models for edge devices using knowledge distillation |
| title_full_unstemmed | Compressing medical deep neural network models for edge devices using knowledge distillation |
| title_short | Compressing medical deep neural network models for edge devices using knowledge distillation |
| title_sort | compressing medical deep neural network models for edge devices using knowledge distillation |
| topic | Knowledge distillation Deep models Edge devices Deep models' compressing techniques |
| url | http://www.sciencedirect.com/science/article/pii/S1319157823001702 |
| work_keys_str_mv | AT fmohieldeenalabbasy compressingmedicaldeepneuralnetworkmodelsforedgedevicesusingknowledgedistillation AT asabohamama compressingmedicaldeepneuralnetworkmodelsforedgedevicesusingknowledgedistillation AT mohammedfalrahmawy compressingmedicaldeepneuralnetworkmodelsforedgedevicesusingknowledgedistillation |