Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification
A deep-learning technology for knowledge transfer is necessary to advance and optimize efficient knowledge distillation. Here, we aim to develop a new adversarial optimization-based knowledge transfer method involved with a layer-wise dense flow that is distilled from a pre-trained deep neural netwo...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-04-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/11/8/3720 |
| _version_ | 1797536983295197184 |
|---|---|
| author | Doyeob Yeo Min-Suk Kim Ji-Hoon Bae |
| author_facet | Doyeob Yeo Min-Suk Kim Ji-Hoon Bae |
| author_sort | Doyeob Yeo |
| collection | DOAJ |
| description | A deep-learning technology for knowledge transfer is necessary to advance and optimize efficient knowledge distillation. Here, we aim to develop a new adversarial optimization-based knowledge transfer method involved with a layer-wise dense flow that is distilled from a pre-trained deep neural network (DNN). Knowledge distillation transferred to another target DNN based on adversarial loss functions has multiple flow-based knowledge items that are densely extracted by overlapping them from a pre-trained DNN to enhance the existing knowledge. We propose a semi-supervised learning-based knowledge transfer with multiple items of dense flow-based knowledge extracted from the pre-trained DNN. The proposed loss function would comprise a supervised cross-entropy loss for a typical classification, an adversarial training loss for the target DNN and discriminators, and Euclidean distance-based loss in terms of dense flow. For both pre-trained and target DNNs considered in this study, we adopt a residual network (ResNet) architecture. We propose methods of (1) the adversarial-based knowledge optimization, (2) the extended and flow-based knowledge transfer scheme, and (3) the combined layer-wise dense flow in an adversarial network. The results show that it provides higher accuracy performance in the improved target ResNet compared to the prior knowledge transfer methods. |
| first_indexed | 2024-03-10T12:08:36Z |
| format | Article |
| id | doaj.art-6b205d99bc574718b0580d124625c467 |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T12:08:36Z |
| publishDate | 2021-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-6b205d99bc574718b0580d124625c4672023-11-21T16:22:34ZengMDPI AGApplied Sciences2076-34172021-04-01118372010.3390/app11083720Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image ClassificationDoyeob Yeo0Min-Suk Kim1Ji-Hoon Bae2KSB Convergence Research Department, Electronics and Telecommunications Research Institute, Daejeon 34129, KoreaDepartment of Human Intelligence and Robot Engineering, Sangmyung University, Cheonan 03016, KoreaDepartment of AI and Big Data Engineering, Daegu Catholic University, Gyeongsan-si 38430, KoreaA deep-learning technology for knowledge transfer is necessary to advance and optimize efficient knowledge distillation. Here, we aim to develop a new adversarial optimization-based knowledge transfer method involved with a layer-wise dense flow that is distilled from a pre-trained deep neural network (DNN). Knowledge distillation transferred to another target DNN based on adversarial loss functions has multiple flow-based knowledge items that are densely extracted by overlapping them from a pre-trained DNN to enhance the existing knowledge. We propose a semi-supervised learning-based knowledge transfer with multiple items of dense flow-based knowledge extracted from the pre-trained DNN. The proposed loss function would comprise a supervised cross-entropy loss for a typical classification, an adversarial training loss for the target DNN and discriminators, and Euclidean distance-based loss in terms of dense flow. For both pre-trained and target DNNs considered in this study, we adopt a residual network (ResNet) architecture. We propose methods of (1) the adversarial-based knowledge optimization, (2) the extended and flow-based knowledge transfer scheme, and (3) the combined layer-wise dense flow in an adversarial network. The results show that it provides higher accuracy performance in the improved target ResNet compared to the prior knowledge transfer methods.https://www.mdpi.com/2076-3417/11/8/3720adversarial optimizationlayer-wise dense flowknowledge transferimage classification |
| spellingShingle | Doyeob Yeo Min-Suk Kim Ji-Hoon Bae Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification Applied Sciences adversarial optimization layer-wise dense flow knowledge transfer image classification |
| title | Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification |
| title_full | Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification |
| title_fullStr | Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification |
| title_full_unstemmed | Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification |
| title_short | Adversarial Optimization-Based Knowledge Transfer of Layer-Wise Dense Flow for Image Classification |
| title_sort | adversarial optimization based knowledge transfer of layer wise dense flow for image classification |
| topic | adversarial optimization layer-wise dense flow knowledge transfer image classification |
| url | https://www.mdpi.com/2076-3417/11/8/3720 |
| work_keys_str_mv | AT doyeobyeo adversarialoptimizationbasedknowledgetransferoflayerwisedenseflowforimageclassification AT minsukkim adversarialoptimizationbasedknowledgetransferoflayerwisedenseflowforimageclassification AT jihoonbae adversarialoptimizationbasedknowledgetransferoflayerwisedenseflowforimageclassification |