Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition
Knowledge distillation, which is a process of transferring complex knowledge learned by a heavy network, i.e., a teacher, to a lightweight network, i.e., a student, has emerged as an effective technique for compressing neural networks. To reduce the necessity of training a large teacher network, thi...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9495804/ |
| _version_ | 1818694031760162816 |
|---|---|
| author | Duc-Quang Vu Ngan Le Jia-Ching Wang |
| author_facet | Duc-Quang Vu Ngan Le Jia-Ching Wang |
| author_sort | Duc-Quang Vu |
| collection | DOAJ |
| description | Knowledge distillation, which is a process of transferring complex knowledge learned by a heavy network, i.e., a teacher, to a lightweight network, i.e., a student, has emerged as an effective technique for compressing neural networks. To reduce the necessity of training a large teacher network, this paper leverages the recent self-knowledge distillation approach to train a student network progressively by distilling its own knowledge without a pre-trained teacher network. Far from the existing self-knowledge distillation methods, which mainly focus on still images, our proposed Teaching Yourself is a self-knowledge distillation technique that targets at videos for human action recognition. Our proposed Teaching Yourself is not only designed as an effective lightweight network but also a high generalization capability model. In our approach, the network is able to update itself using the best past model, termed the preceding model, which is then utilized to guide the training process to update the present model. Inspired by consistency training in state-of-the-art semi-supervised learning methods, we also introduce an effective augmentation strategy to increase data diversity and improve network generalization and consistent predictions for our proposed Teaching Yourself approach. Our benchmark has been conducted on both the 3D Resnet-18 and 3D ResNet-50 backbone networks and evaluated on various standard datasets such as UCF101, HMDB51, and Kinetics400 datasets. The experimental results have shown that our teaching yourself method significantly improves the action recognition performance in terms of accuracy compared to existing supervised learning and knowledge distillation methods. We also have conducted an expensive ablation study to demonstrate that our approach mitigates overconfident predictions on dark knowledge and generates more consistent predictions in input variations of the same data point. The code is available at <uri>https://github.com/vdquang1991/Self-KD</uri>. |
| first_indexed | 2024-12-17T13:23:07Z |
| format | Article |
| id | doaj.art-20ba579f6abd49d8b045f9e4cc4d1136 |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-17T13:23:07Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-20ba579f6abd49d8b045f9e4cc4d11362022-12-21T21:46:48ZengIEEEIEEE Access2169-35362021-01-01910571110572310.1109/ACCESS.2021.30998569495804Teaching Yourself: A Self-Knowledge Distillation Approach to Action RecognitionDuc-Quang Vu0https://orcid.org/0000-0001-5458-3713Ngan Le1Jia-Ching Wang2https://orcid.org/0000-0003-0024-6732Department of Computer Science and Information Engineering, National Central University, Taoyuan, TaiwanDepartment of Computer Science and Computer Engineering, University of Arkansas, Fayetteville, AR, USADepartment of Computer Science and Information Engineering, National Central University, Taoyuan, TaiwanKnowledge distillation, which is a process of transferring complex knowledge learned by a heavy network, i.e., a teacher, to a lightweight network, i.e., a student, has emerged as an effective technique for compressing neural networks. To reduce the necessity of training a large teacher network, this paper leverages the recent self-knowledge distillation approach to train a student network progressively by distilling its own knowledge without a pre-trained teacher network. Far from the existing self-knowledge distillation methods, which mainly focus on still images, our proposed Teaching Yourself is a self-knowledge distillation technique that targets at videos for human action recognition. Our proposed Teaching Yourself is not only designed as an effective lightweight network but also a high generalization capability model. In our approach, the network is able to update itself using the best past model, termed the preceding model, which is then utilized to guide the training process to update the present model. Inspired by consistency training in state-of-the-art semi-supervised learning methods, we also introduce an effective augmentation strategy to increase data diversity and improve network generalization and consistent predictions for our proposed Teaching Yourself approach. Our benchmark has been conducted on both the 3D Resnet-18 and 3D ResNet-50 backbone networks and evaluated on various standard datasets such as UCF101, HMDB51, and Kinetics400 datasets. The experimental results have shown that our teaching yourself method significantly improves the action recognition performance in terms of accuracy compared to existing supervised learning and knowledge distillation methods. We also have conducted an expensive ablation study to demonstrate that our approach mitigates overconfident predictions on dark knowledge and generates more consistent predictions in input variations of the same data point. The code is available at <uri>https://github.com/vdquang1991/Self-KD</uri>.https://ieeexplore.ieee.org/document/9495804/Self-knowledge distillationself-learningknowledge distillationaction recognitiondeep learningconvolutional neural network |
| spellingShingle | Duc-Quang Vu Ngan Le Jia-Ching Wang Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition IEEE Access Self-knowledge distillation self-learning knowledge distillation action recognition deep learning convolutional neural network |
| title | Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition |
| title_full | Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition |
| title_fullStr | Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition |
| title_full_unstemmed | Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition |
| title_short | Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition |
| title_sort | teaching yourself a self knowledge distillation approach to action recognition |
| topic | Self-knowledge distillation self-learning knowledge distillation action recognition deep learning convolutional neural network |
| url | https://ieeexplore.ieee.org/document/9495804/ |
| work_keys_str_mv | AT ducquangvu teachingyourselfaselfknowledgedistillationapproachtoactionrecognition AT nganle teachingyourselfaselfknowledgedistillationapproachtoactionrecognition AT jiachingwang teachingyourselfaselfknowledgedistillationapproachtoactionrecognition |