Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation
In this paper, we propose an online ensemble distillation (OED) method to automatically prune blocks/layers of a target network by transferring the knowledge from a strong teacher in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of each block in the tar...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2019-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/8918410/ |
| _version_ | 1818323650493808640 |
|---|---|
| author | Zongyue Wang Shaohui Lin Jiao Xie Yangbin Lin |
| author_facet | Zongyue Wang Shaohui Lin Jiao Xie Yangbin Lin |
| author_sort | Zongyue Wang |
| collection | DOAJ |
| description | In this paper, we propose an online ensemble distillation (OED) method to automatically prune blocks/layers of a target network by transferring the knowledge from a strong teacher in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of each block in the target network and enforce the sparsity of the mask by sparsity regularization. Then, a strong teacher network is constructed online by replicating the same target networks and ensembling the discriminative features from each target as its new features. Cooperative learning between multiple target networks and the teacher network is further conducted in a closed-loop form, which improves their performance. To solve the optimization problem in an end-to-end manner, we employ the fast iterative shrinkage-thresholding algorithm to fast and reliably remove the redundant blocks, in which the corresponding soft masks are equal to zero. Compared to other structured pruning methods with iterative fine-tuning, the proposed OED is trained more efficiently in one training cycle. Extensive experiments demonstrate the effectiveness of OED, which can not only simultaneously compress and accelerate a variety of CNN architectures but also enhance the robustness of the pruned networks. |
| first_indexed | 2024-12-13T11:16:04Z |
| format | Article |
| id | doaj.art-249a7e0bc00b4ab193de0457bf30b1dd |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-13T11:16:04Z |
| publishDate | 2019-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-249a7e0bc00b4ab193de0457bf30b1dd2022-12-21T23:48:37ZengIEEEIEEE Access2169-35362019-01-01717570317571610.1109/ACCESS.2019.29572038918410Pruning Blocks for CNN Compression and Acceleration via Online Ensemble DistillationZongyue Wang0https://orcid.org/0000-0003-2409-7065Shaohui Lin1https://orcid.org/0000-0003-0284-9940Jiao Xie2https://orcid.org/0000-0002-1488-2118Yangbin Lin3https://orcid.org/0000-0003-3407-4756Computer Engineering College, Jimei University, Xiamen, ChinaDepartment of Computer Science, National University of Singapore, SingaporeDepartment of Automation, Xiamen University, Xiamen, ChinaComputer Engineering College, Jimei University, Xiamen, ChinaIn this paper, we propose an online ensemble distillation (OED) method to automatically prune blocks/layers of a target network by transferring the knowledge from a strong teacher in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of each block in the target network and enforce the sparsity of the mask by sparsity regularization. Then, a strong teacher network is constructed online by replicating the same target networks and ensembling the discriminative features from each target as its new features. Cooperative learning between multiple target networks and the teacher network is further conducted in a closed-loop form, which improves their performance. To solve the optimization problem in an end-to-end manner, we employ the fast iterative shrinkage-thresholding algorithm to fast and reliably remove the redundant blocks, in which the corresponding soft masks are equal to zero. Compared to other structured pruning methods with iterative fine-tuning, the proposed OED is trained more efficiently in one training cycle. Extensive experiments demonstrate the effectiveness of OED, which can not only simultaneously compress and accelerate a variety of CNN architectures but also enhance the robustness of the pruned networks.https://ieeexplore.ieee.org/document/8918410/Fast iterative shrinkage-thresholding algorithmmodel compression and accelerationnetwork pruningonline ensemble distillation |
| spellingShingle | Zongyue Wang Shaohui Lin Jiao Xie Yangbin Lin Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation IEEE Access Fast iterative shrinkage-thresholding algorithm model compression and acceleration network pruning online ensemble distillation |
| title | Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation |
| title_full | Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation |
| title_fullStr | Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation |
| title_full_unstemmed | Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation |
| title_short | Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation |
| title_sort | pruning blocks for cnn compression and acceleration via online ensemble distillation |
| topic | Fast iterative shrinkage-thresholding algorithm model compression and acceleration network pruning online ensemble distillation |
| url | https://ieeexplore.ieee.org/document/8918410/ |
| work_keys_str_mv | AT zongyuewang pruningblocksforcnncompressionandaccelerationviaonlineensembledistillation AT shaohuilin pruningblocksforcnncompressionandaccelerationviaonlineensembledistillation AT jiaoxie pruningblocksforcnncompressionandaccelerationviaonlineensembledistillation AT yangbinlin pruningblocksforcnncompressionandaccelerationviaonlineensembledistillation |