Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers
Pruning and compression of models are practical approaches for deploying and applying deep convolutional neural networks in scenarios with limited memory and computational resources. To mitigate the impact of pruning on model accuracy and enhance the stability of pruning (defined as the negligible d...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10485421/ |
| _version_ | 1827290005959081984 |
|---|---|
| author | Zijie Qiu Peng Wei Mingwei Yao Rui Zhang Yingchun Kuang |
| author_facet | Zijie Qiu Peng Wei Mingwei Yao Rui Zhang Yingchun Kuang |
| author_sort | Zijie Qiu |
| collection | DOAJ |
| description | Pruning and compression of models are practical approaches for deploying and applying deep convolutional neural networks in scenarios with limited memory and computational resources. To mitigate the impact of pruning on model accuracy and enhance the stability of pruning (defined as the negligible drop in test accuracy immediately following pruning), an algorithm for reward-penalty decoupling is introduced in this study to achieve automated sparse training and channel pruning. During sparse training, the influence of unimportant channels is automatically identified and reduced, thereby preserving the ability of the important channels for feature recognition. First, by utilizing the gradient information learned through network backpropagation, the feature scaling factors of the batch normalization layers are combined with the gradient to determine the importance threshold for the network channels. Subsequently, a two-stage sparse training algorithm is proposed based on the reward-penalty decoupling strategy, applying different loss function strategies to the feature scaling factors of “important” and “unimportant” channels during decoupled sparse training. This approach has been experimentally validated across various tasks, baselines, and datasets, demonstrating its superiority over the previous state-of-the-art methods. The results indicate that the effect of pruning on model accuracy is significantly alleviated by the proposed method, and pruned models require only limited fine-tuning to achieve excellent performance. |
| first_indexed | 2024-04-24T12:00:45Z |
| format | Article |
| id | doaj.art-52040c5821b24e7f8813cbc13524834b |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-24T12:00:45Z |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-52040c5821b24e7f8813cbc13524834b2024-04-08T23:00:35ZengIEEEIEEE Access2169-35362024-01-0112488654888010.1109/ACCESS.2024.338299410485421Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization LayersZijie Qiu0https://orcid.org/0009-0007-2461-2641Peng Wei1Mingwei Yao2Rui Zhang3Yingchun Kuang4https://orcid.org/0009-0007-6839-9037College of Information and Intelligence, Hunan Agricultural University, Changsha, ChinaCollege of Information and Intelligence, Hunan Agricultural University, Changsha, ChinaCollege of Information and Intelligence, Hunan Agricultural University, Changsha, ChinaCollege of Information and Intelligence, Hunan Agricultural University, Changsha, ChinaCollege of Information and Intelligence, Hunan Agricultural University, Changsha, ChinaPruning and compression of models are practical approaches for deploying and applying deep convolutional neural networks in scenarios with limited memory and computational resources. To mitigate the impact of pruning on model accuracy and enhance the stability of pruning (defined as the negligible drop in test accuracy immediately following pruning), an algorithm for reward-penalty decoupling is introduced in this study to achieve automated sparse training and channel pruning. During sparse training, the influence of unimportant channels is automatically identified and reduced, thereby preserving the ability of the important channels for feature recognition. First, by utilizing the gradient information learned through network backpropagation, the feature scaling factors of the batch normalization layers are combined with the gradient to determine the importance threshold for the network channels. Subsequently, a two-stage sparse training algorithm is proposed based on the reward-penalty decoupling strategy, applying different loss function strategies to the feature scaling factors of “important” and “unimportant” channels during decoupled sparse training. This approach has been experimentally validated across various tasks, baselines, and datasets, demonstrating its superiority over the previous state-of-the-art methods. The results indicate that the effect of pruning on model accuracy is significantly alleviated by the proposed method, and pruned models require only limited fine-tuning to achieve excellent performance.https://ieeexplore.ieee.org/document/10485421/Neural networkstructured pruningmodel compressionneural network lightweightingautomatic pruningpruning stability |
| spellingShingle | Zijie Qiu Peng Wei Mingwei Yao Rui Zhang Yingchun Kuang Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers IEEE Access Neural network structured pruning model compression neural network lightweighting automatic pruning pruning stability |
| title | Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers |
| title_full | Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers |
| title_fullStr | Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers |
| title_full_unstemmed | Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers |
| title_short | Channel Pruning Method Based on Decoupling Feature Scale Distribution in Batch Normalization Layers |
| title_sort | channel pruning method based on decoupling feature scale distribution in batch normalization layers |
| topic | Neural network structured pruning model compression neural network lightweighting automatic pruning pruning stability |
| url | https://ieeexplore.ieee.org/document/10485421/ |
| work_keys_str_mv | AT zijieqiu channelpruningmethodbasedondecouplingfeaturescaledistributioninbatchnormalizationlayers AT pengwei channelpruningmethodbasedondecouplingfeaturescaledistributioninbatchnormalizationlayers AT mingweiyao channelpruningmethodbasedondecouplingfeaturescaledistributioninbatchnormalizationlayers AT ruizhang channelpruningmethodbasedondecouplingfeaturescaledistributioninbatchnormalizationlayers AT yingchunkuang channelpruningmethodbasedondecouplingfeaturescaledistributioninbatchnormalizationlayers |