Model pruning based on filter similarity for edge device deployment
Filter pruning is widely used for inference acceleration and compatibility with off-the-shelf hardware devices. Some filter pruning methods have proposed various criteria to approximate the importance of filters, and then sort the filters globally or locally to prune the redundant parameters. Howeve...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2023-03-01
|
| Series: | Frontiers in Neurorobotics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1132679/full |
| _version_ | 1828000916707475456 |
|---|---|
| author | Tingting Wu Tingting Wu Tingting Wu Tingting Wu Chunhe Song Chunhe Song Chunhe Song Peng Zeng Peng Zeng Peng Zeng |
| author_facet | Tingting Wu Tingting Wu Tingting Wu Tingting Wu Chunhe Song Chunhe Song Chunhe Song Peng Zeng Peng Zeng Peng Zeng |
| author_sort | Tingting Wu |
| collection | DOAJ |
| description | Filter pruning is widely used for inference acceleration and compatibility with off-the-shelf hardware devices. Some filter pruning methods have proposed various criteria to approximate the importance of filters, and then sort the filters globally or locally to prune the redundant parameters. However, the current criterion-based methods have problems: (1) parameters with smaller criterion values for extracting edge features are easily ignored, and (2) there is a strong correlation between different criteria, resulting in similar pruning structures. In this article, we propose a novel simple but effective pruning method based on filter similarity, which is used to evaluate the similarity between filters instead of the importance of a single filter. The proposed method first calculates the similarity of the filters pairwise in one convolutional layer and then obtains the similarity distribution. Finally, the filters with high similarity to others are deleted from the distribution or set to zero. In addition, the proposed algorithm does not need to specify the pruning rate for each layer, and only needs to set the desired FLOPs or parameter reduction to obtain the final compression model. We also provide iterative pruning strategies for hard pruning and soft pruning to satisfy the tradeoff requirements of accuracy and memory in different scenarios. Extensive experiments on various representative benchmark datasets across different network architectures demonstrate the effectiveness of our proposed method. For example, on CIFAR10, the proposed algorithm achieves 61.1% FLOPs reduction by removing 58.3% of the parameters, with no loss in Top-1 accuracy on ResNet-56; and reduces 53.05% FLOPs on ResNet-50 with only 0.29% Top-1 accuracy degradation on ILSVRC-2012. |
| first_indexed | 2024-04-10T06:21:31Z |
| format | Article |
| id | doaj.art-4d56bef422124d9ea0f13c76238e66de |
| institution | Directory Open Access Journal |
| issn | 1662-5218 |
| language | English |
| last_indexed | 2024-04-10T06:21:31Z |
| publishDate | 2023-03-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neurorobotics |
| spelling | doaj.art-4d56bef422124d9ea0f13c76238e66de2023-03-02T04:55:06ZengFrontiers Media S.A.Frontiers in Neurorobotics1662-52182023-03-011710.3389/fnbot.2023.11326791132679Model pruning based on filter similarity for edge device deploymentTingting Wu0Tingting Wu1Tingting Wu2Tingting Wu3Chunhe Song4Chunhe Song5Chunhe Song6Peng Zeng7Peng Zeng8Peng Zeng9State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, ChinaKey Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang, ChinaInstitutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, ChinaUniversity of Chinese Academy of Sciences, Beijing, ChinaState Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, ChinaKey Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang, ChinaInstitutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, ChinaState Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, ChinaKey Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang, ChinaInstitutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, ChinaFilter pruning is widely used for inference acceleration and compatibility with off-the-shelf hardware devices. Some filter pruning methods have proposed various criteria to approximate the importance of filters, and then sort the filters globally or locally to prune the redundant parameters. However, the current criterion-based methods have problems: (1) parameters with smaller criterion values for extracting edge features are easily ignored, and (2) there is a strong correlation between different criteria, resulting in similar pruning structures. In this article, we propose a novel simple but effective pruning method based on filter similarity, which is used to evaluate the similarity between filters instead of the importance of a single filter. The proposed method first calculates the similarity of the filters pairwise in one convolutional layer and then obtains the similarity distribution. Finally, the filters with high similarity to others are deleted from the distribution or set to zero. In addition, the proposed algorithm does not need to specify the pruning rate for each layer, and only needs to set the desired FLOPs or parameter reduction to obtain the final compression model. We also provide iterative pruning strategies for hard pruning and soft pruning to satisfy the tradeoff requirements of accuracy and memory in different scenarios. Extensive experiments on various representative benchmark datasets across different network architectures demonstrate the effectiveness of our proposed method. For example, on CIFAR10, the proposed algorithm achieves 61.1% FLOPs reduction by removing 58.3% of the parameters, with no loss in Top-1 accuracy on ResNet-56; and reduces 53.05% FLOPs on ResNet-50 with only 0.29% Top-1 accuracy degradation on ILSVRC-2012.https://www.frontiersin.org/articles/10.3389/fnbot.2023.1132679/fullnetwork accelerationfilter pruningedge intelligencenetwork compressionconvolutional neural networks |
| spellingShingle | Tingting Wu Tingting Wu Tingting Wu Tingting Wu Chunhe Song Chunhe Song Chunhe Song Peng Zeng Peng Zeng Peng Zeng Model pruning based on filter similarity for edge device deployment Frontiers in Neurorobotics network acceleration filter pruning edge intelligence network compression convolutional neural networks |
| title | Model pruning based on filter similarity for edge device deployment |
| title_full | Model pruning based on filter similarity for edge device deployment |
| title_fullStr | Model pruning based on filter similarity for edge device deployment |
| title_full_unstemmed | Model pruning based on filter similarity for edge device deployment |
| title_short | Model pruning based on filter similarity for edge device deployment |
| title_sort | model pruning based on filter similarity for edge device deployment |
| topic | network acceleration filter pruning edge intelligence network compression convolutional neural networks |
| url | https://www.frontiersin.org/articles/10.3389/fnbot.2023.1132679/full |
| work_keys_str_mv | AT tingtingwu modelpruningbasedonfiltersimilarityforedgedevicedeployment AT tingtingwu modelpruningbasedonfiltersimilarityforedgedevicedeployment AT tingtingwu modelpruningbasedonfiltersimilarityforedgedevicedeployment AT tingtingwu modelpruningbasedonfiltersimilarityforedgedevicedeployment AT chunhesong modelpruningbasedonfiltersimilarityforedgedevicedeployment AT chunhesong modelpruningbasedonfiltersimilarityforedgedevicedeployment AT chunhesong modelpruningbasedonfiltersimilarityforedgedevicedeployment AT pengzeng modelpruningbasedonfiltersimilarityforedgedevicedeployment AT pengzeng modelpruningbasedonfiltersimilarityforedgedevicedeployment AT pengzeng modelpruningbasedonfiltersimilarityforedgedevicedeployment |